pax_global_header00006660000000000000000000000064147665014140014522gustar00rootroot0000000000000052 comment=1ea32e8cb61b44f1e64de34ee9cb723787956a79 pengcheng.mind-mindformer/000077500000000000000000000000001476650141400161125ustar00rootroot00000000000000pengcheng.mind-mindformer/README.md000077500000000000000000000200131476650141400173700ustar00rootroot00000000000000# 基于MindFormers在云脑二上训练脑海模型 ## 一、介绍 MindSpore Transformers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件： 我们在此版本的MindFormers套件上做修改，形成了脑海模型的下一迭代版本，目前2.6B和7B版本模型已完成全流程的训练，具体训练配置如下： 其他相关配置的需求请查看MindFormers套件的使用说明[MindFormers](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/src/branch/master/README_mindformers.md) ## 二、版本匹配关系 | 版本对应关系 | MindSpore | CANN | 芯片 | | :----------: |:---------:|:-----:| :---------: | | 版本号 | 2.2 | 7.1.0 | Ascend 910A | #### 脑海V4版本，2.6B模型请使用master分支代码，7B模型请使用7B_shared-embed分支代码 ## 三、模型训练 ### 基于modelarts启动训练任务的配置 | modelarts配置参数 | 配置举例 | 参数说明 | |:-----------------:|:-----------------------------------------------------:|:--------------------------------------------------| | 启动文件 | /PengCheng.Mind-mindformer/run_mindformer.py | 训练任务启动文件 | | train_dataset_dir | obs://path/to/dataset/train/ | train数据集在obs存储的位置 | | eval_dataset_dir | obs://path/to/dataset/eval/ | eval数据集在obs存储的位置 | | config | configs/official_training/run_minicpm_ckpt-2B-v4.yaml | 与训练相关的配置文件，SFT一般数据集循环训练3遍 | | remote_save_url | obs://path/to/save/ckpt/ | 训练相关checkpoinr、日志、strategy文件保存目录 | | do_eval | True | 是否打开eval流程 | | consumed_samples | 0 | 训练启动跳过数据集起始的条目数 | | data_parallel | 256 | 数据并行 | | pipeline_stage | 4 | 流水线并行 | | model_parallel | 1 | 模型并行 | | micro_batch_num | 8 | 批大小 | | optimizer_weight_shard_size | 64 | 优化器并行 | | resume_training | True | 是否加载已有模型继续训练, 默认为加载分片模型方式 | | resume_merged_ckpt | False | 加载整合模型方式启动， 此时restore_ckpt_name_prefix文件名为完整的模型名称 | | load_checkpoint | obs://path/to/pre_train_ckpt/ | 加载的预训练模型路径 | | restore_ckpt_name_prefix | naohai_2b6__rank_*-113809_2.ckpt | 模型名：*代替rank分片 | | total_steps | 1000000 | 总训练步数 | | decay_ratio | 0.001 | 进入WSD的学习率衰减占总步数的比例，SFT时可配置为数据集步数占总训练步数的30%~60% | | learning_rate | 0.0001 | 最高学习率 | | lr_middle | 0.0001 | 中段学习率，这里保持和learning_rate一致就是固定学习率+CosineLR WSD | | lr_end | 0.00005 | WSD下的最低学习率 | 最新版的鹏城·脑海模型还在实验探索阶段，敬请期待 ## 四、模型 | Model | 模型名称 | |:-------------------|:--------------:| | 脑海-v4-2B6 预训练版本 | [naohai_new_2b6_2048B](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/modelmanage/model_filelist_tmpl?name=PengCheng.Mind_2.6B_2048B_tokens) | | 脑海-v4-2B6 SFT版本 | [PengCheng.Mind_2.6B_2048B_tokens_exp09_SFT_124871_2_fp16](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/modelmanage/model_filelist_tmpl?name=PengCheng.Mind_2.6B_2048B_tokens) | | 脑海-v4-7B 预训练版本 | [PC_Mind_7b_2TB-119639_2_filter_adam_fp16](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/modelmanage/model_filelist_tmpl?name=PengCheng.Mind_7B_2048B_tokens_SFT) | | 脑海-v4-7B SFT版本 | [PC_Mind_7B_shared_v4_2Ttoken_sft-123102_2_filter_adam_fp16](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/modelmanage/model_filelist_tmpl?name=PengCheng.Mind_7B_2048B_tokens_SFT) | 多芯片适用，NPU模型转换为HF格式GPU模型： * 2B模型转换脚本: [ms_to_torch](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/src/branch/master/scripts/ms_to_torch.py) * 7B模型转换脚本: [ms_to_torch-shared](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PengCheng.Mind-mindformer/src/branch/master/scripts/ms_to_torch-shared.py) * 一键将NPU模型转换到GPU，转换后模型文件叠加项目“数据集”当中其余的HF模型文件依赖，整合全部文件作为最终HF格式模型转换结果。按照HF模型的使用规范来体验转换后模型，可参考./scripts/hf_2B和./scripts/hf_7B_shared * ms_to_torch-shared脚本里0.ckpt和1.ckpt分别对应为rank_0和rank_1保存的模型分片，项目中自行下载使用 ## 五、说明 在这个项目中，我们对以下问题进行了探索： * 在模型规模确定的情况下（比如200B或7B或2B），确保模型Loss最优或近似最优收敛需要构建多大的预训练数据集？ * 如何构建确定的超大规模高质量预训练数据集？如何定义和定性定量评估数据的质量？如何合成增广富有知识而数量稀缺的数据？ * 在预训练数据集规模确定的情况下（比如2T或4T Token），如何找到确保Loss最优或近似最优收敛需要的最优数据混合策略？ * 在预训练数据集无法一次性准备好的情况下，如何找到既能避免灾难性遗忘又能持续确保Loss最优或近似最优收敛的持续学习最优数据混合策略？ * ... ## 六、评测表现 我们评测的数据集包括MMLU、C-Eval、 GSM8K、 MATH、HumanEval、MBPP、BBH等数据集，考察的能力包括自然语言理解、知识、数学计算和推理、代码生成、逻辑推理等。
| Model | MMLU | C-Eval | GSM8K | MATH | HumanEval | MBPP | BBH | CMMLU | AGIEval | |:-------------------|:--------:|:--------:|:--------:|:--------:|:---------:|:--------:|:--------:|:--------:|:--------:| | 脑海-v4-2B6 2Ttokens | 49.51 | 57.42 | 59.59 | 19.04 | 37.20 | 31.40 | 35.84 | 55.48 |26.53 | | 脑海-v4-2B6 2Ttokens_SFT | 48.93 | 65.83 | 66.94 | 28.18 | 38.41 | 31.2 | 35.79 | 65.58 |33.56 | | 脑海-v4-7B 2Ttokens | 54.95 | 64.95 | 69.07 | 22.48 | 40.24 | 36.20 | 43.72 | 64.70 |32.56 | | 脑海-v4-7B 2Ttokens_SFT | 55.37 | 72.56 | 74.15 | 26.76 | 62.20 | 42.80 | 43.4 | 71.65 |44.03 |

## 七、许可证 [Apache 2.0许可证](LICENSE)pengcheng.mind-mindformer/README_mindformers.md000077500000000000000000000477321476650141400220160ustar00rootroot00000000000000# 欢迎来到MindSpore Transformers（MindFormers） ## 一、介绍 MindSpore Transformers套件的目标是构建一个大模型训练、微调、评估、推理、部署的全流程开发套件： 提供业内主流的Transformer类预训练模型和SOTA下游任务应用，涵盖丰富的并行特性。期望帮助用户轻松的实现大模型训练和创新研发。 MindSpore Transformers套件基于MindSpore内置的并行技术和组件化设计，具备如下特点： - 一行代码实现从单卡到大规模集群训练的无缝切换； - 提供灵活易用的个性化并行配置； - 能够自动进行拓扑感知，高效地融合数据并行和模型并行策略； - 一键启动任意任务的单卡/多卡训练、微调、评估、推理流程； - 支持用户进行组件化配置任意模块，如优化器、学习策略、网络组装等； - 提供Trainer、pipeline、AutoClass等高阶易用性接口； - 提供预置SOTA权重自动下载及加载功能； - 支持人工智能计算中心无缝迁移部署； 如果您对MindSpore Transformers有任何建议，请通过issue与我们联系，我们将及时处理。 - 模型README：[docs/model_cards](https://gitee.com/mindspore/mindformers/tree/dev/docs/model_cards) - 任务README：[docs/task_cards](https://gitee.com/mindspore/mindformers/tree/dev/docs/task_cards) - MindPet指导教程：[docs/pet_tuners](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Pet_Tuners.md) - AICC指导教程：[docs/aicc_cards](https://gitee.com/mindspore/mindformers/blob/dev/docs/aicc_cards/README.md) - 高阶API指导文档: [docs/mindformers_tutorial.md](https://gitee.com/mindspore/mindformers/blob/dev/docs/README.md) - 在线文档：[mindformers](https://mindformers.readthedocs.io/zh_CN/latest) 目前支持的模型列表如下： | 模型 | 任务（task name） | 模型（model name） | | :------------------------------------------: |:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [BERT](docs/model_cards/bert.md) | masked_language_modeling
[text_classification](docs/task_cards/text_classification.md)
[token_classification](docs/task_cards/token_classification.md)
[question_answering](docs/task_cards/question_answering.md) | bert_base_uncased
txtcls_bert_base_uncased
txtcls_bert_base_uncased_mnli
tokcls_bert_base_chinese
tokcls_bert_base_chinese_cluener
qa_bert_base_uncased
qa_bert_base_chinese_uncased | | [T5](docs/model_cards/t5.md) | translation | t5_small | | [GPT2](docs/model_cards/gpt2.md) | [text_generation](docs/task_cards/text_generation.md) | gpt2_small
gpt2_13b
gpt2_52b | | [PanGuAlpha](docs/model_cards/pangualpha.md) | [text_generation](docs/task_cards/text_generation.md) | pangualpha_2_6_b
pangualpha_13b | | [GLM](docs/model_cards/glm.md) | [text_generation](docs/task_cards/text_generation.md) | glm_6b
glm_6b_lora | | [GLM2](docs/model_cards/glm2.md) | [text_generation](docs/task_cards/text_generation.md) | glm2_6b
glm2_6b_lora | | [LLama](docs/model_cards/llama.md) | [text_generation](docs/task_cards/text_generation.md) | llama_7b
llama_13b
llama_65b
llama_7b_lora | | [LLama2](docs/model_cards/llama.md) | [text_generation](docs/task_cards/text_generation.md) | llama2_7b
llama2_13b | | [Bloom](docs/model_cards/bloom.md) | [text_generation](docs/task_cards/text_generation.md) | bloom_560m
bloom_7.1b
bloom_65b
bloom_176b | | [MAE](docs/model_cards/mae.md) | masked_image_modeling | mae_vit_base_p16 | | [VIT](docs/model_cards/vit.md) | [image_classification](docs/task_cards/image_classification.md) | vit_base_p16 | | [Swin](docs/model_cards/swin.md) | [image_classification](docs/task_cards/image_classification.md) | swin_base_p4w7 | | [CLIP](docs/model_cards/clip.md) | [contrastive_language_image_pretrain](docs/task_cards/contrastive_language_image_pretrain.md)
[zero_shot_image_classification](docs/task_cards/zero_shot_image_classification.md) | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | | [BLIP2](docs/model_cards/blip2.md) | [contrastive_language_image_pretrain](docs/task_cards/contrastive_language_image_pretrain.md)
[zero_shot_image_classification](docs/task_cards/zero_shot_image_classification.md) | blip2_stage1_vit_g | 目前在research中支持的模型列表如下： | 模型 | 任务（task name） | 模型（model name） | | :------------------------------------------: | :---------------------------------------------------: | :------------------------------ | | [Baichuan](research/baichuan/baichuan.md) | [text_generation](docs/task_cards/text_generation.md) | baichuan_7b
baichuan_13b | | [Baichuan2](research/baichuan2/baichuan2.md) | [text_generation](docs/task_cards/text_generation.md) | baichuan2_7b
baichuan2_13b | | [Internlm](research/internlm/internlm.md) | [text_generation](docs/task_cards/text_generation.md) | InternLM-7B | | [ziya](research/ziya/ziya.md) | [text_generation](docs/task_cards/text_generation.md) | ziya-13B | ## 二、mindformers安装 - 方式1：源码编译安装 支持源码编译安装，用户可以执行下述的命令进行包的安装 ```bash git clone -b dev https://gitee.com/mindspore/mindformers.git cd mindformers bash build.sh ``` - 方式2：镜像 docker下载命令 ```shell docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_2_0:mindformers_0.6.0dev_20230616_py39_37 ``` 创建容器 ```shell # --device用于控制指定容器的运行NPU卡号和范围 # -v 用于映射容器外的目录 # --name 用于自定义容器名称 docker run -it -u root \ --ipc=host \ --network host \ --device=/dev/davinci0 \ --device=/dev/davinci1 \ --device=/dev/davinci2 \ --device=/dev/davinci3 \ --device=/dev/davinci4 \ --device=/dev/davinci5 \ --device=/dev/davinci6 \ --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/devmm_svm \ --device=/dev/hisi_hdc \ -v /etc/localtime:/etc/localtime \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /var/log/npu/:/usr/slog \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ --name {请手动输入容器名称} \ swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_2_0:mindformers_0.6.0dev_20230616_py39_37 \ /bin/bash ``` ## 三、版本匹配关系 | 版本对应关系 | MindFormers | MindPet | MindSpore | Python | 芯片 | | :----------: | :---------: | :-----: | :-------: | :-------: | :---------: | | 版本号 | dev | 1.0.0 | 2.0/1.10 | 3.7.5/3.9 | Ascend 910A | ## 四、快速使用 MindFormers套件对外提供两种使用和开发形式，为开发者提供灵活且简洁的使用方式和高阶开发接口。 ### 方式一：使用已有脚本启动 用户可以直接clone整个仓库，按照以下步骤即可运行套件中已支持的任意`configs`模型任务配置文件，方便用户快速进行使用和开发： - 准备工作 - step1：git clone mindformers ```shell git clone -b dev https://gitee.com/mindspore/mindformers.git cd mindformers ``` - step2: 准备相应任务的数据集，请参考`docs`目录下各模型的README.md文档准备相应数据集 - step3：修改配置文件`configs/{model_name}/run_{model_name}_***.yaml`中数据集路径 - step4：如果要使用分布式训练，则需提前生成RANK_TABLE_FILE ```shell # 不包含8本身，生成0~7卡的hccl json文件 python mindformers/tools/hccl_tools.py --device_num [0,8] ``` - 单卡启动：统一接口启动，根据模型 CONFIG 完成任意模型的单卡训练、微调、评估、推理流程 ```shell # 训练启动，run_status支持train、finetuen、eval、predict四个关键字，以分别完成模型训练、评估、推理功能，默认使用配置文件中的run_mode python run_mindformer.py --config {CONFIG_PATH} --run_mode {train/finetune/eval/predict} ``` - 多卡启动： scripts 脚本启动，根据模型 CONFIG 完成任意模型的单卡/多卡训练、微调、评估、推理流程 ```shell # 8卡分布式运行， DEVICE_RANGE = [0, 8], 不包含8本身 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_MODE ``` - 常用参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的{model_name}/run_*.yaml配置文件 DEVICE_ID: 为设备卡，范围为0~7 DEVICE_RANGE: 为单机分布式卡的范围, 如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` ### 方式二：调用API启动 **详细高阶API使用教程请参考：**[MindFormers大模型使用教程](docs/README.md) - 准备工作 - step 1：安装mindformers 具体安装请参考[第二章](https://gitee.com/mindspore/mindformers/blob/dev/README.md#%E4%BA%8Cmindformers%E5%AE%89%E8%A3%85) - step2: 准备数据 准备相应任务的数据集，请参考`docs`目录下各模型的README.md文档准备相应数据集。 - Trainer 快速入门 用户可以通过以上方式安装mindformers库，然后利用Trainer高阶接口执行模型任务的训练、微调、评估、推理功能。 - Trainer 训练\微调启动 用户可使用`Trainer.train`接口完成模型的训练\微调\断点续训。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 # Example 1： 开启训练复现流程 cls_trainer.train() # Example 2： 加载集成的mae权重，开启微调流程 cls_trainer.finetune(finetune_checkpoint='mae_vit_base_p16') # Example 3： 开启断点续训功能 cls_trainer.train(train_checkpoint=True, resume_training=True) ``` - Trainer 评估启动 用户可使用`Trainer.evaluate`接口完成模型的评估流程。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 # Example 1： 开启评估已集成模型权重的复现流程 cls_trainer.evaluate() # Example 2： 开启评估训练得到的最后一个权重 cls_trainer.evaluate(eval_checkpoint=True) # Example 3： 开启评估指定的模型权重 cls_trainer.evaluate(eval_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt') ``` ```text 结果打印示例(已集成的vit_base_p16模型权重评估分数)： Top1 Accuracy=0.8317 ``` - Trainer 推理启动 用户可使用`Trainer.predict`接口完成模型的推理流程。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16') # 已支持的模型名 input_data = './cat.png' # 一张猫的图片 # Example 1： 指定输入的数据完成模型推理 predict_result_d = cls_trainer.predict(input_data=input_data) # Example 2： 开启推理（自动加载训练得到的最后一个权重） predict_result_b = cls_trainer.predict(input_data=input_data, predict_checkpoint=True) # Example 3： 加载指定的权重以完成推理 predict_result_c = cls_trainer.predict(input_data=input_data, predict_checkpoint='./output/checkpoint/rank_0/mindformers.ckpt') print(predict_result_d) ``` ```text 结果打印示例(已集成的vit_base_p16模型权重推理结果)： {‘label’: 'cat', score: 0.99} ``` - pipeline 快速入门 MindFormers套件为用户提供了已集成模型的pipeline推理接口，方便用户体验大模型推理服务。 - pipeline 使用 ```python from mindformers import pipeline from mindformers.tools.image_tools import load_image test_img = load_image("./sunflower.png") # 一朵太阳花图片 classifier = pipeline("zero_shot_image_classification", model='clip_vit_b_32', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) predict_result = classifier(test_img) print(predict_result) ``` ```text 结果打印示例(已集成的clip_vit_b_32模型权重推理结果)： [[{'score': 0.9999547, 'label': 'sunflower'}, {'score': 1.8684346e-05, 'label': 'toy'}, {'score': 1.3045716e-05, 'label': 'dog'}, {'score': 1.129241e-05, 'label': 'tree'}, {'score': 2.1734568e-06, 'label': 'cat'}]] ``` - AutoClass 快速入门 MindFormers套件为用户提供了高阶AutoClass类，包含AutoConfig、AutoModel、AutoProcessor、AutoTokenizer四类，方便开发者进行调用。 - AutoConfig 获取已支持的任意模型配置 ```python from mindformers import AutoConfig # 获取clip_vit_b_32的模型配置 clip_vit_b_32_config = AutoConfig.from_pretrained('clip_vit_b_32') # 获取vit_base_p16的模型配置 vit_base_p16_config = AutoConfig.from_pretrained('vit_base_p16') ``` - AutoModel 获取已支持的网络模型 ```python from mindformers import AutoModel # 利用from_pretrained功能实现模型的实例化（默认加载对应权重） clip_vit_b_32_a = AutoModel.from_pretrained('clip_vit_b_32') # 利用from_config功能实现模型的实例化（默认加载对应权重） clip_vit_b_32_config = AutoConfig.from_pretrained('clip_vit_b_32') clip_vit_b_32_b = AutoModel.from_config(clip_vit_b_32_config) # 利用save_pretrained功能保存模型对应配置 clip_vit_b_32_b.save_pretrained('./clip', save_name='clip_vit_b_32') ``` - AutoProcessor 获取已支持的预处理方法 ```python from mindformers import AutoProcessor # 通过模型名关键字获取对应模型预处理过程（实例化clip的预处理过程，通常用于Trainer/pipeline推理入参） clip_processor_a = AutoProcessor.from_pretrained('clip_vit_b_32') # 通过yaml文件获取相应的预处理过程 clip_processor_b = AutoProcessor.from_pretrained('configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml') ``` - AutoTokenizer 获取已支持的tokenizer方法 ```python from mindformers import AutoTokenizer # 通过模型名关键字获取对应模型预处理过程（实例化clip的tokenizer，通常用于Trainer/pipeline推理入参） clip_tokenizer = AutoTokenizer.from_pretrained('clip_vit_b_32') ``` ## 五、贡献 欢迎参与社区贡献，可参考MindSpore贡献要求[Contributor Wiki](https://gitee.com/mindspore/mindspore/blob/dev/CONTRIBUTING_CN.md)。 ## 六、许可证 [Apache 2.0许可证](LICENSE)pengcheng.mind-mindformer/build.sh000077500000000000000000000024541476650141400175550ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "---------------- MindFormers: build start ----------------" BASEPATH=$(cd "$(dirname $0)"; pwd) export BUILD_PATH="${BASEPATH}/build/" python setup.py bdist_wheel -d ${BASEPATH}/output if [ ! -d "${BASEPATH}/output" ]; then echo "The directory ${BASEPATH}/output dose not exist." exit 1 fi cd ${BASEPATH}/output || exit for package in mindformers*whl do [[ -e "${package}" ]] || break sha256sum ${package} > ${package}.sha256 done pip install mindformers*whl -i https://pypi.tuna.tsinghua.edu.cn/simple cd ${BASEPATH} || exit rm -rf *-info echo "---------------- MindFormers: build and install end ----------------" pengcheng.mind-mindformer/configs/000077500000000000000000000000001476650141400175425ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/README.md000066400000000000000000000317041476650141400210260ustar00rootroot00000000000000# Config配置说明 ## 配置项顺序 configs统一在run_xxx.yaml中，排序按照修改频率的顺序和一般的模型训练流程顺序（数据集->模型->训练、评估、推理），具体顺序如下 - 非模块参数：seed、run_mode、output_dir、load_checkpoint、resume_training、auto_trans_ckpt - 环境参数：context - AICC：remote_save_url - 运行参数：runner_config、runner_wrapper - 并行设置：use_parallel、parallel、parallel_config - MOE：moe_config - 重计算：recompute_config - 算子调优：auto_tune - 性能工具：profile - Trainer：trainer - 数据集：train_dataset、train_dataset_task、eval_dataset、eval_dataset_task - 模型：model - 学习率：lr_schedule、layer_scale、layer_decay、lr_scale、lr_scale_factor - 优化器：optimizer - 回调函数：callbacks - 评估函数：metric - Processor：processor ## 详细配置说明 - seed: 随机种子，可以参考[mindspore.set_seed](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_seed.html) - run_mode: 运行模式，可选"train"、"finetune"、"eval"或"predict" - output_dir: './output' 输出路径，当前不支持修改该参数 - load_checkpoint: 加载权重的模型名或权重路径 - auto_trans_ckpt: 是否开启自动在线权重切分或转换（目前尚不支持） - resume_training: 加载方式，为True时会加载训练过程信息，如优化器、epochs数等 - context: 环境配置，可以参考: [mindspore.set_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_context.html) - mode: 0代表Graph Mode， 1代表Pynative Mode - device_target: 设备类型，Ascend、CPU或GPU，默认为Ascend - enable_graph_kernel: 是否开启图算融合 - graph_kernel_flags: 图算融合等级 - max_call_depth: 函数调用的最大深度 - save_graphs: 是否保存图 - device_id: 默认设备id - remote_save_url: 使用AICC训练作业时的，目标桶的回传文件夹路径 - runner_config: 运行配置 - epochs: 迭代次数 - batch_size: 数据批次大小，当前在使用yaml初始化训练时，会覆盖数据集配置中的batch_size，后面会删除改配置 - sink_mode: 是否开启数据下沉模式 - sink_size: 每次下沉数据大小，-1代表全量下沉 - runner_wrapper: wrapper配置 - type: wrapper类 - scale_sense: 梯度缩放配置 - type: 梯度缩放类 - use_clip_grad: 是否开启梯度裁剪 - loss_scale_value: 缩放系数 - use_parallel: 是否开启并行 - parallel: 自动并行配置，可以参考：[mindspore.set_auto_parallel_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_auto_parallel_context.html) - parallel_mode: 并行模式，0-dataset数据并行, 1-semi半自动并行, 2-auto自动并行, 3-hybrid手工实现并行 - gradients_mean: 是否在梯度AllReduce后执行平均算子。通常半自动并行模式下为False，数据并行模式下为True - enable_alltoall: 允许在通信期间生成AllToAll通信算子的开关。通常仅在MOE场景下打开，默认False - full_batch: 在auto_parallel模式下加载整个batch数据集时为True。半自动并行模式通常设置为True，数据并行模式必须设置为False，否则会报错 - search_mode: 策略搜索模式，有三种，分别是recursive_programming，dynamic_programming和sharding_propagation。仅在全自动并行模式下生效，其他模式不生效，实验性接口，谨慎使用 - enable_parallel_optimizer: 数据并行训练时对权重更新计算进行分片。优化器并行开关，在数据并行训练时默认会将模型权重参数切分成device_num份，与parallel_config中optimizer_shard保持一致；半自动并行时默认将模型权重参数切份data_parallel份 - strategy_ckpt_save_file: 保存并行切分策略的路径。 - parallel_config: 并行策略配置，可以参考mindformers.modules.transformer.TransformerOpParallelConfig - data_parallel: 数据并行 - model_parallel: 模型并行 - pipeline_stage: 流水线并行 > 需要满足实际运行的卡数 device_num = data_parallel × model_parallel × pipeline_stage - use_seq_parallel: 是否开启序列并行 - optimizer_shard: 是否开启优化器切分。优化器并行开关，通常在半自动并行模式下生效，与parallel中的enable_parallel_optimizer保持一致，默认将模型权重参数切份data_parallel份 - micro_batch_num: 流水线并行的微批次大小。pipeline_satge大于1时，开启流水并行时使用，此处需满足micro_batch_num >= pipeline_satge - gradient_aggregation_group: 梯度通信算子融合组的大小 - micro_batch_interleave_num: batch_size的拆分份数，多副本并行开关，通常在模型并行时使用，用于优化model_parallel时产生的通信损耗，纯流水并行时不建议使用。可以参考[mindspore.nn.MicroBatchInterleaved](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/nn/mindspore.nn.MicroBatchInterleaved.html) - moe_config: 混合专家配置，当前大部分仓上模型不支持，实验性接口，谨慎使用。可以参考mindformers.modules.transformer.moe.MoEConfig - expert_num: 专家数量 - capacity_factor: 专家能力因子 - aux_loss_factor: loss贡献因子 - num_experts_chosen: 每个token选择专家数目 - recompute_config：重计算配置，可以参考mindformers.modules.transformer.TransformerRecomputeConfig - recompute: 是否开启重计算 - select_recompute: 是否开启选择重计算，只针对attention层的算子进行重计算 - parallel_optimizer_comm_recompute: 由优化器并行引入的AllGather通信是否重计算 - mp_comm_recompute: 由模型并行引入的通信操作是否重计算 - recompute_slice_activation: 是否把保留在内存中的Cell输出切片 - auto_tune: 是否开启自动数据加速，可以参考[mindspore.dataset.config.set_enable_autotune](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.config.set_enable_autotune.html) - filepath_prefix: 优化后的全局配置的保存路径+文件前缀 - autotune_per_step: 设置自动数据加速的配置调整step间隔，可以参考[mindspore.dataset.config.set_autotune_interval](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.config.set_autotune_interval.html) - profile: 是否开启性能分析工具，可以参考[mindspore.Profiler](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.Profiler.html) - profile_start_step: 性能分析开始的step - profile_stop_step: 性能分析结束的step - profile_communication: 是否在多设备训练中收集通信性能数据 - profile_memory: 是否收集Tensor内存数据 - init_start_profile: 是否在Profiler初始化的时候开启数据采集 - trainer: 训练流程配置 - type: 训练流程类 - model_name: 训练模型名 - do_eval: 是否开启边训练边评估 - eval_step_interval: 评估step间隔, 默认为100，表示每100个step间隔执行一次评估；配置为大于0的数表示每隔所配置的step数后执行一次评估，配置为小于0的数则表示禁用step评估 - eval_epoch_interval: 评估epoch间隔, 默认为-1，表示禁用epoch结束时的评估；配置为大于0的数表示每隔所配置的epoch数后执行一次评估，配置为小于0的数则表示禁用epoch评估；注意：数据下沉模式下，epoch所包含的step数将从数据集大小变为sink size的大小，不建议在数据下沉模式下使用本项配置 - train_dataset: 训练数据集配置，可以参考[mindspore.dataset.GeneratorDataset](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.GeneratorDataset.html) - seed: 随机种子 - batch_size: 批次大小，当前在使用yaml初始化训练时，该参数会被runner_config中的batch_size覆盖 - data_loader: 数据加载配置，可以参考[mindspore.dataset.ImageFolderDataset](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset/mindspore.dataset.ImageFolderDataset.html) - type: 数据加载类 - dataset_dir: 数据集的根目录或数据集文件的路径 - num_parallel_workers: 读取数据的工作线程数 - shuffle: 是否混洗数据集 - transforms: 数据增强操作 - tokenizer: 分词器配置 - type: 分词器类 - max_length: 分词器输出的最大长度 - mixup_op: 图像随机混合，可以参考[mindspore.dataset.vision.MixUp](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/dataset_vision/mindspore.dataset.vision.MixUp.html) - input_columns: 输入数据列 - output_columns: 输出数据列 - column_order: 输出数据顺序 - num_parallel_workers: 读取数据的工作进程数/线程数 - python_multiprocessing: 启用Python多进程模式加速运算 - drop_remainder: 当最后一个批处理数据包含的数据条目小于batch_size时，是否将该批处理丢弃 - repeat: 重复此数据集count次 - numa_enable: 设置NUMA的默认状态为启动状态 - prefetch_size: 设置管道中线程的队列容量 - eval_dataset: 评估数据集配置，具体配置说明可参考train_dataset - model: 模型配置 - arch: 模型类配置 - type: 模型类 - model_config: 模型参数配置 - type: 模型参数配置类 - checkpoint_name_or_path: 评估时不指定权重，模型默认加载的权重名 *\# 以下配置针对大规模语言模型推理* - top_k: 从概率最大的top_k个tokens中采样 - top_p: 从概率最大且概率累计不超过top_p的tokens中采样 - do_sample: 使能top_k或top_p采样，为False时top_k和top_p均重置为1 - use_past: 使能增量推理，为True时为增量推理，否则为自回归推理，使用时请参考[模型支持列表](https://gitee.com/mindspore/mindformers/tree/dev/docs#text-generator) - max_decode_length: 文本生成最大长度（输入长度统计在内） - repetition_penalty: 重复文本惩罚系数，该值不小于1，等于1时不惩罚 - lr_schedule: 学习率配置 - type: 学习率类 - layer_scale: 是否开启层衰减 - layer_decay: 层衰减系数 - optimizer: 优化器配置 - type: 优化器类 - weight_decay: 权重衰减值 - lr_scale: 是否开启学习率缩放 - lr_scale_factor: 学习率缩放系数 - callbacks: 回调函数配置 - type: 回调函数类 - type: MFLossMonitor: loss打印 - type: SummaryMonitor: 收集summary数据，可以参考[mindspore.SummaryCollector](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.SummaryCollector.html) - type: CheckpointMointor: checkpoint保存，可以参考[mindspore.save_checkpoint](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.save_checkpoint.html) - prefix: 权重文件前缀 - directory: 保存权重的目录 - save_checkpoint_seconds: 设定多少s保存一次ckpt - save_checkpoint_steps: 每多少个step保存一次checkpoint - keep_checkpoint_max: 设定保存ckpt的最大数量，超过则会删除最旧的权重，以保证数量不变 - keep_checkpoint_per_n_minutes: 设定多少minutes保存一次ckpt - integrated_save: 是否聚合保存。True时表示聚合所有卡权重，这时每张卡权重均一致；False时表示每张卡各自保存自己的权重；当半自动并行模式训练大模型时，通常需要设置为False，以保证权重保存时不会因为内存问题而失败 - save_network_params（新增）: 是否额外保存瘦身后的权重。默认为True。 - save_trainable_params（新增）: 是否额外保存可训练的参数权重，即微调部分参数的权重。默认为False。 - async_save: 是否异步执行保存checkpoint文件 - type: ObsMonitor: obs数据上传 - step_upload_frequence: 每隔多少个step上传一次，默认为100，表示每100个step执行一次数据上传；配置为大于0的数时，每隔配置数step后执行一次回传，小于0的数则表示禁用step回传 - epoch_upload_frequence: 每隔多少个epoch上传一次，默认为-1，表示禁用epoch回传；设置大于0的值表示每隔所配置的epoch数后回传；注意：数据下沉模式下，epoch所包含的step数将从数据集大小变为sink size的大小，不建议在数据下沉模式下使用本项配置 - keep_last: 检查obs的文件与AI计算中心平台是否一致，默认True，表示仅保留最后一次回传的内容，前面几次回传内容将会被移除；设为False则会保留每次回传的内容 - metric: 评估指标配置 - type: 评估指标类 - processor: 推理时的数据处理 - return_tensors: 返回张量类型 - type: 数据处理类 - image_processor: 图像处理配置 - type: 图像处理类 - tokenizer: 分词器配置 - type: 分词器类 pengcheng.mind-mindformer/configs/bert/000077500000000000000000000000001476650141400204765ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/bert/run_bert_base_uncased.yaml000066400000000000000000000103041476650141400256740ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 128 sink_size: 2 image_size: 224 sink_mode: True runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: MaskedLanguageModelingTrainer model_name: 'bert_base_uncased' # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./wiki_data" shuffle: True shard_equal_rows: True input_columns: ["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 1 dropout_prob: 0.1 batch_size: 128 seq_length: 128 #length of input sentence vocab_size: 30522 #size of vocab hidden_size: 768 #size of text feature num_hidden_layers: 12 #model depth num_attention_heads: 12 #number of attention heads intermediate_size: 3072 #hidden_size*4 hidden_act: "gelu" #activation post_layernorm_residual: True #select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False checkpoint_name_or_path: "" arch: type: BertForPreTraining # lr sechdule lr_schedule: type: cosine learning_rate: 0.0001 lr_end: 0.000001 warmup_steps: 10000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: BertProcessor tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' return_tensors: ms pengcheng.mind-mindformer/configs/bert/run_bert_tiny_uncased.yaml000066400000000000000000000103241476650141400257470ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 128 sink_size: 2 image_size: 224 sink_mode: True runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: MaskedLanguageModelingTrainer model_name: 'bert_tiny_uncased' # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./wiki_data" shuffle: True shard_equal_rows: True input_columns: ["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 1 dropout_prob: 0.1 batch_size: 128 seq_length: 128 #length of input sentence vocab_size: 30522 #size of vocab hidden_size: 128 #size of text feature num_hidden_layers: 2 #model depth num_attention_heads: 2 #number of attention heads intermediate_size: 512 #hidden_size*4 hidden_act: "gelu" #activation post_layernorm_residual: True #select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False use_moe: False checkpoint_name_or_path: "" arch: type: BertForPreTraining # lr sechdule lr_schedule: type: cosine learning_rate: 0.0001 lr_end: 0.000001 warmup_steps: 10000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: BertProcessor tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' return_tensors: ms pengcheng.mind-mindformer/configs/blip2/000077500000000000000000000000001476650141400205525ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yaml000066400000000000000000000136011476650141400314260ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 80 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: True parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage1_vit_g' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: ["image", "text"] transforms: - type: RandomResizedCrop size: 224 scale: [0.5, 1.0] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 32 padding: 'max_length' random_seed: 2022 truncation: True tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32 num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 return_attention_mask: True train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False convert_param_from_bert: True checkpoint_name_or_path: "bert_base_uncased" arch: type: Blip2Qformer # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_qformer" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor step_upload_frequence: 1000 eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32 pengcheng.mind-mindformer/configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml000066400000000000000000000135761476650141400317270ustar00rootroot00000000000000seed: 42 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 100 sink_size: 2 image_size: 224 sink_mode: False initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: True profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_to_text_retrieval model_name: 'blip2_stage1_vit_g' # eval dataset eval_dataset: &eval_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "flickr30k/annotations/test.json" ] image_dirs: [ "flickr30k/images" ] stage: "eval" column_names: ["image", "text"] shuffle: False transforms: - type: ToPIL # align with torch impl -> check interaction with 'Resize' - type: Resize size: [224, 224] interpolation: "bicubic" - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 35 padding: 'max_length' random_seed: 2022 truncation: True tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 35 k_test: 128 # only top_k takes part in computation add_extra_itm_score: True num_parallel_workers: 1 python_multiprocessing: False drop_remainder: True batch_size: 50 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 return_attention_mask: True eval_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "" # pretrained model dict here dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False arch: type: Blip2Qformer # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_qformer" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml000066400000000000000000000116121476650141400356060ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/blip2seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 40 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'blip2_stage1_classification' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" num_parallel_workers: 1 python_multiprocessing: False drop_remainder: True batch_size: 40 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config freeze_vision: True max_txt_len: 32 checkpoint_name_or_path: "blip2_stage1_classification" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: num_hidden_layers: 12 num_attention_heads: 12 query_length: 32 vocab_size: 30522 hidden_size: 768 encoder_width: 1408 bos_token_id: 30522 sep_token_id: 102 pad_token_id: 0 max_position_embeddings: 512 layer_norm_eps: 1.e-12 hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 chunk_size_feed_forward: 0 cross_attention_freq: 2 intermediate_size: 3072 initializer_range: 0.02 hidden_act: "gelu" dtype: "float32" layernorm_dtype: "float32" softmax_dtype: "float32" compute_dtype: "float16" add_cross_attention: True use_relative_positions: False tie_word_embeddings: True output_attentions: False output_hidden_states: False use_return_dict: False arch: type: Blip2Classifier # metric metric: type: 'Accuracy' # lr sechdule lr_schedule: type: cosine learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 5000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False lr_scale: False # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 weight_decay: 0.05 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage1_classification" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # image processor, tokenizer for prediction processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: [224, 224] interpolation: "bicubic" mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: BertTokenizer pad_token: '[PAD]' bos_token: '[DEC]' add_special_tokens: True padding: 'max_length' truncation: True max_length: 32pengcheng.mind-mindformer/configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yaml000066400000000000000000000141061476650141400302130ustar00rootroot00000000000000seed: 42 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 16 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage2_vit_g_baichuan_7b' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: ["image", "text"] shuffle: True transforms: - type: RandomResizedCrop size: 224 scale: [0.5, 1.0] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels padding: 'max_length' random_seed: 2022 truncation: True add_special_tokens: True tokenizer: type: LlamaTokenizer unk_token: '' bos_token: '' eos_token: '' pad_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels add_bos_token: False add_eos_token: True vocab_file: "" num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 32 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" prompt: False prompt_length: 0 vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: vocab_size: 44728 text_config: type: LlamaConfig seq_length: 64 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 64001 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: False offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False checkpoint_name_or_path: "" arch: type: Blip2Llama # lr schedule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_lr_init: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.98 eps: 1.e-8 # 1e-8 weight_decay: 0.05 learning_rate: 1.e-4 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage2_vig_g_baichuan_7b" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 32 run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml000066400000000000000000000112251476650141400352560ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/blip2seed: 42 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 1 sink_size: 2 image_size: 224 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ImageToTextGenerationTrainer model_name: 'itt_blip2_stage2_vit_g_baichuan_7b' # train dataset eval_dataset: &eval_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: [ "image", "text"] hypothesis_template: "" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 128 checkpoint_name_or_path: "" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: true hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" text_config: type: LlamaConfig seq_length: 160 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 64001 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: True offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 1 top_p: 1 do_sample: False checkpoint_name_or_path: "" arch: type: Blip2ImageToTextGeneration # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 128 vocab_file: "" # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor pengcheng.mind-mindformer/configs/blip2/run_blip2_stage2_vit_g_llama_7b.yaml000066400000000000000000000142001476650141400275220ustar00rootroot00000000000000seed: 42 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'blip2_stage1_pretrained' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 24 sink_size: 2 image_size: 224 sink_mode: True initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'blip2_stage2_vit_g_llama_7b' # train dataset train_dataset: &train_dataset data_loader: type: MultiImgCapDataLoader dataset_dir: "/data" annotation_files: [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs: [ "vg/images", "coco2014/coco/images" ] stage: "train" column_names: [ "image", "text" ] shuffle: True transforms: - type: RandomResizedCrop size: 224 scale: [ 0.5, 1.0 ] interpolation: "bicubic" - type: RandomHorizontalFlip - type: ToTensor - type: Normalize mean: [ 0.48145466, 0.4578275, 0.40821073 ] std: [ 0.26862954, 0.26130258, 0.27577711 ] is_hwc: False text_transforms: type: CaptionTransform prompt: "" max_words: 50 max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels padding: 'max_length' random_seed: 2022 truncation: True add_special_tokens: True tokenizer: type: LlamaTokenizer unk_token: '' bos_token: '' eos_token: '' pad_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 33 # it equals to Blip2Config.max_txt_len + 1 for constructing labels add_bos_token: False add_eos_token: True checkpoint_name_or_path: 'llama_7b' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 32 checkpoint_name_or_path: "blip2_stage1_pretrained" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" prompt: False prompt_length: 0 vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" qformer_config: vocab_size: 44728 text_config: type: LlamaConfig seq_length: 64 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: False offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False checkpoint_name_or_path: "llama_7b" arch: type: Blip2Llama # lr schedule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 5.e-5 warmup_lr_init: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.98 eps: 1.e-8 # 1e-8 weight_decay: 0.05 learning_rate: 1.e-4 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "blip2_stage2_vig_g_llama_7b" save_checkpoint_steps: 5000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 32 run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml000066400000000000000000000113241476650141400345720ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/blip2seed: 42 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: str = "--disable_expand_ops=Softmax,Dropout " \ "--enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: &batch_size 1 sink_size: 2 image_size: 224 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ImageToTextGenerationTrainer model_name: "itt_blip2_stage2_vit_g_llama_7b" # eval dataset eval_dataset: &eval_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: [ "image", "text"] hypothesis_template: "a picture of " transforms: - type: ToPIL - type: Resize size: 224 interpolation: "linear" - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: Blip2Config batch_size: *batch_size freeze_vision: True freeze_text: True max_txt_len: 128 checkpoint_name_or_path: "itt_blip2_vit_g_llama_7b" dtype: "float32" compute_dtype: "float16" layernorm_dtype: "float32" softmax_dtype: "float32" vision_config: type: ViTConfig image_size: 224 patch_size: 14 num_channels: 3 initializer_range: 0.001 hidden_size: 1408 num_hidden_layers: 39 num_attention_heads: 16 intermediate_size: 6144 qkv_bias: True hidden_act: gelu post_layernorm_residual: false layer_norm_eps: 1.0e-6 attention_probs_dropout_prob: 0.0 hidden_dropout_prob: 0.0 drop_path_rate: 0.0 use_mean_pooling: false encoder_stride: 16 checkpoint_name_or_path: "vit_g_p16" text_config: type: LlamaConfig seq_length: 2048 # sum of max_txt_len and num_query_token hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float16" param_init_type: "float16" use_past: True offset: 0 repetition_penalty: 1 max_decode_length: 512 top_k: 1 top_p: 1 do_sample: False checkpoint_name_or_path: "llama_7b" arch: type: Blip2ImageToTextGeneration # processor processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # input image size tokenizer: type: LlamaTokenizer pad_token: '' bos_token: '' unk_token: '' eos_token: '' add_special_tokens: False padding: 'max_length' truncation: True max_length: 128 checkpoint_name_or_path: "llama_7b" # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 10000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor pengcheng.mind-mindformer/configs/bloom/000077500000000000000000000000001476650141400206525ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/bloom/run_bloom_176b.yaml000066400000000000000000000100051476650141400242650ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 14336 num_layers: 70 num_heads: 112 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 16 optimizer_shard: True micro_batch_num: 48 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_176b' pengcheng.mind-mindformer/configs/bloom/run_bloom_560m.yaml000066400000000000000000000100171476650141400243000ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 1024 num_layers: 24 num_heads: 16 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "bloom_560m" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: False parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_560m' pengcheng.mind-mindformer/configs/bloom/run_bloom_65b.yaml000066400000000000000000000100021476650141400241770ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 8192 num_layers: 80 num_heads: 64 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 12 optimizer_shard: True micro_batch_num: 48 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 1 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_65b' pengcheng.mind-mindformer/configs/bloom/run_bloom_7.1b.yaml000066400000000000000000000100151476650141400242560ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False micro_batch_interleave_num: 1 auto_tune: True filepath_prefix: './autotune' autotune_per_step: 8 profile: False profile_start_step: 10 profile_stop_step: 20 profile_memory: True model: model_config: type: BloomConfig seq_length: 2048 vocab_size: 250880 hidden_size: 4096 num_layers: 30 num_heads: 32 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 initializer_range: 0.02 param_init_type: "float16" embedding_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" use_seq_parallel: False use_select_recompute: False checkpoint_name_or_path: "bloom_7.1b" eos_token_id: 2 bos_token_id: 1 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 arch: type: BloomLMHeadModel train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset processor: return_tensors: ms tokenizer: type: BloomTokenizer type: BloomProcessor context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_id: 0 device_target: "Ascend" graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # parallel use_parallel: True parallel: parallel_optimizer_config: {"gradient_accumulation_shard": False} parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # aicc remote_save_url: "Please input obs url on AICC platform." # runner config runner_config: epochs: 10 batch_size: 4 sink_mode: True sink_size: 8 per_epoch_size: 8 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 weight_decay: 0.1 lr_schedule: type: cosine learning_rate: 0.000001 lr_end: 0.0000001 warmup_steps: 1000 total_steps: 4000 # -1 means it will load the total steps of the dataset callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor trainer: type: CausalLanguageModelingTrainer model_name: 'bloom_7.1b' pengcheng.mind-mindformer/configs/clip/000077500000000000000000000000001476650141400204715ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml000066400000000000000000000101341476650141400303610ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_b_16' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 16 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_16 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103601476650141400345000ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/clipseed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_b_16' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 16 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_16 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-mindformer/configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml000066400000000000000000000101351476650141400303600ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_b_32' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 32 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_32 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103601476650141400344760ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/clipseed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_b_32' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 512 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 8 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 768 intermediate_size: 3072 num_hidden_layers: 12 num_attention_heads: 12 image_size: 224 patch_size: 32 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_b_32 # the loaded model type projection_dim: 512 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-mindformer/configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml000066400000000000000000000101471476650141400307310ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 16 image_size: 336 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_l_14@336' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 336 - type: CenterCrop size: 336 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 336 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14@336 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 336 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103721476650141400350470ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/clipseed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 16 image_size: 336 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_l_14@336' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 336 interpolation: 'linear' - type: CenterCrop size: 336 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 336 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14@336 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 336 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-mindformer/configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml000066400000000000000000000101361476650141400303730ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ContrastiveLanguageImagePretrainTrainer model_name: 'clip_vit_l_14' # train dataset train_dataset: &train_dataset data_loader: type: Flickr8kDataLoader dataset_dir: "./checkpoint_download/Flickr8k" stage: "train" column_names: ["image", "text"] text_transforms: type: RandomChoiceTokenizerForward max_length: 77 padding: "max_length" random_seed: 2022 transforms: - type: Resize size: 224 - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False tokenizer: type: CLIPTokenizer pad_token: '!' num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: ContrastiveLanguageImagePretrainDataset dataset_config: *train_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 224 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml000066400000000000000000000103611476650141400345110ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/clipseed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 5 batch_size: 32 image_size: 224 sink_mode: False sink_size: 2 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: ZeroShotImageClassificationTrainer model_name: 'clip_vit_l_14' # eval dataset eval_dataset: &eval_dataset data_loader: type: Cifar100DataLoader dataset_dir: 'cifar-100-python' column_names: ["image", "text", "label"] stage: 'test' fine_label: True shuffle: False hypothesis_template: "a picture of {}" tokenizer: type: CLIPTokenizer pad_token: '!' text_transforms: type: TokenizerForward max_length: 77 padding: "max_length" transforms: - type: ToPIL - type: Resize size: 224 interpolation: 'linear' - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 32 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: ZeroShotImageClassificationDataset dataset_config: *eval_dataset # model model: arch: type: CLIPModel model_config: type: CLIPConfig text_config: type: CLIPTextConfig vocab_size: 49408 hidden_size: 768 intermediate_size: 2048 num_hidden_layers: 12 num_attention_heads: 12 max_position_embeddings: 77 hidden_act: "quick_gelu" attention_dropout: 0.0 drop_out: 0.0 initializer_range: 0.02 initializer_factor: 1.0 vision_config: type: CLIPVisionConfig hidden_size: 1024 intermediate_size: 3072 num_hidden_layers: 24 num_attention_heads: 16 image_size: 224 patch_size: 14 hidden_act: "quick_gelu" dropout: 0.0 attention_dropout: 0.0 initializer_range: 0.02 initializer_factor: 1.0 dtype: float16 # type of tensors checkpoint_name_or_path: clip_vit_l_14 # the loaded model type projection_dim: 768 # feature dims logit_scale_init_value: 2.6592 # lr sechdule layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.001 learning_rate: 0.00001 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: 'Accuracy' # processor processor: type: CLIPProcessor image_processor: type: CLIPImageProcessor image_resolution: 224 # input image size tokenizer: type: CLIPTokenizer pad_token: '!' # corresponding token id is 0 top_k: 3 save_file: 'results.txt' pengcheng.mind-mindformer/configs/filip/000077500000000000000000000000001476650141400206455ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/filip/run_filip_vit_l_14.yaml000066400000000000000000000116011476650141400252200ustar00rootroot00000000000000seed: 0 run_status: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 15 batch_size: 64 image_size: 224 sink_mode: False sink_size: 2 runner_wrapper: type: FilipTrainOneStepWithLossScaleWrapper # parallel use_parallel: True parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: itr model_name: 'filip' # train dataset train_dataset: &train_dataset data_loader: type: ImageToTextRetrievalDataset dataset_dir: "" shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.2, 1.0] interpolation: bicubic - type: RandomHorizontalFlip prob: 0.5 - type: RandomColorAdjust brightness: 0.4 contrast: 0.4 saturation: 0.4 hue: 0.1 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False project_columns: ["image", "token"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: ImageToTextRetrievalDataset dataset_config: *train_dataset # eval dataset eval: &eval_dataset data_loader: type: ImageToTextRetrievalDataset dataset_dir: "" shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.2, 1.0] interpolation: bicubic - type: RandomHorizontalFlip prob: 0.5 - type: RandomColorAdjust brightness: 0.4 contrast: 0.4 saturation: 0.4 hue: 0.1 - type: ToTensor - type: Normalize mean: [0.48145466, 0.4578275, 0.40821073] std: [0.26862954, 0.26130258, 0.27577711] is_hwc: False project_columns: ["image", "token"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: ImageToTextRetrievalDataset dataset_config: *eval_dataset # model model: arch: type: FilipModel model_config: type: FilipConfig text_config: type: FilipTextConfig hidden_size: 768 vocab_size: 21128 max_position_embeddins: 32 num_hidden_layers: 12 vision_config: type: FilipVisionConfig hidden_size: 1024 image_size: 224 patch_size: 14 num_hidden_layers: 24 token_learner: True num_tokens: 24 num_token_groups: 8 token_learner_dropout: 1.0 dtype: float16 checkpoint_name_or_path: "" projection_dim: 256 ratio: 64 batch_size: 64 recompute: False # lr sechdule lr_schedule: type: CosineDecayLR max_lr: 0.00005 min_lr: 0. decay_steps: 780 layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: Lamb eps: 0.000001 weight_decay: 0.0003 lr_scale: False lr_scale_factor: 256 no_decay_params: ['positional_embedding', 'class_embedding', 'scale', 'bias', 'ln', 'bn'] # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: FilipProcessor feature_extractor: type: FilipFeatureExtractor image_feature_extractor: type: FilipImageFeatureExtrator image_resolution: 224 tokenizer: type: FilipTokenizer pengcheng.mind-mindformer/configs/general/000077500000000000000000000000001476650141400211575ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/general/run_general_task.yaml000066400000000000000000000052001476650141400253630ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: TrainOneStepCell sens: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: general model_name: 'general' # please edit by hand. train_dataset: train_dataset_task: eval_dataset: eval_dataset_task: # please edit by hand. model: model_config: checkpoint_name_or_path: arch: # lr sechdule lr_schedule: type: cosine learning_rate: 0.00015 lr_end: 0. warmup_lr_init: 0. warmup_epochs: 40 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # please edit by hand. processor: tokenizer: image_processor: audio_processor: pengcheng.mind-mindformer/configs/glm/000077500000000000000000000000001476650141400203215ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/glm/run_glm_6b_finetune.yaml000066400000000000000000000133021476650141400251330ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: False activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: False checkpoint_name_or_path: "glm_6b" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True arch: type: GLMForPreTraining trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/glm/run_glm_6b_infer.yaml000066400000000000000000000131631476650141400244260ustar00rootroot00000000000000seed: 0 run_mode: 'predict' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: True # True for incremental infer. activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: True # sample in npu to acceleration checkpoint_name_or_path: "glm_6b_chat" top_k: 1 top_p: 1 # top_p not support for npu_acceleration yet repetition_penalty: 1 do_sample: True arch: type: GLMChatModel trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_chat' metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/glm/run_glm_6b_lora.yaml000066400000000000000000000134321476650141400242570ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: False activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: False checkpoint_name_or_path: "glm_6b_lora" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: GLMForPreTrainingWithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_lora' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 layer_scale: False lr_scale: False # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/glm/run_glm_6b_lora_infer.yaml000066400000000000000000000133521476650141400254430ustar00rootroot00000000000000seed: 0 run_mode: 'predict' load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: GLMConfig vocab_size: 130528 hidden_size: 4096 num_layers: 28 num_heads: 32 inner_hidden_size: 16384 seq_length: 512 # 推理时, 输入pad到的长度, model里的最大句长 embedding_dropout_prob: 0.0 attention_dropout_rate: 0.0 hidden_dropout_rate: 0.0 hidden_size_per_attention_head: # default "None" means hidden-size/num-attention-heads. layernorm_order: "post" layernorm_epsilon: 1.0e-5 use_final_layernorm: True use_past: True # True for incremental infer. activation_func: 'GELU' position_encoding_2d: True param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" bos_token_id: 130004 eos_token_id: 130005 mask_token_id: 130000 gmask_token_id: 130001 pad_token_id: 3 max_decode_length: 2048 # The maximum length of the generated words. is_enhanced_encoder: True is_sample_acceleration: True # sample in npu to acceleration checkpoint_name_or_path: "glm_6b_lora_chat" top_k: 1 top_p: 1 # top_p not support for npu_acceleration yet repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: GLMChatModelWithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm_6b_lora' metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLMTokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/train.json" shuffle: True phase: "train" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] max_source_length: 64 max_target_length: 64 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/dev.json" shuffle: False phase: "eval" origin_columns: ["content", "summary"] tokenizer: type: ChatGLMTokenizer vocab_file: "/path/to/ice_text.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/glm2/000077500000000000000000000000001476650141400204035ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/glm2/run_glm2_6b.yaml000066400000000000000000000132321476650141400234040ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" # 59GB for 910B save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: ChatGLM2Config batch_size: 1 # only for incremental infer num_layers: 28 padded_vocab_size: 65024 hidden_size: 4096 ffn_hidden_size: 13696 kv_channels: 128 num_attention_heads: 32 seq_length: 193 hidden_dropout: 0.0 attention_dropout: 0.0 layernorm_epsilon: 1e-5 rmsnorm: True apply_residual_connection_post_layernorm: False post_layer_norm: True add_bias_linear: False add_qkv_bias: True bias_dropout_fusion: True multi_query_attention: True multi_query_group_num: 2 apply_query_key_layer_scaling: True attention_softmax_in_fp32: True fp32_residual_connection: False quantization_bit: 0 pre_seq_len: None prefix_projection: False param_init_type: "float16" compute_dtype: "float16" layernorm_compute_type: "float32" use_past: False eos_token_id: 2 pad_token_id: 0 repetition_penalty: 1.0 max_decode_length: 256 checkpoint_name_or_path: "glm2_6b" top_k: 1 top_p: 1 do_sample: True arch: type: ChatGLM2ForConditionalGeneration trainer: type: CausalLanguageModelingTrainer model_name: 'glm2_6b' # if True do, evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLM2Tokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 65536 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 1 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: True # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm2-6b" save_checkpoint_steps: 1000 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/glm2/run_glm2_6b_lora.yaml000066400000000000000000000134511476650141400244240ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'glm2_6b' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # ==== context config ==== context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" # 59GB for 910B save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # ==== model config ==== model: model_config: type: ChatGLM2Config batch_size: 1 # only for incremental infer num_layers: 28 padded_vocab_size: 65024 hidden_size: 4096 ffn_hidden_size: 13696 kv_channels: 128 num_attention_heads: 32 seq_length: 193 hidden_dropout: 0.0 attention_dropout: 0.0 layernorm_epsilon: 1e-5 rmsnorm: True apply_residual_connection_post_layernorm: False post_layer_norm: True add_bias_linear: False add_qkv_bias: True bias_dropout_fusion: True multi_query_attention: True multi_query_group_num: 2 apply_query_key_layer_scaling: True attention_softmax_in_fp32: True fp32_residual_connection: False quantization_bit: 0 pre_seq_len: None prefix_projection: False param_init_type: "float16" compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" use_past: False eos_token_id: 2 pad_token_id: 0 max_decode_length: 256 checkpoint_name_or_path: "glm2_6b" top_k: 1 top_p: 1 repetition_penalty: 1 do_sample: True pet_config: pet_type: lora lora_rank: 8 lora_alpha: 32 lora_dropout: 0.1 arch: type: ChatGLM2WithLora trainer: type: CausalLanguageModelingTrainer model_name: 'glm2_6b_lora' # if True do, evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False metric: type: ADGENMetric processor: return_tensors: ms tokenizer: type: ChatGLM2Tokenizer bos_token: '' eos_token: '' end_token: '' mask_token: '[MASK]' gmask_token: '[gMASK]' pad_token: '' unk_token: '' type: GLMProcessor # ==== dataset config ==== train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset # ==== runner config ==== runner_config: epochs: 1 batch_size: 8 sink_mode: True sink_size: 4 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # lr sechdule lr_schedule: type: polynomial learning_rate: 5.e-5 lr_end: 1.e-6 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # parallel config use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: False loss_repeated_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: True # optimizer shard strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 expert_parallel: 1 optimizer_shard: False # optimizer shard micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "glm2-6b-lora" save_checkpoint_steps: 500 keep_checkpoint_max: 2 integrated_save: False async_save: False - type: ObsMonitor keep_last: False eval_callbacks: - type: ObsMonitor keep_last: False pengcheng.mind-mindformer/configs/gpt2/000077500000000000000000000000001476650141400204165ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/gpt2/run_gpt2.yaml000066400000000000000000000105251476650141400230450ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 4 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0001 lr_end: 0.00001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt" save_checkpoint_steps: 1 integrated_save: True save_network_params: True save_trainable_params: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/gpt2/run_gpt2_13b.yaml000066400000000000000000000112311476650141400235050ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 4 model_parallel: 8 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 5120 num_layers: 40 num_heads: 40 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-13b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/gpt2/run_gpt2_52b.yaml000066400000000000000000000112331476650141400235120ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 64 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_52b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 8192 num_layers: 64 num_heads: 32 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-52b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/gpt2/run_gpt2_lora.yaml000066400000000000000000000105231476650141400240600ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "gpt2" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2' # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_lora" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False pet_config: pet_type: lora in_channels: 768 out_channels: 768 lora_rank: 64 lora_alpha: 128 lora_dropout: 0.1 arch: type: GPT2WithLora # lr sechdule lr_schedule: type: polynomial learning_rate: 0.001 lr_end: 0.0001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt" save_checkpoint_steps: 1 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processorpengcheng.mind-mindformer/configs/gpt2/run_gpt2_txtcls.yaml000066400000000000000000000104201476650141400244400ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 # 0--Graph Mode; 1--Pynative Mode device_target: "Ascend" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 16 sink_mode: False sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True search_mode: "sharding_propagation" enable_parallel_optimizer: False full_batch: True parallel_config: data_parallel: 1 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: 'gpt2_txtcls' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 768 num_labels: 2 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float32" layernorm_compute_type: "float32" softmax_compute_type: "float32" compute_dtype: "float16" checkpoint_name_or_path: "gpt2" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: GPT2ForSequenceClassification # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0001 lr_end: 0.00001 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2_txtcls" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/gpt2/run_gpt2_xl.yaml000066400000000000000000000112361476650141400235500ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_xl' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 1600 num_layers: 48 num_heads: 25 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_xl" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False arch: type: GPT2LMHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-xl" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/gpt2/run_gpt2_xl_lora.yaml000066400000000000000000000114771476650141400245740ustar00rootroot00000000000000seed: 0 run_mode: 'finetune' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "gpt2_xl" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'gpt2_xl' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True tokenizer: type: GPT2Tokenizer max_length: 1025 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False tokenizer: type: GPT2Tokenizer max_length: 1024 input_columns: ["input_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: GPT2Config seq_length: 1024 vocab_size: 50257 hidden_size: 1600 num_layers: 48 num_heads: 25 expand_ratio: 4 hidden_act: "fast_gelu" hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 param_init_type: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" compute_dtype: "float16" checkpoint_name_or_path: "gpt2_xl_lora" eos_token_id: 50256 repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True use_past: False pet_config: pet_type: lora in_channels: 768 out_channels: 768 lora_rank: 64 lora_alpha: 128 lora_dropout: 0.1 arch: type: GPT2WithLora # lr sechdule lr_schedule: type: polynomial learning_rate: 0.0005 lr_end: 0.00001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: FusedAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "gpt2-xl" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: unk_token: '<|endoftext|>' bos_token: '<|endoftext|>' eos_token: '<|endoftext|>' pad_token: '<|endoftext|>' type: GPT2Tokenizer type: GPT2Processor pengcheng.mind-mindformer/configs/llama/000077500000000000000000000000001476650141400206305ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/llama/run_llama_13b.yaml000066400000000000000000000120751476650141400241400ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_13b_910b.yaml000066400000000000000000000120741476650141400246720ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910B parallel_config: data_parallel: 16 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "57GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_65b.yaml000066400000000000000000000121001476650141400241340ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_65b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.5e-4 lr_end: 1.5e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 128 910A parallel_config: data_parallel: 4 model_parallel: 2 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 64 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_65b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_65b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_65b_910b.yaml000066400000000000000000000120731476650141400247000ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_65b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_65b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_65b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_7b.yaml000066400000000000000000000120711476650141400240570ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b' # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_7b_910b.yaml000066400000000000000000000120671476650141400246170ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b' # runner config runner_config: epochs: 2 batch_size: 6 sink_mode: True sink_size: 4 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910B parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama/run_llama_7b_lora.yaml000066400000000000000000000124071476650141400250770ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: 'llama_7b' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'finetune' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_7b_lora' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 learning_rate: 1.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 2 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b_lora" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_dtype: "float32" softmax_compute_dtype: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 2048 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama_7b_lora" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False pet_config: pet_type: lora # configuration of lora in_channels: 4096 out_channels: 4096 lora_rank: 16 lora_alpha: 16 lora_dropout: 0.05 arch: type: LlamaForCausalLMWithLora processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # cfts init config remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/000077500000000000000000000000001476650141400207125ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/llama2/.ipynb_checkpoints/000077500000000000000000000000001476650141400245035ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/llama2/.ipynb_checkpoints/run_llama2_1b-checkpoint.yaml000066400000000000000000000124701476650141400321360ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 minimum_loss_scale: 1048576 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/.ipynb_checkpoints/run_llama2_1b_32-checkpoint.yaml000066400000000000000000000124151476650141400324410ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/.ipynb_checkpoints/run_llama2_1b_loss_scale-checkpoint.yaml000066400000000000000000000124151476650141400343440ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/.ipynb_checkpoints/run_llama2_7b-checkpoint.yaml000066400000000000000000000124161476650141400321440ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/predict_llama2_70b_910b.yaml000066400000000000000000000075541476650141400256760ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'predict' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: False micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" ascend_config: precision_mode: "must_keep_origin_dtype" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: True pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: True checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaProcessor vocab_file: "" type: LlamaProcessor # metric metric: type: PerplexityMetric eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_13b.yaml000066400000000000000000000124171476650141400243040ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_13b_910b.yaml000066400000000000000000000120771476650141400250410ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # runner config runner_config: epochs: 2 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 16 910B parallel_config: data_parallel: 16 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_13b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 5120 num_layers: 40 num_heads: 40 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_13b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_1b.yaml000066400000000000000000000124701476650141400242200ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 minimum_loss_scale: 1048576 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_1b_32.yaml000066400000000000000000000124151476650141400245230ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_1b_loss_scale.yaml000066400000000000000000000124151476650141400264260ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 8 model_parallel: 2 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 100 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_200M_cosD_tokens2b.yaml000066400000000000000000000124201476650141400271060ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 2.e-4 lr_end: 2.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: True eval_step_interval: 800 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: False gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama2_200M" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 1280 num_layers: 10 num_heads: 32 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_70b.yaml000066400000000000000000000124471476650141400243120ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 1.5e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 128 910A parallel_config: data_parallel: 4 model_parallel: 2 pipeline_stage: 16 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-6 bos_token_id: 1 eos_token_id: 2 pad_token_id: 32000 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LLaMATokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_70b_910b.yaml000066400000000000000000000124371476650141400250440ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_70b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # runner config runner_config: epochs: 2 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 32 910B parallel_config: data_parallel: 1 model_parallel: 8 pipeline_stage: 4 use_seq_parallel: False optimizer_shard: True micro_batch_num: 32 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_70b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 8192 num_layers: 80 num_heads: 64 vocab_size: 32000 multiple_of: 256 n_kv_heads: 8 ffn_dim_multiplier: 1.3 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_70b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LLaMATokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_7b.yaml000066400000000000000000000124161476650141400242260ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 2 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 20 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 4 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2560 num_layers: 20 num_heads: 20 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_7b_910b.yaml000066400000000000000000000120621476650141400247560ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 2 batch_size: 4 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 3.e-4 lr_end: 3.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910B parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 100 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "58GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 4096 num_layers: 32 num_heads: 32 vocab_size: 32000 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float16" rotary_dtype: "float16" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: True use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_width_256_dataset.yaml000066400000000000000000000125661476650141400271440ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_ratio: 0.01 total_steps: -1 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False # important input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 consumed_samples: 0 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 512 # 2304 num_layers: 10 num_heads: 8 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 minimum_loss_scale: 524288 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/llama2/run_llama2_width_256_dataset_resume.yaml000066400000000000000000000126621476650141400305210ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' restore_ckpt_name_prefix: 'llama_7b_rank_*-500_2.ckpt' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: True run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'llama2_7b' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 3.e-4 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-4 lr_end: 1.e-5 warmup_ratio: 0.01 total_steps: 8280 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False # important input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False prefetch_size: 1 consumed_samples: 64000 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 64 model_parallel: 1 pipeline_stage: 1 use_seq_parallel: False optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: True select_recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "llama_7b" save_checkpoint_steps: 1000 integrated_save: False async_save: False - type: ObsMonitor # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 512 # 2304 num_layers: 10 num_heads: 8 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: False offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 20 minimum_loss_scale: 524288 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/mae/000077500000000000000000000000001476650141400203045ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/mae/run_mae_vit_base_p16_224_800ep.yaml000066400000000000000000000113441476650141400264660ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 10 batch_size: 64 image_size: 224 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: masked_image_modeling model_name: 'mae_vit_base_p16' # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 64 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: RandomCropDecodeResize size: 224 scale: [0.2, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: Normalize mean: [123.675, 118.575, 103.53] std: [58.395, 62.22, 57.375] - type: HWC2CHW mask_policy: type: MaeMask input_size: 224 patch_size: 16 mask_ratio: 0.75 input_columns: ["image"] output_columns: ["image", "mask", "ids_restore", "unmask_index"] column_order: ["image", "mask", "ids_restore", "unmask_index"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: MIMDataset dataset_config: *train_dataset # model model: arch: type: ViTMAEForPreTraining model_config: type: ViTMAEConfig mask_ratio: 0.75 # mask ratio image_size: 224 # input image size patch_size: 16 # patch size num_channels: 3 # channels of input images initializer_range: 0.02 # initial std of cls_tokens and mask_tokens hidden_size: 768 # embedding dimension num_hidden_layers: 12 # number of transformer blocks num_attention_heads: 12 # number of attention heads intermediate_size: 3072 # ffn_hidden_size of encoder qkv_bias: True # if has bias in qkv dense hidden_act: gelu # activation of MLP post_layernorm_residual: False # if using post layernorm residual layer_norm_eps: 0.000001 # eps of layer_norm attention_probs_dropout_prob: 0. # drop rate of Attention hidden_dropout_prob: 0. # drop rate of MLP decoder_hidden_size: 512 # decoder embedding dim decoder_num_hidden_layers: 8 # number of decoder transformer blocks decoder_num_attention_heads: 16 # number of decoder attention heads decoder_intermediate_size: 2048 # ffn_hidden_size of decoder norm_pix_loss: True # if using norm pixel loss checkpoint_name_or_path: 'mae_vit_base_p16' # lr sechdule lr_schedule: type: cosine learning_rate: 0.00015 lr_end: 0. warmup_lr_init: 0. warmup_epochs: 40 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: type: ViTMAEProcessor image_processor: type: ViTMAEImageProcessor size: 224 # input image size pengcheng.mind-mindformer/configs/official_training/000077500000000000000000000000001476650141400232115ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/official_training/run_minicpm_ckpt-2B-v4.yaml000066400000000000000000000136041476650141400302320ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False reuse_ckpt: False # If true, data parallel reuse ckpt for optimizer_weight_shard_size 64 resume_merged_ckpt: False reset_steps: False unload_adam: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'naohai_2b6' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 1.e-3 weight_decay: 0.1 # lr sechdule lr_schedule: type: WarmUpCosineWithDecayLR # WarmUpCosineWithDecayLR learning_rate: 0.001 lr_middle: 0.001 # lr_middle==learning_rate is same to WarmUpStableDecayLR lr_end: 0.00001 warmup_ratio: 0.0004 decay_ratio: 0.15 total_steps: 1000000 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 1 numa_enable: False drop_skip: True prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 drop_skip: False eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True optimizer_weight_shard_size: 32 search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 8 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: False select_recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "naohai_2b6_" save_checkpoint_steps: 1000 keep_checkpoint_max: 1 integrated_save: False async_save: False - type: ObsMonitor keep_last: False step_upload_frequence: 1000 # keep same to save_checkpoint_steps # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31.5GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2304 num_layers: 40 num_heads: 36 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: True offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_2b6" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 depth_scale: 0.8 init_std: 0.025 embed_scale: 2.0 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 40 minimum_loss_scale: 32768 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/official_training/run_minicpm_ckpt-2B-v4_sft.yaml000066400000000000000000000135761476650141400311160ustar00rootroot00000000000000seed: 0 output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False reuse_ckpt: False # If true, data parallel reuse ckpt for optimizer_weight_shard_size 64 resume_merged_ckpt: False reset_steps: False unload_adam: False run_mode: 'train' # trainer config trainer: type: CausalLanguageModelingTrainer model_name: 'naohai_2b6' # runner config runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 1.e-8 # 1e-8 learning_rate: 1.e-3 weight_decay: 0.1 # lr sechdule lr_schedule: type: WarmUpCosineWithDecayLR # WarmUpCosineWithDecayLR learning_rate: 0.001 lr_middle: 0.001 # lr_middle==learning_rate is same to WarmUpStableDecayLR lr_end: 0.0 warmup_ratio: 0.0004 decay_ratio: 0.15 total_steps: 1000000 # -1 means it will load the total steps of the dataset # dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] # "input_ids", "labels" , labels are used in instruction finetune. num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 4 repeat: 3 numa_enable: False drop_skip: True prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 100 # num of epoch intervals between each eval, 1 means eval on every epoch end. # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False repeat: 1 numa_enable: False prefetch_size: 1 drop_skip: False eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset use_parallel: True # parallel context config parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False enable_alltoall: False full_batch: True optimizer_weight_shard_size: 32 search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_optimizer_config: gradient_accumulation_shard: False parallel_optimizer_threshold: 64 # default parallel of device num = 8 910A parallel_config: data_parallel: 32 model_parallel: 1 pipeline_stage: 2 use_seq_parallel: False optimizer_shard: True micro_batch_num: 8 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 # recompute config recompute_config: recompute: False select_recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: True # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "naohai_2b6_" save_checkpoint_steps: 200 keep_checkpoint_max: 1 integrated_save: False async_save: False - type: ObsMonitor keep_last: False step_upload_frequence: 200 # keep same to save_checkpoint_steps # mindspore context init config context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31.5GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 4096 hidden_size: 2304 num_layers: 40 num_heads: 36 vocab_size: 125696 multiple_of: 256 rms_norm_eps: 1.0e-5 bos_token_id: 1 eos_token_id: 2 pad_token_id: 0 ignore_token_id: -100 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" rotary_dtype: "float32" param_init_type: "float16" use_past: False pretrain_seqlen: 4096 # seqlen of the pretrain checkpoint: 2048 for llama and 4096 for llama2 extend_method: "None" # support "None", "PI", "NTK" compute_in_2d: False use_flash_attention: True offset: 0 use_past_shard: False checkpoint_name_or_path: "llama2_2b6" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False mup_width_base: 256 depth_scale: 0.8 init_std: 0.025 embed_scale: 2.0 arch: type: LlamaForCausalLM processor: return_tensors: ms tokenizer: unk_token: '' bos_token: '' eos_token: '' pad_token: '' type: LlamaTokenizer type: LlamaProcessor # metric metric: type: PerplexityMetric # wrapper cell config runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateWithMinimalCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 40 minimum_loss_scale: 32768 use_clip_grad: True eval_callbacks: - type: ObsMonitor auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True layer_scale: False layer_decay: 0.65 lr_scale_factor: 256 # aicc remote_save_url: "Please input obs url on AICC platform." pengcheng.mind-mindformer/configs/pangualpha/000077500000000000000000000000001476650141400216625ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/pangualpha/run_pangualpha_13b.yaml000066400000000000000000000115541476650141400262250ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "31GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 8 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # dp 4 mp 4 pp 1 for 2 nodes 16 devices micro_batch_num 1 # dp 4 mp 2 pp 4 for 4 nodes 32 devices micro_batch_num 32 # dp 16 mp 1 pp 8 for 16 nodes 128 devices micro_batch_num 80 parallel_config: data_parallel: 4 model_parallel: 4 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_13b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 8 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 5120 ffn_hidden_size: 20480 num_layers: 40 num_heads: 40 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_13b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-13b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessorpengcheng.mind-mindformer/configs/pangualpha/run_pangualpha_2_6b.yaml000066400000000000000000000113251476650141400263640ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 16 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: True parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids"] output_columns: ["input_ids", "position_id", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PerplexityMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessorpengcheng.mind-mindformer/configs/pangualpha/run_pangualpha_2_6b_em_f1.yaml000066400000000000000000000102431476650141400274310ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "" auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: True parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 16 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 16 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1.5 max_decode_length: 1024 max_new_tokens: 32 top_k: 3 top_p: 0.0 do_sample: True arch: type: PanguAlphaHeadModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: EmF1Metric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessor pengcheng.mind-mindformer/configs/pangualpha/run_pangualpha_2_6b_prompt_txtcls.yaml000066400000000000000000000103601476650141400313640ustar00rootroot00000000000000seed: 0 run_mode: 'eval' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: "pangualpha_2_6b" src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--disable_expand_ops=Softmax,Dropout --enable_parallel_fusion=true --reduce_fuse_depth=8 --enable_auto_tensor_inplace=true" max_call_depth: 10000 max_device_memory: "30GB" save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell loss_scale_value: 4294967296 scale_factor: 2 scale_window: 1000 use_clip_grad: True # parallel use_parallel: False parallel: parallel_mode: 1 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: False loss_repeated_mean: True full_batch: True search_mode: "sharding_propagation" enable_parallel_optimizer: True strategy_ckpt_save_file: "./ckpt_strategy.ckpt" # 1 node 8 device num parallel_config: data_parallel: 8 model_parallel: 1 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: True filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True # Trainer trainer: type: CausalLanguageModelingTrainer model_name: 'pangualpha_2_6b' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels", "attention_mask"] eos_reset: False num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset # model model: model_config: type: PanguAlphaConfig batch_size: 1 seq_length: 1024 vocab_size: 40000 hidden_size: 2560 ffn_hidden_size: 10240 num_layers: 32 num_heads: 32 pad_token_id: 6 eos_token_id: 6 post_layernorm_residual: False param_init_type: 'float32' compute_dtype: 'float16' softmax_compute_type: 'float16' embedding_dropout_prob: 0.1 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 hidden_act: 'fast_gelu' use_past: False use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 5 top_p: 1 do_sample: True num_labels: 4 arch: type: PanguAlphaPromptTextClassificationModel # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.000001 warmup_steps: 2000 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 lr_scale: False lr_scale_factor: 256 # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.95 eps: 0.00000001 # 1e-8 weight_decay: 0.1 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "PanguAlpha-2_6b" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: PromptAccMetric # processor processor: return_tensors: ms tokenizer: type: PanguAlphaTokenizer type: PanguAlphaProcessor pengcheng.mind-mindformer/configs/qa/000077500000000000000000000000001476650141400201435ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/qa/run_qa_bert_base_uncased.yaml000066400000000000000000000127161476650141400260330ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 2 batch_size: 12 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: QuestionAnsweringTrainer model_name: qa_bert_base_uncased # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: SQuADDataLoader dataset_dir: "./squad/" stage: "train" column_names: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True input_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] output_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] column_order: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] num_parallel_workers: 8 python_multiprocessing: True drop_remainder: True batch_size: 12 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: QuestionAnsweringDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: SQuADDataLoader dataset_dir: "./squad/" stage: "dev" column_names: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True input_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] output_columns: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] column_order: ["input_ids", "input_mask", "token_type_id", "start_position", "end_position", "unique_id"] num_parallel_workers: 8 python_multiprocessing: True drop_remainder: True batch_size: 12 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: QuestionAnsweringDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 12 seq_length: 384 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "qa_bert_base_uncased_squad" arch: type: BertForQuestionAnswering # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 3000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: SQuADMetric dataset_dir: "./squad/" # processor processor: return_tensors: ms type: BertProcessor tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' do_basic_tokenize: True do_lower_case: True pengcheng.mind-mindformer/configs/sam/000077500000000000000000000000001476650141400203225ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/sam/run_sam_vit-b.yaml000066400000000000000000000152701476650141400237600ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 768 depth: 12 num_heads: 12 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [2, 5, 8, 11] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_b" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-mindformer/configs/sam/run_sam_vit-h.yaml000066400000000000000000000152731476650141400237710ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 1280 depth: 32 num_heads: 16 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [7, 15, 23, 31] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_h" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-mindformer/configs/sam/run_sam_vit-l.yaml000066400000000000000000000152731476650141400237750ustar00rootroot00000000000000seed: 0 run_mode: 'predict' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-dataset, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_segmentation model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: Sam model_config: type: SAMConfig image_encoder: arch: type: SAMImageEncoder model_config: type: ImageEncoderConfig img_size: 1024 patch_size: 16 in_chans: 3 embed_dim: 1024 depth: 24 num_heads: 16 mlp_ratio: 4 out_chans: 256 qkv_bias: True use_abs_pos: True use_rel_pos: True # 仅支持默认False rel_pos_zero_init: True window_size: 14 global_attn_indexes: [5, 11, 17, 23] layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" prompt_config: arch: type: SAMPromptEncoder model_config: type: PromptEncoderConfig prompt_embed_dim: 256 image_embedding_size: [64, 64] input_image_size: [1024, 1024] mask_in_chans: 16 compute_dtype: "float16" decoder_config: arch: type: SAMMaskDecoder model_config: type: MaskDecoderConfig num_multimask_outputs: 3 # transformer: decoder_depth: 2 decoder_embed_dim: 256 decoder_mlp_dim: 2048 decoder_num_heads: 8 transformer_dim: 256 iou_head_depth: 3 iou_head_hidden_dim: 256 layer_norm_eps: 1.e-12 compute_dtype: "float16" layernorm_compute_type: "float32" softmax_compute_type: "float32" # std checkpoint_name_or_path: "sam_vit_l" # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SAMProcessor image_processor: type: SAMImageProcessor img_size: 1024 # input image size: int or [h, w] or (h, w) mean: [123.675, 116.28, 103.53] std: [58.395, 57.12, 57.375] pengcheng.mind-mindformer/configs/swin/000077500000000000000000000000001476650141400205225ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/swin/run_swin_base_p4w7_224_100ep.yaml000066400000000000000000000140701476650141400264230ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 128 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 use_clip_grad: True max_grad_norm: 5.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_classification model_name: 'swin_base_p4w7' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset seed: 0 batch_size: 128 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 0 batch_size: 128 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: SwinForImageClassification model_config: type: SwinConfig num_labels: 1000 # num classes image_size: 224 # input image size patch_size: 4 # patch size num_channels: 3 # channels of input images embed_dim: 128 # embedding dimension depths: [2, 2, 18, 2] # number of transformer blocks for each swin layer num_heads: [4, 8, 16, 32] # number of attention heads for each swin layer window_size: 7 # window size for swin mlp_ratio: 4 # ffn_hidden_size = mlp_ratio * embed_dim qkv_bias: True # has transformer qkv bias or not layer_norm_eps: 0.00001 # eps of layer_norm hidden_dropout_prob: 0. # drop rate of MLP attention_probs_dropout_prob: 0. # drop rate of Attention drop_path_rate: 0.1 # drop path rate of transformer blocks use_absolute_embeddings: False # if using absolute position embedding patch_norm: True # use norm in SwinPatchEmbeddings hidden_act: gelu # activation of MLP weight_init: normal # weight initialize type loss_type: SoftTargetCrossEntropy # loss type checkpoint_name_or_path: swin_base_p4w7 # lr sechdule lr_schedule: type: cosine learning_rate: 0.00125 lr_end: 0.00000025 warmup_lr_init: 0.00000025 warmup_epochs: 20 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.9 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 512 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: SwinProcessor image_processor: type: SwinImageProcessor size: 224 # input image size pengcheng.mind-mindformer/configs/t5/000077500000000000000000000000001476650141400200725ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/t5/run_t5_small_on_wmt16.yaml000066400000000000000000000107011476650141400251130ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell scale_factor: 2 scale_window: 1000 loss_scale_value: 65536 use_clip_grad: True max_grad_norm: 1.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: translation model_name: 't5' # train dataset train_dataset: &train_dataset data_loader: type: WMT16DataLoader dataset_dir: "/your/wmt_en_ro" tokenizer: type: t5_small src_max_length: 1024 # this will truncate the source input sequence tgt_max_length: 128 prefix: "translate the English to Romanian:" stage: 'train' input_columns: ["input_ids", "attention_mask", "labels"] # determinied by the model inputs output_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TranslationDataset dataset_config: *train_dataset # model model: arch: type: T5ForConditionalGeneration model_config: batch_size: 1 d_ff: 2048 hidden_size: 512 do_sample: false eos_token_id: 1 has_relative_bias: true hidden_act: relu hidden_dropout_prob: 0.1 initializer_factor: 1.0 initializer_range: 0.02 is_encoder_decoder: true d_kv: 64 layer_norm_epsilon: 1.0e-06 length_penalty_weight: 1.0 max_decode_length: 128 max_length: 32 max_position_embeddings: 1024 num_heads: 8 num_hidden_layers: 6 pad_token_id: 0 relative_attention_num_buckets: 32 repetition_penalty: 1 scale_output: true seq_length: 1024 top_k: 1 top_p: 0.95 type: T5Config use_cache: true vocab_size: 32128 checkpoint_name_or_path: t5_small hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 embedding_dropout_prob: 0.1 param_init_type: float32 layernorm_compute_type: float32 softmax_compute_type: float32 post_layernorm_residual: False offset: 0 use_past: False # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.0 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.0 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: LossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: max_length: 77 padding: max_length return_tensors: ms tokenizer: eos_token: pad_token: type: T5Tokenizer unk_token: type: T5Processor pengcheng.mind-mindformer/configs/t5/run_t5_tiny_on_wmt16.yaml000066400000000000000000000106321476650141400247710ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 1 batch_size: 1 sink_mode: True sink_size: 2 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: DynamicLossScaleUpdateCell scale_factor: 2 scale_window: 1000 loss_scale_value: 65536 use_clip_grad: True max_grad_norm: 1.0 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: translation model_name: 't5' # train dataset train_dataset: &train_dataset data_loader: type: WMT16DataLoader dataset_dir: "/your/wmt_en_ro" tokenizer: type: t5_small src_max_length: 1024 # this will truncate the source input sequence tgt_max_length: 128 prefix: "translate the English to Romanian:" stage: 'train' input_columns: ["input_ids", "attention_mask", "labels"] # determinied by the model inputs output_columns: ["input_ids", "attention_mask", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: False batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 train_dataset_task: type: TranslationDataset dataset_config: *train_dataset # model model: arch: type: T5ForConditionalGeneration model_config: batch_size: 1 d_ff: 512 hidden_size: 128 do_sample: false eos_token_id: 1 has_relative_bias: true hidden_act: relu hidden_dropout_prob: 0.1 initializer_factor: 1.0 initializer_range: 0.02 is_encoder_decoder: true d_kv: 64 layer_norm_epsilon: 1.0e-06 length_penalty_weight: 1.0 max_decode_length: 128 max_length: 32 max_position_embeddings: 1024 num_heads: 8 num_hidden_layers: 1 pad_token_id: 0 relative_attention_num_buckets: 32 repetition_penalty: 1 scale_output: true seq_length: 1024 top_k: 1 top_p: 0.95 type: T5Config use_cache: true vocab_size: 32128 hidden_dropout_rate: 0.1 attention_dropout_rate: 0.1 embedding_dropout_prob: 0.1 param_init_type: float32 layernorm_compute_type: float32 softmax_compute_type: float32 post_layernorm_residual: False offset: 0 use_past: False # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 lr_end: 0.0 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.0 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: LossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # processor processor: max_length: 77 padding: max_length return_tensors: ms tokenizer: eos_token: pad_token: type: T5Tokenizer unk_token: type: T5Processor pengcheng.mind-mindformer/configs/tokcls/000077500000000000000000000000001476650141400210415ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/tokcls/run_tokcls_bert_base_chinese.yaml000066400000000000000000000132771476650141400276260ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 24 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TokenClassificationTrainer model_name: tokcls_bert_base_chinese # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "train" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: TokenClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "dev" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: TokenClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 31 dropout_prob: 0.1 batch_size: 24 seq_length: 128 # length of input sentence vocab_size: 21128 # size of vocab embedding_size: 768 # size of text feature num_layers: 12 # model depth num_heads: 12 # number of attention heads expand_ratio: 4 hidden_act: "gelu" # activation post_layernorm_residual: True # select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "tokcls_bert_base_chinese" arch: type: BertForTokenClassification # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: EntityScore # processor processor: type: BertProcessor return_tensors: ms tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese pengcheng.mind-mindformer/configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml000066400000000000000000000133171476650141400313360ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 24 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TokenClassificationTrainer model_name: tokcls_bert_base_chinese_cluener # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "train" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 train_dataset_task: type: TokenClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: CLUENERDataLoader dataset_dir: "./cluener/" stage: "dev" column_names: ["text", "label_id"] text_transforms: type: TokenizeWithLabel max_length: 128 padding: "max_length" label_transforms: type: LabelPadding max_length: 128 padding_value: 0 tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese input_columns: ["text", "label_id"] output_columns: ["input_ids", "token_type_ids", "attention_mask", "label_id"] column_order: ["input_ids", "token_type_ids", "attention_mask", "label_id"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 24 repeat: 1 numa_enable: False prefetch_size: 30 seed: 2022 eval_dataset_task: type: TokenClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 31 dropout_prob: 0.1 batch_size: 24 seq_length: 128 # length of input sentence vocab_size: 21128 # size of vocab embedding_size: 768 # size of text feature num_layers: 12 # model depth num_heads: 12 # number of attention heads expand_ratio: 4 hidden_act: "gelu" # activation post_layernorm_residual: True # select postlayernorm or prelayernorm hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "tokcls_bert_base_chinese_cluener" arch: type: BertForTokenClassification # lr sechdule lr_schedule: type: linear learning_rate: 0.00003 # 3e-5 warmup_ratio: 0.1 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: EntityScore # processor processor: type: BertProcessor return_tensors: ms tokenizer: type: BertTokenizer cls_token: '[CLS]' mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' unk_token: '[UNK]' is_tokenize_char: True do_lower_case: False checkpoint_name_or_path: tokcls_bert_base_chinese pengcheng.mind-mindformer/configs/txtcls/000077500000000000000000000000001476650141400210635ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/txtcls/run_txtcls_bert_base_uncased.yaml000066400000000000000000000105771476650141400276760ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 64 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-standalone, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: txtcls_bert_base_uncased # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/train" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/eval" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 64 seq_length: 128 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "txtcls_bert_base_uncased" arch: type: BertForMultipleChoice # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 # 5e-5 lr_end: 0.000001 # 1e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' type: BertProcessor top_k: 1 pengcheng.mind-mindformer/configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml000066400000000000000000000106111476650141400307020ustar00rootroot00000000000000seed: 42 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False save_graphs_path: "./graph" device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 3 batch_size: 64 sink_mode: False sink_size: 2 runner_wrapper: type: TrainOneStepCell # parallel use_parallel: False parallel: parallel_mode: 0 # 0-standalone, 1-semi, 2-auto, 3-hybrid gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: TextClassificationTrainer model_name: txtcls_bert_base_uncased_mnli # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/train" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 train_dataset_task: type: TextClassificationDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset data_loader: type: TFRecordDataset dataset_dir: "./mnli/eval" shuffle: True input_columns: ["input_ids", "input_mask", "segment_ids", "label_ids"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 64 repeat: 1 numa_enable: False prefetch_size: 1 seed: 42 eval_dataset_task: type: TextClassificationDataset dataset_config: *eval_dataset # model model: model_config: type: BertConfig use_one_hot_embeddings: False num_labels: 3 dropout_prob: 0.1 batch_size: 64 seq_length: 128 vocab_size: 30522 embedding_size: 768 num_layers: 12 num_heads: 12 expand_ratio: 4 hidden_act: "gelu" post_layernorm_residual: True hidden_dropout_prob: 0.1 attention_probs_dropout_prob: 0.1 max_position_embeddings: 512 type_vocab_size: 2 initializer_range: 0.02 use_relative_positions: False use_past: False compute_dtype: "float32" checkpoint_name_or_path: "txtcls_bert_base_uncased_mnli" arch: type: BertForMultipleChoice # lr sechdule lr_schedule: type: polynomial learning_rate: 0.00005 # 5e-5 lr_end: 0.000001 # 1e-6 warmup_steps: 0 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: False layer_decay: 0.65 # optimizer optimizer: type: adamw weight_decay: 0.01 eps: 0.00000001 # 1e-8 lr_scale: False lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 1000 integrated_save: True async_save: False eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: return_tensors: ms tokenizer: cls_token: '[CLS]' do_basic_tokenize: True do_lower_case: True mask_token: '[MASK]' pad_token: '[PAD]' sep_token: '[SEP]' type: BertTokenizer unk_token: '[UNK]' type: BertProcessor top_k: 1 pengcheng.mind-mindformer/configs/vit/000077500000000000000000000000001476650141400203445ustar00rootroot00000000000000pengcheng.mind-mindformer/configs/vit/run_vit_base_p16_224_100ep.yaml000066400000000000000000000136671476650141400257070ustar00rootroot00000000000000seed: 0 run_mode: 'train' output_dir: './output' # 当前不支持自定义修改，请勿修改该默认值 load_checkpoint: '' src_strategy_path_or_dir: '' auto_trans_ckpt: False # If true, auto transform load_checkpoint to load in distributed model only_save_strategy: False resume_training: False # context context: mode: 0 #0--Graph Mode; 1--Pynative Mode device_target: "Ascend" enable_graph_kernel: False graph_kernel_flags: "--opt_level=0" max_call_depth: 10000 save_graphs: False device_id: 0 # aicc remote_save_url: "Please input obs url on AICC platform." # runner runner_config: epochs: 100 batch_size: 32 image_size: 224 sink_mode: True sink_size: 2 num_classes: 1000 runner_wrapper: type: MFTrainOneStepCell scale_sense: type: FixedLossScaleUpdateCell loss_scale_value: 1024 # parallel use_parallel: False parallel: parallel_mode: 0 # 0-data parallel, 1-semi-auto parallel, 2-auto parallel, 3-hybrid parallel gradients_mean: True enable_alltoall: False full_batch: False search_mode: "sharding_propagation" enable_parallel_optimizer: False strategy_ckpt_save_file: "./ckpt_strategy.ckpt" parallel_config: data_parallel: 1 model_parallel: 1 expert_parallel: 1 pipeline_stage: 1 optimizer_shard: False micro_batch_num: 1 gradient_aggregation_group: 4 micro_batch_interleave_num: 1 # moe moe_config: expert_num: 1 capacity_factor: 1.05 aux_loss_factor: 0.05 num_experts_chosen: 1 # recompute recompute_config: recompute: False parallel_optimizer_comm_recompute: False mp_comm_recompute: True recompute_slice_activation: False # autotune auto_tune: False filepath_prefix: './autotune' autotune_per_step: 10 # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: False profile_communication: False profile_memory: True # Trainer trainer: type: image_classification model_name: 'vit_base_p16' # if True, do evaluate during the training process. if false, do nothing. # note that the task trainer should support _evaluate_in_training function. do_eval: False # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: RandomResizedCrop size: 224 scale: [0.08, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: ToPIL - type: rand_augment_transform config_str: "rand-m9-mstd0.5-inc1" hparams: translate_const: 100 img_mean: [124, 116, 104] interpolation: cubic - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] # [123.675, 118.575, 103.53] std: [0.229, 0.224, 0.225] # is_hwc: False - type: RandomErasing probability: 0.25 mode: pixel max_count: 1 mixup_op: mixup_alpha: 0.8 cutmix_alpha: 1.0 cutmix_minmax: prob: 1.0 switch_prob: 0.5 label_smoothing: 0.1 input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: ImageCLSDataset dataset_config: *train_dataset # eval dataset eval_dataset: &eval_dataset seed: 2022 batch_size: 32 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/val" num_parallel_workers: 8 shuffle: True transforms: - type: Decode - type: Resize size: 256 interpolation: cubic - type: CenterCrop size: 224 - type: ToTensor - type: Normalize mean: [0.485, 0.456, 0.406] std: [0.229, 0.224, 0.225] is_hwc: False input_columns: ["image", "label"] output_columns: ["image", "label"] column_order: ["image", "label"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 eval_interval: 1 eval_offset: 0 eval_dataset_task: type: ImageCLSDataset dataset_config: *eval_dataset # model model: arch: type: ViTForImageClassification model_config: type: ViTConfig image_size: 224 # input image size patch_size: 16 # patch size num_channels: 3 # channels of input images initializer_range: 0.02 # initial std of cls_tokens, pos_embed and linear weight hidden_size: 768 # embedding dimension num_hidden_layers: 12 # number of transformer blocks num_attention_heads: 12 # number of attention heads intermediate_size: 3072 # ffn_hidden_size of encoder qkv_bias: True # if has bias in qkv dense hidden_act: gelu # activation of MLP post_layernorm_residual: False # if using post layernorm residual layer_norm_eps: 0.000001 # eps of layer_norm attention_probs_dropout_prob: 0. # drop rate of Attention hidden_dropout_prob: 0. # drop rate of MLP drop_path_rate: 0.1 # drop path rate of Block use_mean_pooling: True # if using global mean pooling num_labels: 1000 # num classes loss_type: SoftTargetCrossEntropy # loss type checkpoint_name_or_path: 'vit_base_p16' # lr sechdule lr_schedule: type: cosine learning_rate: 0.0005 lr_end: 0.000001 warmup_epochs: 5 total_steps: -1 # -1 means it will load the total steps of the dataset layer_scale: True layer_decay: 0.65 # optimizer optimizer: type: adamw beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 lr_scale: True lr_scale_factor: 256 # callbacks callbacks: - type: MFLossMonitor - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor # metric metric: type: Accuracy eval_type: classification # processor processor: type: ViTProcessor image_processor: type: ViTImageProcessor size: 224 # input image size pengcheng.mind-mindformer/docs/000077500000000000000000000000001476650141400170425ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/FAQ.md000066400000000000000000000117151476650141400200000ustar00rootroot00000000000000# FAQ 目录 [1.1 如何给MindSpore贡献代码](#11-如何给MindSpore贡献代码) [1.2 多次提交记录，如何进行合并？](#12-多次提交记录如何进行合并) [1.3 如何提交只有一个commit？](#13-如何提交只有一个commit) [1.4 多人协作，如何解决代码上传冲突？](#14-多人协作如何解决代码上传冲突) [1.5 开发过程中遇到问题，如果在开源社区中进行提问？](#15-开发过程中遇到问题如果在开源社区中进行提问) [1.6 PR提交者是否需要签署CLA？](#16-pr提交者是否需要签署cla) [1.7 PR提交时显示存在冲突，无法自动合并？](#17-pr提交时显示存在冲突无法自动合并) [1.8 CI门禁检查提示的错误不在PR提交的内容中？](#18-ci门禁检查提示的错误不在pr提交的内容中) ## 1. FAQ 在查看FAQ之前，请仔细阅读[代码提交注意指南](https://gitee.com/mindspore/vision/wikis/%E4%BB%A3%E7%A0%81%E6%8F%90%E4%BA%A4%E6%B3%A8%E6%84%8F%E6%8C%87%E5%8D%97) ### 1.1 如何给MindSpore贡献代码 请参考bilibili视频连接：[如何给MindSpore贡献代码](https://www.bilibili.com/video/BV1kg411F7Pc/) ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/171942_e6202765_7386274.png "屏幕截图.png") 视频会手把手带着大家如何贡献代码到gitee、如何fork MindSpore仓，并进行代码提交，解决git中username冲突等的问题。请一定要先观看该视频哦，会给您节省后续很多麻烦！ ### 1.2 多次提交记录，如何进行合并 - 问题描述： MindSpore transformer代码仓提交规范要求，提交记录仅保留一条，所以多次提交时，需要进行合并。 ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/171522_cbcd9488_7386274.png "屏幕截图.png") - 解决方案： 1. 通过 git log 查看代码提交的log记录，使用 git rebase -i 选择要合并的 commit 2. 编辑要合并的版本信息，即修改需要合并的 commit id 前面的pick修改为 squash，并保存提交 3. 修改注释信息后，保存提交，多条合并会出现多次 4. git push 推送远程仓库或合并到主干分支 ### 1.3 如何提交只有一个commit - 问题描述： 为了开源规范和方便审核，注意提交的时候只能有一个 commit： ![输入图片说明](https://images.gitee.com/uploads/images/2022/0414/170606_2dcf4944_7386274.png "屏幕截图.png") - 解决方案： 1. 修改代码之前，请先git fetch --all; git pull; 拉取最新的代码再修改 2. 如果已经提交PR，此时可以基于上一个PR进行修改，使用 git add *; git commit --amend; git push -f; 3. 通过commit --amend可以把当前的修改叠加到上一次代码中 ### 1.4 多人协作，如何解决代码上传冲突 可能是已经有同学先合入代码了，再合入代码时，会报冲突。需要先把个人仓先更新一下。更新方法是： `>>> git remote add upstream https://gitee.com/mindspore/vision.git` 这就把远端主仓添加到 upstream 里面了，然后使用命令 git merge： `>>> git rebase upstream/master` ### 1.5 开发过程中遇到问题，如果在开源社区中进行提问？ 在[transformer仓的issues区](https://gitee.com/mindspore/transformer/issues)提交issues，按照模板来描述问题，我们的工程师会在下面的comment留言哦。 ### 1.6 PR提交者是否需要签署CLA？ PR的提交者，每一个都要根据提醒来签署CLA的，CLA是保障大家贡献的代码都能够得到开源保障，声明开发者是你本人哦。 ### 1.7 PR提交时显示存在冲突，无法自动合并？ 由于存在多人协作开发，因此在开发期间如果没有进行及时同步代码仓，则有可能出现冲突，在PR提交时无法自动合并。例如A团队的commit4与B团队的commit5存在冲突时，A团队的PR则无法提交。因此建议在PR提交前，同步最新的master分支，然后对开发分支进行rebase master操作。 ### 1.8 CI门禁检查提示的错误不在PR提交的内容中？ 无论是使用pycharm还是vscode等IDE进行开发，**请一定要安装autoPEP8等格式代码规范检查的工具**，每一次进行代码修改后，需要规范化代码，然后再进行提交。避免CI要等和阻塞等问题。 因部分静态检查工具引入或升级时，代码中已经存在一些拼写问题，而CI门禁中检查工具并不会对仓库中所有代码进行检查，所以可能会出现PR提交内容之外的错误。该问题涉及的是静态检查工具引入或升级后，该文件首次被修改，一般为一些简单的拼写错误，请按提示进行修正后重新触发门禁。 如果静态检查提示了一些错误，但是按照提示内容检查PR文件却未找到相关代码，请rebase最新代码后再按照提示进行修正。此问题出现的原因是三方开源检查工具小概率失效，导致错误放行并合入代码后，其它PR再次检查该处时报错。pengcheng.mind-mindformer/docs/aicc_cards/000077500000000000000000000000001476650141400211155ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/aicc_cards/README.md000066400000000000000000000330571476650141400224040ustar00rootroot00000000000000# AICC上使用Mindformers教程 AICC，人工智能计算中心，提供[ModelArts](https://support.huaweicloud.com/bestpractice-modelarts/modelarts_10_0080.html)服务和Ascend 910算力。 MindFormers套件当前具备AICC适配的特性，用户在ModelArts平台启动集群训练时，无需额外增加上云训练适配代码，套件将自动进行： 1. 从OBS拉取训练所需数据至训练镜像 2. 统一训练输出路径 3. 训练后将训练输出整体回传指定文件夹 4. 训练过程中，通过回调函数将训练输出回传至指定文件夹 下面将提供在AICC上完成GPT2模型的训练样例，用户可以按照此样例学习MindFormers套件在ModelArts训练作业上的使用方式 使用前需对AICC的[AI开发平台ModelArts](https://support.huaweicloud.com/modelarts/index.html)，[对象存储服务 OBS](https://support.huaweicloud.com/obs/index.html)和[容器镜像服务 SWR](https://support.huaweicloud.com/swr/index.html)有初步的了解 ## 准备工作 ### 模型准备 本案例使用MindFormers套件内的GPT2模型作为教程案例，请参照[GPT2](../model_cards/gpt2.md)进行模型代码和训练数据集的准备 - 模型代码 ```bash git clone https://gitee.com/mindspore/mindformers.git ``` - 数据集预处理 以Wikitext2数据集为例 - 数据集下载：[WikiText2数据集](https://gitee.com/link?target=https%3A%2F%2Fs3.amazonaws.com%2Fresearch.metamind.io%2Fwikitext%2Fwikitext-2-v1.zip) - 词表下载：[vocab.json](https://gitee.com/link?target=https%3A%2F%2Fhuggingface.co%2Fgpt2%2Fblob%2Fmain%2Fvocab.json)，[merges.txt](https://gitee.com/link?target=https%3A%2F%2Fhuggingface.co%2Fgpt2%2Fresolve%2Fmain%2Fmerges.txt) - 参考[ModelZoo](https://gitee.com/mindspore/models/tree/master/research/nlp/gpt2#language-modeling-%E8%AF%AD%E8%A8%80%E5%BB%BA%E6%A8%A1%E4%BB%BB%E5%8A%A1)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ### 云上对象存储准备 准备好代码与数据集后，需将其上传至AICC的对象存储服务上 - 进入`存储`标题下的`对象存储服务`，将训练代码和数据等输入内容上传，并准备好用于回传训练输出的路径； ![obs_prepare](assets/obs_prepare.png) - 进入`对象存储服务`的任一文件夹，点击界面上方的复制按钮，即可获得当前文件夹的OBS路径映射 如：obs://huawei/xxx/mindformers ![obs_url_bt](assets/obs_url_bt.png) ### 镜像准备 ModelArts上的所提供的预置训练镜像，通常MindSpore的版本较为老旧，不满足MindFormers套件的运行需求，所以通常需要自定义镜像，安装合适版本的MindSpore包以运行套件 我们在[镜像仓库网 (hqcases.com)](http://mirrors.cn-central-221.ovaijisuan.com/)上发布了一些经过验证的**标准镜像版本**，可以通过几行简单的docker命令的形式，直接使用验证过的标准镜像拉起MindFormers套件的训练任务，而无需进行较为繁琐的自定义镜像并上传的步骤 - 在镜像仓库网上找到当前版本的[MindFormers镜像](http://mirrors.cn-central-221.ovaijisuan.com/detail/78.html)，打开显示如下（截图部分仅供参考，具体镜像版本以链接为主） ![aihub](assets/aihub.png) > 可以复制其docker pull命令拉取该镜像，在镜像仓库网上的镜像权限均为公开，无需登录可以直接拉取 > > 注：当前版本镜像为每日开发版，稳定版将在后续推送至镜像仓库网 - 镜像列表 ```text 1. swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 ``` - 在一台准备好docker引擎的计算机上，root用户执行docker pull命令拉取该镜像 ```bash docker pull swr.cn-central-221.ovaijisuan.com/mindformers/mindformers_dev_mindspore_1_10_1:mindformers_0.6.0dev_20230615_py39 ``` - 进入`容器镜像服务`的控制台界面，找到`客户端上传`镜像按钮，将会提示如何上传上一步拉取的镜像 ![mirrors_upload](assets/mirrors_upload.png) 按照操作执行完后将会在该镜像列表中看到上传的镜像，并能够在ModelArts中选取 ## 拉起训练流程 ### 训练作业配置 - 进入`ModelArts控制台`的训练作业功能区，点击页面右上角的`创建作业`按钮，开始创建训练作业 ![train_entry](assets/train_entry.png) ![train_task1](assets/train_task1.png) - 名称/描述/新建实验名称：按实际填写即可 - 创建方式/启动方式： 我们需要使用之前准备的镜像环境，但拉取方式仍可复用预置命令 ![choose_mirror](assets/choose_mirror.png) 这里选择自定义，即可在镜像一栏选择之前准备并上传好的镜像 - 代码目录选择上传的代码路径 - 启动文件选择run_mindformers.py脚本 > 注意，在AICC上拉起训练任务时，即使需要多卡启动，也只需指定python脚本作为启动文件，作业启动引擎会自动完成ranktable与环境变量等内容的配置，并多进程拉起python脚本，不需要额外的分布式启动shell脚本 - 本地代码目录与工作目录保持默认即可 ![train_nodes](assets/train_nodes.png) - 按需求选择资源池，规格和计算节点数，配置作业日志回传路径 卡数=单节点规格数 \* 节点数 **下面将较为详细地解释不同情况下拉起训练时所需的每个入参及其含义** ### 从头训练 ![train_args](assets/train_args.png) - 训练输入： - `train_dataset_dir`：选择数据集在云上的数据存储文件夹位置，作为训练数据集的路径；该参数名应当能够被启动脚本识别 > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 训练输出： - 启用时，需要将文件输出在超参传入的路径下，训练结束后会进行回传； - 由于输出通常较大，默认的路径下磁盘空间不足，我们会将输出统一放在/cahce目录下，因此MindFormers**不启用**该项参数 - 必需超参： - `config`：填入对应config文件在训练服务器上的绝对路径，在本案例下为 `/home/ma-user/modelarts/user-job-dir/mindformers/configs/gpt2/run_gpt2.yaml`，亦或是通过环境变量`/${MA_MOUNT_PATH}/mindformers/configs/gpt2/run_gpt2.yaml` - `run_mode`：train，训练模式，将不加载权重进行训练 - `use_parallel`：True，在所选规格参数大于1卡时需要设置为True，单卡运行为False - `remote_save_url`：obs://xxx/xxx/outputs/gpt2_test，填入对象存储服务中实际准备的OBS路径，**训练输出将会回传至该路径下，需要显式写出obs://** > 其余可选超参可参照 `run_mindformers.py` 脚本入参进行选择 ### 全参微调 ![finetune_args](assets/finetune_resume.png) 以下列出需额外增加部分参数： - 训练输入： - `load_checkpoint`: 在finetune模式下必须指定，需要finetune的权重的obs路径(通过数据存储位置直接选择obs中位置) > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 必需超参： - `run_mode`：finetune，微调模式，必须加载权重（在训练输入中设置） - `resume_training`: 断点续训所需参数，正常微调需设为false，或者不设置 > 其余输入/输出/超参与全参微调一致 ### 低参微调 使用lora等算法进行低参微调时，与全参微调没有区别，需要把config指定为使用lora微调的config - 训练输入： - `load_checkpoint`: 低参微调时，该路径与全参微调一致，需指定finetune使用的原始权重 > 选择后，ModelArts将会在拉起训练作业时，将**云上的数据集拉取至指定的目录下**，然后将该入参的值从云上路径修改为镜像上被拉取的路径 - 必需超参： - `config`：填入对应config文件在镜像上的绝对路径，在本案例下为 `/home/ma-user/modelarts/user-job-dir/mindformers/configs/gpt2/run_gpt2_lora.yaml` > 其余输入/输出/超参与全参微调一致 ### [断点续训](../feature_cards/Resume_Training.md) 训练或微调，遇到训练中断的情况，如果保存了ckpt，则可以保存处继续训练。 如果`run_mode`为`train`的情况下训练中断，恢复训练步骤如下： 1. 重建训练 2. 其余设置不改变，增加超参`resume_training`设置为true 3. 增加训练输入`load_checkpoint`，通过数据存储位置直接选择obs中需要恢复的ckpt 4. 拉起训练即可加载权重，优化器状态，loss scale，step等继续训练 如果`run_mode`为`finetune`的情况下训练中断，恢复训练步骤如下： 1. 重建训练 2. 其余设置不改变，增加超参`resume_training`设置为true 3. 需要变更`load_checkpoint`，通过数据存储位置选择obs中需要恢复的ckpt 4. 拉起训练即可加载权重，优化器状态，loss scale，step等继续训练 finetune恢复训练示例如下 ![finetune_args](assets/finetune_resume.png) ### 权重保存相关修改 保存ckpt相关的配置需要修改`config`配置文件中`CheckpointMointor` `save_checkpoint_steps`:可修改保存ckpt频率; `prefix`:可修改保存ckpt名字前缀 ```yaml # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "gpt2" save_checkpoint_steps: 500 integrated_save: False async_save: False - type: ObsMonitor eval_callbacks: - type: ObsMonitor ``` ### 训练结果查看 完成后拉起训练作业，耐心等待输出作业日志 如果一切正常，将会输出如下日志： ![train_logs](assets/train_logs.png) **并且在 `remote_save_url` 参数所对应的OBS路径下有输出回传**： ![train_outputs](assets/train_outputs.png) ### 常见问题 - 启动作业后报**module not found**： 标准镜像可能缺少几个pip安装包，将相应的包名加入代码主目录下的requirements.txt文件下，拉起时会自动读取并安装相应的包 - 回传过于频繁导致影响训练 可以通过所训练模型的配置文件`callback`中`ObsMonitor` 新增参数`upload_sequence`：表示多少step回传一次output目录下的内容到obs中。 ```yaml # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True - type: CheckpointMointor prefix: "mindformers" save_checkpoint_steps: 100 integrated_save: True async_save: False - type: ObsMonitor upload_sequence: 10000 eval_callbacks: - type: ObsMonitor ``` ### [权重离线切分转换](../feature_cards/Transform_Ckpt.md) 注意：`mindformers/tools/transform_ckpt.py`转换接口仅在`MindSpore2.0`环境下使用，可以在`NoteBook`中安装`MindSpore2.0 CPU`版本，进行离线转换； - step1 在`config`或者`训练作业`中配置超参`only_save_strategy=True`，拉起训练作业，仅生成分布式策略文件。 生成的分布式策略文件保存在`remote_save_url/strategy`目录下。(`remote_save_url`表示obs存储路径) 训练作业界面中增加`only_save_strategy`超参 ![](https://foruda.gitee.com/images/1687916968422798842/7975fa97_11500692.png) - step2 启动`NoteBook`任务，运行如下脚本完成权重切分转换（注意在`MindSpore 2.0`版本下） ```shell python mindformers/tools/transform_ckpt.py --src_ckpt_strategy SRC_CKPT_STRATEGY --dst_ckpt_strategy DST_CKPT_STRATEGY --src_ckpt_dir SRC_CKPT_DIR --dst_ckpt_dir DST_CKPT_DIR ``` 参数说明 `src_ckpt_strategy`：待转权重的分布式策略文件路径。 若为None,表示待转权重为完整权重; 若为切分策略文件,表示原始的权重对应的策略文件; 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`SRC_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `dst_ckpt_strategy`：目标权重的分布式策略文件路径。即step1中生成的分布式策略文件路径。 若为None,表示将待转权重合并为完整权重; 若为切分策略文件,表示目标卡数对应的策略文件 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`DST_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `src_ckpt_dir`: 待转权重路径，须按照`SRC_CKPT_DIR/rank_{i}/checkpoint_{i}.ckpt`存放，比如单一权重存放格式为`SRC_CKPT_DIR/rank_0/checkpoint_0.ckpt`。 `dst_ckpt_dir`：目标权重保存路径，为自定义空文件夹路径，转换后模型以`DST_CKPT_DIR/rank_{i}/xxx.ckpt`存放。 - step3 将切分转换好的权重传到`OBS` - step4 训练作业界面中增加`load_checkpoint`超参，输入转换好的checkpoint的文件夹obs路径。 ### [profiler](../feature_cards/Performance_Tuning.md#profiler数据采集) 训练作业界面中增加`profile`超参，设置为True，即可开启profile功能。此时profiler参数均为默认值。 采集完毕后，profiler文件数据会回传至`remote_save_url`路径下 ### [dump](../feature_cards/Precision_Optimization.md#dump使用方法) 参考高阶特性精度调优中dump使用方法，创建json文件，并上传至obs中。 json中dump数据输出路径指定为`mindformers/output/dump` 训练作业界面中增加`MINDSPORE_DUMP_CONFIG`环境变量，并设置为modelarts中json文件绝对路径。 训练作业结束后，dump数据会自动回传至obs中。 pengcheng.mind-mindformer/docs/aicc_cards/assets/000077500000000000000000000000001476650141400224175ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/aicc_cards/assets/aihub.png000066400000000000000000002161031476650141400242200ustar00rootroot00000000000000PNG  IHDRP3 pHYs+tEXtSoftwareSnipaste] IDATx\Te?k+ iXuh$\((" bB*f+jD]fi?f; e QE$3g`a9gsϙ9ǹ:}r9tuzA, `A, `A, `A, ` ?O~F}ƍza=O.wu_s3N酸|YvaW,}$]LpwS""+圿yŭʵ̖Sn%WnXֆ\3սھtehG*WtبX!&l̛?DTrfiɅҊő~}H>Q,}OP?_< ixު (Z2tǢߴOe Zg;sFŚQ*ܗwN*hDdl`5Jjqf|>=lx_Tp0Neg/o;~@*jFU]g f"\Nr0'd3QHص$'E3rҙecK%"_ZU;Z/nl!Dlgv rn ZQq_ZQ9}%\6N@Zm(b˾ӒK7-bu9DA{<.t7Ϟ~'GDD<bHY,P>U%GĮerZ1e;)XjFUO`jYR~8e D](wk{](^tS]K'H~\Fwt'OC~OڅF%bUVW4[lH8UHhgu+)&tb"ufT+d9&rn֚Zvh?]/}?WQeUgq脥wrSVN>7: tT ]kٿrkUpufr5꥕Ѫ&'8e׭V9(:RҿUwwU^5finÛyF}_qg>Lyo˟<3vsuq7 Dܥ_{(S- v e6fY|9 9p&"JOw>tkUbX~;ugǯ(ܱw".nWle't:B"Ouu9;C0Z@GsvIltݭ2yO?FDTS= SQ.yKD;v_,Y;7twDFzW[ݪO\}Rk<],zb&B ײ-%R?y*6 , k=]֪/() uͻ\1%4مgMl&=(Y`KeMHDDyw!Itl٪ qRf*eivr9q5HTjk.r3mk[!"xo"6. }Ek3ʥDW*{*|T_qYW#Շil[Z<(-+Uǚ4rP푓^ǻ޶m ;7]9YI^ɀn]ּ?inz:zAU e%:]cg9ݩmxl)+x<׊Ji2iΛO1rv;KOs ?QGY__}߼)<S7'={pG'^!-^ej<ϫ[leyfz;T{|okbݽ}zɍѹ-9:H0[cL@%sdBY(/(ۉSݽvT)޳[مx~WLXQ^"9L1[EBUB(\%9[*UߦD'_&d焇~#`}ⷪlg{l|-L]^z?V/-V^)_rCeYZ;=1@4qxv% Νp0;DWd؉$8 W\s壯hE8DW{wzl#uLQ.J{Ǣu9s^av@yw[%v=WG4`K{m*ظ`ˣGCK`T #ގ`,Lђ+oމǦby$4OZi_b" pPIK;K|Kv^x:YRV{Q{ K)fiC1=}o1wl%7o%w1ܶk+.49+Φm;zdLpf""rpką!41w|AÓv") k}~iD"|UAr:u$S6m>PN ƛќF9yOSsb'uѦ1qCrghŜvTԏopV㻣0^Un3vD3W_BK"*;,vX`THFǓkQ`FIL-NB= L-5Nax3|;?ײoh4S//ot)!^5nr˺v6ܼMd.x:3hH{"Wh6iؚ1L7qxdWHӯ=1q _mu{+f|GhV0!@y8'Vgbi":Yq%evuO&h/ ɳ?UCg?C {G_R՚K뚍Ysj]bZZqjZk"-{-qrug_:?&tՔθ[/<޶Dlg̪+ '`tV'MXnf>@/4ֲKyێbl9g_ə349yL5/⯊ݫ2a~i:5&9o22cٝu2f\vx\\N_+w xmH`'viZUc76yVN Vmg:̄몗V۹04N>YE[s[g>lx]|cB_Ohٵ9+/֎jrg9a[9s4#uV>KCt ~ z<|&obяw}jٹI&V 9癊pYv4Vs/_znKQ[ܗyֽkȹR媋T.G{]VoGmVZ3ZsRMf_Zs_zi=]* ܚB@ ~ܘq]-^K[pYṶfFI^UCsҽa~iEE,=Es53WvP[m.&FV95qFfFؖx$+͎$~9o66)8ME+)Ig7tnql>[;{:Kݐq_fE Y^-n^ꥉm&uK+KĭVv\Վr7}k!1Wccq]݋n[Czr?UUQQ`ƩVRV.RK+ˇ9ΚluYN3=5VHw7wůX6$M/kݻH/k?{MuXl~x Ӧ6kؙFx4b<;5M.ƈߜl3wKn8+3Ϫ]*R˚g+&Кf>{Zr+$W[Xaܯ(G[4YTy):=lxyitRK7œ $ f;Ӓ  O6ɸ_߱CM;ؘwu_Kͻu= y< ?&"# ϵekS ʫ7>};w <c񒉟n0oBGv4U>|,/IЊ8 @=i|zN]ayͻuv]Gߨ S'i?tXt={&ox3ym<2Ǹ_>}0^1t,.x X0 X0 chٮ׻ cWw!/ `\.>$K4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK4aK[\ IDAT4aK4aK4aK4aKƨQrfiɅҊ'O;aw=qc.,v1^xrCwPr]i\]^0ס6R"'{(ދb':IFZ IrjL (DT$QuvjBh@ `&3/pΆ⺁w{jl)?~-L˓wEy/-,_#G4'?zQFjS#=EFKB$&,׈T Pnvo~#uɌ ݕ}DNE&g ЙLG/ K[u(gDtAW5p橤Z'n !LEXWOu[\tUrhê3״w9$`-;ѻpzͬ~Z.(dg䙮UϢu 6Vf7.k(=jlf?؅e9Y9w'r\.r7kl+g3 )#=g3+gYM2r6&Yͬ&f/ q}~T{g3I+O8fGrG$({/c-y4W*tMP\ѓ6@ʃ^HyL"jnf eyTl3+g3#2#lzז4.tR4[m;Yqv.z_nONO?;@^ tu 9D>+sWG_4{+DuH,"ᴍO5"""# c&&O8HDu1=s?p!*O;H$#]atjMjSwY6֒ҏ ˬݗΧͮ!"Rp:"/zb|U"埮vw6+>>UAZk.fHCJ)bbuDՇ37F_uݾMs${{ ޳lݹo$K"fx̗R dir9$մ .=#SA+3Dү2۱\B, =S"迧^VΏP@E1LpKO5KvBʔSUtWr*FMFdOX9IV,}`{imzsS-Ԟ/>y凔(/j-Yʝ⪱Y}硓u*m- V¥jż̥[^'Zqn\?}>OkʼOB'Y9ۨN<&"IãvOA"lvGJgeR=WKW:vP4qx8٠dsxΨ\YمPfK-eǏꀗ -F6{NqHv*T+2E#&lJ_KaZ՞>-S{ cɮݳš>]!$bW""2}wĸ?gEk^o5d;gծl}k|mkduBS&D+dD"woe=XYK53-=}~;ȑ 3{wiZr:C#*=҉ߝ$ˉhukBtʝD9aNګwp0~QR#2Lfbh{o$"5*gHEd)vmN"ȏ3~u,О8\Z޸Yˆ]P|oJO`|)3͉rNp \|^j$ekL<$~ɟAvQFZFb1't5QEdO"7B\HRZZlS ?S^,,""Eb.11AJ"Z( 1UkuݾuATzbp^ tS=Y:MûS^{qFr]AjǪx,b63o{.†p\)lLu⿂ s[9sor(,69e4GҖlF]!]X~dW?\qo_f`Z!V=주 UA8t2bT!sF`+"}Myxhgn6jYDtlޞR?ϟ. "s4v @bWkvsjdej1'*8vVbpfD3:UZtY~s j1Կ8Ƚ$_. %MAmQϥkF0M-Xb"N\9/hg^Qk\7~~|cg 5廗D&e&c !fOiNTL2~Iff[S?xwDN5QeT9=ӏ&?꽭)$hw&8<έfīe&*g>r.?7RF.\3CO{;dDmpzCu"[iA7g+2Զ^&),4=!pUCM8xDDC֌Ү5F%y%n [صD}_eǏm&ϸ閭?]Y e+^(""w獩ho}X;*^JLzZ֖]/& Y=%"^Lzz̰Q_>{/7d(=!}{ym"/=#o؍HH;I]ގesJ*"~mB&?:|YTcI+눘ٖI޵l1'&ƦDuDK7[<¹諾6''YPh+3ɩD3-HGR0b{{ԾvQNDDD>޶Μ$'-'8 >ɛGDDwwGFMΉXe:> bKMzoT_YX]i%޸f͙7ؚ!|ED4EӪ\ZCD"f6JDwˀQ4Z+:rHW!#eZÏs2"j"r?̭o/CfsNʧlFiI̳ "2`?2 ].|UkjPQmcC3i[.o$D8y caК}Ak$;##!TvFueaNȁ?l7`l=a}x)]~FE=Y;o\"+K5,0hAD;2"p\Jد$!}[ Kq$./VPITi_lD4jTg$r-5ojeeD4jf\ܢ0Uk7jO8}L3%NS׉|Ttγc?eh .ɑj o&NZc*Ɔz""S~-^qQQvB1].ҨIˉFBD套5y?DDNY[mK/yW/kIb_FmF大֑dUB;{sqco5R#5Y,+ BB"j%VxCJDvB6_ӥ >Y:UۏsfƳ MS*[uz} C'. rxT%K+ ǫ_A.ge6}icetD~]M҉ԖkBD9k~;ND5B)%&i~.zE|[(`>k3[FCE\OxnК̈Wg>M܉mK$9~"]-XK!"wwe65?,MJ}IYe3hcoX}F}GD>9 HV/j3ӉgxBTvFɉD+Y(1Y}(F,ify|"rs>iLiik'%4%8?71/\s#J>z/(1"" GEoVh6w椐%P}[Y9?$5I'H9FH W/>݇'K8 !"2jsg5W5ra.#ĵSEXO"ѶDmѾ+o(jro$/`_rq왓Nkdʩ-7Rv r>vlfFFRy!2MD6^N~Ԣ(=FwerǒEKvY-qi sM LtWd}o6==˧cFܬadb"sWkEch`M}$j޺-W0OJP:{•DB߹AµQӤDL|O5bxn+(=fUT{L/=dOu?\u⽨9g}N-f7c?lM)>l ]@2]E^6($lQ%[+7s Tzhv\3i%CK D]$d:Nj?oKGDW"E%ߓ}6c&-*2 +svYy((|GO(xbN? j*>uh;o1;a[[/\mt#,M,""Jrg{eOf1ً5R6ڞCC=&˽z_9SE;+TV\]@w>TLrstcTqlZx_m18&. dQM|Z9fV@Dvwdo{4>_A;琍UVWRm\{1{4aw\5f/ثcy, _hkrVaf\tlTz8.*Q`y?򞐈lmsIgXZP5DdU!ZẋK#3(7TI>&"6 rtla Z7n[}4hIBE#[, IDATpް@+F%"m{ .aDN/cuZM $_&49Í7[,7=q[NnS13)_ l{E<̌ NJ}1Ay#c&OK NޥzĎjV"פ3|G_,${Ή[նO7b>{$:+qa.9buBnx}׵<#}]}},LR(4Xx͌;2~^kz14ϊӪj^E1gO7)Yx͌K>[ZTg_l^ӱ,m1Aދ>s>FϾJhzX&J"w׈}RֻhSG۠0 |KgT{NI'rS53# jɛ¼M$WIMzjWsSs;.H8;cI coVmz$L챻m>)OhX-);JW?a I ix}G_mpӝpt}낄":IDz4|˦wpqcj$] ;Zz{R=(G1"ܝN/ N kKQk&7-l>[Z:3XgQ>ViwfY  mz2QZz{G <;_НNO{S9A, `Xڸ_߇ O:z/=Æ'vu/@/K_{R^^:<^4YTy):=lxyitR &5NK.V4wA, `A, `Ћ]Kc&q߱ʑ^m&L r$"-gkU&n6tu7:7|c,6꾴^EOgFFh>3M0ds/gEd^}f"fع׼g e/#쏠vZޝh D'$)؎FEJM䥙Uӷ$VH#åĢ+|?'Y%Id;N)ЁjgLګ7{D4?򛬽bis_!g}ZSY RcnaFʄ6}E/_*Kz 2l,e++ڿ 4<ێ˼=JfMP\s+Ϧcc˛F%/zFWfrs܊$6'T_UI4D=`&"z ٿ۾ݡ-twzf+qJ/Gopy~dzϷ\﬏qeG-z*{RMְKwL\SBTTW@mh\%aX h@+?P+y15gU28&+|ʞ#pl:-{gZ'=;݃@DS(pޯO<~RD?ZY>?7{Z]3:wl}"TY/u4m |Zֿ=>{:CKDD?)_Ȱ_g"Nw/S|Ҳ(mfu.1/3?ߺUPa}?j o/EqN2hnX[yx6Up?,}Ͽ/G=xс/bލxhdi|, ާ=[/|w=3GկW5)=xÏ=/tcg~g2~O>AŞ?/9;oR|{'.W_|FP?q?y8O{տpt}>C.Rxf|g?'F3{]~}?*"zD䩧裏~7[W}4uo}di|<|OgC>w+WogvOܞ_6>wk1ȗiYM3P{[?2'4[kOPvn_ e?f~K~ %ʳ_bnпڷ?O|G SS}/wnolO}"2~ԑO|_秞[] <{~&򫍊_WsGW_6=?hB>15%7j+2=:~}"y#r_ko~sziݡÿKIs`onŅ?낄q̧?̿w䏵s//y7KsC]DCǼ$|BTß '϶tiޥz}=#?џտ sx|8KeGW/4jK?p8ągD&-g~)v'jO{Ȑ+g~ !V":㵥o|?_>zcG><؏|bP~nkoGDɟ|=Aho7{]'Χ@?vÿo~|mNkٝ"vlW~^_7?~?pW;3Y,݉8 Oǐ?~gzTcدC 4ᐥ,S{]' ;? g` Kc Onx)RO"wfO_p 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KcP/G^*EFzh8eWdždžv]ʫcC}<;<6}M/4vFE/<ўȾIGο}"b׻g~ݨ^5;&)/N/>{sCrfFrcdi7Gο1YQϺXYS\zѱEDnNWZf==vc] EIgm:יE;oůfoӇ#Zk槻>k_z`qrli.\:|s'ە `ϐ1n_;".1{$%-9U"cAS)-r}ɷE4ߐ1} G' yU2:vRr5/>83Q8{Y췼r5pPͼGyƝrr7<^H:F[F|pukvD^r9[Dk{[}; ${37VZ|5M;{sj1Rw6K Y_OQCNWWЧEc-N9vM_9«? O,"jMi`7{ܿjxn(rlTnjLۈb٣[.v~♷M_;ډcAnqz82;w40XTߩs#)R(o2XBEOب1͊vs3)U@^)`tm;Hzg嘈߿*C}Gi!Ȅ8#MQ_V˸v+ա;Y@YIQθUЮ7+gޙ:-i}ldWVM\Dku=M540ᐥ, @8di!KYpC 4ᐥ, @8di!K;Ѣ%[̌+m=iZdh#BƀKQM[/(Ĵ`ʹlbt˲帲˫ىGMӺ_0,A^9cN6-ޥd9 <-U阷M3rVVdEekmN^#bbhsʹZ6j- (ꥱoYn۲הڅ-~!YjWAj6U Uw7*o7J1-S*vZŪg-rrTDDF mۯũjnTݬUd*Q'jimd")YΤnHtc]3FA~Q,֍q]DקR3ED6juC _*ֆZ1n܈_OEoP15wm/6떗ȨaNMr%Mivل ėh=: G w㶋e:(;KSk^MZ1,VUˬSqí7Sz-[F}5#OnEtczꏲv)$έ[ ;|I]QTg߶Ȥz=yVܥϗUTEYQV 7bnWkb*k*loE Y,"vXiDٍRLTr]t>o$c(۶7*[(e+[v{-=0OjF9--[`z*=1"JED6JyeJF&[xFbK^۔)mEI("eR4R QDiKcǍbw#gDrr[\ۥu_q ۮJ3P)M/YVv"ڲ!Fzy[Dd{T\mj~ ۮF}DeR7nd ا&uC*k䬌O)mY.QeF4MDDrM}LsU""vj$,." 5is""ƂL|E&٘7ڬQ_hsSZ/' M{/(O=x`< fF3MwF)dإhfS[P_ rug](]Ẻ.Yo;G>j6W9KcKQM1L~x| , @8di!KYpC 4ᐥ, bfXiC̴+Aj+m2e}!ẺrfYkAR݌Ζ.Jokf,XYV]o^{;7ײz۠#l52QGFg8U(zJ`"|cw{xe3k>mm[Qn57ՉCkTWjq㮻.JBMs5hs*խ щ(dlDc PJ9̌ru۵j}tbܟ4M]VYw]3+lNTkU!g!sSb,;ZYVfF.:"SqQo?f.ĂxNIZQ[nfmG≩zd66GL$G+E6O4H|)^;n-eϖ"fq2}ܐDܶ]WYlJ#W6J1M+*園l{_gy x-Ӽmdj7Jm7viG, 9Q|Fmzi¡^pC 4ᐥ, @8di!KYpX1ED.Ĵv҆n3Әfui_hVdZfMB,hv)iY X] vrx,d*g3[]3+K.Lkՠ'FeRZ5Wu7EO#ƗsXnY#~B,bN{RD.""lyi&`1v!YtgI^_?1B@j$ҔVQgYYMi-1XMUTZ1.2Uږ8,/y˚/vXJX#pG[Lk+;ٹ8"bnumb'ʂ4)"fFdtlZ&(V|Dd-x1yˊb'6n[Rk-ZURzJF+ uRt[Et{6TJ+3ΪR4 6,6W.1Cg}(E#uq(T'ja3󪁙RoB,#bQ0V&)zbVd2-[F'ƝIݐֆU@di >/v`F#j#geuK*"v!Wr֤HPE"4^RNٳ수i-e'q;IKьzV4rlbY(9[T"j[-% nQ$VpՉvml]WkqeDdͬ#H`01Z%J[Ζ-MS˲Q'?ꨈ5S5z>k2k5 q{J[?[#$g-ǥKIkz_3E"bLb-zR=R=QLuJE-0Tdֲ^+%ڄ9c1:ZPL弅+דXtШXaT\U>OTOhiex``Q/6rVV_~ƸHȉض$g嵌_ 9hkV1 Z/[RɎ'>uڒ6ېi4׍UQDZUlkmN^_GKK1-~.{˖J{{?/ύ~׺Qlv!YTrZFb ?E@jqjhݐl5s?1$b[l`TK[씶uvũʚNkE4iN;`j[TgL\WO9kv$t]b' dGմ7jwsf/HӮuu֫榴TɌx1|Cd9V} @m=k.EXjr@M}EDu9زb",(fZzmעޘ{*gYR4ݷr@M<EQ#%+2Gi :97m5z,[eYzT3>ݘִN2d9E5-Z[׵Ҧll٢h4cިUו#"#]Y5̖WD/E*ϪN?poCo2ZG9M&T$cE՚-mn20{e.Ĵhi7Jm2Fkgp> Kc? 1M"J|IiZx]}-'%Prjf׍RL4-#[x-+[8wy,e6Ѵ(کmCQ9c56'^]]0Ӵ$1vP^Oʢ;:[Ĝַ,"9xnNnxefdMӴ@gi nqh_Ϸ.:Ԯ @[[^[>|Koz}ymEL:8?HۢV^;+X0}mY%v8^B=NPd!ux - ?L7 IfOu1Ϋɪ;m1Vn_?p5V7F]ꋽ_y׮d#Kc M,"+LJ+ͧ[~{E)k[>Z}+cWNwn#C~uۖ3oܹߜz_vcuR^y+"r㙶H^~]= Kc\r~dBҎ9vcur]7 >n<٨>wr#=t:Kkn^tW_gxW.ԏL+}-u#m9s5Ai`o4o-RqlNNKp]%Wd.S޹LD̳'v{}/-銜A໑:]3W噕Wǚyxelm5{sCɷD})?9$Nru3cg]9|_My.l왧 m{lmW\BKRs-h?l diO.,*9˲,ZP'/n}?R)3Kr\##%k)>檡O3l_skfXبUebCQ9˲ KH~͌'lbsvz.]+$VQg˖e-Ճo;Wq/gt9fZKɲeYUSnδKH-DR4eϖ^1]ʳحѼ56G̴Z5of84WkhNoۻL.7i-SpFhҚwz-LkU(;e|W[*dB,VeJiKo]iR=!⿎޺RcE6W|D˘n9K]߮Ʈuu6&Drb.<S}doإh05%fQ/P d2ǴUpčRL˘ ICվfVt]-'VhEhR!ifH~]*s4h}u9YP>;V8|hkD.^߻-?T (||(y'X?}+n,#Ѷ؞;-W c5MKSXקY8Z>ĺ|)4MEZb[NWDZ{W uEmν4ٵj}4p/YR3A4U15 e&aWk55wt}JZ1Ƚm y2?s6Jź*^"[ެU[2>:lfc-OVQz jzl,ze]rqmKK7DWUxy.FǢo?xӘQSE^Njo|6(|jKgn7ѧ&1Ywýۼmh62ﵖ0h3pxeUcW~r:m{'կ5ֻy%tu=FV#wǴsկ;nk,r_F[jc63O+j.H9qNFy|7O~WSb] whZs?}|X;אַ˻mwٱ[anJ8Z>Ă|N_n9jOn8-/r c9]V?Ϩ>x Q>u @om%TsJT0`^iͬ4-ijU~P&eQ_,bJ_5f.RFmfJՑxvV%n%z<]}HJ|iDSj{V 1kMU'Zpu5Iݐfgt˥qkż4v9c]n^Gė{q] &:L{6r/YT{uض>o5g[jj;u#7_Ohd|bTCUx[zQVol8%F2bo㜟YF%w{yւ[s@56J,67P7ו[oy/=P8-_I `M[zt|˱.ﶢIݐF˦ZuϮ."jt T#x)'XRzKQ*f`K^+m-^Ojuow Y5MTq~liiE^Sk^>{]`'-[F'퍯ۮh*7E OLySui-ٶOP2r@q~w(2,{}S3eu}iH*zǏ㮛}ۜ^hy45|=G'*U!YM6r]Z#Yz9F)v"vdPfb*E{wR1O9m(lԪ2 ,QSQT飃/ʶ벚V:U#z*g{o52g[vڶH98*mll٢PЎ 1__SaNn"}ܐ9ӜZu.eEmK?}~Os5hw^S//P"]TNvZoi;re+e+*/ǻ?a,+W,l?]G|*խ 'M[DزG'Lf-+v+XϗшorJYsFM-o~3}|K=*WF)v"De|B]4y}s]GS)s-2%+ݖ &t ٲ5؅X֧%Xz yQu5o\U'+"=c,XKbѳu9uqX#g-bfYaosi֚m\Yf)mUҌb*x`}|L8ChR7$ee#xXr{oo5#7 ϗKЗymH2F?^o=߀Xmۗ-EM^o6HLS53;\]3|Oxo5ugե( 6HE֛͓Emi4~)GS'fʜ3J9{ز;3_,kijkO IDAT|[wh(EQi|8̆8sm]\ĝX3+r1`fYUZdQu mzo]F3۹?ѠJ|ٲ\`Ơ֡N_hخԖr=;ɬ3mݬ!- [dovrl4Z ImfTV+ryiж۷-UD=M-[TEݱ֜{-[.N۹r)Xc<"wkD^Y8v) oϿO)]hPogFB>Ywq]JoA:mԪ+di DF)d*3 c=i6<Vq[3&)9D*%oDFeT\z XYx]J=Xq.*b{InɽmcɄDO4~+esۛ#mv.g򋕖/kKq[ںfV'{ ŗmcd^k׫nûӧIۯcA-]s[\3>llᛯ(g؅ֵTV?݊tXGzw!q_ObU5Z>z8=˥ۗcZOIe1_3=UnLGQu-sJǍ d^gN31\EH+7Gfi]g0hėsZD[#9Ow=k2t_ly64g&c]-d;[EN}l4h(Gew삡yrjjdN1lkÓ! 웶Mlrb4mܦ蘺]ԍTD&/L榴!q4Y^0"sFfq.RO:0nG.E#Uݙ"6^oֳ)3٤V-FMs^qBD%]rrɻ݊nǨ,Q{Z }۷H|ia^Yaf^uef)gkc&j,; Xζ_.]׊qCV+6,"uz3' WNNƂloly.k-d4߷ykۙ8Fw!EdOſOZ~W4?dyg0z^h ."vb >3dǹ(NT'fڥhF^ߕI}6CQ-6΀j:_km.7ڞ]Fo*E!7[4sטi-p ]h;Ic |Rjph @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di!KYpC 4ᐥ, @8di .Kcӗ'V^;{kͳyꗣCKw{Fzh8+otdi 滷EUgWN7W8zYN셺H=stlh.X#Kc`mnjan ç+mٸxfENMLgn~ޝߵS"vѻ.>GƠZYtKEyygDlgD\sޝͷQ]ľ9:?xF[joJv<ў;rWg^jwHr:j W';&+=vuѱ{$mJb{[n>>tADy;SUۋ `1VoH7C+ރ 'Gο"b[s3 ο;[kt뗣/T.5'+"Ƶ{3"r낓 娓cWDwWxD" >>rzBs"/Hݲr5p͎c/;q]ݛ>.p~z3.ͨ"J̳7qZ]iD}9Ә ,AΝWDV^;)￡UMi/W/}u"'vl~~Ƶ{; `/Eӻnt湋מ;wol>w'~ ;=ɞew(sjձW͐ޗ#_ 5,A}["oN|wvn&x/~mmivʄmuM?~Jw2 62YhA_m(/i7C+g޹/KnX-7%B-N̓! IE4ڽa|-߿wcܪ79|3WnnqzxlS]9=6tt+ 3di 闊cO5߽8ke}(rojtbkmfܿ7?}8=|N'ܮ{W?s^ri7tErWݹJy;+v")HrthܕݹO4[熎eBWEDK_:v!RR,Z,z fF:ʹQi-;?x=ښY(6ju#1ifKދ]hsu44مXĜ(/ź`4jgYWi)j#Z`e'[KfP̴Vc[ll٣JD[l_SN#H}N8LyQz9+-d Kc ;x4MmΉum5pamQy,eZ[[hHkK1eTLXh$֒ʹ6~VdtQ&473*S+Z4#ge;{YKցLl+" /!9if=[+ ؉|ˎqF)vbs'[<H̒5#"^`Ué[VV2ZM2ZQ*oFRDlzD#+ ʊͭku#j\.d°>_Nڑb!(VQґU╹LP=kYY_܊⎪r(V['uEs.4})UxTDdYq-")S9k=obZ̾n E@K9ֶ-VSlbF)v"_5y})>"v!DgVkTB]$U3AuU϶H%lŷVd-Zm-^.2Wmm#1gL7S{SF픦Ljūԍ׊u1D/޵V:X d}qA$,޹q2B&ƺH|:錟hƂ."z<(zGj{s]ͱyTmj I;#"DZ|٥I1Ӛ9ne'Eƥ^'bfZo6{w}ѳ4ۀfiV;V&w`(88kQ _502A/ cl*9^ClPKOϥEnD*W$r.(AAA>ЖF>,{ߞmCv osKܺdf[kI667HAĚK3/~  @[EkMO [fǴђȌv vm6GPi*ot'hN ioqZ{߸s!L=F TF_Dʺ.iIuoECwl wFK]V:ˣaۋAAmiC'cxSxW쿕D{ߘyT5QK;6\{'t 葝o.hxA~-;zJ}%O6U(hzMa֥šϑ )լni@^ );AAAȇ=F =&h$*mfdޱx?P:^i ;]'Mi@iE|~5KWM9S=Nx68vhy .Y/Qq'=}+`2싖>C  ZA)w}{ ٔRJZ%Aȅ#Z-Vc:<_L|ӘQ{9$>+xjFw$ϻ

yȋN<AAA8e@)i .AA[@[AAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@['N۟/O")17~J+657}c ee*s;),jdv.iA+{ߨ?ƼJ+Ӭ@VeEuIMjS.Cuv*|^}n43=e>$߃WTA{|9 %hK#3{\Zb|7\{}^ p0ĵ/O^(hi5 ى1ճK4zd]g 7hm$?;&L(Zut%Usa.TItI"16ZFEhv B)tYG!m;w <b>GQ-G4]y*፼y:F1_*yqnWeɑ%$Xȟ/zkS6}ƙF2{r{8('tG]m+qkm;Pq&m]>OeJFz澻A4ZD7qJs/ kk62u07ŽF&nSea6*2FPjDߠ[9姉hy"Aߘ/D )Zt=}Fym@UB5Eds(@oz*2}%p㟖-0'*s?#t,-~~ mi"FJݡ$(Mۓmr*hVȬ1 r,7y7$ϣ<sÆ4yp#Ƌ %|n'9ˈM_E 92t k5#%a)KS^e*35uܘ!|eZleJԛ'KmNMciR174UQ(shyK+dۦ8 HJìT%!jW~p@i+Mer4 {8ٔuC1]i`ޥMLH +uJ]+LzwK4O(oi'Yz\ W^22A*s_C\wU|OG>vw\S '%AÖM7q1NHm~O}Б7éQj@6nGĝ͉-l.T<}4u=pmZI`ÑeI-kzO -O2ID^Rom:/ ֲReWS\9엛W2& \-`—v :ٴj[EB6k>v6#+mqS_ǜJ7@e܈C#$s@R6k$˜P~DDNUY5VZ("Ӹ A>&ou3i&ߋ|n Lfx?o>i}l>&7O^Ef?Aky}|nZMσV%r)3STQIpo7 h&1 -`;)Q՚lR=$sr('gVUQ 5 ||NCoD9 ϼާSҊ>O&)QWH*ϛT<&~vL÷ b}eieVECB˾ɬ uq퐵FTzoЪ0ܕ*FTݐiIm̭&e-l<,ȥ+2TrQW@jͫnO8 nPxKf$n!SK ?^+WS韫R7o>߆iD3Nܤ;geϗ볲V^XQeŞVR,9|{bJ)0Sݡ80^{#!V\;'7AH`_eitU E0cID7,/c=P>:c޷g`}u$]y@p:.I\d5ѵl;J)Mw6!pmZCY#]'ѡ@9ƺδu}6QZX>`%ŽX_vi/Fk#4uX &펖&BrTezxP_tN/;Twb)IWPk4a)qC;WVg~ ]$wY:I3~}SdN4KW(>8l?x'YX IrLMrtEȷh.i7O wf=wg5w.=,'/{+/Y98bwh5=bW- IDAT8t5@7G{A:2HfůzP %N ͤʹl͑=*clb=92 ei!]CX؇κ H]S韮&9?i*u>߆.A(fp"P/}VSH a{[g0ʹ*m(TdȲ`:l",]0'NE[Y󠙌- sXthwNڬX\GBS(4nHR;A[A`mWbRn!}jv< ŝn6S]FJoܜYKǝM-@',S4+.inEQEOq Ղ UUD|.( vKj<^ƒoQ5{Mj[ٲ^&N3MMvF{,}yx+I%TvJ6t~$S[l~0r|NEmSWh%{;jsc{x`5)I̍؊NnU3IKGmJaq˹'b\Xy;,\a[\"e>bmӰFP .? ; %Md"ʆ(! 6ciTg8l u%:umSe$$0زpӏ4ea>N9SFj& V ;rv%ARzYZ_Zȍ};jj6iEmRC1|Wk;ѣP:RݟVU{mӐE m%@\Lүn!e2;=]It(rP_+/Hʹяjy#}Wuɚ"@688#m+{@>Tr>RnB8oE~}iI).Gw}$jY|e5`84;9q[s' |xk8K cˀA)^ el6Q3vc5Iwеhٱ[ǡSl8?q_4r/H<^xzjE;qb]Y;8B)gQ|t늦Z_ z8@VD& @)kt6wѷ̯x8ᮓ Obmn4V= %rnES"@W0B Pxy=? ڭqew&yoeGIcNrLw NnWh3r #pz+G դc?8GxCn37/I%˃m"md+W/ɵa(Cjb&p]'r%>%zNH!rHOꔋޱx?=5j93O0X~]:X~;9-c|yM;Ѧ?t?vœ˸ՃW\Ǯ|<:9K^,Ȝ'>~{a/Ui7בδGS9>PCkmHJ17ًQVs$W@y%L#{Ϝjg,IQi-F ;/R||z|LHSC"Xftu  $ iDnl(rd)mdI'%b>O49i- _/~4'y葃I Wks'ѧOzp-A';zXM -e2T|p!4,+L&?MciSiT(e:"c= 5etд#m| VSXC q6ZL$4]Hk\9 K \VCjkTK#Ժxt1&^[1I0*4ї4ea >@>qsj,QydG`!ix$x_5LI^МrбPލ b8CV,ˎ\)jw@ T| җoϸ5C? Cޟ4Hj,8{A#wF=enIt s ŶQ] y*G1ћ.8T\'tҬoZ-EpI? K#   u>   rhK#   u-    ׁ4   \   rhK#   u-    ׁ4   \   rhK#4 MO3/`ms{8!{]'Og+h'k4ҝr]}Rcʥ 郼KFnTUL&X/I{v{߸sA \Ѹ OXmw@ijL|sØ=!>oMۮu:qVGE+2:2JP6 㫉Kuty.I#qȹuv17ӽԻMuoGP_wկ]up)rwzYuom:oTx핲i*r#*˿+{G'4zWr ЖF>(ܦc1ZFD/<̀N}c? \"aBݹmks,"~l8X)THzϮE}PvkӂH>}17;M 5C$ZE_Q]q>1f&Fj4П"Ƴ$ggSTL6}t=sG]9Qff[9_S~nFџ3SȿWωW$ff*E_n9=/޾v*oϑ郰LPhK#5@*Ƴ/pofUW+(Yr*RjrL2eu~(l늍-ڔߎ;ք ͭfٔ>Ĭ0L% l?9tI-(-4̔pٺty7g)fЧGJۺe4ɫ?R-{炵fgoXZ3gņǦ!< 42̭T]-'s]ѫ\zu_l!-ikǮ":~AE.>Xbns'䝹9[ F嵸x=XؠZ$Z6nGĝ$3ZQG,2sd$|b\܋¸**^=$ak>.ڦicwaB_2\:* :rso:RcP; "PL;|7~1M:_MTnN/PyKP$N7h&ߓV|I5~~7SS+%WVky}}}LZ$edgc S+d7[LxLRi0K3i1>䟇ok7BԺ.*g3iM6ؚ|k5֧SA'q$LkdKO ZM|U>=*3hyTI[6|xRZp>%UCfD&+VQYjW Jg~ƊaA\XQm>++QJf&բܐuH2vQ$I(#p|O1d?+QTzoq@lx=o e|=ȯSo^{jy)/ͤUzF֑lxKAӤ$٩m6.>+m.o^sY(oÊ;B,TקB[U._J{G^T7QeX06|AFN6֥S3^TSeo{!7fnʙm+0nMk(g4ch#':yA0r_>`-pܯδxnLzC{LOr+-I^tRd?*ގ(T;rRy(Xڍ4]=61(#MKGQT7}KQM{?Sښ,G(ZZjUq׉DQb#,+ؔsߐWl-I0\ZMПb2)_qiS1qsڱx Makomhw[N3 KthZKyҚʐKmIz]ڄQyzUr 0ӍVTOX_r/) &bnW4N_~?%m\?4ͮK 6Mv>vG[q ;]큇A6XEZ{N+YLIfcVjBhq/Z3~^Bū|ݟh,rXM93փ %!mq@Dڝ3UV9In m&%zfY4.-!oy(`M]Mxպ}z8-3geoej/xOY,Nx (4BN YLA\P4Xa{;X/_kŮJLk*/PY $MTy{zrJ+qww6aC}{4 7N8lGNm G8`wx\5o"oX;|&;%#G횯놈;`xq&|֢dZ6Шf_:gN@ǖDG R7JQ@z8ԣud#_gPFcj먪ٗ(̲I!ar 0> >?C>+U-f@iZl?E4݃h:Nxpq}V-[ 9;iw&ܛJ(7 NQD M:܊/G Nƍ SC}3U*^omiC|(1v+wVE&EKW>LY]LlWiq{pQVⰅmAex\//10 vb'mȭO#V~/Ӟ w26>!Z3;ƒxRI/Toj.q.4Lgdxk`5;7Ba6kǟ5[*ijrmqmzco>d›Ai .*j0-`xKK$ ktf{O-}>oĊ֟EI{!A|B476PA]z._OհWϻ֘rlYX3Mkdw;m0ӻ%GeǬC9D!,-Lvؒ6a78 k)结r CLoh*?KiPRu>CѼ%vpqj*̢C=jrk-|XqBas!'P3brgd-6jbu"{! 6/~$'@o+tH)[cÃ9M~2vX6dk${5A:%r/ٟ) ֥Qh~1[oO@y%}š]UrgT(zk_:1;P^Kl[Bb >~pa\tOAo:=ookr(OyesOvp ʃ8 j#w"32j ,i-AIdQNes=1b ]i JÖ=F @kD!t=yKE;qp0T'&Oa(@CCAڋ2ÅHR,_QRBR{|%R 8oV}9_ӹD#FN5.hLeu5 X66G?At f6wR\3 }fI͸ODţx\7rCJ]Pdi.}Rf|ӹ |%]?0 ly N2?1Ȁ~==TS~`4AE IDAT '-v.JEVdV`ѵ &O bÑI.DǩAl8҂ܦ^_,>?8bˋ7_ΒmӀ=! U֨?=D2=#IK;.ڡa̅TĆ#%!L=YJ *2; Obhe 1ZREhgTF2pCEQ|O?x@&[YM8H̼!Ϧ0Ѣ}l{yirH4- O"8:BV k5"yxS:ZhSJ靻\5]{Ng]DJ{߸2?!B^O|Sy /P>tTm ;w+ uRɭr]Y@n;#5 ݡjKO+Djst/ uUW82T}oe+!AVuu\PGY?Oņܐ ո!;v<ڐfXν+dge%  7֟&ȿ$ v3PGıpq]@6{+.h;]坡G'>h2'D f.9}:Nuk81^UUS4!s/\$.apy;JiY=Ndx1e$ə̚jjO:s<}LzY_] ii}֠/sG;H@ёrqR.G|Qa{[CMܸ!ú<߳Dl0ئQ7]$TEt#W+#-,jħSh" j'4;FĨY/PA.oؐ}#U9Qv S̉tLO-Yy}J37V$܎Y"$ niܼ u_͝v#v擎x;;11eF TƥXFtDjB姲z1 MkOP%h\f@4CT^=+r_,!-].!(_4Sn\X*>ЛFc4 s<|u7؄_ச:ngH#szXHmPē\uV*/fQ1 0:MdoJk!83\\R4#H@'kmȰs/qV}uU'HjFdj8="mmt[Nr ygF\}ls윤N exRvOZ겊e 81c'' %s;TTMܦ3T*N<_sTUeI(BRCR|E$R!ci ԦrjKro4OZ'+V{[r͕dmS lGdR揆VV?SG!8!/?^__9 1$~J ^%|Xq^D[7% 4sY!4F A dq)o!ސߏڏr/D]uF%y#hK#   u>   rhK#   u-    ׁ4   \   rhK#   u-    ׁ4   \   rhK#pR{@AAЖF>.ܦ ]Fr٘:=w]y?k'ԐQ7b%\۽o ykI=]ƛbsXR@AA~hK#nӱ-(%Ʊy;Ɯ'>$wȧ6siAAW ZCsyZPI'}&Yޏͪ<'d\WSLd՛__x互u6r'(7V+M0Q];-!!u9}6h?A5Y,2XD 2 Nx$EIzIL0YSz|+0xRUՔr %qAA?Ͷi%,Szj$[<.\3 p~|;Iy"z6 poDȮ3UuIi G䤎\?Ymvڍ sߦ{jsg IQhE9Z)6^,|ejj'ۄ!k) %7 '!m`+@׉+^9oWZR4b L]'Lq̗;Y1nitb Q{Enw@UaWAA74Avq1M=H^S6em};sZjq- :Zh_x(-?8Jia:6V=h+ׅ|uBT,/rWpq~2[fQXo~ISI]HinYhGQlauKGAA~@ ^>in43t$]'z)񭉱-Fw&<4}7RSt ;5eߍ7Pm?r ڧ.,/Jz݌{g92wIr\Ajs_я5O#"75zIe~6wΛ9I4ݢwS҇R?ޱx_*A+/[!i-k؉<=PUr^9/qԺ}Xc~t932aܧea>.A|>6 KkqrzaS (@71AAZ6B_6|MLqwVqEbAA~9 E2nkK&  KAo9}wO#4Lj:^iS/D8  /}AAA:AAAAmiAAAAAA:ЖFAAA@[AAAAmiAAAAAA:ЖFAAA@[AAAAmiP:W?i~0YOh|,5Nf'ŧ+Z۵ j$Bm(qjZ"U|[IT17+EM*Qٴq;L0}m:2~U ,KY jkp!3}b&Vk\ū+K oߓȿ țrqOU}{9+MG'=7b kO Cԅ_$:^k4r9tk{>d5џm"+B 1 74el>36J뼐} Qh}MSS7p .-xZjMӠ)r@")V0"p=8ˣai'%,M@ 5=M[y]Ro:/ mPС?O</qMkl4c>&+IvY[-I^0AUDZ17N)d.btEt#ܦchF?j>?nHm:{}nP{z kuե]>% _jyM$/nzYDM-|P?d=3nӾ}v4uea1w >@6r C}TIM 僻.eZ4.4)|wS,dVަ?E0g!so :2-J4aE7l)'{Ǥ2ݖloiyD}j#Pt7_L)u O].y H6 UJ1)Y؟Q}8 m:_5L R񽕔%\Lʓ5 oB I&q<`$8-CE_sVL=E0y$rF>&p,SA*~8ݡZ8϶ꟹo轟8M]y3̜'d%W!_W1Mm_^/{Nxx%sIFT&vÓr\mMBz/ ܠt OQ}wKbDKQjJèv.Қ2z,Hr.S|c4S? zvhW;wCNRvr媉۴0˲>^, [BTwЬ (0e A4TnK T17<DSXZ\)7bU9vuQ%nh_{ˎ$*b wjo2*,4 ҪzMt3f6Τ %y=)lg?n_=٘ScG ūvRQ\?Ħ?3 E1|)/,[IG!;曥<s:|i&C} k>G9b֞ӧU%fTZeiZK[.UfI8ɷ:$wU熿NO Jm[r!&%i2")hEBtXz@M,7R L/ݘ^?oJU.6Z~O"*^óZK[o+>i}ޤ ?_OZo}j۾OZm?|OO4VkIDQ*I5.%QU|nD3KQʲN [ Eܚ|KL^s rObIkyL> 'ϵiN甯$ޠ5u~6*|nTg:nxOKXz4Ico& ѶBW7fu`,m]δG>7mE.Z@ΔIkOJ_dL{ԛ=ޮqkZ\ hÞ;ӽ}.Ջ}H6=@q&Y/Q(.EM)t vF)ӔCqO(poF/DOLc6Z.n۶>#*f,̷4A۟8q,[˥32LV6$(nq4usYdJLs 'ە{A3OF=伥5 9|_͝:?Iy 2=dS7Sc=(+9sDhdgFL6\{Br]K'O4œE̽ k;v}oE\e]:$(4R,Bf-[5k\.٩17MgV EXe@v[rשJo"ii+x duXA"e:xO.aJF}|:u/D1-|hZ :8&E!g!@i.h @v@S)j<F'(` |J0EpP97H$oݜc aЀLٔz_%S3O(oGNẀ-Mu9j5- IR-6 )@P'L8嘇~=Ėj-F+&L~'[% 7\8G^31L6$ vf,/vp=DUS0pB5 DS 5Ցjpa?.k:`c7 *_unةBC6 @oWܘlקW/c~"M8T\Fje{jgwlW#v7oK\Ҩ1էW8m:VLPUzUjfV7nv@CMS撪j9OJVT(QpKŽǪ3T]`_y!s ָs8{e4E/>TU{)ld}>rJŶqyIf<]{;$CJ^V&m 􋽁$%lܤ`7Bp~i⪓k0tCh%H*gىmϸg{ں*,Ar9ΟsNŮ֦v8i0؟sgΫpki2ZG;Bxt_4?28@KaLZ߆+%/+ IDAT;ȊNu iM/|V䌰_hnr/I1qa]^Ź]ƇH|cp05,9=1Gz'mnm7kZZZs{->]].({z?ЛjaumgJ(Ն:3B*i6ӚE2ejZ\Z]Gw8A$iӸRl aWr!{*rK̉ LjQ%)i̎2 IhzZTr!yBM Bδ9ӥiQXte)iӛӹA [YW3w/s pvF,b1*sF*u=5*,ƖᄒL)9 hOsGjv=K)%~ǚZIzGB@R{"+ügИJ-emq[I,ʾ 3#;W;='"7;a=FYC ƨ$V{[ܑ^- Χ62kn)\*2i jn;eE-qM/K%'\ =u},emMK e]q|ChP@G %ΝzIE-H&$}=VnRd^/dѥ!DQ̷-NiVei̾F+OX{|/G K[ E6TVe;[ ګ-usdz>27E])<[]WGSfu.՞-yBd!פ IRq2gYC1p@ ;5]dPa/s_$E(#<{3iQi *F 'f%"?՘8.tOG/5=FQSΘIdlN=ѥLM"ۨ6i|ok(ܒN\X0ccZMbs&уT55]gz) ,$fHBbRZHWWŔ1v S(3KkDB,_RկieuʢH^w|.y66Dԗj9Ym{: ʸ%& e6B7KȰ 呌Dh$,w >AQj>i_tp))_bOL׬Vn%+q}ĺb7H-D:|!ُ؇g[Xh.MЛE" &bZafl94>N^ WVV|%tl-V JzHD7JZKc*;WӞB_μ@c`YQdXaİ-.XjCB\>KZGŻRLҖtͷl7YcXigL# f5}ў553_P$涙i9[J &bT~R+KFM-uǨT [+TkoOkhu_fB/_s/߈S>Ϻ|}oVw$-~H|w2,J]]Ι8=O?e9z.#>d5f?03N1e,?z,vJЪL3]E1kXY`Sd̏XRўLٲ/\V YQZ5|Rw''5thkkkINI Tc^`N+X%>ְZ/]@0R~kҞoxո x _$d)FG\Z+Ggm~g+hOx/  }i`З}i`З}i`З}i`З}i8@Zl }שAGFN9fp. gP g'"Z- ܭ7Eh3ne!si1r5{*_夜=O p`.W UjNTZ=WP Q/aO!'q0{'.F2 /?vfkSLSfzD/$/R[dD/i1YdrqHGgTͥB+4i0_i1IWk^8ߞS&[KbtZmiJQcSL^R^ (&M򻳓r9I񲵌csci<+bGx2ɴ9MWlb4[n׎3h1rJL(),ܓ Q;JלR-G G72>-s}s9vbuf:F`ܳ:.uy@SsXѺ=qi۸vh"]HtmĴENrnt۬ٞH!V&j!=)dki`nas|v21൘BkF2U-ݭ9HG>mWVjiX*g11rm.$r{zSc͍vrAjV^<^)07R/W~BZA9IƵQ.l1vQNcWͣdFtv'ey\HY[ŢEy.#\)ƺ3D2C\>"-sBd}\]֞ѿrz3ڳL6*槆q@"yd#%WQw|Ch%2 /?5,H1JJy96KDXn|6'KjYEݨ>_£ӵud<1A)Y~ښ>zbz:cBEu=re5.)z>*?NY\AD.\~w^C {3h1z8K^.-TA Y.{f"yBejO!u4ja2euͮz.&P&m%@_sfPM'>-`Z7 BY_Y^JaoMJ(J˴[T+l(e="a0x~/$BNV ͒{ڳBkxyM!Jdڈva9S\xkeacD.H٘-DzZcH O)Dڍ%ɥUS4+RXf2>&+(3!sc^qǜyva>tlћ:S"ҚB-!5L4Fc>1gq( !"ɞemG| Z𼕶QV[@l 9w2YmٱϞ3#e] 'huF3JLF Z/ @_Y0N\+e"aNf} -J$R+KZ3v}ixE#ӵk\P #/(qU9%&ĢZIR4(ݤA[5'ÉknMLJ聿3Աlm-ig~e=?H[zeZiWJm;F͌%-lmDP|V̥űX~ཝ59VrJdk$/D.H+k b_xfX%j;&Չun){i2$t RK,Y3Fp7wZ+UMvz$.2/ ؗ %ml>u :9,5{zEhcc:[+H) ܋YD'k;ub]`e8r[eu,z}os .VBU䌰>{2B) ry33q؎ #pFqOLU0Kϥm2dHNi,@m0ЗOӨ\ fsieX =V^R*O  }ix56ը0/Fޗ /HڸRP+Z;T* |*P",˕"tO{FhL'"y "㵉uc`JE^r]TZ#fXPǷ\זbZu7ќ\ LQּv*[֍t-Qc;>'NRxT};Tfo5I4)Vڸ$ļumrZ%ֿ-ź[[:v] J,% w!B++,V L&D ݎVkbyGOd3KbKu"uSҐpo4f]z9-&幌$鉉Dp@AEee`7ZuBRY[[r󭴓@1󸜹8Fv8n'5Wt'R{rAPܽ>uꈈMؕv9#sg𛱕@_ǂШ\r{/ͅ=~'0'lr_Z >G*HΜ>_ڶxfg "`o8"ruVcKŧklTW.mYX_&wR^G R.̆f ܤ yzż: HãbA[U 4<.W D{],OGXё&"(SܲN$E8-ͰqľeknjRRYަ>>#%uN}Q1.C:N6L ,ղZ!~q"v1ʺ֑Rf&+%}R__KtuHR(ٜ[]|U 1& *- smh1i]uY{GB4+",r6eE2[Dv1;W8Awad8ϩHK'K$gc˹v4'\6!PhkquQH[֫w2qKo^wo&+ܖ=o$H|ɩ.k[Nr!I<[xv+M_ꎥ\A[Y]:`+*,h->5Lr=hS\nmuGKsРT_ /$?e["ZVh0{.L`(G3tǨU|r'fc㵴Eu:[E39>P73;_U2iQHmj5HZ˨kI찒z+ό6wȕbX&ҿ~IDl2 %adX.-!CĖQQ۴ϿQ_rfwʺBQ{SIRق,Dh8!+iN69nt!efHDDrIXqzm=927Lr`N.%6T޶lmEq=l4]&ePCKh"\&r/Չbšz ab=xIDAT%ejٴn3D 1+|9Q8FyͲJ* TBKv OOgZ٦̅.0a) hƜUdz w,rJw"i?\h2/ +rOww:HfmN{3ߜPq۴ 7"vR->=Cj lrUU^FiqLZ 1m, KlFSEsޠ\mgHhl2pAؙC[[[z|:澞0$\sX$nˁ bOVW9UJa MڿOrf /˞Jao-^>ޛ|t[i3§=3pF/,N@ǩ.+-5]yl+8}~r^7vreU]\..ְM OʶUjj|+ö/ Ga?ݭ 6aJMO=twy /=G]ӑp༊t[<2rJL‰z3߰}RrcՑAx[/\ͥűbxtA`o`З}i`З}i`З}i`З}i`З}i`З}i`З}i`Зv|do7o-? ;~e4Hh>J}iG OH#= u4@0K4@0K4@0K4@0K4@0K~gl=wa#߷ɟϏ:t}.u4AK=ѡ߾Pn0_`"Ç>އu-uP7: }iݿ6nnlQDg~.~ܾ$6n?+n毷Z꼻AD.$X'S¹ܵ*Y"%7o_\d%^,XbrG3^oxRL,㌦͸̣'^ن.߿[l/ .g\khvWhODD©NG7't/bu19gώKۋ܍Ͽz5_vW<(}roSwio?+xЗז𵩿qYk`_yz2R|4qwJYkRrG¹:7Qv~|sCVn,j?/Yo/l8= |O>8;ۮFOOD[vCF/a6>YfeGUꉸm;͸!l/}囷gEo A ^-{fv6|̱,n47 ]dѵ'Zp}㙎Գw}գj5J ]z>\rGn_SoOm3|B|60hԯYn~9Q:SepN'"9yv4orwe7=qc+Szy%x %-▜3OK䛲ecP~vG:Bŧ_T;n@Yr[TT z"p9ώRpUҽc_ϾѓQbȩk\D{Lez+:~v ".X]3jN?uhͦ StD:]\Xonlo;vRQfSxeY}{n9a}A]l^sdyfvQFe)'| ^OW?RNF&>iV9lKoUK\ZZ8sƵR;XH''ݪ?ZݸizhCe=Z^xtE: C k?m|}^hscх!Svɋ+k _M qwXo\A™=qOk^W͟ -L]̐znGtΦf}{;g&c }nSU-$Ŧ իU;SvqĽ{'^ ZS]ў11vd]Sv%"47vritq8ґGXv^81s_m6;;M-L ]xv΄󿱅ٍ޺M۳DQ?3X=s5ٍtI]͢zdz2EzR"W\%̗xn'2ſ5c%&ӆD>9!#sӋvb/dE(r6Z^UW;{v&n-Y:H+uw.O -;؉{C7 ;o9D3 I;.M=ƷRJ~65t"' zG1J-Uj^>bůq\u*\9D=!Z|߉L/ S-po\׾e["D߿i4$Ψ|󔎾ժ#`8ζ>Z1ұ탣ϾҨ"'Cshf(_֫G?T2_O^KkxB갥/lb_=qI{{ImozDDPbք`xdۋOz~z"YAP4囧._} 21?s'~j~zVZK=f5]|nڎH׸4m#,qfk7{$B{FUϰGܣHq͖V]Ϲ!cqwg[m ُ*iQ2Fwڨop!=!o4'BDOr=9,XHԭ2h;"m bճ1UI#=ɻrUmd{jxφ_l">_,ٵq"R2?my e淴M͋W<mQZt=Y/kuRۧIUG{,^ VK cu|6~E['%_6W.u5M64s]n5 i~Kzh;˜$tns4+k׈Hĭqsn7/0nPQ=<]u%_nm6"؇G03t5A~g?-8d i{nO5ko{ Z-><EcN~}PmX5bGm_PF$ݠULɊnw=QOg>.SGO}P.J֗uo7swl?] #~*1POdVW;޹2X@'>4f)t65tHY{Ƶ'.}:Z O"Tש5"D938:s3C"s.zamGnl 4pzϟoӽ4NOtڡL6][9}~5#G"Uo郟j"b׽c.9`W{ё62u,ynݦ_*6Mjj1ffMkqktT ]U'JG3/7ɓJKRGI免 QLȊO9QJ{d1W RC ~\*9lhg?qX#7D4-eOYݶm-{ܥvfzkةoNÇgow'Ç\${?ݑxA{5ou\Z5 ֟T{ ;GjSR?Qlk~:qX~LcM7WuA.w>9zބ=G Z`ؒ{s!D0ou"h6֡)uu궏B4GzRnɱm -e6_*l]5Y4_`UQ@Ŕxa ޱFqkJV"gsJ+wH}sF}qreɫNѵ8 fq׉0xWKl)oHč[l6WWK`8{hQX9ŦyGO說C46nzoqwn'}5/ ȏW7©wjesCg-TxiE6qERWWsQYS "̠$wkEP/hXy0M.'|UK2'9 WI%uw;|15v57}jPL9g`:QOgWnF[Um­ci\ uPN6iƭYqx8n[$^'tJa`<-7K72TsQgھy*t'!cm)2ctЗ0"Y\abnWpZY.?^Cm?hk_0w`_M TW|sBN6Uա&芺G$zNv?3t/}|klw7v?D=R9Ao4d`qkl~^ڨ9 |i{z\w/=])-O^"uM8 _s2ux4L? 8fSr^xPkxniF@O.ۓ.PS-wq'Lwu7mlG©r׻<"'O/wN}trQLdŀ8QxݍvsUݿHN"Cm;3{a[ĝN}t,<?|T.c{3D ER%Y㘈~vKsnDi{dT+-֦[q\S]S?}m>8B|FKG8z6s7a~| 7[|LXx_.Ů  _"`bƭ_ǻ" >Boyjo{;o;=Z ʡ^1x/ N>Z~I/Kz}/Re2?Eݮ}a{R?q_AH&[lbc[[]`/ #xIA$oI!@::.#Cm/9~Tv?KweK?`hH/ێkXIėu/]D};j_7o_\lG*Ϳ>:CyǾx= o7x\ /  /  /  /  /  /  / w|?aC*{Λow(^Row凡RiΛo0ߡxI0$  /  /  /  /  /  /  /  /  /  / "MjFIENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/choose_mirror.png000066400000000000000000000573561476650141400260170ustar00rootroot00000000000000PNG  IHDR_L IDATx{X/ EU5EmZJ%jpT&ZEqSM.>ZѨE QĐm"&bQP@cf`Tv9g\3nFB!0=!B! B!hHB!FC­B!0n{BkȹQu0v>/㘲!:(wnߩC b@e<a s$ʭ~=/Ut0שOÑp+D' zUwC!z$ !B!V!B B!hHB!FC­B!0FnywU9= !kj O!vqFZ-[LnBNˆÊ@v8+6|oGCs ðp͉l>Ɂf|.v1Ҳzl*Z.ρ fN7W?ryO8.WMJ䆰6w쟯g.`>B[Y¾-k1d]Y8f˫`s:|Zl]\s|_ VL}w\_J#SePӖed OMe JƵx|C1kᠱ38XYZ_r2ÐESqz sÜI[6i3ex9m ~%W@U-$h[1k#c>OQ|R^#9^u'x nNl'"YeX:M"`'-WuBfz/q9I+v?g|Ӫe83eEOg1e#UCL|> z6"~Bt.(7q>xPT+U?72;9j*ޠ ;e# 4rle_ηS<~$ -Y̒ܶ+Nl+m1g` UT@ 'VrJklo6|=r{29v02űIUW}uv~643sCtQͽH}w|迯& XT U_*2`NUO^)W}YuJu@]"aRKTnKᳯ1yz*Tߤ&BRYT}]ŋ1oO:.puE9bp T oj^ޢrn;-ThGxU HpdYуlUym"VPպWv+ ߪn{A9mC۳ݙR*Su}zN[Bw]2^s">xUUU^uĦ}MS 9AgyWܨg} .y~6p_6AnV-_nW`T==e=tv a8M+y|~CG'pk9eoAd%RG#I(l#NmhPEs>@F 9 pyRFܶ12̉]cGP6 iuAQ?h@UIlce{R6CsZ-髬XӬpwf:lΞ`mYCԕ1,KnT68k>BZlv>\v  *Z@˾^ę|M_U=`y|~"MeB>5WS6UxE73U[1PB]wBi-uiht$Qv~O_PmXS!#ZMẳ._b"\ y"H=m j#h3x}R_ct3.ĵ *b5͘9ׅPۊ>_ĒJl ُU;@UŕM\+ άOzբż+ܜ/RrOuLw/_ U_lՉg|/> ƍLr]JSF]nTXaCQt&йv8j+ & HoI)g/o//\ܦ+IHDGgGhӈL2[nEeh[@Xgp*ߦ\bZ"|^XxrF̔3f&0Ylն[tQR(#St__;s9v]p/+,`;3c StQ7[,1A;r3v3^ȖCѼ]UWcY GωkY ˓qlY Aoe.2"VΫm _ e$7T漚}AB}Mq3g0E}BL{DeVG)ުBnM"@wjƁ;G?ߢB@]~C:;D/ոH`źdACKŎ-sVU[&##$ОVT'7jF4f#֜#(Jf^=m mANpDP'lr[f:fN[ײ \L9r9 prgtM'Mr`܁`׌ּ37'UU<^z{n[I&^joT\kπ|9A16yxּ:~V!Ć D*b=-ܛ*~^>_92EOCS{Y| lױv=Bv=qq"qvW!B!z B!hHB!FC­B!0nB!ѐp+D'*_X79W!:J"|щ(=/׵گ79lbB\ #E!会† aBCr I]ϣ T!z.Q,v9m`[^`GPFf$Z5܏-ۂ yd,)=T9F sv̫; 9c5YBPm>-fSl.>!.a\KBn]n͆aRoXu3<7̊y_O}ۙۢF+D@s(v \MPVϷqu$hߟ<U*}2w&sv1 Gcm@l+p{V&L'j7O_S_KfF3Ss#5+ U(wfWCe7O_c\2$ˏJ+37Sy=E csS68u}>nE9rEp[}Sﳨ'cvT;hKPeM-V$8C¬JؼF rs5U9E`+xk˪\f6W|dlIl ĩUks@l/*fPZd˞(MŶ{*-]4_um>}Z)57TN;,A"2I݉ºF>ަ,c2yLO\O^FLsb;A1̉l3<C;+Doj `wWsfmsF˪6wLYn# BW;3+ܝ)߳%k1y:p-Vn[0p1R ~:C{pk9eoAd%RG#I(ڙ-9楪&8olOڮ<%d(hM?o vGUAVj31ctgh4'Fջiac((O3=ZuoSpש~<纰b4ck>΄,~.X?~b6h>Ya[ W!LܚRvFaAiCj6c5CAz%w\P8_`S[kBl31\_ȱWW-Hf8Ep+z~,0A@h;m:s@PA$'iBr분V74im5B_cp[ZNbbG3l5א̈"v>BRٸ!űb!E,V+u†?BMvC*ҕ{ tv &ot% [gFdv+z|PRh -jiV;Oo0ð!hޠ: O6gYk:tBJuZ>[i{ 6>cHVa$(ܖP_`*ֳ(ۛrЙsVwr:i$óST)puio?ܑ6 Z='^^M]!zUUu (Qbm"yf4MS~iĦ<}v.{i r0^ef /nL{m 9B#EֆsltMlcdL@%$Nd7 V& " زt]p~lYV=/.~l7KB~!8ۭd,HVݞ߯`+Dd?Ep+B![!Ba4$ !B!V!B B!hHB!FC­m|ciHQm{'{܁A&F0BJr!z,&}'_Dzu0_g!/(ٺrj9{a5Vȇ~_=[!zI(+✒©z/|=v.'y/Tg޼MHbwC yXSE|[=}# u%|/iOO$IW>FE|H!W B1 B@>MgoV ųwɬ,kc,PmEXy'8,HY;P]ժ!<'.!lP=Fާ[X,[Zүs X_K_B*X&/USSmGgǑ9d頩f/bWPf~!ov.#l'!oPX>f1^=O%g Uw˯G[HhA&P>8G?/@֛lM)l쭹첄+}t71D?as}|8I] ^[0>ރ([U^[PucB|<"H!|+Jx\jUb /)4?/.nqQ&tZ v%R4 { NM"-x  'c˻ҵCǸs}̦V2X>q¸.`xu CۚZ~&0 Ͱ IDATC̾I.e^χ0b'rB=MJ | ~< ۉ5_j+bw^ @Yg:BK.i /H:~1?gkg9j8 \,_U{yAI'3oX&3e;RddlY,_/p<&[l2 (KL!ӟ`7 ƅ$)ًyc;\>q_+SZ~-4rK%TF3dȠC "|@调\tWƿv` x1'%=wU=ٝ;w~ܹ̾sGԻHV%yJi};:&X˷Y+_ޛѳ,.+k_SnO}~wgZΒ#;3Gu;e'~`5ݬzZmz\!1췞ŧ Rz:sWx |KP@lc9)`'|ıWuc=hёctz0ynuJ%eYHC[艑$|L +߇]G7jUKqT[!z򓜺6/-C('l" ;$5[GRf5Q~Ϊ<@ vlh>L56SV_qq }qj{ҙ/;r {ǔ|x!hnS6FA├|98$.U-UݴgGLaU]C5^с$5Tִ?0_K8הS0!@tR=f}$q<'sQs2e 90 e飼 5TWn㖂2D\G0O=CRy؊wҰf^= gTwYd_Zۖl r{Xnڂ;Ka$ ct23ݕ%/?9č*?i;#5_繀HEN~S}񞷓G`&{}<緜!O1@ڍ7y+oH.df4hOs,S˳`#<~Y/O;#YD~;=yi'} ԋkxLa褱J^v-_gگmX> >dY:VoSOV_.!q3illlAa>I$?ĩǻ=fog# 0W ~- I+6{mv&K?m9sc6Wt[j046s+nyV#pHJ)f|itAvA@$ ϶\Zo1[!x:ƊDPiGvH1P![!x09Bt)9L!B B!hH[BB&tTndܽWܽWy卍X0%'F%ЕW>ǦTnd'lyjjj{(B<4 ?C߯뎡 !X0CӾV͕8!_ !vַ/ʭBY]N_Ph566+V!A _+xt!U;jiim_ Җ ۆV#UlMLL055mp5u5fRB!ASSS,,,Acc#M?ϧL&]m!444Ѐy)DOfbbyϧ!mCF*B!:D7Jg111v_Րp+:y&У)jWmAznBaf=Ю644PQ)WF}[cJϭB'Yϩ[!=!ESB<+!W­B鋿,m$ !谾qE_p+/H!Q_Yp+ժA_p+`}U_p+/J!z*V!B B!H= !D !y2+I-:cJ#F?B>D{B3dUe[NY=cQJN4/q1ȗuk\6\nnl 7ʚа{ǭ'5`6jh׻|nd7ydz멫W^y<Ɓz;l{c-oTȭn(ݷJJn}U?׶߲!7VKeu?~a*O0sWب6O8n?,#XZMݓ.- c'[VC;%$ .e^adapg}}s#ե{͒ {k&bZN&Ll?(,&.=3Vn}iW~=nb@-h\~g+8pK&7V߾f3Kj-?XZ3i:j(+W0` `>G| %ʖ>?q[Ϳ}J[F!n8" ϸʫx'Y`'7NN}ԊT}[]n&_UVFrC>D­VMKWaoV:??+f<<# f*7lnyW!?guǖ3jӼ@.$l2` y׉<&?wu>#wxZ7P-k^gr7FvljuQC.ݷԡ Pkgx|׵xjMv^5Cb`snWsj2 zj`fůUShĞo;,(W1e'}>\85*\Z5?"c6j1!Z sVmPÍ> `uie\΅?Tfsm0@e^~kƔ?IuvQ0g||PBꑽ%B!fb(6n_ګ*Vcu6{yj? jUre~h+VTڿH1Kprxj i {&󕦒h>cW= |LO%=WBNV'vۤ]7`(07Sk_}lS+eέM9z8_Sޮ4h]k`I$yۣ-zA?ORnñK @5bE7xwX?zKjw5{<Sk¥xB !LMp%R7<\'CYIWX p\KM`ЯqJÍDn.nfMCk \H+s,گ͉{ç'k)EnUOVN.10iqO{əsN*0xͨvbK)gu+<_lFQPɅcYDn *D_;GԖSv=hڳz&d#V!Den 41I$rw7xǺ5R`UgJeMֺq Է˶)g6>?i3?0U񻵔 d8f37{+UUO`o7I2a?uTZi:S:G ȑpBwk)5s8vl؍`yS\Ȅ+n`KTI ޻~Znv)Ϻ Xt%!&.) @=Y٠YaF+l嗅U I5\,"T_eF1lyFO$ܳѤz*x[e-.n޷ Nkr.}uJq-#VPEڹ4Ldxʡkgm4`auwZ]-ՆJNܣ); `m@񷷸V<5rnGXpONW>NUYl}%>#ВPJF9Ń,NOBa T<߆Jes^+![!K{-?o;ׯ|53ʦm-@)`ڏwVD`fe%n'0iEĜHs WI 0sޙmTF-/; \W8WO1n$ |W/u"~pnR-#o(),& ,b@#W]Cu;(; @م|?i@q/ _)g3A>ohoQWۈDgVi^'+L0#>YL$Ւqj ޘ7+*9a{τF7(>[|ؑrY6fxIc_](D#!C1d*<ʎo_OQ zU6/Qy[Veڲ&ˋ177 333  L'Xt(;Gܦ@~FfkH`E#;y)=yxř6IlkXi$=Euw,%Éf=WWejQ<)4iUx4)uyܿ*0dԶ7@G0|dqyGԷ&q(oBMa8og]mVvymS 2Ooɼ'qd1Lb7l {{X~GͶ 4 CGB;I([ҳgQ_ d%Ux*02%sH/%&ja!}Q"2I݉ºF>ަ,c2yLOƼdE/fo $8o /HAN(?A2/ e dհJӶp5U[uBG±=\NcgjjB"xCq} Iz1Ok^Ϥw%zQ<^BWjb qɑXGI _QA\RcD9嵅c,C_r h&%+^K㐒O@uN¶0TkI^ȎgmYAܲ5d iR>ϡ9УܵZ<Z@+K=j @wﾼyΙ9dCǪB! Zz[5YTH%!o gD0'FgpՌ9ړefh "D'ilhZ5u!uBg)\ n1%LǓpo)SLcbNK_\qO!xd/E#;>1" x<>E Y/O!sWKY:T2o,F['IJL`w e*ü;qN:x,?{jBB]skZK]9IԦ]~#`~N3g/8'9nͽGN{W>r-U5FnO_b_ZvncuU6A|j $i[ue $]?@q Ky ]'?B. hr{ǨP:cs4/8j!hjKH&ʭ+A|tU[E4ǂX.|>H2[3?`B:21i56L#ӷ 32z88.PTm8crFh&.,,!o$!CP_P^B+yxcˡyx2(?Zm??Aġ /!lEeP0t]/'nU[gYNyNç3,>n)(T'|V_GR ;G,( gؿxd;.kx ϳj44iחߩ^Hɿ}<]wHJ/`Kx$ Pƿh /=w/ʭ b j1[Ձ9JƳqj vlKU m j{Y4q|V,HPNMڮd|xSx!.oG6q9K~7wZ3s!!(Nx~ ٱRS@>PgDȎ}%y fNdF0o"x%VY;QxꭳG8o='IYz`,&.iþdKldRvzpk$;\Ipn4ЪKB_#>=M0'&S :hS}Tv6"rݚ/,lyԃīZdf;q$=*sbAFzC.g !:H.x/ۨ1*c- #l]_X6N?GabbBa_ԓQ'ݔc ~kV#^t:D^b4큗@ae'cﷄ~߾xqK!#p+D`bb!CSRBG; ff^/;%P!DʊÆPO`iiY/B!Dאʭ=%_QajBr+DP(a![!:9wv0B>Ad'lyjjj{(BtFihhP5wZȯyBL­e.Z_,Z8N9 _{Vc&祫 ]f!]$KRr a$`VQuȅ<=qkΦ7kmrgap$]sIDATXҠx?j'vҰUgqdB x.~\?HĖsiѽJRBej{*Kȳveh^ɯN"Z˩6~~I,29|xz(唕y9}'moekRΦxdzӂ TŮ@l:ejTOvW,K8,m;Ϯy$\Z|\Rt|^k}H%em<0f ,yFsͿzu.*ᶵၮ:ʣo K@Q kjdE\/O^-nYcN_gQ3SWrsmlrKY?oz(>SOȏ$9%Žn#(+okc&n|d9/HЖ6_Óp+KA蘱#9q9>T&-Q?k{o#O$far_lLҾ8#;!iJ>^C=,x%i)I, E#p.*U/!v/ˡ8ٶui9u9+?TiWM,+*=OPdymSTcLxv~8#~'1mWRWĎ'KAZY6??ȹl%]Y`+}\u00c w@+ h& :9$tܓ+å4Lͻ[]-٥,E(ڝdc\%Ij û4`+@n 8 3C Cu]yw>[EyKX?MF{n TaU;N^0{dށ6ԷVȗDndž=}2ۺ#V!Dߨ9JFf Hc`PjqMai?Ԁzh|\K=7itG4h# fS-3Yd7k,c c]J\N"l K!X-PO/Oudt|{%5;Xxzۚ%ƟPM6Qz|E箇IYTLJA94r331E:R7}&~hg{FEqZ2poFdis'h]$]d[--Ԁҗ^du.IͭBq IR6Lz)1{اK? aw`{Rqf|XLI10pΏG惾u p⭶zhk}RNB3~87[!} wf#]ͨ xN2.fwbRJUefǺXq_}!4<&xq2S`d^oeIRK;4iNL { vιkRm& GX{0aN2o_bCs#ns{!\hXkGIJ慰 T@p[#-ݗ^cJ\1~vXt%!=rJLߏtbLMcjÇ..>Wabǚ8r܍Y3] Hz"a35̤47"52.3|ٽ8? ^jnYdشhhE=حs>BwW}?DGm+&#y~&!!Dg3]lXZewYՠ 1nF6݉Y<`$c?eQp:Jtdh1p~H bcF)O\U"=Tp+aV1#t)Igڹ)b/xE.;@^HBC6f'+CF6Z-Ճͯ/Δunϵq?xu445?RV-QUrR=}0G^Dyuڋ%1h iL%^WEXT>GbeB>{ k$o\ cf>Iuٻ0+3<`LVnØevtFhrfj6'gvwAxhˬ6JUL!q6oS;&gj=gw UPp%XMG *R lUXPZB!,سBȑ#B\b:Ix0Kzz>v\\\pvvRRDPdT(}˞!5B! s5Clhؿ23]Y#~Dn{s.4/&%Dnn 'SvƢbCB­BqL&jD ]+}݋]+'ebI?)Q 7g&e B!:%e BܚrY\P&B! B!ߐp+B! B!ߐp+B! B!ߐyna/q466uWpXT`@KK`?ǽBXZ|H_wH:%5 }h_wE3^^ !uR Dkll`+B B!ߐ!0',W6Z?6mW(pV*mYo=(fۜmGz6 ./Ztp˓p+f\V\a56۸N٩mtqse!~yxcH 5 ]9mq`| \83|\ io}=k899T:T*qvvIb p{3_hroV4PLs39⥥FȧZCC=_?Gŕ`hif VSS555 899BJB$,Zo ZQ}('4QlߗJחٲ1_ Vn_ilhnrZZZhllp+n/[/Z{[42eyrمw_&AWe?A/8Pl 7I8v{bG3htWIfkW8֦bN]~L` !Ѽ:6 lL=1SܭsہdbH_ \ 5:_4 ]EM_wO!n)n~K=6^{!L}F̈ ~` sp19z `ƥZ&M4;}eBS_x3֘m5]W  'fs43܀qlx?>5Ǣ;:;#oE|gjG7S)4Ɍ#Vd6>hX"bŁZ;p/9^ڵltx[=B܊z=:@e.QDvL;6A|z.ORgn-wR:3)봁?]0+xCݍ7?8,Kl!x'xLB+gB02}Sd͗ॎ*8j{jv,Z͗Bj>>#6hXL2p#x}ӈvm5ހlAfLyҶ^! ib C*aWm>}FkPED.$\/ANv\^Z4?רڜxG^A*vo5h-ԗM>%ֱZo~hSm}Wf1emT10- Rb;H['a,|3=j)ٲ_枥b&)XGk\fX8;Rm\֬*W3bp dw* iߟjA͔$Gb:A;v)$E D/"i|ھ;W@IO&@_Lj:~E#JDֆYO"8ɃԸ  /|o1(AӺ^ac?u1v۵WIz.77z,7 RS T1š̥PI3' ͧJ_>nJ: 7r׃8 ĥPG.NQYFqpe}@)>8M7o ^/+,Ky5D]+Ɛh^k6{6w>7\dۦtκԼW( BtC\L[𙉸@.דYHZLKT䌩%5:$Gv8NNGoYk'-db<jax*M`|%|{}ٟ Ԓdr q4 ϗvu_Sٷl@CZbEŚ8E;qLK$q\_35Ͽ!;+YJR3qƫR,<1ĀQ=Nя0^u+cཤfj9yɃIPX VSC ^ˠ,|UO-v#\uJ1PˢU7))/!exָe:ҿ1]VD|qsn+ufƐ}8?tM"Ts ӛE+w]_`[VQ#&+rvNn2W&sUTo‘ nu ՝RLTB$mjy65TKPz3 g:Bm%?:il pN棬5.X%jO]<"3%/UWn簡Q m8֚ SQWvtd8ڐC/j#4 ̪ j-kaLz@xb <ijݡ pe4Cy}뷭^tV+ils!B nݹmv[&/ve~̈"Vv$)2PFcHZ+{RVnRJ771DOdft'kaJ4}u8+^wR٤μF(prr܇ywS3v *^ =r9!<*+~^3 j=lڿz$xi1B?YEBۉ$3),Z3]Gμϲd 7e!i*AMm tGd7?q%rN%0-ƅnCŚU D=(|(S\|m^U˺*Kh5ҹ4bB~Oa0U=yjeTD@Neez,fBY5o\$"Zs}swu&e:ؠ/кWu|ʆu GwVNcQrҗw +u\t;;₋ U R4HeWAj*V{`u\Fm]={+7d _ϭ%^G;澇x\epV-]7`Eb/MT9E߲h͚۶syƨ}޿#{_>qvvFTT*Q(m b7DMN*y@]DW睧Fhxilo ۴TľEDhjl2g%׮ЪRqQPTb ..8tv'Zbb --4U"R5f\{{Ǚuق8nꇻsw2;E{h 446ctr2=ȭfBI1z}]aT~5!UŽ(vt.!:B![!BoHB![!zJw ĿUL6nÓ ZgK0 ZZZfM7i}I!D ŀ nacx3vL.y%D 5ؚOwB39)KBa@}˲BN%/Ih>|8s"w|}{+ĭFUTT-кNNNm])KBF) q0!3nBŲuϖ&6BٛLjnB8Ecw:ȭBN) /05 899ȵkmu+ !n k~ZZx% !|4HTnnvWqVpkp-Go([!1-2ߦjpeVem y(֜[!]j-M ̿@­=2Z\p;fOp+!$x_[Vg-v6ky{ EB2ZvlFú( V!eVa돤,A!ݬߚd 5mtMz IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/finetune_resume.png000066400000000000000000001220051476650141400263220ustar00rootroot00000000000000PNG  IHDR0q@BsRGBgAMA a pHYsodIDATx^pg91q:ޙ9k w×5^3{Yig 渍c̲`wӧՌ{0i $Vja~ɲ%e3¶0B[=|3|2+TU*TW7TϧL͸teDlƅKMlq&@6| M3>lodȼ'F2d< y 0@`#G2d< y"o֛ooyY=W,GV[&I_jHa//7((ƭ~MCV^j%37e\m|[.Z72#˩Xcn94tL|j?f J2((W^h[qhuH ~ZnD|DDAo C@AQEQT)65կNJ_GA_?B 0#ȸonW;  (ƫn?5GA8Ĕ. BAfi*3COiIznEF}5-m^-6.A̮n9wݎIEAQEQx gm0kπǩڕBa׉L^7=aiz{>L?& 56">%>< uy<}& EQEQ*_/*,X\ D 0.ɓNpc4;Vdk^ij w2qi$/] 3r獫i`2`PEQU܋vr݋얞fzZ4nR`75{`8!FVD8U)Z'Ѱ"1J<1lU?#/+Wy<&i ȃ]EQEQk*/SSK4Rzc`7O!)`=(cͶWED<(Y,{!i}| z͋9&yIWpAmSI& EQEQ K>+ЪSjA 05) Za<_ViN)"#lώzaW7r~9w7`3o۳Zo=n&;=-/Njz[]I~b۠JZCU"_O!a,lU)S{cbP;SFZ6:̿ hO N=GÉ^YÔ A_c/?79|..;gS2 6O+ .|hl> %9f (҆~&jy?`Mf̿+cWkjy/&]0Lr&8 ncl*b{_3!̞r?z1OŻ&F1VEhXQR9zdMcvy?/MJGQuKMBvI&j/jT9 rl8idԥh 3$6~۸"-KWJ;XVUVLjj~\ڽߧvZeQ~\Ez̭KWʷ_KZ?':cU M婜eZZi{__<+fw8 8NUs_#+󯃡ӪduyYjYe2GaԒoWoyS F]fyÝcUVWQ˝++>839wȜ6XMXۆ9\,~'zV`U2br* s-elS.j~m4v/y ְq45VWʆh`<*.t 22++벫uQ`}iؗ.J 0tj6K^y|r&K%QeC m2V`84h`_ipZDS׆+`ؔX0`p}v_|ܚC%WLUuVRxO BP_>ڛrI$U_vy}2ay|Aƒr w}Lkbm`N#kK˜{gK-$kk̴Yٶ(x4:C4K'?)kij3f` tx4x[ny&[϶JSefzv+@*lrMq̼w4Ɖ6u֬ݛ\(^;v\,]ۥ/9q]gKLOL~'owrSVmO?l!vc@~`m"G&MM8 kS}ϟҹv<׺6 y.7M~-ir!2~!>*td2Ϗ6^xfz4 <_Xiy IۣCm8i^/6_ciDC|$Y]/waiH/B5{I$U_zzqB|8wf9́`xBFt<&&Z4nZ7xH,ם|h{͟Jo:G!306F=n`.9c_Grfne@XyRҲ,/oY6zBÇEL`}{ۆҿs].0Qkl@3 do=ZW*xMc岢ߦj ;Npu[_.f͇nz:?˙[ݛΠa.iX2[N_y_:B˶m07wd;}H1=7OV4q ޸/sGȜ 4Wp~ԭz\4 =hr}a1Ҷ^t;nʶB@|}{tٙ|˾-2gi VW\וi$hؑ3Ý۝16*g~QaGrϢ{ vrΝg5r؀kA[I56RwT9tpAVRۏkc68M!Rp\n.uy~c^CW/ 4ԑ=5׊ީ,z:Ox\8-{qtZ\5A鴪䕏]/kuH/jW_}NL  OwѦ 0ks#_׮4Wp~cvV_`4l&oe5yͅu`J*wY.i̶4@f97r4pt斝YujQ28 B06*G {`h<؆ ow ~RZ.?ΫftzkdQfl:`4涡ouN!)lKyoS,|G)Ƕin >7n_}ѫFn-uyOՆpX旧w; ;BIuIND{̼G5nD x+ iUgGto>%Z俔H/jW9_g^[.3]c^`ii\,_떦%\yB=i2 K7 dVbYni1ßDl7,5mi7$zَhl{ޞ_oƭnY\?AG~\r$z`˽e]/ۖm4|C)kp,uN)4x~ K^}[{?n7h4:a]j[{K_'MK͸+y}kBͳSD܋yz;,lKyۦ0Y**$-GB1mtۆ_i7|-Ky{lحF |WOo?"{ iUgK䙮q#ReU~`)7U"dNmmM= &әm"aQOMիJ O_Fn~Ua0Ư16mk_{zaF)˳^]dx旧w ey-_4ax=0jS* g*So/dMկJ&%JY-{oy ݹ"YS#0 Ӫ:+_&_?/]U 䀩H/(J؇+ey#D<Ƿ$=bEPΗD__mm@%QC<7m pgBiAnX@m`)7QŗvcPw|,_mjgyfO)SOыw:M#AX@m`)7ŗvcPw|,O=uD;=O ">bEPD__mm@%QC<Wl} i#Z:-1P[@ Ldi7nڹtVqҺ6VC}HT30gyfK\B{_#"jHa޾ 8⤦Wu+ d>$Z. _ē噭*>WNxL@`)UEQEQW!U*ۥ.Go-="jH(ᫍSw;&Go#=  (pw.qjJ ^䡇_Dr{]EEQEQn)Z TmK{Y/ra[W\EEQEQn#8."T[* .4맾4+ 00((r+7zBĔ.w .txsU@`)`PEQVp 8dTeh/ ,\EEQEQ ǩJ/h(*/~)i:U@`)eEEQEQY7^Z{a516m-g?%(4|ڻ1'4Tho ]^g׳(8pOKPc@ e|iQEQEirw47(kvﱃ.]?g[^Bt]i7Id|<P;@ |Yb|Sdh\P7D~[wah=G($.%KD^;)cB08{Vd?S{Pqt p#- \4pB+3ihtRXcB0i>-~^{hhþX 8* Rn?]s: _7>MqL_VڴROfi~PzJ7a =mqL_V`B qL_V- 1!P[@ |YcB0DŽ@m`)eZ~z(jt1!P[@ |Y=q&EQԴ+ Fj!(jj#R @`P5]@`)eZ0(E0P-EM"Pm@ |Y k`6 T EQӵ0T_Vϕ/+~UVsyO?tswj@:%o{۵:\ěP#%=HKQt/01&uM'?5TcdUH/+sDdÒVdϔoENj{e*Ity| +^k*g?-Go:\ѳq`B,N&O@]wpu>abMl&&=VF}3st;j+`XFk,/%>VIvb+M6N*qLu`EPꗕP"0 5!/e4|CȞf4xt9/(F 1e'.K׉&rh(a/(FJ 1۪[aYie!uE?s3Ea[v\\Mz*%04 Uy0kOR1QaW۝rt8jBR+?|FFG'^Ѥq*R?6H1a ~U0"R~ kjcBl54_ Jٸ6ixRghϊ‹w:F6F{t{(A_.=-D{V^t<Il;ˢ')Ip-$[6`\:̾C{@Ooo@!i1otș/ytfn{!19y䰴: KßǖfA7Hic./ơ9~ȼ3_oew{l>`tOqY~3z@:>wƞ~^?o}#Y.{sޅ~y[k|@r(i\]fcL\cN[fr.^Gubwsz>ooeUy݁#k3moA]qru<]})sȾY_af-r_oڬW]W-{r>t-y|xo1yfuu1fc> f9{ݮJJ>k٫`@B_V'_SD഑4r-3,Jh:< HmϽk`x _WhcZAh,^"U@mC+)$(F>/ضA(|Ng=0b PmGGaO9$Oj#/`1#mt{Ny46;%ow; m|b{5xB+Krݟ,i N\45f}=lZ .~r(a\}4Ͷg']5ۜ`=EU9y, 5:vskU|} bP`str[ftYn D{涮y}Mxyu(47exۑkm+g;q<)/%c910TBY_V p0 gE@ƖӀMhynF/аBx!+,:ܰr u+X 1q%}ϋrW1}=+)\ɈJ1co;,o~Gj밍 {wK/ڹ4^s|xL^iD8h{{zq@h0~I/yXN~A.`ixG̿[>/8X-ޯ <{]YKK Up8KP~/eyׇk0>M~t`כiHuKBe|K9uKN'o= w}x}"]{FqI;z?^b='9ek4vNv-VH/+P3&E^UilMnE 5zf:'ۘ%=$` z^Kt|}^)O:*r ;K|۪nKҊmo91X!`zlп<`n{Zi̘~󼁼_KFN7|gS{\ÝiD8G_3zzJޗcI=Ao0<(͏.H,X2zZ/|NCىvO9O/DY/lmޏ-_7ӿ~/vWv\#zCZlx4|Vh hpg)%:ᓝA y=ι+ FP֗U"4mtuO0@$1. קx^() !v!N7A?y2`xzNb~"Fd؊ {uYU"!Pl޲^(6#W=Ta zv)W>xGZ线_#rPAü雺h{: 7[{DXzEi)$=wϤky8SHjly_~{r{-|y Av=AN4m=Iy^wŖYt4s .ơSHF/< oy.]~tLZ:?2/'`))/!oRNA+^%y*^ rUa =5 <_z~%W/]{\+)NW/]01, FPV !~#KKCuhsb \k'5b E_~`togWZBRJO!1#X^AC۬wӅ>u5૷~jv Ipng#o;.zt=oa{ȿaKיX쪼]n9y<@Ôvۢ<$oyG{J>9Y0 }_~F^@  ϐ^L׈7 N@3|;rk?c6^Q/w{]3hP `~+H~9\ NS6DO_ur¾u2zENuE% 1:h$3q +h_4&}!i{!>g s5,XY+9åpe-ԠEQӵ0T_VӋڳBOѠ":\?H#vC{\C4.+ /P k`6 =-Dm԰BO!ѿz_wT-VvTEQӵ0T_VZH/+BAQt- Fj!(jj#R @`P5]@`)eZ(jt1!P[@ |YcB0DŽ@m`)e "R E8&jH/+pL_V- 1!P[@ |YcB0DŽ@m`)e "R E8&jHA((( @`#G2d< y 0@`#RŞ6((ꫂ@ErrEQEQT "R=ˌ((P{@ EQEQ]@`)`PEQEw`G@AQEQUEEQEQT}P{@ EQEQ]@`)`PEQEw`G@AQEQUEd}Ûj[]I~b۠JZC^UN@m`)VYJoܸiӍ[]I~b[%-!Q,/ 0&; sKm;}H>R= j~)0J؇cyN/x~#R_ sKm;}H>RWutto֦8EP/4vcPw|,饒sܩ~9w>CP/4vcPw|,饒9 +J!zd`)T/Mq@46TU}ȥ[dsYJB,珔sEv."=֥+[w̯%r }^ꪏ婜eZZi{__nyղGe#OU{Ml\'Edu7>P0*BUfCZeO+K՘:+Mz) 6VYƫX WUIt5Q 9m51-3_+;4H`\ڽFwPS|G)D)n֖6_bx#XO`(9헧'0`p v%Ϸ-o|''<ִ0Ǝ 䕏d~:&B澜v8IUf5{CY 0DFΝ1{*SXSfx~ܨҲ.+(p\!_my,blT΍8(P=HU{g`h#%|`EMUU>d.?uɤxlggj4]J{,im A&9buׇFh6܍sbosVʶm>U3ׯPfʝwFW#$B*|G)LMہ6vi^!!hmx4OAݘmZ/O7t}V4d0ϏtyΟrsd}ƫ:(r~sSed0jaIƫ/L=@]r~!n"+eE?ǝVܦ\Y} }Cv[_NYfy1-gbYۙ}[ujn{y2O}n=MğX͸['Lj'rϷ"T>U;"]?q ?x2Gzq 9 j~0֥1 FP&3zٰEYtpKl0 &hc-.>G:a'ﲿHT\*8^w3&B|^w7!{0s_:)ן ]3O/7<E6ܿ@Na^o-EBq9"m+HG#>oKZ~o-2?O)37_gT;}HQ- ۠qvϿ ]Ӹ5}63ިi<;?uxO"w* "RD֑y/HjrxUk5.~sFh|,v P_0嗦Jtz iF슆\KzLXδSx 6o~4Eۺe҅@6"b[^ 0yp%\6:'_m$ח@6PG.7I˾-2g?om"6u}V( F4шO_5 tx$p_3g;Rᶔg~Q!"R ,!N/[s?Iq!ϛ]Z6.4.r>sfD1LJه\>#D-t[Ll+^`\W0}$zq{? Q̼ǿk#*y>*QW=t ?&.`)T/MzA(.ip纠LȳvPxאb|gpCr D/l)oq43ԙ]lӟm P B pmM$6V6 )bәm eipQOS8x#ˬg@ ۘ;'h2;B\^#ik_56`8̝*S10+EIll0r Oy1rt*9ϢYfG,3;~& ~4t xs35UwTg#*6h[:L?Cl}M9aFs5MinE F߹:h(>|WjߋdO~%_ryH.[}[e8f'@ KSa^Lm|{zHiE/hq/YaK^}[r!jfY?xtU,Xh҃$0}ҴԌk/]oڍ^2C/hޏ{ARB4 ^yz04m׋ٞ63-zT=hp,4EBu5k|^LK|iG=sifݿ2wSƩ,v`F`D|h-9-\H, 9m\)ˆ]87,Du,E>sf *I jҿ_ƣ]_kt707kZp\ӻBgF^ x8r$FXz*yxBLe*Y}o35KS^XJ7n׫@P/4U)*ވncnoܯ *_ugeۊS_niO,Qg./d;_jH_~i//6ƶ(!XK%3^_0嗦œncl*b9T>jH_~i//6ƶ(!XK%3^_0嗦œncl*b9T>jH_~i//6ƶ(!XK%3^_0嗦tȍ7`u8iݏWl@}d>$9T>Ed}5J׭u?^mPT}W%-!byNt}K 0 +鋌(("jH((=  (. 00((0#R (("jH((=  (. 00((0#Rv9/3((jz "R}OK(( _XA0oG@}x= ǃ@`)P8jH/, P{@ |a7#R q<_XA0P-g?=OQ5mk:x= T߸q(jj#R @`P5]@`)Z0(E0P-EM"Pm@ |a k`6 T EQӵ0T_XZH/ʗo*<ԧWp}|qk;5 o7ZڽZ}c`|-xʬSmr?ꕽ{䳤J*y9vH?KzE}v=Z3Qmq*U?7.xUaY`6 򿰮%[ KZ咽?>SuGb/%t 'Ez6YQ/xvWp}\+GƕS w;8g?ŠvmD~yUr_o`|SyZ昴9XNtfFK_]mHt6~˟KU܏lUeM0JҀ J$f86d7%:^9!Fluet}:D %::^)!Ft[5{s? 0Kz:`,dg|"|zEF˕i\K1ϛcb€ -M?|fP~MLQltgH8TQ=F`GP*օVِkix;qmCO ўnxϽSHXx:xxa|m9lGbBgxE@K:oQIۺ, nϺ| ߂OeCt<$-{W-3~@w> 3k)`|%#{[;@.z cro\I!Am4ly Bs> V}4:6/(ovI F_M#To跍qz`he1>F>{59gnfNQswx@JOzs>g~#RKKy/miܵi]S/WkӠ58 uo~OƵ6:m5ƽƪw2Uԃ-ރt<>|GްE塏]x4pr?r>wAXVC_签pErrIvJ%Ni͠e׼y-GS ;#mmw=`nY%9L2N2, .N*c䀹"Pm@ e}a!HFW^# Dc:B B{/~}() !v!N7A?y2`xzNb~"Fd؊ {uYU"!Pl޲=4d޵_h<{BƒvzAv X}|L0 `u%Ol 3 4]ٓ{U;`RF~i u/RO9%֠ hN!_(,X-mϿ$'j˽g},8$o5&aL&_žny#ހ5po*.m;aLh#9\֧w:Mn[*붡/>?k9F/~ or{; w۽'WuϤM}v9!9Ael>u} /BˡvWrUԙ2|?}+r>?( FPV !~#KKCuhsb \k'5b E_~`togWZBR50?p`=z {T m^l#[OG׉׀)$mgEi`|lm3 AT%KFH~9 ]踦~LOH@|Y!AI]`RC%ynC{/kBh#:_: f": S/a+9s<<׹E~imPnZy(@^(|]&}f~6^McԛV[<%o:bMw˟]goًO&4M#:1PQp}u]Zmmk7>_p"Ὗ4c0~FkCe#>}eXh9YU^O2䓛2a[f]4 0TB_Xa,lĹ-m>[aWpIE {\XKtҰ"艡BtO~v: \@+g&\nU -\~O;m;9,ڸx Gb$w~З7xe0(9m׸Zd~{8zôO#tqnr`6 ҿ`6""kAC0,x8qxH  3DK7x:xF vADt}'?g]xën&ߨz{>$(zyϋpIG\FM^?t넜OLMfQ/LjՓ8!{ `)VUvSGyU=j#R kn>7nxL#|m\炨 .8j|dH:{uÜEͯ:/EJijp)\Y 5EQӷ0T_XӋڳBOѠ":\?H#vC{\C4.+ /P k`6 =-Dm԰BﴑHO k[:5Z}>i# k`6 T EQӵ0T_XZH/,BAQt- Fj!(jj#R @>EQtA0oG@}x= ǃ@`)P8jH/, P{@ |a7#R q<_XA0oG@}x= ǃ@`)P8jH/, P{@ EQEQEw- y 0@`#G2d< y 0@`#G2dP%__@}m2@~`<'dw\xh؎1u˺Kݘa3fȌ>L, ;ȷzI>f} /=mY$1Ww\@0,>;C׺+:8"$:p}ތ }N$ֻnxwEwb4ڻ\xdH\h`A8\>?aXN~ NϖB95w?g|:arp =LQxH/ "SlPѽ4waAÁuF1?q4aDJ'r!׮ɵkO>vYtΌUn ?f}^.(`׉ߏLCZݔ`5r;$"@*NoA㍲KOf,nﳧPg& 0Ean;BM#x;=wÆke~$)@"Rqm{mGz4PA~ȕ34$ ~yoJǍ01/`k`)]!}{QOwZ8F, x׷p{>D8!#xܴ;@ R 4^ E{`am ?8צ8.L Bt .& 19y:IACߖo:پG J̰^Fx? H (0/ȅAaa!^HfrNcw|5ֺF5'ßfrPQ1PNwj$N' @۞z? (-"l8]"|Jk'Bֿ.FgOo(1G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`d77FoFΰ?U=[&nvH-|5-m^-6?gvu˹v$d׆w?k_!jz6y .v>{NJ {+MBnPѷU7v]<:xW~D~PΏW:AyY0|h?X*O Y8W6cLuݲ}7f،2cmkjpB0pӫ ! Z'Ѱ"1JӍ&#'G)n鬒t].|ȵ˟72dܱ@K5; pyo˂%?3]a1ud1~ҽ}^e^'1sfXgfs^'4Δ?h-yh0QȄo?i`q\^Y;G6I`X\xF20f`rQ6 A{Wk| ,eww]G.C{RzP7e?>FSsF<+tZJɼs7GG4s-eߏ7/?'ԻϞZ CN!)`x6, 0f6t^#7-7p"0zchaK Xss)`l O闦ɓ#kƤr@BX@G/  5 0IF䂟:nq#3tkXư7y8/t?e AJ%%{xy^RSK;DnJ/56зNkҳ33졗Q;@8 :rg>Z62N39;kÓaBP`LC~&jy?`Mf̿+cWkjy/&]0dµ%Z/ɾɽ2 C7p@$\"†u->RDN!Y_C_#3#37#c]s-==MuKMBvI&F]fyWGzlsٞ y 0@`#G2d< y 0@`#G2d< y 0@`#G2dǃC~:,~ zD=}BQT~u?HA/7nܴ~ ro=F?z@oY/H`?^`)`    2M\(䛲ŎugdXڟ^VsdZqO)USr} &40`"vm]n'a}[Ɏa{( jQ("g#aEi?D 7Ѱ6h=2aEd|{H@yjp;+-Y ei; "s"2xʿCc۱]5w˼o OBzjY#9>~s['=(f&h/09ԸU=Ui( [[Ϛ#)OlkL͏.0mLoD> j9kiV珯:hgzbKfi,q%U9޴ iSGD^y3||ЌNכt (o6md(=L8hqlcʝcAd\yϺۉn7ގSb6*w'U)* 0 ^]%ibF"[ٿ0WFt"@jpi[#vք5ry0Yrq44~`ם㘷v4Zר6hNGHaN,ܼ.%x8ojZ6󣡊ij`O`8#7LiA[yW{ܬ; >}mhn;+f{ӱsakba77ɣwOͰMwV׵7cϵ,Wt{ݼ0}~L<;~cڛNn^/=O0}[y{uclkYN Yooiq@ `H4=\=:r& 0zI7ȻE녑 /4c=(U  %3z`['Ȭ~h!ȳ dMmܻVNs.^)s{;7ɝilIT큡BИ`*T6")?wIH׆iDiDJNokI;,=M܆}s <'ί%.ŖN=+m@; tntoSit.3Wc^^b=b{Lt5|FOދ`%ΛVzޘOA[|{̕,șmߓxGsv{`]9IX>r/\o>N+~e^~@L77}e\VlXo9:h#FJm\j C@ϖ{dqt|kcѪ.>7 i`3U.Ӑ pily\z4}NH׶oƠ6T͸lh bLҽFJPC݃AvuI)T^cL5ZƸEC]s]N^k:bړ%w ӌO/x/p 0e| <'ۏ) tbn2o7لB4mon:">IfEuwݦ]>fZz uxsMW̺bύ̇yo E˼ 0F'qhR{f:f !S7r[2in#+= 7BKƺi 7"$~c*)G<; RCwx -4w4M;xNde]tӮ"R~ә7i86FгŌ0Y(wLSIkC[>fyM0&3_nT|1ƌinǺNJPûvO~%_ryH.yx?50W@ud ,ƮHˊŲ^Ěr1t:G_HrL1&4t qd,鱁D"NgxRJQC*w֯{>&Bf'ds=~bY3%JR޿4 K~zOdJ$ofsm -ܽ_Mc7U^O=x%>^V͗7/esujVY 0Ɔu N~5lGA&Ŕ끑0(*Uq@ Z`b?^`)`    zA@@VP/0vȍ7=F?ȂEO*Po@ z+~ zk/E__P0@`#G2d< y 0@`) |<({Z;{(ϿEweiHAnܸi7@?[G00-+G@ (   `ȊFd~c6u``)`@ +< ==ǝs7c00h0 ƮHXQZ#@`)LClk{J3%mհ\Ly폮w <"6.1~R}?P*쐖ˬ6Kw9_C7Wd璕.YݸXfݻPVq=gʆ}ݲybsY2 voƆ2szi`G|BB`qom -㶜zqyeCw 0o0I~bՔMP| {˜3^όvJi7[f.!nyϒ6ݷ[(+^?+f{r2ߥ+;_[/ ӹNYȾUrƾU%~izx4aiz<氽GPVqB|E{d:^.9ˣƬ'ddܹ-X)MӆC7b6 63dHn/5MCeY؀r'`fqt4?[+eeHqyΧ9>l{htj0H(r.M&w0(]&Ov`<@ eA~6 g=rQ=g.H .=6z`$'ۃ#W/PF'n@OzJS2y={&”0"FGr8n3@1 t Qva=&b* 0O*Qÿï{6/"N1G:-kkSʑ &.. KG>O6AvYj?r`ܹUpG`DS=r띲 wɜu<;/;S^(MvSVE£e#ȃGmTO)yكrNjH @jFO6ܱEz=S_UGu^i0*\|c@`)|pU-iKn"O-~lh\#-ꃷˏ[,Gu`3e;dibuBYv>& -r7띲zlYrEl[Uұ:e͢s%'탱}P&i:LƎ 䕏d~3ݶL  P Vil;7 1n60rlo_7n|߹¼^3eE%Mi/5qY ^C4SiOF Ƶ6N `RU%~izx4jcoP{@ Qвb\,~O-ȝ qFsVx\lyQ6pq.`hz`[tH 0.ˢf;ee3] Թ4*W? 0.Lq ?gs~sS1(]&Ov`<@ nx5m).U}s=%׊SV43 } v`]as^b2:PƓAJS2y={&B e}rJÂQ9qA^Yr]/w.M>#.H&9rO6ߵI:ICL0GEH<=X.;t|'\y}o0d78m,gi,ˆYzj[G5.~?4`zb#R(@|̼w4,yQ:mhpSV S W=r 68Xx_)M~0Բ{~CzYS#hbxvkeh;$ףvY`eQӸ_dQ`T .+-׺eҠg$ /Q^#0骵?:gsXqUk|JZ&d7\jnj.v?|e!*700j`9K69:p YQ{[eYbU2qk@@~`lX[']Pg]H Y~@ R@`?^`)p@VP/08P@ +@ {Z;ƍzd#eiHaA`A(J?GUߕL4 y 0@`#G2d< g )HaA!FQT~u?G72c:/kI%EQWYD#RЃ7n{~u?G=fgԶ޾8&ee?Pl>~s?T?yq zA` %K  ` K  `ȊFd~c6u``)` U]N˹1D@ E>{Ȯ`lG6wI#Sh슜U=2:EE)`ZgoɆU6eUrjEUfG׻GWGÏz?R嚝 BKWN_00E`X3bߨc\h;e2Zٹd8>ٰU.{SEn9gyFUGu^i0*\|c@`)|`-!-//Y/fˏ˜2gbze2e@N$ d~\~`o$̰~5ny5xZ-BnKe y{<5ȑffުf*~DHX);}i`l?XfhK"d1q̺wn; uz,}w-yg,[nzh, am y2ϿO6ܱEz.ѽwi0=sxRhAvy}2K_-߹k ViXr[x[><`gҹvd>qB6ܷFگQpu*7è韱WJk,!0 "{WÝVAn`2Ӛۆȳ dMϻ3xOwI =u}s^|9ӊL*KˤW?L{s#R(+p"l~O_Ãxp~]5ƀA·uedC:0e+z[ iEÃبyEizoD^$G L}S.(u*q޽u^z^[/lשOFz/h 0rôƨOT`c샆^[nMҗk^Nܴg :PƓd45.޳w``) 0F; 7z[ SC,n7S?<0JioNgy赕|5k6O3=8ԧf*`<{+}aN!)nt=,8wA y~/ݺ>*ip2Z J`!Cxm<=/af?ƻsOY~Nou jm4ܿ vO3jo4>ql>.ݲaO@Gw[, WJS,2~Ss[ske^p1*.45xň~E<z{1Oо^쐦g`jY\3`mƧUa$@ Yi==mEa_Y껲?F 7nܴ~ -K#h@ #  `  0dH2M#q<@ 4`Ȋ.HOq L"  0dEE+r6VVCP{@ 4`Ȋ׻GWGÏz?R嚝  4>yMT85[dNBS ;Yg?&O6"MWW{jHaz}aI\J5r;20UM#d>wl{Wqyq򱽟yo&0y0J>P*쐖ˬ6m9ˆQ3+}3qƋ%eÖey‰8}MҙSt 0| hD~~{l"ԸL^xI@:0 Aԭs}G *6^.p(A</;eE9c} _#SGCO!)%(g`׸x{A^)&00#˝KO!s#[ݲf9ʑ 0ԒYzz @u\^HKػvY%00ETPjg-Sӿs\~@MUktnj"|? 0 04쯮+e5v&i?#ޖ/.%/J-r ch&ir78t>Pf<k-zp>seZq]V;[f5./7j L5 grkݲmi)}?رNW/bVmUh@ YiL e]2dY6u MmF^fi WƭC+HaJ6ɶ-/HuNzX04运_t(@`ȊƆu N~ 00h0   `  0dHzACnܸi7@?[G00>,`DQ]YD#G2d< y 0@`#G2d< y 0@`UadvZ0 (s?~B>pΕ-.ev;cy&dP $\>,xyPc +; Ba1udQj$y3f̈hL1 <ΕCdֻnwEwbP?pl|u C,d - ?!>,A@C_ 0Fw+@`|-Wq,uS1(`]+y6zAnG5_'k"v BXNHGF׮ɵkO>vYtΌUn ?`xfh{VH.B A}?(TdƟR>{jIB #r N!)`z`D·{zJY5DA"rP LßX´sEzyN[I:&ҵI2(W|wC `8a':,Vv܅>[C/Ј40y0e{DGu@`@^z?['Aٿ]Kg"\mH<l{dEÉlp ,"}wNd^zIx!DpMp0|^cao\ O rNm z`"K2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#k@Lyo jwludǰߌ{:>{rHX5ڱPmrK?!K[C@yfmd^H+MOE{/ 09IaCV&>0 I FFAfB=%$"w7lA{Wk| ,eww]G.NB9q!@$,  }}\.:*J3mt*vYtΌUn 7D.@(#H "9W&jLK_OovD{Ę=0vXTH4DǽڻC ~. `q{[ ʾo?^*2Odw{=$WQVc>w IX,>6=3 !q^$dD oy y^oU b~vwSi$9 @+$OhN'NG}ްsF=+ =WfbP%xz4AB$8'B &Ëċcڻ?ݛݓCe*"=b`?}P % yyp‰=!I8=( G\ȠloWUߵ'hƼ׃aw8 DC 6\pC-׏_B!F/ //B iLI0 A 'wrzgx0v.LEQF?|=2xmPzouf=|0js!CR@yAG^ } 殁ԑ ^p 4ĆyyME^x@B3} `];O^.ީ;+#_<J6Tp{=D"㹞 @%虡eK"?5#5d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2d< y 0@`#G2dޔ 0I;}[Ɏa{wco\owiowJ7O+M mrюqM,2E9~}iebO[Ei4uPϦMAML Lya N9 ;!jXڟ*^>n^z.&ӵk#_M !YZPXO 'nWr=AȠ#@@".c#: 7 5 3"mf{ |4(##Wz[c@"Mk@==0" ,[y&聑 b;]%BDB},u" 'B1'H)}Mzb7 蕦0p_I=%О`!>h7Vb* N`T 1L#t 0on,RCl$x"y|IqhyVtN%POdG'5À@Òz8遡i6'<_o`ig^1?h׷(>'F)'nw%]bVv0M} G`oT%x})AzrsaDyz1=N!FdGbį8N#:mOwÅ!DR/_nFf,>'F)S5O}ߨځF=0P  ^s#%NEF`@mMN!"R"P[@ U- _$|7鲬tx`)]6zbc /5`8CO [E:5:ZkqzDTH{CLdk@"D{Foʨt:-&.; y 0@`#G2d< y 0@`#G2d< y 0@`#G2o f|>|& q"LVYIENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/mirrors_upload.png000066400000000000000000002515631476650141400262020ustar00rootroot00000000000000PNG  IHDR r_E IDATxyxW&{B HHHACm/j9xRd9G--(K QD(k@Bl31L2l$!u"sߟsgIj^~V:@D*JR@SzVW?IV/{b own,`0(((Hr=4W%<'yIRhh5eee ]BT ŪrT%W`` `088t-HPPA;0Z\nHhPrHj*ld\?\-աkڛwM_7uhbbbFѮ];Y,L&ѯhZLnD:Cd5Y$sef}wUJ*srlh? k ôUTO! +88XVUfYeee2L2LM]f>\K Xm.Dr16D+OcBuWd=wMJ˯3n\~~~ T``؅ "w:@741Z~ ~:Uֶck&ߠ)_~VYU2;ъ ʼ fhmndztDYMzkblFKrRu\)mu{D5:T~Ъ5h=2QavLu]5#>BE&}W#EEuWdџ_ӑ6].! 4hhX, >J{S$JGU7')Qǵ' VhhhSڲZ%?NԷo_[mڴ$Y,+++K{n֜5Cv̬:k&ϣU:E-8:pYiNlc^i#Z^*oB@t]z#myX5z2Ũ7UҌ5G.3'[He7 В %$tSD۶:.p0뮻*٬>L'N>nz ѣr`0(&&FcƌѡC{nYV4Ofk1LсPX' U=<Q(?>/E^E>R1AzS.6kUTjج m;5sn:ȍxnI%{:zT] 4swJ\sU<Go9V)s~P;5=r2 CR^m|MZC Vh4*66VfY۷o׹s犋)ժ޽{{k6Ae-+WN5?;+bթ2}zL[.z/RN!e\끒rMT`ү6SfExzn۫iNaGcd U_ﺶf3^yLOܽ"gT1?[L*kMWoIB+>^g;{K+Cv7?"""K.d3gΨGPppJK绚f/KY{.T躆(cMzUef/6+P_ 6_njUd~ÍܑYקs5s>ITMJ-lcdTsjTy Uer ޷WY+rJib9PˑAU;uĺ*sdhpIII۷6nQF)22Rk׮m&OB}g5s999Y7nTQQ$i„ :x;`uqgI6o 5jT\\>cpz0ͲX,2 ׄ+ @&IfefYe}S**ybU\*ӣF&F@iܱ2ߠmT]>'%c&>C| U7" 2[G')d!UA0Wq $.Ct5#['S\VJI7j .7}nuzX-V|>b!^ j"gi5jp7aaa{>LIIIСկ_?Iҷ~[XII֮]/*""B.\pj2p@O֭i=EDDhȐ!d K.]Vsn_kݺuVTTRRRظN^DF'x]y]s)kgG1ɓ O@XRKTZ]zU{UyyKx]vK.^C5Uu,t]HEa[40|رޅgϞ:u6n6fIҀ\:M6\1*..Z ŋ*..V.]С9w]vTRR#G[nQ>}}-zsI\VE m|҈]4[ۜIv ~U-o{T-@Of,jm,w/4#]\>W;W 07§{ϸ6 OUo wsPt=t|TTTB 01>J5++˧=Ծ}{8q1ctqرC/@9V$mVŎc={TYYUPP6@'Iz@mXWKtnt&I/_nappeXo߾Qdqb mdX.5L7$ ~j')?T'Ϋ@2/6OSERoհ߶HI1efhjukNkOxy._:VW 5=ž+%00e<ٳ]رqff&L$;vLcƌх ܆ꪠ@7nTQQc<`egg+1H',,Lڶm |֨X,.y|ڰaCS`M.Y Ƚ_ńSFFi_Y75OdJoI!SÎu|mI=J>W#>UxS:&x.LOU1Q^eOyx6"V9U_c9R1Q$-ozߪ{V/I9ʑg5}t.s~ Vd2)##|KJJR߾}qF瑩vEEEO5yd7N|]C1z]Tk KeXJ 4P[x.bUXё;s&\=^')ј'5izJlsd6_4CoO1 ںI=${ mZ>'UM6۽*j׊6:I$*t*aڛJy^wo kt6]IM+V9P:{Nė`\ddd[TT$ɤkusv7+..N/^tt֭e65j(^m|,ЂԫW/U;g6uQ]v */BCC5zhF޽[999M]R4r]4[d 0(%_iWtK1V;}MGjSwkz]ÈZ˨Qԯ_?9sF-r9l%%%W$O>qu';;[]tqۜ;vڱQFd'|8n2ܹx/ 0@Q@c{&+((_e˪oWY5p e4߷$7ZzC5V4޷kRrLSUzca4`kgCBB^_^Dޒ̇._&:*_j憒_luEGGkڽ{Nt iׯ]›fl($.A)k}Z;j SYY6oa\wK*X4$_CA^o?Y-V*}P=p]xQ_KxNu[re ty߿_E| 94f5UΘʵ2T3H[/t zd^Zst6Jvv u U5tuYbF _4&_UVU[,6ٻ@Kg2_KA+:}YdVQU?ooA_3iOIy`И`x^JxЌtt\%&P &HwZb3Yu}.O BE+蜅MOV+ CIEjp POm?H1SEPl_[J*X͂b_SZ@nOioOA~~$?ŪBU9f7[K?s;XVeG"?Y%ȪsxO [^|w O^|k%2{]3~k65~uMIn5}hk}1P7-4'jl M]MU"@cVkSW4u@sb-/rsV??ӧ  8tj\$h\XuZΞSؘ.h2.'-ȩS'矓lv{bu# r>zoH{L&PdvJIN^{+**KZ%@֮R ~Z;8<ڶPHp {u' PwVEdWd Pl86u)L M]Z,2\ЊI]y.]$Z0$YA:@D*J:@D*JM]ɀӚ`K@3B'4Vh?j{ogWRJv@h؁nZ9 [ R;孟釥Kd))iR !!0A9NhY~{ܷAw Q76KI~XhY~{B7$Q7|>Zx8uN)zv,{jB^n]sfnھ#s$-Xۆ8Zz枫#Zz>&cmXfݧ>{j/(ܧSXXXEEEm8#Ih{P7nvsQ^x;w~[W8$u ֜UK5aki֕+W=_- FsyL}z$7uNG6{`l9 vIQ(kgZQi9iN IDAT= Z߰Ya!c;w~:jс΍޽zTXX.Z,IpBK\c)I:p jl$[HwuIz5S>&Z9"mt1Rf|m;vZ pbuUA~ZAmTj1w$<#@WE=էwOZA]^M^3{QzSBBjj@eԝ?8:k:eKbhꔉ.]eg$8B{uzG':.rm\V;ԁ9zg|G>bbb4$%*(c֮Z yz湗uSl]̞vR9wd K{VVo$gu(=~GN>#ֻWvVIRjJBCBӭmok#5lPG;gFJ2lPo7_^ՎtќӇ^R#CCjZ?\]rvR%Y_zf^|aNtWUsu?J;~bl4 saܵWŠ!m=ISƹ1y0@W^S4u1h^"W.SStU؃oRHR{W967?y/<*9W5tv(oج{PK4ijWWޠ4uDZAyyեs|1H:͹`Ỏvqq%޷侩vR4S;Ȭ³GuWp+m3Zw;@3A czXQu|+u}}z|׀ҩj=+cu}_.U#i:v[?NIlnÌWs.Jf.P~g&ṽ1@W֩kۣS}]5K)ާaǘi^m$)uȟh˯):>7lPu;{NN^xe y)7$[IǨWTw#SC ց9.ᵝ;ʿSLLL)/(t<'{/&:*tpbZASLB9\Ãcu\y;k uNlGVw )[֟}Xiq_:]f/{FΜX{d<<=X8H-y/iɍ!Nns37JLqj'8 [j[,e힩C:Ud{m)s6YZ43KjTG>\F _u[E|(uVB(CKJ|ALT *DWקؖQnq6#Y6&7CV>,Z3N^@IRWƪ (SJ~ʶ39GϦ)EJ7DF:: zxή6!:U}M%*$[ɪک1eΣSޫ9|VM*kj9jN,̙=KsfҪqE).V=o.;2}G$UZxS_:9KMIQi/mSo4eVbb"?^aێ}BCB4')`zݬH.}KU>KW;ԗ{ ϵ ?oܫjc\Ut*9A=ۨ׳_ٺ%2ȱ_strG:F=rκt;WݞŏjQ\o9S{xHG6~]djԦ&MT0Is';3nT jdciiG'HI5qiNaA*("4HW*Vus۩!=aFif+뽊NktizE 9o1FjxAksWq]H]ֺۘzv鐤:6m|\с7uN"w 9wes$,Q;BCBӭZju}]g@gzpb۰YF%%vck5tD(6T_\ϸ"WTTVא.9FҁޗIbUǺ:YiXK>4"ֽNv]ٵcՈgP/EISyVUێX[Aiy/W:źn.Q57uc\mS{؟Ǫ4bxV~Zwd*.bm! mSǷ^$]׼_StQ#G \rgw(\*IiKf%VI%UV pSuDLnͷ2NC-Շ#\wu.ۤ_ ם=wұ.ϧr͞ŏ*kzjtARv.&u^:^r2][.J{`us:Pڪ_+htueK.qPVacktpVT\ێuCx(ru o 5~I2V slNy0uJ3#뜏UpVuoZӻ# vs:j>&ʥQ#~{+9vNPܼ@M̪rS5]&>tY )RfVEι^f?"-z'Kߢb+e.Uϡ>צQufL-FWZq yzONA:O$7O Z sjwѭ򚈂:>bW\?km+:EVeĵl꣫n=XXm)cǑCFzvU9S렓.oeD#\m4I\jaV<-͔KW.oX+Yɒ23]t}2BY Oi\mȔ,rbB'H*"ҙK$u=UKER6ІӔi([]9V쁹3 }@!5}-zhL=TQ/lTGWs+uS'_jvGO% O`7RtVTV[U9+ZcjFEZޛ},R򽓕1|}bMj Ved(l\;5a tlg];)1#C]>S7e(^I:+u{^Ũ{[^tϒTѭm'7$[x߫tI.t_?V_:VڽTidI)|ʯkkjYS^Ɵ]4{x.e^sҞ3ck*% 6_׎AN u)yrMW;ŕ0: I蟨Mڷ=+M=we%:8Ͷ$1Y߮UzW*9* T4sW}45 c\",#zc OGhbzxX<x[6Ԋ6q}A6'CiGRܼ5=I3HM՞lHS ק[YVw:x )RfN͕+XNYZe>Kɿ[?[)Vwf"p@Y=8ʥݖW|Ϲ]VW8&:hu` yS`sFzDoOWQ{[͋ Ђ[ٕ t6W'2+mqjgruX\p'7r<1V:I2%)YCSSr]ܿAKw /@QR¤ ~vܨ jb9Ip보h\nv]*Cmc꼱SFHEST "wl=4jQ ]5cwf-)NqfRjbr{ tۿޯȏ6kgmխOzvjUx:2)h~jGtZ,np%:8. :lҋa5IR i^Q'K+l)~KZΣt)IgVl>X;o*s \]q.uc51ee.VZ$u֔rἍڕ2KgUp}Q1uZ'=T7vګlB*6i&SߘGʲ6uuu\:4u@Y  ~~_S@pN:AssZ:|Ly} A@qsy׳ee @#2Z>-K}oO<%0At !!$Df>X|,}=Z#GQM])>$CS@S @hZ%tVU"@hZ.ɀӚ7(thKpc+U]g5u h:4u H.ԁ쳺@u_b6 ;!~Y=`HR/f>2 NKukumoNv÷v_~Ҷ@5XY JsM+jP/@g?Vl:k5ƾP3 T7Qdg~YYY:]i9fڄ,&t^ JG#kj><'pʗv:8^GOKB j6u hA@+P]HMe&UGӆt[ݠz)~eu; 4r$a:@c[.*A~ج̢4:v(qϕXrIE/ii<`ٯP E"ŗ{EtJc=J ×1SWuNۺپ>nN>Cԭ@ 4抎uʪo?w /ھl>-xd |sySBe 2 2I?ڶ ip!8 x._-i)w/4׼8:kbj'"@@c"4 a:14t#\+ES!j41BsQ1NtZ&P:heN:h%LZtpkiV/>SDDTl:ӘiRIͲ kt_gͮ\-Ws\ /K\H3/P)I^@ρAmyޟ}v<}j@@ZDkJ{@{æ^#H-:h+FPT0:Fh{[j#D-:@@SXa:@@Z @sP}kCB-Sco2 adXƮdh*Em&@G{$I}rZiiCn5 jO tpsk&iæO4sjOwhUWstHrr8~5:V!> Z\GV48M;,]IcC5a?}z+v#8A:BtмDUj͚~^t߽aJK;zs$X U#C}GQP5\S ^S9a<hU"19E_cC-?h,;Cs~{:59X 9"Hְ[]%j5>ƯdhæO? IDATS}Ch*FXe=eC#D:;[ڸMacGO@I҄qceh~ :+g˶O޽zZ鹹u`Emئ~k>X>>=w>нVGOOI;۴Nvrskwf>9~I85Q >)4 Z3kC~ 6{Z3UKLN$=6T,^(@qZwmΝ>m:|Ԧim۶ZwDc5vt9x`tqQ;76Fu$6mzvGh Tp<n5 9z^CAjѡQ <7,f}5kZm>U#p<t4tu68ʵ0]ek:*!8hy@S0YC#$zX]E$HX!:xwkh=ʿm j7p28<@WyJ_|GNW4C5nտ/-Nc_K.  (**Rzz,c#|M#K4R?_ #zQojB} Zڋz4G^ `;{@K⢀ |/ h~V.?vM/7UAQ/ 1 WWj-QVpgRsh\\\VkU[}S$A>WosV\57F*qr|xh ~мի]!NｳQq>th߶VVj#L|ح1+W5}{N9 @͚KsEc^!" A7sM # ͓SON'nӠ; =^6rKBhȉ4UrDЍZxOjh~+RUL^S$lN˛tފjbN_?3]+*ʳ~Iʴ{2zgȸx WD*ܓc0zWW [dhrm^yWg A>@k{e@ZvQ|_<~f {*RUqIҵ|wO75s,U/^n0S// 8P HR*Ǟ^ rJ:I{&@*]u5*I9Z4lB[}Z6$%] RؐE._&++G]jv{r|'R1 7Z&Gb|> @kSUͯp:rQvei]^]_fVjU]\7sRި8m]vWDPW}*e&RZBvեB.W_W )Vt/99I_𜤣11Oj2Th^P x-\C,I%yVoޓW|#4\\B'#8hrZU#KtR* bYA.upaE~Y}t>4z=|/Kj A.9Il+5(:&WCRI!0T0.|l:d@VLl e ҭKz}B=2QIl烞Z_?SʋHrr@C٨>^CUs |$iKM+ЕIEm5 _eI7P2X*0W _R$}٭ITM ˺$dz ZAqC4+b@Sd,KmwuW9ʔ]Y-:jHLQf}6/~@nHT$98(/͋Ь2bHmT 6pc5PG>tZZpi+ewT3$++^&_>1P [}*`^>h>v_Z.Gzg^iekضmIj`Vk*I2j=>R/^Ep9$AQ|Y5C;*EhVetLb_TRxfY?1Ze26)W;vrikc7T`,{\V#p^6OH ߽s>6،eud]*g''999ɩMX栝+pj:BsFUqr~rw?BH.//_NLx|}z}IRA5nWu~U5|hmCWka>K4:vEdizotXlwX7|Դ*6m?[a<5-_t/U~Z&Go__o͘6I'MժS^Ϳs'MՌiV5y/]Zsmi$I02.I*5-_=UO^:i{[Wr$SPdjN>KW -ǏH$yG4y>W]Ѿ$Io IR O͞@E9G}(LJ{F.Iz Zu'${ə96[]9h`:vE.؝讅@m؉$:):y*WN\^֣Smu tT/xjZroh,֞?j?j=7@m@ck$Ig5hxz64[վj*.){$=#$s<9ӦmkCjzsDV@G%ljfnf)P[USmy+p]ب4&CE%zoz~tg'}5MDȖ@U1\v]{nЪwyIuоD@Z/t5p%Pk>^y7/SAS2ԯGwsin,VnY93 uQyƍm1(_Nʶ$Owmi_77ɾZϳx@~z/9u ř!P[ŽC{l,1±Q#%4́@BmNu}9Muq6w$IruuX%r2_tNOOwK>S7[.xl:;;)RSmNSRJ=9]coY3d\l-?V Ym3 VС~ܓU9bp#`{9g/ gX vWvhL;7n)t:vNuD#T2_4jx']׍Ŋږ&ɶj$ 򔏗$)7HOB',W%IZsN2Nk/`Ҩ$ZL dnUq@kw!Bkvtu&Gnt Se+T/Wz^ڻ9},+7}rϲ[6jy0|O_0a9VuYi -S~FwnT@ku &t >@K{^VuhP|s tpp?jtd4J:wQN{\I'Y-'v| 4J! 5˅,=4S^W ,VǞUs5^M;j*Lqׯ_מe4+sw쫒\jfƌWzGiƭ*.)ѿwWWYA$7/_qu7f? zɹwV9B ɓղ߿Z%̺M%%εvhޯ4>}VNNcJ;{NS@e9h@ VUX|d_֙3gw4!IG*S PUTTd e̹~||ueߟ˯{WyzzUw9LԿ__5}.dff!d0d(T9;9G yMII'T\RLIR±d=yz_km&*8GZ ^MSAC(ۥ!.oM-H'IGݥnݺH9WTnکcG s/\}، k7* z?U<^u0ZV+Vg_ZZ_Js4,Ϝ9SjXk}`P!ɎThך|1oo/62G@^>@FPWΕu$D/ѡzH+= sTеT6qhn[ nݿӆM׽"zۜF>u r8.@We[V 0Y[5 sB;:[;Ѻz5WWsĤ"a<=<4եKgwkWIj_ڹyyT{|t=GjMʸ)m}x搆N{luqtH$ݽsTXtC׮]өӒ^ԹouJJJ/WW Хo]c#V93נ픷hq>5Q@>>zfΣZ}e*..$ 'O[Y8XG_+I҄wk/šZE__\Hhĝô*,,Ҧ*7/2?Ywf^x7k$L@kP]IبuW~kk]*>n{c<碾#S.d)7/_Աïս[Wkڶzwno־}ly{uT`|%[yyrvvV]doo/ '@W%+ulSziKzH7/~~7YA ̆Yނøhϩ[.zpDoMի/Ug۫WO5aծx\_~97nЙ36c]vΟPfE]vMݺv>'/ݳTrb}d>>ӫW^~W{B9Iz7\jo ժt_iiڴiSt[*n/lZ$);ev}sQΕP"!oHMeok~fH]Pmp5ŋt4!YNNz+r%&++ݹgΜs~!wxۂ!h)6s?Oz?ݺud*,,Rm=*Zyx$?gXpkZ svvVw몎=)˱Oɔ$Iyyz+ѱ}f|| nu %ݽ4k)<UNE}v]&} } ŕ Q(\!CmݔttT\Rbp֤cF[;z>K?" Wsrvr%I**!@z IDAT-O*m]7-c:t._b ̋J;{~^*,,ۓaG>t]i~ $II'|zMv^?1=Gq T.)2n}8o,o/9sVVժkmmV>>*,,ԌS'u+ʺh {I=*}h6(=ݠ$I\]Ag(A]%I[FޥW_o_оoD,[+gT?nr޽ʋ7n،|mPƅLE߬]qU\Rݯ?Ox+u ЩPދW>8LqAHMԭەmVƍ:]{<=ݠ7n7'}A?T`uwW7nuUjŪC: ?ͱ{W`O|hJ=#G,~ =Lo߾V{cNr~}eϞGShMN4ik^Zk$S WkgҕsU.p K=uЁs czꩿQ\tE}._ m\p#3j)J>w@1_~b}m RfNo" :=Do%GxQ+.*X)n^8@?k/h%\]]ѡ{<;'G_퍫0~߽tYNwןÚU28Xo!թkVݣPHۂӦM U.^-*,(Ԩ2d\΍ d]*g''999ɩMMzA;=5.ݶNc?ڥW?^:"D|}j=Hvr򔛛n*_XXcQwyG۽7|F~p[rn^^N|!\ꫯ,۔MZwUq<hmUWv:@CW @hZ%tVmco)2 qƎTށ$!YK}Qx{Yo424D ;~r~JLNfϚZu"ԧw$ SO[;ϸ;PߟNGQ[d4VXwz$ |rrOZyuPAa6EЦ68qA>:,I5!}uY?WQ53-:;Φ4zԝ:CɫfK+9tge~brG=4kCrskg` }3F={=_#4mޮ{mBgObri34e׮{M%%Iԡ[coT!;;[׮]SiiiߦMuA ~SF# nnJNN})}5zG r~҆7l9K IvtQRn5yٹ"+7:9SgTTtTV}zɯ{:MY}^߼єgwvvJKKս{w9999xw&%%%Qvvv +'99n>I>^ʹknU2W-?Aފ[5 T9/H2U3|"Ew> thKJ93T <@=={+@WspxCєkMY}v_vAs$///eddkf|HWsۇh,묙HBS4a}:o+9QNN^nn=k%`w`}:M>:<*  4Y==Ua7ym) ϙ9995Xئ]9Fc6lDg;>׃W9JvSC;v`֬f,{/'7v6s>$B[W7vČ rNRKٜ;PFcIJLNcC'Ir+9ݐdj:{TXg_їl6Edj9[u&1Wx3 zG 𫲅qc,[}=X$o+jbmUVN_.D^{3Ŧ+9 SI۴W{U@sEΎ ]i͞5U~ݻi˶O+k4胿m$=c:qA\?n yuлf&kM?Ou9qڔ&蜜LJ=:ť_ :]["r{/dˑܼFʪK[g_Q.5!}:MZ-P)\͕&N>u$!Yqzl*&ԬIB [;=7wei(jB=*TP[m޳76-ک32x:zz꼫yJ8vVsW j*`K - B͎ 4We#雗hj4#^RZ0}ko[qmckP@d5Zdd[雗(,{ܲv#>guliʄ`RE!@W=d xa5O@(Irsk_ZﭶPJڵ֔ݶJ=|R %]3i$..I2$ea-;.׈ZZaWIRGOUK߼D 6I1)Z+wðIz`yZݥR[VoW&_Dg< ^ۿ`^0pi̝!Z^)Nt*ؐҶkZ6_鋖(Gk (mעme?D #M_Gg鯪V5K߼DSZBEs%IhV͟e?WF -X]V3ي~JUJ5(t|whn1i?XmN*eK7(=6^ 6hֲ% :bfn1pNE6U*p#*|κ9=EfPz תpzK h2>k4mi֌@icU̐O_ҎsʞGslS^)n\k8+?MJ\>(Y{OъE7GmMe]Nӽo&.)nEjf*B5Kk45&ivgɢ9RƌxM+|ABՖqS7~%S^ikn]]\ =TUl4]VQNAgT)>Rì  RXkiT?O]MIUv?OYW0~z5=Q1БUbm#IWsrѫZqƆyJ%7j>m[I~Mji|v*K2@)]^U>eKU6]}yt%WM[C-3pa:@ +\\2p1U3[n{$Jw^4vOuQhzISUU3d^;ɿY "]ӱ9쪿~|cZ8mܫ TR@JJKUZ\"I.J$T$ZlyHʮ&T8UeLṴZ_,LF[1*B\TF=_+zT- *(PQ-{` M9*M)wYײV˧RhՑA32w]պvRK:yYlR>_ЬrsM_t1s#_wUwt~{_^T%U--ԦkT<%]dnSNNA"ɐl}9\.(q|GtJ=|R׺S}PWe-]5E+6U|E߹Dܘj%,gbmfb]%_b\ڍB4{Ylh{5|*}'E$D)\oq*]/m u˩%]5/D))3J}4=]ZLmOH\N4edWŭ6jjxNua4Ѧ*d \Sѣ5kizzV 3yg(|5>ͿVҕ8wZ-6aO:KRsYµv\/5W)}ʱڥnOE(q`( IDATh\[O}7Wkh W|UƻYyÒ_ו(pM rVQr:Vqi;0R~)뒩\8'EGTp꧊OV3t_aۋTx4\qv%c4R6$_afxNr\R{۵Ay-m~;S p.\N}ֶB{|*:Ff}pM;mUnRpQWovb8Q=˲. [(&kp@q;w]3e>H3`U˝`1*O]E(k54 {,+L%&gY67gjcp y[zcyg>x>\G4ڷ5Di c lT ׼aZL49i@eij#JҍAUϮ9S;TjCu VB .@,xMF:Mg(:  8mBK2,F#op\޶D+ֹ _ZξZZw/sWk!ڗlxWofh; 9]{?}..߫u޿w⴪<W]]4G:%՜^U|EI޵>0w})H i#64g+6Ў9_UrXU2bI,Zz ]ZUZq WIҌpmڊ4Ϯ\>5/nZشĪX^;fa.,k$]KgΗ|upd}𼳐.`˝%\E:+Ct>kBg,+ɺdMAmGWܿJ*[+C(u&QnB'Ht~GW:Uu^}~HwW:]{Pӯgm}&@MMs#@גk_f۞j3U<ڟ*9L^UPOk uE7_ !2/2B;wQM&E"#\7@ ֳJqV}yv<p ^m= hokӻh_( TPͺZfHGtsgknfMu+Pg+՜sF?xv$zQWx{'; ~WoXj#0w||h @'ʳ…e oon@'Eg*=sJ>>U?7ڳ\\hZ .Ȼ\pA^^^:F{hxK0|ҭ.s"#sԜotQȈf%76ڳvBp*** x6@\Xh/ji+m\΂ti?~\/^l1뫠V=#@XPPP ] 0.Tt?FGՄ*+$_ @x ר+gB-;5V1.W6{Z:,-k^zs6`JGtmHtԵTG:yh :@qbi̇UYUen/9Vѷީaٸ1YcTPC I㔐4N;4όYN.j@!@}Y(%ܾ.cQDDoƸ'((PV.׾mWM?Qиz wT>]ӧ+4}Ɯz;`]eU[N70h]zJsSey)Ok^vۊh-%ICc%I;KsdX乳4wLIwߡXo% SRR Iv}ߐr Z:@ﯕY64/iXƎR,j%f\8ͺn=4#.[B{ouNP^=UYYvk1J_q`$ x:@RrTJVqzl+A'Vz6 ts93qO׶RZK/jn1nCJg֌jnbʿU| k5q]5ah־3U$d *>b F+66bgi t)B3JuY͘,$X3㌐cΨF'Ioe.uzKI]dw4 n\RCʬWڍ;JƎ2-]Xs{ɱRM7NLeKh⭣5h-[|o h}RS*564'IOWh9kQ%Yʍݫ/=^Tq*/a (}ɂ=|[֭uj?Smo[5t1n͙s%\K0ΞSh5ݼ/.u:v>6uΞ`.QW孡JtGR=:_=:@'E)tJR3UUW0@}Zcͳ3{&Mm딘0X}G=v~7F]=:-Q~P˯ݘoS|Wii dX >rX55tOG={Оx,@mO!3*W_%ՆWgHڥ5jo@xαOSg?^\I G~ܲRRF{c !##9 j?k7G$嚟K}5$jˇ_tw'6ieϞv 5wb- ch~aYvޯ+f+(s\ F%w*󓞚~Iw?MO{Q8µ)%&=/X+oUDg}_+`;o̕;lMA@]$))1U'I[d'InfݭJv(/-q㖕vvfMn e+2myY%pϮ\fB7;]iuRҰڻ*g*C RḐRk'OZoUY4DIQT:UNױJן?өh{E #h.[gB-UdkLkIG~>ޤHT]O鱵(h=;*B'+$U(te* 5gN7?ӤI'--Z?4"8Zy.{_ߺ70 l*Еg+MQ>bzlh+_B5t-_{wKsH*?#H=4{w+ՖoGOIш@4/@w@95EߺmI; A(kbfi\ f*|B-(IVӖוg^۽Vi٧4HEt!kmhI#T-CR!e>OW-"3%sRȜ~UxI34/cxYqC^ й9Vs/ɮZ]yvzRVQ2W1DOi|k5ExN}@K8Jג+Kd8Oo>p[qQcoKhŇRr!1;EkӕB=hlgokv8E6ƭ t!##$jkc+ZC46)|iؽ:P֊rRyTWέyjY4{ ;݀ sf Tx_Mz2EZ}۵fvW!)P{KG(HҬ}J]ڱ1Gohzn_̓sl㖕wvfMne+2mY+yyI7ܦ{yn*i<[[yNRyQc4k?g/_=4lfUQ:=iNhٯ'Θmy>^W?l 6@5R.t ɵ g=^%|Gڐ\$Ҭv9`$t'+@W(xRjҲI#:}eΗe%^hBFI'-Mzrb@A կ@W-IC:Ơ4u~۷ek.pkZi5~GsX{׾.%=o~7rU>^=8eMEc4Iiͳ!!J/˞]A\cbz|˹FԁRўFd9$elZ#{tފ6[h KjBG%/k9y7`{$%YQk[$*o_7sC$խr)/ɭM6̣.ӷn Mֲ:f/zj҅Jyv=4#QwfJ׳z0dּa >ue\)_ӞݫYH!Fd_!ieZ-a;#5nh+'cYN^_ݯ|<b Wvu-v\eEr):idɗ֦+Vrgk?w@%s.')gTdצ+d~^^?i1!#6_W[JVԴU`}΄SZwNfi{Ng$I^y_=m=*9^&~8uFWx{[^^2hlg]p]^^Ffo_(!!YǶo7o_tD[Vj3.QnE~5۶9.n9O,֬k{^$iڕ+}qcG| w(Uzt7{{﬷/%uFH_eU&*B7mKK$I^У<e,Ҍ_2O?FK?g&KyڭǙ.V;~'Zv%JWhoQs/,mҌY8r6Wg*(뜶1Y:T|DKm{(((P˖.15yZ0wv57k tVMz8MwT{n{uO>wCZU\=ltRxAvn)tGkA:..q_BzI(:Ed,<*>.VO,VATA=ӧ+4/z&Ks0Cc ]bllVoQ J1NKy Hҟ^^^ŒVS%昕UUy)}ƍe7v9'0zWb̻b [GW_wYh̕(gslx`1 Rb%%%W:3h0jFJ?c[;tI3|<^ر㪮>kwܺիWFII ڴŢc:+B4EReO>ڹkjj^.ӻO+ 930Mq30dTrfTQbI>/}/e-s6n2_Ʋee:T|D?1/$Ejn$kżYk͟ÏU7|={(f`:k @/0ymxPq^3gṜjo=oc>qmCB=kMLצUVV=¯wʪ*}UMv֥)_F$Q t.;={*g%OzԄc-jT{zG]vI˂;QY)]F54gKЂ?~[}&:sjVZtgW1ж٣.V/O曫4fHM}+9 ݸ[J'N/F[GWgn= k%JUTsƨ6 Ѻ7B *n3f>:>RݹuV`f4llIwQL S.b5cs!1c)Əѡ%JL;gj(P~7췢⌺[h9z)}z /ޡIzUxXe.kT[B[FH2|}# XR566FXGb.#bMt'cNkC(22gJ'ʛԏld,mCnkq:q w(>.|ܩO-Vll[v#@Tr6n{י@g,)ЌϨplNw8[zQcըnVLL,TllƍFjlaFcIλMևrn\:[~U t6gg;Id K~:>F`qA5#g;v˞. l;7omR0P-d%ǜ]5$q9CG3XEJ](-%uFָ̟#jX9l+_ 7iǎ"Q#7W~ ן^^^w*GЩlb]&a'W$4Jeۇ qFx@a7 \eUΞ@r?% TPC[-1ZPCh3h\/ΖupRJӒg7䰻}YNsֱ\ll^|53xqmlא9UAz} |/Ѣ*TT|'//^fN:T-==>pϏ mi\Rtm=V3r7@לvn;q_cm/G[GۅZXRj tXuUNYcK=U،]&!ʪ*ʪ*%YC=ڿ]@+_n\3,Izg-5td Yr쪚5d\ml h乳4a([ThHOݔ0DUg5y}vϫj禲ZvN׌qvlu.V]yBz7;MXHݿt kvc<tBXFe;v%P=Q)KkxR[ӌeر^Pm{)y,+f`f "&ϝ}EF@*42.mv;PrNe0BcSR/]Spׄc4jDY Ψf;oci m2jj:,Yo UO;gf?@ ~nP/a[:.n{^A.csiRqOJT%l2Ue,Ң4jD^|5`1QN^OӜm]ؑ}}:]Jo#? %ORKgQ,E 3 ɸosfޭW_7+3rİ?@gvFws~9MW+$ɷ[{.|[FWmmݺռM:s#@>k:;SuΣ;FCUжƍhUE@ﶞmS"@:%tNK[O.5ݿh>~ mz^ZO{\8#cρC.\ׯ6w].s_~uom<ە>l,.*:؁g]w% hmTtJ:@ԥ'vx,VM&E"#ګftgu:=Qs:pX P&wJ;w!@IN&:ϥzhhNԜorxN7j: Ѕ5ݺ[o9X|es?]*9?ָl=J 6K^e*]6Q_| m~u?U?egϺlsdλ|\wu6Pb&mw Gtkmזft\tvK: g)W@[+Cr6nRB8$.RT02g;6*fRR完YoeJHcZe'\_t{}[CRh|KN%2%+,厲Meu ]إ}[Q ;g"FhaZ-]XIRWYU_}]OWhxR$ 0~$k0\svWHHOIRd*> G}իW{!=TQqFǎױcPr$=>0dG9~e`mJ/mz6C]qGUuo;3I==OYΪEOUGEh뷷k]@J}z{s&/~lK@ۼsgPʛdwP,4Xgt״Izouv.k9vhǹ6QӯMfw&iPߚy/o꽯}խ^y||Le4̖3g*C0 =+"^}Y=1k$ۡ#P3Cqf6 [oe.U.9 4P|BC͍;J?]_yM)=mh.BoD铏o]eUf|XCG+B ~{qZ|%I!ZOV|\,9.y/$xCz@9qꌮ$/ݿfCym܎ܙ;.q)s{{NE~5q舶njLm9x>Sw?s\ThTtJ:@D)tJRxy[OUЁ\?pxGos [ݸߢ)ޱғ2YoxymooഭMg |tO"@l/@cy]s9}.Ga!=ێt kvc<]:*tAtqCyOәqgrLޢ#R^uܼztNS"@:%tNS"@蔺3/?O\gzb:S^.}w;m~Mw_OL-e\ mzx̅?l3Z.c%r檞.\r8SC{YUw_kxz(cJnN!t*th6xG.B=9 y$@w0E-u =n_ޣ5>3h\O @y/^ӲެEOLРF kܷւJZ:,-Ҳ te0=0^~n9su*'qբct`jj78.P^: {*Ҭcw@ۿ Vr弙H5 5qp6zuJީqh́;>~303C&4@E ZByS2T`+S+iV8kڦE"bjcWh-xx"K4F2Ќ ĵcV$~zֳ54iݪjjo:oEv}SlaNKR8~!͎~H+ϟ sw=+**(o~.)-1qzpdlqcguuKP``:7&QYf7^nUVf]Bnk9?8>wrLp`>-ǔ/l9J1Ϝss!sCz[Ͻe_g}\}X.1nQ]c}kchۇk'Ӏv$Z\xDC1el˔,;Uy)RoO jWT[w%@o_-',wB :Gobct|7]aan&I|?ĥ:t`أ&W?;G4JNY2)R[W#s$Eԇ;B kaE$x{+-u522B9Y>mbcB EZ#G+,,T;Mi=a0!hwV',Իy7v櫪{%v:d"itHsg9}N<}}$S֥~V6F:qn'$Y W*zo^h5m0[|\Uj-VrzU*--Suu渻[n4gTOXP ӖB E;B~LDUTR³+e7wU #{5*R)ɉZ  U켹JY%?_.;yKp>ͮ*zÊ7)> hw2ŗf?3ج~/wӕ u^[f)6.A N`[Ĭwe58WНc;[:iuelq fѨ/R{q[TJQQF g;8(@>>W I+1 IMʸIRچͶ:ePmMl_p:_΁Ӿ;/ʮNc?^Qn5za4Re(E4CFCQƶL"i7))qsrUUU:G/$PUU**NsFԇ;aETQQUHHgaanRh߾nB yn CoO P``]fev 6%v،5¼W;MiaM2QIK7\ԨH|7]A*,<5JJ\С{tqc^wŝNJevm:,_973߸0i[nא5xhY^aazAynYDbcHm[7$I ;_25…[n6uM:u Q?yD߾[ߢ]KTuZUgu`/ʼ4Q?_Ґ<vdZհt[rz3$5 nxXʺTs%ڷ :ըfǀ0IMUF jRQqJUUZDCy6r.g;~ꔉfP(5ۀQͶe%ojՋ/KꗴevIRS0~B RsK\RS%¨S͵VTTJJ\ms;UUQQilݳ;os A41cDG%>oϖit%j|3CsVyE%f;[/$0äҕW|rkÏXq1lIҳ񼤦cEfh=+IMtP -u -\2[k|1}?zUQ~#:/iV2KF5,YYv!,GF LWsUTTNڷd[2sWɒzm9B%] ;cVۛeQ\l]Qv###5%F#<.\&5U|WEEꗔ徣F*mf0ivm7Zvnٺ]69QG*,hT5!AΛv-]3UTT!ASl\ηLuu ZygN_ӷjz7c2eOkڊakkn/$رQKhѵpeꔉv/#P?GNJ*)<GFF(%9g9|;E?_._NJtXWϜ|eewJL͜~0ܹ<׊Jʢ6:IѳtAgtAIh@ 6{hynNCo[{C1t@ #?UUUن֌}dd-wYp߄qW<Jpox{̌^ߤ ͠Zss Ym7v*s|ڙ8'56nJא[.4*<2 Ra*pvmTn6wkl( $57{a*Xv_}o:$tCzhrfw#j=OKcy{{Xiz^n!ګf׮qX= mv ι ҹusz_p+C|Et?bv[ڿ[⚿j_ TtH:@D!tH:@D!u ں c|=n<-Q^嘠X-Ƴܳ[3>mvm>7h%m܅Kc+,\嘁oSqrL`o?1'OsGp1-R1#&j+|l9f{nJ5Q[@m]W|rIRIi&=v̐ac~?aEjYةWƶL~KEmC檤#NS֥ڝOʺT 6F2ݞk*)-wyN%)>aWm]kC|r]s[()-C3c[kpFV󻺮|QcjwVk7d&e]Ӻr5vxm\砹3'>ajJIiSQQ I+4uDKUچZmzں:%(go$)jTRݪJҖEE&+:TxDk%IO%'??nW``)ISL)%5/w$//O-_"Iz>oHv5',׾>;_|ys%IF*mf=V45S (?Rm]+9{ѓӧMl۬zyukÏٽO}RpPO%hmrBBŗUTT1q%)9eWbc4:j]g<=e]޺]s!ImJK]c#.6<+Nۖ۝7N6YIK]\l>@@SZV#GtVU)$8: 7@ᘛ0sRSk? I  #gˈCHYj͎4_EEY=W!!ؖҵWŗ5}dM2Qôo6Y3g< ~%//O j?J|n婨SUUumj  UچZ8^aa|? 9:u&=y?  p '{aWMm@;@EEvmjllTfmvＩ媨TJrr0~]:\N9kg#e]8^ik GOQϸs60ȸ\9{4n1|jAgx@iw&|C ze%| }sU+]͵4iT3a~h6}j2 }wd=ϻ z:tڝ .:<'I>ޭZ[ӡ#*,<:ej q9IZ2I}U/aNկ~NXV*++7nM6Y,xʬ"'ITW.Gaڝfe9oo}k$5Ukڢ>avߚ3ώڅ[U0녤cے֕So}sl{#X]IWYy]8x{G8kTUt cvs=wgFhӱ5dq}VU|?k$UUUY3uIW8JÛ:&tc(ETl9B޻ m^+7q1KTTT?~iM5}k@Eadl˔$ΛkWQ.f+UdeUQQں:klj*m9r eUTRIi޺]BoWiiَ9}C4bPl\*&,eW?}_bJ|P%>uu ůVLҡ#7vtZh:VuK{}jn~kWj +.6*نg1ﯻ{ IDAT`[/,,T+UM<9>?mi,rlooݮ3PiiTUUm8FE}R(7/#@#Z|O꾰X.Yi8rLΖCt9y<Ͻ嘺jc _liA&Y3;+nۖs˨fIɌr0~Su>TϯYM_Ruu&'j㦷z+N _VoNNJ]FpɶNĵ1Lu߄q:zHK'5`@+RUUS3w+뒤>{i;o:mt3ZѦmج?Z ٛ6eTBsg_<$affyܧ~CgfSV,)-׆OY}/,,T3g<`W/К~ϯ$-"_kժ3[N###{ We+Z6of;Vd9G?na=w]`9J1=}RxX',ouU"-ǜ:]c9kпTI^=ѽGw1G >Ss#gY=b9fchZ[} cWL]hY3춏вy߄q)I.+] WRR%ڲu<zfS*+Pm]r>SUU 7O~^~/*+_Vyl*lI{sUWר!JJùqZ0ǚ:eJKT]]#//O$'*8(@ĩܜRQ"5zTS;*** /OJ\5NXQ| !8P_Jjy; YYٛqcG7[J2[ޝ7ֹ#骪`sjy-4Ё=:t`vNL{}rijƪXGW7"ujk4bPmج<B5g󫔳7|%U(^e啭V7*:hS&6[S^mǵmJsq 6mٺ 2ee(-uݾt:zȬd=EP+?m+SL3gs_F͉k2ej3pa>mFFF(˰7BҊfCMHbc7$\i*2r5<,ԩRZvN$itH3+(0@aazfSmϬ']f\#zΛkm9HM-_hw+2.6FcO6٩j#d5*RcF^aa|?KΚ kVrܵZ\=wq1:VTlV8a,sLcA=%~:f<o>C_Q%e7غ/^UQΝչsguId²9-(!~ Ԅ zkmµmEL[µs jvnZ ݺXtݬْ%ڹ oر &cJw(Mp ;V7X=tڝ@e+C"@:$tC"@萺p-YKKi+_Wrohoj|?W)7f/ WL[}F~FFx,:*&:ە]US[Ϩ{zͭ'lPnn/(߷KbIV?ܮ}[v華VK cTZѨܮ;<+R/q1|=_PXTh_ӮJ~93y~A?٠'gk^*TUyjomx{{+-uNc3 Rߐ )RXq1nCzk^L l3$ݻ2紽TyH}:!ؼ?{+OWk{o;v Ҏ*s{qY3Sԣ{͙OOHyi٪U\p{|綾֥$oH?e.׸^i;aw/-t\WէWwYGOzsK8Y}`slї;<]6.c:W?RRSXk雑u9qW6va\݌9VzsYe\MzPt _K;+\Ґac\n߲uzt%|VP޼fJRXX^[ysqyڲuݮV}E56~atd e}IC\lΜmγs5BS'4=Wsf3CgN$TCW(o'x|ϯ9=h<>=\#Iɡ}ޞ ֘GT_7q^]ힳ Ci\9 ])+u{ltzKMJtyN+֓^?y\|zsm7\Ԏ]4޹b+F*WlOuOO1^~Vk̑96: ;ʮԎ]]S߬9hkG~Z毡}UpQWZ4>HGZg|RYi,;o2e[a˳Y-߷hڐ~Bwm%:#Ё=vZv:7sܼ|m$)(0i|ƶL-_t썛޻@g{->^ .hTAт'9ӑm%}ѣzzz8ÆU@s據LƷϲHjxDl0|=}M/3=2~k$ ]iyj#k6vwx9]Z]U ''YOZ@D1tn%en>op i_^]|M{o/;'*[l:+urRj {JU5wkkG4vn9pۃo =}YsiU9Vjݹ+W'g˳DzrvꮕɟjQ Jڂ^?Ul+K$34d@%>D\|~D3H7uDM2QRSnt$'Zl?5خ }v*/Ï ΋`Q)wwUlꯀ.˳V}cr*kMDۯ{=W-oK ]NJ]ٕ;PUK#л2΁>ݯ5uzz8P3vi`UV?:@#G4[ a{C^ݯ}mk NiiudkzѩV.ǺzZ׺?k}XbڏI2/\K-=@{Ռs<ۚc ]M;WR痴NU/}gA ]|z~S1erTpQou©u1VQk΢~5NlIK̿$|Z3 Y%T]V3+UTRIir>SsK\UU$)go&?S} czzs'<Va;þfly{N9rS;P-/ϮɜZ@mdKڕ]i>szEyS%8ǖ935`T3*ܵp<}Non= .V3*͋w2Ku[.:[uA!=Ty.RNrjܟFV۵,UCj/iWvb~kbNHPocp]6wg[:Wl54wloS1v*]|л>RM=ztfӲUv-YeMsضpY5}7v=2$I=qe`O#=2SN?_-[4eh`_Pk+rJ(!APʺT s IMU+25өkIi%-]3t 0kwSg遉wvW>=\ f%9Wc˳Bz5ӫz謀ݝec8*йRpQ_=ݯ|֌݆uOOko_Ӛ>w ֩3'MAy[5똑hyE=zԛ5 U+9EC{)_^]8.\pІDؼ< eqrG˧=f/f1k$]oLSo~fokj Е֝1[~!y:~ T螻Z7\4+jj0vb)땜^RS WI%I-qga-w\jhg\+_PQQQ"]~%;Y֭!I[G茀تCJ/YfV ]UnB=iĝ~zl+9u]:RSE~^gOm!ſ;_ͿSom/ш;4t kk)5P/yyv5O8P6^fAaalYeN6-Xj;v+cg|=jiH_mJ~v zk{JUUs 0vR8~:V\P/sbQW-\#"iՋ/i7$ȩꜫvҕ t^~%M^z#m tk70[~c8[sfSsCPonZB[qYAPogi٢jlBVhn3t2ۊu{ ԣ{gz:zٕ]nV4jQ̠f+I2<3 ;Pe:6gG.f]ceޏϪ>?wIouBFwg~3fLkuueT47ڞqF?^pGs{p-7Qx :sN=] 3߬wF9y~7+1LI/>c`Eկ־ՖUz|>a>=\կփo48ߺ r !CjsE1. <-@g0ڞʮk6̛:eH:t`YƶLmܔDx{kO%?Wةv9-?MN{Duuou 'îhҒڕ]{-]U(sd`?_=va >Z~Vkm#Btť*U՜jx KFzwׂ'ϛ4cV_3xkB>3ZʀfBs6>>ښYj7m+׏ Q{to Tiy`9Ưg|ɱΜnv7h%ҙӧ,r̹ ,tҢ5Yc[9ܢ5(3bP1~ݭ\d}}s?o9 tC+vBG…ãSHpuZJhw-҈wǻUiO t&t^hw.\$xjگ^Ňouf/C"@萺ZNc}Re9rgyt吰~ޖcza},nfv[q3rL^>c.ԟsI17X=v/.s.ue1-QXc9Қj1~>-0嘆s_XW嘳gz[=C"@p IY!(e]1uu;_s競齫 ]'c[25vrURZIC荴uӈC7$躜m()-1q***vB~.ڸ-j@xNՄgX[n7|F n㦷4d@}Cߔ?&+jTOC(.6FlqFFF\\h͐y֖ۯ[f/ETJr|-S;_9{$IiJJ\ܼ|=jI I+4uDel%>m$ivyyyꍴua瘛Y2xƎRNVkwV&'jdd9Nٛ~2n`~'fdHN`xpz)։04s) 4cD3y씏t111 EMZ0؂N9c ݯ^׺uZ\ٳ< FDkY"[ %IˊJM[6~g gJ xsp0N8.rAp85`E+YJJLTn&3*ƭ._S\ǎW^ 1ε54k̀\]]O[njjjV cLU ?_ZZ/[{I?DC"U S-8s?Ok<pŹ96F~':u0nDu㵝7JfhuJ7v\̻ڛf;;tݚ땒luEGE{v䚧1WU!bѠA=x5ۺ7s,EE p9> s51) ӇsgTKOobʜ2]/Zm4h=MxIs ?Zq͊lܖ'=v˞|keݣAz쓝3#tVۦ4n|7f ~q?SfݢHUTV+.~Jʻ}eJN:UTVtLԢOzkm=|kyTy33T˗iРZHUĵTvl Jejhh](nU@w#@5h@rgڌ!)1Aw](DI*mK VsIY&RAN:X;wP퇵u;#ݣ^PB}˞ 59#]!o^[%_KFRbjk*.h .KvSyVl+)]U﹇ qÿ%IZ]R"ݸ沴GΓzkuZZYf?Sqo6+-_VdrD?x|Hh3qBh JJL[ۃXM*zW#_P`+8(H Z]gFں_kxov̘1]:~$cuйߩٝmHw@sU\Ζv} tGӽ=U:Е~rsW;~|NB]}{].W>ro_'u]s4?@z{Эz9?ʾ(\Rph:|𵴵u_n󳽽}Ͽo|y*("(kgm}}p !(t.W.Q[Jy;8^mHw|==z:@@"@H ~3g\ۃXIYBz{@9QpP}((kwUidE];/w8Zӥ t<5ٛy7):jHVs֘Qt=ZU[;n76ٛUy}oHI<[o[ g=#Γu3VMCժ77s ҙgozlkS攇f~}]}U>ۋ歷>G_[6}C:p!}~a]jwq5רd>PW J'N|c:vxgNUۦ/wq Q۪/_}AS~t}=tz\l{C_WG璿lՏ|7|]Jʵ?ߥ[ ܧ~H_|vo/:T߶^w@=ݭ5ٛ5}Ff^*)VRbBo ~TTVkn~bbU%TP--GܳZvjw۴uJ}ݶw-ˊ4qB$^bhg/t_,I8|k_I*mw&XcyV;v\˗iuJ9NY5-+gn{muz,Ϫ ͱy3ZZRǎ7Ǿ~k>cP=1W4f~nkkS^[v@OdCpp{g_x_m7\x|md2\p9FΜ9N8jll Ӯެsku/ڵۦ۵n&{ZZxIz%ߑ+Ͷ:xArVU/-0CS\iYȺHO<> [ :6#a'Ƥ2Cө_ݧ0EI PiYfkUۦ6/07z{ \ QLLKށw#Gy|鉹Nx͞3_ti9W Nc_}?n:;{3+WGLRddǶ6+66FI >bMMRǦhUnU7xt%W3>:*RSzkuϝjUo{ϝ:<`?٧EOn?,KHXeݣO,"/)x~YAcSTjc7BnC"õgW3\myo?a4#tiz-9N.FGΓwUX,z<*t~X Uǟ77t9p8{b43Da$3\0ٸihrF3zX--GUZVnSIU~5FP&bQD`p2B5W\!^kɧui]_?2+:Nվo3+ߋMfOO{Le嚖in_ɾқڽQtTjmmUTm}g݊A0$sg墢ZmZI*%ߙ)w|=LED /a5jz| Hsշ=CUGe.7]-^[S&*.~S&jNS9UQYm+-+W:Zъq3Eݣ_F騯&{&M5g̖;cd%mT5ٛmV{rgc-0|~~܏6iۻ'sRIeejk:-^WسJwEO+:*RknPkkɃ˝ˊ~&gZ]&KHb4qB54*ZjM+zrgQJJ4S٬89O}ٳ̐߻jTQy~cnk챽j~ >} \2_}?:̯?P$תP\hjh4Rs <Êw}׾oS:Ko݋F9m|`^/_zr,]hϮ*t-^߰Yhj"sռm[+""\CDjۚ8!UI ^9)iG0EeOEe54jZ=:[oڽza>\;ɾ ;5i$%>u%_mݛ9ܿ%\劍%N/YO݋55jmm7KrUes:>gRIJ&{j߷z~%<*L#Iݦ-ZW>*m:̷gyRva}Y pI,Z$别WI,%ߑ~ /g$`X$k~g}kHdGiYhI 5ٛvƤY>`]Pp3`PWW6Ѯ""(e3_W~MjV3Umi9㎎:i{muzM\Taw MHM8N"IRkI=yvIG!D{t8^.;w؛3UۦjY\ _#G1&%&葜e]PҒb._wH+chm=hu:/-)$IQQJ<}OU5&kdN4z .'N۴wkj:iࠠN j@m:vSLmdC%Ǐ:ms ~?JlEcg[3cJ7FSk|<=Og{1bQ%ߑ0EMf-)^9X&v-J}~aaF~n5h@;v\N;:*|}ogFCCGH}ߚZED+/w$7ݨm5$WlW":GKWh劥湸%\BKjh&V񨨗|gϒ|] x3"]SN{vU2tCSˊeښ mܴcdCI|O~G+PR[nLij4eUTV+.~^X ݗ \A]m雒?}Jk4-+ӬO$s?#Q3=W|U$#0"I!4YAXa1C fiYaLiYԇdou!6+66Ƭ>}/6ٛM[̥C V{Kթ]Uoۡw*iTEGEj>زEr/s60YYrmzMH?,vY}F,)1Aw$WY09#] *zeQi.(Ҏڥ䔉.xF7DGi>s޳ʬTZun;<ϝ#%N ?;U>@C_tt&{f??t7\˩-+^:xAa""UwBƶ5kכ_ahKrbccf855:|ujjj}֬ݠEGEjTr%[InJ#QyV9NN~|iKppz^MMjU}}q?]1XK ۪SMH״LL3ҵgxÆ}S7m&Lќُ(/wvfR~,ZZꇓra9z[ym"#jp:]IeŋTZVm>fIRhH^|N }NViY˞ݯݢd^iz\ڒ?܌p`ioHvJP]4/-G,-+׎].̢g7x6mkիo,驹R}jr:Oo}ОҲr8|ΙiY}zyrI=s//ޙ?D--GTZRZ_~$|WTV0!!^59iwq)k\򐊗,4n>Ⳕ!RI JJL0q;,sY&{s޳%%&u*|y͛Gykk*~!m;I fBrН$yӕc%%&no-԰sDE QkkVY,ji9(IJ4U#n-$W?]iYU;%>' !B/x6ٛ5}F59#g 1wsmi9[v}boy;#ݼne> SsJsߋ7WiϮ*IRvl¦f6TTHr=[z0'O s"bqq IDATD}~9Ns9VNrm3 5ww=kDA{lt aȗ'/8X  濮rM7 tu9iﵿ~k;f̘.Jo0pN^z'*b^[ @ΨZ^/ZhVAjhNYxaߠ+v*^Rzţ=tW?Mi7iruaaij:iXGn;9uTmv6n C:me;2hN$m~o7k>6Õz/fҲrnӖ @{%ю;F>󷭽vhN81tt+āp;v\|YQ2\׿MgϞ5#@1t::Jp%:uw%\g4p`;rEWӉ.W)((H>wQt*EufwVK﹕'c@BWJK]ѓ˳vEWg.,'+Mpo@K`N)6iʷ+.~<^Se JHտ$wVR3KNV|Eƍh;CrtQuu!MNrm$mܴE3T۶[ ;,F4,fjM,._&{ T@݊TiY$iTrZ[O*%%7 @ pWn2L~X?w,C ==L:Tn7Fe/>5k77ݨ|k$ɿ+;g**}rgHY ttoUhhBCC4dHyTߴE<*I VK Jkt>8g@ @@7:t'HN 4-+SQfH=sV|ES4,fh@ֿy8Ţ+8!U'*pUTV+.~JJW)vXC ~fa/2\iY>͛hO C:zQUm~[Q=:@@"@Ho@qUoՖ~}{槭Ϋv u$gcA6>^{5P@5=&{EǶ6i~}H뗝oH. gk$)8(HUW PPPP/ #@Kb_0@117꺰^x p%r8ʷ|IdoֽY9k}ެK>ﯫ\e;N:??'0:B8Nzߺ|}.Bꔜ2Qiެ|k**/xViY{=sq8Ni)]KX8NeV\hUTVk?c1쀲sf[iYG;Ҳ./]_Fndo֏>(sn g:upE}Tŏ5ٛ6iGs<92BQ55%NI !.r8\U#+)1 By>MfM?Teiks/xV׽on?3Ҳr3;Еo-Tvl7k*)]䔉).~FJM[4n|9kXP9_9Ɛo-M[TRsjÑ4-+Sw*%%Yk֮W29eb9sǡFߩY9~wEo\_%\FSEhnf>/zo%%&pN9&M*-{U>!:s/MgϞ5_˼֐͙@e/G2]va]mUUTTd>PK^/,QtT"۵WJJLR;Ξ=3g3k.fKQtT**5[T'tZ\pmۺ^_Y/xd_'>B?_Z0|=XZVUZH'zkN^}KdIrg(,>$pW'jL}rg(wV54a͊QRbji9%|GV4VTV1>'өoҴ{d]P'i%ZԱ)kcyVI};=p:4f()1A5oUnjw47|?Dk|Uo9+g\gtZq{%`.UNIE$Y,=^.ԉBդ3nig՛}cil6ͷ_T}>}>ngϞթӾUvqoP4IҎ]܇C!'|)ڳקNt^iuօ7iwwt__~wv&{lд{$I5hXPIX-q U=7Shxsf[JUM{vU1sEe֬]ҒbY,r8ͳvMqIRLP@ $ g<6nRRbV3ʷ~}h9r7VJJYy*x>;.Ghr34:yJJWy=N֬ݠiwI\zjc}k.~ ײEѻvx,Xn|ߩښ.-ZZV#ڳJwj"ҋwj_N-^] 473s >""UmIҊh״~m]Wsվh+Qy}Nk[6--G6nڢQi}$׳?+44l?7@qsqu_,9O{Z=QKkTmK6ٛGhn~JS=x77;'s1[W~&g+/wԞ]U2JQQC<+|yMQx ^y,3'/@_CK`'59#]--G3'f(D3pᄄx%wGIJII֐p_P_uuQEhZ=:xAMfݫ1)ڭf] ! #t1n|GkL}ɧji9*I!:JLjwP͙9Vxn>M}'D89#$9' '8_Kufr:O*!į0oSEZӥ-U:ެ5 IS5D?<hU)%W07.~T:3>q9NqcTsZv~ۍfɿ+;g**;}ϱQI\WT}ٳfz%LTǟjrF4Be$ݓ'3\pڞ]U7Qs[Ee;vOH9&-_V#Gje6o/uH#9+;+J t#G)mT5ٛuQU>9N@%T x>OC ~M%O(44Tٖ;+GwPCG{.g_4tGp=^պ-:*RUҢ9e0?!߿E-]έYARpwV[˴~ƙKq3rD~fmܴE!!ך&NH'M#Xu͊g+WV! ѐ!7Q~3rxT߰cZԲEzkueeB~ t=I [U;Uۦ_(Sڤ9*M/x νe5n3˻^[v)oJ)UoѴLsIG\Caݢm[kmhK6z-MH eXݲE:hzEFMf[ JfY~Чo6]VH3uP,EDz{U2/5~f}d h qjjjuYY_F嶚Zs5k7(66FQz3$V?xOJ#QyV9NN~|ilk WyeXPվo44a{kSRb^*)2ב |}]ppK8˟w%6ðRU[SoVvl;\JNҽY9j7ެ{rVgVN3eŋ~RǍҘO+CW7xӲ2=EGEꑜb+q91ݗK^mݴLL3ҵg tn7q`͙rghgNZV /Yإ9')$ZIcysUP3f~}ò(,5Çzϛ~[̰a3kZVbŘa’U*)]JNhV(ͷtHM˺GSرQK/CŢ-Um'%J},Ϫъ#<骶hͳj䨴sm3eWݷ kJ<mEO<H0\UyTe2_ZRQi/wV54j4-_VߠARIGXBCuJK%sݦ-9*M ԏ33$~mͳvM|Ysϵ^!C"_h:տ_?:}Km]DѺRWW/%DÇa|k>f>|mNr󬒤K-)^+ED`=$3O^ IS3CSEWm?qBKv]IeŋTZVm Qًi~Oz8*-+}ٳ45[4l+mT磷;͹HWZoמ@j*<|?)"ͳY~sbhTٸI=s}*~ @7KJL;T7v[ T[SwB ]~!m;I fHDrUru{5.EO=Lk{;}a:~NN>g<߿5$)HAAr6={VgϞՙ3mmm>˂5>sݱګVzxtk2cV,-+װZVH{mu):v6Z鮥n1Zsf?UTVkzI Z`g_ϗ_b4/wpXEejhw.f}9PáF!jmmѪ0E--G}e3Iijm_dn?,IRM ~TӲ23T^}Sw&*,,,do +^nwMfMFMHٿBCCݟQȑ˳J2)f0EVox^}UО={.ƺi:|TC0^s`C@{w<>O[W&8l_YorQfϙ9~+']ΞmөSti˿t/:mӯ_OiLmw3gtY}k OW ];=ֱ':msԩNۤi[xctmiڤg[SN:κ;#jgҲrnD:ji9 \</RoW+ohsV3n~66z˻vrkǽp#@py!@wyH| }ɕFXiQ]Z 7$pymmRPЅȅuzb B>λ`X_܏ [+ $tDR}՗CC:Fg=p+ $tD:@@"@H $tD:@@"@H $tD:@@"@H $tD:@@"@H $tD:@@"@H $tDo^ۚ4̬MsNv Rd`#⁊<IDATx E*AQ fJalO<["$զ{B \Wzu|@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@`(ѿƕo}'n޸1 b8y8cC? :r<qƍ 0J9[o@t$I@@t$I@@t$I@@t=\x㎜z0|J5<칸p""Sq8ϯRSw'Xqw⍿^+[[@tС׿8rg|#O&""Npzu\Dޛ&__":~RZ p&ZٺZ+[rL`Kk@f&l7|QeF sg] :(miƠARZo-#lfI@0skilQד^9(6ۚ`t3h9Hj`,ptWg\L`v`Uq%22!@j2@l:kb F+{#lo`׸8 %k߼׷v_[k A;56`@g}6 ּ;yZq͛W `*׺yaoH`&":fgoo/^xk>0w ى[nmVy_F\']oŭ7kC5ʊBͯv6'mP:Q. E}:[OjQWEok⼝}^cmnVzn5E5sm֬c'^ eSyNy#([^3'H:S6Λ&90]:¸m)8oqŢnuL?WZL[܁h.-oWu!6SED+]vALy &Xn4qJiu'Uy. %.&m!:ٹ_ t6i8Yȕ`rM>XۄtKyg4 Њ5 20rLRWV"-? _xMeyTi[#N˸fO<0;:eEiۺOYDQm]E1==0mRj8{\tUUm"z!]w-EzA]-9I@0}LkEN[8 c͇- }ńl[.k9qLZ]-_*銎ds`֪dOt* Fn#<.r\u\FiSFikO|ms`r?/ " Lʓ/R&M@0"ͧUv."#v "ckg1u`ޚDyV/+K"")t]DvV-|uU8DT +ҍF]ٽ"tO945qeeN."UBO:qB;i%kUCT9 P)tm#*ϰY9خHDImU'!3}`t#5."qǖm 1Yʱ9 FjVp-QUessֳX`CTWu&ptn]yDϲԉ6_ns&67֕k?˝@@j]Dnql yU6EvWL_8L`"M-W-[⵭5 @ښYL`B"N-;ahuК>|L`bڼ6u*7nq߆w,H׺6: H@0QE 閺l8Ded7kPkYf%\_AdU nЭvvD?ݺs&n#B}W,Sm{Y7L`wt]tḃ`FF0Fm9yv@9Atk1v091`{ks>7&I@0S aBMu6gZ!p`t37  O8@" "HDs)$lsPP@ s, X1E@@&1S'RˍG!S *JXhB@@#U6(EvtA@_t: :(06C? A@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@t$I@@Q%مIENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/obs_prepare.png000066400000000000000000001574541476650141400254460ustar00rootroot00000000000000PNG  IHDR<T) IDATxy|9YX@6IP!MpPA*3*`iB Ax&HaQq  X!J4la ,INN }}]} L={4EM"u<Cxx^Ƿ7P_:*G/*ձ7zKy]묁Ȏ k'kqRYm^+ԁsyubASo40SϞ=DMMzBd؃ FEG]%CEm)9&'HkO]ӎ˅Mx|6z_P m;r^.W?JÃ[IFE-Sy:q6?hhx7@^/[k֓!շ b 0 )ЮkE@m Te䀮z}djקdr|XK_[[ oVѡX;.`AnZ_[jVh:>-.ZxO/b-yTrlYAOآ9-oTj]qfNܴέ5CK/ō(x\ް6T6dTvrw2r 묫ʷTݡc׵"_u ̐*"Kc<hD@E6nf[),Po VR,%IT_Տb\(aםZ7#FQk@ ]~SٓU帻5+(O^K׵\Ӫڪo몋WlzSY׬a~(w)R9NQJVR49 <=,۟P^qifNǯ^П.PA^'~b>U˅2i__[5"i/V9[l_[85[ws`x < Th \*P*ǽحBeYWo1*]Ϸu*OCRuˊooZ]E|zU #1}&VT}Yʚ~lB$,QV>Ŕ@qӂZ aM@nϛڭ'*Zz$_L|@KUΕo5|kRhmV-.vmu*DC\5D>-W~ņHRT Hڧ%a5^Y)Ye:jzbJU_&>!ZO{V1*}c;3Yьת]gga<&md-*Ձsy.QkU'j\.vWn*Ȣ}51OK/jw~sWzk Hef ۦ 6u l]ڜE.2BR#x~NgDr'bSa_?բViѮW#8i[Y(DxQ+_Yo苜swiv6:WP_ӉVPojFpKء-R PGA R?u-܆,FYIm %[[% WJ[MU(*rT(J\5Jh2\D?UV8_Ng#aw|xL(*QwSލbM<~EsX -|Sotj_kҾHEY٥Uɨ")7C;j^ئH!Qu(TPو ku  DEB*xR& g[#iIHIRԤ,>SxAm wD=QR!S-}$""f~۴DZ֪G|ɰjz 6^+RbWjV!0Ժ&EBb$+KKj3tppGD8+.u`p-{f+/ؗ}v\!Ցl߃3xZ@S\sǯ+1!Q!U jxkkծnU`&NqD-Vo Q6^X*} 3+f럧h yV kRӵn5emS}Kns~wAjɾR5#r>! >Fk%I=wFrsnOP Vm] ԳV/߮Vb7qb6)$]l$i*J!}~ 45>mkxmەWbb3Is͵ D% *.xLY<_*<P:ũO=Jq̳J6xLaI:y]}4]K(RL&aH%C~sRT},Mĕ\Vv!ګw(#1V!Qu:&0ger3^rDs(EϖS?&fEeHP$*1Q:k><؈9" n'kݥBMξqZjlZU ;NsG5>OaPyxGVeFU\謉 Ւ,MZ /G$Ӓ~9xiR&*dL Ҹ}$)SվߦkXh7lkٛYh26M:IY5)K)NQӊ]A(|YkRf$BU g +(=`ȰQ|' bV^/+7%r-k#ަ[HEt{5wLEvTW]󭻑ZY-%OHܕrHEFzxn*>oxMu@zi4&{;"P[Z{A>WV5Uzm2@ *PjX_,j_FIn <|Wl.0[5X衵Igۅ+,)G5Ӿ}ךz)bUZjPK_M:^lUI Ms-}laȪOK ipۘzej;TktĪFigs-}lWXjxE}BUSPbK׬gŪw -:e5|5*O& Ð HEd{}ManHʵBU <8uN=56s|||c6l6t M<[IijVq9,Վ!JcoܬvL׮ՎToT;3䑇|[Վ1:r9U6ճ{%OP(Tu_jnxP+S@zފk'{㊪D5M_ꇶ[>$z^~MvTj)uu'5'-@nua13qU8<pU)GB5F\9%{_Ӟ+}5::Da(/6 _PwՋ۷h`z譮U]/ѡ+{qx)z~8:훮K[ъA^AC闝si~kX"*v=5RC:֎V^ U *Y̊{RJGO+'"B{n;O>>j/_2T;>T`HUDKZ:>[@PᡂIkls *(0y.%̘_ϞVgubn}nmܔ\V$W3ԩS}yOUk*=CԮ][fh78ϕ׽{Wi2uvwP<{W[vq\&.צ=UC/C@JMr|E }\C: ԐL丹Wx=qq[/E T\ʵ8&#~9z?W~ h⎬`2/?o 2w[A.kEc-ZcaN݋G6eM9+Cl!h@]KݪVXYP"tAgueL6کJ[`kbZ7vҭ{O% 2P >L{Jܵ5ugewrC*yڸ)Va}_P``k((0PI.4f *Uڵkr@7."Y<_Aھcڟvlv;zI\Paaz~tI&x%_RN =vDٸ̝v~KqOҞOZL8G*4S/eW혖-ZdՎ٤U~[Vټbl< 7a=ig+((Haa 묍j\*=8,^8ҡ|']h݆?v:k26/?_ӦֱY0IAZ&E!Iz{ζt p/]RqV/_C]bBi;ح^]FǴHEI>Mq;bVLSɁ儮ھ:յM]*UnkӦ;Z 7mS#/J~fd '0 Y, k$0&&ѢjMjvc;H]:4Nhu <Ѭޯ(ï)4?٩=;#= HZe+֨{.r]롁s¥j侍#D.4fHmܴ=[k8T.^upT Ѻݮ"iMPSR8 d?q-Ik'ǔ'I {R^{G@_9g뽵@B'qܹ^FTJk:b*[l5jȄu3u_19&4 w rp'iG4.xTxŋWP`7V߾]ƕH'NwP1񻥫4iX޳WlgQ#px?S>Pq>Ij׮A:~ 򻥫>װX8眻2JBX?٩oZ)I ߻r$޳W!!Yl߱[7%(@Qe3ʷsηnWkü/}ƪZfl߱[% 5!-_}AÞýgpIe4L&|>jٲnSu+ڝ4_U^/Ua$]g7ح~_͘1+ IDATGj{+*ZVzGWh NՏDi7ٳ:uꨀZFo~Rs/A]:v8UKR@-dW rԩS~󻕚0mp!/?_Kְq ;a _xVOVk?ۿZ۾Kx"l_ᡝչSGmt[~ԦMZrZMVk&hiQo͹ ٳ-_O2AӦLЁo4ul4Ux/ĉ$I澬ig;i(ߟ-]Ov{ $Izi֫{*V񼡝C4OjίTS:v#$IݻwկfLG@ Ed:šzʹpE>>>1e6e2ђT;׷ V15k%Q> %jfj0NCm<Cxx^x<0F$~8m4M@uNd2Nؿ9 TY!Թ۹[-5X[^ӋuyE>7OSnauhr.\|ffLJc ÐbQIIC+WhO[V]o)y=Wk˖:s|5z-%Ҵ!uC1 z{A9ڲ5G1^TwË㯬?I_~^$yۢ]!Gp1}3pR> RʻEuVlo/[YiV ԦH%{a׈{*+ZskJE:puйMӮSQO ܄Ą!eU ,i>N;wUxPoM[{Ptfp-}AqʊEmf2 èۭb~3gcsz4¿f8o=C*>As-3[΅+,,TiaX,*))QxhpjKidm=Gg.ݺƒե==yMYXx\;Д<F@sen u<Cxx^:MpN> F-~A4*<ҵ2}Os-W9M |&cݭ}q՚rv"l,viZ2~"x]YswcJqҗK<$eoW6wZ*?AUNZSɚ5U9)-IcVN%k_7Eߦ4&25P;^v.|P*V~E\Z~kQaj ;43JZf{n{AuSҴdB-\3Ӓ4&m,r>]Git-C ^򃛖k&/Wm /UUF]3Eõ=tp~{Ah溮ڔ1cm!!S%iC4seEUথm~IZ1)6SC3{!IjQ\J'8zhX !Pek!I҉l޼sCvJۗ 'Uz)gH@DkL\? k'ҿVRH] *TؗqHSE)q>נ#/%I+x^ aJ/lNwP/&(]K*THKҘ$ Y9WPŻjhqFr4Y̜>QmIQN yTpVx8sC1:5WFD>vPs!IzM[Ƀ/S>8qTrzV?BHm2ɰ$O>')"aؿ;(I VB1j*}rm,Mю 7OsW+\IR|@ |WXӒ4&\r*,†Ux;JUЈm?uyJE6o !PSz=rBI;?B!g冿~*,v鵹3o3Fn%K\{减Iу]J̲e7t 5u ͸{(C4s]cTx@3M);tSe?W{4 [ri]Dlm\wZKoP~%[>0ᅾLޭ"ʝ4c5I_/ّy8$ƺ<%@`m-IfV*␦ip07<͈PV-]n۩m|oՏzڊwhN[ͽtY'ˮ4Eƍ{Ysm'ߣߓh'Νwf vjEJ;в9@BJp H˗*>$WCx8[cڸn61Lg4f|-uSJ?gB|vΦ,Ԙ*NnVEc-Q._A9i?)m2{}ޭh_hϟO$t y7uZjlԔˊxnFOP֮"GKqPTK]BƲA/S>H׸l>g;zu"Vk`jzܞGk>>1e6e2*1 CE%%%  n]6O?WAh<4~jxsSo#^7O\<u<Cx߆am0E_dPkph=lE7Rՠ /j< MWIudgMxN-ƿ?!&РVѳw꿼ztzzg+OZPK;AfiтPN{ҪwAVwzUx(~s5z-Kcwi#1R4w}fЪРOk-3a3[^~]p.AL>^%^>V.lld40 Y,(<4 v WG7̩uAZ7_;GpC!_[Rܢ+G/%]4S÷?).[k;T][eZ~K1AP0 w[.4[}꼲$)9VIf,lld40 Y,(<4 v WC#͕7P[׹c~skmp@Axx ɧr+<~]U`W9|zfC)mm?63J!-ny_R}g(H*7хb,THREЃPmVjeTHzM9Zp-({l9 R"T얻4`0 S~f. `x?\;sfW<1/"""""""""""""""SCۦs۴t<ΫoSûC3:{쒝:5!#sMDDDDDDDDDDDDDDDژp=q۴tH0nҏkRҝOaU[m?! Vz5&i0ҲpLNm~8o3n\aBk=ֶ' IDAT2[@~&Ԗ5R?8nt0錙HMm2w<9qn6:֘u :pzUXIs~Z>Ӳ{\(rqZA7`OHְfs5G!.t lbk9'\{ JYcÞ شV~(vp'GT5>8b0&5ہ#"uC}b]܂=;äg6S⤲]pڪx)ړN^~ w@Nely;?!b7SȚk$TՐ۴0Q &PWҖf;pL)ͮ=>10>3j7,Ƙ2sqdZ:/>+IծtX_G=\nj3}/BpcCӃ-3#% hS⦦MwԴTj2u;. hܫC?moM8(,g[w OrjZ=ń~.֔ 8)o/A.47B[)qC;`w4{(s6h^/6;yם'uhvIƴymZ36x}e7W<: cn {l ۯ*/gc[o"{ i|Tiy,Dμ|+_D gȨK_~~)"8<2ȁ7=/2Fqy(j|h;-N7Դj":KGG֔w:+e[1,_v͖Lo?B鯶aoa$_/'ɖ@㑣7a"8߻ 7454L!7B?b0A+;i1ؾhBg>8k700vX]MQf9cmUt<=2pjPo#6p'qC1y4},8dݶYj˶PC:MߧpSXxD^t[ǽzmKpvlM#NG72K߷GNM nm2'mslq綴]֑B߬St$SkP\Y{2|ƖoˉIJ<FpFyD>|N,uDȘaͽ>adM1t3YQQOiE{.?0׃ـ}\1;37|ݑe.{F*2k֣y|@ QE!z"Pϟ;]cIu'>耑!òEv3<_aS`//.Ox~Yc vik xˈ1"0t5a#u34{@G%(y$)ì)~F*R`K"9,"~ZܛO%OV`aߨln#m Nge}xv:q [`d~B:=Hjq>Lzj i}MipsGϘb6ZM߅J<{~`Ɔ5L /hURӰA-}K,33]ԑ +mYvYD#˺b[*YsI!0?]AS͢lX[dn\@l<`\'|u&AͮZʽ `@ M b )֖1=O7.#D`c#EFDؠˌ "0kMgh$6]gF'bS䵑ChHE-n9=遴TRI¼s}mܩN_ 20$?exdͽ}ý_gvYbS{98$4/X^ 6%%;| 3##mm5܂=v3O-@Kv1hr.k?ޣOhnN۲5xl!e.g M1dsӜz p[}1⻘񁣚gx/=)19䢚Af*K; e[hlYm|c`ti/?|pKpF?' ε}9ƚwӫe $f4 "9R[s iF煷H=]BwRvk8)5m``Ky}Ɣ\\{|/ ڿj2WY:Fr$v9qV>K|#g>|M˻g'uV ʶ*!$,P[f#5;\-q <=//GՑGbic-T I9M.ă}rsQV j{IE˽^޶XY}#ØMR>ʚ`~q`  ^n_œ RC~; 3fRDDDDDDDLp6UqF],,_9W{~H'Z)kݟO=,񧗞K]=@'mḵ,- dܮj(ⰣqyaHggDjvΙM 4M񰛒ЎQ[W>"LunZ13%N\ePwƈ4uRD:kص?s꽩pn5aJ6SO-tj8q.p0wMwGp#h<[6Ey_ 4xBGs<&g*P#HghXyJ@2'CygN{'!_-{kn1Gc`=۪VG{S'R` "tƨV;bc$/pC l1㈍tՇ4>|`SH ^[,a,6{Sp4S(/clccfx0AMm$27Նgc5vϨUdtro5.%| Ӧe:䛄o35Ph! {uk<a~ M${om9z%b\[4)1!^[ {v.5:QY_k̰ܳbazpaӘBMœ ES$FVkCHo o@f-4|+! \ i x8y_^$ m3O;bxj5 Tz ;tMf'(m [؛:I}3S7s E篣Hk;I;\8Wg7F%n&}jWTTHveՖW][_*6X:KZNoGOfni4tF@B8N*F#t5Β)戾H,s:)S%O1OuZ G%-TCJ@Q7ESVzʇWHj,2_<8mrH.\9gf9FP478|0hTj -,Tf d?e|P\asbQO-4<5G'c>HDeևѤO!+?Y/'6S8Qlnۆ=Nu? 7G:ȧR~N% ?1:]R]ԒB]UUtoyr?q,c2<w- T|y&A 5U~;_CE)&9L- H%9kNh~4qɰs|3br FojM}Y2,!bوN!sG6@;UMQuXҜ(vrɸ].EeVNhpθkPƒ-*gM@@A5t7&Ãl Yn Rt&IPIt#/sO+B&mNnb,\[P{u5Ic\KmTF^ 3k# h{udmF{/zggAPygx~?=w[uMlg"|KVp?L8ߪʹwa)1|oǶkP{_蜃f4^JƴynSOoΓ``4ы'^E/T>}bE. WDDDDDDDTq̣UD :2sqdvw f'/?H)t3CHSk\;-̛D{}5%ڶ\ ýnEUZܫh4ZQ+_ ~2ϬeNa;nj׸;y,J8lyo&Ã̱ܫ]QtbkU5&;Z=$?`J gnhQhԞ$5%!{U^06BUa-YdxȘv9?2COO ,Ω$WDDDDDDD<4d_?_#};yOf%kbg3$9."8Of>t \uGh[kd%H] c#RvFs3#kt&ߨk#ぃ\Nbe(-r4cgo| Ꮑ9Ԇ}j30 63[>CyC$OM i 7zkDaO0錉qqeX;O ;5ӱ::v 7;j\8 HQޜsom?mvY"fxۆl)s+ВCRH0VT̼dY:|zdHBD;n6Tmazu TҔi -K{ֶ\ƱS RKC BEm!7.R ?D!ߞde}Ҳ%hl%}{{?-6SԠ)-SXҎb?FRU { F;??sk6sUO}ɀҩ k}M 3xm٘CDDDDDDԩSZf6{gb ^h0+U 3o>c>A켒Rp.lADDDDDDDDDDDDDDDѺ/s@giIpƵm wڊڀ\2J06{̿ѣx%L v]3IA.6-[ +GڲBsW:fó " KHʜ{~k G%]5zwu//yS=z,KH̀KR,?PwזrӚݶș0L?b>oU* ˹6lFp8j?Ü%n/0z/ǎ„oJjj ɽصz(`Soѣx7/ջI!$ ̺C샀+\EϞ;{FKDDDDDDDDDDDDDD,PC ݘD"Lᦳz$L]/LOqP=f ;FL dͿ}3مKgdP/3Úɿ9qcB>灺w EDDDDDDDDDDDDDDډ

z篬xs/-~տqMwb"V8qlWCۇri%1rniyˣ>@?N5;őd Z.,"""""""""""<0.o 0h`?tO?Pw]җ{g]5KMM-GOgqŨM?ז/-5Vm p)܋ˆ fq<,7/ e1FVv]}%I=vU:b(y9<>@oŷSm_ x9)! |ԛ&I sH?LBBW}M[)߸k;?]Cc w|71CL]7k׽Ϯ*弴U/׳g"C c)%GyDDDDDDDDDDDDDDD<ɽO!""""""""""""""rQCfBrz&%r)3 =hq;e[ i￿cV\a xvVWuq)],/>O%7d|qSoӇ7#wf }_-`WEU-N:ulW~ٮ9GNBʜ:uFNUQeky?6uzgk{z#V#ױ{$p;l|9o|lW@DL(]]ʔM vW3Tb &ò$X/ppkC}%M'w0/\9A֥1_پJ''gz?JOb ExV1mZ#*JArߋ/es.\2Qx@gdv{:R$XJw&mX. v!6y0>{.DDDDDDDDDDDDڄD<f?]Ţ6sy)C_l*5,֛; `h@8Vd RŢz67U1;<=)"1f0r-.pVF:gp>`)ӆ"q\}SJ^ge}j) vk)0B}"4ŧd,K=8JJ&Aa5/==/vEDDDDDDDDDD#Rtx&3`vXUF**dat/꿔 lxn0-sxp+a< fӆlEQ8 *G(s)s2(:ÿ< ÃyLT 8<'եd0[z-ޒ@[.@UT 'UC7{55I`cbݱ7YC;튊HGJCG3/eы;`Q+;ìSy3Xs)wWF'ð LD* aLKg{Hyc:k-s?P *(eC2%d`|JŻm1#Sb S S bv@u KYTkoыu)9s):Q{)s53DX5m1E3:n?6j`]_~E nuO7SKI^(ADDDDDDDDDDDV<|ſ͖쒜<ȂYWEDΞPx)s g-e,#`>= ; ` \?LJt)7.4C0/\oU0 EӂXζ`yk;2AxaL泠ZU (liI$N\+ob{w>x3]?H;퀇 W1?o™7[cZr^ϑ;@"r<]`Lo_bFS:o1> Ar&d0{1-EuaJ ̝ bcΠpp$ϙ3(y R^>yNNg0 =k=  (>Gge!S,lLv.e+U al+)Bwk,ĪvPR<$\DԹv'"* ؾu_Rz2 Fex<X<ۼFl:)/#ς, gP8$6'_ɠ >(t9y%@a)zo-YXNo<g-eQ^t{NʥQL-:6&0/[&2ٽKG1۰ x """""""""""m촧9X9S=ʒGZ.co_1%,ZXJ?WhT\Is0Z2\>H3=>m }_Y`m+\K@:>.];>xgtg({F$WG𼷎_5!qo8HDDDDDDDDDDD:NN:u:(7h#99(C4GzvD:CG\\:u )s)9y$}{ V1-/|VZsu^=9'U蓜N5f_W"""""""""""籶¢p)+kU<Ǝ؅6q˟qɐL">;Gn繧\|Ц{X.$ٮE xvsi%\~ٮy.0.L89k ZDDDDDDDDDDDDDDDqN;w *vǰ`riOiq^m^{+cZ(|o?ۙ[9u:uԩ>)ޣ-s5(CU>2/nt@5'>.8:uRԩS466rIR: igxK*OL6٘H&*[_Χ5q)X dy8߶60Cg.ÍB~>c_gWEݺu4K CBB%""""""""""""""M eyi+\܊7W1I!e.K׮].\˜#y`]\Jp]*,r=Y_„^^:: /9ז#|(!H1dS)O5^Xqր 3Ç71IF:\Ϝ5C|?șƳ]ic%wHj#E ]k""ґ&rzЌo `xV !GkqWE׮]aoLRRe/,{cJ.IK1Q~9/¾s5G7ЌpY Gꝿʝw±e֕}#8vǎF;j*֭+WFV:L;Zz,n`0مKlU u^ͧ5G~ۥ[ ߶60Czeg;f/-ן8tI7\KϤDׯ܋E3&k HHcFTƇo3c/СzzLdIA 3[l``o0Nѣxy pCyv부횙!.I+9v쫀 ]ɬ"!\fï3,JDDDDDDDDDDDDDܤ0֗Wg\6l?XSț29KuMހen6dpcNa1>,FDDDDDDDDDDDDDܥ .\<۸gTK_Ut֕oPq3̹:`%C$ @ uu8v2J<ō }}i? >\1W-(AD*DpŬ/4~OF1mmKO-xZosdg{u=o/l.R.;"!틈HYQ'A|=.C<>FνKXF}b,W$x8@Sr]>~e>Qi ʾ~[}FsrwY[\}rdg|ԀQvQƾOMk!r&>hV0j ~LK|X5+AAFV~#mjj_vL?xǙ)Y\ xFYCaoH]p#DDڞ%P_?}f0[om`ꬱgomF;l`a#L x> ,*#8$´o3;6FDx*xoSbh~&w-}f?~Yͤ_z;ӌQ'p%i^K4.s/]`D䛬T_W7/ nDjU7]P|]c hFt/KQAsW=%gmVBX[FRC!RS"-TvHIrNg3W,! yJb߯ 7CٿVXs,f3xejJxVҏo&m;2qFGkn$8طjI!Rv% K^'}. ´_ emotEm+>>}߷׀Arnt'5*H=g9Ys&OoR8PCc޼UᓵWzMx^wqwZUƓ=5B=j,֔\ϯʧz> WN#,jWa'$ TiJI!Ӵr}M|a1O)?fȿXM4ז[!Cl c[f-֊e[|/#8/=E ZKy5Z+<\8Iأ^НT϶L HxVxjm)֨YO쯄O .mVМ?m٨}Rs#oN^c_ڛ V\tZ^=lK{Js~\M@3o_&sht-E5>37Q#kTڨeibBW~۫}NTCEJ<Ir$}Swx=u-zx"Dg{,D˓ErMJ2ZvMK(eOΪ n "\@a.<%WWWEڵsSYY U/h޽[n^]3J2]{o7re'. UnGvN`JwrF/^/qZHeoFPޞ%_*~\=ixy2@?mi@ pm6p:]xK. ԃ?`-;})7ّ$xf=˗khFۻo߾2LM:9Y=Y~&]g-k|∖=;Y.֊l3F*r⫚2~Tnxׁzif}5N.ט?VY@#4:^2 VWItIj.ZU[}Cw'?xιq]ZnM۩jY;s_JG 2Z׺UO>PO4js}ǝ uX>k })^ew*:&ۻQp;y{ gG^hi6_];̹ܔ Z׀kӸCΫȗ* A{ Th㼺v h Z׀k\KzK_Iڤzovjw0ER4P[ӲGyJFhWw474&nJamfǛ{xІM;g;}eH]׹Kt Rݣj֮'X9˿ܴ̅*W]vNc*++UQQrapni\E'9My=1voծK 2HÇ;/:Yo/gֹ/˧qE}iiZF>pmeeضԽN/Oa¸1Zw-PtXqթS%ND?6ROI:uDζ{7/]ꃄ/Myړ&Iz9v>ܴY&? {ﯼ:|{\eesWu>?ݮcq$iF?36_ w/j?|~{{3ӿvXѡ)n2-;[Hn<==$U^$۲7mokU"]Iݽlm%$B C郄L$V塦d-kWYYYܓh ݓT?4ã<$\xJruqڵk4R*//ϻf GK3^KtteTYYB:᭟ZguUtX/4[N8MyړӧPЦxSKA=xwwkTHһKV*1HUm)Pۂ4!I:3|XSCoMn-Zdg(;;G~?!$[uk!j9z.fzNK_k{7mVYU!PMbm$IRa tP}}~:|~8OLh*HXcqؾZZxxt}?[oբwxkK/X5$ ߥA TZl#5Lf-U@[l,)|\ IJҌMx4uIIZ8SZ:9 %vhIRZ|X䭩 #ԡj FPT~}iu~'.&}> YI;iRaI JunC1֎A:k9g%z<[ŧjB@CSZJQ|E{I]xDI)}4M%aҴkn(J}#vr;Xuટ?UZRDsIG$oͫ-ȖMKωvKraJ\ ꤩKhgВxV]pQمթ_w /gv\3 })f%j]l%Eh^~ffi` sistͷJQ%INRV'EH}$w5ǬxU<[嘲 /La .ꕿ5ra}rMIZ㩏SuK-#B?KeG!U6=Gӈk 0.B"y\YkD?1͎PP=^G7v=9^= e.<%WWWEڵsSYY Wsl/VB#u5}i-4OU"/[36{haI9dh:榨 ؜Mͅ$diFI #5cA5nJH҄L/ )ф rH *@@h<>pO.X:upuwAI? Sط̹rxx0;<@a =a5tk<+q\n+eۚV\#SZ@Kr}*<ݤo,},<_6zdzr=)xu9hîKK1k|} |Ai.mV+JR_7a@մ/M [zdvn#6IWVāNhMxpheQ-@iCM,h"MxEuM'jGT[+4~9#6;qXaީ.yѡb>u-m5Ykqz*cXߥOC 嚶^4㲲Y қ/_֥KK9JN@Ks<jWYYYtK>/OzO4׿єAroI2No{Ӕw^%+>d o51\]\v9TEEenY@5q.-[gke 4$]Cb&O}AZ14ѣLyBw k[_ZZ+k¸1QIw$[%+e2ꉱ6m">It^'+:uٻxзꓤmz77m$[0LｿR&F ˱Jt{s <#JݛiŘg4|X=tT]~lU0UM`򳵭PVj'9M?߭קXkK C l2}S©(m~}oȟk`_%+%Iݻyig.cG+wTw* ^Ԍ ,֒ȣ‡7Z\سowI6Iu߷ _MTxZQᡚٕ! ɷ tǀc@_[X"g:W{Dj<:tРenwAZIRA ƍѩ%}wN5PmaC5=N;,~ldwRkPVvQs?]vNǮZ<\:uD&IuVxi]n^<<:h¸1:|{IRGOO](-h*+++{s<_A{6E'5}Y֞4}V)<̅*W]vNc*++UQQrk i IDAT|<Xk ?"U6 J5ge}oV.Bᙩ C'M]⥴2#ZmVBzZek.UC5 ֺHSUUAXƥdiFܧEOEGZH{5,A.F҆TMPyJ̒YG4aeLE2n5_ɬ㮾<@᡾k$Y4a.]Q_(ڊcJ2c%2IciŒ,AD,U.Z[p R[\ aVZQE$8zZ*(b!5#ֹZƄBZaNJ }L~ ZRb\g)-"TISg+q*If-p^ Tp|4'Кj;J-yuy4;.7~R?++]<\'Kzqvwi tA;ȳ4.أ*i}A`+K%+fvwYܵ\*n}^Eڤd)ѷuDK-8Vd\կ@[b9A]ErfM+ɶo7,U&ˢfl-b`mT$Fx>)T TMa9oҴeFא:i8)1ߤcFk4&T-%)Q^"8V˨}U V+ɮ2OEzIZ-孨H)H&g)Vir>.BS3S5!P")8hfX#QT[~P6ꝤnV+.&y^)K*.FV1&dzv^TtAE?~zqCu6ᨓyɤ% g<21+RT BI W I9ge**<[Q>5m令2G#rOH Q FSTub-5pSN[YdZM&uq )Œ[* ;ɼ C5"\l4B5]fYI)Y W>?&Ma)YZZ.S]y|'FpLr=_ꤩKLk㾞Uպ"9SRQr󾺑<@+R q}k]vRWJΐydT}p?餵ۊe+P+wNN-3 Ώ*--f[#".2&>Ufij*S!)5F4-hѭK*"SgE~2^RTkwhQvW)9 n"yU/ᶟO)E}4=E9Z0_sGݖIolZ ^Za1RB}ZO&Z"^x$TTٱg53PYni ;*MnG ]҄ 5XL#iZnmC e9ֶfwb?&B,-X*pTJ0%w7,&Pi l,裩3--P>+Q5lQK rHRA"VΔuRȠJ!\[{W㪎WTznQq|$LE=$5dCWaڕvUxnf}~[xڲe4y7 Mt7ojGQuARVxkݕƉ+emuP>Z,ǫm!RJq1cEdYN|}@F]̅咿}AitT-_d`ϱCGR4!ڲ2I2gHcAcBE^[~pS~Z}DIw|=ǫ*^*C5Ui5au_P5̿  E~R]{g2I*E挳*oR"7zJʔ_xr=Αe%[`HZH4|م ZC5UxPnH}X")&j "bC3*q,U:i8iiWjiҼ2teiqjå9wD'ggu4)8tkԇ\-ڵ'ϫKV(}l4@H݂e-FnC|r,\]]"k|U|ee***T^^.s靛'I5&j ߶E*+wrzՖ. [oRW .)/@}DI*ժ8:ΩrN̟lhWű% ֶ a rVk>_ޔ,-IϏ(M&).J`99oJ F1NUİ\H瞝[n@-ܯᦨGܕUC ߷4 -N$Ƕ)Yᒱ5^7Ul3[Z%q5I!kN͌"{2dR{Ck;_KÖGn^]3J2]43ԉM9R]sK4kh]v,.cyAϬ>:7W|8CniV5 ɤis{M*<}g^?k~`neO.oXނ @Z[ WlV ;Ԏ*t.I]~&m)ީ1+imriHd𤲏tRgK^K3^ؼZj7l56ovT4 k|a:6X6o_A由GmX[Z>ڼ~Z'4~z[UT6񁇫r=:H҃j|Kۂԥ(ћuV~a<c{ᰰl6rNEn@jOh٦Kem˺$5ɣ m@v )hi)÷,|cϦIs?̑]f>mC6kufW<ƮyUXs/)t};hiq&/];|-J3GkVv vm]eee5zeo5%z|J{ke]0;ԣIe$j){4%WP/ OU..rqqQvTVVB2ЊnBMxں\mu"o3R Eߧ+_N|U3GɽIf ԍ֪.l9]:vjҐ=@ݚ;sl]㦇cR錒 KQN]s@wvVƦS*pZkO'k=cM&2$ɻfYoCfY/)t`Mb"%<߰ X=MjXQT]t7v=akb>t(iw*Qu{k&][}n1zCwh:(;c)_Zz|:vQC>ݫ=ՠ-Znh>/_Ϲ])B@ (i*XZV:&a"^znzk(] sXZu>uK:Yq$]YFUlUqC*jHjHj48`:vBN:j*cE#^znm֚ʷ|L{|UC52@ Cd WLTGV{Hqvi}t.Mbw|oܿF1(bHڔMulo>)feU erñJ3k? 㮾<@Atݒou{5ɿck͙lwyvFk~PF+9n?bikڌNI s@1Sn\ kkgT=B@t>]7B+dg=b%HpNw)A_;!NoW=qp>4Q ߻ᑒL8לeļVd[3$b"j!#$?vVhT;XҾv3JRGMuO*_7],2͉feES>/$O- RmVg+ťR]֬GV&ܪGVU[p]HU2)ZZ~]̶o7-Ub^3={0*K*6 K~û^;nUOK-'59^;Fא:jXik)h߱=K -%9?cjW̰`̒*30)aHWiH/:&TzdSڷRhUɬб5t,*_j}43ЊS]g#W˘ң'La]S*zڕ/-ד1wڵmJ60 _X )evQģ]iXӇoՌQeLﭞ/VW}NcZ#X;i=%Jc3bo]?Fl<3ѼuV?+7끘@|:kA +$3TfQc;ɿ]% 3d$zI: *<ȻQnz8Ggd;_$6GNćK ,5pSN[Y6[M~]AQ,%$ojU$拓%ԈC~eeyV qLqHq?GKU w錒 a{iUQ{ P8?*I&wm2W{XkZIzq_?\պ"kI*[ՍDZ@<ҧ1K`'Ԑ/Ԥ|]%OMER tӮ~z8Uԣ J%U(k'CʊAMTT5+.D+IyϭYt모v$}]] "<.. 0R@ P~M#kՇp`%l|9UKowh`ckz>zfɨu24I(]/Z]&cWUB}]g{plWi{>tJnC+EEYKb!|BҐzdM騌]5Y]s|gŭ?m3 P▘c2xb%8:vYↇYh.h:ƜSz|\p_z =]3rG") Ϟr>]ҙ :LͿoZä_O\H=OYZ_z;a}ci{w닗?X%nm @A ]}aqoS3lZ ^ZgKkҵv=ڵα'U ^:vQ$T|J_;cM =lpñkfeMz+n%lł7vMnԭm(8k(p[8*hPov'Kl ULx֤ ;X`8U?jB UCl GY]w1x! !dD F7uSmܑ!i9 ˞mJ=bۀ۞ifd ٍMz%uh̪HFt` ;>̽b&s2ǎ'DG"lz&ӳwI&O(/|$1g]H%)ͬmw\y.wu{nn  `զUnwzt.2/}8/>>J끱#E I:.Ȍ>/_D:Icd'`n|:*[Z;c13&fVR .G*MxM }0#O=5?p[[iOړ:3)Gm?7+_r０Glq; '_ZxW&mTr*L`pU&*\Um_o=|'z6kJnrW$4/ssl>ci~ża•Jl{_6lYùf!ݯC|+qGwY|6; 7xD[Z\Ywӣ̿3zƶg_U7n1Oɓ'Օg豓=depr ]xG<֙ʬ>/+?R LmaPx]0V9>SWg--zml𹚼m&k?ßz6fV.~mW7:g'?wt=P<PW^yevVqe&rW^U@ʗ53@a֦v Us%x G(P8p@<#x G(KF{k?%c`_1kQ G(P8p@<#x G(P8K/?y.ouNMPʒ_yW̾@+.6P?sYk3z9FĂS9V<{˄IɴI06xsdzs Qx/?=ΩSJ9熙`=5HZ gO0J)gExvoI3ydʸ>^eyu>g|ȶ|ԂlZY}A }sܲh Sp5{[ctSJuvgvmx1minfmifjvJ۠{۳{/n՜d4%I͌siٱ&ټrUr>spfv=tӒ;qǔ_" ~EٴڇwKI4g̮ ϧޟw85摙]ڟYgvmh{}w Ik>Ci={K-?w<9Sg<$ɑ|ysG3j~6_ho}<!̮`l)YeqV˝ۓNtmiyi{~&iJ66r iyun]ph^q4;u^H2&ʺ9-ܖT<2ܝ8 P'{֑ܳ|oӺCI*!|We]= l^יAOؽ1[;f{/15S n:{R6k3-i;ibM">ns)7'0"b_\7xZu{^ ;p"r$3{բ==2bղmoIӲ}Oٕ)Y~bޤy0 iGs*2F`,;~0i<>]Gc^=U33'S*KN̥7=:r>~mzܗ:yb{/ʢa͑W~ Krݗ>['ǚ}=7LYC/O>8{O{fܵ7;c|g lӒGf.-y>[L[fF3';T.PbޡNJCcR#]z]K@llwA6 I3 .ևuɂ̯LMjYB~mn_7 %ELk^nw:CY@>jddlް/NJ[f~y2?iВ#lUѴ@V-ڽY`j;(<ao<9!7Opg;2p'>|ڟɴRl5 s׵LJR}ysݪi߀}9vizA[3JN%9~ _Y˿oV{?M|6G'&N%U&gS|*c-vIr:ǎ5v_P6U4_KˋRds==O{ߥy3<\K?tv]13w7U/~|ϓ󝟕oZol n?}؞?˓#>]7j 1#E+,K5摙ie0ung&rϺ#_?'u-ԯ)]MNdΡXHj_=WiNö4m,>,ˊ}s`pQ a݊5sҴ@V-.0{a.iև;iـs{"Cz‹[ 3u߄ ?9~3K0fSrGGy ST%]':`v~3}aiZ\L4Siű<%?'U%jsUM5+9_ݗy3\< ~gmx(rEAQZ]~`,*O]H2t\'{?џFmٜ 3Y(Xc\p/oAHv%iwT{$ck?aGdOkmZ\Z(Z?wԯ_ϧa53r˽սAB -1ΠoT1Ԅ xˮΊzL̊Y.|9ȭ]R<ҕ,Ny*5)7r$)RTLO%F>ԩ*zO.{$ɴks_udfe'9ce򤪤HR\΄2QaRZ5}>a%w&kܦ`]Ź=>H:QJ${q}l7ɞ>7aH5rn=gt$ 5!iw\&}h/|=d?~|WU*ƍtӧstuuwMnAV[H~ t`Uf_ w}GJCy膞)`}0LH,g_$mŞyu~0}ӫgbF--n&踈0>fzs:ç7 2(LQ(%*t~l JY.ЪHjsw(yWμ+Fy.P5 8Wp@<#x G(P8p@¹d]_n%c`_1kQ G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@<#x G(P8p@$\.3IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/obs_url_bt.png000066400000000000000000000371731476650141400252720ustar00rootroot00000000000000PNG  IHDR% IDATx}xTo23HB 8P`E4Q/K4ڕn-,ժ(찵2H4̣DLJ!@b <93gޒIs]\dΜssf8E32"""B!Hgk+N]]!G`*B\HN'NV!"B!Do U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@u'y#2$!Dwү 5_nV' ؿk|>?"0 }|zs.}=q#Zillr;DrUj|2l1wM}a iGE[m4 z^7h9:ɥop:u١KSej@ddD1p@Qѿ"u:D͗`X9WZfӇ{?8JgK+N'ʲVN'Çt;/t~u_'\Rv1II 8˥I>K뿿bIUjmvvY2!/(/(zMl_MNyo齾Ft* 9Hdk ?Wy-x?Le䟻3GS&3#ֳ7MխZ[|%scС|t[C?ˏZQCGεPu AQm!53m[|?~u/Z8Tz-K$D:Pcg.3Gez  @4XW&iw=.c'&;t$7HnM'Gi uX [0{i{_V׼^'JÛ/mS|5+ (i 5Øe~9#{39t;@>G3y%kR6!:]V5ӫ@gaYŻU 7)_2'NH &M^(.jJZtT:N?٫t:q:eLw<Ǫ.2fWvp&mP1gݷ1^):#ﾋc͘[kڶK.CETGg4q^O9xMg|)g[h[.NcM4ݢccgδQD5c85xHo7P(Va8.fFsק8ZF1󮻘=!Z)vJu0&.(./`,8 >}L=R2nfw! +?Čs@zeJu;Ǧ~H?9č8[ ׵G9R~hqĎ~X dS;-{yg Y˿vh ߺ)wc>g}bJ@rD%*y_`8|G@%|㙵wڸ@L,wp`xRU* CvɌmAG/Ԥ$_ٕpAHf'n'7FiD=v^y48k m^-ΪjO :xzk 9qSM^;!G@GNRpc]1{_ǪÑ( %ET!BщZƮSI_0/ﻏ¿_,ek!x⯃c<$G|4peUM h$^=_g޽=4qPtGy1 1M8rZk?+ܳ4:8\M~co_ @~/1o Dp@Bt}I@m[ԪGrm/* 7{wb(ٙ!We"D3ruDOWHHŽ&R+neƬ & W(3D? DO|d1׷-uEuRaM|VK.π:D88PnADd[A)[:i{?Ng{7>J@% !"HD!PB;0)$ !H}&1"BKPݟT!DQ餵Ï1TѽI@B\C| W5?-|M ~FBP?(xxo^mn[̺Z$ !壿ๅO&#Cݝ[ϗ3z}; /ṫ\V!:CTVd΅O_ݩyoyOL ?WkWKxͱX!Sg:u:H"##D餥fG b !zKK /_477Hzӯ_?NCAdrV7W! B!DH@B!@BPB0*BT!" $ !a UP_A_~zܭ9 G_`ZXK-,IOb^"v׎4WUedR.?s [ؕNXv#Pk~v'$WP 3x1l}4 :c%ټX4GfNBB5@gI8>>S yim&DG'N6#lhuly?0>@ّp[bpN 3z^@f.J)ϩ/0Nux0H,XL `n?*-Ff\w f&lwױDw| <9c.9On6c!3-#'g{)bLpOv[ h8}EEF<'^VO-\_q:6seϾkTN+AUtT $o*bRX,G{JXGxzڊ<@T@2^?ӣH #>oUezEܠ s zRYffz'@u %&<W߀mP~--W]#:]$)O͸$ 2*:gTGUܙVS6ysgpwvJ5/=\ũ{M BWvRR"K{ \2)i9lb=ݓE/iSx^`׳{i+=ٓaɝLŷ,u'%lSkJwL1ygoqjjZ89y5$/>p◲)>oP{F2 pFS/Xo";ɑKN<{j&ݮ;I=wگ_?*TtF n_|J•fN^3~}"o/=\哝F({#o< u<缺2ל"03~HX7Nod&X8 @$a~g KYN-vghN5dg@>nh3Pm@ LU -=6nĘȃTyu2n  gB 7Gh|9h|t_;tE/rשT*g|.鰽ŷG6oyu,?IK٬,;Mdq<Ř~z9zR69e> 6dA}UgH>Ǟ "^Sӹh1מFDDo`n_=3~Tk-3-IKK ďetnqWv=jMYi[iw @o7oghOPW_٭jp[R_f ΐڅzj#֓ϫ4տ- 2U@V* B/vP rݩP~|f[ضRPAX}z{]G]@3ю *¥'wuj㺺B\U,UB B!Dtú?!+tjO^!PIBPB0*BT!" $ !a U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@BPB0*BmU6IDATT!" $ !a U! B!DH@B!@BPB0*BTqmbɳ-Y-!DJɰLҍ>OڰngJV:6`'b:}=2Y`H&kϫڰ!s {[UWB. 6>qz\\37sl[4JsK#sV c򎮃1Â%Y88x~WuKs|6Efxjt4ج;)[E1-ȱZѼB('j (mt?^uC8:a@{ Lւ:NQN޳a s}׃<{ur|ЗڃgI[Js4?<+ Sl'rw4K<~C`kY,>rCk&툕9  dF)h$^WB< 0z_+IzH[CFط]y`E7ճOjQ9d 2Z>XfrO%1؆Ţ}L4{UvG#cmLg9+;\QP-h ucg4|:L`0IZ4Ka QQ:Z :} G'iiݠL2=o-s\ۈh$irD&~VYFV5By:\rӬ$* I2zlϳP1L nvuUd2ƾF`)^Mo\DffV/Ph<7 ~;v=ewToH,1iA5G*a,ԥZ6NB\3Y޽{V뙽6gjW^F3i"c:YFM4jܬLfZŷSG8wk),o w<_Jk`PHê71ejTHu :YvRZA4#mL2ֽ̞CU(K TXe3W7+gjr|65mi.vK;HŲݷMigut}p(7{ C;PS|'}bt@ ueǶ\Z) ׊O.W>k3iܿM,U1,;1gFS6Ob]q>X.)Rwů4xQ@?`(/gH&P3X{qOHhuõBTkZP $iqO}zaa׉53+ZS{tT7fX8Ck*JWWvZk%&)8H)ͮ,*q9 w0)F]cķb˧%՘sChu 讏A4ܟ)0vhqeT7V{$TX}I'kD 5qOo\JZw$iob5BcoGo4)G9Lo65cTlwaj, $<) `P%ʢj-Hyt鍷tC jLgJ[m&z<>t{>艿I"B=E =YBM+tAO<q 42"4l5b9 jX!̢5_ T,lzn/FpSfݤ*U% zz*JES/ZFpu"xǟbηӿSvD~;+յDs,]95gfhRfC;qh{m41%lt܀Y{>C/ir-&V7AlsXW0ޯ7-dD~IuP"=g1c_%p ӐF>QwJ`F_%]S`UC4٦}z3_ko2PiO4Wi߽*k{ 7)PYd<#f:f5i݋MSMkƨy]G56Xڣ|1\_~WWF 8[hQy]o&1(e(3K~[IscV3I:ϻͫ+U}iLG4.6 ʣLf f`ʹtG{΢+%f nkQWuNM C0gf`qt|x],ay8#fe5IN5V*ǘ?gLw8NyoWǜך=\Rs?\־cA`"}Vx̮Oܬ9$Yakv7PSt LւL?M \ko'חI{ܳ*=^kmXK= _^xSWrkWw*/jcmLЭQ?tYgfuL ϗ vہ eR| =bŲݷBi)$i-؇R GO&pmdZk.vcn,wwdbPᕱrL175p[<-bq__Fχմ K1H 8srk]sC8J* .FW+3}FlTA& Ӎ1<X37)&r,3Ժd >1KL0}REu]df'(Z+`L20f2B=>HLժ:C%=$g%S(C7W"1pWg' iAk{_o=ςdWb wa׹oc]1T6wZkc_s*,JeZt6*&͊gGA5$1c|.pUl&VY:VxuԫM&wYR6-v b0g4{JQty\lUzLoVDf,Pب)LrXfY[5b:0L>]I5Q{9 qppk:|cf޻qwF*>'3gQLZ\i04VSG ec Il9kK#u@[ݧF/aeʂ8p} G<4=(u^3 : ǴXC*UJ=eI>4JDzlV+?b0{jP }w?+5;36;a :a}±o `CV#@ZFt2YX j09jggLVV2Fd.ݵYS`UyX\mf2j[;P>TGPo$@K3j`,߮]O7%NvaYQ\{*ĀcN !B+Lf&Je3.# >`'y^&w~[ DЏ2q$i5CmǤ;`@>TFu1Iwfp_D3YYe$]+ K W<ǜM'5(TP ~)vnuUi !f6` 0˯4;5Gj\m/ ?+6p1}:>E4Xw[B=tp}vtjĢtUlƃOA@) ܭcr½1~.4XKUoQ]ebQN@qef)_\*M%#jxu/\F%cSf{2*bmF'MqecZQ\l3@c~<&]ݦ pzCEjmpIN'm7҈+nLyg/AbXAi:ԫU`u 퍡zuլe*CJ{h2#9?6 ȻSo\5NZo,n8sMX&_: ꠍOh:4ڹJ^Ao$Zd§-ԙNN.2H"""^tBss3-BWB0*BT!" $ !a U! B!DH@B!@BPB0*BT!" $ !a U! B!DH@B!@BPB0*BA.SE镢3ٔؤm'{yW7zJֽ"#:TO(z^24fHWWo8Oa't/= |=Rɺ˗[o*.mjjJ*3SY籥S~t}6;^T!7*BT!" $ !a U! ۳z&d_" H@g5iokHVG7qf;jw'n&#bg_ףɘ Glg&]Vg yO x/cѶ\o\R߳:'VaX?Pg7S`㐲b3e+rk 5gFe FP^fD~o&*>+RoޠʏX%WPͺylʊ5̻g nS.aW&S50RpMJ ;pLضf 7[T|ed^9]AK'6K;7L}`CY.x={g?\E&ϽDMY=0+PW8#ٽy ܦtۖV'eŪ ODș{+ysqyMRXm-5 n'f eo*5U,#u0rv+Ip^2׬c`޸eʾ/Y2cP6iPEGCk47*CnolzZ hy=Qyp‚a\n39+zo0|V^qf3jfʏs5H|_ώ?Ofɘ:v5uPε)+6Sv//{"-ef22|~ Xse7al:/_q+&;nƻyDU||j qc#UJPC[H4{t*#y9kl6Q\mgPٔR.#Iy^y7( f7K~?nvc-,d G n9Y@^ !c^H١5x{7k~?)+65!ޠĩΞ!fmQ$}w?FN{a*z^;V7&Jc]3RW{۳ ]W㣠t3*O?:ea{<0fo!Z{V$}jok8ذϞ tkt ):H՜cc#Lg9k}\Nqc6k A|F w, f1g2`L}\wOƳ1}GW!!'ԙNN.2H"""Vr:La-@Ύd, '¥3XKәs˯ClJ(RTT&R;qP=^RuHwu1Dס !!a U! B!DH@B!@BPB0*BAW}..B!zPB0*BT!" $ !a U!.n=z?;՟ &J&7uѣ̜q+Ccth}"U!DIENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_args.png000066400000000000000000001762351476650141400252740ustar00rootroot00000000000000PNG  IHDR * pHYs+tEXtSoftwareSnipaste] IDATx{\Sw/[5R F"^F s_f&/ie~6:2NB[V[7}9[?f%S`\\Hȅ~ޯ2++k}7+':ؔ:8xh!`c1F "/KxG^yB~y y#D^lLW{A6t.:8*"/ `/ "/ `/ "/ `/؋\7f%K8nu[8h¿0͙yxP9::֮}eW~<0A5>>a\\\BC4l<yxP#su `/ 6zP vT}].8z) U?D^4oNņnO,d?v=7 *?VHy Z<.[FDiY)[l;pB0eF^~&M h=rK/7Z8)C%[l9lh'^'r_m%ڨpORlOٲ=e}qgT+eW5ayi&7MZ,DaQAѱq bP͖[6.RW~anw,uf_V$&+qe3ӏ^Qy *^W?:ֺmzuʔjX4?+׽N{mzBayaQWuDTfiS]ySVM{Mijk!:xŢ?N+,U3ű<۟OVjD ڸ ş+(81%.DEUPy_e+iVsZ_~y[TQȱy`*3BD[~<ה_DD!¬cB"uJNGۏWQ)DD>y!NɫN9&q^Vx狳75UP{];vƛctWW"%n-SPO \٣|^[NꈷBƸ oUwgm[`y[[(m899qmAtQx+m+f]flF^/|t/HR,x}ۯK^Gte63(q;8%Kiц]taגl|[&c+‹~޼dtf>M&ծ]i'",Yl=:]X\_G4; N`u[!1RUU54mg(b5UlOٲ=%vɯ+؉0͟Mɫ ^T][v{Iu \Z>Xfo( h,Ԅ2w"fL/ s=&&_`CL#&&&[CG /`lۅ`6_KdtTW,hXQ|v?r4[6~a[ E ӷ Z~PJ|iuZcTIu?-Nr۞2::/x9&aLitQ[]TqE|,ٰ l̿K\8Mqi*$0BEB:"抋]k+-Vh3J/||8e5$x#I)gMqv(kҕ?RO`zB/mڱRVSMࣼO,^1 ~X6#2Nh抋]zźbuli;̓Do̅]Zډ`w/ŬIGNjk*ۈi斎M{-f-j)`KEAoZyQ-#~](]KDhU747VSѳCX'̼h 4vbO<(?K}和+B|EB)?ǽv,H=M p.CmE?K5Oͪ,\׿* |&?( (-&ǣ”{U~*6%ۚ[7[j꼊'<%)c?&6LDž|B]\&9I-!"U=}ϒ?I>6ǿh#h>_h:x\|5s^Z %qm4Q״u3^.bP20V=фjhi4u ^7["L\1p E&1tWʯ/)vrVǧ+z yY~cF|tJJ. XS*H=hzkq-:^X.|-0R8&4F ^zdqoYڲz.^^ pُ9r 5~tS7/{/`#pé˫ r{mјj>ippP5<<66>>>[s*Wk䥥HEhPbNY[)0+[$ǒ /7dֲx狳2ӶR֯8Q3:/m_+D,xuLSU]?88dTϟNz{ܝgSoJcc#8:::;;3 Gy?2͟=~Ӽwihnӄ],(;":wXk5W\Mh֋ +7͖[6.ŽF*`aL2lw˅{}qٗ]W(6n}b&.vvzcc;wUC^˯RL]T*ս{êa[7vJ1b5 |j*=IrL(QHGV,)ȏq~ 1"I:tOl=wg_C~+i΋6̈́WIz&Qe >#H!`+&=kmVo[G팢.""NydD׍M":[YXznHnA. |8];;jP^DW^Ay{ýT*Jekq1؊tO$D.2YzL 9(OQb x3"a8[]&JNO*;z4d##?"Jl)"K,2 ,krƿBe5^ӤRYM$/Q18ד#Jyux)*`/FFJb4= T+_IE7yLxa#oJT ^"ڏoع30O#/ZE]_3k"5!¬cj*>?S[<9P|s_.6tݮ][߮g <”(]pe-{K[s/M%^lܻ`G>lO=#ggCC.mk8y[b4}?~vbQj½wu[r5}IZj( 'D8Vm̍(K 4)LJk-#C.HL*(>S?OmY|#'ȊkTxA?&Ggj3#,9w$룞s8IT|'MښSYO}f(M*OlA&5qDY Lݣѓx@D ($I[i['Gg2.Ttcg*aOD>7qyŖa' vT}].8z)|[3s"ڲ9i2MQSg͎ )@ا^QχTJo$YUSRU;+kd 9c_bgpSi~xT:;"p(LQ  D'dZԞx8@#%Nrwta~(,=&Lyan.Ceu'1݅"PCGQ);HeUio$m =`:aLZLldJJDYgʒ 8"xN,K?ίI>;]gDҮZ{`uא[7(ڻ9E]O8għ/N73L`NQWIi%!3Q"H34 ˒jt9]7#Ę|{m!SS__f39D/sH/ GZb#A!D˫  0mm4A汱}G_y^""͏QKb{S7˶(|L=䞑״t 1}cS T0$ma.?x~-QSWjN/]T;wj(9\$oT1|~-ADTz?)j4?Ľ_u|IqYMcrȋ6ignKbK#tes?omZED4t]ܢ<{W1T0:Vh4GsRK}I`l_%∴mރo_Ӄ`X5fitw, fY:S-³q-³O.۠9i_.4򂏎JeU^"'On`T/*"QATI~P+GY z_CgǢ>"oؓ 2%{\JId?|vX RDe}Gr'bi~Ly7C9Ev~nggTәgz' l sgebj[|ѝg|'PřV嚺:m7%<欍 Oz'W!34~%NS_ˬDv.xC*SoJ%4iTd8JSc[t vq5ddD/b:Y_5K2qIP"?"$Tm H[@S*J(*D+.4gV!X?يORYV`g 2in_q(u2 DDcSO8g~ qvX̑}OۻRҖOl]JD^>DPqׅUo ɽ^^ږzu4~4;]C+4$J,-'$:Wqο5E#;yS/ח|q"zYfPĶƆS}[ULu ٻ]yDTO8#/A~U7>_j 5HO~~¢z9jH}U[؝Eir_E'DQNA&y} Qj sÎYdxTއMf}ʹzzˈz^PkISn̖nL;#"')KĿN. Gb?X ޒ=%}DO-9$SƦK~kno%ՓNQӍ42.MfeO*+g_P#1 ,iBDN'_4mKBN[Fiwb-u.R*4i+HnܚDů(RB o XؒQˠZ4SN'Z36MofVZ]"IV^MS7ѨdLIF>ɤx)* m8ɉ&`b#s}:ڻZyOD8 "z?V/W?zݷwVˉ|| "ъEQ޸u%gU;t 7mEw䧉>DDkI~{2Yh9?B͝C6FݙO*'VXZ^QލO+iz3;'-` _Zo)9jkU&J%MQ-װGDl+Jɗ-)WiX%}kmƙmADD~ "by SEMU=u_B8pVnT6>PWsih"/1Lծ:'k]9EDNkh_uNֺwri ɋ.TSεri9O]>L"b"UG󤝫ROgQBZnkx+e7Jg;c5 ʉx1k4y53ilW" i<*0C|K2yZDOLlrꢈFQqތ34u59/EtB {%TVwGkwz5n cdBѤ7HBa~x:29!l4Mf%aI)q -StRa Lld ѰjHWe}[/Ѝy " 6N "/0p3l?~6I+nˉDDnJN-.uݶ?yqe KW[J:y^Tl'JxeX4ye뫷=n'}heQsHvSES IDATB%%KPT5GRRKIѨ-^U]܇3 );PD3)6)/qS]f_KY Mډr9 ϨIU!oP{]mi^);đ4޸Id]nҔ3p"5*m0 z"J F[(Ti@D8sU]NDTbaၨݳe/Qck3tZi M*׈"\9]X\7ZD򃂈'fQygpٽv`g8^AIIcl1 r/W2n'*/u{liqpctGoh~aqSQRӉ. )>;#qhʼnWZ O3GLϳxil={ZS2䓓)LY%jm=.PODPW(ALZQI,?Q;_d%L Lg٨LZ?<n~} p7ӓkKj2eJd(Hx7PtcxYUe|j_h(<~"~,G5+h2ef54TAvYVHnfkJJLe܍Zv)EEwLhݱ""⬍Н &4(~"/Do&"rvqqvyP#??ߐln蒥'ϟ !"bO:k%pzql|LOD#⹣UUI)~m<Yz?^ 22 ܝcIW,ÈZ|BAw:y(؛L?h_C̨9ǥ&.Q#My&i6G/<؋#4*nDVɳ4ZNppAb8EGE3\QS©83"F,%m\j !?OIl4~ɏLla"n5_j79Ԉ9/f3x/ ">gw"Z:!SlD>r^` PXu#x" {7.)/\.wXl;,;M`jHRR?Q l͆%39aewou.w@X<8$Jv{'I/>(rq?z_"vd,z1?}ے6,R(jJh{X+ʤMkfΡI+3?z>9||n>Ϲ2K޻W϶_NR-k^z $=a6O0y?}ù@^8Ev;<" ^9'3nNq7)Od|4o.8zg|`c>yTZr)f￈.$^l>kӾ~Doa{yIɗl~5y^T)[T|O@8~g _>P1א jyTKQTx"x``H|)/q2Z~ofz57UUP n#"~iQjqNN^fFӡi4KT$ɏyYUtDC|FqauH'>5FJDUeĝs2 &@dZ~rv*ۚxnI?b][P#Bˬx{ tRtI-m-DD>~yӅ /+S2?<~Ɠ+Sф)YYw2~]lgv8;swͅZZ\P ""sgGy{?Nߦ:Rt/A-%YMb|̖rޔ}V&U|rNz)G`;]m#s&>s'ouq+]Oa{ kH\{yqJTXY9|gVF]3β%>Cm|(]"'On즬O$DLχSs)G8k78MZ1k$6T~g|ФLN7XsTʉ)VdZùmdXYyfzx fݝhM"Dza.?>a|||3Qoɾ_#`{SםzTN C;m&aU<| Ziswv~X`"M.nN$7nM-6qONUJ9ܸUh^h HJ,Qo۵4\}MwtM=P!G"̯?> Ba>(ܥΈ-Hdumӓc2i lJ4s`mb׻"K*lT[Sk$ھ3 k!Z%~7>pfu+{3LC4OC9i"$ZL|DX<+PY.ŧ*ۥ[\{a.4+s2 S.|dǪu LJ9QP(l,Ȫ*.Wx' lJiv+Wޘr=*b"/RHO*9gjCҊ )&F21T#V8e3NN%ڶ̽C4O7eEw>gz*( !{i+3 qbGk?T7hy__x=\`Ӫ3rttppk^@蠮+6}Z.'7hyr֧}_5Z?n- pv7qB^yD^{A^y;rrrRRl)tQ[]TqE|,ٰ l vPUUy!.00&[CG /`lۅ`6 0-u+ okkC)00 ,k ̶U\Qm _y0K6BdoHA k!zP"/PB$ F "/ `/ "/ `/ "/ `/<(vrrV<yxP98846^G!!nؘ+qsx899,Z<\ lˣ`F "/ `/ "/ `/ "/ `/ "/< wM 0pBuA s/::{ h#{A^yD^{A^yD^{A^yD^yssRƁUo(yy:3CVE{u睾{-흝޾!1 OOn < B 9LHIeY vi8"r|fs?S0sc"˙#ͷT5? K~kT/} {u . XL Uvtt./0==?}}uxҧW1m]e]Uňև{rsnGE_iRy?nYÆhzez0jnV3?{|{/~9+8Y+{]ʷշ=#<a7P{d3yQDXJD%∤b;o3e9s0_ͥ{S@! j"Kk7l{f_"C}/^qhHz_Zg&m@(̧8lE~DAnLʪ~cgX,o)8b}DܩsfeJcgƝlШ掞1VH,v_¹nݴvvyr z޹a\G/,KF2=n5$i99YzL7LFyԧHqʊCE˒g|D^UQ7˪/ 櫢m^NOվ Oib4=DQʛyg7o6Z՚YW/d2P$@EMR f+"  2SLuKii!34846۟U]a({/8V6:dwX!`CCm=vީ ~o786xvM}8v H V ʈWN!A!'?LEySVFnR&3x L9|⯷_OH^s؁Oz6ʛr0jhtrP%4~.TJ҈ƝgDܙ7bz]Z+.jnkV=)c||vGD_&ޠemG Ԛ@qfDI8 @$iKOmw7wųQ7E/UU]gaaPk?;Gƈȑ::M owo֔Q|uraKjkn oĪT|_?FD. Q" HOSM]{#DD,NIJ0_G"kZzghz" ݰ*BGƉXCU6P+'GgPDTtce&(@M zmΏYjrM9Ή_aeϐcw*l#gVQXǵ[1"G8xZ'qRסT';2*:TT]i%r YT J}R5Fn 9+VYYt'D}*]j4}Ù,|Oq{plYA,SJ(?s -WJO_詗}·h09:{â"DDc=\Fm\R9|Ev띷FX ̻?}]JO++TtBnjP:D,[v8i:jcG30@`͓X|O6 ~R Çr9[k=<>YȟE.8=i^_xNM0n=}Oy7gJS=3e C#k$1ԻӔ' 1 74QA,mTDrL 7*jCN%>["HS`Ww~B :qeH-C?NYBzn~ugH~DW}EThlPȍtO|j1?Ln8"-?1FqBou'"Zdd7ȥu5;RYM$/kO/,kRMaeM8gʒ_d1Nj3&56!)`qb^m'y90܈ӝI;=B*]|B| m?'M !`POKCÁ-׮(C-wıo/ z.yܹզѩ+3$ןEw/_qڰ*A?Xu51ٍADgTQO,v62]T_mVVpw]sR 'dsXڬh7SF403t2yps3_=> IDATt[#!a̱[-W=8h =N,[ _t#y2Z., pomhfcq֦5^#+]rVERw*GFyn .z:u(ڮ0>p]CzGύ+ȹ_yw)׫s\: u V VvthOb]¥1OSWO|g˝A=#UWﹳ2:~k+O߶aA#]M WwV0U]MתP`Z>#V9FzjR4/<7Zhǚ40"c":uwܺc2<݈:k9ⶈD֦5WXIsP6|weԗ0FSq LR*o\*$E!ۊ6n?8x. .tP[{c&x hS1*ǸZuUWC9Q$7!e.Uߋ]y(dEFːaGRՑ)(YCm{him_rppzUSZZg[jA& =7(-(-\HEźT#Uqf&ym`Jd`.1å$25 9/VXLwab4޻Gdsd:TCLvV֨gyR_In L=mW+]-c/tqU[Mde#r ⦳S DOG2'XrmRkXu?w"ثlSq5{^>>PݹT' QWg0y2@X̛m;nvOք]ӍAԯ0}=]\3)=Y8:3BŞc4u#?͞CWOu o 30X3p!$bLyFۺ3IXfL||LA}n K< +h`H?1qqdx_Vv{Mm쮁%nVۭ1̮$!(i( ^qɸC}bL#da.KK ꘎Ñ)z"w15 fq`cYXY¤/QKJZyTmD;f-@FYb^ WG]>fztvvsG.1ȋȝ:d6ħ*4Z#ud&I-|X\eb[f ^B qq10Uy`ڑa[5o{ jc}k]Cqf(νE{ݑXw߿6Ҵ'$by[5QergßmeB?@0(Iʩn"qrٻ˼ADDB@Zjynz Xyz2VY۝eNk[yo)iVq-xfRT0BD4? 3`;s]׉M^ct&ޯ:s94߁ٕE1ȉw-G%r^qv׹Kc !nfNxFΝn 3Ftq5QsAPkGDm <#8׎[ p(NXT$JѨ)o͘>Ͼ[R4@*yILNX氱R]{~qe1P&cj2iWuo`!u;AXϱ]%"c-?eYqAs]\۽,)l}П^/ l:ƆY9 TV\$"kRfʪGtNGϵUJ.CĜW$QuO;̵N&ۯ{_G tX?wM %mnlrma<7<ၾ}#_;)^AQ;TݾCG}5pϐ w;s=>khi8ok xc򱼴.o ;ʿi7<%/jˁ,k l!ENX}xXӑF 9qcNcZfZv?ag{,e2\wnj/ z pk8V[d;`єgz~;q^D$9)yewXR^q5aQ^>z苫uRk}͡l/LMR'mܾ6}bH5/%u5nky?kק s;gI*)T/Mw9kYqQ:8Lyn݂g=1u &]r5ύwЈQN>\("bFS%`p߽|X/wuψQwYxu,Ӿ^č6{y;hDt]e#Ʀh?"g._jڍse9\l'BW|8/u' u_3MŮȢ+pYwYqȉ&A#MS͉5!Re{]Wߠ1w:닽ǏV71>k8oh]<#"E\j^A|x]G=C"Iw<]nlih;>^i="rM2 pkh*J]S)"%ƌeE3R,gzvz sڷPQ:;/lZGv1 QXO'㼘7=;,F91Ě][$R2]i5cTKe"X[yQy,Jj8^9iiTjW#kd}|"sJ7 ίT/gy}|Tn26,(tY٦֮˗EiC\h 5_䂈 çk]+N5;'-9rnVDM;D\p7(!!|F RTr,^>]}\/[\;:wIu^cS{gAA}6:~9x o< <Vr`mjʔWVsG6K,/MiԵ Udf~U.'5"j+GEJK7ЧŠ@;O^&oj#E"Rhsh~{;$Xl3#w>se֦& \~~Wg\yki{e^lAqZNj^D6\KYVsNg~;:k^n1~t|Ł3}Z s%dp&Nc1TD|%ۇs?'gOkjV9hȠu~8jtGjrNibא~.w8dxHPC?<,PD##꿮> 14 o07ϕo'gk~rѝ1/oN6~{wp >&zG;{cG. ~ur1u 2loin8q. c& {\ ;Q; j_:cmcgfCNM^>rђʝ4~Ywqa.6>l<-1c}ןgO7?8qrxG%Mu~Bi.몏W|kDpϴ:oKvz>l~Æm>q`o>BFH<"R[i6,R?bszM%2(|7_Wmm>V[SbڋjUv7$.1nU~^FZb pZʾ則ފڨ}8{Gqiߚ2(i?ycH\R엔y%dtRb:҃F =oK^ ھLG#yaOW18;!*.ң/_G׆]Wk,/rȶp켌5]nhᒥQbIYt5/st >e?8;aQQӺ-Xj^19zყ|FwT̄sǎvwݣ8oW؄/8pw0"nJ#Y&jXVo`p̽N}ڶȻb[_}Y /?u`Q/ǙqkoU'Ĝ l\ut]|׽+vkk4o Sl>eNםl{eGo_WT:X+?|l܄ZwDŽ|:xZD%"h>.NPy: rj?k:+8:uo]mQ0펦/Km='sc""}n7%0r$ 9X/}ub1ھ}Dzyx{yxy{\i )L&%z;mJOG<65uhu 5\#c9oطkbL6j*7)[ѱ?k<| xԅ?srG~ɫrePaCzݙ_XqWzVݎƏPϞv?ޕ0>Q7<׍[Y],WneN7 z[ϕ]l1Ɋ0vXFNidtUI"إ{1_Yj 2:O쮻5oQJϨ}X-Hnf7':jpۺod{"P)ЅIW963emš>ZϘg2  \@Mzd36XO~lQ\VM[S)v]yt_q%˷[M[o\󈀀Ç {T 6$ >e,Sa}ܵL7er 1"*]֭h?^{3yUŖd1jSh\7cKof}.gs#xQ =;y{SU '/ ip[[ \~k&W+|7F-FIR^)6E#+wp܁hebSm=³C{܎_xq_eqe9慧nw/yqIOc.3˚]9 >=T86i؍$0ElK\ke!wwWOb{7h X+. VFoșso#Fg?:]廧ۆRcŕkwe˯4킿\1~S}7S&s1KKgϝۼew=_I^]H^H^:U3ywg.\[3SrX{rg۱g,w hK w߼eݓƻSՎݷtW ܢz h@>jk{=ȫc/ƿc9JbPT(__ϷF96wM&U3{cOD7 ܺ{p~)wČ8[DN_=_y5nj.7;† mCE6bذȹ}tDcÇ 鮱]p#y[!}x ;)ݱO7_}F od.=yBsctsky 0dȠ72ؐd='ꮀQniRz7-O!yO!y6pD}cO7 H^ M7 zx x x x x x x x x x xt\=m&:0L=-f1YSH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^kuGwgE#ynd[dfC2ѿ`.iogg.A l) b|^D3^A 'ֽ%s/6kE^df.v<~z>;C^9%/l}fAIɛ9Jm^ge/df\aSq5_W>/dtI"fI^;sd\^i퓘)Y`UiBR( ELV5Q( E k葑oضY S)r%$i""ڸ ;2ӫ* EVDni9@M%+5"qjra֘3m  pê^9iC76.b9'\?ҬBHD\ Rd_풪1L&Se4[LS3ƧjJVuEșa*RbXdA*rʜi.w^\bڝenk":,nəY4MA>;Wa,{'{"`gyJSi2VrV+#VY=4(e)0T_yeKtJ5ީָRR⬷u*,t%x.y9p:@og0VDXg61+s""u;DdTD̜DX2Dܬ5VKD T"2F]bR&*cIMFiZg TZ~7,,YY]RKIj9rUi ʒ1y4Cn:i;r"F)z)4ODD05g~ݪ>FED&>fZDEȶkn1.w֛" 5ò[RY1,|N'WJtATHƴ;7i(""J>GSk3M.bI+3bW@mܸBEj-7/n8[똸xIi٢V$88G퇬uȳL&d[CO2|м"".jDPtڃDHQM-_T}[E9ibPכV&9 ma*(#^M)}vRrhPUj7i"&nnieAp Pk-T_EO:N$53g>k5$/ +NWֿS~qIPY_$*Sb7%PT%:r̫mPmre,܊4CX*m13,496]^Vmh"^\513;3bkޒw:,~(pԐò/ؓ΄AL^#:G*JE'o0:]_!J Dzu@U,UA +@R_gϞG>G~a~v5/U=;[Nެ.V$Yٚ9n^^P%/ 髪 b*|\^8u}bUym_Ԧ;mt5/ FH[.%溜kϙzCf'&Y_`κ7:]tlf:*J)ه&vc ¢P9&6"MOf:}#Ŵ>[avc2+VD Z"}o#wk^ +ҵʜ i 2Ғs$/ {hGIUf 3ooX[& jҢEL"_PHs6˲ Vg駥؂>mĦ>1쒙oҤ|'騅 IR+P)kN;F/Hj!m5.,dIӸɰ")&[YP||TMrդH pH{$eaAYi~sIPĄ9?ef8NN[m2v^r%DѩҵʜܤYzQ.tciCU%vPm  2-ya*TZ|w\e'Ͱk*Y %lS/&7tt\W:cT`=MO[ rF?붅$-{ :'JYRVX|Jd@[5m^\ֻ-"~E5tZSҶ_TiDNʚz.ևd3-\Կ\՝A1zKS( ,>;=f N{ũKv(;_:l֊<\x<}v]"*sJ^*rÛ7s<^jEDdSJ }wmT&#Sslx_Pf?sפYKJjAĜX+URE>5{u#y\+?2ŚO_BW#-lۮ}EdOqz7Gyc~KiAK(b/ J{城)[>G6>"[-t*BPdZBP$nm{ũK 2"WR0b(X\&""ip#3:5IH׊h-3g=Idy&U$NY.YΒE$^|dۗc)xh7f]Iwp_QM<ۚȼHk~Tf5zd ΝSQ+"Ӓۖ_{lk磾a#7{XR|s5O?6L⸼ vː9_!GbRP(\*r]R5&d̙fi*sTM괎93PeZ ^ +,NEnV0.ыKLsmMdVb4973nf?0eJcYn֜u""bo.S2lcT̫!{4mLMz]2˾D$lT۲%tͪ:."qr0ݞ׳ٸ>wHމnn>ބY.&j1H8RC2K$ZU&9V$񨆪Rt /[YPf]3T ޼N?^%""J<ҸR4q:"-O9MώQdm7-3SbRW;z_R#""S_vNS0%#5{9a1n|ٯZZj;Hlj.kN]3jyU7b:H)貖\X]'"2vtڲgEF]u Ћ)Jw/:. j^,g}ܫTęݔ@mw4:P1 UN隡C2ݰ?>h?nϼa +k=*-ip6dUhW͛'J \у&ffgFl[NrA=LH14ȴU:R+z#(**}viRJ,T Kv~|kAN^jtYKmcut6L}>p*/uj6n9סQzFWVu3xzb?ps˩FSvmj'u"0=%(圌}j[gWRٳ!ꢫaE5gW8Isܼc򡔫K^ *WUPU"qz﹫$#"ھ8%UMwCg( V tޜ4湄} #gJH#x%O-ѕe7^RRcۨռ'H$rz.o4Uksc~M"SKK#\`9\op~%4?0isֽaSd4+O4&pn6(**UُLх=e5:qAD}RL7.M&M-"5; r9K*J TtoJN%&uϸm+3)POѤvcGR5T[9Hf-ujE"[/m~DJ\N>*rƻ̦eN)}κk Z*t}ɞ,7>a~">~ #R>8wAfXm)~-f˓8tşW\gwP}??̟]H^nN7SuZ.^W;&l؀cSmMSH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<SH^<S{ޮ+=v⥞no>> 1=w_Z_@;aB醹. IDAT=@a2z Ʈ+/]mz|Ş}x>_pt[ ބ/bѣLj]|}|b#= q_]=t_z!@ ynT/^"vpQz\@ߏ5x x xtlU}iQ6!̿O͡/o\.Imv{pP-A}o[6ԼnS~ӍP)eJFfox**?Zk/>2[IMpG񉙱,#޿[ZtƩy9˚t}KYv^RN4%!*.aB QӗfMe3&\{O07>!-ޖ7kSLHZZtrNIda|wNuT,G3^mim;o]5(ߵ nԜS}!Ls>qqvvϔ /2{f=Sbq`KZ'|oxȷJRúgsMSq6tk\"U3xe怞nG&w[{-.կ[4.x۬|%Mgt}[Tys{5xNW_:|;9U7翖" njmܴ;9véo+-lZ7k: 񫜇DkIwԑP7TiǪؿ>Z-yvN ]@7jJYcEbED~i xYp>+Ԕ(]n.qe9ߧaSy."rm=-#鿔e\u L}}}b_gnc6FDx`]SQ]Siw'U&V1y  hM- mƥg^Պhh>'DpXsfŖOP4?0jj??v9m#Pp{[@6@v0oq;E\~['"RWg bۣ'04g1nd[HR51sWMH]K#i6oQMXmK6rRPC7 =ugree)m*^섨s?԰-uJ|BT\=?-7lH/Ԅ vfB0[[j)3^:~7;uq|?=?BsqYyX4*VcyܙZ[UvQ]V\}ɶΨqBԏtp邩SLN]Z\}IZcܙugr겲KvweqgrJ2QZ\<=@DdYT\¯Ki3&ۚEhB=m@zcS 39e*럘3giݿ.|r&$>6wYY}[5eƣK̦Qq/}!ER$DݟY;K$[Ͽ%&6Tw nW-> [^67uG?67yt~%RPȥE@g<ӱOOӦBHd;<֦Ƶ ,R_jn+K~h5ZU=3,/<]13*.aq,+[DgXLHYZ:Pz)+ۯtxwA]~[s8-QSfukVb^??uM51'¯ss.-.]/\4cM WfѲ剁""[i@r;>zݥIϋ#"7[_}c ٺmk-] Ƙ'}ms"?S*m\5 bggՉs-]FĈs{kH/4iXW6"#bHUKrdwm _޸mvâaD>[ziӈ.hEy~""Ҵi.aZ?n|1ZOn[bW27) Wm.\ϟ~dga>?S~{ᛟ7gysooٿoT]ʺqmkgI5{nuxs,Ew~E&gY3=ug[J$9lP=hҢooٿ鋲֋""k{s?7}mJ6c!W̲a#qO]a@萄 saU>5U[<xߚݟ}yܺwѣDU>?S*>zm+e]~[3&¥LY5w۾Uɷ odx~:Zd֤{lykV}%"穥n2ߙd|}@rwl&?OG~o ?h Gߝ_F>睜KmZlS:=>򗛜#wGz}]']mpjwxzcf|diuw~?#Dv~K՝1=13.'/Dƫ@G{:8鯪 vO'/8܋借;u3Jͯ'ͯ73>aqwdjz͗c妶 WxGݱsTӶ\ٽޮs>bwWZmnmm{?(gw|(6>ᕯ춏|=;|2#x'vQG09UvO2JŮܲQ7eK.]x… QϨ6EF6͎ۙ_&6:˚‡fnI[ǡM;%ᇝ;PVWb~0c-7R_FH@ra#DNֹw۶.<8k..+ LNx~ىq.&E%K%M}g:cFŖyXƥgJ &x>!aENȥ-H`r?l֘m)=ڄ/s|Ht[B؈"Nt#͇ĵ!CCE6Vy}G[ɶKΊ;z9MƸvGsu&L<{vu툰`9Tig'"9 )-r k/tɮŁTNb.^VajQ&MON@Ȏm.Kt w^ac#x۴E1K $~8tgâu ;JEE ɒkد:zBW ~N1j*.쏄 X.a;V-jEfP "À<}|0k>Ԁ2sטnNPM;Xy8T$;FxR;in2̆mM^fl6mр*r X=0v~0Gd[JׯS{FD|[ u"ELC-yC` .{u](ݟy "#) \iV x^ƿjRD!""Qz GUoȨ ;%:7tM⮗JjEju luf5n=w/2-riW5[ QH"ѰܶCl/lټ6ܳo J5,IܷTqBE'n^cW])k,CtԼm[O6G{}EDq{ɋe9qmcg{m^"=xΛMBtSƝ&2l[d@۹U0B7TVa"Ju_I"v*;pVnBb'堋 5I{5 ؄✢xHOpgWVK%HA.M\Wy1 (^uD{*DEmsf/o_"}^zIǩ\{ٍJD䋮 || oBr9r6 /=3u'RHOfTgƒ_I%yꥮmAiTSseOJO V\v,   _i9`\}~~<,biitA~5$"b@⦒ נ D k]r _"K~{wSF9 \34#rDhwR}=CJݪEWLƓ[DG}0C0l˕W[&g\x7f?ߘ;u.Etqav+4m 3BBZuXWVwHF;)ϭn\KsuQa)]ƺ17LR̘wr&Tr\=To:lxe>z"/hY0| 7a[qunqnuեe}֖0i(?^~HrdI/Y.5ÇoGԊ(gRGcUQ5G""jڟU룉)~ZXJQDlyEkN_!`_4ԝx.5~&_ܴJ'nj95qR!it#4rԜc(OX.UN ޤBLٻs6JSMm[XcZDBthu/.ޚ١ wvr\G=̯_%|6PD֧ṋk_-FKmMi;p*&"HΖgfk=uYiK~Ux^Fˉ}{w.ƺXx=׼dݦ6ƶ1eSMQ^iOmz[D6k [Vn\y;`lL}5џTSؔY{ԐmzxlሴeE +Iool.|\[=%}zcK/sb]tp9H `"=YY/Ao SױFf3[siYe3/<SEU uD9/Gpqqi\'ޥ3fSqfj# Q'&?ؓ,k.Hz]mԟ2xGC+7ZV$kmpkfIߨuHaM<;n1Ar pq-R;!q ??9{ﶮpǡ ` x(C5{J"⦎JڽaD &WM9[WZ"h?iEʟj\3 \~'^<1':6xɛn߽!y`oSNqt`RS'9"0QKgXgɥsf[M@/2gs ]^H^^H^vGWdmowrR8Q:hk*n*g8 H^* l )^u& хʎnOٱ `Zggk0Jg>|G]x _~_]8y;7.ޘ ظ;y ؋ 0zg>|G]x ).IwдNG`4|PUȢ kWQ(O\Lq ܏ޘʔ!._npt!#E7.dܝ IDAT"y&۷;x`S:;O;\`]E`/$/B`/$/ g꿼kۼo>p@E W~/;~uuU+Vs=@z>h2[5c|k p 0G܅6/BbCNշ(1 ɜ∝8j\ĠQ9S5'O6T9B`r49Z,mRu{MDa܈;g>Aa."2kw]0H^&kʭ7[xdY9e'Hݵ2_e1NWg>E7~%=a`B47A]?6$fĘ!DJ Bs߹Yji/u9 M:!!荅u1?6$2:d{<$rAD "b9.\V'ŞC4ON8fG)#yeD䂐+X{k#tyGul,KR[-bL\ $e+ySE <.Ѯܐej=95$2J BiǤbK1L:j~f."cߒ;2_:1aQyߜu?~U/UE{ 6UU9EN\5'r6 sICLB>Y/fȹ5s? YidԙrؓPYQ/UDDZM[KQ Ya=z׵MZ09;y VE0 6V&x㡒\HDo<Ӭ}S(D\?v'C\-',L~^yؘz1穃t`K֨} bffܸ|EDm{ֶ8Ya~H^)C.02 s)R絷J]/=i=@U:O_ӉE--ɍrѹՙ6+Z"u9VY{`rBD"4U>&%\DYۏ6o-ʍ66lyPД7v|ED-{ 6?*E 9YD=<{A3JVlKɵ6yR ,Y"7V{H+""^_68=Y*+v8w.""gUW""HcSyG{T$MDDLeFDD̵\woTd\ڽin*2gYeٯ?he^kFo$/Z--._b]aњE|DDD!\/8޾"-"B{2oݛeW96 $ʰ-nQw#HbݨcGm\TP{wV[_ZnTxnij:~jG5kȭ[suGƣ&`jSDlHl4 ܣ+kRꛚ7yU7 j͖ԄVio5xL>.qu~ޮvO;.9Y-=Lnn]kgn5+=Cvl5䧭7:ΙK:\pD?vYĿ{Ѽygϻ M_`b#ih4{1or1,u]bV%4'}Kol.^*I}1/t`sqqi\~46*""LmqGo#S_yn5+wW3`Wn z  y0:ꤢI.9ѱadzOvG5cP0 `RS=硗bc#wK^$/Ieҧ8`0 `/$/B`/$/B`/$/d䤰; p$k{UmDm`lU@oUgk|]Ȉp0M60uvv:t˗npt!NNߥ; `*tm`zc$/cFB`/$/B`/$/B`/$/B`/$/B`/$/B`/$/B`/N.۷;]8" w.dh&EC:;;]8AuGW!Ύ^uF>)s0M6p?zc6&˗x`*S:;, |хmShcn<)Np\҆63"s#\fȬE}$/Nΰ{,hlN=C]O~j:ڛ51ur؋X/n?q}{W#m3u/܁6/DD" 窱#\>9utkñ[.go6ns^E RxF&lk3gn(Gȥݵ?~W=; W6&&qݿeB%j 낸ÂƘZnںvw݆dTsІo.ڭ)E"⡝^QmjZ|d_DbR>͋lZ21'ehݷde晓jDsuFt)ֆ' Ec+DD s|$dEY- 6q:ǗT;{0u=ZrrU"sfܘ>+=r%rCnw؎T\""7.=m@DD|<bڛsa@玛.JxLp`;]teCE/ [:JL9Q%"xe ۺ̠ m^nřIL۶JV6] }--"#hSdsco+cUM-"Ācyx#v4~X޻`1&;]hIYpd\H`S˯mCܢ9Zݳ,""4y8#yMI)9/ l]wJ~ɋA]}i)oFBBbCC}ZePBv0J)fLK7c\?4(>uG0\Lt~|xUdnϲʲ_lK9>,"20H^*2t~oti9Ŧj"ԁ"b64Ia[ܢJnnJ^4SPw/q}cV?ڼuo_5!/>^"ڹx9u5n?`sY~ѵ51 0eT".~Fs}pw9QR\}31*55"r*"JMjm{[DDt]`=[S;Ul gOo\]پȻ-_&L@>~""lo*"|2#ɞ{Ň7~(Xhi1gF9_6n?Oъ"b+ZsJ I{uÜycBK3~r7ED"k&SNzp1>~"jnnNQV7[ \E&3Q ]E)rΜwW-mDDʍF&J<&1yMؖh\&t>jF]uqY%5kȭ[|t㺈ȣQ 0u&f}E{ E.Q [m}|ε& s:Dk #vS~Ro[dQzCWtm4ijPMkû7ݮ7FqS?_r9sɶlWC?Iqլu8u=/ĕn-xPۖ[0Du2hѰ<\aMqÚ>0ؼȋÿM{%˘a(q]qh0Ff?ie -͛8{enPh7RDdaMtt FWߋqtC1l}0aMaR'ؕ.me?:pQB ѷ6;Wʮ,->1Kg&1uHc""Fw60:u9u[rG{ǻL]~>~rʵm]6SCA$/FGTtys |CA$/A}kzjC/cF'Ι9n5H^˜Oqt`a^H^^H^^H^IamowtHv''.0."y&ߪ50eY۫: `\m`lMtt F՗/7ܾBKvt!#ET6.,=H^ ^H^^H^^H^^H^^H^^H^^];A ow8p''E@ǃ]HxKW.!o.u]tt!Ltt F_, R:;;pk{{b_ Mnu}#,lƌi/x,=mLV/7ʔ!._npt!#3.2YygZ]$/duv )N*;^hLnݾq_0!~ #x8)O%W*)tt% [e8/ѕN[2mZbi<UncLBz<8n[--5Fk*H~!vwm3cƌQJCm뗷ʇ;] B1xrcֻ "mEJ˦Xױ?vSUo[Y"Yip?p<q|""k>~}홿oiS@oc"`OGƣbLm$/5c VTz&Ҳg^c$?q뫯z1) ]rsEإ(%>M^ˉu'K[cǍ͖g?v4W%r\ GSɸ9JuL_T=P-">m:׋Z>f۷/h+?vBD"||47&'}6.'p`i[6<C4~l7eVgg ˊ"Hx.&lݱAߝhus )ylwgzsm}S̙"rs͛_̴e͛7>b1BLccsU[Y󍩯_Y,+[Mido*\W۱3]qYǷO=˩ߚD$w7u6^r2_ G;ۿEܶGED \}S~]"} }.Q [\u=}=~e:ʠ{NHʐ\% qQe25,' 2v#Ǎu]g4mL.eik^Bu_RE.ScKr،*ɌKC<+0"Z絶~q=Ե 0nH^)cHC-9fuaYFuSYϽ(%"s }w[칦}ʺ ?o.0&%m=Xh!6 :S{*3 ŸJDo<Ӭ}S(Dմ5>MZ09;y VE[7RUDDDyuA\aARcy-7m]nic)Q=4,ǎu3!"b)5|/pߖ˙ ۜWs?_QUAI|`¡ Ճ IT9Uٴ-9طœӗK^커T}7 bTm?T IDATuE9ihVܚ'͹1Ykvjhj*{m5+w}ʽyf'W<:no5-Do:ڤ-|O4t_X3L)ئٮ"D*7$%ZR{0';97/&ynwI2b77m96ƽPNNt $Wh1n2zop>\mnae:cϒ{K<p uޜ5VX۷o4΀_$/pVTtoZrrU"sfܘwܾW%VD)>zPÿ\&5'ޗe"RJn&*D$"JXwyocR]<$PDW\C_:\VHgg畆O&u2PU!|MdWn)ݷhs0K*""B!wO5Q8{z{(ED}Ƌݺ梺zC 0gJ!"n^=)89}>l;H[\*H3g7-M(Ģ 6NuRqo1{xrg (g( 锯ٍ7|}#0v4xZt//V7r&"JDՇ{ޠ,Я}}EEDtN ="D i(?5x1̵\r7*2H.~X}#)zz-[*R}FDimXL˻Vdg"CEqE Vdԝjy: UbtOzV:*ˏ. ߯mŲkEr>?US];Yޫ0b޾}I=@7]\};8Qf(~Sbu:8vr?g >\:N?cپUWi.bl^KTtJjN:\7eVy&&{[ǁ{=t/?-o޼)"2MO m^E﫭 "#) ܨeTo.U/ 1uG3wT'R6`q"2]zCF)9kw `ܛSamSHOhuw8%hjEju &vqvC,iljW?W[( ;mK+1Ė^–kÇq¬ft >8^龡^S(ԁԌ"""C3=NDr}O$5u"!5[~5//L,1{dH|կލ-Y[^Q\ﰻ]\Df?7$/T|,r)FL,iȔpʘYXP`XE`>2(!;pZi3&zvR=FsŎM*dYߤ=/>c 8Pϙsf=w8l*uE1+\3WI6\82sqc\sMUr_-7љ~Vwz߈9׿C)?To^kӧ۞|s$/%rF{/R{NDݻYKDDJWEd=qpՏ<3k?eE]Oƈt|SSYB7E+r[F$XZZ0oݛeW9*)P*WEB=:jϋY\IګIg&k7=Qt;OjvۡGX]mkI5%}ϘXnA:P}F$(XKLAgi-=Ay\T"MB|M~=/&el$:c$|hM6m"?Q:;;oߺ-"V͜q^)my2aĎ̲us}wK+eV5Sǂӊn˗蝚Qs=mgkjn]S]<}_D() Tn*ڏzTy4&:?RimH%ԢhRb4v6;7MK,~=JMe&(ˤprs" #z׸j5a**7ʉlT" XZ' 5ڹ""J7WRW;_us^I[:k{tIAQ7Ҡp7HMŚZ]6ù^>9mU{]9ԈcHef|h*i27EhG 0&lK4M]x:N5r6$lԾRME4We램_!9JUEiwnFhW,hr+"rz5o6h+K*{E{4br'/5U=9sg|╵땊leQbҳ]IGԊ(gRGcUQ52'Ru֯:[_!"r\pP[XYo*F%߸FxZusϪU9r"&5´ K}bԃ㺦Zׯ\ӑ5&yX/:Po+Wh7ss93]'mFŖߛh+qo>.">bۓŪ5COtܩ K6/wZۿMD'Y7D<H^ma_e0ty.`o?hg%-y a=ͺvZլ IXvզ;p~vg5jmzۇ;Eakl!OrRs\ڸE; zƑ0%FߴY:D5w~oю|VTeXEMp`sl2Ɔ"Cv'"R*^]EDMgN4[_M1~M6M6Ci~<3:[TUAIHnqS6k{'[o񶴒gM.cכ{Cw2 `r];y%E{=OuB䬍M*DH!_ 674^N^B7v߶HFjkgc\5~[̶ADmr19Zu8Ozll!G}dۚ :1ލ&L#/$-LܷkE4U{bB ehE*֮N)]ށxb/'QW΢t>/oܼ~"|kz,=&wRJEUcPz 2V(]Uw|լxuq[siYe3L_'j]D 4vh̙"mm7{*{c~KhGg]uwu9u[rG{5S˱[qdacWmm7VA5ǎ]^E*t7jcN0ΚK*5/$56{Y& W{jVǾ q_lxd̙]x6e^tx#y0̏/wtꤢI.eEʟl,:u8wly:PrW qy[42\vMD"!>ߜDJO)nBܗ[C϶Ә6mZdys6n]H^=QTTVxsՎ!Z@3~ﮌXwfssR0M6ks6х`]{!y{!y{!y&+''U#Y۝bDRNoU#~OxU@oSlM@ o.u_LY_tgg\Bҙ/_n}"''E@ǃ]HxKW.!o.u]tt!,=H^ ^H^^H^^H^^H^^H^^H^^H^޽FEyQP ` KXP\6IIfEjhg8vcGac1^2J4\,/gңFKK \"0ɪbwy*y"]vZf-h@+$/Z!y VH^Bh@+􌎎={޽ԩSrd„ 3g|E m۶ {v9wޤ$m69}[GGǒ%Kϟ?hРݻwWUU=zj=zfϞ= ?%K:::ܬc6l6|Y]]kX #B͖۶X,a;b2L&S?? k׮y7_B}衇zo]ʗӧFX,gy橧 ⇘L&+]e+&dΜ96f%''M!yzڵɫWs9ok׮o,ctB$Ng4+**/_>jԨ)STWW_9Iql^\\---~oӦMVUq 6l|MmmmOQ]]sdYR]]]RRLYT\\(-Z4|~zŋz>((HdJRWW'=x`0̞=[/^[n&Nl2!]R1Ͳ}d4}Qvvvpp`HHH,=:))I:tj9s%&&VG}XsYYƍCF;wB 8pРAr˫j$%%)a2E,-----M>M6#B򹠠S|)Lpp,l>|xvv~06aÆ+Qˊ+_=q_b 9Ḑ3}ػe|{z-!D@@C-αdZxqVVVII ^B$/@_;cǎt:v.Y,]jjjv#nܯ!v1:NsN#;BUQQQQQQ{zz_~C W^ݳg'2D^<&oVOSy]#yN/$??ԨQf*((u:]qqqKKKvvfKHH͕'""",' Ws?~|˖-v];uT\\2b۷lr^{M#ʳMnn>^\\,tv]IHH>|f޺ukrr2/z :}K/?i͚5z_oG̱~5kg͚~̯j׮]aيFl?-绊EE111ArGn/B{5/@tҧzj…-r8v$zE.\SO-]:ȑ##Fљ3gjjjN+=_rrr0H6ʕ+)))s؆>/3ׯ+((Pz̙3fu%ho7n|?䓍7>3?1cΞ={ر?B{3A&ҥKwzcі-[BBBƍ'_ZV&{/Uy_j>IOOMJJRˤGbbbY}҅ >ޘԃj/ƽL7k+C?iӖ,Y-ZBFPZvͩc#ݬ!R픹?4ay*+i>ڜTA>`ٻ(Kun+pk&''^f.~~~W?sϹ_!tHJJRZ|=IL&SNNzD~*-----:))iʼn]%41LBltNp M6zF`FF:qxj* :yLxxmǻLpwR‘Ҳs^9BSXGF'G)ZinnQ9&w377788855U߿>ӧooT՚ӟ붋+CJ$J Q>(%%%/̴zk޼yFQ*iNbbCwu#>BJOOGa0nOc}^O.0>JNɤ !Vd646546}yȐi MY].RZvk]d[>trwb̘1#2֊V`dd3,Kii:&Iӥ !nܸb˗2hiiPYVq.撒:4n:јLޕ˥<~^yxehq٢u97n<"῜.W&()*:,n=(ٶuW]~J:蓂C'Opɓ{1xvvvvvl...޴ib޼y~~~6m)ɓ+WʪSo#OɓVr8S4bYpa^^^nn:dC_ ^tu,Blꓗwh:˕ M뮤U+䳯РԔ|.w|Rf!n6lX{{XbŊTLkkk{{a/iGWwUWWܹsѢEq?cDbI,bccՋLJLLTIKKZ]%Ų`,7WSKW'_lE϶5 W>qvhPyEVۣ:ёWYz3喿?_N<@EFFtcǎ=S?vNsh뒬p:ө#F|. !^~CLS5VQܬfW/$Ue5ꫲWgYFW_0{si94D}͋$۵SNaϿ]徸i.#)D5/F+?V7dȐ~n}={ȏx!CuETWWBBB]tە/f #زe5/BUc6ׯ_uVrrpps=+˚nXw`\4٢ё᣷}!r6?:>JܮEM&2gڠ/YrfaW5/zܹs.\e˖Et:9wԩb۷lrᬬ,^߃{~j֞&oP45l߱KIտoݸO۾cWD3/T8׼Z{饗?YF׿2|9r2nk֬INN5k333ŋ_vh4lيFc~GIII)))999zEޣ$nݺOOOWyג,EYjռyF0lѸt'NIZ =rzġ<Z">`~*:䜀}ǮA ncΩru6Ʀ5Kr6o߱K!Ҳs7ry^timm… O:xb#!իWWZ'-]:2B#GX,ON9sfԩʈ%''G]!܈\K駟&%%uuo~󛤤?XQYYi&tultNIz˗ !:dd2;Wsb#Eo}U+~tkUʳO+* `^˥Gw8$hllnk%7\z}@ֿ,XTl]y]f!Ĕɓ_FGn\{篣?<^qO>ٸq3<㏏3ٳǎ(HIIy:d2JJJ.]3嶶l2n8jl6EBZJ~ו+WZZZdT[o\?44T鸬d[l*++/e`dZ6m޼yBuP{tsmYfֆ#G !d+_L4 ʈ^R#J/v_/nP/ٿHDDSRR};Zʓd֭[䌌 82yqn.g7{s8+d6eَ>L)**#.{*!DJJpUe/Jbݖ'++i#L07}Uz3ם6p:vCLL<@ws7W;9׼n>Fh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^Bh߽M\ohhji~F[Gg_0h#i=lٲ{.efmy'dnThmg[Ç drW+^=ec~[d5k裾۲ ѓ/^kNo޼Mm o яj^WVlo}/)/_{˿>́ёnokkw`DLfZi9sѿN{+WTYY1ew}W hֲ~ÿKOըŽUU5ǏyxT7SKle{͉]+mmVd mnnQOpSTl]a7SҜ;eTA16:r}}}_e>ޜj&rC!đIU H{gt 'T^V^䞛FJ>ni\d;p9|Oϣ/J\9+/9tutM>5>/n6}^@g]7 7nظƦkZ|<{B7 o!1ve{ }[CGODv#^lEŽ7=>,wOyF#jL!]]/ݽO\dpkJtTm{tEEG-9sUU5$/d1y&&JZ!Рeo}~qӶ$8_Q~hg}߁?{ 7_k0>J=yawuuJMFw$4dFWutv^9?P4yqoO3g7gI'D ,yx͛>m0>PƦA}KyhHS$7CpOD{#ij%/c#}i]ik"rsi#% jhl:_Y+/u{ø_&/Boo_GqI':6rpK˫/tUQ7 /v3ק3_=9P6rnyq|eۯks,)-;g\~g]vyT_stw6$xjZUx{iUM_oL^䩲n}^ws_8hqtcen޼9 v7[>yzkOl??_VOgtljg*/T !ƄΔnGB8NQb~to7ۆkV_GqI'lPV^퓃v6u e`"y1(e/O"OѮeL#Z } Їo1b͙sw™sUkzz_5~~jo{xTp#l=6qv@E.^=}lGgڀc8dE۳.TU!c". \Wrƍ6Ro//??߀AC1L@+ Jh@+$/Z!y VH^Bh@+$/Z!y VH^Bh@+$/Z!y VH^jW1ڄIENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_entry.png000066400000000000000000000463631476650141400254770ustar00rootroot00000000000000PNG  IHDRz- IDATxyX'*.$GM-4SNt,Ǔv44+V'c4dVCǒ$⊢"3, 03?ƙAAy?+F_gSǎ !ssyI$ppB88 !NB@'X][{zJNjv!@t緑AtC_^B}rJ;㵒>e!Z4e#wezLgB0ܸ :N^N:PQi^'HˇMnis!Z|yO}e˫Kd[_]+xp']0TRo8[Vy-[aQ97x8qY];3zjx:`%n rEӃ Rko6b׎7酡/_կk]+ڷa㝿orK9TRAFVP/B\DB-4E=ЗdI6Ȳ33;@O:RD5MbٲJΖdLW+tl/,o"DB[_1 SNx TrF>:۵Z Ϸbv sZ]S| !n-nЋ:e0,>%~ƛYCWZoܜ-¼ uin)z[++q[iQ!0?B|(,,6μ)*)g:BV^)>o7VՊJۅeh0o*ddd* 瑜AFF2ln(K 삾8KQ< G9%.pI_1lWuևH}-Š70׽gvq eቩdddXZ0 R/m\.ތ}|ŝń@`oB.dKH>ތ`_7"u]?Z\.ykFL!1{M2 5Q׭W1kcHe,&>2x7!5)-a0r?[3x22ċ>v2!z*ؒ_h6G9Wn]qZ݅KvѼ/\G0<1UO . u.ቩ$IΘWhb`▦?Mn,aӡ>X?~ESyo:OYrgbB`H/zm%Y,]39[Zi^ZJW*xˍԥ|W\a^J3Å+7muZ4[Ϳ ](/ښ[[y_+c7B*0|3u޿G塈* b27:On,ټ%)U? 硩נh<]їU3мm˕rf^!{K*[Rsz0#b|׿V`0dbi8< v*.5<1G1(cQqL+l;$ (AK6md3X36`#=ں@H^Sif7/Œpz?6U[^EǙr)(nPDQO-& ՊoGIjrx ~ qn.D:Yi`#`@uo|AobҗX\ w R!|$zApP,R'[^xƶ Ǿ'k&Q3H0iAE=w.UE%DxTo鴈@WZ_<]imՒFI׵Wp+n*ze] q$ddP/Cԫm3DFƤz1^, u;t&"M,C q$9/^ōhGǾG|`jTL%M/꾏35i! e(uz])8_ݾJ "ܜWV-GQd62&7_W2[-₃i{޸8U<âX%U'Rlfo-"iѕUڠ'!—ٙ1 _ӝ`@I]@ѶlUn+Doj  2][ԹF{r*N  qkq!p\?b0sAv us6v,AsDuYzUÕ{<݆T4U:hW9;a0Jg{7 q{hqmUʌNSgsQ9;,nΌp%B,df-:rsfB+q`ނK*H/|]]uw1_j:.[k!`ʟrv'ph?S`7W'iT:_iur†&L+\rs1 2P s|.@;mupqBd,h iZ̰ᆺ3#ABjqNB@'! I$ppN/d^BD:O+'7''2חUL!$ zs44 $z 6/! mBL7]@B@'! I$ppB88 !NB@'! k;x|}XV[@l#h() BBƲspqqgg]HJB88 !NB@']-V[c'8}&+MvMMځ 650\eZ-R~ %ڄax{{X//z;ֵznnn :Z\EBoo߽!P\\}G^^BdLjp RK-Ͷ~nhX`0IX p>reנͣ'Pt3gαj>m>޷lEs6Zsرkee˯_/wQRi3L姽P7u?WiWO?s%:}ShA:p?LjG~Gne'Ns/V/)ѐ)|Ïh4x3t...WΟH6 j9s _z}i[ɹˋ򻇇p}Wt)qwwˋ ڷٙ 6KvW9q:oGAu_/紃Lyf~Z1~lN J =?DT. J68;;ƅYMOqqqwF͍ 6y,~{9fvia¸'0 'Z10Ngk7xEx(;kjY^#?EӫgwWg{}(?zWT~ܝ:=GAvqBCZq>K iMqI %$$n:"!pۓ ^J%]{ɹMOX8^o`;۝tډrN^K->r##?C`=o{u2m: ^})b۹ps.^0hLDqO6n<=y,~s?!|¾ags'.B]ӻ'.f~X a}ϼȣ7@ii)iS^^~zvŅ؇b^vC5b" N=GKxy)8wV77W\\E眞~_~9Nm׷7m۶|?ص~z^4a >7&;99]Aʷ;P(y g ;TjѺuCGs vCٹk/> sMBAD$<,^On^t:ڵkS-O63O589Y>UAW,ŕ9yA.eӱc6~~P2ɉA0O|Y_l?̚1Ik*8L@?'O3_~+u]quuJrNlNb Qݘ> ?lA~Wu>"{Ov39Ӊ-N!0ПfLAJw) ɉN ߗ/>d3~,mx{Ssiʴ ?:gϞÃȎvM3(hP(ܩ4mڄ奤XC7FEu 'yw*c16)|z=:]ݓ>AXΝFS*!p //%1Cѵk$ղ.`萁|.0wG+{T^>]:G9zZЭk'ŷ'qriL:w7Wpr"%i`PL~}oc'8zՙlӃcO來(x)>{:]{()@B&ۻ޽i_4mB[[]/f fNDun];Сmڄ$-_} nnnTVVRVZFP`S'C/{+=qN:ҶmΜ9?RG6 зFccÿʏ?4ʀ~}Hvxo;n~N;hƀ1xm"oFqtt?z=.fR9.0 \tٱ155?G5=w^ly~ڳw/Jytt;yiYG痣 zztJFAl~kNxX(]:GZ|j.dRD%4yIqq {ϙ3ܹ#ÅY>r¢b:GŠ̥Ѽf! !ncz1!DI$ppB88 !NB@'! 9l8[yAB4! AB88 !NB@'! I$ppB88 !NB@$hO;Bi{}ۻ7aDKzzR _cײ!-g6H[*`1 Xꀽ'ƅ. 1,)?ba6W91s߰LYuQGBI"ԝ&q|Rr&IOZtag㉪vV\5네2qAi1gK яIsqb$߽:!$G4ap/veVrөe6g`ujm&dlCs9$RtmDi@8 k8C1|^0W0DuDe^ړ38נ3BIJJ|Z ƅ?h|;û\|qI{˜VeVhv + t彤(j &7+K^Dun!줁m=8Fˈ;};_ 뻏H:V^VqSf9}z2FghN@a' " hi/bٲ0f;B]HptEh9 &z&,u%|i۱#ǎgb\6:LB@8IB@8:{O@'! I$ppB88 !NB@'! A)%a:T\2:W__ZcYuh4ޓd6'%%kJ'O9}^J%~'TUԹ3yy|uy=0{椦>u!EB{XjϿfy<:Ǣ%+hJ/ jVHddݺv2/S*=Y 4п7:3mSlNJfˋyG _{OOOJ[h-t`ދsu5h4,~g%g3,Kӧ=mamBFI>N@{Iyt A|%k̂Rzs-2cCkh<<d_QQ^Q @Xhnd]F=65..\tfBp*NNoJNGk65d_UJX`ʴ\fDSJ҃ 5][I`@+_P7~~>RXT@/>^s (.1!(o/OДJ6!(\VQQi3  ՕKY9zڷ ٹƿ;]B (,|L"*k w"  I!ztX IDATAߥl}SR8٠w@8f;qex0c]̚ذBG ᭵E̘7Dt.CI۽XR}O>8IP#: ^LB@8%! I$ppB88 !NB@gG452nޓF#̞9'ϰo5cJyF㸇2xT\@^^AǺfϜTsBB kSWiMӗb`c uڇu!!`;$`ZXL:궦}ZrT.w{ao4x` |ݨ ѫG7Sd?ŐҸ/" h4Ѯm'TRZZJ`M{O۶([q,|vI7!)2٣+N3_ڷ5jn/S_W" &8(м^s$I)%m=|ӿ;{vvaV Ǎbk/4IBV.\|5OǓGxzҘFcOFK8& Z~j4 a#K'GOںj@ӿON q !MB@'! kAmiRwgRB؁}KH6?@ss5z$!دw@w K)->_g3mbr4@zhQkF(zLa":` !q!%uF gʏxU}G`̹7l4SV@kԑv4:hNWiu'-`D5bzrG׬Sg g.$oPNX 4 0H{u`CHьƭ]l Igܤ+b263kT|GN 6>@bQB4NB)u,OL9+Hx/-"ػ|x>>ɤЅXzҷR2E ӰPkEp'2&Vm" E϶ d^b9}z=rpqv]**+Օ`rP&$R-9< ZP\R:8g'A~_QPX |ͻNQVx+Q)b䏗҃k:WTWTYQ3ȶ o /]EFء](.tvr}P*u:._Ņ)/ q+am)+r{z:M* 8ȟb yVЊ덟yKJ ˓k<4ZBTp%GV[@@ w.g4ʥtz=ۆ\ĝa!ҕ?#(>$tD0B5&1X/b Djp`aFg1IysBz1ں@)2M#|0(]@1;>1 $/^2ƳeaLjEIG!`9 F@vȱQKn$k!М$kQ!nOB88 !NB@'! I$ppB88'pl&>06|cagaI5ٱ+iS"a:?luߦwI}łWfd|;FSwVP æeg32Oӽ<-GB:˱w?I?d] 6cuԹ|+`?6sb9dvI`IlNJ&<,dيuv3EBI|u[3yyX)xc{V=o>l :F0l۾y/N'oٵg^|~*'~=b. Xoayz4+ 'Oڰ9)aP?żSѰ|zfN@pPEu q' Y,߽'"7c>T=i*IG@ .>8 ck~mw.S=|{ kJdRZZ:O^Je:q [Y3&Tz?:F?eg {fçIڷSM?ø'GQ+ 22N;psPԨeu]:2}+8JץM64mSR{@&?g%!Pô)Oєڇ3?ƿoRv}{IcXbaVIDS^4b9@Oj=h/P*=?w=hB%+c(5YI@ ֺLwp`ދN=+&]G.<:o&X- T_Myf%چ Z6S@C/ih%U^3 /wTzR璗_@6xzzԸ m FG, \{\$!PMnyi4;_jӧrUҢ4MN"##V fHtΥjuCHT|G(F-~a,~gyIۛΞ47Vg ғ`^~IXMiiY+]a q !ՁbRW.uMEPяa~yHa?-x\{ ]N{SqؘTqK5:71}i#f^Q̔{>0#{X!ء:ŝ)*mϊmq(ԟB؉(nRPqb!-#B8& m~.!_@ƖHrl4 ђ7tIzw dd@z3)-CEv=NMP̚vgK]C!Dsu@Vcmy9ZiESAB,,tѻYy(!Qb.7ǮB4-' I!dhavqx^MMV?MOh)Am6v#|H AZμ0 ]v&aP'! I$pp2a5YL$Z|52bq ^mðC˱SV)74.g_5F RL7w@d#om_nƶ;Ա_nV͡SdD#UzjѵHBw!ܿ_'ϰ^3@ʳ_\Lscj{t֙/ޱ('^=`.p}pHs +6'%s)-UԹ|a3Ǎ"8(:oܥض}'\HX2Yl`iw@ۘ=z"!NB@'! 5I643)ćwo!%$N| ?˦a>']k# !~v]mw!֏'.Gd-jנX3aE|m !q!%uF(ά&q],}i~4 Pب !!;M℧IƳd~z)s6f$;ӴKOfmL槯_ ]WBCв#ޝBE,ah a }n +m/!DiXnw dZrO/( @>\tO5y_s2/^1Ѿ zY98;Ӯm\VJxX0Zm9WrrP(hDiUJO(.)Ek|C'A~oxKAAכܼ{ h:EAx)=ȹKi :WTWTYQ3چŅ6ܡ-.]ɉBȺ| Wچj\ kLY׌ÝցhPRzOQܼx+ hb χB J un%Ƨ'*M)$8OOWrh DprJIxX0n\Aоm5MV8IF^զ" R74pEߎGI?ߠ3BIú|Ĉly;b'jVxgE5#U3P"ܕӢaϛIu v^Dt@a lȘI\{-')jU !ĭށ`XBjN^LaŘ^a'6\NF pWWQeZ-ɱAkߜ G'3IעBܞ$ppB88 !NB@'! BחRZ,{qޛ8yLBTEP.g_X)2_b&b~XLjG1=i|^PT}طpOkcq`!6ҳGWfN9}`cl۾b^=L7J~rVCfދӹp1R6)vI`IlNJ&<,dيuM!8q $:ӧ3߱MT\@^^m`@0I[vgďxG,}0DURpLRhBfڔ~9)su0jdj@=iPUځ#66'%mL @5Uv. l۾O$;;bѣ!8(<`4 VI!#USu@.|՘ض^'G0_'NCa} ?K}ڥŀ:XHHX|GS/Nҫg7x߷0odjxzz3$ғ`^nkBEJB";&! I(I]./Bk !OtIbɻ_'5Qv4!>H;YADB$Ձg3fA m6Ava# @cq[#n/'3*:؂$Rz$!5v~x=k,jT hՀZ DKeV NBHv [ՀZ DӸP#eջU .1E|~ ud!4*R;z{E` #;sh!4*b{{H^;eem&)ao9N7l[,Kdl%QߞQh$X# _EB[ί xRqkv"P@^L{^i1XM> =J  !ZnhQ R6 !Z olB{׋ $ppB88 !EXJ˧mIO<Î]5)O\ 1Cޭ3M)Vc#[=o5cRwI}łWf'(5iBR{1Ϛ\U+ jп7m4Pl9)<5ɅY՘ttqtґyy|5g~rՆ^e{ɦӓe+B8 +c'9qMhJ&e'O٣+3O(MT|:F0l۾y/N'oٵg^|~*'~=R4扱Z,hؗOBSfAp1RI^^oa,f "}Ւo-0Og5 GJowf># *WoM#آNW@̐h:ouklS} י# WJcO&=6IXP*=Y .fϜJRinH/CVu~rwoR65&|M)]tdQ rM'p 44bpǮTۚ.^ӀwwIXMiiKZKEnyptyH'! I$ppv'$iE*}Uuw"}(!ح$=cGC=jTqKe@9s'd$[Aj|6C2N@1Np^>φZ ־Ϙ >R.>:mMEd}+DsU^;y),_';&wGD?Fwmv@T|#X'isz1b^,بrgWt;HHld)3)zi #/o`ɾ*#/o#E !!p(wK,v DQTuI }{qH.WT\mKriTwc IDAT`̫+\sQ|t!4KE|;$>>mC!Ձ4x,. h@=C S#|/jхv҈B=yijbV5r@az#}~p!4j`۱Ou))]\p؜"olܱa6B:kt7v1#gZ!H(!w?szc3D ; hqB88 !@SjY4%JOGgkGgI?]@70n'nW[.qE7;W>*WKN;ۛJĂǟďTˇ%L-_FJ^yYgO)bHt0ZS&qېDwv[J4|U~ɧ{g_Kg6vKY,t!th/<[ĢOX/ ?\e߼tioqE Ap|Ӆ+gn|=H+_7̎"8׎C pwASc2d֥+5J%JU.di,Q*WKAa9;͑&!ЄJ4,~$Y~5s8/.Du%z9 jMJ?>g*W˞֋]9 Ȑ`vZcҕyqxCBsL!~6CBY$ҕ7_YF:vAE邗բoMʮGiFMm}{o.ldwXABSi)+\eAB: +P0_'Y4~0^JOtC5ea 55>SIUH"eZt ^ʮ^m.qcDSIE48VerzϾ;JUH"?Q1e]7-jў1#ڣ_8*C 8qbl?q%\a3]kXW Lx.o'`$n+9eL}A u05&V`v}7_iwKP%b386 $;HA`*L2TV)``t"VNZ*du(3ViPD $\r#w<0cK H Inowo~8ᾆ)MBش`?UYx'wylQwYbt*7c6%sw[f ߕw+ @W3%q&cDJ^i˖ 1`$KI sMtvuׇ`Wl -wZ>pp5gyG+eK 4 bl? !Pͻ6c'k1d*/z5\441 گ_"/ +7~l3̝3;& "~1nI@:zy {,ϪKn]&!ƦfRT5OWbHMMMŎ*J̢ErDJdE`ʤ׬\^_f)g;spu4c~$pXb V7'N,1xT$?@vܜ,2-InS:g5-?Q=3Jbֵ٬W=#;7.iV=DKgĴ ZRM[-/n6*ukFy$CGǿ?(G5qCP+Ex+VB} b>Zˁ,W0 "aB蜄:'! IsB蜄:'! IsB蜄:3IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_logs.png000066400000000000000000003126661476650141400253040ustar00rootroot00000000000000PNG  IHDRG IDATxogO-zl6֓!:NhiҘF.dGE" E^ ΍hy,HŀAPvlAzD<'ݝt'}Fsvf^^TS$E۔~DB>O=.1<]rE0vB+lNOJo_-j)/J_|U??]}O~?{YOnc=j=G?盯'?+P~'2|`ӯW4޻_cH?zE_}+ɿp"7=y =J^YoW߿Yӯww$Io~M D?XFg<w?V ǟH)P# LX_ʟ^gNcK}0ߙY0ߝ1G ]?ú>__c>Q;ϧAw8]_hWWzQ?GЯu8?/|Wvu<\hϧ~r,I 6YXxIEs9,:(/R~NJeX]-KꓰaGkwcAwϤGn$}Hs_~|{mrý7$}򱾘N˰NG.9>^x_{Jې 밤FP$ֿ~?~ƟO}7X{EwwAX?~_w_]YrGX?V~ ROYx̿wt51s c"fMc:6]lʻv!K]y@"` 35`&p?= L7`\~O{Cp\ytzQ0XfkL ` 35`&FI)#R#8()eU_5}b*v2Ym>ϧsndLq۩|֧x`*1nGty5wkQ$CX7t練vk޻ZM_t3TbɒJO]FeSY~P-zjFVZULuCٝ^YG'qݐhjN;է$i,ϛjH·KE51p}ywc7MLX҅^cgX_~Kw`c]_Gt=_{/h˟"S=isU Ė{t;%0$=)nd}Mx{rtGEY;\?IV4I<5,i\\ZcOEyO&yTF(yw 8V9VOZ55>RHZ/i1uMjU[ lèB6Zݦ{E5ۥE6Tbp斫l;޹yB]zk_6ﳽBA ݛ?K嶬RzQa! f64䔋hIMm8~=7@گ}BAzNYE瘗9蓞`׭^0zWfYk12sJ+ߞm5G7}Dp}kҧvt CFFQ <!sv=OeIvJSՔ7תfzT=hZh)kGژV}J |2w=ySں!;wJY4;!S󒔐 @ΰtVWWۯWjGǚ>Xt{:M5>/{B' H,]Qb=arEpa, W@Z-}J ]j5%tN凶E>9f 'ͨ̚/u7?A0H 4JJCwsye/T^Ժ֍1wSnxgvÎE)M:ROviZ-VcʖBvXw;%ch}vOY>PXz)oTjq/ YkjuZƘݑd:\#}οzW81ϥk O9c7l9':}թ4m}9c>N鏓B}h&w;%#/yP)vK!<#uCE5hL踳sÓ5U\>wSʪnr 읜}$(L翖i\ٴb+XC#\Ρٻ&X8LUn8u!Lգ,;2k<@_q׆3'~(gTNPDt].1zEe3m7nґK\$a Ö́Sa9+i3k{}a-z:2_,jJה,DKK:jV˯i=w'-ǧz/ [NXaVRYm(c3 :l4sդlmÍ2,W~`qg_ 7^`=AfofmVGFud ӼHh0DZQOlbs>>V,ltILÚKLd7eR뫽6Fd.eųu:3t/c٫*}hKon*6ϟ^X aQ`Lɨ|V&+]rҴP6nuZ_Ak~s sQ[ǃw >;JdV kLhE5`&@0Xf멫)oJj2R*5CRF^![c'^>JXj N0g=vin^1x >ytOJuP'+T踧=hgTbR⪤̑eTw>_Lm8ژ˲zrYx? qJ(<()e 9wX .Y^Y~Gs]'~yc;3.5;yO׽c+ey;QG?t55ws]s;|~oh ` ^xXbh_:՜^~%CleZtmkѵ;*YInǚ*>;N6_ {k2dGrr6 >{6q䑣߹h˩O+$ٱ^Y9iQMWTTne/S~vqFZ;(#ّrgZ[ lj {r7s3Ӳ7suܓ~90(C63Vjm頠dyV4j;i&PteבV]&#kMFsnl]XRv6索+0Os9-k'7Y *r`~a.eUk:ئQv&,`塜?'hokNsfߥo}o!H7vVb?דwHG.Qr%L<5vO酉`7ffY/\_x#nhN|Y2gud-?Qה뇣κ/0ǿ9^{Si]c f*XJ\=Yߋ~hKk2 ;)t)˫Ix߆eu`VG_; x^A ➬ 73FdIՑgYñ4eL`3c#<[YCp>Ϟ1eXʅڙ7F u{&,w<kn}5eQ8Vz),2= T[3&\O[js=_X{`vM>V1'RѱN+U*=hJ%-Z\~%EJ<je6ދOx\T]ZdCGE)V^okvng;7[+(E%7)(HkQgz6 qDξʶ$7MU͟Qy ÐQTe9;9b;_%;ڠ3mY Jٯ[` sGimr8a~C-ǒW:V$o::7U56VS_g3^ɦ._sq=&T\G=yɓ'r03jCRUG+_!sv=O穹e=MUdȫ/wO!x{ S'nf_ _Ӟ޿3]?|BL%%]MU8֗jtE]Rn}6R<_Ad}vJaI aҍ!~ISʇ~yp1$HIm̾V&ojuuUr.g g_qYVO.n~a8xY]FI)còap9|V J;zg_穼s]'~yc;3.5;yO׽ҾLeV) X^C]s_[=5^ w+-Mo4/[هyT pS[OeV.(% 2k^'nd}e j7Qnj &T[7}\TӋT^;I97~vV&gRz9 fenG5I^]RQNAΎYkAg&gGIjm5` ' T9݈\N:P0AqOFy X=am^fb EWvKiۥk2$a4WyO=劥\d9 \NfVdg¬ Ԯ/7 U:,jPT=Cv<Zoۊ\.~cyX'j"rWPʣ}0Y6WS_=i2):azE$䌲 b%W{Ӗګk~]|N>ytOJEu/tXRASRr/tc,.ZApMblQQWZ)J0nѦvMG 3}7pZݙkMGѼ6MSհ~vT^0ddex$NNX'$ogo(6hcŌr[VtE,HRV/ïg?C$QZvܭAݑoXJJB!ؽ_熲jjVnf[d`._sq=&T\G=yɓ'r03jCRUG+_/o=ov~8wMZԥq/kqžnk?O2Yf*dIW%'d.,?8l=NQMy#+{*n躡NNUޣ/FIr}Ͽa+qgէ$p{<]ʀ{֬̕msk/c~Ck;GDj߶_h.cMb}hGEY;ٞW?vvh킒a QRj E٣}[vJFSRMTR9TMΎrI;jܽrCjS~#@d~jrvV36p@܍崬 dd۞gLi/ʐxC[:((fflml)]ud/nZ\=8[+rv6索+0Os9-k'7Y *_xj&תrZ\F@T=Cv<< `מּ7vVb?דwHG.Qrt);K/L<y%7v6sO=udwg8rye<2vLYY˃2;9F :7J}pYF7Q7kojܐqϬ9vEb Bwm,%Ĭ ?wlr r9jR6QpΕL{v+AܓUdՑgYñ4eL`3c#<[YCZ~g1+ra6vBal竉(]A)Z8~f̖q>RX&q۲C{/$v gM-㟶^%X{M.Ƒg `}szpɣ{T*|tS}JEJEa;yNJ偎} RQQy\-ܠqݦ{ jKlh(+ b zu u?ecXMG 3}7pZݙkMGѼ6MSհ~vT^0ddex X'$ogo(6hcŌr[VtE,HRV/ïg?C$QZvv9sPyK䕎 ۅk3B{7 eUԆ׭a|T<=sJi/E9*.<ϋ]ry!uתm< RKQk\Zg0ѦLkL ` 35`&FI)#R#8()eU_5}dMq} ()[өmoxXP'oڄ UJpSq˒Z]GSJn:meupq, W@Zͤ[Z]]ͷnzթj Vp1/Ye'(rY%uX*s<ڌ\!J;zg_穼yHzRT}dhjZ(Ȭy^@ǔT)vOmPqQM/Sy%:l&yM픲4\eÀc${'j<Ue)opM2w=y^U9Y~c6v2>LDsP3Ux?*X}K)PW=dǺw[4Q/zTcԐ;eU/vmTTrn&D}_r]:*,8θǤyU{O*{U4,5GJϣt?n_yUWY^-';>ha<`L>yw-qIV}\rJ/iVMΎrHI;jܽr+m3ւ,ATMΎ=8jfSNuUsuPW`㞌́)E٪ٝT{hK% ،-UAu31*û쥴5Yk0'>dλQBANtDs? ҇ݫwf/( J9U@JQA/w11ῢzlٜr>hU?}?{6omg'Ufڭ:rx;7YC9x]?NN:՟7^O;V}Or6R~'v\" Kxj&잒n^M<2עl f{UuB,][KYTǿ6ƕg;s6Q7ko9:>yBsz ɖ:صU8zV? +Ж,efX!/&enRnIRQQ\mݦ{ɨ㢚"?*J>Ŭ z禁 u?ehs#d;loֵ(3w=UbFѼ6M3* 0 EeY&ɼS.VI5ہ mQn˒XQ+en1z3A2*Uo3pCJ̜_/b˝HL|~Pe=TdT9x˞YSpTjsy>/ψaJ?0'Rt$zE/4S%K2GnQy[TAں!e)+|@ajL>7en& 6o&3qQKoiuuZ|U΢.) fuMsX}q1_0{ d}vJaIE8 mVbjOJ4P߆o jp{E]>WdzԆyϚ.'u7?A0HU~xWay0<^ن K++}y̎^Զ;ޙInر7I@)ΘBAle?l9*,oebYјG{=8|Y>yxɊطUbozѹYcݟv_;56ԧs>~ 8#d^kzΰ7kk0;>o[LDsP3Ux?*X}K)PW=dǺw[4Q/zTcԐ;eU/vmTTrn&D}_r]:*,8θǤyU{O*{U4,2G cv:J4,iwݚyxM-.'}MP?n]4i/aK!Uc)27rҎ$w.g_hm`[G⌬ wPF3U#nCV36p ܍崬 ddgLi/xC[:((fflmNN#{)ttMF֚d?*>ߥ$nPpY 9rjϩW] ,uQc~=&c3wE^9zc+wğ0U3x& Xy(cdJX5 }JEbm7n 6q}a Ö́S"a9+iYf,s'den5Y+(zV\GXKY@:jJOj1Q7koj\kfZ\]} ˃rR Wjgb5jҚlX)9}y5)?pl͕zܙNGPC'9͌[#Tz΃yw s0+-EYnXo7GO{?3{s>X %3o<AaM{Mm|PI)zw|Wɐs~K#o+g${'AhUʧ-[ ? j}> Wum8J]OյX7A4쫪hԎ*,ρd)+$rk6V(en_x~= Ҫ!%fίߗk yE{$Xn&>?2{jrJ+߾_3A&T\;V((RũM}qԔ7l%{̄6]`XfkL ` 3Ք7 w%5JJ)FI)#ڐvFZ /&-ݧޓaN{^֠uNհ|Z%EÁjrv,]FNqT,HmmWsF֚giבV]!#kMNcI=9nP*>hEgiYriAKţ 9^QW 6`~F6xAF.jgCEd/u89aɄ.T_q/ͧ;jgv~3w -%*Qr=yUmgW;Ұg(9cVZ #߳v(`}c7'<ӷ7\dفճX懶f) 9}y5)5(kwexȴkO;5ܝսb)wPW=VOّr7pGͻ֧RD]z4Zyw^=UC[z;2%t34=̞u0x́+OFeҨY/Lt]M>V9Vo=U*U>:֩aJMI_^T~~K? \8KE%e2rva2wJEelQQm 7#u斫l{CzT0d,ԕ`Z}7[(ȑțfT>J Mdӆa(k;9 ~xOxN;,79S?ϨOCRzQa! [okTAk-IkQ01vVb ui$U_.tb3<o25br*2"0Ȱ@0XfkL `=u5 C]IRFJFpQRȫ6klCEV[7dX%Ϩ?w;%ںg+~ٹ q<[=Wte-Եh>PaC]--~Q2S%K*i>!sv=Oed pjYkUyw3G ewrze>r}ruCv;_g!sד7_)oJkc7;L[)˛ļ$%dB܄/֗q~: fnX6G;M t%- )ݙSB*K܀J&}46Htȃ$uO-ƛRUp}<ۛ")by~X-rs+\\{^aw{q s߯ݲ u/vXhN5wTznmNA?GDDDDDDDDD0^Iktg׻KϱgF%:?}ĝz˃ 6M/KfTvQ5C U(.mKXڶWn)oh3ڰ&JQKN`6+)?Z.mzeQ"TVB>`nV~aH X@ 7IǻzVoYPCzOD)t+He;ŸQfrM X}fc/ X, ~=pppyX:*oTض u*jl^9mP 5(/#mn9wNy 6P? we/ T(iA. ?8mvT`Y2զu'մ'i&j &ToYmJ oPilPgg)9Z*F܆mh-`v,TKSׅ6uwt 7;y߉(,sߟDԨNtum kka?ڄJC˫<_Bi:5`(S 7a0"h;Y}L%*ef˲ߧM3!?.:Q) j6 bY&ARVJPOhZ͓Gsu.?Հ~>duaɌw""""""""-`[a7}?=x pY:Ljng_QD7~T [E8 %w""""""" S7sZNʀ\/Wro>WHGTEѰ8c~XWQZC5>IL?Zsv_xS}X>LNDDDDDDDD+C_˷Nm꿟 qkU9.O7lb,ב+ЌA^:z~MQjө1?lwoi bh6pTEu9`}w9֖ V{{9wZƧO\x%Az!}w9aoDg/YDDDDDDDDDDe#`}ObinQu9`}OiOj"""""""""" d X~@H)`}s?u_!-Qs/ޕ{\ϸ_3!""""""""".g۶-: 5yfx%Az"_.=}_~yK;Vw>f(3,>O_s,9!S2e9ߟp|{{kgmHh*L>|hfe X>VMGwz{9`36g*S7qww}?ݹ9!aMDDDDDDDDD42԰>G;`!~]wpoi wX=gwx4Z;ְ&"""""""""y9۶m>՞)1Pȟ`sBDDDDDDDDDDQ2dX$ tXofhf}!DDDDDDDDDD42=th&"""""""""5h&0`MDDDDDDDDDD3F"rr"n MCζm[|ÇxA.ޕ9||w_zk0'9^t7-`ӿn wcm)}_`̾Wz%:\]S?DX:ռ2j$H5:X.=}ǿ;"+7gͱKDDDDDDDDDDk%:? #oՑ[IDDDDDDDDDD7%Aԥ*ó5/#'i3.Vфm5m$?ρo|ka k"""""""""+C9՗ocsp{KkxV/ŵgH_DDDDDDDDDDYζm[|Çxu'[D+_wSh2ܬ"r @}j"""""""""FeXu#2cfDDDDDDDDDD4&"""""""""5n*r"ݬ+0P \.w 4: S?jգi6@UhOnÐnrUKO=Y_w"""""mC?\:K9ƹsrCin1X6,uzG H…}x d $:?䨚|pr0D)$=cT+;OB]&lZ+aö-eR+O C%c=CAL'ۙ[^h2F8qϝD .moM=up{1rM`9<񝈈f9`}w9?,.7#:m/d}aW!:oӂ\\c\UQVk(G A"#mh.6"*uM=/*R!k/=K4b jYmB0Ķ{m3It{nQ'DF핞z ھ.㒿ֺVf[蟪?v&z+{e/L^޴qhB@ThX0hc> 7- UvBDDDDDDٍW,Y{as`WN넶Cܿsrf2<s~5_$j( ɫE?2dij>66)(01l&# 5WbZp2֜`giEIMdfǷ'$3oЌ`?y/(ٕS u( 4}.Čn\q(eb]^ Pp̼A9Twݶ#2GNۃg g8@w]vZzѮh~T/0Z&\cXb&=GUT5Mč:ٲ{l3<)滣 v2I(Ւ)q@Fxͱw"""""[d%:?G*Ⱦw\.WmZ::UпԼp)(nx\XZ(c2AKp]Z"ͺGox*+=4W?#6>,hPQ Z(['Nf~.\΄*foP@ζFqZCP[0-hnc먼s`Nֺprlnfu*d(Y +Y A?v 7չ'XVbn`>u3uZ:t/yfU4G5 &vB@X4c<zoB0=M9$cگ3@۴P =n%g;U){{v[ mqe ƙWJ`N񝈈v|k'>&d_-:F5PU#lu6zy]h gϼ z8BuDK),=Sm׆N/wjFVg_5+=52haQA rYn2rz9ma:3p1僠Ǒ)W،zs?Ȟ³cVޓIpH&W,Oe0Rw 6 (W'~^IJTq3rT͙3ֶ>΁J *h#vJ/t#8v6ڧ* q5&AYOǜcG&JTo7ΌGE]&Ně ,?Z񝈈ԁ}-2}Ÿ kV_TvNֱtd$A.K:@5#li p<ũ ,gAFg9B$b,2=a`?$s?ZvT8L=`<A'hC6ee4CoN֟4.﹙/X}.t v=_򀵤zlV/CuʈG#J7Xx\^)YMޑK TjPhPW٦ڱU|N'ttQl1t<y>hQedLL IP;rQFN#HܦM'l_%n h4'\2g%cmd|fÄ++N)!""""" l۶>|\W{mhĠ2Dc`5 k"""""""""" HDDDDDDDDDD4i XL`fDDDDDDDDDD4&"""""""""5MtTE.WDuU5BQ_bBC͡z4F m aY q[ɴg1+[Ư){| s߉h|{簃Kg):8|!: IDATFh w@ $^g. :G mHtڛpvT". {m&jϘUyă86a'؜ -4^ l)Z>7m . b:?IؕeL:7q;68\N2n{z;=\?cpMc@g\޶gD^o@?Igun<Ԍkkk⟁{OqD\LǽiAE.] CGU@ ~PBEpZQ΅Р]lZET.nz_TB֠_|{ Η@i@3IA ղ>ڊ`mPG Uwf$!ݢO VcJ}m_I?UtM޹QWĐ6Mwv ] ?5nՁm}f LÂn%P{V̟n@ 1""""b ` s+?v ϰKy3j| L9ot`]/O5`te %ȠޒB؄Rr֛ۤ/kI@~J6PUaYRp]P0L t- fd ++xCum7b&9*p,?k8cpYN&}Rjüɥ nw}yl"~+#^̶WuӼ稊ʾutuǞ g|wtA=aND `:*Q&TK8WDę9.m5gm}W:&ڭ:z{ۊ-W}DmS?"""""1֗,b+!z2]r*X_j^ d7hB. }-1Oap|Dڭ˖@u&3}9m{~0D T[B_:q2s9r&T!K.6ұOmvڂ nAsgNQy 3Gu?)+4r) /8rzx hFBЏ݂MoGun yyUḘO`֮}]ΛFfiD#cIjQa'EϿ8/k:M #۴C9?{93 ľͨو*r8~{qo?&LgՅ6Jւ]qxLw+%yvҲc6v-Xv,T@n>=l`_Lj5r=6>˷&~ s>KkB" %R `z96zy]6*(ᔏ$ LIj_:ܹЋ:6Ϫ0jW5o=e{7jd"@nA.e9yaÎu!/f#dc>A#S)?s?Ȟ³cVޓIpH&W(Oe0Rw 6M&J ~%Cu*| YndEstf>XFT8K(M(8d\)zTVJҍmvho$Zț姚;LaG&JTo7ΌGE]&Ně ,?Z}(1x(z8ɴAە6߳ FR~{%[S/`9^ ~ u2 aJ, Ts:g5- A8,,8T\DE&g:L7Luk? mGÔCDq1tֵQs|zAG?3˶ӖIdb15\(鼞?8 P 2oƈuÿYĺJ h~1XfPGS}:6MԎ-s|3Qg "LB5Hɫ O"""""<yQ_ݶQɨf=Lr֮=C5݌ @iOvb .vkN4P5'8 p uD/!d +/tn_7ƣ4nn|6nZ]Ng0 m݄94kҭ.LvjP0҄=Nm&uo3=ʳx7߬"*Cޞ| GzU em C1b^-JI UQ'g%:onخФ3:p$sj69ѭ6L)c=Z4g&SX0mmS^?]xƽk&eX62 aybZq;^ݢS78( z8 (OLOjJG<{` rTs>nT:=AyXF!E !NRE"=8]( 1m&!{NM /+JOCt 3nh^E)uq^ctE*R2-9?c¶ xbZދ$4vf@,_^*oԃ3hN(8fSKnĔqo/eRw0 C6l=1dĉ8p?KX⦀JsE(yVB[8Ng6Lؿr*BŒ> Zǥ$>4D:ޓf2e)9&DDDDDtrm >|=D6P| 4Y~F &""""hH4{,4^֠ g!"""  )=FDDDDD3h&0ÚfDDDDDDDDDD4&"""""""""5h&0`MtUf]jGUr9kMDPmPsMc{/ڐk\ 㚚7xlC?\:K9ƹO)7]6,u{ #}тƆލ`u7 %7:ⅣVyfPnLrOj}IBͰaɖ2xsS: CAcG m| ]Yo~'Z\k^X;5gYc`g??"Oԍ) l$6q^ODt͢wcmmY3p"~>B֗_}WO@w2 *z}VAp訊¶(x\C\,uT Cb]nh.2"*uM=/*v֠_|{ @i@0ueZG[1hzm{m3It{nQ'EZkb~X蟪?v&z+{bDe\ Q BcWoBADELK^ PُXq+$~&Z@qKjOsu2)y=E?Y4O*ODDDD$H5:Xij pϿݠ];-f2| L9ot`]/O5`te%ȠޒB؄RrۤgL#""""%X_XY~{?No{wkW\ 8Bb Z Lr?u2,hP\x ^ lNk?j &T_~Y\K8vCg:Y~SaWiڰ4);-8F #P`*ciQ1pz^^9jn`>uJuZ:u1#pT4G5xHo4 7*lS^7cx?afhv}_6!nu)'c:t8ByXv_8lR{}m݂vgto0S@qMȢ[9 x@cmo}-`XH~0;GUTleq48cم?iYI6󏈈|k'bx}քkϗCI" %R `zV6zy]d6.( 5LIj_ZĹH6Ϫ0jOo=e{7킛}`t~w LbuرN  t4(\rUͨ0C2y{r? niФ{Ѝuǻq` J ~%C}u*tL^m(ӒYv igm}*%"q{nH7Usg~00)cRPg~\GmqFA;V('7ǙfH}76L]n `9R/qq8 lp5~ʙM= Sظۊc03OOG ft߹}e'8X~&yn`M~͚lOр}|'kق}- \dKٰ0L#vtϒk*[4b4fq*/Y{Kv.A"}vbr{ث! t|cg%iOt?b>*%0uECEz6LpFt^Oz^Q㈗]*g]o!J h!-7 S r(/[| &}X ٿՅo,I. ˗sBYkX_C*moP Ε2<y],=k;Ճ!NFmD0+%#gx_Z:t)'\Bz0r^On;dOJp`x kJb =Cۓ^m8[ u=\#zZ[ET:5]Gy)3_i`lkZݠ˦LOW3pkClrW?]p'5tgb푶0PŻfQ4P '?G̨ nl:P(8 DmBiEA NaAx߻~hW}?t+vGK?]!m}:JyuniР|KQZQ  s_c2i}e'cqKLߧ""""2!=~>=sqxU`kآFNk(D=hU \ 5#^F0useĩNk.ǭPǷ'(/Q(ϸ𞅚4y,Atim3 Q~71̙Dst*Jz( ;.ҕVyΣ2=q:sg>͠^T=zGo8ń:rm cQ*ݤ}Nt }[ƥ _#?#A}wMa2ĉDf|2vupQBE~rֲSV&S|Zc#34 ,DDDDDٶm/|<@%МDD_u#+ r:/ve"""""2?thXhA(BDDzB3fo冔#""""ۊDDDDDDDDDD4aMDDDDDDDDDD3k"""""""""" XL`fDDDDDDDDDD4&ht@5*rSܵ&P[(6PL}Xh9Tl۴vX08ssr>Ku7Jy%""""˷888;ts(.{@t6,u/c\lz :G mHtڛp1zTb@UhSn3IT{ W\Y&ߧf >VhJfذmdKo)P34=+˹uo=wZi?ľ}cu-heه T#n>6P? 7cDykT{ {o2{K1rs?=Y]oqGt]ov *z}VQm-/P𸆂@\DuTrpQn64t~nۦ)5%P1|j_V2^[ꨡtLݞ~0[IpanV`Ze/@}<*򏮻;7{;*r0Q{ӄ~g۱Pa3QYgv@YG&l)/[nc#oĬ6i^ci{<>#~&25h53vuGD}Yy H0XbmJ p_7V0VyB3Oa۔;n,:x 2D+I_o sz4G&8mB &tTCBylb?0CD?q_>a@j,wJk%d9m3~ {Y6H0FR\] {ܒ!?X}o`qi 6z;@mj?^Qy 'nh0Q'k 89teMʈ Q@{Tf1دfd)h-hvTf [.̧nrNkW.fB4'4 7gިTϿ8/k7<Ӕo}XB m̋Jeeٶ !<"*Z1Y(ywto(tǴ}Xg, ~򹦣F ~bfg72c3 T(ci vA 4_ ˳:}T֥YY-o,w4oܘQQ?sM7eX~kp@++Ѝi -o]tǙ̎L%q u6HcjSUh6ZhT3v3M{e{S`6#""""@_ox}/pZvw鯸+SMZE'X R:R6>LjNG,^A/`5fp{" >?1AW=a`?$XADeh;YiC_8x !5K&{ ]WLSO:'ANmuB|=Tzqě1"g]Xm>WeFPVJ)ˁk5AyѿP_CG!<2nxegnsFz8B5OɫMV"""""y<y(\scu\>88';.E g{oĺ^=*]zkV3e5}< u2Ο _:p`x k܊TeCۓ^m8[ u=\C XUN5p)y2ƜFx ֍:|5z s4)jn tsh/Ē-Տw&5cAmٙfb{- gnYET=<ɏ o$b3@uh{5l'R(ucyE1lsF"~դd3\7{Yww,q%f[mRzniРe>!-BuJt)m3mr""""2;Ƨ ֩S}ppÎv=`"`WNAo/ŻPS:KmSPN:e!Ni~)+=Ay89E AA} =T]2;m&!{b&@¶=~9h]E)u]yұ0"Uil.3LZm¶O9?Zhҡ6l9R&/|nJ}b;d"Ư8nlL# q(D:m7Q#[~2QD}9h`Viޛ66'j~'Z憈Kϱ,dwOeixH۴^gu`\UQV.P/\k]GEJ8wA/ȶv\6PHY~q+/҈^ סjYmnj 6D'$.}u-U+[ys"-+u2BTEuc6yfoG[N?_&j/V?>I+4vv,T ,4 `dB?՝޵ޤ썳e8nB]GDDDDDDQ+ {0B?o7p2;"e u>&_7:ioA2ʫz*pя *Y-i=,M+%M J+*Lu2as+[r7-h1d%e:ҊrJoOR?H@ Y4塞>pnEe6 YRvEa/z*ae 3o:{Ю}F]-߈3,Oa^k\kNkS9Ve3=hsۅ.'' ӏXo 켷8E6C} 5QI}3`0[<Ng<.{ȄXCOu_"""""/Kt~Ks1ޕ} \57 Y::UпԼp(mڋM2p,vrRyA \/N1O2gǂ~e;H瘸yBmӟ"XrV**071[| ㌲+y fŕѐǙ3h(c  VB%8l~fVQ7Cpĺ ЄDV1|k'>?u`xt7iXE<9g lu6zy]Ȯ ׾NyLmvJA눺RY\{@ rS"npǶ5?)ۓ 8rL/e2Q/3`@x{r? niФj eV!uǻq` {J ~%Ca:U-&%df!Tϝa?vP A"Rgeڜ\9 S­ۼ. q&8p\~QBIn<2/ `\tfhy㶜Pyvԁ}-2S?}&eRWv܇1v98^MZE'X R:b 0 _,tϒ3c- a8,H,CSBT*A"}N>3^ s;ʣOA.CQ0D4,4^Q xثnи0_ 6@v:ۓ? ;p`x k܊TeCۓ^mD쑎Bpd9hО'O~-BF꺉quqSeV{nge>%M)K.pKjP]r&~4|Cgb 4voǯpᛓViY>,}ן\YD3phU&=M|87ك_J Hv6>G;mDiO큛I}Îl7:pZC!ADMv, math 5#^g-o.\ \Y@qk: KHKqQ?* $=]=m&Uc2ᔭS%x*[Ml3[I0ZUR(P:Jw\nW޳(} k/hwOק &iky/dB=ۯ-!卺_vF,l9w.Ug&J}49;e"eeܛ6QQ_Q<+|?Z;wJyϩ,37qrV%eٶm/|<@%мAe"q""""""2?thXhA(BDt$F)L!""""""Y k"""""""""" ̰&"""""""""5h&0`MDDDDDDDDDD3k"""""""""" XDGUrE4YW`~Q\)Zh(eUa\wCfq b<ˍ0޴IDDDDDy|{簃Kg):8yqˆHs.pjR 5Pz_.Q ]tnk s 6Ppڛpq|T HDL՞Uw够`,X_&lZ+aö-eR+ #LC^>:iܰTN`M'<Ue43!k |?(Ȣ玱guэ5rsMop9dA2stM}fs٘HL 4 m_WO0LoګdX{3` gQENI}/,Jd^_W={,{[3/Wⵟp9oa`2&yطI!>|A=t<ОGaAKpJ1Zpb#@4A%M:Ý+/f´j5xZ؅m:n Ё3HNOf=bm糙h[Fѻ 9€ӱmsr`!'KCE|цaI_h7Z+Z6b_3l#\2`l8?B&  X 2yK<7'xR.ѷMS5 :z0cb$JSe)|&:cdՃ⸰M`:8$\ֻϭĸG?s92>5ꕜ뢊| ސBhz\٩ m ~`wyZ FI.7PFwUձ+m[.:p 3zg6G'8Ј;vuOh'cE6mLL3m#B3Qvj@a.䡢j>+\\~ׁу >7 G &.})8SFwEw̪#g>'ń(vsl8- _#3G*ptsY`̆3Cedvr` o߉Tc`v DSh+R`ʻ`c5Ԝ`1`Gk voo+7{]Nrn%|@>%vi$@a˂-<vZrnUǑ=a]uΡXr `a 60Ll(1J~#:HFwLEni܆9cN`ˀu0zl)\ ٙɁVxNBvZ'MWiY>쩕9ܹ3Y}çLOzw{N&%쇾wTJ^Yg 8En1, |yENOvoąwu0,3P'K*Wbm4! XбE=} 㨕=zX;pJ\%;Yv` сӆTge=X\x 隰TU]{ڮu |I]񶼦KFہb}Sl'%|o5a\0=3߶AAATsX| _x}w|'YP9_m0ձdDi"thڅ]s]j}ς*#q\ũCېU[P',TDRrz{؁l:s?tʄu`=!/zP#4wT/+ȱC*f3XZ1cƪGXPqj/9"б܈+1M$ّ`LjNAp~|yN!wcy^@=?<ܴ]02ʠm#   TtX_(G;\Oq?>OlMjOL CWw8; 5f5Wb=Ws<."+ꈎX]9T~~ knE ͒Ko{78>.١?_Uٛxe&2[2zh0#. mUAQb.GxXJU(tX!UxBh'Z-8ve=3mtx7`[SzO*Yo6͛4O7E☇3)Jpa/L:]`@JZL%Ŕg/!E ΈV=vf"H+DG]L-yuW.)s Ձ{yfUFAAA0jl&~?/|h~ q)y   SEX=|_t³A|(,SÁ   CAAAAAJ@ k     b% 5AAAAAÚ     X aMAAAAA&     VrX}F@\.l'6jZ H(A|ب}жj;-$n$ٙq\;wm#I=)AAQSa+.]+k\7H< Z I|&Pg {nz7cB~b+M3HJOU`aYZZZY)f][ }_:f3w/mJuaG#'ȭ{}?t*Vf/;M x-;VodgvqeN!{߻!J?U!wW=/ąXG]؅L@Ǯ7ЌeTl\md2  (oٳoϿz_~HpYO^{Gw$hhY=?:sB IDAT/GVhخɁ9φct w ܝ`h1<`ۇ.:p9Zچߍk_e^.   /$5.ԃI=wOxP7 9Em3ia==sudnZ@0an fT}1:`l5yt4 x*baoZV7Z; ?-3 S'DžmȔ QO}n-%}4>aѨpU\U;^|B FpWۑ1Old(h}4LD |3rzS|L`6a*1.؁ȷ3)Y6q}NrDYn0FjwhOaIyo}'#u!REbR󮞱;#BxI|?kE[h7T,vdEaˀu0zlH F~">9ϋS8]?@AAwNe77HvW_%3A} q7x ϞOsWo~m~$Km \u"8fhQc_WEa=BvZgLOϒaO܉t")RO/?>yɝJ}9xyy:t@pċ bjGZl:dQM\xGG= i!ENlMxjOYCΟ]NCY*;btD*ꈎX9T~~ knE ͒KCjFL~TǷ̑+,\1SpM㨃cZM0c$8+7n}JH'3   ט}]OoxcD㳯 <%g?**db[h8 " Oߢ?'hrrs9gRD%Ñ Lß;yPjB,뙙oe ߒP{r5Yxw.hnbN<Y2 cΤp+EeQZ62 t0W#4S vu W"MV},4'-uz,"TK(Vfm.z-u?ը!Kуv|p&k8/LF46j6\pP#4,nxNxB- C=c0KLC;QZz{TSs#  b9fL~_~yW!x (AD*uJ&1}d   > ]' 3rfA'Nl3əN   cAAAAAJPE     X4&     VrXAAAAA+9     AAAAAJ@kب菫FV ] %V }#5ۭ. wOoϺs;cuoSȸlme:ﾪFmg5[eIqc{kH7}c:fn{EV/߉wo^X{uw*._^:5^;>I8ҤPp(Őg9,(,3MUfUW@O~A{BgY9vuC,wٌ鋺GmRPꔻ2R_\ޛI!3o,:;%0hwCXs> ,L b|~`5XMOG6kk8990}NeJ1_I{7Et츏K`xh]Jjt}O}Q"!Ӛig>ɠQG2l;h 78g؄6к4MлiI/==%lA,C6r7l6AB݆T.Yт83z͆\֧Ͱ. B…ށm3;P6>Gw$hhY]9k΋HvM|nimw>,s6a-Z!Hwlݯ}ޒq?qwm!CXGNŸAKe uNZLnciqaׄG09ӵѹHmjYg#3gͨ' tw'\8%>%+sD*K֏mE5uN-f8o޴g"4o`.`1 7We18_s"ㇴwIUE/$5.߄N'x"8??kw<_4Gw65 :d}$>0KWP,%"봛[A)p=XDW#50:xa$CX{OÂԅm IgO60\CMk ~r>e*}4j6\aPZWwG`[pࡳ☾m6Pqt"|'U!7PdEӕL+m)^a r`{^,]|WwddĤp@5|͜ISq[:zJQ/;#ٓE؊ Ȓ}Wگl=B]%OFxL娪ءUFkXLccfGqu1]F*α3je}<5} &nFYg#5ganG?ԁ<^tūp΄{mh}JV))ƒwɟ7>i󔤐tgyc؜ȼ)tZ9vQE'!u7wp0M}ōU~a[7R$kޔ=-ܻ O塾OؔQWZb25^ u{O|]I89Юp.g“&lMovq,rAxw^=s܇}"䢃&W̬{0fCX~jԭGZ/8\ ^ a*Pɳ  SV% `ce9',:FG+ :AV0?Tq;Խ  o TrO1aCN<8BXGS7o"},C ljp+ :wN!%|DxW ]}ǗpoZ6ғ :!:&# U`릥 bHFZzĸ{jA)#擑t]~]LDe)Lm6/6A11&/ N1Ui2WPcEGZ4x?dúmI;U~P&דhR>.f*1k3V3RrMo']2\Kxڙ;b[0ayY柎vO\Yk: B_*|~~ĆU\\^VTvuܴOWdJM,$R,xTd!lguut+.HƝ0f t\1f|q]eMzʺ<+mMbchA@ 5az3b }NE^36C%w6*!g14a)%qXX'(a:L韎u֟~?2Ps{s J;%u 0%Bf-zZ ݯnOs Z53GX,UvT屌>~F$Ra%|OG@:`(rHXuahWٍD߳`+Uj ^sK ALYa^apC^ A A2a (Gc+N[5dv^4b#lYe2)ndPc|/doY]-!l.w]tP_S3_ W$`$e* c"c3E6`)Uu͐lFbQb(&'}8Gbn}c-pduu9CJw_ۅUd >'s7W>Z{:o|kΐߞ-$32!6⟧sZ3o8Jup0v4t/WQbXb)pyc*븘BUWs;I~KqY]PN+U|b_%G%_=Lp:Ko}|shZIZ9WԞ-.Ϝo`C9_AUl9l+=ՉVg\kc??6  V~'T)+*IqRӂDs-癊Sp֕P DWԲbVŐva=x[t>[Zwh/nØc:zͳbX`Yl؏b{ ΐ$sdeMЅK':CZ"NbWl]R.N1ifm(vtآHYJ`3s# GZl v70Gu/;bVLƬVw6A(B5|/~-~|exI'@1XF m0=|}Ѷ81'ZbhktxMA_ߧ#m`HA[p``"lR.n}P&=Dv 1o[7=w`Xz( u907gB;*.?eoQ}ߋ· =~Bi gtRŸprQ,Wb{GJVEIꞮ-Sj8%LrnEB~C(Dm6~|wq2Z@ng!6<)pߝ=8Qr3Q1+ĉ}=~GBZ|gSwJִI+`6;ڢnҁ0=}Q7mSj>JqwxU2F{ 11X${dDu/͆%|.ZbS]Vn"};9y31T8@RWMH,W>eƟ9 aub QuZMk%NAqO(oٳoϿz_~¥?S|Ͽg00NUطAى!>|A=ОGaAKFf,0 nz}/5]`:j4{ <:9@F -f331T82#59+wDhGN̡PvEE:N3lmeNAqO/$5.ԃI=7W_m?Mcm3ia==s:CM 8DŽ)N5&Ntt|N=[M&-hr9=B+y1J}Xz(Kͭ\8,7I'bd{)oHq1HHpNq3gm9}4>aѨpU$8;s+0Cg=I(2 ԀQ߹;5];KRc ,m8ysh[.:p 䘢86 pݡҎ+w\9XBQz\V\QzU(֊7>/Fh~8pb\#4]iԱAy'ƶ)nJOž26WSX{$d^} #9Ŀ:z=ic:ǀs&dCc+/Ce#3R35ulCڢm!%y>.&7iG\gƂv(;/vHyPx]9{^Y t4?*HeuNAqOa6\_&5Sg?1О}_8204a:H;Q%eZ4O-E1L/dGP."DŽcVX5=wm& v%(;S`^im:UYtrp02'Kᢃ&W'{0fCX~j˫4ż ɹk g\E7U]iKRyF)+gG-O)5 PhbɁ:ܝM6#u"c`PbF 1Uw5o+zG+ut"<^y->[Źjw8jOR9q0R bQ ~i9vc#BQ䠩(:qй`v4r.;ڤ 6V1>W=RmmQ Uyig"֙*>nB~56fq@U89/kLEkv܇}Ĥ8/=]8vk,jG#YalY6:v$6,>&KW=foN2zRm;AA Sa}pKgſS?2:`!!J0YX3p0a@*;*|R% Jꎴ,qʝа- [ dw60]W-y6_W&ϑpY a- RjaD"3B bXyH '`(҅w$gaXCD;k4! XHѱE?} Oޮ mAK|eD%}wbܴbYbm ۭfa<q xi q1O2^.i R؈h0[c9hY :/S5!|U.'6j&0/4 bM {|>OGĶށ'91[=(47-^Jƛ0kut..+ZXkYQ_hcUEoho V|_LaUt EBș!':h;⟎;"ךhnx˃"/TΗr9;AAĊSa%|?ԓ.^ɟdg5S\OOͳ?Ƀ7n6X3E4 ~ Oڊ®9Ю+F㸢+=GR5-J v>9/0aQ9i!:L'L!S9/Mf)售@ 'IVFxŸqS2[ʳ"б܈+1M$Y`JBanŢ*J:/f> *ս\w$CT{ӒɶNa{`uLuƉK 6gO͝^A{C6_o3#VY-hƟns̚qwҮ|6,)"*p 6D ':bH/D,`13\z>ej߅n]Bp<͙6nd\/^q[;}' TrX_0G;\Oc?'*栖KjOL gΟd]NsrjUASx\S<=bXS-q&hK-S0 tcX(h' L5W<_BW %7_EP21Wvq w.":v![ F< Dmac r@5սusl._L/bA+ 0 ITU,O}`7K"[=u;QW; `qr^g?OA;L݄':%]kImlMGV[: Qmܟۡg\`R &ԧr`]XmΎd՘l8/,sX+_e)ߟ(i|NAqߩాj?~zsc6?kƏ#:> \V|W OB[|^Ge=9V]]ݓs-1IOO^B:#<|f#mo{[tpM C=^d s lq!=X@_i '+.F`H3y*l-$ÙЎmR H:-%aͥ!QxMecܖHݵM ;в^Qeߤ/vjZԸI ~Q_"CYT :rEXfOXUX Pv^♧wnZR^D!,, `s9N/ 3'`sPY W:ɌPdwU aq!0wz]o,6K` <HzgA^Fղέ[ikw  Bm6~|wqDAiu27lDsbvf*L#  с|Fj kla6}[>=ČAADHaMAAAAA]$     ECk     b% 5AAAAAÚ     X aMAAAAA&ȉZ \OljP(mVuB2`i `wG0TYf>12!gp%E9SwΠJTjZ/Nfs->/X ;u{6v?w_)uj*6YF=OX Q1 `.ɋqlR>uഇ yP[9=_g9   b(oٳoϿz_~(pYK] ??ӇI0v8Q1 &;L7l;h 0̵&z~MiMKWu;^͒!i=raN Hl:< \ 9sF3tb~ °)lIu$69:/"vf10=t&AۜQ]:k'KhخɁҝM>F pBw?`hPFAA3_Hk\)s:X:p M՘ժ:.#Fxzn(̱:cXp+QhF[TUe1sеX၂6]5럎hvFFݻW(Y*Z`eƱ1R㊗amVۃwE# B\X0Ew%)PՅ9Z6ǂ˩rw}t qFal%x*Y/O Cm,x#4XsaKjq]S΄${Yɱi>YȧXٙ(AAAטS8, HpȢA@>-P{Ճ!: L!*s|˸t \VݻObp0Om4 +< –BLhGebY6XA;aKIyXP=خx\S;0Ayy^'lFdKdF:-xmnŖ.wY@/fCgy⼳U~f{5 xa#{2,=j\iTu>]f'۞Zr`C^؋i[AAAwLm6~|wq A|lhOa[X {H,!   %]$GEz,< A!#4 !   bu!5AAAAAš     X aMAAAAA&     VrXAAAAA+9     q9Q5W [Z5J|(݆P?3QبK츏FF'}Ho"=o`{k)Q+P\<~:ZWGw$hK KhѩخɁ֭8 a]tZ-KD:w(O=]OW`e~W IDAT+{ s[wcɃxb$=.`L謓Z%6zќ܆5_0!!RTm{3 G" bՙ/$5.ԓ/w3|{C|=~w?зh2A_F8`07-W9L' [_O:>F&OtЉvOhl>;sTB`VPJ.}Vz;%s;0uB./@ Ò 3c93`l{q ~r>e*r<zc ފBhzC٩ YݩCo:e)_h7"ǭ7@oÁwc6A\}WXW|GhUՕz+Am~=بvwHq'ٶV?#CM.(QOu/#o>%.WH$[VZT]Sufgғ?IR%3ն%TS|b.? #;,;U;0A]UQz6{Q>~Wķ_s/{<|3RO @V=`464IU6O>8D1oW!=ت{!J?‡v|YvCRWWPUQWwFS޵O0G[}s7AD;9Sm+Dȑ%m '6ZB{<ȯ{ɰIj"iovU䪏y_uՃ!Z =ˋav>/=" 4xi7.YeQ-ڶ'Zj}iwCo}Et /fv+utcqTv^*y{bRzX33}\E}4Dm>#=E|"y= $) ˳~f>g*,Ҥ\\tPOqg?AU|>1`9UU *;߽𯯾w61)< paXHy8fhPUtQX ycҳ$}S+׉ `ba'l}("x`NRib t3BUEua^$we]=V0JXyHj'H* Hvhʻar=,eS^#D I5k:vho5ap%N9ULv]m+e2.51PWGh^-2zϨ>&BT6xF X0:u1y9Ngzx,ĔPSc1TyS{Yg&S3O}?arë$XJAZZH1IE-=23ǭZz `ۀ%,xV&m4\ l˻S$/'W b5䰾5ĦS ?_젊PSV l1:p MBԘ\$xz#(8kc?? fa*iTU&ǰбA7lsT'2I W<V^Qqr/yI8VeSDZI.Cm':B`{v$LHS{%uuA[e}h_t}ӄ ZkQ`A牟γqq'c˂ /%Խ<HͱJim>JI|1@}O+|ǘx1z uLz*AA, kL)(Xc\s<)wl>Ńɟ#SRH8m]BbPnL?X{,=R D%s l Q a4lYpȢAҶŨ_L_&p".CA(@%ŪS9uЕeB\c * r"6>lGv$%U2IʞRA \> ^Ss>q_Fȓws.G6-ϿgH XyyVT.U\fA Bҋ龈uUTN~%= r?A]Qf3?#>U cGSy"A$@tQD!4Ȫ1O`M/vnRhK $.L2)2۫e^]QH;Ȭիor]AADUm XxvŪAgZ@یJKkl0\>X 9yn0;^+wqp[qO8sU7  bP5AAAAATz"AAAAAA, XAAAAAk     AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD"@ߌfq9y=|l *J9/q%j}}]ە˙z qYypݸCCgd mYdILu>X~'w߽oFgqKגH):3}* Kf L(^p4,Dص}0G@U62>땲3lvE]0@ ۛa6c.E]ޣʥ$ >Ly{\v@*sd>y{o>6\.g>Bt]:fn{[QjM"=3Yl^hm3unv;6gnSlc?'GZc\D.U:( u[UL  EM"w(W|4˵dyzVf%W,ܞ!Ӗ̿\Kߖ#w=V~o_-"oj|ܷCa=#sUrbX,aPכºOgog>Xָ߳ybk/@(oٳߓ? >?nş}n.x0{>-yfll\C'3C# f uЈ$@y w]f *Tm߆۶4YÍ@ '_![zp,܅>MCA)lV,8y:}7&_I(#JO!*X59 о)w(X $Gxp[TQ߽6+ 6KӯqΡvSԬ@ouƎ;rsa,\6\m͛bg#3=PEavxȳ5OpC]]W)u%8oOڅz=궏4|| T>)A<=SUt MݽdvbTb`Z^?L>/Z Fǿ_ٳgQ6Ǝ;u1vh`ug,H&܇m26៏zL]n7}K^';o/{'vXЇJFNLcOkqM~r?i3,}4k$l u`AZnєwf0LN[/:|*^fdG 9hcX!Л.ܭee)06FZ/ gV@Q֛I1iO%dˏGhctAo+؇+߳nS+uYK!Ϯڭ'KϠ/> .3y>UWəW3&l>,YH2lf>\K~SVY3pp+{V'C cW4T>xʪ:z>ͳ,ho,cT1ebؠMy-/*xGwX OR]g#Y /0`,|iL;\9@VU3Ah{R-&R.B&Sh^P[Td9(BDʦ^ LMl3m XhS Ѝw#-SЈ–m;)'m8#rρAQ&G&[t6._W&&(]ն&f?B??taO9:svd t"f7\잸kR$q^wߓGY#>SBf ɶ[3E3W}8-LcqW՜u,[$̸8OW (bG f<4$g&8Oؑi%=)l㼎fQ!#.S֍6d=bAU1a4:Aɲ4/$Q&)|) Tj56bKAqpB- MpOcw/۵ jE_m.)I=SisgC{Fت z@}K34, Dњ2q`s~c WuyLb=)bs7o'ΉȺnM`9_GXbĀ-r{tm>~sS 8i&fW|}9T K>Ô{"^Q9`;+j1%AR}y M4[vnKx2u3L0ŝ m[1R J;TYY>{H1ܹ32٘n |>&QO{ڐjP䍵!_LQE=v|9c-?;ècYN "[ pJȺjyxA1[0yZ}^g\XP֩{^*!f''f(ZOmci,v 1 ^]1ܨL=4Xc=IϮc .Y1,`'fG ٱCBkK Gf1uvuֺz/+HR|_VJg=vD7Gl93%xHPsűjgt7 @=aFU=-֖5k~\U:}֮Ed3qpw!}6| mh}_` Rc4,g`>UN{\fq) t6 h}jɣsp{=@Hq-ƇoڨÈlj#fCS(&;.-ua+Vӧ'pgJO^Qo[h})]ΡOu6.3j2%Mڵ\wO/WQ $=ƃJ7 M")dC2SsQ;>Ɓ ;Z Sv;VK(5_nr;6\7-Gf<uW̰(9rP{=|AѱNKko &*ᡛ׋KV[ݽ԰>&[p,ة2bfg'QHq^/:Z.; O_(ޒxBrb~?sѹ I}t2="ga0U+{JPp1Nө3ϏdNbċ/GxNۭ}FݔJXбk0&9m&+7.E(ZkDeVԍDVJW> pVc;FlSnEJÔws%c}g""c U Q`-50CxI2%[XfMXme"{u a6 V Ϯհemb.vxrnX <-hłgBI3ZZwYrc\\_VR:D\W62Ig-*2-V0bװVuTTH1ÎG;ދ$,OjEk˙~aRU $fpejׂ:Yr{";iU^{\ZZ…w}a{{fDŽ-0QqߙMrC:o,֩|cL}LߔZzp_j0/|t]/dOߵj} :Az^g_Wo#?L(^TX`|iwzWO',=y~7}wCG{>GJA8h, X΅c@ uӮ;xX.|s^]k9ֆ A |qϮ#o[l>xњRf3?#>U cG0Q$ 7]9tp^|ium *Ǚ˫0K;n꼲?-ѢܭZL B>#zY71!>>^{ײNAۉćO.`Q#,vu],,#mbzf᯲vua3۔#8{hMxLyl]8ً!zFD>6{OKyeXAAAAAkeXAAAAAk     AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD-$1AŹ^M+H  X5NOO~3?{_>ӒHc#4@6jpΊdWUY&۞q͊^pt&X(tT2ړeۛCIƄj\";fX_f3X=ܔdaGS z{|\sʼU} ͜yga_oM"v}3usԧZN"o7X7 fXw( IVow(o6f~'=nL!2d/ymwYGםg  )oٳߓ? >?nş}n.`;}&&š;g< w]9hsφkspno۶ddV e/nEmj:gc.̃DЁ#AmJRNoO^goO!735+@:lE C'RK/Vݎ$#2y8%GEF0Rt(@N 늼s[o$lrI^k\eQ@(uؙzO| t c%]>V{F':S݆]%Gk#8D:< QGfLRAAD:Fǿ?'gk;Vla{[5`N|`ש4ՙ~d`tn6hmGEÃ>%ۙw!o] vQ=]RY홧 [!Ua Oy4G}.6#wGŹ?DmLS21 _vſjp-w23⳪Ykkþ-ʝ`ޖ "=afު:C"N.ǍC8i'3ДFUʞylIzcډC"[s#K[|M}NM4Xq}5ٳgµf<2&'HLcUs7{xpj 8Qy g.:6*L*yۘ" ڝ/mKdIi)%rL*J5k9Hpw}ΓwFM  b;\xzoxُk|'< w{Lo._a"4aH s1#'vSp&8j%= U~-di]Ncw1 LAؑ:9}G/;>ْ>6( ?x φe()r 8@).ox (pew=֙Qu1 :BimyO J`4ڼJڹIދۘA u~yk\K>,G@X3n- w(nMD>4טɓL Էn0Ptiu  ærw>~W)J/q[?g M4[vd=85VB6j` +8@j` JhJX)#=Kb܇3s̡M)tS J l㰡8qxtBt!HuhIz`b`gv.[H\_ AA7ֿ\g[F),X_U VOmdR)|G~wvƁdvz<aGbg}ܞwGJmQ. Yx60 9x6}`pw(o`׍[h|"3gAk7)d_˂ʞ׋KVؽ԰>&a6uAYTE9v${֛8#53Xq>ܓxq賠nF ,t꤀>j) enJRbtB|ʂeP[8;qR!I!}4z5R0Gݒ2gR彈bVf]*qz2:X9S+@jrb^VXTm>x9hCWܓU*E=ΆE(_EP2Yӳl}U>4GﳲRUkW)/Ǔ1\,To5 α+0ulЭZoOBbW~ Oyx8n#; :9tG3 Iu8- fb UقRt8 )8zgNoڦ(0ֱ3R/] n4*1a kzFe6R{&sNk]ocw훇"k\Z]NEmX{[W*:k9f\Hb {:k|   Lz36kVW?}'<LO>?=}wL?ʏv#>›ᶗyTx&grqܖp/{Ly$0q~)qh?^ $GSo kCe>M}VyBF ɣiѮ72$ҋtkMԺDi 11g](̄1uu΀_d\jsREuطRK*;"2vm m]D8&b YX/1؅^tZ=KZdsmVG?{ׅ{dWa?Э)>ޕuwї]梑&ds<(c3z  C6f~G|jAǸRE pu bnJt"WDBux,zMx&Y$  [ubqx>`5Aw :l$X/i4ѻ`5AAT(Ú     X *t            AAAAAZ@k     b-5AAAAAP #gj&ਿ?sPբnD1no Αy_sO, jfJ{>xǝ   Btwqzz*{}wk\77 IA 7W  LepǼCګ 9%E9S2ў~/DJ2DU`6Ƣ.ʅ0Ƣ.~Q$>JW 52wK Q0.0 `-΋ql{ՍYKq͊t]H}HZ(Mm;_ӓg9   b(oٳߓ? >?nş}n.`/?߽M{01 pā3Ca}A#r0ԟᮋ|f *&ڷ=ަ zmn0Z%0 {03rA=t}tҸ D'߳ 7ۀ'ʐw5XXfL9VK/Vݎ$#jǢHn0laQ҃M\2M7NeAAA;Fǿ0 f?m+βڨÀ.kFP lOADfvV{wGVnuXfy4VFߛqM~r?i3r:!W Y{~TB.|t6('1 3vNExr@`f~ƟϪfAF )Ph\%Nz,.?lP]1T+,%tOGɞ'|i QYRn ϝűfFϟa.ښ7*aߨ ehޓ:KqN's4c40   w𤥆o7_~"5gt][=;`z 9#:L& <\XvNwDpL{ox?J&-L 82;/;xanI2ų<@*?SjJUYY>[R%M5(֌ %,^=U=4~ymF=e.o5D22pac8s =v̂5x='bz03XxȆ-_ɦl4!(ܸ0Ncg=0> cMÄlE@W;K߉2pޔKF _F]WV|SXp>/hct~ilNuvz$+H9h4[NB~/OS}%Y:d^6"  XOR !1 |ɗxKʇ)&:,#Qr>‡z4\ԯHƁ ;rq2@}K΂ ~qZj >Lnr;m- hu_ )[\|T3)w]{riV+;K(wEAAq?S_ը>==O7O( J$޺e 09 #m :#.)/v 8zŲ- VwVC֐j.lhX*+=qy T%^fk5 /Wpc`2 E6   o^wU cGSz"A>Ss×-}!*uJgAAA _HGDy >5f!=Q+`,Jon=,fypxns2Z|||,Y_mk [.% w߽oFgqKE&opz7EoH)R`;hoeDoO`p,c\嶇;U 6^sv?O^QЁ<%ÔH}JM:a)Ι$Xjg`+7l]G,h*ʏ"s͑J(6ca\pV?nޕ_c(1$)Bu0!͆Wg.Fۃ }=oa?Ì\AZs]Aʦ"XЦžSӞ|?܊?\l |s{01 =Љ3CБ{A#ZԟᮋXf *&ڷ=ަ zmn7H^jhoyp,܅8 8Bݦ4.Y0=qܯ-bW;9:/1ȍ5dI!DUc= @cJ` `:59 о޵ ӗw)g.F۶̣}tW1L}5qW`3ֲϕUc|,}pO'vڣǃفڞ&ewęvT0G(Sc>Ջt_Ѵ ؠXEA*Fǿ0 fp >-qcNJ: ~{f W u: tl}-&mhnNhlg } x)Y;ygv8JC6{YT BOL'wGOڌ0K<2L=14pᣳiT,̊۫fşϪfk v(\w!b w uu*iPF'H7#>KK..w'Wl0N}4ά1gpmEGEߗuNi-UD"ٺM,!'Le} UNy|pZ S|~Vȼ3ӕe5lV{x^zuC7QOAsc7xK_/pw>ϟ p!p[e /{T6;q^*g>XwA ld_ a`N\H{A>)9cSإ2+r)vaΆ|EYY./68<g[Bɏa!l)6OoѶ+bp҆AQu`rdJQ._W(5bLr11(qJ#M0a$9FKFq.덬M8rw/_6yEmX'GRE+:=0]]-9l˅ ؚ.n`>υކ}ghd%V"{ gRDm9A[i~|KhRYO.R8U}תLJTu^xS6LG$A ?%摞E7֚>؉hQ%m9/k.Ӭg?V]gyUwOԻo6y$T2Y=sP{U?H޵LWqM1C]״ȜO%OޣSNa[9Sa&߫ÃGq }Q{e|U :O TXo jxxIm\+E `%68oaR ڀ{anI,tqYDȆL^ج,q5P!DNj@ AaC`ƣl N=mnTu31 p+ac8v =IEMa4fֱ,'8|%ӫ}s|5ǎ݂3qeUׅر3q!nQ΍R׿̦ Z׋#;)4m}ڧsa%{:,o?݂y9( Q=p'aƂZ{2;P-jo:0.`tn~Ğ=006old5tQ/.AF԰{o堦^NSXb]v4[IY@gav9/9UUSMx`lSznz:H[Fgr׸j#=Kj<696~N=W|&,de<8y6Gh] A/$ ֎j_3-'S=ƃW73~)L26YֱfƯ?\ԯq`Î8pzHfr$=Zj ֘&Cnr;#xmeN{ ֮km[.G&:0:0yS*-'GgZL[-u $lS=q6b3zs>7#{oAJډ CkϬn)@VWn/\>NY*gqpQDAA 7{w/c| >ŧxY gˬNqlO9S&\aׅ[ QkV󌄄G՜!lXSuY4 ǵK,ݴaj0VxUqIAh.e1*:ɺ`|X0n\~l_}s9:K^I MOu9^n9e8d'r4.PW|V65(\t— ;L^):pYAT\/ Ap>Ԃ>hUE2{/] ouٶ*2AꃐlݜxNVʚqK=ks]1S䂰Vwƺԓ%Ҩ.] C'55d8Y&m)c%{a=$STZkY6JRBkX;5>JfŞIIjIڎzxhȧ% b ?pgQO'<ǺDbgjǬM~H]y7.X_{xGRtKH/)p3,hH/BiNu̎4ȿJ=DD8V_(C|} =V,(xֹP඗yTx&̣2-_Hz1PF[fbo_m=nkO m *BIP5Yː]Yv,ĥ1֐n\`('io#vkGrYMeLy,#e^DZ舲Ѐi0ݿBE9}^tw1Si.>g @kGg3?a X-V>ʞRA \> ^Ss>q_q:Ouϙﻀf?hۓ,T&WIW:&Ҧbuyr>Sɵwk10z!O!w" ]WԷ;V?y=5nc= E6s>=9#[@` WtSJxGIޞ9||?#Dbg ˬ۠ARf3?#>U cG0&.>9B3'1?tp^&dT m1w2)2۫e^-0sn ͌z[ +UAATVX/:0ș!ugW_DAxˮt,ϸ͈qMV5샥w ]93z[uC  eXAAAAAkA.AAAAAĢ5AAAAAP     X (`MAAAAA&      X}AD\gjZ PHR蛱\>V2=852'<ߒr=㜭AE 7%2tXϳkr9sP[ٽ>۞!3˕=u},[ݝEt_ݶ~kwtwqzz*{}wk\o߽MћҤP f(E g9,)(a,8涇;UQ^"=sJ*ZEy)rm嚙a+<^3胬^);fX_f3X=ܔdaGS1=.)Fz92//N!˙y%. `-q8OlLGH v{bc]`M9s1,Pf#o M}\ kqp,BV)d/m(kJ:^wWX$xWŵCȶgⱞU3 grȴ5OkG>fG|{ޣ뙭&?u|?܊?\l̳ZFAmhMлmKB[5>^—Zuԃc.,(l:p>Mi\fѰFbwmj+!DUc= ȌeI UW]do ׼!Ġlc.{Szu4>]P2 ZBVo-i]*"WdWx{<(]cNJF|Lgu]3Z6pGg Q7iztl}-&mh%r{Q/ylg } xٙ l\944GɃ{f%rî.Sm!%9W`*s{c-1iя\'mŸf́'JN3N ,ʻ6B7ϧ0yz&k1a$΍Z[>FSJ)˳uWoғnKgmeMb71]L3gqg oU}Nt~NV>_T߻ɜmm`+l]{Õ662춅yJs'7_ 76Q*v;LoDiBi_ > \cj+-0W1elDVvP&3Ah{RmL=Ed)MpOcw=+۵ oXsMWOʙhL:SFXhӰMqNpd)lT Xjf󥈭|IG-m.̵xQ=L0O sƾc a~YGpb =ZۢρߵToC!XjAU=> ͩ7 `KO [W*,*}㷿xxُP:/Z&-LJ;7H<ո &uWXL΄T붭8L)q#=Kb܇3s ϔsgd61T !18l(Dz3T|_LaWqlJم>Z b}E=v|Yă"g%vwB6YNbJu/fd\ ٨-Tc/Թw随f/uvuֺz/+HR|_Vg=ȊNz^W.[7H#<5`vF.k9 ȶbydTgV`31%$6 ld?yUOdY>L=CM~hׅ;==#ص;{X< &>tL%chOѯ}/O1l4;mBVʢ |c`S_i98a$np{Om& %Jl^49է>Y;ۄ׸ܔf7p1uj_3-Hzï:>{ dQpH\ԯ8aG+vpYɝn ^k[=Xr{w܁-m\2 IDAT#3: + nxE2^h'Y)i4-iWH vQc1#{DWe"0մroſ+5-]X9:ȩ.]B}XEgWt)3$K}H!]1RZS8ZB\Wmmv_vXjeO .fxkCC(fvd֑XG7DƆQ7sd dj+g &j5{fQŰE+vjGx{MatLA[9e7O$͜kO\5>L";0%Phg\XotuJL ms^8!UibvTc l5<}Z~l_@&]~؇]0ΤVEH;uHȣR7@7ĿL9l'#%f{zoźa]|t]GYw"xjOuDejצbGccE0wn͹3.Y(.vJ2VPZD/wK _ԥt&_EP2Y̐u_?2_~[ԝ{ׇ{dױAAR"DrV z>BkQvf?@z&Oz +2z[naȄB SS_!irwj٫3fx@kEgK:}Z~OyB}x70G/ؾUMRjV'XW>VIBӟL WkVW?}'ƖV8*-K|46.0D[K~͸OC^39Uכ#Cę±LXꠦa66>g+z :g]]+6qA:^kDr Q:e0[m3b(v Lj|o{Q݆>/\̶S> bx5L _on#ͱu/ǎL9>Z*߻f!EDyȥNw)v½dbOL% u> ОZѺ*k#8lu*Օ:Z[z҇lmpo &턵Boy֥w -omedźVY~49ɞs\uQ<]rS6.bD!jl&|hT~ Js}EݿeA Ŋ6<8ʋ"EE/8g?,¸/TMŀ^ {*9Z[#@ti1gX/=1+X"gudecbK$B.fl,(?A@Ȅ|Ļ˭]y(KQmV3 + dp63q9he8lMxLyxqyR/ଣ3֬Lb'(P5AAAAAP5AAAAAP     X (`MAAAAA&      XAAAAAk     w7O )~>OC N9s     ⣤6f~G|/p3:=Wg{oYf&     WdWx_~Ȃx AAAAAD%Abo++ Ak|'O{?Zӓ-`-U+k?g5uNJSҕz֚7_M?35qܑ_w:?TW$ S zOQwsE ==_%95Gdi> ~?+]vDe"牀5`"L@0X&kD ` |k8κjה 94k|+pƂr|Ts ;`}|W;;;ݶN;-sjuA};;:4DyԐF)`9\N!p~[jxQF縡 ' 1O& Qzwڙ X_WM.ruoKVk U}y-7qC >ռ[v}rX 1vXE[_oڑ㻽Ƕ^ǁZ \Ѽ/hTxx~KרHcm$ܡ>,7fy| l;/6UƤ{B̤;`==_ҕv-{ݝ:n0{Vw|[~VSSϴTRj4VОf?xCOyYQIwam,hӦ6_.N>,=+7hk.O:YhV1[$@uu0~?]RM\(-ZU[ֽmmv]ovg4<{+2R﨤ݎgmhTA܉fu0>΅]^|/||ʒolQcmڨ~~4il!0Nx 7PoJRi> H-pkoEFό:~H{/M#>1>կy}p;%r]M:L<$H@mu'&_A0sk]](F<7pAw.Lm|fH+AVPVUzl:'U|AB$E߬rYj=5U]vTPvIֹfEU\{u;(-J=Uc^ag%|S7ti-˙+|T)CUܾp m[UuU:jo.0󸡅RC:Čr55},>}7䞐/ʐPƑoٯ^5Tp{Mlּon' mQ]F>;^9[ڼao{gǯpqL4'N?Y3NjóVB~'|MXb|w{Y6,r*Sǽj7Z*7ꬍ-l4d:e43Z|,k3љqwj}ݿlԈq״3?GjoW1F_oЌRDxxYEY$32}Cj(V߄GJsO'j"˳ǽ~vWK!W%S CMUIЉyd7t ݔf2u NfzՔzGeᛂt&Js6FM1}'0/4 CjfL,uk;ot3o:|श)3Vu, >׼5NfwFGun9Y7L{:k};fUᝐV|%?nIv{Bj ۞3jzo Njik hSiԖ%g'F>ܚok32o97eZ_LcJcF8ww\s͐~t3qkcA{w 6`jZd,}oE?,3qlmXS5UbS[VV`P<32̵JΌg%{rv]ZI?fc{~MGyoW)0kXS``U}A8nhFOUߌ2+0Z|V~Y1 փ ͨKֆZҏ]mt}7U`6TW"KGQc<~Ǒ(w~q7LߊY̸܊ߔupܐ88s<Ay'ᛖwApYK->'Mj$P/XՍ}"T-(5nnep3gxY퓷`:~l #=両Z:l}BvNz7kz՘ /0XTX9mbJojފ}1" m`pAF t^`ۼ7}zdbє8= ieoЇ*~쪪> fh,Bi jz'e#TU}?C<龡]gZϟ%VUu.3KX]Ɯq6˟S 2Y0{r\}CsQƻ؟ ?d]ƳRUuyT/iG/TTZu!5G#p9ilWQi,UF=eΕUD6}'\븥֑}nFp{wY5~$w>osYk.$:pIS\Y~{=^t.Yb)pa,T*08ޖS̹8+P^-`-ՑgsVSN-}`g>#2hl _$g`VUz ,7u)Yd\v]` DyPRAKV'AQ٭RU'Qv֟}2Yٶ7ƣ4AS^/6h,<]cx71-|S\His"S*00{$8_d,}%;?wï;\_m߯(F K" l3]Xf(kÐ%\!lXv ہp`t-W{{pP]I5'[j#NDjljk8[ȿG^-B_P E;u}fc_tHyZ5nMK@7nV-}Ƕ#qx^X;n'(٤{װI; ^KIFm)7d֍ryt8հnnץ}BkOC>|?r=1ߔ ՞cISׁo\Pڟ2{%UMw_Oi^)>R~DO޵Wl~ӿ#ʈ$N(o ЍgMYЮL%m[^+k/=^y͚R{5J끩gfyϢ i9;TFK:sX#+X5A}v_[vUL%i&ܪ5ԬzJoeS_U2m->u>`L[+70 ]wq%AAzofT bzef7tC3cQD#~oeGmeenѽ-JSTt}'gԺsTbgW[|nT1MEHtnU X3|I5xU]/yd:Eǒgr%$?{黯Hv㫋 t,3!e$$;YTO |B 8_>nc[[zJ׮]gsiC *̽!'̘HcnfO ?F&x }oXjr/ܟ5qQƠ\(57[v'wmy?=@!pkQ!CNy߷\ fxC_fUF%݉ŗN|8o`z U|AB$E߬rYj=5U]vTPvIVL*`{హqm[c? `g ڙIّv|;*T" Ĝ | 39seoj==ۚM\|qO* grB`SbF] dOg*7Y2d~#*#>4S D]sm8m&6k޷\7{]QRU.# ;^9[ڼiȼ "NX|,k33G fq}@J_6]toϏQ#4ׇV^K̄9b[ݿm gwÍLZ+]7}3X$ϬS zķO{CqڿHl~vWKӃgT_=NO; {kÐqsS{dRo*Ro(Э IDATfw8ӫ2; xK)|S@QZ>"K{ڨu:[9`pn("3}=UGɂzKJ=U51oh ͪ5үJ~ܒ n=gԺf*f1Hz"O-73][ꮗ1/F>ܚok32o97e 2%m1K#jq~.FA?TcT uۆnL{+o{vT|=P 3z c3 G͞9nh"/><[g_l𕆩.wΖ/S~Qg䶖x3T? \#θn;Cio8əYQ1u+xޏ2gkd@sSﾋ 8S= Է>yv9΁RVdgMqT-gOy×}>+a #jsA#˖=7SUFDY~I5v[[=f͵`I^FꬶQ:To^pg]toS`|pAlwj5e)/ )^p8^a:{Um4d K_yAewu8-=' =]l _߻u~5}+3J[~\U \c=m (Yޘp ۳v}Şƨ<]]WV3č!s1`2e Xm|sU)YMӟk?fjv$q.kc:8a3f _$g#VU{,΂7Yot3|%HuςtEN`miUK ƃeGd;WZ9{aϔ4J SVRKyPWy(hGƾxvEQY1|A$)c}\ZMLp{vE3U|֝;6uN>/ԩ^wu~ά8 g'5ºUYT'zjJ׳ DbzOjψ4n[v%]E[\#*\}]zggG߯Tt*.J'O`8gtTL͆mn /jWg֑tčg֤- r@ ho3_++=^yMƸUFXgYjܢ ߴ}ɹp01eZv-_TLQ>S 7zzߗmnU[:óUoJȋGl}QrֶwmMbZN=勯[?{K+,Cƒ2R8gaxc72OR~^=g̝kKo+rbi?+7PٶT>mzόqZ"_碌Muϟsٝ 3p/qjOxZZGu3[pr~իWve8,3Ss9սI߹ Jp}P37]&ƃ CY||B7 ώ$2 kD ` 5`"L@ה-quM߯) 6sh(iq6ԼV;`}|W;;;ݶN;-sj)?tD >w/5=kQ ޗ˩~=4{[jx-F縡 ژ;(ޝviA`@:n &RչZ#Cf_wp^ ~B7;sľ0} QTq)<΅]HjaOr&]Su}X>;[I*Luq键 mY?~}f½a߹ 2o!i[U}*|Izً *YYijzjt촩̧`U68asU1ڶn~Ej+椌8;YoVk 2[j3WVӳQ a|W騥J 5<3rZeN3΃n ߈6 e]sm8m&6k޷} JUգq*WcN6o2 ^o?IXv}]4}LRp.- eG6 77UuŹUmdPw]:ϱ>5xmR*}o*Ro脽)cTU/;UC-P(,|SQHvs6FM1}Cىֆ!5SNqwO SPd5TAA:[a0\8ջWw]0UAyCd_hVe^~Wd瓻٦3jzY /-'=ԚVOU5}1Fv[mWsmF-'LkÐf9﯄BƘ5]?qjXƂ]\x9MrR3lw/.i@W>yl\jȲ%v0%" }oUnDGxr並6!hF*Ҟ5RO2To^P{ fT+f5e>fg1HZvxqSc;0讟RiݽPЪ *=^Pw7~ʽ6oF1U,Bud=_q}O*HO}vj󅳬xa}]ʼ_'+<:;o9kḥ֑lnF#to+bl-(-`-U+k?g5uN^^^>!(Ƃup3pT-g("93 3_4il ,f8a }vb\v]KB؁VT]/,5T?Fu(;>"JhFv]rƕm(M9ׅd?fyIy(hGƾxvEQS\,&P;LXo/뺢ҊSiP]h2;?wAmםN}/";ynszEAoIލ"Yl3]X6iYT'vM`ҷ';/c- a52򾡺*Ƕ?ufqKpf^qT ͠ gQ렐?"T_iWplAֱ;մts2F%[4J]r&j?G!8Og&'Iʸ_j7T[3,jIҚ3#Ӂ!c g{>7q==0i2;ybڻ;NɏRq>y;xIwJ娒"Ú 騮ZRwL_7uzxgMЮL?9PǷ+//spugQn0(7e]r. nLVA]nsK:sXM7$6l%i&ܪ5]S9uvW姾/7#/GeX^[}늻̚)1ml4 4vٙPq@g]cɼ_WbS;&ި.5!e8U)y$q G+^zk׮]V{9nhၴ!,ZS X.RA],1 H40+@5`"L@0X&kD۸o8y'STtuo`?/LV>:c痴9S3OE*gvZmi~I/B/N5dSO?tןٛhdX<7; ԥm֒o4D_tGK*~Zﭤ/ b>E֝;Ww7ZԴK*mkP*-7dFW:z'*~9Դ4eXw}JW7>c痴NcjZWg3wut%` :C V[_R3 I>ynwwWwgwug&]DVm[曫*'MN0=Я e\S'z]-S `$F*OOh=dvwZJ j>5֝;WwC7KVqiYE߷mJ%M;-'`5 $CS<:.ʟ}E;Ա(^zk׮]V{7]<L@0X&kD ` otONRQ׽x<+˺3%-{wZ;} Vm[fTJxo@;ԙٱ ؜$H@mCYZWgꀓ{*=O )CD_R0iehu{vs*E- Wl3Oz]G@~v7.KZ^^~΁:Yd˰~ۖJO>N{k~ELSWzܑѢ=]t}TZ ݐqW) V)`Q)]>* =❭ww"@}^ҵk.=OT.pX&kD ` 5`"Lqp|WOfuw\z\=s|Gc71;`==yNkGCAmSi*j ƂqC rr5ej+5FQ߸<{eQoRK皪96.kuqTgV|¤c`o2ǥ|ZX: p%C3TvޗޖCΝ-56ÇZ/GมPmwш}!< QNS5ܪ~2jIݢ}b$iX7WfPVxӔ񴬭/ Z5;}.aH-%꒜uqQ=UޗKZ^^.]i޶evO=S~I/GnŦVkY+h=^R/bhc`72*N-t}֠_|{RyUojc;}w|'wJusXSy8  Q~P}~W7Kڼ?3;*)ec2}^@f c .[O ]uK%h] -{eq nWz׷o_E!gKg_`YC(A{ݎ=-7uו1iTF]XYPxOkU,qTsv&=7pPm^Mܨ8Sb )$;[I*Ό~z D_L)UگphU޿ i4ܪJM/I/{' oV`9,*.;m*|$i\ODqc.=ںIjfD֤J Q;M*^6ep-)[&dg7Pj1Xfz2*A}(C72FBmfx׼Smi5Y`:"2T u/p[x̵̣͛f}{ {߻=}h6=|QNJ>xۆjCe˅l p6]Y}/a xAЬ}~72eh1(w=|83=UyP Ew8̚)=^.q:a8cDxlYG[>tgPU%wܹWRU);Lb7<*(6#7xfo\V !&=Qv]#;87oE&sx._Ӣu,j7Z*7F87|>QX +ydP5eɷOgɠʒ]=zvn26,7 $f9FiF|}o&Pr{`l#7>T6{Jvק#"]:3¿)q\3}*Ff;U/6UZI#x^,{ lwI ۆ~?s^3:[͒n}9x5Is.T5J3_{ru_SQ{Y㙑q7P}>=iH C8x2z*9w!NAvvY%'glu*1':1sOAle@hs?~`_IX+|Y4f_{4;_eZj NL\YAঢҊƞ]ʖ]ud fT?rϫpԓ%KU.atRww ?eM|c[!,oQW8׽ǛςAʭtŬ4}/;bV5]qt=2zz_ 1Iq@gqmmn$}wm=5J5s ƎPUu1k-;wKP g/Sb {߸l- # jYxIZ,Ա#MDYQz?D#ۯ%36 ޅ _?K>_`2d Xm|sU 'vOvdpc/jJ#;5~'TAwSê/901a|;\z_99#؞ܮL[ƟrXQu XjhV7^հޮ(:#̞1̺qo+=ɞΤp Ϩl&A{nguB~kW .(ӐRzIռzTw*w=c̖5$ 7$^6ևǾ,qK-w߂*Q3b-[I@#xf~&=)ĭc7SrgqC1(9{vḅf{TM*8Jԅ̨U&Cu7-jZ%fjjjVw7V ȉ~{s+_φ?lfgF܇=2n}S 7 NB5wq]v=o(X tGlJrl#jXlnm,>Xg|2zkMEx4~u>y[I[=[UuŔy4,`mT?3=گu.]֠O1>mɼ[}[ ݛ֏mS#NHcLգ=& /u`=mp3ջAseofQg{aӾ<;N(G)o|! Mߋ[y{y%Sm=r12n9̙Qmr'GlKV:4d m۩~W7C.=3ǔSr I3i~6`gMGϧ=9Lֻku4tܮK1%Ϣ %ZT榺Q %Ґfmg`<믒ʇxu.m5iO̚"ݕCo {BOi7V]|{z˂7:[7BY:-9ԡ9޸=Om=C?8n5A3[ֺ봦r,vW姾R/7.W2m->S(pcVSfƮ.w7=Y[#jގ!S ęJvK׿`߉ `gkT7:0 %R9uYȮgpM1^.WnS6ZQ*4d .j:S.1\SS@Ms)@*S ZuŬľ"C(Ss~S_ߓ"cjpmpqy!0V$m|p?o\bXʶҼϬ(MOn&p#wEV[ {[NP_H<߸Uؔ"f>?=|~ , |8K{.\bi3\_>]~Y 1nnfǭy߾a夎cYKu~s2L 'ہLK3)#cۘإ8nh_Z榺E/'p\=v_ [|Vxxnbz?2!M8o5`"L@06NI4Ԭ.5~ݛ;:i/̸|lXO/iyQ`Zo2ҔYY fPjZ{ ~zxuڇuu!|>QX Wijg֒o4D_tGKlN-}L)@@uPyZťe*|u߶Wr>RJt}JW1~2"HW^ڵk'*M8O5`"L@0X&kD۸o8y'STtuo?Q{:^ry6CKZv:۶4|oDiIYvGl%Aj;3;:鵮φ)M'Gʐa9Q,iehuK*vd#銊wDAt } 8wgwC쪭?9P'klo2\ۻc( 0!P)b;d(l Ę`H9ݻիA3,n:R:O$q;,8CGNMYSn_LA]ek:eGG+ףFn" 0$h,zg襋kN` I~P IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_nodes.png000066400000000000000000001304101476650141400254310ustar00rootroot00000000000000PNG  IHDR! IDATx\uab. +vI)ӡGR[eq18Gvq5ѤLvHRkÃۤ'XGS1]ƒ"jA% J|@ˆv3=3s]~_jjjjBq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BKtl\OPͷ\e ۸'ry#mvO".B)ygں""""".B}*R[wADDD䚠BDDDDDDD\B!Bq """"""" !DDDDDDD%BKtlX63"jma-'IAŠ}le;"ՑWtIv܎a\6>ʚɋ`nںɎy#Y'zJ턯`R"^?Brm~O< [Kl[k:a=ݓ%` 6ɜEv#z$3EYGy#ܛoXD׶oV#""""NWMyblfLL l{=Q!$``C'azo'wiҵmQՌ ڵU0y}$fooXUbG.s!yl.<5;)&B5q.z{_c͚UV)xh֠ jU|}|l?8Lzd= ݹ m?6Zy6>/O\gO+lǃGy6r{x2gQ_Ns DDDDEjjjjjUd1c C˦'xf_aQ5ZfܞAxlsyzդͪ !l>Sdˁ;m;u֗V⤚`k+ lBrd=Tؖl:|W]lʊ5Z{T?ewGXuxgۨ*▛;?plݓ9l@*y^UpEAK.0*Mj q\AX}oROՆJ rkȿ,"""rpq%D9߾N[o}B.|tN FK\6La6Tb{z{6$`m~czԆ%mVF]lo>aCOa-mj"m`; %n܆!#`;M~>.;e69:g$__F |VVrhKruN^꺊 u|=8TYaWN\{:Љ5acy77lЖ۳cb,ʸe!~='1AI6}>ogܲAD}>'#@ɏˆlkcq<>FU6$1-+7 +&/9#Sjr\syV]Eơڊ{Ǥ+cl?4A'C% ϝxΆ[I_tkCsnVpgЋÐOz;t rX7s||_~F~1أ>'szmDDDDŵ!{]/ﾥ0kJ<7Nz{ZaWJ|V+4*a[ȫml`8x8Ή`[-$΂0oR}T(&|Lݺ{V+4*>Z4W QۿG)>(ׂimUCF[_X a_W=V!C%t;c%G}h/(kyO[LLyE.Mh:a#tg @<|mUB+j>?O9qr:B׼sTEf &h=OpG +k lWXV5 H1g!x zԙnn.'㍇_)?toTа{eDӀipP7WPqhpPmߗ 78qBDDDDڧ6 !*+pd 8Ι}fI1Dogq J8QꉏB†aF8ε#n:H{sUd*5'Qe*+~fđ\GG)>WߣݟI8 Ìam>m7aɯ6:Q_MP@Cq[w7(rN8d3sB8[g4ʹ{5eal;gcO9[ߚ  nG]U!""""?MsUYOХ+u8]=ΖeDi9>]Q8'Dp<쳅,iogTws06c-'3'r%""""XTBtO7熁 cFˉw>eKg` |=v.{E #l xS5,d;+lp7iW%۪&aThpI2װZ㾑^_a* ?GQ6tw$ּʻGZ?ؖl"WY[X$^e{0i^g4go}^׫ݱW5[7졢sW+Dvg sXVBwg$nr}7~FsOhrvsxM,(ɽ@~ ~/O>K^J>Hky4ܲ'm3'ĕKc\'b5l_T' >޷q3 ޜϞD}P? fd}Cӗ{N18l2%D~1==ηMcCeEĮ9J6a; FX U1l\ sXt(}r( :"I>};Ww;ƾξ<# Nznjk\Io%5aGĐn78FWа]?gG*Tr[w9 r᣼T?α睟$ē3v}?Iӫ{T˨5555m _VuGr͝{^/o.{m0'\BK(P!""""""".BDDDDDDD\B! С "߃WiBMoP!x#ۺ""""ׄ5555m BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bqmɹm£>D\L;BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!"WűMOgަ}ĥ"ƚwf, {MsQWE%u={lrl:\6%ٿxN&"׼mimJf;|iԠ<4y!0 >HxһK,".At^>u?όn)fy WҰVompWD߫eoKĭ*hpUӉ[U;{r6?; dyY ~qn6 ~24D]S!"?`K ^`)2z }>˸&=k?8,DnefZ^쏷pT }#~+!4kdؿkO}Wm&p8y[9 pjq2V@\)F~!j{= H{BDcy`K%$[A=p'>p8j:q@M3IS#"׆aHl>P޽|_҇7^Mely"aG! OԆcHNv¡82slS=p2y\m89hO1oUCzK|Ы')IBwjJشN0#z*䩦gN_f +V4?{?d?k/c1LFe%O` ,{3ccC"9~F HSKT3:ln6±}߾m IP|3?96ƷɶEڡBDF|ƑVٳ3"Ҟ58ʹ&U V8y>gĘ졦 /սO;T 8vs:]+- ?p`qe$8saGd~<"59`為3*lZSNܾ}j+)Og)g`D}jllYӜ<UNĐ|~z˴k>H^*LdX}Ơ'N=}C%v5x_Q!Vz?{(MS˓ާ=y D*2y vv bl;YE˓ŒaF àq lW˜ }7(S>/wvr<h^rB7EOBKx0ҧ弜ԓgj ledCwoهf 0l6ѽ`MW"F״bafCs 9UH8KvtS c&Oo ؼĥEqӉHd&O'mmuTb_6ʩӜē W|ν: "BDJ7k$ 4n~fļwCьWmbH{ lUle?bM5;R Vh=l8Ahmɱ1w3d{o.Wnjal!ųFjy4' rdU"SBDTt{4u`;[ ~W7A/cq2~ {0Axz)E*0}уYG'aSm\^bئ|`+ WQ*:"ם5555m Źm£>D\L;hNq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\CMMMM[wBzQ|d[wADDDDD(p q """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK( ?gKܑ71|un.^M 555tlޠVM\@*؍.nt+l[e,[bh~XT,έw14@Ё;2 ]jrρUD3^ a|û'cXɌ?vAg? w-3$XzEY mgo' %a0~fْW0`D ۪z~ʖ>ҦFf01,$d9(2JTzq>ȔQ& F=ʶFUENQX*?ȌG 3+8) ڕlI1a!Hڞs4_I1CpQq˱=ZÀ`&5u#ն,3cҺmiys4"C00 A*ad@p4Rltu0`ž4lI!`",HDϰ#ÂD%Y(ҿ<Ϲ4֝ca& Sxal)mRd2g lB> $'w &,@؄wQ!Ҏ>~0Xi%iN1m-8BںS+ sTg[wDDSbw ~ǃ7:]9o:tId$amZf2ϸZh'nԊB̉w."?%-KiڡL{1+L~tT[Q:Ř7W WWͶe_?fI٬F%kg3c_ .TGrF|6ja͈dxg$e t("k5іqd`c {yKvsXfcVoYlo95$cfoJL#kT2GڅȄ`An=Xdg[D4=l}#ag_zq b{ۈ>&gy1N]6pt_7 IDAT:͏^PqabaG %䅲c>MP?m1=%71Q׿hK!FL^'YVyD "mO7oƒؽ&.JU2LRȋQD>RK1>- hيqn#+LH\"{Ծ&((*=SeY3iil1n} e9)i>;-tyW\tŋ5lZ?p@>;U?))ZhORϚ\ ;YbK?\ągI6j_)3)ji3?5OƱsbSI{j4"b5ì:aDݸر ڡ*h\ aY0Ԇ1ȓٍX,1Iv=H" BZV(LD1VEPVX5zl E)0j&!53gf1n/ 򦕀ٹ#K+Y9gڡCǑ]t``BF^|ʲ}Yv_,/7t/Ǻʉ2Q!Ҏ}奺`_c=n&_\ mF9V,c`sm{ = h/0GM>b9f:8 y2@ /"WaolsI!'/UsOR}W4.gSr(=I$m6mBE)e#~w;> @M5g{7>wo/ٓOXI{,qb,D'wexRTf&^vBv掜.ui۠_Te_Eeݺ@]Y*ESeT|8$_$ W0$ko)*.l gBPgIօrgrCgrO-7tKӕ4Pn% %^Zli%Ê1{1QW0)um4Dg1iљ>|$ [2-4.3F`~/- tQdh@]axrS{͞Dfwg! A {4 !ZuصDYS<dy8^KF׮P]h=tDŽ:fp. cYelMPƒ[.q?#mTA^&<9BmIb[w|mo*ht{#mjKjDS?=4޶Su#o|dsdlCDeex b|* /4J*N1%{Nms(b&5;$-3Y'ק;\JQޟ3ς;pȊ,Y}2ɜOt'?ʶ1؇c\m{Ex9!2*6@1Dڵ7qVxcF_<ojYϛ_E ^՟ؿ?Ha>ڄ[x0~h.qp >* i33]n fR{+X .]`⭝[YUKD0dvK.tKulc꒞nT瑕I?H2۾gaWOO #xs  kө9z!G6bS۷Rv:= $.lvzyxc'dg=M[yUne{^Pډ+$pY\mZh@p(尮 7,tҺ,o9in@tohsʏ?@msl˲PP{G{]ҢlKf)t\nē<ǍսUT|9JN؟we1V &ۼ8iJ&Q-\-2U,3C0e\vK?de[1F:O#`NO[]9}e\]ZF~3GOۡ0|O_-B>؃Ё>)Λ["71}TW ~W3[%Uwdz-abiF93KUөK jj4VNas]ܡLbى9oG !?^ <-g3VV@keW<է*忞샻Xꤙq*Do/ rzPh%; 3)JWJG%q,˗3wV{mB*ǵ47@|!)]]%8PlǾ $?p[4XNg7+y)㮰 J,[)m~D-/C(K̼ 3/1aL?pciJ1~k`{'fSOCҵ;婌٘ZSX:3i <ŋqFAnIL]WlǶ~w:r a]L7,We?yR7n+!#az|SgHmۃG%"O]KILJypiK̳),g@^ϙvSŋKv֝NtfzYخ2ӄyn16.gدĨVn&`mpwB ˏMe}+2 (fƟV:q#),?5,heU;~?oCEU^CX@ PS#O9/"?ow5<>mnZo*+3f܇UTS:ŋM'Լ9!薈\g|trnO[0|wvm^23"XYNՍY2cc#%m#ߙk:CD̔<G)V9um3zQ'1 p~ ܮB_kNrs{ WWÆ%~?Uڲk?9-EG BDDD$&$˼JďgboEs$p} ~RoֿU3>n Хҝ.rX?```@عuE$?r}o.{n7"""rDOLͭ#>~|%:`>Z}t/oH5tЁnCtFqܙr-"""""?6"""""""QP!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\c[w@zR|d[wADDDDDtiNȵO1DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(P!""""""".BDDDDDDD\B!Bq """"""" !DDDDDDD%BK(=v|FUŶ:uw oOB:g ϻuWDDGgؕO!H34Cą.*ywU@!Bq """"""" !DDDDDDD%BK(h'!D κOe?Y7X~>;\;dIQ[e^zCXfƐvѝ9I imHVϲw.`f Joϛ'ZX"f6k\ҦgXaF,8n}9P e@x&-R@1iѡe&"]!Q]H)}ҋbeFkTs$m$,2PYW-+36s o{? @,/)NuœGM RIS 12,@Lvd-O3#&E1Gi;п & фL<⠽m?)h+%[Rd2o lB>=w` lߕAܭٻ@|9JN/%4>_r#k̢od>B0qzvfqv>ٖuvP5E=O7 B W[Dv>M?ⓕ +P\#^FiG:oComߜlsĩ}ı%msd31S>O;6u"GyV"W! Zl!ܭ;2KJ"gg|efJ F,&k嬋M[bL-=rz}6J~;?1 H9Kl9%M!n7; y$OϢ_b|9uv?rUaaI8p=Omop(9d&? 7ƎxU;%i+L Rvƽq㙓55zWVfD%QTH^}1bi$p$LYr& $;57cV ik@ud;=*JN؂żc*;ܫŬqqPcZ6knd{MRҵS}Ag~Ɍ+q`B/3}<͒;,,WeͅG/ y|-7{; ;d39/ 3xfW[NZVvô0g#m(ْ´%ݝΗ*(17ڝn-(Fwa{ g#_60#KeTȶ~4u-0$,'`d}8``]"2 Cvr,KV9ܡ{dҚʤV*a鲱AQz 3.[s9Ŝu1vF|G ]ɶO;=nm=k +@O`Xİó狴_c^w `Y"vUJoϤn6 TÎAD^$cqĂy( k=6b MXGUZ{HY1ASè-X'` Ï-Y1`eFt1 8V.P񖙬> =u(w*ϚIK73vפO`3}D7+[vo'nD3W<_ow(A,e?,1@a;'N~ 8O){&a/0,yв~qRoZ5ע`Wv{c/ |Rj_3B W'i%Joڕ.pCa}`]`Slh༹q9Ǩ<ͥOOu |utΤ\ߦE?8̞8lfL\ x(bH;9ۏh?£Ls=ްK 1s-କ996{sIaB?_{MY|@3V[B鱼kaoP,Y2W2%,Yfvd>X/4?FbYeU˄_fDZ4Jʪ78hQ~V+@Ln-IFD攍wi5Ϛ3frs7dH^[?4lGV3ҡ[I{䯘A=LnNk':ca= E hc _- *X`|ɒ'H{(ё8]sɩRbv_c}Ѣn IDAT*֙ɻÀ7G +{o﯑KojTqL5Ha8RIM -ķ_gŘ]dB,33 H*̇L?Oۇa8__ɺ#U-gݿbtڇ?Nsl?{e%ijj訙TW?3]]Q\H\SL-R3kZZJ"x(S7M@a30h9Xu\ļ':mM?r9ڙJ#cpYN uɠYLf+y u)ena^ ^v}wP:;rnKx҂m3xke*'JQM7]Cݤ #=ǡ0k,2> >M.X>:Kz6s4IߢdrU'm1ã9=c yuLBG'vQE" sxZtDWUgmf&?Bwٹbf`ӝiH62,;J aoB[ ȱ^r4nb!bD':Lu N184{=!ݔ o'ޛ"}c\'yle'(=Ϋ1eѡmH7£8M"_+u9Җ31*yi7}:b5P=TM?9bXcU`00Ye4O>:p6GiSxc>+Q YCl55@!ќe}'9l)h0u^jPLet[UkXRH ]41[/݋Ԕ20αh37=Ϟ7FhZKQȕi0w/ǥ "\;EY$N|eKb 䙈+FHkPl4}$2Lڟ(!W杺q7(CΞoYUCOoݝ^>;='I{s42,qo)TMSjԳ̳Ԕ0&G٬\Oj|Q ޛ@?SA@D'&39k-J0UapG&0`,i ê]O}˵4=YDzy `:t L!];=ݬzG Jo`RzW'P[=vfBx(FēKRb77Lެ8xڋ)Z:X:XBG%0x mH&=XLrI-Ͽ)3v솄 KTsd)NID]vJFSHn?4=z]njjG^K;qS0#1jVnKa}q5ǾAuSg?FSC'ЇRXv?MaBV=gd2p O6 _V}2Qpw'VF^f%5U/glWObS?jC8l֩ւۆ?9[a'F db]s-m*ymG s]ԩ؟RLɽ܇ON۷CDDnzXʋ;w4n4Kѽ9*ϔ_R9>23L3hg"? )ya L!y7Uвe~ڹjY2J eаly+c.Q[qbG1͠fzO0~Ɂ,KxQqgzV1%2q$lߦ1Pe[mRhF߽y&(u3 %txNg^} M%n^`"~ٝF[ϧ֙tO0Nuݼp]zW'. &~c ASGWӋSn]zbwJ9d~I98g:N$7:.o d0BwQ}jOE!Ҏ׸L+YƉ?%!@ھ1ppHԹ<!_L˪dn9W0no&x(} ý_roDJ"( CoX+K/KrlZr-gЁ :^cZJ|=>G XGъ -bԒ8pπӮ3=9eOk] 8RoWf0}_Dsk!lz>v/SxXjERC%:Ѕ`"3>xi׭fpd/64BDDnz @ xwHnv3a:؍A@s6D>߷M+73r9#oB+$5=a 5gЀ`Rr=m/! M`C820`ޟ^$jA3f/`4i̺DЂG A`"6ljfՉwrlh].qO?\ۣˉ֕*2HM fTŒ ɜٹ?*^?,rGrǎb\mo7 q ?'NMOh~8Sz~^ϞфngY41f/ 0$x+$ x*e=5nr?_cL{x1~bFt]myN#Q-?O c? ba٧'U #~:ގը0-i>&OYCYZf%.ӕjN`LKğhCp$z֕~֑v\x4eu!ƿeuNcdACn$D߮rM3ZAlVMDǙeF`VVȪ'3Ycp!p/glˊՖܟrw |y _O1۹C^x#O^`C yfs}7&V=GAF~3?d>ZHt"sB:"rE$$_G;|xx;|BeP=xY.R%(RNDαtNŰM\tpDOҵ]o79vlK[:5a  m۲q:ƚp] !E`J&<YTGNg11~3cğG[c1h39Gar0F{֜q~N N0w9~l~""RnBD!\coLcEՊzF.I_kṿBWw{:Ԫl}$k""""ӤBDDSwL"""""""r+R!"""""""BDDDDDDD4a\ϫzmESǮů50-ek;3u{ORxr _^k؟/N`x|j=#˸rf,jاdgZj9<~8jmsW_?ph;*vNH='tahF߫N?go~}TwggZse&].CΎ,${5^ODDDBģР4l//TU&qW?ˌnԜ"j 9.:QEqU{cklih=(MFصx{I(Ink[;VX:WU½T6@C6󵍎[Gޡ86Ķ>RWDžڸPs=6lެ:^X v{| ϧ<٩o |NdIA4رu.$J8ngZ9+;е4ZTQNA-s{,x4as| M߸wN{)MLꡁе4~k oR.QyvY4W+BxgH Q0w4jn&DXKGҁw :Q ?vqm(v?y:\`Nd 1~ oiF)Uam]g9!x^n=qDY›xב4;'܃v.qI|m34J;\ٜt5q/*2:%o-Ԁl6Su.t20j[)p U $\5걐F=R$e{|??i+#M- ;{0%S,^w#p zE8o^*Ak~*""""7"(Ԇ71۽b'c/śvM4J:pYBg*:Sv\G:UƵҨ7*w@Py0% ֱcWt՜ŮW< 3Ƨ!W75֩:vNt0>.0k k.F~$IB|>!v8| yM5~6x,$y뀥BZ.Tca(h"`<,Ϭv.:#8*8}{armϿ"S&=TGi؈"܉O3/&'ߤx]Wf)7Uɪ){z$l;L0Tt龅Wu,jCWmd7mgo6{BֺWP;s)I6Gi;φt{Ms ҸjkHbZaL)O6|<* Z݉N GPf>)4.q"졲|οCY!O =Kio;K3nW5!DDD'B!w\ծt6$'$G7T}2jt8֞|;i8qSMDv:hn ,\DIB8 bidwޫ-:֤8iAG|,JF $!t+erB[`x*Sh9]-v6qegOmS G3@;\I{z|>Dcɦ`Jv`(Tv\w4|:CIA[ިªQ,d'NEC:7[M̕3iu˫DŽCIs%8}.8i0؍NmXOáy;i(I2IǏ\J\h=/8.sp레x(m^UtΡab].ݎd >?:SS:͕#(F,(qT\W lyrɵERuLl*, ƨ yrR*qy~oŻGG*jb4zrׅ/h:Qy&L2v2BE7;3Y*Ө@l'S6O'-smSC4j rMTXW2cL+o(߇Fy(W5lhԅ'9-~$T Ka8IY TzLPOkk[p!NVG^M{ef_c(Fl2{8/:4&aQ+[!M4W3O!@y;xY `Vu${<odM*ߤR#:ZVge2R=lB,tl`HދݧΫrX}daՔӊjp ]Kn:WdS`k=4m)ϜfKmL| wdMHR3 ~!mk a0׫m2c+؍^(SDDD@!Hrtf/L/֨_&Du/@ x܃w=׸ŷT1j,TFY@P[Xdvg5\v[>we$ev =hheՍgO~p(F>weLgȪؤH#{$".CjXG:'|G!q||:=(9{l֕u)+6g];l)\<+ߨMDj[j[^ZS\ȭF!U||bBu at,E:,')[bv؞GٺEFFGQ 7׼:Ɣdb5n) XՓFG\<`wn|1I/Ogլ*`. q?-Uiޏkh1=C%,`Av)dWM!h&<Ch\ݜVz ):v[jJ`J2©eFX^w]PֈH36FaJ%'k%&:%'ye ˺^x+ i-֦r@7u=e&qex,nV/ $dkz2fDFp(`h ӧ"""""SBDDDDDDD<» """""""""""""" !DDDDDDD#BG(P!"""""""BDDDDDDD<_F0}ㆷ؟qP!i:i͡Y;^""""""" !DDDDDDD#BG(P!""""""" !2xebm{I"rc4Myn!L%KVJ|>.d811?qJ&0-6f\#sg$1f5\HVp%R}7DDDDDDjӠ>/~\a wN~bx%#9sߪ|{_20!O;EH3a+$|>"30&b8ޛIǹ?DbJ+"#N*ί$㕌[TC!w)BBDnIr^۝G3#"""""?*B49^A82 $}g"nb@-eNL|)#9A^h-a Zb ;rT6=7`K0>;݌ 8.c&)!і/,@A-M6ҡ<qv)11+ \ØyvSD^gb9E B [) SnE`-mssyآKcQ/8>AOx_ƉHm<Bw硲;!3}|:ݍO2y&^ByϮ߿oU !nAu$DH7F_[4ȉط*DFs5NFc&&1)nȜ|!AȪ [ Ns[=t:݀&nn|—惭"xT!_hN;<<YԄ.~R/^#!n= u9C%Ʊ-5#`)@7 7FwҶ>O+zl-z2m4KDDDDD^BҔDlOçј{=.íz5!)(4"ㇴoU -X[FK$r-!""""""?2BSy|GMdq~Jj !`^ +ݎ.8R|X%)(tbUMHbLfO#H".?Nj8)"7FVQPӊԻz!h4viڽÊҘ?1%5]`׌=İr{YyH'1+!udEړ#DzV¨oE3&)D?yhyNQת5F,99G]-}Ä\b$_>ۆ]23(I)ulۂ$$a[]^PcDH=#hX}7J`(ۉ6Z.FDDDDe6[ ~|JounG;)>7mrm.m!xq/c"/¿]p{W/c~/&NQ#3$ؾ^'e罿"aH t lۿ"a^^@6뵶9mwgGP-Hkq+B&G2wǖtx9kx;"s=H=2 չwg&m%Տpz'~η/PN#B92V!ng6ObdSb1{"3-j:6bF:H/ /ӈPS ^#Bz9swUm^+8׶ƆD@XbwOړfӱ$W%a]slW,]F̐fŸDkjzN"""""rS0x/F%:BSU@` .lVfZjAA9]˲uFvwuz!"""""!hʜ +Kc9%e} 4OaL|,_]Ge,ے[zJz_+ vkH3m;f^FbdUHu.[ 1 Rb GC 4*0LjlCHn񃍑iO~_"""""RnnD^ƄO*Th򍉼16.at[co@-yK~mXA֘5eb.8ئ9_US=(p,^)t t%w鳜9-M XaV jE&hԤ5ZѪ;\ػ]Տ~HjEdLhswq:XGOH{z$_+gTM1*{,if]ݕQnVƨ5!"""""7Gaug7ƏߠW) msnoLc.oa?x.Ct~XªEeqa{kqDSFD#x2Vmm=` Sp52iQۜF|}ՉCe@UMˑvKj:Mǰ.y4-#IKvi۟a%:ʰ1aLpV5íHC9dP('9\nޜj8rqr/ZATGiz-CZд_4yΡ30̃{<%s(= VXİ;˖jSKnM]GI\LO4l=r4\F|mL:,(ؾ!^;mY漄GA@kQhWl__3U(r:ue4CUȭXK_vyv;Ә&a*8֎p5c !""""נP ONY#Th=Ѯ?*8[,}| ͒\P8EOư<ہ}wil}ih.$cʴrn%f0QbӞchbִ{yj܍9hͥ)#_EE~CfW?omkvokQ_0+q3jNc$mF3l&r1;e$W9J(l?zi1^x [ity9A1>xҍ]ް[#z0"aH6\DDDDD ɓ^YD~aDw|.`!y1\cg,ՉcEDDP!7gs19oGChфc0R#m,;s "-DzayAW3mqEɟAY4uVUۆҷjqë?=+x+G E;bTqwg/fWe C"nj8ŋH"ãXq(&L[`MaȏZm:Eט͘/kcS1AC1ُW꣚%""rlQzYɇ\-&1߾xy?(S9 [Z5 Ey/eJa2{d2fnF5ݨ,AEj$G) ` %#2Jeio IDATd1OLa#Ss"m7-$ҨJU땕\71/^L!ژ{43i2a3kE&R!k-#nLNe_PLM;ΠǺpb^ [5gC>L!I`VGbc1.NUL+47əayPCYbBYcWg|J]Gb,fٻImi rD~NOD8rMFSCޏY.gC4-L#F8a"[fφդ%I*q%tFGPt"bh'Fd|5wRNyQRFۇ1i?wiiX=r?_oL"z&.*hWEm/a>-lOMr;_G*rhx $?!Zx},S2~?;_ (#&<IvS4-,Ę,h݊GSC4'r # PRߍ;s'Nnۙqu0Ajv.gNpbVEMeu({38fpLtGN`Vr0Ͻ? O`AYT-8˙B<EoUpgۻ;o$ +7D-mcԴN\Ѭ4}DĂ&ے2/f YGK7>(}:}3z-f{b.n<ေ'gpﮄ>84<^=;߼^y?u'<msvw313/?镧(:u5OZ5dNQ%B "OY i:z):_UPC7ޢߣƴ=~xE+"""3A|)w'CRIna͌0ntĞ%5阰0FN[DsW3/vuCnJ/B`y}Y>U&?}5K; "" e>H䷱{͸O}~nO|Nn9g^;~GI3 2,A:i >Dtk+}|eM`>^!MF;EXU3x5qـ?~>yxj>q׳<7\i=3˛ӡl6-rMw?@ݏk~Qɍ06Rxה9dd1eϿ퇕R%A4c;hK-ΣUϪ31&_.wyJqUY:5<kpy_xU5਺?Y~SBQ׫Lno~ͿoL3p=Ϊ5nܸ{?G׼хW_?>I.\J>'^~n3Pt'_,~59Vi磜<]g$u[ϲUBqee~_T2!DDD.S'EcidfF?XuISZ;\T`u}T4yoil؊/Ex'ƄXs!CX/9'_;"fꪫꪫ_]Ͽ&~yo6]'{+RC;G~'ܸĉEyIcpg[}G(Z^E??9w?ήL}3@^>h~ޝ<܎L#]=EhX>s'7\1u-vc4 |M;"21-ؐݫך`ЏΝ#9i;ym‡~ȉ{~5܆O}f[-}3(_4x){ ׿;.kǎ#.|&p߼7cm7>q 9#?%_NLs1(1ɍX&Gg~/eq8po1,Y2/_?#0ZYtc;So SgLn96[ZPL%cg^P0"""" ef̝[ș3yGB:y W+;cM)&x#7[uPB24m4 fSP0;oۜ~5\wݵ,t֬TWW1ܼKDDDDD. BH֦_s sq e 26v8OA,,(r!\suȕIAiӦqxc+US]+Ɍ=1+ f̘~A]8"""bP&H漙:禺("r1c:9ou\\C1L;nEDDDDDD䊧""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""yqT@D Xxё>sdӦM`L 沰tr^ɘv>WPw=3yYz#DDDDDdJ(!2E 0o^!XPf& 1v N8FFD.;}POUH^\AVHPuj9t/Ҥ8˫N^<b+ʹEM[ټG[gU| PlX\ӎ*<+7Hʄyu-Y{m?}e6eXo7_ɦ6ōy ll'Pνo5}\qU>FGNxޤq!ډIDDDDDb hfADLE{ra@/wG3[f)}BfZEznl 5I3la+p-WgĆᜌ YNL""""""e6Hse#V8YCW٧\AFfýɄ&" 1o!"""""""~"werak FOteB~x"̚3:5u];u:52%jtϡt@>!/"شZy1{*W¢>C9/H>\A+X&|֖)}BdXbIK٧^MM& !$ K&tRCtL""""""p!&'ĄŲ%cڲ-6h6Aj졏Pq55,|dUT,w3<~kN)"""""2!D51o^!XPs\]ӦM`L 沰t"""""2j """""""W!DDDDDDD$/PBDDDDDDDBA !DDDDDDD$/m:S] q TGDDDvT@Jh܉D@$:sWI2= ?mP_[.XvQ<}G n\%罅P9vmx;+wKtGk0X~sn-in \,S5^+=W'g o,*́Hlc$( JOYBe,^0BźU.3}i5 PB%lUx2/})u^( nؔkJ8XnZVX*|^.Q*a<<1z\̲&?Oٲ̫!0kfB Fo+ c-ʼ;6%mK=n3Pnf07}'@$;M Nq웥2zr)if,qmO< UjPC}OmtJRnl@O,2gL0~ZkgG+𸭥\;5[;"87-oﻉ,^E)qvⴻeJ$!a9?}&dڽqq0o֮~pعo^uX~%@.*Fwpzp'fNlGrnvQqoIg}{6R6s;eV=&u7[ ' #nI9DDd)!o.\dtRFvR8DGR\nO[Ի\8k'<%Rg,EJ%=R d(wY9J$H|Ge 1џVϨ|/PV롪'9nwyg]LILp  lM2٤;^k>xzY9/O~bAl0Ҵ [FU2A PMIH~;Ƿk"VeJ_] dVs]f9RG9pT'ь4oP1¹֠'{]RKtk=0'䥵n,C'#?e YHlTk,31?9caXn>/^&q?WmLm!Ba,z7 5ߗZaRs;u0ǣ8~Fp#cge[#.q]tj8 fL| Hލ2z"yPm˗Xi w@4+ ƻ.C' f0c4]4Gin83 C=^&6`n^N!ʶF"^i7䬒~kfXJ%߬;8fw'^ G,б(eI7 @ہ"V'tɏ/!+K;;12MDD" pP{"ApTrM/u͐JX*F݌,sStr\oSaW&LC w?IziVZ{*9fs'c'tqkmj4Hrf'20pܥ'R88ʝ8wyF + .?K"De |2탨 p?s/Gv盈\ɷvƃń3!wa$oi;q'to9"ʆeK?Fy}B,:`Bn A~QVư[oG)M/nT]~B *&u\'[tQ_ _4\iX_ђ/dq {i' 5I=MN&3"䎂"6’N:4BdnUrZ7yV".<hz9} =ˡ=0J$v0*[F~_+m̕.Lk_2K)K<[dhXn2(mQo5@81"LVb}q$coį}wkh?l8 };+pqb vZ0Z3'ļ>&"vY@&5i3!<ߌ ϺQ|{Pn,ED"f Qls[DjW%0^&^-V>)ы'#/޸%K7GppT2[o`oJg̡Ni҇$CsKe} `7㻔'kjsu-)欋WJ@tXPOJxa]m4L`FI'}zu-ĸ^ۑ~":ڃ|ow˗}j (vpHy X\Ƃy\{lNcph~Gsn`o:EDDDDD2QR{Ļvܬ8XN͒Ln@jJDX˗USUx{@0eP BDDDDDMAKlۯrl55דp7o|?u,_2{}0Ek:We,wFDҒb/(eɇbtW7)_ǽ3qE{1_˅xqKY& ]+IGV]q[p:/$F4&\¿^\Ʌ ;}y 36SU8}D>C(ԧlɋ"PӴ/wnXC9>{߇Zxtc3}5 \tl9{VS&ZʟNVϫ[LNFrcݟdѧV1xO?i??o-YIN PlX\ӎ*l]ۚqoKYyu[y}GE+m)S fZb[i9^~s޾E'z~?0c/ęsΝe*n=,r"9y^Dž,k'Z& Bd ѹ v 2z^6]̏}i|L :$;`|2^PR^`ϴ| K.Z0!.aϜ 9hDDDDDD.&MVkBqE5 _Oh=ڗD2o] ?~U:%&1t/N^ܸpK_|%5}w=fN_neP t{b¡``="ADq(쿴̚3:5u];u:qP fDDDDDDҹlwL tJqhע=tfKsE֬j[qKĢ_f_T9<_S_wߗ3cnrݼivb\ 2v CÓB  L""""""p!P1}N삕_UMLX&&/Yf5?|%:BտbO_+u',3_`FƸ-,]@I :ڟ2eȺ9Fh]mli+;yq+l]ؙ 9d=\[2}Γ%y*LZE"y0ãtp hN]0KsH\6Acq-f4y#Z:tb6Ŀ%FdYP0O}mb/ҼzϷ{V1b%{?dJKZ.<0EDDDDD$_?TBl|?n㻝97k>;Sd>9t3P>Ƴ2~GX:cFDwќ ne5s{icz0E>{>˖sz\7אK51o^!XPy0ce{ fq-KL !D.Pƒ겜6mfRX4 |) US]2(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""" BH^(!"""""""y 䅂""""""") Qh,IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_outputs.png000066400000000000000000000250411476650141400260470ustar00rootroot00000000000000PNG  IHDR4?H pHYs+tEXtSoftwareSnipaste] IDATx{TTdF:`p "PrVihNX'#Ĥ9ҘܪiIlRly_t4$rP"5#PG.'f: tY,ٳG˳'tuuɐʏkkCK*00`ڴȔ+(۾lmowzN i&L8QvmL6ph=G(8Pq @!B3g !B(8PH/9xN6` 6ođ3g !B(8Pq @!B64tv2`@`@dDD|| &tuu =rt%&邂|ucU]2>'@Y~;imhDDgmhQ⬳2 (yաgI^)wE~4Hwy7o YrR#a9*;"RcmEiy&!F S2{"'zěX|yc5>g~Xsoטln>ntT~g~孜VY~ #􁬈aF!ۙf5&+z}s6lfΪ .@EzCGBLo0|7몊Wm\kޏq~+θED?'w򔯜[L/ԹE 9ygl6a}e_X֚35rtד$"'sD,7no1?ޗ_8:sAN }sqW8gy[nIu/eW1gz yt w{YqW DDuyW kED )oQ }nh=~{G$8*ߺ/nӇ"b?f [x]qٹōCf^HT\>\A58,l/ ψ,""elgMxlbzn΢]܍+:hfo1b+qƢ-/wReӝG}]D$zzf9WLO[5:c|Y')l{VrԛZ"شluGJQ>|s#%/玅Z-"qq{}w{~MZ|M$,a,CE$$:f Lb++x{A^|_3Ϲ!L99/OUHTuavtԂﾬ;SXx.2iŒ5""!3w/訬 L}8[: Y|m7[*v>Uu0#AϼguN|PyD\$5ظ|%"ML2mk f]y\dv=:(8۳Kguzs'DĎ=k%gq_xᐝT;,jfg=秠=i綕4xhsr'333W6GqKBrRg-^l<¯rjk[M#&*-9~""2 ޷9~'97${{]pqr ؋>ȝu\]ш[TiC@Hi %gڪv>_hNY~6!r)YI:khA[ )Gѿ| nE^>3I.?!7dO]kC-><ţ-+q6Z5W=ab[V&ݞ3Dz?sf }ю0g͑-ǭY)|[f"2$ggE|P=Y|Prڙ$fox ?EDkJȹ}=[;Y\޳GUvrւ'{8Ozp[H+w,bNZ˷nݮqLZ5}a^o+ËwYřg~NRYrސ=xsW/5xDD;֜9[Svxg%G`􂄈"suI󆹒,y^%|SF4 >DZNcNlMIhb?7<b9R&)Փyq |rk=Lszޭ1"n7}Iw[k U'D4q 7CD\|1oa;/5 " z_u!UFtl+x ̭gܹo31gů≠ G|eܳE5n2#sr>YG[W'-4,oKjcm{xGiY(|{k('wqVڕ/O WѶ;eLnG|9&Ҟ/)+}>h!3}W!=+)zd?~mtU?~k(C%r]z졧dw 2\ӻ%Wjrxv[h SE;>End1oopGKLj3_]vq}Edne&D* wEĔ(WɎ>DfnKmEDD"f&q;7~& ^屘Li$=VtgVDĐ5pics6<9饍ng|wq۠ۉHkyaQjv|E#nߎe69c+l\p[ϼ\l7}enfMZ`i%ӢFw<q3cmYɫn齂^0~4jM9ߊѥϛRHZj׽TTwIDdr?A? @r/Pad7 jZDDp&?Ws>[K3?[f.!mX|~[4Q9?]e߿?/u]""㼕W̾͢DD 3MRy1&wx>,?K1Dܐ륷UQmYqmwtZ]Շ~bCo' ~|QMgT7LpׄyN7Qvh 1qoqVYY9A|3ٳǎUuɄx/O*v(R%T88JkCCg/oA|3 ? !Π,9Pq @!B3g !B(d93 vp9t|0OiMg !B(8Pq @!B3g !B(8Pq @!B3g !B(8PqL9=2J%kXshjXyc~Zzv/ޫh8X>ZT%bt˞O-r\#!꭭ ""-6 Z|W_]`.}(RDfR[2.Pq6ޫh?ݿb}J>}4\(G\-kJ}hM2rdQ_i{Css{֤G6 Nke~Y6cd˞{MQX9?]}uYn$C:pl-.Bs?&RE"ݽ[#EZvo+^Wg9dMPq66C߉˽7]cJK~Ydfec9Q\&"&"]PBTzPKP"wrv2;| ȼ^sV_i{4b-7V]uf[pTg!cM]{Y*"le1zk"ޣq6Zc{̬;bwaļ<Yr}%tpCpv M떧mքɶ$vmz歱""%q8evknJTquOk[Lzk=\b'Jz礸Bsˆ-7֕žӗHL-9ز6D 8=hVp/N'ob)A]vQI" f,nYS҇­WZJD S҇Nmtɘ,ZT$vmHuovLp"||MDrK5W3 6&m"n5X: 0y-DL\h8=Nqo'Os5>8fՍv?~;cGgYMaibӹZTURz6kB[8K$HXն5m"^Y X#Fosg}{W[>5Ս+g IY{g蹑c퇶Z0[}U[{nz2YB(n\刳gϦ=r]~8^q=yDo+7DD45M0zl?jEㅏl}3mļ|wv?QJt4M\x|vs?U4f5yw;Wxjbc꫷%!꭭X=LmeP#z]~$DM˞OW+5=unnzqKׇ;pu1}{"[;:r+w^2 wkBO-} >~,!Ց'1ֲGZ}D>K+H8K}~X‗,{̥z423n2ٽjSnYSW05gc\!n8{_gu2> ݚ !B(8Pq @!B3g !B(8Pq @!B3g !B(8PH/[j.8 "=&B3g !B(8Pq @!B3g !B(8Pq @!B3ˏkk>l0mZdJR˜ &tuuˏ;;=tAA>l:{gU '~;Y[kDD8;:/̺邂|;%:6*O@p!Ǝ1rzٲQ061lAϟo |@q٘2F9pa}m %t` 5g Xp:qipDh96ŧ(M˛6U7N[wK#Po*jN;'?n8E4x=#n2Np#FmD4\Y\4ٺbir?hzU37/$""ߛ-4%Mِ=9˿~ظ%\Ht]OrE˞Ok$&fj]R>"")I&ӯœnVoկvR."a60:f{G.3-9}zliLqWDzU |\ߟqײđQ9QMFiOՊ\k9Oip{$2iHikiq, nӒs_gNl!0JhMpJp l̘sr?{]5G#1"3Gk"wM"z15"/}0Pq6Z aîcsn}xLJs<CFNDf\#*87ɧqZsii[橪w FD#S'-Y󶝛6}+lch*8̜D/"bXhZG{[FO1;'G0=Bg/aVXN1Mo+hj꤄u;~鞄cOB _KwMd_Jq8Wô"nqˏG!3;qGQ~jClJ޹/+Ef6H;`xKzQ*mnCBT?0&tuuˏkk\0mZdJR˜Jyo|q. !#KC8p8oPq @!B3g !B(8Pq @!B3g !B(8Pq @!B~Rpy15B(8Pq @!B3g !B(8Pq @!B3g !B($Џ]~\[kai"S|T~4/o\~I asٳ> g`Wi;:nO`celw*"Ҳwӡ+Kck9Sȵ".^M<ߝHg{qIcN֠_)hb}Նt߯"bkD-_2s=Sĕjeq=mD'ƈhi:Cth\eJHvkK$bHW4#}#oυJI ֊ojD/KmAVq|\.I"0"οX\/!s.n""2E)4(҆=詪wytӔ%Ϳ3OO*X(mq6ɚ 27*М7]'9BOo4d^g;,!ˣ{J}õ)]F/㺨HhV\@}in-{o>OV'vkCM.On8-]GO rrK5јOo9 Ě2"Nt~֏<":MԐoyD4"7?oS{48= 1$k%+|KU zgii+mKD035&Mzv;'?gLiCIh/><-{ۡnA "2/og{˖~[cw=s1xuR_i<ۑٳ :{60o4|#TH'hM\86-gU|3ٳ>6-r6rډD??g)I G̅&0mZdJ/ b(L.]s 1B3g !B(8Pq @!B3g !B(8Pq @!B3g !8˖3Ȅ.=8  @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3g !B(8Pq @!B3 c)$!IENDB`pengcheng.mind-mindformer/docs/aicc_cards/assets/train_task1.png000066400000000000000000001610421476650141400253510ustar00rootroot00000000000000PNG  IHDRk2 IDATx}\g`$kPP˚aL  w2Q~5yU&N6Lhj[LB ^L8?9pZ*q͛\8O_.]5 v.Uka iqF#'a%[y쬩P;wY6u'MEEECl\ a00@֘ԷZa(ka{Κn:g a !#5v;BX`Gka !#5v;BX`GZ4tnOvN@X`Gka aapo]WRIsKW9[m&pw]чghحU7MO}[94Xià]sa-8k5Tt֜LՎK R@cьšt@9kN'>!^$Zm{]ؤi jW|`B8M4haJI677vWFI=5aP6LuGZ9a2$iAI5zVhOeX9Ʌz˰9N)9[E O. q~T]^NUwBBO0ݿxXRlּ$OF{9/}^QvgGW9P;ձhDmXӢ+~7*NjlIR%hyG/ѹJl!WߣjiuD֎Y?lUJX1 aG,$)W)һ]asj2< j`'M7zki\5miN*Mz3t:Urv:mzMS䞒M^;JtA> RRFkI$&LҺpYoJT!XkghXI'7)t5 -_wv{F=Vh,Кy %)kR% WIi+F%9xbWR/)u PhcTѯnұrI49 ik8FD$=.z cDE_ V۔|P3QkVLhJLtc|MY:t0_o-?46^ـ2-gZܮX9cUPSW[ʊ/xšoUvTC]{S.dԟtiq*}!>jTuS|{3Z9Qq0(N}u,k儣%+qGibw% f`wҔBq?(z %!׌ | 6۹g9۔PAג5wrzm@z;I ZS_ө;,rRHOnRCvZm{/|jN֭M|Zl?ЬG5wfBOVdRo{/v$T}Q"Su(T.H'7iZeE&%œcGS>Y5%oҺ9_=0&7zu7hd#v-S)y]5fo<ԯ3S/LCmbGd;U%!a jcvlBN)20EYj3 ?EJr?O6|Կ9]`7Ha$QAȁtW4vm/aϔH:џUzƾzҭh,&х5%qM~$gJ\C't*%YH*/ԧe*> S[Ij$\Lr#V쳬Av TV&֥wIW M4W=0jgs#)<6I(IJdp&ˀMS;s=m5pr9>9MhZH9;ItI{0QZV"3@"zϼZOWӌKJ%rB O֝0hڐ[!}z(ai2c Sl6k-^S1I3C<|CcjMI.KVB=&H@tw-wF***T^^2ytcFr%:AXmUdy,7wԤ+<$7jp_7tM߉uzv(gOOWu4Hum􋪱ScGUN Д]Qn<=%x'%yp4T6\ӒJgּQ:9uhidͷMgkdA4m5vcK]TehF.^*cˆ.aaQ}~X7̈́cKhCݥFѨ L6dmàa !#5v;BX`Gka !#5v;BX`Gka !#-';'O_J7̈́cKz4t) a H|y+ QKE$0a @=cH P}s/ PyK@fsM!'lU7{4*YI^S~1Jd)<]^Ynw0uj4jUÊ7S~Az7H S[0}Gga._$K '>o9/Hߤ)̴3&=7QӮzECp/OzsC;_wulߡŝ T%;9ZŗqprBPSvz*r*?iN%u_5S_9m|yNR#ԹWɁ2lsQdXt- q>r:Ncj sǥIwT:IJeꖱ@9kq9PA#hmДg-(V(UiZC*ܤаuJ8Fzwwfـ=;ENOtqAz7LsS:r?Qh@8uKJ0ZOJ1z 3Q! RhL8vBצikT ^#JStښISBa5S{Vjxz i+=k@,3)zgQCkn6ވsJ^h300DcUvxZ;Q1۵xl"ӋєLO0-N4dfw6$5[{_rl =\uSAXim@h:x}x/D!z6&]%5̰9l-u EXQ F}yiGib>SL:+2~Wl?Ir2COQPW[ҵ/]!a>Q Iӑԫ.]5Z^bPXl?ј{~m?ߟ9~iUo?qFY?gogx0q\@{1aT;{qFӧ/l35)w\6VfqFƔNq㣌Ƣ{>>hؑg4Ƣ^5G"˾1~mu_[l;g?`u 6Fg_Wfew=8r}aq'VeqόʌF0L-Ӡ?7⏌ FĘ2oqFsJqШwO|ݪ:>8uFUƑUukL|V}QͲ-i F^T2ݽXφ-5W#5sSl^b~|T?ᏗkV]n4g 0`+Fce;QqQ(F"?j =w#k|S~w׼9~?{p74Κ\%:*ߨyR智g[߶qxN+jE>2_sꚀW*uBLf@? Vj[(Lt.ZyN/$'#Β$'UBS7fSNv,tdwJFSH[]K~\U\%]Cu|wZekЋzzP"'Wy"g^yZSҔn䨠t@,M35M㻙4n}IW6>t]INN5:S)DRy>(ЙAw_1:5ՠ7;~DJ:|%)@ mpт o55;aд!B"W9VxM?FR)N-$cּ F'wH2~ޛ֮ܳ|ݬV:>u\kRI[*8ZGƛ6QA\c EL443D3:h?ZPŮ/ToS%NrtZBAs(VAO}$Ƕ=t~æc4w\Jg+6&K^ F'GI4 j(HzMB$9xbg}.H]iJH7oE7ՠv^Zawc>.KW;FQ*//WYY<:rO$O) EpC;pWW.Y!# #5v;Ң f%R:P(j;MkVR/܏c':$F^:>yEhO~}qD_2wi_Ψr]<$]ۥ*_*oA<KG9 Yoy˩P[h{J.X")YH*9k<'7RU9>5Q{WM9LZ/H7rwv)_?'ՠ/µ.%Y|/X^ަ0]~>:LAyA{:V?կrSǴt񸦜Sߚh=]uUNR7 {஠rW.SnvkUFsM}WmqN)U;Jcp:Nb~WzYia v@Oӏt]a*9~K*Tr9  I*:vF~j2nҘ~.VWGV ӽYCq߶eM;t=3thc4UQQrɣP_36):}W"'I|i+@]Ѿ?*zjbUtۇt/)%CRʱoU">= Λou0 }:<0a ٿhk>_ +J=LPЄY`G#5v;BX`Gka !#5v;BX`Gka !#5v ]h @cѢ p{Xۓ  kt$]+=h#GI|zCC5@#pjyܡƆ.45@#2\o2w B[##5vTv#3%%֡5c-D?<'ϖ [ )*uz'P7Z@aw ]2>ݣ=JSԞD5MMy__o\9O?34Pc"o(/VVM?7BM`;KnW45KÚCQ$*q2%}cGo6l Lƶ+FIծQu}7iRs[k4z䜷YǨaW]flnk ?OìJŚ^6J_Coqu zNJ}KB5ԖI7-ݖ{fs6թEiZm^ J7/6S/h|XQѳQ+t"*B5(J,Q7|{Vix}e OoT^Wʗ[2^Rw|w2be2_,tC[oFERP-~&%Qz^Mr;^h#Dm8/5N֮.KkJ4bwt`|JZ}87OW3[yۨ(7/ҩ5W;JآIտhz7a"n ՂiZpO:;[^C}%`wOFhXtp$עIٲTo|eOZahz:Q3[_+[c4SM>QP+#GKV)8 YZΰPc!heX$I&tyG8(qAѹ>-GuGCj遯wM-oIz~$_+vT?{_́F*]Ѩ(/=&䬍.m?]/kfTb)rӈ"]9mwXn][jv4W/EҾœ&inΊ/#lt%xGӰIK0y6 ]ҞupMrj7AK|5:5"0)@9>gsoe_6{QX{t+<^2TELL$eSci?1!Cr=asnQHN:wwN:K)odU0cA|z?=by+DӼL\n\/:@7Cs;t ONJ LwZӕmsR^+zeAanWn/4Ao-K[Q2[v{=rW)wmbHiؙ-J/ӗG4bQZm۲uoWXM|s*o3[9tg%3MPiV_ܽ!MD'm+ϩQۻDswכu U/hftgM1>}z^d GƘeXd[sk_!9c+fl^q\x^W$ZQcǙRM>#d/5gu[V_[>nG˗$dKO=?z@s-Tdc4w#z9k핫ڸ43At4%搜K ktט\6݋ 3vHnZĭj+WqYfh֦^Sؖ IeMCTJU|.k~5 NjIzw M;%ݯ]'ir%ZgwĮutǘTKLsUS٥-I/hKn]S dkؤG~CId-F <|:bcW]OW}qJR¤%&myV9 ʹ0[ k:ᥬnS88@כ_P֤{P kRQ2t^`hԡ}kvԹ4|;kͱ6e C|kt?,wH|%{ʷ1eKUހ?HOyrNFT~rt{qgtȿղOO~iMEtUA8/^a^Q%k'=go>V&ujͻf$XT\TRNU]ɰc'ֽmv֘MUf c0u䍌UgGedފHǹ ՘ C=heE(IrJugyvv̵=_sX }Pu1}Zg+W_21-}^3iWTw[gZlSwʴkYFEҩÚ*碩v-Ssdg9qӚo1Ie7OhmV5j$~4l~l=y)m<1I 魤e ?gx6;kj>w<9t)qAߨ2q-C] 1OZ<}Nz7Nww~(`$Y y"Mvv{5NzBzicok<9pi0UvTJJoN [EzVO)ۻD0B^e"SӰ ʚHU}k%*ֈ!9k۲i츨paj"^PDϭϏ/hNUhTEEUVV&Nm2po6Gz%yV>9M5o,7ja7mS *S y\%.\\WF U;[ t3Pƹ`W; םTaH֫.YR]?):MNMU+BY [ziiʱ`zUTcLۭVrUrZiD:>Vϳ>]]tzՆ&ZeY2u2xo P׭:gL_љ-ؠ̛HP[uT?pU&vsXpt7\a6n2^j#GI y4zm!Sw_=K3 k|]w uz'߇R7tKk?'i*אa 쏽5k4aP͑CG=HC>SRnأw̝-?PyaR4a-=hLiBm @;BX`GkFr@CC-[5~c.pkmKW;FQ*//WYY<:my;BX`Gka !#5v;BX`Gka !#5v;Ң p{s屯TZzKL8:TޏʫGCЬėǾ ?yȭKL\TO3ADi Cnt4;BX`Gk { nJ[ʙ:n 6z><{DߐniB~ʡJh 0 :ΐu*iq5I*~FkNjǥ h5ޣ:$?ښJQq0(N2 8=3&$yVhOWUe`{VWo6G멪EVN<,Ҙx9IGykhA7;7wAI677vOשhdmXsmt]W~4i`dX9[().2l3&~v,PPL5 ]*xEMWxy$/Sxr|pfV<ުРœ IgPRZ=$EuLh1rJS#?t-K9 ۥhNKWԽUΥ*(Ԏ|zuFִhmuT=Y%cyU"%+gOj9SZ9a) 1tHYbFެ˦J֎X~9щ3g[uNK5IR%y_ы}tg%W{')nAqVNhJKh_.kh8;ԫPΩTm^kyQGݪY9jDaMrl$>3u8G-1YIwNꜾ&]+%PyKze_:kBLAGzisNצּ(+sGy+1kF)2Ԙqݱ@iڀZY?lUJ:z?쨘Z-O2nerhA5Mپ9ijvv1].U%I=_BtMyuKrDx ҃ô(W~N%n9;v])|<_MY%UhTEEUVV&NmJ*;W,マrp)T˙RuVSG5 +>qN9aM6欱ݕR%czir}rQJe5QAJ G9JJW.c"ɊM8!?(wJ1àho^Vɞ NVp}v>uarI>i r!àf/sI҃zc[djy)Y{5P=yOhOgS/LCJ3St-!WF]Sˈ@rKej1) 1V/P^hUNJiyNы55/|.YTyΕ|ޣp$;(vtTM6)4䯚ol 2tL(o"]c{P.IW{| v Ӱh߷[(gB96p/=>.IW7o1TNdHjLF5P١Oes9*U7Q:ܵ{$uHx+6ha8^.Kp~fs+l2 fLגgcyfY֩ U_EJʈ7(<~U'N3$N廼2f%)kddSLgI:D+Vjf uA2,L/&7QRU>3&wUw%H[" P :*ԙrl9HNW`eDcN*gW0S h Ģ9h" &jhsNv|B+s7#۷`%N3-KKvSevyysscϡt$IjZ]Xsak&Е_85_\4.IۘL|RV2^I)y&=7uVo_YS? Ծ cC<d#e,@MM e`y_JnO!M8EV:UmR<:'cc*zfsW_XzM ?_f9wm-1HI]׾)=\/ ]?vŜ$IZV֜ t ]cn!vAN*5l~&w<џ^M@HI| J#E%} IDATm`[G1lՠKqL c4Tߔ|7M M~P(2?֟Ձ_O)}~y7šRy(=dbߚ(.RS:Ȕ]$k>51:$xyLF/I t#e:}gim>cX2ȪI]fΞkFϡH&0czRy,z&&J \ܟ'6fdϚ*iKش6/`GTԄz4P35*MWpzM`t5Hē>9j=[h c'8 CY~N,~t=n niX;e[cYIjkqcwv,NSJuɽbg)It),\a+BC׈nfeϚ .aIU@&kzɤ*fBjf1sJ԰ e̜S@(3jOW4VJIwԇyma{Ry߶h{~_JJ;߶@$IZֳt]ݒIג$I$Ij95$I$IQİF$I$)H$I$EI$I(bX#:ueZzڐ?S-= I6CK@R5[z(ڈN:2zԈ$IRdYyuh**++ ~@$I$IR4(I$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$IH):Z\yKERѩSGFA$IRbX# Ա\۫gKERS.wa$IU4(xsIWյzZ'I k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)e\!gSH$I$)6$mR'\ӹgM3[͞'x8xlыō<͋?^sS;˪{ܬh6Cܬz l fIԡx%ߠWv_$IR _\[Kkd(2!3 w3edr3Ic=;:,,f'UZwWJz&- >'x=kdlR~dPj9ϓ@3<_sdam<:M,Ӄ 3({6-1'Rx8n?$sgrڝÙoL{d/MHBv\H \þp^WvB ɰo/EǍ`4_/LI|D<^%ISFS3HXΗO0q kr_ލ\:ʙ57'suߍ; ~H$IFNW6 iɱ} ټx+BHaPj@)SI7 wd ~ڏ#NߟN.ޝEōLOks2?wsEŜo]OmN7*yJB &ljRYyqQR[' L|RGQR|3I@d/ ;t}HK,Zz$}߳M 2"5 V2tiCSjOсޣG@iΆޭ E~qa{~5uMϪ#&(uǝ룓.)-a؄[0q5;Q gXqg/ةLhmdmbdlﱳ(RP%Nge$ҾQ+gaUn l$IV+$K&~q6&80g"eu:/.#~/y&=7uVo_YS? Ծ ffd3: t6Xj+B}uƵpw 7'|:.%BQ-ePF^/oTۇ=tC~!76#@?=+Í\,t۷ծDXBvc2ihgE` UuPEQ`4(zqdns|-1cI@!?5YOǽh)i4*aCF?s^$I$]Q_ᰦ8,%U)iJ_60̭Tv@Me ՔYΦ{"Ƥ5h6)n&C)&XV7ԥ]ό?ޤX_M7zF&4 {47*Rаe*kw2?"j AU Ћ3i,},]u)(ݽzeA|Fo$qcr7B5 zWFXqQ1{;bIb` K㉥R(}.5m7NIbNNKbҔ@@v,+ks?a ;y2s؛_!"I$ 5 T~6I&U( sqؘh TϚ2Ӗi!l^̏ @Mp&fץAtV%tBܩ<"79))=@:sҕ5-R/Dōc{9dC߰Ǥ弲~z10 NCGxx/I`M\;4MFYMӰp!;_.$vT=xa\_ag%p%V 3PȻAls.i^w5^TI$IkΊPcg'5⿛1tٳ&Kش0!0)V0v6SRo:A͟NѩS$IRdYyuh**++ ~@0EGK_p?oHj#:uQ#HC$IjS k$I$IӠ$I$Ia$I$IR1$I$I"5$I$IQİF$I$)thH82ێNfWߒv?ҧ]nI_"?PAL]mn>`{Rq TvǤaL$I2$(穯3gl~vejb9Gg(yQ!^t*#[3.>@o|ALJq+.T$I p$IQ PƠQcy mc.pӰsNp$3^\Ӌ|\rSgѹSLg1G$ItUYY#IR+㓱ӭ[WNv|'>Ĝ. >u={+Pɩ^!돕Wveȭw_{AY >/)>n_Ĭݮ J$)a$IQ8Qbv7 jf>z u}?y{"\7Q_w_񥿐$I`q$I^?lϐ./o#=oƄc:=Ow\ˠ1|~tI$]]5$E~uS'kg\p5Cf071mͅ]uо= HkNSc$I k$Iz}k8~S7n9Up3ts73csCVŇ}I}32 9x8!\w$IRHeաjMPuݛ_TB l E|W &nVXk1uDv Nph˰ot3$It5HԚ}_8g`KE$I_ AIԊ]x͠F$+I$I(be$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQ$bXjB$I$I@c55$I$I-AXӮ]; l$I$I&z֘H$I$]m{ִk$I$IdYyuhJ*++*C$I$I!|#kw5ڔF$I$ig ꚔI$I+AXӮ];kl«m-$I$I$I$IrD kB g$I$IKwK$I$eH$I$EI$I(dW$I$I"5~Xcx#I$Ite5kƒGC$I$IheMUU$I$Ij4j1I$I$YW2$I$Ij56$I$I"55$I$IW[&PQS5$I$IW_I$Ig$I$I&{H$I$j3$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE kVLyeKAZ$Iҡ yv=ͦ0t]9ON^lᑨ9 s>kᑨ9:uHsG$.VHJ?ÐVҊDIH@lА$I(dYyuh**++ ~@ۗjϚ 2Hs͋:d/{mdܔOIh9?QnTVffAt7j ~C eV")G/mypI_okxrz.-&7{ݽVң{;ϖ;w+hp<$~S*WڕCGܻܾݷb[4G ;}sYwCv{_[Ľ]KT`c s>kq&{K.B!L)9CH*u8j~Kx@bv߯@(̇oD㑕=>^sEEe+w>y;P01iW*#1oz6;[bGqY\yQQYSON&y;PЉo!X*PM{or-*a"T$PW&ߜs;>`5M[8Ij IDATE5|hWЩo]? pNckryzN>)Ӄ?M`Y4H,*u*k9jtoGYݟ gŮoPSGͨ^y)(l,c8VYMLϣ?%Xp"åwaܩ`<4tMLjI6EU&?Zo{@N NMbvM䗻T\1nWщPa>j̆'lWˀhՒ V϶;gܠ@N -s7T֛]k-\Ł&HJ$r * ~KŎ_q[cQ˔8HUyG 5cL%{B 2n? O 3BϚ96-.O4Yu25'62ոS@{np6,΃c:p7dʞM7KYD֦4es0Vyy%l QspzyQ&;~h3a};m٨44.&gà["9]ss7уWwǧuI$&f$ >mPL58tޏֵp9/.L 5Mg9idx%J̬ۚGH[_S2nа8&q ovV U0x25 5/ړ1β RK3+ktww!l9 V V cH,* #i:9/xu$=741-5Isױ@#YWؽ/_ JB4H9.Q;~-s^nۡ]0 9~[Ug/r+I65!|NŞ7wtb;9B7%J '8%S&z҄*kPUbJՆ2Ӱ L>%(z5{T%DYXǃgY~ځw44rYԬ2+y=϶j@vz@EёX1w=sӏ #M Tİ5@/h *0{)|kNo8MRVve@_SAV/уá|ɼWMaJ$V\ؚt)t&nMǴ"y*?EB YI)y&=7%/QYq?H ~`y0޶5R͝Н1~rΊM<O]>|\FeMJ\> 8YaAN m&W2hD_呹Ø!Y}b&=V[Kʚ£Eqkذ>wM rfhtdt|o'n^ɔtO·f nM]tۃ8wx$IT5:gzClZ8_bxb`>4ߠFQӠBӗNgu!5KN`iCXnUK5Sy''|~L?u7N^~LЉܽ-6*rXFsX.v$NdE {4K<8r'x{çIKA X81ع"xVڨc" %0O)܇^kس]Ns|gW?'qLxS`So }-L~Me$I5/)?yYuO[Bl-} {6-L45Mϓ=p6jn}i§bAfMɜyx_i+]YJѥӠN K%A%L)vГ;+C-XZg`&ڄs5IJ)#;sꃢqak'Y6W2aYK4OLRf'NfB8سN5+KnteNUk#NfDzCy4Kk:iz Aȉa~F:;v 5<4bϚ:8.} C0ŪfzVXz0򖖒$Ijڝ,+mTWWSUUEee%ӒcfZzB֮''/HsH\]vi!H$]ۼH$I$3$I$I"5$I$IQİF$I$)H$I$E8S^f"I}{#IZ#3vt$:۷o!H$]v'ʫCTUUQYYIEE$I$I+k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1Z3-=)UU$I\ٹ4~P3]9ON^lᑴMݻK-= I$ FjΔWoٖ*\ I$E kV [{9ÐZ?w.$IZ7I$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQCKj۳&KxꞄ˺ed2}fxO%lefdlLgeNrNa;^Cb{ϖr,˘H{<xdIv:AxS,* +>E,?xuر_e泂<;7Wၹޛòo5#1onsw-e_2PI$I5ZmXsnB3y/.᭥uw[̭;H[ټxpONk/Mޜul}Y@j:b\/L̋+㥬jӇJx|[px;ŏ^8vEw%ypIO>]UKqi%sݳVʬDlؘ> 8VnϽ˄u@s9]7Y6w cuLH|>047@I o$I/Ն5Ѯy+)]y!]$2i"U@"s/ z}E~$I$55NPQ[*vsh{j^lA;nu,#`fVS7# 7Y gҜP Gj \2CX;:Zk&nM!ޮഥѽ=wCwc~B oN ʚSW3u H8Vpnx0SĆ_1iisH$I51#]OPn܆b9Gǹ?~ob]AΦ@Lv:YSJ|JaLɱRF BpCi+M?V TzϜK:)yiJ MApl9[1h4>85=tSʬDY=,i:vϭs{.gщYabmןc]}|ņ+I$ ' s*I|:_+*MaT LK&\OtSLɱ8>p< #>/l'qVÙb6O 5rS;s]ݘr9a`);g95l+Gd\(i8 T5j{[=i4 0h]V J{Wa:eED#I$YZ]Xsak&Е_85hX\D Sl^4TVJRIҦ\y$vm)O['x(m+aj›< u!|=g6cL:U^|8Z*Rme5 iiخWe«aOdن"̺͝3ZU#I$ZYXs3qƃt9R|HGIivR_AJ5iI7)jk p)ПF rŒOEf^.?57Rt Tt%1=GWi ʚ^8ծ@~4@MC6;~-ıve߰U,?ǺC-`wSӠB .-I$/št:{=^9=7<졔iEѷ9gia.OyHjټxV*9N #xxkfij35^xrz7xcpӇvIJfVDVKOD<,Jv֮iP͚¹wt6' 5&}wJrv̄e$ItI,<{򎓖WԌiPek]O8ҾFR(` aϚUdͦ% =3/M3yHfMɜyx_i+]YJѡGrپӑ^Ƴ6?czvY=⇰vL@ga,ĘOj; jG!4}iWN p 63 <ΣKKCj:v$IԺ/Z֞0V!ưF$I\eաj}Zrl$I$Im?~$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$I$Ia$I$IR1$I$I"5$I$IQİF$I$)thHj%/8ŋP/S5A g\M}J+ϰFj%cWϖ@KѣUIt9 Jj%.^ob]۫g3GS_$3$I$I"5$I$IQİF$I$)H$I$EI$I(bX#I$IE k$]yG~A_p%dѶ^{?ik /B__61I$ICK@ҕtgL'kOxԖFEӇ} -:|:XHYvOL{i'S*:~6`؞YȬ{Ȍ^[~{oo;?A0< Om穌Y) >[¿$IRc kCzn8 99LeRP&מ /4{K /^~aPAmv$lϯ&GV cIC<Ɣ`ǘB!I2$IaX#}?^Y&??ML 6W޹Cxr2Sx)__7nf"rD7gƗKڠ2>v?[.})O"l+)â I$m9GjI*I.a$oMbXfM`?y3;T39<`#? .\B 1wR9 3a&Ζyax &dma?qLCM0\*.#68˖=[G=>VL!ļ<n '$NTx{{,}m2߶2oEwX1lyl ~9`{u+?gcCֿdʂ$1+צЧ=f$]=l]Ċ͙N@Gho[YpTjNCo pxAx0SHֺB<G*I. vNX௿ɔO%CIb熟@P޶}7Fu D 3x21;,^ 4?rx\@Ps.y7wpu6ٳ4%-9ghX=o{g24ooeҥu ҧ޽E}y ԠDI$`R;gK7SUb7TBc5IUX5QIwla . ѴhFԄ ~>-]fs &S0[Ԩ(愈A Bμ3`nŶN DƎ=cf4m9]]C5˲ IDATA5Vq`gMuts50řT/[&wǸP Ydf}K7̴]'0#{RlcdNȸ`z1M!S W3;ي%v[N%F^Lgf2PB9ַ|*f5Z^e"O8vTGE*a0,Y`|,yw>EBE '$d!Yx,dz20`dVh0wu"?712ƜD*:i5%1BȺA|o赌"$'WhPs 0%jmcx~PwN"B1Mr_ KyCaK{j.$ExB"bV9feiݾKMn{5еmZK .S5h:Q=V91fڏ$""""2䳋Wihh:<&"'̿=XhCɋxn֋,gg20ļ˶Wȵ)SYAyHb5wjy(l;Odп;"""Gkֈ8 g)9O,k1/MYM9\ȸ)MU!""""rcֈ8I0Œ%٬"""""%5""`!7A-""""" -0,"""""""@ֈ85""""""""Daąw7Dnʅsq]E?rr󩩩ﮈ0gg'|}N?""""|vJcAcc# SWWDŽ17ێA85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""DaQX#"""""""@ֈ85""""""""Da2 RSS]qhNp$nv{("""7NaQpF}WwEDġ]9974ﹽqEDDiQSS{~hwu#bn޸"""rֈ85""""""""DaQX#"""""""@ֈ85""""""""dЇ5a6'ϘߖqPr` W). |e5UpE>-sy>*:GE'),:IOZ^&>_ "Na$dg@ ) duTL0>}uPWDDD݁u?% 燳;.tƌ98y\{z=G~fp9M*K%ͪٶ'"ط<^H(e˜;wŧxdøe;Qe!JVG %Ʒ^VQeo-R9){pi[-Eg ϰ12Crz"қ l.0BQX`6jMd=L=ɀ6؜F|vxn""""}h5|q/ yh6`?-˿TRv`<&SI|9 ,"P&OrA[_R#ًyY0nʡ4L;0oj d DǤ1V$`ģ '>O?RosNO^qqm W2Q>KS l "̞/LC^`匙_A ¾‚?<6F3iT}TɁ5-+Po%oAD<||I^]ǣxrBR[x-֓򁁕mJzv;޶0LO> ]&=!i1y.!8 Z8Ȑ42TLO&=tX*->ygeFW|dG].|JS^Tgu{!ٻyKSZ|BO=4KҤQ>wJܵ'ǻybЇ9~Etzљ N5̿3Q则aOa:vˋ@6p_\ _PS]4r%1múTr^J Sڜ@Ɂ?P6[k80&^@r C83Z2FD1{ ?3 !  *o!ѫ gSw"[P1)kRrW&2H0b#9`8Wc\6nVa FD|]w /.!A]/$bM޳w]FO@;Z?"r 7AHR&9J|x  #I9O5+a-4ZK#4N6dQ{4w<'\,C1,|;v/G1cQ3xL {6gJ)i`v(! .v1ۮW\͔@DF|vtoyE%tmAZ^\%%<>|""7TLns23l_V֏h5S=Ӆ%氹iU"""2p ڒ̗?527>>.[2385kvqCyA~Vsjș&xO-SCu(F]pK`l*)P/u57Wfi:z0$>NI+)&c:gJNQ*)(G@(26]@7}2 9ei.=6"uWDDDn\ q]PĥT@۰Ro{x1/>F+1m1[3!F N48e+>7^ʧ t^S+g2wr*J¾N鷰KBÀa>I2rB(abPV9Go~*v* Δ"1)XWkpw@x¯|Ɋ ƶ޽PDG0M{^(3r@X޶_ ZNV|odAӞf5Smպbf>qYmZ6Z%""" khu3CsR6Gb ^$7eFSs9˓C)Wn`cCς;>mO`0`twT0DmM$"q=bxh cFcw$}ʗsX..?ėo>Wd%sod2c!/.{yF6p!''\h e"G2 ," [>i_ ^樑5"""ې.^il:hllz0?+o[C%0xK l_ykHiLg7)ԧVֈbeb[[[uBxPmN#& u}l$O=N$VѴw6"wDDDfЇ5"-VGm۷aMEv*[mN#vjv\nvқm1htDDD%"ǣ.z9G^cBx5Xr/Zٖ X kDd1C"""""2xur kDDDDDDDDFdpvv""qvv6n޸"""rCNn>55섿Mq=7䳋Wihh:<&Ͼv4 JDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDtϔQpCsvvχ&yp{noCq kD0+""Ϳv{("""7NӠDCHOs=FDDDDDDDā(q kDDDDDDDDjKl9KHwNhwԿ(Nj VR{;~C8QK;Mrn6"x.*{wENc#466hFi]iEa`S_6`čcO3_ aZʬaӮ=,9ֿlKm\+(%%H&iշxHʎc@9We!JVG+jj$/#I1#a?`+f^DNf>5"""̀ kΐznliw;e' ěsYwv:'쇭F 0;~LٔuXl5|N|^%<Eq_y9:!ۖDo 8g/+0b[vgG|zԴU^IcBP\:Kcm-daݑ5Ykr|2 ,1#)6}*^dㅈH@aM%_ C#bRi_ opFț0S O<&#oMy?6? &e79wa=۝V'ؼm͝5r4G}ak8O[e-W/Zԛ̌#}A%Ln p'ᩓ߽|C #diE24maO?Jz 7^wg$BFQeP5:{Va ԟj6X+Ņ//%},Yo ,oBWbP;VT0#f¬Dƾxoy%KVMEĦuN$~W>Wbw*EYE ɼhX2;ŝ=ql+Kp,_62~`oȭQ<7f/e v1θ)§A52cҘg f+F 4MDDd08aMy}wz2*khĐ2ZWt.[RÄ]& }G*e)8 -36{/;;Imusm1$]ycHn.N0N9^O[l6b椓'?xbfc|%ef)ζ9w u$}8ǒqʁ,.mOpB 1FVꎽt=FI u];',z;2/&dz*62f=] HL(,N x5v7YNvip2 Kfް:=Y~Iu"ތcV011;i<8CUϞH6?[wz+Y "'/sLtfӠy(rL밧 [(LlEDDd 8a/_/)RӒՌ6a (98 hlאWk}AKm,XhiQnq 氟F/B|iX㤮5si<1K)!;ùNKZXlީ]TtY]y`.˜|eS,32|8;1i|Y1v-|5E1pf4|0~^WTׇ5X91JdetxeD|םyy5X{~pyA|>ȊpaTRNY5|w}mdsEq(gsFD|]0vR.;7e1yA+' e &M9n)j%>mXO ]DDD k |g=|;hCdӉ1AZu@r84΂[4(ێS[T{i_ ^5""" kzS-cXF7ztÀm-:BXrRw@ о3TӵlLf3c(Ό 0z9 _]C'0'rL-e߰iq.x6z&#{x%ql2 4:93əښ֠zJoiMaqGet/ŋ-_ ͶmS hܔ剩ӻySwRSx-;=q򏏑w4e_½A<Ľ%3 韻_1/O#ھxժ9UO,)vz=úϝ̸{̝nxL.ecbZI{A2ݰ􁞼gE@@DD#kazK<>fAe}gfx!&;*<9H׻4͏|7W1tPak㒫՟?}! h`1G]wĒHwBDDDD(Ĝ55섿Mq=7䳋Wihh:<&Ͼv4 JDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDFDDDDDDDā(q kDDDDDDDDtϔQpCsvvχ&yp{noCq kD0+""Ϳv{("""7NӠDCHOs=FDDDDDDDā(q kDDDDDDDDjKl9KHwNhwԿ# 2{""""""s ksk};NR1d aakhh5^{x\SP֋f/e 0F1`i-e߰i_6FHmmm;aQ1|REDDDDvkC+9Jٿ2-]MnbW /eOczq؃ڊ"'vNxbdr\+;:KgV49Wr/Z:r;"1m/':Y0|-3V{8ÓojdNHk/w'0GrV?3m4 ǿꊈȀ5`.KrgwZVY*[BK^yJٿrOZ<5!M' 6Fe{ĂI:w@5^bA7Щ-~K^`dKf[t [jG̺hM)~8s.zL]]]wiPjhh'ߊe$Vنk"""2( ذf Vd Raɂɭ3fP#)u1[C3fg3 1Mn1k.? qKx|+&T'6p?~su:#CǷ-dY|'~L1 z˖}ye>-;eiS:H1[(?_eN^ccvYN.ˌs(Q+Z}*ПeVfG$TӳJJ];\J[8A PV ]LMq> ``sQ3mB `bAO%9@$4O%ËtCChT%_|/?CB?-|jⲓLlޝ0w<~>cǂǁѶǭh (cM@1>f=,\72UZ1Һym|Y˖#+&3cj (iۤ |'k2u/s829>&at푘X"C*+bg"4NGcg$mzZzA  g"udNeqhmMu;c™ccVP8Q{l0l 7c4Lded0ZYN<(R>jC{b1'$9!^?a%?w0~{1Ƽh9Hn Fe'>Z?'/0#=lىBle6ٹbƬ@#Kv\҈ƜpBBKV8+Dى a1D5d4ތ#p6~`޼OJ"f'ʬ~+:q!cNhsѵeneOt8>ImqM55ki*G.&2E9ecagUvmi{ &PŸ)BsQXcg :$G&"""f>O?RosNO^qqm W2Q>S;”wucR_l`t2M)3BP;-;bTB@)>dtMgu_oN#sl6b椓'?xbfc|%e` dQ\ n6n^mecɸ Kr8#QK+yu^O̞h#[ᤆ HpdR\qGY[d3;`Bv*g#2ˣr8yT6<r߰gQ8N,,Uyú8f-w_%։\z3Y"(NN? \N$YW1w@>{#n.V{l!ܕSd-N;ݟY&vc>LbSQ/NdWvnyGCcNvma̙4||/YK0^V|E3`&y?Ě2% 2/0y/Wl8*ތ0i}I?7?`0z~O/6>œ‹n |2 (V:{= (5[(O5AG?^}KX;]MЇx}m`M":=L[ibyڄ9%/bG/l?kĶ[^DDDָehL*8-Y͈iΧA0)}Aun̞/sx[%L4;&:5;ٔe~xybs3BzqRך94Z݊kJ+ ܫoJ:kዯ)΀33E$Q">5Ųo8U\&+SU@+l.D|םyy5dovSÊ{< SdED@%Qw٦lM60w\rV11iD` c'sS37r"iYkߟK0@PAU[^x[)>>|snVKGfQqx[FHF%ovƈ(fgW<$٣C4^n .C2p3>ŊvbM 50c0;ާxy$skؚ[Š8n..\Nx%7ø)M5erCćHdI*ȡ~^AoqEDD Zqʷq&çぎӇi*9R~fudecfuã,LF^cR eL'flӫw+E@_ȯ*[tAMYy%B/O\ztBz_^jNY.܂Iv^]]oQxws[ޓ b!Q9ө,~NV&nm5x- S:R#lѡf$@Rͯ+*a+n){,nSZk/<Q\L\N\%%<恤k|PY)^fȻ`|GO6'3eũlQ3#I.jQ0KQXbN26šjK2_}2wHl-̳xZ,M;>]_f[CZ( 0s.x+W+9$!\մuCnQť*TTTW xCs`s=mnIcNNgT\Y0ˡ &t]a ^u\nqO{p3F0K3fU5]5}ʑj+4od&{*M?;%M9DLd{Ylȹu[g-0 p*pZ#Fq]U͋>Gm Ye$//l99(;[i+PIYW;E_1׎{L7qgKV0RIfv5͸;r;w"kf2PB9j|)jP Yd+[_ 3/mA5Vq`gM5c/2ѝMP`0s;10߶n=k9t`f7o^,Օ/eXȜhvdEO Δ_}BYg?__Dʮf KP`MEŧm?#;95M{›TW1o 7K.w'<&Wq4ld#nufbhtv1 pJmnG ֑8ecl;-e=$Oa obGDiT{ly~J~pSk?ϋfd`$ӆAKU wUy{3r߯tn(9lur(?>/'Qj y+~KI Ԙ]ȏlu g;Vvݐ|6>ո@(B%cنNf++Y3`>a`OZQl~6y6짍s(Lo}&PzmFֈ nC>x頱멫cB_{dqkiP֬iX0+7`lk܄Qf?lkXe6錙xx6w/e̥n.wb >6:&jRwebV~)=%gn {i}Ʈ=2q;C aС89  iTax;ރn;B2yx5 G=/'HN#z굝:4oI&6öS[=am臘Zn%w7{:D-#iʾ{"x.*{!i@/1/O#zje{5\eС j W !CZ+CfV}v< |g#CUI?|ļ7$Nei<@}2?VmG k3AI ף_Ѱ1[r8/׬F5#0dziR2p'wСKG9mN|_ͅK_Xǿ?v?5y$G1!Q7m&.#PBո=ˮ'fP#"""2)ĜiZ!y/K/휟`ὁ; FDzMw EDDDDDDD75""""""""Da{- b.~-=5kD?rr󩩩ﮈ84gg'|n=7ܸ!]tHCC1aLMDDDDDDD䶣iP""""""""DaQX#"""""""@ֈ85""""{ui~0]+pkZRN4d$Rfat3+&Ql$côx֠EHd;g2'2. 59+C~5.TOF|=N=ufs8y# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?rL "ϵq4RDDd $rR8lK(di ]3]L>٨FDDdhRP#"r ])""2Cֈ5""""""""~DaQX#"""""""Gֈ5"7}Xt""""""r1L#`e=Ekn,em78ۢm<s6rY,[ʄ=fcȳwz&ww",{)mBp=o먈/v=EI׹0IšI%>3* Ô58f'ah_ѩ[ }*.ã䵴<fggWԵtxc)ĚL5-feM4m7N',@x'/pz5f*YNU֒kTDDDDdrfuX3r&Nڇ>f~a6 Y+޳Y qb E?\–V?eY9ɜIwFK5`91S]GS8p{~>;ZgR\M;jY?CW(ڛCB04M'J:Ow ݋WSLsa>B}F:nP!GV SH1 ևݘf:t3~3pŦSҽt1Tc IDAT"Ԉag&ˀ RVڜ&֦}@V&Xڴ wd7ys~@ h (\aNB3y3i5\ PCj.![(:8b5fheCVp5q,?)dwr|XFn6옂g9|rl?K!Re"Nv;ıŗ4-yl~Yl9OPWGz5gHx"˱)Gt ol!(:BA1%`n$8F6/hsv`. ytr\Ӽ79ȩ&BS9?CD-ĖPsSj$Dn! "vS[.#rv$i:Lhp?\L}[ ] tu仾FaYp.7d%"F2~PNS oauwNzlG FہcU cZ>:qa[u'6t .:ylg;!2/)}; _2^DKlXObS7ؿ c.""""rfVb!4=NV YZvYy̳rܗj .eV:9kMpچNZy&>ib=gXr [< .:' o|hH/DZp/,@Hj\S$Nǜ).q3rg1 9.%, e1Z#u9;4RC|]fDZȫɅx10=b q˜1=T_I dṗe esu1 wPTj04RQNz" 1`vg1w~XBSs&b_~IDu}vb,2ߞ22 ɴ]!GX&Rߠhy| KkvuxyH_%*q[/K0eFy_盵<#DDDDDn[&9g'mZB+y.>8ŗ2/׋@5ǵFcĽVma˲ҲMdY06XGn,%E `㨓1F&|c3f I_@{b6DEΓuhi ˠP/vL'J0c_G0d 9 rj20#Oo&`,Ĕ횅}?ikqĬf{&!f5毮!8y4p628/Cé;P=:`bdX蛯=9v7NM>Ï`GOIlLl (8ԟb}#_;H1o 7kvGkU!{3;zh' l!FXMY1zd% Ĕ%z֣y$: ! 'M̟I>wt@4D|]ߠƈ@-*^|9XWM8,5^55A{<"kX|7˚ZhLX9'&S~:!a3ނ DDDDDYִ4<;Y {XgeFdrx~)%`&l#-^@X#nLU2w{!wNIJsh3Aq,OjYN)ܥIuM C&:o6dP課S{&q$ ز M\gn9I2s͋:0$5\VLMcMxb8R0#"j}tम! f5 t;餽%C\y܄c&b)3NEjv$N.,91?oc _ .-4EsqlY^]FFDDDDd6HOEMbmZyNײ,֦BJ'b Ota haGc4©訥]Y!w?`LC;{Ȥg2+)I%aKLiqc>} JRq–)5'Y&!̓,Ɵg2_XkT44|~k+HcP琝ϫ!bTŽ}hob9S;`3E:}oYc `[)lwYd?dᘟeFAL)Vr9`61¸2H6"0.eP˗!b$2n '=[yv`¾<{%jc[o.!`.|-po.|׽K1dzZR!e G)?!@Ǥcrz2珟}yիWʕ+=ߠo>ÚQyϻ7 v:?,*hb2<BųƿN~8ߘ{̸Ac{h< ޾|ԥU.QEr2ZN\I;FYcj,Ya1M*sx#{6mmEdzrt˅5"2=f/)!cwYgº|/(SX#2fgˠDDnT"2DDd.~9wt"""$m0,"7D;L"""Sp'˗t"""9~σ~^J__W\!ЙMDDDDDDD䖣eP""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""GDdb~ML""$rR8l}ꟹc EDDdֈ$N6&>.5ӥ}N񚂆[3zLA$z{t~iu3bn1"""2u kDDDDDDDD?FDDDDDDDď(# kDDDDDDDD?rLpc9y-q-'`&+%LњEy+)Zh(emi}`6[%KC0o, "PALK$I'o5HI+d'@ϙ5MdU`.Hnd:Ͷr2?gp*'Ca0-m pVDL\Sck*F1ZFi5r}>u8WZ糳xv/ %մk=~59$CtrtG9;нx5%{4 -tߗov o{y$leq`=4S%]BDm1[Ρ*Pl'AIض N͋Z]! yo: .iޛNmTE`vrzd ZbKC9MR\yق)5mS dTyw9;\4&C4k?.&>WݭUd_Eks]PѰ,8T ILZDYL#?(70&=^ϱR1-}8v簭zzm:x6}ї\ȿ//B\H'v)fy_{V1'3ͧe`ӲC؞8L9޳ *'(9yfȌRX3f~89a\ulOhH:ՙ;"^kj \a)exa 6;`YyaqTfxql%*2MnK,e'44_a8[i%9C9z4'R֠F^M.7F6Nj=q\p &5T_I dṗe esu1 wPTj04RQNz" }8~G7;  7595H&NSDSv4cy*ǒ7=FgXZc:xyq :_C۱c;l]F:ذ'c[zDe~6{u ܡ(|g=c(7j|,gΚu5&pz?aḂx}/-G̉5(z;1VuVm89ά)kn7g,,d"Z:1c v }܌tLC *r "3HY 0TvtޯaHW3#!CR3 $/=1`h"SIBC\4Qg4wCTd# %@@9@8-kf8KY(3uD&I]P< ퟞ'_YtόϗS| O <3X:9c ֪Br4gvNٖCF]ҵc7J0b)#vSeKh%D,a ''xW6`x>}7u`>ZIj0VLMcMxb8R0#"j}tम! f5 t;餽%C\y܄c&b)3NEjv$N91oc _ .O-4EsqlY^]FFwPi6dA/-F Wg93r\>t76N֦e =:]&s1fN$̾+O\T` P넿1cx*; 1ő`~{wFvR&!~2y As];jN}H@G- ˰0I|^9x3ڝSON#4Dѷe~6IF:9ZNMN?ec ba!=؎u=LFGs ;,z82;Czb|LwӼ7}''ϲp"(#929-6P?3s/M);fLY[wbd9Y8V9hca멻0[&~E6eydMkF @L- |Έ5L$`3E:}oYc `[)lwYd?dᘟeFAL)Vr9`61¸2H6"0.eP˗!b$؎v=LhΠElݾ~Gp].Pͪ#kI.H6(/9!zs&IKMZf?/_g3PuPOݷ2˽h&s:ipZfֈns\z>\BݓLKRs6rm (^4&c1^xl޵ ٬ܗjhuiܪl^&[yWeOvdby(^GE䡛KIv+z'-f |fވ}qlJy}9iT*ϭ[FL̬kDf\Nr2EM, 9~SBSuKYY3z x}A2 kn6eP"PuplGtnE6U+o6;̛kȱ)دFDDDDnM kDDįv3Ǝ;"""""%"""""""Gֈ5""""""""~DaM" N.~L!".~9w^SgC:m0,r\zg#RDDZ@D._zM}ꟹc EDDd/=z*}}}\rCg6[A5""""""""~DakIDATQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~DaQX#"""""""Gֈ5""""""""~gX3g`5JkDDDDDDDDfĈfΜ9(vcYFDDDDDDDd޳f Ns>^u= p4vs j҈Pwz6)fֈH#š9s?0{MֈPdaRIENDB`pengcheng.mind-mindformer/docs/feature_cards/000077500000000000000000000000001476650141400216515ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/Chat_Web.md000066400000000000000000000074161476650141400236570ustar00rootroot00000000000000# Chat Web ## 基本介绍 Chat Web提供了一个网页界面，让用户可以通过类似线上聊天的方式使用MindFormers大语言模型（LLM）推理能力。 ## [Chat Web支持模型](../model_support_list.md#chat-web支持列表) ## 使用方式 ### 单卡启动 示例命令如下，将会启动一个网页app，使用glm_6b模型进行回答生成。 ```bash python run_chat_web.py --device_target Ascend \ --device_id 0 \ --model glm_6b \ --tokenizer glm_6b \ --checkpoint_path /path/to/glm_6b.ckpt \ --seq_length 512 \ --use_past False \ --host 0.0.0.0 \ --port 7860 ``` 参数定义如下： * **device_target** (str) - 表示要运行的目标设备，取值为`Ascend/CPU`。如果所安装的MindSpore版本不支持`Ascend`，将运行在CPU上。默认值：`Ascend`。 * **device_id** (int) - 表示要运行的卡，取值为`[0,8)`。该值只在`device_target==Ascend`时生效。默认值：`0`。 * **model** (str) - 表示用来生成回答的模型的名称，取值见上节。 * **tokenizer** (str) - 表示用来编解码输入输出的分词器的名称，取值见上节。 * **checkpoint_path** (str) - 表示加载的模型权重文件路径，此参数缺省时将会下载模型默认的权重文件，默认值：`None`。 * **seq_length** (int) - 表示模型中的序列长度。默认值：`512`。 * **use_past** (bool) - 表示是否使用增量推理。默认值：`False`。 * **host** (str) - 表示web服务运行的host ip。设置为`0.0.0.0`可远程访问。默认值：`127.0.0.1`。 * **port** (int) - 表示web服务运行的端口。此参数缺省时端口将会设置为`7860`，如果该端口被占用将会向后顺延，直至找到可用端口。默认值：`None`。 ### 多卡启动 暂不支持 ### 网页使用 当`--port`参数缺省时网页app将会默认运行在`7860`端口，如果默认端口被占用将会向后顺延，直至找到可用端口。 通过浏览器访问Chat Web网页地址 > 如果服务启动在本地，即配置了`--host`为`127.0.0.1`，则访问`http://127.0.0.1:7860`或`http://localhost:7860` > > 如果服务启动在远程，即配置了`--host`为`0.0.0.0`，假设服务器IP地址为`12.23.34.45`，则访问`http://12.23.34.45:7860` 在输入框中输入文字，点击***提交***按钮，稍等片刻后屏幕上将会显示LLM的回答。点击***清除***按钮可以清空聊天记录。 #### 配置项 聊天界面右侧提供了若干配置项，可以在点击提交按钮前自行进行配置，每次输入将会实时生效。目前提供的配置项如下： * **sampling** (开关) - 打开表示使用采样；关闭则表示使用贪心解码。打开后可以调整下述**top k**和**top p**。 * **top k** (滑块) - 从前k个可能性最大的候选词中采样。取值范围：`[0,10]`。 * **top p** (滑块) - 从可能性加起来为p的候选词中采样。取值范围：`[0,1]`。 * **temperature** (输入框) - 用来调节候选词的可能性得分。取值范围：`(0,∞)`。 * **repetition penalty** (输入框) - 重复惩罚因子。`1.0`表示无惩罚。取值范围：`(0,∞)`。 * **max length** (输入框) - 输入与回答的最大长度，不能超过模型的`seq_length`（注意：多轮对话时，输入将包括前几轮对话）。取值范围：`(输入长度,seq_length)`。 * **prompt** (输入框) - 提示词模板，与输入拼接后传进模型。输入框下方提供了一些样例模板，用户也可以输入自定义的模板，需要包含占位符`{}`，代表替换输入的位置。 > 每次启动时，配置项将会设置为模型默认的状态。pengcheng.mind-mindformer/docs/feature_cards/Eval_While_Training.md000066400000000000000000000052651476650141400260550ustar00rootroot00000000000000### 边训练边评估 大模型的训练效果需要评测任务来作为衡量标准，而当前大模型的训练耗时长，等到训练整体结束后再进行评测任务的时间与算力成本过高 本功能提供了一套在训练过程中进行评估的流程方法，以动态观察模型的评估结果，具有以下特性： 1. 能够在训练过程中执行评估流程，并在日志中打印相关的评估结果信息； 2. 具有功能开关，根据开关状态决定是否启用边训练边评估功能； 3. 具备调用间隔控制项，根据该控制项决定调用边训练边评估功能的时间间隔； 4. 支持多种模型，只要模型能够调用Model.eval()完成评估，都可以复用该项功能；无法通过Model.eval()完成评估的模型则需要额外编码适配，暂不支持； #### [边训练边评估支持度表](../model_support_list.md#边训练边评估支持度表) > 说明：边训练边评估功能需模型已支持评估，并且该评估指标可以通过Model.eval()完成 #### 使用用例 - run_mindformer启用边训练边评估 描述：通过run_mindformers脚本参数，启用边训练边评估功能 测试方式：`--do_eval`开关启用边训练边评估功能，`--eval_dataset_dir`指定评估数据集 ```shell python run_mindformer.py \ --config configs/gpt2/run_gpt2.yaml \ --run_mode train \ --train_dataset_dir /your_path/wikitext-2-train-mindrecord \ --eval_dataset_dir /your_path/wikitext-2-eval-mindrecord \ --do_eval True ``` - trainer启用边训练边评估 描述：通过Trainer高阶接口入参，启用边训练边评估功能 测试方式：执行以下python脚本，其中数据集路径替换为实际路径 ```python def test_trainer_do_eval(): from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='text_generation', model='gpt2', train_dataset="/your_path/wikitext-2-train-mindrecord", eval_dataset="/your_path/wikitext-2-eval-mindrecord") # 开启训练，并打开do_eval开关 trainer.train(do_eval=True) if __name__ == "__main__": test_trainer_do_eval() ``` - 配置评估间隔时间 描述：更改评估间隔时间，以控制执行评估的频率 测试方式：更改配置项，将 `configs/gpt2/run_gpt2.yaml` 文件中的 `eval_epoch_interval` 项修改为其他数值 执行`run_mindformer.py`启用边训练边评估中的启动脚本 ```yaml do_eval: False eval_step_interval: -1 # num of step intervals between each eval, -1 means no step end eval. # 修改此项eval_epoch_interval数值： eval_epoch_interval: 50 # num of epoch intervals between each eval, 1 means eval on every epoch end. ``` pengcheng.mind-mindformer/docs/feature_cards/Inference.md000066400000000000000000000337231476650141400241010ustar00rootroot00000000000000# Inference ## 基本介绍 MindFormers定位打造训练->微调->部署的端到端大模型工具套件，为了更好性能地部署已经微调训练好的大模型；我们利用MindSpore打造的推理引擎[MindSpore_lite](https://www.mindspore.cn/lite)，为用户提供了开箱即用的推理部署方案，为用户提供端到端的大模型解决方案，帮助用户使能大模型业务。 使用MindSpore Lite + Ascend硬件进行推理时，一般都会基于预训练或者微调后的权重进行推理部署。当前使用MindSpore Lite进行推理部署时，主要三个步骤： 1）进行权重合并 2）进行权重转换 3）最后部署推理服务 ![推理流程](assets/inference/inference_process.jpg) ## 设计概述 Inference采用的后端是MindSpore Lite，为了兼容已有的MindSpore的前向预测的流水线接口，对外部开发者呈现统一的推理接口，保持调用推理的一致性。Inference模块按照任务类别设计推理流水线，支持任务类别的可扩展，针对同一类任务兼容不同模型进行推理，实现模型可扩展的设计。`注：当前Inference模块仅支持LLM模型的生成式任务。` 具体设计如下： ![inference_framework](assets/inference/inference_framework.JPG) ## 接口设计 ### pipeline pipeline是Inference模块对外提供的推理API入口 ```python pipeline( task: str = None, model: Optional[Union[str, BaseModel, Model, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, backend: str = "ms", **kwargs) 功能 获取推理流水线 参数 - task(str):任务类型 - model(Optional[Union[str, BaseModel, Model, Tuple[str, str]]]):模型参数,在backend="mslite"时，是启动inference的推理能力 - config(PetConfig):微调算法的配置,包含微调算法的超参或者需要实例化的预训练模型 - tokenizer(BaseTokenizer):文本输入序预处理器 - image_processor(BaseImageProcessor):图像预处理 - audio_processor(BaseAudioProcessor):音频预处理 - backend(str):推理后端,支持"ms"和"mslite"两个后端 返回值 返回推理流水线 ``` ### InferTask InferTask是Inference模块是支持开发者在Inference框架中拓展推理任务类型的工厂类，主要提供get_infer_task()、check_task_valid()和support_list()三个接口；支持用户查询已经支持的推理任务类型和获取对应的推理任务实例进行推理。 ```python InferTask.get_infer_task(cls, task_type: str, task_config: InferConfig, **kwargs) 功能 获取推理任务实例 参数 - task_type(str):任务类型 - task_config(InferConfig):推理任务配置参数 返回值 返回推理任务 ``` ```python InferTask.check_task_valid(cls, task_type: str) 功能 获取所有支持的推理任务列表 参数 - task_type(str):待校验的任务类型。 返回值 返回推理模块是否支持该任务类型(bool) ``` ``` python InferTask.support_list(cls) 功能 获取所有支持的推理任务列表 返回值 返回推理任务列表 ``` ### InferConfig InferConfig是用户调用MindSpore Lite进行推理时，需要配置的参数项来初始化推理任务，具体如下： ```python class InferConfig(BaseConfig): """ Inference config class. """ def __init__(self, prefill_model_path: str = "", increment_model_path: str = "", model_type: str = "mindir", model_name: str = "common", infer_seq_length: int = 1024, target: str = "Ascend", device_id: int = 0, rank_id: int = 0, ge_config_path: str = "", 参数 prefill_model_path:全量图路径 increment_model_path:增量图路径 model_type:推理模型类型 model_name:模型名称 infer_seq_length:推理序列长度 target:推理硬件类型 device_id:设备ID rank_id:设备逻辑Rank ID ge_config_path:Ascend图引擎配置文件路径 ``` ## 快速使用 ### 模型导出（增量推理为例） #### 方式1：针对Configs下已支持模型的导出方式（以bloom_560m为例） step1. 修改配置文件，在配置文件中新增infer配置项，在run_bloom_560m.yaml中添加如下配置 ```yaml infer: prefill_model_path: "/path/bloom/bloom_560m_prefill.mindir" increment_model_path: "/path/bloom/bloom_560m_inc.mindir" infer_seq_length: 512 model_type: mindir # 参数说明：参考InferConfig的说明 # 由于配置了prefill_model_path和increment_model_path两个路径，因此需要导出增量图，因此需要在模型配置中打开增量开关，如下所示 # 如果想用bloom的后处理加速，则需要配置is_sample_acceleration开关，如下所示 model： model_config： use_past: True is_sample_acceleration: True # 后处理加速开关，部分模型不支持，当前bloom和glm支持 ``` step2.执行export.py，完成模型转换 ```shell python mindformers/tools/export.py --config_path ../../configs/bloom/run_bloom_560m.yaml ``` #### 方式2：针对research或者用户自定义模型的导出方式 step1. 准备好模型相关的配置文件、权重文件放置在一个文件夹 ```bash test_model_dir ├── test.yaml # test模型的配置文件 └── test.ckpt # test模型的权重文件 ``` step2. 在test.yaml中添加infer相关的配置项，参考方式1中的step1 step3. 执行export.py，完成模型转换 ```shell python mindformers/tools/export.py --model_dir /path/test_model_dir ``` ### 推理体验（以bloom为例） 推理模块提供了一键启动脚本，帮助用户快速上手使用Inference模块进行模型推理。 step1. 利用执行`模型导出`章节，得到MindIR图，如果是全量模型，则会得到两个MindIR图（bloom_560m_prefill.mindir和bloom_560m_inc.mindir）。 step2. 执行run_infer_main.py脚本，调起推理： ```shell python run_infer_main.py --device_id 0 --model_name bloom --prefill_model_path /path/bloom/bloom_560m_prefill.mindir --increment_model_path /path/bloom/bloom_560m_inc.mindir --config_path /path/ge_config.ini --is_sample_acceleration True ``` ```text run_infer_main支持的参数详细说明如下： device_id: 设备物理ID model_name: 模型名称，当前支持bloom，glm，llama，glm2 seq_length: 推理序列长度 prefill_model_path: 全量图路径 increment_model_path: 增量图路径 config_path: GE配置文件路径 is_sample_acceleration: 后处理加速开关 add_special_tokens: 对输入token化时是否添加特殊字符 stream: 是否采用流式结果返回 ``` ### 推理APIs调用说明（增量推理为例） Inference模块是由pipeline() API来对外进行调用，目前支持四种方式进行调用： - `模型名称进行调用，（以bloom_560m为例）` step1. 参考模型导出章节完成MindIR模型的导出 step2. 在configs/bloom/run_bloom_560m.yaml中配置infer配置项 ```yaml infer: prefill_model_path: "/path/bloom/bloom_560m_prefill.mindir" increment_model_path: "/path/bloom/bloom_560m_inc.mindir" infer_seq_length: 512 model_type: mindir ge_config_path: "/path/ge_config.ini" # 参数说明：参考InferConfig的说明 ``` step3. 调用inference API完成模型推理调用 ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline # 定义bloom 560m模型的tokenizer model_name = "bloom_560m" tokenizer = BloomTokenizer.from_pretrained(model_name) ge_config_path = "/path/ge_config_path" # 定义推理流水线 lite_pipeline = pipeline( task="text_generation", model=model_name, tokenizer=tokenizer, backend="mslite", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `模型路径进行调用` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline prefill_model_path = "/path/prefill_model_path" increment_model_path = "/path/increment_model_path" ge_config_path = "/path/ge_config_path" # 定义tokenizer tokenizer = BloomTokenizer.from_pretrained(model_name) # 定义推理流水线 lite_pipeline = pipeline( task="text_generation", model=(prefill_model_path, increment_model_path), tokenizer=tokenizer, backend="mslite", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `模型文件夹进行调用` 针对于灵活切换模型的部署，我们支持从指定文件夹目录读取相关配置启动推理。首先需要按照指定格式准备模型文件夹，具体如下： ```bash test_model_dir ├── test.yaml # test模型的配置文件,只能存在一个yaml文件 ├── xxx_prefill_graph.mindir # test模型的全量图,此处需要以prefill_graph.mindir为后缀名 ├── xxx_inc_graph.mindir # test模型的增量图,此处需要以inc_graph.mindir为后缀名,当前只能存在不多于两个mindir文件 └── config.ini # ge的配置参数,文件名不可变 ``` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline test_model_dir = "/path/test_model_dir" ge_config_path = "/path/ge_config_path" # 定义tokenizer tokenizer = BloomTokenizer.from_pretrained("bloom_560m") lite_pipeline = pipeline( task="text_generation", model=test_model_dir, tokenizer=tokenizer, backend="mslite", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` - `自定义配置config的方式` ```python from mindformers.models import BloomTokenizer from mindformers.pipeline import pipeline from mindformers.infer.infer_config import InferConfig prefill_model_path = "/path/prefill_model_path" increment_model_path = "/path/increment_model_path" ge_config_path = "/path/ge_config_path" lite_config = InferConfig( prefill_model_path=prefill_model_path, increment_model_path=increment_model_path, model_type="mindir", model_name="bloom_560m", ge_config_path=ge_config_path, device_id=0, infer_seq_length=512, ) tokenizer = BloomTokenizer.from_pretrained("bloom_560m") lite_pipeline = InferTask.get_infer_task("text_generation", lite_config, tokenizer=tokenizer) # 推理 output = lite_pipeline("I love Beijing,because", is_sample_acceleration=True,add_special_tokens=False) print(output) ``` ### 如何配置GE图引擎配置 #### 针对910A，推荐的GE配置参数如下(config.ini)： ```ini [ascend_context] plugin_custom_ops=All provider=ge [ge_session_options] ge.externalWeight=1 ge.exec.atomicCleanPolicy=1 ge.event=notify ge.exec.staticMemoryPolicy=2 ge.exec.precision_mode=must_keep_origin_dtype ``` #### 针对910B，推荐的GE配置参数如下(config.ini)： ```ini [ascend_context] plugin_custom_ops=All provider=ge [ge_session_options] ge.exec.precision_mode=must_keep_origin_dtype [ge_graph_options] ge.exec.formatMode=1 ``` 针对于GE参数的详细说明可以参考[GE options参数说明](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/70RC1alpha002/infacldevg/graphdevg/atlasgeapi_07_0118.html) ## 自定义开发（text generation任务为例） ### 如何支持新增LLM模型的MINDIR导出 在export.py中，已经完成了增量推理模型的导出流程，由于不同的LLM模型可能存在不同的输入，因此，为了统一的导出流程，我们支持用户自定义输入接口，添加到已有的流程中去，从而完成对新模型的适配。 例如以bloom为例： step1. 在export.py中首先实现了`get_bloom_inc_model_input()`接口 step2. 在export.py的增量输入表`PREFILL_MODEL_INPUT_MAP`中，添加bloom的输入映射： ```python PREFILL_MODEL_INPUT_MAP = { "bloom": get_llm_common_prefill_model_input, } ``` 完成上述两个步骤后，则完成了bloom模型在export工具中的添加。 ### 如何支持新增LLM模型的推理 由于导出时，确定了LLM模型在MindIR图中的输入顺序以及shape信息，而由于模型差异，因此无法按照统一的输入处理，为实现不同模型的统一处理，则需要将模型输入接口类似export时一样添加到任务中。 例如以bloom为例： step1. 在`text_generator_infer.py`中，实现bloom模型的输入类定义，如下，我们定义一个通用输入处理类： ```python class CommonInputsOfInfer(BaseInputsOfInfer): """ common infer inputs of llm models. """ def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): if not is_first_iteration: inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) input_ids = np.array(inputs_tmp, dtype=np.int32) inputs = [input_ids, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs ``` step2. 在`text_generator_infer.py`的输入工厂类`InputOfInfer`中的`MAPPING`表中添加输入类映射关系： ```python class InputOfInfer: """ Input of llm model. """ MAPPING = { "bloom": CommonInputsOfInfer, } ```pengcheng.mind-mindformer/docs/feature_cards/Performance_Tuning.md000066400000000000000000000306661476650141400257730ustar00rootroot00000000000000# 性能调优 ## [Profiler](https://mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.Profiler.html?&highlight=profiler)数据采集 MindSpore中提供了profiler接口，可以对神经网络的性能进行采集。目前支持AICORE算子、AICPU算子、HostCPU算子、内存、设备通信、集群等数据的分析。 mindformers将此功能集成到了套件内部，用户仅需修改配置文件中相应部分即可开启采集。 在配置文件中添加Profile相关参数： ```yaml # profile profile: False profile_start_step: 1 profile_stop_step: 10 init_start_profile: True profile_communication: True profile_memory: True ``` - **profile_start_step** - 表示开始收集算子性能数据的step。默认值：1。 - **profile_stop_step** - 表示结束收集算子性能数据的step。默认值：10。 - **output_path** - 表示输出数据的路径。默认值："./output/profile/rank_{id}/profiler/"。 - **init_start_profile** - 该参数控制是否在Profiler初始化的时候开启数据采集。默认值：True。 - **profile_communication** - (仅限Ascend) 表示是否在多设备训练中收集通信性能数据。当值为True时，收集这些数据。在单台设备训练中，该参数的设置无效。默认值：False。 - **profile_memory** - (仅限Ascend) 表示是否收集Tensor内存数据。当值为True时，收集这些数据。默认值：False。 修改配置文件后，[启动训练](../start/Quick_Tour.md#方式一使用已有脚本启动)，在根目录下会生成相应的profiler文件夹存放各rank的profiler文件。 ```index ├─... ├─output │ ├─checkpoint │ ├─profiler │ │ ├─rank0 │ │ ├─... │ │ └─rank7 ...... ``` ## [Summary](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.SummaryCollector.html)数据采集 > ### 使用约束 > > 使用SummaryMonitor时，如果使用Trainer高阶接口启动任务，则**必须**将代码放置到 `if __name__ == “__main__”` 中运行。 在配置文件的callbacks中添加SummaryMonitor和相关参数： ```yaml callbacks: * type: SummaryMonitor summary_dir: None collect_freq: 10 collect_specified_data: None keep_default_action: True custom_lineage_data: None collect_tensor_freq: None max_file_size: None export_options: None ``` 参数 - **summary_dir** (str) - 收集的数据将存储到此目录。如果目录不存在，将自动创建。默认值："./output/summary/rank_{id}/"。 - **collect_freq** (int) - 设置数据收集的频率，频率应大于零，单位为 step 。如果设置了频率，将在(current steps % freq)=0时收集数据，并且将总是收集第一个step。需要注意的是，如果使用数据下沉模式，单位将变成 epoch 。不建议过于频繁地收集数据，因为这可能会影响性能。默认值：10。 - **collect_specified_data** (Union[None, dict]) - 对收集的数据进行自定义操作。您可以使用字典自定义需要收集的数据类型。例如，您可以设置{‘collect_metric’:False}不去收集metrics。支持控制的数据如下。默认值：None，收集所有数据。 - **collect_metric** (bool) - 表示是否收集训练metrics，目前只收集loss。把第一个输出视为loss，并且算出其平均数。默认值：True。 - **collect_graph** (bool) - 表示是否收集计算图。目前只收集训练计算图。默认值：True。 - **collect_train_lineage** (bool) - 表示是否收集训练阶段的lineage数据，该字段-将显示在MindInsight的 [lineage页面](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/lineage_and_scalars_comparison.html) 上。默认值：True。 - **collect_eval_lineage** (bool) - 表示是否收集评估阶段的lineage数据，该字段将显示在MindInsight的lineage页面上。默认值：True。 - **collect_input_data** (bool) - 表示是否为每次训练收集数据集。目前仅支持图像数据。如果数据集中有多列数据，则第一列应为图像数据。默认值：True。 - **collect_dataset_graph** (bool) - 表示是否收集训练阶段的数据集图。默认值：True。 - **histogram_regular** (Union[str, None]) - 收集参数分布页面的权重和偏置，并在MindInsight中展示。此字段允许正则表达式控制要收集的参数。不建议一次收集太多参数，因为这会影响性能。注：如果收集的参数太多并且内存不足，训练将会失败。默认值：None，表示只收集网络的前五个超参。 - **collect_landscape** (Union[dict, None]) - 表示是否收集创建loss地形图所需要的参数。如果设置None，则不收集任何参数。默认收集所有参数并且将会保存在 {summary_dir}/ckpt_dir/train_metadata.json 文件中。 - **landscape_size** (int) - 指定生成loss地形图的图像分辨率。例如：如果设置为128，则loss地形图的分辨率是128*128。注意：计算loss地形图的时间随着分辨率的增大而增加。默认值：40。可选值：3-256。 - **unit** (str) - 指定训练过程中保存checkpoint时，下方参数 intervals 以何种形式收集模型权重。例如：将 intervals 设置为[[1, 2, 3, 4]]，如果 unit 设置为step，则收集模型权重的频率单位为step，将保存1-4个step的模型权重，而 unit 设置为epoch，则将保存1-4个epoch的模型权重。默认值：step。可选值：epoch/step。 - **create_landscape** (dict) - 选择创建哪种类型的loss地形图，分为训练过程loss地形图（train）和训练结果loss地形图（result）。默认值：{“train”: True, “result”: True}。可选值：True/False。 - **num_samples** (int) - 创建loss地形图所使用的数据集的大小。例如：在图像数据集中，您可以设置 num_samples 是128，这意味着将有128张图片被用来创建loss地形图。注意：num_samples 越大，计算loss地形图时间越长。默认值：128。 - **intervals** (List[List[int]]) - 指定loss地形图的区间。例如：如果用户想要创建两张训练过程的loss地形图，分别为1-5epoch和6-10epoch，则用户可以设置[[1, 2, 3, 4, 5], [6, 7, 8, 9, 10]]。注意：每个区间至少包含3个epoch。 - **keep_default_action** (bool) - 此字段影响 collect_specified_data 字段的收集行为。True：表示设置指定数据后，其他数据按默认设置收集。False：表示设置指定数据后，只收集指定数据，不收集其他数据。默认值：True。 - **custom_lineage_data** (Union[dict, None]) - 允许您自定义数据并将数据显示在MindInsight的lineage页面上。在自定义数据中，key支持str类型，value支持str、int和float类型。默认值：None，表示不存在自定义数据。 - **collect_tensor_freq** (Optional[int]) - 语义与 collect_freq 的相同，但仅控制TensorSummary。由于TensorSummary数据太大，无法与其他summary数据进行比较，因此此参数用于降低收集量。默认情况下，收集TensorSummary数据的最大step数量为20，但不会超过收集其他summary数据的step数量。例如，给定 collect_freq=10 ，当总step数量为600时，TensorSummary将收集20个step，而收集其他summary数据时会收集61个step。但当总step数量为20时，TensorSummary和其他summary将收集3个step。另外请注意，在并行模式下，会平均分配总的step数量，这会影响TensorSummary收集的step的数量。默认值：None，表示要遵循上述规则。 - **max_file_size** (Optional[int]) - 可写入磁盘的每个文件的最大大小（以字节为单位）。例如，如果不大于4GB，则设置 max_file_size=4*1024**3 。默认值：None，表示无限制。 - **export_options** (Union[None, dict]) - 表示对导出的数据执行自定义操作。注：导出的文件的大小不受max_file_size 的限制。您可以使用字典自定义导出的数据。例如，您可以设置{‘tensor_format’:’npy’}将tensor导出为 npy 文件。支持控制的数据如下所示。默认值：None，表示不导出数据。 - **tensor_format** (Union[str, None]) - 自定义导出的tensor的格式。支持[“npy”, None]。默认值：None，表示不导出tensor。 - **npy** - 将tensor导出为NPY文件。 训练完成后，summary日志文件将会输出至指定目录，目录结构如下： ```tree └─summary └─rank_0 events.out.events.summary.1596869898.hostname_MS events.out.events.summary.1596869898.hostname_lineage ``` ## [MindInsight](https://mindspore.cn/mindinsight/docs/zh-CN/r2.0/index.html)可视化调优 通过MindInsight可以完成训练可视、性能调优、精度调优等任务。 训练可视功能主要包括训练看板、模型溯源、数据溯源等功能，训练看板中又包括标量、参数分布图、计算图、数据图、数据抽样、张量等子功能。 训练完成数据收集后，启动MindInsight，即可可视化收集到的数据。启动MindInsight时， 需要通过 `--summary-base-dir` 参数指定summary日志文件目录。MindInsight启动的默认端口为`8080`，当默认端口被占用时，可以通过`--port`参数指定端口。 通过命令行启动mindinsight **可视化训练结果** ```shell # 开启 summary-base-dir指定到mindformers生成的output/summary mindinsight start --port 8000 --summary-base-dir xx/mindformers/output/summary ``` **可视化profiler性能数据** ```shell # 开启 summary-base-dir指定到mindformers生成的output/profiler mindinsight start --port 8000 --summary-base-dir xx/mindformers/output/profiler ``` 启动成功后，通过浏览器访问 `http://127.0.0.1:8080` 地址，即可查看可视化页面。 如果要远程访问物理机上启动的可视化页面时，需要修改MindInsight的Host地址。首先通过`pip show mindinsight`获取MindInsight安装位置，打开安装目录下的`./conf/constants.py` ，修改其中的`Host`字段为`0.0.0.0`。 **关闭mindinsight** ```shell #关闭 mindinsight stop --port 8000 ``` ## 保存IR图(中间编译图) 中间编译图又称为中间表示（IR），是程序编译过程中介于源语言和目标语言之间的程序表示，以方便编译器进行程序分析和优化，因此IR的设计需要考虑从源语言到目标语言的转换难度，同时考虑程序分析和优化的易用性和性能。 MindIR是一种基于图表示的函数式IR，其最核心的目的是服务于自动微分变换。自动微分采用的是基于函数式编程框架的变换方法，因此IR采用了接近于ANF函数式的语义。此外，借鉴Sea of Nodes和Thorin的优秀设计，采用了一种基于显性依赖图的表示方式。关于ANF-IR的具体介绍，可以参考[MindSpore IR文法定义](https://www.mindspore.cn/docs/zh-CN/r2.0/design/mindir.html#文法定义)。 MindSpore提供了保存中间编译图的功能，在使用MindFormers编写模型时，在图模式`set_context(mode=GRAPH_MODE)`下，可以通过设置MindSpore上下文来打开存图功能。 在MindSpore 1.10版本中，支持使用`set_context(save_graphs=True)`来打开存图功能。运行时会输出一些图编译过程中生成的一些中间文件。 在MindSpore 2.0版本中，若配置中设置了`set_context(save_graphs=1)`，运行时会输出一些图编译过程中生成的一些中间文件。当需要分析更多后端流程相关的ir文件时，可以设置`set_context(save_graphs=True)` 或`set_context(save_graphs=2)`。当需要更多进阶的信息比如可视化计算图，或者更多详细前端ir图时，可以设置`set_context(save_graphs=3)`。 可以通过`set_context(save_graphs_path='{path}')`来指定图保存路径。 当前主要有三种格式的IR文件： - ir后缀结尾的IR文件：一种比较直观易懂的以文本格式描述模型结构的文件，可以直接用文本编辑软件查看。 - 通过设置环境变量`export MS_DEV_SAVE_GRAPTHS_SORT_MODE=1`可以生成的ir后缀结尾的IR文件：格式跟默认的ir文件基本相同，但是生成的异序ir文件的图的生成与打印顺序和默认ir文件不同。 - dot后缀结尾的IR文件：描述了不同节点间的拓扑关系，可以用[graphviz](http://graphviz.org/)将此文件作为输入生成图片，方便用户直观地查看模型结构。对于算子比较多的模型，推荐使用可视化组件[MindInsight](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/dashboard.html#计算图可视化)对计算图进行可视化。 pengcheng.mind-mindformer/docs/feature_cards/Pet_Tuners.md000066400000000000000000000052121476650141400242630ustar00rootroot00000000000000# 低参微调 MindPet（Pet：Parameter-Efficient Tuning）是属于Mindspore领域的微调算法套件。随着计算算力不断增加，大模型无限的潜力也被挖掘出来。但随之在应用和训练上带来了巨大的花销，导致商业落地困难。因此，出现一种新的参数高效（parameter-efficient）算法，与标准的全参数微调相比，这些算法仅需要微调小部分参数，可以大大降低计算和存储成本，同时可媲美全参微调的性能。 ## [微调支持列表](../model_support_list.md#微调支持列表) ## 使用示例 1. 修改任务模型，以GPT2为例，主要有以下步骤： - 继承GPT2LMHeadModel - 定义替换lora算法的替换规则`self.pet.pet_config.reg_rules = r'.*dense*|.*linear*|.*mapping*|.*projection*'` - 调用lora适配器接口修改预训练模型 - 导入预训练权重以及冻结预训练模型权重 - 为了使用MindFormer的训练流程，将GPT2WithLora微调模型注册到MindFormer中 具体代码如下： ```python @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): def __init__(self, config: GPT2Config = None, pet = None, **kwargs): super().__init__(config) # get Pet tuning model. self.pet = pet self.pet.pet_config.reg_rules = r'.*dense*|.*linear*|.*mapping*|.*projection*' self.backbone = LoraAdapter.get_pet_model(self.backbone, self.pet.pet_config) self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) ``` 2. 修改训练任务参数，使用MindFormer的trainer进行模型训练，主要修改模型配置yaml，以GPT2为例： ```yaml model: model_config: type: GPT2Config ... arch: # type: GPT2LMHeadModel # 替换为适配微调算法的模型 type: GPT2WithLora pet: pet_type: lora pet_config: # configurition of lora in_channels: 768 out_channels: 768 lora_rank: 8 lora_alpha: 16 lora_dropout: 0.1 ``` 只需要将arch参数由原来的`class GPT2LMHeadModel`替换为现在适配微调算法的`class GPT2WithLora`，并按照lora算法进行微调参数配置。 3. 使用MindFormer的Trainer进行模型训练： ```python from mindformers.trainer.trainer import Trainer gpt2_trainer = Trainer( task='text_generation', model='gpt2', pet_method='lora', train_dataset="/data/wikitext-2/train", ) gpt2_trainer.finetune() ``` 至此，完成了一个微调算法适配过程，最后执行上述步骤3中的代码即可拉起微调算法的训练流程。 pengcheng.mind-mindformer/docs/feature_cards/Precision_Optimization.md000066400000000000000000000124521476650141400267000ustar00rootroot00000000000000# 精度调优 ## [Dump使用方法](https://mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/dump.html) 异步/同步 dump的操作步骤基本一样，区别在于json文件不同。 **异步dump(训练结束后dump数据)** 1. 创建json配置文件 JSON文件的名称和位置可以自定义设置。 ```json { "common_dump_settings": { "dump_mode": 0, "path": "/absolute_path", "net_name": "ResNet50", "iteration": "0|5-8|100-120", "saved_data": "tensor", "input_output": 0, "kernels": ["Default/Conv-op12"], "support_device": [0,1,2,3,4,5,6,7], "op_debug_mode": 0, "file_format": "npy" } } ``` - `dump_mode`：设置成0，表示Dump出该网络中的所有算子数据；设置成1，表示Dump`"kernels"`里面指定的算子数据或算子类型数据；设置成2，表示Dump脚本中通过`set_dump`指定的算子数据，`set_dump`的使用详见[mindspore.set_dump](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_dump.html) 。开启溢出检测时，此字段的设置失效，Dump只会保存溢出节点的数据。 - `path`：Dump保存数据的绝对路径。 - `net_name`：自定义的网络名称，例如：”ResNet50”。 - `iteration`：指定需要Dump的迭代。类型为str，用“|”分离要保存的不同区间的step的数据。如”0|5-8|100-120”表示Dump第1个，第6个到第9个， 第101个到第121个step的数据。指定“all”，表示Dump所有迭代的数据。PyNative模式开启溢出检测时，必须设置为”all”。 - `saved_data`: 指定Dump的数据。类型为str，取值成”tensor”，表示Dump出完整张量数据；取值成”statistic”，表示只Dump张量的统计信息；取值”full”代表两种都要。异步Dump统计信息只有在`file_format`设置为`npy`时可以成功，若在`file_format`设置为`bin`时选”statistic”或”full”便会错误退出。默认取值为”tensor”。 - `input_output`：设置成0，表示Dump出算子的输入和算子的输出；设置成1，表示Dump出算子的输入；设置成2，表示Dump出算子的输出。 - `kernels`：该项可以配置两种格式： 1. 算子的名称列表。开启IR保存开关`set_context(save_graphs=2)`并执行用例，从生成的IR文件`trace_code_graph_{graph_id}`中获取算子名称。详细说明可以参照教程：[如何保存IR](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/mindir.html#如何保存ir)。 需要注意的是，是否设置`set_context(save_graphs=2)`可能会导致同一个算子的id不同，所以在Dump指定算子时要在获取算子名称之后保持这一项设置不变。或者也可以在Dump保存的`ms_output_trace_code_graph_{graph_id}.ir`文件中获取算子名称，参考同步Dump数据对象目录。 2. 还可以指定算子类型。当字符串中不带算子scope信息和算子id信息时，后台则认为其为算子类型，例如：”conv”。算子类型的匹配规则为：当发现算子名中包含算子类型字符串时，则认为匹配成功（不区分大小写），例如：”conv” 可以匹配算子 “Conv2D-op1234”、”Conv3D-op1221”。 - `support_device`：支持的设备，默认设置成0到7即可；在分布式训练场景下，需要dump个别设备上的数据，可以只在`support_device`中指定需要Dump的设备Id。 - `op_debug_mode`：该属性用于算子溢出调试，设置成0，表示不开启溢出；设置成1，表示开启AiCore溢出检测；设置成2，表示开启Atomic溢出检测；设置成3，表示开启全部溢出检测功能。在Dump数据的时候请设置成0，若设置成其他值，则只会Dump溢出算子的数据。 - `file_format`: dump数据的文件类型，只支持`npy`和`bin`两种取值。设置成`npy`，则dump出的算子张量数据将为host侧格式的npy文件；设置成`bin`，则dump出的数据将为device侧格式的protobuf文件，需要借助转换工具进行处理，详细步骤请参考异步Dump数据分析样例。默认取值为`bin`。 2. 设置Dump环境变量 ```shell export MINDSPORE_DUMP_CONFIG=${Absolute path of data_dump.json} ``` 如果Dump配置文件没有设置`path`字段或者设置为空字符串，还需要配置环境变量`MS_DIAGNOSTIC_DATA_PATH`。 ```shell export MS_DIAGNOSTIC_DATA_PATH=${yyy} ``` 则“$MS_DIAGNOSTIC_DATA_PATH/debug_dump”就会被当做`path`的值。若Dump配置文件中设置了`path`字段，则仍以该字段的实际取值为准。 - 在网络脚本执行前，设置好环境变量；网络脚本执行过程中设置将会不生效。 - 在分布式场景下，Dump环境变量需要在调用`mindspore.communication.init`之前配置。 3. [启动网络训练脚本](../start/Quick_Tour.md#方式一使用已有脚本启动) 可以在训练脚本中设置`set_context(reserve_class_name_in_scope=False)`，避免Dump文件名称过长导致Dump数据文件生成失败。 4. 解析Dump数据文件 可以使用MindSpore Insight的离线调试器来分析。离线调试器的使用方法详见[使用离线调试器](https://www.mindspore.cn/mindinsight/docs/zh-CN/r2.0/debugger_offline.html) 。 **同步dump(训练的同时进行数据dump)** 参考[mindspore官网dump说明](https://mindspore.cn/tutorials/experts/zh-CN/r2.0/debug/dump.html) pengcheng.mind-mindformer/docs/feature_cards/Resume_Training.md000066400000000000000000000051201476650141400252640ustar00rootroot00000000000000# 断点续训 ## 介绍 在长时间的训练当中，如果遇到意外情况导致训练中断，可以使用断点续训的方式恢复之前的状态继续训练。 恢复训练时，从之前训练的checkpoint文件或文件夹中，加载网络权重，并恢复训练时的epoch、step、loss_scale_value等信息。 > 局限：由于数据集暂不支持跳过已训练的数据，当前仅支持epoch级的断点续训，即仅支持恢复到某个epoch开始的状态。 ## 使用 ### 脚本启动场景 在run_xxx.yaml中配置load_checkpoint为checkpoint文件或文件夹路径，并将resume_training改为True。 ```yaml load_checkpoint: checkpoint_file_or_dir_path resume_training: True ``` > 注意：如果load_checkpoint配置为文件路径，则认为是完整权重，如果配置为文件夹，则认为是分布式权重。下同。 ### Trainer高阶接口启动场景 - 方式1：在Trainer.train()或Trainer.finetune()中配置 在Trainer.train()或Trainer.finetune()中，配置train_checkpoint或finetune_checkpoint参数为checkpoint文件或文件夹路径，并将resume_training参数设置为True。 ```python from mindformers import Trainer cls_trainer = Trainer(task='image_classification', # 已支持的任务名 model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 cls_trainer.train(train_checkpoint="", resume_training=True) # 启动训练 cls_trainer.finetune(finetune_checkpoint="", resume_training=True) # 启动微调 ``` - 方式2：在TrainingArguments中配置 在TrainingArguments中配置resume_from_checkpoint为checkpoint文件或文件夹路径，并将resume_training参数设置为True。 ```python from mindformers import Trainer, TrainingArguments training_args = TrainingArguments(resume_from_checkpoint="", resume_training=True) cls_trainer = Trainer(task='image_classification', # 已支持的任务名 args=training_args, model='vit_base_p16', # 已支持的模型名 train_dataset="/data/imageNet-1k/train", # 传入标准的训练数据集路径，默认支持ImageNet数据集格式 eval_dataset="/data/imageNet-1k/val") # 传入标准的评估数据集路径，默认支持ImageNet数据集格式 cls_trainer.train() # 启动训练 cls_trainer.finetune() # 启动微调 ``` pengcheng.mind-mindformer/docs/feature_cards/Text_Generator.md000066400000000000000000000107351476650141400251330ustar00rootroot00000000000000# Text Generator Mindformers大模型套件提供了text generator方法，旨在让用户能够便捷地使用生成类语言模型进行文本生成任务，包括但不限于解答问题、填充不完整文本或翻译源语言到目标语言等。 当前该方法支持Minformers大模型套件中6个生成类语言模型 ## [Text Generator支持度表](../model_support_list.md#text-generator支持度表) ## 增量推理 Mindformers大模型套件的`text generator`方法支持增量推理逻辑，该逻辑旨在加快用户在调用`text generator`方法进行文本生成时的文本生成速度。 在此提供使用高阶接口进行各模型增量推理的**测试样例脚本**： ```python # mindspore设置图模式和环境 import mindspore; mindspore.set_context(mode=0, device_id=0) from mindformers import AutoConfig, AutoModel, AutoTokenizer # 按需设置模型类型名，高阶接口将根据类型名实例化相应模型 model_type = "glm_6b" # 按需设置测试的输入文本 input_text = "中国的首都是哪个城市？" # 获取模型默认配置项并按需修改 config = AutoConfig.from_pretrained(model_type) # use_past设置为True时为增量推理，反之为自回归推理 config.use_past = True # 修改batch_size和模型seq_length config.batch_size = 1; config.seq_length=512 # 根据配置项实例化模型 model = AutoModel.from_config(config) # 实例化tokenizer tokenizer = AutoTokenizer.from_pretrained(model_type) # 对输入进行tokenizer编码 input_ids = tokenizer(input_text)["input_ids"] # 调用model.generate接口执行增量推理 output = model.generate(input_ids, max_length=128, do_sample=False) # 解码并打印输出 print(tokenizer.decode(output)) ``` > 注： > > 1. 首次调用generate时需要进行mindspore图编译，耗时较长；在统计在线推理的文本生成速度时，可以多次重复调用并排除首次调用的执行时间 > 2. 使用增量推理(use_past=True)时的生成速度预期快于自回归推理(use_past=False) ## Batch推理 `text generator`方法也支持同时对多个输入样本进行batch推理；在单batch推理算力不足的情况下，多batch推理能够提升推理时的吞吐率 以下给出测试batch推理能力的**标准测试脚本**，仅上述增量推理测试脚本仅有少数区别 ```python import mindspore;mindspore.set_context(mode=0, device_id=0) from mindformers import AutoConfig, AutoModel, AutoTokenizer model_type = "glm_6b" # 多batch输入文本 input_text = [ "中国的首都是哪个城市？", "你好", "请介绍一下华为", "I love Beijing, because" ] # 是否使用增量推理 use_past = True # 预设模型seq_length seq_len = 512 config = AutoConfig.from_pretrained(model_type) # 将batch size修改为输入的样本数 config.batch_size = len(input_text) config.use_past = use_past config.seq_length = seq_len model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained(model_type) # 对输入batch进行pad input_ids = tokenizer(input_text, max_length=config.seq_length, padding="max_length")["input_ids"] output = model.generate(input_ids, max_length=128, do_sample=False) print(tokenizer.decode(output)) ``` > 注： > batch推理的推理吞吐率提升表现与设备计算负荷相关；在seq_len较短并开启增量推理的情况下，计算负荷较小，使用batch推理通常会获得较好的提升 ## 流式推理 Mindformers大模型套件提供Streamer类，旨在用户在调用text generator方法进行文本生成时能够实时看到生成的每一个词，而不必等待所有结果均生成结束。 实例化streamer并向text generator方法传入该实例： ```python from mindformers import AutoModel, AutoTokenizer, TextStreamer tok = AutoTokenizer.from_pretrained("gpt2") model = AutoModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextStreamer(tok) _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) # 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` 上述脚本不再对输出文本进行统一解码打印，而是每生成一个中间结果就由streamer实时打印 ## 分布式推理 **说明:** 由于MindSpore版本问题，分布式推理仅支持MindSpore 2.0及以上版本，且暂不支持流水并行推理模式。 [分布式推理参考用例](../model_cards/bloom.md#a1-模型并行推理以1机8卡推理bloom_71b为例) pengcheng.mind-mindformer/docs/feature_cards/Transform_Ckpt.md000066400000000000000000000472441476650141400251420ustar00rootroot00000000000000# 权重离线切分转换 该特性适用于：1. 权重过大，单卡无法加载；2. 权重为切分权重且与目标网络和运行卡数不匹配； 此时可利用本特性进行权重的切分和转换，以适配目标网络运行； 使用场景：1. 分布式恢复训练（恢复时卡数或者并行策略发生改变）；2. 评估/推理场景（权重需要进一步切分或者合并） ## 方案1：源码执行 - step1（默认已有待切分权重相对应的策略文件，若没有，也可参考以下方法生成） 在config中配置`only_save_strategy: True`，正常启动分布式训练/评估/推理，生成目标卡数的分布式策略文件后将会退出。生成的分布式策略文件保存在`output/strategy`目录下。 ```yaml only_save_strategy: True ``` - step2 运行如下脚本完成权重切分转换 ```shell python mindformers/tools/transform_ckpt.py --src_ckpt_strategy SRC_CKPT_STRATEGY --dst_ckpt_strategy DST_CKPT_STRATEGY --src_ckpt_dir SRC_CKPT_DIR --dst_ckpt_dir DST_CKPT_DIR ``` 参数说明: `src_ckpt_strategy`：待转权重的分布式策略文件路径。 若为None,表示待转权重为完整权重; 若为切分策略文件,表示原始的权重对应的策略文件; 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`SRC_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `dst_ckpt_strategy`：目标权重的分布式策略文件路径。即step1中生成的分布式策略文件路径。 若为None,表示将待转权重合并为完整权重; 若为切分策略文件,表示目标卡数对应的策略文件 若为文件夹,表示需要合并文件夹内策略文件(仅在流水并行生成的策略文件时需要),合并后的策略文件保存在`DST_CKPT_STRATEGY/merged_ckpt_strategy.ckpt`路径下; `src_ckpt_dir`: 待转权重路径，须按照`SRC_CKPT_DIR/rank_{i}/checkpoint_{i}.ckpt`存放，比如单一权重存放格式为`SRC_CKPT_DIR/rank_0/checkpoint_0.ckpt`。 `dst_ckpt_dir`：目标权重保存路径，为自定义空文件夹路径，转换后模型以`DST_CKPT_DIR/rank_{i}/xxx.ckpt`存放。 - step3 将`config`的配置文件中`load_checkpoint`关键字指定为转换的目标权重保存路径，若转换后仍为切分权重，传入转换后的权重文件夹路径即可；若转换后为完整权重，传入权重文件路径即可正常启动训练。 ```yaml load_checkpoint: "{转换后权重文件夹/文件路径}" ``` ## 方案2：高阶API 执行 参考Trainer API使用 使用`TrainingArguments`类打开`only_save_strategy`字段，其余步骤可参考**方案1** # 权重自动切分转换 ## 概述 目前分布式训练/推理，当权重与分布式策略不匹配时，往往涉及到离线转ckpt，大致需要以下3个步骤： 1、获取分布式权重策略文件； 2、离线转换，将权重转换为分布式训练/推理所需权重； 3、启动分布式训练/推理。 以上流程对外部客户来说易用性较差，因此设计**分布式权重自动转换方案**，在分布式训练/推理时能够无痛转换，提升分布式训练/推理任务启动效率。 以下为原分布式训练/推理流程：模型创建后，只能加载对应分布式策略的分布式权重，再进行训练和推理。因此用户需要按照**权重离线切分转换**手动转换权重。 ![img](assets/Transform_Ckpt/original_process.png) 以下为改进后的分布式推理/训练流程：中间插入**权重转换**，方便用户无需手动转换权重。 ![img](assets/Transform_Ckpt/current_process.png) **权重自动切分转换**会在`output`文件夹下生成两个结果文件夹，分别是**strategy**和 **transformed_checkpoint**： - strategy：保存**分布式策略文件**，主要有以下两种文件： ① **ckpt_strategy_rank_x.ckpt**：rank x的分布式策略文件； ② **merged_ckpt_strategy.ckpt**: 所有rank的分布式策略文件合并后的分布式策略文件；（开启流水线并行时才会合并） - transformed_checkpoint：保存**转换后的权重**，按照`transformed_checkpoint/rank_x/checkpoint_x.ckpt`格式保存权重文件： **注：**每次转换结束后需要**将strategy和transformed_checkpoint保存到自定义文件夹**，因为每次拉起新的任务，如果开启了权重自动转换，会将这两个文件夹清空，然后保存最新任务的转换结果。 ## 适用场景 ### 1. 完整权重转为分布式权重 - 适用：**使用完整权重进行分布式训练/推理** ① 配置`load_checkpoint`参数为**完整权重文件夹路径**，权重需要按照`{path}/rank_0/xxx.ckpt`格式存放； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起分布式训练/推理； ### 2.分布式权重转为分布式权重 - 适用：**修改分布式策略后训练/推理**，如：16卡训练，4卡推理；16卡预训练，8卡微调等场景。 ① 配置`load_checkpoint`参数为**分布式权重文件夹路径**，权重需要按照`{path}/rank_x/xxx.ckpt`格式存放； ② 配置`src_strategy_path_or_dir`参数为**分布式策略文件路径**，如果有**merged_ckpt_strategy.ckpt**，则填写之，如果没有，则填写任一 **ckpt_strategy_rank_x.ckpt**的路径； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起分布式训练/推理； ### 3.分布式权重转为完整权重 - 适用：**分布式训练结束，使用单卡推理**，如：16卡训练，单卡推理 ① 配置`load_checkpoint`参数为**分布式权重文件夹路径**，权重需要按照`{path}/rank_x/xxx.ckpt`格式存放；（注意：rank_k文件夹下可能保存了多个ckpt文件，来自不同step，每个rank_k文件夹仅保留一个相同step的ckpt文件） ② 配置`src_strategy_path_or_dir`参数为**分布式策略文件路径**，如果有**merged_ckpt_strategy.ckpt**，则填写之，如果没有，则填写任一 **ckpt_strategy_rank_x.ckpt**的路径； ② 配置`auto_trans_ckpt`参数为**True**； ③ 正常拉起推理； #### **注**：分布式推理的并行策略目前仅支持"数据并行"和"模型并行"，暂时不支持"流水线并行"。 ## 自动转换案例 ### 一、Llama-7b模型的分布式训练案例(物理机) - **完整权重自动切分为8卡分布式权重** **案例描述**：基于一份完整的llama-7B预训练权重，使用8卡进行分布式训练。 完整权重如下 ![llama7b_autotrans_1to8_train_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_ckpt.png) **步骤**： ① 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b/" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 8卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 1 pipeline_stage: 4 optimizer_shard: True micro_batch_num: 4 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 启动训练 ```shell # step1：生成rank_table，rank_table_8.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,8] mv hccl*.json rank_table_8.json # step2：拉起训练 cd script ./run_distribute.sh ../rank_table_8.json ../configs/llama/run_llama_7b.yaml [0,8] train ``` ③ 查看训练日志 ![llama7b_autotrans_1to8_train_log](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_log.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama7b_autotrans_1to8_train_strategy](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_1to8_train_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **8卡分布式权重自动切分为4卡分布式权重** **案例描述**：基于8卡的分布式权重，转换到4卡进行分布式训练。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 4卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 2 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ③ 启动训练 ```shell # step1：生成rank_table，rank_table_4.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,4] mv hccl*.json rank_table_4.json # step2：拉起训练 cd script ./run_distribute.sh ../rank_table_4.json ../configs/llama/run_llama_7b.yaml [0,4] train ``` ④ 查看训练日志 ![llama7b_autotrans_8to4_train_log](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_log.png) ⑤ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama7b_autotrans_8to4_train_strategy](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to4_train_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to4_train_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 ### 二、Llama-7b单卡/分布式推理案例(物理机) - **8卡分布式权重自动合并为完整权重** **案例描述**：基于8卡的分布式权重，合并为完整权重进行单卡推理。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True ``` ③ 启动推理 ```shell python run_mindformer.py --config configs/llama/run_llama_7b.yaml --predict_data "I love beijing, because" ``` ④ 查看推理日志 ![llama7b_autotrans_8to1_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log1.png) ![llama7b_autotrans_8to1_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_log2.png) ⑤ 查看合并后的权重 **完整权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to1_predict_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to1_predict_ckpt.png) 注：**transformed_checkpoint**请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **8卡分布式权重自动切分为2卡分布式权重** **案例描述**：基于8卡的分布式权重，自动切分为2卡分布式权重进行分布式推理。 **步骤**： ① 将**完整权重自动切分为8卡分布式权重案例**得到的**分布式策略文件**和**转换权重**保存到**自定义文件夹**下，本案例中保存到`llama_7b_transformed`文件夹下 ![llama7b_autotrans_1to8_train_output](assets/Transform_Ckpt/llama7b_autotrans_1to8_train_output.png) 注：有`merged_ckpt_strategy.ckpt`的话，策略文件仅保存`merged_ckpt_strategy.ckpt`即可。若没有，保存任一`merged_ckpt_strategy.ckpt`即可。 ② 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/" # 配置分布式策略文件路径 src_strategy_path_or_dir: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/llama_7b_transformed/merged_ckpt_strategy.ckpt" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True # 2卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ③ 启动推理 ```shell # step1：生成rank_table，rank_table_2.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,2] mv hccl*.json rank_table_2.json # step2：拉起训练 cd script ./run_distribute.sh rank_table_2.json configs/llama/run_llama_7b.yaml [0,2] predict "I love beijing, because" ``` ④ 查看推理日志 ![llama7b_autotrans_8to2_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_log1.png) ![llama7b_autotrans_8to2_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_log2.png) ⑤ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下 ![llama7b_autotrans_8to2_predict_strategy](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama7b_autotrans_8to2_predict_distribute_ckpt](assets/Transform_Ckpt/llama7b_autotrans_8to2_predict_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 - **完整权重自动切分为2卡分布式权重** **案例描述**：基于ziya-llama-13B完整权重，自动切分为2卡分布式权重进行分布式推理。 完整权重如下 ![llama7b_autotrans_1to2_predict_ckpt](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_ckpt.png) **步骤**： ① 配置参数 ```yaml # 配置预训练权重路径，预训练权重需要按照{dir}/rank_0/xxx.ckpt格式存放 load_checkpoint: "/home/zhangsenzhen/2023Q2/checkpoint_download/llama/ziya/" # 设置auto_trans_ckpt为True auto_trans_ckpt: True # 设置use_paralle为False use_parallel: False # 设置run_mode为predict run_mode: 'predict' # 打开增量推理 use_past: True # 2卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 1 model_parallel: 2 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 启动推理 ```shell # step1：生成rank_table，rank_table_2.json为自行重命名的文件，原文件为hccl_xxx.json python mindformers/tools/hccl_tools.py --device_num [0,2] mv hccl*.json rank_table_2.json # step2：拉起训练 cd script ./run_distribute.sh rank_table_2.json configs/llama/run_llama_13b.yaml [0,2] predict ":你是谁？\n:" ``` ③ 查看推理日志 ![llama7b_autotrans_1to2_predict_log1](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log1.png) ![llama7b_autotrans_1to2_predict_log2](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_log2.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`output/strategy`文件夹下 ![](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![](assets/Transform_Ckpt/llama7b_autotrans_1to2_predict_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。 ### 三、**Llama13B模型的分布式训练案例(ModelArts)** - **完整权重自动切分为分布式权重** **案例描述**：基于一份完整的llama-13B预训练权重，在Modelarts上使用16卡进行分布式训练。 **步骤**： ① 配置参数 ```yaml # 16卡分布式配置参考 # default parallel of device num = 8 910A parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 2 vocab_emb_dp: True gradient_aggregation_group: 4 # when model parallel is greater than 1, we can set micro_batch_interleave_num=2, that may accelerate the train process. micro_batch_interleave_num: 1 ``` ② 训练作业配置 ![llama13b_autotrans_1to16_train_modelarts_inputs](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_inputs.png) ![llama13b_autotrans_1to16_train_modelarts_params](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_params.png) ③ 提交训练作业，查看训练日志 ![llama13b_autotrans_1to16_train_modelarts_log1](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_log1.png) ![llama13b_autotrans_1to16_train_modelarts_log2](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_log2.png) ④ 查看转换生成的文件 **分布式策略文件**：保存在`remote_save_url/strategy`文件夹下，由于开启了**流水线并行**，会对所有`ckpt_strategy_rank_x.ckpt`进行合并，得到`merged_ckpt_strategy.ckpt`。若不开启流水线并行，则不会合并。 ![llama13b_autotrans_1to16_train_modelarts_strategy](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_strategy.png) **分布式权重**：保存在`output/transformed_checkpoint`文件夹下 ![llama13b_autotrans_1to16_train_modelarts_distribute_ckpt](assets/Transform_Ckpt/llama13b_autotrans_1to16_train_modelarts_distribute_ckpt.png) 注：**strategy**和**transformed_checkpoint**两个文件夹请及时保存到**自定义文件夹**中，以免被后续转换任务清空。pengcheng.mind-mindformer/docs/feature_cards/assets/000077500000000000000000000000001476650141400231535ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_Ckpt/000077500000000000000000000000001476650141400261075ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_Ckpt/current_process.png000066400000000000000000000276171476650141400320520ustar00rootroot00000000000000PNG  IHDR9J׀sRGBgAMA a pHYs&?/$IDATx^ y{K/$/I %b gpvGZSلweR؂M@ v1//q,َ-B-@ kv~g{nݧqg:5=3=ݧO|;,/8YwB!MOQb8evm%m%ڼpYQEQEQEQEis*Afc>P%l|ѼjQ{#Ae潢(Al^5"P{OQF^mn4"HQEQEQEQeB((((A EQEQEQEQ1 A((((cEQEQEQEQ*)(((TREQEQEQE#((((2FP!HQEQEQEQe[!苿sP?eJiON kJ^8v~f)YCk)[->?}"Sz~2vŸO}%ğZs6XO DQ$.s3a|+8P`lЊBQ5qR }Bn$Ž;J?ߕ 㧘E{Ċ.IU!϶m~04c̪~+dztrYRIw`` |BP{ҊBP n*e`wJp\[,ͻ_GW;,"|?]J1dg AYϙvh6r@&\[ԬxOl&חҐSyF'd^HL'ZIҞDaU+4p,=Qz{};1Kq6.mzK?)7k_#naidAr63/>o9}9i;FThFyv"4f.9&v(oVکsR9M.#8),ƒd@!" GU{\EQFN\0`Ϡ;y[(bXYJ(6m1B?+6/dɩ$""3T>@JY|7bHt9myM^޹k7lYlWz7@WQ r@5ks#)h0LD"TlJ4К;*"|3dӣG:k(es-f9 ]#&#tuY:[]"2I(T:7}eㅄ -R}`( n#}pMЈ'fݝkba`a}礵Tbb-օkF{᱗b?W$2v u-C-9,"'fm>${'Y uCC+hj囥?sK\R,KÏsPKt !VZdidB#!Ξo!)׷#DoGCR`EۧmSH:Fby H睽LMʼn>塛v2U|ĸ9V!ޗhBlDbd( {"!tv &N",zg02㗉qYG$L擗e_,3B# [H_|w)ٻ\O+BP~!hBwm -_7Ē{/*{Sm2tn~В"-zfZ9ά XQàXzS `vbiQ^sU"wJ$<04qĎIcvTLq3i0+)Gf}͒óY$t3J=EZQ!(1'"2NK q")c3H3~6 }#08s04',vE^~ǞRkKf^9'(tN?fyg."Dj ;( *DJR"T4B嗊d HmH;'VM }|AÍ=,EcN"pFqu̺U Dľ)߷02p IxgewhM'*RXRzցh;/M&yq 3| .K ew\cJ|;Oտ)فD"txDy@Vt}\@භw2:i`KƊ","O^]]Wh!E͏EL!Ro_1xt6h'm4~z fhX, :`9UZ N1 aD@ܣ٧"SEH^"J8Q="buZ(BoʛNXTG#Ip#ɴ.EJz&fpb(V!S'B IQ1H{JP$?iO'$a/uMޢhNO)$VZHcYKxVl"~B9mK5fƖ`ҞȴY!R B6:N.Y$cs;^6A<=ȅ'[;a3)7T"$5;<\FM.tIۏ%S}ybߠHkA&L߼JSH _kXKh)!T UIv5>:M5QCo̓# {v+~g $#30IgWe(VwKHâ\1$}t7}(R(2N<-53,چ} AHΕ+%Gzf aʵ;Gjת ߙ￳aa_{ =G6~T]n 1vbQ I7;ن}IaG}+ɓ}m42ΞkmGm>}-`&}^9[`M:#o޵0#mw|qI~}}}Im\!Cw05ciCoHO;"{8qҬD2G-$C#EJ+!`R\4Eă\WΨr'XUw^\0b\Š$o֛d{I ҰTs 7ɌYs #%0 5nzy)>\S!4nїߘe7>,!E_ٖ]>3Łx=d=߆{ *wJo7~37B#8!|>w]yl%tl~%⇎h &;Fkٝ.g,ΞKkYl}HCJH4}>}l{oiא6ގNݵ1-1@Uws5JօG%6m<gǔ#q:ֿ+>&k!In 0 H,,ꡄrCL溩l AqqAh\]sҾ$5"qg{24.i,7gs.Y9OܬgD ?RZ+^Xz0 {.ai,BDaJ"|aS=REIχm7U #ZIz7{):ydho{g>^Jfp`x]xw_2NvvGtvق<})΋DNcc~%5=:<mZr²ҒUžټބU!E+}jn%M/#\!$#qFSʆBcKyF%C#ɥt~T릒pݚȎ!T,& koN@}/f[^XIH- #a#ۯynMbd,){JB8^j (^rޭ^6մq A;?{a {aqBDH\ND|;wgpJ;< A۵k!_fkDB.5p뾗yډ@hoBA o޽_v _p6YD,̫ ?"}ɶlJNir]E?[ j.x[ o i^dGOُ8</gG$D` F Ai9&YD;.>'.\<4od."Ds(]_g 7S(J8` qKD zębywNs}ɕIc^e,4"upk5~{2@r.υ6#=} dG\Ң]6]Y׎p'~q j./(ukDQ2v$Kŏ>3lR (,&)?M_jg={ĝ7_YeDq_a~\V"-9q\ !bY>#pI㧞-=ռbQ S NV1ӹ!H;@ 8D೬" yqJ 녴R B"C=C>W-("G% ?.GW "Qw|#q)ƒh<# "GHi'tmRwZ46^}&̜6μfi+|'e[@+DZ-i\@#e߳diX&t5 PĚ$m a$mFA@`@0+<|)0)äڍg\RjAnq쇫=29al(, YȲݹL4t&;w Kd{mVE#j@Q:Aooi>"PX@#!rSObG=*W?W>yOFoED tW3Z/% +}]^]lr5}eh %-YS-lW~3gvyo(v, o[yÏSD@`|i qAA>h.?hDjJ6>$Էmɓ:+ڂϱ$ApB/|P EЃUJ"P{>Yw˾5|& DCtr0mhߺG6Dˈ@WB"Go"" MQ.N 1덈@z 1$|M0س_^]J.~6(+8!rΒDDPR++RF;Gf7"RIS8Ǭ"Psar߈HvTfnno|n hQvz[}-́`6yxDzrBPw;čFdrs`K&+mɬԬ~Pgj4nїE qB1bP3@0æ># O+D,W/{>ؓehC<̜6iZ9'&}u&z +H,I~A:LLQp"qxC$1,E'^]듊 Wq41+IRYt :QO)DzJ >E*oSo+JJ m7mza>D ʔ+^g?2=m|5CFH1hG{F;G^ͦ .v23čƅǴx\`z=OS} VшMB}J>"4k_>?(np!LDCLK(@A.Aw-) Xv}'_g?6B={?s9K.* 'DYD~R8wDrDi)X 3-.KNJW1l?:l,ER\ؑ:1P\OHc\z^[ bΏ ݣ;tR%i7%'H"H}͇\WA{ 6;oKԒ'D~O^i%uqm̆"_| X1"G$1ZvsF*eׯ v:K+r9Γ \EBn.:z ~d,0ͥ| YӞy}V.>~a@uÁـp`18o%7yS47OIeߛZ Dh Mʿo6B0nReXGq_v,$aDb/ aI.QWZYxbD1!` Oi Z{"j#Ap 3.H"\K3 YrgQa/7{K}Pwp#iF0tHݴ1Md.K6FFB Y2 P.5o[V^w];ޯoޙVf{W~*o(((Uz(((aWGqF"Czz@@!==D"C% Zx}g {(((͔єՔE_tppͣ x(((ZR6SFSVSfSv~y饗M)((Ƹٔݔ"/!BQEQEQE_)õRO_"e+EQEQEQ)e8e6SO_"F(((nO<ߤ\'/K (((~6իWo ruKYNESSf|O{O'ãW.+=n(((~ڵk-֬Y3/\n))5^w‹W{W^~53wxΜy?g@?Pyaz=EQEQEQ=gW칮wq7އ~px7P_歆# wg/}y򭷇_:xtS?ORUjިKv[6v{1ˎ[鵺^#뷔gj~YT^c˩1?>fǩexW^ͅ~˔7m?o?K.ӶW(jt7{)|[Fu_՗+Qr]˿ 7; ۿ/~uF"a}da(?ԳzsA^5m~@:I=Op-s߃ߙ7M!G@_T IZOkZ]z {} z>[8r{~UzmÚQTѸ]Vվ)))|JN jPEu%^N7ܰ>lհsݽgo6]A_6E5އyo:v8v$7>v~O,a sǚ6P$+BN@u]RWji>VukN}v)O1UiL5yx\9nuj,T/F:}tl(vCaʮ5#/9 ݐ]O-Xz~sn]Yow#M̅P?=}2量?; kŹ}cM< jjz_yc}YطTO-W>揲ɽUUpvkŎŬx\Z7 sTu ^@_gʂ-c6mͅo+mڑ<=VS|>nml>0_lO=?7mwח_OjoZ5FΖ-g~m}\n}khCQE Yҗn̂?9\&. @0OmeREwF?wlw@/e'v>ɿkuP_3LzfU_dTc}sk[ujY/:2uOۑ* amʖ}XeӴ|<=^McW_Uam|cm4Y6jܯ2=RuIciu6c6~]֧k}+^ޯ_&ۭ򯽟~4FjgCRc3 6O~lFZ/Ee^ok۠ݶϞ۲O*MK=F{l8ٿ:CY  /[Pӵ*[<8d[lkzjcg-_6.v*M+תL*M/ Z+gzіUɲqu~˽ ZvMNXu>NZɮNN=V{cMɴsYqi6ZY'v* NomsMb֙vvoX[=ײmCmy vW W\n3}r^+&mIm)+-cIm||?~?M՟_t}|;緻1P5DAenj~ߋJ1M=:qyoox,SYVufOeO-YpWC?AYehWZƦAԖ|m~jlqߚɪθX߶>0}2;?}h{mEQ5[_ϓ5z^/?+ ]ku©v| ꫝ(k4tҫpU?ZN}i$ڥY'S}~՟mܿuZF>:m?}[UVchh]~l^\֏kos?-6jk嵜f|M[ڸYY`ѺRUzlSoWߩqx~l}oJQQn~|ԷaZ};^>ϢJ'URPϦYemmY 6_ ֏zOmUӪ_?ޏ/&_blVZuibjKU}-x\q6PEQ{} CG~7_5׾z%KyuۯNTuOҼo=dUm_=lgYa.O&}s=V>y*VX(~|[UX{ۨo{_oyJY⾋9^ZVml{UZ^T_&/h֥+ޮx} l]m܏}KU>ge닷_i;zowLUYTu-H-U}n!Ж~>j^2LmLY&K|'=aeq *S':V}e':R?ZNN}zo-+-[T x|?'I'~[mQ~n־vkY[q?EoIQ_N{ۨx]uae>ucߏ\mY[>JZWǏR'W_+.KԺ6-xe~}_vAexq~mYjo,A/Ki~YpUS,rOjY8dmiև#}=qY[/UejM6o˔kjY4N66|{n}:(U|׿y'_^}n媮UP?˯so}g$a^JOrR?ZqOӬjSi^ՉJX~ԇowꁵ2f˪[^?ޟۭ4׿q?EͷcЙۨx]ua]V1}x{Uu1K/k/oa^Rj\/g-Pj^2SӬ}<_oj~?EecEm۬_6gm_5_}[?/x\(]+?mGuv ۶7 Kwc 7w=+ux0󩡇y9_d6= x$V':as}ՉN^-Pɼ([X[MO ]z3aEQҮ/իWguƯg/`?}髷e%)*LԢzẗ́+Ktr<5EO-=us,/"SE-O+7}{ /հqaڵቧvϛ_UrtM藈^{-:z(j-cUvr)w?nA[Uv{j(=EQ߼}bAKͅVe9eK{I)((-e8e9eK~sgɃ(((Rksu^KK/\(((RfSvSk@;eC SEQEQE-~))))2oK~BR!EQEQEQxlwczz@@!==D"Czz(wIQEQEQEQW!==D"Czz@@!==D|}<<^ '9`PC oy4\7o/+s׼cȗ}YGA?C}/Q[ok[|gcTMtߜ nϗ|8jtBmaڂnJڏQjRڤWo?r\ٶk~w|A^ӶOoge䧥Տk<}mjl;էӎcR#R6qi{K-+6loR]Zr\hQ<N+7ݞ_]M~qxzON6N밓ߪ~NFt4!tmg6se8L1P2m>-z8X DRf髟_mض OuJmt\5R5ϏM|;#OcMcЖc۷i(;nRRcڏ^gULQ@Tr_e'HsU>'>l_?va6of'qln? viAi'UU뱐1v;h?˶?UO~}~l[8hy]u?=:֮ [wѶYұmz}]Z^RCWv4a\֭}I=qorHjZ܇-cbٱ:Ǎ~~c\mtqZޟD؉mٳg9'vBh'bj'k:'avie'sbZYڏǟxvA۸O4]'*E'h~Q~c1ԯj|8cѾhĎ8s[\5mLJmo[϶o8vZF4/}q_ElԳu~Kj~Ѿi6FuճcE fRdӴǕsܤgϛڧkdXm'8+veDNer~;!e'svNDϮeg_mce[ک}l~Ѿ zB_{QU#v_-k6qTuʯ'dz2F>շXim靖awˏgLmooǭ-h[~RD3JmC"/For}2S{چmW=몣qce4[jh{l|Lyܟ_?@t;y;Q*zml׎d'<;SqjOtBh'C_S}2~{v)n2m뫢סmIIeCnmN䭴q@2~lM=2u)!;ԿDmuٶj6&6F8ӍQ+o<_5jcה^?ͷ}cƖ59Fߖў>EFl615ZO:o=嫓'M*n_W$,uf'bvf'v"fm=:!կ~SxRi'/#u$IMԉmW&><-iZ^hƸ Uh6<~}TXپ~i9AAXuzض4->~m諵Rlw_/m۝ m_;رeEӍ'X|cY46uV1c?@ɿZ~eǏ).$UvZe|MuvWRǙ-[Nqyy$xUɖ?QJ<;AF0kiYN6'eʦ NdSmRl N<曺K'mi[M*ڟTٶk?b}hͯ΁̖\m4Vkyem'羽kyb۪鶝֯JAضk{ĶUDŽ37mO͓zoO[xBۢmטhJuk_?]ϭ{\v6ZFˊs;yEeSll%F-皮Ɩb(^W$J?KIog'v"f'hvObS/;q9_vkOxmyۆx{5=._*۷T?KU5߫&e'u>WJn]׾lM!B} 2?*]-|^5}EۤR_c_FcWZxLb~L/w5B z6[X~w> ^6..mk̶ӶóL?x۲}"i;nw=־jm<5/_9Wm^? ^ru|S'A3zѣg˛AQM^Sc 'NL-8h<`Uqcb|z/*)ãu(ԷyN㜺GJWRaƣm~scvb^˾Q uS2ԷAG׊`:%m~7Ƭ{0p>Q' cx0nQ"dͯAq_uƍ10 eI;M km\PXkEȚ_u =E?[cܪθquWuSΣ2O@IՆ[Y&_;Odb쪰[j57d7mٚ/aZuRkקC?E?=TY~%ǸUqX(j|O.@ŗ+vOh2nئ[yAH j~/V], y}AiTcJj޸UqŘ-{02<Kɓ±cwy7|'O9_miBmU׭ڑozovBطu7{{8VgL)m\d4@/7WY;}K em6Ve^+W]7oSԟFYHmkUjml[SOx^~t8O|2/[&'v}v\-;*iS)5Oq:ձWH˪M܏[ی稫q àJo/5\;4w{puw|'u;wxne^:jٖyԦ ^[§*_~ŵ7ʞ7U7k}$&*+睻¶=@PmUTF:Wj+. ViiƨmznSKC\>|8yާ@_TMC;շnH[{0賱)'WghÏ}/}x>e3aɶm~'Yh֥uïgۯuh?;Q>MKmosQ_ d~{ov+GOva7oͮJg_dۆzU~p:\+Z# k?\rEe.}ܴx>NXLCǪvUGq+ ei5N bWϫȂ[W_ q/+Wm}UQ׺S쎄UW!hj^>6_W6UuRq1I͋ˇ@|tDi`UUP՘I^dž/M;Wz-USnj Ӱq à&RU06˂)Q͋Y,Ǵ ,ZKa>_FۙCkY}iǟ=n[-Y}6sYY4[5¶Lb[o?Ju)kwN>O=ş WZwL`,x4766k!=yzxv)WcqӲjS}Q]j=K `e?HA} Y b[M JMMt^AS?Ȱ0ZV`URB*2ZV56>ZΖsM8UA_Rߗkjˮ+*, > ޞTVC mKj~Q8XPmEЯy*.vȅy3K05<. un~cC^?t]Fm,@ ~0H䲾,XQw< zKI׉uQW5v ȗ1͏{z1~|ml5~E?p6X_qU*duuYl}@K->Aɸ2[Uo^6o)z7:˪,kLLqW<7$֗ԭ) >ǚP0 sѦiL` :x(+[阶ꎂ*?~7sZmmAf?tL69]WyБW7))r|&9M ()x4`^E_Fq ָYWV@;_O}ƯG˪t'@hf_UZVhDתmz7G_܏_Ȟ@i)=oS,Y0~҂W\ronƁ&6@_`U_nW譴~Wޞ_MYieTAeM(+HWwϲZ®&[}jz\?ahF댿rM>[n#{F-oz-Oe`ˈYmyho3ٶYkS:VYxaO)!З wuOxzV})2u&4ԕ{WOy~Dmn}7+oμQ$׭e+V,U BX?ծ4 ʪ]MͫSuRq1J͋ˇxn~Ԙ]j^z;>ez*KO~XR6) >결ʫ?©}.h6fm,Pk-HSA֯CuUk ߢu6amSqخ6)/x{e_-ߴonr{ޔ^" c|=.ܴ }N'q('Vw<^&cⲫC%5]DU4n6_a(ۺ~^u×msj^Q5y]V,߽O\U7Jc1YpW0K}UEsunPj2nMƤIk9-_X5l\4*=W7)~=N-Ӵ0 U:k櫍ª(\-/ֿC4]-j{ۗmR_>klovZn˨ct?)VK׼r*=#ZDm,z/jAӲvŹ^Ufjx47M ԺSAlaB,v]WU~hYu=tͯ:h]u-Sjt}|O}P(-VZɭ@_TZ_V&cMX~+~W=W? }*x\ovޞ뤯XQ{ ?״^܇gQ*M+*m°>.շfE޶Ch=U{n}m6ⱱhG!]^Wu%އvs6≶遇Uk" uK+ qMXqx4:^qʖk]c*(j#~PTE^}krE>^@o۩uYJתiȲSRk&,Oi@}UyYp|nPj2n:}x>jrO6}cz }ש:XkS/"X0 l¥걙g UW܇sQ;]ۨ/@,4]5: ZhDm}/V.&c:v}!֏i!ԷBw%e!+[ kf_[ԧ׫.-w8yjῤӶ=Ԟ41~ӖaJˎ"?ΪEGqӋ]5A]Fe/8BUs jǵh-6ƴl=:VMCW6Ze m?+ d sjBYY_q JMm^dϵ~<@|qB]_U8y) >+V5BoLWW;UUh[®}4^}/PǁX.cqVMZVm,빍|M(*O ^cm_S,kFXA׾ʷQEhY8뫞ͷz>V?8Y[Gm{W8_Sk[uW.su|~U"ZxKEh2nke?go Z +d&㦿/?4J˧3nkUy~"NXV}-ba[ t ~9=V}ār8ک ZV}ik4kz|/36-m=Eţ׬ ]kBm臬x0nQ"dͯAq_uƍ10 eI;[}YnPbWqc̺) x{@?u_+BUgq 0^F$ЏG׊5%m~7Ƭ=N`8(ߓ1Uw<ZWݠĸͯ:vnNk;Xu¼=N`8(ߓc6Qߖƣƭ??p r,ƣ\7Ƭjr0p>Mf:dF^G[?kλ'p+~˽4:sUwnq EO齸mǮ,h0KOf:`"Czz@@!==DPSEQc\|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TzeV|VSK8)jkj9TcyxxDJm6'EQra|VS88E-}M-jcѣSMIQKR)j8E-}M-jc>GUO.xe˿~st*/<\O^}WYmOv$z]፷KΧ(ji׭>55?UyoNLjzYUZ.ޗg{#|{<ݾ7~oשEdT{|-w)*UM:gqn񽋢R~HU>7M=ִ[miNT/<_/p~(gIjz';aFk꘷iqY ul}o{"Տ֕Zmo!e^Om6U~Cػdjz?~uuSTj@?7а}}h_Zz%u }~QTc܎Ϫ|Md}z6ꫭ;.'|Rm4O>=Dzs> OٟlCQMz/1>}i%{i6}7T}7,W-1mHS)Z%驲e'Fg5$uu2gWc\eǧ|{iee@ଶnǯ+Ӷ?|6#~ɟ7uZV[&UMlyRMZk/^q0s5j@?IJo|oVpH)jy)Svrc! jT&gWcuˎpj@ol =䗳{?zӖz~>߶gU}mՎ3{ߨ4M!ݞ1mǷVQ=FQì6C.}諞[;}CDS)j) |0ZN4 R|Qq59}َݢ:R}:}cW譬3cMSiTc}hzl˩?}.+/{od[jVʿk.~cǞϳn߷TZ^!?SmT!RVԾt=<чM UH}qKQK$5NCS<'3^v~>iT\Mq}>>WxS;>o ^ʾ[Xs>-Hh]~ϳ~U?ϲZe68.M}1h[jo\}?Lo5y_x:RW5~ئ6֏ԇQm;.7D5F }o~}tUQU>4,EQK$5,fGU OTUgUVMqzN[Z.V֯^* *|jgض6X6ml9{!l?F<}|{l[|޳@U}Lώ{*jZmT޾!kY޾ڇ}Vշ.? {1H)jW>{ɼ}Vs+}V鶬?!Ҵɪ3*&Ǹ?]cn˵qy=ky|^e/oci۠y@?f}=KmoK^͖o?`)U:^ux{q{ο{O_5j@?қ>&k׉,D'hk󁢟ow.7R{Z|՗n7IIQT?KRNrϟ:_EiTY59S׾jY8e&os ozϤ4vkoo1z՗k9ۆ&UO_Uo2~wx/}5c(gIjzQ百/; giTQ59|k!ٞ!!.^õ ZF?܏c~ #@= :m{OBL4֛_UlU^PVM*vc\ǽkg}6B~aV!E웿kЇ <7n EQK$5=U9?RZΟXtXviTqɻi;uu]˦t{UqZ~FhYW[?Hx;{MVS;^GN=0OQuoG{Om6'EQr.IMR5< ޤFmS}RE-g5ESqS+r6>SEQR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZ(jR/qZzZF~F8?C:͔Rj3?oѿ\Wo{'9JqS+r6:m$ISL럅pC@] MSK?h?_&7~< m~o>1@#wLZ#X<|&,> @kz5=g3h@`M56.(`>O:6{p{R] W>l‘V23`=}"yu{p}W0?zX>QŐ qoy N m)t}i; ̯d пxpV<&uXFZ|&ufW>.xey/C`?#?,@$ʓs p‡>Ѭn|BO~.[Vm+_o q6Oiy?.yw}Me9=R{{xt'.XIݾQi=ƥh):}>RlmUGmƫ۲YW^WW\zt@?>ʼiV/ Kxz@Epd>Ia>:g|BߴLN dX'qh¯ :,Ǹ΅RMF}S_P}}7Z(c?zϩootN}B^e)eW5^PW-V5Uz1>۶Rپz{2y~נxԏ_mRǀc]>S"ЛoR?z- t@?>ƕB={W>\Lh$ϟNNt8I?\1c'JQxLD˪){m޶Çq~{bUd6TUXZV}\ ~\OfF۫qԶ}MuѺlkW:*Z+A]`ZXpu*sr vh[Sw]4W}sgFK'} iD;w=VM;'vxڨT_mZ0KqcQRa+{+cSԧuaMTS/]:^֟˔훨>'fwϖ>}[s*ЧDl;DZ6O?fm4X>[@ŗ z@?9[^xz;':ɋO`Nb|Ҳ!^lmc(~Qz_轠͋#RӲb[,{m߫tl_mZ~XJ}6<{svxq.[Oh|bSieRZ%5=Je?5.=wߋ?:?gox_Q[wiY!A*ש* W[ v>`@fU6 *pOcߏJ%eޫ:՗=~GCg~6^ej}s!!FjZJ:.ƏCǚ6E}eRmnd, ]\WMcR^}l]]z@CCz#f4ݮZr>RȌîMá깦 pܯ8It/?nޜxIO,#1u}G<IAm->0Z:ԔN`J[&ta],؇T*bǴTH{#ދ*z5xLcYR$XkiEՏI܇ki3Nx}?gcگ1K-kW+.*UY 5Nj"cC$#k-/4Mh2qoO}s 2-׏Mjt)>-omRH}P}ujyۑo[Ѿc+zL_~N>u2Wt++v"MɪBA܏˪ҩ>.(`똴fǷ>wM|#:CޤHT8e8sAOW;Sem D_}Կ~XjW ?ulՏuqh{U m_6%#Ge6-Zml=W2_c/FZ|&`įvdEc'~=ZJ9_XǠ?-0vVu_$͏Uel{ޣZ_*Lӫھ-J~sRDžW}^O?zh~_*_x>k|<t |{-믪7q׆MRz ~^\-|ŁO4O(lTSN|p|-v>kpyʯ?ޞ8>m,Tnͳm͏jxl|mw;6E1k#^V1V˪?t;O-klo~;ki񙀥A'}7)`iFFWiSWoʀs6&ooB+ԗ?a@?yHuI./ @kz5=g3h@`FxLZ#X<|&,> @kz5=g3h@?y'E-3((f X<|&,> @kz5=g3h@`FxLZ#X<|&,> @kzDlڼ5YqA4~Бȶp߃ڨbٯeֿ۟Ο/>L a̫;ޛcѣg.T_|?7`8tͼ%̴LhKd5HAXh=wg~} oㇳp0ޯ23{4r>(q+[׬{G 7qx_e<'W _>5LMMO+W]>fu7szzsٲj^ [x븶yZN˛qEλ0[}h-{Mzs k=p>ʶOJi<4.EM8YO,G15bm$>hۭm|U6^ؖϺ ү.ڬ䵜s+!]WuO~Y7_OB.7'HD+G9>{wgFO'vRh'E':iD~](X1ga%>ݿt.j7z[9~rEq{N}k}~Ħv3},N)۾̥hxoqZOٶE_5ZCu~:k:L0Z:';rOFW|L*(XRS'Ѳjes~[=-|ܿߞX5/)G}7oczXKj=[N_9Zokcϩ@Mk}}iYVuN1WbtZHly?O?/Rm^{)ziY~1E-EmvʶW^:^~-g?S>~G|8'V4>12uUC _%2Zsmom`ۻşZO_uٳ7 oY-zRC T+]]ۥXW}nJLSǷU[TY?~~Tj/t-ko^ >֩[$?*:S?*߇T"v[x<~c`ۦ?#C<~;TZ^ۤue*Lhd0>!I^YÀ? %>'vOHկm.XSТ,z:&(MOZ{hD>1~Ͻ enۖO=.//5&^O,8 1_+?݋!]_. ض1~xl{ZTp˖,Tf0^º/Zx2]hCP7K@/ЪpÖQWQ2X:(~5W,T6YmF[>/f}-_տS6ݶIضY_\Շy ֮mWf>O/e|U6iʯg{HVTӊNSk9uxL;LˎNC9.`ziZmic)R$E}m_-U>s!!FjZJ:.ƏCǚ6E}eRmnd, ]\WMcR^}l]]z@CCz#n-h|pCRȌîMá깦?GY~lR$UmxsvLyO&5>x$#^֯@ϫ)j__x=ֲ Z|&`tB_);M'LKկ隯䏻XkҿS_}xcjH<αLӊM:cCgJQ&$_EcZj=}W\vUXp=k<E(臠I@F<B[8l)Yc[&X:סzn0~^F}۾I4.eۧmMj|bʾxYN-vپx;R⾊;[cc'vv򩓹]t]hNV ~_VN}|tAA,խ6_Ǥ4;}FnY}[_,{ϕ槶W}q[=_f_ۦZq_0&UG*¡,[ޞ(>x\PUjoNU][mC=/W1߿7ֿi;l, Ïl;ۦtc}i~ܴ=^-gM/uv?Emշ~73g'~unwX&[֟,;)ѲW 0ǚ?lɷs* i~=/cSRogj^U_l9Uj,ۭǾox':.|3zcFTјHcQ_l=}+EWSkYU˯6lڼ5%cXC$7 Pj+|yZFa(>iS;5O}<[NoOy_6*^vX5ݏS<66؝oJ/ceՇWƎ)Sey}Zގ7LҠ> ]) HL`R޴ek`ҲmV~T)+0m1.қL@ #}e9v{7[ˊߟǰ}Lv/E~FxLZ#X<|&,> @kz@ŻBP>t6wc_Em?OQ_mySπ}#CCSEQEQEQՠ ==D"Czz@d~¿Cn4 ΟD+osB2bZߣ Z?NB \щV>Ή_iնMN֯~2ߝeUYk+Hcxj'Cf(yշVY{itu"}=.i[̰͗h?c1@$$M'Ret N"Z'%qNDtSx^u)2~9AOh/;yIIg'n@: (l qjU~^S^K}Wir۫{Ԡog@1H}/K9OqkRC}?! cy@Yv:': e=;Q'4$rgd'v'Y|{l,SBN 5ү+_quu愳κ5~N2g͠_:>qO~^US'jW-=A?lmVqZ| ST692Fߣ:P6u4m)'6峱/y}I<$扛b{vɍ}N+e'E')NsHWUۘV?mwܿG1(lWL>Ihr|[%/qWE/lڼ5YqA4~Бȶp߃ڨ-PI' ?d)ƛFm5M9>P[Yos''y;gE3Z'1_y,hoo$%OME'qEJscm;Zoζ$`,5~ILLqyۙؾ?h_'PWTp $׹mYOc7OxfP4+zy~}*Rwܴn-7oLk53m/!U~<5*>.mc<3ꌟ~E/;n1>V_˼c\`8>+=z,Vڨ}{o81q^8>N4|)}sOIQ'u-TfFjg O*>e#nc'!G-,qY&UW8Mi_޼cuFceRS#s޴6^޼i/qInmlT=)Z_v "ՙm6ߠ?ظhxߧ8Ai8|Dܿ;}WYל|gU?tu<'zui=nw9X@d5H۶NOs53a 5;^02:o:'vB2 XߐIEɒ:.N|ooL?s}Dif8-@͝<'0 ui"VI mLJIӍeZc}qOvSՏ?Ί*~M66Ӽ?S6O}tya{4r>(q+[׬{GԮӗ6*Q[X7i~"P8!7_n| N N<h}eʟ$dNOl_6Y6n|R'Csq'vjo[̟֗ba˧-K꿬GOUODm ʤ;5m묧5 %m>c\T=&k(c8ˎjRP6m">֣XΤ4}^1}jegfZ ߣ`S}ݞr:h78z?a ү.ڬ䵜s+~pjwڨ(mx"|St"67i}:? 7ԉKQq'ڦi>dDOPuv;nm7oܾFeW _Il0Wζ=#h,8N*TΙ$Wͽ<=NGm}j?fwLv`nrcd:~~d2U-0˂uFxZLQKQhf5(1+ާyu3R6{iƺd:~oSGw}z67yz?,cߋ=}6<ߖqR2ZWTezm~?4g\W]}e2a wlϥzRQjEpdpޝ!

y uj͂x|:!΋Qfh߼?~yqM+o>'1-SGNl"EI1TN:Lj=u-˕O>o7wlk?9[_/5M?g}탓ZUg,mtXm z?ůgGIc\R;ޟ1fcJ6]#-OOzDV9b1_&chъdXSj;JMB¼m'9;ALtKdNɂͼ]@ӋjE4MThm'#>Y;9d:1N<}NNƻh0FT:K(VO:˧HZ\4>v\O<_rgomvVUYǷ:Rh5 }wߣ83]jR㜚6ö557|;Sylu#X0uٳ7 oV>e! .^:?ܳ2Vyp;U;fo{oZ!|yv]| WZ(/Wuzd)h'&vay7[~ I*>MQy'~k{RKk֦I(jmkr0)[Gj*JT'^&'Ec`UgHUXƠ~g̺{:moam3]:~qkF3DžqaQj[~m=o=1^40o l{fy>j6MAމ-Y5mͺau_,e6C;vaNm Fڂx-g&o>b?HX>WtB0}7X7  oՁG .4|Uׅp}MS=E uǯrc{`<!==D"Czz@@!==DP~EQEQEQE56B@!==D"Czz@@!==Dmڼ5YqA4~Бȶp߃ڨ- ۣ'oWwl7 zGSo '& 'v|'©n ჳRFMd5H۶NOs53a 5;^tͮJ}\xVu5^;@?0Jmptu_}]~ŵYqY)k9@W;36*QQD+<20E77޼V ZcS;?&>ٮv~*e4z'=uU W1 M?tXQj;JMf=i1_&chъdXSj;Jm?p@:?gox|B{]ZzuV)m;ge2)>{wv^-Ws=+r첷cAze|=Sa-[P->kښuKx ֿh_Ytmh/!wW'zR*ƍ{ ZNϻ\綬xz=|=z,\pѥChVW.LMO{{xn-~>1[ izV*ЫO@cM:uꅵy/ BgA߮+tpo`$ }W\vUXp=kܚZCD7%zR*qWVmz@ަA8mUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@PO8+[쵬W\6mޚ?[H1,+ԙzQ[/yz=B8S޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-Kzzo[6{ Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th jsѽ!?Oȩl-~PquU W1 M?tXQj;JMT@5_b1_&chъdXSj;Jm+ǁ^ZXL]goOY~KW*<c'L:U:gaՎ٫ ΢@Z!|y>d@~Lu}>mX  nْj^֬XƫXXEB_ mC{<c':V_ `o!^nmmY *q4z`i8zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^e v^;`$KBqW\vUXp=kܚZCD7%zR*qPVVmz@ަՑ N/nOv8,Z˶>#:C}N-V+ [ SaOK|`Tt |{-믪7q׆MRz cg&J:ufneW[u뻅麁^*wJ*Ћk ћl UZmW;ʯ?*y~`>?m"5MڎcԮ󓁽fڨ-mM&߄nW/+~2~Б,j{R`)@Ѳ[y+~ "Czz@@!==D"Czz@@u'NQEQEQEQTj+"Czz@@!==D"CzЦ[Úu0Lc' l=Q=}"yu{p}W0?zX>QQ;pbpb]}"z>8/` XJ>j{[m]Q?WS>0YO^ξs0@DžlUW_>c SO_ڨFm GW^WW\zt@߱ӯN_>8nR O, ]zs[j@e;6ajj^FmW |>ϻ3 :*Z+}Ǧ\:(^ds:?p`v%ٻ ?2CCV$zRQj{^ S O]goOY~KW*<c'L:U,01{E\ZܮBmw}v>Ы_-S' nْj^֬XƫXXEB_ mC{<c':V0n-[rzgmmY *z`8zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^f v^[X7 }U0o@>+.*UY 5nM!@Q"w􁛒aNm80+T+\[u~@o `6kêknȂ}l[@;;a wےaNm80뫿®Mn'` y-Z_Uo+ 6o͟-~ }Lޕ udIfq^6z=*07m8Xl@v_`/U 0}dDkR]'{iʹQ[ܛM ʯP_V"Czz@@!==D"Czz@?yu(((T\"Czz@@!==D"6m֬۸a@?yHxdDJma̫;ޛcѣgڎٷ;>K|)MR2^Wm['0ɚ~py/vfW>.xey/Su~~F6j `8 ү.ڬ䵜s+~pjwڨ(mx"|`P"ЛoR?z- t@߱SlW;?2m\toOf豦}dl]}e2a wlϥzRQj赼8kf[4`\,Fz c?Z uJmGz8 Kׁ={ƛoS@߻իJi;C=+aNmٻCXcj]Wxow?||vvg C2uBxU ={'†[dZ|״56*֭:ƳB^6CN:U,xmB<-k>^*Ыϲcႋ. Gruajz:ssTO~.o 8\WMcR^}l]]z@;om2שS/{)fW= v^ۇ{m? W&a^`U~`Ttˮ +>Ug~7'q_zcܔ uJmānomz@!ܦՑ ZnOv8,Z˶>#:C}N-V]+ TIWȋnײz_qmشykl!Kǰ;vfdXSg&?LEAnfW\*wJ VqFA!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1Zv}|n==D"Czz@@!==DO:6{p{Ra{tDX}-ꎭs@`~Y}jvyĎD8ҭ!|p6_ )t}iɻ0v}&L=~a8}` sٕA /^٪f} 08}vQ˯6+=.+y- c_=ڿ>|pF6j;Ja%'X&ƛO^]#wlj'O彌F@6c`t{-Õy cӏ.kڎR@o=8p.?u8 _|@!wlG+aN(5 F=uhk9`\tٳ7 oV>e! .^:?ܳ2VUªW-4+]!|y>@~L_m~`wb*leKzM[ncibak< ~. e?|dXSj[ł. -k9=֭-k>^(aX=.4+W]==<7M@pK-z}4=Vo+է u@߱S&z:ڼbEW-Uzn-c(tUI@o&ⲫŠ_ـ_%}N)VBµl{^7 6@/@t}|}a57d>Zq_0;vmɰ^ԶJ TzmFIWȋnײz_qmشykl!Kǰ;vfdXSg&?LEAnfW*p;%譽mG]S޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-KzV|@@!==D"Czz@@!==D:^((((Azz@@!==D"Czz@@!= haͺ GMިFm`>V_˼c\`8>+=z,Vڨ}{o81q^8>Ntk͗0j ݄l,%cuAڶuz(̟] S_'/g9khv|P‹WY @1]/mTj@++JJA^= Ow;O￿x.пr{8}̻Jmv6<J>OL(Lw͍7o՟VzE } S;?&>ٮv~*e4z'Dywpx>]z^EpeP;[]= za[?om2שS/{)V-Uzn-c(UI8뫞[zⲫŠ_ـ_jWO)Vòi >\ !M#zn?HB>0*>>`k-8/`y oK:U@ k}~P7ԫoc\b+[쵬W\6mޚ?[H1,w+Hu[?3yW2ש3&%5Bb?PUq=ƙB-[kmNJa^:6FmG}aj^Z3m`6&ooB+ԗ?a8ЫiБ,j{R`)@Ѳ[~+fzz@@!==D"Czz@@!==y<:EQEQEQEQ Czz@@!==D"Cxh40Zo 7wz@@w'.\xJ=D6m֬۸a ,/FQQ;pbpb]}"z>8/` XJ 5H۶NOs53a 5;^tͮJ}Mkߡcu~~F6j `8 ү.ڬs+j}+SׇμۨFmGi!˄txZ)kYkOO{^m.7' 3;Cs|&0v;qկS^l* ,NL lB@ikm, U,[ug/Ӆl@߱_ uJmXYaڂx-gn۞۲U\车_G \tiЇ?Օ Szzsy _8ƁOnj+ `:Bة}kaNzamK1BYз+ Z½Q諂y]~X,6ⲫŠ_ـjD7%zR*qWVmz@ަA8mUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@P &`t |{-믪7q׆MRz cg&J:ufā^V{&W*`@NIzMm(A!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1Zv뽿}[`!==D"Czz@@!==DP~EQEQEQE56B@ >>wFe@{' Oλ2pyW`T v{@,ց3y`Zzq<vwSNfئ[Úu0L@?N& l=Q=}"yu{p}W0?zX>QQ;pbpb]}"z>N`(t}iɻ0v}&L=~a8}` sٕA /^٪f} 08}vQ˯6+=.+y- c_=ڿ>|pF6j;Ja%'X&ƛO^]#wlj'O彌F@xG6ϖ}uU W1 M?tXQj;JM!b/L;6ɰ^v^mJ:?gox|B{]ZzuV)m;ge2)+䟽{jn{+ v+t]Eײcλ2u>W0;16ܲ% Y4Wn5~L}N>u2)b_W`m!^Yp MùRu?:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W8(UIO@g}eW* ׳Sq_zcܔ uJmāޮUx{^7++ S?LƉnOv8,Z˶>#:C}N-V+ BA0t |{-믪7q׆MRz cg&J:ufā^VKjv^~0Ptz=0.7m8Xl@v_`/U 0}dDkR]'{iʹQ[ܛM ʯP_VV_˼c\`8>+=z,Vڨ}{o81q^8>Ntk͗0j ݄l,%cuAڶuz(̟] S_'/g9khv|P‹WY @1]/mTj@++JJA^=BWnyQڎ҆'BX .-Sײ@05vSy/&kl:*Z+}Ǧ\:(^[ [@ŗ }Ǧ"ڎ +:O\ׁ={ƛoS@߻իJi;C=+aNmhU;on!|y}WZnz}_yTp˖,Tf0^º/Zx2]h;ɰ^Զoqpm_9co۞۲Eԇ0`:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W8(UNO}H쪰Wez6׼5BD7%zR*qvܞ !Mk F_y|pXu Yõm{}uGuv0 >x[2)z}Wz=s-em+GB^t>uW՛kæ[g _=@߱3w%z:3YA`@/ v%5Bb?P=M[*-6ЫnK¼B?0 c6FmG}aj^Z3m`6&ooB+ԗ?a@?yHuI}h٭v[=x"Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th {C8~2]}e2a wlϥzRQj:n'_#e=@߱H:&+_Pkv/]goOY~KW*<c'L:UtuwjGf+B޶{]u F%`wb*leKzM[ncibak< ~. e?|dXSj[łQ`o!^yw ܖpO/C `:zXKÇ>Ѭ\u]4ӻ[1,}ct?6UXTW.[Wׁ}N[ u k^zpς]W(UN"/tˮ +>Ug~ŭ5}1J>pS2)zjk nmZͷR{?t}|}a57d>Zq_0;vmɰ^ԶJ_aW^U =ƕ]ObeU&.ڰiB@a!w]ɰ^L8Ћz@ `@.BR޴ek`ҲmV~T)+0miJmvv 5Fm nsoR6&t+B}Yd\Wۛڨ-Kzzo[{ Czz@@!==D"Czz@@)(((jPmp"Czz@@!==D"Czz@ڴykXni#m7*Q[GO2zo> G˟է6j;jgNLNOS/gB7!Kz]mokm+ jjg&kɋw=?]9╭k#wL~jaKڨ-*JkRr;@ρ;végmTjVyeBa@onyK,5}Ǧv~:LM|]Th4 OcͳǚAׁޮW2\0;6aF(5 {C8~2b/L;6ɰ^vzn@_uٳ7 oV>e! .^:?ܳ2V-;U;fo{WmB޶ܭ v;}Y`G nْj^֬XƫXXEB_ mC{<c':Vpm-\[rz|ޝB=e-PjG .4|Uׅp}MS=/ҧ?F7{cs^_5MJzuu!wԾɰ^N6菱BgA_=(Uai7/`\v]V|,\[S"c;}dXSj[%PnWy@(з z#F_y|pXu Yõm{}uGuv0 >x[2)z}Wz=Tof0 ]!/^M\~ŵa/BؙɻaN, 0qq]s |ߞ_aJ!zӖaJ˶ j[R0 GMd)QQ;{l_u~2L0ͽIЭ ecX>O:s]moRj,Ec1ZvZoezz@@!==D"Czz@@!==y<:EQEQEQEQ Czz@@!==D"Czz@@`@6o km\P0}OL(Lw͍7o՟FOO{6λ3l31uU W1 M?tXQj;JMgG#e=@߱H:4kyG]goOY~KW*<c'L:Utu\WW˻o!ݮ߶;>;{C8~rvW|<0n;16ܲ% Y4Wn5~Lچy}N>u2)b_)W`owmmY [S\G .4|Uׅp}MS=/ҧ?F7{cs^_5MJzuu!wԾɰ^N6菱]W,ss-λp}U( KmJqW\vUXp= nMm(@Q"w􁛒aNm8ەr n諮_rOmUܐ8\kٶǁ^wQgw@郷%zR*qW]_\s!@UUmQ+[쵬W\6mޚ?[H1,+ԙzQ[uz=.k M[*-6ЫnK¼B?0 c6FmG}aj^Z3m`6&ooB+ԗ?a@?yHuI}h٭v[>`<!==D"Czz@@!==DP~EQEQEQE56B@!==Dy8״j.UB-M$.\R h F!2RQAFh4aPD&%1]MZyo;s3y݄3wjH@ z!A5$=ԐXӶ{k7ұ'{Xkɉ4{GZ?0zԽXkmĊtԓy*nJZW-YNF>c_KYۯCm1`gKJg5iU;R=V׼eg.tb3+u=M`TZqks&B>"+64sygwϞ&ayJ{DtA΁AB}Mϴ2Sz`J>ҭ;`Uy< +6}Xbb0-e>">b+'`A=#+6Xfb0-%#N/\l/za9466֜2saCiw38)m>5=G\c9ߚvGAwOv#}աܶ{Nrx_ Fb?~}|Ziw3YD}q]|ksз?I\M75u3z]zǛScz:>bΝŇy~x.WY=kU}f.m/nf,,GϻŸ"5bMw_M͸jv%&A_+oz7k;iy} A$_ FQoyzڼfطu\?QO ^WL=kq=CD}\m!vځ/;bw{~ִwx⾂A;y4ֻ C}Gz١!1SwQqFUD]M\oǺ8W60#NL4?W&Kqfei/:cL|/(Dbk7aL|3\}>Ÿ1{"=ԐPCjH@ z!A5$=ԐPCc1c1z!A5$=ԐPCjH@ z!A5$=? |O^HG?>ĚX 0hON7vc?) }֣ŚX;ls_L'Vۧ,SiwS7׺ nr2AZ~Nj8\j]U:_Is\m7wΗ*j͆-;[woYϭibM[/6q]PA_ٟ;ӽ4&ӮS#"wtu&+8ljL<ߜ~u5oIi?? NI\cgA>byw1vz 6_VJ+O+`A=#+6Xfb0ŀq ˯yX*5L?~}\PL)=AJO{ρq~>v}Sz|Š5qOǽ^F_iͨŠسhwc=?k4^5U7#+vJc.Gc9|m_;qP|sXnVKy~c x|M~-1({`<]rk XG&b>a؉*ay)5ά, E5X|̽)E叨_l<2A?yF3c5zX};A5$=ԐPCjH@ z!A5$=Ԑ<'c1c1=L?@ z!A5$=ԐPCjH@ z!A5$hmǞi~ڍtD:ɞ&ZA;~r"Oa0F>#̧nu/ab=">daJ3?ҽUEtlmRkvP[?ƙRҙbMj.@xsTqUk65mٺ ,X}4}n]Okb-0U}ǽoڜ~t@<t0\ޙ5vv}1]EgosEǵP5A_gSc3 GANJPU}}$3 M}<ֻX;L}\~+?H`A=#+6Xfb0ePU.6c_k={UkN~Fݹ4ֻXIuS?2|y> ߾|JOYq߸f_(}CۍkfTbsvY4;ɱ5/G*kNrxw#s;_Ϗ9۟/>|c`y^X.=yu1=s1ZqX揼v?k<G,Szb3z73]w#‹r]bg̀Q)qxm=/}ySZʦf\~5#~/A0 ^y4ֻXI{ w^Ľy#)ѷO|=mf3:}{lj⟨'`}f+n&vy7z~FI/_v>-bƭi֣}="+vwPiw3w' k!ڿIo;87b.*`TED=0._ĵ}{e10ADkzkbݼgVX ,M>"G/6~AvĚX =Õy0=ԐPCjH@ z!A5$=ԐPCj'RQ^IENDB`llama13b_autotrans_1to16_train_modelarts_inputs.png000066400000000000000000000517361476650141400400510ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRNsRGBgAMA a pHYsodSsIDATx^o׽~CHZ2KA%lZГ$]J!RI7Mˏ1c @1s5"R!}Ϛ53kf3 g/-Ϛ˺z?kfH@"0q#D`$ GPGm?BAA;V.#iZZ;i߽AAAPH2V1s!`AM#N  adG  J#<0AA,Gy`!  (Y0%ʹ7/ϗ8o6ҿozS`!  (Y06umHoSOӃ-rk/ySG  JcZ^ֱ1\MyҍChRot  Uq-k L6:@l֚aF31)qȬ^(+n[RS_6A\֫~5@}~J}o=$^漳y~W #AA%mA3LU^W?w! #Wsf*—o8W|W|K}w:/ofށ-~=חz8BAAPJ7g G1ld3YK\Vnmy?+Y,'_! k$W??;f_3LE`˴ABr͑k횔Vv'9T>`]82lA6tà T1~bK3ΌdeYN_l#xw%~5Q\0evr@PIn$=\sԵrMJ+\u*l*83c *Alco0`4FYo}\U=F_ }c:b m~i^ffs#@)>kWA}4:h273Q0NV8:VW,!8;6(2A5N,@0{ŵxP>`S~Q8nJC %ϟ9xp!(y rg-?Hr^=^>TĂ( AY\u6 Ʊ>x7rqu3oGU,4:ԖJ8.o{ꈏ<.TĂ( AY\u6ќ!t͚83̤Q_ KSI#E(p6[ȿp9E:P 0Lgq*9(ؔ_Xg+G:P=3<ǦQ;l/<~O}kK9h7djtOSjX4`2(>kWA}4jĐg}\y\32̠2omcS͡<~jt3eLJ|Nszr#nF+4rw`JStjx:i6uEj붬 dP}׮ArMiEqoS֍K1^1>>=HBM˴rk/݁i|lƉU.Pem[!]K'ol+A'ǓoVŨuOF{uOq@V>Y0ھlsJҬn{4N%4:jdrUv U^(KCۖ*IcM[1 ܚ:6.qZUm2]B꺖p|h|Ukmo@UNlJ/`Cz?}u3cEFSu\$T*_h/huo+Zia*<}CYqasNdԁ7@c84Nܬ=S^Z6}5wS@~e2FD3=7JwWҢb\\1!Pg&}[n:j/^4vr]DӬsiOW־ t>E[(o[G8}ﻥ7rN=Δ{-U8&f/cMC,j1tơ}!ud,#yZ_e8C14Y|H)sm09-_y60Ɖ%z{{JmBzbf=1o!fju{^v6=?-M-Qq=Bz/ cc?Ӗ_f?=X:^Xg?zTܜz^㪧9wx<ا8 iݾ}ޡ/s5fC)쟽BeK}yM>˯ߍ7__INfs~\hHg9GMԌU@ zuw:;k S"7߀{DΗJ$U$qŴcK]O )M3?Ħ90@ǵUi+-2l9[6ki9KsF^ehNw|/-{*|xN=Ɣ{-ڕ􏀑Z!}/ϣwdJ0)DU$l+$>Ɍ`Bɟ6Ó2~8Pn)Mq5M5N܁ϼ ,IVtQ _q֍~C#>4s-ko=85ͪԭe f9z|;3yǘ}IӰi3ws酭#lx]:I7_+җNf_~ǯPޱYQIi7uR{f4nڱb^վ;N4QΝaϙOy{sVA2Y.bL0mz둁i6N S *UmrMi#i*}b]>x(Yu}e7/-qK!`p8FQAԧXHsKxTULo8{c7-@/gdK_cGBƱ6/(l@2Rrs a֯]ef>\}MЮJ5´մm2s5kUF-S 6MiS|ZIOnuM:o;V|pE=Gi?reC^;i3MW|Zq7 XGq6Ae)>[ʵ0O6Y*΃ٝEAB00i`5xF/'hM97P 澂8oeU tRsCfj9G[?Xe;Y1/W\QdqkcT~By}0@ordŴ|1xNer_ ii?C69۔1Qs?לS[9G%uNj}l#FcZ'&r`rHhc GPTZi61v2h]BZe ǵ/ƱmC=x&&,8&;P:z/I(,%=ٗ_Q Hq˂1U=0]_v#7 =6 f=˦Krm wW'#uP.GPmfo'-{^!4yv,q͓eb;~XG :E¾4Gl#$m>~z\G]@!PWGW=#hIwvT&|:8_]L+?TyFn#U SYl׮Ԉ ^rm+>:.|{pG&\]';,">m;)^'X/[u>e ]._P0mܬMiڲ7/y$Ls8Gp̥6Te]F1V;Uuô/0A;]c:' ;7^hrJ&$3fxk:g6OҾ4ܚ^>ng~}i!*h^:ju5yްֿV:+}4MGٌNӳ Ƀz1n HexV\GcćϩP}kWiCi`Q}i7'NVMd]ii1dνǃ-miA*y/fzJ1t msdwUrMi#iqbUBFsі~>YC_^9Ojظ2OΥ'~"ۺT}z_ Nsvfz nZ._O嫪*/y:Ҽyhӥ8f.cP 'i~_Lēq˂fMu^_d F d&5pl Xq4Y4łeN'^;cXqٝbB ۏѴ6-(y ΃WJ;Ń+>G:k.o8MAQdn?~Q לAn[HwHku*m}"\8QxKY'yy&異zZfTV(>[ܵ+%߼;jV<$~ޗs{nۯr 'x#7A `͡Η\ lJ/`AMS ;e_0BR8nYwU_ʬOSݢ-70`#4k6gGjϸ+ꫪ4c&AH|}`=Z\GǸX2f7S^kװvB۹]lfsO rn9HMfq~fYJŏW?BΏoA6`V/̧E۹Ncɨ@,J|(/ c=N:W33%yq D#:8IGS4O1_ێ;s*+{-UAsm?kU}z#}Xm7W gY}Gu*:_=BhKQ 3x#zȽlJ/`AMSqЧyV is;uOyf5}Q,S1! 0iY>ʽsJi< Y ͌NNASm"l u4n;DxmbmOQƷ?[6`)Mq5M5N,*f ~LM&Ä`S~jjX4 sW|>rQ5)p*9D dG(+:w񲺑H zk\{lפµ8ΡJ$B8FNAAA#Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG  J#<0AA,GyZZ;i  K2f.#iד_  rʅq5M,2#8@20 8 #<0$2#8@20 8 #<0$2#8X;\A=#<0 f `&0 80A#d q 8X;8BU_0dGy`kG l2cAT}8M`Aq`!/G #<0#AP@6qƱvq `&0 8qA۷M] QtκU+^JwoS[u49*M`&0 8Ec=hV@C ݵl+Eien=ZC5B'm[kFS]{-OrZ4a;|=Gw6ݛ]%"ۇ.CZJW瘡#dڡ(x`κ&n_ m(qR̅qz*ݞ^Wv/T07|c雋$]mk ^7h̲OeTbJ>Dc`2c8{;h'ԺP7OQ[):wZqzܠ]Բqs qN0cW7;KsNtqW[xփgfllq9ekm?J{tjkuj`=pӗxec4`3KT*w7x6/Sk0㉊9רuZR涾k4>e_ ǵqҮ7q]8췋.pw]q쵯3]38FWO*~;>NJ|G6֦b2tznS߰ؠs[YM2ogNY{ !:Y^=_p>$3^|N#/pQw'Mo>[St,iWyi~l>Z-|玻qI_Pmyua^s1]lZn!$)FlUvkm P$i0Isǃfe`(WiL dݻ)GCyeM(& 5p8J-3tK̐gm <6qII6~mDJ=l6kM_|m^#TGΰ1mȚ2,#w9Sl&mm3:lהּk2!3ͤ2u(C-& l2OqhRZeڱl5u4mzXKi!G8ނyӦo^]2F٤ܩ4T"m_DFcӢoγScHFjXƾ2H~FB͈S×y9qEȚş _4#ԟ`Sc;.u!aIȿ}Ͱ 3I[Tr~M}CA]{Y"3R}~p1O1{nRimn[D#E'w|I<I|=Ef\Lߧ8c-4AG1xTf4h"<&rMle8;wӞw3/]L6nLǤ4Ã}A<ÝvV?):/C||Tu,L9ϟ*s`qNO9hI#־ְ|i\rgy -v3:S@ǁlPa*eƱPߍɟ}kUo?VqttRK(u~;zO*hfRlqۍ/ۿX8M`A)8ʣ,u&/<;g /d1̜˒g$4hf~*FҲQpWQAn84PtEg =*cﳙAO;!N:>#ä4_Gx߱r]c51jJowq;~V{b(_Idp߅s>lɣ{ꝴ;q#ױ}agpm ZFȔ~|P{c;}/PQ>蔓;wLwK10 ^(ć'Lu8ݤI|ggq ySUm"}(F~,s{;ϦO}4ctJ>,%W/\/e9ׂeNCf9dnk'J=9w=7v*uwy[\h#[?0#kk+:m.qf)%_,sR2ˑTkYFi1R J!픗G2g1۠`&0 kIæ7&13q1ՠob5<,ؖ)ܫ:vc)8B/1j d^L3J>skj\eTM]i8M`AIo]3ċq͜ո9rfˌ |LQTq¡YGUJ+Ȕ2CQʇte);OFlU2j2CT%Huz&kZRy}zqr :9wRE l2O3iF*ll_P o7gM8`5q3B?Όc: '/atMQU}Č#A%1Ol۠$8M`A)8x&&Ic T eĈ C8ݸCǥ}TՏ׌,+܆9d9u1<0A#dgbq 21M5D}ZL6X OЬ9xY0hj[Q\,Ȍ;mq `&0 g f1Tz1S&Ѳ5QC3 -ά-͸ywc۬洂8BU_0dGyJqSG l2cAT}8M`Aq`!/G #<0 f  =`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`Hd9   JV!`$ G8H@"0q#D`$ GPGm?BAA;V.#iZZ;i߽AAAPH2V1s!`AM#N  adG  J#<0AA,Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG(+:w2m?<AУ\sc&] uU$q4rN5ɍd@kXIikWqA%M0 8BYQ >TrﳸvlJ/`AMS 0Lgq*9(ؔ_8'MEa ʽU`S~jjX4`2(>kWA}448 h* dP}׮ArMi#iqbATɠ,]Ń: GPT*_9Z=sn+QߡߡmI*8N6H `U_@`hRZMPKiT=k^I wN|6*}DD9*{ʵkp'Rm|6:u6m:6Ѵ5y+@ϭir8F-O4:R׵FCF8^%n{~}>`S~jjX+G/OqXMf6'شB [i;.`S~jJXJemx\6\,u?k՟hxǍLs~gT‡L?[K_r9C{^ͧu|(,g}QM%Fmh z/׽HMGquaFeex*o=,\K;oJB/hEcV'#FOx53*e 790f:un9l]͕kiiY9ml٬U95`nڜ6mM i'=561r,\Q-~j&n߲ R%u{/E4Q^G0i4c$ }.F^D@ } J\ؐhu4A,ɶg،uܩkuzj%,6YW'@j 1gvW+4H*&Y1ʛ γۍryܶ)Dq|po,yN=>%6AϘe~T{-Ug6}l2`[s)>d LLx?XW,ֹc0\qg$/՘QҒ󋗽qH:_qwA}44>r l0:7vK_qL.6gJs im%q~i1|b!+4m#84FXI|tlBcci)z&{̧=(*~ 7ޠ385/6qM zޙr nz0fUC9 <0#h|aP?ΠG(>[+ ޑA_y0 }H_iWO"2Lb{iAQql3g=%_F|1p[UOL{W)ؔ_8'V;>(wh:ïT=q(fO1m_~lki6Sz:uǹD/kwcxf {_c.K qRVGYn~W1d?n61oh-͎{18/mUq12$[$\:zݯ닌AT`?-}5͛w7_9ּY'Χ5{he;#q[c66yٝbrsl,K 62O-w$6Rc>x0@[Uo5Rtoyf'm M[zqqueu>Pbk aQӱGlP^r} ~dȭoغ9'U9e~T{-ڕ@(1%UA=<m@٭Aqwr_|<^+A|UrMi#iqb;gB+y>L[ܯU;ԶEiݏ]=9ыn^ZXO9>s߮'G|UuΓu멿1^}iIvr9L9w%,qryp]UxGg2ONӡI=HNX$xy0HxYOu/zzdffWU7sERX}UffɗR_r5`WUg #kv]E[]>YV#'XJ.N'm M[qh!frlvMY^!>Tk嚔VvT2#xYHC= 5G=kRZUPP%Mw!`AM#'   #<0AA,Gy`!  (Y0 8BAAP`Aq  d8AAAɂq  2#AA% dG  J#<-4|}z@AAP%ce34/AAAAcB8&I@qd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq ߂ dA3AUQ0dGy`kG l2cAT}8M`Aq`!/G #<0#AP@6qƱvq `&0 80A#d q 8X;8BU_0dGy`kbyjn+U;y$z\TCG/nz\M8OUo3qnڶ)G #<0CQƱs `E5H+:HS:pڋo~5tKԕԚ ۶HtϺJ_K?F<ݴmC).f٧"*1_U%y8dƱԱ!]؄R_P<5գ]mnPo{ƶ2qJm>4NW/^Wq0 8{46pvR+%wCVuӱuٶ~Rh@xZX]=?qtq.{i\+״ yj'm}ǩrɶtGW:=6j3QAN.=p( .']38FWOPe' g`;1pZx%-.Ҙs<?HG\S]h9-_`J=<083oa(sb;eS@}޿Ljlukx$ߜWctbT%/x܄cs%m#K-ˠJs  k1vo@!Ž1w[\=UhwEK?/{@ݒq:'aUN?sz A!L3h9B3{^a'&t>[cDrWa)@dMQ8/o4d(+1v6{3Ž$  櫓bbtyǙ̧Yi4y"<6  ^yf5Q̌gra11yò͞_h㞽gKG?&cnҟOy3r 2M?*x%z.$=_'W|8ׂ}#}7ϐ,(qx@6qh71Z#FT9f3O"'Iٿ80bU^O! anM8ո p(8(Ի\ί7?Y/s`6UzLH.}p6VaEכyySit\Љ[֔I We[8ff}=9@_'/|\|Xy ШKoTfONVxYW][vG7v9^)ӯnk9C!3ns{̇[_w=i[Gr kE:#I}(e-8Η.|nˣ^=NzRGn}EɖDžI#dj8ڍcXbtQѠtn5 :8p E`փΌ{՛q ن Kj;c8/og)JÑ/H@Njf)L'٦×uى;w9XQUn>"c|1U0 Wd4#(CcoGRFg>aO/Q8&Kpl=;IPR=wjt\ԗ69wԬdR\8d|O=F3;N\ϓ_Iu-oq;yAKC~'sX7=FB/w6;/k~FE87my[ >ݣ{%Pw)%;qq]uYIݾ4DWG]8M`A)zQ̐,lc<3pǀ1yeQl&l0 HINRɦ8b]:xf,ڏz#|{\]>t>뢘ʝg]8M`A)8MoMbf; bcA#jntIkmyX-)S]IPm.5*U8>=:D #dbgӸT<t >L93o:#qXJkf@5~puX+N^<`0xql?)GWP8M`A)8x&&Ic T eĈ C8ݸCǥ}TՏ׌,+܆9d9u1<0A#dgbq 21M5D}ZL6X OЬ9xY0hj[Q\,Ȍ;mq `&0 g f1Tz1S&Ѳ5QC3 -ά-͸ywc۬洂8BU_0dGyJqSG l2cAT}8M`Aq`!/G #<0 f  =`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`HdGqd`Aq Gy`Hd9   JV!`$ G8H@D?/y[UIENDB`llama13b_autotrans_1to16_train_modelarts_log1.png000066400000000000000000005561631476650141400373750ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRsRGBgAMA a pHYsodIDATx^ϫ%GOp߷oK}oMf$h@<3"V_}g>#¿WzW? c>5b <>O7|+owqt?:SL=v8O|{G?}8im'jg~msy/ºگ=cmsvk Z msUӳG㬂~o5tӯԼzirϦWf?>ٯQ8a9_ulA=p!6q/BOX(.k1W/je{BkX .f~?>2xE 2ӻlp]DqUߵ;*x""D= _R]Ԛ_QsӳmƳ Nc7~'h;Lԥ>Ŷr7l,cl)&>5pfAߊ>}e7||# yoWXf6N }g0A_^}kg֗@Ee?$( `f2Z^WId?/>佳 %,XY(Fx^`{Vp$#A/]З5L?ga|TjḢ,gў$/ꬻ3;(8l~iFg{3y̬.cV#]c[/srbA?Jd]u ZO-BwoL|y~]>ܶ>| Q{~}{5s{)R~zp:ا8OrFa*A9"^ֵ"VO+~+D3+q f3Ue Ao{B^@\ιȬ}&B/}d֜oח_Ϟ[q0 wqwsܙ;CؖojQ '焂~|#yk Q Qha'|W\･0($\B<ƌm尷s}A{^ B9,p)H pչ֊vZ:o>޳QcoW7y]Ϗ޷*+~S/v> =z>tz=\kH7/5ao+l1~tb ʷڧB]Z?*$GŻr_ s"A?p˛˵(BOX>vXI=4n}Sl; c}3ʸ6k쑽.臿˳͂>CHz#8'V¡ΏaNwjsx(&Vٵ|Ww.^&!bϧ`F7x?FO/?O_6@+[1׎)7|C]Ouq}vϞ 1+lm^8 QkUqA*u 9"W#my$hߢVKD=_H='0j^~㬄^P=cȢ/|淜;bf}h\ُiPSWϏ/G^gF96T:G}!F-{ƭpHD<ݿpG ";߿_OA/x{\G IĵTbk=G a]{}IԏX70[(aDevm+'Nؼ zu M^mԽv 1_P (s^w?݈b 挄,g_[э[d¾}-ozv^ |%ۿ~F& QԷv/F9 U4<"<15)t g#vKn9gx{C{>ثЂo_wq7E`o]-Z~~`\-ys,?:>#nXebķ,_ᨾ~8=PV9=wF}GGoEV/^~NA9}p|7G"^3Ou¡5UV?׹.m!Ayz7yx^[~x~tԌ@|f7ϟ3u<~+uq/<.O~+hQ;ao%@"'ή5_܊yHԂGPA.*?ٷ`XD^>ھ7^ ޞ5SC"{5GeTasW].?۾rw[o?pzos>RO}Y{c3Db{~{Ѯz#(g!?#_랠ZS67j\p<~A[N kD~_bE"P' =A? ZhxUxC>+9Fz k)PyyȽ<9imdV[?q.R/c#{RGQ}-:w{KsoO>q7+ԕ`oӑb{c|3|=hgl3vÊ݈Ձת<%[+R5fR_="s?} A_ľ.sg(QE}SV?-d<+ܦ9'ɍߋXm\"}#[Q3Zcp63TKF/a~kq>tTbۯYؿUMz /bmz_ \GGu EDurY拨ϏI$@Eǂ~K_-"9zo=okkcT\{pZA=lwin۟=}@3 V EhX+[EeoWX?X +}g0A_WkG=rޗ=aOCD JB8 ?YBϼI_oET~Pr,qB^{;D“3?>2}YӾj=\gy(}5xxFg3r6oz(y[5s~-q %'!ߪAd۷>DD߼A{~q3A@VD] t 킾IC#JoGys6A s4rssx.u9{]w `KdR=m9$ii2~GZ>l8^%."oTXO׭\"nQ|B3ܛY-tO̷07<`S^ iϒg/C5(O ( pȨyP3?] W|-"O+~+D3+a+w0_'諿/$Y5EV}jܳ_7BP4ٔuZ[s s-X^uŶJ6=6oc?/"9 *X=jH|ckO%u|6@A_&Ǻ1a39SHs eK}j9Hgu}n~7{3>wяq?K_*?4gs9yA-g儂~ZڈQl Qha' Ʀ+RO[+wF.!ocƶ%OkǶ+菮:$ak@ԛF룄W/s]N״Rtm뛨zg+\*ʌmʊro}qAa޻9\l +g ˾@[p`79L^罾OP kV~yH@~rۃ~f$B8 YF,"ءT~Cc0o/OTC1Xb>74[z+αblc2') "h7$j~41W"gdt}# :wG}3n9r HЏb#\;bs9vm XWsov1MBujw~ `A أD8OlsF#H55G< •[]leXXA_D.臿˳͂>C=Eeg\gPĆ㜂hQwd+ƹGջv?-c*n^^D~B?O#kF"ybU|̊tV9`sb=4_eAnF }?cAMPC_EciPҙ֫G\%/}1כkz[X?Ӥ9J=Ooo'8yqA?]AيVH#vA`V7>a{yf{k߆ӳ1[~Y7?^+ɦEcs%'+OYBQa0/Bn,ڷh'3+A?y9*觿_o8+TB?U=biFus>(.>L'DĎf1EnQs^VvK_-qʘq-'R -ƇԺjNw$Q?bhGJeP;vSl݈ۂ{m'lk-uڶ>/k/! JS"\"5|:o~1#g1+4{+|,Nط/AZ]?(W_џǷ B݋zaM[}H\o\-o8xCVY`ΦWW\oz7%Ͻ_;CXr [l^n|<p> 97p/+k.-a+W}, ĮCD!~?`6QbUpݵo;5{P~x 5輍,C.`lf!nbی';ύ9ql@/'\iROl+_ϵ>31/~z3\#8A'\+k" $YynzlBЯR=ϣľ("y8|~/mZ\?g]c-ǽ_)5>> (NJ7 p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:{霜UG0Χ~ՋOwH~62ky~5vAN,9^ :]!8g ?>,sY8w%p>Ãy {]jq >$Vr/NX(n/n\1U?[>Z_v1H +r\yUe[D2w Trְf{g\+gg'7WqǟO A=(g!?#_랠ZS67jeb E;x_]v;Yl_%G Ft,~)ЯoPWNGt^op ? +w#V_BÖBn ~H[I}q藛o{t>`j,9y+z>=QIzWֽy~zh^}}G~pڋ91+s6iE`g>{"g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\ͽk=#ЋkFr!z{Cȁw؞Y| 1?򼸟-3Rٖoƙ}+چ"4wD{_M`-m"Hϲ+,n3s_/﫵 #I9o˞vA' !1"Y(7kp suIC;A˱9> p쵈1y78#A/]З5L?_RB\{&3C?tG{b˾YWWz{Hvb%xnJ^zq7tFɏ={w|q;>~)Fg;JsKo^u;c?]zx}kms9{ 9'oJ BΔouNgۺ?Lk#z&^Z\aoE^bZ*~}[?2g/ AUB"&A5p݊E/([9ӼuApBG|\| #a{à =a ,ybK9T΍ O񮵂@okMFN j?5|e=a뢌 ܇=A_|!Zey XQ(.>ybG_sl\=a+\済H\ BQeڮ^,`z欲1ݓHsFzz\-Cb;ڑ7[2?{ 9m4tcoOv3m=@tT wK<rH|D%^xB+4׻n{qkkVDu2wr7yY?\VBNc{ܳ1S3eGƫ>{!{5ٳg6mymݳ1:>tl5~tmo"hδ~P]q<`m4v[{B{^f5lz$s7[}Z+37CriOz''N  ve+[!Վ)Ywcߠtb~O uq}vOne~xh?':VMtuB\qi^ߔPViG4܀7 7' (V ~|9 0j^~lRϒp{j(N5")1Iͷ}Zy~zڜY+?~/%^yS cYk٭sؿhW|9ccL17o|+6!]t}VUc0=yWz䞽m'Q,իώ1c 覆O)ga펺ڛow}Qr`sbc"~fp$G IĵTk=Ga]{}IԏXzR-"۲{7`^ CZ ao!Dϼ7KN7!7xc\iXTp-FnH7H4- ?DJ?n} zX}/{xnb 6^`m2>kq=uވ8{TJ#KTroXFzs7NޞxK 8^'"mEᝏp9G<JXAz@rWC~1.DO ٷ/~CIx\_ȀAx|-,櫟􃠭szKTܭ]$|#_~c? \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpcMTn_u_'{l#\妎^:'gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]q4I-:;|H hA_k%X Q^$.bA5|6bG#;/|W> 8˶eAaθ*Vjm[ϔgOoz9U?@zQxB~FWT=A;`m&qoԜ ńv"66y#T/>wؾJS<>Gx8 J7Hks]n1dg3caEn@kUwzWȭ ?/ 9j⾄/7;FM߮oErM:sX $ASk/dEmSŇl{㷍B@lyw?ؾ?|LtE(Ȕsz8bR賲~w꣛=W}eݛwuNj=}-=go)V vW/[)q;j~\vg[cs<x~)c<qb\t{%7Ǟm˺9N{^[ǣrB/]6tMɵ!=ܻc|>f$OYǭl/ܼN~؍O=b*~yW[gm3-3 V EhX+[EeoWX?X +}g0A_WkG=rޗ=aOC.bDP`ov{ 榻9 AbŇwE4csd}|bkc|o\uqF/^&/kڷ~CnzujM--f~L6pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴ]o磝sRsRT 'Ă~zIP c=;\"r "tG9 Vt4opof]?ܶs>W0HaO{u؂+=KCռs%^xw<@okMFNj?5|e2p޶}7$/k恀G.>yaȥ.=A(l=\X: l*a=4gZ׮2k-yc>.C^\9fOC:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|2xښ[NN D~s\ olkE~+$0nŇ} >@tT wK<rH|D%^xB+4:7٭@xx"nuͳqccm ȱNPFNκ&/6 7J;"UWil{6fx(xgz5ڹ=?zF֐9:{lUƾ-!;{6FgԇƏm(_k=}moǃFc+efPѾ^G1^M~mg\/9ss`!N5OǺNN6%A` VBS ǾA?@6\6잞 Z^N6u.B/S'HsӼ)Ҏ i~o/n,7OPT":Ar6aVA?z}s^U-bZPZUak+EIo Dmb\h=ybi{y8cV9x9L#Á=A=b>\\]oW_P湾= [LYW?7o{*A[S9'[IG|9c? \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpʹW鳏{g?ն9a] lw19[ed;ǵ~m-3H{1Wc;Y}|:?j^4|~9~]g+ߟMWc"StI%Rs*c2?s~W\{*80.̋w7-x k!5ۋEL>φ](R2}deׇA`^wѺ];ս5WJ-{r MB:G'H~QxB~FWT=A;`m&qoԜ 8h'b3o=A î~G?Xap&Šy̩_fsDz_Y]y]b{O5i/(lK[ʧA#-{WceS vf$OYǭl/ܼN~؍Oε^19_Z[>5YΕ`,[6|#bmBooA:~]a`wx<m~A_~_Hqx{_ r<6?Lp fIB\!( Ys.Dߊ?/>佳}-"zYor?U\g$:}e }? YrzB\{B?tG{'Bƿo_U3^c9RX+^ QcϞm<_?Oe_νܒzXl19`~Zya^8BIۨ/Â遐3[ٶɯ>W[}~Ǻvcs8:!ŭc3c}㵲; AUB"&A5p݊E/([9ӼuApBG|\| #a{à =a ,ybK9T΍N7 wOqE{^K7񧚟97C:-ԼMna,Ue Ao{B^@HJq@Խ3P8έϨ^Kƫ ֈ4;^-zU~=&.(yebG_sl\=a+\8\jSRwG4}pmWš^/xY0esV^įIg9zv=Ɩ_kurUDJI6{:Ա7'zMrr}p>xIUKC=doo=׵Oenٲw˫K~q5߱UO>㓡ןC~co/q"A?pa7۵}`]͍"y{k7 þm ouG߁%Ew9$P7x !8ƹ#[ĴR8ë֠WhD85MO}9gkq}#V~^Ke`sb2uv%F=UlS͛9J li׾9-mijULl>gkɃf' z|ظ¸SYX{Hܼ-ҹisk74k~ZOpp`G IĵTk=Ga]{}IԏXzR-"۲{7`^ CZ ao!Dϼ7KѭD"ě-rS.̍jt#7ҭ@4tW&XzEE+?DJ?5kp*"rx+r6giWC~1.DO ٷ/~CI}{ >A'~1>T&9G*g۟xķ\Xp~}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L?%N!ߦ_}i|0}Ɵ~^5ѳ2ONfZ+|Zy鳟ju~Ϙ|ۜ2Bֶ}{$Ա9`O>Gͫ~/ۯ?>lzej삜Xdrʽ9n18 Cq^E}X>TqK~]ƥb?6'n#9ZZ A,Bja7 +}P- a/@Qd9*J-uw{9kX3Z[33+8՚}9 $H?(g!?#_랠ZS67jeAbxPDAz}aW#Ur0z#K>+ <>xC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7 !~nv憹ܠ}E=7M~Fzɮ_bǃ_w%4;)^n9oea` x6;~l>p&JmJ9u=zbRY;>+C>jk^̉Q\ٖɷO+;K٫:^廧yN[r=۝Yl Fl]h>l6'-sMhz$핸{-8ym =t *JV{nbl51G6&׆=|zqH.dV[?r^yV^ocr|kZ0̂mg;/&D0xqgxwXx(83ѯ~a'﫵?0푔YuH7EH"@ AY8bFn{MbFoElپX=d7}`k{*3?>2}YӾjp, 9=!pǽZF:M=u_q7/ޙJ{s׎WMK/(g6}s/zǧ_}/l^ank|g,G˘[OctmT׬~cD~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2nt+B\qy~zڜY+?~/%^ySֲqP9 1|:u *_o6}zl)͜o%T6k_Ζor٪jv 6Q[ܳ׵A3꓅szuq>fl\a\]V),=n~HͭЬ-k?? -'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>iP7Qx;eYn7…q^nF4tW&XzEjmi"{7o~y|/DQڽNx[/s'}55NF S1%UUtpvy*z,ŵ^>n/X[o`cȾk3XNٲn'翱߳=VjlA3]lǜ9^g>n} zX}/{xnb 6^`m2>kq=uވ8[lEZ?'y5w{Ğ߭~u &q7NJxRol~h=X;蘇3Vt[ ׵iu8|O}p!zbpu;|!BmLN/,kp&>Ŵ?|S~u<508?/sniG|9̵A AB}+A ABwӛ?᷻w]LwO8e<~{oy0yyͮzQ]=39?y WNO_m>՗vWjU=+c~ol—_L}ޛ>YojN`ɷ*# ?-kmkٷGߋMۨ FtӯԼzirϦWf?>ٯ.ȉE!+K?Uއe~>CUpa\>;"q?|wтJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO FD= _R]Ԛ_Qs.B{ㄢͼ%z^.A,c+ϥϭ/5~A9 J7bHLk. 2z@3ݰ"|5~C[ R5fR_="r6?݉H 7oq}=minM7Qdc+&Z! -zN~ݕXnӜj(>dە}5m㺘/M?/A7EzN]G( }V/N}t{J{.м.`xsbWl-ӊw}βW)q;j~\vg[cs<x~)c<qb\t{%7Ǟm˺9N{^[ǣrB/]6tMɵ!=y =>^\3 ٧֏s}n^'?BƧW09_Z[>5s`gh>}6!qWķŋ?ޮ~;NJCAƙ~ 2}YӾjp, 9=!pǽZzK:M=׍e俬zg+=Ds];^W<7%/:gǞ=[y; 8~~u{%%7:.cn=r<>ҵ9޽^Qq}ʷQ_aٿ!gʷ:m&˵eg_[[/g}c-Z[u/1`SBlFN,WNc=;\"rⅡp^Q(>G!ߊsc[8ܶ>| `z\=a ,yc ==q)A?xW'L#x({y#XFNj?5|v/~{3~YƂ~FDe<pvt(:=ҥgF{*P uL)5>:s3**/C5b'ΡWK^p;__23sG13=Gej^}︅VM;ks~ڜ4yL{ٿz:߉mkam֯˯K +{U9ɽU䰚44YCbnﶶ<~N({kۊQ@ Qha' Ʀ+RO[+wFr6fl@A_{lkZ3/󎜅7Սn% x+4Bq ;йQzg+Ɗ&VLo}mG f} '+~S/v>aȥ.=%uwLGmvZbG[6g0W~F3Zkjlю~Z!~/WUNTDiSH{xi^Gk$Gmn~)#GߛD_k4ÉAMs]XƜ-{}/IkW[=>T3읃>aM~ث9\Gv0z;;noeg =clDgaCqluG߁%a?‡JWi}<;~uͳqccm ;oed olC Tԫɿ#RuJ;gc&@USvnk=5duΞ=[յow@ȣ?ock{[=HƗAZw}_[mm yd靈׳=@́Wl[luk-0"K\o%tzp&dž w[q`?'Xob;>bf5}~=mǹm==cy㵿CAV7=!G Uov!Fz_ >nhߢ&ΛAr6aVA?z}s^U-bZPZZa+E4R"o&a>y~zڜY+?~/%^ySֲqP9 1|:u *_o6}zl)͜o%T6k_Ζor٪jv 6Q[ܳ׵A3꓅szuq>fl\a\]V),=nkM[Y[֪}6s8ZO"\9Z̍u՜$Q?bdnw 8[voMs+'Nؼ ZVތ=Qm{!03-}AӠnt+!&w~0To sZ?ȍt+x"M&xoFI7of~%rDaB=-Qd??o([orn$&f{ةd2D߀NZ?SPZ'},s גvwܩ5R#9[7{gQ49hr}&ؽ혳b30k'@ӭaC eqo;{=MLFޫl]q]Zc-vwH^۞=OnXv78 8)ᡂKAApND(BGfb휣cXEDCo%^NX\צL= p~jo#5@o19 yd@8?~ϋ<އ A\j9Rq?4|#~r}p \!W>p \!ӻĻ.of2Lw=L؉ܼ~<cMTn_u_'{l#\妎mgY}|:?j^4|~9~]g+ߟMWc"StI%Rs*c2?s~W\{*80.̋ w7-x k!5ۋEĕ>φ](R2}deׇA`^wѺ];ս5WJ-{r MBj;n@޳A/uOqN)XI52sxC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7;M1߮o57Ex}E=H7Z YlsxrT{G!ۮ^n9o@lyw?ؾ?|Lt(ڔsz8BQ賲~w꣛=}W}eݛwuNj=}-=go)V vWϏuwO3Qַ6?z;2ŻK}وmN[H+q%9l[qj:Ez~TؠkblL Y{⚑\>e~䞳s:b7>59ů5o]=6pfAߊk"w~K|[  c;px<m^A0ZRHqx{_:|}"F$ V\p,d1#|V7F}+bCƊ!sd}I ^ˍ<Wq]I.˚moP/gY s=՚[Z hrmƶn=.&e];s_5v!#u񊕸)y9%?ڦyN#\T+y<-9yיevs vͱ+Tr+=,9S9m0_l/^>z93>kqw{i['ŭc3c}㵒Sw6`#'ЫE oo[P8(BwoE9 -n[H[s>W0Bv`=.˞ZWO{<ױPŞn2Mye/4o{FNj?5|v/~{3~YƂ~FDe<pvt(:=ҥgF{*P uL)5>:s3**/C5b'ΡWK^p;__23sG13=Gej^}︅VM;ks~ڜ4yL{ٿz:߉mkam֯˯K +{U9ɽU䰚4gmcvk~sBA?[VtvߴX"m5G >I56]qzz޲_{7&WϷ1c c[➹_})w,.77-F<e!A8oWxܨwE޳QcoW&澶CԾA?t);Zc [0ǵPģ 8: l*a=4gZ׮2k-yc>.C^\9fOC:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|2xɈ~cD~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2nt+B\qYH}?׫WcƵUo%RC6Isҹis\[*v--'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>iP7Qx;eYn7…q^nF<&n7 #ѤKϷr39Zh瞖(W_џǷ BKP䏷R97}w][=[o2ES"\[uLo@{ g'rQ\bmվU 9ky;nUTޚ^-yw=3kf4[>vYs5}z ǰ&`#Վ ..-WShϻP$msƐk=[Vl2'xCऄ .=[:z;ᚉsy8cA9x9 bq]F\34AЇ!a'W[A!$‚=gA [LYW?eo{*A[SsNJKTϾ? _o?vA \!W>p \!W>nz0=Ƿr{?=v7='o7vbz=y 77ϟcpʹW鳏{g?ն9a] lw19[ed;ǵ~m-3H{1Wc٣sVAh;N}W/> "_N_!}ʬg=9>{sr?xcpv 5g} N#K8YnO7Gr/NX(n/n\W?[>Z_v1H +r\yUe[D2w Trְf{g\+gg'7Wq5r'H~QxB~FWT=A;`m&qoԜ 8h'b3o=A î~G?XaxC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB 7;M1߮oE"x"[g ؛bw#=dWk/dEmSŇl{m\"}# +Q)9pĤgeRG7{'Hʺ7R{ݯN{1'Fqe[z&R>P,qgyS vf$OYǭl/ܼN~؍O/G?ǘ/~-,{{Jgh>}6!qWķŋ?ޮ~;NJCAƙ~ Ím}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/XzulfwV`#ߜ9^%."ox{;p݊EyE|+:gdaoЏopB):qYԺW-~ڳ乎-/ĥ'LsS^K77C=>ӢO+~+潱bA_^_'諿/$Y5?NtG >H:{>:>SzJN.x?doIsҩW%);{WChⲌ z lkYvWg;nU?ڜ6'M6m/4wk[;znG۲#w>^UNr/_9f3M>;nnk;焂~ZiEj}klz-Ae"|lo$M !ocƶ%OkǶ+=s-R>YP|\V7=ey+7;7]CleXĪmm&o+@"n>bG_sl\=a+\8\jSRwG4}pmWš^/xY0esVsIg9zv=Ɩ_ke_rUDJI6{:Ա7'zMrr}p>xIUKC=doo=׵Oenٲw˫K~q5߱UO>a?+?`{U`?'׎Vog'u>,[!8}v,>{h`8.M;p{>LGu[`Winėk΍t~um$6k72wr7yY?VXc{ܳ1S3eGƫ>k>ܞBz\#[k?[O={kcߖGf=?CVGzh/ʵ؍k\23hMgg{#]&ٶ3Z\a|EЗJY{Ank;~XPlG;o}:v1'8׾ gclB?o~_s]=ꆹ'HsqnWoΟMyHO~>nhߢ&ΛAr6aVA?z}s^U-bZPZUak+E4R"o&a>y~zڜY+?~/%^ySֲqsOrb2uv%F=UlS͛9J li׾9-mijULl>gkɃf' z|ظ¸SYX{H5Y(6{C淬Ul6pD\K%sP9g;H~ BApnm/V\O쵝y?Я֭{^B`gZ̯lۃAV7؋or72M榼q^nFnwĤtW f[d_-QksOK٫Dxw}!%(uz>ﻮv7):7뽇T9c(zv~16j*C]y7*w*oMHΖu<;홵zTc3tܭ| vb;9L>I=tcB{^vSj[h{\X+_Fb{(Y;Kno5N>I \ z u"Bw>5kp*=amI4: >\ {|=1fR 6&'5 8Abڟb~yxPA?:?pRTM29G*g۟xķ\Xp~}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L?%N!ߦ_}i|0}Ɵ~^5ѳ2ONfZ+|Zy鳟ju~Ϙ|ۜ2Bֶ}{$Ա9`O>Gͫ~/ۯ?>lzej삜Xdrʽ9n18 Cq^E}X>TqK~]ƥރy솇 w7-x k!5ۋEĕ>φ](R2}deׇA`^wѺ];ս5WJ-{r MBj;n@޳A/uOqN)XI52 t,qg,{g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\5#}Z=n=g{u#n|{k09_Z[>5s`gh>}6!qWķŋ?ޮ~;NJCAƙ~ t#}]쉾!~^|{gZDHcE9>ߤe.Ht$oeM67,jM--f~46pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴNj{+PK<,޲9lzL3Ҝj^WcЯ؎vֺ{rr$BN= zD؛}FghwFƓL[Z=xT^&9jsK>8i<$_!N j'27l;E{MեGo^㸚ت'ugw* QukGlsغvmVvX>vXIt=4n{˦Zw_p&H#|؎ѭ OqőF7HCnW<[7?f~PFNκ&/6 7BJ;"UWkl{6fx(xgԧCZkdk Y{;'y{+gVumgctG}jV-Ҷ~P]q<`m4v[{B{^f5lz$s7[}Z+37[ 644yﶶ<~N +([ v|N9j#8Ч#lo{y?ls۰{z6v/+k55߃ѭn{B+47A7 HOp! ~7[ēys?=_A?y9*觿_o8zԫEL+S+=<\+ zFJSm߄8ԧ>AOs{7"k'e+o/{Z1N 9g!&]gaZbԣa_Ko{X1ܼmE 9>훜xͻb'um;yЌd!\^]|vW~W|J9 k۠& aG:7mnfoYq{т~q-Zbn|@攟ͯ +@'&0Tv ſ#lز{o[q=vP@BXf|zm Ai1mJu[E7[z#\Fn[i2{0M|,7þfh瞖(W_џǷ BKP䏷R97}w][=[o2ES"\[uLo@{ g'rQ\bmվU 9ky;nUTޚ^-yw=3kf4[>vYs5}z ǰ&`#Վ ..-WShϻP$95w{Ğ߭~u &q7NJxRol~h=X;蘇3Vt[ ׵iu8|O}p!zbpu;|!BmLN/,kp&>Ŵ?|S~u<50؟9Rq?4|#~r}p \!W>p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:Q=:gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]qXA?_;4Rx+9ZZ A,Bja7 +}P- a/@Qd9*J-uw{9kX3Z[33+8՚}9 $,gDKuSk fF͹ "^Lx9h'b3o=A î~G?8A>ooNVw}+oW7+ވ#1 5b{c|3|=hgl3vÊr3*Z;=l+>dpH[I}{b˾YWWz{Hvb%xnJ^zq7tFɏ={w|q;>~)Fg;JsKo^u;c?]zx}kms9{ 9'oJ BΔouNgۺ?Lk#z&^Z\aoE^bO:ѷ9)9lsrbA?Jl]Dv / "tG9 VtȜޠ ᶅ9Sp-tlu[pgs[_K &/f]k-87Лs=>"O+~+D@1~Yp*~Oغ(cA?m{OW#_H"Vk8;ktOju=}|twigTq%U^~kNCN*1N wֿ Beg]缿Pcf`{\X2ն:Tq sڜ6'M6m/4wk[;znG۲#w>^UNr/_9f s͑!˿&dk\g Ahڶ}wi9ZخI뽷žݳ <72x>PЗbG_sl\=a+\8\jSvõ]U^,`z欲1ݓHsFzz\-Cb;ڑ7[2?{ 9m4tcoOv3m=lGAOs{7"k'e+o/{ZQ oUBL&_nĨGþʗacy3[ 9>훜xͻb'um;yЌd!\^]|vW~W|J9 k-I \ z u"6 ._<=GcaHDXムo1ϙg1_c}mO۟͟_rn'mSL@x|}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L޽~ 8O|{G?}8im'jg~msy/ºگ=cmsvk ZZfbrS6G笂>v>n5^|ƿE>nCYϦ{ rb}) 9kxa㿟P9+.=t)KCyc{Alsv.[тJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO A=(g!?#_랠ZS67jeb E;y_]v;Yl_%|Co#\9xC"A] FVxvA_!~F;c=V&ѯUyaK\!!CjO Ga^XB bt'"ك| 7ދ߸knM7Qdc+nLoeEmSl{㷍B@lyw?ؾ?|LtE(Ȕsz8!YY;J+C>jk^̉Q\ٖɷO+;K٫|85?i}kSg3-s}Aȱy9^ m1gde Md]c϶e=QT9[痮AEjM &ڐ]1>^\3 ٧֏sgBLŏ2}YӾjEX8^ FƔÍm}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/X^6}[?_2]8!'ЫE oo[P8(BwoE9 -n[H[s>W0Bv`=.˞ZWO{<ױPŞn+`ko;7Лs=>ӂO+~+D@1~Yp*~Oغ(cA?m{OW#_H"Vk8;#]B}|-Ry@Խ/RoN.x?doIsҩW%);{WChⲌ z lkYvWg;nU~.6XfõNtm\cG h[~^~]zG=WX٫I",؟Vyl 1~_5l{͉3N  zom[}> c;-lW$At[b{~EH\B<ƌm(K6zmMW^{Z~|ޑp;D7-f 0(4#"^ʨⷉU+ ? mG 4} '+~S/v>aȥ.=A(l=\@ۈxY0esVsIg9zv=Ɩ_kur刊zi_:fڝd7óV4^}?յ׾IܴRG879ziHI̍9[}Q^||yuۯ8;I݇{|YVx= 9L5zN$G19̭Nb7۵}[YBr;ocqa&Y|q\.jw~ } !a;JQ'H#xhѨsonG<1ʸ6kJX'(#{'g]ge^M+4Wz=3E@[AȻF[Ƣ}D.!By 9/W[gcTϕzU}i~CqjWALIjHԪCs札Z{Y/ʛ^EDԟ{[׮[-1Ѱ%fsط=ƘbnVm`CNli&'f`z8`={]N4>YH}?׫WcƵUo%R#|t퓍qqD\K%sP9g;H~ BApnm/V\O쵝y?Я֭{^B`gZ̯lۃAn|Jfx14DFPyt#7 {Ov_a$tcYJh^۟{Z^ |%ۿ~F& QԷv/A?KI{uMlͶSo5eL NUo1ſ]]=$~ާʡ vGqףO9VV'X%Z̻1VSykzFrolϬգilsnL{1gV,]_g>n} zX}/{xnb 6^`m2>kq=uވ8[lu՝sd|3UǨ^o~N>I \ z u"6 ._Lst:xˉgqd!8?p57B}B 7ژ75 8}AbڟbxPA?:?ϡ?H O7p \!W>p <>O7|+owqxx20zc'דraz;]zf/rᑉ߻Wy^#O7\?pzos>RO}Y{c3Db,sY8w%S IΓZuv.[тJ8 bRIp]D\jnlk}̏"'Gv^&>q}UQzm5PYÚqUڲ)Ϟ ^!ĩٟO A=(g!?#_랠ZS67jeb E;y_]v;Yl_%}}? >)ЯoPWGb^kvfzff;gkUvzWȭ}Z0y#',Xs? ӼY'bL9Fd}ow豀HF7_bǃ_w%4; Uv"9oe"}#芞L=#$>+z>=QIzWֽy~zh^}}G~pڋ91+s6iE`g>{"g9om~lwfe/(96;dzыw2f̳ۜ,5LWK~sٶԾu<*g+5(y[acA|٘\ͽk=#ЋkFr!z{#k_=b*~yW;}mfqfAߊk"w~K|[  c;px<m^A0ZRHqx{_:|}"F$ fwman.!)}^|{gPr,q|bkc|o\uqF/^&/kڷ~CnzujM--f~K6pc[_7}Ꝺϑ:wxJ\ܔn蜍{lmy<'w|*Rv<߼w2d;HXs{zEarN*F}dN)ζu/F~O/M~mmjm;ֽĴ~kLh圔꜔9U 9^%."ox{;p݊EyE|+:gdaoЏopB):qYԺW-~ڳ乎-/ĥt\]k}߹Oל!a ~j^[&#\U{E Ao{B^@@ktOju=}|twigTq%U^~kNCN*1N wֿ Beg]缿Pcf`{\X2ն:Tq sڜ6'M6m/4wk[;znG۲#w>^UNr/_]'Cl b{3asmN9pBN({kۊQ@ Qha' Ʀ+RO[+wFr6fl@A_{lkZ3/󎜅}n{+47M-75GdqE޳QcoW&7~FϏAhRNקxC_k}'l'K]zjPTn{+Ӌ%uLoٜU\={Rih^]eZlG;}zk]\9b^!͞N="u>?;#ng{i~k}i\4q|or}Ґn'5[ubs&#_q\wlՓ$[5!ɷ#49_É(:µ#6Ilv}+;VHna,Nc;$:X=eS;/$d>lGtW'H#xV inG<ʸ6kJX'(#{'g]ge^M+4Wz=3En} zX}/{xnb 6^`m2>kq=uވ8?usl/q^PʽbBcs 8)ᡂKAApN:<zx#\3v1g"!/'Viu8 Ր߰G k jcr<׼2@}xir?mC hxj?#w+@? oH`>ムp \!W>p \!ӻĻ.of2Lw=L؉ܼ~<87}Sm~~voUF~\[2ods:Q=:gtߧ_y0/tu=篟M_6}޳_]CNW8'NdH=Yë菽|և*]qHA?wfH-:;|H hA_k%X Q^$."A5|6bG#;/|W> (˶eAaθ*Vjm[ϔgOozTkO' @޳A/uOqN)XI52x1儢ͼ ~^.A,>s7+ԕ`o葘j]n1dg3caE9kZrk28k-{D&ȉ%m? "Ow"=ȍp[콈1mFd}E=7G7آ]6ͩBU]ojuY` x6;~l>p&۱(Ȕsz8!YY;J+C>jk^̉Q\ٖɷO+;K٫|85?i}kSg3-s}Aȱy9^ m1gde Md]c϶e=QT9[痮AEjM &ڐh]1>^\3 ٧֏s;HN[E_oɁ>#uhmmYзm,BsGڄ]/"n,{X+}g&W/}F=rޗ=}IB][,ft]1BR4csd}>|bkc|o\uqF/^&/kڷ~CnzujM--,~0W5lnlƿo_U3^c9RX+^ QcϞm<_?Oe_νܒzXl19`~Zya^8BIۨ/Â遐3[ٶɯ>W[}~Ǻ6 qth圔9)srbA?Jl]Dv / "tG9 VtȜޠ ᶅ9Sp-tlu[pgs[_K &/<ƻ Zqn75&~#zz}XEWV9C/ pU_'諿/$Y5?Nt He RΞXTzJWY:s3**/C5b'ΡWK^;__23sG13=Gej^}︅VMtbmO&{6Z{X;ѵ-s=7m{u;\ae*'⯊Y5˵͑Ab_v۞ksrBA?[VtvߴX"m5G >I56]qzz޲_{7&WϷ1c c[➹_})w,V7һ7-f 0(4s9zVFM:_PIh=?:I[8Yÿ_N_ }96sD.w BQeڮP2Fģ΂-P ~F3Zkjlю~Z!~/WUNTDiSH{xi^Gk$Gmn~)#GߛD_k4ÉAMs]XƜ-{}/IkW[=>ijӚ[NN D~s$nxc]ǾE~+$0nŇ} 岩A~z܇ 2D%^xB+4:7٭@x1̭y2n,~Z9 YfA@pYWG 9vpLϔ[+xhZCI^~ٳU]|<6Q:?գ^|Txn>XzОAMFn:{=ۣ z5ͶVq +^Vq|[:9ЗpJN +([ v|N9j#8Ч#lo{y?ls۰{z6v/+k55߃n{B+47A*BM4܀7 7' (v JdtA?y9*觿_o8zԫEL+S+=*l zgJLRm߄8FVE}6l-.oDOzW_,"ުLv:kQ}/7þa=6sfηlrڵ}NgK79lU5;wŨOvB^8360z+r\N~~FɁ͉-'R -ƇԺjNtaCeP;v[ۋ-{m'lk-u+o瞨׶>if!wo7 +.s^w?ȍx{i2W&XzEA9Zh瞖(W_џǷ BKP䏷R97}w][=[o2ESS[uLoW{ g'rQ\X[ϊs5l#Zrޮμc;Wj$g˺y~Z=v:MVwsf5}z ǰ&`#Վ ..-WShϻC]9Gϗ8C/\joPޠx 8^'"mEᝏp9G<JXAz@rWC~1.DO ٷ/~CIx\_ȀAx|-,櫟􃠭szKTܭ]$|#oNL@x|}+A AB}+Ayz7yx^[~x~tԌ[Ɠ;1<ӛO18'p7|L޽~ 8O|{G?}8im'jg~msy/ºگ=cmsvk ZZfۘ>X"LY}|:?j^4|~9~]g+ߟMWc"S 9%Rs*c2?s~W\{*8/^;2Rx+9ZZAX"qa7ы+5 Jkwf~tBn}Л0̫."Zk~5;Xeo=S?Wb0}π' @޳A/uOqN)XI52 t1儢ͼ ~^.A,cd"A9 J7bHsWhvA_!~F;c=V&ѯUyaK\!zͿٟI/F_w63 8oEY ;kE0^r[YvJA`ۉl& j-ԇ={ܬ$r>Y(7kpo⛛$=FHJ~Pr,q‡/\,{-bL.Ht$oeM67TX4^ FƔÍm}{\M˺wkCt?Gܵ+qsSҋs6J~ٳMGW_K1:۹Wx[r|Z32##]kc=?W9|VzX;=r|s:a\^=|6rg|> {/X:H++DG;/IÉ *uyi'pVd-B>Q|3-n[H[s>9X:qYԺW- ͞Y/lE &J7޵Vs97C:-ԼMc;G2p޶}7$/k恀G.>yE]zzwBX{7FBȁۘ}ZﱭYC\q?cfLOto[~u`\QiŃ-7ӵ}G=[5V6|e~C'm&>DXÿ_N_ }96s$MpmW0"UK<,޲9lzL3Ҝj^WcЯ؎vֺ{rD^BN= zD؛}FghwFƓL[Z=xT^&9jsK>8i<$_!N j'27l;E{MեGo^㸚ت'uIZuiMag&;|8\Gv0\7ݮ X>vXI=4n}S5(Aa(G#EMv+47#[]leXXB%r.臿˳͂ R&Hr+=");ʷ0^{vnk=5duΞ=[յow@ȣ?ock{[=@ŗAZw}_[mm yd靈׳=@́Wl[luk-0"K\{4N5O}6%A`튣VBSdNo/7ُ|~=mǹm==cy /'fۣÇNn{B+47~#yA_nfoMM?z ~rzUO/^q6F\WgI=5VzxUJϔ۾ qD;nj+kJ>{fӭowl؜H߫WK{|9II$t'ʓ$F$BZN< 'k?iFkdLgO~)w}wwwy^(w<pjEXZ#c7~X ,RYn3zH{\;ADynv_!d/??_;!އ2R]>GK(?/1ꗯS6Bu?q+JuODyimBcZ }`W`F+aNT_1g `ÃdgUrpz~|=i6v=!ЯQc<۬2p }V~ ύmC}hC~#:.rƷ˟`>p~\ }.>@ D"z^_zq3ݽZ]tLt~s7ݼ&tn}܇=}yiæ55{qf{׋7ӕ#ړ٧?4w1ObW}?־Y]w!Ҿ?.ˮʏ+۵Cvx1Z=z.muu> _I} 1}Un' `z?MCO?jzQy'??z>}ϧo/ݣC= )m SE>CW}"syN蹹8ڹLJ@߅" C1^B\w \P\N۾6Mza(lZcGAyk*70km|1еyj?f[xX !B}Jl_N?dCqp/;v.lm4zڰi蟤싴Geuث0@u=m;^z@kg(_f cVgh07zX/?j^Qs{译+E ̅57ӵ W-V cvKW!dk#y: }Q0qc~߮ܗTv/8F[~N%Nz7@c0 ,w\GWp+Trkםw:*8m9+%Yն8:,.{O/GVOޞʝ-e QFuϱhwQO3p,ͩH\+e|SSܲ?B1fv9(qaUvNJu?}2p[}x~XzrnGZsz-\kz,m}HDž]k\dk8סF k]`< r!́t6"bEpQ7|L vwHT>@c3]/]C//Sp 9AGI҆Tw*(1؈R_1|RƤ_0}W1_s8}q=C0_ pq_[2A^>Q^Urj_X~w^vu%uwcW/ <.~WF}鹍1-F;Fr}v{} k;.IrL1}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ"0XC0q%~z8| AD}n66}})0}p߶}7/NNF0IT8G)lr6aQ^Sb7jT[^nߣ~Ÿt-rquhͥ1_E'Ikݾ=N/Ov9zr>f2ʲE]V)ҕi~1aPwܪ5vǪr䱾Y.{^ZXgPW1&i-_9l΂sHq}~MHa11}Zom|>512.< q[o rCUJa^x1/@ߍʹR4 }B}mΰ>V}5ė px Ks=n3B3[e%b_n$ ~"YF >hnNG{ʑڎW+ۻcC.Qw^S+z :9 rĵ֖l'פlgP]q\C9ֺ;+#Tl͗ӪN1A޵-wrz>GrecSs}=F۫1 mRWs G9ٺ^e0fK~]mmseKK|Uc5)6)0o1&՘ 1v CkF쏵olm)۟*wmBUs=GP0f!/ GCF/u@R}X!n*+1b:)B18 N2jTeְ6f9<制:uǶu+kĝ{bny|o^ĵb[m=}[e:4ʪ,{|_ZbfGur~]FUn]YgNzZeֈ@W5lWV|}7fXI/ 뱜WcLVK{!k3a!s|{9z7m:a~+n뺲cm藛Y=n@߷^݇v%D_[AT___ H/6'_//mV( ?Zm9?@?`)iy` Zz~BBy=vvִF2ܹ'oM.=9D[[1fOj6]E5mmc.;+vofڒ1*j {z܅m2f8Hu~ּ6\3/_X[Mq[3y]n>9<zL=& caiy sC*+?_n c&Prܾ#ʮ~?Je`Zֲ#=sAzZZdYN}|!;|o!U\鼛}B$}uzI/K miB+yn}"]P_{<cI{9[WX[e߮avs=}^]n_h B1-L}Hmc^k]fV8w[eFQUeuØtuf@:9?ʊ5mkUv'_Fy eֿ>Fu/F(=x|o~ӽquӯ1Vim]Mkmn7] y/ ֆ]8wt>oS}m\G<1C5&:U:4zb @F\n#X)?7~ }-Sz eDIm`>p~\ }.>@ D"z^_zq3ݽZ]tLt~s7ݼ&tn}܇=}yiæ55{qf{׋7ӕ#ړ٧?4w1ObW}?־Y]w!Ҿ?.ˮʏ+۵Cvx1Z=z.muu> _I} 1}Un' `z?MCO?jzQy'??z>}ϧo/ݣC= )m SE>CW}"軹u7G]; Z>X!q`7K_BC r}_O|< 7B|>(oYeT~sm86bPmQlo+ !=@S@/^V'L(.e΅ͼ ~Z[]]6w} );:.@8=/7^ѽ͠Օ>QmSZ\;C92-eWjS=;[R7c^/y^gV|=&KYjSVBys?eӹebZsP6\ê쪏9'~dηBsk[KZps~ݎZUkˏsf>q>n7ʾJ:'uh?vZEqs >`X\v u?l{m]5P،m pF nk0C}n~A8QAR!՝A 6ԗp;?vpԵ1xk׽LU mNAuq#kK~=Ы'k*[nW k+μ9.rAuǥJ㣯==\w:ƷhhSn/wmc_{r|Ϝ_ɹc,3u֊臮IcR.aX~D1˖CWrj|Zcֿ>Fus˶vy}ZוGll@hKcǰ|Lvh@"l]B^v v ~[>WIovbux:re{~i[5k 絔LDxa06yN&  =y?O >q\nh PTپU+غW@m{+#߉!r(z kO:>f0%הW{9V()h迩>FvGrq[rКKc0N ׺}{>_Zr8|:e#eeu5)qR+kucRjUY55,;jUc}=,=e]m]ΠbLZ*r̜ɇc> &^sюZuli~lVo|!Q*{%0B/vXm}`i\o)[_>׶\gX_{/5e `P\kdی 9VYۭؗ*+IuHjGDۃAjzor䶶c ǐKԝԊ(^eN2q-Ǭ(+5)Thq t_5@~{ cv BpDZ_//+-Ye%f_V:E1^yo|u \UR56,g6XgV;qzeso^s: WUlmq[v޵[FYU;eZKAlnWn۠˨ʭ[=2IU^7j֟QA|nޖNyɍoO(v} GuXbgc_d? n({} 㺮Xe:2_Vh[/2h]/Vb#՗>**mBVp_p[0"f0)ՖXc^M /ϯ8|Xg1kN 0ꬕ0I׍G/*ChggMl/z_ɝ~Ɲ_s~VMn9mQozE[|>[ފ1m/-YbΎְ]a&km['j͋m1e̟zkSn}rcy {Lo#W+憠UW6ܾA @W=Lf}G]uףeGz &sɲ|X睔;~.Bw2<|1.4,B*-y7ӕ"^a/җ^[hն` B+yn}"]P_{<cI{9[WX[e߮VXvs=}^]n_h bW?Ε|z2R[Z׼bĸUkl9o>Vٳ|fU:np?stݹ0f:,]jY5iΏb ;es5k]ɗ9z^ByQ|rʮa{M__y^ݸ:_ׂ6|_zs!i >A0`erڰ ·u}m蘇3\yADҳ*\ǁF98=}\v?a>b~Wܨ1om8>p~r?eJoϡ?w@러W6xhw#p@\ }.>@8gt޿֋ngj?慵og5yw˧>M|6f/˵3 ݋^G}.מL>yG}U~wYNNiS|r/b}l}XI8>C@pޞck}^P2$0z qKpBqu|t;Aoڠ7]h1د`CC -jm}m`${ + 9D2c1ÅBڹOkK|끾kæN/q6}C >pxCޠ.{Fh07VWGMq_hq L]~̪ CkVAs=jbnO\5~pٿafv_߹/gwKW1r>p -߷:E<;UK&Qߠ|d 7?eF~i!c^S·g:/5 Z)uݾkh>/A-먮 N[}zJecV-A}9Kk<>oޓǧc㓷rgKYjBfs2Z]%*#d)Kms54JwT;ol{:P@]r y܆kX]"Dݏt2Vhnmk_^_37|ܯۑ֜^N?kY}eB[]k$+*o8סF k]`< r!́t6"bEpQ7|L vwHT>@c3]/]C/mKp 9AG҆Tw*߇1؈R_1|RƤ_0}W1_s8}q=0Yf8Ǎ8ǯ-fBb(nl]/qWol;X;躻zʱ MV+k#s~{Mr޵Jc|Z>s>R|9~$Z<ΨZ+^'IѶ޺bWc 1m+t.[5˶^^>skY[--"=iĚ'4&rL1}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ w^X<q%~z8| D}n6}})0VuV/FC\Q^@5i󓎗!:Αk+潜gvXה_MkT[^nߣ~Ÿt-rquhͥ1_E'Ikݾ=N/Ov9zr>f2ʲE]V)ҕi~1aPwܪ5vǪr䱾Y.{^ZXgPW1&i-_:g~E&=FkBz5Wq zk[!@!-vQOofzK=>`7B߭W#kǏն~nVQU<,smu nyr__/ e/exKY̐Co}ݪrT_7dv4# }WӋxCߕ#ۯVw8\VEE.uTHX?ZPkKYQ?VkR3|y_UYJ{]١M|ko*ˏiUҘ{UO;scI=V٣S9mn걩K޹>՘GkΜl]kMrln.ʶ綹%o{ŚpKV\gk5-1eW0Ƥ8N~sVܶ:صC-1esW }@jǁH| A0 %/ GC]$۬/v_s"Jݍ}7?:?.ANU` kc3Qh_3YH}lNY׸_Fܹ>s: WUlmq[v޵[FYU;e%МKAlnWn۠˨ʭ[=2IU^7j֟QA6ɍoOXcb]XuPl ֛GAme[w{\וk ]UF q]F>+% BpҼR@[@  E[0"fBrՖXc^M /ϯ8|XgKN 0ꬕ0I׍G/*ChggMl/z_ɝ~Ɲ_sCDckӵ[a{[[^6=OalnU7eIDATbL6kK*c5qzkXɼe "yZbp}t~bm61:orɹe1c1/5gttC\KcdVq^qLr1t_0RQvSQ*Km:\qo?Оkǟ7Ӛ(ϭׂ'm<`wR!Ÿ|ӰB|NLWx$] M}i-X_oMZs+!;g}'_ϗsN.D5l̷8 ȄON*vk'?S5Pn/cQo1~ԇ<5o;1nZ)1[Λs*Ul4žYUv]w.;KZgV=q cXN\ ZeW}e}Pv^jc/_bm~㮵JHuODimBcZ }`W`F+ua;:ᷩ6crj!HϪp=q1{ m{B_sĿy.Yd@拟?އ>Ж)?^2Fu]$|o?}@\ }.>@ D"t{w7ݽf{s==`wMkڻ{g[d@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdk\L}.$<9&)Uy4 _?>'ϯg}|Vb]9_'[,7w~?~7{19X{O7Gptk `U!^F/!;~ ]c.B؂?wC(n'm_~=0T v5fQU̵u>~<|AEB !׻*)O\v+qNˌ?>=}yB_fz3;m3ė=/7^ѽ͠Օ>QmSZ\;C92-eWjv*8NZc-f"pVĵn{@UFGg1.+`fzy_cz{Իog/ozZҁ|1WaV5t9:tNx~^ۜY(K2׭yXcWu@?.D'8+2ê*0UmFe*AO-B*Uv.PkVV/FC\Q^@OHB^u=m̶ځ~^ysqX _~z-o~/k9֑L;Ǯ}kW]_o0f@[:t6ߴr?` lq zrx+7C[\(n^;~ >0wr-}߯aPk[3ma{3Zp`GpEYNE;:[e%b_nWV;\dո }WӋxCߕ#/,jפZ~[9^,{UndhsZ@_omk IU@7^֜ 6e}"ƟP)a\a? \W]Vv[E?>l޵aCW8o>@a<r}ul*+1b:od8}ncqhYYqo@ߗ+;+??W7rcRu>̉+XuPl ֛GAme[w{\וk ]UF q]F>+;{k[Ba7?tFUb!Z>0/m:Fb"Ǽ>[ǿ__qR~-5'* ^_tʲW.܎ױ(Ɋc^?/Sl{e6V~b_59gb̙̜ZNp0~x>lx0caiy sC*+?_n c&Prܾ#ʮ~?Je`Zֲ#=sAzZZdYNJg"PZ|!,A^B鼛 :J/l2uf_Zm rL;Bo^|3uB}]ka3{ u}t}}A>^Î\ǩYQ~u@YG_-cSy ϟ~ƽ雿v'_~+m I9^Q~O=s6uk,ך}}yq}pS?>@ D"p@=nW^LwovWw;]?StIdzO?]}g?zyO?~2}oV=o=}Ï˲cƾovPv35z̆zeGvhϵd[|OcRoL_I޼``z?MCO?jzQy'??z>}ϧo/^kekt%; [}1o}^P2$0z qKpBqu|t;Aoڠ7]h1د`CC -jm}m`qh{ + 9D2c17Znhk@uث0@u=m;^z@kg(_f cVgh07zX/?j^Qs{译+E ̽{3]?wj~S}x+8o {.X{}5=[}\-2@"! z߷:E<;}^B=qj@'o8e kc|­G')[e[2o[=Vki,m+MU7}Q W-}^Tāmg47nXQ!ngW+ˎgrXm͵+@F ]WѼ"X.r!o~PLkmKA 1ӁBl9 6S>L[ܗqpNi%K[E]/`֧b>wop|chk6CYukou.żH|pB\o86c:^ӻU?˲|c(Wio}u yq_+uȚmlC^:'eE?v/ۉmN~Akm<1 :SRZZ" !٦@ˬ NdI9| D}n62E޿moo;1_U,ѭ7Ӎ_ao[.X^nϪs84s;|#O_9v }jCgM `'x3m0!V+zj[_}7N+zK(*:Xne32!x>v{AUV"vJR}Гe$uh^|~!_^Ga_?a:zEmԧ2DUg}J2ƭ89neo7,_OYNnGN~sVܶ:صC-1esW }@jǁH| A0cmVYչΚ@_put8}eqk5uםe-{ߏȼO%FY@z@Mvu3&\}6QX2Ƿ7ُ|ۦ#^+?vN~ }>5}(`W c]1l{>3*|seϝPl[(NkEy5}6Џ/.{\)~|o|Ƿ779eٗ*om~o"`_g΅S}O )O?9 p6G=4=g_xGmї{!p?V.YdbceF>@_[ѲE_>tx {o~ZO7w˧ƾwtsw7ݽo^4.tes 7s?suJsc_LHyϮ[+(v7@?-@~qsC'v~f;w2п乾'{pj;9 !>ovzwVh9gs!/@8}+Do' ~ΟϤsuY?.пzLǵUY{y|OAhOGÍ&Q}$}c,yܼrTs nL\eB~\t9e>WTwhu^{˼cEte\?{pj{1kPqٯQ9/p^'׸@7V췫@?PjX;mסǻ9fhk1q,Ĺzc[{9?/Ǻ1s:kƚt~-Gﺐe[DžmiѾSCq",C1m=o~zu븭B\؞yK#p: sr :|;9!ף4C޸dpj,ԘPTY},04]=*@]qm?+͐wMj,\_/jxL.q^\?Nhno5~͹~8>v g:eiʐ1Aㆠ-}ׇs[}:܇_>h2`#]q5Ձu^Y y~zWc[𾌪mز]GUv!ݾ~Vw} }/ ݾ*$q~3roRy;ݼV~Xuٲ  Pu9]uK0j,~6]Gۭ貒.ѿc×rT۬9>뽵heoо{?A[S潹exZ@?}^'/BU-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $7gtknp臾/n~Nό}'N<}Nm@?!cw~電Fh9gs!/@8}+Do' ~ΟϤsuYu?!/eF} B{j88n6y !kc1ϓ浐H+Xpc(_>mq-弤CK}X7+u|,d}-OhS;ЏA ,^0~xHr8 }~oD[[A߮@IcٖB]:r瘡y;V߲\DZ1knYX<>(m\k>;Bm[<9GN~Qa O e(vZ ra{{/9(v@˶'u] 9-V\Wb!y\I_u6eTmS񐉒%+?r>x Ư9GڮL>,C"<C2ݱ>htCn^mas6~U wTye2 u 9,_Ecwn2cv7UمPv[G߹w/BDه"4ؿ2w|~ ߤߵvy% e+@#s|;Das9Xtrm$%[me%]{·/51kYss}{k5h)ޠ}Nm@惶`{s=] 4ہ~cE}*a #s#CI7y1fb{P2Pzeeư|[6\_F#.֡o@;ki>Zer>ZsO >ԎVtPqPoĽCXo7[oA;kp?0O9@ }8}h!&}8}h!&}8}h!&}8NoۗO}n{s==5=}y;߼?Ti\8.o>~8G!4:P~7{1=nm^8|!@}3|d N?/Z@g%>C&=z7fn,֛:t>>ssp}/?2_N?1M/MhRσtX R`!h#\;,ZFXeDɅ8g"vjhy?Fh\~9&@."/uXy>z}ͰZ:R܎op<Vgc$8κkΤ1YǾW}eRay5p2Σy=aS\]O ](x'CqLʐ1A_L}}>0DžIFWBk uAH^i6oWcaj|UT/v)q`ZhXbV0 _hBsXx"™r \JkuH@u?-k6臲,l+X6zZ3!tc VbzXYvnKnvqv#Ls"[C6Uq>[ۃ6 ~Xetmۣ֋=Џ|}khUѬ6msy9f@I^!oV CX4n:g0V;jncZe{j\Mu kW!Cޠ_@òX4,p!/j[8lW|qQ]eUy{i_=kvlx'ӿyOg HۂmCl\~YG 꾔!Qm0f[)O9 rȮu8ikYJc/>aXЮxncֳUZL\lH{^X:r_̳\߮mqeԿQm9oSsu}c51]}IOWcݸaU۫a]m祹=};!K__}UH>.6cߤߵvy% e+@#s|;Das9Xtrm$%[me%]{·/51kYss}{k5h)ޠ};wG1_?7_Yv!XE)N;ɡD/ikDkSؒۻzl} iRa.p6lol67{XZ:cm{^K:lL[N]I 뵑`Kw-UuY{ׂ#vDžsܵmc:qZX-\+׏*_w v>gE׌r}(azY3~}x6}uXnZvݽ yxnY 30kޛZU?ޏ?;}*a #s#CI7y1fb{P2Pzeeư|[6\_F#.֡o@;ki>Zer>ZsO ?>K[cџ׋Q~S.H" \Ǵhp1 S0W6h_+Lkplݞc{mXz-]?Vܘ2x'a4P^nC9L߬ubWp_}0ӛZcڿ=oA~z}sn2W=8=֩kf=}^?.с>*qjq?: }Fc޲ߓ{X_>[c`@M!̇"!úm+;6N%1.S)T_zю[~KJ8#zjksf 95?mg8fyI/s<}ggz01BK!/]/64@ks:?]ze!P7̭qR5TomrٗmL+hZPd\\[,+::NC mb8g=d%P-kYr:aa~~ \^#tN?Mۡxk5{'4ung}{G횾9rcA[몴RkU<^6^EY흷Ǵ跾F[b}cueOM<%"`x'?sw.$?Mo`oKTXƐb'0)!a޾mP DŽ2v/pN:v_ys:ݞ4 q]{zRHoIm]Q.ۯ76kh^zǬ1wﳱl߿/%+ kX۬U[ s]KZ/1JZ!ֽۏOG_o߳ǟA?4 6ZF}m 6:0 ot_{xYK9{k w?aU&,ꐲXϺ=q4ۗgK^/af8`je=&p-6u>k'>kܻ%'[٭CFZ?cE[\qi[LqI}7&>AMO޻_?+yYޞ QYjM1H1ڷ08IF!cO6l D2xY8R_¶-P #ơ;hNuܥqk[:^[cz~8ڬ\7_d6O vrӚLxoS{{CC(g (ւ5~._.3ucǥ?7l=1>e\GX?޽9ۉ@NqdjM\6촤-$'b2׎j]Vhi2MV[0l4hNxCVZkx_;f=BǞZBµ!b8=ޒ:v} ǦyMa1e쒲V{F1Hm!z-c"[k\vuord0^Ro,CN\}h{~'ۂV`'NR(O.Ȱ$'޾5pm+^P]joHt8OY'iKZk724~{ [Ŷ>}ׂ\߮/ش{c&h&v,y,so^R{Z=d0 fs9(Ʀ5:v֮Wsjn} KBM7?ӿWQT B:NږCX剠hXֶl \R9@ΪXq@k_kb&sc0S4yܖkɨ}Ӝn cbe \;\xϩ͏Q\߮/a8/6[q s'ǡ<7/e|}yjؖ݋}(^V[\}h:現Խ^0QK!ր-L3VRNZ}91VfF}W+hNuy*I^x=9R_ c=Sxr6yJ3{\֏Y+٦8ܷ_Ͻ6eH-ǡ:9/k%in]WZ#qC&c֡k\jC^v=٣,?M/Wtzzj{vϿyaҸ7ӕq)\|8Ͻ/A~1V>\ uR4qHLVn?7H匎 RׄMش(z0?~ ifgY;m9\7Vϣ9"kPPwL_$c/Y#Q&翿֯^~YW=Io^keR}}\fP52Z}'},&+ Rެy:hݓ[m'.с~ s:{}9q?_׾=՞]OVPo {~{/>[Ntwbds}O9?>v?l#³F ~0 u^8:~N!e bړB^Nk?4+v¿TtjVغcPid8Cs64_J8"i^u)%^ݧ{_Ͻ6[qpԺ4+;S{mXO=m˵aٲƵԎVbN\bzw$fί?=oVh9gs!/@ݐ5S)DGojyh eTd9tni}=´5!qt£Uǻ:n4i^/k7~Ucl[k(Դ'ўe c=_"?azn=VԶ}y{Κ~񿫖Y\x[!b|=op,&=J3~jev ?˟~//_4v1s1oq1sk;LZB_q>fx}'?/xVa~>ǜo>"F{1kPqٯQ9/p^'!hT7V췫@?PjX;mסǻ9fhk1q,Ĺzc[{9?/Ǻ1s:kƚt~-Gﺐe[DžmiѾB½GY;TY9=H0DžIFWBk uAH^i6oWcaj|UT>4q@>>4q@>>4q\tt~s7ݾ|j{\M7wwݛ驹n>慽JLW1py_?<>9:aPg1:2o_fz:ngӭ^  9!ss;]?3;_\ߓ8bzw$?oVh9gs!/@8}+Do' ~ΟϤsuY~F ^o-ӵUY{y|OAhOGÍ&Q}$}c,yܼrTs nL\eB~\t9e>WTwhu^{˼cEte\?{pj{1kPqٯQ9/p^'ɯ8}{n *][bpBPԏyhk+hUq5v,R蝶PW}]]34Ou[8\=fa}-KԽ˗LJc~5cG:Ֆcw]Ȳ6z4hS)!8ti` ESގCˁv]_sxС~ho%V\W! nƵ Wc,u=?c9N>З牺s9[VIho[@sNm7SC _S2d }P`ȇ޷m}` Cseۓ:ԅ+@+q1 ymߤ:2ǩxD9~~| `4êȖeXYχcV颮[uqTc ˵:_nEv;֘cǠjUfui!^u@,{8r5t-*lߗ?7CU-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $7gtknp臾/n~Nό}'N<}Nm@?!v~;}W{Fh9gs!/@8}+Do' ~ΟϤsuY-?4[t0}~?^_ShSpccT i_y<7>7F\GP~>nF]k.-%u^2oXѯCc!-]?oY:w8Gށ~ e`yZ!T\pN@2uOOԵ=c~# vj\M˶z:Ux!t; Sݹ|?2}-:8WY{tXuu/X7Fn_zXo]l븰;1͹~8>v s:#]xZE`(C0в뿙N@.wÃC|\G/yNu(͐7ٟn \ 5fa11U}Vq0DžIFWBk uAH^i6oWcaj|UTܼ>\mLAaBd:d0sXƘ:eTm ǖ*/?n> S*@sNm@_/EheU!&{ IktZK0a d2,VCձG@tQ-vørHtKlڢJrD _Vk1cP*o:4:kSAm9Lzihy;U-FF7/TGRn,b39el׽5L ~q]a+mF\C5n~w>l|n_?|:Ju"t?e}R=@Ї{·@A`o2ނIw>!`x'?sp>ĽCMp>ĽCMp>ĽCMp>?M/Wtzzj{vϿyaҸ7ӕq)\|8ϽpCh+u̡ $~3gɳ M|/}g _N /yIN~1CH[l_jƛq6@ {hS=зB&z"wYLz>WY3_a ezF} B{j88n6y !kc1ϓ浐H+Xpc(_>mq-弤CK}X7+u|,d}-OhS;ЏA ,^0~xHr89jg*;?n *][bpBPԏyhk+hUq5v,R蝶PW}]]34Ou[8\=fa}-KԽ˗LJc~5cG:Ֆcw]Ȳ6z4hS)!8ti` ESގC6sq{cB\؞yK#p: sr :|;9!ף4C޸dpj,ԘPTY},0DžIFWBk uAH^i6oWcaj|UT*qjq?: }| vM[8p>/o' A?LcY"Ԛ7{'2bW?77ߞ>Ze1c_ѷyev(o=fC#_+=כ&IOoEgF꤁B L/w(}~O>w_M/>*GϧԹVžKklqvgo]m2ËX.*?"'}m|crԺ?Әck NNF^ځ2@&n8ZT(Ko_,eu e.יY;ߺmFv\^ꛜtcF߽5ܾlk leOr .k\ }v({xmŵ!T]>'}?:՘7{y ZZ$ǩךm͘lY3qcR:_/՗[]?/A|qmǔD*E}oLR_1ihoOvz߇7ƏͿNphL~Bm9ܺ_cBدG"y:q5r͝I,;?E5x|$̏lXXZ~/˗cYOiQΙ׺eo^ ϫڭb[eu-s)jL/_ot\37~-:e&x5,{YעmjLʲVctKj?FgmR׀Ԏ0nr .Zϩy8aF yvxsf[m+ˎgrX 軾08K%|ɡ'mK{Y|m}_ta,KQ?/mHu /%O_hۡIrݗ ?9?>8dy Ժs,)_5-\٣{Jf^7,=?|ckԮǚrK1)&ko5?^K˗)(%}l_VCWgKj.o d y3^9k9?/rƀmoo1{ǿܰ_N~E"YSvb+Zp幱rPWnQkAFif7vT?c}|ǩP>esur(i:foUSx- kT2jL1\5~?q}|ݢOu|/OɪYB(>.Ez\ASv6Z3p:ehzc֣+~r>uۊ>v}+B%o}oסlu> i:s@4ΆP([B^ougG9c^kۮܗeNEE(]ɬ^My>q\nhR]]E/U? wb+ʫ81ʸc8r|52UwZ_:(bk0QcSN̾Xc~ˏYX\9✥m\ժۍN{vўЎb[8j>fك^ڥۅݚnݗeF9Z9OaM޲ǬwnYvٿ^D=uo;`>$ڸv?U11=&a{{|N٣Uc,۵K#~y+[͏יq,&v}~٪g8[N⺨6YˏoNƹ{Fǻcʵ׿XW~8^PֵzAl˲c2_BT:oڇ|8FE=AyRs.z#}|^ɇԣޏ_}@\Fpuu9W_p@|lf`Q}-Æ_\__VYۭؗ*+I_jGkסAjzor䶶űR;4gr]타Xk_~kbe+{kDٽ]zd/jםʙy}Zw*Ƥ[wX U|DF=E[ r<&N=fkC\~^[s?{(#k)b׬AٲOQ.eu<}[}+ۍi_3n.+]̹usS☧Zd8>~,ezrd\YkP О{37s)\a?bCܶ:T!H?>l޵aCi{\ւkC [Ef`Q}-D6;u=({YcveVٹct^q}Ȳ{J{G繴M(on4118ƺZcf./0/+/K~; BGlѦfUی@mk.1΢=sxmbœ3emh}Tv>l'5ʩa͹ƤVm8ɗ//h]FYv[7Zr1e[-ʫ=s_?fb>cs]fPre)`o@{:Awc_d? n({}ǿָ+?֣2@\_V[Zs.n)////ԫ/Eygw"<(V@_>/{_}m e5¿ՖXc^M /ϯJZ~cBX+ȌC&ǷV:k]:-׬'"N1,+uc2]#q\mz=YK?_sn9fkGMֶF%MG|yc*$Ǿ/|)/>±?wk}+t^}}r ZTDjgno'_ڒ11ו/ѷZcNk~K,˗r{"~s0v2 Īy9Xf_v9Fjz}}me M0n✧1Kfpvq[qCk!WvM_G:m׬ՖC,}-1s꺊TwXWjsn8jE\#C088&l~}:l{GQnwD1Oe͟o[ˎہ~c-IJsz-^]y9%~:,_W_鼛}qP_7O_znٗVۂ2lh)a>B^ |뿃F޿.wB=z<)s_@B cE 1eZpte١MѼ&0%xn:kZg^+O7uUw﹞r 빨wrS}cRnݡr]b̋XqJAGv:t|뚓k$$A} T&]lE{eq621Mu~Ҽә IJj 8:Zew={JٿbLzCilTGu037i:=fl,7Qt{a|/՗Z/_/]ʖ/˶60(s=.es{sܮFHD,w\'S.ǿxܯrk#>KòKy-z[sꃪ^QnwZ,eʶz}|T5|.W˥啕-ucC6o~O8\ww1u*{֝Uv5^|nП뼯}}uŘDŘ.ʼݵSE׿ y=Z \s([ύo Xm:볚Y5Xsz:m>>P p|XׇEP_l}cYoSt6x^XB(<. ĀsεpºGa:8dWgkk>K5&a[/αoxb ~ o Wo _TPݗ@߂/ƻf= wVdǿy.]bPz ЯVI5ͯ= .)r m@Jqẋm~"Ia~ UݶK #rYpt|xN_Oiyֿ$q/w>@ D"p@=nW^LwovWw;]?StItd@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdkԾWzs=$b-Ͷy>!4$Z4>G<~:w_<*؇{ۗOmW/7ӕ<Gx(]@*Q˰4K%u/o [ nb!U}\_HzVߛv@cGAvk*7K}djӖ6 Q{}cۅ{}\gH.{ + 9D2c17Znhk-Ax(xC/{Fonu=m;K~ɖ+BH3k=[++כ?5wQm8 ܛۗW>?{ ž __MO1r}s22@"! = I}s_^cQ!qjw@'Kh!/7ख5 埤8n k[QomWX/n_l@k8_۽vs?Q[3}L_āg47nX7s6Ca߂o[\vBgrX _pwdʅqV@Wr)A 1ӡb Br >/mHu}/%O]x{6Gյ<$0>o}*}wN6O qk8a＆k3UQ;[uݷn15Q}-Sk#=}+<џONV1<`=. ٣, p6m@?eP2t6ߴr?oYm2.pz sUJ>~ @Skz0#ps?luxfR2Km`o@$Ze%b_n$W V;|`)m-@=wmm I^:rg WГ9ORkG'Dvjzm|coO_Ϗwၾ:f}Z 8}eqk5uםeUS]k^#G(DWT{bPv5&oG=|>_Bzmh8^Z}x(q3Xz?~dn?>;0C.[o}MG>̏k\וQvf/+ -zBs-c|S1.6= BsU`E- Fab=|pZ+}̫鳵~{u'ׂX3_OŚBC5 ZAvyݗZ'~cݝϺeŨ췌;.,w???OGZ# p\C8<# ×p-|]O<lBxffP}` h{W|y6Zk. w s&w{CǐV+†y1gogzs((66o-;nu+뵠y]ww)0+-!b7Ky7ӕ vtモ^HfTu ;,a `O>v_!d/??_;!$V&iiꯒ_]{=Rٶu?1wRy~ٲ_rգŨw S ?y|ӿ@"|(f|(6i|㣇>@߿9}[pQ0TvQ4o>v]g3kL古[.ܣA5PC{kžv}h<ǡ|ZvxCUzt~`S [RGp3)$_egWZ C!9ďn`iNpK mCi"6)\&/m7bJ5^_\lXguKcsU㺱]'o/g@?㇏#Z?6{^ :!c|غmn-uLv {byt5Xynu2סtlIwnVrZnݸ^\屲ܖ.'~즺F{n&.D-p?e*V;Ŀ /p?/ p/?3sje/ G|;Po[PcǹW7ÄhK澲n㰬ŖafzGmΎEkõ-׿y-})mrFbȿi=qs4]Ic@?_.+1ײ>ZkKou=FKÿ]B١=q8ݺlkf9oSsyxGI'~^ ˣ?;O|7v!hpzm3 cքpdžRB*\:x>ÿq:ok ͙]nk։}=5 >prOK>?'F{x#P[!C*f$N0/} w uV[J~Jo!9=/zokb4=e1iYZk&b[6}^Yor*ƾk9\};Hy֮95?Z%?})z/5ۤ uQV#k-B}1(ҿQ@Qn~>i4=dž^km^Hcoּ}Z먧:B4{w{Ҷu?օ5kmf;rٱy\ίtN:Zn ul)C'ۭ0gZpok[w默e-}}wο{cKFU"Zmсa;Q. ɡNpc*3&~;:`Jc_קqp$lhhFs n aк^ZY;ۇ+]6k5k_CRCkޜv{֯cwFߵRF -1juƦ>^u=YceRwyBҸ \`eo@"5 G£R/9w@l?.ԚX۟02̒!W1+cq۵pk<ס΋ݜCq[{]ḃy v(&pM6ug{s6"t{w7ݽf{s===ߤvW/^q5cp>ǟw,<lA@1d}جJbQ_p2~!Zӎ_r,R:q)۟Rm@kOgǼ?}__}'WZfu8Ӈ_Kޛ>,>f+??le7=SlWvdkԾWzs=$b-Ͷy>a}0⟦٧?}~5<=>SӷZp\: йdz nO?{OĿS\ޑ%@9@o;4v5Eg]ڟ=u: +vXڝK%n|Ri>؝3joٮ:+c%7֭i,sfIخ &nZ~UYk^e޺5T^"q\ӎcw\cMML-c}32o\}pԇ2, ǰw bKV#Xu|1}UwRQ}ʨR|y?>ڴe l0a^߾ߘva^?,p9!!)O\v+qNˌ?o_>5 /]A+O<?Zc 6{VN?LCR?\tkCu|[n|1 ZsۑU^oPoh<&;ưl)X}u:>-ZT)q5mߜ4k$?'[Kcz5iU(:ot]wٱr:.͍{䘄cӼ1k3Z*5~!_*)vebzTw|/0\;C92-eW2Zgh007zX/?!WV7Ikq\u3]7m\8wB;ބ P›c>OVMF~ /yLu"STv!qj@'o@?lw._umkԭ_W5~NFru҃Juv]n3!]=jK^}O*/cI=q4liS\~:~LQ֡F!C?LCÿl\o/lKt6Js1`)uՎ1j6Y~9?ۖcXm] m\k͏0K {!{oNJfz{!u[#54w-[ǹv>tS]a+|9&߮/a||/6[K%}NvAs#톼px<n3ao-eG*t/~&py~w}w!,Y uPgm[C./RY: 6S>aܗq`Q0n9ΰ0}W1_s8}q>Π=k]^?˿gZL_:Eµ',KY,_uef^iLZ׹${,##߻f~vLjqp~m(K~Cj(dF+<߶>H9:Tb_DN)ٶV9*+R uo-Ǧu]g|=1am9C,6 cZiϧۯw5ֶ0Wuc}UBPտt]kЁy~o2u!ckãwzqe6zBm-1 vm,K1]3fidZdk3pv}+B%o}oסlnp^AwCGm@4ΆP([B^ougG9)Ƽx*0iS_PU|U:\My2rE\a[V`wV/FC\Q^@@"t6zwց}]fv`[@_Sԝ%Qm,T=N}]{6_Vw|^:2ڟ?p@grz#c0!Mŗ_Zl3 Bh ,db lQ27u@>f[Kr\9^v}NkT<55^-Ǧv庵Nj0K 湏Wmk!k֞xF񽠽Z/~>o[kb{1V}2ֺ~%kn-+qa[\FWfoY|1irX{-zm.ێ~\ˠ~ aeli~-߲d]ԓǛz-:7w|H= l~٧ HaG-@^hu($B>3Ǟ%З#ϷJ>3uJfdv4@S[#@=wmm pci\_-No_olk&1I/ukxp ~<=ieYݯpN/e7=r `{#A}:Ycg5|`)#|iTk6֞ls4j5AzjԯiiІCG':(й}5u^/e=2NNs-s?i+FXgC}үSy}{m.N~C};mC됺>c i{\ւkCS!wc dWkf`6l~}Z 8}2_kXACX]ծplٮry/rDv C{,՘rAu1ot_o2 )8Ġ-)*VpyEB<+RJ6j\^8aC]>sG/`Xߞ.)IswiKq?Z#W粞Fx6,-=q8^]rwlLum[ŠuZ(y8{~fYz21ئ]C =xK1dAr&owtD8_v_!d/??_;!<(;.5YړcR׬=42asYicXcSRϰOY6}i0x(x;@~cxo` o. ud;\ߕU7}\=a>m*ToGNxg?_!;>E"p@\ }.>p~Ϯۻ7ݛ驵ݕN~L,f{_{2=履F>bW?77ߞ>Ze1c_ѷyev(o=fC#_+=כ&IOoinɝ4пno^m}0⟦٧?}~5<=>SӷZN, {@tTe?^M/.uj=DvrtRX!@]B.A;~ ~cj?wC7B0T{";j[cV_/'ӇV̵8s.,ߋf0'.Wbr8'epcNr pb{_o {t 7]غ6 %PNdKٕ !5 ̍ˏ~qҟ;@utL.sݕ.KoMW ^_гqp͕ [֛߷:E<;ȩCnݯ ;ϧO~C_owQo 8+Cg'˹-:OCowVd\yHCveo^D_˾Ysmm>r+[;KiLāg47nX7s6Ca߂o[\vBgrX _pwCй҅q:Ȳs ]\CжEPω.q_K8Ɲ/o?g ?9?>8d6|\%2:k֯"O{1EY|ݍvW1^iy;uWϝ5fMsωzꋸ Mte΁rEHMp:A 9ss1m^ۜe{+Tc ͹s<'Řw9|YAa#ʳMfy"TrE\.&q鿱~}@_|'˹ꁀ T`Tk[Uǚ5 |ݝua_:޵^ϣE/#>6|UsWm)i@?eP2t6ߴr?oYm2.pz sUJ>~ @Skz0#ps?CVUDeyf քq[+o/[UWR u&hIk}@=wmm+c?_^++!7XE[9Ok8#ϫ9Ic ȶ:p*\oW+?{J}?Mxv CpoF\%Q}n[*Xy$|S?wUP=Hzd\|vhG߅bUVbku3 CK>앷q&Wg?+0ڵkaYs :<_ƍc֯;eƬZ5پpc Cj|s]YǏMT1_/|l g`C =xK1dAr&owtD8_Bc۷nqldݪE=ŝ_>PM wo]X6bIVuXF-{|~-_I|+^n#\$}i|S#@\ }.>@ D"t{w7ݽf{s==ӗ8O7w͋u%ިSs_%ݿ:"cnw~w^aLN:{SzJdzO?]}g?zyO?~2}oV=o=}Ï˲cƾovPv35z̆zeGvVJ{~7#Ml^+E9F椁=|`z?MCO?jzQy'??z>}ϧo/X:Cеi݅.&_@?3ܕ*1fg f9}G}~J#]x-Sː8K%u/j [A?Mc>.m/~=+ Me v {u8=*{c_O6mk+[? "qh׭>/O2B}Jl_N?dCq̩\CoKޢoqϪٻz6j@?{߾+ BxU S l~]jƒkg(_fA?fxv s㡇5c2pezs}%t}y|oG0"5]HKWg1rveFsu vyLu"STv%Ǩv~o ?}>}~#zkY1z>=!tYmс~|ܵ"kG#e|Ķx}\uͲ}5׏3\Q{ ?uWS#UX|N|,ez>~u⩲1cۗڦr!ۮm?3&S[8ks[s'4uVEzz kiߠ_R=Qoاe~˺c&׈nsUk͕gpm5NvAs#\톼աlx<n3v-eG*t/~&@/] hOb9}9Y|m}__ K86S}/w~ OBZO[}@[!@ߝS?O&o3$G|Xeݕ:k֯"(J^1EYZr[rm~-#xcdxm==X~0*+}٦::qײ/֝f?Bsc̬=}y#ˊmmvt[Z:z[G(Waj\Wn.k٨:&;]_̽k5zMvx=yCY\G([#^#uYbǨ_i[c-I:O9AN@߿n]9wy!!|Au;7ǁe;i;4I1汾,6һEx3 _W_/ŭ/gUy2rEܗ"1VpV/FC\Q^@1|keKR׮mUkָq̮5'Tǹ6~ܘ5MylT\{]u;t1_+VDp6qN_ۯdQ<7(KNaoXØndkL;7kea̚`\kaP?v>k.gV-wØU eUWd^[e@?eP2t6ߴr?tq zx3\ax|^ɇԣޏ_)5s={a.vC,_X[_zo"|m|DV}㕤ք V;\d퐣e^|~EqlWZ!~u/w4xV5 q mڔNT@ֹ~<'e:fcTyC;\o=cƴfχhX|?/Ous1>c5Y {-U0)Ҩc0'U_cSźruw*KeUݡ>5cUt:f^N~HEQ}n[*Xy|S?wUڰq~~GFkЀbCۉ/֗ۿo*Kq݇!b[ v/v_sM!H[Ɓ?_o0l0c5۠5k8f7vk[' ^uPvm UװVZṁzd1[~r-u]ʗClhQǰ:֎ru걔|rZqm7eg/JLBL%(@ENR($\Ĥ%i.tD *8I ! pYŨ).AͪiTzֳ^6;v|f{gvNfnE݋E{ =ޜW휭ǎn;0C|G~9#c+XZڰNYS{u޷$޸o*[ozFc>.1(޸vmXC\7[osc mI6 "5@lh^~~}yGWhlm^A춲҈{ <'8^3yu]mmW}}rxmaהּط{YP E$ȼT ⽴֬ FZWSϵ_O[ȯu~nz/V9/5Nݶz7׊*s߼v>+\[wVf# 5r~Rk.с~ i,`ϑ!l0.;gP?njmǰh?6m~%'=,2é{oyh;|ukWK0h5zNƯ-q׸kʹ׳gsk:`-eyd!^YκOt}׿7eo^kSk{yki+96צkK__֬k~T8fzNcіy9^0Z̳Z +;|CU.8jo*NZ37Jk%P[; 4poQ@xs.!p7W.7 \bUˀoa!p@\ }.>@{ϯ׷+7W3y׾z~wŽpvzyl_W͞]J̹c֣]tt:j z_<E?>cqʎ뱿Okfe?"{k/sg?ڡg޺f]#{^>|RkuO$ǧJ67]3ߝ9i?w_}o6I~~_ώ9VÉ=#m3 ?{]W6Gׇ=ÿ_NaS}/?qz'stkX!^B.A; Vc^ ZvmR@{ӟ?LSanm޸}=N8/ŷ˧1mYkk[Dru텭|7RS@׿L2{|C?*n)Wx_Ʀ[ili YQ0n`n|ad^C_kZ/o_oH~y薿޽y5],~u}Z#yV0ި~\\^sl5n=+@_-@(~мxz=4ǺV#au^{iV;xPW{x^o9Vجz.x47r\_v}VZ_zɎE%vټu_Jnd5"k>Z_K }1sVw{S]*_gYmUMֶuHż}/\DzVێó}(w"Ǟ梬w]e\m}mVKMZRs2 g׶ˍٝ+tbUZP-O:6V,k^l[7k@?Fpm, kQym]*X|Ӷv(tvc~y==PЀAlެl /5,,Yo[o.o:UunVbŞkqS}h~&o>/ojRQoMk:-cQq㜃`!mwpQ_|o>T*U~? 85sj{Q<#3P^T}Gɍ#-yM~ݟp텾f~ט֣`qYUGn7cw..iiˉnhv}֫H#h^1*ίֽWp\4nǦnWb_uxva\o x77ُ}sLG=W~Uױ۵a#k~Bn'T_Zo&8|x8ӿaoo.6777bjosc mUߔQmor?ף~{e73֞kָ~e:<WuQD}]k_GkF]Sן+dMe͚9Zqm}Nxm}=7`#_^O7~,s_־ZZWۏ;s]jm7U}t;նqY؏z_C}[s-{m׮Xs$rqZ~W;;NfMu~_"s-ZskY/x~@rtCZ- sd> Ym~1t}DZwD1.Om__ Ϸہ~e/ĶkJ뽠zw>6qޛO˛q˲ ŕ7_ENM\jc 7}7 !{7E5@ nԺ{D/wv GS)TXe{tomہUV|5TzfŽ]EXV~}:RX5Uu^"0kլ^gٜsu-rNWY 8Ey5"'1Q6oXo߶8?!ͻ'7;Ezff\~u^/1;Γ5l=Uˢ_jjUıBk^ת\zc[הl~ mLGVԆuUyp }`Wox8:!ԃd$w/8r_ڇ. >4W5{N>?Ep }E"7e9H5{ 8zor0~=ٽl>Fb*>1@\ }.>@ D"t>>tz޵～yg/γulg@Wn/p{tıyogdi^GOwяy2Xz~tڱYvǏ'Hޚ~˼?uvh:YWȞנ4Zkݓ$齒M1wgN7D{ߟ^Nܯ??q?^$?W?ku?/kgN1K6C@LAoڳ/r3]YǰM{\3~XGl:)>;:wN}X,Cn/!~ V ֝1Z;6C y{x&B7Զzno>{mo֘? ":T>GVț{ @ߊkVo9Qnl xBͳ;՗o֖=l%a@[`{|C?*n)Wx_Ʀ[ili YQ0n`n|ad^C_kZ/wt[FC_{~콺 Qyo+o7ŵ[)T~\\^s7lKp Q:o?po >Y뽬#-Ok8]k#!]5޻׷Z᤻X4qw]U({mQ}^khb=w|_˼׫v߾,} K]jf5uGۨYsa\żϦ q|f:o<3iڃ9}s>w^|}gY^(kXsp7N?vAs%\málo}߂n;R{N39,_ I|ܽC YoaJ қq|8IcFRߩ}74p>vp |븘ٵOg15f'G/;r*2W`96>R͟Yq) 5fg^n߈k5){y5q:yTv}֜W k{^ mk[?_ڟ>7;-|FS#粟z,jN͚rƽ{%_[/جYۇyk_&qZ5\μ_Ks\.~ 9yMxNgڛֹ9 =Dubl.Pmg!C {سϫ wb\++>p"ZgMw'Y{kDe%il.+ü='tU-=uS1WZcއx66uP#]kxrxOv5ج4֫w{'ztq##cլ9n1.k=z%쿽Vmpn}횹}7dchZ{H6k@?yP2t6ivlg{o R{_uS#k0ګM>pnokb=z|_)o_3ޤ7[m%Xˆz%@QG취kW''{|Cߵ#;t^f+ pmcmZpTx6>;ڋƕ 焢}a]meckʱlAVgy.6sf~=VFMj}pԼg01[1TjfqqvYsrRE';^cIj7^/0pMއWWWmkV{^^sܭ6s 6otGueB`Rv}LڡٍaۇmSZCC?]'Zϕo77sƛV =gsZ\l |Hjow@M|8_o4j}zkZӹn[ pjdۡ>mwU71mw{z2nq Bzw͋y|nns}o^땵Aw]7?t=G1^ͺ.V,c9qMxnۮY{uзެus^:kc=/oӽva\o x77ُ}sLG=W~Uױ۵a#k~Bn'T_Zo\s͠x?S]qäkz>*(̹ȱ$Pr?ף~{e¤>*}E"0N*܅}U\ }.>@ D"wv}uefzzf=^OqtUowx}| ~xGOwяy2Xz~tڱYvǏ'Hޚ~˼?uvh:Y۞me9l{>O7>'U4 kq' 9ؾ?w_}o6I~~_ώᬲH7ӕux@po軀T>aiP?KIvXnb!H6Bďq<ԇlj빽qVθo·CAl[J~o}tX!儔CS@׿LBj]0nۮv۱uj-\_NK,KoL.(B{.K.$: ԭ#y:D7$7Kq]sy {8G[뼿a%7yBw=\=kS{.v=㺎goiXƮw׍'[VkXsywāc4W] sf ێ 8f|s= k}Pk@?tYZTd}b0߹Ű):[*zC"J.m e ԾcHs)Dak@::sjߟb?ko߉qjnGZykzmԏE 75KsCW.~^e-xioOӣ6l,<}M?h<'} w~';;>,WA}x`8 W2o- ~PF}[@ Bu_kl#:ߡTozTd\k9|uz|=z d`6R! zD[ǭ*땤FDFmnLߢ]@gӋ }׎|y(E\u'9:.q͆@ߩoW&%9&)Xk-|<~d>0Og ;]8}>pXǶ(W}>?8CP~0%|lƛCa~4bgM}S_' R| a;0rj2JQc+15>zEe @?wTg]g&cx@^_+|g_,0>ܽj^x#Ƈ }sLG=kgvۃm̾oVPs:1,cӿm W%.3_5"t3wP]VX_C<l^\B[Հmc},}gӯG>Ng=!P43 BmEl{y{'Cj}u\b~Ǖc㵱uk޹^+c}gwOvo^Gra Gߝn}<zϧ_ oa~ ܾ~7;9IV k-AblX@Bޖv&Z5:`%pbc=lOo[=:1 z=R}Z^ZߺVaxgWcǣ^zXBʆs_?Al@v[h;E-ݷ}^*j붍?kӞ(ֺ;Sցm\@e b,hoy\CTº~o:؂%0 Bo^|kcB}0^'TK=|ʶ{#`N;ӿ?=.~7f?s>8G5?4{٣?\މ tp/²FMSZp-kqsζͿܚ<mZj[cԡ)>\?>AJ2E?Zֱ|~A=>;|Cؕ7Տc|_L46"}c}hy{Ň?`PYoׯu@a~}lc t>إ^`ֲ}ֽǁ9]}bA-yQV-{<)o>/@>.F=a>8_Bd;2uVS};$?, ?qqkpoK) EB5awW#lѶ-a} {zFZ{!a~Cm G(Vir}]1X-ziM$Ѿ?PԶw>>"ܯOP_??%v^*C|[]!ySߴ`ǾAiXa k n1DkVcpnA5O,6RuH珆&baܦ}QYk'imqc߭1|-ui1ǚ^dZ55l}H'Co"DA?kkt/X=7 D u;o}[Y~7$;!h-Ko G=צUژV󰮫:XXv-j{"0׽ϯ s8cn}[^u]cG{~@3Zs'@@h.v;n3GѸl7xK߀C2+{6iúr2t[SzS֚Ik9Qx[cEu8߷1:'¾nC@#G<(O?Z{wݷAv=0S-dsA`-녦t {5 Feb=}'5) {ǵ~_7;0t+6)Tߛt~k?X~; ӣ=~dP>/CS!:&]s@o sn[KgZ:M֚Zq hSVCu[P5֏1^ힲROYlnÞsz@t3؛jn۞|=߼?T.tes o.I;?{KοYxu!{)Vߚt;~;S}!WYowOзP;5@2;~Lyc~I@֫PO}k X{^ZڷpZG!4\٫+CHD_OO6 V ,@~qsNίq@K_jkC`8)D{.$A]+uFBO-]g Vq9F+| !l\}aGeq\SeX}יּު_,קUv֫5nB?C{,*ַcx>?7ܱӮw0{k;kB}|d ӻ"냯?rz۪o[䐗Z =|cd! #;!zR.x}1ڥR`*]xi5|Zߴ/'됀XˬuZ~NZ/kZ.m6<Ҏ_LUyk@?!p^Vop~,Mzn+3kԟ|G>^q ?_S Tp5ssұ9:y9k\M41V輮1׷\wuk]̿wӵݵ8[wˣoM:w/fP[ j~~$L<^V5B2ԭK_sNGժk8jOZ:?&˼isdoYܐqFK5ѯ~oߧsҜ=}Ⱦz-w@X^!`:2 ׭dO>@8Q UP벆?Ϛc^'D5[ڼ\{Io)[@ȟ=~4]+̊eAZ/XLt=D,m o;ceP>کY@yK|0Cj@v_׹ANA}kmSaκgR=҇VbBy_[we)@T oe/)C2F?(} 9>0Džؓ2.ԅk }8'$/TCoү/jab|u>dqxuWxUtpP4!e#8r.ͮL>C,<-C2ݹ>ht:׿pn7/0uyWӼ|*eFػPu5u]އ yvr ,pז!o[87WǭGv&]wh@_/Ehq|0wNJ/7?rzVcJ `4mXYcV[}hHtK{2S 3!` ּkf=O7em0Z)n^ǩ4&]-YA^3yl5L yq]հ|[6\F%Zow@ki=^[ڮ?z:J}"t?ept{E|N8;] }@A\4Nn׃@l+ ]>p6ݣ?Gߚl>r _n3v@v p3Go1+B>zz/'c1F pVB.?{z9^J팅O>fzsO|{yW,B\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂hOdx_?0wJ|]>j!K@m%^Ծu;ͱ{?Am_c~-` e>-i 2wAqS-8/ ottcԣ6.'} Sq^cVܗ|}d߁l[^w8}<<<<<<<<<?^|fC׋6z=]?7¾ys'r7#::^]sO_O{9GძfO:ca@ 7CO}l'L~ήpwK@9[!|oot~zn+3ofjB-uu==18&嘛ܦ6;>zh˲]Gk X0V_^9luq=`%>v:ȑ{~LFGV33t"c@?2z%.S@,~Q敎7"}pA?Cth/jK}y~oV{ol]vs}m:sEҷ^Zȍ@nv &Y,o RyvZTB(;CެGe\:^MϙH}qksہյvn1YcߪV&=}7+e/)RF?(PwG~+܍yV@___@A%З#ǚf<П5]>[#rIlLBIa@3 ݹ>=pZy= fm*qee!C/q5ec苲fε9ϊ#7ށcxځ;cF_k=c+5Z-ĸ qǫcka Q{82}v*c@2MZ"0[}8 {_NMAa|ʅ7fvG}g1Q>@ea)c/ʱu+LK}m>@-uϻ~с>s~}~}~}~}~}~}~}~}~M_N_>3t{{a_޼PzLWyps g9:>7G|1e1r>|p5)Bg@9f rzNO?F<CM_O/q\o}{o!_.B񳗯뵺a[|m~KmVC\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂hzn^阯|?{y#BTc;Dׁb DK؟ڗm]֥>n>;Q?W~,}5x f@ k.hBP :ӻ3q*(ΟQfw`Yʸt3?l }kZ}cUdM{aoWBW._S2h ~PϿƱ#UV.ÁJ/CsG5 51.qx?k軾}8G>٘Tb- îL1,gais}[ [BGv+!!xfYʸj\sY zy fX*!٪smsγb-{c9ͱw/Bx؇"hA8^vXmcaoTr^R=c+5Z-ĸ qǫcka Q{82}v*c@2MZ"0[}8 [|=}E0zPT ΋yBr3]ǣ3]I 0~wi}Xf{m&6_w ۖ׺]?N@q?>pp?>pp?>pp?>pp?>pp?>pp?>pp?>pp?EϦn/f^Oύsp/o^}o+܉༎CdzWWC?Kg?Θst9>n3v@v p3ƱC1F pVB.?{z9^J}\<6!{֍yǹ-5q.<6wC+\:r\Ě-}q]ǵz*aïos,әG c6j=r^ߏ{1,a+`t9fA uEp=ʷOyc.k{\P5]ڋ%}/aj_۷w\_[GN\=~0r%5<I5 Ae蛂@GSAqE%0 <zTƥԗf[_?P] mƘ5vj%k"#p~CpY",AkhouLg;[%o182k(24wX]׌Ƹہ؇sk{D#)IH5)֒@9tR~fp;ׇՠ;/cGWN<}Me\\5,Dv=l,}Qlչ9Y^~;!RY/R;w\p}@B,~@Cm\uj!%whǘ 8^[{ a~#_Sߏ3Ԗh52p^)x>AqS-8/ ottcԣ6.'} Sq^cVܗ|}d߁l[^w8}<Ǐ`xw{wWO@`a"$nAw7vǟqaM0Xhy52"{~ٻM'>Ji|#\G Lls'ϟW :8İW2Yݰtr?u?$\>PK{GWj_#je?pl@jJF> oխ>ְS"p?5wo׏<~>u:OXɶ~Xcls{F5o}=ۭ}@k!n5@amOx(k- rW"r?2GߚãoOy6{G/=Tp3H{ߗ{e0ƫ ׽k 1lJ?q2}])̻<-{&J^jvA;~٣?\_=>OY7a@?O-6Ǎ[2Pcנ!vr:.%P7=}||އߩ}!g][椄qpk>o-=>W~8?}޹1/3/.`Md+O[KXxLQf@9|)HwB=bZ9kRS>Ѿmau^g.>p~Ǐp}z 'ѷ<=k>v됀tf ZTͪuk3gWόczq3F98@ؗ7/7ӕ|Dߚpޑ| sJƻ/5З7-q=Я}v\\>M/d]<>7?Qj-ۮi޵9_s1p;YιeczN/xhu ut58R@/w?fz? fnb9\M7x;@Gx K8Q N/v47CxR_rǥЯVeG9-0UCMc\ߙd?lse[r~sk!kz.6bVQzpv p3Gʓcnzz}z#'Зߞw`OM_}k}4~:GX~:ךv@ZI蟂?9-6GeyVx@ VB.?{z9^Jf6!{֍yǹ-5q.<6wC+\:r\Ě-}q]ǵz*aïos,әG c6j=r^#o+M.?+Xu}+ lmw).1ר>?:po#ۛGX+5C#g|.ށ~ eyJ!]6Y-p]\ow+,wJ|]>j!K@m%^ԾpasmMF=?:UcҘO v~%t 9EX*|:v(?J wcpe8>PP ehȱf!&%gq}w=GR|<>:V!i9ym-["$'7c s5П 92",-}w{A󨏧w>zzu@zjia@/˟ƕ=ǝ`/ʚ:6<+ֲ7ӫߏwGxG`?gh}8WV]Z-W)Y^{\_ uc sxvRPe Rk<ٸ5z}sZ<ѻ/d{"}X*V4EXj u[|m}n+m弤/5z847PWk]/k,Zq16*Wk>+~sAp/eTyw{Vy2j@C` k1l zڥ`,ۂ_^dz-cJsj(=˵};j4k`8\/l7~_=(nżr!Muу5WIDAT@uzDY@;؋r,y~6R_;mk1Wﾱo݇N0&^ޗ@n@~|oc{ƞ @?"c9=xo kXki/4@p8r?1GߚwN {vhUҶn3v@v p3;}9ɗ; g q7y~=q‡& A^^R;[Bӏw7:=W)B\W3s[j]ymk3z}V,u世5[㺎kUÆ_Xc3otmzdk=<3}N'R9cX,ΫW2< r<)@͂h>}7te !x f@{V[%O~cR }kɟ+wz؏FnvS5`f! }SzΣbz/'q=? e-*!pe!Xo֣2.d/dĸ5@_ZhV_7oU+Y>sՇ2a ZC{V.ÁJ/CsG5 51.qx?k軾}8G>٘Tb- îL1,gais}[ L[>bz˺zjia@/˟ƕ=ǝ`/ʚ:6<+ֲ7ӫߏ{"}X*V4EXj u=~u!_j?pho6׺^Xj;4c?lU?׆}VD_@?۩Goj4k`8\/l^RT= }ZǨGm\NA N(ǒ׭7k3/Ⱦٶ=!pZG΍x{x{x{x{x{x{x{x{x.:6]u;~87mz~n}yB=}3]NnGut58Rj@/s1]·y +nzz}z#8}+!Mv ~Ώ|_mvCӷ^mhSg>X7ĝs۔fgCpYq-ku˫8 ο1ṉNgr92tϏڨQ{xfzN~?xsǰXWdxx hS2uU}1=$ٻ8dW:^ވU3uXj-}[q~yWuuTKz=k #7^})_zd ZT)HCAL７wZTB(;CެGe\:^MϙH}qksہյvn1YcߪV&=}7+e/)RF?(@_#z(~n@::pn܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏C@;܏Cpсg1tnoos ˛y_J>w"ws?8nՕ:PO|[[s.W<"tNn y`{@yl'g q7y~=q‡& A^^R;XjKJ-uu==18&嘛ܦ6;>zh˲]Gk X0V_^9luq=`%>v:ȑ{~LFGV33t"c@?2z%.S@,~8wJ|]>j!K@m%^Ծ-k̯e,p ՇR:_/GՃZp^+T= }ZǨGm\NA N(ǒ׭7k3/Ⱦٶ=!pZG΍x{x{x{x K<|C_>yLޏ9[W~d0=5LP{Ϧѷ_<^  G4_a@~  O}ky? NQzzow޳ݩۛ:t:UX1íhzuy*{Rد~_QKWۃ^?F~ה=j~2滹7?ֱ'_LoYfO>fzi;[h{\o>6uOTbz=p2O\QX2ּN0{{)"gG\5{_= ,aCRV?_鋛lA\ tPg@}oC_YK/yl ~הj~]oMMݛn#!‰X!pq\v^@w~ 9s[]ؒ=x{@GS9f|~~xor7eb!8\.q&^JMJ?ߕ7S\5t~SSsy f{>6Tyj=*V},ڮEɵG־o(A_ڜּ1wm 񜑱~Bj/>;Q:0qL?e^kMu׮r׺seu:aY8yu6za\m~xx~km9{.Kc~Zq]|WVMӺZϕk}&[TիY6&ѵ^ȱg5WǺS[oM޾趎\&T|k C4w?ؿ/]~W'xWvq<̻2fٮ#_*꾴ݟ^Ͷ5Z}:_mb?Ʈ9f/k\[rPm{!`qyno[u/̎]ĵSSW9߆g.qoem1-✁y9jtdkt׭sLcgt,T3{5qf5!ֹԾ{<ܯ͵3Uk[waͭ99l=v{Azڞ|m'wcP W2QE RC!$ YAM_?7fBТ=nmM5)C|iK>Dަz:߹/yMxu±|5n7DxZޮu\k{}֤L yW[^U)DLs5cZ^PSw:Dztҵi~h-uo3KiܕP~u^xZcݗKǾuqV3?79C^ox<l^7CSog5YB,po¯ ,ƛ AHweT~n\珨\["e:בk9WO6zcpmݝW`FsYfev1l5Ȯ+uIeY-6ʱS3qڼm1A~:5s}zeǛmsͺ}km9o>#{_) kd.ýJӼ7"heoƻśy(ȸ1.%q Ezm_Xcj7}}׾;rת5^kt+\v;k0Xc ڢl :9N_o^G7 XY]U?텁+uu4O:k]QڲkcG;R>s}zبSsf珴\>[NX{Awu=\f[X6}T f> vx¢X( ׬Av7v䃡lrdp6^HUoa.z?VGh92T* lYTkrаv`^HŹmγ7ͫ?W76s^>Y>[ 1+/kuKx9cj&:|L؟'g|NV3vxow863mԧ;9ֿRy?획 h}nus(BYFyq??~FP'>ѧU'1_?yhaj-m;anZQ51Wǚnǘ[0vdy}xN^{^Ǜj+ t]Q*1/Gg{AؾfV=:cIe/li&yy~q~}97nz4$Y?'[}Q} lLWR:p=sXYi5N1^ck6ȷپ 5П-AN|C<Ȃf(vpD=*o*^yaZ77,x WAU=r:!er1 vD}4 n?GgW5u\?z;W~;dm#X_;sױ\ `oo:{niZ빙Q*dskߘkv~Ѯýܖqu3e\%<X~T1.\c)v3ߟ:7{teC-cZklaFn^b-ks--tk ڼ?VқWZg^p1>qV#dϕmcֵIo?^[w=s͵lUjkfkamw?ckjc8nեk>blo3VCƇ"u!h(СPy(AF~^<}Kp!֊Sc1C`lժvBm5é+珮T8fb\c]qsG->׬umxVb󶊚[c{?{M)k7vFZߒ/[?K7 ^3@zSĸD~& Ĝ\kT͚Ho^-&f%e휬fq-%_|r}2Źf\~m6{yEBw^+5q6;s~}=L_ItMtmX5_@\k^A^W(zyhY3YGmj} 7jskEmqVwb/ל^cGkkjk#kK1^'5 =П"uWo "5p DxSFoB8KCtV5`uՈNnpz4Z{+,UL_7vvΦy9ܖ{6Cg.K*'p z`L_ >,v(xJ!$opwR!a;@p 8 ;@e˜@{ _x?} `G{'N#8wсg1zu^M7ֱnYǰ^ ޭGx轺2ι[ݛakѳO{~XθQ_OO߶Oy:6{ϿXǪ-pϻstu>}Z^\ k^=pr^K>|3OsR&{O[~.mfݼwxl@GF{Q9K o_Ooo+ OPqlwUrlB=r7Λg7O臶O-W3I7w/.q`!Qy2D>f['gN çrl{ }\C!I n˽Ѷz}lA\tPg@}oC_YK/yl~c~הj~, A7=wP7ߑ eD{:g@%-soh6?Q{Ty^>Pk =зB.6D .|R;Mg |lm_ X䜲oZ(kvX0S0̏rS비VcQv=/(OC%W@'65/w̝[BF>y1^ZqNƞX/_d}g\k;_ǕUӴsZ'jֿ&~vjnEz!Ǟ\ o`ۿ&737oi C=ؿ/չ&c(yeü+c:eKyl{\Yoɵmy9V}%lljch;쿠؟s ,t׹~Is;fBҵ3㮛|۵kmϚYa7Ws|~ ^K*6i.F~LzYjNX}WY6]~ÜB,w?^נ5K_ML?G|~bNej`ks>׼+{鸼Fϣ>vͶQ[z}f~cz~C_ o\_Y}qqoP zҘ% mY "aCxk KҜq\yWqvצsg>Ss yq},^?Vk^05~xo3KiܕP~u^xZcݗKǾuqV3?79C^ox<l|_p}+8ɂdA txs!po93"!aeXQhS͝›K9=r=zR~#]=FsbmI@K+ǡڪͻ;"΋׉Z˼b>W5̟kƑ]W\˲>[#mc;f2רݎ9: =֦(NSכב{k>1F=#}=Vf:d}mc(Oz{aJzZq1$nغԴ\m:6gTܮ#ml;]_c9ipmc)4Qf=@wvބ635kP]xსlrdp6^HUoa.z?VGh92T* lYTkrаv`^HŹmγ7ͫ?W76s^>Y>[ 1+/kuKx9cj&:|L؟ vjf^QWv}%GX{#o?W{>kv[ʺ5Qfj{'j@9i% Hh۝o K^<yaZ77,x WAU=r:!er1 vD}4 n?GgW5u\?z;W~;dm#X_;sױ\ `oo:{niZ빙Q*dskߘkv~Ѯý;js[yΔRskٯuBjx %} cLV0enSf5μqHݰeLk9>cc1Hc)Zk[[5µy97ֵΦ5jc5}=ƛ2Q;G<ɞ+*Pko/ [jߎie_O1z4gk~ lHz^1r׮[p=ܪK9|,x=\ـkKp}0C. E\@1:,›F7Bn$mӷ1a,ax>9V3 V:`'F_3-]:ksbmzIxm=H 볌7/1vxC#{Fyk 薠tV׶Yy?t=<5꽗3;Gp}wlfkdz]U_o| ~MQ^/5kJq_ykjX}YR(~ob7j(b83(Yo7wh:7]/Yk:Y3+edl{],>չvEj z/.t9ּBgQۿ8k5њ8f88:Tko->N5n^Я9ƪYyiʏG3:bbOkkXTCTxº7BǣHx")#7j!%CRcI:NclM~}ܙۺ~r"0J "O6Gi*aEy{|nZkkY ̫\Fk=^Xjf.d6ۨq?c^wn}},zi콚YHs c}וkFǒ-r~wpPD:v56&zzԺȚ׻ȘV gj&:6~@C|*|ư'l[v>9y T/lYYv5iөY֎;s=;s^nemq}hS%߾9}j56.½ډ>]վ_X~oʚF4/'vzi^Ŝ9ul>[o_>utϻPu[efl>y9RSEY//[kyͶ_.իY:ľ ::eCX)CK~t/ǃ^@O) n^_R<={'v>w@sC{'R]oor-r7E+K 7N#8Gp4;@hw>x?} `G.:6]u;~8QWUët:v7W3vvya3ܻWW9wk{3ux-7zv9jvy>ֱtz7;Yfo1X7|1eޗvy׾w:[|磏XK!{_ckNqݻyׇow?q]|d϶iŞs,No}MSy_bxzz}{3]YvgڀΖ@^}XdG~הk>||^7w!o؟2m&Թ[>دg텓o^8])CX:e|֘_S$!$3:-FFr?[WÖssB}AsWK[B5娚b?o@B78vY+6 \/!7u΀tK[O9:jʑ5spJeN8‡p !ku[`H~S%D }0@\^ÂzMa7CLZ^0JF}r.5"@'65/w̝CxƾLa}:^.u\ZwCg=סa5z=5]F;k=^M_߼Yo8'ceui/^˾3뵝ʪiZWr}_j?z5kfa#Z/d{YձstvZFv oso[gzMutB2`w?ؿ/kO"7cbKOy/6vA?gcٿWu-_ٶBtcux>psǛ+V c#q>*UŷEK۲}YFzLqy8gྯGo^j;ֵa9pvw=ƚӸqU3Yn{]t^j=[ZsΎƻVaCYqk彠Af-ڙ={'j@?2xz%CLP/ q:?Bp(c!hAR6?%m?"oS=_ܗ&[xz"<-ڎyXoֺ^>kR{xk8ӛW+9}~U>k/|_kB4|Q#?Ju 5u}~,KM羫,]VaNZk;ˍOsQkkК%/릏Ukv>?1o5O~0zZk=\_t\^jQgf-z>[J>_}x<ֶ~3;\]oqo) _szeCoT4xoS=KfUe\KX$Ϗ˪8fkXwD}) Rs yq},^?Vk^05$cyk_#\ bn Y{ [?kǺ̳\X~]mY/Y\}\=/]:9KZn+,4k쳼v_Y?Sz] Y5}t_g5ss1TڏpKGß+jazZ>^wv ChA .ɂ.zCvzSvMC $̻ 2*PB@?OQ7w GP(aN\2jȵvkhќ'keR=q1jΫ0kkubc2owU gzDqd$ײϖB؎8\pm޶WV^? sUͶf˾׵Q`KqY^ZYCMmoqeiޛB4̲7ƛ<}dq8nO>kk,[cj7c8%_]l{}z%rknwmkS^{^PA[_')Ƚ5oX枑{Aؾ+?k˶1'0vRFIg 8v\[vM7wcl}g]jgOն\b3~snw^ _r }ͯ{GͶm^Oh# vx¢L$cHh/\%c-Ovt+/rrdp6^ކ0~۵L!j=#*yWms[X_ZhX^;0Z$z\׶k˛Ս9/]5ʺ~[y15w[>U3E>'qoxxx<^S;j6ӝW\}wzM)뼟vZ|u[S;߬KEz̲77kg!a /C1.1} ON9cfÚ3ZH5vG%em{kb l{5ݎ1|-+{a`7Vқ37B麢.Ub^|4}=Zͬzfu(+P]^vwM*p߯.-r nhIN'J}*{خtHk/{jϱ؇rakRYb>׬%]wZ^wv gK&[ݹ!aO[&Htύ{c6"Z/JC=[NYg#aТl[>&6wڎ请Ϛ@9OW#3썫y|? }{.ȟkm=ם׫{|TogǭGvdU31.q?ckϹ| ;Rs-:5qߖ;ިuҞs3>yU4֬1]{5ksl6e\LY e=W *a?|_ b\"'0dSVf?un\3 [ƴت}&&bhǟ׼Xzms6cB6O=gպٴ\FmL&gxUƲ}=ju?se[u텁{kR1O:ףlyk8)]Ɯzls-[}گ횙Zi]+X|N[u)皏5kS!wuX v4!H7ě(֛E3n$m3d[ku<>YYڳPdo1.цu!o&X8+sM5R5k"yza fIY;'YGK}sq6}mymsټ6"\} yٴY!LV3C#9'5 ӵa}^\~u_msryy]Zףqkף5gqgqqu5[|.ϭ= jݺT[a߉_s"{U۳ZOgXu,xŞ׬-LU =П"uWo "5p 7#͉ƌ&Ʒ%,IԘbhSADclM~}ܙۺ~r"0AJ "O6Gi*aEy{|nZkkY ̫\Fk=^Xjf.d6ۨq?c^wn}},zi콚YHs c}וkFǒ-r~wpPD:v5 kJ}uj]dlxL^x~ɯͲ7볟ǎb7˟׭7p<&dc-s+"=ۅ JPŐ%1 "޼9kβ&Ͷ{:5+5Ts~1s.vsskm-mBjEZͬcpFv,pkcף9/Gj=[Affs6綴ݻb=ZU)lPǖ(6ck\G[ Uu_j6Qi{Gc+z9W^򲵖=с{lk+=Y_e5&p}\`p~ 1?w>-+е9&z@п \y"'S俒^@_x?} `G{'N#8Gp4;@{ϯ׷+7W3y׾z~g/_%Ǿ^^}wgM i]M7Ϗg9d}S?Ko+Nv{ӿ.OVv?\/NZ;6+?~[~]?.mW=S5kc)~[kycsKu>!!oO'NVo?~v KM8UplׄS !k}jM}V xp}p2, u >]+ A:?ߛ\7gҼzojۅ빽qVڸy~gg~>Pլ"_y]Z+8i0 { @ߊkVo9xB  wqZW PC Lvث@жGMq[(qvm&[.t>Bj]0nց~Sŭzk)/1Kյ'待{3]oܽ~yk7t7]xjƞu< @ +"t܏zˋxNw4/a8G[뼁R#WKh!/o7?7?}Q~s?X 97зmǿ}}=k z+5ǖϋq>q׮[Y3y3Yж^'ϜZF~.v]_k4w\YgfYVz cnQ'u6}@X `{D>_mGKm+晄q}k-u8tYZ (7Yp_|EXyMq5^ϖkbM10Ŭr~J!kի|^v9^^}fmzχyZ\jrZ˶soW\e,e?p:0j@"t_Wc?5wyf C2os1z:rHb`;[Zu=1y}܁~|c޽/BF&<'7NXa@vN ?q:n16})a{6y}o[N kE{𠤀. TBEM.2^;6g\5pY׫oV7umַ;^Wמxdr]],Y'" hjGY?Y;߇?w>vrWsט諾Ni0v wY|oYd~@o>X68Bk`m]*km;~|Hc c#>o^Ԩ7F|48 fe0[m%X5JRf6jpcn8:Ϧ?\Eün}=nd y׏kLaYqn3 MR{x&xMYʼb#5 llصvzESg=\vn~ֺ}nugG5{'lST#$5fn*[ߜk|ky[c`z*p:kmҷxRZn1'u5\oķ:0:7{j+1bϵ1C:G}7pGKq?6~P皯%cC^+쿦tYMbf#;`M9vt]lCv1ϣk__g|Y߷I{=ʽ ?r}P=޲jopJNn@@Y:[*anM ro7tDact6/Mʘkoܸjsޛz3\oķfy?IEd"| [0"ڪ P%g\4s>~=ǿ__qR~/,u~0uʶ{Z>(>sSBryMq׳kN; c%/rCۺ&8;5[>mrZm[b.XV{Zk=d9Y:GveB-9sֺ9;~Vֻ{OvZc)>j^l.kXN8с~ q,tϑq0.;' ogvT}]6=ul9X5࠾^6Fho]R^ލRUX-u7ӕ{Xq_7uNѷPKmlƱ2YsBo^|kcB}0BS}ԼNvm^Mj?;Oۛޓ|S~w|9v5]u5z֤_s֚rn&i:DM_~͚Y{eYh[b!kkg^/nl߮j! :7i}b9ϊ\Zk[aGM5^-Zec^-I8eGme opЍa?Pxhx?}\~|@!!8w> ?$N#8Gp4;@hw>x?} `Gzz}{;ݾzq3~u==w{8ھ Xwb{kVw*N_>3dzu w3]Yi=j;;שǰIkYݻHu׃Ӿ>mgr1;0_SHwr /߹cV^ϗ rԩou^{ݻF'x >{~.gר^+ʻ5iZvKzV6v lSܫf'߼6/gUo=]㏟L?ǟ_>vlV8/ұ2oڮ{k֦VSI5FǮl]?T4m|rGhMߟ^Nܯ??q?^$?W?ku?/kgN?@lGЉ=o&ܡfCjj7ays9˛;tnu۶ew4dp O=|7G su?nsy^ k{jz|#Ͽ9M.t*?k~8g\gYewwo=П'vz?Ͻ}>|T췑J)kOhx2o_SN5G6>|?m_jstƷ]τohg [\m~wrghfB ! m9Pܚ֮b/(jQ]{Rfvg߈W@_#yƿU1 Qz~\\^sËsj;oӍIƭu@_-@(~pqƵgr?vzo{{g>[U/{S!Yl̷)ǻycc똓VF"ww~˺gn6/:նY_q[^m{_sYYg=fC554Vzsc˽l^d} GGCqFQ{Xl*ƿilz=ךjf^kMmI5{i5Y3yZ敿nTuyUϊj^ImBY]rzj5z sO؊6jEi{IE_ȽǝkC=_e&ueýcs?r\# 9u[Ⱥ\Y윬]Ƹ#s/CZwt}oYGGlKXeXX>Ըp]v5/۳5%OyRv1so}'_LoZ"#*qޙ۴k=sˎUPk/t{WҺɹZ=&_zַ~~Ԯ]3ۡ5K5YrXFvGdG{>z=:t=߼&=֦^/B9v{+D o#kjZY[EVjk_Ƙv.Ym=lfl%7uY}_clQk^.zk֟W⎛czh}y`=T}uʚ~;5d='USqNdZ1_zj'u6}@X `{D>_mGKm+晄q}k-u8tYZ (7Ypt1?.k:vVo=j55E_Zۏ1¯vK7ֺϜ^s=!] ~6UBR,:V\Yg[xk~,Q52ZjW|/>.ϋի|sy#4I=/]8ZU*t_k<:y_+f\[#^_S|}&ǖͻ%<.j^[[<'KgꩭU ˱W o{'~H5Ic5LךθZznauz{i>mq{P ׶㹿t~Gx j"3oZ)k&du_rz"ϫqƏ3oysߢ}סty>yumB>-9Wk/I2׮YQgױ5c5+jOJ5hnos 5j5kkUZȜG='v{\r=j/,ۓ3ߖ8߮h˷oz,m狵)QqY)ETj#ocv:J:}gb>;l=ykӵiZNkWWW̱[׸:+C_>5y A 3kApP|{X??m'_9C~Oڬ5̯:\@ޏK ㋗xnjFׄD{\unz_'~y _z3^ϫ wb\++>p"Kc]7rY?@epjmY/VfR3I1^mݙ6^if=en]Z <+5Y [s>D6~5Cw!=n{ͱUUs/lS%׎x1̚N 1⚍sZq;K{SHk̍C]gۄ)(Vڲ]9޼>X)׍Cèʼl}֮ٛ6Kqs5gϫyWnx^_eV>5z&_?}U]e {'~(ڭۼ'.jbա}Ȇ,jRs6|sqMڗ_ӾG:.3&_^26G|>뿭uz5Vmk5sU4C2jU >uk+kY6G|-+>wqkfc!5k^3[5sPZA+uRV~~v:p}گmvr,7F]]-Q墬[ZQxk:|1mwcyv_ _Sz{7/{}u95kk=u gE/jXc!ǯ}WZ[?&qXu@^EP7x+~W+_(/ {A`c? \e_E/mQ$m:Ϧ?\ݡZz®n k]3oP߇2 Y||^39ؽWGkS/=Qyy0ƚ]y \2:=sm39 F=eqv'?o^O1kf˯z8#Xkչ}]I7ܛƹ͚㚥:,k},^_:sT+қWX}ͱlS\/+1Ty:Y3ߺ6*.UCg_eA8͵~mPk~>Ae YCwn1:?Vzk\]kb;1CѮ߬k[o֨;džUԤ*j1XOծMj[g־7nnz^BzVVu]گm}ͨo:[Y ^Xפ׀W\fR_؍?_ZJi}ܞnzSe qm} OUqLfEMkffr=''_7>o Y̯gX[׏{>oᛱw`[{\G)Wd;\cԣXOjZٞ׺q}̫~q]e$kqoYѶ^oUף[껼6f_kǝݗ[\.SolST#$5fn*[ߜk|ky[c`z*p:kmҷxRZn1'u%^H {/sfzv/E=W}AϏZ\4Ao/v͇Á] }+' ugez?[l4k{0vos)k{\5) Ξ -٩}065=ͬ}<_Vj {\-a^53k9gq1s}m]_k9u/gVn7;6kf/go}yիrMe yyݿnO*ەcͫ?nqg[_+MQ-{ΐh[R{5kFmͫp==Y;'e~6X]s~'w.ŽuLe;*3-yޱFLgf͚ZU_k>wj󽫞Cڗ:ucZV=rVYQ٪YkSQfJ\Mt?s~p:,uϹsܼ+ktz{]שd_gb˱ce=/{USkF[|5]^{ϊyY}d1| ۋ{>8:ׁ>R skbo~9#Ep=?XзǧT}}oWduV^c?UċѭK_ҋlx.^P͹ȱ? A/@lh^~~}IzC@!-~H XjW {0toǢF c5|[kњXmy,L}gcQO{fMp?^ocʞu3ͽ#qlO;|M涾+k^ޒksg;C^cX)|j356'U[fug5^S{#{!+o+b~o[f6ݦ5-uWƼjZEoh\qV;rg?_b˪VOzܫ?gyXOIFrɷo-7N6&~Wofp}Tjڵ_++j^:{aVν98?%KgT{zZW_+1k3{tz7M~;orl^>Y_tm7V͋1x{q;Z]}6ϗkv-Oo弊yj5k|jlpDwkׁ|i^>ԮƯפ$x?/pbjz;Gw=e-y4~?^X'^jqc/֡3/Ǐ10aFϩ캝lw gmk}j6PoѷOym>Nekkz*o|0Џ!92 epW?Al@BߎʷѶ˶tu_0kc]j{Kڻ1^J^(fy^k~~ Pu. yF&g__? vOi>3d9Cuٯ e{8y?mg9?k{T}F93Yft5 K1 b1nyܵZF1|d{-ׯpO{zi,=5^6Xؚ⵩0zFYW?PcϯsoϦ{sRrn.}k%JoY3Gk꿢ؿF?{AEk7{(Um[9M;m}_7y1Z)}7׾[>n׊5Z}m m}Pl-ܱ|n1s[k&/;QFKEMRߝqzd׎~͊w~%um^뉱tY&5l ܛ3zV/;7CZǔ[z5o_z7t߲ubmjreUk:ɺ\M2]޻Xvli~}] m=N)tGP䉘 I:- Bx7pW+qzĶghu{XyؔZOz)Gv~kAe.y8ew _=%չȶx883^k%Ϯ{ r6wOe{اX>o= 7?(@V0?q;\xOaȓ,8͔xs H0b(gFwg[}w1_A< I>T矶7bI@ s sv`t@q3O]@yD;{&>Z [>mB*^8>,y!z9|z.>C:;{! Ti3Yq[x?qߴWVRh߳m`& A08;p>A08;p>A08;p>w-迾10۟n÷_/vko sxvϏ%Oo9OyfuyW= {9?g{cg~[?| !w?^>q'g]'[z|7ߪ{uO/5^\Hz\ϝg>~Μd3mi7[,4g'>l J4;U8;&M;,u=7_py덝ΙuJ@lzEUl}Y΅eOsoPk uD% zÞ W?^ 5ED⺦|U93׬E}Grڳ=.>;r߄,)R&J铞7CD}Aӭf{r_d7~Y۞OP|O~ltx"ߴoA {ݸn)n}5Ͷoi]kqr{9]D,4'a|yď2j[>!=:t&cʹfxLⓧk2gX`)1,%T=WѪ'=VcʈQOާ~H¾k7EXA0Gb.1D2gƷ4j]1:]?+k\-?'/lm\sA=Pc_Ί_X΍;c}j?_x73odK^_x־s֍l:6ֺzl+}I{(gHf :+|ȹR>WEt{tDH`볨Y_6OՍ}_ڷQqx?XYWtD+˜9o~l~ɸoseL # ֑-/59 Yk-u5<֣Bv#/Kٕ>YǶiEηk͙z"v܃X}鳼~]cD{gBܷ_Uzk2f1+]b=̯qD=Dάߴ m{9[/B-E,UԫBz]gY$٨b%V+nk鵪^pm1kZS2u5Q zsR|]}a|_W[l}ۮϊpN }߭K0k[e5gu6崌uLKvssAΒLm1o^X^˾$_ίERl<`N/>\8\ m`_|v$A"@":xQś0(Yq[H%!8Yh"K9#x=՛KZZYCG7V'ͦݏ ZlEq2fT +X4߼ZJoчerb[BXguϖzd ۑyn=Zi%rc gʗ.sݼ۽[:q;w]k-Uߦyr6O_Ÿ0BÂ8Цq)C# |\5lmmnAקJmA\Vumk#_ U}֨o;ä9|^PB[Aq?~B#qgx/0ףAmmQ/L_ vdIbw|#-]s:3S0u!}-s/͏)u\q6}Ե%}16>s6OYN }_bѣ`N&l'8<:$ &qHmN[DHIsci,C Z݌⬍Ts{}ԣ{8vD\Q-*o6l?\~zHon  Ͽ}#3G!osyQs~}$"c DΜ{#QxYc{(Vgl0`$a{s_}9%ߣql1Ql6 &#z,r{o֤w~?c sp剮9"WڳQ&rl|}T&NO$5?^,ڎe 3 ^_{|(Wbݺ%=z-xyaM^b{X O}ڵ\o} ztbc^kx#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g3}?xtPwX`~1=…8/3o9R9 "-?' #e?g;Bњ0b'l8y}#lkZ^0⒞gf 㢯z|l3"g{!_TtNt\yQ>^\&l9^\WƠz}3KHף3sr!\9KC 5Glc9GZzzgWzb TDg'^Nxź/o ω"v\-ԳQ(hR){d!1)xFDؖ&1y[]fEk^I)ӲȡۋZWe-X}ѫNJ֜9̩3VD=(}9c~-f>Yܸ>:(gZ]v1fm̉͟|FȱʖOeA'bkY]sƟw|j![I,x>B1W,bmaږ_%"}EBЌsmbIAL C77Y]~fbZڶ"Vl}Bzr6 ND;zpݸe_Ǖj何5{ٜu3ŜMq2hmeOmLLexoPczQ;⓾kHש^*o/G9[u&T}[5+:&ٿya{OagVd\7&bQF7}`/}p_HQ91mA=Vts>Z I6 xŠ8;p>~}\W`?pv| Ӣ~?eAϺп \~{!oN}@8 N}@8 N}@8 N] /ykg [{<'@z<]=q1[gr޽cF^kOx?_>B/ݏOIٷA~Vyͷjb iF>^:׮sş_x3)lL[l;}qz9< ÂN?&y=7_py덝ΙuJ@lzEUl}~?ͽyCApO)gu$B {"`W/\U@p{jR+!ŋ"qɎ[W] zZlY &6dL갭tՙIϛ !QvΉ> G[Aqς=V9/g2?SNY}scO} P-߃uKt+[Lo}OP_sr~g'^N=> ''IX^'񣌷VEHϟ,xe|BVq>&>a&إoJ,Jc UU* v,1*i]CW~M+Q̅^\4FoK(sf|~BW^ynq?ۂ5染Eki.Ϲàs/gůKN]Fj㱏>z}?ϙ7]brzNS//<_k9Fe^Nk]kgMϤkjm9ȹS_.x%{ $ޡz:` X9_śou_-sT\b3|v +j{w<{}36r͏-O7Wmvϴ[ỞsZ=+#[_jl;rumкş7nzTȎS}de}I6=RR9>v]{mײ9}\Q\_ٮ[bY^ڮΊ{1"U3yun}z|[ksVx׸"goZq&H1N-o"M)*UA!,uP#g>B;' -ZH,yx:qg60e/Fԣucm;OB|V/ut<1peWzM񭐽KlmXcYOAzu{^`G%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>үw/cy̛9{ײ/ ׶k!E˾?*48g#zJ|v$A"@":H\ C̐JP"`*GΟ!3 ^55}tce}l8~hV0.#`FмOͫm}X/W(/(9~u^ZlG&۰#Y\"8J0 rF{|2wYͻݻگ^pW5r^Em(g<ު~ & }nC͇zqݯְqOx g[|onAקJmA\Vumk#U>rkԷaMH/p]>uO(:;j hyu{[[e $ &qFmN  4i~.s,6?~HA࢒B`4jnϜz}yg"Ю+EEmO)V͍?!co$b(mN56oucַ#9c|>5~B#q!r2޷];G:c nN~qs$q{ZY(>(6K=Asפw~?3?Ξ^_?+bxq=kum"zCk /mljM2?|= VeWS+1 n]gb{Ȟc{=z|a=:115OgeIF W:v^푴wՏႭ1qؼ5;_ SU)H#!4tElfqD==|; E\Fjy؄2+tX"lJA+6#ή7;lQuY7Np̽voɲ//-{.#}ksi(z!!}vs[?f Nzp0lx'jϫwm0`[7Nb{xe>! (կ5!Hy #X_3YBńs 'OrSsŵ 23[8<}㾸5bvw<禖Ql#lIGz!k(ZbS/]'7.,'l3-e{=9G\a^G0a{Tpoٿ2ۯ۽ZG۰F}9,g`?gn_3fl4hkk|Zu{pYƩQ|vlA_5I`"/ EACbT*ɮ6jq&{z=1f Wk򨅝V8%-֋ZqIGc}lF}ElY ȥ>x#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g[!gxל,ĴCv}P$(y"_̆7pke b[Tκ(lHlY9+}'I~3)2N^6`5Vĵ白ø 'Ħ^E?7ۃșƞz9Wks}Wz jWF1uv9>+qD~:wH'|+D}PavQw?([X/gN`ѨDFř|1l 6/XW%M9Q$nK~OQ(hR){d!1)xFDؖ&1y[]fEk^I)ӲȡۋZWe-X}ѫNJ֜9̩3VD=(}9c~-f>Yܸ>:(gZ]v1fm̉͟|FȱʖOeA'bkY]3.FC~5 sX|8oE`g@fmK_x܋$h JEdU4ZZ,*Wo/n,t},2mElSo,*?̋mZq/}[֣Rژ2fu/3nƶ).Bƶփճ̗⩭ jl]/avX|ҷ~Mx:+@(g ΄vX{C$?/l|~)a֊+=fW5l8 Â>/kච`A@ >hA/J_&Y$$oϋ| =AnWnu}_Acg'><-WkS7k !(_Iw&N~@8 N}|g'J˿s1ܱ/1A0xv?\b˿uO^:O5,g >!/1Qu{ r.K qz?gw\/ܢ.?{սq Ly?J\/xolr=oLdzOu/?"*ӡ#-Y~n `g67Vݴ[$,o`[E-SSb?{9MOЏ5^k?y^[ [  yS8}_C ykoio<nmgmv-}?˿?W^wc%GdMѯ9M cSж۰'kRL.}V *"zvEN*Q@Hkb!y(=Xk:_?}~&<^/Yѹ g@ϑ/sg~y7/~7B~~L֟3?6:O*O7xV\C{z!G Wx<ov<%MiKm_7'/Y"g9;o^]I>/Ƽ5sB'O._ *E3"UD*>=隍bXqtJ.X>*kգٸ??+j]Қ6={4~f\9Etj>\zcs][}a<A~c?m)r[޼?8=:OKC=* p5Nk)D⺦|'[W] zZJ㳓Q?/_'^=^UY(jb*lWEq6 O\/Dx}^ }tMjUOZXb0k3H&ޱ)z5Q7yI>yY~Q[zzrن[ͩ9k{Y3~{:&뿭&dL,Q$̈́t;QDA;c?,߳og0)g,}z?ɏ?O6-@~~-ҭܳo3}fksř9ۊNqX_p}?$­?9-4Yyb+ab^=D_=3cF$e=P`ۣE쮩"kk=j'VmT!tDV_ 9/f<qz^]k#d/k%_ԋh3+4;[>4N=> 0A:eն}BztĒ6g>a&إoJ Kc UU*c.ՂFJ}>CZW_S~(B/.S3N_V{u_G^gi}]ӱ-zYYj9m9yak_< >rVZ4unٯ6Wû}#%&D^:nd[ֱջfk]Q}L޻F9F5Xй^Eؐi8/t/Oyz"@$Y/|:eow3KXYWtD+˜9o~l~ɸoseL # ֑-/59 Yk-u5<֣Bv#/Kٕ>YǶiEηk͙z"v܃X}鳼~]cD{gBܷ_Uzk2f1+]b=̯qD=Dάߴ mXn-^!fa 1\p'1-D OG3'&𞁴E4'W؊=jhyڷR;m{=?eJnz]'soU\/v-l (^n=]{W}Mr2ۘΙ^!m9Lc1K%{}٣֓ -ep_.Z4Y YǫW϶HyB@ŎeE Iߚ%rPW[ ^]9MޯX*XrnO=aYYƛ^+5eN_/cq\z_A9({^QI{:'5Nu֧l=Pzt:뾚꿶UZsVnsZNXdk7.9m=,zvka^K¿Z,cqze/#ez8hWE`Ob.Ӌ?9ϺO""^ׂ^,ՀONBGDT.݂˟0 V^ƵtK(("E!QƩ~/m+SNXݮfq/%z/\׋Dt?Wx~Zs5ECٻ~G3vkN8˹@\hkQk;Εbkj.!r gBh D$.hA_C#d!$m"dyJgD -Wo.kMke }XY4v?N,hi=Qm44c|j)mE n= qb96?[6oGrkm}&Ο%*_bk]ֿwnngvM&W~~=<&Zb_ž% 0nCCD=h6 Op!lk8 D /w 2?S~{UE|]?gA*~Gu=g= Jh+:'o^Hs|d5 z4:-ٿ2 A8q+Z֮l9PozEkNG}F{*ۼ9}˜9e_1Yۻk^6æ/ƦGq$Z}U/"6Ɗ\bY!- bo O=!ļH$Hz]D,rmdI?ٶjn^h@}k,ս<[.\_3m_\zEU۸èW[,i폚6?^?X~7ӫF=U"5㾉}l_1>TN/a0'd{yMJvD p6'_ G o?9NB?EpQH!HxM5gN=ھ3hhNբfs6yᧇ+Ɵ[h}71zb6ϱzqۑymIe?G,G +2B̹7eo5_wޏbuܸ HL1ܧtlL,FX߬IA?c sp)ݏ1Ҟ5׌_6ck=_'/mljM2?|= VeWS+1 n]gb{Ȟc{=z|a=:115OgeIF W:v^푴wՏႭ1qؼ5;_mӎg' >?S˿J/Hq_/r$XeˎqU O}yl'"ļ!M97t|a}Us ǪeOk\zb/4JX.k^zzYSDK~Xq{G^/4Z?tux9dxlXgKב9?й~}{ԫYD>r$HE ]Q$ۦY&hXz_G![ˈU-o^fŕP[ͳ#Ri1h~>1f7.3!C5@=so][jKƁ{˞k}\7ʻ^E~G=;C9c~֏>&zӭ g'[4ɵ][p3+#Ϳol+ Ql_k.BA Faf  7O0nĵ 23[8+17[#f'͕97bfsM䔯= yO]lUFqz:qa9a{!o/a>⚵e|: ۣ{~=j?r߆4͹e?9s1c֧y&Fg^[{"ýX/_mc|vi/'HD.b$.5E$-<]zK}?9귾7lXz~ǴmD\7,ٷS!=GJ껨y>9G!rŚ$f0!DYQD"bJtHqw]#$m$L6W)ǡ ek*0zb0Q ;97~]qʊN^ZK:f7.gP@.Y|w;y_+3Zzs(=E(]~ٵil=9c~{=g;ۣD^?\9~{K[&'i#5 n1zn9ۆ5ICQpwX`~1=EIJ#X# 2+sе=!rVhOKf6Sne>m6׵krkc/qI3{3qANM~>yng3 =s*:':<^(t/}^I/+cP~=_ťsы}VTuL9sOV„횣w~Pr߱^Ϝ_ce{ޣQ^==[3׫be=16ⳓOk݋u uU񄮑԰bᲘ=#yU>X'`&Vզ'P+zoqyƞl^qAGb^M?/=ɘ)UoqW^k>F׫_57ϫkk|qd=Pce_to$_-LT5[1;NԼn^<=+ Jb]7Dl .0&L"{/<SMx;z,BT6_҈r$&bkb W+>i6~>cZ־90v{q@[s㲵/6zXQš3 9}fʞGž/g/}7Q{'C^}[.X5{޻99V?D?S}M>9kD3 -LUqa~OD|8oE'.cLl#|[Xc/[&trQgBg`Qcbc˟6_ Qk>_vkkEƕ~m+嬏ׂQI|v4hyy?",k$R#DD̠}abXWX$z"g%6Xb~ͤORsFb_z?#',1VWu"֟>k:1 $A{<=^ycUnkz{co&ްVt.<ʵԿڷ}Ge&&bKaARTzN5p[P0EkEM..𼰢1=|忾 #&ߛ+piHpbPٞ.*EG>_mgNJy֬Pc`Y =?s=!}^^heo/?ՓΠqV^.~x^/䘈CQTJ>Sfr#Qke$ʃWz͋lۼ8|<?=<AnWnu}_AcN˿^}HwuW!m2]9Ɖ_j39y>y3?Z&53O+sjZ5Y`}Jcd A"~=5&l_#L?_<9%ǷE}7Dπ~6ע􂳎Z#ž׳˿3Y)&rVH]A+<Dž&:ƶba"Rv|T[$x-<y[ox=ĺ/|$pu]γ?~|kgS2xę4uw∛-ry6}%*#C `G_.o9Sп@ -_v A_Пϔ3r&c6z{o_S϶#<-{PhaOꅫ n/\Sqa"xq]SD>ݓﭫrfYlnC=|-YaK{9Io"I Ja}fB~HsO?pbg{A &c3h8\xK?'|}(P=n\HrϾU#N_C=|-,邾'gp!? kIĶV۪v&$lӚo"B>a&إoJ Kc UUt* v,1*i]CW~M+Q̅^\4FoK(sf|~BW^ynq?ۂ5染Eki.Ϲàs/gůKN]Fj㱏>z}?ϙ7]brzNS//<_k9Fe^Nk]kgMϤk~@z.rƆOU{loC|o|0;Tϟ| Y/|<~i2G%;9gnqz|wmϳWЗ9cs(Wb$|qE&lO^L9GzA" #[_jl;rz@P˺ߚ_[ GQ!;vLɗ%JKsnc["u]Lq=FqewomAZbY^ڮΊ{1"U3yun}z|[ksVx׸"goZqg;N-o"MH*UA!,uPΌ>0 Тo͒PQ9-?iS]]kD=xX<+So0.yz7j"^Ϻ^йQ\W5=?DƷBJ/ɾ"bEg9J>QsJ{˚ezr9-sVKȇ4^cQ{(0g?ze)[N_}6]Aǵ|Y#>0e$}~=g_d=ɇw(~E}N!zO*$_z] !pTΚؐ_~bI/~qźW<~*dө+dA*[ioxD~;{Oخ}V֯ת{{JiMWXױ^׸GyFP䃰ƮzTIus|]m.[l>+9y{O,:}:.àNĺm֜սۜ23/͟Ka[ϩ~9K^0=<ǼyZcWz-m;9KX9^d+pًHsY.s'r3Qg'^N 5 !4"x~ȇ`ЌBHۊE%(eg'U⏜?C$ri1gzs ^kZ+kٴqbAKc>(a\FjyeWKi-_P_pQs̽ᵴRLa};36ϭG^+mD3q`,V]eww_7n'?3k2q弊3Q$3#ՏQVKӽ7FhXt8.sb~yK,\=,+<-z .Zl۵~*zUx5q0i{&_{V>uO(:;j hyu{[[e `?wbqbˬ!bqyvD*-X܏'F;F}eF4dgKM}8~yosy[KFy cȴg{(g/v?71gDplԃd;V{^k N~qsغwb]ã1n악hkxi{دxb9…'Ly[HZW-sj 6Iq_1;i̹e6kl&|^k~g2 dɍ ۃ +| |^h׬-CQ/L$ܛp avV6G'Qmeo/Kϙ׌۴>31>k6߻p*}1:=笏ַ{9[bM3K"("m@1%:Ta~]HvW3\ރDUe׫EXG-ܰu))h^ZK:f7.gZ-5r}V?Gq1Gޗ7 ^8JnJb!gvm?[)cΘ_㾥{O(}+dzk"GҖIZ}nBv={)澦~ d+ʑ$ 10[^\&l9^\WƠz}3KHף3sr!\9KC 5Glc9GZzzgWzb:GAΈ/XW%M9Q$nK~II3сI/<SMx;z,BT6_҈r$&bkb d7fS^Zڱ89.[bWu/9s SgvzQrb[z|}q}bQc̾:?c-qO3CxȜ͡sƟw >%*؂8DŽy1E-Y7Z W+bBfWlkMG rfjM_a23жfM_V׫X3/^pjmĽ&ڱ=녣E(:T6&בֿYL-lGk`,xjeb*s{[׋zؽ1_^GN Pyk}9ق331^1 5{ ;{ص"6{r6F,!>;rX#q aE{=187*}Qdp <-WkS7k !(_w&N~@8 N}@8 N}@8 N}ܵ˯߿v,oU^//1pT;o#???u<-ݛ9Zn^37[.}חϾKol/_So,x˟}t'nɛZ<|K.!=xqm#酫sz>\zcs][}a<A~c?m)r#/Oso{9?Ip}˶#<-ID^8WV)BN 5E=y޺*gȶV?`곈r|v$A$Y8S:&o$Iϛ !QvΉ> ϟ ~P{ٷ 3ϔ]'A(>{ 7-ly0&sO!.}SbX%KzUV=Vc߈QOާ~H¾k7EXA0Gb.1~[B3[:w#w3Ǵt=tq6/ZKsyupC}9+~}_^bv:7W}||쾑s"/z~ZY7-rZ]>mz&E]}u.й^ᾋ!>&x%{ $ޡzf{ X9/Y_6_u>~i2G%;9gnqz|wmϳWЗ9cs(Wb$|qE&l˘L9GzA" #[_jl;rVvO[yC1/k~yla/,G1G&_֗dC߳+5/%Ϲ}mӊo׵v-3qE(: ܽi]6gyj;+jLj7W9Kτ:չoտΫ5nebWd[f{_>zYi-t9@Fg'^PE?qa" $x^vI:O) X# -fC(۟?.{5eծuZqF,t>E Ywps!|7=_gp߳ăP>׫@l2u9bɊYկU,1ů0._XڞYl:5_~rlTDŒs+| og 5Z~o/_5)sRx: (Aك|zUJ[_9)qھ0>~->emgE8'oRCǥsԉXTךwrZ:}&_s9l9 g_ Ƕ7/_ sJe_m"g)6C0'Ջ}e.{qTi.˅q=oN${YHB ZyApÆ=8FBrE JBLq\%3D"sFzX7൦nOM'4يę6Wֱiyߢ 8^K-dַ#9cz6K>A ƒAho/1޵].߻y{Zupv3 &ZΫM?lD-﫚_Ÿ0BÂ8\>k5lbBfq@^d~ |}ʯ"vmʠ^Gnv\:Lڞ%i7`/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>a2r/rjkh]5^/paG][c8gz@g'^N/dHW0"ypI>r+L(‘ۜ|%A*$hҾ\X8m~ E%k7#8k#i6^9hD];WT%dRBouH Qۜjl>ǭoGr&}j_+Į3H2^Xy?[pv3+#{x3r~6cb[0cA~F=N2/9…ݏ1Ҟ\x(_ \8=xh;NlA/.xk|^>x*Ҟ2_^v:CLѻS3>5 zaaM~?%vokrvщy}x u?wo/K7g`?g3lS~l1:}li ˩*IbHMs8ŞB~8)FZ(6!̊+b*gGbЊ}bDog]fDCkρzxdٗ=4owq>;VL{vrbs}L[AF=8OhkqmgW0GЉ'V1=x#ߛ_]zy]K~qxG _36͟GBՔ?g/vq'sg{뇾k2gq#zoi$wr>z!ӽ?Fs_Sm~?g=;؟c^Q$Yi_( 8 ; b6{o]5X# 2+sе=!rVhOKf6Sne>m6׵krkc/qI3{3qANM~>yng3 =s*:':<^(t/}^I/+cP~=_ťsы}VTuL9sOV„횣w~Pr߱^Ϝ_ce{ޣQ^==[3׫be=1}{"远xĈ/o ω"v\ia$LCD_$yT?8ʧ"wYH2mL .5ILk1Y,'W`}l}6Ǵ}+rhka2=bUYK; ek_l걢5gsA s*̮==Qb#}_Θ_,7~KoO547>ʙV']l8jw[g's"r%.BSC~|hsV牉wUn?|Z[=91z1+·56k[^&@ P("KW((:&֚tmԚ`9T1z|qc5eg&fYm/bgz/~{/dW1(g^ls:~ڒצ}Bpzt"}WUVF֬ess6EVۣzzR<219AE=O֯ #]z{lAԙPnqkzdWBԚ==Zq忇lF9#wve'߯!EH\sX^@@ >hA/J_&Y$$o + z\oIAc)cG3`?L~<8ƤOBǧp@n'#:p3fyZ9|sp;N1T[o#_?8st-7݇kOxqFnoO/solo:=Ӌ/83o,~8}Oŏ :Nq|7ߪ[^Hk6]Γ_~9|31_6-M}lv/Z_#gQ*+mWmqJoxܷ(|ywpzΞb% ~=#y *l׽쑳z997(gߜHq?>Fy\3V(^'C䏷cz )HNRב9yu({Zd7+_GOR\&>dpO|[Jh~>$]OQܧ_Jzm6A=nYܿe*`2+_S.Ym.ӋN~lxߴmq [ݸ)n}֟so[Q\/^Sl\C}O" w~ɗD[mU;s:p/SA i/7R_ 0dib]ģK0P\[kz lǿ`'wҺ VL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽk>6Ϭ\pEؐ|?]x!_/襷7@b[`߈Y}^ߦ[w/$P?K9*.q>s}=KKk{s}36r-O7Wmvϴ[ỞsI/⹕֑-/53r VvO[yM1/k~yla/,\Z ٱcL/Ɇ>gWj^Jܟs}^[ksVxǸ"goZq<ĐHq\x 6J_:?΂I^6cK,hQi[>z`W[1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRq͢4O[f<ׇxK֗%}2<_{B@ۣE~A<;IHES>S$_\D-ykNjL|~+1ֽsV!N͗_'URJ;Ͼ=aYYƛ>ڟ -WLk..8z_A9({^gգuNk/jOubYsb!>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{xY~#{n2 ^ӽV6/}b~{"T,b=, /w 2?>Wo{⢵ʶk2W[l ‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿe5}kc{Qזt($ɦ8þ 9Y8z $^^D/v$DG{F\D›#NDbMڷ OcR"dfpem$X<ͦ3m_ÙH@k'jQ~dzwb%2 FfPBlTcV|vIG'&>Y:)޽,Sk^=N1>\5&a[cu($8L_XE9 6RCW ɶinJ3K"iWC1pF fVhVE<;VmG#ov~>2# 2^}.g7KM}z޲2X߷6.b}g#ӮbE9c~֏^&zӭ g'[4ɵ8[p3+#Ϳol_b[sw,a"u_kK($0T^~j8Gqy~p{#0}㾸5bv\sS(m6$LNKz!c(ZbS/]'7.,'l3-e{=9Ga^G0a{Tpoٿ2ۯ=ZG۰F}9,k`?gn_3fl4hkk|Xñ=ˋUbuz8$ɦ8#}-$a $< $S!xTHvW3\ރUeǫ 3S"YG-ܰu*)n^ZK:"f7.gPLւ(.;!]}^gW8<ң[DӅX]#j8?g/vqҿsO(}+dzk"GҖIZȽnBv={~)漦~fɹ!k=򽴽pB~߿bB!?GH5酿Lj :H+ʶ,tmbsG>ZRhQ%V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"gbQc̾:?yc-qO5ŜC&\O2^#T]xnaʧ~jlAo=1_[1&6{E-Y7bA $R1 QB3i&]#9sXU/^3_Xd jw,2mElSo,*?̋mZ/ xl!zzt"}WUVF֬ess6EVۣzzRH219ޠvow'}ׄבS=TZ_r L l8jWl=tL+!j{OvmȸlF9#U[ \ή>- mຠ`+ Ꮘ`^O#ɂ&qxxZXV@q.ܮ\c}]qqng5sxҵsO??^u&p=p@nqq 7}Ɓ8OFu+g rtt7voO?ݝ^yc`^ޘÓG~<,1&޿q<.9Znջמr=|ߞ^||uz/_xc pzgXwp}t'-oU%9xqm9O'>rz+gc2lbO[lt_qFUWڮ"oˣ_$}BΗw7)+Vbz8 Bo/~kʥ988@/c%zŞ8?^8]qz 9K)^\G ʗ~6:=wd-/᣸4gyn~L``u-z]~pD!zώ>R O ~P[oٷ~>>}q|&c5墜e%_}b!iq>z({ q]S,-V#Ͷo[Q\3ߵn6>B=A>HQD$V۪v h [m5$b9&svH7%%_*`_ ca;E#L>"CW~MaVL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkmh[=tW"jlH~E"~A/zx;A` X9 xr9e"}X]ǥ%ݵ=7lCK'ᛌ+6a;_gZ]ϹZ!:vFζB)u?)eo/-셅KQ!;vLɗ%JKsnc["u]Lq=FqebwomAZbY^ڮΊ{1"U5yun}k2f1+]b=̏qD=Dάߴ m#arő~ =lXW smg$DY RȢh5KCG堎xMuzxAWFq^ +$ֶ 59(˫G!K)]Ǔ/kN>kk|c<|H5U{A/?ϛ s?g[Ƒb`1ٽUgڱp\:ר889ۆkt>/ŋX/ɽ}N~!hG,P>ϫXl2syG *ԘW/b{umB6/{`g*[i7?"'lW>+xgUs{JiMWXeЯqr=a QI{:'5Nu֧l9Pzt:뾚꿶UZsVnsZNXdg7.9m=,zvca^K?Y,cqze/#ez86d_GshBt AD$4h!nP^̿!"9n+ ĚUH8Yhr"K ;#x=F"KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-UߦyrS?>_Ÿ0Âx1O/ʥ0`^_"C@>U6P!lkP -<-z٥=*"vlʠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e GYf`b 0Ȃxin_5i/'y"! cE )6^w?~VJ{\3~]DĮ~MHXk~X'6Yˠfl1^>x*Ҟ2_^v:CL;S3>5 zaaM~?%8֮=z֣,~f^tn~yũgl5^/I{XM.1:}lS=a*I‹IMsPK^dI1pF fVhVE<;VmG#ov~>2# 2^}.g7KM}z޲2X߷6.b}g#ӮbE9c~֏^&zӭ g'[4ɵ8[p3+#Ϳolܗab[sw,o PH`~1g-\q+ 9z5rG$a}qk8e6kl&|%1VeWo-ȮWv\#Y[Ƈ0^=I7_i#mXNL>ۜ^533i}gb}쵵mw,*}1:=lבq)k& L%AaV H V H&5FR\-dserzTV'Ld^ȓsEzQk5.tOMߨϺ-kcQ-gs{˾KܳAҫ_G-B, ̮MgP5e3;Ǹo'sg{뇾c2gq#zoi$wr^z!=?Fs^Sm~?g#r#kx+ʑ$S m-y~(xbB!?GH5奿3o9R9#-?' ]9"g;of3V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"gg曨=e ͍crl[,c=ɜklTpP})&Zl5>~NtFŒO~jlAop}A#Ce'4cZ-}q/~-Iu_%4fXkҵ=b3SkP5ōEրvM̲^+C6_j6^|^ϭ*?̋mZ/ xl!zzt"}WUVF֬ess6EVۣzzRH219ޠvow'}ׄבS=TZ_r L l8jWl=tL+!j{OvmȸlF9C~x*skU# \a[=1|ɂ^@x$Y0$oO + a ؅s+ L涿}qO.\~3ǂ[!o37}Ɓ8>p@ndWN޽r,o_+oNNw_zcg7vp9^8yrq/r<&&ɶNWk 8]}%* 57<[r;}x?vgOqX[}ő<q~|q6^S."zٞ97(gߜHq?>Fy\W쉀zP1إG ӿu|9tN^|nʞ;͊Q\3m&dpOl[B{$\K4?'(/%|rz6,߲o{0),RV&[7OP,EmP-߂uMt+[eOrP.᣸0gtγ/\wPq䳈~ Ds/x[mU;s"fH~B"9/7/_ 0dib]1QO4S\[XA؎SOR?uua_5$8#G1zqO]3[eW>ynq?ۂ5染Eki.Ϲàuq^Ί_KN]Fj㱏>z}߿ϙ7]brzNS//<_k9Fe^Nk]cgMפ5Y4W\#3kuW=s56$GoE^{ x[`O 0 ؜v/_^#g"}&ׂ%k,q\^"]+˜9o~h~ɸoseL KzAϫl~mveikyY[c {azTȎS}de}I69RR9>v]lײ9}\Q\_ٮ[{ֵo3}ׯ빳^}|~qtMc^[j>/ڭYl+WlcQ378g=څ?GE4"ě\Q"E>vL:O/Sg_@Ȣh5KC 5i| ma^|G|w &ۅ4O/&[^dY :wUjcz~o^}ek[r| գ%ɗ5aյ5v}>1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRql镼?_>_l_$i|<_{B@ۣE~A <;IHES>_$_\D-gykNjL|~+?۫k{?g|}Q!Kέ+nk鳪b~ϴR㺬5Q zqV=*io}\i@9T-]᜼='J[aP'bWS׶^kmNi̿%簭T%Y/c޼|,̱+}}I翶?x,T/_}Qq,^9Ay~j p Y@#F!DDABr6;]_ sɢH G%. A0Fs$ϠvFzX7൦nOM'4يę6Wֱiyߢ 8Μ^K-dַKrkm}&Ο%*_bk]߹y{Zupv3 &ZΫM?G'+VK=7FtX/ovWakظ'Bv϶=k=-"TqmA\Vumc#UzUx5q0i{&_{I/(ѭ |H븟lQ\B#qgx/0ףAmmQ/L_ vdIbw|#-]s:3S0u![^Ο)u\ql>ڒN9Q6=]9,RTl/ioQ?g}0шHZxs bMڷ*(54!- LnFrFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>u[z-X ^ۗfM爅}+0?Ξ^_?+hxq=kum=GUlnG5qz"ambvd-^]KzW٪J{|%{!ۭL^BX\l3^`lG؂O$-5 J8o~[s|XNL|tS{{Yҹ^9Οx`{$bc4}`kLÞfQwim۶^.0qau$HE]$ۦY(OB'(2U&DYC,BaVZZO(vvaq\Po,@ee䱾om.]/mN9>vs/ͷ~0;n-9`<٢Nű՞W؂a\A'nX|c(g}hM1:D~ q; 0ƺ/5{x($0T^~j8Gqy~p{#ֈIseM-F\c09k/酼Vy*zkM@vܸ=xϐ·zq2>y„QM½ gnnOkoztb\}͘M<fmcp//6V8g=<q$l0YDJnjQI/ BkDZRq=HPY[v03+5zy"O _WAEո#`>i6~>z'gs{˾KܳAҫ_Gэb, ̮MgP5e3;Ǹo'sg{뇾c2gq#zoi$wr^z!=?Fs^Sm~?g^Q:9HEb[/P I\^w"_̆7#[1ZuőQ\ٖ2!rVhOKf6Sne>m6׵crkc/I3{3qѿz}l3"gCΩx]|н{MVs$22AgΑG/f6Y#S1 ̽CZ?[9&rs k߽A}zA_s ~{F&zl5?,\8g}r^q,~@m7%$%‚mNSPxYTE2mL/5IXk1Y,'W`}l}6Ǵ}#rhka2=bUYK; ek_l걢΅5guA s*̮==Qb#1Xn>3Dy,khn\{}3oOdbqyOD^#XeK\}M15b笮 )q 1.sܮs39Mp@nqq 7}n/NN㳼?=twz'/8L݇v-z5"A y˛zx8}xʙsC7t/?vqk M/~zγomx~hlf?c~i'ikۡ *:g}oO5rޯ5Qs]쭟K[Pq? 3 f?N_%??|O#b |ZHԈypteׄ#+ MV73U' IiSW`L~&ϊv(gQק_y{g\Pz6 m$G9EoA~!FM9]!%[5 ?V_E d=q;[6OYpp!¥W/>+{s_-{%$p}o }܃v6+[lԐkPO_c[ʭ^75oz|ͣZ{(]&އwoP4~Vdy)OB779|,EQb@_U sx{w*\9ouť_~7ow|{ /9V>>?z*&۞X޻zy=uBt[߮{_[ubӾ8.ՇU]9tmSl[׉k8Chܻްq:ڧx{E϶wʼn D@g]{ٶ (팙8+ٯb@䷉ZkZL`.~р,-EԢUscV_+a\Ă~-ϫuj eƒ*HdţZŬ:gq]g,OkU\숙s`g׊t;r=xݿI6p8gfl]Rrj|aٺV_zVZ5?E۶~ck^5->sk+i]:#{6n^=5ʗKjmy1fO*[}xgu~l!)qD_&"l[X$vZLn<MBtv(1f,o(kk>_x?DnQCgT\[bq>R%~1~yIO/_/ٚ|wD3s O:.0ؼ7!e;WE"qEuIO}E_2{'EXI@ 0j\:Ĺml9׎]S g\"*ΗWPk4}53mri>{8 UasG>i>o6*V=Ϲ~gsxZ$ga6wzq!V1.r}/ОDfG-sBH܏lOq+Eʁ -!SZ{?>&' /w-!YK%BKy!: VX=[6 6<_+h(ȏ]<=Ov7I:F>y/sܽ.|𚲠{X|x= H{wvњMƭZI L2km[{֚}]c]X\FmϫkO^=\!qTG0u4W|= 75vmL\_/NgՌU`71WuovOho;91g}y}))fŰ$Fhj8٪+h]{3O_O?As>vs>Dp뉘kJY`lx=cڏku냵m⼸ִ>{Ÿo+z^`/uhmUzoX2Cbgq?{-QDu#_O؞-Oc}?ʍݻ|?ɏu!Ֆ䗐_Q{^wD |OX>Ӛe( Fbpce[0X$u$~?Ίחz)I'sj`DW8Eqi{J'P##έ{G+b. )<ιx= _s¨I-Eĕ;9YAui~].uO7Ƨ,:^r׳Ճx._ʹl1f_l^G=D8oP9yZ'đ/Hۏ3p{*AW#X9:1 ķo]`;jK~^ JO1x9 uB%/Q^ sO>ފ_e?[c쭟z_"RX/ٛY_"}0s% gӟ_}" H#m_בMg_c6y{NO,]9Oxw围\3Y}i9 66ZJT+5g6=gZWZG}&3Wr7׷fKϯuZs_գVZ;5=Lt/(55nkF%WyGھ(#m?f7-|)8h/W@7I?}?% >@8qq 7}Ɓ8>ŗwoYޟ~;ćݗj|D}9S>ѹy9y^ N޽rëOGD7J=L=z3I{zÎ׉CsF~5s=8϶%w<3΅?g[YZۻ<r.SGŵ0Nqr{YλV߷*Tsy}|>.G_\Nqgs۳_9Vy?ս?}~[6|&u$[??OgAڶsyZ'vʷY[SkMԜ%f}1{RVùEJ禉3??KOoZQ>קTk?}ϑ}1v$/7MD_:.x;l3}9-꽏n}lyYyt,WyaPϛF\4Wm\w׹X>{|?Q/=0ߵ{s"v_=dWroUM6^v,maEs}S.Km]r鵲׼X{zn\رʛ}resʱ?S7!~rkxMuJI9y\o*Xʄ$Js4[U0_?sgEZAUӯJ3.q=ObPd6wwᣜK M#㦜ur׭{]zqX"|+$?ylMC,B?76ᰇ5ۗ?f8ӗ \k>̼<:0zv>'uin|JÏ?O_\|Q{½p{ϋRӋ9|%=8Isq={_vέO|0ֳϽ}V̵\# >KxM9*FMvg\saglVgQh9z5u{s_-{%$p}o }܃v6+[lԐkPO_c[ʭ^75oz|ͣZ{(t?f|S {Җ E..H,R9u,~3g衏zoysoZu|E{'߃,=[:|r62nk.]QɜmѸa[8gz,L4uQ\:&>/-Vcvbz >f[Q=MMzrubӾSLje<53ο>^\Zrc.{^\rZ`歨^нQmƽ moٚr>a>Ft66ksI߽K3[ԚR<UXu~hlهUupT 2o漤9b']'Kԟ[Hz-^cA/^NMt=S9靛y8_cy3σ6}/d|zy;Og|^ιRas|ΝŦ3BbcAzaxޫ^uz[sާu~ޱʚ~z۹99YmY؝A\Ys^]].]ʜD=D5ms|{O8|+1OC#:Fǣu^H=lNj{i2gn\g|1ߺoQ9r?OY:=Br&{^Ϙv^Ksk3~(.{@qJ~Z{y ׶udߍđ\l'\m=|5k &_goӷ#[3V_lzG*9R|$߃spsXq5US>7DÃUjϴm |yٯ=n=G98sw_+yθ$n~|KӟhϋI~jBP~tu {g'k ^4lߞ VL9W9Km8n&93y{]3Q[:&sfrOwAd{f=v1+BաQrkԤɶ9>k\Hykf+Q[Uիj7Esw=br?F ⪤::8%z<>ӶؠΫ7_#kks gK(kg75բnPy$2~aZ";br剮.htkE4cq|D_\ILC %rڴ>Z:zɇ/!˅]L=ٚ|nf~9XZnl|?6|Suzs."~g$o{>?"k=իgͤ߇Cd/D67oJ?asEoVAv.Zyi>g#TW]kg?3I[{1߹fqAZ?q7,d]lμ1/ Y1z _<'*G~Vﳺw+Cڇ~\9 $[Zk2kw#L~hmoׇY|}[͜|_^Wqώ^+rs)mxG}U>{5힓}MN< {7[ȆzD=Lql=QQk&G?vgُ5soq韣c~d]ûe9x}U_ҟgB:&Ytqr5>$?k{5jHep[k{~5󵟳㜜-ڪkf/g 'NԠrAS ۻw1]GVڻv~\ +G ;\rRk_rq_s~pۃzǢW?Kÿ=3ˏs.Y/6VHضqF5hz4֦)z[bi!g:pkeڄ#?i"q? η[ ο}]q[~)w$2;bo}F~d{$\)R}cۿd97KNӟ |R Z>f>t{eb?sS{9̝r#_Y6v^/š|q?_5@C :iX{ pZgd[regݜ}>vmn~}eex8^F%9[09Q^x=.ƶzx&86u^~ {wՖI{/7fi [\_6n;^GNÞ{5N>L=yݭ_ѹ_+q%z_ߏX{&κ =[ۯǓ3c۱7p|\]+l@냅g?8/P.~oogFƮI¸v%) _0 j͎ M~1'&.bI}/Eֻ 1B.4j8^Xۏլ%:۳;cq?~rS<?Ca}&ݰ{L”rS:ss禌k>g#>J 'v:V߽،g8f† >{fآ?7Es\9s*9jOX^[sK~[nΜI}N9>kuMC7~/T後֨>zX,[zAha[~usQks7k{Uߧ^ c>mϾ?Oc&O"_Aϻ&'Ko>`r rj0FƉh8/kк:=i~}Dܲ.||vf s]|ޥL3 ݜׂY\^xwձaj_:纱ke?㼜-kf/g|N2կ俻6[ʉ;u>~d7ߣ=ݛQ\]*-gQ=gczˏ5Y.ql_'gƶsgq&5mJ9"џF;s?{-QDu#_O؞-Oc}?ʍݻ|?ɏu!Ֆ䗐_Q{.l.tpNzaLM{+Ŧ]XuՍm7ƩFD„~z!z}X)}ݰLœM᱌{9LHlAPa$[-ٜJ-/5uoዺ>OuQZm8fԌN 14Aӟ?M>-~_=^\^oq'wiT_wqr|{6W~uYerAtL/^5Kcb#m7n|MϗFWDL.D?5Gԣ1>$l bsP{/ ק~#iϮSO>?rg[cQOɾ77^N~M7F~m$3}rMvkenzy/ z}6879[wvE\ҧv빿W^/_]scKvID Wdg7R4~_]`._ `Ĝ*btO|w$z[۫-z-+=;<僸 {G֖D=`z)=x+~ҽHŘ.B6 u}(%-^Ի7zkw3A^O/s>~mpRֺכаϜG7.7x/J?*66 ݣeOK 9 ]zהQnMlok&osSU{wxm̶zʽ!jAcW9탌LHjjDk{kgzx3 lmy3 ˰lMm΢^0kEM z!W+Y"o-cH)W<Z}&{kwfK'ZxKP:sJnz^+Q>[W^&^v/^z˵GB9GFn:g|I9]16gDrP:zDS*'¶I}%UUrپ׿s=7)U?h=[S֏. !/0ׂl]a q\n%չOȶ883^k%k^MS{}ډe__is]0S{M??4@+A1[=4_L=l!o9wLCO#[>l=njΟ凸$p]^B ੃~@|ދǾ'7ٱD/ozv#xD' }p gq~>8>y~ arz+gc2lbO[lt_qFUWڮ"eoڋ#^/XxE!˻Ӈolw+Eq_SW!7gc5咜\X8@/c%zŞ8?^8]qz 9K)^\G ʗ~>:=wd-/᣸$gy}_p-=aWχ <;K ܷ^q?-[e?>_k/im~>C@~lxߴmq [ݸ)n}f7-z(v]p=p5=A>77I$_>'!Vs&dʹf xL/c2g4.}SX%KzV- Qz lbRܥ~H¾k/HXq0Gb.12gƷ4j|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkm$[3׌^\ )⫈E^{ Ex[`u| {}x?#>O/}Xwnqz|wms/sPRI&7M1+|s.Vl~mv^̢u?)eo/-셅KQ!;vLɗ%JKsnc["u]Lq=FqebwomAZbY^ڮΊ{1"U5yun}k2f1+]b=̏qD=Dάߴ m8#"p{.dd_:?΂I^6sK,hQi[>zTjϟ!Ż׈zyW~`B]HoEnuQ=s +jcz~o^}ek[r| գ%ɗ5aյ5v}>1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRqmu5 -]yK"/Kr%toYzIBB(TU,H6Z8YQ9kC '5&>^]9M/X* Xrngv]q_MU+}5eN_/cq\B=5rP ^/0ΪG%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>ҟw/cy̛9{ϲ/ ׶g!E˾?*48gؕ}OD "BBFr6>KɢH G%. A0Fs$ ^Zaů\ך>>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{xy"GDUtMmzQ.B |\5lgg[!hx>x[))~{UE|]؈ՠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e r+L4"ޜ|%GblL,FY/~&/Q2/>"': ^\iϚkFݯkȱsT&NO$5?^,ڎe 3 ^_{I/ <[Q\iOt/duK!{ [)`š&A\ gokמrvщy}x u?wo/K:7k`?g3lS~lc{c>s6Iyz,VQ|N'Hm-^d EQe7M0BKX.0bms?bQxYsA=soY[jK=4owq >vs/ͷ~0;n-9`<٢Nű՞W؂a\A'nX|c(gs6Ql}8 @Hc /&#b8>y"D#ýX/_͑Γ.Q)k& L%AaV H.t/ɮ6jq&{ x=af W$kEVP%-֋ZqIGc}lF}ElYɥ>x#w_]z^8JnMbQgvm?[)19}K=g;ۣD^?9~{K[&'i#5 n1n9!1}؟c[] e9 饹½#b6%Au~ˑYWŕm9Y /9"g;of3V#fs]l}>&׊6<7c; |'M=#r{qN=뜊Ή+׵>݋+z_ׄm5Gҋ+#Ժ_~Fqzbfcn8"?US;c"g\E_*cƤ_Qa[Al~v#r{'ͦܯg#xL7"n/.(vk}^cqr\F+\Xs\2Z(6Fo3MԞɲбWG9D-G1nldN5BUE~*z~>s^-~ꚘU3~S&vAWc }r<Ce'4cZ-}q/~Ӌ&"}EBЌsmbIAL C7Y:]>71z m|m>{{!muE9bF`Co^ݸe_Ǖj乕5{ٜu3ŜMq2hmeRmLLe7ucI5uT֗-:>-c[_~JZS8][+27&bQF7kn )0=%FB. z#b *}HIV9@ +Xq9nW9@Atm&t\f.g5.Cgnqq 7}Ɓ8ɈNw?=>{匁Y޼Vߜ>᧻+o tsxr׏%7Μes3:]ˍzS.~x\Ӌ>=ۛNϿ olN/ !9N|oĵtͷjb=/|"p8Gׇ_N/_yLMiKm#pJ@UDOM~˕$oQ۝=cJ@lz\GUد{M+g9/~s@{G1DPtbO o/.8@d=R/#I?]znʞ;͊Q앳l 7! { VJ/~<}jt=gGq~) eq}g2+_SvYK3j$scwEmP-߂uMt+[eO7۾o_GO}G8T,"p'b2j[Q=A i/7R_ 0dib]K0P\[kz lǿ`'wҺ VL룘 h槿PՅj|]1:]?+k\-?'/lm\sA=P㼜X΍;c}j?;3odK^_x־s֍l:6ֺzl+yIѽkokm9ȹ/䋐H c/t``,>/FoO/}X]ǥ%ݵ=ϹCK'ᛌ+6a;_gZ]ϹZJ9 힟Zb^-^XcT|Y_ }ήԼ?y:O+r]>۵lycW/vֶZշ>WYQ{/>FDWY&1}ۭu^O~Z,|E+^l1șsAA R1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRqi9ρk?~)?%}"<_{B@ۣE~A<;IHES>O$_\D-gykNjL|~+1ֽsV!N͗_'URJ;Ͼ=aYYƛ>ڟ -WLkʜU`jcq\B=5rP ^/0ΪG%_8m_h?[Ŷ"BCxw9 DjVkYݻi9-c>ҟw/cy̛9{ϲ/ ׶g!E˾?*48g#z/8=-B$у-mAa|E Yq[H%!8Yh"AH ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr6@1#{n2 Ey!Ef2|\5lbC:^d~ |}ʯEkm_v;6YeP #F};Q&mk/.‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿe/_rjkh]5^/p飮-鋱Q8þ 9Y8z $^^D9y_H$|Wh-9J8TH$֤}̱p4!- LnFVFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM( 6ӯeWd șsn$xv/kv-q;̑=<bkO?g=t<|lL,FY/͝zM_7x >b#b8{zycN~|ŕf_=Ck/mljM2?||(Wbݺ%=z-xyaM^b{X O3淵k@~;LJǼ>Kx<ٻ%53~^qy G)V?FӇ ı=ykvN90qau$HaB]$ۦY(OEᾈI1pF fVhVE<;VmG#ov~>bųnj>3ƛſ&˾_=~yosy[KFy i11|lbQƓ-Z[yu-8tb߉U7vyr֡PŶ58")X $D PH`~1g-\q+ 9z5ď$p߸/n|snj6)_{I/r?u̳U[Klƅ{lG4c֖!&ljM8WvvZc|֣Ө6粷 kƌmZm5{mm|8{yJ_N9KЄ5Hq_ 7I`/ "EbqCb_T]#*m$L6W*ǡ Aen*6zHQ<97~]J[V㒎Hdٲ6K}V?Gq1GO=/t-9q".Ģ`m4 US9c~s{kE\{Mzٛ1 ÞgDt/0ΩS9q6G{qEHzqed^Za(.#]^lG䧪cA{~·rL,^&l%{烲+ jMj$X^/qζ`sˡ2pW)sIė<$~9GT:_sGU|Qq'~iDMmoexVˉ+X4rn1-k߈ZLoحzUҎ9qzsa͙s]Ȝ>keOOԣ؈b?/g/}7Q{'C^}[.X5{޻99V?D_SyM>9kf9CՅ_6_֧oⅿ[$qx~QoKv6抠@fmK_x܋$$h MEp(U4ZZ,*/n,t|}nbZڶ"Vl}Bzr6 NB|¥E(:T6&ϭYL-lGk`,jeb*sĽAE=O֯ #z{lAԙPnqkzdWBԚ™=Zq?6{rXvm!Hhs+>z#b *}H`I+>s`nWϱ>s}ԟI!oIυ+>oBX^s9Mz恸8>p@nqqtûWkKo ޟ~;PNo1'W|xXbLy\;7sӵȫw=a{9==Ͻ_N/>Oϼ;>?68O\+O[<|K.o!sz!' st=:O..~}WΜdpĞ48j=|G]E 2K$oQ۝=cJ@lz\GUد{M(g}ٽq+!7缼?R϶x${"^8TLx{0v"둂dyR/ɋϭC3G"۾Yq> sF,߲o{0){,_}B[; ~"}(PoAw㺦X[G{P}4Q왳?C}O"-D.RH|NBHokjK=}״f>Ћh>6oiӗ^߻׻řcZ?Wtl VָZNN^<:zסy9+~+~/V;ww>~>~v>gvv9Nt|}ٖulx9uZWT6]{(gHf :+|ȹRW/䋐Է޼z@0,>/Fo yb~/[樸zg<|5V8.]/ye7?Xj? d\Ʒ ۹2?oz% A֑-/53rUvO[yM1/k~yla/,\Z ٱcL/Ɇ>gWj^Jܟsr u̫svW߼_117_ma~ !rf\}lq)р os&%x>vL:O\cYtѢf}(۟?C$e'wً(aXNTc%dՋ7{>zxAWFq^ +$ֶ 59(˫G!K)]Ǔ/kN>kk|c<|H5U{A/?ϛ s?g[Ƒb`1ٽUgڱp\:ר889D|qos}_~ID_s E{eG,= xv>էy}H6Z8YQ9k&*ԘW/b{umB6/{`O60bɹv}#{v~7}V?\[̯֔9~eq] (Aك|z8>sR|]}a|_W[l}ۮϊpNߞ y߭K0k[e5gu6崌uLKvssAΒLm1o^>X>˾$_ΟERl<`N/>\8\ m|qhBt AD$4h!nH? KȢH G%. A0Fs$ BZaů\ך>>i6~XX?d+{3hh^YR6~>,+zg?:ssx-m~#mX.#Y\"8J< rF{|2wYjL/+L\k9"6{x RMު~ {&#:,Ey!ыtb2|4z{"T,b=,A݂O/BEkm_v;6YeP #F};Q&mk/%i7^Hs|d5 z4:-ٿ2 A8q+Z֮l9PozEkNG}F{*ۼ9}˜93_!Yۻk^6aG][c8gce-:}_"EaN &h'H$|WhE$-9J8TH$֤}̱p4!- LnFrFl>s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>mdͧ3f`b 0Ȃxin_5i~y)~82VD0?Ξ^w?~VJ{\3~]DĮ~MHXk~X'6Yˠfl1^>x*Ҟ2_^v:CL;S3>5 zaaM~?%8֮=z֣,~f^tn~yũgl5^/I{XM.1:}l-8/ ZH^Hmm↥"R'(2U&DYC,ZBaVZZO(vvaˌP{ 7,-5Y{˞c}\7ʻ8^~]VLv[?f{ Nzp0lx'jϫwl0`[7Nb{xH3D~ q; 0ƺ/5%_LGp|D*/w 0 uőA\2GsWfoFN+snj6)_{I/r?u̳U[Klƅ{lG4c֖!&ljM8WvvZc|֣Ө6粷 kƌmZm5{mm|8{yJ_N9(F`Wp &J 0+H[$VL x%BkDZ*@8$-;^FOx#w_]z^8JnMbQgvm?[)19}K=g;ۣD^?9~{K[&'i#5 n1n9ۆ5I\sI(^k!@Cs,D,0 o9"FI/8F-G*g]`Wda'QY9+}'I~3)2N^6`1VĵФ白øANM >yng3 s!_TtNt\yQ>^\&l9^\WƠz}3KHף3sr!9KC 5Glc9=GZzzgWzbmQ_s,~@m7%IK&9GT:_sGU|QqS/~iDMmoexVˉ+X4rn1-k߈ZLoحzUҎ9qzsa͙s]Ȝ>keOOԣ؈b?/g/}7Q{'C^}[.X5{޻99V?D_SyM>9kD3 -L_֯UR[-Ez/=cb`W-}q/~7$QQQ\uNM51ș5rb~"k@W&fYm/bgz/~{/dW1(g^lszZbCo^ݸe_Ǖj乕5{ٜu3ŜMq2hmeRmLLe7ucI5uT֗-:>-c[_~JZS8][+27&bQ|-fWqR`zJ6p]P0oG@0/' zTdp8Nj <-+s`nWϱ>sܮs39Mp@nqq 7Γ_~z8}xy9?|8}鍝OwW7ꑯKo9~fuyᵧ\s'G'k8>{sox-zN/5^\DzpSů^ʙ /Ӗ&>6;G\3(v/^@ 7<[r;}x?vgOqX[}ő<q~|q6^Sșl8s^)geYlfz(Ya `&dpO[#^7@&Cxvn\}lzܲ˾U dW\~_Oȥ{{,[#7-r<=n[G/}nAw㺦X[GƻP}Q\>Pq䳈~D3b[mU;˂$lӚ_(ߺ䢷y"M,Kߔ0| U{K-@0.CZW_AŠ)?b}sWʜߢPϹky[9sOǶge䅭m~Zs0{j׻k|׹~gx죏^_osflyK wκmYdžZWuE?o5)wr6_Dй^ᾋ![|q-BB_ 1^ou``,>/oy&:_ڷQqx;%k,q\^"]+˜9o~h~ɸo72M KzA #[_jlgl[/Ak]-u5<.G1G&_֗dC+5/%Ϲ}mӊo׵v-3qE(:ݽi]6gyj;+jLj7W9Kׄ:չoտΫoޯZ˘Řȶwŋ0?}9~Z~.>s6\s1ph7`Œu |U\Hl; &y'DΌ~y@dEJߚ%УrPW[1|sZY]E>KGٟM9ӟѳ-H1*N|pQ\X\z8ukTRq=U)3't9xK%/(~E>էy M/}."G8*gMxȟWV¸|cݫk{?g|}Q!Kέ+nk鳪b~ϴI+e,^׸GFP䃰Yq bS]v}VsX(un=t\:Au_M_*{9{9-eg2^?öSre`zxl;y0zY%v,rb8sR~WGő\x=OIg*@B$у-m{IQ^CxErV8(q)7UH8Yh"K ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr'ůɯ~_Ÿ0Âx1O/ʥ0`^o|n{u [KzK,b=, /w 2?yni{⢵ʶk2W[l ‡ua/Na>w{c{=~^֖_„ 8A kW@T(vǷa="5>=m^SN>aѿeUl5}kc{Qזt(YVi[8/dĊƜ@5L{A#+L4"ޜ|%q*$kҾ\X8m~ &k7#+k#i6^9hD];WT%dջ+Ɵ[h}74zb6eַKr&}j9_G⊲+2B97eo5_wޏbuܸ HwM1ܧ>_[z-X ^ۗfMO/ I"! #E) 6^w?~VJ{\3~]D2կ k{͏&kŒ|}>^ VeWS+1 n]gb{Ȟc{=z|71kDplԃd;V{^c N~qsغwb]ã1n악hci_3x /&#b8>y"+uőA\2GƑxŭʜZFl&ar^ yOlUFqz:qa9a{!o/a>☵e|: ۣ{~=j?r߆4͹e_9s1c֧y&Fg^[{"^^lqXl}8RM6Kì@"mX1%@TŃ\HvW3\řރUeǫظE"YG-ܰu*)n^ZK:"f7.gZ-5r}V?Gq1GO=/t-9q".Ģ`m4 US9c~s{Lb{9ӽ8buNEDǕ\k¶#ŕye jݯw|k?tt=z11J)a 1t{=T]sV; Skl/s45ѫgaqz5_'9s 8C%ddS?l/&$Н_D/O Q(RkȢ/J1ncJ|/-M V?^ϊ@`9Iv:>i6~>cZ־90v{q@[s㲵/6zXQš3 9}fʞG~^Θ_,7|KoO547>ʙV']l8jw[g's"r%.BSC|hsVt9T]xnaڧD~YW 5 ^܋1ߓz1E-Y7 Z, Ep(U4ZZ,*/n,t|}nbZڶ"Vl}Bzr6 N<zpi=qʾ+ժs+#kV9fl9"dln=X=|)zqoPczQ;⓾kHǩ^*o/G9[u&T}[5+:&ٿzapfVd\oM~^9cgWqR`zJ6p]P0oG@0/' zTdp8Nj <-+s`nWϱ>sܮs39Mp@nqq 7Γ_~z8}xy9?|8}鍝OwW7ꑯKo9~fuyᵧ\s'G'k8>{sox-zN/5^\DzpSů^ʙ /Ӗ&>6;G\3(vv/_%v|q- 9_ޝ><ܟxcX -_v qq?u),_ܤ{ʁs^$be sVEeM0ɢĞeoMBh~>$0]OQܧ_JqaAq-V_rcI3xߴmq [ݸ)n}^(TE=|쪾C}OWB'|2j[ά1RDB}Z3 <&T1D7%%_*`_ ca;E#L>K=}_`ʏX\Ec4?2gƷ.T{s_G^gi\ӱ-zYYj9m9yak_< ^Z4unٯ6W}#%&D^:nd[ֱ;fk]Q}tM]}u.й^ᾋ!t_eX/H c[{@0bs~x^ߦϡ_ڷQqx;%k,q\^"]+˜9o~h~ɸoseL KzA #[_jlgl[/dA힟Zb^-^XcT|Y_ }ήԼ?y:O+r]>۵lycW/vu-Lz|#"߫|\,]WV:ڧ˿yvk-ccn"/6ACMk WGE4ƛ\b:^*.u~mbʋwEB[>zTjϟATxQ2!LH5 i:^MZ~.t?ze)[N_}6Aǵ|Y#w qos}_d}_{/B>w{B@ۣE~A^{v>էy M/}./prքykNjL|~+1ֽsV!N͗_'U0QJ;Ͼ=aYYƛ>ڟ -WLkʜT2^.~{kA>{^`UJ[9)qھ0>~->emgE8'oωRCǥsԉXTךwrZ:}&?s9l9 g_ Ƕ7/ sJe_m"g)6C0'Ջ}e.{qTi.˅q9\}OD "B A 9wN}P/,.#"9%!8Yh"K ;#x=՛KZZYCG7V'ͦݏ[lEq2bfT +X4߼ZJoчer~[BXguϖzd %9cz6K>AǒAho/1޵].ܼ۽[:q;w]k-Uߦyr6"&{n2 ^=Wg / vh^5lbCf@^d~ |}ʯ"vlʠ^Gnv\:Lڞ^ Jt+:'oׅ:;j hyu{[[e s=Dv"7KΫw)V͍?!co$h(mN56ouǭocoM(9q>W]1"gι(}eڽ~3gW0G0k9>m$f`b 0Ȃxin_5"JGp?l~|ŕ"D׵MZO鉄NjEqb zav>k/_g2+)l3y aq=dtzc ^>E:CX$+Sm_>a=:111OgeIFx 猟W:v^푴wՏႭ1ql{~plޚGqζa*I‹IMsP^] E\Fjy؄3+tEX"lJ C+6#Ύ7;lQuj>3ƛſ&˾_=~yosy[KFy i11|lbQƓ-Z[yu-8tb߉U7vyri{(կ1!Ny!!&.b">"'Ry[HXW /sj=w%a}qk줹2禖Ql#lIB^+S<[Q\Ħ^ Nn\XNg_[zDsq8fm¼za&ބe`_k{ڏ?Ʒa=:1ls.{{Y~ܾfئiYֶޱȇc{pbvHq_ 7I` "EbŔ`^X]#*m$L6W*ǡ Aen*6zHQ<97~] P^ZK:"f7.gPL.Y|w;+?+pеQzth>kz$TM1[wxI?%oYox\[29I׭^H5uîtϏ5Ŝw6t kxW#Ie w-y~(xbBW{/;B/fÛkҋ~Y#o9R9 -?' ;~*6wY5?a,NLqF\[ɵ"`&=ErBl_sS?=xψ^`S:sumAW5a[͑ȼ2;>Q\:G٘g%OUǔ0ioY:Ͻ*Lخ9J~e+})5V9՚ӳH8s/_maW*/(!# A<'dM|ɟE"I?|g"AΣQ>\E_*cƔ_Qa[Al~v#r{'ͦܯg#xL7"n/.(vk}^cqr\F+\Xs\2Z(6Fo3MԞɲбWG9D-G1nldN5BUE~*z~>s^-~}=1Πm”O+e_Uc Ž'F~Qo~Ohpǘ·56k[^&!Ah(KW(D (:&֚tmԚ`9T1x|qc5esжfM_V׫X3/^pj=-fizpi=qʾ+ժs+#kV9fl9"dln=X=|)zqoPczQ;⓾kHǩ^*o/G9[u&T}[5+:&ٿzapfVd\oM~^>- mຠ`+ Ꮘ`^O#ɂ&qxxZXV@q.ܮ\c}]qqng5sxҵsO?aܺpqy 7}Ɓ8>p@nۯO ߾+cחcsԮgA\vD~ᱞ͝Q|sn^3㡜3y9?;sͧ/ >?cJ~g.s{G$Y3myqk.`In-=̼= ӿw!۲^o~_85Tk. Aכ܍%?a^fw}q^v}1wl@vd{W^Ӌo-CX&h֧~< ؋櫴=șF,~%W78/+>tnl}y$ I`jtWA(KWalH{5_o|ק_y{g\7$~`mܑIK/:_~{_+o̷9^~85_G|OE]w#k~7s{n=s̆3PjyΙKA<_ix_C{ypF?zq؎}AfM8yG>[˱9{j׳kz&xg!ysg'3קx( ~9>A g՟ޱ[%|y_>8N̻B?=(ǥȿ?uVqWJٓپ*妳|_4~bӋ"$ח\(][V oʿXѿ(I6bz1ۗ 6!AI 8;shmbG?/a~ ^1tahM7WN~yŷk/l<ƃO#gςj!+'V<ioQ)NBs=#oQI9sO~d[ϩyʿL˷+1C{N9sUvk[:\j62]gOwM99* FߤV}u=׍3ݽ^_/W׳@I{1nO?{~XY%XYcٳKol$_3z!$񎕜:M{L-)fMU>wx~оݝ{J?3PYu֍:{_s;b_<oze+A_4'ӯi]a^R8<ЏG9s|{=Xl{;c3,7c F0ꙏ3Q]wZ~9'_.U$q+}&rB9[scbo,zQ}"bZjk޿i6;Y:"s5'&]rp ~>pYԽ3=|R~X`C_)y"XY!<@v{~Vgz>W3ŲO}F{_7_em:f ҷb1>y{g~-3#s5EխA_BjozlyuyM'gU-u\f}Уgo" KY>/GfR+WWcc{8Iù;?ةR_xyجy=gt[Ww3AB.ŧf'5 5{= 'fB>" HL\NKd:@_&lXO{ubO87~N%ͷ21ي"p=VDs&S/|X/v.פ k=և:G-&@ z4M8r*@V!ʏb{盨?,Rm~=n{}P3L}&`_l1y<+=dVcOWd3)uc)m_|}RlJ6.>S!?ΙQjmWã7]sc=*A]FXu@paޙG\kۇ.n΂m*GOY#rץ8=_iwAδ˸>Cj.3uq<W|y9PƱ!GuBX&_So*yqyF页Ρ<`~P釄z0\i^^~N쓟SL9f9k)Υf4gwx~YHPoQK7ןŶ=_ ?/h7{]_ǂ^6V:ـv99.¯]-ָ_zMÿ6I?Gb蝄s<%wp{ֶx/aq|y$.əxg⌧lDZn\Jc^=x.Ϟ@{MU>r0g~|hʟgm]=gחGswP96c3I[+c=Yh;ߓӿ;H&,֟uf~1yA--gP=_g kv6͉&esdIW7Y%Gs~OcjL r6kbsB~Wy!~ 15ȡ/< ~j#lcF9z{e u[7g9&i}LNgcύO[Őz{-xy(t{a Zi8qyLrT<o_Cjgmoоbӿk1;MPH'@2K҅ey_s/ogy/j=q_q/d g&^|AmnVq n[0 _r{)TV9ἃiFqe1r'w'=Ü/VVŊ%v^Y[֦Gþ69H3{Z;{z{L ,=e[q/ '?eyF]_̦g}!7<3-K:!6aCaD}M:n{X9ȗgHvX.*unژڮU\"Q,Xh'>1a:Ǧ?ǜC95YOk;ǽ@͙?ߟ:nuB3ov ck9aވ䏟w7_*_ ;^`וNv;(6??[y0C~hW?z?6=`W/gZD~x'7Pc$TW%|QZ^Ir7//8~#|~1C|۾v:֨(g^X %zه03.7Utp+D'۝ sox7~ 24?~,;"#B /rN zXNkbmI`~1%s>81:pt콄_iV~D>C^sCGNhNǾdWŸoh՚g.ױcN>ɽ Xُ>;71~ c<5?'٘'_jm]N<>[:Dֵ4h;/F{|lM:vAFkyұq~M'2n9GWًZ%"1L] ?LJ;ֶ>#_e9K1cr6ӹ'䮌|̞k,z=t`vjʩ{VL`o3{wܷ_&ml-9% cg9 x2K-}599W{~O3,mi8eY~rּQ=6?[1z0g=X>\~,͎9qu.͟ '.k?Ⓞvjzl͙޿]<,{߿lǽ [W1o|Xx^e/!3I~偕?G *m(^A,3B}:'u_Ǹ?E7'Jk腯Ǜ7ל`Y&ɯ{oI {8]Hٚt!.E`j"Ρ4?wm9Z☿jےs/?; -M`}6ϰ{yN8ty:ҧlt}?=%{׿`}yv?_3_ Ѱ.eعރsKov>-|-aM}\sZ]{fk}_rd)?=LC<^)ש{ns<G+~1_HI}7wML&iݻWt9ci~g֫]Oڸ1zMơj[kԮPm~̳$bRB N!+m%`%񪭭_pbջ>l碜RצAFw0O77Ad:9{9Vqɽedg!;3òփzY6}ћÞ1q4YI糼W\"a~%MsLy<0j-?ٯ ~|/ևm[L>ˍrvpk.F@>sf~96Ψt}4g) K_H;zԚoN}zثkTiD\`d94ǻNr&smG AZ]~G5Gyz\zl͙wDWweAWOcLj!>l\q_+^.u?k;P {ڏ͍~`!:9ʋh]K$pJ\Af{K!|^j!޼npmI1kkΔ o4>&ٸ3B^p].\w|MDٛ`}8ߢ 4Mj|o:i97:"kp*p8b[YaR~f"{u$z+ܶi!}[僭~3{⾱sKy{\_5K_ !abmH}WkN~N^sؚ6.ݿ4lm!|X-khޫO$n^ Ja[s>13)aZt 5Self93:l=;YPt6fw=I؍uLzUtybqzwS/ʙ&ZYm|"*=I/~ς[+66+ WTa9yWqs|9 V%& =1c\z9}!quoI>}6^yg>?#}z֫Ǡ֦ϤQκ}Ʈm1#+wA\sgtm{~ډQ/5ۼ <&RZQ}F(ߋֆxjݜvk}/6guiGƷ= sXq?2!5h ]ϹЀyS% LBņ'/ؗB|a?qsS粏SOVo7үN'7Tkqqwn>h!6;>z%< ?]? գŌ|4'ACoJOn<J7[x?ԭ/ ?9?1y͎c #%~ I c^t {8>ft9@@nqq 7}9@u|lׯ~19?|8}pۼzpr|yw7xD^~zH׫7Μt^]z~5e}/Gݣ޼8}x3|=p ?'(ע??$t!߹/x3Zsl_:ĞNߩ n_z&_-9鏿bv?/gG$'wڸroZ5k'Ϩ`ᬟ_vO1߃cv?[z-稾W9;I׫c=:<;ۯoVp^De}A=ri,z9$jMJZj!KjQm^$xH5*E/v%JD(؂yoO/~8}> /Ͼic_~r?_N/_%Ɖ]R羉}|ܞfmi'gbsޥV~<=[tg]iq x΃П>?}oRyv-o>O__o|?c~fם3A]o.ڷاZ[Ưcd/ W߱ks^gG] \N/zu eĭ@}pFφ3j%^o3wV\wv?[sα?ӹs#=W&Q_e?g,y]E V'w&?ĥ&(-WnY󷎈 އӇb\0za(F.%ꏧ$dY]_E+]_1{QB`&m޻Zd^|l<c9;Xc/S g4Yjח\_{<2IB~x3PF-Rʅ,讂~YWQxC_jۯO*θ"oH&$`#*_t>v&qMMW@ߘo;+L\ږe\ ͍$_5 %AȚ,~}y=;y-'?>se?DcC\֫ͭsQ|zcvm򅭕q>۸0}gAfV`´vVܷ([l'؞7֨_RФ9'Fy<_^֕ڍp=̜Mѻ.{#Lי]9?{Svk!'[e>H_7kvzUC|޿z]yGTFfe'YOS4$ J@@,LK#O6peBJK9P\Ӥ ю,5("dhVA49YAYk}z=]k]3Y !d:"DIaTv/)rۧ9^"m_=`qv˅d!S3|ೡ:E{sN7[ܴ\uhdPs7e8Tc>(yQߣs(gٿ_Vf+}7G}z8f\3fSlUg.n<p=P#h]xzk[^2Y=vp({aYM<=C8dCݼw6x9{eiknz^Jes2/= E ob"אAqxOj7wﲸ/?p3eL`9|,VBڋz.*J&U#* r#)AMxw156c755N$, ]t} 72}"m"qߎ5w&hǿp+_8eWrG0˵&m>>uD9pM^UY`fmر=ykx,u0j%mطt+絼%Ea-j0*:v(%CqPWD"w}㘈nc1,؝ہ>aO{;3ū.r_x]~~7~t۷Ke<|Loclg| s[}e\ij1lm| ݘ炵>\WY\g|g9f9O-d<6-^1;=}[ fVusMcP_\i>x MO+xK]V.}݈8*Xي1K._3uh-g Kj,:ܾϺ\D;=^ub[<:Wϻ_|=\nΊ?TgXy]/3s1Xqx%wlru6W/&F%yR&W[;,̂YHKe6X[=7ʹME%EmfހC6v H5oo}~-0o,"`К1D7vOx'յ4?3; -%bV6Hq{LJT!BϢR65FgHC[%<H}} \]"NUMBt[29U oRFjopEG ksuܽCe2&F_+e[1b9jCשl/}@}L̿{mPevކ8E"nSʥԏ" <ɘ>iL3uNZ3I7>ƴ}ж`#ȳbkgAusTc.nm$~́g}{̅꼐x1 0c8v φywiN:Zw^#smjBgTf1ڿ^rrG&ɍu@N<>!D66%M>"t";.cOkcά}ysr}Luuߺ?S]}8D{hx}Nuk+~WhݗŞr#'ɶhk9oQ:L[>tr_'ımEjxAl(r!}vsHm^6)c0=vn\4f^GkEi$mkr5ǣ1>?<3B#Ώ%ponޗF>xrablM\v}6/7~&"UsM8&^ay%ö/-څlO/|0rQW" P\4'Zk^CS}wb+[9ʃq}{e7!p'6d ۛ>GYƣqh7\`},٣wVi|FpܙOw\9^4(^|V/_سLN2^?)~߂ZߪN/%wgµ7-~AM,ncP7q/ƛrs=dJՁ@ނ@O}(ۣ"%eZlWƔ|VcO!qgfsf "s8ao4wo݃gז_{{ozTb7Ҹq= "\W-\t#]q~^cSs% PA&c 37߻Z{Ľb&&`E{E|{l|1{ڐm4FfT W"ǥkq]̭O{n'ƺŌ\JH{lA}S1gv0X?p xF0zu{bi9gS9]s9w(>yy 1hckgTf\۾O8V~u= >2c>=7OQk_7Oh0\ܚ!]^Jw^;+ΝNMsfC慾ͣbVuȿx,b?]ϗ//nCD%~7A}6YX:[T&ٹ%V*W { <܈'6e{fZе%hZbtV&¸#QB|DF[>YBc 춼.<m,.M{^n S}tmlr]₷9|=wQEߗbrt@V؆_벭C.o^"|_KBg/=h[پIv> r}170GF۔Tkow ٗ]'9a-dYw)rŸ>Z=zq=c3ţ1>slǃu{.91׏}U-RgeOiިv}6=ص?kjorym;L6؛ElunXS_[ۑϸ|.N44vޑr؝U9=ʁ ꯽~f ۾:> V^~;g&m\gNW;tz9<[咮ϒkN~1 |ydl5aO޵o3տyzM}/9I~sm>эX?sep^bϚ@qnsdt3 6i-uGT}m So6^'GCjw&Kc斓 ] 3˅%ꮟe<7']X}#o8y_7J4[i~I"d1j> ] ˡ d~{mcˇzk!\[؞^AޡldEg藕tѹ6|)8e>rljBmn88(ZatnưE ӋgBVB>ƼEoӯ[i!b^q;_$>+xw:}K q Wk*S=?s,}(VW{Llwpqz_%ޭۙRIkd׶wvXfgeݗ>Rc]gXLC\;awg69LƗe=k0m~ڌ6c{+tL+!d_q~>u˜(n?#fOSʟxx u`T8593 P }'qqk> w~n˓:ϼXrV>}9r$.?'Wϣ|\y7ZnnkMp\EeM}~{ĩ>X}̇߶u^>>~K}ك+o+ڿRgo_gW>~iNwo>ֶM}^{v|n6cG99DH/o,O~˯opI-O'/_园U($ʇc}_<}p6q a| \T'X,JK8ɛMXg+ OB;^m+ʼ CxW2D;EF럽|qζ~`yo¦^ $QXq"I/~/~L\-?m*,y`^EK|Xq?>5 Ɯ #"jO-pn{^ TUdggaOߎK Nf]-?opV.sJ/0Ds8IDATcvWBl,:>w}HY/_3, ~Xe-[UIN~$T_֏&Bt'1C! yi@p)ewr.i^$H\ 7&]VHk|5n'21WfV mv5v,#q_D?rc t"aXz֞Fo_ښ2|F׬COgl2'd9Hy$pnDt/]جHXS_ C ;@o= #}}i Ks*ɖkZy5UE_%,J/3˜BA֍yQg''G}DYUX)hрK1nSk2-Ow_S?ZC_/n /1+LWΎ|]5>Ϣkf.$[ʸ{o:xt7weʧS ~V;pn۶s,VO M1}s)l8 =Bܼo ِ/8ERx׸&ۤkR<;okQ_p.\][bXɢ_3iqا(چK:7̴ߙm"5+kRg{Ps ]{G{,a29$V׊NA6R_wm:ya?߉~Ⱦq?;ʧG_!vx]X*bX`E}4M)%n¹HR_n>be$N q<[󿯯1΁_> MjţX>׼9eFH̊?Ʀv-QݞP݂ί#-־>Δ8Dy 'lQ#bh^AVDYTj}~A_cbwKܷ i~#v]s }JH_S'N XN`mK)+LD"fq 79m -f7vyIpWfj-$~П#GKɞXsʼnu̚:*G'o1V}}\MvQ>91[~]N9۾<0I~e+۞c1e־ '>'~U }p_Cqp }\8 >pA񧏗y,yӏ>9_-7/_,׏2.O_,/_\n_^/2p%WNW]ߴGYK>%\/Oea{gęD{r;|}Lf{ sw.?6_^[;)8gpxk?_ ˆLҳlL1>x~~vT')K[Q wqӹ'YUcʘ&i=rve29_-q na?rSY>|m[CQT;k'=X"x!jWK[{~:+Nwo.ֶM%.>#rT7)7sZ)U ?oűN$XwD?}8bmsKr@蚹٪r37߄9Ccụ/1`0K$Awy;ώ$Yak|>< "Á|wM݄n,GbxpT̓F?/X'9 3DqiGK}_Q_xdw~H-nj4Gٖ܏Ǩ_-(ag7~yH}9`IbQYj7A? J7a< M5Ǫ$a[w gHTtokݑ,j}в]BxH?HXH9cϋK<u{`XO'}Ʌ|ỗzhl>s@_`\`6l-||:pځ}Ag{)}5{2*?wyc~1=kR<14s}  wl28JLZbܷX Ag: EQwsp_;u|ڥ|su8P}N.c}ݮ:s>I.$q[gA9EE˃eݳuRg0…]|Ȟ|K{ 5~y"$샽,o0=m5յ4WVR8 誇?DJy}Q}#?؃l{}'{cƣ^dm~uGe}3,yLOdq@nnہN7'm]:m|[E8Z8x>جHV{bt[Df78ͧoyI> g:/0ܷS_qkV^a]/̋.s ἷ=4fLg6hݗiY*F՞D\Ǯ}}Wef?k8<,)[ڧ}/+H{I>X,i[b/k^.$ZkyJ:.7-6WehCxzd{9uDƕ|݈Lyn):Ν}ڍ~Ӽ5"׎جUE޴8"^ vcDZ:1㘖6-95 AGgsĞz>[?W(m3}{/E=g'♮?9Dύ_$۾Mݵ޼>AoO6sL碉kzy73k7օsaKYYg rxO516D.qsrtثO]G|>Qf9;־+?3+; |J@()"'lO4cɽ^oӾk;9c ?ׇG4m]͇ M?~-}Y='(ؾ ? .Jf$öZvH)uتn,o47gSƎuE[.Xf㏉H6Z[pn+L4v(",t#f^^cCw+=~Nqr/wi~SJu_N kR:Zsmsslf/|:>n1}o2qĹo"1mE*qvڪ}fư)tcޢ ng p9\9dm3U˷>W'Zyl^[K\=k,*rW+=껗_4_d稞?vZ˴֯^~+W־n62/"'뺶3uAm~&~UUlbq7B>#8gIīHvg#VqkU9l|t >HOR;1V?o3z^~CJgfR~y+|˼ b95 kdo{OyyN.A!oeZ0C:D!2^s7arnSmQl;:7;[)a,7PASћW;4U !m6~]_iI[Bt,pӸvl9K ٯLO LXJd"ޡ? q{4j7Ao[}yM V0d.3cKxSX&/sYK^x^ z" IN~$fT_s{{)_I4_h޼4bG=~Q~l'5^_ƴ֤P~߷(ۺ~x{Hw}WNLJL^6{D١ Gqj+EܦK=o+3EL/|y1f83PUZ3I7>lq_3k%e%ܷ_/#ǖT'8ڗF.z-vNncԷIk٭y!r@U(JжҜt߱=܎p <#4VDQ̯ ~|G^R_-?e 5: T58˶S?zhz~]utA 6 f' 2pA( }~k"z3[T=#XCGCoX4jkA<}s3ڰ \7h6Lޤujxe4;3_|lGDx-o¼baB]Gb U´yC]"]ְ[}\Dr mcmӋ36g>oA>WѶǒ}Y)$ۺ~G`|?3hODqc%ڊ8ԺX9Q B\m!<ؿ6|f*g,o^6)c0^Zƅ5%I|h7oKr}vojv\aj9"^lOmYïyqL}3WLeW_J݆ݻcbL |6Kwn|ܚK1փ x_ecFy1ovZk]\ Y؞RuBٺ h|';כY ;Cfv~.tvs>փۏ?M5Zo`K=4/tpRs=wX}}PDxP}BNDz*1CApqvVfec6Dj@V\CN<&U>9o2tpTsߝ/+ME^믽AiܸD@g$~~YNjX'?on;5W_Ih PA$"%}Zo_ʞWH=]mH6k)tX8b-iOĵX񽣗 Zߊk3:ΞW/_kRԵ+P}~E+i6pY'Җb46w#]ԶkGkVԷyv~NňH@ Sbר|,78Ե>ЏGЗģt}6ӱsVlemv?~|&2^[g~^O^9uj =okny}[%1\"!v@]3}_*nlmo ͹ӄ |6Koݗqk.9dd+)lcyQ8zh^dvVy8nw O(A.*MaRD_燵V\,J1@oу[.K{Z;}?Vɾ3rѾ%5aHEވ4| m 6q mrps=c~ӵ}k׺lKio:x3WK1<lD7A_R@D|M}68#{|jK4.!)m:ǰs;<k(~au0ygQ\\ NO|׹G ˻6R[G14vnh?{}n^>~6ڻ1~f?[#Cf!n"|@JV7}Q|K>|/z!1) ^őD`vF Ra-VTVBk=eE7˛/)Vv+o`fʻό]Wb \ȔMP8FVwݸerlf6Lw7A \mu&7J?%:RXM/v++,'}2yo}w/5 W/¼aώp?,V։ƽ{^{}K qWk*S=?s,}(VW{Llwzx:1sr .f҉E=Jڬˏ{#G+,n;zHeĶ~\.u^ݟzgqgsre^y{g@]U%]ʶ9և?} 6p^Μ g{ev/9@(j2+rkdG|N53؞}x4~KܫfYoƞȅ9N/'Jol40&n[>CwN spoF/IS\[x1q^4D._Byk9s!zɂ^(^xFu{Ww{} { rG~YonƷ ?Da >Lp>.3.0zW '%/oG,f1y].plO L}!23q@K?@ܿIҝݷSVw6g<BI'n@ ϛ/uyO.OL^}# k#n<_Gr?Kc~oyh#`}: .38% >p@.qp <? jG\/GQç//_.7O׎Gˋ/Qy\+UPݫvǮo#^,WQ%sv~{ˋ2=́3Q9AދS$N?Cp_JM } 6`{w l뻫O>Y#-<kamƣ~i{ՉsA=;at-p=o|k+}k>;l~u\/\y [qyg?}|1g[Yc|^}xL_ti>izy9@͎jɣU_r-DK DqycYÉrp4"B|m[CQT;k'=X"xڿRg߰eʮݴ{mS.MM3}lߜ^ث=w~Y:у2=ݪray6~q9P 9h%9 VAiyq -|q6rV\9"ٞQ2ʧ[߃ X/ Ҥ9{|]8\w[gTuPlN{xudѵKtOo[ʳ{ygSN=Ε,`n>,5{~|],J,n?Y~+K' FmmvdͧvbymvIqnSDp, )?P5RkI@`AxO"򯖷$<ťI:.I}G/Gy ,]Ҟ^!xM~ue[s?SD q?3r(#X,JKQ5ɛM\0禀ocU[W`;{$*^:7nPhE?{wp.!<^M$Y@rd}y/eC۶ B\{`{PsUØ=gF4.Nк" UsҞ>h?y}NqJBӫ(z-ezNsI~/~gӹi G܃}묜*.}^~|gֵKtOgE[YgΧ{G3ǑuLw.&q՗k:FW틿8fPA7ƛ7eifSrMm1&J +聮zھNo,G>_> A_Sþ影gQ}m6\2a>7dc&C"t>KewO0}:OkT KNjVc׾ُs}w~5L*BO^VNulr-}㱎Y%Ҷ_g \H62e8{[rm9u-\nZlʺg4vwm2f p1siXcIg漨ѹr_j׏^پ#s >=v3z3kJ7bFsnW^g\g/}N#S·\;3Cڨxy&1>g2lʼnxBm x.^<-~׬SQl8boMS,ئh~ion޻dI{Ҝҽ5cb=E/9D>C1>+eT_/0ae#goow x 7!)77t큿h-l.>`1[ۭcg>4%YJL>[s}4窚Y9uޞi \_~62k7g-rc׏3Q#>e$\D'Ϧ\jh3%1>W^z29ۉdmu["|7wztCkHǂ8P<'nwY˿}z`XG&} +|KE=?v~fEy LjA){}1֚K>>m@o>ehnk5KmSݞ+xl߅Pܟ΅akM||N6ra;GPY!cءs{xQsx߭8!;}*}Eګ~6u_N kRr@uoޯܞcc6{ݿޗ1as{qsv}lanKv/R3V33O\v;[܇?3Q>OgR'o}8u򩅌Uk>FtǶϲoc9,Oʳn׾i ˟+3v?|z#`b똽xn#͹E@aGYx(1v?%]^-Yl9g[E\7|ʟU2>c{T;A>3ؖ#q_brK2>;063I+.ó_o.&7x%^Oڇ}\]cG7'ӋG3l\?mky}# S;y+}ne!3O簿e {>aWE"`}0G9cZJ'O_mg^y+Zy Yǟv +eft<;3#9ͷ|_\hԽ9RdjbkgY0 Ɂzk炑Q9ȶ 0p`hfnxɠ o?hm~&z91M`חe>e'gھxkJmԐ4UڮH&yӹ'>DmD;"D,+ulsPcx!;t5[h4{`Wx̿e-{u{!*L'!:MPCYw:'|ES{ҀM[7WE!>TO`+}T\3)ZkRa/.o=Pu#6{Է&Ufm?l\T[aOmOml,6\M9x[/bzcuV:W:,g:sqd3PUZ3I7>ƴ}ж`#ȳbkgAusTc.nm$~́g}{̅꼐x1 0c8v φywiN:Zw^#smjBgTf1ڿ^rrG&ɍu@N<>!D66%M>"t";.cOkcά}ysr}Luuߺ?S]}8D{hx}Nuy*h36|ҽƋn|^wfl7:l%d)땷H"7a^ 0K{!#׉*jaڊ.G}kXTn].y}"ض1Rb}\3aIZ}*KH7kpOҺ/=/F~Om]?s|l׉u`|?3hODqc%ڊ8ԺX9Q B\m!?<3B#Ώ%ponޗF>xrablM\v}6/7~&"UsM8&^ay%ö/-څlO/|0rQW" P\4'Zk^CS}wb+[9ʃq}{e7!p'6d ۛ>GYƣqh7\`},٣wVi|FpܙOw\9^4(^|V/_سLN2^?)~߂ZߪN/%wgµ7-~AM,ncP7q/ƛrsͤ#!;tn n~`1Uf1h@s5mdy VmIzot1whAf|907B;7ƳkK^믽7=iܸD@g$~~YNjX'?on;5W_Ih PA$"%}B{%q楮x/D_֚{}gH9jC\s_;QN6_GVu1>麝3wr!_+[qyѵtK(;{_|mIQfSO{6{v>2&lj<]?mkN>9k\>k 9[ ]?v~Nمg`)haɧԵyPzNzԺiKޞhK;]jsdfZ4C\Gb+m=ٌZ5h|v`ldT>$>K]c6MsLTt吏ߡu`N&b=Q[^sm>XM!KםdF3gQǣtsmrktngty͟+ymD;wr;=v|6q̙CZ37f/rZ!~gu8D=R\_?t>_^ qŗ~9ږ8faqXnR= TdX\-B- _8r#ؔF^i }>? L>͢=XᲴ'5{f/%Z耬 e[P]M{E߼79D<ؿ/zᗄ^m{ ќh+3}/yPq9~.$^_BοNuq~Q}p=nƸKtO;FgmewjYΙ Ӯ=/^?~ qT.XjMjL/pOvyf1~}|8M0B>uW -nߣ~36s\<3Msy<Z~^sW2+~Vޙoaתm`3Ћ]s&[fczms[V熵;DsOc)i\ӓؠgo;_Qc:o:c:yfr8fuxL_ù^.,dKGgG{X]Av>3P+?C~hGFg?ߟ/:kAqn~/r@:_}Du~ּ0fu8xY_?'wkt;&an9u0SY\XYQ}sG 1 ^87uM&] Ra-VTVBk=eE7˛/)Vv+o`fʻό]Wb \ȤÌεqHq-G3eSjs3A:\muƕ4-Jxu^ LWWYOX1: {d:/K_jN+ ;=}_yf;z9PqзYFϭv z_=`ߗ@&m#u־?UzTvIYn;Pϭǯ/xlVgu\iچڪXrZsHc>7r8w6Iڞ^|9?3.V;Ǔ`-ۤ-^+~V}oS+5ZUP1~k}uD:@9:z8=x'7S=!ȁ^\̉3bv=Y/g_ \pMSϞ8t: uiP@z_8a.m:L _[ нU<\(BY={U~ H/7sa^gA8&F}Wxe3q@qP}W w}sqY{UL D}ԟi9= $!HR٤{S^LJݲ.qbqOO[^Hܕ8qp1`+8+?_ >N }|8/w }W. %q>*@xe ,]//^\^>gyrzy]#SEpà y?ϝ)oDecbq~co{ç/jYtpBN-,d>([ǵ,_i|]彯>}:cNuv[m*bMf;|/*}wYfP˓k\_g˓?IxfUvFRdqnO^9+{r8AK<\?Nx8ObYq7A? `L75 -ѕM$AX¶we^Xp &͂Bu.ٻˇlgB-m;y7ʭWHhHxI/~/~L\-?m*,Ιxq==@KԿW_W+=1񯹭Ǐcl3P}WZ~=};/5B!}B澭^ֶxqC}j}Cp䋳@Y0~I)؉F$IllN- vc3'|9O{'_t95jI 39wcos릢>(1uf9LY+'f r1&w9_Ϋ3\ū3J>uq3 [>|ϸ[a:r{;C{=ߐp]#A;z1 ;}}3&$o~, ,?ߍȢ wY)—BѨ:EB>g ]FDMP.@\VP"<%anjm~T_x;=q7?yq_EsQQrAƶԷ][QlaşXɳ&٧۞abr-=D*>cz/lL瑶O+RȏTq7y??oO~tHaמ3, ~Yle- D UNp#1H!M[? YgK*6mq<˸3ln/*9$|!DQt 'H|# -׸x^[FS"]?(]>+H>Ͻp.xQ Sds>gA[OÍgoe鍿'վ @^"Q]pG/-r^V /^no~]<(kdiI/\Yuj.S{m硤>]@{e9'sÉXފᶼm; {k} ZĮ'ڵ-;)x mQ9M`cYT$򉺡-pQM$"ZH$tzM8iM2Ű-gaB:ϖ/f~`bI|pEu(@wũoc'cmgF|ߓ&mNm* ZI}ױ"}SC'.״糩 YOvzJ\_by0l:.W[ݽykq:y8Edmu)ЦD;UbT; ٮ%TN 'Z⾍W4籸ȅ ߔproC' |I)Kapb.FbD"WMv\'4Ҳ-8YDJiI7 o|SkO${vl,X |RʳP]}\P˳w+; 9ȿf07u~xz-_pP2هX_٭C=ův]q>[ȳyv,gҦ"&[jٞ<'|,[]7 gK~NI 9%.w7'a/PIȕua#dO,Kq^"vL},Qaϝ= _Fq;4>NIaxkgj֊|A_z͛@ @/՛1r_VzKxMseį7=NxZ߳_QlqTvb鹿H ඀.@~󟸗 yO s@yǘ"^?Uh[]R<_&/ ZxOBF ϽQ@${`GBvts収];G,@x1}jk LbkO j^oAsns{⾦K>`~ˈ4fk~s>}e3e^G/gRCŔhm~඀reS?y (ߙW[m#A9zzq?חi_(vv}{ߜWܟhǛٟmXa )@ ,$B?e/_ ^/h5YLȧbN~yv'^bK~sS^ "G:gdGއVy^}>̷x՛^ O /i?_h 1! Im=ܗ8UhOvŌ4VŁC8\'o/'R_ޖ2F X@;7P/~F>!:D(徽"l=Ѹ-Fa~gD先rLlc#Vm_'[/'_`{Z~HcٲylcU Mqi ѯzGo孠T-xH0_BO݆/TDBVdWXqҟO+?,նՖͧVeD׿~bo}>\4 !Ljq^k`;CKp ƭv}oQDes/6Hz4BOcJA>Bc?['7.5u>ţ˗Wq,On_^/?b~d ^W7OLYYr%\^<}n̮/eW 71 {zE,4O>?i PVߑmP9몟>cq-[1.vmZb*$~'S_}<;'m뼽|(}*q-WZe+~Wy﫥t߾Θ|>v+g}mmwmSem,'grVq? ?^-<_.Ͽ?%}~<{|a7oWevɂ !ky1;fh=K ՋUqNqWT~>F_P= -7|o[+F8eq]c-<$yN]-3}݈K2σ|l)/Vqi8ObYq7A? `L75 -ѕM$AX¶we^Xp &͂Bu.ٻˇlgB-m;y7ʭWHܟ$E/o둸凢M_~_}L/CBw&䟿r?*;/Yf'g)j! [U.I"H(lJ/Xa}b<,`x\_xz?MbQYXq9H\./ :sy먱;,ބ6Qmu.A1#x&3Kx1_T _^O]4E$wݿ2`n懋r,[qvyV/{}ckۘWH3^eզI{oKYO:_Ӝ9=|B䴄'hkc?IOBoXoV?/}eev|IBČ@_ܟy@ d_Z!~?}+}5qt:v(" z!-q%'Fvb@wӋfLIߒRIY_Y(#]@SD+B[u.so_d6إ˪ Eu}eĦR?*uz7o~, >ߋ_sc\u#ظkG%* N}ICDܯ}Se޵;{'uÈP(&a9i ’*zE .l[-TQ/k^oV`[IXHO۹nF^K̈E:VF纭8f)ߴ^AEԜQgVxbk.kN%(OtDeX?o"\ILP?c^Pi#硸_ ƞawv{yDW!q͈Ĕ, =BJIDYRBĠnh)6LJ|bMۮ.}qE|~$SȂV^H>+zɵhPOd 1#[{NeO^۸iO']jY".[R{-0졈 I0_Vo} JRhܗXv\c_ȶZr}m+,uOo~\4PHD}hqn@^R8u ?.0˙$j~[MªHH[S]]?[/F~IX$ȶV?o)+_Xx1Kj]E!&XBʼn n(8I5(T,yQ+ԵB8x-Xwr>|<-ܿ4\0)\}Gd;i^Xmtc/lOЏc$OeX>Qx6)>' o/}$N,7} V =l[@>]{^"v>׮m|dߌuiLOijϱmr?(΂W ,0-PMb"pXnlA8i[)"0!sg~u{?Y\h ÜO*'$YTEs}ؼL;U'5ia WΧ;f^wgϸE5ZܶW;| . vއ`[}۽/ ;P?F.nw0oK]~>! P$* }.n+,X"EC*pc siٖD%UҮ{Z|!7Z>'I ?^9/X{}7\7e~Äc7Wo_c,J R.qX5^:v[ĸؒ{]{_Bw>ͣ]/^Kx0*zMר3G ˉIaxER~a }\8 >p@.q|'7/uy\?2x'ifeW_./> N U7f\O2ڿNr|~^׋\pN-,d>([ǵ,_i|]彯>}:cNuv[m*bMf;UOD<_.Ͽ?%}~<{|a7oWev"/s|!/Rw >p>bdIW/zd.~4\Gg[?{ڤ絏oi7;ˮx#"qR\$Q@y&Gϖ6g}Q3sq?ç7"'>iC^¸3./i({ʗ6ZSr_~]{#"ǿn?@bL,1x} ?mE2'/طNjcL*S[5`=]`x\_xzIH8W$ &[[)qC)lu˓\GMQvoc]A#x&3Kx1_TK~v@^n7=+d"!r,[qvyV/{c16}.6MR}^ʴjt9{5s7QGдM%z.lWn{l7sXbj+kbnQܷ3&ѹ!syK_w+L4v(" z!-q%'F~b@wӋfLIߒRIgcY % 4Eık#Eu)DD aōMJiEP.KA%:ԾV2cmbS{X,ϋJ/ڃ[$)ύ5S4r׍c#+<3;%EM?ښۋ)rF\ao,V#kgwc=Ǝ^}(DZ{~}Iǹl'Uxto2=3LwoY}vuc^(+MeMڮ.}qE|~$SȂV^Hzy.\A=E$#{F,ƌn9RPc =zm+>Ow]jH+Zԇ޾|c< >,׈cY<:Kuֶ޵1;|>}$'aU*I틮IlVuȭƩ/˂v  j`<[}O"Hg>~+Z7Qqcs1~^ 'HA(Ķ{A-d(vJxph}hu@<)~OkmPcEV/ޅk4r=a [/Xe妍&( hS 潛ԇ׸oKWp?aT}s]Dh4/|7ֺO9f ~S4~/Gbo ̱X=%D7mD{X$w/3F/H3cq>y#6$@ĈD*4xE}¹sumhd1/O7/ ]Fz(׃kqiGb}"KDwpn}a~3uA U] 潛[DoqgsAv_\5aȉi8657;ݖ&CMf9+[1ܖm_wo_wm:ya?ڵ}3":qv_[՞c+_>!̋!ڤkZP6(.'&4Ng7 -SFrj&}lbVn8+ "g O*'$Ikj˜e݉7* |S׮΄v[a_d;O[cum;~eslzo\~z.4/]sK^_)E[g,y~%`3Ͽ?w;wt5?ŸO{km .k{9 מYD(Vבmz1@Suk +F,J/}ٱ@N}؝<вᛒn].QD *O%!ڸϥrEw%lz;.4Ҳ-=q6$qFm=-IpWfj-$~o ΅՟_9>l^ڳK)ۨ&DZ#f܂y}K*kf1.n;G iqYRQڮO%|<kTӣt.I;]^RhcluU}.n }Xnw{e}kωIaap(Łrm( .O"=:qY~へ\8 >p@.qp pY<^^||~fy0N3/G\=x0,\-7ޣlИdiO_Խ)== %9fsǩ>X}̇߶u^>>~K}ك+o+ڿRgo_gW>~iNwo>ֶM}^{)yݶwyG˳;~z}D^1X/_^>?[|OO=^0m7[s"ʮXri"˃ O=_=Lӟq{ŭ'I,1R0Nb&gv&gىnC 7oa )Wy=`y(DZwd7-hE?{wpqm^h>-zw;]E?gL$Hq\k`@የ_ q<#qgE3 snx>qNwq?*q\>"ZF VXf ;S?> {v ^j`B}[ mG}SV@;Gyp8,OxzZ=&ӱ,cB}Ă|]Y]u.Or5vKQmuQMGM!gݗAe/llȜJVfDN/ӷv$*8KL߶hmYZ>j\^v iΛAԷi+s즾d\U833]g_m7:>+y,^~ױm}-v^z|ϩ[a:}C`7- ܾ%dH܏^ Nq_pzqߌ> [_*)>qk ?̦==R˃Й?|?VŃy]y.+sIu$6ԏCOϟeA^ܧ6W_~{ؘgJ[yVYa09啉̛RW˳ç7ܜӗ|XǾ}zĥݳ}4-yyU[<3g۴L#E3mg|pPȳx<#Ks_Է-agWw11Vbb}r{`-##mz.0c/o:\|=gY&ؚZbl*]]Q$ 'Q? s/}yObB#pشP/cϰ;n,zna=7}a]?(]>+H>ϽpvRpy%Nu]q^g0TW)QsL,ɗxzdWA3dG{>o>g⓾5E+ڎ~GEP}Lǫ19jG{;Z }=+ ݉yI;y]y& nLhJhؿ~ln߲ Et_M;* ;L~VR߾fUGjR _ j #q$,B{Ad ~+Z7Qqc >6}3z-ŗAx=\wlP>̖ m*U(5+d-;Mrܶ b ?[|ܧ~6ϋbdŎ@iCn]Lq}d޵-Dm _`zsK-v+m|Q:oOjx⼪l8iYV:91ʻ ĺ[ή~qN1q~6Utc;!}Al疗iZz>/ʱ}#S僯ߚ[xg]pq?P\W Yx+m/0bݤ/7Ɇ^ `ܽ ``$dne7sN!8CBBほ}XogaE}¹sum~ }7xZܿs_7JZ1Œ˥bvs%+ׅauA;.џY7^qC}u獜U>BɽKkњ>򙪷3ٱڵ?NvQc[R"?Mk;rhŔ[k1!ʎN Uv45(* ;~Sgpop[öݽcYߵ鈾=׮_;vj[T{lɯ8:{!uz $C@:?`wYlԷs-"ĩγYq?㼘yp̬xB$%^Y{K:pG6oα^ΎjNkLӘ/^ef-tgNjDy#o1Hw}'+x<ڹ򌯋SF.nw0oK]6i(PRvWH(K?n<."pc siٖā@^?? IpWfj-$~Og[F=X0>眱볶y9kUPDX^8K$v.DV%mÿ5>$2hM۔~q=O./6Kb]F L+OC\8 >p@.qp pY<^^||~fy0N3/Gpq<|͓rԼNW!W_./> {|UwEZZLs;m nڽbm}ֶ~ϋ}6e:].AXmIzor5_| ./t$ ׿y|տ*Nz<\</{NiýQޔT zrRqb!(-$&o~`7o"p(;rS?Qm}˼<wM"v;EF럽|q϶~`y/Ik=$o6W< br%I/~/~L\-?m*,=@YVܿKX?P~ _rĊT qp_+=1ñJ}ϋ~cۿ*ЂL,1x} m?opVp/sz}kQ_y8>>ܼ`B |z:HY/U_]_'/&qVՑ" ~(ZaX]}:'a|$޳"f[1⾁EkZs*/<7LBJ Zw`~d$G}DYUPeGI:d ?[|ܧ~6 4\D_ܷPb~od5ݾlk1%~m־ȼ* z hvd_4h܈}u ~V6:W|6^`)!I_?on;%ړ ǽ@"{A0z07vG/I ^j R\RKl!) V'M]u-%kK4D4F3iqD[b1aX ׼"tyE1bvhW!;ɞ9PrqmFܟ%ׁMa½myv@uV?e}צ#>׮_;}mK^m>;B"N$N"XQ @bQ+ҝgDEږr)4D-|L\Ŭ}}ޏqV8D8TNH/[c+2y 6'ɌZBW߹Zly׵w_y B4f|?^}Ҟ˔ORyӧϘW}}~-J8u@DWN)B,'zT; ٮҗ;[⾍W僽~#Zv7|Sr۝ +*nE' #'(f"1 !,4ľ"q 79mMdqd*Njs$]Bn>|N'o58VBm$a{u:{g;SƏl}D2_ZhR׻^PceB5Q40>D[ݿٯhڇ~nx1Xغ{ɉIax5FkA$Чk{_zn")f~KFHW~U8~ ?O~& }\8 >p@.\/_./_Te2'R׫J;pwZY>|m[CQT;k'=X*[}{_-eo-}u|}7^1>cmkT|žo2.mwwd0˜V8$X/_^>?[|OO=^0m7[2p5'$b ~}3+8ڗL#&y]/#}B}Oܿ\{x^9ObKYi7A? J78 Np¬x#KN C|W$b׺#YnlD.ٻˇlgB-mNVr%ڕCHT8Pt|I"| $gz$lhS}fukbq|ńOo.G' PY,r}o~q] Ze\Z>ԿW5{bt_sS¬f ;S?> {v ^j`B}[ mG}SV@;w_nKOh$~~p" ϯ_FudcE%:fpk+Ŭ"P:sy먱aݛ*nWq7_w_ge67O([Yn֙x r@D{F^{rmx/^զ2_*%Ԝ:6Ģ ̣]<دHv/x^/]~Y=/Fټ0ߎpzNzWʓ5U}ۼhܻ}'qܢogLsCD 5qtء3w+HdHܷ!0S^7cOyֺv7$xdDHt "? # ZfCE`rYK#E#Q3&Q] "=q7?yq_E{p{Pg.aJ٬EףumM`ص3b|1>ݎ 6>_&7r-#ѵɹgY&/sYK6mG= IN~(P~Y?6/<[Ŧ-`;?qg-\ q_ $" )+*DJ(ܮEDj7WbjAY_G7,omEpklH&5GíW).rrVԺ鬍h xb=avdҧ}ϑѼfc].2TW qwnFnC?;a\s.ފQ.>>suX$pnDt;Z߾fUGjR -ѯb2#}6ߊ lo/re,\#zײַc=3_fx n(tV ,HxLBoFȑ}h@6q_ }g˓ԏyBSև[*S2V #tuwu+l x#lCcקic2M[ƪeDy\N!}m|TWONP^-"?|>.:~{D~gS?y װڵR֗!AOj ³|EH ^mx9&}qhO6eܼw#_؏ P"IWHDHTkhp.\][#ENXC}[obMև[*c1,D]\ ޞ@ ilcRpd\~^XexεPq?9A}>c>0ۡr%Ko>>jWg .xD1k޻\yQMa½myv@uV?{;6Ivmlrdߌu/.NmK^ER~HHE!'*PhbDԷ(¹HR_n>B"T˖0!sg~u{?Y\9c|R9!) ˜CE9Ss2uz[ݮ=-v\]u_^Qq}} yBǶG4/kwL}uc]`yFvҟWi]01 z!U)ϣKuy(f_h-šWnѼ}sRq?%:RM/vNzd?aԋowDYvءmzB^q -)9[mg!MQ!=lPHo+znChRq 79m>r8#۶랖$3_?ωDms/}2nCijKf|yx_ʜ36Dlsm31ۯyb؜1>)c#&1JofV};۝&C닒ύ_<^?xEƯY[7ؽ{BE䭺Nsi.IA.>`cU\^qξG ωIa6]`Q4'k 4}p_CR}ˈKZ9}`_?N<8Jqp }\8 >pA񧏗y,yӏ>9_-7/_,׏2.O_,/_\n_^/2p%WNW]ߴGYK>%\/Oea{gęD{r;|}Lf{ sw.?6_^[;)8gpxk?K=w$l{+~W~-,;ya6Go^3g.~a>Nk|0g~"ye;o>;Irkx?xt]nj <[\Soy_Xʁ488{ĕI6r2"B4S89|/v?-_,PYlW[ٟd{ ھ߉]>GxGqrnj\~4f3e&E󞈇y/8}},d>([ǵ,_eo~ZF<Jy﫥Da ܕ]'iYk[ۦ9ށ]*փ=:>Y>7|~h؇|cH}a 'c9fq9P| ݵi2ѹ8 VJtylUoBƜ!̉~}z{{q0B;,/4S@|rhcgo>}g8Xwun7O|FNϡ~gƳrw|;{͗ڥ|su89yN1Z~gO9I.܁_c:p:0=_8T>C~mmvdͧ cŖkB&4wcinpWC$lNxj>'QZ=Z׵%=vbuIÿ#ӗ# xy_2YۑQ!vvSwr@0=VbXkW9|Fx㗇ާ$ Ei)&1yTƳM_#xjKv ,a;{$X^:7صnPhE?{wp.!<^M$YHm𿕛y0or{ w/|/8瀾3}k3FfpMZם|:pځ}Ag{)}5{2*?wyc~1=kR<)N^> !u8xO[?!h}ЙByg{}f<+'~>>x _.`3v4l%N/TX_ΜO Iܿ,#nl8]f{`Ml+?rL~y"(ϯ@|3{?]U5|˃R-` #]w-G^/ X}r09z"&xݙ{M0 s`F=ܓe6)隬#h|`m~;ЉfM}+ZG4To˶z]O٧Fd ]/ѩ1q7®-NְςٮK m?yA˱kő&MrxTn?GxZegn_и/^/7އkN]?q%_7?S޳0{sg_v_g4jȵ56uU7&W®]qlNs8Mgή` 4yPQ|8{/%^/<`+gXJ[9LCSsI)[qa+o#iN(`#}_Q7Գ8ZW=Ë~WK>ă(\kn\݊u~nnmLg&{t띕s ׶|2N7{f|Q~Eч? پi4 bмր_xLGsMݵ޼>AoO6sL碉kzy73k7օsaKYYg rxO516D.qs?bPC]χܰe"?uz+=vv6#[~(#9\6\I Xm ]፵RUYhnΦ늶\.%mطS+<絼%G"m m0>ءs{xQsx߭8RCu5&gk6?}x_*cK{u_N kR:Zsmsslf/|:>n1}o2qĹo"1mE*qvڪ}fư)tcޢ ng p9\sZW̟y6)|j!cy>o4.s꯱x^r']I|`^~ќWizi=t.ӎK[WzbGk[\Y{ȼ8.gvbFھ\֭w9 K@C 2w_M,lƒ𸵭>#g-G, tu鳝gc۷рn<<-Ѽ.2δyD<loc  /oǶywtgdʶӵ%%|cyv1шw2g:G>+u\9=|];3haHXoGO_ϫy4$l>}gQi;Q?yf/5GJl̻ yQsйHa<Ǩv8~C}/9!T/_A5/Tm-Pmg`\orT]<yb{zZu( 6,4Df%}nEMvH|~1`b>}nsQeQ BV.kF-pԖpbm*L'Q8zc1}o/%?-ڛTlsC^mޑZ%#pþNgiIѡ,&.o=Pu#6{m212Cy"G8VSSM)vS?zVf^.'cz~\_yk xk$\g}P*)mʖ(?Cs~d؎x[^P7{k_m|9g5{^DS&ůe߯[=exoD']?附Ov?߷z. l#zf7c9ـ~JžuY2w8Gd[vKvcę_pV}z+39Kp.ѧ>/">FqdʝvؕiO,lg.#w&Uck[R_ύ_qL߲i\f xBvV*W~!aM>ckj?^^\. Fm5Yb2WFPꄶu})vlG7/' f~xL s=jBL7`j[.;OFсP~_0.u T Riz~]utA 6 f' 2p@IW*@ϩE8iU3VᲧDyZ` akkݟo#^9BֺKFmb>ՆɛW`ƣugvywchEMB/ NMHuqq70mPh°].y}>GV?yA"kZ8qfw/:E=}_{A?'ɶhk9oQC ؇-e:?9猯?Q\X"` v6vN_Cz>W[f;9cݵ>y,'6I䳊` %5|1ـ~m~uEVVܡy sEh3GmIޙI Y,QyŸE=<<$SuDڞ9FB683y۾/ݼ_ٱ|3;Pg:_7dk)B }6ᯨ( wŘlrK5t]c)7F#Vǔ̕cߌ5YO1׺ǹ@4=܏"#u/(bm%MlO?v7wR C~]d}>{'`~Ш˳`6utG'1Q5|P2C2t <(m= 6>Ѿ/e[b@D u]UņhCs-Gu|a|5\0MUpnbwx%k73HPA"7"؁uޯ7X,]q~^cSs%aܽ  9q_)$/5P^?Bܟ=֚{8E{E|{lov}W|;t v0:#mp:q}?(>.Xbn}u;q-G7-f|BVA>YFlm/_kRԵ+P}~E+i6pY'Җb46w#]ԶkGkVԷyv~NňH@ܗѰ|88#[o|s4S׺l@?A_I2WwC>Ńǖv9w+Yj砯׃__ [{l!-X+3Aq?zg5yO757v-}Js}.Y}oZ_; LNƮ>/_7L7لZsiq>ٷ5S<稿hQ\;w/SV觎=f>t}rT-ן͍`^?O?f>3Pݛ'u'4vnvn['O~#<}1_Nu,w0/}qdvLK24ހJ~KόT!xV L"M ̈7dW079_ϊ>s<9b>ݸع1DE0y#mx#6h6L۷%%I˵}ϭc{+WvXe679CKHkc}s=x mac4cO9]{%f^(})!֪/GGm.=o/潱!%/E ϝ9oVgd%O_r\HX녜{MNnƸKt@y},cd蔏p)~u{q5qƫ̣'i޸s^6ymo$]ys|gH0$T'|PU}Ųږw4%z~Y-g#0ϚkANV[ooEv#}zqmه[_֏˟H}72yuAog?}q'QTG5utdYu.؆ԧmþ[ρ;׏;</X ]/ P)/j2;}_TkKڔx)$z!G~Ibd9P@Z*&b7X_f6)'\7 !p Jhr寕ࡳ_٭ C-O-m3 \pJ׵l7QΔ<ԆnpQ6hatnf_IcgqޢDW]G q~! +Fg!yoLc[_GfjN+={`_yf;z9!y_m3znekǏችߗ &m#u־?UzTvIYn;Pϭǯ/xlѶPxH6VlZ3lzm Dp-2r v1? 걱sf\v9kf~|<-yV!^f>:GX'f `z^.DA?LI"VS}1W.qRu1t>d|Kי̷3c:/Gzcow~aClGi^(cӜzw˽+xt}'[cm>+c~'ra/Ӌtxuɦ3! 8'Cnӫ\.۸K)^TwWg}OVqk 0K QecfxFu{Ww{} { rg_"#=p3M\}f"ņ{/8-@H!}{{=Ξ[_<ʿH:~vϿ7yDσo5^r/<:!}pp9ɥ;p?eugs{^K* .t-R~|:P}RʺcqO_^1ܕx9Nlei/~oyh#`}: .38% >p@.qp <? jG\/GQç//_.7O׎Gˋ/Qy\+UPݫvǮo#^,WQ%sv~{ˋ2=́3Q9AދS$N?Cp_JM } 6`{w l뻫O>Y#-<kamƣ~i{ՉsA=;at-p=o|k+}k>;l~u\/\y [qyg?}|1g[Yc|^}xL_ti>izy9@͎jɣU_r-DK DqycYÉrp4"B<Ջ t_'-[GX>ؔ񯶲?-Ȕ}|T9Ը4./3i̊IgFۭw=_-ooe-L=eFo;b=9̶iq~< ˣϿ|۶ˇowײO>{|EwkxKy﫥Da ܕ]'iYk[ۦ9ށ]*փ=:>Y>7|~ޥ7*s엥=X8!Cڭ K+Wlce}38kt`,ZJ=(y4=%Ҙ FS|s+G>>jCFt{0?<>5pq~q3A4=b]}묜*͉58xOCy㱟n,vtmKyִ79ws~:{iѹE q͇QEfoY܏ˠӢ]Eɠb\͢'o~$ȶuծl4َ>^,/n01mʜ&Q< A=MBs݃u][Kc.^$1?2}9R=-ϔ8L>|vxr~ݲ#=L౷xC|0a Z^!jNU:)\\ |zќAY>s=Li.oEoq~:7ͽ=>{}soS>?",ݺvtlu>4s\yfq8ΓYc!!o8]7brm=[(u?Z}vq':"zxF~y",lʭ/W?] +f59ʃR/xA%FBp@]Q+6m q?YaD@a@6iyGISy{E7&1GxZegn_$+~c2ٟΫQ>{yX9EKʵi:fH{wa]3r!4~Ȕ}fLk)Ce|qX`6}檬;\}Fcw&cր) ǡ:p>U>>Dyh΋+GX?Uz5[82c1:1˞gZ>vدt)Q9`46~~zJ|V@02|;= ;]>l23~&SVXߎgy.ـ;z?A|XZ\yͪ=ņ#Fn $?my!JK')+K[3v+SsA34:ORFEaf^>|F:<~盀p}ҟr|s}COr=/R[Z/㱵z~SsqQ*1xOHVW-bgx{wLW?t,ţ~R-|Q/ud9grʗ^s}SiWg6QY8(PI jK"1 ZA*? #V,ϵ/#m@o>e"?QvPٌk4_l}%+Z~pVڤn(xt K=[>̬M;6׵19o F-"$ 6xjgHxg;C{{}3]q!|GjGX1'ש59q(K\nAZ溍uɧ2vWƞ?˾-ׇGll?+ϺKs\1/H]<}և V崙Q<إή\J+־nDZ[}l鍘%c˺`4}!g⡔zƐɇV0;w}xUgvֳ[(n5sWng]."[;i:{vfiOv+;lϝ8.jc$ܞ6[{(V}ŽSЏCgrb[v|}-ľKv6$ ~16xU?iquxƞO/<9ss Y6ϏOmg=l{ς.7g*3.ΗѹS|j8D^;XDt6}q:OīGkS[D-nd{zZ&>=|!)2 Y =q{Ac V@??qUOHۋTG/̼t5%@+KC0c,[4rM.2F\)Go=:k\CK~2̬ߙUu{NwOeefdDdTT9n]Ӿ#{1Z݃Q!D0aX}tR?>H:L8 osSpN:`IQw-/kW}UR.E}[z?MeR.ŗmԳS_^جuݖZ%mE#m4ˢcQLx{0xI綎c !acwXYڟ?6r6nokgyz#maiTnAˊ tanu2FqyPύ:= Ts"c2N7M@&kV}=F^9e?8 w~v`N^k XU~5^a|ihkpFmq&ZGtk: u6:!T>%}I܀vo9Tlh?d{4ʾ+^ȘQ1>',oKMQnu@lʇܱ/ u [ܫi&lY Ztk ~c8 Ά~֨&nwq-\Hgd Oۈtmrly``L3MO?*jO*,MƱdtc'wV6+U6"1uI#%ع61epvgtoҺؒ/ý|l>)̫N1>,X0?^>Hq/?X /cRF_ژ&es;h37.n@i 96wGq\utݡn(w38(1KPIG'x sdbA+iv(b~PׇO7"[[yXBS?V}\G=q: &8t2c3,䋎J g=:j~nlQr Eϋkaz@ :*q+´, vYf׭9aqob,柿ϙyi׀wzE+1Իk sĔ]/F:&Ѿj+1G2=z|d=Ĉz~Og`akG`ŶKgNZ P|`H sdC_R1[ 0_l;mtQ}|őٽ?u SF/n>wz{\WطOmi*? Cnv_6wvuWj8h?CQYgAc'l93\5Pg-=XJ>?$4W{J݉AM&\\dx~h<6c\\at|al[d-/T:їbPekXns #Mb:[Kۿhk?``-/X:!'LYq[PgX95v`Hgmj7amٗX:CQ*H_91Gљ [惿qW(1'Лx sIWdD+jQ~1V⾲Y4^}>_.H}/_\n^g\bg{(m]^PDlcS|xڸdžó'ݱpB6u08ts` nYAw0k:F,2 bvPwMV\c:'ｭ=_%+e^{q (Um^V /+X,+d뢻]D} 3t? ̃Epe°Pϙ+6%`l]Wܗ@b &OF^Xa?u }8#s].^ 9=Vp\n ~~ˎ.u޿tޏ\3cFf߳/G^e_=Ixڎ(}lTf~Fxp9 dsxG8 TGx͑?T%v>|㜋 C 6'߆3aᚅص~^=̠SIܘ7n\FƗJ"˦1rnѹ^-֨ϥN#RD6-5{}y5t6:MbrrίŰǦ1 P5[vt|ʺ|Ho/:ٺvdScF:696:A[M{ +m~9u&y\lc,c} [ S}|k^yFDbwb?>6p$g}(_s}0#{i]Zelv 1\?x/9QjQ"yIg0Vqj^>󈍅c/s )cȱ9W*lz9`(qh *kfRI8Xw}m`(:A.cm&OehXNdC;vkcy!/WЇ1>X +igM-$,9^g?#[ uQ;E>Y Mu.~L1'ҧe?{s /PYPg&(v̑?z(t\*JV؊/!p/Əo~26ƾKXA2_*× }/|u9Cϗb?"7#zUkc_(Z^Ffn1`{ݟ_m)JG~{Dsi?ڶw,ԱWt۪ϬBg?zN#Zp6/HrLElJOSl]BUtt,eF[a FkC-TYw\8[B8Џ[F_HDZ<=؇zG}!gxu4b%g-Fĵ.8(W5Tfs zi.sؖ ^e^ӓ0X39\/^-HH{[|c~⺩τ[[߸ϔh/#E5u?rJe.q_8ҏ,8@&;[u&y c/5)zS}:?˼׆\0$y#VP:}Ml<.ڲAɯc=?<Y{#;$ӿ[q*S_=%a7fiӍE^]>3|zlt:ñXk#K9>pj_%M/փO(rPn-T_ `~͢p-VF .dkuQcXkgZ0Ln>)|mMzz[[d؇xA`Cڑ8 n"QL$L6/!*{ُmk/QSq!_Vtjb'QF-d18^:0v-dc5-Dnf)w僭Vɹ/8} \7[5ٷe??z :.HBFmƾg{_'t~t.|cs9cR{,Pٖ>Зͥdlk6!\cc*487_5vϊ(=,KB6*!WK_{%3ku=x 2|_[hFkC~\|_±xm.ѿaţ7ڿ&ۛ~`8?[mS+~r_96{Y;څ6lۣabsoqT;ݼ,&-c\k4&rxLeY_< `6z:ScOft쥍B8ܑΤ?;> D6Ö=h X6?>X~ϸto}.,]ͭdݛ+iMaB-Y8V΍ C~2uØ תWS-|k)|b͕2|rM,{oӣEϋqtj:ѼNT6D81uRS}9s-x 8?թO}O0| 9=T<fO)_ßlK$97x$O1J"u*IsrRP*!y= \%s$-c\_7ǽ:(~s+0'˳ "Jo9I/ Y?ToK~&yid>ojq%IX27/YO WoPP|Y=jg%Os$yPqSR؟ix5}1@q%q )/M/d~4_~[<_}$yXUzqzS-IkGȎ}X}?=6c"7OͶl>En/A>|0K o牕WDEho.%޹8>pyOW^~`7,cO//-ON?YΧ?O_m:{jg)g{e}v#(}w9dwa{V=ɗ9}q~s񭬿Q lyc\U}9[t 4vNSL*Je΋̳_C8L;D~~uzO;|zGϧ'Vyע߭ӟ~<=oINO>~ځ2kc>noWV;=b6% }]I;k^@/;esJ4!jbOOT~+yϣoHG{Yoǧ/zƃ 95׼gv&v^V: 25h{bo7Oӻkx]?/I;;A|`%?ɍ`\kiς#lgd %}GJCn|a;%H3@n)K+eOs\]q;xY.cg=&׵co=kF6  Yfq2p>c7q+㻥x_]+n9sL~ ,ۏ)ycsq߾NnHv!qe |I1&~ ɮҀ/FR!U?U^hϧFb5F{uz&悂y1CrbA(a^|.cyUR~s1ncuAuѶonNW2f_m4D/Q=] }) `I$PT%T,Jj)Z] pZZ/suYJq7(E_ǧ>ɍE_zhxrq͢M:U7[ b{*Kn.nK1w=h]ZlȮ [y[m֜ ʜß. )7w O;y=f?$l!qdgO*^9f(5^&>{Y5]KҋO=b@is==g ,n4֗ɶ@D̉W؟$c;-Q3B +XTvˮC[{ڸd;]dִ{j`kۂQo[EߊXцP}-M}0ΑxS1)Wו.;X̽|^|]?xMj1|r LBgm=.ОD֦z 9:|wR U0}v=Z_zV0[ɵpBv<+켅{8_q~P]3c N)}i7-k:ۿ&.Ru-5=供ٛagz*1_98Xϕ,JyLy>|?;\V&C{x>@r@=oEBk̮)~+`eH7+9܋s"G^Njo+Ykif\2:}czu Xc|][.\?^c~&6~6#zXZvkfQ|pK$cS0ew~V(_*[ܛB D1cBM__`z26) o`XgV:ȭl~T Qs\%{.XoI5\ɠyM纀3$Dlҟܧ>RʿqaY%8 q"09CX!C~cXEohC갘 l=UNnrxeH\\Y˽Zp&۷} tpA\~XpmCTs㽤/usp.8Xz)yVx]`pcKAVg,E9c#aqkczmڱ>DzeukB=- ⇵?g+XHd-y,co8wqMcu|[~sG,u1mn28]Z}v3|YYܦK~5L^i>Qs*7üך\el'RU>tqݗ@ #YokE;yLuvec8& uYƢv[CO'tyG!m_dYuP3ikvy-¶{^}+ʱg^< s1YFc۟yquɳw֣Uk'uRƒ:^ !?h\[ՙ#jhȱ/xa|H@ku€mc _{s"X3 <30s,yMf=/KQdy@v?(rA:|1\'=y/ʼsrn\s=pTgL,G:ñHy->]YGO=cEaEA&b>3[O\q-\Hgeз֏ue5v39~4l^dsbۏ)tu5)TR#{XMhl3'M/QhO-vL= M{Nj-owMt$ا(X ź)I^D:Q !*C |Kx_tNSaY3\A6ͳx% &lBC[os>ic.<~& }n+䃋r6!17<8Z[,)خEVmNhਈR.E} ={YrC l(0Jfp_ep.)OksbCC%uk^fL%sSM&t0V40cEX,6S:;u&2c2Y@JVwhn+WȜd{?hWy(ym,{/e>w4xN`rMGqlW_:]tPgw5Xyxc4B qLe(vX٘87Az@[;{g+ف V:z`_󭛟Ծ=dq}),!εPE?rGA]BsӾ/8״,EYGC勍ΰ~f_v`3D}a3lm_:3lC$cbزWk\0I=[3&@kT I]\5Y@_C|ejlJyp,Y|>oF0dPϷXg}l2%c 7{g/{)/!K KXGqe] mf I>{V; pRnA`a-a*/b_f]kme1_~&܃ W׃zWgF[w[k6m ׋X: J>?%`f%|g |\W;Jr:j~nBLj_ hߊ ZU.Q\JAiF4 8Q,v}olKJW,x۲iA/PܟtQuJGg 1Eqxx{ΌwӉ m)|a,r3L4~ jo.ȵ,6Scqߙ?.Ѯ-(9gnj_" s%Z®\b^mK-;'-`b5~Gs"Y? ue3eu1`xρ\E2k'nw䣥9e\;x~;ze}7dQ/^Ů:Uc{-긮8zX?`X ]@Jּڔ܆,^ers-Q~c~agj]d.;y`nIb/k&;Ɩ~´$L\7sy+yg[g6~?ʑT1o"cҹ<:ܐ3uѼ~C o/XY[ EceO+lcaYMm#.4Y ?&:W 6,LԷkQ۵I3t4ڍy|r> |g_Wl90"(rz+xz}LF$s'骉!@cIE".xB[+Q>6(gOz}x相\:ᤂjt P4bUtѽ>2 eLA{2Z}/syp:(*ѹTl7{l[J9,gz z 0a ܇`6J* ql+'ｭFåk7oK3ˊ3f,EwX'NP݀yH@,7e% ÷(뺌9?jx ĽgfRn3 ♓h{{9XH&ʹ`0ֱQ`ί:iukz,)Z16.ܻuEA+9IYyXk#صUv?xu:ׁ> rCM.k"sLGs@U`fs|\83]` whR y+gFxZS:s3&GNs%AI#RtGxu9=M2>@; >cS_P~1~nFg*y>T?[].2&M uѱ{k^7$锭::3*k cwblmtbb-kQڇtA~̍kk}Eo/ݣuuB_YgjlWHgM龱틭Ak}2s5Fl:<˽}r{nʷ&LçE_@z;.̥Vo/ *ȹ&%^b]w JpRɄp9t~%*`^yPqz:>C%+ZkCF@돇t}}~ խCJCH+ 9$Uwplȱ9eFO}D/76믻׹b 2{e?ۖ sfw\b@. ;E2x W?2Ɖ8aƞm z"` sX|ñX􅭵}@yd 9%=i35#[+{6WYo[`ϊnP#CY7JnĹ󨼨ҐG*GKxs}P:$ԇA_88>Hgwu:ĺB[:H^k dzץpOѷzsұj_c>Sx0zw?WcI/u" Z(W8}Pnqa]+p(9_, Vmyw,Ը=3*gb?(o?Z="}_9+gD󅘍c/P_අ{!<\ɂsppF:?#sMwہ=L/XWOԣ9s7r9(s.UA0Q4DDM|:z/$6d$^*l@P ABzP `V_3XV,[6/jxhL N*زx}I{P±AtsK 6)8c(؇kmA Mڑ8}(/ of`kUmM9kq^ŠZT-_ =}k ѵuLY:ցk!i)px)w僲~+ _пw@X_om=_6p/&g39UzQs˸rvX/aSr5>u$wom޳+c1m},9& r|gDR?PɱGy%s sĹ}^2qK/vW͑L!ڞc`D1ٗ|D:{ {!#NǯغuJll[\`>'\|N9K[C^;v-*6yEv&c*lAM59;瘠ٛA-" Y+uu!oV_S;9cI푱 @/Kmmڇ9őX|1>Bhh_߲ !$9~.9scUy r$1A5Gـ.WYdyLDϯx~Jgr[; A^ھ;;gۇmæ=tO`|93wܺ/$U>S?skc?ff5:Ǵ4FPG: ʅjc~2u/c\uLQiS1u*$M({oӣ%{Ee }Q\=w?}$ r\$IryΚp mpr|nȗכ"Irۮu q r>Ϭ^kt#,o/H|n=n[|N,sM枋o)sPĶ,^ZKҐ Xrڷ!Ѓ&2S1Ir^ʞ_WTmnIʯ_܋o_aY_ ?sʇ}!{c֭~nڒVo)sP.T_dӃ?c=ؚ_%[|1I k!c"V*B\y"?b8[/wK?n_yg琧6[r٭N"g}~$WOIDATDƜ7 $I$I$I$I$I$I$I$9'YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r.PNC'i'W移ׯN/>ڒyͫׯOw_o8Ֆ$Ey5#d>,>^|1ꛧfEx znry k>y]%h,;o_|z琻s֋XxwIJ//u:>pyW^~`?p6óGEu?s-nyP~sr9̹r.p긕sV{$&{=5r{$3畛ޓNFmzīYlH7ͻON?@pJF=t^sV eS*nln|A=ӌƦFgr2lTӠ\j{)g(>;uzO;|zGϧ'Vyע߭ӟ~<=oINO>~ځ2kc>noWV;=b6% }]I;k^@/;es:]:NpǧO*7߽}OH lwNM5u]BlL.ڞ#=[?u_ӝ+?/mӚF1 8Hq}k9Gl}Fvlyȡdn7Zٰd vtV^L++3_(\vpߊ|Ӻv{ygn<.y׳'^;Yl7ʑ.Ņ,{uq}:}'Y/ElM' _>9@^Xq/U?Up@-}>5\WW|^ԹM1Crba(a^|.cyUR~s1':&Y'4_q{Ymfte-c FcK.OIH;y8AhJ kaw-kke)] A}(til(b{Grc~4'\,T_uؾ(ϖY[_X-|.\ei;y0Ŗz=9WX0TWD$&ȩ:;! Wp`e4k݊N |Ol_K|1 l/%cIɽ/kC"-)s=Ӌ.XĎ%;ygUyV{Iѷ}]o oEi]V=q_bqt+_`MsK}؟_;٣ģ{,{y uQLgNlevgOȗ\mo/sxb1FMHw_̃{~uzZ`Dhd-ZWV~Uѿ/%8F\kvKֺo&5je[mI}a|Xa/M1?\7|}X"$eG]$4MEYH_K)$Ke|CfFu5+ދqTnsw _cGil[" A\[ {_mGoOa(E{ZwleL!cgg'^^`ϽE'˞m149ޞ3D7xTd{bd[qx"Vx+cU|!beÕ`e,*eWK=m\@j.r elk[νuoY9Uo{ߊXцP}-mKq &"|E-(,ߣzoߝ^,󕱖˚^9՛jo}W=V?"sDZV Qq]|lAȥ+wyo^B{P~e_7?0(T}⦡~yS"{Yg plR80u`rq;Ω>Ś`^:ζ5[ɿ(N3`O;8IxؕZ,y QD̈]Fq(zڒ؁i]@y%_|h+H{ Au|Eu&|ot_do rP'}z Bn)?k{CG7KE&?}-{ ږ>"Rט=>>6N`~ cp[s(xm1eϢz&o긽_Ǯ/ȹž_b~m S7%,Q.~ 94eBNC܏t|%u'Nzhu/;wK^FNn׭Ac`Et{l1Rʿq&fA zi}Pb /zpР`ZEkyʞ_>C\*~Aq_3tM,k0+kw] dr6Q/"NCnZ/} -a v@}0$f4)*rnvn5R(:Ud?-,L0Yd7s Ftq  Eۛ^LN?9a0ܿ٣/\w|ص|^eAu{s;`\TX]gRK;r9BU?";};CxشCq򬁶cӇEx2Adg||`=UPuD-uqla ];f:, /YW}.gOJ|汒Y9 E`lx˿ց|vTXju2ѱC~@h-3ʲUtesK\rt^ȱMnon>ߍ5ϘV{k_%=XC}}VKqb/Z'm㕁"^QL|3ӯ~G޵+eݤ+h`[ 4N~h{Wr|L1O$ 1Gِ sy!^k!W[wG= ލOه/;F'ؓ_=g/@u{`4ÿW@+m?OmtYí|rUqa `w` {pPCrȪ A*+!mNH+:끳7w=`<&yV5D؄_hh}Cu؇5vb`)L~& }n|fr6!171hl `YZۼbU]1/0]¥}ួ~)A l(J_`#~Y[ĺne.ŊF/_϶-//ql]32gȱ,yHl,Tl/ot0V40cEX,6S:;u&2c2Y@JVwhn+WȜdyCˡ]Q9Jif!Y>M^|L{9%[cOsP~V:2 YaNsݫ0䍥*f./'Z_dnW-c;6n1R\ $7ܻ1XI;z`xg =y.>`,U(,.aucc~^K.+ޅ}#S![ѽXb?]. !焅_c![|V({b[)yC֮ A٫- l~Dd5? #͢uk kMX!/E Q#۪v C?z h!3sstؐ{fl_ѽe>Ož{v˪&:،_h:E¹ l`:Xј'çd.g3bѶ06osXn;F},Cѥ\+]2X~rw&s>~Vq/sg˔tpfj8i6Xpr96ͅ㰍 e]wn(|'6>G:]CTh|~?ϣ~bK;ՅrQ}{Pazp]١5j(o}x6ñ-zK^)c^XI@Ge{%HNI;1[>`Qb&RJ<-BVKuTWERxaB;a2/~cXeCݲ5Zo|soxbvt-sf{L%v~||6cn[󅱲FCO%?XK#췠\˂i35c[2JbMzNp%:<_\-:%F\-E~3 #q;8>&;} E6KzqQbkwe[%8ddO/s=~sU}5Ժ2kt!t:sGrgn<(AQ=<Xc9YƢg&A%653@V ` ֽy t<3 {@׍xKؽbʣLx<"dP2\!Ƴss~-ӵˢ׍m 2j[0l-e@F?;`?,O-X|[ze P¿ {6| /,{ Ly/]cu}ú}=D:#>a'\^YmÚg_mu _/U@F?w>e;{q~65|Sq2=,{FcKe݈d[ |g2{ZLP+*{^G^ŧx'?X\:6p:1><}<|lp= `M\w0*{FJ(](k=md8 ,E{])Od(T1mؗEhfhu7Nйyz{l7{l[J9,gz ЇkJ}N $@?w;HNxmIy1"QxօKr@Na,EwX'Ng~ |,[-[_X}C#rxo#c3f3{9XH&ʹ`CF΂9DG륲hy Fڸp f~1u?ƹ:9A?]1E3ܛA_@:YSQ=ÿ=7>#{bGTG u zI?[ڕ?]zvP❞] `)<3.yPu\Ǭ3>?aؗ){7ı+ލbr#[Jr/RŮ;\$6Xjc)=>(h sC-3hW:#A]ЗA_G:ES}ƙ9^ƶܤԻ1ei= C/`*ׄy8~鬌kܑyvF{ك<܆_Zy^}|{c `퟊;lﯸK:s}ϊ9ZQtʯ/fea\__R;6_bdU2(O?M.ۅ́c/s)c}y؋AN6ڌ%IvƃO29gqQVlwʃE<(;Ex=v!`:ޡt~YCqVΈ|?!{EmΡc@Ĵラ%JrlI20!~:rlNY,ȧ>" X^ݽe|e\=_w Sc2mT9A~.˂28_ _ iD};4/sDོ#cf0>E/lpl:xkQ^rƒǴ[@N=zܫ,շ-}0gER7(ۑϛ'Jn%UyT^{iHRGe|y|ǫoH{^?؞D3܇Rg.XFvFm#{~4F c ]_m=?U}2&[{hdӃ

cyHJٛ<PT,g+yL)m~ 1(ECD65͗ܨEߓs9w[#cW?Ϧ>ׁ 3{Gb.j_}~~10Zn}nkdo9gh + ȷcG2cv;DZirgZϖ,Sk@tG:>GgflXgjl.͇]L]>\dl˶+2Mxn>go܏ʴ-s]׾Q.>f`9ޯ vWero/p`E_,w07-W7nLg!wUq;_(pm :~Ifƅ!UظQkM1~dhͺRJ᠕CʓWe,P_lټzhec1i+8c%f1_-u6<Ct?$؇=.I iGZ0(K!HNL6޶[ƾ6?$x9kq^ŠZX-_ =}ߨ,FB1gXƮlwlmї0/|A ^q\7oc1G2Ǝk4D@x& z7\/jn7j&be4\MUAwl.>f'ˑܽͶz:S*2YsMƥTH)kcJhL1AWdz"f*?E/z~⸹.nYst\ڃt9󍁏gm=Y@7oݯ3Mˍ(boQ#&Ц|kE>nF&s\Nmz+Yf}dE;[tbe'[ĕUoc庨=y@u#Fkl!HO%"C_?i[ƄR6s*DX!\|?^ rI!g~Sg;@A*LƐ=.ˈ\mu.K{B3f{S?i)/ Mz?/AdI$I. <`}Owoܪ#&pgci߾D6~8tn`ɦ oj tyBBx3$IBY,^,Ag&^[B|7*31M{'2S1Ir; 6ؿg/糸os|7_}$ɛ93$I>.T_z'l $yc`*Iʍ|M QNݦS!ؒ-bz}3G Vdlsb9⤮/A$L$ /'ߍI$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r$I$I$I$I$I$I$I$WN$I$I$I$I$I$I$I~$I$I$I$I$I$I$I\9(gՓo{4~+XWYmIr<ק/^^}jKyzzkW=^#^ߝ[m}KQ_>٦@6AH>?+Q׋9y3ɗw˜%쵷wO_W^/ng<:}g~s-' _ӷm׵SQfA]]x?_&u!V˾%>5|OEh﷛Α+doR５lY|移oKtNĸ}9[_"+4˒i~&%'_:U !$,?O?@[_w>qm7wߟ~<=og.<k}n]iSؼϿuFYE+y5?GΞu څ/Lzs*ĭp<~I7<:wXO~| i_m<4dyz_q{&hkgralWo)/SjoO.&N,("U~_{mwqp%x(`^XD/8/,k'GB{WQ,Sv؊]+O7k޿_~ 0gbXy4?,q޶s.9֋)zvP:'Y_~qxPo9GwW3^L6t\.{߼rQ{}Z׮|lA?iÛUbY|uC$0S&"ON?l3ID\&U?Up@-}>5^Q /hWW.ׇ܃ px1:!h)`xhZuˀ;=KFj*ZN)ҼKQ{b|yl\2cЍXWdEcQGץ9ҙ`?M>J@[̞ŤVZX-䵠_ ka_ X#p.K)ǧ<8ɍE_zhxrq͢M:U7x}a6}q,V*sr?J^nb!_p.\^0ʉ^}ܐN`W35ۃM*?e~!2Cn-7ۦQm71Yb_w3^ܿOgX ?+{X9aYsӫb 'x\7Ϝ;%ث/PǻM˵?k׊{s}W8\_k>/˞}]/p&2a/߷힭ϖȅ/p//F|8wZhn]vP ݢjPt١/?}ywPY嗅 \?{}͋6|F1v$FI| 9^\> g`6pC4Lx_:_N\>t_$0̌mVN1̽,-yl-ccqil*dsPg(U򞦧e K_Kj;ێ ¾bT[gN/e1A9J=rx$sc!ڟ/S9ho*=}>,BY%ZM2g{P|l{ J"_(2Zՙ3źI_yڋ47ْMy_}tW}ԋ/]A׎g#SF_0wYSGa P]*Ƚ ۶ i۴t2􃾎v#k(R6yq6;d|~!gx@km sv-avBE:~޶,016!CrLg\'E\l%17\ ײ~{?xA3w?\*zxEp鋹xv⳱=3f=GEC>!>UO+u\F~ |&!ޚZt#: }_T'^-[,=FN;[)ֵϸͶ9(,ïںBglZe*L>#5][z;rI>ܮ>^0} -; ֶ$f!kS2Ux/tK~K/"# /sE??(2daUennB5ckc}1gئW ,:+|3x%xPN&h׸㕊92Ƅ1G-b_2Y0i]#^c1kB\Ǘqba r.J;.+יCZڧ#)c( 5 {&ھ!O ɃN#|^50/%ܿefw)Q@ؑ}%}txv)~/ֽ2ݪ=lHuC>|_=k[x5<pA⽟ӗZlV4{(7V8V:ctVqas^ZB_c܉!{΄N:ڣe?O}I691Y)Rųcrw7\0-' _gz7k8GRs|lt d1˜l0r_u򱥎lrBXD\n9^gTRSKq?鬭ߣHGͶ1uW׹ .|9\܇Έ=e{Ό=|[k?41uƖ>,ttKZXgu0ùuZ|ɱ3ii~g"~uU=Wl萯c8B|!V(1&1Q_1h;{qFH I^Ky·ޏm-kHLhҌڱO%UvaM j|pw;4( P}txcH;HŃYU<ɜ )cd{B7nP0[ Cc}yBQ:^93@z7N ^]c؁ꣃ w~y K3 Y)u,Z<_6؍u1ǨٰR.\ ={.$/%>_|i/0U gm.*/%8DZXɞ%x{0Hj2i;c CbHs];ֱ0ɤ`lkm kXoq./>vNq@6/v['ctT$%){OJ~0D\lFiƶȟY9ٵbga(w[ ac Fs[O\|<}7l>~޶gƔ9<;aP,?z@[xu/-J ~Y,$?i3k+6G:teЅygQt LlDl7ldm?;ώM!9I>+1}>A"r YM"c|A`r ;PбJP5c66#Drzrier:+cr>5c{rOl t69F_ sYeRX7xk`-yayާ|Ao/h6kdWF1e uVǶird kg X_h 8׭:4,=TB_؂,ӿ|_r%? !$-1:{qҾ˿L >2&dž|8T.@:ЍO n t~H?Z>?VcO!qcZ}YpYtc}]!5ޯ3!/gZ~؇ `-ĴA;{j~ncDy~-BRpUѸi[ tLЎcw}olKأ=mп `%H>6G_FsvǗq::#}l>b̖v?~Dʵ{Wj8[h_Wg n4ZqWs=AKk];U 3_wUdׅyFc2=6=Rg[עC:3A/:پ7ϫgg@gd)?#ώ-)s3H.`be, {E3Lyj#Qؽ!D.2&gPY@ 9ZKR[uaml 1uY[e }Yc8EȽ6@gcb]LZu@ÿWF?zhjc/V׼u uAIBYiz._ Ï@u>rp}XFR\(=woJg0vRtnz;>&zKŧ%?OWM 1aIBID'FC[+14M>6$?<{ғO#Q6u0v(jxuYA)kv;zkWm_ʘ6wpÊp,^CgV8P9=/E:sv CUv <58v ޱ*ZO$ʼx`QF(<㽬^I> n,u.V?26Z@`[}o-.[~̻wKqP4ۓ$$͊s[@W7s].N@dtO0¾_߲gkr/1# }ѽ"z>_ޤ_|QrK'I]1_l]>[m?/ܫ}ooǗ}+䲉k5Ku6 mH?օrqt\ E$-Ѱ}{7,{y*ůk)6KjQ͓u"ز:s!3;~-o=t;%{뵇`$%1}/uge}/' |ʩghy3-kj0&r1l<n9Kkv1Q۹vסο}\V3:Pg@iy:ue]rRVD=6lg33 ~Qc[mM">8}9}(\3?ƺZ2V ޻/Aw_|} }a ]ꓵ~X??Ix{{#|{K%pJNK rgig/bp~ऀrLVʂU ^~^ŸQyjQZ˙A}l#}]/0ֆcYDrhCtk9r?[_E-c %}0[ bYcK N M-?{qAW).~p/6=X_, s_4pYWc{Ej 05F(ضL֯5ߟҏ2ucp(V d粕88:c 㛾27y,`<<{ ஛%?~qq>3XB¯767s$5^d{#TR Fٞtg1"h-A)oVܑ]pmG/y:eq<>y]m?_$Ց]SwgbܢcaXw~vPyU 3_;ϊe^I]m[ľLϳj"?csLgJ'̇Xg#M(W 7r{lc^%1;r% @<,u`L#kFfi󳚳;y GieL*T[GDξ{]\Y=rcXcsw}K\n3nv..SߣDNG3i.9 2|=flMf#Йg‡- LO5Y' ݏ#_+Nt@g G}ݮ=c807/7qLdqN.S_ҵ&῍DxMjuWj(J:zoI;m>=vyI/,J9DN[79kѣ(\b v!,bJ[Yal_F m2:sY}^;H&=K>\Y%p}@ڑÃL h"[ƾv:wr8/a]~-Ȗ/~>[oA~k!y똈uccYKfkZp~nQa /5fcnĸ9t\*D#}Nx|>6@e[@_6=5oGP庲2s[ :?fdi;EgEj|gFGۓߑܤM{g!̭MOev5&^ sOĚ `z]@]C?_$b+7}P6UaHY"mӅ}@-sh|c] Yynץ_9JDGhNt&$L΅&}aTr}pF}?>tuu3ƴƌ&U< sRp>{u܈L,a<;&wJ$9ȹsY|_4S(?[y@.>vsaJ"y-c%u@.~Ϝ Zg{A6/G˚X=15_#i?sc~pֵ_g@akˑMQƷYdG: ~1#+Bvѧ+g2ޟbwBgK}ljeLKja'Gl"d1@Eu֞UuSmh98|}0.v$*I,'̄/`%Ir^@`!O 0`cn0z,=Ƴ$Ϩ 7~rd-vr'}?e`k$ycԫ:MƑx iPy{E}$: p5͠/?c/?)xSLN{x}iu|V[$Ӌ_cB^>7gjf@<_/z7O6 hvF9AEȹp07Kekܞ!N3^?<}!-|ӷkg秗0|@>vߘt>-V9_2\6GMvO0z'Gbp֘~ sߺG峲=/w/}^_^yZv{̣ӷ_}v5-ON?@'y兊zӕjުUBi<|^W9(mlj$\.VRܯÉ8"BEg uzO;|zGϧ'Vyע}ᝏ\1<~]Oٸ O>~ځG}WT.6/vQuVQziE^ ?>WGߐ~k׏O?!m_6್ ?mmk3A[; cr/}+H|]=GT{{r1udA߳Ao:zl 9qeL kZXZ8,/\XFS/# 9ro;;(W>qC#xEdLޘ)Q}(^< #S9"7\o\8qq=-j}[\6gzs؝?;q.k}s2ڵĝ[o\y{e.`1<WbQr_1{{ujtشR˵XT)/qdU?Up@-}>5(dž>>^^n y sA x19!h)`xhZuˀ;=KFj*ZN)VKQ{!\2cͨ $6YQtuuh;|zt&$,6$eP,&T,j)&Z] pZg/suYJaw} =hx_?>Eї6 ~Hn,zoCÓ o}o*ޯQsl/ g^9+ +yɃц7p KM)9ɼ9liߐ-XUn\8Y lH:0 =2rH }ϯH_M1?\HI/˭`}kHY|dp2!3j"Sns }gz!`X\cGil[ ԙ1?e)8|җ=үζ"!@`,3Dqb[i3b.όAgN/e1A1n{H ?UC?_3mtҾqTF~)M#:3ֶ=۹n<}Ž[˺gYL1vp ; }\G![o9noC P}n?&/=_27Zd\n>WlB>BYokѺbE+|tW _T}[y{Ŕ?*(q}M\ڤ,:󨶒YC>%$<_ߘW>2~ܔb 8<߿r^s~dpм:\ ~ĕ>e>RʿqYjrXm$Q=$ͨ% Vȥ+V ?CT< ,.>ae?"@^? k׆2 qZk:0ptmlw }]k}J>ZÚ`C5Ioٽ3M012UxL98K<,|+. ːW"YcX?06" Ռxŀcvz_~0@W謬8@|L(oq~<{.A%^{kJc˜#T/z{,lc w5˸} A۰9PUYuL!m-m[ɂ1wtTjޱ5GIG yJOvWj-mB>kL/,]X26cR|VhI6\ZO #{=|^׽1t<[:r?掬Aۗ9#{ֶa0y"ȡ;c(fs±U;ogr.l9rmQ>uG: {@۶w~}\c˚8IƷv|/ ?syU.sh 㙊:T=e_M 97fO: LNغ+rg./Z c_WaWc4}"{~&}xk4 , x~YM u&*< ;9Թ7c_sA`dic^BM/M63yVdU1/,7_| _Eŧau_VFy}ψޟg׹Xr {Ep;O%Uyv(;6Wvۘ h8Ic)cpЎ},)(V9u4 lKv`@1ȸA*ۊ <ɜ )cgaW8yB_\s/O>J+tC念>ԉbAkd꣓D>/?/Ed9x՘p Kk_ۼb7^dd":K!sb6e}/߾ o60Jf?,W.\z;;a[ %1.*x{0xI綎c !acwXYڟ?6r6nokgyz#maiTnAˊ tanu2FHe~lΕʾTs"c2N7M@&kV}=F^9e?8 w~v`N^k XU~5^a|ihkpFmq&ZGtk: u6:!T>%}I܀vo9Tlh?d{4ʾ+^ȘQ1>',oKMQnu@lʇܱ/ u [ܫi&lY Ztk ~c8 Ά~֨&nwq-\Hgd Oۈtmrly``L3MO?*jO*,MƱdtc'wV6+U6"1uI#%ع61epvgtoҺؒ/ý|l>)̫N1>,X0?^>Hq/?X /cRF_ژ&es;h37.n@i 96wGq\utݡn(w38(1KP@IG'0_ۿֵx+f_0->Qt5L7֗k /F5ޯ3!x%g4k> XL? n5㲽>FY;1%[:F}QG)Z^Wץ<*q-Ҷof0~ ʗlG{,۲iAy̵)lX:n%g{1=ILbcƱs(cGCA8tV ?抾 ]l[|?DP>wY@f9tW3M͗\hCm砍αc>?o8r#^cJ=Q>6]p[3sσ -\a>ns`[?΁\{.j[ph? =`3xq7GtslأŀKIGۛܘjXi;1ɤb 9t8?^flvK# n/,m*Pg3RW,?l mcaIP|]sk)} {m 157]eKgC?S>䕛)kv7y,b+lL-PT-csLvM?q}# J}06ʩ913>lNF gì+&#:5 k9({ Aߧd?1oܸb%/0mD.2fMcfܢs.:6[Q]sK#b'F6=l>e[*k,Pg3|kWlmt::_aMcVAj~ knuM^uuY+dtmrlzMuփ1/\ZhWr=6L}XƺkȧL %o9:Jr(~|n;mHPaFҺϗNƝ`Ab$>_roբDç϶1`Բ|^ ^RSƐc/NrU6r Pz@<U jա,[kUqřA}l#}] J cm8^OT (潿8Dg#_"0a3,ɱ%ā% 8us ؜N!jyȧ>" V{̦b9~e|e\-B/Gc2_[/|jH?$iێ9 N,EBhRU\dV^}y {A7~|165XB&>W$<{ ஛%?~qq>3XBRϯ767s$kdOgto{7BQ(q홏]|#؛hPǶ_Ao>~c 9bh}kdhC u2}+?fLru V9dcбm=&yqPgNlss?>lo>Sr|?tA@pnl)Ί~Oz9E:H?暐\\{lՙ277`њOx :`,J#_"sMBT<䍘GZC9 7e }<Һpk%u_fuF;hhOzo} TL}a}ޘM7{wh b,`oA| 4@Z>$X Ѯ? `Ot-4VF 0eVGeq_w`Xڹά{T>Ί$Axm%3>\  ҎXpIb"q`y U~|lc_{J;eZ.dH?qj-7j uLY:ց,ƥc5-8|R^0_oyA ^q_ `zqE,eϭRt\*#}Nx|>6|.V2M&V0{l,b+Ɔqx?SaLkի)>5MTP>k JEj>&=ـҷQʢE͸]Mplboh^|*_\}:)>u9<^NJT٧'>K*ǔgO 6@ˇJ <[%:$9})_ g~|f@9q1/^ś]xv9ɕ@YGg7@Ɯ$ID,*%aʹoez7n$o,YOn _C,' qw\~w,ss'Jo9I\)t?4<ؚ>I8Ir&p2?H/`Q-zw۾gUb*AZiy[*_=n|[>\>$Ր1'Y\,~) $I$I$I$I$I$I$I$9'YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕ$I$I$I$I$I$I$I$r.PNC'i'W移ׯN/>ڒyͫׯOw_o8Ֆ$ύ{"Y;!F;=ڮ<ϗbK=|}zSM:l>x>p!|~)V^Gճswf/9/K ko>\twspɱw`Xυ`y>ύ\rˀLrrmG{LceOgy]gHg~gwp O%//KRk; $\BWp^XVǥNUj{"BE?4}9ӓ?O~M{~<=u}>#Fޗ[yINO>~ځG}WT66/jgǿ7:G#^dl_7ΤldeϱyÂzPG'Ja~>'~|`wNM5u]BlL.ڞ#-Ѹk]g|]PπGTw{/f"}Yq Ƶ@۟9a3+V%Yoּ͡aΔDh~XBmӛ/(;>9p_ pIBK hg0;hgAoTq}?}cs/tPo9GwW3^L6t\.{߼rQ{}Z׮|lA?iÛUbY|utCⶼ`@RL',v"698Uqj]s"{AZH楇x`K(>)Hb,EI-V,mI% 3w5ZQ;@ZЇ/~|zK=%IҴ`\kw*ܿk-F\]l8+kA}(til(R{Grcz\p_~{S~ /:Okg]χCY3Bľ)&z F2 %m }2EU7}O`N`W35ۃM*?e~!2Cn-7ۦQm7gvɱw`g0N=o򊓇80\?F9k!{Ru9CBVy&.^vxQ~:޽o_|wi]V ՗:,XY|>^.뼈ޓ9.O~.s{_>[^nDȅ/=b |b1t},ﾤ:3?{' "x05еEbZšeцߑEo NjX y_ :XvYc{b g`6pC4Lx_:_N\>t_$0 Qt5Q1̽_C⾞G9[q{ilɠ:3G/ 8-}c/,;Zצb[e~r?E!sEy^xߣ,&"GVG}dcx0^8D3E33oVsn*Ї{Mpb.[߲rcCb]ۚlKIΣE(m="B{ |8+Y+uBd^<EaK.? uV?relk_5 Ty(tPm%sU{yq;vֹ'|a 2a[R 닁yM*皴[Q%?LX_|J*qf&f\P68 8{}4>Br=`bV*p.R\Pǰ{[a3[ORgTlcΰ@*sϮK~q?l^W>ܮ>^0} -; ֶ$f!kS2Ux/tK~K/"# /3~^ڽU~ ",8069>@_-;ZsYL/59~~6. E'zzVϯQ{.A:_W*92~W׳#du]{ŏe8Ǭ q_=_چ/ȁ(ԮR\gbikiFhsJ|#s+kB8[^:{7'j;y`D0pbͧ},ߥDGcGk^8~ۥ rgZ+Xw~ "u&׉_ Y}m}cSL)Ή~N_i [\\r򹟏؟.!__G,iX-u4&c׵΅+|^DZ:w.{Jԙ{^J~vIf|ՙ#Js[s8rYĺϛQN>Q]_H- ;_Cj{3#u{\6:7ڣe/kqplrߙok&Μ҇`_,A~8NKVߜo39~&|]zw!͏bsL/G\7C-?;ÖulБG/ E^?:1K>mg/cI&HZ2_% I WxrJjؘB;鲰ʎ2XIdt5>8˻y::2X0 t>A9x՘ Z[*)EWeumx]]B-0SFzKQ+&={Yߋ~_/%>YRQ/ 0l+SPR.%j9/ks Ⱥs~YDg$/+~xoOIYM&xg?u }8cH c`:6&?-bYxm `mrK-;څ%Gک0_&ndLToy)98{7O"_su.64}c[owĬv_ Z|d0Kױc~euߊr]i> uMub?On~|o[W3[cvlNA p& ?ٹǐ83vfb][t6iQ.YT6T%g ɥM({{B&jY{k;gcko;:0+|&{Skl~Ɛy >@K GRk&r1y fZyV~r ñ]u"A9=TNj9xCX9Q6:[u׹2)CAt5B0FS [^5YV+#_h:+kcv9i g~53qQA/DŽ|`VK*/lA_/v\&2cI%MB.6$ =qd)e]wƧƆ7cNPk:PP(ʢuM{%x+Ǭ,klYpd0yqN)n;Y?snop:#ZzXN J%St 01Hq^1xg4{[O->r2"8LqxpX$JtL}8V|sce{TI0ⰞY%+Q/XvmE;{k,Be:Wߗ4gaS1[ m砍αcF5 %>`%H>6G_FsvǗq::#}l>b̖v?~Dʵ{Wj,P[*AQiEeг :v\2dy={k5TV:a3ѺJs-hQR8|vؾm[bɘ{{^c%g֏־8èC6IJL XXKÞen>s 7sSfHT>vo1TyCΠ-ǒTz'flXCt̾jE֮}ƫklB_Cth:ro10,XӱrC$U=f0#F~=5o]CE}8PglyZ#uh'\$WocG]ҙ]ƁTx@ζڒ}iIO$5yI$I8Y ㇧oD]CؐIO:H|ڡT(Fa8h $xkC'Hn˘6bk6_#w A2=8XQ6 x/+_a;FnX dl;VПI`,/3F_0x} ̱̽~1 3,! ^cЮ$a$Ib1UM=v(0oo嶰ٚ\'kKL5C=ft|^k=9/-yX9k$mbAv-9^&? qL _]}qȇ}ک,3r<ϧzf9>ߢ-m"6cpd!~}mk}pWeU: u6/o- ƜgYܱXZZk%'e+Idcqx<#?S5e/,蓌ӷΊ5c]a{qqک!{lՙr}%̧}@p>YDL4e{#@{K%p0_NS$,΍(_;5I ȃT`h`mtZlW(-q`,#u.sJV 6II{:+c륲 U{r^o(%5a b+ZAL? rl)g?c>ʱ9eE>Yc6/׹ 2{ҧe>{oSA?3.;HXXq@*y.[#c+zӾ<?J,!ca|{sþ|I_K?a=㪍/^-/-z_`ss_a^B#+o7BQ(q홯~uLm$GrKb [3/~=wdcFK~i/&qjlܸ.Ϧ~^!l[O1AulWic]Y?"}`-P8Qqg ^L-gX8}54v ץt<y#|uM߀J-AgϏwЅ}~^mP|L; S+N!)ZN*{Q:rn~޻|~4e_Í`\'_ܓ1hCz=sȑ\^^!r eL*T[ m}|H{ȱǰlK"~ۻx.7ҙ\]]~c"(o;6LGKNȷg`[S0ta~?SutAItBNJ}.:(rD~7CQ+md<>*srت3u͋'/p\ߤ+SgYӇƶiג7HuX&t3ͧ./<.V)9mqop< [ +T{Ђw=uWk1Cz(Khy[D9lS -u*=36~PgZ6lU kA؇k k>H;҂~x390ټMdckUC@NCU%VBG_ZH:&{/XlM N.K"~+ VocHئ^}?xjDxX|!:.H"Fmƾg{_'r~t.|cs dcX-}/Kٚ#(r]m @CRP-{Oe8޻CH)>+V+h,=3?ڞl&mrk?` `nox(6jH}" Rb Bg<&[dMֿOۀG߿i;h=]< gT.f fs+^ }ylYb#u?SrzZ!5Mkw=mDK5FA?LX9=V<13>HgX9f}dE;X޷ beLSN |Mm@׽ ru [-H_?MU>=(Wڳjbʸ>R6o{_P"I.=&3I{$8PNo spvKR$v)ws{3 [ G7ܕ{U]`ח`pow~KJn?.Y?/Top9I-|'x|\YOn _C,'y}-,cT/L7`ϛz{ګ~ @/^'yFuFd-vr'}`k$yc,N$n06B%CZ@3IOL^s3(T?-/ykPۣa/s19vaO< Z(ṯG$\$I$I$I$I$I$I$I$9~$I$I$I$I$I$I$I\9YO$I$I$I$I$I$I$I+'I$I$I$I$I$I$I$Irdq?I$I$I$I$I$I$I$I,'I$I$I$I$I$I$I$ɕs~v*_=9ָM>5??ݽ~uzՖ$o^^~}n/NO$I߿x|n 1v4ll^_~No]o|uzwGD٦3n>QIٸuv&CcW0[<:ő'G'Ja~>'~|1byz_q{&hkgralWo)/SjoKD~6dX)c݂,Vn}~Y:®Cڽ' ˈk jayl^V젴r_y=i;Q3{cNprD_xc0p^8N _/n,pqqŝ@x]~n]s.ۜawxǹ=S-ɬkJwo-?s9}΂Ĕ>P^E}987F΢{.שa.g`J/bscQnġIcTٖ|.W({uz%`50etdMN@UZ:5J/4)&üPl8:\(b50Y>_0P:C}w=`*1\uqu}`AJq {2(1iBk-E]-³[c,?/~qzW@كvP}`P"/=4<׿8f|^){)2D$dOobqf<m/u`YiIoRuOƍ b%c-;E9o]aν;w7K{׮ 5X|u: |+/\?Fc| ;2Q8{[q?z!8E1\ ~.s{_>[ Ѐ:Y7,xk \>}I_fU)d-zs:t`a$( B8}y*w;-1|xaGkY>kDۮ3klP\'tL> M$/&{HҀICK$$IfVG]$ڬMEYH_v2s8qߐ!F8*Fj;[Ȃ0gX%Fk)g/kY+6zVrY7rc3<IpDgx&T޷g;u61ǃ/^}>xY,)2Ʈ4xcOSG ea< m|!}(vה5ge+l=Ed d u d `-ǭ~MsO:d7{KTq@^1YG(k-z-ZW⃛b%{/Ju9oXSsRGe3/PIK7@gV2:sqڧ䚄JڇBo?TQh6{NP}n.`Ml9٫Bv{??ӇX_N۫{.Xo WPtcU#ҟܧS*W73TM͂$ `/ۃp39X#1y85~Qa"eq knϺ>(6T9麡҃FL؋=ԟmlXeu/g-DcZSiާBm?IzmiB"{e2FXbaPu[tYxKW {/{a|bEdc#aqk#۲5u rM=dq_>Sks=-A_W*92~W׳#dds]{ŏe8Ǭl_=_چ/ȁ(ԮR\gbikiFhsJ|#s+<5qS?umOvWj-mB>kL/,]X26cR|VhI6\ZO #{=|^׽1t<[:r?掬Aۗ9#{ֶa0y"ȡ;c(fs±U;ogr.l9rmQ>uG: {@۶w~}\c˚8IƷv|/ ?syU.sh 㙊:T=e_M 97fO: LNغ+rg./Z c_WaWc4}"{~&}xk4 , x~YM u&*< ;9Թ7c_sA`dic^BM/M63yVdU1/,7_| _Eŧau_VFy}ψޟg׹Xr {Ep;O%Uyv(;6Wvۘ h8Ic)cpЎ},)(V9u4 lKv`@1ȸ^1Zvt^U[Em Unѓ _<us:Y:kn8fkY٫=悳A*cɌG'%1 }3_~^sD r6!17$@-`їaע+ynk)#0] ѥួp}ۗM,z)_(ŗmԳK~I{ΦK#c?kf$"beVEӍ瀋'I tY~nHq0Ɛ=vul>|`lkm kXoq./>vNq@6/v['c>k{=-O*Qo $5\b3J7E v_ehoX=]gM ^{rY| ^~(^@91M/wfn\܀rlhP)C7PfqPFcZPCA(reQܺF&<ĕc_1Uܗ22e5;rBLgi]:ù\dh]/R> XL? n5㲽>FY;1%[:F}QG)Z^WK,ƭ@ pieљ]}olKJW,h۲EsY?sq]o\]>O~ (Rb#>oyyeSvDq=6ʀa{#=U񏮹eV/Cw0a;ik&@!/]YcLp9cj:f[dD`9hs옏m+A{|W8ƘROT6rM}q==¾e~jK+Wmrs p˸VE>gڏB*X? &{pg6l=鎅}&;ơk[)Ư_f]8h5 ]h>Er_ƴaX|/iՙF}:Ʈ:X#Їk"B}~`4w[0qޢ⽷$vk7jK3ˊW3fƎEwX'Ne,/3F_0x} ̱̽~1`(uJqLoOF^Xa?u }8#s].^ 9=Vp\n ~~ˎ.u޿tޏ\3cFf߳y Ңxyw#u$ik;sf$Qc>֚?Nj}Zƾ+욒Y}/ R5G'LPsSE9`lSs.cf3 }0؈~FYWL>/Gtkc=rPz3O~&qcj߸qIJ_*a;\d.K6ȹEz]tlX>F:=OlH{|ʶTXf֠NP t696u:Ǭ2 .Cnl)!mhd+VL m5c,:_I'v#{lԙru%>n$OJstzM*_Qމv 5 ~Ìu1/k~藍;kHr}~?ߞGqE%O?mcXũe{#6̥6.ܧ!^ \mhơx"(@q &* fYpcKKplȱ9eC@O}D/ŭMr, 2{ҧe?{oŅXozqpn%s(VJ>QT {<űi_C^Ѝdl} %0σeU/ A,^&}s/~\Dn\F6ǾP({q [ Qq#We{ߞ(7iڏ ul{3g9ЙS-gܿM8 R.S`T)7[o?C;Kٮciњg ~A]3} W00{@=:gxcp93u9qn}_Y91oݣ*-XY)n,{C"q6gʿ@6z&,Bg3Fڃ\l/&!.c}G:{gG&w$V`<b-׋c =}$^;>_:7n,3ÖV73%'3~Hgd d}fdGWd3#! =h yu;V o^,,sX ~c AT924!2$D?Iވy:<@ {S 7Cl/ lPXGp~o^g?Vܧ BfOIYڪtcѱ{ό6p,Rơė@IA `+aE~Z𮇰vYddXa,iRn䔲CN7;\ԩXhAi|BIf}>@.&?iA&Dd"[ƾ5wr* Zp-_ =}ߨ,FB1gX~AbkZpv^ѷ[a^WczLQNe  $!#Hc߳:_i9%OR~_+V磘$vc/ ٨,_.}NM1Էum=Kzc0_s~ &㵹D6F`hnov;P lNcOѾ}0kGrDg|kڤleY_Q<2v~"V rӈgE1MNJg-nZ6עgWDCL=ak {б6 rG:`0O[3`xc5?*cнyPt97y`gdvoXn4kd HgX976cuɘB c\^ULѶi"Yc5WU.:VU7MR=/"lmcx NG:]S ,ZIUN x1:VPoT>=^j\LP ?=?}zm`/\>T`ْ<%($I-KB}V80 7k7r̉8q}J(MI76|œ,:7<+2$I&/dq?bWL OKH5??ݽ~uzՖ$o^^~}n/NO$I߿x|n 1vu~þ7|yyXq^{{p}jW^~`rS>wmo[uT|⯟\;{'{5ƳKE/u!V˾%>5|OEh﷛Α+doR５lY|移oKtN\_,ۗ5y&eIjm約MpB҂2E:.uPCH*YO~M,;x;Hvz3r|z;Wv{.ƖmoM&5swzڟWw><Ǯ K  qCOOT~+yϣoHG{Yoǧ/zƃ;95׼gv&v^V: 25h{J?BZ V^h1ӌkL 1&GTw{/f"}Yq Ƶ@۟V~5 ir+V%Yoּ͡aΔDh~XBmӛ/渾bXo#^L;z&vIٻON?<@d_y|7Ϝ;٫/ʎw&}:.zٽo^=v>?k׊{s}Wu􀠟y*1,}:}mIh! g fߓAaI*ܧ*ҟ*/ϴSe 긋C!4` px(JR 0L pKNUcB{Gs{'fk ƃiGPǒExXƽZ r" ~^/zvAL0 &%҃-fOb bRDŢ4-bZݵ Zxv b8W'"/N({.~|zţ/lޑX'\,T_uË8'#m/Z= eݵ#{ v8NA,7bYK<m|K/+WK+'L{`?]p󾂜l WaD, ro6=/l~-@lC[\!ȹ(4{/Kgbq cb  Ys|{z9E9;xIAuZqwζ /T__`gRܿJyM(d~.s{_>[5 'ra=rE bY>}Iuf`O Fh,P 6oяNP1|@~y8 ɶ g`6pC4Lx_:_N\>t_$0 Qt5Q1̽_C⾞G)[t;ZOcؒAuf̏r_ @plyOS)[^UYvM#n+.؏eM!BQFU_ch=!ܝ^}u=b*r1n{H ?UC?_3s6:%T>{|X8>Kڵje, |A\3DPd/:ֵ3igW5or}HS<7=oJ{}͋]S/v];iDLv~eO{+dCewث .߃|n6x;mә^:}9={k?C|/gGtMWep￶N̽e `[X{v|Skq[m=s:)2yʹuG6Wh Wעu:~[ceNh:(l)ץGq!=gRnm f[D>pjO61n:/qsJ|N| |-\fKRKrJt_|J*qf_Vr1>8C mcgSLI?S<5uq9g]=EN46}wz E~gaG"z;.E'vMۊr|Kͅpx)0>X~Xv>ӇMTs㽤/usp.m,|+. yi Vcow/0YL6s`;* [ igؖk&{ڨ?.;[|x8^iVq~%xPN&h׸㕊92Ƅ1G-b_2Y0i]#^c1kB\Ǘqba r.J;.+יCZڧ#)c(=خɽXk9Z#_ ɃN#|^50/%ܿefw)Q@ؑ}%}txv)~/ֽ2ݪ=lHuC>|_=k[x5`g}/9דv<#9ŦB+H\ѩI!'5}i@O߅~$:BƱ#yy8Q2ـdzubXOёsb-lb3I_suϼx%i9y@\uF)#wfZ3g6a#Xu:kyP'Ӓ7LI;_ǝwH\?񋭫оbOΰ@|t B׏)D6yAًARIk< DK"5iJOS s_tYXYeGsC\wXto^ZCTqYY^)/xñnm^x_QXܯ_"l>7kiJQֹ/m,}XcRã>0>7',x`@&d5:J ky]ۼb7^EW3`pK!b2ܳh~RE! "x,_R)ȺQ_ut~u8.]<",_$e51U`!1$9z.|dT0edᵉ6u ,}[8k֗Ci Okr8| ZVd t1Qqۖ u{JkfolC]/1٘,rױ~:V{o4ڿ9[t\5dzwÞۼ.Sm*yfkLÎ#9AknN}IO[g~. s+t =q^@-@RB/MI$+@g/ <6ձck1ރpHgcB{}or{}B;LAR5w<{3Hj-"D.2&,sP 5ʯT!_c8kc<N$('֑I\V-o2/S7`16w 4@gck:ڙ^&e(uWOВƑ}ackѺF: >@J}e SPgeM|l[.Q~8L@?|x8ޚ/*Й}cIC%-:.W_A3uҗĎ&lw7$ާE96ǡrRun|jlw3fܸIZe[x\.ǯe Ѻu=Qq;^e^! )壵}x):ZAL? lA3Ṋ9F`pJ<-Z^b!kGEpU4n^E^.56',{c+_V`ߖm -ˢzy ^~1uZc_4q?b<\&1eWcƱs(cͧN*G\}m+җء;ϰ\g. ,1W܇JoHD`9hs옯uMnv|5|/үMQܪes;{~@|mM*{:?4sSikcnfym8Wq(s3H.`be, {E3Lyj#Qؽ!D.2&gPY@ 9ZKR[uaml 1uY[e }Yc8EȽ6@gcb]LZu@ÿWF?zhjc/V׼u uAIBYiz._ Ï@u>rp}XFR\(=woJg0vRtnz;>&zKŧ%Y?]M U& %I LY㇧owKó'=>"eM\wk7;*QXfc2 ;䵾`AñɽkRe^ښz-l]Z{c}=Vby^N@ԟS,7'g2[aM"cRyLݢs,б<߯3bsC1Jgt&=Ә t6;6K[x䤬z9#{l?Ϛggvƶ%ښ\CE}qsv!YQ>6f+;7u1Yu>;N[;?d`:w_.rﾄ'k1~X? +U=^>^S\ϚK7$_;5I 䐴^+lQmдMXgq͢ 8D&O-x]^6Vu_Xл->uȱ4_ߒ0RH L9t1t؜N"ns ,_[1\/ܽ_o8B/Gcm9}* G[o]st4bJiSϖugBǥ٫l%,Nn'cclk(M|/% &A,^&}s/~\Dn\F6ǾP({ .ُ剃i˾Fٞtg1"h-A)oVܑ]pmG/y:eq<>y]m?_$Ց]Sw=;|oygiS7BF,kA`D-Ha؀QKfD_@V  m< - 4UsaZ5fwys""'~ͼuĉf>%7kZ9u `[Nm=;1ʴ}֋6>U(}hsPlYotEgm:ީK%>1mלno=:~ ضWWb/@p__N|_P}#{}ڪK4.!)mǰ}@6~~u۶፟[ٖÜm}r=~;tbkSF@msTP[y~̎ ;d\x7 Uİ7~n50qVEŸ'r>b<:`G+:>S8 eط=< mvc4'O/JY{V{98d6mj8.C5 TʺoEj UZ;:6^7?\DMxImD /NZcA5|_gWw#Wf=>XqIo6m7m:-21DHgbV<ԂY Dl-6\ޖu;r(Lh?%*'btTԵڗ.է VO¼aOuv ~^j+͓raֳ)}gޫ_n*S>?}Lu(P-]Cm1_b~ۢl偏P񘑶ej5:aŸcm:b}qf_m}7^K$FZ^3S=EexOby S{Qfwx%<1}~Adۋ0}7>k+n[#٢'Ec>L3rP<ߗg LYΓ0b A|ytȶ[u3)U }@,ԭweٹ]AlZ<7ظzPYsr,yEKF֜:z4gDي)By^ygql~!{{K컔m}L"̏r;λ/ğge9Oa%hCwbXZ>+C!oMԯ}FOz>>zV1YcgPƣ3E{>kƙXKsh,ە;Ÿ2uӴb)-bF{2c53=u(+(&B.tڄzX[J6t=K0F)A4pxBc 818/$n'(n cpFI=Q֔z>6Y J>2wOy߽gn`<6EP?%^ѵhpFu:nyf1$. }0 ` -#܌O b3q^€@, />;\g1/o8D:}=ʅ>]p,xC>?3߭?۹͉彉#~锱uk_=OpF#Zqr9ɥA)35C* ΐMR2?F,?o6suˁ<'g'&oސ8Y kŚ?=^/4bAM34}l_[?J>r:={8 _.G'3>p@q8s |vo?:>_M/OOxi7?^x1]?_:<dnӓo^z0ǺG]={1=|M+`8i?k?Мۿ7(֚GǛ3T ֩Ns{ӧ?|4=}K;1ݛu|sT콹3 @ge߾\p/s[v^㏹w fz~л߃~}t_?=?^mket$6w=N sѶ~^y$qteu=7:jqqn'J5iPRrʇn%Y#Dž]Éq4"Bo/yϿMw_;އ&;/.×ӫjWg~zniׯFc*إzߟA/RvXwcx#3,\C6Bߝ|Ȼ[4Ѥ+`}ePgߛ֋O'_T .Jܯ']đ UnX=n˾ob렱陯 }qc=a=w\i;闶8sw 5OcM9y/ _x6,9{bq߿.&, 0g&7M a] 2;Ѷ?^82;">۱B &^ƱEwCQyJbn="D:E[B b B>_5M~a7e'wsNwu/s⁥G7`!l8.c3V8gƧ{SmW[ xH듽];*"fhvlC]Ď9EOsD=}MWm6s@8&n,1#ˤ%V?ڰa8tIQ+p}"!o#v/]v4֝] [b~g>##ۈ:_x}3 xLo=ȈEia8.{:?+|_w<ɼƅp}H>,,ʣW`+WVV6'VWW 7nouaoi˱-e]FrK9~TFtc/M/f⟮sgz6m$?A_\.7޵a8qM}TQ;$z ֻ̽JL$珯sT CN(VcYQ_su7׫~U &S~%!Jڧ}/3{_$˽$]eLi[b/LaX68^#-oQ^_8=Rezq߬QĦƪ۝mzm7&cIsۡ<p?}q(!sxz=t̄- /}u>pf<]Cun_WֳxSݶ_YYZzЃO=`.sq]/Uk]c J.^ ?my%T/C6{,#ЧpLeM۵51D}>C->Ki_OcՍaGc[߶gnBSo/S;K뀽~nV{y-yկZ\Ve}"XU6ss|ǧգl{x3ŵxE9ǂqϟP?n?Oﱸ?Ly(IFydqrbA ӏ]\hP%;4Qzn$iPޜLŸ`H|( 9wӯ/㼰ЪӾ4_Y^XDOף|NܷK[юZ[eNR1Fχe]oBi>,GrC3kSĶyfL[mQ /`ož-Z53T/x~{GP dž u3MqC^7^[5K&.c_*VƂ[󵰮:{—Qj尾 IdןnXs\slf-]b<l.1u/2Y6~n"(0%;!]3V3ӆk;>hǂu1vn=<=Zs} uD#jȱ+q~/ՙ=m۱߲ncQ"cַY]fR纩 W~x$UfzNyVB\Wj0u-ƹzr=>3v|-`ƒ6[Ӛ6s >`Z y@>H<3-o;>3m볊`gng(®T{vg6Ekn{vLwun q\[=Ke}[Ooe[ne9tN/vQYn#$׽zVzW'y)oߪn9>m{X7hCg8tm[ݖ/]:Kwꋉ nc_'6E^cGsuqڛQu7Zr/CRrib&䁣|!֒ H\&ۢ"6xk\CyvHj\Ӎ!ӶaQu*_{+\Ko_ps.DvzPiKٷyX]5cYD<eźAz8mkC\W<jm?hv:[k]3U}£여m}i=IŧkƺA{įdM_( ik/fտ]Ăn},0Bk~|BAɱ2R3-Q[Fk| 86ӞbN.kGg-Dݢc涤(Ȼ=ܟCȬ-}d}J{?/Wn":VhU,lJ_ld}Jܶ%3Igᙜ0WNRՆ,;܉s p=.ڻ^!P4E[` =簔Z@dGdSV!<)KԫkK^.ر=.|&Azh¶ay r3-fe-ndj4oD_ih_[mk^0/^+y-$'q0ߊnEguݫ[ku ]¶o[)[1gq?]O_|A@^L-VB,u;i wZwź]kQ/5rMezu jOGaulH')7.XbrN{ٱ}" y1Х wl&9/o#-ci pm+^+HmО޶?oXez{+]ؽW&DvzP/wBPo2>u;>֖ݵT{|HsSSuM+ռfJ'%5K9p_r=vf}ӵcݡ>Xcl Xvw}6⯢D̫_?6nz>į{}5֗0l~iUox -n1TW&Is֯ZtʡcY6#cHwQN]X -|bcAW9߹7#>ByC{W?_ m[Q5{iOlh{kzX>t.+܋1f܎x6}??>=Κ4Ed EAԛ릉r`qhʉ#'WQ.`^;PcsݎQދB-.px㡵i3 e=_Hu/h17"2FFCxpqܒsoM؅Ю__ #b;3Ubx {+jDŽRt8?@P_vX.^f^00} }^\_\ h2B5;m4׊FAQW7Z\}Úˉk?c]cXxPqP{0Z{ߵ}b+^+sRԵړ͖G$Qq dW69|Q *ΟoǦX蔕>k11@ѶAXdF#~,ً5S=)a[8;9(~ULE;G=r j 3<YC.zk\_"o!ufʺ53Avy/Y7V/h5ỵ+VڶbSفc#c nlx_Xglﺶ޸{>S1]+bC)ǽOկc2 Cu7lmlZQa?8uAr#}[Go~->W5a?buCضa.c%Ţ}ü2QnbVƿZlݖAmgnVh`ք~7Q'}eZUgp/۬іATmP79CMHic]="~K_ٷV/X9ZeV$tD6sUܗ Zʹ.5o[_kcCXKD]ߝ}Ћtky1/DߢK?a7uc\ߎ@^_Ok;/y^g$]˟hƫO1GziOj;Fcn'3n3o[Aw٦skƣ?uƤ;bTm PmGZDk[7_a;D3N難|}686ܴkc>(Ϸx otvFfIߝLܗb ]/'fH@Kֽ+R{etN`Dvt'E|^(%zѬ-Dۈ8,ә場S>VXwuzM_+n`8՝= PY.[ӌ}4Vحmv6ofFZ? $|x"G3iQjq3򶜮ۙ׵Z1΄6l8oQ«#b(g~"+F!ymLK]}AR}on~쳪fOITS34Ol^S}`OR@lļk3/7tz})]s>A[uy:(.϶/nmQVg]93Ҷ UmZ3l|~, SBo=NDsG޿uĎ]AgItg.ۤ-e%=WqV9ǚ?|]5z r]zo ߄]t,qTQbe+[ԗ2Szԃ˶op F3p/$(>ua3@=Jfʱ`(=_u ҫu+(^PbCz)y>ŎA{,68Tq[ }p j_p@(s %/vBp+؅n8+n#G?Up#BQ{n $3$<.6"^$~Klf}x6>>p@q8s gO?~8}Mr6~Gg/ Ӌ/G~Kǃ'oL4 rz ^gWNtmߴFsXg/iG"3-`P1Z;byˮ znk s~ӳ{ӧ̟Oo\8i[3=?vn0-I k ?|4=}I|¾:rg=:CRΆ>0}(JkUHg|5{g>;bXty8y_njN77>7غ]I隴Ou?m ]iCl>XAt2"Bi|68|ʼsYfG_ND/LyU~u.僚?ޙ{҉`Yv!/R >OO>ޔzzu ϛcx]Do [1;P7rp塡*)EnRߕ];]{Yk[ݦ4.x6 mtُxczQ틾m [f G&[vxVw#eq6q1ިغGb+%FN [Ul}b~ 18iצVn]oPŋ6^7}{/}?a{NIL>x rLmϮyr2-B_Q׳W;Q.oeL>8<;9yQħw\6# ggxpƚK,܀_cڸ͛p0 ݰ ?D}4}+M*o&?6g \c}k|z{ X8>$bOwEv>d6#z]6. W=-)5S C䗭m~uƫE7/rK9dcIbQYq7Ka< UՃʶgA87 \wHtnsޞ,¯}!Qqqc,4 eeޢE{uYX3Lux3yȃZw~9p1? c ͷ^=7{ij9Ә~3C^ɽsox1Jy퇈XF8+9TkS;׹Ai<\[ uvqX1)$F9zF{ y͝(w eng \g޵szۼ /T_YϻBv?=\׵ՆdՂ A)7E=a;Uڶ;& BzNسUMu(ۚ坺VfgAOHtc`D4tM4ް2flDypҢ#ߒnn*,˶z+Oj؇j%%&U5vm5u0Wm1/)XgNlWI~*PcACui~ :|ւ=hO:EWxKafdk8s^"mk_=`) &]>Ls)4?ezy-r{$IYMU;?P?zH+2#- qn Tk}_9wTf=2Ć عnSmT.gTqxB-szy9>=zB?yf(gX?sc=x}+ OVkġqQ>!~c}f9S)[.ZutTj>5F8^z#s\8*d?k;x}M9zq^_kc~8彗w13t /i:QM}H+~BZ[/b,F~uôvS_9{:R?*/ohK7s%Ùgm TXdgOۅ{i2X 6vkiXĘjB+?n+jĘ@,qs{&MGn*︁\~'Qfڤ"_au+}XIvxi4S0 BH&A~+٬ޯ*>ug5j尾 ݠ?_ ^k]ck۬vk_ϒ›֥?ZW1ˆ8o_Ʋd"qv:۪}fڰcm'Xv0"3y/ٹl#)TC]{hAR=W(<RLbWJݫ__dX(v%m9g|ߢkX^{H\l?Ő׹|cKL"~LV- -B[|rL>׹s۩b;u^S#9߭طYi}!OQN>Ь;}'6en->HPqgmܪrl#q/0iO vo`5w Ky{On~bu6U}}|inh;\s~Mb]§nv&f|4"5P_ÞRsb1{ظ \9ʸ;u@pOޢ̫cgd~&=;#)ZV9$ZLΏ6˸u5ĒMgkw/^ʧJ,j>m׳\vŜ[/ӲL:;oW3kה1O쳻!NҙYxOk㋼RN~%j'H8h 0SV|>~|)}nY8:2,ʶliN{-ڹ9G˵p#S}筎egʼdk}lݒ'e>FD9QdwcT%AjaS ^joA=vdCد)*cvx4}ba;kv>*mmkiP*o+OP;nc0]RC_׶Rˠ obWgߒ/*5cgu2ֽZiӎ'ϛAQ!Sv6ʲ)RMyHkkzuk=xe\GݗŚrƯ\d٢o}9el+6g.ۯ,Ҩ>G>?VӞ3vvm^ tiBm!m?ƨJm6xs#Mq@{k9.4^,I] 5XKuyv^eFۿ֑ 4ˆUZiYUe$Fqul\o9VBz8c>H|g mr^kK]O sįN w8q.}P}nYw%l_n ={fҫ¿[|A:h6:>Pu=f%[{& x{PWU^D=kOLsH>Kԩc-d=Co[eүc8l۪}Su~{:5E)/yu\E"ؽzl1&:>ŭ;]7R>%\WlAv dybCou }O?=?7qc(%ag6~vӨ6q|ozm ͦ6߾'1jIDAT/o>وW/J|{*Q"jA] ` -pkEݎQދ/X{|O>k{xcߢZ6^-1o2sTsݍY3ME^kϽA7][GvDTΫ%yM һ})*n.;WAɋ<-*x/Ih>aImu/kux֢6'&6iZ߷5P-iov)6h͵b)4ƱXo|Xu9qGsk Z6}wMSrVVH3k.Kzzn5lMe)oy^KWbȷО6smxJ"1-8;qDb;zbx]kbD$#NJѨ9+ڔy'9W_gfPyUݝv˺7)"Sw\] ;[|3T}ߎ:yG]϶ٝ y?{='(d)'m7k.QuP5>Yk3㔓cW>,_j|d}] ftmQuJAvJx?٫kgE,w0ExdWg-?E=1WnqW38hl8R=^4zRe1X3~4M~۾~D1^͆}XXé>Z|ڸ}{w#6=^1|{nRbsj7dyhS 4e@Ɂ)QC #`/(W+`{LMn Ŝ&d0^+6p;XP; [ӵn:Vl{mTgO XoIucRQbo!2Ղi8i9wnV1ں5ͥʴX^~)Y%+r+ڴqa?hXrVX _\0u:/۰;>S ux)ע\bqTB9eYP^M}}+ym}k7~湱7BT-ʊ,߾vs?|(Z8ƥX=ZDƘ$kvHfc/)|'*hp܍)ryQo7/n 1'>pR?jk)+US[/0c/5I!}T]$y}<3$bDUQyq0<3:cg5jqMt|hǖ-ǚυj~+|gݶ|^5 x9?zxc8`41mhٸⳢZS:ؼ{cgWr{Z}]yNv8P}#{}}BҶ} n?>@ؐK7[m{m8gz/C'6SHPMy|u ;9OZdBjoߍ&Ί~|BSD^,[';>֣}h]g G{cܷǟGtxں~$[7kzz=Gܟ7KZz/rn*eKްϤʺC|ok x'692VTNxkwЇqq[,11o{5ˍ^_eyGGלO~]ʾe~y=&sKo[<653Ҷ UmZ3l|~, SBoٹOB=R%v ?#Xl/P`@Ι)Sߞ"2rAp x>5I-lQ" 1"J}s~~(o+Vw={Ѫr[~WEro%zͦ3+C!oMԯ}FOz>>zV1YcgPƣ3E{>kƙXKsh,ە;Ÿ2uӴb)-bF{2c53=u(+(&B.tڄzX[J6t=K0F)4pxBc 818/$n'(n cpFI=)t2V [b'9e/k_ps9ΨN=W::,<Zr/7&9!e)Cl83 Kf XxF!͉e9//OrES }oh~]3 {gq;]"gT;Ar$qK/SV7gpkT!`rc?F,/9oDفE !pXY7׆18e83!q_MD].Ļ~> z=K>r:={8 _.G'3>p@q8s |vo?:>_M/OOxi7?^x1]?_:<dnӓo^z0ǺG]={1=|M+`8i?k?Мۿ7(֚GǛ3T ֩Ns{ӧ?|4=}K;1ݛu|sT콹3 @ge߾\p/s[v^㏹w fz~л߃~}t_?=?^mket$6w=N sѶ~^y$qteu=7:jqqn'J5iPRrʇn%Y#Dž]Éq4"Bo/y̏_U>5ӫ\OMwU3>|_B_ڥ|򙟞/|6)t'$[7dP'l>tcx#3,\C6Bߝ|ȻϹ7$W{qhöHgQu94iggQ0(؇93YThVJOpuٮ,7َ>Oϗn^00-'ʃT";OPYem^{UP샪S C䗭 U_jэ#›y6MBDeQZ AL^D(./_(scFAIKܯ:6յaq58x_wWlE>CF^v\i;:}9G:GuS03g@ܿ8z8NץÜq\,tn#Y+?{!V8yMٓy |XYG(j堔}{Bw}mw>6$UD8s鬕|O*/z-r}֙I_9ZP׭}unWif=1%Ҷ_®Zl2Eq^9srw潢q߬QĦƪ۝mzm7&cIsۡ<p?}q(!sxz=t̄- /}u>pf<]Cun_WֳxSݶ_YYZzЃO=`.sq]/Uk]c J.^ ?my%T/C6{,#ЧpLeM۵51D}>C->Ki_OcՍaGc[߶gnBSo/S;K뀽~nV{y-yկZ\Ve}"XU6ss|ǧգl{x3ŵxE9ǂqϟP?n?Oﱸ?Ly(IFydqrbA ӏ]\hP%;4Qzn$iPޜLŸ! jП?a9p-b16b'\r_=eJ['ڹF|x]Zrнm<-_AyJ|aݣZ*2u@lpxãM(͇eU[nxcfmض9݌~k-1%ط]+9&szU/:VmWdwE1MDܟ}`QmOmo:Z+2yɥv_ k^BAy/^5Ǯ15mR~ٵ/gclSB}-+e#/r cY}Z;zm>3mرhyv,X [XGÙJ=O5ؕظ ̏ͶoYp(1,Y3vsU+m?b<}և O3Hm-]sv7벭72:,l^s=+reoUUض={hǃ!}f3q|[:v|u6-nˌفc.Y?ƥk/Wa"9ǧ581rԱ}tuYT6Ommgޖmlk8ښ&6沆_AoqHQv6m):A>uf3+֒HX7(]bmHja[.Qgkk[Oxu/W'tX7hG+E ɶ۱K/ƺh?Av>E=Lgu q!m ڬ=ݶX|֍FwsٯZ>29VFvo3JuCu/fٞo9#PɅXu[X]sܖeyA8'~xEQ EMB WO3Qd7xw#< lv4=ĵ{OVm_.pEyeo>ukprЏq*(73bVƿZK֫JFkϻexޙef+ hmN)_+FPRL%W>Y6FkNWh:/uk~" +rފbGŠ^=WV4n{g&bM9\d٢o}9elyg.ۯ,Ҩ>G>?VӞ3vvm^ tiBm!m?ƨ푚ssIъ 87d'ϛ"V*v3A㽹q~nǣҿ>2epeCr,J=ބcc@Fqnu׹rs}yo\ݵz_tXwG+[Cz%]h?>,׏ۇa~%-_Ztk<[3Cm~{c>gE U }\rƘ|VȘ,ҝv}kS7!(e XUhw.oLjϪPՏ?F[4Vix ^Z!Z{ڣ7{?}&K!1zug7c72^_ϟ0~OO M>_sѩ'31©jbQP7=&-i"}X\jozr_ V>DՂ9THK ]Gu;~Gz/xphce/_ >QMhO|kxp܈o .ƁڳsK^kϽ7=_lbB~}5#툨WK@.Yw)buSU~K\v?P;,/3xb[,T^yqV/uOI,o(tS֝m6kcӨ׫_r\jc-aM5ͱ1V,ki<ڸw(aonn#23xIfShORo?7[voXGQ볊ǁ8]>Wژ}D]+l:5bSVUoČEaet!ng/LI֒{LŔwrv?Uɀ胷V ݰ3JukEٶOx%k{0qmX3 sh#c/cܛ?LþPlzO(C-m+7AƅVƫ~lD<_Ux6bϿՃ/o韦N!6;1Nv`P]!mzc&=UL\W {*s &nm"& T9qB<( y #ArjlOО -l Ҧz*cGak֭ JƿaA 2^-n #FŻaxYjqu[:9ZyGm[DEiU1½RKoJ,gW/WPiC~#bѴ~m@``;u^aw(*}9ݵ+2sUܗ Zʹ.5o[_kcCXKD]ߝ}Ћtky1/DߢK?a7uc\ߎ@^_Ok;ssIҌWb%ҞvtO:?fb >i1f߶N3M;֌GIw<6łFڎpTkolîw YW1vg'>W!97y`tŖ^\#/gP^W'φdy@> 3uÿc7cyXy^3s38\W[+z֍'{i!Zlzj]~Ɩlg#c=ms{Ӷ[PR3N =]wHvvULb`ϟQܺmymƏ=>;^@~cmzug831byVU39Vk-=J2il[{m wx_?~23MuN?ze:>^=n{?<)Zžޯ| mpli ?6=^|6 Po&x8%řWӿ;/Ďh5'q^N̐-'{;Vʺ9R彉lOyÉPJY[H2VfvA׳ub![ C>KJvv)ץؐyHgޫ_n*S>?}Lu(P-]Cm1_b~ۢxsr?/XqݢԖ)֣\53|0𝱇{ )WDy]X;?3eeAU7;oPE8k^ݭCXYvG׌u:yY8_g/z]%ڭ˙RIsdײv:vXfgi]L6] 5&o5JuxȺ4 vns?V2at|F,~ϟ:8 uȹy}Z|Zۅ͝qfF&J# kƫ7Z>k_+zt9~SdbM)ǢNOkJ}~D>Hh?%J1m{gBGkK.cLKkmAtr?b]sr_)\ww_kF @kzƣXψהg t&75'ؓKtKaa,.vګ`O e8Uy2|0xAAgݻoL{'YEЗ{pC~I 2f٥+Z Kf P>G]X,axqs]i.6pY5p9O3 7gpk q8!pI шC\_%T,br{͈~LeCrp\}{\g0s{?G'3>p@q8s p^9/F֔[I+/^=I619'>nNQwl;ӻ"洏>;6S7D^OiNo}.os9muW1>kcm۔|?NݛNXҶ,l\7G~8ƁtO?~8}FzӔڽ[s7}ٽW3U6m~4=}-gݾ{[`'AϿ~EsD^;n͟S+'+h#q_k~^ļ-~U'T,JK4E巁ˮeM Ŀ1f%F". ]|U$R=YG.ӻs}CϫY}p׹{ҿ.wq=$¿7}J¼⺸%o?A/P]~6_^>~EݔrB?ӝ;w]ޟoӽmܶ}.nWxoޖkn{\-1(w iיXSwA祆 X[9mԧ {}c6{ƭ;]ڬy$?mIr+>^ßuIv|yiOWӗv )Pdv=}yper.mNqnB~/[MOk@VhS_/Ix-M^/9g 9XA%DT_0kݳi^mj1\ل"ƿW"Wf̄7a?r2NzЋAЧ =_l1܆錵u1ު'<#"wfA~uљ}M9idKӯ5_φ;x0q_;3nd+u }+o+>}|O[a:UmWdu,V31&{{ŷm^O? ?lg0qY8";ŚM$'Av|%,qLBLSwyu]62Yڶvۺ;1c 6mIy|!|U:MM6V{1Fc-za/ٖgn{nyv ?՚}ZXVGWqi}v=곃̌ Ǹx$:u&|.xuMge?g6{'MCo4s5{B:Q/?#, ~]V|(OY<Btvk39xD쿵9~!ƈjl DnPclH?Xƙz1~<ԦC(x(Cr砹9VZk\NȋDlMڵNIP_'a^t= Z̏ŋ\N$`͔G6]^ۥEb@ mhCϸ} &| 7 2.41Ƌafmv:_Womxuxrk1fey󼏎" XmC͔@RO1bG`w?LW?'<W2}{+eS=Hhs]Kq~(}`ͭʵpY\;Ex9f{u%DZU+d5;*MaQ!ݛ)$?ox!o)_˼ow :_u9g8,]8|(I']ڭlnlChv. XY(Ҏ!Vo/%UwYY' L_mX+VbA۱=GKus'6k10[pNj%=~&5Tvhiwx_,ox\6}?&=!;mўZW$K80 :#._Y/ׂ}Qo)aa\2}X-veA KN:Ro9}"Q!pP8\s .,4|mk^] /~_J[ݛ>}=:Dg?> D⥸_EvY"og0A(EoEgk[ҋx1\P6mR̒[! z wM(*^ 5kw-2Zio{^N 4PGg0^-L6#}[gL}N\wf\\|JY$3 ԗ,4 ʷN/r[}UYh~Ugpop–E%#oiӆDžB|kCu۝BYѿm[Bi[| Y{xqr +$" Q=AX~m ܾHB]Cl?/,,ÂZ$+qfϜ/G}Lmkcc7aEeCuA+ΟRѶtbx%^ ^g,qB?m}5\`aqZ~́O=_1zvp#3.mђz_:uogo{nx}hQ-1cKܵ mS?4ƫ,@=|5AΞsBևP> {+5 _b~+|_u'To-菇KA>uI~uz/)](ZhH5 ʩ 㓀yPB17~}<}ۅ/-(3|@_m*Tm>8Kҗ w1x9H_BI_N^o2_5X7& !<,+ t߶C{iO«M@ O!)5'nbOOB1}=^R\}Y~*ώ}IY qm>>}OWy(ZDB4~hk-6?O/Q>v 8!< {Y'WRށ~BLį/ʏ/nDyG@_o؟A+o16#:K>3+pQ?d_HOzB|+|ջ\8/?";5z}3$ʿ`aQz˷}f䵿]Yq?>m @paП_)gX A|nS~oRwˈOPk=zS}~}}^ _GODd):c"^~z_oq_sӋ??pI@xdzٕ>ʣ7Ouy>=y`8\ճ3:ӓo^L׏ kӕvf3}Ƹ qz*nz= bt^<1 p5]9վdTog}Oދ3{M9/}|l~%J&ιeG}|Dw/器_~o5೑_;=Gw߱yL_{>ZLYwwߞz=:];]{nS{u8`ފ3/;O&F9oQ8{W_gtzkpʔ_N?կҎICIbb$w6_~}9ݣv#oC܄w7~2Vw9y"LjW?ΏG'Uo"Ύ;߸&D}Ntn5VhF;tbݯAgu3"ƝO*_ ★2%r6ctW8x;X:\՘1%zLvB*;kVj{$ZstNi?=y/v޿{Ƕgc)ۡXy^zӝϷkn7˿+'%Bdx9}~ 1cDT,/ `)[{Q/E2?p *+D\YEЏ"REdpMtu1Bk6_mo+R`WM"x۳C]t7w>nݯub["=D_~+dlQoDcq;9G;d!.\t}a?'W؎aE* tAW퀋K]l}%M5vÁRNgg)$Tx :,H5%ˡl,Jh樱]wE7ٹ tqs:-vv/X3v) qIzkTn=0~՗3zmKl oUaťXBq`v8WϋZYlu6xKs?$z/? ٽ/=1g_K -泰Җ߆ ~ _+J7nKf!57=. Udg]gbMg?]]1붂{SĽo{B}Q7{oݶMqx#~zЃ : ⃤?H(?V4S>en:yR:$Q̣_c:[CE_!͗s9g2N/ 9r|FDߏD~6(XjV|XXH<7f\_biISdakc{ʁQ3 | iyqMsvDo{M1qV̿Z g3sIafMR8B'nhie;2:Z㳝LPf"[c"8I} 8Oprgg<ҵ`Xk ~)=c){V"ģWh@s>i|Zkwmlgmk8ksl"P̏wqzo!y;eSŝ\lzmjֲvFgDP/5_%vֳV,[bNz2&5^i%-{ S|-ܯ-/[y»g3W=ֈkjßp9o/~SGsۮ#tDf#\&`WO^ /El˺O'ۗ)WҟonEz!5=oway {!烕@R_B*xyik9ItL?i}*S~ߋGs:ņ7ǒ8 era8SwȲ=7ת>4Ӳ&Ҍ3b0#TٜY:u8#kv [q)B[_Oi/E_Y4 1;u2 9^a_I9:m+l\-97vzo+Fś{9!)Αڡ4sucmK]Գw!o>⳪ϼ~57[4לkfC D{2qDqE_!//jb7_k YBt{=1b8}qks'1C" yi@M]ܯӨ[y8SbH\c82ò> 2|wVZ˛q9+6:0ui>ж;駥-_{?(/eE}G#YBm72]n[[Fh*w-Y!6/ue_+/Q<@Պ5>v8 4q0r֔tZ;.Qi^ѾPB8Nu33ΩN*oAy-˓Y@kkώg:v\5MؒcTa}(8|㐮uL%^ݔ;*Gh1Y&] ij;us^0)r}1mA!o#"Ş_sb\.X;V1xo[r|Scr<{'Qb璏6~JӦvE_àNJ۵]YϺ pǭ~h_e<{tz/j_j]rY\xei8¹5#ĺ_-V"j' :+׾/z~A"v}6ߊoؽ8^?V&Cx:b0rM jPV9TqZeyHvJ%R{#/`A[KG?z/y cjC}n6>(ĉw*p-KqT]m~ub_95Q/sl3O!%6x7}g)ޕ=*]%ǘG+&q ; Qr Aj٩[-26n_|vޚ&l1*K1k<uђהA7;q+׃>u/Fuovz6OoڡJmX9.uD|͋)qáG 1OFu9}{&c-Cyϓ,eGO^ڱZƢ%bܿgh؜h{>+ӥE_vWʶ?-l}o<Ҿ?~Pڦ)VwcZW}=˺޹r{ pj jz>~|Qo)y|kh [C=ٰ~xA{Rv8KT^((!=ykA|.݃Awآyu%ܾ}-+_aU}?pX?s_Wqe嵵uÝ;%AGo 47s6?fcw,VViց ~/ooMpZPUHiU8yl|#mݡbn_k|v8X)l9h376#i33`n ty?{a}cWf:u{T .tڎb`h͚bx=Ƚ-I&cY(O'VvMkԇιn}l7͉iwgoH0W%14կJC`ݏҷ񰄴_u{f~~wϷg@'Q #ƙޚ"aq=Wev^%N{m^lw1Y,mݯ11G3x9-9F7qFế7dÏ(]*Ы뤵U;(<6;W[ݩƦmEt6ы=͊5grc<δ<(DI1SIث{&kOhPVn~}g )RCaK^\.cW3;7V8 x _P7=ӯ*_nZ>_kNoL:uy_c.^D#oIcͱͪ_sbUuXxu{p^y1Gr .h cRrM#1۾3*W=ܥ|Kݖuykv[mJyn;ucc(|>ӊ [OG/k'?ӣ_]W_pi~vDR| ?! " jY6OO>s|6IirzU^׹dQ2 FW?˽s?'nپt9pӕvl|V e5ݔs]wQsHއ54av 1n]Ƚ ]}tˢuQ|]DYʿQvY?oL| HbsW^U7 9onk>Qu⻠Ƴn[?RZ7oRXD_~+dlQoD#1 ŕsG`%tDĔT[a&%[ Qq#[(8kqp{).y~,s0e[y;]{K|elz?}hܴm)v:ka#glV溯E.璎7379D\ pvׁF[#>v]17M/(.#, ~x V(O4EtEDLy>-6'/<$͡:m!Ɔ,ccM]7ukd~'SRVWp,؟b,iltjR=68-чLJ 5 sSP/ȹcgr*[ڡ}ڧ#Rl{V/u8ICLuXTBNw{w?q?[/(?"Zhp_# sV\ۊiNX!/ kF'XbM?S8nsQ+Əw_dX9\)YaŽ1Ϗ))VLn*tYȏ}lh?m<*U;k_'AyR#&`+W}V?Dܗ|zqmSi,}~ x> $ W|( ROUyHvJ%/,:jvT`qF[=OG}G^sp⡶,+[iʸ1QƸp߬{͟94M;՛Gjc5kHM4)|6#IzB}5pcmkUmݻFz*4ADǢߋNF{woC9b8\KecXn@V@%Kj'|(>!"FeӿUូk;Ҏaͱ9qc1D^K/"R~ j; [Rj_ -To döeA K> T>cL !k*B=JM^uͫ+kQЪow>~8,/qcOpycD? =F(! >Iwn5$}ڧ/3eacn{_D1&M!W k˗Bf)*8ZzX elߋNFgw~x1U0̛s,hxj^i[OY7+նח#UqԠl|SaGJk*t-cUrYܯ n[زWK~ wƛ6m{\د~mXr>PdE7p8N(BTw ,["i[G)۴pÀyN_6[ N^oƣ^a_l*GPދYwn6=aaϪc^Cu|,}#R!E{kKh q+ a>L'D_C }/mӺhp_T}FΩ?8MK<7@J[**])pp\S8[neJt~Ap_㹝`ij9ĺX+k5o2ӵ٪99ӌDu<¼qI ߽.]("[جHa7 ʩ}s1/1um1bY\Z5@-I0M랩~4Vve]AIiBȏrBe;1zߛ?9mъD#;6en9::GǟxqL2/1LTбfuAn3; ^&H_``J⦛~~s_bFċ@ }p!qN/!)͓`pkS ^,nC}.~ a`' g}́93>p@L_^om2nLu,-lcrTq?L_|O8OGLٿZ*(q01䁺/OOe~%=._wUO˗YrORY`j|A? ӋJQyo&ߘ^l{#]uPnsޞ,Z|֣f]ݹ>45{>GS 'Q?AW}*{ߊ2[gX q81q[նc>eY~Nue;6UiיXSwAG/k&n+r>o{[wƷYmA$~M1 r<.ϮԟySKaK9 U2OJg폄g%ݝ"}O b^HmƣQi] 8 Թ=6͌_Q0}wS_ S ƃh;:3p7k>M>'0*ܶ+,[d#Tkj ,m~['V|@/qlo}/XfBBQHɂW][>)KE+-(R EbCj;"VbZKC哨D"^NIK/FA\ܧ2W(4KOv-b !fblxyZ+!*0f~dٶ23a>|-ʘ->kXGkLK_Sd3B}ݏX7>8*{߃|<8Ϣ7V(0 au!:~!jl~F݂cCٶc-Bh'"L!I*;Rt&+E 幜D{"_[EJ8eMۮ蘭">'/qE VDOmSxG2F)GS#DϷÙYw/wUhOޚ3P8\xVAZxV>dϚ81GY5}fDX E ڪ{|Oku+}7$Ěl M[FSx)_vF"/,k=^⾵Q/m+`*kΚ(7Ěuxp=}uu\3_Uoo{M_RڠhSYwfWq?%68K}ky@/.ϒ#K >p@q8s g}́yŋų+?}GӋoLT?|zpd?~wO?g)_Lwmkw:7xYSn='{c$瘷wRܟ|)_;=Gw߱yL_{>ZLYwwߞz=:];]{nSjwų^f619q ӏN߾/c6_3׈__ϼ7}_7}K>q.?7xzR͏Ų|zA/<I4qe_o僚?rW?~~ԅ H}~ǚ1oh>vI,B* RX b"GvV)"GY .[ L Ŀ1f%F". ]xU$=YG.ӻs׋dӻg q}t>ҖUqAx!/),K]\":M0o.q}IgqτDT׷嗗Q7l뾽\P~Ϧ7t'B{݇DCתt/g  m߰K d/l%%F[;ƣ~َ@_Avzu&]y`W \S ^طqO"z΃NcDS>/)I_-6]!z$A) EP"#T]{];ݶN s[<_~~=}:}F/nz@ga-xј?`?:ǰqKeީ[ޫ距?my#Vk1U\Zx83c;1.5Nt ߇Kl>B_D(rZMVL΢|gGt_/MTObDҀ~ji{U3qMQCqQ&\As?^s(M׸<|na9kUO¼zwP".ShI8T)Emb׭&*%Kmpn,S9i}qyLJL/?I o?e\i ctڤlu(3ۯxcb^cˮ^y*NE6ۆ8)a灤 ӟb<=,؇#,>\/]hH[^S+׾/΃UV e _Kob~E{oކO\󫄅vq:H\+s\לCaWWBeM_@Q$ڽ?Ob{SV=b5K|oz[w}Vb|}ѥCBq? ^ۥʖ^ Љv<6lЏ!2)b՝O6_N:iC,g&6^ ug}+|Mm ֞%˹Xl-ڒDaq};4i ;D?!~K7-!8,򪼾X][C[R7Ɇm/  _(v.|l/u!Bb=/4{ 6fWWע,R8V}{AgO'ϲO2?x)"z 5C}?+CG#JQE['Y.Vbx%^ lf;Tdd+Vȿ`8iCf9+ƣt]tƺxٲ',4}_ϿYD/KyY3I`W\9_܏l ٙT]7&^nŠ*XVƝ? ^ۥPmJŵ>Y L> ~Bk>~'g8~1׷Qß?zc*B^bGg]۞%tC,=[b6kڦP!hWY{j.7<=c(u)Ed`u)D9[D?=bA՛m}˶5qߎWK}MܷvW|bh7ōV{OQÁDJ,=]a ZIEۂj_j9hWeyw,'Er+'p݊nېD~-ǶDzʣʟPܗUđH 8)ƺVHOmbW)4bX -mv?~/u>S By/{"q0m];cE4.㨱VMuH!vƺ[_̝~3ѩh\>#r:m~sϖ^}==QqڬǽCg㌈mZ;DW9/ц,O8>Lu'ְɾl')QwN8}IaBK>8K w1x9H_Bpi@ O!q8s g}́93>Ń'/ϮQ]O/y2S ؝Xܟ|b~Ϡ7mXx=3q&<7mSqYE}^j>9}YqPWkyl9^l9zv>sgzW?UÜwfʺ;()ue>?m.ڡ3gmmuWH?#8`Vn˿1&ӛqh:*C dL_|O8OGLٿZ*현ð/rAÜ|6M×ӫnzdo!G_Nț?w7!ݤ8}]Ͽcg~1Qaq4ׄȽwIczߟ!]h۞ѫCmtz_v%bMw>~)DOsҧޙvWqv֍mueدtnѮ ޽iǙ_̹)׳"Pl7_AN]5֫~.]}M@̢Q]WʿQ \v@n!H˂sW_U79onkvߘޝ^'ߞ}8]>K[t;c_* yK&%}Fᓨ q>=qoE3 s,nKVvb]:PH A4qF+wӛRe=} }%ιM5vÁROg'?xeR㚒o>ÿ ZWMs߮p,fV}8[ZZ)Mko|df|ƫm8;nBo͉:݊fݽ6F/(}ȟg8zq-γۧqFkݓԿ/~.W[b4 կh<* T%;C=:k.>+y붂{S)'ط>=Tܧ~/nZ>O6zi'O%>| 2z@M_ t(JEGFbQ>eaP:yR:$Q̣l$F Cc7^-7Og?u2_ޛTiN}3z",mݎ^e8bvXlاGq 7Zc5l䷡a}ZEە9wuha|:71)pʶ6 ?~l^Ʈ6 [n>f*1U/cQޣ5Z`ʁ5h{pPܷ/&ѹ"{"M>gܭ0*ܶ+".i5>Y>@/suoMzծ%N+LJ CRC( l`[~`R۩~>(1-xP*Aik9tHtdL?i}*S~ߋG#PiSlȱs,[6C&8S{=qW[U9:niYCyѦB4e8 5po*18~D{[Sg%#s8$L+n1;Juo3%;x]H-|a,<՘.@ CovoO[9{~*ƶX=dMYχW rDz:uw]w' 6BbX@m5[{9n9&iěN7kL>,i2ydׯ;-= ߅Qt|f?gao. }]&ǯ w=ku[8z?-]umݢ_6~9?`L%|Ϯܓ{pcN12/$(h*E`e7n#(O4EtE(DKy/^7omN$_xHD1/ >:/vGX-Xr|〶B<@C}P-VsQלt}x幜||na=`!?kUY=q_D倣2DpB(1<;7v;]K!H m٬zG0ו}GvA=,[8AP-KaG+)m/ooMq8\pJMb(E$gq+dgj33o۝SUނ߼~QLǎ [r5i|%|}c~57DXt=}5`q&IioBV)`Zl'_1l aN ^Ыgͺ{1e P ; Y=S9~WLPL=kSS|ai%ֽhWO˦ϊvXWۘ+ۄ=M{c&ވX0^e#NXKm j1X΋W_9Vzʴ:Ike{N~}shB $t~"xU)rv(ntv+ n_{ mi׉V̯HRdW.r5b>-C0`JDX~OG}G^sx:X˃.s:=k΍ÏAM`i/!K=~2žrlX*9pFl<0=<[Sr+'{TKIh|k'!h-bCֱ+gn xИvVY35MؒcTc|)ibG^Si|Xx'1 ox Me31Nb<==wrۼϨŶay\݉fݽXe8L;TOM.pbë,f5_kt/bkuZͥ,|(Z~o޼=lǽKegq*3VbMoav_w/=1xM6=.G_6ֿ"Q؝^06Ea^i x8(ztNux恻80j}芕C0 sW~u9 3cmz8lz# nR;7v:Ci<61=My_F6sžb x}Dw0v; C;:,1<M_?^'a*۴Z-q||hkPg9-s טϘ#Gu]–r q՛8窼'E 4\g."~xܱZ Rz{]{ݵ/+tq֫ feڪKu~ 1 ={5u\j[MMmqTo.2?ܜzl钞 {َڽ71n5K]pK?s:cQuϵ $~^7z@C)ֽxn kesk_x]U1N+gU܏"E:hËS~ -bt7PwS]mWஉvZ/,[+I1Q񃽖-/uBA:C>H (SԷNszv0h%վl A*[ϻ/ApWgjN zO1"JZqvq6LC"^.[m/4aHVnUL*c)oyAs0.!^+`m֡)⌐V_SJĎq&=1)^w1iݹoI;V<ݯϘ,c"CS}(lkeYZqik~'S]6D{vęD,8{>Ǥ:}03rou{M1iǺWw+fuwb! Z^ܿuN+l7ꭎK֚cɏpn졺k<\'CxJkBD{To:龘6|׀xA%5pr륯sۏ5z8XgPk⠼g}6ɾ붯'{n7k{ý4kKU2骵ҮǽKd/O!y0r;ph bzy l=g|ܸ8ہ Ye3p`vnk)/qa1cn3]g }p!pRPVZMz܄Sܿ\Tfx U9 aXSo\Z+WNYg3>ęJU,p@q8s g}΋O/^L/]<^|d]ӓ&!y5]1}Kۉ|8l4 <|aoqӃ/~1}yH*9N?͔uwx QSګ[<}~\N۵CUgX6%߯9xG\qݖczMwH_7G [OG/k'?ӣ_]W_pi~vDR| Jj/fޟo/dnJO>#?rU#^JW?./WJOO{&ϝKo痜WgӛKZDϿi6z"GwnZ۾Dgn}бpa.1"w?q׾ll/{0"#h__sxT-1(5POĚ :/53UmԧRPOo+B}{i{Oed^>_Xuˤrnw}.c%$h1E I -[}*E+,.+vp9d<[^C/V 4ʟ˛5[ęG⭱VUC/3ZI6m'GxϧC}37 0:,|64zs1~RwǾ/^ZP^] }vP /s#<bltu6^.N([\=v&3V`n]{_4v|$nenp?6'8+?"QH=tB -;ӊ2 HlCؐNLcZW(Oش)nuOIK/FA\ܧ2W(,k`ܦqVTW\^s }j"E2}4>T23Rɥ_ϲm6O^8ds+JWc03Qv+>"@[. 2|}F| O b,qQh`<9܎G3:GPObѯ.#>p;fuP`grf_ e^~/ju[᭑qC=sUAeMT>=e1be|6 uJ OK"(2KEY%FL' QG 2F%Ğ1'ǺvU:J漯oT1ޯusOMeJ[_"h sdn.bk}$#^٭61I8`o.{k3,-{~0{E/i}^3˶rq$q\#ڇkZAmAx)s+$HKxlc'Ga%cevD\vcxX}^7%U}˳3E*Wn3(3St}׉qSΕ`wMU۞>s[{}s_x-cSxYܯ n[ز_7mP_~mXrľދ\GawzI_iz )ׁ>#K1r<|--<NyX-uYa&ЋZ?wa{RG/nB]6Z>!k&7;7ܑafʸbJsT3chS~s+[_]S}vX ǯ1Dml4gez} ? o|SsO{tߝn[УWQ5.ο!rzx uv äϨYw6Î~oyn70~3n"-R=>kZ6DR,Vٙ1]v)M,:rNxcΫ #s.f UzOXVX+@՟~9DlĶ'GBzSư?3f+m5+榵M;07ktǣ wM5бr==&Ϋc]p"ir1ÄoO\bxAd>|vr]aT;LH Rf//pCė{gµ/Β=}8N/|:>-Wuj/n1Oqe!>gg!3>p@q8s g}΋O/^L/]<^|d]ӓ&٥t}9t好<~Ii׏teu\.'8}] _p-~wz/"cܙ>xU0}釵!ʿ=zJ{uzs]w?xivjYk[ݦ>':UoLnU"f'ӣ5_}Nѯu?.+S/}8VJ;2J$ }Ly4n_{Ih]1}V>*X*RupEЏ"RE«\v(k!H˂sW_U79onk>Q>B%[ڢ%+ˇs% LoAO~T\Oe2(7"ϱxqrc*em%}Gc/Fdukk e;6UiיXSwA祆 X[j~^ອT mEO~/zߋ˺ ӓ5W,/ˁ$%ߋy鏟K"F֦w5x0P43E3ݵN}-L1\b΄V$:W|O]G_vEvWA#Mq&z"xl:MI=k/aO-BQHɂW][>)KE+-(R EbCj;"VbZKC哨D"^NIK/FA\ܧ2W(4KOv-b !fblxyZ+!*0f~dٶ23a>|-ʘ->kXGkLK_Sd3B}ݏX7>8*{߃|<8EߘV`Ȣ|Qt9xD쿵9~!ƈjl⾏]~{vk1Ջ!oY8}3%FlÍ 6 X܁mIF_H88јI~`:rG̤bB v{ժUw9]]UVZUw}c .SFR Ž'MICBuD2 l-{߿owDAn+쌋qEVDOUەm69^QyӱmyygaW]#KX=k_['ޞ3`{X |L|]u h|ڴp+\3;^U}\΄D~"x:Ri Zή [qf{-_{b=,q\ι8{9k1q &0:.I?UV FR m!$BloJo5VD&v-iG/h~v/m澳#9}\۬}sNBɾed:eۏ̈\$岞}Mǫb}ڗY6<=g5c1Eܷ±fߩŬ;-xXbjC4&mknfp aL }#]Xମ-VOG׏ %5Vm=$yK|yEMBPZ) I"i]3$K$ˮYXmVѴ9}q?=o71w4椈kdd:e|.\VKR$i_o3ibZo_ukbQ&V'Ŭ;-B%CǩΟ½uy ݶCu|MN=.`\> QP>/v{QV9_b@Z5pbcn a&$B->D"s,ҷ`Obf@u^L?ۗ}\dyESffmϘ9Q.J8DS0E|̄oyxꎉP5Nշ+S׀mUg-yl,}H#ϖ?jt| 1fm;FT܏"&]:R vYmpSDohEmſlG!lM&jJlRN6ǝ2\sؒ:NL!^jJ.L R$QQ(TR-0JDzm { A^O}Gݽ)Y}:b.U\=/ض|~Aΰp,uv#x2?9jtYw-ZKei|,׍y]ͳ_־ԙv93r2^=쯛N6~s`S];ZaU8a3'O Y޾G>[my \?p),6¾qqA8s}3qA8s} |W_.Gugj|Z\MѳWK~r*mBm=$=]=bOO_~n닳keL8xXX"?96'qqtjlVs {΃?N_E>X׹?}*q/+RշjeJ>~OhTvovY[_|~m`o67S>W:+(#ڷ|jɹ'ӕ%>_5bzY^ϟ=~Tm5YَdBsճGva ݝv߲p.PBcWD'fs~O;RS&v'E!9=`l3 P) g)7b ڒy0mھ׷]޲>l*;X^`jg7 ӳ?]>+|Ra!({G4^l ūl;{ۋֵ(^\qyp l׎,cj>n嵋w 'Q?}![?6 f@X-wr/~0\ bF;,HoL2;,D*Cra׹3'> 7֊_Kks}6н^YQ|\_ lvpfuMule?74 v6kltj(kO+{r~\c5Z 5y#"Ÿ"W._7{˅"_^o[udG\:IX)Ja6:uid;{>$uz\'wB?7lw6oyqxsm6}\)o|?9&羴Y68 Kkh!5b,m7 H߲|Yv8 [Y8Я:ry/a|ᚬP٨ u,4׃/ 1'Lph^AgU̖1El}@ڟ ,8|>1Y\ع\=hЇm56H %2^v\qՕq-k/ٖ>-X먺]{rw(g.+_ͶNpowGѷov3#8ψeMSrg>‰k4_OlɡMUy[yΆۛqQ³D8nxug6E|& JLz~`ksmYM3G:fW̅Nl! xxI!k_:9ٙ04چϳd^CXP2.Q}U~!I?<qqߵ)^ȹ2ZM(s)̋YmI!Kʾ*JTb[6<'p9O]>_(rӎqȿy|!ZHRc}J1],1~Gq$ǿmev僘1[(q]׽/"WbRvγ\k%;cCيg\JeWc\ܪ5mvvcPP9˅#s!ڬge"OqI=b!,W)8r^=9'kʽNœ'}OwνQux:~L• _{YWs\{}6IA[/uM{Dv#L5({WU_Vu[mp bvx{mGENt ?¿G?4 ˬ:֢|7>m&/c;~>"ġuk>̓Aa&&d{NDouXlg03=Q~}/-q_(mွЇ:ET}rп#Ey7&[v=j`M3fVO;R2(%yQ'xYJ1-/_6h̬}=s_BXqv&Tl#](c"cl0˸mfg@-XqiƽF>ۻu*2/oֹ;ڶ8ګ}vZm$AxmAH;gy(@7ڛažF{GܯD?,ac`6Bo{\/s}3f8ߋνvoDŁyԵ8.偤?(BloJ22)!Q?4q_LO?}gG^salgμ9V1~>B2:6sKtU_;l@%1b][˄GՆ1^k⺴ qزmvΥ rkOG3'Bg=one\:tu_*ƹy=\0nĭKK5M^{_[?%icnG;#Uc}C~}W .kE.}-_ u,\s۶f{ධ]C5g_7{:m/fu)}!p*Vkox(3o!du<nvͲ0bh9$ϖɰn~F1_2y3v~nk"Ő/j`Mk85[K:X_B3BSsc|OUo^0Uuk-a@%<~9wc)msYg{kGg@-X~Xs~Jvθz+W8K0뻌qnG^oχ:6V翼vm_i\{elռvIw_uiֽue>nbm}Zߗ1_a]%bJ:{m;Ş7~Uux[d,o7}:_a|uj~=fѦE&CVokuxk:aHxH[?\t+!/ǹ뼘~/a#mB~gjC:٪|ϽM3f1q >?ׇs@!Ȫ%Ң >"_0{#0 2жL :R:fiS{R l/u5v(\V8Yso\\̡F_%"D\>K|_okR}snK͑@b19QΏocFkcܻZ[ޫmLA>QYL0.Ԥӟúߋ!sZM}~bogI!Sg[ֱ]M&u :q_  ݅ſl@$}ok]]Q].:u[#2w^ +A,.i^ -b7U Ĭ M\ju&-.}p|Wj3m}1C~ jӹwhݓ=Oh[ܿ{Sq7q|ֽ?K;(ѹ->{Hrc>a>W+["rܯ=.2u9ן:Hٵm*u'{{ptD.KBd.ٷD ̽`C,Io#3Ŏ16>}3qA8s}3qx|9zzz.u.ryϿv\LyzzYg>Nݻ\LW1ٻ.&bdO$WӅUVc1[|ט)t )y0=~ȧ:OE/eo?YʾV@iMM~.y닏|ܯ lW>cF}jɿ'΅& oa MLW"kKN=.ʖ(`m3vAGg+{}[]يYs\n>s:qb6_;ߵ29م?^?k;UGy$P;iUzP@Iޯe,Oc6\${>huz\qzC~/s&yzq6 k>_yS .||KAͳ J*`CYrX D_/f=@Õoؔy=H}kGSs3;.NgUmWvy0̢>b,}|r͓}VUeΗ%_T\Ը֬#13NuS91Ai`wQׂD犀k~Ix~_8m=JGdvm;b 3zWOܷ#}%q,uSӉ{\߮u8@u}m ::s<HAuU^痷AdwD>ϟ7''Ae\;IPZ"P"c!iwt^-g[mwr@}>%gsߋY;Vo5xYd{q?hsjo>hW%_?lcq< ee9Pc[ޫg~?Z'a]̏h奎ّj̎35|l>1t1 nq?jC(X桸hCr㠹Zk<z7wlg==Q~-do'a^"b~@xpqm*'p-`fP*]vMTJX}~99YI?܏oop[ŸeGrC {nc尻}luh>q{^mGޮ>w-Ulsu',OE>es9=@9$enx:)ְX6$s>\SB~"xm:RYDdYkoLJ ֛qf{1mo^)goY﷕g2fZ?BBvq:H\+q\ԏ׌Cae+!Xe@ڨQ$3?bg)o˺߮1{i1a1R桫EL1\?3Vߥߙ/-|ms|8f;C1 mRb;?L!qt3Wp+G3י+br[AFK۶˵XKjNX-|ʵ~=O\S۱y搾vOC}A!s|G.) (k!im#jPqq׌5:T,bL?"9zG"8(CpYA%x<٪nvͲ0bhX)p7}gv_";]HB >{ 5C?Lj^?s{?q+oR̒vWP[ <7ʆ;ΧsWG+5u-ҙ29Ruk5ZR߶T8Ggaޥݼ~[.|:2oGƶ*LQ=A?} ̱H߂!J|/5ǿE?GqyS^|[_$g|cq?"{_^-uc~mMhQSNe<,:q?d"E&hQC.a ZI:ձ| C=k[)^_${\ |~¶H".ǾDyFq_WG,I1Eε<]]c96eqeq[l"AZazȵeȟetsySo5Ң~It ~;s]QS9j2Y{F1;9"VkÑUKCwqy08fLk}M\(tNPЈ }8K%;{7|ඁaஂp 9g>p 9v(O׿\NξtXE-'Wp2mϣSl{={uc^*;33n_={dֹ1Bm^7\Ma߳_F97*sv͸M^'f[<?Uӏuӧ,e_}`zVvDIemuG]~P1k}>sߓEYWBBfƥ~oz[8Y=LOs~|/Ogy?dQO?ge{UZrbw{fydk!u[;M71Q9mzQz})^fn6OS@NFgyڌmǞ&{v۹PF#߯GSF|G>6*T/JKa5Yug8 (mKѶ݆}}Ӹx1*N`_@b֣+{-oLZ>泗g =.lC pKΗ 'Q?}![?6 f7bnj; ·rPq5GN02vAy8t syyvxG,=%({!T̏<HYĶ}5or#VKI-YY[SݸBuMWqme5?¯[^Fn6OSRy"s@mAw{qXջ }>~$|uݣqNog73^-1:öle?7?&3iL]}̊x^؟5ǥ3l\ؾ'i_=)[>\t6j?wVܥJY'{.Eö#طtCoO~dWjM b_]ri.~w! ghw難C~lYLNfȋJJZyJMQiǸo{OQyVZ/gpء(!!#}1P#3518Zc}Q9|9{Lg..Cys?Ka۔eϯ˃^ĭCy'-Mz~* =Řfvgm`{zpkv6~c}(}6"s1I%Ů1K뱌o9bQՌw_OۜT^Y'R_}>+y8i>TFn7*$.)Sϓ4aQzFq_ +/&A]W}"{!rGێ4h=D'ۂ8:G=ߎ{>.WW/Ʀ7omw?x8?ƒoy!z,Kc u\C>_ƣ\T*/?.6[]ߍ0\+\ܐs}sI:yX9Ѷrn<ʯ2Pm+z{<$rZuק;4!:_Q~;da_I0XU۾X5'bОc֦g[s|,} +{e/}&ukه{YdoDz\p]Dub"z!:ʥpu{-2;>x:V 䋸?{܈s6f:qwʋewܺAom5a:)kƏs۪vO[e;;-{߿owDAn+F0Sm72ܼ깐E& 20Ɔ@ߵCZENųirbg@0ޓr^9AqH ۱)Q,%̭K!. !n3gsOG6TvV`9ElkUvɳdL{~*bO+r{ϵٱ=6]\?W&[֦~qrՙ|G#4Iӧ^9Y?~1 >+M5G[+Q`7r5mۖo-̇cIUc/nw-8k_E6}l';^8W}X ̶ fuȝ)ތ,7ՃUqf{[V߶m7ʳ9o78Cqk8+~0wzkf8~`{'![l[U+<$?|Bh1)ߕ뿓=*%֚0Gar(X.+gvV`ken$7U1q?k怵 _u12bޛz"Cg9^+DЇOӟ5yu3jdOqt߅rܿel{k2FN^sa\/ξ+G/bVYX~\y>_}:oV\-lA9ShWp|kjޮ#ͺz엡7+#ogu "!붜Ղ=K=m=}aicǸ뷷hBYurGm09_CogYBG-!VH5{;a,!W:Z]l`֧1ʃӥNKdߛ"ּ͗Ǘ-HFe 1Ob>κ?^}X[){%_\)s"Ժ\==w+W/fb}l eShGkyuO^CʪqCۮ7r|ѱ3Yn_3H~[p½uy ݶC|o~MN=.D}/_W MDziǥ1YXEqwm惛x0P 9 !̟[LXoVcpD|مqy18*_?o_C<5.opQa (5p"LUœjW9b~5R{ǘ7Ks_ayfŦ< q^d=Wez]{Kyɣ^߭}g;}XQԝRq̳Sh w˒r y[<ӢqLwBl7Ԁ3"|sWYڧ4){͞ѵٟ<=]{z:k2>:VNC;{L{\형l|L}I>ЇFM}+krG'*ٌ*J7ic}kg>D :jn6ɄӼs*euЫ -FGC(2 '؎wa?}"uMrm-Wra16)'^uu ˵kzq~0!A1H Mao~Ǽ:x?܍~E߂XjCN8m{=[? kr1YV;p طyɱx qOx{Scr\Z7ּy͒Y.9 @Qar=!sUv:Y7KJ[1XSIѳ} k`>˵к=~p 9v(O׿\NξtX0/͋WOjmI38nB~^={dcZ"~ܟ>叿d)['4}N*7}mn/>r6]Y)$/R:47@ȧus|-*_zg`z뿦|_r_LO?'ӏ}2f?+ۑ_s;tG_IAkU3Xܿ?}?u=Okw`_ovMh ky4]rCLv ,r~~1֏Zw3aWPW~ ȵҽOLᗟu<{yvZ𦰩DU!zQZAtE(¬?Q5DaoBU73Ҹx1*N`_u-f= }K>(j>{y0u|ʡۆ%  "|N%chmF:;s@AAoսDq]`.E ̢Jϋ;_|{mzkgC~&.i#ͱ]?s8zXVY^0O3>o*}^PyvbQD[bt7ŝqaGpG$w~;m5 :"YWW׋>s|tϪ WWm /ȷ[x x//GUGylĂmBiCY'{.z8ePb՛NDY"Z10?A\tO5 3ήC/V2Wi&ʶ3$˷^W~ J)H54ka:rxx:k݉Gu|߷FP9>/YʗΆ&s6)Zoa>}/,X+kTak)R}\^'-q_DA\wmʷ2}hr5`\yVɅT̋朧ԾW>>hhS)_j*Y ٯkr7]ާNKs+WHDAo?}\sR 5)!> q+Fg=0ߏG5y6M}s/1kƸܪ >XrS wsx|mnɇv:t{CZC \O|X{v%4yh+r#~>%eɹp=k_]%溯om5cy}\ª#ƿ=ѷkSz Y3^?kזBoZ~}츏|9{*\t<~Ͼ~.N\z]맑{lw/y!URΤL@!w:yvZfs%ڇkJ5l[pVGY?,B{M,7;b-b+| otoi~mYIe\·;|zapQK]lCYC(kay/[l[U+0跈kbzo;;ZE:V捶[iYy7sR ^e%%!Ŭ6'\{r ou\^sU$J&:rv*][ܗN2rz띰(?a-/Zn_q0V,ko=ySjj]_?mլ$.E$ʦ5|zwJ>XkH̵ܟݿe> >k#P{%~ټ˹׮_ɳ7 !?+<3f9-ˁq{ۘi(kD1a˅˛%ڏ`~7klGe7kr9.QJw:ޓ7\o#`[Da_WLZ]{9/su\:92oʵh]˪m˘<{#y~TzyvRf\pp]B[o,6|:_v|~9_)|:vҧ5~u\'Zdyn hP {Ϯ|`:XPk:d-cv ? ⾯bqT܏~޾c|+H|7Q_/E\9B{+g͵ٸ~&V̪sڋC@XF̥k9C1VZ@Q uE:z3&n}Cc4/mrXn!qbkb^ފ{r$ƽ`K䰽~h>Vj¾'>\3"ms|hyhϮ!~9ZƝ_Sf쵹`vC?|XQDdp^GA:˟ϯ -FGC(2 '؎wa?}&RB]+0ϔyY/:D~q[= 0S)͟SRg&C_ݯ[PKͷ@~\@8mu%/y?(AvwǻsV.(~sH}B?!ǖs#lg*?͢3vb1c[vc]o1b9ge{ AoD),mqP!"It`Hw磘 s]\Ys yn=k3Bi7_<|kq _~9طk: }M~Twr8X?Qo5?k,+/G9s,kt>`[gIza`a#Gby}Gx;>0}Ä?đ7YF܇qL1,?>9p 9g>p 9v(O׿\NξtX0/o[n^LW׵p5]Xeo qwqIeWO6k<.1ƺ ΗwLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵ѾiZo@,+sI3z |Z7®~!O>߿_ן=~j_Lv&7JS@}[_wLܯ=$ŷǻ*'1fDU!zQZAtE(¬?Q5DaoBU73Ҹx1*N`_u-f= }K>(j>{y0u|ʡۆ%0]0x'"|N%chmF:{1ǞyrsqT%>ogDUJ,nIwDžٲ}m&.>fE<_]^/zVϚuӉOb?(']MmW.#Po,^={wˋP:>p.V$V:uid;{D偆,MyZ|>2n~e0_>:ՕvTVRky#⭫#!׶Y.7׭(SBŽXٱt{Ѱ}cWQul{.S1WT1sש)k;: (kAvsE5q_$Y6EB#2;i zWOܷ#}%q,uSӎ{9ȍP:',R׺6 |RT94ZK)ϳdߋXQ4ei,kD :-Q}͂Xok:iLx" kSE-?A+懲+Z` 9p V!̧Y' R}^6݌$.׼j},1}'d/ѳ+F6G7c(Wo}d扶D{?klœc6 :1G,{a΃}-"?(";"z"z!:59mG9ӱZ ¿G ݲ*|c< q? /BS)#)\aGšH!:d{Nd?Qo[fEplF·};?v8El+ֶFlu/HѼ6DNo k>fVAZ|w`ie>bmZ8֌}/+>\SB~f"x:RYDdYko|`oOm[cަ㿷SHo+J*:v>`㢏S^k`$&B)T([VcJdlGܒ6j~T跈kbzo;;Z7$[ M_F۬Sx):̌Eb[X.T~*[Ǭ}[џeSs^3]}k.kZ̺ˆ%Fya;IcҶ6k ~\ ˷ormsk ôSlCUgwyxXC5 qz$DK *Bi5CHDe+ak1,nR<g]IDAT\M{ӓaqF}SpGcNfJF۬Sx.eU4.E~:/Xo Zb.ovo7O{>`{\؏^\;>z? [oFa:{-Bh%i!•k3Q)T=ζ̱H߂b8U߾O9r>t_U䱱p-#<[11&Pz>z؇>1fSq?ȚLH17]VgD?hEfwaQ.'[o}=_Jֶ+~%fcr"/Y{\v]/ Y)K-\WӅA$*~E߂XjfQ(P-Ap|!7C7e<+ϕTG̥ʓgۖou2/4|n#OBp 9g>p^<^~=~yajzȺ{^MU9@B?zjɯ^{[]dSy^?;U=Ͳc}q6oӃw!az@[gO93ku>ZĹ=|Zߟ<|{#y>n;.N j.^=߱`/y0=~ȧ:OE/eo?YʾV@iMM~.y닏|ܯ lW>cF}jH̎(?˧fq~(ӕ%>_5bzY^ϟ=~Tm5YَdBsճGva ݝv߲p.PBcWD'fs~O;RS&v'E!9=`l3 P) g)7b ڒy֮mnU7'VI=M}' ؋R0,Gv^]YhlMxmW~ߛFyF/6wU8V.>g'3VL}ţ?4~y$ kAO~‰?B\Lm(Qg' :>: ŚP׊"-qOc~=пu9mgH x8ۦbqU/(ec"T[FI+|;/s'Bv ;Kƒ8?D>R|d\`>"e.wx9zXrY)##WW_Ƶd[`jҿwɍKgߡ5{W;QdYEYk͌<#W6O˝p 'گ|=Ij볅_'6EO^N pFqxEH;9UCߦ[ܣ Gێ4}=D'` N@X9bU2޹CRwHEF|0gqg}pvf> -C;1:}: xT_Ahoxy-q_DA\wmʷ2}hrLy\ :mVEj⾹n)kA뒲J륍?$.=l>_(rӎ/}'!/]WZ6bPl7:bYbrI˴1cP·'B$J8;ZLNy axg}([לKɾj+[T^an̷y8gp|dt.DlSI#.iٸB,:GKռ'd}Wx}D/n|^bй;UǏ)Wrko>"K*|k0`&b={C°|k{Xl&vh*Y`o,}Q1./w_|]ެ vkS<<$ 2EA]\^/dcf\ ġuk>̓Aa&&d{NDouXlg03=Q~}/-q_(mွЇ:ET}rп#Ey7&[v=j`M3fVOu-n%yeLAJZ4 ~2OP5_re[kێG0ǻPDƸ7..`Υ9q۶`$ ȶ+6_#^?_o\ Mv(KuX)c8f>fXg\[=& nWړu4vLrjߗ\'6_SEfCn昳8q{?sgw>6øq6ZͲ-Y}9e+l.vDp폹k߶㬎~d}ShP]=~Xm+p-_:}l}ù85ֵ R<Z-[ QfV>E9$6j~T"]:g[kuW]{\cb 1۾}mOul vj\A쵅`\nfyB)}!p*Fm/S7(Ǻ@<[ZʃHyf,[ s,XCR?lۜ g%LЙ7h&R ߏ?::شމc렼-Xe& kKNRu!ƙ)֕"cL 0]k |/ٟ艇tk^[|ޯ.zmc؇J}"c pl,W}[[s} ߬otqa&vƵG?њ@ݧq'a!Yy97Άu !mpY9L#5⾯bqT܏~޾c|y y , /s[g~/3B?:6͘WFzL ˰ĵ;$]/az[cTo5]_bcc*~Buh[ tE*E0)ƽqXqZ}zf~l; _+^q|XŹ7N.tPGov.Xv|X%/ѷ5)9H m(Ƿ1l1]{r-W[V@uNt6 (Ƭbb}ja|9ŦR_?wBA}oY#zY{*6׼uÃ]V|D?hEfwaQ.'[oS讉zmW?JL+W2<|\ZjNycugCLUsjw5]E}kp&aiѯ[PKͷ|(j l@8BoS1q~mwwq|.p9IP_E%=܅%"nbQ69ʸew yp2[j}"´ܷwK@v\}KhPq(|8_U5'-}1e?,cWӃ,7D{Yҳ]Ӱ:#b7U Ĭ qz]ͲDe9ߥO2N\/ո\-}涍:"fZs}(pBflt]ZdS}!ewu*jbtnwˬRܘDϕkOm65d+[zm6z|{ΕHww K7̅=62Hl>%{}82~6qS։7XF>p 9g>p /W_O_^<^r9=;WcU3w.?)_O=27ש{w*7{wdX)wjjl=fKOx;e<.>7;8}c](Ž'KW>ު(mw=wRٽose>o}oTuhߴO\竰K{Z*(qv5}O|~߼Ͽ5+}Rk?K7?2kH}>^‹oreݛgo rMo|kO7]Ύ7S H}}\O,Zbq?>6*T/Ja5Yu$1- goBVuo/6w:Xe Kݞ^PŬGWZ/XΝz\곏٪"n !Х./"< 5o/^j7BD9> ۮm?O /8ؽ?;=w}od<pl~Fx:[]qqGp_ v6kltj1+޶׮/V|̇e zd._29م?^?k;UGy$P;iUzP@Iޯe,Oc6\${>huz\qzC~/s&yzq6 k>_yS .||KAͳ J*`CYrX D_/f=@Õoؔy=H}kGSs3;.NgUmWvy0̢>b,}|r͓}VUeΗ%_T\Ը֬#13NuS91Ai`wQsx[U)߶"ǿ p {hۑǙWG/""Wl;/@s6! Cx)$Ctlj@L C;Q ePI,B2,Vyy7R_w""UT>ߜx}W=Dp$Ai|NB̏y-sy`o9\Зq~/fXSG/fy@aY~D_|mǡ0ŗ@myrWkm{q>zv%2?}:fGރ1;:9.5xr ;KE_!"?n] N] DqoqOEnON,^717d]o'{m\RyEjC(X桸hCr㠹Zk<z7wlg==Q~-do'a^"b~@xpqm*'p-`fP*]vMTJX}~99YI?܏oop[ŸeGrC {nc尻}luh>q{^mGޮ>w-Ulsu',OE>es9=@9$enx:)ְX6${Va70[8Ց"w"[{3> 3֛視_=H)`bo>9+QѺ6#^ƹ 5.v@4b</kf>LQ=A?} ̱H߂!J|/5ǿE?GqyS^|[_$g|cq?"{_^-uc~mMhQduu ˵kB XE jpZ4h'@(TR- mx~Er-'[S "N}9_E'\9ײ?~w)vO#;wgڲƕm]̲!k{NW η#'?"yVMuHq$1 ua;G/N=ѱ]m4ʘ9y^=[2g=OnXGW.=ލYd1a76?PpD60 xk7/4"mqΒ=>qz ofo;o _}m & _;ƟjG 9g>p 9g>yrz]>ӫ/#뺳y5]>V{Q\L{D[ht*۞G^oWjΌ=LWungMg!Fg{v؟1u]d{✏rw8;R6ߔ7$;;8}c](Ž'KW>ު(mw=wRٽose>o}oTuhߴO\竰K{ d=__5bzY^ϟ=~Tm5Yٞaz>0\KNl @9af?q;?LBڻ?L]?18nBɃכjZG7;O_D~9޸'D~QN?\yF6K5.y&_/qYD͕~Uge+R޽=fvmV/;}m#7\9z|# q!z^T>K1BnkIvx/xTwRx+ ΂~gQ՟E(DWv !m]/}ogq=bsUc¯8_X;;-qur`.ޤPwk "|N%chmF:{q9;4·!""u@W8BO&ln^@)݁a5w2ןgKWsdb㞒^ay,yoFmTndq wɳsB\\_M> ~YO+}?ݻw{#;\B!*Dpgز͐iqV^3ST믖 G!0凴-a)lV]ĴgC̷ُθl>}h,Dz4PǵO?e<(շÀ%0(菋M_~ws:农gK`(A?s:3ƆyUڝFw0c]w+z{\y8Zs=Kn|?q.yֹ99_c"< @x/Ee?cgf:{t5 #V޺߄-r\=fOumvlͰu~s8ݗϥ_Waב T6Wu˔w>?97#"2_S2ɐ{]J_ŶK^c\6vf}]1Yzn=kl5|tŬC-VҗC[Yķ]=}+{pl6Nh汣E_!"?n]N] DA=mG91C" Ѐ,ۤX'qZ^W!s>~v8~t?ċB<~5A-vDZO[e;vr߳K[" V+؍xbne"wmcNBWW/w:.M~Y.z_+݊Y^9 H8+|"k;^ޞgQ,%̭˦$Ik~+ q9h,ߋ흿?q.y玉 _{X{L+9aMYbk,i=P"9̏b|Ș&ToŞRVvkc{|m|/N}="{c>c@S{ nYLA>ϛ`]{tudgsOݘmO%%}ӊmoI}A1q 9\CY4i˷kUc۱=sK:^;VxO겒߿6pIB5%šoa-6#E$Mf|g7;~ mcPMDp9+QѺ6#ǵ~ke]::kdEΧ2&?|6w{~{{J@wpxx`X s"\Bh뻲vlgu Ccq{\b~kO,9*c~e ļ7%:ot؟絴=y6$Cc~*gr{ϵٱ=65qLAʷiVktN:ߕ@^dyٵ1qnJz*Ve I]!P]_вhr FrwOB9>a>ܸvZޮ"2\+w=>]l}7 "!ƕшxqQV?pVuk9{E[ʃв9}q?=+ym嵵ea -@ۉC([K1gwݖs[?0x (a\3+ȉ+^ޞ e=ZU'╛-K m#}"<X0םEwrp_G#N퓨>mn/Tm=gv|~Uh^Ẽn!?׷p?O'B_Kݸv|~9`?|D'ߧǥ2\(kG~~>n6"}S?55a[3 :/G>sZ=̓N'|@k !OՇ0%}#Y=Us[9P R{ǘ?sۤ w(~I"ʎyfŦ< q^d=Wez]{Ky9{#n5s\g<f—%G򪷮yC\_ >=aBݗm;楱F-&ڽ&ak;Er߿dSq?ȚLH!<< euЫ ʁ<`"wm/Lg[o}=_JD׶+⾶m(ZC[g\\a!~#PSkqH$͟SÏcwZ?ӡ:[`Z9\Mk { A^O}GI(9|eeӡC\x}{+ˎ'ܸo80٧_GbJ?v\}&ejLWϖsԘxJ1Bo>Dفv#~F̊c?i>,-/ۛ:{e{܈wmu{.]6~s`S8;1pyo?Z{ȸtϙ碅?H,7o*6ٍ[@mϽ{ ̷-i.hgIzaZ9C;!j97͕sLqNplȳM3ؔ-ssSv:nOp9ʾ3?NM:$F~7b<Zyc S6&=-~o_3g>p 9g>p^<^~=~yajzȺ{^MU9 sܼ9}*ۈ1؍J!gճGf8wLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵ѾiZk_Wa5!rƵ\ʮ5wk9?>'w?{2'jI%8Cw\5_wxF;oӃwu!azذ}|h~\MVYϧ9Xaֲ~VtnᓟEźTl]нOy=sN<;oxSTw©Ix[ Yu. AVB{ۺ^(HzCl;{ 굘(^ѹӢUq4AO~‰?B\Lm(Qg'|H1(7sϒ(^J'ׅ-i / J_ok -=Es k6aZcNs[A`]oc_:>r#>ՄyWa'?Wľ b73›ՙxDԖ_|S p~;m5 :5 ~uEo[Ώk}lzz+kCӲ]P+Xg^4o݅A^^Lb{,h٨ ۄ|6N*~].cyd}5 QGuBKCdnWG~a )_s')V.ٶr!9np(sjjlrO^K9]⁖d ێپ1~Al;\羼f+_Z8Vܶε cvT˳,:yvZfq Mq=(0'Û*oSumGgbL/ K_9ίC pv7"*A&  }i&! 8>N-"Imهwx`qqӁ>,V}"EoC%0(菋M_~B]#r3*TytП7ʷ  mj"łKQ0w9{~McaՉs|sʳ9h;_ rb!꯲B'@EPD1Qh`>5 9j9}m1^c"֦quU|r1{= px3]+WzP@R_8cSt#ѳ+19Uur6%Ƨk$f䰞W傹k(!}ye,v+YdoDzkqxM=cD{}qxێ8~scD=YfIN,^717d]o'{k\ Zġ;0yh)ڄka}v8_ԏ״p<= vT'-~ ߜlg^}o_/mqߋ-|K# s[ax8`/|<\sOV+65\^O:O~`qDrmhzB&G1k5õUc~1Ӌ#R̵ "TG2s$0.,oin^! ELzӟOlIsr{9|WCﻹk[X^>/kOڨ’op <(c,+,:r6ko\z~k>ќ=|UN:_Exg?qk_b6 )z3=˝<;oxIB5%šoa-8#EYvko'Ayzۻ#lR >Ǿ>9+QѺ6Sױ?_^>\aR׺&ۄkP qXj([VcJF iG,-ڥ#~śΎVqENyuZ0Be͜(s):v;ȱxYffdwA*=z1͉1מCkśf<"e1b fU蒋ֵ.2HOZMP CEo>b'r_XK{vT9Z1fê#e'oJ%W#ZLkRNGt!/*[ك9JT 8||;% 5$Z߲os~5G(=ZselgwEw,-<;oxA"!svbG~ƳzkhmD{il)*}u^.ukM&G b?Uuk9{E[Lj!-{ӓaq}{| fN9g.B9lYqgX'`Ƭ6'ݵ%@>|9P1>.}o3MFp[ߦ4Zc :ӛ]5P#[抵{W;VSS G;7ל%{TQznǡuݛ]~9]쟫kҡ̑ySW>FZ^V]o[ȩ=`I}~Uh^Ẽn!?׷p?O'"_Kݸv|~9`?|o>ia)[:yN@`Łw8XPC֢o9ȡlG/0 :/G>vǻ"*8|tʷ3RdE n.!r\obŬ:8{HYel\9ouoCPY侭kFlG˸iauq?oU1:6/mrXn!qbkb^ފ{r$ƽ`K4U(ǹq̶jk/`>|P;{GǕl5峫sȡyNlqz0{m.1;.Cn.kjm>K*y67~5 בBxx0@,=((*BЃ ]ڞm?_|)[ۮoڶ!7#rkoq2'!sP2f6sjw5]C\}Pltu} c+>m=? |c#xw}Tbʅ܏rRhg"Rrn$_BPYw6N}9rlna}5Ҷ#-R|>bl$0H=/sV2S>PI{L\bRXӾ|sa #kn;gmF(>ro?2/vmb_Gݓ#{nWu{%;ڢh[\o(b.e \.gչ.prH\[c1;.|[z-Y93Go}M\oR`8/xg귛:.!CA=r փ3 `k,Io#>lDzZ,o_og}^`wy8R&ۈp )ƟG8}3qA8s}3qx|9zzz.u.rm͋֔ ѳW1<.ifc6G/X΃?N_E>X׹?}*q/+RշjeJ>~OhTvovY[_|~m`o67S>Wk*5D޸6ȥ=U"_5bzY^ϟ=~Tm5YdC { op뎉ԱxW^3fU 6*T/Ja5YXu. AXB{ۺ^(HzPʶ=h]Y¯8_X;;-Ը|_2Fa L- Dn*,hQ/^ !M.\#/s9W 5`;A/]]iG9l1)Q\;Ҿ)޺zy<rm똥rs݊1%T܋uiϟKG7 ڧ;v5Ua25E3^r6^pg':WE з)-ԧ ɾ}ML{+uaЮ}1.I?݉6Da EQs"x-uk'Eu)XHLBQ6/C'>ƜWuܕ̷Y\υ˪xiҳ]yy5틼-c&m\^]+-ޞ3`{X =ʤ!D`{\؏%n\;>z?7܇o~8@mʺڷB],1H &\YlM!ԄBqHdElI ΋Qq?y5,Ϣ(s݌3'J2[9E ~q|>PFU-avcV11/Ʃ}ex1%oi2GP6!l?ѳm>ш0QDdp^G .^}}QPT ݅ſl{a=}^&R]mCoG D9*2:rfq/p%_MN`W&)u} cE%S`[HB=? |c#ޔ>WR1*OTl[Xx gX8:Efl< Q}5]Vl:_֬2kr>Ƽخ/k_G;c`љ]e[c \')5 9AL~x6qΒv=>o[ovqqA[Ogɷ%>p 9g>p /W_O_^<^r9=;WcU"jP叞ZVicja%iަ|zNbm;=sX_?}]|w~t(fz{g?PǙS7 Zqt,'?ϱ99vێ˧Sc~wl؋wLq*ǺSQ{YO}0U+;P{GӇ爐{Ӈ:}#.kUx똵Ѿ>qߞE0iү,^'6V91]Y#|0=_ӿ?/{ٓG߿>PH&4^={dmaw- -}?a}UOtb=(D#Ů=}ookw\؟SL!6y.~zU`^~~!&vR k;!öڱ=ޗ-=%/kn6X,b/J8ɳEY wg5 5շ]!AXnQ9noıBᗷJO ̵~B>9j5|9jL Ż@ᓨp-q_Em3 ?EىB|;]y{.||B?yKs"ލ]wyhnP\bӉ\·SBY??Zq>~`ѶfF&~r~c4N7[ܮ |qqGp5NfbC >fE<_]3(svϺtjy丶'W._7{˅"_^o[udG\:IX)Ja6:uid;{>$uz\'wB?7lw6oyqxsm6}\)o|?9&羴Y68 Kkh!5b,m7 X=&œx3ZeC`%MH!9\Kc3&d|&XsP׊"-qOc~yp̶֜3e{EsmSW1[bsĸ*Io. A(?)g/s'Bv ;K\_HWsaXr!lae<\]iײm_ӂIվ'7.Q}rl]Gf}w}fn73⻊[f_t>/w~+kF$~ښ?z>=ķ̖lu;=\gىuo2/ Yt]ߦȮamG|Ǚy`U-^ߞkϚd:PK8ܳ>C}8;3ߖCy!xXB>Je< 4շO<_(rӎ/}'!/]gaNKRc}J1],1~Gq$ǿmev僘1[(q] qg!q%-m'<ȅVB3>ykΥd_֏~5ƕϭYv*q0ng7ۼE e\X8j>z:~V)4sl\!r|e#{jޓ}>|}_<>"ޗT~7X>/1܋E\WЉǪ+\7m~}>W5uXOސlkam{C?uxp+QQg׿jbn Ihp j:Qm;)բu=C!V9'6:7m;8}+8昝*8Tu̓ R^§ypY9(݄lɃRџ+@fg7ʯE|%SW<XǷț!oW?\n}h?}ˮgt\ uѱi̵Q*) !yf+.??`ԋ6B 3S,+mjѴ/Y;` Y{kc(I\-m#](c"cl0˸m[0~dŽׯ/SHcZ˕7ri.e6enjk$ۍqJ{ZƎIS][5}}l~s0Z{}֠c}.sf ]0Av5q5;= ֱZm$0~d}Sh¯f kb{MN+V[1cfПA_?|3>f+Wù85ֵ R<Z-[ QfV>E9$6j~T"]:g[kuW1:s~v,סR1 Ywv#n^]T֯{oR'Z̸/I_pc'8YٽRu?M_ 뻷gTE VA릾]wK:m=lYu p[Ab!³M ()Emv5SlCxoW7k{L`b?0B?q k-ukA<@Wuk9{E[!~mN_OOs?3y&̛yhs[)}GWktlZcık!'5!p'B3S+c]k /FCk|62ƝqXJ\V}^ڂ3 v,?9?` Iٵf;B\j_k,qlì2ƹy=\[)-r;S̳Y>{W>ع'5٫c|XٗliUGi}_^;\o6a]%jmZmůIxuT>:.sgcq*{pp]BfQO} Dž-j{#|U~us)1s"Yy97Άu !mpY9L#5⾯bqT܏~޾c|y y , /s[g~/3B?:6͘WFzn;.ū bKZ,4(ȗ kHLT"ضL :"cТI޸T,}8>=K]3?dxM/.Xg-νqur3=-8|s#,}KeϹ-5GoWlD9?9g]qjݓo9>ؾ{t{1D1f3MHPN~/ߗi-6 |+z4 A=C۾6]M&u :q_ -F fvNlKmyz-3ٷEH9ќkgK۷<K>wh١7N waD<,-u} cEPA8mhGܳ3_jP>&;v88Ne94'W1pdcYm_,†_Wy3G>!8BXD9nWE".["*TeO?fL?e,yzh?KzxyWru~DP]즲J5siۚz]ͲDe9ߥO2N\/ո\-}涍:"fZs}(pBflt]ZdS}!ewu*jbtnwˬRܘDϕ>\?[pl|,n#qA8s}3qA8sjo(x>IENDB`llama13b_autotrans_1to16_train_modelarts_params.png000066400000000000000000000441501476650141400400020ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRn`\sRGBgAMA a pHYsodGIDATx^k }p>A1^`q86*82 V]$7 b%-1q竦1b D12 NpƎqc8nR[^kvgfgvgvV~ ]kf?{gOD H"qR$nR @JH)7)E Tm"_-'dd2ZZ{봂:D7?cb {gXV=ը>"q~PoD^ՒvYSM;~s,WCc+8Hi{:x`b*z,;yr{5b9g5G;$7{uy{!H@A@6m|]JK,f>+j c^/m~aSL6W;YJAl ?[ ׆QljB&&q 7lY5zZo;%ۨR.`N\-;~+Mda[2XjAײ6S;YJAl?[ֆa;+NTQ qέ~: /D:DvCuzƳ^ZFx؎xuY7N$;JDmN(TZSC}?2W.IT']_m. ZN-c֧.9,nz*թ?3mue1O*_} -Cmne-TDO^yzG Ԍ[2^hyIkGOIJlceb1n8 ՆD=WOM[wX~IA w575o~/ЗlC5!rTGܺ,g:u).b_/?b}INt:F.:r^+Yx}$6q#4Hv.Qj?xuשD?ֳdq㖴suĿz(pU3ϨX:"K91}6}"ݾ_r(v7nJcC_//j$?^S~󬸽h6$\HnҫK`?ڦ/:HBJN_/΂3c5?3QS.[:mԩI%Ѳ$7}[ t< ?r{Դ]Bsiϔu}Kq".Q˲1ZlvFܲWYx▕`WZϒm`DMS\q1U\?5nۢ묒o5xVQgx嚲;.{kQDK:'QʌWD"ڳcUπGc5ڰru>:DkQe/Ÿ5nr $@` M"B'YFԴլKe$Dt^BG7:zj.qS;]jrM\D.vDNZZmH-GE}fڻbߊ]VNQWn=5M#73$mϯVzvzGM;#}D]E7Fx@9-mP5Ӆ7mŸ[g\;vq"q }G}AoD>j~,ïuIql'iܲ{D%\@F$y˨vtĵKcYn[ve盤#7qH[j%]?Ung5lc#vEs@QqyaK}l,g['5=Sq_NK&x#>3:,3.U:>~*I2HȰ_ jgOO{W 02%Ѳm%պ-+7_Ko̱ZbD{?vֿʮcm.*bۡB[߷ˉwr.=+'czF,kY~t^om^g#jI]Ϩ*oϝWj#l:&َ+YpeSlY:d{-D۫rڰ y{Bu+/|Fbf^`}Wu#q l=Q_*כ~? _yo𿿜uIi5~x/o8g;UQ$گ:fpD['SY:&6QxkԖ^۹NeT TuZJ@Y/m_ jEزZGշ6R ?i{$>Q}ϯw^rxbruW[q/)NxpmXLQe Ja_j_:ɐ]Q_j=K-.Ӱmi;Q F~jUi法v֤ `H SuGyygοTFX:ZZh9{(yLl}l|>XgAެϝG[D{svLl@ qLƒ>9{(1QOg6 v9&b|=}L"qL_z`5ߘhOуZ:.0(74D Kyq407: {:$n)t("@uH S(7PDꐸAQP$nhub9l ;Ӽ 2@V'ܗS]cCM;4*7R("@u'nο7O+9 q{>9qy;)="mve6ׇȈjtpcZ:+|X>7V_P:?@|^rdCXyL,*SͲ7rbBpF7S X[u=xonމ=:)"ul%n.qIC汜蓩/mQwD |,q3a'e-TRmvab4%JfVz{]P?ҁ.ic}9kk;M'/="C;x) rfHIzr~HQ$yS&Ctڪvdn-XWŨ>%/%[lTNʖCmiv4$Lkݺ(sWB%n\=wQn/9/[V{$wFg1QP27(24qI'@ڳj%c ^j$5VmGaq&/-$:,nD)6sbArzysl|fB e, mwI_MB;Nw'S7Cwltu`XWM1oS4GwırSN6vMkrl-NUwظuXfgX%<ejwwy[r,ԭn|{6d$ՙa33=eɌKGdC/Y&FN~|X:M}*&Fzg;'ҿ5/w5}nR:;tum(~t w{ֻmK-6YoJ/f&vRf5k[;a1mҧ~eU?'g}cf0i5l_a71lOg|kA^2m5~ڇ#mG1nߴYw̨nǴ_ZBN;>I2&^.}ZZ ghT+T*sO̩0<֪(|o:)O+OY4.2"өt:N73&AÙc4Odn z7čhc{l(F҃5zVzOK㜗ppcިp䅉.i2S9Vze"Q>%׏ef0Mz_w[<7!-{Єw{K奭Z>,4tQMy.`cZrg8'̰s(;o{t?=!{zM:uMkɼ=BiCNal.mɤvwE^)\;,w^qLguX)Sys%=eȫYV﬐4pQzBb*o=%^E/mʗ?tIkr(G+؞zky,o.&Ntzu(W?B?d&k;ɜ JNqRLܸެp?M{|܏h,7% 6!0ݳq&rgk£9L'w&Qe75|B@ëAJ-U_/s@U<Ǽ{_z-}_6ޝ=2-R. %(>_Z)&bm},-yIyiNlÌ;lo^ &=b=BM46$ExլmuVⲪ?⾓KQ~lO}cpE3J=}s2vN/Q.ď^12-G4-; ghTϸ &T< *dŁURMGyӎKLT'02AXWMq~vֽdǼ{2L^bZ<^>v{:x,/ݕ+S/17pvn Q+oPKuIKܳ۞}}'ėK;DW=_'J7zUwamM##ѳX.1m>Y0mV }oߎ:|RP[`;_Y>\aZv֡l|nmTMc0x)g9-@I2g$n40jm7sKxwbξpof4 (tڞJ񤴬=|a BPm)iTJb}[*yŃcedYaTN\2ߟw,(Q .^9*&=igėCg3n}t |rھ$D*7kr;kCf5kߛeUʻKXcChq SxT݇+[:o<\θ/3ϸ nmTM7XßgnX^N~.4L76R_r i4m,)^ѻ́iŇ;1I74rAR0ӑ֍Pu҃waw^m.5lsM 1רWM͜;Osu%$^yכz[Yw z0Mr#޺{D$<>})o|ylYfux%S?:o{)wo޹ i/ˉ*lE6YVb\&Sm;HnBsƬ z]JJֲxpƴ~_=n}3nud]$n^c^ଋ*celyTyb }┾a $<^#v-:rlfR\=xէ&}\8EyiG>#GzOޭ/*^:go S~O6*U{yb&L#cc'qӰ|cw_>+2Fu}s^>:c >*bլmLܔ?5IխO2yYw}Lxº/4Hf=M3mM1xrwf{N~<иܜ&\n[9$C!zTp==`=$nAZtY^j}g\gfDC}xyj?,گĨ9&pH*nN 4&n;O>i * al4X' ;Ciݫ6ߗOQ&[ID9""g=%+=A6q킌lsOޜiP8@[%AMQ9ɑ{*:ⴼ˞1sR*h<$n)7PDꐸAQP$n!qLH@CBG@="bT 2>%X}s41ZKsL"qL0000Š1GcOzHsL"qR$nR @JH)7)E H"qR$nR 2[w>~aM9{҈Cc"dv<Ζ`iؓFDk-1Ҋ 2ECLPd @=P7: {:$n)t("@uH S(7P:{~PϿiFA z qE{?Ȯ⡏d ) {:$n)tVƴtvMu[ (Zy쉾,*S d, epRZB|M9s2-7'rҴth?pَɬ<&ޗo&L&}_+2ӌ.АH Si^6,qsv\FOf 2m=$n 44ȴZe *$b!}k-@c!qLpYHu楥{Rط ]&7I.%;,^9.l\32;Y[&MrFt(ܗꦉ?y{4/cr[O@Lw,:'[7]uMGy.3n\밓 fP|O09ٸIC(k'nɈM8KI ðaG 5 Gd{>9q˖GЈyw3ΈܖATQ7Gvϔd27|gܗgqƍoyV6qc-^Kv hT$n)7PDꐸAd07'Cv'G*@mǞˢ\>ѐAd.q3KIM6C@N6.1 r䕩wٹO^6k9zxEv~9Sd 7@=Mҹa\lU˘xРlTD?doPvM_eh,$n)ue(pp>`AƻI.%dʞoI&f20+WףF̰limM]2]oe̿s^}r02Ksݪu&-))˵uXGu7@čeİ7wK-n7f1lp҂;(C'oA*qsվT l#n Odno^_H:|qT:GZx"?8yߗ98)j%?,;eY4z og02ώʖ _4/ݔ޼}E\밴Jq]%nC垍Er7mMHKqP ȖZ&nn'{y[t s?A*qԨ4)/yb.CA z qEd @=P7: {:$n)t("@uH Syy;[է1GcO4DH+7Ȕnq: KZ s41a4DH+7)E H"qR$nR @JH)7)E H SuGyygdИ' k=9&@ZAhgl V=iDL H SXki=D@ꐸAQP$n!qLH@CBG@="bT 2z qE{ȱe{ophl$n)7P:羜 /Hn1QApH@Ci욖H@cOeQI.hH S2q% mG{9ͬ6/l@<&ޗo&L&}_+2ӌ.АH SH[^Fs-ȴZe *$b!}k-@c!qL0/CLJsSNz,׏Ib@M#;}}&[1|2-[󲱹Kgs>,^9.lyar} l5廤ݼK\-ۤsbAc4礥ę`"s#9zkA^j3r܄t;hlvcˍwsLwLqe˳rWGd`KA:y۾ 267+W*%nrSz"s{qKۤE,-ͦ|࢛\dKgd`SLd͢yoK}o -Oog{fyM؇҃6@&a|Uzƍ&`r]2sC㞱D̷<*'5NE$n6vMʜ-'6dvqR'#rޖ#:%C܏h,$n)'nAR&hҜ 򣷆_g˘tl_7|{Z ` jѳ]ҧiƟrD71Wd{`Q.MˎA AAT'V8q+G;)9%nۂOkzN0_xfsi>E5 Gd{>9q˖GЈyw3ΈܖA4j̞7eY9rw=_f~>XAJ/*&`diSOl R5MˠIr ˩h,$n)+M8,u(˸ސs 77A9GѩQi:)ܗ r{|<)oȢ_SӃ:qOV03/co#i} R5w/*M870t9dJ7tMzۤiS^wO˅k1KKK6m?UOI˦6ii'קJo'Z{dke}$nL:;aI^r97K'`xYקn7J6&.de#+R 2%OַZmLܣ)FR{މ0>@&9""g=%whH S8HP$nֱ]z9TcϽKApH@CBG@="bT 2z qEd @=P7Ȕw@<Ζ`iؓFDk-1Ҋ 2[w΂=iDLd``X!1Ҋ @JH)7)E H"qR$nR @JH)7Ȕnݑw@Y54hI#b"ZiV$n)Yx;[է1GcO4DH+7ZZc1@={:$n)t("@uH S(7PDꐸAQP$n!qLH@Ğ+rAvv hT$n)7PDꐸAQI~B͍i욖ؐX'-Ix:+Pyk9*>>,#n0V{/J:pFCuԸ~ g'nA3ǧeȼ-$ɤG.eN:|^6!q*Y c99R&}3+=r [NҁZGIi[1c}9kk;M'/="C;x) 2% $5H HTԨl9tٖ\*Ηʅs-&_j q*Z Mʭ, s-VtYbTN~oqv\MƨeL<hPpB"27(247Ȕ: n'񤞎)'o 2޽Mry3lK8f]6:DuIKn5uk^vOjn-e7w|?zmejwYfì۝y)/waTg-m^w~e'YsˬY3m}.A݇܉eL2="+\ /h,$n)'nLGz߬\_c96},]cOrN-ď L'< I<& Mȼ.e&tJ_ '蝒3uq~vN6>;rQ;uߟM=Lyo^_yTZ'nm@ _AsrUU/'6϶M,~2ħ9/'/}ƹGJM-ƿB\zpM|qeĊyz g]2ph(yְ4i-wAy!a M"~͙Ή{_-`ŵJq%?,;e7IrOW/j6M; L@k~D70q`WHUQO`U1ߒzw1-of:x[:θqѼg5۷-6W7CvcQ~鲼UˇNtQgK<>}>yj 工ee.;˦]n^OGݿ 2qs>{Dr[dwPѾ 2č/g{d莎|p-!& Ow98;P< ]oٿ`Ka|6^$Ý=[΍9rvԞuމ<*-76+]G jyRKԫ$68gd7p04=Hl3ę-X4V^/j:dB\󉌉Ҝ 51PX(u /-d=۬tLܔlK3iEWHFe|~|z-> oD }0mmߎ.m/o!Z. .'ď^2WIeGLu H SO:BNG=uK\NXt1.* ez^iZOa^e;^zT>3ku/Ip:iT`OEV~t,X'6|K‘ai.I8a~_l rM;/W_iƤB= bWH&|gxL9u .C/.22*ͪoze`d[,h[#\0tkmkMG9ƣ =bˠ~qNΔPN=uK{}Onc: {'f6t"r/`}AJ UUzs_ltq;8j>uV߇{qq+-ۘj:)4:_*C> :\&'Jh)CmY GٹSy9ձ<.ݛڤ$o"qcOd9OLs$:OMr*e;r选3f͒r 1OXlQ]㯛p>/sHz]ӎf]IK7m]/=brrmV|$jb"mnJ6J#qƈ> OIgH-˴ޟ/l1Cɔٯ6}\8;ƾˢj}\R 2}9A~p^b$n)i=xM}glC4CHmgZ &ڪQegG}-'F+?2R 2e}$n.  ŝS-Y:D#qC/qV@+>Az+ɤ%qc3;GdX0";"47ȔhHXe9ժuiyq^=sR*h<$n)7PDꐸAQP$n!qLH@CBG@="bT 2>%X}s41ZKsL"qL0000Š1GcOzHsL"qR$nR @JH)7)E H"qR$nR @JH)7)E H"qR$n=o e_ H ~_>%?)yv 2t),}-GY>/O2pmO'r-;8s[J¿YFp0spj@~RDs8yxj@~?%ok_)&UܤMiRE&"q !r=uK(dΥ.?yߖ\7vHz/ْM-p)߷ :[OBEsׄMptDsPHH c.~J~{`Rĺ[y?)Jθq(&k$l|ghbK(ϸq9θ2ɘu H c4+yf?w6*>2č̖PMY:\6 2]orsK} w߽>[{ofR*_0md&t)V! _*qtv\gdCysqc J=8Tuc$h$n9Ĩ\.zxMy/E_ C<Nܔ}HC76)猛čϙ {o>} 2Hms\WEmuIh2]&EJBRF/*MS1q%}g܄O3{i @#!ql:go:ßH~&׾3r{oZ\錛1Va|LK!)o>}!w+dڿx_WOS!gyp╿Ǎ&PJ*ghb~xpCD H"qR$nR @JH)7)E HTasRGBgAMA a pHYsodIDATx^eUyƟ1?{mvov[%1s Jt!dp6"PR9JH(DA(*^,)@yq'q;k~z\sZcϧosӟGsV_fc3010301030103010301030103x7gyyG￿B!Bh%o&&&϶ 0-IGh^_+B!B(y3y4y5y6y$<͓O#B!<3zMY>DˈBw$3;#wkofxB~L6n6F^q7Wڼ[Qskg[F|Wun~! 6ۮsչ|^mu.XnUmj#mW^2zmchiFSIsNeݢvzu^UѢc{hS~2b B=>;wc{.~U'kÚy{ՍǍPs3S?^ys_Σ WY?ռiεKα;mqsc5w__[.N'GNçΟ18M1-NJ՟b_Øxchι<19PufC2Սqhmci]oI:uOsTs>#O0C7dl63oɔGcnd/2Ihu^qBͮh Y̋LGs^yQ] c̼6ڤj{mxaFWS{W9mxY[his7ޕT͹7jc\XmPl\ώqVǪr}Ph"oH޹/XޘշL^UO:VIh<&{]q:&6\:Su}sq8e5ξ|!ueqzDf>m9')~2{׫mFe!VM:Cxycf^DCG7˴e3k 6zWoUL62&̹⨾7Q?6UeV|+b{X@wl:1WէJcxc8JUH]-ΏbKէ)^s޹~g~*6]fu|^rrync1ng1c粙u9OHY'b!V~|[g>p=bz}w43/{ձͷȋb7KqTqw\_[4EӦ8FevL6'a]Isc7+9(U/^\F^eU_cq;y˲r^n6X8n'Σǒvj1)׻,h'΍9oGyYe&91cf>>BosfhW19f>>!z}A@!~ˬ߱3lU};~f7忙h=Kcp]ڽgF^<  fUqwtj 6ޔU^m\m&6qS]mIQ1c4MimD36z;$Uh3hmGByos饗6۶m骫#343S?qڤ8tnf?rHQ.)Jcm24} 6z)xʦ]wu>ޭG!z/\/rG\}5\rI=P'y9yEOgyyC@m~L)"~qԮlug^wUV!j٫odd*/'Oo4=Q#B!Щ<<"`7MΓO(B!B'yE &ǟ-#B!ƕ<֤F~|UuƲyS9;Oxކ1h?7VWbifsͿY7nsf?3fׇyi r&}d@-_kFqjAq<ʹm̕Tml>(͟˳h3%]q-"C`֥kTuQxNu:^^MFe:/2{l]X'*QYcخ߶7|9?mj4/imi 6`ehormw&@yP,_ɹ{Lyql\C6͑ԕK[dq^6X11"lVSU^vDZ\9qׂqKq$yFDu.p"VбV,;<~1zOVnۖ+Xm2qsvyɉ7^K?kyqmVɆ37S7G< oF/J2m̼9]XG1ʱj?X>ܿک}|ֺ^sg)GU+n^}1z1WuMjMCq"Ku+w*=W8 a]w_q>3j'9ϷοkY878={|IQjc3Z95Ҷnb Ѽ}C^ujcY=Ec.k_uZ˞XwTdif^Fޛ3O>jcwU.+n2qsF2ob{:1ܮ}{XyJ&|l} P6I8s]m=^_7k:G48n+r{:B}IxI܆\ՇsS{Ɩxu\G>΅r<4Lu켜#ys*_w;1/c|&Jq"{u}tYT̹|Q]+W56"[Ku6|?U從c׈⻮9\ _'Bu+Ϩ_Xc>˪ZuO8U,̋_Es]z|7x}ěPXo46+nVYܟMmkU;忪&ҵTȐƠq{,kA('I{#{ݷ<^u<'jԱV*gif^>rF6 \S^Eomyi4C`yc֮z/E }ezա>deif^w:tzt2 [U!0o'jȼ3govmZfiH~:X63;􋂙_CyCD 5Iۉ2o4tSo@6^+ ։j5dޘi:X63߀ 1v7oo.t7׀U2ZaNPy?$ҧ>}Yciȼ֖`XϤG7TW\u] >xi2t>̛.= >;;Z*^e,ԿVMICMZ^hnK}^MCmYk a`}99Ma>Ds9cqc|7͉ |f;fe%Hu^4 2Z-`|VeSPt*Tliȼ9*NV4_4?E_z~vq~&_uֿ\Ơl35˿x]76۠N\~[3?j穧4]ryokg|7Ge-ޱjg4vF|gsŘyӵ9N S܈#Cղ "f^λaPу'?yg}OgR;DCMҘy/l]3?**<}2oZkYcבMq|<NV+`Gu4j~᡹8{|,)6棚;ooŗ~Z[/ ]@ߍc>'ϏAsͫ>?s"oˣF˶p?B՗s!h Ts8_)lR|,Kf,棶w؆x˚s8Y?f75ʑy!3_k 5gO6^reݑW|n3? ho(E3V8c|.SEW;_$!l|ug5G6&￸%ӟ~WmkfwRדCղ hU6F㘻μж15ʲ2;wuUPTͩ5O2ZQ:ﱧ'mnز]yU5ZzZSU1TfUd.e4eLɌvQq6rm-lWyWڈ6ʳW:s鼍m束c䱹1wm`zfwwտʫͧ럖kv6]p/ }G{9Kf]ޫ΢ly3/MMZ:cMs?IГخBc1֢sMuզϨGïvMyW]|TV=`O1k,u̼ j|C5P4f޴ie2K .ټJ6늱Iy*Sձzc#i:X2sIfCu^TFgd:ͨFWHe#6Ul˿P.Uy<69~Icv]0/6xj_rgԼx݆~y77~gTڐGtQ6L̝ͥw]֍3oє}3Muxy͝Ĭ}F|D:Q,<*/tCzemuY?h9˒YF=E4$7eyt;l3y.ut͉'n`TV`{l 26z/*}.Ω`ڸ >#k>sNш+=rQ|J9ǐͼB<:*S7859?םqE Yn~o_775O3jL a|f8P{9ƫ?2f8`Źj;*o+q?j-KFfTev]ҼʀH|GU&]Kaa#j6zu 5IcsTeE߷I,Uf3CG4^WƭJZK3F7p>:W`l(6OnfeeL͡Mmml&Zz2'3Z؇Nu~^m33WnEsʹf=6ͷr"|r]˹]/\߫G}bu'},^m/MeymB! :cMzܜf9_Ͱ _Hm9U|6SeisU֦1uZҿ?/̮4ԸVG3o釒MLYW{Y=hjI3ocd\yR~Y\n?'y^4e1Қtz_`l(2_Ud8UG5mrQU6r}p|ևybކcGsRh57~۫ɱP[svu9;_oBＮy~^>z]67]u3mwe>gTw2ˬ}XǘyӅcU,L SF>;y~#_+CU?mt*CpP_Cv1Z5Xzt>qS/ z?vomʲo;n_!j̛"f^sݿyeǹҫG&~f^Ҙby^46ߙ6|mkE}.G=5SUSU0v-d"wyW6mD3*C8 p\eY/ bh;X*o3CqԏM^UG?<֫;FW17*S<7>!d; (ݟכOԋӆ/r֭.ۘ)zX,r^n66zFrCc̼ 1OyQ\k̾+/$G H@h#zR5*S\k3o>3<א} Vc ʪUw5k5VIdJ˦UiI3oZӋ6h\5Fx6՘\UǚoX/!6v-"Ks16^OcTj:Xtg;;:\Gsm+T}Ӹ|\}\M F:y|ܗsf>8G_88#!'s9~qdu-4{_?|m WՍ/r78a~FEYW3}O 3odd6ڷ1t>̛.t wIF_]h/l|+C2o |1V8mEշ9fGEr! Vc GIume4>UfLF.e!늕5$Uy͓58V6ͼ1}Vy)Oodo ]@ߕC̼fT/]V=W;w9_,ҿfXh3D+b9؈s:7 vBhNcF]rT_h_yz1f4OTh='2qP6cfGϹ1q,}+sm^nԦUiAwb}:> 5IcMs/'S*NT4mզV?5?,ϱKC2ocPE#sdy>xߨNV7"3m6}@}422i"s0nq)V1ϑJbylc==rcqXrW9‹3е¡_n_6脽 =| _ 6 Cc̼t)yݕ͏` V2XSP4ffT5dXku GF3;ߊ#)cۨFۆreTEMj'}4xD4luvoETc2>V\?j|q՘}xy̭2m̰~蚍AwƠ`W`ަk:QCMvs< ] T~&1M u$D 7lyNVAũLb7 CyCD 5Iۉ2o4tSo@6^+ ։j5dޘHm:X63߀ m: V5$1o'jȼ3govmZf/y] T~&7mmy| ycs_j^{'(̓c;:or4f}eůS,xHMCcNVA1 7l9NVA6Yyg&[wW)gifN yyyyy437BhjGh5 ZF[AG-"fk'>h+hQ0!Ew5b#m- f!F]q6\E#h5k({;;w5|QRb"V*Hb#m-ʆ52G6>Q*&Bmet+jM{Wϩ<,">g/ 4s|s׻>\ݲ.ssV",- f~Ij0)6#p,׉?cl]DhUJ޼7EwM lY!P֘5.y}jWy۹.H2j^9~ȟ5!?kcT;6y|f\kYguϒBԢ`W Ћ?PwmmN 1tj?F|k<š~Ʒ@^fk69O̫OQ1;晥J̢^tN^^ߒRQChZ(v^uv2ӗ2HQIEhIT&@O6U=owZwƣkMWGn|gr ̹ĺZ:'ܟ%Ǩ1{8{S̘]Y=nww)ۢ1k)~O[kOf^>UsKR}}fF!BѢ`,t?DC^erKf_}?D"$C}O(nbX-񝆲ƬhxIMsަ ׳FA|?U&B}Ŝ7e+=fUߟ?W>tc,*wm˶hjZzkYk3sE8?S# - f~I_WCT_?|u^j'TW?N!ZEmn|l >J߁L;/}; uiWqcZϪk7,y[wfކ"U;q.rj+?s6zq^Uo^hMjsжޫ"?E/Y!Q1z I};v7W~s<QDuOyWؒiwUWWs_֘ϵ?^T<~|Ο) ԧE/YlW/)Pokv5[3˯BӔηgz*,; iAq6YgڊAul"s6"6 T#:LWoUާ3g;󹖴^׺yz֫TyAhZ|F[AaF[AG-"fk'>h+hQ0!Ew5b#m- f!F]q6\E#h5k(K3=!4Em|seif׿F!B!- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!Z'- f!B!kw4_vI"TiQ6涝[}(V1Wvi.ʭuB!BӒǶ\u.BeÚy}(ym}8jUU2YǛ_:R@!B-WlQ(.msժ{}<:j}͋_+ B!yyU6F>yʘ- f~{-Ӽ3CG8!ivF+'"+7t'Rc<ԁe.W߱mr,/j| $ij=zh-u5.sH\9fG}>ZL'+1+?ih<7R1湫cmKAׯMq^z6Q{2νϔ,3;u9Τ]W==*&B_*+w4g_zMs%42\Mku9ZaӜu{] /Y!t*a aY3o#ns,8nwm~imh]x-yx[uX6p׮UeyFg|ڎ9bsVcWRW~}:g|mεy IsvzU_0#;ZeCU)Mqb:vl7*da˿xݠx2[!d- f~{8c_?5F﻾i~ck1h= ^iO7;`ɔIfjU&ōfބJUlR\+c֭fPyk6jfG=43/\_m۶YW_*ާ*NdwӖ!FMXYw|ul)G]Sq>^կ}N)w;gә|n6Lw?}KΣm>uc5бǤR̜lfo\qu3Chuˮ4}Qw|I}Ά?Y5]eh}(%2}TYFNPF-5=826n6RWǎWطꫝ˲1c}_@TqbYW[uW>'R̶)WMy.=O}xjշѫʻ6RٴGiՆ]nںZ9֭tm\犑׳{̭:g)^*W$ʯ1{kSט$soS3w)9FU\cL;ǶXN%c1'nETsw3>f?<'3[8nsz/_+fﾖmӢ`旨ʬSͤڨUjsޫd*E3 ^Yd4}g؆W66c{F9*]7+<19ӫcOۼͣ?|F^<&S]IUPm>y}QmLTWVeqMf7h*ͮn+զ9NϟG:ϱS5&~?W2nSgt[S|M,9ɹQ竒ڏxSyT}Nz|}ԤGOy;of%XXy3DUfOUJ2٘K2j2uԬXʼjh|'>A&}lӫ׶,qurގ_Us"s=d3R$Rj>|1I*reN7*+UODtnf卨ꩾs!W2]k6uyM:?7ZY(VU^mr[|qLu2Gorr_Tl׉}(9Nq >;3Xg3'C'|P3DUfOUJ2d^ԉFMNul6ccwzUW՗M^}9xrU]R_*SݖgͼW_߮~G%[x9sV~cFhz䍤 b]1uYFV9gǓrI6aqHq}[g!kVkf**O*S˯oܿڹB\Tc>b\9Zm`81V߹]LA3vz^u1: ($̣!ZU>Uqd)ɬɴe&g3*3;ժvL:^4Y6}<>+y l:c:N/gX+cۖg5&|<=\~,׫߷y[9麩qsx!ЩS2mU@onGȣO.jNU"(߷9rDm? UeTYL y4[M| ژZS*G(>?Fz|_K{y4DagLڨmsպaҰI8f13Ӓ%3Vf!BjQ6Gk3~DңzDqTy|4zq2G!BhZ2oD ku}Qv-T=Gl~}楟eo>cV:k!}ͣ,{-Ӽ3CG-67]~IigWno~t;oSO{Jq]PsUY%{E[o= ZZ^+Zˑu5.sH\9fG}>ZL'Ot1ӕ+ύ?gLy*7yTs)wJm1ڈ[ߵ8),3;u9Τ]W==ce_"y;/9kO]yyi~Ʀy 8e\æ9~'/ ^hs7}qCصAfw~qňs݌mu7x;v  %uNo^E^s>혃N19ulؕʯo"s?ڏ\+γdwO+ӫAعBd*'cx8D.220/^7(̼{ f~X̫o\{Lc9fmN@m~d7q)IfjU&F3oBE~-hZĵ2f 6ϋ^#6:_Q>mFm]ƭ|2}cErAyv]8W]TuK.!ΫVܮ-붾{_~βͼ:ܑU_"YZY_9\/iNi^kUcּc0E ^ *Uya42թF^s2lj[Y.cߛ:n3m}UQS1twŗ;+zjW=W5qy5H!sm)s:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏1f^xy<ޠVgSYFm#ͤ7S=QݍqHe^[22Cmc(`Ɵ5#Fպn[ymts屺ՋWbb}FBC/ȹXuD_Keu>(vߜ}>c*6QWgNf^~l۶mE``He4~Ǵe Qa_,Jn9T6b{W>9<.}Mgn+Tsmuяeuyes]X_yy_qFV19@mk0o0ҷk@k3YT= kYZnV&AM^ju눶܄εuůL\+G3];ml_:`QU3D*ާ,|9DhddFu ǎm[N#Qʼn9d<<=&뺯|"ϗȹl;>KUxLqzoTHF*hjiGiTv&m uZ93C׭t6s=V3U9]=fGטr]>bj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIQU U)ljahd%wKhc_nL Rد\4UF}蒉F^Tc9U`(}Oo6zjcTӈߊi&3O غeyV9NϟG:ϱS5&~?W}scY;ݖT9_='9WW6>|U?>G5Bqeݏ9n[,4n$T'D2"Mqcsj['UsTУgoο3S.}6z 7t~2`U`He4xGVw=}Q96M1̻^u:V_6zo\]WyvK}\Ow[yL"y+!}ȱueпW:9g|O3Pz䍤ںHnI?a׊Wۺ,bۼfVm"yϕ8\?*S7__\Osz. *cy#νk|Z1~S1Gv %&,g x۝zՍwp96ۯ1?:*/ʬi 6m6j2|62sSng(/㌍anϫsnoC2N&?R}3m˳qzuOWms)DZž*OQ?ɱDV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙn-dfuGs ڎcÚy#G^ Gmv={{7?xG`pcaaͼ/Zڮʼgȓ͡ï6׼qgw׊b|Cg-O^y候_"yϺ{^~ufw%[Ӽ9I8팓͏{`~t"6U,Q~rsUY߹_Լ|֔TO֖̱֔>#qM1j}3ϫ<>5LW~bh<7R>?cbusWcmK ϿWjFM$Oƹ]LeyW3Ϲp&L,{֧ќ}5ٗ\|-gM76O)6Y;Y}OBø¾ rD3>6ޔwmO,q;' M3l~6.vƵ~w8t>=uX6pw|c*򺍨g|xmvϩc<ƮV6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯h?f oPͳΩlFnܨkCrݍF5⍧6kKh-uX85Xez͟3ȆQnmmb]s~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhild eԢYѓsՉvGs؆Fkb±cjV}9c}_@Tqbc-O}+o}XW**y8Oq* ` &22fWm"T{7Xl+ i6غx5WkIe[yPcn9xHIʯ1>$S̽gRlsT3qNb1ʝc[XB.}ɨnEr9;ͼ^uNe2ey_"YPL_.$TmبJ fh֍ ^mX%wKhc_nLownq+_y;ȋjL:rm>yUQw,Vo2c~S}vB.Z;m;5\6d:>:>};3Xg3'C'Vf~TfOC!wdz'5hm6ccw" uul:;>ιcR7*4c7}]_oc VL(gr\9g߁ [Vn}>#g!kVkf*o\l#2e1ϪH_~}<ر<5>QWs?)iCksǘWt3tg{Xƻc8 ba`He4MIfM-5>Q9ݩV]q0ge|տ9!UMswLs}ضYI8_:O+q_qz9}+']7T~9vV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙi1qwl=63̫ѶhhU^>;XG魜2l=030103010301030103010301f<{!B!B#(ܙyyyyyy~ˮ>IfÚln۹wڨzp=ͥ_S~9pLlU'k;n keʏym~4Q2YǛ/U"ÍƆ5˾(ki*u#O6_q7o1_+>yQf~T=i^xՙunNw$}3Nzj+7?щTGF>UeUQokSOh~}?V"ZS.S=7ZSZ[.2n5c|磪D>~D3]H 5]&1qj.<_-Fq^z6=^t3eeu_<™K2񪧧t L0K2YrGs4g_rM+\=7o4Ͽ>?+~4gmd|V6q7}6g|'m)ڎ9SS6y]y+:wQۜky}W)/}9W.nWi?sqU)NCǎq\1u3eda_nPWnn9?q]]4q;־y0j:mq}dUߛ6 F3oBΫ<[qx=jmE ?/z|UGu#~7fbmcQ(u\uSU.| 8o\[s*8jrq}:6އpGVf~TfOke-f~-pDQ͆vj5/^ג_6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯cͼƮ|`axZmuNeC6zmuFUe]jn 7o<յy0^[ZmFkk2Q(vߜ}>c*6QWgNf^~l۶mE``He4~ǰI:ٔF\Sqڈ^ԟxs6Pϵf7ױ 6ϗb("<31/뺾6>2yN21W9q,{CnS]G7rqcS,hcmumzՆ=nd*dnڌ_\f2vѨN^hM\ZWżq+>3߅C[79G׼r]2~qD1sw7|fZD3s_vuF  5]e0M0K2}ͧ͢YLfN&mTGFxدpؿƾU_D ty={ܪsFb_hѕ_c}tI({ݥ#g\cL;ǶXNr]1Q Qsw3>f?܏&0MkMo4,R ht_!-ky}6;k׭b5?7Ze*8H_~brۃ/8{2u>Bqeݏ9n[,4n$T'D2"Mqcsj['UsTУgoο3S.}6z 7t~2`U`He4xUw=}&f31y!b_Ы]WRk#g!kVkf*o\l#2e1ϪH_~}<ر<5>QWs?)iCksǘWt3tg{Xƻc8 ba`He4Mf01:Q6ݩV]q/ge|տ9TswLs}ʼYI8_:OgH>4.kayp?m#|uMUO$PcKEƩ뉀E &VEzu;$]̫ӫ$#/*6m箅~(ڮY*fs3QhfCwgkAwOx`1ƏAw/dÚre#Q{J&_%s۝j!ê;:39ٰf3~ ʏ'GG/k<~>;U`6'yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛF{%ǞR\6'\UcIQok֣֡׊rkjMim<3t{ַ>]ct's#}kMcn8\ y]R[6wm"{2νg,by΅3}dUOO``He޳>}KiξSW^?o8{o߾i}~NWi:K}M_vm=f\1\7~b[Ǎsr>4GkCkkko8{ŲWmD>Ok; |>SL}N1v)뛯O9׊9]<8m`PG>v:<6ʼn}ر=<<8~2̼ '33D*ާ_?1N﻾i~c־y0h7tj#fLM6U27y*k@"1Vh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽< zZZZ {M/'&X3Ҝia/|`xWmDU^mLu*}z3#oj @&VxmVXy-Lw[_lTm ]z{_ŊUm5Fϕs9F\^=_Hr{"(csATϔY3'0R0K2}Z+cWE[`a<6zmAm&t1n Gb,/&ڒn#G6Ls0 խu[ޢMSms_+/^渪è,g57Bu_j8/ziGMǪ$X*34GLsGTY͌O2?s26wd۶m3U,Z3D*ާ1Ȕ'> ֱͦ6:z>Fl>Żm|n6uc^8xC{c}w=~]ɱ219@mk0o0ҷk@k3YT= kYZnV&AM^ju눶܄εuůL\+G3];ml_:`QU3D*ާ,|9OhddFudFBu+WطꫝqGEKs]sxc_myzLcu_|87SܘQ8&s1q뽏UMdoWwm4E4Ig i6غx-ڜVeֿik׹bsk\D[kLB9ۈ}.V1H5>$c9Ŋu*"ǜ_*S1{cqU^Q*3Z0%R> ͤڨ@۰ QѬMի X:wh\Fr[mMg58gzU}F^TFioGuw?yyL:] osFOcjcQ][7dy~~![6#,@?'Ciq[9sʱď1o.s6<+|R0kbԷ$FG ԧ/G__?SE?vX3i 9fN5%R> E1s!&[GZ6UhF>܏1PkMo4,R:x*W=m_ә\77Q\y?e~֭xgCmѱq*v^MsF}tŪʫח_ny /^LuP\pz7ϑoE˨O*}(9 ՉǦŪ̼cS؜֫ugt3|~/gc>nͼ X%R> E,ޑ]Ohd(9ٌmb!f}W|/cձԫx8Uʳk\KRxcXy [Gn8R~/oU_T?4c1 śͯ7NH mڌi9;6aqHq}[g!kVkf*o\l#2e1ϪH_~}<;9ܻ'u}G9Ÿzs4m`(q^r|mbyܮ~yWx7} gsaQ\<z`)f&g3*3v61w;6*tirx/W>Z۶<1W6Fqۜ}+']7T~9vV~8u=ت@onGwߓyczdeVņ5ܵ%Q2}Z%ln&*CVؘluv-?>F1}.alX3ك3Sc6jܰsOiۤdyݙi1qxl=63̫ѶhhU^>;Xqwo`yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛF{%ǞR\6'\UcIQok֣֡׊rkjMim<3t{ַ>]ct's#}kMcn8\ y]R[6wm"{2νg,by΅3}dUOO``He޳>}KiξSW^?o8{o߾i}~NWi:K}M_vm=f\1\7~b[Ǎsr>4GkCkkko8{ŲWmD>Ok; |>SL}N1v)뛯O9׊9]<8m`PG>v:<6ʼn}ر=<<8~2̼ '33D*ާ_?ƚy]~i~c{ fmN@m~d7q)IfjU&F3oBE~-hZĵ2f 6ϋ^#6:_Q>mFm]ƭ|2}cErAyv]8W]TuK.!ΫVܮ-붾{_~βͼ:ܑU_"YZY_9\/iNi^k:E ^ *Uya42թF^s2lj[Y.cߛ:n3m}UQS1twŗ;+zjW=W5qy5H!sm)s:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏E<ޠVgSYFm#ͤ7S=QݍqHe^[22Cmc(`Ɵ5#Fպn[ymts屺ՋWbb}FBC/ȹXuD_Keu>(vߜ}>c*6QWgNf^~l۶mE``He4a4j22رͦ6:z>wl>Żm|n6uc^8xC4QX_yy_qEOX} t1)ܦVcF.n #} 6ULճp6Y E 1VkUk2vѨN^hM\ZWżq+>3߅C[79G׼r]2~qD1sw7|fZD3s_vuF  5]e0M0K2}b66r22jѬ<86T7kWطꫝqGEKs]sxc_myzLcu__m<Ŭ\gnCm򮍦6Fq]1ILa9m[E33tLW7mm:Wcn9xm_hѕ_c}tI({ݥ#g\cL;ǶXNr]1Q Qsw3>f?fyAr,/#IPm27>MrQ5u3y#zu>oȕ0Z;]kz}6;k׭b5Tk:gGQʫ>}嶊mU8Tŕw?}V侌Tl؇rиPxlkQ̋<6q,9}o:\W}AW?SA;y9L}6֪IU_"YPdYFMNl613>D+zձX}tw}s*w]5.rotN}C'`(um~iuFRhvWLm]f$LH W kEmzv? m^Z6S~JdeOQ)yV}F/v'P=ر<5>QWs?)iCksX}v3tg{Xƻc8 ba`He4M65>Q9ݩV]Me0ge|տ9!UMswLs}ضYI8_:O+Osq9}bKk뇌SPL v4{=9HW;=?WIF^`UlX3] PR]*ާUf2n5VwmׂN**cߗ^VɆ5=83>Fj ;MJ6םiߝS̬;[ <cÚyX;c̼mKf퍀Q} f`끙yyyyyy43C!B!E<L <L <L <L <L <,kw4_vIX5x`s]-߾sFm׃vi.ʭúC-`~fg:A_q`ulX3/S~kᨍڮyϺ>|ȼn 6l66]EYK۵Py9tf~60yZQ~謅 >70K2Y|wO˯ {Xtww='qSS_}̏jT7O=}`^r*unCqrUe9t5o6;hj=zh-G֖̱֭֔>#qM1j}3ϫ<>5LW~bh<7R1湫16Sͥw+h#K^o}&'܋~̲̼+֙\8wI&^T f~T=WhξKi>u󖋳'qS'za aY3o#nu3'q;' M3l~6.v֭W,8;1{yF3dcc;ԱMcW+Dk?js:Ϟ!U>s\N c_ cS؇cgH/̛˿xݠx2 0K2}Z+c-f^Ni^0h7tj#fLM6U27y*k@"1Vh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽< zZZZ}FcQ3{WO>V6]e*6FT:ވ>k7] Mu+6+e{Sm|6j6r|bETO*Tֶʹ#.q/$tN=eU~]W[xױ]9sVs*ugʬyX)%R>x?|_3js*kmFuǀ32 nkKFfȺ~,S2ϙbd(TZmy61O}s~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhildbj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIQU U)lͺahf%wKhc_nLoϱT_maoG~4BtNu*Jӛ箍4"nẔStB.ymFrY^j#Nѷs,cc5E\myVnN:aĨoIU/8_j?OQ_,Tu=~q?5켙>fyAr,y= :?*0K2} Y#׻>Ѩ0 &b݇}B:v]/NʼkhڮP81V߹]Lo^=VnFSFMfTfNwUl>WUn<ކTe61M/gX+cۖg5&|<=!9W?PjطruS=!@?'9 '2[h{rTwQ3vz|_L ?ذf6jTfO$DeȣjTݝڮ=TUG(>?=/C%< k<{pfu}Fm׃v) {Tl3;Ӿ;-48.Ywзydž5vƘy#V| |8|>}^2l=030103010301030103010301f<{!B!B#(ܙyyyyyy~ˮ>IfÚln۹wڨzp=ͥ_S~9pLlU'k;n keʏym~4Q2YǛ/U"ÍƆ5˾(ki*u#O6_q7o1_+>yQf~T=i^xՙunNw$}3Nzj+7?щTGF>UeUQokSOh~}?V"ZS.S=7ZSZ[.2n5c|磪D>~D3]H 5]&1qj.<_-Fq^z6=^t3eeu_<™K2񪧧t L0K2YrGs4g_rM+\=7o4Ͽ>?+~4gd> m/n6m'N zS޵yT?\q7c7ͰкZך=t׽bI_٫6v^}'s>)>m]WDu9׊9R^>s\N c_ cS؇cgH/̛˿xݠx2 0K2}Z+cMsMXcDP7;`HVm3"n4&T㹵lak׊ףVD͠׈ꪍWuOj[jrP]/Uj8U]Ÿg`jƥ>rc溭/׮l3}waU`HeVbbKo;flepX3yi>V6]eWmr}~9iF^f:ojM.@uSfv?i6Wu>5iCW~^|sW"v*kk[s\cWc8:=VXu|uu.Gqe8gU:r]Ǿ~z>_"YZ̯c|e~ca<ʆlƍʺ6*xoT#x1Xk ` R׺exZU\9SlV-o&o:cu?9r0*9t͍Pׅ+ ^sӱ1>4|;9Q9}3UmC3ïϜͼ2ٶmL4/ʬi 2e;iGǸwcM)>ڝVh23ޫ_Żm|n6uc^T9R XDc}w=~]Ǚ*>q:cRMu\FlT2 X:~[r^aY ? 6ת)6y]4ꪓi#r:׶2q1|ʯwVy5o"Ϗgs(Ə*>~\WL;|Q̜lf7j꺙ŗ]iQw|>g,E {WLzbhildeԢYѓsՑя Z~c6l\yԟc* ` &22fWm"T{7Xl+ i6غx5WkIe[yPcn9xHIʯ1>$S̽gRlsT3qNb1ʝc[XB.}ɨnEr9;ͼ^uNe2ey_"YPLZe 6:' }6R4 5uc;6zauKhc_nLS]W_z!_yK&yQITWsR osFڌmeLܴz6ui[߹,@?'Ciq[9sʱď1o.s6<+{ERՍTĨoIU/8_j?OQ_,Tu=~q?5켙>flAr,c<&r*B˸N\ܴVȊz\M=u^1|2ZCZ[.:lw=?:6[k2n\3(VU^q̑r[_xq*e|ʀsܾy|+r_F}*PYC9hIN<65(VeE8>η^O>砫G<ߜgg>k]ql$@od/ʬi(2d^zD&Ci'l6ccw" uul:;>ιcR7*b\9k08/9N63Ϲ~(.fV f~TfOcєlQӝju;@xgl sx^}VWp{Rt:w49ؗ+^am[՘ԏիxr(cy[9麩Kk뇌SPL v4{=9HW;=?WIF^`UlX3] PR]*ާUf2n5VwmׂN**cߗ^VɆ5=83>Fj ;MJ6םiߝS̬;[ <cÚyX;c̼m+? hU^>;Xqwo`yyyyyyK3=B!BZL <L <L <L <L <L <lvGseW$Uag6;GImv=aүک[w?8txwj}7KVdž52G6?ڨzPo~[*`fcÚye_] yϺ'C_mv߿ygY雷ZHs(3D*uw4/yKM|y׻s>p'=5uۛDTWmX#XUFr窲sϿyᩧ4_+Z))-uc}F1c>zQg"W?y|kXyn}rǘ8N5Aׯ8/y]Hs/22ïXgsLz%xS:X&'yj~//ͯ~ojj?}KiξSW^?o8{o߾i}~NWi Cm/n6m'N zS޵yT?\q7c7ͰкZך=t׽bI_٫6v^}'s>)>m]WDu9׊9R^>s\N c_ cS؇cgH/̛˿xݠx2 N2You1gWnn9?1G~ck1j:mq}dUߛ6 F3oBΫ<[qx=jmE ?/z|UGu#~7fbmcQ(u\uSU.| 8o\[s*8jrq}:6އpGVE~?uԀZZ Z}cͼze5]u(^K~CwU^߿礍fѧ?3{͛鼩6mdSMet"^2V<-Lw[_lTm ]z{_ŊUm5Fϕs9F\^=_HrXc_Dס2ֹ8ulyU9u2a>fV`3/Sf~c7}cyP0JA6:!q u7^Ո7z <-6Եb(`Ɵ5#Fպn[yv}ۿXݿhz\1byv]s#Tu¡fyt:yL2:{Nxo>|yALe{̨+3'c3gLm6SŢe0MFyTݡ̟{Σ2{ьꮮm6b*Nޫ_S>v2*󹶺Ʋ:s{3iEqtcr:dȘTfAKocTn3_ի6q# P'ufZ56 E&WF]u:uD[nBֺW&.敏#].6jMqlZ܇O*v{瘏#sp̿[W]73"y;x6/Glcea*i2KcOfD'3 ̜LۨlCu+ c6j'r\s{\׿2W[uW>G(b)f괵U>[=ouTȘ^fSG(N܌g\b6:Mv`\}.$[mmA1zzU>"*'+PN16bKG1R9Ƙ*wmb c1'AAf|^xcw7z9_+fﾖmBf^o`OͤڌIBe~߆j65uc3黿6Jz:<W(ǾV}9oSY)Ι^#/TW!7귊6By}rF^T9UU9@osFڌmeLܴz6ui[߹,@?'Ciq[9sʱď1o.s6<+{ERՍTĨoIU/8_j?OQ_,Tu=~q?5켙>fyAr,܏&0Mk j4,RbU^/U<_n{gq\G( Q=Hη"eԧb >휄cS]bUf^)cylQ|3~ zt>q?|vf곱VEf^OOfDd]WfL0D&b݇}B:v]/KsX宫<ƥT.ʻ-<&ͼאbSR~/oGn+skMc7BjsLUsڀ?;pҐ~aa׊ח֭u5Y,ĶyjLE+q>~DU,Y˯oܿڹB\;9ܻ'u}G9Ÿzs4m`(q^r|mby~yWx7} gsaQ\<Nd`OFSfLfTMwUlEcyY_vm:Ufcb_x}2myVcR?Wb˸pB}R?1I%u"D"T`"}O.jNUX W__F ᶝ6jTfO$Deȣw ݝڮ=TUG(>?=/C%<ͼJ<{pfu}Fm׃v) {Tl3Ͼ-48.VՎwyɆ5vƘycV~|=8|>}^ɏ[9Sf`syyyyyy43C!B!E<L <L <L <L <l毿֓kW%f`iz|: 3~ˮ>I6ۇϙ0=ܶsWs˷%Q{Krkn@sy 2߹Nv|ȼn 6l66ye_] yϺ'C_mv߿ygY雷ň7<|n`9l3Y-Ӽ3CG-67]~IigWno~tQ]bbU}˝ʪ=歷ߞx>|hE\zo\ek:njGU[|^1fCc幑AkMcn8\ y]R[6wm"{2νg,by΅3}TOO`tyif5rGs4g_rM+\=7o4Ͽ>?+~4gmd|V6q7}6g|'m)ڎ9SS6y]y+:wQۜky}W)/}9W.nWi?sqU)NCǎq\1u3eda_nPWnn9?P]?zbah7tj#fɪ7m@čfބ Wy< 4l-Zz4ڊh=T^Q]i3RmZG4nU?o|+GPʳŹB]r q^M߸vUnq\3um}׽ьx#gfy37&lj⺩NkZ]ʟ޴qn:괍+C/9_~XS r1Lj˫Gb Snr:Tf:ǣ2a*Sc_?Sf=GJ/ʬi`׏ft}y*<ʆlƍʺ6*xoT#x1Xk ` R׺exZU\9SlV-o&o:cu?9r0*9t͍Pׅ+ ^sӱ1>4|;9Q9}3UmC3ïϜͼ2ٶmL4/ʬi 2^;La4c22~LfSrs=SLi#{Eԟxs6Pϵf7ױ 6ϗb("<31/뺾6>bf>8:yѱi Mu2ōadFOu*1*Ui3~ql"j׫E:y:-7smk]+Ǒ|i?o5L8]&~6qbxHCx'uds98fx披jͼ]|՝Fu#w62X԰w4/ʬi(66&?hddFud} `[l:;oW;*;D'<ƾǮ>rΑ<WyEs>DFlM6j8Bqfbj|&I5TslT(E9ͿU:|7css㸙׫ Tf^1l3`0K2}JIU U)lj_hh%wKhc_nLovِD oDӛ羍^u|7Rߙi&3Wm7^.FS:GWyGy(`v̯֐֖˵Ƽ>jsǝwϏV̟j-W2UyWs/V|^ q:2Go#9ߊܗQ-TPsMu "UyǦ89GW9g*|37_ٙZu}>y= :?*0K2}LW׻>ьP{__}&LgĴ O(i|Ѐ540M i8e>D@WLT"3 |@ozpu{{Y^ɕ㺮H،}1b1Dυv[]+~kV*Ͼq)eCn$׵wO$'Cυ-LxOsrU<M_כ&o$667ʙ^eڀ+s8٘kEח֭:lb߼fV-"yJdeOnQJM_+;tj+򋘇u_!XEДY f1&Qg1*'j~y0X\1+<ߢSu>M/R{xmYIq|+ƕS i}1/gv#?'/?GX/N}o,C%0+Ї9=wQˊyczIK]+>zl(Xu^%x-f}COgAOO`E,z}_Nv?}L)">wD)[dy=}h1qIꉸp!yɮ}jװmWwQ식D@17AL <@L <@L <@L <@L <@L <X?}aaaa زd`b,-z ,-?ޛb=Ov?mtwUwo>XOđ]}&EYJ{,ak7K3OE8uy}^+y=K]'?>/qRl`){ y9m3Z'apq6b^nO =m~xoJapq=ġG޵b~7qǺB>; nCgޘ؇iyXwkw^`=`}Z1/Q曋uyQѬPXT=ϝ~?tg_s `{` lkZ1˲P Jg{sod__>d#'E>>٫/Հ_{z{fؓ{#VxO/nwߝ\?z'ɼ|-gVէ.mw~3(k邾rի.$ {N"ZS;7^o:of߭ur>c>:wy4y.W<>7L_~b<7xt_}WosREކ>st8/ΔY~g>W;kU_{qo{nE]owǿu'^B,;; 9X?eaѦ1o_ fɎϾ7qS8 w:њg.?6Tz/-uE5s>V|OVjP^q|{0Z[^-׹IȏiU}}cO|䵬YVޢ'ٺoD_~NwGř*O376/يż*9p*i_E"Wb[Ou݇oDun#߹h>x>n{ k >Gј pɽWYˇB5utY,uSԷ;L!γԶOsQ{cT NS_ekg\6{c&V~uqil(FQQY{a ˣu!u~Zy]Uk8>Y({H)rM,0sTW7$}ۚEMq,ECxmOd9|^;w3"y^!> ň; f,+` W(,*>dcMX0Z^bLG 2 C 3--Gd!(B?]}΢?θ61*qr\δ8;Aeٯ5.y_D}_UT9eqs?uݿn"ߺ_z 4o6Fm& l(cjpn8L^zZgV߸#Mܪ2#1F$ǫr-} f?Dߘr1oň9F8'1Sα+P˼fs2rP݋my1t.,߱V- b~¢C6,t/^Kik4~mt.1}I8O¢EG8*ψǑp{E<$؟R1ȱ81\#j(E5}>2ܗz ݰ'~ovY);ƻH4 6_ EJSm6ڄz޷touָۛYƶo {>ZyOH򛟊VK^㑪$Yc_-2nYUQ ($rkBx/GPW/ueW\7֛gb.6eEJX$$$Di: HW&Tq*i8ۈwS*}hn_Wq 15GX:ls5wQqTsm,zIXJ6)`3Vm*T?̟_cYF_4Q?<jxWgof2lξۢC~l,(^ߟyƵS@P:&ͪ(Fo"U}W w/~P[#Wq~9ȱbʷP]4n$&^3W%Eؗ?η'Uxř zu>?q?7|}&곰Ve_b^o X)&XD&H:Oh%<~B3 (^Udj8[Ys^ꢠz_}̿/b6aUǘc^s/ԧocvq>'yT~EW8p{=FKHB" nϞȇl˭<#AU7ĉۂ9Q":o~skYМ_ntؖ`fmcMve'].zB&!uټbShuj,/s(^uo{P~Cv<c<7>QWstϑ/2]V̫^ӫLB^`]Z1cKGI}w4qCM*>dK "F^g ,| v{nC w ܖ5~i(#Qoa#lγQVkf<iq|~+?OjN~(^tzjtOPց_m_%/d׊gD/bꣾRLwIn{Eͳi*T *܍cpKжiyVyߖnu>A#C.vٸ\!vV_4z#Qz {$j-U]yWqeRbch['xPhż8?2]G֏y/|Y#visnτy_Oy6V>Oݮ/f][ mm͏A qS g螋v a=E.ǚk's$f]iy)9>a=;? [1ݾly` ='19(.h=An b~&7|) ȍx仈Ź =%//ӚӚW =1Sm1حn㙏qHXDALļ72X?C*ļlfbk~=y\-+ּ<17~@>1?-+Z3y}Ĭ+Zb|10 qy!޻𞫐ml޾z|ߓxc{۽7F>6!E1 [{BbOϘI0`17}2W a>1/< czmˀy219 0 0 0 e<@L <@L <@L <@L <@L <@,ɡG޵ͮϜ>z{Ǐ-dꣾ;}GOt|^{3so{yc݁y}ZkżDo}~5Qߝٞ~T;^N$ذصb^Oٗe;}C%޳= ݹ7?}{Xq=1.^}o̽ {z{fؓ{.'>-K/mw~3G/DOk6ߨ_+v 98We_k{wGPkEk9ҷn\eݺG>c>:wyj}劓'ƘOFy7yTs)wZ>Zym賉8/ΔY~g>W;kU_!x;tWrwwww_y9r}]h"~'+bM_mg.?A?f'u;' fhmh]x-yx[]u/_p(򺍨goؗoh"jsԌ9׊9]]6}u6FT6ވx1ț>Aɢf^@V^ܴ|%[,Ԧ5νz|+_S յVc\9טc#OABe~YXPVYڷw\c# Fպn-Z}bcNs|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d Ql>ݕIF&q(xɽ2Q۹L>EsOyn2՟=&T"x#7^kdS,*\ g%y_jin#Z ֺW".敯#} >Қ?o5L8}&}qYH!<~lj'}uD>sw; jżzWBbl }u0M+C6EOc!')Ś8lq}cWy\tn_:mD'dciT 5\_QL-,TeQ fcBO-xml*;W(c,U}u|14w8-?鶩M%gbssUQňUQ!1@"}mmTu9ǩk 9orqD35'cQıgFK1|1C˸M̛VWOj t&䍨کsTyސ+0Z;}oz y'W}㺕&M2}BjC}Ͼ(uj|J;dc Odu| eO1F;QnkŲO}sתw[7.RlhQ9=V[q0Y _w?tirx1ߐ`m+jL|utϘ[14.#?՚[9sS;!@?'/?+9$XJ`W$s{ Lm*XV?|RQR *>d$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"dO%1' ~1صb"b^m- hU^.b_x}[%yb`b &b`b &b`b &b`b &b`b &b`b &30 0 0 0 ['1011011011011011$zX7V̟>s{?=ztϽ1Sǎuv#kǮoxG}wJg{So{:F`^c׊y=e_xԳ/tx;.c,?|d|ǧ?{R1+ߚ cO>? tا.|[SqS̯jToݟ^<=|J~]-'\U}ɮwߝxCGHߺ՚r}^wk:Ѫ{"+Nc/?1WY"湯1>Sͥw+kh%&'܋8SfUb^OknL"^VV b~T=W8]}7}{=]wo q/X Shs7}qCطAff\>ynQWq7nau5>nu׽|Yɾw6~^<ڎ9~ԾMc_/_R3\+γdwO+Q~Z?0W\{~-DƎ"M~b ]۷Ǜ!rgHHB̛۾y(j +C];"b^Gu:|]@׽޼yI:mqݳ'gVM&QŪO_ͼ Ula"Vh=#j>*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< m𣓋 vka,~^jW'jhNujS苧^,jƮ[Y >aM˧[*BMmZcܫΗQ;P]o5Fϕs9F\_=$T܎eU~}>oȯ}sAe8g섘?"a WH%ևl wyѾy)A6*S]kj3鍦TgGO%@?}ن|- 16N~Ok{u:L>`j[Vޢ'橾9ԩ?ΗjWb|CFCsȹZmP *O=q<=4g:98g8Seb_v&%[׿_%YŲu0M+CeUm ({7-6e2s|O_k)8x|̢3sYm7c=qj K>Oƫ<[O?1/u[6C?яsT@} `FfvT׆UZYdjjMfQ^/ u4܄Zk]+ב|i͟o&syy~>8Gby$Ds[>w:"9?Nwb^eoWanB1΂?Y :&R!ŢŧPbMBO}FB? (+6 k 7V{ Wy\tn_:mD'm$B1B*>dcp\HI~lCI$JO|q,hu{_[تTu9ǩk 9orqD35'c1i[3Gqz1T ˼iU]|u^Tv MgrLވ;G p<'~֭xPG{b~7[k>}Tk*3'*7_+y+Ύ~\Gȯ㨝s$[cŔoC9hIM6gJ̋<6/9oOn>3|~~oLga˾nż:.+C6 DV_PIPFhDcļc+ uV,N9xzU}R,ˆS3Id1k5&PпW oYi>7 0o646y#)Al=S_׵iOVdq}֭xg5j17W" /|rU,YŌ 74_~n'sv c<7>QWsd`)BMbTbNO,Ea0cbVWy,:}^7$X+aʳ8_ӗ?#?՚[9sS;!@?'/?+9$XJ`W$s{ Lm*XV?|RQR *>d$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"d:>֘$fD<>AO VYD̫_Ѷl8\_]ߩuz~U yyyyyy+Ϝ0 0 0 0l[L <@L <@L <@L <@L <@L <:|;x][ `Z1ǺB>wDw˷굇?ӝ>ƼLG;ڝw<0XVKWQ *Nu?9D {]+}Yw;T=SϾН{'"UKٗo j@̯Jg{'W^k&='}b-oMvǡWO=3Uʗbd䫊a';WUwߝx?x%8V"ZSS;7ZSZ[2n#qMg1*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< m[XT̫z4'~fϒ` Uྺ 6.?_nhF}KcLMmhEM\7ЉxmVkOX~S-݊Ug61w{_Uշʹ#~*Snrs|y,G~u8gU*S>ǡ8Sf's1k1B*>d1shfu jyVlZmFUu}ji7o<o `[Fko:?3ȂQm[yVg߽חߩS/.ޣٯz09͍Pׇ+^sӵ1K%U{Nx{htsp>QqʪL卿Kb1gJ80e` WH%ևl$>_Ŗ(bOt/wڂP>*s;ɧuv)7,,s_:f|36۹pxC4ӾR\g>ļ?8H~T|tj O9d"=Ă6jcTo1_6q# P'u~q,"׫E6y6PYk%b^:җߐ!Vy7o"Ϗgq(U,whn+}"1_G3|?n󞙠Q̫w !,_(FY6`YW1B*>dcX4kcQIIH: j[ Ww1ګ~GE?@T~bcc_c!rlշT|r<{qNtk Sm"#lM6/?B~fd6$T|ks?z^2K/K%3>wUy_!XDŘ^L&T(1uc1駿Qt66c1>y*jLqtTEP[˂\Ek|nc9E!/Lm m>yUQOlVo2c~S1~Nk}纼+"'ciu麺7y圲EG:FyEgV}Fm#gbssUQňU5z1{C}}f߸n;|Tk*3'*8P~|r߳:; q:!v;4ϑoEeSb q;'6TAD*1/Ǿ<6qut>b/TЫ^3Q.}z of@̯JX/cQ$Zl>1"ƊBG]RG?u5^mg߸KTycYZym;G}s /?ǣ5WsoMcl7BjsLU2m9 lL\Xŵ"Kqy~6 o^ZS~}%2'(^%bUP~Cv<j3sĹOT"UG9E:9g08/Olbtϑ/2]V̫^ӫLB^`]Z1cKGI}w'(6 bާ+9tzjtOPց_m_%/d׊gD/bꣾ;}GOejNg?S' b`okwDw˷굇?ӝ>ƼLG;ڝw<0XVKWQ *Nu?9D {]+}Yw;T=SϾН{'"UKٗo j@̯Jg{'W^k&='}b-oMvǡWO=3Uʗbd䫊a';WUwߝx?x%8V"ZSS;7ZSZ[2n#qMg1_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d *_c~'2Q۹L>EsOydciT1 \_QL-,TeQecbO-xml*;W(c,U}Vͧ7Cj3iM3qMf̯x*fi\p%Zd:?]W:ϾSS?WCs}ܪڨmLb{Nr*6q*?ʣXz|Cq^q u]qLXoA^ 1 E1 s!1& AD1χp"0 MQ[Eo.RX-R[%qtԜH@G.bLۈ*~xE]jcqvYܴVȊv^MUP> h_!-ky cO̯6u+yMZUQ>XWy1G};ϗ}Q؏wߡy|+r,-Tc(9 צ "Uy&?9Gm3}q^OOrw _,vכC7b~Tb}"1 (-6ccw csV׊eax:Uʳo\zx[y1,u6#b9'Ős *S<容rn9<o646y#)A6QT*\񜳟ąEX\+n]בxg5j17W" /|rU,YŌ 74_~n'sv?..1ykضUBrybs17AL <@L <@L <@L <@L <@L <X?}aaaa زd`b &b`b &b`b &b`b &b`b &b`b #[bf׊gv==2Qߝྣ'[P=t7=`<ر{y-صb^7>NPlO?woН}y/X'lkZ1˲ۡٞzouǟ>ٽŸ̞ogZʾ|7^Vb~T=ۃ?=ѽ[3A=v?ԥoyk;uzU㣗|Ӌ5oTɯ/ٵԽu赢[ZSZ[2n#qMg1{y\4nE]uK1v m/n6l3ߌG o?m3:nwM3}6.vƇ֭㺗/ 8`vyF3|2^1O>uڷcS74__jQߜkyIs~:OkϯqUOkx8DL Uys7Ob^mb~T}ȶ b~XF̫oڏN GDP={rV`Ed mYՉ̛PQq,ncK~1i ֺV[>d[db Q(\SnmCW34._[k5׭X?ν?8SgbO݇Pú@̯Jvَߎ8>uo7/ y{e?/~ ^ U}a4j:FS/͎G L5c׭ڬjzS-݊Ug61w{_Uշʹ#~*Sn2*>JWx׾]9sV2s3evB?V+C];Ǣb>yiǂj2յ6z6hMpF~4^#їm'ڒn#nG>w\c# Fպn-Z}bcNs|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d Q\vj~k)sOT:W\_2WeYǺߌĩv/DK>Oʫ<[O?1/u[6bnj۩cgkIy>gōadh@k3YT; kY *^Th-Jy,nV&@O^uۈVnBe.yH_~Cb[79G߼j#([m%:c\a1j~"U1ۗs[Q9d<1V+O}[ȇ|E]!US)cֹU?+E6^U߷fȏ6~*&?SmUMvk Z83c׭%Zߴu.y={ܪ2#*ǫr-} f?Dߘr1w|+F1R9Fw-_Mr]5QCrP݋my1t.,߱V- b~Tb}j2P[XʢQ(֍ [hEn4c1>y*jLqtT1B>#Q< ƒtǤ2՜Tc|zܷSژFVq+՛;o< ؿuyVEOƮuu 9e_>u|14w8-?鶩M%gbssUQňU>jo\פ}rUfO(VU^o(W|W^Nm_ pQ;Hη"2)Bu1rиPxm "_ylc_sT8::V1}g*K/wZm}>y 7t~3`] WH%ևl,d񉬎xP0O1F;QnkŲO}sתw[7.Rlhg!B#,1śͯ7MH m[Ou-!aFZ9gx",Y\_u5|/ľyjZLE8>x(yV1#C ͗㫟\w3sĹOT"UG9E:9g08/Olbd$ͽD%ȣoT=n==TuWj"} >:ٵb3Q싘Np`oگ"d[Hꉸ<c׊y>y+6WwGˣ;5ncV b`yyyyyy219 0 0 0 e<@L <@L <@L <@L <@L <@,ɡG޵ͮϜ>z{Ǐ-dꣾ;}GOt|^{3so{yc݁y}ZkżDo}~5Qߝٞ~T;^N$ذصb^Oٗe;}C%޳= ݹ7?}{Xq=1.^}o̽ {z{fؓ{.'>-K/mw~3 Q[|)FFFqrsU]oyykӋ^kE/5:S{5.=tѹˣU[D.W<>7L_~b<7}7}&1j.ϯ-ǵN#?E^9vAs\OGi_q;<61tmous__)#! 1on=I̫-:@̯JvAˊy{yAB~/w1FP={rVHVm 7y*TX 4/Zz4ڊh=#j>*(jkwQUOf(E+XUB9(Ͼ/U/?U[f?0y5CRUűVs݊3uV-}< mv ,*CżSBby~ 1xWm*Ԯ~eFS/͎y37&E5qTB'Y>a!O]Dw+VUڴЗ߹W_/;~+vWVj+s{$?HL{ʱ/ϡ筲8oyU9LL=G< mα7}c[<oPͳT7fj7P^O%@Qx`m}g㵥u0ZK}.ɏi V}|?N|GBmu[<Ns|yv~.ˇQ]l?4Ρon8>_~^9,&I X*1ts#Cs}ߏ3UV-ogB/om^y;UrU,[1B*>d l>ݕIF1&q(觺2Q۹L>EsOyH2sYm7c=qj K>KcUޟ⺭?_.͸_syrsx#7E6zjSm,ƨbjWmFBO%Z5Y E'WB]m:mD+7ZJżu/!1߇CZƛoD8Q?X18VDϝcg51~=3A-W[CXۿPmV b~Tb}bhi,$$Ƣ Hsu-Զ1_}clW?*K繭2dv[CExQW\rQdйbjYdWm"T7X+ ʴn/^yUeQ \khA>zzUeFbHW$Z>z~1 soc.Vcss>U[b k1' Q2_|b^G\Y*1/cZ %R-$r}Ge2:PEE4~kLncS :FcyLG[D U? w+ |Lm m>yUQOlVo2c~S1~Nk}纼+"'ciu麺7y圲EG:FyEgV}Fm#gbssUQňU8FYhZ-Xu.8um"mU.9sDz<8v_1;}xE_jcqvYܴVȊv^MUP> h_!-ky cO̯6u+yMZUQ>XWy1G};ϗ}Q؏wߡy|+r,-Tc(9 צ "Uy&?9Gm3}q^OOrw _,vכC7b~Tb}"[ŘeO1F;QnkŲ`9,pVyKT/Ow+<&żטFx29GŐs /?*S<\9koMcl7BjsLU2m9 lL\Xŵ"Kqy~6 o^ZS~}%2'(^%bUP~Cv<j3sĹOT"UG9E:9g08/OlbM/R{xmYIq|+UsT i}1/gv#?'/?GX/N}o,C%0+Ї9=wQˊyczIK]+>zl(Xu^QTs;VOmN *:X|5~}aZ1ٙ(SEL}w'RL^z'Bc`qB{]+a,"֯a^hU^.b_x$.y b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &b`b &E[OLIIENDB`llama7b_autotrans_1to2_predict_ckpt.png000066400000000000000000000153161476650141400355630ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRsmu? pHYs+IDATx_l#}t'$Kq촮mh,q[4 n%SuC'KQ?Ku(")̠!R G61ڢM6.؉Dqw'N/q%R~`ofvhcchr9)dR)dp~81zksykN/GpB '_8m 07Ƙmj(4K4s|j_sOC9Yd4vNaOc]4yrV*zs]wtn˩\YFF0lig;;F rcLW:YY#"Mv}Pi49K&e ɜ5)kƖ}i-_jJ!̱MΏ.QH9]DwW $ZCEY-*~<َY[ Ue߈IVUъh/@5BUX(Z"+d:L!Ǐ64Rb(Uj]}ݗbK^fYs_wjOn1V2w-lZ۝kk]O!Cnň{>{PEQUe껉yJ;=Ƙ0\fe58f]?;);x*@3^)ZAg-DTg=enMDfk5jUΛE-KWZ ":)!W!<,= F܊|0{(wU!F%CݩRhq!ykz 2dwżDD$vMJնv,pq*u ODDNk+/h˽7Y1N aIM5q'5t F 8OwR;վ4|${:պ:3_y7n7޸qW^Zj5yh=eJ7[s4#~f3x[Wp'1(Fо4.V43ͷoߣ^8$4 emm|Ў3g =o`<_.Vڨ#2!{ׯ_W}ƍW_} Z !'UnXyyd=3VF%7m{JRf:F<>1BwziG4RXOustwTic%b/zb& )b2f T۩n}qh>;*Xz?VH랗VmdL&p6#wMQa@Ճ/),q\c_j5e8nV$(SU2'o yHNE= %XEԩhIVَW&N}&3:@mgfhW;$Tq* ʤ 37fTT* j\L腚5qB%j/<Τ[]p'l^IesjSJ0T;|wt9s+yKWj<&8C?fr\hs[ip p~Cckל쬷pʕ+WdVC7\Å9'M8pg wX+q.Vs`T,- G$sa`n l +i׮];zD= 'zOI9G7 p sa`n l m 076C9N&'s>|œ;-?gjn[t]ɪgWY#*.oPrcn6MEgLW:;gt/4FۙbvR38AP+1X3Չ/ZG]I\-rUm0T1QtVe+{'bIW*=dJot; UA. AG~+qW~腦n}3_ UeY4/QR\PY޴ˆTH=dmXW3<n*2Sh unĖ͞v69mMmf%ϧo9[&eϹREW0iCc(=kS1J|8B ;w\ )V4)>|5`I1,V[dc޲m]`lH iE(Wqbcڵ-_nMv9nwd,ղ&]S {J&y:-tF-LW2nekoϨ}gٗ־F.R]#"K)n޺""96I'.\ԲO-cl""z8]l̞yT!)ۧaR1X9׽WTPGx/NymZfY{tj2V?2պ2 勝}n:5թkdw̹03u*l-tKW.=!aG× yhL:[v:PgTaY]qΘL!W腚tG'(2;-}V1ݮ5s/)HHQWꝔ w<+b㼒Ϧ/=x|is{nh=7 he7ֿԭri5c?K{J'իvޢN2TY5 gsFuNF]KqV13BMTD|@;;B'>Wz/wRY$"KiV7=v-Ut̢jѻE%*Y)E_&j˂u=&{n4j'5vJd-jdm/mX VG4*w^'9KM~f'~T.Fr]ɭlݎ;N*jI>*'F[`q%|-Z̈{ `R$*%]ق6@sa`n l m8GgfM;ͽe魩pM?l?˽/KS/༙~_zsy~qu88;Xԫp6M4_ƅ~u'1w%"z_.~BeyɆ['hw?6x Ɇ'>zghhYgFsc͛7oL'wg΀%Wٶpʕ+W$WC]WWM3T`m}87.8]~> Ȧx7o6>Ç?xhs'~K-g3ئ?O3S>5 UKKy#S`sa`n$&AmϵG_ @D̟Pj{3 [ߦ_l?MxbJٶ/f[ ow" UU 3kv٩ W43{R` UZn̸4?WaĩC nUƕp l,5cLW:Ynxpz7d1ƚj34ڢ+@Z,R&%.*r\:vWʞJ|pf%lVoߛ4T&{_ԫżDD >JE2R ogi[n;ɻsRN].|t%3fhdqO4 "v* !k#JئvgU$wg.j2,VުYj]Hf?87F 9wlgN Lo)\ͅ1& ekBmꖽBwۚ(ԶT#$~RMd bU2;o+˂H& /uNTQ 'ڮN,SxRk4*u:O UvЂ-`w2j5s]N(͎N5If&l; UjW Q1Z!dn'&vP‘'94CnۿuFѝ"~ck7D^&5c4j 0"ZMFHbٙ'xӐv+իR>Վ_#t鴛f'F)_l`]t%٧tۚ_?~MX$$[y@|i; \2Ti9AnmY8dJe4jmj(@;f?Mx,RV=يSOzUm"r*R`!:6[ne*zX Bd+}Qo&ԄLFrS?73NP7M|qbt*t.>;"5g%dg+ͩ_K0V9"`x0@K_1 _?L}O< [gÿܳϿ[WTؾxx?_m,]_~yKV]ZG_ы/{ 71 =Isw C _^Nt3)z,/= ~`۽Un+[GXEkGyQjһm8{52R"K<$yZQ;9oЈ19bGXMbdDw6.oz{^vzAx^o0pv))3*"{iN*'.u.1pӷ;=ƚjvB`X=ttsZiuV4Tq{:hZo(l~Av ڝuvY-w^!'bBTD3mOhżzhτTah ۯ~եVV..,^^}x}=dg"|1kmG]Y|>pB*Z졥Ɓ9hVvY9)Mj_3 DQݏ;9s_gnj<7yjfsi8-65ιV/MM7Trfď cjr04UC^;{Awָ~?n!>wYO5pt=Օ--~g:"=1휂Vi3"Pە<{dͣ.M$b~) лuͮdGj$n&H$l8-,^^Y|y啅ťԓ?mF<J-OY|⎆*d ֍LK[˶L'7<2T|ќt? lZz*rg,֌'IGU漽϶',O*cM[m/mDRJO 59v~+ ޻18;wwCp/0h/m,W(zkx$2y$ss ƯO]Uxas֒pΒ|:)RM4ԭ2)v26W&"syJy4TYlfƟ:|{h,+pbUZ24GHqIpOXKi:bԟ!HNl֘(zirMɩmll8Kp?ޫ[.]XX8ƈ Ņ"8|ۻ: o'FOzd#ȆyT6@<3qL?lu_{ƿIObIENDB`llama7b_autotrans_1to2_predict_distribute_ckpt.png000066400000000000000000000156131476650141400400210ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRnSsRGBgAMA a pHYsod IDATx^ϫ5YoB7YĕlFE,BD2daȀψ 2\Jdpvc4}Q2!0Cfl_TuUW>zOWuO=]=u{`xw&y/ FhE/L"H$D"ŤID#h&Fhh&Fh@4a8Op25׬]~M_.b%wݝOdz?o8cX\>ڌ%ט75]ԟ {$o|x%\k,{bOL>,/Tjk%xJ頄^<s{w4%rX/3K[WD0sijf/1^k\ x_O6׊ZיKx*{~ ݢQAA d¤tzRG\-OK|RhB8L[w->NW(\eTUJ:naA~w$vhLTR_~Okqt(e_M?Bm<%äNQB<0mjsWca}{u\+>k׭nخƏ$m.@h<.UJ,rl hIO]0k&ۡmX{d{;T5|} u( Sbf9PHGXlRu=7i_V~Pvj䒾W|Ph~6}>\yzJ[Vо{Xt9;R'n6~؄>'k>~{)s6Q&Jtұw*IV]3|Ys.&aWafqՓfmd}bX'(gW]|6JSwT6=0H>I哊g9Yshžle(٠\x,fuIg=r[K܊Mͧ\/@}K/ o/29'u4}QOj gQ(g"ru/qUV?)W_(ly}B[;e㢬| %{GK8s竈F@n' ˧=],+m{/H%N@> O=ƒWVIEc?J±&.јLsXk)va{Z_\=틞@帹)Ldum OG H6&N~m!=6JƂUΩ˒KѮ!o[;l~S(]>;gDȯsHQ>جkuv1.ؘ|,@ 9(t8I LrɺP;knrާ(ފEs@'5L}5d&>jo趟K).f>\'㗯S'AMtʹ_1Uýv0u*6IAkp$q5}2Ķ?K6^[zH1V M݇δϞEP'j`GO6v6H;9eMէ,#9(ttx8'DElD2$^ӕ2~sMMdV4H$D"H!ih sۄq&F ¢qMmh ,}ۄq&F ¢qMmh ,}ۄq&F ¢qMmh ,}ۄq&F rE~8;9pص? D<ҌIfo D<$=C)Dscn )- F<_'9 ^HK4T݃n=x8ǃzzC@/sf#2 V9 $v)̝nB4hF\W+~}p\Ţ$I{,waQABO`"4;^ъ=\Qq ΅=) l;U?^ǣkk.KEcq q~dQ㕔blO<OeQs4yB c,}Y9O1fZ!@1C!Fc$G3.>\ItZH^~?fɪt^IdnьXb4Y'عϋyp"),ufca85I/t.K}B1XފIx>jɒ?s7O䏮 НCұ. ]dsݑypڌ;ei#ۡ w+H\ug]_Xl-nt|IꗎvVo̍hKKޖbwC1} 6a0vˋƊsCzB sgGu=EZAtC" ǯc{.kuOGCmߟ`<1m[1?s6. jB#7l]=sǾM`-tQmnϫ]3]^s/2j;v8W`V4,#o<C|}G*F<:V';gn\+U<{N6[{3dh8ٜhy)]:q&F rE.vK6a\ `hAnu 6f6a\ `hAnu 6f6a\ `hAnu 6f6a\ "?{|o`"_>gom_*q}AVZ"wI$uຬ"Kb7~Oyur("C UD.7 Fym6uihh&Wi9TDw:SL4g'yWD$G!v#Fh|7&B7I[.Ə$mຬ"`  #?p~Z^OYG4~K`ˬ#E ïӻ_OKh,%)/?{`F4V9ǻ&~u`w_ p82w7X[vOmH%ryѾ^X/| #q ϖ};4FhS :W3z W_3on^б`O IvAXzX?Zg˘;.] Gd=kҺﳑN塿QM6|ifw;Ip|enn[=~9<]ݧԶkס,m/lL=e fSٽ6ޟλ3iT_#}8(|ՆǶkm< W4ڠXTz4TGoGF.c_cC}x@Lʼn؏ߐW./wz6v~^}F$_ *ypem722Nu~Կ?UY]@ֶͮfM>:.Y{E΅utޕ(ƶNV9^;ݰTma֏ebO˃n5)w2/]-K`E:~~y~֗(U6Y#u06+,.kbht=K2qpN,^nQ꿴SmA6^f{=|,{XwϒuD Ëo-KF^\x7^oFP#*],5tڕ?;׫H֏VVcRwe,Fn˶Xfg^f{=%$BPN@zR.m_sy.|*Bh2E~٭2$"K(.uheJl|cy?|l| _~?Nc0zY=-ȓtj۵Xd62i^KN뼭VjG-qPv myدhAU7v),i`GBGmavqe׵k265WP3fֽljGol|+.P2~/*o߱ITUpA'@ǮcXtTeuaV+XZ7dg9׽޾I1u7tB?e8zvR-~[?^]<-Rغ۫Ԥ HK8%M6K>Kr[\(eJׂ]c0-]s$6++ž˚!c%vT~N㬿nQGiۂTmz#?JY $i &" -JXG4~0x{~ֲ$mȕH~u;n1F5w"0i4}p?|k^yj+h ˪?bφo#\ƂpMј G.D8$uUDrgM+|)0{]WvDNEѸJ4nN4~a!R)sG',u7P[ j3#)<]q\~u:EM՝5lg< _: (;&^kCx5FďƯR?-=쳊qzE7s曃Gs_;*;cR49's)}7ol?(KHہQIeMl27x't|L|o$lUn~q־& 雚(u2lLl'U/ i H gɼRNv>*>͓%:|=a΋՘.zt4s\[`2p-cg^f{=Voïc>Հ y=)|Fc} mMSi~P ײ |HoywkZJek]ƜQ$t ߥNXl FY+g#:8cߝ^7#l_EC`y|@|x`־_9s\CvVL ؟+/]g'\SƺY܋d$(&t~.#1 g^ssz}Ѱk翱,m/lL=%,M9+c}u>WH}>n8_~:Wiݿ$rhAV7s8_9R͵|.-zkK_av!)wa&F5 0B5]iW} GC}Ϙ_'r!/K֧|.g[Y5.bp[liϾ|y=w& ڔLq s~ٴQD=?瑹Wa6ێ{(@2I&y/ɧR2:lNQ̎K3o+kv^os~cXo\iW( _naMږNlU>XJ͏ #<\r1(~!Jc=Vm)`oPo(k~GNR) >-{R?/uDiN&>pcݱuL幀=r)iGj机++ž9-c%8'AC_NqU(Ty[ ^Ebkv$5# ܂JF8nK40B65}c|;OYU4~k^y$uiF7Fhh&Fhh&Fhh&Fhh&Fhh&Fhh/Ζ[IENDB`llama7b_autotrans_1to2_predict_log1.png000066400000000000000000002744131476650141400354710ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRzk sRGBgAMA a pHYsodIDATx^=fǕ|a`"%Ӗ ,4c؉6j '# 'Fn4&0Qp~vڵ^j/ٻVZ~ߗ~gN|kr&եS|\|}u07./_r6//4-gNFu>L}Os U^r-ggoW̝=Lr1|0%4/o{Ό^ɴ 111 ]-߼Y޼gy~j'JA.o=Wwm\\?91?Y>.|h_|HﳖDi+O[]Bڶyv|Ȥ~xGuu8~}磧vx'8 v> 8;Y`(`/_kUtad_E z/_sbpe6&$1Q@.e!n抐 y)&F.MO=Bw9CmDoIwדx|j$G~0ca04{>$4~E*W_b=1˯XY?7O 'vI]v 7?bz% עYm=7EʯZ-#T+5FM9>X Twy*V옾G0ߛ$\w.1O/^]/<%`Ol~ Rz$'ogM,_u" ,6kDꐮYL8Ӹb ّ}"l$.U\"n ﹈#ZLݨR7j> *Et*i-9O*_E)-dBΖKM'`ዯ߾k@E$ϓ9\ۭnÃyΠx5i#rlʉ@3-g1 QW⥅IdmYb_ &Dnj}h6G?\ >>b,LcP@jԖpx +̸bGz'Wx]O8qՔM̈ zNX ӧb0~3∆}$\cM-t|:_9KsגBdbpNox?3Ak5Ѻ`N `B Zd=v $`x#u]^\\cqȿ=~S^wG}u? D/D*x$U7DL=m -ۊ0f!(,LY1>,o8+4ؘQxcS6+)eǪ#&2u4_.VK4_}o6&5c~Uu_n1 tc 1p!<\B2+'/Y#q2-RSB﷗g^?OQ~6A5Q#kwWw[sf,옾>&25Yࠋ:cr!\]/IЂ UE]o7i% 9ldV/pO?sbp" AcaAqb9m]V_._Y^p$:{Ʉ6&E,m\27!/"E)!Y|UN:g-]M"zRO?[m;~_|}bo~f, ƙfO_'P"$+ 9Z2B\}k"k~")I)TImyR*ZeGյ F5~BPO _x^ NeWX;+6r|Cqmٌ)EA2fi, AM :pN18# pVF|blfme[54t^&`\~Xu Facl|[з%亦mb&ezā3)}m~ei8G=t:^ulm0$WNǎu",$r"5'C "e?%DK~{y`hӊTZ1b]qwE߽~ո:gB<Ύ`bBD"& YzLbH`)⍩7afm|lYeoJ _&~XDscz+pcեk X۾L_yxYb#Ƣ7is [[7kJ*/Eo3Vw3Dz? ѐh|__>(wRصB+0b0Aꗽ 5I;Ogxh@ pK@ vq~&ƈD? b0< b0< b0< p\/^13¨]j\~Y^Pogj`N O ~d+};Qڊ='ִwm>=<=2!+*QCj_). 9B4ΎsV1.o ?X>,?_)=|~!yytZ_>]~ig!/EWϭ]{~5l_qA ^\ cNllKvΏӎٓOv v,U+Wn{];p:frg*'&$1Q@.e!n抐 y)&F.MO=Bw9CmDoIwדx|j$G~0ca04{>oByɢm+I /_WLC,~O侈Iuܕ v߻mv11G,i܈\EZʿvM@S{mggww=#Nv&wٝΏ]3L!*ѭ'^~-Y1֣|SYB,>Nur[#jlt3PL5~'`X9Ύ} r_ 7/E\I4B_HŋB@<5jmy^~oAGuM![%pW<>!1ݟ6G- iV?G we~\v?3ϜTW:O%3R=^w?<]Nu~/Nv\ST܄(:aCcM~/ҷܼN}]Q1R}Oq\~Ljis϶;~]Tݱ'1,ǐ|6eUvUq6H塾wm˾ʎޅP`Ol~ Rz$'ogM,_u" ,6kDꐮYL8Ӹb ّ}rHrJR^t^@rvɾI~|uWt(Vے˘ i'`ዯAY"p$~H폿)5 O b"PoGkg׻a vA\*E_ %.<^xy*/W>|Ke[\Vb#}woL/}7E2{FL>3)$d#hNJ}|3ǔq'IKkO\ 69lQmX?usTWC:﷍+Ѯd[I\<)i73Okyߛ>߮~;?F}=My/MlRUlxk.?>WihjWfftg2fn߮hV=xݬY<>ٜUC1ÍdLvkL)wEW,E,<\4w@ a;5\wcB1|h\b߇ 9Y#2.=vw/-{!`e;~g0y7{=[^NJop$ȏ^F `;Ne09]ۭžٍq?Jq᫓焄p6;>7Kn:GsEѫCظr͇uxX$̖kn«7T]Hfyc3S?X?ٌKaS.Uٳ+ ]w~uvIeJY\pW]Q9L}/.SKu^]~Oenfۣcl}xVX+uv5N]9OmutU. ,9"#TZ|1ꊒX )7Kהț8MGqvL_ (׽Kx`3rQPy;VmVS6\#R]{'g,oi1V3؁bZ/8R9 ۘ#yf;yNFǶ. 揻hFs,}\cC^PَmIL!TGu#-Su_ؙ,}2~l[OmgBN75afxVZ+3OVKy׺ yT~/jO/f03ߊJ*#Y63h1;mخj[c.b4LLhT:NB Rt䷮˫üZok,cokO1{hy h ]E\\$$p}˶ N\\Y)W׳bp}`Yq3+{ѷ4V^_h}[b8P$s M;1q .~&:~z]5/^g9τQig8Fu?m^w~c>ʚyӛ#3 k0?}=3_Nߩ-\7xGC~nK߰ve7/ӞK3^-}Ԯ8w2Vjݬ/mfgЮbUڗnfcB ."F|2貐ʉ.tZaO~ ?d[ הZV汌Yg O?fʯՇA?~84lq궫뷝XmzY&#^VDGCM { nEv}#ww4 /y J8w:no>]?@ 7HB0 qb0 JK ;^@ o4g7`(n;)? O{n    Ϯo,o^]<^|{\x}rLpK\|YCr9m&okbrKs兛f\.quNỏ}\qtGˇ,wE{x|'J[|ݚҶӧ緛G&=${E;{H;=;!X1Vv휜U ?˟.(WJ_-__~^/>]VeOFE!#9R{眶wPo!lmߍq]` 6O ߯q߸}}{vsow=17J0!'Q~$Zu抐 q)&F6MXgW!;霡߃tIfc7Iu<]>l}cV #eQg o瞾Oxb U$1|{bo_2 mső\'+$k]:0i?/?즽'x58:U 3L!*ѭ'^h~-Y1֣|SkYB,>nN"5FM9>X Twy*V83}K݄/WNKj CjĻ(zл/+˕*<5jmy^~oGu.1ç#tr|w~ Owq4hxoƞdڏ:rriÙp~ mm뱽ߖf||N ŀDexL`mBʛlsv=1ڠOmJmfTD~mq|SmoFڠ_fK 8b 瘤g;Kʏfj.WƗ)mmi5 y[9<h1IYh3˗kݮ{8c%̄S?;.0gڞ2vrNrJtpח%v9.}!ˡ|SP妤ն<|PKLkk^:k׍WES//ʟ *6rAFcIQ/7#lZz}u|+6f+31ƷO*T ]ꍽqpYM:x*/^9ˉ_9~y)*T]D},_.6W/6J`}+_O f (V3bpZ\ь?7f_ve.1䙄^;GcX#❉'{G*Eݿ5pXǷGS2~_bF>flGq gcn؍q4^[w;8辭1bծ5nM9ƶۻc)jL}cƥ9fƔ"o~WĹ> vFB]_bEQQOy~W5_̶U\wcB1|h\b߇ 9LUD:}#ء]ˊ{ae;~w.&y {*,U\Dȏwb1;ŗ?Hvx`&鍇n Jq_$c:Ue8sj;#s2Տj+#pRmOym9>~ۼ؞IcqzF g'>2Q̦;ǶGc36S@#b}KIQcD_;Yf-&qi;co/xLYs=Gd\QR$E^_f~1y3j]p1~gR8޾}Cyøwr/ZL~/Cw<[-~[M΋]{NX ӧb]^LESϨO#6hLhbf;y"_a_';;7R @6csB)Bs8wi\{l1{Hcx/֣fdG-ݏ-v߃5>Θl 8j ۘ}\`GQ8gJ/jΌAP&`E8cG!WC~뺼; %+׹z?֑{*XDӋMNÅyءK{2yȧ J=K@$ޡ\"g+]Fܶp߲-R"]|0!Sgѳr1C}) 7ס/-1&5m{0el[4&Asool8'?nm5VAM># dugھcҜ gݑmjW)[ ,<;~u;lo>?p16nsrz)#W~6qփm 1p!<\`B2+'/Y#q2-R:"$};^,c%?Yut#h]Cc@vwbI! w/rN#v#Hryvqx= *N^>pwob0A׻ ՗Ip  HW x % {oƈn xKcʩx4~! 886F]n[Ԇ     Mjyե>ͷW˅>'^/WT:}ۈ)8޵r7NbrK)miƯCX{ɜŋ,>@y|/>zWYKwk{OX߭i.!m<}z~y>{dCBWT:SJ;]r;|lewY|=ϥm|?篖/d/?/._˧#"|Zp&ax1ԩ8pvRai^.4׮חNScΊcɄDU&hEM+B&ĥYH4bΟ'{:_|s~%͏$r'tնGIX'GaB/{>8@m\Ib.bef <50Is #a.nr_߃qxRl}Y] ru{/I lQ)R~5,o!@7B'Pc˦qb;<`%nB N_]/W//&+_koe^]/;uէPH .teeWmUyj:uuw/eQ~vݬ_|| ;kǧ\;'-[1G WVQCT.>ZY>w8GI~}"|ry)cu?O~vg[ٽzGJ(Synrշ27ime{eZ^*fbMoWo>^}-,xE^-{b##18 ~?+mbfRw|Y|_]1Xpq`5\SƂSS.Y.υnLs_rq՗}kPr\.%%I幈a/[]EYn/*د|wVLXPb;rCXeD ƒ ]7#lUQ>[\\s,}m֞G1a{&aN_S]SrNv:ljVkYi1-iROC2~]7W$9<ČecA7YcC"f6.v$z"](-^ OPm"i.=Ϭޘ@LxiqIχc<@758k׻Ir e',be=bowU!^ 4k~3eY"lpE/K_zb0MD̞τBȂolJ*Zw!|4x˜Y{¾DŽmON_>$[~L*y߆bbeoNeNeeۯ_^gZVq1 }fƔ"o~WĹ> vFB]_bEQQOy~W5_̶U\wcB1|h\b߇ 9LUDЧ %V.b]L]\Eչ؛squ/¼gҚߪ<^ZeۈH>zM\ `;Ne09]Be}@}Rm;}OIq៓`I7[ά=a_cpb\59έ]>և~64Vfj}^]ZoXZLUv.;F:ьljSa7bpa*Y9W⥅IdmYb_ &Dnj}h6G?\ >>b,tǙ3ob.{g/\z=!s.lUܶm5e;/ө37|~3SʯgԧroEؒ"ş;1!{}BGq!-;]3E~ke$Vc5mҘoc٫ҍ f3SyghYʶ޻ȇa9Sbp(Z0"NS#w!u]^\\cqȿ=~S^wG}&o'B,o8:~Y`3g2! |zyZpmy0lo)Ouya2C4g 1BXNeslTLþDŽmG2Y׃5EY=S^v赙5J;"?U?)ϡo坈ˊ 5F䷘#)̶ z&"i'p1 ɬcdɐ"gH鈄 ) OѦ8c`m{|quΌxqzBp],Ӆ\h2_*]ާ Pqkcunmcn|xZk ? W t"iyVO?//yM177 @ 7HzB0 q b;˝Ty_[ _y>'_:oZGlsLG{hg1?ֿile7atF&Px6x\<ȇڙ= DZ."O o?833f3}AY Ԛcƽb8؞\)d-˿޾:jKs_b0gǢ!_%7-fe_vQ1Ըm~8cʬpGBakekO%nh+7%s"16 ZDºL{x&?z_+W!O;iz~@ 4QW1hߕ=T/bp~7[_^kqA b0gVhJ39k=xB[|= V  nїC1G bu_o?8#UOd!O 3iHzpYmmQ/+7\ ֺۗ&zZ]OfN _G_1͛ͫK?}˛o }O^tbᮕ9ޯ_^w $h |@k 1W_NTx"kB9]73 뽻b0KC~8~{bN O ~d+};Qڊ='ִwm>=<=2!+*QCjܯSخ6T{u_Vft<gYz}>?X>,?_)=|~!yytZ_>]~iF@¹^.? UM0#<2G[zܬoK«Rm_be&f5O\D_R4qJ sE(܄ @Tv^շ'{:_|s~%͏$r'tv^gO)gկ*{'p|wAm\Ib.bef !׍5%!Cpd}pNV0Md_Vl+=s&`V&E~uG_:zs}/Vw~Y \X\c {b9KP(/EX]v#AwgACbng#3/F115̫)EZPQ%-dx6q\ E`)i-9O*_-d6ҵHt,?|UT~˿D&Z4|pU:jYzf#=y.ik՝JVi_zc~'Z[̚FbXI5FUkG0li {59' `',֖ Ot{Pݎ܇^վ)_|[)8vc$G' qc"P0A $,ؒ!\xA ZPwT ;>|KŒ-D"sE/K_zb0MD̞-EF^sMEٞDPKsIn~2ʷ}e b^kQlGM>v֔eW?E,@= &|3i<# !wŮ#r֯eqrL ַdE$UHqj1X2׭*mq4:({bpOʻ"3bpn1%DSCeZ b#:b P% CaI_ =b_ &Wه&ns%R Xg$1w\ VyŻ@,!ܶm5eO={'g,oi1|WDWg~1h1,&Oe_9KsגBo|lhe —`byz,0f57A}S@곂u{µ.[ ?W춯aEbmKb}MϱT[;b[ߚl)^_a1%0( ]v ,kUߺ.Bp$ŵ+b`=mGS}~LqԖ3`!ŖTfOHqDdzJ&HA+me[8S|"0!SgX`cFAI NO0W8SˎUG@d1h="J]nϗh5XvdžQ!T#j?11h,uD?r,ne ]xb&'*|FdCS3hצg3Rx\/=Z}qwO5_s8bB ."F2貐ʉUqPuY\kȂ{cXXyQh H)eE/+1bpm/m,8N EIuA["$xˮS=f<ߚa| .Yxcy|Ynv8o]ۇїMT3bp:׋hY}:<M1 ^8fʡb7`x@ `x@ `x@ `xܲ|\}fyI{\߼Y|{\}.^^ެ察r9w?NɄ&t Ĭ(my='^]-=1%xkC}zz椝R } Nʴ,pk#Ŭayw姿fy%w=bqB ``O*_VIlZն_c鯟wϗ'2y#+L\/ԗCBb)4#7e&7#솢V7z~MTy҆'>PuB}afĢ+˱QuT.E۪4XN%d+׳E＾.2a|q +$42ķWތyOFA8cDolL.~]zaOPt1RvqLӱQ!*+~_ .B#._q-j[z&DL.rF w\",ԮFqik&Do3zX-nlgս >ϟ&[}}\~'b09('˄[[ݺ?Bb0XWa鄀*]Ir&E_ {@@,H!ԊRp͢l7_Zw_b~28o_ ւ&ێ|iD~ZyD`~*kP2o/Ǚbp/V u~pB"LT*܊ Bp"]"մK*=ϊTg-psA~-iϭ/2n^1Vee36/ާz  1}b0< b0< b0< b0< b077˛W~,ϯ7^-dx\=S0m-9vg@~_}"?#jܯr7N; ǍK;̊}\qtGˇ,wE{x|'J[|ݚҶӧ緛G&=${E;{Hu ՆjlNw>z?YSY|r,?]PZ><_||/.?4hM8t\}{ܚpNqub}}{MWw^w 18L$ъ\ĺMK7hJeZ?| ~%͏$'tv^gOS[[jס"]ƻUTz[ɢm+I /_WLC,~A躢 `'ؾ>~8\ >3+`!*!'0VA鲀9+zo_<[G2MnSWnkC_Sy-P'?K^WFuQc[P!z|%]O5WW@sKWۗB^FKpZHp^yN~]M_gۺq5//{%Km>~ZU93W+icѳp`-s)#18 _sf KݵnW%q0v; ˏ [D!]R@d#ہJKg,غu]%64r.dEIzywpR/t">r/y]sն<|vkk?j~/?X~"ϋ ˗gQ7ִkV\w:uzM#4 ֔X϶2!%ƧL]/n̶zȎkײQ{=^mw}5Zbgn_gjs嘾涼|'u|x}mmӳ{b@n18&mb# %-^蚈OkBf.)ϡ_[ϬCbK\㙘RuqH sc![:.iإ.] `/l%%ѹs.<RVf%!A٦ꥷ w~'f$W~8Q,gd*4AJ$涱;)ֳZxEL;q'횏Uڶrl_mY3X[X[Х6~ |kX_?kʊz@uwvMGVR=hW{&fO c:뗏Ǚ_g?/ Əwd_+etyJy` c@i18ꈉV,b]O, K}/\N6qkJM~|u}h6'*?k[ҋ)_sڵK{]jgfXgo)p/137 =|3T^w-+zxMEX"yOb;^zF:="عޥv}Nea"^6Wp/۳ᶅm\#dB <_-_ϊey[Yі$'9C+Q:Q6J>Q s[h)!ic?VZ)z\BkN_H/!_DhζuL6?1kzi~ב?n7|g֖ĮgOn3;E;LK[9fGc|Oqߠs۵_v}Ti6c΄\DGd.ey여P/`3/۟(T?EV\Q@1xv$c٧rMh,q#Bpbt͗C:3!+cmϵz)L} 0nԋ7afm|l7i% ldV/pO?sbp" Aca-\ivvԮj 5o_d"8G}WIΫΚg]b2ltLj_Wjk_n7fXbmgIו\ߣq&q9nn66s^V=׹?ظSb.ۼbƯ} 6=nqclo~~bQ:ی:r 79ɂ /p nu/`o`  b0< b0< b0< pkixNbm +ty5/mOo7gLzHwTZ ^ݗ> A݉|o r,?]PZ><_||/.?싴s]2rxs-qͭ#.syᘓZZ{>= 3sF'\oΞε4o| ycʝꃛ#÷/x,ܽbc}\~jr QD >4 1&.tbuڦ}ST]kzKk|7fzvɵg\uvyUoC="x,HY3g#~ưw>FFu/:}vWU_ﺨ%QZ1RcXؖq32q#ױ]p8tˉuԻ`[~LZic73Sݟ1\N]\mk9T'-]To$ۦ][{[~?,|=:ska~`cPctgf~`-s)#18 _sf KݵnW%q0v; ˏ [D!]R@d#ہJKg,غu]%64e!K,u xNlPxe] ͇ZwO`YKZmKΓ _з5(.7GO _x^ NeWX;+&Jc^b\zr6 E}?df%y錥S#]G^:xy2}?5k'xv)*fQ_G Y3fLQX1lNI}dv)O3^[Mynuϼ<=Zꌑ(fr,u8+~Ōcn.;'ut"?y_u|k_ucڎ뭬}o~̵;X&3g7m{q;EɷGbǬQ%*1Rcn}!߃9?#5PcR}1ko:AݪˎT3VLV~f<<|ݎpع&k%ؚ?F ~. Yڷ=|{3=:ӳjn18&mb# %-^蚈OkBf.)ϡ_[ϬCbK\㙘RuqH sc![:.ibVX6]̤dʄ3D/1/Oc7gbE]IYb0'Nx+1Ll/qmUƅ7|徑}Lf˿Am?ayج?lulvWRYi>f#l_u_;1D;پC;kOO61>"Ƹ3b莅cʒ>kJcjlT ;ѝ_<&GM#M?fUVgb>5cƂS,d+*kr_F1&>hmM~(]f gcx}ˋR"_<Җl?+TfʺL}K0ݠmba@Q诙8~DsSq{;3 '&_^e-b0}!\ESk!_g>o~ۅ7fV1KUeqb%D'RTt_kŹ,<\4w"6ޗRpߍ!as$R t9Z;lsbpXHP܇ΝAt]*S,A6E]ͤ$ξO,. ^޳UaioVJof6"?O@ H_ ]C۵SĦx>ڎ3^o&r+M9tƚ;߼m~gٶ~ʷN(?X4>wez4t[{ YC0LGw,8Ec#MvҗLׯL\~>fnZ^jsL-l\>tIuwvcFkV8sK?jW{&fo}po後5;3}m6Yy^!$ےm\3[׻V}ks#a0ZS[㲙Ⱦ^ hakC{}l7_`_CmލQ3ѫ;iI4XbVBs**[{ELOm3e~]ƖiZm2zrޗ$ g=F}Q׀+oZfw/q;-gT1 E뉒xɰ/X C~M ɏMDgmKz1>'T̺(۹,c[Ԏٺcjg cA87)?ȧַ+[SƉ~Ɵwqb|u1_m[smH亢cax[&zW~bHX "G ow %m4o%ȶ7n|Mϗu@#c]ƕӮ83}vaG(~zvRl^)>AyaGQ{}(M/'>foַx G!*ɮVꎳ_q\F+q6Dnkv.f;"[_JJ;W?W$7goik?Z[58<<ޚ2g9mo=L:sSU}w6f۶ Tʺ}ҤiidZ{yFR6&&ZoQvh cf|}M6m9]2gO|6fX0e8+Fo=#r߲Dqk79Fx5Sa34D?sJl0f'~L79 0N|eht#F?:$TϝؓnbaDj̺bpmٜ nu/`~0>z9pC>}c>T "R1_ }=~3`QLd+};Qڊ='ִwm>=<=2!+*QCjܯlX~磧,ݎñ0|?g(t7?˟.(WJ_-__~^/>]VeOFE"/LjI}MlbAB `B NH`B[6ѯvPobd"єT,D>Y*{'3{Ж.I.ʗc`0‟P#F¯Q9+zoϗd fSוPcmDT~$JD5_1z|q%[ EdƈiI{u)yWZ &/HdbeUXyj:uu/ Mbyߺ].t._n1/0by60Rʨ㴊?'6QvЗև W[*79 m{?^D7殗h h-kH,풿moRw|pw"gACbng#3^$='Ԏ1ݟ 5vw-dAImax=d(ߚ 7j>vE)i-9O*_,6ҵHt,?|UT~˿gƂ+6>dL;~tU8l>xcXk.m1;3</YU7lN߃jff?#kJ`-m %-^X(&T,Fgo*]}fm"@[.A%=8vЄ^֟N?aeWXЂy铭!ߴ\ &zb+а2<8"D5sDP WX},_e[H$HE/K_zb0MD̞τ׮8jĨE>O0Ʒ|_c۵He{>bx9ma1߆>pogl CޚRb3Xv$,*70W"b_bÅKQOy~WBxl_z~K5~7fYA+݊#q|cmKf!9Զ XK1{o}Z=@ &#}XR8 -ΊS\l9"^޳UaioV@b7Bb0Cr2.Wd>bƠ?2|Ə~z/3jW ՗Z^C芽+dwe$DLxCaZ B#9b h8Pd*7KהX:+nV#*pqCiXHQ?p=0q$ E0bOr1X<[-~[MNqD37|~3؏'bZMo4#տs x1K忹\nn3l36-uhMQN̼1 `R `B jep}}+3폜_"_e5ѺW1.~1<&":j(p/ ɬcdĿU/)!Z3/۟(T?EV`1xv$뱠׹qi)]_ aYfo{宗$fh!'TCכ0u3¶De5jDBadV/pO?sbp" ƓOO"<)uW6W}_l1#A$oi2Uzބ[WRCmٶr͞m-6Cׯڥ$mM1!bj ׺ur 7% R M1111bo>l?]>m\߼^yiWo7;/_/_^i[MZ2ϝ<=jW~,ϯ7^-^]\zig$-CXno&G} ߟth}y 7=#vjR]Pvh:>eѳMhC9ƿ鷅Y]b~3zM)z .|UFg\.fCrGCrm븊=p|~>_O]V7OS_?=)8|y;/oy}_cy˟ayې֭ۖ9,f6aCkh]VnҦ0W n'kI̗AC%$5X}r'Dž?P<>~ty/eB+$ E_MZG+S^^ouҦg%Xt]ZY^(lߖ'^6 zB lBf;=HiuCz|[mATw;8SAԎ^D|(uJ/EMlqɱjmҾfy $iܿN.hoBAM/gUsG]sP/>z7]_=+!5$P ["h޻I99=YH֘t|(}dp{L}'wr\ !;v#g߻zqW.:R Nk;ߖ;mw[8mo rb9xg=`_18z/E&Qx˧`5J:jb v(`~B| 9D ~uµ>_?wMu/ҞpX7 (FNT]-%No_8QͶSvd̦nZ u'Tꐐ;7I/KǶmKv_^-eQxy\=W4W'+b.ѡC{i^qPY9?Ś!*,멊]Kntݶq݉fҬ]MtgD鋏Yս϶ƈB=c[iȇOm'}A$lzϣ`_u‹NW֌//ֆѭȭ|_vK=w"ظuڥcI1IpFx_٘G񻖳֤ͭwnF&C&R}n[ؗBcz!{\v+/=d2mvpM:Okm8q_>Ơ8pzOfWbp]|ΗyIeUM@G ^mW ]2qڀt'nhMA_xMFFS_JnZe7Boz荿z8$fZGB!ɥlI;©n {C/~+-Of18X|Yzk]1-΢^JF!xRP^Q"Ub5Vymڒ7*Wr]ӣ/YǠ/Jg}uDZ,OE_ VxĒ}Eoa%=[FDb޽bpYCSzWuPkֆS|OյN}_.{fJz(j'_vqAFmUtE~l)lA\Nc8?w5ޭ"ţQԘ<>>GVF!IdwQ-? nlg^^{Y.gJ<{ݭ{.5.5&5дgwwWPD^kS{[uۦ%{Y{i<{:2q8dӬS/{ g`[?9Z1\׎y$Gb6mոK3Z6 O oVkE|'$+ktesEmͷH=$f$jja`=p6v ")XWNm1~V086 /;./3 lmcE罳K{_sRo 2NWLfҲVώN3]~MJC'̶+h{gc۩߾; /Oj~'Y珰5Z{klψi2?ðY-F\,t=bjŽG;+fe 5]q:?_ _*>3]6m94vM2޹l|mMdi PvvOJOa+]cw~?kyY_OTÊ>_cc>.fvt?=^gK߲ ]_h ,_ş]Nif̢m6 Ɏ+6xuXPHcX jQ6" qz{ٲ:Cb;Bݼy>)g'QPk5bZfO?i_\ @_6"L;fAem1O{}? M~Go,m^9ݴfٚE$景sS퉲\L(!zn-$n9+Yi:;3g@y CS9<1wGs,۱:nj;wEkUUBzsqְxͱT[cd}䇘;q&[m8gֶyEs7}sy,:͎)Fq@}`VA2mN*6ݘv\%-fvnذn=<ع'qlUz5hT}]g.=N!k1Y\#31/Su{A,%kY+:?cm)J"4qW[̗I!w'&1k dk[{]?z [̚c׳r]׏h m_g_; rsekOB byO=3/Ow~83Oy F+3g;?M8|_콣,r,bp-_ݣWm;16];`{ac"|ߊog:>|k_g}2kU5Gu޺=ZQp-{o%F~߶wl N]~U?ro=[1muerw(ݍY= v*ydG\̎)QPxY#{erғּ!ksJY{vl+پCbOGKi*{|opQUҷ;Y~8^ƚu?7^U̲gkDkhkz[l+B6ۘˊkc`WhbrPcd>.f0cgq}捯!9rN#EoTѿmQݜ(MonIږ c? AS ꍲ '8bD[m>Z!4_[vE<-*Eʎ "瑾vwfz5g?~ FF_" $b2F _^fSR]k6Z١MWAYYV x[ ۪ޕO-0be3R%a#hd+Jo nKǶM+(xByd[^8>cC=9<_{{ @~y`[{^}^1{s-֟_s-:پ:{\KStmm1ap.$v]/Wo^.#ƙI٨c޶݌a޸XƶDcœ\)W\p-➸{m1j7g~5ٚTe+)C/v,q=wZ v-n犫 d,?&}{Oq޽/x yfTܔZم{j]X1u{@׶Y]v-{!Mio|Ѩ:6m;0571rp!}ǙZN4i7y0fA{#A q弴ߛFBDWǤpy6B%g%1n& @,7G9.v1 pϘJbM1Dtc+`.18D|~kmƅ p< XBbppg 9. BsOE&}du' r7yGӔ _ b0< nz矖sy ;,O~ ?/jw ]uNab҇ZjԻT//ϱxψ{sNrNI\mu)pA_eKgB"3m")yEW$jwn-oͽ|SL^|.<m?}||\y\=,o0w6.^^^pȳ^e;yz.Y_/oZ. zI[;Lj`?%63'_BUi=bp.Tu\}[\=p|Y.I+|ݣ_Î$oeHӻK?oV` !lݺmsX*m†&v黌?#*^_%xu _I̗.UTa=j>tO ~?x}wy_&>ci+nh·ȣ֣k[ie]AGBoowV6磧'{<q{c@{݋r +|><1?Nwx[D>{?g]kiΦ{JkSw3:8R O`Nu1V~R;z1aF7*5}跂b%ǪIRHɎCW|8t!_(/gCTsG]sP/>z7]_=+!5$P ["h޻I99=YH֘t|(}dp{L}'wr\ !;6'oU_Z[wsW.7R Nk;ߖ;mo?5kPg1ChMkw]k%” bޟ.3?=v }?G7ԶHTC8R ޫ/p71[>-WO:jb=eb2>Xsj=Dj}~Ud9`us.baO89)FN(U)O$6܈)|;2fS~7RAT9 ߝt雤c۶%H ~y\K j>0Ug:NLxbr+AXhyhmCXJdA|VL难Y1x_lk(0V m|!?vxkd{Z Nbn'<eSNK?R^nҜ^׿f~yYր691"byz$׍ޚ}kZsoz=_zYTbTs/ 71Q!ʏ}Re7ۻm3[V VG?ʆhWJ~_S38ۥs`^y2)>w_‡9[?ٵ3:2|lʛ@~<,ЭqDŽ[}WώosyǬI!3g 9}Sοȶz}}^ުMƯ}Doe;W#W<-/L#հG\g OG (.gep}o#}_*E_NJ3 칊}w|Y 18_<-Zi+l z"%;Pqऍ98L%*cJ./{ ʆn K!vqĐH.e[H+H>붰wn;D{hlh[[-K^M0:W^!u_tHpLH\TF`Tv )vE~e|%[!F׭ %ݡhFZ+_їVc޾G-fqM §/+Sbɾ[ǰ#c"1XRepA;DGBV%~X֞|sݝuZSg6לzy'.ž̯\=3rPmiuyW/[;ϸl~ #ԶGu?n"q鏍GqX .D?Lx_1g^oi9k.&x$عgoNog]c9Aw܋VOعڴYuT[[T;{/6YqNGI-Ltl5O^Z׳r.6˨}SE2F].w,ʞ^ 91x\i~$_jqbHkG'?=7Cg9Z "<\ f_7{7 ͇Ne1N>xN1x[Hӂ,iqYYŮ!&ћ_Fnp qY \;l0&=:ͷsH@!PT>m}-/vCUzO;q~ ]?kR#;b;$ ?-c/7b8.f ꀣH~PBJ~v@trE7cHD&Y H)mT Qpߪ')ܶM`lg%&n,CHƃ|-w 5!S[jL{bk|6%ͷB<:pZZX5g}C8'ږ֗2˦9DG*ͱn[KNg+Z:]1I~[=;N; lLwM6u*U3ۮ= wmm~p~01b4'}h}w;VW0SV%\eK[#[^]w!9viFoVbęhpcbPn9㬗nYb x۹f=yg9v-]Imεxc@qM}|`q0gW9gOyy~LUYۙ:<Ҥ˳c45}{_As֮g^Q7C!ruٕNj}ZPe>2ť s}?V1]+yԵl~86z5\ϜUObuS])5mKJ %rĚW&f?D__h>Cѽb&fWD-UsoLB'_ӳK?&ru6 18}}Leߵo}㇪+dV6NJs>h凙~e87w]ƫ? v^?4:J[-~C۩]ŦΘ΍vi!˭>0ع'\faĻEyH:sA*kV|#!x$z&%c"X"Jֲ?Vt+~ƶ R0Di蚩8/sBv5NLccmZ(Hkz=9 xk~A95Ǯgm-y#u׻:蟿κ$v)F[ʌמ7?߅>jK{&f^4﬿qfOFq93g;;͟T%&;l |s{_fh"]Q TVAƴO+3!ތc/f{ZPwg+3k]N-ekXgͺ)1k xC~NԢc_,6(:~"_{촰Lquo{+jj_5oo)?xItgYIyGd{1&t!!1x8&6IC7稞[`ɾ;AiA1pCQgN/۶lE egK Uh }(IOep֝y춹kC]n-ɾkL 9 DvA;,TGu;@u9uW|qtH\39#q ֱ=GԡmJDk1MlC Pyk.G;;יXzkU5Gu޺=ZQp-{o%F~߶wl N]~U?ro=[1muerw(ݍY]sg;O?/Ϗ ;ٟ<ښOۤ|%dws( *׿4T*Q{C%Ըx> C*&4Sl>k06ź6ZZ9kqUeEWCK"mc1;Z _zn}м4`TzN&6?{Ev瑾vwfz5g?~2ݟ~g 8.i\ȃE7_^fSR>4{*  EHA=^BuZtBn+Y%-gV60z, m=zm+ BUh2sXRi:DŽka+d%غvދ?eǖm֡LϾn Zjok[ '-v%z2~r1LlFuDf e2`Ecbg;y˜\Ѱ+ws֮TNzK.Fh=v0&n~0ɘ>Mc=Y+FAo=+keQV %i+h-LyqwH` _k%5ږM/qGI~պGbpQڞȜhq[/VO)(68 ~/8LO(i4zt{l?̃ A!۵!g=voG ##1J8mCO >b0`?z1 $4nvIl6t^%3Q43b4A=+C҆NN)C D$a7HBruSRQ^&"gUf`߶+~"/\O u1*ȇs13QNO,_6q/YeqSk-qcF =&\ȾXZ -{ _7!o7-)br Q`|([|DmU `Z '-:y6w͗3/ qsg 7 ppDȡOSo?3?_˓{iX~J|<<_le<x͵<5f+pK G-W|b̛okzIyJgo(o]_/!T1\buXmr>*ayפ~Mބ|{ucV>xz0/<[8oCj{[n=@hI].cLH0W n'k}$|(eHq尊*Аk>xx‡lix]ǟ}Xڊ="[;F-('mZaWP:.;e[ݟ?)O{G{:{.q(X׭*3m\J!z>M菻H^yw|k뒷ˉ|:yN]kiΦ)^19?&ixsn*'lGnc.s1υ$lMU[78cHx+b\!QZ^o~e[|∾{`^^ouqLJ mYxؿY^(l͓_yz+mҝAuZz6b!CG!W=vzχƌATw;8$PAԎ^D|PuJhC-.9VMڗBw,OvӸKݬƟoBC?d98tJb.H`Ԣk{΂]Gዟg5dPY0s[mQ{76'1 I~|bԾ/,~xDN+!dSmZtp9ɸbpZ~qn"}WǍRs]K:f u?86rTM*xx)GZD-+Wk5^1Ϟ21x}} 9D ~ub>_? +ZP׃J {a̡O1Jnxjԡ* DjNc)N~N` hs;7I/KǶmK:;Гj.s9IA%:(4}`Hs/ͫ2t*+G*0:XvW1~Pv׍6;;4kW*Y1Qkc0g}uﳭ1tVvZ185Ɔlwcn_}Ei18>ۯ `֥uW}; /7iN_Y3^,k@[͘yMgK]G߶c -m[ eC+pLVΉ٣_:]ZZV`-_++CE^?^@3}RxujvcˇD0G{s٣yNWz}@/+y $VqU2ghsa"W[&;Abp^<-L|8`%CQb6WA|!*cdI_J aQ]0V?X/_%Er)귇!]AGYs!CcCb\mQXj:mrH&mԗ|Yʫuen z_e*P v02 C:I++[.k>8&tݪ]T3/KCgm*Wr]ӣ/YǠ/Jg}uDZ,OE_ VxĒ}Eoa%=[FDbݔᕴ>!Z`aY{ا;zU lm9VOյ]J}_.{fJz(j'_vqAFmUtE~l)lA\Nc8?ω/5ޭ"zʞ90=#=E8IM]l㏾1I\RIx-28-e׿SAb0_k_GA/@Xi8{ 繑7pc{t R6L}ΛqCpCa*m݃h;w5N|}S{jw-2={x~3m~x7{CۨAFnL!Æ{Ybܶ库}? M~frۼr6Iք\. &1|lOeB 3uk!q#_JЩޙb³m wLH>1b4'}h}w;V\E6NU 6WȖ{׿:g ZJ5F[I~Sg9[A挳^}fm[QuZ/߬':ϸ]`;gnE+m޹vTw8?^o |~8N4*5gl8>/Ϗ ;|gtyvUam&[˺.fuvˮ<^\"dq/7ݮ'N}t_X۹N5͎]1t鬕0¬w>;׿:\a|estmvb0 |Mb,$Ïʵ/\[\"l,2)2/@XHV*asXm|Ce?y-C?3~@@98zmc?S҅e±%QΘ΍v!6P]'qlUC Ow}d1n\dq͊oVsD"DPdL]CDZNJ{ضA|qC H8M~e.cRȮb&݉t|_ͮ369 xk~A95Ǯgm-y#u׻:蟿κ$v)F[ʌמ7?߅>jK{&f^4﬿qfOFq@)%͜6^Sl찳xv/oMvӤXXA~Xv)FWSqX{.W?v~X?k:tRTOެoys.!d=9ipPt<`}ki/-kmQ~gxF8@u6j;7.ln2رUQ#!9ǫ` c1x4NJFtVw\Z`+6oˌufo}.־^l#mukn֯8}T~%\92!}ɶQ߮+[ǂS׸|n\h_}y ut]܇ 7JwcVpםNczw>3u'<3??S6id#:&e+SbH7翍ۭr{ICŬ1Աck~˱j'%n^_ϩRwhx~=ۻsڥF뽚B-]m{J^CS"kX;?P&ο-;!fq Z TK) -N#7K[LzŔƾ",_7j}vJY~ l\>* Q7mE)CX{GvA)o^Q[!|ai[z}p[&6?{Bv瑾vwfz5g?~2ݟ~gnmG4.a2_^fSR >MJ4!5#}.7rjHEsYv SK" jXDٌDI*/YJ*Dﶹp[ےyfm "^+Pcbp0Ϊo+$똹P&3\gaw Y a;\eǖm֡LϾn Zjok[ '-v%z2~r1LlFuDf e2`5'|e֖l'ϕ^y3+vc{?ZzwWۻ*&˼&&>~KښK~V9-JHŌ/tZ۲2+rpwMZͯov$y/^F:X?yKmHmambVm6T\6Kev ~Uw*ۭq2}LeY^[Nx<{݋r +|><1?Nw|~ٯ-h|^Ns^Oi=+WR\xozc!&86|Ybq&@(E,F_VԦ節-NZRK؞#7 YaDՏys[[A1cڤ})$dǡ+iԃBM/wq!*L#Q9 vMKk/~ž֐A(g-oEY$Cۜ,$gkL:>J?xSx|G ;9Na[YkeZXzqW.7R Nk;ߖ;mo)_zgٮA 3O,)cpo J/LVAyh1XM\1ɂ'rwxbzK>p)(Vo1Cm#wS(bpptj+S&&3/5CTY_EY7"*ܙCb:pY[P5%gmQͶSvd̦nH48e=^K$4/۶-鰵Eb˫\/^^/WX'1P!ͽ48ywec]qEFKmםUڮ& UbL,`Ίg[cG!HbpnkC Ƕݾ[ bpsK}_=A\Qbpoʝr5˲!ٌtkΣ#n֜%_ ՚}Y]xL$׳h͢| ?_?֞kaQ~.j/+޵m1p|Ҷh:P6DRZ ڬl=uYY90?Zdjy.Ɣ|Q_&?!THq"״`3C߷KknOL7dDSK>C6 }Tﳰ?v11Nqhb=s'N+;lZ߇K:OokGfQ}>Ҿ̝+?4lo+ʈq"d:-R_ q;kfe61~B0'ZU)΢iaZT`%zA,WvVb#1od{L$K[1bpwʃ.Z`aY{uwjMq_s孖X73k2^Ks]̔zCQ^Ol<SM躽F?6Ra݃01}-p~7j[{E ?y^tbϺ+g`-a~jogGv?v!/2?EA}2/Z^폙u_#z=[Y+kc~nϼ7^$cۥk x_ Ti&'4i9TO o/:Ǐ _}B1/'-/Xu1xgp1}SZlD&4"S: ;;m!M ֫@ŭ.fe_< F Go~6f6r+zA>8R+0&=:7w搘сFB̡B}8zlЇbokG=Tw Ӫ/E=2#Czn|n12r#{`b<@8G; %gD-WPtc9MtkP$R ~AuOݑ: ~mvQb2Dnsk]Gڅ[cd}䇘;q&[m8gֶ_<vYOtq2wΠ~lWRۼsp~0<qxS1XiU8g]tOy'czw^ tV ԲftyvUam&9k3ա9޺JE>- Х s}Ac&!صM]cWy>]n[?$YZ7uٕRYOƻmS~Y+,?U~yaF_!P4M=KfH &YdQSo bB_u߿>{ھR2ZF7>?Co2 +yc9LvL@W7Y[Foފlysez!1o7Amq굱{L||e #۩]ŦΘ΍vi!˭>0ع'\faĻEyH:sA*kV|#!x$z&%c"X"Jֲ?Vt+~ƶ R0Di蚩8/sBv5NLc2NJZ\apN5=ZfksofͱY[Ks^wGa6篳/oֆa2' wO<'꞉';o}SQ\к֕cmn9~wzgsiR,:Ngp:)ib.}5A=Z}?q٘ x1!Z#.ϡ7\Vf^O;[V'ΚuSbmd#Zz؟'DMEv]1ǛK'?+J-@LN[=J%k]$&'?78 1oNZe*9G/KޱNo-M?s~ضe{-~L/ߨ?[y诮oF;XCI/;c\_ ר<:u mI]c:mȱH&׵4!go:{|)Ffb˩;f⋣s6FI͑U<'m7`urb /7B歹\ggbOf͑j[GW>*vuY͐d(oWCcko{c>SG.~g׾<tMLN1+NZp^Sv#y?uOۤ謚H>:ȗL#ߜ6n% T1Y.SLNVB$-SL؏^}&.)ֵzYۮ.>u.7]i;;x~ 3bI? ˤXLi].r[qgm篔a%wMO$! 0zn}4`Tz8/Ol~|;>fAV #}Ckx .1?ON=q]>Ҹ:o^biͦ}_x:yz ӥ"MVD+}jiUA (( {Q@%\]6W.}Kp[2Ϭi1-]˕ˈqfgc6 ꘷m|7cX7.8È(|EJqJ/^gb4W4)[y~ۚl{}^ucf5-s'vGw2^;kjVg;nX}ѝ3`d1S}{V9 |2gzV>vSAKMj3W>[F{l{ۈ;!3B1!S >)t'9t@}cָ2]ng6{}xL zϝl#($q1.I Z]<- _|sbb0!u@}k!F%wJ-$F.[NC|iwf 13;_Z!-k/gon,@ @{e@}N]xtO'>r~zz"._Y`l\|~yngW7i\?w_]<^|{\xiwjr饝/wch9)~Jҽmgcyc/{XOoG^˓ߖ]o7Vfy>rL{] `hK!H*wymkgGW?[#xq{Wd7'o?@[tϖPj>'_BUi=bp&Tu\}Z)|U.I+ ѯ,ɇ|`_cy˟tq0(0߆n6{<,f6aCkh]ƌFXazo:O/$H?DZ x/`w-V؅L-|4CC:*@f.A(gB*D*R7U=]{y9dtOuULu͗y(NaUJڡw<*|'[ۣ'-|ծ}D#{V獯,6ouG׶ie]Aïc>^veYsb.ظwvo['ObaR{tNk@`c9x/=/Mx>α96k\4ξP+Ǐ0|lz¥sz\OY|>`;^}wD7g`NwY`<VA| *{M ~+ \o~j~zcG+Kb&G8Vbh{AlI/>=7NSr7qF,d8wƢF8=ssp݊*Ecs8zs$SnŞjhC8ۼ\/; zR7k]XdBV8Pb.P`Ԣk;& upoQ5~Xl ArV 8Ez&P{8g!0gO9kNVP|G wB6 ;mA/O8 RKpٸb0hg},ל+y.;s{5zkߩ)|;*_WR ~7S Dbn|Wh8Q Ϋ/p71[?-W˷^1 &&3/5G7/"$3 g (J {aLѧw7<[[='p±m9Q1b}Aoq:iIzmZ:m,P|+Q ~ry]^x|6x>)g ڦ x*NL^vW1~POycBڬ] UblG0g}cﳭ1tOV@V X#ܠ펟~{l޺$bO zZϓ^^\Fʻ>9CrJxi-73Kk">:ɼ9S U|V_D2E9 9XN_?k/kaZ='j=O*^vf{ѷm1}kƖ@]@qA[5Ƭl9EGE#j վqi<Q~ ȧEUcmAAպ d\b/]j6|l{ƫǜwþmZϹ_gzWqC_ϕ@0G{s٣@|^\V7 z_TjQ ŭ0kd!EF}: W@k;υ-T(mTւ_umdB(=EGh;$OE_ V8%Jl{$l`iK;E *ЗU]!n~XrO>5_ռr{rC?ԗz"oc6RZ:Xf2_oX%/}e> (?6v.;6:}dñt_kqR+s)=9ӵ|mryn>VT+7J=߾Կk]Eͻ,cL|yK/Ţ}^{Bkz]q#.b_p[znعrʙ\{Aj@?50.CaK 1M<^_8GؚCb+󵝮m{'.cH;"\֎`GԖ\׻5#+WĥċRw/#;w=9^k$6sy&1om>/‰k-㓺e\gc<B/(t^ԿWr>1:bG41x<9޼ߏ>q|+MA\RQx-2ck׿C`|@+dykXBgp7QAl56^(pP՛(/o?2gަ"WsujQl=)<]_Y1? o6X|=06j]Ϳ)[lgOxBϩri>ߎ ]'qIH"[r(b:7=%dO\˅K@ 3ck!q#͡dMOT̽Em wLH©ě9<178ڡi27N*=,z/N8<r~g jAcdp pfֶU'rM>}qA;5hw"ͫkGcǙo\;9 :NTE`e|U|8vbl6o}J͋~L 7׊]ECd{z]mZ想&b^f%p\#Lcx{>E8 pJ$۞ x>H=b0}Խd==h\ \b'agClAQ:ϞxV+jxO ,+PV?w^Ɛf7+9"`G'0/Sc(Y^3m_$N_Mo`eL _ րߝ9A6{dx{Fd9/W+:LαR+ ̅4羙1W̥K1jZM̓ Zm>t\b^kk[1#{8OʆFe{n>E8q0 x>P[PAb {EkRgխ:{.1x;2kfy2F'I=_]/J-@nm%¢u#b0&Syl7A7Ơ(5lXơFϱМp€|km/FZ,~o"/3z6{s>El@~تtKw۳Oc\Fcq[`mKlщRY,#M_I!^Aw@c9cW|qtH\g?9#q ֱ=GIC(/iwlǜWU{bϭ2㝳:.r_'?#mkѹۣ_ s8 fH_ȶWjݶsz/8cmόxA5ɏvl{@cc;AnY';ZP/SVߔ#[MMyVJ5??S6mh#Uюlה9ctQt ^PsV+ڗjZ/dž*?O nT>h{<~efg]xqܨޫ]KOO9˥*)\/[kQ{>1] 8[8^1͕_kds~zG V$;b1wmo}KFڦ3~̋*SLB25ꦭ6VsaXqFm}$z]8-m#=NPAdܵ`>7w.[Ě]/zgn>^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCgWd/zWXc=P^~leѳ{̥vqϺ =aפ .Q\o~!3>;g bywsz˹h]{O䖸uקW/sFϊzL5[Սk}b|aL.ֱ[37/j oMܷu\;/.{`:/EQm*2zb'@`O`{΁Ḟws%Zӣ.&7ؓ=7i"\$ɕq#Kh%qSN<&?t@QmOE $K gi;v͊k6&@G#G ndf?Z_$ɍ+q |KkFIrØJoz bk/DK$z%)UlMTb%AuܾZ܄[u n$jP^vf[g_୴uw[9)?Q|1 W$QI\J^Sƛ!dm|RjYmI~r ?qWH18ހPs |Ѱ Ɔ*Έ=[\pa1oPsgH6[Sa6޶}ybp$I$I$I$I$#]#\1 Zl_*[@-¸^=M =+Fb9cI[Y!~u/)'I$I$I$I$;b${baZ!`:ф!1?]^vykK$雟x-I+OϜ>=o<{:^zmY+#KDQ smb=.wyy7'rw>4'AQ_ZK |}} $y%:{ٓ=~|{bKo;o&xIDAT+~>@[vK[mc7/! g\^xB&˰f)\nȴ> /mnQ mTxZv[0}vQoCj<[f csV j^Dw9g X5Z*]"v^ˎzvgω`]~۽70XN]/ЧԼ%r⃜*|41x8z\G~kyEz9 ըr)^1Eu@6|j-y<$`8*/ATbo ЏCb\AQ@m^o~@ͯ[y숾{`| yݒiG&do;P;4wXW,:O-tن]icQyo9b*n8O0DZ9V~b9cFVlBsbҾ߯_( K-`wQta_({/ơ" \&EvL]>kٱB2[@qn}-M2͡pB`/Ϟsqל@ }uwl{՝'}%lv 1*<Kp]^nn 9ڙkK5*s{5zkߩ)x`m71}u9CL18I>,N ML-OEb:FW ~̯;K`1˵*j=~ *2X?gݜKhO8)v䪂REՔLp#±m9Q1QaY v>BTw]&鵹kض@B1uy %,j.كg^+;2U`c{]qE\ul㱁^F_WitgDٮ鋏$`Ίg[cG!螆8LF>ԹA?6 u HĞA-uqE~s ܷ5ӒZng05"OE}t>ysȷ<}!>d>r^s{*^0~۵zNzGqqTocֶ-!₶=_kY 9sv_TA#j վ6I{DR~?|TU+y̠j]zs2.1.}D-N{Phlw p=ԵsdžsVI.9؆NE /ުMW>"7{Ib0U\cZv\EX8_bj/ `z$' ,1@M!gV *c uC "lt1VR8q3:Nurmw|ֱsnb|[mms\m{Њ!x^g`R7 Na ԂD>[ ,nI++[u,+ڊ=+, KC'PQ>[ ~+׵= } tV7H[hhZ>}1X|0+~z+7=&-ub0䮖>I/KkN8"*%PMau Sc9eyb_׹Ʋ5zC(yyW/[l~!ԶGqU]s}Y #c=fZ{Foυ?\tL׺+] MX]~SmR{˕zl}߭w]+"x/Zme9cγu\Y-^S~`~1üSDQ^̜{ϑ}(g+3r΍t|Y5j{#K3Ę$Id1xgK}̾7{7 G`lk1>xN1xK޼. [Mf%ɮ!hzh/*pq 2+kZ|~G U.Wǁm-{[3o۪f-8r~GlEZLw7l1kˍxܢyA8 C@Xl⇢;cPD&$Ć$Z$=cG$sܷ3I 7 m(seI@_j1:'ҏ=X1 u~6eoυt19濶A.ٛ$m_lz5t!"cn˥^|VL銉ozv8wmyj>'YqϏ5#YvFmp}T/1vaG۩]6;@ňqq_bQ{p={Gw&x?}| rɍ3Ę֭_/q?okضK '߻ DSWxT^p˿ oh;$>2X::В/Zt ,LM]p"q7QAlB6f<ެ}PUqb2}V̙WEWhFTg3z`u.N}j?Q#Jyu {/{;~lLv|ǾRlu { p\dkB.A3_kp (!zfl-$n9 ꜹճm wLH©_ω7Gsxb0_oZq,{u_ĞY#{9M)\g嗖#[_Iw'99oNAD:3Kwͬm}޼(xӂyߵmlX^hc?g:znj.~<_ s|/.7VOo:>[ɕO9[rȳ&oJLN>Cׅ+l[n"pܯ*|~Z}Hh *&q1ņ76?h?׹u>("Z> PMe7ZfzWWg(2}.m>s]E]gl-!_ӉYk7TiC&o:{})FVC^9#qݟI0HH*XX 'e m=aY#%Gr^sk^sYgc|E'sz}9:yy{+a.{R j#WCv][gUw=/ю^|tuLz,p}' 9+ߠSzyZzOXXg~*ئ mD*>1PPFq+{ICY%Ǝj<}F_p;SAyi9=zls3Ʀk|?7uǔ}+qImkSh.ߡZd݄T%$vqD׽g$ڹzї3ȺoD7sןC7t[|:{byoX[]lOBk#cz֗4nmJY~ l\>˼IK"w릧 VsobCo7^f¨$zl~=9 -D] V #}qwPqM_e5٩EAϣ/daQ/^fSR k N1W(ˊa.m¶w%*lUA (KHA=~R@%\]6WHܖ3+mvW=(Zqȶz1 B3=Ã[ޡF+d,b}ų?y[:_/=:\jok[vM2{ zyi>3s6 Ƙm|7\ﾬZ0"!=a+[N֕^Q-_=+Vb0Qc 2ԩؚϙ#y]sfFGs₱GdZD6g{߰;QsvcrԚ릱=ɕ}N(SY!ȕ}Fq ~}Sڄ8{-p&x*Lj!ӾWt|[ >$k/́'Ē_mZsOzWD\t1C`|.pS OIrfm7PtJʸ%]M6/7w1)=w2Ƥ |1xNhet$D"?o u-Z _Z!οDµ`֏ӛs\װ`-m7وg"1b{U1In_S~wМ]\3R NƌPzk__~k!)'VdmWjX8|=_7{O-dk"3zGb0͛/`_-sꭥg׵~]bpC7>|Jt$ɵ@7C|$ ԆcMGWk)^6y𾔶?!A=`$ _X4G~=+!j{s5nlrEL$I$I$I$I$I15i$rlpY)ֻÈN nm&ʃ@o>mu,ًSH18I$I$I$I$I.- h?step',?㈨x~(zWU n㍿.}ح"$I$I$I$I$dB1xR~W ַ/WX3bp;>׿RCCa])'I$I$I$I$%s =1t#~1kb/0^i ܋o>v--S\?6sbp$I$I$I$I\2]]8חO~Ο&kjUpoW#_8R烾7lրbj_~$I$I$I$IZTD_"v$DJL1ҭ?S_ B^SlokKeZ ?);^]޾x%Ir<| ӧGS}gry]g0S8r~>pxh9j~NལU'?/w?r?/|<[n'I~sr9&] p=b0{9 [!S_rĵME5ӄ__bp7n7ckK}*Vo/>{osUL}Mps|\hk.|}z |F%_նˋO6T`yVhU@U [>2mOMg۹_ԍYB^FL}#ky]ېbֱ؜UDl†/&]/FV㷷V[ 浮tAxS8rXE#Rv?/ IyG'_k=c{D4y;o[pbѵ-bZaWP@]oozV 6w[y{ 3Ԝcα|3Zm^z9 ڱyX{98i='037uOߓl'KWzlBdxw) O#b)ٯGtha?쟯h^_9޳E saD˗ r\bh B (E_^/7H׭\@:gXLح(] 8v0O7G?zV8v\>[s UIR~-?x!.~q| Na;gϬEXۻ9ėq#`]X9W"}]wJ>j~mڿS5!17R nkMo.zr|]I1-0+ޗ(A)?b8q<)ʾ4e0nۀw!WN ML-OEb:FW ~̯;K`1˵*j=~'ș+JkR^4{aLѧw7<[[×d{ '6~ N)CG)NMksұmcB~_Ab"33IAwe*]ĥ]6BfjHЭ"e/>9+{m{ƶb0Pmwc`/5 {Z 1gzS v[s-`3濒3޾xZr@ Z9G2orBso:<Ǘ5gQyS}oWO_o~~VωZq(.]޵m{ p|ڶ=`ctP6D\ֿck1+[9gnQr̖wX\DVZ79힒#J9G_)RU1f03zs2.1u@KQwhlw לzYkr^]Lz{Jt=YQ} >/^vjOֹ^Du1xu2'{1&vSH~N(ZU)>VݎQeUE-|ά{H L9$& ' ,1(JM!fsE2(RM@!6 Kc(9D1&EO /6-s೎sK mkj OK^'&xn`~Yy>V7 z_TjQ ŭ0kd!EF}: W@k;υ-T(mTւ_umdB(=EGh;$OE_ V8%Jl{$l`iK; Kxat7~n~XrO>5_ռr{rC?ԗz"oc6RZ:Xf2_oX%/}e> (?6v.;6:}dñt_kqR+s)=9ӵ|mG^X]~Sl'8[\`5.m$؆ٻMh>"c;^w}w @ZXYHL fWorhӤ,&,7^7z-0k[I3,"x%^l` ]ǁm^.8j ݢlE- YK1FniOQe_nij?|,fO$zN8 C(W3R⇢;cPD&$Ć$Z$=cG$sܷ3I 7 m(seI@_j1:'ҏ=X1 u~6eoυtr$-O|(r;IL<ˌîsHw[.r+>e"OWL|3m䇰ճĉkfthS[P ;q(\Wlclk9ۏ3~~<0|3_ѶT؅zoV4נtb1XـjOq^6K 9|{#;w=9^k`_q }l/}e30kN\82+*2x/CMؗB/{W>^^h jOkn_ m`_k_GZ!-]d8{sܠ.XqDʆ(4h30ہ6ab6#smn!Z^"yE8*w~K8f{x~39Vco¬Cۨu!#7? na=Y 1n[El@~تtKw۳Oc\Fcq[`mKlщRY,#M_I!^Aw@c9cW|qtH\g?9#q ֱ=GIC(/{8 oe׋}U{bϭ2㝳:.r_'?#mkѹۣ_ s8 fH_ȶWjݶsz/8cmόxA5ɏvl{@cc;AnY'Tz7뭦&zXJ5??S6mh.2z ?:(:nŮ=CY%Ǝj˼?eh$$Xnڊ`SjEX;vnP Q[x6-^θyKgϷl+Tw-X%MݱKC5f|=~9pzgn>^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCgxrj Y{,g>u2^"zu>r/19Yֶ'운9e ͯ#}fgl4vA1ona}Yc9` o S_!q])+QWz}z2ghX@YSUZnck"?g` uΙk[|9gqu ū9Qs%vb7׃mqtֺ3xYǾo}̸޼15؝+adB5#qk' f4hN<#7q 9ocoHXx_D$sZ, ]boo=Y^G˷wڹ*&˾t&?n9>. Oi_ľv=>^^}#jۀqŌ':*TkzGI [pQ??aL[並vؑfOmnG^.>kkZ6ƃرul96gŋI.}s zU=l}y2]P !,VQTz?/ IyG'_k=c{D4y;o[pbѵ-bZaWP@]oozV 6w[y{ w|rS_nj@`c9x ANl{>_<|c=#wl;Zmq Puj^1 eN{l[T >X9,b _Q&C?fEGLbpEa.ŷziE5n#%vKJ X_\_@nTb\P>e,mplBfw"EE:qzG :T<XqHbF݊^>[s UIR~-{I]s/=b0WwO<^)l|_/˅|q]^nn 9ڙkK5*?ܫ]kNՄİ^H1}C#;{s\b𻧊 J //_ ?QQٝX7__|}U;g}hg#2uo?1D18:" o\\!_,߮c>{W/߼\Ϡ"Ls͹=nSN(REՔLp#±m9!-%TA QS ߝv雤殥c־ /Egjg xPm`gr>x|?T!Unw##|qFkױzE}]jHЭ"e/>9+{m{ƶb0Pmwc`/5 {Z 1gzS V6'5j—xWrOKhYC??fzks`wϹ|Y'5~gN>`+;V7\{7u_1v4PO&VOwdQh s`}/G?m,1x#q̊VgmpF1*JWv-;",/n1xcvfCb0QaI ]Ml z"%+B\ .BPqA篠lt1VR8q3:NurmIs೎sK mkj OK^'&V >+o?Hp qM$Y FOXgq+H]_٪c1_VTU\aV_: *Z+_їVc޾G-ؒ@{D´J烹d_sXm඾1,msg!w5wb oUK>_U*,7sN}y'f=fcm/žs)ek&8/QjG_8/qBmc?n"lG8{0/Hg=0wk:-^1 N̙uWk M3yn>VߔGrow~EʻދVwٞ1mmY:.տGQ9YS\;XNGz3sw.,ϑ}(g+3r΍t|Y5j{#K3xD"1 K`)rMTe z]}_"y~Yj?qWuuCʳ- Eȥ>\ f_Fmۄ#b05\|GR9ݖK\'3rL!l8q"97V?xl\A< >ۆf`x} ~}N㌟ak&Gth̉ ?N"w\Ԗa!wj-F|Okb0 a#/H&zrqW牸z{ȃ;ͯo7gj,޾mE n?>(-' F 6Y1yJyI/$q \1|'%虱WҦ'ts^W϶%2!Up ='|ioDZn9p{&*=,zlNϾS89,96F/\+xv &Q^>F׭ '_kfm%#sM>}qA;5hw"ͫkGcǙo\;9 :2SVߔ _ȩaYri#fUa6wڵc1t|׮>޼(xӂ2;7@5vIp/ҟ3yo =75??M9>'7Dż])㙜|MN8|41M ھZps`W8ug.F"ϟb0 ?^Işd`}0]lqQW`xxѾRk1Eո7ͱ}!v~u[q`]_Gl+d78z1v ?`]u)L, Q^nu`=5Y]6=8Q V?䘃kV|#!x$zC15v,Z+:?cm)J"86\Ƥ]` ݙ9c`B&I3Xr?L9ȩCzZ.2s|0c~u #m1 ו _wOsyO=3g^笿83O}y夙zymj7oqvꗩ#n{ DZ羙1W̥K1jZM̓O/L+3!~fc^Ǜ3iQ`^oVfr/./#5'ǶZ ~}&rymZgUw\o:}E:.M}{ ;RE.Jk[EKsr\ /rha63+Gq{_N\G8 k I} wblc_rҗƍ:M_)˯͗gbU5iIdzA!ъv7p6 * ڸzs"6F%덨zdsZG&%zX3C˜k8z=ӳSvG/p_¢>7o_dmͦ>T{/E=\1>u|c?-s} d.{LE{e෵ &mqz4~q9]c6{Xw_Xmo-FD0Q_gѳa+C5yP+Cڏ 9-;gm;Z|wOlqأw2~WkSl7='w΀zLΙZ4G:I0ej^>+(Z?d\/o Vgz؄8} 'oo/+=Qh+s^z's|zbh~i?v7$7,n} E$I\ n|C~'Bs'cL .F sB++'ͧ}+E*zcKW1xZyݦcocyD"^x+ci/lܖnw[+XB3\\ =$7);h.I)' cF(5|{/`/P?` _?W&~zǹ ="?}ѻׇ5aν}VF_F1VZb O)o_}•$,+Ik/oI!J %x'Rll?})m2Cz-8( |Q1_"ZW)bhg6{>y[#"" T{\´$I$I$I$I$q| Rmm?hN㯶{__ƚ$I$I$I$I$I*~kdmmï[;bcr5D"$I$I$I$I$$(?AѾ ߇%8&ח$I$I$I$I$I>R N$I$I$I$I$H18I$I$I$I$I`Z ?);^]޾x%Ir<| ӧGS}gry]g0S8r~>pxh9j~NལU'?/w?r?/|<[n'I~sr9&.$]boo=Y^G˷wڹ*&˾t&?n9>. Oi_ľv=>^^}#jۀqŌ'*TkzG+*vS~~*-§&ų/,b /^ #n >\|. mHclyl*"6aC͋].Lx{[+ǭZeAB<y)Rz9 )X;T|Gtwk{b哯ڵ}wOXۊ="[՝F-NQ1+vz^vˮ{=N̿;yQz* ּ%r*^JO!z>_1|c=#T?zw\ҫ2@sʫӵcsQ՛d~x~gn*'NbMY`<VA| *{M ~+ \o~j~zcG+Kb&G8Vbh{AzN/>Sܤ;OuنÅƢ"W8=ssp݊*Ecs8zs$Snj/Eam9Ť})_Q}YB"2$7á" \&EvL]>kٱB2[@qn}-M2͡pB`/Ϟsqל@ }uwl{՝'}%lv ۍ~>Jpٸb0hg},ל+y.;Ͻ_۵ȆFrzGדSJIqW_nbj~Z ./o1jbLLf~YG_ko^ETIVg)g ڦ x*NL^vW1~POycBڬ] UblG0g}cﳭ1tOV@V X#ܠ펟~{l޺$bO zZS|&۶\"ϸ/7L䌷/r=yy*rC=Pg5eM$YzET[S߮sC\?eow}߷mly4 ZcVxΙ_QmZ~o\/s=%GzsASj_cMK%gU20{c{b9'}aGTCƦx9'#Ʒv)}DUK9abd=憳Gzyi=YzI!rF1*JWv-;",/n1xcvfCb0%d*!19Q`AQm+B*dPPۥ&?h aXE)'H$PPH.׶m1+u,_"Vh[W[8^j:19s7ʛ9?HЛR,n^# ,2곸$ŮlձT8zl^y.tNnB1o}Wk{%E鬠o(:B $!Ѵ0-|*b=`.WVb#1od{L$K[I_GJڵw\*g}kyC~Ω/ozDެlؗu.leP4J^^U%/}@H?Q\lG\wpmtcc鬇#9wW7ŸSz.:skݕڎ/S{5s<~SV?w򬮕w6ﲺcγu\3T>zW .kzX_Dzȶ {~/rʙ\{Aj@?50.CaK 1&[Zy˲or$b&!p1}c on"l؎bp}Dߝb0&"H<G/ įω~~|#lyEN׈ E-'.cHKv[K_ҝY&sc +R}}i'^G0wzصs2.N_soIl"z?Co_1Lcڮ}_0['u-˸8xu_/JTǵ [~B9|'߻ DSWxT^p˿ oh;$>2X:: Yh $++0mvƊ R6zLqA8(M]S䗷3ow +@Ĺ:O㨘r"Y1e?o7gj,͞Suh.d-6'^Zb C[_Rx౟$B/;Zk1{vOmci[+\|ha>/gKW'q-Šlg_8=ӹu_ ~ 8Wٗ}U|qG^^;#2jR8Y Fk;  ׵/\-. 6L/WxM^#ƛ8}< ,a)G8dXm|Cא?y`/*8~|dxR1@)L^2~1.~J{plC\Rt lh>bs*yij8*^Qx\ga_:1 Ǹ25+Y<=xɘGDzmⶇL%p ~jze.cRȮbY=g {\xr^~39泖K?\7u(Xh_]ȿ0G[ u%Ck]S\mmc̙9o?>SCh^DeT!^6_/Svoq[3y[Mvۻ\XAkp1Wh? hŏ/qẊ9l!ZXby@/W60"?o,s/‰k1w~D(g_8=wGKԯmM툵U{}sbyHG ~hQt}|+1 qױ(&LBFbMܠ\TLDhhڬo9y/!h-[X`͙8* ȷ֦9lldŢw&2gk=+ب |Qd?J{=4kt=Gֶyh.B0RDP!h}I Z=3v!xGlu~&Ø#!9ǫ` c1xt1aQ>-Or|bŞ[e;gufo=]N~29GڪלsG+n]rp.uY͐m6"q5Ժmص^p9[p׃kG_Ǥ"h w`ݰݝ[ j;Owj^*C^o55*V 6iC*~h@_k1?:(:n^/i0.¯k^>ྨB-/^fSR kM%JHߢiA hs^yAZ-0be )ǯChd+ 7ے}fm `1_+PB/bppUVHsCg9^2Xiϰ}ų?x9ׯ[9Gmd|M%}7 }rsH >H\1q )b&{g&m_K"$2nsi_4X;^#n߉ǤPzܽb;?JIr} Lǿ65x=I1x+k!~m$7);h.I)' cF(5|{MgM.~2y5S >O)o_}•$xiT$׀f/Y$2=lۯ;oR>:!9'6U$R Nn9{} \ " |$I$I$I$I$Is:S~4hN<#7q 9웗ɥh:GH .1+ϱ8/ `3*%j|?W?ZU1Y7xqkwqQ}J%1|UG ./fGr_[( ̷!5Ďc̱9؄ 5/^L"v黜3^a{ook] ]G~#_Y~m߂D[mj_e|zz~׳ۿ]q.z^>]2"N}yKI/9U|ibqPdqjK^g_iF+Woo>kr{k㚊+S ~g1σh[ׯI`}k&q^W#9*/ATbo ЏCb\AQ@m^o~@ͯ[y숾{`| yݒiG&doAzI/>=NSr*نO\u|[Qu0v+ĩx9̱͑OŌ7}`~rwG[x^j? BI߻~a4Q 2F-cP]sg̎֐(gBs[mqwnmss}|dd^﫻g{`\,x>}'+!dýSn Yw/2WNp]^nn 9ڙkK5*R}ڮjBbX/op#8xo]wΐ+S ~#Q1x&W[MܭUo(>jƐD1U{^}Ei1B X]Ǩ}^01ug=} 9"yQ%YǯA_E笛sy) e}Q; BUpY[PWH|nD8vm8gS~7GTX"N;7I]KǶ}E/^.ˋ',aPt}Z܉Cܫ#F=G.?֮c 0JՐ[E>+&vM_|$sV 7>#> A4mi`5 Ƕn_k@" bnϮ'(ZU)>VݎQeUE-|ά{H 0*!iAIy 7B!'|)9(AAelBnrWP6weXEF)' ^:ErfbMca8BR׉ɡCʛ9"n=,@\S|FY bWWXW1zlWXtNn4o}Wk{%E鬠o(:B $!Ѵ0-|*b=`.WVb#1od{L$K[I_`˒1O;UK>_U*,7sN}y'f=fcm/žs)ek&8/QjG_8/qBmc?n"lG8{0/Hg=0wk:-^1 N̙uWk@|Vϗ~|,<r[7\`; ]Eͻǘ,i_C![h-.,)yˣI^]9;h>}Q糕\9KF:noHΑKsbP}.z+v R645qr2b&!p1}c on"l؎bp}DߝbH!ay]$$J_]C&ћ!7ކ8,a#'T8S_lRLеba|oY#sF/ _Q\j8mC%boUp}w]-~[׬GHX |~ϟ-{}Ϛ[tX=O4>G~b-WPtr u9(B[G;$m_lz5t!"cn˥^|VL銉ozv8wmyj>'YqϏ5Sk;]#6>Z#rk9>5-ݹ%wj-F(Ϊe1E~>+G+`ܢF_0>Wcgޣ\qW̓7Ûo5_fo_6~G ^#m^^v,ؘ}aؼ$ք\. g DO rPBZHms+Yi:9s/yūg} 8Sě9<178ڣ,.'< `*/-G9!Ns#1r< !uu+sƪ M'g1okWx&'!$I'(>xm'1^o wE]!IEJ)kxU+dX`9+N1)>pLFoފmo>2g[!ʼnՋ[zXJ[& CWi瞫o҆φ(z/`Ϟxˬ.xO(Ͽ`rB5+Y<=xɘGDzmⶇL%p ~jze.cRȮb;:^d9/W+:LαR+[ɕO9[rȳ&oJLN>CI4#?{(>_[@]N8JEXG`Mv!^#1xja@dj!o8[`Ѿ;pb84gz>("Z> o诮`F+,tQep祭\ҽ۞}5k[B9ͥ-"!)_n"tLAu4 S$b;f⋣s6F??ȃU3x mm%8w _n {nQp>zϻ}drU9G:/oW>~%v\2!}!^mDjuαk5^s>3]%?ڱ׋=IEa;gv<䶠^*C^o55*V 6iC*~h@_k1?:/Vu0.s-JCyi9{zls&mGv06E^gq]=<N_K"mS_Bs1$Iu|c?-s} d.{LE{e෵ &mqz4~q9]c6{Xw_Xmo-FD0JQ_|djYѰZ[Ս7K7)]Ox~Rd$I bp$I$I$I$I$IrsI18I$I$I$I$I`('I$I$I$I$I$7,? F{IENDB`llama7b_autotrans_1to2_predict_log2.png000066400000000000000000004055411476650141400354700ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDR4ITsRGBgAMA a pHYsodIDATx^eG>x@Csϣ)YrtU4Ʀq E^H3щ5+k $6 ey5>#bεk9kĈ1b7YWybϖ,|m2i\wlaR>L|0Md9y]qYQNWIo~y<5]]ݫ?//~?]>o./~Gk( @]˖,_v7S&[#ԥK <@Ysu,(.k:˱lCȌ a  m%vZf3joIb]m$ke;}aHmw-loO+-lտ$WO3MrcRoC)9*㺡ݿj]+_cQ5 C:3Nycr#py;gEfg}6:gKS]hEKG6uK+eRR;n%KO(q˨!:)C3/ {J׮m%h}}ܻ npT~ 8mϖw(R Jo-]0 N ewh+PtL@:t4L,a:JyY&%6Na*ޥN$sa|HȲl oϔrag9ʲȠ s°߄ܥ]*'2ϸ_ [gyuoֹvԶ>Vk|Z~(q }s ;㹱韗TPtGyt~vij?*PunƳٮIeٕҋXN·B۪:?{:.V@o_1FH{zئC>j/m m%h~h{cKY,ӕ= $0 lMO^X`P}ՠA;EY,*~W,~~vվF2w)ڛx`;6>DB誦YO=3+́;-oSꇼ^yJ\Ϝ6^ST7Cξܖr;mϋ\a]}ʦhB^K/(jJ&Z\CP/Xg s*P7ʨcLeLNibjжG׮m%h}߉LjvxGeA%p@'Y2 LOϵxº׶0r֠WJg3 iGZd:-B䳝 hҁ޺xM֨ 't&a>{ig3.=ݷT_단+w՗ү7-Mσ修DHg / E赇e:Uu)(WN ]7'bژN.U 4/66-Q+(Lbmm :FE[?a,A:?6ye,p}[kp`+Uގ=v.^nեNuRپ}R6U#meGO_qm('_Vr,nl!ƭ#e\0Vϴ-X5yYk/پl+p qtSg7k9U g=;ۙ:㞇E_ә) m4;}GYjCͯ~ \/iJ3=#W7mͿ2MlE\mec,l4ygL{&܋x#[@[Ac @Җg{l&&;}\GNAApv.-I+?-R?w({#@i+-xu׼鋲FiOםF_}/ԎhH;(q,殳wҦrlNKK|H': # ecԛ荇Ę-BO} iՓ.<8II󖤵:9š3 \6*iW t.7KcuvW<ضO-M;9"A 64`-bi:U=gmRЏкx׸nnvVW >o 9䶂gG]P|[ˎ`Qb^l`>Qo9ns[׳C^6|/T1Qܺn9'4VM%G[Z=oo#uNm|2Z:kh*yugRycuuf mPџ@ 5ޮ%0үV[ۓ愦oϴQvMsGVy{xP4^l<};=[MӠNv: 4cU. xNŭ \pQt7[]]*w`.Es'dt&8Gb  ;9/(b/_Gy\f 0fC~%+XA}XC9v}K >`z!+R.l|C) E~W%٣?XA}XAAAAAA                              zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'`zH8֞|FΥc  8*~~@|jy+_,?lJMk xYKv1(UtΓ+[.g˻o\,%*ΖX>b:8t+ɂ>|A3;k|2蜬y|Y9^zyY3ۿ]v-Rmm{r`glC!ǺlЋ#N/rCT=OXs$yw|kgK5>/^DLg佁Ы {N1:AshX{9¾x}+`AA`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_׃Eltܺlɉ{嚧dC~982 U [.ПVr3%WgxYC`ضc!'#?9vY ҳ#\҇r PU/IWOR Oˣ 0z)8| :-ndL\$j ET-8Oiʻ/3y ږ`d6ysM2]/^:ȸI'+X}hb{K L8;|eD>}[v[m ȋ#цξ/g]f}*<:UIA r%`4[^͡3&`AA|$ _ZR|2V,v|Ql {vEhD{/ɞ؏o\vk(3;:g9@ 'G9Q:E$wp9lխd)s%wY? ns84s,u_b%޼]nZ\yak /J4*3GD 8V 6Wi+V|%-]m6s^ӿJR6tK mk{[pAPu :PtC;ݯt ==VG'`;l/l{kzf:ۀfҚMug<qѳ9I Ce!>?q{,,Om0"{;5=BPrԉV~/*/)_޴Vs 0dim>x/HaXcE];Cg6Dus`:o3XWeJSG>><(!T6RgkǾ4κx6Cb;a{O:C8M XCstB;e~| YM/5_Mӷ9 y=62_g廾n{ގAAp_9q N%]<xg Jx99K*WvҥS&ʡm©:8t, q"Ύ!'R՝cBtK^F>C lK:ʎ=:K?0g bKC,m[!]F,Bޙ`sYWq R'X7\˺V~\:oQ[N+S(N_ՙ2ܹBǪzY~QWiȚ4CV@-e;sJےwGJ po_3K:A/f$߶G2H.S(3`<;$_?-uXzusĪK><~s6JiVN`*3^/ aSgb6\X1YI[4x~iKS[ﹽ4]^+,'_*"~Sշ3?O~yr8& WW\-sN5`_&m#:{)l>0{$m͖ZLZի'b&N!c{ŜgqP_C/߱|Z~(!nS<#4(3]Y!99[:{I.ڨ}gQe[{RI+VaPH3?-dž륫,//_as(a96F_RW=llZtc1 Ow?[AJ%`$Gl**Xڠeߐ+!}uAH!8ƁMGSe<7Y0GΈ˔6ma:.́LrYYN(gC/=%_φ޽-IB<KgUnV|Ӝ&z9ՉȈysİ {ڱҨ ۿ}<ڱL}qƗ)@\ ^р0T pR}, "6Ӷ Tܺ))봂;❚KUƕ|V&T/ubh+F;Y2&uV.ٔUYľ4ZMk86̷9ڎ-@v2L48GY[}N3]7_< w*'2ϸ_ [gyuoֹvԶ>Vk|Z~(q }};aswsAAp$)nq AW\ dS]U ߥ ko⽃C}XKlp;:[ 2IY`$'Pڱ=:ӎ7-W]g:`8/[nЯ\Μ+Fµޅt.ck .< d{]rP;=Nl6˘_8`QlSm`Ȟ@L @ր_}YՃ8i ǎ/2UyͶ'Q;&f|<6\aVIb3li?6hő9ڞ5[ru]WȎu_'_ tT0,7>W&)GW5͒Yy`tiy̬R?2S"01il4^ST7CξU6Cn~":w{Xi@:ʦhB^K/(jJ&{C;K&>v6 KcLYX͡Č-Mx{ƻuר큍W2~"GcA`/j M/g`P X2==vD ^Y^)(6siw xBDhAٺxMĨ 3݇]:0=43ٞ[Eߕ;KϖAZ_׃FBlj /:Uu)(Wus,!R;^ _Q@hCkSo $*ٖѶϠncP3) yn'NZƒ ׷ᙽŽov[Bz:o}l_>)*͑β#姏ʯ\/+C \Tu6JVߑe.mmgy,H嚼Il_IHcθaiw&~a  >j"; WApє#<.̈X\3K?Gl W"(:XAAXC9v}K >`z!+R.l|C)rVJd?cHz:SސXAAXAAAAAA                              zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'8&>5{k3}geo@=8s{rA_-w] | 2/?k鏟;.Jy|yKly-DZ˗Q fˡ|$h'3/˧*-󧟔clM^ADЬ\=kKgifSeߴӼOg7VЋ'w.;uGP]{ޑ+[qZezr E2ҥ֮ˬ?\~E??>i/Z>=MtV+I[Qu2W~VdsʱlQ^6ǛϿi;.evi#.WGvx.n}0N/Ǹ4>1z2,6>1H:?)6:{@:{}Xa4d&W rkFkk _zg"h0H7G_ t:Nnt$e< ]yG@Z[ɱϴl^eSnR]F`ێ؏Xdɶ%ҳ#\҇r PU/IWOR Oˣ ^%8$|P3Mls, 4h x~OS~ݟɳ@5dж$$g͛xnaݔt}|jIGM:YG[P8~_?\>dy 3-#(!sa~0˥EF;8r9w8/%yngiJo΅ӡųsޞ= r8A{/d,Xj~m{Ј\_=߮|ZoᲣ\"i~勷^89#bAd꺄#9Knu+9llJw0iqoOQe\6N/Knp9{v77kyak &X{m ;Mt5ڿl3޿ym@ i&ԺƳkfsdiٜ$kj ͡I{Vm=l6J|֞j!(ZVD+?Iuo?u9״6 Fw<QOϢ._fBmMIV9|%^9{s/Ƀr1Lk#u+L~{Iz)Au h׭x!/y=oI=,EsuF0e*:ZS}y\Cx.О [/`kv}mV2o_@{g{jߖyz{JZ}>4' ˺V~\:G)Y +S(_Չ22 eE]-"kr Yٶ bҠc[W)-k5bI7(4BRa e=~z{G~ ~ݣX_1zXuwom>.2Ѧ pifۓzثb7M1mۜrYVsfy&m \?/mijK=їFk}~Kő۝ .{ԯ6CΓۙgxT ^x(ѱfs8=w敢_oiZɳ.*uZysȹNg&oJogٶ^7"ʛpG܏}zȟǣES")٨7@讫?{u$Py1 }i6J[-ɲ>21}S陽& $WW4? ઁ\t,(<8Hxnp+|DǻC}A}t98tY7:W,0hqrJ&;Gg a:7(I'M;~)>;uHmyss}Y|9RnkyXv~G ՉN ][vZ3!-O:a|y'<>ˊrv3t _AWc08Q5-K'7A   [HDT"3#d[ M}`2pL][0$,k~m`˚=VOwMF.u N^YfDvR:–1;]PIKrzD1:$7)V9D{~9r2kA@;Oe~X]3}?͟miP;/}BzD.on99[{+ ڨ}gQ˯ߺ2O\>_Ohg\2j.@V>輦 <μ܂c/6#+]:V{|')s*P=}Sv: Ow?[AJ%`$Gl**Xڠeߐ+!}uޡ ~wl8Y<*3!299='aDt=:BQd('N+zJ\=^ {.W]s=Wǒ`-ڢc 6:9܉qf:2z1 }J`Q!gFIOK7G_͒ʵ 8@C @'` o3mIŭ*N+#ީd9VW~KGY:S *ԉ94gpdgHa:vx Vd.mw&ѵoC mGEecx{ ; Q=FVh4L׭/&.R=yJ8ߪ8vͫ}εAXsG1cGd Ȳ+9]ރ  ,N?7\«A^wX)}UX x}eR7vt!m|4y`ؗ~O:?Luv8evl,}uvAnY ] u:̹8$zm*\A]IGF=ְO:4M//?;}I+G~=0`Cj;Cbj ŐGڶ|˪ru0 ϋhP7YKq|+S=vuh+@c+NfrP==*{?  ._Ԁ^>"dzzֽ'4R: lQL;<&ӱvpn9"x$A˖ o:F]<}إ3sHl-M>Sq9辥Z_D]~lizu u=8$zm4G:KxI(@LGF=7,ӹK錦-@Ylvf9Itt?/䯨jA~Q!9h1j{e%P ȶ}u \I^  tzgisx]|9Yv{dQז=ree(7˒F~b;RmL"9\7ўV>ɶi W7L7u1ϟTPt^G%yk:]9muxgUdm}!Pu:`=C0Wܟ9F }[2Ml,yK&#xcgRGiz:Gl´giȽ1 4Fd/miy`f21oSutAg`ߒӂ s 7dFҢ'Zw͛(ktk4՗B0ˁd9Lqr,`:[~!m*@F䰴4PyX -T9{"9<\6Nx~M)V=sN(0/Q7oIZS]?sy89erV`} Mr4VhwsmҴ!4`CIA frH L/_Nsv&l ~Vf0hgk zu˹3ϊ Ln+x&yHAڦ1{;<]DQ3`zݳ?k GͿHm_%z~RxG_rct8[_7[%7-`Llv1_]a>ּ,7lsqI9Wd+v˔Z B3h[Ko8+9߷u&$Ӧy=zI}}~A#~vrVLһ1:36O?ކoWXWܭPuXsBӷg(;չg+<=` (YZWdciP'EuBh;Arرê {^jSqk(7;"%\ݍVy¥XKap> Α  NrG| $ʿ؋tYApfD,.Х#b%Up&D, spg!>%S_D0= ?s)q6 >fz"+@QT NӃ PF, sP,      ΍AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA`_|#}/^A_/7/oYiAx!N7Oʹ go^%n<=J_\,~y<J;G_@ṉ߄~Lh},d r#ftyvR1AA`??{ ?X~{Lߕ/w%5?~h*yr:-ugݷ\.ihgK,|1EG:ܕdAR>o~X>UitN<_^w,'9I$tflCJݶvze:h< };r[c~<^˸L_bC^/]bi%ܟQT?//2_QQM{i_/^}wHڊ{׶已&S}ѭieCR-/A/=<,}߮H߉t)涋Msb3#lᴭ+9eۅ^Wv}9FŔ|8x#ϻ_;[Zh|^>do7S{3WK9}n.AshX{9¾x}+`AA`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_׃Eltܺ=r 2!*@BgWПVr3%Wg :v.%eXȉO漊AKU듔%~h%^$<߆CN+DKA۾<` Ud= E Nӿ}6L!% &A?kcew 릤tWg.WN:2n _?ZX%;>hiD3|3es{Z'"/F;8r9w;qs?۵ę0X\}-y/  >Nދ/j)j>A+Z_(y="4"ŗdOǷ.;5\vOY )vV89ʹҡ7("]p$!/_aۭn%'@MeK |~'HqziXuŶK yd雛5$^4yikUg2 @tqhlzW '+Kd7Z۸m缦E/땤 V l O.)h3%2ƒX@?u,ՓFv_7j1H3s{z\ O# Nemt{V{kzf:ۀfҚMug<qѳ9I Ce!>?q{,,Om0"{;5=BPrԉV~/*/)_޴Vs 0dim>x/HaXcE];Cg6>890qK9{s/Ƀr1L#u+L~{Is۟|km+~1C^ʝ=([{` XC:#Nv<:ͳ_™kt3-x{osw{@mdh[w}# rJ@x"%sksl뫗T j KtL8ܕCۄS u pXXDHg;zRug8?P ݒےhqc{̙b]gglk Y2r<@fkȺK]?>Z}>^ֵy+:G@pXBqZ:Ve-V',J[Dm/,YSؖ;UJ~ GiMX ~94'= ApyE`%oÂ_(׫{#V]<ǶɆ%?7ڬCL[0۞tk\^!lLlC R?f73i+ٚ&/mijK=їFk}~Kő۝ .{ԯ6CΓۙgxT ^x(ѱfs8=w敢_oiZɳαE">r.DlR;J׶*y7|yVބc:s?[#_"O}ދ'0= :Mg}R{Dc(ڢϊP¶36CaS]ݷs;ҖmAsX HUC`+RpS a:q`Yv tm K#8x/S衷b)Z J3QNqYفtY7:,׹&:ẻ,&t*]ʃNz<2{ϩr3^;ȫƒ;:W~w/s!F/tZ>©cI'/k w*A|qiA|] >A R`NV@0fҶD&Ff&Pu'Y'9M/l#cu؂ a?\0/>V;C˫=f&m#:{)l>0{$m͖ZLZի'b&N!c{ŜgqP_C/߱|Z~(?3NS.GvΊleuΖ.fhEW ,:~V<)sE噙 yܧ9Mr ~#\I=a^ceQ9ۧycf/)=RЁY`XYDmm/g۞9U߉uY~.qPµ% kU`č~MݳJ&^eRAs^~-^ Js]4O髲wRgWk4,2Kia/M4";W K@e誦YO=3+́;-oSꇼ^yJؕbszٷfo|^Rn= Bql&+eTi`Be.Աdcg=iAAp.v0I~ 0%smG=0 5Y``38<y{3/ Xt~m$IXgL;V;1 ^7G׭u!dkԷ>zgisx]|9Yv{dQז=ree(7˒F~b;RmL"9\7ўV>ɶi W7L7u1ϟTPt^og${}Mg~"+ҰB֡lWu,-r}!tV+sgdε/:P[:TzY6_7:mc-zK%.nw_W큍2){u2c~V,1XAtoIZiACY2L#_iM_5J{5K}v}|_NK8I 1JrZIԲig`:FRvΪ~'CpX;&z!51ftINޅ s$[G@CpcYVq IpY[9޶}jia. ` 3h9&KB?h~\6ut33b\yk̙gE&<ut|_eFXAAAA /{zYtv @Ne`C |6P+0:XpYc1]go}kʗffހzNq9dK U(% ̱r&A.{[d w7X}ouӭ򞜅"ϖ y)+1cKb~.s7?7fm>&,sum^ 4m;43OfıSe'K_-w] | 2/?k鏟;.Jy|yKly-DZ˗Q fˡ|$h'3/˧*-󧟔clM^ADЬ\=kKgifSeߴӼOg7VЋ'w.;uGP]{ޑ+[qZezr E2ҥ֮ˬ?\~E??>i/Z>=MtV+I[Qu2W~VdsʱlQ^6ǛϿi;.evĭL)c,t)v# Wv}9)94HՓdq8Gɰ5ó_;[ZgyHg4N h WK9}n.[|g%=;!`J"v/Q{EFهt.Џ! ̲+\f 1;LYcח8d`!k1VJ{VyG)-A*;D{n@Kn{s[]땁-h1nX2'\{}/`{Lӓ+$W.߂bԇ$` @e[\)(G˭e^0<|_CEltău OCp܋=rXsqr[#)YHHbim%^<~Z{uSnR]F`ێ؏Xdɶ%ҳ#\҇r PU/IWOR Oˣ ^%8$|P3Mls, 4h x~OS~ݟɳ@5dж$$g͛xnaݔt}|jIGM:YG[P8~_?\>dy 3-#(!sn'ח`6sgrr0цξ/g]f}*<:v-q&L ˕79~\8Jg={"v[.?Gu:׆ , /-v\noEKS_}}P\`~t&)ysy i%isGS6WqC0l|^V&iZC[?kydJmAOn_D7pQ.g4?euZ.ڊ3r,FPNK8SpVvΦnp!ϙHf4U&e̱4}'p37o<}sN 1'Ё)iґ*3uűmJ]10a~FK} .0ӿ:JR6tK mk{[pAPu :PtC;ݯt ==VG'``ϲudoa@KXӫ64ېlBko.28ہ-JrkJ^Lis _b2$GV&?9ge?3.mն13PA 7#wR&褾ޢkO-!gr Q滄О [/`kv}mV2o_@{g{jߖyz{J21}S陽]zti>Ƌ֍#E,^pwY[/xqՓ(0κ`C|&%}MS/'%˙ʔ{AFA=3]LFm#^p` e.mAӓg\><_]^3Of`ml0I`5 W) h~LKA*UO-'X eQ+е1Vxp&@WY`w;3\޽\s q '8P.F:dr' YA?L%i/gn=-o#ˀQvCRnkyXv~G ՉN ][vZ3!-O:a|y'<>ˊrv3t _AWc08Q5-K'7A   [HDT"3#d[ M}`2pL][0$,k/w`K3]ĶK+aA9#osmԲu=e෰eNTfҒ^=F?s̴N7ɍuJ@g_N+輦 <|^mcGVvuh+@c+NSUv{d{ l0^ʋY{tߖ~mΓ4iC˸m2`9mϖw(R J~~6h7 oH|&ewh-oC8vD>̃z͙>y|;q^SrCgT:?;mỲ>v9ItA}˲+% e!r\c1Y^鄎U/`cǃ]tl'Q{mch+@Cߓ;\bdٕ}G ^v.ʥn/mE}yemONAw|[ƣ?Ɨ4P.jn \W+۵ٹJ)Y:o 5FתOiE2ucߖ6'έ9w{}S(u6 ˍϕ ?ilx(3}UMh{FgVwZ~(3y= ؟9!mcٱI;23^nեNuRپnuH[gGW\[I.K:%jqH2U3m<xrM$G{ZKs@/$J1\}g0Yƴ;ϟTPt^og${}Mg~"+Ұ /d6je °_ ~ ݺoQo9ns[׳C^6|/T1Qܺn9'4VM%Am|'`AAAAAJe/x,"AAAAA*, Z/z?A@}p(                  zj/F_8>^n>_>҂ xC7oiA!.?J\yz?gY>vyj3嵕v"܁ܰc o[1.37XJ3տc偕zyT<'`zH8֞|FΥc  8*~~@|jy+_,?lJMk xYKv1(UtΓ+[.g˻o\,%*ΖX>b:8t+ɂ>|A3;k|2蜬y|Y9^zyY3ۿ]v-RMmNwc]]zeOܗc\LO̡A' Mc9O~fYh|^>tFdK< ̀^-}_(t6z>C+ړ[ pN`,_A r|> 2"g"m̮uy.l#sAYJ>=]KA r%` ~>LΛ{As茉Xp>䟗XlK'qY>_?LZn]U˪k+- |V[s?Z9A{/d,Xj~m{Ј\_=߮찯7pQ.gw4?euβ 'F9Q:E$wp9lխd)܁sQf4O&e̱4}ˋm{v77kqk I%,+i JG bic ~Xӫ64ېlBkwCkn`b`+H~>ӡcSt|myΧm6'87v{18<‰`p*%yt>cP<A^Rog6+.e0pWmN%18caf!GH՝cBtK^F>C lK:ʎ=:K?0g9zQt_岭$w9 ygar d]Uǥ.KN `݀sGX/Zq]Gm#t~cd8qL8a|-Vgp eE]-"kr Yٶ )UtlKr*%e#ʹ{_,|F U =Sw>H9m匍6Gsm6O UP /IUX>쒼WY$XO2={ؘ/cXiS[ V.iSѾ-Q~Y믺%rYcGs-G#Yk/Mؠ $VR9xǴr\5Ԃk{`eXe]`m?z-X ?:HLszi $%9ktY7:,׹m(ٛlcGgѾ~PǭNɛ&]ʃNz<2{ϩr3^;ȫƒ;:W~w/s!F/tZ>©cI'/k w*A|qiA|] >A R`NV@0fҶD&Ff&Pu'Y'9M/l#cu؂ a?\K}}MjIbȥu^J1!}n){Eu-cv7`u1cuInSrX=r#0^1Ye\7Pm`w,?xf`vӠw^6؅<\YҙgG-M]wG|u@^_Z){ܻfLOuzl/ =9m ؜Y#/)M֫d ~MM`6^ yR1xF4Pޥi_c@jZa{?e0 ? F1*uZ\`AH ɖ`(6S'qs^RtVɞ,#]>;I<_o <5^=i`,@JH0TU Aˮ!mW C 0)轃iS]XH/Nz1G ++ʀA'>ͣ7g|I鑂ʵ 8@C @'` o3mIŭ*N+#ީd]~Zuk\ 0/ g` dKX: Nc`}&ǴIQf#MC)1]Vdldk6}b0h;[6g[9ɰ3ecdPn9aKstlq0hߩʞ<~%ooA]YQ Xy9iyƱ#2%$ol8>/ϩm=f[msտ0Y6iCI߲JcE@,'@Yn<sZ;<7(:9_@(|-5[`lїX'T^oL ]eWp' apgkz @|`P}ՠA;EY,*~W,~~vվF2w)ڛx`;f)DtBgk ]&:k;2V;љvxlZ}:9; kA9F_<+ݺ_NS9WgGvQk9 =\\yػv*{@]4m1)p #_@0 =H1сnq@_emo D0 Ϳ6}ܷ/Y\n5Il=0퇲Wm86G۳fKS^nʑ1 jʄ՟46t<ܙ>誦YO=3+́;-oSꇼ^-elX*bl6E.uj=_ʦhB^K/(jJ&v*ЯYYY2l_Z{}dp,E͡Č-Mx{ƻuר큍W2~"GmR4GrXKthͬl>v(ڱ%`ehp]✷{ v2t6vK ]/ 4|E+A1 b\'{mkO h g zt:¼SLA*|赓-[:([/XucKgR&Z|8"s}Kr'\})z$]jg ~!+h_m~mZ^Aw| Tdq1-mA; 4W(hgR><%H09Or%o{ KIeC f躵.wzg\u쓲9Ri,=_~k@92eIu[gD_ 1n)[xܶ|mTɛhZ^ied[4O:˘Ns@z(wLqâ&]9mu3Ȩ6=Pu@`_:+ p9V}s-8\is*=:ļϲ\6l=BG$[Bvy~x!X/3guu|g!R\̿XQʋ}r%!@Po[ #ha=h%FM`O[+K\J,=z/ f9][VZbPF WZ4DyeҞ;}ƿR_!fz5.98΃t(Hr֮tlji3ŁT9{"I<\6Nx~M܆CKi$9zDwB/ɼ󖤵:9;C(X1lUkq*\nxmZrK0DB&l(i< Z) ti{.ڤ urk GͿHm_%z~RxG_rct8[_7[%7-`Llv1_]a>ּ,7lsqI9Wd+vZTymr?6aY:tv[4̇ eݞ:6zE#rر<}-`pE{wEK<-ltYK; ]5@Iا3 pY `f `,.Oű`/{AFxִ/m$MaNA ik8Kxإy2z^foZ}RI9Il!lKQ8* ~IqgF˜ ]>`[AMY XʋtKi 3;SK`l Pm J%*RΟ @ϞNR|F y o+Xidտ,'ʽW/ sLA|} 8C0Vϥ\8$Rڥ-MJd?cHz:SސXpyьɷ|!)Q0.g(BTM蠗| ;/{;DЃ%OOXtP\ֵEwIJ"x-:      8tY\)_{"|eQ?^ܯiTNzLJ.G_6R?2C> 8%y ʞH+G3ǫ[K/ꭷVT~cG    A|ۻiS"<#f.فT/D/i䐧_WgWaV+]ݧ@'i`htM=X9Oi1-sʖ^97{9=P~"#Bʵs0zD,    vۡ\^vnSjHDE ȬRbY}t/eYd }aji`d8w>Oo#f2瀖ۜyРL՗wA'2'va.Kk`$J^:ӳԗ5o hZQ tsas|%U89_Ӭzc3N5˚aADI 9GAfyy z~yJ v*O߼_޿yjAp<{*qsmQNf婕v`oVډ8rr>g&}do c~G_cy'+?,W_VQ-V|׬/<+͞92skI nye2\ v//u?~ !l߭}oz>H\#j ,#/L0>tv<-]z,=|U9nYח6ipn2b8 Ykq1r-/>VVԿ5=,_Z=ۻ@ySd^~?w] J<9d 3[. r %/%A:̖C/ HNfg_6dgOUZO?)$39:82];̡YzזΜm˾iy oO\v3괏N0#5V赌%8/l=ѡ@d K]Y<ˋ|T^|w|{׭W|+-d|pÁcyDztf}ZP/CKl K7Ϸy<|F+w"]ʶV^+;}Ҕ|8x#ϻ36- stF>;l5VZP&l.t/x윷g#"6UG9] Vz|6;¥7 хξZϪ?:rVA kh'4i:(3wwsɟ3=K<<D[`uNz +@f]ޑ5D䧤{,{N`,_A r|> L |xv۳'`/i/ cVu< NX=mzdd+,` <"i%00 |5o"Ȓu2Oۻ96f=F3X=ueK3Ǵ>ڪekYX _ZR|2V,v|Ql {vEhD{/ɞ؏o׃b>D7pQ.g4?euβ"JqbF(q"u Gr*2Vr2ٔ @#9;߬ƥHqziXu뾀؛Mv`Y^ ̓ [K4ڃũecX6\NG0YOv˾z_z%)U:%K n aOK}d(Q! Ze Rn:ܞW+S0_$ elmHk65E^3#H{F$^S[(lMڳbhLgNS (@ϣ^R}zmu}ʕ~c=A]t.#0N8Pÿ!XT|G8BL>:T>Heu@`[-NJvN~Yz9<=g^Dglk?i9V]Fr;}[ :.u\judМ4X/Zq]d#t~b8,L8B|-V'(tZRuI3ddۢ_X.ImIN#_8 r䯙t% Co[ۣ@J)P0]:,u`m|9b%as n!́n: a=L׺*vC؆)e<ngVPҖ s{]}idWGYj?OTyrʺG*m?4錇k6sl^)neFy.luљavF\d@΅(w:+7ySz;?˶*QYy_OUZ<-%}}}uou]5=T-ޫ&鿕7Ƈc86'I٠+me{7o4&Llwg{!XNRjeԕ^5doi{tpstҴC_Sߖ7?We(w)7ӊj,Y;\#]-YF;plɏm'ZNheE9;:y慯1ߥt `\jc-m{AIk"*[|admUwu-Ϧ>r8& WW\-sNʵ {iNp]ĶK+aA9#osmԲu=e෰eNTfҒ^=F?s̴N7ɍuJ@g_N+: X~u/yR I:" 퐺θpe\m|' yM*z]ʯ,{үЃ%%7Y[EǾշl urtdcuJ2oCΌnїtoV9%蟕k+p @8N@fږ[7%UxVpGSsr5uO2N,ax1># @@ؤ("P6c0m@jym]9İav]dY6g[9ɰ3ecdPn9aKstoB._gܯc3hwܼ 7\;j]+5X>-oC8vD>̃zol^s@z(w:ߣ`gT:?;mvG 8k3/k!A?[])y,h( Ṃ?/X tW:b XfXzv}^DJ`=~'dYv%+{{AI`zٚPk_x5h+u@ <ʾK]ѰL]ʰ&;؎.=P.\%x/qNjr:dId,}uvmsv<@*rtxrWl=uCrN;s^ sPzzґk5,3N"Mw7Nicv:x)p #_@0 =H1сnq@_emo D0 Ϳ6}`w1;<& DzPڠőBmϚ-My+Gfd:篓/ :}+VPrgf?ά<0ﴼQfOzzF\Ϝ6^ST7Cld͐ȥt!{MdL_Q:#զaeru0 ϋhP73Hq|+S=vuh+@c+NfrP==*{?  ._Ԁ^>"dzzֽ'4R: lQL;<ӱvpNJA*|㵓-[:[/XudaΤ>g!M4LqEk}wNRƳyց,%12߰L碪.3ڢeʣi$YLCҥvEцŨ}Ǘ@E7#2-Bsu~&>{3/ Xt~m$IXנh0i{B.zȺU6;I}3wf:I^W)_lu/?}T~ŵztYMತQ߯-shkӶcaG*Mrg-տ4dOD'p Mel'9U g=;vNadҶJ:|dmpu(ۀ ~ !tV+ݟ̹ҭa#eJ=YM6,16Ϥ<=3=H{GEL-t1J |iK3=n6T}ם>Pg  8;N؟;e 4 <ѺkEYN_jGY4w$aÝL~lG@84csiS9Z6x'`>jvPhyٓpUNΑᲱ~wMCkbx ND>I?tuByQ$޿yKZCS.t˴+h:E+Slۧ Р Ja0CJ`B4*tڞ6)eh]Q|Jv5xXcX%ɯ焲9d+vZgjѶpVsjo-bLIJyiMpI{Փf6R_'e=f_3'՝Ius3_i(O36x:tHZnmOÚ=6GX7ͯ=[iCn@xM'?l5M:.G  Η>6VE]'R+[C9A9.nlȻ.U"]CN1 Mp8AAw"; s_X%Q^n 3#`Apa. -KWR9   sLA|} 8C0V ϥ\8$RKGwGS}/8O*0@  A       87"AAAAAA+"AAp}ޅUWAAA  {~l"`AAAAW"AApXAAAA#`AAAAAA4"wų?gY>vyjg"=Β_>,<5҂xci\[ic@"a$G7߾X(W>Y *ӫ<>-JSbH:yJ4?6q竗eZ~~m'Ga93c{Ǵw~b::wj}$|m{a'.ǽwoyy2σ o߽ji/~K[u#Rj%y:OfW+зxy,6gK\yx\:%' ^nֺ^~\/;`?=忋_իO߼6߼I`hnVZ#a`: q tw;|/G]Fl=v&vX@ m3q8̞#l| r|6^"fbҤ~KX=ܻ}8ܿk|tY wP/F{H*:T^9HQ`rP aۿ]=Z~'PN:e(/] ɍ@Wֺ;e %׏/Vl B`)0VƞcX_o\ J|O[cI3FdnD̉u{|"6d{ėwq r11NM;c[86983q.Z?]"^hڛA]<_%Ǻw~u9o j݆]>s~|{m'E 4*p:JAnG&Y,@v E7vن|l-kO5W7[68RB,fJ]…폯SCy ux˳ b\o[_sv䮗$[v`Y\`_|ufG$u=Γog29eyw,{L| h+L/Lgz= ^1da{ws)~{cgn }y0+4eqz'"ݜ[Drq:Mҡ΍TN)A;bYNC=JNZuf;d_q%)?EDnHٜ~t;8ɹ.T$Xs K:Rٝgbyg^Z?\<\sj_QfkR^Sy<7#u-u[?G9Dž csНgt,p=Frg]w"aH b t`w):t>\?r^ O9+_Mc~a0{CԵ9Y<k~it*u19*?t/uW=uΎDŽ|@:w]_"?O{[6D<:}R-awb^k8O`:(4u7][@\2HDŽ-\'.i%|.Qw–g]PZ´ /mk d&c\v#v0Ѕ_sY0rЫ%کcƈsorn=R=r^ͳk}!Jf_6Gs^\puе9Z.֤׀^@N齺TllgJ&^3cP.p0Zq<̧wʱ$i Lګu\:% &~I'Bؖ/>})e$g( qZ!'8N\:"Y`7ԧY&\?`R|n67 L/[KiC cճK>NfF5e;kN }}CB>glxxg =n<뒕˔<^l*?k˘xu9pP, @Ab2À G,tfցk*r|]dh=W~nL$l Dx0 #ݞ6$N_VҺw^Leךּ f_쾪{o8O_`Kq*SsOiuq yuwݹ0gU7]3 7EеiLgۙ tMr)~6OZ~]w͡tL t֝s?\cmHeLΞN MPN郵N5z{ y Vaͅ P^o~yJ3QeL&B<(c.wj{%HL u~ve;&ֺ'y͋Nu9/LH.ŷ]}K_{oƞ,'a"y_l,xyIc |jsu×`n;:r>b|2~|/g2#$Mdhk%ڲN;㷣9LgS̅43Xű>t&dºhxIM>+_``@pRM;6Nv,Ў~]GmY*c&鬵u)Q2~vYGoN:cd׍ne-0|dqj2>Mi{OkNYt7=y5{Jmݟʾ16ef[ 9;:2u-Aҝg: u~!}v!@'/OG s yqL3aGr] {. %tֲJ4^IϴQϟEPcЉ\SQuv95Z?†6%Qڳ(r&Ho?Ui!Kkުa{<_kIY18&m޻yNw%cnOnc(lj>͎i[֞RL{rXq'}q)y3*\Ħ՛gVƍo'U\i?eE.̵yJ_hdL~gFy;.t21l}33ϖKmdii[O:.ÞhZ'U:X ;5@^~.r:0,A=g4Gd:,0T_DW/yPN+niiZenFʸNǨ"s&G̱}VGY;2m@_CjuHyLՋ]?ҾHA#c]1sxlϊbtG녧ڞWolʾ} 3a@Y7צvϳB99͝Ws闚XGoJԟ2vBZH^7Q[׽31](`~siͷg{yq~ܾHV^klHu29 PQFkEl~tf+'€]I'KRez2~8x0uÞgBoduY!^Gmݘ8;u"biC$s[@څ{WO_uoL?c`Apn 5$d=f9<CO3eD9=0{s&\ {1 ػ>>b:bbwLJ+_A1mҿ~po_! p9_"%pΗ @<&}F]ww}lewO Cc.5?NXRЉu(/._ |1p ._Gk?VbػJ]K]A#.$`AA`AAAAAA     K'`AAAAAA"`AAAAAA"`AAAAAA"`AAAAAAW/_-6s}s˳]/k+ˇ27kU|MҀM-ayHOO Nͩ<9o 8V婑G^u}c7/3~ ٳ3٫عK:FusLHlFʾKvk.wonkNzSbaO{B<>[\P;_%~Zi Yic$|yk6[[/<E7.f+;y&|t3;rVl_)2>pzvxL99ø=#>}d0= &Ah+oA1GӋg|bh ȲyM2/@X˺!ݩk^(zJ:=.iLomôJ'm僸-Oʹö,^N;tlA|@S za/ ? `^?Peϟ ~#>{З0lr v0(Zwֺ!0fC΅`!:mJny![pvp;6 iﵣϴ@:gCn<.s}?0Ǵ3˩le.C*0 d\ׇ+@߬9Gb'O5 HJ޲7{ku8V whyfw,dؾ6 vwdrO<5a _ZR|2V,v|Ql {vEhD{/0̈Cph^33 ip? ?>|?2_HT\B~NO'uA)p|Jx`z )#gwMe鿔rp6YA4(esnE:#*vZY]ayj4xԖiYt}6Dzw s)nP%.D h&i"@v ¾zz6@RA9x?j1\`:AX>C[?jH6u \z%`߿ymºnvҵO텶dr>nhy#2n9\~nm{//eKzr71GnWq.c <1imr & u'\X13<:'r,I!K^1%OnwfiZCYd\}a|hn:cMpwƹWn5dLy}moH9IK+ψ|'W{Ҟ _X<zS]KcEt2X6Uj$< Ŧ'>G˚16u4*+m5_ӫ۔ hr ճVz(QcRY<~]ٶbIzkga@Mmӱ}zC?Y5/y֣oǰy`yEVdk͊?4b.=5y6 gsj˦vaη5E'AqJ@x"%sksl뫗T j"F6Cy@RF5ץ6сO3ݨү{! pa=;;:T0'6aG9gG74,Mis΂ld4snE#g#YW^kn%+mۥ_l]#v!=tD<B3#] @ :;ȕW;2AhrA@Dtr`J=$ 8$?FN\nϫ{\:-cB/d,??}M?E,لlvmm-us%m }.6ǮL3;.엲=i) ǫ31{PwâOE+G*A_;o sknY37ܸ!eD[A с!{Ss8 mM7!try7hnw.zmwǚ`nruۨ1~Q=c Wx{w Xúx|Xۋo7ViQes|VsY7QOkz-ҵN^uJӉiT6K{d+i{2oҿ'^+$t2r.g5~ sot&Y7AϡYXљ7Ym9'R]i356:l㍕?G鬻=qٳKgtYwsgEnx92?MtUoC,΃2_ dsmH eZAbуY;uݴ9h]^Lq FZ`XR`-dP,VJ%˔_Nː;勣%en,:eezꮺ.诮Nd"Bg5Vy19lo^嚱ǜ$6ө׻A%[ m߈ PѰ<?׼%]к#xuxڼLueynU%\h]wsmY2p:ni~>21zrd /omOw1vt:;^m<7w'l <+}t5+C]&\5Oyc}w7M:Gg^ʧm߶[@_L6:w+udžlUA@ue@erظ,9ܟ7Z &< 9I"T);ccj>:]RN'W_g[+u0XO[hҝu ȺNY?7Ap9mϖw(R J~~6h7 oH|&ewIƍƑo Bo(7.=fwU'U˭egYp~Q2gKaFnOBo =MkՋmј3vt:+}";3?&<´aՓq_Ȣ~+7^c[<;@ܵI粳zrsnKOudb.F:J%'Űvf~pc_J݉`/v;s%] uCY5/ Ugl sf>:g+s\1׿I_gf{RŽlšf3=j:H6:68085= >n0(׾jРW뢁,xJ_}+^??j_aaMw7 j & `׽2(A*tΒm%:%/$NaU 9EAuֹQ 6[F^9g3g;utXsSzyLt @Ab !"Mmڶ|r g&r|]A#ԫ8 zDJ?aOgFl_hށ7l#`]]U76V_o{o-.@Gy,\.N˭[ԭk[w:*__:<r1P<+` JOamvX:;e`PT;9Щ[ ۪E|t:ub{VwƵÃ~le_ٔD\Tʉ!پ6 8P"wǮ(\ 9erl`θ tU(\nkpc4\X?hԖ|.toIZiACY2L#_iM_5J{5K}vgZ 5hQn Qˆ[WyAQ[Z C TQzAFڵ5փ:ik1=:KtuZO% m ;ʪuK3C/3Gh:Sd{*ke[ kqX*X#[_)Z090!)yKVK*88<k"'`}A/6Wls\ ak``j`v<9Ϋ<}콙b\m?E>Qu۱G}Iz~tGq;Lmۯ{g w#DYgwbAj!i[.:jy{U7t=o\j8'|Gu6͂MӃ;3~>9-'H;֣PfmxPm^[Rn_Jf3Xij3IW./&} X̧~ 1.650Xl}@^:iu@5-xP[6m{6dzߵXQK9}zuZ_J@_/kZCyh^R1?yǖى:g2kZ6 } l}qܸ981x!9\A>.%W3G缟 R6q)<)΅Ҿld٫,8Y{ Dl/ 냏!)Z#?Ao<&.w<[AM̳[&`GDؗ9UO_wN Α`?xNtpt%<-_D96~\M\A̳[cvJ狏 a_N6~!s> sXAAAAAA                              zj/F_|<.o\[i'O߼_>|Y^k^nj7OEz~ZpjN=ay9_.O<=p}b{}/of0Ϟ]ޟ^]\7gA!c}p]˹R#qƛwxK0 ،NpTk/ qpgׯwxTOmR%ONO]¥9ҙSn>Ͻ rK1u"\\yw."j=Z~GpyZ|G?ͪϧ_>G=ѓs|{#IG! .Au,}|[s8:} R> pt(e{7RDž^^sxp[oеp FgCd:{qإrʅO%-MKXuܳW#ٯC$[qrߊxlq(v0[ܡ3ᣳޱums"eZN1_͒s`0rO_Ap8Y~OIxz|$[P 0bX0lky^+hK#е36|06ln!/7r0k`? ?u: +=ڻ\LG_4S5V0`Ɲ h }uCS,Quz\Ҙ'iNq%#{[ȟ/sm5Y~ w,sI5ؐ4K ^@~~Z?@G}.02!/= `ȍ`PգN٭uCaP+͆ }uwNl<bm`׺<:c .؋Μ^;L{sv=d287gsL>V\]^J?$N,GC P7?-/3#&G;0Ç_/ @5_NO'uA)8xuRFL)փ wm>2!iQ ܊.uFU> Koui)-2Lme@R+,%7ݠJ \x{MiHE@?h[}\ucm89r~Ԫc`t&*}D7&^ X߼6a]h;ڧBTi{9u^P7CZy.?6콇i=ޘ[#Rm7ѫ8б阴6xͅb.H{y }SL9$%ӒSg{k34-Ay,ac.оC0>L7&k ;\vƫ_c2MO۶7OΜ~@UB@3jwBI-r??$\8VMuV,q0n3c}Tm5WhP7g:N/ZB-koQdm|Nsd#s6s+>kɺXv([_i.Jg+u& 鹦 zd%OAn/fڑ @v "*3;> T!n $0t\/H7r{^Sՙ %l{`yx+d&d'm\lk(icmuY79vNgRqamgMHY8^܃+}*J]>=W{ XuϺ9 )# oΘ9\mCm"оў[λA[Twӷ\Lvћh;[gGwկF1anN`4dƻU ;Gڗ k?2`m9k>9쬛Aݨ5cZ'Z:D4*]%r==7_Os ~A:K9G?d7:,RW LY,ж.4ЙCFy6OVʟ#t]Ξ8Yå3y&O9 C]\ $' ՀT^<4F1-W <ڞc'eG@0mن" | \MNRoK$)<ȴRW {.;8@Q_`9nhzs"7M_g[+u0XO[hҝu ȺNY?7Ap'˻ǟ- P0# 6`,fmвoHߐ>L:z=\sC'#$Pouy6-N@8㗝)s 4zIcbnHOǰwX9sew:w9Hw,.~k Yāy 95кT1ClwthU  kX;5XB;AzO궂vж Јwn`g9J Tҁ%F`eKKG[ci{kA jkө׻Z봬}66aÎ3;B1*)gK{@*GkKO{~]1ǽ`Ier&mYayi;LeeZ^VL6_TLi/sR(~ۓkgOZa"zuۨq4L<JȎY=xt!Yzk[cq 綝wX6\v֞ZOzmaݶћl[HrtԡX[d”ծOyK;Qaag.䲕ks}(7+Ys#;c G_g+7lO@ؑMX쟃7yVcV&AA`r^~-^ Js]4O髲wRgWk4,2.]W D]CĀlW#HA,I3puct K>0v/'A/ʭ< oT9q0$`r 09[A>k9Ś읪׃5crKE/SlyoWn+ж593'c2 ^)?]у &R>c.x:3e D ~{o`2@QQXmE6S(;BEi}Uv:}35>"Wg<@esk8Z?yBXۛg3o4^.F^Zꗧ3ue5?_)_Әg3v zBKw`nuYw~Dֶfbl;&˃WITݺ-=}>,TUc&mEo31g'G7Ao00/Mg-2d@G垲3` γwTXsڧ3-4<Lzl{k*AnηA؜.v0I~ 0%smG=0 5Y``&N 2h ~_f5b!v_p0FX5P%[9CS"| !w4ްS}vJm.1!:Sp%{?b5t?z}m~>1I,z_ $g=d>밚kD_n/6/FztKn0&j.tesWihyYg]fȺ5I{3в u#l]i]^[i===*bI+:Ϻ]57VA=1yV¶HftVNtobɏe:mܺ՞3B#!hKʄrncMͧTl?u`bʶxnۃHuo/KkC+\|q ;c/P?mcncXo[r.Pt@n'lu:҅O~dV։e h_dzwYg]$;GYƝWdfddDd9'F\YZN)u1XT!ٿ7<ԭ]q|~D8efBO9# llLlߒg>W?fq&bacޣ}\[<AnbCֱ= G༐ `PEVk?>Tm,c{۬iů RMO&q5^IBI,6?uy72o`g ﳧg3$?=ld9d`~УF uFs:I"j](鸽z7@}5Ӿ uWg isvQ_8ΪJ=oi#.LlH ;hݻ?\?%bt;N[A?Ժ՜6u 6vR:#A 4Xm~}^L>U[\=)1nSh | L&З0z{FҭiW6z Cl}O\@$m7k+q~qf &w 9fclJ=KjM@YKmFcJYû_ދֵlϩ6D_i!m5ř-Ӝ#lt6h6$ `LC׽~8'g̓I f\I>q_]58lx/o1^ v$£/Gpvc}pٛ6I!cn恤}[#o~. /we&~$ɹXd2YK_ָT;Y5!u,' `I$ |s챭ȇ 2nIr ?ˋ|qgLPcyJӴ/5/˙_\n.9,~!sX$%rR,I$I$I$I$I$I$I.,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$ `߾*W5<4գ[+qÇ*ׇ7ثo}/D{~[rn2n$ ۻ OD;Nܽgյoû^3/b]꾊۽7{b{_/eHxK䌇v.pqsg܋7ٚ&I__>ן~ӧZqx2Okt^,J>[dLJz"h< `X$=T#Q{`2a^T7x{Oې?xxϭ?~<|Ԯm |sF9x*lй_M85I~~%\ ʭqPh7||6Һ΄`q\8ן^}_D|>ڿv=mË?m옏3z6ShHo'0zo0{osOispN-Xls G=@pBܻv?Kv} .G|$봘/B!Q~D^~%'Vk`!6zym,óޘCnl/OJo6.AN0RHz `_&EW36& P:?E?KZebajrxl:ksiN7JG\&[_!x:/t q8@!j4^X` O 0/X_(F% CX?%~ Eb϶42b brq 7t/oA7 ?pYg=ߖy=%Ps{SL6wer1}V9W>Y];vNXKW/gɹ  27\;%O+f(jwgu/qZ@0qV]_\ٱo6/pr. CI `uK^@Ojъ~h}~zЈ^O/$'6 $>n<~fIx܄B9o&7??Tj$<ȃ(/J`% ^,>!e0薱RڈG6#tP\F3 =ڌc}@& RԖm)l}6}V~, \xUJ+L>+Lyn/~[N$.z.\gn{i$U듢VO(bLaUM/ދ ߾9a۷;ާB2E{utPwvCx?N9쳇mHxu>"=w L}2L? "6? 3,L؛df'ʼn.RgFzvP_h_nd/]/5/5+T+"6 ~[{S͡H_D6\4 MPN]CQkC]a 1?$0(8/t>?cQ|n'^^ `o_S(gu9 \,MT,un8F4 yXC<'/(奒]5|_&!?x@y,0tQf̂n$vnې(8vhZ8/e wHKc v+6 Ԙقd\nZՎ,`涋\PQe޵ zO)hQ)p5"ձ0Wd/1j3Y,+Ӣ /^- rB;s-6ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAX͕fp?8g .?8`SWCʪ6s=uIDrX/HUC `WW?"ꅧQ\;8)UQurL'uv{Ӊ; %yʛȌ"+r:{엗!P@-D+E+|y&,ZȢD/\(XRu*/}wyG%ul6ec"Xɲ WW&LE]hY/=.}^_ҍ}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&Xγ\ֈ ` 9<*Hs;ow{o]{?m)[r<؋Fcc,:Ooo^uˀq㬂FہvlV sqN񋔩)XfK8J׍2ebRd?[Yi} ξ9R6A)[DZq$>ӧ~BZHԊU`o6 mGohL:RN&G~H>P Rx$֢$_H/(7]t)[!Z|WR|bH'3Y702.)ׄ;2Ƃ56fd=[c"n8,k-wRr'> O/lkPt;W90h^ab =诅u"*ڙzܲ@#ᾁ81V JqM# ^NAGFIJпڗ_|0Η/JPgwX3Vsla#~Ѹ1nqny׏@ܽIc{j?99mىa=M-b9Eގ6 ȕJO+L]mftZB{zS:a ͗\vm3qzq+.́GLjm6g}2s636ydVkNf 'Ibr^sТWʢ,Vx**X)x51l.݁<i1`<'p,{c21Tұ>,IA>)pć ZS+7‹AV7mLb+Is$8e<"2qvkB>YO$I. `?c[͑%%)e-`7"65ܗ70 2n/߽;Ƨi _j^*,3L<\rwYCN/<$IKX$I$I$I$I$I$I\YK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$I'}UjyhGoWV>ToX{Wo io_›ܜ;!d$Ir @wFv>?{oϮkpÇw/Efg/_Y}߷{o&+)p_lXiבkuę:Nu 7dG7yWh:|3㏇p@t8᛫Ⱥ}Q`7j_`h EtP젴bǹwؠj|6Q77{XIߟ~'i^od|tdӳqBDDz}>${{Ü{Jѫ-frxaΘ_ν2x>8m@Bum>dxx~-c<b:\9Xyp{3/봘/B!Q~D^~%'Vk`!6zym,`,_0BP__A_4ބ>߾'s&Ihbqk)$= `7{ߌ |lxA)eC)Įq[?8[%Ij^$ѧU>Mп/sas ]u ח~:ah^UhE/,D '&pjjr/vʂޓD@p1 U6b϶c `ʑz]w0xvnkr/oA7 ?pYg?\6n<(9)w&2o|>+W,Ɖ _f/Fs ۜ#ʞ)3&t9vw=7}G]<x{}t~8,9->dΕo,Q;#' Líkpٗg(y/zz?YE+V{~a[sR@#z=ݮ{\4qC\7q7mΑHMn&`PM=Tj|vO.x鿼!?}HnpFJ+Tb̘&iHEḒµzv Ƃd ;Eݢٹ肂،w{FM*o՜[̝t|spڏ::hoUe;!c<ןp'gKy[w<~:]} |>squ qq&Mm`3'җЋ6|3ۛm*Cy^Xk?t 6 B`MX1)@wDYOyi9~%6,>vLg)?9[L=c?H;Ў͆ !j{+PwUc6zҵ.<ˌLm~ ر/Hk.PtJ;5>lE7㭯k1=>zdo> e _NbAMϷU6L%~찹]8[ُG٣czZgJ}_rxUA>ۈA=G<Ư|/ .m-dQrva[_^Oa,X|5x#_V.[g,pMBOZ܃c߶t%|3 JWE)z-z.m}M*բ9 `&t >nj4q9xIKBVۆ!0^K%> R<Flf+u3DEm GEHn f̂n$vnې(8vhZ8/e wHKc v+6 Ԙقd0a͋ Y_vKL!ŴOU `N/ѢF/`m.MdS<X~> a˽%ANryN9&(cdns-{gy\/[uu3R'0{"cSd<\%66hz{˸ $ _zֱA/ گ =1vn&i2&suao?3{gf71wkmcEŲmq]kr>L`_1;AP({R=՗UkGab#d-^BUW]V䐽ʮmM]6/\+&s#Y_!sexn'/Ѵ|6.oM_=:ϕۖsYl0t6v [;]iC2^՞p@`'bC nf{+̰|[ 2G1<RI_>%y+093[W564G凊1(cP瀘yĞKc`| {,|-Jwwٿnl\!fnz7w]eZ,_QϺ>X%ˋa adFĦ8Gt_D(g>kzAJZ#W/<10l1N:X `3 8Lp^%V]R{Y ? r\#vXsg VR<πUX ///1ʞG>Q}qcEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0XraՆm}ا_~OP(R+Zc _I]g] lE+w. F{2` /g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښwpeB ^N˼3)~oPxuu5O!:vZ3hr39b Ei:`ۥp񠷩V(shiyTom-Z''Pl( bC|;07(n Cw_`ޗ9-@Yg %;c7}edn541/cLEa/=6 `2VaG}oX'{fXйR؟}~W0 C6OC_zb\#d1 t+=9@q{0o}^sТWʢ,Vx**X)x51l.:\J @BnDc, y0OtF,{c21unv NzCp`vs"7ĉŤ_8Qo\f /FjOJ\IL32fgZ6/V kJoRPE$ȡ #?'UV!*,Ll\BۮbF-~iP_'m,Q"D5fYj SW̔@c ` mrDx{rwC:V8,2MuѮEsz4<~Ytn+g},EOj<8SC_ SۋK}y;]ׄ8^i4˶Qzq"73azI2Sg~^C|sYy99sgn0 j>o3X*1pc_M]f;Fl]5ֲ+)Rzk:~VK|MY6kN/dvk^^Q?WbXĒsGP610,UYa `fd'ss#`[`'IJ7ܤJXyake9(WxoAV=0*$I_9T&?@<{=m Q1W;o*Qpyd~?.EoGvmK]Y`س+~d?[D2ccJmqZ0z˹]vejͮW[oV'Şj,K8Wr;_l/.`os,tESۈ\)e+sc͋7W9?Ǵ <̼O LعfaU25u`"{!g8p.ue-[Q,qL0wÕ+f󮮣 cA%uBI{扯 Q<9.e۠lr.&{%ARऔ%`-v/csR cݓj>o:t]d[_y?C+m6r#mE*1p(a-L>SqPUl^rG_Q{ E>7.M\S)| ևTjL_-VǫS3\ڰk_g]^lM `U9NK3@ 2ꃝ/s쳙\2?E0?md|؆:9$|. Pgt^}Qmݍ> ci߅ẫ39/UygU%~}xG ۇ뷴 @_s}& do6mWk`1]'̭ j]jN:eʊKau) Iumoe^HFeRILYMM[ q$hns\MAK>G\9ρ.#ץm`ړ}Gz]'d^y`Mw1O(gZhT.۴?ٛ>v{+M@͕, f|ۍX[`y1GΟS{ .=r0xu!?!l!K⤃%Ip9rsn_.58l˴c{eA=R's=>bhr(f~/ `ǒKdz[!m_/~c]^]>jKιc˸IûFܾ}=q^ @ٷe#.ɛ=Cb5GL2j2.Sω{}1&XK>nvsub{ ;xqҾ>S3ʑs?'gsD<$˼q0\)}ww=?7gq[J]Ű,gX!gy̾NIPx-hE1GaWR|bh mW)qۘFe@Nɰ\W0 Ȣ|2quk~ė^ dLx#^`6\[h+>R6| v+LħO`r`NGf&o?d>sR0te.KssQ yFadi*d;|~G/|YSP4HPբ }ka;d&Q]yvl `;C.|mYwd]| _r9~`KsR3<fr-ݏ3.w]:1qnn{9}q]&ţC&swkNw|wf wv ^\7;;{"]vcyeux_hXی-,|Үbs8xKw"{[O=vDʖcXʞ\I4QP݌xٿ.:2> *Q>kLdM"q޿}Wels4~T^>VLX r))mSoSx[Wl5Îmuhvgߛ1b@+A~wL/j,^"P|"h7&1Ÿ==Ѿވbتq ؗr,Θ5<c2Dm+6}`ݷ*6x95LkI޾p9;7 u g1gψa:γ0(8/t>?cQ|n'^^ `o_S(guf0,B+A2g[Sn>tA~F(@%nZ,пDMEQ0) ٍ8x,hP.A9eQ'k$ ) ?9se#nOC.]vKGڿSeFXIXꯙEWKNZ /_/ʜWq; `rmaf6=_LIJg9 `>.7Sߠ]u|,_9̆mWoc1>wN>)fԅc۷=ݰ(cɺ^&cfgk :x?t:6bU5^Aŵ">Am+̈́M7]wjٶA_6 kùR{q0[W<_ ai#;<5Vx"Vͤl)K$Aق$ZK\aCKKsCWn3h66c}[Xz7&My5Ƃ}Rbs-MY+>tԳ0X]?krl 'Y6l]v^J\2?Mu:_35kH+_NnBLCb l^ n_bN){ b659p=\EogM0>YՅgy¹ٺ"[5li:.FscoUf^Ndľř퇎sdysOlmǝ RxUmH?zi׎)a;q `Uw*t\+s<ھA \ pNq `po&XԶ oVǼhײKl\YHsE'@Ɍ< dh`@ B Q]euqe}0օ,o79Kɦ}Q_1s72;֫^,v|_ _na@nfٺ P_bunuվs3=C)32Ǿ;cX=M'VvB S&̽0ma`[v׈[9y_6a,Lƿ*Lmf5S`,ϞQTB?(}14~ "u:}6;%ܺw'wE<3FFe8~>_75=RY,pY~,8ԋsy9hCrTO۫w:c` X1_zsP>Z7!?3_k^lC_Ab8FFӏa-}ا_~OP(R+Zc _I]g]JƝAZ #52`h ځaa' `f78X ~H8}d{` =/:xE" N0Sc]"$e.:':w &M+6jAɖLܛtu][o`@Vjf|0aKnc'ցaqDeOf+W]#~7eMk-_ ߂a,L6&L7 ˹  ;=o?D&kc"7LJ<#XE3OS ?q _St"0AcAɋ9?;W\LښǂXKe3/35y`b9[O'Kq<܏@WIi11g_}1ctlšoৣ/Uh!ʯ~8V ^|>~cm oẋ>&jA$ rn* ͌1 ~֠Le7YO 5 ^]mnXc'5\cʘ~PzIdӾ;M_CD)D^g1. 2+֯{ltA~ 1汊b!lUά hb_ϰ @ch(0=FW}yDz4>3G9G ^ `ܬfڥ陠Elb٬X>uɞɕ=`Wͳӧ*Tm+6X 0ax3*FB8~j Wl!q L'_66k-;ma]m,ưk$MYѺ͘{'6 ca2eS^k~=;^5o>ԃln&?c `{<k/l1rB2o<|nۙ=(oC˓f,õR⽹¹b8F&KcXY8~lzwHC92l#Ң'*-(k4/ ;) `x%O .<by7*Flvc|I5o|H˰0>쩒-| 5j<˚ k7l}d.ʹ5i"]ۃ&r: m|ߛ7._Vy$8Q 0m/JE^Kba&[+Ivʓ]wx)u-Iٺ_SU7K+>vdO! r >-"'TVreg,oPC3| j]kyV~}x' ea#K5Yc-LmZon:A1kbx+=c H6{7}b}o ĂOx6#qiugWy8C$>fj{~3m00=&ޔ-Ǔs?*~.[S&'$Δ͸}ho.qAœ{"{g Xİ y `N =AP7[:4q$ܔ1kPgtX;?|l]wۋCݻ$mv}l?i,kqqs_Y;,iՍ֨tg¾bXol\KN82Β⸏/aQ<囐^=As6WW`^6ڴ-/]I~Iqs_'&ϱZsIg}[~8)\:Q\*_C^J {,_%I|I[\WrE;gg_3Ì6\x<="h' Kc3YuR,I$I$I$I$I$I$I.,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$,%I$I$I$I$I$I$ `߾*W5<4գ[+qÇ*ׇ7ثo}/D{~[rn2n$ ۻ OD;Nܽgյoû^3/b]꾊۽7{b{_/eHxK䌇v.pqsg܋7ٚ&I__>ן~ӧZqx2Okt^,J>[dLJz"h< `X$=T#Q{`2a^T7x{Oې?xxϭ?~<|Ԯm |sF9x*lй_M85I~~%\ ʭqPh7||6Һ΄`q\8ן^}_D|>ڿv=mË?m옏3z6ShHo'0zo0{osOisXz=9|vXo)ŗvmjU7d|h}d$3ǫܻv?Kvn<~fIx܄B9o&7??`T~'( (ZxiO/iRf3эnK) Яxdc:B%yl1cȺx1tim:,?khπ/EmV2:gnӗ aRFKo‚M\QEc_-# 0 ٹFE)?/2EA O s8ߵolK& \i0{Qw5'ls'4_\ƿ(y㹎7|n}9Y8}R2-x>GnbWq_@}tO|܀ba]vAG}fq {S,،lj%#x >lf=i[P~^09"an]Mk?s+m|L|$4ݹ86sS^_Z/lsG}W YU+"6 ~[{S͡H_D6\4 MPN]CQkC]ad^({;bbB-xIXJHViL .؅.r.޵ zaFhQf+^sտenVΌ8;x2 5?@0I;d%s q^yαe<]?,r+Nr?n?@d&zOdw 6{*aFMo|Ăk_Yϳ:61f~Ml mȮ>51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1dg +g]=/bR`kf0ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAq\ʏk6 s&z#3 :e|5jC1\$I$g)T-^>F~uEc[)R^xŵc `؎cXu ]'tRgHoq';HCP̘!b+8-t묨&.8VRxp+{1 \O"b7*`z{x6m%z! uzq1t `uc\˦\'QjZm^ V 1F[D+TJ!(`uܯ c~Af ʒ:{61.Yأيid7%c8/Ǫ 8wti}~π9ޞ9*ѱ*gTr+t~#:lx؃Ys}F>}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&xyYy.PkDlZcy)\'5vڱqޤ1J?5(۲S<x.Ӎ:=Ytyuppԭ2 /\ՍaY1Ǎج:q!`)SSz l5͖q+-JLř-k#e3u7I9o/?}'(AAdMXEVfsбvT /.%ћkdrD|@E I^gLX0Y `[+qC_}@2C{`]O)|fx7FƠ<@<嚐cGXsцl],uk ] ǐ} ==eBPYJ'>)^xQ(Y)XBBXE:mULYɾbƘ*(ZxCdL⠧w Ʉϻ%lm-5Q0A^'^{jӱ*gFgy5VczsuJ;>/g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښge_vQ!_4n5F[`y7woXvO~e[vyXϭvsXFѣ >rғb SW[ٟ0&㞞T~lB%׭] m\l\{Acs8c~$81bf?bay `2Vo̜O@]X`4δo>f29 $Y `Xz^ P ~: `0nk@^*X쇣 `h8ưw\O|P7:ĀIx(R$aPߐ?4ޟXY7^ Ŀl^^qcHsC`1>~ײ~b]Sr};X~S `gd&$z aD_0e[*YsG"JaE8ްzVeG/t& `ޗ.Z3@ɑH~0riAX3\ȼJ6dFX|9dy&Ȟl~o=^[_[|fLdk LGmVnNGg'_NRo/Vo//vn_xX_c,Fiيܰ$ چ!pX\XZzZd_N k,kmB4_@<>q{oXNe[v}=g7E5>c |".tZhHFl7ɛ,6ۣm@J{)?8sFL6ÿzյ}Z(?:iF{j`N)]sp$6+]/,4~ Ű%>()lv_Y$ a?W"Êx&i}/_ZUa)̙mY.~m`-4=]:["еKoP't]ލe| π쳙\2?E0?md|؆:9$|. Pgt^}Qmݍ> ci߅ẫ39/UygU%~}xG ۇ뷴 @_s}& do6mWk`1]'̭ j]jN:eʊKau) IumbtoƯI:;ɋ>|\ްb9\߾7?mɹ9wC ȸIo/>|8-sޞ]W׶z^{$^;w*oL6'WAS|}#旱Ү#y ;s/"3͹&g7s/~ߨgk$I'~}D\rOkOǵV@}^d^~Ŋ\*b{^6PD8C8{iS!s|͏79:Q_10Nmd2ٟFA3ڦ@7h#'p%&T;{(bǹBmK:_p$_~x}OhO/zYc>:|w2krٸN}"">ca=aQc_~$}T]?V},`:_O*{Ϛ׳w9oxS9ݻ}{p̘B^j+>r=+4~տTUVd3j Qؓ7o-DZ`}+z ז7C pgBΎ{+Z4n=Yj>bqg#J `uZLoD \u(A?" O+5F ]@mJQƼ6 ]GYoġ 76WЗ'wy H7\}c Ln`' dV)$= kcy_&xmX0[Y^ 펬U&v .V~)> 0Vdz#tuo%~;prONJzA@h ME^Ȃ}BQK/׊2 zOáDo,_?dHtLQ-[ڷW.۷dg j*zuϗp8j_8n~#,΂;{h-G{Kߙl!bL?r\}ܻ|wYl' TB9w>5@?~9f2~Sʕ7A_t*Khc&1Vu:/Jޫx6_[s2sywra=|ԭo˟Ezϭ鹛UP_/g17`XX]Q@gyg& 6q"}I(/lçO =YqV2?W%LvCl-d;e(9 ?}wn/N͜u/_ U]=;yyZRˍ@%H!&~q `W5o+>uo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AWWD' zMJF1x9PlcMu4V6s1b=1 \kn & hjͪ?T\ q|Qcq`ڋyCƂfr}~1gǏ5U$/񣍏-B̏=&I8s N ].xX"[ ע&T7Y-jFCy?$K+&*K3P=(K, ]CǴ$6:<+?u3ZV:7K_l#RvuQ; 2Y|7'6牔M1Ui{5+_s˘({T sqN+#<_18cf~pV5\q&cZUm({$,^ū҇ȯhl+E* OvL qSVN@  :dwJ7s#DV,{cw `X}`|Rxp+L n]'T1n yn0_Er=`=Z@ՇstngȐ:m`Ӹv01ef PYJ (RBE-H6؄ H-WNcNZ:_UF 3eI=M٘f,lE4}撱qlcUW_knbo~`gW,.={&mGdFXkѶ:x,}yMTӊl\{ҁc\lo:ǞFq(}==*vЗ7۳]cǦe(=8[bw!'9=/s3*BƳsxem䏣t`y4r_+6ÿ\i~ڃ>8Ӿڝ} rlfbSS&I-}O?>"((,a/l:vڎ*>uvݥ$7zsL}8I" Z_&_H/( [+qC_}@2C 4Ky"%όfY\2{ b K%nGl_*Kɝ$؇>  1 x#6K"Uh^b"^'7+2p@WSEbȢ,S>6ywQĔ-ƝQ\;s6|)X/b4*C^cU0[ͱ%<9k<66w b}^. π5@.K(7#. K&UMƒ8  >̀2/k:p {tX?0N6/R~le{:7"_|Z hŲm\ bƵ5gܳn9g q=*d0/7#-0<7i,;{O'g?-;<ɹE,{цbRt1r-Oy\K]hqOOv*Qa6 b֮m6Po6`٠9Ph1?13 $NJMbI< +9xƙLY5'A$19K C?K OG ƍ_x hѫ]eB+<_pT~}| Կ6u@뉏` B4F0I81E*,jՇ% { [?fax `fU@x1F2{lx1 P~zƍ!bp Wl Y>sc\˦ŊvM`yN-8‚UE.X}”oWohdQH+-QVxþZ"vVZ8Ӆ0 /d4[>_Z^'G ~9cUΈs!"*TmZ4WGc` E"{Dz`QzkZo?m1k-3?Y9mI|;eH8[\Tڗ3ۅM|MOc}lg+r>X/h_ KX"S.<2ۚXix6|qޤ<{Oʶϭv{X|o7\k}sE/\v%|?nA7KYmG5ڀ:<{=m Q1Wo5s "4<H]\mK]Y`س+~dH0d{]_JSb\.;HгfW]0鷷hgLkc/KN3_~bX>Kdߚ礽YX\4DyޕEٛc`fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQmYw/sbkZbq_,sŸ11e6Wj N8W/d:d³' d Zd{{&ɺf׏õR(b b˵aCo6l?)y[ӻ(bqˊ̈́r/GgW_l&6So3Es 81lѾe.yfG-I+}GO T;cQ 6+-Zx{AO.FRP;{K^c>FB(O8~j]z 7Q!_:KF*TQUm#j:mf6+Drځ/1meN5VP6S?D|;93,NJe-w: KůF-0IL6,6HC|M._SbSڥS `(ߍ._bƞफ़:_ 1v_ fH~{:sɾs4G c;;tDպ 3@Q<q{eoDv7,k}Ҷ좾TqU(t{f3lF,}]j3_l?' ِvкw~^AKfwt0#3u9mjm++.ץtF&1if]|6+&,:sSc6(n9M?/a:v ͥ[Ӯ|m81&0PKk/H6n5^$WkL$ssb3@(:y<{:"0Ԛf"uیGŋwkٞSm84e3Bڒk#3['9G؜mjbm/I)=v9{pNΌ鏵uvA$}㾼Lb+Is$8e<"2qvkB>YO$I. `?c[͑%%)e-`7"65ܗ70 2n/߽;Ƨi _j^*,3L<\rwYCN/<$IKX$I$I$I$I$I$I\YK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$IYK$I$I$I$I$I$I'}UjyhGoWV>ToX{Wo io_›ܜ;!d$Ir @wFv>?{oϮkpÇw/Efg/_Y}߷{o&+)p_lXiב[wOΗd]\{3oԳ5sM$ `|"?9pOvᇧZ+e|/2?~Xj}j;÷طn?}bE.ցyvx½h/HD{FdV !9Vo>Q![x]/'jk?62GsUn٠smS pkpٓ؃K=[1{n6geamu /fq??߉}|{g,1>;59gl\>fO`1 `0zr(R/?>r .>\m| '{w=}Ɏg}|ٻ<ϩ{]Oyؽo8f!i/Sڋ{?_XQײp1|>J>bqg#J `uZLoD \u(A?" O+5F ]@mJQƼ6 ]GYoġ 76WЗ'wy H7\}c Ln`' dVy+(6y/ˋg2)2~ϿѶ3blhwd2k05hp9'4/ H,2}Y7&o $o>|OE&U7 2?X!?}H=D7e,+o$ă@6⑍T<8LbnC6#jyXХn@>>e[EʠnM_F7UKVh.Q 6qQD9"yn/~[N$.z.\gn=pX)`?r,̥׾-_ӗ '狅jN-Ni9Q^so*Tݲ1spne~[-;?|nHĮ⾀B>S9ӏº삈 886YKGE|e>m}zӶr/arD,еfwo!0&W,FI is{q"m˧~9 qajgyʙZWKo^kr*_XԽP{h"g&(~mQݘ͜Gdl&u9^5W\ҞlS'`5]W+hQ T_@A5me|ZG@S"hg0 r容/@.\}'6Pq s,ٜ\i]6bΎk z/#I,^G[慘{`MqZAx]꟱(E>\EMַ^oZ:GI&WLU:Tg7#yGo(%.!cڌYЍmsrEGRVX+l)|zɼP>vy33[",BӑvӦř\h 00]R]k,B¢\g/ѢV-I=H7-S+gB&z{7!o9+Efޒ '<'rks127y]9ǖ=tLƳ<.ʭ:캙 ǓY=1|hj~gqq4e\ }g=Z ظ&c\A5!ܿ'p D`~c>a[dN=-|q&}MmplX1aMnǐ +>xz1p `V\Y @v~'{SĬYŰo&N[{ڵM]uʰmT7%փz=˕t>ٷ_'a ډ/e,ʏ?X͉y"et "mU@yfd׫2&,ib3CAX͕fp?8g .?8`SWCʪ6s=uIDrX/HUC `WW?"ꅧQ\;8)UQurL'uv{Ӊ; %yʛȌ"+r;0VԊ `]/=]nâ͵<$2*ƍ" @,ѿAh0VbPwC\7Ƶl:ތ*K}yZEV8aIcEJթr P Vл<d,g)삟=OVPz\26rs `-M_x% >039ۓ#`>] :V J.roD-u4:O{g -ұn,JL\'5vڱqޤ1J?5(۲S<x.Ӎ:=Ytyuppԭ2 /\ՍaY1Ǎج:q!`)SSz l5͖q;re+6ÿ\i~ڃ>8Ӿڝ} rlfbSc&I -}O?>"((,a/l:vڎ*>uvݥ$7zsL}8I" Z_& ^Al|E>U̽/Vv.'axR>3n e}#cP~brM#c,9\hk.MS޺5.cȂѲ~w,%w`/<(|,ڔ_,MPXT}x!SPS"xȶ*ެd_}q^1cL- !R&rLqӻdES춖wFq{@X: /JPgwX3Vsla#y(Yb$yNXdq{oXNe[v}=g7E5>c |".tZhHFl7ɛ,6ۣm@J{)?8\+#}bb>-Cti}So=z5y0ͮ98I^^Q?WbXĒsGP610,UYa `f& NJȘG$yz X9&E,yh;_xXS*n<ҭ݄˛󡢌o(xqqM><{=m Q1Wo5s "4<H]\mK]Y`س+~dH0d{]_JSb\.;HгfW]0鷷hgLkc/KN3_~A|g͜%gos,tEm"WJnI8_;ZPbܡ׍_i_ |Zv5՗ `^c!4Dy[FWh4eZ0*B%*Z62FP98FhmMoBO_`.І9]Tc h3Ct̷Ùoh;Tr%ny/9xy7+~Lst[g.v"Qy>ylCQ`gwHZAx3a:n޾(6F|ʹpC]ԗ*μŁn>slC[߈K@m>d76Z+u5Vydz.`5}Cyeť⺔H$:6V:(T+g }b3|..\wm4[asx&{Kcbme{Bsƴ+_hr=̅?6Ծ'Rk.Zz Mk5ɕ8Z8r;ܜ Pv3o6r^%OŞ? &٬ȥp]6p}%Ql8WjkٞSm84e3Bڒk#3['9G؜mjbm/I)=v9܏99993?nM1{J%g{)|Rx{$e~9{޼I$,Kv>x4$ysDXx/.ӗ0q$IE&XƥzwsGdl?YK$DN.Lcm7G>D'Hp߈LK_q_^68eS x|=|yX2rp9le9f$I. `I$I$I$I$I$I$Irid,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPd,I$I$I$I$I$I$yPTׇWz~חݚow}8\_Ymo>|P>a^7 xsc?޿}! o~ےssq$%^}"pZ=m޽I|wx!gRU|lNSF}/c]Gg;g1ܐ lo}‡v_:<AuxGϑ_#?sxg<|տn{߰~gp35nwי=N{wpN.ׇ>Wkx^bQ/b&}>>׋D[U b,&M?b$Č$ [!sIJ{?M=7or 1Gs6gL}]ٓZbBҊamGay~'ë+zhݗGxzx>?|'͒ᓿ_sOu my~F@ f s)mwcޝ657ڥ?ܧĒ<#l}dg]7?-Cƍg}|ٻ<ϩ{]Oyؽo8f!Sڋ{3x?3}y>h)EfE,O-:`]JٍcΜŞq)̹pL xE}v{S!&5ntWZN{wpVb8| $ZGQ yXxzuX0Znx.Urt6Q:6l|yA}}}yx*fϙ$I"eWJ䳯z~3;K_h_R6rHnL'UK8,aN}*\Cd KR=wV8[p}S&PxņZ,L2z E8+O}ŗ>NY{(=&zcagdb@gUΕO{gyD/gU28cəms`eO?lzK x;>#.?_>:?csx27Baf/v v>vFN[׸g]9/;P_ZR-~VV6G볷礀Fz]~%97h↸&'n~o3ۜ#@$ M—z@\ry)FCzL6xꔱb<āدn*C6fszlYW3.mMv[r m5-*R]tLm2qX*`aD!,\RY+Xp_:m(H\ŝճs=vyһ 'fsw ߲^V"M* /N3ɘ0X,.X(ό<3aoj8$}^ϗ\'ެ8m+P+&G]; k` |Mp~nŲmOҟƾ;'fκ|K+,<ŧѮ=<Vx!:1:.sƷf{2tHv /#B\_%WS^ƵLjm|٥ k]vsyk,mQ@ljc_\|vk3 | Coo9+[_b{|37A}Ă?̗omLIK-asS_ qH]YczZdIo+|%1~f{yNpټ?mms\Fمm}]z=E{oXcq;pY{[ncnбҗ6\S+(B ^3Vx赻5իrVZ(TI$纩ipE%- Ymvy33[",BtvmCNqL! ,,.rRQzׂ'X*=B_EIUqLzmXn/ka,D#9cʃRxs[r䄚 w礝[nmy9F8K<زGxER^']7 ux2 ';=MV0Ub#nwk@b/Yab͠@ڐ]|jcaxo"01 -n2W>Ӹwnvs&6k8[W,FE&7cfS6OWVκ*R ` r1|? ^urK(׊BUm+kEdn6k}_vv9~s,<φ=N]}}̻dLr37|-ިg]0q2Vh#bS#/T}m35Y ` UW-_]VTFqVuGB Btt&8|.)I=[QX9:0c=l^lrh7 p>7dv6n]fOcqDBqڨ>Ls;E`Ӹv01ef PYJ (RBD-@6؄Y+-XNNj_zW_,ŚafS6&?GYi7;c^a]frT^A0ۓ#`noDǪ!sPɥ^F^g>`gW,.={&mGdFXkѶ:x,}yMTӊl\{ҁc\lo:ǞFq(}==*vЗ7۳]cǦe(=8[bw!'ȗ r'Y^b!{ğEYyFhX1hCGhwյ9O!:vZ3pokOcga [ڌru[,躻lf67ö[GKԛ1tuѲ$$C6' krb;0@ [WgXrUcrx^t•yrZp~ol} }ql({fEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0XraՆm}ا_~OP(R+Zc _I]g] lE+w. F{2` \'I_%ukBfcAD^ui֭1tE,Gl_*Kɝ$؇>P0riJFmJqbR Al+NKR^'Up@WSEbK7,jE.\ב*SoiFKm"VCh1px9hUwX3Vsla# hoؔ K@. \_e3yn,X]wyĎIb]X"c9@Je;c^X, l<}M[5T3_krϒ0kqr\b!5Y `Xz^ P ~: `0nk@^*X쇣 `h8ưfs*5y0 Y$Xsc\˦ŊvMpR ]dgtѥ$V~mlX}oWoжSH+-()@Vd2qDʞyxa+,ǂ>X'G ~9cUΈs!"*TmZ4WGc` E"{Dz`QzkZo?m1k-3?Y9mI|;eH8[\Tڗ3ۅM|MOc}lg+r>X/h/3%x5D=Y{􇚏0=J52ѡ˷13q C;=f39zZ/@lfv@uڃ#QGeS; З6d{T]]$F8)G7YsTXIslL3 :yz#7bvޚU^!<H]\7#@헺f97[[IgõW*",<}Er.{nůfYDZ]Po^AL_Q&]g 1GiNYQ_ч6wM0rٷ9iwC:V䢩mDyޕEٛc`fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQm_pRJjXh\T˜q􇚏sTFAr|E(}&52&m؄);f3@6)X(6mySf[/ $Ζq09y 6}#}Xi&,_ٿUez +=>FShl_AUZ~`^vLv!t];ro)۲=l`St!lf ̥bnG YuK:.-!6Du}K\v9 `ߑӂ"nȰJ޷lӲ˯/^I0`,mP; R  NZ@LW CbKJJ²y偧trp^lc۬˩7bSrh쟍l c[TaYbCǿe2֖;z؏[-I%qiJs-R>QgGb)*ʘBG 30DorkTӓ BGj+,29Xo0ĀC1~ M2?>i a%!c\6J6'O6H 7_2ѡ͗m؄V;Wuf0" & N`ٌrkc^M")shZĆkn?~;sbۛW2n 5 VlFc/YXC}T P7?ol^]e#2B9}]= G{Ǚ1?/8ubmN՞T{rM咽M c=Rd/\̲{n?nplfgA7݈1 v3lϞ<#J \7OnbC`$N:X$9 q_]#7fW3K>OML;^#%ɭqR<'ɹؓ+=?K .]޻^,ɉhv,Y $z>T޿}^8muzu}ۻ -9;y/&I ro^}"^;r* zy}3eߖĺ$o~0]1hyl?N='Xk;󐽶Ⱦs~G)֮:t\k4޷E&ǣ/ROm|q63^%:0Ϯ B?{n2-P}_ó?OϿOXm cG_tU/m/'7 -K6K(:8Pxvvzc!⭾ n +aOGO6_Dݗg߷?<= r;zt3=9|Fd=!lJCu]_ ?}mҟ?ig)2Sǩz 1z X >sq.~>mgX9^^D&Trd\(c.0ѹtG{6; {m#⩽'a\ȍ>L컳i}YΗ1>?m=zVd*%g9}9 `uZLoD \u(A?" O+5F ]@mJQƼ6 ]G.}pjM yd@ئ擉7\_#"_>q oDT!B[L󮗡IB4> LIdPg cS&lԥ?sqn@9 "(,,4"l/EY!k M\X ؾZLٽo-б `d3Jk3mc:a$$dKuK_.|_rE/{@yp_Ddsh} Ś_Ļ"xf~uB<{ǕW}כf~Lg`ߝ>ޟ-1>n>pW1,9KLܾ?C̻.~i HX-ZO/vŗKҩ/|8T#m*xXHVfA }7 FD2iG9 (hZ|Yzz9!{9 nzuy~k~O<Ve=Q״oL 01U6[2Ik~T°C[)2ؚmmk@`BtP\0@X_R҅bgۖFTO% m&Tl¤`i뽈׀zIyg `pw1W\M|59CWT7(9+g{rl`\lGvڍ-ybFeq3䋱LELqЈl"90F3|akn3>I!_-݉썾ngWt>ꯋڙbo)[bm+{r%]k#c^"FaBu3f\) $jD1"f6l&u̗Bgh{]lQz]2NX9^3#c!`o,ȽllnMm^tnNmDm)_7D ;nwL!r}obtC;f~,gS%zS1Ԫxe7R |Od EneLc?{<-{s9,}U[hţ/=Y|1G)?ky']VeAVlFM/oU,l81 8sk\+ A׮'5 !۽}[zyq!svn*bx1ΰtg/ۙ `Ppj%^}tƢ Opv6Y߾zQjQ0l`XAW?d8A|}RYD7-loO~GF_[3P|0O` )Kzt߾.iNaEY`)០[OɂV-BHLËTu/Ne4+u-u*]R2|ٌ\gr]@3K T7hWo4Wg)!ecnzۘgϼ|g umzd79Xnoɘٙz^cm]?=~ͬsU@fǶWPwq!ekPb.i3aFݾZ-sh Zpn;CDl(E,cXȎ>O>^ƴn3k![6yP9ᾮG"eؐRRЕ ک @_ǖ<ލl)|slkX}`KSʺ',1 jrE8&s[mClG5 )?({,W'9w1)jc]Nŗf_9R|/Vg{[,͚?k}ޭ]K)ebxA,؆6&B8X;M좺 f\60[87[Wdm3{XgӥH~Vq.}]w׸Կ+!)׳86pu1=q>3oMzAJZ#W/<10l1N:X `pns~C8G7H+oI@"N n6DT6_ 7>uS d/J9qRdx2ay0~ !wȮ2:8>z@c̥dӾdȂ(D`RpMX|E*Hvʦ8.CCtq;]Ab)$ EZ8Z*8qkW<;D/s0=?gꃯ_0r/>WX;y>Aɦ`![̧dӼys6;l(=]qm/<dyns3} LGh{c嘏Eֵ>f&uqZoOSQęέxf6+/ % [0]'Y\'_k:70t1.n{3EЃtne'D0eq  6[eq].5edn[2f*.`6X?r{K%wC)[ǛSwjSw~Qj,c}QzpBX{a#2_?\/ ,,^?{EĹ]Gc9؀_;^[{1ld,/f kM(FEllf5fen//K l1^_0^>o/?}'(AAdMXEVfsбvT /.%irΠN-}04lsZ0Y\,VM^^J8}d{` =/:xE" N0Sc]"$e.:':w &M+6jAɖLܛ_D_WDZV Uٷ]5 眊Jh3Vc>ۭk}LLnx2l}T6ױژ}>^Ǔ>j=Vl.{g.Jh:.-{Ϳ7?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{x[!Lfxb cڝ/4 G N^عm]b2\/-e/$>E*] ,Y|1l-̉:~:?XF~JOL?8jvc,0,xk/(?07~5EwE YT~UQRk4Snc| ]lTx7i,`lDV `&vS}enfV]8c_Yc&2RC'^bHߚx]s/ʮ67`1̓Qɱ}e̎a?(=̤ZPi_Co~M/!tAA+K-~ =1.",ˆ,Q*Jc(6X]̊z ,Vy6g/X g( ^Ί˃-YFW}yDz4>3G9G ^ `ܬfڥ陠Elb٬X>uɞɕ=`Wͳӧ*Tm+6X 0ax3*FB8~j Wl!q L'_66k-;ma]m,ưk$MYѺ͘{'6 ca2eS^k~=;^5o>ԃln&(o}t<7s4 q O gz #8א7kqLxDЖ\'#WYY5K1_6YL-̙͜iDڰ˨Ww:~pLCckmzի^B/gQ`Q X|:4гJ;+lQL? "0`DnRWst Y7);E`Y{}{ŦV9?% +:E1#C'BKrm[R'm{'= /muqFf}maP/sjQfUŘ"K_OI,+(~mزWdؿcc/0Y1W~(3EǑ7q$>뀹ߕ\>WqpJ~KȖyWFl'^!>n!3|]^sxyp 7͘,\z& @a2Md3KǮuevjp]f_.ߕ#5@# F=Wy[R|Kev׆$Vೌ + }XAfC%uCK0_W;Eڶ sYc"ۤo)c8aV?9q~o]du y caiU?6[+(s5WXNMrVel6 iRʽ{w s3N h;52 ł(_9f9)k!(s6ZG붝OxcL81-qXȞƂQ`D0GȜb-"'TVreg,oPC3| j]kyV~}x' ea#K5Yc-LmZon:A1sfr2~gkLO$[]. qH[*["3Ekkass\ӽSm"@&֡2]|ceJn=c ~'{Y#~#jn wٛ'?![ѱ*{do{6PҜt܈Y,L&rim&(.g >?[T<ﵽ6v}o`z*6"A,X1MwE]}/dET[E"qʮ~l2^?5b1r v>)+aیĥN] &Cz ٓ `l{SO:l-Ooד8S6Ybx s1baf,>)=v:%x$AЏnݗbI:pSCك=bxh\yxuݺo/oc>;uٽ"Ʋ \g[` 䦱|PEeXyT7Zҝ b q/9`8K>GI_oB>{gYO^^y]kN@{w%%{B\ ?.k%ӟ1ntrDq| {~{)I.G;o˳|q$E8'Ynq}|^ (}].+|3#NGڌs㋠;3/ad'OI$I$I$I$I$I$I$4$I$I$I$I$I$I<($I$I$I$I$I$I<($I$I$I$I$I$I<($I$I$ɍ6~31'Aif?w ̕M:Id=2&wtܩ68u|r?86f{cfjl_i={\爃ٜdS+sY}踧/e3q~c%YK$I$Iơ"7[ =׮y7fp.V1f5 ٟ},T*mC?{(v:ne nz1eB}/%>{X(cZж$o=}%nj1VjƲ)u^ w\ќѸ)9M$ww>>n_wu?{)ړ$ɛԋQ~6VEYfLJ__>ד;ï?cۍOS___^=?j{ o3 |}Fb]lGy3c6h:bE_}n"k3}9fe5^׬>yrOZmu yud՟B=q/6a1{QVYX<ɞlo~|V~?1gWNi񖼕>7luj?j?=Ǟ~Xs6^|U?>O_~h{ymZ?=#>Ck߹l'&{Cۛ{JC/?;M0;tzx<q_KoxQ?%O>kpMpNgS$:q]wZ ~qxۇ/6 (/ٍ,eY۳c]V>*YЇD^f !u/XX=CnjagOyrj1xzJdշa߃'Oʖ+95J^myc1k? 8Os 1AqVwm=㎙G}e?C\kȶEOy[p{ϴNg0j7oA͐+}n)&i׭+-kwcYcV,ٙH ԋx п#jQ+N_ (AAu_,L% 9CXd=כl:V^.RF],؁!K~ <Bas#1Lc⻛fsqk{ɦ[%Vwmѿ=V4h9]-f}Vd4Z_oL$ Z_YiErS<['g6v!:n9C?{uz-’m}׭^$C7o~B+v .+}Vi쁎dk>[WEyOmߒ&sk>V9ݸ "S .hZآқNdRù\e6|Umモ"G3N>E]%o {/*Eo/A> /ANSNh%8Qwc-Wfz[vً\; `ֿ f7̊{R:-@ЗIg_"Al}4Dz пzN_Թ0U1*U>k7y $&nֶk㡯<51 :_iյg7vavz׵7lnT'y\^vlL@bߣD>*`ݨYN̕kԑ~^ڱPYܙ+QNoK&Y},f `?T}YvG\<ǚ\*_^[Aʔ=hwZÓڬlӗBۢ~GXg[u׿0{L]O0[Xmc܎=DiQk=pd/KlgЛ( 06#7@ᩴ6$=o^7dflh^R7&[ɪ7zdz {F4}P!N?}Mmcx6wcבod\_It}/rlAXS{6~AAGߜ\t\G$t{?Xe;zMSyc[[k)frL5]?:U=V{q'3'r+Yg/YslEo61$VD1L5oOwm\g}"ۣX}{_-(~bo'fP۶k͒}Xs 꺧g}"M5ۂ?zPmt%{sEH # Xtm^ V0Y/"M)֔b,4NEһLֈ?ReI= ZfHoZ.7|⌍<E1ylH7_߿^3^oKn27& B!"םh]\V?Zc$Ik7x":`g3CшMcԇwgc,7$'} 3(~*}?7sgƳrԜodK7vޒ7&n޵V"ha`4nHx&}so]|B똟usVZ%1zc]<f\Xӑ켏c o7Wo+!pje5x?>~4n+;ZtL*u9'+~0Y6w"-9XGa?s-tjYet::=VYE// 'kOT7A梣n3:ke޹=qkR;zm,ztV;<k6Du1{wyt8=uڛM, Cr.Sm}1䎖Pe^K\żVWI%ڇ}r֮~D댡;6tss؈;0UWs7eٜ=֢m`bx`ozz-no-_kCkڸ7n`yevkxx1k}2˂+_zcIPwM| ^[ <b״V3!x=nyOM׈#_zuv~&x^n'ZǺ.?g=?s,u4-jWz]8/QW|#+O,A/?nͽ֢V:OVu6=e1?:^NŢc /XSjiu_޿7:.:ijECUt< _" aS}I)&u_VWiyJ|{Usm=Iƚ~< llm y7 !Y fǵMps,<9^ dCYi=T_9cX2FK(b wokfkмuX=.a)ɿj~//F~Wi2 aJC[+?DMLMHzC7C1jNaZ\ŹpņcIZ$SyiH9o-(11c?KCl1<[s_39VMǮG:IRЛzbs^ު:i mz6ZG.aKڶ{_UF}ޫGV]\бZrXO7:~X,֖cuN~^Rj{[FD^ DSwK׫F.^.?署=z?[V?* z4{W[ϣxD~HL= 25]?kBkBRkf-,ZDZvW#v['F򛱄0MetMoZ `d:?DU?F}9d;c-oRXkjDNYCE }0u4$XTO7L!ϯ=x]:F\J}ט|{M\CYg;om1s_]kՄ=uzqj,XZŷoɅuZ_N.#Z@(o)EZrX3z[[fkZ9V_q.*=u'>?Tn?"{?[=ư#~N9VY~UzigS޿;(^8'}qF[uZ ]!hvos7n업!xևB UnS~04>zc~~uυ=xV]BvK=z3͉XV]>ת_;OʷPkĚeVF-x} N.[uZc Xen驳Zޜ${>?hzmVV*[Ysx}UZ6%uxݞvkr Ǻ8k  0w)lH%TR * }85r/ֺxk@u9ŭM-N+[-A[VG\hxkKmmxUyeX:㝷 ~ӊA؁Z9meX`=| _Gʭ2FC}^ZM8ˢVNqď>S\S߽w{zx5_5ύw:ujyu/]ܼG|{}7>T^}h>+kR;#/]敐C:KкI8TDߕ?mg/_ YۧR^|Qvp[ȡBÃkx{l+Gƾ K`D#sVcuv/fpX)P`2! .pdD7''bBO"tb§wc,Ușh%~BOy[5nI;R؅x 4L8[%pS+H_z1 {9ss8N=`0 p;}bz G.K8¦.\e?*7E :EBm!}z9ȶJ"E9+R.+W Ou6:Fo~_vF-_a$UE{m c֖?FǼZ: 韬}jGnnaŘSqL3 ^Gȕj_;8$t ^c\>$U(1RT9sR X!:F:Ku3a p9Di9qDH,$I%ɜYv'ƾMHCTy' I'ޚT'Q`Lb׃iܵpr81IGjj6m_ڬ:ms7o㼙]1BmxʁqS?o(Y<:=m8'0G5 o/ۤޗp,XhS>Ϭat'\Q 4׏fWn1 ,_8^u~Дy5b'Gf}McHa}Ƴpy2x+ܧL"JtbynbÓ9"^3t<˟$æ~HP}:㚤Jn/Ǿ 5ft<5*yY8NS:v,*_)"sSD}K"a"hQ2O6!a/fqdMZIXA7ǿFX:g@ MqoѼeiCLñwӸN-/ 6Mkǵ| R!wSF =_,9bFl>ҼQ{9,U~js%-“Y'Z`z|1$RIRYRrZB$ 2mCL#UWZcJ|Ee%Fbv`ro;"g)XwKPawԈ9ϖ VbǷYj\Nd$0!gJ$kv<#ծ3-I9} lm[z7w`8g"̱c-nT>v-z[[ֆxq״qnݢ-{%yETb36^簈UܫkHGc-= lOtH$źDae.-YRSMH10Nqj'Uբ>( )2`_x XIRɲݽ_RdKd dG?O.Y&yb|&,d$P_u9Y#jxU<]dA#åDgS>d -fr.&m_W/f(O1B~)K"SttħyknM`ɴcjw=\Mb."n*Xt߄&0nz9IJŤ;^k >HL= Ib qLeB Ibh@ky_k݊i\륮{5xi}(A+#X2_$Zi+.,&Q%* ĮBSYD r<2>ɶ$c.dHdDiIőbgIX̗<1>Z94}m(`j]L''bi顩l=*Xn $Z}Zk?{eL&b-%-Zs w ]T  8K֤s F'z$K5BƒR֓Z4{!Hs\$깤VwJs=?p@͐,?S} 8tNx*H2IuHYHѲD-a7 j`{'h,9Ԋ5?dK~( EIDATi,c=O,`O0OiG}z9"عy't[{ϧ5LOşU8' ؉(68M&[EKP,u ƅ.#fSwX8j=sBNl^+\`艚RaR!ck &oSl:us,bL̇3Յ`] o޾޾{yv=j·7gp,y9\{Һgϙz8j"i)$8 ASG,fV^w`Ӝ4k-Ų_;?F;v]|s_|(\72vO>w:ujyu/]ܼG|{}7>T^}h>+kR;#/]敐C:KкI8TMgpx߆~|a,U8~1<}//U~|<|edz[zQc?r`@C=Ԣ&}RQyDV/zϥk?WRQm ^t ,&r! IFdy3P0YNx(] .T$P*B')|y7R%9VQ-'hUsTc+]gK@:Ĝ=ٔϲhaszM|wi} Ib:l.r{/Mf3=yF`u}LkƑڟZMYNZdY v.Fu7wbڰxʁy3;ݡNhOi#|kħVy߆1؇_d/Biχof fCqC96)<É="8>obBFrD?U4 PY =>f1Lh!r1ys]Cxª[#M,ғ\uآmʅsaA%y) ~4$kТHª_)g685%9|``pl#~(N\oeB,ujƤQ6͍bLX 7ޭt.c?VoΛflp.s7OK]E_,,^! ,!ugy1X{VīcQ.[3SyFxy[kWS?"^:Ӓjݗp+6ͱc-z#ʆZi 7^=``$|YS9:yo4EGO:ZT5NY-h .,|T6coH&u_VW)yԌ?^~ռ\sO&<:]+l͚HŐxE!iϛJ6ͱR,W,`;^ dCfz' {ݸ^А9O'ߑ{sR'c\qꌹ^urW17}1#c +X!dd.%͛\k}-Xm묬w[X|i㛵)Mzb4n~:6Nuo;/ĴX me6ËI+w},${20 ȃ#cW>M_>BNvox"}O#꾷Ƙ̊Y>蘳0|˸ p&l`BdT8g COpA~` Y 0pV@ `xg|)IENDB`llama7b_autotrans_1to2_predict_strategy.png000066400000000000000000000137321476650141400364640ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRhUsRGBgAMA a pHYsodoIDATx^=4UBlK@2rBbĒJXByɰl6[6qr + fim5uOUWuUyU}SL2>sP$&~!BP( &IFhBMH I#4Y4a8Op25c{]Ӗ_חŗ7}l˟cV#{3F}"Jϓ|_:}y8^F-._p+LZci-xK5P⃰ȾM\V%ۚ$^Z[cu}[.渕^Qd0?G6G'6aokL zs2,%B8L_w%$AOW(\տeVUJ:o!s1M 6֗2>dM[kۼwKY%ž->W2k#CY/&2]ϏLhbscM}ƞQ˥%6۠;i<[pdk^ϯcp k Fwr[9 ۶UӾ>U_Y΄!Qmjb~Zt믥bߤaLAkm֋ԒgPx.av =:ף>xa^sk il:Գg]6cdԞU)uU1Pɳ>=kRu|dgId\۪~sdXݯ:4ΗZz~e+܄j>̷HM_Ҩ]r KYȤ]%ve{@ hk6[ڿD.oV cp>m>mt{$ڿ\O@k˦ʚ+'gΙl֞S2_O25=Z(t%ՠlɇJV+Ӑ-ZE hymvcI KlA`s]mB \$ SC={m~ESϛ1Η%חȩQ!_[=jyLfSh1t{nGav]n% ={Ji?=)8%BWҨ4@.5 F& 7 B L/ q;Ajq #8nkl_l[/ח)%{QG;]^O~D=~uU#֚$f5)z9l^ E ǯee+ ansLX*{JՂn4 nKHF+i|.ZQ I ` H80 $Ŕ&mX_~4@FhBMHlHP( BI#l/_1a^ I#!hu4lf1a^ I#!hu4lf1a^ I#!hu4lf1a^ I#_|NxN0^}}%v\d>&e.hƤboT%2Iݽ|3s:xW-G3pk{Wdn^m˜<[5X4zOweRM]]i5:j,&f뿵b'0Ic!gxbžsP) l?% {iX#y{-;jvS.Il>J;C'0ҷEkt}6+$&Eo~=_2)> ]XpPI+mMmؒ> 14|M%kųm9$|VmXd$,M82|?9VJq׵=|^F_ߏf.LiH.\_ńQߊ63y*/}lQ `͈MX̙&M=|2yNF ASg~c'/K&>=n/].K|ZjZV;Q[6Q'K|\ϖDO :N?ͱ=)]6Е4Z|L:vit}7ei#{CޭĢBl/_uKy?|:kr;dn^g=m/Mp6sif]eYYwӷ`?'o„C,o/^;+aW3 -uK A>IR:$q"IΙW&SVk -OFh[1Op[mBA4ʼn}c¼nF B?=&6!i >c¼nF B?=&6!i >c¼nF B?=&6!i >c¼nFhJ1'?$o oL-JͿEӟ|X%iO'֮}Y%i,% {9?/]Fy{( c(@2}K(o& ߞ&$nIci$ nIu)PL4^W'u wMH(ߢ1BpVI?'DH_/$_G?EE}Y%i}CMVK___?Z} &{(|7$dSILE648l$R$q Ed,˽!O ފƔڽQ$LХ}(z//Ë- 2Ϛ𨜏~ gDKޔc+}=ƾtT7jGG#n^c&iҾ{Z m%`=H֜7G&jgc<$y: KUϮ|:=Lb73~s^=`m;gn{Cw4'pf8K$%g7lmn'P'yFN?MYSDNDu)jAP/]+v ?kcY[!NXg7k Y"=FN^3|^;lv2/bۢc>LbaQ 377lKwltǮ42xZV6MJԽ05.$Vffdp3vH{XBK_+v/k+;g^T^_1oODƸ92mKFVcW>l W kWMNNJzlqrO?jO4?򯧣j'fUB|ۜץW-rOǵ1DѷRڛ10[gTyU\vmX{-0 廒@w!%N6zBZ.-dYW=CL$I*[T:Ga-svP+gy}6Lcn賡 WR>< ʙu~mb\Y#I,}MFK1~N5(XUBvBdv/ӽJ{YTޟ#<y,gN?|:ןJou+Ewf.I=}UDޜՒF/z(߿{ViχOCA64~$[dsI##L(EP~mY˟{);{SI.7IKiQ\ly8rCP"kgDϪ%oJ>ZKQGu|Ck^}Bv~O#vh7Ic/ӢxrC5 Dڟ1Q*]^Tf\19WTnj=kء;KOL0lMNgow/䓕՟d6O駱) c"ȉn7E[<Jش+sŎpgm,Wdyxެ gG#Uϯ 𣏕S/ۭċ`c)fXc{kM"ҝ2>c2뱫1M ށx6MJԽ05.$Vffdp3vH{XBK_+v/k+;[h{'T^_1oODƸ92mKJo_v[ tFv&Yl边XO KŞƆKdWIN'!ק)+3t,/{%?#B>-,d~;{*?~7u {gR3.}-\#7n/7bo% tJ݋uc|{\;Q/lF8qsY~z,\),l_u::6l]59վ(Ϝm4Nc<TXL(S88V򯧣j'fUB|ۜץW-rzߟos{7c`>J4/nڱLZaA6w% dC*J&yl' \[ɲpt?3{ITju*U[xIgcP ͦEnQ֕M p^gC9') xTy#ea^R\.ݞ˕5Ă>D[?U_+yST6^{=4p'9h$?SG7xX Ϝ4Y(t? ޲vMW\,F{EI#9/04~o i&$Є4@Fh0 HFIENDB`llama7b_autotrans_1to8_train_ckpt.png000066400000000000000000000160021476650141400352450ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRl!osRGBgAMA a pHYsodIDATx^ۏq7💼kp5@ȋ /da(Ȗ%Xa[:Ql6-R"%ީm%;)#',ŝsZgj4/"}8 8H2CvA :%w E%L9H2@ $tN$sCM'5Md2յ>0cs.@󷶢5fJ635̵iYVK5Qte.꘩]B;vJ' i|f}k6ϱlL'ź5eBr5kVI }z>:]3:(] Ðd2B yJH];F דړ?kc:yOsSۮy 9G9LJoAoX6o'bxe(1Mi3N 7LMuV~Z=nDlIN?ݲfCzۗd gN-Gt|sxy {I=Ҙ,_eo~>K6&pv1~d;!|i "19v95Vd{'3Y&߂E$n jȇABк鬚cd|m@;PRI'Kt{7ӟE(N6[N*iM͑ yhk*=$%v-P'C!hKͅ>@!s:ij:yP0[OJ#ӑzV&աْ=r)[SռJʧ[r+|!E6tͩ$ѭ)#>cGt99'vuj=rZ*% Kݻw=ܣviZ Y)-D vU9JDj..d'%﵋[9=5~y jD> ~$hibRsldiO'fT5ݥuSjlA2yKI[O}hVnŮS} O,;D3`! ~6|,oLMXg>9Eu>&a͇KbiyodX|p1km/$/MU==Wr{ 9Ûo _7'מfL&~,nN)U"2ҺJ[:ѯϱ*z,f}b2ѬJ0FI;/j`]_wE>}@\mzZ^6jJs֤~7_S\(tF+/9y/yMAdDBDGܖ7nlu$d֘Mc?:ycOI'<%{4cX>" hBON0gX!lY`R"l J.LI&#xKv%W(7dA :I&$sdΩL2QPPPPPPPPP-d\z\*`n=1.qLj@0k-縀?1.R@ @ f̀Ǹ?EH2SǸHI&5 5vs\) fN{ c\"$Ԁ` ة[`q c\$ fj:YSk2DM7L':oLd2`[j )}-!:.q?FdNt0ӱ` V f>MBnGkvc:PI|?6Ddz7Uvڲ=;X83d̷!|M9ǸH%};p1$sl"l@ U%!NJھuLHp:xQBm]`7- ,j܏)}$ fN9[c-ף et=퓚`ezs] Mg[Qx^:Xzd"\a*g7a=5^`]ڒY+F[KHgp.6]rĮesPf?.$t`ILq9]`I&{' TM253RAL*Tҏmt XzI2:T)qlZ/X,zI2?Cuѹ E/I&X-dA :$:/Ww`'|zeF,$Pxyw Q$3\)]@}2U KâƷex?(H2Tָ_`KsiMMv'&Y#y]ΌݘN$uZ7&MmH ]˺$ fٚgLYSYr% 5ē%^ܠK'qe]c B%PPo&1 * /r郪 Dp`!69lRd*kڣH2"q,=M8I]vd{zM<:Ǣ+N!̛æ^k ]^_BYF?ldzLu,V \8M6vruA2Ŧ[ړVt&]4v^jo72➩:y>n 2̥v[=$@$TiHhD {mP􌜅{8)mٺ͝XM윰K686gR>'Lz.X۪/ GzSYH*6zYcO0py xH>C0Y?7't:i{8&=#gqL2 a9H8,Np(:s&eu,/#~6zIT!&osݳul'7Q6`XRJy=$뙦@ [66vS5zFb'=CٺhorËu{^_$aL4mtƓ˺u6JD`dzBu-}7I=CϦBOV4ÏdHV$(nl8oG;C@H%mLL_δܴoƦg, z:ֽvrP];XE8лd,>Kʛo=Gjҁ\%cQ nշHlT^/iDU4iOa?~@6Q7v06Kژ!d9>nΈX'<I2w?O]Ts F}3se1^C@7mdف?"O V $sK2}ţ$ӗɟWo??U/n\ﯿ͟]=:, &q:wu]u{w՞益/Ptɷ7{^=Lo(d^gO^Tg߽oUQ]v}StyzJ?{S=~L$k﨟l^W7/W_R^Tj׉ 껇No?o0O2`[Y} 5|eumuЍ;:|WM]W3V>>_7&M2V77 ֪^ftUS~o6%zcPN~X|~Yky=.Wo1_8;rF=qC/ʞyI]y{$s=DP23o\L]J~J؃5vvo6 7_r?WgeQ+$ϑwUÿ9ۼvb~ĦzL'_y-~/ɤsؾ}6r?}$ڴH{|9sly][LW$4Y۝[wŨH$dhd.iT2}Cj|Kɧ?\JLe.x='yFO}nONw$V̻a?"c^]S_>YF?ldzLudnz%um_/W?9rZ} .[꡽?>u~ܽ[&37qbe~LJ6wuNDZçԿ\W?{{_9>kco2CGQ],9:(tPk릺Bu؟U8lA2+Sh +٤~uz~uaM\u61MT(ǶM$|ޭu $2'Jز XBIճ d>zdCv 7lzKo]T:zN=t{17~$N,*A:Bgۺhwz uf+GEP#6}@*AZ5?%!Do%7h$~O^E$iA)ЦBOV4Cd Qo1Ϟ~\lRw$Vͣa? a} DD.5;?bf@=:vzI2oR}:rV}3FoQg&@ЁA`咍͋IA !ߴ-e I{&gL/(Qw ~ @ $tL9H2@ $tL1J?u9rIENDB`llama7b_autotrans_1to8_train_distribute_ckpt.png000066400000000000000000000323201476650141400375040ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDREsRGBgAMA a pHYsod4eIDATx^5Ibxƕڴ;g14pGD- a|ۆT]\5b-yebďXF=iDi̦SV9byzdR'HRG2{cm̗ۂJY$}-+k!Ծ/mg:|%S\?O+w/&[umZ6Zw?е0|<~뺶,_m&hf{˿//A>e =s{A^ )AOOpԧǒtX/1PPKObA/qCo*ds:3c7ɪy crt|e;/\u},4tQ14&aٞoiܭl E>5ͮc[_t.ꊟ,Lcߡ"b ,qIjO$Y?6qykL8?ќҹ "kajmRtLE5 WݹS /P d :xI/6nlg?_x;ՂbDd0}>NQ'(R:֟tNyH7D]~_jlV]s0j|/FZkYww$E% =7i\JI5>v@)1"b8)FmcŀIt*-OE.x:nj1^>ע OT c?LP'Qƒo8+ }1y'ͯ129?u>AUbz;le!1v=F.kj#}{./[DnQԣ}\1o)g`ĨIQZTF:`·X|`b}Gn똍a[/o(i: %sIG@;ݸLY񫘛GXҹ4_8]swieu':鲕8G>]WCz_Fhr^kʹ}W9zdRӱGV_6,ObbTKbRDA6lsP2TK'EGp1THJ,"""""GwR")&E2~m85`}n )`>`}n )`>`}n )`>`}n )`>`}n )`>`}n )`>`}n )`p NNU~ lKCn/ctRϓ; .1!~M wo}>>H _M2 ǽ@ԟ]t>r6a`́?n϶$rw\cR Bid&:Tؐڨtt(Ty=7lF|Oa{rUR?~7~<+F箅 h.Kˍ)~8Ǭ~tRO fNX;}Oz>'ur8h,9&E v6Y_Y+|MsGI hdЖlO>r0٬3 0dƢ-uYpׇ_j!'缴H'E _j܏X@> jA$d(}YB2 5KTtoFl=O9`a8'\\ۓ~y?C_>2~6lEv?8_zvG66k-Bhl_Ӻ"Z>zpN2%Dt|+C3dsdJKe.q&? [V`8K1nuj;3-XMұGu]jccc?0'E,[I\ />Ѧ%[*{C>I҃s{.D.ISJ*6.+)bf}|pwdw,) }jLCZ1\C%yEqnyP>|u>Oo?KɧoZ2hruArbbͣ@IC?-:fή~q"A-?:XB>MV~zA=ZjRTgq=:|un'F[unOJS#Ή1Wf*mJl_(R]fcžr]$\~l:)`kнHwoZv~c )"XoSO{#~b!>_ۃ);u[I ;u[I ;u[I ;u[I ;u[I ;u[Icݯޝ ,ֿoߘ? Y<)7yip[OOhVvYH,r[ޏi,`PB %HZF{zRDO3NQ> )ppRDIQ"gtKIpR"*d)ux~zu8BuV`3IJRzZ@RxR_d pO~:B}IAR@RX>)|p`O|~Ѽ|YX"J_+>$N'EdhzṇٽGH8^_q[ Zi=:?J4Y+ke>:Gčy_ȚjtNf ؄(hXā8UN7tsyTN z/M WNJ*q_};s$;l ?xؤtt%#1/`]Щe*čwYɏ$tbAmR$Jc=Ve)9RU2d/O+s$^;}->66ĈW={Mƒ3OR$|Gu8'@ĺ+$8:3䟏f'M˺ETCčDoR߈+ԇ ",|?kg,l$E$E哢ߝ߼ ,}y <'E=>$N'Edh܂9Q^ $EC·=._zlj|#|WxI+ zKoʮ._c|15GվaGH:Ip%9ov^obBU`U=_}* sd4?xؤtt%#1/`wY{],Ht֬J:R{+ɨVB_YDsNJGr.KwbkC`L1d6-j{L7)M: 䨝Ɵ#% U57ycӟDuܧ(QYoJ|e:G1'JsyLA9%qC5ъa=ڋ't6פ9/P=㡟e A"S)ف@\7R^3wO.Ԗֿ_$;Ҍ-kvYE%oySa{77u_c,s51Y>"6 h%Nv.DoTW4[k퍟̗sKX>e+9RU2d/O+s$^;}->66ĈW={Mƒ3OR$|Gu8; uTHpkx埏f'M^uFi/so9G1eFHѶrαڛZV9L!1cq8t H#9,dްĈS9Ѕ^KTkѷM)Ow^eN3ߑ_fc͑*:W`1g,:D}(^ӱR!*llccz哢B}\hCB<;Iч`|P{7tk8IF|RA8)(XOw>:/B+|R'ς,QTT^8N,r _`G{.@ 7*I+LKre7e_B1?+wȚjnƣNw8ᅎ\(9pĊ[K*0*HκI P%nXk}OYЖaΑaG.;-`L/ idt %'ɛZVЩ[I9&G/,N9c9%E;pS v5xM]`fz2Fٖn>Mm^u? ̦^S[_\LǓ_$;RҝaͿfa[h{:˩[7zySG52^3#)b㈠ V2N)zi9)˅6-d3,}x wCg6#dO'EIIcw7B/"'E_~r>h^,,I%@nO嫏IQ))`eNy8IQFxqCOrwtɻb˲қkqcW|15GվaGHz8r<(zݯ'Jb;'bKS䬟H P%nXk}OYЖaΑaG.;-`L/ idt 0č^+3Zɏ.;R{k]crR$Jc=V̦^S[_\LǓ_$;RҝaͿfa[h{:˩[7zySG52^3#)b㈠ V2NeSHIV64>=Xϑx#n_5?K`?I-|<ཅB믰kx{ץ`"i!FoDc}<9)4zpGsuތڷBaj ߐ0S7v1{M7>a'k&FTfNtAO'*~ۦMޔ^eN3ߑ68 d+ԇ ",|?kg,l$E$E哢ߝ߼ ,}y <'E=>$N'Edh܂9Q^ $E`X Z']a']/+-):Gy_I=T(*z*`GH^|i/H9C>oS׷\͘b͑4?xؤtt%-`L/ id.`*!`XÑPqYɏK$[ }1XhЎYKC7`k~^ej f'am_tP{ec(wIQm`$G`4lBG ^zI_ :[{Jf׌)t{Bϑ0Ѡwd-PMuojbɗ+f4}5iKA$Thx'E$!RƑrh}k$@A-&*ꄔLꓻlx0t)wC}Pc]T䓌iX=c` )p )p,}|u^VdO7ןY>)|q qX>)*%C# g?] )B.bh4Z3.yWȕޔ] #Xbƣ[[o^lIQ0ņ֖ɠ s+|k)T^YEJ*q_}* km<&aG.ߵ|#L'pF֘Zz19Z })fq%ϱ+Ms9Z1kyb 5xM]`fz2Fٖnr21;xܤ6QJtd0XvrT0!Ch2&Uĵ(6|$H :[{J]{_M L8cFݑqV-j|S{O<X7m9+tIs^ j'D{fC?)& E4SF39B-8r-77wO.Ԗ$tg99\)x,G^rj@-x~qׄ dH8V2NeS99ۈI>K>uN_˱ 1bq^ :Q2h hK9( H!x 7q*EAIY"n$zF97ÚczpgsuތڷBaj ߐ0S7v1{M7GR a'k&F$tS9 =עo6ySXE.zf#m9 d:/B+|R'ς,QTT^8N,r _`G{."\ĻO]1_ W[zSv%tNc~*szL&-./ &4Rr9C>oS"%@a>gB[5GI؄&Ed (10wm!pcz9_8H#sp)P?SNJ219Z })fq%ϱ+LsbL~-ڣ,:5vcle[oȅʸt QqD)ѡ`IQh &P!E{$'GbqSoJ|e:G1'7dch0/O:Vdci kҜIjўOIC@#? PVL: )ɻ'wԫt_\LǓHOLtg99l)&Xd-ySفk[#f t  qc=dyI D)k. nי^39f[z}S"З)Sq1'{z^N#Zk9!F,ݾ?ak0~"Q;=[=-Y@ kh *EAIY"n$zF97ÚcRѶrαڛZV9L!1cq8t H$#9,dްg^2jZZm&ohDYo\c_+e1總4٘PceBUmdMݹָt 'Eo?-';=xІlg`iv哢~X>n 5<5O2}` " |RyX哢/?9|4/_g? wb [02'̦^b:DzBw~7̥;#QaKq|%,nɛz_BM7\{MxGVo h%NS"nEC*ʵ;Cs[z}u]K\zNzZO,>K>uN_˱ 1bq^ :Q2h Y \&.^S(|<6;i2#ZčDoR߈?cP߻3Xq/$#`X>) H'E;ǯ?, =X-)7x{.??܃m'El?)R=dRDXHTWc j{ǣ%&EE E.z)wIUލѫK$mدK󮥬K};_ae↭VUZGw U㟼;/b>)ZPtƃi{oHԫKmih7 =R zBrVr_}* km<6;~܋Gw ՟MV*֓wrt&@|ɂeg;4j-Lk da|EF,NƠ Y=]h,6vW5a<=~b3hnb}3oB&ud͊Gr慻ȉK?ѩO[ }iNc=:r::Dͫo[q`L1.㶖mW={]|&E_[ĄØ Nj9j'>KX^ԗ_kP;Rq5R!\c=~QczIPܕtL$twD-[=%JT//ŏ,Q8ƌ#a AdqC5ъa=ڋ'_6פ9/P=cդ_hHO:&h,)ɳZwq,%]VA_k/]/Tg\[z{fuAa\&۸ r=> ǿR^3wIaC/E/I*bC|%,nɛz_BM7\{MxGVoOP4Ac=j,TyԷZ1KwIWonEJ#vԭJa#:%stXRtc)+Mj9f5joO}4]sTsr=Z%y:} }F̌X}ú`,9ՓVK5)a%>7b 0wtt뤊v\ŗ)cOcAm\m=l,rKN:ϋra݅u埏Ѧ|+e;@{L/sO;;SsvWhmm-6S7v1{M7X5)Ri3h-'eM2h$r\<69l,і=ke˖gslٲeswgggVO?75߼skι\O_XmӫOy͛'_\N59.r ϩO~ynw~o(_8yČ7NᯞZmw]}i"d,8I$A! N,8In/.7o7/Q^Mo~έξzBJ.֙Jgs5]ȶ!g2 YݟΟf&sh?꽬K7Ov;|`v٧_=rc{)>qѾ}Zo[.|{ʳlZױ8 WO3dzo~yy'Ïj66|5}!&ZϜux8inBFR0ū(F"? Qd 7~ 1HӇ_hHE(vS'l>xzcoQ&N|0=|[CwǪ&;Oxkw NIJ8!Տ/sKvZY6$/xu>]^+u dLRS8#oe-dT*Q)\}[3iLw=Vi ؔbtz_!(\G^{Nպ>}5]tS~u X">ͣI b Oj~].kyds!,(nFYCzhnVYO-o`5/8.!y:3 YldCwdgIҎ:{(MUsVgDZ6`{GN{b88 ,ʷׄ,K&ri>MAioa#86&5!kyMȲĤZ.Bo\,iǽ_̅NlC,Buaz[Y`ϱU] C;Xmi- ǡG5S]oQܘ՗w?psS9Y\0ɨ?7Y%Xs8f(lu,kW2|IZg}x-ѵi!*oGcsP! BlŌ($kqb!bE K6jJ!| }fMuQb&y<˨ ˺ݏ"(<[^\e<SBn?>39ޏ+~>o[ vVTׅ-dy߮,X(^cBr.1#Ϥe=g%.dʳ-ǘqhǷuW9hv}s;2߲Zkӳ?|l1>839,/N(!yvY/'燸gu f|~s{>f4r?mG/9Y~o) cҚ u{kH[vB.OjOo.FDqfO¿;p8u&9ϦV;+# ] 0A . q+`9uq/-lm/5aF&!X~g S8 61mTe+a^k1*fxe,W?#I鞏}FΏcwJg˹bZ{kMYm+#]9[??~5;/ߛ;2Q#x;R RbCao=~o3!{h.ޟf]H?;ӽ_Oqq{_ϡrudݽg{:/w]rf+kxl0.bm!m1i!vkj0M7_[Q[taJ";^Ѧ|:&S`+n^Qxi[mUT6"} bpU6c2 d>/WBV=||9Q rFv?jk9}<:;3,0s=o8Ƽ; Y/{Y-;Z޸y},jsJ{z6vrsL[u M_~1΢XWaW%:Z1ݼg'ؽU̽wO~ۼ]Ld|KHF h/wkiϙڶg؞O\>˱m +jacD8 }H5FňtΡNyr,̓eۓcsԿ ]㸰r5|?yR5l`n} 1.D |Q h8F.,m]i+Sƥ&di-1f{!ˏJ9;gMqjͺ("R8W_d٢+͟ӸuA1[\c^G1oPCYC߸+u1!,m𹰋-Z,߰P·h ,vvq1}AuBb"e:?wNڰ+fK>쳙 ɒ\#cucQ;5}{eڸc9Ԟ<ݼs»w+N] ,8ҭ)D,'ܨʄxƾsG7r*ޒA}Uy AwBA _,Pe~+* 2裋-^} 7avAKrf8ŦႦSY{YE󖍼^qw>$ѠK>Xb#EfcՋ^\ҦDڷ՚?Sg6Y"֣u|K1^>״bkO-v:u:uݼ̱VbرX.8!ލSbTRCZ_9w(ZW}n99K~M]w{X@.r:BQg)bK*r U@|UYל OKM*H;G*,ɽjZ6!EO]hxޯa1z'cܳ<FaêxWr&Hk ~lKk?3π5׽}9}cǢw.8MZ7#!t|[+ֺRnz! R}U~-r1fRITzBRHR -@3iK[炑gC`)SK|v[6Rm K;ZV^o _in#yXl |iZG/"~i7.=`>e×ck)\>sb,D;;_>xsπ̝4H'?xرXpX:؟-{|jƻt{K! UN,J_U_N >}ZBE(-ŕؒ ^LZQ*BnXSq.Pũ%E}R[5C( ݚRIw-k| (7^|M],|~fFb;C}X˵b_֙i]3ϚMN_ïUs׈=>cQݐ]]2eT em{$/ugto Y7g2U;*j,^K(-ŕTl |\q&jŭVѮU mBfMR+JE:mŬu:uEDR{@X_QZbWc=KopZ_c+O>AN}[E]+oڻb$c_Y2,9_x3'}_& }Mm_b|_=&އ¾VC"+sJ~ך]G,_*t7RQиr_"\h\Yhѥb[B*$dQi/-X;YWdԼn+>kve٣QK{~Ok]g O(оy mȹO\cvSybG} 撔Xb~_wzZ8qzZdۂmb䜺my5k4a7NIs0<] ~Ǽ p⤯*~=Uտ=74#EQBh)X]Yv&qX2Ey{5ʵYhޫy>Zg1%/k{_+&"j;t.H͇vu|{;WnOܚzqty^_kO5~\JT<;A]a<;K.BEmu{8V=:ʻ\E{,͛ }Wӛ.s빳﹞.PPp:7s~1]qNeFߩ/cڹp8Q|nz[_B]EX߷*n՘}о:b!ϲ[+J">EHk[ie }Yv9œugʩ^0̙ukF'V3OK]5*p(u:es"S残e3>ybٳ<i:K,z4B5c1Xk%&w\k߰?cvXV]-z[NBG翙<\`z(Ͽz/kͳ,?>(7},]Wܯ؞}Gx\ou ^&L] ~|=.P%  {o!\,"`,Պliο 72MU#t- ;^j뮱c..IP+9Z>dw] ¨c)v/5jEJZۇasoYdTG^%-ɒkB}}S1y$$kG]yd>1zy/ {K}\~%ǎE-!oz|ڬ};v `홲zOQ;7.? ?k!y¿y ,'1 `4WQlE;RnbDuўuߋt]QdӧN||pޢLͯMp`zxn?{y\>y)}Yo n$`%%d};}-,hQKn'\\.QYo[׍o;R۳ߟE*YE5XDؒ ]Z.+^me7R lVU-2Sk`jzlԇ|QWloǑIX~>;|5:OoeyՋ}5]Ư sTKF_Y儩^>ͣ,x$>ui-zM=E^(kYMb !ɷ '| ˭ hq,x͂q6j4NX_s܇qm?UN-jKD,{Yӟ,^oAQ+O#,_w[RPcb%-RV(*aU(-{=ɫz=hXkMp>.vo,ĎM/;$7R}"ƥ3=~lD- ܑBߴWc?߳ |ѷߑ}cǢ>YS<.6}-)ܙb F<do҂wMg YZR|{Mj"F0᳛6bA4 YmB%&m>֚pd{ߔƖǾX0vXuKibk!e۶烳?md.ENj زBg3qZ{j}й̻ta"!ĦŸBo ;Nj"FXD &>72B,~uNm;5ߤ/Eb;˵]`sV췕oyvV4{N[U5[g!gSsiv6G-(Ytj,7"kGڹPya.X,R,uƯ9S^C쩣xsm`BVra`ެ "C[`BI' dnE STq;( k8R{j),-.4m8U:1`ټqw (hBX_8*KSYg1&<P[Fo񖭄h3~NDUOH5?*8E\t,ӓCrv: Oy`QBga'duDy^C~9[ Bp.pM= Y ۳p*Hsc:?-cg\2 7\\T֡pU6Kɚ"Ӛd,ht|^|m:>sߠsF4GeZ~4v;,gj蛵K^8vXwVRޮٗ%K{;r!+`C(,GN])@~kh2l qMF9v*/<rs6(rUd6 EZMBl?>3`f,[ s-Z!K-{/d߼Ѱod>"0<3L8ڛX;n-E2G;v}KҾZZݿwkՍx,n1phZD UbV`ijEBdnP}KLc31~)ZgZG/$ E)rLNͣRfWN:O\AƏm{Lr< g4KDOLy}M۹λ4maUIWŬ@V]X`SVKMҶ[bNB׹r"gE.esY!EnQ|^싌PZsW/}""Q-[`.fyB/gcEmq)+!>߹_kAvo xm}ֵ+YW/HXgscfٴUƗ{ryYs/lOyP61Ǭ #S c.>3G8 )6p,pc`! ‹v@Ȃ!ݚByͣLȊ7n3˦r dՂIĚ>! N,8I$A! N,8In/.7o7/Q^Mo~έξzBgo-tau9.z3]=J_^wn߼:&!g_mcoo"?}>?|}~mGiO?JmM>m}?.0>nߑ3[ݧ)wa4RcY~N@XloK: c]{8 Ev!-efpS؋Mf!xf* ϢBj"B R73Һ{+Ƭ}}vԉ?x8>456vZѪYKg__W wuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7X,펲IRb?YsT!9# 8VBHƄk7Nfbm/DJA?7$|ճsg{m[yӊy=&k/]%d+ZEJEVQ/FX*>ui"BlCzhnVYO-o`5d-<˿*sͤ-svP^ƦlΦe[\gO;,oXd={u~ZLK?R{ #1kSkܶCj+ײ[Wg56prUٷŏ/͓|k7B+g'T*Kj"F0᳛7LAg|&{ۮ .mB%&m>֚pi^p%<=Kܬ&|,N]aLլgi\DE6s,bX\ -QeP鳂^lrl.'7,Tؗ*rs61cy"}*1b|3 ߹_kAvo xZElgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y Y@.x135۝u 6湊Ecla7gYnxuqeRXm013+뭽rq8^$ F sLmB,D[DLNEGqsĊb{uDH ^Ftngq %VRxH>7'!! Bc ^CJBLEWe>^ھyvzֶӧO>}us\`Hzۂwİ^ڳ<шpT!b-|2=?zo{<ݓJGg5Y:aGI,0l{u87W ۅ,gNcoCYb 0,'1 `4WQlERbDuўuߋt]1fӧN|pǰɶ.LVg/ςX:ڴr.aZ) X 'dxn YN_1 fRy ptB1l3Gmﳮc}_eQ6 Y)W.w9p:`%fNEH1ÚGpG gӶXcۋ0zҼA)M!-dlG*\ec^{헳k+G} kZH|g֌x*divKEK&ri>MtFk<&dYbRvcI߸(Yγڳֹ Ukp df=Kcb*ЙcL2\mه4w iZB7>B<\K,еOϞոƔY?u/mGp^|/QT(_->]!eu6w_^ u gw)%ʘYs;خ΍)|tsacƬCw[ckM{K}9h^zg(c`Sߔ/wPi3|)!aK1Is)rB84 ^m5a?"@9Q"%X5qN_v]ů.6Չ"G :l[BVmgoKr?gZU% PV!zV=@i|q,7=dM'-~ \NΊ8-dy߮,X(^cB;q1m[sCBoqb{35eu6unsoȘXC絔rEc]\1w/*}u˶aV`^GJ1L1KM991MVcxҟbGrp0!+@5Rt0oVq-$l7udST"H\ضķU=H)diqiؤoũXZXy<{˃wk9, DMBY,V7Xż0-[ Ѷ*H |R[ĵKG~1=9?č,gG>}][XY^EŶYsT͵9D;fr?lٶ"c\zM4ϳq؞c>{ǔ9Z{ 󽕞7bz|rgͽn{]OoY;ة濋/U6plcr|{Lax/11bދ BV F|\78 [2:}o mQfmc|! 37|65ǪX-" yfUZ-aЈ(@eE,ZQ\Q ٳV\bWU` Y74,d-l:Ogkvd8fnvK[\guc ,&@IqsyVX6?˅J0e-{m bZM9rlbfz+2/u$ql3.bDx[koj[oL5}l{Dݚk,WGE&AiZ{?BD* EŢhB[,E'9n(͟[3s-ҷNqe1{.Bma@}^LF{Xrs^DQhO/mGK!X8NWϵŶc f3z]Ҿ=<^O׆}?_"C9{l^7l7cm8Hn׾7kʯ+,à߾Ͳۜ/{oC@<~m6(h7y)bqB|H#h4,m]i&6Մ,m% l/dqX)'y{y,l}QpE v & rYOtqP4@us k ", .eq%';WwcB#YsaϽ=HTəm<Ҿ5Ŷ%+}MBs*;*6f=T*U?1WGI e1_#,Rl~f3;g&dϳǬ^lwֽs,ؘ*vylB)ݜy~d6a]jFMһ013+뭽rq8^$v G_@؟cnf!޺%gJt/?$&tm7Nh!QJ'q]! Nt  jMv|fPn&e1733i_qr d;>T+,?{넍sX7n64y{򵁉[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y7i+UȺvG 6LO~9ۿ}~1=9$~>4RcY~N@XloK: c]{8 Ev!-e?{IZwB( )`0 ,vD&+?NO~ {=#b*gOSΏa1HSkm'\>>ޭ!­^tbn]eXE*ApJV Y~|![Bַb̂E}\>z Q~D5#dYmwMBV>ovpT!9# 8VBHFr׷j8?^~c Z9ho I/h!g;UA4koʛ"泸ޔb X%+oWMwQrEfm]j|\7RLIޯe-Oclp,[f !YZ!d=NMh:ض<*s,1PfRWTv9b;^W(j/c[gӲ-b|Yl.H7?:|Uҏ^iH+ulY 9rp7+ە)KuuFfš?R{o\k`GX?;"VYČ%T4Z ܦd :#5v]Zo,1o;$KV1g峰[^C,T/©+ꢚ, |@g~E3Y ˘E gɾ,ŶǍO?`75R6 t-!/x5.d1/~Oci[Q,ng9sT9ʗ>b[PrZ[gs[py}zPϫϭ5}ȑkG7>f:te;Ƹܴzcuw2f 6p{M"ŚRtK_I5.A,/pijŽE"%5q1}fMuQ&yjAY[%We_jlٌnB#dzQNwҟbGrp0!ˋJ ϔ`ެ J[ BI' dnE칰mo=!f{RZI#f[SuV-^Zҟ1_G I& PV!,NE+Y,ryb^^񖭄h[VkJF͏ +-ڥ#dF#پ-,Tb,wq9Zv3^l[qygXx5 ;c3S >+huv/}H}CN5]});g~Lu^(bbļ7;A "o&pa=rHI_[C[f_kBMClͱ*/k5 ܺhSao7QʊnYY<@i3j~g{XG8اGB[Y=BͶ{${)h;;?[lGcZaiQ{b Ϟ7g+gi׉nk\ c[ڲF>k\ >{.Uٔ)ǖ1Z.fVy>OKB??s͸m6 YmqATZzQA0TӧͶGڭ6}~7lѱI+}Z+b|ض/㈂(@YDX/Y,E'9n(͟[3s-ҷNqe1{.Bma@}^LF{XrsA"otО_d?:BlOcCqX=km@f ۭǥ}Q{;f=1xny "b1vcE/'{s r!P:nn|^q ߓ^}?oaמ_WY]ǫ硏a9_ކyڞm6 YQ0d7n>R)o m-DC"rgoZLM[5)ڨIݖSuncY^|}d{*BV(ع$+e>]㯦 WAYӤQ-[ /L I4XhG߸+u1!,m𹰋g.v R%gHۖ|s4 & qlhܓmv6w'VPa?\Wq\i%ELb.|H)%>Ͼ䂧:{]9[:ϱ`chQ]= vsXwEzitLJۘra^tljp8od/ޔ=vPD6 uD,Q?SSE|1b07Nh!QJ'q]! Nt  jMv|fPnZe1733i_qr d;>T+,?{넍sX7n64y{+ł[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=JapiI :lof!xf* ςBYjB R73Һ{+Ƭ}}vԉ?x8>456vZbѪYKg_mYwuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7,펲IRB&ՂJP03"p$@2&|X(-Vm&K4oC{SHrxG Y=ѯ0|{٘&7YoM KWD} X%+oWMwQrE[#<*b+oڽ_Z>&X=E^bBGwXBzmy)|&ntmyTX~cbͤ-svP^ƦlΦe[\gO;,oXd={*bZ+\Y_]w嶽R[ ?b=!Cce#~ Bٷŏ/w~b|kXQ,-VN` 1o Z ܦd :#5v]Zo,1o;$Ko\,iYV7@kpꊃf=Kc*ЙcL2bamه4w iZB7>B<\K,еOϞոƔY?u/mGp^|/QT(_\>mAOWinWorb @o?>2f܎2k(s#GïA \X1ڻr^R{1ס;ʘ5ؔa7;kZI42?rF4_qb,w*dyCVvs-,)e,"L]UW\S}!kwoT'\%&mo YtW 9#+VB(@YX8[=@i|q,7=dM'-~ \N 8-dy߮,X(^cB;qu1m[sCBoqb{35eu6unsoȘXC絔rEcr_9w/*}u˶aV`^GJ1L1KM991MVk<Bۊ51 k~y{8sEw Y^PAxDfUWڂOJ>I KvC_L7E%-dυmSY5ۃB֊M6ZU}񯅕gW}][(LJb,wq9Zv3^l[qyg&8OZClαZͽc=J1q=C>eP鳂^lrl.'7,Tؗ*rs61w!7PĈy/nwYA0UsYL7L#[/-lm/5A!&!X~gn]U̯u7& PY-b=+Vg("{fBoU03bg8زwpBͫ Yoodt 5ΓYZlkҖ?CF?|9[y<+~NÿNw_B^{ݲ݌ז6YbSwlͦN9v1bne%!^Ζu$ql3.bDx[koj[oL5}l{Dݚk,WGE&AiU_{c(" nebYQ xn\4ڢ53GQ Es`fE) ,fEh8 Yϋը+_qT|.Q|r"('j϶Σ,PyzFϫb1.Gvlqi_ԞgY`O p^kȟ/XX!H\bk1߶e$dkϛrص~gycvQÏ/ccjmΗｷ!cbbŊ?(h7y)bqB|HA_iY"Tbj q YvK_:RN,o_)GDj)+I\V<9jpE.]E=M,:յ| Zqv }K߸+u1!,m𹰋g.v R%gHۖ|s4 & qlhܓmv6w'VPa?\Wq\i%ELb.|H)%>Ͼ䂧:{]9[:ϱ`chQ]= vsXwMЗ΁)Xwab^gȅ%V*[{!б]`pH&?Ǵ&BuKTT"`tq\x;bG)*t=w,8ҭ*D,867]v!%v:BI$4ͤ}IAhRjO6υ2c ![b4^-[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=Japir(!9噰yp(d9aD!^(Y,2EǮT :}<}%^1lG_ݟ>uO-泏;? MͯMp`zxXj?{yWm}V"*r ~W SNz c(j}Xc>fg]$ '>'!jDry=] BIpT!9# 81ÚGpm`m3ƶ]XyVڛB >Zَ~U6yܭb~kBV(^"W,~ (pPm]j|\T0}RrY';K3"BlCzhnVYO-o`5d-<˿*sͤ-svP^ƦlΦe[\gO;,oXd=+TEZLK?R{ #1kSkܶCj+ײ[Wg56p֚pi^p%<+ݪ&|q-N]Lլgi\HS:s,bX\ -Qy%1\-g_)O!d1s+cf(cb<:7r:ͅݽnَ1+7/yꝽYM9^|SA5y/e~::~)&i<>Y;T f1 &Z8Y(GS0XD &>72B,~uNte0o+Pf]N#msUƦ͘ &t+e<//9*p$>8Lͪ  d}@톾oJZɞ ۦj),-.4m8U_ +ϖqvys-j6˜/#Å$Qp(i', e1,񖭄h[VkJF͏ +-ڥ#dF#پ-,Tb,wq9Zv3^l[qygXx5 ;cԄz|rgͽn{]OoY;ة濋/U6plcU˷>!7PĈy/nwYA0UsYL7L#[/-lm/5A!&!X~gn]U̯u7& PY-b=+Vg("{fBoU03bg8زwpBͫ Yoodt 5ΓYZlkVY7|b Ϟ7g+gi׉nk\ c[ڲF>k\ >{.Uٔ)ǖ1Z.fV[dk1r٠$w~z_Gr7f!-.hJ ?"VO4*TӧͶGڭ6}~u [taJ֊X9rn+& PV,*EEX-\sQOuBՃZx h4,mK*15myʸԄ,m% l/dqX)'y{yyV# nHB$YA.+5]".ʢ&bnAu-5yaVM<9q)+!>߹_kAvo xZElgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y$֑ 䂧:{]9[:ϱ`chQ]= vsX6c\:`ޅy!X\o@ǦwFA"ᶛlp8vPD6 uD,Q?SSE|Ctm7Nh!QJ'q]! Nt  jMv|fPnZe1733i_qr d;>T+,?{넍sX7n64y{+ł[,&Yp dI' B$Yp dIp|q9]y3y}ajztn=w=` <;Ǵ}um tӛVr+w1 9{hۅO3}yP^gkm;J>X0}Qj{oYnq_=vh>-HqL Y]yqq*d]LW;ۣwO'?vW?_>{??^?hf?k{[X' 7%mﱮc}=JapiQry?^lf!xf* ςBYjB R73Һ{+Ƭ}}vԉ?x8>456vZbѪYKg_mYwuԂ]%NIJ8!Տ/sKvZY6[˧cRa?j{u7MlX?BvG9u̽@؏ ̩I <7op~;m5*r^BVv"%^6ﵷnM[BQ߸օ/d[E^_"ll"Wb fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcmņ7'9BQ{:m3bsA= Ybl~ ԪWiGCxsa$fm~umwߕ:}HmeZ6xƆ9Gʏϣּ:#oaM_A7Y5c0w YZR,Ab#>MAtFk<&dYbRvcI߸(Yγ٭ YoBBtQz>U3?Ǣ,eR gɾ/BglKk }T v\sQ.o@>>{"WBS޾rg19澘uvy8GUS|s >]!eu6w_^ u gw)%ʘYs;خ΍)|tsacƬCw[ckM{K}9h^zg(c`Sߔ/wPi3^C_tuRLR d pP! B0/pĶkdM(ca ߘ YێkŦ:Q(1iKj,JЭ/͑>w\*D*Zʢg("/<]g~,Do+)Y17gut^~,/XU%~ūyWa'N ?m+w}nv^-Q_;?[l/b柷ƶlΦ߭1.-ۃk⼖r_I KvC_L7E%-dυmSY5ۃB֊M6ZU}񯅕gWJ\(abS3㟡9K3!+}}O=fufs>ucqѾ}Zo[.|{˳Vj4zUȺvG 6LO~9ۿ}~1=9$~>4RcY~N@XloK: c]{8 Ev!-BV|#&! Aa6 YNb0Jh0 ,(D&0+ ?NO~ {=#b*gOSΏa1HSkm'\>>ޭ%^tUvHz'\G-U^B q ω`kGȲ([,?7pT!9# 81ÚGpm`m3ƶ]XyVڛB >Zَ~U6yܭb~kBV(^"W,Y$sE^_"ll"Wb fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcyyͤ-svP^ƦlΦe[\gO;,oXd={*bZ+\Y_]w嶽R[ ?b=!Cce#~\FQgd-i#:w1Hf>˵f pT!KX*%H[Bbag7)(zM]ۄ,KLێ}5 Ҳ%KyV;T f1 &Z8Y(GS0XD &>72B,~uNA($,1 }3=M#dlR YZ\Z+6ilkq.VſV-^Z>lTŭ(U4EJ\#re+!Zei Q/yvY/'燸guk + իضr0˝r\qV9?hLg-V\$~l+Y7.g z uv|l<ǂymGwX,r͙Gocf-΁)Xwab^gȅŞ^tljp8od/n/ 1m Ed}no]GU3%:U_Agͅ+&ZqIGxAȂ!ݪBcx}eG"YbǩC.[IYLM#LW,&Yx+ʯ6 :ac\(!9 %F %;mʿ2! B$Yp dI' B$Yp d6_\NoLo^_]fzj& ݾyuLB^<7v!LD~|0=_xiZێv?L~ޛ>}.y~\alq}ݾ#g<.ڷOm R܅z/{yCC#7Qjg{nN?'ӓwOG+5Mgmo u&ð=uL\P_) n޲9QY~l>[6{^lV=]b!xf* ςBYjB R73Һ{+Ƭ}}v /x8>456vZbѪYKg_ ۝ܺ+XE*ApJV Y~|![Bַb̂E}\>z Q~D5#dYmw-BVħBw ߊ I@2&|X(-Vm&"L4ohC{Wz_a.1Mw+o<{-b+q _Ȓ_ . \h"a+}dGZE=_,cu fOj~].kyds`v\}z]7[mYMb !ɷ ulEQcWe.6Ըݿe?yNα4ٴl9_ ҿw_g}oTEZLK?R{ #1kSkܶCj+ײ[Wg56p֚pi^p%<ςnϥl*WŵP8uA]T1sOϱh&as0DC;Xmi- vnk.ʥVmZB'g_Dj\rcW_N,Ɵ:ӶΣXr8/s*/}.Ķҧ+l+᷎l9N1䷟W[3knG5ѹ# n.,}̘uuvq]i~?Pel0E5ε{\/$Ű8OO!ĦYLGl ;j0At VM\ B7_]lE.6V:ςb-1)D|"U% PVq,E+a*PZE._+ y@=M8YD~7WSbn,"~fi Y^,!K#WXaN~A~LVܐ[w~^?omnMwGC[c\[;2&96֐y-x\W˅JA{ݲcy(X׮on'摶R SL*RcSwf̅p[cxҟbGrp0!ˋJ ϔ`ެ J[ D' dn/E칰mo=!f{R6ZU}񯅕gWeP鳂^lrl.'7,Tؗ*rs61Y9=&0f1 s!+`#j. )zőT뷆(Ͷ188 >lؚcU^=ju}8FD*+EgeJUrόBU͞ fZc[N_ya!ޞnf=y=\띟-#1ssݴ[:+ko>_g1gOϳ4z~^.T셱-xmk#5.q=E|֪lc˘l-o3+փۡ$}ɹf6E*-X=Ѩ ֚SM6j`\>yCm{{ҧ"V6{+(U@4E*/1s,h"\EA7ϭ[,ΖEh8 Yϋը+_qT|.Q|r"('j϶Σ,PyzFϫb1.Gvlqi_ԞgY`O p^kȟ/XX!H\bk1߶e$dkϛrص~gycvסo6ې1P11Ϣ_3F!+ &MGA>-R!Y-\DC""v-T0TuncY^|}߹_kAvo xV lgR.Urf*iMmJ9G`ʹ=vkgsG~~ouO̕QR$BY Y Y@.x135۝u 6湊Ecla7gYnxuq7y΁)Xwab^gȅ%V*׋xcS ~# xpM68}aiM("p{:"⟩E|Q1Atq\x;bG)*t=w,8ҭ*D,867]v!%v:BIi$4ͤ}IAhRjO6υ2c ![b4^+ n;o dI' B$Yp dI' Bmt>ӫOӹ\O_vXt:SL{ v#D[v!LD~|0=_xiZێv?L~ޛ>}.y~\alq}ݾ#g<.ڷOm R܅z/{yCC#7QPоz*'ӓ;sΫw_LO=M~|4}VYu1Ο#Y5z斄LY7{V8.:޻F! #B F)Uf(?vC/}ogu"]WYe;^jp6}<=1,ij~mǻ5ĢU˳ ξ;uW*i^BZَ~U6yܭbpﹶWӥ/"/+!E__ĂGytq8ҳ?!oBڽ_Z>FH${.>oQ?ŪB֓oXM KȽf^̾BVeU{};S iG=NJ <64} }/ YsoZNYx& ;t<1޿w2UbXe 3PMh&m J3B^mׅ6!ckMBl/8EɒvgA W 6`+e(Z, j}~q6ه4w űZB7>C~Yk.Vmz[B'g_Dj\rcW_N,͡NA|d YgQ(beBBu}6:ufq##N=eO+w>6W>kLݑ=pP! Bl/pĶka։D `ĕ t ޘ YێkŦ:Q(1iKj,-s)̑rH-B()ğX8 f(V {VQ*ngEd^'-~ \QYΊ謋-dy߮,X(^cBr.1#Ϥe=g%.dʳ-ǘqoguU}h}w}U dl{1/`r0!ˋJ ϔ`ެ J[ D' dn/E칰mo=!f{R6ZU}񯅕gW6j~Te1o]#dF#پ.UVGE'BK_cf;f<⍬NշB;%`Icۍ*zD "!؈fG!d9uq/-lm/5A!&!X~gn]i_5+9q!K͞YZc#BѴwpBͫ YF։$S=3 Y@J!ەb͚sJ0ŬU+)BVaAҋu7fnZ@BV[\"~EhTok Qm덩Oma5Af0Mb˼`=LXZ+[ｕ GBV,ΦBu,ET_d5 RTuV\-2Ž^!_a d@}^LF{XrsASt"/rL+f3Y`z\ofN8Ƽ; YMZs̊|\ʷ=ڏwuYn9H-BVL4ٍ18!e}[>uBZD4EfwaoZLM[5&di-af{!ˏJ9˳y݄Ec_ žp ' bLao]M X)"Q-[`.vξ=,A\~JOwDŽ,G.Pоby{P·h ,v5gC1 =lٱ 8od/nk B.hgUg4f dnU!b:9?wEVimA]eb~/Ϻy|h9WO3dzo~yy'Ïj6Q♻~y>?xޫB;*'*dC~1x?3v?{d(d9aD!^(Yb,2EǮT :}<}%^1lG_ݟ>uK-泏;? MͯMp`zxXj?{yv' V) X 'dxn YN_1 fRßz/+ppBg??5E{~=]ǯ&Gn>pXՂJP0+"($GZɘaͣ|SX gӶXcۋ0zҼA)] >Zَ~U6yܭbpﹶn\\M/].ʿ/ !WP~}1VhgV8N7FX*f>uioD2be !YZ!d=9S7]zng}{Fƽ{rzqޟW_#΅$lK_l"59">Y_n߽b֞Y~ YZR,Ab\-9MAtFk<&dYbRvcI߸(Y3,Z\jXu\W}S1X1$XBЌ*,/Ey!͝"r,:ǶǍOEi]DVsjqZB'g_Dj\rcW_N,r8E! ϩ~йۮak$ξ/yE_:jmo}GJQ쏃p㿜bwNwHfHp V->|fQ&/Ë7!B baěRq_~"#~))yE"]2^#d,Tf1 &Z`Xu"%X5q%72B,~uNo[vV"=[~,/XU%~ūyWapcrtsp9-%碟8܏ۮǷ/c}*O뛟yȵy̸/bٯY-A6n=D>DIZ#b.kVVn-L&jDEy_i[];mV1DR d&dyqB T̛UA\i >A$,1ME=-'dlR YZ\ҟ"f[Su/lyj>ֱ2n^y0\4USYZYe+!ZeiN#m|q_LOq#ّl_$09akc41iMx|OIeq}ZHv1<7q7xN.MRMIḆoհEdϧ_Et։X[w>{p8˜ㄬ:zBV&56%Ҿ՞* kNL#BA "o&pBSGJ"Pڢ6B\n/lbkUypO_yf֕e^ܾRڛTe+a^k11"d M{/d߼Ѱod.׋&N2[YxDT°UY庋o7ǖӳ?/{dglj\lNܩB,B)X-"J]̚EE "I[PG&P8?,A(M-{e D~c5U_5APꋞV5Z87l3.bDx[koj[oL5}l{D 2qmX{_ [taJ֊Xz\8 1ھm->7ZoV=.dYq|ӫQ!+V>\9$ BP/rL+dm(^=:6{/^">1)Y8yvL۝Ѷ?>韝q?~9~BKa8 bݾgáHZCDV_ U#C!ezh"ŬE׾Ʒ5bwwZ{[]D}L!Z86 YQ0d7n>R)o Q h"݅ſ k15mĦm,?s+D>o/#wv"CE!v)/2 oQ9.\AS3}RNP=[P]Kͷ\օPЖc}K߸+u1!,m𹰋>cg]Kʾ9<$OU$VvrMe\im!ڳyq;1Îg73m(N~3,7 zB?1p+I)!k"ĂI81*f㱅^~. -dǘHTEM//Y ,Em7n--Z"Ǵ}81F 5^E+/dݲ`}ٯgo #ܰWg⣵0*$8!cqGz Y,bH)-R)OJ{{$FJ],62W9<9?E+c'BUo3S&6[3捚U*%d[JCF|~'n[`}5+p8ogIѦ&P.DBVZ`%o,އ}$,c=C mEm}(.}[BB!I_&8tL'K>AWqV/vHȏwz{ӿ=[iVl0V!+g $2W@j%_䚆E1O_ȑd7A,[Ȋ7:98W߈c㆐pn;RXsPn4Q"vj4E@n٢bjHM,MQD~#C[.&u-ĭG BB-s;;nXs5c7Z"H{WvG-r݆j./kXbMBa>/kٍUbZYNs(Ze V1`շ[esB֚u9dQ2l.K 8JZ>k dA-/W}_Gdj -71}qcM|oG l$,?N v~?5d‘ YUc! n[;\~ 685"&'dEdwD(!nMp>.V;7%d-y$EA,;ƿ;gnj?4^ *(b>CZVZnbP^Ҥ=k=̛ǧ'dI}BVj{=4iZ}Yp|q9]y3y}ajztn=w=nʝIB7S ~g?=Xȯ"qA(BVKzB!khĈU?*jD(G-LZoih(EȒ~:+coo"?}>?|}~mGiO?JmM>m}?.0>nߑ3[ݧ)΅!R\T<,=i+U r #|2=?zo{<ݓJGg5Y(]<]vBt!A+,aŬB"d%{6Q!k< fDMY#-HJs_=Q&de^O Ϝ?2GWYQr€!E ̂EfqŎ(2Ԅ?vC/}ogu"]WYe;jp6}<=ɶ.L }6v?=!+[qjE|\wcwK,^Bn|*Lz\8[}OϞոƔYpB8leQ &RϢ;$wD|uE[Ȧ#| w YZ /,Ķ\8Y(GS0XD n)pu YێkŦ:Q(Agk>X5~]*7-d­+Ƣ*!+QڳBn?>3 y8"q`-o`lg(H߳7K[򂕿]e Y7Q {7Ɯ.^_MW>7R2[z.ys,~|_gzh1I[g;Ü_X"&Y  ESJ[ "H' d"$oυmK| Y5ۃBHU!dxʹ)r(ڋ sij!ˋ:ov!VֳvABVnJoVD/+W6j~Tz1o^#dF#پ.H,/ϑar#:aLZ|dt~4#wS7?iAGп )"dd[8ĔM9N_[C[f_kBŽM0em2ǚckybdތqB=l%̵k-:F,/GBuZ\/q?wB*#G1w.d¯cyxT︙"&Y"~EhT!vl1i!f; KTZtc ~:qs{)mcs.6ec YQnEm_6 ls-ҷ`Z Kt;.Bbz5*dǗs "g>H> Yrx\dY/dZ}~B|nSew4mx/ckt{yv9<ӳ p(dEDH"`b\' 8Zxi2 lbj q YvKЩOϔͽB֎"w18!+sy>qWӅ+hb/ƼbnAu-5s[/[{Xƕ\ ܵ Y fi7υ]<DʾZ$?g$iqjQ)m7r|8x'=/|oۖmnN7Χ˳ᙞm;~# & <+ų6i'B-o3S&6[3M56ipH;tL'B$Yp dI' B$Yp d6_\NoLo^_??MC~{Yoݟޯ(mwc1GgOz~;R|}޶ ]й0DE'^ Ydzo~yy'Ïj6Q2qWr5]Xo׶0}I!&ɳQr€!E ̂EfqŎ(2Ԅ?vC/}ogu"]WYe;jp6}<=ɶ.L }6v?=!+[qjE|\wcwPE)ο) X 'dxn YN_1 fR'z~yn.NL:En,w`} UJ(7}*Lpʛ4-V)Q)\B`m3ƶxXySګ"X‹IZَ~Us0:AiS+,,w\jt7+޸EC(֮"̨m]7O*BV 7RWIޯe-OclPxN3gqMQ~u썲,V|[j _p3ԄrF %ul⌶6iW0yxvs5,\l>bi7Bx>0͓;4}r lwO4#=~9Mghݽ{f޽_lrg{1w ˽ݚ[b _x,>=P[{Q,SpKE D0sLAg|&{ۮ YmBV).؎}5 Ჰw q3B{7H Ʃ0 ֳP.$Y|>®f-]gɾ/ǂylKk }TPEJzof%d}2}Eƅ,7}b<5hRuy~Pg14b,τΞdL[bp&wDly3`I휫>&}˽X;"?w׎J1}ܮrB™ޯvώ/b{_3","J)ЂJF}lKEy=k㹳|y|mT,fxa!Մ(t"@9Q"%pp}LK_v]ů.6Չ"G :lX4ENJRi!+SEu^X4.P (s̏`1-|sEdeϗs߳[-dy߮,X(^cB[@1Ll׌<*>U˅_BQ}3/dI?܋_bMy幖 :V3;[/boJ_p?34p5_K Y{a c.#kjH|SKZaivWۍ ",,H!|IYb&)*ɛ>sazBV %y-mpUY1r.;:g|)\ZC򢎻.),}oEȒvoJ6Ssx%wCڨQżУ#dF#پ.( 稌qz?[S !K_>Dr&eQ(6s39N C ʈ_nㄬxrgM(#ĭ[(N*Jc Y}i@#v-)d-^cwɎіߊ 8CB!neLSg6:}o m!kmc|! qp 76N”Eȋ~ˀkڎ[ {3w gk«,bEeP_CU5bݲwpBͫ YF֩byMg%ay6A}~wIdK`BVh_z,e,k4;ؿ{ 7g}{FMkCED8 27?qgGĜU¸ͲE$l =1ݯOVD(|>e9!I"Œ!U!tN9 Q5\bn˗[|3-c!s~ ~dUs{5Urlnc̳mlӳ?(Weܭz{_~ˑ?1v YE8Zn[I&ذkl٦odDŽWIZG(Ha)ģZc{#IvZbC]hϛlbn_F!+ &OG A>IWKK]X`SVKMҶ[N~B~hvTY8닌X>qWӅ+hb/bV us k ~ފO;BZ=,A\~JOwDŽ,G.7i*er>WP^+r&1U=iINc];14H{<{}K90$OٔgiGD1(%W${ӿ>?J~u+uY0R&F"_kMsܣk]n\J׈jx=,[`pUp8D)^DEb gZܘryٯf~)~׷\r1~; ;PrL[iZ$ʛB -,[WEI7ڷf'Z8haHǴ^~Z$U i2?ֈ;;.! 8A(_uG7g_fqܿ;k-[Y>*d9=BpWJEOKE*YqX!w[,_kjDao5Ǯ1nײH~H?Y'?=Sr"S~>;n`[eJ Zb@n7%b􅊄}!"Z"AHE<> ?:ߧyz[.b{9𮀐ZߟbWg>8obeB)TCDmPrxBr(#=QhDArm,ǰ  +K y#K)>"Bv~h-Ī.~ d6_\NoLo^_??MC~{Yoݟޯ(mwc1GgOz~;R|}޶ ]й0DE'^ YS l>釟s^=ķbzrI|hz鳚Ȥ;1Asu>]t>vX署x'.YOg_UhM0Exzc/Qkm'\>>ޭAm qi~zB>Vռ}VQ :) X 'dxn YN_1 fR9+f"dmBtx d,w~S/w*,6-|Zc'd5u{ͤ3v#kYbTnN͈ qofs{p9W WPӧ"$IbEr! v6kl{']U7 ;*%WEK>h_0E2ϲy Wӥ+ֿqE _a\1;ݒ)ڮH"+}dGoX7-d\4I=k[fE,V|[j _Y9r^ӪV~ow̲O}3 `_F=B)f8"VBViEuv3B^mׅ6!Fl>֚pYqXm YȅCU=}BV,´+л".ZҘL@]s,*E u,ǶǍO ,Qsj"vKd(^ YnLy黟[z?9D!ysR&%ir|V:Wa{ D:3;esΊَ=cYOknޤwJ.;f0ƟeT,fxa!Մ(t"@9Q"%pp}LK_v]ů.6Չ"G :lX4ENJRi!+E*`EH"{?>Y$8QL~7WV3 u}^~,/XU%~ūyWaG NHб&AdV1 [v/5>W+QSzLؒxb GWwoJ;L "B. ‚L"+m' Y$%f`"S<-'dlR YZ\ҟ"f[ W)cjsȡh/Η.̥=,__DX[EbE*`EfwՄ,}KCl%DZW"ZiG瓜 t,ӓCrv:wϫseSz*91?06#ǕH~%[a̎kϽ+vݿ]g?wJ>+׍p`^h2ĭL bJ̦[/-dmm/5A!&IyQۏxcM1.簚E$?W|=Q ׻95yNq:}jpd]v6jXx.us k ں m9A\~JOwDŽ,GcSSug?<*WasaK9_2grvMi*{1SziW>mrlѶ؏w5ѿXE5ȹպ3;ō"WWUY]&)g(UDo+Yp"bX@* [Pan4@!j#/`g|%!$A! N,8I$A! r~fznz{/T;-qٓ|t4~`/;y:)!Y8P}.bz#?Bo]*w.^'L/UۿsoFu]h]t77>LEWe>^ھyvzֶӧO>}us\`Hzۂw{WB).*Wtힴ{ql*d@N?'ӓwOG+5MgmG&iۅ*uB֮!^W_e[\ֶg[Ǡe;ge|ߴ-sn\W=\~{i:wL)]d! xCA8(6?Qd 7~ B"Ӈ_hHEسvS'&l>xzc/Qkm'\>>ޭAm qi~zB>Vռ}VVao;ApJV Y~|![Bַb̂E} ~m\1T t| Í Y~vy?\kkw,2J/Ery5yvq3ϡyvBVK2הּ_76r 2J %OEINyơŊ6<7?zCm&O4ovC{UKx1I Y=ѯ0|2Y'(abee1+\M X( .(fB/T6.dE/(ؒqOj~].kydsbc=D^CBGwXBzmy)|V8Cg"/~t~qȹH39G!'wZS:U۵uDL޶:rEBϝ9{cr%kk a\ǽ2vg]3s?f,wlVgq/{xקDo{ouc< ~xj ΐ#k{(υD{B'f1 &D#N2&.Aİc \]B,~uN f;#s#z9En9TlMB CO~ϖں˜kn؎a/G1[ 3/˘'syܘxs}1s}= $u%&j\su-ƺ}C"ui1<ۊ]Ozm"~~ /p`^h2ĭL bJ̦[/-dmm/5A!&IyQۏxcM1~vlW/s^Bh/Q-6(kGs]>ceyy#v6W˹ji&mo{ٕs9Ng]c+/dĿ5=Sm" ?"~EhT!vl1i!f; KTZtc ~:qs{)mcs.6eNA5 E /dX(*Ά.9b->s ̵BKp"I٣2 d>/WBV=||9͠Falv.d{su f۲ n ((2x3'^3-F!+ &OG A>IWKK]X`SVKMҶ[N~B~hvTY3.W", |˜*FTR-0?v"N6q)+!>߹_kAvMOם\YWҹ̥ v_*<.d~u`*ʵ 3ݳ ߑme|sʘyF j­mQ(Ιk*9 ێ"\Ym?ܮȝh>˱eM=; h~tG}|?ZSd"jY#ΦS[<\k3wCjgb/Xqjwy6={l_b_^>kG--+84,PܮwP4+7@8BaQN4 -H[y'ۚg6'wYp"bXE! Vo({Nu3N$Yp d]_;7I YW͘_pX_p dI' B$Yp dI' Bmtgo^4K{tpՒ%ej"@hP!CiCjh5Bɂp0Enb<(qAh@`h(wrl߽eZ+Vg䓙gGĊ+֎<gñu;·T;s;gǝ{gÉն-m_&u]8NΞZmmN^_ /Ͷ~ 3ꪀMMe} YESGoFr ޽?nIy6o|~c{;vHO\oefĸ+c":qQP6׽u2w-,|/%?^<  >4S/ڶ&!3;,2li Wbo0ln@/90xPB}}8P"a!NI$PIP"KI`pcgAQ o~{]u׉tMg볻}J G?v_lp`xx\Ҍt?|l9yUBb Y>[BÇlLB ֧>ې\rfap͢)B* [K[>oX6!I!d=8*|Uʮ}yN}#\CdS!3H`) PaeOT"Ms) k/rqvc mgc+˅d*d]H(PQVA$+ĢV,B:,E[*דqhEB#ŵ>4>C|&G5XbkԄwUEc>]r{W^$Dq0}s$戅#l>)qsTlw=0ykZ~c67ߜ|۶{bcF9-}vc;QYMp}b>zZ]뢘:\wk5x (p_K0ojS~6ڞksk1dU!KᄅVv!$@UHD/xnĔ-dn;+MePOD--۱fiC ǂtRr˷AP`aE'pbSBgw–($ƱK~3ߨHlJ~}ԅ,'X,!īi, aa(dW+z(_$v㈞7t*[9__ZT}|K=}~}1Y9' 1snl/m;.i^Z?O]o<ĬȺΧ>y(0]__p:eosI_F8U`5!ˋR .:"WꂏAD(E&I0MQɉWu !d\~=2ZʄO>Ws&ˡ`/仕U\Q"J+XxU(d lEX[[+)p%? X*ܿ_ Ox"kev].9\|mM-*s3\{q=-(vim{d8O,;7/eUd*"|,ֱ~Ss8 Y|i~NxFݔu>VR2V3ian F95Ulo'N9\}3?2{͸aCqB!n q&h'-) 7ִb_ʳn3QN($Xk@Q(.+ P"Y"ZfcEW[~GB֡FV-H%&\NbeOW⃼+9oPtdM~Y- E~oe|]^s9szmGwnR7b֠_fk]Z4۪6ۘKs3Ũ9܌Ϭo|378T YEˡE*-X-(#<57ĎOLU}ZlGŹV},َR]ҢÅ IQkM? vֆXa*Ŕ! H?\(Xd_^ )ZoޖY"fQѶ Bbx+dÒ9@Ͻ*jkn*csAsmK;a91-uNqyX/ޣ5= y=cUX~{`z0G)-v]εvl'5Wh*k33cy*N{|Ոw˶؏=hƗovޯ-a;1fpkږoX:ا80 YA0ш'ad.\x} Y-,4DoXRiER`Tջ1'{,ܺ)u`W*/&Bc*^1uw6PBL\P0=/(g*4OH#Ɩo\1IΕO"nM3.Gx;DŽ3gOt.9 ҾG۶ۍKϟ d__qm⬝[B]Q:^uByX(.G1vv}]77Ήn86.7Vǩ}1"s#Q-qN~UOqkdؗrNv5G.)ռΆlgݮN~b <9ֹ j*dOWn&lDkXJ*Nliu]v,xQ owE9|k! i1X*R5ŵ@)6m aӈ˝E}s8sB! V1FW1;v𴶉5D,/HŹ9:'[6!d/A! $A! $A! $l;/.'v{/Oφ/Nc:wv.v^7Jli2r:Φsb8ylL/.V\ִ񶷍_ثpׄB[w?> <|W3Nm=;Qjm7x`Vl=&m}ڼ7N)#1>qѾ}o|\"ECѮۈJ\ΦB/mSpn=<g_J^=۵)~1<9?y^?^}hxd_mBq=2;6 Y[bYw6Xmk]'W87em+du0dEC F`IDT/"J b,gw$&A6~$6%&ťqX sVN _x) X^}]٘DOg}!(OYv b!Yy@SKaO_ J %I8[pbE|囂>!D~z;u9.*l~^"NLBVv E^k?>ok]_َ9U } ~ x>i?~=w|~v3qX\?oսkmfJ6 YAKAT -{akLA^mt}K=CkMLl8;s%7>_Y.לL$T!BQ/,yM@. b:t!ob*yRϿ|sG v-""aou흚AhL'_rAze+rl/ Y}s$E#lłz|b\6m{O?)uk?:lģاn -?SΘ14D08wb0q2\u^w!f_ !ڶ}_+X^˼$xڶ-b}ʼnIpBh+ ;AI`$H"]b 7~bJv|Ŧ2Ar'ٖX4cAtsu ی͠"X(U5 YAٳYp(Bnwx%oTSE0ebF~ YNrOWYB ,"WXf/6kQ,8VnPȿH 7v|LVE)uk?:l _l1uL?re$̷~z2ؘͩl?7Oqѿee/O Rgi8VrX'> Y^D$,p!R|"D,2M oJN0KhY%۝BpU&dx%3Y{aw%߭\⊑i8 +UT3Y!ٌd?\3nflEX[[+)r%? O|,MBOd~̮2.lܞDB`ڋ)s0E(ضIo)uk?:l7Z*{ݴ\yG9>67&Lk|=8_6wmwdBO3}VEb~ ys^c#k0`! `&CLxBH\ǖi@~s Yl3q}N[d; S>/JQ/=ni;ľg%ZgY;Êbhլky!ٌU4kF- ,[s-ZB Q:m!էCZL]?=-7ݓv 5f=lukk߭XA(MTa7w?$Jk7|kG-Ywmv,ksTcЈY4֘YƦֵ+06aUZO2ZSCXTէŶ{0[kǒ(,Յ -:1\X ™FYno`\mmXQ"Y 抳Y*Y1VbZoVB+8|ل!y1a[òX+rYeLz.ܼ|],mΧ}77o&BF f[H}{cbZ~tV}$qq~|/攵Һt~oa%?>@ϦoᚈYv/+:l9=B1e-w,lce{alB!+&$Å /qO51ˣH-K 12y\JB]Jwz3Du5[<(FŴPK1_"g*1uw6PAO EݶؖK{ݱe}Yi[ccg/y(ceٚB[ܨG>'} GNaU@-j+f#ٹa;#1JzM2!Zrod UI^,*vm,p ħ b+d]OD d8@ ؏=͡D,p@ ,p@ ,p@ ,.NӳغN{p_v$̳1Ue} YESGoFr ޽?nIy6o|~c{;vHO\oOq]'eڶ 5rAbe+16NS~ ؼ=<g_J^=۵)~1<9?y^?^}hxd_"]pb_4X,p͹=>{}ܺa[f>u[oG~hrַ_ogm`;Jѿ[kjqB!&8(&" A'Q! %ߋ(7u.D4v>7xgB FUmO5gۣx& sD+Uu"^pVW`-!C6&mQYW8y Qe3 d_0pv~(2 2ɖi~] 8?>|> /DZBOԶ3.dAX[ YC`", DQvϣ|SP^qv6#sl;&]U j(E ^wXί0lcG!+帮 = u6ғT = _ޏiJ'ɌBYIAȊOEaviZGߓ(DYEzhdfYO>Ο _`? .ݣk9ݽe(/ao?h]Tv_QPuqθAk~l7FBP{5<+ٞq\e,0~oԲmShxE9Ksp YqEƱSc=BT{MKA, Y*Z O{st5vYpIח Y<;Ū!KqHTOw"*+d]"q(¸.Zצ,xzY62oh_O>Ĺ}'>phk}h|,"kb]IMz{x ^ Y4&/xp $6⏺ǺL8q /{ߓ#>>>~+M3#ϵ0guDϙF0B3ϟ}wϣLĿ}m~ݠ-d]X+d<+DSkYP*zyzz 氪ʼnIpOf)$ F$x (>OL B _IȪD.|NJ]3Fٶl! !RC_k׷e.+Ǻ[yvE*n@çNjl>/|- A⾋.d9=]e Y7 ^McLHu!X7 GE^PG%kt/ ?/6FV!YqK%+g [ƊDQKWÂ">škrdT='SA]a-͢ɤ;\ֺT o=5Vv7Մ,')_|:| uNjO"$zC"No=J;Ʌ,4~=3f]Q,O2*{(5k?u-/BqgqyxlEX[[+)s%? O|,MBOd~u4/ޏ쩩Ǘϯb5ݓ'u!]]}&Eg cbe|#;=Wz2yj??}s[_}) zRF$H'_Ř'"'jk}[ݗ)=7*B M%ęO:',"PB2L\cd`\PbL..JBVݮƁ YV(:̵kkV-d߼Q:TJq1ܽ"rX8I0 ՜ڇ_K^3ϒ@o~—ƹŌ[[sY Vv K)oG+?sW6KIMSj$u.DQ.dۈ!KK&ZBVoL` "~Eh֚7|\ŧŶ-GSz"34’=>Պ0~E,vM'5 YUiTuhC!V:O]Y z͵p߼bayf\l{Cr}^ z{Xr|>Ǧ\`9p$ruRƺ\5I?GeP+|){/cB(7):IDATZ|lǼ1@!F\ZgeG<󱸔vk]}gݭwdpt iݕ(XV‰œUkӢ qX+kTM?Vz2.>86S؂BVL4 ه ^cD(j3bG /~vZXv)-?OmX-(BVv~Ɯm=c\jZbPa4  XdqpBM]tEB:@D67oP-cy*.d22K~yc4OOOj\Boi.%k"OPG7#zߎ:w=_;`a+=s=s)n!'EWUs,Q+߯k [x<9%<9uvYCf` W -o*$AE+fſ( jeK*=Q>"E(ZS$,%ZUM?Kq.^~.d6Y! *Xa6y៎鞬6$$Qfx>۰Ev[&"qpʲk1l0Au{?>>U.d d,fp0zFW '(CtFzĪZ[v,W/"PgVkYWi!+0_I?[I<#j`m dg.[ W\7"JL$O+l$[a!zFZ$)_U 5}/J\>vXY\AZbؗG_?:mōEį#ZH~=mBSA!蓷̗m- \I-/X'r1lZCY68B׌?=ݫ(R]?=* 3[24VU#႑_1n Y5 m=f1 RJOX/}ϋ7,տ"%?2mMMJbq@eAG|eЕ_g?ѧ8/MO6"/ D%U;S_`W?Q}j?@Y\#}a Y+Xťړ@%z%!+2 _<5!+Mz'T/4K$q.biqk! k7;f۾IJʂJ'?a Fu!|Dc y^ K4*v|]Bx4bkrBpph gñ^p1?{jTb\ȴC^a]!+y5/~bԮE:x?.[ph_W >_~QS'_`A=c~|- ,<*&̏Z{u-f.޹RƘbQ;w_ 6VǬ+-R%`})Fa}tk?/Kv_pV?;B^ajI4{Q(b@,ab}$d'^P< /1^/boIsmK8lj9|3(F"bbcǐdYkqrBN~1sҺވ|fBpɬDV,9/dr^\z2Z55zHHkGr&&x\KXI>N8Q7>ON%Xbو֎W/6f-!k~xCxuY{ uZP]^B4 弸mS_W`Q'_7;ND;d1 |]a*&aspqS`G!+CB<467W%]l?2o|f ddXܽvE{V,wBk-Z/Yp^r9E _D<^+!k:9jBaa 'z?kzO'96{Ţ*d܈9"n.vEۊ$UAe]:R(mWW`NXPv|q:r +䗅z 1˓Xo0h:ס .bƮ{d~H!Ş{n?#UCݤbX|, ("~SBVu?vXȬPmyĹ ʰPWL8 ״@åz" 8@ d 'ܓ+^\~@p@ȺQ_eY Y Y 9`!x8b8yl&Oφp˳꺋p¯` ,_O. d/7_w3ćNџLl+! YuBW?pG[¡U6B/V,ɋ[ӓK-*_Oiw~?{ʮ=7ý?N^(t JƺڦPߩ9cE7ľܯIc{6: {u$aVv9`^u<"$tl~"TB',b]!+$^ڣ0!p!ˏ"9;.Z~߆B+$Z{mGZș{} })M4,7gc|i>Ltr90ͭ~b !dod>H`.vDT-]$!=,)(D^()䂏TG/-L}5Ss1ؘyBVo]b.ŗ|m}wso~%ڈ~Y}#͍ϖm5pַ_Xpoo}@3ε%u0uаDMx-%|iX5D(Ӷ7'BִN,E\64s=E,Po_Mp4/ &DxM 6IʟbY\Z)+ޭ'! @#gG@0W[ O)yD۞Ġ 2Yv -2WjmK:nb>E[j N>sc! WY?oG_cْDʯ3ŃX"dMK4LD#S:LBV|j5]>gW%E2K\j}_Q[_g)е-H p@_T[_~'n}{*_5!+D$&m [?5v!,Fzm1+OBz⬟jo-k`@A_%H" X$ D6|BK?}Ħ-/\8k,e9"V$](BVT[';5']`l;/.'v{/Oφ/Nc:wvy:KW-J87^1gOm9ΦX1o2{pb5j{S\'ϣo8?bw桯^'Jb>XGȒOp7ijv}]!%r>c\*>N%~-f GITŠ,;%e0ci|! r!뭻 n7({0+?>m'ZwhljOʴm #:jtE6ȃJL/M,_DQ'bKɫo?/'Dz'OG+5 6p@VvgVeg–ō{}z}r,ګoz]S\",a7MaW}BkX%IB!Dɏ_*c_-y{j\$ۅ]siMB\DҾ9R_.c? BBZ07&" A'Q!!%ߋ(7u.D4v>7xgB FUmO5cOu*mZx) X^}]٘DOg} Ղ3,Yt޹B d!!~-S\",a#/>I/VW0/!+.,]/n|m#K,󕄗SWBLJBI9AB^YnQc"+eysÏO3:CgO_ J Lpll <77Oon32ǶiXy(ۋ"X Bluj sʶl?vy!8Na}=<hd?>͡ x[|, viZG]:H([HY Lb !XU \[\3R#Cb6~pmK);RY/܋˹Xcmڷ8]g~9|;٥j83u쵷QWbE넵545g|Fy,jڮ $0iQ)BdK_׭,8~o~s%j_B'`!g u!,{ B%[%1>➐_eB+ɷߓl\HskqneGsT{MKA, Y*ZinSЂN7r&;. Y2!ܟwX#di>ɞѽ {_Dm.0EŤPS,ֵQfvaK(ɇ8wϋ- i=jp]{&d=E,?}ɗl>MMlR !Y ϓUߘ1~+ǯfXU$f'B[I BeET#AxL\ Pݞi YkEJBV r9cŮ{Rl؇:\9އV+vbYbc5^2 Yp(YE01ލ46fQTvł](rR`垮,XD^&E,LwbF{Iۙ-옉ڙyVWxej>qw3Hn;ߗ{]^{98VPyx߻;O_U1soÞ^O)6b=E"H#bo2) [-zŁ$-)2ܹ02G~8,^鍩%diJc\*p\jګX@)PM:| uNjO"$zCt "No=J;Ʌ,4~=3f]Q,O2*2f PȊ%TR3 PFWg*>޲am$W$gFɏw~a!dD,_8H0iH>~g`a3eD~ypsWVEMsa!l1y scҽyՊ"Y;kXY4f*l"jV5 e#Y s-7oXPk%vawK:,W %slI /D˯d/Evk]Cݏ 77Α'joƙcYSKo_ĺ\uL&?1KuGڶ/䏷[7?]\kbW-klK[>iޗ|^r)N8wyO.XcKۋֵˉٻG1%__HO"Z,/x$M{{\)ͻ<1٥k̆_ڋ,w3_jA+sc!+5'9" CfxF…8' K!&XRiϓǥ$diۆ`$f Jղݷ1'd[X9?<ۋE v dqpBE]0q Y욈E=/(g*y >"([.^\q';W>!b7\.bkKۥuE˾l쥼=ݟQpߕoҮrfγ*I1itL^Gk _~|n5fVL1yet?Vs\sxM Di?Mߗ^~?X 1kW(߫^{0֦sA{=7'ƖkKۗ~}oz‹^o$dַ|Q!BVOŤ4L4)(S<>¼f"X\/E==~!+eZm /x-r!Kl|Ub`z" 8t|yi\ l! }a,+ F" YI M_ZzDi9b^gŚxGԑk-o3Dů[$%}mh!g>+3An)z^CB8U\ d$ d y=elR_5s;-!?9U'(IQL'a$[SJ+.ڔ9B֮LcůII,ՁunXt*`qX`BŸ|AL<+2I-3,MH%H.Y Y Y ep~q1\>{yz6\|q:[ɾ|8}OM)o.ӗr2|,ݛ2'Yp1<6ppjFq 'V=0{m]/ϗ툛;>xNl3򳘻~8Ӽzvwx6n=6pv{LynkS\߱Gb|}}r=)Ӷ-\ѕ "Eg,f{x{x/ϾzkSzbxr,$>~4Rc?Nɾh ?]of\kvւV\!f;QʽkJ5{m]=oV<[7 qsW8ۥ>ug\z:o?Kѧ{ îqe.޺t?㯚E<{o=gޅ,JUPRgc(d_Q)B䧷Scۉ4r^PE,!?Cr~9eۋHz?vX%ZLגyB7pJBƛIMJ[<C.| rb }xC)|0rfPfx:Mkyia܈{*>o)W珆O&j ({{}ҝg<͌Uaf~yƪޣu9x5J̜Mz#ڒϑZz)}^y{W˞mƗQɘr1b^3,Ǭ#]1-:ֺx?6Z\9stGCbl>*{S,~bf5֦}ƻ5vKg|5f|}<ֺYsnutu}߄K%r.HY|Zbw FlS.ܛodW{YVE_npbΑ aa[-/ic xD1Eyhci1w߂Du9HI KXy7Dg+"&{ӏy/g?s3Ԯ}/d;[qv'~w8jTpg{(qm1#TΜ?D[!jмU}k IwYj'c8;UvYd/_yd*g20䞦>[u!O8x<ﭳ[E峄{ XR=U6%-t#k/y|wU)^=B907܏E>N~} YFoƋo^@1x78p÷Lo߿|sGP UhKoԇWsM\orEMz{x ^ Y4&/HEcb3\nMo,ͬ 7]o8\-%esc|kQ]jG)f-u k6˳Wkyưu?_&#v&unjlubRV}/;_ gXZϡkƺܜٺ`=ky7^oY8;fn|O3eqs-[:cߟ<5 f3}L85*!`#^{ۦ^#sho{pyɿLhkU!K m%a'RIQI12q%@}LI[v]+Wj!+vܓe>ԩBȯ7!} Yco ցfr7~`ZMqa~o>Fġ"XѾ_BU~4˄b+E7<Ƚkљ#%­~܌d~~myc"U1C4۷ּ o-h|3ƕ\#bŒ!;v1xQn0_13ҵ &{SHSƊyOkYgd`ߋU徴ۓ#[cn/6}襰.zcvsx̩{Njjφ+bRڒ^ψWGrl {`aK!2ud,&Vs>7[ȌbRʹMN~8}?vgޏҽ33?{M /`8fzܓYQOi5!yB,njTcĪ06Qυ}Dž,e}o(߯l$[ufbF~sA!֜=Xnw/emw5l?hd빲WEXNౣcH*G*//N>Q Lvk a#8ɾRC*$|\V|̘ub#D?ʨ\rj+wXo<Mކݐ :C bp8T7 >޲amd (Q|fqmܿ_ Ox"kev]״cL(Qeu$Zk4Pa7G5J9eA}gEN>13*u`}h`]\ |O:|qo+}1s~D{<^i]ay.1ڄO]Kk]m=}&>|]&<[;r-rxٜ(UXC#Zc=%u:&x o2n]ּb=滕dž/f>dv|7f0}v 0eŤsX-S}ط)imzg޾5Z~1MR?+m=?oak/π?j\.ߍ,?<*۷󩰎 Nk}\1)%1bZz9ss\Gύya嚜cwۄUX_9˺qsƿS4-;|pn;U,/X qK3^yBDE[*kl3q}pʊ/o BQ3(Q, YZ,؛> <@3ꛎfe'?ܛ+Y"Zfc;jV-d߼Quc|_vz'^7Lx-:sxfx*fs}\Kè4ݯ}W5o>F{{+)u?fct;Sͅ9eAmmh_Wul̒巻%~[8+ܛ՘xk]Mc >TXJ~v/ufcT=+5M99g c4-N=Ϫ19tgLCJl)k~G[Kv?\N>vz)gFVukГ3%< z1bU?Q˅zۉR]殺܇487f@g|Vs,&̬ŬN?Wgۣr+B1XF@}}<|dh݅=<]V౳X*X-Ri_jFA'>-m?Yq>^_X7m]jM]b5ƶ!ٛu#\!op3ṃ;3-:L90}D. Bbx+dÒ96G<`}Е#}6Uy{Πޤ,F2g:ْoޘl3[:Xƹ/|*ޡbVEcl˭w&asA5~}9MP.Χg#}Z[l;ק+h1b|s65~R1pF;ς,W~6by.䂈5޻뒶|ud#m7flM˗s@#cXWZe{}sXQ?b?GK{/l,ߟ'}v5XdQz1QOξ77qd6,&vYۜo&ݓGETɮ}VbU3_帴GaI3ޭGUT,s߉hǺs8[l_(dD#}pE06N􉢆>4^])<壅lbjq 61ڶ!舵ق-dlg 3V_~ҋA!nXop>~< B /GТ%ŵ|Lo܍72:d$~㊉Ow}B,mws\yf߿qeoZ$vc"#a.昑&&s>IOUUb֢&"}y~&ٚr#[̱¯q|g Q<:/Ș?rjF J>{ތg vvy +{ֱw9^ˡ7gֽI͉Y6bX1[1|m՘H1w5,xU߻vqȮ8+dr,Gzm59:[5[[Zʽ)Ǫ{y6zsCI_)rm)f LH_^x u<^J9܎Y{s|,$gv̗f{"ݞǬ Xg;fjv9-g$+qVjbjaݯ f@"(>sBlJr_Ř1N9~yrݾ%'<Nڜ gp6XmEyc} YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ+k{ݤgbncK6|A0ŀC>RۮOɱɓJG;%m;@űaEBc̈́D9! -Y(dH'(pxĈIpT/"J B,gw$A6~ Y /'6"^Mp`xx\}^gLN3s? y!:(`EHz v>>dcf>}'PO)n.l d9p_-蟾B%("Hz|囂mFv\8l}{S(rbz!b9 BPee^h]K1k?AE4u4s'B0(]T珆O&j ]{e2/s"uF!uoOm;=vi]O{@쉹&=ww~kD7fRgS!K5 ,*n(-|h'<4m  L/y|wU)^E! ڶMq U c>JPdum*r~סo_O>Ĺ}'8"uhk}h|,bbuZk/b]IMz{x ^ Y4&/p` %:|a/kOE7̏Җ^1qQs;͎[9%{Aɦa?K5Δ>LCp*d9Vu-N$ F$x %q*|Ti YkEJBV r9n>V욹'%/U!kgI+W`~̇Wd2b/ A ܢq0c7*+ۢw~}ԅ,'X,!īi, aaY/a/kOS_of۴KXu7V蚎?P$:8E_!.Šv9Hy YNPD*//>Q Lv">, bKo# !d\.BGoŧWUY2Vq1vW'CE`V5ş׺yx!KxV%Zy"Bj湍\[,niY/'k_,W\lE̵k3 ]:m!էC,;c5k_+‰4,cŌi_~rѩ{ "|1ˣבBq.Tm) g3+XȪ Za.e􉩪Om/':}l8*OE QlX#xa]SI}kyV]16~CA:+Βb!}Qzz]+ky[z"3 -Bbx+dÒ9@B*aGqz짅,:Y.w*'SI-3-(|DNs}0w?|{\(){@+P F<$pŋl?c F5Wj%o=+ ɿK15mmCk]^qu Y Y pe ಁp QBOo0 ,kG3oɭ[Wnk~%pn}c=™sk㚷i>Fv1˶3'Fz֤g5|LFpSpuxׇ?=پ~$kP^TڶE~8/L 6S~]TԸxcHcn{5n{}qU_G?*[Å,:Y]! ! bx}blt8}pj 9N_m쏓:}Ο=ڐѿeBczPM/S]pmI'OS+Gc-~{kg(ih#Ņ1fQB#*ZO.bo\Ȋ6@6iV&AjÅx^^8zisBg} YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ|)0̈mBp6޵0zx{x/ϾzkSzbxr,$>~4Rc?Nɾhې d+DɷH)7XrgpbX!P %d5sjWG4cc[J,l^bqD~;z9O)E YQLoQ~n+ :'I% qRKyRyU"q)5^/.9XB!&8A!&* B 7v{!qWq]wHdf>;'pc $0r". 5X7De[+UTDA| Y>[BÇlLB ֧>|8g!}>@;(}cE"I K\j9/U!íͯ$XZ|FcO@ZWE^!kx~rB]=W 2PRgc ^ yo +Nfdm'˱z&߷Et >ί0}v{,!ku= u6a:y⊃)+R'G|dTOE Olw~< }Ih[=Y^~(Y Lb !XUkE^ Y.ObO9ee<˄/6i U_)bվﳹ5&!1{9ԽWg͘*>'{Ŷblo=[X<+bb<ږ{׌;_ZO;f< 1y$Bˇ.ZYl)dW?\ԿZFV?0Gv+Ĉ>񩚞v` P[GM$?0B}X*~{/{o!dL YcL c[yf>VPIpGh+ ;7E I(6,qȄnB_$r9n>V욹'%/YM~+#7ogCkyٳ VQQ>#:8^f~o_m7(hKR`垮,XD^&mŴ_~~пeuV{]˳Nvd>b }HPߛ=m{ o #5[YĬ+ϊ5)1Od~p}d6_z 3,I! Y?8[_~+8kv)IT* $ YP" 5@ [8]jm\{$~q~ޮť Cl!dD.'/>/'d9Ziy]_,b5! R 4dWꂏ?D(E&/]MQɉWv !d\|AH"c@ W~5Ĭ] ^?R"MsOMMCR~*>kAbk/i[TYSȲE֛(}{EPDߟ_4O-:PU#YY^0UsB ǖi@~s Yl3q}QamBp㩬FAEB^45 VQ,W\(͵k"+N[ryn!~#Ǐ)~;ߴQi[]JrsF82!oR5/F7{asO"3,b6˯ _O$7Us| >&rZ#fmЄ]y;Nѷ_ʢ5~SG3㣙%p|$ ICH(/-9" '+ʼn@d=S+|{&fĹs(8Ic$$dя@THwuFs'u_>.R뀐v,"CTZZQFxZkojKvVz3!EǒQkoNjstRX*hf2+b. ɆU ت [+Pk[AHo %_6Cr}^ z{Xr|>&xJ$p{f&a٦'mw|u>L.䁱4xt2:'o"ÿ^\?FSEnY<+bu6(p?Vl<^7b|rxj̤m\\8D! ?~~I'ߏk巳ZET.9Bd a )ɷї)$6'fHAen[U c'C,IףRDzOjk >},1(k|no#g9u!V{n,`O>\|"'OGՃZxY h4 $%߹r_8 Y{O#r_b{e`< 0WhI?.byPn_9GX <,Yͩ,c`*tsŊ zBV/?_gWL?G.**Ro =㼽_)X%iqEQ*ܯ!d/׿c Ĉ7skk 1!n->F;\䩢Eϯ]AOwN,׋!h\+żfkY@OA,p=鮟")~! Q=ݯ=pgED"ߧr\Lѐ_qnn4\i I*VkB۳EüzrH :5/W-4VQα>Z{|Hm 7:@pn0kYk muǪ /HƐ"A~˲ =bJ,.#decCdpkCME?QUd*_i^&hlL>cw\;-bͮ5㓿O9Y׈eB؊?=ݫXP ?=KH@} s! L!0D KL1[77|"6o//g˿ii!+g6zl\7G?Q+HNZ5VB5D]lu \V$q.Y@)6EBz$>XBKQm%z91 1LV O(F/|1gbd0G`/|~w|<}bg& u+ٞw6ؐy៏fj_Tp(sz,~"?/8%EPdR*†g sټRĊy?:NZj :(VJ1冋Sَ[1}u d?mkYbH…(D8-8-xI6Iȉ%D6?&V>n.<'drWZ, d(N[ gW)JH&NOɅ,짧X@A! bx}blt8}pjsw_=Umv 'pl[+ٜU/u~'bkr!뭻 n7({0+?>m'ZwhljOq/>oS`/W9*M,_ܪ|x{x/ϾzkSzbxr,$>~4Rc?Nɾh%9;E!+-,."XuN+,d .GOvN! ןB #L qB.MU1 Ad) n 1HBȽM/| :)̲|}vwOO)dٽAH`D]ī . k|oAyhMWK,!!/uKxIhA}-n$n!dQ?~y6I˛'d]9cVjA|RJ Lpl $BV>MA{۩ی̱Dw9Vq^/qb۟NUn/[~%dp8y⊙_EosB{"P i,"%bOlw~< }ܕ'F2ڪxnS?>ĪB֓'ؖ\˯c~V(xaxvpgF1e .d>m$S;BV<8-ˤh6xwvgxl9+,ִq 3bS!K5 ,Q$O6-|t!dl,d˄,sߝbUWQb¥M.Cz,GT!+C12,uA,*&x1z 7]$|sG -_X]UsMjg;5!/x/dјOd u5 &<ض-9鬒ҘBK>FƳ.d9Jt鏑AL1~u>zmzC>>.v?>;wU,'T0`3JŽō$@GJ K\!2ᣛWo 4\fI K~;{#>: H*/Yec5f/@EQ2 Vx͍z6fQXVWT.d9=]e Y7 ^McL{$ǼG~K@1ٶ-s [hi]:s%"hg8&;FȖupbb%&-(=׶oe!'Մ,/4H`0U^\ >^}@t7E%'r_Y%~4Nr!K @!ߊ3-\kXZ:Y 8icVBQY|ϋfᓏlEX[[+3 nGWf/6,Ó5";Z W~hAW \7ٶ-5цQӼG3G38GJkWc$aBj 4zY'랍m[Bug! `þjNBCxB!>2mou!kmc|&12q# UV|y{ ,Y~Q^p}\]}< Ƭ F*Ⱥr^e+b^k6 \!fouBͫOu YCDy?5۶Es am"GcNm9nls4΅1d _M\kMm^_ !.:X#nί||;^D+ kcVI3/Ca7+[EZ"+:am B^ Y|MqPA\>(;JnR VN70sv^Xj]1vY*fn ESǴ4J\:[ht.u)lroG!k{sKۋγ YA0шld.>x}|rᣁA-,4WKĒBLM[~<.%!K6#6[PNJ9!t.yU T ETLTW0cLF6&7Ά*œ+ %ŵ|LE7o ru+&>߹r_8 Yݽra.kW6==> m)UD?a屲MBʹφS!r~}Jhc~]>z778s2"}nٖ~<_ <|W3Nm=;Qjm7x`Vl=&m}ڼ7N)#1>qѾ>u9Ԟt^}ʧ4^3C/׽勜(_ })ymXIx}hx~ᝒ}4;͂ 29MDDiD? YB XV^> dgEC Fx!@LBYJ;C RroxS8+킸Nk 2l_R|vox81G~9gj,wbE2*Zu]B:x) X^}]٘DOg}FBVxgyBֶs"8^Ix)PnW 2PRc(^ yo +Nfdm'˱z&߷7'" E^f.3KJ7D@ X^<';NZ>b@@=HY^~!I!d=8*|mɅ+)P|"#oY΢ _բ]((L׮O4^ YlL{lUƏϒh6xwvgxls25i/?Ĭ?܇oU{_-M|5\Lqi6l*di1ý& }RH %ቜqnS G7BζBVL2)Vx,&\jD2dWȢyx_?=5g E/*0bd,X–um*`rݬס yS&̇8wϋ-񱰧j[yqqԄwUEc>{^֯ OPew0mΑOgƜ/ca<⁳)F}1~u>zmzC>>.v?>;wsIüBޛyƩ-L1򵈽$`b0W/؅U,'T0i3JŽ$@PE/lX ~B_$r9n>V욹'%/YM~+qT +vQiua,1/"(Jf5ix7|_FPe[ZCa.o+t%d"x5e2! PLg//TO`5۶{νa9-KgqDL:dgٲSX!H3pV=׶opp{ϑ3{in[ko{Dkns]}fm&dyA T2J]d7)*95˵yBVv'DonjkY형UpNV.L"^Vų g@y,|kK~xE|f\m(l,MBOd~u+{'rݸf۶~4HuN5ϸkWc$aBj L :.cݳ6um^<f$ʶ5/D5wnƯ̊x*B }՜+!Jhe B2L\cd`4G2AX~P5v`g+/; K׌bU,[s-ZD@9V-d߼Q:4(_kO^6Qum0ⶬB=QӼGg#sa: ;=}2='/VqXlJFݶF;=f$ʶk/ݾ)sx[r]}ccX*X-Ri_jFi!-}bb=BXI錫:2PKEǒQk/Kｕ _3xl&g4f*nV8s56{0}DRM6Bbx+dÒ96'_.KnReyo`za|=wԺbؤ{W^w~k@̍a~ꘖ[^g1/ zݻm%%V\c@f43f~~|y0 ϝۊ{ND,Uv_L<6&7Ά*4"()baJKds3k) XnނO#ǖo\1IΕO" c<(46[m]ƽG-jH9Lz77nm9\m>lK_c{-)-t˖%8{/I6K?sv}~8Ѹvmʝ-mYBC8,r,^/@! i1XJO짐0Y,jAX:@ ,p@ ,p@ ,p@yt8.^<=.8dq>v8~΃͠9R:+/پcMz|zOs9x~>s19r!뭻 n7({0+?>m'ZwhljOq/>oS@Wyz.mɦB/=׀Ó1|]Óc'珆wJEۖb_bVNlax8b,dg^qyYqEuIk[vdEC FxĆIDDpT/"J 隢,gw$A6~,~9gj,wbEЄw.ܗqe w6k!!/uKxIhA}gk@YFA, ,Ղ+!TYBIA-hN>MABv6#sl;.]U 6 '" E3LU1o* k_b( * ?+P1݇9tlGx!'"y:Mkyhm 'qQ~umQ@.S?>ĪB֓'xgΑ S?`O-6~pDmЩl-ֽv=U#U{Mvy`Vcy.dU3ca{L_4wEv}dw\aΐ}/mH8su+_˩uۑJ~<3d}TJ<'z1&+sH?}m~62<.{((~a̷,o#4ve`nǺnYy1.ӏ{`#p~%?ƺ)YbV~4V IXOROVy.0xCYdjb|eaq8RdJ;Ʌ,-$+>f p%_w_e?y7OhA+ T+U14բYgq oيZ+W$W(T '>oU!dD,_.d{mJ/G1rkyW싼-u")ӵ*7/97T;(gov #ekKz?dUg=f 9<3JNZ.{]Y)g> OQ,ždq>%d;cP^/ )/:j{uHuF>X>{-}0X/͓q; Cq!n q& M>L[*C]Zfk9 UV|y{ $Y~Q^p}->ڛcOiOXuͦqgC}|gYX,d9ԑ/bZSCXTէŶ{0[뎫:2PKEǒQk vߥ[+cd"Mh @IYa֌"RV|"u67\ *-Mm-:,W %slH#ntR^|7渚Kτr-\)O!F*i'c~W-Dҽ }H#fb?bd%s[̣vJV}+{~/3E9|SPEˏb\u{i:':"{i|*c#υ5y#zNEgBD!3>xlWFܾW]s򘭰K,0d6G6ˍ|?Ξ򹍘 ÿue39?֙֗OsYY Y Y Y\6gñu;·T;8~yٳvUeѺ:+/c-d8 } YESGoFr ޽?nIy6o|~c{;vHO\oOq]']ĸ|) ƫ<.kڒM,_nmx{x/ϾzkSzbxr,$>~4Rc?Nɾhy_lw Yy<"kp儬?$}^@ ,=:B!&8#&* A'#,%ƍ!^7EvA\]'5YI|)Ńl>7<# H!h4,]g\[c)WKr߳_  Y>[BÇlLB ֧>`x1.(\u8iGA8b$B{OkՂ+!TYBIADЂDuyo _J$䧷Sc pr^MƉ}lu*sg-T,sc@dz4

aAOeWG_\/VPĦIpBh+ ;7E I(d,q>eF[6~u$r9cŮ{RBz/lTH N[,ΊB./fE Hǻqpc7**nJq]zԅ,'X,!īi, a`x3W4Jh?ay lBKU>J0χY\ske] &?r7]Z48=M6 Y9a ,D[_#í?]H䡯ƿ}Mno-=}c(En~Ol8nq- &dyA 2J]dk)*95*$4_H13•|a:];N,Ҿ,)F=`X["-5nGϛŵICr~1<9^,ïY.PWqOԅ,B?E]>{-Ѣ$4}6.J>ds8*FOۤ=k-3Y9a ,O=58𿎾x=G_~}DHUrvFxll;kI枋yZĥ@tk[7e O5!ˏ^ׇU,/ 9W3Adle"PB2L\cd`\!-|-Y6qyjbgWT_qͲ1b5K %hi Y7>}-d7( YR!кF{^`n&簱Cۺ",~9Oh<}ngz._영Á aoNBد6ۭ Zd \a!L+_}}ӣPOkXȲh6QP TDWzƊ$1K _K,zuO,?b! ŧH~%eر􉩪Oma':}l8O4EǒQk vWؗ~X,B.Y#0 kFYYdkVs-7o+[OP.Q*oG^ Y|MqrFI'1ˣBM$.ԴRmCk턬MBWKlnV{]XZ qY-qpB?]sžBa; D67o-c70߇o\1IΕO" cqqow ŘD{/ZW~¶WG!y+˘=)cm350w7 _5/kTq! ;>IpBִ &i)qUȲBד~# UȞЅҵXp#(mU" _}k( _Rq8;q$,5JR_m5.Bǯ] [@n;RBQ[7 ē>i`ӯ;<H+dyb۷b!|],?&iF>\O d5wMWA`W+?}smu,'’*d{ɷ],'E? JvYMzR0R%1iKJO;@ˏ5,OX널B(}G'~wU'c]W|*i A$ҿ+#~i|6IB-.yB7Y*`㵟}{mYI01DYDV{eEs)2+.MB7`oG_sOkY}=e49)5hN=OW駙Z\kE]Bw&~O믎%d[&d_=bPIۢUspww9N ,AC#YVL!l~I+dQx˜%2iz,o|[$i7+puep~q1\>{yz6\|q:[ɾ|8}Oj38~yɥ9o9yϙhĵ=u]W.}=em$0=b!+>עүm"dv/%y p5Y.duw8~wu;C~mѳvwoŶgvާS\CI1| *˚dS!Rx{x/ϾzkSzbxr,$>~4Rc?Nɾh{%s\~1J0SȊ8Qg>I+e!+B:B'uVV^!K2v}ڿZ,Hab0rh0 ;Rnb{Û_^iuu"]Se;pėR<{я9 >. k|oA~Rx55z h;o6^pVW`-!C6&mQY7ܽr>OpFep/ꉙ#Zˆ/WM}/dոx=sNjz ! DQ*<5DOy1f}1 2ܭz,[4㾽5=Z’jT>$ֺO+JUPR bE|囂 mFv\8l}{Sqbz!b9 \e62U!Kc$#6Pl8 O <Rݓ dzF6JPm,>u4s˧SџQ~u鍊$V|?UL9SGVoWat} ;3c-ih}k|8'T8_Қ;yx\..m{]*.ٸ v}FW!_*ro&^ƌ\6d''oZw{_xO6X#GuĬ$ԟĊE PID–GȪh=g Q:~ pTb{MKA,q%!=Y9Bg5pXUrB&1 $ZH/|$ċ.^Em!kw\aE BbfI K~;{#>>2tun+U3 YA,/eW]@ m#fᐏw\K~3ߨlbbF~ YNrOWYB ,"WXf/ޘrY(n ]/wT ~glmg6yLݿFfmזֺz:I\ P_9qq~o*?& ~}Py94Ӝc{=BT>f{ۍe}M‰=OQ]7H vXĮBfE) L|ʋ+uNjO"]ߗC"Noo-!d\|AH"c@ W~Uw>:+.R"Y@K}͂`h |e+b+)Zr%? O|,MBOd~uH~ѿ}~9m9O*Bu2uq=_oYzv{udinlV X;c>2sBObg6rq'}?}ucskNql@>6l1߄ 9bK?';#wm{WnĬe;XrA#-Ri_jFi!v,}bb=BXɦθj[ίMuѱwqs~ck@f̂`h抁  YlXBU0Z1\ Q$\&tYϋU~K ,g.DNpR˺(dD9}6_I(+I{kj\쌸NϊDfQϳrokXr~Zَ\p_p1<čB1ыkA AcJ9퇋aGi/y>s7C>6Uٳ}(ʉ-mհ fP F<"p‹`l?  KEh!& lbjq) Yڶ!舵قvBV ٦s!+%6l.U,g*5f|pB&]drB0D67o--}!-]/.߸b+uc{Ex1K.xnQhl9˺R~,!G ]iǓ|,cN69 }S|rZXhٓ<=wOa|okx$M b,n硎E/Oj?XGϭ_ uos,ouZKwvPL[Fuŵ ;C;+'ZW6j*dOWL@ "],WY@OAP,ӵdD,,Y+Wk,p@ ,p@ ,p@*B˿<}0Wu㿾d88N߳%N^_ w&_/6%p~AѧFnppl2ngÉնlpE.C̿ytquV,o؞W.7=9{wOǿ뇋;l쯯|4k/ÿwh[ʖw 1m:257W܏mcvʞgdVCZ&|U;v`W C% aox&(? w+.o6CdBg@ 7'|~?=9ui3m;۬3j}j"{<.O{f4Pk"ubm̷rw]Vy݈Ye]"^#~rl zcC_fTh>$ ;uw8~wU?o=#o>fm#oh'Rkkcyվc܎1qĸ}:7؞:ᬳΫ"\Ý; xnP!x-kY߫X[P<vy_gsPsD?x<s.X_Yd]BV}"@5W lϭ"F '!kd~\2? Bڄ?w֍`fs!˾:(ƃ>h͏O:PaNC Z?q_~@rw>OVBV+$N0,aD'bߒ;a(½qEJT ] X"Kz3ekdѺj(Mk2@15bV[1-t Ǜ{3]qfM^;(bB}}GBjҘ[ㄒ:Kxgۭόh \v&h FO|^6c|V,A?C1gZ]ҽsC9w& Y5j\c!kNk5v\TdR Y?϶^~ߙ+7BVrkܛ uuM0ްøy(VF4)0TἋtLpSϞç'{ LW?B+ #w~-Q62?oZ_N(* (h]EB izD,.(DLZ1kYo Wxy:pT=Fy{CmyN yǃ!r1qguߟ]=7Qßr/FE!d.&yW7<$x;Dȗ@=t%ئ00!^7w/>qM^kōCvQq1爢Ngl+/3h//Oz}9g#yn ? ՙ}Y<3Y{ksl?mZc=Dq"5J?7#'|_MV})몜)4ٯfۏ?oG#|A_Ŭ7|#=̯,}s̋Kk,NOz/Lm&gbYd6݇dLh|ypu5qZcFvzоS18lA쫙Z>{ѿٰ]/j#;~9XGq_*k[~86* 'MD5y_cޗ~tǣ˔ja>]QvCZC qn SUߨ|ת]kR{ےj7vHNS/kW1l1/kYbsUrn,frK1nĬ܏l]2~(퇋3zϳ?3 ?lދx|zOn|F2T>8OXſ+gÞPZ<w+fJ4Gl|):\AatM][FxL\Lt'cR.}pΝLt}YY9ϲ$;؇Ba5`]?HCyϜMguǀ2qNGy?csk>d6KEcn+3WXY}I5)03P g gu\O yUaP|6(~{?<-YL?{3>?Yݿu,u|g)t^WG-YЌu6ƪ|2;cYl<ᇓ~N0cSyM@HBں^ mNpD7[M[CVX+ YSq:UˊkK,P fo kl-uU#_d۷=ɶ̯PD2?W\gHH?0w_&a0ߘ{pKt}p6 k}#fo~x?y.B`퉱Nbz"D6!d.%Èe[ W\V|z-\ZүN~{~g|]gzϰθy&!K I1/~l76u(v/ٗ;@WY,5?= vXgy~d? dK[vY:K}_ykGum?mBtMײJ_z_2{hMLv\3]_d^/lm1my(vw>6ZbP^?⚩o~_{/{S&_{Nƈ((X?n1G#G0C~D(g,nʝԼ/ך;#Z }M;k~/iZ1q(C.ZZVWkbFkZǬMc]޽gڪ[egfޟg ܟdx=0[nك|]9do\+~kQi^wcb?q_~7cRnsm/d=h> ɖ?Lt=ͣ9g{+;_Os:}8DKjUfH Me3}I@rA¾EP7GW'F+Psh]p~}ЅkAcOq7bcV_+>ͭ}*t`DV z7?? d6 -gLe >SIreck YSkBSinz[dئ}lvٮ' ֈR8K=3w}~xv|Pδbna{^:j{$)GZI݇G~]LƸx9o9-7ָz9RgQ'Z^N ]luC}_w|$ ~ub#9fWQEs7Y9fNu4 svKZznFｚ 5m̵7 ק|ow{!Duy1t=?h|[84՛=7N/s@|G5OofH C7d!bDE֦W_fם}l*<68!G2q4]We?a;}ޞKyj1~Ĭox<ׇk"!G/OY[))ޡxë7[#X{o&lyGJoć G54\$G I%>)cP;hmn" T6g+c[V{9Y%^g=,3~^egxlbRPν~v^1{8n?s-:9밑,c"rU_~6sc6z ߿ GiۍϬ?}ǫ6b+7l?G/ 5vKyb&Nuk]Nm2~nb-|6u6ee1#*Ě6+N=mC9Q =Ba5gܱF"H5̶h_mѽj=㵧J2|\\|,so,f\YgeGvdYC8~{oJCg+{_?V}hrlrz@T" \Sg2y2{mӱ&r:oߵ|.`0ГFpu)dK 8 A#W)Wwz[@\#`t.zoޙ~Uq׼B%rIE}~1ےK؏K !k)@C>WLn UGqB΄E,**'d?MWXo}?,Kۏk)d]y=Ts Yӣ7 L:5 g{VD;/=5Ԉ_<=.8Cƭl86bM>N%z8".W7,[9|׮WsΓ~ ugw6:K=yZ;Ë/7g4wKa~Ln͋_ݻ 7g+ؑxV;zM7(ɾ~$C{:q) 풿k ޞE#Y4q[Z80ó !X9:΂Ђu$vG ,ce`d5ש"2*"22~0weVfdddVV\S32t%VH!+|xpڨBa,"y!~x"O>en߇?GS̗2Hj(#IeMMTݙ/*slH}1UƸ56a[}.mGL6*Ry_%(YC\/ _ZW%9$9p퟇"o2}wd}#o!F[66O<޲mTx듧|ˌiŽ߶$]koY&g"`mW6dx[S$gH{j}#r5/2Treple82U~x)sP鬳"BhV=nw3VnGFd,nY;+D6`N6P߫d8fJcX#d]<~UH!std۫(NX9J}cX|<$v5{X쾣9x>΢];S̚8\' H㡍gTOgïBDRBVPmvqM7wGb;Y_ ~OB)&h,»6qO?_ɶT$@,Px2h_ wN@ą-/^վ߇'_F66D츦i1*0meXPFZ>=-hmhG<-l l}0"0oALE}ɣPwc?դѷO. ~O|d'.jdOh8ƥBVF/#uA{}a ϙ9!2-{Zp/_Bhv|wc)ؽEjY͹F½.w8kr]OwD;W!+sޚ6 kNϕ =ߏn)ۑm;a|}ubV?;ĝ Y}]o ϛ޵3Ŭu$EAkg`{1,l<0N""wz/&XFa57PFfmk+%+=S#$+E"sӸ*J96)cP>jg* 8f˗͵y8`*Hqfzݩ}X8tnOQq܎7Q;|QYo&\喺^fY&$SOl) t'dw)q;BȢMv{l4 ֵ&vI~qʗ*ߓEc,ZW/0 aug+:v?Fm ]ɽ96}HSYiI}> c{)r:Yɛ[wMߴgVGxR0N۞Fg^cZ"ŸWFH\QųF9d0qU$:ؓ~r,P)]o+kT~ӽf 3|8}6~vdyvG_|ٕܶX:~NKӿrE)LlgqKɌ{ϘgtS>-gq5)wq?:{ ͱ r?+g4]ٞS"ilSI]+}^;Wsoynў}VѢ2?՝ۧX"ϗl,gG9cY_rnƾj<c䳮83kSʨy|6e*1UEXH^*{Bn^3`#}ʹm}=q"!+=t:=\#7Kwz Yc[+V Ya#)-4u' p1/Ŝ/)`ɓsb!&T@% [i9_3mO>+ql;qicDƇ#=q8m|~xC(Eԗ5<[ ۷[Y zT0-7 +oJNU/yzYַ/ [EݔږŒl$֘W*m/f|ݍضB_-~ĸK}~jyj,Y`Gni'p%U(K&j_d|U<pQ<r\7E||jeKm>q]z3YVu)$3¶ſ^)֬ՈvQ9D6jU%|oJWI6#'8ܷGf3R~&뾭B_ 8ĸ*ޘY?]g'[85mV3Gik~ m|PiN!>9N4|V%y?m*dMRB)d⑪eAza2}'"xS9SltLvࠉlaD 6 QЙj%QK XO ba&JdIy,>&=~+V4|>BXW9K:|>gd 8tG+΄m\6qC>H'7LCHgXZA@v;me;>`՗mXĦYLQ)F,XQ&D~aI%1 =Mm%譱}r5zJu{H>径ɿ'Xkb_~76yn ;&FK\ށx.{_s}wKNWg}:,ءڪ㌓q,iݍk?iCf=_-}[_;85yJy$R?1^{5-Ο/ZmΧ>4ɧ%uʌܧ?Ʊ\Y+[Om|~Qg]ж;댪߰>ّ˰PRT,W2 KVǖ;bBm|e4|րe{wҾTB{\nt{P^[ٻl?{~!ϡ8lW9ϑ?jLYiwχǿ,,w ev|Q]5S1I[<n igT׌Q U}R[®|W)gWBڬqߥZ>"B4N#dUtHKXU2@jB BVں^`Ʌ%Є&|op\s91%AV!vfo51{Ӹ1'_d[sۡ\ϑwMevM܍t85%qVDyI1nyH>6PGiIώ|Ly}8yi_,HО/!po+Lۖ4?aŒw`ڶ•݇^ WgsxE=R%bW]$s|ὔݟkoPk'Ds3F~Yv~/ߘko<҆<SڑAy~-4N/[xxSI(Hz3PT2c@_L)ƈڤҽQ683宿1#muzr)󑭋4'(G/9Yn7Cw߲=k}S"]]UBϽ,}LC)bm#c.sYjٝk|= _9b =p^~B{w'|{{udW}j\ ~YF1^{}V:>1~Gf%76rpFl+D[ gn H>ؐzbI7҇giCc>/(E>82rZ(_lo9Ӹ{6~8T`A?B֢=5P/V Y볧{A'^3\T@liIn'B5& BnNJd'E$x71y4O~tųUڝl}6r\žkcymvSk<ŦԵ!X#Q\~% R߲='4VvDZX v隽O^ƫm}d蛘ڥKg3]aޙs&yDymuÞQ+.7M;me}neo}*i/;i|>dU.0Y"˝8 ens_ű7sz});Kcoo(3Ov*r^{|,XY*vяhgOO YןM F8,nؖS?Am}eX#;Q`и^ [D */ d_KB 5& $B:@ ZcG1I!Ȓ՘$&VZ9aY}\kM-F3occWs"v]Ř鮏v!iw 'g9.sy͵elWaS֚ZFiُ݆8sݪ-+'R—Ke?pmĻ`'1SC46 o_MICQbϲ{m>"'>k}P=q`hl{\+Fz/{{{S}{.xsh}!=lD m!bT9cOBzpə߶b+>O_ ؎e}>>J׭YgϖX+#.o9zuR</dɃT|öBVvdSIm>4BE4@/`=3e>JhuKzIQf_j_vqg]LHdOY8 U>3ww*AsJ/q$O'ڮղ-6y祇SD^es`WayѶyQc凜,v2նw,?PEQ"E((_,$8_EJID[&ڝۊJٶde˹Wm;Be$f:xY<ڽ2 ZT|lŃ*B+;7OvL9v{_m ڷϘwKnQu#5iC~/]sx^"Ͷ8,cu~S5?6ױdbE3|X๘q.ǡm(ׇ߄gڣ#}yke26՞ݹ~>5jQwѾk%Ϻv>R[Oi^?{漋 ȼB-wy 6CkȲ^+j&{_ʟ,NG->:R=%I*ؒ{Zk{${=s{.'m~ֿ ijYڦs14lo27.='6>˞r|H^بǘ.R99M)}/]3b[fYgZun(YmNBY~F[BO̓)L}\Ngnc>,[֖IX"[gU8^նW Y{HFi<w\Gv'Wt{F'BWۯsq㾱e>q}v,t;xg;=΃?7 Y[#!+m"Iu:q̀3="{0ɟ[Xc>V! k83,๳y5}|'Q٘,l}8CxDjɰ"vOc%im"MhZgEY<HȲb8s"0!@`! Y,Y,Y5nW_{^v3\zש}p̔p0/_q^k)KQ}@p|mݻ\ 1w''gOyvJuٓdSO\cIq9%ۅw.o<|}[(*[??{'={y듧|k'~/IMdo>_'ۢؾ\g8CA?|I׈}_xG*x:J?=x RRφq-6eݻh1MȪ|;|_~>}_rX!+qxsi|qYx d}aE"X8A&A# 0B'& %] >AD'l{uJSYv K%Pϟ v M]|1ע4L*ϽJYǽ$PX,B> F$8}67W/8"(JjIn0EWG1VV䎸W&c(&rfu9잒Jێ#+D̗O-QϿb6QKgOE®(Xܣx-M^ϓ4rTV p݄,گSl&e6}-qe{Hh]K,:jY7yxE"ھBk"m}5М~:6q,kmS?ęG{}D_Qx"ȯ?ksӺoR3lŌk18Zc>ĝ}'P! !X J" <7  AB Y%*P3\{vlHtEHa> ɖH7i4)&pKʤL=ِNs"(&bٜ:tJ<הL6K`W!!2L $$u_cIpCu}ַm*Ŕ*Dkſde#8` G/&=L9I~$plt}2$fm42mD'jNIP_[%J Yt}pOP$d l=!6DC3yQ֣?u.%P^߹YbEG mh>i^my$">HHȒq9:^ 5k>>1qe{\6͇mO<}xvp,49gB_i@zK >v5A&A}Xqt͵|Wrk[oSo|,G$DRK ,"xNRK.rR-OpU׼Xf d$0]?ciHL_\%$LRWkg"ߟNBGB!k=q$<(-wLx}v+[Z1/K6j~M&ne VGr6|w)ǷΦsG/隵ɻiuwxL]u-Fsm/Nsmu"3Ewٮ_a2Vm3Nfc왏x\fπf!W.VO ~5wĎ_LUmvR"ӯm?]eS*K_mV= -fXHq@Y"E'i)6i&,ao;i[hZ$$Sɖ=aWk͂QȚ%²`52%UŶ^MugBViu&lczv9y;㮟 <)miB.ŵ>o%S}>$sV}U'Ԅ蛕O(^1zp#s[_94CP 3[b6֝㮾ohB[k/8}_|4܎]|Vhp۞`(^XРH"`>U'‹#vڝvɧn[/v.H\k5*{ 3%5-p+?!9"Q 9 S ImnE󃟝/rA+K2_-"+_CO8>Lmd U pmHIu$悈Bi _ (ǯ4Ar"}B@:9?7 |xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN,{|P#60C^0_o4?|.GbRs|c}Uv$A^<"'^^'wGnydR'8uHVe\\96^ ʮ/uIKuqO}~ _mgݷ'nЈ:>$ $BOtwUɘg/;Fޥss^yp !4ڮW2qڿsT3 BVZ ,o,]&\+sӜiO3Ͻ³KFB #B`B A8D(6L՟(wa𶭗`+4 WBϢ/Hx)|dpc$iٵU·QqJܥ|-$zCS$d q9M3cی֏u ;$.{y$4^HȆG/4%($@2 /6Y]EyNT)/񧏳L>k|B@SRl =%0oZz Lld1^Lg惮>6.{_Z'm8"ƍӎ'$5og^^˞t0OC-;ӋxH:/^F͇oq<8Y']7Xcշx/N\|fY_3*|I#g2Nݿq6יqtUiqz"_HlR0Rsį^sKSz ݕ3Cwg*airŵ,W)&βWUqVJ^D蕱č}g>m GIq-qchf:"[c"8x>Ix2rV߯zvm>7 '.˄GܳgV"ģV(탇9~P}qUk#~>\X۽>4d4c`jkU8)3NW 9ݶn_(#棣1:g:n,=Y:՞'퍝g왺^\r>seMKA_ D.xo#N1fu۾ oJBUE2a^],|9߉5z:Ħ"?wKka3oLmdchjbV0ᯡFfKmoUk>{B1s[QogάDq'iyQY'6UbCPB,+ ;|݊2AT#,%&%t1h Y?+E.,E:#9[ ݉"0C'\pXTì~݃hџ7-lCi[xЯmWwOK]b,`QeBaėzsō-%D?gږ1܇6v_P/7] ' KgڦRN/ue_+Eݗswe$]6̿Oo{~[{JJ- 㝒)@2&O ”$bHɌ4]SU݌9{x?;"g:v\=M2ǨZ%o\>߳S9<:ъKaX\6qwfu߁;ǭ3kOƖ9ǠC%sٱ@o׫YڟlBs}8w6W~MMHfd=/BߙLh;{|.>ն=j棸W~zk}ԟx~+Ŷd YAD$,H*u'!NS&뻢RI܈ܾ'4IM/dc\ JvwG 5\祐~.|fj#'\p1@^[ Q6m*H FɎ|.hYW{|Eke֏d:ՔV^{uܵ#n/`BaResWZ|] 3ٗm5qx_fXiacY%!s@pSr#[;KqJBz>sdNn),׻HX]?ms\FJƵgg4a¿TKy$gV,(#<2vO>ߥsD\Y彲}67TgmY1OƖZ ;Y̙|vH}.^xͱO#d3[]f <_pk?+rwq"]/칩 cִzbmT׮9(G `Q|{˴ER%mm;g $om"8]3;r]jø{@e 4;8p5u5;ٽ$,?~[:#KW֧ykKۖ1Gu'N1ke5l|nGw-x.rLcp=_=ʊ{-J儃Ӛ0  kƬwdwjs29%>;$:io} [Vczb, 䔞YSvkBսͺ}fqh8.yDWumHޑ4"^5yYb(X ?"VK4ʈ_LcQiѤ-ĠN6v_rw|L_nTܖG5񠄬xLxPsvh;Xm GP\B7By1+dKߟq &d_j|T_1> mkcG Y{Jmsou_F>”l x)}ϷJɾ#MkOu]QC)ݰ_}u(`s"wmoqvgkޢ/gAqQܗF3eQjbuA gimQ$S vieݹ롴6{>X)彤<3g!U\t;tprdˬ%)-J?vz;-aMmgAz-|/&'_]Za Čkmkw>طY,Npc5sNLgs>W[+3(dEĢu@D0qD {}Q< D.͠ڝvɧn[/%!˶:,B5H1dbEKg@4y8q2A0=Y{tx ô=n%űl Lh?^!bmxZ3Lj,^:vc8P{c,/[O|$D/K'*0뒯Kw*&|4_Y`wLewFLf51} BӳqV}i>A2{_dI_ 37OI{>QsisRl=ֵI@̮3f8KHD [ޮ<֬}8 طr16\6w> Pl5s[i\nY4a_Xs9ri]Y:FUASy^suUY^i2bvkL+k>+^~kQ}˱}o.}+̘n_f#'{d clC8WFnawu>{v75s΢=#vgj{'xsq>,v! p(佰 =I5/Y;S~]r{4ў3ENf,N*vٝ{8=1 N=~3$,ܳ3&! .X3O>Pwnv<)d=Unx KsMjyfc~ZWӷ^N6$3'Μ˗*B;>|xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN,{ngm}UT,V*d `xw߾~*|_>Xj_H/~;BBJ Y/>.SzxG| -3N>\A_kG^_]ǖvp|W^4 YgIkGzj7ӿj룍^mcQ7U6aEˆ X8BN&* A'A!,%] Ay2m%J"E(h=^J6?>X* MZ }f౔lmCKre$`%H'dsgƶE*ǀBĬ"9 Yg%`HdaLP*A"XmEHR>{mC2qid:;sB9RaM/BŻmfk}1ul⾌63 Yξ[eqfS;6;ӂ+%T) I_ePJޏ@B L,iţhwW pBUYp`} Rؠ8l(~;.˫7I%%)i5nQ a\F)閾HAJ:r먾 (Dc:XQ|fY_3*|H&dEJs$/jЃjۋTfr1EƒcԖ2~j׶ݱ6".%)Idf\jžt%W$DeŹ|5ޓʵY;_]T,e[P^#E2;8 j=!L1{!Јei,ݟ!95%#R}|Yc_jBֻ}ū~!ɿG!.^H Z[Y!R"2.*sJ\2ewm*RZ\3>Tn3UIDATji\fly?:q.8WTg D!IO)8^gb,1Yv?Ԏ#p4]IOis('Xk'/ Uj Ae'ykܯ]Vs8"&^ 1p7BV(eyc}԰kmC[z,޵J^Ymߊ1%ofmrċ[z|&햻ؘ q>]z5ڶq}3B B@2+nYZ&>O]UnŔ-ʬo;nU+ Y5Ŕb\u\' Y)~1nN!s\ken,KZ0P9/&lr""dR@'v (`Γ0v5~ԅ,*Oū^^&S̥/͍=[X)aM&y?:ھ}.^d6=v[tLUkƤvhi눹s-pO$c2 @Sw/󕶻 L?1:'7"X敗wg1LҀk+Uz 7CG}o=밸^k[)'̺Rk@(s \ؒ0Oإh^w_dأܳ)Yaٳ=Ξqh;QpM B&A~I~Yč U' d$^#jDamn!ʔ$pe/d:)qbjhey|N< 7і%V$<% ^[ Q6m*PI?8!(QbbHC?,jG^+z\1#qcbaݘc\mH\Um ^y۝&؝ڝ -ękf+ PG$JbO%ErtM%ڿ~x<Β|I vGAP\L؋&%]=g9g[Z Y|cQAopc\=ֽ{ufסGI6Mbؠ"+\߼R)/֢|-T=z]BݾQ6r,ؿxq,bT?Y^4#uz`,a[N67_|+Ga1[--.lk۹?8a=L,"BǸU/Hxײt.$9/0u5;ٽLtIZ{n!,{|.c̛{7&ɬ41bp>ϯ܀Ҝ4זCڧm2>6}3Γɔ,7m~ʾY޶OHh>Wo+~^Y 2w~Tᶌ-[1W315zlNk9NX>,u:}=#Pu]WQm[Gx#}~jL1kRDMC5mZ1(f!(b1V"FX_kMQmSU6#N6v_bߵx_6}Xe~۹bn")KZ$4p^̩ e>֟ !9%ܱHI,1R8!~8/#_G{=O٘@+$ڄuf:J-YqN[~{HYo\hoqo)Q6!R2/j6o^>$9U[-$Pke"N/|a4!bɾϗ:r] c5~FG{J{k7@{p8:~-6W/2bk]?:d8KkEkr$|U5b8c.=xl`bQ_:R"OI(fp +jSOFvm;Xj| Y > m,xBl*?EЋv-C%:K ^Kre>X~ZB8m)6 r} ?d+>!bmpߍ$%IXhH}įMpxI :JK7Æ}9|lڮnLf>dD[̥Xz>;|,Wu oJ*dB;&SJDc$O: sm;&o9. qm-[e;~d;c٢J7/uvXSkw _vXF߸\!Q/ʊhYbQm}DǨ_emUß#Wf1{ BG$a3ÌMsg5Rűq6Ro[D,`X%J@Ӌw+u\\ᯫTL! _@`SX#M'Y?#$yY܁$ĸzB,p@g ,p@g ,p@g ,g7ëׯ?]\.>jyf8˗8n\[稾 8Eػ|ѳ|xP_~H\RHvｓ =muڼSO۵Oۓ_x엏$ڦmN_ou7/} Y!IKgz3>T~t{ >B /.Z۠u$J>7!f烟ї~A -\bۦ߽s]$4eNLYZ=$PX,B> F$8}EAKԵo%d}<MbR!Sѧ+dw%(|x33ņ\zC0?9\e7!˼b8[ Dl_gz\ٞ1_}eZEVM'- ^HfGo:4g_}NcM-ZT6Ə|~8q^CTǵ<ܴ|n)#[x1~1Ze ;>c֘ d9q{mI8TȲB& BTōxOhV BѤ$d׷ Yv~4Ԇ/Ys~4bA퍣<}IJdKJ$D4K `q8wL^%eR&lH}9_lNQ?%BlkJn^yt:)D_a`4DAP2?OxXz}vzJ0;9)y:'E9&:mٸqτ׷۶݈1\yK[8Q峺ƚ5k,F/ny@ga-D|QY?`^?6ql3RѶ|Vo~:MEg˗b. #Wqi}\83s;1.6xt ߇%{&Åe%a[Qf` $P/m!k}vk_IȪE.,洄( Y徭ɄKL8z7$ dBId5'cSt'џ7-li[%8mWwO*PRȢ볨&!Kd+,n)YO$Vg؝Įv)b-:bnDOI;Znw|%iEDGrGB =A bY/+?i>}幫geu뼍"3\mC@Ol#‡µ?kѡ"B ڨ:I KLJKh!d$pFȒoz6ؘqcHb\~>QdHԒFky"E&iJV?^s8čVBvJ%R=I2FɎd;ÿ?+?g'!No)le_IɺgF?"mj0_?!ՋŤq]?^߹ʖV Јv>vN Y'하Bw93!kA b's[km;9nǶ)ׯ8/o`ú/֦%2\S8#-g|4CL݇g!+V`B#n)q& _,J-- &mm;g⚠*씡>~^ٞ$o%;HEڈHx7%RdIxEN NܘV?^baU?}_]@?BID RJ_yXIZ(d33'Eq⮟ neK+pfCOĭlw HΆ>=tBւ(0]6y"ϑhN~iMѻ[dq."Lfʸۢ8cfl=P!,dE, VY"lETն~1UisێQ"ӯm?aESvMKZ_d|ü#*")%{PHIgJ$M G"KjD;[E*&&8TeOXll`q/=,nɯ?5$%dmz|I#d-dglSmݙU{ @䘞:]Nθg;OJj[1/KqmO&[~8m{%U_d~I?5!+feǯ6t>JGrL(=+\hNy,7P6s1B̖u縫ڦ֚5Nj,@zy5A=3c(3ڶ>ܶXF!+%2:R "OI(f ݙپ m|궵TȲmT|!v}zչ΄%}7] #S /Wx Z(%lDַ8mdk$Խ庻b,L%Q!+D%kp YRoC婎B,DY`l߱~w͓nflSߺW/obmyR~[35_ݶ Yd>?,ҼΣK[7X#5H 1Yۺ6lvvFz*1ܶ;0lu߷Íg9M4N>gD# ҸuɈ>43ƚ1XnG"᩾l pmHIu$Bi _ (ǯ4Ar"}B@:9?7 嗟`/,ʎ$0e,qI:zx~xGN)ySd]&0=9aξGWC'a:8;?z7 ũSѧO~cU\Ai!qK!y6f]3_h)LiM,;ŋ>xhb8_9 |n!A'a,1GsgWe&tz?Tez/4a{Q"B,I#xņIdPp.L dT WH`),j;bx?OG;,Cˮ>ضw\ }] YA,`6VH;g$]&-ApJV~y!{B?{flQQaaa/BpA/a?QSy9P`(!>2TaLlt qOI NOg}5n`b#$dD":vgD'_8oL2ÿ{lE=edsӽ9`xlY&6u{ŐO(; {vstޟՈR۲ϮO AJXD(Ij_T1ZکX60RQMv0btŕkkytI͛SzyfCNCB-y [G#ūTq|yO/qN*gn\:%Xj[=Eci_+BַLb!_`U=i$:LƩ79:1*m}?Ny_ZK ^8 I%~ڜ7'|]~a{c{BC 8^ BM䴐ۋkY8[SLe3$#4爵,)chAߙO[m7{Rc\K|s|8&Ȗ-H5' {I;6O-~s8,rϺߊwYY4EFt,Hssd]=ݯgvCkׇ׽ޗ:~ ,]m 33]{ Ҷϙ+s[sL år3nM-D\{^7gs(_{V;KcVušBB7 ,*nx@c߮]tY. YmB ==BV\{HL_5;ּ8C#{HNtHPa{P/~O6S|(Xȗpȝc^VR};#[⹃5._pTL E8 6'^O*accvgF@">?hݛIsг=?#iXH_OeOƖZz ]kpvӼ3*k9y 4s̐ϝmMcnl-]; 4|gޞЉ}]ئld[~W|ٍ;?3SiWg>xL۸G]Uߟ!GqKԶB _fŲ׭0 P˄Y KJ bJַm*SJ׶?u)Ew]XJqv^p:(%{spM蝗~D|}A'v kVx;姥.d`_WyB?(^M2!0 -(&̋/[4v.dyse%k[pؑzBnKɒ!V,_ʾWƻ;}JN.㧶=_=Ns ZA(%30̔OI4]SU݌9y/'{8k3;.ޞ&lcTas7\lwt}dwdi2Km۾e)cJ1PDޝ}ά/Z?[fk݇>zbk .߮W?لPZۗő?pTc-?4m4[^;>U>>f<69_u[sZs=\9{Ҍg[B 1@u5gWW{>ɲx3ϰy[j{le7!#Pk֢eU@B"T$%vgBw +m Uj\Ȳ!KR1.m/l8141=}j!{㋳Af/jnþg/Lc!‹5]qw>{ά JE'clLyw~yY,L>;$ɾ\Hgo/nllh]ӚxMڗ㹳ir9kޑZYKP3a_\<0>ީ(8e.BV&GRL:-JP|ܟkLye{x#5߇0~AW-?˃ A7q^5;ٽ$>p۝Տw Ygd)4^ymi2pƎ,9c/J/&Zki+T>6[7)X.fN刹[FC@뷵XBYqbEi,i:J{IJp81]&{.jA.:$Lc\K|vHu:$>+GI`hv(mˋ&5뾃S YOƶY_8DWumHX/dY?kq X?Slu3TgOv t胅vDOk-h}LqPߋ#⢂ze&/kr?+]g>؋BVQbpa%†AZkj[ڴmG(}YWƶ}&)Y7{ umPԽ{R@!;4CbEڪʽ΋^!+XcX&6ٗGR$hp(ᮍ)FϞ aNݿE׈gQ2s1/^x/Gh/yb8|S4x]g|~ mb7Wyv]%$;9껶ڎиG5IїI38(K#ޙC2ǨB\u18IV>*I-קiKk4X0 ^SDޝ~JkO3k{9oƵim6ֽ~b9^u&vq6:Bx1Sk3zK}[ۚu=߸^e+bCT٣J'a/>N?Wy`  t)Aܧ$Q^_@Vxwhl߆K>uZ*dٶsE Ys׫u&d龽P/yzYШ:IG{p-8K^,c)~x&̔Kq%'\?8 YDlǓ^4|pȲ^*sÁо?se/)䑈c%brDe&/{ץ;FrXﯬyA;2s_P˭ӧ <Rv8!c8J8~"v'jN5|N*}7GҺP1Eq}g )RCa˼۵ǚg!f,^uމB,s3O"|_=޺Orˊg7͵šs̑{Nʺǂ$]0Y3"۫v \d<8K/W6|&W գ6c\s]|>2׉uψ9|>t8ȟmmrW/oslTL>5zvſ!/s{bLf?Of# ]`E`?Z/wLz :WS~y׉^7s/k;½qv'޼iqNOL.KXσXE*B֙xxP@gB "'̓;)*vqv7iqN9&ou|"M̓J|ś|VǽNBV7s?lg'卍3"D+~ WB8K dB8K dB8K dBwͳ럮^o׿ ujy5<31W?ͫvkl'ZowDsgWYv!띋?E>|y<|(/?yb.Vl$oOOIe:m)ߧڧɯ_p|yOSȓDYTp[v=<~cU^rcD{p|W^4 Y>[;,PXi-5`ז-O/"O?rXкgBVW qbvbE,X8F&* @'Q! %] Atz2m%J"Eгh=~J6?> -\bۦ߽s]$4v)dt3ZAA%2vo S$d q9M3cی֏uc@D!ynbr`3sr0$ eSЖSRDH2k{?Itv~Z:Gx:&.93)/ǺUiDk/GvK[0W˹wBVW H#>ol41?xhO AJKj_T1~Avvm&"LovCyS(b,d|8m]-a_eM7Uyٯ{C&s)^^ 7/O—(,9LIe^s+X6JIDB&RLIlmmĹ$\Q{k+PiXx]+%/:kҽv9I\$ M\iO]?ˉ"@OLw?_4ݫ"nʺOx׸_gqDLxBRZS= Yv=zxBVZ˳=֙l|4+k1[:ͬM#Wx1~}WτrSp/ocvٕ!'0Y$u+QD ` Ŕ-do (Qb_Մ2}ms]GъR~;#8+x (/Q&ϲ|Q \nB&'"~O&Dl< mWwOK]b,`QeBQ?\3U6Tmq\ۗHkS/ܣkEZUнfLnni?K1ײm>['Dr0%cb:$EQ&ҩ~)d1SP^cwD%5HϿƚwg1LҀk+Uz 7CG}o=밸^k[ YA8r*ZUc P{vrg6*wWz1~&u|brϦ_ng1mg϶:?g1 YiE9Thrླ‡µ?k* u'6NS&뻢AuѶRC*I.dYj%)Wsm1kX۟㢗x;N&D[Zc󤛗(P{m%DlWB%aDldGND~" !br/yqWƍm7STywc"q!qUm{ɟselw`wjwF&|VgCnBN+9z ?L"اDb=-y,I D}$8]~agз&$Pճ~Sxu %ϗ*:DV 7Ci g]gvydc߄ZJЇkÔOo CY? \!K%Ihmcӷ?Gټu`nűQgwx&O{cXԵmslNF g5 x}" .BV&GRL6z-JP|ܟkLye{x#5߫Efw-{IWIBR̾{BISW]JcǚKDHd;m!ϫ?Ȓ'i8Ƽwc"q I3*#&'( >+Ism=$}ڦ/#ya>s.0v):H\rw4~^Y 2w~Tᶌ-[1W315&$uL1q_$ǚV[_59 v4BV}dXm=)ưwMU|Va]y`.;X ?Kh֚;/6mn2J_duյ ,y_ {lJ_5|%v'"џ tIYJ'E<DŽ[bNmx/dw-aEHjf:d1tY\C?,}!xijKk.1mo1bYܳ^')YIe߀/D\m߸e^=͹M6v rmo wb}kvƲEG-~8n<_hW;A9יl߱>q繂G Ym,| W^Y1{lbJk;;).dp퟇"o<>嗟7!f烟ї~A &!bY0"$B,$k^XlZ[JJAˊd֯Ц t}u=<ɗ/(vp~? !!!yh]! i 0(5# 8:2QEcl3P}O 3So3mae_~K~`,dPr-_$Jm8Feo{?Bb&~'B?EI8-~(2 Y 8dldTvcuTo9GSRI1vq"ydi%>^&1jSE{reybC=!yJݲe^uJ1ل]ƿWb3=l K>2|_G-"+&WHDW|M$ͣp3/>[1&Ζxm*G>?8h!ZOGcmnZMj>7q攑-sq-Y2{|g|^kSԜ`=$*dYR8a/z9:+įƾ]Ai!dmMJB|}U _87 Y% عgjg~/B7 ɖH7i4)&pKʤL=ِNs"(&bٜ:tJ<הL6K`W!!8LC$u+n,D0 %At VI:_L)B^必E.[]INdnQr$ Y1F5 m}q6{|,G$p(^M{5r*YxIT7]dI'v (idV"N*i򓠾J",ji**I" !{ Bm8*vgG6]J&sp,}>n'~IDZ-d}ܑ%mOsuk&mg}} VC:g&d-:^ uc} |Mm V_9XlX5TDf~c[gl|ڜ1y"d!lbzf{˴ER%mm;g`0#>~^ٞk{|)y`&T@"&"^Mes*^rjy7浕p5X$DU gWW$OEb*D')deW^c?k,t? YC'L DQoAg;[Ҋx1\P6m2q+]B=k}wO8u6`>x1L׬M޵Hcsdk5S_o{i?%vk@mY)¼vuٴ2n{ƶ(Θv7c|46{}x6 YE"ly#vlbj{Y~umlŲ0x}o);^9xEöI=n(K$ 3%_b&܄#%5C"l-p"m mZ dd*ٲ',6}6_bY08 YPWXLbFƶv3 L*ͽ݄M rLOs.'og3'%-^å'djadsΪ/x㤟}}Wz[:%K9\O.v}qοs{k4?,ҼΣK[7X#5H 1Yۺ6lvvFz*1ܶ;0lu߷Íg9M4N>gD# ҸuɈ>43ƚ1XnGpi%Hd"h;\[&pRIg,p& bpuZ+sf܀p`%NAY,Y,Y,Y5nW_{^v3\zש}p̔p\O)vG6W?/_W~~;\yegƑq&n;!{멸,">ѭgq8p4rѩP11=r1Gr瘣H9 1<.ds1\~ȇ:E'T_e=*m7x;sv^[<]=~HO_m&Do}'v=4}X8Gs^ 1;| oH_ _]|_~>`/,ʎ$0e,qI:zx~xGN)ySd]&099NcS 1_^3xy_:e}tG<#ajю~!вkIZ$z]?Rbٹ Y{%2)}o S$d q9M3cی֏u ;$.{䗒 zi$ȆGBv^@ʩ͑[w߆$ e_Ogg*0&>qOI¿KyMk_o I}Hg|8\S}'7z&<ۚo1S<9uhv%PjUC>NAcާ5q s1{a;⹴9hp v*#$F_PDݏ@b%NĒYm GIq-qchf:"[c"8(4}F/'YE϶Ȯi2'_S YtX!94!C;vBk=ِN4cYK#_6!wkzYIr|*jBֻ}ū~!ɿxQNsbC۾Їزqh_dhs;Ǒʖ -; kmx*iCu>ϣ- ' <1n=./:\w/Z{ ,gM3]oMuMڦlT}{T%$'CR>E>;$7qsLgk|1w5zK&;lk{G|~к7sgo{8~(F<0Ǒ<-kKұa^=͋9ʿi0Os{v|Vt߻||L~={[~)]K/,+],0 %Ix׭1 PD! $P/m!k}QFo D) :u':{FEɑ$d /H>SBةxxQ;@+ ykA'_Dlmu\iS~ZB Vu'dth/#b¼EjB?_Ve ᮍ'ۻ~!+o;j s]ٗ"qx7E5Ī;}JN&In@SZInQMBK/s9vR?9'{vTmU7cN҄$Rw1E>;$tx{eQ1K`sYHqSG<0t-5'8^"|w[gJ-kC%ر@o׫YڟlBs}8w6W~MMHfd3uiݭދ?|U|HmϜ=iƳF!ߞrگ*6޳~dwiW3|opZvX0UfD %*$R|W5FT?&뻢#1& G|k Y;E+\$%1á8bZ~8M& l.Jn3VDIQ~36iYW{|Eke֏buͫ)QkcGˏ]';^* {/Q6w_e 2}V/w#Rm#Xc'"όub:rSr#[;Kέ7.0'NȜ]$.쟍U݌9 .c#%[Zj x{eQ_*XH*=1NѶ6{5/3/sgm~s{1#յҏh2Hϰ/Jdz+GpĊK x|+T೭mL\d`ԇ+ۓxm,Oa˪sL\[BY֍70HCGEtqWyfXW8BVQbHe~%eDcj[ڴ!EV_]~fZ_SŲ%/^7_y"Yg4mҏ8wv^6^j!8{(8dpȞ~^,ܱHB[{AB7By1+dKߟq &__Np6ܵ#d_m lehCSoQ.¬5"Yf{L K},̓Gh/qh e.(!~dEد>{K{Vߵ}v=ڝaNL>>1Gq_–9F⪵qV>=YX &mEޝ~JkO3k{9/Jڴ6R~uaמ oǺvO.%幘?:Wܧۡ{n#}TF_fU.ILh,W!Ӌx^ ϋ>zWיg3)٘ L}-~ q.mmk|{їZC|fʽ쫞g|{ Ǔ3{%(dEĢ ]G $A:I԰ƱKlKD3vɧn[/v.lۦG|L:gT\QtuM}aʇ2:4~oAq,%=Ϥr} ?d+>!bmxKfY;p cyzbw|1Qc%brDe&/{ץ;FrXﯬyA;2޻EϙN1} ‹!b_p8#dgRI^go2H{>QsisRl=ֵ@3f8KHD [ޮ<֬}8DI@&*>Ss1bvٺgRgSx$4mu; }[V?#Mi@֜csmW}" s |{{V|fm˛&-}Ml٧3M}Ik½ő i`_rvyd/3qu8.[x=gҿ6s.ܫ8ہ{gw͛g겄<["d|7 <_,p& bpR<@ ܹI^L!AplgwÛg_ h Y=M!! d}+N0;| oH_ _]|_~>`/,$ŗK!k!%\U,}}zQ/Epu>`.kޏ~x$?Yu?XpMsݟ9OeDzp~'kvrÓO?aA YnRzۭgB B`H#A|ī((L՟(Twd𶭗`+4 BϢ/H*|dp_2Zs^ $]?RƱX8xIL[ Y?B\6_E"QHv7X=g! D IC,DKTm'W&GԶ[6Y YHymd991> Bta׏o u')4NȪp +Q8b6<xhO A UF( Ij{1ZکX6.RAP%?OPUx>6.>EԶ#Xi΄,J7KyjMyD daJZ/[u?B$QJ/'2qfN*gn\:7#)(N+VTo?ĪBW̿ _; Yѿ'˨Z,b!8cl$!+eb ڵmwH5 kDg_پ']v:5 ~gq._qrm7}c!*K7M7 +.-* QUy'gWk Yfc:{۞5'~YqX_?S٫7񞴿zjc DM-(fG\ [ ۩YˆhZF| Zkwf^I-bcĬjgG/l۶Nilei X!~ ơB+7 ,QAWOkN!kue!kMȪIԆ/nJvsiنlLw*dŗM/Δ/޵<ۄ;& l_Ɇwj_cB#:tJx<הHe} YE,'7x#%hmgXHuȸyJ+qxnSLܵH jqPIϨB{&q>mmĹ$\Q{k+Pb%ec[YEf+SyjY|ۖkڷλ~^Ymߊ1%ofmrċ[z|&햻,a8kmLOZ!" &$ TVv7f>PD `Ŕ-do"L[(1Qb_E.D6IY",%!+H|t(? b-P^MeI+*9%܄MND|L Dna%LyFۮł] Y"x5+˄`4ѱgb =%ld4G'B۷/ Ց tצ _G#nɵ{͘T6-y1be|+d 'SP'Y DMm& JޞbJN8)I8\$)|g9"cgVݫnzaqN׶::zCº-l⺛65Vێ5 i^w_dأܳ)YaٳٲLBVg|xvMbڟi2#+u'6FT?&뻢%~4R۝䢌YquјSNAigQ} Y/V<ѝxj'n-KZHyKlیU+0~"pBQ'"EF~1|uYԎVfbG+ƶթB¼1Ǹې@92;M;;#| Z>+͉3TW!7AXH^g=xI(BϔH&BR:KRzmgdOӅm- R:}gWYNٖց|6H<_XTv[1Wu/vu(Q}a t\qWǾ'BĵfӾ9e*"FeӟU<K;RaQ׮ͱ9qcc2֦8k=X .BV+/}O~oT೭mL\,I@}X= SAWyA۟d ]^U/➿sRU׼Xf2'jN[ϻs?$nIZ1oݘc\'BҌʷLjI !<:ksJs\[r hiH^GD8\ԕ kU7%Km;L)kCJB^&19ɩZ/8 }KnRlÍqQ[:Y,Oc<$nB\K /t=Cij4ߢ8lV[e1l]Se}`?p>ͱh{ų f!(b1VO2WЦg=s)ZgϢŽDUh 9%xj/?s܎_Np6)nLD_F ϓ}/uSpk\\S5y]6?9E DS=wo#ui.UO4Xfb]ӵc[ ti-}hMb)GXI.(X@H$`>U'AC;3۷E,)m+$dٶsXe6g>RLcElzѮ%cb0DgI;7Ӌ}I̧k/QKHR-0%FA[/A\RǕs\X'dA̲8) -v\o c6/)A'P<~ 6z}YB/ǜm^ەuٍi̇l}_g➥XdB&cB0D`L4N=_o[Kd"&oYhHrr#R\Ƹ[G3-?jqR'o׍5F{}ϸޙ"ɼumga;l G:Pqbܮ=j_ IY;uҞǨ?}u߽^Ř)& PƂ}Ϙz 36~.׬r Gę/ڮ=xxd /0"C$C8WE"F譮-,p& b1@ab<F:"y4BEId!B8w dp? 1;! %Y! %Y! %Y! OW~y/׷nK:ϼnr+k|-W?~=zy9o{w678_)>W^Y1{lbJk;;).dp퟇"o<>嗟G_^߹')'NJ7Hy}G{B6 Y$1\0Đ(6L ՟(wdƈRH`q)$-j;bx?OG; | k)>;¢R-ǩ'wz(xB֚ ERhge겐$q{Ib!k!n)!+-+Q[^OB%ڦ۶.QzB߇'_PȢڽ/~7;䡝v,kvW3J2NMHB+ T#3So3me_~sG7ag>}vYa~3Bⰻ= ٛEȫWD%?{$Hjī#+d?+XHprG+IuR9u3:Q}ǿQvOI%Qmu˧I|kJxx1^ĨBֳOaW ,JQ<&gg \a*+ ~rwnBy)pfv2_پ=c$.%uh|hl\8[㵶lp̣>/sky<Wi7ęSFbbƵdYw}{1LARs΃>pe MKAebEط+(9BHz.. BmBV&&|23%˨>&[Y@/7}=i%}'&A(A-)d9n,%hRM1y՗ I{!DPL|Ų9)Fu Iy<){yh5E,Ab\T>B?HJb!11)Ȓ2>fR긛^ej]?^nv#[tp/?oO8D pk ??UM6ZGe-zÎaX/- HFYE쫶67a{/_ފ,̏_ťgpgǸ|$m&|ۗB>&XVv2qe,$3p BGShWo l>p>Qe\D()̛R_i<C$p(^M{5r*YxIT7]dI'v (idV"N*i򓠾J",ji**I" !{ Bm8*vgG6]J&sp,}>n'~IDZ-d}ܑ%mOsuk&mg}}Sqh-'^t'DR4*\)2I$Ui&nlیU+I6Jv d $,]<Y9?+< q~Ka+JJ֥?S5BĴYneV <^l/&]VbF l!EesoN=lĬΙ Y b=qݘke_Sl1q;hN~|y}3} x6//٤沟!m?6hyb>p<YA8pGJ_2mTo u!k[&X2A߸w"'*Yc`pI/&DL9I/I<ÉSW]JcǚK,Lq^ttdz+Y"1I~qH2AJ^+5BT: YY tơ|& N3ۭliB.vq6.\[5Gy>\@:QZ0Ek&Z1yU92uյ)ׯ^;͵i6z},ae~]lZ=c[gl;ݛg>q}=>pS ImnE󃟝/rA+K2_-"+_CO8>Lmd ,Xm p*Z :xזo(TG! .X dwJc/7 bG dX ssCB8K dB8K dB8K dBwͳ럮^o׿ ujy5<3E)&pSݑmrp{Wg0;!UW^ٙqdu[xqz*z? (btY<1<ΛY<1ߏ3*v7% g}pR Y\ y,CQ~#Usg/Je#y ":N*{4\i'O>m>mO~_>k[nSSW&N\qٷOey3)8TȺnǶO߼;| oH_ _]|_~>`/,ʎļGN;ҋq/,Ó;|#<R'8uxK! 3_x96^ r.{wPV s>xhac۳Rb;8k]N %DV}B|ŭg$uƳS6 Y$a#)dC0 T .nSL }ewEFo9N}X>עox7 8%+AB=!=3(jWQ(N5b^X8pA/a?KHy9$B^@F?HIo]yz,@$j[d3ׂim70*"S;|U _8oL2/%t๗횹O撽<9`xlY&_ñ\Űi+,~,'_xxBV|N䫳<o;?о$%+t{&`$!R!~mکX60R#˛B >3>g0eLWy7o](m'$5o?YU2f~X[b١ 'OlϡOm|ʗ"!rfu_[1vq6IU|fY_3*|I#g2Nݿq6יqtUiqz x؏/ YrbR3֡ 6 9!2 ݕ3COw#zq^yl'Eu#^Z=YJ1<*L2I6Y1"&iL9ZwV ~g:f&ߨ3!e9&R3Ig|V'^s9TȲb& VyĄZx^c߮d~x]V]ۄ,;?V)s+c*=S3Yҵ)0CY:Vww|Ճk7%n+PX}wj!+;0/ဧ^ÿ{^F|^u龄9WJqdwwb;7;9nY;(_~2ֺ7}K1B)&l|O{UBw77]5ܱcj2QgY g{SLM3{Mԣq6~R3?BٿkWTض1:s׸p%ƪjQo۵’Z٠'{g(í2bo{\BlL8"5#Z_Vq.EՑˆgcd}WT*I @jY\!Kbcq) Y4q\NFwu :7}uq`!9;8{8{m%DlW"$(Qlf~m/."ڑʬW0͵|i1C{ᥥm:p^"$w* {/Q6w_ne 2}.}i=f[/1%;!3^m{ļg?]+=݇=vY] "8#V_E/ZH~oTmL\d`G|J>}&uJȆl^6 y10χ%5;ٽsZ{n!,_Y{ז-cwm숟#cHH4\k-m%41gssP8zkER=0/r%#֞b e=K |G\<7fS)z.jŅL4ƵgY>Oa˪sLQR #'L̰=M_qušͺ}fDZddl[u)Ű]Gvi[vqr6'q7:FԷ^>φ6uJY3L_OTu>3C3kоvH{B.~bؖUlUZm虣U=ۏ³˶"r_`3X ?"VK4ʈ_L=cQiѤ-s&W`ob;2a]̻`<\f&Pc! ߵ w,Ҷ0XxC:/za>rmھxpJ}E#G wmDxYyə(b} {=&`>_ '\Vyt'&ʎ37=El(׏uq$~د>{K{NV㭶#4l sL>>1Gq_–9F⪵qV>*Id%ބ ޖ^{wnz(^x>m,uYCK1}4MkWָ0i'dWmf5bn4]yS;a?(J9݈>3}W?nGTY9yJk.='^{'\_%oYҞL=w^78z ؆bll fB̈́Jg}XY1fJgwN]"s6 YQ0(AבbD}N5D1+`~fPmhSBmۊ:}>g*Do~ݷ̏SZ., Lܓ7NE:f퀚_"[PKɶ>6r} ?d+>!bmxX3Lj,/r;g1쾰l&)#/KR:L>/ 9_y,Wּ ؝S]^R\ӧn ⌐VSJ%bx}iyd5'ٚN>'v#i]{IH}\m1YBDԦ&P2v-f+řULN|?Oڗe H1]0GD<Źs4.F|\{/94ڮz XcIRZ #wH;bϫbܧhfֈLעo췶eI;gq<Ԗ}Nsh]=~O~3$ˉ؇of,C8nBm&Ǿe>k{lCLVUc$KޞۇxS`X:WSd/E/'67'RzAhϙǢxoz)>U=;M3pzbt zb%sǙ3QgH@gB "'̓otȝ-f Y{g;8޴8'd YG;; bo2gԥD!gkܛ1$dq#}83vqvR8 &cEd8N @pB dB8K dB8K dB8K dp<^~=/vxp]W3SsӚؼnǘvb&pG==p\|5U8l޹.YmÇG/Ϟ_ oFDtxThxn;O}ڮ}ڞc|$/ֶܦ4&%Mعno of&-Spud_o4?|.GbRs|c}Uv )^^uY )'?g鳿 _wȗ4_?<}CѧgO>mWxO{⃟CY6n+;?_!+/yxlʙ'%dْQ~~uτ.r=<q2QvbEˆX8BAN&1$ @'#,%] BA1⇄m+Ҹ.X\i Iڎ>#jю~!`6]vGXTjԙ k*|-ˤ۽%NIJ//uO7qόm3Z?U9)d繉YyEsJ4B-268ќl}oD;HrY YHZ/ʄ,3'˱nE ]%ڒ4f60q.a*:i*4#$_P"Dޏ@{>";C;6Kf&]U7xdyS(bgl!F/ Pv\ʫWo˓%J KSj|܊)&R-}90uR9u3:Q}ǿQvO@QmuZ"d}$V-gVUڑLȊ>8I6_Fbն cd%YY-KeԮmcmNE1\X%J>S͸\=JlI>s2k'Օkhw6 Q?'>% \'RЖ)d]dgWk^Mڳ~Z,31BE=mz,pϸ t{կٟ˛xO_d=1ݦw!)ZrDza(Xy_V,k}Hm #&f57V;<{qf۶uJHg/}1\WӞlVR} YV$*O_ UOkӏ˂|}e'* zeL%gJvQ}F;KqVv7BV|َ/LI ]^˺Mxc;OL/E>lH}?&4bYKCGHp1}MɈT+_ؗo_D_Ȣ{:|qQȹ8>R֖qVT DyW|6d]׌t/kr6[ަNK99w9Q8qN,Sr1$:\^HR:oJ,v'ϫ4N2IJkb+b tA3ݽ*Rָ:7/|0{G@Ąދ'd٘uhMdޮUEX}>4=7 l[:Z nelVĿ|-x3kHU'^ }z3oܥ#+t?Rm8F|IpUȲ$&Y$q"0 PgJW:_L)B\D.u_M8 EIt T1|ߝZȊ/"%Y/!as^MHDDɤOԛQ2'ajNi Y,XU%,"WӽL; K_{&j0S& M6N~t+}r]p[@wm r{=r\ ׌I>ms)Zg  D`$ />" mM+5~߿~/2)~<_P]mEܽ*PƸzZytm+! iE(XVqmZZ[!fu?]^[^RWz1~&u|brϦ_ng1mg϶:?g1)g||vMbT-/R|HaD31+*č$Lx_ 5I.Yj%11ɇ޸, ]f8r.X#zo,/݉v20&ڲՊ۞'ݼDk+!fX* ''d%; p"R[LliYW{|EkeV+&p2nlYZ,/̻y j Kc᱋DG pĊPK-JPܟkLHOg ԠoNQBِFaq9}@»$t!)f_=礀ym%ܱc%d"N۝Տw YdIؓscܻ1ǸNfoCy~u\椹>mӌq{_s$L;q/9MKG(Z&(ZXIyyg9N,xR2opc\=tֳ8zvO}ԑ{LZíquP\3rMunw\J8[mIz{J5= Y/gmWRWZ_WZs?߫WcuWLV5Ǯo2ӥX٢59bޛ:xv(dEĢ]GA:I԰Ŭ^b]"A;3۷Oݶ Ym+v[DYu 2?N!hыv-C%:K ^Kre>X~ZB8m)6 r} ?d+>!bmpߍ$%IXhH}įMpxI :JK7Æ}9|lڮnLf>dD[̥Xz>;|,Wu$cMdW Be˦{ 46y;%5q Br 2'Մ+dٶ{;GSZ\t%Ѻ?Grb̍^YH?({cþgL=cZ?fbke3+q6Y3uB,vS~5o 1'&"W_W-7"Doum d3!}U B8 K6IFx͛>O.4<(FH& 9! IqY,Y,Y,Y5nW_{^v3\zש}p̔p0/_q^k)KQ}@p|mݻ\ 1w''o]8|3%Fچl:0>Px脓,G`%.7 rHߙTuLUuyfgW~{{zY)w>\^Y1[|bJk;;)·dp߇:Ǣ{/粧ΆwJe#>? iy6|~mc{/#i|Um;/Ҙe*ztvBɗ6_'y\M5<3 _Gït_' 6BOgq_}~}˺7b%oUѫ>/߉{>/?r{_!+xsqby+]@J! !Xp$ nLbH&OBBJ߻@ 2`xKrEtMAf볳! >  G?B b1|4&5Q:f#I}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB>&1̊e%a'(Zܘ(_ qW#䴅mWDi B0nb̝Sovٷ{#8` G/&=L9I~$plt}2$fo42UJw(%,>ZZߧ okHvvD"nuơݙ<(um ,Fk1ܢ#>KtC'y$",dqGrCB =A bY/Wooi>}幫eu뼍0*NEf>ۊ8aׁ$ џl :=,lfBV@0k˪  ~:I KLvg1FwE%9AKhY%۝䢌 BsPG#fB*{3ӝLIZҨp-O$TkNl%DDj'I&m(@>vhIX:{4_Y8E}%%SC,o_dS 0NQZ0Ek'Z1yU92uյ)ׯ^N NsmFoř"̻Ȩ㶧o⌙}{>G?fπ؟BVQbHe~%eįOm2/:Ia) P>k A_Zƭ;f}1TI/&DN=S%&)mM8YR3$¦Y/ҷ`Z$$Sɗ-a+5f(d_@{Xܒ_a?5$%dt3 )[7&d^n&9wLxmIIK#p)uV2^hsI9/x㤟@<&θ(^1zp#s[_94lo}mx}Q-1sqW7O!kY{b샾{>gc(3 S,cFבC}N5?=hٿ5bIaL][|\JBm1mI9&tڮC0[HşJxzċMՒF)e%ž| $[#-fa*J,Y!*_CVJO_ؖ IBVlK:# Yr8e9Nbfcd<'3+1ܠ/Mߦu_\m٘u\"lWΗ`w5v^¸eqo5R~`vsv7S[ Yz*3b\;0luÕf9OFc:ΈF6VKH%#ڐS(& R67 _dNVN&xkDN# pۀNUD,nKAwJ9x;H_n@@p  b ,p@' ,p@' ,p@' ,n'W7o7//^.7] uϼr xbwd9~{3\_zes  {ÅWvbgrݾ~qֹ1BmB,ʨ]{w8s"71wmmfϷq'l;s >W8*녬Ά><<ֹ?<S~9=}v6S*m7TvoxLyG|kӸ~y/I?.ַܧԧ2mۉ~~e:0+n]z}_o3~>|sM}|1d_y)%&6x!^X.?9|=Ӱ]]bXÄ?n Y}&h`|">s qcMGz6h/BB4`1AAܘĐ(L՟(,dAȨ{u"]SYd;lxHOi<я%6t%>7IYT#tn+k iF4x 8%+ABO<=!?=3f~H n/,y5C㥁|nC$n؏B"goQȪBxI^*|mh\[4 דa΃Ƹ>;/v,TY6~Q 8#dl1<=wOj>3{ݖω$1[cګ=޴dv}гb6ӂ+%T1E8_` Jގ@{>"?,Ec Z~(/` Y>엙 c;L]2(ΈYU%_\/c.:K,;a~|QQ:X+_Ӌٯ/j;| l+c:n"d}$V-wVUڒF#dg;\g?E6q x؏/ @LuhBپ>!;T~a{{BihĂyA% Mq! YqK뷾8_)gs$e6K9Zv6- vג18t Lc8&ȗ5H5jLZcF[_cʳͱ͇ߨ;խ})rϺ_wYY.Z?fN,2Ҋagy̑9V1e?44'oLKzkaOHxHq 賹׆x&-`3X8vg ߲϶zFQ9k~{s9q6,6Dm.Gjа]͇nۋAsǰ}XTY;ȮBF7 ,KBZJojC\ѥ_m:!+yg{H2d0!ˊqqlZ Wt`,ҁ;tx ͇{!8 cYKCC}3mMT+wG&d}0Q !ߍKl$u 9o%2bơʄmGz*s_X6u߻jƫ"w(TӴ|G[S/lBnjo(~1;o=Oqӳ&5k&9A!ݶ>z.~Rz.qxޘ//8Dq&"@kBvn4wj3sgb$ƹ*p{FUoqJucs"Ao Uqjŵf%k湞U>45~i|ѬѶ+&;pk~{X_aS! _fŲE-.2aJO\!NHh Yۖo $߱cU~MkE$dJqq#t`CYvpS/=y/c#~OFE{ƶJaSł] Y"x5+eBn+gq#c+\Y&22pƆK`abo=4UŗX s]Z=%x0% "sͼ(ސSrI,ANbGSbT)nQo\dƜ$ƔO 4lsNmU7cN^4ON4hv3;.ޞ&|cTasxlhY1wq^ęn8$71)X̝}TV["bݹ`wqA{NvkݛLkb{G7sxm}˴Ge"^h^g1M35Q5QlҳR9fj9&_A\xwFVJ8sÕEݶ[%AuA/jOP^߇[{e~ N{&qb3!+ ZUeU@B"T$%&h!z^/BVv'e {);9bC8Cws١?Cyg+!fM_x%R{=/OFɏl7& !_dyj泹/2ty^Zm{kcC8q&H/Q[mVmAGܕg|1LsuB#D5v$) -qSr#[7%brbdN@GJ&.#}mφmU7cNذIRَY5=M2Ǩ_*Xrw>{ά J=ֺOFfVl?3uOjُkw)|7sI=ǡϙٿR:̭GԯSy~Lc\^o'9חCi].7y#_}gϮNzbXO_=Tl8z;b>ضwds>ُ~WdWŽYArʿBC 2mTo u!kmL\d`FJcVLwۋ?޼>м.0bpa?FC`V<[ /~_{ ]5{߼n!,5+--cwml'{ܤQo+ I5ZKk/\ܺ/5o7EQr=]F^m㞯֞b e=uz [j/[O%[䢃L4dv}[—1=8 ƘyWϐוۙcdJiԾu߁`/cX6>Rag%ۙ7VDT=ݗl9w#bOj[E{/jgj~VF>ӿW{}9rʻg~5LP.̱_Auz?ҞPg{= e_Ubm7n>z =P܇+}{x炳fV YE"~Eh֚;/> KҗEj|(k qdc(-CV=e:a9l /9H}Q~By>+dKߟk3Ń̖W+16 6ܵ!kk`%f%gCYЋYkDl*@;y~, :h2ٷw'3olZ{&P_?^\a۴ZSHvsvmoَPG5 oіI7fNqQܗF3eQjbUƘ䔗]{H5`!cc΍W?_ ϧ3kh{9ojm6}=568dlD[پPaC嘒kVsP,ލk3R +(X ڍC}N5?='4]SVI* YֶZ%%e/c; [syD_QD,BA~>a:{2{txPC=lh%ž| iЬ-ݖ .JO߹?8 YDI`4#,o~Ѭ9@ϳ²x$D\.K'*0y%_nULci<J vt2Ώc^*ѧ}!k=ٯTҡ/\ޘ9ִ|N*m7m#i]{IE_1c6ɦ&eZcW3N1 Tc|VUJ,h,&zc_=޺Orˊgilk/C1{9KLu=[q?Wev/qfH(R{dq@X,)22Z%m_}Ι{ٲωh>vt=\ixF|vjDlA3Z!lUa1ɾ&km7ٽwe} Œho Yl/ۑdh Hq?'Uq0ndܚ-I$;N4m?CGB8]8*'}@ܤ+Ԓou'Uqm3pDNQ޹cb"Y6sI]JM&-I>?qk8Qyk xɰ?:)ANY$NY$NY$Mjx_淫ܻNS sؼǘ6b61{{9z>pX/dw6i(?*˹鳳RHn沈{gv^;=_؞~Hq>>%|iN,;+83݅p v3|0| Cp?χouﾉϿ~2/쫲I%xܯ{C@ZHI>4W91t8"_].,yQ*Tֺrx0OMy{>Th[<}=\xe{\+d~Q,j/%qWHD:OdknU6gDvdz ;J! ! nLbHg&OBMBJ ߻@0`xKrEtMAf볳! /  G?B b1|4&eQҹ|xs\7t) X ~幸 Y)6?D"QHvX=e!D IC,dGRn;$my9þw. NZ:ER>99_ y1]a!Wf=hCcuB֒0q).i(HʈULAI*kly;7 v6Klx.Uj7jy(%X2P6f_H%1-ԅ,2a (~=\?p$|ȒÔ_[uԟb!llnˉLٯ/j;u߶_+*B_ Mb!_`U Ʉ8r9Nx/jЃB*qz>"cI1BVeb Ov펵9pa (/o*INxe{MB/Y_=\1~g˒1=% )؟WMIk|[ۦ$gڳ~ Yfc:{mz,pO t{ծٟxO_d=>v D\+8qFKuۑɞ4OO>B@:>dmX11knߋ3k)=7#1;4 3bvKc*dYaR$dٯX$~5!҃B|} WRIVmW?r23]T1 :M8?i{rb5ϗvH5[꠲޼5n׮+sL?"&^<NJKŹ+JNILBmE)-EkkZF"u kIǛYG":b8o;@1햻,aCӞ/ 68F\ c%61eV,+ ;QD,@,%3BPnD-q]rAhOcU~MkE⋵H@y27y%zH䜗rR69}2) fo 0$])?.u!+ĿEDjW˄siƗFǞZ,Lɦ쓍Ӽ _/l#MA>~\-:LUk~1m"#R̵l{ݘ-&S2) (198 H1-vL< ͘Lj0%Eiv`J.Jnwg1LҀk+Uz 7CG}o=밸^kkk!]k*Sgc푴:h^?~۪ʏҋ0|{4ו{6r?1lc?{Unce&!+>gg|l*=l&dA T q H] dwcd}WT*QX{_ 5I.dYh 1>i ~ mowx̲:<Ot'h˒V$<% (6}PI?8!m(Hn1Q!d?9,#9_1#qcfujЃj0nL16$vȾs&_֘ękf+ PH^`z@)mڻV& `^2Mmk-%3\]=g9g[Z Y|cQAbѯJ^=:PΣ$*<'\q֏P %24߻~E9Uw/EVmg,H=E]7.c45PKl)<l"d#+**~Sf&L[*[B]Zg۹?%ĽҘ2b o+% ZЅ}9J}KDHdۜ_t YodIؓscܻ1ǸNfLJY+pǬ4'͵%6x1>mo;f.QE g/1h%GL Y9ɩf{] u?/}y,lj?_-K~+ꡣu&ѳ>0U5Qk.=Brۦ5;Qm[G2[YO>? `Pæ5UV1(hvdU+RY_jFk#6*>m ’eS=),Eo FD%VqsZ'DRIh~O/1ᖽ e>֟ !9%ܾHI,1R؇!<~⿇o-^(8F[5<\2fckb]D+o̊s@K1r>,[[JT,6yPr<μD6'He3'/DrO_UDo\X'dA̲vwǻ8)y-Zdv\m c6/)A'P<~ 6Fzǰ1f m_9ۼ+ċM\c➥/$$u,1mkE)͹MPv~$(wb}kH;c٢J7/urn5ڻ{_|qNuC;v ظ\A~T*s_홅BHu,i{c!+17SLxexD¿A ~=c1i<7,?d]c8R3|1Tzvp`X$>%|հ8`|6j9w-\ͮBVH }9$Fϓ|}T~h/=d?/ԽFh,;ݾ{-&Q Y5o=>.)'7Hy}g,Zb!+Q,-HR$$Dqf^$8D$ S?4wm^)W~َ>;0Txc~M!-sʾ?ׇql|!x4EAKԵo%d}2<MbR!#Ѧ+d7%(|x33ņ\zC0?9\k7!}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB>&AEXVvY"bxD\P/m!pwU#\'1wƪFl8LL,=<+BVLP%&dk2!$U ЩnhOԛQV"N*iU򣠾J",ji**I"B{ B}8*vgGmb׵]J&sp,}c;ctE Y2.41\Njaf}~:ܿ|G_Y(ׯۦFƗM׽F6:.8qo+l$]<6F5Ld8 Y,*\/Q|X$,1ٝYXr(BVv'e+{)%&1CcNy'dYJ•'t'DR4*\)2I$Ui&n<[ Q6m*+II%? OZ y9{V$,?g'!No)le_IɺT!ĴY2Y)yRh/&_VbF |!EesЏNט{RVY+\m!dy{Wp/{1ʿcMm V_9u ,R}3+} nx6//沟b3BBmƴ9D?SC+G`8}'L[*[ l;g 6T@9@PP}Mh܈sӇHq@Y"E&𜤊\Zčyn_f d . _?"?,W$:I!+9Db_&!#ku8q$<(-wLxm~+_Z1/K>M&n݅k+"9߇{ ) Y 棈t]4: B̖u縫ڧ`5Nj,@~q1A=3۱kmv )RȊE E&Su/m\FVxD4;3viL][K,k :}c Yֶ)cV/ h!+%4*!^6'.(bdQJvvmA/%:H*{u7ŦYJ BVJPӿmoC婎B,DY`lX?Yyj;OJ 7aӷmWo~q[6fj"!in-ȷ%X.hyG0nYd[ch"%\g6~vr?Gvַ޸njmr6׭3[{p2Y{ј3" RuɈ63=!},6%h%Hd"hwL$xK: YDH_tA!븄/xǯ4D,,!8n"@' ,p@' ,p@' ,p@yr5~fx/zxp]')`/J1)vGV:7W=/^@oW~~=\xe'ƞq&n.{뱸,">ѵgq8!{sv8of|;qB϶=Ǽ-`8s%^zl8mcQ~=Usgg;v KeϴNw>{i=4>b}}J}JaKqnmKsnƭg7ήBp=>Vv/7O~?9&>b_ZļGҋ~/,Ã>[GnGN&^oRF%>z5?>oƫA5ե.?8;xRoȣ?uꔹӽj?ξmgݶ'l7hx_~g $^?j̝}WTGv>_5~8W",h$d=_6GL <&s-!cOlPO'y!z,H=O=#eVR"aD,"H"'řIxEp.L <޵{\uJSHZd;lxHKi<я_6ȳpI,*B"TdD_Ʊ(j˱4Jzo!wApJV~y.{Bֿ{f(jQg/5b^X8j8KKH.oUz_Jf)1LIo^urqv bƄjQ8J O{J2kA۴Fu Lld1^w|U1\ĵ@Ln L/d4l7N19MsmɘlJ9|f-Ico_ 3v>항}7ll3J2EMD&KDݎ@{>"?,Ec Z~(o5,T㟍Y>#JXCIJ5ޏԼeEe2:̏/O:ϡO6J\>{{zuR9u52Qm/mtu߶_MRUĪB7ο _[HuęSl8L8&}?Nya*_󗚉U97'|XT/wrp|qVH< b0? MqE+v)&βWᙃI$A čmgcڲ oGI~-L46cb|YsTLƤ5f4U99Gv.Vw,rϺ_wYY4lT"#c[*=.cz椽)sIc~- ~)d}60Ϥl ǎLa_Xx[Oو]_~u\h.y\v2d;ڒfv0¿I`)5"Zx^cۮt$IIȚ_1v1Gf1"DS/j׊q޵=xàwse@\[XЗ$[m>ߓdPR_B?틘ik:`^s0?5!?UE__cqn^b#iSlȹs,[6T&l|n8S²CWG4^Cix=_<@Jl%t:/:٧k̎[kOSC̎f_R&[úymS2CGwl؞I?$fZ4fYFxcFsk\ƙ@1 ?N0υ[؋4\|닃guo&AϺ_pPyca#}=ybAY6[>]Ʉ,룺FTGcۅX];w;9vͳL1\vL=*N`_1KyZWf9נc5sZ?5l\O{I*\њF4lghχh}Wyam}M,+|L $DE !ET#,%3p:_L)Bᶣ_5~%!FOcU4 o.q YzVjCPp!ߧ$!I>m)1n*f (wJd[OR %30%1h}sR"%<9۶U݌9){x?;ע1%tx{eQ1Kౙ5faygza(`&w$޲ػs֙RE'cl{V a瘌Nm^,uxTeN-sxm}˴Ge"^h^g1M35Q5QlҳR9fj9&_A\xwFVJ8sÕEݶ[%AuA/jOP(N\i~{о)ٿJkr/g/>{vm-p?g|Jb߉#nwe~{Lv9f!+V(!ܛ O~oT#v5A&Am$4fE1Cc7}k=/8a4?z fuͳp5p5pZ9m!ͫNR+{/R2pƆ%$$\k-%4>gssPػ jI3N^ix>dHbqokOpR9)ǏcąN>P#qOtR]eǶL4dv}[—1=D KH5?7{)*BFoNƾ?dw>6ߋX~2֭,r=Wo|YDܸhS|R2.{Q<lT5g˱Tu>9fb=r9g kc:]8tŰ-k4u3GYK[Wz}s<mfU+RY_jFk#bj=BX鋬q5>r{R™Z_45b(U/rҏ8yztìs$5֟_r]{p"}|:,|WȊ?oc/fǙ-W4b8mlkcC%;JKDч0׈f%T1/wJ J"x%OLgشM<~g!q:H Uަ]:Bkxv=aNxLޯ1cv349/s=,Uk]*4&dUzlW?}bi+/wvc΍W?_ ϧ3kh{9ojm6Xce3 ׬XhMyiʱRDy7csء{5u2wp=Reگ֘e3梭9;Uv%yc4iQڇ{QaP#ۯXۙom{Lhf%>s}nHg?`bQB#E TKW0Ŭ^.͠]SR!ڶNߘBm&ńn7)ZVq6EA~>a:{2{txPC=lh%ž| iЬ-ݖ .JO߹?8 YDI`4#,o~Ѭ9@ϳ²x$D\.K'*0y%_nULci<J 1OI8?Axy|"tU_lgW*ÇǙ7/I|DIusRilIKĤv mjy%tLxH65q /^orkֿR"S1S$ŽYuHB}2P6٦c_=޺Orˊgilk/C1{9KLE=#6my+VsL5+8(FjF%wĞW"bMafֈL,omEvΤcȖ}NDsh]ݧ~Of3|öS;/'b КQfݯ یM}|0YhCM}W} Œtٿ&4lLyZ_w{4js/l~"fĕx,zV8}pk޶8'&IDNߓr8X`&  YDH_tAਜxRNsSKB֝V n-~8EB^xӎ#Xb d̉'u)4$ƭFBV7ۻs;lgG孍3"o9fWԟ'9GB8I dB8I dB8I dB7͓7Û~y/Û߮s:g^WOL97Cbbc+ۈ "9ap߇:Ǣ{/粧ΆwJe#>? iy6|~mc{/#i|UΗ1&pm`xW{*d!g`?Lߜ}1d/_ -We;KŹ_h}hscp?D\X>ãTu`x5WT#83u%G|$; .i}{,G5#5ہF$!gO=[&duBxnbEˆ X8BN&1$3B'&,%ͅ"~2*+p~َ>;8Sxc~M!-sʾB@>c YŶM?YZ,$M&n-ApJV~y.{Bֿ{f(jQg'8H&f--iOY:A(8'CP& 9Y>%ѫIF`m- NiLN%TrNZ:ERk#T/>, y1] Ymї3oS]w\cuB֒0r>Xnc?-QB/XD(3ILT T#3ةL,Qv ِIv-6$>#|-VZkp.uGH6+=#i"8FLknvyڶuJHsz,a9vKc*dYaRJ"cbض+(A_m:!+yht\!kfEJ*Mk6A$~ 68SR¾({צnt/Ӌm~'R>DŽF,K} u ι?){R>(^ YtO?x#%hmgXHuȸyJ+qxnSLܵH jq͌&cF xMNl͖iW'%􌙸wv*1m̜Xˍ tF B>RIڤ7V_ IMuo5\ sw&c4g91VKf|{Uí~uPY o^ka[9z/dşPgi]mB KIkRd(ѱWbp-k:ͬM#Wx1~}VϘv]ư xiضqV.eI(/bYI؉"fj0 1ͽydVHQ1 8Ay|?o\wJs\[r 1mӌc, M[&ĠML/}pQ&㽤~H|INݎ羼~Ă/UؖhÍ~Q[:Y9]+pYp`QS!/dYȇ޸?G}oQUle>G>y,Cb >zTYen̖ǰOs5l]AV YE"~ly {oT~1UimOX,W#'E" k}*k %dmaO,i"^c-{1'|?'B"r]K}5YbN-YBy>+dKߟ \X'dA̲vwǻ8)y-Zdv\m c6/)A'P<~ 6Fzǰ1f m_9ۼ+ċM\c➥/$'ltʛYPe^B;͹M6~>$(wb}kH;c٢J7/urn5ڻ{_|qNuC;v ظ\pD>*NUui=CkE&)yڵIkHؕP]oݻs$,xO1UH eo,q0c|ShuYH1k6~[.YlBK.B H_`{brz!nm *u"y%BVזiPB8WU` d0} Y#}tLk׼ibN! "3BYȁNYNB-D,@' ,p@' ,p@' ,p@yr5~fx/zxp]')`g_e7nZwc|cNИ7={ʝ$ÅWVbs̖$RN7> .~8`|11V_k<]ArI?OOS٣WG QRͧx︞_jtͳD&d||j?ws>WJ+ޜ"qk}ފuWD?ධR"aD,I 'xIxEp.!L TWI:cq)$-}}v6<$4dكb/a2]bHoǠR AD{<עmxB! $ς N _e겐$$TtR[JJAˊddIhBVm*}w.?>ξsxcnvnؤ?O F@ UFA]8!"Dގ@b%`n36 pXvK&1jhSE{reybC=!yJe:l®?Qf_Wb5f_ٞ1UZEVĝ3#}GaW!)!hoX bUU/zOMmtr5__'dl]!Ke*_!+&Hb%%l"MMJ ;&2)~O>}NW,bTO4cښW'F QWX$"Q(eO(d}MBU YٯPO fG'%O(D1T<ږwLxm[F c[ru>[cV8d |5![^e{!+0"Bk>VEr[>l?l2p}oؖ*mO'~c[jݷ,{Zea~*. .qfvc\Ym>h  ۗlB B@2+ )I FY K2$BuRm!5~%!F-X!/j7oc5sw-d%XbK qO&BNR%9 k|L>Do%m(⤒])? k$BI!Ϣoۚ,! d dq;ч[qbw&O z&v]ۥjk;;Z ?'k=f|3 /H YDܐ%}Osuk'g}x`[rm}_yjk|tk:o2SOj."F~u ci'[DN#۱D-P%!D}~YD U' d,T76&jY+\!B?[Bnm+1^LyNڽN&J-iT'RdId5'LxlUW"ד$6J~ d $,=/sHX OBRʾu鿩!B7i/\eRf'8z^L K'3ᵝ|iplC41 VB:'&d-:^ uc ƚbAFsXsfVnmbm__"Ie?žgیis>F!>g!+V@b#n)#~>F{˴ER%ԅu3qM}Pe xuMJq1~n)dHq@Y"E&𜤊\Zčyn_f d . _?"?,W$:I!+9Db_&!#ku8q$<(-wLxm~+_Z1/K>M&n݅k+"9߇{ ) Y 棈t]4: _c1Y#wJ LɗI37HdI͐~gm ܾH߂bBkaL%_Ŧï%O5%aqK~e`52Gb_T/lݘU{ @䘞:.'og3ᵝ'%/^åyL&[1 ~Z{%會*~jBV8jcXoK磄~z)TҳŮwo}Ұ:>Ggm;]}>[9^gh9؎]|l+ضcnOBVL,Jlu0D0qD {}Q X% ۥ1umvq) YֶuTʂ/d5cVYXŀn_X~C #S /_x Z(%lDֶؗodk$Խ庛b,L%Q!+D%kp Y߶R ! ZZȊmTGD!KWG,0I6Yw<'vdf%ŰԶnX[^Ը-35_] Yہm~,ҼΣK,nFj4nc3nc?[;fja;[o\cF̶u]}-׽nzRq,ǽChVj|_dD~řd>XfF1N+AZ'@oõe"[ 'Ցtm@'B "Y%|;~q/7 bF d8YGss]B8I dB8I dB8I dB7͓7Û~y/Û߮s:g^WOL9{Q\L;W/9|܅+;13n_8ww!6[Mg!qeT=;9ޛy3Gq8z9mI+B{gFhw˞>;)|@hx^*7<|muڼ#Oi\[SS(跎wcUo**ztv.o~h~.͹7Oג}U'8r^\{a?/8r7BB̍&0:bWaξGWC'a:8;xRoȣ?uꔹӽj?ξmq۶'l7Pp.]?ױZٗAhŵoY9' Z; Y}p=aĽ窪 -!cOlPO'y!z,H{}=#՞uVR"aD,I!'xIxE.L TWI:cq)$-}}v6<$4dكb/a2]bHoǠR غ+x 8%+ABO<=!?=3f~H n/,y5C㥁|nC$؏g}гb6ӂ%T" ul Yy;7շ_ ?,Ec M~(o 7,1#9eEdk+!EWb5WO~q_0/PFZ[G9F`q|yO/N*g\:%XޢIwX@XUkK8qgsGD>/,a?4pb%I֡ 6gKSJB! ]ۢ/"qo%A{! YqK뷾8_)g3 tMVDWslL[=ѯ%cqFqL /k1j ԘƌƷ8w-?~v-s2v^Ykq3+?+]YdDwlמYl>.Ms^PṤ1t>{mg6wcG{|,Yo-llĮh:.űJFaO)_7SgRjArޯ }||趽؎m4i O.߇~V}70~{]dW!)! oX bUU"1 +/rƶ]AI :Ku9vYȚx|%[Ȳqa/cOz.Wl3ā)0CY:VwAMuwIGz*|ołSֺOg2!]pۢl=3_d{箝;LJLY&qC;\p~~v[0vܯR<+3Fok[CUܱ9q-Y wyl{h{_u3_4Cm g{|g>>]z778M,00 eV,+ ;|RD `eH$_L)Bᶣ_5~%!Fq~v!;_8ǸaR qsʲü< zA{}0*fot4Urwԅ,*Oū^a/v#]8[ЯBb7ٖ1܇66&X4] o=4UŗX s]Z=%_ZK!ٟ&6u;֞㦒HipBM@Jf atN8y`smwM5ls|^4Oq^ע1%tx{eQ1Kౙ5fay {P#R ܅`Ȇcr۹53bݹ`wqAfmֽipJֆq`}Tvk1-xy{p75Di?D=W)KώKݧ昩#8wVq^m3G[+a_ Wbگuۮo:=y;Cy}nga͟s oNzS?N}Cl&dC TIQ_R_V.ABIYb ]XrD(BVv' d+{)y/ IVlcycpHD!L!;{}_Tá߳eߦr'iGIk||/ȎV~5\˗Vs< /-={uܵ!q$嗨 J*-(/2ɿ|n~X=?XcG+XbmqSr#[;(|brbdN@GJ&.#}mφmU7cNذIRَY5=M2Ǩ_*XyՙЇԷؿ➓a>rw>{ά JE'cldoĉafw1wl >j"^K׺l=@F#r.893W`Z5^~y=\}*ڏim3gr(f=8okOٵu_O k)mGo|'V۶Nn$L_Kl8ا)|9CTc̼[PYrel{N?!^ǒnm}d;Ŷ9]s"n4ߍ)>m} 6uJY3L^*:3C3vVյK~vH{B .vbؖU|UZ]虣{ >,mE~X}3,8mfU+RY!–(%XDQmSUVJLV},\=BķПR<_c1Y2{ .?.alxX~ywE-?*k}^!+{X}h\/gR_шQᮍ _[+6{Eœ͂^̺/^#b՗POBq{(Ƃ{;F{}e{ƙ76=E|(řM6MucpHݗjZczxҚD;V gjh;>k>\_G=CilbmgIV3Ǜ +GaMVxJgγAV YQ0(AבH}N5D1+`hKD3(3+lԵǥ$dYF0R*pێYalRLd}_c@o2ӱv uq`@eg;7gj:$vmA/%p84{u%K߸w}B,kwX3Lj,Ǜ_4+vc8쾰&)"/KO:Lc^|9uץGrXﯬyA;hǂ_BUrΏc^/g#!c8J:q&7&jNұIsm{$k/ځ<\W{̘,c"Cc(|zXE1&jZgP%{Ծ._Lc_=޺Orˊgilk/C1s$p=Ʋ8uvGN5+8(FjF%wĞW"bMafֈL,omEvΤcȖ}NDsh]ݧ~Of3|öS;/'b КQfݯ یM}|0YhC}c?knUm-m3p|b0At?=I*f@qϐNED,ʉ'u;7J<)d nUmfx S. dw8X>,B͜xR@ӟxKkܚ>o$d~#;36qvT8 6cvEy2ON @pD dB8I dB8I dB8I dp<^y3yyry=j8}pĔp\<$6/1/jmM`!ž8q_Ϫx~ Y }xysx,?r.{lxT6nq{𙶝ig>6q_>Ƨ\oOOû:3ELWR6=U:B/7O~?9&>b_Zv$ŗs !k!%\U$?|u|G߇u?Ľ5}^%^@?ÅW7uBֽ{Q,j/T|{8/-HmhۗÃ?G*@ nU6XxvpgttIENDB`llama7b_autotrans_1to8_train_output.png000066400000000000000000000315571476650141400356600ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRCnsRGBgAMA a pHYsod3IDATx^5Ybbp[<6ōq!jo Bm(U]\.5B̸k˴żr͈#D8?~?p97"22NȬ<_SA5P,~ӟ=|,!GCxjP A1z=_^rz#v:җ;pSC'a[tǥB~`ow5-Tm·}8ޔO,Zz|[^dܸ7W]ϫ92j1}PH#Tm_͹։zR2iM OX$0Ц!OS']'=*wiߴAy˸ZtC_@Mۘ3鿾)#U^?FZ|:5|t|ݓΊe`"rqέUb~vԹ<ڶr)KQN!fZz.gA>URqqy!Pgd;K{ڗhc_>PROb^޴/ƻoo*$"l&,} 0*VqeHm}-O}iC%|'>ؖNJ:x7p_E b" Ϻޯ7%'ƭD,#3q!d_o´9ۙs_o-}Q EHVIoZX ZL!H_;}mfd\lcc]cK0w۷b%oOD+U 6&YΊ5uݬ od[\H3\h[5JsM21|f3mk*C/l璎˗@ΧDNisHy~nD\cm:G>|8'?9#34Fߑ -Y\U[zŸWK\l'}[>[N{kyq_+d(t>鱉=c)?N?ǣl/['kbY2||xf]4l}Xn"$tu6 qs ! R!D4C1rb;-O&/n:{6|1ɋN=7N,|w?l?K-&Oa3{|y]_^&oO= Xă!=Gykϱ]68>=]O'1o*x4vNoH/ZMs'or c+㑬)iKι޿[Z#lP !z.}QPT*Cq65<*Ȯ8w憂zj0< (ԠSbO !<5(T!4444444Goo~@ 1n A1@H>m N u[`>n!@iqzL0q{P Ocyۃb|@|C4`^$6-0 |^Ǘ777r:kN/) i\|lz΋T>)&:?Œ7:g_/pLcM2M)NWy Os:BQٻ{;ŗSq&f1r8^]%OJ'^,ĸӶuNi!!e8ևi&.4|M[\I|Ly=oֿ}w-@6dqTҊW^r㇓_ r1Xsv>Cs|Ov=q?n&h(C3qLcг|%S 1}z.|S9?sX{i<1+D""$b,9d }?>ƿ-q&-^8B 'm֣&eR󔓃 s|L :?'Ea{[ %$y`$N:d39V4.>8QdjkD?)E5 }&ךsq1DX!'AM8O)D89L疕PB"i<“Og˸$u,rM૔%t\^ Ѳp=>ކ?)m!bo] ͱ(vUoGnD&Dd&)-bsI)܁%hE1uw! u$swݱ u֣ܶ~kmkvHi^7~M%V̬=PLk>(./2ʖ/ z!u&jŐ5C'\^@n#MÞҼ.GbB8B4*e_ph H;IH X e̮glq<\T'|RunmG"F[~nOrh| GzΉ"*cJb埔]cU].l*l`kԒϽwkZq-V/DQ zt 'n|h>&mb\ @|C4`^$6-0 8=&m=(hɧ 1n A1@H>m N u[`>n!<5Ê{whk2Wh?V _p[Ct?hָl$pC"⟝^6{Z1DO{ #-#=XA .tV Q&ڠSj1n7mC ɖcp>5*~{:A12(pZ܃aw___H +!<5(Ԍ+~|>6m`C|v>ɲ}ϸb 7O ĸb(W[@/h|9_}k} CENÛ77k_\Oo/ǰu$?2v_/nh b#FFL/rn G( P yX1aq=˙sB+%/a_[m^p&vW QpS"wK1pIɹƹDwJQ> %_u Muf=J?hƳگv8*eγkίEeeb' ٦i}o`P/}`g-ꄴ.&u!yZknPd~.$ B(^gsw#|-ӁUي/U&R8  Ɠ8>vd(98P J9,;JT}eZ_gfNޝLwUT_L瓀r3w'g9%oڡx7mxX7\`{MgGүb!X%X>Zm.֕XuȿߦByTm=p\@ٓ8€eu&ƒ31$|gG}+(b%$˶,)lufȿb.mVV])l@6үW㬗 ;yp/dڦFُ>k~}C4£+$.p~'_g<ݡERX݁nW~.妤k=^~z_;mvkVsL؈X_؅l>~qлENK#] ]c\1akCwCWɻ4O.'d`<! bO !<5㊡ߟ߿M=+|e /0q>6OO ĸb(W[&5>&N#.FB/DYKI~jvso.<.]}1;_p&P U #9Og޽ؔ.a[x-$v$y Oѭzfw[`>]1D9N -ń'ED'\zx[uӠZiKƓHi[uy;_wy\@l~kmS=:w+_y<DfF\Al ##Jr4Nş( z t1c75얞 E 뿪ߪKr!|Q 2Fj[e9>m%_~_a{Mǥq*A'q|Prp>Ar*Xv5W@ɬ~៑;yw2UVr>R}3k+,Z1Y=sBnIУOrwK޴C }#m7|o^oϘ_a.CrA'Jz|l@\%Nfhִ535NT05Oʺuk|8Hs }^q a:~w^c}1ޜΠ2:65Vj ;yp/dڦFُ>k~}C4£+$.'ۧ@z-QsϭE76 <ݡ1{MNf(t)7%E廠3ߋO𥉉ͮ|Mjݓ K g{]0ݯ0zyii y 1FÃǸbì?|HtHCwiF#\O0y>C!!<#32۴ȸb/?YW QgԾP0N+rEPO/a^^0M*A 1ד|חxwIl]~ydG>Ku/[ ݌]x?%'1]8 cd< L ntbR55Y/| n G+(8)p\\"K*n %'*rӠl֟ҏ4ڥqKiߪʾSd{ [ksrȿ]㡰L$57 /g{]0W  *8M&. zŌQ,_H:n[z*)~.Ʌ^=BD*<˿k>n1bV|}5I1a'CJ0Qʩ` Ԃ:zRm'JoNޝLwUT_L瓀r39,,|M;1v Uf l H] qP|B?+y4b+)\=z6?535NT}wRnJ:wAgW+N]\'26"'v![`><_a\1|HBbyW }~ih. h #Ǹb`SbO͸boF"㊡o>;dپ g\1DOSrC;1A=t @'{ͳz}|4bz^i~ \g%MS3en=yOR]#&~/rn G(JU7W|Gb]8 cȷKӮ?K!oX|g ~l3-0MDb擢ss .J-Ӕ덛G(bԺ2UX.ֵDMU}T@<ڣk'5qCa 1HdkoE4^`>2 * 8Tq ,%HVy@D@m]-=,=SozX۫ l.nx8^s:жY"[פz\ 'bx. %*D)uYaCrZRm'JkoNޝLwUT_L瓀r39,,|M;1v Uf l H] qP|B?+yv\%Nf\ %uYO.Ѥnw+hB>{򵾳r'BlNݶXr/DoE$A%rRYC59-t;dپ g\1DOSrC;1A=t @'{ͳz}|4b~#.FA/DYKI~jvso.]1D9N -ń'ED'\zx[>'G](AG4(άGGҸ%RZ5V]W}k9~kmS=:w+_y<DfF\Al ##Jr4Nş88]\C\O1 ZmvKO"_o% o:'<˿k>n1bV|}5I1a'CJ0Qʩ` TT}/Jf ɻӗ鮲j<ꋝ|PY.ҵbǓޡGdiA"Fn޴a,s5a1ICjw%W4(mp8ŚJ:K:S!X~p?W}tHn44y`4Ʌ c\1AP A1f\1iXq7_~l}3'}i`W 加v :_G>`>@1TDbh4Y3&/~ͽٺɎ}1;_xw[`>@1T^M8Ogr7tQ\,!B]iWAr⥐7,ktk^?p&vW QpS"wK1pIɹƹD^%^}҃b Q13ZEFxm>M[uy;_wy\@l~kmS=:w+_y<DfF\Al ##Jr4NşPRk=: kQ_}Oi6Bžg|R/ !`{d_ߍ5s|NV1Kd+TTKA$TO員|B(TZ& $+ȵۢO;}*v9cؙ'"]+&˻gn9sYYNnɛv(/bM֫2^+~V.(DI/XPH]]/_3SO.qSnw+hB>{򵾳r'BlNݶXr/DoE7%i򓉑 ea#^s'+z<1X2=YH~.Q=JB_W[a0Tw!^ɴM~} .&(iGe_WXAI]@{4N'jn";T1f˼2߁Vo&n_m]ką]J~z__ߕi[͵{2!c#bIzb~al ãC>9-t;dپ g\1DOSrC;1A=t @'{ͳz}|4bxfk;Jz.iⷼܛ/숼ѧe9\| C%‹ ztw/6zb!t t< L ntbR55Y/| n G+(8)"%&n>) :98rS~Pp)ӠliKӧ~.o+N9 o9xʵG"Ntk22bl߈6hwu|d_1T:@*pK(G4A>XYKP E}_=Q>얞 ErNI{,Uy6}7R|b</9h[,R kR=.P1b"25Oʺuk|8Hs }^q a:~w^c}1ޒҕ eaC,$@^s'+z<1X2=YH~.Q=JB_W[a0Tw!^ɴM~} .&(iGe_WXAI]@T}qdj\Ztcs`[$HnX׵!BզߥK)1d/f?V65mvO&dlD,IO/Bwu|xt¸b"..v F`10_5!ݻ+$D>0a<|+6!<5(Ԍ+~|>6m@+2O/xCT@1%km3b^lJ*s:|:~ w]Ei]⥐7,ktk^?p&vW QpS"wK1pIɹƹD^%i}7n"G= &@QF-iߪʾSd{ 7\{t~-DW&8.x(,;!lͶh ~^GFC h&?k PQ{t֢/(Zmvb3UV]I>|Q 2Fj[e9>m%_~_a{Mǥq*A'q|Prp>Ar*Xv5`N%FktWYu/vI%/\kdy-g4|<zIYNnɛv(/bM֫2^+~V.(DI/uT[ASY$*B~ ruzM:K4_[Ƈ㊴8گЧϞ|ɸ,ۮw5|0!;;[A]P&&eZZ659-t0a<|+6!<5C?'t5}kܗп;wm?\1 "c "bS5*B#Y/sb.yGC&6W 5'¥Oܕ "?;97d0-%]$?5oy7[_k1&g]Kdzu9.|EFNċt?A7"Ok1^9[H+0*H**ɖlb~|4beJ/pr㶽7(|PSwYBsq!oBj?3%4}xX9Z/&\jl_ߛǞc+N-Qp(.8߹xXN8n:ԓl 8)H]>sҼ[uy;ߵ1\Nzv9Ǐ|9yy ])e b!mAY_w[={[gg<3LSHa$Kc2F>øFB03xbV|}5I1a'CBF ]&e &}Dߝfodk`ɹ Hݢ/d Z,T>OD-c_kё=3'Jqw/]e.wX/vfͫr1Y>>fgOF㖼iELizYk>c>~+ruثor%P%,D#ARWۡ~Gd^" Cd~'X㏎>nu _x:joM^ЧOWrq! X]nk`,9*d,"(*J]O%g[,'ŝu;(6k}f?Ec7y8|;YAQ`%v9Dr^VV]7&k^W1Xzne3i6f?l ãS3.ZsJ>QrbO$$'dC gf>g'K5|>gC,qFe#mHl YK UQ#G>nylʫo#ye.[bV.]^rtdEdb_XdNl)s8Jִڨg{]0ݯb.~tN&ѱ-bkii!Y;>t5z2b|r?7;&Om4<Ë!?ۦFC iZ l CxjP R%оՔIENDB`llama7b_autotrans_1to8_train_strategy.png000066400000000000000000000374701476650141400361620ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDREsRGBgAMA a pHYsod>IDATx^ͫusȟ88?pN6w$8I#܁If?6Yk&ՑA_HHr9vUZU|?YݫWg//A (M8p(2((((E^_^^o//iNoo[>+˗tz~+K/-R !Us4-6ǔ_|z|v{xO==x_cEQ~PmYҖg']p /A_nNy*J?ܢ\ۮ>n>#^[_zdYz硢 =Q9«[NU/Jɿ`h}Qz:fl)$ (6^x)>b/aꟷN?]ʾ.UGEu"R_m˲՞ȶv}IO$,WUz^<~U:_>O1XWmwg,euZ>νû~d/ᡢ~2xr:~C^dtL|979+,/i[Ζ|lIwDz/ Sw}ek*"Fn}N_kV9 ux=~+g &_ bwol0rTsvS͗(b={>#X(ƉJ4uNrW|b΂lfVt~OWgbm֢;lsfi^| J^aƳU \kb@9fdOm#@,S;CWCksn&՘{pЯK|VkV~R5E21WH$ԑmqU+Ӳ$I~a{͞~fZb)D>i:隰yH7d_,=쳋"gf.՘AVMc=|J+}+u,sUst-2Irr6l^>sQɖl(f_;lU|dRP12z9<_rl_&[ϱċ*91 1AMgx_yk>ivwJKFA9Oυ?5RY "b(:(yVB<DZ뭮c11F}%6l_"R-󖱁Oati/Y^^+:5ؕdl}e6\keyoIBn<"5͐e<9I{fZce_^K׹n~%J斗"M7ȅQ "VEA tQ@(·qơbU]6] ci"~ɷ((((( E8pW?$fQ?tZOq=ǃ ƀ z,E ?]X "&@~&@<>EL$4tMcx|<(Ih xP05A\(`зח~%]~_W:Y{}{yy W_T<}'cx|<.^ByBft=%DZ_zP^빹/?Ifwy 湿rq9;ݐC؃U(z^ī %pNZI<4}Wy ӹ]3gKEɰo!l{7W~W}3Gkg߷:N$*7gs\ X>{_| xh=|ُ򝞬 2OB7_|8E ?Vkb"^U8yp/8K/S_=(믷g͜]?X^x*3"rU~Ͽ~ϫ>{^y e6sGM@>ZI)' !=6eB['L|ZII"-/2WE5Oddoq&V鯴9*e >1Et[}-$?)cfZNMn XI'>S#Wa+Xֹ"&= =S~EV()Tdy1TÓlԌkICjz~ֹGoqɝV\OCXr;o/y[z>|w!xx>(_=J<:6|6|ˍ؇U r_RScUaE5Ϸb35%^_F1k//6OM+uFȂLn{ TXR}]r_P율( oMnU[i\y;Blxʽ̤ROU4w6YvsZ[@ =}{EEL5A\(`$1k AQHBcOq=ǃ ƀ z,E ?]X "&@~&@<>E͋??}?'E~WůN۳yQW~?C'-EoY^͋[:xEWG:h,3ؼ(8"(E߾}{޾{H#nEѿ?ﷷ?qآQ`;]0V( |,iB_֑lO kr{9 ESѾ8~rPO͂AߋKk_c~,y _c=޽v=%Mgӽ-\D_\s 7ӽ/!?i"r\o줔 <9!uZei|KԯH]b.-G4_ ;MFtM<|b'B} ۟O{hDFmKThx5D\j!VnIB/zE/Yŷҵ.PqڢΙ*9f'fNpbJk1+Zz-d aj/p C97_Uk7QsN4m+ٸ퍰(kLjO?O#{Hb&_W]/&oR]棹ڏخdlkߓwOڵ|ՑqY{6#Uõw3Dbr P;ϑۖU#6Om.KNWtQhߚg%h@jqxB:~ /zzDwYz'XەWG[<3{Z 'p}NKΐk2YZ۲SœiȱzWqTk`TqIس˓#I_>XkV6m/ض+^o/\~"Y/،=eV9U7 =UW7v#AkZhpH~۬nrZȔwz,O(QO/Ak!_Ws5?V?7[jz@UgsR+g^n/v<'n7sewQh6'gUr,8{辰h\cygS~rH@r9QU}d]ݵsHvE#L:Yg_S+ tNrx(btuwdWZ}Pح(3C>T KQp4N~[r/gF،EqETcq,[q޾jNB<8lQ$Eqcϴ)v{fC|waNi? ɇ9[m/z5#߫-jwcE9ţcϴ)<$#~_I-RS˳|#yOԽRn8u7^-WƷS@G״+gwEQ -zPS_H*T8wU!Q_M 5$S٘#Q1"m]\5 _R/.o|xZ^˨Eޛ|Ϧ{{[S`,\g-]o={_C~Ejcp9ayr]q-C C2 Ӝb5ب_=Ũ]Ziu͚g%h@jqxBIp_(qw'vC[q?1w]MgrQ'';Č;6ֽ'9\Ӓ3d0L,A0g8rCU9g876hjF$a.ON?4K_Y51`Za_^Oy. SMF%f}If5ַ[csM;aV9U7 =UW7v#AkZhpH~۬nrZȔwzXHE=y+c|ͺ^ͩPTX(Z@UggRpf5 s[<%n_l<{ߖUr,z"ۃy!owޚc'T吀hO":7sPɺk=t -Ft!'(ϾW9b@Q4<YWt[ھ\(VNsxV}d*BE>>_My-٭(j}~O?0f(GF؎CE(,(C$cʟUs1a"y<(z{M?Vr`jlJh!IWl՘RsRWr$*!]r_HkW%CnOoK^M'+_g:{qqm c~,y _c={ն{torK`*엥?e{ t apڢW-49 .' >O"+yH]{(VcYt3_1R<KQ?͗qW6Y;vOlr*?}ۉUsֻDWcKA5: 8諽9-Y"^be1?+_T[vCtץ~jAW~8:QX^/Rd=yWƋk#:vmF%OjOB7KD8ߊĞ0Em+c)-NN1vOwl={OƓs> %Iig`5X,m)aδqX=jNsθq50olԌ:Hž]NCf'yD3|P8~xsi^oO7W/1HVKbt56#ek ӷr͹HVYg'tncN  pZBCfvCBӓBC*\>h-5kjN"gDq]?F9߹<'n@h6~[b>W:Vɱ` TmCkl=V 5?[~,$\DfN>q`*YWwg}A<E8$d q<\,(]c' v*oK7 E@ʉ`{݊"=0tCE EG%L2ן~Fa8lQݏ(.P`+YAQ>I>Dz?7Y$cExJQG?:LrVF_eܞ &||ӱ ceOCp zޛ_w{[Cq{Yx (5xt6嬭R"!۳|@-A)Y '^Gn8u\ [ R5bo1꧀*5iWΔ$g1ĉTZyTCq%B$4:XkHR[5|۳1'{%G?b%EH]*xr#fzs ۼWmVE7OrUd,x~K6z㍱t?<^kն{torK`*엥?e{ t apڢW-49 .' >O"+yH]{(VcYt3_1R<KQ?͗&yC{ze5#nGIka'7Xſ%*4^w[ Xy_A_!VnIB/D/Yҵ.PqڢΙ*9f'fNpbJ8k1+Zz-d aj/@}nzNWdڇ~4mȸZYc>~ZvD ׊|^evr|z{jI569FcM{H1mSk]=n/֮뮎2U;%\-J8r P;ϑۖU@meɉʟEY==Z~R{ =q\]"‰V'yO.`mWnߙ\ɉ11鎍qu xrg$) L&E-;%̙6Pm~i7N QIس˓#IM<HOY5i6n{ح8~xsiSzDr_㭱y-\NkmE:B:?W]t:h71{Onw8i!mY!h!SIPb~!RM5z5ZCyS`38kal.oUɟIe߹-};(|s4Su-1+XE!56+vi?{MU T.s3'J8ѳ{N  pޢhIg|2RkqΉ`_.EN1U~Hͅ hD=WnE_:H`]"qwؒC~~O?0f(GF؎CE(,(C$cʟUs1a"y<(z{M[;,FngCt, ߱2!p_ g뵾)%tA{s뱒^ӡ8v@<@QS<:LA[)6ieܞ~2,Hu/~#7W P FYvLNrV{ C(_AUEJw I%<}^"$JB)$_Ucʷ=sRWr$*!]rx8XԵ!7b7Wu+3Z|%۟OY:{qixcy^kն{torK`*엥?ʇY)߰{N -qB#r(Zh5eN9jN/Q z#usQ|y^.t( >6:%qW6Y;vOlr*?}ۉUsֻDWcΫB5j/drKjz'X~Ywlǖн'u)~Et$GT1;)5wSY]_%\9SlI%eh}U{ssuo"n$4mȸ퍰(kLjO?Oގ6Z\V_«̮^Wy{O7/+5"HKYNݓwкiX֛:E}5BAD/"FWxG Ȫ] kBQr"؞+c~? $PQ0.E8;Ee|l!?SDQ`3[}# #lǡ"F؊CEtPalO kr{9 E*9z{tLo]Bcuǫ Wk^b,rOg/*َ-]ߡ{ORP? 5 +??-xIcvRj'fJrؒKV &Z|EV}gIӦި5^eoGtP~XW/UfW/gNjɼƛTcׯFcM{H1mSk]=n/֮뮎2U;%\-J8r P;ϑۖU@meɉʟ8kܣqa '[{~u%" o}bOmWnߙ\ɉ11鎍qu xrg$) L&E-;%̙6Pm~i7N QIس˓CI??ej<^mw\חWS|sޟKÔz}zDr_㭱y-\NkmE:B:?;yMvsHv~К6>2坞%RQO/Ak!_Ws5?V?#F06P՟ٚTlT~rO>\ٝl<{ߖUr,z"Uo{qkִ=&.Br9Q@ā}d]ݵsHvE#L:Yg_S+ tNrx(btuwd@V-mh.EA+'r3UrNJĺl;*pXKO%#̣UeȷMBI׽Z9qz.šg%7#mV~D\W~$55j%U/&oR]棻5"HKYNݓwкiX֛:b/}P'G%1:X߲3)ӷr͹yѿwrb$r Aَ9{x$ty?hM oUMA NO 'relԐY׫9ʃY c#vyOlN _0^xNvBmY>|_NzOtX~r]k<=; qXωbO!ўEtnDuu:zv#[0CBOfQj}M=7s9⁢i>ޑyjׯZ}Pح(3C>T KQp4N~[r/gF،EqETcq,[q޾jNB<8lQ$Eqcϴ)'mux QT/+C? ɇo=77+|qF{MjEQjN3mwJ̖q{6bXnT~{Ooͨ[{@o1꧀*μΔ$g1ĉTZ!,ߩ/$\9,}wIht~56%א+jLgcN JDe8ļKvUr9FҰޫm?NA~)Q?e,A=hŮ7Kcc)yo=Un L"slOwSa}A< N[㪅&GQ?tŵ4+pj,4Ns՜^b~Gv%g*Ϋ'sOkk%*4^w[ Xѹ!VnIB/D/Yҵ.PqڢΙ*9f'fNpb]MJrؒKV &Z|EVskl܎@yYD?3"+X~|^ev=[y{O7Ʈ_ yz ƚ}b%,ɻzh4^]] ױeн'l3Q;\( }wJhZX9#'p:(l9Vv#-Y=.Z%$S۲˒?hbs<+G/fNds1ez~k }7ĉV'ız'Xەw&urb@xcci{x0Y8-IJ;CbdnN ssPm~i7N QIس˓S ӪOmvšG1qۉo7^dc(x#[~okoYǙ[V<_;91]gU9pulǜ=A<ͪ&g񇠅Ly'A~ԓK26}Zj׬՜j OEψ⬅L T'6'J(Ŏm'TܖU'DG'Wg qXbO!ўEtnDuu:zv#[0CBOfQj}M=7s9⁢i>ޑyjׯZ}Pح(3C>T KQp4N~[r/gF،EqETl> U1a"y|=ѱo–y*,7+_w{w ɇ9[M/:7^ 狻klߑޟjwcE`͎yφxtƛoDyPJ7 .[~1)17WAr^ajUo1꧀*:kߓޟC-^^Bx &$=QMs9/o枴K>eywlOwGa}A< [eg>@809pq)ƻ1U@*[|@ĵ+pj,46͙/Vs> KWl=h];ZMYgvb姜D YfLo]B#Fjl)hbcFr35ݣdE?5 9GNa'1V[vCtץ~jAW~87;%l\g"M\V_«̮^wK_=^LU7Ʈ_ yElݓw?h֛:=GqxV}d7y]"   Ó^.}IENDB`llama7b_autotrans_8to1_predict_ckpt.png000066400000000000000000000133421476650141400355660ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRiGsRGBgAMA a pHYsodwIDATx^4IR?ă^<,^ד/aQd=="uƁO,mvWQ/23, eF䏊짻~t?TfEFFFfFTO@{{ޮļDb"QPPPPPP6Z$f"@"v 9cD p:4'W[05܀}E?Ppq>ٖ?v1*FCբ/~| p)7=v~܋ƼY*%Iq[kI[n5ΧH2@C->] ]S;\$uSWo^Kqn⫗_|8{K]5(S9P?'[v1?|h2yJ$~(=v?:yN$zDfS}VUNnҼOe=2ISjH}-'{":3_n z-gҼܜ=ROm4PU>򕰞b?1,F6x~Z>ZxI/n`cnO3+m߄5Z쎋(b5u ~cvk3R%X tX7K\{mM_El.Bg0A& 5?K|m7_u}-aa\/M|e!$/3| Ovud6uzclh21o=~^0 /N*Pa#K]Ϳ4nM;d9ɪ|ݷl){$rk3 `% )_~(b̄jl᪻t*LC̍t6aoJoOB+뙰*ۄRGZ{b &EiٓA(;w _ҷe"Ԗ|Sҥư|$x{7yleS.'|2kûZF?ˍcmWVv1VfKڤ"HZY=BavfK?i~-6Ղb |2<nsI-k WIҿ'Br`ي`qe_9}[֋ Gk"pFKK^5[(%s+݉iܤt=\|ǢL mo'#jBI`=vʂOA8ҵI|1bLNOzf=sAUcX_Ys>ilvmZ/U}ǹe|'{-;GqAv!?gShlkINi#&Rw ̧ly7JJE[/}`>& M}n~@)tJw<9LYGZ{i\qq$#ln]Ϯ'i8GϿOc}_m)L)7];,Hg#= gCWi`Kd^KℙN䏅:,݁{X(ג8F~ W'CX쭱,DǚkU* ;1HA"v 9c9 6e^z@ >n$r:@ k~Oڀ6A"6M >n$r:@ k~Oڀ6A"}S-b/ ^7Z;&3$mDN_+}(L5{=|:$r'8ynt^ xu{OzοXݕc't/ޟSQ9Ж|OkqY5fg>`/~#~6ԟ +v Ÿb7$-^&HuxK02ڧlD-%aqu4y`GEW&<9õ'/?w \ۓ~9NZ?C_u>}E~?qԏ|$}6wFDNͿ8%qicœmn!S*/}M @$E ރ!4I"qj/X'ĉOҵGu]jk)mJYBw剜@F>:'=>~pȉ?\PGГ7Csl}Jײ hJL`ٞ8}#}7ʼݣd[Bt|tC%pZ~kݏꥤP/]~nںV?ZKn->Rw3wwkpD~ t{X0;Ɗ{N,21} 8gǑ= ,@w"W|kH$/R"RO:>&CںN#B闗c䒸{;Ok*X?8;JҿGFqe֮C?.j,e6tx4oEb{ ΀ԓ%:ITyysK1!8|pϹut Ql\#QΩ+\#sbsz@uX?8Nl@v+Ʒ PoKmn$r:X/do-h!6A"6M >n$r:@ k~Oڀ6A"6M >n$r`,?~sRK I_ heDŷ8|o|Qyw'9ƤX FOï`dDNoٔcedꉜ+I~ν`dDND옔<;9}Nױd07r*e)[49T'uL"B>^V@"Oo$9 ?t>P_,p=D>y3/ 7ni heDNI~\3dd,K \1^F"o<'.q+<= S<9kwh̲apzDh; jO!}g ١H AoHtb9K3٦6=׿b$HIWLVAR \>Hq,?|kLvƖY_Wk+㓺41uelGD?Hi98k3D->J ~8p,93MMMl:MlA6ΕcNZoq:έ80'mwkVkХ:P?> dgӽ'[ PRg/IoC}o~2eo*X>j ;fDïo_M|ϲ`dDC|W#Nd'r0DNy,$K}VIņ*}|&ϋp2g5e2Aou2EkjwE \0T@}3FS_c),Ϗr\VNL=['Q/RAmM[̕Y"5D'*GwȮx;'4nk9y}cDp7Xboh=8t% 娝tpxX;|ƾJ:X&8 (ِHuҘL v5YǟFM_%-B#i wf1[n[{OTK_M$cCHIW_LW >Tsq" Go3[f~j"dMJL͉\;.87@R.aD`NY.R]eIDx:eLoME!{l r/> ŤղR^3ޓoiǶ~QH۵ DӷH.WþuS[z}SG2^Z(w'rvx<4p1R߹E76XKZۈvK.R{ynTQN디SbʺBuv}#m0t\-'J5+gOoZïWֽ '_y?aq$?aSݒs?_PORoج*u wb>̓UjmlϠ6w=S}%Ӥz>09wNOGѦVP]=@rs_:WĒX%Vۙs]Bؿpn4nۄ.f;t/Ū4%RhA߲ eg(QOr@:+E6&mr-as5l7W* [S߳v?7_4Kojb2䞎?S^U*?5߿w~/%瘝PkX'}cDN^_Ψǃ[wW"OL?~KڼGn|?Uj|kLe!YSR"XR"`W ;1HA"v 9c|"lSsIENDB`llama7b_autotrans_8to1_predict_log1.png000066400000000000000000003171041476650141400354720ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRU_MsRGBgAMA a pHYsodIDATx^4DZ 1 f[G0&ƣy@Kr@(1Cr$_ @oF~TEfEFDG'̌U>1N"/zB'+zxBh++g#|lqz٧(^޾^_.__ϧO -:^.ƾzyi W:7-se~SoE6cIϠcwտ:aB92Zz>/'~*~; %Oa-|hkI窸LLocmљnc?;Tte{^(E&>b>Ztfz>_)~wgT>lѯٷ3}~Ep^nʪ'PQrtUEp~E:bIh3Hn'5 w.:g{/Ipz >"|lJh a|1/SZgzR=*r#A~,=]šhEo>SH+y__\6ЧĞT s+R7d[KߺT;G$#/4)^,o9>oWmwM^}Ӿ,Zp蓼QtYհM>ֿ$o*FF{7K;ob?$ꗊ SA, 'B|PD`a۩Ev@WZu+;g=c[?Mk/~wG4?%;ve:i~yLǟHu!樗%_'؟-zw? {/}/${L~y6 WѹNM3첛Y p{_>&9:4W1>HO [z/={1D77jܼ{pb5` D78 oQtVEE^[+Vz+N]cO :Lx?}Y> gfXYk<b;]+2.Tc|(:?;?s\{/]gB<>_ MOXmQNf<ܴoغgevxkwv(CStUnb{qsqn/NZMa@?Ë j[[H7 ZOY_ Ɓ1^s[/~k{mQ"xsOD_~]O#`c0ڽFo w]fsI47PN?uYާA#rQn9j[P1gCi 'a<ꮥ[TmOҳ|OmTܪo٦NQdz>ufO~H#[IO/Jr E~O|HR}'Tng${*@߀~Ϲ/}멐íR{Kba6(j.'=~G~l/]v}痍bљW\?$qwܴx:@ܑ^{89]O?<'(: +i;UOgxw6oמ,~h=@x7;szyb}}br}F_.__+"w{=Anr 7n pwyiolOunon| -a2,Oz[C?Wd:ɖwGg}2=EQN79\wן/_vwSǂB*o1 oΔSO\t7m*ҕ\D7\T x*X/S{3ށ:ar ![ku7~<~s}=z52N/'Т)O--\3~-: } Bsgpw|y^,)poܯרoX"ebݼ%_un$2n Ri|X4>8)y:sZgzR=6r#A+z;5[)?`%-Њ^?|dmCE篯F܇yI__l9i|=mh67D~4gѯOl/1B47΢sAx妒 7f|蓼Qts^հM>ֿqT8h '.-ؑS'w:EYCTVO^K|Tā7i|[y]*YOzG7A/:](③3OIcF0L|Cgooor_}_KFW*xE??yWѹaL7mt3ƊI b|&Іp8͵@*-膜n4yTqUh&_r1|(ltOEh텴>K{ } }Ƕ~b0ιWX[tujCi,ů7Or_=݉<zRэ1i<f9 k{3oJ Fq|$.o`ͬ"Oտ^b],z'Ai>.$ZDX]>?whﶯSr'Vb;]+2.4vKȆTl·>鼇}buљ {o^u74sYSsb;;:c<TԢs`aW0r霿ܔE/M6[T[twxNLE΂$O;džk#ݭh#_SrL(/EcqsxӰLm8EL?ﵿy[Ns{]ӗcqw:EF&0&Mez}t\-gaj[[H7 L#?kA2dRQ)$g+Jk *z ,GΟ^3EO}LHy'AWuܠ Y"V]?Ae_@ҏ[Ngw-~5&y\t.7e M^s[翎t#T?hB?Snl> /QHտ=j#OݵE I~n*˹ևGͺEԧx֧._ˢM#GmR J('G;pStNO.>E8\&S={6/i=7ck"/8?-59'I9粣 p7 y&,"W{v{EgpG 4z [j&L}<|4@ܑt&?1F_& `7Ptஜ//Oމz:_/v O<9{ROߗVnNә8р}~wg=`Sܿ7)v61.6>uٹ(x].j6{{oo-=F}}~v=]Ϫ Eg?1~q}`ESH2ӷcom?];C{sb0ǟ^o06qSٌtcW'0S["ee"`y:ċQN79\wן/_vwSނ?a=EDܿ;\О8ߐt%}XvUY<7sS.Nr[I֭o CWDJ`K=!<˚Lc_gY?ET.AO<7_I0vs<]"G;ށ:7zտ&YwSy|O|>ψxQ|䏛G>M~sU\&?7|1 ͝y'6_c5|.JW\5;#5x~}8o_ϽO [j,|uu<6^z}g _qo?gY?_Q(Ag}gWmp;<;luM'^'ɤ't'Ymg<%> EOIǠCm>^LJca]8Z?k苃iOOHL2:>ʓZůg 5>tw kR~`IcЊ^?|dmCE篯FzWJ|Im,wTט[h[Y#U|;'mƟKXod|kҭ Jc2B.#ߔo?Vbw;x7awƳoe'ǧqIs~ }_pςg|=n_{߳/׮vOKl/1B4 -{,:7:YN#]n\'7my'$Nl d6v5>n>D<%p\y@XfCG݉oZdUn-ȟ&^t~Q #Egǟ;p#p[u4}\h^llM|7kS d9쏼_J{Do](ΕluBZ&v?A8Gm}W?`ǧjc~*w^E(eĜ#*_:6ɗۢ?]-V 3[YMjl{dcqĦn?#{Ώsq$.~>㽵g/_;o~?sogi("g <3s X * =nh ,)9%.h1_ J4yTg?|2 /tIcBZBiFd~+snU>w9g>~^t ;><s'AmvL =ѐj{8;GoFkk~% ޮ78}]O3lK؍I烮O't,->w_9Wc<\gx;;s(2#s4]2V;DZ/ٸ[߄ֲƇ/Đ?~&Q_<+7O(m>M@\><[G_?̘E},M@G [oBRb"|5~տ^oS~}7u*7e?l~T6xҫsw|"Ƞhذ~V9R'dqQ\5XK?ZZ==7⛾ {s^?שOK+YfeigѻD9_,Ч#IO7>fnbD> T/ Dz/SƯͿV{垦w<U=N{1^~-e“ ZW?Wu}9?>q#gGǾȏ mĈ}k8+pv4S,eh}R|d"~iQ eWE͌낾 eGzcM^`DSRI}Y OLtBLKI//=>>mM KXEǁ,1k'c7}y{9?!nʷGՐ4^K'Im,k~%zE m.j2> o?SԷ>}|Vcӗc+t)>X9::ZN_^~ώiQw?vߚ}3oҹ;BCU?R{CQ_aIhgFIoo!_l܅IOi|;߫$cH7Ē-lqot MN>*YSM_&UpW+^hMg9|ң?7ߥgΟ^?^;f}^s?e}q}úr5]NY|[m?U6ٿ|kU@ze^ ʍvI:(ImSc}W[oο'4W}Y~Ϗ_UsYۙv1JL+<9sI:q[|x{u/qM }vC?Y>7w<BO~$۠ySڭWq_*v__[?K5"q}Ox@CjqlCxjs :hP|F}Pϔ1oؙ:1տ].t7=$;ߺ'}I~n9#=sql%o֥sQQl|$ܾy* 7Er3Pts|j=oC_Sqdx e|/>w)ruoɷW>ir)imR~cG1˾`k_]_oͿZ<~>s&baK p! ~|47}kQ@=߂Ui}RR+,y >=Ogm.>qV_X?3# =-.jlniDHO:}}!~\FyMOѳ/q=-~@EgpGN콲5?)q17pU+G'G]? 6_K?=πvEgrk]s7-/xoޑP<7IK?oo)ϟ-N/qp 'g E!~p<7- 4~n9/K(5Y=xv{<+xsl[t>m9HJ{b<];eB{s4ǟ^ox6f 4 oNz|["ne"`y:ċQN79\wן/_vwSނQa=J7B'nt.+Of^/~M_3֯or4{/u[#>1,z<Q;e{sOt}f N("/"!vs<М"G;ށ:7r ![ku7~<~qv=q<#Ea?nzB4גUq ^?ڢ30~*4w9|aў3cnn2~}?pW`|8G#G~_۽u,5 `_Q$S_[tZE?fpx3z.$2.\VɯYt-݋[f]!|=EFٷxuIh J0>ql/uJ?=A2(O)FTleɊtǠ~چ__?TO="&Yo2ٯ1%%F%vO`fI{CV_ee/F_GK mwg+`B?+o{oZVlp~xΟΛ˾^2]WW_/c ~؄?-D,U hf3iͲSӸj9:7B_%?1\Y _ϟ;/,:7'l$M N:nmM~[o/j£!dvlƎ?-g|Gxq|c. A@ l(c;] zw@4-=v}[~l 'Ǡ?_BHљ{#>(P3Mq}x+7Z3[/u|_? tZ![>=rNﲿ Pspx7;s9m~{}}&9G?UtlB/E}K*[~VoƇH6ɖWO_?kڅr+g߆}3ϮM'ypƿ}*{kkUqY._x Wѹ>vRNtB m) 0MCVkA7\tu9ϣo_|>L &59Kq:1Z{wA/)mHVoQtڢ.ǯS݋΁p#xsZ^c|ay]q@Go?k ۿqL:gch?V;/+1u|?s9kyWO9ܮ~ٗl\oBk\CtbHcMd}@YtӞTGߢ|;E|OE^[`IwTOY.0LoUvG7>̋>_ZxkEmlxAE/"o*:SY*:B G_5kum5ҾzL#_#IOwĈ|"[a^[$_:e_l?__ ʣM_x:cS{Н cKz[˄' v~LO?;ƍ4}:;8k~>ܿXfuSOj=|t wMV&|? `EE8n}Vc]EzBi&E~v,s@tY1q훾[蟏q?=[.+O2:7.i,=v77'ڿu|Ls/ 9[ۣM_^OBucl?Đoh;]K|yE<+ώiQw?vߚ}3oҹ>;BCU?R{CQ=c`StD^~\T;K1>/GΨIC?#] WB:8/KzJ3݋>{}6:}?E<.jn^}tEO?w>moח,s97'*T6ߟt"}Dp̟)':% ᦐRd[H6n^Iv|>[q\U0[c;}6pIO"5|SׯimPN<~'$w2OE蜞\}qXMmk>,׷oNqi8Y<(:[^) >0'g?@xzE%weяȴ<17-se~SoE6cI(I?T+lN>+>[C,:85r~^[1շrOnﮟ"ھvx?on/>?x񊞠\T x*XBG;ށ:`z5ɺP<^88p}*PVt|ӓak __oe)J/nVt#k*:}P=0Ի"v_lM,6u#}KlA1_kۯ7}VF6Qɷm~HG>-A,:žv^\˯׿S:{nM!p|_ʨѽ9nhOίDMR񠚫a|b{E^pHE)]$[#q$'ͦY8."B:0N,ZNH~׺%żpˏmA7VgR():8tQHڿNwM0_)$^|M|eXk?_9쏼_J{Do/iȉ\Ɇ^^οq)47E_pQ ׯ?u,:B2S2G5 |<Ut.7ӱxO7fqOnY<4qvxٮ}i4yTg?\!PB?OEh텴>K=>Ӿcz_AssnU>w9g>~ڞ$DžIzA?رm߰eHAv5j=蝣WQιME__~=rǢnE'psnIi]6E^[+Vzl+=}ؓx*ޓ?l_;EgxҫS4w|"ȸҫ hR 7a,}y}f{/ kzźGl߈o"0y^?-e? CKQs7E P&uW  oh^aaM~-:;F''oA?Սg5ޭ,_O>O:֤oڎQt}>|(:?;9]^cc|8Gӗcԯլ_h__=tLjpg/: +tVq|;d7}S.z?]>wNv81rZVEab_SQ>{}6:L.:7:0}T^Y_&UpW+^hMg9|]t|^.:zczFjs?ɿ>?#čNg޷ڭ-V跋J|U[j#,o{UIdZѹ}s?<妾1=_o1>:X̟)7q L}ZV>BSy0?U+t;鹜ێg+q}.[([l|>vYc/ n~x^K&\g؏_%ɸLkU<1aoGh%Z]XԵlTb-N/o6>U%rA܈ɏ(:m˓ \f.Q9}"\hsb`хêIed󺬢3|w43~Dbnnr^ 7NܸO/Fxex#Rd3o-} uGUn&[Fo+>[C,J8r~^[1շrOnﮟ"ھM~sU\&?7|1 ͝y)/V*|7G߱=^Yc|)-<Q%;fTkp8PlXtf'cg<|Sfh9/YtN7iA增io<+-OίDMe5W6Z$o߶h"٢7@[ ?{/Vx~M7a sl w'$IkݒϫSnwWzǶ қ?k| z_E)iJw(^_y&GN/y&?ز? }}c}ʯ}Ηs#+K7gљi|ikq.͈r/^ Yh 7(C,?KMMVkA7tu9ϣo_|>L &b|y,ǧh텴>K=>Ӿcz_AssnU>w9g>~ڞ$DžIQA?رU툤d?ΏVio7be3`;'_G짾|v̊g}O)\oozӥ>E籂s.'= .:׺oǢQ:5 ެ$w_>ӨL'{k̘E':*tn -ʓDӺlE?Ƥ+VzR.=}ؓf*'8ƇNQ5,b;]+2.4vKȆTl·>>Z3k=xS5y+V?+f/J9)}54c__ Zw<xvJ1Eg6ha Ze^Zt][osaO)Pӭe'u z4٭;+Nu)./[V?//ofUp.to*$|JXE&0jʉ馒s `cWoz޻U\>>mM KXEǁәu1>Wns4}y{9Fw?ko?sc|?[6k1@mؾ[?5?zPt#LЏ/˱Z;GQ7'nTtnkjbIzcE8ZlĭǕ'o{4jƈ}c~tXƈ㳯;𔛜^~[7fog}fZbwq뽸}/֖˟  E~W|HR}'Tng${*@Z[Zƿ^5k/Sl"~~$<wiW?si>'L}J?i&?c-y:_<_ go\Gy񔦗FrzH~a.v.ڙ£gY-za9!''JA3oUQ'rn4b\me/yhvJ_~ˢx5Z\?$qiF**ݍRD~t nڿo\Ⱥ[QAMnVt#rOԱ.HOEö_WE4'/BJyڢe<ϟ*funQД RA@뇈g= 9.[u5 =EOaKm E{w(- ~nC.BYLvų>v2"syJ/޸6zfP̭SЌ}oB[?Šf_B^Řg_#@EgpGNדgE+ _jyrMَ[Rti2Hin9y},umvA3k?g=[ZkȠ [MsQ뷪(e}΃EOYZK f wo2uO胢3wt}y9m:CIT٥ ޝ 6+J)H +`n㘢܆/EgGѻ*jY4ol^O8ҾijZψ6謯m'Z#@Eg>E6>~OΏୂ}#z/H. uQtv]RxtKxŞ S'Xł&YR\U<]۾zaϭΏFM| 7Yӻ+zۛ/se~BKo yE6cIĦ(ա+ dH!ZﯯO)γ_//Lo^Q$?8yo5_&?cs3˿}wǯP!3Ub~Si(ŷ} ϋN."2EIQH8TzxqY?D[O2o\f}Vui=JljL/VސJ$$k/Zx}|P[.xc^ T`pƾLxOWzQտ&YwSz= s_~])7}NX|=r8L?xG_K*>Web*({ksgܙ_^W>~ ,H)!O''K"[Y3b+XYA-T[LcṆy5_Q|\XTn[ 7gGsUk7v?I& hZi/X!o)gax')0im>^>ǂIn^`+T)_Μ1 ~^TA+z;^)?`%-Њ^?|dmCE篯F܇yI__l9uT޶ߑ0v>Š[b;x:ӭtEt;\7f|蓼)xΤjm7mQ mEEˏŚF>++eRب$ꗾ4u XB|h,9Snib);ȏmA7}tgR():8t;IC~aEN~‹*{.VM+OE3u?K=>Ӿc[^_VE|-:Or|:4{Wǿk{^П/,|ݒ?> ?̢sf˼iǷ(ObO/]>K4vE$3Ǟ_!z/U yƇ׏5VtVs)JƓ^Že_lxAƮyR}MXEo*6KECt>`L_t7O`dNo's\Jn/A畛} OZt7RQUp.tnd|rջ&_;<'n靝+K7B OX#~;n+kkЧ`'ش@0} |8Pt;uEgj}e~k7 ov_O?s~Ǥr{tOto~2>w&3V9ܠտt|QXgڢZYߩ@t^ۑ"G^ce}Y"mM #X¿]tޙp[ %nmeߵV1,!E}yj5ѻn뽷:r9t3p6gt#W?Sn> /Qտ= Ёܿ'ZX) tYI47CԏߛY_Oцdz>uуf"\lo?-E)Zz}WN|'$w2OE蜞\}qXM-9n7h/tl_G<{oS,Gx{v3F?Q{G糷:8)nJVtzL(:;"_.щeo-Nu[z 2;|G׏bU^w^E_(EgVl:BnOP'\)d)>BdOվ3 =^1W^΁SRmT ボk\`MQ,ʦ"뢚r*FW..7hH;QoWIW簦^?q-__N26Gۂ _n/֧KOOw\)ĩ_eeh΢_^tc,KO:7O/T+J\ȢY۞bT{\$ͪ1h E繀"ajE*n~[ ϱ T_M] Û#1=5H_$(7Y|@2_Fg}пO/_p'5w:O /+ziIE8ƥc,Y +<$,ͅPn4f&% .{EMſ]T ]Ӻ5`} fƊΖ}~DLر zҙ?99__L^.E+^VlkJ_p~k d/SB(J\E^QS/-\b+./=:Eg|<7VtVw vwgyc<8̞(F,V)\s)Kep~%GTH\R. ȟrBQ0,.OAOj ~gYG~Yϣ;a/^IDZ!X.zb_NZi}DS~<-̟mt|pӸ/yN^@(-ֿ38e?_m,_u1EۯO[t}'ٱO[c"Nz9O_tsatN^Sa*-Y˜ZA{kAeN]m'T[ok##َ)@׏|/Hn@;^_t#mQxT1ss'ќޝ¬i_~vpr=EOPr}j"|ţArfTntk,^냢38}΋: L?Hw6މ  |Ģz\ L췖]T+}ݵG*Hm`WĢBqqqo(EgCcT\ۭC9/;{t=bۑUtOk]\;'plr_RdnQ6wWMhX_t&ٗ9KfJN~:bs=Qɟ_iG>8"$'w_uOjoڃ&X; /<Ɋo yTkSјůTpD. Vtn&D?,sSʅ\X6j/ד*WttwxQП/,|ݒ?> דi׹JOyt)YEN+Il-cuyr++.$:Egx6no's\Jn/A{c8~x8ܾ~ TdDhh8RYLw:~ o]yS܏sB~ͷ]Wn !X.ʕbgۑsLт^S~<9mt\.X_t "wv(s^ZI9N` @_tn sz5Fz-A>OE9GQ~q$TKF^?,o惊^Zyxy~ѹOXAn?A?o[^ XhO Oެ% pq:4j7(: vEgrj~qQNKADvtf?k~wON痵<nlA?/O/m뷉np;E:st񎃢-@)?Xv[t>i好q8]ϗDShon, a2.7A8hAVꄱo ɖj}ÍeeA@t}zyYȷb%6˿|\wן/_vwS'{b5俶HE{~(oU- ^b[zGzuI?X s(P\Nj恞`㩠zoƟI7LKm_x_ 7?ћ K~])߇r}pB|=3Xo>dN!V|hkI窸LLocmљnc?;wJ,9Ι٭([9J;Gy??[~5*ͱȧXQxpULh.{uMʼn^'ɤpb4lZ!o)j?:=N fMo<|1B~lBiLOIN2:>ӓa-5&Ak|2bz@+:{ ~zEj(_j#b{>Tnd\#x?m~bcj%>s#Wɦ2$ߩՈe_l/cڗ,ǭ?3΢sx墕 f|蓼)xΤjm7I|@7fmюEEˏEF(@(ehM?TN7Ŋ_q|?c|T謓niSnwW_ۂIo\&E~Pq|_ m_Œ_үߔ?OZGy~?u~-:5"[?EO~~GO.4^(n @am|CwA{p^UvZ dpiƛ I~SQ+7Mgv}hʹ~ߤ{.MMa)19i^HMxOoؖOlF9*kӻ3Nm(:Sϔ?㓈mS޼=2#%x䇆} '?=+gOPfS>;eBa+v\>J֧>opйǷOjrᎰ/; |[%KѵȢ(kt_n6<]_vK??|~6Xy:7Wٲ/se nIR 7a,}ysҌ^o?BA3mUF7K?Ng8?[M_x:a'墪>FxFh/տt|qc/ٻ to.N7I煽-~VDd}\ҩ)&H7OپKk *z ,G뱋Ο^3EO}LHy'Aλuܠtl\:^rik򏭿/C;wooٯ{lg,ٿ<:;mYEg>caֿsI~@;x/ErP_ҍknk1 I;3¼=#\01j[Г Z膣 lltqBw#ine7s/! g}BZMc#iemz/ #IESQ):'Csc.V}Kr[ Ϗ,c\<4VPJisԦ[OXi3͒}l'&5cUk/m]kΪ}^p .LqQ 4hn(O?./ܻ'|)i xg7{Kw?xC `7Ptஜ//=55#ޝ <y EHv=[[<>{&{v.<8Z(VOnU\3?#@|'=^/ 6텊t7q]'(/D*0vs<Tח_/0g*ؽ^.AqoIԇhrߕBo Z}b5><.F[ {su7tQY 5*A\beH8]gGswuMe8$ V~NjV|H[ʯ&%>NO`üٷxr v'y}Ob :ImjCI~A_Z f=]BLhEo>SH're{[hۋl75~~leΟ'<dK?R@s[oרu\U+wT|5cr~Q8]?K~9ˋqAULj?-~,b{`aa_#}e7d-_7!~\k}00}d|7o9Z/s3*ϥsΧsM9"i/t;0cϚ'?%so39j_%/O`q24^Y|h2y7>eyhowJ#K#;dx~/=dx#Ckǥ1c16tFߢ|R̊E^[`f_/7SE)>|웬QorsZfL.Ƈ;VtþN5t,b;]+2. JclX Mf|3Tqt_eV|Xk/"0dJc#F|N簯_INx+)>6DYߨ_3v1XKGct#ii_qC/IVCW,8ekKbEcanlϫԯw\åW>ag݅S7[]r$]4 Ka%y9C_Z'SǗr c0ck~ѷM_^_uXď˿}^~ gh58.k{vuOkci~FFն}s1]{X 6//Y3ްK~87#dTm~XF?.s1]{xoN9G ~' "Jr…4m߉3ݴgbqlNIoo!_$IOi|FMNkg}k~yaoG~'yc|Z>j?菬/KI'v!!41#W¿믟nAIZ?[w=3 !~q̯ۗj][0>ߢ;g|$#~&l%OKUms/#Kzh¯/z~=CUs L:(ۤ?/Ӆ3_uL yCԉl}foyxg %6IG66οX$0*inOe';1x֧>d˿E?SN:.hmk믟N<'$w2OE蜞\}qXMmk릵s_w}]-]ߞZ_˿:n,Tx<3࿎}~?@~:[5sq?lؚ;|-:M_E;ć?N1X[Oo` o>2Mi?b n=*g뻾ɭ]X+CkC|,(hsv}FW?ǪM!A#$'CHE 7C`Wv~[ }[HOIKmxwПN=*>9[F=_U<l3#iOZi|~'Fe3<-x ڀtޑVււ3ݰ'pE&~ɹroN0AO;׊~+9:?|~A~uH'c4~јEgz޵Ei32 6\/):C~xڔ_-۱dK #͎{חGsc E]jC wzP_|*q8}:7볤3[gó1Zy8/OCq_b`}\'f7=MkqܨL` v/&W:)!>)yMĦ㣰MAYM מ !/~SOdv~>@%kwr}|{{N9],Oюݴхl tJOu p=?\Ƕ|S.sm.J5HEPqn"dz ߡE"0dcM:7Ɩv%>L6~܌aU~ɶ:n '6A六KES݀U.["APAOm=Պπ'qVҿ_oq379&}i}-m?U#~7ϵ?_5? ?t\XҮɧ6i-^S yA s/Xx1;w qEK zQӼb|m"V?B>kErVKZ[^k$r 훑p׵/Z^4ח ǽݖ-Ȧ8ٿ\?`qpY/E~OJ ɐt#d6^0 s|C ,IkS[]Z|X򵱌y7b$}olZ73] EFȞ3BbHa=?:p|w9emB2<)5ʡeRKOQ\Dh=?mC}9L?*VoW6o:޳!5"c*>hr,,YOs}kqtck4_p~ ڦ/?Ln7?]$_ mQ_1/vYSZbG˒1g?fc(j?bncox+6dCrKuxљqd Eg؇;uErQ۳rQݻPٷԝ*?*c gU9ڷb\XJ>7 j2bl8^#O(\7}ֳk[V׿]HG_ZNݒi[O߽<ۯ/97 ~b.s`gs~7M_^Is]M1N NV}~X|LSm|@m$T_ mP uԴO2Ks So.zviQ(+ 50O6Ư_ z:_QW}gHo]8emW;Q&\EEVw)wqo_y)󕛘e7i^y]ѹ2}wџON U@[ XnN2CHYWS[SKzM: |Km VHKvK>Slѯ>^|Cr_qAϷ߼P;u7oڙ'w=eX۠$agnAskWg/>mG~<_?P|zTJrmS ŮX;U ¡œVa6,I[ q8ŶsYG̟U!ڝij:׊w|=Z&]r,e:;La#bA:ﺘrk# 7$ʛv-;Hǣ|\) ^. _$=X|30 o|W3eg][Ks>/oo~?S@/{Ih)>3c^*qo>ɾ({v4J|-<:Ow.׹D.mbo]}K>Oca ߦ3-,INn|=?> ^RlW,:ϩ8'xNbZtr}_- {2'*fJ,_4YIr3k]f > a:MmЋDp/#`0nRGb=%sDlLb^#%eaϗv[k.f;67Dh{h+QEg)耰^3sҳY{ـ3t]UAI9*:6Em.H] 8Y) {%f?,tKĢ+w*–"<4q@O,:~|}zDYo/~Iw 7Eg)槣!qN x b\0?~\^ׅuEBKg),KT?^R[PcC#鎢3@r N/a hA&p7]-\[t~_[(E#ȅ~==io+{Jүp˂<@GoȢxKLnu g"8hl;[y}EۋW!5;> ǏS_OT3P݂k!(i VPP#mH. %ZJI^lk =BѹmP[ EDp,F>o=meb>ۜvkEU*EXL[Pvtx?ZT9]gGsUk7S{ƾ?BO>at'(,S;5=eMfo7Xi0>alNi}˓:>ӓ7u=7lIъ^?|dmCE篯FzWJ|Q WbSW?| \ߥ$=&wzi6q_ןD'OYOC?3_O7o[_vU?G;̛"#'K=F*NY!UkEE1 ŏv\^|U|%O"o[[N+C]^'=ZM蜊XSϺ_Ќo}upmT ͵ͅ$,Ra BO>vt7|# #UQf8>w&Ο16.+x^XX۩@Eىi_IkżpˏmA7?kыΟw/JxLSҽcG!ӻ>֟#̟G7~_ώ_k)a_.mX.??3YQQt_s={l a󿮟plg>czT/LhO(FE\/ZGrxJA_Ksu4o mSz/cu5lNQZ(q tlf7x*Xp;ݡSq]^ ݔs_ ziz:Z:ȦI ?V}v04o.ǧtIcBBiF-~+snU>w9g>~Dٱ><sSjDݺQ􈬐__Doq*ԩ"`+>eQWL+:3_6V-V +q*>g>'״_o&`sVL6'>az-֓͟}^_AOɯFzE~xY0sa7)- }#tn#oo\Ζ|?a-S"uhE/ 1v+: O:7/P7L_Zm(:>>{-룮wc)kvL$="~Nݢ]?mGG /KEݟtV v-`_k*Y2y%ڢfZ⬻VTQgzy/UsO\ X(<)>VǶ\~3y$CZ[Qش;d^ޣ{*7ɥn/ӫs~'@QP?#'QnWB:(,KzVDG~⟂Zܑ}Fnwџ&V J:KE&BSTJeYWt׺ zU Y.c?7ߥgΟ^?^;EOo?Z'rEzC_>/< w2"^m+çO"NN:vK?SB ܆\H4 r& Vt.\IӬ[^\t-e[1/(gѹ|amBsG.:(Ec|*t7eL1oH_^-yzx)'$w2OE蜞\}qXM 1&^6>1sA1?osCQ}u".ֿK=ɆrzX>ߛпzE,~oE;?ρk[@q]P zWxMSt1+? ^Qq nq_$ =GOYKmgEgpGNm^t>ѓs-Ne]k4~ܵAcxN @;C7{Pz!}NspMr:_m?VtzyٔXW9(ۃcC?{7` P!\w;No{G-{?Ͼ>[q`f-c߷?UO}ֹx{Eg90?/tǿIN[`_-q_~t퇹 :ӷؗXhK{ҍY?\]^pS)LycA;tO tQX_O6ٱ=tx{dۘ+}cUHN l(VnM|Y, k.a/W[l]S{o_ձ/i#âk}Asɡ#B|:o| ĽyG>ÖEh)&ϊ,]=ӹ: C[߻kmw{?yܕUEg*V̯_ (믦~Bv[4ER|^  Eo/ &^} itw&]/nCb_8.ɡ/&Yx?N$F?$ &҉֏d2~b4D>׾߷__ |!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-)El[?qw3(gޭ5v．>Vy1^߷ȶe]v7kN<7[o/ws-~yܕUEEq9,'ۢ3 Tn *;o3nOEtQnc!n>l}|8@ݔ/ ŧC5&==K}K/_p/k|CLҵv>c~@>Pa,|]r>Fh~'Vn&R{3w?CŰ؇]`I.k։&ī1a}*JknW[m{=z~R]V>\?揱'oUk~V =K]߆'n+jc{dПǽϔBL>Hc2>H/ƓŁ|SHQϟ~-jpԯΨǘSK,*'hy[-mJ|_KI.XA[ae1_5x0Vϓ?b䯕/o,h${ne,om}4#JuG[k ;϶̆aaٿU>xjVqzB9;nm8'NOJq^tO`Eo/DJE'zLI%$x/ P$tn˛6}/ik+O&ͷ.bi)?)DEo-KM6j3VAo]s+BprNt}O?OxA$#s y.r"n$/cOYy@klޟՎoOap~kn\ k[$h]ٿl]?Xo-D"&6/l_ҧO9L%dE~T\GBy~7W[e91ƛ_]B>6>O?9O%WuxgtII{J|[Y+י/{S'^q:Ov(8$~vu o5M?ٰ?,z%<%S7uTm M[, bw:ғJ:b{㛢s,g=䧟_HL }K.6YEJr *ٌ6tn,5_ɲ`zB,l*iwZbquD_;ud*wk'.|v~Xi~#'cm&4>_WX̋V9/>鬽rClSQ9~X6) W)IP1D8>3Y~  ^B\2xC=>ӟ^:u?ݢo]V/O~S_KΉ| /i E;=?-vOo|wkmտ^%`XN$˱n9Xo.7tZ{+S'rL9[=&#O+ :8g~u} +jpOkw$VWus";1՟ͻo:#olo|Hw{q72o*jS 1g^8O3w6E_(V+R3cl%iÿ@SRdy=kݤ`6|)I"/l 7U>^֟ZWn_ NkHno m~XBY7f>tkY?珌co~jM׏c5[BǿA[3~=0Vto|mGЧjբ[x~v.?}I~מZh6c~z)]hﳯzGd, }ߖ~Ԯů>3K/w6DW|5~\oxۺ>5wXK>xJVS!w*'˻](B謷OXUtǧ'Cc᱊I6l-9= ֽ)KW|.}eLOk;Pq?SN_+w?_{g;Lf0Ur1~,1 ߮տ-k}# e?#$kOC7X^9 @?_ioWOFȏJ7'#罌_mO'[;#O/Ho9i~#>eg;z)'#qiot>d_7_xGGu}"4[M/į4^G_|uL1#űw~ v(8󓦟MgtoZK{ۺ>xE :aѳ/)YYtO2WO_H}+wڣE{j+Eou?;>by)jt_,$N⤥lDsQ:Q}ʆ7<|oJۓG~3zs5Ȯg̰H'jO?-⿰.$N)iozfK!h!w."[,y|zX3gex?etZV>zom|7sQ?[Y? sk:2l-b;~/v_5dW~=jSQ˨cMao?~a?YDw2D_$xD́8=*"]XidίԆa!ڿ||.:/ pK{8\ fM{O|&ݔܘa GEo_p+ (:o(wWxhWo?]tri|o|^ܴmrs^?={W?,JzBcjПmi C_[q ,(: }ݱ{{ߴ?rXG7!.[r|MuG~9:oPxWgo~f~&ٝ~]h;]?$#7~?p(C5`=sˋqZlN˟汣Qt>{qt?o=V^|O6oj/~wE[蘈ot{O܄~*?L8[[sha<ߏ{ςiߝ汣|eY+{W`GQ=?!_?c9z_ .>ucIuh|gV׍얿,}+b79z%QϽ{|>nu<%Jt}>_AOꝛʲ)CPw+&R8N3'6Ԟ^j6!'kLԗϵNACj'<{-znOzv~m{EFrF_2v;>>\?)|r._/i 'ϯϭaϰb{+ț|o"W_V?e,V<7ڴ?-MOzGv+G?G9#[8ɞڷg~ ױq7ο6'b]׷%ߊ !EF,v*ķ\c/Eʿx}F|l6pu|Z4~y}Zt>]?8__|ݴ寯TߴEО.+]O/XOoao{kWy~sMmXFc+ݢ%I Ĕ _ϣrP΁NJ!b'^†=OAM6e}Q "Y[?k&J)rH$ZNQ셓$#[ *gld8sqo"ϯws|3]M|4&):WoQW_8^o]{_c-Wœ 5=/Ƕ0so/ײƾE|<]vb8]m]A+Fև::w0X1LNφ7Or x97ώI"CS8ڿt\[߭Q?3="~͸>e,7)㽐VG:o?m֟и#W >Af5񷚿[tBw1񽢳6~~FmO~Sߕu/xET߇irQ~O^~%Rab|D9Gc$;vaOa/+76 ? ="~}R_vlZuA:aDz~S/39&ɷϽPS0 b^NwmQgzE{O!8.7kg\{c]<|`Ai3 Ŧ>F̽/[5?xMo?0YEoRuss6Slϣψ``t%oϭܟx~+.Stqݵo#wFjkfl6G,Tdp 0Se37|PoMo}%Uw6_rf~:7ֳk[V׿],~ft6וkꭑwT>!6YeH5݇cw^|y0^CjVT/!uvt5cT;{Ng:sh܈#h]Kkޡ#HO 7;y6l۲M4~dp~[uIg'=׶hyersы bWŎϛޟ`C?JƇ1&.{ Sy8>WIfaG!7ov|PylW|(Yܟy]K"2ni}v?Y* 'Te_O̶[}22O[u!5~ߔo}3o_P[_}Z|Y n~>m;sѷީ^Ut'mk|n9Gz㿷x+-<3WKtp&y gu%OhXGſQggZ~[۟oʷsC0]x?o}Qڟ|C}ng?j<P88W^GI^1_翙}o ~Y}]?_3Z\`|%?m?Zr׿JlY߉%hk<-J@HlHjK]P.zeEF|zƃT׿ @aWϧ#!PT|'߈<":͓x{tz~R/x`Pto+p_k5ҟHtEBO4CҾ>Ay83O 3Ћ'zҨ u޻yNח6xRv~)?❕+yӸG_gXOl?~8]N 䯱]u'>ɩH&y{.c (Eg3rKÝNWj7iju@jۿw/I!\w;);_eiΆrҲ'toܼ{ysKX=V [W7(Eg uo{uYGt?d#XAщ{}=+9(5kk;?_A短rx#*:S*f~\J@UЍE]5$ۢqG(BX Kh/z~V~a'&<.__I3o}.]hM7nnl/8' vp2MZH6]l^/(MEvOk_zۯ/V.|!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-)?lgv݌tmͿs^m+Ażpv[@e_͛_ _[?t}Vn?VȲ\|^oά/o#7\tS) ^dzBXqLsLH0bV ]׬M,WcTuVo=Ij?).l+dڟ[X{*5[?+Y텞%oCǿo~R~׿4G'?Sf@W+?_baŁ|SHQϟ~-jpԯΨØSK,*'hy[-mJ|_KI:R69W5~$J2JfPׁ6^?K.~\BQ[DZ[4[sb795}m|c,rhKZC=? IBE9}pű׸"7u}:ܾ+~VZobC6p2?x,XG8?/ŏ>NE;{eblOƏk<UE穀 *6OWݦ-w1;KIhp1νM9ѳϊ/$dJɈN?(*ItNlFJcۛRJzӍFI̒tžlHIĠoddSOq:Ao 74ccW7ue]v?%f"^W>c+Uic AsşWeA5mݿ5nVOa-wS|9oOFrhKgNm18^ں{J_;ud*wk'.|v~Xi~#'cm&t|$kmg~>3 32_]BMC#LJk?} svF|҅je~c[kp9?gH_sߤ~x/BT$nSQeR,E ݎt^BL=x{)h+ U+TKɺc<# p|fз t13*NxQ!̿e\]33yE-JUљ 5E 9x7KpbP9߃Ӳa@? o`qM⻍K׉dw9?#fܿJټcEgE}O7!?_2x"AOszL(G&M7^V%;tq|GGVn zXa_>UYc ko['7#ŏ$gm7b&p_?HvuGKoo$sKD:{3~ʱ-I􎏬˾fx1α@ޙLOSјxEazs4syg\Պi?qo\/*:3+yR wpMLI un]ӷ|)K"/l FU>^֟ZW.F_ .:א*o|Tֲb Ue6[k,5 ?|b°%-l݊Bi-OTmpZZvkOeG#rs=]qf_>%>1U}0^_]kI.>7c}\}Ttc6:׏ o/y|ӹ>]˱Js}2]abx\دb?g~!+Ω;v.py!xQtۧ_*:ӓءO1XEM` $ i93wKps>p~Ǔ+jMrg}&')b}לF#}'\C;?8.e7]ƇѯœeeODˈ Or9Ӛf}joH"Gh gbX˂dw|ZV:7{r\]yٿaC6hqX^K: @lCY''ȏJ7'#LԳ76nStHS?+%a?ݷ_]߂4Kǥѥ,ۓ}Y0S/dp2OW1ٍ./oq?!ž#$oe>>wsY.ý~^M&U_'پ^XmeO=8[ge9?\=9}F"ޯi2mSԁćo~q擦]}sH6 $rɈ'2VK\>tD?ϬK'0=l>9CE*,򗵿/}~{lx±xc.:/ J\?opRI f{O|&ޘnIxvAAXÁ1ڂ-ߋG[{@< p \y4$C-,@ҥX g㉯}SEE6M99y+ PpON ~M=}#6 Y6X/;[`(: |' B[t}y [l'e뗝# g??߳:=]h=Qxo0v_;1w@巻\?7NgeB_}L;⯬nA˦*e{pGso,տS9pIt;uZj{+@7ouQw/AVX˗4]3_e:Ʈw+ȣlz$+?:qoѿOǠE?_ _[([OvM7VnkgxUEg*V̯_ (믦~Bv[4ER|^  Eo/ ^dz}}Ek8Rp&חR p^8z=v(2N/gC B?$ dOQro#ty4F?}}陿o,[[>W}ݿҟmN[_u~k-I~![kuWYo7%;SN;մ!k]oN9ϿG b^c -/mlMp>-NNqqLa{o=m&pAXUt^#pyyѿ-:|pYOE2xѓ aF71qe󞈥X |>>ܬVnʗӡod[_ڷϞEQ7ēco]Ϙ_:u?_{ܿO ܉]P1,a$Ie|:Ăx5&O_s{^$e}Wϭcfp~kx[fuBO}׷_7[Yoo ~Dqg)3Ly GyJx*${*8POOcJ)?3ٯ_-VP|U|y~ji_E´d;M>o+4_߽MQk#V矔c1?kYkH~d(DX͑?EƯm]=rd[ =%H9tђ7CY^FR KmZ_5)s:Os<ߩOBt}O?OxA$Ks y.`r!> __7rԟX>8?G"Ɵ!ݚ'~I^7텕2 ٺw>ʸ_ 9q|ퟒJ*?6TOߎ_G=u~*)͌1 I?}u\}$ ork~u} Z?X<іz~օr/cqEOotڹ}U{OW*(?ņme(~$s~sѮ2'#=>7\ tJ?m,׷;;b?S?,h滭b-v_ ٿv s?gOed{:8~0-ڳY~ֶo/mݵCrFo']'/!㴋!䮝5VIx7XI)ɯpۙ_fLÌoWߣЧjPsgeq1>js{D;Oc|*Cs2X~[5;vuE\7^K;Tf}窽yx*:'Wn(QOE1CJ=oJ:wq7IuxF̠odbf?]_0\]33yE-JǃE 9x6KpџFRPԲ9߃Ӳa@? o`qM⻍K}7c:.Ǻg!0>#~(~LbЧ/[j_/ٯlIg>Mi~gұEGKDYy韞x|+g¶ux]ÎmO_[3^i?Oʢs*Npy3 \^^$vSt 6_+w?_{g;Lf0Ur1~ёxO˿2>A_ҥ?*ʸZҹ$g0?Re}_Of"?*ތj|ʟ2QmO'[;#O‘Fu߂s~u} F|:/v,RO$jrF lOe}uNZu>v_dg7~C]OGZ[<ۗ?>"uwG/`_C<5tu2!g>KߚX'tYYtO2WO_H}+wڣE{j+Eou?;>by)jt_,$E"6C2≫}<s3kR¯sГhX#NK{83ft'+jO?-⿰&$N)iozfK!Ȳ'B.Z#۵Ynog4;61"ke} ooqg_+砣~ Im?:4??YWCklYE?,_u%W˟\FkC;1h cɲ'j]ח!ħg"9x}$bA'?G\O: v~- ̯ܿu~_ԮtH۶AhYCXo/<޹k3Y>/Llv|mx‘?x0V%O{8ޏ tzK='~nJoL$< Xڂ-ߋG[{9opzO6F1囖mJcNMvxS/]%}ۏ/x67^@Qm^ߔAg^7 3xg?p?;N@~& }@):=9]O(x7qZgR;x`EeI};\>3 퇂3oR PA9J!=8֛@pKg]E/:IÂy9]_^IewïONOZO?~8]N 䯱]u'>)׾x{.cC_S?O6"?}tqɾzk9.+:ӟ/_,zۅgm?%uœ^^6xW9(ۃcC?{7`ş) Vz3 @{ʏ(:O[}},V5kkl"G .r Ҷ7]t@Y)Ey?ۿ?׿~~$'˭dćp8/?V%O+&pOioXCqOd}y-c_Me0]<7KLGzEvl=AImL>);_?yiΆrҲ'tokc;ğ]_,ٺ§ߘ?E߽,:o[c/_~8t}G??.kA7&In>t?d#XAщ{}=+9(5kk;G/wPt~g}k?gݡEd^ѹT~3|nWhTPTXWSOb!-W}q)>/tqWh/zb2}c5\Qq&]/Kp,ZSq o;_MeDc|rNɞC_*\'F|Kr>׾߷__J|!k}cϕ7Ɵx#ho>'bۣӮߖ?G_K߼mHVdݕx n-):ړX_w77][o'qs}1}%6  _[?qK%%Vn?;l8ຢszz mCx`$O&s4p= LHWV1Kt B&[+*/[Q΋rdY.O>/Eg֗.:kU/zr2=S!,9&WM'etS qt!OH\O&{(OjMzzu-iR?{T"G`]d}ɜ|XڻE}$NЭ M$fb=+ I.k։&ī1a}* N[zȓK ه1^38D<-J~:_V{'ɾ/ě߭Yڷwg)ELy GyJ#%>5d4?Zg_ѝQ1秖V-XU'LO=B[$]ms^?ZkHd(DX͑?EƯm]=r(ŇA^߲1vHF4ry.'ӿSx~sQzҙ-tʈEv]B$$t I }D'ٔGsv]B]D|9YICLJM ]mNK&}:v7 ).X vk?WBOzJ+y 't!E NG=C4]AI}5@l)rE|!wLI +^q퓍s4-b) o?ҭa|b{u^X- -7#]/][D~Q8ֿҧO9q@2Qi/ʿ q3cBn_o#y-91ƛ_]B>6>O?9O%u!k\>vn_ x+~O!oj~T\[Y_>gQHݸ~~-Ig?ڧc~kHG5q'ӼkG><~1zz{3{r6O}֭.5(^usJA_}@QS7uTm M[, bw:ғJ:b{㛢s,g=䧟OS2dD'tOtxb{HB$:`'J6#]PM?%FMqfISHb_N $xbз|'8 ZCC?Oٺ?GԟWoOVpQ/+б*g|^Z{Ck SOeh_El}}+d'u0k9wS|9o3 32_]BM#AE5Ai>?y~s9U4f7 $n\?~˱-5k8㜟3GFx-\ܑ?A?,xXۡȼ,<5u޿蜋&o)r"qS¬/\b1/OXEv `J({HGM^Ip.2_ZJeOP3m始Pqŋ a/ĵ/7}Cσ^/ƟnQ7T+DW'e ov%_jv>/ۄDܟ ;o|wkmտ^%`XN$˱n9wGD;k牿Ч/[j<שe 1Oy~P7Fx[!Ox\БO=[X>Vk|Zk%¾ i}F\?ה߶OlobGIx3~woL~T\[Ko֥k}<#['b4OkƎ:rqq߽ ?ʕ] ,=\5̢\eVα=ؙ&~):mzg2=ELEcz=́sW+J秕ƽqsT`>*:yR wpMLI uf˷|)K"/l FU>^֟ZW.F_ .:א*o|Tֲb Ue6[k,5 ?$5;gw|.nEg![(_/:z1fG_sAO2G{׳OO0GsLm}/WZAbXWC&43~آͣӗ藬[)>9VpO!l_  5c[Sǖv" M1yYOS#S7V*:cB^t.{zzzϊɏStNܩ.vv >bUY}*:n!^OcA0_]dyt|׫;\IVm%?3.?I!wC'G5?W>JYNiq/Y7>~,-{"\F(OxķΩ$77V{C!ܟE>?FO@?_$YߴOtﶾA~>ޓ"G~_-Z6ג>'91/_֯7+E@3~vQoF5>POG{5s۴H}OtwF絟Iy#t_XȟH/8..gٞ$'z Ѕu}:\= nvi}y8y)#aG$Zm>ȧ+%dW'7aE4_9>QN\Go?n;_h.:ӿ~A-1*˞_jlg㻵jذ?+P(gc/'~k÷ǝ}'>|Zn6_l-b;~/_5d4fwv-ozm2X?D_F8_@O=Qꝿ QG%>=<^qϑo;}F|>\V}gc!Đw~ efK?[5pIFH8Eg\ RT+j~]`=];SVzpYN\/(qO}޻IݳhooͽoMg5^<o^=zjOps+h;hμн ^=+u_:5XlqEv,ys܇{d蛼şH_^ #=-/x67^T@Qm^ߔAg^[?ȯ- vuMѹWL]A)VY~[,O;P gK|u^}5k9.:7K_WodHy):=9]O(x7qڀgR;x`uih/oq~ok# Ģ3 j| *]bx.dJA ͶSF*: X R}q0[ lt>ѓs-N_ɳY.(pp䯝_cdx]OIͯx+bTxEgƗ8˿>vf/w:9_Y0M'UNP Xr ႓wye뷽#x=cg_kbMo={^n-薶=\̯3E[b \'ߧ?6.&9Yn%7'>c 涚/D(Aзh}Q1c8J{ ҍC^d}y-c_Me0]<7JLhGzEvl=AImL>)a;_?yiΆrұ'tokc;ğ]_,ٺ§ߘ?E߽,:o[c/_~8t}G?G59`<(ħ  NGw1t_E?_ _[8Z|݀߷w_?8[ǃ(oEgUEg*V̯_ (믦~Bv[4ER|^  Eo/ ^dn_1t.(q8eF.חR p^8z=E٤_qz9 O4b(!_9Gɽ/.X#%LEv)ϵ/=e+gߚenߘsߍk`/)koQߒ7o~տ&Yw%vo_C?e+crikğ:n[پt/;ư{}"%//mlM#_ӹt\x[_[*AdUyQ\, E?qEgm<BEONg*4>JQnc!.IKfc{(OoiMzzm-iR?{TzXɱݮg/:şK_'O DAjo.y2fhmbA֧ү܀jտm[OROZ2Wϭcfp~kx[fuBO}׷_7[oܑȑ|Y{Eshsɤ?dH f*#D"_(TԅJ @,m+::̻zzJ>Cc7pȇ9RK,#r?~EwD3?Ϲ?Qc?~'IGq[?m|w|K2?GGH绔k6󣵿X#C!o/7:hn}ya^/.ܿgiC1=&i%GS/U>/Źbs~K~MvqzB9?nϴ`͓XÓEы΢||-z nA QgH*p*`LrP ?d x9>&MC"=`4&dD4xeC2 1|`s+AOn}{&TBpv>OUѷAI{X@/l)1$|#o;z?ʲv;ecl^S08t@72a`evh| Q\?l2bluE;~|럒=M~mTOwQi/sϕ*s7ܟ8 l>rϭLdW[s(~ ֖C=0qql?0v~z~S#cvIUPk? ~}CQe0/iީ߮K<7Tw/ .>k[g֗158c/ʮs-Ƣ.> ᦰKbQx{3'Ihp1Β<):"zփypc 8mCiHGL!,In494ub_NP(-?Tt ??  ߐ.bq+:mzg2u}#AvS!vpq!+:μ|z;St 6\WY~~2㥾Kq<*/&W|}eOR;b;.>a83xz: 70|Br/U߬Wf"oaEc,N1$>] i0yH>}4?}Lֿ!~O+B7K/s;?5yP^u|~)+$\~{|vVoD#h'{16~nwݿ}^>ksj| t/R$y\ډ. pOߞ8q_ "ؿȯx=ڱ~M{Nqv.?}GHX0[C>3_0ׯq!n.w^~Nᳵ>#I7#ȏK>먟YtO27O˯H{GמڶJ=Eǧ򉏟ߥby{R.p[tɂ%U1GܸxX}8S뙭})0chZ?lK{83~fp ?5e0ze5~/ 1hBb~!v(8M+:~{ӑ-qY"˞_;QSZ|vֵۿz:s|WBJ݄m(؎= Ŷ?ɿjL?\;د_-62>Xu o/gd;X+"X_| Y^_y  ݀q!lkl`E~m?7E>A>2|7h#Gn/:/?cZ_㳸>I7Ʒlk}\ޞ~gc?ߜrcSP7Ӈua<ť|G'TbqM/ͥǗ 7­(01Xk/n{.z5zw9 Žpc%q|uos1C1~w\sI˵4/IO⼿X.'.Kibqk_o(;)J4;fݳ >eP|d9w#JyX,.a<a})_,d#V!Z8׾oV/a ~ݳC|xSǔ?9W|`ѹ_ \_EbX,bX,b:΢sK;XB Eg[tb3)^E;ry+/'tNm[9bG Eg5ϧgJ*(IDAT 4p??8ʫ?~~{+6ӗoNw'g$|V6}MpG%wqkk} ~ ߧd?8k8=ٗR(LqcA; b(Ș4>X`׷Kw6vb@I(^SOW,vk}{A~ `2x矈2{}Ygc/_~8k|G+!`oz~YٙGto|urAP|σE<"?_ _藟|g־\{?fVhآra+nOZ芞iQbw [-:ylkUt.lO:o^ ߦ+MA7ߝ~ ٴhK! C{++0x% 8E|ODz.l}as~Nj}~'FY>rPU?~p-`_.g|~oP[o?}n;9|~r|үm'`7o~ֿ&Ywe>_ _?%v1߶Af ~Ϳg_?:g/ aw|_4zlO׷þ]{?س1xK)>ڽȢVe /G- }z@|[wt'=a)8;YIi̧* =lmsDgl}7rG}eYO6E)SZ:0?׿RRb2 ;e4޿(.z6V|G#-hsg#ӏM~mЧrTˏJ{{W̼0?o?`G~.uc'95߂`Ccp. 1nMЎk%]WA[47 mG4{#ſX8>ꟛ?j|I[;k[?O[Gֿ%'tqS@Q5~ƋΧo})fQeVPStnɛhq_t~vk7uP%O7]ۻ>qOB+QtI9ѳϊ/8j2dbPF'$Ԑ6X@d~HzuɽlO(i2IW/dl %A2M?AǺAPg!_vb9>+L{9U$MY{XIikd|8Nҭσ6TOחk޿]l8< |?,s+Ώ= I+?F4ݿpKqQ[סO.xcT;KH8wKvu]gd0.y+~1[˿xmey 0&գs('^609;;(/_oaG|v |}ZXq찯h~Ub́Cz6N7~Fߥ1gBko.&:x}̟vfgO6c7t}$>.5A }3(W贊|+:s`Gy]tѢ|V.H1ՊΖEXMLh O<0ǹ}9@ټ}t~Z+*KrS+0*:n8;*5)>{[ݸ@aA>w-¶_C<>v XlB+Q>֡A/ V1Qed۟?2ejY&aŷ 7柡C_waEgfiw'{gi`^氳ha֜k^m.I)? s}kjV'F_5 پ7#>v>MDy9Zџy~1/Y86?xr?/~?ڌ}G^iUuI\苜L)|3T=i_):{@frs*n|rڙ..wEgbWљOOb{*:K_!>ïI((.}/׫;\IVm֧.?I#H{' +T}~b_1eOoWyo;S;$tj'!)>,1:3qX|Yu?i.o7cr|mؠͻ܆Ą/:|6K;ODˏJ;74LSC?D;@ >pW0\9{o971?}g;z+5~vKӷ'a|9N|$/+^j|v_GS/ngpcpsߑ?}%!ȓu5y}/N}0qtO E'?"qK߈ /ASNs+n^N*nO}mctR d$g{ NGԶ;zbw-rB-"si*[tO27O˯H{GמڶJ=Eǧ򉏟ߥby{R.x[tɆ%e1GX}8S뙭}ihaƀ'тm۵X!wy 8>?|l\Fo!]H?S oEg?bu쉀?خŧoO74;&gcx,kewp o龽ca]˭砣sktrTn۳O;2H؎= E?HUOd긝x~sZȀhc 0ӿ涟`zǗ`uT'~!dy}}$b灔ϊc!,x}eĴ}ǷkBb)Lw_?j%ӟ_ދUq󇠿Td%A[s ga+f}{Z.ƥ|K'o?1ar&KRʈ_==|gB\a Z>[EY)ZMZOٽϭ 6b~M_ukQҍe7.^t.?Ƽ6Km2ׂCk#=-~X7|~[/*}oy~_7-5韕_ۦ):3WE)kԐP.q|yX,.a<a}_,d#6 Z9׾oVudvkZ,n):[4h.bX,bX,bX(7}I~/w1omX,nC/n4|j}zY燳qH:56{f:˿#_~Hg?t,|1w:_6~>=O__} mNNAǩcV^}{pVͥwQ_SNxz'~`-t.5A緣Ҵ[ߟf^?[:Uٹqs,ܳ^ӶUt~|#XEa7o?= ӗoNw'g$|V6}MpG%wqkk} ~\?h8+_KR)˸džw[ MOdLl,}p %lo;1 _$Mj'c?W;[uO??OD߽,w@ձ/i5#kPp Ge'eA31~ܩ~u|kķR֗eu徕Bo]oL X {yH"ʶ];4.x矌:>z߼mYdݕB|).iG͏ܶ׿=^~iͿsǟe~ɾ.8G_͝_ _r6* :v4ƿg׬}=3+X\ ]q9˓߰pY"7) zr2|BXsLdbލR qp=|6X~tۤ'ٶ_>TEK7vf1>8?_H_\_pO"=t+_*]a8ރ tĂx#WmjO[yI'Mz هlkoZ,>J^96o~gG^^jpf?ϐB>ʱכ~X %>jLˏzLkW˟ 0?M|Ϲ?Q;[=Nz= dk[89?JGҭ=ߥ^6YOp_,c7GPțC4yu?GΏw/L~w9t`?K"Q1Is(>y}.#?k6Xmv}\ntƲ.r^tm[G lPp?$5H g?%BHg:^NO(IӐo{ĤH<ͷ.;CLM䩝?H/W6)]߰ vk?WBW1lO `'#?T}$mb`rO7C,k^69:< [Ot#cW_O_f{aƗŅY/5~3q~EM/܍c@r~?Ҟ:}en2fSݾG.ڒbg>.. cǸ!75>B;iw_EޚVm7 M;_?柪egǭοiߚ_[eWѹpcQ^\pS%m($REgIc=?&CH&SAolI!j1!ktJK)$)\JzL}9Y*@Izbз |`SOб&x ?0P0kCN3GqEi|w?GJb֗$öI!kS}á 5i2m~, AyPWywע_Ƿ\Wޚ{[G?؞aӏD~_GkM6>83:}xC{wrGNʱ6}o]ɯF)v-nϐ 3"OWџi{f<4H|{/wdf_Ƣ}LߴY蜋_>?׷/Bn-p- {禽qѸb<  (%,AWC~WTgWH%Vx8=6 3C҈ѯ?SYg[׃E1q/V@1FBS1߃Ӳa? o`|wg7.?N851ø~@+:kϞGVrbI(ܟ*n׿ ~ :zW[%bֳ7V;k3_s`^5'4ys*wo̙ЇY~ܛ/= =>g_DX{%\5SǍ#]>";"??8!.i;}x>xS 1g\8O3;VO+EEcI`s3(JƦOI)UI93,})(,>b\Ex_+chG|p՟[ڛXMh}E6h ko~˴5io|Fٱ>珌#7o}Yhֿ 9~k|- }gh`,(b]_+:3O_b枦M EQۭϸ?k;fif0e z'٧Oo$A}[gy~5֜ /dߌ69a|k]GQ`hg(-$ryg%br/]¶u8.jAfSL|(O(_Gg_Bٞ~@ .vv ]YoUtӓអcẊҗ@H-s~/]_QyW/IjMT')_ܰrdiY?<=v!u}EoO?a+3f0" 1Or{sjv.X4W<$\E>?F>~&k}Y@߂ow?߂ο'ץ~\H{˽w[6 &:~c ͿlC_i'q ]~TY|ɟ>e'Atnpyy~-pwK鿒?Pqi'4=}{/ƗNZ~xD_zP#c? \|q;+뇋/ I>c9޿g/`_C8B d]p Dhgn9AE.? HETl 8{soްc)Z{e8?n.#0lUtr$,:''g$=B{kOmx荞SR<ʿ=} )jt_tI-:dA*I#n\EVK<>Ph Ty~)־{1I`v-6FHȥ=Nڿ?3?8Dkݲ?l4!1?b;Na&ܿH?zdKC֡Ȳ'D`oj?jXch(gc/\'_|u/(W'6skveP,b;G~o3'u?Yp\?_?\sv}WZ]E:*HzG"@qHc7~}bZ$_fMѯ_Ю:by{8ͭo;}}:뼼3k3Y0>ςg!AĴ}R5?,y?< X'v` ñ\ET<=o9ԃIԗa,tl;o&]t/ bq7.k~Q}m.= ߸/Xn]E_{qKw־;aW-G.L@IJ[&E >芴 3{!}Eg x[usI?M9$'q_Y,C%k5nv/^XEߔnt%߿n[8SLj>Sљ-TIU!ȻPs@N.byJyX,.a<a})_,d#V!Z8׾oV/a ~5qГXEgF;^tNEY5Zљ/f4c>!>.oVUt^,bX,bX,7.VrLu>̿?._&rtd{Xt|^t~' LۿX=ǰٶbQ9׻wrӸ=@Kg! 확{o>/?]d}̞_I)o[(LN+Ҿfѹ+B_<Μғ6;/?'/>lsÅ6WR' 3+pƾ=8? Xׯ)+yp?W:7E:Wy?n=>nUٓfƽϭD{Ci۝ .jIg ss?W"k>q7la? =LtTbxGG?Fљ_|s_2|sGt߿?;?oO?>#Oo'>k?-Qӗ[[ H0S8> 5H<ő\aȾ/Ba ylxEGƤwlOܞm}\wuLBٴn2?fyssX {^[w^;DBX?_+Y;_)U?,so,&gegUxk{C=/P~\'Ks2E{?س1gdSbs+nӝE S {}E2\{z gf5#7Pm ߾ߟ~@A9tcQz/M=LѸ;]@=_ L_N// _^! 3_tz,.la6l+/ >'e*A6g\́ꗾx'F| g|~o [o?}n;9|~r|үm'`7o~ֿ&Ywe>_ _?%vavC̞wEt_a9q2~td_ #%hV~\A.ۓf=G}+f{gcfψT Z5AB͵<'K~ӧ Ϛ^82$.ضdz7Utˑ\|Egt/n冋Κ<I!\eГ$cq%tnܗX\7oM?~I~tۤ'Y_\T !m36sk I[TI>HdbaTצ&FlkW[lT[OOʢ{; هlkoZ,>J^96o~g%o]zz >C(._oc6 h'2.?3ٯ_-6s8WtG4˿i?ܓ#Z@ֿ6;Qdyy}$rN{-pz~K~$r}(DX͑?fPM^]?ϭ󣇡=Ƿ2ЭY?}(6DclL_jR!\6>ֿ%=b0l rAsWtQ`\dW^;qWBX/]aͮV NO( sQy˖vxR>"zYO`Eo,!M0$EH4lDB7Q3/''iIאo{Ĥmo]vvP#?4^ِ⦎vC?[}vr?ɭpcA  NG31H>i[(68ۃ- Gtl cQC_Yn'l쓍st~Qy ;Ƿ֟F"ԟA// .o/ϻ?(.7sg#Oo+䧜߶y}*'Oʿj|̛ csO tzo8Ac}t9%Pό}\\?~~a7DGh5[׊E/I|o<,|C FtdZO>fև1~8o_> E{Λnr~ҋ|p>ڮ~/l{79QUtXԅg@<=vI[, obw:= #FY'EXDzO?+liC$PCDkhcmFU_!M&i<$]]ؗL'}6Aǟ poHCډ}(3hW^d6?dayo8Tz&_O<9Z;;H>S?J^_ nyZtk6Okʼ#ly|0'毮={N޸^5 ߺ}*wk?'"F]F'r c_ۻrGc eoF|3$Ì{giڴcdOAߌ?iwdfOƢ}Lϴrm> !gۯt RuIgǓŘF^mKA_g,:aK16+Z9m|NoO_>1Z$&ZFM{)q*:SyIg/8 kQ Kp]$$.2^I5+(Aax;<졂@ 1ey_3~z˳Aj(Vc_~kϵcb;? z6Pocaקe?:"wKvu]gd߽?(+:+yZk+9S`LGE8TQ7Otm`npɛWKc΄>G]~s7>o?sfD3?~ʵq#qGdGooA5B玢 \5Rqt]";RtuEd OH4g_9h;)b(ϸ( qnfw6oE_(Vƒ\ g*:#jQ̕l7B\s=n\UP0} ʋsI}aۯWuo;wV,ojofcu6(h JikdFg+ߘэc}2UGo|b5r0[RC3gW{Xљ]k{67Em?9?}/ƚIH1ufd2?0m}rg9Xs:(ྐ[~3jw@umG/Y86?xr?/~?ڌ}G^i3+ï'bQ/]_QyWwpW0\a@Aނs|5n|c~:vz)Wj<.DoOe}r8|Iʯ^_WX=j8C;_uo#$|?ܷ<^bH|!`;)>^;;DԋO~:Oۗǭ/}|#>˒O=:[X<\ ~^,>]Eߑ3us[+:7v͓3~e=ѵmRFOQ|wXߞ} /:$C!tI$r 7"%.V"8S뙭}ihaƀ'тm۵X!wy 8>?|l\Fo!]H?) oEg?bu쉀?خŧoO74;&gcx,kewp o龽ca]˭砣8~`D@7!GE6_ʠXv,i죯gROd]h=62>ڹ 0ӿ涟`zǗ`uT'~!dy}}$b灐#y_ `_ Зd/k߬) Q_ p绹m@ojźXG_g?g`w| hkJL'/}|{̮Kp3Q=k.rZ!>9RtNxu' wT3g ??$.y_,.)/2ar&KRʈ_==|gB\hƅhiYo=O:߳?xsIe~sYd'yݯ-?..GzZ`k5nv/^TXEߔnt%߿n[k?+.Ut<VsX,~֗BƱ>kүsf_Ga~xb*:/bX,bX,0Ͷ-ōr [ߟYOZO^?p6_1oϬBgwogwNgϧg 4p)?8qʫnUٓfƽϭKks>zMVyEg~A~ۧoO/ߟ>?x{m~rM~S>1\lߝ|Zd_@EA(/ "T 2.,hVlBK߱=l&+_;q\п79r+e{?'3Qܳ{_f̬bq%*:C)f~J|@SЍEw}B6-70ER|tq^ :|J/ zb2 _$_68C$g(XѰͻ/ZCqԏrO|C HU?HTTOb0/}Bj3lܷr׷>7vk}{]~9>WDٶkށcQg_O7 ?_2vW//;mаs{'}>vr깹藦+?9n~Y׏Ιys{0~My=kv^zW^!ޓ{=;`55kkO/W®sW\sw?-:{q7,7\tM * (&y+Yw#澔B|.?1C#_/>ݥ6Im}~b' ,yB[}f >P06|"~q}= =t+_*]@1,ރ tĂx#WmjO[yIRIYtog[!pc-[œ2o_)~V R=Ɨ |l1G:.2ϐBcM?ECp~,PHAw(#r?hl~EwD3?Ϲ?Qe?~'IGq[m|w|K2?GGH绔k6󣵿X#C!o/7:hn}= _赙]rA6DcP}4_5)\+F~0ldWy+'Sx{L ƹ(iK(68ۃ- Fo cQC_Yn'l쓍st~Qy ;Ƿ֟F"ԟA// **Owqt}ΝL?QrOr~>]~Ts_5d͆19N?}}\s뤃8Y[c}t9%Pό}\\%/ㆨؾ]5ykZOs_6p~{O/튿DsOWdZ_Yeo/N~dXŭ\ π.yz)쒶X t{Z)\G$Oα~V|Q!$)q 7$ҐI5B:P%YVݔir.I %=&iž, $Pz=1['X<(!yj'㣸4矣]z1Ka$婾P~F?Mkj <(hOeh+y}Ykv?wCBGqAʼ#ly|0ʏQf"?M/ܣu&XAuT>FNO7XO'Wn(QGE1.%f x'֦:+B\-?9`F~v>=/?ݢO_Wl,Xw'~zo֯?׎x /~5ژCՎ];ox׵M?_/헼vu]gdpcqS2VK>>go8vg4*1!j~Ol's?\?#hUߘ3bL~l?93"/rm>nH|\3S}*xBQ"sWt&q+,^Ut&@63"1<iygVՊi寨h,Ly|3blm\—cϰ|WtpL>b\Ex_+chG|p՟[ڛXMh}E6h ko~˴5io|Fٱ>珌#7o}Yhֿ 9~k|- }gh`!S33u'{gi`^氳hak&!ZƠq1}O\ڷ}Wc頀BC&<pk@=):,} d R?Efw#| G^$6?S}`wv]|`sq?ʑg=gtn?`څ _Y?1ćѯ̘E޲'7+_ukA5>B;֯ip)pݿ~1 >"_m _}f_0ׯq!n.w^~Nᳵ>#I7#ȏK>먟QPEb(>kވHҭ蜊m_@ybdn_H+ =m㕢7z: O?K(/]}!$K$cqY-@1Sq"3[Roaƀ'тm۵!!p8i:g̰*j'ƃav>k _bфCPqHpVt# A-qY"˞_;QSZ|vֵۿz:oY%OQ 5C؎= پ0?C:n1@@C>`%7+o~}v}}d+#/nn};bWwG֏wƟñ~-}w/ GY\Zik6X#&GqnC{s)tKK1WЧOΥ|G'Mu1Ew\, E7|~㾔2bFu&C~-qEϻF/Y0]γ>w-kh,Wk~ sQt.?ii-B&?bqWȿ8/^ŭqK)*j7p+urL)Ok'©0Fz255!`?Nړv*:t@Fvf3<VsX,~֗BƱ>kүsfRk[\HD mI_T0%6> dW`sGtHcg`bX,bX,yA? |_ Sds|FUsqSh@L 4-Ɩy%ޤ ӎSt~' LۿX=ǰٶbQ9׻wrӸ=@Kg! 확{o>/?]d}̞_S S0 z9=pk4oL7~Nkd+jy_!ϑE:Hpzw:_6~>=O__} mNAǩgV^}{pVͥwQ_uSVxz'~`-tn:t~;*?>n}#{iz|,f'΍{[7e]zM'j42W$ }NJY)7>+:סk`6cp3S緧bzO}D=rϿj?ߦt)DN_>nm-/X Ob vBGvs "T 2.,h_lBK߱=*p{1oq 1.N 1 eӲ+}ja/|9/{lS{OQw/ auog?ȟYXLQ7K*L>s}*EPlss=-"Eg(6o_O?h ([XȦEh\ϝ.X ڋ^A/_iCOL&pHL8=˦ qEk(9|yžAD,PE9(f _~` IfE=}n}___[)uON?_瀮l+l۵Ccwɨ3X'͛߆I]z+ėOhP#=Ч=^~Ϳsǟe~ɾ.8GJ/{K7^A.ۓf\ y{5Ǟ?\4N\h (*XB!^Q*]:OFޢSt(EbnۀHѹ+.Gpy򹻟ѽ.:kP&pCONOP_<<|ǕlAER qp=|/ʗ^`_|Kmӓ{z/ŎcÒ'X,]J柯]$//'|ڞ /HN.qA N~~m:AlbA i+_vxN$,}k}揱o]?+YcK >wk~6L~qae3䁐rb!8?y(${(, r?h~EwD3?Ϲ?Qg?~'IGq[m|w|K2?GGH#ZK~h/H~P#( _~z[#G y5-s񏂹=_iCP:$v*~e^KY_(zpmgZS@WT,gSz.rqbьm{tЊvc׫qzB9?nϴ`͓XÓEы΢||-zȾ`)($EH4lDB7Q3/''iIאo{Ĥmo]v_F4xeC: @n3Ywt'=a)8;YIi̧* =lmsDgl}ұ7rG}eYO6E)SZ:0?׿RRb2 ;e4޿(.z ڗOg|_'ʷ)䧜6O%|2\W27yan_{_#y-c'k5߂`Ccp.籶d`c״K毂"o_+inΏ,(u ſX8>ꟛ?j| v&~1k>ɏKO*EG󇷧߹c}r"/WȬSx;;P'#u؊(isWמt=Z['o\/mqQ[סO.xcT;KH8#Ո1ێ7#>aFj=34mZ_h[314NƢ}Lϴrm> !gۯ1*POn܊,owKvu]gd+n4?eQ[Qc?G?GoYWF@a I(ܟ*n׿ ~ :zW[%bֳ7V;k3_s`^5'4ys*wo̙ЇH)_W.X\\xWB!Y{SǍ#]>";"+'gǞMh_,h("E֍rϸ.=Ή߈Qo|):7OMP*/iNX}xWtdx3. {ۧ[W+J秕$W0ΈFZ?Rbcoup(a+sI}aۯWuo;wV,ojofcu6(h JikdFg+ߘэc}2UGo|b5r0[R_X+ۂ=?M.5yO=M Ջ[qv> ח~ךt-c?>e~z'arM>stP}!۷fGɁ(;/\<3>0Ϗpr2藬[錏<~OFt}egtm>u}# i SOsEQ\rM/vvS!vpq!+:μ|z;St 6\WY~5~NOJ_jRߥ<|zu~Ǔ+jMT')_rdiY?<=v!u}EoO?a+3f0" 'K{N1$>] i0yH>}4?}Lֿ!~O+B7K/sǟ͟6h.Ƥ̏4žNwQigF4&{zJch:{[ ?/[pƷoOgbَ^/Jy\ډ. pOߞ8q_ "ؿȯx=ڱ~M{Nqv.?}GHXeFɺG#seIع& Dhgnt>/ŏ[_F|y=@%zu>^%Ax's!X\E$sD_Yh}tmSy|*]*Go!E!El]R&}čj}Q6.| [Y.$BlCڄ3~u쉀?خŧoO74;&gcx,kewp o龽ca]˭砣m*AD@7!GE6=jP۱<ľ'W?`Ȁh>Y'L"~FO_/QEǠ9PB~p?>n1 v~-@_ ؿȯm~/hGFB|1=ַ?}ĈQźXG_g?g`w| hkJL'/}|{̮Kp>%A\qO*Zj%]t/ qbqw'M&k~m.= ߸/Xn]E_{qKw־+O{aы \~Rs??C&?~-hH?tvq>^ŭqK*7p+ur_YuX,v<VsX,~֗BƱ>kүsf_Ga~xb*:/bX,bX,0Ͷ-ōr [ߟYOZO^?p6_1oϬBgwogwNgϧg 4p)?8qʫnUٓfƽϭߟĿSܳ^ӶUt^,"Fљ_|s_A~=}zw"ޞ~|Fr_koߔO| ~[w/, h}Q1k8J;#aȾ/Ba ylxDGƤwl ܞm}\vuLBtn2?fyssX {^[w^;DBX?_+Y;_)-Z2`#B|: 7tM>V`v:o(Aؿ"EʯW{Խv޻=ߓ?0ŕ ߦ+MA7ߝ~ ٴhK! C{++0x0||di8Rc)F.l}/~7xx|bhb(E:~HPĬE_*\'F| K!5}n}___[)uON?_瀮l+l۵Ccwɨ3X'͛߆I]z+ėOhP3k|r깹藦+?9n~Y׏Ιys{@~My=kv^zWΕ ;zvkW3qk־׌_\?N_cWѹ+.Gpy򹻟ѽ.:kP&pCONOP_<|Õ,AER qp=|8,?1/~ۤ'ٶ_\$ qslmks1@/Bۓ8A|0=@ϯM'M,72!>~ ک$Evw128X<-}g/ sl| 'nφiP]~m==!|Uȱכ~}8X䡐ۃPX@'2.?3ٯ_-6Ef~s0s,*pO^hY?6NGe~Ətkw)gײ l^ ֟}~G+ׇBC_ ou:9?Jzm&p׻:Prfo F}9d{Mſ?e}b_sQ??oNt9oOwm۠+\\Û ryUt T^>ӂq. 7O:cOJ9GD/: #l QgH*p*`LrP ?d x9>&MC"=`4&d5vh!?4^P⦌v*x حG;:\ ~Г[_ǰI?Ճ $Sgc|9Pmp!3[A> wN{'v) o?ЍE_)~?1텝2__f=׾̻_:[ľ}F(rOr~>]~Ts_5d͆19N?}}\s먃8Y{c}t9%Pό}\\&/ㆨؾ]5ykZOs_6p~dA1u3L4jT~qoW%bHN[{k[g֗158cG.vERtyB7˄a #آs~8^!8Z;F-f'{ zWѹpcQ^\pS%m($REgIc=?&CH&SAolI!j1!ktJK?$)\JzL}9Y*@Izbз |`SOб&x ?0P0kCN3GqEi|w?GJb֗$öI!kS}á 5i2m~, AyPWywע_Ƿ\;o~ʼ#ly|0ʏQf"?M/ܣu&XAuT>FNO){;lb_VStv<"wҎv}DvDLTp\ .:to蜞Eg;Kg+8α@ޙ OCxEaxs4sy+jEWT4 S1Tt6} NJJMjo쏱gXՍ+: oá%mZm??^y|p׽Y?雉?لWl#16}aWZL[& 0:+XGn등oȨ:R|֗gㇱ߂ЗP33u'{gi`^氳ha֜k^m.I)? s}kjV'F_5 پ7#>v>MDy9Zџy~a(-$ryg%br/]¶u8.jAfSL|:ɕ$5ߦ~{_Yg}؎ln?NG9GxN}?_\K7'Ɵ0[D~q'99CLb;OovB+LcϢnA?灵,ooA__ka R?.v??Y6h.19Ct|N/įy5{.?*ވF>Ot2j|s[]O?ܽD-lqy~X-pwK鿒=~vKӷ'a|9N|$/+^j|v_GS/ngpcpsߑ?}1όc|;5?#MᎿ@ԋc)Of>)F|c'p'Wr b42eg9?<9,>#Wc]{j+Eou@'>~~Q_HQCHm9$ BTI"@q*ZaBcЧDLy߯gÌ? mac\븞3C4O)(-|_G!Cq !m[kA9d,{"@DOkS= ͎??2~Z:v~=\u=y[o7XXnr96~hS'0-@7!GE6`veP,b;G~gQOd]~=jJ`o?Tpm?#']iu/OO"cЏ\H")y_ `_ Зd/k߬) Q_ p绹ma1bus8֯e;>+Y31mc|{~Kp]DhO:"ws5t֮G4Utf~6. H<ƿX?aeS]5Aȿ6_oܗWF,p`}ȯ%y?k+yG,Zнձm30'-i5C&?bqWȿ8/^ŭqK)*j7p+urWN*ƾ9Unga_1S~-bX\*x.RX8GzBp}q߬_ wo&\$4W-Oo/FW3cTtɯ)XEbX,bX,'-&o^Z-:m*B°>O}}o~ ;O{g{Na}mbqr~9(?wq{hӁ:?CA 䯱3+'k|_~=vۂj,Β.W,0џz=<՜u4r_\ew:_6~>=O__} mNAǩgV^}{pVͥwQ_uSVxz'~`-t.5A緣*:=ϭb6?햿_=kvnK*:9$EXLHs*{yAo{Jl_'xc7o?=O_O_9#ߟޝ󟈷\nZ'7õ(ǭE $)~VL$NCn0Rd_—J0]e<6M|#c`c;'nO}\6f>.;?x؉:&lZV{EO?_<`9=es/;|j"e!|MgG{,&gegUxk{C=B~L7ɯplOgc=cʿ [At+fѹu"dO vB?~Qq)*"4*:C)f~J|@SЍEw}B6-70ER|tq^ :|J/ zb2 _T_68C$g(X6]8/Z6~7xx|bh(E:~ ÝIfRHM[_zƗV %>ovݾ1ƿ.`/|97/!(v;p2 I~淡gkuWn ;CKGƷ8vEt_a9q2~td_ #z~oM;c _r8?0kk=3+vrT*"w9{ ²UtuvT@nm`foKOq?0#}=6Im}~g,yub5|alE{I[TI>`(qAzN~~m:AlbA i+_vxN$,}k}揱o]?+YcK >aώo~m==!|Uȱכ~pȇ9RK,#'2.?3ٯ_-6G&ܟ(ܵ˿i?ܓ#Z@֏6;Qdyy}$]l״f[}(DX͑?fPM^]?ϭcO<ЭY?}(6DclL_jR!\6>ֿ%]P!8?+=}EbsʗcAX):rnu̅ZEy?n]E@PNO3-p3-t}D(n:/XB2N`Hh3$%g?=؉oHg:^NOh`!IU5&du#H!Gi!M~ G;:\ ~Г[_ǰ?Ճ@$Sgc|9Pmp"3[A> wN{'v) o?ЍE_)~?1텝2__'aH\C7|_382Dy%?}*'O1p _o=:oϹ?5pk}䲟[G|8sƷ ؇GP-)z^c2^l?0v~z~S#cvIUPk? ~}C#gH|o<,|C FtdZO>fև1~8o_k F`+>RwY[ѹw)"r#zC|JYz]I/>e~*:n,3Kn -w1;}➄V ,ɓs,g=_pd 6t$PCDkhcmFU6$E&i<$]]ؗL'}6Aǟ poHCډ}(3hW^d6?dayo8Tz&_O<9Z;;H>S?J^_ nyZtk6;?~~WO_];%{N޸^i3@?C]穈?w|,7ῑqxkWoAu>S/[~33ݞ!1fDߣ?sOӦ1Ѷp&>?y|3??1?~˵H 7cmZs+JSh>9< -b._f l!:4?u*E`_9m|NoO_>1Z$&ZFM{)q*:SyIg/8 kQ Kp]$$.2^I5+(Aax+T<졂@ 1ey_3~z˳Aj(7Eg8 ;}\7 kv<~mLj`®O~?tE7xZ}&&1X넳\O\V PNJΊ}sDQo-uh|!0&գs('^609;;(/_oaG|v <*cűþ.>#s@1Ul|b;;A7Ɯ }ϏE/ÿ]~s7>o?sfD3?~ʵq#qGdG)D}y`~٢z.LBuW'xVkOa8α@ޙ OCxEaxs4sy+jEWT4 SQbd;Ԥ<{[ݸC`A>w-¶_C<>v XlB+Q>֡A/ V1Qed۟?2ejY&aŷ 7? k~(zXљ__b枦M EQۭϸ?k;fi?ggk^m.I)? s}kjV78Xs:(ྐ[~3jw@umGCkK-MtG\kp'#O>q_6c>fvڿ8CwTrvS!vpq!+:μ|z;St 6\WY~5~ 6mfw#| G^JoS>wI ~GlG6GټF#}OO'>F柯]p]_Q.O|ʌY-{"L8b̓e~!&ħ7V;!&I1g?ǠZ_з įI{Eu濊<=?܆XN'7hMDy/G?D;@ >pn \5{o971?}g;z+5~vKӷ'a:isA5>B;֯ip)pݿ~1 >"_3k e>y}d}kɾsLKD$q})~7#ȋ:,D~{gg9?<9,>#Wc]{j+Eou@'>~~Q_HQCHm9$BI"@q+ZbBcЧDLy߯g D mac^_B?3?8Dqkݲ?lt!1?b;`&ܿH?Ȗ8CeOv->};~վ1>cQXO//_/O~Kf _v=#o# 9 7`veP,b;G~HUOd]v_ϿZdmd}?_?\sv}WZ]E:*HzG"q9nL`~>@_ ؿȯm~/hGFB|1=ַ?>AźXG_g?g`w| ]Wb>q}gv[c/wť|K')_0X\ܗKRʈ_\-]k s^X{(:?!_juX ȿ8/iCָۥzQaEw|Sӕ|in9NX,P΋bqINX,?Kb!Xʹ}#?]bXbbX,bX,bqz4,0nbq<fF9rP~\ɭO֧ ' /u~8䯃_cgV~;巋O;x3e3gm~yx|8fUطg\ڿU9pw:w BA緣Ҵ[ߟf^?ku|_s֥YgmXD3 S緧zO}D=rϿj?ߦt)DN_>nm-/X AOb qvB GyK} _*t?n+6鏌Iネ=q}ۘ7mc'똄X>5d~ujtϽ'w(חֿ:V7wĿR\̽yZOG N\ <Wij(M~+e{?C[f=ߓ?0ŕ ߦ+MA7ߝ~ ٴhK! C{++0x0|Q}di8Rc)F.l}6SJNj}~'FY>!rPd _~ܩ~-`_._֗eu徕Bo]oL X {yH"ʶ];4.x矌:>z߼mYdݕB|).Puz`L{޽4<͏?9}1/\p{%_ɗ6|\'K,ti\ĽY^3{fV~vr/'i݋ۿaᢳ&EnRWwN{'v) o?ЍE_)~?1텝2__'72bl}{fG(sɏ.mCJS.?*eoGo=:oϹ?5pk}?޿LdW[s(~ <֖C=1qqiF_; Q?=ڱ}Mk*(omؑƳI4jT~qoW%bHN&:ڧ/c|k~q#ǒ_,n]EZE]x}pMaŢ.fxOܓJ:b%yRtE !OY=$tH}ĄHҁ*MVݔir.I %=&iž, $Pz=1['X<(!yj'㣸4矣]z1Ka$婾P~F?Mkj <(hOeh+y}YkvW?o$yW rl6~`ڳ}\_GkM:>83:}xC{wrGNʱ6}oO~9s>S/[~33ݞ!1fDߣ?sOӦ1٠ǐw}O%bX;;k=}]sSȭEbREat/ܴb1.Ws?gt^Dp%OJz*㕟T[<" s}cзt 搂a?BsH759<{<3:vr?]_z_`8&%WXV\;&3ؾ?HhcvV;{v}Z#/G /^S|7_% gw埸>G2p߱Gp?߲<R1ƤzTsOED7Fc?gg?=-숏nGaw8vg|(1!j~Ol's?\?#hUߘ3ͽ] sfE__?ڌ}8##  1g\8O3;VO+EEcI`s3(JƦOI)UI-|y1 ˷qWA(/%mZm??^y|p׽Y?雉?لWl#16}aWZL[& 0:+XGn등oȨ:R|֗gㇱ߂З?$,[Fϵ3sf~e=دؿ&韹i|zQvk3O>{هr!ZƠq1}O\ڷ}Wc頀BC&<pk@k?o+O]~Qu (G6o5 Ok/_W ~fF2cy˞/"$g߾w:vIl'NHsCR|Yct1g<֗-vw-k^?,|]_f"?w,06rkiLΡHgKJ;O*G^&)| ? o?`Gw~[w^{o971?}g;z+5~vKӷ'a|9N|ꤝ^_WX=j8C;_uo#$|Ϭbl=lwkG ;Dԋ܏)|gd>)F|c'pX\;;IatO}S6^)zT>T,oO>BBn!YJ1W| Ty~)־{1I`v-6FHȥ=Nڿ?3?8Dkݲ?l4!1?b;`&ܿH?Ȗ8CeOv->};~վ1>cQXO//_/O~Kf _v=#?, Ktro(9AXbG_O?qSv_ϿZdmd}sM`o?Tpm?#']iu/OO"B~H(X\ l|$_fMѯ_Ю:by{8ͭo;}#OtwƟñ~-}w/ GY\Zik6X#,Wť|G'^0X\ܗKRʈ_=->y?k+ĿO%%IENDB`llama7b_autotrans_8to1_predict_log2.png000066400000000000000000002435201476650141400354730ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDR/sRGBgAMA a pHYsodIDATx^=4;w cx-رb'I6><)wqF3pZ_,VuWu7þbqq}lvw777p2nwGr>xpm|;y>?l;lqذ<5?2np [~ol[?w{v}QQƏ[dWtx*Gj&W'|[mm_B>v_K9_ew?n~4ƟGCb?Qj Ļ-7⳥~kFײsNlo4~؟~mG? ^|OԧoB ֿ?x9rHg*c,!UG n7yڛ?'o%Կ&YwQmYף_.yH>{b};GJ:d^"m}=\n?Ӂ0QEm|h#7gڜ)wxygK֌_Ake瞝bh^khz G}W} QquM4=ԧSa|!Kc1{yȿL߾sM&;s6}aZ4W _o|V>'Sw:̛mȾZS?A{_Ad^oR["!1N~#[O7bQ~d+OںbѻG]ͧpO=rϿkom}()v/sjQ9FgꯐϟJY8GIm/8C߰jh_9WY${ k W5C}&TDgB/ֿ y$.7i#C"Esy]cM~[_O¦<$e6՟;1~ip?`g˓SVq(}[QBhww7gW~4o>?*qJ~w97:EW+[[{9 ס+Q~u$pӎ,du/?KOצIJz%V3i m!Q?M2[6cT-n?i2t-vۿV`_7-/zO0iUrRgO枂iߠFǶ p9k̟[j;rx◯qE|0UdGf| zR?kc=~O% ulK=16n>v_M_}cygY iT,ڝ+}Wم ,>뻞p;G}noHӿ{\@:}d6ICHN IIөxE_ݧ(Ǩj%׉nY#~"`W?oC8\v'ٲ/׾xAI_i[ޜ߄uM\bƯ+~)?CWLB=~?'}5K}ް #oֿ~r?y헨[k8zƑ{g?BI+B7֯kY)przQ,ZB_])o?9?GQ_ٍm.O\_J?韀_K?<> NNZ+GEQu2s{B*X_A<7{PuxE>W_Aԓwe93'u~M[x3L<>W7ăOS8=sIsQ{<|y">mO&ۢ}urm1>MuSŽG?.Kk/?=55vƷv߿'n,\a?⟌?'j<3>VKrAKntGA=~OFt*Z^[yg| ʔxq-mNH/{>^]џ@}8}YSw)S \p9-HZGnd,FHw~&P⩏ከ7o!]_ʿgj_I~&׭?(>yjR&-7yX/4{ʏU-?hgl8gQ$M$jg mn1[/ vr{G%__O;)m#kW!uwo@}>sqwі| \a$;SB,B?S c.ܙem?:Ypdw>VnCjezw?y,asUlW, M+L7u~q 8Qlkp[߰A~{g&Or$rp9|(M@1[W&%{ZMy<_,p4궔lmq~-?oE]'!lk?K7$;"?mOvvƏn(:Cߨ2b1-6&+[?o_|d(07:hN,P_Y_vTs>gpwgn[)pa'ؓz`/x_8 ol:wB\j1KX@}?x^wS\6WLO!GscA}'>u>}W~o> pb]UoGOz=Bor\ c@} 'A}| t9ՃݜwpFA}εc>A}c?y^˷Omۙkbϫb U9~P_I 6/שz>Q:5__'yF7&(y_YվyUnyQ_%~K%2T_{>uP2gUw+P_:/ڋO/A}DF;}p 1y~o߁_¥?6ŐƧ/\>y]\cΎ_ 5 2Z_P:O7}8b[A} WԧӀh:v!XsgлwzeNxZgЯb.3ȮYn?jWX׺qu:̝Y'gd fWA;K{MOG/w}x9/SNK[Ec']~s%9%eAfCهJ|Z}5~iaN1{"O6^XIWu*^__kXa꓍'[4&EC?py~UrO &ۘp4ĭ79WこO(4aD)vO(aj ]LvIzi]o6KDU.T]^EY}$h|}dgGbo@}Q˅3wbW/ֿbKG}Q˅I2>y>o<\ #O:)rTgV7}K1=3,ȗxHVOGݜć%Q~gS =ڿ]3g 1Y;G}F(jUv>XP3Roo>c9u/Ls~c{*P_R_)^z7}VTO4vr~_j㜇MFsӢ&/'OhwwbvM^ |f9g*O@/w}K%,oCI&Lme73M}Ư.~(?גV~},P_R_J6 IHEme73M}Fz?TjQ_V_J6 IHEme73?>/oY%OFϼ8\>AurUI5M^\.$AOSVOj@`|6/J<縖6$v#Q_3]>Ek6惯:ح1@}k6^^_ '[/'bgʵUG[sDqVOg|DP_UG/ ֿkw-֗3~ڪͣ? 3'-oqVOg|D>p9`N^1l>$S_ $jgP<:|_X.dK}ibhG8R1FϬ?+m@Ϫ* &.PU?;_} ZwpQWQ_9+a8c}bl>G!JΪ]>Ᾱ/!؇*rVl#~GW!sk|B;ɭ ezEEcF>Cg9}k03at*s& -SK"fkL^v|>;yi#{#ƧwƟ#k>SgiA}~e}Źؿ֭|? Y&+hGOu?s p}c}??!gB};Z?3Yfs Y\@1bu_`XVr4턶/>x럷P/_ԧ&.KX@}? 9?oy!P:'ϥ lT >[<ԧF@uJP F@}'>u>}#7c`uuE=Vл\>#lP_ lP:Xw6OXbKpՃݜwpFA}εc>A}c?yp{ m{pkb_ϫX%A}>  |Q~B}/y/X>cY|G˯u 7E}>D;ޡ~K%0X_{͠>uP*#ؗꓞ=m ?Kcs}Ňiv{ӌ׿??.WO^w ӯ&5CA5O] Oԧ@:z;ר/`}9@}>EW\g/XP_/ `sA}sW_pȇ˷c~v,[(݋J]@-?pƏGYu?왯Y'&||ҽ.j~8 k/ _kF~},rp'R_9%eAfCهJ|Z}5~iaN1{"O6^ONr|^____kXa鞈P?!ȏ>[ 9^sl \p'}J8*䐄Ccuф;RGQ%z KA(^O^<1^,8yvWObRvJca<Eg$]eC+=#KOv3kۨ}Wم `C}noHӿ{\dMK~ezACęhSJ3&O]UQN7)jo|G,7Ż]D_AŗG;o*L$r׷_’6nj+Qͥu>ژScTM`K2G(j+QͥF{F}6'H_);F}6gM>p^%D_&]E_5M^\a.;&bgʵUGb~HAqLgB_3Q_Ogu ucz:#<3>xy}e%P\n+Vm'3>DqVOg|DP_UG/ ֿkw-֗3~ڪͣ?,\@qlZ(!5!9WV䬓msp|l*zӋd3}GgL2Nr+9v}z l\laڗ?޿6_ gO?>R VrVbtOymil$m!y3W!sk|B;ɭ)zx7ymilֿN^ٿc 1XP[#ſIn%g.?X3Mhvpubi жK V%%w H~J ڬ襝 'Bit_ ?4h'W_pGh_nGOu?GOׂQgY ?!_mJRGk pp}9ڽS];ρe>=YD??'/<?r?b||g9PLg3ߵAT+ޏ؂u}'zq==O>>u6u!P_:->/SA}]ԗαy .N~LhЧv60NXоe6cA}l DP_ "O`)A}\.߄1-|WI F: zB 1>ud ]> S.P lpS|϶WLC:A}/o۶G˯蘼&F~s\/y/xqB}/y/XP_SE}>Ue*hGG+0yGWQ>}q@}>en!vϹ6F|kԗKfK>uP2|>~B}^| ԧ\.c{$=`ymW4@}IR~"LF}x+S)sڏ?}/XP_/ `sA} .~[e;vL9G?'/_-gVzq+tߋ'Ӣ)S{(_.'/-~#'/5~Oߟї';G/9^8ljr_"G}}ęs_|*r|K,3ʂnG!EV__kX+ ٵ_B ?UݿʟWӿo;Ƙ>{_>%铚T)aRb՟\ؠ[er))?SQ!ߔ`~_ݣ+ɓMT.\??Ec&Ť63']/#IzrneXH宓z/WQQGiW 0>^1$>kYYQA}>XҽԎ >tJ_/?+I#ߋ .pkNMdרwfMދSO^Y.a*P'jxgY9E|f99;dć%Q~g)^z7}W!_Ћd՛ 1Y;G}F(jUv>XP3Roo>ow.&jUv>XP3Roo>cbdڰHzK69=qB2O=WBPA2G(j+򏫽Qoo?Z+~Z7r.0Cn*8[/aɏ{O2G(j+Qͥt~w5լզeֿ¿?"~mnX̑;ngnsd}b2ڵB{G}X}E*&%#wݨG#Kԅ!&gx 4>mzSƈsJ׻8AuWvMW+Xǃrmџ񁀘&˵xq-mNH/;/>^]џ@w}rv*ޛucz:#Ǡ\[p'v rmџ+/voȹbgʵUGɰFqVOg|DP_UG/ ֿkw-֗3~ڪͣ?=3){;$w-ALJ591;3kBeMH8/y};'~]U֋']'JΪooq_h?֛}fcl}[lsUU{o"9YϿgk)ӟ]DP_? 1XP[#ſIn%g.?θ?ؔ\&-=I 5c5>R VrVb} x_P,7O51z#k&hzym4A6P1pȐhO^4x_h)2M.IBoů 珥2y|W&1>u3Y&3IɿIBZ7 Q@}2Q_ο/G?b||g9PLgڊaSh=\Z7u3Y&3 rcE{zhQؔ?2j߱y 9P_sB>uNׅ@}럷^ .Cؾe&/zop1>A:%OA}`#N DP`;m)!WлLn! lP:Xw6O`׻@}'es.w䇁 R=mMu?@P{~~.~~~nI L޾]p}b9/-39rt}-{ GgO7_\]~Џw{w{ǯZ~N :_t~uPZz߰~P:^|Q:_>oFЏ~ ԟۋO7/K`<$I,zqn{w`J6i`qGk+DaDdO_ԯRg\A)X~;\/Ir Fזs|'0O c,&Ld6d&C'}~m)>J˅K}.O?o>k7dA"N!]_ftq䏥=.{g]=5#>ϑY>C&Tv|ׅ ?WݿʟOO{ w"1.Q?w }^?Oo#GU믑^9cLD|َ ?7Z~LW!''%"-:'3KBVi4J44q .nIcJb?L{+~.zP֟r \ L:77?,E$;sD_GFɧ\}.ô~KjghŠ=X?ӽb|Pz|s%~]?co3 UQv/O.ӵfrh'UqLZ-vҏSh9'&oWǣ?sϢ͑.HVc ,󤯺9w}_(j|/=񛾗V;Co]eK/=#KOvus"rc|*P_R\8'ÄL_ XM*e!~If ? _?x<+)F_ =r[᪓W{;t߀Vqo\a懋/R5o~ԯAjg(a|/H$so\~W[Z~~__ҿ}gmWv|.3[|Ȳ&[_-d ^#+]bɁ?VnnA_j@@.뱻/S=ǽ9'BzUv~KGo5uGZ;2EPUG kw-֟3~ڪͣ?wVQgDPUG /G :_HR VrVbzdg'Q//B=o_p}eJ>v(YS'w5HkU_-o"&qo~VHhhۥ G|@~rXwb?*'ߗo?Gb|g~p~gp߳ׯ^ګ{D^e~=O\1~m[b6?_Ms^en_puh'9V, TϮ\B~8P7B~8P7=|fB}E2;8٘?འ~A;O0~A;O0~uqe[л8nޯj. lPc`# ؠ~ F@/>\=m g9.k?uyA^kr<1|`ϭϯX3{v+PpD7_5pƣ߰~P&G^|UW~?3Op06I,=_U~ŝrRs T{G~ -D>X?Q?o/>_ԯTC_toX?Q?o/>Hp{R@@,t&zFaq9%V4N@~ ʳ7?ԟ́U7m@EOO `F9\WHI .&c~6 \n?n4鶇A`kiߟA@$ -L?މ s_-&m^3t G?7ʙ⣕K[u}ݎ ?O}OZp52kF}͟#?$|/هM\~i# 9X?e ?WݿʟOϩfQ?u72ڞQkdμ~@<\,cBP"DS%Ϣ]*mr }*h(>=.h?MP0.&F;ɥ!=Y*D=nkT?|=~NJJaQoa/",ߨ#υvoiEV{?`LAQυ7Se[&X?_ / >p9Ok3N.+%{mT,xHocZ6L.'svد oG͢_͏WR|&X&oWǣ?sϢX.I_p'~-!w_ޞ不;~jKNF1?7Q %ޞ不;~j[s![MԾB}I.~?\'bm&?ݝvݒ&]3Rd< 4cO.;*žgjRqٯς[;XdQWvOW{;t߀?J6 -!aɏ{C2-߿nons-g<>ybPj+Qͥ/R_Ǯ3{7ݨ˰<2(a|/HęO |r1iHEOӢU4[@ڪͣ?i̴8}'VM,+;:/2ˈwuVvM;[+XQ3(^ڜ]_w$~^]џ3&isp[k3~ PsVvJG kw-֟3~ڪͣ?뗡p9ukpOW_[yg| p\vN\>m$Մv[Yz(ro/\,ˣ\Bqѽs_ir#ho#/șX7=ǯ>}+~};OQ,t ÷20#hcOڑms!~Q[<;귥W~7=I+DcǸbb~~G!JΪ]]'hk㛞a$eW >pQ 3$}J ͯfBljNƉ뤮߂>:=-|K\юߧٗz҄5,,?!_-zWDkSȋkV|%fh2!8Yd8'ȗs BLol9ڽWww\B֜~P 6A>O'?_hO_u,bB֜~nSV>O'?_Ms^>pUpE[N/wJՋ 891P; 1P; 16Np|$aCZ,,+ȵqP "1~P "1~P=\N_]"/M+] -7|UPg lPc`# ؠ~ F@GO.8.l{E|NnN;η0pAP_~5:`~M_*V&/H>/>kWO^msn/"N(:O\iGaeE/ԟ o_&X?_ BΊrkFUNJE{N{HZF˩'We<3ڞ9prYю-Iߣ?sϢ͑.HVcc=ˏ֓J_޹_ȀEӿǛ]eK/$}O]ď%?$Q3Roo^~kMԾB}I~~g{ߎ^Ҟֿ7t_V#?7Q wgE},") л1y4xG⩍Z@E $sx{\,x7(\kK 0F>yڔWg'77?Ϡҟk/ __;ćӃ뷦[>\_" IwQ_ٍm.YIY$F{GYeJX/竰t^ee4p[<+VmӅ'Qԭc@W_[yg| p˔xq/mNI/{~^]џ[_`puGZ;gCVm2X?_aXNk6v__{QgDk eP\;W<3>?pV6>s9'NyG}L{'YeUX&-o t$2 jB;ɭF6?֛}bUMzַGeMO8+hK)ꏰc\ 1kܭ6xW>ݜ_t_ZF7ozo_!'wk|B;ɭ72g'hkӷ'~X?;zfQZj = Nζ]@_dwiO.D{RAjE'^7 O1h |*դP!W_Ker Cofj=(2ZE/r_d)fp?Gb|g~p~gp߳o])A>O'?_RhO_u,{ 1l\!ȏiWX+pe/XN?߅@q71N?߅@q71N?߅WS#Lh_:%Aw `%Aw `o8!C{p@G_IVлwVP;O0ﬠ~ w `?\>p9>'fz7 l{h?w {~~@P'ogpssM/2_8Ccs}b4ߐ?G/ gĠ~Qo 'wi_ [oX??<\.a&0)<7zvKļ>9ӂG9W`q?o>g/}AT{G~EnX>EQ>8A~E%(7Ur ./uҡI'g~ `q[!t!O5{8X?~uP<mKE;|(jUG{MaߙI99q Z~9(ws;{Q~iXMK/O }O ? ;,.=,EoeY'Cݐ/OwίR= ok{G}ķ՟!w"1k7Q?w fo_%?O[{ wb_vCC^~xdsg:;`q&rXr9 { ).ir7{z{KH09.ҙɅ&ibd_?L{w|{Q[6O bҝ>9L:Ce?8~%u$pӎ+'ltԯ)^SD7G}4bT@ԯ)^] hN6bv?p};\NC,;Mti49ssq ⚮YWS2Zi'WefDdҟk 3?\p^ka žЏڔWg+6/oIhQmX?L)Ime77MƯzF{F6H?g);F6oG}X?Cm1\p J2NI9ј4iT׸4{ÄV7JC =D]Bʮ9d{C-;W<3>Åϗ iukpOW_[yg| p`L~vb9;W<3>Z X?}]DP5|/귺w:nN P>p9$e9\UcѯrCVޑb4GE\_OJT`Oϋ2?5-L"$E`N.gp_$jǟdmab>Y n;&/ozO%_GE[稿W)Nr+9v1wMNk}g~%?pN^ٿ_V{߭_$jǟWCA~X=Ik)I`.*I@`J[XIh۽5M-}m8$wou?MI5ϵ_ oۥ ֑Glqcrgr0n7$ҟt_~hI !7 W^'~PL_ן?1~u3?8PLo7'XaM~{e+?@M|e9ڽW];;D{8~s|6g?puh'Ľ9Z Yco ho{!Pc ho{!Pc?x2XqɅ{*_ckQ@gP $_A$_Ap4L_Ehw.4܄[1i=v#lPg lPc`# ؠ~ 8\Ѝpz07^S3h0~C|[1 pAP_~5:`;p._C} u~C]?߰~PEswܯ>9SNh]Wo_tP"^|#_ԯYg~9A~E8\vqv|/xSW;JPU_*4bP]X?_=>JP~Y#u*P_R)^z7}X8s WX?D.ߗ_'w{{FM߫/]tc#'Ӣ'ѻa)M1&@x>ŷZmŇY Ho K~ɷS_=-.NNZ/_}թզe? .n?߷_ҿ}g9GQ_ٍm.Qq*>eosdQWv~sK7o-?ib߻O |rQZq{U,k< b*^MkLv#Q3b|]Bʮ9Ln}wꛡ[k3~ Pڪͣ?_g˵;ωrmџQO[k3~ Pڪͣ?_g˵;ωrmџ3'gQgDo`g.Sc賂k5VrVl\:|_X.n\1{ ٗ\_R!K36xW>ݜqr97ƧD6WYM⃅W#ſIn%g.?W7y?8~%m!y+_!'wk|B;ɭ귑?,7yɍ?DP 1X?Q[#ſIn%g.?m%7VOO˔brDϓLis;NlӊD 봀99²W&@;>`_,ܷb韹0gZ;,^7z-}uɒGlq\w~T&4WkS@2Qο\vU@2Qο'YLJcAfv?Gbj@2]ߚs!k&'W!N,Xk@0AkǃC]G C]G  $Lh_2;D~wC~A;O0~A;|r8>}\4ᔯSwa/߄1PcCYAX?61|'_E`|">fz7` A^4A} X?? _p1N~|ض=8Z~E01|Xn???s}oxqC]?߰~PEwe*5y,G&+09-9F?j iށ~:/Q:?Co/>ԯWij{ٗnGaD$`GP:_Fp~o v=h.h'ta}qPBauM'gԯǬtw[!Ëc `tAK[`{s{?C]P:S9P&M1?X&^K߉0tOMC/ }OI?[dO99#gT?YIκx~}1L C&T[9%bAņ]?7هK|[5~kaN1{"_6^]cL1D5׽hQ2kX?a鞈P_!ȏ~\p9)$Fb'$}*x*t+"$ Hh"6VzпU0hυW89vWObҘyu懛v"how]owW~HDUoԑ|Z|?L;귴]HƷM ~*?7JX?_=>JPtfAtAz̾7i4GZyX?_=>R(_puTӵfYh=$O-ԓC֫k'g=s䰢[&tGE#?,uM3=/;C"=#KOv귆RHUjrOpsد oG͢_3_GQ %ޞ不;~Ruۨ}Wم `DnoHӿ_ꨏX$U|zA濾I`IǓBm'_HHEme49Xq7#jM GoonR3g& ߊoY[%Ϧw=Tױ+ Ho K~F6o_;žgjS_7[/R>3ϣF6o_65}X?Cme7/pE*X?L)Ime77yf')YhQ}t׶I׻A*Ɂ[Vv698W<3>@|a3&ҧP=ǽ9'(B6v_{ԭ??_}mџ/3vD_j@`e쓨?ጟ?꯾j@t@rs"wƯ\[9f3swg„+!-<)'¨'y.E{zJ>+k5VrVl#~rqbѾ薟0v!ENKo#?S'KnqSNQ9/aǸbb~~G!JΪ]]qKM^rcg'Q/u1X{߭_$jǟ\گ}ު6ur+1{dhq^?9cֽOl$ט3a)MNŻ* BXZ H/?b|rߊ2dOnO_u,bB֜~n] M~{S@2Qο\vU@2Qοg_#M>Oofs{'Ưww p/p \>-@x c_|v_xo{!Pc ho{!Pc ho{!>~2C8aCZ8%_A?A]/~ F@]/~.QCH_gH_Hv: z~ǐ661~Pg lP?\>p9s&?\l{E|nj;X?? @} ?p~l-cqzh0}oX?QKG~@O|?~y-L!zzK[wrŝgFaD$`G~q.~oaExSE$= yo& (;OY/V kA~uPoJ?bk{d8װ~N}'Vk7Q?e뗳{U 9X? A~?꯺?^MϩDc'q$ w1QD_86D,Ss4d[>\[IJ44q 4/: zɓf(cc2,E$sכݕQ~u$pӎ-o_&X?_ 돑gG?J_G}4bT@/4Am!>J!rmez0Y޹䊇 Dױ!oY;{ɡ]e0MG3M ߎGEۉ/\2z]ߒQ~g)^z7}W/X}[X?D.ߗ_'w{{FMko{vZo]eK/ >ˁP'}e>,DGꀻh z=WVJioYSOp/Zɇ S߳Nn77?-\_" -!aɏ{C2G(j+Qͥ;7>˼d(a|/H$s$?\~Q~<A؇=¿"mX?̑@ar=eK[ kYM,+;6¶/2h|tTv)cw X5-\nc;&k6NW9?Eg~HDߨWe귴yg| {rv2ޛuv8'Ϡ\[p/v rmџ/3voȹbgʵUG/1uGZ;gCVm2X?_l;\%ȠwW¤LTA~ӏ+^~M=ZNNqJ9ٷN^(N0 Q,I~g^#O7ⷸ/\cǸbb}_ts~D4Ewjǟ׫ls6}&/zo_!'wk|B;ɭ}>ҍ'Kna$eWyO.կQrE[AkK4'0i 26=![dcX?KX?KImižJB\hry/A;+_`'wVP;2n^w\=Xm g9.k?uyɍ>V`upybzL4ض=8Z~z~Q~~zcfK_}DSDp|U¦,x>|kW~ǔj~̟WۋO?AʘKP|oX?Q?#o/>n:Km[OyJ 3>?dU/Q<_iauM4?ԯWxϓ~7?`}a})}N _0q~6*r6'vs~Qܜ RZ7y;_O2xҽ.K>My}ӧ?h9/tAņ]?7ʙ⣕K[?)mU|z5~kѮO!~>oJK}Or(u!GU^9cLD#|$2mħFzf$nhh6k7_]n. 06,[_l ONP[b35p^m~?kZ[SWRoN?/MZXHw(:O\iGwu?`L٭~r7 զ6Li7/a>YlF3EOvo,=3,ȗaȶH/^a3&ݫM-E_>qد oG͢w]/pҗw}ߙQ~g)^z7}/U hN&jUvX?_YbcᲵrӦ,a|/H$so[?،p4>6\歾j@ Xw9o|e3~ ˵}eDJ׻n2UGLJDߨWe귴yg| 8}]Sԭ)SW_[yg| p`L[Ġ6.aIpcϢ=p>KDjB;ɭt=luFp=ܜzcp_w+mrO/E?m'~TU~eJbo_!ƞ~-#ʧC o?G~ y7 !,맇bY- _U{oٹpI6.ǯݩ>)_S7 㧯vc&o)߂>:m@289qn,Pظцq`?'}mQt_y'O!kV|%f7> W֟?1~lg~p~g.o7]!+Xvq-._vþ/[ξ/wJ/OOX/nG C]G_H|ЦIZ#lMOa. DPc`#.%7g. euu4k7 ) 8y@/A;+TKp pF,)0ηlnS'{~)X5db'g:Y[I.d=|),˿u^?<$?e5߿o=p;P[ 9:kG×&_^y lmL6}Th99ŃCsF^v0 78NXg^/[/΁C=cy/X?ƀ^IoF痑8?ve>!?_?յ~?UoP~ym.dT -_%/Xh!9 }wn+ݞ =!u2 $"@nBۤ3xH-t~l@vUޡ[ 6NS `h{9YsQس(}b0vo+u/?TCAu5;R6<~cmiE{MZ ݋UHutH :_7t.׀$.c[;VBk凿 PWPnVF>{&rqיZbޝ;osA;pH. ؟9Kcgvi[̦!rd}[>ɼ=\}[8IcqPTk}ɏ[\Z3p3dCx-[￴o_~xHMv"5ݓpI(EТL2OpG>۟ )}$Ϳx-oVneho'}3Ɏe^,_k%G- &KWQf{Af4?xsk4)~o+.>MHF>tomG;ן 66a۟|_Ν.rQb_`ξ5EHo~te-sEkGaBN{/g%:}I53 is|O?EvEgO?z˝um$sČ?Km+ȢU???1ơ3 vZ,)ZH}}z{[/ uG/H񿫽fmg?['׻oUO ]YB+pw_ N1H~\4u+twb|bkƯ;?ZL|W~o~G qZE?}$3?C\mZ7u~$-^tT\Z>=aO+0;;w_.>u̶Q⣌h2f29?`GFNJ߅E5})5WD? Qu-O>mU>iꌕ9>}Egտ?QJ{"+nPɾfpό2+Ϳob4ft<[z6^6mw~V2=sdr@1r<,zf,nҨ'sAZLg9)M?huZԷ=-ۥ>=/raZ!:Uy}7?bMif1wkl}l·?+ر#}:o|"bjs79cgud+Ol?2?C!ǿclx:^Cpx:\n\Y\Ai$NM;VzQ+r_\C}Q[k6طտf,}8=--Ʒ'|r$5E(W)?_/*m^?WƘ|_>9]m_eWXA'=!/uUOt5V;X#4+?2R#&B~Qw©k|B7B?U_5~\MO~²#c?Ui}OoWſo"&ֽ Z\sN4Tw~?w|ŭkR 0E}=u,> i DR7bW6CkkIg7)&>}5s- F| Z=^?Uo|K1MhˌjɷBPJnˌ475sgY idVK|Iӿ.EF?N~mdk~̘ (pW}otנ~{_%ai|S?u`W8t'|(}zll-~~Fht#WqG#{Y?5bjtm7"Wh6KnMxlIejgj>=b2?}:St 6uLbpM*KxH>3QƼY<9;7ea"YD?%8gK`G9db"@^x\W.+{U|h Xj_gI-qWyMכ1F|1kH)?Z?AgHdl?߂Ie^&Y[?dsr۟ߒ]Onl&G37/n{g3-#=&H!6m4C(_g|پxUF!*9(?Ҟ(or1%4z |Q?3p5%7OvSܗUCcl?2'{HL?ڣU{j+էmğ/ӡx^:#nhLϏk_ysȷc@;}h+ͩlI=7g`\xzqKWG_S>>acߎr s`ql%_//n%m*epgns{nޙE??Mf`c9?޼>b2?6W\~u@`>?˿>s}C|8\.`s7O LC^-n?qpQt|gYkWc?6? O7y6~=Xb<|Rpt.S"MQ7l/H [#fzi?Wƍ~̯&G=fdOo鿺Q1>=n)|;@?(@V) >:Z9#' 3?l(Ui}nO?5ߞrҋ'-~OA8_0sz"@"ϯZ/?Oߟi3ۿ@:>E'_~P6~}I@:&e(?WkK}??ıZs_|ך_ڋW,g}ճap}|;Ok:hMݟ8\?rn/tvBh򷝟X/_swĻ֧vܼ?X/i!>/@wXH_+޴?Ojy '^dL:qɹO/~xX(>B]e|g3ɟZ {b/ڛX, 'h$[σ"?ƷҫXZ>\ܷS7G5U[o?5AFz q⴮e=׊e,3Bo,I>>M6O1o-jg\k!.zc6OWeG_J{y?_oɺ%!Iylj=ڕ?EȆ)KcI!͏WOh 럾|pq0[Տ:n_?F5~ʷ94^ ԟV1χ+ރ|tiw=K?(AG$+lO>м =@%\]v=1պb?,V}o!f[czMQﲿZibŮŎcgEOu}s7@>+s_?l';JBD[lR3O~_$ޟ mҖtXTȡos8nUJfc&D;~{uQƷܬ0? [~@y=67+A`7ۧǖ_GX SQZ6j%rAzRF@\z(kMcFOsꩯO֠//Kioy{-DU"Ys#R~~yKY8 2ߖ/Ҍ\\+h>~%Z!c工Xf8bZu\_k`?1 d7V'.;z^_/GznrCtV_rck3-+Zmw`?笏t!_v? ߸<#zG`]WD~jW>KŢoq5v?\^3~aQ'O8!yVដe1/Q104^&},$[_ O{C4/ZwS@ [Vwx$n Ռ1Z ]10)V5ql?B2~l?~jWO>o S6o6NI~GMݾ GZR__ǭvNߛt/wЏQ^?u~cg?a~mƔ濪_R> ދVφN &i'ߵC hП_u˵_p9O 'Z|l04}&z ?S殉_h'{8KOojǯS]Q_]c2^~^`ϱh h^[=&8{O5\/Ĵv%-U_og~`t㷀d7̸$WoO%?_|cwggk\;wiWc 53O^+MnYo''u7w޹įf _Ez8]P$qzL{BY=M7~_^]g/S?i|CYF i~wtg 2>˯x/+yތݎ;*:WZ{k忧8؍Zppyii^ f~?^3no@kVY~UFֺqZ5<58;>3M()L/k[&Ŧ5bgd$w]UYc|WK??SlԦM~c'xk[~Vv?v?fg)t]eVe{o&,ug0r%o "5*ovo}k`٦٫%vIrR-\U=3cC}L0c{wMv.ܵ|M57nr&a/j^)G_k bL/[]_b'e҇ b`gon̅6t8;\Yֿ֯] O縬X3k,F|v+~f~O~5h\SZ,{n[SGn I6ϒhg)ROU?D=AN`N_;ǍtdltD?] _1~Hv/p}maū"+? lկ&鴚t"+q-Zԓ_5Q?< &%v* ި<ZbM{Oƒm4^_ ]ۣ>4+Yۇm? 6>W 2ynZPTm,u6uq_@v>M8xA2_@嫢CNk~l~;~`dj/+w%uM?~[c/r4W.bʯ}GXcjkaͯFqJ3v%AVHֽ0X8G_*m_ES߶~ ͏"=7߿ӵ_{f1~e]oW_˶|H;75!# {X?/[.Y}?[ԁkƲ ]}½w\lU7~*W 5 >p| ˯cGB Jw8G؟;?b L<:|7O?zwTd/MzF88)ĚW&=/pJ:!nW Cx\88 E˟~np p9>0wu;gQ>ns{fo?B6Xsس@Css~o>ْ>Gx?x>NP_^:\F p.sgLV~>_P}e2~pl ^\܂?wٿ =ܣ_}Ogk}pc~6,sߋ϶u0Pyg>ߋk|K|8qʷPe_lk,6Շ-. .iqY_OKC8(ծ~K66.ڋ8Y}?[?V+=b;0P;;S'qkti. ؞'S! Oߨ[ȇ76XصŇhC}|/JWOｼ0/xto_ߖ2IU~i/AKþ I׸À"_?KxTۻUAG;U |GG 䇩_s?ҏ ?н\3ת_'[¿9oGp9^/&[xW/]O说׸q-'ׄCI?~Ú-T2YPKϫP?wt]˷VNO|W>˿a{':5kwW9>|\V^\/FB c4^zq?C\&D9w,"UV:X1.pkq.O"NJrqg?]-{ZJ7.~DEJ.:K/^ o/矢7w'>oO~_`B[^P3'wbcT~&ryko˟_8{{^K{я}n|߲]n(U#;pgsHN|X?u3GUayJ$^uUyG23]-&ψOdb4F|5.Pm<5>Nxun3]sC ,?{rg|>>|r 8-_]O|Ws`BeݿP^8.T/- +嚅"?/QAsn9<_I\Ⓢz؍|vȖyW2Ls=+W_w*_Ͽá8~$ZmD뫁z9c~сr#|2a- ?ay6Pp Hw8O-0e&שױvU 'AΏw=/OOte6iꇻoG3|É!>|Gv?p N0'` o̯OY?{i>7ZUg~oϟ5[^u"p - cLh3{m ?xŋТ{<O:@ k֍s:G槧~vO,Y't>:kѾϮbrc.S p9}MQ! fuyGoȮ.|̦@w?_wmagxqdRm>ˠg{־U /~p_K9_ew?n~4ƟWr]>Z-ݭǐ:{r|+=CWrVwgQ=7'zo?x9rHO$]NRI\,tPYäW@41O8 KM>mE({$"~>Qe~_ vז[:\V!okrw9qћ)⼤͝tUc?׻٘gV<{}\'0jBma@1A\u?Kq7{wM֏=\q?cy8G`WpsXdm?xImEr;I}3IǷoQT.xs9|4}/|WQ C?FSR'&q L q\>K1O7bQ~`O.\u.?->0G5? O9YU =_1|ezH]ٶ C݇.+k.vM~%/ސ^w:~{RU %N\'kV*dYGjcwV~a]Ӷ7GfןX)ma =]O\S>ۯa+u}!`š?5]1X&˿_Hf<埡_o..+}[EKbt~Wy7' :f"ZhŘ|bדy{dX~i2މ=wr$/tO/d78_=!DPPR<_bv} YOzw%B?\?~s99\~k;?Hu0.?}5CHV~݂%F=Qs5ڏ~>;k_Ib> |9{sq'ǂۍ2?lXoyC{o2;'}ʁ⛁Ԯ\P-M? %~Mpm,\mYf%ŒA?GY?AfL}F%?וX˿oGc:}sHӿg6ߌ,<™JD^OS-LD]?A;M5kAMm&ߠ$MH&'E.<>ԓs5) ɯؿ9>Z{!um?sOoP~lg_q_pyzr-pوO_ս]X姑^j+ C] |ׁ+~Z۸av>\1hk4_#9p~u"O/ޓ}7>G]LEݟ˛|m?$KTc끓oٷ.lOp$7;&y~fo`K"L|8}&N?.rkS3-3C}v|MQ'`.Ǐ{VHjVE3_/w6m]nY;7[sUpſe7o~0W.+#l.\I/a2^DATINOMXt.%7]X[.Sϫ?w_͐} ¸5~-$DU  lAo7?7Ao(Oxg_i"O~L:\vůʿ#>,]-?ohw駴26x g#%~e _σg~ob깅dt~uޠ'?$mO#^ԇdk|+'ʷؿ]?:VŞs?XRo ugLLc/ϙ ԓsnL~#m.<{ }M+ ӛL۱8=s۟lߵ׸w'?gV՟fȾm^?cLJDz.*6f3?mjZȯiaƮ{Ba9~,7d7Sfж@}-_ 3FU2/Y?~=(VnֵjׄkOѩK7"wͿܠ_g#ć/g┿u_%~>D=1>${dG°1?3ԇkwt6e\2J1Gnb,Fw~& gbR(&Ie}<=!oYMR} Y>_E2S.@7TMɦEA>ht=GO?Qþ,G_o}/<Ip.9=&# TR~I(߅_5?s/2d_ԿV[7x7coW?K[P}p`O1G5j| 1;?(ĸlw7u'wAfqHz޴hy3 o٧cF_͈RCs G<⇐X"g9.[_]eC߶Oɛ5N\$O}$a߃ [>p9&]zò@|_{ӄS߳x0 eLIT`ErB,Կ=htcp\NU;!oh}³؜?@/T-:bOlW,'wM=ď}I_\Z,.(2/N:OO:m<6ណ8Ɵ4ɞ_*'/!E1eL#~r%*߅_V~ԿU0cBmCOud[G]_9kΟC2!G׏D{b"OJ|'OZ :68ynQkןhQ[_Fo}2z 'Σ߄-b*^vX_u̟#鑟WxC߲O篒?ٝ_F}W>WWτSq&L.| |C/n8XS |8\oS\p C7O\mt}2_?F?]uk~4G!?5A.p m˂epYCW/Gq_> $=N -A^p S.ȟd9'vd|zb Go&jaQȺ{x/|f `S38:>WϮ94`S3p_:kѾϮbr.SClt=dv.ѾϮbr8˒ڱWDEo"SExtdI??*~S}g{quYc?>i~#wpcsVvLli|H=eܶ7xtߴy#wpˋ[XNTc_\|%חsvH%G8+@@y21 ߳3E_K-3ou޳}ߒ_j:ccCi[ؤhn.in?MUV&ist ?"V|J[Ec)CC>Ϳ. ']nS>ufM3Ga:ϬwMW~*s  @}#h|AVI=F7, J}sytY*]5Sv>Jֿت=aO?+֟?2.ԇ~>g?貒7]/Uk'Fw)pz}1vUz:\ ȉ?'%JH⾜GL.e8>-4q rճ8~p6nnwć߾˘b{ЧLｊOӆ(f0N?a~,Njl=pm,T:7?m=tb_ߠ,F o>#钟߁P1j龹}$ {3R}oFψZ_| srsr+>=bT_ԎOogOa~G˿oGc:}s;?%?#j}Yu/y c UߺI{8ʮWE?\f4',Nybph҈ >r2Il_ŴG>39,о~isyOy&7v=CL58}貟F41nO~i~y~fo`K"L|8}&Nb_.~tQ{jF7>}E8b~f8jWɁ-귁vY$.g'?Ma ć/g┏$õk:Ur`KmS`.&75~m[or[w0y,`]) #lG~](#,;M}4Zx7?B2Jc/#]dg1dWŕ׿!Ɂ%F.1pk-9G?o|Wo7 9ݥ>ʨ-d}lЏ?2]~VNbc9-_ω .Qv*rLft'AK=qk]fAKGuv3>_b oN9YŴ=l[Z[׵}Mt#~GҿONlЯ3֌?aɗ3q狫 >D=1>${dG°1?3ԇkwt6egяźwCǛZ[WZ~8}&N>$]#)?~,ֽ>|W sMn_|i PyK$XR`?@Lzܝ_ˉ3j&#>K`Ym.OzOA?k;q(?0)D?i߲O8ӻs G?reпy"g9./{qC߶Oɛ5N\$O}$a߃ [?|=N][??OA[XW3g$il]oۧD? G= }}`cڵ['cpyfI%~Խb$D]c6cU4Dz~?/&0um$%wL. 8~ra'do\FnuU/W>} ?Zcd[a5`?q@|ìˋU:ψ{ϑOsб/R[o}[1>GOzvQߕ3!ߔo3kڷ3}&X]Y淳8|}8gYTi [Ł5f_d۹ɭ&.QV˵m_F.7ލ|_|W3&J16\L?HK~*s  ߁pٯG{ȩgO;1gg3Jo..+}[EKbt~bz̋+%~kgY3%N|>Jֿت=m]?HavOE~B6\}te%o^t(wORLLc(Zb? _WG<\gdP&^%OSBS%9Pr uGƪOJJm$.$Yu`ofVA/7'yA"wo2tu~AQ@}(t>=\R}oFψZ_| srsr6߆%#__#ڳ~GR}Q;?Nm? 9.ԇbmQHe?!7'1(W'Sד*;'KEd8}#.ȉ颞<\ %R<1{V<~8}貟F41.>'Z@87u+2[suaɗ3qϋW^rkS3-3C}v|MQlJl/Aw/WɁ-귁vY$.g'?Ma ć/g┏$õk:Ur`Km`x|ݢo.lO? c%a ؿM\fr hykV,0.Q qAɫt?߁wo@#_5^7굿mKP])߂OF8YmY`#߁wo$?7Oie?\2|>N`VW(RAϧY&(\*Y5bOѩK7"wɉ 9žA[38a2K8UX{ZJQw =2y#a捏õk:R2n|i?ꥍ#ߺwCǛ~i?~vdq%LI}D%H6?|MGjS X{7A}ո[u%99߫g=L>@/8~HԧQ a;ӱkǯf?)Z&n8YvAxw8De3"W >mH_#?ɯ?/ϕMtG= }} R㪯1.~zRv1w3bZ_fb}o(?0)>˿6zjFD? >mHV!#?H?$lzRv1w:cX=ӱkǯf?)Uia~mF:KC0J{KtS&p=MfuFm_=~ '/UP5:=F=lW>d-Eb{*O}?ٝ_F}W>WWτS7/]~!jpԗ+_CdQmgl -g^fF}@~([_*]@3G~ْJ~Die}Ps^fF}@~([_*]y\p7p-gxq_0~צׁ\` e0-x \/,-?p摾8=S.}g!D}\/,`pl"}nw@ ^ԄUu!گ_[m*$\/Xp@7ϭ;/$>&#a/%*"0}R7y?7pz:Y9\2Qeq= z캀N _\M 6:gϞzYB~-k#裳틫o,6] 79 gO,Y s/Et|*cŠ )2HFz_?g|quY=Nu4n;Kq9 d+?;Wu gYT\pe'owO c_\)_T!յ-e~;+חsvH%G8+|\G,?|.cz׽Iw##K6'bkj–~ղυ"?O* لws3LpefA?~g~yw#鶜T;'Z-?b zխ_f~[}[EK՟QRLyqoum,riߒgCi[-"<_ԇ:EFÊOic,ehcg0Omkؤ_? qw>\k4>?A|zѭ%_?1:{H1k3kjs?4`]j`#@}#h?8>-А\&69u~|D\>Q?Bb6}\Tѵƥ}iYNsr! tZ8{6L跟O.ȾV,<"HdP=uې9{+kdC{W?]/j7_>gįsYVfgd ݸX7? ;P}4F#7cP3֗%_ܜ0Ơ\?z O_O.cY`Sؠ,Q@}(t>吼ybY'\\~)hr_.{(ZFh>枚&6ϊz|?H>\K^F6?ŋl]Ur`Km`x|ݢo=CL58}貟F41Z?Hn*N -:0K8c3p.{o{Ҿ~iA~""L|8}&NL?tx\_'X~OB\edrW_I|=Ln,ߘ\o^-?,ʷؿ]8Umۨ]Oie?\_JH'NAfX?&B?%A?NVtw>$][9)(j?|>'6SG׿ _? qG˿!8='=\1誄0ɗ3qRAaRKl%?Lx#a捏õk:R2n|i?ꥍ#ߺwCǛ~a?~_,zPd݂85ZjStҍy~H?9A[3 G'_)G]dct6egяźwCǛZ[WZ~8}&N>8$796?[>Iđsɥ!W"o\r|]ɵo4pӿ'bp?Ldd(?'_G~ҟrby_Ar|qU_1>b\8vn/0w j}z濡Xħ w|-t,Os߼ z?OcC~>%o&R|U>?Aɭ[Ԯ]L?Xħ w|-t,O3??46PSf"gK8\#$AlU+r0靵%  [k WL.A_iL.y&65?\IO=+;|毘 |kFh&O]~!jlZ@DxKc@ί5j F}{Odu{>aѢķo}`{w ӿ&z gArs:ܫtu̟#鑟 䯒??Q{vY+׺Q7-J|1֗ Fw~0ko1u̟#鑟 2Pp2 j/@ܜlxX?A ?H`Wp8nxOv;G,5fS  N pe`pe [H_H_^W9htC_㿶uUH2_$@29@yXakNܔmƿA7֗F˿Wif `Sqt|>] asi|6:gϞzYB~k#裳틫o,6] 7igMuY^Ўυ9WߢYl@n`s9ZN,OCWIOq~gYT։x>VvMI@b/\n_j[$M2ݓJa`ra#`um tߴy#wpˋw ?_ ?~gQ=??lgg>t.OE]W}KR|wNuu~\nͯ2sT'4]_lU?&ߒ G/SM}j?mևVXǶ@ #~CI}rF]/>c~%3 kzt"?`!?.ԇ~>Ɨߑ`cORLLc(Zb? lׯZȸ;P.5J>wtVQ=~\͇?a2O"%Z[|uP"U?icreº܃>UxԇkwO"_?: 7pI r4h/yis`K{W?]/j7_YPL?a~,I'߶B+~3s/I7. 9b2f32./( ]~>֑GǒaCnNcP=uOҧ'UvOlڱpm,)lQ(tw>mhZGon'gD/K99aA[ծp {,F]\1) ^&W݂_sEq~/GM.Ʉ'Aw/WɁ-귁vY$2,.{d #jl=p-6e??ib\6 8Hn*N -:0K83@]G!ﱏf$9Mx#g[/gp.{o{Ҿ~iA~""L|8}&NLa>\K^>t.埐x!q#%95)x[rL.qk6j5~|ғqbCK?}Qp[F?Q稥_JH'NAfͿMlO= a Gd8YmY ok~yq@}HƷrr   Ocwv[P])߂OF8YmY`#߁wo$?7`2%MU d&Sx0&CW)P=ojSݍ&gʻm6~| f~Poo!a0G =lYg~=(Vnum_ůf?E.݈o̿5pzDp%LB~}=-GHmʸϢuv3?7Cw dq%LI}D%H6?|MGjSݍK0pL3sG GigD%MG?@?" }} N.Ɖ1~/㢏)k!O`خWYXA?k;q(?0)>˿6zjFL:1ECel/^Og@~UcC~>%o&R|8{+c-_Sj.nOg~,`S>}:mՌ'BW}()y3O}BG~~~2Hnu_`c1nkHq0;\oJ؟"~4}x~}\cP1Lm|}r/L*>og6M2 Pc@ί5j M.{?d6o?Z7 >:ȏ%oۧcD߂MBo~3J&W?W?#?G#?AǾJm!ogGO] 5)n`ԇ= Gz%L[0 D=H~z9-GOpeA?p&#W{Z| aװ`^@2߳&8\vkG,5U;t:/!3 ?`W 8."}僞оN7Dk[7|Uu\;g ?h`W 8.f2~~nʧ#F:Go~,%3*^JUE`l.^>^T~*A?`s9 Ԍ*0~+?\d'F1틫o,6] 7iݿc|p ^j9˒ڱ0G>[;M N?S/KVh_ >?>hC?ɜ?~gQ=!x>ֳS^ϟIzJΪY||G,?4?_ /GsvH%G8{QdǾzS~w* 9C`k[8vV/;Kq9 d+?;W8|}dp~wV|3~ Ү/wA_XLhn.in?vnrk67OXYX)ma =l 407w)n⳾16WO?a:~{T,d߁pٯG4>-zo3Y.貒7]/Uk'Fw)ǼR뷺6`.QG$m"?`!?.ԇ~>g?貒7]/Uk'Fw)pz}1vUzzpvS2ĄDo&QzԞ d"< ڳ~sAP,w_e?=S&^m<~u&Zh9„:#PG:ɷP6,_:|IqoQpuY7Stu~AQ@}(t>=c7gD/K99aA[UoC̯ Y#_ev '߶gBC߶ƨuvJ~F_؟GϿu]a)pۥv8\ə?%^aXg-Ʌ&0yc%ďdɷ@D6qx|/K- 3[suaɗ3qϋW^rkS3-3C}v|MQlJl/5^%g-f}|貟F41NdS>?l8jWɁ-귁vY$S?UrŲMN%"]) k_6I^-8zrkykVѱM~#Ky~6,F_l5?w>$][99?W%(jA/zoF/Ćv~J(`ky'A?NVtw>$][9)(j?|>'6SG׿pr7i0>5~`?~vڔUB-ߴ =gY~=(Vnum_ůf?E.݈7>l)̿5pzDp%LB~}=-0lG ]5Mw7zq_T6hKGuv3?7Cw dq%LI}D%H6?|MGjS X{7A}ոx3)ￏy IK@!zo៹m.OzOA?k{]Y0)D?i߲O8ӻ̓ G?reпy"{o>#W >mH_#?ɯ?/ϕMtG= }} R㪯1.~zRv1w:cX=ӱkǯf?)H$`C߶Oɛk=/{Arǖ)kkO3?0) >b/}"R\(+BrOrV1%QmR&_'V @L.Y׺*P[+pmuwBo=MfuFm_=~  gr~1_M^~F=">BZ=?=_*>ևle؞J~gw~]\_1?M={ڤ};g"S_.{Z7s}0ÞEo=FRM-?s$?=ϖUw'O{.{Z7s}0ÞEo=Fζ\`5uO+\ ?H`W{vy/pE{|#_̚.]3g@ O~p `pe`9r(0s?֯p t)ov    :x,vлKoЏ{ I?c/%*"0}R^/B|\*?fhx.A?`s9 Ԍ*A? ?\D~vw. z캀N _\M 6:gϞzYB~mk#裳틫o,6] 79 gO,Y s/EtC;sqH}rӃ*=/.ljC^Y: o_pvH%G8竁||G,:jpeQ=_} ^֏}q.~QUs0V׶p?>__.#wpcsVvqrqI?8\}L 9?]ӻݿzk~ UL.7ލ|_|VM3&JQ16 Zl»ww>\k4r/zoF\k4>?A|zѭ%_?1:{H1k3kjs?4`]js>>LE&] T@Tw!x~p&$JĪxгh2/uWpghAږA@Ez}~y AGɥv%r- ]~>֑GPNe/7wd_+_ψZ_| srsr6߆%#__#ڳ~GR}QYZ}? %~s'߶gBC߶ƨuvJ~F_؟GϿu]a)pۥv2|>  sԟ)?xL'0"OnM.mt/ - ['FV'&>|>/G!ﱏf$9Mx#g[/gp.{ٔD/7_t]g_t[wo3u>H6%ďdɷ@D61? \Dp%Ldc@}v|MQ`÷нJliU ϴ[ te??ibW&>qr $#K|1$ңw 6coIͱqrYl\bј GdZqzw&~w>$][99~]cx[rnm~k.B}Nlhw駴2H{ү %~e _? qG˿!IL9G?o|Wo7 9ݥ>ʨ-d}lЏ?w\AH&mz +z}CN݅?DϜbڠp?NM|8}&N|qU!ևh>枖G]d+a 6o|P%_ӑڔqwWwwK۠Q/mֽ>CHa[g׃"k5l]ia3_K8cKltԦ,Xnh7x1Yq`OKt$JH}%"mr 2˦w3hA}i/pԂTN&n8YvA]& wKOL<;e\_⪱!wՇbm7)>k?=NvM6}bE{g _ )'$W\c`_ E/_Sj.n߷vvOlW+,[5 &>{o٧cF_͈RH!wՇbm7)>*D{g _ V-_Sj.nOg~,`S>}:mՌ''4;.8"ѻ2-r}ɝ]2jT u[j/U@7<{-3/F5e<_*>ևle(Ul_%?\IO=˨joʷֿlf gO}i{_!_oSI6lc]nu3>o ?Zc/|.aD߂`c ?G#?Al_%?z첾W9Oiu3>o ?Zc/|.aD߂`c ?pYp{v j/@lxX?A ?H`C»/,-?p摾8=,}TBD}\/,` .o$}"}k H_٢= ~mmW!8  p@7?\/;g+q,z|9E-tNs ꟫st|>] 9/`s9 Ԍ*A?`s;ezYK= 6:>^8L 6:gϞzYB~k#裳틫o,6] 79 gO,Y s/Et$`vF?\;4ADURn8rYߝK, :o_pvH%G8竁||G,?48\;e_}irO0am?~gQ=??lggJ/;Kq9 d+1O˘a 6M* ل7KR-6?I:ou?a6.5{?˻I9~+y3\nͯ2swtVQh~xc^\)[]?ˢ)qZPڵV59hK+O7S*Kzhuao?C/> c~%3 kzt"?`!?.ԇ~>g?貒7]/Uk'Fw)pz}1vUzWO?d\}te%| }rC !$zYP\sAS"_@BAMRN~۾4,'}:}&4!DW˄ukr|⒯a/D*o]Wm ٟKG>W5ڳ~GR}QYZ}? %~s'߶B+~33~n\,sԟŨ#\dgd ?]s]a5?| srsr+>=bT_ԎOogOa~G˿oGc:}s;?%?9p9oN\CG>!M.4e{ci KG϶'ߚ\8jFƅ[Eu6hE*9}W60<>nQ7kߞ!YBH[|˾ tOdw$7KDs\Dp%Ldc@}v|MQ`÷нJliU ϴ[ te??ibW&>|>|&Oeor M.cCI|=ʃK7 pe :k 2_5^7굿mKP])2RqBdi"X?&B?%A?NV#/H6yۚnGscwv[P])߂OF8YmY`#߁wo$?pi.eoq(C)1a>$VM ?H5P%_ӑڔqwWwwK۠Q/mֽ>C a[g׃"k5l]] W3SnDC7ɉ uߚQg8="L|8}&N>$]#)?~,ֽ>g Ej-غbӒgɗ3q'u _F?\7 D|?%q$o)A_bmcrɓF]>3Ngj%&D>1=/ [+?1~/㢏)k O`خWYXA?kʏL| b}߲O8I?(;ؠ_ןθ?ρt'Z# d>qJ-̏L| b}߲O8c#OcC~>%o&R|OA?\ԿI&ůIż%? w-5ڀJlegrEMl~AϺlӷJ?>noʷֿL3i QvoɈԗ~_j} h??d6o?Z7 >o ?gzW0l3N Ё'u4&Wh81= ?`")5E?R7}X*RZQRM[0b*V\OgQw9iBJ|c691gj/u3QRM[0?s$>='~hb 6⸸19 7tx83=:> $pg ~*(.J;s'y^#F?9;{翉ߡ߇gI}~.&}} sJF?򟂽?a@S25o%+m\qDu>޾8~GA?Ok~Kq]'!x FR/KVhGsb}}t{;M' ?8˒OCi;eT}Jz}'X:Ӯ7qtH%GV0%_?1{HskgCiO_AT o?u qw`}#h|?A|I.CS?\>JUr8enK^DUjD\Ӊf1Nԑ{/a8[_ZQ3/K9=aA[+?I-񯲁bOoggC?@߁oGc:qv?%>#_؟GB\\'-aڸo6[fBǗ͠Q/|:qXo_3es}|˰cl%T'߲&tBr/OWq:wg=ua懓/g┟}W^rG9ևh>昒(76"|1>3ԇkwt Ub}S:i(#i~""p%L̲0P%_q~/:VMg3?l5NKuۨM;6LK?}Qpk[L C2x ÿ&_L:5Fdn?wNtkF0ɗ3qU>D=1L}D%H 6>\K#)nE?؎v3>7Zۃ+򟚬g|>|?Ft6e@bnp2 bLa%̠N. [N gD₹~/E/_nm.Uss\ ۃ>3+Xo*c3?n>˿6'3}+ܺǐ~(;/~?q'ρtUeCZ}J̤i_#>o?/υMt8=/{Ar@c)k.f1>03!ӱk_Ɉ<#HASևbm73i~>?A-k-_Ԯop|r|$%8Aoo6Z˷Y6iq]Tf1z6e06NBc@^bq E ks bB;V~"H1~&Y] l^'Qo9b}ȷ1k߂ēdk|3~9YV\OgQw9i*>~>_%>Zҳ9]\_1>M,Y#Zl| $fK>o >j-h;0_} gO!_%>Z?Q?1 dΌY\<g"M8ߣ @pfGgt #\.K ~ɗ~1?*}>G4IDAT֌>[oǝ/,0/,Aq06z]96m]?ӥ82_$83_$32f).sZ.2{ߛߡ߇?,%}#󲊷|Lz߳7g^*>?f`9~t;C1@s?{$P\ޛ}qv_]b b~NC*`nt-d;k#裣'g(6 6?R/KVhGsb}XI(dI?=ȿ7OhW7]z7=?~GQ=v?@"?_ WuI?~GQ=v\\]w^\})߂<Vp?>v?G(?GsV?+|~~sZ\mmu{u\-vnsK{_!O<~}_ \huaߐReRM⳾1⯜iS˛߇Lߛħ"?B6M]X.5G޼c$~3<W]\)뛋J>w[|!>WJqV ؟ePڵj{PO+le}A#!'m< لwqw`}#h|?A|I.CS?\>JUrBX\A3-H $CFj!^H տ{Ps;:!72fgd >]r_PL4w`}(tܫ}$~ {ߒ\6ߌu}Yu/q c Uz]6d/yh*'Kڑd8}&:s? ? MX}4F#jGP3/K9=aA[+?I-񯲁bOogq!s`EnQwm.Y̓үe6;g'j8~A'N?"p%LI_n3S2#f[/gp.{IJlSt|4X%z7կv>H>&g'j0p '_),]5Go/ciTg- hpjD9\1_/p=$I^-33),0.Q8YuYBɫ_o|w`}HƷbr ~ \|K^+ߕی.ZFo/ϙv~J( 5< tɪLևdk|+&r_f sݥ>ʨ52|>Gq;^:-gԒ jX0ͫ_o.|"2Y{n.k, Po8I&݈z6P|{D~fqg懓/gɋq}{cK;?m3C}v|MGjS݌.7ݕ;qj?ꥍ":݌?c(V`&'_)O#.A5Mw3,XvA}g Ejl]OScS5/Cn=syfK3xu{Џz^|~~m0Xd?i[&&,;/~^gs"xY7>#ʆߵ>mIF|8_8^ޟ D8qE{g _ ǵ1xqǵY?Sj]L?Oc|,`gB\-o٧cFyF#!w~>%nf )?+}$a߃ [[>O];w17?񱀙 q}Ͽe_}JSc[?zdr_-ty\cR+2/&T :-(tOmn}:M[0g >kObת_NAFub~^'Uj}]}f*O}?_.|Ư |do_p/M&5c}[?d6Ư?R7`{e~1/|.aDAc ?G#?AzWOO}&5LnX75c_ ]> x,Fܚ Q3=:Hv>j V_qAY·y2Aux; ~`\@aq~\g}pdpg 1˗ޟ#ts pQ.9wtC??A pn|y^oڿ`*JU|#0}R;y=pxlYs|x?{ߡ?Àd ){~ξ]Av垍('ף@?Ok~Cg]'!x FR/KVhۂ6>:}}qv_]b bA@s<-dv?'fo_G׿Gt@B/.Sw)Oo~`{ے#^\)_'v=uߴptH%Gيg#|5o_{8$z?Ԡ'\K|A؋O?wE@/>X}QrrIa [l ?~GQ=v?`?r 7߿o>ePzU>.>#]aqOZ|z7N=omUaVy or2KG"tٯG-#ȧ7ߌ^szIեz/3~y>貒7>V0%_?1{HskgCiO_AT o?u qwOˍ [#=T+kAFaq^É6<-ԶesF% >qϛbɿg!ǥ}is[nگ%pӿ%ӆ~^+Qח%_ܞ0Ơ\?zUoCo쯑ߞ񯲁b\ ,kx8}_k^ndͬ@|u'k?| sr{r^W>1Z_evi}Q; '߶D~gGɿp|Q-Kr/=z\*Ytɻe'g?x+m.4asZwh>昒(76"|1>3ԇkwtKo.q!|:ciTg- H0$S[|˾g'j0p '_),]5Go/ciTg- hH9F3?|>? g:])_31gׄ2e _33Ac"կ7>#Aǖ/ϙv~J( 5< tɪLwZo (1$ft͉ 9~ͅƤEW|NH܌9}&N\ bpLj{cKl%>LxG°;?b|fHmʸŕso_OmG[[vA}"a0Ggׂ"k5n.bׄkW2I&݈g|>|?Ft6e@bn?1Yq{uESC/g̝EH&01xd! fLHmӢ~&Vx~L<8e\߶T6wID[\mIF|8_8VkI_G~ҟbby\ArRZ_q| y)k'~fߜxlخ`YP|,`gB\-o٧cFyF#!w~>%nf )?+}$a߃ [[>O];w17?񱀙 q}Ͽe_}JFDʆl}3G)X\SJuBƌo¤o֚7{X؅Xj6P~>TJ||z6Weշb|&D- >?_ Wy\|5c}[?d6Ư?R7`{e~1/|.aDAc ?G#?AzWOO}&5LnX75c_ ]> @ݳħG~ڃb-/n\ ~2qqbrIx83=:> $pp  7{$3f?A?i;J걻FlS%y |iTN0VY|6yxGQT-WXy<|%/';J걻 ?o~]4қw_ Mq\aZaqvO w!?˥z/3~y\k4>J>w[|!)]_i ]?٪!AS_)gG߁_]V' &k'FwiNQ\N\gQǀ ,7WDP⸆6oՋRN[6??mX-G7 'E6A[x?Ǘ<~tc=gl./j7׿>g!XPqYu`ߋ>?H7n.;s?QGuY7>#钟וbɿC߶(u^J|Fe?!'1(Wu'3%U6Ol#pm,LtX;>mhRG:Վ'gD]_| sr{{g7cW]SR-z7Sekyh-}st"y˓@Ou0<>nQ7ɧ 'Wv-C#z[ud??NTst X'ګ8;3㿃:0ɗ3q燾+CcsLHrΏDmNõk:^B*1Ӿ~ ϴ[A4d??NTs[f~8}&Nf8jppeY2\g~/am ^W+;]\_o$dlhw駴2C2xmG\b2ftq圿[׷S۠Q/mֱf|Pow`?~$?-ZPdum__ɀ&t#~{;':k5pzDK8@\cj>$._ӑڔq7ϢulGPdOM3NdS>G]d#k:R2f|N?cHYz掱cҷ&meOro.8VkIO.=/[\$8E8w>ڵOoN6OlWWYZYoq}hP|,`gB\-o٧cFyn?$%Qvѡ_\_:O$tʆߵ>mI@G~~2Hnq\l  _W%kϵtu#鑟}/R[C_1>M,Y#Zl| $fK>o >j-h;0_} gO!_%>Z?QC?ഘY5 lqqD{AY 83ߣrf;~:.qϬJ4YX ? NJt;>`I Us!?ۺ4w83_$83_~6,sp7=[\E;blꟳp8}8_7gK6MܞS@S2=?gߏ~ 9zYBɵGQ틳wNu7^\`nt-dv?'fo_G׿Gt@sgK,Y|PB??~GQ=vzQ 5oZd.NׯoQ: o_{8$z?lE݊0.㛒I[..|t🦒?SO|Q51>c9$z?lE1rظ-sV]okȲn{Woɿ>MNДvdk892 7m#ڲunk5EjUp)}_@|D KzhuaߐReRM⳾1O]p`?%MsMxև~>Ƨ=dtHf<埡_Y\]V' &k'FwiNRl, &QG?qr&sMxև~>G3tY玛?t+=9Eõ3k'[/ ?8#钟וbɿC߶(u^#KN?ַf˒aCnOcP= Og:_Kl./jGY ϱ,(4w`}(tܫO@ψ,81*o$}c8Ŀp~:98rEufcWzi[ӱ{l.[LEߵf67v-C#z[ud??NTs\8;3㿃:0ɗ3qIEŕ׿g!ﱏ9d$~9MxGͶ'_ ]5G]T8cg+:i(#i~""p%L̲0P%_q~/:VM`x|ݢoq~G_q к-'Ek&'cLGK?}Qpkprgby/O~M(G:Yv5D60(J~K$ԝ E=~s,'>_g!5.D^| o̝lJ<^^c7XD6ӱk_Ɉ<ӷo@K.?/CtH?%mW k}()q3=~pӿ%bp?6q>?A-kcJk}9G?񱀙 q}Ͽe_}JFDʆߵ>mI@G~~2Hnq\ld-rdUu'=X߅3!ߔo߳kF}&< Oygl 5g^faQo9F%L`? D=H|z=OO*Ѻ##^򟩽Z8Q˞Y ߈X\ETP=jP7qE3۽>e?]rq9 [)kAey8z{oVXea*|xaqyeG1  :"qj羻6_Ő| ^Ur*L{Y΃_WїRq9(HO_4H+&2/}10zs?UʮWO~bq9= 1?q=c+,P\>ٻz5Z _(Yϧ쬋1v J:)ؑ#"P IֹOOM.W\~J[ySzyz.Nj_UB 񶒧ৢe胊Vu!(8Kp; X\\^G#URqQ)E.ڊ]t5W).kxcq |<m&}y$S[}epKcqs9썟cI?u\C. s']8n}r_+k{;lTL}sqӶl#a]EALE?']lB-֤szrmƊV@Fqve8%^d'V/.sr"z<(DEIbldWӞ E~N~Y\|c7(./>)r' Zu'\Q7P|}>ng7xGe÷o\X%+.7'0'}^Id[ \Xgo]@. km^C}P\!/½~ T>!ګr}QQƏI$tz`o Կ:ۉچkѿK}6K~]V})3`sy?[?W4/_}ܞŋU9}9{~{`|WlE1.,xmE*- g} erzE{4.53_âzmw'+.i5^#Vqei|{^xXz/ LP\||q9$w*]^S,R*v?No^w~g*= N$.S!yU ,O;Afqޞ?t[q|{%Edb.򳺸\]_B,st.l&C%E\\0ϼ _?4AAY|_!kN| [h.y oҹ¨V؜"T\UE1٭ͷVܵۥO_;q$pmjx ~,Ʋ8GV\@b]*ǏIx楱K>TT]tP#ONtG"}~ۊ 槉<=yNwG/o MaS`yJ?߱=(13|q=A_ c..aN:})ښ|~qs(Fo2ۧ)2qwǗ/#''M}cZOa;g3 oL&+gKGr(3kο~\1:?󧯏h}}koү8 8&[hOA+\T$|a˯Xk)Tz1Gx[]~T#3uVNJERW1JXZ#œP5zZgq9]>.iۙM띹㫅gtJ P G"\aS[A}U_Er.|~ 8v _g+//8B4a;Ӆ8ͷ_K//sþ[~l g_{OTn).SgQ߻?H>n]heŧ^J ]k<+/s;w=C<~)Z]ȷtL,_^k_I\A㬏oưO=c'_3ZV 낰T d8g1Bz,J$=ZC>XKbmyGl:_ }YFF*.9i-b%QT m!nzpBb놿8Y=R.^:3,qta'/E.|~)/7||?\\wJrR}"vO ӾA-?.rn/Lm/.S/^~ƭ{+~G廰KO#sǕ 7A2 M:W/Ik-kp7(k_:7snoo}13~[#yC'َHe6P\^Q˪i\`:!}#k gda9lkcpL}'[MX ]?IZ6um1^qJ,1_/c4|A%?K&};s&\_e|u me~ⲫ?%\[w|"ȸ?'Է˧X**s]\(>'z>[ +~Ke1by+oddh{cba7/U߲׿~tW[/8S'1å G]ˉ.:绾P~^/.{ hj6dˢbXд<{ 5;yI@+ kmk yi epNW@Sa)y?1M^#a3}o93ab;(/ 7+.3O޹\B_1y틣/ ?s٘oBG.Uy+o¶etӘ낱33wE|;9kgƿ>FEVŒ]o>Kwc=*nQkTݡWp2KrvL8 ,8W|qy\%XK9$=j՘6 E(.g)tlOt10Cyg)}LLnߋr!-ԿU/Ԭ1!ȯY]$%d?,4}Y=% m*.MU2ұ?QȡS|dLg_9[PQL=vq7^˿y.#u9-ŗ;wW@O3oesK? wvϵ%O׽(}+v_큁ǭ|?m6<?gk=hڸ'' ˱'{ǸpfpϿT۰%6"QuwsǬ Y&*JE΢)ZN^ U\.~n]c*lLjqVB'X_\I_H+^ ~lOO sy'.3lB&%Ǔ~GQН+.81B.Hp I~E}㲏"1Y]ַ_(%}>HLqf=Xt qY^…>-߶\ߟT쭟<r9P\c${.4 c$~/yzK.ӌ?q﷯ctXv|bMƧqS]ɗ^?i~WQs4Z]'?MH_F.mQE4*I@O-; 1b\lϺdlQZ'ҡ(.tfN|qT݌iy^vy/n]UNlqrBqڎgipp/#~ޱ˗p%G/]/ozf_h.J2~ d4߹9=_@u zlXq}ogsdτTo6'"cm_0=志r럗O?/?Sn1~_ > 1?za юty3y7ͷ8|qr{brzZ.֕E*#{Ben :W3 ssq$Կ&YwQ={{oI3Q\c~,Ʋ8G}筸@"*Ǐ6:ޭFc]7_| EEџQ&!ONtG>mEQUDࣗٷ ~Mh C0ӝ?SÙV\mM?>9`aǯ+_꒨},נ|..R( ˟ڲ Ș-ȝ$ ]4Oř8v _g+/ 2T ~(la [EQh3aB~-?dMЋ˿a*_y׳ ~C߷~ fx,?c"[p BhT.dW30E"]\_/Gׂ T|X_,Mh?W'i,Đorш\X&bGkHӟ9f´oP~l϶ Feu]cY~ ҅Iߤ3l_~C1~[qYOz ˾^.Od6~OEe[X( qz>K8ձ\hGG\pW׺[y_ hF-.˅xh*,؝O̘Q&|/[0W̙xp_ENϙo1B͊L|w.WPW_^0m(.c~KCq\>6x׊/~ Nst~z_kG(.g>=7lU2_n;?sR)~gblVP?߉/-Y'5"m_BOb8o2qMEQ&+Uq'Gϴ~V|uHþK.cKz셻~IH8 ˸vWbſo88^}_.b]7/JEC2]6qz*,^B?3]֟N~e[cV~Z\dV$.Ƹ#HLzr~?6DM..(Bl/,O47aǚcڿ?.hn7Is*.;1 8/(MB;{*nXV߳~3?_Ǵ@h[ys~vz.Pqq `X\.ת(]sv72 t^Aq|ޕ8*(.8 _>{\ ]܊ގz ɶ5zab&ŗx6+(p}v/P\z1 |;q k(./N`V: W,m#Am~r 7KʃIENDB`llama7b_autotrans_8to2_predict_distribute_ckpt.png000066400000000000000000000162421476650141400400270ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDR]yw?sRGBgAMA a pHYsod7IDATx^=U_B7YƕlFE,BD0$`D;",810 Ņ( ntP2!0CflTWU]޽}>]UΩ:uߝ{zzk_D"H$I]p@.t4`E؝t>uéϵiur,Wχ9cXl?r-F>fcYUz+'&T692װ~%,ʰpuꃮ/djXX%eOuxäwJ݇c_"F} b껜\^75բp]=gyېoVLtK>67˥: 'ۨitI\RG u}Z3X.}\c8vpkpANP)ɗ[AW)݂yprtЕ`-/8ɚc]+r[Z*eq^I}=RAQWz"3hx{Լq/ud .2:^^ۮ'խ2Wǘ'okȶum4ѽe_Pt-I8a+/ϯ oMo}&C}-/K97xݵ8,R7kU㯓o*Id&L{m >1ƨ_lY.w& } :_2m%Z/UʘCҿ.O7qKm{1e|lcHhҏ1ҽ +|ҽZCEz(g/;'GIX705q4y5t26r֡뗮<}48g$+(3sJCF;7/C5x_-ҕ߲O$QgĚ.Sc2JS}O\t5)|=uR.09(/.VӶ^))t\vkDo )W]3)w, Y?%XcJʕ}I|+ҽ曾dD]z[IvI+07PԁfOr;NWnْX95n}l~xHig؝yYOIK:{j̛]Z\N@z՝U5tmk%H5{- sY\vM98E.k|Re\BUЕ8 |K׮]p"o+wEh"׋';xhlV}tlX-=w+9( &c (]rO7c#SܞM~,Czpr+te!1| m%k:jE\kǓBrrdLƮ/ɗ'gK  -9q2^k笇vCN&}Oq, _(G__9$qNضg2KTO%)jJfq~>^~\NY|YMϣtʽ8_髆?Pmt=N1?Szb{eƯ}ۍELeyk2W],Df8=,,ɑvrՍ9&˾\B5p UA"sJŕn .ˡx2rp5=:U pwt4 ] hA@f.D"H$&]<{=s O.Nt`}= AՁ/>!8:}]']'WzO B@O vct\\>{. {NxJН}7}[k8:;ߏ_ϻ_%IsqH)RExjqNP'opa惮¡fCrkf7UAk~//1?암)9^?i٣Tƙ_tuˢF::I'wj{%%:VefBȻq%Ǔtw?K]`soxh_@R];qjYɜ윐@r&s'\]AWBGapx@e(_Sg[f]F]ޏ-NtuG/'En+<ϋo\_hHmD]/xx6.9sBWpXvtIS8<~WfaNuN"T/]˧s+AlaݻYt_J=Ӓ}kCl0xw>rhp06^ȥk]pE.?l98D&AoHsH ;>g{i0Jқ./]$_{vܪwuϋޖ.دK,-Y#>UtZt=~>7X9wJ^^d4+ !5G-~/F6k!'dڷQ99 BάjݪC\y?dh9IPaklUgqpxI :T:)ɓgή;>5ֱ˪ZTXq^nt8 B[/bݴ> .UܵOOa'ѭMAAVv v] Bpԁ/>!8:}]']'WzO B@O vct\\>{. ='u_`}BЀ&A}LKI|?$}oM볯}I{~K]]Iu:NLw^}nM.y{%WHutu6 oO\%Kon.ltuɟ:R][gKR^N/GO؝}v)]$Ȓ?1Tz!I~wH&i p4 >ŇOJitt@Ѐ6AGwO^K{oO?5M?pߴ $_)?3ht5NݣFܪ?s{4{&>;Ž5cU>\x:mC*0nPMVy`yc_CKǭ{ .qV9j:LcwCPq$1Q?-N_Qϥ)318|U0GCC~_`fT%=$7w[M9it Mnaܮdo3H ݠ)+>䃲KJgq1̵Ȝo.H,Ǽy4-U{˿;9w `_>5~=Ss1܀%{ԏ{-K-G WCi!1QtTeAϝl0wԴY`ƺ$m#$&&}[ [=itp@Ѐ6AGwO^K|_ &OMӏ+7m. J?!'3]`kMU|hA"x `>]ퟪ}2,KEFY== [أ]3Dy `ꟳaCۗ!cOÿ Sn:1Uǜc?J!/G3n*ܯ%:&40sL8~M{?*I+JR^/W;9s_%{|6=2@5]-\̸Q߈ $\ >8gqATLjp~k1|b{G4GxTFk,:sGr70q?C,+CʥҬTy!ܛD9^D#zml F@'yAC(,m;3|*cN/qc钤ܪoʑyK{\nov vu {KܘGo]Nʙhe?4c[A.L(H>sɞ\,wFW|JS' o򹅼|q,C~ܫeo pXm]*@%o)U9RFx~+0:p{}?l>(*l0wԴ^Y_/9jpe&|ѯUrc<_aʖqq%Óp^^ᾥԤ|0R K.J$ tسCD[{,잨IC,5GWP!%ztʯuMCB]:ۂ=`ƺ$m#$&&}[ [=itp$".M׵O_JW_3׿nnA_t>/W n] gЕ^n.Ix$u&Aq# ~vƭ{_أ t*]M,:^-_olV#zҿn诪~̧/eWwc(I<`1+ߣ?l>[أ̓~<7]-,}C:^)_~h/% Z-Pyt t03zYE])318֤U`QƱs>|6:%G36yӥXJ֛.[7N3 OQ7PoH 'D);Df_;wmv]LƐ/$o](KƬ>sxw_Sxvx %N'xȹI?y=S7H ĚuPZOYH\~Js˝kX$=*s[KDpz˃_q-أP̓7~_b W՟BMɁ.7zR.2\}uymݢ)tyoTH\1$bRu̕gsYsާFs`<ga^hu0mpҖorZyo($|6O\cyЙ?d} cY3do2\*A'zBK+c):ʕM48>)sHa ,L ȟ+/jX7{lL¸“Ur굠zf9==]XםgXkrZ?8]syWIRn7ȼ}.77ex\q:O%n#7`7]7=$0[t9%+%J_jd&j 쒬2ܪ(_wyo gxLMHLIK`R}<`ZtTePڥf;jך{%g}s81\nPܷ+ ןw[l:[=^b~XC.Փ#l˧˳Ocه˳WN>V? ;Sߜ}{\F6yR<+`ňqTqۋ\/3?-^gTEYh˯ϗ-EvW#Y31mWLN,e) &AOBc^-NĢ.zQΙ1c{|}dyns{ϸ\Kf3,Ħgug^vĉ~}ÞkwMF B߰:\r~V >WqL Y{xJ7x !8\=.Vm'XZ 8'HvE`>%mէ_5֓}O.u\vܼ\~+o^F]E X/ef!Qi֞|YeR.ϱ/fqDO&(&ёnֆ2KRL͢Cr50X/jK4lɫc|WA&[̍-nk̵5#P[KnMu7= @(T X\HK!T3EQoԮt?V/|sb]mR,n7ό 0Wk>[F֏c;)n].YwZ[coIU홦me])nVcS6d)KAv\؞sIdtwvlѾM~US#޸㦋#}g~0.V{Jaⶡ"FX>Xl-I ۪@KREc2 nS7˻Qq;vپP.DOWCس/pL[<\x譏D&+\5m2/n{9ڂ3++=G7y#6s9r0qL{wĄDEI!>uq֮N| p{o9~;6cmcʔbxhÞkڎ-ֹ@lͳ&$1&I^!ၑJaztBHh:Jo97OO_[LBofl|Lvx[= k<@:c\#Vv/ybzWe{Wvȇs;.#6}7RT@ AW, x6\Vn6͟=FxX?K̋; nw@p q6v}zjО;r(T\kvy{!/k3b-z٦ ΍9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQ(ȃ#wQm/]πgˋ_{ɻןQw?oe^ϗw/>_>eNkOR(M[t6u\je}~HݘԾ?ָ}cBvb)HIDE$46 ~j vB tD֋rΌߎ+'3'WvcL;-s,WO~wZ2;g!6=ӷ"ċQԎ8qݯosMqHAV'f޼\~jㆸ ?KoO!GQpк*x+חCk<R܎v'd-s/z؅{Y sb;e:6$![.D׉UPl; w[7OI_XnpEn]<-~>#rdXYX+|O*h{IkѠٮ\ֽikm}Ψ"V%V.X9m)`k'B*\F6- Haֿ~"4a%Bܖ}Ήssb} "K MPE(qQ I E.[XC;'!-6,q#E&Wb}|B0;8-qo=.n6-o~giy֎ DhP6v7kk{^ke~=.cBfǺ-v˶n66LPL#ݬ eKϥEi.jda ^Ԗ"h|W;6i-M[֘kkFqo{&|1LHT1ܘ=M gՍϨX^D*ayHĪE4~m&nvA;e*qRxkD] ] .jM6Ct+ј!^>]?Ү.0㫩ڱַ7=6^Wo`{-{HEq^_IUn:WߖA#ns )fo]zԏ9޷Gф$v#PC6LP\WX"8SFo(+S+k wnڥ#n OUX\HQę~x)$∺3VD>֪cDf ֲwp6χmA9VgL5gCqLq֕7ׇ[¹uڳv˶{WwydJ&Ѽ,`)Ȏ b. "nY=[ێ`\>7ɯJ}j|vvtq|/qƱ D!&uR4!JB,ꛡmU㉴;M?.6yk> S/V7m#Vi{Ӹ~jr=܆&ʊìfZk3 uhO~^ͯ{K}1[mz>%nlxTxe+s7>c{DEI!>uq֮N| p{o9~;6cmcʔbxhÞkڎ-ֹ@lͳ&|1A* $^h"ၑēJbθH†!$GaBfc|$mB>\f6fOc ɠ\XyD-yrqt`mVq fMBQoMm\U_}e7oy\=)<{wX>9񃿾{]qpC,ydO읽 /n# 2FQx(xD?GB_H|>?3 -)}3%ɏ#W9ht @%@Ulk9{Ju͋Du,Xbo]IvVsqi1+̤x] \f&"pv9&>V_G?y1s7[U嶅fez-b*!.f xZJ~,k$S49bpkqZ)U\G\we?>PI➷CC6v}zjО;r(`+1m%u\{}zʺ\,W./7c19{B)#HB [IrBCl ,p{[maTbTѿϾ4=?v_&#IDݕEH< *>,5lrykom8[Cy98,}'=񿻵%l ly_缮bNjkU&9ĺrn?0g&9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQ(ȃ#wQz=*=n>[^K޽ʿٽ~("xme(/̉s.ð :Gq.&,/n߳O\ό_Ns#  *$ ŕx}=Ѷq8>RH!`GĜǰ&űq?e<!vYy{ʨ\Ra*ލ|:..X1r!9}kDg$zbN)ˈ~>8yG>qُ$&cBvb)HIDE$46 ~j vB tD֋rΌߎ+'3'WvcL;-s,WO~wZ2;g!6=ӷG*^vĉ~}ÞkwMF B߰:OWa?j{˸YN@>>'q[+{MPq8W;z(jiY X{=,Ŷіqics߭sCsE1 *y%ʹ"otc_#}icټvM~wmG=#EP=ֺ/7>;~ߞ-k! ^Cn}+lu?7Ul=i o-V/?x-f# zXO~v[_b)WdS /\9an]4ib 3F^Fȃo-9ѼN,'X^:EߍoIcur\gOI_Xnɮ۶te9VN!%b}'4y=N.兮6=`65,fΗl3+מ;4o=KϷ*ieVLk?b΅ڿ?IJ:5^{_ўYak?-/~|/e \k$ǔBﵟo3k``\ †v !\CmU^-ĉT/e'q\Ps>'qC}z4//<$?*Rb_iHR039>ߵKk|>rYqG?-Fa2_}H&nK^;U2BmD usU-⶜GsNܞkĭmS=O81=mZ }ˢ|}GtG(cu\(W:Z][gA{\vm[,?ڟ;ǜն̣ x9\ r:2}Yn_X{l3:]qe6o2Y^ pLQvj%|4.vs!Qc֡;-!^k|/_q={Y]9ξ _}+~ȗޗ:?|JBa_`Y ’%X,>\ĆfqEYk9qEG.m+N+i0^x$m^+QᘳEH|ȁٜiiY}c;'p:3wOqXχVmm6,Z-sreto3{c`M'H7kC%s)fQڡ YBqu6UM+zZv |B-5fښ%n7:q`J5qFl,ឋ=E5~3ܡ\.D Ha$_)?-mqDl7ANmJ?ssHTZRnpQuT+fնe6eݕc˽z+c(7e_Qb>CS;p÷%2Dv҅ͳ=Rm/;,?}،ϩ 4a|jԳ7"4:ezܴ \<5ƽ_9039khl?l dyyXs\) byk<ٿزy{ؘF1-n{qQqI|VBbmںM- bm[@n{mmz,P;Ciqh,?+vuqDgC:}U[GO`вwp6χ{۽9݃e=yR<\3zgp\_vk[Oy"sg"*GLX6z`ل2-xk%%.a*ٶ5]9s,)B~iguLmu}?TOH5|R#z4,%Dm'̾iwww㜹i4oc)q[a1zg0>ͩ#5;X4j'C~lb`|⶛s;]s&:źsߗ]CfJ&Ѽ,`)Ȏ b. "nY=[ێ`\>7ɯJ}j|vvtq|/apv(DUh޵I T_烳!Z #*L \$XqYލuGr匀ȉHC|Xf[Qxyl#w"_ێ% ۭ)ys^ Ut [^o~/9%BNJ\7l7co8pdyW"h\!fmw wWݞbmW=11Q"ٷCF)x*zvC/+E={L%(HBJ4Jd^cPsvcIm06Y[՞ueӢWsivwm?=D I/{2Gq#?8z߇^wZ~^W__Y q;B|,p ]Q' AH16 rvlm4*Ɣ)c#^<gm=j׹c[sؚg' ҂^Pۯ`w?Kh uT}3̱XyJl/8iS#k{2(xjBJ[A%#gy$}c e9:)GiWݷ:'mZ+ =\/U?WGN 'v||Ž\[;c\9m*X0v̏"7oej5xiBu\ZY #i>rD!ccC#\-g>y'}z5@7cwm~5u06.fn?i.O=8rZ>7'8IX1DZF"9ЪOe.;mD8 Xy?9l;\#sysP[Oh3Rd6}C>γʖ}1`7$8}Qq;Ǵ K8un\GDZ ADaQyq\h1⬣F:cGZx0n e/m_h wk=aip7v(wh~x\~ z+'|>&3CE?mLhr<5{\Sif,}~/ykى~19h^|FԷţ"Aj vTW?3B^υ^ w,&7F˥c \9Gv[1>ϣ<5Vsye^s;z>aͻ6=fZkV^yd&^o/zgVX 8KuFħ$Z(WKIplx<=s됇x e!x"N'|?b׿c \sѩWǵF8s90[{cW_GY:K1.ڌc["0je_\jS|6 xd>0h렘:FfN;#$0?(d8ZbͳQvh%pϩ=F# oȺPH\ˣcw㹭9ъےRu=>q@AbPD8X6{m^>B{7ע<;Ǵ}Vj|<4}<{Uڮ봱oqco s1>ٷZed}'EY@'r.Qz=*=n>[^K޽ʿٽ~("xme(/̉s.ð :Gq.&,/n߳O\CۇzO.sKƗd 3!n; $&sЋIh m4]$s} QRbY/B793~;fl,Ϝ \ٍ1}Bbϱr_=]Zߙk|_ؔXq}(QEq_߰皸k67=ӱĕc>f2c}m0>?m쁲S܎rm/;~ߞ-k! ^Cn}+lu?7{ҶfZ\_~ [F$Kq;fuϿS둾w*q 8W|t? D!աd5:A4֖!׉zcy}vwXiřS'?7bymvGu?mvX Mo^=Kqy+}nM6%pʵ'?[u,[CSbZv.ĬWvw7!٠bG\my#|9k$ǔﵿo3k`xLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧ; Ӂۋ),ڳ&š]}\޷,ɇwO ~8_ǵ~O}z塽%n|MmצsyQm< Wå W#(6èw[[ks}}-c{* G_ny̴ulg恐QAcbh7r5fsݲbl핻ZΗuQسW5ؕþ٘&OR秃CcSS.௖僿S O}gM'H7kC%s)fQڡ YB6quM+zZv |B-5fښ%n[}_ZSv(Y}e4B;ҁ>K*ՍϨ?XX7ʅB FIĦ[ؚ~SԖ٫3/mG.JZGE}No챝bF[m[[kS]}1y߼1ױ2"γBh }OMT}r3Hw[)eYvuK;e]ccRe̋Ů͐ jŽ2F%~7ohUs8sUΏ>!QZ>H#|/gjjp'"<$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@>\C8;j*XTY~< f+ʲb<^߀(Xذ0 Q[ŵKGܦ?,yl\{0lk9XNݮZGf #e[ ږRuKHϋvDC(/ֱ1uZ}[y#&T3S F sݲ˱s]'X,̋ɮM;fdvC/T1QbދHf=`n}>q>,?땶1_el$|9o48A:8y/ԙW_whȹgk $x1G9w_jm/.*!ϊ]^[a]m -~o9 q@04و=}o*S17cC8;>C"Zh_LG41x:DcZu -{/noh|Xܾݛ=Xs'ų0s~ 5aiEܔ7gh-FPLĸ9[eɵ_봷QF/:Pe=s 3Q7vb+c6Xo3-POu9>^'u$q\<3KK8~(a9'8A<'c7-?eY ODy1(h>p=d'˂՟GS#c?{T yȹr6O1qi<9]_+{!}_21%nhބdDži17"nY=[ێ`\>7ɯ&S#޸㦋#}G\پy١TT:ȖD왣y^XoQaRJ”Rvmfy7*n.}[|*r""Vٖy!^zHWǶc f 9.Gvkmvq^Ĝ{C"֯כq~Q>Xl+)Z~/d^HJ>)lqǘ9Y\q!n;0&Ť= ,MqF}F3IS-ZBs^7^ox}7-6KWgVN-b.@se,GXc~y\2#1@1C<{Aik3;Z{#>~NK?S6C6kcAsy#v"4A.ELdn0o@~pC )mӷ/g wM!?M|jX|) MAIo1v1{{bX!O$|~c3މsX7OD39Fi܊[l.oy?9뼅%Xs}!qpD!íq*i m=χeUZIWH__V}1ƹ<텬Y1#DGTmsBlj[~| }/5rfmKGlӊ!s"1cmTNcv7|#}،L9r.kC,斜B,xYI$4"k}.nK=4 n΂|u[~xs KDCbzKa! } _G|vZNmɳmYE'p3:>c_ZyA${Db$!PHǘ{ZޙmY8 =l%a48-zmݧxhYubFD} +b&|f_5.k*< +QdϲO)%fmU1y67ʲRyM!c彆37&k y>i7kfyRA'i J́=\-Fc؛VGLk1/Wg}Q1W7`N,!_Gȴ2oCR5Nسd1ZqD+i|:vEzy)i͑P9~{*DBž/qE˰F_KO91m}#q??(u6s\vrB-=![yY-"93ַ<1+>?[[\q(zeG/bęٖ4'Pr1e$qo'[~>㈖\^Np`:- OtI%incyݫwߙXȄBp$46 ~oh(){1!3WWOgNƘ>!1wZX~篞._̵dv>BlJq{oM>u,qAE(jG7&n|d t N!h_,WgsqxGz?x̤zhs-Zs9FG<4oIFV_;}[aTZʨO(31b^_iEV?-_s. U㋶FD-Lf*fʞ'ɧ_wr^-5._V82o /cr6ڼDg{Š_QQ<9V[Ayt3FsadLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧoHBkڳ&,R^A/x߲_'b>GA eq,k#/񈾒kw)ϖ_ ԷY~C(͋۴nZ%ۅ*`k]-_=Ǒ.Yevw{0k6wEԯϚ1uWʹCںoЁvH3'fkpcl|8D-Lf*GVܘF2d#ޡr/Sn;ON!&j0r^I㉖Y ~7$c3h?r3}~1/on;m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qb6(R X̞&hW!d4qhOhc7'4 ۅ()+姥-nM5q#ک-Wt"] &(e?xV=7BKs6jɿe㿽T /n{D+bcۋY9| #}frɈiUŖ1ᯅ?s|yr%0qze\vGcP#ibT~f[X\GI {-Zst/[ ?=.1ۃ'-%suJ⼏ȓICC_skJ,jP$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@b{(MDQJm)|+~xEb'6,? HlbIGܦ?,yl\ Ae`w}$:bAsIRUQ})QǠj{̖+w!zs&o֦u- ݶJgT\g# Lc܌qⶨ[[rMVkm2/ngl3ΡmiδqΔb}51=y|c6=qoݱ 6aB:\p8kWԉB|>AqkW~OVhnیieD.clTK-wVvsv[1e:<+qE IDam*{ulw\:A $.O&do97OO:BLE[asLM?5/pK'ra\w'eo}2On_簃qs_z }.b#)_km\V%H:q[0c뾌K򻳧ȹ~+򩱟 ̷/q̵yLۻf6`?( nH? RLQ3K6kpzB 7s[`4ft;ڸ9 @Ol@ЈYٛџvp nw@p q6{m^>B{7עp\,Wu{aW\.k _je@ɜ_})||/(\'GVJm×OY/gƲgJu6m1?|dw.&7ۧ89\˹l/( .'_ۢS:M}# ya'Ӟd앪O'&nK^;U2BmD s;N[!m9?眸=7f-S۷^8 "Rˤh=K bіC;'(h,q&9CJZgA{\vm[,?ڃ;(I_lK8O' ت-ַ_+>:[a|5cbsb^E.="yZU)}:,n.m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qbvX2%nb J >P,T7>v\bzomJߋkIi$b-lMfvjU8@[>/hm}%w.3_Lu3k#|m3я^||![޺>ԾCbMv7z m߾ ϢIv(B3Dl_{~[96Avq~ٷρ|= b`X a*w"nsk"LOH!nÀ=o-%qsm uTRY_9ꆸ-[)}kUtpg3(rOJx>^jRWP$>+w!zs&o֦u- ݶJgT\g# Lc܌!JuUAcU Mx)bvKxԱc:Fm^-{/noh|Xon+ʞ[6"ڔJ3p[`~Խ$b;˳oOYAT{2} aJ&Ѽ +`)Ȏ bn5S}E~z6}on_Mv٧_gGqoMG}2TA8UJa$D1(CDP??B4k AXoޖ%,_€Mu,FeBrF@T' 9 !U"2dN>Py˯[Z{/6>7:&8[{aQWM~c2r,-hY.r2!nQQRe.8za+DS>+`'+vlm42n" 6;+n;vPέ۲m `z"n{F`S ulw\:C $lZEdo97OO_ç/Lv7 -7~jd-^vOM">1Z mWE8^F'S&6]=nI9Ne\+v \GҜxg[12&r|ήml% 'JU]h>(Z-o^C݊3 > q6&P~T3m ; nw@p q6v}zjО;r(x{,\]n^ח/^/ZYrrR+{X`j>ggv^3ĝק]nθdόDWo<~>!=ws?CS4F9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQy㟜c?EG'lůZ~Ϩ7ˋO/;_/[#S*瀲cRTb{Zl8ၿU}"j¶,oOyOviCqwz|;X{Ls]󞐉߃cp͞q1U's(lcBvb! IL¡$hI £!'^|osfv^!__=Y9Acic垿z|3n3ג > ))?:7qPhog\"t#N~\\cm2f9{8}Nܕ!dz>%X$K{4滴~=cWUgsݬխq5׻~qcMnX|l^s~ξEZ>{&l&{*hO9۳?KoO!GQpEɺoi=i o-V/?x-f# zXO~v[_b)WHILw;%Wt@voܗfo7x zcwof2B>'֖8xXN~]屼 uC~q !n~ qŏM~@Z99uy !+epq=vZ}|p? ~umO8 RYn#ߴ֞u=++;,s6ҿ65uxXcUĦ5.?u/s*ocn[昍1!ϊ1׮kҝmN㲠\dيb[ai3!礁W߶W?"b,3cڞ?ͷK}1[6͸,l`0] !X.#tK0_?W4a%Bܖ}Ήss]`b>7ޘ[+ڳŇux3`ou!Ӈ"ű:}?X_+g:9-qo=.n6-o~g폆o+|>Jʻr8`M+)lԷᗧa{ YUd;e|i1[ oS+v2τ/8Mx8=%ឳة:|-<}9?xr1硙 }9ί,9O~]k1r,!cT*:VLָ-}:1kεoۗi3;:Z"46c?dg5~5ZwO9e=N\e簈aKWNϽON{\f癳<~|m73wzN3{2/{!٣|yV%TEn;%%b`O{.qcٙq=Vo$k8l>fܷ27WL)XnК/* }~sLcur,nɄ$:Pf\YvBFfM)F\u3{l"߄ymajkV#Û{suoBM\{V!3{n7΢n|F؇,f6g~WOK[&nak6MmGS[fǏ }Y?ا|2hV1?Q ۽P/|^A>שŠKWo#Ǜ;{ȏr*?jz>-Z8F%:q氆[o͉ ?.|KRg3ٶkɵM k:˳-꧈kpqQbB' 8gE.[/v°.ֶ}_}8_JVlĞھ7)cr옛1:X]zC֌77^yS-i"XVm3VE??7iP1tr! ?>lĬY.C/Q7lMߟzWrǠZWWˎ7{ڨb篶]2wo>X+yG;"rKQlnK\v^_66מY>Զrwþsgcuҕ9nŸ7m ĬSG˩:?Ng\2IouMi7dP-ZrW퇖j+p+i(5Y'?1hy]StʝM~mw\&s}:g#9\m9{o)|>Hqs||(ǯĤ> #bJ&Ѽ +`)Ȏ bn5S}E~z6}on_Mv٧_gGqoMG}2{bo<ڛ^Yo!7aFlh7Lgfcy[懠lIΛݨ~_l_qCp5J;\3V6݉Yۖ*Fo꟢95רEs|!q׬}մVE~q1Ige2*_X,3j }7磙vͿui} ƕJd-#ڔg͹^14Vڪ|c #l1y= mտv?|q Tޙ62VYlwy!;9ٚk͎m1+-rޏW<#.ܹ8G6|㢹eLq=˄DEI!>uq֮N|֮c۱ݶh˸\&ب["טּA:cl˶u.yVꝋ?7C^v&*(I|ӕ:X,\f6fOc )337Td./pXDi_#m ۽x9/|ӺeemUR͇یEo,Vò/e.W:l+orLÓ'b7;ѯ*&"7֘91swmۑ8e #}D;Ų"&r/ľeڐceUic8f+\:sbu.p?\shD{[' lW14,c* W\~ߺesW;J۹N{>qsh3շ5mR}wʝj\F&`PYiΣb~F9#؏ygP!y^8{Vj2NuwZ(r4Pĥe7d^cESVds$e<,#I1}[9m|M璈Uz^ڮcq`7$ޟJ|/-&Y}?6}*NvȦgCqC夏a\:vAͽͳp{y- 76k>ELT>"܁# oK!Pjz6{~q[b]ϔ9?>80 mw >cD{fiIs Ny`spLbX+7G]?w249.Wq6`; nw@p q6v}zjО;r(x{q\v\ vLr\?=l:9!Qcrcp=˥Vvp|LO7.;_ Q~~Ӟ}S=sE>@y;?9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\LoOq|ssd_IQy㟜csTqBůZ~Ϩ7ˋO/;_/[#s,q~qDm:1;~E]-LpL\Qҳ-cв]2gG]'xO|Dp9.;ڢ]k߽H?'$bl'yTw- 'p򦈿$n~rO~™x=k{&ώ>w?#5vG&n7KV=+o7}4|tvֹ;~ߞ-k! ^Cn}+lu?7{ҶfZ\_~ [F$Kq;fuϿS둾w*q+w!Q 6vL}]Et822B~-&:d?@:cy}{".N[VG-uZ!!n~ qŏM~@Z99uy gf+?߮[+ܖFv)>YXYY?4gJ[A-X{qd}:\kݞB)N1~m>ܾrOq+ĨNߌz>$>vo!X$m/_e5j>Xe D]5~~ٷol1ܲC,ϵ>?>}Fs̡9j+ev1Sf-աc˖oԿa1+QG(9}9+ƸZkV]2_o׎16g\y,}tRNt5`Umb[@.¬m\*{8|q[Ώ#9'nvYuo~A} ^b0p^! ^ՇH: x!Ӈ9ű:?D h۟-~qq۵oi;k4||ћ!2] ?LR.x6S؊Ưlc-k+Z:qmx\sڼfc-!Csqgcׂf ;~8?~|or~ʼ{h*ےl~:H \Ę0m[Wmu,rŐ|s!.ƬlF<[}zfQ 97"2F>W|9]f0鹌{?fy θ: զt~Z9Ho}n2־x"ϼdS!{jlaХ>xӇbJii$b-lMfvjU+O|2hV1JsK׏Ic,֔)c_X{ʍh^ܶm(}b\ζS1c}÷y: eo e9׵^ŧ#\2G7Twm-Yֈ_quƨ+ c.*(~3W~t#o6Ozj4z~WĘKu g˹LˏgY+&[%COSGUEވYQi؈cQ&qF6os\"0}E13fQWfA3پ"lrqmH1{48[6moz!D{V?I^aeڇu'W$9܆凁[ŵKGܦ?,yl\~:ㅯ=ԩր&;dm#9RQ٩R>ϲ:e|mf5iסcZp||#?>x>WU1uBq|JZ_GδEE0$Y ֋#7y60mE=}$Wb>:<goMeʘ;fCH@{V`'>*C|aY>֪c0D[N_ܦ}@1|M:r|ȑaMlǞu֏Hzk̕sVF27|y}@ס %m=FY'R0Ww?3~o5moSgǶ9|z$|yofOlί測:cTyǜgڷA?'?j?YiŌ,} L7vy߬l錋^%ڸ~c0M6bPШ7ʮ̈{sYWjc=r"Fh-޸X,|='<2sԷxej-01(k= q;B|,p ]Q' ŭ])=Xcm3іqMQ/mDYqaϵunmؖm\ ;qەCKu~p>?(~U s,ot | Tܽ\f6fOc )zp$r  /yz)Neo>:Þk>kk(3rcl:e[sOmoa1kpgc&Z}pB3 gXVgl.s,85^D9few7=b·Qk}1h4\|(# ,g9<:ܷ"x.]ޱ/uJO%> S.pgh|:gG¯Y\- 5!6k`֛ 4 nV nc n/"#85g)nP Pd `p@p q6{m8%__-7K|oW˅'nE9w=gÇR+1m%u\X~]_je}..7o/Բ z/Eik^o1'ndxW_:/_o?,?!}URS٧٫v]~D}ٝ)oξ=r.#<)2)ms&$#aBvb! IL¡$hI £!'^|osfv^!__=Y9Acic垿z|3n3ג > ))?:7qPh|b6E(jG7&n|d t s0 4vE'k_:sȳ-z9|:>Lq[-{C\~ 0Ǝ%g Zt[u8dݏM}+חCk<R܎v`'d-s/z$JbnCKOcM[?ux!}\8i nܵ_94Xo$bo;aMY]jv)V`GwKgJ:kPj#׶^F X9:Lܖ&v!enrfk*Tw;lCr~9q{n Z}ݧ׃;ą;|,G]VurZgA{\vm[,? p];[jI8BHFT7SurC}~yrgm5kͽ.J~1[ |یn[Ɵ^չa炷%4Xs8<ߚV|2_9ofgSïm;ĘVt9Ŕv.cEdWǘĶq~xNR9\%ܦg;Pke m=DGY,K15]¬b)EЈֶwfm]["o1713<@m-qbv88;Czpdm3f/XVU?/p!^rg [siնryLy.c׉u3hsxOn[qr3cq|Il~ou8Kqscb'j*>r΁:? 7+ǵ"bP$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@>C;Hj/@V>E9W3Svf+ʲb<^۰0p>~ʉMX7?Pf5ŗA menZРwrs=ccN:S ] XrӍ Ύm/ ;{6JΥ9[E::&99c9۶d3ĶzqtPID4oJX DT_@߬{mvq{0MWu]oq#l߼찢Ճcu)-tx"gC;T|KÕ:W&VΛݨ~_l_q d8h9Cshc΅7!V6Ო-cjuN[J~. ֿF[$|2ƿ6͇\鵘{+{^v,'W}sʔ uȸ-46n N4;ve;rl=pCVn+S}?._j먃c?c^+yZ 2!nQQRe.8za+DS>+`'+vlm42n" 6;+n;vPέ۲m `z"nn@Sšh!Fa\=Uo97O:Hutp+ufl|Lvx[=6xRr>̜\m?s*"G} N˵U>9amjzXb%ck\lĘv]Du\. )ڈ֗q=L|KKSfVushĬAe[;˶n2Gk]we;\W]9\Ǥ ?\/!b33idbn5ј N{2d?( nH? 8_zcDV}N8/1mx:;s'h69xEz` o^C8 n[yELDLq9aB<7 nD nc n;'ƈAl[sډxdP)~đ .oyUk6@6{m; nw@p )j]n_壼^nZ.>q\}-#.9а!8ލrNrBs\,W./YiKm7~a#eNru>^b~XC.Փ#l˧˳Ocه˳WN>V? ;Sߜ}{\F6yRdy'Uϖiw?~v,/.?//D_~}|n/ʎ ApLÑ1ECq|!𠶏)*^w\jeڸ{/`?*sl% PHb2 &AOm 1׷%E}/?]>"{3c •'$N /嫵qgYM)nMqa׹ۇBp߀q_߰皸k67qd'75xǰ}:s{{ v,=[B<EV%~oW_ۧm'XZ 8'Hvc>%mէ_5#}'1ݍTⶻv\7o/n۴aމL/Nu+W B$dq:axXN~]屼 u~XiES'?7bIw/#qNjUZ[G\~];Vx.|p5rw?mk^e,\i[E˽۲:{ 7._؎>j׆k} ~ ksc܏olW7E+s{{{6mych߽^ڜrE/˶7 pLܖ&v!enrfk*Tw;lCr~9q{n Z}ݧï;@~1pО%q\@^<'au!a?CYsQ'R_I8r6?[>&kSfwHP.>w,r U"FT)lE-k3cq\Vm˵'_usݛ#04p=Q 9.Ceԯ$/mRb߹Ni(s~~\=/%_Janω~1#Y.oۗi33Ʋ/z]楃1#mEhl|,sdM'H7kC%s)fQڡ YB6quM+zZv |B-5fښ%n[}_Zj&iB*T" UL9Nv\`zo; ۅ{1~m&nvA;e*qHs]4^FEoWNRà]u)U{9׽]cǬ1׽=ERQە>8[zL"~ Uu]7Ŕd\6jS~1c.c_}ؗ<}Ƕe-0$nbc|k[oó(fF,L""v:WߖA#ns )fo]zߦms _@0vPfBvVJoEgA\=+~xEb#aa|VqoK6*l*$Q|МDܴA[ x/S Pv8 <nW=\cvwb֘]6Zz,>8h67zU}#ɽ*m\s$3<\qղ|wj}aZ"ޅ`Αu[Zڶܢv۞۾+1Q Cs3P۷2eLs3;(3A;hkjqC&gA<3VD>֪cRp}4Maq~sS/7vɷ ԕHbY5f#Xsmҳ7;ح_ۥW5Wߺ_VMMDzWM3U.Sjue{m=|8@,ST|[k@ VǦljsVm͛R츰=-Q3W7yk۱]kG6Ud}j|vvtq|+7/㰃2*TDy&&΃R C!T}3Աp߼j26ҷyt|}\lWtlq1\C_x\{Ttn\~Q }zYq{[M lhgG ZLgo[U n; ޼ p_qaFS0%֋ oHg1P"lx䖟nQmx&i pH nc Bs!n?VH-on#s={s;E߾˛ۭ1źx@6,Ȣl}YZ#s [3H -wVI96`7b=#nKxo6,"&Զ~ zY[w.tÜ-?Vl͘]ߴn6, \7-a'8-nپς+q'l>o-V[ sP_I.H5X{9Q .D Ja9 F-#\# ^}9q'?,Rx_|aQWOv/_,>e.^:ml&b\X-/\Loҧr1']~G#2ڴH淟y8G/K |O/{{fyqQEx}|y's~QNJzoo/~ npopk{Gn!ǟp~"uQq;mmfXY[Rn2>V]˶Mq[ eBvb) ILS$ȺI «!¤է'^|osfv^!__=Y91ciSq?%M]_\pOGAcwy1ʡ&ؐp~>QԎ8qݯosMqHAV4/Ch=onD-qElXႯC\m՝eD܎B57Vv,QnBpGRlS#W&qm'XZeV$0ϋ䗬7w=QaGO~L ݚ^ ['\? xkU42B 6tC9^'_Wy,/Coc<.N;#:S'?7b88 韻kT-=G]:|}9rVnWEqbޯ>n=뾌w(CJwds·oG{aWγ$ QdHs^ab~]|[%ڷ(r/t YPcYَ?q_(hLܖ&v!S%˛M,kߪȬ34a[3m9?m7Ł kx7&1\DܾQ^X&Yk ul*!'D,[rC;'A:F,q$9Wc}\^%n|Mmצ=,9r._lKJy_ ~qXb}5O7o<l1Qf18874q; $n-Q^% ;+~Zve{/7#YֿPceQ/V6RL){YYvBFEZ:7%ߙ=wSn-MHic~4ǥ!7/mw"n)ў S1`#ERqeoNh Q,bJii$b-lMfvjU8hZ>/Xm}%TD`! {%([~ QՍZ*d}gX ź6ٶ|l)2V<&e|c.el?mg^' C{CMCm%wT9ABzf`/m/,bbe[hQ'd7*4[g}M0SL=ذn@wP.c(zm^>:]5Am$LQ\W{V 0\<lZPPlEXY[+Ӹ ﷊kMX7?Pf@_Gu,%Qݮn۲yM:k3!n;l6XdWIΞ}(sioNkw Y=x:"n;fY(IvVL^LTDC$>-\Smmj Rl -~o9Tp}Fܮ)_VLr!L~ˆy\ugCcZulZN_ܦ>W,YY_ Z+EnCy3][嗯[U>S@or=fe /}Ӷ5Ȩ!=Z9cJ6mB R ęI17Mt9`\;>R-q-[otV 4My;m&A$P)" W~|J)cy[4)IΛݨ~_l_qT(X(xc>ĸ\UQIGN| }5_b\cP_kYnYy-們?+^g>wǴ)vQ% ZbYN: b!۱ݶhS%nKB蕶 uq|ؖeW5'>_XD'0!OAL$ױrD)paWQ:X,q{8E 8בb{MpۼSSSEg16wǴ)AIvS1 n; m; nw@p qLR\oo߮ ۛvmR/ĸxK18x q hYL\yĹK࿗'ԙFM)%n;S_^orRvd}s ´=/Nqs /?/zX괸ݢ/n,xBe@&~ۉgkS={<,Ow~ܦB,_ޟ:釋{눱8ˌ:j;6Bň⽋#){}3c˸~27ط*^C| k|$x+i^c\pl(Ż\[*B!8e?/O y/Y(zΈvΗ bFADZ>fZ+=GUE8>xM&* Ǿ}-X`meXI:ӴN}^u1 s)eh[읏ֺඵzBs8Ovv_sBb!oEwI[~99 s7C|-S{M.hbjݮmG` !xh 3:Sۼ;ָ̊v·>ވ^?=c(  d\gu* nިh^vB=O3J_(j82Ι*~\plY)ΑW Xg\p]]&D?GJpr.WHtRPEL֮ v zKX1󸶵 M6*|ү݉棲]mq3J4Y*?nn'1^yX#CboQv~,DB?8YT9CrNuThenDd=|"/6C{lY1/c$cP٢g\Bs a[yseBSĢDv\q!lq; p{ja+a iQ \ucqJsEsO3>=+zqٶxoAZ""j C -P̷+,n7cøVCZmr>~ľ!@6xL^Q}I;qsvnZcQ ,`@ؾCr> nm; nw@p q&rrB)]^^/k؀9z'eBB_QW_*pq??(uQ8!r^Xp npLYaDޝ??^^>pW #qse ^N7n'/ޤVڊvď/%{ryYSY KtZ.HdVƖb27&o[P}@Khvܯ$9K}{\ݼ cɓߔ/ᵣ3nG.@e26b>r^Qm9O]).s!ы;yDuM91 8!npm[|b A QT*!M7M?MRމM< (Z.d/27a%ET7>S}1X9qۮ,:2c !@pV퍊%Mm't˛ۻvK "raq[uCZk3.n4nۮo[uy 1&\B\$RT ! 1kq%]uܖ?UrxsSB-Kaw@f1¯Y!(K n3e V _ A9Q,iJ5=_wAIH++C]veNDl3Nb6Gv[f CDvY``błƷUܖ>»*7}kGϳF{DϗZ}8-;-k10CHsaWQmMX"r5/Ƹ\/WM}ƶ%_( av8eK"Vx^ʶ=6o랻~ np~Lyq6{m; nw@p qSrs{ܾGyyv\hϝ}fZoo^wag />1/r\R8`T{rksOX|+âr,Ye+_>eX}>\*mu~MĸZѷ_ݹHߤO\bN4>Gd.i"VG0-/~%^F^Y^\z^_<_މ|ܲ_E)Vb . s}MnNĄR&XI dd]$4@,{a!3WWOgN@Ƙ>!wZT~9yOIoSQF9ڡ"CŋQԎ8qݯosMqHAV< nC>{z{.n6?K"DGEԥφQ0HA*xD+חCk<̊syqcp><*Uh|j>vng^/W&K OЭps:tKyo@r*eS_ٓpN+4ϗ=!nyF_, \N\?| bd[[{\.uw?k>{_>oV-؇/wvyq3f̵̙-3L뒕c>4֋kg|V:-߄26 IjTno;7 [5qt;lk"qf-gP8Sx7&1'D݄ê;`QF(jϒ`aWu!sA.ű:}<{ GĜX_+g:9-qo=.n6-o~g#!k 1ֲkWZWոey?yf%uR'O/jR[|p[dG5벯^s/vw|6=i4LrviGb,|3ܖV?ce_a/mkZ,nK2$62K g)fQڡ YBUk1lɫc|7ꖸ"߄yyV>7Os\rO!xu~پyUhQϔv0.S;~fzoFm/p}/ƯM"6mڎ h^%GE 1,Bk$[Kڮl){$zZ{Jņ̂+;^ELmѷ+}%N53Hmsm3ߧZ4iG?;k{η:bιv{cgRTL"!^l^-DvεFJ?m^?εp`8<m/&*"( ])f6P)x}_}_*>#nϔ/+&pZyf $O?ī!}T ø<<[\`w5٦>.:boCe[x xLۦMHX }83)憛"nY=[ێ`\;>R@L)ns[MHݎ[%\6MnI;DlxP:]t J%JꛡmP=d@ܦ:owv}}Q\o:(xqc5c+}Ƞ>SG^O'p6EΞ"ױ~fs  oo[$ىY=y$mo-fOqX=Ik>Mn\殎i2RY^:?kwwME7qLۻuLw3DTI!uq֮N| p{@[vlm4*Ɣ)ccҶm]=81ehU *;.C?Me/C;HCpp"do97O:DQO+^m of3AZn6Z>&n;- ʅ5w qӶ5h/1b^-S^J>q<+j~D99&}aIHs)(˷mL˕3o^5Nb\G|[-{yVrk7Xm?z}|b^y_|sm16 _I6)O5v㘶w똶2f$na/@A@z_.7!lx@~L@| ۏ ; nw@p q6{mNWrR/rr=wkQ7V/˵ݻZg3q/]n^urrR+{@!KΉrqTm\oomulO5fO3wCݟlΗ\?{@uOH`g>r.N叾W}r}x:av8#'}/sOX|+âr,Ye+_>eX}>\*mu~MĸZѷ_ݹHߤO\bN4>Gd.i"VG7>[^K޽ʿٽ~("xme(;֡=Fռ#JkkǁZȍ_r̵|8|Hr֣<[aXiy${oćGٻ<\ qaT>{~nyyY{)5K6\w6.~$ oJ˄R&XI dd]$4@,{a!3WWOgN@Ƙ>!wZT~9yOIoSQF9ڦ\q_߰皸k671>,E~`s{0|X?XNr`8~vTg9Xiy备xs#{;^ܾ3q.E?{}qA?|/\iBl׸|gY!:*.x6(7٦G %Vm'XZeV$0ϋ䗬7w=QaG;S \ʿqw.6I،:YF7ϸ:cy}oIbur\ _?_~JvS ?=0r8x>7gϷo.72r[GZ1]?lq/n1.a[/#|+c6v1EjiǬ3+C15|y֊Yo\rL\cV9>C*0۲aT?囜VԶؤo=I\1܋239磬+گT"Wd.YDv.YzzX1rXPsBaѕ͘|- /<ڔ~sqG\ϚUýW,9~.lq֥NѿEy+ں܉gAGytZɹˌuhyVWDg}(ELZkӷ+֋g={J'_\7/v5?g/\>Z4V"Gkg!rtj}eFrEyc"&ʺ?-MzDWCrK$7۹Y]ߪ34a[3m9?m7!ǟΧ6ų05p>!&lnC 8nrU7"7>uؔ*6'F_'b>ma eyCsu\oPoTl'%n|Mmצȹ<8`Qn7j}/CY%yzU2f|}+?L߅|X1/c@Y_o_\=k<~{52AsMqɘE~21ʘyGuש6moz̥{B}y}+yfy۳EoNK|!*HB|7D3RF͇苌aDk:~xNkÝ8aWsO%Wy&m P0sA; bU["_~>&%v*m~^c}Q\ 5<ɟy= 7g_z-`q;f22:>ke\g5N.hsmSy+zh~s[YܖeOIl eR͢Cr5X/b|W;6n -qE )2| o64XB%e}ma1k1{ڢgfՍϨtX^lH.Jߋ+姥-nM5q#ک-WG">5Wȋ*ox)碝`X6oZG֨[fӏ%֫UlvW\9xzys(i$1o_3xC=\nsXy 9U:>L=r|Leh>: h/kSHҎC-nY_f҇c6?U.(B]Ko\}=U=G16:?ߪiY;f9gfNo^Wc߾x.m>+m/V9=|F_U}Usǵ>/F>Tvqkw-n.sdb>X9gؤq笽bFbTǞ7aM9vkhTA۫l~(n (j&)oswq*9!m -gV1smQbe| ڹ^q[sl̥ѶX#roxXIP"zۋm E(G,L":n*wlrqӱѼm]| Ǿ]QνN}{u;k\T^`Z$ mło.HCwS"6Koي+W$7s'o.qv!nnq^cJ s^*RY3fwAޜ12ɫz/nv[e_^GխfCkUq5w-ߵ"_s3f/ɐ\<t;lbR1-n{1QAITB pM7SV/J- ݶRqq~|Y1˅3K&}BX {e M='ܔ3kug:}U[GfևްZN_ܦ}M)şu|Γm }Yqkx< =^ىY=sy${Wg\-_j|-F2HXReʳ*\?.hb>Kٙ2_2U[>?u/T'vUhTɑ(؍XOd_ț8_J|**&zi[o;}䑉FyˎK| ̳$n!Beޗ/(meSW>Qٵ?/O/LvMYlvyNFbf@1mCh1| r{?FVNYvQ% ZbYN: bl[m۱ݶhS%nKBD uq|ؖeW5'N7&Bá͎6jq2{ul.j1M,~Ơc| ؘkufl|Lvx[=:l~7n)sonTwMH+r8?EjێT9}_EUgǬiqЇTV\A}S7/w6՘Q_gOq9ܣg]"aOsa l|5 mc_އ|͸dd@?k}̭l`>eoې:cY=63(VV+o3\qTī{ej\UkdԶ㽽UbYr x&=g9JeUqkͲxwvFos1e_b}sJ_]Iʱg$o ٮbR"-Q~*߻smMg\.|Y='zu̬\y66WhUyg$CWJbC[ pϓAv|+bc'岑_f1Neo%Mu pA4C`8o'l [@,ޓ4P ,4fMq;Xswmpě_.80;Cq`M^Ц|B,a=2~vg@%ZYqྃO-?t| QH2N~we8W nw@p q6{m^>B{7ע7#/cr^͋귻r}5Erx̉۟>Y.^a>՗WO~e߿z|dԶ;|byi,py]ic6jG~Avb"}>sE9lb缈͈qTqۋ\)-/~%^F^Y^\z^_<_މ|ܲ_(Zs|#n;qZ6M n}AΉ qۉL$a^TLv&'1Hm'bQߋOOd(혱BzW?UqZƏ3|%˃@.=DE=Xau/✠/mWOM+%1^T1v- _U{:~CXT!_7oeVI41X[~s< N,'X^:Eh;u[BlS'?7b+]ʿH̕\9(4?^=/";oT}mJOb~c +sIFD=<Y[Yr+<<&nK^;xPK $kߪ\s4a-kpvm;Qܦ8/7\`_}{N+nz)/NPnjϒXEe.ioYhD \8_ǵ–xD_IЌr)z7A]7OGrAOe2C_Goz(f(ߊ.h^ڒ9;3soY7l_L˘s5eӛ0o=ŘlI4dcIx eKϥEiǸ m/zQ=]?Sc-nMțL+Ѳ]֙c%n_f4mqm .Lx%Upbm3.x "ꘂg|FXc7'$ ۅ(FBS{1~m&nvA;e*qHs}@80j:zs]ԓ9VUV (BsFZ'm:U~\2D^'EO hu[BϊKeq<p*)Dz50u )CzIovfLbL26-m Y6I91͹ 8A&_ 섫$T3^v˥g0y#uT}3̱XyJ1#<m煽mxl&HͦYmcӍ'/w :ŗ.3s͕J侯}|49:`ߢ3 ܩA9ʽNB4tkɎaI=&"ٮ5R<·_KIVҞRG9C9=$!3q7m6 nw@p q6{m?X~^~Jg0fZ+X\]nvoojjuRElR/rr34R+;yX#[mw}vG!'GLH,Lԏ/#x??kCF;x.rGaC˓(q 1s Eߚ"vs1ʯ^{|w!ʹya+f&n+CX/îG6)}dxW_/_<<{~C>l[xdH0(|+ډ1%|flU* Ǥg;)rEy=yV=>1wmsegS)9 q{ӓ/ȡ1Ix=Ww~F:*-n#)?BKM‰"Hf_ q;B"| K/7*"Mwܤ{-^r<íCo3,>D7 m[Qظdqh]^?q18oVVŔctPi ""+e_Sr}{|LcsJJb[/qvjS?Z8> :A $"$H^||ڻYª)x []:}KH,=8S R\KoP\vWM:ŸF.chڮ!kXڎs}ٖڸ|S-'??odcVfbTX̵nɳCv!̃[a{_j]gbϑu~T=vBp~!Ϭ==9+'c0*VDGѾ7./klOmW9P|Oe22|8b\61 [3%f[|,\9YܑsJ/WX9iG/u7#RX)џ9yxqv]]Wl[-l|otFm&?B?g @}%c`|ظ8R#e|јd`m;qr{ۻ꼿̮M}>8M$wa{/e{L,^Q[o)H}8,iQ딯Oוk7uCLs-4`zՉ=vZ1sk^?u̪\\=؎ n7ܿad.7(g:yH|Z3j/77@KаѤw8# \  QRڢ?}"}f@oݪ"$?9ڳN V6YEYAxh1!/9!ɏ_a+'c$cƭ̭u75Gx~ƪ k1Prs 4ŴE}a+|C-7 zSh-M/EEC6 Zn Hɐ7읽_@$MP+'`PJc~EqRi7.Bw#Mgů2.ߙyڇ)^OggzU9Xp>F{ {G3/2|)3"n*@%pbnz9^q01%as8:bu}Ki,3}+f.ϗJs@h;m߳y1Kzgw9 ֣ucO HzzWmhES\w'Q!2+rql^GaV`:3|1YBCm;$ ݴOolOJﶭo-{|gV}!>3Nwky+z5[!]63rrM;5bmg{srʾ疻}yVر8<+n:1\q:اS4lo~LO/9+j _3|!M湃{aκN@qٲ3{E#C}MƇ"LS8(ZڡL 6nGe!+(6يhـ~`ѿS>< y(r&ZD&؃\׾=-j{q} +.0=!Hv{=cZ8k"llneuoS&zlD {?aw^% ^=s}(i[)|PCc9Zl#/bc$T"g,؄J+:}\TѶg3HDy(n(4b˲|ƨo@g s->hm;2Ŭ?C:狉}Ι=)bH 6'=xdc-FNlbӌJ١ اvT?ڽ>gy>_ÍSIG2k@WŃ9(ϰzᵜѽy}֊z)~Bƿb1GwVٷY9.Ya?|y2 &C~mtˁ9*6ρ?'?cJ[NUH<_6!S; OEAWF/}KHuaz{mdѮ}-߯Bw͗x\ƿJ4 Uث"F|L{?C֏9>`,Q2`;M(#v;rD/5}|nW≭3щQA:\^ -kgcw =wK{.+y)6Ϻ/]߼G-މ{uP{=O-+"n&l;~z|Lo{몽-q׸[ƕyX<#㾏s#xYb|6ZW['zg&ñasgrŬg!萱2I@!Nx~" nY~Ƣhr'2Ei,{c+¤\fP, >#ڹ}| 3rl:a^3udն2:XQWYUK7M /ᾴ%ck}V52 @]OFUdk!*!.gc\ؾ|q{>`b#?*k_a셵x1|`2(qvp[{rF etu~]žol}q{96Eq{;Xɥ/;S;ʍf=і`ئNDԮKvkoo4?zvNv,p'>+8\X}:08agl+{+\ZN{ߤ%XSMžg3طxžomS1DB>(ϏuY&ﳒ,j狉.>8f.2ƱScP Ŝ8wus$ٲcژ+ibz<#l3bk=_pO>C؄}a}՗>DE잃o7 e,-C|PfvL.g\Lp恫mHl #A xP`'r{n?:i!/b1}+/.8oEc>'m]ͮLf \ϼ@ ~>c(wȃ<#^֧߿ ph0ڶ(XrhUیK|bWN1HhEFaaƲ Ǝ5H׶bLP6㱭zvv^ZϞ2Xkh^ʝ6r;@={^}Dc㶎uz~8Dgag=_&v3פp2/wy^;e;B=g|g j=kߎBPbNo]~LqG$kOJG<n,cFqxːXv{-2\fZ~,|ļZ;}S<տ`x]w7ڹjdm/cL|wd|<-mg{SkunvsL\qcu`4a.3QoXGvs8ML,E3sm3kܦ![Zw3]}B_ʇCI9+|W.N[#Dr!/I\9r_dH$߷jw(3wqɅkr) y$Ir8_Lq$n-_y)l_|QMFIrX~o{2䢤}_0!9$In玕'^p7Q/I\8DeIrAq~G1)s]f$RMnSԹk0ۛnDeIxϹw*_޾z%IrO>~)9K"LjWo;lVsqvQ/ߙez˂yVp. P$]wGd?=?3{X˞~h˟d{_>}ϑʿl}tPvfN;1͙?֧Ͷ}\ ><=U>"ϙ*1ٔٸCq~ |pǭwT_/y.S/"G6_?α7~@7ܙ^g_e 6~ݩ=c+8٪Pq$eU܎KDO/@$uio'CQ͠-Elض7Hp6/UopA6,afvpbK|~'\Yu_)Ity$P{|ZJIb2'rueΧ8v/d~D(U/9ñ 8wVҞhZ<#-N| H"1i/Ȕ^qi)D5 ͇=\7ծi2ۂb u;j~鏭m{!D8U;'ؾZ ֳ̙}aZk練NƿKKt|%fpܻ)/kR܆ϥc :wSl.^r/WRܾܾ;1ygsXv$3۽ *ֳvhf{H®ޗږS_Ia "âJ(돈1"]kp^̃X ~,cB8=XMv(QW%eNhcg~]|%jlH2EmגH5O@o<6Rϑ_a7&=IA%L|&@`_zߩ_2`"PϪw'W?QGٮe?ōVn6%עEH/0{lc}[PL| bPC>ۧ>rȿҧ1%m2k߇ΏqkY[? 1p=Ӈ>9 cB.>+Uhϔߣ䜟}dڥl .Ve?ΝXܟ >e0Gcy)v:m炈Sk]hcCg;@>gbY1g Uz6+QeV[gtnm;cL߽s5IwKALfXغ5 Ee_Ϥ]3*X$n %h@͇B}{ _%6:UKU _&r_p2ᓜr Dz%W`hs4[($5T{hgb _JXvj7DsP>>Ӓ;/^~Tq2?#'HtT,k"mt2Ś`~U}%lžzI̫pb!`[(W"}!^H)_96^1X= ]h3ZvsEF938>b,q}Ǟ;}Wc {({=8n^092uSqXXJc x]1si-uxT:vwBi͋_"{=;Ye8{dGr3LܼnUF{/bҾ,9a;W\q/kD-9#}^su츼o}he0P8og-*yal뽎ovs/>vz8q#P#W'U2ɳ _|O3hOj}5r-862ю:ط)1OFԤgOpdF-Aƿ&Q>&lr,_$(Lf;Zrh[ABc(&:"6FB%ŠMBGNCm{:D"N#,gzF@;s+nٶ#cZ۸WxW㬊tm_^{ yG8bix.^I&8͸Q?7׶JU:ŌQצkU"ICZ?Zk[G4t=99c,8l?O c4'%IDAT 9 #bk`igc$w~}w XbQHf!Z~M?1dg(;IƟ|嶵uvC 219'x8hNl2MbdS $\G& \p@vD4($ܟ,J<˼]ֹI~Lcx2e,<{5}֧%2~n P {*6˫"*6ρ?'?_inQ;\A R މ@LP- E26}A[Bbݳ%%nK&cpX֘gv%Nk~M~?kRΫ9xUIg;#Uث"F|L{?C֏9>(B{C흂ʉ&yTI9g"ҩsxbEt׭7s!DπON Ƈ`_Csj5;yL1"hͩ#9~Gl5qZŇ0Q5 ]ovD~1ȹDYY볁xCu`M:B{;cINv Dz)4'+{n68qӶ)㹘|9&Ir8rO|mh׾nHA~.}IZ57'$fQ۔o~8I:\Pb$=AI ?Lj0ugvظќw`^Bqofn ^6s\2;7~_A}G'ƜEPV}%KBS8WKc!vdW!nE`1d߽XY}{Ea=m "I1g[wO!bO+╌AA2o.xb)GƻY1ο.|_Qô<$pvTW,aEu'Y??eb.\q^/?SEYmrHy~V1W#aLNq^h\q2VK~As}:=n;+=zH"CL/F*0.aƳX9!3穼?qEl lRqMgc$w/V1Egk#^PFږuLj>)6`g\7HܞF/)%הw*:[{^ a$+2D &/'g8ї2%U#>y vmΒhد {%J6ɂr{:ܗ춽OcXc:[֐-á|ƞȈBT< lz@s ^\ؾ|q{>`b#?*k_a셵x1|wyQbi(3|˚xAֽu.ۯq7O.辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt C˜3m=}?m|ybo|dO7'5s_ߵ veG+&.cOp0>cz*ox>sFmKFB|4[Ov/G%{o91 KTqmZ= Uⲍw|Mqwnu<4oSPI5+n%WmRT}׺%x}Wkme8I"a>z jP c*J ZO(r=6꾑:7mW(TN2|($m>'m]ͮOfe=|~>c8iye (y(k}j8c}~0$GCqжE/(3j(prm*S}qwzbZOi"f -8(!,lXر 6Ul fw<{]/ΎbKu/Ck|͋`b+C:/B("jOm~u!bԿG> t,g =2m &͇y-9s>]Pclwϰ^Xv' i~LcT=+mn, 8N(nC3i!r399yCm}[g[y όŽ%ːǰчEgz,̬IH=2sN@(Y+SAKeMe'shqG}Bٸ:$I$Lx22"Z$ƭܗy= jw(3wqI/ ]@^C?I$I4)nmN,7In[y& #$e=Oq3w;Z|BI$N/I\${;ĤMwKeomG-Rm?+}D~Av$Irv$I$I$I$I$I\7)n'I$I$I$I$I$$I$I$I$I$I[GI$I$I$I$I\:m,IN!$I$I$I$I$ͶWۿ /_ziw("vsl~h6w\1.Ѻ<RN$I$I$I$IK6 {_CM.ۖSDsd ݟ~ [ba}~@ 65%qX}Uxh{P'Oo_lo~~~%I?o?ީ<~v{qX$5gWěAq*/l?8*l/K,5 )͗#}?':֓_Eev{lݏty=m9BqXy9Șqq&.4I4d &VPn"ڙEߑP=pܑۗ_f Q33h^n+'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ xs0~OKQ=M>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[LKNͳ=#uE퇲_.'H$;0%'%z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|E3Uc)ݳJ'^9EEN[器^\!^KEE͗&,sMr(o@7Y{AqX~ ~};]:Z hϭrS틟IWA0Qnẻd\?z}~ñE)K(٧87cˎJcpv=X[/|@"W!`$3Z4NJ AB"vMIjW4}MmAxUyֺEEP?}6?"]lp-CY̾m5wUXo_e_ƭ!Fs}ùA+X{ns|l.>+r%[ӕpM7?Ť}yD_KyUfQl'5]M#9KK~A?qn4Ϟl]M1YΙ}t]YqJ~}u=y$qa׃~Km)KW"NXtIC{9`)xm~` ^$l.PnGؙ_g# Lؐeyz~㱑ڇ}^ȏζƠ_^4 1xN j/ae61%m2k߇ΏqkY[? 1p=Ӈ>9_AJۂGgJQr>R6MKb/Me/Lw>+ܗrRԑ EocM>DLDD登֣oطueCG܎o33/^/ l.X π }LcV3j /(v\s&X9_vze}|-˨#X@^$m~ږ#f~4?/~O6_n[Y3=ur 6,k[bvFY\ }X♴~~Vm:`E #dsRd% \`MUy_3N0}QBM_pBrT&_Iy>oUr t{%/Jpӌe׾&qӱO4?z)}5콱_&=6QmNr^щ&MkUUM_<Vmm"ߎ})*M`,_8=aыcc{Cq ȫs۱|qAzq=i>g`_d3#R %e^' a=,^ڽW7LL El~:b]m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#51H_>}1uwl.:yb6q䌐5{e)ױj۩~a\|(rX뽟leꇹ{:.{<]filzˮ_h?B'VĺxVdήٙ Ѽfqƕv8lcdODQ|M<"l3W5/[^G # {) dQq6qE:R_KXFto^WQ 秈Xm:vmD6VM/J2sR,чACh JKPu HdַL&j_ܾ&2IqҗWIP`^ҡRf3yG\'M~Ugl^K)RoAIllned&lC{5?0;$5$J$J%3 VkLf;t¥mE;>( \(&:"6FB%ŠMBGNCm{:D"N#,gzF@;s+nٶ#cZgfR/ tm_^6цÌYWg>9nDu`O2?9D\k樜8+WpCD.-*'|nmtraMPל >E=:~('ֶkdGwzvKkuϖs/ysȞx/ǀm~ǿq#p6P%lƹO2MuTEf# ^ `D`Y 0 ,D/g5l e'3Oܶwh<8gobCYL(`RN?DȄN(ȮwIV$E%D ܟ,JbDDH*L/8gDdždjXyzk.OK1z^P{rIcdnGΙefo//J]υrF|B۶;\'A}%&Ͳj.Hv C-P}5Q>A?K|23`wrh<_`i.uda;M-gvp.\|q.q֋DK^׊m&4MĎSXzrTܮ<@}yVWO܎ƶE&n샽e 6qڵ/RЦK_RV2} I6?>yԾ"6gn[!Nb!2O QD{!%rC%#*!{8@L'NtqRgr0QR'J%"ßɾGc]>aL-a½N{O5lDI,{; .kPщ1_EPV}%KBS8WKc!vdW!nE`1d߽XY}{Ea=m(珉k=)b=Q㇋W2y(YTW}ovfq88Guv󼒈Ck#]S_ղW|Ձg`x;depP XsyzL1"g-!=YOC2;~i*sa0+WZ-aǟYڧs?p]bZs>F{thӈSvM"*yXT-}Jw _n-'}~ / h,תu^gk"#q}in.p^q*frm (PrQ:EFq쬂6+KJ85ݤ֞Ʋ<6&Q.$6F谍^Wv`>ϗp>yĕm- xA̻g6Vol$;mΒhد {%L+?A_p_ҳ>%ck`l[`K볾ȈBT< lz@s ^\ؾ|q{>`b#?*k_a셵x1|Dߝ%bZ*@(}E^׸U'st_ܞcuQܞů~N73@rnr#b=<rދCDa{VP+6Q_=bh?Tc]B{}}A<OK^Lܶ<6T hg)6m~ Gzsy[| >g;WJ߯Z 7,uK3q/DLCU4tqB-'LrSzm(,}dDԹitRB:mʧv= 7.3zu7gᳲg '9Ls'7WEYSÉf,@&m=q@ڦZ[/ "O'EHQd6Yl, `X\]}m*e;ۊ=ڮig?ۺ!v>YDZYEl y 9@={^ˮnvc:1#F:3|τ_ɼf{U.1vgXG\s};Z A9u!3"(\{Wڶ}qW, q!hڌmb7$r?s>399*KOK& mDžk1|DS&RV'[툑+q}-c8^MJh|i{Ww`$c`_K+%Irm}_ ̍V_o$bQgQq]/ C%,O%+XzD.BN ~f&zqۊ!nH|]j,W6:TNX`n;)nmN,ïAVÿOa j0RN[Ɗ{stIr},""Y qzmK c-<͖&m5u~B]$q{; 7$ʼn)^$7 1)s]f$R3x_~nS<>ꟸ8򧨒QR$fA W/nK~G܎{}푀|?23{8,n'I$I$I$I$Iҧ'[A,:Nk(p>??&@{\^ץo4O\B~"ߎ6w$I$I$I$IDP+N"ֲױ80nlEW1+8P _7#q[ ڿ,x=.|HI$I$I$I$Ir&XlO)5r[ |Z['Wzj LGPXf<xM&;7 FI$I$I$I$Ir6?Q4iN6#rNQv$I$I$I$I$Ir8,n*|=s7`/7?^?ʒ$sTz}8,Kyz{3+ˠn|oz{dpoQ%qsgKΑS>yޟxh}Ic{/Ï?=_E~Guֶ?my ,x^v\4k87$$n侹=tA⸮K < ޓuJ }5W.%` sD./2pS?眀/=|lc^%l>(g?P'||RlsD>+>Qʯv٬_3aZDg]@`Q엃II/, LI/z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|E3Uc) ⼉љ ;/kQq;|Q乜CL \T|i8zD˜sqWer2mcQqF݌>grC;SNqVq;/b?پXץ}EۏBXV6"b^"]< D6Ta?ڈ۳I\/i{ dLpe9})d&奾& ʶζ"i*%jZHnם_gN;[9ؽd}BiǢDT &w?ZI{jy|6_8D"ĤI S{ĥqRRQD(Hk7npNVkdl ū'̳--?{O삾W`k)Z2gm{hٿ,=NƝ3nڭ!Fs}ùA3F1gOi\Iq3~ƘsNqu9CLq;Iș}t]YqJ~}u=y$qa׃~Km)Kxal%aGĉ\$!) A,_X6h? 1qzPJnw;ʜ>$Jؐ@eڮ%jxla#{7򣳭1(nz)Wo/M2{ KpM`*ϙmSdbDUN `]m>60J*7ۯE^06"a,0Cu1:lYS/1Ρ8Lm;}O}cOOyVc^bas=>vN龬_o_(1ڳטP1ֱHǍQ!^Xhb3Cr|x֋Y^`%l{˿}&^"pvY~6/ih_>t~v^h1>Լl 5սlsrQ^r5LjL=JG])QܹgYƎ~9y)C]]w"Ķ6{Ƙ6l\zD} }|]3~&-*1|s@_٩_os18h^luF禍;3;W$ywqdx1^,YTE>,n}}LE?c?I6Pҋa|,$ʁIWr3]⃚M&?\<(}5콱ƴ˾W&Uψ|>0 ՆKŚHa b{&X_Uc [^Rf6&XtJ;UH_6|἗E/R 엿Cc m' 2h!nǂym7k֧;Șų=wv3X)^?+&"6cr{Tk1V}y/6X>g:^W\%.qhK%/vھgb^sVG9NǞ\k 07|ا/> b1x;W.l?QrF2qyBTK\|(rXS3yal8Lqt!MYO~|۱3l@|1ZhJFnײ97Y}deƲ sLq6qE:R_KXFto^WQ 秈Xm:vmDv= |ˁNn{&QBLԎMKSE"R5ɊME]'H"\BRmD}F"Ndg8U3)RP~X&q\6e8 =وyȽ%5$JDŽM%DeI `GKm+9H0y Зs`DG_HDXXѱ @Wu(m6?YgܝPQiĶeQQ߀XaZ|nZ5vdIQw[,b+272C5''!O70~Vՙ!{'|*}<?Qy{keAYŁ~^/s"ria(V9)ssm.Xe11{U"^ ivG^ ggy2f,>h^x)<_[\l4H\L>${_?ؾb0B2 mY!?CI6 /ZP!I ?dR ^=aUI΂_/4f.@ P~ۄL)bD &?"XFX_펾 -!1͇ Y%1_8bVkEx&x 50g:NJT|>BWE|i1Ս$~1s|i/Q2`;M(#v;rD/5}SG!.w֙ϋ [o嵳CՉU;$7AZ X%c͑yL1"( 9ud0gr>sai$[wm 羒G{:l ޟ}Xγ>Yu >3o8O,>h^\=s3IYN31Im~}&nCevC sKJT9I&1'Wئmx ;IL c1< y/+؄$mHȄ$#P%+6Aq"Ň?Lj0]LΣcNL Ix"(~I%!RE gۥgj;p"0pz{Ǿ=)*b I3DbE?\1(CFZ,:{x70c>ץ/ >sD4ZE/⳨<{$+=LXŢ+I3ň^ն( ?gVs0]|c,< D>B/F*0.aƳX9!3穼?qEl lRqMgu{޶{y^X~o/:_w/ ]X\3.s^q*frm (PrQ:EFq쬂6+۳`',qkʻI=eɆ |k^tk| 3^K\֒*{͌בATʼ|fc`|;Gm[_kggWq%dA~=x Kzvާd,@[{1I}kȖ0 @IcOFUdk!*Ih= D9oc.l_\|`0Ecv}㯰~sbZK{ .1ؿrt龬Do^뢿}#ؘsl,~gwܟK=_vv -wy5̺{-? Xu;Mp=]hh~ XNY7,}Vc87ɗן)n[NMtsR3W]; blW}t;brB{1/ EvlzvtyE3jS,=7b|ϐzTbuEllY]קkxYO8mkss3a}S]BX:Hnm?/Ͻ{ bv?)j_2 @?Ab@2ueߵ.} |Z}Y~q;NeAXŇx#NV'rĘV1Jv\=M%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#Ge;M:63;W A_38BǶbpOeٱCO{i={;e`Ϣyp88VΧY<$Bq{'nvc:1#F:3|τ_ɼf{U.1vgXG\s};Z 4d&1,{&ym{B(PnjmrZ9o @rg&'1P|b^0vW_{Ρy1s@6Fo}RD:Ǣ̚ߏ$~#X9>4dr(:rl^;?]T6}|/IrHq;In+mS>Eŝ_sOIq;{)z}:n)n_/'?q}s$I.gI?bR&ɻH2ϷK}6ţEm)6dE Hq;Txݱ}ܻ`ۧg@PNq$I$I$I$I$YEAK@o{{]*\9jVi#ݏBc8/V͓Oqb$I$I$I$I$gF( J$6')ϘHܮ~ߋFP{ρH!m fXvDܮuSܾ4RN$I$I$I$I3bEMF (n=/M蹱ܽO}?e 1_Kdﺬ9mv6|Rܾ)n'I$I$I$I$ə{,"-L-h֑`mV_ϋպ||Fs+pa;ۛHq;I$I$I$I$I.&Eʙ}HA2Z%F7@=b&^Gf:Fa$m|e6}4CI$I$I$I$Ir :ko'Lm΁s Î"r;+?r\mvv(K vm6>-S>/=Wu'QY$=^|sWoeI\#O^ooxEy^o "*$r\p|9rj smc=le{QToGN:!.w댕/xnYqUB,z1"p:YQS n!z,⺉$Sm/WlQZČ;p-qѫ#SY۫$n侹=tA⸮K < ޓuJ }5W.%` sD./2xR?-qd!Gl9' D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2.95J|~9("”|Lٞ>)ZG_$s}\g66W`M3sع㧏]o4>mZVXɬ6'!yTَyΦsEEN[器^\!^K6sQQed 6 q2ƹ8+w#(:z.sPf0^*_#nF¥s*7Uܾt]0_s^(&SW_F/YؑP< sj_3۾WhzMDb7Sg?{m+L}U܎KDO/@$uio'CQ͠-Elض7Hp6/UoEspm١ ^=Șr+BRY_%be[g[Q5s-EQX$Sί3S-ʜOq^>_TƉ^vT5 sk%ƪs8x~ & |ĥqRRQD(Hk7npNVkdl ū'̳--?{O삾W`k)Z2gm{hٿ$-X_ƭ!Fs}ùA+Ĥ=79m.>+r%Y.78/"nTR>?=ᒿb+^Vܖdk/ 1x9@+c/ j>Xh>O/3۽ *ֳvhf{H®ޗږS_Ia 2$D˵Kb |$%ޏel[h,]ݎ2'3F>0!'0 P4>c#km]AqM&z~ic^ćl{Uyll% &r ~wq`j^vShvnQbe_!`lDJX`bmA1u3ز*B_dclϋCqw8*n7ȭǚ#J򜭮8V.}Y=ǿ3޾zQb@ g 1ybc㑎Cig:'*K5LEk[l^&3_Ѿ}1 ձ3c 0}yA3^k{/KL \|W؞)}G9?K@6>7;~;09q=;倻uhws= #sS?\}yuHsfk3/^/ l.X π }LcV3j /(v\s&X9_vze{' MI<%kqMm"\vP u.`s/>m /dIuuSܾ\.A1b-b:^̮(8k~{__VlNrdAᓋK\$f`R+'B倛8M.F |Fߪ%Q",/mO3]⃚M&?\>~>ONXgD>dfGa8{&:F'f4^ VW7}XVJ[;XtJ|;UH_6|἗E/R 엿Cc m' 2h!nǂym7k֧;Ș3a&Fr0V<,Kㆉ)ᵈ /^?7ZU_K4ϙྎ3F{?RyKcw9 ټxl%r׳UQ:zӱ'~A|$Z=t&&|ا/>xs3w@'\̗5/A2qyBTK\|(rXS3 yal}galjyHY5O;v\z=Ze}3 ~8N\u=1Ȝ]3y0+0qR<9x9N*˛ oOi\ Gm/ִ2ً#{s|.mH$e;%(6"f ]lv0Mu-߼rOtb;m H!^dXڇ70IQMvl2 E[&/n_X$Ը|_~ DLΫ$(0v/ )3 #&Q&=ZhOj}Hfcs{,$sN6dJ3Ǟ ߑ$ѿ&Q>&Qr,џIZ{g6ء.m+AIB10-X61*3VtlB%zG>&v:h۳ ƏE$"w<GqeYT>c7 3zzع[q{xͶ{uEy^\'ElRwr6t!*nQx%}?X4RFvFD5>=d7iƀf<:1vYe7pcw T=ox~Q}!Ʀ\+r8!"h{n>7׶JU:ŌLP׼w9wxb,".1p=I,uHP=[?ѾOats#4{& 8ibi:x?qH!'@o:r~qO>`D`Y 0 ,D/g5l e'3Oܶwh<8gobCY`kGl dRE?DȄN(ȮwIV($E%D ܟ,JbD H*L/8gDdždjXyzk.OK'k[5Q~E?K`i be ##gKwgjKsm 1_;&p2Y7&Jb*)n'gQ}^N2  ڗ )hϥ/)R+Q$q30O˄\q^/?SEEqQރZ!U|>LNc+Vwvyf Z%?(1v^k gxyC|h.yQQl3`F7v/{G0Yݷ٤b$F0:=&"\Wm˧UF'ڈQt]12u8Ym*W !c )&U 4ɟM&5BmRsIxCe%lkIfȠmeu>zc&)$=iv|F~-a_7ybݶi, ^{Lc+|Vs _Zm|\EFaBl읣_6狈 Q4v9oWY{=? {7'f/ċ;%jvx(qʐ0VEQ,ob76辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt Ciz_@'_^C)_o95I\1Jvخ 0W\τC;OۀV 3fi^3|b_WGf~^X2p}+c|쉰ѐ^G}/$&۳߅Z)C èjOA\__2 @?Ab@2ueߵ.} |Z}Y~q;NeH??i蠪rFNQ{J\=M%/辑:7mN]H'1q>t5MԮGf'zn&oX$ge߁N4s{炓Nn^֧?&Y0Lhz‴M͵$u m* ^eADZOi"f -8(!,lXر 6Ul fw<{]/ΎbKu/Ck|͋3lz.N)0'bO~߫?yZx̱q[:=Q"Fy@R0zܳ/}k|8;[޼lݲ3W5zv K~oGk!(1.d&\S`~gw3k;~_:n/fhM"J{ 30qy㰱kkM'mw~~Bݵ%noOq;8.pP>A$w L:]rItE$I[/UK ڝD, =*nEAru_I$]##\͉e5Hj)l_|QMFIrX~o{.I$I5N/I\%Ir(27IeF/?76ţ*9 'EKRN$y8,n'I$I$I$I$I$uv$I$I$I$I$IrHq;I$I$I$I$I޽m~-o7$I$I$I$IDlw/C{h}HO!&_#;і?j/BI$I$I$I$IrKHqTᶉۿzVqnnQ";{5s$I$I$I$I$/2s۵/#: ֗կG"{\jǡ?MraqgϷUAߟ<s~IT$I{aY$ȓ۟!^o^uFp_嫼|&./`7AX6`ŷIOo^>߾POo{,Ծ Z=9. \k>پ7_l}R)8G)7%oSO = M<>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[L{Y0J|~9D"”g{Zookӯm,o˿Zۧ9sQ\׃폶6̩b>揟>v9kYa'ۜǟʇ\9Se;92gΌqzGEr1B/'rQzd qm\;|Aq|Tfo/.grC;Sܾ} r 'w3O 6</Hfݦ'_7gR(Q x]7PT~(e%n3(rK(u.eͳ@>*nKm=C=Kߟ@@ƤGC䇒D^mEDӒUJ͵EaܖP;Ov8(s>ű{2ۅ$GEzM~۵Dc9l/ $vOהWEOg[DQ[Kl /n>#}9 R(?e ZWqJUnG)$]|%jlH2EmגH5O@o<6Rϑ_a7&=IA%L|&@`_zߩ_2`"PϪw'W?QGٮe?ōVn6%עEH/0{lc}[PL| bPC>ۧ>rȿҧ֏c֦'n-n`}gmY,"zfQA3i 'BI/!p!0B+"'E^9WIh@I|l_5%%% '>)\0p,\r% 6GURKDXC;m8RgƲk_|P'%f7٘~>&=A0`X)lALx/꛾x,a+Kf^݄B uǾ &0/EJwȱzU9Xp>F{ {G3x6 5%U8K졜psabJx-b8(OMc՗ҿoc)s&u̥Qy^R}m{6/&\C'ގ5\gz8FxڏEvVbx^66uem̹i"#;/3gU׭ ,vE]ƈ,9A`vW`6A<~TjV/=I^6"xUl.4ՑZZ2M8?ErlhC$b7%C\=x7tr;4b2dg}`|MvlSZ. LVmj\/b?A9J`자'vcȫ*N/'OK@9Ychs۔'g#jҳ'Ga2# l_^ (Q69/A &a3-h 1s C_тe~#?caE&T]qdcb=`XdA"rwCq{D^E3F}b=gkGklۑW♈.U:qtFwO9'=xdc-FNlbӌJ١ اvT?ڽ>gy>_ÍSIG2vrqW\Zأ56JUN \۪+/VYf `CG1Ehsy(inGpn5YpD1٤F1ƾi< O ޳s F&M?0ǫb,$QV PTWSܖ"q(xr9ۿc>}c{# $9,n_V1{m[Z_?ؾb0B2 mY!?CI6 /ZP!I7ϗן)F"E{!ls%n~\wCwm 羒G{h-9_~ׁ5< z}S𱝡}ñl}}x5g D j鞛M*bܴmwx.&aWIGVqxNq;3Cۑ0#-md#}2+7s\31Q}^N2  ڗ )hϥ/)R+Q$qsd\3#JD6!ßɾGcZFsށvoO>@"F{ Ž˺MS{ L^mr)xqsbbH£AKZ, JL-gW 8_.=wPۑ=ކ#\=Ɇǐ}ce#?MNS1g1CĞ(V+30O˄\,ʹ☴/?SEYmrHy~VqU#a ¿67F՝n^Yjj㟏pYYG_x!|;^T`\ZC7g;+rB4/gSyVؤ⬋0ǫ}q1{&mV3?Q קR~®'Bx<խ֞iQD~I6uom9]q}c"n8vVAʕh}Yo#*^SMomǎ a$+2D &/'g8ї2%U#>y vmΒhد {%J6ɂr{:ܗ춽OcXc:[֐-á|ƞȈBT< lz@s \ؾ|q{>`b#?*k_a셵x1|`\{ۭ<;/?_?gRGG!> OfY߳p^%.xzfPWK=|uuDh}ߏRPXDv6+ʽG}mG+kq"x߀}޷eAVĶej)*˾k]+r v$FFOhN1 %'c%Q*Is%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#Ge;M:63;W A_38BǶbpOeٱCO{i={;e`Ϣyp8Mc|9FgPD mޟۺ>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-NҐp"(T䕶m_ 8N(nC3i!9[DZL~1 wY@sls1/+n/=`3'Ƨu "rw~fcgfMGAs2F` X9f6Ο z^.{n*>a@;6^-V%A ,Fk$F0P2$sp}|0QqW_lKoMGvPvʗT޷}B0|Aq\!7Ғ䖃 LF$]-IkVKx[m;X~o8Ǥbi. n ǂ$ -ՈNGÚDOPZʧXN")>sDn~`OEܦ:Цݓu-Sq|w|M= MG&q#M>|/IrHq;In+mS>Eŝ_sOIq;cI_Av/cum+^JU6mdbʸߣ>ML^9 +! lS?q}s$I.gI?bR&ɻH2ϷK}6ţEm)6dE Hq;cH'XF_-W46=;~`"s߱#-=QV։~6߶M4-ї'm1~'$I$I$I$I$IH!v$j:񲈐X:+ø#Lѓ|U_辚G[k&V3g*<$I$I$I$I$IRvT~h()ї6 =mYqaė,XGvضjRN$I$I$I$I,q~.ˤE/d7cySHq;I$I$I$I$IpNn?)n'I$I$I$I$IrRN$I$I$I$I$wOY\%/=Wu'QY$=^|sWoeI\#O^ooxEy^o "*$r\p|9rj smc=le{QToGN:!.w댕/xnYBIr8I}sw{|̓vq]WxA+'Er'Xk]J6e爸]^4eěZC |sNnTJ6vv{Ol_GCa{ﳟT[[(Ux>O>~)9K"LjWo;lVsqvQ/ߙez]rj9 ,j?rPt9eE"I(<)Ӄ=-|7S׏I_y(q}mlGGejgTsOaޜic}zlǵYmNOSC.1lQ ζiteAq×K9xɾ2qPnm\;k{}=9(n3*_#^'ͅf>*7Uܾt]0_s7Iv|_ *}"K&xC=OmEn)bŶElyGm|z/{k#n59*nDž]/-߿ޓ|7/e_IW8c!%WIHO `)xm*^ ַK4T2Q6v'"o56$DƢA?jxla#{7򣳭1(nz)Wo/M2{ KpM`*ϙmSdbDUN ]m>60Jk"Hi v]<6-(p[VR;0 qT>ozt~v^h1>Լl 5ս% \|WsVI{% #{.eؔ(q\ft糂}{..uSDԹŹ{֣sAso9ws;yYo륟me+0`vJpFE܎kd0N/Hr 6,k[bvFY\ }X♴~~Vm:`E #dsRd% \DIW=~I8 VE 5}I O7;Pq,\r% 6GUeKLDX+I~1^چD+L +f,5M~yKg}d}6 $/}/ψ|>0̎jpbMtN0iXүoⱄvl3/l;0-vHQ@n#I;헿Cc m' 3۱|qAzq=iV֚5ؿ.n1Kr͂b_' a=,^ڽW7LL El~:b]m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#51H_>}1uwU'\̗5/䌐5{e)ױj۩~侇bpC✪1sf]`CqbҮCavGӎ;Ӟc$7Ul.4ՑZZ2M8?ErlhC$b1B lzQb> j܄Ll&E5ڱI&oLԾ}M`e lS}Q'1/;ؽ#N>#WY|}$_dg^K)RPIllned&lC{5?0;$5$J$J%3 VkLf;t¥mE;>( \(&:"6FB%ŠMBNGm{Md=9EFl{Y vVȱ^9ۺq>;gtm_^6цÌYWg>9nDu`O2?9D\kg%^r%_8DmQڭrRV]^ }J'1uy_~^~siMt23&O~Xʾ> łA,9eܢv \Do2T[ċX a}el;۷\7'@g>}~q[2#*oƼ>_>[k+f15xiB=|*@U_Zlu{  bX?8G1FˀjTN4ɣxL9A Ey\i'|^D'G9pz}30(N z$q b|x5`,ѹz䕘g9h/ԑÜU#`N _υ ~1<{۶:_ r+yp[i׎_wz?/uHnk lDm2,0;9F4G04:2&Ֆ`3;8.>Ǜw/,q﷯jv}_7?;/:__yԾ"6gn[!Nb!2O QD%k@/+؄.Q Od%sd Lj:,Vb{?٬{4vU`x&*Y$n^6A_L2;ױp,Ⴞqe ^0/i$D*1]XƝ4`AmGx0,v,7 wo=V=crK -a[34#{R {Xd Q|s1dzK?0 (Xqu p y%͇F0>:e c,w`}5/|}k+#Zd:2j[wl؀IJ;_wڮ%>_AFKW /ᾤg}K ߷'ȗg}a)bhZ|{@&={}q"}FNU^Ϗ͉ kKgƻ0VEQ,ob >sl,~gwܟK=_vv -wy5̺{-? Xu;Mp=]hh~ XNY7,}Vsx/yO|ybo|dO7'5s8*ic2cF?Ňv ˟oۍK<;/?_"hߧh>0ZP7{=\cʻ{"y4dQ߇ ecwVm{ (>0GKh^۽&T6qh/im"/PymFs?Z|$mEϛD;`nǴv=G5J+n%WmRT}׺%x}Wkme8I"Lt NB>m8&JVD?KTn?J>e_0}#un.yNb|(k>'m]ͮOPfMްI$B1h2'?o_eO '>ϛM,aжikInUr8>2$"E"<ZpDRCXxd,l3cqu '2bʗe;ۊ=ڮig?ۺ!V@_=c!Di{{'/>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-%օτ9̣jWڶ}qW,q!hڌmb7$r?s>3991:yyY$<\ Ktty׭zH?m3k;~_:n/'c=LLfpɼqصᵊ&Ҷ;I?HqZ?kV<2.u#%,䎄k5VX4bOU1WW6?:䎁IK.$ɵq+q*07Z}AVEVGw{{(HKaPb4ab'mPDY/2I;ѥ^Ҽ}Mlfe#yFa\_G"CY]$$I$I$I$I$93=Q"n j"9^+x>w;{w:L7c[=&nͮ7]BU l{~a!igpo,QWׄ׳AI$I$I$I$IrfⶸӾܶjkau"{AWkHkExNܹS,г5rz,n`ney[yWII$I$I$I$IrfVhz, [֊=[ܥZ4n}d)8K,͉ 8A}M9'l~.+ +6Gry|SHq;I$I$I$I$Ie۵@mAۑ`iF-6I}7 N YEFmv5ӿZXO3i~}x|_^>Թk0ۛnDeIxϹw*_޾z%Iro߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ ܔxs0~O9@>z9'`D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2e<+s8YX~(`y.D#S2~ҋli㿙=eO~O/jm>Eَ{Folcs_?>:(T;3?~Lf>eensb*raw[ b)X\e;*n/ʗ[*n_48Y"%uL!n)F>@yGeQ܊#'¹mɬOhm})R. D(*?~² f Y b!Fܞ%xNL\zOv c+ˡnC@MKTbe[g[ dvs-EQX$ί3S-ʜOq^>v!4cQ"^rcpv=X[/@"WDb$_)j8))P(0k$5K{n' ]5e2b#wyֺE0EP?}6?"]lp-CY̾m?uob={7eVt\ ŌSr=79,^qyWNJw ~2wbeE ( jQY׽v,ЊsQdfoޙ$ spohWř}t]YqJ~}u=y$qa׃~Km)Kxal%aGĉ'0.5rH {/A,%? l!HLFL;ݎ2'3F>?56$PDƢkI7}^ll ۯ^oo0՛K礠&>\f {sf`[/X0cgU+؟lWF+s Jk"Hi v]<6-(p[VR;0 qT>ozR6M/Me/Lw>+ܗg]ꨳ6sy}rͺavu~w;3g3~&-*1|s@_٩_os18h^luF禍;3;Wo, W#nc&n(=Z_s`ϻ/6(AV0 =/% &XEl]Njgq͢2/aq{g.YO酒^4 CCf!a>VDNr:UKU _N0}QBM_p2ᓜr Dz%W`hs4[($(J=|1^چ/%}f,5M~yP(jY{ci}|L8ٟ|`$: ދ56:DbMJb_̎mM8 0-PwHQ@n#I;헿Cc m' 3۱|qAzq=iV֚5ؿ.n1Bi ayXb~ Sk1_F*nu}Ki,3}+f.ϗJs@h;m߳y1Kzgw9 ֣ucO HzzWmhESRǫN^/kJ_`G!kʜSԯc~ SB0P8og̟2ܽ`t?;z>C'ގ5\w>c\lg%VNec#]7[kƜ,>2cXz9^7J=Bmu?^ Y,n6Y*DQ^' #W'U2ɳ _|O3hOj}5r-862ю:ط)1OFԤgOpdF-Aƿ&Q>&lr,_$(Lf;Zrh[ABc(&:"6FB%ŠMBNGm{Md=9EFl{Y vVȱ^;m|QT\]'ElRwݓmCT<21KXsJ~6LioP} 扄k}zn;xyuc ʼohϯAToz$ZC͵vU?\ɗ9Gklvssm.tb\bƨsk5g~cy/ )izEggɳcI bd<3}x\@4/g/M.6~ .&ag$ܞV>E̕(ވke]"~P QdGAKsr~qO>`D`Y 0 ,D/g5l e'3OܶwhAY&3' N#J62AMr$L.8Y ZQ"qZr©MTrX 93M <˼]ֹI~Lcx2e,<{5}֧%2~n P {*6˫"*6ρ?'?_inQ;\b R މ@LP- E,26}A[Bbݳ%mdNW[klmX&e?18Ef {Uė#[螏IryH1Ǒ=^h/vSP9$12i#L2\3߷:_:ur~OlNrf`nQ^;?QPI kbr啘g9EkNݍU#`N d04A-߻߶׉bs_ɣ-)Cu`M:Bd=v>)оX>eƳys5t&g1nڶ;e<0sY9_}D*Ps6qg$.Ю}ݐ6\"koNIɣ9)?s5q+y?4=5q {&Jր^"W I% IF\'Ǔ JVl!()}A4#"Q&lN=2̵׷}|7K( ،^mkfrnKYfS//5ĘC`_Ҫ/tITbj=@};ail+ڎ6abYXo-z{Ǿ b[/8|u"D".^!bg-R]aQűnR@9LJ"\gwa|Nu|U"^YT=ox~Cݞ_&,bQ|ybDN/j[C{s]9d.a1PW0+WZ-!$_L{.wVC{>,< D>B/F*0.aƳX9!3穼?qEl lRqMgc$w/V1Egk#^PFږuLj>)6`g\7HܞFK>85ݤ֞ƲdCɊL>5B/:Q€ɋs xCe%lkIfȠmeu>zc0ǭ@۵38HdhMy%=mX2V5dKcs _'l#EX-^u$gctڞ2/.>_D^؈i}Wc91{a`?h6d%|˚xAֽu.ۯq7O.辸=뢸=_Yn,gRϗm Fx^ nhB0|ݎ|elS'"\j%;ڵ7=v|;';Akt C¸K(NL17vSrjR/ic2_61ێ[dm8tft鼈?gRg鹑?[{i=*{AllY]קkxYO8mkss3a}S]BXJq^q+Qh'HlX&֥/<"_k/ nɿL9ka;oZNP}2FIs%"+&oMۥ.oI&7mS>Q`l|gᳲg '99Ls%7o_eO 'B&ph(nڶ(eFnYVeo3N/QS+Ic:-R,#G)e;G:63;WP"C*|lQ6㱭Xzvv^ZϞ2Zkea,:gʋ0Ľ%Bq{'nvc:1#F:3|τ_ɼf{U.1vgXG\s};Z 4d&\3k?} b`<1xV1ypTvsSٴ ZܑAPw67IwCn(%-LF$ɵq+%`^Ov-ڝD, ]cRwׄcPO$IMG&q#M>|/IrHq;In+mS>Eŝ_sOIq;I$9IOwAo|_2I`$7}oI.3~ R?>M@QEgE/hQ/RN$IpXN$I$I$I$If{ڶ{ ˒䮒v$I$I$I$Ibo:IrIq;I$I$I$I$Inv_U'v3q~}r82%pC0$I$I$I$I$9#$Nx' $>zgMڸ~5=#5RN$I$I$I$I+Uqija6wR}{Rܾ:R^#$I$I$I$I$BzgK":" <9_Hq{ϞoG߃:w?yzb{,I/y9NWò$I'?C"޼ ߿Wyf?QMf{]g9_n8NyhD9s9ױ?_=(* ۣe_~TamQZuAƌK<7ᬸw!@YD??"F~8`y̋ro~&R>?'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ xs0~OKQ=M>|SΗn6U?l}jke{/v;gI䳂Cm羝j;.J;[LKNͳ=#uE퇲_.'H$;0%'%z=f=?}\#?e;͕=hlS̜*vc#lכ3Oom}2͉y{|ȅ3m<is|CbᐋQq;|Y乜CLʗl梢Kl9&ҍ9\;|Aqx| ~}K1gr~N.7?ў[妊? Kq~& qUkʗ3A}6qVq;/b?پXץ}EۏBXV6"b^"]< D6T=õg~3LzOv c+ˡnCf=KTObe[g[Q䈒vs-EQX$Sί3S-ʜOq^>_TI^vT5 sk%ƪs8x~ & | qRRb Q`(Hk7npNVkdl F\u`:j~鏭m{!D8U;'ؾZ ֳ̙}a.5p8?)nX)-\_ƭ!Fs}ùa+es$˕*zLo*)n_U4i:@B[ۿ Eջ$n[Qǭ Kek3UZsdF~tu}6W7ExIf`sRP{ . =W9uwL,1Գ* bQyOq۹F~-Z) cڮ>ַE`WbPC>ۧM²]5G>]ZqW%3yArEe JxE-6О%ܾ\䅊9G:nbBBsm6qHdz^2/a[3[ޯmOzyI|Ecܷ{ƀZV@C1\egxqΦgØJ8F$>gJQr>R6M/MeXܝ >e0Ǝ~9y)C]]3D6sy(wztνp=/.m+k>v|Yyzg@yޙlǬg_4QL^os˂H\)}KtX UeQ܋wUgO ˈ6%n )ܹ5ZmY,"zfQA3i 'tЋZFc%+8&!.$l_%1$Z%% '>*@űlrD07o3V%-1aq$uxi_01$n:6A/%f77x˾W|r²?#&3; Iދ56:1äbMJR رͼ`_ĢC P۱"EE '!BPܶ_96^1J J nǂym7k֧[YkG`q[q/8;{rCqxqĔZĆqQn㟊|-ƪ/٥]RLp_늙K?mұNl^L<6y*@= p>ka^mhES1'Q]~tbb6(}%g+sNQW/DN X/υχ"9U{?ssf]1aCqbҮCmQc n˾u)tbJgA욝8ka\i6ZMi8/nc_v ^$IqZ3*TOگiͳ#ǰ"e g\i#m񵴬e$@wup~ئ#nцH>(nc`@ x$>'}>D > !Lj"caL(Z2}q$Aئ.3N bJ_v^%A{INH0)vq}4W6Yٮג@{ʼT;[DoER6cY'+}u'PlDM<Նџ!e ?d@`+ Br~VcȶPv?Cmk~˃~?O!F/: vDɦ@&UId€ 'dW;JL$+P(G"0ɢ>fP㜥wd9ƓI6~.cI?߯|>-񰉙NfFB="]KٷڧX0(6ρ?'?3[N$mBwCw"ԟjxK#vG_wh,g/nK&cpX֘gvzZ_gkml{ƽޙ5xy/PLx0VbbdyH1>1z^P{rIcdnGΙefo//J]sd Lj:,V:/!lN=;H*0^ gHXPgn^_`^8oII ΅8W!nZ*am.S//uVƣ:O䀹1iQD~I6uom9]q}c"n8vVAʕh}mHDG85ݤ֞Ʋ<6>$6F谍^Wv`>ϗp>yĕm- xA̻g6Vol$;mΒhد {%L+?A_p_ҳ>%ck`l[`K볾a x-= lz@j{\ؾ|q{>`b#?*k_a셵nA%jtX&~z c ^ž^΢{~k*|ccs̱.۳/rH.|ٙ-жPQnD0V,cW6u"v^]{{3`Ƿsc;Agݰoa/RM;WGf~^X=vOcx{"y4dQ߇ ecwVm{ (>0GO1!Tk{Xq;üK_nr4Ff ս9H(_ $ o,_WKQY]җ M_믵ݗg_&3ә: 78 iђr&JVD?)I0i\E7RI $&gNsҶ)(zUm^ Dr,;ÉF{{/S\p kQp$2 m[OVe˿,1S+Ib:-R,#G)e;G:63;WP"C*2Q6㱭Xzvv^ZϞ2Zkeaa{~Juqi{7Di{{'/>>Q96nX7CĨ;}XF{eb>~M'rg Лכ[#sW}6Xasϱh-%օ8:\{Wgb4uNm697$r?s>399yyY$<۟u^0v_{qxsP>vn}LCt ߭w3k;~_:n/fhM"J{ 30qy㰱kkM'mw~~Bݵ>X<cBZz LZ-nϸ*zQq@]{sQFb^4FON'n kuP%b\]-IkVK|U`n}v'>s: ~Q\,ayFpپB=,sV|D6H1Zϥز>P5e_Hq$nsb~ $-~ W<_T䆑v2V۞ç}K#`ES鉇=R"^ඉnGrr-|]fQkonr·b[+nt9_ $8?KPt!&enˌ_*o:mGGGUr2O8ONr[mIn=NȯG*}lp~ŋFmv$I$I$I$I$1ݯdO^"CIoMK"$I$I$I$I$yGM6?IRN$I$I$I$Iƃ__v$I$I$I$I$IrHq;I$I$I$I$I$u$I$I$I$I$I:_<{ /m5 oO$Iz;ǯno_=˒$F x23"._Ge7ETvI|8sԾA;z_z|,nQ./Oۃ?Gu5'({68~#%&@>}$In>'?>o߾y|Cy?ݾy8R* >h~(8sRdr| |}K ئˋ ܔxs0~O9@>z9'`D7vO%;_~'r@W /ۣ!O- <'s% jn5vԿ򫷝v698(Ln}2e<+s8YX~(`y.D#S2~ҋli㿙=eO~O/jm>Eَ{Folcs_?>:(T;3?~Lf>eensb*ramcZ\Zѹ7/kQq;|Q乜CL \T|i8zD(y;k{}=9(n3Ǐ 5|ݟgrC;SNqVq;/b?پXץ}EۏBXV6"b^"]< D6Ta?ڈ۳I\/i{dLvd9!)ItjrBll+>0Rne( \?z}~ñE)K٧.$;z,JDKpl|ݮ'g# Hd*HL 2%W-@\'%  Fqd&q~iM$a+xLƶXl.0Z^&?}G ^sP~h=˜ZmNlqnڭ!Fs}ùA3JsC㛘csI^9͗+)n`ϮC<,5lt+SN6r&qw|!]"rzV.mcpwO q\`Rr=),w~^dX4[Ieq"BKb ž $Rc۠jTC*(sB;l(ScCOa,jDyz~㱑ڇ}^ȏζƠ_^4 1xN j/ae6,n}}LE?c?I6Pҋa|,$ʁIWKv@%m_5%%% '>)\0p,\r% 6GURK YIv/Kpόe׾&qӱO4?Je_>#{o1-}I'33"LzDGa8{&RF'^ VW7}XV쫗ٱͼ g!})*M`? zwȱzUVrq;#n^#H/g>Z={J{F a=SW7LL El~:b^m,|u4J\і:K<_*}c/뽞ݝ2X s= 2X#an^a*=Oi_ |fsݝ0Gu+K8˗5/32u츼o}he>F>T;/S?ݛ vl1",NÝ`p{;p=V|v8c(sƳX9!tlٯesnڳHˌec-{ڶw/}&nWTGkiYH**7q˱MG. }Pܮߔs9m$PdHd$0\$"/n_XԸ|_~$rJ%.-D^T$&p~U?g|r̠=e^]D;`ߦ >g>Q=9?aZDɱDl0 hmE; &cr,  +:6 #c.v"E7$"w<GqeYT>c7 3zzع[q#{!ǹ KstO{>Qw[,b+272C5''!O70~Vՙ!{'|*}<?Qy{ke6Y|-\Zأ56JU|nmLPל>E, [u)inGp5YpD1Q_xF1ƾi< O ޳s F&M?0ǫ޽mk#޻yCD{-[h~^@zw~}w XbQHf!Z~M?1dg(;IƟ|嶵uvC 219'x8hNtQ) l^e' dW;J$NK.P|0NKnϢLk/ Ï%h{DN|9Ko86'i{m\ƒ~׻_sg}Z"#_<_ 1 u9l*2iYs:(&~)Pڢ ^R+cݾ%$0==KQ2N}_ܖLᰊ[1|x&ڊz{̜=ܵ {Uė#[螏IryH1Ǒ=^h/vSP9$12i#L2\3߷:_:ur~OlNrf`nQ^;?QPI kNǛ{.zy%+3ň^5lcs 7?a;Da!?۶:_ r+y4a߱sg:&g};cv Dz)4'+{n68qӶ)㹘|9^5U5p׿8m.L Huܿ+xj(x8?G咁h~Ul6 >I\?a]2!m%Ej%*ߜ$nG+rlS~6j$V&hzLD`6<#Q@'#P%+6Aپ aLΑsx(^6k'ky>aL%JlF/6{O5l3y9ɥ,{jytb̉! ^0/i$D*1]XƝ4`AmGx0,v,7Nwo=V=crK 3FƮYm+ݓ"fŊ0~x%cP⛋9>XJuunhGaǺ}K=_|T0m7+h>6r=݅9U-_{gQxIVu{~E9WW#rzVۢRރZ!wi*sa0+WZYxpYY2dYP5|;^T`\ZC7g;+rB4/gSyVؤ⬋0E=^"n2zhwsNb{M>˧UF'ڈQt]12u8Ym*W gKJ85$֞ƲdCɊL>5B/:Q€ɋs xCe%lkIfȠmeu>zc0ǭ@۵38HdhMy%=mX2V5dKcs _'l#EX-^u$gctڞ2/.>_D^؈i}Wc91{a`?qabaqt_ طesu~]žol}q{96Eq{;Xɥ/;S;ʍf=і`ئNDԮKvkoo4?zvNv,p'>+87ɗן)n[NMtsR3W]; blW}b|ێ}S-6`3O {Yzn$G!>cllY]קkxYO8mkss3a}S]BX>e~neE=q{ol/R~78(_ $ o,_WKQY]җ M_믵ݗg_&50P|7|pB-'@I(h>dstQ"R.{bFܴ]:v!d P~I|N6S]˗z&>+ |pӞ<:Q}kQp"a7H6m_~Qf\QnU68>24"E"<ZpDRCXxd,l3cqu '2b/e;ۊ=ڮig?ۺ!V@_G!|Ҝ/a 0~߫?u}|?slֱNoQv1,бԟ1n߷}&4N77kglG̕l6vA޷=z=璟cZ(!3r?DT=+}^P,qBq꘱MZg/inqm}.cw%:czR lύ`=0w>3k?} b`<1xV1ypTvsSٴ ZܑAPw6f߅$q";ؗ+n=)H wsPHK[&<+tE$I[/zbn $bQg_& E4)_6]ݷD1G.R>5ۜXo)l_|QMFIrX~o{.g.${J;@/Y:qAW>0,Oqq=w?|L$ɍs!&enˌ_*|+nS<>PԆ?mblYKv'Z wi'B"ѯg_Q"nsG|I$I$I$I$I޾UC?-r/W>/'$I$I$I$I$LX#R>*`B?Kbb־sۖ]=?ImRN$I$I$I$I+@RPElD ^+M+Yo,hG_q^RN$I$I$I$IKZEȑ\"+uJmDDZ'WEI$I$I$I$IrɠhkY@~"2w[ܖcXq*/⥴QZv$I$I$I$I\"+|;{;)1llhA`#7Y+ 6T$v3ġr%+}aH(|<]URU]2՞#Uꚿ$AkY<ޔƜ6> ׵bw=VۯXgt͸['{0rjsŇgv2googf8wȓ3+ҸGnzrf38j%v܁߰lr|>].w/ɯVO~Y|\k\.''g[Ƿ]ہ[[WջPcb\o "no0@HܶSz$TsgI$d-_ǧ8hf#q'G!?Ο/o˹$kJxs&K/d;J20q;hIwJg?pLԛ]T,׏trN{nxQ~YN{I_8.w=g!ƶ싸־'ەb΁/*ņnk]^^̚"]6zM"ܨx9y/v ϙ?7O)燧د5g7hy"1.sJ='ڻlu%kH9ȣKö"g[ߩm\v!KJ[l&]>ByǪ8Ь+|Pf< q}5_mj=Ͳ{W oz.+r+ӜZD趸 "vj|,_ē؝%A{x8Nm<AT~ױ:fEo&, q;"7[/HpB熈VƗ&hdJmj^ΰhhXv!zrB,[EԎ?m`>8]s( b*ucqjñY[SU,~ѩcˎ(c8wVkn܇T|@!_Ctf$U!s6* A,dB! rLr?"p+,7Pomj=xFW\ܾgW0'n"n*n/̋Do]: 'KZ1Gj9M@s3} ϝG$1Ľ q%}yFis ;hjY JTf;/J{m /ٯh{E+S߮fIX1s++{Jokg~ ȧx]T\P׾ {yõ;^# kuX`9 zԾm>6I> qñ)N4zRmt~sx^³!< p]DD*r䳘 # 3,ҳ hW\lTzl5/y.l[m"}\m+n/^+yC@݈ܶەmW ғiO97t6;K̝IA3Wl?'Y7RP9qe:둁kct֣_kq񜘔]+X13]NU[6z9< ?bu_ל<ݞױ`u;]ka$ ?Sl pr,h P 4ފ5co.v"Qmu $"7{Dqy>g`i-\c̵\k#r6}Cd K|3su"򑺾s[rxU1U.y:Qozvy"1]7qKjrEM ԵUgXҨ5j ޠlj?c\]üz~4bgG ZVgoBo_Y1|ǀ-y p Iv9/1s>م-{q{A,NbQHNB4rcohHƿ}utA)?.!Z/5MVXqUl2xQEv^0` ض 5JDqHUQ̬"*H*"lb!9xE>Vqc8>YRxL3B~ċt,,mbA~F9kf$(&~]S}Qb)o m|-݋ۜ?8̢l5N3E;_e-chLzq7_; `*/%Gks]1rX;ys)r7~g'"jAecd/L\v-OWD_3}[vfյúƺuqx1pU`.Yx}g{T,[wૢ+ka"pz|x뾁tLMw ޹oz^zMؙQ߽Ę'XW97\FTaW?10kx$?*D_lc_~#;Q} ?Я|]6mqZ5m~rZ">G7q]Md\0/DmJA`_ XZ\DljQ?ՈUn{&f5ptǙy~}Usu|<O%XMr`ͪ!9ۨϓ_5+0y[5FH5/Ũ]@k5VsVGu1cg\[kmR1qyn˕h~s1QIA28>[q<("?xAjNy%*0&6g]{$nכTVQJjo9T 4>Ncf/l1BRgmEVD39UV }cc}PEikN=8(Vla_*a\5?O~QϼKsޏݲj<k^:7].w/ɯVO~Y|\k\.'' t+3remk7~}qg#q'G!?Ο/o˹$kJxs&K/d;J20q;hMwJg?qa$]7#ǯ隗(-x>k῰= q\z+f17eCmq}O+Ŝ_T &׺zYPubT ":޷- |JJE:vQ`m<>k c*8vXN1}7QDXT#]+a/f=྿%>P("h/T~mب( (0c$1'IjST ֠X&0|mL{}1.%d1ƣ)NBxXT~Y?hp̅UbN5qlx)yUYȕ,M܊}yY;HqrK=.S̈5V _Ղԕz9#u{pm }gmqۭ ,G;_(H qtyH׬?9e88o}Dp˜|%$l` $DVEj%H0<뱔oп AbpcUb*!D]Q?1*JPXwT1c#ن\_+o͘Aqrե*f$"%,|R.ك*g+;LI0c{  cżOy빍zE%}jY JTf;/J{m /ٯh{E+S߮fIX1s+Ky/k"Z9 ,?'^ʵ0*WD?I͗϶W}=< lzʆ69+[sF̎^c墉ZWWZ 1,S|Xs>K@>"V?=5rts>3Եo^klsr^"p-׈2Ă?׬o\bc.bhҎY'5~ ό9}6Wwe[XϚ[5}SGlUqW?5.r9QIENDB`llama7b_autotrans_8to2_predict_log2.png000066400000000000000000003710721476650141400355000ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRER^sRGBgAMA a pHYsodIDATx^=^I?00 V9\U7f 1TBQ 25 Y۴S#DY7j U *-|'"=\/<u//ۥW?|ɨ@_lo޽^?ʒ$Ix,I[wۻʛF;GM'{kzQ~ׁ}no;~(On# _:lk$I$IURI$I$I$I$I$I$IȄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrUW?|ɨ@_lo޽^?ʒ$x]ΓWoeI"O_oo߁*yiԉh.__oO *_nAoNco9U%@16Ƴ_=Uf_Vۣ/Zu~:ړ+{ex~羞c$I=j >l߽jS=}pOF3.&ZZ|63÷\/tehHBD遭C9l]CJ=y{u n^ v}G}s{Iٿl} {WL&{Bƹ]f7!mgvպ7JgW IC,cO1Wy z9`tAՂH)L'yy=g~/=wmxW|+Rd:wptc>udc~c|siL|Y/,d&{䋏Xy9o1Bf{tj2V}]ĥ.Ez姯ܗK$@L̹IH%%.R|un{ܙ3/`B~D ղeL9!uڹԞ|i./^ʄT$Ign찅Pf/Xe{I Dh9s^`ҭ&>}|?#8(L@~~6C;9aۤLfr!wx[傂]Lm~¾˱XmrcF6yK x 7``I/J/ Ş3&璐x}\߇#)Q!nBȚ2i[ $׭]6{CNH;} d9Aʆ4uhp<! 95Yr6{u=缕xtd 0_a)ܣ9>ӡ#G0Lإg}m6C?bo{]&Mݍqv>ofD|@.繰l g}ܷm1ױmdh_c#O$IRB @-ON@Ϙ$"["$.T3YM2*@r|pj dZ #C=EZ9`D RV4ZL`"76!@=5 lKM^ D~V߳f\VJU5۷ChK}Am;~@Igb`@GX/Zq_Li#t~`bUF Zà tZAQ Y-k ]ٶ /HTҎ~( 'ݼL߰ 9ֶN#LH*""@BN]&!d"8g c}NB?/pwHL_rk\_5!| >oϤkn-}f_wD_/|ԥK۝=Wn?vY'}= k6lieF}-luLuw3*%wse=+M5{ď3#I򮿴H``=,23,[nD}>t}ZoWe 9fjk}z۞>>+ΚCWHsO1wі/ckvH_j%>$I$INFR=ATIMHo%XVFA#uJB '!UuGTāCԏo=C% "wP8 `^pf Ӄ!|VjډnE9RK pm/Q#YI=Z}DVCXw>x)>P%>9Ah 6ig6T )' . [k2Ϥ<۳ A׵\\y6a+AXh9<0&M&XFrp3#mCe:&5[I9$"jB 2|F$ ijsrǺO_ki._]]ķKBȽ$b>AC?c𥖯)Em? _tCodEr~"ieJ5@/?Ssu}ն>V|i}QZ郅s]yyǴ*YX"}B{/n:s{K}Rdzқdv m6 h>kl_>gHIJo9bNPjk\I$I$>|O q LTAR3O!%u=wqhjqAκ ДiЭ+@$,4dNkȶG'`,6fО ^s7OzVel>S *l[osmڡ``6A ,e=y .{9бʨYۧutWfǙ_z`]MȘ IRPw1E "6m}{ƕM) ՟4RsɺVB2iL"d1n㳞*kbjDݯoe%wg k#G]ItÇ>Gcx{ Ѓ ;MQ?F&> i3W| '}εAXsƱ#:"xg~\yy}N5_$g~t ϬtQgB#Xٛ]hoNcE@r֔qe/b l6ϩ-f~Cח07khתZh;J'.*h}L$Itn4!ɚ%!ӑ74 U~hb%ʷ;)!UP??}<ԟ;vIM።/1| +"q@ALƁORrL N*F6Ӂجa5fi뫠g1vTv6'[~6i6 El2k5|gr:P&c:E~ LceN׉ &LIHMTI͚g n77E۠Xw lɎm2|+I6{& P9h6|iP̏2U>Lߛ+ VPzW}ʱU/_h{fV3o?ig g1e+sAuΐDy i}c"T7-UrF%;t;z~@Vucg&=i}Q5 +D7Rx{Ƴu׬퉏W:~vu 1I$Iӹg="T$&dy uJB ZgOBrhC U9x{nQ jxvFt)W"se2oA AdUVt܈DZ6~}G_kgfe>Cqq YPߐ .u)>Ȏfhٜc>[KF![F=1I"@2$G$fm7f̓Q'IH-LtDi:NTja ?oc Ӗ.R6c{tZ43։3w|o>G:o_>{z\uٙM٨k ƭ?#ﶹ`56_i[Ա0ǣ7-zZ_eed[2{L6g,.·yy}N5ll.>r>TҶ*C qgo_О!lpUK0ڎ'W|{pm#4y_um ώoz="˒X۞x(3YgyA^$I$WRߒ1A҈y><7VXFDATv[q5hiZ[Q\HHهPz,?7x-Df\iEC%:M=׍˷ouhle^.'R6x6׍TGMxfXO\YجQ:Ͻ zqTHפl(52apݬ[ 4uެxF `[rBB5YPIL,hɉr2m'D(I80d9IZݶLxI#9^=ٶKHqf3ki3}bgk~~ZK;={랭}寈z}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|gK~{8`m6/wfj_3L?W]/=0R}"-[[xL-۔>L(y>~1}$I$Irqn6!!p0!u)f\%{ G,܋$In{.eMg{xt_N,5z=ЁDj")f3L8Y$I$IVɄԥT ߺL{MLFy$wLH%=c%Ys:ɔSm $I$IR៵{>Iv~& 'gpM(YSj0tInGTLd3TJ$I$\%!$I$I$I$I$I$I$LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W* o۟:zS,I?l߽jS=}pOF3.&ZZ|63÷\/tehHB^Ѡ.!`FA_'_kIY﷏;` =ϤLF9? :`ۥ.LlVa}2Xۖ|fW{tf}Qz/ߙ=d2>*sŚ\ `롮I]%?IrtonOYߋ?exohG2}G>(3Ν<~-e80{O_\?&S&}*6}j<#Vr<[̵xkϔuf{ _r CS+sn(j qc8+we.>LH5'MH5/d\Gb;ωp7LH%I${͍'4Ϸo CNzD&m?dKH501EO@mz$}o繑x:m?tg6;Talr$zݐc"`|. G}<%&))%YMR@?{ݚEms'_ PlAk<`>G֑^P_(|MhL}ꃙ4k'|қ>J`Eu%wهXc{/m90 m_PaxB̰ك1y+g\jƜ=KЯz1N7rD?sն1?;޵AZ"jݘ˶+:~\NIDž>bק3h]0gW:t_k/{3=o:ўS(yYa#6M$IDsK )HO<::?c|n x^PRoS(g5tRB\8A AX ա>d)jHy)~=ܷ7̉lt~:R-6y5X0:[ lV>\X+yV;fV}l -9=8MGݏXQYC?i#,Y`k}`҂}@w-0kU@T"m~/ RNGMon^r&LHo؄k['K&TgT\ !e'.RQ_DZOk|Ƴ>'!ї68YSL_rk\_5!|˫>oϤ~Uҗ s}TkQ+/GowR_ mŏۙg`<:Pq.H0Ft.óy{mZE`3yݑ>?+M5{ď3#I򮿴9zUϗp|.9bK7C. 6D{mP}1 Wg@c/ntn^~v]u@h6}xW~D7n{NꢜSe4AKlAMSԯF `P/c`[ncyouP(6M"T -Z3R'~{&.10z] d|^Vr?&,#ő!Mht$؜sR5iU># 怇kLHcw'{ǓzI%ɋw/|oO+jY2+eE&+c1L(Sڭz~9z3 A@3VO2_?LJ,Ykg,ʕr&p}Tov&ٜ;-VYᢍG,d@]whڦ)^{'R8mWQszq_7D5up?թ3O{WMOI$Itn'!~HL`%7Ϭ 3Ÿ| )8pуpH.4ۂ4y;VHs ~[4`5d[0t<uvfО ^s gUL3 ¶yehε-j>ރs+g4AL6K03kH:?zVtt4+}KU0.G&d$S(ػ" [i=ʦOKdJHQ&mIUl6m|Ce~x\~`U\Uͷ xG0}@jt<>}b0h?2.,3mlTؙhr12yo`Ks̴l}wk3W| '}εAXsƱ#:aθW6bqc,n=xx<={rڋhS_ Ϥ>.AS˕$I$7dO?HH{P*?WY4A[Wߝ* >ws;𤇦<9üwH>3d% HEC{ d@) KʖANJͬ`I6kmX|*Ā\~c}$X&*ܤYfR۳LHy7GpHHʼnmPxkdǶ[LHMp wMA|+LA9ڟ _Zrß.ԣ2cu8Fʂ՟26t<ޕxrl,^U@L~|eY:s>i>LOΟ]7 I̯a i}[cjcJˆ`V(cdNyGso Cs̹lBDžϋ1:LO]Pl\v ;#B:Q|&j fzP=;*d$I$I"n>!YO@aH8U0II%Y^>vD"em„ٓP%PBJ0| UG9TK[sk>n`Cu`F빃ul g*+`E~n@tj{fmymо#G/뵳doC@dvBbz6(\28,mQ_zcв9E}(B귌zb?DdTIHAo>͘'OZ舉ed+YuyڷbXjR\TEԧ5 Ⅼ[uEL*ZDWOi[ Ƿ#,\1޲ __V;!lpUKpf ^Gdsf;OcLezgb^ۘBwtϗ}`g2or٥]UN$Is oIY i<| +,#b *-߸m-( $vpA? |Ё:!r:M tr) &Pk@e#o\/r8yn̦r<*,0e*}.DΕ2sN(0/Z0ЂWIِi$ f#LH\7+j[ WLJr7+cE;^jQCMx =%#Zrbjm 5J YNcm-8^HxOmR\ Z|_]V "_;B0bB랭IBz}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|O$$`B٘rgC&J֔2\q?=[q*O LH%I$I\$$I$I$I$I$I$I $I$I$I$I$I$IdB*I$I$I$I$I$I$*J$I$I$I$I$I$IJ&$I$I$I$I$I$I $I$I$I$I$I$I\%!|[yxQ}OO݁>޼{~j%Iwۻ\;'no_=1˒$E޾UxҨ]7ۻޞXewUv%.܂ް2J >rK%cmgÿs{oۿ"?m:./G_$?t?.'!Vd{3}=$I${R??} ?ؾ{çY{lxZny'}矌O?g]L: mgo^,$:ΑT=[9ȵrj>|z4` /3#,,h?>,LDsTe)*oBkےuoά;J; YZbX0 l=r08邪'S.N ~p{?n*^Q/{~C?~>WlGɞ-tk)Á}Ⱦ1Z7^VYLSs5bDlꇁĥ.Ez姯ܗK$@L̹IH%%.RΖֹ->rg-8bl&OM]-{SľfODA$*I4 {HBL ϘKBq}>DII !kʤm &xV^&bw.~ۜ":!U^~J/)>֡}n$9s6~xm 8i]^{y>ڰݝ6hwf}('gC틄gɹrz^dB}ÎOypdB*I$I+s )&ROT'R5R6Z >& -() _@߼ˇvѽrآ#*r8{|F%t] %02qaz BLz>N\E6i4`fmC@Vyאc)lm#]G3_ˎJ7f1zz{S'l_ )Xc0O*l-U+X`]y&먄H @*cELw/o(,e.Jx@'yDbA'̱ɖ`$Hx.YEG[ߏz n$PфiFj?q,|`)Mq _{&b<~QD]#`z| HxBs'k쳐>3lz2y+g\|;RƜ=KЯz1N9_? 4Ͼ VVbF=b`Ի֡1HM̍O[C]U"Cqlr6>υfWX}K/ަ}}q{Ff06m$I$I.-% <<TI"%yMBNHyN*?$I r T.vZ@ʁ2Y=CR:h5j DV olC ԸaȠ @7l/Y= nFeYsk_ mhK}Am`TzTOH>m?zʏ:`N#b4ZPb`b\V_jm]Kd ȶE`]xA"-%vD@H?%g„MȱuaBJ%}VIRv2 )%Ig<[sR?X>}VSz7O1?vU|ЍחLˠ]^~^q8zgjm/~g<с:Ӷ'vAP6cp;ޣިmӲ.ɳMY\x^n:@zfsЗ9I޶] RRϾe>"gX뵝K7C9ާ#Zscb|,Glc/iϊP+>CS]xs;җmg1I$IӹфTOdRCR[I>8DHvr|IHUQ+t0!/qzAPI.u `d3ώ `>d`d{ID7n{N#ReeS^Fl!zV2b>|, -R6} lKY},#5eئI=YDv^H9^pAE/XZ~&Eɠߞŗ]ʳA [ B[I(5A`%o22 Qi,k1O7hJ9'!QJ|Y3"OkxȾV˄;֭~@pewMF.Y !}aHԏ}×Z,| տ1cmzHo)=D{LjLwy֙ˆ T XuG&k73j;gq?W>4ч+tz3)\W^1ms6zuž]hnG:WzYӳuL5:޴~ Kh ] ~_%ګϚCۗY1h)eR{6x'Ե+&c$I$s; O?~H@%pd2?-yD!U~fmw줄 oHI]gqZkp.4%h{V>be<.sp}AE X`pˁUFu:'>7=$hBLL:"I,hɰQo3lJI4KeRIfp!qP_c,>?g=Wc"f֕5x>UYĿ 9ZϴOk8>9ڏ/-3mlTؙhr12yo`Ks̴l8uTOdy gp< uUǚ7Ű7с/U3?u_E}͑s%9 gzrE}LmYyg TaYbJpq[Xvо]Pwaq/b l6ϩLu-f~Cח07khתZh;J'.*h}L$Itn4!ɚ%!ӑ74 U~hb%ʷ;)!UP??}<ԟ;vIM።/1| +"q@ALƁOR|*%(Ao ktc6kmXA|*Lߛ+ VPzW}ʱU/_h{fV3o?ig g1e+sAGvݬ3$}?3c3װc"hCK0+gZ 7tMY: l6Aw^KYsK~/g<[['~ڞxglG^@$I$9OH|Pyy,NLRaRIϽB{[$0z$T)g;Z?Pw;XB%Xֶhi}FպAf A:[P6~Y3EɀO~n@tj{fmymо#G/뵳doCƸ8@ oHGẔAWi{dGm3lN1A-up%-$p U#GwPwO3ɨF$&:bG4JV{޿mn*0փچw+ʟ Ľp~[z:|_ds~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY<;gqY?F꿜j]L]qϤgҶ*Cg Ն _W[9~گܣ3鯹yyڜϵ"k1g.C[<7VXFDATv[q5hiZ[Q\HHهPz,?7x-DfyhEC%:*V۞[J:4BA q e3ٍ>qݘMxTx耊m6^n@DΕ%ܻGuT޾zMʆLN nAP# ~Z`}mASʘli%'$4P$}iɈ,'AvBdCXem4%ޓm({i<m&063.+&,K;={랭}寈z}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|,RI$I$w%?kg9/}$9Ln'ROܗ <$2Q?~a>*g?H"?(ߦʄT$I$* $I$I$I$I$I$I$idB*I$I$I$I$I$I$*J$I$I$I$I_?lۿmb7V%~~ۯ~eֻ-~|tϴ?:>q#i%RI$I$I$I$IrpqݒAr'%RgO6eB*PȄT$I$I$I$I#hb]v?4? Sٶ1KHe-Q طou2!|\%!|[yxQ}OO݁>޼{~j%Iwۻ\;'no_=1˒$E޾UxҨ]7ۻޞXewUv%.܂ް2J >rK%cmgÿs{oۿ"?m:./G_$/YC}gmo@=8s {rOHgG(e:NKeBceqamYԵ Cj >l߽jS=}pOF3.&ZZ|63÷\/tehHB^Ѡ.!`FA_'_kIY﷏;` =ϤLF9? :`ۥ.LlVa}2Xۖ|fW{tf}Qz/ߙ=d2>*sŚ\ `롮I]%?IrtonOYߋ?exohG2}G>(3Ν<~-e80{O_\?&S&}*6}j<#Vr<[̵6ݻU_84!^+WܤQR q)>:gq=W@9{]|<9j>j]-{SľfM߾M8G yzLA,!M<U}.UdFhB Cob{EBjvÃP=:>cPJA{V=N?tC򄾫tJ%>S!UmvröI)ͦB.}cEڰݝ6hwf>i?38 ʳx/ϕr'Rv|2lu|8oCaO-U9r^0b4m @BIL`z>N\E6i4`mC`Vyאc)lm#]G3_ˎJ7f1zz{S'l_ ̓ [KP2XڃuU癬u-СmJA#u_r1emхbL)W:# 8)'`}L AtY6*:~U3Lw$2&N6Rfs˾Ԧӱ>#!5g {Wu>xE75[P"/ϑu?~#9E Cu!>?q,|`)Mq _{&b<~QD]#`z| HxBs'k쳐>3lz2y+g\|ά1sz+^LSF|}oe. ?+o}ն13PZ7%1>o]vcW̵Ti6pGlry&א?mKLJg_|mCewƾmY'ړw%3?+l6}Ħf$F"/6ے:4.]^:5".KHP#uZJxP]ĀhMH4Db]ly$!V{ӲcDƔ3deRd4r(m\-}Rj xut$A&'vY\X+yzofwkAmɱ6Ѵm<~IgbFe_+?$s &h _=w:m-eѵMl[ օ}Qht:uh"|g$TƟt3aBJ~&X:Yr0!> );It$xZ3- M.IVSz7O1?vy/|^_Җ>3s;/wyM>y΃Sʞaџyz;G>N۞ Cو5sx64{2{>M˺l&Ϻ;_gK8ry@z3#I򮿴9g<_B]#r>u{k;헨o݇rOGS7n@ m60Q!['ۦoj+ip|^'_3|PJ_9eL9x MWd$څ%$|j(QLEc'?f ]`F\&-a˼o"AYXa+K}JYcaϰ/#YuߗLH'ܧG]vKYi7lcg$7 L*uhB|+Dz4rɮSRX8 ;`%& 6?)8`CO=a i|,C/?:c 4d>?k='uQMT@0Xp%YI &gx)W[xn0O{d1r- fm:([a&Mfk*pa@|GTl mȓAQ=AuE=@ .VrO2D> +x \Hې`YIMxAVl9 PB*xZsC\&ܱn;?]ﺟe~uwMF.YL^XDh;|x{_Q—1?P[)+z0Y9fڦFn C{ǜglx@U/zZ߸aV`!}Yϕ+e!,M w9וwL[✅% 7PW O~+͓6WzYӳqM~B;θpU\m| uM:|gʣ?oT9Ϫ-gjm Иk[}>{V>h \m`f` .tV6>i}iVfїp`]MȘ IRPw1E "6m}{ƕM) ՟4Rs rL0l'c<%wg k5DqiD׾q1|sK6@v*L49GYC\09fZ[лKWD֙Kq ͓݀YY Xu9i}Q {H0 Lg\W?\yy}N5_$g~t Ϭ♴hy}.]xٛ]hoNcE@rvA]uw ·gf>?se:.\7 =ƨiO5^?c64 |'Ǩ1QLɎnb;\^k~-붋Y^<㒽ԫAvn%/gIY ybQ7KC ̓ -;q4O=ԞqY/ٮ'hBy}{Qu6Z(oyk!~v|$]FRy^^B> )xo|j@P*&X"|껓R%ZN[x`4?ԙ4qwP D0A @_=I 1.pm!`^{l; 52{_֭_ +_0ǜp ~oo”KD!`;߬Yj{ )Fhp 8qSt8 *qy6v%%LwAGwI3|?>hr>GKK]߅zTf~̕/بa\YSƆһSzB+6i}QVOu=LX}gDW84־^ML'~xbG~sЄƥ*_WKtyq?*d0VRb%A"B;f8YtH6@g=!5jx69ovid}{XY۠Æ yWn>!YO@aH8U0II%Y^>vD"em„ٓP%PBJ0| UP9TK[sk>n`Cu`F빃ul g*܈DZ6~}G_kgfe>A/ ,Cym;Q.epXڢu/hes. Pl1/o~&ɨ9<|1OF0:'!0=>Vo;vS 65XQli"e=F^n%Lu"o6pYsw\_UKm qb3n F[q;jE s<{CߢWl_IU(cθaiyv^ύʿ69(=Իq.gRr'z H۪ e3Ybe1^o=9OU1~&5cP6m^Gdsfzzc_tjevu TeD~JȄ^tB][ L+H@{Vf)I֞e]؟ɚlfաKYL4qzL{ln͉SɌ4Sp oIY i<| +,#b *-߸m-( $vpA? |Ё:!r:*U+7]\ʷz 0PH׋Z/ f7uc6Qa-Slt9'p, l(1tuByт4޾zMʆLN 0| a0aB YAPjbRsY-T?2䄄&jX!-ђdT#hNQp\7`r+0lmFr{mSe/f0f겢1gs?_-.bB랭IBz}E~ɗ{ziv/Lֲc2l8[_oJo^3UjYƺl|t 3@&O$K)GI `cCljՅ':M;}|:uDܗhs5$MH}LH] <<$yK?4I[^K߭. UĢ\RN5z]cRI.zAxB.qYX/mɕv_Bw';Mo&#7{줄i'" )̓G-!.k!Rt,̓*.Ű58$}&RR-\ $OhdQlKIrȄT3V5 #gڒ"ROl0F"c?.G'xBD%V%KUe"vV%2xRB# n ewsJB2ҮH4I-ϓϢSI}{ZkhQ*u$R៵{>Iv~& gcpM(YSp=oĵ_OZjMSy LfB*h~q= 2JHYVB~ŒNIHՃ(=\x(vkW'~nAS 0X|rrςn3)/ardOhAt8K]b٬&d-̮Z&[*_3i{Ȑe|U)5#@0C]/.ZIx8 7'Eg{7w>#Ev^B_n2n=ѧ{̯xy}.u?>eed>5|+9gY-Zğ>3e&&.uuO/r+?}\"b}eME:_-!>.1wws[|s y1|6R ɧ&J)rr% ΥKs}V&$I>s )9M@"HЄS}$g$R LLPu^%IFyn$&0v{/7Y$f- g*<4{/ۣ4H`BA{V@:[M|@FpPmʕRmvröI)zͦBx .}cE7[wgc ؟M}`V8i]^{y>ڰݝ6hwf}('gC}Ɵ'y 7El;oΚCwLH%I$Iren0!=D dB&XBFs4!$Ez?%9%!@Up.W[~Xc,a_WbVѹן9ş{3痠_b:o?9~63|նQx.uh RsS@8f>6оqlr6>υfWX }K/ަ}}q{Ff06m$I$I.-% <<TI"%yMBNHyN*?$I r T.vZ@ʁ2Y=SA:h5j DV olC Ըa @7l/Y= nFeYU5ۯAE%ǾڠtӶJ_t?JGe f1n y#} Z:?0P1*NE|- :m-eѵMl[ օ$*RiGMon^r&LHo؄k['K&TgT\ !e'.RQ_DZOk|Ƴ>7!')~X}O9Sn毚>L|CsR{?7gV5C7^_Җ>3s;/wyM>y΃Sʞaџyz;G>N۞ Cو5sx64{2{>M˺l&Ϻ;gusyf=}C_z${vy_ڂKJ=K8}aOx~v%gI{'Evb2YLiT>/-g͡+W|9§h15i;w/5]W__f?oۯ+|, -R6} l rncyouP(6M"z~B *z֚ |3/,&OE,=xu-W eJrO2D> +x \Hې`YIMxAVl9 PB*xZsC\&ܱn;+^:I%u!^1!~wR6/c~RV`"9F?s̴M2hY9:sݿj]_>cq(dB8+WfCY:Rٙgs+/Xeч6x d@]whڦ9UdzқI>6.&w}YpYs`9>F/LU:x{t&0tV{\}kTBW{$vVۋ([>a"mi6>GHԧW P\&Iar; O?~H@%pd2?-yD!U~fmw줄 oHI]gqZkp.4%piA^ʐ>wWd2|k=>}b0h?߷| oԷzSag9/1ӲaоS?u*8ߒoAqd7oֹvֶ>VkzZ߸8vDҾT]Ht_E}͑s%9 gzrE}LmYyg)TaYb iC]hoNcE@rvA] l /b mD6T&:m\ΡK5FkU`ă~-xr{Rn}>5n"!52m([Rڙ-m%'Y^=gIOmzb 6!~Ŗ%!gg7IMHa'{yIt$ MB,XbNJHOok;koṃxC}xK q0 4"q@ALƁORrL FL>V_`ֆԘßQQ8Vwleؤf>G6X]`O=Ʌ@l=/t53}i9_'.+0%y 5Q&u7kڞ%dBʻ>Z#DB*NNoce-@^';݁oI ?S<Շ[k {,_a寨:z#}g×g +#>]u_Q` w%ާ[2KWlfՁ9>&zvx\OZ?/ϩ9ӓg:Cg3vN|lMKc ]*DDK)V _'>^ۑE.ǻYL@ַ7e'dr$ %h.MфT]CW?$Cg6m #ɥYBjڧTލ4!՞$ɇ'^> (L<~ &0$ގH콭SR={DJH;ȇ,w,K[sk؈n4>#j]|3@zl\-(@- TYmFs#ڠV3kke>j| \%u 62dm@~C*8 ץ J[ ;jes. Plc/o~&ɨ9<|1OF0:'!0=>Vo;vS 65XQŽp~[z:|_ds~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY7Y\빑7缼>E6z8S}3k9]=mU2.-e콃9~گO9>kؖ7 f12NUx:c-zK%D%ױ=B'oQ3g2H{FK<\R4%2B<]VkO M&% R1Tt>mM,nؖKڇO?GGRI$GԷL4bOύo1Do\ Z{ZvVR!$k yx сY{`A@ F!qVR-qxЪBH `vAۼO\7fS9f79:}c=seAn`F b<.@Q#^!8k@ul5X_[z2&[¶e*m MɹR'yfkERΣ+Z-δ!mcf_3~E rJs9&dE;|'0e|f>3?W]/=0R}"-[[xL-۔>L(y>~1}VZNZ0$ %t ѱH NVb.ICNJI y2/3 -(߳\_;v %eSR([Mh&IxdBRBo]&ɽF[&nb\J;F&䞱gLH!Sɩ6˄T!rGoMbIڷ>ymɶ1d?$>\"!fgB*IӹZB*v!V~'IvA0!}γI!%kJQ?=0{$s|mLH%JHDL ZmsIpgk.r|}3Rf,+qtnBj*u>ҺI$WIH%I$I$I$I$Ir].Za%!5#Hɍ*F7J>%(I}vlrTg@ovNSI|xdB*I$I$I$I$I! &?G^A=M^圤ڥhXb>$sDڳ%OC$&J$I$I$I$I{ M=QђV`d^Fl x TF҄'$+IiPuX`59sRW+'+I$"RI$I$I$I$Ir ^BJRPKԉeגSV©$UJ"d|nc94Q?ic85!%NMH1jvRlI$LH%I$I$I$I$=Z 4I F4$gkOr,H/Lhg$ABTa'i<,[7)X9O"KHUNh9I$+dB*I$I$I$I$I KI#EK8Qv!*my֦./H5=My'}V1DvV9 )븝0I$WIH}V^>dygw/7nZeIx.NɫWO̲$In_޼4D|aﯷ'V]f{a]/7̱߂*~I }Y*3OQo:%_w~iPyYs)39ž|x(JH-1OPd+ ^BNx]'":OHqZBN|BʲA$Ɍ%~~@<}O%x~O?~h$juj9d'rXIu# z9Drl'׃A} |݂'}g_>r 4h?2`Hqn3ŲYM`m[]MҙyGUv|g!X_kF6s`^'tb$q%IKy=g~/=wmxW|+Rd:wptc>udc~c|siL|Y/,d&{䋏XyYL̵`oa`a84!^+WܤQR q)gK\Gs M&`W9?Ϯ#yK|{D&FRh5%(3KJ5^J"#TL,49:i'%jNIH͓LHU{5@.$IOHirAo@b&#I?*I2s#t4!~z"!5;mv` V}1 ϽP+@Ҏt0K->SxH!s 伇m2eSl M˅8m }v?aXQ}z1#E ]zAo${:)؂$ǤI$$mHJdLږ`b5Iuk"&z)|RNa٫zN3Mnf teEu%\˄hvw/wH݇KSXPgy fPs⼽dB6idhI!;_0%J&[ D귁@V㯪aB;M dxܲ/tY M%Ľ53޾zQ} eԺƳ sd鏸߈|NPݤ?k~Dܟy> ߧ>J}~^k!(_T'_~}he96F8e,d =sY<:Wrά1sz+^LQǾ2a۟ٷjPZ=%1>o,a25Gg|C+m)4I%IarK )HO<::?c|n x^PRoS(g5tRB\8A AX ա>d)jvІ  RVZzoo<Y%.2PbJS ,F6+g.|K|>׮+ٰ|~3ߙ`*3^o S7>]^c~}&m *חLˠ]^~^q8zgjm/~g<с:Ӷ'vAP6cp;ޣިmӲ.ɳκrΕ=* gF8 }陓m]i rn/+쫞/]#r^nD}>t}:ži af2{mP}켊=9=Wlhl-.gsLoJ?-qrQx SڔrMZodM+ j'Dl$%nҷvO ].!5d!MB]*Y$ύ&z&J"J$é'F딄;NB:X ~͏oA XSz7x@:a0A !bQmI]s,*TY* hA ,8v۬$fxE3<Qtԧ@9ؖmXFk[-ʰMm&{f~~0Hk`*Eɠߞɠz ຢAAy+m9'F\@OH.I"k,k1O p1$"jB 2|F$ ijsǺIm\Y\?L5I|d3y1cC?c𥖯)Em? _tCodEr~"ieJ5@/?Ssu}ն>V|i}QZ郅g!}G6uݎI+VBЎ7"θpU\mA}q]ԡÙGpSNW84־>e] g~87?O'e|+'Z(^2RCE' ٤{l$%I$>|O q LTAR3O!%u=w.zPIe|[&!{`MgРv/z C;&;dPhv}g6dΟ*kUP<2f`cwh,+@cvmQ;[f>#@A*p}`"a\^{,@wl?XeTгۧuYG_­ 횑| 3A3I<.&JHz[qeSJF%\k%(6$n4B6c#I4̯1gc]yg6ݗQi ϧ k5DqiD׾q1|sK6@v*L49GYC\09fZ[лKWD֙Kq ͓݀YY Xu9i}Q {H0 DrΕxTY3cxf1IAWy}.i܅:]+hŰ1Vzf>?se:.\`U=rs׏ǃۓ|'Ǩ1QLɎnb;\I$Iq )LWC+&9 s?? ]Ge\ pzϕ?elx(+>تY/b3́7ee4P׳˄ųsyy}N5șȮug?yzvLM1tgaDK0+gZ?sus̹lB8Tج8W}BΈЯN}%ph}>ßoOĎ/=I$IOH|Pyy,NLRaRIϽB{[$0z$T)g;ҁG9 {_;Hn TҖڶ/?#m]|#zl@-(@- TYm+s#ڠV3kke>j| \%u 6F"23kGẔai{m)>&CƼdRe/3V́dTI,&Y;̓Q'IH-LtDi:%Q4VJ@zmxo(WY_LKrB֭DqNq3's~9Y~{;/{ToץȶF_1nyJۢ9彡oc|Z+s@/$*1cgܴܰY<;+y=7R圗TPr^gꢏ{&}-g~P n<{6@_1^oo8i eγJ(a#uZَlmo1ƾy&mT7˘L"Y܋x3_Y@_As@җ|lxLS?M.c*#I$Itn>!-)k$؟s#ae[L4DeW֞]ſun8<A;t:PX4D`R\P%jfKV0QX*}zA8yn̦r<*,0e*}.DΕ2sN(0/Z0ЂWIِi$ f#LH\7+j[ WLJr7+cE;^jQCMx =%#Zrbjm 5؟# YUYƂem4%ޓm({i<m&0v}ݻb?|l1aC̄^lOR}У?s澎/CH /{L{e-aIxzVz lUڮ6e+a6m TK_ sz\)/gkER|!CBL-gZo8W~ܷqm& v1ȣ/G7-mNm8`g_V;E1B9&i$ O6{n0F]x(^ʼn9OlZ_;"_i׉7te\lɼ|;e dQ:O'I$Iلԇԥãyq$ xɔCI:r]nuvOz'r{bN$w$I$Ik KsK  R'oe2(6˥$cdB*I+ɚ~τ3m $I$IkqTgC>O$$`B٘rgC&J֔2\q?=;gZ+TJ$I$WIH%I$I$I$I$I$I$I#RI$I$I$I$I$I$UɄT$I$I$I$I$I$IrU2!$I$I$I$I$I$I\LH%I$I$I$I$I$I$W%RI$I$I$I$I$I$UɄT5޾{]7ۻޞXA>v{T'ɝﶷeIrsk2߼怞ow<ؾ{矈_m??}`i>~v y}Ъmw>SõEO5e7UfsN@ sxo{ן>^_۟w\?w1gw˧w]}|B7'egepWeO#wiem z/ivc{&R7+DLй!BL; mv/ ]GV}5/ޮ)Qo9!˽ ˙~nrL/m]x]d~ϗzuɽK..yN]Rܻg )HB&h"$)zr&Yz@Mhx|@RG/ɐGDz^u)GA]p$PfӇ9( g irxyzҾOXy}%qQ HHyn%$ ̚RJx%s#Rw\'RYLHK3RBS[1-\.ޮssMٍ2~:%rCgǝdR:ݻ޷Kv1nLH {72^6;weB\=7{וdHh9 {=!ۑI ԷUBAِLz[Ji7R׬6R8oggok\e遑{)~~L^M#%/_.<B/~2mQyDQ g womSg3s0Cm欑+]q9ylG+b5ǣ%Zhd;Ç}>֎Wog-@ϣ2&|e<.sγ|&y8Եsl{kG=ox6;msyjow]?\z_qJ"{=~sL>MYP5w1&{wrnz=<+y?31Lcp;A[{/䂞r~{QqlF׍&d?CI`ؑ߰ )߈ɚ m313>CSޛ뺒R(cu<'d[3h}>n^4esTlrмl?KTs<q=.>CkKzZ3ԁei$m V*?~Y|9&R6 _6/uq~@Ƕ՗'s?ar:#1dgٖ3<s&WojΎ :]?߬EW[uhkŖ%މk&| gleT w!egV{f#Xl|6/m3(xI6d8 4АrH^9^X]]f0ϤqG+(>g-.c?Vǥ$&b., }|6<}ǫs|Dj]V!:@rڡ+gz9jkA%O3]3aALi>ǂjDL ُx-V[ź,E?$[/ՖwDx_ܳȻjBJ&zR˼MMXDH$K"TIDNJoGzΑTDMV!J*x}Ȅ9Zda};4bϺ44/e '/.q+/Q{ ߣ=RoMt P!c\wfRxo;Y )D5 %TBZ1,cD~soqnU ElyV~.O˙t6E`s֥ 6gŚq u v)d38by {y([CDd371G~gra$ٻVF;5Wl\06m*J&M|n}e<$MI,:-rG7C054[^: )DY!%$ 6{&g-GY\ʽ{7X]LE ƥuy; )zq -Y R6}j:v_NRw}roH~qK`,>?k_̳EEXB.z8yǚuh5'jkڄ2keD}TayB}_anW!kkhj>͜9 mfߙd3r%hXN+g]8]Ņ#~y۴ zK+{HI1ksa<-A֑1c؂w<;y|{~Ը9i٫;+1+K Fg]ӳyv3t94onh{:|l1O3hEl:2clqH `'q!go.. _EmWׅ}<2&Mgb36d-,]Z?#xv&}mT+NdZPgٶ7mɸgۻ,( drcl@ص}֔v崏a:z*ektΨHg1,7c3&zk˘+GE9'h˘Ktlտ5Cd21|3kͅ9W0,ak1yu,PlAzw>XFL$pFGb\sucLtkmYʦю]ќgyBlZ~Nu,y?Ol&tBY~h3. 59̒C3 4N~%Xlr]LQ=PGuz@CQY.*j, lm(ϋrƣm"یc(X63LgΙ6Pc e`N4y6v@]A/){2"} qGPsnCVm[h租u{RwkS뿲6=5֍xrܐyVڽe΅_ORG뀶hvY EnI= 6.,DIfXPRaXotٮM,֧LЈDx62񮩗?>f7%*B7%B8TR|3\^Q.]D}t+KYB]$EH?÷%r6 $9V^/|'.yVeY/&}&W.UӋs@*?!oGץ_Wsv晍/<3i3m.< d1F6eLXːkì ȡ ^>9E6?(TݍulHMg`r{a3a^m[mEϦƕ+`^Udyg\/Ǔmo^Ze&O^޽{Wy`^ x޾z"o怜7I^H_oO:m8_? {/ |xv{Tk&٫عK=:f9yFZ!z>닀bo_Ε3*lϝ{$xny\km{~}k$WKHx`WgYKh2?v1Ir7XnOz(@;R=+y~xe}v/ CpC158|9p0jS^V6/܃/f$=[/__x1-I/2KvGB34ShHseW]oU` dMǪ1u0!G;y8c{?ϝs|y^zs/b={3͏Mιkή%%&w  $vh©?TPDгg b &h sC$<7O'&fVJbƛP;[O8EՋ]>h@9qK1@/l>Oq/IpA} hѾi3MpiI(Įqp3NB5z?6fGG(+ (c㜼;W'AcfACq^B%-a;w_ ˻'2  )ro1"ً1̂YEd ʻMm΋߼ XRw[ond }uiy %,*RxLmΉVsF"%oj"F^M&Ȥ 쏿z۪}ea )k`Fx,У SϵCI0Ã=k.`=}}Y}/IyI{g[*oUe;m~}[iߖ?/xc{]žJǂ:g:& ހº삘 ;|< 8f^3aojy"ǒzQ䲍1muͺӲCY䄹@m>0cMpd;˶QcLƔI w扴/ra9bikSB)D~6EM%X׹ 2MՆZ#E5&9|sq@~w(o+Ѷn#2@6)}ﺂf+^ҞlS'`5W+h΁c i.:~l_ q1b]m9 _ym:O :PXgi/9\f6ۑ}6;\ӈxDl2G s~#6IRj xut$A&'vYr c쀟Wl@sff $I8?f'oBG&QC5 hN 1"ϐm&'\2!ec2&qt[K$2~<?b'T'O.lِϱe_ʷ.OfaDvfOEU# mm\0ڗma=7ސwD[AbiC<Ss9o&6A弛)t;m7ñ&6k8[XcÚܰ!ͯx{wL1^n{lK7PMH529ꏣ?YbΚvD6i/rmmW]2lA e~^t}|%mO-1Ǡ XʹP?q$LK0 9OlCH[P7Um9',4CF<[xck6 ׃'zp #gr.X!eU$FR=ATIMHo%XVFA#uJB '!UuG )dm ^/G;6oS)g&7,FޙlxW Hh &ׁ2,QobPDV9G6}kI|F`ތ yW5b^92p끎y1tQ6rUZ6}ߜTu9J2@IDdLDQRUEo0ILw }+QB*$MEr3"HO_6d ?_0!/=k ^wϐ>RWhFtRGï>`g׵W,/<{&mGdxƼLmuYyp@6z\Ƴ,8̵9<'QN<b: "5!VikNBŹjԹʞ8^tROʶGk9n tsYyuz[e"_/Ɲglxh;0ގx\'^yN)SSz ccj>b-IR,?7Vj/ r &^\3=|CAlPlqIp; O?~H@%pd2?-yD!U~fmw줄 oHI]g]]xN;vKtc9t#[I7UBo%Щ dvZ0Ԃ&Mֳ.eL )InR|!,ƘC.r>y%X0jXYwzS{C 9 YP]#Z6]/ڡ\'uϪAb&o:$q Zg$]('9%)JRc`%fz{ ^׉s+%(j8hLcgj1׀~j q9`|{>hUXѰ׻ZW-sk,?62v / ]>.k#ߌ@ްdR]zI_X2ςỳ.{רukbGo}:UZ;xG9lOF4ϟ"zb6jy"7uxlTw9Re?ϣd$,_ynywkegPe'uۺ~`7ٶ(zrXTz2] _aj6kdZ}\ӓy;ߨ0s%׭] m\콂5/%36g|93]a9uRr$ٞ#?o@HN}Bߍp6@Y]%YVGc` Eۙ"{VDz`{ϴz{DNзpCͩuhL }Pۛg+o6^.F]*˙B&cM+?>Ʋm<[֙aHʼnu"IPOÄk~mǃV_fk[0WX|6kzPem]߷ÊZxm{Wn/WDˮc:_ϼ/~G#ǠɛK˙mvD>ڀz(yT%|}Ld<:GVB`ifzvCR(xLucԫɃ6lIr׸g="T$&dy uJB ZgOBrhÑA[p~ӝ83Wb;6LBDOgrYh^0voڑ]0!Eee[fP,16Sm3u$Nrx:y!yc~#Z!CK#>dY5Þ ׮^2 622M%`7jz}+9vzbEﳓg$Voo71ن,,ov+sN75LA79xd:>^̇]ዦR~Wg-L =˞cI;6ujn A^g̳<벰-҇u٥-[Q,+[֜vLhw9D[R'ԛ}5!OEj?c6([Yc=2>JFmrIu5A}ZdX6Բ_{s*o6>)2kN0NV/d:ɲgOZ~^ԭо2>3O5u|yS _lDZ˵aCr|m6qRPwIQD=ǹ.+6z6|9 $qLK(֡l3lѺ-yf\'-I.'%ec|xn$|&^|jӲ˷7xGo AN;,|#]z`bCM,{:䟃O%F; !)[NB)IV.Qfkܱ=T6+)mL *gtU[. D:2SAoജfT{<ʻQ.zt}Cն1qBnUSI|QAK@dvO<_F7@Ak0DodٞM,ާRYrFvwR읕RY7m4 }|o>K@װuo/R~X{J$_ nމonYx]_cHّg^U.mlCW1/9j3_l' ِvкpy/6Gw{c=/`է:eo5/tF9LX*S!HBf1ߏֽtBtZFp1 od9LzkFҽiW7l8'㱣=sσ"n׸__ KŞg`r's!7'6݌96Eٮ枞K0Tf"uی΅7/xTy}kٞSyhfbO iK~Xhe~bs͙ڦm%5لԇ:ō}Y&W\(I>sv0~Hͣ^ &qI)I 9;k'_}w˝>yϴI$ R&R f Hc/$Jg7L&/kU;Yq !dsKdB8J$\-!Y?>>_ jH0/W漅o`EryM RO)|q~r)w]&n+̄T$ * $I$I$I$I$I$I$idB*I$I$I$I$I$I$*J$I$I$I$I$I$IJ&$I$I$I$I$I$I $I$I$I$I$I$IdB*I$I$I$I$I$I$*WIH}V^>dyg\/Ǔmo^Ze&O^޽{Wy`^ x޾z"o怜7I^H_oO:m8_? {/ |xv{Tk&٫عK=:f9yFZ!z>닀bo_Ε3*lϝ{$xny\km{~}k$WKHx`WgYKh2?v1Ir7XnOz(@;R=+y~xe}v/ CpC158|9p0jS^V6/܃/f$=[/__x1-I/2KvGB34ShHseW]oU` dMǪ1u0!G;y8?͏we~Kxٻo~njv] `vv=Gl SyOKkϩv=-6z-+X2|KQ׈! G8loDd]24Hp-p5!#{ѵڗk8Mç՜;=>%%&w  $vh©?TPDгg b &h sC$<7O'&fVJbƛP;[O8EՋ]>h@9qK1@J6?xS Dǀ_.h> ?iL7 $8Ylcvt~ވ2[zY2=ɛcfACq^B%-a;w_ ˻\hq{YCIr_\|'tjbד Db w{фOF86T! p3sPeoؓ}4e01 !A#d1!u7z;mZA!e\Dn-"clVm]os^l^|Nz[u#H@#սKk3g/IfYV2h?gosND*ߟK01-/yS17rPj2hG&-X7vGeU۴VC, dHNH^ 62ci<g؟~J9X@fx߾z]}¾n,@>⤼ZL*Tòsx?6tԭo˟Ez1JĮb_c3ocsa]vA̅>3c/癰7ɂ_({9ȅ1hv+6 93O$DIT7Fbaߨ m'dbg6RRr|.Ҳ1DD8g-%Rqi,#ֺ+ |B;}RBy>Jl )[ȍ |+}N|?{n?@d&zOdw1lT4Y?= FMooc}gs yG5!66dσ15saxm"оўa[λI["{[Lvh;kmceۨ1~Q; 2߉xIH ܃KZYBj&@Q֟M1gj;f"XC6ꆲ{DP/g'~dcPd,\8y&%̅xߜL'R6AϡU (LۛY*жEFL!`o98cfzpV.xLr~5jC]\$;hB'j2ԡ )$ZJ?z"h$NIHa9$x:;!,s Ahm*&h!b; /kP"$v h&F@3?o6Y ˡA h*Ȇvs"7 OبZ̛A=:/Y+6WÑCn{=q:/;.fCS{C˦sRr.66GQI(Ƀ4, H2"ZBT!v &wo%JHEdUHΰ2|F$BL4 &e}? 59|fewgHT|+#:lDf Z+n[=ж#x2~_cX}Ѷ:x,u<8 GKP ZOk. XfY\[KZ(4^&F]O elce(=y"7d 'o?+GBSb3;!Ubyl{6ۇbq'u;&kS퇲-;mZ{1\ǟǻEm|^]7|l^uˀqYۯs87 c:3ɟrSETXfKqՏz=+5Ǘm?Kt.x[Y Gf6(6e8g$ԧo?|$n 82ɟ<ߐ*?6PvRB 7..wϐCUc! 9fټs,s,Իf ,k-sPYJ:dg 1@T7Jad3].F%1h3=xDιΕRXF @F[u|~Ń3͘kgc},=8j/!U.H(D0Y: )W1p~Gcm ڤsY{j=99mىaݶM-r6m(A+LWڲǾ4مVdN7j;̆\hcuk}6CWy7{`͋kA@ɣ#;cLGlxXuN$%\1׿Il3=iGaxL[ ?Mf'FRy^^B> )xo|j@P*&X"|껓R%ZN[xbop8L'-dMag6c;$)Td!aR&g4el4ßIHEqXmށ@=8yQS~Gz[#QW}1s縖M׋53%׷r¨u_:Iœ,%iߚ[dHH6qjʶGNhϰ%R 8kX[H)}ީA軑(dS@ԑ=ڊh~,|lh;VdXqo=^[߷xϘV5 35>~hz9m6PN$獧ba;79һ"Q>[?O^je۬PS0p% 2i#(ӤX~׷c'(V>;qFBjծؾ6[{>vmrf[h2^nQc_3k}; Ⱥu*}ޙMm#|/^[(;: 6{==={ܓvm?vA:Y 00gyea[+6K?,[X;A W9#>Qj? r.N7)<kB4m~ƲmP-^&l{2 e|:ږ%Yߦ "_(G ;9Vb2/q?fcc X[r.Ps -mB3,[xuuL@:|,/=]q_3_Yןk>P/ |\J==$f'%ukz~Esb3lCΗ9kcefj[Gն wnIr9n>!-)k$؟s#ae[L4DeW֞]ſuL;zSvja{}FW(jbCֱ'D*!U>OI:pI&|&ZLFmq2P۬0Ot<156ѡWn&HˤNr&SdꋔV?~lGe^/ǁW2zw"@u}uRvdc晷WUyli3aoKs}& dm6m} '٬ȥp]6sK1oH?|(e_lGCS6{}ZH[xDL-S#lt6hsmV/I&>'>p.n22xmt@I|9ln{*0yݻʛ+/? eMQ~x_0I.Gz{bԉhyaxoxû맢^3gO_oo^]\73 xs5uX_\{rHVa{tS%Y|.pZk+˜uoM_$ɅZBw/~>kϟm=ZB-QIVs>߾Mv |-׋%DY9wCn< {K ic0Y?mo|7~Q_ɻ;&_/ۖﳶ }KXѮAs 6U;R}oy4'*yZǣoionOz~Yc>~WdBDDz+Rxog MnB'>VַsG̅ <ڙ>fa|#f{_Yǀ;璳==>^z.gs5]y&9w8Zs*rO`/+=p _ec5byc>[/` 6/\ !u~8g͟tH?qtmsZ?Nl+}hB6$Rs܁D7ء HRA="A϶I2%'궿JH<Y=(Mo^B]l冋<W/voؓ/@t)ү<Oq[$RP7~Y.G^^jI(Įqp3NB5z?6fGG(+ (c㜼;W'ARۺƁD0IV5Oq`#c0cMpd;˶QcLƔI w扴/8D{]FJikտ'yB-n*!UƊظe>wm6)y4 3&(CQ}C]|5Guc7}I{0{U_ȗd:Ci"_iWݨ7ZG ٖWCɜg[mZw6KImОkD.^/Ͻ4;ꜙق'Id$O#߰ od ~ڶC21sH3d )9>LHi٘ Ifk3ݖԄd ~`]ea'C6QH`lHز~Cg~\/[ssR'0{"caɪy|O66hz{x d.c˶?v氞o;͠ !{~Ϝ 7kxrMr}g:DXl5-W,F1aMnXǐ WLo_񎉓X ):zMH529ꏣ?YbΚvD6i/rmmW]2lA e~^t}|%mO-1Ǡ XʹP?q$LK0 9OlCH[P7Um9',4CF<+#5WX=k39~Ր E[t p]p ɤR&ȷh},+I#pꉠ:%!V섔v 6}w3|mȊeL6~}+I^ "#R7^dDu==[ .rD{ԣQ9jżbsze9=d&cl6t1媾7l9,%BacsdhI~,d_[;mD~TB¿O˹{A& |}W 郊/|oD-u4:{`~{]yҍrcgvOڏk>VZϞ.dSw ^Aiͅlb-IX9Ll3Yʕ6#xsqc6u κ9R6A)ǩ&I$>|O q LTAR3O!%u=w)ΆvΊ;ak/,3&mе> Dvh|VXР:un4(36 f,"KssB|RcgHk塪1K憜l^9̹V]d^Cɢ//\<.-9Mnq1+FvTd[Ѧ:M;ͱmQ1q@Pm%ڼ|#";s̯5W0~h͑cDn,k-sPYJ>bϪAa_:äp g](8)JQc`fz{Fs (j߽hp_fo}DJ~خ3報WI~` {NdžyRsla>#䬱ۘS)y4<8ݻ/k܌7L7,TW6KW,;3b6ʼl_ג ,6nc:UyD!eosS‚ًPT2"I8%h+Abr/Nd%J,{cE(_a`!~ăx99w)71CEyT n: OG"p7捘#^!<H]MKkuR,f`k75Þ ׮^1 62-`/jz8v/f +z+sBAjծ85[>umve[hsJF`hqi_3} coqʜewC:V䢩mDyޕEٛ eG `m}zeog5 g= !!u qĨwu b.M5O|MvY]g,es>6ܓX/(ږ,@E e E} >/qg﹮acw|g7S}ooXfs b -sBO-<{ҾM&nmXEg+nk+8\). \J=q=$f5:"֏纬L)rqXT[G\b,D,lL{oK~QvKq_H[1Aш|r \7 VF~D ATv[~q5hiZWQ\/HydчHPvR o]^MSo&=k?^1 P° BR~p/$nXjUhmMoЁ^n.@֙KCHa%6PΞn"j] uy۫/{#ʿ ga,_3p \wuƐ=g3שׁDqo3a}-o%\6us2ٛ ia{XnxC s<2}=Z澎v|mť⺔H$:6Vz~2~ LsрL6{\E lQOm1r 50ٛl@_t0wo=.KW]pn/GcLa. =Zs׊_lXkIbǙ3I,f{Qt*y*t,Ea5fE.Շ2/[^e{Nm!ДbO iK~(δoYasEY$$-H=v9l9H820?sutC$>q__58bx/i/iXGJ[Q#Y;Y걾G wxýM$Y:7Y VB6Cތ?^%)IK$82YJ_ָU;Y5!u,H' RI$Um#j[$Kی[o &-/Pjx(/RE^r 2ήLPbyJӴ/o˙_n.9,F#ŴT$ "$I$I$I$I$I$I$idA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭ|8χVŷo?s{^6ᮏI_2n$^@^}"pZxyw{l?=y)48{F*vߥ3ٜGZ!޿NY}/c]Gg;yg1fy]xy}x<鯭>Gw?VexG}f;+%&g=ҽk\wי==N{wpXA꧗O'o^y#z]qx2}/>}X$j}j;طnŊN*HфPnZm&1FK*FxfqX9b|Wy{C;;`!8Ƌ0b[7v(z!/@A NA j/x[޻|^?pgqgIDAT>kOOhWO/z'Yc;|'2KrٸN}""O])v}~WV sMnNUvmË#큯)m{|19|܇vNyé/Z.>-\l_z.õs^{StFy=f;B_]*>r۽+6uAęGlSPc|j XL6;;M%iYqZ|T~?(dA`xxxb-Q^gI~uMwϞw{(Hiq A-8H<,}~XA)Zxnx.Udt6 tdAjGو5IA>MW\px$)L6f,7RDWyW_?>,*cnypip)Įq¹P"~,a;:wo[o/ˇ#1&y^(@_c`XыП.WȨcwxRE Jo{lgP2V~# ;CZh[*\PoKǠc(VuxͿ AzX)g+urй",^Zr_ wer.b'˽q"BóًJY 6`eO w̟Ix.G63|~r/bͱ9:e.C5 R{I~uMwϾ|słw]tjaZDbo9Z_>'-( a75Y@)~#fd7ⓤ0٘EYoB:;`A4^>yz;?nE) gÈNۿ=en]1'PzqQҲyyLGm_Z~ >C_̲"eutfLD7UK +8YxS N1eyn/~L^Y6mU\;hG(K|Ρ}).y볊횖5aWV|8SA oԜ[{ Qƿ(ylkSm ql>d7Ώ>Y8^7Ky[w<~\Uz&v* t'c>nc. ba>38΄MlDzQ<_rنO[@{f{{㴭@ep(?K t07&5[v˶Q>&>sH^H9)/ǯK8,vEA|F2#p Jom-ٞ 1?}hfCȅw7|Tyכquc6zҵ.<ˌLm~ ر/Hk.PtJ;5>lE7㭯k1=>zdo> e _NbAMomLIK-asS_ qH]ֳ\_|X˞Vg*ˆA=G<Ư|/ .m-𙍌2[.l),{q%uߓrIi]{pۖ]ͨ]  WD>& gkk:}^KH*/1nk1yO)(/<>?c|n ^P RzQj`cP`&B c`m( #Â&Z8 eڡ\ۇne؁?Tqhdgmxnlxþ(_ M%Xikr.WCImuDzKY3Mӹq^Njv#Rv51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1d7y&wLT~(@E($VAaCW0!ËU>X.{ eW`_uZCB(궍7uۼp̍fm~nN"}㹝Gِ85mF|?(:^fl]E֘ڴg*~ Ơ`őSd3v6JڔfȵYBz[1C̣ R=<Xj;!)$?3`wX3d*:@l<>#zd7 _׈k>VZϞ/}XWzZ0`CksoY:p,k1<](/Gs{ }t_lg+r>Q.$r!_gdr/LϢج<#44#q4;Zo'zuɎcja{"6>]lk95.4Ͱm9ђ'c ]]le?ɐMs )|Y|:Vq\#vXs-GɎ<V9c-dn8?Dηl} }ql({8}qcEpsq1@8]bkߪ]d\ω]\Nm#u,5>Kh_#or9ܿ0y?>=|pV,TAnCQ)'ή@ w HQ視P3 \;<䡧A 6A JAlemX"bIkPI&gH۽9bt%!`ŕ ,Ի ak!ِ} ==eb*Kɕyb0ۇ>WBTAa_:"ɤpC~RH8~MCVm'պOdu.V0}\Bonp>y/ PqW>ب/J8yow0Wj-1y5VczsuJ;>/g@G{%{MvX%&sYcIJsqYVZf@5k8zZr=m_TJy~) ?x6= ^>-{Eb6ʏF\ ښg,?GJ|ޚ^ b`?|$&~y]]1Ɠyݺd1<ڨ{qL|d}Qڤ4?v)f3s)7~5EwE KT~uQRk-Snc| ]t@1MԀ!d0ƒ4nަ߈eoL3ay0mx_xXӃ=dP}^NNA*yTÚs z(?bcj?HIL3r-3g̍q-+5%׷u̩'t+@0/,HԬ"SOLUؑcؗ/ruaψ-Wƛ.Zacu Aޞ9@HNsBFy_,2MuRGv-+֣ sȢLt[=c0(zr5̘V5 ƙڬPݜZ$N2^_.F_*ۙB&&J㧱XӈaI C Eʎ<_3sĊw%cTx75a{vkdCo]cf6>v{jfsLcٵp_X1 ̔ځKG@ˣ6=G˦v?D0/m&ڻu?q\~MϪ)5>KVf=ө_ ^ϢT"d{ uLA Zg/BvVhRAt>I>?bmJ܌#ಒ# 7iAe:c]?wys?Px99w |ŋ:gH ޼OG"p7捘HWĨwu b.M5O|MvY]g,es>6ܓX/(ږ#&/grL ՖQ@S6Px1\bʨ9HEdCѺFIJ x6e\lƙǽ9 vtXY_WkOr%gq?&e^6W Fܷ-o{lqe D٘u=fq\#Sx,Ts NuiTKlU枯7=FShl_AUZ~`^vLv!t];ro)۲=l`St!lf ̥bnG YuK:.-6 R!mE#1p(Xa-Qٽoՠͧe_k_E_p] 1:8J8 &yR6 Ww}0D; Xulb9腟y⠑W_P:SݰhXhmnBmVSnTgDœ0O#m:xrqr܂2֖;z؏zjέlL~m#ivnK'Z+Db잸 u*Vbg7plϦMa"CϯdŒon9 RxcX"m玂T|D!(Z)%!c\ c (gOo7zJ$_ <n"q{eoDv7,k}Ҷ좾TqU(t {f3lﾥX f3~N@&{!m3up^\o:anGgPVs1.ۜ[\ bXjSOT=wͳTT8) i `!|ПmǑodC#5/۰ ɭv*(ye8a.Dg}M&ޛE -Sryz\&'FY:7Y y6E?SfCB~k%Ina㡼\Av;Oe1XJ.<~|,H%I$I$I$I$I$I$ dA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭy/}{Wf?W~ziyqx1ƥc?v'Mﷇ7/>/ޞ#{sxa2Uo8/ykHK7?bk-xq9?<+2vo.ǿǿOׇgll`s+m~ܬzQܧw_8Fo4mE |/>ͳwOmKm|:v/?{N?;~d|Ӎ;lo;B&|Dyϝ[va6xiHrEO|nw^>?N^^8ܻp 27.jkηO[9*VN{燓D\Qטe{:|i]{*}uD<$ۼ9;q.軞fʙ/raYuia\Y閹8z\ e_(}z ;TǏ"#'R bh mW)qۘ/ӑ!kMN^8Ar{ ) 擉A^X8ó7 /E(2fcn8~j0I^桼8u,|H|Y5/}'`~'Idgg(Y dMȘ !AW/ E`.UދϞ=t}= ^>r5'gf #fP/~vgBb}@>p.[cΈw9wveݫaryv7e}1aGݻ $Ld_GwkGM)|i]b|.P>sA/Tf:)rjS}1wgj Ϫ>#=+%W,Hy/zAz?YE$Vz~f[sR"z=ݮ{\`X&WNC>$yfyy1(RQHt_7{yeqʣ0|N^wMvH<[1 ;L"+2&)b\)z\ TloIkŇT°K[)2ؚmmn@UP бVHhER Z{dkdO6*mUt=O)А>n>䲍x`Yb԰>/8_3aWR' xSs+^M|59CWT7+ٖ(9+g{rl`\lGvڍ禊yeq3䋱LELȑf~[Mm//&Oc4?y׾?ֺ6s/ X(">ҝV~6|ES)6X.ֆ'W5}?p=%bƁ.T7#l=ϕ"LOF.Boffb\g|)t߾29{?*_K +Njv&}d,7Ջ΍שH-+?ƚaͶ: 4~;DXnvhg1  ڠ?ՏZ jT7L|es{wDD毤X}]>F¾>b8Q,[5A?Z<CЛ7s#:ƚxul5]FmfV†3gUP `tZ?["_۷1g\1X} RPj^x}t" Apv6Y_A*"ӥ R&>x2di+A'hP~p4yaT1~Jm@J/3ۍޔY&)~޻qQ|m@<1"c}_?>ƃ ԝ,vnr.yĭzouF7 R6__VV$)EQ+ ŒYѡ#""UىU0amG68 R?S~;= *M5ˆZi/c@>>Oxq3>O=a|  }O||&z2R6涡_-z89˧}RmFLvÞ%y3;8TO\k}m/}Ȭ .9ljC_%m&lW˶eY+mǕ{ ٺbx K1穱˘m&}-dKY&84='1H]B R_ ]ʞrA;ؒǂԻ1m"ec6o oliZY㤞eYՅgy¹ٺ"[5li:.FscoUf^Ndľř퇎1{ՂT/bRC RWI?pꅠQ: 8)HUQtrA|.ˁ `$`7ds_^%# _`ܐ4YcI ˤsBܴxit lbDclل0VL1 ? :O>{FR ]x|c?D |ϣ8!Fp/us{{{ \?="z\^Ps.#ƱUl/=X6k2|z3&"g6Mn2З}rQqdOAO{(@pd1?xD!UfsбvTA /.%iHe#(42F-{bo n, a{ Ʒw+D>^dVk! =/:B;y{l8$GLt*u:՛к@X{CɖL;*X0 d¢NȓzqS9pmN_7 `̣\W6 ba؅ R\c+@Ymf! P8|R>M>7[15tu][ϰ?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{Y 0ٸtщ_mOS ?q _St"0AcAɋ9?;W\LښǂXKe3/35y`b9[O'Kq<܏@WIi11qՂk`k/!>)7~5EwE KT~uQRk-Snc| ]&jxcc 5(XmozJh`-87xxV"c,+-8FOP^n$$ (6a 9AorC1;5)=M;a:D..uV0 u(ᡋWv!IV .2d9 R&Z*zE5@צ}-HY% JtxPP5sJ.kJORy%?ǯ h}&gc?$_zs0y~tnb3 ҏLvP61vlV|kd,Y˺dOʞuQy0+pnG~*zѶ uE0LGNtn#M@?5ǫym8oMAp ˖ȶ6CSc5h]f=eڄ0۲Eg⏩|}S5O?lʚ7]dxuNm6ӈɻ{ާ `k/l1Xllw?J{#)w"ڣvck"&d3&?+P?=rrS!BQm2`m;Sǘ~qbe7=[ymhyr݌_3]7WT d `B FcQ6+&Z{AO.⿊Rທʖ(Fbx\EHlma'~aXo x~Qo:EdyXSݸ|f(L4#{ͫZl 9i'&Yn1:8R:W'Cmڽఋ\>ȸ|oi{7a._V@b /nb nVQ ,Qu+`r ] l{nY`qFRQBq܍OJ:g,`y(`gA `y+F|G%<Ko^9+9cUQw< =%>nDZZ^93ClyMXK} G֛ہ~_m9o3nW'm-.^wR/Z;_ #ozϺ|}|mtnra.C.u_g.P.I',}̍n"gfrb|uNp^ hs}a\/ 7b#Y}O-JuQXbDV/z,ca߫X#v*G8`br=H\Zd]`2Dߐ=6_f.Lɾ7e:_6I|=3e3n_ ڛK\$gб0'Ȯ1|݂c`wA]$]@`nݷ fI\{xܣAv~kyxuߺgob8޽$iE{eώW` X>(s"c! R& RGu3*@o+/z 73Βx/EEN/~]W|fŕׅM6tc+/?r<.V YwV֚Kn?S\&qrDqZ{7{~'InF;oˬ|$iO\q}|^b..gW\9i3΍sspߏ/fv^p {=On?$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~Go?^[m߾=ϕ+^xEڀW>o_&}y@ ȸIx{sxahiak [x:g6v{Uؙ;|I񮝿 s\jo{~{F=[3$Ir&.Vq_^k??|t\k-޷EFG5:/Z5mgzh< Rx텅hc`#>'%J̱z37#o<95ooshw?Cax9.ǿ!/<<֜ ~\SV>_74_+zzhgG_ާzzx>9|͒?_s7u }JÇڟ>{m"u:I{79R0*,:5zrFXesl@_z_z.õs^{StFyAx̘wdT|N{i/VhlF~~<XQײp>gQs<W봸O_CaQ~D>?DA -<UmX"714l;𙾀ԺE(Į!(8s0;qw;Vⷭ*Aq1yח (p A X, b' pYK!g>N٧>@Ϡ@dȭ6 4Gvs FCaSq)&Ges y,pԾ<;A熟8h}O{ ]Oٿ+ _&2? Ǝ|\}|vY;|R/B9*π<3Nɓ!W1[.zR gōf^Gl2y&J RuK^ЩOjh}^O/$'6 :n.~fIqBY $ooD>Dx`x_,Ha_rGۤص!epx[Dopsoܻf\l~zV>Huo Rdm)l}6c"AX*BQ_xśZq ({ ẹofN^Y6mU"Y 5~A e@6U\* ݠ` $vL875'lާB8)_WYy6_[s2G,Cne~[-;?|*=ws:g17`XX]Q@gyg& 6q"}I(/lçO =YqV2?W%LvCl-d;e(9 ?}wn/N͜uHlhWE"<|ɏ_suo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AW|iOO0+ʕV{Սcs/@ƚ6i>mfbh)v{ɳcrrM >U9,"sXkސ2ن\{_cM#baU%Dq69}hc˼cI\{*HAxxDs+Z]d}իrVL(HIsMz4a!7l7QbT }XDcRVjeavctC  "Bra榇Х_sMbn#[*Aoymӹq^Njv#RvL`_1&5V"NAj`gpOѳu R}L;F}?)bb7hc&ڮeD6AJ:[/0}Pė2gGgr]I,f< :V*j,A+h=XƵ,8̵Zpk׏ӣl}z=ۅ>vlXӈa(vr /:KV* R Vs{3!>g/ ŞsU֎}׏&սWCٖuh/sn41Ȣϫn]l  n\7*88nLof՟7ɏ HJ[>em䏣tXyxW*ƗmKo}.?fq}1;HŦl'M\)H}>8|(,`/l:vڎ*H>uvݥ$zLV1'ēR<ΐ6˛ƈ!ĆgW2Ƃk`mEM5iZ } ==eb*Kɕyb0ۇ>WBE6 KR`n@BW,)a(̬1fz{Fs (dǍds_Ɗ); Vz_;w\?y9hUFjӱa[bϩ9k<66w b}^. π5@.K(7#. K&UMƒ8  >̀2/k:p {tX?0N6/R~le{:7"_|Z hŲm\ bƵ5=>N{9kY*.,02)+W10<7i,;{O'g?-;<ɹE,{цbRt1r-Oy\K]hqOOv*Qa6 b֮m6Po6`٠9Ph1?13|NJMbI< +9xƙLY5'AƸjA 5_|x"Tʢ%V*樂T)@1l.AځLB4F 0If11 E+$p Be|>ѯ@v)>o:xbqӱNP~&q(@~zݑM+>~ײ~b]Sr};,?0jI^JQ@xu@u9WS~L>A6rTCԊ?Rʋ=guzCK,w?ޞ9@HNsBFy_,2MuRGv-+֣ sȢLt[=c0(zr5̘V5 ƙڬPݜZ$N2^_.F_*ۙB&&J㧱XӈaI C/Ņusg(}ܟ8c-Y g'wM˓)lNܺo7{ȵY^'zl/(mc RX={Bӕ RfR$ Gm%HL#Iw̼\eo8<% =!j*2u7.O<(L*KssbyS`>T Ouꦓ:ɸtq!bqcވ99һ"Eߴt^'/urnf}SS0p h#Ӣa_!c/f} UZ:NQ@(-Q0S#`Mk_Gm狍1rٷOeβ;!+r6"WJg73icцW:O8kV¹VleuXlE1 W:vLbT̻1 &'&DTh3mEʹxpImr yZ a-9uejy&^E;٘瘂}{S25Oli'2xlٓm2ukt-=]qd]\YZN)u1XT!ٿ7<ԭ]q|~D8efBO9\ fb3=Qĸ:׮p13 [撟g}ݒ|\ LP4b<׍_1B_\ |ZvU R^2c0x[FWh&eZpρWTAc<Y(eZdrڧo/*m,c{S۬0')L *3U7|, D:6SAഝfq**?X[hub??M51ݸBnU\S)||և"j1KℼL-JP&Lٴkz~%[~jQ)uSxӻ\b+_-FIم\0 R |Yrs \2?E*+<<΀rv㬏W2t^}Qmݍ> ci߅ẫ39/UygU%~wxC ۇoi#.LlH ;hݻ?\?%btN[A?Ժ՜6u 6k+.ץtF&1if]=?y)\4g,͞3Y|^ lQOm1r 50ٛl@_t0wo=.KW]pn/GcLa. =Zs׊_lXkIbǙ3I,f{Qt*y*t,Ea5fE.Շ2/B򰏽u-sj l{ZH[{5|Dq}43M-Z%%nA1 dxK˜X;M 1{J<}yy~dӋὤpc]c)InG_fd&8ٛ6IdAdAj7X4 y3rDXxق/nӗxs$I.E&dA*yDd~YVd=;|ֹ' ,H%IP.V YwV=5|Mn .Oo3n71$@㡼Hy582AA)_<{wO@dUX,g~ryu^\y|{3 RI$$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~Go?^[m߾=ϕ+^xEڀW>o_&}y@ ȸIx{sxahiak [x:g6v{Uؙ;|I񮝿 s\jo{~{F=[3$Ir&.Vq_^k??|t\k-޷EFG5:/Z5mgzh< Rx텅hc`#>'%J̱z37#o<95ooshw?Cax9.ǿ!/<<֜ ~\SV>_74_+zzhgG_ާzzx>9|͒?_s7u }JÇڟ>{m"u:I{79R0*,:`z~2~K'7t32H)=Ͼ1>TEn@M9l;n8C2kp38 '+?fGG\Nm-e oPFcL^$c -HB@F Iak9lxRy{ϠSO6Pt3(rM?\=6ѐeEaq(ktu&g@;_ʙ_xÄٹN:7CIPbAʻ.~ :0ɂT-"7ނ/vE$6TMpо,)Bh7+=$>a1,\)bo1"ًhRfэgۿEdn]1'ƽmEKw1{@nm#]`T/-?khπ/EfV2:gn3&*ߏ/WpW Rlf=i[P~^09"an]Mk?s+m|L|$4ݹ86sS GyGg$y:>,3ɭ_$RAظ2mקM5#EGp>X66A9u]t E um{o<"d6+a5/6u^uEjQy6u@Xf]6ͧt\ tm1.r6yv "WB 5ڇb0BgvXkސ2ن\{_cM#baU%Dq69}hc˼cI\{*HAxxDs+Z]d}իrVL(HIsMz4a!7l7QbT 2)H7z)ePVZlM;@2 "Bra榇Х_sMbn#[*Aoymӹq^Njv#RvL`_1; sUxkh>aڨn(K{+|o\'O${A!^_X ~u &D&ZEڪʹɚWeLY=f*98['Εfp?8g .?8`SWCʪ6s=uI^ Ť҇ȯhl+E# AuLA qSV䂔N,{щ;T %C7cnȊeoLT|WJF,"!f"lzhs\H"^57A܌ Ѹd1*,JӸ( ]Lj -7cRr.曽Bx xЊ^ P IF[D+HUJ!)`c8E6= QUզb=嘶k-裶] s RH~0rrA%zB7Ȗ:y'=X5g׺^t#X솶qcGJٳqC6KP ZO+hqc-K%s-g{eb0A_xnv+m4lEn'݅d'~/DZfσ3}{UqRc[;]?ޛ4FW^e[vbXǿ"S>u]& 2q=l85v`0U\'?.dS"e*yJoaw>f#ْ?ҵyyT/ r &i\\Lchw5ȑ M:N$ S}p 7PiYO+_Ht7U¿} )KI6f'xҎN%NzCvyS1:ؐq0JXsu -Իf7CX Gl_̽Ce)2O fЧJ/FT hQjtE;!,V"LoO6ވ}"csœql[ql=>7X1eYu09_˱yOrث$?^c<\90SrXym)mt\Uok]H5QnF`]ʿL[_e%+q|f1ZiCe^6ukb_Fo`|Q*m_0(٘tnD&x&e(?q"7:kkڣ;{RQxrQa>~)HYqny׏@ܽIc{j?99mىa=M-b9Eގ6 ȕJO+L]mftZB{zS:a ͗\vm3qzq+.́GLjm6Hyߵ++7mf'm .x0cgڷj3d}֜AN),|^ B}4 R0nj@P*X!ꛣ RZ8ưiNOD3I YD*$MXd<ʯl(D NE}!i!~͵͵K;,5p3+/F:nF_|ƸM욒a!Q ORHŋ,^zUi[d +VBP^\9ި |Ψ eۓ#)siA(kEU [ȮEsz4<~Ytn+g},EOj<8SC_ SۋK}y;]ׄ8^i4˶Qzq"73aA\Y*H9 g1B³>f!V/ ~ܽ7i,O^S-;s3_"gzuz\ ] |:|-G#}Mbv{bie6%꽔?WIMa98z,ʈifXvOK8]ZT'hϠ^M)ek~N[ן~~ $>(mc RX={Bӕ RfR$ Gm%HL#Iw̼\eo8<%(t‡ ԡߴ#<)*KssbyS`>T Ouꦓ:ɸtq!bqcވ99һ"Eߴt^'/urnf}SS0p h#Ӣa_!c/f} UZ:NQ@(-Q0S#`Mk LΗlr8F Y~g9cE.FJ)]k^Pvt/``fا\ly&m\3ڰ@:Y 0gpͳJuY8YÊͺ(8&Vgz3IyWQ :ޤ=\ׄ(muƲmPY9c=BQmYE/'ߦ"KxVAJ~R簋c< gsjo69zޔjC@ [ L',[x}uL@>r]l/q}dW< V] 6߱ͽmV[>y"iԩpd8s{e-w:Ț[n G*_>>Qb %i}ZqB^_xU(q s&zGld5um =- ?:)d}KuI?|gLfCfAy.=u* B_洹c]9_[q)l.30Nզ+H-1ds{F?p1ldo}i ӱ[ ܽloTh.^vkù1Y0dž^j\+^|D vc&^gV`r'n7FAΫӱ4Tf4^ T8 yxW1{Ѻ9نCS6?--?8Ӿeybs͙ڦ-f\ ]z`]YO$I(+H;?ۚG>&HpÛL[_P^䋼d] /߽;Ƨi _2*,3L<\rwYGN/<$IE RI$I$I$I$I$I$IȂT$I$I$I$I$I$IrQ $I$I$I$I$I$I\,H%I$I$I$I$I$I$% RI$I$I$I$I$I$EɂT$I$I$I$I$I$IrQ.R/K>[xqxwɋo~k/?"m]/D} d$Ir �90Dp0ε^-~~{xR?h&q퍜UKW}7f9yBzO변b_J=*̝w$@x9n.7̽^?ϽCk$9+H/Dy:[ODOm|z63pXIu`])^fɎy/9g:>Uv_YO̽6QsFGd~vjV3'-( Hl,$}偋YR7&oV{rC9I}x8#9x`5x/  Rؗ7nE)Ƴ"2|s.ǘ~~6e= uF.F{ۗ5gǗ"lHt`3] ZR  +BS8@٫)c] `W7sb~˪imh= P R.C5/6VL 6W_B_p9ދo}Ss¶o^}/2E{g[joUe;!cq~N9>^V"~s78W/^|>squ qq&Mm`3'җЋ6|3ۛm*Cy^Xk?t 6 B`MX1)@wDYO)_"񳮜]~^+/>˗KP>,6/H5Eֵ RW5o+>uo9)8ȆⰁ ʩc(jm+hsT7f3 ɵw]AW|iOO0+ʕV{Սcs/@ƚ6i>mfbh)v{ɳcrrM >U9lZ`q`ڋyCƂfr}~1gǏ5U$/񣍏-B̏=&Ir= V'Gg,ϭkjwAjT7Y-2] E'y`5dIR`$3ݐGeoPB䯒6ʤ %0z)ePVZlM;@?D Ɂp(|+`+7?`nz ]zl 9$6:"76X;6h="eC/ry33[b,hOOŁnrQ 聿mV`>AY^ sTAjplAۗ_+6~Kص R0?C_@]ej~a2'!(cdne6lslYKVz~vԅ,L>lT4Y5?3 FMo|Ăk_Yϳ:61f~Gl mȮ>51ܰsC7ߘOCq7Sd{i;s7C_l5-+m|vX1d7ޗ+SX)W9=K/\ ʴc?ٛ"f*}3v8֮mPMt?h.ԋYɾr? x]N|)c~V~$q&%B|nNl&)chi#6&kV^1QdH2lj|59WXp Ǚk2RVKW-HQ-&> E~Dc[)^Ůc R؎cTu O'te ^$v6ޓNlޡR(Ls#DV,{cr}x@bJ2 Rf"lzhMyڷ .xAkm7p~4.YJ*?b1s4#,<~4.;.fCSZ{C˦\ f/P1^G)"W,TB RURr 0_G9z7Nc+HEbUX5f?Fo9Z/bg1?ycW0:N}Y 'G |ȲAt2\ ߈ [hu6`y\{_zҍrcgvOڏkčfjm+gҗ>j,A+h=XƵ,8̵Zpk׏ӣl}z=ۅ>vlXӈa(vr i9V^^~Χ<Z9=)H\'5vcIctu~jPexn?\`/u{-8|(,`/l:vڎ*H>uvݥ$zLVAø=[ ~Z+HBӸ,a_u5F[`y7woXvO~e[vyXϭvsXFѣ >rғb SW[ٟ0&㞞T~lB%׭] m\l\{Acs8c~$81bf?baYa_}c& Dم<LLVc&Ϛ Irc\ Ś/>KAyFA ƍ_@ h]e+_]}sTA~z|Z Կ6u @鉈`&i!kC$ɁޘgPwE!Zp¿/F,X!~͵͵K;,5p3+/F:nF_|ƸM욒a!Q ORHŋ,^zUi[d +VBP^\9ި |Ψ eAy"Gx{r#e;!~mȼJ6dKٵhX3 3mEe;{״3c"[ט'X[g?hz?@usk8>8!wʐz{q3}o^uhq>珩NڷўAnIr>_ ik?&(OF ȯh!}/m>-Z*J)/1P Nj V] 6߱ͽmV[>y"iԩpd8s{e-w:Ț[n G*_>>Qb %i}ZqB^_xU(q s&zGld5um =- ?:)q__58bx/i/iXGJ[Q#Y;Y걾G wxýM$Y:7Y VB6Cތ?^%)IK$82YJ_ָU;Y5!u,H' RI$Um#j[$Kی[o &-/Pjx(/RE^r 2ήLPbyJӴ/o˙_n.9,F#ŴT$ "$I$I$I$I$I$I$idA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$()H}ŗFw߿ѭ|8χVŷo?s{^6ᮏI_2n$^@^}"pZxyw{l?=y)48{F*vߥ3ٜGZ!޿NY}/c]Gg;yg1fy]xy}x<鯭>Gw?VexG}f;+%&g=ҽk\wי==N{wpXA꧗O'o^y#z]qx2}/>}X$j}j;طnŊN*HфPnZm&1FK*FxfqX9b|Wy{C;;`!8Ƌ0b[7v(z!/@A NA j/x[޻|^?pgq>kOOhWO/z'Yc;|'2KrٸN}""O])v}~WV sMnT'ixsGB~P/>{^_W$o΁}3jqݛgC|ٻ^<ϩ{]Oy&91?`wm^<<8qg9Ws}m 73,b|ZmARfgU>9=R?Nˁ2,H,ϟO%ދܣ 1W׸pٳ1|):-@!Pء:~?b+H50E O@mJ|a,H(QAݽ<)ȇ[)}~jϙ$ƌeWFjʡ0?jrbq$pa7UҺS]sD.dYYvtuD[߶޲_e.ZOɎ;Z8cE|h_4gD ?A}yv ?pYg4fߕb/wcǿT>Y?^T3yG샕=y.3&J x;!#.^ȽƊ5rrZ|T~?Y(H]&}6ݕn>Y RuK^ЩOjh}^O/$'6Iܠdo4~fellh߈OdcoדydA _*:;`A}%[${Cw0SoO[{c̉?f\l~z'tF.F{ۗ5gǗ"lHt`3]эDR %~ X 7 P∘&{~C+/ަ_yv@Q/H 1)-k?qDQέ#/!4~~훚}t| %~VmmW:܇;9 x[ߖ?oB/K.ivloos { #b50&{ ?7َb6ħ|NOcߝۋi3g]>%84sY9KH߳_TZ|&hk1%d C;62^F.;KЧCތk#e6Гw5Yfej`߷=Gcg'Ǝ}AN_sҧSa?f.d,o}U^@Ik {[I([r j0_nz~[eocϔOXboJ5Ex %pK|]3I}_r8~s89eͶrο-gu֟Yj8G\R=-1X+ֵǾmyEO،uoN~Etc-?mR`.zv9M%} mbXsTP+ P3V W.Q69 n2,49xIK Vۆ!0M=,h򧅣IAJ}>*IY0n}}VC&JIvֆƦ1E$\&K Qq!q5t_GU>t>X;6h="eC/ry33[",BOOśn_؄@G(sE03#fAjpzAJ*H"POu G_bM@WFAtOҾLū@kߕ\99)sry9F[fC9ǖ-:qTnIgH]8DfOEU8# mmΗq?H,e۟X.{ eW`_uZCB(궍7uۼp̍fm~nN"}㹝Gِ85mF|9?\+>/d٧SyzORwX6`*|զݐ==Pc0a s%Sver_).7ml7h^ Ť҇ȯhl+E# AuLA qSVTLɂBlbсWI[AdM7[b/4`;`q8.[ 6} UAV9+ree<.0ɑH~g~*ѱ*gTr+t~#:lx؃Ys}F>}K7mnh<h?7֚}=K_;dSկba6:޲tX2bx&[k'Q\?_&F_O elر骿b6JO#V}]I&e$G {WdyE} dfaŠ5q^Wײ|c<ۭk}Lvý pW ;df9:`[Y-uw٧mTom9<7ce+;0DI,HlOGv`L(5kWAN>ya IJv0k!s!r]IJ3ű]ίč=ϙj~ L?"tr}rvs='vep9 |ֱt",Y3?}p<0>G xZGsӊGRo6 mGohBJ:n q2x#ipFZX\z𐇞:%pb).ՃdKHrR`+clNZOg58C<%C. ,d1g`EM5O CX 9̆,k-sPYJ>)^hEQT$hQjtA;E,(Nd[Eޞl1׹PA8lbVl7lcجA`0Ϸr>gxg73報WI~'o:Jͱ%:sLocNi{ z[trDzɎr3R0ްdR]d.k,^`03J (FG_K.G5zTi82/G]s#5˧0W,Fшa a\[K:I#_L/*yk4z18)~oPxuu5O!:vZ3hr39b Ei:`ۥp񠷩V(shiyTom-Z''Pl( bC|;07(ΆU\!s{&BY N>_2A7nΙcLs͂յ}GؑGx/ޅ(1tT3g79S^3@e1X&,Y~ G G hzyZbl ~cm oẋ(<8,X5x9 0 kz0,Pq8 o 8!wʐz{q3}|S/ 奢~ȘŻl7LHcݳ(^#|Z30cSk6c*ˮĎa^8flT=X:Zqd9Z6!}nC6MiܵOb4saW53ODaFJ3doU o87lX6ۀ({Z};Qj U3}c^ /z.a#hkٕl rXq=5?X&,YQ C5L~{ ?_|Z?STXTsGP61),hUYJ)$v6)q3`J hoIJ7$RUdsȮLxQYb˜{a>TXIS3$o^Iƍ s8F[sw+D;K-ףc;:~ksu5_ {6\zŨ,"ʴCW +z+sBAjծS;^_7U7u -r_L5x49ֵQf7T^P%yNG Y~g9cE.ȕR6ϻ27ּ(>{s^ <̼O LعfaU25u`"{!g8p.ue-[Q,qL0wÕ+f󮮣 cA%uBI{扯 Q<9.e۠lr.&{%ARp>rfR1 cݓj>oy9kGNm e[>M\l.DlTha\#aCGE~A/hy>9+l#b *-k-( 1FGiZ$oXBWh'P,B_Ji/7[< 6߱M( xraXtmROn5?BA$?=ld9dTVy>ylCY0ugMa:n޾(6F|ʹpC]ԗ*μŁn;slCݷ @_s}& do6mWk`1 ]'̭ j]jN:evR:#A 4Xml/'OGk>I*'1e65m,8qs8>qMCtyĺ&te6!Ue%'̅¹|5D6ؾ{_eZll2aGۏ\?Xg[>~C%@8QXCjgȅ mP@>MAK>?\9ρ.#ץm`ړ}Gz]'d^y`Mw1O(gZhT.۴?ٛ>v{+M@͕, f|ۍX[`6u RGHep<$}(ZȃPI;q__#7fW3[>O/9^n y5CjIr)ʇtpϒǂ"^s7erlsKU6m|0O# RIbqve KdA*I $Ce+H_`_o%{(9-tǹ;,HIqUm#j[$3em-<$V֚?˳,$ /YRw4% Rͱ+s%Y?iLj:?ٗŨc) Tޫw? $I$I$I$I$I$I$'$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$I\ _}5~G=^mW>\y ֎wW_[ym$7;޼4D|sx{zvkmgזN|ݞou_&[a#ϝ?. ɽ{:3z.M//5j ;+H/Dy:[ODOm|z63pXIu`])%b@S{d? +*q||y]ޣ?؍gctN~<?N^^8ܻp 27.jkηO[9*VN{燓D\Qטe{:|i]{*}N8"{mދ8W`]]O9_ư0tW K=^>=| ZpG yX)T STݶ\ЫmFȂleɵ& / ߽ydXnĀ gR G_xx}Ad/#G 7F*p"``'>Cyѩ˕qV)IkɁoXb`kz_N>O( ?x +}Q'@Ț1Z*@,cwɮ^| BNoa BP= >RȒ=3! ۉwřD  Z=#¸(H[cΈw9wveݫaryv7e}1aGݻ $Ld_GwkGM)|i]B}#_6vB<#+f3? c.yϖvU?W}FzW K=X_"N-L~ UH 6G볷 礠Ez]~%9 6L/|HT[ Rxy(o $/H/K-#]V0|N^wMvH<[1 ;L"+2&)b\)z\ TloIkŇT°K[)2ؚmmn@Uh 1-bBZuH!,h푭> Ƕ_VY̋`76EXv-Qs큭qqʯ/)@q&l}k s9 Rp75Wnuߤ/j_C> Qyi[Mumy򝝳|/Ȧ}DmרXqn8P7Ct[8_Yy/Plb90F3|akn3>I!_-݉썾ngWt>0鯋ڙbo)[bm+{r%]k#c^"FaBu3f\) $jD1"f6l&u̗Bgh{*cپdrngGBX{ONn >oSxڈRbcovl+#@C8ތ1fvr]͠ SX ϨŠKٿxk YEneLc?{<-{s9,}U[hţ/=Y|1G)?ky']VeAVlFM/oU,l81 8sk\+ A׮'5 !۽}[zyq!sv u ݇ V'Gg,ϭkjwAjT7Y-2] m⳹![@$~؛ƯK YH GHĀo %e_f)!M ~s[~aI7:,yBqcE~}<;Y\w] ޔnl.!ץ:2[#F`T/_?{' bxYdFT:^v+ޝ^_k_JF R)\YAg޵zAJ'~ϗ>.Χʡ@4Wg)!ecnzۘgϼ|g umzd79Xnޑ13C5{~A BY1zmZCl6{]f¦}l[栯hv\8PzX+/ǰ}+}i+fB%llAs}]%D˰!Х칡+S끾-y,H&RV)9uA:NYc q,L殟9چ٬fklS~P.rM;YjNs b.S&ƺ|/G͚5?ۛLT*Oekf|q Ѿ`.%攲g1< hlC]!cUy`zvQ3`sIU]x?׊-+Uö=Rl$?8>k\E_hnLOY~ X OyɺÿW-HQ-&> E~Dc[)^Ůc R؎cTu O'؀& PھA Y 8SL6E:Yza8^2TJw5X͗^}g7D8~_r?qNO0?vXI#᰹Hʢ8Q Z6ns0-)1Yz.nSŽjdcRr X̐* PB+HM] k}`q٥9,HԢbjic/x(*H~XEƙ.gvfyzs/R~6 k'ו\:Y?4l* Oɦy ZumvغQJ7>{*>_|y? :|cc/*3ƈ7$Z1ыۭk}L6"dc3[hl6[W<_Kl`N NԿtn`:Sc1\Qwg dNցaʄ1⚹&-,?0lx]`k '8c&]e⏩T\lбF ٳ4JJ8S7_f3|S'ܿs?2Fe8~>_75=RY,pY~,8ԋsy9hCrTO۫w:c` X1_zsP>Z7!?3_k^lC_Ab8F=q?>=|pV,TAnCQ)'ήɆ# җ䎠`m 707 ( E)/ oo7`oз|g+~$D<4 ApNO%D|6M@2ѩ):?VoB"kRoc %[ʒ:2YFR辫`щ8F1bDXiyRV/yFQe,fth7^qŕ >Xk1vi|n2 ukdJ~3`'gc8/3A}=k)yNuidӽPcp?Jޜ?*Ŋl}LJcoCotu][ϰ?frm+n3>W0%n_l׸\Da"²'an}Į&5ٖ/oA0&scI}Lm}oAރ܆sB7"exuұINmf{9ea q+L}$?~.v E;a.ǂs~,vDu 5׋v a 1?ѱJf5:_;f kv>h %sb3N䏩7⬵y@b8Fc),|^ B}4 R0nj@P*X!ꛣ RZ8ưLs7@jP &Qb6l^puLřvE?/(7H՗+?~O@G39y`^&2{E3+sa_h~g^!Ϸ[dK>^#c\ֵ>f&{r&Wʃq_@7Ns8Sыl`+/f?utҾs m1<^ku13xn ڀu`X\Dv-îx4eE6cY,&ܖ-:Lm{};ޟy 0g{VDP&SǛSwjYFL޽ >뀹ߕ\>WE=s-̍N> )sB|hB0f|沣 6oc1Y&CŞV/:,=Yl|A"u1Nm"KجeWB9?pns}쁞w {]\U.ŷTviwmHb>P'Dl6^X>ĺD uX[Ym0,5&M2cQ*mZǛEƱ\g0^՛cj}?ٻn˜_2W_s5|$gp}ox_fݐf+կqSz $_LhN@i_=P1ϘO8vG D(8@<1yPO9X !G۷ъn?Xv]L}scʼnalqd׷Iu3)v\R1f%d _ ik?&(OF ȯh!}/m>-Z*J^ R"([xmrOK # y-,H2p.7Ca}ٓn^bKȑ\NK=1p'ԑđҹB?YojnE]:A{Kػ wr϶D5Z/(",nVQ ,Qu+`r ] l{Ţ ?u\*[2 ^ 53.sgd,t$9?TS\B}K \'-@l7Xh*硆g( u#պ*ŕϙ/FfF8kZC?ڴt,bnk|u2?lCow/ĽW־L}Z& lt=x{k {w vrB>㱲r%gz =@ǬhڻMG|PX[=Ge7ͽthsqwriN:f gn,v9Ǵ6k{s*߇K^@{;?z7e0y_͠mZ&{jQʿ KsjSv=fc^Ua#XI]f$.vj0"oȞ_moޯcf fdߛxraG/eky|ݤtDę/%.3X|OdnA1 Sή]nG[ui{3L$IQoQg X;?5n<<ٺo׋wȷ1_wu^ك"Ʋ \g+zrn,kq籐s)ͣL@쏕=|gIrs<ė""'Ix?ܫg>3~f &\vB1ޕ +H;?nk%)q~8u?ћv=ݓ$7~ eVIw4'D>> E{g_3+ ֹ93;ywa_=Þ'OE RI$I$I$I$I$I$IȂT$I$I$I$I$I$IrQ $I$I$I$I$I$I\,H%I$I$I$I$I$I$% RI$I$I$I$I$I$EɂT$I$I$I$I$I$IrQ $7?|Wv*?u#oo^$Ivpw=۷(~1VEYf᧗O^x|ѧ7OS__~kq$3> /`]W8.G[\2<|V{r=3*Ŗ 8{9q"(׷?y4xϓ.PË~rod'Ic9>/ -{owO\}lK gQ>HݴNMn}3hۣ4\_oE -HW O_O_|o/xo=?׿'8^\RŁmN\׽B_o_m@2fB//)=E`vтV)rEey^FfA"9||v25~W×}˄9 >sqe{{qȾy RP /(@HыP0Ћ пjA+XȡK%(=z{dozz0JnS,.Q^sӃ 񶆣uWEO7"yY")Vj@A?7UoJ Za pԽ1;X3RKx[FpQ,].1jAIy{\s>\y/2-N}ೋoxπ_ E ]^ 椓%e13=?Y R'ʯXI%.N&.G&[ 7dCm *kS"QLGYum:c؊=khʏצA 13k3"ȹݽ5fuňX.gc:ٕ݊\<͕`Mϯ~f"ɚgy6Տ1|!w"6ܱ~7w n@L"dx`^Г+;żKBJ~@ZJYYAJi8zFdO![(ə+<ogn,Nx|^RvAc׾h} P Nb5j=W0@X'}_6軜Sͦ{p l,F`Ca]׍ܓOGBj,PC"+śY-*ȂG6Y2'z *H YA*م=0sbAj 7U/\6_v$_E*T5 x]>wM>/k(@Z^b¾Oyw*QA?_Eѿ/2b,+kOuF G˞1 Jm-lnv$ZOsZi#{iXb=L<o` tEܴM )71]! xSnM7\vdvoX6pMB2M)'"Do%'쏓';1ibB6(aʎtvMT_<(I6]c庴-mՙE<-{ _AcOcFo|>]{(de{ʘN3][-CC2vAbb!?Wױc.^|Ue]=`N}ti㭹clډOٶAk9; #?N{K|VtFO3ݦSr.%סbw-fXi S_#|_0`>Ⴢ{2per>.j_`k[|9tcg lXvöi͢\Yj" claE RXp /{Q)Uۼ ^ "^R<)(H@ce=8E(XQA*+#YXϻe1Xݪ鯻yqK^Z/͗0}<~׌g2ۮ8ҙG nNf/םh.qA Ok' EQ@-BD*h]%_v4&qx E_*YnhŲ R3Oy!?y7m&og7fJ|\q3Eop.d ল3nYD$_S׵.?G=|__?b~ˮփc%ڊ5:1,{!` /61T0H}z9gZΕfزu߀ƻP؈b_ѫeZ=yY;εgBLsd`x{|casYn,rffoCL7Ke\ [muҹeO`iL9rΊ%aq#lm],ôG!9aO¸`]sΗ-=`m?ڞ1 Rժ^WjN(ߝ|96Nn:9e%΂\NJl'F:vC`3Ix_GJo k-nJiF;YbϿ-1&z{LA"eϋϞl՟.r1m%ӜG <09FCcȞ MD9VT>?o/ҲۚAsڊ; Q~jOpe/l"zR x@m=sD97T0T8 g .xqFBuAޜڌk}YL` vI"(\`(R"K)\룋9v$.D>^v2[P95Q{IÊ׊ZP&/ynb>CAJeܗҼmmUvâA$ R\~_ק8ײJ;Ӷ6T,se/2[WZ2~Xڏ?1kr ? dg1*.){c2Y 5nveOr ڙ+ߝÅxFmkaNQXqHg];!ϙʹx/\}q[BdFQELToǎӂ': 8 hPG_R'B\㩡u[.F9bT ezm잼S/A/A66Bu+zVA,7X-H_+(E5n 8"B'ɞܫrFfLӸF `e G)7:k7~F{-ח2A7gƮ܌7*[o|CSzP6׵f8C[vqy~iϽxbAƨ= AǐM ^/RBë5?$q;C=k\־_+ ~ރ+""u>vx=c~iׯZu\P)sO.we]C7xC|j]Cfq"eֵnq4UIb|}lajC9oQ7rɺ:0n&ǔߚmvlcGM?.:^mDB?k}ui80?p.){PYԩ8/R`>)G͕=)r+i1 ~¹g>:mB6ƛgތl:Ɛ=⬶+~L5h}iL,ƙ R!az}h8֧'O+/Lz=6Y)pTj~)[~l(d&>?my$ۑ~/a熼u8^E0wx/dV5c^ z)ӭ z{(E""E$ok_+H(%^Xܬmw/ /'48?6YC6bݒ i]k+Hmԛ)7m봯PɎ(7~CUnh{Yo쀦`7>v7z3|k#|P߉8a]qGoe/CzW d5PvJ.0d}znؤ= Vۛ.,l,] km_k&ba[7f+ ^On;f3&}zF>Ɛ=!Լ\"lnB"??kH{#>d6U}8OzԿ Ȯ4sM`l^hq[C|8ubWˎeMLt y_l߇!&mҸl#-pINix6+7[vS&Knfm{7o㲥 9ތ3goq6-ó>eaͤe\OбK+>RIr(6T$BQ(yX\ ukvm u8 {9}_̇$gb0azMc8l.IH<<{?K_&; (VYJC9w 3H= u?oY|>}߿%Ũ>9 qh9XQ\Rv$|/T,ReA*I$9Yz`dKO@/2nQnt|#&}Fl[~:9ŻX:[Aj F$cSfIN!$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ$I$I$I$I$I$I.J$I$I$I$I$I$I$I$I$I$I$I$IdA*I$I$I$I$I$I$(YJ[۟>+}wº򑷇7/E{$;yuz[n`޻;ڒ_^?ww^$I$I$I$_=='+Fyr=?==kв'Z{e>1ާF]|b(wMnZ&>in4} 7D"]ʋwYO_O_|o/xo=?׿'8^\RŁN\WvrOóZmrI7o[Av RSEgO`%I$I$I$Ię RP /(@HыP0Ћ пjA+XȡK%(=z{dozz0JnS,.Q^sӃ 񶆣uWEO7<pp Q (H\ R_ɘYS㿐>7L.WzɂT% Rf% RI$I$I$ɻŅɾk(VpE S {  *.Pq7aaYZT+mn0dO >AUp[[mU$ G+){`Ă,n syqZ_ĶE 2\_.BUC/CǗom G_طV%*H7^1PA__{EvMi1Ach3FAMc>c܎=DiQk=pd?uT6w m86\d=;W"o+'u (D1kA-&ycJq +M|E:osтn6_*:g 6$pMjHHvo.g}Ѕ,'3;2n `y-I$;~5_l?N/O_nJƭ4I$Iu.Z‚yߋKh]iߟ(xI)DYE([)Gj RX_~5/)HBGV}Nu6 erz!l$_旗셬\o:їd>oMtB6{q/lFu RXx_;Y)j%TAB,8>I${F-HѢ{vqZOsZi#XKy>F{&^O˹/E= ##*İb+ AqD/XD7޳ut|X1, `ؠ&늡zȃyQFPG{,H%I._;?_qm/Mau,sV R8V"oQ47_/YJ$IqT)^~b2M)4>rG7CX0"DoO")HQ ^yٓSExݍXT|qX ?_b / erzmV/ I!^ RE>@[m-J`ɋc*t*No9ϱk48l=I cnW}=#5s=՘nk=G[ dҽ+ PT'ʹ R{_ ,(~0B4zE(^(afY VXdp]3b_5Xjw}[ ݞ; RI_߱B (&e ' ZlyQeuVZ*y ]_(` sI$IPHA7 X02TR ~eb]>s]|ApMeaRQE: ?lRO r1ͱsB^(֋me>g.H‡ER"XVwI?Y80 RpldmSelQa]wg7rb93a~6?y|UvnbAܺ*)[?nkh@@FNѶPKoPۭ/ dv!$~JHw)'RދhYXn_{%ޓZ$竻H9Om~8u^L}QR]]%:#܍Ǖ|ߟy?l2_R>n$Ƅ&[Af8Ө)R7qZT9?~p$_Ȁ$/|bShW̗d>>r㸬{9T:uoHgC)@(Vx Z<01 f $IJ|l ЁN \־U({@j@ @ TaE|<vdԲjK*~-\HYe)$!Y)q,a}+*Ӗ@Jϧk@L#_&-_f|aTMuu>Ykp`h 'Cepq\ e(52@GrGR6Hű>c0HO5n{I-)co s@JC*lwͽ zKdkq#:)`%zDHwvH`S(-ch_ Y!&p@ʸ7 .vRI;{g)4[)D;t۬,8汾CJTy= ;U=`?)`xZ.)sQ ] 4+Oqd$=K~>R:@JPwHXA\ǐ":>uK Xl>T,{Hi$LiRRcL^ZLY-N>F<Ղ7)"n>L]G>{3]Y]'uwzu~ߣ:p#aN Ze c@>WNn Ė*{eWW?N.~^y4 _=<;oܯ/O_ʺ;"nx雗yezOۋ#OnݷO8֯;sgzʼމC}B?//ϯWy?/MoյlVvAҩ_˹)QjGB P 'E/qR# |I{hK~ M.=iݣZù.wF-]R#HB,P@!XC%dYW~ @k!GAAK#`} `R9}샾!>~ȐpXm'vx\=p>"X&G1\ H>"ItK8Ϣ .GWͿ3'Hq\2#w$ex$Fz@Į{|rO([+ĩ;&c4Is9HU ka$:KAVhS0 I~J%h @lGMqQfS.lYm͕k+5p`'z~Ya>vR~t`\-N7o\EoyNbQ-ܾ2[qgV$シ,~ݤXdm0~!(u^@ճ5tH= p,̲w%T|zG]sֻ3C% [_Y}÷ۋ5'1կ9sqŰBYwNBl!eI~(%#LV w;؁4JYGZ% ;!X(c_k_D`R>A@عc5i;~gnx,;d` O5}BV sm0B";8 wxN zς \qGYʾmo.nvY:X 2 eX d>}mg r.-<2xT[z,o[`?F+z%5n??ާ?Cqm˜Ql8.T1Έdc\ϥq{ޙI .$.-^d ^Rwag'C)8` %N\-@d{ IXe;9nwA+jY*/e5yS`nO 5,,y׉g?Vg#Bl- Ţs z3nub]'ʥ~ »_8Y@O~H߈rjVWZ H4B!ܻ9_8n@ʿ{;fSh;+=g1)_ xdaT/)nswK2M21zSx!˛Mu;:=@JuGuzu;n< ?ge}@7*z xUW$ΙoSWR Z?*\dżvu)srcu {^[Y o}/sRXo ߙ}}wob9ck?s"x0 Bd>02B,d E=xIk0!ڡOKB@ͷaqcֺpO(1ֳo9SQ{o= 82Zlבܸ:E|Z7sRGBgAMA a pHYsodIDATx^흱N?!?Xȟ>\ F7СDzFN]D֡ϡpk88Y_Ƨ[V%ҡD ȤL9^˸ɞH.8-igTƶb C3_*0m'c& O1XWB_K:X뚛kja޳o=$H{VN.|Klqr{vC^_`!؛,c'y9%9/_GDPc/ ߵU)",,tHwf5-Z|7oIȺ0CC*l!$/ '7uyb=5ӧ/Q-z_ī)ֽʤ LU#[ԏ?юe*̨6O{yJ4X5=yNR{{(yόV&tcX'sZd=IȺu]ՓLiflUaKj[M_/k]3s1 Vu/j_ ꫯ>b=%]2NV2v1WbNŁBݏ{:{3+_3g}+!yn XI.ֵII}z2تj WIZ3ݗ뚙/=v-{xy QtZo>})Z/x\Ŝ]gxR9Hy/_sQ`tȇ?зZO#IA&t71ú?Xo9/FWȉy܀y%P/E}T_+%bLs}$yy/s+:kr%F߸' Y1P_T*`!Sin"}`lץuAه]2 T %:"UmO\mȜ_/I=Ycí:u;oˢP{+w:DP*3syP6w3trMn;oxOy" s[tϽn7Xm&N$C~ΙM6̶mpf2>AX+l,ڒψTsV 0Wu[ˢP-HG:7qQ~Ń r9dŕHX&u@gd N%afmUP<}.s$X~)N't\)r3^~M6mEv/vG66[zFrs6]_sma{_Y>zW9[)(ȇu/p&g.e8=*$t(Eywfam2a0n]~Tv-o]OsUcO='d|,6~R!'ZU> X$am?.D:?З3,'[ײ ;Rt1)'b/0IbTC`:Iw빏a-񰥪g7n]~T=\k/=2_/eK_+}ҳg-gSMlav·n71)IM&L{S"Ib*mUK@_J8!+OH:R:Zc`'yP&8ui7Tw͜_#?0@puSwZIYdJsf>5`IW@8-/@N6u_R;7d./x/*K%}+qxѳ")Iǵtx|Hx]1(ynw}YأsbsUҦĒX+lW>YK+KA^OO/x?. Cgx[:7.B#p8u[ &H8tꀝ -m  WtLml$]:`cn c F ;u[I7A&HruN~6AЀ&I/E`R&I׷[7/%M;ʧ?4_$]߽ix itY}g_읇$]T(2읇$]v/8ᢟKo~$] xXҕE^T,Y󰤋J^,֗3X6C.*dѯSvet`4I>'DP_$W_?ُ*`4I$] ht'?z[c4˿߽Y~ !T#1\҅ Gd3IJR!Y3 qӹ_'sp4cSI,wIkiSw// 2@Kmi+* Rݓ ka`תCqjz>}Sp'5A>D,2?kiSҽ)s -4KmڟO){R {{{͑k&1x+9ARk/XFSGghΒ]w\-DAil B9QU߮y!OEx9iLHT" L܀zrC9Nb]wBme/g|O{v%Xxvګ̿n^xϤj|َ1`bkS:Zvi2p~,!߈]%]d@рlX:C]Kc.!egќf1]mG%-jrNyDy'U4|2yLdOF{^%֍[٧۩.f~{%cb5g=ٌG q\]eQ7Hץvh/Jh,N2nYM3 b5N%KVp y(b!tߛ/˪M=Gfu~bm^3"msOۧ)yq\ޫc[_N̯x{վ7毙%OQMYn1jq__cD߂,s؏u?zt ”L(*M"qHuKVKl[ #6^h/c&vkܵ~vBEX?^Sl.7EUr5 [-皘;8n{zG09Ưe0ڈr=d-L70gpCӰ950*h[~''k ӇEG3vt/ ~{pNB _91'mPļ5ApQBn(<ɞ\Ϥ߶ H jd =Aew%Xxvګ̿n^xϤj|َ1`bkS:Zvi2p~,!߈]%]d@рX^r+p ֤/8L79 Oj>r,ymQv/KH}' J5_ogqz16uB1`dɮuV)rbovg_rXcŠX,,hO6c&CW-dtY c?u)f ;Į.6Zfe;oaLkuŒ2\n yXq='˲jKX&{<~bm^3"m1>Nn'?m§qyme93oUޘf>QG6Yg\ݓw!{P~=n~ zc?( .S621F 6ǡz#-Y/m%7$[ذz|[zYrg zsRGBgAMA a pHYsod$IDATx^ϫ6U?ąnnW .fY$daAf!ĉF&.\E]pՍʀCdȌmS?:~~|?PU]u9]Ns_̎z>I^ @Hpd׿u/U2p ]dw !5Hp &Ca>ն \Ӻ㖺mE//B8v)kT-:Ő]J:_'Պ<=cϑ9^}5OO!)op%Cqjк֖SdbSA!QkC?%N\D(<_[\u M>|VO6!3cs 7PM:yc-?K|l_&!XÝiButQ1laһ5֑Pi#^:ƼK碬ɝwRWq:8%[C}&Ul眝tZxwњҹdZP"Z_:q" fUw.Twl2bn<$ֹө|Vτ5W&t2JΉk'l.rL=Iz>qTl5at`=P[MIe#9̾C%u#Yg=vg[tڅ!>_?,P>dH.>kұXkEj.6.tиْ ?"_g};BŲy 8Q'u': IFتk=+'CodE輶u}ǩ}I2TؐCR}O Yp"+",PZ+(y\ $_Kb.rq3@ }_:i3)lcLL' 8cg,t(O:'_ce$סcUłvʚClIdֻ1_ZH_ܾ:p5O{pݦれKОD&DDNb"F5 t.O{1Q?\nv>uHr _ icÒ?[~>.7sIF@NЧ,~UMΣt?\/W\x 1FduO6uNe+Gq46 >]+n}l#L9fH>=dA2L&}:0ђC/ILjѝ *pd":? %AD@2O <Ľ nlcdw !5Hp ]dwM3BAAAAAAA"ɒ!@> m^A2@>}N zY ;&e$Cttqz Om^A2@>}N zY ;&e$Ct_y/1;9N_g[f? |/ \d2΃7TuR >t͒ ߻'g>=ks[ܰfgq ϮsT^nHkT܃Y'8= =L jo(|fɸs-ӱAguVdtpP2ly [ 7ltb>'cpq`<)tt?IV= T*y+7J|t ?ZcqtRO eg{1'{Gf'qN@-7du{ٜ#_+kw%ژ7t._S(r6L m#wf՘bz (.h>#b?ux򛡷'YSqtNKtN$~vhM8 jbX0O98P,>KpZ%V*9:7PF'#HGk/-IN!/TqF-"OΗ>ƑZkDSV״.}^^kGWΓ!o)S! ۜS//c܎woY%q C,gf8Hk[pW6"KzIv]/ӧ9*~<=e{Nj&q}47)}oSMKU |,+Y S" OK%'eO+b&s>('J˜=&P.|~6MҽC%lT&Kz{~XvRkd|bcҲeseRӓs6x9MA pcT7yyc{N܇YtޏGy_)D)kw1,ڀdHtKerǹĐWMͯyzDCL?\"4MOؖu;ef+6NuV\5$Ca܂Z9vƥ׆@#W#XB*e&l8,YPo6 ,07UG%xz[IŖxSē}N zY ;&e$Ck6K>GwΪf //7f=%CW+O*{{A i,o. 8%C$wƟ̿pvKm%BPbpvKmo}9!s[2D?& ]k2o~LFDZ]!*d)2u0?c Q~TKPNfwU[h,9,6~Bcf%dw !5%C|4￞Nv^7^Zw_ gdrys gbdS@_pC8{*yzb~r ӋWκ2Y+,𰌍ŔA89"nǚs՘swr&v2UGH*sqPma\gFq>P`k~Ϋeic&'7ے$+zпKjNC~ ,?[#uGbd[ X!nF$x9Af%Џ#l)j|S{_|yX7Х3dIs^ jNU;|3M. hO52x߳drN>>ʦ\K[>^\LǓ_DM1(x۰Uu zz1.rvZnO'3\w2F"ԣJF G z)}@< >[kA -~\aUjǕHſK>u^c{aĒ+z\2$dGu(;FuTHld2kx?&ѦS.ebQ{Ǘ2>,npfsuތڷBδ4:;RK5S6vz# ;rPpɟ{"vPOe@z-QϯE߶lfzBm,uDBy/g)6ƶoӎ,6l_w8Xڿ~vd\FK1tv{Bc/ir1l }ì9Vh =:#2^o.{`n%C'C!#=]2Gyؑ풡6O &풡R4RN0ZGHjP 5I~ח_V[v99"nǚss}Fpz?@2T8 /b뉾-f+B U)WA|T5~=)}%3t l%Cd *R #4.Qz^ *@kN^{CC:9Z㳱ZWQg9V|1ޣ,+-ڃ*:5s\cɉd4ɠc()P_| dž*TI$vYl,7#ꗝ׿B2Ζo&Z%ڣDʿY߆.M!{ORP;uAgv~\囡lr1LF?/B7`T`[Չ[ɧTٔ`iNjxQ)zs-a}9Ǻ˥ϛ2<7y=aᏬຓ!6X%#>0 DuDXd[r7+s*U1/뭐K|916ˆ%W'c3dHɎQD{w kb@'z RxD[=a(%XHj_bKzE6onk[!gZ߈);f=d Wc2 xU4QvoZB 5*6zkѷ-ٟPE*(&r<u_+tSlm~ݷb1gg+áxM̈́P5e2Zz# K{ ~|ܘ?K>a+4ƞ "^"]2p ]dwv'~`GK~<Һj=%Cl~||zu|[}u8'Cy3?lƿ_1!8 "spɐJE%CT(!Ҙ]!Rw_V5̹Z%Kd)&C{莿g6/ rO>N{e~]]1VrK\8'G/c\5u߁V[] ~_j1 ma =oj|y3ϯ[{8~F=-P-VNJ>*5~=)}ZsWtߍsǏsqz?͐qXXo('T. ħ؟ /BOm!Id6>cJ}d\?/Ж,iW+ϓY>mFDb*f,p8MF~慻$fĿDOt4^Pik|6V*,Z{tLnќ':n-yc(`dtXb"c`s´I/>qL.cy XXvzI&׃Sŝ tȮHuäk26k#4u=~6yUh|~Ϥ[?|ߕbl'J~iƒ>9@׿BY\-PM۫K ]z?C4祠vThp%~TV*1NdAbf*w*֮]SfHAKvI&mK =פWٽWuAAĒm~ϡ[? ōǿf9'N\||cyLX#\Y{%Oг4Xv)llk} {OXg#7drNJ!6|_f$Ly4ex%$mEJG>-[K eJJudIJoWmԾo߬>}[b z^N#qB?~l#fF,پү=?Az3u2oGT'=[ +;8:(1YIf;ƈ@”rjˢXgP;Wv=A]nviUyQ=ߺ09lw@ݿI) ~񥱌_suI7hƺn\}_X:nSBl3eaެǘ=?`dJJc29~@-'eFMCKSL"eƈmr.hKɍޠP)50!g~HHl<|SHD,wWn/~ Cӱxg*UPLX~QHVXxƶU[ؘX|v(_x%*l6T}Mٹcz]!ݠjmv K2$m%C:Ev88'Ev-cdK_^*opN6K.$Ck A2歷ޚ`Ցe|QfIENDB`llama7b_autotrans_8to4_train_log.png000066400000000000000000006151621476650141400351040ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_CkptPNG  IHDRe@sRGBgAMA a pHYsodIDATx^ϋeז /G%.51 E4ta-"G1iOII4ȁXC5# m^A@ x 4QvtCN$T75;}s^{{^3 >{Zk{W1?￙?4=Oj\x=z4/o76}*xz3TWL_\m%.˫sG?+?^6ϗ}~&{co]w;[gN-Y>3[`owgp9e[^w~w>1wlo;7ȷ!\P~6?D|Ĺ/fϙ8`]%*?~8xNo N]'<؟-c ׿D/?[sp~1gZT!+\s̃\c;C&EƝ3_齘9p뼽B{ཱྀp ?eA3 ߋw pR \IAp '͛ͫ+}7ӛK뾳y=]`%WMkO=dۭGtg|pL_~N7K=ޭ(mw1S;G>}?.0~hK>SM1q򨜴7ݩ#O-fٚSsT:X7O=pzyC2x}xz N?p1#׵?R9SXp6Dµi8B)Vw8 ,"F&*+_? >*u=bjWP_e;ݣ#'"b+f~kqu5VuEl9W3f=s IN8O}KvJY6&/^+r~V=WLAx+8 W!r+b[tDžrR씸b3iLwKtj ;?pµ۱QGV1+YM8QfٛΫWmO~wԫ(}dGYҊ1V1'{<}Y`}z]xC~x~WO-߰n p\B5몵?f/VɆINa7<3)Q<~ψص9msb8Y יZµ/W}$_޶u>u<W/\7'x9eNBu/-:)ZQb"܅ه4wŤ(ŶǍOBQݖfa*ڵt=bpƔoW1};ҌJ-}3E_h=nπFXDzv,.~(瘤[Ϝ+9NgGҦ,ގ~|]^y`+g!\{Ppߘm5!߷C"gEk-'\ ۶EX-?M"6uouB{:uc!j wLHēL){c1(qXoNQ3M C;K[{wX;X= nʾHtnjܓs,׻\kC -w{02,qDMRWSRl9pE-9޲mj2^LҸOr"]:µ!޸vv:353rOXDCbK>µ3[ }˜Btnޚ3!V?Kʩňw8hx=izm⏧s oMaܖOo.n⏦S`N*\!hٙߞx [0pEN[f{txCcm{-q`Ysµݳl%̵k-:Z╁l[N_Ya>3x\Wno8xW ޮw;+ƭd*UƳR1bųx7_^noێ??vC كb<0q%=7ϴ]p8'GZ- ݹ!o}#f#"lmz\Ex3yȆw.J[1="\9 [.{N^l4&/DbN!9%"k[iW`xLkrTg/8*D(xg |ՊhW1mﻮp_ΏWqdYq6ݱ^~9+f8h=szcmXԞ9fJsijsL[uˋ?2_/~3AgQ[ 5ra [%;qYC}ķ{=kDuH5beŸOD:h3v-V6 D?}-\kUKMVPՎYQgu! QfE,B7_q7ӕE`c(TR-0 #~޺PD&9/* &gQ-f=wcµ#kaXG6+!Ӛ2)@>Gj/z}*ZW-\]K5;:c=[ u$&7Mj)ۋ'x/~ sLO8!]p}ynX` ;a[~ƿ.|_w Kywӳ=׽]\abyqMyc3ZCP9g akpzh1"{h:<~wnyFDžܖusl}\hwُB~CeYzp~ \dEQGW>j>w^(lv¶{;SoŤG8}AZgb-3B" 9l|z~#H|}OB߶P0f ?N[R.vGSOZ!vx$kO2"iZ9ZRn] y@sy&p} s Gkmݵq-Bl?y+)wY 5 'go/ {G[W|pu.w9&9r`mw̹NP찼' ŮE.kb7%t^ɚ9,!V3r-&1sЪB⇿V1D?}k;zZ-m;TnG} ~{3AR %(49\0D\JJ*D^qw>$C,j?ھbK[KڔHV{btZs{jOBlb[[7y_ӊZt{uwu꾣vb<3yeB+ϭil7NE!ĂgC)p7A8amp8_zcKCqT}GH@يw[M:h^#H{$A`eAUPej-2i꣱9",$_-z'K1v^^"Qo_Z=wn6qtXszX5YcwψTO{/z;}Ӣ3Q?fdehG֊dzTTR8O9^v;a^a?߂.>m9MQp0Cp@Ƈ{Җb`g@!#K|v[6M-Z;[4e[0?5.B[tߥ}| ReϋK5u~L?TG۵h1O{}/2:\= k'1RɾUw˺Cj>ϱcp?9<yAԒf&zkg2~!>O]|hn[(ˁ5_2>)[c_"ǰB|ÿbKb4\]Fc!/|,G~}Z{vϔg+5]Կ_kK䫾aرn~.,\/z59]V>Gj=Y#@͛ͫ+}7ӛK뾳y=]U3o/.UtÞ:C.럗>S/_ni^l'd;Q꿖P0Faj:\ op];̶V?8֬(O=b>q&&dcYw)hC|<:b j-žbbGZG1m %5%m8VRPO1/{7>/o~?}⾴ZiZ{p4~j&ɿsM3?' sy;ZsR8N'\h|g>Dd}}ݣZێvO??>z?3}]n/?rc{74o=ޤG+I /mߝ:j+[un{`?L?هe'R5Y?8Ѳ.Vuu3]Ym}x~|9/P}?w'^ȶ,<0BBTp5q[ y J*!n+>+Wо٣qr{OMܱsY{$^5#-6fd'/Y ~q:o⣎>$mA#KlKҾ[W 7ZoZl[|k$]#F:?깷s~֬i1۷H- x&z1}DµiC) w; ,Fj=~ QiC,i]Wَ~h :V;&.;Wj,6(\#kU 5,zv$X'p%\;}%,hQ n'T~q5]\[׍p o7^v߅/>[9l=4`x9LWr`n&yP;&PjDX5\_V\Q!%ZKr_ hҮ;]-j~Z-织߱r9qkQRmr_+#WLvx.}F4gQJP4[kEnXΣ׿x3iLw-:( /ob FZOofu3]Ǐ ErBH#h*[Gyp?&"MRI"ڽ_Z>򣔕߳('(np,N|[avl[

zޣYs4fs퐤[-凳}[5f@!D-2=Tz9idi%_cBL//ݖfi|z"A3ԂnӜBlɺ S.V`>FPU:e k"S֮e3>ybY;鳎AϚ<ǎ>Mj=ˬ6z Xct~C~:7wzko6׎}k~,1 r}m5],[CڡzwwM0F:`և$P|z1:ҊCp%bO2oѾ|JZuEaXl1l.6*ы1YjͮVx4R5#iծ~?jmR~FbvOȩVc=jf\o9$C7Rx8 BLkll -n.9=(D Xu}߈ ێ+`<6~g։) m~8G?MZW$uO0fTiž%~㽚@cM8)DoLU)7opj%\vDz+7*(V7gZHqjoT}M7N 911>ġ\G۵|}%e\;]Dlحq[`0XBdp5l[ԉz+ؐ|m[-BPyChLkG}U/5jDV~{>H#kkSbK|V$V:Ooey{B=|=<ΎWg`a\/B|{ugOT}M\5`闳y-t  cx~On؞P`M|܍<`Epw<"\zBrǮ%PHҡ*VBDnl;c}BFzyLyj jc)EiĖ2^%5vHwB']uҞyoӏMV_K3)vd7jL^,hn<ǎEyx\mz[ON*Vs:Yj@rR:Ԇ FC̶ސUc딢oNlãmG;,Q;4WaW vw]q=7gJkZu/vp5q}fL3so=/Άukkocy9"ح-ZjWnvޭyW7߳-_CH`t\p-ÿkkN}kFK=T^|IY>]bg{M$|<߮0o.lK] aֺS}[y[-s=MObcQH;ſa~tm+9fZ9}^}֟7)N-UGy6vOeNZY6r2*Z{(^BН"v!DҾp]q:5^ݮVv;n_C>+: ;IdIBDe`"([`ElID9Kd?.µ|z9*\Ǘs_ fQxC6Q1m.<KKpUQڛiE.~׫WSK=v}Ms [5*gf~cN޼.ز7?fNސk"kxqt_j:to[oݟeMlG:ᚠ_kZd>#u0>CM m)>m;5ߤ/@QM/kDB[wWͶlBD45hCӪ:m)[QYƴF1u${ߌr:Ys؟ ,ZRj)^WԱ8v8Z>HN$\GTy)V[-_AZԌDsEfwaoZLl0\4V`HVC~/5ZuuvشAD`F J؛Ӹʛ|^It:յ| ⅟.\{';.ro#wT YO]p⵶{kZ Xk[W {h_+|+̨92i+/#^lOuP6c֩) #쏅{x88WۑC~_㠶v=MAuMȬ 5k%>R28WyC5ߺ73S=9/&3,{j}QĖ}{_*^{dڠoV.}po?|^`LLS{u+ne#Ҟ5y5Am IHEp/ZeR맢u`hf']ߖгU(4CH4"LIlZ皟ZYbRƎž8j?Rik`Ƃv;ƱĖvjjmKmY] ;i?}v?SsLr: {?4KDOkO՞uѼZYh A;Nzc ogu|VoV؂?}p#]a#[~v`t;Zzc؄۬C!e 5QI?1p 'N 5k8)pR \U>^y3yuefztiw=U;juvium?n+tVr5[?cjK q:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘8\yTNZ|iѱs}T*\_M7;ۣ͇ӓ0Kg~pϧ'yK5ǟOgmwu:`{8a:G8W6x+`p}~wWONa?j{u38kힲIV߇p8 W!r+t[tDžr-v[Nfbm/G5ha%U#Dn\_~};~W0Jm}?rXMqډ&ͭ+p-? 07! T6>nCy|;>uixkmD^ / Y^!\?|ú)t'ޒ`먬jLD$oO-sukڛm3bsA= Yuo'LViGCxka$fm~umwr~>~-~sźWcC #vG|k1XUj̾5-~~q>7no\k`NY Z\ o*N7k'^|CwQõ,йzM]{)\`U-zC>ݡ?B;ׇֽ4f>4ׇu<$|mC;Q؈mi-~k>OSȷ=bpƔoW1}3/m]GQj8ۊ}T9 c\b[Pn[[{sw}Fc[@=SL7V̚Qv E<6r:٭\lמrr_Pel0E~5uй wse"R֧Cb/S3 i9 BLkll -n%9=(D Xu}{/\o;~c$bl3Q|#:b淎su-lztKzgZؖ}w>5eܿe{#cRcc YRcՊ`7οr^-T u˶aVԡ`M]N#msURcS wfI'xVyB('k\|9p\Nd&0o!-(A2lweSžnGm¯w_uN7z w]#G:N vql QRCm{y 9޲mj2^4߈ m1o.o\;;^JajV fS+Q* |lٶ"cZzM$ q؞c>{Ɣ5Z{ 7bz|jg\ljl\ϓDbveʦM;=fNo=_>9C osLj _NI P 1z˷򆬤[}pEf[-\o=:`uݡ1uZж{ښvqG{!<7 P3K͵k-:zb‹-{/\p}㺗-lN{Qb;237 M-|9[y<+~NÿNg_B^nnkKzָ<{.UTéƖ1Z.fV;c{eLx9|?בk 8p]=ZBnovo }M6Zk:.ZǾC6|uQnלq.-Cލɣ Sjr([G0X)^C/Ga@}O/G}rkAԌSV_d?ۺGA uzFϫb1.Gvkoq)/*ٳC` j}ay^B,Hd^-j1[f̷庌d~}Nv^|Tî=;+80w>W9Բۜ/Ͻ!cbbEo3DuH5beŸOD:Eh7p3 lblesb~/ަӶEbj{<ת֮kS> šr|vjk1@k us kȅ %wT YO]p⵶{kZvŘNR-Ujf۪iMmJF@ʹ͜l#?~ y\床'Jy1PwabS3㟡9K3:]ϾԂ:쏭k,ؘ*vulBݜy}d6a"%B}`ļP KTr!б9w\/pP/1m )<ӭX"DaPpqZX1«aO:b>]8mDk86oo{yP %,&MbyϿy#@53@܇Gx{wNܿxH-Q|ty݆Кog^pE{5k8)pR \IAp pWzzՕ>ӛӥu?Wk!8Sr_׵ܻ.oV[m۷I/.6M~?}Sho{4[kQb'G臨wm}'~\alqaі |0ڛTcsQ9ioYG:v+pUnvG'?a񗜗>_டOO.?חj?=> u&qð=uLq^mV!vyV9p}!k'`(@/Ja0Yuga6PcWMb}TLMzԮvGGNhPR1qىjlW޾' WcoUGQkתkXG+A`Nu /z.[Wb̂E}%1>:g!f?}N^?µvO(\Z5G+Al[q-f:c9O[t5NG@w-:( /ob FZϯk7q \7\ˏvL@R >ͣ[^c!}: }RzY';lg~Ϣ]!\?{<}?+'ߖoX7d[ly5bCI-}v b-c[{Ӳ-b|Yl.H򽧟$Xݛ+ VEZLK?Rs] #1kSkܖ!k㭟+ֽj8n9V+?R=ٌ~Uj̾5-~~Wno\k`NY Z\NPMډ+]Dp]~tD 5v]pF=TEo?LB;ׇֽ4f>OSȷ=bpƔoW1}3/m]GQj8ۊ}T9 c\b[Pn[[{sw}Fc[@=SL7V̚Qv E<6r:٭\lמrr_Pel0E~5`zf/e}::~)&ib?>Y/g!\{ASߘm5!߷C"gE뚠 }Wjq_M"6u"ouB{:LW>pŁ%XCja$Cm^kAjq\'-~ ܁7QѢǼ~vi ^oO[µkG^!|9}nLVڐU.(/-j1[c[[{SQP֘ߗqpI5dq^KU+ݜ㟋{P3H7->y[Q5v}s;y_7bbUKM5ٛ'F/>*r8RsEu"sA:h)DǶ֬OEM&fF!}oUF9" }9w].ܵGJ^͇?0|p,Ł%+heWc{|$R4wS/#/U6pgo1Y9=&0f<1 8pjCb!f[oVEȅrlR ©3mqxpx%v8 a#[jn27o\XBOvu@n^k1c^ h;8}g;׽nfuXjۑᘹVn-m}iڛ^Lٓl9 :m~~mW >{ae-6Yb)vVfS [dk~YJẌ$鼎\3nɄhѢzu{ =C|Cwoۅ̆OIumz\Ex+{Ȇw.J[k-^zdC_sN8䟯[Zo`x5NFq}Q:~u>Z1H4D"u?= b;U3z^=ێu9[{SKyQ9ϞSPK ۰__O "b1FD/'{} j!P:\7l7c-e$svY]uŘűu,{2*&^k98/N$\GTy)V[-_AZhDsEfwaoZLl0\4V`pVC~/5ZBᗞ;om]J/Kp_7ӕ;b<$A@(TR- 51f }Kbj߻1km𵰋^bL'*5mUGضc\I ]h[e{Gs]fNjܑ]~0c5Ʀ}` YjaV.:685'k|+Z96A42puKD\[6$tm&VL(Edؓp@;Nzk bj;۟%n;BIx8I|Xop k!+!}DrQ!ޝ22c!~K]h^!e9ٕOnp 'N 5k8)pR \U>^y3yuefztiw=U;juvium?n+tVr5[?cjK q:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘8\yTNZ|in֑ZWFLOQfg{TypzyC2x}xzN0\loG; c]9olXϜGoµ? $\{Sr" B$ 8Q^XX-6PhKbjZ#/vU=>rBNUc38M~:"\9=:>v_ sN~sqc(bspt,YY1> 1As{&:K_7ۇvq&EC@oV鶘l :Vc~+p-rLz8vԯGyvߋcb|;>uiN3gQNQ~uݮ=o7By-߈\lq3ܿe?OαDel{vkoZE̜/_/YXݛ UQҏ\iH>+:|HmeZ6xuƆ5GʏT9}]Gٷŏ/ ,ך1opµT+tSv7t5\M#tkpg;Ow+zhkăiw ӃP}nKc}u o^Cry1E׳id_ǶǍO<|[rFCkn'],k/h 1q&xH,hY`]4] ێ+W&cُ:7:!׽og&\KK|}ht6֐y-e^W%/56pgoZtҊ{S6:F9~9rr: =R`4ߔ c[ Bk' "x\u, ZPUlb Z̿zvn;.*@E. ܽ>8L18>"re+!Zei/Q\ļE\tkӓCq{uk +UTl[5N9GZ쟃Nc&Yeۊďk%l'!gXO1e|n!ZYA3-۽;$jxy;{NYOG?:gax.11bދqR:ԆFC̶x [}pL[-n=:`uݡ16Z]{vqG{!<7 P3K͵k-:zbba}g;׽nfuXjۑᘹVn-m}=$f/DA'Ysub=?F*}uv3^[rmԳ SwlͦN5v1b=FuZ_k 8p]=ZBnovo }M6k:.ZǾƛC6|uQn\Nq=]8[)"?\OQ)C|qQ8nku,:ma@}O/G}rkAԌ"5g["46AT]y\[l;b6hnM=.E<{vLA-1o~}=6/ϋB߫@-frݰ݌\q ֯.kϛjص~gucv!ycmΗې1P11_yq": LTH2b\'h xBk]X`d+- {A7}-\kUKMVPuVYPz9d\g^ܡ!& z^G1o\ I$\h#wT YO]p⵶{kZų"ŘNR-Ujf۪iMmJF@ʹ͜l#?~ y\床'Jy1PwabS3㟡9K3:]ϾԂ:쏭k,ؘ*vulBݜy}d6a]c56S6uZXbj M6Nɿ x5}piMLn]GQ=%2WV/ḵp7bWG)"Þt|5q[ӈpl+Wd,qܡJ›YLx?Fkg \ X#ˏ*- [%F 5(k)?2up k8)pR \IAp '͛ͫ+}7ӛK뾳y=]PCpLۧ侮k w3]Ym].L7Oۂyo]P_\m ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7đʣrBKwtkNVUnvG'?a񗜗>_டOO.?חj?=> u&qð=uLq^mV!vyV$\xM5w N"E(Ydb,0EEC+G? >*u=bjWP_e;ݣ#'x([_56+o_䷪(bΕs~n ׇ:>v_ sN~sqc(bspt,YY1> 1As{&ZqH[q&EC@oV鶘l ͣ[^c\Iޯe-Oclw=r[vp,N|[aϓMovmuTXFbCI-}v b-c[{Ӳ-b|Yl.H򽧟7K_iGCxka$fm~umwr~>~-~sźWcC #vG| Bٷŏ/w~b|kXqµT+tSv7t5\1!m.\Y_&\`Q mM8A(qb>D1:7!<ΘŁ"ه4wΣZB7> <|[tsݲcu(Xk7uH[))&\TÝk!|I+5Ba}Śv5~q9rr: L`4ߔ c[o{IOv[7Ecy_6Dւf{Kbw<(||_tYwp!d^P {8`Ζ`!<<-[ Ѷ*H ~"-ڥ#\?\kgGޫ}][(ŢQm`V;bUkb;юg˖m+.?6|Dp?Ik>VgLY0[~#&=gȧ*}Vuvu,': 7b1 /p^NP:u,Gzgp\+l7햶3k^[ ٓl9 :m~~mW >{ae-6Yb)vVfS [dk~Y^O7N2/gyf΋ UڣEi-zl(X }M6k:.ZǾ[C6|uQn\mf28`Ay8^Rn|HP{>[G0X)^͢vkrTg/8*D(|=e5hkZEh~d{ ]wgz-1r4lr=;ܷa^E~!bT_N @BuXnn|[8HdM5s1cǗ116sm{9r": LTH2b\'h mm-ſ k1Mp1Ӷkj{<ת -a=:лpIv{9frZ x^(TR- 51f }Kbj߻1km𵰋gE|k1ZUiӚbR>s&t9nus9vk{sGwuO̕()bk3"Ŧg?C?sgu}O=fus][sX1U(.`S 9jmÌ&UMOӳ013+U\tljpjN ܷ%,@`?Ǵ&FtN#b*b܆CĊ^ {pIoM#ZA\~uGxS޳sZ( og1o{}rp 0b>H.?8߻sBPCr7o⣫65|;[ up k8)pR \IAp '͛ͫ+}7ӛK뾳y=]`%WMkO=dۭGtg|pL_~N7K=ޭ(mw1S;G>}?.0~hK>SM1q򨜴7ݬ#s.U#'un{`?L?هe'R5YWcGk2sbݾ{]z3~qi:N$\;P^ag1 bkMb}TLMzԮvGGNPR1qىjlW޾'\WcoUGQ bY5A`Nu /z.[Wb̂E}6|Z՗B ק?"\G+Al[q-f:u9O[t5NG@Ew-:( /ob FZϯk'jL^ sȕ^XĒg!8qU>ͣ,FiE8>x|; cOj~]/kyds7>#߳'(np,N|[aḄkUk+\n[''ݾ#;LvޓϤA߄7 ׵}oZLYrV^3 (Z𰮣XcopgɾZB7>AAt[֣Ej??>~<µS]/b<ztGopͼ~u&*^7wg@a#cYZg?4#߫,9 {3-TQ}[ϔ06o>'Im7f7g u ҋm[!ʱuJ7Z̴Q6ۣ+Dg{^c~K\-mwP㜹pf,[ s-ZZh;8}gk̈)}FcwpJqvx߅Zӣ5M [F,^1[vsj>q2*Z{(^BН"7>m="\u\}lx;ݎ9jn+(U A'_Dg[\s-ҷ`=Na d@}O/G}rkAԌSVjE_䘶]W/pǫ8Go{꛷vX/\/k>f?zZ`{3+nfqL+1Gg9p R';Ns": LTH2b\'h xB3E4n ^Mcmq_a o>RyU-~sZoj!y"/BQD`HĘ|LWND*NbnAu-50=,ALξZ{DŽkG[.'N'Nm5o2a~=ϑK96z=U{ؾ2y\޼ex ,s,8kOS͏ fsߑ?k싵foKss/ZWh?F)Γкe/bE\=1g jG[s-{ܽ A*tyfC3 ;-!IfqAiB8fVIkC)],xjxg7vL71Rz5kwUvDc*bM)̓֡.㑭{jo=Q~NX?g2%5R|A3>c_l]/π%Ccp MůC??حS c}z꨸8Dl8g2VbL/5Frţ5H!-s$'[bk}͟$z^j!8=yO,uY^׮q"kk }fq7p߶?&&snkhO/~7/ۋ_Oo`ׄY`:ZW ˾% qG&5b NmOvM3V۹kT֘q-L}lm6׊R nQu"z5^?/=/ ׭ugNw.\'{\Ҧd6ĚS@O/ܿ]o]KoHZ8!&षֈ1P1p_cz!yKB3)VnӺ1ۨkڴƮECşL/"FA@io(ZSc0G]0IqbD_6&ε}[D'}$f>Kߝ#|.xE=wM^UVV|k%i_M_wwkm;J>@dwOwď:9<,ҼT{jLy<*'-tM~7H>*G8vT'?a񗜗>_டOO.?חj?=>V{+7ӕwTT!q7RJvݝAނ8q#I\Wz?aбp|tbk/uO4u䭌"LoCDϦ>6s%9k=N$\;QP^-EYdb,0FC+F? >*u=bjWP_e;ݣ#'x([_56+o_䷪(bεb NE{$X'p%\;}%,hQ n W^O?n5{`wF'37Kϝ|G}c0*(m7d^qr""ElKs$f~$5iű^ZJt4QVZ֧cROݸ:iL+g[>3OE,[8,ⲴM$(u:}ځ`<|*ϜckOGwe|= \mq&EC oV鶘l :Vcʛk1ݯ_\M? ȷ?=N@zu5]VтQқ wd^4oF2gJQ~u Y^!\?|ú)tQk8SWհFVo}߲'_~,7GMe{^?uAsϢ|=>$@;ⷈcQRip_EzVPezEOJb?CM0.)kdX:@7 r~Xz,Ibޫ Ɉ`7o[ٷ>{BYe![ּKQu~y~HFr8J^,m4rhZcgW},W?$9KZ q)\;rܦ`:5v]pF=lKkxĨݿȓ׺ds(BiwF IgɾZB7>PZ4RsTou:-gϣX=.\1_L"j8á\ оۮa>k$ξa/yE_jm7ž#{^{VᾛY m$@w th#ZaHB{(mebÿqYR 9QZHG%\@_ǩo}ȑ8Ys/r.kh=xq|k+ݚިy?[q߬ݓ>hfjK nX5[ĭGmL8p\Nd&0oѱ-(5lweS,:m-]-*jk-_~pw~oȃw_uN7z wmB}E7 "SLwkk X[e+!Zei6j~Tp>yvϧ'xّl_$05a+_}1Du˜\,Q̿`>$Fܲո>-u[~ȣhY sU}| C_-k= 5'8,[DV놠7*lHgVb=5@89kKc4(XYkyY#n˜_ۗ웷Cm)r?֞ HId.vּhI[>8p>Z1j׋ߥpUxY,v&oI P 1z˷*B.c딢oNiãmG;,Q;4WaW u[3Ue+a^k11"\ M{/\pw\֋&N[ףxDT°Uׅ庋o7ǖ@)@)@C6FfVq>nZ'$܄`1:VG,bt ?:mLqk[Ķ7k7ә7[罝0>pMV){`- _GYKi=/FM955vuW_'jQnlm3"\6k%iǣF/duUhQZ !;7DoD7}l{Dh_hoenEwz+7_sŭ 9p PC"!"-╽[ƅkKd?.µ|z9*\Ǘs_ f$PZEi{uuM5a Pfbe[1.%oŴEڏSgǴ?m'şC ?R ?Zz(^nfwaoZLl0\4V`pVC~/5ZBᗞ[ߒ-I/D/*fkӸ ZEE JQ-[`]u:Xrla brBlS?{>&\;xvG(>ZV ~,'Z"_={k*N;n )֞OVj~ l->?4W?]z/~SO2a\ܖor&y@>[ qˋ?7n[M_ bAMnF?">kůbE+o/Bk=&Ku}sl {ZD!1_޶R{XAK\ڊ#q'ƞnO&x|Ko{>c^~ZJ["h,ub:^%-ɿ xߌ8Z_D1iqf ^hvo[}?U?8x_C_i 7R-Faζ2,JbȨcYx̬xY,XVrZ[j/շFZm,uxNH}DGnPv{W[1g'#vkgͷg|>lyVlgm;o4o-&F94)?>9Igcp w45e[͙ߊ5ߘ]1m/^Vµ Yq 3mk;{-x^EՃ߆QzIt@Flj$'$ֽ^Z1 - Ns+0qoE{mѨņ"4fF-%[o_du X8譳qm:/Q%} 飲ߊ8l}9<1S YKǗBc 83&ҽĕg=feֽ0e`Rݸx8>KN_ K2'2FwNZ.%}u"ŸceoχJ }>B!>S|J4jZ@\[Sg:RFOOukgN~?s1O"ˣky8pվ5^l 40*Jzɱ>Y' b\Cc,8ĺܵQ3-XV_Spk!k 5|<6Ը>tޗQ`[KZǚpȣ]ke p \g6(\S(Áp 'N 5k8)wϯoLo^]<|=]Z}s'jq{ڳrtjSs:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘,dUNڴs=2wmG-fټF9p1>_r^>з~>=$^_>^?xf?k[%/.{WvLWQAksDµiB)Hw8 ,"F&*+N? >*u=bjWP_e;ݣ#'ܛ"b+* \GElf2{%\{6Ÿ@`p˟pmF/D5^y=Xjq;59s&EChEXlp]STt~;m5.`V\_mwcw= Z{uy ۾VfJ\7c_bg@qګ26DŽWĸwBDp'{<}ۤYL}z]+P5ggqzp Ge7v7O]=ʏVrokY}OZ}v|]o?mllXR~@8ORtՊ8x9 ӣoJ׽m["^W~uף9VxyZBrv6|VCcR}f 'E -NtBxJ"c]GJG3(|[p=NؖI oC.Qsbxok6i N?{qڍ)߮b1nWQu|f ‘B(v7kϚHދ~%⮛늶fQ]#\,k/ 1q3&(DԦ@Ad W~om"_$bl+vD9 +'М@aN"*:{2BR. Ex?N Tb>o[*e;Ģ'}o.m iKab}?nN$!?: ]}ZbW}rR:xFC΄[:px%v1V|]!qr-v>Fq-b6 n3Ue+a^k11"\ ղwpµ |yj{ŞZ~ bܹp=j{9{bSa>f: \3'8Y}XZ{"qA|Cwo[߈n[ךh>W`s*F1ݎ#})\GA7{elJ u1\-تT=Oae@}O/G}rkAԌSV V߸Vcg]]zS C Zhػ>VBv̊:+ !8 1ʉ ,5tLWNDZb1[P]Kͷ,yWz;ج}LvZ۽5|-p-Vբ5<$OU$VvrMyo9S1;m[K9չ=+] nZmpnN~tב>}N/fiGӻ~ 2}~j{gv٧ϻ_~~omi{I5& }6\]cшY fG58NO~/9/}ۿ]?\/O/z<}\1f?Dp]88'(D@/LJQ0Yug15я]!8AmQ8S*M9^CJ}e'/D'klG[ƽ3vcW]f(~>b IN8O}KvJY6>/.s4 \%•h^m UyJSo8?^~kÊKz@~iѹr-c5cjX8fvoNq"UMfc*yY#uo?:Z,ẝXԃ-\\-\ P),IX-ur;li!(^>}/D,>n|дVV)EpQݘ/EXNmCŴX-8_3a/'YiӖ;Ѿ:[q}=bRdOX{?#cyvHϫSG8 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p!3E؋"Q!ha :ůoIG'-~ ߜhl2Q)=[%L*m iKabjd]w3-T!\{׽i?eKߓRXs񖭄h3k?W"ZiG/܉yKGG~>=<ΎWg$2R|oE7k=6y_>˞}GVBN̊"\1Ƌ YtRbL.4fr/hQ̒us k?oŧAtc}Kbj߻1km𵰋^6aR b^^߯_\g z5iM|,{2 ޓ˘4ֵCĜV-LD>gSi`ON۷%,ApOhjŚN5q[ӈw 6ۼ!(#\p" -1@p 'N 5k8)pR*_O߼޼Gyz3z;D\xm, ^^W͹<_3f .L7O{L1X;Yc!Nݽoκtgw{~P o[}]͹{ϺCKقFp8p璘#}<>_wwkm;J>@dwOwď:9<,ҼT{jL2*'mڹNǶ@ڹL:;p:Qf/>_r^>з~>=$^_>^?xf?k;2ف>n?9twP|u]1l2!io )YWQ-ۊ`ϓ cưuU~w=c:~<R6{q qY6[|Yus(w#HvB0)E "cgE(*DG?v1!zGEമG^L lG_{4}{e+* \GElfdo>؍]QGw >;W_\ܷoĘm3?|!=xD>skyo߭k'o!\?{<}?+'ߖoX7Ҫ@>t>e+|ч3i?|qnȹrH~umau=7-c7+ֽcq~{ynlY|JD0c>I|Ҿ[w/2'JUj-Gw^gw7וi7i7]kS3qI%սPhρ::Ѷ/Kqcyڊ5l+g*)mT}mu,yaW&5/u%uQ;39oe9[W[U>1/vSn =/]zrYR@@;ձ_YlZ l{] EV_sl;{׍K7VNkQϞVwV k3:' j~m:pdr/:s9?d|MuǸTЬWTܼr$? t2_Y*u[Ҏں-;5smVpO}V:_[%Mz2粆`kdX{4ZwF^Gu]3.ef:q-;2Y c.]|vZ?~ul1΂>v[O{aOeV2c ul~epsOXGv-] 0S̬ˇWaݠ/.1ԝi9 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p8_0/ֽma8qE'-~ /ʶzo.m iKab3E-.\ޥtS3RG^Ǵ-X6fv뒶88[5E|tێʺ| mǬ1ۯ_\ve|w]5Cq1s ;K{_>+V͔$:a̟39.0ZkyИ[vFX{V}4h!:32٬?雟GצWe^v5g+3s\/luev|s~7o/u~mOc0ȵ7~}z|se]_}UץQkoroķu]{m>+ {|&u):qrg~k>fvZ8:~~6,,3;~QԊѽ>I_[kՙ&Us.kXs5}_‡uUsi:o;|^9^rcgZ:-?O#雋`||Dʸ1W>I Zb 1hSS [~kh lw>Xzoc;.bcg'ns8#l˃/ـXYZckcvÂwVxX>w\d0|tja9>dC΄C|6lwbVԬ3.@SwX{ge>WV{ڔ1(cPsc-6s#k+ U[zgvc]V1kz̶{s][j>jFwþS^Zܸ5c{rLfH:}*CO[cẼ]s ۋ194lɘЊGQ+޺J?H7^3/:Q_[KqNYy?0*ss{[e8FjoO|Tϓf8˵g9n֪I;_^>7bRe9 pLd V-Jkw\Q7>m="\ǵ&Z{ϕ56ʷ|ofkxk?0myGb*~~A7i/~_/pg:(~vu1\-تp:Qa€p<^ O/(ZrkUV&5l|\a8H}]~XDbS=ssw/Eh=+S]>KL˃QN|s;*bj#iIz~}"nsbb̦,&YHsx齡mE'w=N>dqik=淴w}\)lkixϳO?o=>浚H6*|yŘ%~nv8w\/nߖ KmĊIWw8VugCWxes{JlC[=O"wgۯ7~75q[ӈp$A跜vbĝ3 ~kr#\kj9?>nwkGp8c>QbkwQp9A}N/fiGӻ~ 2}~j{gv٧ϻ_~~omi{I5& }6\]cшY fG->_r^>з~>=$^_>^?xf?k;2fC.\A[c\Ik.kѳcвΎ?ngږɇZ{O= |h^gmAϵ߽gy a3a&vsl~v<󟏭DµI) p8 ,2FQ&:+G? >*u=bjWP_e;ݣ#'x([_5VX`Xzm>2(b˸7c&|n:KXZdA`p˟pmF/Dcq BϞډ r*?N-[|vu?,\絨5;Juv(^SgG<j>opݨ~-?RφoκL>*\ `іXMq;%3iLw-:jq Dk;.2;a;^3fyV;L 8 Xݡez8prTglQXyd^4B=\>o)8B~muS> p CloEyϳCȷ{+#D9>^W'?>gXE)zXb)t\{O·G?Stl]癴]Kx%s?ST23̇&nͿKNm̗ט/Ѳ]]W-Vks1LyIqIal[YHmvMԚژۖ\ 54Zy Xծ뷳D:muu>>?/[+ձ*f3~jJ9˟ӍuGl,c}Hvo#u庬c=M9Ԝp q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&`Ń)weT0K6:h]/ ɾ?b-%fpYk>KPh N?{qڍ)߮b1hZ^>uQCrA@64/Of+뒰kgouש^'nl/] *;7b:t>>WowFhưw#'M'+[XO3 ?num k*DLUn۹NZ1\蹳55>~WQ wP+XCX[` :]1P;뚩ݏޱY}\:?Os]vnuc: ~xj ΐ#kbByXo+b%Y;9/VN}|b{l֙[K]8 ڋBLLll -&)g,BiYhՄ3$m"_$bl+vD9 u&r߶p-a<`LX*a/;XZ+'|:SC5Ph]µӖp-yWG&va(K$ηb7i|{ٞ2q6ud{Ok#((ks9?5J_.g*[wl~>8Y7Ob[ا)'9퀽Dsrݱƿ\-YŶo:Q̗לU+h#}+4W[`v{] c0s<1+|m]J.5q9Jq=>Zu/k[Qg<>?}7yT̝ՍڛEܜW]1p&\7ex[^IEMYuͷ;u +pw~o[3vWF].ܵ!z1Bµ?ZāШu<qZCb9޲mA\S+FzOr"]:µ!޸vv:[Jg'=7O1?:ω|9Nr⃻~sxl [EvZvּzLs#ξQoEeg> f;#sj=\碶*&!O]Vgvm]aLu5Hc{Q-bVւj}(2qr}~nۺnL}蘹>sNv{] k}9xb]me@ϐHm]~>wGi!ƶbWIM\O糁YsnSk7c;xW$ۖ1o'IϘۚ5?9pDKC b!f¢mʱuJsyo mzKTcCr'Z|l[~dۭ 0F{a<[qe*ZtٽA{XʃNײwpµ 'ktGI>Cvܛz?@c̙3 +ZVe<ọ۰;l}E uuGswjƩ _cA9[Tٱ]]W'_ceuy#v6G*ji&mo{ٕs9Ng]c+ϧu>||9ˇq`VBN̊"\vw 3~'ߨ7"PrbnAu-5ag-ʱu(;ج}LvZ=6eUTkD2>(Nа=}p!zwo/ߑme|3_e<s4-bl a(g_IQs;~KE [6Z>ܮh>˱eM=; hY>:k=χ\k_wLTTm5r侷c l:3%M}ǬQÝӎkõ[gԖ=D.Xqjw7=sl_b%|4Gq֖6jw?,jM#_ 65(ۖuH߲N2sp;oKX%6bm%NKpdX@nQp w o \IAp 'N 5k8)wϯoLo^]<|=]Z}sp"._k?w3]Ym[9Sr_׵7Sի7fYSy?7p:GO_G>Ty8}&/|'kMv;|re>~us\yXyL&՘,dUNڴs=2wmG#fs WoNO~/9/}ۿ]?\/O/z<}\=B`u=R$l?&Z9;,cϣ֯--Ϧg%\M6`8p@!zaRA(E(,B?QT~ QblCi]Wَ~h :V;&.;WU~?X%V':^ e;2͘>|A6{$X'p%\;}%,hQ 8^X)R$ \-p?9sϡYaþsϟ;58 W!za%,VESg{\p,)Nok v6kl{7]A +.ZgEN˵c"t׌Yd^ckw^Xq4[WWᐲGyaf@:靐o|Cq'{<}Ûv|h*(uhm=o7B{Nx52;XG!n+rHE_]۫kxgʏYݺ:c&*l11-1iW f[b-aǼuܯݴu6Cr&e ae{Gwz\*_c?~URU{.֙qs>Z(k p q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&iw7bT}i]lmTC;p3>n|: z\jou:-gϣX=.\1_L" }jG殂-J<|Ā雯ltpdBsW 4lV={]g:ʏevD@m[_ʘQ`K?;c{}k؊]ȥ;}Z?[o=x"Me:Dv W59Q y6e;G5luxٚڷlݘh(YoƶۢlQq"5Z\M>CjA_߶-5M"VRo`nz]>Jt0?*\g"m @U2[iq0-YAvv`f8q.oPTwd7opj%\vDz+0s\$VmPD8|wԸg?waWÚ,ށZW~?Íίπ}kiv<2e}eszJ̼Զe^.[WxIdЖmy0F6baNkUuyQi.f9W#uPph)ȶ+ߚH.om5k1EXsx*V|̘s"5\r ѻ!⛅⠰8@AyxZhq-[ Ѷ*HJ5?*e1oJG~>=<ΎWguuAm#7zh`-Tj{K]mylΥPs8[i"Z0A_a5$9D Y> {ݸ A4@/6bv!,jѦ[0~|D:ޮ1Ɗ;$.wr](ζW1O&x+݊suPj1<&NZPZ=V\bc塩?o;8}gsVLáO/5:f|Mhqb*ؘN 5{q "c &U!y2Ͼ.3^ eszmpަĬCͿ%7¾Xк,rV۶\9& |>k{[>$>FX|}bәsu PLd V-Jkw\Q7>m="\ǵ&Z{ϕ56ʷ|ofkxk?0myG H<<-/I(#fqW3X׭Ss-ҷ`< v<4i0 \>ϧu>||91C"sS] o~^9,l[>؛0װN mj]q4DߣG[ı=z{sGߢ%n:_|5U݉i>ne 7vSc-?7Xq]3`#'XDżY-e}^ {۶"۵&[56{!:~w=מf>&vbV=:K|X)4frjVvᷞQ-[`>$6Clla brBlS?{>&\;xn<}vDžΚZXj j* Y>Ҷ-TlwjXﭛV僜ȉ[(jXimk7RL*6"QڭaELJ6Rv{]oo /?wo-|qma]~So]}#@,ί\z96~^c"rZ )OĉjXmd1԰'ns/~R < %2~;oKXEGfۈp/Gxq^bDK01m:k'V69ᾰ]49\r{ 5q[ӈ@0)qI|PoV;<1yKkXռ"mskkG Qb4ʏAރB"jnW6asճak8 \IAp 'N 5k7o7QLo~.ξtj8/^5{ߨCpLۧέjo/.>rtj[!mi/mx ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7d!rҦ둹tl;1 ܬUNO~/9/}ۿ]?\/O/z<}\/G1-$"n)|}L?n+-ZyG;~Mp=\o7'(D@/LJQ0Yug15я]!8AmQ8S*M9^CJ}e'/D'klG[ƽ3vcW]8d߃ 0':?=-+1fAی"_>ǡ]#s \:\߁:<\o}~oG 0ę|T -a*:c9O)vJ\-^gӶXc r ZuXq ~<]/-:wr]e,vvf"Zx Vtx}ˏoŷ`z' ޘ}iE!?cۂI}RzY']A!\?{<}?+'ߖoX7a=_'K|Y#!H!ۍs{E2VGNbov}bu;Mڧ;/z]+~y{r(n?Zcp5&ߢ_/=Se0~]ޱYZ=-U̲g%_wu͌WͶwZp q)\;r@T wEu#G {ۮ |ֽ됯GsjvWCcRp sݶp&C$PHQaݛ`,:>ه4w}lKk }x)<, VX׭?>~<µS]/b𵠄vd*2۫Da7_KI[|}2[I@K%a/tw,Y˦mt\Ϟj{=jF[L~{BKXGbW׶\sK%q}q|{.}+9!xI3ﯨymVpc=!㸠m,f~mr-`qµ 8Vr}[M"jS X R gH@- ۶EXUI !V A\GsGLm: 1ALa=K({!{.Dx?NDo;W3 0lF@ߞk֎(VcB>f3%3EIR۵ #5n9#lq0xESu/3_l1kLd~vΖDCo|1~1onNe[c#~~1scuT2wjAp+?şe0uPph)ȶ+ߚH.om5k1EXsx*V|̘s"5\r ѻŠT5\!)ƒgC)m+O~+Ϟj{5rݵƦZ뼴[>o~7561fߓ뉤[sM!&1ϪD?b;j9ceềOMӃ]:__-Dp757T!z aQ6:px%v1V|]!qr-v>Fq-y2[эV!Ct^)Y9_rq&êƘ*Z|98fAw-Ou~nofJag#1Ң]hOcClTLcӏY5םgJ>.՚P[Gk͢Mo-c*e1+bv;#.¶NԂ.)6q*vjܓ_eo[[.k+Z1`;7rby!wDXYV)V>ʹ6ܷvNd}{ *nKbxtl/HkLRC)_xy\ϊU5 3$; por?n-DDVہ*BS>t|w~S #5Am p]1m9n>N.[2]`=NvcK`.hڭ#/(7GK!'[>}3ƒ~W`svzjGNG5q[ӈ฻|9p6Oq_^,:A h=a߰L׺0E.89։}#p@{ByA> Gik]F:c`dNnzwpk@Dߗgn|wv~0 ib3G$-"o`&_[iX"ojQnh@VR|[vkɽJZ5nlKMlR{= 5@wY'}b|${^q=}$!޾Bbx:֣,l|f1-ƩJ"^hƾ!nir}l7׉rRw)F[5cqDLw13uyf8 5F?^ʿ%o+u%7VGś1JDXi eCh ~max.By¶%ZZӭ5ZQ "}Z_#8?8oe]ܛV|enS.ż5c(F(_0W6&Mk[ ¥_߾UkMKkr564#p@83n0v^1,N)nr\<"ʭcoj-ʍ毵VZ>Uݗ}t?fkfߝKFbj-Oo[f8=k;;Dz[Y" r[IN$[/ɷdǒg`K$Ihafz[r nWZ+͢b"[C.'9Z9('Ti7^|R@m -3a,=2}#p?A|~=~fznҺ{^Oןv8C.Ӿ=ge.{zgqE۩-+\@08-:"e) |ޟw isxQ-1hמ 2Z ӺH"\X %R[j,kqt1bjF--k9FՎ=}ڃ; ?.~:٧3~;Y{,m_hzֶOOmL}.yOܯ؞â-`zO7Ƥwb8 C Эdcen WSyl>_r^>з~>=$^_>^?xf?k[%cte?/^A9v[喯#d!$$\vcjMĞ--DAr5\#i1Ӟ-ŷ$-medFrA2Ʋղk15V^Tei_e?}OC~ kmҳiO6zi;D9pDZ!zaR Aĝ(.΢? Q5ZBtTy"pZ#/wU=>rBJ#5תwlG¯sj|b1^?1n$X'p%\;}%,hQ n+^! kUůw7G1őEزſEKEk>#nZ;(lrRx<2Wz9m;>ɯ&q !>.v|:>{531yM|w֊fY{>?G+l[q-:c9Mk8?^t~kP@#L|6_=Y^61طvݛo r?wn'N= ĉ+›>ͣfiENoz'Iؑ}RzY';lg~b78B~muSۡ&V=>f-F]:r5/mҮ+ r_qLkn?۾wύg#ˇՎR\qvY+?\wװ(EE ꋃY(SGZ-QNpj%/-i!D-ޏRc|Gm/_-4|HGd#p8KZ q g>fqxQ4ͽg^4KD<̎;vu}|םmµϻvZÒarؽk_$Q(*Ewmb`,sc*TɷUIJ>Y ee,/bL_^{VкMZӷ?yqoW`KH ^Rƹ.׏{Z,믴B(ϲBZ iIV!ֵ%Nkr&ۗS]G7oB > Fѷ7"w m{R !BС97!g!5'KOJubQrQNƒ*}D9Ǒ't"^[=5>Oy(qQoܖb?'m eKzꔱcx[UBfA\ |emUbJȚRW̜on/\n; 6ƛbSȱuAD6>\tΥI,D9 Z-Ha( 9Jd+d-׸DD$c[ ^Y;K[fߞk7,V/mJ' Y צ#q8 -8nFkI)^mo-~j]{QmG%{ _JDLJ ձ+AA݃a!۫+I,E"Z E沾E-c6"F#=Q:-QEi1#Ǒ`Ǜۘ붼4;>uXe`5^rp8pT-HQU)9KU*rE\0{ pm܋Οy( 1D Ks߯͟ ./VB"W]zjqE Cb ^ERY=[Qm*U(eFG$ڥ#\N/5ّbI먎 v>µm__a&ehA&b5XlIΞ"}8sod55MٗE4ŢZܗE~|c]樔"EGrͽBK+O,X#{p9pDCGdaDґ7o1N+(mw?ۣɇvn 9D1J/6޸^D$u-vw,X.BY 7?`X߸>W"Y_^+*h Ĺj5w/ɬ~m\+oBI:m[c+ [o?},$xEXۂAZ\,8 #\9ۮi%Rh-j!T_QkXG> K~Z9 I׀ۖ1Fc`غsɄPf(mqѺ'W䷱"i_G##󱼙݋[0nJNm'G%(bŽ;[ I2% 5y:}:*\g}3'$鯏D9CUG؜L}o="r_oϗh~m\+Gp_g[Vُy-@?3^g)'c uxz_ӌko 1"V|M6(7*u*qq_< ׄ盇'F )uhV3 ʣ`܇5b Y uu)6&[Su֯7D eweoQL-/ dX-m]zv{>%'\rA~Eg!`I<*,eİ3Meo\]O$`XE@*헨c|K, eEZETۅ$&Q-fw1H⵵{kp.<ώitoCyv-\ *#eL >5 ~e/v_0.9o)웑e:SSSEg%6'c (3ɷ.܈x,r}(gw;0"\r춮s<|kbgi]oFm x"W )S FB-F5㔷!ZpWp @pR \8)N k'5rͳ >p9]x1jp\<^/+%.ҩs o>ȉ$BK.@x_Ng~3=CV޽n!\gK sϥؕij܊ԗ[jp=?=K)Ro^4rK(iE#8HPFN?kE-׭xi>sk qjc+^9V59Y++J4mپR:o>Jȗ}磵.m^毷\'k|o\E7 P1)!-spo^g.\ /WQsNc $8w۬}eO(۩>&~ sA)|jXX1p?o:N,\v/fpk[H5q]2#BۢN%I1-5$b7&@!oPP  u;GFU;bpE(f}ߌ^bò}g3l'/PpH8V p+HL%V˜Ӓm˂AƵݡ1\?}epwc r\鈚 &.}H13aGẞC f j[l-~kʶ"َOoe5yB,, qmQ>(\SY%_׋P͟9=4n(Eϛ0%>8 GXU-}oxz\-;&l#:U|0Marm=Q\gJi NQ@fIp;t>נF2 i58<WSC>!@pR \8)N k'5rͳ tyr=x"s3];.5^2N$\'oHg_:vr>:u69s>"t=S@ *%0bzKկ7w顭*V{5@D=8,&o|f)_Z[Θ8?}Oĵ'??.۬Jt3<%&Tw)q6e[U>+q%JkŜ-}'\롈DX)jv@ /uMŖ뷎%꘭t6po.{RVrmFmGe[N@D5p @pR \8)N k'5Ip ]ݫŋK|'Ӌ/ :gnwM9 pfKݯzDGmψ'~qNjمS8`Tziq:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u qo*oͅV~g It!oL?W_j> .gBq|i?WeBP{±c v37&/pMsuX^N$\H+H DE"*RED̂o$*r bކ˾e\X lg_?z4IbUJ?~m)[EWe~OxG+I`.uO?*{Ϧ6+f?^ \C>rqz \sL*`ot[%ۣ"׏@GL3i,l͢k'zqI~P< ':[磍7o*ͳKulɟ_.o`yudc[MvῬSٯu,OrweQN1~qݮp?`x7׏Va?M섮ׇ.+╬S֏o$[. =h-뺻5I"~roG*'4?r^~܌lgsm0sly"ZyFD/KQǼ\iWggrf{fm36nɟIP iO\ |$h~{,Ydug۱pBdpq>z\vZ"tx:h.wmy94w?A$Ňw/Ŷ\VƒPr؞u(6_ki˫ {oWŗ7׵mlǖplBfA\ |emUbJȚH ܾp'oM!w[is.I%Nڊ8l!neYI䁶#HT5;V<<z7=mgQD.mj~{߰&X*T!J ]^xf1ui-]r,ޯxUxq_ٸ3qLƪPۮe߹Y3lj{@|񓱣>\wn󭎿msJ;1\N P)9KU*rlEOu]v0בA< =?omWP -jc:0=9uñ;sJT0֍zY@{Pg޳Ue!U(i##|n?CqMv䵘^vDe.5%}L)36,nm~%bk!s:r#1Kxbfm\S~8>E|9ir;o3uEy:溓Ùp TN#JM"[uۘZMbc}qxCk lf+lcc^.P_(?w[Bۃ|nk;U[=>N_Y55hQփ%}LO)5N_~ elƵ=^̖6j{5뷏,5^B{>\kS\uXm޵ʶ3p|N&\5cEi+="wD}߈n:?kOsSlfv/nm{qPC{:?S ;&T"K:_>W} ܱHߒ+h|! \pn_qT87 '$<鯏D9uL۽q-zR'*Wvjnubq? i9mϋ} |Q|lۘ,~vbBmY7ב?n3r縔b,t=ǮeҚi3]o8mXcLc溗âN;g}lyq^:S 0HEH1ޢ~,&tS]YvbK= 3]zv{>%'\rA~Exg;.A6. lѦ|.K:$|xʡ헨c|K,oL=d۸aIbBl{.$^[^:h\Ŧij f^?i]^DAn|vmW=#Ϛ=gI_fcRP܊eJv1reY-s'\R[/{yq;3/n,oW6S]}|)ce5TNO 8 kp)oMCpp2g @pR \8)N k'5ʻW͋Ӌ~(O_\MuLWrr)p})wOc6>f1^byv9 /'^=s=]zun)in;ufݎpeƌ s0~pi[mgSN|~-/wԱI ôv8-#zw^>o늹1iKx{ec_\ȳymz4]|;o:wD{>\ͨlE7U^Hۮ{oq;al/qaUVM>fʛ1]{Pyn.P;yи!ޘӯ|\SW0};ʎEzخ矮9C =y`G_@r̵۸F=[9 ʥSt9Lw[9܋oYv}2GwSހ,ۜsY۾ϧ1'JSi 痘٩]^·ۼʱsz^?s^sX}kcl%GYrO$\H+H DE"*RED̂o$*r bކ˾e\X lg_?z4IbUJ?~m)[EWe~Oxё7I`] ϟpG͊E~ t3Ń9=,ڗc<;r{8t%9;Xn0XCrIDATQ%?9XOc͗[#%jn8yh9ùp~ϋin)ßJ~zWu3> oDp|n~A8zIGflHyV~ɦʷ {pS5FʷMvgXs1a{ZaDG7gq-’Qfb&mf5]]$_(tiF˛V7^u=]&v2͒fʍ hsXZ:وԃŌllN)gֱhlk߼5F֦[5٥qȸ\ך^܏Y~VuܪuIcc m,~g\CjK}ϹS~$}oG̚K,_˳3'v>C`V1iNZ_5ϩz=)gST-_v5?3=ګ?ݿעqvf~ַ* r uor̛WNXULۙ󮵈صce]cVbآ%cwy5n=mxuLeAt[s?y"Iƃ1u|핡o9g ֋[4'c)'s\1P9 A}qֆdt^y_k6 3ٮr[g;fqe˳My<0&kQ[~rnµ>3 8 $YUos}%; l;u޵CZl7G ]K:oxyeEu&6u_l ^5sú&m᭏ꜣ>\m晵mr!V|`KvBW^b`k"y7o3RNJO[&llMVi ,PӞp-5ꥭW Gb}:_9/yFE;`D67jUYLc)*ۮj8ǬG?ͫk?gmyΞ!z :{PnF9| r:ۑ>_Vm6eh>: M4חɞkҎaڪyVNc}hkkѺh%)~u右.ָzv+4*89X+{"m~'ZK.: Z^Bto.U")"=ua}ø#\Gk+^p]C/ȷmo!\Pk6 @ sH ^"X*ހ@y LXe N!mD~O*]:5o\y-fq-s=ǔIsugDnn LlSWh9U:~DycY9VYw ٗ,7ȊkNM6\prʃzCyzK|Lc掋juYEu='eNoXwn3fL3qGbθ-ȱ)jҗ޸~'mM^Wb}Itz\.Tv|7>3S.x0θ|}D6=͘,C;sKIvj Ʊ ilׁxlŬ>Lgb&+ٚj<ol/t&`|q>޽˻&zMGj#V{ۇ|lp=n;9bÜL}6N<^5li&qpc~)x뭶yVv,,Gwpх$:) ˎ$[E$vZ1Gi mO>tvm6 v=>Fm1 a@nZ{r6ok.[Z%pSkVjL33͘{(N+V._NDuuZwJ^R;guxl{d}*&flŬAߛ7f;ar@1xf| <,%#4zftݯxrKRqbuڟ^y~'?d'§ԇb3֎YڇymU1{^H.ν3.m+yM޸16=_Fu+g\uVz5q{cl!i,Ʌqq(vl晱퍳g\l ޢoODuH-m]]GIT d4[_T_Xm (9纗 :f+«MZ{J/RysimUqkMޘl}`Seև *ӾRY֘YlW1ѾعH;~xD9܏Y{wWd[IvK79Ŗ1rjl`m~!V I{cH+WZJD+Ï Zoj%96|r.ƌ(qqW9_MWe_z_ZۚFzkRə*v119gČ|tAʘ:fDgr0{~f6lkDٮE,vxrrtq1m{hqsLWc3|Xs:ǚe~rݙ|3K> kFet^Rp :1lv^<#jIDygr)ٯn{78(,n:N?G Paas9|7]v 6hsҘqkL>/ Ι;ltyv?o}ߘgkW+_p i$(obp:Cq{2 ~3Vo 0}y ٙ6@1i_P8L~ஃK`M"wwimSm~YsDX8KA|ݓk5Ip @pR \8)U޽n^^*q=b!+n}& 6V~Mۖ_bޱ3C휼].G\ ן~T\M?mV,bQ8 -"_fWp` cW7b0mQ~7b&~oo ?,f1A  y>엙 e{n΋!L&Qzo%zs>ȹrX[]۲9;3W?.]6oYo%"|l|)<,Cq5 B \k~:EB._wEUM"k2^a_+M)v1jۇک+Vɣ~0"DeO8r$&]LrX&35n' џ-|#VHw5~vi ,PӞp-5ꥭW GkW Ŀp͵g r|W[v\?3fE D"3> }̇7O ֞c !qr: Z,7xA]oYYt,;;ud{OcK{8 ·}{µ3MXp7?3âr\w]4) @l*o3V;Bœ7B:5o\y-fq A-<.t_bTtu99V~DuhlmyXoI.~2)Lϰ]ߙޞB:cO ItDHo\s]%,&t޺mL-`&Ԋ#BgL[h{alѦ'^7m#MXR S5&]kSW]lܱc8ZN_Yk?Zuk^/B)1c5+kO)2¶ ;:j,}ooO1F̼p8N&\5cEi+m6R}#޶G8}a6룼ǭ%\E>rn_㊽k]Z-haŗ"dcF(I*PPElU)l< \pn_q lgVBW7ε[ W:gӸlޙ^o.~̬0c|*k7ml;(>#lT"_:m-Ѿ1Δy&N$\gԢE]GIT >hbK= K[omkW{ =u.0O۵y-Y ^48S򹴻.IxKK)B헨c|Kh#*EHbBl{.$^[ǦOYNs96Y.u/Tlk76g*R>cPGN2=>/"lZc,u9}:$溟+se{[(cq+eQb;#Bm)FGњ@ p( \׷}%wu-kBgW^oID<:!k@E\$>|-5*zBtNq9*?^coeZb G"p>gzp >ʻW͋Ӌ~(O_\MuLWrΒͳ y?)篟xe}~uco{ų̻9Vh7D3W~[l,&måxNµ]"m|Q [P*PoZ"h:o{o{sbQ׊ˇ$ʁ45s$2F9pG'ߙϼpzG_*õMߌfj~]kz[͏No~VekI:ޅq?_?Ο/>>5mӷ#_\ O[- ׼'\O^؍y ]?ni_~_fzيͮXtb?o-\/«[Eߌ G'VC&":n\]Bes筱Mа=_es"C!(*E$(.bu/!/lᶵ b'!"p#R&׏MojalcB, m˯1tBr4ޛ#hswODI`.uO?*{Ϧ6+f?s$8/n1?{{|IԵ}K](olmގ-\KQ_EGX rFEe-}cp]!yՏdz [ε}\^ϝſ-ڒKݯWy pmILj <6EI{BxW2= ; (jC$kݟw5F0<ƈpdAD,>QNJ R"jg!x`E#DP.YɢI.+cIu}U豎IW]T:-۟Ho\{__&}Io29^zXAu/\{vqG"Zq [T[D[\ T A\8 M!*."y"!lB*&5 ֑Ju߈VmgߍVϛbSȱ (Vm`8pb4v+q6)'nK΢8Z- {F(Xe[q;)Do-6,S~kiOÚbVثca#9{Ug;u.$;6\,Ngg\0rm=5im-Avdߩe*߃-1ʳ=Fbgky֜1*Æ<|m1srDwG?VEݗ$Vک q.qX"$.cU@.'"'|K4 <|ee-QUap]rsY?^ 8N.\'U ETUBtoZU)~V[wE,7;u>pmΟC/&z,\Kls;o }|>g{Z׻VRqOD`Z Ll*o3VB!-H췊kp:=8dG^y\f^k-撯:L@_*#LjѢdoPy]}ᮎ[~ kFdL$k|T|]8)#fCy":N*\mWO+fA͘99"(.\o圅nzξ"|p3rZqr'~sE`=d;Ţ-A⻰klὠƐlY_>ֺ\2o%'\r{Bsɛ(TN#:$UblԢv~Goagn͘8܇.~D]Z- H<Eµ w,XD(xӲwp5>o\<忿,*,&{s9ٸ|>ն͵fu977 Sٖk־_p֗GM-,qN/bh> π@M~U&V{kk\C։Y3G;1srDևpan|miMbO!\{.z}Ld%V˸r> u 8 XIb A7V.{SycěwG{]_uB[5ɄPf(m^[¶^{oFtӧmQq\{H6f:6vVq X+PTo"dȆ'U`Z;[-(BQ·ȗ0 \sӧu{ֺ}Q0sIˆgF9\/٦7m;96[yfG-y{{Y?"IMa]+ڗOP԰k~jPkr<#Nj-Ͻͳs'f'%7cmIẼ-Ej]J$t+u^uJ$޶KZN"}V5E2V{ָcJI ؤҊ)ͻ[i^x~2j3&'/^9Z(؉p/;D͉,ZHA4ޢ{UVhDs(Ŗ;`EV (}m|1 (9mݷγ~P XIx(BaF*Q7xӵxN(_%Hc|Kh&ފ=,ILVZTc5kkعYӉREb?ۖgj+ yΘx[}Nw;/ܵBuX sFD u?z]uk@g3:[.mVdZ%t9$[*ɲ&*~A> VDŢWwE=p ZײH<<)l˼]ϑ$"ˉV;9߸ }[oD-KGp6~PZ?{{nCwH<!IQ<u3žhd Vl",ߥ/GB)QUKbn!QmJ42~6X-9>׉ֆ+Sޚh 8K ۇx^!9pmi h1++ԫ>elpm9pcA~ٗNxy>zb*tǾjڏI$Pu9[r <hkQ B_C.m?c7ݲۦ-1$M V zX˱`a01"nik<g_Dh ŖA)2ח:\22i^b?p@{ .I]\=p]D ]&l#Xoіx7"&sp="1}]22"$jbۂ 4>;ףb$Gk".xݢV4W>!K$.Z+2.ba~խcE"RZSK#o w< IbԷ@ .s&'o=Z%lOaT@.8y[Ko2hmtp@bBXT&Z^X(x#R۰H0֩}9v /69_l+ p@B`[UU nRh>p҂Gzݕ]`p+̩+vXEk@c +(`#߄>Pk'5ww//Q\O/.d5xv)+k.eͳ P_e?l}1]}@p, ti;q:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u cq65ybzKe|*n7''u5%6/ޘӯ|\SW0};>/ !\kw}~aZ z/ I91kq"A!Ph($.bu/!/"b#QnM$6D^}T.zBjWd;ѣMmW} ܱHߒP iHJPkttTZ8* rBSG4cn2^ < ^N˼wps}v@w͸J"\q짍iԧZܶW;x / vܻ`mm\v6i{lO>'@jQ¢#$zvN&mKLuQ_%nlH47(+{؎bZlGa \ecQ# µfcQrBuVuˆm/!:χx> ,‚hS[>v%ۃx>rxo%X"F"*uKߨb;spM$ڽ58x";Y{}\7e u"m_ӿl{o]i\bEZױ1_RowΞSW(Fs϶Y=kTkX=mo\pX Qph \;Nyc5wN k'5Ip @pR \8){5ݼx1x~闏zzt]'t)ų^WOLYYt).n]e*Ag gc_r^w{1?.yLGEYZ=`tM|gz?۶Q~ޫײ?z4}3*mwxuoW7?Ҷ:}[NuK\?|X~O&aLxnmk9ZmdԘM9pT:8`=Kͧ?>?_?Ο/>>5mӷ# .ZdvX]1fw_f~}v ꗆ$Hµ%-ayox/yAωk+@,Y\PYDEnA5yQ ]awG7IH0J1_Ӷ֘K:cf|/:m O΀$ E.pOuO#f"g?uaHq : ųߤ M{_08g|_@P}{ `3Fx,"l%GEԺ㛁ꏊgӶYbw=Ot añ~omu">xr^rīg{|aU~;%|5zov~/^):';Q~/c]!\"No~ú)tR Zȥre]^L=)ہcp.¯#f*?V.{:Gv0KZ˸WɅdmK_UiLs|%fa&JHBTYu_l~}Sz6Wd;߇uWF6ȟIP ǖ("x(j[ moW숬1ζcZ];pM6-{>%vO>&A!E޵Ev{p7-&Jž rYKCAiL_R{嵘p{7o45߮=EBQ"e^7Bhm{Xkב`;ْ]=ۋ)]#Ȱ7⫍{cǶ~잟U'*׻7ۙ{Ƣrwb1~u}UY,h QqYTe}QmqsFȚH ܾp ySl 9#`8pb4v -{81ǵ+p(cC{&]P\U|ٞɃyџ-|#VJjNY淧=Z k"K[a-3^0Qkmዚ>==_ Nݲ']mily bb\5I6xo?ǣ9aFd|,1Ǻ-cͽw-N2ص y[};_N.\'aP ҋX(F$x$N:RU[YH_{ow}lnfa7.׾i!_؏?8}%r`#>xgkՖ"ik՟}0,\o\S Su)\w=ҺD sDOwvnU}a;#u*9'/;Vuf"m[ op+-oKİo<7њ]ۋݒƟ% 7'Cњzmy ۶C~{M="\Gq#qۘNj4֏?wEtMS= 9A9;C%m[ElU)tK_D845y:}:*\g}3'$Qcm\&+K'inYWuL̸E4SsnۈQ;]6ȶR.qzӸt[XiJ9-|/7[fyx{qc;'eMGvأѱ]Nc IN$\gԢE]G ITMږꢾJ .(Ŗ{2,&QEݎ0 o㛍YG =[-bLU4/;t+YE򹴻.`*C{C9U߂p,o |Iƒl=,ILVZTc5kk\y a#jnr֞m_5MÄcc/꾯+LiӊĚ^c_c}9.߶>Ɇƾʅ4n(Fs϶Y=kT=[:/^ڈc~}j\\{ڹs{e昶{wk p{Ԣx,@w6Dk%7[[o \p h ^V \8)N k'5Ip ]ݫŋK|'Ӌ/ :gnwM9`|byv=.kZ~p4;D{ͺ{8$w='}x?}پ'N֣Lgyy8#/{UZGoFe3o.ڿ5RGv]7{ic{o5S$IXxMع#Rmۖ `|ޘӯ|\SW0};ʎ<_ٔg+D]LW_&!\}g)6gXp}8p}0M{l=g~ 'I%RLb"VgqB" f3OboCeGE2G,vM=$!*uD`~Mۖ_[c.錙E-qfs4w>H4;'t)I`.uO?*{Ϧ6+f?sl%נ ]8DAkkp&_ވU´/[e[M䨈Zc|3P}6xmf5]"%;4χcj ێD}hHWed5^OW(o]!Ғc[vա,?T_gi2Ɗd#[eyp[Y_WX:$ELuv\b'L/7B1Hb3# 2QYg-D5Rf c۶l֦ۙ5i5n5skӮ.ɮ<ۦElyUx̪<H:6m1|Ov.Em}=v?!_f}yͮ{m[e5F=YnxҜ_Ic9^NnAY0y)/>}>|ù%ՑIXcZK}?`Zӷ?yqoW`K9ϔ\Ws`+32oJ]iggZUc"-;.wCd;}_ [q󠄓ߣDb̶qؗhi\vgWٳo:ffy|LY1>_hǬ\g~"FlK//7ARbeqci\y}%և,}EHH_.K?L_l_{Jw\.x1tlY,h pYTe}Q]qsFȚHL:oD+_+M)r6Y0gѻlǟBx{έ`;0PwB$kpppe{n'Emu0jNY淧=Z k"K[a U8"@U@uM~Uq^9ȳRGP q}ה9n #?=kAZ:V ?=iTׯ1K)%&9 9"GlcEPSǘe3_oc[_7ڏh|.ٖݳڔgh}d\g\?`?uOlcbU6ql/0Ʌ$jAJhtߜNg[M⤪#P*xu<$vv p-ǖp}AOdžxC-k+TWµ&tH+hqH*zXPyH>xׇ̾Al*o3VB/8VH7>CqMv䵘Esdgv|Ȣ)uyW[<w݂g_m,xk.ך(l>'tߢ/b covQ:olL뱭\{yƳ[MgZiG/Z1+2uXrv0"I_'ӊ.~l}'أػұ~.yњ[Kkcmox 8LmeY-Qq?V6u7l1[-݆X Zog=ڞ|&Fs(jCĿ@~okߴ /lͷ#S!w<8L9*cZ%ik՟}0,\߸VuDhN{sR40uui D/a٪oj+I+=T>[ؐcD4]gQ~8~J#Wo{y)0'Cњzmy ۶C~{M="\Gq#qۘNjq} ܾTqCr>-:tи:l&sշ- ;ٖC1 \sӧu{ֺ}q1s\sf"ѡ[liK 3msr) "-s,}Թ0<]k:odJg2jǥfdu;7ߤ}ծM_}ʘ`ͳÜ}Slm9UntqDL٭cF pоןx{_ֲ8,ul[xdNu`{|TL6J$N78,p:s3]kgų^O^:^yB.n]e55ntM|gz?۶Q~ޫײ?z4}3*mwxuoW7?Ҷ:}[NuK\?|X~O&aLxnmk9Zm9'u>uޘӯ|\SW0};#E^'[;=:Cp?)/_`p]p ;H&R,J2XEHP_S? }˸v7ξ~hzVcI5m[~m3fޛޚplwr ,\O"XH'\lhbmf:Hc[@ڽdr ;垲+98$g{޺jvl _3'mw?x :],k+ gT%Jږ$r~v;"k̀pXy!\ lDca_}U,HxE.b#za?YE\Va(}^|(}Ree,V۬E+r1[%\1}YMv-RĖzzY)ʺk3X?rM]g!2,!,گ֬ku} y5 ?أikj>^ p =_wg!\)DE@dQ1EB._Eb*&5 ֑7}zwwnrlq9U,3]%B C"؏M!N /)^8ZU?_Nd{n'џ-|#Qbbh ,PӞp-5ꥭW I"cc-FXl] _Tk"jj#0C>kŵ).F~ <>n_O7oB̵3??fxM?^uAW^+h^]ǯo_y-/${_ܩ6+S܇-Ke,_j '"*}s: m7 BU?E]9z8 } p|'\˱=?omq_ӱsŴ'\k޵,B%q:^~UpmVvCd~B9Pӿ\\|cΗ>⫧+aWTяµGO-rK&J;uaнp]oΉ8'N&\5cEi+m6R{w7>m{DGܷ1/zk; בcsb(8"+5}r&Nc\`\ un:?ʉ,ZHA4ޢSIS]xBe?$ݕտ=lG1-Tߣ0<#(}m|1 (9΅ 3vk!ZBl '"|.XDS>v%C~>" ~oA8ȷ" pVĝ1FߩDŽk"0Fy{Jk,qgJovO-㕨=_WTs+Yq l+~ݝ"Qھٯdh:ؖ:1c>ٺE gg3ߙ:!aEfèC?;z?e<&^ B bsiuICkW sOΝk /%F^K'>k[}΁pmݟ?6z=&,u"{ׁsk,6xz*7 _ E뤱[!_.q`'-T-\o00bKEzY mS )!S ~_ $Jjp4ʓWӅw37ջKͺ&]tM~;{ϼn]B\}% nz??Ho WكsE`W[uKyµQp'}쑷Ss:[O3ym[(xUUڇk=Զ;hJ٫ӛiu>|-n:%>JקLy0&u cq65ybzKe|NqT:LwI$:oL?W_j> .gBq|i?WeVkNI;,gnWL:L0'q^6%\$ֱV~ oc_^!B=4oS{ ײ^}Zǫ;*\_^m7ŬSq"J!Ph($f.bu/!/b>#AnM$6D^}T.zBjWd;ѣMZm1/Ō HFV77 M؞'*u7%kdp-YbM}qZpwY" "!vqFȚHLMъp ySl 9ŪMG.y@ Ǝ?p{!sG0;w>`{`ϸBlJ_O[`ls尟ES~kiOÚbVثopk,H: xkG^puʵd6.|[-/wmʘeN6amϞ޸F`I\P]ݟ,pɏo9~vCqMv䵘 kuDO9[zU9lʹJk p[;ng r[l&??tam9WaU k:Km{DGܷ1/zk; בcsĽ~W@Q O0I->ݱHߒj< "KQQ:=kݾp.1p>Ј4(2Ǵ˸("dg}Lerբ׋D̈<^=wBڟkñs{rWT6>ڏ=5O,뜯>]So?7θfZqeӕgm~hcQ9̇(Oؾͼkaw랰"V]D}bf{h[sUƮ쬜^pL۽qul~o n(Ua8඀p 8mp Ufpop.7!Zp 97WRW89N k'5Ip जT8}W'oMʩsWc_LWze/X; ngn{5ݼ*qĩӢ~闏zztᕝ3<+;\Cɿrj<#^gR^WG>I?'?#s 1o7?f3(_?=}1mIwk O>G=|z< kΐ|&|W;v`ss?'q8tOo;p]?Et]?D[aP^8PUA\/n]>Rt>6ͤ>[ 6󗿘^~(W7.LSIQ铪5.we?yߟ(jxufLͱ`3ݶmj=q#IXX~XY?bozpG'ߙϼ)Oޚ,uzkvoGoZ [Neƴ`ƷPqsk>l cq617Ql<1tְҹ-{N? ʾ ״Wr`1}h=+;k>D);Sͮ?n!+3v}@~MZ{i']}г<ᘸ3fm>x \;e3;;g*\sg)~!\v9PW(;&\{w\i;Vˇz0`Cy_-!@MdWzؑ6?v>qL7˦6/XlVc0C?Ct:[ҁ?z}zg[շՅνH_}bQy/m̝<Ġ=ډُ%`?3k󹲷63]3t_LyOP9}jމIsهwec(Nȉ$חcGq=qp}cs~3ߙ;3o?a^s:򝟛n޵stOo;C:X=GUw·|}7>Q<';oN'ac$FNti4׭q:}Bs AކtY.X_Vd$Nѡ9.'HUjį5>MH ͘gWux/]OWO͞IF%~0(egNC媴;<؞|W?k[ouGe]KhX3mIDMv)/ ohvlT+vi+µo;lq$ZňѾ*:6bo^X=G y{ͳ˼{CZj]ȥsl}$.^h>$Dz?,n/`KkuT~u73Dysη{M2S=Kh=5WFx\QFD)=7X3~5S]ٯe;P,&׈l}?a|P71 榝od^<Q*lϧ-.mcnk~odw> 9G>Ovtؖ!]k6m}棼PpvR.5 KnϢ~:j۝ⳬ\l2* j|q̨=}qEGLE}Us{qf{|5棴LO;}@ڷuF_Y!=1~X彳\c}KsQ.3[[|{S9>+D|.#U3yY|Ҙikz|\;SOJ\usqTl6*Uw%}1 Q&]=Ko~fL߷[՞V5sxҘZk\혵H9i9,~Smcr:Wոl͇x~ӿֿ3C]:nb>燐s_RR{Q}hIү3UBu'^|o-m*_Ab)_TGLo@\lLO8]!0[a۸5X|; vs P^sz+;˶uCsZ/Zy>!9"7QF)^mrYˆVKnی6^coI7)Z1:G"nSiG^z[ߛDT#$[:ngY96*.Gµg\nrd^N8`/_>wަz}lqop*Bޓ)_})g7p[Okyu}g&3/}e/ 97i}-K:B>]ڙt|K}۷^Hόz]ٵgϊe3}ː}3]lgGc/$v׳j^4ǚi]2ΎV~Q :1c#Vk9.72v48Z۹OĀ3qcHF7 J )m,%$ֶ{~җglh3yNnWX6FM qU{e[k{cc)b3k|LבyKО|u~zcp-^6W?D{j~#_܋sM/=xoɝg$\þ-_Ԗ?tclL959K{>vٯ;?u|8dPߗ߽-֟1ҙƏ]+\?D~^iį7=_m]n#3Os3vZy|4~Ʒ᫛͘Y[sܗ|{NZxO~e'j~p(u'L%P>/-,q'QA\ mw]ދYe7[^? $H6v6k&Qo/aO2ʇtro43G~17:!Ќeqy'I4Mji+&qSߑ@;Bq[~%tmko#ƐƷ9\&gQq|9;~FrK_ps-so[Z. lGkRm-p 5&]C~dz]RoUej.T)K~([-;8w꘎ݙkf>T6GRp.yC?CyM~Nrdp O\<2Y6؎-On_ٯ*wM ｟upcڠg&c-o+xe\W66ctMg~3:[~7 A 'F|TIydp-CjbZ3.-s)BgW-򙚞Mk'fu,6<#g&u 4ƞGrGZ˵%|{zOmG V{rcly}E#6[$CKXnOt\7hG+Uɾ۹ЍK?Cmu9_F\lv͖P]4_(_1>)۝\bjFws9^Z16kvo3*ent![_—.2È}XR.dT>x36`a}]XpހZxdS,Pa%C@ OתBum9>`<;.>,澜{µ=:}^ǖGr *(˼UOԧ8lLɏ?^jt>c7fFc{Frʝ~mᗰ=_=Ljvq;ٯӭs!V.̟M ݘī_\X1;΍;^m8cK{帽9bͳ*cƤݹrvcNZ1fdNVN?ʿh?B:tͶתj_F/#t&Tw}{qS|tC #'NGS]+P q=벬~FZ8/͵i}FחG{Q%zmKxq>:9c0r+uzN}+k!"z~Tr,4F3.udtd\\, ng%\ Ќy؈Y 52\>cb=ʼn9_UzQkbxzVﭵ?3slgᗰVۊ=-1%m/!K}ZףӡP/=U=xPCD~P=D 4&('`'>]KGtN{Hh>63!-ysn6qeY | ioJ *ѵ3.EUkǝ(c{ǽqM#"3v!#7fiط&hƺLk|6Bs]]~KgGoEm[|qR,P$Z3E oni_$2_mm=x>R[O4~cۊ'A5ޕp]v\H[{Bc9=xd,3<޵z {g^jkO՞meϱ{o~&{\ۘ0M~ _/_9#۹bwH}NڬgBr|_%ҞVfy]ONʙ^̪ ^Xx=qqKS)Iۇ5򺍙%peɷOe:k'̑6ŝē=haZ53PP'PL>ėe\D gL΁xޅ}棎77+0Vs\m _ygxN;wGt-Zؖg6ƇX?|Ot< i*_/;{ͳ<%|yop>]n<+9T|{]}78ݽ1ls-=+c3s¦s9_k#ؾMFz;4o1:7Ҷ3. &@;U9=ʁQ\v<ֶ@Ϗ͐s\/~d.[ϟƬg&+6>[T~|ybBAo1j/4cdklr|kQ}(me!>DJKWNA_BMP.:1]k?_QyN3 b#ryLu[wӸ=T3>ڃƸthٷߢ̝RGk/7jCmwUG\8ܵGYvmesI󹲳63Ќ'ZZqK&/v菉i/o2|H_>cׇ짵DRVZQ4k'S͏溋t7a?s'N&/[)}1m?9~1u*s30\r߭|syXOFe :1#OLL#t 6O'Xws'ϘѵIRϚ}Ȋ\T[1k+l5wc=El_W{ '\'? /ɣ{oM-r'ץFa.pWR0c  >@>?Cqo\ 쇦tc}h}{A35ˌ|@~ \pXxO( ey^GzT -o TwgS /q5s؁[g=?Cqrz*:54'3CN3$z0gi,h,b 83VA`Ϸ'J9Ku/ u>h\wXrZq>)pH+ v;_~k ޱYNk 9p hkW+_p p @pR \8) /*_9u`̗] qkgt-w_%8uZ/Q\O/.s=]zeG }1]}1Kt(w#T\ڸm^v>~^_yD{!#%l⼌/G ̣_P׏tfc|YH?1)'#s}٧gÞyϽ3ȣevHj\s@QogN@s=#qnhpqN.\VuCL  '0)urRlbh"_J&xyʇoYiO^|}~ӿT_LДϼm3?V~/Wrz(ʺtՍ˃'Tf|R>~*kK]fòG)**^A{)Ss,X|3vL@j=j'a<tȸM[O3ymSK5-YGk?>e3Q煏Mߴ6|󽷸ˌio;|xnmkco0yBw~;t3Й;=;Q=˭]~;|;߻O#vҳ~)5q;\qcͱxGkhzHµ]0P@~`&PLؚi}2/bu>$]e!۫НF ߺ\c|)bk`=m~W(ohEoaz(Hzߝ8H%V ^?'1ܞ>_j;$> VNm$^} oM6C7sIg^N˲)z8يG =yG;;~~;!<6+S[٦~fkwȧa_bR 7[5cб^55gG=n_(> Ǻ۝cŬZ_l[/}lKW&osSg$t?RfP/s >X3hnM%I."Ưq-~yZ1sDvmٮ~HkOK3μ;{²^{s.3O{WD8mz>[Ɣ|~x"r[ϕ}џ`Gc>ڧ\~l.{Gy0Uͳ1x]DZp{5{P2o0'Zc>gUQx+W1jKe}HcѶ53Uvr8m+W^;q̌ _9}9ڤqg"`/`Llg[cW?^-yC y٨b3$dݜ $>Bu'^|o-m*_Ab)_bhl  V:DP:\>K_sm $%\۹1sz+;˶Yhk>ՙsDGQm r6w[/kz r|%lf\ܶ.ޒn8 RXMo"u͑ ᯗ&Uɖ٦kVh Mڳ[mcjgS {W~LQuwfOq?J{}|}-پoWD~`Ʋ^Ͼ,\[[W|YRg\K;ӗooW Q?صg~f*D(mgŲ^*Xspߌpdƀ?Qkb[/3M+$/9 Z}v\ٙ?_ws8k+G;'9"f.Yhgb̿*ɗ1K}E9>qq;ޙk?Ǭ_9V<yͳMׂ^s |˼˧U6}Z gi䳿.Rǽ8ۨ+1qU"_~Zk͇@l8mw=7U7םcrecs@cs^-qV@𐩲_sWےBQK e8-_Ҹ|8ȹl8/j{<K+ڥuCZu;{w~nK܇q2=>Qϑl_B5&{CGߖO|P;~,R75_TW=j~.pl:0Ϩ mhɖ3)u=Ysm}׸隷^̚s{v#ͺԟmgUf]s\A^s>l\>,8j}Wy-كXk~ a<=Y>OgEz6TepW=}1=s!\ٿ ͟M?ىQ@!27T99*_(`;/ڻ.䍴^lZc/yB-_!FsWRYO\ յUXԂ(r Vs:]De^ew \@)ŋ?^jt>cǼAf׹ i[SAY&}3e #Щz]Eg)ܺB.=V Gf=5-U.y^m-Ҿ6}F7 x 8=g=~fi=9ݫV[?H=I}9=G=F[ ?ɨgs3v3r gt笶] ߙko<҇S~AJ_h^^1Sj~fZf[|sUV#zkʣQ5pOKpr7^wHDݵ|T"_~9GA؟f\z*m1x01s\b"<2#8)\t *۞>q^նPuF9hm 0IIثx0dm hRz׈5,u{ x?.mu}߹/uɇzc*`νs 'fvx\Q}MF4?NL5xչ~t$Z-PA׳$Uh֛Ǡ|<H{c&W6 7jf׹q>2۟ o.dQ1C@$ZBa-2oB/ײX[{'ж!Q6~5DP+oϣ1uloȳ=qm{{?JAlV9~~#B篣_]皮vZ5^wNbnx">C1Oe_V_]>ה{2y&!J}]|xDc|tHƈjP⨯ױMU7ץr'NqTIQBsmK UgDg&\xډW>|^qkƈ}LmɅxmuӞ\Û؛Ҏ6:r\¥s!nw+6G]b/i͵xcﴭ9y;'fԱהh11k{Sť7Oqϳ*;BV.Pu cD?ۍ84cFm;y껝w\Uvp^ӟ_Z5Au9 j^?zؙ&\ۭZ7/Q~"T@4JAҏT=p G~.?xr4Q5˃NkoLFԇyt 銝|mg>”+R(IPRPq)^;\w^׻Z{Uڗ=u޾ۿ^p/_fb{smcbÌ7-|XlC䌼n纊MMQ_wϕz>.s,YRVV1ؚ8熝e9Q]vFrB尌A\c]sr9qwzʣ>vhRx1mQ^2s~[rrBk.caTk >ꬂ|g>> j[\Hb1߂9gbh;RL7lCu>8o1u5"@Vz/{{{S}˻/x}hc!ιl3X~.]o;vgQ7}ܰ1K|9ʟ\ѹ'}YtT^<{r7>?65ammy/ wrccٞh޽ຍ<׷k1xЎme3pwYĸsŶ7 c|P׋/v}>ד>-z[? $-WgN"+l;@.\$o!\ˀq?nJ^0 ?IKRm/^З~t\`yſePWA^>L9ヺH\e!6S ~}컂h=T3 >ڃƸtd"E, '`DTK-ׯ. օFElPvWV[]޶N(PGt .1KWpmsv\npY6,p{?jܓG^@گ+ao\K{ݷ^ڇ-v>^S[s]~EFm޿JC ylֶ}FrXҶ7-R^\2yC+|\Xy.ǡ>.a YX&u~k^|pMg30f\mwyҩy񶇗='kг:`m_u6|{ZkۛSg.my{lX*fk.BU{ǽq9Yuܗc!czDF;{?\眜cF0ъlGRcF,c1kSnjQksn6*y֚TV?m! Գ\Mgng>:^\VcKhlsQǭ&.3+l}e[?fƶ*}GgL{l9roO@;]ep[LsEկ~uW \A!巬p=Rgkpu9ky{5څE(;%ۃ|( \{!c-3$'W_'36By%ΐIua M@u]^h, #^*;z"ZTlJohOE q G5|O}N*\N k'5Ip ]ݫgHڿ#mI0 [nzgAhA:r$r%FC#Kk`duꩈ̬|:u̩|:שQnNWuy1>5/8n\[Xc@p|zvWb|NO狑3啇M^YI'1ﻓj5N~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}R;}_UQ9pHO<-]_FOzT~q9B/M\N˟4[ 1˜N.b\}vm˺1MMn`y6]9">~_hu\:q\űph58pMY,EEb"RE#2vh!"p]awU֯/wIHvW+:#ӶeZ34sa`Q}/b`- ,g揦/It6uY8׸$Lpˍpũ_n lW(z仟s>$ m=:]|t _ \W{.p 3SAPT´iAQqm_B dggbM,F90_]5?/ti^"&mr.cc;8 7>_<ħIfƨWG1V?^Ir2WKeTv:jSDٽ$Ec}$0o6/bsE]۞?+|m O/Ř®( kJS<&b3ņzXj/M26.l?Qf_+Lϫ83fB =o?]|:$}kD7<nCkEgtٺoMesg};Ds^ ̓}M9edK35_φ;z0Z3n,k+^gT+cS$| fQ4= ΀p!\S~YU?Tkr=,vy 36zbғ16q&SB6%T݄c ةNDw,IpCS"<$|X{eH9eMqLem>/p% I^-\?1]$dcd|dRkqqς7݉1\yK8Q糶ǡ{pp͈-op~l1̟h[1Gl>v6~ɶ?3u;}{}gyK볍6ǙYqgHtL>v.p͂1 TV/#+PD o}GۍVϫ|MEӺ(Hu X[a=L-Cc="aK (0x]7+ڄk2$Q񚓰R7]v2,6JUbv5^PoS'Y t=Z/{(A~MEq "fơL)zon5$]]h+{ pfI;^;>ܗ?) o? 2.1Ƌafmv:_W1omx}zvk1fe}" Z8)a 3b>G3'$oLۂb?UoeMhY{xI[ p-x~ XD_X{Ú} Qzpef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iT#Joy[Sv5mO6\ц\= q1."CYQXH$uET&$'&pLrU׼\mIqNL o./~oU dcFu"R.h%rǷq,Jr?W CGl"J@['Y.Vbx%^ lf;Լdjd+Vâw?M!t0]6y".hI~M1{q.ۢkaƸ۪8cԶzt}JN&\WEkƊV˶[K}GD?M="Ư'Z߇#XpxݼQ_}[DIP/&|ĩNR%E $w1B{\-p"m }U+Tٲ',.:oZϟY ( gنlI 5} FOiS~pm("D&86'5 %aܺLp@p@w? 9Ip @pR \8)N k*Oo/_N/G^z;]yשtԔp,j1Sv__Nw7^\=qs>Yt핝nj3o_{z[X<{^/u;~Wd>g.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'nmoݥg@Zc Kw~鿿k~.+]__WM?ϵU1 A\dl7ӓ>[[$zƍkM/w>}{fx9y-3Yޓb=;`zB)#ӭ>99+wB~~W}WN$\(D@&(E":REdbMt+GS? mӼ.X ~}1KB+R])ס-1.>]E/:>u>KѯA`Nu~~&{?6g7QX09Pq$'e.(I "qvT Gz:@6PR) u!n%;NPQHx=R6bYJbq,$"dgJ ^/q c5`bV}8;W)&|D/܋Ƈo}qՇce̊Yr u&}s3]W[81ڜ{_gؼ^Az !\ \'YYImG~9m [^ `8?ޮV´'ޖXa :*8 T߾ ^ Lhw9CK煎0޲":'mr.c1m>uxE1n㟧z$aÕV,cl=h+RNIlmMO}+ĂcRC_8BM{O:ɹ8q꾹q,c]7&B~'6?D vNd =zo!2 ݵ<3CUeGxMd_8VhcהrNG{mgpfSЉ#ץF+.|:5qԙg:2owCd!c"8#W޹_8L5ӳ~'dZݤwr?O{J~ߙݻYoo>{N͋l$3_`<C]ؐNs >Ų4PG><3֒Hah LYDžkjS]_G`JkbCcI$TBl8<۾s9T% ~m"ZAɬΧx3ymy9|wIx I-d(sD{;Sg%#{rKYgUu6a55#EKlg丹=U>;Juo3%~sW|,|/J G^ޙBxukuZI^gozʽP73Eg*\ҧ췤c~k|쮟98;Vb&L7=?^f,0NaޓE?k|~eo.Ws=<_-c݉2;u}pz wzwif[}xv~l윕kk]W1[Z,k ,hƲ}QVJ,5VHOP}֊yoѷ?dg~z>*?b|pu|I-a}m5@$ Zͫq;p&6z6}E~ ׶;K[fߞkXQ^ڊ hĤ'bm="vE2h&wv0]oZM"_ʾW~/Q8OwI55|϶ϔw`-99=*]swLuI NAkNu>4::}YhnO.tgG3;.ޙ&l1*ΰ&WeuI^|I\ 6ցΉ,M]ZNb=O{~*L+ zL\V# J܈C4'32ʙ32ٱMw*}4QX\gܣYдri1_}vTƮ7ʙэ]ְs:׶3UփnC^flK}!7#9_OrqiɅ jA:Jdtߜp[ ⧪oeM\{8 p|+Š=?%+B{ztLR k~TF<4Q]]&ۄk!Ulen3WDo$ Q&?@wk髫=޸~:IxkjCTo6?-qQ!$E=X:žrmk?|C'$=rKe2Iݏ9%8;S+'{TKqJމz>9YxZD~nؕӷ[26(VY33MؒcTc|)bk1y|aͩ>G+u {W^AEwХskx#|Pɺ9N{2{s]8ش{dlR_̥Z*S==f?ݏҷ_l3p B/vYx7>m2zz7?;U=xxeV6v}l_b _-^KN cokּMp޺8س}ܞﻂ~oW,c-mF9pRz[Wבh#E;U|%m+"hnPf}]iK]`cvϟGD=~;o[ozRLYg {.uz,NE>d!=Ccz؂\jJ0 eۭk b?ljspM{|RBE$wz5@_cev9%?%"Ypqtm؇nҵJ/9URnf1ߓŀbđӻE#!/ʙqL6C2gs^$X o37O8L{>QkRi}ks=DWV*Sy1Ϙ,c"CS}(lgeYZqiwM"nP5֪߬5O2㱷qMק 3#Zc{gۈ{׌[ohg^-ECq9yeDLe⌨"qok8?3s҆=9ݲ_X=<7/FeHgly>Xw8X`+UK>eg^*.#6O_ce{b9/g)nW/}Vz O˳P|p]`qeP24GB$6 ڒ@9eq5gYx*vIy (@ۙb4s< \Nzk5ʙ'\_ f+ڤMy=Wg;8; []Yޚ>G1g: 輡řONI%q漓p !<8ٽx}H? o\D=_G5Ip @pR \8)N~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}R;}Zqd}nV8>=)GA{exgo>?U|iߟ?\__sK(C^IG*U3~g.c;osOE7O?e7SQO/<7&l[n==tEO++Q'o򓿪rftLODmۣ7]2u8ӱX=Rac̉k H Q]PQ|vi!"p]awU֯/wIHvW+:$-1.>](bW$X'Hg'\sh}F?uǀ:߆zqCE)+.dX TAH BkbXĖ_{]Erc}q?r6~ۛT>_fm㏉Fۤrow}.c%$$(ʷFXXH"#6VX#mTlYaPƋájbq&푨xkyˌuVMܓ#a!LUĬlFoW-1~w׾/^ZP^ݷ })8bs  +1c:8R"&A"NsyM&r fQ4|]#t_7]x޵=k/ B`kr=,7)j7?1FINvs|)AY$1gEzؐNs<&cYOB5c- T+ %\3gQMvߦ. vGJr7*Tq]Nqez{S:?v)G>$*;5+g4X>[$*S| 9$Tq>ts͡cɅ jAz )DǶOU' ≥(ʚ>]qs[FsOs {<^?]C3/+B+֣Zy-sUn϶U@/3É3 twM dI5'&e{(vJ%x#yG͎ ~ϦxlWd[7ߢN+FPWݍ2|r5/OX ŬȢ[-ijyRXcxµgprGƀ<e;IۜrX ]19{b`dK27Tc;\ M zwH<x{HZƞ+Iqekab.#/u}_yjk[ hLYGDؘg5,{ : Bqza28".1gx/t>93ۦzQ/vWcdў^[[x1]bubin1nʽs~g,.sӍZ ǘG  ?'5cEi+= wDC߈ntp#um_{}8?2YEo+w|^PL{o7!^$Egj$#cH2rII?@,s$;y']k<~gۗcw%aNpt3R@ x{1u\MVϖł5emhwo1Ƅ!zXuŎ^ 5l<==}CrMyﭸ'Gl F0{cR+.OBOk{[gk2{J]C*0}=14۩cb筯) >|쟷4Ƹʉ(ZHA4ޢSS]X[Lo&w@5أ? D=u;o[ozRL6PLj"3$$KN)[>vw5%(mA)i%չl DjŦxmu%bc5kaw'bGѯ\k"]&ѹ1S̍K[[$MڛEygn>,s.vc_c7!&b-9xfjG TDFSy9E D"RIDS=3,ףX w_Xs:7cXخڌP}BDzYٟ & cM>{O޴w[PL޴v9tף\ L5бr7gյ.{f8yVyn϶0лPVBZcۇC'9+Z\a m?؂pmx@@4Dk8N737U VsJ@fŸ Yp NkOCV \8)N k'5Ip C~(7w_o+:ϼnrw7~VLqrzʭs 8=\Ow1qv''șIk^c$Iq5N'\u1]}?ӧ޳uO7?e~}1Q+)rzThzkwY_r;m>}/~qQ <4M>)fXRvyQ됔Eϗ|itџO?뙏/S ]ǟr|4~s9/o6c~:=mF۠ǗuOϱcpݲOӇmx۩sD}}|;uub9Rc_:ќkcq"DA!xXXEpFGd iCeGE4 R__LlaJu`~Mۖ]k}}Y_-VGl(Vkz<kf6]y#Lz8}]7\3o yZBm|kN|?JIg,d6 ?^/ozudck3ᕄ-'sE[)$\IlmMƎo1EKY1vyG f"6G[Ե?ypR)슢iްd>ų\ooz)6:Sl}:&bPk݄,oSb&e6پ3c&ʷo8A"yIfEH2|l߲&8A@6iʗQ$VٮPO NK1Q2>2)Nu}ٸʸgmp<%sYW=Af7_?z6B4# Ss ;ad[V흾彊>3?&}-^gN.\U ITUtۂb$U$'~xitEKEsXཥk} rna~Ǩ D_XEdwm9IHĕkeT&e$jQ?^sDWBe\ifGI;0OBGMBrVhb4}BvKz[k77ݾlY-2^/@.Ɖw,xcv+[z1ߪ&A5Dp]JoqlY"\g\3D"N]ڭlJvy6)~WP_1E~;eCmQŋafmE:k]]^#SW]+mђzvkǩݛct(L]E¦q;2Uqd۩m1ǑmqhL֌k[ضK}GD?M="Ư'߇#T[Ix7 s˦c="*^rMS6K56I D!b[E&W&9\eOX\tߴ?@Qβ -kÓx뙹_n7GkLm ׵ױ֡Xqς7v)Bh[:1/kqmO{ɸa0q)h]8-:fcǯ6v=jGzL^+\̟K{{>NP7K9רAqϖusܵ mSkZUq ^< G;c(};S^uN$\GԢu DoN)m.O[Q3+?Qwͧ/5vM,ަ+w޶$\(ă;CD*dD%U&e6!J(TR-$j[+,.g!:,[9o;VNZ}φ$`k:%Sk^E@bk?4v)E#;Ű2[޼LW$4+Oߑpmɶ{qget5v]ߊ)ηi?DrZ}~S9?U|iߟ?\_ .{J/v3=o命E.iȓ:p)滆 kō_>k8/mSvqv|N:>miw?MU_orewLKy }8$µݻ]U'<mvi^1z֍wo&Ř,:bB ;N?C,{ÞX0=Kb)֣wV2MWN$\HkB)wT[ iCeGE4-`j IDAT R__LkaJu`~Mۖ]k}}Y\^-VGl1䫄8;_? ?L\Nm2(boα8arnhI6j~.(y'%Dҁ("(BO(19^JwRcdՓO(1''?2I^oH-"HL LI8[ɂˊi}w0!b5ٙO}u[Y6stbԞ1Sq}FkG;d3qq'W{l=3GQ5^Az !\ \'_s?sx9<+92{@~2r&*<i#xY"lDGErcꏊdggbM,F9x`W a-U~[~KAĸkN 6JW[:3|KI8Y']y.7;?^ ~K Ϧ- oX7=$LƩYqtqz$dΉ ?Gܿ ;R?Ἳgc{J(݇)D#Md_؏ )KuM\ܤ^ۿ3Yj1g8.8 A.5Z1vQ#Ϥμl81Y|"[^CA1ʜֶ:g?=Nɴ~Io@~d3mwB{qpm+'.rxiO,{)S˽8k9L/{OnڽHqiQ]yܾo_ޮ=R櫄t-e=@X|m= pµ/3 k&gxvKpoW7E|Ϋvml7'Cw?X֡.і^?.\?!=DyqzKXY봜!>h "Q% <Ahp㤖`#݋ޙŸ8+ٓ[?2Ȱ=Őc* 6c{RULCr=U>;Juo3%~sW|T%d2}Y1azyaz(AYK'?G} [|c[Ln9S$!_IhGqKl19nR,y,M9mZ;1ʿsjΎظ 1cOǏr>D3/cX짵dwŽ_ۭ>kzW XwC]\^,>Gi]ڢǖm+>8~=yBt ?3O̪yxI-a}m5@$ Zͫq;p&6z6}E~ ׶;K[fߞkXQ^ڊ hĤ'bm="vE2h&wv0]oZX_pȟ>yy!򅹮+}'a6ip,ۗdQ#NẸƟ>rΔ%'bGe&Da$Dܞbfy%%:pY SU݂|yQLǎw [r3=Dz ,E:{z𺄾}QgzqCa,@FEX޾߅8S ;$s>-x\nړ-._-K1Fyhp8D_q/񈁳c{T^iS߱N;nWߓϸGݳriSiSg8b.kmd]opߕ3 ao=zthmgqx!.:c-ճ*|L ?sM4u Mi  :IO,f[wE[I,]pp]{Os {<^?~5}AbjOwÃh][hV?tNr*#خxЮ\m5VIXX* LJlW"7X}` ;5o\S?Z$<u5?*_ӷ_woMJM\dB#Po8*֮?Af\[2Xρ N;‰2U!x\G{{gJ@wp|@7-(|Y"X?^yl[- U9Aڞ*OKY_4}k]}߯{!C@>R!W|-<$s!Ν-U{]X̚-Ga@:ϦFYal9ly ҇7vcg aMp׈8f5gֶ8c!^*v{%ŏjz &1<M_?^řML819)[x{QZ2|)x=Z€CؒcTa!zg q}֫G Z)Z瑋_?w*ãTz{W_sQ 'vv)/11`g#3qn{չ/d!=Ccz؂\jJ0 eۭk b?ljspM{|RBE$wz5@_cev9% $"zqh؇nҵJ/9URnf1s*.DA#!/ʙqL6C2gs^$X o37OI{>QkRi}ks=DWV Jy}g )RCaK>^.cW37{km=255㊲@7["}kRݿ>X֚vҞlsCڞ+E23Nkν(lƻGɢl[{[$M7Rxwt2Bн'[WkcS;^գC_ 錍={.ף Lc弊~ǹLKYEߕҥwD&y=r֣lO۔y^`Q'o= |Zs@BmiΌ$NH3qt,Db-/Zk cR&Ws葅Al`ܛ-ĹL!|L  8w5xष!ZpylMz+DS~%xPqؕsq<@| Q9z<딤Yj; iÌ@+m)c5`OԳ\+pT \8)N k'5Ip C~(7w_o+:ϼnrN[bzc+ۉ|vVp? x~.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'޳ݬwKҮv175ǮrTĽWw~鿿k~.+]__WM?ϵUI%82~rZ5}wLJ|?=HuDߦE4v9Xtwk,ru+g}(c䯹|A7MOfz"ʕݠI>qv:N ˉki@( .buc/"b|k"]!A0mluBjW]w]c?L_L¯i۲kroc;˫ꈍ=&|ĬD89  .dlQD#1 >^ep}@BbK ,J/ufe,+zdjvm9y G'IԖb!n7DЭOeߥ."bKqx<ākͱccz.:ӈ36gJ6(q-AtTD-1Iv~}&m'z~ vy=>خl.X[E^ޏpMtE|mTH8t{udc,꣖OoF&IuR9ur먱Vv/yQmu"BC_8BM{G :W$ۼ׊T^,2$ -2d#\L, UMݛKH3+{Fg_ٿZp&]~U5 gq-ymR]>vc8F‚֢fEjEA#(ׅ̖~i e[1ʨ~noά&ogYٟW }̈[g%t=9{/L %uo A*f?17}kK^xiA}xmt6{3٦>dL=G ?g)\[?Z0&n>,|7 mylWdqzsYj=A' tR\DZ HC?dWgoښt6Y @XQK2"\}~7r{H譇˔㈸zoKԫҝW{b3AS'd8pCgO?u{譇kz.r:4_7^ v9ztYuCڍgqNss=ߤ]lY!w{u^ML_ix:s vooykr?gʂ73϶ZƑs3p": # zvNoKLuQJ -7;OS_j8[Y,Nm7 Ā)&ʵ6>Ӷqf:"QYMdDdIԉ6EnMR51(%4D1:mH4m1YBlO5|L&xm=:N"V,(vkM$:_!ryx }qľ)Q{(mԇ%trnaLf=dD[z=|,uV ?=bZ,QhM'+3I`M\7{LףX w_Xs:7cXخڌP}BDzYٟ & cM^ 'N}v EG*Z1]gXLGc-kcn&Ϋk]p"i1maqVB1kv4Ha1g@+?l^yG[~ޚh 5؉f憷!\"}|5jVc޻xW Ls!_: p p k'5Ip @pR \8)x<^|9/nzt]SSz/)_N/_u56סgw.7x9S^y;t{̞$|;)_qX.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'~/77kݣ jNwfM !)dk/26?5~3M_?6hr_ߪm tzvm_==Ilzhu?M~Gmyoʼnk8+`Q]DGoŚmW~{"/>*y](vWmbzZW;&ӏLۖ]+}NrkֺG.w4C WOޒ~MaqX zI`hDgScqK4 ןN1[ ׉ `[QfnZ¶a~uy"'\LOiz?9Cֳ7L|?U yվ72k9?DA%Laߜm^'qm?*fv5}(.`~x~ vuv97]"^Xyү"1ϯ9)+%ßt$Dx1ՑeτWmDpY']y.7;Qv/IdQmub+<̛͋lQ׶Jhz6_[K1+yÚreDL!^ @A{vM-=OJdz>*ΌBO+ߺ)A&|_56>>4К~:6q.[}S?ęGD_ќx"ȯ?:{}ߥsgNْk188v{LEVk⮃?p9K Պ`3I:E<+ w:µϻp]O.\wbhP~Ք8[7I)&l瘬K$lؐNs7&cYNSj!IcZČT+/F -k(d#h|/INbj1 @~$$  #T]^{ݗݶN X>_>y=}>݃kF/ny@ga,Dz?b`?>ǰqKe[ޫ?cAGg;U\ZmW}9p8kG3gs ,ku, |eM<*J17AȴBz't V+UIDVܚZ$_'v׺`,~(x[{WD–Q`nW dI5'akndY'em$6}0'jNަNzP?/P_"\ Q|pppDECR0&vݾkIV m?vw|/OwōS6@ߘ$'e\i ctڤluh34;bxkb^cˮ^y*NE6ǟq6SILg>&}-^gN.\U ITUoǺo-RIxb7+":jC;uA< W??.!td{8}$pef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iT#JX B۽k. vZѲ)tvi =ڞlha6ڰ84׺  D0|-".^rMSwI2aˉZSW]JsZ%V&91o/HP|UMkՉ$2K"\(E^)\og0(QoEg[ҋx1\P6mR[b 4=vʆf9 +֣t]tֺFVޣ%׎S맻7 Qę"WoMvdnɶSb#љ!(9p]+TZwivoc/76p]מh=~3o9M6yoUD&DU$A䮛-:)Kmdk%lBs1-UM`LsRf˞ i=f(\7e[O'X 3s߿8oJl ٙT_'kkc7aeCu&oRжtbx%^ ^g,ēqM}G0c/\[uy{z@,By&k9Gc{U PJMw5%ZmB$Nm;.Q-Υf[@';Idն^wWX\BtYsv׭p/7j/ IuK:= r8e'Z1`iR)ŋF wa3el=gy)ZHiV.#0\m?X;:jqSo=Ң$丝.1{sxLkt.6Wψܷ.{}?g}ǾnU\1q/m 5FOiS~pm("D&86' %aܺLp@p@w? 9Ip @pR \8)N k*Oo/_N/G^z;]yשtԔp,j1Sv__Nw7^\=qs>Yt핝nj3o_{YjXW5C'ϖ.g!`st[շ3}=[(*\=/wJew}u%veiMSޤ߉Qь5'AQUvUrg3}M]÷pCM_]*~lϟMʎIH$F^bbq0KLOcGnydoK7}q q);Zߦ:3ƫZ>bqFgB:u<䷥4=V.}ō(#F;SnR{^v%BIkw&Ny$o5G.Cw8kzmϘcZ=z7PO#9oos,x8l3p)V9gE!\+,{yո3ܫSSs~?'I$ R ""VGq" F&(r⯩ކ˶i^,wU}G[%ꎉ#ӶeJ\ŚѵˉBKМ$X'Hg'\sh}F?u P!ix ?4]PB#\ "9vQuIk0;8q$|BI1k_,K/Ne: ,LL6 MLIIC^/+2-'=+Ʀ=joFD9v%NxkA~W2Ƈ6:}\glJ C}Wڞ5.\Z  K2B ":l̚:y6sNyz]|8p+9+\&]}vv0cvLTxxZ F7g+e[׉zc۷kv5}.`YK+ w`^zu uo"q/7ӓNjRIF7C{ rC,-e*λkyf88$2C1I<7} bRNH=",d\SBK%G{mgpfSOd\(ǑgRg^k|s,QqL -|T|4CZƽaY׾й0iO~z>9iw޾߁<)=~gXS5qd׃}I?.R8q=ǖ21/_q;~O>ƽMƈp^踱ߡ14l=DbʞX뻺_ey>Qݛ Z0cWUxC)+DZ/?q'ʵuXhD#EqOU9}b{pѾAy_XyXY V$VDĦI",7|] כ z<u]>p݉CIp{t.[z payP y!燹 \B|XK$=KK~gϢX=.\SM]?a=ZZؐk=Xɫ[6$A*}rms$B //9Dkkw>봜!>h!~|wEǠ|b7ɫjB~w%JF8bԞ׌E sq>Sxmkώgl;\c"8[!8֓ܵ{1]6vk+q$ן#V޾߅-@-&V)%5 k't7`:X'48c{g"~s iě.7gL?q,e2}dϯ9-= ߅Qt|f?gao2?9ڙ{r(wzIzT⹹qWѻ'e tɶ|q'Y,h pXVr/ʦxTbnYiD!tm7Z?M""n|":Q=c8^m #a,>{ՇK4c{Q?^=P{u5n'x^[F~o jNY淧=ZDB>atKm72Ā,>} N ג] '+:ClYM,/ue_ #`b<'rbrNJDz}I7Ċ޵*ŋ#ܿ3%M#ٍ.^9;$}%%s(ĘgNRNߪnAN^h7yQLǎw [r3=DzJ |Ő$tZ:Gx->+Ƒ>,o_=.i]L}jccpyh]yW׌ͻ ;W(Һ]Z:w>YUB>cymsr:ZLz;}[6m7NKY]EM!#\{1b}AOmMGֺ g/VÍxO{r*0<{al{> ^_ Q6sJ* G͎ dwCp#\M_]5#>NXk<?*_ӷ_wo?WH$IB&Zۃ~)Ϙp1,k[cQ{ Jȩ]챽/VDl"G{{gJ@wp|ɵ%@VTKbQ[<6vĨP՞*sam!3@X{m1۵hQ @}=qsY%;kX'~l0N*\!OYXvl%,!s޲mR bcG:mGۓ ~mUQ{GkÍ pu>ۆI}6|s>@릮pϵhK %g2ow5lX>ZW ھe %s;t(1X۟}[m7)h9| 6ILNEH JHX~/oLp|9ך'h$[#G#cݡdb35>;J l['悡8-Y΅ }Nb=ξJ>{F'^[Bm==_چ]-3o/λ]qa'%)ؕ[3{eSιF~zȮ|.e@:ϦFYǡ 0kqfN>ܽ#dAsH/P!Oc}Ǭ%:I&"K#|;V+ch]#>1<M_?^Y p&Wv_ qc?8:{}Gh޳2I|53HqQ=f– q8[c__lr8{жҷ*ٗ+փ]ebxOlwR=ۛg}}=g !v>YP[a 1 ={=uM}G0a1=z,v޶kk.ouYO_B(ر3kOdxp $-SAe4>؂\jC~xpmu%bc5kʧb=8FdY<5@_g/r811:?+q%B5Bc&Khҝ ^e,=/v14{ZӻJ 0QΤXf=!c~8J8~v'jM=ϝrMcwgҾ*e{^}1YBDԧ&PؒzXb=ߕD߯{s軼oV׺ keco'TO3fFf`\bcmqFgq\]o3Gǒ=9BS:"*c~Iń|k-L㌞= jNsΟI;4emsڧݷ_GZ Uȹ5^"uvpfߜ@8q .4h&JTM"Ph '"'%cуl+wgɃxqvspo^8G v>3嵠.,a9 <, \Nzc5ʙ'\_ f+ڤ)My=Wg;8; []Y; L9OukGX/p䷃!ZWs W^DSyf; iÌ@+m)g-Ip @pR \8)N k*Oo/_N/G^z;]yשtԔpbw7^N7_'3ƶ!g+^N'\u1]}?ӧ޳uO7?e~}1Q+)rzThzkwY_r;m>}/~qQ <4M>){پѿYXPsߩ> 23}M]÷pCM_]*~lϟMʎH/%!\#{֪?賷.cPMQjOxth]C'jDL;wӵW9Kẹ?'?O >u'c4=Ne#>eotgG-q 3'I$ R ""VG1v" F&(r⯩ކ˶i^,wU}G[%ꎉ#ӶeJ0\Śѵݡ=<1k ,A`Nu~~&{?6g7QHp H#mG,\#$RXD[ 9 AX#vcp*B!i|AQ"96"\oYaPƋájbq&푨xkyˌuVMܓ#avW/sݸo<ƅk#ͽ/y4ذL/}zk>}X@gb9މ3'T,46ppmOLjEl$> A"j|v9̀pp!\ םX ;:G+\,fՉkI6ǵr#>mk#x0(zu]tK:Z({`==<\G~5Gп{Ob1.#>8fsP`adz؋O%{o~!*ᝑqN^{^cuٸě;[ekg[bخ-XNQ ~]ucY,h AqXVr/ʮD dZ!=aq^+UIDV횯Z$_'v׺`,ډk7wC$)v^Mȉ6ZV~f3erFv2'6JUBv5~i ,PӞp-k"K[_!| XK_Z{&ZAqO6lvx roXzG>{UAm͜T>=eb؇Cς`Ee#(j_d#\3Y(Qo5鯇='9ֽPc7}|ոy {bljcSںSt]bAopǦrO}#^Ckmblqvvyk,-kDe3uY{uP DUvlo U$'~)"r('\{1b}AOmMGֺ gw%^ 'DMd5&\ G'^ԏלD,m*H$e5;*prS[KG?kG^y^Qme~:XA_Yw7&/~;ȹc,dw7#Ă=Ykf+ϐZaHYGX뼞;l᭶RcmQE&JayRYyF|Qn؟Gw9ܢ1`/υ}N6\k -':<{o=y'Esbڽߛ%q?=9"(V>7g!\!|˱9 6{#P{F=;,t, ~#Ο AvS1[ Ayw[cfM-h{p6窨=B+͘qukM&Nr9pNNW?ע-1p"k?ٰp}nǵč=Io=W7&\(?Ӎ1ϺoŠZ %p E 1sceqD\vcxX}^7%U}ƫs=yjTcx/Ặ?ޞ mݽWw譇kz.r:4_7^ v9ztYu3uPg|AcɄhX kL۶C|{臾ݴGuD?amAg}kQ[8nBH6fIF$^d$/~&X-p"mI8vNl9 5y60*\Z/8 Jœf<륀2fC]-!bݛ۹"ƛx>i )ρ>#XKr=|-CDQ-a`:[)ỸEam`=,Î^ 5l<==}CrMyﭸ'Gl *}VsʻN3/ۧ#sݍ_S\^ o]LWilGs٧__Lofʾ;D}+=Z]ǗN۵Oߋ_?\qO7MyOJ'F^kXhcjbKUIY$[|ϧ_/O_#|=e*~kSn󗏦o.aOg|>6eslĦ&\l~wD~.vGc?y/N]:p8X{XAWEN4XH&1Q,J2@EIGe iCeGE4 __LjaJuDŽXi۲kIvn;"" wGWOޒ~MaqX zI`hDgScqK4 ןN1[ ׉ `[QfnZ¶a~uy"'\LOiz?9Cֳ7L|?U yվ72k9?E%La%EX+N)1dggbM,F90_]]q#z8}]Mk(Z*V'g!1ϯ9)+%ßt$Dx1ՑeτWmDpY']y.7;Qv/IdQmub+<̛͋lQ׶Jhz6_[K1+yÚreDL!^ @A{vM-=OJdz>*ΌBO+ߺ)A&|_56>>4К~:6q.[}S?ęGD_ќx"ȯ?:{}ߥsgNْk188v{LEVk⮃?p9KڊՊ8H}cIxXvB#ZIkµϻp]act=" q}S.O{m_bғ16q&SB6%T݄c ةNDw,IpCS"<$|X{eH9eMqLڦ}*_>GKXZ~cdB=%;I,"D!Ȥ8q^e*olk=bn<ϗq~gm_C[d:dc? Ѱc4+آO},1uXY_U)[>x>Mpef8y,$qZ8IY'Zԏלm7Qen3WDo$).Ql'̓|q9韦gQ<M]-ҿMyk:}Vb}}AC"q? إʖ^ Љv=HſaK?C> mR1_1}#les̄1Ƌᑸo){^wwk6~ٿY} uxԯ/Qؤrܶ!mv׃̧}iTB#B(*[SAܴb/x{Jvma?WEqh$\s QD-jsv_`IP/&|ĩL;ITOL$zDy}%ܹs-+✘n$(\N_>ߪ&A5Dp]JoqlY"\g\3D"N]ڭlJvy6)~WP_1E~;eCmQŋafmE:k]]^#SW]+mђzvkǩݛct(L]E¦q;2Uqd۩m1ǑmqhL֌;.Zm;ķtyM{D_kOsw?b1ǎĶ~חF6c-=.$Hz]7[$NuR,(J$+0bl;i[諚^r͖=aq~zGQo86,f˷Ogqޔ3 N&\^n ZbM? إmJŵ>Y'> y˜kĥ>u'㴄蛍O_1zvp#3.mђ\:}C,^=[b6q>7M-hWY{zn7܏ݣ&>#rߺ-nUqƬǽCg(|v=4r_2b ٞCq}2'WB4| "m t_SXD5x7!Zpp}7 [[ !Z8' \8)N k'5Ip C~(7w_o+:ϼnrE-&Wpcݙt+۟/`'U8<3q&Wn*g}q,"֤g|~@\OwZ{g}Oދ3{>:x=8ptLF>xy<} ʯ>~<}Fl{ODݷR٣ݯue>o||]EY7dzӔ7wbe{ݿ6Ư/8=5G+9 L_;\gWW>~0mgӟkcD=r/1Bɶ%n'1}#m]ٙͳµ{{dBȽVCk 'nϡ/6Ɗ{9y4&?s>@|Cg_A>/N= ~pۜg<:OliV0Iv~}&͢k!%UBGj/uR9ur먱ïS% 7l"NguSޓNrn doq:KG׍пI1i;'JB27s\KKJZ!*ɲ@8)7<7}/bBC}LoL1qVZ g6ř#] '!(ץF+.|:5qԙg:2owCd!c"8# ?3\5s/R}oО|;s>'&}yS{δϽ|jf}񺘽+5m_k=֎I?[`,~ޙx#{Ǐőv)\7}{qۯw]{{-IW y^~/|i.~ :YJsfl/>Gó;}Z?}EuUg<:Rb%&A"N{}cIxvLFoW7]x޵=뺰EcesiמmqFx+\@!{vXz!|lubC;I\B<|XK"ˇ%\3gQMvߦ.N)u g%P e o8S@؋ރIi?^ y^G[h&~0Éo@᤹tg1vqRK0»^YȞg?DDMF}My(nd?GY۟z.6bY*%:7w+Dq6,Dro=kq}CXk^]k» /|+zoBb [ΔW;~3,ئ};3+cs97GEēcP^YfX318t{-3Jj<81%~Z{O]^ia.|0u'vs+T$c~*g vX_- \IρTk$b~΄5_֞9*fskܗ,.+O9=eodп+/qs|Uzn3<\ǻV+ih,sﭴυ`^rsn/ i_^T~-}s6M鷲olwvp{gKg<@ A4tS1[Ayl[}XjEK_-h{aPp6窨=r.jx?pԺDzh"={$|,$ݤF%%5;E[bebD[N_? g\+oc{cG $$LKLjݷv=-pF'jYym[Nx -J޻ #ܿ3ʪg=j󭭵+XdX/o24v^XRs s ҙ%io [6}Ym1v Qqv$֣~Tꃿg3ů}q$ &No՘.9Mk3bNHQwd7q7i{Keo&TU{{0~hoa e5[yٛ)ރB>X;+Gݫx6'5cEi+= wDtW]EækGcmĶ~חmѻIDkB<4"}H-C?4[sF}E'u*W, 1ƮK,b<딾%aӉxSmmX;Úף n[.0pWϰ> "W,\w8pRzVבBeE;U' xBkH47~3پ4@$ _S|]gL" ^a?*C<= Y$}ȔCztk%չl ,Da"<ʶ[zdT ?1F'HdYP[+k6Áп9ʲrJHDb81I{ݤkr_sT1!׫ܾOc$$Zӻ#]LJR8k&!k39Tq,Ǚ[!0$=5)>w5iݹoIKX+>cvOM%v/fřݛzxbsQ>v}N??޾O5_f,̌kkTc?Y=Bk+]6F>Ab%kJ8JtΜDLe⌨"qok8?3s҆=9ݲ_X=<7/FeHgly>Xw8X`+UK>eg^*.#6O_ce{b~ߦ}K{ U婭W|Z3`%'?6Ȍ$NH3qt,Db-/Zk cR&Ws葅Al`ܛ-ĹL!|L  8w5xष!ZpylMz+DS~%xPqؕsq<@| Q9z<딤Yj; iÌ@+m)c5`OԳ\+pT \8)N k'5Ip C~(7w_o+:ϼnrN[bzc+ۉ|vVp? x~.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'~/e,1~~ia9p}+;W?5?|?ՕϦLZ숤[BJj?Rhs>{?%_ˈoy}v_:.$mx[~M̚qA3yM^Y,N ?7Qt_syko27Ux)#Ү@Dg}qbt6F:]|.c鰱sD5 BdQTA0\(.."_D(.DEnB5hyQ8Ehӻ$[R])ס-V/ni熾/!>/c;ko5 0':A??=E3s$8߇6ksj” ]B\`Qbna(I""A KqB9DC9oaX"rso~~ff >Č`"okC{߃>acwMbc6d>8sOliV0Iv~}&͢k!%UBGvcpM‚֢fJ\>-jof!6oL޲/w+C͙-L#Q8+$'Gz񥃉aY?UiSw/b[w_}q}-6}kK^xiA}xmt6{3٦631d9,k+VgT+'@.7v7mylWdaqzsuZ]C [0V?1vgw-qːtl =7ɗ)iϟE"yVpXX`[Nϋ i?'c20eP,P3֒OrOZ;ӟ}qڔoWmkq|-w N 5OIԾW^s|j eh|eg?ޓ˼|ely?:q-VT`grm8i c-^7- lW`[(E:}q:op]ۨ9ݳ}"yuW 6/\#և;gO9 {d w&O^k:h_mᝑqN^{^cuٸě;[ekg[b؞{=w鳸0-a:acYV\$4RYMȍ¦'#+D LS{;M_޷/Zq?M""඄׉rb=#AºS=o ŚX7"!HonBN DJ<5)3=?1Qbx׶;K[fߞkXQ^ڊ X?Z3ՊM ~9dG'CwϷKHp{K m=rܫ jkvh(눵k->g,uo)`T,L ȒWM vA kĞ1'׺vվJx澯oԶױ>s{OMmJ[i8e#{s쫙S׭GcPZ$[E]3`gwF}Fgxbcv{Fq:)fI^Ο A@ՂtUeg&m7md7 +"[S]}{b{O='\+wcp:$|џ4ϧ:_1E/3É3 twM dI5'&e{(vJ%x#yG͎ ~ϦxlWd[7ߢN+FPWݍ2|r5/[A{Ka` -&{pձ[|?޳?[XsEy {9#c_ kiz$mk@[Nxuyފ{N1WT}Fqo Pg!\!|˱9 6{#P{FY9Ip||M -c^Ӈ ?'h舧,8:bc9nٶN)Ԋ8 XtM ik*EadD8~S{,S#{Ɗb9/QJw3^C;PrτQe {3 #1.F/\죺N,6MW]z5]Vo{̉)59Q]o ?'5cEi+= wDtW]EækGcmĶ~חmѻIDkoeK~ ",?S%xEN"Hb1m$I 8Z( \èp?kݾ(+ scShN˘ uhދYwo6lo=?3<7b-ZYz 4@O"\ h=3.{c^-8|/[Ű-_C9+܇=z-ފ{r$ƽB+L+j?0{ө[7~n܅ʺwbJ{T3ՙchS~{+[_=S}-9?"t<Ř6=66P9pRzVבBeE;U' xBkH47~3پ4@$ _S|]gL" ^a?*C$*̐,:ѦoMהI@&(TR-fBQݗ &Z?= DmGIrT|ފEڎr4vDK>X.O17/o5n73jo坹㳄_ι[ۍ}9߬c_♥ΪBgSL‚EYGKgOWKľ݉dgQ>,ףX w_Xs:7cXخڌP}BDzYٟ & cMN /(7 #0EіIciUQ{XpXn鳉Z=3iZ<7z{sgb jqFX+bTZcyS^Ha1g@+?l^yG[~ޚh 5؉f憷!\"}|5jVc޻xW Ls!_: p p k'5Ip @pR \8)x<^|9/nzt]SSz/)_N/_u56סgw.7x9S^y;t{̞$|;)_qX.og4{Q~#Uӯ/7je3e{"_NﾕM~.yKnڧů?.Ҹ'ӛI'w^mkmĮuw9c_OQ됔Eϗ|itџO?뙏/S ]ǟr|4~s9/o6c~:=mxvm˺1MMn`y6]9">~_hu\:q\űph58pM"?Y-ED"RE@oMP+_S? mӼ.X, ~}1K½+I5m]Էk;km% ǡoٗn> p%,%Φ. i?~bA8~ֵpm7EWO|8~"r܇g_o.?~}oe>r&*kf5]`WW64Ys [ֺgBijdubn/_sSWJb?c$Izycԫ#+\ $l9+J%N*gn\nb5v|)^Ȣ>Vx79 آmϟplk맗bLaWO5%)zxK֙bCA,He5Zc&dZ {6a(/|U3.Wu|SIɃXMµk"m}|h 5u:kml]9~É3H9D_]tAK&Μ2%Njך/cpgqZg=p]rµ'3 k}}C7|.t\#벹pm#'\[k6jhgjΞĤ'%Dcr5%mMlJ [9&ǒ2 >/6SXTڧDxH1#$s~˚, bڦ}*_>GKXZ~cdB=%;I,"D!Ȥ8q^e*olk=bn<ϗq~gm_C[d:dc? Ѱc4+آO},1^DmsW5:r~}1; "aK (0x]7+ڄk2$Q񚓰R7]v2,6JUbv5^PoS'Y t=Z/{(A~MEq "fơL)zon5$]]h+{ pfI;^;>ܗ?) o? 2.1Ƌafmv:_W1omx}zvk1fe}" Z8)a 3b>G3'$*B p[ #P5N_e}WDfQscbgG='\+aycpv|8c"Z7{Z˵c{ڄ/3cDm%+ĩL:IԢ~$lZ(vJ%x#IqG͎ dw8aO4=繭,h,-hYMo;o\}Z eR_]Y.VbxN l!Dv. XYhʎ!CLۇu'cLٶN)!֊̾8d vu|AAdh%d@vlo϶!" %w݄o8 {' D/5u5;E[beRӭ˧[$/Y&XQH" Z-y-5\klvR0VĉK-^55oc?&ʽ+ưhOcl8m6pb=x1L׬M޵Hg˸kkdk=ZR{85~{@}E)º~uZش1nG*Θl;-8-m Uњzmy ۶#nFtӦkZ2z+c׋x#:mءm]3vc(E$QEK "qfIFQ&](D^>c ܹHB_ʤ8'k ˟Ng?u~Ya1[emtxp=3ˍpͶ^Mu2Ặ:vV0^:]k2Y.EmK'Wp-ubO<Y \#.8%\Gl}ư>֮G R {asio{ xf75(nm }mCk*dߋt|~6зag Ή(Z H11ޢDL{}QXS"V|Rc` `կ7]_g?m?gCoHWސ Jpۄl7]S&[Iܶ؂\jtRDVmuw,D'`+<'mp שSoِl-\DZDyߣp-׫#QqRusOQ.Rhp6s[x~;{˛:f{; 5ٶqS9ӺƮK[7#-:HZ@YoowJ;ʹGbnqU}r7{zVe&s'gD Ҽ}Ɉ1d{řac˜^}N\ ,P: "m t_SXD5x7!Zpp}7 [[ !Z8' \8)N k'5Ip C~(7w_o+:ϼnrE-&Wpcݙt+۟/`'U8<3q&Wn*g}q,"֤g|>+^Pk|rśL9u4 pnN~bO#g<>W?Ro~>bzV6S'[׺N7>vڮ}^,x2iʛT}}զ&NZ}^g3cUrg3}M]÷pCM_]*~lϟMʎIH$F^bbq0KLOcGnydoK7}q q)ЊX䯦$xkaX}Oιд;}[Iلv; ^YE(8}xq^>@<\)Agp}aAڳ:c}6WޓXw??z´[e[ } 86Ev~}fMkh k]]:/X~f]Z} }gjiw!\GEo聛* FSNЄ>jzfm{N:zQ4c7s!QvymR!\"N?WuSޓNg xoQ:KG7;NI<-=ɇ'P_? 7KwL$yw gNR%d"K Ě}N`%mgjy1\g+|]3/=ZA{335ks*,k# =&YXYְX^C{.4vq6?ža=s2|7pyOA;)e~ zk(DQUuL caeVYɸmKpLjx^ȸߡ1$d=DllgO];]eaZC}ݛ ܵZPcjUXS+$Z藟vNƦcP+{/ȩdTr8G=B')\kq>AM'B "'\kۄo9]WsZk6bdgn\?Bomv!=` ЬI2?+)ɠ h[y`عzK,%G$Or}|݃𾴄뷧?~,uhS]_YÈ(Vd_ྞ?S,U-V sGz.;5B&Dk(sҵ;#?-gϋuy$O#^vXI U37+DJ^Y׈݊ޙB8ٓ[l:uhO>ӂ"Dqxlkv^踊Cxmk՚g[Z`KAp)KdZoʭ}mU]MWtp[[$?b ٢bu˙BYU轨}Ӹ/=e_+-Ɩ}h⹰sL1;+9gvOL\wO.gr>])XQTl QPu*)+o$p; Cl ^߃Xµg?|4^?DZ]3kV6w@؛}5JgExa:= YZWPƌw!lw2ooaen5W^R&t^Prʈ-F%ۓRLH)~=ڣEab͒Xd#cc;goQ"'FZĶVفY33Rbo(O%x?Tf{PFυn[$Sym%O;NŶ \E'z,~- ñwL(6?B/S\Yo^]넟yĦ$Q{ s}^?n%C=Av%fm|*m\q֤7<3%Zl>Fbxgw^#}c{:]˻<]`G*\GO q 16 om}j7 ZdQi}6 hY Cwxͱ9* A˺kaxk5=WƜ=תm`(9SVVɰp}Z/o%};%s;v|o/JJ9zokvJDb3m;xך ā^ޙe`po#vQe-#c{v@OC<ҙך5;$ik2[6}Y ĵzn󇳷pυۋ3=/'[q꓿g8B$f{{V̱s3-km3xN o?8kuqR_[b:ۉ^9ѺWF=%kov {bͪf|*sdk9yNvCmX sBӲoGa㐏v8RgX}'ݓ{-&\5Ei-+F3Y}CȮ|.2 \Sgwuֲ}=1t״?gQ_Љ aq­=p"ƂkC'Q޽HczI>[#}`BJrьVJQ'ƙ63Em׎mqu Wz_Kń\[xDlwHyqP?sil)1ϰW}݌=}'T4=3ҷ:*X2sã8?sJ˽+Z{Js@~|g1äsϳg1)={3וu:ubS}?+{*ͷ6/n\yޚUu޽71no\;>/|Gε?4T'9=3Z[ 1'ͳh>ޓZUX{=INFډ:YWZ AsowBޚأ0HW_}]sLTv7Ʈ.D{x  Hx9aq1A{KhҜ :JƞgD =~,hӻJM9ʉ6IYǚH8k:mD2j47}}m%oEՉkFg9r̖r\ǚϋ37+z7M6n cvre5eZ?޾$צ 3#{DLa֬7_]k{}&L`۩ce _rTKv<{w;TIcx<3!!=_׷'3֚3 ]S8pjlG/y2~M_݇k'`imv>ˤ=nged'7/qq .@}W2O]!Q6p2C ;JDOGHhυǢ%V$.{g;p,<8ćK;s hD0 \2IHQkpoECNk!lM~D)\ܫ8yxhqveyQgeb%\ q @ O? Oݛ9$\yC:g;8SlΔgwS p @pV \8+ kg5^L/_^pcvzں'^N/>Pt;Sl'ڛ nyڬ_M_y8ޟtG{7D~}wcs;_Ⱦ:}~j'ڧe]?}Tq#7UywM"_mVq}G]{Eͬ#9Tqoϯ~}lZ;??}2_H/ƥ%!\%; ?fo};=}gONoX}vGNomi捏~)8=R`-ӋtϤs;Xe/[\oӓZ[{x:=b?d-ebO܀M>qq81:@;:]}Y \2gHOEkxx"&j jBU yQ8ҮdW;AluD ܮ7k՚w| !uE%۷ǦyV`Yps3Ag%\k;kS}&:A1(/e\p}q}=Hbs+u=D%ڮ(WX\p}yѭ`pb$ Ln͸*1ytGyP+m}""w⢢g2fH]. SW!L+%XEp]lS-]`ggfM$&9(%%WkV9٥-ٷP!wƮ֟}un_\o(!ߖoOE$_~S{/# C2 cN.'^׷u.i oOYDžЦ~oWG}Q|䤯.7̉\DžE97xCk 1/_v95e`BN'ykVl}aI 6'_rბ5cmk#41IKpG[tOnbEHJZԋo?sO|ߎd;k/s4g ۛV6edͭ:4Nւ9אh顅:\=s;{ga/5z#x?3r?2JߓŹ7"f^[k~zo%sc5̧!8 4k \>!\kspHeH8BFX-pfBSlWZ'5_!:r~u3Β?a L ;+f&XxNDsU?]I:y9S;cznf[HEBv1ni $PӖpu K[_%|'Z{*ZqO6Gtvx zohzzbi߻=֌UAh$v5[@]}ښuHW hP",#acFlCJB???9'uq_b8.ڻn{| mݸN0AUͿ=6vh֧uR,Vq\b*ٛX:#=}^W?ј=kVdLm7Y杯q2O#ZE1|M,uY+L4#Qad.aVj.a YBxb|+Ӻ֧ۀčlw?P댎Gw9ܢĽ\1ǯυ?ouZ5`0Vv➼Z%`c2{(3{crgX`S0 Փg)\a!k~>K|E=҇=^ ǬY9Ypfl:qiW,N|*\GO q 16 om}j7 ZdQi}6 hY Cwxͱ9*q\%2K]oğ\kF"f1bϵjI3(ow5?/\sg1QǸBc$e>>k]yy>-~DL] u1(@5)f(\8CcqX\vcxX/b\szO_Cڊ[ߛg-z+M=bžP׈ƸJKѺ&1Er<t̋a[. ׮hMhQZ JLV7O}#i}zRDo`zUz̶(n[j ;S|-(N!V'䷙ͬsh$#SJ2RHWI?@fsᶍ$;y%]=[a@:ϦFezCvN^ߌG}dl+E5V̚{q=@v&֚>vMHP$`rBSfslDcLL<~}bLl\>&\x=ZN"YV,vԾ$:_1Jyx}qľ*{*mt ;k9s+{/Q+qlK?,Y -ToM6Ŵ$,h%7|V12v] v1QnG s_Xs1lmF>#cY[k?i&鞜cK;"NH+7'(Ct7maޤ晖b,k cvz>8w}]p p"ͱ5Äu/+ckv!NĜ-K`|y]DmB⵵9@5䷢!Z1@;V1kZNķŘw.u50\hLPOY稱P|zvL)qvgš#gk}'n2IIp=5OtIy4ʯ?|Ci)w^Lw?=fL>1컮w>vҮ}^GUY7xzS7q$"զSvĺwwuL"t~ ?O^ _'õL>}2/}o6Ljc~<=Jm쩴A랟 67e-tQ@vv|X:sX{ny/"\gp pgH˄@0EELb"80_C% QՏgC,%G<+j_\MU+傿OU[nWj|X;J{V}$($sa]/q23Ϊ. i?BD[⡯>/º}GNqz>G|g_/|lguް| \BTx2@o:t[ }86E?vq33o|8LXr c՚kNv)}=jO[VyiyQy|O$v3F2;?,oZuxk3N–-_rETGbJ$2d#ElN6g!\ez6_[Oؘ̮$ _Ŧ|H_G y-eVmlF?V㟯W[39̘(d[qm8D(V|_Z@^oy^ScM.>]kgbw(?8hk}<ºٛ'.5Wqf[JuQZy pQ;e'zϘJT m&Q(-r6O4;}Zg]&\eh|c-Q}B[HIϐM՜8։[3I9j&l甬K$lؐS7%SYI:}N$yk3r}N2 [A .pC+A[$\3Y^-\?1],bx|rʸ+}*g[ۭpmgs[ZES`Yd:ѲI0}fac4+"?z:~)?3흾*|g{yQ(k!,\h~ [!9'cͻ}쎤 qڜWf?(I0s!z ?z[k97;}6mAP/D naăk--?/y, Ylˎb}NoeC,&\6V u¾kb[ρ۰GkYu'4nbmoIrwn[,m ?5#ЙO^CPsV: H²!f qZm}j7ZdQi}6hC欷ck_$XK$5U'$%&PNRV׬2\Vm+┘nAP~d )uՙ,2sȝ"\('pM^)\og0 )P ފ81ς5vm+bسV۸P]b}qzd ͦKWŊpMd]Kt|]]+h_q<~{S=8D~eZٴ1nG*Έb{h[qk[kj&\5Ei-m6RVO26 ko|][.ت^Շ/,Y7îhxfI&QB'M!P1|fsžʤ8%=[叧қg3oCb6Z, zfO7Gk%ؙED_g=hbzuk YƮEiK'WbŰ:bO<_H/F\#Zq2NKNkZc]u m?<#S ޽D8򹶷Gkt>9"Yc͡=|ꖸkҦז?Uq |~3AG;()}žvINbFfYQfD$:EMMw}'iK= $~E5EbcvjTμ8 %]7&C9"/:aN7!Ѫ$qNnq w.mT$E[\," k:_7hYu: _UHU̵?UyrjbXm[3[ּĺ[3DiV|kmX;Q?J_UqSo=ҢYPq;QW{sxDkt6WJ߲\vz!2=)'s5;9ij#UKۇ8k2'ןkv|+ Ҧ@kuD4Lp4=' %a(;f&Dk \c \9Ɵh p @pV \8+ kg5^L/_^pcvzں'^N/>P+!̉}mzzU?_^pVy~;XeƑqf:g]q,"W>N']+ Q槥/ye3u2\N}I{YOUeyMU]WO[ޙW$f/P:>vCx{仯ߦ>?><}dNOLeGE=q'1m%ٞNmzo7"[NO¸u{;9B#q2]ƫ/N}uqvHs)O>Q獏~Yھ͏#Q=eeѱ_ݡy }H,µ޻FyUGwo{dou8uk)k\{ڛb)>*y]PWljz'z=T_.}rB߼Uk>vޙ8OX C` w?=c-_Y393&v*K0t`~,@vq! X "d2!xQrK$bG!1e'%>.2I7&$%,A$Q '";Qe*6ok݇مOaϕI{l,KLnk&b# k9нxꫢgFؓ36k(\7qX+חp.gv<]a+{??ʅƭ'V-jGnqG٦G.v~}fMo k]]/d,qZz.- 7}=jOm꾗k=(bHJ 鶐uxg!Y[ pN.'^Qe(Qq_ ~H|)3q?1ٞת5;$:f5*_=&ٰx=גcTnkȦ.X|}]_ JzW$?b ٢bu˙ؾf?$}N-ugw7yΠ8ηKYTp9gݱΎ踉1bOƏr:X3/bc짵dgڥJu5kzknY>臅^,>Q}׶Hy[g1N߯A_S4Nw/Ort{ƹZ Io2OM%n[<.rrQ4Qj&E}z{ve/AɤփV=_jwfK] %U4kރU7_vግm[-\.o-mjz{$ziHI#Jdoe=soc4;;Nw-8Ưtz6g%Z¾zQ滆v,"Q/ɢFP2K%$jܩ3)99%'b3(xЈk&c [Z܇wK{Fz9<;tatGM_u_>N£u_UBژ{cG^=gM\dB#PD8|O.cX}KC;N+VB(c{Az#^(T'J*hl\{{gJwpgzqgSĒtdb".1籱ӷ[Q<6J=Ukvy֌LceS| 6:3E=̟y' q;<|(jO;N1k֌ qOJPp?3$cxMki ] \J Μ{[:;v=C~Mo6q25˹\#d{Xk{p!p~mz?h3NEx9l}Y74yMl^ PsV:H²!f 1O-FS̶81q9Cm5zc|לvsy_n1 K8Xֵ:I(I3$1Df1bϵjXXVVɰp}Z/kk1<7vyt3 N2N%%ݷoB3m;!eČ2s0V{{g&g=RNbbHd_`82gwbq.1V:Zf9mOf˦1kξdeG+VBF'[q꓿g5+Pok2{ 45N6|^C?1nyC>=v-Wr8{+-21/gԶCk׶M,lo۲Pe7ރp}nqsx&xn_FMvEkBZ-tI\oD7m:_WO^ lUCWaWk^0i~=4"|HC=4燯[=x>Fϭ8s.ܶؗ 6@-0 \Sgwuֲ}=1?ř.otb8Cc8}{cG܇8,*\Opk4SΘH|/(aьVI3kmzg$௯L'mA0Wz_KEF.bJo9;=]aJnT=5#3GKs0[J3,Uo_7l!G?wO6aEj{ pb%;Wi/;g{1<:4ܻ5kSF_S="YF8h["FN}gOm9s:cg8:OE 8KFbp'XoٶP޼_Y[ ;޺XXk{X=]hXg(8tf }꾵mmgy=?P۞_\gsXs.Jn{% gXTs)bcY;QGir}m[-jno5}Ga _ѯ7Hzr\mjߝ'x)u~Ȗ,NUB?daxtЫ Tc3ܹxE"'qև޵D1Yj&6SM{.QONHdjrm?˶CnjHI }LҜ :Jƞg4?G'(ad/;w! X *Fb9I%㬙l8 XIƱgr*!g&nIe>iݹoJXkFg9r̖r\ǚϋ37+XbsUk{OSiW +⾦cLGw4caf~pc{p{%Fkr&MAE¶z˛Ia1YmoKi[}\ƘY6ty.z|i?vh'ܫBzk]ȱM>/y] Dc弪~2޿1/~fU};I_/gn߷i9XiìPt;Sl'ڛ nyڬ_M_y8ޟtG{7D~}wcs;_Ⱦ:}~j'ڧe]?}Tq#7UywM"_mھ>nmW@{k^*F c]ơ{ oO~|TgײWןLߩ^@r|1.- z%/W5{7Ms>{vz#9N u7>)tq33iN7VY"w,1/[i6s6tz@2~9}%s&:L$$񐋆Q]$ .cIEjBUKQ15늄ڮdW;AueՖ}Z_o;Eμ-+ƺXb@"hLn tIoAP p# mK,\_"A|Z+v1qDN[E,#cQxѷ^"UDDSlqW(1.|?qo6m~!Ab}")(F_"28)1OqƝ>Op.O7n=2@oQ;t[ } 86E?vq33oY8LXr c՚kNv)}=jO[Vhlun_\o(!ߖoOE$_~S{/# C2 cN.'^ ZOK72Tlbai⢟l68Ǒm6w{ ߫VIԭkڧ%55ij(Ѡk]Wۀ"_-E1)\ .C{Ru/jhϷ֍TkTXucc6j}j[ }ZonN{BWXD - EfG>"}b2b;˼0WiQkYT1Yx8CQdzg. QX4 B!,7x )d4[W^I4dk!,\h~ [!9'3&}ݑԗ!^RX$$^* 'XLd5&^ G#^O׌D2ح+I><;(ƭֵKGM_u_<$@wo=Ke XУfO _K]~D Hu/?wXDzM7SVx}{Qc??ov$.p?q/{x sa-}qr漖k /Ueﭸ'Dlfq[M7y=~7|VtyETOys~cS /mH>FX{-6f};kcD ӧ \>gh$,bL֧}ةEf[-\h` v5xk69X}{ ~q,ukMFrpJNWƜ=תm`$9ϠDf ?dXq1c dRaDR 1Oؗu5 kkڧ/3u&2k|#όv1j%a&={B=n +־~s^Kw3VS;@{ÈzDQ'!B ADleǮ5|z\LWq4_7^v9Zd\3E,49kzHG' \>g]њТzuy ݶCz{oݴGko|][.ت^Շ/,Y7îaѺ[8fBJ6ϡ͒LI*HIH#^%Ta΅6p$tl9<1I(F:{3Z@ x[1kٹ"ƛXk ρ>#^Kkϡ5KgQ/\}g{wD@q^ыafYؗ'sV5[0-]ފ{r"ƽ2R"I[7K@ʹ]ZJ?4߇U{Ts3lVy,{`ٶk;AbKki}Lu+5O5:\l7k'dS__A#ZvMHP$`rBSfslDcLL<~}bLl\>&\x=ZN"YV,vԾ$:_1Jyx}qľ*{*mt ;k9s+{/Q+qlK?,Y -TSSLK‚& ݾb@]Jy&CMC[򅹯r,Bf6#},qV54vhtON1%*!. wf-(澵MgztQ>ahc1?]~zT uqpݲez˨s o| ]|;hup9VDq7kqu3@83 Ad hI|\DP8h]!6QmduEblWP_w P}cpU[nWr͛vы3EoW` [̮8DgUcv4 O}l!\gEa] Qf}__ O8=J\> ֳ՗MO>v3}oe>r!* /ӳ">?xdv%Q1{\pPUGn m&Q-fz{,;BOG@ lxֵhZ:~.kDkn ׌s ׵8IIϐM՜8։[3I9j&l甬K$lؐS7%SYI:}N$yk3r}N2 [A .pCl)A[$\3Y^-\?1],bx|rʸ+}*g[ۭpmgs[ZES`Yd:ѲI0}fac4+"?z:~)?3흾*|g{yԱAtS@OCuP EU!2oNGѱ-(FAUɂxf鷈)Z귎uַnG<;M|yL3AHN[ L]놟C-5f>̋ʿPaK"0? ص–^ ЉHe}<~,eG1>mzo7!ytΉ Wp+Gkaߊmޣ5~vމso} uX7P$|9;q඄՚v̧}h!9pDC$qޔxnSQbO{}aLm5[Dd7 1%YJ *qF(^'l)QkV_s.\IqJLۋ (?}2[A:Lp] BNoqlIǓE\3VoEg[ҋX1Lvy+m\[b 8=2ʆHf% +bpm%:Hj{{Ư{8V?ݽ}U y~[tl#s[gD=8|̵55gbЫXHVIDAT[K}CD?M'="zJ}:B7ت^C,wnkDkswzX%A䮙2),$J$)0*#& s.ܶؗ^g˞_zzL_p]pmHoY?c!\}pM;shlµ{-PL/}bM? ص!mJZ_5Y'k1׈K}D:Ni im6Okbay~c*bzWY>h?N'PIJ7k9}ԠgsoSwsCڲ|*"o9v|~6ط^ם3 I AR"h#Y;QGIr}Ok5?$+Db }{kc{B#\'!Z+ꍭ-\]W䘐Lu9 noh V8'jl;϶LwȢ_wWH\.BtIsv׵p/Zg}ϊ,`K:sk*XYI*Z쟪<] 9xшc1涌-}-k^b֭W"4+[NO µb\6wǝv˨%/qݪηi?,E_W}~S9 v>ӛ&w.2&8KOZ9Tr=}MUoS߿M__}>_2}O?z#IL qo~ X8 ;$ KUHv/$dYr츠Cb!$~6ױDHv2V? I1C,'(qvqvSwQ,YlcR!œ_OVwQbV}H"CR}:3>.s׭=#ǑHT]'{EsϘt3gNq8 µ@SgvVήꌏt{=]`O.OǷj0J q-FQrTDQ)Bѷv5}X~%%WkV9٥E݁VZc{WXv"F7 oy@#\ש~[\'hB5=l='{x\Nv(syꈱ޹oa(6)djWן~ú)tI'3g@}|a/bN4vt}joO2t6̱LɌ{< wLjx^ȸߡ1$d=DllgO];]eaZC}ݛ ܵZPcjUXS+$Z藟vNƦcP+{/ȩ-Tg~8GSs?pKz5"k- :4{µ~I+?M-- 6 r$~0\XԢm1~^lca.=/~?xykpcs~RGPY%o3>/qV3'ę^Dq_3-*-W3q?1ڞת5;$:f5*_=&YؓCq=גcTnqh$?b ٢bu˙쟏 %C <9q}t|Z;={BJɝg,~s iFś,WgL?v,e:}ϯ9=Ɗ{Yl[A=}}so6תp}ΔvcS1+u'[&+[Cp⹹2iUWNu26ޣ_oKq6z}kkEZ\DS'&e%^#417QBz&1.޷-kq>ZLZj=ؚk5e7#x.{[.q/㰇0ĭIR$ Ø^O׼jnF+v3ƒ[<CnQ~kiKf:-#N&oe%xy}ȯ -̽#|w]KTw:a=}꺰^ ;drk}I7$кSg;SR4wm^I$ .9X!fa01gIRmNߢnE8ok՚g2vL3Rba1=_d֪\/+g42O|M{~[uA/s05&e`KV^H}KW>ԡ 6eXߡLer͡wߘۮg}WT܄.;ߡ<뜢kzüعli_jسVߵ?f"]_'Bw4uhΫGo۶?}\\S6z6@M몮f1bϵjJk2,\_[~mm<0Ǝ N$&0:K9bڷݤvc$vjVi -#f:"S&$LuoL2 b7_Jɤ"ń' bȄ#.%> 4Jg^k88l=f-\x} q?1ξߊsV=Í̶b*Mgg/ |߇1eX3kn}cXI}mk|l'zD^x3_>2?q;'쉱*;7UuΑvZ ;X9oK۽/c1 }vN˾;C>㌢w~gX}Yyeݽ` ׮hMhQZ n!7DS߈ntr#7hت^C,,B֎3ݖ>Wc{xbփU=ʇ1z遮z }Xk`|\>͹pb_H+r,5y6}7*\g-cC'|MCqN/Y'K`,6Dn~k4Ӿ5L/n,0厎ьkR *ƁqfMLDu[E-Sk鯘ԟKB7v0%:X*ؚٞ=f̖ q8[HQݓM}QwD<4jiS.;|/ qB"g^bry1XT<~oNNlJgsOv-~?=[^sû1͘kQc=ZC~ś(ֹ9}WqFAkc24dҚ&qWAuO}swƙ$j((p4 SIk -ğBkH4W~ ž4%\GNBW[^c71!kvCCz8+&kS|.[7PpY;Kfb?1:kv#^<ʵc8l%^N'%Fv{?f@zc/4a^u z_s1URihwDcя%`.gSLr"'dIYǚH8f%֚T1$cIc}LVVZlϫfq1!)Ikl)gui8{w#􇾇wz7n)3-I״~,i({>|_f,̌=_3Ec_pf}r>+ޖ+}2 to_rTKv<{w;TIcx<3!!=_׷'3֚3 ]S8pjlG/y2~M_݇k'`imv>ˤ=n8:SSlf9Y2#Cq\ $p:`nb2IJ -P"b}8BD{.<-(ɶ"qwܫ8ہ{gg98$>\ڙ{ϔA/$gIBj / \{N~5wʅ'\_ f+o&rMK韺}ejyo15=%gG|<=lg^^"(?,~=랹>b_"=A9\«WNU-JɾPh}Qz:RIu?Qlx)>b/a_n2t)[~HкWͶYQH}-Ӻg{"Y/%=(R_YW5CxzZeW|3(6R<b]ݝ,m՚k8|޷G* Ic` CE"EPt>/3'r.,sVY^Ӷ- {< Le_?U"tVm"lij9Z|7oIȶ>0CC-_9k(6 ʾX{cņ+ƘjGN>ξ ?OgskE8YFµi⛎NYP}- ׊WCZtǃmn ڿX+1W,xk}F:51~_Eٓl[CP;-霛o#cuu5b8_ZWG?g?K=GHF/d"H#:W ~1WeKIW ]OI==Bl\4S!8_n׫ X\KׄͻAҿ'Cr`نqe]9c5u\d'LԢdV KdgY]$ڰ}#ȋJ [f6< U쏵@_s]r1tݐC؃Ux<*.˛xDV)I&<4k<{0W.)dח;X iŕ:i&9,*?A}tpW9{dRWA@S\ӟ5D=AE,{vڋ=9;9Yeotl0ZZ7X}Ñ}sƇyƩxX'}X|)F<]L>kJ X]cїcF6;gU?[r!X! O}Q27Vҡ}IAfk-܏U2)) *U"'\/Nrΰ|6Xܨ~9yїgfy\&_/!?ƑzQfk/|/ k"F'?wGQ /BJҷ[>'x`}ZII!-/2gE5O2<oq&V_isU}b/趈YI?(cfZNMn XE'|<h~@ANc>`bR\LFWG̊4bSPaV&^EKݺS3%=M[zGuet'wBZqm9OCXr;,Kޖ<=O?¢xx>(v%oG> |ˍ؇U 򧁀TTqT@UHm"HٚK֚-/#絿xI:dOKny2TX˿(}6]r_P g_MnYp[i\y{tv8?#%(r/3ACS  hAV4Ny)|D%|bӊ+Jzω}M>%z>/xb*|Xy_Z8T%(aWj{ z,vǓ(hC w$ī9A\ǃ"x?='@<  ?='@<  ?='@<  ?='@<  ?='@<  ?='@< $A󛿹_ؓՋ߹ֿ?$=A~{:~?MZE\_$AVqǷOO?+<`"ݡ(TiyE_sD X9 / $AsA0:χ% AttvMY`AtPCˇnPEEO~Ya3zX5X]XՋ #"/&EЗ/_no܃׿N#e"?^y> !laC!8TB[q"*A4 }Y#FʟYgI)J;Lpӧ'>|sq[:nY&C᧷$g ɇ:[il>\Deu익jH^mpV{ Cޱgڄ m^>rCa"2nC&Tg\/eDbC|$Ȩ7~>lluzH)IjwcAPHQT(Y\/$p犘r`j󾇤_UcʻƜܙpyh6N׮J.(?h~yU(FPO͂Aߊъ_o|5V*ڬǻזj[lg%(엩?e{ Ð4E9ZXZvP!+ yH]k(xfcYd3_ 1R<KQ?͗NFRɚ?݈7ެH;;Q|2f>IoĘֻ@F㫱d߼k Cܒ^.E%۱ktIw] 'D4E/63iU2NI4l%X9SlI%eh}U{ |;D!tݛȪ8k'6 Ҹas4o2|ZWoUdW//&ƛ<&x{vו֜zɻzhwZoH,s=aEPRl",r P;ϑۖU@meɉʟ.3c[۞Wؾ"Loi9 zp74X꫽Pjr]:91yf<ݱ1nb{x0NKRҋk2X˴3œiȱzWqTk`T³qg''G|!֬67aͣmZvYvu*P\EF%'+~㭱L'תۉv\sn~gO㝜g'tnc=A<<&Ly''A|ԓK26}0Zj׬՜j AO}X #]ކ?J_ ?u{c^0olګ}AG\yu>u&KcȎxs{hjO&{MU \DfN>~d*YWwg2x$ty.fٝg_Sg`tNdx"HWxedU_ѵzmiFsZ9l3c"~ ${W0&E8;Axl!>DP`3S} !lǡ B؊CAtP!E[cʟYgI)J;Ln[˚q8(OZgb(4}c!p_f뵾ouױw֪Y#q{Yx "h{Ǟiaky(CL.<+WkX,/e#_?'UTdԌ?nA߸]#q|'9=!NTWB!!ߠBJ}!S8W$GFWcלּ_UcʻƜܙpyh6N׮J.Iۣcz[뎯ƒ bD j/drKjz'X~Zwlǖн'u)~AtΤW0;%57D:&{`XKO%#̣U)psuo"Ú7c]-6gNcVHM{;ۄtM} "zyN}x1?5D1qxtƳK|d~OCVzuWGuf;t یxT.BߕXfdcTpXyܶdZ\,~Om.KNWAX׬yfrmOīilSFp_qCZu74X꫽Pjr]:91yf<ݱ1nb{x0NKRҋk2X˴3œiȱzWqTk`T³qg''YmnšGXvV7SUaȇzsj_/O7W/d?Yoe=Q/VN4״os[ws3N^uul2{x$ty>M oUM- NN 'rel`ԐY׫9ʃV?#FM S'v&͛moN^;*q 6aEKyM1w}4H;~cͱj<^=I6U9$ ڋr9Q@}d]ݵt )FXCBMfwj}M=!nu9˓E ]}cU~Gͅ" hD=͊ =?3t]E AGSwUF /`K6+Z _()PE !l! :"lO $cAXc{Mx?Fo q<IYq!'{dH>zocn[#޲Xw:W(Fw6R[w8A~q[Z@7ϊE\`翪9W<獟?Qp.Ց8S'*(!WB!!ߠBJ}!S8W$GFWcלּ_UcϘ;#Q1MUŃ/ӛko}^|d(W^ε)_ն6R_m7M,"2qlOwSa2x"GU K O ?dŵ$+p l,4Ls՜^"~@v#.xfvhwCvb濜4=:.Pj, 7A$`B+fx}?u|Qvl{]I_8MċMLZx SR3x#8M$cbg V[aj>`Z2坜%QO/h!_Ws5==~Fc-vyOlL7ۛ(k~$NvB-E!)殼Ofɰ` !56+l量ړɞdSC(ѹهJ]ٽL  p"h?$dvXqYYWyto[ھ\(VNج7CU Ip4N~9[r/B،A_}Bq"ETcVX'sr{Ys1a yR}ı=&\l/2nG)},{0=2$wzocn۠GeZ5K}u$}68=!Pp3m¥^fQC?Alա@|C2| ap! 7\ R47~>6F1#ջWGLNrV{ C_  %+$N\PQ_MR~EV);wrUx|?{sm WczW"Mgӽ=-E`Lhg-]oؽL )QӼYq-C \C32 'Ӝb5Ȩ_9Ũ]Zi`Z2坜%QO/h!_Ws5==~Fc-vyOlMl.(m!JvB-7poyM1w=74HmCkl9V^=4K}'=ɦ*D{Q.s3'J?ѳ{}A<+lU.]wjo^}@q+v4$ lַ1Xww큣ޟjwcAK'8K|d;vMW7+Q.|=7!\_46DXT+P1l'KNի 9Ɵ(f>6~ss4E[H!Pr\cRIG83C* Gv5~/]+`,NdA\4֫h`jl=jLY1"_/m|9䗼^xXk׋WH~TS^ \Xmho_Lk?ޑ#ߧ~9'E'Ȓ~sޝ=-]oؽL )2ė69C[]y*-N⓯s5<, 4gXy/.y^oKMF}}t=6)E~]!.eJ6W39zsTM(j=s%e<ߥf,~TXzS:ِ.rMF822Z~]pqCNuxg,Ӻt|Fcwwrr'50鎍qăJA%ę3TrWҧQ$ %yj&kU32iadPj-՜c0qWEFe|8Szxu-kk{'ɓ~}j? \2㭱L}+WG;os[ =+W7v,ӽ'GB;ظҧQz !SIPD=y+c [5z5/~Fc#ֶoo`aoAe|#:bl?j͒acul9V9i/:ޑ9Lk9AӐs؇J]ٽL  p"h(~&]A:$w~Ě7DZW6O+d\J\^1 mAyLwt! ^i(9A{`_f M ˲Ip4P%AE^K9IENDB`pengcheng.mind-mindformer/docs/feature_cards/assets/Transform_Ckpt/original_process.png000066400000000000000000000205031476650141400321570ustar00rootroot00000000000000PNG  IHDRB9SsRGBgAMA a pHYs&? IDATx^ ]Uy  ixF H+mblBeU̴2U@qap5vZVN ZB-FLWB@ 9]wϹ7s{ZSϽy,1 _knCLdL '& c2ZYD0 0 0)BaaaL92 0 0 caaaaS3 0 0 Ør!daaƔ !0 0 0faa10C0 0 0)Ge ɭW%-y}~kn>%-ULϛR-;*#'kk9Χpܗ*սaT-m׃:R2H 0m ]}*Ȗ_T79w|s&߾bQr߼NwA~M\w7z/>bJD5=9Йi ,92"𠟳'I g7C΁1 h9v/HlAg[PhAk>gFdv>䰺s (h(+e<dW5ćϜ"E9~#º fuAV&DzxA٭YN}t!#k; 2v71o@q#=<}"eU5/m.)8\zJ*St?~S~lP2:a!8S~塆k>qR1jTYs'Y/L)^=I:6Z:κm9~!E<,,y g+J&*;ݞk4AÍtBY)"StfDbeMYC=nҦRuۨw3ͽ =|-MjEZvj0Q2:vР]=q|ַ0|((($u_N3q<ŅNWt4۷B=+Iq^vaa 38D@!:!DvD}m۟U8^]Ȅ! sp xU>F|DVa4`,NDHS[C#/%zP:?wP>ՇTy^1"$qxepSʠSFbG-F KH":ك&)ddpRyB}O K=k9_CWM~ DF!e %yM;L_I*bt|3>מ~B#AȐ!*2n5 /€B'tއ܃]Sc8dhQ"FwW?vn.& Q~dF!}(Z!X]bP oѵ4IblNqO=p"Y2)U;:e *Fǀuj_R |C1hYv}kdE=QTAi Ͽjm\@&Sm_t[<ʰt!}esDdAVD(KkDROY!fA ,]d^uywUYY1޻| oܕKU#.!=%m4=ڥ7`dYk-ZE'GgSXo? m%=Osh KɝLGk-*c b!cz7^:ò7}A:7Zfh+kFPL+.4/X>`2/0B\ܴ;3~!T%KW۷G_<#*w 2ϧx]ZlkzȐ/hB;0%O~K}rCHo0W]{MKF4E].jd*Bqۘ)wLYRSug}ɪ wvϐg,yв,X*ii P|72ɿto|E]Ҟ^ u亵- =yǒi_gBy^hMc^bJAy]1Ԯk["[:2/=0H !Fy ?O);va弈My+!+dD{1Rd^I ZKƶD!%PNHO)k1>~]KwrT"E,&1f|zW!-P}ƒw ?ʌvƀrN%)Y` n+ѱ68W"e4ITQ>'Ǧ;gQ#0|{@u6:}K7XGzm_ ?ɋ(:??"ktc(u+(rd6+6CkN[7ٷ.6Ԓ刎c$5?V g.n锳!K  E ⍷wOM@BIF;dG§QDշ_vH7|n!$)"%OS x]E,EHMDoii,_R*t5E#7NB-zOȕ{\snHq (S;k** (w[P*d= WhD|Aw 1b޺pN˱YyN\w"0DE %٧+z. ,Af`g,ԅD((tZtRh `̸ITOCc'o0@ Y砑Ϻ+7t|)1uw87@(uZ~Lkfun5ԒdNO+$7o|h%WE}EiLyv֊ڈ(XȢ HF.1Y#>B'E>hᓇQ?ۓͭx"F"FP;1PH* W m按]dpozQ3M piFEmC{7AAUay2t? t`tD+1 u#E2xkG0PXx0+˜$/EByX]K5д}z=y^ @J|,z%y.v=|"hB0-1sBg726r(DGoz/^dnKBnEeFޗ1v͸!ge,DBgWȋTd]Hj3|$Fʵ{EYϯJe^]q67jӖ݊uV۷}rDHZF@N14mvຘu/ޭK1d?r]7m;"z*[Ɛi绝2!pZkڒ`io+,1z98d5e;0:dՇ[<4PI"䩩2鱧O' .M q=Uy:hyt1F/ 債p@?Շn.nt4ZȪ Et}A#zjdӧ~V[7eS8BH^3bsT2:|'VoʋH]uYYF+ӟJFP>Tj55^E;Y#Tuu0"k.桏ECn!shyp<;R_qhg쪒#'}FJLwDHx-kR{pEv= +T43Z0 Qo޻x0;1(0FP; _AЊ'\N}ie%Ķ_ K)آӑmgp~RU!ixHC|EVR۝}%&:Fq5E\dQ'R qnxTQdu7`24"BV$UלvUi|%>IوP 2{e_[:3ׯk.W]$R ɍc 6=3w~ BޚYdD(nzV}V'JP,Rrɥ}RTNa"hd;xͭɠg<畐R#Ȩ>=Av m:^w/RJxfK#6Ev"FJㄎSvepe8R(9RP`/ ՅNb$F2%t)"93./jU'bEVN7`h8x10zl+l|AVtkOǏ!onZC!p0>aaa ,x/>o~z04iQ_Zp0:NM=_j@./TOèt>t(ߒc84 Ҏ?QvE\N-+ 1#]Lԛ˒E  {;V wVC=\/ S nnU;0ځD-0VN;uƒ@=856MPmhE\~Dk\ נeF,cqTU 0@U!&7Ńg5Qc VheCXP2 O'$0A Ti56tV#~fʋE"*T+ (|2K_dmE-NܢP4#T7RN(XK?opA  - 7g6ک =Vy$vxc1zpjH0"mÌ0$ GHá$0A o#hY% ǫo.O< uQΝlRH6y5pc:NrR͌$nGˢyM]Ι眵2 MK+M䵳|7@zD.?V/ho )(1lew^bˠBYXxn{ 3 I%@"豘teI5"S$ڑkϹc υh{1BOҿJ r-EtyI{&O=G^ۖÈN=Zfxwe_G cS1  !Lߐ1BBgٯ_ܝ}Y@CPGP)oL{'n !Cc}  YtSԝp < W4<*9YvA0vR0 !D t:e=i8sFdW?8A&+CHgVn\3dpREyAy`(ӆ+mƗc ےA!)9׹͘$BdyڡC%.:"$!miCg֭8V#yy=_dufO 6'4<w|F uRȓc(y”eP !䁎4zRJ3/t8Ir$"$7YClhG !Qꐕ8;1+yInKY~xP;҆tNg\mM?gVщٛ0ɃzUf" fiAO0@#[t'\ A#hX1`x$CPܑyi!t'_%@-mnC9*[(;NrZyIU`tĠ4uwk. YSi0 R03ɘNLd8X0 0 0 c `aaaS3 0 0 Ør!daaƔ !0 0 0faa10C0 0 0)BaaaL92 0 0 cQU Ø>|3lL & #E 0 0 0 0 0 0$k\(zIENDB`pengcheng.mind-mindformer/docs/feature_cards/assets/inference/000077500000000000000000000000001476650141400251115ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/feature_cards/assets/inference/inference_framework.JPG000066400000000000000000001367751476650141400315110ustar00rootroot00000000000000JFIF``C    $.' ",#(7),01444'9=82<.342C  2!!22222222222222222222222222222222222222222222222222" }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?( ( (gTFw`$-/<7"oiQpC1_ ͣ|_U>W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`Ea,|K' ǫBVu,P&W `\)QEQEQEQEQEWNol4EPIUu)!,lbY WpV8H!3<Mj I/oRjmh,>zjϹeHAL&iPE$}@IE((((((((((((((((((((((((((((((((((((((+:Au}N\7V hEe-Ρ/I {uV(_бR<),R,GC=>=-#7`  {ER(((Oxe鵧[&72Z-=&}MnWMT-$ zz Gb* ( ( ( ( ( ( ( ( ( ( ( ( (ke{{c]hka.|X˰i O#*9`yS7EƭszEH(%*T*cle `a;^'Ԯ, 3L7*֚Q!.q,/]s yBa$S5+p ?$c̓n2Nyqk$vzea w0w*H GZ6ne[%%Iz|C|ǞO]}7l bXn *X.W9;x=|est4˛%m,51b[YM!@i,UnxLo=saՔMy=1ø(CqÞIz3}յı-{a(4aGq UKsOfudtyn.[wޡVy0\-|- >u -;i C mR3`p7 $^M6Px [vڲc%r:3du=/&d?MBCgvZKuu!u 690ak_[DuK{AK q`Iݽqֵ,<m`!{;mFQ lk Trz+B,jZ&bt9c9gdxn(/<Y6\AmJP J|%c_A=7"yewF_ެpW N?At0~&Y 8V-Yv7m[BI#K,Le96ns Gz?z[1u-noXè-m(3#2,nʠ2GB>nZڵ/._ i(lߐ5"n5=CPݴe|*qEz99'Ok_GsiivƎ؁OM?Z^=և.ɨ\]I*Y;$_kr]-xz<m6 O1]5zse)e&qVM;خEfl1sӯ?§[[ح7~0^^0ܩ%u2ǃXQ ]/D%+#B]}F9 :w}De-I|KW8[e`{u9uN9wsahcTMN\nE&0.cذל شXW6T|lg,KkKR)#0t < {Y~U9*[#t0AA(]68OsM{I*Ki,drJIy\pmV^Od#y{pUx *'=+RD>EPEPEPEPEPEPEPEPEPEPEPEPEPX 3DGr Xr5-6ݥK/ʃd$-ER((((]#gn:[HNZLwF9-T,Wf9#%'WU >e)+ϥr,ԯ10)܀ OGTV'L6qie_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&KSF}սWlߜw8Ep:+ +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h_kFLG|WA2M]ݢ h)h/5_I˸V|WA2Me_o4Ywv/5_I+֍&.nX_e_o4}ѿS/$e + +֍&Z7 e,Ea}ѿS/$_kFLEp7h/Z7 e> h)h)7k:1\(ۊ?526m]}KT3BQf(`(Pw=O@ tP=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ~'x0X_;Osmt+ r:;1r*!_H躷l[/ˆa2<:t3<C&jA<F%.+= sЊ\2j^(5;UUw!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vWh,(_4oK}V{hw;z+_4oKj7 %>P=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vWh,(_4oK}V{hw;z+_4oKj7 %>P=;j7 %?|WA EUފ?|WA Eھ+"moEqھ+"_CFdGjs_CFdG!_HC\G!_HWѿY/$QZ#WѿY/$Qh,(@vW5ob3lkn-4 g>ď'szWBX.fKk"{ ?RGjV-4SER(((U/}?*B)݁||Cx:U1E8.ު!GBF~\>hWlPƋ`T <݅QV@QEQEQEQEQEQEQEQEQEQEQEQEQT&4ymV97@=ak8JrJIl`Mn-TV.cwzsӷ>)5uaϣA}SO,]L3n"2c>/ϩ^ 81G5|dBzWmwSTE|Q=Δd։l TɒH};uz"ƊP;\MGURzkc5}֕th[G ye`bNHBy|+t.4it"8.eW+>e/"jKd0 Tʷy6e_ZKHQ4eYT0 3޹_%E-{ϏF`z׷%XPN7!U'p?OTjK?vtvWM.ϿTA'թH;#h`ñr? ){ Q*#Ưm`? <;}%I{+w'Мql’p2 ^CYu2K [CFS؊=.֖8oSӮ+Hԍo^mxUVōoR}cb8WB }{U<ik.mo57$]偟8O=[j(Eeͅm2O1޽_U_r7_K-,%X~Ҧ8-UK2[3g9dԭ!we8y}qZVSjjrC1!vT?fVM3Qleմ9bjiXU\9YK{H/L`@uWd:6kehdd ]19,ǹ4r{0屡EUQEQEQEQEQEQEQEQEQEQEQEQE 2CpLo5q&9WI $I=,{rPB΂0G:{5o֌o#+;(((( cEW2C2zeO@#:u-:cuf4l=HQ=0?kZT[ΜgĺjQ`gT#$S(t k}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3(}]w<wLƏJ<=A3o'd4}s>%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y%qx{g5tQ y>u0p֤[;l z--/T`r@F>W]F×ZY5V @r,2{JGE$䌊먢RSwb((((((((((((((((((((((((((+ ſax@ݢ((((((((((((((((((((((((((gTFw`$ck%b}FeܑzossX8 WM]B_;Ws'$oRkzT%S]\-AFA{QkOoH%je#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>OiZ ߌ˘b 5ڜn?V9-g8lV&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>/*k}cŸ3hG&FǢ#c_ ͣ|_USm>=.rSrkt(g5n+I%dy݅QLAEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPP-h0jBW=y%BeJpB5r5?7_Us]YHVbBp~#Dկ58n-uo-UsL^jj?S5+,Ui2r:U֝W)S[Q\\gsH=Ln(쮒 *Op}<*zZd-gӧc% [tccyVtetarZE8|:( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( œRje oJj1[TTn4O4i:us m빘~olU3_?+jqwHF͠cǽlG$o{[ޏM)-RXas󞋎δ4-֓[ZI34e ͌qjQB[{6>CWIoh06g&.~h_kV\b4rrYWnsv4RqMI76Yy]\W}]m㍸q4|ͨ_]ȲC1c t4Q9HM LOܲ"tbh(((((((((((((((((Gx71i鬟*C?_^k4:R,T}zNu6c+HI+,[;W.&Q[|ƅQ^qQEQEQEQE^x>i{ 4chGOr'ֺ*'t&GYx2R/*+N{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REPvqliﺷ"͙"~qw[OOhN_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8ECA3REg ({c_??U'$Q;OHŠ>W{(v8n0 ۊ?qL, ıs޺(u>إ6:DeXvgrIQX}((((((((((((((((((((((((([ k Y nמN񵇆av%5(Vu`wazc0PQT4MZ-wE`x!K\(Wz#~ ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( >shOwf!AP̥y$y N7+bȱo?UTL_4oKj7 %u }/m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$V][Z|6Z0:E`FAGWCEŸ`F"cDD@Ut@t}h,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$Qh,)Q)m>|WA Eھ+"EŸ`7WѿY/$TZ_Ϧo{; 9"u2H&VzmV'TTMAY$QE杁EPEPEP\X*:akkc g?WÚakkc g?WÚakkc g?WÚakkc g?WÚakkc g?A't9giѴػH֨YN:ET,t+p#\~5^[˝6W{bHE`$?/cpMiEQEQE.WOHأ#.x<Xl4q#lb֑=xۤ2{_gf\Uy`Lat䈱د`v'ZtP*YGG}|glcsljkg0:rDXWv`;|:( ,#>uk>qȶh9˶{z~5^X53u9",v+;0>pVʖQo8[4X=?,:;8d}+NK(ϷZϜl-,urޟA$ᝌz%Q\ N;}+JK%a&O\>5խ.KY' 0$wzW9xHaA%R R1@^F&A*s00CƠt)=QLAEcxG:e+eqo9W!Q Tuy+I E*tOֳMl_'tzE_"K{F/7 W$Eɏ>֝'DG?MxmZ}LñUL9$wWi~![}(^[d|1;U }A&Oxĺ9 9IbpzyY5u]lWiu=J$񥆉//u-帔A8-#ɍ94+mfKp C+!~Q\jV:.vod/=* OWmbҭr=1߿*m+4/kx~ey[VF@U,OL?j'ZJ|?&qs kJ$ZXW:+μ)]OZȶV;M#\ MT}G_ƭ]v y6%H c qJZO9&Q\/#4&*B QvdsY$ҿ3ic';Gv֊*mtWe㛫Z@[.c9o*RӺvc  Rujh+>!Ǩ\Omqa l0ocYJOz:>5a[ɣ%.[sp?Hһ:Z(fvم;y>8ϱGpM?O]COdn pr? YZ(`ީ!\CjEQEQCCc #4P=)z쭦M[S0%h&;P~_3=hR +esc]kޯo2j #$)G9j]_\-b7A8 @Vj[/EngEq%Ԛ &7A3G|+ܬ$w{T~9Zǡi?n-nDb7nrGJ=?\3Cg=GUyHMܳ#0vuPϧa4o Sej'rEm!l8۵ͲPݝ[Zv<e=9Hךɦ]YڛfA Qi7wi]h^9XJH4hg;q@1T_JLl'5:EIQG~ UF$g:kvWח öV}n i8_L|U|dgZIYgCy$j0J95.=z.aGxnL 8ȦYQ::V4^K_JTd |zHHFuWw\ֵ9|Qc3ڃ0ѰyJLJ֬h)<Aa^KK68sm}?L֋)kCШJƮj6Z%en"٦,<8@eiG@Sʎ+a#8ҫҿ$wTWxJᦽbH“;Vn5kK kS}lggS搻8#$<(֋6vTWx{V[ҥ/m;a8QLqu:G8 ,"- w PIR[؏_FZӨ:ROC 2@ {7_^m2R88jkTCVv*Z&UsqyV7PHm'dY6)q+J (k rl|A EAJl.H_^yz ݍ cw+kx@D(((((((L$$r@%88<{]&.%PReE%\~ GNͯC)5MS0ϾԆ^3o[Ug~U?Q^AQ@Q@Q@s9c*cQUש2YEWyycqj 4m>c}kYde ʿݑ~W? Y3gH)67 hϒHw鸄3ZEeu Xdr"9o$k>wP"@K:WYECmX&8?xGVg[/ xJؒ2xP~R k&n;Z1叔½ ^i>vrkZ7:ʹ-c}Gu۷PفuB9r7u;šבEuw,E4 Mc'՗xUPǍ,cx[(LP|sqkRi_p9Uu(u-~YFGr7 јm'ߠ'%-;}ذ8Q$E$~QۡOC9?\ZYKt@x#F1=k"2*Yw ,a@l/ҋV~N9I|3z߆oD^G8,rĠ_FG|U@.4_iW0oM[g9q]N26o[S+md;; nv*1zqxfMxh4D9bxlN0QwBro@* l{p ChI[}ɔ0GUTPUF C( (^I- 4,-Qԭ$bf/`>8888[̑#me+q!^3|EG&уLH^UZMBXRH˺7 ` }+Ӌo#-tmmJDQIc D;t0 ;~E2w I]`L»:){8smX4j{Qqhdo,b[4ka->]Fmu}Zko-,vU{fѼ!i֪WDe##瞕IQu$?|iְZw+k(Ga@ ?-|AW#J[Y H40Do(ϡQKi#> ԏ +ϴŞVYC1#z3j&n%X @1O]mN^ַ֖8-hhZrYlAqO 玦h^twF.K)&F(T31;cZ:(ksX,|1#P ݼJt2yҪ';}Ak[X!}$28_﨤E]xOV5mfɵ+6;`(ny>޿Ax3Q>ޟZ]3om{⻚)q_g\2*Yw ,a@l/隿/o_[hX 4psoN;}(uuǖF@T?8"[/M>أb֊^?AI.4/ Of,O*vQ*M! oAII K/_ܙ0-&߼YvOVꭄ|c$bUIʹ&PESVeR gJHZHd*3YsQӢŠa{ t#;'929r=0 (Q@zł&wЕK<ɸ:zv -Bu}BHEG6r3[tT8߯k^e)XEA[ǫEq~ĸV98; EdQ@Q@g3^Ks#I' F;eEgD8,= +KX@GeR-V'U3?O%?Bh諸@(((+cQWG\X*ƽI( (2Hwg ?dƓ iqL֭s'!^?h+ZO1V̟xzg d}? ?c'j( ?h(GO1ZP2~1Q/Lb.d? ?c'!^?kQ@\GO1G1V̯;c?LCۤo %1z -|j( VЬ0Fƃ 0IEQEQEM< LGTPN~_O8xRdL2բW{DŽ ],T\^zƨ? ?c'j(<$A2`ʤz׌~5G!^?kQ@&!w U#%c?h+Zf 7A X&2(QGO1ZP{7xIe1HEj'C"~>í]l+3 ē׽LUB @[G4Vi6A yIn*V'Vu~]?Q^AQ@Q@Q@s9c*cQUש2YEWyEPECswme("^W 5C-9c7s3l~j4fjY?X Ow?nH熧7E֢" X Ow?nEdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]+p5bT[MJXӯmi z( (((((^_/.XLq}SQo1EdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]+p5H熧7G$V?S]fkQY?X Ow?nH熧7E֢"PE}:GjT -R}VG@Y袊b ( ( ( ( (wL{iE2:UA"4VQ ?u#ߍ2:OH熧7JfkQY?X Ow?nH熧7E֢" X Ow?nEdEc<5??"  3Zxj+tEc<5?.f Wsxj+t\,j+'+ WsYVO$V?S]>#t[dʹP 3VdSEyn~`TLfV=8O`&\^H ,GVQVgN_2\?n[ڐJH̗-%;omǜ{ U(EP (+[in'pąݏEP2MCj6W3 Fppx<a۩jl$Q:h 31u$n+ 3iwb``'=;Qu{jQEQEQERMgNRQgXrj ̆{ 6N{'סzpqN[Ywlvz{yFLlρ#b`H"袊b ( ( kƌQFYiՓiBY Cc[dX4%b0ʩ^F+ ԍKEQE1Q@Tmq\GnFH HRS((((( tx㕮twMCSdɹsOP{ u]kYq٣r?*sn=j3?O%5[f KqQW~SQEQEQEQEW9?Jȱo?U]?z/QE{gkÚL]ۄ[h|@)e&w#VW:bٟMF.RCN( /j7ZW )|J>AvOyJ h$gE HY沬ڧ+ob۝_ͮxH{S-u7),Sf0n={bi5/kqk,:nr [+gA5q:ܺ[BŢjj93p+{M-4CRy5,g9IoQFqZw|W]Ak=kH#&qa}X5/mlӧ>Q{Tv݄8䜞wl = G@={W9=ox>5KwhzvjYG4憯a񎽨>+k붶A@׌V6 -umΤֈYNf"EKۤl0[9 9?J~[ vI/mY+9?zRzo <.DHD{]o_~յ-ZNV8~̱R19\=kjz5w$ 0#>((( ZE싍d)x56tZ}ͽI# 6ʠc2 *21^+iAB$i %,Ib$ud#hڲ6-b*ġQ'Q\4/ uIF $"&@s ;֦2 ˅JqZeޓXm8mu &Og짽_>׬$8QZ7ZKq;$.zIZæ.˵ĨzrHSѢ)((+*<]dvzKZ67\Iam) QE1q45m7S-kۓH+}gz+>n4>{;y,wYU̸ZQ|Kxs]4Z=Y`)+  =NmwŚwmm=άoUT#>l?q]7L漝.._50q09%kJk(e2ŧrZ$.3Oy䩦 o߆Sx"{IɹKW$eQ{Tv݄8䜞kڎ"ki =x?M{ A}WOk$ ;9:]c?K0&-*_6#a͐ITӌUnzyxCcK]b4[sss5q#t=io <.DHD{]zs¶ͽܓ]Z^ۂ#˔)W8ŽIM}C6|+b NyuXdn, @N:`V<]GSwiprp;>տe;9}KU;@98<ԺGlH/{<忸Lk/=M\RZv: (2 6?ԎH"~ZֿҚLNZP!c֪3o[Ug~U?Q^AQ@Q@Q@s9c*cQUש2YEWyYr/T4mp'OLa\ (VPV$ IpF^ժ{Yj qeA= ?ULN0FO&hjY?1}C~cZȾSvf?mh VOE z4d_ǩ߻o3@X֢/cݷ"=OƵ}C~15?mhȾSvfEdd_ǩ߻o3GE z4jkƌQFYk0i!C#EJdiH*n.'4b+rF<doNs*PKEQE1Q@Q@Q@f\C=FeNnmש?O^a2 [{|iT*z؏CS wE}S`O`_D*H4;(g[K9Ynd2> ЍVMbdDh2#5:yc|ڴQ@QLAEPEPQW0< x]( _K"SY7p;+~F 5Uԫ(ea"Ś*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=GO?Vúg>Ś*O?V'+jQab['+jQ?iǨϱf?iǪR%:h$@ygp?wfúg>*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=GO?Vúg>Ś*O?V'+jQab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,UyB?<[Szk=h<[SzЭO=Gt}4UoЭO=PިYu1q,o*'kʡPwbab['+jQ?iǨϱf?iǨ ڟ{XwAسEV ڟmO=;sYmO?yB?{9,o7_-<[Szee]xFm KYg0`$cQRLBJIwtQEyghQEQEQEQEQEQEQEQEQEV,mn,mnEQEQEQEQEQEQEQEQEQEQEQEQEQEV oa=?K!o'd4EPEPEPEPEPEPEPEPEPEPEPEPEPEPEPXW_>i ѶXW_>i ѶEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPXV>j ѷU^]|LZ|ZnWQA`#xi aPt[9( ( ( ( ( ( ( ( ( ( ( ( ( ( o'd5^}7>{{RbrXW1yAhZiCu4P$G#B*QEQEQEQEQEQEQEQEQEQEQEQEQEQEu#`k[3Okɧjqm-D̒WTo9P(((((((((Og䋉.mѦBl⠂q8ֵjȗ"x]w,c#0;skpcc%,[|qFx!PikgLni˓Fpy|WA2MiC4V-b'3ź) G4G ղ_Pf{vhqas_kFL[Qp*Mt7VA٢ drq4zDp[%gfwp76a}ѿS/$֔MulTݚ-n>XppyG'ۢSPPmյ۲sn)\6qN6:osԂī]lufrNM4xvbdyO;T8E* v}[ :Hs?֯l~:Xycn8۷9=z(W {kOLdsJru>n9Zq*[X,#+cXppyG'ۢ/5_I)#/3=4[|󁿌N9E;RhẶK n7,88gm}1&;#9E+VxDKhm191H-Jgմ02$z\̹7?C[4Qp .ݻC4)l|[p *U5u'S};psb1lw,1qJ.US&t91Z=@QIղĐYI>C<[C:7PۆKdeAc~>%׎TxX6G;Wx h)kv9=+9aʄfcq$u>jГm:,aAh;|X'MbM'z7(h(EjY ʏ}tU>vls@$:@ k{\7r kna(iOE̐zz3=!v}?Zi\Jws'/3]tk_;u.xI'|WA2MnE_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I+֍&h-R'Wј*h4}_L"AlG'zQEcmL9Re8*}Na,sji IpH?A, 5/-3t$`#h QE (wXѝ*(3=h;,Tgs+#SW9spX.~SNRìݩ0p0#SVчL,N=?YAѣl?eʓ+֍&h /Z7 e> h)kv9_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I[_yInQE~g-~gKuXY H?X涻fK["^JH ijP.Dws AAL VvIy\%Wۜs h(øο=r4V wI|p c;zn5ԍ`NmA-V(@I8_6LU(TQUz Z/ /Z7 e> h)kv9_e_o4}ѿS/$s00 h)h/5_I(`a}ѿS/$_kFL[Q/5_I+֍&h_kFLN)Nq-Okn.E}̇l{x> ?>^K{Xd]d0>it@hsV ſp>QE ( ( e5%A]y"xLMFk(mLMiG;`8g,s -nc: >Tӭ$-#FAVkt}2Z.OǧVot U)8x4גjkGV[$b 'F>n:m_OE3N|1w wmJIl䔎O8| TChC=AZx~kuޯ,d||qz֒ˠ鳾KSp-y[5.뚟VƸyq) X`$~#_ן_בTW)F6K0n$|Wh铐HJ (~"k^8́%O6ToO]CMA%Q?֓RK .R8 yX@I9֕k($p`>iQE+3:rGi?$?ܬM$~$ ,7*1օc阛4P6袊@ [ڼZt8齕2*j(C .P'6۽k.F9<uW=J&'Y5tB#H¹ Dx3Q;[ί4ƥeܖp ܙ>8k>Kymvޙ5*Z,vVYi+`&v:yzѿ뭿Eymޛ} |-D>xV.*fvӱNSNϟ>y{&ъlSTsp1;/]i/g{+2iEpGn BOA7S]Gg>ֳ2B#d(%Gj#_ߗWh~)6姷}6h[U20{WVt'{,[[6ֈ%Мp1vSШ:<:o.tX?K"2R(YĒ)qC`⮙jgWwJ#Gt|y[GFːM =VNVV- 9>zgTnɸM&{0}42?x]/aՠ_ku!@Y w98`Xsnɤ 'n,H+x >fIW#fޭ}~' NͮDδdJ[89'Tҵ5i|vn#͉ 3\vB߉;Gy}-l>C(+9lsk'WtffSǗ#G{[ן~`Κ(toQOFUXfX wmZOH_-e C(g4+R,Ι4Zv`^e~snFN&Dž& "HLe< C`sO?; EyEunM3˲r smͧ %gs/^En[[#Kğg &nXz mR;+͗MyK*H.x ֏jd2~o{+;ܑi,67 =_o|Eif iP{7i ~+ooYco塜qnNTq0{]I(?GMIxN;?색:5X0?HPoZR#Q%8 ۢ)SQ"-y^x`'t,jy=2>VꦣE[$+NdXze}֭ER((/z[SQd{<(xE=5袊@Ī]SVilI8I_ۏ=nUMCORMYIB H-n:|P12?ۈ| >)G|65+q0|9X\ۢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V&F?5 7Q+ _ ͣ|_USm}|cH[yd SQ~M.[XO[xD~v?^rk_5C[?rva[SOmPfv2K,s1xQH5u:vo(q/@!'pw=mM;2T}&u?`2x}'Ǣ1v `Q vɧ8д8Xm#(`7(6񟅕t`}'&F9_`EakOo_ ͣ|_U\ݢ5 7Q ?fѿ>/*W.nX_SmŸ3hG+7h/M|)C6TkOo V#$%l hm.)C E"Y_ \}V4 t7$ *NTw$ XKcf詽.J1JP =SlMGUx@|b$m b27 ($Z<(@OUU,]?50/Xr=\3}Ap˺s?zZ4 }e(#]LeeLd$z' OHzΟsd4h۷uQ_k !ң$D?4Sm} ?fѿ>/*M|)C6TrsvKͨ[nm'~P1}+/_ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`U4+![tLJϩk/_ ͣ|_U*Wĺ1bpzW.nR3R@P2I< G\KZl*6H}\O~${LGˏTRAHȢ]!N\k?s$b GXAdQs[tQCw)Vt9NY''%~[n?5U5 > JE6da$RvN:2ǒ=$A + Ꚏj2B:^dn!t0t1` d">hasR B{;-#hLX`GIi IcFzұ5 7Q ?fѿ>/*W.nX_SmŸ3hG+5-[[b5۫ۉUBp8qVk _ ͣ|_USm} ?fѿ>/*M|)C6TrsvŸ3hG&F9_`EakOoxxC6 fM ?\۬;kwYy :4~?vBz>ժ%ep7 8|ֵlHmc™$rI'I$rI$M=QH /z[!Y<,q70wݸڧ(EPEPU5Ga3‹71Q<)պ$Hc$9¢D'O4ߢ~u | ^l|.zh_eeLc8FqV`^wgYEeYZe̙ qN9< &ũE}l.ZeL䏕cEC:&:k}̻;8{,ﭮ&m2D# ʜ4YnƟH'1I#5d`pA=ޝiZǙ^X>:ɳ9vF0/QP[^^e"V_*@\ddk'_ ͣ|_U`nU4SOՠikU4,*Hލ?Th}6m%"8ʒ38f+ _ ͣ|_Uii@אli-Y6Tǽ;4*:qJi,LAW;q0k7_ ͣ|_U`nU!44P[I`$b w D:qJi,LAW;q0iYnŸ3hZP}ƚu(o ne1\;Nxf*ڦoJkh DE[Nqr+7_ ͣ|_U`nU&4}4jS_[G`U\]<"*wDڦoJkh DE[Nqr( ?fѿ>/*4}4jS_[G`U\]<"*wE誚0,,K5-q gYSm⨳`nU5 SO`Y+k8Y,3,j[,@ڍCT}Jm%83x tV&Fֹ~u | ^l|.zfUGZҴ/OSsu~1n#8T8 ,ҰH6gbpI=fwiAy}mo5liVPNXj:֕z/3{qG(5sf]FrFpd}~'v:us&vCn큓O‹05hj|ZZl2զQ+I`Ե>f]FrFpd}Jʳ?t˫3.wl rx{oڡg}gonxLfVU?]ZǍw:dd!5i 7X&U\ g IHދ04|lw fS*/@wVӮ]-ncgr22 fDէh4cOW{Gmr0\)'#zwC%Ͷ>pW#Y=?5mb H&bI' >]6(@QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEpengcheng.mind-mindformer/docs/feature_cards/assets/inference/inference_process.jpg000066400000000000000000003506171476650141400313230ustar00rootroot00000000000000JFIF``C     C   " }!1AQa"q2#BR$3br %&'()*456789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz w!1AQaq"2B #3Rbr $4%&'()*56789:CDEFGHIJSTUVWXYZcdefghijstuvwxyz ?S((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((_jVdbK˨m#' ?5Gzw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƏK  ~A;?/7\@VG% hAޝqkY?ӿ.?zw4Ed_oNƝ4I]Q5=ݎZբ(((((dGoK,j2>I4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O +;M'_G$O,!?ƀ4hI4 YBi?Ѣ$)e 4IRhFI' &AK/O *[/bmx0a ( ( ( ( ( ( +&źp]:}4ak+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7h/N5CTwt +  ål_Uk+bv_0_G'~Jؿݢ; :WQ ߆aҿ6/*7hh|iV(u2YQ/#$j٠(((((((((]{ÿx;+'0N@i({(ڟ$Bkk5PD4;v94,= RG_Tð|K$5x]r<"0 |JlhEQEQEQEQEQEWX&xV֡|-M&fkk{QdWg_+xtʳC?DW~?7~ ywRu€5,?d}Ÿ<8_iyZJ$ܒj2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP'|7 {5G2w#UP\~x瀕X`5f2 p:u Z.jWu_N+.$ 图]`RHKQ&ixoa=E>r *( ( ( ( ( ( ( ( ( cO{_N9iu5?rh(((((((|YKx_Vn.Y.OkX^]8)('NN2|f|H-אiv1yWw8H+0x=Bo(.|9m8_8Lln(8$mE|QxVǯ+Kllg]cn1iɂI1*[T8iB/m;mmV-qrk~I?>n? |yuZVQCe%qȻz0GҺz߈b[ύ3=l,%ѵE=ĂO6M 1{}'{_׾ 7DKNK[4IRgFٓM4敖 j __#*|e|;Ay-^kCffZ< v1$W|g\!~ź?tv᷸k_ [R>/LԵ ib2k:W$ܘ *]A?lZ_fhƑ`" mZyQ l1z֛۽JM~}v#yEBwb/=U犢$R>PC93Ҽ-1=|qMYw]ۿЉ\Mc~:*[GxwZwuƝa9amv +^Bw\ҏK& i)r/_-?l߆0][G7%'U$S]iMQm%55oy~0 )?ц)y 2ZomWCbM،qYc۶U>| ssh|_qclX[ۜL/|7_*kWjŮqkqlXxBm, ӌiyyg=Z,LR4%g&Nn5mGvy*,}v;#V0IX@A'D':?d|<1x@[cZ[ǪN,R&:&ѵOG?F]+{5>nۦk<kK*X##'x >Q(hJQA~ h[ezGs75 o|__j?htrZe5_j; _ugw?v}W/xJ_Yi^":rk%6Ww/1![i\G|oxHnl4R1)#t3$ ?j?~)~5|a,Q};CU񌳪Z4ecfc?oοh.5=vIw&h@REYkʦ1rYr?hM__C ?/P$P$PH 38 ??E>$|D񩻷?|g'Ԡ 4e_n$WUJ׺"7QEIAEPEPEPEPEPEP H?u&?D>ػ4t|v!GI;OYif>&\c 7RE&t)AF(+,T6(cSۿp?O#nj&?? +,T6(cSۿp?O#n¼g~>5x'ĺZ9u=wfdvF'cN@=HӪW秆l+3o k4oo:G{[JT4Iv#Io6k. #Z^:WGxk:[=Z1g`p~PsG9ׇ/#eRMlr>uYUzW kGĨu?"Q=ZEqk@-|ˤd"ѳsJ75n]Q٧ zO oM-)[Ɗx|ƹ鞕濰ßo`!Oum4O>D uFI>e>k6Ǿ)NԴ HfU d)Zu2?I%;~MQ7koknè1۟AhzoAiCo\W~,m[I-|9 AV>4hm蒀Y9A_|Qׂ,,o~fy."2HCn. Ú~/|7M ESu3Ğ5P l*23xZxx: p R p<:kYo%IKy8nm{}mEsFPFoXbJˈcfb2@]~\P|!o?.mm>Tt{E̲xjA`ysѳ } g~uk_ƟޜxVeGEe|E$>T,~4S-gbϗ}OR>'Im폇5ˍJ̞6x7P-xAܞK /uj;U^P#o]|?~5xhzֺ6%ټZ[Ƕ³\V*J0g? |-? ~~ ֧;GAu }F73#= ;gMQ݃r[ߢ龾ZVQzk]?F) ǚφ;Z |Fռ|]|8Y$_5A񖜍xDY Ѻ#GY$8-~"-jmkoO?> %zßŏ^-j%$UeH|tHmmVbk`5/¾~ _kk|d`bђ]ss]uImk K>| xu1NԾ"kv4 KF2wJ;˟:˩:'Ŀ?0gu-Ib-Tb^/(5Qc;JJ)_#R]Ci6zmo4q RH7pLu;bZN|7ҭuG#k^Gmm{b%$54}Ky[])}˚4[Ixq]"h꾥zE x$rz t_~'|6FpUvMsOG7|B[,<d=CZE.2eM댞ku ꩩjތ׾ZSp-G`]Enݟ_~𳿲?._19Zԥ Wq\}j߁Bq gs1G ;ѯ*Sx2—~9ð ꍨ?fŷo}k/C+q|b=<|,nwЕ'ߍ+k껔ݚ][󾞌J߇w]׹W-~>xFt9/'cy|;3]g4WWM*Q _7=@Ex5~DE_CWWM*Q _7=@Ex5~DE_CWWM*S&x`IOj y!2s?QOmԿk~W\w:_|<.^lnU /rMpGS'U;mhn"N$GS ((((ZK3Dh\5-p_~\W%>&'ھ]GW2mG[>gvJ㒫YU4$|u̫6t)E_2bg[i#?'V-Ojxn1򴻍R%v@Ar#yo28xK~~|C{_k)c'PĖ6q9S?> -.oƿڏAZ1E$|HKKJsZޗug}S1W][.'u?Is¿ tx_J$/5aiH$ w !XҼR iյ~%x{YҡK;g68U}K)y~,|{]2Z~Ǎ֯WƱG{:Ȼx9|^{'O vgt]Fl ]q$gh|1EsOkBٿu>ѓO59iok"gH eTPHZk޵^s)Ԯe"DkvM.3_5o 2|P{&~j)ŤQ}to,v`+ zҭ{wgjFkͽĿk-1πz7.t~Q]99~qcUnu\x~-t "Y-QD3Ӟ~#O^&Փ0jFtn8#׆,-o4km{QVmn,B %vp3~ឡo[~j%99E_Wz5@+`((( ~.ػuWWh??](ܨ((((((((+u= ;׹W{wKr((((((( .!K9d." ^&#0*H2i=Ʒ>7<Xo??WiPKXIn]$Q-C&WgK?z.h-`<G9췢&&-_\> |Gog>hۉ-iY7Nkg?+xSߍa]o ‘[j0-B^2HϽ}o_b4τ 6MyEw2㿙0vϾs^/WuOj:%ԭN!o Ќ@#iwڸgoY[k~8??"Cn|?{,[HPd@2)S 75#/ޕ{? xRD,6WĚOt>[ӤXP>\ |+9?d~_t?~m,+mJ@i|,4$OGv/>Tj:)y&KrIcn85蟱g<WZnmn|35 A2vPdr.߯ޗn{tjKKt_iuXNÛMq[QX{ F"w`pAXuXI,+U.5%/[QxuPn.YUVEQVt]r\wWk_-7RYXpA51Ʃor;8+ߍߵ7ƽZt_ l4 ]EBXd%ƽ? X5y~(x :fZ>#,Fr>n꿱Q^?׵v_}eKj-'s[\'ey[M(KDE;M +DZ\uީ.h]4ws[Ew,* l*|ɴ74Zg߳v}>l5B$Aq;w.Wv7%֡o+ַOl|KS7gំ쵹+v9QJ7Vu0\G0!ػ4| !;I(((>2fLj/ C::EgnԸW 6 b㇅_iVz.kNª} ^K$?iܨ_ |Ix?UC&fz[fpR|Guğ 5M-W$KH&fWW(u!,'s5x/?e˿j/fޙ<D+XLۏB8 .MuKRM}ڿ#}s_┟ ;x,{x~=;lfn#UV%MCV /E7-?i6OOeIF ^[qngOx?B\sK[-# 8?W:ׁuxqxsTk{%.nUe2 jIrVjkL)++[|ӼM[Ukx,˹396ki{qKxSx> ꚑ1qCek?e.?ukȕئI_&}6v!mGs)̲/b2iCӢ//|Wlao迳ō7Y^/ kPKK¡ XAi |y/~$|9a?GsBF湧Ǣ{pIXaO8 wFfgQ{nqH5O|r>#MI4QyO3'v1U =BKw}OX_j:TbM>[{[UݕŤp",7$~~{~6Լuyqx^hl{mkyiʧs_B[5ߎύ]wITq$`$R 4n2~e GBEdxgh>"4h|S|M[ǿ1=U@]άO$ .e}6~w{j{Ǖ}Knk/>w; u %tZ$WveIRkSߌtc޵MSZtAtXNJmT˰m7τ^8ѥյˍs^lgQ$房p>T9~|Ÿ?þBhCr - ďref\8`SWJ: )|%i_K]/˵,xоg/fx KA+fA C+ ټKrĒ q^[ƿ4?qϐk^!%ֱjgy(bd[r<` 1q_O~#.<- Ų4ݖkep]9>{?*V;cI9mʡëCAOKO-H~YG>< BGAd]wCMB%Qr7+?e㖓cKv~ 䯄`dSɼD8ҭ|^~4jk_|}i-eڼ+A&"A'+=;Mt]>1 +QEEP?G7եo-zzW-;! N/ٗ¾,~ Ȋ ;O5UDQO!\N{T mq8ɯ- kZsBF^IFд!vh(ݜl8 %#G-'?_z0۔\=K*v FPe.@qf>-uuuCw ʁF2;.tM:Rn,-g, k!V ppy7]IuAG}(nOSƙi :?į\ç#]sM3N4 *Gw[z]nlyex$I )WpH# A:/ ٛ]'MK P,(Xm泏^ۯ.'e5.?55ػUr w'} z{Q@Q@Q@Q@6Ef[c@`3N<Y׏\0oxkC}TGmQ%^:XG>>;\׸׆('q@EPEPEPEP^y=.{xzH[X|]9hvJUyC/߁5mDkM&(( \d`VUbS,g7^r;'4 &ƺWTu|D'q ^8I )  cW$m?~M`o;;lyy۳+_~Ǻ8_ݎ}:ՄLo8h2! KdKy!5 /ė7oo-u67B28?NKnvק,Wc.XΣ%߄/j__.'"yvF=0q\t_ KɦUyLn#}am#)K, [xVYcWTeiUN$ppsp1F4xᶟ1C=At0+AUc&{kK> ּ|ׅlc Y$Ρ I}C冏;#q^ sx7x~ uD{tS`  w~ELj~2b[ ETxvVxfE$t%Gt6Ι(,o_|Yic-užʹv50ʳ40j?~wvwKKMW/o\|ǿ ך+O xeb0i~Pஹ_N|^xo7w>jc"fE+ſ`r$fڕuixc8?¤_pEWڜx/rܺ.k~n ߈Z 7yg;w_-\vyh~ο}jv_| sW"_N(?hmیdyRw1V{?´!žRma4%f'vͷLk6cZi%{+`((( ~.ػuWWh??](ܨ((((((((+~+EW=E Tu/xvW3Z\Z?${Y#nd{bxz'O #C 2`2A +,~ly|+C&Vi_&xCyf`iY)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}K)A_'Կ*=ʊ߰ҟU}KQ5'Ǿ>3|dKg[@p ׺ژ-A}Ujw|9<+Zmz1%L|V~7 Lſش#zlskf#;p +((((¿kYxXx[]L6F)'c[#c1^[qo*O$X2#%PGsk:4R2IYOH x63l~xM@OƶjiQ̓GOi+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʊ߲ҟUK_S_ _*+~JKW /?GiO |*~ܨ /)A/_/Կe?%r7쿴R}¯_~=ʼM~6:izm3VWsԚz\,o|g%Dt|E|9>[?i^'xB zײxG:71xwt:N֖v j0=9$I$@QEQEQEQEQE߳]~wşځ6cC ΟL1\G<th8ȯy6?MwkvmxGp6Z)IbZS0yI1|?Hix|kV8[ _C4;KJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʊSO_O_Go)C'¯'/Nr77W'QJ Kܨ ?dUfҟ2|*R*+iO>?uٿ Կ:=ʼҼT'}+S*%iycEVn'GS_Pޣ#'S|?Zk<+bz|? ;b5mo叹5@1ǻ `0?EPⒼV{Kmkl\\xfX4S!Qprs0r\ƺ`fv'7:4vE.!ʆ 4JW>,٤SA^"Ѿo-_KykSR&;Cil$6b< K+mzƋϋ6 Хli=ƞҕy hʳ0%VlJ+WVՕϾ|eKZ}Ǻ-׆Ѵzwv%y3my=L!PgK[ʿOڷ,mG̕eWfdL*ݝP+[>|RO 4(#ZIE<~T1;{L2 |3R b7xOEs 9(!q=;E:z.>?P\yu鉒Y7.!Sd -z <*7b3@HvgHtˆմ-%Uc8]c(@sߊZUosm%? /w5Y> ((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+$|+o-nV-m$N|+zp~}嗯@AƐ|BN {FIqi7䋈$)DC]r4wx+.-`ԥ 4ڥڭ]OUY&o.Peů 6P6Zڌ͢m5vky@leXxuRs֒9o~Ğ7`Ռ-r .BkJ*6xv?ų.^kEӦk H"c$T+ CcQ+km&eG vO3[s$}E ],yj?,}Z}3Eh0-01RIw|,pv[vK@\_ ;-;!UulodHfbe,~o+$@ϮJ>蚺c$b9$\MQxaJ5mswq&W,zܞp?/~!Hw1Gijϐׄ(f@C q&_7-{Cu(QEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEG"jʣUoKտԿ"zeO տԿ"eO տԿ"zeO տԿ"eO տԿ"z{5tkMZTtf r;zv^'LՉ,IQy.81Vj7 z@E zz$Q zz$WQ@C zz$R+|8S YsR&+'f_UWK]&?oxgg*}zK"4O-PcMEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEU s^Ӽ3jz:~n"OrHu$957 0):l2FChT=e-J !Q ox {WCx[^9^*?᡼-@+᡼-@W׿ g?atzeW׿ ho +?@kG=2 h taT-qGf.|]-3n-n4VIbR|U?xA ]4*_{%櫈'WntWCx[^9^*?᡼-@s{9+:nL !Q ox {sVGQ^g ox {sT{9+ (3sTCx[^9^*=WCx[^9^*z~Оd\W:*Nq&E8{9atzMgݤ]Z\F<.9UAuOY((((((((((((f W(JլbKXSN-)bZcQ^M2s9/ gſ]<}>Uz?|[EUQ gſ]<}>Uz?|[EUQ gſ]<}>Uz?|[EUQ gſ]<}>UzoţkT<[E|=ʪ+׋OOUK gſ]<}>Uzoţk_^-?]<>Uz?|[EUS[ߋGOUJzU0=foS,*9.?=foS,*9.?Ey7)t{TœgOU@|N߀_xSMujy yd)ȦD߲ώ|m7U0=fu9.?9.?=boS,*9.?`z kŧk3.>Ey;|h*_-*((9.?򪑾xX/o]*|[㧏JWϋx|?ʪ?LVϋx|?ʪ?LVϋx|?ʪ?LV⼯ϋx|?ʪxI^^?#n+zҐzי~ΚƱ|5^^ԬAM]E Rwk :q|QEQEQEQE5}w-u G<4Kgخ *|WGp:ZWT7[7oTPʿ_n_o/o_UQ@*}չ~C?᾿ܿh!}}UE| V@ r?U7[7oTPʿ_n_o/o_UU[ NUY,༎)^ Ep#t$XGPA>_r?}չ~CW V@ (_o/oG7[7o᾿ܿh!}_n_>Ur?}չ~CW V@ (_o/oG7[7o᾿ܿh!}_n_wXՙ*f8z-_\\Gi <8G$)ui>k5;=5 [˄0%aq܅bd(᾿ܿh!}_n_>Ur?}չ~CWתοVJxW\M֧oZ%7PȭHq!`}VUxm( ( ( ( ( ( ($|'dh [{HXH x4Fv1\o>:_#ⶋRKM6~Hl[Dly69k,ɧ躷k/xz>t[_;ےw e?v+W~3u| '^/ NSK[tu<Fӆ'q]q7s9|9#x,5[Z+[8.J*#ikG_>u'۫jpZ7ra)P}+j|H<%F*@iHn%M͕ƙ=ŸI[fb-E#Gx6BVg1-gmgQle7mcRg;G9Yn|ajhw1R͵,QpEŬrgB[)GMFI Ym&] Ǭ GMJKe=y3yR"(+'?uxC1ykVе+t3EQa)Iy5=PxzMKhqNs na6(Cmȱ]I#O|(=jz_Zމx MbbmUc¹y3 i-4ڟ ZK&=io+>o ]S}ڸ1 u!n8u;M]?I𽖌׶V1xHY'ްI[DBr#͝i|1OxBϖ=>OW(yde8jHp3y<]w\x?z,Vo Gw@bAШXqRQ'[f)0usstzcqmbƻbyrUlpiڝv7p]CPH9bpHlqx^)'R?E/)o<1evƚ$ģLVYe;DʬLPn@o/KZ I5ln춓c'2y HLj'ῌ/kMs3kQHܫMgl`i և%-'<-gFhwVz {q9XB.d8߂>4Q IS#mKЮn(Q7mRVhdE?|7ڶ[NK 4_.#ށ3U2K0_۟D?\Z/n5B+4LGSKe d@/jMwChz k3)$DLѲ=,{Uco\ׄ[h_UJ~zhibH. Q8p/W}AմWIԵMNgG-M)5I%F0*+֑ 5K='~/4ZV[8&YɔLr^o_ 5x]JM/NT D3ܛ2"eIJ s9|9|i8;w:mBu[b[nyq+U>]3,7.6.\բ.XUb'Pa?DӼEG_ZW!avݼ"OWO07n,U j:^u}{MbHtb &Gy{#E<>rIGu'~#fy7MŸ .tO n?b}F{%ZHHkb ˢk⇉xA%tFn#I,#C3YE`"9"Տ,.ޙnSTQ;s֓ {.<$M1HX_DQY<2$7$nYO {c>z|=ߌ_ ?xv>m"ׂkdJ 31cXs"E[dfPXn&`ndRrBbyGcBE(( : yg$Wş5_O_?CתEPEPEPEPʿT1? ӥ}U_*Q&3NUQEV&KgW~U@oX~$x4tE 5ӨsoKoG2|㿌߱<)g'GJRHԖ bM",ޭ\%vZ7"|w^Vk>1Yx|-?khoh{leuFe~nGQxr{KgOhIs}x{V)FkRCyU2VvfUܠoGh3cxVU1E%,B+x ]̌ Qݹq1-k3_J#/:TJ/N+)umFV݂BN*׷wweZ+Brۙ+M6}mJ蚿i+o)I&kb,2b=XbB˿uXbo\M{ƺ~71٫_jzV5f̲0< *>62k^,s+3GrU}BJe_?g/˫h^n|;ksy)nXz8"j4YEl[i^jԮO{YmukȱEs ȄdA<Fk/V> ymcrx:G+d2-rbY-,Q,w.Tnhz~;?}/* ˑ2vgGSѫx2c⅗-no%&My%O6F%Uu8Z/ۿk+]_}>߈~k_?"iv"Xk:lIU\XA.1Lc=W,؂ocGH;Nqxz:Fԓ]寇mtXLl76H mmV-G×/%%Gk%Ū5V0grDOjN6N[z_}M~#޻!{i+S"Ad_sO# ^F0j퍧 x'M焾Z6>;;Yb& (} ]ݿŸڪ/dGzTJb$7> g ~"KW+$Z-$ޡݝYUL(1a*n^ӷEoe =\}۳O/W nT 7DeCRFto2de w$_?G>!FЛ~<𭇈$0%7I֤ Km9bI0&];1 |?x q &þ04~YJftH)C na8' o|{:.~l[MJ j.m<ӵ[ bKdHBhiY_ B1X6~?e%šƚjl]9*@昺 $(,9هگqS/4OMue5ݳ.%#,\+_|-ٳt:Ωί}`kKY/Ir/2"H"m˂6Fy߲SmKh\IǣKy߾Kx'<'x4Wfo/G'Ӥ{v4nRW- m/ |P{xwH%nj]1mR1yƝo_ᇎ| uj6(c.9I3ò1g7<7y\DWƧ Z5YZb+@R5KKO<ձn:o뽖w;>1|`u/BӠ5}.#yw(kufjfXF'yys|Ok^Po#w4bYyTE$~{= طj2 f' i竾we뮧tV4 (((?5ǎ;VY"7RRCW캶ȋ^> ˡ|@~xKWa]Zj,vLl4$Gxɒ~ 3 ouԡ."O2JOFw6T9͇hAj)LLY<;fqB w1儣5sO߯W66^*U R~16)eᾣh{c<2Pʫ"o7 Źw{KxGŚt}[懦]jx'QYgh {dѨ 8v\5}⯀ɨE DvZo5K 8")(Prh|1i׫]i$sk:i#7SI!Y ;2mf~?rygo_'<ן7kl'vyݻq Ys~uXޭMz5"P[gt+6Wݭ=DqԗWnvqzoA[hWOV;|yZTF-!HѤ[*7xy^iɿ;oU~lu;3]BVƟg}i0wNRim僶XH,_0|?=KQ4 Ϩ 7k%S,FC1nW$(aVڲK7\(Q@A|ep\l"wU|L⧢vwWVu'(ʮ⽺r<GÿYWWaTW9?|p u@EyG#?;]Q(>TW9?|p u@yg#?;]Q(>T=URzo i<\-?64z6#J#[g s<9?|p uE S˿EAG>8Awf2s{˪?EAZ#UY Q8AwώP|.,Zrx,rٯ/s{˪?EAꎭ 7dҏ(*ߚ߆ TLu*ZʿEAG>8AwR=W8漳ώP|.s{˪},OTώP|.s{˪VW9?|p uLT^[(>T9?MOւ(k?EAG>8Awm9 8RqG>8AwώP|.e(s{˪?EA=^G>8AwώP|.(s{˪?EA=^ו9?C/~6o{Q:~?$Wş5|Ïdjz:Σ6j7miy}=ㄅ\vk((((bKJUɉMg-+(((F(_g~syq}u,ٽ۝(㾖kaE ]* .xRMINј]m5B ( D\ߢWj{}Rΰ5sam 6_,љ4oV`m$um/:>1MҬtM6m&0DCT6軎1_+i?h |5π->,Ƴɮ8Ф) ӂ۞F1*,VC-/?ōwsx-<+-֚- #Xo$\xZ|Y5.Wv&OxoZg9m&;Ir2IEேk˦G:?tf7YYHAbFq(|3%HncJdR9 9sƚ~x+O|[h7n{Ȣ8m6a'Wr|S_IOSh`>5#x Q[ Z'v{ ~+|)o|5ڑ<74K!6#wIHW_i'ǛAUώ'5GG÷5o ܚEmVV Ŕ*r+BԵُ4g/_{eYPA6]_hzlzV4ZH.DJ :’צQES᧾Y7viV| %LóO|o),? K?;^Eg O2o sEh}daV:,>(t-;RI^6p9'*]C|ۃFFȪ+ PEyqWX5ƥus]Iu; 6^{Xd(PNNpŀK mmkvײ/~ =sρO|o),zexvS᧾Y7viV| %LóO|o),? K?;^Eg O39EgRY?᧾Y7v2/g =sρ>\)6_3I J!G 3pz4Qxv隆/xR[[Nh,H:d EL3Š(((((((((((LIK@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@SW|߳?1Rz8կ*s G/[ִf?-֝R[t0EE*q |ی]>7i?7;/:^֡)XaDr:,AUߊu|nZBR9!H*ot/>;ybqJc#Ițla`TyyBT~Z|}DG./gϧ>Kvo|~;.8>#j6zwLvΰ@EŴ1vEe7_źOzWlb~_Pa J$1*N7 s߇mO %u&ϻQ^O܈m\(t\V0W[ys+}ZTjsѹ;{¿~6|L?t<볩/mu%<Ģ \gzU5Ipq1Q⯤UJ-//DiJOwy~ |}i(|m|5[ >MB|A)FbGB ;gC}6/>PPX_,Rkw4x[VkVxdR\FT;W?kOhvm-`,.x8>씶i|zdki˾)~n ;f5xM֣gJX5$b2&IuCyW ?cwIxwßfY-pXKe Y,r(1hb6ꊘ-ti?MϚ,7-Sw Y4oxVt Ci]>kq(Ie7m?6p՛j_S#Et @ڎyFUP[$9#覮}Ǘ_sϜ~ I~M> km'xmʉ7 D}._QE(((+_n77޾`_,#z((((((((((((r<GÿYציxN[ǟ(wK5( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( >)xYejI-oʶngc7,x|ԓ9$'$x SW ZI>7<wOO<)(ww}j?$D]^I>7<wOO<)(ww}j?$D]^I>7<wOO<)(>#DGǗ4?::j 6vo4vG 2=c(VYFo+xc^jɩM:Hyt㹐 AtȎde!0 OI>7<OO<) r#÷j{2I>7>~><5|n{) z_sz\ZZU֧s.oqrJw(g۴oPNJ_h<'xúzGn #)YXVRL+OO<)>#D20\ܜ:^e_[/@+i׺kt)ho^^<-R]7>:NJukgk6$`H݈PYFq g/kk9SHT<{X%EWcд%SF7Wo|om7xu`J!5x&h>|>4Ha)hORy=jS|+]h8|P0m?j7[Hę9 5T|8֦ZIcTDSG gW"$lAp5?$|?E//K]5/KᛯmmXWZ v$*xKnA>*xs]tK;]Q֡OkgHXhĊ3,ǺDʩ5@bIǗ?O|?|5lxDŽ| ?^SG[tFY6UN#ޅŷFM%k,8vacWdsmJv9H7=I5ljQEzeαpPt:D$*1hFʒ9PpqqSm7¿7=xO|$ѼB4=G?t@[F/H I-^7gCSn ki<$pA Q@] ''I?5z|ny.MG$x SW@Q ''I?5z|ny.MG$x SW@Q ''I?5z|ny.MQ@O>'W8#^Mn8:_SL(MOK7Q$wٹILq Q~Jʿf'Z~,ԇQU((((S |i&iԼ}X<.T>x\aC Ex:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq134î?f/&_?&>U\~_LMًWOI?G:bgSk(_uD+$TWq135_~v׆_WMmwN>vX/L@ ( ( ( (K'D#uC6MM,{&5Q$M&ݐmSD[?Ώ[^)- gWߗޅs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/La׊xtYec>k犤>mLNR4aOY0r?/+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7ASD[?Ώf=2?[^)- gG-?ٿ/L׊xtY k?Es,oAs(3"9tOon:=\+mx7Au|qmVXUݔ:ς{kmt%dU擋Js*QEQEQEQEQEQEQE??| QfZ?/9\; ZJZ(((((:xOŚf(v3ne0ە]]͹@9׋o.|3|s`/<'ۨŞxZ6GQW̐6 >Wo|iZ;o Յvnw4N2X 0_Exீ]:^OkZ^N4iMJB (xVPa)7[[~h$F*cDe\MYn|8㉵=GPi6kSI^ /.eɑif#l%^X*5|( NjK*FgF\kj1OxYKKoE: h-ZKeBI!AȥP V6<}mݍI=I-ЖoO[RtYu-GV]NI˴{}~'x_uدM_FmW+}amihv O[k2 鼹Z[[kRNӍ^ [{ кՇ۞v5߇oc-;ܕ6`/!UH8;3uv_oR}ݭlWX8$e&۩Oh1 q 'F^}WO bL\6v2ZmIqٴ~Z$eaxZNJ|3k{CqoԩW<3E1yJt_N  u+ s͍R6GڀP%=]Ϝo5>>]k^+W52I. ItKEaK{wy)o ̮ѯ7,Mφ|?;/eu +ᖡbFRtîC)!'np|'𞽭iZi6X.|LT2P? xW^uH{oPk G^8 ,bsr@T_Iamekׇ. N[d/gyӴQ3 b5X㌢G.I;\SG]Y<#Պ"Onx#5 O,O[XNfih((((()ө(f'Z~,ԇQU*|Hu Z((((((((((((((((((((((((((+-[0~-cuuj&zeQY((((."yH`K$Q@$Zk}77Q\$XH 8 ({|7!mt/aӬ"yuq+$q8hH#uH\%3D+i)Vqj:k$ӾȤ]Er 1d9]KQ^_,ixQUml6L.݊3pTX|8=3nwW d5m'DԬgQv{\dxE."swUVzi'4+QEQEQE:(zey"cw?Z/]OL+1Q@Q@Q@Q@Q@Q@Q@y'ǝ?_:-wA֡GI%写=7 4FrstPGx8PSqx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=Z?qx_;V<=ZF!^?ֽ^!^?֗=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(ücCk^ ֭=k௏t"{ۦdAm$U5B͵Usַgܓn>;tS~k_Mj8=qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(x=Zx=Zz(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;=Zj(qx_;~xGO9tP~Κ.5u"AԯuwU:u4]w02Io*h+h ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ?_?|L:zeyů qZ' EVc ( ( (>FҵCoEe׈cjZqIa:.Ti4#/,ӊx]K0]ڬwYխdhe' ̘>xȯyoƫ5n4U`6ZfrOKZFVU.orvm-k[%Eu>=mR\B0(bz)SMo_o"JXHs9l.!#s=}S_ -~ Ҵm@ȠRsԨMM+FO5/p©-ڞc8'M뿗JmmvS_OϷ ׭x.<=a-χ@[fM]^FK^+h^|H;CϏugc'Y/aT5MH'fYs)? 'MեP(I$yT IQh<%nbҼ/]M IJSFB+cݛ/K| 񿅬Uf $&iV%?#ovW 0|Yf>x'Q'u|9IoQ$#@_RR$F68/,o%q(ĺx[_eUY/Mw 0bxJ+kNeݷ;%/]]jՎ)ZĶlEݶrn1JF,o?3 oh&jsHiFx֣U?2,YܣhK{IҼ3KIll"ex'"P2JSs̿IbQ#,) #__/D)7$s~?so|-F:Qi9^m3iVsZE%$^Εote?btiP5Kq4H_$;6Ox^_h/!+$Lki"۵7IIsxB/ IΏCm&dd́I3k(+y/Շ-]{h|_ڒÞ3mbMKŚ5݀ko&/fDtXfc ~{0ջ7o⯷ژ|kxPҵ˘ ҴЫZe"FV&KM2+;xf(b F 8~RK:"*+᤼#@n|ETIxG_?֨% !Q %|@s/BZK? _G4 ψ =j_i/+^W7>"*+᤼#@n|ETIxG_?֨% !Q %|@s/BZK? _G4 ψ =j_i/+^W7>"*+᤼#@n|ETIxG_?׆#8J"*?᤼#@n|ETTWIxG_? !PQ^K %|@s/BK? _@Ey/4 ψ i/+z^W7>"*?᤼#@n|ETTWIxG_? !PQ^K %|@s/BK? _@Ey/4 ψ i/+z!`:oi/+EqHxL++;  #hׁ-ss|Gz-ĺt[wg=,<#2$$N2I@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@yg iKFn!_6x!N7p2k誋w< !Q ox {(;?jy47 5B !WQES᡼-@W׿ 2/g ox {sUQxv?ho +?@kG47 5BLóCx[^9^*?᡼-@ze^5< !Q ox {(W׿ ho +?@k^Eg O3sUV?k3jwrZ xc| cFu>ǐb)#}ӏV2?Qkt^5 !Q ox {(W׿ ho +?@k^Eg O3sTCx[^9^*(;?jy47 5B !WQES᡼-@W׿ 2/g ox {sUQxv?ho +?@kG47 5BLóCx[^9^*?᡼-@ze^5< !Sg.bs<6WyCk%EƱy<,JZ9 ^EJH(QEQEQEQEQEQEQE??| QfZ'o_9\=ZAK@dc'##\|R]c.dyz +~RNvTT1}9~ҷJUuQk?<#KtO PI$l$0:*8NTr \k^24iuootʤv)c BhzSW|l\>h=rzr%I?xF_]KV4Ş H2@,IhK1v%%~BbÂ=1T4i :<iTI-,,9ϳRGFJլ0]Iq.)CT^:dC+,s^F"wVRABDO!ޖQX4'll\ot!bP۬$eJFu;l>,vXVn1<RI9鎴XikWJ_>m sOb+Mr3,n>`U= CiV]a4{dib ޸Tgw5ѳiO"Z+gOֆ_x_Ack6$"7p8l:ִ.m:,侘Z,v3쌜o}`sXEcgOmaĽ:p,|&fUUK K~-?Pk5![KmgB4-JzWY~5mFO{FU+;o,ڨ$bH{vZt_mY7[u9C&&Q 'X\"kkubymj#V!˝Ryvklh@^Ip9bĜ׷&EyH'~(x/K` Nc uXBWDZ!+_S $Ry+؊LGKEd}wc ׶{>oAaƛoۭSUtD2Oyy2 +vb@=3@VEjmw\SȆu"@7k)i _}Jem-f Ϙ#dQY:~<o}\B_[̯0 !r8e@֛{MFxht2,)`ǂ=O[UadhՖ@ 0GjͷFy^3kqG=ΝYP#WsEyq(+PxƧqݶow aa07&E;%%gp>dľ+-oz7 J?uݾ'ɃDb+K$${{Ɵ5;OT]K-2m~6D{$H!=6GYC$T#u_/i&떲^;ݤfG< X BW!Sg~&wbg5xT̑CqZkO-PL |ұX& 7|qx÷EYZG%"0ao1o5E;OJ4,F2,,m)fY$.xg?:Zi7k'L֯9Y AsH>;χ汦Z7 l^K )&GHڻA*|Ӿ,o?~ ZY(l$mݡg*āoG~(Fh˷©nz܋?+̼K?KԾk~-tYL3ΤTK5̲2Z5PJl<ۤ0p*4\.р?N1L#V΅#/-+X]]c쐙|/V )r:݋k k,n?bn1Zi-$RDM!RO,!|VShvAe 71[S7^W/4o4֦Ka!@.dm͙l``ſ>?|XCeZn H..;i.+[pd~ jg7$u'Lm&Z{ ,w)?sڬOB__Z2=> 32I*+\&1𞱣ko j-qmiRO2({9%{ q\ B6zxU/m}RY:ׇ,@ƓvXFvyh!~X^ aEiqivRZZ#o:"Wm?1ug}si_+X@|ȴY`]]'FeWBV)@> #'_Y

=/丆9Ԡq euE=es>!j~^ƾ`w S3:!eT*6O2ċ/7կW/ _Jzeui{&bb{kvi2DsF>A'<ٓ={Qu\k:h k6P$H' aq ģ $}X:x<x.U5khwrx]ԭ4mZ;Fճg<ۘeBLK+ݟ+4m6tӵ6=2P?=䚵'R1;Ɠ,f:?_^"?xM? ,Ml_%Q!p7 _i_E@ sޥ~w'> O\i_ZO89y{5B'V+譞d;œ_k~::ּOm=5?X[[)Ng;TLOu)y6wZotioQyח \OncQ)xO:$^%s.$pJ%FѹC۱[ᔹ o>/ӴV)-nug{kK,z ^z5e\e/3A9<-iͧhZL- 2Th9θeYc| %[nD76@jdEV{/;Eχ?úWVu>}kgs{fRf0l4Lga]Z½ |au˯ \hm& _ݣlK]cnDjQʅO'cGϬ|EE/_xkPK=7m"ǘ`2J!(\/~ɞoW@Jbpۇ#ʞXuȑzRvɔÝWj 4wƚ?[Ax;Mw gm;eQsO7,{ܶlGk:O&ׅ|U8+Wmk(X,VJЉ^䤞`PekZt.xN7he>p#-Ŗ'%VT*q7_G5hڦꏨ 2r-/.P*Un'V7/Z_X-jVB[iLI#kiz]#o-lc▊i>%\h3;\*,f(,*nWzƬk76-ݮ+x3öNwn)-/x?89jzJw|ǓjZ : 0׿6g W?_ECS(3y|sɴ<[9^-Ey3υ+/M&hww+|-A_m>׿6CW?㏈?oU4r +X).f,0Y͇_?^<[9^˷~Zx^|Uw jq[[_(;p(;j{>׿6g W?_ECS(3y|sɴ<[9^-Ey3υ+/M&hww+|-A_m>׿6CQ^g o {hy|sɴZ5=2?g W?_G3υ+/M!W<[9^?|-A_mw OL& o {h;jze>׿6=\\xnoo5e{O}BK}bU啙ܬqf$&kQEŠ((((((('o_9\=׫Wd++GQKH)h(IOE>E@SIORB)QEQEQEQER-֒E;WKE%Š(EPEPrg5mvN((()ө~?NY^^U4:sg:zQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQE~-cuuj?_?|L:i O=2(QEQEQEQ\oo w,6ڪgY(] vU2M:j+|iwm2ָ ;V($pl S!X[_&}4uIK?pf_̷B%3d4rF^6Pɢ~Exi#NjK)BKy:N[[wJ2#g PZO~9Wk㫯 λci:E71"YaS0"$6e޿ vo~]7=ʊߊ#m5]/*η9>dGۿc~1ŸeMb .u[t4D8bIU)|¿W(pnz5𽞻YxQoyoego;9d/ 5ȻXyHV1䫳}4ƺjZ':K737IEջ~ “}rW]Gx5Z׊$O ]^kv|Yak/ fq+ uDmK?% rBK.A%v2֭W_ފ~~_#x7n_>9kZxR]^[w qڬxW]uMLg:MAaTDG!W W/|ݿ̉KJO >_ŌeH˰%Ξ#15Oiۥkr;/=yJFGaN?s~>ExoG~ K nNc0kH徳IV3bgf7U&B>ʆg'6Bgߗiߟ@7WxjP^t :Ouw/6ok8,X@6w ~Ծ#\֛2N,˱I}># 3( Rg|C^*IBiK)r\x.Jd7|Vjm͟7q&en€Q\uOZf. 麔PL|M=7RoyOCz o:VRJ$//%ɱW䨆Ow͕郞Tl.l/ỏ;-ޛqK.CGt,GbGa1,$~!Z73CUŬ6d%u8h\Vxiho8#}_Z×"_.lI6 7L;~"xo`h?Vޛ@/< =~\}~(F=Cw&ԡt Oa\]]͆Sڗ[/Z_tN87iMO#"1RSo;Z=kx:{ fd>#hKWUaw͕}ޫ |csg xInۏ\e ;t$`LnawxWgcK_Z<$#R,%3` >|-~6Fo><9!^ַQ|Tjas|oޠ̲o_W7o}~B?DyY[8NoiqMNg@l<8gص r`-$;&|͝_[نikK_^𿆴J8wiwg]O#poTpϳ!_\^gV)IYK)|\xۗV9 7|ī [!7V>ϊduӦ%VPźC;̒?g̲Soh֣u[mCWU H$K LO7:h\gu/A2 \B\yVg*Iv2oԋ{O/Ch!goˏoug?(uh:ICO'75KS,vL ;?_te kMvox{n$>wIߔpVZ9<{qyo ⤔-$//%ɱWQ uF+u~$"ӼQiz֚]پH-#ϖ($.P?x4f4`#LQEQEQEQEQES[S[,j9RFViuHuZ((((((((((((((((((((((((((+-[0~-cuuj&zeQY(((+ᛟxGPgӡ{j6ROF\cI՘kTxV7j^|Qcex_Yaw}4Gr$ D0lACGE/~w6?7:(?bc:Z5wrHESKo6VOG?|/R<_M7P&gZ *H]J!X1De߃7S6sxŌ u]/!IMtuS!8dPU{K^]ooofkLdյ//Ɵs(ܧ2{stQ\iڽ}VicVBt ?}zנPo}G}|crj Ia_;?YQ&yuy, J,hUڌ:/_6Ru^3KԼOsccٌanm![,ybZo&m}E}nx3~Tzfjw0^h^s3ݙ I׼ }><;ojD T-=zt iokG7M=#XoE.z}Ƴ-:y' }y,l{(etUKP}nM#/m HIYD`+Aȫ4QtV]{ɟc|H[wSޭ#W0,B5%ۋ-AY%B%6(@q 8Uς< xB@lofh4&74.*b[kr|'}u}^?zg4K'Q&~:#˳\8L$j<ŚAݑxui4;#ett%[i23dU}__p) (((+gu_Q?DW__EVc ( ( ( ( ( ( ( 򟌟Qs^^SJ7hEA0b8Qҧg@w_F<5675%ajMwWK2/uWp4  8x\Xns5K{ ;{ן^ (+d+?vMouF jgүcqhƻmI2+;kn\D2$< =/K-Rƞ&}.L͆vͩ}O ^-2 gΐҹ^|S^Լ%ƫov7k6 +lrP/]~ޗ]+\G1^^j_giL[[u\2D# OžKx=MOLJt/_D-bsk4Xf$а76B |-)i h^#R~ }sGk>׵]:Vr[!7AdL;LAaHJHCHŸu]^A?/"oɩY[Si͕A;)"o/>#twĿSVHJ,rEkq+1k+mzj(k m2-7L]e4vtMdVMEZeVVB-^+⧃|A_g6K,"!OW}6V{gEFyOPgMt5Im%i' &In$T*E4D$O+?k2&#I!fxc Ѱu φj~}o0&=As>*+H3w߇> ׏m'5r+Vyg;K9!flsCvW7.5"HwO@%b+W'yQ(n_ѡ mMONl 5V4WGԦHQuA~T7V;̪2ivy3UEM״,;KD@jQܒG|1U֓7I]sPM>Y%>tDIgz{6ƕ`/%2pR6jtgyȬπp)@)=ºm%I3p J [o >)kx>KKx^R,}QuӞdQX:ĭnrӀ؄o1t>3J٢^k5|?e[S/}pGR ILgxgcwc]uT%L׶Zk2dF-;Zr PDg=sj(0\o RhM:maNI`\;%k]ʹ|7Tֵjↁu+Cɐ7Pby򯘓77@e/TB:튖 ( ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@r_>'6]sT棢V:-h6۷q e\=A hN3K->9-¥O*y*_ih?/VWQG~_r g ?Z|sZ+(K->9-Q܂Ǚ¥O?RO Aeze{G |SE?T)Y^E},y*_ih?/VWQG~_r g ?Z|sZ+(K->9-Q܂Ǚ¥O?RO Aeze{G |SE?/¾6o _ ?LдNCZ[F9-Ag9/xfG;X_T)YG*_ih?Lh</VQ ?Z|sZ++(?/3K->9-¥O(AcRO Ae|SE?2=X?T)YG*_ih?Lh</VQ ?Z|sZ++(?/3K->9-¥O(AcRO Ae|SE?2=X?T)Y]G|;z->w{5c[)2L 1]%V QP0((((((()%WW?k5??| Qf=TT AxT<g 6^׭-=Nn. 74i0 ?,{{ς<)m/όtĚYi [ܤf8VQ0z7ƞr< W&~r|?^w{nߟ+th?<>|c#Ӽ{Xjpymo®]6 w@]K (sC}ÿKu=Ǟ4wzvo<vj}3um>QmuG[ yN2lC(zVesz] y]x6Y6cs(. G1q0ʩix.~;߅,&z:ͻX!1ѳ$Ci=dQT_,߅<-};Zcq*]9q?Tq&Л<#χt]O^t-/Yze棬[[Tm3%]!ܙt#/+x'¾&3xA|GG5Ha&10e]`@Y(d|q֘''C:姊L _I``{կ|RgEkEvxMI8g(\d*ԫpwso&ԃOQנo_izƧc]X~өռǵ|#b)s`b6?j;_u/^x_Xj6cDž]+2Ͻ./5|?#axf·q[%Ġ)³*@TG4|>~jz<1?jf dvȇrá]nP~\H1J t 04j>_شKT ˭c8wsp`<5|>vjZ<3z&Kux"*6 lw!<:sU|XOQxc[񎁣xR1 -#P᷻#1ͽ2,6bGhN3&C8p"v_?ϛ;AQ֋U *S{m#6rS zXaFsj08my-'Z? z~u;7Zxlc*<#b9X B_4MS%S=7Ro4[kSR5e%È^_1פ2/˜Ա/=h;y{EQEQEQEQEQES[S[,j9RFViuHuZ((((((((((((((((((((((((((GmoBӵz-\xdG"]4F@$g]T0W1*קE'QW~;i,'m~;i,'mvecϪhwDÿXO?wDÿXO?__9M5ߎxw [G5ߎxw [GV7}+o>VOa?+hOa?+hϿ} 4W~;i,'m~;i,'mX῜[M<;M<;+>a7T_+]]ec>s+ok4k4oW}SE| w&V w&V0Ϫk̼?'-|;y5ߎxw [\ƟC^@<:ښF}ǕYe7gsہ/9;oW}E| w&V w&V0ϪhwDÿXO?wDÿXO?__9M5ߎxw [G5ߎxw [GV7}+o>VOa?+hOa?+hϿ} 4W~;i,'m~;i,'mX῜[M<;M<;+>a7T_+]]ec>s+ok4k4oW}SE| w&V w&V0ϪhwDÿXOڡ~1΅ 3eIw8l!5.T `@q֗V3}+Ʊf}qEWwQ@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢖREPEPEPEPEPEPEPEPMt|/ U^SM0 t?XWK.qn)e毞ys-բ8uQEQEQEQEQEQEQEQERfZ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCu/SOOn [+'"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ G&>"o%[E}Ù/g% BtkЃ/]uQg9r_ [^(+k?Mn[ŝϝ0KŮL 烐?A^]:o?&xkKjZ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuxFOn_6[\dm}>jEFaEPEPEPEPU-;d+K/l,Fpq55ͥ]˫4jɦMxn Em=sZďxU~*D쮯-1VqY S Ũ7ӹ ;N_5[Rԭneic,I $$kڤ> ֯A^1#y¥ Fpa>@߈t=cͥxX{"wubK l`4)KDin\뮾{?wŚg:|ҙ-dOʹBCJAA+˾^Aq/MY%! r|ylb T 'SFT{8Z -5e f' J}~0xW<fZZlI6BB0snlܸXӾ/xN[9u?OF~w}nYA@$;W0 3^ST}'Ǻ\ Fn4}f{62*,G, I)Az. \6 5RHre J qSTIBiiy俙SW{3cP<#k=Mdi|J\Zk(3RAGn5h4'}0WV$A(umWK{?J![Kˋ}>r.KA6t|97Y}5ukip֠0`PXCZ.˿ϥ#iaA/GI2o,mW#y 7;@Bۃ\L69d/⻁.YC '9mcz φ;j7~$sdd扦E0!A%Y^գU.i+GgۢTv dSi8,3^vM]ʧF澺n~Wu}f9"VWRFhцTFkv_M֋TKk[q92eh uzX6ǝ;)Gkz_@* ( ( ( ȷya_NW5%E rH|mr_!뭮KD=XSשKEW'EPEPEPEPEPEP^SJ7kի~2FczI}?Jxdc':?O˭rv</ş uߏ<1k[͵6l72s*@TxWz.x:A2Pmgp%M!ܙ$ލ/еr.붶nP7`W!zv|Q=~jWN[eO%YK. {PşxSg5:?5#5R/.Hq3o}̬a__gLs"?4|:sqkexk7ͽ WUw\*0Ѽc#|}ڤv\#MN0Hꗇ~6|=񆓬zŖ^>DžlZXc`{9u|l{6~<ůo6zͲ \F<PsW I#o=Yxe |Em5&PԬnY%d#*Ўؠcğ?Ku={Ǟb闺m WF;*mOXO|MxcX񎃤xQ1 -#P!{ˉ{ee3.Zn6(r8?NM?}keai$EmgwD3:c$+PW&z{R"q~bv|)t W h1=;VA=ݮV.\; wM_^Ӵ B]\ij.'X&vTsҶ):7O0nʹnc@~6|=񖓭j~?F:ޝA<61vŖ~>υ[kz^m$+6?Ⱦ 2by#h翯\gjXxůt]BM-Mg*))SrGFB:v&~P bO_ƅMV-5D!K dH>h&Gc/><_|A5NkWjqؕ\pZ ĚWֵ+-.;H侸XIm 33z֓ާH۸LŏOg57u/(Yi )&m﹕p9a/-ot< 1E8xm|w;>nc=3X]?RK9 a?9 >5,ZU u a"7Wn 1H }⟃𾷧}Sӵ'2w6=Z]l2-OOfdxT FYhZ}Σaai Ouu"Q,* s O1g֫Yw>[]kjco+lUIVck֟y^xYk6cs(. G9q8AQ] [w"NciwU.a,-ơowyqK<̩h %wצ^ʹ1_| D;<:>W>6cMHĶZFCߘ8q3o}F߼oZoq ŋ%R+DqsصcZWd6".dh;ʳhH,nGQc4 G g#Cm-|e{Ny^w6۷S/ W h1u_`ҴR mU&`p$zgiqjWJ[8;2xa=1\^Ꮕb]s7tA{lo- E3F*e ~6<񖓭j~?F:֝A<61ryȉ1s`|Ynu>Ŗ^>DžlZY[c`{9uGč>[FMLz-7ihـl2g8#ֵ`,.K2yu H) XF]T@6v8Hrg7>.-|+}9&mP\r6qH v|j{/ExgC5>צ^ʹ1_| D;<:78'Aa5C_] ]$(,a"c)|Oi|c$ԌBHuH`#1ͽ2;|m{ _^*7zr_[|$KıaV2.Q~*Y[|d&o.yj ε9%r쑮!f/6Q_[i 9>N%0<tLk>bg]VԜw8Qmv1@u4w\P2Z*1׃xiө3?f'Z~,ԇQU*j9RFV ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( C$_ _I? e*/kܿm߈S5myskcbF7D7WKc桿_Mj_G- |E֥sG{9g[Er_4Z# CG?5=}\- |E֥_Mj_G4{fuW% CG?5=4Z#ٝm_Mj_G- |E֥sG{9g[Er_4Z# CG?5=} ZH:uG@@mJ7 3ӭkW% CG?5=4Z#ԧ]GRֳ:+ϟԿZ? K깣^}å\EiwAV{D(fXN3& i æ*ʳ:y)SOAޱ?hh/&/GW伖aך$YdxsQ.ł_# ﵽ C{;S{3{M𮋢 i=BdKmjm7(P0v)o/jWP^i7W0.b{dwI@P$RFC7pұhh/&/G>~"kRz~-M΂EӴYml-mwXaQ$g9veNIޙ4>-6; VkURHe*a_Mj_G- |E֥{i9jve|;|Wt}2AU)#UFž {S6{ /4fRNSϟԿZ? Kyi5eo{{wJ͙=*Թcnc[j3ڴ+ϟԿZ? KԢ#oV\- |E֥_Mj_O=ϳ:+ϟԿZ? Kpsζhh/&/G>~"kRz9=3/Z? KϟԿhg>k_"9\ CG?5=/O [}SԮ;^W;ߩW%{I? {/'/CӗO^-Q_Q@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢠz۟^?/§^N9 q\u^@x—<zdֶvNp$O3qn[7~ߋ5}L⽸[WX:IZ؜RZp7^\ּC˭N˾Ylɒ-ְjˇqIe?>E$i H S$ˆ Z1 Q֦JGscǚ|C/~!x.B;u[Μn#,<֟ϒM6KmQ]>ϩ*~ˤWf6 DF4Ʉ1Mb2*@Tl2*HcOkڱu[z5 Yx(edEXź2N?|J>UT 7OxO7 .dyb=VAXt0\n 5^쏁J$#P?hmki~ROԡ^ -OswH2:r;`+C6.xM񍅯IҼSioo!FZ"[+{~}1W&5[HaèK ";q2f}; r 5jn]gݿ;=2q5YbA,H v 6?^3x_xIxoEav-..EKO5a%a56%^5S |,K_/4Mk:O:Wj:E]yzboqf ZWT)~x/յG^eܗE(bœVx<(ח=ݥl ŮoeڏG|-]x~ǚϥZ9,;{R(eH*ZIfbHi:Gƹk^,CKsi?.E–zv*Z!iw|ne$`*}+# #P{6Kq/X*X/qW| ce=/#yy<R9{[_h%Ӹ[R Go= :C?t ?ᆣ[UH-nms5pM<Gb$'ڭᾛχ -"-esJޡ2-k'2_|+Z}Bީax4qɭIF].U\J U~[o"|-: & ͞g#.2R['xGŞtio-l^k!=ԑrFO[4IU٤ꬅr prs)DSk w !:E-ZD@w] dfx V|ljPIoGo h>h:Ecw$yhm'c{4rduH-|o5ψ]xOXMkM5Zke{۝>r¾s4c};Xڕ׮Cmun.hr\+Aێ;GCZЮMͫL$CiDʸ b\OV_%]^F2Jp·4.+=Y|AxMm;N׊i  tĝ#w/ zå- uN-/aOk(q(P)-kr~%? }:M&;/gXdHzs̘!n lciv{W&Xj)=ΙhY&Y"1I"ч[>]Ƒ|-mg|x.!<|1zt?gth!vCE:(J$}w!<MeP΢v!yvKL;y{@Q:o?&xkK I3~y ؙ.kӮK)T103ϯLi n5[@ь6ͽq]KEoREƱr?mR嶏\;MBIoxɵitWdL7Js]dR?mߴcukV]9u+#,l-ϖ}6"Ƌ>YCz*5$Yuu-Z|w4Hin5;nJE` qb}ARxZ <"vHQ$mҭH*Ɨ)7®w|R'|/Ũ[dHxan(?>&<}nmIl oZXݘ.},b0A+Rd ş@إk ,o5g꺝*۪:K6+&rJ$_Dqίn+/oɟk{mJ7y[0݂2+>ņOOеmD?ƫ;Smv76pyW0 ,eJe(E4n5K餸T|IX((((((((+?x_+GLOi?Pj|P>z~먢((((((((+$ƿ]mr_!eJ?Ih((((((+~2FczyOO(r{YU>Oo"9=y f?5O'>Wx?_n_: kƫys˶}B{X. !hWڿ*uuh*fMnu y30{c,?|mỽ7Of^m{m꺊>dvnAd ʲ$%^&R|L '__EoxrP%đ PA;@ JMoHe|+}OΎO]H}Jyr- `McfY.cH7>&sj)mJ ?vw4mGk3 #?'+_g|>U_ޅ薟<u<|^y,47k9c>}C8+ZRWy5H>/5e-y [PM~vw%KHqwLU/b 5)\J~Q5%k!?i4#KO hv,ѧ67 XTfُ>L^rHh|7 C-{w|Ar2G]H#ݴ:B_Ǻu(◈-|AhR[Aui&K0*&% oY\|.Y_Ĉ6~h4uv ,Hmt"$Q.W-82@~<Լ#gZ5}SĒ_iFx3a`d7v F72{Q,oXS}Fx4{ /2GopB.K81n~:~ - L#O'Vc˞[MGiysq]ks+BMJ8fK?α ¨Ž8 INF@ EӖ(ԀcA֮~(xwYP;gVwκk'好2^뮮]o XRe Qs<\J)?z3m@LQ۶}֩(|-fxFΫIumv)s#aP7 t+O{m#FWTi<'),H ֗s'}u-jwxaĒ¶ڳ5^seHYfO׷X> McÒFm4uV&1]* m RBT5} > x:ZaU(ıPybF9/PִkKYB8ܬpA#?h~w|Kl|E&'Ӵvy/p?n'dUcWu'iĿ ;?.-gBHSRYcn- fa'1AGe|_}ɬ%̺ \c "5Bɴ,l|Ǎ4=/1xQa Neh*#:#XNGwmki]i jEE.<ۓ[iۍ+}'O?4C"|2/cN5 $ydm9$"sg̹vhE[[3 "fGa+Ci1Xd*~L3nvס٠cL_ |#ռ[.|>fg}i^ūf!U=B;XSgR8|e_>JҴyud}rE5Ynϵ +#a kKݹ-c=?Ct4yg$Wş5_O_?CתEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEP__+_!+6'W,BWpS;mQ_d|QEQEQEQEQEQEQEQEW#Z'4W]_0'=O<7yȩ4pLYۼ$N GVROW3'R3EOQErQ@Q@Q@Q@Q@Q@Q@Q@r_!뭮KD=LiO^-Q_Q@Q@Q@Q@Q@Q@yOO(r{YV)%WW?k4ꢢ>` gcRZ? xoƿD ^ֵj׺5mn+U%p2vS;*Wݟ@/< =~\}~~%vPxjP^t :Ouw/6ok8,X@6w ~Ծ#\֛2N,˱I}># 3( @ O^x[qx% iC+,rlUwIߔ[-[RJLWYM˗ثy|Ow͕蘒Tl.l-ỏ3-ޛqK.CGt,GrGa1,xUmh~ޣVwZBHwPj^6l/9څ=9r%Tjt˹SSVB>Wݟ@/< =~\}~~%vR@y>"x߅~!xY߇tMGVm?SKw{FҢI ؤpoD9a$W5_%o zQgLIqwԛ_I4#Jv;~Pga&gHfSژ {=煼7P-4|Eq%|y.Mg%D2}lq|C6~NJ䝖M%VP×! ##qٖZ+m;DCFhwj#RDλ/6ֽxiho6/jpȗ˶[9RM .N߽]-?7vT6?>^y{?N6~Jv}9MCx{A8O~=ܿ8پ٬Xcg|;:(M_^𿆴N8wiwg]O#poT@eվ!ۋ xn/$m4|Eq%|y.Mgʈd7|^nOq6\2$r4wBw Dvn;2Lj_Vv.o :mGqk!a)wy;_q>l/}Eqzh|MAۘݨ_k×"_.lI6 7L;~"xqj|?eCj۳h!goˏo֊4;mJk}PgŔ>Ok7Y}{u8ݦ[%v7<`]!Rcc(վ!ۋ xn/$m4|Eq%|y.Mgʈd7|X _kaK8^K Eu=l;Venk5z995%PxNvQlWNvZ+\n(ӂm?+]"luFmƭ_j#-3!T bzS Ѿ"x^þ"7=x"U<J6,e2{qG=օ=?t?AcH[j կm7rVI$:PI;FI=< xI^ZkH$+p3D%wE&oS]]XoWsgtkNFu6,^j .JU$W[?_t7ÚridR] H."!`[daS,X@ө~?NY^^U4:sg:zQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEI? eIkkD%}wU>_ ~no-owWr<7R"e`7Ƈ쫒ܿmCU?x ]K(_爿Կ"(`K~?"—R?W/)u/H9c=_爿Կ"U?x ]K+Xi>W/)u/H~?"—Rh=O9/U?x ]K(_爿Կ"(`K~?"—R?W/)u/H9c=_爿Կ"U?x ]K+Xi>W/)u/H~?"—Rh=O9/U?x ]K*/6cfAk @K"Mw4RM}r_$Q GKEuSa{Ig% GKE$W[E}¯_R_G*,IZ ,Uq9ZԿgxVIfO7܀P@%5iKw-ܷ2zu"嶞}w4v%^]>7#/V6R@KBHG *8$23#+÷zbV~4}as7/xU2 Cg$񎕭[>I!W.A_򞇃kғ~kz_e [eAX۶1m6 1 D * yr\TVk_ t[6k+\m> M&25! []_|4? =@6 Oq:Ȣ- FyΓbKZ$ mfDetr<_r),q-:~dۻ_zҭIy-R |I+tRHc.<6ri0-粄RaAaYa1Kd_]kؤ8R7;>C@ /;Y|;)$/rd{'ْ̧*#N 7>TJmsynYJu[KGO ٹ#>\ wzG]*I[4@ .2JO|W1/SxVS|;{u}bɨZɧiU#B= ,mãʍS@@_B(I?k .˻9j7z$Ѵ=Ρyb3j,Gb|凔$#pk5% Mn_ xFXGg+IVz4o'tW*Rv # ;JtˍE;L9#vr2IZϖ2~iܮhr[o>[~?_kR4V k2_Vi"张xKyd])6gz$)#P&2U7q^r3x}2ž&{L;-|)!1I@Q^^..ʼn](9((/-oWӕu|/nzj~4_~[\$5%zk$ƿD5~E 8QEQEQEQEQEQEWd++Gj򟌟Qs@)i-QEQEQEQEQEQEQEQE`zw;xwZnYMZ"nR,W*pqrusߥz~8!?#AqCP5G?k '׫@Q  @ÏO??^EyG3_/?>f_}z~8!?#AqCP5G?k '׫@Q  @ÏO??^EyG3_/?>f_}z~8!?#AqCP࿃~wjzu=N-5~Vh5Ti`޵1h?SQ@Q@Q@Q@Q@5u5٧I֯c?!kkʿf'Z~,ԇQU(((((((((((((((((((((((((6'W,BW-~m|!OY脯>w9!:k#'7/okOx]|LyuD]$GQԂ{}>f-'Er_k?xiG!Z"o"Qw|?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4D+=8մf-m4+[i-0'&8xĹ٤PNiGQ\!Z"o"QVCƛȕW}ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:++X_MJ? ?~ʿ#/BE~4Dcߍ7(?:ȷya_NW4/%lxcx_G[n/?}5]l$Y^YUQv8 wݔmsV/Ik`K%{I? {*?Ih((((((+~2FczyOO(r{YU ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( kt4kt4埳O_?Cתו~?NY^@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuQE}EPEPEPEPEPEPEPEP\%kğ~]ur:gKcS⇯aG]EVAEPEPEPEPEPEPEPEP\$5%zk$ƿS/SשKEW'EPEPEPEPEPEP^%GxD?> k>#,4 4tnutR9 gU<=+wj1O~(^?OQckԣ@Gzw@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?X*,zG@W cG'?㴣$>1ckjl"~1ci?~d?v>36S+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETX?~d?vS+#ETXcٰO_,q+jdc ?俲e|&}CMg.mo-$Y"'ׯ$GRCR5+H9((((((((((((((((((((((((((ͯ?I`K/K__+_!+O~|ζ+񮱪ix~HU^ۼd|xcvy+[ s;5f???&;J\?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#/'?|wCh?:+ ~~M71w 6ʿ#'DhzMCIF#HXefi@?|wCkZGĝO52+h3y)†t_pdsLjiJMcT{7=俳|wCh ~~M˾7*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6'.*o俳|wCh ~~MrhK71w 6xZ׼>n5^{yg# `hlΆ/Ik`K%{I? {|,t5~E 8QEQEQEQEQEQEQE5/RրE"n?JZ(((((((((())hssqk嬖5+)Vtrd0|?cMmz~w<g_>KXmP54X>tZ[bN_b9=4%Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@RdPIK@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@Q@~m|!OY脯ZC$_ _]O|sCuxFOn_6[\dm}>jEFaEPE6I(݂"1|`XjOiVeŭ/@b[xS sU!ߙq𫝝/ГRZӖ15͝$2&rbïE+2Gd/#󝔲 $O>xYocj4z5+{]O=YKy.;Efe@WcAMi=qC%庫n$jYs3N3va(2D5{W6гJSj:eF9TM.ћKwp]+2v +!I|7WJ&6R&< Z0.{?&q<zJ(<,-H˗sѨ<ī]J@LTdGN8I'ף~K(u]z7t9,[vZ׼A1obny^ּu:Gh7 ys}ne5cpSN?Yk__W]?+ɬ&jv77ޛq}KMAd=Inwv8k!oz%?hӚ—9]S#*n8}j?ԺK+ͼq/WfD@Kx֚D+#f.F9{ךiߴvu&UR8wLc5Gl?vw*ı>G{_N9լ})EWqQE|/nzkKwsS:/'/C[\$5%z%S(OŠ((((((G||C5׌&\[˷AwW*m{uyGo(r{Y/M;۟ XWn/2w?rmߊqbKE y(mLQ d+bjo3COD]Y^jCgm<3Ƒs %)v#eW+X'p;eG|C/aƫ5i-=}Bקz8- Bg &{cxÕɾ4 X|OuU/+ͣk"YiC\,Ā- rQ aR?dh?T?;%?yQ^{-ii]׈8]:"Uǝq9Px}Kö 7zmˢ˰7?1ROq7ky2#^R?;%?z2VG'3yӧ"]i_%i+]TDhjKs| 𮬉`{+;8SMSR[YU,[xΚ7eg ;#WUZ?x{^R(= {5^)?">k[6i%֋jHo(`d K(C7Ea]x_ 0n7ho⸷@Db%>rfL nu>QDw?? {5_0ꟴ?M tH^^38!902FNp=;BH5;䷊?e 7 A"\FJz eG>T(= {5\~᷈~1j#5MSH.J6F4C"ɾsȬw(lIEơ C׼ u%|Cs!kٓufYU[r6B\b2#d6xZ[O)ZBx`GŒk%_i:GK' .6æO4#"i򶹎²2*i?;%?O ? {5Y:0|9XlƇ-t[TӤF.$Xcs+  x['\i^TԼu%;5;K6㑷o t_:t 5b{h]7bK;GW'ȑ~*+Us?~^Q ?? :=wǞ6ˁv-4Ѿk2E_I4ʲn5pC?'OE.ak [xR[%ź]Osp--+q=H0΁@4C>QD{{5G2#Z0;1h4O h/j7~u401W3 n!!  C?T(=SLW7ĚnK[-mi= \:ohc88#|Hoψ=%֡G{g֩q40acE;Q' cT݀~{5G2#^VU/CzZN?-1$jI+昁_)yTԲ~%潼5?[#@o >애;uH&1,QKP!DwK?>jo|🏾k} \Zk:x&i"iz2rFUix? x୞o6^/mJK@e.U7 \ =C/aƨQD{{5^q5/$>#t]vkxFuo_HK9I$w1Ď$D,+!MV&u?|?^R?w{5\q M_oj [+]y:${C$8 OƏ_o/]>iVV:De $J3!t۶wvK!?'2#}0kبyS=w_ׯsxq\^=5O5ҵBQk>D$ѧV,#rƳ?~"uANjn b_ofeqU]>9dFv%:I$$F3M7k5ﯙO5yxKhе}M.|MυI2.cs4bd@9$o*Vvᴉ.K7@|cq 8[4WE:J~Unq?,dvPf:l/#*1EfT l$<GZv&MVtŶ+0nt7'iS(H?5{L2>vzmiN[_}GwZe_E6eŝZqxWQ-?{JjXY|?öo.bb;{'|}ۭD_Gg)8# +'75_K/.^[y_ZÍktKEu6!s$"8A^8-w GZ7Lu]DĖŬkLI82B^0z`{1 N+<M7-oFך֧[I -+,B{c 'ވ$,JƬ|[ז+y#zEhv\H8 xg(XT*׷^!wComk ^&Œzm8#`ydo43b; #k6; OG6MޞAzEuʲv3(${]M<޶{ºVIX8 qIN>wI#a)šu;XHnau'OF$0 pAf:k5Rqw;S; 'V唓+yY/t*cM@m(ݬ諒Ծk^FZ.i !Ť_G6ČĹl4͓} EJL>$Z/33jCt3EtyW[ѭ0Y'hxʉ xcwoZx5OX7k}>n[VX9p@OInXx#koa}eWϰywl7`gvϗw3ӊEqQ@r_ [ޯ+k_"9\hN/Ik`K%{I? {|,)k%+p(((((((% _W?k5| Qf6~)YxP.xgmݼ֚[l⸍ nhZH UkK|_+ABռat䶺7;[H&YbJZIX' xTG n%<6&t5i6$KygC1=ˀ S%ur|2׵_8=jZxVZ KwVZL̍#!"Fy*+dON Goo!wcc5)aukF#B*#G38pqQJ~4}[']Y5iggm{7qrpV?5?4a*xIxMw}#Mud`sHd%ih}/߳λ-1xW5χhjܵ*d'jJd 14z?Ÿxu]WVִ__C5ٖ4nsE'ϙ#~~$7zw\"٬"Q8T#';v=On/hM]ß0>. izZ`ԦH<1VpD9bebƞ2xHww7˙ ^ VO@1ak* o1,;-O<_|%zkz?ƺՕu<$.R;urZ2v1 8[o*? :/ϯ_\s-Z<Uˆ'Tg y n#1KT3|Q/bޙ _i$KWvP|Ăp++[_|?dåvrAEd<K.%Fa7'R[GzEjf8<% pp+W~˗ƽx"VO{ ]ޡNAwT/:2E&p+)- zK3o x+G xD--u[( F<*+73ϓ>7ݪi^KgºA54` -b"X[yuxDSyY1̢:MuMxMG%>(2v1neLVIcrp64rW>ߋ^,ּ1/S5m+HKM+SZ:ige:p4;] Xe$t%at6t3hz5X5ͬZ>1mir 7c ~Ù*tk{[+bZYC|ړG=6rMnfDflcĬDhO[#s<#>D<xEs> N[{H~CpJQFǜ}ѾmIaVIvT]FLh;СKT67vj)5p>Lw웫÷_iƿA񅞓M_oDʕd WD86szυ|%x֭ JFX'i)mI1(zg?f~ڷϥrAmB`M.tA dG,"\K 'ѽD%Ն `2>*wnm=VٟCjmE2 +v,H`WtU.zZ|K|4Χ?ͯYJ(8Z) orF+,xM|7vWVz)=x?$HPd k(k;_I٪Agrt64Yk;m4(|yy,$}c+?ፆ[xvGo<CneMuS+7J5g٦Ӿ,^:sqm5HmwI l h00= .~>qk:ޥ0b`Ks$ !AdMM~Qyg$Wş5?Oc?C׫PEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEP__+_!+6'W,BWpS;ms>5MBoY]Z^{ۇ%V7rGtWصsw9//й('*?wBrr_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOTixA?!W[E}Ù/g%<;G\uQg9r_^;sÿPOUs-o[F43K8@̪[h A] [[NWVH+$ƿ]mr_!O^-Q_Q@Q@Q@Q@Q@Q@q>_aca]m7SӮ zOm$r.ce#v쨠$m._Vo?'3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k 'm6xH?ˬx3_o[Q@5|*)Vܦi,-?kZZ\ҹ@=??G^*e 9)k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqC5G?(k '  @ÏOW<??G3_/?>^f_}~8!z(?#AqCoՃ [( yq=z|;&°B 6 .]]u+I<<#K,b~nEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQE\|5 7ekV@$J-VH=j+crpI߹ŊCjݏϟ'x_~BwAW~/3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~Bw'x_?"C?ϟ'x_~BwA~/?3?~BwV]'OW&.mmo걇ePY®IE~QI՚"GZ| +p(((((((((((((((4u8RC~'|Swz~YtK@x,`ZIQAl+?^0O[|>iK=$+{c4W }̺%=Xx{a')~3-}E| f{u9 ;@DV'uMCAԬKd[+eX'| Umsӽx)?svS6O<{u?AqοN_|vO pF,ziD:TbU!#$Fs~A1|3>#Hk7cqx3US0t>3W`ǿW_?LCE)?svS6|zUA:PWQ|4ߢ|Oq+{smBX ܃׉\k =C@z9UmݧЦL" =抭\ygLHՕԀAx#:U((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((+Z|?zd4Y.:yqFٲ9)޶+?jeu;dz!yc"4|)-񿉭xĩҲͶht}ĉHG+J]U F6n&Mm"+*t#a4((69>$;m2?+ԝ]G;q:_ԞgXh6HTZY&1VbL\`AմMYŠCy=?Cx\縞~c} s|<%^I!1s˪*$ aːUEUǗ! @Tp@8mڣ!Cc'`}x'_Ɗv)Rʒ3Hr{0Gi('37#χ vhNgdR {*H>*σèjW҅Tn"={d8#EQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEQEU=B4"yȥ28eo?5rs7RᶾDvi? WHzڳw~RI@gXĜܖb 7{qt0]H]cLP$I V Ac5Mg'?_\PgEq~4?wWoƟ'ٌ3('' FGaרâحضI4q A ʈ$ 6i,?[Fߍ?O+i%`z^t~ Zt1[(P*. r 7K,:$/+ q;an,[?i,?[Fߍ?O+jG<ߏ_Ɗvi,?[Fߍ?O+hDY\򶍿?x8o/4iV_M?Cu42 \}^1;Ur%ƭXNQ2/b_[gcZk)om+72]cr-kk#V  /k^""xm6P[ K]i %ÄwSʤp#|ж'䃷6w9bI9===SPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPEPpengcheng.mind-mindformer/docs/model_cards/000077500000000000000000000000001476650141400213165ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/model_cards/bert.md000066400000000000000000000131151476650141400225750ustar00rootroot00000000000000# BERT ## 模型描述 BERT:全名`Bidirectional Encoder Representations from Transformers`模型是谷歌在2018年基于Wiki数据集训练的Transformer模型。 [论文](https://arxiv.org/abs/1810.04805)J Devlin，et al., Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019 ## 预训练数据集下载 1. 从[zhwiki](https://dumps.wikimedia.org/zhwiki/)或[enwiki](https://dumps.wikimedia.org/enwiki/)中下载数据集。 2. 使用[WikiExtractor](https://github.com/attardi/wikiextractor)提取和整理数据集中的文本，执行命令如下： ```shell pip install wikiextractor python -m wikiextractor.WikiExtractor -o -b ``` ### 数据处理 #### TFRecord类型BERT预训练数据 用户可以参考[BERT](https://github.com/google-research/bert#pre-training-with-bert)代码仓中的create_pretraining_data.py文件， 进行`TFRecord`格式文件的生成， 如果出现下述报错 ```bash AttributeError: module 'tokenization' has no attribute 'FullTokenizer' ``` 请安装`bert-tensorflow`。注意，用户需要同时下载对应的`vocab.txt`文件。 ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行一个12层的BERT模型训练 ```shell python run_mindformer.py --config configs/bert/run_bert_base_uncased.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wiki_data ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import BertForPreTraining, BertConfig BertForPreTraining.show_support_list() # 输出： # - support list of BertForPreTraining is: # - ['bert_base_uncased'] # - ------------------------------------- # 模型标志加载模型 model = BertForPreTraining.from_pretrained("bert_base_uncased") #模型配置加载模型 config = BertConfig.from_pretrained("bert_base_uncased") # {'model_config': {'use_one_hot_embeddings': False, 'num_labels': 1, 'dropout_prob': 0.1, # 'batch_size': 128, seq_length: 128, vocab_size: 30522, embedding_size: 768, num_layers: 12, # num_heads: 12, expand_ratio: 4, hidden_act: "gelu", post_layernorm_residual: True, # hidden_dropout_prob: 0.1, attention_probs_dropout_prob: 0.1, max_position_embeddings: 512, # type_vocab_size: 2, initializer_range: 0.02, use_relative_positions: False, # use_past: False, checkpoint_name_or_path: "bert_base_uncased"}} model = BertForPreTraining(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='fill_mask', model='bert_base_uncased', train_dataset='/your_path/wiki_data') trainer.train() # 开启预训练 ``` ### 多卡训练 - 单机8卡数据并行训练BERT-base模型 ```bash RANK_SIZE=$1 HOSTFILE=$2 mpirun --allow-run-as-root -n $RANK_SIZE --hostfile $HOSTFILE \ --output-filename run_distributed_train_bert \ -x NCCL_IB_HCA -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x NCCL_SOCKET_IFNAME -n $RANK_SIZE \ python run_mindformer.py --config ./configs/bert/run_bert_base_uncased.yaml --use_parallel True --run_mode train > distribute_train_gpu_log.txt 2>&1 & ``` 其中各个参数的含义： - RANK_SIZE：总共使用的卡的数量，采用单机8卡训练时，设为8 - HOSTFILE：一个文本文件，格式如下 ```text 10.1.2.3 slots=8 ``` 表示节点ip为10.1.2.3的服务器拥有8张设备卡。用户应该将自己的实际IP替换掉10.1.2.3。 日志会重定向到`distribute_train_gpu_log.txt`中。可以通过`tail -f distribute_train_gpu_log.txt`的 命令及时刷新日志。注意此时8张卡的日志都会输出到上述的文件中，造成重复输出。用户在如下的位置查看每卡的输出 ```bash tail -f run_distributed_train_bert/1/rank.0/stdout ``` 要完成上述训练只需输入命令 ```bash bash scripts/examples/masked_language_modeling/bert_pretrain_distributed_gpu.sh RANK_SIZE hostfile ``` 即可。 #### 计算Loss ```python from mindformers import BertForPreTraining, BertTokenizer from mindspore import Tensor import mindspore.common.dtype as mstype model = BertForPreTraining.from_pretrained('bert_base_uncased') tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') data = tokenizer("Paris is the [MASK] of France.", max_length=128, padding="max_length") input_ids = Tensor([data['input_ids']], mstype.int32) attention_mask = Tensor([data['attention_mask']], mstype.int32) token_type_ids = Tensor([data['token_type_ids']], mstype.int32) masked_lm_positions = Tensor([[4]], mstype.int32) next_sentence_labels = Tensor([[1]], mstype.int32) masked_lm_weights = Tensor([[1]], mstype.int32) masked_lm_ids = Tensor([[3007]], mstype.int32) output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, masked_lm_positions, masked_lm_ids, masked_lm_weights) print(output) #[0.6706] ``` ## 模型权重 本仓库中的`bert_base_uncased`来自于HuggingFace的[`bert_base_uncased`](https://huggingface.co/bert-base-uncased), 基于下述的步骤获取： 1. 从上述的链接中下载`bert_base_uncased`的HuggingFace权重，文件名为`pytorch_model.bin` 2. 执行转换脚本，得到转换后的输出文件`mindspore_t5.ckpt` ```shell python mindformers/models/bert/convert_weight.py --layers 12 --torch_path pytorch_model.bin --mindspore_path ./mindspore_bert.ckpt ``` pengcheng.mind-mindformer/docs/model_cards/blip2.md000066400000000000000000000413601476650141400226540ustar00rootroot00000000000000# BLIP2 ## 模型描述 BLIP-2: 全名`Bootstrapping Language-Image Pre-training - 2`模型是2023 年 Salesforce提出的一种多模态模型，它从现成的冻结预训练图像编码器 (ViT)和冻结的大型语言模型 (LLM)中引导视觉语言预训练 (contrastive_language_image_pretrain), 中间添加一个轻量级的 Querying Transformer 弥补了模态 gap, 该 Transformer 分两个阶段进行预训练: - 第一阶段从冻结图像编码器引导视觉-语言表示学习，强制 Q-Former 学习与文本最相关的视觉表示。 - 第二阶段基于冻结的语言模型引导从视觉到语言的生成学习，将Q-Former的输出连接到冻结的LLM，并对Q-Former进行训练，使其输出视觉表示能够被LLM解释。 [论文](https://arxiv.org/pdf/2301.12597.pdf) Junnan Li，et al., BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models, 2023 ## 预训练数据集下载 1\. coco2014数据集: ```shell ├── annotations │   ├── coco_karpathy_test.json │   ├── coco_karpathy_train.json │   └── coco_karpathy_val.json │ └── images   ├── test2014   ├── test2015   ├── train2014   └── val2014 ``` 可通过LAVIS github库提供的链接下载annotation_files ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/configs/datasets/coco/defaults_cap.yaml)) 和 images ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/datasets/download_scripts/download_coco.py))。 2\. Visual Genome数据集: ```shell ├── annotations │   └── vg_caption.json └── images └── VG_100K ``` 同上，可通过LAVIS github库提供的链接下载annotation_files ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/configs/datasets/vg/defaults_caption.yaml)) 和 images ([链接](https://github.com/salesforce/LAVIS/blob/main/lavis/datasets/download_scripts/download_vg.py))。 ## 关键配置项说明 ```yaml train_dataset: text_transforms: type: CaptionTransform prompt: "" # 二阶段中语言模型添加的prompt，目前仅支持固定prompt max_length: 32 # token ids的最大长度 tokenizer: type: LlamaTokenizer max_length: 32 # token ids的最大长度 vocab_file: "" # 词表文件路径 model: model_config: type: Blip2Config max_txt_len: 32 # token ids的最大长度，需要与train_dataset.text_transforms.max_length保持一致 checkpoint_name_or_path: "" # 模型的预训练权重，在二阶段训练时，可在此处配置一阶段训练得到的权重 prompt: False # 二阶段训练中，是否使用prompt prompt_length: 0 # 二阶段训练中，使用的固定prompt的token长度，具体值根据tokenizer及train_dataset.text_transforms.prompt得到 # 以上两项配置与train_dataset.text_transforms.prompt需要对应 vision_config: # ImageEncoder的相关配置 type: ViTConfig image_size: 224 # 输入图像大小 checkpoint_name_or_path: "vit_g_p16" # vit的权重文件 qformer_config: type: QformerConfig query_length: 32 # query数目 text_config: # 二阶段语言模型相关配置 type: LlamaConfig seq_length: 160 # 语言模型的输入seq_length大小，在训练时该值要等于qformer的query数目+语言模型输入文字token id长度 # 即 seq_length=model.model_config.qformer_config.query_length + model.model_config.max_txt_len checkpoint_name_or_path: "" # 语言模型权重文件 arch: type: XXXX # 推理任务时需要配置 processor: type: Blip2Processor image_processor: type: Blip2ImageProcessor image_size: 224 # 输入图像大小 tokenizer: type: LlamaTokenizer max_length: 128 vocab_file: "" # 词表文件路径 ``` ## BLIP-2一阶段 ### 快速使用 #### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](https://gitee.com/mindspore/transformer/blob/master/README.md#%E6%96%B9%E5%BC%8F%E4%B8%80clone-%E5%B7%A5%E7%A8%8B%E4%BB%A3%E7%A0%81) 示例命令如下，将会执行一个39层ViT和18层Attention Layer的Qformer的一阶段预训练: ```shell # pretrain python run_mindformer.py --config configs/blip2/run_blip2_pretrain_stage_1.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir [PATH_TO_DATASET] # evaluate python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml --run_mode eval \ --device_target Ascend \ --eval_dataset_dir [PATH_TO_DATASET] # predict (单图片) python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml --run_mode predict \ --device_target Ascend \ --predict_data [PATH_TO_IMAGE] # predict (数据集) python run_mindformer.py --config configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml --run_mode predict \ --device_target Ascend \ --eval_dataset_dir [PATH_TO_DATASET] ``` #### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import Blip2Qformer, Blip2Config Blip2Qformer.show_support_list() # 输出： # support list of Blip2Qformer is: # ['blip2_stage1_vit_g', 'blip2_stage1_classification'] # ------------------------------------- # 模型标志加载模型 model = Blip2Qformer.from_pretrained('blip2_stage1_vit_g') #模型配置加载模型 config = Blip2Config.from_pretrained("blip2_stage1_vit_g") config # {'img_size': 224, # 'num_query_token': 32, # 'cross_attention_freq': 2, # 'drop_path_rate': 0, # 'use_grad_checkpoint': False, # 'vit_model': 'vit_g_p16', # 'layernorm_dtype': 'float32', # 'softmax_dtype': 'float32', # 'model_type': 'blip2', # 'batch_size': 8, # 'embed_dim': 256, # 'vocab_size': 21129, # ... # 'is_training': True} model = Blip2Qformer(config) ``` - Trainer接口开启训练： ```python from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader from mindformers.trainer import Trainer # 初始化图像-文本数据集 dataset_dir = "/data" annotation_files = [ "vg/annotations/vg_caption.json", "coco2014/coco/annotations/coco_karpathy_train.json" ] image_dirs = [ "vg/images", "coco2014/coco/images" ] train_dataset = MultiImgCapDataLoader(dataset_dir=dataset_dir, annotation_files=annotation_files, image_dirs = image_dirs, stage="train") # blip2一阶段初始化预训练任务 trainer = Trainer(task='contrastive_language_image_pretrain', model='blip2_stage1_vit_g', train_dataset=train_dataset) # 开启预训练 trainer.train() ``` - Trainer接口开启评估: ```python from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader from mindformers.trainer import Trainer # 初始化图像-文本数据集 dataset_dir: "/data" annotation_files: [ "flickr30k/annotations/test.json" ] image_dirs: [ "flickr30k/images" ] eval_dataset = MultiImgCapDataLoader(dataset_dir=dataset_dir, annotation_files=annotation_files, image_dirs = image_dirs, stage="eval") # 初始化评估任务 trainer = Trainer(task='blip2_retireval', model='blip2_stage1_vit_g', eval_dataset=eval_dataset) # 开启评估 trainer.eval() ``` - Trainer接口开启推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Trainer cls_trainer = Trainer(task='zero_shot_image_classification', model='blip2_stage1_classification', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) # 加载输入，一张太阳花图片 input_data = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 加载指定的权重以完成推理 predict_result = cls_trainer.predict(input_data=input_data, predict_checkpoint='your_path_to/blip2_pretrained.ckpt') print(predict_result) # 输出 # output result is: [[{'score': 0.99999976, 'label': 'sunflower'}]] # output result is saved at: zero_shot_image_classification_result.txt # .........Predict Over!............. ``` - pipeline推理: ```python from mindformers.tools.image_tools import load_image from mindformers.pipeline import ZeroShotImageClassificationPipeline # 初始化pipeline classifier = ZeroShotImageClassificationPipeline( model='blip2_stage1_classification', candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], hypothesis_template="This is a photo of {}." ) # 太阳花图片 img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 可批次处理，输入图片数组 classifier(img) # 输出 # [[{'score': 0.99999714, 'label': 'sunflower'}, # {'score': 1.315181e-06, 'label': 'tree'}, # {'score': 7.0368844e-07, 'label': 'toy'}, # {'score': 4.7594781e-07, 'label': 'dog'}, # {'score': 3.93686e-07, 'label': 'cat'}]] ``` #### 多卡训练 - 参考 [快速启动指导](https://gitee.com/mindspore/mindformers#%E6%96%B9%E5%BC%8F%E4%B8%80%E4%BD%BF%E7%94%A8%E5%B7%B2%E6%9C%89%E8%84%9A%E6%9C%AC%E5%90%AF%E5%8A%A8) 中的多卡启动部分，blip2的任务配置文件默认为单卡启动，如需多卡启动，注意将相关配置文件中的 ```yaml use_parallel: False ``` 改为 ```yaml use_parallel: True ``` blip2现支持数据并行 (data parallel) 和模型并行 (model parallel)，需要注意根据卡数和并行模式正确配置`data_parallel` 和`model_parallel`的值 (dp * mp = 卡数)。 ##### 评估一阶段Loss ```python from mindformers import Blip2Qformer, Blip2Processor from mindformers.tools.image_tools import load_image from mindspore import Tensor # 加载模型 model = Blip2Qformer.from_pretrained('blip2_stage1_vit_g') # 加载处理器 processor = Blip2Processor.from_pretrained('blip2_stage1_vit_g') image_processor = processor.image_processor tokenizer = processor.tokenizer # 加载输入，一张太阳花图片 image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") text = ["This is a picture of sunflower."] # 输入预处理 image_input = image_processor(image) text_ids = tokenizer(text, max_length=32, padding='max_length') input_ids = Tensor(text_ids['input_ids'], ms.int32) # 放入模型计算loss output = model(image_input, input_ids, return_tuple=True) print(output) # (Tensor(shape=[], dtype=Float32, value= 4.45203), -- overall_loss = loss_itc + loss_itm + loss_lm # Tensor(shape=[], dtype=Float32, value= 0), -- loss_itc # Tensor(shape=[], dtype=Float32, value= 0.681192), -- loss_itm # Tensor(shape=[], dtype=Float32, value= 3.77084)) -- loss_lm ``` ### 模型权重 本仓库中的`blip2_stage1_classification`来自于LAVIS的一阶段预训练权重[`blip2_stage1_pretrained`](https://storage.googleapis.com/sfr-vision-language-research/LAVIS/models/BLIP2/blip2_pretrained.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`blip2_stage1_pretrained`的pytorch权重，文件名为`blip2_pretrained.pth` 2. 执行转换脚本，得到转换后的输出文件`blip2_stage1_pretrained.ckpt` ```shell python mindformers/models/blip2/convert_weight.py --torch_path blip2_pretrained.pth --mindspore_path ./blip2_stage1_pretrained.ckpt ``` ## BLIP-2二阶段 BLIP-2第二阶段是通过冻结的语言模型引导从视觉到语言的生成学习，将Q-Former的输出连接到冻结的LLM，并对Q-Former进行训练，使其输出视觉表示能够被LLM解释。 当前BLIP-2二阶段支持使用接入Llama-7b及baichuan-7b的推理任务，并暂未提供预训练模型，以下推理任务中，未加载准确的预训练权重，不保证生成结果正确。 ### 快速使用 #### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](https://gitee.com/mindspore/transformer/blob/master/README.md#%E6%96%B9%E5%BC%8F%E4%B8%80clone-%E5%B7%A5%E7%A8%8B%E4%BB%A3%E7%A0%81) 示例命令如下，将会执行一个39层ViT, 18层Attention Layer的Qformer以及Llama-7b的二阶段推理，**Note:** 当前二阶段仅支持推理任务，并暂不提供预训练模型: ```shell # predict (单图片) 当前二阶段未提供预训练模型，该预测代码仅展示使用方式，不保证预测结果 python run_mindformer.py --config configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml --run_mode predict \ --device_target Ascend \ --predict_data [PATH_TO_IMAGE] ``` **注：** 当二阶段使用baichuan_7b作为LLM模型时进行推理时，需要自行在配置文件`configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml`中 配置对应的baichuan-7b权重文件及词表文件，如何配置可参考上文中关键配置项说明章节； 权重文件和词表文件的获取及转换参考[baichuan_7b](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/baichuan.md). #### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import Blip2ImageToTextGeneration, Blip2Config Blip2ImageToTextGeneration.show_support_list() # 输出： # support list of Blip2ImageToTextGeneration is: # ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b'] # ------------------------------------- # 初始化模型方式1：直接通过预训练加载 model = Blip2ImageToTextGeneration.from_pretrained('itt_blip2_stage2_vit_g_baichuan_7b') # 初始化模型方式2：通过配置文件初始化模型 config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.model.model_config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) ``` - Trainer接口开启推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Blip2Config, Blip2ImageToTextGeneration, Trainer config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 config.checkpoint_name_or_path = "/path/to/the/1-st_pretrain_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) generate_trainer = Trainer(task='image_to_text_generation', model=model, model_name='itt_blip2_stage2_vit_g_baichuan_7b') # 加载输入，一张太阳花图片 input_data = load_image( "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 加载指定的权重以完成推理 predict_result = generate_trainer.predict(input_data=input_data, hypothesis_template="###问题：\n详细描述图片内容。\n\n###答案：") print(predict_result) # 输出 # output result is: ['###问题：\n详细描述图片内容。\n\n###答案：这是一张图片，上面有一朵黄色的花，花朵周围有许多小黄花，它们在阳光下闪闪发光。'] # output result is saved at: zero_shot_image_to_text_generation_result.txt # .........Predict Over!............. ``` - pipeline推理: ```python from mindformers.tools.image_tools import load_image from mindformers import Blip2Config, Blip2ImageToTextGeneration, pipeline config = Blip2Config.from_pretrained("itt_blip2_stage2_vit_g_baichuan_7b") # 可通过config配置更改相应的配置 config.text_config.checkpoint_name_or_path = "/path/to/the/llm_checkpoint" # 修改语言模型权重路径 config.checkpoint_name_or_path = "/path/to/the/1-st_pretrain_checkpoint" # 修改语言模型权重路径 model = Blip2ImageToTextGeneration(config) # 初始化pipeline generator = pipeline("image_to_text_generation", model=model) # 太阳花图片 img = load_image( "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 可批次处理，输入图片数组 pipeline_result = generator(img, hypothesis_template='###问题：\n详细描述图片内容。\n\n###答案：') print(pipeline_result) # ['###问题：\n详细描述图片内容。\n\n###答案：这是一张图片，上面有一朵黄色的花，花朵周围有许多小黄花，它们在阳光下闪闪发光。'] ```pengcheng.mind-mindformer/docs/model_cards/bloom.md000066400000000000000000000670441476650141400227630ustar00rootroot00000000000000# Bloom ## 1. 简介 ### 1.1 模型描述 Bloom (BigScience Large Open-science Open-access Multilingual) 是一个开源的开放权限的自回归大语言模型(LLM)，用于对用自然语言表达的多种下游任务进行文本生成。Bloom系列模型涵盖从560M到176B的多种规模，其中176B千亿级参数的大模型的预训练基于工业级的计算机集群，在46种语言和13种编程语言的文本生成中达到比拟人类写作的SOTA效果。对于训练数据集中没有显式包括的下游任务，Bloom也可以通过指令的方式，给出令人满意的zero-shot回答。 ### 1.2 仓库介绍 `Bloom` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/bloom` ```text bloom ├── __init__.py ├── bloom_tokenizer.py # tokenizer ├── bloom_config.py # 模型配置项 ├── bloom.py # 模型实现 └── layers.py # bloom 层定义 └── convert_weight.py # 将huggingface ckpt转成mindfomer ckpt ``` 2. 模型配置：`configs/bloom` ```text bloom ├── run_bloom_560m.yaml # 560m 用于推理 ├── run_bloom_7.1b.yaml # 7.1b 用于8卡训练 ├── run_bloom_65b.yaml # 65b 用于96卡训练 └── run_bloom_176b.yaml # 176b 用于128卡训练 ``` 其中Bloom_7.1b可在单机单卡上推理，在单机8卡上训练；Bloom-65B训练至少96卡；Bloom_176B训练至少128卡。 ### 1.3 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0 - MindFormers版本：dev --- ## 2. 前期准备 ### 2.1 数据集制作 这里以Alpaca为例，数据大概21MB,用于调试。 首先去官方下载[alpaca_data.json文件](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) 然后调用`mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py`脚本将json转换成mindrecord文件。 ```bash python mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py --input_dataset_file=XXX/alpaca_data.json --output_path=XXX --N=51200 ``` 其中`--N=51200`表示将json中的52002条数据中的前51200转换成mindrecord(推荐)，`--N=-1`将转换全部json中的数据. 在执行此脚本时，对于每个prompt如下操作将被执行： - 将问题和回答按照模板制作成prompt text; - 使用BloomTokenizer将prompt从text转成token ids; - 添加eos_token_id直到seq_length。 执行文本后，`--output_path`目录下将生成mindrecord文件。 ### 2.2 CheckPoint转换（选读） 作为参考，这里描述CheckPoint在HuggingFace和MindSpore间的转换，在不同分布式策略间的转换。Bloom_7.1bB的推理、预训练、finetune对这部分没有依赖，可以直接跳到下一章。 #### 2.2.1 在HuggingFace和MindSpore间的转换 Mindformers可以直接通过高级接口下载转换好的560M和7.1B两种规模的ckpt,无需手动转换。其中560m的原始Checkpoint来自于huggingface的[Bloomz-560m](https://huggingface.co/bigscience/bloomz-560m)；7.1B的原始Checkpoint来自于huggingface的[Bloomz-7B1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt). higgingface到mindformers的CheckPoint转换由一下命令完成。 ```bash cd mindformers/models/bloom python convert_weight.py --n_head=xx --hidden_size=xx --torch_path=path_to_hf_bin_file_or_folder --mindspore_path=output_path ``` 其中`--n_head=xx --hidden_size=xx`根据模型定义，bloom_560m的分别为16/1024; bloom_7.1b的分别为32/4096. #### 2.2.2 在不同分布式策略间的转换 通常训练采用分布式训练，而推理采用单机单卡。所以训练后涉及ckpt从分布式策略到单机策略的切换。 策略文件会在每次训练前的图编译环节生成，保存在`mindformers/scripts/mf_parallel0/ckpt_strategy.ckpt`。在转换前，需要将每个机器上的每个rank的`ckpt_strategy.ckpt`拷贝到相同路径下， 如`./src_strategy_dir`或`./dst_strategy_dir`，再将他们合并。 ```python import mindspore as ms # src_strategy_dir/stra0.ckpt, src_strategy_dir/stra1.ckpt ... src_strategy_dir/stra127.ckpt ms.merge_pipeline_strategys("./src_strategy_dir", "./src_strategy.ckpt") # dst_strategy_dir/stra0.ckpt, dst_strategy_dir/stra1.ckpt ... dst_strategy_dir/stra127.ckpt ms.merge_pipeline_strategys("./dst_strategy_dir", "./dst_strategy.ckpt") ``` 然后将checkpoint从src_strategy转换成dst_strategy。如果src_strategy或dst_strategy策略为单机单卡而非分布式，则不需要提供策略文件。 ```python import mindspore as ms # 如果src或dst策略为单机单卡而非分布式，则不需要提供策略文件。 # src_checkpoints_dir/rank_0/xxx.ckpt, src_checkpoints_dir/rank_1/xxx.ckpt ... ms.transform_checkpoints( src_checkpoints_dir, dst_checkpoints_dir, ckpt_prefix = "new_", src_strategy_file = "./src_strategy_file.ckpt", dst_strategy_file = "./dst_strategy_file.ckpt") ``` --- ## 3. Bloom推理 这里以Bloom_560m和bloom_7.1b为例，介绍bloom推理。 ### 3.1 基于Pipeline推理 可以通过一下两种方法实列化Bloom pipeline：`pipeline`接口或AutoClass的`from_pretrain`接口。两者都会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/bloom` - 调用pipeline接口: ```python from mindformers import pipeline bloom_ppl = pipeline(task='text_generation', model='bloom_560m', max_length=256) ``` - 调用AutoClass的from_pretrain接口： ```python from mindformers import AutoModel, AutoTokenizer, TextGenerationPipeline model = AutoModel.from_pretrained("bloom_560m") tokenizer = AutoTokenizer.from_pretrained("bloom_560m") bloom_ppl = TextGenerationPipeline(model, tokenizer, max_length=256) ``` 实例化`bloom_ppl`之后，可以将问题传入。 ``` python result = bloom_ppl([ "what color is the sky?", "Translate to English: Je t’aime." ]) print(result) # expect print result # {'text_generation_text': ['what color is the sky? blue']}, {'text_generation_text': ['Translate to English: Je t’aime. I love you.']}] ``` ### 3.2 基于API接口推理 推理也可以用更底层的MindSpore API. ```python import numpy as np import mindspore as ms from mindformers import AutoTokenizer from mindformers.models.bloom import BloomConfig, BloomLMHeadModel ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) # ############################## # # bloom_560m config # CKPT_FILE="bloom_560m" # SEQ_LENGTH = 256 # config = BloomConfig( # param_init_type="float16", # embedding_init_type="float16", # checkpoint_name_or_path=CKPT_FILE, # max_decode_length=SEQ_LENGTH, # seq_length=SEQ_LENGTH, # hidden_size=1024, # num_layers=24, # num_heads=16, # hidden_dropout_rate=0.0, # attention_dropout_rate=0.0, # batch_size = 1, # use_past = True # # ) # ############################## # 7B CKPT_FILE = "bloom_7.1b" # CKPT_FILE also takes absolute path to ckpt file, e.g. # "/home/xxx/mindformers/checkpoint_download/bloom/bloom_7.1b.ckpt" SEQ_LENGTH = 256 config = BloomConfig( param_init_type="float16", embedding_init_type="float16", checkpoint_name_or_path=CKPT_FILE, max_decode_length=SEQ_LENGTH, seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, batch_size = 1, use_past = True ) def chat(): tokenizer = AutoTokenizer.from_pretrained("bloom_560m") model = BloomLMHeadModel(config) model.set_train(False) question_list = [ "what color is the sky?", "Translate to English: Je t’aime.", ] while True: if question_list: question = question_list.pop(0) else: question = input("please input your question: ") inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = model.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` - Bloom_560m的预期输出为: - what color is the sky?_**blue**_ - Translate to English: Je t’aime. _**I love you.**_ - Bloom_7.1B的预期输出为: - what color is the sky?_**blue**_ - Translate to English: Je t’aime. _**I love you.**_ --- ## 4. 单机8卡训练或微调 我们以单机8卡训练/微调Bloom_7.1b为例。默认配置为 | | SEQ | DP | MP | PP | uB | num_uB | GBS | |------------|:----:|:--:|:--:|:--:|:--:|:------:|:---:| | Bloom_7.1B | 2048 | 1 | 4 | 2 | 4 | 2 | 8 | 在YAML配置文件中，具体设置变量如下 - __DP__: 数据并行维度, `parallel_config.data_parallel = 1`; - __MP__: 模型并行维度, `parallel_config.model_parallel = 4`; - __PP__: 流水线并行维度, `parallel_config.pipeline_stage = 2`; - __uB__: 每个microBatch大小, `runner_config.batch_size = 4`; - __num_uB__: microBatch的数量, `parallel_config.micro_batch_num = 2`, 常稳训练时建议用16; - __GBS__: Global Batch Size = num_uB x uB x DP. ### 4.1 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` 若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成。 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 4.2 启动预训练或微调 通过`/configs/bloom/run_bloom_7.1b.yaml`中的`model:checkpoint_name_or_path:`字段来控制是否加载CKPT. 请安如下方式区分预训练或者微调： | | pretrain | finetune | |--------------------------------------------: |:----------:|:----------:| | `load_checkpoint` | "" | "xxx/bloom_7.1b.ckpt'"| | `train_dataset:data_loader:dataset_dir` | PRETRAIN_DATASET | FINETUNE_DATASET | | `parallel_config.micro_batch_num` | 16 | 16 | | `runner_config.epochs` | 1 | 3 | | `lr_schedule.learning_rate` | 0.00001 | 0.000001 | | `lr_schedule.lr_end` | 0.000001 | 0.000001 | | `lr_schedule.warmup_steps` | 1000 | 0 | | `lr_schedule.total_steps` | -1 | -1 | | `callbacks.save_checkpoint_steps` | 1000 | 400 | | 初始loss | 12.xx | 3.xx | 其中`PRETRAIN_DATASET`和`FINETUNE_DATASET`都可以用alpaca_2049调试。 ```bash cd scripts # pretrain bash run_distribute.sh RANK_TABLE_FILE ../configs/bloom/run_bloom_7.1b.yaml [0,8] train 8 # finetune bash run_distribute.sh RANK_TABLE_FILE ../configs/bloom/run_bloom_7.1b.yaml [0,8] finetune 8 ``` 其中RANK_TABLE_FILE为上一步生成的rank table文件。执行后，相关日志输出在`mindformers/output/log/`训练过程中保存的ckpt存在`mindformers/output/checkpoint`目录下。可以通过`tail -f ../output/log/rank_7/mindformers.log`来查看当前的训练情况。参考收敛曲线如下图所示。 >默认情况下，图编译大约1.5小时，51200条alpaca数据集微调大约4小时/epoch. ![输入图片说明](https://foruda.gitee.com/images/1689316828651589261/75882b0b_11484396.png) ### 4.3 微调后对话效果 在`mindformers/scripts`路径下执行以下脚本`combine_ckpt.py`.这个脚本会 - 对strategy进行合并 - 清理微调ckpt文件中的优化器状态 - 合并微调ckpt文件用于单机推理 ```python # combine_ckpt.py import os import mindspore as ms CKPT_SUFFIX = "300_8" # 300(sink number) * 8 (sink size) = 2400 step CLEANED_CKPT_DIR = "../output/checkpoint_cleaned" COMBINED_CKPT_DIR = "../output/checkpoint_combined" COMBINED_STGY = "../output/strategy/ckpt_strategy.ckpt" # combine straegies ms.merge_pipeline_strategys("../output/strategy", COMBINED_STGY) # clean ckpt by removing optimizer states for rank_id in range(8): input_file_name = f"../output/checkpoint/rank_{rank_id}/mindformers_rank_{rank_id}-{CKPT_SUFFIX}.ckpt" params = ms.load_checkpoint(input_file_name) new_params = [{"name": key, "data": val} for key, val in params.items() if not ("accu_grads" in key or "adam_" in key) ] save_path = os.path.join(CLEANED_CKPT_DIR, f"rank_{rank_id}") os.makedirs(save_path, exist_ok=True) ms.save_checkpoint(new_params, f"{save_path}/cleaned.ckpt") print(f"saved {save_path}") # combine ckpt ms.transform_checkpoints(CLEANED_CKPT_DIR, COMBINED_CKPT_DIR, ckpt_prefix = "combined_", src_strategy_file = COMBINED_STGY) ``` 然后执行以下脚本进行新的对话。相比与3.2章的脚本，这里有三个改动： - `CKPT_FILE`改成新生成的ckpt文件 - 对问题使用了Alpaca数据集相同prompt模板 - 模型embedding_init_type改成FP32因为SFT用的是FP32 > 以下脚本针对Alpaca数据集的prompt模板。如果使用其他数据集微调，请更换对应模板。 ```python import numpy as np import mindspore as ms from mindformers import AutoTokenizer from mindformers.models.bloom import BloomConfig, BloomLMHeadModel ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) alpaca_prompt = ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:\n") # 7B CKPT_FILE = "xxx/mindformers/output/checkpoint_combined/rank_0/combined_0.ckpt" SEQ_LENGTH = 1024 config = BloomConfig( param_init_type="float16", embedding_init_type="float32", checkpoint_name_or_path=CKPT_FILE, max_decode_length=SEQ_LENGTH, seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, batch_size = 1, use_past = True ) def chat(): tokenizer = AutoTokenizer.from_pretrained("bloom_560m") model = BloomLMHeadModel(config) model.set_train(False) question_list = [ "why the earth is unique?", "why the sky is blue?", "write a job application for a data scientist and explain your related work experience." ] while True: if question_list: question = question_list.pop(0) else: question = input("please input your question: ") question = alpaca_prompt.format_map({"instruction":question}) inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = model.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` 预期的对话效果大致为下表所示 | | Before | After | |:--------------------------------------------------------------------------------------:|:-----------------------------------------------:|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | why the sky is blue? | light from the sun is scattered | The sky is blue because of the presence of water droplets in the atmosphere. These droplets reflect light back to the sky, causing the sky to appear blue. | | what would be the best way to travel from San Fransisco to New York? | take a flight | The best way to travel from San Francisco to New York is by taking the flight. The flight is the fastest and most convenient way to travel from San Francisco to New York. | | write a job application for a data scientist and explain your related work experience. | | Dear Employer, I am writing to apply for the position of Data Scientist. I have over 5 years of experience in data science and machine learning, and I am excited to join your team. I have experience in supervised and unsupervised machine learning algorithms, data visualization, and data cleaning. I am also proficient in Python, R, and SQL. I am looking forward to discussing my qualifications further and hearing from you soon. Sincerely, [Your Name] | | why the earth is unique? | it is the only planet with a liquid surface | The Earth is unique because it is the only planet with a liquid surface, a magnetic field, and a protective atmosphere. It is also the only planet with a life-supporting atmosphere and a diverse and abundant life-supporting ecosystem. | ## 5. 多机多卡的训练 这里以12机96卡训练65B为例。默认配置为 | | SEQ | DP | MP | PP | uB | num_uB | GBS | |------------|:----:|:--:|:--:|:--:|:--:|:------:|:---:| | Bloom_7.1B | 2048 | 2 | 4 | 12 | 1 | 48 | 96 | 在YAML配置文件中，具体设置变量如下 - __DP__: 数据并行维度, `parallel_config.data_parallel = 2`; - __MP__: 模型并行维度, `parallel_config.model_parallel = 4`; - __PP__: 流水线并行维度, `parallel_config.pipeline_stage = 12`; - __uB__: 每个microBatch大小, `runner_config.batch_size = 1`; - __num_uB__: microBatch的数量, `parallel_config.micro_batch_num = 48`, 常稳训练时建议用80; - __GBS__: Global Batch Size=num_uB x uB x DP. ### 5.1 生成HCCL文件 - step1: 参考单机多卡启动方式，在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； - step2: 将每台机器上生成的RANK_TABLE_FILE拷贝到一起，执行merge_hccl.py脚本将不同机器上生成的RANK_TABLE_FILE文件中的hccl*.json进行合并，包括server_list合并，server_count设为机器数，rank_id顺序增加，生成合并后的RANK_TABLE_FILE文件， - step3: 拷贝到所有机器中，保证不同机器上的RANK_TABLE_FILE相同； ```shell # step1：在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json # step3：将step2得到的合并后的RANK_TABLE_FILE文件分别复制到所有的机器上。 ``` ### 5.2 启动预训练或微调 在每台机器上启动`bash run_distribute.sh`。 ```bash # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE ../configs/bloom/run_bloom_65b.yaml [0,8] train 96 # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE ../configs/bloom/run_bloom_65b.yaml [$rank_start,$rank_end] train 96" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` --- ## 附录A 更多推理技巧 ### A.1 模型并行推理（以1机8卡推理Bloom_7.1B为例） 这里我们以1机器8卡推理bloom_7.1B为例。涉及两个文件`chat.py`和`run_chat.py`。 ```text /SOME/PATH/ ├── chat.py # 负责定义一个并行进程 └── run_chat.py # 负责多次执行chat.py并拉起分布式 ``` 加载ckpt有两种方式, 由`run_chat.py` 中的`DISTRIBUTED_CKPT_PATH`变量来控制。这里默认使用`DISTRIBUTED_CKPT_PATH=""`代表的`Load then Shard`的方式加载ckpt. | 分布式加载ckpt的方法 | Load then Shard | Shard then Load | |-----------------------:|:-----------------|:-----------------| | DISTRIBUTED_CKPT_PATH= | "" | "/path/to/distributed/ckpt/" | |说明| 先加载全量ckpt，然后切分模型。|先切分模型，然后加载分布式ckpt。| || 不用预先按照策略切分ckpt，推理时可以灵活调整策略。|需要确定推理的分布式策略，并按照策略预先切分ckpt。 | || 对host内存的占用较高。| 对host内存的占用较低。| |适用| 适用于较小模型，如`560m`，`7.1b`。|适用于较大模型，如`65b`, `176b`。 | ``` python # >>> `chat.py`文件 import os import time import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers import pipeline from mindformers import BloomLMHeadModel, BloomConfig, AutoTokenizer from mindformers import init_context from mindformers.modules import TransformerOpParallelConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools import logger SEQ_LENGTH = 256 DISTRIBUTED_CKPT_PATH = os.getenv("DISTRIBUTED_CKPT_PATH", "") # set context context_config = {"device_target": "Ascend", "mode": 0, "max_device_memory": "31GB"} parallel_context_config = {"parallel_mode": 1, "gradients_mean": False, "full_batch": True} rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_context_config) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() # config blooom 7.1b config = BloomConfig( embedding_init_type="float32" if DISTRIBUTED_CKPT_PATH else "float16", checkpoint_name_or_path="" if DISTRIBUTED_CKPT_PATH else "bloom_7.1b", seq_length=SEQ_LENGTH, hidden_size=4096, num_layers=30, num_heads=32, hidden_dropout_rate=0.0, attention_dropout_rate=0.0, top_k=1, top_p=1, do_sample=True, parallel_config=TransformerOpParallelConfig( data_parallel=1, model_parallel=8, pipeline_stage=1 ) ) def chat(): # init bloom tokenizer = AutoTokenizer.from_pretrained("bloom_560m") bloom = BloomLMHeadModel(config) bloom.set_train(False) if DISTRIBUTED_CKPT_PATH: # find the sharded ckpt path for this rank ckpt_path = os.path.join(DISTRIBUTED_CKPT_PATH, "rank_{}".format(rank_id)) ckpt_path = get_last_checkpoint(ckpt_path) logger.info("ckpt path: %s", str(ckpt_path)) # shard bloom and load sharded ckpt m = Model(bloom) m.infer_predict_layout(ms.Tensor(np.ones(shape=(1, SEQ_LENGTH)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(bloom, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) question_list = [ "请问为什么说地球是独一无二的？", "Translate to English: Je t’aime.", ] for question in question_list: inputs = tokenizer.encode(question) inputs = np.array([inputs]).astype(np.int32) # add batch dim outputs = bloom.generate(inputs, max_length=None, do_sample=False, eos_token_id=2) outputs = outputs[0] # remove batch dim print(tokenizer.decode(outputs)) if __name__ == "__main__": chat() ``` ```bash # >>> `run_chat.sh`文件 # define variable export RANK_SIZE=8 export RANK_TABLE_FILE="../hccl_8p.json" # <<< change to yours # distributed ckpt path to load after sharding model. # use "" if load full ckpt before sharding model. export DISTRIBUTED_CKPT_PATH="" export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./chat.py &> minformers_$RANK_ID.log & done ``` 使用一下命令拉起分布式推理: ```bash bash run_chat.sh ``` 日志可以通过`tail -f mindformers_0.log`查看。预期结果与单机单卡`bloom_7.1b`推理相同： - 请问为什么说地球是独一无二的？ _**地球是太阳系中唯一有生物的地方**_ - Translate to English: Je t’aime. _**I love you.**_ ## 附录B BELLE [BELLE](https://github.com/LianjiaTech/BELLE)（Be Everyone's Large Language model Engine）是一个旨在促进中文对话大模型开源社区发展的组织。BELLE-7B是基于Bloomz-7B-mt，使用中文问答数据集微调出来开源的中文对话模型。根据微调所使用的中文数据大小分为0.2M, 0.6M, 1M, 2M四个权重。 微调的模板为 > Human: {input} \n\nAssistant:{output} 原始的开源BELLE的数据集和权重可以通过以下链接获得 | | 文件 | 链接 | |----------| --- |--------------------------------------------------------| | 2M SFT数据集 | train_2M_CN.json | https://huggingface.co/BelleGroup/BELLE-7B-2M | | 2M 模型权重 | pytorch_model.bin | https://huggingface.co/datasets/BelleGroup/train_2M_CN | 数据集和权重的转换参考2.1章和2.2章，命令如下： ```bash # 数据集转换 python mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py --input_dataset_file=XXX/train_2M_CN.json --output_path=XXX # 权重转换 cd mindformers/models/bloom python convert_weight.py --n_head=32 --hidden_size=4096 --torch_path=xxx/pytorch_model.bin --mindspore_path=output_path ``` 数据集和权重的转换到mindspore后，可以按照Bloom的方式进行推理和微调。 pengcheng.mind-mindformer/docs/model_cards/clip.md000066400000000000000000000163321476650141400225740ustar00rootroot00000000000000# CLIP ## 模型描述 CLIP (Contrastive Lanuguage-Image Pre-Training)：是一种基于图文对进行训练的transformer模型，在预训练完成以后，任意给定一张图片，它可以在不用微调的情况下，完成对图片的零样本分类。 [论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. 注：CLIP训练代码未开源，故MindFormers提供训练pretrain、finetune功能，但不不保证精度，目前仅对zero shot图片分类精度做了对齐。 ## 数据集准备 ### 预训练使用数据集：Flickr8k([链接](https://pan.baidu.com/s/1LRlQUL1MRipPL4MLOdExzg)，密码: s4be) - 数据集大小：2.2G，共8000张彩色图像，每张图像都与五个不同的标题配对，这些标题提供了对图片中物体和事件的内容描述 - 训练集：6000张图像 - 验证集：1000张图像 - 测试集：1000张图像 - 数据格式：RGB ```bash 数据集目录格式 └─Flickr8k ├─Flickr8k_Dataset | └─Flickr8k_Dataset └─Flickr8k_text ├─Flickr8k.devImages.txt ├─Flickr8k.testImages.txt ├─Flickr8k.trainImages.txt └─Flickr8k.token.txt ``` ### 零样本下游任务使用的数据集：[Cifar100](http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz) - 数据集大小：161M，共60000张图片，100个类别 - 训练集：50000张图片 - 测试集：10000张图片 - 数据格式：二进制文件 ```bash 数据集目录格式 └─cifar-100-python ├─meta ├─test └─train ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 当前clip多卡精度有异常，仅支持单卡，后续版本会修复 ```shell # pretrain python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml --run_mode train --train_dataset_dir [DATASET_PATH] # evaluate python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import CLIPModel, CLIPConfig CLIPModel.show_support_list() # 输出： # - support list of CLIPModel is: # - ['clip_vit_b_32', 'clip_vit_B_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # - ------------------------------------- # 模型标志加载模型 model = CLIPModel.from_pretrained("clip_vit_b_32") #模型配置加载模型 config = CLIPConfig.from_pretrained("clip_vit_b_32") # {'text_config': {'hidden_size': 512, 'vocab_size': 49408, 'max_position_embeddings': 77, # 'num_hidden_layers': 12}, 'vision_config': {'hidden_size': 768, 'image_size': 224, 'patch_size': 32, # 'num_hidden_layers': 12}, 'projection_dim': 512, 'ratio': 64, 'checkpoint_name_or_path': 'clip_vit_b_32', # 'dtype': 'float16'} model = CLIPModel(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化预训练任务 trainer = Trainer(task='contrastive_language_image_pretrain', model='clip_vit_b_32', train_dataset='./Flickr8k') trainer.train() # 开启预训练 #初始化零样本图像分类下游任务 trainer = Trainer(task='zero_shot_image_classification', model='clip_vit_b_32', eval_dataset='./cifar-100-python') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 使用训练好的权重进行评估和推理 trainer.evaluate(eval_checkpoint=True) predict_result = trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 从obs下载训练好的权重并进行评估和推理 trainer.evaluate() #下载权重进行评估 predict_result = trainer.predict(input_data=img, top_k=3) #下载权重进行推理 print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline from mindformers.tools.image_tools import load_image classifier = pipeline("zero_shot_image_classification", model="clip_vit_b_32", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") classifier(img) # 输出 # [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, # {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, # {'score': 2.396818e-06, 'label': 'cat'}]] ``` ## 模型性能 | model | task_type | model_Type | datasets | Top1-accuracy | log | example | |:-----:|:------------------------------:|:-----------------------------------------------------------------------------:|:--------:|:---------------------------------------:|:---:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | clip | pretrained | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | flickr8k | \ | \ | pretrain [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flickr8k.sh) | \| | clip | zero_shot_image_classification | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | cifar100 | 57.24%
61.41%
69.67%
68.19% | \ | eval [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh)
predict [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh) | ## 模型权重 本仓库中的`clip_vit_b_32`来自于openai/clip的[`ViT-B/32`](https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-B/32`的模型权重 2. 执行转换脚本，得到转换后的输出文件`clip_vit_b_32.ckpt` 其余参数获取方式相同 ```shell python mindformers/models/clip/convert_weight.py --torch_path "PATH OF ViT-B/32.pt" --mindspore_path "SAVE PATH OF clip_vit_b_32.ckpt" ```pengcheng.mind-mindformer/docs/model_cards/glm.md000066400000000000000000001033341476650141400224230ustar00rootroot00000000000000# ChatGLM6B ## 模型描述 ChatGLM-6B 是一个开源的、支持中英双语的对话语言模型，基于 [General Language Model (GLM)](https://github.com/THUDM/GLM) 架构，具有 62 亿参数。ChatGLM-6B 使用了和 ChatGPT 相似的技术，针对中文问答和对话进行了优化。经过约 1T 标识符的中英双语训练，辅以监督微调、反馈自助、人类反馈强化学习等技术的加持，62 亿参数的 ChatGLM-6B 已经能生成相当符合人类偏好的回答，更多信息请参考清华的[博客](https://chatglm.cn/blog)。在此仓中，提供ChatGLM6B的推理和微调能力。 ## 仓库介绍 `chatGLM6B` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/glm` ```bash glm ├── __init__.py ├── attention.py # 自注意力 ├── chatglm_6b_tokenizer.py # tokenizer ├── glm_config.py # 模型配置项 ├── glm.py # 模型实现 └── layers.py # glm 层定义 ``` 2. 模型配置：`configs/glm` ```bash glm ├── run_glm_6b_fintune.yaml # 全量微调启动配置 ├── run_glm_6b_lora.yaml # lora低参微调启动配置 ├── run_glm_6b_infer.yaml # 推理启动配置 └── run_glm_6b_lora_infer.yaml # lora模型推理启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0rc1 / 1.10.1 - MindFormers版本：dev 推理可在单机单卡上完成部署 全量微调训练需要最少单机8卡，Lora微调训练最少需要1卡 ## ChatGLM6B推理 > 需开发者提前pip安装。具体接口说明请参[API接口](https://gitee.com/mindspore/transformer/wikis/API/) ### AutoClass推理 可以使用AutoClass接口，通过模型名称获取相应的模型/tokenizer实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/glm` 首次运行pipeline推理时需要进行模型编译，需等待一段时间 ```python >>> from mindformers import AutoModel, AutoTokenizer, TextGenerationPipeline >>> model = AutoModel.from_pretrained("glm_6b_chat") >>> tokenizer = AutoTokenizer.from_pretrained("glm_6b") >>> pipeline = TextGenerationPipeline(model, tokenizer, max_length=2048) >>> pipeline("你好") [{'text_generation_text': ['你好 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。']}] ``` > 注：`AutoModel.from_pretrained()` 接口当前支持 `glm_6b` 和 `glm_6b_chat` 两类模型，前者为通用模型，后者具备推理加速特性，仅用于推理，两者共享权重，在推理场景下建议使用后者，以获得更快的推理体验 ### pipeline推理 也可以不实例化构造模型，直接通过指定任务模型与模型名的方式进行pipeline的构造 pipeline中，也可以使用 `glm_6b_chat` 模型加速推理 ```python >>> from mindformers import pipeline >>> task_pipeline = pipeline(task='text_generation', model='glm_6b_chat', max_length=2048) >>> task_pipeline('你好') [{'text_generation_text': ['你好 你好👋!我是人工智能助手 ChatGLM-6B,很高兴见到你,欢迎问我任何问题。']}] ``` ### 基于API接口的推理 可使用如下`chat_glm.py`脚本： ```python import time import mindspore as ms import numpy as np from mindformers.models.glm import GLMConfig, GLMChatModel from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.models.glm.glm_processor import process_response config = GLMConfig( position_encoding_2d=True, use_past=True, is_sample_acceleration=True, ) def chat_glm(): ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=7) model = GLMChatModel(config) ms.load_checkpoint("./checkpoint_download/glm/glm_6b.ckpt", model) tokenizer = ChatGLMTokenizer('./checkpoint_download/glm/ice_text.model') prompts = ["你好", "请介绍一下华为", "用python写一个快排"] history = [] for query in prompts: input_ids = tokenizer(query)['input_ids'] start_time = time.time() outputs = model.generate(input_ids, max_length=config.max_decode_length, do_sample=False) end_time = time.time() print(f'generate speed: {outputs[0].shape[0]/(end_time-start_time):.2f} tokens/s') response = tokenizer.decode(outputs) response = process_response(response[0]) print(response) if __name__ == "__main__": chat_glm() ``` ## 微调 下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法 ### 数据处理（在线加载与离线生成二选一，优先推荐在线加载方式） ADGEN 数据集任务为根据输入（content）生成一段广告词（summary）。数据集可选离线生成 `Mindrecord` 或者实时生成两种方式，两种方式选其一即可。 ```json { "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。" } ``` 从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) 或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载处理好的 ADGEN 数据集，将解压后的 `AdvertiseGen` 任意目录下 #### 1. 在线加载 将任务配置文件 `configs/glm/run_glm_6b_*.yaml` 中的 `==== dataset config ====` 部分中的 `dataset_dir` 指向 `*.json` 文件，`vocab_file` 指向词表文件，**跳过** “2. 离线生成” 步骤。 #### 2. 离线生成 将任务配置文件 `configs/glm/run_glm_6b_*.yaml` 中的 `==== dataset config ====` 部分替换成： ```yaml train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels", "position_ids", "attention_mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: CausalLanguageModelDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: CausalLanguageModelDataset dataset_config: *eval_dataset ``` 使用 `mindformers/tools/dataset_preprocess/glm/adgen_dataset.py` 脚本将数据集处理成mindrecord格式。 执行命令生成训练数据集： ```bash python adgen_dataset.py \ --input_file /path/to/AdvertiseGen/train.json \ --vocab_file /path/to/ice_text.model\ --output_file /path/to/AdvertiseGen/train_0604_128.mindrecord \ --max_source_length 64 \ --max_target_length 64 \ --mode train ``` 执行命令生成评估数据集： ```bash python adgen_dataset.py \ --input_file /path/to/AdvertiseGen/dev.json \ --vocab_file /path/to/ice_text.model \ --output_file /path/to/AdvertiseGen/eval_0604_256.mindrecord \ --max_source_length 256 \ --max_target_length 256 \ --mode eval ``` ### 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` > 注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 全参微调 #### run_mindformers脚本启动全参微调 全参微调使用 `configs/glm/run_glm_6b_finetune.yaml` 配置文件，配置文件中定义了微调所需的各配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm/run_glm_6b_finetune.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm/run_glm_6b_finetune.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 启动全参微调脚本： ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_8p_01234567_127.0.1.1.json ../configs/glm/run_glm_6b_finetune.yaml '[0,8]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的glm/run_glm_6b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如 '[0,8]' 为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` > 注：由于GLM6B的模型较大，无法在单卡上运行，此处仅提供分布式启动脚本 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint #### Trainer高阶接口启动全参微调 下面提供一个使用高阶接口进行GLM模型开发的样例脚本 `task.py`，用户可参照以下步骤熟悉如何使用高阶接口进行GLM模型的训练开发 ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, run_mode='train', task='text_generation', model_type='glm_6b', checkpoint_path='./glm_6b.ckpt', train_dataset='./train', eval_dataset='./eval', predict_data='你好', batch_size=4, dp=1, mp=1, pp=1, micro_size=1, op=False): if use_parallel.lower() == "true": use_parallel = True else: use_parallel = False # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=batch_size, learning_rate=5e-5, warmup_steps=100, sink_mode=True, sink_size=4) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) if run_mode == 'train': # 训练 task.train() elif run_mode == 'finetune': # 微调 task.finetune(checkpoint_path) elif run_mode == 'eval': # 评估 task.evaluate(checkpoint_path) elif run_mode == 'predict': # 推理，仅支持单卡推理 assert use_parallel == False, "only support predict under stand_alone mode." result = task.predict(input_data=predict_data) print(result) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--run_mode', default='train', required=True, help='set run mode for model.') parser.add_argument('--use_parallel', default=False, help='open parallel for model.') parser.add_argument('--task', default='text_generation', required=True, help='set task type.') parser.add_argument('--model_type', default='glm_6b', required=True, help='set model type.') parser.add_argument('--checkpoint_path', default=None, help='set checkpoint path.') parser.add_argument('--train_dataset', default=None, help='set train dataset.') parser.add_argument('--eval_dataset', default=None, help='set eval dataset.') parser.add_argument('--batch_size', default=4, help='batch size of dataset.') parser.add_argument('--data_parallel', default=1, type=int,help='set data parallel number. Default: None') parser.add_argument('--model_parallel', default=1, type=int, help='set model parallel number. Default: None') parser.add_argument('--pipeline_parallel', default=1, type=int, help='set pipeline parallel number. Default: None') parser.add_argument('--micro_size', default=1, type=int, help='set micro batch number. Default: None') parser.add_argument('--optimizer_parallel', default=False, type=bool, help='whether use optimizer parallel. Default: None') args = parser.parse_args() print(args) main(run_mode=args.run_mode, task=args.task, use_parallel=args.use_parallel, model_type=args.model_type, checkpoint_path=args.checkpoint_path, train_dataset=args.train_dataset, eval_dataset=args.eval_dataset, batch_size=int(args.batch_size), dp=args.data_parallel, mp=args.model_parallel, pp=args.pipeline_parallel, micro_size=args.micro_size, op=args.optimizer_parallel) ``` 因GLM模型过大，**无法在单卡上启动训练**，因此需要**通过分布式脚本拉起多卡训练任务** 在此提供 `run_distribute_single_node.sh` 单机多卡标准启动脚本，用户可用其拉起分布式训练 ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_single_node.sh [EXECUTE_ORDER] [RANK_TABLE_PATH] [DEVICE_RANGE] [RANK_SIZE] For Multiple Devices In Single Machine" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) mkdir -p ./output/log/rank_$RANK_ID echo "start training for rank $RANK_ID, device $DEVICE_ID" $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` 全参微调分布式拉起命令(8卡)： ```bash bash run_distribute_single_node.sh "python task.py --task text_generation --model_type glm_6b --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 1 --model_parallel 8" /path/to/hccl_8p_xxx.json '[0,8]' 8 ``` 参数含义: - `"python task.py --task text_generation --model_type glm_6b --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 1 --model_parallel 8"`: 需执行的命令，此处完整输入task.py的启动命令 python task.py 各项参数含义： - `task`: 需运行的训练任务，此处为 `text_generation` 文本生成任务 - `model_type`: 模型类型，此处选择 `glm_6b` 模型 - `checkpoint_path`: 权重路径，此处替换为实际需加载的权重路径 - `train_dataset`: 训练数据集路径，替换为实际路径 - `run_mode`: 启动模式，train——训练，finetune——微调，eval——评估，predict——推理，此处选择 `finetune` - `use_parallel`: 是否使用多卡并行训练，此处为 `True` - `data_parallel`: 数据并行数，此处为1表示不开启 - `model_parallel`: 模型并行数，此处为8表示8卡并行 bash 脚本其余参数： - `/path/to/hccl_4p_xxx.json`: rank table file路径，替换为之前准备的rank table file的实际路径 - `'[0,8]'`: 占用的卡范围，0包含，8不包含，表示使用 `0~7` 8张卡并行训练 - `8`: rank size，一共使用了多少张卡，此处为8 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint ### LoRA低参微调 全参微调能够在微调数据集上取得良好效果，但存在遗忘预训练知识的现象 因此推荐使用低参微调算法，冻结原模型权重，仅在小规模参数量上进行训练，在微调数据集上取得良好效果的同时，缓解模型遗忘现象 #### run_mindformers脚本启动LoRA低参微调 使用LoRA算法进行低参微调时，使用 `configs/glm/run_glm_6b_lora.yaml` 配置文件，该配置文件包含了lora低参微调算法所需的配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm/run_glm_6b_lora.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm/run_glm_6b_lora.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 #### 启动LoRA低参微调脚本(1卡)： 执行命令： ```shell cd scripts # Usage Help: bash run_standalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] bash run_standalone.sh ../configs/glm/run_glm_6b_lora.yaml 0 finetune ``` 训练的log日志路径：mindformers/scripts/mf_standalone/ checkpoint存储路径：mindformers/scripts/mf_standalone/output/checkpoint #### 启动LoRA低参微调脚本(4卡)： > 注：如果需要进行多卡训练，则需要对`glm/run_glm_6b_lora.yaml`配置文件对应参数进行修改，以4卡为例，需要重新生成4卡的HCCL文件： ```shell data_parallel: 4 ``` ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_4_0123_xxx.json ../configs/glm/run_glm_6b_lora.yaml '[0,4]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明： 对比全参微调启动方式，仅将 `CONFIG_PATH` 项修改为configs文件夹下面的 `glm/run_glm_6b_lora.yaml` 配置文件，表示使用该接口进行 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint #### Trainer高阶接口启动LoRA低参微调 可复用全参微调部分所提供的 `task.py` 和 `run_distribute_single_node.sh` 脚本 4卡分布式启动命令： ```bash bash run_distribute_single_node.sh "python task.py --task text_generation --model_type glm_6b_lora --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel True --data_parallel 4 --model_parallel 1" /path/to/hccl_4p_xxx.json '[0,4]' 4 ``` 参数说明：对比全参微调启动，仅改动以下几点： - `model_type`: 指定模型类型为 `glm_6b_lora`，表示使用低参微调算法 - `data_parallel`: 4卡启动，数据并行改为4 - `/path/to/hccl_4p_xxx.json`: 使用4卡的rank_table_file - `'[0,4]' 4`: 使用0~3共4卡 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint 1卡启动命令： ```shell python task.py --task text_generation --model_type glm_6b_lora --checkpoint_path ./glm_6b.ckpt --train_dataset ./train --run_mode finetune --use_parallel False --data_parallel 1 --model_parallel 1 ``` ### 多机多卡微调训练 多机多卡启动 首先在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； 将不同机器上生成的RANK_TABLE_FILE文件中的server_list合并，server_count设为机器数，rank_id顺序增加，并保证不同机器上的RANK_TABLE_FILE相同； 在多机上同时拉起任务，拉起方式为 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_MODE RANK_SIZE #### 参数说明 - RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 - CONFIG_PATH: 为configs文件夹下面的gpt2/run_gpt2*.yaml配置文件 - DEVICE_RANGE: 为单机分布式卡的范围, 如[0,8]为8卡分布式，不包含8本身 - RUN_MODE: 为任务运行状态，支持关键字 train 预训练、predict（文本生成预测） - RANK_SIZE: 总运行卡数 #### 4机32卡参考RANK_TABLE_FILE样例 ```text { "version": "1.0", "server_count": "4", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.141", "device": [ {"device_id": "0","device_ip": "192.1.27.8","rank_id": "8"}, {"device_id": "1","device_ip": "192.2.27.8","rank_id": "9"}, {"device_id": "2","device_ip": "192.3.27.8","rank_id": "10"}, {"device_id": "3","device_ip": "192.4.27.8","rank_id": "11"}, {"device_id": "4","device_ip": "192.1.27.9","rank_id": "12"}, {"device_id": "5","device_ip": "192.2.27.9","rank_id": "13"}, {"device_id": "6","device_ip": "192.3.27.9","rank_id": "14"}, {"device_id": "7","device_ip": "192.4.27.9","rank_id": "15"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.142", "device": [ {"device_id": "0","device_ip": "192.1.27.10","rank_id": "16"}, {"device_id": "1","device_ip": "192.2.27.10","rank_id": "17"}, {"device_id": "2","device_ip": "192.3.27.10","rank_id": "18"}, {"device_id": "3","device_ip": "192.4.27.10","rank_id": "19"}, {"device_id": "4","device_ip": "192.1.27.11","rank_id": "20"}, {"device_id": "5","device_ip": "192.2.27.11","rank_id": "21"}, {"device_id": "6","device_ip": "192.3.27.11","rank_id": "22"}, {"device_id": "7","device_ip": "192.4.27.11","rank_id": "23"}], "host_nic_ip": "reserve" }, { "server_id": "10.155.111.143", "device": [ {"device_id": "0","device_ip": "192.1.27.12","rank_id": "24"}, {"device_id": "1","device_ip": "192.2.27.12","rank_id": "25"}, {"device_id": "2","device_ip": "192.3.27.12","rank_id": "26"}, {"device_id": "3","device_ip": "192.4.27.12","rank_id": "27"}, {"device_id": "4","device_ip": "192.1.27.13","rank_id": "28"}, {"device_id": "5","device_ip": "192.2.27.13","rank_id": "29"}, {"device_id": "6","device_ip": "192.3.27.13","rank_id": "30"}, {"device_id": "7","device_ip": "192.4.27.13","rank_id": "31"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` #### 任务拉起命令示例 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [0,8] train 32 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [8,16] train 32 # 第三台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [16,24] train 32 # 第四台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/glm/run_glm_6b_lora.yaml [24,32] train 32 ``` ### 微调后推理 #### 推理样例脚本 下面提供一个模型推理样例脚本 `infer.py` ```python import time import mindspore as ms import numpy as np import argparse from mindformers.models.glm import GLMConfig, GLMChatModel, GLMChatModelWithLora from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.models.glm.glm_processor import process_response from mindformers.pet.pet_config import LoraConfig parser = argparse.ArgumentParser() parser.add_argument('--seq_length', default=1024, type=int, help='Which device to run service.') parser.add_argument('--device_id', default=0, type=int, help='Which device to run service.') parser.add_argument('--checkpoint_path', type=str, default='/path/chatglm6b.ckpt', help='Checkpoint file to load on.') parser.add_argument('--vocab_path', type=str, default='/path/ice_text.model', help='Vocab file to load on.') parser.add_argument('--is_lora', type=str, default='false',help='Whether is lora model.') args = parser.parse_args() if args.is_lora.lower() == "true": is_lora = True else: is_lora = False config = GLMConfig( position_encoding_2d=True, use_past=True, is_sample_acceleration=True, ) pet_config = LoraConfig( lora_rank=8, lora_alpha=32, lora_dropout=0.1 ) def chat_glm(): ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=args.device_id) if is_lora: config.pet_config = pet_config model = GLMChatModelWithLora(config) else: model = GLMChatModel(config) ms.load_checkpoint(args.checkpoint_path, model) tokenizer = ChatGLMTokenizer(args.vocab_path) inputs = ["你好", "请介绍一下华为", "用Python写一个快排", "类型#上衣*材质#牛仔布*颜色#白色*风格#简约*图案#刺绣*衣样式#外套*衣款式#破洞"] for query in inputs: input_ids = tokenizer(query)['input_ids'] start_time = time.time() outputs = model.generate(input_ids, max_length=config.max_decode_length, do_sample=False) end_time = time.time() print(f'generate speed: {outputs[0].shape[0]/(end_time-start_time):.2f} tokens/s') response = tokenizer.decode(outputs) response = process_response(response[0]) print(response) if __name__ == "__main__": chat_glm() ``` #### 运行命令 ```shell python infer.py --seq_length 1024 --device_id 0 --checkpoint_path /path/chatglm6b.ckpt --vocab_path /path/ice_text.model --is_lora True ``` 参数说明： - `seq_length`: 用于指定推理输入长度 - `device_id`: 指定推理在那张设备运行 - `checkpoint_path`: 指定训练出来的模型文件路径用于推理 - `vocab_path`: 模型词表 - `is_lora`: 用于区分是否是lora模型，设置为true表示为lora微调训练模型 ## 评估 ### 模型权重文件合一 微调所得到的权重文件为根据模型切分策略切分后的权重，我们需要手动将切分权重合一，以用于评估和推理 1. 获取模型切分策略文件： 在执行全参微调脚本时，模型完成编译后，将会在运行路径下，生成名为 `ckpt_strategy.ckpt` 的切分策略文件，该文件将用于第二步模型合成 2. MindSpore提供了根据切分策略转换模型权重切分的接口，[mindspore.transform_checkpoints](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.transform_checkpoints.html)，执行以下python脚本，将8份模型文件合成一份 ```python from mindspore import transform_checkpoints transform_checkpoints( src_checkpoints_dir="./output/checkpoint/", # 原切分权重文件夹 dst_checkpoints_dir="./target_checkpoint/", # 目标路径 ckpt_prefix="glm-6b", # .ckpt文件前缀名 src_strategy_file="ckpt_stragery.ckpt", # 步骤1中的切分策略文件路径 dst_strategy_file=None # None表示不切分，权重合一 ) ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ### 使用全参微调权重 #### run_mindformers启动eval 使用全参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm/run_glm_6b_infer.yaml` glm模型推理配置，此配置下评估速度更快 ```bash python run_mindformer.py --config configs/glm/run_glm_6b_infer.yaml --run_mode eval --load_checkpoint /path/to/glm_6b.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` > 注：使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 各项参数： - `config`: 指定用于评估的配置文件名称，此处为`configs/glm/run_glm_6b_infer.yaml` - `run_mode`: 指定执行模式，此为`eval`，表示为评估模式 - `load_checkpoint`: 指定要加载的checkpoint路径，此处为`/path/to/glm_6b.ckpt`，替换为需加载的权重的真实路径 - `eval_dataset_dir`: 评估数据集的路径 - `device_id`: 指定要使用的设备编号（从0开始） 评估完成后会打印评估指标 `bleu-4`、`rouge-1`、`rouge-2`、`rouge-l` > 注：由于默认评估指标的获取方式为生成完整文本后与预期文本做比较，评估速度将受限于模型大小与文本生成速度，评估流程可能较为缓慢 #### Trainer高阶接口启动eval 仍然可复用 `task.py` 脚本，启动命令： ```bash python task.py --task text_generation --model_type glm_6b_chat --checkpoint_path /path/to/glm_6b.ckpt --eval_dataset /path/to/data/AdvertiseGen/ --run_mode eval --batch_size 1 ``` > 1. 当前评估时，batch_size需为1，否则评估速度下降严重 > 2. 使用离线生成数据方式时，将 `eval_dataset` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 ### 使用LoRA低参微调权重 #### run_mindformers启动lora eval 使用LoRA低参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm/run_glm_6b_lora_infer.yaml` glm_lora模型推理配置，此配置可用于lora模型，并且评估速度更快 ```bash python run_mindformer.py --config configs/glm/run_glm_6b_lora_infer.yaml --run_mode eval --load_checkpoint /path/to/glm_6b_lora.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` 各项参数同上，路径需替换为实际路径 > 使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 #### Trainer高阶接口启动lora eval 仍然可复用 `task.py` 脚本，启动命令： ```bash python task.py --task text_generation --model_type glm_6b_lora_chat --checkpoint_path /path/to/glm_6b_lora.ckpt --eval_dataset /path/to/data/AdvertiseGen/ --run_mode eval --batch_size 1 ``` > 1. 当前评估时，batch_size需为1，否则评估速度下降严重 > 2. 使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 ## 模型权重转化 本仓库中的`glm`来自于HuggingFace的[chatglm-6b](https://huggingface.co/THUDM/chatglm-6b)，基于下述的步骤获取： 1. 克隆chatglm-6b代码仓，下载分布式的模型文件。 ```shell git lfs install git clone https://huggingface.co/THUDM/chatglm-6b ``` 2. 执行 python 脚本，合并模型权重。 ```python from transformers import AutoModel import torch as pt pt_ckpt_path="Your chatglm-6b path" model = AutoModel.from_pretrained(pt_ckpt_path, trust_remote_code=True).half() pt_pth_path = "pt_glm_6b.pth" pt.save(model.state_dict(), pt_pth_path) ``` 3. 执行转换脚本，得到转换后的输出文件`ms_glm_6b.ckpt`。 ```shell python mindformers/models/glm/convert_weight.py --pt_ckpt_path "replace your ptroch pth path" --ms_ckpt_path ./ms_glm_6b.ckpt ``` pengcheng.mind-mindformer/docs/model_cards/glm2.md000066400000000000000000000435741476650141400225160ustar00rootroot00000000000000# ChatGLM2-6B ## 模型描述 ChatGLM**2**-6B 是开源中英双语对话模型 [ChatGLM2-6B](https://github.com/THUDM/ChatGLM2-6B) 的第二代版本，在保留了初代模型对话流畅、部署门槛较低等众多优秀特性的基础之上，ChatGLM**2**-6B引入了新特征：**更强大的性能**、**更长的上下文**、**更高效的推理**、**更开放的协议**。 ## 仓库介绍 `chatGLM2-6B` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/glm2` ```bash glm2 ├── __init__.py ├── glm2.py # 模型实现 ├── glm2_config.py # 模型配置项 ├── glm2_modules.py # 模组实现 ├── glm2_tokenizer.py # tokenizer └── glm2_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/glm2` ```bash glm2 ├── run_glm2_6b_fintune.yaml # 全量微调启动配置 └── run_glm2_6b_lora.yaml # lora低参微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0 推理可在单机单卡上完成部署 全量微调训练需要最少单机8卡，Lora微调训练最少需要1卡 ## 基线 测试环境同上述环境要求 ### 性能 | | data parallel | model parallel | pipeline parallel | batch size | sink size | sequence length | accumulate | per step time (ms) | tokens/s/p | 优化器并行 | 重计算 | Memory (GB) | | -------- | ------------- | -------------- | ----------------- | ---------- | --------- | --------------- | ---------- | ------------------ | ----------- | ---------- | ------ | ----------- | | 全量微调 | 8 | 1 | 1 | 8 | 4 | 193 | 1 | 1894 | 815.2059134 | True | True | 25.2 | | LoRA微调 | 4 | 1 | 1 | 8 | 4 | 193 | 1 | 476 | 3243.697479 | False | False | 22.38 | ### 评估指标 | | rouge-1 | rouge-2 | rouge-l | bleu-4 | | -------- | ------------------ | ----------------- |--------------------| ----------------- | | 全量微调 | 30.784298224299064 | 7.073415046728972 | 24.773958598130843 | 7.466147757009345 | | LoRA微调 | 31.05639289719626 | 7.1753861682243 | 24.229674859813084 | 7.229435140186916 | ## ChatGLM2-6B推理 > 需开发者提前pip安装。具体接口说明请参[API接口](https://gitee.com/mindspore/transformer/wikis/API/) ### AutoClass推理 可以使用AutoClass接口，通过模型名称获取相应的模型/tokenizer实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/glm2` 首次运行pipeline推理时需要进行模型编译，需等待一段时间 ```python from mindformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("glm2_6b") model = AutoModel.from_pretrained("glm2_6b") query = "你好" prompted_inputs = tokenizer.build_prompt(query) input_tokens = tokenizer([prompted_inputs]) outputs = model.generate(input_tokens["input_ids"], max_length=100) response = tokenizer.decode(outputs)[0] print(response) ``` ### pipeline推理 也可以不实例化构造模型，直接通过指定任务模型与模型名的方式进行pipeline的构造 ```python >>> from mindformers import pipeline, TextGenerationPipeline >>> task_pipeline = pipeline(task='text_generation', model='glm2_6b', max_length=2048) >>> task_pipeline('你好') [{'text_generation_text': ['你好，我是 ChatGLM2-6B， 一个人工智能助手。我背后使用的模型是 GLM2-6B， 是一种大型语言模型， 具有超过 2000 亿参数，支持多种任务。']}] >>> pipeline = TextGenerationPipeline(model='glm2_6b', max_length=2048) >>> pipeline("你好") [{'text_generation_text': ['你好，我是 ChatGLM2-6B， 一个人工智能助手。我背后使用的模型是 GLM2-6B， 是一种大型语言模型， 具有超过 2000 亿参数，支持多种任务。']}] ``` ## 微调 下面以 [ADGEN](https://aclanthology.org/D19-1321.pdf) (广告生成) 数据集为例介绍代码的使用方法 ### 数据处理 ADGEN 数据集任务为根据输入（content）生成一段广告词（summary）。 ```json { "content": "类型#上衣*版型#宽松*版型#显瘦*图案#线条*衣样式#衬衫*衣袖型#泡泡袖*衣款式#抽绳", "summary": "这件衬衫的款式非常的宽松，利落的线条可以很好的隐藏身材上的小缺点，穿在身上有着很好的显瘦效果。领口装饰了一个可爱的抽绳，漂亮的绳结展现出了十足的个性，配合时尚的泡泡袖型，尽显女性甜美可爱的气息。" } ``` 从 [Google Drive](https://drive.google.com/file/d/13_vf0xRTQsyneRKdD1bZIr93vBGOczrk/view?usp=sharing) 或者 [Tsinghua Cloud](https://cloud.tsinghua.edu.cn/f/b3f119a008264b1cabd1/?dl=1) 下载处理好的 ADGEN 数据集，目录结构为 ```shell AdvertiseGen ├── train.json └── dev.json ``` 将任务配置文件 `configs/glm2/run_glm2_6b_*.yaml` 中的 `==== dataset config ====` 部分替换成： ```yaml train_dataset: &train_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/train.json" shuffle: True phase: "train" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" input_columns: ["input_ids", "labels"] max_source_length: 64 max_target_length: 128 ignore_pad_token_for_loss: True num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 train_dataset_task: type: KeyWordGenDataset dataset_config: *train_dataset eval_dataset: &eval_dataset data_loader: type: ADGenDataLoader dataset_dir: "/path/to/AdvertiseGen/dev.json" shuffle: False phase: "eval" version: 2 origin_columns: ["content", "summary"] tokenizer: type: ChatGLM2Tokenizer vocab_file: "/path/to/tokenizer.model" max_source_length: 256 max_target_length: 256 ignore_pad_token_for_loss: True input_columns: ["input_ids", "labels"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 1 repeat: 1 numa_enable: False prefetch_size: 1 seed: 0 eval_dataset_task: type: KeyWordGenDataset dataset_config: *eval_dataset ``` ### 生成HCCL文件 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` > 注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成 RANK_TABLE_FILE 单机8卡参考样例: ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 全参微调 #### run_mindformers脚本启动全参微调 全参微调使用 `configs/glm2/run_glm2_6b.yaml` 配置文件，配置文件中定义了微调所需的各配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm2/run_glm2_6b.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm2/run_glm2_6b.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 启动全参微调脚本： ```shell cd scripts # Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] bash run_distribute.sh /path/to/hccl_8p_01234567_127.0.1.1.json ../configs/glm2/run_glm2_6b.yaml '[0,8]' finetune # 将此处rank_table_file替换为实际路径 ``` 参数说明 ```text RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的glm2/run_glm2_6b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如 '[0,8]' 为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` > 注：由于GLM2_6B的模型较大，无法在单卡上运行，此处仅提供分布式启动脚本 训练的log日志路径：mindformers/output/log checkpoint存储路径：mindformers/output/checkpoint ### LoRA低参微调 全参微调能够在微调数据集上取得良好效果，但存在遗忘预训练知识的现象 因此推荐使用低参微调算法，冻结原模型权重，仅在小规模参数量上进行训练，在微调数据集上取得良好效果的同时，缓解模型遗忘现象 #### run_mindformers脚本启动LoRA低参微调 使用LoRA算法进行低参微调时，使用 `configs/glm2/run_glm2_6b_lora.yaml` 配置文件，该配置文件包含了lora低参微调算法所需的配置项 修改数据集/模型权重配置路径： - 数据集：修改 `mindformers/configs/glm2/run_glm2_6b_lora.yaml` 脚本中`train_dataset` 的 `dataset_dir` 为前文生成的数据集路径。 - 加载预训练模型权重：修改 `mindformers/configs/glm2/run_glm2_6b_lora.yaml` 脚本中的 `load_checkpoint` 为预训练模型权重路径。 #### 启动LoRA低参微调脚本(1卡)： 执行命令： ```shell cd scripts # Usage Help: bash run_stanalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] bash run_standalone.sh ../configs/glm2/run_glm2_6b_lora.yaml 0 finetune ``` 训练的log日志路径：mindformers/scripts/mf_standalone/ checkpoint存储路径：mindformers/scripts/mf_standalone/output/checkpoint #### Trainer高阶接口启动LoRA低参微调 示例脚本如下，需要指定训练数据集路径和微调权重。 ```python from mindformers import Trainer trainer = Trainer(task="text_generation", model="glm2_6b", pet_method="lora", train_dataset="/path/to/AdvertiseGen/train.json") trainer.finetune(finetune_checkpoint="glm2_6b") ``` ### 微调后推理 #### 推理样例脚本 下面提供一个模型推理样例脚本 `infer.py` ```python from mindformers import AutoConfig, AutoModel, AutoTokenizer import mindspore as ms ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=0) config = AutoConfig.from_pretrained("glm2_6b") config.checkpoint_name_or_path = "/path/to/glm2_6b_finetune.ckpt" model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained("glm2_6b") inputs = tokenizer(tokenizer.build_prompt("你好"))["input_ids"] print(inputs) print(tokenizer.decode(inputs)) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("请介绍一下华为"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("晚上睡不着应该怎么办"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) inputs = tokenizer(tokenizer.build_prompt("类型#上衣*材质#牛仔布*颜色#白色*风格#简约*图案#刺绣*衣样式#外套*衣款式#破洞"))["input_ids"] print(inputs) outputs = model.generate(inputs, max_length=128) print(tokenizer.decode(outputs)) ``` ## 评估 ### 模型权重文件合一 微调所得到的权重文件为根据模型切分策略切分后的权重，我们需要手动将切分权重合一，以用于评估和推理 1. 获取模型切分策略文件： 在执行全参微调脚本时，模型完成编译后，将会在运行路径下，生成名为 `ckpt_strategy.ckpt` 的切分策略文件，该文件将用于第二步模型合成 2. MindSpore提供了根据切分策略转换模型权重切分的接口，[mindspore.transform_checkpoints](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.transform_checkpoints.html)，执行以下python脚本，将8份模型文件合成一份 ```python from mindspore import transform_checkpoints transform_checkpoints( src_checkpoints_dir="./output/checkpoint/", # 原切分权重文件夹 dst_checkpoints_dir="./target_checkpoint/", # 目标路径 ckpt_prefix="glm2-6b", # .ckpt文件前缀名 src_strategy_file="ckpt_stragery.ckpt", # 步骤1中的切分策略文件路径 dst_strategy_file=None # None表示不切分，权重合一 ) ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ### 使用全参微调权重 #### run_mindformers启动eval 使用全参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm2/run_glm2_6b.yaml` glm2模型推理配置，修改其中`model`字段下`model_config`中`use_past: True`开启增量推理使评估速度更快 ```bash python run_mindformer.py --config configs/glm2/run_glm2_6b.yaml --run_mode eval --load_checkpoint /path/to/glm2_6b_finetune.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` > 注：使用离线生成数据方式时，将 `eval_dataset_dir` 一项指向`.mindrecord`文件，如 `/path/to/data/AdvertiseGen/adgen_dev.mindrecord`。 各项参数： - `config`: 指定用于评估的配置文件名称，此处为`configs/glm2/run_glm2_6b.yaml` - `run_mode`: 指定执行模式，此为`eval`，表示为评估模式 - `load_checkpoint`: 指定要加载的checkpoint路径，此处为`/path/to/glm2_6b_finetune.ckpt`，替换为需加载的权重的真实路径 - `eval_dataset_dir`: 评估数据集的路径 - `device_id`: 指定要使用的设备编号（从0开始） 评估完成后会打印评估指标 `bleu-4`、`rouge-1`、`rouge-2`、`rouge-l` > 注：由于默认评估指标的获取方式为生成完整文本后与预期文本做比较，评估速度将受限于模型大小与文本生成速度，评估流程可能较为缓慢 #### Trainer高阶接口启动eval 与上文类似： ```bash from mindformers import Trainer, ChatGLM2Config, ChatGLM2ForConditionalGeneration # 开启增量推理使评估速度更快 config = ChatGLM2Config(use_past=True) model = ChatGLM2ForConditionalGeneration(config) trainer = Trainer(task="text_generation", model=model, eval_dataset="/path/to/AdvertiseGen/dev.json") trainer.evaluate(eval_checkpoint="/path/to/glm2_6b_finetune.ckpt") ``` ### 使用LoRA低参微调权重 #### run_mindformers启动lora eval 使用LoRA低参微调权重时，启动如下shell脚本，执行单卡评估 配置文件选择 `configs/glm2/run_glm2_6b_lora.yaml` glm2_lora模型推理配置，此配置可用于lora模型，修改其中`model`字段下`model_config`中`use_past: True`开启增量推理使评估速度更快 ```bash python run_mindformer.py --config configs/glm2/run_glm2_6b_lora.yaml --run_mode eval --load_checkpoint /path/to/glm2_6b_lora.ckpt --eval_dataset_dir /path/to/data/AdvertiseGen/ --device_id 0 ``` 各项参数同上，路径需替换为实际路径 #### Trainer高阶接口启动lora eval 与上文类似： ```bash from mindformers import Trainer, ChatGLM2Config, ChatGLM2WithLora from mindformers.pet.pet_config import LoraConfig # 开启增量推理使评估速度更快 config = ChatGLM2Config(use_past=True) config.pet_config = LoraConfig() model = ChatGLM2WithLora(config) trainer = Trainer(task="text_generation", model=model, eval_dataset="/path/to/AdvertiseGen/dev.json") trainer.evaluate(eval_checkpoint="/path/to/glm2_6b_lora.ckpt") ``` ## 模型权重转化 本仓库中的`glm2`来自于HuggingFace的 [ChatGLM2-6B](https://huggingface.co/THUDM/chatglm2-6b)，基于下述的步骤获取： 1. 克隆chatglm2-6b代码仓，下载分布式的模型文件。 ```shell git lfs install git clone https://huggingface.co/THUDM/chatglm2-6b ``` 2. 执行 python 脚本，合并模型权重。 ```python from transformers import AutoTokenizer, AutoModel import torch tokenizer = AutoTokenizer.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True) model = AutoModel.from_pretrained("THUDM/chatglm2-6b", trust_remote_code=True) with open("pt_model_arch.txt", "w") as fp: print(model, file=fp, flush=True) with open("pt_ckpt.txt", "w") as fp: for name, param in model.named_parameters(): fp.write(f"{name} {param.shape} {param.dtype}\n") torch.save(model.state_dict(), "glm2_6b.pth") ``` 3. 执行转换脚本，得到转换后的输出文件`glm2_6b.ckpt`。 ```python import mindspore as ms import torch as pt from tqdm import tqdm pt_ckpt_path = "glm2_6b.pth" pt_param = pt.load(pt_ckpt_path) type_map = {"torch.float16": "ms.float16", "torch.float32": "ms.float32"} ms_param = [] with open("check_pt_ckpt.txt", "w") as fp: for k, v in tqdm(pt_param.items()): if "word_embeddings.weight" in k: k = k.replace("word_embeddings.weight", "embedding_table") fp.write(f"{k} {v.shape} {v.dtype}\n") ms_param.append({"name": k, "data": ms.Tensor(v.numpy())}) ms.save_checkpoint(ms_param, "glm2_6b.ckpt") ``` pengcheng.mind-mindformer/docs/model_cards/gpt2.md000066400000000000000000001062701476650141400225220ustar00rootroot00000000000000# GPT2 ## 模型描述 GPT-2由OpenAI于2019年发布。GPT-2模型是继承于GPT模型，GPT-2是一个非常庞大的语言模型，它主要是用于预测下一个单词。按照参数量的大小，原生GPT-2模型可分为small（124M）、medium（355M）、large（774M）、xlarge（1.5B），但在此仓中，基于GPT2扩展了13B，52B等规格。 [论文](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)A Radford，et al., Language Models are Unsupervised Multitask Learners, 2019 ## 模型性能 ```txt Mindspore: 2.0.0rc1 Ascend: 910A ``` | config | task | Datasets | metric | score | train performance | predict performance | | :---------------------------------: | :-----------------: | :--------------------------------: | :------: | :---------------------------------: | :---------------: | :--------------------------------------: | | [gpt2](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2.yaml) | text_generation | wikitext2 | ppl | 22.11 | 1265 tokens/s | 4.66/11.37 tokens/s(use past True/False) | | [gpt2_lora](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_lora.yaml) | text_generation | wikitext2 | - | - | 1428 tokens/s | - | | [gpt2_txtcls](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_txtcls.yaml) | text_classification | SST-2
IMDB
AGNews
COLA | accuracy | 0.908
0.934
0.941
0.693 | - | - | ## 仓库介绍 1、模型具体实现：`mindformers/models/gpt2` `gpt2`基于`mindformers`实现，主要涉及的文件有： ```bash model ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── gpt2.py # 模型实现 ├── gpt2_config.py # 模型配置项 ├── gpt2_processor.py # gpt2预处理 ├── gpt2_tokenizer.py # tokenizer └── gpt2_modules.py # transformer层实现 ``` 2、模型配置：`configs/gpt2` ```bash gpt2 ├── run_gpt2.yaml # gpt2 small模型启动配置 ├── run_gpt2_13b.yaml # gpt 13b模型启动配置 ├── run_gpt2_52b.yaml # gpt 52b模型启动配置 ├── run_gpt2_lora.yaml # gpt2 small lora低参微调启动配置 ├── run_gpt2_txtcls.yaml # gpt2 small文本分类模型启动配置 ├── run_gpt2_xl.yaml # gpt2 xlarge模型启动配置 └── run_model_xl_lora.yaml # gpt2 xlarge lora低参微调启动配置 ``` 3、预处理脚本和任务启动脚本：`mindformers/tools/dataset_preprocess/gpt2` ```bash model ├── txtcls_dataset_to_mindrecord.py # 文本分类数据集预处理 └── wikitext2_data_process.py # wikitext2数据集预处理 ``` ## 前期准备 ### [mindformers安装](https://gitee.com/mindspore/mindformers/tree/dev#%E4%BA%8Cmindformers%E5%AE%89%E8%A3%85) ### 生成RANK_TABLE_FILE(多卡运行必备环节) 运行`mindformers/tools/hccl_tools.py`生成RANK_TABLE_FILE的json文件 ```bash # 运行如下命令，生成当前机器的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" ``` **注：若使用ModelArts的notebook环境，可从 `/user/config/jobstart_hccl.json` 路径下直接获取rank table，无需手动生成** RANK_TABLE_FILE 单机8卡参考样例: **其中`server_id`是机器ip地址** ```json { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 多机RANK_TABLE_FILE合并(多机多卡必备环节) - step 1. 首先根据上章节内容，在每个机器上生成各自的`RANK_TABLE_FILE`文件，然后将不同机器上生成的`RANK_TABLE_FILE`文件全部拷贝到同一台机器上。 - step 2. 运行mindformers/tools/merge_hccl.py将不同机器上生成的`RANK_TABLE_FILE`文件合并 ```bash # 运行如下命令，合并每个机器上的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 RANK_TABLE_FILE 双机16卡参考样例: 以下`server_id`为机器ip地址，不同机器需设置不同的值 ```json { "version": "1.0", "server_count": "2", "server_list": [ { "server_id": "xx.xx.xx.xx", "device": [ { "device_id": "0", "device_ip": "192.168.0.0", "rank_id": "0" }, { "device_id": "1", "device_ip": "192.168.1.0", "rank_id": "1" }, { "device_id": "2", "device_ip": "192.168.2.0", "rank_id": "2" }, { "device_id": "3", "device_ip": "192.168.3.0", "rank_id": "3" }, { "device_id": "4", "device_ip": "192.168.0.1", "rank_id": "4" }, { "device_id": "5", "device_ip": "192.168.1.1", "rank_id": "5" }, { "device_id": "6", "device_ip": "192.168.2.1", "rank_id": "6" }, { "device_id": "7", "device_ip": "192.168.3.1", "rank_id": "7" } ], "host_nic_ip": "reserve" }, { "server_id": "xx.xx.xx.xx", "device": [ { "device_id": "0", "device_ip": "192.168.0.1", "rank_id": "8" }, { "device_id": "1", "device_ip": "192.168.1.1", "rank_id": "9" }, { "device_id": "2", "device_ip": "192.168.2.1", "rank_id": "10" }, { "device_id": "3", "device_ip": "192.168.3.1", "rank_id": "11" }, { "device_id": "4", "device_ip": "192.168.0.2", "rank_id": "12" }, { "device_id": "5", "device_ip": "192.168.1.2", "rank_id": "13" }, { "device_id": "6", "device_ip": "192.168.2.2", "rank_id": "14" }, { "device_id": "7", "device_ip": "192.168.3.2", "rank_id": "15" } ], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ### 模型权重下载与转换 作为参考，这里描述CheckPoint在HuggingFace或者官方开源github仓库和MindSpore间的转换，在不同分布式策略间的转换。 Huggingface权重： [gpt2 small](https://huggingface.co/gpt2/resolve/main/pytorch_model.bin) [gpt2 xlarge](https://huggingface.co/gpt2-xl/resolve/main/pytorch_model.bin) [gpt2 13b](https://huggingface.co/cerebras/Cerebras-GPT-13B/tree/main) 其中，13b的权重需要将上述链接下的`pytorch_model-00001-of-00002.bin`、`pytorch_model-00002-of-00002.bin`、`pytorch_model.bin.index.json `、`config.json`下载并存到一个文件夹`torch_weights`中，然后使用如下命令将Huggingface的权重进行合并 ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("torch_weights") model.save_pretrained("gpt_13b.bin", max_shard_size="60GB") ``` 权重转换： ```bash # gpt2 small python mindformers/models/gpt2/convert_weight.py --layers 12 --torch_path gpt2_small.bin --mindspore_path ./gpt2_small.ckpt # gpt2 xlarge python mindformers/models/gpt2/convert_weight.py --layers 48 --torch_path gpt2_small.bin --mindspore_path ./gpt2_xlarge.ckpt # gpt2 13b python mindformers/models/gpt2/convert_weight.py --layers 40 --torch_path gpt_13b.bin --mindspore_path ./gpt_13b.ckpt ``` 另，`mindformers`已经提供转换好的权重（其中lora权重为mindformers训练得到，非Huggingface官方权重转化得到）： [gpt2 small](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.ckpt) [gpt2 small lora](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_lora.ckpt) [gpt2 xlarge](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl.ckpt) [gpt2 xlarge lora](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckpt) [gpt2 13b](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_13b.ckpt) ### [模型权重切分与合并](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md) 从hugging face或官方github仓库转换而来的权重通常是单卡权重，基于该权重进行多卡微调，评测，推理，涉及ckpt从单机策略到分布式策略的切换。 通常训练采用分布式训练，基于该权重进行评测，推理多采用单卡，涉及ckpt从分布式策略到单机策略的切换。 以上涉及到ckpt的单卡，多卡转换，详细教程请参考特性文档模型[权重切分与合并](https://gitee.com/mindspore/mindformers/blob/dev/docs/feature_cards/Transform_Ckpt.md) ## 基于API的快速使用 ### 基于AutoClass的使用 可以使用AutoClass接口，通过模型名称获取相应的model/preprocess/tokenizer等实例，并自动下载并加载权重 `from_pretrained()` 接口会自动从云上下载预训练的模型，存储路径：`mindformers/checkpoint_download/gpt2` ```python # 以gpt2 small为例 import mindspore from mindformers import AutoModel, AutoTokenizer # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) tokenizer = AutoTokenizer.from_pretrained("gpt2") model = AutoModel.from_pretrained("gpt2") model.set_train(False) inputs = tokenizer("An increasing sequence: one,") outputs = model.generate(inputs["input_ids"], max_length=20, do_sample=20) response = tokenizer.decode(outputs)[0] print(response) # An increasing sequence: one, two, three, four, five, six, seven, eight, ``` ### 基于Trainer的训练，微调，评测，推理 ```python # 以gpt2 small为例 import mindspore from mindformers.trainer import Trainer # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) # 初始化预训练任务 trainer = Trainer(task='text_generation', model='gpt2', train_dataset='path/to/train_dataset', eval_dataset='path/to/eval_dataset') # 开启预训练 trainer.train() # 开启全量微调 trainer.finetune() # 开启评测 trainer.evaluate() # 开启推理 predict_result = trainer.predict(input_data="An increasing sequence: one,", do_sample=False, max_length=20) print(predict_result) # output result is: [{'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight,']}] # Lora微调 trainer = Trainer(task="text_generation", model="gpt2", pet_method="lora", train_dataset="path/to/train_dataset") trainer.finetune(finetune_checkpoint="gpt2") ``` ### 基于Pipeline的推理 ```python # 以gpt2 small为例 import mindspore from mindformers.pipeline import pipeline # 指定图模式，指定使用训练卡id mindspore.set_context(mode=0, device_id=0) pipeline_task = pipeline(task="text_generation", model="gpt2") pipeline_result = pipeline_task("An increasing sequence: one,", do_sample=False, max_length=20) print(pipeline_result) # [{'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight,']}] ``` **注：**要提高推理速度，可在对应模型配置文件中进行如下配置，设置增量推理`use_past`为True。 ## 预训练 ### 数据集准备-预训练 以Wikitext2数据集为例 1、数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) 2、词表下载：[vocab.json](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/vocab.json)，[merges.txt](https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/merges.txt) 3、参考[wikitext-2处理脚本](https://gitee.com/mindspore/mindformers/blob/931cf93045473d5827ee26638d83fabe94058d28/mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py#)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ```bash # 训练 python mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py \ --input_file ./wikitext-2/wiki.train.tokens \ --output_file ./wikitext-2.train.mindrecord \ --max_length 1025 ``` ### 脚本启动 #### 单卡训练 - python启动 ```bash # dataset_dir可指定文件目录或文件路径，指定文件路径时，读取单文件， # 指定目录时，读取目录下所有以字符串mindrecord结尾的数据文件 python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --run_mode train \ --train_dataset_dir ./wikitext-2.train.mindrecord ``` - bash启动 ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2.yaml [DEVICE_ID] train ``` #### 多卡训练 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 请提前在yaml文件中修改相应的分布式配置，配置参考[并行配置](https://gitee.com/mindspore/mindformers/blob/dev/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md#config-%E5%B9%B6%E8%A1%8C%E9%85%8D%E7%BD%AE) **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE ../configs/gpt2/run_gpt2.yaml [0,8] train ``` - 多机多卡 在每台机器上启动`bash run_distribute.sh`。 请提前在yaml文件中修改相应的分布式配置，配置参考[并行配置](https://gitee.com/mindspore/mindformers/blob/dev/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md#config-%E5%B9%B6%E8%A1%8C%E9%85%8D%E7%BD%AE) **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [0,8] train $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [$rank_start,$rank_end] train $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ## 微调 ### 数据集准备-微调数据集 - [参考预训练数据集制作](#数据集准备-预训练) ### 全参微调 #### 单卡微调 - python启动 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --run_mode finetune \ --train_dataset_dir ./wikitext-2.train.mindrecord ``` - bash启动 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2.yaml [DEVICE_ID] finetune ``` #### 多卡微调 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE ../configs/gpt2/run_gpt2.yaml [0,8] finetune 8 ``` 多机多卡运行需要合并不同机器的RANK_FILE_TABLE，参考前期准备-[多机RANK_TABLE_FILE合并](#多机rank_table_file合并多机多卡必备环节) - 多机多卡 **请提前将yaml文件中train_dataset配置中的dataset_dir设置为处理好的mindrecord数据路径** 在每台机器上启动`bash run_distribute.sh`。 ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [0,8] finetune $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config.yaml [$rank_start,$rank_end] finetune $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ### Lora微调 #### 单卡微调 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2_lora.yaml --run_mode finetune ``` ```bash cd scripts bash run_standalone.sh ../configs/gpt2/run_gpt2_lora.yaml [DEVICE_ID] finetune ``` #### 多卡微调 多卡运行需要RANK_FILE_TABLE，请参考前期准备-[生成RANK_TABLE_FILE](#生成rank_table_file多卡运行必备环节) - 单机多卡 ```bash cd scripts bash run_distribute.sh RANK_TABLE_FILE path/to/config_lora.yaml [0,8] finetune 8 ``` 多机多卡运行需要合并不同机器的RANK_FILE_TABLE，参考前期准备-[多机RANK_TABLE_FILE合并](#多机生成rank_table_file合并多机多卡必备环节) - 多机多卡 在每台机器上启动`bash run_distribute.sh`。 ```bash server_count=12 device_num=8*$server_count # launch ranks in the 0th server cd scripts bash run_distribute.sh $RANK_TABLE_FILE path/to/config_lora.yaml [0,8] finetune $device_num # launch ranks in the 1-11 server via ssh for idx in {1..11} do let rank_start=8*$idx let rank_end=$rank_start+8 ssh ${IP_LIST[$idx]} "cd scripts; bash run_distribute.sh $RANK_TABLE_FILE path/to/config_lora.yaml [$rank_start,$rank_end] finetune $device_num" done ``` 其中 - `RANK_TABLE_FILE`为上一步汇总并分发的总rank table文件； - `IP_LIST`为12台服务器的IP地址。如192.168.0.[0-11] ```bash IP_LIST=("192.168.0.0", "192.168.0.1", ..., "192.168.0.11") ``` ## 评测 GPT2支持文本生成和文本分类两个任务的评测。 - 文本生成： - 获取数据集： - [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 - 处理数据成mindrecord格式 - WikiText2： ```bash cd mindformers/tools/dataset_preprocess/gpt2 python wikitext2_data_process.py --input_file {your_path/wiki.valid.tokens} \ --output_file {your_path/wikitext-2.valid.mindrecord} ``` - 开启评测： - WikiText2 ```bash python run_mindformer.py --config configs/gpt2/run_gpt2.yaml \ --eval_dataset_dir {your_path/wikitext-2.valid.mindrecord} \ --run_mode eval \ --epochs 1 # gpt2: PerplexityMetric: {'PerplexityMetric': {'loss': 3.24, 'PPL': 25.55} # gpt2_13b(需替换yaml文件): PerplexityMetric: {'PerplexityMetric': {'loss': 2.35, 'PPL': 10.49} ``` - 文本分类： - 获取数据集: - [SST-2数据集](https://dl.fbaipublicfiles.com/glue/data/SST-2.zip)数据集包含电影评论中的句子和它们情感的人类注释。类别分为两类正面情感（positive，样本标签对应为1）和负面情感（negative，样本标签对应为0） - [IMDB数据集](https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie-reviews)影评数据集，包含5万条IMDB影评，评论的情绪是二元的，专门用于情绪分析。 - [AG-News数据集](http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html)数据集包含496,835条来自AG新闻语料库4大类别超过2000个新闻源的新闻文章。 - [COLA数据集](https://nyu-mll.github.io/CoLA/)数据集来自语言理论的书籍和期刊，每个句子被标注为是否合乎语法的单词序列。 - 处理数据成mindrecord格式 ```bash # 因评测前需要微调模型，所以需要生成训练/评测数据集。注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/gpt2 python txtcls_dataset_to_mindrecord.py --dataset_name {select one from ['cola', 'sst_2', 'ag_news', 'imdb']} --input_file {your_path/train.tsv} \ --output_file {your_path/dataset_name.train.mindrecord} python txtcls_dataset_to_mindrecord.py --dataset_name {the same as above} --input_file {your_path/dev.tsv} \ --output_file {your_path/dataset_name.dev.mindrecord} ``` - 开启微调：因为原始权重中不包含隐向量向类别映射的参数，所以无法进行zero-shot，评测前需要事先进行微调。 ```bash # 运行前请确保run_gpt2_txtcls.yaml中的model.model_config.num_labels准确，具体的， # sst2/cola/imdb: num_labels = 2, agnews: num_labels = 4 python run_mindformer.py --config configs/gpt2/run_gpt2_txtcls.yaml \ --train_dataset_dir {your_path/dataset_name.train.mindrecord} \ --run_mode finetune ``` - 开启评测：评测指标为ACC ```bash # 运行前请确保run_gpt2_txtcls.yaml中的model.model_config.num_labels准确，具体的， # sst2/cola/imdb: num_labels = 2, agnews: num_labels = 4 python run_mindformer.py --config configs/gpt2/run_gpt2_txtcls.yaml \ --eval_dataset_dir {your_path/dataset_name.dev.mindrecord} \ --run_mode eval \ --epochs 1 # ACC: COLA-0.693, SST-2-0.908, IMDB-0.934, AG-News-0.941 ``` ## 推理 ### 基于pipeline的推理 Pipeline接口进行推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model='gpt2', max_length=20) pipeline_result = pipeline_task("I love Beijing, because", do_sample=False) print(pipeline_result) # [{'text_generation_text': ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a"]}] ``` 以下为基于pipeline接口的自定义推理脚本，支持多卡多batch推理。 ```python # predict_custom.py 文件 import os import argparse import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindformers import AutoConfig, AutoTokenizer, AutoModel, pipeline from mindformers import init_context, ContextConfig, ParallelContextConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools.utils import str2bool def context_init(use_parallel=False, device_id=0): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=device_id) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, full_batch=True) init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, device_id=0, checkpoint_path="", use_past=True, do_sample=False, max_decode_length=30): """main function.""" # 初始化单卡/多卡环境 context_init(use_parallel, device_id) # 多batch输入 inputs = ["I love Beijing, because", "An increasing sequence: one,"] # set model config model_config = AutoConfig.from_pretrained("gpt2") model_config.use_past = use_past model_config.do_sample = do_sample model_config.max_decode_length = max_decode_length if checkpoint_path and not use_parallel: model_config.checkpoint_name_or_path = checkpoint_path print(f"config is: {model_config}") # build tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # build model from config network = AutoModel.from_config(model_config) # if use parallel, load distributed checkpoints if use_parallel: # find the sharded ckpt path for this rank ckpt_path = os.path.join(checkpoint_path, "rank_{}".format(os.getenv("RANK_ID", "0"))) ckpt_path = get_last_checkpoint(ckpt_path) print("ckpt path: %s", str(ckpt_path)) # shard gpt2 and load sharded ckpt model = Model(network) model.infer_predict_layout(ms.Tensor(np.ones(shape=(1, model_config.seq_length)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(model, checkpoint_dict) print("Network parameters are not loaded: %s", str(not_load_network_params)) text_generation_pipeline = pipeline(task="text_generation", model=network, tokenizer=tokenizer) outputs = text_generation_pipeline(inputs) for output in outputs: print(output) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--use_parallel', default=False, type=str2bool, help='whether use parallel.') parser.add_argument('--device_id', default=0, type=int, help='set device id.') parser.add_argument('--checkpoint_path', default='', type=str, help='set checkpoint path.') parser.add_argument('--use_past', default=True, type=str2bool, help='whether use past.') parser.add_argument('--do_sample', default=False, type=str2bool, help='whether enable do_sample.') parser.add_argument('--max_decode_length', default=30, type=int, help='the length of generated text.') args = parser.parse_args() main(args.use_parallel, args.device_id, args.checkpoint_path, args.use_past, args.do_sample, args.max_decode_length) # {'text_generation_text': ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"]} # {'text_generation_text': ['An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen,']} ``` 以下为多卡运行自定义多batch推理的脚本 ```bash # >>> `run_predict.sh`文件 CHECKPOINT_PATH=$2 export RANK_TABLE_FILE=$1 # define variable export RANK_SIZE=8 export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./predict_custom.py --use_parallel True --checkpoint_path CHECKPOINT_PATH &> minformers_$RANK_ID.log & done ``` #### 单卡pipeline推理 ```bash python predict_custom.py ``` #### 多卡pipeline推理 ```bash bash run_predict.sh RANK_TABLE_FILE path/to/gpt2_shard_checkpoint_dir ``` ### 基于generate的推理 Generate接口进行推理 ```python # 使用AutoModel.from_pretrained实例化模型 from mindformers import AutoModel, AutoTokenizer model = AutoModel.from_pretrained("gpt2") tokenizer = AutoTokenizer.from_pretrained("gpt2") input_ids = tokenizer("I love Beijing, because")["input_ids"] generated_ids = model.generate(input_ids, do_sample=False, max_length=30) generate_result = tokenizer.decode(generated_ids) print(generate_result) # ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"] ``` ```python # 使用AutoModel.from_config实例化模型 from mindformers import AutoModel, AutoTokenizer, AutoConfig # 可对model_config进行修改，如model_config.do_sample = False model_config = AutoConfig.from_pretrained("gpt2") model = AutoModel.from_config(model_config) tokenizer = AutoTokenizer.from_pretrained("gpt2") input_ids = tokenizer("I love Beijing, because")["input_ids"] generate_result = model.generate(input_ids, do_sample=False, max_length=30) print(generate_result) # ["I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It"] ``` 以下为基于model.generate接口的自定义推理脚本，支持多卡多batch推理。 ```python # predict_custom.py 文件 import os import argparse import numpy as np import mindspore as ms from mindspore.train import Model from mindspore import load_checkpoint, load_param_into_net from mindformers import AutoConfig, AutoTokenizer, AutoModel from mindformers import init_context, ContextConfig, ParallelContextConfig from mindformers.trainer.utils import get_last_checkpoint from mindformers.tools.utils import str2bool def context_init(use_parallel=False, device_id=0): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=device_id) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, full_batch=True) init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, device_id=0, checkpoint_path="", use_past=True, do_sample=False, max_decode_length=30): """main function.""" # 初始化单卡/多卡环境 context_init(use_parallel, device_id) # 多batch输入 inputs = ["I love Beijing, because", "An increasing sequence: one,"] # set model config model_config = AutoConfig.from_pretrained("gpt2") model_config.batch_size = len(inputs) model_config.use_past = use_past model_config.do_sample = do_sample model_config.max_decode_length = max_decode_length if checkpoint_path and not use_parallel: model_config.checkpoint_name_or_path = checkpoint_path print(f"config is: {model_config}") # build tokenizer tokenizer = AutoTokenizer.from_pretrained("gpt2") # build model from config model = AutoModel.from_config(model_config) # if use parallel, load distributed checkpoints if use_parallel: # find the sharded ckpt path for this rank ckpt_path = os.path.join(checkpoint_path, "rank_{}".format(os.getenv("RANK_ID", "0"))) ckpt_path = get_last_checkpoint(ckpt_path) print("ckpt path: %s", str(ckpt_path)) # shard gpt2 and load sharded ckpt model = Model(model) model.infer_predict_layout(ms.Tensor(np.ones(shape=(1, model_config.seq_length)), ms.int32)) checkpoint_dict = load_checkpoint(ckpt_path) not_load_network_params = load_param_into_net(model, checkpoint_dict) print("Network parameters are not loaded: %s", str(not_load_network_params)) inputs_ids = tokenizer(inputs, max_length=model_config.max_decode_length, padding="max_length")["input_ids"] outputs = model.generate(inputs_ids, max_length=model_config.max_decode_length) for output in outputs: print(tokenizer.decode(output)) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--use_parallel', default=False, type=str2bool, help='whether use parallel.') parser.add_argument('--device_id', default=0, type=int, help='set device id.') parser.add_argument('--checkpoint_path', default='', type=str, help='set checkpoint path.') parser.add_argument('--use_past', default=True, type=str2bool, help='whether use past.') parser.add_argument('--do_sample', default=False, type=str2bool, help='whether enable do_sample.') parser.add_argument('--max_decode_length', default=30, type=int, help='the length of generated text.') args = parser.parse_args() main(args.use_parallel, args.device_id, args.checkpoint_path, args.use_past, args.do_sample, args.max_decode_length) # I love Beijing, because it's a beautiful city. It's a beautiful city. It's a beautiful city. It's a beautiful city. It # An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, ``` 以下为多卡运行自定义多batch推理的脚本 ```bash # >>> `run_predict.sh`文件 CHECKPOINT_PATH=$2 export RANK_TABLE_FILE=$1 # define variable export RANK_SIZE=8 export START_RANK=0 # this server start rank export END_RANK=8 # this server end rank # run for((i=${START_RANK}; i<${END_RANK}; i++)) do export RANK_ID=$i export DEVICE_ID=$((i-START_RANK)) echo "Start distribute running for rank $RANK_ID, device $DEVICE_ID" python3 ./predict_custom.py --use_parallel True --checkpoint_path CHECKPOINT_PATH &> minformers_$RANK_ID.log & done ``` #### 单卡generate推理 ```bash python predict_custom.py ``` #### 多卡generate推理 ```bash bash run_predict.sh RANK_TABLE_FILE path/to/pangualpha_2_6b_shard_checkpoint_dir ``` ### 脚本启动 #### 单卡推理 ```bash python run_mindformer.py --config configs/pangualpha/run_pangualpha_2_6b.yaml --run_mode predict --predict_data 上联：欢天喜地度佳节 下联： --use_parallel False # output result is: [{'text_generation_text': ['上联:欢天喜地度佳节 下联:笑逐颜开迎佳期 横批:幸福快乐']}] ``` **注**：要提高推理速度，可在对应模型配置文件中进行如下配置，设置增量推理`use_past`为True。 ```yaml # model config use_past: True # 开启增量推理 use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "gpt2" repetition_penalty: 1 max_decode_length: 1024 top_k: 3 top_p: 1 do_sample: True ```pengcheng.mind-mindformer/docs/model_cards/index.rst000066400000000000000000000101561476650141400231620ustar00rootroot00000000000000.. role:: raw-html-m2r(raw) :format: html 模型 ========================== **official** .. toctree:: :glob: :maxdepth: 1 bert bloom clip glm gpt2 llama mae pangualpha swin t5 vit 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `BERT `_ - masked_language_modeling\ :raw-html-m2r:`
`\ `text_classification <../task_cards/text_classification.html>`_\ :raw-html-m2r:`
`\ `token_classification <../task_cards/token_classification.html>`_\ :raw-html-m2r:`
`\ `question_answering <../task_cards/question_answering.html>`_ - bert_base_uncased :raw-html-m2r:`
`\ txtcls_bert_base_uncased\ :raw-html-m2r:`
`\ txtcls_bert_base_uncased_mnli :raw-html-m2r:`
`\ tokcls_bert_base_chinese\ :raw-html-m2r:`
`\ tokcls_bert_base_chinese_cluener :raw-html-m2r:`
`\ qa_bert_base_uncased\ :raw-html-m2r:`
`\ qa_bert_base_chinese_uncased * - `T5 `_ - translation - t5_small * - `GPT2 `_ - `text_generation <../task_cards/text_generation.html>`_ - gpt2_small :raw-html-m2r:`
`\ gpt2_13b :raw-html-m2r:`
`\ gpt2_52b * - `PanGuAlpha `_ - `text_generation <../task_cards/text_generation.html>`_ - pangualpha_2_6_b\ :raw-html-m2r:`
`\ pangualpha_13b * - `GLM `_ - `text_generation <../task_cards/text_generation.html>`_ - glm_6b\ :raw-html-m2r:`
`\ glm_6b_lora * - `GLM2 `_ - `text_generation <../task_cards/text_generation.html>`_ - glm2_6b\ :raw-html-m2r:`
`\ glm2_6b_lora * - `LLama `_ - `text_generation <../task_cards/text_generation.html>`_ - llama_7b :raw-html-m2r:`
`\ llama_13b :raw-html-m2r:`
`\ llama_65b :raw-html-m2r:`
`\ llama_7b_lora * - `Bloom `_ - `text_generation <../task_cards/text_generation.html>`_ - bloom_560m\ :raw-html-m2r:`
`\ bloom_7.1b :raw-html-m2r:`
`\ bloom_65b\ :raw-html-m2r:`
`\ bloom_176b * - `MAE `_ - masked_image_modeling - mae_vit_base_p16 * - `VIT `_ - `image_classification <../task_cards/image_classification.html>`_ - vit_base_p16 * - `Swin `_ - `image_classification <../task_cards/image_classification.html>`_ - swin_base_p4w7 * - `CLIP `_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - clip_vit_b_32\ :raw-html-m2r:`
`\ clip_vit_b_16 :raw-html-m2r:`
`\ clip_vit_l_14\ :raw-html-m2r:`
`\ clip_vit_l_14@336 * - `BLIP2 `_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - blip2_vit_g **research** .. toctree:: :glob: :maxdepth: 1 ../../research/baichuan/baichuan ../../research/baichuan2/baichuan2 ../../research/internlm/internlm ../../research/ziya/ziya 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `baichuan <../../research/baichuan/baichuan.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - baichuan_7b\ :raw-html-m2r:`
`\ baichuan_13b * - `baichuan2 <../../research/baichuan2/baichuan2.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - baichuan2_7b\ :raw-html-m2r:`
`\ baichuan2_13b * - `internlm <../../research/internlm/internlm.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - internlm_7b :raw-html-m2r:`
`\ internlm_7b_lora * - `ziya <../../research/ziya/ziya.html>`_ - `text_generation <../task_cards/text_generation.html>`_ - ziya_13b pengcheng.mind-mindformer/docs/model_cards/llama.md000066400000000000000000000545411476650141400227370ustar00rootroot00000000000000# LLaMA ## 模型描述 LLaMA是由Meta于2023年发布。LLaMA模型是类GPT模型，是一个生成式的语言模型，主要是用于预测下一个单词。LLaMA目前按照参数量，目前有四个版本：LLaMA-7B（7B）、LLaMA-13B（13B）、LLaMA-33B（33B）以及LLaMA-65B（65B），目前在本仓库中，支持了7B，13B和65B三个规格的模型，权重文件来源于OpenLLaMA。 [LLaMA: Open and Efficient Foundation Language Models](https://arxiv.org/abs/2302.13971) ``` text @article{touvron2023llama, title={LLaMA: Open and Efficient Foundation Language Models}, author={Touvron, Hugo and Lavril, Thibaut and Izacard, Gautier and Martinet, Xavier and Lachaux, Marie-Anne and Lacroix, Timoth{\'e}e and Rozi{\`e}re, Baptiste and Goyal, Naman and Hambro, Eric and Azhar, Faisal and Rodriguez, Aurelien and Joulin, Armand and Grave, Edouard and Lample, Guillaume}, journal={arXiv preprint arXiv:2302.13971}, year={2023} } ``` ## 代码结构介绍 `LLaMA` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/llama` ```bash llama ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── llama.py # 模型实现 ├── llama_config.py # 模型配置项 ├── llama_layer.py # llama网络层定义 ├── llama_processor.py # llama预处理 ├── llama_tokenizer.py # tokenizer └── llama_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/llama` ```bash llama ├── run_llama_7b.yaml # 7b模型全量微调启动配置 ├── run_llama_7b_lora.yaml # 7b lora低参微调启动配置 ├── run_llama_13b.yaml # 13b全量微调启动配置 └── run_llama_65b.yaml # 65b全量微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A - MindSpore：2.0.0 / 1.10.1 - MindFormers版本：dev 注：推理可在单机单卡上完成部署；全量微调至少需要单机8卡，Lora微调至少需要单卡。 ## 权重转换与权重合并 ### 开源预训练权重转换 从huggingface下载英文预训练权重（权重来源于OpenLLaMA）： - [llama-7b](https://huggingface.co/openlm-research/open_llama_7b) - [llama-13b](https://huggingface.co/openlm-research/open_llama_13b) 注意：65B权重OpenLLaMA未提供，如有需要，请开发者自行解决。 下载完成后，运行如下转换脚本，将huggingface的权重转换为完整的ckpt权重。 ```shell python mindformers/models/llama/convert_weight.py \ --torch_ckpt_dir TORCH_CKPT_DIR \ --mindspore_ckpt_path {path}/MS_CKPT_NAME ``` ```text # 参数说明 torch_ckpt_dir: huggingface权重保存目录路径 mindspore_ckpt_path: 权重保存文件名，可以指定自定义保存路径 ``` ### 分布式训练/微调权重合并 分布式训练/微调后所得到的权重文件为根据策略切分后的权重，需要手动将切分权重合一，以用于评估和推理。 - step 1. 获取模型切分策略文件： 在执行微调脚本时，模型完成编译后，将会在`output/strategy`路径下生成各卡的切分策略文件，用于权重合并。 > 注：lora微调时需要确认配置文件`parallel context config`中`only_trainable_params`为`False`，以获取所有参数完整策略。 - step 2. 运行`mindformers/tools/transform_ckpt.py`脚本进行多卡权重合并： ```shell python transform_ckpt.py \ --src_ckpt_strategy {path}/output/strategy/ \ --src_ckpt_dir {path}/output/checkpoint/ \ --dst_ckpt_dir {path}/target_checkpoint/ \ --prefix llama_7b ``` ```text # 参数说明 src_ckpt_strategy: 步骤1中的切分策略文件路径 src_ckpt_dir: 原切分权重文件夹 dst_ckpt_dir: 目标路径 prefix: ckpt文件前缀名 ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ## 推理 ### llama推理 以llama-7b模型为例，利用预训练权重进行推理。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline(task="text_generation", model="llama_7b", max_length=50) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) # output: # [{'text_generation_text': ['I love Beijing, because it’s a city that’s constantly changing. It’s a city that’s constantly evolving. It’s a city that’s constantly reinventing itself. And I think that’s what makes it']}] ``` #### 基于API接口的推理 ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task="text_generation", model="llama_7b", train_dataset="{dataset file path}") # 方式1：从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="I love Beijing, because") # 方式2：用户自行指定权重路径并进行推理 res = trainer.predict(input_data="I love Beijing, because", predict_checkpoint="{checkpoint file path}") ``` ### lora推理 可直接使用以下两种方式，利用lora微调后的权重进行推理，运行脚本时会自动拉取本仓库提供的lora权重文件。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model="llama_7b_lora", max_length=20) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) ``` #### 基于API接口的推理 ```python from mindformers import Trainer import mindspore as ms cls_trainer = Trainer(task="text_generation", # 已支持的任务名 model="llama_7b", pet_method="lora") # 已支持的模型名 # 根据alpaca数据集的prompt模板，在instruction处填入自定义指令 input_data = "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{}\n\n### Response:".format("Tell me about alpacas.") # 方式1： 传入lora微调后的权重进行推理 lora_ckpt = "./llama_7b_lora.ckpt" predict_result = cls_trainer.predict(input_data=input_data, predict_checkpoint=lora_ckpt) # 方式2： 从obs下载训练好的权重进行推理 predict_result = cls_trainer.predict(input_data=input_data) print(predict_result[0]["text_generation_text"][0]) # output: # Alpacas are a species of South American camelid. They are domesticated animals that are raised for their wool, meat, and milk. Alpacas are gentle, docile animals that are very friendly and easy to care for. They are also very intelligent and can be trained to perform certain tasks. Alpacas are very social animals and live in herds of up to 20 individuals. They are also very vocal and can make a variety of sounds, including a loud, high-pitched bark. ``` #### lora推理结果示例 **Instruction**: Tell me about the president of Mexico in 2019. **Output**: The president of Mexico in 2019 is Andrés Manuel López Obrador. He was elected in 2018 and is the first left-wing president of Mexico in over 70 years. He is known for his populist policies and his focus on improving the lives of the poor and marginalized. He has also been criticized for his lack of transparency and his authoritarian style of leadership. --- **Instruction**: List all Canadian provinces in alphabetical order. **Output**: Alberta, British Columbia, Manitoba, New Brunswick, Newfoundland and Labrador, Nova Scotia, Ontario, Prince Edward Island, Quebec, Saskatchewan, and Yukon. --- **Instruction**: Write a Python program that prints the first 10 Fibonacci numbers. **Output**: ```python def fibonacci(n): if n == 0: return 0 elif n == 1: return 1 else: return fibonacci(n-1) + fibonacci(n-2) print(fibonacci(10)) ``` --- **Instruction**: Write a program that prints the numbers from 1 to 100. But for multiples of three print 'Fizz' instead of the number and for the multiples of five print 'Buzz'. For numbers which are multiples of both three and five print 'FizzBuzz'. **Output**: ```python def fizzbuzz(n): if n % 3 == 0 and n % 5 == 0: print('FizzBuzz') elif n % 3 == 0: print('Fizz') elif n % 5 == 0: print('Buzz') else: print(n) ``` --- **Instruction**: Translate the sentence 'I have no mouth but I must scream' into Spanish. **Output**: No tengo boca pero tengo que gritar. --- ## 预训练 ### 数据集准备 以Wikitext2数据集为例: - 数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) - 分词模型下载：例如下载huggingface的[tokenizer.model](https://huggingface.co/openlm-research/open_llama_7b/blob/main/tokenizer.model) - 使用以下预处理脚本生成mindrecord训练数据 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.train.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/wiki2048.mindrecord ``` ### 脚本启动（LLaMA-7B为例） - 用户需要首先clone整个仓库，请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动)完成启动准备工作。 #### 单机多卡启动 - step 1. 在仓库主目录下，运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`。 ```shell # 以八卡运行为例，生成0~7卡的hccl json文件,不包含8本身. python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` - step 2. 修改模型对应的配置文件。 在模型对应的配置文件`configs/llama/run_llama_{7/13/65}b.yaml`中，用户可自行修改模型、训练相关参数，并通过`train_dataset`的`dataset_dir`参数，指定训练数据集的路径。 配置文件中各参数含义详见[Config配置说明文档](https://gitee.com/mindspore/mindformers/blob/master/configs/README.md)。 - step3：进入`scripts`文件夹，启动运行脚本，进行8卡分布式运行。 ```shell cd scripts bash run_distribute.sh hccl_xxxx.json ../configs/llama/run_llama_7b.yaml [0,8] train ``` ```text # 脚本启动格式： bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_MODE] # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的llama/run_llama_7b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` #### 多机多卡启动 - step 1. 首先参考单机多卡启动方式，在每台机器上运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`的json文件。 ```shell # 在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` - step 2. 合并每台机器上生成的`RANK_TABLE_FILE`。 将不同机器上生成的`RANK_TABLE_FILE`文件拷贝到一起，执行`merge_hccl.py`脚本进行合并，包括server_list合并，`server_count`设为机器数，`rank_id`顺序增加。 ```shell # 运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 - step 4. 根据服务器节点数等信息，修改相应的配置。 ```shell # 以llama-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。 # 配置文件路径：../configs/llama/run_llama_13b.yaml parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 ``` - step 5. 执行运行脚本。 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式。需注意，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/llama/run_llama_7b.yaml [0,8] train 16 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/llama/run_llama_7b.yaml [8,16] train 16 ``` ## 微调 ### 数据集准备 目前提供alpaca数据集的预处理脚本用于全参微调/lora微调任务。 数据集下载链接如下： - [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) alpaca数据集原始格式样例： ```text # alpaca examples: { "instruction": "Describe a time when you had to make a difficult decision.", "input": "", "output": "I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client\u2019s expectations. However, due to unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team\u2019s resources even further and increase the budget. Although it was a risky decision, I ultimately decided to go ahead with it to ensure that the project was completed on time and that the client\u2019s expectations were met. The project was eventually successfully completed and this was seen as a testament to my leadership and decision-making abilities." }, { "instruction": "Identify the odd one out.", "input": "Twitter, Instagram, Telegram", "output": "Telegram" }, ``` - step 1. 执行`alpaca_converter.py`，使用fastchat工具添加prompts模板，将原始数据集转换为多轮对话格式。 ``` bash # 脚本路径：tools/dataset_preprocess/llama/alpaca_converter.py # 执行转换脚本 python alpaca_converter.py \ --data_path /{path}/alpaca_data.json \ --output_path /{path}/alpaca-data-conversation.json ``` ```text # 参数说明 data_path: 存放alpaca数据的路径 output_path: 输出转换后对话格式的数据路径 ``` 转换后格式样例： ```text { "id": "1", "conversations": [ { "from": "human", "value": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive three tips for staying healthy.\n\n### Response:" }, { "from": "gpt", "value": "1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule." } ] }, ``` - step 2. 执行`llama_preprocess.py`，进行数据预处理、Mindrecord数据生成，将带有prompt模板的数据转换为mindrecord格式。 ```bash # 脚本路径：tools/dataset_preprocess/llama/llama_preprocess.py # 由于此工具依赖fschat工具包解析prompt模板，请提前安装fschat >= 0.2.13 python = 3.9 python llama_preprocess.py \ --dataset_type qa \ --input_glob /{path}/alpaca-data-conversation.json \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/alpaca-fastchat2048.mindrecord ``` ### 全参微调 以llama7b为例 - step 1. 修改`config/llama/run_llama_7b.yaml`中训练数据集路径为微调数据集路径，并在`input_columns`中添加`labels`。 ```python train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "/{path}/alpaca-fastchat2048.mindrecord" shuffle: True input_columns: ["input_ids", "labels"] ``` - step 2. 修改训练时学习率和优化器参数，与预训练不同，微调学习率配置如下： ```python # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 1.e-8 learning_rate: 1.e-5 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset ``` - step 3. 添加预训练权重路径，修改配置文件中的`load_checkpoint`，配置预训练权重路径。 - step 4. 启动微调任务，llama-7b模型以单机八卡为例进行微调，命令如下： ```shell cd scripts bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama/run_llama_7b.yaml [0,8] finetune ``` 多机多卡微调任务启动参考[预训练章节](#jump)，添加预训练权重，修改启动脚本中的`RUN_MODE`为`finetune`即可。 ### lora微调 目前lora微调适配了llama_7b模型，并给出了默认配置文件`config/llama/run_llama_7b_lora.yaml`。 #### 脚本启动 - step 1. 修改配置文件，参考全参微调修改训练数据集路径与预训练权重路径。 - step 2. 启动lora微调任务。 注：llama_7b_lora模型支持单卡启动，需将配置文件中的`use_parallel`参数置为`False`。 ```shell cd scripts # 单卡启动 bash run_standalone.sh ../configs/llama/run_llama_7b_lora.yaml [DEVICE_ID] finetune # 多卡启动（以单机八卡为例） bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama/run_llama_7b_lora.yaml [0,8] finetune ``` #### API高阶接口启动 lora微调支持使用高阶接口启动单卡微调任务，示例代码如下： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='text_generation', model='llama_7b', pet_method='lora', train_dataset="{dataset file path}") # 调用finetune接口进行微调 trainer.finetune(finetune_checkpoint="{checkpoint file path}") ``` ## 评测 Llama当前支持的评测任务如下： | 任务类型 | 评测指标 | 数据集 | |:----:|:----------:|:---------:| | 文本生成 | Perplexity | WikiText2 | | 阅读理解 | Em/F1 | SQuAD 1.1 | - 文本生成： step 1. 获取数据集 [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.valid.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 2047 \ --output_file /{path}/wiki2048.mindrecord ``` step 3. 开启评测，指标为PPL ```bash python run_mindformer.py \ --config configs/llama/run_llama_7b.yaml \ --eval_dataset_dir /{path}/wiki2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama_7b.ckpt \ --epochs 1 \ --use_parallel False \ --device_id 0 # PerplexityMetric = {'PerplexityMetric': {'loss': 2.1142693907022476, 'PPL': 8.283531529594038}} ``` - 阅读理解： step 1. 获取数据集 [SQuAD 1.1](https://data.deepai.org/squad1.1.zip)包含针对500+文章的10万+问答对,是一个阅读理解数据集，由维基百科文章上提出的问题组成，其中每个问题的答案都是相应文章中的一段文本。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/squad_data_process.py进行数据预处理+Mindrecord数据生成 python squad_data_process.py \ --input_file /{path}/squad/dev-v1.1.json \ --output_file /{path}/squad2048.mindrecord \ --mode eval \ --max_length 2048 \ --tokenizer_type "llama_7b" ``` 预处理后数据格式举例： ```text Read the passage and answer the question below. ### Instruction: The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated the Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by defeating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made eight appearances in the Super Bowl. ### Input: Which Carolina Panthers player was named Most Valuable Player? ### Response: Cam Newton ``` step 3. 修改配置文件，eval_dataset的input_columns中增加`labels`，修改metric类型为`EmF1Metric` ```python # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] # 增加"labels" # metric metric: type: EmF1Metric # metric type设为EmF1Metric ``` 此外，要提高推理速度，可以进行如下配置，设置增量推理`use_past`，并限制生成最大长度`max_new_tokens`。 ```python # model config use_past: True # 开启增量推理 pretrain_seqlen: 2048 extend_method: "None" offset: 0 checkpoint_name_or_path: "llama_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False max_new_tokens: 20 #设置最大生成长度 ``` step 4. 开启评测，指标为`Em/F1` ```bash python run_mindformer.py \ --config configs/llama/run_llama_7b.yaml \ --eval_dataset_dir /{path}/squad2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama_7b.ckpt \ --epochs 1 \ --batch_size 1 \ --use_parallel False \ --device_id 0 # F1 score: 48.48954955952303, Em score: 26.850507982583455, total_count: 2067 ``` pengcheng.mind-mindformer/docs/model_cards/llama2.md000066400000000000000000000446011476650141400230150ustar00rootroot00000000000000# Llama 2 ## 模型描述 Llama 2，是Meta基于LLaMA 1的更新版本，基于新的公开可用数据混合进行训练，同时将预训练语料库的大小增加了40%，最后将模型的上下文长度翻倍（由2048提高到4096），并采用了分组查询注意力机制。Llama 2模型是类GPT模型，是一个生成式的语言模型，主要是用于预测下一个单词。Llama 2按照参数量，目前有三个版本：Llama 2-7B（7B）、Llama 2-13B（13B）、Llama 2-70B（70B），本仓库已全部支持三版权重，权重文件来源于MetaLLama2。Llama 2 的7B和13B 模型结构与LLaMA 1一致，70B 则加入分组查询注意力（GQA）。 [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://arxiv.org/pdf/2307.09288.pdf%C3%82%C2%A0) ``` text @article{touvron2023llama, title={Llama 2: Open foundation and fine-tuned chat models}, author={Touvron, Hugo and Martin, Louis and Stone, Kevin and Albert, Peter and Almahairi, Amjad and Babaei, Yasmine and Bashlykov, Nikolay and Batra, Soumya and Bhargava, Prajjwal and Bhosale, Shruti and others}, journal={arXiv preprint arXiv:2307.09288}, year={2023} } ``` ## 代码结构介绍 `Llama 2` 基于 `mindformers` 实现，主要涉及的文件有： 1. 模型具体实现：`mindformers/models/llama` ```bash llama ├── __init__.py ├── convert_weight.py # 权重转换脚本 ├── llama.py # 模型实现 ├── llama_config.py # 模型配置项 ├── llama_layer.py # llama网络层定义 ├── llama_processor.py # llama预处理 ├── llama_tokenizer.py # tokenizer └── llama_transformer.py # transformer层实现 ``` 2. 模型配置：`configs/llama2` ```bash llama ├── run_llama2_7b.yaml # 7b模型全量微调启动配置 ├── run_llama2_13b.yaml # 13b全量微调启动配置 └── run_llama2_70b.yaml # 70b全量微调启动配置 ``` ## 环境要求 - 硬件：Ascend 910A/910B - MindSpore：2.0 / 2.1 - MindFormers版本：dev 注：推理可在单机单卡上完成部署；全量微调至少需要单机8卡。 ## 权重转换与权重合并 ### 开源预训练权重转换 从huggingface下载英文预训练权重（权重来源于MetaLLama2）： - [llama2-7b](https://huggingface.co/meta-llama/Llama-2-7b) - [llama2-13b](https://huggingface.co/meta-llama/Llama-2-13b) - [llama2-70b](https://huggingface.co/meta-llama/Llama-2-70b) 注：Llama 2的所有权重都需要向Meta提交[申请](https://ai.meta.com/resources/models-and-libraries/llama-downloads)，如有需要，请开发者自行申请。 下载完成后，运行如下转换脚本，将huggingface的权重转换为完整的ckpt权重。 ```shell python mindformers/models/llama/convert_weight.py \ --torch_ckpt_dir TORCH_CKPT_DIR \ --mindspore_ckpt_path {path}/MS_CKPT_NAME ``` ```text # 参数说明 torch_ckpt_dir: huggingface权重保存目录路径 mindspore_ckpt_path: 权重保存文件名，可以指定自定义保存路径 ``` ### 分布式训练/微调权重合并 分布式训练/微调后所得到的权重文件为根据策略切分后的权重，需要手动将切分权重合一，以用于评估和推理。 - step 1. 获取模型切分策略文件： 在执行微调脚本时，模型完成编译后，将会在`output/strategy`路径下生成各卡的切分策略文件，用于权重合并。 - step 2. 运行`mindformers/tools/transform_ckpt.py`脚本进行多卡权重合并： ```shell python transform_ckpt.py \ --src_ckpt_strategy {path}/output/strategy/ \ --src_ckpt_dir {path}/output/checkpoint/ \ --dst_ckpt_dir {path}/target_checkpoint/ \ --prefix llama2_7b ``` ```text # 参数说明 src_ckpt_strategy: 步骤1中的切分策略文件路径 src_ckpt_dir: 原切分权重文件夹 dst_ckpt_dir: 目标路径 prefix: ckpt文件前缀名 ``` > 注：`transform_checkpoints` 接口当前仅mindspore 2.0以上版本支持，如当前硬件环境只支持2.0以下版本，可以新建conda环境安装mindspore 2.0的cpu版本以执行该脚本 ## 推理 ### Llama 2推理 以llama2-7b模型为例，利用预训练权重进行推理。 #### pipeline推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline(task="text_generation", model="llama2_7b", max_length=50) pipeline_result = pipeline_task("I love Beijing, because", top_k=3) print(pipeline_result) # output: # [{'text_generation_text': ['I love Beijing, because it’s a city that’s constantly changing. It’s a city that’s constantly evolving. It’s a city that’s constantly reinventing itself. And I think that’s what makes it']}] ``` #### 基于API接口的推理 ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task="text_generation", model="llama2_7b", train_dataset="{dataset file path}") # 方式1：从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="I love Beijing, because") # 方式2：用户自行指定权重路径并进行推理 res = trainer.predict(input_data="I love Beijing, because", predict_checkpoint="{checkpoint file path}") ``` ## 预训练 ### 数据集准备 以Wikitext2数据集为例: - 数据集下载：[WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip) - 分词模型下载：例如下载申请通过后huggingface里对应Files 中的tokenizer.model - 使用以下预处理脚本生成mindrecord训练数据 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.train.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 4096 \ --output_file /{path}/wiki4096.mindrecord ``` ### 脚本启动（Llama 2-7B为例） - 用户需要首先clone整个仓库，请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动)完成启动准备工作。 #### 单机多卡启动 - step 1. 在仓库主目录下，运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`。 ```shell # 以八卡运行为例，生成0~7卡的hccl json文件,不包含8本身. python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` - step 2. 修改模型对应的配置文件。 在模型对应的配置文件`configs/llama2/run_llama2_{7/13/70}b.yaml`中，用户可自行修改模型、训练相关参数，并通过`train_dataset`的`dataset_dir`参数，指定训练数据集的路径。 配置文件中各参数含义详见[Config配置说明文档](https://gitee.com/mindspore/mindformers/blob/master/configs/README.md)。 - step3：进入`scripts`文件夹，启动运行脚本，进行8卡分布式运行。 ```shell cd scripts bash run_distribute.sh hccl_xxxx.json ../configs/llama2/run_llama2_7b.yaml [0,8] train ``` ```text # 脚本启动格式： bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_MODE] # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的llama2/run_llama2_7b.yaml配置文件 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_MODE: 为任务运行状态，支持关键字 train\finetune\eval\predict ``` #### 多机多卡启动 - step 1. 首先参考单机多卡启动方式，在每台机器上运行`mindformers/tools/hccl_tools.py`生成`RANK_TABLE_FILE`的json文件。 ```shell # 在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num [0,8] ``` - step 2. 合并每台机器上生成的`RANK_TABLE_FILE`。 将不同机器上生成的`RANK_TABLE_FILE`文件拷贝到一起，执行`merge_hccl.py`脚本进行合并，包括server_list合并，`server_count`设为机器数，`rank_id`顺序增加。 ```shell # 运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json ``` - step 3. 将合并后的`RANK_TABLE_FILE`文件拷贝到所有机器中，保证不同机器上的`RANK_TABLE_FILE`相同。 - step 4. 根据服务器节点数等信息，修改相应的配置。 ```shell # 以llama2-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。 # 配置文件路径：../configs/llama2/run_llama2_13b.yaml parallel_config: data_parallel: 2 model_parallel: 4 pipeline_stage: 2 optimizer_shard: True micro_batch_num: 16 vocab_emb_dp: True gradient_aggregation_group: 4 ``` - step 5. 执行运行脚本。 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式。需注意，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/llama2/run_llama2_13b.yaml [0,8] train 16 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/llama2/run_llama2_13b.yaml [8,16] train 16 ``` ## 微调 ### 数据集准备 目前提供alpaca数据集的预处理脚本用于全参微调任务。 数据集下载链接如下： - [alpaca_data](https://github.com/tatsu-lab/stanford_alpaca/blob/main/alpaca_data.json) alpaca数据集原始格式样例： ```text # alpaca examples: { "instruction": "Describe a time when you had to make a difficult decision.", "input": "", "output": "I had to make a difficult decision when I was working as a project manager at a construction company. I was in charge of a project that needed to be completed by a certain date in order to meet the client\u2019s expectations. However, due to unexpected delays, we were not able to meet the deadline and so I had to make a difficult decision. I decided to extend the deadline, but I had to stretch the team\u2019s resources even further and increase the budget. Although it was a risky decision, I ultimately decided to go ahead with it to ensure that the project was completed on time and that the client\u2019s expectations were met. The project was eventually successfully completed and this was seen as a testament to my leadership and decision-making abilities." }, { "instruction": "Identify the odd one out.", "input": "Twitter, Instagram, Telegram", "output": "Telegram" }, ``` - step 1. 执行`alpaca_converter.py`，使用fastchat工具添加prompts模板，将原始数据集转换为多轮对话格式。 ``` bash # 脚本路径：tools/dataset_preprocess/llama/alpaca_converter.py # 执行转换脚本 python alpaca_converter.py \ --data_path /{path}/alpaca_data.json \ --output_path /{path}/alpaca-data-conversation.json ``` ```text # 参数说明 data_path: 存放alpaca数据的路径 output_path: 输出转换后对话格式的数据路径 ``` 转换后格式样例： ```text { "id": "1", "conversations": [ { "from": "human", "value": "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\nGive three tips for staying healthy.\n\n### Response:" }, { "from": "gpt", "value": "1.Eat a balanced diet and make sure to include plenty of fruits and vegetables. \n2. Exercise regularly to keep your body active and strong. \n3. Get enough sleep and maintain a consistent sleep schedule." } ] }, ``` - step 2. 执行`llama_preprocess.py`，进行数据预处理、Mindrecord数据生成，将带有prompt模板的数据转换为mindrecord格式。 ```bash # 脚本路径：tools/dataset_preprocess/llama/llama_preprocess.py # 由于此工具依赖fschat工具包解析prompt模板，请提前安装fschat >= 0.2.13 python = 3.9 python llama_preprocess.py \ --dataset_type qa \ --input_glob /{path}/alpaca-data-conversation.json \ --model_file /{path}/tokenizer.model \ --seq_length 2048 \ --output_file /{path}/alpaca-fastchat2048.mindrecord ``` ### 全参微调 以llama2 7b为例 - step 1. 修改`config/llama2/run_llama2_7b.yaml`中训练数据集路径为微调数据集路径，并在`input_columns`中添加`labels`。 ```python train_dataset: &train_dataset data_loader: type: MindDataset dataset_dir: "/{path}/alpaca-fastchat2048.mindrecord" shuffle: True input_columns: ["input_ids", "labels"] ``` - step 2. 修改训练时学习率和优化器参数，与预训练不同，微调学习率配置如下： ```python # optimizer optimizer: type: FP32StateAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 1.e-8 learning_rate: 1.e-5 # lr sechdule lr_schedule: type: CosineWithWarmUpLR learning_rate: 1.e-5 warmup_ratio: 0.03 total_steps: -1 # -1 means it will load the total steps of the dataset ``` - step 3. 修改seq_length，seq_length配置如下： ```python # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 ``` ​ 注意：alpaca数据集最长不超过2048，因此seq_length采用2048即可。 - step 3. 添加预训练权重路径，修改配置文件中的`load_checkpoint`，配置预训练权重路径。 - step 4. 启动微调任务，llama2-7b模型以单机八卡为例进行微调，命令如下： ```shell cd scripts bash run_distribute.sh [RANK_TABLE_FILE] ../configs/llama2/run_llama2_7b.yaml [0,8] finetune ``` 多机多卡微调任务启动参考[预训练章节](#jump)，添加预训练权重，修改启动脚本中的`RUN_MODE`为`finetune`即可。 ## 评测 Llama 2当前支持的评测任务如下： | 任务类型 | 评测指标 | 数据集 | | :------: | :--------: | :-------: | | 文本生成 | Perplexity | WikiText2 | | 阅读理解 | Em/F1 | SQuAD 1.1 | - 文本生成： step 1. 获取数据集 [WikiText2数据集](https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip)是从维基百科上经过验证的优质文章集中提取的超过1亿个token的集合。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/llama_preprocess.py进行数据预处理+Mindrecord数据生成 python llama_preprocess.py \ --dataset_type wiki \ --input_glob /{path}/wiki.valid.tokens \ --model_file /{path}/tokenizer.model \ --seq_length 4095 \ --output_file /{path}/wiki4096.mindrecord ``` step 3. 开启评测，指标为PPL ```bash python run_mindformer.py \ --config configs/llama2/run_llama2_7b.yaml \ --eval_dataset_dir /{path}/wiki4096.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama2_7b.ckpt \ --epochs 1 \ --use_parallel False \ --device_id 0 # PerplexityMetric = {'PerplexityMetric': {'loss': 2.1142693907022476, 'PPL': 8.283531529594038}} ``` - 阅读理解： step 1. 获取数据集 [SQuAD 1.1](https://data.deepai.org/squad1.1.zip)包含针对500+文章的10万+问答对,是一个阅读理解数据集，由维基百科文章上提出的问题组成，其中每个问题的答案都是相应文章中的一段文本。 step 2. 处理数据成mindrecord格式 ```bash # 使用tools/dataset_preprocess/llama/squad_data_process.py进行数据预处理+Mindrecord数据生成 python squad_data_process.py \ --input_file /{path}/squad/dev-v1.1.json \ --output_file /{path}/squad2048.mindrecord \ --mode eval \ --max_length 2048 \ --tokenizer_type "llama2_7b" ``` 预处理后数据格式举例： ```text Read the passage and answer the question below. ### Instruction: The Panthers finished the regular season with a 15–1 record, and quarterback Cam Newton was named the NFL Most Valuable Player (MVP). They defeated the Arizona Cardinals 49–15 in the NFC Championship Game and advanced to their second Super Bowl appearance since the franchise was founded in 1995. The Broncos finished the regular season with a 12–4 record, and denied the New England Patriots a chance to defend their title from Super Bowl XLIX by defeating them 20–18 in the AFC Championship Game. They joined the Patriots, Dallas Cowboys, and Pittsburgh Steelers as one of four teams that have made eight appearances in the Super Bowl. ### Input: Which Carolina Panthers player was named Most Valuable Player? ### Response: Cam Newton ``` step 3. 修改配置文件，eval_dataset的input_columns中增加`labels`，修改metric类型为`EmF1Metric`，修改seq_length为`2048` ```python # eval dataset eval_dataset: &eval_dataset data_loader: type: MindDataset dataset_dir: "" shuffle: False input_columns: ["input_ids", "labels"] # 增加"labels" # metric metric: type: EmF1Metric # metric type设为EmF1Metric # model config model: model_config: type: LlamaConfig batch_size: 1 # add for increase predict seq_length: 2048 ``` 此外，要提高推理速度，可以进行如下配置，设置增量推理`use_past`，并限制生成最大长度`max_new_tokens`。 ```python # model config use_past: True # 开启增量推理 pretrain_seqlen: 4096 extend_method: "None" offset: 0 checkpoint_name_or_path: "llama2_7b" repetition_penalty: 1 max_decode_length: 512 top_k: 3 top_p: 1 do_sample: False max_new_tokens: 20 #设置最大生成长度 ``` step 4. 开启评测，指标为`Em/F1` ```bash python run_mindformer.py \ --config configs/llama2/run_llama2_7b.yaml \ --eval_dataset_dir /{path}/squad2048.mindrecord \ --run_mode eval \ --load_checkpoint /{path}/llama2_7b.ckpt \ --epochs 1 \ --batch_size 1 \ --use_parallel False \ --device_id 0 # F1 score: 48.48954955952303, Em score: 26.850507982583455, total_count: 2067 ``` pengcheng.mind-mindformer/docs/model_cards/mae.md000066400000000000000000000110631476650141400224030ustar00rootroot00000000000000# MAE ## 模型描述 MAE是一种基于MIM（Masked Imange Modeling）的无监督学习方法。 MAE由何凯明团队提出，将NLP领域大获成功的自监督预训练模式用在了计算机视觉任务上，效果拔群，在NLP和CV两大领域间架起了一座更简便的桥梁。 [论文](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2111.06377): He, Kaiming et al. “Masked Autoencoders Are Scalable Vision Learners.” 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022): 15979-15988. ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode train # predict python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import ViTMAEForPreTraining, ViTMAEConfig ViTMAEForPreTraining.show_support_list() # 输出： # - support list of ViTMAEForPreTraining is: # - ['mae_vit_base_p16'] # - ------------------------------------- # 模型标志加载模型 model = ViTMAEForPreTraining.from_pretrained("mae_vit_base_p16") #模型配置加载模型 config = ViTMAEConfig.from_pretrained("mae_vit_base_p16") # {'decoder_dim': 512, 'patch_size': 16, 'in_chans': 3, 'embed_dim': 768, 'depth': 12, # ..., 'decoder_embed_dim': 512, 'norm_pixel_loss': True, 'window_size': None} model = ViTMAEForPreTraining(config) ``` - Trainer接口开启训练/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 mae_trainer = Trainer( task='masked_image_modeling', model='mae_vit_base_p16', train_dataset="imageNet-1k/train") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 重头开始训练，并使用训练好的权重进行推理 mae_trainer.train() # 开启训练 predict_result = mae_trainer.predict(predict_checkpoint=True, input_data=img) print(predict_result) # 方式3： 从obs下载训练好的权重并进行推理 predict_result = mae_trainer.predict(input_data=img) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("masked_image_modeling", model='mae_vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img) ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:---------------------:|:----------------:| | masked_image_modeling | mae_vit_base_p16 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:----------------:|:--------------------:|:-----------:|:-------------:|:---:|:--------------------------------------------------:|:-----------------------------------------------------:| | mae | mae_vit_base_p16 | [mae_vit_base_p16]() | ImageNet-1K | 83.71% | \ | [link](../../configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | [link](../../configs/vit/run_vit_base_p16_100ep.yaml) | ## 模型权重 本仓库中的`mae_vit_base_p16`来自于facebookresearch/mae的[`ViT-Base`](https://dl.fbaipublicfiles.com/mae/pretrain/mae_pretrain_vit_base.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-Base`的模型权重 2. 执行转换脚本，得到转换后的输出文件`mae_vit_base_p16.ckpt` ```python python mindformers/models/mae/convert_weight.py --torch_path "PATH OF ViT-Base.pth" --mindspore_path "SAVE PATH OF mae_vit_base_p16.ckpt" ```pengcheng.mind-mindformer/docs/model_cards/pangualpha.md000066400000000000000000000250601476650141400237630ustar00rootroot00000000000000# PanguAlpha ## 模型描述 「鹏程·盘古α」由以鹏城实验室为首的技术团队联合攻关，首次基于“鹏城云脑Ⅱ”和国产MindSpore框架的自动混合并行模式实现在2048卡算力集群上的大规模分布式训练，训练出业界首个2000亿参数以中文为核心的预训练生成语言模型。鹏程·盘古α预训练模型支持丰富的场景应用，在知识问答、知识检索、知识推理、阅读理解等文本生成领域表现突出，具备很强的小样本学习能力。 [论文](https://arxiv.org/abs/2104.12369)J Wei Zeng, Xiaozhe Ren, Teng Su，et al., PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation, 2021 ## 数据集准备 以悟道数据集为例 - 数据集下载：[悟道数据集](https://data.baai.ac.cn/details/WuDaoCorporaText#a2) - 词表下载：[model.vocab](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PanGu-Alpha/src/branch/master/tokenizer/vocab.model) - 参考[ModelZoo](https://gitee.com/mindspore/models/tree/master/official/nlp/Pangu_alpha#%E6%95%B0%E6%8D%AE%E9%9B%86%E7%94%9F%E6%88%90)，将数据处理成Mindrecord格式。注：训练数据处理时，长度应等于模型接收长度加一 ```bash # 数据预处理示例代码，代码来源于ModelZoo # 生成Mindrecord数据，其中output_file需以字符串mindrecord结尾 python -m preprocess.py --input_glob 'data/*.txt' --tokenizer jieba --eot 40000 --data_column_name input_ids --seq_length 1025 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行2.6b大小的pangualpha模型训练 #### 单卡启动 ```shell python run_mindformer.py --config configs/pangualpha/run_pangualpha_2_6b.yaml \ --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wudao-mindrecord ``` 其中`device_target`根据用户的运行设备不同，可选`GPU/Ascend`。另，模型和训练等相关配置可在`configs/pangualpha`目录下的yaml文件中配置。 #### 单机多卡启动 - 运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； ```shell # step1：机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：# 执行运行脚本：8卡分布式运行， DEVICE_RANGE = [0, 8]， 不包含8本身。 cd scripts bash run_distribute.sh RANK_TABLE_FILE CONFIG_PATH DEVICE_RANGE RUN_STATUS ``` ```python # RANK_TABLE_FILE 参考样例 # 单机8卡 { "version": "1.0", "server_count": "1", "server_list": [ { "server_id": "10.155.111.140", "device": [ {"device_id": "0","device_ip": "192.1.27.6","rank_id": "0"}, {"device_id": "1","device_ip": "192.2.27.6","rank_id": "1"}, {"device_id": "2","device_ip": "192.3.27.6","rank_id": "2"}, {"device_id": "3","device_ip": "192.4.27.6","rank_id": "3"}, {"device_id": "4","device_ip": "192.1.27.7","rank_id": "4"}, {"device_id": "5","device_ip": "192.2.27.7","rank_id": "5"}, {"device_id": "6","device_ip": "192.3.27.7","rank_id": "6"}, {"device_id": "7","device_ip": "192.4.27.7","rank_id": "7"}], "host_nic_ip": "reserve" } ], "status": "completed" } ``` ```text # 参数说明 RANK_TABLE_FILE: 由mindformers/tools/hccl_tools.py生成的分布式json文件 CONFIG_PATH: 为configs文件夹下面的pangualpha/run_pangualpha_*.yaml配置文件 DEVICE_ID: 为设备卡，范围为0~7 DEVICE_RANGE: 为单机分布式卡的范围，如[0,8]为8卡分布式，不包含8本身 RUN_STATUS: 为任务运行状态，支持关键字 train\finetune\predict ``` 其中，模型和训练等相关配置可在`configs/pangualpha`目录下的yaml文件中配置，如数据集路径，可在`configs/pangualpha/run_pangualpha_*.yaml`中配置`dataset_dir`参数。 #### 多机多卡启动 - 首先参考单机多卡启动方式，在每台机器上运行mindformers/tools/hccl_tools.py生成RANK_TABLE_FILE的json文件； - 执行merge_hccl.py脚本将不同机器上生成的RANK_TABLE_FILE文件中的hccl*.json进行合并，包括server_list合并，server_count设为机器数，rank_id顺序增加，并保证不同机器上的RANK_TABLE_FILE相同； - 在多机上同时拉起任务，每台机器拉起方式参考单机多卡启动方式，需注意的是，多机多卡的拉起方式，相对于单机多卡，多了一个总卡数`[RANK_SIZE]`的入参。 ```shell # step1：在每个机器上运行如下命令，生成各自的RANK_TABLE_FILE的json文件。 python ./mindformers/tools/hccl_tools.py --device_num "[0,8)" # step2：运行如下命令，合并每个机器上的RANK_TABLE_FILE文件。 python ./mindformers/tools/merge_hccl.py hccl*.json # step3：将step2得到的合并后的RANK_TABLE_FILE文件分别复制到所有的机器上。 # step4：根据服务器节点数等信息，修改相应的配置 ''' 以pangualpha-13b模型两机训练为例，默认配置2机16卡，如果节点数有变，需要修改相应的配置。配置文件在../configs/pangualpha/run_pangualpha_13b.yaml parallel_config: data_parallel: 4 model_parallel: 4 pipeline_stage: 1 optimizer_shard: True micro_batch_num: 1 vocab_emb_dp: True gradient_aggregation_group: 4 ''' # step5：执行运行脚本 # 第一台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the first device} ../configs/pangualpha/run_pangualpha_13b.yaml [0,8] train 32 # 第二台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the second device} ../configs/pangualpha/run_pangualpha_13b.yaml [8,16] train 32 # 第三台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the third device} ../configs/pangualpha/run_pangualpha_13b.yaml [16,24] train 32 # 第四台机器 bash run_distribute.sh {RANK_TABLE_FILE path of the forth device} ../configs/pangualpha/run_pangualpha_13b.yaml [24,32] train 32 ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import PanguAlphaHeadModel, PanguAlphaTokenizer model = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') model.set_train(False) tokenizer = PanguAlphaTokenizer.from_pretrained('pangualpha_2_6b') inputs = tokenizer(["今天天气很好"], padding='max_length', max_length=model.config.seq_length, return_tensors='ms') output = model(inputs["input_ids"]) print(output) # 计算输出的logits model.set_train(True) inputs = tokenizer(["今天天气很好"], padding='max_length', max_length=model.config.seq_length+1, return_tensors='ms') output = model(inputs["input_ids"]) print(output) # 计算loss ``` - Trainer接口开启训练/推理： ```python from mindformers.trainer import Trainer # 方式1: 开启训练，并使用训练完成后的权重进行推理 trainer = Trainer(task='text_generation', model='pangualpha_2_6b', train_dataset="your data file path") trainer.train() res = trainer.predict(predict_checkpoint=True, input_data="我喜欢北京，因为") # 方式2： 从obs下载训练好的权重并进行推理 trainer = Trainer(task='text_generation', model='pangualpha_2_6b') res = trainer.predict(input_data="我喜欢北京，因为") ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("text_generation", model='pangualpha_2_6b', max_length=50) pipeline_result = pipeline_task("我喜欢北京，因为", top_k=3) print(pipeline_result) ``` **注：**，要提高推理速度，可在`config`yaml文件中进行如下配置，设置增量推理`use_past`为True。 ```python # model config use_past: True # 开启增量推理 use_moe: False expert_num: 1 per_token_num_experts_chosen: 1 checkpoint_name_or_path: "pangualpha_2_6b" repetition_penalty: 1 max_decode_length: 1024 top_k: 3 top_p: 1 do_sample: False ``` ## 模型权重下载 [盘古Alpha权重下载](https://openi.pcl.ac.cn/PCL-Platform.Intelligence/PanGu-Alpha) ## 评测 盘古α支持TNEWS和CMRC2018两个中文数据集的评测。 - 文本分类： - 获取数据集: - [TNEWS数据集](https://storage.googleapis.com/cluebenchmark/tasks/tnews_public.zip)自今日头条的新闻版块，共提取了15个类别的新闻，包括旅游，教育，金融，军事等。 - 处理数据成mindrecord格式 - TNEWS： ```bash # 注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/pangualpha python tnews_data_process.py --input_file {your_path/dev.json} \ --label_file {your_path/labels.json} \ --output_file {your_path/tnews.mindrecord} ``` - 开启评测：评测指标为ACC - TNEWS ```bash python run_mindformers.py --config configs/pangualpha/run_pangualpha_2_6b_prompt_txtcls.yaml \ --eval_dataset_dir {your_path/tnews.mindrecord} \ --run_mode eval # ACC: 0.646, total_acc_num: 6458, total_num: 10000 ``` - 阅读理解： - 获取数据集: - [CMRC2018数据集](https://storage.googleapis.com/cluebenchmark/tasks/cmrc2018_public.zip)是用于中文机器阅读理解的片段抽取任务(Span-Extraction)的数据，这个数据集由近20000个真实的问题组成，这些问题由人类专家在维基百科的段落中注释。 - 处理数据成mindrecord格式 - CMRC2018： ```bash # 注：生成的数据集文件需以.mindrecord结尾 cd mindformers/tools/dataset_preprocess/pangualpha python wikitext2_data_process.py --train_file {your_path/train.json} \ --dev_file {your_path/dev.json} \ --output_file {your_path/cmrc2018.mindrecord} ``` - 开启评测：评测指标为Em/F1 - CMRC2018 ```bash python run_mindformers.py --config configs/pangualpha/run_pangualpha_2_6b_em_f1.yaml \ --eval_dataset_dir {your_path/cmrc2018.mindrecord} \ --run_mode eval # F1 score: 21.12, Em score: 2.10, total_count: 1806 ``` pengcheng.mind-mindformer/docs/model_cards/swin.md000066400000000000000000000132241476650141400226220ustar00rootroot00000000000000# SWIN ## 模型描述 swin：全名swin transformer，是一个基于Transformer在视觉领域有着SOTA表现的深度学习模型。比起VIT拥有更好的性能和精度。 [论文](https://arxiv.org/abs/2103.14030) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, 2021 ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```bash 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode train --train_dataset_dir [DATASET_PATH] # evaluate python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import SwinForImageClassification, SwinConfig SwinForImageClassification.show_support_list() # 输出： # - support list of SwinForImageClassification is: # - ['swin_base_p4w7'] # - ------------------------------------- # 模型标志加载模型 model = SwinForImageClassification.from_pretrained("swin_base_p4w7") #模型配置加载模型 config = SwinConfig.from_pretrained("swin_base_p4w7") # {'batch_size': 128, 'image_size': 224, 'patch_size': 4, 'num_labels': 1000, 'num_channels': 3, # 'embed_dim': 128, 'depths': [2, 2, 18, 2], 'num_heads': [4, 8, 16, 32], # 'checkpoint_name_or_path': 'swin_base_p4w7'} model = SwinForImageClassification(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 swin_trainer = Trainer( task='image_classification', model='swin_base_p4w7', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：开启训练，并使用训练好的权重进行eval和推理 swin_trainer.train() swin_trainer.evaluate(eval_checkpoint=True) predict_result = swin_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2： 从obs下载训练好的权重并进行eval和推理 swin_trainer.evaluate() # 下载权重进行评估 predict_result = swin_trainer.predict(input_data=img, top_k=3) # 下载权重进行推理 print(predict_result) # 输出 # - mindformers - INFO - output result is: [[{'score': 0.89573187, 'label': 'daisy'}, # {'score': 0.005366202, 'label': 'bee'}, {'score': 0.0013296203, 'label': 'fly'}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='swin_base_p4w7') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) print(pipeline_result) # 输出 # [[{'score': 0.89573187, 'label': 'daisy'}, {'score': 0.005366202, 'label': 'bee'}, # {'score': 0.0013296203, 'label': 'fly'}]] ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:--------------:| :-------------: | | image_classification | swin_base_p4w7 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:--------------:|:--------:| :----: |:-------------:| :---: |:---------------:|:--------------------------------------------------------:| | swin | swin_base_p4w7 | \ | ImageNet-1K | 83.44% | \ | \ | [link](../../configs/swin/run_swin_base_p4w7_100ep.yaml) | ## 模型权重 本仓库中的`swin_base_p4w7`来自于MicroSoft的[`Swin-Transformer`](https://github.com/microsoft/Swin-Transformer), 基于下述的步骤获取： 1. 从上述的链接中下载[`swin_base_p4w7`](https://pan.baidu.com/s/16bqCTEc70nC_isSsgBSaqQ?pwd=swin)的官方权重，文件名为`swin_base_patch4_window7_224.pth` 2. 执行转换脚本，得到转换后的输出文件`swin_base_p4w7.ckpt` ```shell python mindformers/models/swin/convert_weight.py --torch_path swin_base_patch4_window7_224.pth --mindspore_path swin_base_p4w7.ckpt --is_pretrain False ``` 如需转换官方simmim的预训练权重进行finetune，则执行如下步骤： 1. [从simmim官网](https://github.com/microsoft/SimMIM)提供的google网盘下载[`simmim_swin_192`](https://drive.google.com/file/d/1Wcbr66JL26FF30Kip9fZa_0lXrDAKP-d/view?usp=sharing)的官方权重，文件名为`simmim_pretrain__swin_base__img192_window6__100ep.pth` 2. 执行转换脚本，得到转换后的输出文件`simmim_swin_p4w6.ckpt` ```shell python mindformers/models/swin/convert_weight.py --torch_path simmim_pretrain__swin_base__img192_window6__100ep.pth --mindspore_path simmim_swin_p4w6.ckpt --is_pretrain True ```pengcheng.mind-mindformer/docs/model_cards/t5.md000066400000000000000000000101341476650141400221670ustar00rootroot00000000000000# T5 ## 模型描述 T5:全名`Text-to-Text Transfer Transformer`模型是谷歌在2019年基于C4数据集训练的Transformer模型。 [论文](https://arxiv.org/abs/1910.10683)C Raffel，N Shazeer，A Roberts，K Lee，S Narang，M Matena，Y Zhou，W Li，PJ Liu, 2020 ## 数据集准备 使用的数据集：[WMT16](https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz) 对应的文件路径如下： ```bash └── wmt_en_ro ├── test.source ├── test.target ├── train.source ├── train.target ├── val.source └── val.target ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) 示例命令如下，将会执行一个只有1层的T5模型训练 ```shell python run_mindformer.py --config configs/t5/run_t5_tiny_on_wmt16.yaml --run_mode train \ --device_target Ascend \ --train_dataset_dir /your_path/wmt_en_ro ``` 其中`device_target`根据用户的运行设备不同，可选`GPU/Ascend/CPU`。`config`的入参还可以为`configs/t5/run_t5_small.yaml`，在 这个配置下将会加载`t5_small`的权重并且开始执行微调。 ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) #### Model调用接口 - 模型计算Loss ```python from mindformers import T5ForConditionalGeneration, T5Tokenizer model = T5ForConditionalGeneration.from_pretrained('t5_small') tokenizer = T5Tokenizer.from_pretrained('t5_small') src_output = tokenizer(["hello world"], padding='max_length', max_length=model.config.seq_length, return_tensors='ms') model_input = tokenizer(["So happy to see you!"], padding='max_length', max_length=model.config.max_decode_length, return_tensors='ms')["input_ids"] input_ids = src_output['input_ids'] attention_mask = src_output['attention_mask'] output = model(input_ids, attention_mask, model_input) print(output) # [5.64458] ``` - 推理 执行下述的命令，可以自动云上拉取`t5_small`模型并且进行推理。 ```python from mindformers import T5ForConditionalGeneration, T5Tokenizer t5 = T5ForConditionalGeneration.from_pretrained("t5_small") tokenizer = T5Tokenizer.from_pretrained("t5_small") words = tokenizer("translate the English to the Romanian: UN Chief Says There Is No Military " "Solution in Syria")['input_ids'] output = t5.generate(words, do_sample=False) output = tokenizer.decode(output, skip_special_tokens=True) print(output) # "eful ONU declară că nu există o soluţie militară în Siri" ``` - Trainer接口开启训练/预测： ```python from mindformers.trainer import Trainer # 初始化预训练任务 trainer = Trainer(task='translation', model='t5_small', train_dataset="your data file path") # 方式1: 开启训练，并使用训练好的权重进行推理 trainer.train() res = trainer.predict(predict_checkpoint=True, input_data="translate the English to Romanian: a good boy!") print(res) #[{'translation_text': ['un băiat bun!']}] # 方式2： 从obs下载训练好的权重并进行推理 res = trainer.predict(input_data="translate the English to Romanian: a good boy!") print(res) #[{'translation_text': ['un băiat bun!']}] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline pipeline_task = pipeline("translation", model='t5_small') pipeline_result = pipeline_task("translate the English to Romanian: a good boy!", top_k=3) print(pipeline_result) #[{'translation_text': ['un băiat bun!']}] ``` ## 模型权重 本仓库中的`t5_small`来自于HuggingFace的[`t5_small`](https://huggingface.co/t5-small), 基于下述的步骤获取： 1. 从上述的链接中下载`t5_small`的HuggingFace权重，文件名为`pytorch_model.bin` 2. 执行转换脚本，得到转换后的输出文件`mindspore_t5.ckpt` ```shell python mindformers/models/t5/convert_weight.py --layers 6 --torch_path pytorch_model.bin --mindspore_path ./mindspore_t5.ckpt ``` pengcheng.mind-mindformer/docs/model_cards/vit.md000066400000000000000000000124171476650141400224470ustar00rootroot00000000000000# ViT ## 模型描述 vit：全名vision transformer，不同于传统的基于CNN的网络结果，是基于transformer结构的cv网络，2021年谷歌研究发表网络，在大数据集上表现了非常强的泛化能力。大数据任务（如clip）基于该结构能有良好的效果。mindformers提供的Vit权重及精度均是是基于MAE预训练ImageNet-1K数据集进行微调得到。 [论文](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2010.11929): Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 2021. ## 数据集准备 使用的数据集：[ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速使用 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # pretrain python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode train # evaluate python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode eval --eval_dataset_dir [DATASET_PATH] # predict python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode predict --predict_data [PATH_TO_IMAGE] ``` ### 调用API启动 > 需开发者提前pip安装。具体接口说明请参考[API接口](https://gitee.com/mindspore/transformer/wikis/API/) - Model调用接口 ```python from mindformers import ViTForImageClassification, ViTConfig ViTForImageClassification.show_support_list() # 输出： # - support list of ViTForImageClassification is: # - ['vit_base_p16'] # - ------------------------------------- # 模型标志加载模型 model = ViTForImageClassification.from_pretrained("vit_base_p16") #模型配置加载模型 config = ViTConfig.from_pretrained("vit_base_p16") # {'patch_size': 16, 'in_chans': 3, 'embed_dim': 768, 'depth': 12, 'num_heads': 12, 'mlp_ratio': 4, # ..., 'batch_size': 32, 'image_size': 224, 'num_classes': 1000} model = ViTForImageClassification(config) ``` - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 初始化任务 vit_trainer = Trainer( task='image_classification', model='vit_base_p16', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 vit_trainer.train(resume_or_finetune_from_checkpoint="mae_vit_base_p16", do_finetune=True) vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 重头开始训练，并使用训练好的权重进行eval和推理 vit_trainer.train() vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式3： 从obs下载训练好的权重并进行eval和推理 vit_trainer.evaluate() predict_result = vit_trainer.predict(input_data=img, top_k=3) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) ``` Trainer和pipeline接口默认支持的task和model关键入参 | task（string） | model（string） | |:--------------------:|:-------------:| | image_classification | vit_base_p16 | ## 模型性能 | model | type | pretrain | Datasets | Top1-Accuracy | Log | pretrain_config | finetune_config | |:-----:|:------------:|:--------------------:|:-----------:|:-------------:|:---:|:--------------------------------------------------:|:-----------------------------------------------------:| | vit | vit_base_p16 | [mae_vit_base_p16]() | ImageNet-1K | 83.71% | \ | [link](../../configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | [link](../../configs/vit/run_vit_base_p16_100ep.yaml) | ## 模型权重 本仓库中的`vit_base_p16`来自于facebookresearch/mae的[`ViT-Base`](https://dl.fbaipublicfiles.com/mae/finetune/mae_finetuned_vit_base.pth), 基于下述的步骤获取： 1. 从上述的链接中下载`ViT-Base`的模型权重 2. 执行转换脚本，得到转换后的输出文件`vit_base_p16.ckpt` ```shell python mindformers/models/vit/convert_weight.py --torch_path "PATH OF ViT-Base.pth" --mindspore_path "SAVE PATH OF vit_base_p16.ckpt" ```pengcheng.mind-mindformer/docs/model_support_list.md000066400000000000000000000736561476650141400233340ustar00rootroot00000000000000# 模型支持列表 ## NLP ### masked_language_modeling | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :--------------------------------------: | :-----------------: | :------------------: | :-----------------: | :--------------------------------------------------------------------------------------------------------------------: | | [bert_base_uncased](model_cards/bert.md) | wiki | - | - | [run_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bert/run_bert_base_uncased.yaml) | ### [text_classification](task_cards/text_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :-----------------------------------------------------------------------------------------------------------------------------------: | :-----------------: | :----------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [txtcls_bert_base_uncased](task_cards/text_classification.md)
[txtcls_bert_base_uncased_mnli](task_cards/text_classification.md) | Mnli
Mnli | Entity F1
Entity F1 | -
84.80% | [run_txtcls_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/txtcls/run_txtcls_bert_base_uncased.yaml)
[run_txtcls_bert_base_uncased_mnli.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml) | ### [token_classification](task_cards/token_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------------------: | :------------------: | :----------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [tokcls_bert_base_chinese](task_cards/token_classification.md)
[tokcls_bert_base_chinese_cluener](task_cards/token_classification.md) | CLUENER
CLUENER | Entity F1
Entity F1 | -
0.7905 | [run_tokcls_bert_base_chinese.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/tokcls/run_tokcls_bert_base_chinese.yaml)
[run_tokcls_bert_base_chinese_cluener.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml) | ### [question_answering](task_cards/question_answering.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------: | :------------------------: | :------------------: | :------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [qa_bert_base_uncased](task_cards/question_answering.md)
[qa_bert_base_chinese_uncased](task_cards/question_answering.md) | SQuAD v1.1
SQuAD v1.1 | EM / F1
EM / F1 | 80.74 / 88.33
- | [run_qa_bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/qa/run_qa_bert_base_uncased.yaml)
[run_qa_bert_base_chinese_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/qa/run_qa_bert_base_chinese_uncased.yaml) | ### translation | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :---------------------------: | :-----------------: | :------------------: | :-----------------: | :------------------------------------------------------------------------------------------------------------------: | | [t5_small](model_cards/t5.md) | WMT16 | - | - | [run_t5_small_on_wmt16.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/t5/run_t5_small_on_wmt16.yaml) | ### [text_generation](task_cards/text_generation.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :----------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------: | :------------------------------------------: | :----------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [bloom_560m](model_cards/bloom.md)
[bloom_7.1b](model_cards/bloom.md)
[bloom_65b](model_cards/bloom.md)
[bloom_176b](model_cards/bloom.md) | alpaca
alpaca
alpaca
alpaca | -
-
-
- | -
-
-
- | [run_bloom_560m.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_560m.yaml)
[run_bloom_7.1b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_7.1b.yaml)
[run_bloom_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_65b.yaml)
[run_bloom_176b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bloom/run_bloom_176b.yaml) | | [glm_6b](model_cards/glm.md)
[glm_6b_lora](model_cards/glm.md) | ADGEN
ADGEN | BLEU-4 / Rouge-1 / Rouge-2 / Rouge-l
- | 8.42 / 31.75 / 7.98 / 25.28
- | [run_glm_6b_finetune.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm/run_glm_6b_finetune.yaml)
[run_glm_6b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm/run_glm_6b_lora.yaml) | | [glm2_6b](model_cards/glm2.md)
[glm2_6b_lora](model_cards/glm2.md) | ADGEN
ADGEN | BLEU-4 / Rouge-1 / Rouge-2 / Rouge-l
- | 7.47 / 30.78 / 7.07 / 24.77
7.23 / 31.06 / 7.18 / 24.23 | [run_glm2_6b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm2/run_glm2_6b.yaml)
[run_glm2_6b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/glm2/run_glm2_6b_lora.yaml) | | [gpt2_small](model_cards/gpt2.md)
[gpt2_13b](model_cards/gpt2.md)
[gpt2_52b](model_cards/gpt2.md) | wikitext-2
wikitext-2
wikitext-2 | -
-
- | -
-
- | [run_gpt2.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2.yaml)
[run_gpt2_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_13b.yaml)
[run_gpt2_52b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/gpt2/run_gpt2_52b.yaml) | | [llama_7b](model_cards/llama.md)
[llama_13b](model_cards/llama.md)
[llama_65b](model_cards/llama.md)
[llama_7b_lora](model_cards/llama.md) | alpaca
alpaca
alpaca
- | -
-
-
- | -
-
-
- | [run_llama_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b.yaml)
[run_llama_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_13b.yaml)
[run_llama_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_65b.yaml)
[run_llama_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b_lora.yaml) | | [llama_7b](model_cards/llama.md)
[llama_13b](model_cards/llama.md)
[llama_65b](model_cards/llama.md)
[llama_7b_lora](model_cards/llama.md) | alpaca
alpaca
alpaca
- | -
-
-
- | -
-
-
- | [run_llama_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b.yaml)
[run_llama_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_13b.yaml)
[run_llama_65b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_65b.yaml)
[run_llama_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/llama/run_llama_7b_lora.yaml) | | [pangualpha_2_6_b](model_cards/pangualpha.md)
[pangualpha_13b](model_cards/pangualpha.md) | 悟道数据集
悟道数据集 | -
- | -
- | [run_pangualpha_2_6b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/pangualpha/run_pangualpha_2_6b.yaml)
[run_pangualpha_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/pangualpha/run_pangualpha_13b.yaml) | | [baichuan_7b](../research/baichuan/baichuan.md)
[baichuan_13b](../research/baichuan/baichuan.md) | -
- | -
- | -
- | [run_baichuan_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_baichuan_7b.yaml)
[run_baichuan_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_baichuan_13b.yaml) | | [baichuan2_7b](../research/baichuan2/baichuan2.md)
[baichuan2_13b](../research/baichuan2/baichuan2.md) | -
- | -
- | -
- | [run_baichuan2_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/run_baichuan2_7b.yaml)
[run_baichuan2_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan2/run_baichuan2_13b.yaml) | | [internlm_7b](../research/internlm/internlm.md)
[internlm_7b_lora](../research/internlm/internlm.md) | wikitext-2
alpaca | -
- | -
- | [run_internlm_7b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm/run_internlm_7b.yaml)
[run_internlm_7b_lora.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/internlm/run_internlm_7b_lora.yaml) | | [ziya_13b](../research/ziya/ziya.md) | -
- | -
- | -
- | [run_ziya_13b.yaml](https://gitee.com/mindspore/mindformers/blob/dev/research/baichuan/run_ziya_13b.yaml) | ## CV ### masked_image_modeling | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :------------------------------------: | :-----------------: | :------------------: | :-----------------: | :-------------------------------------------------------------------------------------------------------------------------------------: | | [mae_vit_base_p16](model_cards/mae.md) | ImageNet-1k | - | - | [run_mae_vit_base_p16_224_800ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/mae/run_mae_vit_base_p16_224_800ep.yaml) | ### [image_classification](task_cards/image_classification.md) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :-----------------------------------: | :-----------------: | :------------------: | :-----------------: | :----------------------------------------------------------------------------------------------------------------------------------: | | [vit_base_p16](model_cards/vit.md) | ImageNet-1k | Accuracy | 83.71% | [run_vit_base_p16_224_100ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/vit/run_vit_base_p16_224_100ep.yaml) | | [swin_base_p4w7](model_cards/swin.md) | ImageNet-1k | Accuracy | 83.44% | [run_swin_base_p4w7_224_100ep.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/swin/run_swin_base_p4w7_224_100ep.yaml) | ## Multi-Modal ### [zero_shot_image_classification](task_cards/zero_shot_image_classification.md) (by [contrastive_language_image_pretrain](task_cards/contrastive_language_image_pretrain.md)) | 模型
model | 数据集
dataset | 评估指标
metric | 评估得分
score | 配置
config | | :---------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------: | :---------------------------------------------------------: | :----------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | [clip_vit_b_32](model_cards/clip.md)
[clip_vit_b_16](model_cards/clip.md)
[clip_vit_l_14](model_cards/clip.md)
[clip_vit_l_14@336](model_cards/clip.md) | Cifar100
Cifar100
Cifar100
Cifar100 | Accuracy
Accuracy
Accuracy
Accuracy | 57.24%
61.41%
69.67%
68.19% | [run_clip_vit_b_32_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml)
[run_clip_vit_b_16_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml)
[run_clip_vit_l_14_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml)
[run_clip_vit_l_14@336_pretrain_flickr8k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml) | | [blip2_vit_g](model_cards/blip2.md) | -
flickr30k
- | -
ITM
- | -
-
- | [run_blip2_vit_g_qformer_pretrain.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_qformer_pretrain.yaml)
[run_blip2_vit_g_retrieval_flickr30k.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_retrieval_flickr30k.yaml)
[run_blip2_vit_g_zero_shot_image_classification_cifar100.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/blip2/run_blip2_vit_g_zero_shot_image_classification_cifar100.yaml) | ## 模型能力支持度 ### 核心关键模型能力一览表 | 关键模型 | 并行模式 | 数据并行 | 优化器并行 | 模型并行 | 流水并行 | 多副本并行 | 预训练 | 微调 | 评估 | 推理 | | :------: | :------------------------------: | :------: | :--------: | :------: | :------: | :--------: | ------ | :----------------: | :------------: | ---: | | Bloom | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | 不支持 | 推理 | | GLM | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | Blue/Rouge评估 | 推理 | | GLM2 | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | Blue/Rouge评估 | 推理 | | GPT | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | | LLaMa | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调，Lora微调 | PPL评估 | 推理 | | LLaMa2 | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | | PanGu | data_parallel\semi_auto_parallel | 是 | 是 | 是 | 是 | 是 | 是 | 全参微调 | PPL评估 | 推理 | ### Research模型支持情况一览表 | 模型 | 任务（task name） | 模型（model name） | | :--------------------------------------------: | :----------------------------------------------------------------------------------------------------: | :------------------------------ | | [Baichuan](../research/baichuan/baichuan.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | baichuan_7b
baichuan_13b | | [Baichuan2](../research/baichuan2/baichuan2.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | baichuan2_7b
baichuan2_13b | | [Internlm](../research/internlm/internlm.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | InternLM-7B | | [ziya](../research/ziya/ziya.md) | [text_generation](https://gitee.com/mindspore/mindformers/blob/dev/docs/task_cards/text_generation.md) | ziya-13B | ### Text Generator支持度表 | model | 模型文档链接 | 增量推理 | 流式推理 | | :---------: | :-------------------------------------------: | :------: | :------: | | bloom | [link](../model_cards/bloom.md) | √ | √ | | GLM | [link](../model_cards/glm.md) | √ | √ | | GLM2 | [link](../model_cards/glm2.md) | √ | √ | | GPT | [link](../model_cards/gpt2.md) | √ | √ | | llama | [link](../model_cards/llama.md) | √ | √ | | llama2 | [link](../model_cards/llama.md) | √ | √ | | pangu-alpha | [link](../model_cards/pangualpha.md) | √ | √ | | T5 | [link](../model_cards/t5.md) | × | √ | | research | research | research | research | | baichuan | [link](../../research/baichuan/baichuan.md) | √ | √ | | baichuan2 | [link](../../research/baichuan2/baichuan2.md) | √ | √ | | internlm | [link](../../research/internlm/internlm.md) | √ | √ | | ziya | [link](../../research/ziya/ziya.md) | √ | √ | ### 边训练边评估支持度表 | 模型 | 评估指标 | 可用Model.eval完成评估 | 是否支持 | 数据并行模式 | 半自动并行模式 | | ----------- | :-------------------: | :--------------------: | -------- | ------------ | :---------------: | | bert | - | - | - | - | - | | blip2 | - | - | - | - | - | | bloom | - | - | - | - | - | | clip | - | - | - | - | - | | filip | - | - | - | - | - | | glm | Rouge，Bleu | 否 | 否 | × | × | | gpt2 | PPL | 是 | 是 | √ | √ | | llama | PPL | 是 | 是 | √ | √（7b 至少8卡） | | llama2 | PPL | 是 | 是 | √ | √（7b 至少8卡） | | MAE | 暂缺 | - | - | - | - | | pangu alpha | PPL | 是 | 是 | √ | √ | | qa-bert | f1, precision, recall | 是 | 是 | √ | × | | swin | Accuracy | 是 | 是 | √ | × | | t5 | 暂缺 | - | - | - | - | | tokcls-bert | f1, precision, recall | 是 | 是 | √ | × | | txtcls-bert | Accuracy | 是 | 是 | √ | × | | vit | Accuracy | 是 | 是 | √ | × | | research | research | research | research | research | research | | baichuan | PPL | 是 | 是 | √ | √（7b 至少8卡） | | baichuan2 | PPL | 是 | 是 | √ | √（7b 至少8卡） | | internlm | PPL | 是 | 是 | √ | √（7b 至少8卡） | | ziya | PPL | 是 | 是 | √ | √（13b 至少16卡） | ### 微调支持列表 | 模型 | 微调算法 | 运行模式 | | :------------------------------: | :------: | :---------------------: | | [GPT2](../model_cards/gpt2.md) | Lora | finetune、eval、predict | | [LLama](../model_cards/llama.md) | Lora | finetune、eval、predict | | [GLM](../model_cards/glm.md) | Lora | finetune、eval、predict | | [GLM2](../model_cards/glm2.md) | Lora | finetune、eval、predict | ### Chat Web支持列表 | 模型 | 规格 | 分词器 | 增量推理 | | ----- | ------------- | ------------- | -------- | | GLM | glm_6b | glm_6b | 支持 | | GLM2 | glm2_6b | glm2_6b | 支持 | | BLOOM | bloom_7.1b | bloom_7.1b | 支持 | | LLAMA | llama_7b_lora | llama_7b_lora | 支持 | ### 其余库上模型分布式支持情况一览表 | 模型 | 并行模式 | 数据并行 | 优化器并行 | 模型并行 | 流水并行 | 多副本并行 | | ----- | ------------- | -------- | ---------- | -------- | -------- | ---------- | | Bert | data_parallel | 是 | 是 | 否 | 否 | 否 | | BLIP2 | data_parallel | 是 | 是 | 否 | 否 | 否 | | CLIP | data_parallel | 是 | 是 | 否 | 否 | 否 | | MAE | data_parallel | 是 | 是 | 否 | 否 | 否 | | Swin | data_parallel | 是 | 是 | 否 | 否 | 否 | | T5 | data_parallel | 是 | 是 | 否 | 否 | 否 | | VIT | data_parallel | 是 | 是 | 否 | 否 | 否 | pengcheng.mind-mindformer/docs/readthedocs/000077500000000000000000000000001476650141400213275ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/Makefile000066400000000000000000000012121476650141400227630ustar00rootroot00000000000000# Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = source_zh_cn BUILDDIR = build_zh_cn # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) pengcheng.mind-mindformer/docs/readthedocs/requirements.txt000066400000000000000000000001001476650141400246020ustar00rootroot00000000000000sphinx docutils myst_parser sphinx_rtd_theme == 0.5.2 mindspore pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/000077500000000000000000000000001476650141400240105ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/_static/000077500000000000000000000000001476650141400254365ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/_static/my_theme.css000066400000000000000000000000661476650141400277610ustar00rootroot00000000000000.wy-nav-content { max-width: 1200px !important; } pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/_templates/000077500000000000000000000000001476650141400261455ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/_templates/classtemplate.rst000066400000000000000000000007371476650141400315470ustar00rootroot00000000000000.. role:: hidden :class: hidden-section .. currentmodule:: {{ module }} {% if objname in [] %} {{ fullname | underline }} .. autofunction:: {{ fullname }} {% elif objname[0].istitle() %} {{ fullname | underline }} .. autoclass:: {{ name }} :exclude-members: construct :members: {% else %} {{ fullname | underline }} .. autofunction:: {{ fullname }} {% endif %} .. autogenerated from _templates/classtemplate.rst note it does not have :inherited-members: pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/conf.py000066400000000000000000000121051476650141400253060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # """Configuration file for the Sphinx documentation builder.""" import os import shutil from sphinx.util import logging # -- Project information ----------------------------------------------------- project = 'mindformers' # pylint: disable=W0622 copyright = '2023, mindformers contributors' author = 'mindformers contributors' # The full version, including alpha/beta/rc tags release = 'dev' # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = ['myst_parser', 'sphinx.ext.autodoc', 'sphinx.ext.viewcode', 'sphinx.ext.autosummary', 'sphinx.ext.intersphinx'] source_suffix = {'.rst': 'restructuredtext', '.md': 'markdown'} # Add any paths that contain templates here, relative to this directory. templates_path = ['_templates'] # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. # This pattern also affects html_static_path and html_extra_path. exclude_patterns = [] pygments_style = 'sphinx' autodoc_inherit_docstrings = False autosummary_generate = True # -- Options for HTML output ------------------------------------------------- # Reconstruction of sphinx auto generated document translation. language = 'zh_CN' gettext_compact = False # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. html_theme = 'sphinx_rtd_theme' html_static_path = ['_static'] # Copy source files of chinese python api from mindformers repository. logger = logging.getLogger(__name__) copy_path = ['../../../README.md', '../../../research', '../../FAQ.md', '../../model_support_list.md', '../../transformer仓Python编程规范.md' '../../aicc_cards', '../../feature_cards', '../../model_cards', '../../task_cards'] # origin path in readthedocs.org is like # /abs/readthedocs.org/user_builds/project_name/checkouts/version/docs/xxx # new path in readthedocs.org is like # /abs/readthedocs.org/user_builds/project_name/checkouts/version/docs/readthedocs/source_zh_cn/(xxx or docs/xxx) # 此处为了在readthedocs中构造出与gitee上同样的目录结构，以达到md文件中相对路径在gitee和readthedos都可以使用的目的 def copy_file_to_docs(ori_name): """copy dir in docs to readthedocs workspace.""" ori_path = os.path.realpath(ori_name) new_name = ori_path.split('/')[-1] if ori_name.endswith('README.md'): new_path = new_name shutil.copy(ori_path, new_path) elif 'research' in ori_name: new_path = new_name shutil.copytree(ori_path, new_path) elif ori_name.endswith('.md'): new_path = os.path.join('docs', new_name) shutil.copy(ori_path, new_path) else: new_path = os.path.join('docs', new_name) shutil.copytree(ori_path, new_path) for file_path in copy_path: copy_file_to_docs(file_path) # split README into 4 parts with open('README.md', 'r') as f: title_list = ['Introduction', 'Install', 'Version_Match', 'Quick_Tour', 'Contribution', 'License'] title_for_index = ['# 一、介绍', '# 二、安装', '# 三、版本配套', '# 四、快速开始', '-', '-'] file_count = 0 fn = None for line in f: if line.startswith('##') and not line.startswith('###'): if fn: fn.close() fn = open(f'{title_list[file_count]}.md', 'w') fn.write(title_for_index[file_count]) file_count += 1 continue if fn: fn.write(line) fn.close() # in _static, set page width 1200 px def setup(app): app.add_css_file('my_theme.css') pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/000077500000000000000000000000001476650141400247405ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/000077500000000000000000000000001476650141400271125ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/README.md000066400000000000000000000123141476650141400303720ustar00rootroot00000000000000# 接口支持度说明 ## Datasets | Dataset API | 说明 | | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | [mindformers.dataset.CausalLanguageModelDataset](mindformers.dataset.CausalLanguageModelDataset) | GPT类模型文本数据集，支持MindRecord，TFRecord，自定义数据集等 | | [mindformers.dataset.ContrastiveLanguageImagePretrainDataset](mindformers.dataset.ContrastiveLanguageImagePretrainDataset) | CLIP类模型预训练图文对数据集，如Flickr8k等 | | [mindformers.dataset.ImageCLSDataset](mindformers.dataset.ImageCLSDataset) | 图片分类数据集，如ImageNet2012等 | | [mindformers.dataset.KeyWordGenDataset](mindformers.dataset.KeyWordGenDataset) | GLM模型文本数据集，支持MindRecord，自定义数据集等，如ADGen | | [mindformers.dataset.MaskLanguageModelDataset](mindformers.dataset.MaskLanguageModelDataset) | Bert类模型文本数据集，仅支持MindRecord，TFRecord | | [mindformers.dataset.MIMDataset](mindformers.dataset.MIMDataset) | Mae等图片数据集，支持MindRecord，ImageNet2012等 | | [mindformers.dataset.TextClassificationDataset](mindformers.dataset.TextClassificationDataset) | 文本分类数据集，仅支持TFRecord | | [mindformers.dataset.TokenClassificationDataset](mindformers.dataset.TokenClassificationDataset) | Token分类文本数据集，支持自定义数据集，如CLUENER | | [mindformers.dataset.ZeroShotImageClassificationDataset](mindformers.dataset.ZeroShotImageClassificationDataset) | 零样本分类图片数据集，支持自定义数据集，如Cifar100 | ## Learning Rate | Learning Rate API | 说明 | | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | [mindformers.core.lr.ConstantWarmUpLR](mindformers.core.lr.ConstantWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后LR固定为LR×常数 | | [mindformers.core.lr.CosineWithRestartsAndWarmUpLR](mindformers.core.lr.CosineWithRestartsAndWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行cos(1)->cos(0) 衰减或周期变化 | | [mindformers.core.lr.CosineWithWarmUpLR](mindformers.core.lr.CosineWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行consine衰减或周期变化 | | [mindformers.core.lr.LinearWithWarmUpLR](mindformers.core.lr.LinearWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后LR从0开始增长到指定LR | | [mindformers.core.lr.PolynomialWithWarmUpLR](mindformers.core.lr.PolynomialWithWarmUpLR) | LR从warmupLR开始线型增长到指定LR，warmup阶段后进行衰减 | ## Loss | Loss API | 说明 | | -------------------------------------------------------------------------------------------- | ------------------------------------ | | [mindformers.core.loss.CrossEntropyLoss](mindformers.core.loss.CrossEntropyLoss) | NLP常用的交叉熵损失函数 | | [mindformers.core.loss.L1Loss](mindformers.core.loss.L1Loss) | 平均绝对误差，SIMMIM所使用的损失函数 | | [mindformers.core.loss.MSELoss](mindformers.core.loss.MSELoss) | 均方误差，Mae所使用的的损失函数 | | [mindformers.core.loss.SoftTargetCrossEntropy](mindformers.core.loss.SoftTargetCrossEntropy) | Swin所使用的的损失函数 | ## Optimizer | Optimizer API | 说明 | | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | | [mindformers.core.optim.FusedAdamWeightDecay](mindformers.core.optim.FusedAdamWeightDecay) | 权重衰减Adam算法的融合算子 | | [mindformers.core.optim.FP32StateAdamWeightDecay](mindformers.core.optim.FP32StateAdamWeightDecay) | 权重衰减Adam算法，与mindspore中一致，区别为优化器状态改为fp32 | pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.core.rst000066400000000000000000000036021476650141400331210ustar00rootroot00000000000000mindformers.core ================== .. automodule:: mindformers.core mindformers.core ----------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.ClipGradNorm mindformers.core.build_context mindformers.core.init_context mindformers.core.callback -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.callback.CheckpointMointor mindformers.core.callback.MFLossMonitor mindformers.core.callback.ObsMonitor mindformers.core.callback.SummaryMonitor mindformers.core.callback.ProfileMonitor mindformers.core.callback.EvalCallBack mindformers.core.loss -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.loss.CrossEntropyLoss mindformers.core.loss.L1Loss mindformers.core.loss.MSELoss mindformers.core.loss.SoftTargetCrossEntropy mindformers.core.lr -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.lr.ConstantWarmUpLR mindformers.core.lr.CosineWithRestartsAndWarmUpLR mindformers.core.lr.CosineWithWarmUpLR mindformers.core.lr.LinearWithWarmUpLR mindformers.core.lr.PolynomialWithWarmUpLR mindformers.core.metric -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.metric.EntityScore mindformers.core.metric.SQuADMetric mindformers.core.metric.PerplexityMetric mindformers.core.metric.ADGENMetric mindformers.core.optim -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.optim.FusedAdamWeightDecay mindformers.core.optim.FP32StateAdamWeightDecay pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.dataset.rst000066400000000000000000000013011476650141400336100ustar00rootroot00000000000000mindformers.dataset ==================== .. automodule:: mindformers.dataset .. autosummary:: :toctree: dataset :nosignatures: :template: classtemplate.rst mindformers.dataset.BaseDataset mindformers.dataset.CausalLanguageModelDataset mindformers.dataset.ContrastiveLanguageImagePretrainDataset mindformers.dataset.ImageCLSDataset mindformers.dataset.ImageToTextRetrievalDataset mindformers.dataset.KeyWordGenDataset mindformers.dataset.MaskLanguageModelDataset mindformers.dataset.MIMDataset mindformers.dataset.TextClassificationDataset mindformers.dataset.TokenClassificationDataset mindformers.dataset.ZeroShotImageClassificationDataset pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.models.rst000066400000000000000000000102141476650141400334510ustar00rootroot00000000000000mindformers.models =================== .. automodule:: mindformers.models mindformers.models ------------------- .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.BaseConfig mindformers.models.BaseImageProcessor mindformers.models.BaseModel mindformers.models.BaseProcessor mindformers.models.Tokenizer mindformers.models.bert ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bert.BertConfig mindformers.models.bert.BertModel mindformers.models.bert.BertForMultipleChoice mindformers.models.bert.BertForPreTraining mindformers.models.bert.BertForQuestionAnswering mindformers.models.bert.BertForTokenClassification mindformers.models.bert.BertTokenizer mindformers.models.bert.BertProcessor mindformers.models.t5 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.t5.T5Config mindformers.models.t5.T5ForConditionalGeneration mindformers.models.t5.T5Processor mindformers.models.t5.T5Tokenizer mindformers.models.clip ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.clip.CLIPConfig mindformers.models.clip.CLIPVisionConfig mindformers.models.clip.CLIPTextConfig mindformers.models.clip.CLIPModel mindformers.models.clip.CLIPTokenizer mindformers.models.clip.CLIPProcessor mindformers.models.clip.CLIPImageProcessor mindformers.models.mae ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.mae.ViTMAEConfig mindformers.models.mae.ViTMAEForPreTraining mindformers.models.mae.ViTMAEProcessor mindformers.models.mae.ViTMAEImageProcessor mindformers.models.swin ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.swin.SwinConfig mindformers.models.swin.SwinModel mindformers.models.swin.SwinForImageClassification mindformers.models.swin.SwinImageProcessor mindformers.models.swin.SwinProcessor mindformers.models.vit ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.vit.ViTConfig mindformers.models.vit.ViTModel mindformers.models.vit.ViTProcessor mindformers.models.vit.ViTImageProcessor mindformers.models.vit.ViTForImageClassification mindformers.models.gpt2 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.gpt2.GPT2Config mindformers.models.gpt2.GPT2Model mindformers.models.gpt2.GPT2LMHeadModel mindformers.models.gpt2.GPT2Tokenizer mindformers.models.gpt2.GPT2Processor mindformers.models.glm ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.glm.GLMConfig mindformers.models.glm.GLMChatModel mindformers.models.glm.GLMForPreTraining mindformers.models.glm.GLMForPreTrainingWithLora mindformers.models.glm.GLMChatModelWithLora mindformers.models.glm.ChatGLMTokenizer mindformers.models.glm.GLMProcessor mindformers.models.llama ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.llama.LlamaConfig mindformers.models.llama.LlamaModel mindformers.models.llama.LlamaForCausalLM mindformers.models.llama.LlamaForCausalLMWithLora mindformers.models.llama.LlamaTokenizer mindformers.models.llama.LlamaProcessor mindformers.models.bloom ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bloom.BloomConfig mindformers.models.bloom.BloomModel mindformers.models.bloom.BloomLMHeadModel mindformers.models.bloom.BloomTokenizer mindformers.models.bloom.BloomProcessor pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.modules.rst000066400000000000000000000025201476650141400336370ustar00rootroot00000000000000mindformers.modules ===================== .. automodule:: mindformers.modules mindformers.modules.layers ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.layers.Dropout mindformers.modules.layers.FixedSparseAttention mindformers.modules.layers.LayerNorm mindformers.modules.layers.Linear mindformers.modules.transformer ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.transformer.AttentionMask mindformers.modules.transformer.EmbeddingOpParallelConfig mindformers.modules.transformer.FeedForward mindformers.modules.transformer.MoEConfig mindformers.modules.transformer.MultiHeadAttention mindformers.modules.transformer.OpParallelConfig mindformers.modules.transformer.Transformer mindformers.modules.transformer.TransformerDecoder mindformers.modules.transformer.TransformerDecoderLayer mindformers.modules.transformer.TransformerEncoder mindformers.modules.transformer.TransformerEncoderLayer mindformers.modules.transformer.TransformerOpParallelConfig mindformers.modules.transformer.TransformerRecomputeConfig mindformers.modules.transformer.VocabEmbedding pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.pipeline.rst000066400000000000000000000010511476650141400337720ustar00rootroot00000000000000mindformers.pipeline ===================== .. automodule:: mindformers.pipeline .. autosummary:: :toctree: pipeline :nosignatures: :template: classtemplate.rst mindformers.pipeline.BasePipeline mindformers.pipeline.pipeline mindformers.pipeline.FillMaskPipeline mindformers.pipeline.TranslationPipeline mindformers.pipeline.ImageClassificationPipeline mindformers.pipeline.TextClassificationPipeline mindformers.pipeline.TokenClassificationPipeline mindformers.pipeline.ZeroShotImageClassificationPipeline pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.rst000066400000000000000000000004171476650141400321730ustar00rootroot00000000000000mindformers ================== .. automodule:: mindformers .. autosummary:: :toctree: mindformers :nosignatures: :template: classtemplate.rst mindformers.AutoConfig mindformers.AutoModel mindformers.AutoProcessor mindformers.AutoTokenizer pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.trainer.rst000066400000000000000000000015271476650141400336410ustar00rootroot00000000000000mindformers.trainer ==================== .. automodule:: mindformers.trainer .. autosummary:: :toctree: trainer :nosignatures: :template: classtemplate.rst mindformers.trainer.ContextConfig mindformers.trainer.ParallelContextConfig mindformers.trainer.TrainingArguments mindformers.trainer.Trainer mindformers.trainer.TranslationTrainer mindformers.trainer.CausalLanguageModelingTrainer mindformers.trainer.ContrastiveLanguageImagePretrainTrainer mindformers.trainer.ImageClassificationTrainer mindformers.trainer.MaskedImageModelingTrainer mindformers.trainer.MaskedLanguageModelingTrainer mindformers.trainer.QuestionAnsweringTrainer mindformers.trainer.TextClassificationTrainer mindformers.trainer.TokenClassificationTrainer mindformers.trainer.ZeroShotImageClassificationTrainer pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python/mindformers.wrapper.rst000066400000000000000000000004111476650141400336440ustar00rootroot00000000000000mindformers.wrapper ==================== .. automodule:: mindformers.wrapper .. autosummary:: :toctree: wrapper :nosignatures: :template: classtemplate.rst mindformers.wrapper.MFTrainOneStepCell mindformers.wrapper.MFPipelineWithLossScaleCellpengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/000077500000000000000000000000001476650141400275745ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.core.rst000066400000000000000000000036021476650141400336030ustar00rootroot00000000000000mindformers.core ================== .. automodule:: mindformers.core mindformers.core ----------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.ClipGradNorm mindformers.core.build_context mindformers.core.init_context mindformers.core.callback -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.callback.CheckpointMointor mindformers.core.callback.MFLossMonitor mindformers.core.callback.ObsMonitor mindformers.core.callback.SummaryMonitor mindformers.core.callback.ProfileMonitor mindformers.core.callback.EvalCallBack mindformers.core.loss -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.loss.CrossEntropyLoss mindformers.core.loss.L1Loss mindformers.core.loss.MSELoss mindformers.core.loss.SoftTargetCrossEntropy mindformers.core.lr -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.lr.ConstantWarmUpLR mindformers.core.lr.CosineWithRestartsAndWarmUpLR mindformers.core.lr.CosineWithWarmUpLR mindformers.core.lr.LinearWithWarmUpLR mindformers.core.lr.PolynomialWithWarmUpLR mindformers.core.metric -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.metric.EntityScore mindformers.core.metric.SQuADMetric mindformers.core.metric.PerplexityMetric mindformers.core.metric.ADGENMetric mindformers.core.optim -------------------------- .. autosummary:: :toctree: core :nosignatures: :template: classtemplate.rst mindformers.core.optim.FusedAdamWeightDecay mindformers.core.optim.FP32StateAdamWeightDecay pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.dataset.rst000066400000000000000000000012741476650141400343030ustar00rootroot00000000000000mindformers.dataset ==================== .. automodule:: mindformers.dataset .. autosummary:: :toctree: dataset :nosignatures: :template: classtemplate.rst mindformers.dataset.BaseDataset mindformers.dataset.MIMDataset mindformers.dataset.ContrastiveLanguageImagePretrainDataset mindformers.dataset.ImageCLSDataset mindformers.dataset.ImageToTextRetrievalDataset mindformers.dataset.MaskLanguageModelDataset mindformers.dataset.KeyWordGenDataset mindformers.dataset.CausalLanguageModelDataset mindformers.dataset.TextClassificationDataset mindformers.dataset.TokenClassificationDataset mindformers.dataset.ZeroShotImageClassificationDataset pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.models.rst000066400000000000000000000102141476650141400341330ustar00rootroot00000000000000mindformers.models =================== .. automodule:: mindformers.models mindformers.models ------------------- .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.BaseConfig mindformers.models.BaseImageProcessor mindformers.models.BaseModel mindformers.models.BaseProcessor mindformers.models.Tokenizer mindformers.models.bert ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bert.BertConfig mindformers.models.bert.BertModel mindformers.models.bert.BertForMultipleChoice mindformers.models.bert.BertForPreTraining mindformers.models.bert.BertForQuestionAnswering mindformers.models.bert.BertForTokenClassification mindformers.models.bert.BertTokenizer mindformers.models.bert.BertProcessor mindformers.models.t5 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.t5.T5Config mindformers.models.t5.T5ForConditionalGeneration mindformers.models.t5.T5Processor mindformers.models.t5.T5Tokenizer mindformers.models.clip ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.clip.CLIPConfig mindformers.models.clip.CLIPVisionConfig mindformers.models.clip.CLIPTextConfig mindformers.models.clip.CLIPModel mindformers.models.clip.CLIPTokenizer mindformers.models.clip.CLIPProcessor mindformers.models.clip.CLIPImageProcessor mindformers.models.mae ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.mae.ViTMAEConfig mindformers.models.mae.ViTMAEForPreTraining mindformers.models.mae.ViTMAEProcessor mindformers.models.mae.ViTMAEImageProcessor mindformers.models.swin ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.swin.SwinConfig mindformers.models.swin.SwinModel mindformers.models.swin.SwinForImageClassification mindformers.models.swin.SwinImageProcessor mindformers.models.swin.SwinProcessor mindformers.models.vit ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.vit.ViTConfig mindformers.models.vit.ViTModel mindformers.models.vit.ViTProcessor mindformers.models.vit.ViTImageProcessor mindformers.models.vit.ViTForImageClassification mindformers.models.gpt2 ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.gpt2.GPT2Config mindformers.models.gpt2.GPT2Model mindformers.models.gpt2.GPT2LMHeadModel mindformers.models.gpt2.GPT2Tokenizer mindformers.models.gpt2.GPT2Processor mindformers.models.glm ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.glm.GLMConfig mindformers.models.glm.GLMChatModel mindformers.models.glm.GLMForPreTraining mindformers.models.glm.GLMForPreTrainingWithLora mindformers.models.glm.GLMChatModelWithLora mindformers.models.glm.ChatGLMTokenizer mindformers.models.glm.GLMProcessor mindformers.models.llama ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.llama.LlamaConfig mindformers.models.llama.LlamaModel mindformers.models.llama.LlamaForCausalLM mindformers.models.llama.LlamaForCausalLMWithLora mindformers.models.llama.LlamaTokenizer mindformers.models.llama.LlamaProcessor mindformers.models.bloom ------------------------ .. autosummary:: :toctree: models :nosignatures: :template: classtemplate.rst mindformers.models.bloom.BloomConfig mindformers.models.bloom.BloomModel mindformers.models.bloom.BloomLMHeadModel mindformers.models.bloom.BloomTokenizer mindformers.models.bloom.BloomProcessor pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.modules.rst000066400000000000000000000025201476650141400343210ustar00rootroot00000000000000mindformers.modules ===================== .. automodule:: mindformers.modules mindformers.modules.layers ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.layers.Dropout mindformers.modules.layers.FixedSparseAttention mindformers.modules.layers.LayerNorm mindformers.modules.layers.Linear mindformers.modules.transformer ---------------------------------------------- .. autosummary:: :toctree: modules :nosignatures: :template: classtemplate.rst mindformers.modules.transformer.AttentionMask mindformers.modules.transformer.EmbeddingOpParallelConfig mindformers.modules.transformer.FeedForward mindformers.modules.transformer.MoEConfig mindformers.modules.transformer.MultiHeadAttention mindformers.modules.transformer.OpParallelConfig mindformers.modules.transformer.Transformer mindformers.modules.transformer.TransformerDecoder mindformers.modules.transformer.TransformerDecoderLayer mindformers.modules.transformer.TransformerEncoder mindformers.modules.transformer.TransformerEncoderLayer mindformers.modules.transformer.TransformerOpParallelConfig mindformers.modules.transformer.TransformerRecomputeConfig mindformers.modules.transformer.VocabEmbedding pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.pipeline.rst000066400000000000000000000010511476650141400344540ustar00rootroot00000000000000mindformers.pipeline ===================== .. automodule:: mindformers.pipeline .. autosummary:: :toctree: pipeline :nosignatures: :template: classtemplate.rst mindformers.pipeline.BasePipeline mindformers.pipeline.pipeline mindformers.pipeline.FillMaskPipeline mindformers.pipeline.TranslationPipeline mindformers.pipeline.ImageClassificationPipeline mindformers.pipeline.TextClassificationPipeline mindformers.pipeline.TokenClassificationPipeline mindformers.pipeline.ZeroShotImageClassificationPipeline pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.rst000066400000000000000000000004171476650141400326550ustar00rootroot00000000000000mindformers ================== .. automodule:: mindformers .. autosummary:: :toctree: mindformers :nosignatures: :template: classtemplate.rst mindformers.AutoConfig mindformers.AutoModel mindformers.AutoProcessor mindformers.AutoTokenizer pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.trainer.rst000066400000000000000000000015271476650141400343230ustar00rootroot00000000000000mindformers.trainer ==================== .. automodule:: mindformers.trainer .. autosummary:: :toctree: trainer :nosignatures: :template: classtemplate.rst mindformers.trainer.ContextConfig mindformers.trainer.ParallelContextConfig mindformers.trainer.TrainingArguments mindformers.trainer.Trainer mindformers.trainer.TranslationTrainer mindformers.trainer.CausalLanguageModelingTrainer mindformers.trainer.ContrastiveLanguageImagePretrainTrainer mindformers.trainer.ImageClassificationTrainer mindformers.trainer.MaskedImageModelingTrainer mindformers.trainer.MaskedLanguageModelingTrainer mindformers.trainer.QuestionAnsweringTrainer mindformers.trainer.TextClassificationTrainer mindformers.trainer.TokenClassificationTrainer mindformers.trainer.ZeroShotImageClassificationTrainer pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/api_python_en/mindformers.wrapper.rst000066400000000000000000000004111476650141400343260ustar00rootroot00000000000000mindformers.wrapper ==================== .. automodule:: mindformers.wrapper .. autosummary:: :toctree: wrapper :nosignatures: :template: classtemplate.rst mindformers.wrapper.MFTrainOneStepCell mindformers.wrapper.MFPipelineWithLossScaleCellpengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/000077500000000000000000000000001476650141400262115ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/AutoClass_Design.md000066400000000000000000000022221476650141400317200ustar00rootroot00000000000000# AutoClass MindFormers大模型套件提供了AutoClass类，包含AutoConfig、AutoModel、AutoTokenizer、AutoProcessor4个便捷高阶接口，方便用户调用套件中已封装的API接口。上述4类提供了相应领域模型的ModelConfig、Model、Tokenzier、Processor的实例化功能。 ![输入图片说明](https://foruda.gitee.com/images/1686128219487920380/00f18fec_9324149.png) | AutoClass类 | from_pretrained属性（实例化功能） | from_config属性（实例化功能） | save_pretrained（保存配置功能） | | ------------- | --------------------------------- | ----------------------------- | :-----------------------------: | | AutoConfig | √ | × | √ | | AutoModel | √ | √ | √ | | AutoProcessor | √ | × | √ | | AutoTokenizer | √ | × | √ | pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/Config_Design.md000066400000000000000000000446571476650141400312510ustar00rootroot00000000000000# Config和API组件 ## config概念 config包含模型配置、训练配置、环境配置等信息。启动训练、微调、评估和推理时，会先根据config中的内容进行配置。 ## config设计 ![](https://foruda.gitee.com/images/1691027467343740998/29a1db43_11500692.png) MindformerConfig是Mindformers整体的配置类，里面包含Mindformers中的各种配置。 - run_xxx.yaml中的配置为默认配置。默认情况下，Mindformers根据yaml配置文件创建MindFormerConfig。 - 如果使用run_mindformer.py方式启动，可以通过命令行参数修改配置中的参数。 - 如果使用Trainer高阶接口方式启动，可以通过TrainArguments修改配置中的参数。 MindformerConfig中包含的具体配置可以参考[Config.md](../../../../../configs/README.md) ## API组件注册机制 ### 注册原理图 ![输入图片说明](https://foruda.gitee.com/images/1673344591842027306/a4d581a3_9324149.png "image-20230110151323393.png") ### 注册机制简介 MindFormers套件全仓代码主要基于Python语言和MindSpore AI框架开发，通过设计业界主流的装饰器注册机制和MindSpore AI框架易用性工具，将套件中涉及的API自动注册至registry字典中，从而实现对API接口的灵活调用。 注册代码设计主要包含两大类： 1. MindFormerModuleType 全局注册模块定义类 2. MindFormerRegister API注册和实例化类。 通过该注册机制的设定，可以有效的统一套件中各类API接口的调用机制和开发范式，实现套件模块化设计和灵活配置。 ### MindFormerModuleType MindFormerModuleType类目前共支持全局注册模块共22个，可根据实际模型或者工程需求进行增加或改动，详情请见下表： | MindFormerModuleType | | | | -------------------- | ------------------ | ------------------------------------------------ | | 模块名称 | 模块赋值（String） | 模块定义 | | TRAINER | 'trainer' | 任务流程创建模块，定义训练/微调/评估/推理流程 | | PIPELINE | 'pipeline' | 推理模块，定义任务的推理流程 | | PROCESSOR | 'procrssor' | 处理流模块，定义模型权重预处理和加载流程 | | DATASET | 'dataset' | 数据流模块，定义数据集加载及预处理流程 | | MASK_POLICY | 'mask_policy' | 掩码模块，定义掩码策略 | | DATASET_LOADER | 'dataset_loader' | 数据集加载模块，定义数据集加载策略 | | DATASET_SAMPLER | 'dataset_sampler' | 数据集采样模块，定义数据集采样策略 | | TRANSFORMS | 'transforms' | 数据集增强模块，定义数据集增强策略 | | MODELS | 'models' | 模型网络模块，定义整体网络架构 | | ENCODER | 'encoder' | 模型骨干网络模块，定义网络中encoder模块 | | HEAD | 'head' | 模型head模块，定义网络中head头模块 | | MODULES | 'modules' | 模型组件模块，定义网络组件 | | BASE_LAYER | 'base_layer' | 定义模型组件最小单元模块 | | CORE | 'core' | 定义模型组件最小插件模块 | | LOSS | 'loss' | 损失函数模块，定义损失函数类型 | | LR | 'lr' | 学习策略模块，定义学习策略类型 | | OPTIMIZER | 'optimizer' | 优化器模块，定义优化器类型 | | CONFIG | 'config' | 配置模块，定义模型相关的配置 | | WRAPPER | 'wrapper' | 模型训练封装模块，定义模型前向计算和梯度更新流程 | | METRIC | 'metric' | 模型度量模块，定义模型评估策略 | | CALLBACK | 'callback' | 回调函数模块，定义模型回调类型 | | TOOLS | 'tools' | 通用工具模块，定义模型公用工具 | ### MindFormerRegister MindFormerRegister 类涉及7类属性函数，主要功能包含：1. API 注册 2. API实例化 3. API检查。详情请见下表： | 属性函数 | 入参说明 | 功能说明 | | ------------------------ | ------------------------------------------------------------ | ---------------------------- | | registry | dict类型 | 存储已注册的API | | register | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **alias**(string): 注册API重命名，默认None | 装饰器函数，完成API 注册 | | register_cls | 1. **register_class**(class or function): 注册API 2. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 3. **alias**(string): 注册API重命名，默认None | 注册函数，完成API注册 | | get_instance_from_config | 1. **config**(dict): 配置文件字典 2. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 3. **default_args**(dict): API默认参数，默认None | 通过配置文件获取API实例 | | get_instance | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **class_name**(string): 已注册的API名称，默认None 3. ***args**: 定省参数，已注册API支持参数 4. **kwargs: 不定省参数，已注册API支持参数 | 通过已注册API名称获取API实例 | | get_cls | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’tools’ 2. **class_name**(string): 已注册的API名称，默认None | 获取已注册的API类 | | is_exist | 1. **module_type**(string): MindFormerModuleType包含的模块名，默认’common’2. class_name(string): 已注册的API名称，默认None | 检查API是否注册 | ### 注册机制使用示例 - API注册Python代码示例：用户可以利用以下两种注册方式将任意API注册至registry字典中。 ```python from mindformers.tools.register import MindFormerRegister, MindFormerModuleType # Type1 装饰器注册自定义API接口 # 此处即可通过装饰MindFormerRegister的register属性完成 YourDefineLRAPI 类的注册，并将其注册到LR模块 @MindFormerRegister.register(MindFormerModuleType.LR) class YourDefineLRAPI(): def __init__(self, *args, **kwargs): Pass # Type2 通过API类进行注册 # 此处即可通过装饰MindFormerRegister的register_cls属性完成 YourDefineAPILR类的注册，并将其注册到LR模块 MindFormerRegister.register_cls(YourDefineLRAPI, MindFormerModuleType.LR) ``` ## API 组件 Build 机制 MindFormers套件在注册机制的基础之上，为每个模块都配备的对应的build函数，方便用户使用统一且通用的build形式来创建不同模块的API实例。然而register和build的形式仍然具有一定的`黑盒`特性，需要大家提前了解MindFormers套件中的注册和API创建原理。 ### build 支持列表 | Build Module | 功能 | | :------------------: | :------------------------------------------------: | | [build_callback]() | 创建回调函数，返回API实例或者包含API实例的列表 | | [build_lr]() | 创建学习率函数，返回API实例 | | [build_metric]() | 创建评估指标函数，返回API实例 | | [build_optim]() | 创建优化器函数，返回API实例 | | [build_loss]() | 创建loss函数，返回API实例 | | [build_dataset]() | 创建数据集，返回MindSpore Dataset实例 | | [build_dataloader]() | 创建数据集加载函数，返回API实例 | | [build_sampler]() | 创建数据集采样函数，返回API实例 | | [build_transforms]() | 创建数据增广函数，返回API实例或者包含API实例的列表 | | [build_mask]() | 创建掩码策略，返回API实例 | | [build_config]() | 创建模型配置函数，返回API实例 | | [build_model]() | 创建模型网络函数，返回API实例 | | [build_processor]() | 创建数据预处理函数（用于推理），返回API实例 | | [build_tokenizer]() | 创建文本的tokenizer函数，返回API实例 | | [build_module]() | 创建网络组件函数，返回API实例 | | [build_layer]() | 创建网络基本构成函数，返回API实例 | | [build_pipeline]() | 创建pipeline推理函数，返回API实例 | | [build_trainer]() | 创建trainer函数，返回API实例 | | [build_wrapper]() | 创建前反向封装函数，返回API实例 | 可以看到，MindFormers套件中目前已经支持20个模块的build，基本覆盖组网过程的各个模块的创建。值得注意的是，我们build函数的设计主要依赖于配置文件（即包含相关模块和相应配置的一个字典文件），因此要充分理解build机制，需要同时学习如何开发配置文件。 ### Build 使用案例 下面为用户以某数据集模块的注册和创建使用为例，详细介绍如何使用我们的Build机制来完成一个数据流的创建。 #### 创建分类任务数据集 ```text Note: 1. MindFormers套件中已经集成的Dataset类均需要依赖配置文件来进行创建，因此使用前可以参考相关配置文件用例来自定义数据任务配置 2. 如果不想使用配置文件来调用Dataset，用户可以自行定义完整的数据集模块并注册，只需保证返回的实例是mindspore的DATASET类型 ``` - Step 1: 开发配置文件 [配置文件链接](https://gitee.com/mindspore/mindformers/blob/dev/configs/mae/run_mae_vit_base_p16_224_800ep.yaml) ```yaml # train dataset train_dataset: &train_dataset seed: 2022 batch_size: 64 data_loader: type: ImageFolderDataset dataset_dir: "imageNet-1k/train" num_parallel_workers: 8 shuffle: True transforms: - type: RandomCropDecodeResize size: 224 scale: [0.2, 1.0] interpolation: cubic - type: RandomHorizontalFlip prob: 0.5 - type: Normalize mean: [123.675, 118.575, 103.53] std: [58.395, 62.22, 57.375] - type: HWC2CHW mask_policy: type: MaeMask input_size: 224 patch_size: 16 mask_ratio: 0.75 input_columns: ["image"] output_columns: ["image", "mask", "ids_restore", "unmask_index"] column_order: ["image", "mask", "ids_restore", "unmask_index"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: MIMDataset dataset_config: *train_dataset ``` - 开发MIMDataset类并注册 [MIMDataset类](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/mim_dataset.py) Note: 如果用户不想依赖配置文件创建数据任务，这里可以由用户自行填写__new__中的创建过程，并保证最后返回的是mindspore dataset实例即可 ```python """Masked Image Modeling Dataset.""" import os import mindspore from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from mindformers.dataset.dataloader import build_dataset_loader from mindformers.dataset.mask import build_mask from mindformers.dataset.transforms import build_transforms from mindformers.dataset.sampler import build_sampler from mindformers.dataset.base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MIMDataset(BaseDataset): """ Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) mask = build_mask(dataset_config.mask_policy) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: for column in dataset_config.input_columns: dataset = get_dataset_map(dataset, input_columns=column, operations=transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if mask is not None: dataset = get_dataset_map(dataset, operations=mask, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset ``` - [Build接口](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/build_dataset.py)：该接口无需用户开发，以及集成在MindFormers套件中供调用 ```python """Build Dataset API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset( config: dict = None, default_args: dict = None, module_type: str = 'dataset', class_name: str = None, **kwargs): r"""Build dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) ``` - 创建Dataset实例 通过build_dataset接口，根据dataset的配置文件来创建MIMDataset实例： ```python from mindformers.tools.register import MindFormerConfig from mindformers.dataset import build_dataset, check_dataset_config # Initialize a MindFormerConfig instance with a specific config file of yaml. config = MindFormerConfig("configs/mae/task_config/mim_dataset.yaml") check_dataset_config(config) # 1) use config dict to build dataset dataset_from_config = build_dataset(config.train_dataset_task) # 通过train_dataset_task关键字来创建是MIMDataset API实例 # 2) use class name to build dataset dataset_from_name = build_dataset(class_name='MIMDataset', dataset_config=config.train_dataset_task) # 3) use class to build dataset dataset_from_class = MIMDataset(config.train_dataset_task) # 4) 不使用配置文件创建dataset实例 dataset_from_self_define1 = build_dataset(class_name='MIMDataset') dataset_from_self_define2 = MIMDataset() ``` pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/Parallel_Design.md000066400000000000000000001413511476650141400315650ustar00rootroot00000000000000# Parallel **《并行专题介绍》** ## 前言 ### 分布式并行场景 - **超大规模图像分类与识别** 这类网络往往有一个超大FC层以及loss层。在人脸识别和超大规模图像分类领域，**分类数可能高达千万，FC层参数量能到达到10B级别**。这类网络即使在数万分类这种场景，**模型并行相对数据并行来说性能也能更优，因为数据并行通信的是参数梯度，通信数据量等于参数量，而模型并行通信的是feature map，这类网络feature map的数据量远小于参数的数据量，进而在通信上能够性能更优。** - **高分辨率3D图像处理** 例如医疗领域的3维CT图像处理，典型网络是Unet-3D。这类场景，输入图像往往有亿级像素，模型的Feature Map非常大，叠加上比较深的网络结构，模型也往往是数百GB； - **NLP领域** NLP领域，目前基于Transformer结构的网络是SOTA方法，最典型网络的大网络有GPT2/3、Google翻译、FaceBook翻译等模型。基于Transformer结构的网络，从bert的340M参数，transformer-xl 800M参数，gpt-2 1542M参数，NVIDIA megatron有8B参数、GPT-3的175B参数。往往更大的网络，就需要用更大的数据来喂，训练这类网络，就需要使用**数据并行叠加模型并行的方式，同时处理大规模数据集及大规模参数。** - **推荐网络 Wide&Deep/DeepFM/DCN**等 在推荐领域，特征数量是百亿甚至千亿的规模，会有超大Embedding层，内存开销远超单卡内存，就必须使用模型并行把Embedding参数切分到集群。 总的来说，不同场景的模型，结构各有不同，如下图所示，不同结构的模型，需要不同的并行策略，才能实现性能的最优。图中方块的大小表示layer的内存开销。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092326vsbttpsalxmf11yl.png) **Transformer一般会采用数据并行叠加模型并行的策略；推荐一般会采用模型并行转数据并行的策略；ReID一般会采用数据并行转模型并行的策略；** ### MindSpore分布式并行模式简介 当前MindSpore提供分布式并行训练的功能，它支持了多种模式包括： | 并行模式 | 配置 | 动态图 | 静态图 | 支持设备 | | ---------- | ------------------ | ------ | ------ | --------------- | | 数据并行 | DATA_PARALLEL | 支持 | 支持 | GPU、Ascend 910 | | 半自动并行 | SEMI_AUTO_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | | 全自动并行 | AUTO_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | | 混合并行 | HYBRID_PARALLEL | 不支持 | 支持 | GPU、Ascend 910 | - `DATA_PARALLEL`：**数据并行模式**。用户的网络参数规模在单卡上可以计算的情况下使用。这种模式会在每卡上复制相同的网络参数，训练时输入不同的训练数据，适合大部分用户使用。 - `AUTO_PARALLEL`：**自动并行模式**。融合了数据并行、算子级模型并行的分布式并行模式，可以自动建立代价模型，找到训练时间较短的并行策略，为用户选择合适的并行模式。当前MindSpore支持算子级并行策略的自动搜索，提供了如下的三种不同的策略搜索算法： - `dynamic_programming`：动态规划策略搜索算法。能够搜索出代价模型刻画的最优策略，但在搜索巨大网络模型的并行策略时耗时较长。其代价模型是围绕Ascend 910芯片基于内存的计算开销和通信开销对训练时间建模。 - `recursive_programming`：双递归策略搜索算法。对于巨大网络以及大规模多卡切分能够保证瞬间生成最优策略。其基于符号运算的代价模型可以自由适配不同的加速器集群。 - `sharding_propagation`：**切分策略传播算法**。由配置并行策略的算子向未配置的算子传播并行策略。在传播时，算法会尽量选取引发张量重排布通信最少的策略。关于算子的并行策略配置和张量重排布，可参考这篇[设计文档](https://www.mindspore.cn/docs/zh-CN/r1.9/design/distributed_training_design.html#自动并行原理)。 该模式适用于用户的神经网络在单卡上无法计算，但是不知道如何配置算子策略。用户启动这种模式，MindSpore会自动针对每个算子进行配置策略，**适合想要并行训练但是不知道如何配置策略的用户**。 - `SEMI_AUTO_PARALLEL`：**半自动并行模式**。相较于自动并行，该模式需要用户对算子手动配置切分策略实现并行。 - `HYBRID_PARALLEL`：**混合并行模式**。在MindSpore中特指用户通过手动切分模型实现混合并行的场景。该模式完全由用户自己设计并行训练的逻辑和实现，用户可以自己在网络中定义`AllGather`等通信算子，适合熟悉并行训练的用户。 ## MindSpore 分布式并行设计 ### 设计抽象 MindSpore目前是基于**集合通信**的模式来实现并行，如下图所示，分布式并行问题可以抽象成这样一个问题： **将计算图（Graph）和Tensor切分到集群并行执行，并能保持高性能。** ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/0924434kcjcswerenfg2yz.png) **从Tensor切分角度看数据并行、模型并行和混合并行。** 如下图所示，**数据并行就是在输入数据Tensor的batch维度切分，模型并行就是对模型参数Tensor的切分，而混合并行就是对输入数据Tensor和模型参数Tensor同时切分**。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092515mmmwvsrt1iyu1rgi.png) ### 设计目标 MindSpore自动并行的目标是**构建一种易用高效的的分布式并行训练模式，融合了数据并行、模型并行和混合并行，让算法人员不再需要关注算法模型到底需要用哪种模式训练**。主要目标是： - **简化分布式并行编程，串行代码实现分布式训练，对用户屏蔽并行细节，并且保持高性能；** - **计算逻辑上保持和单卡串行流程一致；** - **实现上统一数据并行和模型并行，一套框架支持多种并行模式；** - **结合集群拓扑优化性能；** 实现上，就像前面介绍一下，MindSpore的自动并行分成两部分，半自动并行和全自动并行。 **半自动并行**：主要是在图编译阶段实现了一套算子切分及图切分的框架，把模型切片和设备进行调度绑定。在API这层，把并行逻辑和算法逻辑解耦，变成了一些配置。 **自动并行**：其在半自动并行基础上，构建了一套Cost Model，能够了基于数据量、模型参数量、网络集群拓扑带宽等信息的代价模型，通过一套策略搜索算法，计算是性能最优的切分策略，这样用户就不需要感知切分策略。 ## 数据并行和自动并行 ### 数据并行 数据并行是业界非常常用的并行方式，本章节开始介绍MindSpore中`ParallelMode.DATA_PARALLEL`数据并行的实现原理和使用方式。 #### 数据并行原理 ![数据并行图解](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/data_parallel.png) 1. 环境依赖 每次开始进行并行训练前，通过调用`mindspore.communication.init`接口初始化通信资源，并自动创建全局通信组`WORLD_COMM_GROUP`。 2. 数据分发（Data distribution） 数据并行的核心在于将数据集在样本维度拆分并下发到不同的卡上。在`mindspore.dataset`模块提供的所有数据集加载接口中都有`num_shards`和`shard_id`两个参数，它们用于将数据集拆分为多份并循环采样的方式，采集`batch`大小的数据到各自的卡上，当出现数据量不足的情况时将会从头开始采样。 3. 网络构图 数据并行网络的书写方式与单机网络没有差别，这是因为在正反向传播（Forward propagation & Backward Propagation）过程中各卡的模型间是独立执行的，只是保持了相同的网络结构。唯一需要特别注意的是为了保证各卡间训练同步，相应的网络参数初始化值应当是一致的，在`DATA_PRALLEL`和`HYBRID_PARALLEL`模式下建议通过使能`parameter_broadcast`达到权重广播的目的；在`AUTO_PRALLEL`和`SEMI_AUTO_PARALLEL`模式下，框架内部会自动分析参数的并行度，并设置相应的随机数种子，保证在数据并行维度的设备上参数初始化值一致。 4. 梯度聚合（Gradient aggregation） 数据并行理论上应该实现和单机一致的训练效果，为了保证计算逻辑的一致性，在梯度计算完成后插入`AllReduce`算子实现各卡间的梯度聚合操作。**MindSpore设置了`mean`开关，用户可以选择是否要对求和后的梯度值进行求平均操作，也可以将其视为超参项，打开开关等价于学习率倍数缩小。** 5. 参数更新（Parameter update） 因为引入了梯度聚合操作，所以各卡的模型会以相同的梯度值一起进入参数更新步骤。因此MindSpore实现的是一种同步数据并行训练方式。理论上最终每卡训练出来的模型是相同的，如果网络中含有在样本维度的归约类型操作，网络的输出可能会有所差别，这是由数据并行的切分性质决定的。 #### 数据并行代码 1. 集合通信 - [management.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/communication/management.py)：这个文件中涵盖了集合通信过程中常用的`helper`函数接口，例如获取集群数量和卡的序号等。当在Ascend芯片上执行时，框架会加载环境上的`libhccl.so`库文件，通过它来完成从Python层到底层的通信接口调用。 - [comm_ops.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/ops/operations/comm_ops.py)：MindSpore将支持的集合通信操作都封装为算子的形式放在这个文件下，包括`AllReduce`、`AllGather`、`ReduceScatter`和`Broadcast`等。`PrimitiveWithInfer`中除了定义算子所需属性外，还包括构图过程中输入到输出的`shape`和`dtype`推导。 2. 梯度聚合 - [grad_reducer.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/nn/wrap/grad_reducer.py)：这个文件实现了梯度聚合的过程。对入参`grads`用`HyperMap`展开后插入`AllReduce`算子，这里采用的是全局通信组，用户也可以根据自己网络的需求仿照这个模块进行自定义开发。MindSpore中单机和分布式执行共用一套网络封装接口，在`Cell`内部通过`ParallelMode`来区分是否要对梯度做聚合操作，网络封装接口建议参考`TrainOneStepCell`代码实现。 #### 数据并行案例 在数据并行中，用户定义网络的方式和单机脚本一样，但是在网络定义之前调用[init()](https://www.mindspore.cn/docs/zh-CN/r1.9/api_python/mindspore.communication.html#mindspore.communication.init)去初始化设备通信状态。 ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class DataParallelNet(nn.Cell): def __init__(self): super(DataParallelNet, self).__init__() # 初始化权重 weight_init = np.random.rand(512, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.fc = ops.MatMul() self.reduce = ops.ReduceSum() def construct(self, x): x = self.fc(x, self.weight) x = self.reduce(x, -1) return x init() # 设置并行模式为数据并行，其他方式一致 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.DATA_PARALLEL) net = DataParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` ### 自动并行 自动并行作为MindSpore的关键特性，用于实现自动的数据并行加模型并行的混合并行训练方式，旨在帮助用户以单机的脚本表达并行算法逻辑，降低分布式训练难度，提高算法研发效率，同时又能保持训练的高性能。这个小节介绍了在MindSpore中`ParallelMode.AUTO_PARALLEL`自动并行模式及`ParallelMode.SEMI_AUTO_PARALLEL`半自动并行模式是如何工作的。 #### 自动并行原理 下面给出MindSpore自动并行原理设计图： **注意**：图示的Shard Strategy Search中缺少 Sharding Propagation方法，这是目前自动并行推荐的策略自动配置方法 ![自动并行图解](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/auto_parallel.png) 可以看到，图中给出了自动并行实现的关键流程，其中需经过一些关键操作完成整个自动并行的实现，这里先介绍一下基本的概念： - 关键概念 | 关键概念 | 说明 | | :-------------------------------------: | :----------------------------------------------------------: | | ANF（Accelerate Network Forward） Graph | 正向计算图 | | Sub ANF Graph | 正向计算子图 | | Graph Partition | 图切分 | | Layer-wise partition | 按layer切分 | | Pipeline partition | 按stage/block切分 | | Distributed Operator | 分布式算子 | | Distributed Optimizer | 分布式优化器 | | Distributed Auto-Grad | 分布式自动微分 | | Tensor Layout | 张量排布（张量被切分之后在集群上的分布情况） | | Tensor Redistribution | 张量重排 （两种Tensor Layout之间的转换）
当上一个算子的输出切分策略和下一个算子的输入切分策略不匹配时，会触发张量重排，会自动插入一些通信算子以保证计算逻辑的正确性 | | Device Topology | 设备拓扑 | | Group Manager | 群组管理 | | Cost Model | 代价模型 | | Distributed Operator Cost | 计算分布式算子代价（确定分布式算子切分策略） | | Cost Weight Estimator | 估计分布式算子的重要性（估计分布式算子不同切分策略的权重占比） | #### 算子级切分 本小接将会介绍如何基于上述并行原理中的**分布式算子和张量排布模型（Distributed Operator && Tensor Layout）**来实现MindSpore中的算子切分。 计算图中的分布式算子计算流程 - 首先对每个算子的输入tensor按策略进行切分，生成算子输入的tensor layout，然后根据算子的数学定义，推导出输出tensor layout； - 然后再检查前一个算子输出tensor layout和下一个算子的输入tensor layout，如果两种不同，则会插入一个tensor redistribution。 ![image.png](https://bbs-img.huaweicloud.com/data/attachment/forum/202011/03/092548flhx7ubmapjrlnvb.png) **Tensor Layout**表示tensor切分后，tensor切片在集群分布情况。 **Tensor Redistribution**表示两种Tensor Layout之间的转换。 **Note**：这种建模设计充分地表达了张量和设备间的映射关系，用户无需感知模型各切片放到哪个设备上运行，框架会自动调度分配。 - **如何得到每个算子的张量排布模型？？？** 为了得到张量的排布模型，每个算子都具有切分策略（Shard Strategy），它表示算子的各个输入在相应维度的切分情况。 MindSpore中目前支持用户手动对算子进行shard配置或使用全自动并行模式进行自动获取两种形式。 - **切分策略配置原则？？？** - 通常情况下只要满足以2为基、均匀分配的原则，张量的任意维度均可切分； - MindSpore为内置算子均实现了shard属性，用户可以通过operator.shard()来为单算子的输入或者输出配置相应的切分策略； - shard属性的入参：`.shard(in_strategy, out_strategy)`，其中in_strategy，out_strategy分布表示算子的输入、输出的切分策略，且均需以元组形式传入，比如当算子本身只需一个Tensor输入时，`in_strategy=((1, 1, 1),)`，当算子本身需要两个Tensor输入时，`in_strategy=((1, 1, 1), (1, 1, 1))`，以此类推即可。 - 不切分的维度shard策略写1，需要切分的维度能够被切分策略整除(建议切分策略以2为基数，被切分之后的tensor shape仍是2的倍数，可以获得较好的切分性能)； - 如果一个算子同时对输入的数据和模型权重同时切分，假设数据（batch维度）部分切dp份，模型权重部分切mp份，需要满足`dp * mp <= total_device_num` - **算子级切分用例** 以下图为例，这是一个三维矩阵乘（BatchMatMul）操作，它的切分策略由两个元组构成，分别表示`input`和`weight`的切分形式。 ![算子切分定义](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/operator_split.png) ```python from mindspore.ops import operations as P batch_matmul = P.BatchMatMul() # 根据算子要输入的Tensor维度来配置切分策略 # 以MatMul算子为例，其一般有两个输入 input, weight， 假设其维度分别为[batch, height, width] [batch, width, channel] # 因此可以分别为这两个Tensor配置切分策略 # 配置1：不做切分 batch_matmul.shard(((1, 1, 1), (1, 1, 1)),) # 配置2：batch维度切分，切8份，8卡数据并行，只切数据，不切参数，batch要求能够被8整除，建议的数值有8,16,32,64.... batch_matmul.shard(((8, 1, 1), (1, 1, 1)),) # 配置3：混合切分策略，数据和模型参数均切分， 数据切4份（dp），参数切2份（mp）， 其中dp和mp值需遵循dp*mp <= total_device_num的原则 batch_matmul.shard(((4, 1, 1), (1, 1, 2)),) ``` #### 张量排布变换（Tensor Redistribution） 当前一个算子的输出张量模型和后一个算子的输入张量模型不一致时，就需要引入计算、通信操作的方式实现张量排布间的变化。自动并行流程引入了张量重排布算法（Tensor Redistribution），可以推导得到任意排布的张量间通信转换方式。下面三个样例表示公式`Z=(X×W)×V`的并行计算过程，即两个二维矩阵乘操作，体现了不同并行方式间如何转换。 在样例一中，第一个数据并行矩阵乘的输出在行方向上存在切分，而第二个模型并行矩阵乘的输入需要全量张量，框架将会自动插入`AllGather`算子实现排布变换。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution1.png) ```python from mindspore.ops import operations as P # 图1中的Y = X * W 计算 matmul_1 = P.MatMul() # 在X的batch维度切分成4份，W不切分 matmul_1.shard(((4, 1, 1), (1, 1)),) # X --> shape: [64, 196, 3] --> 切分成4份 X1: [16, 196, 3] X2: [16, 196, 3] X3: [16, 196, 3] X4: [16, 196, 3] # W --> shape: [3, 32] # 图1中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y不切分，V在channel维度切分成4份 matmul_2.shard(((1, 1, 1), (1, 4)),) # Y --> shape: [64, 196, 32] # V --> shape: [32, 768] --> channel维度切分成4份 V1: [64, 32, 192] V2: [32, 192] V3: [32, 192] V4: [32, 192] Y = batch_matmul_1(X, W) # 内部并行计算逻辑, X切分后使用Scatter算子进行Tensor分发到不同的设备上并行执行运算，具体分在哪台设备上运算由底层算法分配，上层不感知 # X1 * W --> Y1: shape [16, 196, 32] -- *device1 # X2 * W --> Y2: shape [16, 196, 32] -- *device2 # X3 * W --> Y3: shape [16, 196, 32] -- *device3 # X4 * W --> Y4: shape [16, 196, 32] -- *device4 # Y = [Y_c1, Y_c2, Y_c3, Y_c4]T 此时每个计算结果还保留在各自的设备上， T表示转置 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略不一致 # matmul_1.shard(((4, 1, 1), (1, 1)),) <--> matmul_2.shard(((1, 1, 1), (1, 4)),) # 这就导致matmul_2要求的输入必须是一个完整的数据shape，因此在matmul_2计算前，要对matmul_1的输出做All-Gather，将分散在各个设备上的计算结果收集拼接并同步 Z = matmul_2(Y, V) # 先经过All-Gather之后收集Y1, Y2, Y3, Y4 拼接成完整的数据Y： [64, 196, 32] # 将V在channel维度切分成4份： # V1: [32, 192] # V2: [32, 192] # V3: [32, 192] # V4: [32, 192] # 内部并行计算逻辑，V切分后使用Scatter算子分发Tensor给不同设备并行计算： # Y * V1 --> Z1：shape [64, 196, 192] -- *device1 # Y * V2 --> Z2：shape [64, 196, 192] -- *device2 # Y * V3 --> Z3：shape [64, 196, 192] -- *device3 # Y * V4 --> Z4：shape [64, 196, 192] -- *device4 ``` 在样例二中，第一个模型并行矩阵乘的输出在列方向上存在切分，而第二个数据并行矩阵乘的输入在行方向上存在切分，框架将会自动插入等价于集合通信中`AlltoAll`操作的通信算子实现排布变换。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution2.png) ```python from mindspore.ops import operations as P # 图2中的Y = X * W 计算 matmul_1 = P.MatMul() # X不切分，W在channel维度切分成4份 matmul_1.shard(((1, 1, 1), (1, 4)),) # X --> shape: [64, 196, 3] # W --> shape: [3, 32] --> 在channel处切分： W_c1: [3, 8] W_c2: [3, 8] W_c3: [3, 8] W_c4: [3, 8] # 图2中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y在batch维度切分，V不切分 matmul_2.shard(((4, 1, 1), (1, 1)),) # Y --> shape: [64, 196, 32] --> batch维度切分成4份 V1: [16, 196, 32] V2: [16, 196, 32] V3: [16, 196, 32] V4: [16, 196, 32] # V --> shape: [32, 768] Y = matmul_1(X, W) # 内部并行计算逻辑, 分别在不同的设备上并行执行运算，具体分在哪台设备上运算由底层分配，上层不感知 # X [64, 196, 3] * W_c1 [64, 3, 8] --> Y_c1: shape [64, 196, 8] -- *device1 # X * W_c2 --> Y_c2: shape [64, 196, 8] -- *device2 # X * W_c3 --> Y_c3: shape [64, 196, 8] -- *device3 # X * W_c4 --> Y_c4: shape [64, 196, 8] -- *device4 # Y = [Y_c1, Y_c2, Y_c3, Y_c4] 此时每个计算结果还保留在各自的设备上 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略不一致 # matmul_1.shard(((1, 1, 1), (1, 4)),) <--> matmul_2.shard(((4, 1, 1), (1, 1)),) # 即需要将每个device上的 Tensor Shape [64, 196, 8] --> [16, 196, 32] Tensor Layout转换 # 这就导致matmul_2要求的输入不仅是完整的权重shape，而且需要进一步完成batch维度的切分操作。 # 通常的做法是先All-Gather，将Y转换成一个完整的权重shape: [64, 196, 32]， 然后再执行Scatter操作，将Y在batch维度切分之后分发出去 # 然而All-To-All操作可以很方便且高效的同时完成上述两种操作，在All-Gather的同时进行Scatter，完成上述转换过程！ Z = matmul_2(Y, V) # 先经过All-To-All之后完成Y在batch维度切分成4份并分发给不同的设备： # Y_b1: [16, 196, 32] # Y_b2: [16, 196, 32] # Y_b3: [16, 196, 32] # Y_b4: [16, 196, 32] # 内部并行计算逻辑 # Y_b1 * V --> Z_b1：shape [ , 196, 192] -- *device1 # Y_b2 * V --> Z_b2：shape [64, 196, 192] -- *device2 # Y_b3 * V --> Z_b3：shape [64, 196, 192] -- *device3 # Y_b4 * V --> Z_b4：shape [64, 196, 192] -- *device4 ``` 在样例三中，第一个混合并行矩阵乘的输出切分方式和第二个混合并行矩阵乘的输入切分方式一致，所以不需要引入重排布变换。但由于第二个矩阵乘操作中，两个输入的相关维度存在切分，所以需要插入`AllReduce`算子保证运算正确性。 ![张量排布变换](https://www.mindspore.cn/docs/zh-CN/r1.9/_images/tensor_redistribution3.png) ```python from mindspore.ops import operations as P # 图3中的Y = X * W 计算 matmul_1 = P.MatMul() # X在batch维度切2份，W在channel维度切分成2份 matmul_1.shard(((2, 1, 1), (1, 2)),) # X --> shape: [64, 196, 3] --> 在batch处切分： X1：[32, 196, 3] X2: [32, 196, 3] # W --> shape: [3, 32] --> 在channel处切分： W1: [3, 16] W2: [3, 16] # 图3中的Z = Y * V 计算 matmul_2 = P.MatMul() # Y在batch维度切分2份，V在batch维度切分2份 matmul_2.shard(((2, 1, 1), (2, 1)),) # Y --> shape: [64, 196, 32] --> 在batch处切分： Y1：[32, 196, 32] Y2: [32, 196, 32] # V --> shape: [32, 768] --> 在0维处切分： V1: [16, 768] V2: [16, 768] Y = matmul_1(X, W) # X, W 维度切分之后利用Scatter分发Tensor到不同设备上进行并行计算 # [X1, X2]T * [W1, W2] = [X1*W1, X1*W2 = [Y11, Y12 # X2*W1, X2*W2] Y21, Y22] # X1([32, 196, 3]) * W1(3, 16) = Y11([32, 196, 16]) --> *device1 # X1([32, 196, 3]) * W2(3, 16) = Y12([32, 196, 16]) --> *device2 # X2([32, 196, 3]) * W1(3, 16) = Y21([32, 196, 16]) --> *device3 # X2([32, 196, 3]) * W1(3, 16) = Y22([32, 196, 16]) --> *device4 # 由于matmul_2算子的输入Y 是 matmul_1算子的输出，但两者的切分策略一致 # matmul_1.shard(((2, 1, 1), (1, 2)),) <--> matmul_2.shard(((2, 1, 1), (2, 1)),) # 既不需要在中间插入额外Tensor Redistribution变换，直接可以使用各设备上的值进行运算 Z = matmul_2(Y, V) # 由于对Tensor Y的切分策略相同，不需要再进行Tensor Redistribution，所以此处直接使用每台设备上相应的Y*切片即可 # V在切分之后，分发给不同的设备进行并行计算 [2,2] * [2,1] --> [2,1] # [Y11, Y12 * [V1 = [Y11*V1 + Y12*V2 = [Z1 # Y21, Y22] V2] Y21*V1 + Y22*V2] Z2] # Y11([32, 196, 16]) * V1([16, 768]) = Z11([32, 196, 768]) # Y12([32, 196, 16]) * V2([16, 768]) = Z12([32, 196, 768]) # Y21([32, 196, 16]) * V1([16, 768]) = Z21([32, 196, 768]) # Y22([32, 196, 16]) * V2([16, 768]) = Z22([32, 196, 768]) # Z11 + Z12 = Z1 All-Reduce(SUM) # Z21 + Z22 = Z2 All-Reduce(SUM) # 其中由于Y11*V1、Y12*V2、Y21*V1、Y22*V2分别在不同的设备上，因此要执行加法操作的话，必须要插入All-Reduce算子进行SUM的规约运算 ``` 综上，1、2两点是自动并行实现的基础，总体来说这种分布式表达打破了数据并行和模型并行的边界，轻松实现混合并行。 从脚本层面上，用户仅需构造单机网络，即可表达并行算法逻辑，框架将自动实现对整图切分。 - 切分策略搜索算法 当用户熟悉了算子的切分表达，并手动对算子配置切分策略，这就是`SEMI_AUTO_PARALLEL`半自动并行模式。这种方式对手动调优有帮助，但还是具有一定的调试难度，用户需要掌握并行原理，并根据网络结构、集群拓扑等计算分析得到高性能的并行方案。为了帮助用户加速并行网络训练过程，在半自动并行模式的基础上，`AUTO_PARALLEL`自动并行模式支持并行策略传播（Sharding Propagation），能够有效地降低用户手配算子切分策略的工作量，算法将切分策略由用户配置的算子向未配置的算子传播。 为进一步降低用户手配算子切分策略的工作量，支持切分策略完全自动搜索。为此，围绕硬件平台构建相应的代价函数模型（Cost Model），计算出一定数据量、一定算子在不同切分策略下的计算开销（Computation Cost），内存开销（Memory Cost）及通信开销（Communication Cost）。然后通过动态规划算法（Dynamic Programming）或者递归规划算法（Recursive Programming），以单卡的内存上限为约束条件，高效地搜索出性能较优的切分策略。 策略搜索这一步骤代替了用户手动指定模型切分，在短时间内可以得到较高性能的切分方案，极大降低了并行训练的使用门槛。 - 分布式自动微分 传统的手动模型切分除了需要关注正向网络通信还需要考虑网络反向的并行运算，MindSpore通过将通信操作包装为算子，并利用框架原有的自动微分操作自动生成通信算子反向，所以即便在进行分布式训练时，用户同样只需关注网络的前向传播，真正实现训练的全自动并行。 #### 自动并行代码 1. 张量排布模型 - [tensor_layout](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/tensor_layout)：这个目录下包含了张量排布模型相关功能的定义及实现。其中`tensor_layout.h`中声明了一个张量排布模型需要具备的成员变量`tensor_map_origin_`，`tensor_shape_`和`device_arrangement_`等。在`tensor_redistribution.h`中声明了实现张量排布间`from_origin_`和`to_origin_`变换的相关方法，将推导得到的重排布操作保存在`operator_list_`中返回，并计算得到重排布所需的通信开销`comm_cost_`, 内存开销`memory_cost_`及计算开销`computation_cost_`。 2. 分布式算子 - [ops_info](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/ops_info)：这个目录下包含了分布式算子的具体实现。在`operator_info.h`中定义了分布式算子实现的基类`OperatorInfo`，开发一个分布式算子需要继承于这个基类并显式实现相关的虚函数。其中`InferTensorInfo`，`InferTensorMap`和`InferDevMatrixShape`函数定义了推导该算子输入、输出张量排布模型的算法。`InferForwardCommunication`，`InferMirrorOps`等函数定义了切分该算子需要插入的额外计算、通信操作。`CheckStrategy`和`GenerateStrategies`函数定义了算子切分策略校验和生成。根据切分策略`SetCostUnderStrategy`将会产生该策略下分布式算子的并行开销值`operator_cost_`。 3. 策略搜索算法 - [auto_parallel](https://gitee.com/mindspore/mindspore/tree/r1.9/mindspore/ccsrc/frontend/parallel/auto_parallel)：这个目录下实现了切分策略搜索的算法。`graph_costmodel.h`定义了构图信息，其中每个点表示一个算子`OperatorInfo`，有向边`edge_costmodel.h`表示算子的输入输出关系及重排布的代价。`operator_costmodel.h`中定义了每个算子的代价模型，包括计算代价、通信代价和内存代价。`dp_algorithm_costmodel.h`主要描述了动态规划算法的主要流程，由一系列图操作组成。在`costmodel.h`中定义了cost和图操作的数据结构。 4. 设备管理 - [device_manager.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/device_manager.h)：这个文件实现了集群设备通信组的创建及管理。其中设备矩阵模型由`device_matrix.h`定义，通信域由`group_manager.h`管理。 5. 整图切分 - [step_auto_parallel.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/step_auto_parallel.h), [step_parallel.h](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/ccsrc/frontend/parallel/step_parallel.h)：这两个文件包含了自动并行流程的核心实现。首先由`step_auto_parallel.h`调用策略搜索流程并产生分布式算子的`OperatorInfo`，然后在`step_parallel.h`中处理算子切分和张量重排布等流程，对单机计算图进行分布式改造。 6. 通信算子反向 - [grad_comm_ops.py](https://gitee.com/mindspore/mindspore/blob/r1.9/mindspore/python/mindspore/ops/_grad/grad_comm_ops.py)：这个文件定义了`AllReduce`和`AllGather`等通信算子的反向操作。 ### 自动并行案例 MindSpore2.0及之前版本在启动半自动和自动模式进行训练时，**必须**通过`model.train(*args, **kwargs)`接口进行训练，不支持自定义循环进行网络训练。 #### 半自动并行 半自动并行模式相较于自动并行模式，需要用户**手动配置**每个算子的**shard**接口对并行策略进行调优。 以`SemiAutoParallelNet`为例，在半自动并行模式下的脚本代码如下，`MatMul`的切分策略为`((1, 1),(1, 2))`，指定`self.weight`在第二维度上被切分两份。 ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class SemiAutoParallelNet(nn.Cell): def __init__(self): super(SemiAutoParallelNet, self).__init__() # 初始化权重 weight_init = np.random.rand(128, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.weight2 = ms.Parameter(ms.Tensor(weight_init)) # 设置切分策略。在construct中fc的输入有两个，第一个输入是x，第二个输入是权重self.weight # 因此shard需要提供一个tuple元组，分别对应每个输入tensor在对应维度的切分份数 # (1,1)表示输入x的每一维度都没有切分 # (1,2)表示在self.weight的第二维度上切成了两份 # 切分的过程是在图编译的过程中，在编译完成后，self.weight的shape就会发生改变 self.fc = ops.MatMul().shard(((1, 1),(1, 2))) self.reduce = ops.ReduceSum() def construct(self, x): # x shape [64, 196, 128] self.weight shape [128, 128] # x 不切分， self.weight在第二个维度切分2份： [w1, w2] # 内部并行计算逻辑： # x([64, 196, 128]) * w1([128, 64]) = x1([64, 196, 64]) --> *device0 # x([64, 196, 128]) * w2([128, 64]) = x2([64, 196, 64]) --> *device1 # 结果：[x1, x2] 此时x1、x2还分布在各自device上 x = self.fc(x, self.weight) # 在construct函数中去初始化并行调用operation算子时，相当于用户没有设置matmul算子的策略。 # 那么默认的策略会自动配置数据并行，即((8, 1), (1, 1))。其中8表示用户此次运行的卡数 # 显然当前算子的策略和上一个算子策略不匹配（上一个算子的输出是这个算子的输入时）：shard(((1, 1),(1, 2))) <--> shard(((8, 1), (1, 1))) # 因此需要引入Tensor Redistribution变换操作: # 将： x1 [64, 196, 64] x2 [64, 196, 64] --> [8, 196, 128 # 8, 196, 128 # ..... ] # 8份 # 显然要完成这部分操作，要同时经过StrideSlice切片和All-Gather算子拼接同步结果，再经由Scatter算子分发重新切分的Tensor数据 # 此时每台设备（8 device）上x的shape均为 [8, 196, 128] x = ops.MatMul()(x, self.weight2) # 内部并行计算逻辑： # x1([8, 196, 128]) * w([128, 128]) = x1([8, 196, 128]) --> *device0 # x2([8, 196, 128]) * w([128, 128]) = x2([8, 196, 128]) --> *device1 # .......8张卡同时运算 --> *device(2~7) x = self.reduce(x, -1) # 由于self.reduce没有配置并行策略，默认的策略为shard(((8, 1, 1),),),因此并行策略与上一个算子策略相同，故不需要插入额外的算子进行重排操作 # 直接可以并行计算得到每台设备的x值 return x init() ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.SEMI_AUTO_PARALLEL) net = SemiAutoParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` #### 全自动并行 自动并行模式，融合了数据并行、模型并行及混合并行的分布式并行模式，可以自动建立代价模型，找到训练时间较短的并行策略，为用户选择合适的并行模式。MindSpore提供了如下的三种不同的策略搜索算法： - `dynamic_programming`：动态规划策略搜索算法。能够搜索出代价模型刻画的最优策略，但在搜索巨大网络模型的并行策略时耗时较长。其代价模型是围绕Ascend 910芯片基于内存的计算开销和通信开销对训练时间建模。 - `recursive_programming`：双递归策略搜索算法。对于巨大网络以及大规模多卡切分能够保证瞬间生成最优策略。其基于符号运算的代价模型可以自由适配不同的加速器集群。 - `sharding_propagation`：**切分策略传播算法。由配置并行策略的算子向未配置的算子传播并行策略。在传播时，算法会尽量选取引发张量重排布通信最少的策略。关于算子的并行策略配置和张量重排布，可参考[半自动并行](####半自动并行)。** 用户可以通过如下代码去设置上述的策略搜索算法： ```python import mindspore as ms # 设置动态规划算法进行策略搜索 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="dynamic_programming") # 设置双递归方法进行策略搜索 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="recursive_programming") # 设置切分策略传播算法 ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.AUTO_PARALLEL, search_mode="sharding_propagation") ``` > - 在`sharding_propagation`模式下，算法根据用户设置的`shard`策略传播到整个模型，在`dynamic_programming`模式下，用户设置的`shard`策略也会生效，不会被搜索出来的策略覆盖掉。 > - 在全自动并行模式下，如果需要对某个Cell里的所有算子手动配置数据并行策略，可用Cell.set_data_parallel()统一设置。 #### 混合并行 在MindSpore中特指用户通过手动切分模型实现混合并行的场景，用户可以在网络结构中定义通信算子原语`AllReduce`和`AllGather`等，手动执行并行流程。此时，用户需要自己实现参数的切分，算子切分后的通信等操作。例如下面的代码示例： ```python import numpy as np import mindspore as ms from mindspore.communication import init from mindspore import ops, nn class HybridParallelNet(nn.Cell): def __init__(self): super(HybridParallelNet, self).__init__() # 以下2卡运行的场景为例子，实现分布式矩阵乘法来模拟单卡矩阵乘的结果。 # 即原始的逻辑 # 输入x,weight的shape分别为(32, 512), (512, 128) # 经过计算：matmul(x, weight) # 输出结果shape为(32, 128)的tensor # 下面我们手动实现上面的矩阵乘法逻辑 # 我们需要手动的指定当前权重的切片的shape,我们希望在matmul的相关维度进行切分。相关维度切分的情况下 # 需要对matmul的结果进行AllReduce操作，确保数值和单机的保持一致 # # 分布式逻辑 # 输入x,weight的shape分别为(32, 256), (256, 128) # 经过计算 output = matmul(x, weight) # output = allreduce(output) # 输出结果shape为(32, 128)的tensor weight_init = np.random.rand(256, 128).astype(np.float32) self.weight = ms.Parameter(ms.Tensor(weight_init)) self.fc = ops.MatMul() self.reduce = ops.AllReduce() def construct(self, x): x = self.fc(x, self.weight) x = self.reduce(x) return x init() ms.set_auto_parallel_context(parallel_mode=ms.ParallelMode.HYBRID_PARALLEL) net = HybridParallelNet() model = ms.Model(net) model.train(*args, **kwargs) ``` #### 数据导入方式 在并行训练中，支持三种数据的导入方式： - 全量导入。仅在**半自动**和**全自动**并行模式下生效。用户可以通过`set_auto_parallel_context(full_batch=True)`开启。开启全量导入之后，在自动并行流程中认为读入的`batch`是一个网络输入的完整shape。例如，在8卡训练的情况下，假设每张卡`dataset`返回的shape是`[32, 8]`，那么当前一个迭代训练的训练的数据即为`[32, 8]`。因此，**用户需要保证每卡在每轮迭代输入的数据是一致的**。例如，确保每卡数据集的`shuffle`的顺序是一致的。 - 数据并行导入。用户不设置 `full_batch`的情况下，每卡读入的数据是当前训练迭代的一个分片，因此要求每卡读入的数据内容**不一样**。例如8卡训练的情况下，每卡读入数据的`shape`为`[32,8]`，那么当前一个迭代训练的数据总量为`[32*8, 8]`。 - 模型并行导入。模型并行导入的方式主要针对图像领域中图像尺寸太大无法在单卡进行计算时，直接在输入流程上就对图像进行切分。MindSpore在`set_auto_parallel_context`中提供了`dataset_strategy`接口，用户可以通过这个接口配置更加灵活的输入策略。注意，当用户使用此接口时，需要确保`dataset`返回的`tensor`符合对应的切分策略。如下代码所示： ```python import mindspore as ms # 设置输入在第1维度上进行切分， 此时要求用户确保dataset返回的输入在第1维度上进行切分 ms.set_auto_parallel_context(dataset_strategy=((1, 8), (1, 8))) # 相当于设置full_batch=False ms.set_auto_parallel_context(dataset_strategy="data_parallel") # 相当于设置full_batch=True ms.set_auto_parallel_context(dataset_strategy="full_batch") ``` 因此，在用户设置上述的配置之后，需要**手动**设置dataset的获取顺序，确保每卡的数据是期望的。 ### 流水线并行 [MindSpore流水线并行设计原理及使用实践](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/parallel/pipeline_parallel.html#) ## MindFormers 并行手册 ### MindFormers 并行设计 MindFormers集成了MindSpore原生的并行能力，在`transformer API`中利用配置化方式对Transformer网络做并行配置。 ![parallel_mindformers](./assets/parallel_mindformers.png) MindFormers支持基于`Transformer API`开发的大模型通过配置化接口进行并行配置，主要提供两种使用方式：1. config配置文件； 2. Trainer高阶接口； ### config 并行配置 配置文件介绍及使用请参见：[MindFormers Config 配置]()，主要用于`run_mindformer.py`脚本启动时使用。 用户可通过提供的并行模块和关键字，完成并行策略的设定： - use_parallel: 是否开启并行 - parallel: 自动并行配置，相关入参支持可参考：[mindspore.set_auto_parallel_context](https://www.mindspore.cn/docs/zh-CN/r2.0/api_python/mindspore/mindspore.set_auto_parallel_context.html) - parallel_mode: 并行模式，0-dataset数据并行, 1-semi半自动并行, 2-auto自动并行, 3-hybrid手工实现并行 - gradients_mean: 是否在梯度AllReduce后执行平均算子。通常半自动并行模式下为False，数据并行模式下为True - enable_alltoall: 允许在通信期间生成AllToAll通信算子的开关。通常仅在MOE场景下打开，默认False - full_batch: 在auto_parallel模式下加载整个batch数据集时为True。半自动并行模式通常设置为True，数据并行模式必须设置为False，否则会报错 - search_mode: 策略搜索模式，有三种，分别是recursive_programming，dynamic_programming和sharding_propagation。仅在全自动并行模式下生效，其他模式不生效，实验性接口，谨慎使用 - enable_parallel_optimizer: 数据并行训练时对权重更新计算进行分片。优化器并行开关，在数据并行训练时默认会将模型权重参数切分成device_num份，与parallel_config中optimizer_shard保持一致；半自动并行时默认将模型权重参数切份data_parallel份 - strategy_ckpt_save_file: 保存并行切分策略的路径。 - parallel_config: 并行策略配置，相关入参配置可参考[transformer.TransformerOpParallelConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/modules/transformer/transformer.py#L197) - data_parallel: 数据并行 - model_parallel: 模型并行 - pipeline_stage: 流水线并行 - optimizer_shard: 是否开启优化器切分。优化器并行开关，通常在半自动并行模式下生效，与parallel中的enable_parallel_optimizer保持一致，默认将模型权重参数切份data_parallel份 - micro_batch_num: 流水线并行的微批次大小。pipeline_satge大于1时，开启流水并行时使用，此处需满足micro_batch_num >= pipeline_satge - gradient_aggregation_group: 梯度通信算子融合组的大小 ### Trainer 并行配置 MindFormers Trainer接口介绍请见：[MindFormers Trainer API]() 利用 `Trainer.set_parallel_config(**kwargs)`来为模型设定切分策略：[set_parallel_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L690) 使用样例： ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig ############################################# 并行环境初始化 ################################################## def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) ############################################# 并行环境初始化 ################################################## def main(use_parallel=False, run_mode='train', task='text_generation', model_type='gpt2', pet_method='', train_dataset='./train', eval_dataset='./eval', predict_data='hello!', dp=1, mp=1, pp=1, micro_size=1, op=False): # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=2, learning_rate=0.001, warmup_steps=100, sink_mode=True, sink_size=2) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, pet_method=pet_method, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) ############################################# 设定并行策略 ################################################## task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) ############################################# 设定并行策略 ################################################## if run_mode == 'train': task.train() elif run_mode == 'finetune': task.finetune() elif run_mode == 'eval': task.evaluate() elif run_mode == 'predict': result = task.predict(input_data=predict_data) print(result) ``` ### 并行注意事项 -`data_parallel` * `model_parallel` * `pipeline_stage` <= `total_device_num`; -`micro_batch_num` >= `pipeline_satge`，`micro_batch_num`的大小通常越高，流水并行的效率就越好，但是往往会导致**编译时间过长和Task Stream超限2016的问题**，请根据实际情况调整该数值； -`model_parallel`的大小通常和`Transformer`中`hidden_size或vocab_size`的大小相关，需要满足`model_parallel` % `hidden_size` == 0,不然会导致相应参数无法切分，从而引入**额外的算子重排，引起性能劣化**； -`pipeline_satge`的大小通常和`Transformer`中`num_layers`的大小相关，建议配比满足`num_layers` % `pipeline_satge` == 0,使得`Transformer`的`Layer`层可以被均匀切分到不同的卡上，如果无法被均匀切分，容易导致**编译时间过长**或者**编译失败** - LLM 模型相关配置说明： - model: 模型配置 - arch: 模型类配置 - type: 模型类 - model_config: 模型参数配置 - type: 模型参数配置类 - checkpoint_name_or_path: 评估时不指定权重，模型默认加载的权重名 *# 以下配置针对大规模语言模型推理* - top_k: 从概率最大的top_k个tokens中采样 - top_p: 从概率最大且概率累计不超过top_p的tokens中采样 - do_sample: 使能top_k或top_p采样，为False时top_k和top_p均重置为1 - use_past: 使能增量推理，为True时为增量推理，否则为自回归推理，使用时请参考[模型支持列表](https://gitee.com/mindspore/mindformers/tree/dev/docs#text-generator) - max_decode_length: 文本生成最大长度（输入长度统计在内） - repetition_penalty: 重复文本惩罚系数，该值不小于1，等于1时不惩罚 pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/Pet_Tuners_Design.md000066400000000000000000000072121476650141400321160ustar00rootroot00000000000000# 低参微调 当前基于Transformer结构的LLMs取得了显著的成就，如何更好地利用已经开源的语言预训练大模型来应用到自己的应用场景中，往往需要使用微调来达到较好的。微调又分为两种：全参微调和低参微调，使用全参微调需要较大的算力，这往往对于普通开发者来说很难获取到足够大的算力资源来进行自己的实验；此时就需要使用低参微调（Parameters Efficient Tuning），只需要训练极少量的参数就可以接近全参微调的效果，以此突破算力限制。 当前，MindFormer是专注于Transformer结构的大模型的开发，对外使能开发者开发大模型，训练大模型；MindPet则是专注于低参微调算法的开发，对外提供低参微调算法接口，使能开发者修改自己开发的模型进行低参微调。但是，以上两个工具包均是使能各自的开发，而对于完整的大模型开发，则需要大模型的开发->大模型训练->低参微调的流程链路，当前存在断点，则希望在本工程中将MindFormer的能力进行拓展，能支持MindFormer开发的大模型自动的链接上MindPet上的低参微调算法，加速大模型的孵化。 ## 设计概述 由于是两个完全独立的开发组件，需要结合两者的功能来完成对于大模型的微调，此时使用适配器设计思路去整合两个部件则是最合适的。由此整体的设计如下图： ![整体设计图](assets/MindPet.png) 在本部分则是实现上图绿色部分的功能，调用微调组件的APIs，针对用户输入的模型进行适配修改，然后返回给用户，用户可以将其送入到MindFormer的Trainer接口中，进行后续的模型微调训练。 ## 接口设计 **PetAdapter** ```python class PetAdapter: r""" PetAdapter is the base class of adapter to modify the pretrained model. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): """Add efficient tuning parameters to ptm.""" raise(NotImplementedError("should implemented by the certain tuning algorithm.")) @classmethod def load_ckpt(cls, model: nn.Cell, config: PetConfig): """Load ckpt of ptm.""" pass @classmethod def get_pretrained_model(cls, config): """ Get pretrained model from config. """ pass @classmethod def freeze_pretrained_model(cls, model, pet_type:PetType): pass ``` PetAdapter是所有微调算法适配器的基类，针对于具体的微调算法，一般只需要实现实现`get_pet_model()`接口，将微调算法的改动添加到用户的模型中。 **PetAdapter.get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None)** ``` text 功能: 获取微调算法适配的预训练模型 参数: - model(nn.Cell)：原预训练模型，支持用户直接输入模型实例，默认是None，当输入是None值，则是从config中由用户输入的模型实例化MindFormer中支持的模型。 - config(PetConfig)：微调算法的配置，包含微调算法的超参或者需要实例化的预训练模型。 返回值： 适配了微调算法的预训练模型(nn.Cell) ``` **PetAdapter.get_pretrained_model(cls, config: PetConfig)** ``` text 功能: 实例化预训练模型，当用户不传入模型实例时 参数: - config：微调算法的配置参数 返回值: 返回预训练模型 ``` **PetAdapter.freeze_pretrained_model(cls, model, pet_type:PetType)** ``` text 功能: 根据微调算法类型冻结预训练模型权重 参数: - model：需要冻结权重的预训练模型 - pet_type： 微调算法类型 ``` pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/Pipeline_Design.md000066400000000000000000000007451476650141400315770ustar00rootroot00000000000000# Pipeline MindFormers大模型套件面向任务设计pipeline推理接口，旨在让用户可以便捷的体验不同AI领域的大模型在线推理服务，当前已集成10+任务的推理流程； ![输入图片说明](https://foruda.gitee.com/images/1673432339378334189/fb24c2fe_9324149.png "image-20230104093648200.png") MindFormers大模型套件为用户提供了pipeline高阶API，支持用户便捷的使用套件中已经集成的任务和模型完成推理流程。 pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/Trainer_Design.md000066400000000000000000000005561476650141400314360ustar00rootroot00000000000000# Trainer - Task Trainer 结构 Task Trainer开发依赖于MindFormers套件中的注册机制，方便开发者使用MindFormers套件提供的各个模块快速完成整网的搭建，各个模块之间可以做到有效的解耦。 ![输入图片说明](https://foruda.gitee.com/images/1673431864815390341/da621a72_9324149.png "image-20230103154930330.png") pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/assets/000077500000000000000000000000001476650141400275135ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/design/assets/MindPet.png000066400000000000000000000332031476650141400315620ustar00rootroot00000000000000PNG  IHDRA:kesRGBgAMA a pHYsod6IDATx^ |T/UQ>j RIX6fl %j^Wn\Zƒm%-vcmMDbh c/6oμ<3g>:ϙ3g&w<9FGG5![orN gΜ?{o:"'dO?RSOaĉ'E%_>:?:dރkJBXqqq@1Fn;}7ݑCqG6{{w#BHMMM#Ȱ:q WO_0KN8`=?8Frjx饗טhٖz}WoZv?͋]C|ǿOV?PrR{y$1ʩG}Ԗaѭ[a}Wb[~}w2ssӝߒXa׭w%9}lY>5S3gQ,KNa6>Y$-zn,@!v̙v\ 1^ #D\ٳjyYhPZZGFFd7<3D >`=c#dGz5-;#~3b}||4cV,@ _x{{e/v߃8]vH&{o۾MYrv$+-̗ ̘N,̀fKW#G^M}CV/O!nL\d昷zkOH^!n}:6?TqƱH_jۼgdKsqYOI];wҫxUIVݢ`['_ ?4782!Oƍ?~\!aq\7a,@vUz Bz(3z'9W|TDmhz?0yR2e豾!ħiwN~' y0ǜ|qĻҶuse*RIѐhŢwg{2iLb󹖲XxzUW=:3l޴~tocmw6M}6Yxӕ>3 h$O{y383W_w,@RkO5>TǜhHh{Sb߁~<5|ribkiNo϶\{'q=vx+],GƇ3e!"g@mNJcݰ F1[(wȟvD~x@w{P\&BSfy6 aZN.Յvл{/nb^SYΛWJX҈U=m]@O]ccb*Z+׋_LDN///}nǾ^;0쾝uLzgPlCqG];Esl;[}0m}s}:FYC933/H>'G|J]@.c'<l=͊rcpe%2L\ Q(,]}TL5z&Oԯv0Wh ZW KQqdɻL"ʷxWtx6>u:Li.mZ|>9rkK> ZM^27,iMM+X˾sOfg϶O}rY-S-1|Aɂ, Kі(vxa&JӳDC4i għȋ0ujzn4zhT.r5@X/~h2Z[Rqy{m}Bs\`߇V}w"Uc[L+>g--Il ߆_&[ӓK+l<~Nߖڴ㪫KDq1ǏuzλoirR`M=^'XC>?mY|M+ yipL>ݔIPQ=,a#ZmOd ӑ`2sSwaeok|ۙ0fFe F鑹~ƑujG۽ !3DA즚66NܱEUMߏQӼEu_V̧ye0rf3Ī):~2RŏWD휯 fLehߺx5/Fs猼mMWha»=sG3mN?o7e!>L+[l9SxשBś8-ߕ_4'Čs&W9,͛U  l'ZEñ*7Ϭh̬h>^?f˕x̖b>b9k]|KSi~(@ҳ}lEl 4jKz)}&(;et&eX迍6=UfI࿢#ع@u;͡7g.#'댏>%L-.Q뵡ޅe%ۏA6O-^ݝ:S[?C[j͓mg(?Ѝ}7v|bQ9'FX!mG oqrr*ػ'GWف?zS4nlRw*oM˃L*1w=hq2L>>);c=YP剢՟.h/wvfLH=o2g1]V;cXlֈ~J({x(qXOiW<8Gs?D *aNJKK˝{(w ￬ U+-7/Pt͞O;ܡm{/n~7$}f3oFy\ck^)Xʯ45}f%2ˆvbǚ5 l>٣}k9F50~ Q\Ww=lj^SF:VKYFPkGR`[C̜fͧrk§-fw2F=؝!>TdW<53=144 sXW_^Yq̶@WՔ<^ڶ{0֢].K4'k^{/)tɢe1>%}_ɶw}:2>\\[b{ml0 To1wN^0Ș^d\gOt{c-:g-Bt\/k9V:`-\1B| 9B,y"Fی]ܷb0?^W]_^" -0˹Wqܰώ? #.=(.q+ϿeKRќH|#%}g#=ljPB?MN7wz}+@Ϟ ޳`vHOHo!![|-uZ}WM!RoW--Vy ;y>4[2<%OEY yH%qfnQ5ȩ#o_ߗџ,C Üo?o#t}x~^VWemqر$[nH7ClJZ?TһPNQDxdM^]/0?K4ز08qH4~?~yH 0>,"nKV0]c Qn֬۴~ k}9g28;8|4NIzj| 1>lq۽)%y㤸pU _=4^˖~"e/ip{*lӓ_*ӺObY8}|*͟O'Nrdܨvg-)(`0dX9vzwO >}ndȹ7Musz 7ܐ6iҍie~vڴirE ÐaJ CP2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P{+uDNWG?d_>~x9)h 999 JH '$I~gdV}蚌!ƷosǾ꒯ʩ@p=>87O<)F6N8B!Üt!ieY5~—oTâF8}dgSYv6N8aH 0'!F],N@ߜ?,; 'l21\;m}}}i-e!8`@xaihq,(ʩͲ{ɂCqƁ v8܌& H@kA"  ƒ s̕\$-'c4FlZ ZYp6N8dcXH[D 6N8d Ba*d Ba*d Ba*d Ba*d Ba*d Ba*d Ba*d r&ZX,O.eV}fx3t:kBJńrź1s~~cg 03.׼ʮqտuScynKa J]M-f4i5Je3(}4;D X*fh4dUV[kG,[)K뢊c-¦-qG悲|ٞW{U<@qO`a'x{bK^q\"z*O:ƕjk5_m%u,nbaX~cccTZ-s59'v0ǩ0JMڔ9 ™ + pE{JfBY^2Hlb\2 T U!@U0P2 T UE&Zm7L( {?{` ɰrYh{" rΚ|ϐW|AZH>Q,='_d cVZǯ+P$q!I =ƺvn6 D8ZjE+0a%.eY ͉.d .oq+ 90'Ioz:ZHR{k:{GK cO{EіZ^BaqcwW3>ַA49xuJж|ٖ(g^v(ZEƿG;;1~k9|~cf}>}֮=|ub/gX֕M{|qD4^d44 ɧ1kٚե}n7V:[ 9ٹiQ,El'ݖ7y{ >9aիseAӛ+g.$Əj!vK ЊX k󴮽ziJbT ꘵V<mk Ii6@u첻%r&UksdamֿrJ6>mPnAqZS]q +3n|wgC%&lwzN.޶l'wSAz^EqjEH9-ۜ-߉89-\șO'2}*ObӬ|!HbԒIΞă*1**棠"Ч߲!+ٖHf ,7 Kӱ񃅜>sec֫X ~⁣iUuN7)ǿ~J=.Z6P9`>NR~~~cg pi1nc٠U;1{ݩI92|֤mGz4ěxP.?H9WS_F=NNl.yRMjlWa; cΊF~LN|Yk, $N {c^G#X%׳S#Y)~c藨;Yx{|!dAižǭYU*K,D eMrѾ(nnZb6Iʉ˱7:::sC >__ &M}5:s6gYZW\mٽea{rxOwd!4bKU .a#ĝ@18=ͬv B cF;M9|g^-wrZV=տh[Tƞg>|Lȝ# rm)h_x.ok>ц=: gSħN14}gV瑯E@2Dގu>"fxdUMZO'sj[7;0q|}6׬3^ㅣ}eSڤr mKKgئhiѥ]-2I6,Jx4إ^w4Q^3gDbz-t*jAŭ1mxϸ]c}edςl~,i{a MO!`b|Ob!ʡ_RZTaD,^3]av;gz ȰxİMsVM2L6d:M۳,4J[X`ccc~(4k#쀱_ߘx;V{ SA ۴%bף0cѫMg>rg|Azc*ʃ +2~j*0J:k4y1.;QRw\o4c@ũІmvܵG^]+*|^l] 5/{ƭ'&|̣6ˀ@1']sK0̜Zvc=t l|پp{ѫ\u J}l(6އ_]xu3t<\!ACŭm,4L*ZNȘG݊CZ<N6 k9$؉wȘU [Ds.[ZceYA^~q]c.o}H2 M!&DT(È]eH%}kQKlyn&Yk־~~vIM˞"  ؖL(È#Ǚ$mh5 |>L GBJ*b倧1tj\`F5VԶs $o/9j(gsbHjh.ee^ŷb'QG[lTV0;a^8Wb.4V(Q‘P0bƒ_جo f/=20UwHPFBx qUriL~AdE`.؋~d85օ#B@ZdrW,,a%9K7'K4FGG=nex^7S^!hed0E*ܾm o]ûzCDv&GO]A81ڿQl1 4nݚH3Vhl:ET559/{P0Ta:҆& !7/./''sJD[g;Yqsap!MK/,d}!2ppiLARbB=TT-OEIEW)%84p )!q0^ӷXxm Z&wg5O85? !go46ui0r1wǚv1\0<58+= m͍Z/}_RgR Mh\WXn"[y-׽&l.hfbmX}Ȭo';P,ˁ@"alq Z%L",67$Ɛ;4.0,ˍD0`/(@x`V-R ֖G]L^ľ-" #HУo7 Bxˁelȅf妱U~>؊JQksuFwڽ;ET3[Qc5J_O}Fz%"E< VMZMM"jMXFsXI]g&mnn ca^10T-]]oTha ?$U!%uN ca\I34X\ԕ6BXu Ř6R0ƒ_O[bjd՟(BM3|fkcռW%DBxˁ ܵ[ ;ǨLVS^yDBvГeM<χYo+4 MBLJ%r _g][A\z1j v͌+mV/[E3/3,4YX do f3=Tu:#:{(QXto_=llc-$<㬚v(3/3b Yyc-D~2 `x!jI@0P2 T U!@UgXEazq)^۔je9l,63gl[$3,sjrW@L똼 3HV%u5^U8~1C:P[0T |Z)',7xV]M%z݉w4jh- aN?jh+}ё]%$WJ- ŤG n\&OLɒzPԕЛMrNӎn=9mļp}M[= cW ֖ц(;^sc5[6ͧT+[iѦ~a`+( E$# ӐU4pѲhn#YŲ\ZHݻWQHB^oB1$ror|E.dN)nr_+۲ܸ~Cѥgߑ5calDmq [to=~1 U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P2 T U!@U0P21峄$Fp|]qƁ /=1$ vD>IlZ ZYp6N8dcrrr㏐ c.lq <0̟[YYPܟ_*P$F`܂Yp6N8dc͛wO>":ZʊKd9"Yv6N8dxr^?K K+:7}s(lq FGGgΜy!9!EyW}xRnNNιxK*^6C`Gz76N~;2,":::Z:Zn\}2_k_?~zŜs͛'F6N8!A!Ta*d Ba*d Ba*d Ba*d Ba*d Ba*dj̙3:$'$`{{n <(@b5kօ rrr뮢"95~;2y=Pjj*9 ###?EIahKtXUUUqq5kn@£?sc?yç?9$-Yd Iϟv Q sLGGGjj* iџ?e"昗^z Iv Bƈ~c7:::sC (:+]A2yhQU!@U0P2 T U!@U0P2 T U!@U,f͚% *:+]A2yhY]pvIVVP")7##j;CyĦfVVP")7##j;CyĦf6gΜjwu,DRboFFv8M2l޼y###{/$Z)Z"Yތhp ɹ165TUUdɒb|m&Qx!>olŗo4X_ᅟ[cGnhtW'NaͪU8v$N֭mOnOOoUlߌCqRpflAGWpǦ#=hF o^Ov|?"g_D;G~4lW?~M}@ ɫpUyšyo:?3 ZDa-b܍<19._k3Gг?d IԌoYC^ Fg{8?d v|tӆFhE&“qKx{<;Ddݷa/FeKwnntQV`/q:!Z~:ugotQVW4#. ъwkpwFzSm .8Μ93nlXGɓ'~=X. xO^qb1@?w`l 8v|ljpiVcǧ\{/tقXm~}4HJ$[ɑ#Gw}ɋ~,C.wyc zG_*'}!J|(>nn9sF}׮]܃puشFi 2i|膵. p73 DW4L\.#H,z9 LF>/ Zq!vz?.?##iy8.;l6n܈~X>߿f2;vPa||)9o:=<OM'ӵZ_7q+ >-EjqIGs+b?((=P/ݻǎ'|5hooG<<~CСCrxnFyOy}Ѳ9?e~muhZ5asq3Wrs'UV:Q(fX,ֽ Hl6"̪P(ZadddA'xdrE[lfchhHa|)~]/wNѫx-ܹG"9_ @(C6T&^D"ᥗ^pMD*Uc||2>D `͘,Yu;vdW󴶶fR)9ƍ1>>.3rrZ*ZHWX,ZΛ4ML' dTj*0*qA slsT.5;嘩AY<է)s)u`0~o^>h&j;v4'bb_˕};QVrݻW.˖S]7~ob79R.?ϚX^8dht1,q42ܹJsNzĹF]u;re_K>/rijYMӜu}+я_W6Ϝ`.ÇN. 4MntPx<`&-针UeYy0 CVOaOUݍn-fp"{_v>qT"8==]rcZ<#%Mv!".gYV9RWhLFԻ{addd^V3Q(yCcM%#fUx\ks?^\0dY:%l:::W $/A4Viv+6,˱`0h4<Զ.bqNv!j188hi[SO_'00 9m```;NDsV_ną\T7/t}}}%*|.WDBri:<[W%@t>ƃî]: ðD>,sBʖr_ɓ'K/J[ơ촼>[O]۶m&{1 \L bvwSO=%rd26kfdU}<8qr͛8Yݻwվ}fO Vgj/j-hy}vcǎe]vUk6k`bb(Lbǎ?N_|}޽_^ `)T +FQ.#75=H&x<jPzS-J)pMgX$ƪ[w)| j%NG?8o>:urv^+q˿KSFEtǎ׾>1atuu`C)r"Q iڢ 1"^mےL&xub{ݒ 6D 1X>|>O{eGXM鑑s&5XB"ӧ/|D[[۬~OLLXV#%·C}׿zJN]k"lUEqPZ=P_T1z0^Bܠ>6^+AreDj[$ h|ffrD"9I;x~P̆aػwc144u B PPV&Ն.3nw^! 'HDVirygWԟO(mId{x^KwrLˈFFjwP߿(+-rR9'1dfs}i[G$ia}uPH~L -GޡU+^W6LR)nf=e@,Ps.D[pەSWp8دZ,cN1O4-u}ݤ@/B^~9D5J_a۷o/i`Q,[n`?' iM]-2L:V> uxrAUA 'Fr9(~vrTJ9rD)ZFtz8rw1eT}uriT> XPDK:l bzF6S !A{b۶m%)oTKٵkFFF~a|+_qںqFDQ|K_SO=o|޽{Vm""ZlA0FjFS b `bzѬq{[n:(@cΝ)SKeΝ/85j`)Ei~iHvV曙pZκ/zQ`fk"bWlIF!""7lhd2 w\d2d2\.D"sNHDDT+lԁip\y&Ȉ$ ڵ ^lq/4Miqh]a&ɤ2N4May9n?|7Qva07@bjnjnʍUD65bFjUZ##"HU6fϘh6B!3Fֿ?@D+ xd;@ ~ཽFĺFFF4-b*4͒٘m8w5a&yU}whB!O3r9B̃PUQu,%~#Kx'b<" ,` ^daq+aմBPg&/u zc#~MFM<G:88P[Rjx,V5 6L4X" -]בJUTf6^iZv"K.DTod^D]r谬KWeԖ#JL&L3J-P(X,Vpz>#:e.F/V"zd2F4EX,B&&#!\L,'FdM!20(p1Q#!Kf4M$ Z'*EKRfh>sQ.ԄB!Km$fT8ɤlTR,-m@D7:b8O4O&̆ZcfjM=M6d?P=Cf60,YglPc0ب1ǃx<^2]]].C&|چ{9e6N8FhW'N6˂@6>vV.sNK7نD qQDi9`U( ,r Ԕ冋CzW}Vb*Ewê-˾~1h@Ξ7N袬Xwlڀ/W'ᶍ768^P烮Cܳgu1 z, W3"Uĭa kJ㿚'O_7"jZ}ѿ~6nXH/]//@Du+.^'-m(NM.Q?6>zZf6݄ntQV+WMǧeOGqkz96)s|6(+~o+x . ъ+UxpL5YukWч^np3ogz8{} v܁'.ъ/"DM7O܎h|L߿Ђ5?ğ3G`&wn[66DKﾇP$!нX=ԁ#xGزfl܊ Q-S„ǧqkC-` 6Ի(NNsZplU~3l;?]$Gx/M;>pҕFhٹe:ن-oƃ;5Yu* 6β .-gDWyq`h!XLDDDu` """+DDDTW]NA~Rh)oc={nV~و1XB{=>}zmܸ--#w """+DDDTW 6lPqDH$f]6P(8H$ZY;ax<fOG$q> ,nUZI$A&[a!,$IbD~Ŧܣ6%r-b[OdE"LLL :.iFFF H& a%EQR)D>T T2=8~^2 RB5lP* |it:`0(,ʼn>yF.tra4 |^PT*xebJpX^].ׂ`ttn[>D"3^E˼"N166VR}RvؘM6!ZN٠rСCwpX.788(~y|w .Ibh9ud)DzxfT@AuA ˅B}v.LӄF$Aoo^ (8\c%+Dve5E>UBbѲiT:Q[uYrd2F4dG  5+׾#^4N3h8.ؘwpp$@$Z)fU6E @&A0\-KѮ`>w.vrrRN4M) QEH$F+GFF,UDCCCBe-˨ksQFnQ g#N$b85~Ty:tXLflBL&IKF$`llzDmwboS(ٷC0Hmת@6j 4ez5ɓZ5`l2dz{{{nyb1*UXDWW|Q ju*Fz1<<\Ҩt @"Ç-U=1tvvZڮ)g-jl%g5x#rի|2NK=>CCC*dll pu`0xhbGߏl6;kzŢrj 6@n af{M ,jdMd1]̧ަDG D$bBCCCe$/_իWk1hZQujZz5VZE#fR*> V_4"jF8xk 6Q3[H_`D'<5K\~шh9K_aTpFDQ5͉T)J͋pFD+S_s^KٳhkkktqͥK`&&pe^#ҥKuZ< 6f """+DDDTW 6lQ] ;gql]=~ g]lDܸn < ߺu9ΆiH=s^<tnZ%""s诿 xpGź3\ioMǟ%0}ŸۆqU*35Kпԁ#X҂?Eצ >6S?[gH6n`6{] Z;d0:"@dxso-vMtl# .-wua}w`iܹ"""dKg+o 6vp8{""""bADDDu`$ \D"b ð+4H5x*BP* r^e1 2IDAT a.T( u]>:|>% `Ahi4M˅Nu9=6E@<_R媕C0R W߂@ 6LM3 j0䲢Z y<e*{P,+N2V˫VId2m+e6+UILF__~?x\/_}2hat]7L]t1M}ni4Mj83W܆Zr˦R+i4ﯸ|ث_=N5We>է3.l-/Sn###|>4̤Ų`tt,1߿ik(n%eB%e%jUخ:Muppв|\$i:ڒ*`0rU=#ٳq~" \D`.dYyx<|e)~鱼.U,bݕe!d2)/}}}v崡iSSS|>yD ATFsJA3D\m`uvvL˸nit{''N/" 255e6nNR{B!T _;@Ԟ!aAa V!Q# x 'j`o{TkN&sQ)"cAݻVTKdHD`0(J-3jOrm$B]]x8ԇh266V|0 ji\.t̕'qhPhJ I6(z~ Em*і\U -C^m뺼?|֥6ᰥVAu٣hI0fQicn?RW6K;VߺY"""Z9^~MEg?Fajm6lQ]1 ""bo"""+f6lQ]W_Z[޻r2rby3NN7HDDD'qmphٲ 6v^8Λ-(^<"""Ιlqoga也\<"""jf2WG{q-K??fO~lvl [6múwϼcǧ1^|c]6 6;Pƻ.6DDD|>n-ln m%PqΞgADDDp~ 6hyZWrը$DRnthq˭YWADs3 seLMMy7%"?FDDDu` """+DDDTW 6lQ]1 P(Jg2D".Ӳ8Dr0 ax/ 40L\Љ;LZ.h~_>_ ¾}>yE|d2~R)c)3W2`Νy.D,FDV3͖|>Հ(t? "jj2ˍajt1622~U*1fjE\&Y떬xit]ٜr d2\.b1Pccc5;K "jj2ȋ21PrY ð,'.@\XL^v Ned2iL"o/}kNkgr 122Rzو irr000`ɊطNeܞ={\& ʌSfa)`AT:.ssQmfNV?Dz5 Ðm= behhq~q,^yr?,~?ᰬ4MǡiZ2e|w,i>m~! "Yw"Sw9A,YzppJN+{5QP>Ƿ1 ZFD*4M\pqy\|y뷯O<\bb6k :z]v^%2 b2_4LNNP :G%Q ʪVAe]]]4\QCB4߿U0x <\r!b;UƠNB]N4Mhfir9y캺iXu !̚fѲQ-C3ө?1Ö 5k055 .d'Oti&ZjL&c\8Q)zPF6=mIJN Q'''K2 5P(@uƞjPnߪ1ת! СC( -t:-15X"(HDn# 9C`&$:::*͈ er궝M6auD,ݭrw\ DrW;[NmgSJ/|3=z?GGG^"zF144BBP(4'c!>Wn[vlYTvgVNU͎2UpB<G,+Ɠ-U=(VtvvgaVYƢ TZJKCQ'b >{ϳ,j٫m|>ڦDd/G͎2VQ`lTI#c13h80C]-z{{꼡P ]e'VȈSVZw,S&" N_~yh9rjawС)F#N[=˩g\r"4q՚hd!D"E٩p8"O$0F$AOO}N$@__l?tAxݾ@ Ax^2 Sۥ S֫\ R4hXh  "/D+|D 6V 4hl@ 4lP 4Vl`DTKJDnmmm{" DڀAi2 \ODTkebK8s6?MMDr5D7\֗f 6S1֙l [h&l?q|Bwކ7kHy{?{/+_WE|~ʈm@/wU--~j+`o hт޵W~[6XrF?Z;h96y hph9h8:5 h9z]lhmph9ٲqwn^}DDDDM7`e!""e """+D"\.WG&it1BRD""Մ};HX^/ ަx@ˈreQ瓙 `n]\ À+YVd+׶A샦iؼy ,WH$,C;&twwWڵ p!߁y@ `Ɏr9)p  jbpXV0 4N85Mˊw%h&Uׂlb6a&Ţ㾉$]K  ľRXip0;wZ+w0! "SOhIfDԌJfZR}j{Gd~)g>118]ꃁQlՐzw/.cccZ}h4*J:Q a_Ӵd2D#O]eBTg85*vߝjFIQ=UG,N^Y D^E֩C5Z}" 8Pl@4U+j W p@RT;;թxDu]5"3D5vKM^$sRDF +LӴ\k-LZ{5Bpj!TdRO@{PM"W6;>ZDDDDڛ͋82ADDD` """+v}%""bfbꪥՙF7~ooX:ptTDDDDѩih[-7_~z?7N'q-l{nC; \<"""jfWMO>ss߂? m;_ßBQsz—S?]Nw~.!W0zx֮–mC7DDD~}}_?]>t/6nX'_plj&ql~}ҢˍׯƝmtbm%^<\8ip)IENDB`pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/practice/000077500000000000000000000000001476650141400265325ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/practice/Bert_Example.md000066400000000000000000000424631476650141400314340ustar00rootroot00000000000000# 开发新模型--以Bert为例 本文将介绍如何在mindformers中构建一个自己的模型。这里将以Bert为例，构建其训练、推理的相关代码。 ## 模型注册 在mindformers中，我们主要使用注册机制来方便用户能使用AutoConfig、AutoModel等通用接口进行模型的调用。因此，开发一个全新的模型，我们也希望用户能使用库上的相应接口，对模型进行注册。 以Bert模型为例。首先，我们通常将模型相关的文件放置在`mindformers/models`文件夹中。为了新建立一个模型，我们在`models`路径下，新建了一个`bert`文件夹。在该文件夹中，我们可以放置模型文件、配置文件、处理文件、分词文件等与模型相关的文件。bert模型注册的代码结构如下： ```python # mindformers/models/bert/bert.py from mindformers.tools.register import MindFormerRegister from mindformers.models.base_model import BaseModel @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertNetwork(BaseModel): _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, config): super(BertNetwork, self).__init__(config) def construct(self, **inputs): # bert model code pass ``` 在上述代码中，有三点需要详细介绍： * bert模型通过`MindFormerRegister`的`register`函数，将bert模型注册到`MindFormerRegister`的`registry`字典中，方便后续的调用。具体注册方式，可以参考[register.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/tools/register/register.py)。 * `BertNetwork`需要继承`BaseModel`，从而继承其自带的方法，包括`from_pretrained`、`save_pretrained`等，可以帮助用户快速从模型名称、路径等方式，生成模型。 * `_support_list`是模型自带的支持列表，其包含了模型所对应的支持列表。用户可以调用`BertNetwork.show_support_list()`方法，获取不同大小的bert模型的支持列表，从而选择自己所需要对应的模型结构。在support_list中，我们将当前的模型命名为"bert_base_uncased"。 具体代码，可以参见[bert.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert.py)。 ### 配置文件注册 对于`BertNetwork`而言，需要传入一个`config`文件进行模型的配置，其中包含`num_layers`、`embedding_size`等配置信息，当然，这是由用户自己决定需要传给模型什么样的信息的。对于这样的配置信息，我们也提供了相应的注册机制，可以方便用户快速地获取到模型的配置。在`bert`文件夹中新建`bert_config.py`，其配置信息的注册代码如下： ```python # mindformers/models/bert/bert_config.py from mindformers.tools.register import MindFormerRegister from mindformers.models.base_config import BaseConfig @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BertConfig(BaseConfig): _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, num_layers: int = 12, **kwargs): super(BertConfig, self).__init__(**kwargs) self.num_layers = num_layers ``` 同样地，在注册类的构造中，我们也需要完成`register`、`_support_list`等相关信息的注册，从而完成后期的调用。由于继承了`BaseConfig`基类，`BertConfig`也继承了其对应的方法，可以方便地进行配置类的生成和保存，详情可以参见[base_config.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert_config.py)。 完成配置文件之后，还需要在[`mindformer_book.py`](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/mindformer_book.py)中进行注册。 首先，我们需要在`_MODEL_SUPPORT_LIST`列表中添加对应的模型名称： ```python _MODEL_SUPPORT_LIST = OrderedDict([ ('bert', [ 'bert_base_uncased', ]), ]) ``` 注册完成之后，模型即可通过`AutoConfig`即可通过模型名称`bert_base_uncased`来获取其yaml配置文件的路径地址。 ### yaml文件配置 为了方便用户管理和修改其配置文件信息，mindformers中提供了读取`yaml`文件进行配置的方法。yaml文件可以放置在： ```text /configs/bert/model_config ``` 其包含的yaml文件的格式如下： ```text # /configs/bert/model_config/bert_base_uncased.yaml model: model_config: type: BertConfig num_labels: 1 ``` 用户只需要将对应的属性填写到`model_config`中，即可完成yaml文件的修改。 具体代码，可以参见[bert_config.py](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/models/bert/bert.py)、[bert_base_uncased.yaml](https://gitee.com/mindspore/mindformers/blob/dev/configs/bert/model_config/bert_base_uncased.yaml) 在完成配置文件和模型文件的构造后，我们就可以直接使用如下代码进行`Bert`模型的构造： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import BertNetwork, BertConfig # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 先生成config，再生成模型 config = BertConfig.from_pretrained(yaml_path) model = BertNetwork(config) print(model) ``` ## AutoModel/AutoConfig使用 Mindformers提供了更加泛化的`AutoModel`和`AutoConfig`接口，进行模型及其配置文件的构造。用户在完成上述的模型的构造和配置文件构造后，得益于`MindFormerRegister`, 可以直接使用`AutoModel`和`AutoConfig`调用刚才注册的`BertNetwork`和`BertConfig`进行使用，代码如下： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import AutoModel, AutoConfig # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 方法一：先生成config，再生成模型 config = AutoConfig.from_pretrained(yaml_path) model = AutoModel.from_config(config) print(model) # 方法二：AutoModel 可以直接通过config的yaml文件进行模型的生成 model = AutoModel.from_config(yaml_path) print(model) ``` 在Mindformers中，我们提供了从云上下载配置的功能。代码如下： ```python import os from mindformers.mindformer_book import MindFormerBook from mindformers import AutoModel, AutoConfig model_name = "bert_base_uncased" config = AutoConfig.from_pretrained(model_name) model = AutoModel.from_config(config) ``` ## Trainer构建 构造完成模型和配置文件后，就可以构建代码进行模型的训练、评估、预测等操作。同样地，mindformers提供了`trainer`类，通过`trainer`，调用模型、数据集等。在mindformers中，需要在trainer中构建对应的类，一般我们使用任务名称来建立`trainer`类。 ### 模型tranier构建 根据需要构建的模型任务类型，我们可以在`trainer`文件夹下的新建一个`masked_language_modeling`文件夹，在该文件夹下面，我们可以构建对应的任务的文件，如：`masked_language_modeling_pretrain.py`。与上面的配置和模型相同，我们也提供了`BaseTrainer`作为基类，新构建的模型的类可以继承该基类。具体代码如下 ```python from mindspore.train.model import Model @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="mlm") class MaskedLanguageModelingTrainer(BaseTrainer): r"""MaskedLanguageModeling Task For Trainer. """ def __init__(self, model_name: str = None): super(MaskedLanguageModelingTrainer, self).__init__(model_name) def train(self, config=None, network=None, dataset=None, wrapper=None, optimizer=None, callbacks=None, **kwargs): # your code model = Model(wrapper) model.train(config.epochs, dataset=dataset, callbacks=callbacks, dataset_sink_mode=config.sink_mode, sink_size=config.sink_mode) def eval(self, **kwargs): pass def predict(self, **kwargs): pass ``` 在`MaskedLanguageModelingTrainer`中，我们注意到`train`的方法中，我们需要将模型训练所需要的参数都作为入参传入到该函数中，以供模型进行训练。在`train`的方法中，我们可以按MindSpore的训练方法进行训练, 用户可以参考MindSpore的[教程](https://www.mindspore.cn/tutorials/zh-CN/r1.9/advanced/model/model.html)。具体代码可以参考[masked_language_modeling_pretrain.py](https://gitee.com/jiahongqian/mindformers/blob/dev/mindformers/trainer/masked_language_modeling/masked_language_modeling_pretrain.py)。 ### 数据集构建 在mindformers中，用户可以在`mindformers/dataset`路径下构建自己的数据集文件，其代码结构如下： ```python from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MaskLanguageModelDataset(BaseDataset): def __new__(cls, dataset_config = None): # fake dataset generate def generator(): """dataset generator""" data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) input_mask = np.ones_like(data) token_type_id = np.zeros_like(data) next_sentence_lables = np.array([1]).astype(np.int32) masked_lm_positions = np.array([1, 2]).astype(np.int32) masked_lm_ids = np.array([1, 2]).astype(np.int32) masked_lm_weights = np.ones_like(masked_lm_ids) train_data = (data, input_mask, token_type_id, next_sentence_lables, masked_lm_positions, masked_lm_ids, masked_lm_weights) for _ in range(256): yield train_data # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"]) dataset = dataset.batch(batch_size=dataset_config.batch_size) return dataset ``` 对于`Bert`，用户可以参考库上的[`mask_language_model_dataset.py`](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/dataset/mask_language_model_dataset.py)。 ### 使用模型trainer进行训练 在完成`trainer`的构建后，用户可以使用如下代码进行模型的训练： ```Python import os from mindformers.mindformer_book import MindFormerBook from mindformers import BertNetwork, BertConfig from mindformers.trainer import MaskedLanguageModelingTrainer from mindformers.trainer.config_args import ConfigArguments, RunnerConfig, WrapperConfig from mindformers.dataset import MaskLanguageModelDataset # yaml_path yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", "bert", "model_config", "bert_base_uncased.yaml") # 先生成config，再生成模型 config = BertConfig.from_pretrained(yaml_path) bert_model = BertNetwork(config) # train config batch_size = 16 train_config = RunnerConfig(epochs=1, batch_size=batch_size, sink_mode=True, sink_size=2) # train dataset dataset = MaskLanguageModelDataset(train_config) # optimizer lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, learning_rate=lr_schedule, params=bert_model.trainable_params()) # wrapper bert_wrapper = TrainOneStepCell(bert_model, optimizer, sens=1024) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() callbacks = [loss_cb, time_cb] # trainer bert_trainer = MaskedLanguageModelingTrainer(model_name="bert_base_uncased") # train model bert_trainer.train( config=train_config, network=bert_model, dataset=dataset, wrapper=bert_wrapper, optimizer=optimizer, callbacks=callbacks) ``` 在这里，我们简单地使用MindSpore自带的`Model`进行模型的训练，在这里，我们需要传入数据集、优化器等组件，这些组件由用户手动定义。为了更好地方便用户管理训练所需要的各个组件，我们提供了一整套任务的配置文件读取，可以让用户快速地获取模型配置、模型训练、数据集配置等所需要的所有信息。 ### task config 配置 在建立`trainer`类之后，我们发现，使用模型的`trainer`训练模型，我们需要预先定义好network、dataset、oprimizer等 我们需要一个类来存储模型的训练各项超参数，包括训练的epoch、learning rate、batch size等重要参数。在mindformers中，我们通常使用`yaml`文件进行全局控制。与配置文件类型，一般我们会在`configs`下面的`bert`文件夹构建以下几个文件: ```text └─configs ├─bert ├─model_config ├─bert_base_uncased.yaml # 模型的配置文件 ├─task_config ├─context.yaml # 全局context配置文件 ├─runner.yaml # 运行超参配置文件 ├─bert_dataset.yaml # 数据集配置文件 run_bert_base_uncased.yaml # 总体配置文件 ``` * `bert_base_uncased.yaml`: 模型配置文件，同上述章节所介绍的，提供了模型的配置信息。 * `context.yaml`: 该文件主要提供了全局的参数配置信息，在配置文件中，可以直接使用对应的`context`、`parallel_context`方法中所需要的属性，用户需要了解以下两个MindSpore的函数的相关特性: ```python from mindspore import context context.set_context() context.set_auto_parallel_context() ``` * `runner.yaml`: 运行配置文件，该文件中存放了运行所需要的runner_config、runner_wrapper、optimizer等信息。其文件格式如下： ```text # config runner_config: epochs: 1 batch_size: 16 sink_mode: True per_epoch_size: -1 initial_epoch: 0 # wrapper runner_wrapper: type: TrainOneStepCell sens: 1024 # optimizer optimizer: type: AdamWeightDecay beta1: 0.9 beta2: 0.999 # lr sechdule lr_schedule: type: WarmUpLR learning_rate: 0.0001 end_learning_rate: 0.000001 warmup_steps: 10000 # callbacks callbacks: - type: MFLossMonitor - type: SummaryMonitor keep_default_action: True ``` * `bert_dataset.yaml`: 在数据集的配置文件中, 我们也可以指定模型的数据集的相关信息。 ```text train_dataset: &train_dataset batch_size = 16 train_dataset_task: type: MaskLanguageModelDataset dataset_config: *train_dataset ``` * `run_bert_base_uncased.yaml`: 该文件会包含上述构建的yaml文件，其格式如下： ```text base_config: [ '../__base__.yaml', './task_config/context.yaml', './task_config/runner.yaml', './task_config/bert_dataset.yaml', './model_config/bert_tiny_uncased.yaml' ] profile: False use_parallel: False seed: 0 resume_or_finetune_checkpoint: '' run_status: 'train' trainer: type: mlm model_name: 'bert' ``` 这里，使用`base_config`将所有的关键配置全部包含，并且使用`run_status`确定模型的运行状态。在`trainer`中，使用`type`关键字，来获取上节构建的`trainer`，并且通过`model_name`获取模型的名称。 用户可以参考mindformers中已经完成的[Bert配置文件](/home/jenkins/qianjiahong/mindformers/20230113/transformer/configs/bert)进行配置。 ### trainer 配置文件注册 同模型的配置文件注册相同, 这里也需要将`trainer`的yaml文件配置信息注册在`mindformer_book.py`中，其注册的位置在： ```python import os _TRAINER_SUPPORT_TASKS_LIST = OrderedDict([ ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")),]) ), ]) ``` 这里值得注意的是，`fill_mask`是任务的名称，用户也可以根据任务的特殊性，来命名合适的任务。 ## Trainer接口使用 在完成训练任务的配置文件和注册后，用户就可以使用mindformer提供的`Trainer`接口快速调用其网络进行训练，代码如下： ```python from mindformers.trainer import Trainer bert_trainer = Trainer(task='fill_mask') bert_trainer.train() ``` 可以看到，相比于`使用模型trainer进行训练`中使用的`MaskedLanguageModelingTrainer`而言，使用`Trainer`及yaml文件进行训练，会更加方便和简洁。 在完成上述训练类的定义后，`MindFormers`也提供了一个统一的运行文件[run_mindformer.py](https://gitee.com/mindspore/mindformers/blob/dev/run_mindformer.py)，方便读者训练自己的网络，运行指令如下： ```shell python run_mindformer.py --config /path/run_bert_base_uncased.yaml --run_mode train --dataset_dir /datasetpath/ ``` pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/practice/Develop_With_Api.md000066400000000000000000000272671476650141400322540ustar00rootroot00000000000000# 高阶接口开发样例 MindFormers套件的Trainer高阶接口提供了`train`、`finetune`、`evaluate`、`predict` 4个关键属性函数，帮助用户快速拉起任务的训练、微调、评估、推理流程：[Trainer.train](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L334) [Trainer.finetune](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L419) [Trainer.evaluate](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L516) [Trainer.predict](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L583) 使用`Trainer.train` `Trainer.finetune` `Trainer.evaluate` `Trainer.predict` 拉起任务的训练、微调、评估、推理流程，以下为使用`Trainer` 高阶接口进行全流程开发的使用样例（多卡分布式并行），命名为`task.py`： ```python import argparse from mindformers import Trainer, TrainingArguments from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(use_parallel=False, optimizer_parallel=False): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = None if use_parallel: parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=optimizer_parallel, full_batch=True) rank_id, device_num = init_context(use_parallel=use_parallel, context_config=context_config, parallel_config=parallel_config) def main(use_parallel=False, run_mode='train', task='text_generation', model_type='gpt2', pet_method='', train_dataset='./train', eval_dataset='./eval', predict_data='hello!', dp=1, mp=1, pp=1, micro_size=1, op=False): # 环境初始化 context_init(use_parallel, op) # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=1, batch_size=2, learning_rate=0.001, warmup_steps=100, sink_mode=True, sink_size=2) # 定义任务，预先准备好相应数据集 task = Trainer(task=task, model=model_type, pet_method=pet_method, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) task.set_parallel_config(data_parallel=dp, model_parallel=mp, pipeline_stage=pp, optimizer_shard=op, micro_batch_num=micro_size) if run_mode == 'train': task.train() elif run_mode == 'finetune': task.finetune() elif run_mode == 'eval': task.evaluate() elif run_mode == 'predict': result = task.predict(input_data=predict_data) print(result) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--run_mode', default='train', required=True, help='set run mode for model.') parser.add_argument('--use_parallel', default=False, help='open parallel for model.') parser.add_argument('--task', default='text_generation', required=True, help='set task type.') parser.add_argument('--model_type', default='gpt2', required=True, help='set model type.') parser.add_argument('--train_dataset', default=None, help='set train dataset.') parser.add_argument('--eval_dataset', default=None, help='set eval dataset.') parser.add_argument('--pet_method', default='', help="set finetune method, now support type: ['', 'lora']") parser.add_argument('--data_parallel', default=1, type=int, help='set data parallel number. Default: None') parser.add_argument('--model_parallel', default=1, type=int, help='set model parallel number. Default: None') parser.add_argument('--pipeline_parallel', default=1, type=int, help='set pipeline parallel number. Default: None') parser.add_argument('--micro_size', default=1, type=int, help='set micro batch number. Default: None') parser.add_argument('--optimizer_parallel', default=False, type=bool, help='whether use optimizer parallel. Default: None') args = parser.parse_args() main(run_mode=args.run_mode, task=args.task, use_parallel=args.use_parallel, model_type=args.model_type, pet_method=args.pet_method, train_dataset=args.train_dataset, eval_dataset=args.eval_dataset, dp=args.data_parallel, mp=args.model_parallel, pp=args.pipeline_parallel, micro_size=args.micro_size, op=args.optimizer_parallel) ``` - 单卡使用样例： ```shell # 训练 python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode train # 评估 python task.py --task text_generation --model_type gpt2 --eval_dataset ./eval --run_mode eval # 微调，支持 python task.py --task text_generation --model_type gpt2 --train_dataset ./finetune --pet_method lora --run_mode finetune # 推理 python task.py --task text_generation --model_type gpt2 --predict_data 'hello!' --run_mode predict ``` - 多卡分布式使用样例： - 单机多卡标准启动脚本：`run_distribute_single_node.sh` ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_single_node.sh [EXECUTE_ORDER] [RANK_TABLE_PATH] [DEVICE_RANGE] [RANK_SIZE] For Multiple Devices In Single Machine" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) mkdir -p ./output/log/rank_$RANK_ID echo "start training for rank $RANK_ID, device $DEVICE_ID" $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` - 多机多卡标准启动脚本：`run_distribute_multi_node.sh` ```bash #!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] then echo "Usage Help: bash run_distribute_multi_node.sh [EXECUTE_ORDER] [RANK_TABLE_FILE] [DEVICE_RANGE] [RANK_SIZE]" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } EXECUTE_ORDER=$1 RANK_TABLE_PATH=$(check_real_path $2) DEVICE_RANGE=$3 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $RANK_TABLE_PATH ] then echo "error: RANK_TABLE_FILE=$RANK_TABLE_PATH is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited export RANK_SIZE=$4 export RANK_TABLE_FILE=$RANK_TABLE_PATH shopt -s extglob for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) echo "start training for rank $RANK_ID, device $DEVICE_ID" mkdir -p ./output/log/rank_$RANK_ID $EXECUTE_ORDER &> ./output/log/rank_$RANK_ID/mindformer.log & done shopt -u extglob ``` - 分布式并行执行`task.py`样例：需提前生成`RANK_TABLE_FILE`，同时`task.py`中默认使用**半自动并行模式**。 **注意单机时使用{single}，多机时使用{multi}** ```shell # 分布式训练 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode train --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式评估 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --eval_dataset ./eval --run_mode eval --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式微调 bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --train_dataset ./train --run_mode finetune --pet_method lora --use_parallel True --data_parallel 1 --model_parallel 2 --pipeline_parallel 2 --micro_size 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 # 分布式推理（不支持流水并行,试用特性） bash run_distribute_{single/multi}_node.sh "python task.py --task text_generation --model_type gpt2 --predict_data 'hello!' ./train --run_mode predict --pet_method lora --use_parallel True --data_parallel 1 --model_parallel 2" hccl_4p_0123_192.168.89.35.json [0,4] 4 ```pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/docs/practice/Using_Api.md000066400000000000000000000266111476650141400307400ustar00rootroot00000000000000# 高阶接口使用样例 ## AutoClass MindFormers大模型套件提供了AutoClass类，包含AutoConfig、AutoModel、AutoTokenizer、AutoProcessor4个便捷高阶接口，方便用户调用套件中已封装的API接口。上述4类提供了相应领域模型的ModelConfig、Model、Tokenzier、Processor的实例化功能。 ### AutoConfig&&AutoModel 使用AutoConfig和AutoModel自动实例化MindFormers中支持的模型配置或模型架构： ```python from mindformers import AutoConfig, AutoModel gpt_config = AutoConfig.from_pretrained("gpt2") gpt_model = AutoModel.from_pretrained("gpt2") # 自动加载预置权重到网络中 ``` 使用已有模型配置或网络架构实例化相应的模型配置或网络架构实例： ```python from mindformers import GPT2LMHeadModel, GPT2Config gpt_13b_config = GPT2Config.from_pretrained("gpt2_13b") gpt_13b_model = GPT2LMHeadModel.from_pretrained("gpt2_13b") # 自动加载预置权重到网络中 ``` 使用已有模型配置或模型架构进行二次开发： ```python from mindformers import GPT2LMHeadModel, GPT2Config gpt_config = GPT2Config(hidden_size=768, num_layers=12) gpt_model = GPT2LMHeadModel(gpt_config) ``` ### AutoTokenizer 使用AutoTokenizer自动获取文本分词器： ```python from mindformers import AutoTokenizer gpt_tokenizer = AutoTokenizer.from_pretrained("gpt2") gpt_tokenizer("hello!") ``` ```text {'input_ids': [50256, 31373, 0, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} ``` 使用已有Tokenizer函数对文本进行分词： ```python from mindformers import GPT2Tokenizer gpt_tokenizer = GPT2Tokenizer.from_pretrained("gpt2") gpt_tokenizer("hello!") ``` ```text {'input_ids': [50256, 31373, 0, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} ``` ### AutoProcessor 使用AutoProcessor自动获取模型数据预处理过程： ```python from mindformers import AutoProcessor gpt_processor = AutoProcessor.from_pretrained("gpt2") gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 128], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` 使用已有模型数据预处理过程对相应数据做模型输入前的预处理： ```python from mindformers import GPT2Processor gpt_processor = GPT2Processor.from_pretrained('gpt2') gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 128], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` 使用已有数据预处理过程进行二次开发： ```python from mindformers import GPT2Processor, GPT2Tokenizer # 自定义 tokenizer tok = GPT2Tokenizer.from_pretrained("gpt2") gpt_processor = GPT2Processor(tokenizer=tok, max_length=256, return_tensors='ms') gpt_processor("hello!") ``` ```text {'text': Tensor(shape=[1, 256], dtype=Int32, value= [[50256, 31373, 0 ... 50256, 50256, 50256]])} ``` ## pipeline MindFormers大模型套件面向任务设计pipeline推理接口，旨在让用户可以便捷的体验不同AI领域的大模型在线推理服务，当前已集成10+任务的推理流程； ### pipeline使用样例 使用MindFormers预置任务和模型开发一个推理流： ```python from mindformers import pipeline text_generation = pipeline(task='text_generation', model='gpt2', max_length=50) text_generation("I love Beijing, because", top_k=3) ``` ```text [{'text_generation_text': ['I love Beijing, because of all its beautiful scenery and beautiful people," said the mayor of Beijing, who was visiting the capital for the first time since he took office.\n\n"The Chinese government is not interested in any of this," he added']}] ``` 使用自定义的模型、tokenizer等进行任务推理： ```python from mindformers import pipeline from mindformers import GPT2LMHeadModel, GPT2Config, GPT2Tokenizer tok = GPT2Tokenizer.from_pretrained("gpt2") gpt_model = GPT2LMHeadModel.from_pretrained("gpt2") text_generation = pipeline(task='text_generation', model=gpt_model, tokenizer=tok, max_length=50) text_generation("I love Beijing, because", top_k=3) ``` ```text [{'text_generation_text': ['I love Beijing, because of all its beautiful scenery and beautiful people," said the mayor of Beijing, who was visiting the capital for the first time since he took office.\n\n"The Chinese government is not interested in any of this," he added']}] ``` ## Trainer MindFormers大模型套件面向任务设计Trainer接口，旨在让用户可以快速使用我们预置任务和模型的训练、微调、评估、推理能力，当前已集成10+任务和10+模型的全流程开发能力； ### init_context mindspore相关环境的初始化，MindFormers中提供了init_context标准接口帮助用户完成单卡或多卡并行环境的初始化： [init_context](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/core/context/build_context.py#L53) [ContextConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/config_args.py#L26) [ParallelContextConfig](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/config_args.py#L26) 单卡初始化： ```python from mindformers import init_context, ContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) rank_id, device_num = init_context(use_parallel=False, context_config=context_config) ``` 多卡数据并行模式初始化： ```python from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = ParallelContextConfig(parallel_mode='DATA_PARALLEL', gradients_mean=True, enable_parallel_optimizer=False) rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_config) ``` 多卡半自动并行模式初始化： ```python from mindformers import init_context, ContextConfig, ParallelContextConfig def context_init(): """init context for mindspore.""" context_config = ContextConfig(mode=0, device_target="Ascend", device_id=0) parallel_config = ParallelContextConfig(parallel_mode='SEMI_AUTO_PARALLEL', gradients_mean=False, enable_parallel_optimizer=False, full_batch=True) rank_id, device_num = init_context(use_parallel=True, context_config=context_config, parallel_config=parallel_config) ``` ### TrainingArguments&&Trainer MindFormers套件对用户提供了`TrainingArguments`类，用于自定义大模型训练过程中的各类参数，支持参数详见：[TrainingArguments](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/training_args.py) 同时，MindFormers也提供了`Trainer`高阶接口，用于大模型任务的开发、训练、微调、评估、推理等流程； 使用TrainingArguments自定义大模型训练过程参数： ```python from mindformers import TrainingArguments # 环境初始化，参考上述`init_context`章节实现 context_init() # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) ``` 使用Trainer接口创建内部预置任务：数据集按照官方教程准备[GPT预训练数据集准备](../model_cards/gpt2.md)，自定义训练参数 ```python from mindformers import Trainer, TrainingArguments # 环境初始化，参考上述`init_context`章节实现 context_init() # 训练超参数定义 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) text_generation = Trainer(task='text_generation', model='gpt2', args=training_args, train_dataset='./train', eval_dataset='./eval') ``` 使用Trainer接口创建内部预置任务：自定义数据集，模型，训练参数 ```python from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments, AutoModel def generator(): """text dataset generator.""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(512): yield input_ids # 环境初始化，参考上述`init_context`章节实现 context_init() # 自定义训练超参数 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) # 自定义模型 pangu_model = AutoModel.from_pretrained("pangualpha_2_6b") # 自定义数据集 dataset = GeneratorDataset(generator, column_names=["input_ids"]) train_dataset = dataset.batch(batch_size=4) eval_dataset = dataset.batch(batch_size=4) # 定义文本生成任务，传入自定义模型、数据集、超参数 text_generation = Trainer(task='text_generation', model=pangu_model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) ``` ### 并行&&重计算配置 MindFormers的Trainer接口提供了并行的配置接口`set_parallel_config`和重计算配置接口`set_recompute_config`，其中`set_parallel_config`接口仅在**半自动并行** 或**全自动并行模式**下生效，同时需要模型本身已支持或已配置[并行策略](https://www.mindspore.cn/tutorials/experts/zh-CN/r2.0/parallel/introduction.html); [set_parallel_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L690) [set_recompute_config](https://gitee.com/mindspore/mindformers/blob/dev/mindformers/trainer/trainer.py#L731) 使用Trainer高阶接口，自定义并行和重计算配置： ```python import numpy as np from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments from mindformers import PanguAlphaHeadModel, PanguAlphaConfig def generator(): """text dataset generator.""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(512): yield input_ids # 环境初始化，参考上述`init_context`章节实现 context_init() # 自定义训练超参数 training_args = TrainingArguments(num_train_epochs=3, batch_size=2, learning_rate=0.001, warmup_steps=1000, sink_mode=True) # 自定义模型 pangu_config = PanguAlphaConfig(hidden_size=768, ffn_hidden_size=768 * 4, num_layers=12, num_heads=12, checkpoint_name_or_path='') pangu_model = PanguAlphaHeadModel(pangu_config) # 自定义数据集 dataset = GeneratorDataset(generator, column_names=["input_ids"]) train_dataset = dataset.batch(batch_size=4) eval_dataset = dataset.batch(batch_size=4) # 定义文本生成任务，传入自定义模型、数据集、超参数 text_generation = Trainer(task='text_generation', model=pangu_model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset) # 设定并行策略，比如2机16卡,设定数据并行4 模型并行2 流水并行2 微批次大小为2 打开优化器并行 text_generation.set_parallel_config(data_parallel=4, model_parallel=2, pipeline_stage=2, micro_batch_num=2, optimizer_shard=True) # 设置重计算配置，打开重计算 text_generation.set_recompute_config(recompute=True) ``` pengcheng.mind-mindformer/docs/readthedocs/source_zh_cn/index.rst000066400000000000000000000033601476650141400256530ustar00rootroot00000000000000目录 ========================== .. toctree:: :glob: :maxdepth: 1 :caption: 开始 Introduction.md Install.md Version_Match.md Quick_Tour.md docs/model_support_list.md .. toctree:: :glob: :maxdepth: 1 :caption: 设计 docs/design/AutoClass_Design.md docs/design/Trainer_Design.md docs/design/Pipeline_Design.md docs/design/Config_Design.md docs/design/Parallel_Design.md docs/design/Pet_Tuners_Design.md .. toctree:: :glob: :maxdepth: 1 :caption: 高级特性 docs/feature_cards/Resume_Training.md docs/feature_cards/Eval_While_Training.md docs/feature_cards/Transform_Ckpt.md docs/feature_cards/Text_Generator.md docs/feature_cards/Pet_Tuners.md docs/feature_cards/Precision_Optimization.md docs/feature_cards/Performance_Tuning.md docs/feature_cards/Chat_Web.md docs/feature_cards/Inference.md .. toctree:: :glob: :maxdepth: 1 :caption: AICC docs/aicc_cards/README.md .. toctree:: :glob: :maxdepth: 2 :caption: 模型 & 任务 docs/model_cards/index docs/task_cards/index .. toctree:: :glob: :maxdepth: 1 :caption: 开发实践 docs/practice/Using_Api.md docs/practice/Develop_With_Api.md docs/practice/Bert_Example.md .. toctree:: :glob: :maxdepth: 1 :caption: API docs/api_python/README.md docs/api_python/mindformers docs/api_python/mindformers.core docs/api_python/mindformers.dataset docs/api_python/mindformers.models docs/api_python/mindformers.modules docs/api_python/mindformers.pipeline docs/api_python/mindformers.trainer docs/api_python/mindformers.wrapper .. toctree:: :glob: :maxdepth: 1 :caption: FAQ docs/FAQ.md docs/transformer仓Python编程规范.md pengcheng.mind-mindformer/docs/task_cards/000077500000000000000000000000001476650141400211605ustar00rootroot00000000000000pengcheng.mind-mindformer/docs/task_cards/contrastive_language_image_pretrain.md000066400000000000000000000044261476650141400307620ustar00rootroot00000000000000# Contrastive Language Image Pretrain ## 任务描述 语言图像对比预训练：对模型进行图文对比学习，增强模型对文本图片的匹配度认识能力，预训练完的模型可用于零样本图像分类等下游任务 [相关论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. ## 已支持数据集性能 | model | type | Datasets | Performance | stage | example | |:-----:|:----------------------------------------------------------------------------:|:--------:|:-----------:|:--------:|:-----------------------:| | clip | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | Flickr8k | -- | pretrain | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flickr8k.sh) | ### Flickr8k([链接](https://pan.baidu.com/s/1LRlQUL1MRipPL4MLOdExzg)，密码: s4be) - 数据集大小：2.2G，共8000张彩色图像，每张图像都与五个不同的标题配对，这些标题提供了对图片中物体和事件的内容描述 - 训练集：6000张图像 - 验证集：1000张图像 - 测试集：1000张图像 - 数据格式：RGB ```bash 数据集目录格式 └─Flickr8k ├─Flickr8k_Dataset | └─Flickr8k_Dataset └─Flickr8k_text ├─Flickr8k.devImages.txt ├─Flickr8k.testImages.txt ├─Flickr8k.trainImages.txt └─Flickr8k.token.txt ``` ## 快速任务接口 - Trainer接口开启训练： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("contrastive_language_image_pretrain") # INFO - Trainer support model list for contrastive_language_image_pretrain task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='contrastive_language_image_pretrain', model='clip_vit_b_32', train_dataset='./Flickr8k' ) trainer.train() ``` pengcheng.mind-mindformer/docs/task_cards/image_classification.md000066400000000000000000000133731476650141400256460ustar00rootroot00000000000000# Image Classification ## 任务描述 图像分类：模型基于图像数据集进行训练后，可以在给定任意图片的情况下，完成对图像的分类，分类结果仅限于数据集中所包含的类别。 [相关论文-vit](https://gitee.com/link?target=https%3A%2F%2Farxiv.org%2Fabs%2F2010.11929): Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby. 2021. [相关论文-swin](https://arxiv.org/abs/2103.14030) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, 2021 ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | | :---: | :------------: | :---------: | :-----------: | :----------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | vit | vit_base_p16 | ImageNet-1K | 83.71% | train
finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_finetune_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/vit_base_p16_predict_on_imagenet1k.sh) | | swin | swin_base_p4w7 | ImageNet-1K | 83.44% | train
finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_train_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_finetune_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/image_classification/swin_base_p4w7_predict_on_imagenet1k.sh) | ### [ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速任务接口 - Trainer接口开启训练/评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("image_classification") # INFO - Trainer support model list for image_classification task is: # INFO - ['vit_base_p16', 'swin_base_p4w7'] # INFO - ------------------------------------- # 下面以ViT模型为例，Swin同理 # 初始化trainer vit_trainer = Trainer( task='image_classification', model='vit_base_p16', train_dataset="imageNet-1k/train", eval_dataset="imageNet-1k/val") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 vit_trainer.train(resume_or_finetune_from_checkpoint="mae_vit_base_p16", do_finetune=True) vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式2: 重头开始训练，并使用训练好的权重进行eval和推理 vit_trainer.train() vit_trainer.evaluate(eval_checkpoint=True) predict_result = vit_trainer.predict(predict_checkpoint=True, input_data=img, top_k=3) print(predict_result) # 方式3： 从obs下载训练好的权重并进行eval和推理 vit_trainer.evaluate() predict_result = vit_trainer.predict(input_data=img, top_k=3) print(predict_result) ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("image_classification") # INFO - Pipeline support model list for image_classification task is: # INFO - ['vit_base_p16', 'swin_base_p4w7'] # INFO - ------------------------------------- # 下面以ViT模型为例，Swin同理 # pipeline初始化 from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("image_classification", model='vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img, top_k=3) # 输出 # [[{'score': 0.8846962, 'label': 'daisy'}, {'score': 0.005090589, 'label': 'bee'}, {'score': 0.0031510447, 'label': 'vase'}]] ``` pengcheng.mind-mindformer/docs/task_cards/index.rst000066400000000000000000000053271476650141400230300ustar00rootroot00000000000000.. role:: raw-html-m2r(raw) :format: html 任务 ========================== .. toctree:: :glob: :maxdepth: 1 contrastive_language_image_pretrain image_classification question_answering text_classification text_generation token_classification zero_shot_image_classification 此处给出了MindFormers套件中支持的任务名称和模型名称，用于高阶开发时的索引名 .. list-table:: :header-rows: 1 * - 模型 - 任务（task name） - 模型（model name） * - `BERT <../model_cards/bert.html>`_ - masked_language_modeling\ :raw-html-m2r:`
`\ `text_classification <../task_cards/text_classification.html>`_\ :raw-html-m2r:`
`\ `token_classification <../task_cards/token_classification.html>`_\ :raw-html-m2r:`
`\ `question_answering <../task_cards/question_answering.html>`_ - bert_base_uncased :raw-html-m2r:`
`\ txtcls_bert_base_uncased\ :raw-html-m2r:`
`\ txtcls_bert_base_uncased_mnli :raw-html-m2r:`
`\ tokcls_bert_base_chinese\ :raw-html-m2r:`
`\ tokcls_bert_base_chinese_cluener :raw-html-m2r:`
`\ qa_bert_base_uncased\ :raw-html-m2r:`
`\ qa_bert_base_chinese_uncased * - `T5 <../model_cards/t5.html>`_ - translation - t5_small * - `GPT2 <../model_cards/gpt2.html>`_ - text_generation - gpt2_small :raw-html-m2r:`
`\ gpt2_13b :raw-html-m2r:`
`\ gpt2_52b * - `PanGuAlpha <../model_cards/pangualpha.html>`_ - text_generation - pangualpha_2_6_b\ :raw-html-m2r:`
`\ pangualpha_13b * - `GLM <../model_cards/glm.html>`_ - text_generation - glm_6b\ :raw-html-m2r:`
`\ glm_6b_lora * - `LLama <../model_cards/llama.html>`_ - text_generation - llama_7b :raw-html-m2r:`
`\ llama_13b :raw-html-m2r:`
`\ llama_65b :raw-html-m2r:`
`\ llama_7b_lora * - `Bloom <../model_cards/bloom.html>`_ - text_generation - bloom_560m\ :raw-html-m2r:`
`\ bloom_7.1b :raw-html-m2r:`
`\ bloom_65b\ :raw-html-m2r:`
`\ bloom_176b * - `MAE <../model_cards/mae.html>`_ - masked_image_modeling - mae_vit_base_p16 * - `VIT <../model_cards/vit.html>`_ - `image_classification <../task_cards/image_classification.html>`_ - vit_base_p16 * - `Swin <../model_cards/swin.html>`_ - `image_classification <../task_cards/image_classification.html>`_ - swin_base_p4w7 * - `CLIP <../model_cards/clip.html>`_ - `contrastive_language_image_pretrain <../task_cards/contrastive_language_image_pretrain.html>`_\ :raw-html-m2r:`
`\ `zero_shot_image_classification <../task_cards/zero_shot_image_classification.html>`_ - clip_vit_b_32\ :raw-html-m2r:`
`\ clip_vit_b_16 :raw-html-m2r:`
`\ clip_vit_l_14\ :raw-html-m2r:`
`\ clip_vit_l_14@336 pengcheng.mind-mindformer/docs/task_cards/masked_image_modeling.md000066400000000000000000000054211476650141400257700ustar00rootroot00000000000000# Masked Image Modeling ## 任务描述 掩码图像建模：对图像中的部分图像块进行掩码，用剩下的图像块重建整张图像，从而对被掩码的图像块进行预测。 [相关论文-MAE](https://arxiv.org/abs/2111.06377): Kaiming He and Xinlei Chen and Saining Xie and Yanghao Li and Piotr Dollár and Ross Girshick. 2021. ### [ImageNet2012](http://www.image-net.org/) - 数据集大小：125G，共1000个类、125万张彩色图像 - 训练集：120G，共120万张图像 - 测试集：5G，共5万张图像 - 数据格式：RGB ```text 数据集目录格式 └─imageNet-1k ├─train # 训练数据集 └─val # 评估数据集 ``` ## 快速任务接口 - Trainer接口开启训练/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("masked_image_modeling") # INFO - Trainer support model list for masked_image_modeling task is: # INFO - ['mae_vit_base_p16'] # INFO - ------------------------------------- # 初始化trainer mae_trainer = Trainer( task='masked_image_modeling', model='mae_vit_base_p16', train_dataset="imageNet-1k/train") img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") # 方式1: 重头开始训练，并使用训练好的权重推理 mae_trainer.train() predict_result = mae_trainer.predict(predict_checkpoint=True, input_data=img) print(predict_result) # 方式2： 从obs下载训练好的权重并进行eval和推理 predict_result = mae_trainer.predict(input_data=img) print(predict_result) # 输出 # [{'info': './output/output_image0.jpg', 'data': }] ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("masked_image_modeling") # INFO - Pipeline support model list for masked_image_modeling task is: # INFO - ['mae_vit_base_p16'] # INFO - ------------------------------------- # pipeline初始化 from mindformers.pipeline import pipeline from mindformers.tools.image_tools import load_image pipeline_task = pipeline("masked_image_modeling", model='mae_vit_base_p16') img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") pipeline_result = pipeline_task(img) print(pipeline_result) # 输出 # [{'info': './output/output_image0.jpg', 'data': }] ``` pengcheng.mind-mindformer/docs/task_cards/question_answering.md000066400000000000000000000120321476650141400254240ustar00rootroot00000000000000# Question Answering ## 任务描述 **问答任务**：模型在基于问答数据集的微调后，输入为上下文（context）和问题（question），模型根据上下文（context）给出相应的回答。 **相关论文** - Jacob Devlin, Ming-Wei Chang, et al., [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://arxiv.org/pdf/1810.04805.pdf), 2019. - Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang, [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://arxiv.org/pdf/1606.05250.pdf), 2016. ## 已支持数据集性能 | model | type | datasets | EM | F1 | stage | example | | :---: | :------------------------: | :--------: | :---: | ----- | :------------------------: | :----------------------------------------------------------: | | q'a | qa_bert_case_uncased_squad | SQuAD v1.1 | 80.74 | 88.33 | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_finetune_on_squad.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/question_answering/qa_bert_base_uncased_predict_on_squad.sh) | ### [SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) - 下载地址：[SQuAD v1.1训练集](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)，[SQuAD v1.1验证集](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json) - 数据集：该数据集包含 10 万个（问题，原文，答案）三元组，原文来自于 536 篇维基百科文章，而问题和答案的构建主要是通过众包的方式，让标注人员提出最多 5 个基于文章内容的问题并提供正确答案，且答案出现在原文中。 - 数据格式：json文件 ```bash 数据集目录格式 └─squad ├─train-v1.1.json └─dev-v1.1.json ``` ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 在脚本执行目录创建 `squad` 文件夹，然后将数据集放入其中 - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode finetune --load_checkpoint qa_bert_base_uncased # evaluate python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode eval --load_checkpoint qa_bert_base_uncased_squad # predict python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode predict --load_checkpoint qa_bert_base_uncased_squad --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化trainer trainer = Trainer(task='question_answering', model='qa_bert_base_uncased', train_dataset='./squad/', eval_dataset='./squad/') #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="qa_bert_base_uncased", do_finetune=True) trainer.evaluate(eval_checkpoint=True) # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] trainer.predict(predict_checkpoint=True, input_data=input_data) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - QA Metric = {'QA Metric': {'exact_match': 80.74739829706716, 'f1': 88.33552874684968}} # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] trainer.predict(input_data=input_data) # INFO - output result is [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import QuestionAnsweringPipeline from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig # 测试数据，测试数据分为context和question两部分，两者以 “-” 分隔 input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') # This is a known issue, you need to specify batch size equal to 1 when creating bert model. qa_squad_config.batch_size = 1 model = BertForQuestionAnswering(qa_squad_config) qa_pipeline = QuestionAnsweringPipeline(task='question_answering', model=model, tokenizer=tokenizer) results = qa_pipeline(input_data) print(results) # 输出 # [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] ``` pengcheng.mind-mindformer/docs/task_cards/text_classification.md000066400000000000000000000141401476650141400255410ustar00rootroot00000000000000# Text Classification ## 任务描述 文本分类：模型在基于文本对的微调后，可以在给定任意文本对与候选标签列表的情况下，完成对文本对关系的分类，文本对的两个文本之间以-分割。 [相关论文](https://arxiv.org/pdf/1810.04805.pdf) Jacob Devlin, Ming-Wei Chang, et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2019. ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | |:------:|:----------------------------------------------------------:|:--------:|:---------------:|:--------------------------:|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | bert | txtcls_bert_base_uncased | Mnli | 30.9% | pretrain | -- | | txtcls | txtcls_bert_case_uncased_mnli | Mnli | 84.8% | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_finetune_on_mnli.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_eval_on_mnli.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_predict_on_mnli.sh) | ### [Mnli](https://dl.fbaipublicfiles.com/glue/data/MNLI.zip) - 数据集大小：298M，共431992个样本，3个类别 - 训练集：392702个样本 - 匹配测试集：9796个样本 - 非匹配测试集：9847个样本 - 匹配开发集：9815个样本 - 非匹配开发集：9832个样本 - 数据格式：tsv文件 ```bash 数据集目录格式 └─mnli ├─dev ├─test └─train ``` - 用户可以参考[BERT](https://github.com/google-research/bert)代码仓中的run_classifier.py文件，进行Mnli数据集`TFRecord`格式文件的生成。 ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode finetune --load_checkpoint txtcls_bert_base_uncased # evaluate python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode eval --load_checkpoint txtcls_bert_base_uncased_mnli # predict python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode predict --load_checkpoint txtcls_bert_base_uncased_mnli --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("text_classification") # INFO - Trainer support model list for txt_classification task is: # INFO - ['txtcls_bert_base_uncased'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='text_classification', model='txtcls_bert_base_uncased', train_dataset='./mnli/train', eval_dataset='./mnli/eval') # 测试数据，该input_data有两个测试案例，即两个文本对，单个文本对的两个文本之间用-分割 input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", "i don't know um do you do a lot of camping-I know exactly."] #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="txtcls_bert_base_uncased", do_finetune=True) trainer.evaluate(eval_checkpoint=True) trainer.predict(predict_checkpoint=True, input_data=input_data, top_k=1) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - Top1 Accuracy=84.8% trainer.predict(input_data=input_data, top_k=1) # INFO - output result is [[{'label': 'neutral', 'score': 0.9714198708534241}], # [{'label': 'contradiction', 'score': 0.9967639446258545}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import TextClassificationPipeline from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", "i don't know um do you do a lot of camping-I know exactly."] tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') # Because batch_size parameter is required when bert model is created, and pipeline # function deals with samples one by one, the batch_size parameter is seted one. txtcls_mnli_config.batch_size = 1 model = BertForMultipleChoice(txtcls_mnli_config) txtcls_pipeline = TextClassificationPipeline(task='text_classification', model=model, tokenizer=tokenizer, max_length=model.config.seq_length, padding="max_length") results = txtcls_pipeline(input_data, top_k=1) print(results) # 输出 # [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] ``` pengcheng.mind-mindformer/docs/task_cards/text_generation.md000066400000000000000000000004111476650141400246750ustar00rootroot00000000000000# Text Generation ## 任务描述 文本生成：生成新文本的任务，模型基于输入的不完整的文本或解释进行填充，或对输入的源语言翻译成目标语言。 ## [已支持模型](../model_support_list.md#text-generator支持度表) pengcheng.mind-mindformer/docs/task_cards/token_classification.md000066400000000000000000000123561476650141400257040ustar00rootroot00000000000000# Token Classification ## 任务描述 命名实体识别：模型在基于命名实体识别数据集的微调后，可以在给定任意文本与候选标签列表的情况下，完成对文本中命名实体的识别。 [相关论文](https://arxiv.org/abs/2001.04351) Xu, Liang and Dong, Qianqian and Yu, Cong and Tian, Yin and Liu, Weitang and Li, Lu and Zhang, Xuanwei, CLUENER2020: Fine-grained Name Entity Recognition for Chinese, 2020. ## 已支持数据集性能 | model | type | datasets | Entity F1 | stage | example | | :----: | :------------------------------: | :------: | :-------: | :------------------------: | :----------------------------------------------------------: | | tokcls | tokcls_bert_case_chinese_cluener | CLUENER | 0.7905 | finetune
eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_finetune_on_cluener.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_eval_on_cluener.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/token_classification/tokcls_bert_base_chinese_predict_on_cluener.sh) | ### [CLUENER](https://github.com/CLUEbenchmark/CLUENER2020) - 数据集：训练集大小为10748个样本，验证集大小为1343个样本，10个类别，分别为: 地址（address），书名（book），公司（company），游戏（game），政府（government），电影（movie），姓名（name），组织机构（organization），职位（position），景点（scene）。 - 数据格式：json文件 ```bash 数据集目录格式 └─cluener ├─train.json ├─dev.json ├─test.json ├─cluener_predict.json └─README.md ``` ## 快速任务接口 ### 脚本启动 > 需开发者提前clone工程。 - 请参考[使用脚本启动](../../README.md#方式一使用已有脚本启动) - 在脚本执行目录创建 `cluener` 文件夹，然后将数据集放入其中 - 脚本运行测试 ```shell # finetune python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode finetune --load_checkpoint tokcls_bert_base_chinese # evaluate python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode eval --load_checkpoint tokcls_bert_base_chinese_cluener # predict python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode predict --load_checkpoint tokcls_bert_base_chinese_cluener --predict_data [TEXT] ``` ### 调用API启动 - Trainer接口开启训练/评估/推理： ```python from mindformers.trainer import Trainer # 初始化trainer trainer = Trainer(task='token_classification', model='tokcls_bert_base_chinese', train_dataset='./cluener/', eval_dataset='./cluener/') # 测试数据 input_data = ["结果上周六他们主场0：3惨败给了中游球队瓦拉多利德，近7个多月以来西甲首次输球。"] #方式1：使用现有的预训练权重进行finetune， 并使用finetune获得的权重进行eval和推理 trainer.train(resume_or_finetune_from_checkpoint="tokcls_bert_base_chinese", do_finetune=True) trainer.evaluate(eval_checkpoint=True) trainer.predict(predict_checkpoint=True, input_data=input_data) # 方式2： 从obs下载训练好的权重并进行eval和推理 trainer.evaluate() # INFO - Entity F1=0.7853 trainer.predict(input_data=input_data) # INFO - output result is [[{'entity_group': 'organization', 'start': 20, 'end': 24, 'score': 0.94914, 'word': '瓦拉多利德'}, # {'entity_group': 'organization', 'start': 33, 'end': 34, 'score': 0.9496, 'word': '西甲'}]] ``` - pipeline接口开启快速推理 ```python from mindformers.pipeline import TokenClassificationPipeline from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig from mindformers.dataset.labels import cluener_labels input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokcls_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') # This is a known issue, you need to specify batch size equal to 1 when creating bert model. tokcls_cluener_config.batch_size = 1 model = BertForTokenClassification(tokcls_cluener_config) tokcls_pipeline = TokenClassificationPipeline(task='token_classification', model=model, id2label=id2label, tokenizer=tokenizer, max_length=model.config.seq_length, padding="max_length") results = tokcls_pipeline(input_data) print(results) # 输出 # [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, # {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] ``` pengcheng.mind-mindformer/docs/task_cards/zero_shot_image_classification.md000066400000000000000000000076171476650141400277460ustar00rootroot00000000000000# Zero Shot Image Classification ## 任务描述 零样本图像分类：模型在基于图文对的预训练后，可以在给定任意图片与候选标签列表的情况下，完成对图像的分类，而无需任何微调。 [相关论文](https://arxiv.org/abs/2103.00020) Alec Radford, Jong Wook Kim, et al., Learning Transferable Visual Models From Natural Language Supervision, 2021. ## 已支持数据集性能 | model | type | datasets | Top1-accuracy | stage | example | |:-----:|:------------------------------------------------------------------------:|:--------:|:--------------------------------------------:|:----------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:| | clip | clip_vit_b_32
clip_vit_b_16
clip_vit_l_14
clip_vit_l_14@336 | Cifar100 | 57.24%
61.41%
69.67%
68.19% | eval
predict | [link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh)
[link](https://gitee.com/mindspore/mindformers/blob/dev/scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh) | ### [Cifar100](http://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz) - 数据集大小：161M，共60000张图片，100个类别 - 训练集：50000张图片 - 测试集：10000张图片 - 数据格式：二进制文件 ```bash 数据集目录格式 └─cifar-100-python ├─meta ├─test └─train ``` ## 快速任务接口 - Trainer接口开启评估/推理： ```python from mindformers import MindFormerBook from mindformers.trainer import Trainer from mindformers.tools.image_tools import load_image # 显示Trainer的模型支持列表 MindFormerBook.show_trainer_support_model_list("zero_shot_image_classification") # INFO - Trainer support model list for zero_shot_image_classification task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # 初始化trainer trainer = Trainer(task='zero_shot_image_classification', model='clip_vit_b_32', eval_dataset='cifar-100-python' ) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") trainer.evaluate() #下载权重进行评估 # INFO - Top1 Accuracy=57.24% trainer.predict(input_data=img) #下载权重进行推理 # INFO - output result is saved at ./results.txt ``` - pipeline接口开启快速推理 ```python from mindformers import pipeline, MindFormerBook from mindformers.tools.image_tools import load_image # 显示pipeline支持的模型列表 MindFormerBook.show_pipeline_support_model_list("zero_shot_image_classification") # INFO - Pipeline support model list for zero_shot_image_classification task is: # INFO - ['clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336'] # INFO - ------------------------------------- # pipeline初始化 classifier = pipeline("zero_shot_image_classification", model="clip_vit_b_32" candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") classifier(img) # 输出 # [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, # {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, # {'score': 2.396818e-06, 'label': 'cat'}]] ``` pengcheng.mind-mindformer/docs/transformer仓Python编程规范.md000066400000000000000000000324441476650141400303560ustar00rootroot00000000000000# 说明 本规范以[PEP8](https://www.python.org/dev/peps/pep-0008/)为基础，参考华为Python通用编码规范、安全编程规范，并结合业界共识整理而成，参与MindSpore社区开发需要首先遵循本规范内容（与PEP8冲突部分），其余遵循PEP8规范。 如果对规则异议，建议提交issue并说明理由，经MindSpore社区运营团队评审接纳后可修改生效。 a ## 适用范围 MindSpore开源社区 ------------------------ ## 1. 代码风格 ### 1.1 命名 **规则 1.1.1 包名，模块名：小写，不使用下划线。** **规则 1.1.2 类名：使用驼峰格式，首字母大写，私有类下划线前缀。** ```python class _Foo: _instance = None pass ``` **规则 1.1.3 函数名、变量名：小写，多个单词下划线分割。** ```python def _func_example(path): pass ``` **建议 1.1.4 除迭代器与计数器除外，禁止使用单字符命名。** ### 1.2 格式 **规则 1.2.1 每行字符数不要超过 120 个。** 如果超过120个字符，请选择合理的方式进行换行。 **规则 1.2.2 使用空格进行缩进，每次缩进4个空格，禁止tab缩进。** **规则 1.2.3 import顺序：标准库、第三方、自定义模块。** **规则 1.2.4 返回语句和条件语句中不使用括号。** **规则 1.2.5 模块级函数和类之间双空行，类成员函数之间一空行，注释与代码间按需添加空行，原则上不超过两空行。** **规则 1.2.6 无效或冗余代码直接删除，不要以注释、TODO等方式保留在代码中，建议提issue记录。** ### 1.3 注释 **规则 1.3.1 文件头注释必须包含版权声明。** 所有python文件，均需包含如下版权声明： ```python # Copyright 2019 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Add notes. """ import xxx ``` > 关于版权说明，应注意： > 2020年新建的文件，应该是`Copyright 2020 Huawei Technologies Co., Ltd` > 2019年创建年份，2020年修改年份，应该是`Copyright 2019-2020 Huawei Technologies Co., Ltd` **规则 1.3.2 对外的类、方法、算子、Cell注释格式。** - `class` 和 `def` 的注释格式相同，采用业界通用的python注释语法，写在声明下方并缩进，所有的 `class` 和 `def` 都需要写注释，模块内部的类和方法可以只写一条简介。 - 注释格式详见[MindSpore注释规范](https://gitee.com/mindspore/community/blob/master/security/comments_specification_zh_cn.md)。 **规则 1.3.3 不允许通过注释屏蔽pylint告警。** ### 1.4 日志 **规则 1.4.1 异常日志文本首字母大写。** **规则 1.4.2 日志文本中变量名必须使用单引号注明。** ## 2. 通用编码 ### 2.1 接口声明 **规则 2.1.1 用户接口在文件的__all__中说明，__all__摆放在import与代码之间。** **规则 2.1.2 当前文件使用的非对外方法命名采用下划线前缀，内部跨模块使用的方法无需下划线前缀，用户接口在__all__中声明。** ### 2.2 数据校验 **规则 2.2.1 对所有外部数据进行合法性检查，包括但不限于：函数入参、外部输入命名行、文件格式，文件大小、环境变量、用户数据等。** **建议 2.2.2 必须对文件路径进行规范化后再使用。** 当文件路径来自外部数据时，需要先将文件路径规范化，如果没有作规范化处理，攻击者就有机会通过恶意构造文件路径进行文件的越权访问： 例如，攻击者可以构造“../../../etc/passwd”的方式进行任意文件访问。 在linux下，使用realpath函数，在windows下，使用PathCanonicalize函数进行文件路径的规范化。 **规则 2.2.3 禁止调用OS命令解析器执行命令或运行程序。** 使用未经校验的不可信输入作为系统命令的参数或命令的一部分，可能导致命令注入漏洞。对于命令注入漏洞，命令将会以与Python应用程序相同的特权级别执行，它向攻击者提供了类似系统shell的功能。在Python中，os.system 或 os.popen 经常被用来调用一个新的进程，如果被执行的命令来自于外部输入，则可能会产生命令和参数注入。 执行命令的时候，请注意以下几点： 1. 命令执行的字符串不要去拼接输入的参数，如果必须拼接时，要对输入参数进行白名单过滤。 2. 对传入的参数要做类型校验，例如：整数数据，可以对数据进行整数强制转换。 3. 保证格式化字符串的正确性，例如：int类型参数的拼接，对于参数要用%d，不能用%s。 【错误代码示例1】 攻击者可以通过找到环境变量APPHOME对应的值，并且在相应目录下放置常量INITCMD对应的攻击程序，达到执行的效果： ```python home = os.getenv('APPHOME') cmd = os.path.join(home, INITCMD) os.system(cmd) ``` 【错误代码示例2】 没有校验属性 backuptype 的值，这个是用户输入的，攻击者可能进行攻击， 例如：用户输入的是：" && del c:\\dbms\\*.* "： ```python # 值来自用户配置 btype = req.field('backuptype') cmd = "cmd.exe /K \"c:\\util\\rmanDB.bat " + btype + "&&c:\\util\\cleanup.bat\"" os.system(cmd) ``` 【错误代码示例3】 没有校验属性 backuptype 的值，这个是用户输入的，攻击者可能进行攻击，例如：用户输入的是：" && del c:\\dbms\\*.* "： ```python import os import sys try: print(os.system("ls " + sys.argv[1])) except Exception as ex: print('exception:', ex) ``` 攻击者可以通过以下命令来利用这个漏洞程序： ```python python test.py ". && echo bad" ``` 实际将会执行两个命令： ```python ls . echo bad ``` 【正确代码示例】 避免使用 os.system，可以使用标准的 API 替代运行系统命令来完成任务： ```python import os import sys try: print(os.listdir(sys.argv[1])) except Exception as ex: print(ex) ``` ### 2.3 异常行为 **规则 2.3.1 异常必须被妥当处理，禁止抑制或者忽略已检查异常。** 每一个except 块都应该确保程序只会在继续有效的情况下才会继续运行下去。except 块必须要么从异常情况中恢复，要么重新抛出适合当前catch块上下文的另一个异常以允许最邻近的外层try-except 语句块来进行恢复工作。 【正确代码示例】 正确的做法是，避免使用 os.system，可以使用标准的 API 替代运行系统命令来完成任务： ```python validFlag = False while not validFlag: try: # If requested file does not exist, throws FileNotFoundError # If requested file exists, sets validFlag to true validFlag = True except FileNotFoundError: import traceback traceback.print_exc() ``` 【例外情况】： 1. 在资源释放失败不会影响程序后续行为的情况下，释放资源时发生的异常可以被抑制。释放资源的例子包括关闭文件、网络套接字、线程等等。这些资源通常是在except或者fianlly块中被释放，并且在后续的程序运行中都不会再被使用。因此，除非资源被耗尽，否则不会有其他途径使得这些异常会影响程序后续的行为。在充分处理了资源耗尽问题的情况下，只需对异常进行净化和记录日志（以备日后改进）就足够了；在这种情况下没必要做其他额外的错误处理。 2. 如果在特定的抽象层次上不可能从异常情况中恢复过来，则在那个层级的代码就不用处理这个异常，而是应该抛出一个合适的异常，让更高层次的代码去捕获处理，并尝试恢复。对于这种情况，最通常的实现方法是省略掉catch语句块，允许异常被广播出去。 **规则 2.3.2 使用try…except…结构对代码作保护时，需要在异常后使用finally…结构保证操作对象的释放。** 使用try…except…结构对代码作保护时，如果代码执行出现了异常，为了能够可靠地关闭操作对象，需要使用finally…结构确保释放操作对象。 【正确代码示例】 ```python handle = open(r"/tmp/sample_data.txt") # May raise IOError try: data = handle.read() # May raise UnicodeDecodeError except UnicodeDecodeError as decode_error: print(decode_error) finally: handle.close() # Always run after try: ``` **规则 2.3.3 不要使用“except:”语句来捕获所有异常。** 在异常这方面, Python非常宽容,“except:”语句真的会捕获包括Python语法错误在内的任何错误。使用“except:”很容易隐藏真正的bug，我们在使用try…except…结构对代码作保护时，应该明确期望处理的异常。Exception类是大多数运行时异常的基类，一般也应当避免在except语句中使用。通常，try只应当包含必须要在当前位置处理异常的语句，except只捕获必须处理的异常。比如对于打开文件的代码，try应当只包含open语句，except只捕获FileNotFoundError异常。对于其他预料外的异常，则让上层函数捕获，或者透传到程序外部来充分暴露问题。 【错误代码示例】 如下代码可能抛出两种异常，使用“except:”语句进行统一处理时，如果是open执行异常，将在“except:”语句之后handle无效的情况下调用close，报错handle未定义。 ```python try: handle = open(r"/tmp/sample_data.txt") # May raise IOError data = handle.read() # May raise UnicodeDecodeError except: handle.close() ``` 【正确代码示例】 ```python try: handle = open(r"/tmp/sample_data.txt") # May raise IOError try: data = handle.read() # May raise UnicodeDecodeError except UnicodeDecodeError as decode_error: print(decode_error) finally: handle.close() except(FileNotFoundError, IOError) as file_open_except: print(file_open_except) ``` **规则 2.3.4 不在except分支里面的raise都必须带异常。** raise关键字单独使用只能出现在try-except语句中，重新抛出except抓住的异常。 【错误代码示例】 ```python a = 1 if a == 1: raise ``` 【正确代码示例1】raise一个Exception或自定义的Exception ```python a = 1 if a == 1: raise Exception ``` 【正确代码示例2】在try-except语句中使用 ```python try: f = open('myfile.txt') s = f.readline() i = int(s.strip()) except IOError as e: print("I/O error({0}): {1}".format(e.errno, e.strerror)) except ValueError: print("Could not convert data to an integer.") except Exception: print("Unexpected error:", sys.exc_info()[0]) raise ``` ### 2.4 序列化和反序列化 **规则 2.4.1 pickle存在安全性问题，禁止使用pickle.load、cPickle.load和shelve模块加载不可信数据。** **规则 2.4.2 使用安全随机数。** Python产生随机数的功能在random模块中实现，实现了各种分布的伪随机数生成器。产生的随机数可以是均匀分布，高斯分布，对数正态分布，负指数分布以及alpha，beta分布，但是这些随机数都是伪随机数，不能应用于安全加密目的的应用中。 请使用/dev/random生成安全随机数，或者使用在python 3.6版本官方引入的secrets模块生成安全随机数。 【错误代码示例】 ```python import random # 伪随机数 func = random.SystemRandom() print(func.random()) print(func.randint(0, 10)) ``` 【正确代码示例】 ```python import platform # 长度请参见密码算法规范，不同场景要求长度不一样 randLength = 16 if platform.system() == 'Linux': with open("/dev/random", 'rb') as file: sr = file.read(randLength) print(sr) ``` **规则 2.4.3 assert语句通常只在测试代码中使用，禁止在Release版本中包含assert功能。** assert只应在研发过程中内部测试时使用，出现了AssertionError异常说明存在软件设计或者编码上的错误，应当修改软件予以解决。在对外发布的生产版本中禁止包含assert功能。 pengcheng.mind-mindformer/flash_attention/000077500000000000000000000000001476650141400212745ustar00rootroot00000000000000pengcheng.mind-mindformer/flash_attention/__init__.py000066400000000000000000000001101476650141400233750ustar00rootroot00000000000000# Copyright (C) 2021.Huawei Technologies Co., Ltd. All rights reserved. pengcheng.mind-mindformer/flash_attention/attention.py000066400000000000000000000466051476650141400236660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """the base class of flash attention""" from abc import ABCMeta from abc import abstractmethod from collections import defaultdict from functools import partial import te.platform as tbe_platform from tbe import tik from tbe.common.platform import get_soc_spec from constants import FP16 from constants import FP32 from constants import GM from constants import HARDWARE_MAX_STRIDE_LMT from constants import MASK_FILL_VALUE from constants import UB from constants import TILING_BLOCK_SIZE from tik_ops_utils import TikOpsUtils from tiling_strategy.sparse_tiling import SparseTiling from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class FlashAttention(metaclass=ABCMeta): """The base class of FlashAttention""" def __init__(self, q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy_cls, prev_block_num=65536, next_block_num=65536, disable_debug=True): """ Init parameter shape :param q: with shape: (B, h, N, d) :param k: with shape: (B, h, N, d) :param v: with shape: (B, h, N, d) :param attn_mask: with shape: (1, N, N) or (B, N, N) :param dropout_mask: with shape: (B, h, N, N) :param alibi_mask: with shape: (B, h, 1, N) :param kernel_name: :param tiling_stgy_cls: :param prev_block_num: :param next_block_num: :param disable_debug: """ self.tik_instance = tik.Tik(disable_debug=disable_debug) self.core_num = get_soc_spec(tbe_platform.CORE_NUM) self.M = tbe_platform.get_soc_spec(tbe_platform.L1_SIZE) self.kernel_name = kernel_name self.cont_data_mv_1_bust = partial(self.tik_instance.data_move, sid=0, nburst=1, src_stride=0, dst_stride=0) self.tik_ops_utils = TikOpsUtils(self.tik_instance) self.parse_input_shape(alibi_mask, attn_mask, dropout_mask, k, q, v) # NZ _, _, N1, M1, M0, N0 = self.q_shape self.M1 = M1 self.N1 = N1 self.M0 = M0 self.N0 = N0 self.d = N1 * N0 # ND batch_size, h, Nq, actual_d = self.q_ori_shape self.head_num = h self.B, self.Nq = batch_size * h, Nq self.N = self.k_ori_shape[2] self.actual_d = actual_d self.Nq_N0 = self.Nq * self.N0 self.N_N0 = self.N * self.N0 self.l_shape = [batch_size, h, self.Nq] self.m_shape = [batch_size, h, self.Nq] self.O_shape = self.q_shape self.prev_block_num = prev_block_num self.next_block_num = next_block_num if tiling_stgy_cls is None: self.tiling_stgy = SparseTiling(self.Nq, self.N, self.d) else: self.tiling_stgy: TilingStrategy = tiling_stgy_cls(self.Nq, self.N, self.d) self.Br = None self.last_Br = None self.Bc = None self.last_Bc = None self.Tr = None self.Tc = None self.Q_gm = None self.K_gm = None self.V_gm = None self.dim_mask_gm = None self.att_mask_gm = None self.drop_mask_gm = None self.alibi_mask_gm = None @staticmethod def get_l_gm_offset(batch_start, batch_idx, h, block_h, block_idx): """get l gm offset""" gm_offset = (batch_start + batch_idx) * h + block_idx * block_h return gm_offset @staticmethod def get_alibi_gm_offset(batch_start, batch_idx, w, block_w, block_idx): """get alibi gm offset""" gm_offset = (batch_start + batch_idx) * w + block_idx * block_w return gm_offset @staticmethod def get_drop_mask_gm_offset(batch_start, batch_idx, h, w, block_h, block_h_idx, block_w, block_w_idx): """get drop mask gm offset""" gm_offset = (batch_start + batch_idx) * h * w + block_h_idx * (w * block_h) + block_w_idx * block_w return gm_offset @abstractmethod def define_custom_inputs(self): """define custom inputs""" raise NotImplementedError @abstractmethod def define_outputs(self): """define outputs""" raise NotImplementedError @abstractmethod def collect_inputs(self): """collect inputs""" raise NotImplementedError @abstractmethod def collect_outputs(self): """collect outputs""" raise NotImplementedError @abstractmethod def compute_one_core(self, batch_start_s, batch_num_s, core_idx_to_tr_info, core_idx): """compute one core""" raise NotImplementedError @abstractmethod def prepare_global_ones(self): """prepare global ones""" raise NotImplementedError def get_gm_offset(self, batch_start, batch_idx, h, w, block_h, block_idx): """get gm offset""" gm_offset = (batch_start + batch_idx) * h * w + block_idx * block_h * self.N0 return gm_offset def get_cur_tr_block_num(self, tr_idx): """get cur tr block_num""" cur_prev_block_num = min(tr_idx, self.prev_block_num) cur_next_block_num = min(self.next_block_num, self.Tc - tr_idx - 1) block_num = cur_prev_block_num + 1 + cur_next_block_num return block_num def get_total_block_num(self): """get total block num""" block_num = 0 for b_idx in range(self.B): for tr_idx in range(self.Tr): block_num += self.get_cur_tr_block_num(tr_idx) return block_num def update_core_task_map(self, core_b_map, core_b_tr_map, core_idx, b_start, b_end, tr_start, tr_end): """update core task map""" core_b_map[core_idx][0] = min(core_b_map[core_idx][0], b_start) if tr_end == 0: # 跨head，但跨过的head不会被当前的core处理 core_b_map[core_idx][1] = max(core_b_map[core_idx][1], b_end - 1) else: core_b_map[core_idx][1] = max(core_b_map[core_idx][1], b_end) for b_idx in range(b_start, b_end + 1): if b_idx == b_end and tr_end == 0: # 跨head，但跨过的head不会被当前的core处理 break elif b_idx == b_start and b_idx == b_end: # 没跨head core_b_tr_map[core_idx][b_idx] = (tr_start, tr_end) elif b_idx == b_start: # 跨head，第一个head core_b_tr_map[core_idx][b_idx] = (tr_start, self.Tr) elif b_idx == b_end: # 跨head，最后一个head core_b_tr_map[core_idx][b_idx] = (0, tr_end) else: # 跨head，中间的head core_b_tr_map[core_idx][b_idx] = (0, self.Tr) def convert_py_dict_to_tik_tensor(self, core_b_map, core_b_tr_map): """convert py dict to tik tensor""" # python dict -> tik tensor # [batch_start, batch_idx_end] -> [batch_start, batch_num] # [tr_start, tr_idx_end] -> [tr_start, tr_idx_end) core_idx_to_batch_info = self.tik_instance.Tensor( "int32", (self.core_num, 2), name="core_idx_to_batch_info", scope=UB ) core_idx_to_tr_info = self.tik_instance.Tensor( "int32", (self.core_num, self.B, 2), name="core_idx_to_tr_info", scope=UB ) for core_idx in core_b_map.keys(): batch_start, batch_end = core_b_map[core_idx] core_idx_to_batch_info[core_idx, 0] = batch_start core_idx_to_batch_info[core_idx, 1] = batch_end - batch_start + 1 for batch_idx in core_b_tr_map[core_idx].keys(): tr_start, tr_end = core_b_tr_map[core_idx][batch_idx] core_idx_to_tr_info[core_idx, batch_idx, 0] = tr_start core_idx_to_tr_info[core_idx, batch_idx, 1] = tr_end return core_idx_to_batch_info, core_idx_to_tr_info def get_core_task_info(self): """ Get batch start and batch number of each NPU core. :return: Tensor([[core_1_batch_start, core_1_batch_num],...,[core_m_batch_start, core_m_batch_num]]), Tensor([[[core_1_batch_1_Tr_start, core_1_batch_1_Tr_end],...[core_1_batch_n_Tr_start, core_1_batch_n_Tr_end]],...,[[core_m_batch_1_Tr_start, core_m_batch_1_Tr_end],...[core_m_batch_n_Tr_start, core_m_batch_n_Tr_end]] """ if self.core_num > self.B * self.Tr: self.core_num = self.B * self.Tr total_blk_num = self.get_total_block_num() b_start = 0 tr_start = 0 remain_blk_num = total_blk_num core_b_map = defaultdict(lambda: [100000, -1]) core_b_tr_map = defaultdict(lambda: defaultdict(list)) for core_idx in range(self.core_num): cur_core_blk_num = 0 cur_each_core_blk_num = remain_blk_num // (self.core_num - core_idx) cur_core_finished = False b_end = b_start tr_end = tr_start while b_end < self.B: while tr_end < self.Tr: cur_tr_blk_num = self.get_cur_tr_block_num(tr_end) if abs(cur_core_blk_num - cur_each_core_blk_num) <= \ (cur_core_blk_num + cur_tr_blk_num - cur_each_core_blk_num): self.update_core_task_map(core_b_map, core_b_tr_map, core_idx, b_start, b_end, tr_start, tr_end) remain_blk_num -= cur_core_blk_num cur_core_finished = True break else: cur_core_blk_num += cur_tr_blk_num tr_end += 1 if tr_end == self.Tr: tr_end = 0 b_end += 1 if cur_core_finished: b_start = b_end tr_start = tr_end break core_idx_to_batch_info, core_idx_to_tr_info = self.convert_py_dict_to_tik_tensor(core_b_map, core_b_tr_map) return core_idx_to_batch_info, core_idx_to_tr_info def get_attn_mask_gm_offset(self, batch_start, batch_idx, h, w, block_h, block_h_idx, block_w, block_w_idx): """get attn mask gm offset""" if self.att_mask_shape[0] == 1: gm_offset = block_w_idx * (h * block_w) + block_h_idx * block_h * self.N0 else: gm_offset = ((batch_start + batch_idx) // self.head_num) * h * w \ + block_w_idx * (h * block_w) + block_h_idx * block_h * self.N0 return gm_offset def is_only_causal_mask(self, attn_mask_ori_shape): """check attn mask is only causal mask""" if len(attn_mask_ori_shape) == 2: return attn_mask_ori_shape[0] == TILING_BLOCK_SIZE and attn_mask_ori_shape[1] == TILING_BLOCK_SIZE if len(attn_mask_ori_shape) == 3: return attn_mask_ori_shape[0] == 1 and attn_mask_ori_shape[1] == TILING_BLOCK_SIZE and \ attn_mask_ori_shape[2] == TILING_BLOCK_SIZE def parse_input_shape(self, alibi_mask, attn_mask, dropout_mask, k, q, v): """parser input shape""" self.has_attn_mask = False self.has_drop_mask = False self.has_alibi_mask = False # NZ self.q_shape = q["shape"] self.k_shape = k["shape"] self.v_shape = v["shape"] # ND self.q_ori_shape = q["ori_shape"] self.k_ori_shape = k["ori_shape"] if attn_mask is not None: self.has_attn_mask = True self.att_mask_shape = attn_mask["shape"] attn_mask_ori_shape = attn_mask["ori_shape"] self.only_causal = self.is_only_causal_mask(attn_mask_ori_shape) if dropout_mask is not None: self.has_drop_mask = True self.drop_mask_shape = dropout_mask["shape"] if alibi_mask is not None: self.has_alibi_mask = True self.alibi_mask_shape = alibi_mask["shape"] def define_inputs_outputs(self): """define inputs outputs""" self.define_common_inputs() self.define_custom_inputs() self.define_outputs() def init(self): """init parameters""" tiling_para: TilingPara = self.tiling_stgy.tiling() self.Br = tiling_para.Br self.last_Br = tiling_para.last_Br self.Bc = tiling_para.Bc self.last_Bc = tiling_para.last_Bc self.Tr = tiling_para.Tr self.Tc = tiling_para.Tc self.define_inputs_outputs() def define_common_inputs(self): """define common input gm tensors""" self.Q_gm = self.tik_instance.Tensor(FP16, self.q_shape, name="Q_gm", scope=GM) self.K_gm = self.tik_instance.Tensor(FP16, self.k_shape, name="K_gm", scope=GM) self.V_gm = self.tik_instance.Tensor(FP16, self.v_shape, name="V_gm", scope=GM) if self.has_attn_mask: self.att_mask_gm = self.tik_instance.Tensor(FP16, self.att_mask_shape, name="att_mask_gm", scope=GM) if self.has_drop_mask: self.drop_mask_gm = self.tik_instance.Tensor(FP16, self.drop_mask_shape, name="drop_mask_gm", scope=GM) if self.has_alibi_mask: self.alibi_mask_gm = self.tik_instance.Tensor(FP16, self.alibi_mask_shape, name="alibi_mask_gm", scope=GM) def do_alibi_mask(self, Sij_ub, alibi_mask_gm_offset, m_aligned, n_aligned): """load alibi mask from gm to ub, then add Sij""" with self.tik_instance.new_stmt_scope(disable_sync=False): alibi_mask_ub = self.tik_instance.Tensor(FP16, (1, n_aligned), scope=UB, name="alibi_mask_ub") self.tik_instance.data_move(alibi_mask_ub, self.alibi_mask_gm[alibi_mask_gm_offset], 0, 1, n_aligned // 16, 0, 0) alibi_mask_ub_broadcast = self.tik_ops_utils.broadcast_row(alibi_mask_ub, (m_aligned, n_aligned)) self.tik_instance.h_add(Sij_ub, Sij_ub, alibi_mask_ub_broadcast) def do_att_mask(self, Sij_ub_zN, attn_mask_gm_offset, q_blk_height, kv_blk_height, q_blk_h_aligned, kv_blk_h_aligned, only_causal=False): """load attn mask from gm to ub, then mul it by MASK_FILL_VALUE and add Sij""" with self.tik_instance.new_stmt_scope(disable_sync=False): att_mask_ub_zN = self.tik_instance.Tensor(FP16, (kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0), scope=UB, name="att_mask_ub_zN") if only_causal: self.cont_data_mv_1_bust(dst=att_mask_ub_zN, src=self.att_mask_gm, burst=q_blk_h_aligned * kv_blk_h_aligned // 16) else: self.tik_instance.data_move(att_mask_ub_zN, self.att_mask_gm[attn_mask_gm_offset], 0, kv_blk_height // self.N0, q_blk_height * self.N0 // 16, (self.Nq - q_blk_height) * self.N0 // 16, 0) self.tik_instance.h_mul(att_mask_ub_zN, att_mask_ub_zN, MASK_FILL_VALUE) self.tik_instance.h_add(Sij_ub_zN, Sij_ub_zN, att_mask_ub_zN) def do_dropout_mask(self, Pij_ub, dropout_mask_gm_offset, kv_blk_h_aligned, kv_blk_height, q_blk_h_aligned, q_blk_height, precision_type=FP16, workspace=None): """load drop mask from gm to ub, then mul it by Pij""" with self.tik_instance.new_stmt_scope(disable_sync=False): dropout_mask_ub = self.tik_instance.Tensor(FP16, (q_blk_h_aligned, kv_blk_h_aligned), scope=UB, name="drop_mask_ub") self.tik_instance.data_move(dropout_mask_ub, self.drop_mask_gm[dropout_mask_gm_offset], 0, q_blk_height, kv_blk_height // 16, (self.N - kv_blk_height) // 16, 0) dropout_mask_ub = dropout_mask_ub.reshape((kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0)) if precision_type == FP32: dropout_mask_ub_fp32 = self.tik_instance.Tensor(FP32, (kv_blk_h_aligned // self.N0, q_blk_h_aligned, self.N0), scope=UB, name="dropout_mask_ub_fp32") self.tik_instance.h_cast(dropout_mask_ub_fp32, dropout_mask_ub, "none") if workspace is None: self.tik_instance.h_mul(Pij_ub, Pij_ub, dropout_mask_ub_fp32) else: self.tik_instance.h_mul(workspace, Pij_ub, dropout_mask_ub_fp32) else: if workspace is None: self.tik_instance.h_mul(Pij_ub, Pij_ub, dropout_mask_ub) else: self.tik_instance.h_mul(workspace, Pij_ub, dropout_mask_ub) def compute_process(self): """The compute process of FlashAttention""" self.init() self.prepare_global_ones() core_idx_to_batch_info, core_idx_to_tr_info = self.get_core_task_info() with self.tik_instance.for_range(begint=0, endt=self.core_num, name="core_index", block_num=self.core_num) as core_idx: batch_start_s = self.tik_instance.Scalar("int32", name="batch_start_s") batch_num_s = self.tik_instance.Scalar("int32", name="batch_num_s") batch_start_s.set_as(core_idx_to_batch_info[core_idx, 0]) batch_num_s.set_as(core_idx_to_batch_info[core_idx, 1]) self.compute_one_core(batch_start_s, batch_num_s, core_idx_to_tr_info, core_idx) self.tik_instance.BuildCCE( kernel_name=self.kernel_name, inputs=self.collect_inputs(), outputs=self.collect_outputs(), config={"dump_cce_code": False, "save_temp_cce_file": True, "enable_const_fold": True}, enable_l2=True ) def unroll_data_move(self, dst, dst_offset, src, src_offset, nburst, burst, src_stride, dst_stride, dst_offset_inc, src_offset_inc): if dst_stride <= HARDWARE_MAX_STRIDE_LMT and src_stride <= HARDWARE_MAX_STRIDE_LMT: self.tik_instance.data_move( dst=dst[dst_offset], src=src[src_offset], sid=0, nburst=nburst, burst=burst, dst_stride=dst_stride, src_stride=src_stride ) else: for _ in range(nburst): self.cont_data_mv_1_bust(dst=dst[dst_offset], src=src[src_offset], burst=burst) src_offset += src_offset_inc dst_offset += dst_offset_inc pengcheng.mind-mindformer/flash_attention/constants.py000066400000000000000000000022771476650141400236720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """define constants""" from tbe import tik HARDWARE_MAX_STRIDE_LMT = 65535 # < 64K TILING_BLOCK_SIZE = 128 BLOCK_NUM = 16 FP16 = "float16" INT8 = "int8" UINT8 = "uint8" INT32 = "int32" FP32 = "float32" REPEAT_SZ = 128 BLK_STRIDE = 1 REPEAT_STRIDE = 8 TRANS_CUBE_TGT = 8 FP16_MIN_VAL = -65504.0 MASK_FILL_VALUE = -10000.0 GM = tik.scope_gm L1 = tik.scope_cbuf L1OUT = tik.scope_cbuf_out UB = tik.scope_ubuf L0A = tik.scope_ca L0B = tik.scope_cb L0C = tik.scope_cc DTYPE_SIZE = { "int8": 1, "float16": 2, "int16": 2, "float32": 4, } pengcheng.mind-mindformer/flash_attention/flash_attention_bwd.py000066400000000000000000000646031476650141400256750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The backward tik ops of flash attention""" from tbe import tik from attention import FlashAttention from constants import FP16 from constants import FP32 from constants import GM from constants import L1 from constants import UB from tiling_strategy.strategy import TilingStrategy class FlashAttentionBwd(FlashAttention): """The implementation of FlashAttention backward This function contains the flash attention backward implementation used in flash attention (see paper) `FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ` """ def __init__(self, q, k, v, O, dO, l, attn_mask, dropout_mask, alibi_mask, prev_block_num, next_block_num, kernel_name, tiling_stgy: TilingStrategy, disable_debug): super().__init__(q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy, prev_block_num, next_block_num, disable_debug) self.dO_shape = dO["shape"] # [B, Nq, d] self.dV_shape = self.v_shape self.dQ_shape = self.q_shape self.dK_shape = self.k_shape self.dQ_gm = None self.dK_gm = None self.dV_gm = None self.O_gm = None self.dO_gm = None self.l_gm = None def define_outputs(self): """define output gm tensors""" self.dQ_gm = self.tik_instance.Tensor(FP32, self.dQ_shape, name="dQ_gm", scope=GM, is_atomic_add=True) self.dK_gm = self.tik_instance.Tensor(FP32, self.dK_shape, name="dK_gm", scope=GM, is_atomic_add=True) self.dV_gm = self.tik_instance.Tensor(FP32, self.dV_shape, name="dV_gm", scope=GM, is_atomic_add=True) def define_custom_inputs(self): """define input gm tensors""" self.O_gm = self.tik_instance.Tensor(FP16, self.O_shape, name="O_gm", scope=GM) self.dO_gm = self.tik_instance.Tensor(FP16, self.dO_shape, name="dO_gm", scope=GM) self.l_gm = self.tik_instance.Tensor(FP32, self.l_shape, name="l_gm", scope=GM) def collect_inputs(self): """collect all input gm tensors into input_gm_list, the input list should keep order with the para order in Primitive and init """ input_gm_list = [ self.Q_gm, self.K_gm, self.V_gm, self.O_gm, self.dO_gm, self.l_gm ] if self.has_attn_mask: input_gm_list.append(self.att_mask_gm) if self.has_drop_mask: input_gm_list.append(self.drop_mask_gm) if self.has_alibi_mask: input_gm_list.append(self.alibi_mask_gm) return input_gm_list def prepare_global_ones(self): """Prepare global ones tensor in L1 for cube impl row_sum""" self.ones_l1 = self.tik_instance.Tensor(FP16, (self.d, 16), name="ones_l1", scope=L1) with self.tik_instance.new_stmt_scope(disable_sync=False): ones_ub = self.tik_instance.Tensor(FP16, (self.d, 16), name="ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.ones_l1, src=ones_ub, burst=self.d) def compute_Pij(self, Qi_l1_K1MK0_ed, KjT_l1_K1NK0_ed, m, k, n, l_gm_offset, attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset, q_blk_idx, kv_blk_idx): """Refer to Algorithm 4 line11-14 in FlashAttention implement Pij computation""" m_aligned = self.tik_ops_utils.up_align_to_K0(m) n_aligned = self.tik_ops_utils.up_align_to_K0(n) # Sij <- Qi * KjT Sij_ub = self.tik_ops_utils.matmul_compute(Qi_l1_K1MK0_ed, KjT_l1_K1NK0_ed, m, k, n, N1MN0_to_MN=False) with self.tik_instance.new_stmt_scope(disable_sync=False): # TODO NZ 适配 alibi if self.has_attn_mask: if self.only_causal: with self.tik_instance.if_scope(q_blk_idx == kv_blk_idx): self.do_att_mask(Sij_ub, attn_mask_gm_offset, m, n, m_aligned, n_aligned, self.only_causal) else: self.do_att_mask(Sij_ub, attn_mask_gm_offset, m, n, m_aligned, n_aligned) # move li (ith block of l_gm) and mi (ith block of m_gm) from gm to ub li_ub = self.tik_instance.Tensor(FP32, (m_aligned,), name="li_ub", scope=UB) self.tik_ops_utils.move_vector_from_gm_to_ub(li_ub, self.l_gm, l_gm_offset, m) # broadcast broadcast_li_ub = self.tik_instance.Tensor(FP32, (1, m_aligned, self.N0), scope=UB, name="broadcast_li_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(li_ub[idx]) self.tik_instance.h_duplicate(broadcast_li_ub[0, idx, :], src_scalar) # fp16 -> fp32 Sij_ub_fp32 = self.tik_instance.Tensor(FP32, (n_aligned // self.N0, m_aligned, self.N0), name="Sij_ub_fp32", scope=UB) self.tik_instance.h_cast(Sij_ub_fp32, Sij_ub, "none") # Sij - Li for idx in range(n_aligned // self.N0): self.tik_instance.h_sub(Sij_ub_fp32[idx, :, :], Sij_ub_fp32[idx, :, :], broadcast_li_ub) # exp self.tik_instance.h_exp(Sij_ub_fp32, Sij_ub_fp32) # fp32 -> fp16 self.tik_instance.h_cast(Sij_ub, Sij_ub_fp32, "none") # dropout_mask if self.has_drop_mask: Pij_drop_ed_ub = self.tik_instance.Tensor(FP16, (n_aligned // self.N0, m_aligned, self.N0), name="Pij_drop_ed_ub", scope=UB) self.do_dropout_mask(Sij_ub, dropout_mask_gm_offset, n_aligned, n, m_aligned, m, workspace=Pij_drop_ed_ub) else: Pij_drop_ed_ub = Sij_ub return Sij_ub, Pij_drop_ed_ub def compute_Di(self, Di_ub, dOi_ub, qo_gm_offset, q_blk_height): """Refer to Algorithm 4 line19 in FlashAttention implement Di computation""" q_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_height) with self.tik_instance.new_stmt_scope(disable_sync=False): Oi_ub = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_aligned, self.N0), scope=UB, name="Oi_ub") self.unroll_data_move(dst=Oi_ub, dst_offset=0, src=self.O_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height * self.N0 // 16, src_stride=(self.Nq - q_blk_height) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height * self.N0, src_offset_inc=self.Nq * self.N0) self.tik_instance.h_mul(Oi_ub, dOi_ub, Oi_ub) dOi_Oi_l1_K1MK0 = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_aligned, self.N0), name="dOi_Oi_l1_K1MK0", scope=L1) self.cont_data_mv_1_bust(dst=dOi_Oi_l1_K1MK0, src=Oi_ub, burst=q_blk_height_aligned * self.d // 16) self.tik_ops_utils.row_sum_cube_impl(dOi_Oi_l1_K1MK0, self.ones_l1, Di_ub, q_blk_height, self.actual_d, precision_type=FP16) def compute_dSij(self, Pij_ub, dOi_l1_K1MK0_ed, VjT_K1NK0_ed, Di_ub, kv_blk_height, q_blk_height, dropout_mask_gm_offset): """Refer to Algorithm 4 line20 in FlashAttention implement dSij computation""" with self.tik_instance.new_stmt_scope(disable_sync=False): dPij_ub = self.tik_ops_utils.matmul_compute(dOi_l1_K1MK0_ed, VjT_K1NK0_ed, q_blk_height, self.actual_d, kv_blk_height, N1MN0_to_MN=False) q_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_height) kv_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_height) # dropout_mask if self.has_drop_mask: self.do_dropout_mask(dPij_ub, dropout_mask_gm_offset, kv_blk_height_aligned, kv_blk_height, q_blk_height_aligned, q_blk_height) # dPij - Di with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_Di_ub = self.tik_ops_utils.broadcast_vec_from_M_to_MN0(Di_ub) broadcast_Di_ub = broadcast_Di_ub.reshape((1, q_blk_height_aligned, self.N0)) n1 = kv_blk_height_aligned // self.N0 for idx in range(n1): self.tik_instance.h_sub(dPij_ub[idx, :, :], dPij_ub[idx, :, :], broadcast_Di_ub) self.tik_instance.h_mul(Pij_ub, Pij_ub, dPij_ub) return Pij_ub def update_dVj(self, Pij_l1_zN, dOi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height): """Refer to Algorithm 4 line16 in FlashAttention implement dVj update""" with self.tik_instance.new_stmt_scope(disable_sync=False): PijT_Oi_ub = self.tik_ops_utils.mmad_compute_transpose(Pij_l1_zN, dOi_l1_zN, kv_blk_height, q_blk_height, self.actual_d, N1MN0_to_MN=False, a_transpose=True, precision_type=FP32) self.tik_instance.set_atomic_add(1) # 实测 2 (fp16) 不支持 self.unroll_data_move(dst=self.dV_gm, dst_offset=kv_gm_offset, src=PijT_Oi_ub, src_offset=0, nburst=self.N1, burst=kv_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - kv_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=kv_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def update_dQi(self, dSij_l1_zN, Kj_l1_zN, qo_gm_offset, q_blk_height, kv_blk_height): """Refer to Algorithm 4 line21 in FlashAttention implement dQi update""" with self.tik_instance.new_stmt_scope(disable_sync=False): dSij_Kj_ub = self.tik_ops_utils.mmad_compute(dSij_l1_zN, Kj_l1_zN, q_blk_height, kv_blk_height, self.actual_d, N1MN0_to_MN=False, precision_type=FP32) self.tik_instance.set_atomic_add(1) self.unroll_data_move(dst=self.dQ_gm, dst_offset=qo_gm_offset, src=dSij_Kj_ub, src_offset=0, nburst=self.d // self.N0, burst=q_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - q_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=q_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def update_dKj(self, dSij_l1_zN, Qi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height): """Refer to Algorithm 4 line22 in FlashAttention implement dKi update""" with self.tik_instance.new_stmt_scope(disable_sync=False): dSijT_Qi_ub = self.tik_ops_utils.mmad_compute_transpose(dSij_l1_zN, Qi_l1_zN, kv_blk_height, q_blk_height, self.actual_d, N1MN0_to_MN=False, a_transpose=True, precision_type=FP32) self.tik_instance.set_atomic_add(1) self.unroll_data_move(dst=self.dK_gm, dst_offset=kv_gm_offset, src=dSijT_Qi_ub, src_offset=0, nburst=self.d // self.N0, burst=kv_blk_height * self.N0 // 8, src_stride=0, dst_stride=(self.Nq - kv_blk_height) * self.N0 // 8, dst_offset_inc=self.Nq * self.N0, src_offset_inc=kv_blk_height * self.N0) self.tik_instance.set_atomic_add(0) def compute_in_each_kv_block(self, batch_start, batch_idx, kv_blk_idx, kv_blk_height, core_idx_to_tr_info, core_idx): """The backward computation in each outer loop""" kv_blk_height_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_height) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx) # load KjT Kj_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_height_aligned, self.N0), name="Kj_l1_zN", scope=L1) # for Qi*KjT (算法11行) self.unroll_data_move(dst=Kj_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_height_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_height_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_height_aligned * self.N0, src_offset_inc=self.N * self.N0) # load VjT Vj_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_height_aligned, self.N0), name="Vj_l1_zN", scope=L1) self.unroll_data_move(dst=Vj_l1_zN, dst_offset=0, src=self.V_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_height_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_height_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_height_aligned * self.N0, src_offset_inc=self.N * self.N0) tr_start_s = self.tik_instance.Scalar("int32", name="tr_start_s") tr_end_s = self.tik_instance.Scalar("int32", name="tr_end_s") tr_start_s.set_as(core_idx_to_tr_info[core_idx, batch_start + batch_idx, 0]) tr_end_s.set_as(core_idx_to_tr_info[core_idx, batch_start + batch_idx, 1]) with self.tik_instance.for_range(tr_start_s, tr_end_s, name="q_blk_idx") as q_blk_idx: with self.tik_instance.if_scope(tik.all(kv_blk_idx - self.next_block_num <= q_blk_idx, q_blk_idx <= kv_blk_idx + self.prev_block_num)): with self.tik_instance.if_scope(q_blk_idx != self.Tr - 1): self.compute_in_each_q_block(Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, self.Br, kv_blk_idx, q_blk_idx) with self.tik_instance.else_scope(): self.compute_in_each_q_block(Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, self.last_Br, kv_blk_idx, q_blk_idx) def compute_in_each_q_block(self, Kj_l1_zN, Vj_l1_zN, batch_idx, batch_start, kv_gm_offset, kv_blk_height, q_blk_height, kv_blk_idx, q_blk_idx): """The backward computation in each inner loop""" kv_blk_height_alig = self.tik_ops_utils.up_align_to_K0(kv_blk_height) q_blk_height_alig = self.tik_ops_utils.up_align_to_K0(q_blk_height) # load Qi并做分形 (L1: 256K) qo_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.Nq, self.d, self.Br, q_blk_idx) Qi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="Qi_l1_zN", scope=L1) # for Qi*KjT (算法11行) self.unroll_data_move(dst=Qi_l1_zN, dst_offset=0, src=self.Q_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height_alig * self.N0 // 16, src_stride=(self.Nq - q_blk_height_alig) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height_alig * self.N0, src_offset_inc=self.Nq * self.N0) l_gm_offset = self.get_l_gm_offset(batch_start, batch_idx, self.Nq, self.Br, q_blk_idx) attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset = None, None, None if self.has_attn_mask: if self.only_causal: attn_mask_gm_offset = 0 else: attn_mask_gm_offset = self.get_attn_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) if self.has_drop_mask: dropout_mask_gm_offset = self.get_drop_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) if self.has_alibi_mask: alibi_mask_gm_offset = self.get_alibi_gm_offset(batch_start, batch_idx, self.N, self.Bc, kv_blk_idx) # 算法11~15行 Pij_ub, Pij_drop_ed_ub = self.compute_Pij(Qi_l1_zN, Kj_l1_zN, q_blk_height, self.actual_d, kv_blk_height, l_gm_offset, attn_mask_gm_offset, dropout_mask_gm_offset, alibi_mask_gm_offset, q_blk_idx, kv_blk_idx) dOi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="dOi_l1_zN", scope=L1) # for dOi * VjT (算法17行) okk self.unroll_data_move( dst=dOi_l1_zN, dst_offset=0, src=self.dO_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_height_alig * self.N0 // 16, src_stride=(self.Nq - q_blk_height_alig) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_height_alig * self.N0, src_offset_inc=self.Nq * self.N0 ) # load dOi并做分形，提前计算Di，否则dOi_ub无法释放，或者后面计算Di时需要再次load dOi (L1: 320K) Di_ub = self.tik_instance.Tensor(FP16, (q_blk_height_alig,), name="Di_ub", scope=UB) # 为了释放dOi_ub，提前计算Di with self.tik_instance.new_stmt_scope(disable_sync=False): dOi_ub = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_height_alig, self.N0), name="dOi_ub", scope=UB) self.cont_data_mv_1_bust(dst=dOi_ub, src=dOi_l1_zN, burst=q_blk_height_alig * self.d // 16) self.compute_Di(Di_ub, dOi_ub, qo_gm_offset, q_blk_height) # 分形，为PijT*dOi(算法16行)做准备 Pij_l1_zN = self.tik_instance.Tensor(FP16, (q_blk_height_alig, kv_blk_height_alig), name="Pij_l1_zN", scope=L1) self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Pij_drop_ed_ub, burst=q_blk_height_alig * kv_blk_height_alig // 16) # 算法24行：write dKj dVj (放入内循环=>增大tiling的blocksize) self.update_dVj(Pij_l1_zN, dOi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height) # (L1: 512K) dSij_l1_Nz = self.tik_instance.Tensor(FP16, (kv_blk_height_alig // self.N0, q_blk_height_alig, self.N0), name="dSij_l1_Nz", scope=L1) # for dSij*Kj (算法21行) with self.tik_instance.new_stmt_scope(disable_sync=False): # 为了释放dSij_ub dSij_ub = self.compute_dSij(Pij_ub, dOi_l1_zN, Vj_l1_zN, Di_ub, kv_blk_height, q_blk_height, dropout_mask_gm_offset) # for dSij*Kj (算法21行) self.cont_data_mv_1_bust(dst=dSij_l1_Nz, src=dSij_ub, burst=kv_blk_height_alig * q_blk_height_alig // 16) # 算法21行 self.update_dQi(dSij_l1_Nz, Kj_l1_zN, qo_gm_offset, q_blk_height, kv_blk_height) # 算法22行 self.update_dKj(dSij_l1_Nz, Qi_l1_zN, kv_gm_offset, kv_blk_height, q_blk_height) def compute_one_core(self, batch_start_sc, batch_num_sc, core_idx_to_tr_info, core_idx): """The computation of FlashAttention backward on each core""" with self.tik_instance.for_range(0, batch_num_sc, name="batch_index") as batch_idx: with self.tik_instance.for_range(0, self.Tc, name="kv_blk_idx") as kv_blk_idx: with self.tik_instance.if_scope(kv_blk_idx != self.Tc - 1): self.compute_in_each_kv_block(batch_start_sc, batch_idx, kv_blk_idx, self.Bc, core_idx_to_tr_info, core_idx) with self.tik_instance.else_scope(): self.compute_in_each_kv_block(batch_start_sc, batch_idx, kv_blk_idx, self.last_Bc, core_idx_to_tr_info, core_idx) def collect_outputs(self): """collect all output gm tensors into output_gm_list, the output list should keep order with the para order in Primitive and init """ output_gm_list = [self.dQ_gm, self.dK_gm, self.dV_gm] return output_gm_list def flash_attention_grad(query, key, value, output, dO, rowsum, attn_mask, dropout_mask, alibi_mask, dq, dk, dv, prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse', kernel_name="flash_attention_grad", disable_debug=True): """ algorithm: flash_attention_backward Parameters ---------- query : dict. shape and dtype of input, only support float16 key : dict. shape and dtype of input, only support float16 value: dict. shape and dtype of input, only support float16 output: dict. shape and dtype of input, only support float16 dO: dict. shape and dtype of input, only support float16 rowsum: dict. shape and dtype of input, only support float32 attn_mask: dict. shape and dtype of input, only support float16 dropout_mask: dict. shape and dtype of input, only support float16 alibi_mask: dict. shape and dtype of input, only support float16 dq: dict. shape and dtype of output, only support float16 dk: dict. shape and dtype of output, only support float16 dv: dict. shape and dtype of output, only support float16 prev_block_num: int. an attribute used to define sparse attention next_block_num: int. an attribute used to define sparse attention tiling_stgy_name: str. an attribute used to choose the tiling strategy kernel_name: str. cce kernel name, default value is real_div disable_debug: bool. whether disable debug Returns ------- tik_instance """ fa_grad = FlashAttentionBwd(q=query, k=key, v=value, O=output, dO=dO, l=rowsum, attn_mask=attn_mask, dropout_mask=dropout_mask, alibi_mask=alibi_mask, prev_block_num=prev_block_num, next_block_num=next_block_num, kernel_name=kernel_name, tiling_stgy=TilingStrategy.from_strategy_name(tiling_stgy_name), disable_debug=disable_debug) fa_grad.compute_process() return fa_grad.tik_instance pengcheng.mind-mindformer/flash_attention/flash_attention_fwd.py000066400000000000000000000706751476650141400257070ustar00rootroot00000000000000from tbe import tik from attention import FlashAttention from constants import FP16 from constants import FP32 from constants import GM from constants import L1 from constants import UB from tiling_strategy.strategy import TilingStrategy class FlashAttentionFwd(FlashAttention): """The implementation of flash attention forward This function contains the flash attention forward implementation used in flash attention (see paper) `FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning ` """ def __init__(self, q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy: TilingStrategy, prev_block_num=65536, next_block_num=65536, disable_debug=True): super().__init__(q, k, v, attn_mask, dropout_mask, alibi_mask, kernel_name, tiling_stgy, prev_block_num, next_block_num, disable_debug) def define_custom_inputs(self): pass def define_outputs(self): """define output gm tensors""" self.O_gm = self.tik_instance.Tensor(FP16, self.O_shape, name="O_gm", scope=GM, is_atomic_add=True) self.l_gm = self.tik_instance.Tensor(FP32, self.l_shape, name="l_gm", scope=GM, is_atomic_add=True) def prepare_global_ones(self): """Prepare global ones tensor in L1 for cube impl row_sum""" Bc_aligned = (self.Bc + 15) // 16 * 16 last_Bc_aligned = (self.last_Bc + 15) // 16 * 16 self.ones_l1 = self.tik_instance.Tensor(FP16, (Bc_aligned, 16), name="ones_l1", scope=L1) self.last_ones_l1 = self.tik_instance.Tensor(FP16, (last_Bc_aligned, 16), name="last_ones_l1", scope=L1) with self.tik_instance.new_stmt_scope(disable_sync=False): ones_ub = self.tik_instance.Tensor(FP16, (Bc_aligned, 16), name="ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.ones_l1, src=ones_ub, burst=Bc_aligned) last_ones_ub = self.tik_instance.Tensor(FP16, (last_Bc_aligned, 16), name="last_ones_ub", scope=UB) self.tik_instance.h_duplicate(ones_ub, 1.0) self.cont_data_mv_1_bust(dst=self.last_ones_l1, src=last_ones_ub, burst=last_Bc_aligned) def rowmax_compute(self, Sij_ub_zN, mi_ub, kv_blk_idx): """Calculate row max for softmax of flashattention2 :param Sij_ub_zN: with shape N1 x M x N0 :param mi_ub: :param kv_blk_idx: :return: """ n1, m, n0 = Sij_ub_zN.shape # update_scale_ub: for update li and Oi(exp(mi_old - mi_new)) update_scale_ub = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="update_scale_ub") with self.tik_instance.if_scope(kv_blk_idx != 0): self.tik_instance.h_cast(update_scale_ub, mi_ub, "none") with self.tik_instance.new_stmt_scope(disable_sync=False): # TODO: 二分 mn0_block_max = self.tik_instance.Tensor(FP16, (1, m, n0), name="mn0_block_max", scope=UB) self.cont_data_mv_1_bust(dst=mn0_block_max, src=Sij_ub_zN, burst=m) # 用第一个(m, n0)始化，少一次duplicate和max for idx in range(1, n1): self.tik_instance.h_max(mn0_block_max, mn0_block_max, Sij_ub_zN[idx, :, :]) # reduce_max mn0_block_max = mn0_block_max.reshape((m, n0)) mij_ub = self.tik_instance.Tensor(FP16, (m,), scope=UB, name="mij_ub") self.tik_instance.h_reduce_max(mij_ub, mn0_block_max, 1) # update mi with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=mi_ub, src=mij_ub, burst=m // 16) with self.tik_instance.else_scope(): self.tik_instance.h_max(mi_ub, mi_ub, mij_ub) with self.tik_instance.if_scope(kv_blk_idx != 0): # exp(mi_old - mi_new) mi_ub_fp32 = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="mi_ub_fp32") self.tik_instance.h_cast(mi_ub_fp32, mi_ub, "none") self.tik_instance.h_sub(update_scale_ub, update_scale_ub, mi_ub_fp32) self.tik_instance.h_exp(update_scale_ub, update_scale_ub) return mi_ub, update_scale_ub def rowsum_compute(self, Pij_l1_zN, li_ub, update_scale_ub, kv_blk_idx): """Calculate row sum for softmax of flashattention2 :param Pij_l1_zN: with shape N1 x M x N0 :param li_ub: :param update_scale_ub: :param kv_blk_idx: :return: """ n1, m, n0 = Pij_l1_zN.shape # cube impl rowsum of Pij lij_ub = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="lij_ub") if n1 * n0 == self.Bc: lij_ub = self.tik_ops_utils.row_sum_cube_impl(Pij_l1_zN, self.ones_l1, lij_ub, m, self.Bc, precision_type=FP32) else: lij_ub = self.tik_ops_utils.row_sum_cube_impl(Pij_l1_zN, self.last_ones_l1, lij_ub, m, self.last_Bc, precision_type=FP32) # update li with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=li_ub, src=lij_ub, burst=m // 8) with self.tik_instance.else_scope(): self.tik_instance.h_mul(li_ub, li_ub, update_scale_ub) self.tik_instance.h_add(li_ub, li_ub, lij_ub) return li_ub def softmax_compute(self, Sij_ub_zN, li_ub, mi_ub, kv_blk_idx, drop_mask_gm_offset): """Calculate softmax :param Sij_ub_zN: with shape N1 x M x N0 :param li_ub: :param mi_ub: :param kv_blk_idx: :param drop_mask_gm_offset: :return: """ # rowmax mi_ub, update_scale_ub = self.rowmax_compute(Sij_ub_zN, mi_ub, kv_blk_idx) # Sij - mi n1, m, n0 = Sij_ub_zN.shape with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_mij_ub = self.tik_ops_utils.broadcast_vec_from_M_to_MN0(mi_ub) broadcast_mij_ub = broadcast_mij_ub.reshape((1, m, n0)) for idx in range(n1): self.tik_instance.h_sub(Sij_ub_zN[idx, :, :], Sij_ub_zN[idx, :, :], broadcast_mij_ub) # exp with self.tik_instance.new_stmt_scope(disable_sync=False): Sij_ub_zN_fp32 = self.tik_instance.Tensor(FP32, (n1, m, n0), scope=UB, name="Sij_ub_zN_fp32") self.tik_instance.h_cast(Sij_ub_zN_fp32, Sij_ub_zN, "none") self.tik_instance.h_exp(Sij_ub_zN_fp32, Sij_ub_zN_fp32) self.tik_instance.h_cast(Sij_ub_zN, Sij_ub_zN_fp32, "none") # rowsum Pij_l1_zN = self.tik_instance.Tensor(FP16, (n1, m, n0), scope=L1, name="Pij_l1_zN") self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Sij_ub_zN, burst=n1 * m * n0 // 16) li_ub = self.rowsum_compute(Pij_l1_zN, li_ub, update_scale_ub, kv_blk_idx) # dropout if self.has_drop_mask: self.do_dropout_mask(Sij_ub_zN, drop_mask_gm_offset, n1 * n0, n1 * n0, m, m) self.cont_data_mv_1_bust(dst=Pij_l1_zN, src=Sij_ub_zN, burst=n1 * m * n0 // 16) return Pij_l1_zN, li_ub, mi_ub, update_scale_ub def output_update(self, Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size, m): """Update output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param Oij_ub_zN: :param broadcast_scale_ub: :param offset: :param n1_begin: :param n1_size: :param m: :return: """ n1_end = n1_begin + n1_size Oi_ub_zN = self.tik_instance.Tensor(FP32, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN") # mte1: L1 -> UB self.cont_data_mv_1_bust(dst=Oi_ub_zN, src=Oi_l1_zN[offset], burst=n1_size * m * self.N0 // 8) # vector: mul and add for i in range(n1_size): self.tik_instance.h_mul(Oi_ub_zN[i, :, :], Oi_ub_zN[i, :, :], broadcast_scale_ub) self.tik_instance.h_add(Oi_ub_zN, Oi_ub_zN, Oij_ub_zN[n1_begin:n1_end, :, :]) # mte3: UB -> L1 self.cont_data_mv_1_bust(dst=Oi_l1_zN[offset], src=Oi_ub_zN, burst=n1_size * m * self.N0 // 8) def output_compute(self, Oi_l1_zN, Pij_l1_zN, Vj_l1_zN, update_scale_ub, q_blk_h, kv_blk_idx, kv_blk_h): """Calculate output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param Pij_l1_zN: :param Vj_l1_zN: :param update_scale_ub: :param q_blk_h: :param kv_blk_idx: :param kv_blk_h: :return: """ m = Pij_l1_zN.shape[1] with self.tik_instance.new_stmt_scope(disable_sync=False): Oij_ub_zN = self.tik_ops_utils.mmad_compute(Pij_l1_zN, Vj_l1_zN, q_blk_h, kv_blk_h, self.actual_d, N1MN0_to_MN=False, precision_type=FP32) with self.tik_instance.if_scope(kv_blk_idx == 0): self.cont_data_mv_1_bust(dst=Oi_l1_zN, src=Oij_ub_zN, burst=m * self.d // 8) with self.tik_instance.else_scope(): broadcast_scale_ub = self.tik_instance.Tensor(FP32, (1, m, self.N0), scope=UB, name="broadcast_scale_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(update_scale_ub[idx]) self.tik_instance.h_duplicate(broadcast_scale_ub[0, idx, :], src_scalar) single_loop_size = 8 # prevent ub overflow loop_times, tail_num = divmod(self.N1, single_loop_size) for idx in range(0, loop_times): # double buffer with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: single_thread_size = single_loop_size // 2 offset = idx * single_loop_size * m * self.N0 + t_idx * single_thread_size * m * self.N0 n1_begin = idx * single_loop_size + t_idx * single_thread_size self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, single_thread_size, m) if tail_num > 0: offset = loop_times * single_loop_size * m * self.N0 thread_num = min(tail_num, 2) # thread1 n1_begin = loop_times * single_loop_size n1_size1 = tail_num // thread_num self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size1, m) # thread2 n1_size2 = tail_num - n1_size1 if n1_size2 > 0: offset = offset + (tail_num // thread_num) * m * self.N0 n1_begin = loop_times * single_loop_size + tail_num // thread_num self.output_update(Oi_l1_zN, Oij_ub_zN, broadcast_scale_ub, offset, n1_begin, n1_size2, m) return Oi_l1_zN def output_norm(self, Oi_l1_zN, broadcast_li_rec_ub, qo_gm_offset, o_l1_offset, n1_size, m): """Normalize output of flashattention2 :param Oi_l1_zN: with shape N1 x M x N0 :param broadcast_li_rec_ub: :param qo_gm_offset: :param o_l1_offset: :param n1_size: :param m: :return: """ Oi_ub_zN = self.tik_instance.Tensor(FP32, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN") # mte1: L1 -> UB self.cont_data_mv_1_bust(dst=Oi_ub_zN, src=Oi_l1_zN[o_l1_offset], burst=n1_size * m * self.N0 // 8) # vector: mul and cast for i in range(n1_size): self.tik_instance.h_mul(Oi_ub_zN[i, :, :], Oi_ub_zN[i, :, :], broadcast_li_rec_ub) Oi_ub_zN_fp16 = self.tik_instance.Tensor(FP16, (n1_size, m, self.N0), scope=UB, name="Oi_ub_zN_fp16") self.tik_instance.h_cast(Oi_ub_zN_fp16, Oi_ub_zN, "none") # mte3: ub -> out self.unroll_data_move(dst=self.O_gm, dst_offset=qo_gm_offset, src=Oi_ub_zN_fp16, src_offset=0, nburst=n1_size, burst=m * self.N0 // 16, src_stride=0, dst_stride=(self.Nq - m) * self.N0 // 16, dst_offset_inc=self.Nq * self.N0, src_offset_inc=m * self.N0 ) def output_norm_and_write(self, Oi_l1_zN, li_ub, mi_ub, qo_gm_offset, l_gm_offset, q_blk_h): """Normalize output and write it back to gm :param Oi_l1_zN: with shape N1 x M x N0 :param li_ub: :param mi_ub: :param qo_gm_offset: :param l_gm_offset: :param q_blk_h: :return: """ m = Oi_l1_zN.shape[1] li_rec_ub = self.tik_ops_utils.calc_vec_rec(li_ub, m) with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_li_rec_ub = self.tik_instance.Tensor(FP32, (1, m, self.N0), scope=UB, name="broadcast_li_rec_ub") src_scalar = self.tik_instance.Scalar(FP32, name="src_scalar") for idx in range(0, m): src_scalar.set_as(li_rec_ub[idx]) self.tik_instance.h_duplicate(broadcast_li_rec_ub[0, idx, :], src_scalar) single_loop_size = 12 # prevent ub overflow loop_times, tail_num = divmod(self.N1, single_loop_size) for idx in range(0, loop_times): # double buffer with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: single_thread_size = single_loop_size // 2 cur_qo_gm_offset = qo_gm_offset + \ (idx * single_loop_size + t_idx * single_thread_size) * self.Nq * self.N0 o_l1_offset = (idx * single_loop_size + t_idx * single_thread_size) * m * self.N0 self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset, o_l1_offset, single_thread_size, m) if tail_num > 0: thread_num = min(tail_num, 2) # thread1 cur_qo_gm_offset1 = qo_gm_offset + loop_times * single_loop_size * self.Nq * self.N0 o_l1_offset1 = loop_times * single_loop_size * m * self.N0 n1_size1 = tail_num // thread_num self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset1, o_l1_offset1, n1_size1, m) # thread2 n1_size2 = tail_num - n1_size1 if n1_size2 > 0: cur_qo_gm_offset2 = cur_qo_gm_offset1 + n1_size1 * self.Nq * self.N0 o_l1_offset2 = o_l1_offset1 + n1_size1 * m * self.N0 self.output_norm(Oi_l1_zN, broadcast_li_rec_ub, cur_qo_gm_offset2, o_l1_offset2, n1_size2, m) # compute li mi_ub_fp32 = self.tik_instance.Tensor(FP32, (m,), scope=UB, name="mi_ub_fp32") self.tik_instance.h_cast(mi_ub_fp32, mi_ub, "none") self.tik_instance.h_ln(li_ub, li_ub) self.tik_instance.h_add(li_ub, li_ub, mi_ub_fp32) self.tik_ops_utils.move_vector_from_ub_to_gm(self.l_gm, li_ub, l_gm_offset, q_blk_h) def compute_in_each_kv_block(self, Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, kv_blk_idx, q_blk_h, kv_blk_h, workspace=None): """The forward computation in each inner loop""" q_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_h) kv_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(kv_blk_h) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx) # load Vj (kv_blk_idx_th block of V_gm), then reorder for Pij*Vj Vj_l1_zN = self.tik_instance.Tensor(FP16, (kv_blk_h_aligned, self.d), name="Vj_l1_zN", scope=L1) self.unroll_data_move(dst=Vj_l1_zN, dst_offset=0, src=self.V_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_h_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_h_aligned * self.N0, src_offset_inc=self.N * self.N0 ) if workspace is None: # load Kj (kv_blk_idx_th block of K_gm) KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, kv_blk_h_aligned, self.N0), name="KjT_l1_K1MK0_ed", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=kv_blk_h_aligned * self.N0 // 16, src_stride=(self.N - kv_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=kv_blk_h_aligned * self.N0, src_offset_inc=self.N * self.N0 ) # Qi*Kj.T Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, kv_blk_h, N1MN0_to_MN=False) else: Sij_ub_zN = workspace # Sij <- Sij + attn_mask if self.has_attn_mask: if self.only_causal: with self.tik_instance.if_scope(q_blk_idx == kv_blk_idx): # main diagonal self.do_att_mask(Sij_ub_zN, 0, q_blk_h, kv_blk_h, q_blk_h_aligned, kv_blk_h_aligned, self.only_causal) else: attn_mask_gm_offset = self.get_attn_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) self.do_att_mask(Sij_ub_zN, attn_mask_gm_offset, q_blk_h, kv_blk_h, q_blk_h_aligned, kv_blk_h_aligned) # Pij = softmax(Sij), Pij <- dropout(Pij) drop_mask_gm_offset = None if self.has_drop_mask: drop_mask_gm_offset = self.get_drop_mask_gm_offset(batch_start, batch_idx, self.Nq, self.N, self.Br, q_blk_idx, self.Bc, kv_blk_idx) Pij_l1_zN, li_ub, mi_ub, update_scale_ub = self.softmax_compute(Sij_ub_zN, li_ub, mi_ub, kv_blk_idx, drop_mask_gm_offset) # do Sij = matmul(Qi, Kj.T) for next inner loop with self.tik_instance.if_scope(kv_blk_idx < self.Tc - 2): kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, kv_blk_idx + 1) KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, self.Bc, self.N0), name="KjT_l1_zN", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=self.Bc * self.N0 // 16, src_stride=(self.N - self.Bc) * self.N0 // 16, dst_stride=0, dst_offset_inc=self.Bc * self.N0, src_offset_inc=self.N * self.N0 ) Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, self.Bc, N1MN0_to_MN=False, workspace=Sij_ub_zN) Oi_l1_zN = self.output_compute(Oi_l1_zN, Pij_l1_zN, Vj_l1_zN, update_scale_ub, q_blk_h, kv_blk_idx, kv_blk_h) return Oi_l1_zN, li_ub, mi_ub def compute_in_each_q_block(self, batch_start, batch_idx, q_blk_idx, q_blk_h): """The forward computation in each outer loop""" q_blk_h_aligned = self.tik_ops_utils.up_align_to_K0(q_blk_h) # initialize Oi_l1, li_ub, mi_ub Oi_l1_zN = self.tik_instance.Tensor(FP32, (self.d // self.N0, q_blk_h_aligned, self.N0), scope=L1, name="Oi_l1_zN") li_ub = self.tik_instance.Tensor(FP32, (q_blk_h_aligned,), scope=UB, name="li_ub") mi_ub = self.tik_instance.Tensor(FP16, (q_blk_h_aligned,), scope=UB, name="mi_ub") # load Qi (q_blk_idx_th block of Q_gm) qo_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.Nq, self.d, self.Br, q_blk_idx) Qi_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, q_blk_h_aligned, self.N0), scope=L1, name="Qi_l1_zN") self.unroll_data_move(dst=Qi_l1_zN, dst_offset=0, src=self.Q_gm, src_offset=qo_gm_offset, nburst=self.N1, burst=q_blk_h_aligned * self.N0 // 16, src_stride=(self.Nq - q_blk_h_aligned) * self.N0 // 16, dst_stride=0, dst_offset_inc=q_blk_h_aligned * self.N0, src_offset_inc=self.Nq * self.N0 ) with self.tik_instance.new_stmt_scope(disable_sync=False): # load K0 (0_th block of K_gm) kv_gm_offset = self.get_gm_offset(batch_start, batch_idx, self.N, self.d, self.Bc, 0) # zN等价于转置后的K1NK0 KjT_l1_zN = self.tik_instance.Tensor(FP16, (self.d // self.N0, self.Bc, self.N0), name="KjT_l1_zN", scope=L1) self.unroll_data_move(dst=KjT_l1_zN, dst_offset=0, src=self.K_gm, src_offset=kv_gm_offset, nburst=self.N1, burst=self.Bc * self.N0 // 16, src_stride=(self.N - self.Bc) * self.N0 // 16, dst_stride=0, dst_offset_inc=self.Bc * self.N0, src_offset_inc=self.N * self.N0 ) # do Sij = matmul(Qi, Kj.T) for first inner loop Sij_ub_zN = self.tik_instance.Tensor(FP16, (self.Bc // 16, q_blk_h_aligned, 16), scope=UB, name="Sij_ub_zN") Sij_ub_zN = self.tik_ops_utils.matmul_compute(Qi_l1_zN, KjT_l1_zN, q_blk_h, self.actual_d, self.Bc, N1MN0_to_MN=False, workspace=Sij_ub_zN) with self.tik_instance.for_range(0, self.Tc - 1, name="kv_blk_idx") as kv_blk_idx: with self.tik_instance.if_scope(tik.all(kv_blk_idx - self.next_block_num <= q_blk_idx, q_blk_idx <= kv_blk_idx + self.prev_block_num)): Oi_l1_zN, li_ub, mi_ub = self.compute_in_each_kv_block(Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, kv_blk_idx, q_blk_h, self.Bc, workspace=Sij_ub_zN) # last kv blk with self.tik_instance.if_scope(tik.all(self.Tc - 1 - self.next_block_num <= q_blk_idx, q_blk_idx <= self.Tc - 1 + self.prev_block_num)): Oi_l1_zN, li_ub, mi_ub = self.compute_in_each_kv_block(Qi_l1_zN, Oi_l1_zN, li_ub, mi_ub, batch_start, batch_idx, q_blk_idx, self.Tc - 1, q_blk_h, self.last_Bc) # normalize output and write it to gm l_gm_offset = self.get_l_gm_offset(batch_start, batch_idx, self.Nq, self.Br, q_blk_idx) self.output_norm_and_write(Oi_l1_zN, li_ub, mi_ub, qo_gm_offset, l_gm_offset, q_blk_h) def compute_one_core(self, batch_start_sc, batch_num_sc, core_idx_to_tr_info, core_idx): """The computation of FlashAttention forward on each core""" with self.tik_instance.for_range(0, batch_num_sc, name="batch_index") as batch_idx: tr_start_s = self.tik_instance.Scalar("int32", init_value=core_idx_to_tr_info[ core_idx, batch_start_sc + batch_idx, 0], name="tr_start_s") tr_end_s = self.tik_instance.Scalar("int32", init_value=core_idx_to_tr_info[core_idx, batch_start_sc + batch_idx, 1], name="tr_end_s") with self.tik_instance.for_range(tr_start_s, tr_end_s - 1, name="q_blk_idx") as q_blk_idx: self.compute_in_each_q_block(batch_start_sc, batch_idx, q_blk_idx, self.Br) with self.tik_instance.if_scope(tr_end_s - 1 != self.Tr - 1): self.compute_in_each_q_block(batch_start_sc, batch_idx, tr_end_s - 1, self.Br) with self.tik_instance.else_scope(): self.compute_in_each_q_block(batch_start_sc, batch_idx, self.Tr - 1, self.last_Br) def collect_inputs(self): """collect all input gm tensors into input_gm_list, the input list should keep order with the para order in Primitive and init """ input_gm_list = [self.Q_gm, self.K_gm, self.V_gm] if self.has_attn_mask: input_gm_list.append(self.att_mask_gm) if self.has_drop_mask: input_gm_list.append(self.drop_mask_gm) if self.has_alibi_mask: input_gm_list.append(self.alibi_mask_gm) return input_gm_list def collect_outputs(self): """collect all output gm tensors into output_gm_list, the output list should keep order with the para order in Primitive and init """ output_gm_list = [self.O_gm, self.l_gm] return output_gm_list def flash_attention(query, key, value, attn_mask, dropout_mask, alibi_mask, output, rowsum, prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse', kernel_name="flash_attention", disable_debug=True): """ algorithm: flash_attention_backward Parameters ---------- query : dict. shape and dtype of input, only support float16 key : dict. shape and dtype of input, only support float16 value: dict. shape and dtype of input, only support float16 attn_mask: dict. shape and dtype of input, only support float16 dropout_mask: dict. shape and dtype of input, only support float16 alibi_mask: dict. shape and dtype of input, only support float16 output: dict. shape and dtype of output, only support float16 rowsum: dict. shape and dtype of output, only support float16 prev_block_num: int. an attribute used to define sparse attention next_block_num: int. an attribute used to define sparse attention tiling_stgy_name: str. an attribute used to choose the tiling strategy kernel_name: str. cce kernel name, default value is real_div disable_debug: bool. whether disable debug Returns ------- tik_instance """ fa = FlashAttentionFwd(q=query, k=key, v=value, attn_mask=attn_mask, dropout_mask=dropout_mask, alibi_mask=alibi_mask, kernel_name=kernel_name, tiling_stgy=TilingStrategy.from_strategy_name(tiling_stgy_name), prev_block_num=prev_block_num, next_block_num=next_block_num, disable_debug=disable_debug) fa.compute_process() return fa.tik_instance pengcheng.mind-mindformer/flash_attention/flash_attention_impl.py000066400000000000000000000152441476650141400260570ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The impl of flash attention""" from __future__ import absolute_import import sys import os import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore.ops import Custom from mindspore.ops import DataType from mindspore.ops import TBERegOp from mindspore.ops.composite.multitype_ops.zeros_like_impl import zeros_like sys.path.append(os.path.dirname(__file__)) from flash_attention_bwd import flash_attention_grad from flash_attention_fwd import flash_attention KERNEL_NAME = "flash_attention" cus_flash_atten_op_info = TBERegOp("FlashAttentionPrimitive") \ .fusion_type("OPAQUE") \ .partial_flag(True) \ .async_flag(False) \ .binfile_name("flash_attention.so") \ .compute_cost(10) \ .kernel_name(KERNEL_NAME) \ .attr("prev_block_num", "required", "int", "all", "65536") \ .attr("next_block_num", "required", "int", "all", "65536") \ .attr("tiling_stgy_name", "required", "str", "all", "sparse") \ .input(0, "query", False, "required", "all") \ .input(1, "key", False, "required", "all") \ .input(2, "value", False, "required", "all") \ .input(3, "attn_mask", False, "optional", "all") \ .input(4, "dropout_mask", False, "optional", "all") \ .input(5, "alibi_mask", False, "optional", "all") \ .output(0, "output", False, "required", "all") \ .output(1, "rowsum", False, "required", "all") \ .dtype_format(DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_Default, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F32_Default,) \ .get_op_info() GRAD_KERNEL_NAME = "flash_attention_grad" cus_flash_atten_grad_op_info = TBERegOp("FlashAttentionGradPrimitive") \ .fusion_type("OPAQUE") \ .partial_flag(True) \ .async_flag(False) \ .binfile_name("flash_attention_grad.so") \ .compute_cost(10) \ .kernel_name(GRAD_KERNEL_NAME) \ .attr("prev_block_num", "required", "int", "all", "65536") \ .attr("next_block_num", "required", "int", "all", "65536") \ .attr("tiling_stgy_name", "required", "str", "all", "sparse") \ .input(0, "query", False, "required", "all") \ .input(1, "key", False, "required", "all") \ .input(2, "value", False, "required", "all") \ .input(3, "output", False, "required", "all") \ .input(4, "do", False, "required", "all") \ .input(5, "rowsum", False, "required", "all") \ .input(7, "attn_mask", False, "optional", "all") \ .input(8, "dropout_mask", False, "optional", "all") \ .input(9, "alibi_mask", False, "optional", "all") \ .output(0, "dq", False, "required", "all") \ .output(1, "dk", False, "required", "all") \ .output(2, "dv", False, "required", "all") \ .dtype_format(DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F16_FracNZ, DataType.F32_Default, DataType.F16_FracNZ, DataType.F16_Default, DataType.F16_FracNZ, DataType.F32_FracNZ, DataType.F32_FracNZ, DataType.F32_FracNZ) \ .get_op_info() def get_flash_attention_grad(prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse'): """get flash attention grad""" def infer_shape(q_shape, k_shape, v_shape, o_shape, do_shape, l_shape, att_mask_shape, dropout_mask_shape, alibi_mask_shape): return q_shape, k_shape, v_shape def infer_dtype(q_dtype, k_dtype, v_dtype, o_dytpe, do_dtype, l_dtype, attn_mask_dtype, dropout_mask_dtype, alibi_mask_type): return mstype.float32, mstype.float32, mstype.float32 fa_grad = Custom(flash_attention_grad, out_shape=infer_shape, out_dtype=infer_dtype, func_type="tbe", reg_info=cus_flash_atten_grad_op_info) fa_grad.add_prim_attr("prev_block_num", prev_block_num) fa_grad.add_prim_attr("next_block_num", next_block_num) fa_grad.add_prim_attr("tiling_stgy_name", tiling_stgy_name) fa_grad.init_prim_io_names( inputs=["query", "key", "value", "output", "do", "rowsum", "attn_mask", "dropout_mask", "alibi_mask"], outputs=["dq", "dk", "dv"] ) def bprop(query, key, value, attn_mask, dropout_mask, alibi_mask, outs, douts): output, rowsum, = outs dout, _, = douts dq, dk, dv = fa_grad(query, key, value, output, dout, rowsum, attn_mask, dropout_mask, alibi_mask) dq = ops.cast(dq, mstype.float16) dk = ops.cast(dk, mstype.float16) dv = ops.cast(dv, mstype.float16) return dq, dk, dv, zeros_like(attn_mask), zeros_like(dropout_mask), zeros_like(alibi_mask) return bprop def get_flash_attention(prev_block_num=65536, next_block_num=65536, tiling_stgy_name='sparse'): """get_flash_attention""" def infer_shape(q_shape, k_shape, v_shape, attn_mask_shape=None, dropout_mask_shape=None, alibi_mask_shape=None): """infer shape""" batch, head_num, seq_len, _ = q_shape l_shape = (batch, head_num, seq_len) return q_shape, l_shape def infer_dtype(q_dtype, k_dtype, v_dtype, attn_mask_dtype=None, dropout_mask_dtype=None, alibi_mask_type=None): """infer type""" return q_dtype, mstype.float32 fa_grad = get_flash_attention_grad(prev_block_num, next_block_num, tiling_stgy_name) fa_forward = Custom(flash_attention, out_shape=infer_shape, out_dtype=infer_dtype, func_type="tbe", bprop=fa_grad, reg_info=cus_flash_atten_op_info) fa_forward.add_prim_attr("prev_block_num", prev_block_num) fa_forward.add_prim_attr("next_block_num", next_block_num) fa_forward.add_prim_attr("tiling_stgy_name", tiling_stgy_name) fa_forward.init_prim_io_names( inputs=["query", "key", "value", "attn_mask", "dropout_mask", "alibi_mask"], outputs=["output", "rowsum"] ) return fa_forward pengcheng.mind-mindformer/flash_attention/nn/000077500000000000000000000000001476650141400217075ustar00rootroot00000000000000pengcheng.mind-mindformer/flash_attention/nn/layer/000077500000000000000000000000001476650141400230235ustar00rootroot00000000000000pengcheng.mind-mindformer/flash_attention/nn/layer/flash_attention.py000066400000000000000000000212711476650141400265620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ A FlashAttention Layer. """ import math import mindspore.common.dtype as mstype from mindspore import ops from mindspore.common.tensor import Tensor from mindspore.nn.cell import Cell __all__ = ['FlashAttention'] HEAD_DIM_MAX_LIMIT = 256 from acctransformer.flash_attention.ops.flash_attention.flash_attention_impl import get_flash_attention class FlashAttention(Cell): """Flash Attention Layer. This function contains the flash attention primitives used in FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. An interface for calling flashattention operation. 2. Two configuration parameters for enabling local block sparse of flashattention. Args: head_dim(int): The hidden size of input. dropout_rate(float): The dropout rate of the attention score. Default 0.0. prev_block_num(int): A integer to define the number of blocks to look ahead for local block sparse attention. Default 65536. next_block_num(int): A integer to define the number of blocks to look behind for local block sparse attention. Default 65536. tiling_stgy_name(str): A str to define tiling strategy of flash attention. dp(int): data parallel. Default 1. mp(int): model parallel. Default 1. have_attention_mask_batch(bool): indicates whether attention_mask contains the batch dimension. Default True alibi(bool): This parameter indicates whether the flashattention supports the Alibi. Default: False Inputs: - **query** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **key** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **value** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): A matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, head_dim] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from acctransformer.flash_attention.nn.layer.flash_attention import FlashAttention >>> from mindspore import Tensor >>> model = FlashAttention(head_dim=128, ... dropout_rate=0.1, ... prev_block_num=7, ... next_block_num=0 ... ) >>> query = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> key = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> value = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.triu(np.ones((1, 128, 128)), k=1), mstype.float16) >>> output = model(query, key, value, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ def __init__(self, head_dim, dropout_rate=0.0, prev_block_num=65536, next_block_num=65536, tiling_stgy_name="sparse", dp=1, mp=1, have_attention_mask_batch=True, alibi=False ): super(FlashAttention, self).__init__() if head_dim > HEAD_DIM_MAX_LIMIT: raise ValueError("head_dim too large, reduce head_dim and try again.") if alibi: raise ValueError("alibi not supported in the current version.") self.flash_attention = get_flash_attention( prev_block_num=prev_block_num, next_block_num=next_block_num, tiling_stgy_name=tiling_stgy_name ) self.flash_attention.add_prim_attr("primitive_target", "Ascend") self.scale_factor = Tensor([1. / math.sqrt(math.sqrt(head_dim))], dtype=mstype.float16) self.scale_mul = ops.Mul().shard(((dp, mp, 1, 1), (1,))) self.dropout_rate = dropout_rate self.enable_dropout = self.dropout_rate > 0.0 self.have_attention_mask_batch = have_attention_mask_batch self.alibi = alibi if self.enable_dropout: self.keep_prob = Tensor(1 - self.dropout_rate, dtype=mstype.float16) self.fill_v2 = ops.FillV2().shard(((dp, mp, 1, 1), ())) self.tensor_one = Tensor(1.0, mstype.float16) self.drop_gen_mask = ops.DropoutGenMask() self.do_dropout = ops.DropoutDoMask().shard(((dp, mp, 1, 1),)) self.depend = ops.Depend() def shard(self, in_strategy=None, out_strategy=None): """Distributed configuration of FlashAttention :param in_strategy: Describe the split strategy of operator input. Default: None. :param out_strategy: Describe the split strategy of operator output, it is only for certain operators, such as MatMul. Default: None. :return: """ if in_strategy is None: # default: dp=1, mp=1, construct inputs only contain q, k, v in_strategy = ( (1, 1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1), ) self.flash_attention.shard(in_strategy) dp = in_strategy[0][0] mp = in_strategy[0][1] self.flash_attention.add_prim_attr("dev_matrix_shape", [dp, mp, 1, 1]) inputs_tensor_map = [ [3, 2, 1, 0], [3, 2, 1, 0], [3, 2, 1, 0], ] if self.have_attention_mask_batch: inputs_tensor_map.append([3, 1, 0]) else: inputs_tensor_map.append([-1, 1, 0]) input_empty_args_num = 2 # dropout_mask if self.enable_dropout: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) if self.alibi: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) self.flash_attention.add_prim_attr("inputs_tensor_map", inputs_tensor_map) self.flash_attention.add_prim_attr("outputs_tensor_map", [ [3, 2, 1, 0], # O [3, 2, 1], # L ]) self.flash_attention.add_prim_attr("as_loss_divisor", 0) self.flash_attention.add_prim_attr("empty_mirror_ops", input_empty_args_num) def construct(self, q, k, v, attn_mask=None, alibi_mask=None): """FlashAttention forward :param q: [bsz, head_num, seq_len, head_dim] :param k: [bsz, head_num, seq_len, head_dim] :param v: [bsz, head_num, seq_len, head_dim] :param attn_mask: [1 or bsz, seq_len, seq_len], if not None :param alibi_mask: [bsz, head_num, 1, seq_len], if not None :return: o [bsz, head_num, seq_len, head_dim] """ q = self.scale_mul(q, self.scale_factor) k = self.scale_mul(k, self.scale_factor) bsz, head_num, seq_len, head_dim = q.shape _, k_head_num, _, _ = k.shape _, v_head_num, _, _ = v.shape if head_num != k_head_num or head_num != v_head_num: raise ValueError( "the head_num of query, key and value must be the same, " "If different head_num are used, users need to change themselves to be same by tile.") if self.enable_dropout: drop_mask_bits = self.drop_gen_mask((bsz, head_num, seq_len, seq_len), self.keep_prob) tensor_shape = Tensor((bsz, head_num, seq_len, seq_len), mstype.int32) ones = self.fill_v2(tensor_shape, self.tensor_one) ones = self.depend(ones, q) drop_mask = self.do_dropout(ones, drop_mask_bits, self.keep_prob) else: drop_mask = None o, _ = self.flash_attention(q, k, v, attn_mask, drop_mask, alibi_mask) return o pengcheng.mind-mindformer/flash_attention/tik_ops_utils.py000066400000000000000000000633401476650141400245440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """the common about tik ops""" from functools import partial from tbe import tik from constants import DTYPE_SIZE from constants import FP16 from constants import FP32 from constants import L0A from constants import L0B from constants import L0C from constants import UB class TikOpsUtils: """Utils function class about tik ops""" def __init__(self, tik_instance): self.tik_instance = tik_instance self.dtype = "float16" self.cont_data_mv_1_bust = partial(self.tik_instance.data_move, sid=0, nburst=1, src_stride=0, dst_stride=0) def MK_TO_K1MK0(self, mk_input_tensor, workspace_tensor=None): """change data shape from (M, K) to (K1, M, K0), K1 = K // K0, the effect is equant to: new_tensor = np.stack(np.hsplit(mk_input_tensor, K1), axis=0) :param mk_input_tensor: input tensor in GM with shape: (M, K) :param workspace_tensor: workspace tensor with shape: (K1, M, K0) tensor will be changed, otherwise the new data will be copied to the workspace tensor, and input tensor will stay unchanged. :return: Tensor with shape (K1,M, K0) """ dtype = mk_input_tensor.dtype m, k = mk_input_tensor.shape K0 = 16 K1 = k // K0 M = self.up_align_to_K0(m) if workspace_tensor is not None: with self.tik_instance.for_range(0, K1) as i: self.tik_instance.data_move( workspace_tensor[i * M * K0:], mk_input_tensor[i * K0:], 0, M, K0 * DTYPE_SIZE[dtype] // 32, (K1 - 1) * K0 * DTYPE_SIZE[dtype] // 32, 0, ) return workspace_tensor.reshape((K1, M, K0)) with self.tik_instance.new_stmt_scope(disable_sync=False): tmp_ub = self.tik_instance.Tensor(dtype, (K1, M, K0), name="tmp_ub", scope=UB) # data_move(m,k) --> (k1,m,K0) with self.tik_instance.for_range(0, K1) as i: self.tik_instance.data_move( tmp_ub[i * M * K0:], mk_input_tensor[i * K0:], 0, M, K0 * DTYPE_SIZE[dtype] // 32, (K1 - 1) * K0 * DTYPE_SIZE[dtype] // 32, 0, ) self.cont_data_mv_1_bust( dst=mk_input_tensor, src=tmp_ub, burst=K1 * M * K0 * DTYPE_SIZE[dtype] // 32) return mk_input_tensor.reshape((K1, M, K0)) def transpose_matrix(self, src_ub, dst_ub, N, nk0=False): """ transpose matrix, default support shape: (16, n) -> (n, 16) if nk0 is true, support shape: (n, 16) -> (16, n) """ K0 = 16 rep_times = N // K0 if nk0: src_list = [src_ub[16 * i] for i in range(16)] dst_list = [dst_ub[N * i] for i in range(16)] else: src_list = [src_ub[N * i] for i in range(16)] dst_list = [dst_ub[16 * i] for i in range(16)] dst_rep_stride = K0 src_rep_stride = 1 if rep_times == 1: dst_rep_stride = 0 src_rep_stride = 0 if nk0: src_rep_stride, dst_rep_stride = dst_rep_stride, src_rep_stride self.tik_instance.vec_trans_scatter( False, False, dst_list, src_list, rep_times, dst_rep_stride, src_rep_stride ) return dst_ub def KN_TO_K1NK0(self, kn_input_tensor, workspace_tensor=None): """change data shape from (K,N) to (K1, N, K0), K1 = K // K0, the effect is equvilent to: new_tensor = np.reshape(kn_input_tensor, newshape=(K1, K0, N)).swapaxes(1, 2) :param kn_input_tensor: input tensor with shape: (K, N) :param workspace_tensor: workspace tensor with shape: (K1, N, K0) tensor will be changed, otherwise the new data will be copied to the workspace tensor, and input tensor will stay unchanged. :return: Tensor with shape: (K1, N, K0) """ dtype = kn_input_tensor.dtype k, n = kn_input_tensor.shape K0 = 16 K1 = k // K0 N = n with self.tik_instance.for_range(0, K1) as index: k1nk0_ub = self.tik_instance.Tensor(dtype, (N, K0), UB, "k1nk0_ub") src_ub = self.tik_instance.Tensor(dtype, (K0, N), UB, "src_ub") burst_len = K0 * N * DTYPE_SIZE[dtype] // 32 self.cont_data_mv_1_bust(dst=src_ub, src=kn_input_tensor[index * K0 * N], burst=burst_len) k1nk0_ub = self.transpose_matrix(src_ub, k1nk0_ub, N) if workspace_tensor is None: self.cont_data_mv_1_bust(dst=kn_input_tensor[index * K0 * N], src=k1nk0_ub, burst=burst_len) else: self.cont_data_mv_1_bust(dst=workspace_tensor[index * K0 * N], src=k1nk0_ub, burst=burst_len) if workspace_tensor is None: return kn_input_tensor.reshape((K1, N, K0)) return workspace_tensor.reshape((K1, N, K0)) def N1MN0_TO_MN(self, N1MN0_input): """change data shape from (N1, M, N0) to (M, N), N0=16, N = N1 * K0, the effect is equant to: N1MN0_input = np.concatenate(list(map(np.squeeze, np.split(N1MN0_input, N1))), axis=1) :param N1MN0_input: input tensor with shape (N, M, N0) in GM or L1. :return: """ dtype = N1MN0_input.dtype N1, M, N0 = N1MN0_input.shape with self.tik_instance.new_stmt_scope(disable_sync=False): tmp_ub = self.tik_instance.Tensor(dtype, (M, N1 * N0), name="tmp_ub", scope=UB) # data_move (n1,m,n0) --> (m,n) with self.tik_instance.for_range(0, N1) as i: self.tik_instance.data_move( tmp_ub[i * N0:], N1MN0_input[i * M * N0:], 0, M, N0 * DTYPE_SIZE[dtype] // 32, 0, (N1 - 1) * N0 * DTYPE_SIZE[dtype] // 32, ) # data_move out self.cont_data_mv_1_bust(dst=N1MN0_input, src=tmp_ub, burst=M * N1 * N0 * DTYPE_SIZE[ dtype] // 32) return N1MN0_input.reshape((M, N1 * N0)) def broadcast_vec_from_M_to_MN0(self, vec_ub): """broadcast a vec from (M,) to (M, 16)""" M = vec_ub.shape[0] dst_ub = self.tik_instance.Tensor(FP16, (M, 16), name="dst_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): broadcast_l0c = self.tik_instance.Tensor(FP16, (16, M), name="broadcast_l0c", scope=L0C) self.tik_instance.broadcast_ub_to_l0c(broadcast_l0c, vec_ub, 1, M // 16, 1, 1) broadcast_ub = self.tik_instance.Tensor(FP16, (16, M), name="broadcast_ub", scope=UB) self.cont_data_mv_1_bust(dst=broadcast_ub, src=broadcast_l0c, burst=M * 16 * 2 // 512) # (M // N0, 16, N0) -> (M, 16), N0 = 16 self.tik_instance.vec_trans(dst_ub, broadcast_ub, M // 16, 1, 1) return dst_ub def broadcast(self, vec_ub, shape): """ broadcast a vector to a matrix :param vec_ub: a tensor in UB with shape of (M,), and dtype is float16 :param shape: the target shape, a tuple with value (M, N), M and N are integer multiples of 16 :return: a tensor in UB with shape of (M, N) """ M, N = shape dst_ub = self.tik_instance.Tensor(FP16, shape, name="dst_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): M_16_ub = self.broadcast_vec_from_M_to_MN0(vec_ub) # (M, 16) -> (M, 32) -> (M, 64) -> ... -> (M, N) self.tik_instance.data_move(dst_ub, M_16_ub, 0, M, 1, 0, N // 16 - 1) times = self.tik_instance.Scalar("int32", name="times", init_value=1) with self.tik_instance.for_range(begint=0, endt=N) as idx: offset = times * 16 with self.tik_instance.if_scope(offset * 2 <= N): burst = offset // 16 src_stride = N // 16 - burst dst_stride = N // 16 - burst self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, M, burst, src_stride, dst_stride) with self.tik_instance.else_scope(): burst = (N - offset) // 16 src_stride = N // 16 - burst dst_stride = N // 16 - burst with self.tik_instance.if_scope(burst > 0): self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, M, burst, src_stride, dst_stride) self.tik_instance.tik_break() times.set_as(times * 2) return dst_ub def broadcast_row(self, vec_ub, shape): """broadcast row""" M, N = shape dst_ub = self.tik_instance.Tensor(FP16, shape, name="dst_ub", scope=UB) self.tik_instance.data_move(dst_ub, vec_ub, 0, 1, N // 16, 0, 0) times = self.tik_instance.Scalar("int32", name="times", init_value=1) # (1, N) -> (2, M) -> (4, N) -> ... -> (M, N) with self.tik_instance.for_range(begint=0, endt=M): with self.tik_instance.if_scope(times * 2 <= M): burst = times * N // 16 offset = times * N self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, 1, burst, 0, 0) with self.tik_instance.else_scope(): burst = (M - times) * N // 16 offset = times * N with self.tik_instance.if_scope(burst > 0): self.tik_instance.data_move(dst_ub[offset], dst_ub, 0, 1, burst, 0, 0) self.tik_instance.tik_break() times.set_as(times * 2) return dst_ub def get_K0(self, dtype=None): """get K0""" if dtype is None: dtype = self.dtype return 32 // DTYPE_SIZE[dtype] def up_align_to_K0(self, n, dtype=None): """byte alignment by dtype""" if dtype is None: dtype = self.dtype K0 = 32 // DTYPE_SIZE[dtype] return (n + K0 - 1) // K0 * K0 def calc_vec_rec(self, vec_ub, vec_len): """cal the reciprocal of a vector""" dtype = vec_ub.dtype vec_len_aligned = self.up_align_to_K0(vec_len) vec_rec_ub = self.tik_instance.Tensor(dtype, (vec_len_aligned,), scope=UB, name="li_new_rec_ub") mask_len = 256 // DTYPE_SIZE[dtype] block_len = 32 // DTYPE_SIZE[dtype] work_size = 8 // DTYPE_SIZE[dtype] with self.tik_instance.new_stmt_scope(disable_sync=False): repeat_times = vec_len // mask_len if repeat_times > 0: dst_rep_stride = 8 src_rep_stride = 8 src_extent_size = (repeat_times - 1) * src_rep_stride * block_len + mask_len wk_size_unit = ((src_extent_size + block_len - 1) // block_len) * block_len wk_size = work_size * wk_size_unit # 定义work_tensor work_tensor_ub = self.tik_instance.Tensor( "float32", (wk_size,), name="work_tensor_ub", scope=UB ) # 如果work_tensor有索引，需要写成 work_tensor[index:] self.tik_instance.vec_rec_high_preci( mask_len, vec_rec_ub[0:], vec_ub[0:], work_tensor_ub[0:], repeat_times, dst_rep_stride, src_rep_stride, ) mask_len = vec_len - repeat_times * mask_len if mask_len > 0: wk_size = work_size * ((mask_len + block_len - 1) // block_len) * block_len work_tensor_ub2 = self.tik_instance.Tensor( "float32", (wk_size,), name="work_tensor_ub2", scope=UB ) self.tik_instance.vec_rec_high_preci( mask_len, vec_rec_ub[repeat_times * mask_len:], vec_ub[repeat_times * mask_len:], work_tensor_ub2[0:], 1, 0, 0, ) return vec_rec_ub def row_sum_cube_impl(self, matrix_l1_K1MK0_ed, right_all_one_matrix_l1, rowsum_ub, m, k, precision_type): """用cube实现矩阵行和：右乘一个shape=(n,1)全一矩阵 :param matrix_l1_K1MK0_ed: input tensor with shape (K1, M, K0) :param right_all_one_matrix_l1: input tensor with shape (K, 16) :param rowsum_ub: output tensor stores the row sum of input tensor :param m: actual tensor height :param k: actual tensor width :return: row sum of the output tensor """ K1, M, K0 = matrix_l1_K1MK0_ed.shape # 调用matmul实现rowsum，结果shape=(m, 16)，取每行的第一个数 with self.tik_instance.new_stmt_scope(disable_sync=False): row_sum_ub_N1MN0 = self.matmul_compute(matrix_l1_K1MK0_ed, right_all_one_matrix_l1, m, k, 16, N1MN0_to_MN=False, precision_type=precision_type) row_sum_ub_MN_ed = row_sum_ub_N1MN0.reshape((M, 16)) if precision_type == FP32: for idx in range(0, m): cur_row_sum = self.tik_instance.Scalar(FP32, init_value=row_sum_ub_MN_ed[idx, 0]) rowsum_ub[idx].set_as(cur_row_sum) else: # row_sum_ub_MN_ed 先转置，然后取一行, 替换原来按行操作: lij_ub[i].set_as(row_sum_ub_MN_ed[i, 0]) row_sum_ub_trans = self.tik_instance.Tensor(FP16, (16, M), name="row_sum_ub_trans", scope=UB) row_sum_ub_trans = self.transpose_matrix(row_sum_ub_MN_ed, row_sum_ub_trans, M, True) self.cont_data_mv_1_bust(dst=rowsum_ub, src=row_sum_ub_trans, burst=M // 16) return rowsum_ub def matmul_compute(self, A_l1, B_l1, m, k, n, N1MN0_to_MN=True, precision_type=FP16, workspace=None): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 with shape of (K1, M, K0) :param B_l1: input tensor in L1 with shape of (K1, N, K0) :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 if workspace is None: C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) else: C_ub = workspace with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor( FP32, (N // 16, M, 16), scope=L0C, name="C_l0c" ) # n1mn0 (n0=16) self.tik_instance.matmul(C_l0c, A_l1, B_l1, m, k, n) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def mmad_compute_transpose(self, A_l1, B_l1, m, k, n, a_transpose=False, b_transpose=False, N1MN0_to_MN=False, precision_type=FP16): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 zN format :param B_l1: input tensor in L1 zN format :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param a_transpose: A Whether to transpose :param b_transpose: B Whether to transpose :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) K = self.up_align_to_K0(k) # todo: if m == 1: # m=1时, mmad 退化成gemv, l0A 排布: m=1 Nd, m=2 zZ; # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor( FP32, (N // 16, M, 16), scope=L0C, name="C_l0c" ) # n1mn0 (n0=16) A_l0a = self.tik_instance.Tensor(FP16, (M, K), scope=L0A, name="A_l0a") # Nz -> zZ if a_transpose: self.tik_instance.load2dv1(dst=A_l0a, src=A_l1, index=0, repeat_times=M * K * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=True) else: for i in range(M // 16): # todo: 使用少的循环次数 self.tik_instance.load2dv1(dst=A_l0a[i * K * 16], src=A_l1[i * 256], index=0, repeat_times=16 * K * DTYPE_SIZE[FP16] // 512, src_stride=M//16, sid=0, if_transpose=False) B_l0b = self.tik_instance.Tensor(FP16, (K, N), scope=L0B, name="B_l0b") if b_transpose: self.tik_instance.load2dv1(dst=B_l0b, src=B_l1, index=0, repeat_times=N * K * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) else: for i in range(K // 16): self.tik_instance.load2dv1(dst=B_l0b[i * N * 16], src=B_l1[i * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K//16, sid=0, if_transpose=True) self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k, n, is_bias=0) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def mmad_compute(self, A_l1, B_l1, m, k, n, N1MN0_to_MN=False, precision_type=FP16): """calculate matrix multiplication A_l1 * B_l1, and move the result to C_ub, then rearrange C_ub :param A_l1: input tensor in L1 zN format :param B_l1: input tensor in L1 zN format :param m: the actual number of rows of A_l1 :param k: the actual number of cols of A_l1 :param n: the actual number of cols of B_l1 :param N1MN0_to_MN: Whether reorder the result tensor. :return: C_ub with tensor with shape of (M, N) if N1MN0_to_MN else (N1, M, N0) """ M = self.up_align_to_K0(m) N = self.up_align_to_K0(n) K = self.up_align_to_K0(k) k_tail = 16 if k % 16 == 0 else k % 16 # 根据精度类型，决定matmul返回的ub精度类型，如果fp16, tensor_move做随路精度转换 C_ub = self.tik_instance.Tensor(precision_type, (N // 16, M, 16), name="C_ub", scope=UB) with self.tik_instance.new_stmt_scope(disable_sync=False): # matmul C_l0c = self.tik_instance.Tensor(FP32, (N // 16, M, 16), scope=L0C, name="C_l0c") # n1mn0 (n0=16) if K > 16: with self.tik_instance.for_range(0, K // 16 // 2) as i: with self.tik_instance.for_range(0, 2, thread_num=2) as t_idx: A_l0a = self.tik_instance.Tensor(FP16, (M, 16), scope=L0A, name="A_l0a") B_l0b = self.tik_instance.Tensor(FP16, (16, N), scope=L0B, name="B_l0b") self.tik_instance.load2dv1(dst=A_l0a, src=A_l1[(i * 2 + t_idx) * M * 16], index=0, repeat_times=16 * M * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) self.tik_instance.load2dv1(dst=B_l0b, src=B_l1[(i * 2 + t_idx) * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K//16, sid=0, if_transpose=True) with self.tik_instance.if_scope(tik.all(i == 0, t_idx == 0)): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, 16, n, is_bias=0) with self.tik_instance.elif_scope(tik.all(K // 16 % 2 == 0, i == K // 16 // 2 - 1, t_idx == 1)): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k_tail, n, is_bias=1) with self.tik_instance.else_scope(): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, 16, n, is_bias=1) if K // 16 % 2 > 0: # 尾快处理 A_l0a = self.tik_instance.Tensor(FP16, (M, 16), scope=L0A, name="A_l0a") B_l0b = self.tik_instance.Tensor(FP16, (16, N), scope=L0B, name="B_l0b") self.tik_instance.load2dv1(dst=A_l0a, src=A_l1[(K // 16 - 1) * M * 16], index=0, repeat_times=16 * M * DTYPE_SIZE[FP16] // 512, src_stride=1, sid=0, if_transpose=False) self.tik_instance.load2dv1(dst=B_l0b, src=B_l1[(K // 16 - 1) * 256], index=0, repeat_times=16 * N * DTYPE_SIZE[FP16] // 512, src_stride=K // 16, sid=0, if_transpose=True) with self.tik_instance.if_scope(K > 16): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k_tail, n, is_bias=1) with self.tik_instance.else_scope(): self.tik_instance.mmad(C_l0c, A_l0a, B_l0b, m, k, n, is_bias=0) # L0C -> ub, fp32 -> fp16 (tensor_mov可做随路转换) self.tik_instance.tensor_mov(C_ub, C_l0c, "m", 1, M * N * DTYPE_SIZE[FP32] // 1024, 0, 0) if N1MN0_to_MN: return self.N1MN0_TO_MN(C_ub) else: return C_ub def move_vector_from_gm_to_ub(self, dst_tensor, src_tensor, gm_offset, vec_len): """load the vector from gm to ub :param dst_tensor: :param src_tensor: :param gm_offset: :return: """ dtype = src_tensor.dtype a_burst_num = 32 // DTYPE_SIZE[dtype] full_tik_blk_num, tail_num = divmod(vec_len, a_burst_num) with self.tik_instance.if_scope(full_tik_blk_num > 0): self.cont_data_mv_1_bust(dst=dst_tensor, src=src_tensor[gm_offset], burst=full_tik_blk_num) # 地址回退处理尾部数据 with self.tik_instance.if_scope(tail_num > 0): offset = vec_len - a_burst_num last_blk_ub = self.tik_instance.Tensor(dtype, (a_burst_num,), name="last_blk_ub", scope=UB) self.cont_data_mv_1_bust(dst=last_blk_ub, src=src_tensor[gm_offset + offset], burst=1) with self.tik_instance.for_range(0, a_burst_num) as idx: # offset非32bytes对齐，无法用datamove dst_tensor[offset + idx].set_as(last_blk_ub[idx]) def move_vector_from_ub_to_gm(self, dst_tensor, src_tensor, gm_offset, block_h): """write the vector back to gm :param dst_tensor: :param src_tensor: :param gm_offset: :param block_h: :return: """ dtype = src_tensor.dtype a_burst_num = 32 // DTYPE_SIZE[dtype] full_tik_blk_num = block_h // a_burst_num with self.tik_instance.if_scope(full_tik_blk_num > 0): self.cont_data_mv_1_bust(dst=dst_tensor[gm_offset], src=src_tensor, burst=full_tik_blk_num) tail_num = block_h % a_burst_num with self.tik_instance.if_scope(tail_num > 0): offset = block_h - a_burst_num tmp_ub = self.tik_instance.Tensor(dtype, (a_burst_num,), name="tmp_ub", scope=UB) with self.tik_instance.for_range(0, a_burst_num) as idx: tmp_ub[idx].set_as(src_tensor[offset + idx]) self.cont_data_mv_1_bust(dst=dst_tensor[gm_offset + offset], src=tmp_ub, burst=1) def scale_compute_vector(self, Sij_ub, dim): """scale compute vector""" scale_value = dim ** -0.5 scale = self.tik_instance.Scalar(dtype=FP16) scale.set_as(scale_value) self.tik_instance.h_mul(Sij_ub, Sij_ub, scale) return Sij_ub pengcheng.mind-mindformer/flash_attention/tiling_strategy/000077500000000000000000000000001476650141400245045ustar00rootroot00000000000000pengcheng.mind-mindformer/flash_attention/tiling_strategy/__init__.py000066400000000000000000000001101476650141400266050ustar00rootroot00000000000000# Copyright (C) 2021.Huawei Technologies Co., Ltd. All rights reserved. pengcheng.mind-mindformer/flash_attention/tiling_strategy/sparse_tiling.py000066400000000000000000000027121476650141400277230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class SparseTiling(TilingStrategy): """A tiling strategy implementation for sparse shape""" @classmethod def strategy_name(cls): return "sparse" def tiling(self) -> TilingPara: self.Br = min(128, self.Nq) self.Bc = min(128, self.N) self.Tr = self.Nq // self.Br self.Tc = self.N // self.Bc if self.Nq % self.Br != 0: self.last_Br = self.Nq - self.Tr * self.Br self.Tr += 1 else: self.last_Br = self.Br if self.N % self.Bc != 0: self.last_Bc = self.N - self.Tc * self.Bc self.Tc += 1 else: self.last_Bc = self.Bc return self.gen_tiling_para()pengcheng.mind-mindformer/flash_attention/tiling_strategy/strategy.py000066400000000000000000000041411476650141400267200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The base class of tiling strategy""" from abc import ABCMeta from abc import abstractmethod from collections import namedtuple TilingPara = namedtuple("TilingPara", "Br last_Br Bc last_Bc Tr Tc") class TilingStrategy(metaclass=ABCMeta): """Tiling strategy interface. All implementations should be defined in this module, otherwise, the UT will fail. """ _strategies = {} def __init__(self, Nq, N, head_dim) -> None: super().__init__() self.Nq = Nq self.N = N self.Br = None self.last_Br = None self.Bc = None self.last_Bc = None self.Tr = None self.Tc = None self.d = head_dim def __init_subclass__(cls, **kwargs): TilingStrategy._strategies[cls.strategy_name()] = cls @classmethod @abstractmethod def strategy_name(cls): """strategy name""" raise NotImplementedError @classmethod def from_strategy_name(cls, stgy_name: str): """from strategy name""" stgy_clz = TilingStrategy._strategies.get(stgy_name) if stgy_clz is None: raise Exception(f"Strategy:{stgy_name} not supported") return stgy_clz @abstractmethod def tiling(self) -> TilingPara: """tiling""" raise NotImplementedError def gen_tiling_para(self) -> TilingPara: """gen tiling para""" return TilingPara(self.Br, self.last_Br, self.Bc, self.last_Bc, self.Tr, self.Tc) pengcheng.mind-mindformer/flash_attention/tiling_strategy/wukong_tiling.py000066400000000000000000000045751476650141400277510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """wukong tiling""" from tiling_strategy.strategy import TilingPara from tiling_strategy.strategy import TilingStrategy class WukongTiling(TilingStrategy): """A tiling strategy implementation for wukonghuahua model shape""" @classmethod def strategy_name(cls): return "wukong" def tiling(self) -> TilingPara: """ 反向的空间分布待详细分析 N = (4096, 1024, 256, 64) 或 77 Nq = (4096, 1024, 256, 64) d = dv = (40, 80, 160， 160) """ if self.N <= 77: # [77, 64] # cross-attention or self-attention of (64, 64, 160) self.Bc = self.N self.Tc = self.N // self.Bc if self.d <= 80: # [40, 80] # 内存瓶颈为在ub中对P*V结果[Br, dv]进行cast # ub: 512 * 80 * 6 // 1024 = 240KB self.Br = min(self.Nq, 64) self.Tr = self.Nq // self.Br else: # ub: dv = 160， 256 * 160 * 6 // 1024 = 240KB self.Br = min(self.Nq, 64) self.Tr = self.Nq // self.Br else: # self-attention if self.N == 256: self.Bc = 64 self.Tc = 1 # 内存瓶颈为在ub中对Q*K的结果[Br, Bc]进行cast # ub: 128 * 256 * 6 // 1024 = 192KB self.Br = 64 self.Tr = self.Nq // self.Br else: self.Bc = 64 self.Tc = self.N // self.Bc # ub: 64 * 512 * 6 // 1024 = 192KB self.Br = 64 self.Tr = self.Nq // self.Br self.last_Br = self.Br self.last_Bc = self.Bc return self.gen_tiling_para() pengcheng.mind-mindformer/mindformers/000077500000000000000000000000001476650141400204375ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/__init__.py000066400000000000000000000025731476650141400225570ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """mindformers init""" from mindformers import core, auto_class, dataset, \ models, modules, wrapper, tools from mindformers.pipeline import * from mindformers.trainer import * from mindformers.core import * from mindformers.dataset import * from mindformers.models import * from mindformers.modules import * from mindformers.wrapper import * from mindformers.tools import * from mindformers.auto_class import * from mindformers.generation import * #from mindformers.pet import * from .mindformer_book import MindFormerBook __all__ = [] __all__.extend(dataset.__all__) __all__.extend(models.__all__) __all__.extend(core.__all__) __all__.extend(tools.__all__) __all__.extend(auto_class.__all__) __all__.extend(generation.__all__) pengcheng.mind-mindformer/mindformers/__pycache__/000077500000000000000000000000001476650141400226475ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/__pycache__/__init__.cpython-39.pyc000066400000000000000000000014751476650141400270460ustar00rootroot00000000000000a >cf{@sdZddlmZmZmZmZmZmZmZddl Tddl Tddl Tddl Tddl TddlTddlTddlTddlTddlTddlmZgZeejeejeejeejeejeejdS)zmindformers init)core auto_classdatasetmodelsmoduleswrappertools)*)MindFormerBookN)__doc__ mindformersrrrrrrrZmindformers.pipelinemindformers.trainermindformers.coreZmindformers.datasetZmindformers.modelsZmindformers.modulesZmindformers.wrappermindformers.toolsZmindformers.auto_classZmindformers.generationmindformer_bookr __all__extendZ generationrr?/home/ma-user/work/mindformers-baichuan/mindformers/__init__.pys&$      pengcheng.mind-mindformer/mindformers/__pycache__/auto_class.cpython-39.pyc000066400000000000000000000557441476650141400274540ustar00rootroot00000000000000a >cf@sdZddlZddlZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZgd ZGd d d ZGdddZGdddZGdddZdS)u AutoConfig、AutoModel N)MindFormerBook print_dict)build_processor) BaseConfig) build_model)build_model_config)logger)MindFormerConfig) AutoConfig AutoModel AutoProcessor AutoTokenizerc@sXeZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d ZdS)r a AutoConfig class, helps instantiates a config by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoConfig >>> >>> # 1) instantiates a config by yaml model name >>> config_a = AutoConfig.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a config by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> config_b = AutoConfig.from_pretrained(config_path) rrcCs tddS)NzkAutoConfig is designed to be instantiated using the `AutoConfig.from_pretrained(yaml_name_or_path)` method.EnvironmentErrorselfrA/home/ma-user/work/mindformers-baichuan/mindformers/auto_class.py__init__8szAutoConfig.__init__c Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS %Check whether it is a valid yaml name mindspore/_TF'' is not supported by '', please select from startswithsplit _model_name _model_type _support_listkeys isinstancedictlen ValueErrorclsyaml_name_or_pathlocal_model_typelocal_model_listlocal_model_nameslocal_model_namerrrinvalid_yaml_name>s*   zAutoConfig.invalid_yaml_namec Ks|dd}|dur|}t|ts6tdt|dtj|rr|dsZt |dt |}t d|n| |rt |d|jdn|}|d r|d |j}tjt|d |j}ntjt|d |j}tj|s tj|d d tj||d}dd}tj|s||} tj| rrtj| rrt| |t d|ntd| t |}t|jj} tt | |jj!j| S)a From pretrain method, which instantiates a config by yaml model name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. pretrained_model_name_or_pathN+yaml_name_or_path should be a str, but got ..yamlz) should be a .yaml file for model config.z.the content in %s is used for config building.g is not a supported model type or a valid path to model config. supported model could be selected from rrrTexist_okcSs0d}tD]}||vr||}q,q|SNrget_trainer_support_task_listvaluesget model_namedefault_yaml_file model_dictrrrget_default_yaml_files  z9AutoConfig.from_pretrained..get_default_yaml_file"default yaml config in %s is used.0default yaml file path must be correct, but get )"popr%str TypeErrortypeospathexistsendswithr(r r infor0r#rr r!joinr#get_xihe_checkpoint_download_folderr"&get_default_checkpoint_download_foldermakedirsrealpathshutilcopyFileNotFoundErrorrmodel model_configset_model_config_to_nameidarch) r*r+kwargsr1 config_args yaml_namecheckpoint_path yaml_filerBr@configrrrfrom_pretrainedYsR            zAutoConfig.from_pretrainedcCstd|jt|jdSzshow support list methodzsupport list of %s is:Nr rM__name__rr#r*rrrshow_support_listszAutoConfig.show_support_listcCs|jSzget support list methodr#rerrrget_support_listszAutoConfig.get_support_listN)rd __module__ __qualname____doc__rget_config_support_listr#r"r!r classmethodr0rarfrirrrrr #s  M r c@s|eZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d Ze ddZe ddZe ddZdS)r a AutoModel class helps instantiates a model by yaml model name, path or config. If using a model name, the config yaml and checkpoint file will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoModel >>> >>> # 1) input model name, load model and weights >>> model_a = AutoModel.from_pretrained('clip_vit_b_32') >>> # 2) input model directory, load model and weights >>> from mindformers.mindformer_book import MindFormerBook >>> checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') >>> model_b = AutoModel.from_pretrained(checkpoint_dir) >>> # 3) input yaml path, load model without weights >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> model_c = AutoModel.from_config(config_path) >>> # 4) input config, load model without weights >>> config = AutoConfig.from_pretrained('clip_vit_b_32') >>> model_d = AutoModel.from_config(config) rrcCs tddS)NzAutoModel is designed to be instantiated using the `AutoModel.from_pretrained(pretrained_model_name_or_dir)` method or `AutoModel.from_config(config)` method.rrrrrrszAutoModel.__init__c Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS) z&Check whether it is a valid model namerrrTFrrrr)r*pretrained_model_name_or_dirr,r-r.r/rrrinvalid_model_names2    zAutoModel.invalid_model_namecKs|durtd|dd}t|tr<||}||}n(tj|r\| dr\t |}ntd|srd|j j _ t|j }td|S)a' From config method, which instantiates a Model by config. Args: config (str, BaseConfig): A model config inherited from BaseConfig, or a path to .yaml file for model config. Returns: A model, which inherited from BaseModel. Nz8a model cannot be built from config with config is None.download_checkpointTr4zUconfig should be inherited from BaseConfig, or a path to .yaml file for model config.model built successfully!)r(rEr%r_inverse_parse_config _wrap_configrIrJrKrLr rVrWcheckpoint_name_or_pathrr rM)r*r`r[rqZinversed_configr\rVrrr from_configs         zAutoModel.from_configcCsRt|ts|S|jj}|d|i|D] \}}||}|||iq,|S)a Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. rH)r%r __class__rdupdateitemsrs)r*r` class_namekeyvalnew_valrrrrss  zAutoModel._inverse_parse_configcCsH|dd}|dur(tt|d}t|d}t||d}t|dS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A model config, which has the same content as a yaml file. r?N)rH)rWrZ)rV)rErget_model_config_to_namer=rYr)r*r`r?rZrVrrrrt(s   zAutoModel._wrap_configcKsj|dd}|dd}|dur$|}t|ts@tdt|tj|}tj|}|rp|st |dn | |rt |d|j d|r2d d t |D}d d t |D}|r|st d |tj|||j} tj|||j} td | | t| } | jjd| it| j} n"|} |drr|d|j} tjt| d|j}ntjt|d|j}tj|stj|ddtj|| d} dd}tj| s || }tj|rtj|rt || td| nt d|t| } | jjd|i|sJd| jj_!t| j} | j"|_"td| S)ai From pretrain method, which instantiates a Model by pretrained model name or path. Args: pretrained_model_name_or_dir (str): A supported model name or a directory to model checkpoint (including .yaml file for config and .ckpt file for weights), the supported model name could be selected from AutoModel.show_support_list(). If pretrained_model_name_or_dir is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Returns: A model, which inherited from BaseModel. r1NrqTz6pretrained_model_name_or_dir should be a str, but got z is not a directory.r5r3cSsg|]}|dr|qSr4rL.0filerrr bs z-AutoModel.from_pretrained..cSsg|]}|dr|qS)z.ckptrrrrrrds zIthere is no yaml file for model config or ckpt file for model weights in z;config in %s and weights in %s are used for model building.rurrrr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrBs  z8AutoModel.from_pretrained..get_default_yaml_filerCrDrr)#rEr%rFrGrHrIrJrKisdirr(rpr#listdirrUrNr"r rMr rVrWrxrrr r!rrOrPrQrRrSrTru default_checkpoint_download_path)r*ror[r1rqis_existis_dir yaml_list ckpt_listr_ ckpt_filer\rVZpretrained_checkpoint_namer^rBr@rrrra;s~            zAutoModel.from_pretrainedcCstd|jt|jdSrbrcrerrrrfszAutoModel.show_support_listcCs|jSrgrhrerrrriszAutoModel.get_support_listN)rdrjrkrlrget_model_support_listr#r"r!rrnrprvrsrtrarfrirrrrr s&     i r c@sXeZdZdZeZdZdZddZ e ddZ e dd Z e d d Z e d d ZdS)r a AutoProcessor helps instantiates a processor by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoProcessor >>> >>> # 1) instantiates a processor by yaml model name >>> pro_a = AutoProcessor.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a processor by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> pro_b = AutoProcessor.from_pretrained(config_path) rrcCs tddS)NzqAutoProcessor is designed to be instantiated using the `AutoProcessor.from_pretrained(yaml_name_or_path)` method.rrrrrrszAutoProcessor.__init__c Cs|dr|d|j}|d|j}|j|}t|tsJ||vrJdS|}t|d|jksz|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dS)rrrrFrrr) rr r!r"r#r%r&r$r'r(r)rrrr0s&   zAutoProcessor.invalid_yaml_namecKsV|dd}|dur|}t|ts4tdt|tj|}|drf| d|j  d|j n| d|j }|s||j vrt|d|jd|j d |rtd |tj|rd d t|D}tj|||j }t|}nt|}n"|}||sn|drN| d|j }tjt| d|j } ntjt| d|j } nt|d |jd|j d tj| stj| ddtj| |d} dd} tj| s| |} tj| rtj| rt| | td| ntd| t| }|} tj| s:d} t|j | d}td|S)a From pretrain method, which instantiated a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. r1Nr2rrrz, does not exist, and it is not supported by z. please select from r3z1config in %s is used for auto processor building.cSsg|]}|dr|qSrrrrrrr z1AutoProcessor.from_pretrained..z+ does not exist, or it is not supported by Tr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrB(s  z.get_default_yaml_filerCrD)lib_pathzprocessor built successfully!)!rEr%rFrGrHrIrJrKrr r!r"r#r$r(rdr rMrrrNr r0rrOrPrQrRrSrTrUr processor)r*r+r[r1rr?rr]r\r^r_rBr@rrrrrrasv              zAutoProcessor.from_pretrainedcCstd|jt|jdSrbrcrerrrrf@szAutoProcessor.show_support_listcCs|jSrgrhrerrrriFszAutoProcessor.get_support_listN)rdrjrkrlrget_processor_support_listr#r"r!rrnr0rarfrirrrrr s  Z r c@sheZdZdZeZdZdZe ddZ e ddZ e dd Z e d d Z e d d Ze ddZdS)ra Load the tokenizer according to the `yaml_name_or_path`. It supports the following situations 1. `yaml_name_or_path` is the model name. 2. `yaml_name_or_path` is the path to the downloaded files. Examples: >>> from mindformers.auto_class import AutoTokenizer >>> >>> # 1) instantiates a tokenizer by the model name >>> tokenizer_a = AutoTokenizer.from_pretrained("clip_vit_b_32") >>> # 2) instantiates a tokenizer by the path to the downloaded files. >>> from mindformers.models.clip.clip_tokenizer import CLIPTokenizer >>> clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") >>> clip_tokenizer.save_pretrained(path_saved) >>> restore_tokenizer = AutoTokenizer.from_pretrained(path_saved) rrc Cs|dr|d|j}|d|j|jvr8dS|d|j}|j|}t|tsh||vrhdSt|tsvdS|}t|d|jks|d|j|vrt d|d|d||d|j}|||vrt d|d|d|d||dSrrr)rrrr0as.   zAutoTokenizer.invalid_yaml_namec Cstj|}tj|}tj|}|s~|s:t|d|sLt|dddt|D}|shdStj|||j}n|}t d|t |}d}|rd|vrd|dvrd |ddvr|dd d d}t d |||S) z Try to find the yaml from the given path Args: yaml_name_or_path (str): the directory of the config yaml Returns: The class name of the tokenizer in the config yaml. z3 does not exist, Please pass a valid the directory.z3 is not a directory. You should pass the directory.cSsg|]}|dr|qSrrrrrrrrz;AutoTokenizer._get_class_name_from_yaml..Nz;Config in the yaml file %s are used for tokenizer building.r tokenizerrHz&Load the tokenizer name %s from the %s) rIrJrKrisfiler(rrNr"r rMr rE) r*r+rris_filerr_r`rzrrr_get_class_name_from_yamls,    z'AutoTokenizer._get_class_name_from_yamlcCstj|d}tj|s*td|dt|d}t|}Wdn1sT0Y|dd}|st d|d| |S) z try to get the tokenizer type from tokenizer_config.json Args: yaml_name_or_path (str): the directory of tokenizer_config.json Returns: The class name of the tokenizer in tokenizer_config.json ztokenizer_config.jsonz:The file `tokenizer_config.json` should exits in the path z, but not found.rNtokenizer_classz1There should be the key word`tokenizer_class` in z', but not found. The optional keys are ) rIrJrNrKrUopenjsonloadrEr(r$)r*r+tokenizer_config_pathfpZ config_kwargsrzrrr*_get_class_name_from_tokenizer_config_files    (  z8AutoTokenizer._get_class_name_from_tokenizer_config_filec Ks|dd}|dur|}ddlm}t|ts@tdt|tj |rh| |}|sd| |}n | |sj|}| dr|d|j}tjt|d|j}ntjt|d|j}tj|stj|d d tj||d }d d } tj|s^| |} tj| rPtj| rPt| |td|ntd| | |}ntd|d|jd|d|jd|d} | |} td| j j| S)a From pretrain method, which instantiates a tokenizer by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/clip_vit_b_32" or "clip_vit_b_32". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A tokenizer which inherited from PretrainedTokenizer. r1Nr)MindFormerRegisterr2rrrTr6r4cSs0d}tD]}||vr||}q,q|Sr8r:r>rrrrBs  z.get_default_yaml_filerCrDzTokenizer type `z` does not exist. Use `zJ.show_support_list()` to check the supported tokenizer. Or make sure the `z` is a directory.r) module_typerzz %s Tokenizer built successfully!)!rEr9rr%rFrGrHrIrJrrrr0rr r!rNrrOr"rPrKrQrRrSrTr rMrUrdget_clsrarw) r*r+r[r1rrzr]r^r_rBr@ dynamic_classZinstanced_classrrrrasT               zAutoTokenizer.from_pretrainedcCstd|jt|jdSrbrcrerrrrfszAutoTokenizer.show_support_listcCs|jSrgrhrerrrri szAutoTokenizer.get_support_listN)rdrjrkrlrget_tokenizer_support_listr#r"r!rnr0rrrarfrirrrrrLs   #  K r)rlrIrrSmindformer_bookrrZmodels.build_processorrZmodels.base_configrZmodels.build_modelrZmodels.build_configrtoolsr Ztools.register.configr __all__r r r rrrrrs&      pengcheng.mind-mindformer/mindformers/__pycache__/mindformer_book.cpython-39.pyc000066400000000000000000001034551476650141400304640ustar00rootroot00000000000000a >cf0@sNdZddlZddlZddlmZddlmZddZddZGd d d Z dS) z| MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository. N) OrderedDict)loggercCsbt|trL|D]6\}}t|tr:td|t|qtd||qntt|ddS)z Print dict function for show support method of MindFormerBook or other BaseClasses. Args: input_dict (dict): the dict to be printed. z%s:z %s : %sz is unspoorted by print_dictN) isinstancedictitemsrinfo print_dict TypeErrortype) input_dictkeyvalrF/home/ma-user/work/mindformers-baichuan/mindformers/mindformer_book.pyrs    rcCs<t|ttfr&td|tdntt|ddS)z Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. z %sz%-------------------------------------z% is unsupported by print_path_or_listN)rstrlistrrr r )input_path_or_listrrrprint_path_or_list-s  rc5@seZdZdZejejejeZ e dej ddZ e dej ddZ ededej e d fgfd ed ej e d fdej e d fgfd edej e dfdej e dfdej e dfdej e dfdej e dfgfdedej e dfdej e dfdej e dfgfdedej e dfdej e dfdej e dfd ej e d!fd"ej e dfd#ej e d$fd%ej e d&fd'ej e d(fdej e dfg fd)edej e d*fgfd+edej e d,fd"ej e d,fd#ej e d-fd%ej e d.fd'ej e d/fd0ej e d1fdej e d,fgfd2ed3ej e d4fd5ej e d6fdej e d7fgfd8ed9ej e d:fd;ej e d<fdej e d:fgfd=ed>ej e d?fd@ej e dAfdBej e dAfdCej e dDfdej e d?fgfdEedFej e dGfdHej e dIfdej e dGfgfdJedKej e dLfdMej e dLfdNej e dLfdej e dLfgfdOedPej e dQfdRej e dSfdTej e dUfdVej e dWfdXej e dYfdZej e d[fd\ej e d]fd^ej e d_fd`ej e dafdbej e dcfddej e defdfej e dgfdhej e difdjej e dkfdlej e dmfdnej e dofdpej e dqfdrej e dsfdtej e dufdvej e dwfdxej e dyfdzej e d{fd|ej e d}fd~ej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dQfg fdedej e dfdej e dfdej e dfdej e dfgfgZed+ed#ej e d-fd"ej e d,fdej e d,fd%ej e d.fd'ej e d/fdej e d,fd0ej e d1fgfd2ed3ej e d4fd5ej e d6fdej e d6fgfd ed ej e d fdej e d fgfd edej e dfdej e dfdej e dfdej e dfdej e dfgfd8ed9ej e d:fdej e d:fgfdedej e dfdej e dfdej e dfgfd=ed>ej e d?fd@ej e dAfdBej e dAfdCej e dDfdej e d?fgfdEedFej e dGfdHej e dIfdej e dGfgfdJedKej e dLfdej e dLfgfdOedPej e dQfdRej e dSfdXej e dYfdZej e d[fdTej e dUfd\ej e d]fd^ej e d_fd`ej e dafdbej e dcfddej e defdfej e dgfdhej e difdjej e dkfdlej e dmfdnej e dofdpej e dqfdrej e dsfdtej e dufdvej e dwfdxej e dyfdzej e d{fd|ej e d}fd~ej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dfdej e dQfgfd)edej e d*fgfdedej e dfdej e dfdej e dfdej e dfgfg Zedgdfdgdfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfddgfdPgdfdgdfdgdfddjdlgfdgdfdgdfddvdxgfdgdfgZedgdfdeddd0gfdgdfgfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfddgfddgfdPgdfdgdfdgdfddjdlgfdgdfdgdfddvdxgfdddgfdgdfgZedgdfdgdfdedd3d5gfgfdd gfdgdfdddgfdddgfdedgdfgfdedgdfdPdCgfgfdd9gfdPgdfddgfddjdlgfddgfdgdfdgdfgZedgdfddd0gfdddgfdedgdfgfdedgdfdPdCgfgfdeddKdMgfgfdd9gfdPdPgfdgdfdgdfddjdlgfddngfddvgfdgdfdddgfdddgfgZegZeddgfd"dgfd#dgfd%dgfd'dgfd dgfddgfddgfd0dgfddgfddgfd3dgfd dgfd5dgfddgfddgfddgfd9dgfddgfddgfdFdgfdHdgfddgfd>dgfd@dgfdBdgfdKdgfdMdgfdPdgfdRdgfdXdgfdZdgfdTdgfdjdgfdldgfdndgfdpdgfdrdgfdtdgfdvdgfdxdgfd\dgfd^dgfdbdgfdddgfdhdgfdzdgfd|dgfddgfddgfddgfg3Zeddgfd#dgfd%dgfd'dgfddgfd0dgfd dgfddgfddgfddgfd9dgfddgfddgfdFdgfdHdgfd>dgfd@dgfdKdgfdMdgfdPdgfgZedgZeddgfd"dgfd#dgfd%dgfd'dgfd9dgfddgfddgfdFdgfdHdgfdӐd gfd>d gfd@d gfdBd gfdKd gfdMd gfdPd d gfdXd d gfdRd d gfdZd d gfdTd d gfdjdgfdldgfdndgfdvdgfd\dgfdhdgfdbdgfdzdgfd|dgfd~dgfddgfg ZegdZed`ddZedaddZeddZeddZeddZ ed d!Z!ed"d#Z"ed$d%Z#ed&d'Z$ed(d)Z%ed*d+Z&ed,d-Z'ed.d/Z(ed0d1Z)ed2d3Z*ed4d5Z+ed6d7Z,ed8d9Z-ed:d;Z.ed<d=Z/ed>d?Z0ed@dAZ1edBdCZ2edDdEZ3edFdGZ4edHdIZ5edJdKZ6edLdMZ7edNdOZ8edPdQZ9edRdSZ:edTdUZ;edVdWZed\d]Z?ed^d_Z@dS(bMindFormerBookaR MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository When adding a new Model or assemble in this project, the following constants list and dict need adding. Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> >>> # 1) Fill the following constant list and dict in this class >>> # 2) Overwrite the support_list when define a new Model or Pipeline. >>> @MindFormerRegister.register(MindFormerModuleType.MODELS) ... class ViTForImageClassification(BaseModel): ... _support_list = MindFormerBook.get_model_support_list()['vit'] >>> # 3) Then you can use auto class and from pretrain to init an instance. >>> vit_model = AutoModel.from_pretrained('vit_base_p16') ZCHECKPOINT_DOWNLOAD_FOLDER.Zcheckpoint_downloadZCHECKPOINT_SAVE_FOLDERZcheckpoint_savegeneralcommonz%configs/general/run_general_task.yamlZmasked_image_modelingmae_vit_base_p16z/configs/mae/run_mae_vit_base_p16_224_800ep.yamlZimage_classification vit_base_p16z+configs/vit/run_vit_base_p16_224_100ep.yamlZswin_base_p4w7z.configs/swin/run_swin_base_p4w7_224_100ep.yamlmindspore/vit_base_p16zmindspore/swin_base_p4w7 fill_maskZbert_base_uncasedz'configs/bert/run_bert_base_uncased.yamlZbert_tiny_uncasedz'configs/bert/run_bert_tiny_uncased.yamlZ#contrastive_language_image_pretrain clip_vit_b_32z5configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yamlblip2_stage1_vit_gz:configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yamlblip2_stage2_vit_g_baichuan_7bz5configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yamlblip2_stage2_vit_g_llama_7bz2configs/blip2/run_blip2_stage2_vit_g_llama_7b.yamlmindspore/clip_vit_b_32 clip_vit_b_16z5configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml clip_vit_l_14z5configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yamlclip_vit_l_14@336z9configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yamlZimage_to_text_retrievalz=configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yamlzero_shot_image_classificationzKconfigs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yamlzKconfigs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yamlzKconfigs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yamlzOconfigs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yamlblip2_stage1_classificationzQconfigs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yamlZimage_to_text_generation"itt_blip2_stage2_vit_g_baichuan_7bzNconfigs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yamlitt_blip2_stage2_vit_g_llama_7bzKconfigs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yamlzJconfigs/clip/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml translationZt5_smallz%configs/t5/run_t5_small_on_wmt16.yamlZt5_tinyz$configs/t5/run_t5_tiny_on_wmt16.yamlZtext_classificationtxtcls_bert_base_uncasedz0configs/txtcls/run_txtcls_bert_base_uncased.yamltxtcls_bert_base_uncased_mnliz5configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml'mindspore/txtcls_bert_base_uncased_mnliZ gpt2_txtclsz!configs/gpt2/run_gpt2_txtcls.yamlZtoken_classificationtokcls_bert_base_chinesez0configs/tokcls/run_tokcls_bert_base_chinese.yaml tokcls_bert_base_chinese_cluenerz8configs/tokcls/run_tokcls_bert_base_chinese_cluener.yamlZquestion_answeringZqa_bert_base_uncasedz(configs/qa/run_qa_bert_base_uncased.yamlZqa_bert_base_uncased_squadzmindspore/qa_bert_base_uncasedZtext_generationgpt2zconfigs/gpt2/run_gpt2.yaml gpt2_lorazconfigs/gpt2/run_gpt2_lora.yamlgpt2_13bzconfigs/gpt2/run_gpt2_13b.yamlZgpt2_52bzconfigs/gpt2/run_gpt2_52b.yamlgpt2_xlzconfigs/gpt2/run_gpt2_xl.yaml gpt2_xl_loraz"configs/gpt2/run_gpt2_xl_lora.yamlllama_7bzconfigs/llama/run_llama_7b.yaml llama_13bz configs/llama/run_llama_13b.yaml llama_65bz configs/llama/run_llama_65b.yaml llama2_7bz!configs/llama2/run_llama2_7b.yaml llama2_13bz"configs/llama2/run_llama2_13b.yaml llama2_70bz"configs/llama2/run_llama2_70b.yaml llama_7b_loraz$configs/llama/run_llama_7b_lora.yamlZpangualpha_2_6bz+configs/pangualpha/run_pangualpha_2_6b.yamlZpangualpha_13bz*configs/pangualpha/run_pangualpha_13b.yamlglm_6bz$configs/glm/run_glm_6b_finetune.yaml glm_6b_chatz!configs/glm/run_glm_6b_infer.yaml glm_6b_loraz configs/glm/run_glm_6b_lora.yamlglm_6b_lora_chatz&configs/glm/run_glm_6b_lora_infer.yamlZglm2_6bzconfigs/glm2/run_glm2_6b.yamlZ glm2_6b_loraz"configs/glm2/run_glm2_6b_lora.yaml bloom_560mz!configs/bloom/run_bloom_560m.yaml bloom_7.1bz!configs/bloom/run_bloom_7.1b.yaml bloom_65bz configs/bloom/run_bloom_65b.yaml bloom_176bz!configs/bloom/run_bloom_176b.yaml baichuan_7bz&research/baichuan/run_baichuan_7b.yaml baichuan2_7bz(research/baichuan2/run_baichuan2_7b.yaml baichuan2_13bz)research/baichuan2/run_baichuan2_13b.yamlziya_13bzresearch/ziya/run_ziya_13b.yaml internlm_7bz&research/internlm/run_internlm_7b.yamlinternlm_7b_loraz+research/internlm/run_internlm_7b_lora.yamlZsegment_anything sam_vit_bzconfigs/sam/run_sam_vit-b.yaml sam_vit_lzconfigs/sam/run_sam_vit-l.yaml sam_vit_hzconfigs/sam/run_sam_vit-h.yamlblip2_stage1_vig_gclip)rr!r"r#r Zblip2)rKr%rrr&r'ZittmaeZvit)rr vit_g_p16Zswinberttokcls)r,r-*mindspore/tokcls_bert_base_chinese_cluenertxtcls)r)r*r+qat5ZfilipZfilip_vit_l_14)r.r/r1r2r0llama) r3r4r5r6r7r8r9rBrCrDrErFrGZllama2)r6r7r8Z pangualphaZbloom)r>r?r@rAglm)r:r;r<r=Zglm2Zsam)rHrIrJz1-stgz2-stg)blip2_stage1_pretrainedrr)rrrrNZmt5Zt5_pegasus_base)r3r4r5r6r7r8r9Zinternlm)rr%rr) r3r4r5r6r7r8r9rBrEZ baichuan2zjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/clip_vit_b_32.ckptz[https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/clip_vit_b_32.ckptz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_b_16.ckptz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14.ckptzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14%40336.ckptzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/mae/mae_vit_base_p16.ckptzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/vit/vit_base_p16.ckptrNz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/vit_g_p16.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_pretrained.ckptrWzYhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/vit_base_p16/file/vit_base_p16.ckptzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/swin/swin_base_p4w7.ckptz]https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/swin_base_p4w7/file/swin_base_p4w7.ckptzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/mindspore_model.ckptzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.ckptzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese_cluener.ckptrQzhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/tokcls_bert_base_chinese_cluener.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased_mnli.ckptz{https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/txtcls_bert_base_uncased_mnli.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.ckptzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased_squad.ckptzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.ckptzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_lora.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl.ckptzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckptzehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2_13b.ckptzrhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/pangualpha_2_6b.ckptzqhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/pangualpha_13b.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/glm_6b.ckptzghttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/glm_6b_lora.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/glm2_6b.ckptzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/glm2_6b_lora.ckptzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_7b.ckptzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_13b.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/llama2_7b.ckptzdhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/llama2-13b-fp16.ckptzphttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/open_llama_7b_lora.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/bloom_560m.ckptz^https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/bloom_7.1b.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_b_01ec64.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_l_0b3195.ckptzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/sam/sam_vit_h_4b8939.ckptzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/clip_vit_b_32.yamlz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_b_16.yamlz`https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14.yamlzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/clip/clip_vit_l_14%40336.yamlzfhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_stage1_vit_g.yamlzohttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/blip2_stage1_classification.yamlzlhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/mae/mae_vit_base_p16.yamlzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/vit/vit_base_p16.yamlz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/blip2/vit_g_p16.yamlzkhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/swin/swin_base_p4w7.yamlzjhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/mindspore_model.yamlzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.yamlzxhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese_cluener.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased_mnli.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.yamlzhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased_squad.yamlzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/gpt2.yaml)r$zehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.pngzthttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gzzehttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/bpe_simple_vocab_16e6.txt.gzzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/t5/spiece.modelzahttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/vocab.txtzshttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base_chinese/vocab.txtzfhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/vocab.txtzshttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/bert/bert_base_english/vocab.txtzbhttps://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/vocab.txtzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/vocab.jsonzbhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/gpt2/merges.txtzihttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/pangualpha/vocab.modelzehttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm/ice_text.modelzghttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/glm2/tokenizer.modelzhhttps://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/XFormer_for_mindspore/llama/tokenizer.modelz_https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/llama2/tokenizer.modelz]https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/MindFormers/bloom/tokenizer.json) )ZChatGLMTokenizer GLMProcessor)ZChatGLM2TokenizerrX)Z CLIPTokenizerZ CLIPProcessor) BertTokenizer BertProcessor)Z T5TokenizerZ T5Processor)ZLlamaTokenizerLlamaProcessor)Z GPT2TokenizerZ GPT2Processor)ZPanguAlphaTokenizerZPanguAlphaProcessor)ZBloomTokenizerZBloomProcessor)ZInternLMTokenizerr[)ZBaichuan2Tokenizerr[NcCst|j}|d|D]:\}}|dg}|D]\}}||q<|||<q|r||vr||}tdt |t |qt dntdt |dS)show_trainer_support_model_listrrz+Trainer support model list for %s task is: unsupported taskz-Trainer support model list of MindFormer is: N) copydeepcopy_TRAINER_SUPPORT_TASKS_LISTpoprappendkeysrrrrKeyErrorrclstaskZall_listr r temp_list model_name_rrrr\s          z.MindFormerBook.show_trainer_support_model_listcCst|j}|D]:\}}|dg}|D]\}}||q2|||<q|r||vr||}tdt |t |qt dntdt |dS) show_pipeline_support_model_listrz,Pipeline support model list for %s task is: r]z.Pipeline support model list of MindFormer is: N) r^r__PIPELINE_SUPPORT_TASK_LISTrrarbrcrrrrrdrrerrrrks        z/MindFormerBook.show_pipeline_support_model_listcCstdt|jdS)z)show_tokenizer_name_to_processor function._TRAINER_SUPPORT_TASKS_LIST of MindFormer is: N)rrr_TOKENIZER_NAME_TO_PROCESSORrfrrr show_tokenizer_name_to_processors z/MindFormerBook.show_tokenizer_name_to_processorcCs|jS)z(get_tokenizer_name_to_processor function)rnrorrrget_tokenizer_name_to_processor sz.MindFormerBook.get_tokenizer_name_to_processorcCstdt|jdS)z'show_trainer_support_task_list functionrmN)rrrr`rorrrshow_trainer_support_task_list%s z-MindFormerBook.show_trainer_support_task_listcCstdt|jdS)z.show_pipeline_support_input_data_list functionz,_PIPELINE_INPUT_DATA_LIST of MindFormer is: N)rrr_PIPELINE_INPUT_DATA_LISTrorrr%show_pipeline_support_input_data_list+s z4MindFormerBook.show_pipeline_support_input_data_listcCs|jS)z-get_pipeline_support_input_data_list function)rsrorrr$get_pipeline_support_input_data_list1sz3MindFormerBook.get_pipeline_support_input_data_listcCs|jS)z&get_trainer_support_task_list function)r`rorrrget_trainer_support_task_list6sz,MindFormerBook.get_trainer_support_task_listcCstdt|jdS)z(show_pipeline_support_task_list functionz._PIPELINE_SUPPORT_TASK_LIST of MindFormer is: N)rrrrlrorrrshow_pipeline_support_task_list;s z.MindFormerBook.show_pipeline_support_task_listcCs|jS)z'get_pipeline_support_task_list function)rlrorrrget_pipeline_support_task_listAsz-MindFormerBook.get_pipeline_support_task_listcCst|jdS)"show_model_config_to_name functionN)r_MODEL_CONFIG_TO_NAMErorrrshow_model_config_to_nameFsz(MindFormerBook.show_model_config_to_namecCs|jS)ry)rzrorrrget_model_config_to_nameKsz'MindFormerBook.get_model_config_to_namecCs|j||idS)z set_model_config_to_name function Args: model_config (str): the name of model config model_name (str): the name of model N)rzupdate)rf model_configrirrrset_model_config_to_namePsz'MindFormerBook.set_model_config_to_namecCstdt|jdS)z#show_model_config_url_list functionz)_MODEL_CONFIG_URL_LIST of MindFormer is: N)rrr_MODEL_CONFIG_URL_LISTrorrrshow_model_config_url_list\s z)MindFormerBook.show_model_config_url_listcCs|jS)z"get_model_config_url_list function)rrorrrget_model_config_url_listbsz(MindFormerBook.get_model_config_url_listcCstdt|jdS)zshow_project_path functionz _PROJECT_PATH of MindFormer is: N)rrr _PROJECT_PATHrorrrshow_project_pathgs z MindFormerBook.show_project_pathcCs|jS)zget_project_path function)rrorrrget_project_pathmszMindFormerBook.get_project_pathcCstj|jdS)z0get xihe's mindspore checkpoint download folder. mindspore)ospathjoin#_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDERrorrr#get_xihe_checkpoint_download_folderrsz2MindFormerBook.get_xihe_checkpoint_download_foldercCstdt|jdS)z0show_default_checkpoint_download_folder functionz6_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER of MindFormer is: N)rrrrrorrr'show_default_checkpoint_download_folderws z6MindFormerBook.show_default_checkpoint_download_foldercCs|jS)z/get_default_checkpoint_download_folder function)rrorrr&get_default_checkpoint_download_folder}sz5MindFormerBook.get_default_checkpoint_download_foldercCs$tj|st|d||_dS)z set_default_checkpoint_download_folder function Args: download_folder (str): the path of default checkpoint download folder  should be a directory.N)rrisdirr r)rfZdownload_folderrrr&set_default_checkpoint_download_folders z5MindFormerBook.set_default_checkpoint_download_foldercCs|jS)z+get_default_checkpoint_save_folder function)_DEFAULT_CHECKPOINT_SAVE_FOLDERrorrr"get_default_checkpoint_save_foldersz1MindFormerBook.get_default_checkpoint_save_foldercCs$tj|st|d||_dS)z set_default_checkpoint_save_folder function Args: save_folder (str): the path of default checkpoint save folder rN)rrrr r)rfZ save_folderrrr"set_default_checkpoint_save_folders z1MindFormerBook.set_default_checkpoint_save_foldercCstdt|jdS)z,show_default_checkpoint_save_folder functionz2_DEFAULT_CHECKPOINT_SAVE_FOLDER of MindFormer is: N)rrrrrorrr#show_default_checkpoint_save_folders z2MindFormerBook.show_default_checkpoint_save_foldercCstdt|jdS)z!show_config_support_list functionz&CONFIG_SUPPORT_LIST of MindFormer is: N)rrr_CONFIG_SUPPORT_LISTrorrrshow_config_support_lists z'MindFormerBook.show_config_support_listcCs|jS)z get_config_support_list function)rrorrrget_config_support_listsz&MindFormerBook.get_config_support_listcCstdt|jdS)z show_model_support_list functionz%MODEL_SUPPORT_LIST of MindFormer is: N)rrr_MODEL_SUPPORT_LISTrorrrshow_model_support_lists z&MindFormerBook.show_model_support_listcCs|jS)zget_model_support_list function)rrorrrget_model_support_listsz%MindFormerBook.get_model_support_listcCstdt|jdS)z$show_processor_support_list functionz)PROCESSOR_SUPPORT_LIST of MindFormer is: N)rrr_PROCESSOR_SUPPORT_LISTrorrrshow_processor_support_lists z*MindFormerBook.show_processor_support_listcCs|jS)z#get_processor_support_list function)rrorrrget_processor_support_listsz)MindFormerBook.get_processor_support_listcCs|jS)z#get_tokenizer_support_list function)_TOKENIZER_SUPPORT_LISTrorrrget_tokenizer_support_listsz)MindFormerBook.get_tokenizer_support_listcCst|jdS)z$show_tokenizer_support_list functionN)rrrorrrshow_tokenizer_support_listsz*MindFormerBook.show_tokenizer_support_listcCs|jS)z'get_tokenizer_url_support_list function)_TOKENIZER_SUPPORT_URL_LISTrorrrget_tokenizer_url_support_listsz-MindFormerBook.get_tokenizer_url_support_listcCst|jdS)z(show_tokenizer_url_support_list functionN)rrrorrrshow_tokenizer_url_support_listsz.MindFormerBook.show_tokenizer_url_support_listcCs2g}|jD]}||j|qt|S)get_model_name_support_list)r`rcextendgetset)rfZsupport_model_name task_namerrrrsz*MindFormerBook.get_model_name_support_listcCstdt|jdS)z!show_model_ckpt_url_list functionz&MODEL_CKPT_URL_LIST of MindFormer is: N)rrr_MODEL_CKPT_URL_LISTrorrrshow_model_ckpt_url_lists z'MindFormerBook.show_model_ckpt_url_listcCs|jS)z get_model_ckpt_url_list function)rrorrrget_model_ckpt_url_listsz&MindFormerBook.get_model_ckpt_url_list)N)N)A__name__ __module__ __qualname____doc__rrdirnameabspath__file__rgetenvrrrrr`rlrrrrrzrrrsrrn classmethodr\rkrprqrrrtrurvrwrxr{r|rrrrrrrrrrrrrrrrrrrrrrrrrrrrrr;s    BD  @$            k            k         M     OMV    r) rrr^ collectionsrmindformers.toolsrrrrrrrrs  pengcheng.mind-mindformer/mindformers/__pycache__/version_control.cpython-39.pyc000066400000000000000000000056731476650141400305400ustar00rootroot00000000000000a >cf@sdZddlZddlZddlmZddlmmZddl mm Z ddl m Z ddlmZddZd d Zd d Zd dZdddZddZddZdS)zMindSpore Version ControlN)nn) is_version_ge)loggercCs dd}|S)zCell reuse decorator.cSsFtdddksttjds |Std|jjddl m }||S)NZMS_DEV_CELL_REUSE0z2.1.0zEnable cell use mode at %s.r)cell_attr_register) osgetenvrms __version__rinfo __class____name__Zmindspore._extendsr)funcrrF/home/ma-user/work/mindformers-baichuan/mindformers/version_control.py decorators  z!get_cell_reuse..decoratorr)rrrrget_cell_reusesrcCs.ttjdrtj|d}ntjd|d}|S)N1.11.0)pr) keep_prob)rr r rDropout)Z dropout_probdropoutrrr get_dropout"s rcCs"ttjdrt}nt}|SNr)rr r PTrilr)trilrrrget_tril)s  rcCs(d dd}d dd}ttjdr$|S|S) zreturn ops.normNFcSstj|||||dS)N)orddimkeepdimdtype)FnormArr r!r"rrr tensor_norm14szget_norm..tensor_norm1cSs6|durtdt|ts$tdtj||||dS)Nz>The 'dtype' is not available when mindspore version < '1.11.0'zAThe type of 'ord' should be int when mindspore version < '1.11.0')r keep_dims)rwarning isinstanceint TypeErrorr#r$r%rrr tensor_norm29s   zget_norm..tensor_norm2r)NNFN)r(NFNrr r )r'r.rrrget_norm0s    r0cKslttjdr&|j|f|||d|S|j||||||dd|dd|dd|dd|d dd S) Nr) input_columnsoutput_columnsnum_parallel_workerspython_multiprocessingF max_rowsizecache callbacksoffload) r1r2 column_orderr3r4r5r7r8r9)rr r mapget)dataset operationsr1r2r3kwargsrrrget_dataset_mapDs&      r@cCsttjdrtStjSr)rr r rIdentityr#identityrrrr get_identityVs rCcCs ttjdS)Nz2.2.0r/rrrrfix_optim_global_step_sig[srD)NNN)__doc__r mindsporer rZmindspore.ops.operationsopsr>rZmindspore.ops.functional functionalr#Z tools.utilsrZ tools.loggerrrrrr0r@rCrDrrrrs     pengcheng.mind-mindformer/mindformers/auto_class.py000066400000000000000000001124161476650141400231530ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ AutoConfig、AutoModel """ import os import json import shutil from .mindformer_book import MindFormerBook, print_dict from .models.build_processor import build_processor from .models.base_config import BaseConfig from .models.build_model import build_model from .models.build_config import build_model_config from .tools import logger from .tools.register.config import MindFormerConfig __all__ = ['AutoConfig', 'AutoModel', 'AutoProcessor', 'AutoTokenizer'] class AutoConfig: """ AutoConfig class, helps instantiates a config by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoConfig >>> >>> # 1) instantiates a config by yaml model name >>> config_a = AutoConfig.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a config by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> config_b = AutoConfig.from_pretrained(config_path) """ _support_list = MindFormerBook.get_config_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoConfig is designed to be instantiated " "using the `AutoConfig.from_pretrained(yaml_name_or_path)` method." ) @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] if not yaml_name_or_path.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml model name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif cls.invalid_yaml_name(yaml_name_or_path): raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoModel: """ AutoModel class helps instantiates a model by yaml model name, path or config. If using a model name, the config yaml and checkpoint file will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoModel >>> >>> # 1) input model name, load model and weights >>> model_a = AutoModel.from_pretrained('clip_vit_b_32') >>> # 2) input model directory, load model and weights >>> from mindformers.mindformer_book import MindFormerBook >>> checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') >>> model_b = AutoModel.from_pretrained(checkpoint_dir) >>> # 3) input yaml path, load model without weights >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> model_c = AutoModel.from_config(config_path) >>> # 4) input config, load model without weights >>> config = AutoConfig.from_pretrained('clip_vit_b_32') >>> model_d = AutoModel.from_config(config) """ _support_list = MindFormerBook.get_model_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoModel is designed to be instantiated " "using the `AutoModel.from_pretrained(pretrained_model_name_or_dir)` method " "or `AutoModel.from_config(config)` method." ) @classmethod def invalid_model_name(cls, pretrained_model_name_or_dir): """Check whether it is a valid model name""" if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_model_name_or_dir = pretrained_model_name_or_dir.split('/')[cls._model_name] if not pretrained_model_name_or_dir.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = pretrained_model_name_or_dir.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and pretrained_model_name_or_dir in local_model_list: return False local_model_names = local_model_list.keys() if len(pretrained_model_name_or_dir.split('_')) <= cls._model_name or \ not pretrained_model_name_or_dir.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{pretrained_model_name_or_dir}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = pretrained_model_name_or_dir.split('_')[cls._model_name] if not pretrained_model_name_or_dir in local_model_list[local_model_name]: raise ValueError(f'\'{pretrained_model_name_or_dir}\' is not supported by ' f'\'{local_model_type}_{local_model_name}\', please select from ' f'{local_model_list[local_model_name]}') return False @classmethod def from_config(cls, config, **kwargs): """ From config method, which instantiates a Model by config. Args: config (str, BaseConfig): A model config inherited from BaseConfig, or a path to .yaml file for model config. Returns: A model, which inherited from BaseModel. """ if config is None: raise ValueError("a model cannot be built from config with config is None.") download_checkpoint = kwargs.pop("download_checkpoint", True) if isinstance(config, BaseConfig): inversed_config = cls._inverse_parse_config(config) config_args = cls._wrap_config(inversed_config) elif os.path.exists(config) and config.endswith(".yaml"): config_args = MindFormerConfig(config) else: raise ValueError("config should be inherited from BaseConfig," " or a path to .yaml file for model config.") if not download_checkpoint: config_args.model.model_config.checkpoint_name_or_path = None model = build_model(config_args.model) logger.info("model built successfully!") return model @classmethod def _inverse_parse_config(cls, config): """ Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. """ if not isinstance(config, BaseConfig): return config class_name = config.__class__.__name__ config.update({"type": class_name}) for key, val in config.items(): new_val = cls._inverse_parse_config(val) config.update({key: new_val}) return config @classmethod def _wrap_config(cls, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A model config, which has the same content as a yaml file. """ model_name = config.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) arch = BaseConfig(type=model_name) model = BaseConfig(model_config=config, arch=arch) return BaseConfig(model=model) @classmethod def from_pretrained(cls, pretrained_model_name_or_dir, **kwargs): """ From pretrain method, which instantiates a Model by pretrained model name or path. Args: pretrained_model_name_or_dir (str): A supported model name or a directory to model checkpoint (including .yaml file for config and .ckpt file for weights), the supported model name could be selected from AutoModel.show_support_list(). If pretrained_model_name_or_dir is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Returns: A model, which inherited from BaseModel. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) download_checkpoint = kwargs.pop("download_checkpoint", True) if pretrained_model_name_or_path is not None: pretrained_model_name_or_dir = pretrained_model_name_or_path if not isinstance(pretrained_model_name_or_dir, str): raise TypeError(f"pretrained_model_name_or_dir should be a str," f" but got {type(pretrained_model_name_or_dir)}") is_exist = os.path.exists(pretrained_model_name_or_dir) is_dir = os.path.isdir(pretrained_model_name_or_dir) if is_exist: if not is_dir: raise ValueError(f"{pretrained_model_name_or_dir} is not a directory.") else: if cls.invalid_model_name(pretrained_model_name_or_dir): raise ValueError(f"{pretrained_model_name_or_dir} is not a supported model" f" type or a valid path to model config. supported model" f" could be selected from {cls._support_list}.") if is_dir: yaml_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".yaml")] ckpt_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".ckpt")] if not yaml_list or not ckpt_list: raise FileNotFoundError(f"there is no yaml file for model config or ckpt file" f" for model weights in {pretrained_model_name_or_dir}") yaml_file = os.path.join(pretrained_model_name_or_dir, yaml_list[cls._model_type]) ckpt_file = os.path.join(pretrained_model_name_or_dir, ckpt_list[cls._model_type]) logger.info("config in %s and weights in %s are used for model" " building.", yaml_file, ckpt_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update({"checkpoint_name_or_path": ckpt_file}) model = build_model(config_args.model) else: pretrained_checkpoint_name = pretrained_model_name_or_dir if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_checkpoint_name = pretrained_model_name_or_dir.split('/')[cls._model_name] checkpoint_path = os.path.join( MindFormerBook.get_xihe_checkpoint_download_folder(), pretrained_checkpoint_name.split('_')[cls._model_type]) else: # Default the name of model, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), pretrained_model_name_or_dir.split("_")[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, pretrained_checkpoint_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(pretrained_checkpoint_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update( {"checkpoint_name_or_path": pretrained_model_name_or_dir}) if not download_checkpoint: config_args.model.model_config.checkpoint_name_or_path = None model = build_model(config_args.model) cls.default_checkpoint_download_path = model.default_checkpoint_download_path logger.info("model built successfully!") return model @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoProcessor: """ AutoProcessor helps instantiates a processor by yaml model name or path. If using a model name, the config yaml will be downloaded from obs to ./checkpoint_download dir Examples: >>> from mindformers.auto_class import AutoProcessor >>> >>> # 1) instantiates a processor by yaml model name >>> pro_a = AutoProcessor.from_pretrained('clip_vit_b_32') >>> # 2) instantiates a processor by yaml model path >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), ... 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> pro_b = AutoProcessor.from_pretrained(config_path) """ _support_list = MindFormerBook.get_processor_support_list() _model_type = 0 _model_name = 1 def __init__(self): raise EnvironmentError( "AutoProcessor is designed to be instantiated " "using the `AutoProcessor.from_pretrained(yaml_name_or_path)` method." ) @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiated a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") is_exist = os.path.exists(yaml_name_or_path) model_name = yaml_name_or_path.split('/')[cls._model_name].split("_")[cls._model_type] \ if yaml_name_or_path.startswith('mindspore') else yaml_name_or_path.split("_")[cls._model_type] if not is_exist and model_name not in cls._support_list.keys(): raise ValueError(f'{yaml_name_or_path} does not exist,' f' and it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist: logger.info("config in %s is used for auto processor" " building.", yaml_name_or_path) if os.path.isdir(yaml_name_or_path): yaml_list = [file for file in os.listdir(yaml_name_or_path) if file.endswith(".yaml")] yaml_name = os.path.join(yaml_name_or_path, yaml_list[cls._model_type]) config_args = MindFormerConfig(yaml_name) else: config_args = MindFormerConfig(yaml_name_or_path) else: yaml_name = yaml_name_or_path if not cls.invalid_yaml_name(yaml_name_or_path): if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) else: raise ValueError(f'{yaml_name_or_path} does not exist,' f' or it is not supported by {cls.__name__}.' f' please select from {cls._support_list}.') if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) lib_path = yaml_name_or_path if not os.path.isdir(lib_path): lib_path = None processor = build_processor(config_args.processor, lib_path=lib_path) logger.info("processor built successfully!") return processor @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list class AutoTokenizer: """ Load the tokenizer according to the `yaml_name_or_path`. It supports the following situations 1. `yaml_name_or_path` is the model name. 2. `yaml_name_or_path` is the path to the downloaded files. Examples: >>> from mindformers.auto_class import AutoTokenizer >>> >>> # 1) instantiates a tokenizer by the model name >>> tokenizer_a = AutoTokenizer.from_pretrained("clip_vit_b_32") >>> # 2) instantiates a tokenizer by the path to the downloaded files. >>> from mindformers.models.clip.clip_tokenizer import CLIPTokenizer >>> clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") >>> clip_tokenizer.save_pretrained(path_saved) >>> restore_tokenizer = AutoTokenizer.from_pretrained(path_saved) """ _support_list = MindFormerBook.get_tokenizer_support_list() _model_type = 0 _model_name = 1 @classmethod def invalid_yaml_name(cls, yaml_name_or_path): """Check whether it is a valid yaml name""" if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name_or_path = yaml_name_or_path.split('/')[cls._model_name] if not yaml_name_or_path.split('_')[cls._model_type] in cls._support_list.keys(): return True local_model_type = yaml_name_or_path.split('_')[cls._model_type] local_model_list = cls._support_list[local_model_type] if not isinstance(local_model_list, dict) and yaml_name_or_path in local_model_list: return False if not isinstance(local_model_list, dict): return True local_model_names = local_model_list.keys() if len(yaml_name_or_path.split('_')) <= cls._model_name or \ not yaml_name_or_path.split('_')[cls._model_name] in local_model_names: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}\', ' f'please select from {local_model_list}') local_model_name = yaml_name_or_path.split('_')[cls._model_name] if not yaml_name_or_path in local_model_list[local_model_name]: raise ValueError(f'\'{yaml_name_or_path}\' is not supported by \'{local_model_type}_{local_model_name}\', ' f'please select from {local_model_list[local_model_name]}') return False @classmethod def _get_class_name_from_yaml(cls, yaml_name_or_path): """ Try to find the yaml from the given path Args: yaml_name_or_path (str): the directory of the config yaml Returns: The class name of the tokenizer in the config yaml. """ is_exist = os.path.exists(yaml_name_or_path) is_dir = os.path.isdir(yaml_name_or_path) is_file = os.path.isfile(yaml_name_or_path) if not is_file: if not is_exist: raise ValueError(f"{yaml_name_or_path} does not exist, Please pass a valid the directory.") if not is_dir: raise ValueError(f"{yaml_name_or_path} is not a directory. You should pass the directory.") # If passed a directory, load the file from the yaml files yaml_list = [file for file in os.listdir(yaml_name_or_path) if file.endswith(".yaml")] if not yaml_list: return None yaml_file = os.path.join(yaml_name_or_path, yaml_list[cls._model_type]) else: yaml_file = yaml_name_or_path logger.info("Config in the yaml file %s are used for tokenizer building.", yaml_file) config = MindFormerConfig(yaml_file) class_name = None if config and 'processor' in config and 'tokenizer' in config['processor'] \ and 'type' in config['processor']['tokenizer']: class_name = config['processor']['tokenizer'].pop('type', None) logger.info("Load the tokenizer name %s from the %s", class_name, yaml_name_or_path) return class_name @classmethod def _get_class_name_from_tokenizer_config_file(cls, yaml_name_or_path): """ try to get the tokenizer type from tokenizer_config.json Args: yaml_name_or_path (str): the directory of tokenizer_config.json Returns: The class name of the tokenizer in tokenizer_config.json """ tokenizer_config_path = os.path.join(yaml_name_or_path, 'tokenizer_config.json') if not os.path.exists(tokenizer_config_path): raise FileNotFoundError(f"The file `tokenizer_config.json` should exits in the " f"path {tokenizer_config_path}, but not found.") with open(tokenizer_config_path, 'r') as fp: config_kwargs = json.load(fp) class_name = config_kwargs.pop('tokenizer_class', None) if not class_name: raise ValueError(f"There should be the key word`tokenizer_class` in {tokenizer_config_path}, but " f"not found. The optional keys are {config_kwargs.keys()}") return class_name @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a tokenizer by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/clip_vit_b_32" or "clip_vit_b_32". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A tokenizer which inherited from PretrainedTokenizer. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path from . import MindFormerRegister if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") # Try to load from the remote if os.path.isdir(yaml_name_or_path): class_name = cls._get_class_name_from_yaml(yaml_name_or_path) if not class_name: class_name = cls._get_class_name_from_tokenizer_config_file(yaml_name_or_path) elif not cls.invalid_yaml_name(yaml_name_or_path): # Should download the files from the remote storage yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') class_name = cls._get_class_name_from_yaml(yaml_file) else: raise FileNotFoundError(f"Tokenizer type `{yaml_name_or_path}` does not exist. " f"Use `{cls.__name__}.show_support_list()` to check the supported tokenizer. " f"Or make sure the `{yaml_name_or_path}` is a directory.") dynamic_class = MindFormerRegister.get_cls(module_type='tokenizer', class_name=class_name) instanced_class = dynamic_class.from_pretrained(yaml_name_or_path) logger.info("%s Tokenizer built successfully!", instanced_class.__class__.__name__) return instanced_class @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/core/000077500000000000000000000000001476650141400213675ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/__init__.py000066400000000000000000000024211476650141400234770ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Core.""" from .lr import build_lr from .loss import build_loss from .optim import build_optim from .metric import build_metric from .callback import build_callback from .lr import * from .loss import * from .optim import * from .metric import * from .callback import * from .context import * from .clip_grad import ClipGradNorm from .parallel_config import build_parallel_config __all__ = ['build_parallel_config', 'ClipGradNorm'] __all__.extend(lr.__all__) __all__.extend(loss.__all__) __all__.extend(optim.__all__) __all__.extend(metric.__all__) __all__.extend(callback.__all__) __all__.extend(context.__all__) pengcheng.mind-mindformer/mindformers/core/__pycache__/000077500000000000000000000000001476650141400235775ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/__pycache__/__init__.cpython-39.pyc000066400000000000000000000012741476650141400277730ustar00rootroot00000000000000a ?cf@sdZddlmZddlmZddlmZddlmZddl m Z ddlTddlTddlTddlTddl Tddl Tddl m Z dd lmZd d gZeejeejeejeejee jee jd S) zMindFormers Core.)build_lr) build_loss) build_optim) build_metric)build_callback)*) ClipGradNorm)build_parallel_configr rN)__doc__lrrlossroptimrmetricrcallbackrcontext clip_gradrparallel_configr __all__extendrrD/home/ma-user/work/mindformers-baichuan/mindformers/core/__init__.pys(            pengcheng.mind-mindformer/mindformers/core/__pycache__/clip_grad.cpython-39.pyc000066400000000000000000000056311476650141400301610ustar00rootroot00000000000000a Hlf@sdZddlZddlmZddlmZddlmZ ddlm Z ddlm Z ddlmZdd lmZzdd lmZdd lmZWn*eyddlmZddlmZYn0e d d Ze dZe dZedddZ edddddZ!GdddeZ"dS)z,Operations for clipping grad by global norm.N)ops)Cell) composite) functional) operations)Tensor)dtype) Validator)Rel grad_scaleTget_square_sumapply_global_normrcCs.tdt|d}tt|tjd}|S)NFr)P ReduceSumFsquare expand_dimscastmstypefloat32)xnormrrE/home/ma-user/work/mindformers-baichuan/mindformers/core/clip_grad.py_get_square_sum)srcCsTt|}||d}tj|tdtjttdtjd}||}t ||}|S)Ngư>?)clip_value_maxclip_value_min) rrr clip_by_valuerrrnplogr) clip_norm global_normrx_dtype clip_coefZclip_coef_clampedrrr_apply_global_norm0s   r&cs*eZdZdZdfdd ZddZZS) ClipGradNormae Clips tensor values by the ratio of the sum of their norms. Args: max_norm (Union(float, int)): The clipping ratio. Default: 1.0 use_norm (Union(float, None)): The global norm. Default: None Inputs: - **x** (Union(tuple[Tensor], list[Tensor])) - Input data to clip. Outputs: Tensor, a clipped Tensor. rNcsjtt||dur,td|jd|td|dtj|jt |gt j |_ t |_t|_dS)NzFor 'zF', input 'use_norm' only supports None currently, but got 'use_norm': r"r)superr'__init__ ValueErrorcls_name validator check_numberr GTrrrr"CHyperMap hyper_mapr GreaterEqual greater_equal)selfmax_normuse_norm __class__rrr)Js  zClipGradNorm.__init__cCs^|t|}tt|}|}|||j}t|||j}|tt |j||}||fS)z clip grad.) r1r rsqrtaddnr3r"selectpartialr )r4r square_sumr#Zorigin_global_normcondclip_xrrr constructUs zClipGradNorm.construct)rN)__name__ __module__ __qualname____doc__r)r@ __classcell__rrr7rr';s r')#rDnumpyr mindsporermindspore.nn.cellr mindspore.opsrr/rrrrmindspore.common.tensorrmindspore.commonrrZmindspore._checkparamr r,r ImportError _checkparam ExpandDims add_prim_attrrMultitypeFuncGraphr r registerrr&r'rrrrs,               pengcheng.mind-mindformer/mindformers/core/__pycache__/parallel_config.cpython-39.pyc000066400000000000000000000023751476650141400313600ustar00rootroot00000000000000a ?cf @sLdZddlmZmZddlmZmZddlmZeZ ee dZ ddZ dS) zParallel Config Init.)default_moe_config MoEConfig)TransformerOpParallelConfigTransformerRecomputeConfig)logger) recomputecCs|jr4t|jts:td|jtfi|j|_nt|_|jrnt|jtsttd|jtfi|j|_nt|_|j rt|j t std|j |j j dkrtd|j j d|j _ t fd|ji|j |_ nt |_ dS) zBuild context config.z initial moe_config from dict: %sz&initial recompute_config from dict: %sz%initial parallel_config from dict: %sz=pipeline_stage = %s > 1, vocab_emd_dp will be reset to False.FrN) moe_config isinstancerrinforZrecompute_configrdefault_recompute_configparallel_configrpipeline_stage vocab_emb_dpdefault_parallel_config)configrK/home/ma-user/work/mindformers-baichuan/mindformers/core/parallel_config.pybuild_parallel_configs,      rN) __doc__#mindformers.modules.transformer.moerrmindformers.modules.transformerrrmindformers.tools.loggerrr rrrrrrs   pengcheng.mind-mindformer/mindformers/core/callback/000077500000000000000000000000001476650141400231235ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/callback/.ipynb_checkpoints/000077500000000000000000000000001476650141400267145ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/callback/.ipynb_checkpoints/build_callback-checkpoint.py000066400000000000000000000063551476650141400343370ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Callback API.""" import inspect from mindspore.train import callback from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_callback( config: [dict, list] = None, default_args: dict = None, module_type: str = 'callback', class_name: str = None, **kwargs): r"""Build callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_callback = config if not isinstance(cfg_callback, list): return MindFormerRegister.get_instance_from_cfg( cfg_callback, MindFormerModuleType.CALLBACK, default_args=default_args) callbacks = [] for callback_type in cfg_callback: if callback_type.get('type') == 'MFLossMonitor': callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK, default_args) else: callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK) callbacks.append(callback_op) return callbacks return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_cb(): """ register MindSpore builtin LR class. """ for module_name in dir(callback): if module_name.startswith('__'): continue monitor = getattr(callback, module_name) if inspect.isclass(monitor): MindFormerRegister.register_cls( monitor, MindFormerModuleType.CALLBACK) register_ms_cb() pengcheng.mind-mindformer/mindformers/core/callback/.ipynb_checkpoints/callback-checkpoint.py000066400000000000000000000756561476650141400331720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Callback.""" import json import os import time import datetime from collections import OrderedDict from copy import deepcopy from typing import Callable, Optional, Union import numpy as np import mindspore as ms from mindspore import Callback, Profiler, ModelCheckpoint, CheckpointConfig, context, save_checkpoint, Tensor from mindspore.train.callback import SummaryCollector from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindspore.train.callback._callback import set_cur_net from mindspore.train.serialization import _get_merged_param_data from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.cloud_adapter.cloud_adapter import Local2ObsMonitor from mindformers.tools.logger import logger from mindformers.tools.utils import get_output_root_path, get_output_subpath, get_remote_save_url, check_in_modelarts __all__ = ['ObsMonitor', 'MFLossMonitor', 'CheckpointMointor', 'SummaryMonitor', 'ProfileMonitor', 'EvalCallBack'] _cur_dir = os.getcwd() SAVE_DIR = _cur_dir @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ObsMonitor: """Obs Monitor For AICC and Local""" def __new__(cls, src_dir: str = None, target_dir: str = None, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last: bool = True): if src_dir is None: src_dir = get_output_root_path() if target_dir is None: target_dir = get_remote_save_url() return Local2ObsMonitor(src_dir, target_dir, step_upload_frequence, epoch_upload_frequence, keep_last) def _get_loss_output(output): """Get output of task for MFLossMonitor.""" overflow = False scaling_sens = False loss = output learning_rate = None grad_norm = None if isinstance(output, (tuple, list)): if len(output) == 3: loss, overflow, scaling_sens = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 4: loss, overflow, scaling_sens, learning_rate = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 5: loss, overflow, scaling_sens, learning_rate, grad_norm = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() else: if isinstance(output[0], ms.Tensor) and isinstance(output[0].asnumpy(), np.ndarray): loss = output[0] if isinstance(loss, ms.Tensor) and isinstance(loss.asnumpy(), np.ndarray): loss = np.mean(loss.asnumpy()) # Boundary check. if isinstance(loss, float) and (np.isnan(loss) or np.isinf(loss)): invalid_loss_info = "NaN" if np.isnan(loss) else "Inf" raise ValueError(f"The current value of loss is {invalid_loss_info}, terminate training.") return loss, overflow, scaling_sens, learning_rate, grad_norm @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class MFLossMonitor(Callback): """ Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) """ def __init__(self, learning_rate: Optional[Union[float, LearningRateSchedule]] = None, per_print_times: int = 1, micro_batch_num: int = 1, micro_batch_interleave_num: int = 1, origin_epochs: int = None, dataset_size: int = None, initial_epoch: int = 0, global_batch_size: int = 0, device_num: int = 0): super(MFLossMonitor, self).__init__() self.per_print_times = per_print_times self.learning_rate = deepcopy(learning_rate) self.last_print_time = 0 self.mirco_size = micro_batch_num self.print_warning_flag = True self.loss_list = [] self.step_time = time.time() self.epoch_time = time.time() self.run_context = None self.steps_per_epoch = dataset_size self.micro_batch_interleave_num = micro_batch_interleave_num self.origin_epochs = origin_epochs self.initial_epoch = initial_epoch self.global_batch_size = global_batch_size self.device_num = device_num def epoch_begin(self, run_context): """ Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. """ self.loss_list = [] self.epoch_time = time.time() self.run_context = run_context def epoch_end(self, run_context): """ Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. """ def step_begin(self, run_context): """ Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. """ self.step_time = time.time() self.run_context = run_context def step_end(self, run_context): """ Print training info at the end of step. Args: run_context (RunContext): Context of the process running. """ parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") auto_parallel = parallel_mode in ['semi_auto_parallel', 'auto_parallel'] if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) cb_params = run_context.original_args() step_seconds = (time.time() - self.step_time) * 1000 net_outputs = cb_params.net_outputs loss, overflow, scaling_sens, learning_rate, grad_norm = _get_loss_output(net_outputs) if learning_rate is not None: self.learning_rate = learning_rate loss = self._fix_loss_for_parallel(loss) self.loss_list.append(loss) if not overflow: overflow = "False" if not scaling_sens: scaling_sens = "unavailable" if cb_params.dataset_sink_mode: origin_epochs = self.origin_epochs per_step_seconds = step_seconds / cb_params.batch_num steps_per_epoch = self.steps_per_epoch cur_epoch_num = (cb_params.cur_step_num - 1) // steps_per_epoch \ + self.initial_epoch * cb_params.batch_num // steps_per_epoch + 1 cur_step_num = (cb_params.cur_step_num - 1) % steps_per_epoch + 1 else: origin_epochs = self.origin_epochs per_step_seconds = step_seconds steps_per_epoch = cb_params.batch_num cur_epoch_num = cb_params.cur_epoch_num cur_step_num = (cb_params.cur_step_num - 1) % cb_params.batch_num + 1 # compute time remaining step_remain = (origin_epochs - cur_epoch_num + 1) * steps_per_epoch - cur_step_num time_remain = step_remain * per_step_seconds / 1000 # compute throughput throughput = self.global_batch_size / self.device_num / (per_step_seconds / 1000) # compute percent percent = ((cur_epoch_num - 1) * steps_per_epoch + cur_step_num) / origin_epochs / steps_per_epoch * 100 if (cb_params.cur_step_num - self.last_print_time) >= self.per_print_times: self.last_print_time = cb_params.cur_step_num self.print_output_info(cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm) if check_in_modelarts() and int(os.getenv("RANK_ID", "0")) == int(os.getenv("RANK_SIZE", "1")) - 1: self.dump_info_to_modelarts(ma_step_num=cur_step_num, ma_loss=loss) if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode=parallel_mode, full_batch=full_batch) def _fix_loss_for_parallel(self, loss): """Fix loss value in pipeline or double parallel mode.""" pipeline_stages = ms.context.get_auto_parallel_context("pipeline_stages") if pipeline_stages > 1 and self.print_warning_flag: logger.warning("pipeline stages: %s > 1, the loss on the last card is valid.", pipeline_stages) if self.micro_batch_interleave_num > 1 and self.print_warning_flag: logger.warning("micro_batch_interleave_num: %s > 1, multiple copies in parallel is open.") if pipeline_stages > 1: loss = loss / (self.mirco_size * self.micro_batch_interleave_num) elif self.micro_batch_interleave_num > 1: loss = loss / self.micro_batch_interleave_num return loss def print_output_info(self, cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm): """print output information.""" if self.learning_rate is not None: if isinstance(self.learning_rate, (float, Tensor)): current_lr = str(self.learning_rate) elif isinstance(self.learning_rate, LearningRateSchedule): if ms.context.get_context('device_target') == 'CPU': if self.print_warning_flag: logger.warning( "device target not support CPU when generating the learning rate value, " "please use: mindspore.context.set_context(device_target='Ascend')") self.print_warning_flag = False current_lr = None else: if cb_params.optimizer is not None: global_step = cb_params.optimizer.global_step else: global_step = cb_params.network.optimizer.global_step # temporary set_train to avoid error on 910B origin_phase = cb_params.train_network.phase cb_params.train_network.set_train(False) current_lr = self.learning_rate(global_step) cb_params.train_network.set_train(origin_phase) current_lr = np.array2string(current_lr.asnumpy()) else: if self.print_warning_flag: logger.warning( "The current learning rate cannot be calculated in real time." "Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor," "but the input learning rate function type is %s", type(self.learning_rate) ) self.print_warning_flag = False current_lr = None else: if self.print_warning_flag: logger.warning( "MFLossMonitor callback is not set learning rate arguments." "To display the learning rate, you must input the arguments, " "which can be LearningRateSchedule or a fixed float" ) self.print_warning_flag = False current_lr = None if current_lr is not None: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) else: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), overflow, scaling_sens) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), overflow, scaling_sens) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) def dump_info_to_modelarts(self, ma_step_num, ma_loss): """dump modelarts info to display evaluation result page""" ma_loss = float(ma_loss) obj = None modelarts_dir = os.path.join(get_output_root_path(), "modelarts") if not os.path.exists(modelarts_dir): os.mkdir(modelarts_dir) if not os.path.exists(os.path.join(modelarts_dir, "model_analysis_results.json")): obj = { "en-us": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "loss of model", "value": {"current_loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} }, "zh-cn": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "模型损失", "value": {"当前loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} } } else: with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "r") as fp: obj = json.load(fp) if obj is not None: en_precision_performance = obj["en-us"]["precision_performance"] en_precision_performance["pr"]["value"]["loss_value"] = ma_loss en_loss_list = en_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] en_loss_list.append([ma_step_num, ma_loss]) zh_precision_performance = obj["zh-cn"]["precision_performance"] zh_precision_performance["pr"]["value"]["当前loss"] = ma_loss zh_loss_list = zh_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] zh_loss_list.append([ma_step_num, ma_loss]) with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "w") as fp: json.dump(obj, fp) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class SummaryMonitor: """Summary Monitor For AICC and Local""" def __new__(cls, summary_dir=None, collect_freq=10, collect_specified_data=None, keep_default_action=True, custom_lineage_data=None, collect_tensor_freq=None, max_file_size=None, export_options=None): if summary_dir is None: rank_id = int(os.getenv("RANK_ID", '0')) summary_dir = get_output_subpath('summary', rank_id) kwargs = { "summary_dir": summary_dir, "collect_freq": collect_freq, "collect_specified_data": collect_specified_data, "keep_default_action": keep_default_action, "custom_lineage_data": custom_lineage_data, "collect_tensor_freq": collect_tensor_freq, "max_file_size": max_file_size, "export_options": export_options } return SummaryCollector(**kwargs) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class CheckpointMointor(ModelCheckpoint): """Checkpoint Monitor For Save LossScale""" def __init__(self, prefix='CKP', directory=None, config=None, save_checkpoint_steps=1, save_checkpoint_seconds=0, keep_checkpoint_max=5, keep_checkpoint_per_n_minutes=0, integrated_save=True, save_network_params=True, save_trainable_params=False, async_save=False, saved_network=None, append_info=None, enc_key=None, enc_mode='AES-GCM', exception_save=False): self.config = config self.save_network_params = save_network_params self.save_trainable_params = save_trainable_params self.rank_id = int(os.getenv("RANK_ID", '0')) prefix = prefix + "_rank_{}".format(self.rank_id) if append_info is None: append_info = [{ "epoch_num": 0, "step_num": 0, "global_step": 0, "loss_scale": 1 }] ckpt_directory = os.path.join(directory, f"checkpoint/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint', self.rank_id) self.network_directory = os.path.join(directory, f"checkpoint_network/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_network', self.rank_id) self.trainable_directory = os.path.join(directory, f"checkpoint_trainable/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_trainable', self.rank_id) if context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: logger.info("Integrated_save is changed to False when using auto_parallel.") integrated_save = False config_ck = CheckpointConfig(save_checkpoint_steps=save_checkpoint_steps, save_checkpoint_seconds=save_checkpoint_seconds, keep_checkpoint_max=keep_checkpoint_max, keep_checkpoint_per_n_minutes=keep_checkpoint_per_n_minutes, integrated_save=integrated_save, async_save=async_save, saved_network=saved_network, append_info=append_info, enc_key=enc_key, enc_mode=enc_mode, exception_save=exception_save) super(CheckpointMointor, self).__init__(prefix, ckpt_directory, config=config_ck) def _save_ckpt(self, cb_params, force_to_save=False): """Save checkpoint files.""" # pylint: disable=E0203 if cb_params.cur_step_num == self._last_triggered_step: return # if param is cache enable, flush data from cache to host before save_ckpt if self._need_flush_from_cache: self._flush_from_cache(cb_params) save_ckpt = self._check_save_ckpt(cb_params, force_to_save) step_num_in_epoch = int((cb_params.cur_step_num - 1) % cb_params.batch_num + 1) if save_ckpt: logger.info('......Saving ckpt......') cur_ckpoint_file = self._prefix + "-" + str(cb_params.cur_epoch_num) + "_" \ + str(step_num_in_epoch) + ".ckpt" # update checkpoint file list. self._manager.update_ckpoint_filelist(self._directory, self._prefix) # keep checkpoint files number equal max number. if self._config.keep_checkpoint_max and 0 < self._config.keep_checkpoint_max <= self._manager.ckpoint_num: self._manager.remove_oldest_ckpoint_file() elif self._config.keep_checkpoint_per_n_minutes and self._config.keep_checkpoint_per_n_minutes > 0: # pylint: disable=E0203 self._cur_time_for_keep = time.time() if (self._cur_time_for_keep - self._last_time_for_keep) \ < self._config.keep_checkpoint_per_n_minutes * 60: self._manager.keep_one_ckpoint_per_minutes(self._config.keep_checkpoint_per_n_minutes, self._cur_time_for_keep) # generate the new checkpoint file and rename it. global SAVE_DIR SAVE_DIR = self._directory cur_file = os.path.join(self._directory, cur_ckpoint_file) self._last_time_for_keep = time.time() self._last_triggered_step = cb_params.cur_step_num if context.get_context("enable_ge") and os.getenv("MS_ENABLE_REF_MODE", "0") == "0": set_cur_net(cb_params.train_network) cb_params.train_network.exec_checkpoint_graph() if "epoch_num" in self._append_dict: self._append_dict["epoch_num"] = self._append_epoch_num + cb_params.cur_epoch_num if "step_num" in self._append_dict: self._append_dict["step_num"] = self._append_step_num + cb_params.cur_epoch_num * cb_params.batch_num if cb_params.optimizer is not None: self._append_dict["global_step"] = cb_params.optimizer.global_step else: self._append_dict["global_step"] = cb_params.network.optimizer.global_step if "loss_scale" in self._append_dict: outputs = cb_params.net_outputs if isinstance(outputs, (tuple, list)) and len(outputs) >= 3: self._append_dict["loss_scale"] = outputs[2] network = self._config.saved_network if self._config.saved_network is not None else cb_params.train_network save_checkpoint(network, cur_file, self._config.integrated_save, self._config.async_save, self._append_dict, self._config.enc_key, self._config.enc_mode) def save_only_network_params(): save_obj = cb_params.network if save_obj.optimizer is not None: save_obj = save_obj.network if self.save_network_params: cb_cur_ckpoint_file = self._prefix + "-" + "network" + ".ckpt" cb_cur_file = os.path.join(self.network_directory, cb_cur_ckpoint_file) os.makedirs(self.network_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) if self.save_trainable_params: save_obj.init_parameters_data() param_dict = OrderedDict() for param in save_obj.trainable_params(): param_dict[param.name] = param param_list = [] for (key, value) in param_dict.items(): each_param = {"name": key} param_data = Tensor(value.data.asnumpy()) # in automatic model parallel scenario, some parameters were split to all the devices, # which should be combined before saving if key in save_obj.parameter_layout_dict: param_data = _get_merged_param_data(save_obj, key, param_data, self._config.integrated_save) each_param["data"] = param_data param_list.append(each_param) save_obj = param_list cb_cur_ckpoint_file = self._prefix + "-" + "trainable_params" + ".ckpt" cb_cur_file = os.path.join(self.trainable_directory, cb_cur_ckpoint_file) os.makedirs(self.trainable_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) save_only_network_params() self._latest_ckpt_file_name = cur_file @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ProfileMonitor(Callback): """ Profile analysis in training. """ def __init__(self, start_step=1, stop_step=10, output_path=None, start_profile=True, profile_communication=False, profile_memory=True, **kwargs): super(ProfileMonitor, self).__init__() self.start_step = start_step self.stop_step = stop_step self.start_profile = start_profile self.profile_communication = profile_communication if profile_communication and not start_profile: raise ValueError("When profile_communication is True, start_profile must also be True") if output_path is None: rank_id = int(os.getenv("RANK_ID", '0')) output_path = get_output_subpath('profile', rank_id) if ms.get_context("device_target") == "GPU" and profile_memory: logger.warning("The parameter profile_memory is not supported on GPU currently, so is changed to False. ") profile_memory = False self.profiler = Profiler( start_profile=start_profile, output_path=output_path, profile_communication=profile_communication, profile_memory=profile_memory, **kwargs) self.run_context = None self.output_path = output_path def step_begin(self, run_context): """ Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.start_step and not self.start_profile: self.profiler.start() def step_end(self, run_context): """ Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.stop_step: self.profiler.stop() self.profiler.analyse() logger.info("End of Profiling, please view the profile data under %s and analyze it using mindinsight." "MindInsight order as follow: " "mindinsight start --summary-base-dir %s", self.output_path, self.output_path) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class EvalCallBack(Callback): """Evaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. """ def __init__(self, eval_func: Callable, step_interval: int = 100, epoch_interval: int = -1): self.eval_func = eval_func self.step_interval = step_interval self.epoch_interval = epoch_interval def epoch_end(self, run_context): # if not use epoch end if self.epoch_interval <= 0: return callback_params = run_context.original_args() cur_epoch_num = callback_params.cur_epoch_num if cur_epoch_num % self.epoch_interval == 0: self._execute_eval() def step_end(self, run_context): # if not use step end if self.step_interval <= 0: return callback_params = run_context.original_args() cur_step_num = callback_params.cur_step_num if cur_step_num % self.step_interval == 0: self._execute_eval() def _execute_eval(self): start_time = time.time() output = self.eval_func() eval_time = time.time() - start_time logger.info("Eval result: %s, eval time is %f s.", output, eval_time) pengcheng.mind-mindformer/mindformers/core/callback/__init__.py000066400000000000000000000014511476650141400252350ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Callback.""" from .build_callback import build_callback from .callback import * __all__ = [] __all__.extend(callback.__all__) pengcheng.mind-mindformer/mindformers/core/callback/__pycache__/000077500000000000000000000000001476650141400253335ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/callback/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004611476650141400315240ustar00rootroot00000000000000a ?cf)@s,dZddlmZddlTgZeejdS)zMindFormers Callback.)build_callback)*N)__doc__rcallback__all__extendrrM/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/__init__.pys pengcheng.mind-mindformer/mindformers/core/callback/__pycache__/build_callback.cpython-39.pyc000066400000000000000000000043261476650141400327040ustar00rootroot00000000000000a ?cf @sXdZddlZddlmZddlmZmZmZd ee gee e dddZ d d Z e dS) zBuild Callback API.N)callback)MindFormerRegisterMindFormerModuleTypeMindFormerConfigr)config default_args module_type class_namec Ks|dur|durdS|durt|tr>t|ts>tfi|}|}t|ts^tj|tj|dSg}|D]<}|ddkrt|tj|}nt|tj}| |qf|Stj ||fi|S)atBuild callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') N)rtypeZ MFLossMonitor) isinstancedictrlistrget_instance_from_cfgrCALLBACKgetappend get_instance) rrrr kwargsZ cfg_callback callbacks callback_typeZ callback_oprS/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/build_callback.pybuild_callbacks,  rcCs@ttD]2}|drqtt|}t|rt|tj qdS)z& register MindSpore builtin LR class. __N) dirr startswithgetattrinspectisclassr register_clsrr) module_namemonitorrrrregister_ms_cbDs    r")NNrN) __doc__rZmindspore.trainrmindformers.tools.registerrrrr r strrr"rrrrs  - pengcheng.mind-mindformer/mindformers/core/callback/__pycache__/callback.cpython-39.pyc000066400000000000000000000456101476650141400315260ustar00rootroot00000000000000a kf{@sdZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl ZddlZddlmZmZmZmZmZmZmZddlmZddlmZdd lmZdd lmZdd l m!Z!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+gdZ,e-Z.e.a/e!0e"j1GdddZ2ddZ3e!0e"j1GdddeZ4e!0e"j1GdddZ5e!0e"j1GdddeZ6e!0e"j1GdddeZ7e!0e"j1GdddeZ8dS)z MindFormer Self-Define Callback.N) OrderedDict)deepcopy)CallableOptionalUnion)CallbackProfilerModelCheckpointCheckpointConfigcontextsave_checkpointTensor)SummaryCollector)LearningRateSchedule) set_cur_net)_get_merged_param_data)MindFormerRegisterMindFormerModuleType)Local2ObsMonitor)logger)get_output_root_pathget_output_subpathget_remote_save_urlcheck_in_modelarts) ObsMonitor MFLossMonitorCheckpointMointorSummaryMonitorProfileMonitor EvalCallBackc@s(eZdZdZd eeeeedddZdS) rzObs Monitor For AICC and LocalNdT)src_dir target_dirstep_upload_frequenceepoch_upload_frequence keep_lastcCs,|durt}|durt}t|||||SN)rrr)clsr"r#r$r%r&r)M/home/ma-user/work/mindformers-baichuan/mindformers/core/callback/callback.py__new__/s zObsMonitor.__new__)NNr r!T)__name__ __module__ __qualname____doc__strintboolr+r)r)r)r*r+srcCs\d}d}|}d}d}t|ttfrt|dkrN|\}}}t|tjr|}nt|dkr||\}}}}t|tjr|}n\t|dkr|\}}}}}t|tjr|}n,t|dtjrt|dtjr|d}t|tjrt|tjrt |}t|t rNt |s*t |rNt |r:dnd}t d |d |||||fS) z%Get output of task for MFLossMonitor.FNrNaNInfzThe current value of loss is z, terminate training.) isinstancetuplelistlenmsr asnumpynpndarraymeanfloatisnanisinf ValueError)outputoverflow scaling_sensloss learning_rate grad_normZinvalid_loss_infor)r)r*_get_loss_output<s4           $ $rKc s|eZdZdZdeeeefeeeeeeeed fdd Z dd Z d d Z d d Z ddZ ddZddZddZZS)ra Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) Nr) rIper_print_timesmicro_batch_nummicro_batch_interleave_num origin_epochs dataset_size initial_epochglobal_batch_size device_numc sxtt|||_t||_d|_||_d|_g|_ t |_ t |_ d|_ ||_||_||_||_||_| |_dS)NrT)superr__init__rMrrIlast_print_time mirco_sizeprint_warning_flag loss_listtime step_time epoch_time run_contextsteps_per_epochrOrPrRrSrT) selfrIrMrNrOrPrQrRrSrT __class__r)r*rVqs    zMFLossMonitor.__init__cCsg|_t|_||_dS)z Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. N)rZr[r]r^r`r^r)r)r* epoch_begins zMFLossMonitor.epoch_begincCsdS)z Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. Nr)rcr)r)r* epoch_endszMFLossMonitor.epoch_endcCst|_||_dS)z Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. N)r[r\r^rcr)r)r* step_begins zMFLossMonitor.step_begincCstd}td}|dv}|r0tjjddd|}t|jd}|j}t|\}} } } } | durp| |_ | |}|j || sd } | sd } |j r|j} ||j}|j}|jd ||j|j|d }|jd |d }n*|j} |}|j}|j}|jd |jd }| |d ||}||d}|j|j|d}|d ||| |d }|j|j|jkr|j|_|||| |||||| | ||| trttd dttddd kr|j||d|rtjj||ddS)z Print training info at the end of step. Args: run_context (RunContext): Context of the process running. parallel_mode full_batch)semi_auto_parallel auto_parallel data_parallelF)rgrhiNFalse unavailablerLr RANK_ID0 RANK_SIZE1) ma_step_numma_loss)r<get_auto_parallel_contextr set_auto_parallel_context original_argsr[r\ net_outputsrKrI_fix_loss_for_parallelrZappenddataset_sink_moderP batch_numr_ cur_step_numrR cur_epoch_numrSrTrWrMprint_output_inforr1osgetenvdump_info_to_modelarts)r`r^rgrhrj cb_params step_secondsrwrHrFrGrIrJrPper_step_secondsr_r}r|Z step_remain time_remain throughputpercentr)r)r*step_ends\         .zMFLossMonitor.step_endcCsrtjd}|dkr&|jr&td||jdkr@|jr@td|dkrZ||j|j}n|jdkrn||j}|S)z3Fix loss value in pipeline or double parallel mode.pipeline_stagesrLz 1, the loss on the last card is valid.zHmicro_batch_interleave_num: %s > 1, multiple copies in parallel is open.)r<r rtrYrwarningrOrX)r`rHrr)r)r*rxs    z$MFLossMonitor._fix_loss_for_parallelcCs |jdurt|jttfr&t|j}qt|jtrtjddkr^|j rXt dd|_ d}q|j durr|j j }n |jj j }|jj}|jd||}|j|t|}q|j rt dt|jd|_ d}n|j rt dd|_ d}|dur|jr*t d|||||t||| | | n,t d |||||t|jt||| | | d d td | d d }t d| ||tjt| dn|jrt d|||||t|| | n(t d|||||t|jt|| | d d td | d d }t d| ||tjt| ddS)zprint output information.N device_targetCPUzdevice target not support CPU when generating the learning rate value, please use: mindspore.context.set_context(device_target='Ascend')FzThe current learning rate cannot be calculated in real time.Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor,but the input learning rate function type is %szMFLossMonitor callback is not set learning rate arguments.To display the learning rate, you must input the arguments, which can be LearningRateSchedule or a fixed floatz}{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %sz{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %sz|%%-%ds|2r u▮z# %4.1f%% %s %.2f samples/s/p %s })secondszf{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, per_step_time: %dms, overflow cond: %s, loss_scale: %szm{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], per_step_time: %dms, overflow cond: %s, loss_scale: %s)rIr8rAr r0rr<r get_contextrYrr optimizer global_stepnetwork train_networkphase set_trainr> array2stringr=typerzinfor1r@rZdatetime timedelta)r`rr}rPrr|r_rHrrFrGrrrJZ current_lrr origin_phaseZshow_strr)r)r*r~sv                zMFLossMonitor.print_output_infoc Cst|}d}tjtd}tj|s2t|tjtj|dsidddddid d d ddgid id iiiiididddddid d d ddgid id iiiiidd}n>ttj|dd}t |}Wdn1s0Y|dur|dd}||ddd<|ddd dd}| ||g|dd}||ddd<|ddd dd} | ||gttj|dd}t ||Wdn1s0YdS)z5dump modelarts info to display evaluation result pageNZ modelartszmodel_analysis_results.jsonprrHz loss of modelZ current_lossrZ pr_line_chartzloss line chart of modelstep)nameZ x_axis_nameZ y_axis_namecurve)title descriptionvalue line_chart)commonprecision_performanceZfeature_sensitivityZcomputational_performanceZabstract_featureZ adversaryu 模型损失u 当前loss)en-uszh-cnrrrr loss_valuerrrw) rArpathjoinrexistsmkdiropenjsonloadrydump) r`rrrsobjZ modelarts_dirfpZen_precision_performanceZ en_loss_listZzh_precision_performanceZ zh_loss_listr)r)r*rEsl      #(   z$MFLossMonitor.dump_info_to_modelarts) NrLrLrLNNrrr)r,r-r.r/rrrArr1rVrdrerfrrxr~r __classcell__r)r)rar*r_s8  ?Lrc@seZdZdZdddZdS)rz"Summary Monitor For AICC and LocalN Tc CsF|dur"ttdd} td| }||||||||d} tfi| S)Nrnrosummary) summary_dir collect_freqcollect_specified_datakeep_default_actioncustom_lineage_datacollect_tensor_freq max_file_sizeexport_options)r1rrrr) r(rrrrrrrrrank_idkwargsr)r)r*r+s   zSummaryMonitor.__new__)NrNTNNNN)r,r-r.r/r+r)r)r)r*rsrcs,eZdZdZdfd d Zdd d ZZS)rz%Checkpoint Monitor For Save LossScaleCKPNrLrr5TFAES-GCMc s||_| |_| |_ttdd|_|d|j}| durLdddddg} |rftj |d|jn t d|j}|rtj |d |jn t d |j|_ |rtj |d |jn t d |j|_ t d dvrtdd}t|||||| | | |||d }tt|j|||ddS)Nrnroz_rank_{}rrL) epoch_numstep_numr loss_scalezcheckpoint/rank_ checkpointzcheckpoint_network/rank_Zcheckpoint_networkzcheckpoint_trainable/rank_Zcheckpoint_trainablerg)rirjhybrid_parallelz=Integrated_save is changed to False when using auto_parallel.F) save_checkpoint_stepssave_checkpoint_secondskeep_checkpoint_maxkeep_checkpoint_per_n_minutesintegrated_save async_save saved_network append_infoenc_keyenc_modeexception_save)config)rsave_network_paramssave_trainable_paramsr1rrrformatrrrnetwork_directorytrainable_directoryr rtrrr rUrrV)r`prefix directoryrrrrrrrrrrrrrrZckpt_directoryZ config_ckrar)r*rVsR     zCheckpointMointor.__init__c stjjkrdSjr |}tjdjd}|rptdj dt j dt |d}j jj jjrdjjkrj jkrnn j nPjjrjjdkrt_jjjjdkrj jjjjatjj|}t_j_td rhtd d d krhtj j !d j"vrj#j j"d <d j"vrj$j jj"d <j%durƈj%j&j"d<nj'j%j&j"d<dj"vrj(}t)|t*t+frt,|dkr|dj"d<jj-dur*jj-nj }t.||jj/jj0j"jj1jj2fdd} | |_3dS)zSave checkpoint files.NrLz......Saving ckpt......-_.ckptr< enable_geZMS_ENABLE_REF_MODErorrrrr3c sXj}|jdur|j}jrrjddd}tjj|}tjjddt ||j j j j ij j j jjrT|t}|D]}|||j<qg}|D]L\}}d|i}t|j} ||jvrt||| j j } | |d<||q|}jddd}tjj|}tjjddt ||j j j j ij j j jdS) NrrrT)exist_okrdatatrainable_params)rrr_prefixrrrrmakedirsr _configrrrrrinit_parameters_datarrritemsr rr=parameter_layout_dictrryr) save_objZcb_cur_ckpoint_fileZ cb_cur_file param_dictparam param_listkeyr each_param param_datarr`r)r*save_only_network_paramss<      z>CheckpointMointor._save_ckpt..save_only_network_params)4r|_last_triggered_step_need_flush_from_cache_flush_from_cache_check_save_ckptr1r{rrrr0r}_managerupdate_ckpoint_filelist _directoryrr ckpoint_numremove_oldest_ckpoint_filerr[_cur_time_for_keep_last_time_for_keepkeep_one_ckpoint_per_minutesSAVE_DIRrrrr rrrrexec_checkpoint_graph _append_dict_append_epoch_num_append_step_numrrrrwr8r9r:r;rr rrrr_latest_ckpt_file_name) r`r force_to_save save_ckptstep_num_in_epochcur_ckpoint_filecur_fileoutputsrrr)rr* _save_ckptsb    (            $zCheckpointMointor._save_ckpt)rNNrLrr5rTTFFNNNrF)F)r,r-r.r/rVr rr)r)rar*rs&5rcs2eZdZdZd fdd Zd d Zd d ZZS)rz' Profile analysis in training. rLrNTFc  stt|||_||_||_||_|r6|s6td|durXtt dd}t d|}t ddkrx|rxtdd}tf||||d ||_d|_||_dS) NzCWhen profile_communication is True, start_profile must also be TruernroprofilerGPUzXThe parameter profile_memory is not supported on GPU currently, so is changed to False. F) start_profile output_pathprofile_communicationprofile_memory)rUrrV start_step stop_steprrrDr1rrrr<rrrrprofilerr^r) r`rrrrrrrrrar)r*rV@s*  zProfileMonitor.__init__cCs,|}|j}||jkr(|js(|jdS)z Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. N)rvr|rrrstartr`r^rrr)r)r*rf[szProfileMonitor.step_begincCsB|}|j}||jkr>|j|jtd|j|jdS)z Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. zEnd of Profiling, please view the profile data under %s and analyze it using mindinsight.MindInsight order as follow: mindinsight start --summary-base-dir %sN) rvr|rrstopanalyserrrrr)r)r*rgs   zProfileMonitor.step_end)rLrNTFT)r,r-r.r/rVrfrrr)r)rar*r:s rc@s<eZdZdZdeeedddZddZd d Zd d Z d S)raEvaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. r r! eval_func step_intervalepoch_intervalcCs||_||_||_dSr'r)r`rrrr)r)r*rVszEvalCallBack.__init__cCs6|jdkrdS|}|j}||jdkr2|dSNr)rrvr} _execute_eval)r`r^callback_paramsr}r)r)r*res  zEvalCallBack.epoch_endcCs6|jdkrdS|}|j}||jdkr2|dSr)rrvr|r)r`r^r r|r)r)r*rs  zEvalCallBack.step_endcCs.t}|}t|}td||dS)Nz#Eval result: %s, eval time is %f s.)r[rrr)r` start_timerEZ eval_timer)r)r*rs zEvalCallBack._execute_evalN)r r!) r,r-r.r/rr1rVrerrr)r)r)r*rxs    r)9r/rrr[r collectionsrcopyrtypingrrrnumpyr> mindsporer<rrr r r r r mindspore.train.callbackr#mindspore.nn.learning_rate_scheduler"mindspore.train.callback._callbackrmindspore.train.serializationrmindformers.tools.registerrrZ-mindformers.tools.cloud_adapter.cloud_adapterrmindformers.tools.loggerrmindformers.tools.utilsrrrr__all__getcwd_cur_dirrregisterCALLBACKrrKrrrrrr)r)r)r*sJ  $       # %   = pengcheng.mind-mindformer/mindformers/core/callback/build_callback.py000066400000000000000000000063551476650141400264210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Callback API.""" import inspect from mindspore.train import callback from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_callback( config: [dict, list] = None, default_args: dict = None, module_type: str = 'callback', class_name: str = None, **kwargs): r"""Build callback For MindFormer. Instantiate the callback from MindFormerRegister's registry. Args: config (dict, list): The task callback's config. Default: None. default_args (dict): The default argument of callback API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'callback'. class_name (str): The class name of callback API. Default: None. Return: The function instance of callback API. Examples: >>> from mindformers import build_callback >>> callback_config = [{'type': 'MFLossMonitor'}, {'type': 'SummaryMonitor'}] >>> # 1) use config dict to build callback >>> callback_from_config = build_callback(callback_config) >>> # 2) use class name to build callback >>> callback_class_name = build_callback(class_name='MFLossMonitor') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_callback = config if not isinstance(cfg_callback, list): return MindFormerRegister.get_instance_from_cfg( cfg_callback, MindFormerModuleType.CALLBACK, default_args=default_args) callbacks = [] for callback_type in cfg_callback: if callback_type.get('type') == 'MFLossMonitor': callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK, default_args) else: callback_op = MindFormerRegister.get_instance_from_cfg( callback_type, MindFormerModuleType.CALLBACK) callbacks.append(callback_op) return callbacks return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_cb(): """ register MindSpore builtin LR class. """ for module_name in dir(callback): if module_name.startswith('__'): continue monitor = getattr(callback, module_name) if inspect.isclass(monitor): MindFormerRegister.register_cls( monitor, MindFormerModuleType.CALLBACK) register_ms_cb() pengcheng.mind-mindformer/mindformers/core/callback/callback.py000066400000000000000000000760741476650141400252470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Callback.""" import json import os import time import datetime from collections import OrderedDict from copy import deepcopy from typing import Callable, Optional, Union import numpy as np import mindspore as ms from mindspore import Callback, Profiler, ModelCheckpoint, CheckpointConfig, context, save_checkpoint, Tensor from mindspore.train.callback import SummaryCollector from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindspore.train.callback._callback import set_cur_net from mindspore.train.serialization import _get_merged_param_data from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.cloud_adapter.cloud_adapter import Local2ObsMonitor from mindformers.tools.logger import logger from mindformers.tools.utils import get_output_root_path, get_output_subpath, get_remote_save_url, check_in_modelarts __all__ = ['ObsMonitor', 'MFLossMonitor', 'CheckpointMointor', 'SummaryMonitor', 'ProfileMonitor', 'EvalCallBack'] _cur_dir = os.getcwd() SAVE_DIR = _cur_dir @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ObsMonitor: """Obs Monitor For AICC and Local""" def __new__(cls, src_dir: str = None, target_dir: str = None, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last: bool = True): if src_dir is None: src_dir = get_output_root_path() if target_dir is None: target_dir = get_remote_save_url() return Local2ObsMonitor(src_dir, target_dir, step_upload_frequence, epoch_upload_frequence, keep_last) def _get_loss_output(output): """Get output of task for MFLossMonitor.""" overflow = False scaling_sens = False loss = output learning_rate = None grad_norm = None if isinstance(output, (tuple, list)): if len(output) == 3: loss, overflow, scaling_sens = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 4: loss, overflow, scaling_sens, learning_rate = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() elif len(output) == 5: loss, overflow, scaling_sens, learning_rate, grad_norm = output if isinstance(scaling_sens, ms.Tensor): scaling_sens = scaling_sens.asnumpy() else: if isinstance(output[0], ms.Tensor) and isinstance(output[0].asnumpy(), np.ndarray): loss = output[0] if isinstance(loss, ms.Tensor) and isinstance(loss.asnumpy(), np.ndarray): loss = np.mean(loss.asnumpy()) # Boundary check. if isinstance(loss, float) and (np.isnan(loss) or np.isinf(loss)): invalid_loss_info = "NaN" if np.isnan(loss) else "Inf" raise ValueError(f"The current value of loss is {invalid_loss_info}, terminate training.") return loss, overflow, scaling_sens, learning_rate, grad_norm @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class MFLossMonitor(Callback): """ Loss Monitor for classification. Args: learning_rate (Union[float, LearningRateSchedule], optional): The learning rate schedule. Default: None. per_print_times (int): Every how many steps to print the log information. Default: 1. micro_batch_num (int): MicroBatch size for Pipeline Parallel. Default: 1. micro_batch_interleave_num (int): split num of batch size. Default: 1. origin_epochs (int): Training epoches. Default: None. dataset_size (int): Training dataset size. Default: None. Examples: >>> from mindformers.core.callback import MFLossMonitor >>> lr = [0.01, 0.008, 0.006, 0.005, 0.002] >>> monitor = MFLossMonitor(per_print_times=10) """ def __init__(self, learning_rate: Optional[Union[float, LearningRateSchedule]] = None, per_print_times: int = 1, micro_batch_num: int = 1, micro_batch_interleave_num: int = 1, origin_epochs: int = None, dataset_size: int = None, initial_epoch: int = 0, global_batch_size: int = 0, device_num: int = 0): super(MFLossMonitor, self).__init__() self.per_print_times = per_print_times self.learning_rate = deepcopy(learning_rate) self.last_print_time = 0 self.mirco_size = micro_batch_num self.print_warning_flag = True self.loss_list = [] self.step_time = time.time() self.epoch_time = time.time() self.run_context = None self.steps_per_epoch = dataset_size self.micro_batch_interleave_num = micro_batch_interleave_num self.origin_epochs = origin_epochs self.initial_epoch = initial_epoch self.global_batch_size = global_batch_size self.device_num = device_num def epoch_begin(self, run_context): """ Record time at the beginning of epoch. Args: run_context (RunContext): Context of the process running. """ self.loss_list = [] self.epoch_time = time.time() self.run_context = run_context def epoch_end(self, run_context): """ Print training info at the end of epoch. Args: run_context (RunContext): Context of the process running. """ def step_begin(self, run_context): """ Record time at the beginning of step. Args: run_context (RunContext): Context of the process running. """ self.step_time = time.time() self.run_context = run_context def step_end(self, run_context): """ Print training info at the end of step. Args: run_context (RunContext): Context of the process running. """ parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") auto_parallel = parallel_mode in ['semi_auto_parallel', 'auto_parallel'] if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) cb_params = run_context.original_args() step_seconds = (time.time() - self.step_time) * 1000 net_outputs = cb_params.net_outputs loss, overflow, scaling_sens, learning_rate, grad_norm = _get_loss_output(net_outputs) if learning_rate is not None: self.learning_rate = learning_rate loss = self._fix_loss_for_parallel(loss) self.loss_list.append(loss) if not overflow: overflow = "False" if not scaling_sens: scaling_sens = "unavailable" if cb_params.dataset_sink_mode: origin_epochs = self.origin_epochs per_step_seconds = step_seconds / cb_params.batch_num steps_per_epoch = self.steps_per_epoch cur_epoch_num = (cb_params.cur_step_num - 1) // steps_per_epoch \ + self.initial_epoch * cb_params.batch_num // steps_per_epoch + 1 cur_step_num = (cb_params.cur_step_num - 1) % steps_per_epoch + 1 else: origin_epochs = self.origin_epochs per_step_seconds = step_seconds steps_per_epoch = cb_params.batch_num cur_epoch_num = cb_params.cur_epoch_num cur_step_num = (cb_params.cur_step_num - 1) % cb_params.batch_num + 1 # compute time remaining # step_remain = (origin_epochs - cur_epoch_num + 1) * steps_per_epoch - cur_step_num # time_remain = step_remain * per_step_seconds / 1000 time_remain = (steps_per_epoch - cur_step_num) * per_step_seconds / 1000 # compute throughput throughput = self.global_batch_size / self.device_num / (per_step_seconds / 1000) # compute percent # percent = ((cur_epoch_num - 1) * steps_per_epoch + cur_step_num) / origin_epochs / steps_per_epoch * 100 percent = cur_step_num / steps_per_epoch * 100 if (cb_params.cur_step_num - self.last_print_time) >= self.per_print_times: self.last_print_time = cb_params.cur_step_num self.print_output_info(cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm) if check_in_modelarts() and int(os.getenv("RANK_ID", "0")) == int(os.getenv("RANK_SIZE", "1")) - 1: self.dump_info_to_modelarts(ma_step_num=cur_step_num, ma_loss=loss) if auto_parallel: ms.context.set_auto_parallel_context(parallel_mode=parallel_mode, full_batch=full_batch) def _fix_loss_for_parallel(self, loss): """Fix loss value in pipeline or double parallel mode.""" pipeline_stages = ms.context.get_auto_parallel_context("pipeline_stages") if pipeline_stages > 1 and self.print_warning_flag: logger.warning("pipeline stages: %s > 1, the loss on the last card is valid.", pipeline_stages) if self.micro_batch_interleave_num > 1 and self.print_warning_flag: logger.warning("micro_batch_interleave_num: %s > 1, multiple copies in parallel is open.") if pipeline_stages > 1: loss = loss / (self.mirco_size * self.micro_batch_interleave_num) elif self.micro_batch_interleave_num > 1: loss = loss / self.micro_batch_interleave_num return loss def print_output_info(self, cb_params, cur_epoch_num, origin_epochs, throughput, cur_step_num, steps_per_epoch, loss, per_step_seconds, overflow, scaling_sens, time_remain, percent, grad_norm): """print output information.""" if self.learning_rate is not None: if isinstance(self.learning_rate, (float, Tensor)): current_lr = str(self.learning_rate) elif isinstance(self.learning_rate, LearningRateSchedule): if ms.context.get_context('device_target') == 'CPU': if self.print_warning_flag: logger.warning( "device target not support CPU when generating the learning rate value, " "please use: mindspore.context.set_context(device_target='Ascend')") self.print_warning_flag = False current_lr = None else: if cb_params.optimizer is not None: global_step = cb_params.optimizer.global_step else: global_step = cb_params.network.optimizer.global_step # temporary set_train to avoid error on 910B origin_phase = cb_params.train_network.phase cb_params.train_network.set_train(False) current_lr = self.learning_rate(global_step) cb_params.train_network.set_train(origin_phase) current_lr = np.array2string(current_lr.asnumpy()) else: if self.print_warning_flag: logger.warning( "The current learning rate cannot be calculated in real time." "Only the type of LearningRateSchedule is supported in the callback of MFLossMonitor," "but the input learning rate function type is %s", type(self.learning_rate) ) self.print_warning_flag = False current_lr = None else: if self.print_warning_flag: logger.warning( "MFLossMonitor callback is not set learning rate arguments." "To display the learning rate, you must input the arguments, " "which can be LearningRateSchedule or a fixed float" ) self.print_warning_flag = False current_lr = None if current_lr is not None: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, lr: %s, overflow cond: %s, loss_scale: %s, grad_norm: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), current_lr, overflow, scaling_sens, grad_norm) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) else: if cb_params.dataset_sink_mode: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss: %5.3f, " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, int(per_step_seconds), overflow, scaling_sens) else: logger.info("{ Epoch:[%3d/%3d], step:[%5d/%5d], loss:[%5.3f/%5.3f], " "per_step_time: %dms, overflow cond: %s, loss_scale: %s", cur_epoch_num, origin_epochs, cur_step_num, steps_per_epoch, loss, np.mean(self.loss_list), int(per_step_seconds), overflow, scaling_sens) show_str = ('|%%-%ds|' % 50) % (int(50 * percent / 100) * "▮") logger.info(" %4.1f%% %s %.2f samples/s/p %s }", percent, show_str, throughput, datetime.timedelta(seconds=int(time_remain))) def dump_info_to_modelarts(self, ma_step_num, ma_loss): """dump modelarts info to display evaluation result page""" ma_loss = float(ma_loss) obj = None modelarts_dir = os.path.join(get_output_root_path(), "modelarts") if not os.path.exists(modelarts_dir): os.mkdir(modelarts_dir) if not os.path.exists(os.path.join(modelarts_dir, "model_analysis_results.json")): obj = { "en-us": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "loss of model", "value": {"current_loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} }, "zh-cn": { "common": {}, "precision_performance": { "pr": { "title": "loss", "description": "模型损失", "value": {"当前loss": 0}, "line_chart": { "pr_line_chart": { "name": "loss line chart of model", "x_axis_name": "step", "y_axis_name": "loss", "curve": {"loss": []}}}}}, "feature_sensitivity": {}, "computational_performance": {}, "abstract_feature": {}, "adversary": {} } } else: with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "r") as fp: obj = json.load(fp) if obj is not None: en_precision_performance = obj["en-us"]["precision_performance"] en_precision_performance["pr"]["value"]["loss_value"] = ma_loss en_loss_list = en_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] en_loss_list.append([ma_step_num, ma_loss]) zh_precision_performance = obj["zh-cn"]["precision_performance"] zh_precision_performance["pr"]["value"]["当前loss"] = ma_loss zh_loss_list = zh_precision_performance["pr"]["line_chart"]["pr_line_chart"]["curve"]["loss"] zh_loss_list.append([ma_step_num, ma_loss]) with open(os.path.join(modelarts_dir, "model_analysis_results.json"), "w") as fp: json.dump(obj, fp) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class SummaryMonitor: """Summary Monitor For AICC and Local""" def __new__(cls, summary_dir=None, collect_freq=10, collect_specified_data=None, keep_default_action=True, custom_lineage_data=None, collect_tensor_freq=None, max_file_size=None, export_options=None): if summary_dir is None: rank_id = int(os.getenv("RANK_ID", '0')) summary_dir = get_output_subpath('summary', rank_id) kwargs = { "summary_dir": summary_dir, "collect_freq": collect_freq, "collect_specified_data": collect_specified_data, "keep_default_action": keep_default_action, "custom_lineage_data": custom_lineage_data, "collect_tensor_freq": collect_tensor_freq, "max_file_size": max_file_size, "export_options": export_options } return SummaryCollector(**kwargs) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class CheckpointMointor(ModelCheckpoint): """Checkpoint Monitor For Save LossScale""" def __init__(self, prefix='CKP', directory=None, config=None, save_checkpoint_steps=1, save_checkpoint_seconds=0, keep_checkpoint_max=5, keep_checkpoint_per_n_minutes=0, integrated_save=True, save_network_params=True, save_trainable_params=False, async_save=False, saved_network=None, append_info=None, enc_key=None, enc_mode='AES-GCM', exception_save=False): self.config = config self.save_network_params = save_network_params self.save_trainable_params = save_trainable_params self.rank_id = int(os.getenv("RANK_ID", '0')) prefix = prefix + "_rank_{}".format(self.rank_id) if append_info is None: append_info = [{ "epoch_num": 0, "step_num": 0, "global_step": 0, "loss_scale": 1 }] ckpt_directory = os.path.join(directory, f"checkpoint/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint', self.rank_id) self.network_directory = os.path.join(directory, f"checkpoint_network/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_network', self.rank_id) self.trainable_directory = os.path.join(directory, f"checkpoint_trainable/rank_{self.rank_id}") \ if directory else get_output_subpath('checkpoint_trainable', self.rank_id) if context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: logger.info("Integrated_save is changed to False when using auto_parallel.") integrated_save = False config_ck = CheckpointConfig(save_checkpoint_steps=save_checkpoint_steps, save_checkpoint_seconds=save_checkpoint_seconds, keep_checkpoint_max=keep_checkpoint_max, keep_checkpoint_per_n_minutes=keep_checkpoint_per_n_minutes, integrated_save=integrated_save, async_save=async_save, saved_network=saved_network, append_info=append_info, enc_key=enc_key, enc_mode=enc_mode, exception_save=exception_save) super(CheckpointMointor, self).__init__(prefix, ckpt_directory, config=config_ck) def _save_ckpt(self, cb_params, force_to_save=False): """Save checkpoint files.""" # pylint: disable=E0203 if cb_params.cur_step_num == self._last_triggered_step: return # if param is cache enable, flush data from cache to host before save_ckpt if self._need_flush_from_cache: self._flush_from_cache(cb_params) save_ckpt = self._check_save_ckpt(cb_params, force_to_save) step_num_in_epoch = int((cb_params.cur_step_num - 1) % cb_params.batch_num + 1) if save_ckpt: logger.info('......Saving ckpt......') cur_ckpoint_file = self._prefix + "-" + str(cb_params.cur_epoch_num) + "_" \ + str(step_num_in_epoch) + ".ckpt" # update checkpoint file list. self._manager.update_ckpoint_filelist(self._directory, self._prefix) # keep checkpoint files number equal max number. if self._config.keep_checkpoint_max and 0 < self._config.keep_checkpoint_max <= self._manager.ckpoint_num: self._manager.remove_oldest_ckpoint_file() elif self._config.keep_checkpoint_per_n_minutes and self._config.keep_checkpoint_per_n_minutes > 0: # pylint: disable=E0203 self._cur_time_for_keep = time.time() if (self._cur_time_for_keep - self._last_time_for_keep) \ < self._config.keep_checkpoint_per_n_minutes * 60: self._manager.keep_one_ckpoint_per_minutes(self._config.keep_checkpoint_per_n_minutes, self._cur_time_for_keep) # generate the new checkpoint file and rename it. global SAVE_DIR SAVE_DIR = self._directory cur_file = os.path.join(self._directory, cur_ckpoint_file) self._last_time_for_keep = time.time() self._last_triggered_step = cb_params.cur_step_num if context.get_context("enable_ge") and os.getenv("MS_ENABLE_REF_MODE", "0") == "0": set_cur_net(cb_params.train_network) cb_params.train_network.exec_checkpoint_graph() if "epoch_num" in self._append_dict: self._append_dict["epoch_num"] = self._append_epoch_num + cb_params.cur_epoch_num if "step_num" in self._append_dict: self._append_dict["step_num"] = self._append_step_num + cb_params.cur_epoch_num * cb_params.batch_num if cb_params.optimizer is not None: self._append_dict["global_step"] = cb_params.optimizer.global_step else: self._append_dict["global_step"] = cb_params.network.optimizer.global_step if "loss_scale" in self._append_dict: outputs = cb_params.net_outputs if isinstance(outputs, (tuple, list)) and len(outputs) >= 3: self._append_dict["loss_scale"] = outputs[2] network = self._config.saved_network if self._config.saved_network is not None else cb_params.train_network save_checkpoint(network, cur_file, self._config.integrated_save, self._config.async_save, self._append_dict, self._config.enc_key, self._config.enc_mode) def save_only_network_params(): save_obj = cb_params.network if save_obj.optimizer is not None: save_obj = save_obj.network if self.save_network_params: cb_cur_ckpoint_file = self._prefix + "-" + "network" + ".ckpt" cb_cur_file = os.path.join(self.network_directory, cb_cur_ckpoint_file) os.makedirs(self.network_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) if self.save_trainable_params: save_obj.init_parameters_data() param_dict = OrderedDict() for param in save_obj.trainable_params(): param_dict[param.name] = param param_list = [] for (key, value) in param_dict.items(): each_param = {"name": key} param_data = Tensor(value.data.asnumpy()) # in automatic model parallel scenario, some parameters were split to all the devices, # which should be combined before saving if key in save_obj.parameter_layout_dict: param_data = _get_merged_param_data(save_obj, key, param_data, self._config.integrated_save) each_param["data"] = param_data param_list.append(each_param) save_obj = param_list cb_cur_ckpoint_file = self._prefix + "-" + "trainable_params" + ".ckpt" cb_cur_file = os.path.join(self.trainable_directory, cb_cur_ckpoint_file) os.makedirs(self.trainable_directory, exist_ok=True) save_checkpoint(save_obj, cb_cur_file, self._config.integrated_save, self._config.async_save, {}, self._config.enc_key, self._config.enc_mode) save_only_network_params() self._latest_ckpt_file_name = cur_file @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class ProfileMonitor(Callback): """ Profile analysis in training. """ def __init__(self, start_step=1, stop_step=10, output_path=None, start_profile=True, profile_communication=False, profile_memory=True, **kwargs): super(ProfileMonitor, self).__init__() self.start_step = start_step self.stop_step = stop_step self.start_profile = start_profile self.profile_communication = profile_communication if profile_communication and not start_profile: raise ValueError("When profile_communication is True, start_profile must also be True") if output_path is None: rank_id = int(os.getenv("RANK_ID", '0')) output_path = get_output_subpath('profile', rank_id) if ms.get_context("device_target") == "GPU" and profile_memory: logger.warning("The parameter profile_memory is not supported on GPU currently, so is changed to False. ") profile_memory = False self.profiler = Profiler( start_profile=start_profile, output_path=output_path, profile_communication=profile_communication, profile_memory=profile_memory, **kwargs) self.run_context = None self.output_path = output_path def step_begin(self, run_context): """ Start profile at the begin of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.start_step and not self.start_profile: self.profiler.start() def step_end(self, run_context): """ Stop profile at the end of step. Args: run_context (RunContext): Context of the train running. """ cb_params = run_context.original_args() step_num = cb_params.cur_step_num if step_num == self.stop_step: self.profiler.stop() self.profiler.analyse() logger.info("End of Profiling, please view the profile data under %s and analyze it using mindinsight." "MindInsight order as follow: " "mindinsight start --summary-base-dir %s", self.output_path, self.output_path) @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class EvalCallBack(Callback): """Evaluate Callback used in training progress. Args: eval_func (Callable): the function to calculate eval result, task specific. step_interval (int): determine the num of step intervals between each eval. Default -1, means only eval on epoch end, do not eval between steps. Note that it will not take effects when running in data sink mode. epoch_interval (int): determine the num of epoch intervals between each eval. Default 1, means eval on every epoch end. """ def __init__(self, eval_func: Callable, step_interval: int = 100, epoch_interval: int = -1): self.eval_func = eval_func self.step_interval = step_interval self.epoch_interval = epoch_interval def epoch_end(self, run_context): # if not use epoch end if self.epoch_interval <= 0: return callback_params = run_context.original_args() cur_epoch_num = callback_params.cur_epoch_num if cur_epoch_num % self.epoch_interval == 0: self._execute_eval() def step_end(self, run_context): # if not use step end if self.step_interval <= 0: return callback_params = run_context.original_args() cur_step_num = callback_params.cur_step_num if cur_step_num % self.step_interval == 0: self._execute_eval() def _execute_eval(self): start_time = time.time() output = self.eval_func() eval_time = time.time() - start_time logger.info("Eval result: %s, eval time is %f s.", output, eval_time) pengcheng.mind-mindformer/mindformers/core/clip_grad.py000066400000000000000000000072451476650141400236750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Operations for clipping grad by global norm.""" import numpy as np from mindspore import ops from mindspore.nn.cell import Cell from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.common.tensor import Tensor from mindspore.common import dtype as mstype # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel # The attribute grad_scale is needed for enabling the parallel mode # If this is removed, c.clip_by_global_norm will have precision error in semi/auto parallel mode. expand_dims = P.ExpandDims().add_prim_attr("grad_scale", True) get_square_sum = C.MultitypeFuncGraph("get_square_sum") apply_global_norm = C.MultitypeFuncGraph("apply_global_norm") @get_square_sum.register("Tensor") def _get_square_sum(x): norm = P.ReduceSum(False)(F.square(x), ()) norm = expand_dims(F.cast(norm, mstype.float32), 0) return norm @apply_global_norm.register("Tensor", "Tensor", "Tensor") def _apply_global_norm(clip_norm, global_norm, x): x_dtype = F.dtype(x) clip_coef = clip_norm / (global_norm + 1e-6) clip_coef_clamped = ops.clip_by_value(clip_coef, clip_value_max=Tensor(1.0, mstype.float32), clip_value_min=Tensor(np.log(0.0), mstype.float32)) x = x * clip_coef_clamped x = F.cast(x, x_dtype) return x class ClipGradNorm(Cell): r""" Clips tensor values by the ratio of the sum of their norms. Args: max_norm (Union(float, int)): The clipping ratio. Default: 1.0 use_norm (Union(float, None)): The global norm. Default: None Inputs: - **x** (Union(tuple[Tensor], list[Tensor])) - Input data to clip. Outputs: Tensor, a clipped Tensor. """ def __init__(self, max_norm=1.0, use_norm=None): super(ClipGradNorm, self).__init__() # Add interface. This parameter is not used at present if use_norm is not None: raise ValueError(f"For '{self.cls_name}', input 'use_norm' only supports None currently, " f"but got 'use_norm': {use_norm}") validator.check_number("clip_norm", max_norm, 0.0, Rel.GT, self.cls_name) self.clip_norm = Tensor([max_norm], mstype.float32) self.hyper_map = C.HyperMap() self.greater_equal = P.GreaterEqual() def construct(self, x): """clip grad.""" square_sum = self.hyper_map(get_square_sum, x) global_norm = F.sqrt(F.addn(square_sum)) origin_global_norm = global_norm cond = self.greater_equal(global_norm, self.clip_norm) global_norm = F.select(cond, global_norm, self.clip_norm) clip_x = self.hyper_map(F.partial(apply_global_norm, self.clip_norm, global_norm), x) return clip_x, origin_global_norm pengcheng.mind-mindformer/mindformers/core/context/000077500000000000000000000000001476650141400230535ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/context/.ipynb_checkpoints/000077500000000000000000000000001476650141400266445ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/context/.ipynb_checkpoints/build_context-checkpoint.py000066400000000000000000000162321476650141400342120ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build context.""" import os from mindspore import context from mindspore.communication.management import init, get_group_size, get_rank from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers.core.callback import ProfileMonitor from mindformers.trainer.config_args import ContextConfig, ParallelContextConfig from mindformers.tools.register import MindFormerConfig from mindformers.tools import PARALLEL_MODE, MODE, get_output_subpath,\ get_output_root_path, check_in_modelarts from mindformers.tools.logger import logger CONTEXT_CONFIG = { 'mode': 'GRAPH_MODE', 'device_target': 'Ascend', 'device_id': 0, 'save_graphs': False} PARALLEL_CONFIG = {'parallel_mode': 'DATA_PARALLEL', 'gradients_mean': True} def build_context(config): """Build context.""" if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.use_parallel and config.parallel_config.pipeline_stage > 1: config.parallel.pipeline_stages = config.parallel_config.pipeline_stage local_rank, device_num = init_context(use_parallel=config.use_parallel, context_config=config.context, parallel_config=config.parallel) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() config.device_num = device_num config.local_rank = local_rank if config.parallel.get("strategy_ckpt_load_file"): context.set_auto_parallel_context(strategy_ckpt_load_file=config.parallel.strategy_ckpt_load_file) def init_context(use_parallel=True, context_config=None, parallel_config=None): """Context initialization for MindSpore.""" if isinstance(context_config, ContextConfig): context_config = context_config.__dict__ if isinstance(parallel_config, ParallelContextConfig): parallel_config = parallel_config.__dict__ if context_config is None: context_config = CONTEXT_CONFIG if parallel_config is None: parallel_config = PARALLEL_CONFIG _set_check_context_config(context_config) _set_check_parallel_config(parallel_config) device_num = 1 rank_id = 0 context_config['mode'] = MODE.get(context_config.get('mode')) context.set_context(max_device_memory=context_config.get('max_device_memory'), mode=context_config.get('mode')) del context_config['mode'] del context_config['max_device_memory'] if use_parallel: init() device_id = int(os.getenv('DEVICE_ID', '0')) # 0 ~ 7 rank_id = get_rank() # local_rank device_num = get_group_size() # world_size context_config['device_id'] = device_id parallel_config['parallel_mode'] = PARALLEL_MODE.get(parallel_config.get('parallel_mode')) parallel_config.setdefault('device_num', device_num) context.set_context(**context_config) context.reset_auto_parallel_context() context.set_auto_parallel_context(**parallel_config) else: context.set_context(**context_config) return rank_id, device_num def build_profile_cb(config): """build profile callback from config.""" start_profile = config.init_start_profile profile_communication = config.profile_communication if config.device_num > 1: logger.info("Device number is %s > 1, so profile_communication and start_profile will be set True ") start_profile = True profile_communication = True profile_cb = ProfileMonitor( start_step=config.profile_start_step, stop_step=config.profile_stop_step, start_profile=start_profile, profile_communication=profile_communication, profile_memory=config.profile_memory) logger.warning( "In profiler mode, data sink mode will be turned off. " "Please reduce the data sample size with 'num_samples' in MindSpore data format according to " "https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.") logger.warning("In profiler mode, auto-tune will be turned off.") config.runner_config.sink_mode = False config.auto_tune = False return profile_cb def _set_check_context_config(config): """Set context config.""" mode = config.get('mode') if mode is None: config.setdefault('mode', 0) if mode not in MODE.keys(): raise IndexError('Running mode should be in {}, but get {}'.format(MODE.keys(), mode)) device = config.get('device_id') if device is None: config.setdefault('device_id', 0) max_device_memory = config.get('max_device_memory') if max_device_memory is None: config.setdefault('max_device_memory', '30GB') save_graph = config.get('save_graphs') if save_graph: save_graphs_path = config.get('save_graphs_path') if not check_in_modelarts() else None if save_graphs_path is None: save_graphs_path = get_output_subpath("debug/graphs_info", append_rank=False) config['save_graphs_path'] = save_graphs_path enable_dump = config.get('enable_dump') if enable_dump: save_dump_path = config.get('save_dump_path') if not check_in_modelarts() else None if save_dump_path is None: save_dump_path = get_output_subpath("debug/dump_info", append_rank=False) config['save_dump_path'] = save_dump_path def _set_check_parallel_config(config): """Set parallel config.""" parallel_mode = config.get('parallel_mode') if parallel_mode is None: config.setdefault('parallel_mode', 0) if PARALLEL_MODE.get(config.get('parallel_mode')) not in \ [context.ParallelMode.SEMI_AUTO_PARALLEL, context.ParallelMode.AUTO_PARALLEL] and config.get('full_batch'): logger.info("full_batch will be forced to False when the parallel mode is stand_alone or data_parallel") config.setdefault('full_batch', False) if parallel_mode not in PARALLEL_MODE.keys(): raise IndexError( 'Running parallel mode should be in {}, but get {}'.format(PARALLEL_MODE.keys(), parallel_mode)) strategy_ckpt_save_file = config.get('strategy_ckpt_save_file', "ckpt_strategy.ckpt") rank_id = int(os.getenv("RANK_ID", "0")) os.makedirs(os.path.join(get_output_root_path(), "strategy"), exist_ok=True) config["strategy_ckpt_save_file"] = os.path.join(get_output_root_path(), "strategy", strategy_ckpt_save_file.replace(".ckpt", f"_rank_{rank_id}.ckpt")) pengcheng.mind-mindformer/mindformers/core/context/__init__.py000066400000000000000000000015011476650141400251610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers context.""" from .build_context import build_context, build_profile_cb, init_context __all__ = ['build_context', 'build_profile_cb', 'init_context'] pengcheng.mind-mindformer/mindformers/core/context/__pycache__/000077500000000000000000000000001476650141400252635ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/context/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004651476650141400314600ustar00rootroot00000000000000a ?cfA@s$dZddlmZmZmZgdZdS)zMindFormers context.) build_contextbuild_profile_cb init_contextN)__doc__rrr__all__rrL/home/ma-user/work/mindformers-baichuan/mindformers/core/context/__init__.pyspengcheng.mind-mindformer/mindformers/core/context/__pycache__/build_context.cpython-39.pyc000066400000000000000000000114411476650141400325600ustar00rootroot00000000000000a ?cf@sdZddlZddlmZddlmZmZmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZdd lmZmZmZmZmZdd lmZd d dddZdddZddZdddZddZddZ ddZ!dS)Build context.N)context)initget_group_sizeget_rank)set_algo_parameters)_set_multi_subgraphs)ProfileMonitor) ContextConfigParallelContextConfig)MindFormerConfig) PARALLEL_MODEMODEget_output_subpathget_output_root_pathcheck_in_modelarts)logger GRAPH_MODEAscendF)mode device_target device_id save_graphs DATA_PARALLELT) parallel_modegradients_meancCst|tr"t|ts"tfi|}|jr@|jjdkr@|jj|j_t|j|j |jd\}}t dddt ||_ ||_ |jdrt j|jjddS)r) use_parallelcontext_configparallel_configT)elementwise_op_strategy_followfully_use_devicesstrategy_ckpt_load_file)r"N) isinstancedictr rrpipeline_stageparallelpipeline_stages init_contextrrr device_num local_rankgetset_auto_parallel_contextr")configr*r)r.Q/home/ma-user/work/mindformers-baichuan/mindformers/core/context/build_context.py build_context$s    r0cCst|tr|j}t|tr |j}|dur,t}|dur8t}t|t|d}d}t | d|d<t j | d| dd|d=|d=|rt t tdd}t}t}||d <t | d |d <|d |t j fi|t t jfi|nt j fi|||fS) z%Context initialization for MindSpore.Nrrrmax_device_memory)r1r DEVICE_ID0rrr))r#r __dict__r CONTEXT_CONFIGPARALLEL_CONFIG_set_check_context_config_set_check_parallel_configrr+r set_contextrintosgetenvrrr setdefaultreset_auto_parallel_contextr,)rrrr)rank_idrr.r.r/r(6s>    r(cCsf|j}|j}|jdkr(tdd}d}t|j|j|||jd}t dt dd|j _ d|_ |S)z#build profile callback from config.rzUDevice number is %s > 1, so profile_communication and start_profile will be set True T) start_step stop_step start_profileprofile_communicationprofile_memoryzIn profiler mode, data sink mode will be turned off. Please reduce the data sample size with 'num_samples' in MindSpore data format according to https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.z/In profiler mode, auto-tune will be turned off.F) Zinit_start_profilerCr)rinfor Zprofile_start_stepZprofile_stop_steprDwarning runner_config sink_mode auto_tune)r-rBrC profile_cbr.r.r/build_profile_cb^s(   rKcCs|d}|dur|dd|tvr>tdt||d}|dur\|dd|d}|durz|dd|d}|rts|d nd}|durtd d d }||d <|d }|rts|dnd}|durtdd d }||d<dS)zSet context config.rNrz(Running mode should be in {}, but get {}rr1Z30GBrsave_graphs_pathzdebug/graphs_infoF) append_rank enable_dumpsave_dump_pathzdebug/dump_info)r+r=rkeys IndexErrorformatrr)r-rdevicer1Z save_graphrLrNrOr.r.r/r7ws.           r7c Cs|d}|dur|ddt|dtjjtjjfvr^|dr^td|dd|t vr~t d t ||dd }t t d d }t jt jtd d dt jtd |dd|d|d<dS)zSet parallel config.rNr full_batchzYfull_batch will be forced to False when the parallel mode is stand_alone or data_parallelFz1Running parallel mode should be in {}, but get {}strategy_ckpt_save_filezckpt_strategy.ckptRANK_IDr3strategyT)exist_okz.ckpt_rank_)r+r=r r ParallelModeSEMI_AUTO_PARALLEL AUTO_PARALLELrrErPrQrRr:r;r<makedirspathjoinrreplace)r-rrUr?r.r.r/r8s(       r8)TNN)"__doc__r; mindsporer"mindspore.communication.managementrrrZmindspore.parallelrZ&mindspore.parallel._cost_model_contextrZmindformers.core.callbackr Zmindformers.trainer.config_argsr r mindformers.tools.registerr mindformers.toolsr rrrrmindformers.tools.loggerrr5r6r0r(rKr7r8r.r.r.r/s$        (pengcheng.mind-mindformer/mindformers/core/context/build_context.py000066400000000000000000000162321476650141400262740ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build context.""" import os from mindspore import context from mindspore.communication.management import init, get_group_size, get_rank from mindspore.parallel import set_algo_parameters from mindspore.parallel._cost_model_context import _set_multi_subgraphs from mindformers.core.callback import ProfileMonitor from mindformers.trainer.config_args import ContextConfig, ParallelContextConfig from mindformers.tools.register import MindFormerConfig from mindformers.tools import PARALLEL_MODE, MODE, get_output_subpath,\ get_output_root_path, check_in_modelarts from mindformers.tools.logger import logger CONTEXT_CONFIG = { 'mode': 'GRAPH_MODE', 'device_target': 'Ascend', 'device_id': 0, 'save_graphs': False} PARALLEL_CONFIG = {'parallel_mode': 'DATA_PARALLEL', 'gradients_mean': True} def build_context(config): """Build context.""" if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.use_parallel and config.parallel_config.pipeline_stage > 1: config.parallel.pipeline_stages = config.parallel_config.pipeline_stage local_rank, device_num = init_context(use_parallel=config.use_parallel, context_config=config.context, parallel_config=config.parallel) set_algo_parameters(elementwise_op_strategy_follow=True, fully_use_devices=True) _set_multi_subgraphs() config.device_num = device_num config.local_rank = local_rank if config.parallel.get("strategy_ckpt_load_file"): context.set_auto_parallel_context(strategy_ckpt_load_file=config.parallel.strategy_ckpt_load_file) def init_context(use_parallel=True, context_config=None, parallel_config=None): """Context initialization for MindSpore.""" if isinstance(context_config, ContextConfig): context_config = context_config.__dict__ if isinstance(parallel_config, ParallelContextConfig): parallel_config = parallel_config.__dict__ if context_config is None: context_config = CONTEXT_CONFIG if parallel_config is None: parallel_config = PARALLEL_CONFIG _set_check_context_config(context_config) _set_check_parallel_config(parallel_config) device_num = 1 rank_id = 0 context_config['mode'] = MODE.get(context_config.get('mode')) context.set_context(max_device_memory=context_config.get('max_device_memory'), mode=context_config.get('mode')) del context_config['mode'] del context_config['max_device_memory'] if use_parallel: init() device_id = int(os.getenv('DEVICE_ID', '0')) # 0 ~ 7 rank_id = get_rank() # local_rank device_num = get_group_size() # world_size context_config['device_id'] = device_id parallel_config['parallel_mode'] = PARALLEL_MODE.get(parallel_config.get('parallel_mode')) parallel_config.setdefault('device_num', device_num) context.set_context(**context_config) context.reset_auto_parallel_context() context.set_auto_parallel_context(**parallel_config) else: context.set_context(**context_config) return rank_id, device_num def build_profile_cb(config): """build profile callback from config.""" start_profile = config.init_start_profile profile_communication = config.profile_communication if config.device_num > 1: logger.info("Device number is %s > 1, so profile_communication and start_profile will be set True ") start_profile = True profile_communication = True profile_cb = ProfileMonitor( start_step=config.profile_start_step, stop_step=config.profile_stop_step, start_profile=start_profile, profile_communication=profile_communication, profile_memory=config.profile_memory) logger.warning( "In profiler mode, data sink mode will be turned off. " "Please reduce the data sample size with 'num_samples' in MindSpore data format according to " "https://www.mindspore.cn/mindinsight/docs/zh-CN/master/performance_profiling_ascend.html.") logger.warning("In profiler mode, auto-tune will be turned off.") config.runner_config.sink_mode = False config.auto_tune = False return profile_cb def _set_check_context_config(config): """Set context config.""" mode = config.get('mode') if mode is None: config.setdefault('mode', 0) if mode not in MODE.keys(): raise IndexError('Running mode should be in {}, but get {}'.format(MODE.keys(), mode)) device = config.get('device_id') if device is None: config.setdefault('device_id', 0) max_device_memory = config.get('max_device_memory') if max_device_memory is None: config.setdefault('max_device_memory', '30GB') save_graph = config.get('save_graphs') if save_graph: save_graphs_path = config.get('save_graphs_path') if not check_in_modelarts() else None if save_graphs_path is None: save_graphs_path = get_output_subpath("debug/graphs_info", append_rank=False) config['save_graphs_path'] = save_graphs_path enable_dump = config.get('enable_dump') if enable_dump: save_dump_path = config.get('save_dump_path') if not check_in_modelarts() else None if save_dump_path is None: save_dump_path = get_output_subpath("debug/dump_info", append_rank=False) config['save_dump_path'] = save_dump_path def _set_check_parallel_config(config): """Set parallel config.""" parallel_mode = config.get('parallel_mode') if parallel_mode is None: config.setdefault('parallel_mode', 0) if PARALLEL_MODE.get(config.get('parallel_mode')) not in \ [context.ParallelMode.SEMI_AUTO_PARALLEL, context.ParallelMode.AUTO_PARALLEL] and config.get('full_batch'): logger.info("full_batch will be forced to False when the parallel mode is stand_alone or data_parallel") config.setdefault('full_batch', False) if parallel_mode not in PARALLEL_MODE.keys(): raise IndexError( 'Running parallel mode should be in {}, but get {}'.format(PARALLEL_MODE.keys(), parallel_mode)) strategy_ckpt_save_file = config.get('strategy_ckpt_save_file', "ckpt_strategy.ckpt") rank_id = int(os.getenv("RANK_ID", "0")) os.makedirs(os.path.join(get_output_root_path(), "strategy"), exist_ok=True) config["strategy_ckpt_save_file"] = os.path.join(get_output_root_path(), "strategy", strategy_ckpt_save_file.replace(".ckpt", f"_rank_{rank_id}.ckpt")) pengcheng.mind-mindformer/mindformers/core/loss/000077500000000000000000000000001476650141400223475ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/loss/.ipynb_checkpoints/000077500000000000000000000000001476650141400261405ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/loss/.ipynb_checkpoints/build_loss-checkpoint.py000066400000000000000000000047751476650141400330130ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Loss API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_loss( config: dict = None, default_args: dict = None, module_type: str = 'loss', class_name: str = None, **kwargs): r"""Build loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LOSS, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_loss(): """ register MindSpore builtin loss class. """ for module_name in dir(nn.loss): if module_name.startswith('__'): continue loss = getattr(nn.loss, module_name) if inspect.isclass(loss): MindFormerRegister.register_cls( loss, MindFormerModuleType.LOSS) register_ms_loss() pengcheng.mind-mindformer/mindformers/core/loss/.ipynb_checkpoints/loss-checkpoint.py000066400000000000000000000412271476650141400316250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Loss.""" from mindspore import nn, Tensor from mindspore import ops as P from mindspore.ops import functional as F from mindspore.common import dtype as mstype from mindspore.nn.loss.loss import LossBase from mindspore.context import ParallelMode from mindspore.parallel import set_algo_parameters from mindspore import log as logger from mindspore.parallel._utils import _get_device_num, _get_pipeline_stages, _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import _LogActionOnce from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.op_parallel_config import default_dpmp_config __all__ = ['SoftTargetCrossEntropy', 'MSELoss', 'L1Loss', 'CrossEntropyLoss', 'CompareLoss'] @MindFormerRegister.register(MindFormerModuleType.LOSS) class SoftTargetCrossEntropy(LossBase): """SoftTargetCrossEntropy for MixUp Augment.""" def __init__(self, parallel_config=default_dpmp_config): super(SoftTargetCrossEntropy, self).__init__() dp = parallel_config.data_parallel self.mean_ops = P.ReduceMean(keep_dims=False).shard(((1,),)) self.sum_ops = P.ReduceSum(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul1d = P.Mul().shard(((dp, 1), ())) self.log_softmax = P.LogSoftmax().shard(((dp, 1),)) def construct(self, logit, label): logit = P.Cast()(logit, mstype.float32) label = P.Cast()(label, mstype.float32) logit_softmax = self.log_softmax(logit) neg_target = self.mul1d(label, -1) soft_target = self.mul(neg_target, logit_softmax) loss = self.sum_ops(soft_target, -1) return self.mean_ops(loss) @MindFormerRegister.register(MindFormerModuleType.LOSS) class MSELoss(nn.Cell): """MSELoss for parallel.""" def __init__(self, norm_pixel_loss=True, parallel_config=default_dpmp_config): super(MSELoss, self).__init__() dp = parallel_config.data_parallel self.add_loss = P.Add().shard(((dp, 1, 1), ())) self.sub = P.Sub().shard(((dp, 1, 1), (dp, 1, 1))) self.divide = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.pow = P.Pow().shard(((dp, 1, 1), ())) self.divide1 = P.RealDiv().shard(((), ())) self.divide2 = P.RealDiv().shard(((dp, 1, 1), ())) self.square = P.Square().shard(((dp, 1, 1),)) self.cast = P.Cast() self.mean1 = P.ReduceMean(keep_dims=True).shard(((dp, 1, 1),)) self.mean2 = P.ReduceMean().shard(((dp, 1, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.sum = P.ReduceSum().shard(((dp, 1,),)) self.sum2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.norm_pixel_loss = norm_pixel_loss def construct(self, pred, target, mask): """mse loss construct.""" pred = self.cast(pred, mstype.float32) target = self.cast(target, mstype.float32) mask = self.cast(mask, mstype.float32) if self.norm_pixel_loss: mean = self.mean1(target, -1) var = self.variance(target) var = self.add_loss(var, 1e-6) std = self.pow(var, 0.5) sub = self.sub(target, mean) target = self.divide(sub, std) res = self.sub(pred, target) recon_loss = self.square(res) recon_loss = self.mean2(recon_loss, -1) loss_mask = self.mul(recon_loss, mask) loss_sum = self.sum(loss_mask) mask_sum = self.sum(mask) loss = self.divide1(loss_sum, mask_sum) return loss def variance(self, x): """get variance.""" axis = (x.ndim - 1,) x_mean = self.mean1(x, axis) x_sub = self.sub(x, x_mean) x_pow = self.pow(x_sub, 2) x_sum = self.sum2(x_pow, axis) x_var = self.divide2(x_sum, x.shape[-1]) return x_var @MindFormerRegister.register(MindFormerModuleType.LOSS) class L1Loss(LossBase): """L1Loss for parallel.""" def __init__(self, reduction='mean', parallel_config=default_dpmp_config): super(L1Loss, self).__init__() dp = parallel_config.data_parallel self.abs = P.Abs().shard(((dp, 1, 1, 1),)) self.sub = P.Sub().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((), (dp, 1, 1, 1))) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1, 1),)) self.reduce_sum = P.ReduceSum().shard(((dp, 1, 1, 1),)) self.cast = P.Cast() self.average = True self.reduce = True if reduction == 'sum': self.average = False if reduction == 'none': self.reduce = False def get_loss(self, x, weights=1.0): """get loss.""" input_dtype = x.dtype x = self.cast(x, mstype.float32) weights = self.cast(weights, mstype.float32) x = self.mul(weights, x) if self.reduce and self.average: x = self.reduce_mean(x, self.get_axis(x)) if self.reduce and not self.average: x = self.reduce_sum(x, self.get_axis(x)) x = self.cast(x, input_dtype) return x def construct(self, logits, labels): """L1Loss construct.""" x_sub = self.sub(logits, labels) x = self.abs(x_sub) return self.get_loss(x) class _Softmax(nn.Cell): """ Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. """ def __init__(self, parallel_config=default_dpmp_config): super(_Softmax, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel # on/off value for onehot, for smooth labeling, modify the off_value self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.sum = P.ReduceSum().shard(((dp, mp),)) self.max = P.ArgMaxWithValue(axis=-1, keep_dims=True).shard( ((dp, mp),)) self.sub = P.Sub().shard(((dp, mp), (dp, 1))) self.exp = P.Exp().shard(((dp, mp),)) self.div = P.RealDiv().shard(((dp, mp), (dp, 1))) self.onehot = P.OneHot().shard(((dp, mp), (), ())) def construct(self, logits, label): """Forward process """ # LogSoftmax for logits over last dimension logits = F.cast(logits, mstype.float32) _, logit_max = self.max(logits) logit_sub = self.sub(logits, logit_max) logit_exp = self.exp(logit_sub) exp_sum = self.sum(logit_exp, -1) exp_sum = P.Reshape()(exp_sum, (F.shape(exp_sum)[0], 1)) softmax_result = self.div(logit_exp, exp_sum) one_hot_label = self.onehot(label, F.shape(logits)[-1], self.on_value, self.off_value) return softmax_result, one_hot_label def bprop(self, logits, label, _, dout): """just return the loss of the dout. Note this should be used together with _NLLLoss""" d_logits = F.cast(dout[0], F.dtype(logits)) return d_logits, F.zeros_like(label) class _NLLLoss(nn.Cell): """ Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. """ def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(_NLLLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.repeat_loss = 1 self.eps_const = Tensor(eps_const, mstype.float32) # In auto parallel, there will be a virtual div in the back propagation begins. As we use custom bprop function # we need to eliminate this virtual div by adding a factor "mp". if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.repeat_loss = mp if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.sum = P.ReduceSum() self.mul = P.Mul() self.neg = P.Neg() self.log = P.Log() self.add = P.Add().shard(((dp, mp), ())) else: self.sum = P.ReduceSum().shard(((dp, mp),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.neg = P.Neg().shard(((dp, mp),)) self.log = P.Log().shard(((dp, mp),)) self.add = P.Add().shard(((dp, mp), ())) def construct(self, softmax_result, one_hot_label): log_softmax_result = self.log(self.add(softmax_result, self.eps_const)) loss = self.mul(log_softmax_result, one_hot_label) loss_unsum = self.neg(loss) loss_reduce = self.sum(loss_unsum, -1) return loss_reduce def bprop(self, softmax_result, one_hot_label, _, dout): """A simplified function. Note this should be used together with _Softmax""" logits = softmax_result - one_hot_label logits = logits * P.ExpandDims()(dout, -1) * self.repeat_loss return logits, F.zeros_like(one_hot_label) @MindFormerRegister.register(MindFormerModuleType.LOSS) class CrossEntropyLoss(nn.Cell): """ Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) """ @_LogActionOnce(m_logger=logger, key='CrossEntropyLoss', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(CrossEntropyLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.enable_force_redistribute = False if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.enable_force_redistribute = True self.add = P.Add().shard(((dp, mp), ())).add_prim_attr("keep_alive", True) self.add_label = P.Add().shard(((dp,), ())).add_prim_attr("keep_alive", True) self._check_and_modify_sharding_context(dp) self.sum2 = P.ReduceSum().shard(((1,),)) self.mul2 = P.Mul().shard(((1,), (1,))) self.add2 = P.Add() self.div2 = P.RealDiv() self.relu = P.ReLU().shard(((1,),)) self._softmax = _Softmax(parallel_config) self._nllloss = _NLLLoss(parallel_config, eps_const) @staticmethod def _check_and_modify_sharding_context(dp): device_num = _get_device_num() stages = _get_pipeline_stages() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and dp * stages != device_num: set_algo_parameters(fully_use_devices=False) def construct(self, logits, label, input_mask): """Forward process""" # The add is used for forcing the redistribution before stepping in sub graphs, when semi/auto parallel enabled. if self.enable_force_redistribute: logits = self.add(logits, 0) label = self.add_label(label, 0) softmax, one_hot_label = self._softmax(logits, label) loss_reduce = self._nllloss(softmax, one_hot_label) # Using input_mask to mask the loss input_mask = P.Reshape()(input_mask, (-1,)) numerator = self.sum2(self.mul2(loss_reduce, input_mask)) denominator = self.add2( self.sum2(input_mask), P.Cast()(F.tuple_to_array((1e-5,)), mstype.float32)) loss = self.div2(numerator, denominator) return loss @MindFormerRegister.register(MindFormerModuleType.LOSS) class CompareLoss(nn.Cell): """ Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: """ def __init__(self, config): super(CompareLoss, self).__init__() dp = config.data_parallel mp = 1 self.gatherd = P.GatherD() self.log = P.Log() self.reduce_sum = P.ReduceSum(keep_dims=False) self.slice = P.StridedSlice().shard(((1, 1),)) self.slice_ind = P.StridedSlice().shard(((1,),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.sub = P.Sub().shard(((dp, mp), (dp, mp))) def construct(self, rewards, loss_mask, end_ind): """Forward process""" bs = rewards.shape[0] // 2 # a sample has two bs responses seq_len = rewards.shape[-1] chosen_rewards = self.slice(rewards, (0, 0), (bs, seq_len), (1, 1)) rejected_rewards = self.slice(rewards, (bs, 0), (2*bs, seq_len), (1, 1)) end_ind_chosen = self.slice_ind(end_ind, (0,), (bs,), (1,)) end_ind_reject = self.slice_ind(end_ind, (bs,), (2*bs,), (1,)) temp = P.Concat()((end_ind_chosen, end_ind_reject)) temp = temp.reshape((2, -1)) temp = P.Cast()(temp, mstype.float16) end_ind_final, _ = P.max(temp, axis=0) temp = P.Cast()(temp, mstype.float16) end_ind_final = end_ind_final.reshape((-1, 1)) end_ind_final = P.Cast()(end_ind_final, mstype.int32) loss_mask_final = loss_mask c_truncated_reward = self.mul(chosen_rewards, loss_mask_final) r_truncated_reward = self.mul(rejected_rewards, loss_mask_final) chosen_end_scores = self.gatherd(chosen_rewards, 1, end_ind_final-1) reject_end_scores = self.gatherd(rejected_rewards, 1, end_ind_final-1) compare_len = self.reduce_sum(P.cast(loss_mask_final, mstype.float32), -1) temp_loss = -self.log(P.sigmoid(self.sub(c_truncated_reward, r_truncated_reward))) loss = self.reduce_sum(self.mul(temp_loss, loss_mask_final), -1)/compare_len loss = loss.mean() return loss, chosen_end_scores, reject_end_scores pengcheng.mind-mindformer/mindformers/core/loss/__init__.py000066400000000000000000000014251476650141400244620ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Loss.""" from .build_loss import build_loss from .loss import * __all__ = [] __all__.extend(loss.__all__) pengcheng.mind-mindformer/mindformers/core/loss/__pycache__/000077500000000000000000000000001476650141400245575ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/loss/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004411476650141400307460ustar00rootroot00000000000000a ?cf@s,dZddlmZddlTgZeejdS)zMindFormers Loss.) build_loss)*N)__doc__rloss__all__extendrrI/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/__init__.pys pengcheng.mind-mindformer/mindformers/core/loss/__pycache__/build_loss.cpython-39.pyc000066400000000000000000000035661476650141400313610ustar00rootroot00000000000000a ?cf @sTdZddlZddlmZddlmZmZmZd eee e dddZ d d Z e dS) zBuild Loss API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfigloss)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aBuild loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') N)r) isinstancedictrrget_instance_from_cfgrLOSS get_instance)rrr r kwargsrK/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/build_loss.py build_losssrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z( register MindSpore builtin loss class. __N) dirrr startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_loss8s   r)NNrN) __doc__r mindsporermindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-mindformer/mindformers/core/loss/__pycache__/loss.cpython-39.pyc000066400000000000000000000342411476650141400301740ustar00rootroot00000000000000a ?cfB@sjdZddlmZmZddlmZddlmZddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZmZdd lmZgdZ e!ej"Gddde Z#e!ej"Gdddej$Z%e!ej"Gddde Z&Gdddej$Z'Gdddej$Z(e!ej"Gdddej$Z)e!ej"Gdddej$Z*dS)zMindFormer Self-Define Loss.)nnTensor)ops) functional)dtype)LossBase) ParallelMode)set_algo_parameters)log)_get_device_num_get_pipeline_stages_get_parallel_mode_is_sharding_propagation)_LogActionOnce)MindFormerRegisterMindFormerModuleType)default_dpmp_config)SoftTargetCrossEntropyMSELossL1LossCrossEntropyLoss CompareLosscs,eZdZdZeffdd ZddZZS)rz)SoftTargetCrossEntropy for MixUp Augment.cstt||j}tjddd|_tjdd|dff|_ t |df|dff|_ t |dfdf|_ t |dff|_dS)NF keep_dimsr)superr__init__ data_parallelP ReduceMeanshardmean_ops ReduceSumsum_opsMulmulmul1d LogSoftmax log_softmax)selfparallel_configdp __class__rE/home/ma-user/work/mindformers-baichuan/mindformers/core/loss/loss.pyr(szSoftTargetCrossEntropy.__init__cCsXt|tj}t|tj}||}||d}|||}||d}||SN) r!Castmstypefloat32r+r)r(r&r$)r,logitlabelZ logit_softmaxZ neg_targetZ soft_targetlossrrr1 construct1s    z SoftTargetCrossEntropy.construct)__name__ __module__ __qualname____doc__rrr: __classcell__rrr/r1r$s rcs6eZdZdZdeffdd ZddZddZZS) rzMSELoss for parallel.Tcs`tt||j}t|ddfdf|_t|ddf|ddff|_ t |ddf|ddff|_ t |ddfdf|_ t d|_t |ddfdf|_t|ddff|_t|_tjdd|ddff|_t|ddff|_t|df|dff|_t|dff|_tjdd|ddff|_||_dS)Nrr)rrTr)rrrr r!Addr#add_lossSubsubRealDivdividePowpowdivide1divide2Squaresquarer4castr"mean1mean2r'r(r%sumsum2norm_pixel_loss)r,rQr-r.r/rr1r>s    zMSELoss.__init__cCs||tj}||tj}||tj}|jrv||d}||}||d}||d}|||}| ||}|||}| |} | | d} | | |} | | } | |} || | } | S)zmse loss construct.r3gư>g?)rLr5r6rQrMvariancerArGrCrErKrNr(rOrH)r,predtargetmaskmeanvarstdrCresZ recon_loss loss_maskZloss_sumZmask_sumr9rrr1r:Ps$             zMSELoss.constructcCsR|jdf}|||}|||}||d}|||}|||jd}|S)z get variance.rr3)ndimrMrCrGrPrIshape)r,xaxisx_meanx_subx_powx_sumx_varrrr1rRes     zMSELoss.variance) r;r<r=r>rrr:rRr?rrr/r1r;srcs8eZdZdZdeffdd Zd ddZdd ZZS) rzL1Loss for parallel.rVcstt||j}t|dddff|_t|dddf|dddff|_ t d|dddff|_ t |dddff|_ t|dddff|_t|_d|_d|_|dkrd|_|dkrd|_dS)NrrTrOFnone)rrrr r!Absr#absrBrCr'r(r" reduce_meanr% reduce_sumr4rLaveragereduce)r, reductionr-r.r/rr1rss$ zL1Loss.__init__?cCsz|j}||tj}||tj}|||}|jrL|jrL||||}|jrj|jsj| |||}|||}|S)z get loss.) rrLr5r6r(rkrjrhget_axisri)r,r^weights input_dtyperrr1get_losss    zL1Loss.get_losscCs |||}||}||S)zL1Loss construct.)rCrgrq)r,logitslabelsrar^rrr1r:s  zL1Loss.construct)rm) r;r<r=r>rrrqr:r?rrr/r1rps rcs4eZdZdZeffdd ZddZddZZS)_Softmaxa Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. cstt||j}|j}tdtj|_tdtj|_ t ||ff|_ t jddd ||ff|_t  ||f|dff|_t  ||ff|_t  ||f|dff|_t  ||fddf|_dS)Nrmgr3T)r_rrr)rrtrr model_parallelrr5r6on_value off_valuer!r%r#rOArgMaxWithValuemaxrBrCExpexprDdivOneHotonehot)r,r-r.mpr/rr1rsz_Softmax.__init__c Cst|tj}||\}}|||}||}||d}t |t |ddf}| ||}| |t |d|j |j} || fS)zForward process r3rr)FrLr5r6ryrCr{rOr!Reshaper]r|r~rvrw) r,rrr8_Z logit_maxZ logit_subZ logit_expZexp_sumsoftmax_result one_hot_labelrrr1r:s    z_Softmax.constructcCs$t|dt|}|t|fS)zQjust return the loss of the dout. Note this should be used together with _NLLLossr)rrLr zeros_like)r,rrr8rdoutZd_logitsrrr1bpropsz_Softmax.bprop r;r<r=r>rrr:rr?rrr/r1rtsrtcs6eZdZdZedffdd ZddZddZZS) _NLLLossa7 Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. 秎W:cstt||j}|j}d|_t|tj|_ t t j t j fvrF||_t t j fvrtrt|_t|_t|_t|_t||fdf|_nvt||ff|_t||f||ff|_t||ff|_t||ff|_t||fdf|_dS)Nrr)rrrr ru repeat_lossrr5r6 eps_constr r AUTO_PARALLELSEMI_AUTO_PARALLELrr!r%rOr'r(NegnegLogr r@r#addr,r-rr.rr/rr1rs$    z_NLLLoss.__init__cCs:||||j}|||}||}||d}|Sr2)r rrr(rrO)r,rrZlog_softmax_resultr9Z loss_unsum loss_reducerrr1r:s    z_NLLLoss.constructcCs.||}|t|d|j}|t|fS)zFA simplified function. Note this should be used together with _Softmaxr3)r! ExpandDimsrrr)r,rrrrrrrrr1rsz_NLLLoss.bproprrrr/r1rsrcsReZdZdZeedeejfvde dffdd Z e ddZ dd Z ZS) ra Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) )m_loggerkey no_warningrcstt||j}|j}d|_ttjtj fvrd|_t ||fdf dd|_t |fdf dd|_||t  d|_t  d|_t |_t |_t  d|_t||_t|||_dS)NFTr keep_aliver)rr)rrrr ruenable_force_redistributer rrrr!r@r# add_prim_attrr add_label"_check_and_modify_sharding_contextr%rPr'mul2add2rDdiv2ReLUrelurt_softmaxr_nlllossrr/rr1r-s      zCrossEntropyLoss.__init__cCs4t}t}ttjfvr0|||kr0tdddS)NF)fully_use_devices)r r r rrr )r. device_numstagesrrr1rBsz3CrossEntropyLoss._check_and_modify_sharding_contextc Cs|jr||d}||d}|||\}}|||}t|d}||||}| ||t t dt j}|||} | S)Forward processr)r3)gh㈵>)rrrrrr!rrPrrr4rtuple_to_arrayr5r6r) r,rrr8Z input_masksoftmaxrr numerator denominatorr9rrr1r:Is    zCrossEntropyLoss.construct)r;r<r=r>rloggerr r STAND_ALONErr staticmethodrr:r?rrr/r1r s"  rcs(eZdZdZfddZddZZS)ram Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: cstt||j}d}t|_t|_tj dd|_ t d|_ t d|_t ||f||ff|_t ||f||ff|_dS)NrFr)rrr)rrrr r!GatherDgatherdrr r%ri StridedSlicer#slice slice_indr'r(rBrC)r,configr.rr/rr1rts  zCompareLoss.__init__cCsr|jdd}|jd}||d||fd}|||dfd||fd}||d|fd}|||fd|fd} t|| f} | d} t| tj} tj | dd \} } t| tj} | d } t| tj } |} | || }| || }| |d | d }| |d | d }| t| tjd}|t||| }| | || d|}|}|||fS) rrr[r3)rrr)rr)r[r3)r_)r3rr)r]rrr!Concatreshaper4r5float16ryint32r(rrirLr6r sigmoidrCrV)r,ZrewardsrZend_indbsseq_lenZchosen_rewardsZrejected_rewardsZend_ind_chosenZend_ind_rejecttempZ end_ind_finalrZloss_mask_finalZc_truncated_rewardZr_truncated_rewardZchosen_end_scoresZreject_end_scoresZ compare_lenZ temp_lossr9rrr1r:s.     zCompareLoss.construct)r;r<r=r>rr:r?rrr/r1r^s rN)+r> mindsporerrrr! mindspore.opsrrmindspore.commonrr5mindspore.nn.loss.lossrZmindspore.contextrZmindspore.parallelr r rmindspore.parallel._utilsr r r rmindformers.tools.loggerrmindformers.tools.registerrrZ2mindformers.modules.transformer.op_parallel_configr__all__registerLOSSrCellrrrtrrrrrrr1s2           4 )78 T pengcheng.mind-mindformer/mindformers/core/loss/build_loss.py000066400000000000000000000047751476650141400250750ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Loss API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_loss( config: dict = None, default_args: dict = None, module_type: str = 'loss', class_name: str = None, **kwargs): r"""Build loss For MindFormer. Instantiate the loss from MindFormerRegister's registry. Args: config (dict): The task loss's config. Default: None. default_args (dict): The default argument of loss API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'loss'. class_name (str): The class name of loss API. Default: None. Return: The function instance of loss API. Examples: >>> from mindformers import build_loss >>> loss_config = {'type': 'L1Loss'} >>> # 1) use config dict to build loss >>> loss_from_config = build_loss(loss_config) >>> # 2) use class name to build loss >>> loss_class_name = build_loss(class_name='L1Loss') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LOSS, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_loss(): """ register MindSpore builtin loss class. """ for module_name in dir(nn.loss): if module_name.startswith('__'): continue loss = getattr(nn.loss, module_name) if inspect.isclass(loss): MindFormerRegister.register_cls( loss, MindFormerModuleType.LOSS) register_ms_loss() pengcheng.mind-mindformer/mindformers/core/loss/loss.py000066400000000000000000000412271476650141400237070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormer Self-Define Loss.""" from mindspore import nn, Tensor from mindspore import ops as P from mindspore.ops import functional as F from mindspore.common import dtype as mstype from mindspore.nn.loss.loss import LossBase from mindspore.context import ParallelMode from mindspore.parallel import set_algo_parameters from mindspore import log as logger from mindspore.parallel._utils import _get_device_num, _get_pipeline_stages, _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import _LogActionOnce from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.op_parallel_config import default_dpmp_config __all__ = ['SoftTargetCrossEntropy', 'MSELoss', 'L1Loss', 'CrossEntropyLoss', 'CompareLoss'] @MindFormerRegister.register(MindFormerModuleType.LOSS) class SoftTargetCrossEntropy(LossBase): """SoftTargetCrossEntropy for MixUp Augment.""" def __init__(self, parallel_config=default_dpmp_config): super(SoftTargetCrossEntropy, self).__init__() dp = parallel_config.data_parallel self.mean_ops = P.ReduceMean(keep_dims=False).shard(((1,),)) self.sum_ops = P.ReduceSum(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul1d = P.Mul().shard(((dp, 1), ())) self.log_softmax = P.LogSoftmax().shard(((dp, 1),)) def construct(self, logit, label): logit = P.Cast()(logit, mstype.float32) label = P.Cast()(label, mstype.float32) logit_softmax = self.log_softmax(logit) neg_target = self.mul1d(label, -1) soft_target = self.mul(neg_target, logit_softmax) loss = self.sum_ops(soft_target, -1) return self.mean_ops(loss) @MindFormerRegister.register(MindFormerModuleType.LOSS) class MSELoss(nn.Cell): """MSELoss for parallel.""" def __init__(self, norm_pixel_loss=True, parallel_config=default_dpmp_config): super(MSELoss, self).__init__() dp = parallel_config.data_parallel self.add_loss = P.Add().shard(((dp, 1, 1), ())) self.sub = P.Sub().shard(((dp, 1, 1), (dp, 1, 1))) self.divide = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.pow = P.Pow().shard(((dp, 1, 1), ())) self.divide1 = P.RealDiv().shard(((), ())) self.divide2 = P.RealDiv().shard(((dp, 1, 1), ())) self.square = P.Square().shard(((dp, 1, 1),)) self.cast = P.Cast() self.mean1 = P.ReduceMean(keep_dims=True).shard(((dp, 1, 1),)) self.mean2 = P.ReduceMean().shard(((dp, 1, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.sum = P.ReduceSum().shard(((dp, 1,),)) self.sum2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.norm_pixel_loss = norm_pixel_loss def construct(self, pred, target, mask): """mse loss construct.""" pred = self.cast(pred, mstype.float32) target = self.cast(target, mstype.float32) mask = self.cast(mask, mstype.float32) if self.norm_pixel_loss: mean = self.mean1(target, -1) var = self.variance(target) var = self.add_loss(var, 1e-6) std = self.pow(var, 0.5) sub = self.sub(target, mean) target = self.divide(sub, std) res = self.sub(pred, target) recon_loss = self.square(res) recon_loss = self.mean2(recon_loss, -1) loss_mask = self.mul(recon_loss, mask) loss_sum = self.sum(loss_mask) mask_sum = self.sum(mask) loss = self.divide1(loss_sum, mask_sum) return loss def variance(self, x): """get variance.""" axis = (x.ndim - 1,) x_mean = self.mean1(x, axis) x_sub = self.sub(x, x_mean) x_pow = self.pow(x_sub, 2) x_sum = self.sum2(x_pow, axis) x_var = self.divide2(x_sum, x.shape[-1]) return x_var @MindFormerRegister.register(MindFormerModuleType.LOSS) class L1Loss(LossBase): """L1Loss for parallel.""" def __init__(self, reduction='mean', parallel_config=default_dpmp_config): super(L1Loss, self).__init__() dp = parallel_config.data_parallel self.abs = P.Abs().shard(((dp, 1, 1, 1),)) self.sub = P.Sub().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((), (dp, 1, 1, 1))) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1, 1),)) self.reduce_sum = P.ReduceSum().shard(((dp, 1, 1, 1),)) self.cast = P.Cast() self.average = True self.reduce = True if reduction == 'sum': self.average = False if reduction == 'none': self.reduce = False def get_loss(self, x, weights=1.0): """get loss.""" input_dtype = x.dtype x = self.cast(x, mstype.float32) weights = self.cast(weights, mstype.float32) x = self.mul(weights, x) if self.reduce and self.average: x = self.reduce_mean(x, self.get_axis(x)) if self.reduce and not self.average: x = self.reduce_sum(x, self.get_axis(x)) x = self.cast(x, input_dtype) return x def construct(self, logits, labels): """L1Loss construct.""" x_sub = self.sub(logits, labels) x = self.abs(x_sub) return self.get_loss(x) class _Softmax(nn.Cell): """ Calculate the softmax results with given logits. Note: The bprop of the cell is rewritten, just returns the accepted dout as returns. This cell should be used together with _NLLoss, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. Outputs: Tensor. The corresponding softmax results. """ def __init__(self, parallel_config=default_dpmp_config): super(_Softmax, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel # on/off value for onehot, for smooth labeling, modify the off_value self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.sum = P.ReduceSum().shard(((dp, mp),)) self.max = P.ArgMaxWithValue(axis=-1, keep_dims=True).shard( ((dp, mp),)) self.sub = P.Sub().shard(((dp, mp), (dp, 1))) self.exp = P.Exp().shard(((dp, mp),)) self.div = P.RealDiv().shard(((dp, mp), (dp, 1))) self.onehot = P.OneHot().shard(((dp, mp), (), ())) def construct(self, logits, label): """Forward process """ # LogSoftmax for logits over last dimension logits = F.cast(logits, mstype.float32) _, logit_max = self.max(logits) logit_sub = self.sub(logits, logit_max) logit_exp = self.exp(logit_sub) exp_sum = self.sum(logit_exp, -1) exp_sum = P.Reshape()(exp_sum, (F.shape(exp_sum)[0], 1)) softmax_result = self.div(logit_exp, exp_sum) one_hot_label = self.onehot(label, F.shape(logits)[-1], self.on_value, self.off_value) return softmax_result, one_hot_label def bprop(self, logits, label, _, dout): """just return the loss of the dout. Note this should be used together with _NLLLoss""" d_logits = F.cast(dout[0], F.dtype(logits)) return d_logits, F.zeros_like(label) class _NLLLoss(nn.Cell): """ Calculate the NLLLoss results with given softmax results and the label. Note: The bprop of the cell is rewritten. This cell should be used together with _Softmax, to optimize the bprop of the cross entroy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **loss** (Tensor) - Tensor of shape (N, C). Data type is float32. Outputs: Tensor. The corresponding loss results. """ def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(_NLLLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.repeat_loss = 1 self.eps_const = Tensor(eps_const, mstype.float32) # In auto parallel, there will be a virtual div in the back propagation begins. As we use custom bprop function # we need to eliminate this virtual div by adding a factor "mp". if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.repeat_loss = mp if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.sum = P.ReduceSum() self.mul = P.Mul() self.neg = P.Neg() self.log = P.Log() self.add = P.Add().shard(((dp, mp), ())) else: self.sum = P.ReduceSum().shard(((dp, mp),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.neg = P.Neg().shard(((dp, mp),)) self.log = P.Log().shard(((dp, mp),)) self.add = P.Add().shard(((dp, mp), ())) def construct(self, softmax_result, one_hot_label): log_softmax_result = self.log(self.add(softmax_result, self.eps_const)) loss = self.mul(log_softmax_result, one_hot_label) loss_unsum = self.neg(loss) loss_reduce = self.sum(loss_unsum, -1) return loss_reduce def bprop(self, softmax_result, one_hot_label, _, dout): """A simplified function. Note this should be used together with _Softmax""" logits = softmax_result - one_hot_label logits = logits * P.ExpandDims()(dout, -1) * self.repeat_loss return logits, F.zeros_like(one_hot_label) @MindFormerRegister.register(MindFormerModuleType.LOSS) class CrossEntropyLoss(nn.Cell): """ Calculate the cross entropy loss. Args: parallel_config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **logits** (Tensor) - Tensor of shape (N, C). Data type must be float16 or float32. The output logits of the backbone. - **labels** (Tensor) - Tensor of shape (N, ). The ground truth label of the sample. - **input_mask** (Tensor) - Tensor of shape (N, ). input_mask indicates whether there are padded inputs and for padded inputs it will not be counted into loss. Outputs: Tensor. The corresponding cross entropy loss. Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> from mindformers.core import CrossEntropyLoss >>> loss = CrossEntropyLoss() >>> >>> logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), mstype.float32) >>> labels_np = np.array([1]).astype(np.int32) >>> input_mask = Tensor(np.ones(1).astype(np.float32)) >>> labels = Tensor(labels_np) >>> output = loss(logits, labels, input_mask) >>> print(output.shape) (1,) """ @_LogActionOnce(m_logger=logger, key='CrossEntropyLoss', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) def __init__(self, parallel_config=default_dpmp_config, eps_const=1e-24): super(CrossEntropyLoss, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.enable_force_redistribute = False if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL, ParallelMode.SEMI_AUTO_PARALLEL): self.enable_force_redistribute = True self.add = P.Add().shard(((dp, mp), ())).add_prim_attr("keep_alive", True) self.add_label = P.Add().shard(((dp,), ())).add_prim_attr("keep_alive", True) self._check_and_modify_sharding_context(dp) self.sum2 = P.ReduceSum().shard(((1,),)) self.mul2 = P.Mul().shard(((1,), (1,))) self.add2 = P.Add() self.div2 = P.RealDiv() self.relu = P.ReLU().shard(((1,),)) self._softmax = _Softmax(parallel_config) self._nllloss = _NLLLoss(parallel_config, eps_const) @staticmethod def _check_and_modify_sharding_context(dp): device_num = _get_device_num() stages = _get_pipeline_stages() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and dp * stages != device_num: set_algo_parameters(fully_use_devices=False) def construct(self, logits, label, input_mask): """Forward process""" # The add is used for forcing the redistribution before stepping in sub graphs, when semi/auto parallel enabled. if self.enable_force_redistribute: logits = self.add(logits, 0) label = self.add_label(label, 0) softmax, one_hot_label = self._softmax(logits, label) loss_reduce = self._nllloss(softmax, one_hot_label) # Using input_mask to mask the loss input_mask = P.Reshape()(input_mask, (-1,)) numerator = self.sum2(self.mul2(loss_reduce, input_mask)) denominator = self.add2( self.sum2(input_mask), P.Cast()(F.tuple_to_array((1e-5,)), mstype.float32)) loss = self.div2(numerator, denominator) return loss @MindFormerRegister.register(MindFormerModuleType.LOSS) class CompareLoss(nn.Cell): """ Calculate the compare loss for reward model. Args: config (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **rewards** (Tensor) - Tensor of shape (B, S, 1). Data type must be float16 or float32. The output logits of the backbone. - **loss_mask** (Tensor) - Tensor of shape (B, S, 1). The loss mask of the rewards. - **end_ind** (Tensor) - Tensor of shape (B, ). end index of all tensors. Outputs: Tensor. The corresponding loss. Examples: """ def __init__(self, config): super(CompareLoss, self).__init__() dp = config.data_parallel mp = 1 self.gatherd = P.GatherD() self.log = P.Log() self.reduce_sum = P.ReduceSum(keep_dims=False) self.slice = P.StridedSlice().shard(((1, 1),)) self.slice_ind = P.StridedSlice().shard(((1,),)) self.mul = P.Mul().shard(((dp, mp), (dp, mp))) self.sub = P.Sub().shard(((dp, mp), (dp, mp))) def construct(self, rewards, loss_mask, end_ind): """Forward process""" bs = rewards.shape[0] // 2 # a sample has two bs responses seq_len = rewards.shape[-1] chosen_rewards = self.slice(rewards, (0, 0), (bs, seq_len), (1, 1)) rejected_rewards = self.slice(rewards, (bs, 0), (2*bs, seq_len), (1, 1)) end_ind_chosen = self.slice_ind(end_ind, (0,), (bs,), (1,)) end_ind_reject = self.slice_ind(end_ind, (bs,), (2*bs,), (1,)) temp = P.Concat()((end_ind_chosen, end_ind_reject)) temp = temp.reshape((2, -1)) temp = P.Cast()(temp, mstype.float16) end_ind_final, _ = P.max(temp, axis=0) temp = P.Cast()(temp, mstype.float16) end_ind_final = end_ind_final.reshape((-1, 1)) end_ind_final = P.Cast()(end_ind_final, mstype.int32) loss_mask_final = loss_mask c_truncated_reward = self.mul(chosen_rewards, loss_mask_final) r_truncated_reward = self.mul(rejected_rewards, loss_mask_final) chosen_end_scores = self.gatherd(chosen_rewards, 1, end_ind_final-1) reject_end_scores = self.gatherd(rejected_rewards, 1, end_ind_final-1) compare_len = self.reduce_sum(P.cast(loss_mask_final, mstype.float32), -1) temp_loss = -self.log(P.sigmoid(self.sub(c_truncated_reward, r_truncated_reward))) loss = self.reduce_sum(self.mul(temp_loss, loss_mask_final), -1)/compare_len loss = loss.mean() return loss, chosen_end_scores, reject_end_scores pengcheng.mind-mindformer/mindformers/core/lr/000077500000000000000000000000001476650141400220045ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/lr/.ipynb_checkpoints/000077500000000000000000000000001476650141400255755ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/lr/.ipynb_checkpoints/lr_schedule-checkpoint.py000066400000000000000000000353751476650141400326020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/PanGu-%CE%B1/utils.py # https://github.com/huggingface/transformers/blob/main/src/transformers/optimization.py # ============================================================================ """Self-Define LR Schedule.""" import math import numpy as np from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.nn.learning_rate_schedule import LearningRateSchedule, PolynomialDecayLR, WarmUpLR, CosineDecayLR from mindspore.common.tensor import Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'LearningRateWiseLayer', 'WarmUpDecayLR', 'ConstantWarmUpLR', 'LinearWithWarmUpLR', 'CosineWithWarmUpLR', 'CosineWithRestartsAndWarmUpLR', 'PolynomialWithWarmUpLR'] @MindFormerRegister.register(MindFormerModuleType.LR) class ConstantWarmUpLR(LearningRateSchedule): """ConstantWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, warmup_lr_init: float = 0., **kwargs): super(ConstantWarmUpLR, self).__init__() warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.greater = P.Greater() self.total_steps = kwargs.get('total_steps', None) def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.one_constant learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LinearWithWarmUpLR(LearningRateSchedule): """LinearWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, warmup_lr_init: float = 0.): super(LinearWithWarmUpLR, self).__init__() linear_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.linear_steps = Tensor(linear_steps, mstype.float32) self.greater = P.Greater() self.max = P.Maximum() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.max(self.zero_constant, (self.total_steps - global_step) / self.linear_steps) learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithWarmUpLR(LearningRateSchedule): """CosineWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithRestartsAndWarmUpLR(LearningRateSchedule): """CosineWithRestartsAndWarmUpLR.""" def __init__(self, learning_rate: float, total_steps: int, warmup_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithRestartsAndWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps if self.greater(self.one_constant, progress): percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * ((self.num_cycles * progress) % self.one_constant)))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent else: return self.zero_constant return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class PolynomialWithWarmUpLR(LearningRateSchedule): """PolynomialWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, lr_end: float = 1e-7, power: float = 1.0, warmup_lr_init: float = 0.): super(PolynomialWithWarmUpLR, self).__init__() decay_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) if not learning_rate > lr_end: raise ValueError(f"lr_end ({lr_end}) must be be smaller than initial lr ({learning_rate})") self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.lr_end = lr_end self.power = power self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.decay_steps = Tensor(decay_steps, mstype.float32) self.greater = P.Greater() self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: lr_range = self.learning_rate - self.lr_end pct_remaining = 1 - (global_step - self.warmup_steps) / self.decay_steps decay = lr_range * pct_remaining ** self.power + self.lr_end percent = decay / self.learning_rate learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LearningRateWiseLayer(LearningRateSchedule): """LearningRateWiseLayer.""" def __init__(self, base_lr, lr_scale): super(LearningRateWiseLayer, self).__init__() self.base_lr = base_lr self.lr_scale = lr_scale def construct(self, global_step): lr = self.base_lr(global_step) return self.lr_scale * lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpDecayLR(LearningRateSchedule): """ Warmup-decay learning rate for Bert network. """ def __init__(self, learning_rate, end_learning_rate, warmup_steps, decay_steps, power=1.0, use_cosine=False): super(WarmUpDecayLR, self).__init__() self.warmup_flag = False if warmup_steps > 0: self.warmup_flag = True self.warmup_lr = WarmUpLR(learning_rate, warmup_steps) self.decay_lr = PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power) self.cosine_decay_lr = CosineDecayLR(end_learning_rate, learning_rate, decay_steps) self.warmup_steps = Tensor(np.array([warmup_steps]).astype(np.float32)) self.greater = P.Greater() self.one = Tensor(np.array([1.0]).astype(np.float32)) self.cast = P.Cast() self.use_cosine = use_cosine def construct(self, global_step): """dynamic learning rate""" if not self.use_cosine: decay_lr = self.decay_lr(global_step) else: decay_lr = self.cosine_decay_lr(global_step) if self.warmup_flag: is_warmup = self.cast(self.greater(self.warmup_steps, global_step), mstype.float32) warmup_lr = self.warmup_lr(global_step) lr = (self.one - is_warmup) * decay_lr + is_warmup * warmup_lr else: lr = decay_lr return lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpStableDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpStableDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.cast = P.Cast() print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(global_step, self.decay_start_step_idx): grad = Tensor((self.lr_end - self.learning_rate), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.learning_rate, mstype.float32) else: learning_rate = Tensor(self.learning_rate, mstype.float32) return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpCosineWithDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_middle: float = 0., lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpCosineWithDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) self.warmup_steps = Tensor(warmup_steps, mstype.float32) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.cosine_steps = Tensor(decay_steps - warmup_steps, mstype.float32) self.zero_constant = Tensor(0.0, mstype.float32) self.num_cycles = 0.5 self.learning_rate = learning_rate self.lr_end = lr_end self.lr_middle = lr_middle self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.max = P.Maximum() self.cast = P.Cast() self.cos = P.Cos() self.math_pi = math.pi print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}, learning_rate: {learning_rate}, lr_middle: {lr_middle}, lr_end: {lr_end}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(self.decay_start_step_idx, global_step): progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_middle + (self.learning_rate - self.lr_middle) * percent else: grad = (self.lr_end - self.lr_middle) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + self.lr_middle return learning_ratepengcheng.mind-mindformer/mindformers/core/lr/__init__.py000066400000000000000000000014421476650141400241160ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers LR API.""" from .build_lr import build_lr from .lr_schedule import * __all__ = [] __all__.extend(lr_schedule.__all__) pengcheng.mind-mindformer/mindformers/core/lr/__pycache__/000077500000000000000000000000001476650141400242145ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/lr/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004461476650141400304100ustar00rootroot00000000000000a ?cf"@s,dZddlmZddlTgZeejdS)zMindFormers LR API.)build_lr)*N)__doc__r lr_schedule__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/__init__.pys pengcheng.mind-mindformer/mindformers/core/lr/__pycache__/build_lr.cpython-39.pyc000066400000000000000000000052041476650141400304420ustar00rootroot00000000000000a (gf@sdZddlZddlmZddlmZmZmZddlm Z m Z m Z m Z m Z mZdeeeedd d Zd d Zd dZeedS)zBuild LR Schedule API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConstantWarmUpLRCosineWithWarmUpLRLinearWithWarmUpLRCosineWithRestartsAndWarmUpLRPolynomialWithWarmUpLRWarmUpStableDecayLRlr)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)acBuild lr For MindFormer. Instantiate the lr from MindFormerRegister's registry. Args: config (dict): The task lr's config. Default: None. default_args (dict): The default argument of lr API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'lr'. class_name (str): The class name of lr API. Default: None. Return: The function instance of lr API. Examples: >>> from mindformers import build_lr >>> lr_config = {'type': 'CosineDecayLR', 'max_lr': 0.001, ... 'min_lr': 0., 'decay_steps':10} >>> # 1) use config dict to build lr >>> lr_from_config = build_lr(lr_config) >>> # 2) use class name to build lr >>> lr_class_name = build_lr(class_name='CosineDecayLR', max_lr=0.001, min_lr=0., decay_steps=10) N)r) isinstancedictrrZget_instance_from_cfgrLR get_instance)rrrrkwargsrG/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/build_lr.pybuild_lrsrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z& register MindSpore builtin LR class. __N) dirrlearning_rate_schedule startswithgetattrinspectisclassr register_clsrr) module_name lr_schedulerrrregister_ms_lr;s   r$cCsptjttjddtjttjddtjttjddtjttjddtjttjddtjt tjdddS) z( register MindFomrers builtin LR class. Zconstant_with_warmup)raliaslinearcosineZcosine_with_restarts polynomialZwarmup_stable_decayN) rr!rrrr rr r r rrrrregister_mf_lrFs$r))NNr N)__doc__r mindsporermindformers.tools.registerrrrr#rrr r r r rstrrr$r)rrrrs   " pengcheng.mind-mindformer/mindformers/core/lr/__pycache__/lr_schedule.cpython-39.pyc000066400000000000000000000213651476650141400311450ustar00rootroot00000000000000a of1@sRdZddlZddlZddlmZddlmm Z ddl m Z m Z mZmZddlmZddlmZmZgdZeejGdd d e ZeejGd d d e ZeejGd d d e ZeejGddde ZeejGddde ZeejGddde ZeejGddde ZeejGddde ZdS)zSelf-Define LR Schedule.N) operations)LearningRateSchedulePolynomialDecayLRWarmUpLR CosineDecayLR)Tensor)MindFormerRegisterMindFormerModuleType)LearningRateWiseLayer WarmUpDecayLRConstantWarmUpLRLinearWithWarmUpLRCosineWithWarmUpLRCosineWithRestartsAndWarmUpLRPolynomialWithWarmUpLRcs4eZdZdZdeeedfdd ZddZZS) r zConstantWarmUpLR.) learning_rate warmup_stepswarmup_lr_initc s\tt|td|}||_||_t|tj|_ tdtj|_ t |_ |dd|_dS)N? total_steps)superr __init__maxrrrmstypefloat32r one_constantPGreatergreatergetr)selfrrrkwargs __class__J/home/ma-user/work/mindformers-baichuan/mindformers/core/lr/lr_schedule.pyr%s  zConstantWarmUpLR.__init__cCs>||j|r*||j}|j|j|}n|j}|j|}|Scompute current step lr.)r rrrrr" global_steppercentrr&r&r' construct/s   zConstantWarmUpLR.construct)r __name__ __module__ __qualname____doc__floatintrr- __classcell__r&r&r$r'r "s r cs6eZdZdZdeeeedfdd ZddZZS) r zLinearWithWarmUpLR.r)rrrrcstt|td||}td|}||_||_t|tj|_ t|tj|_ t|tj|_ t |_t |_tdtj|_t |_dSNrr)rr rrrrrrrrr linear_stepsrrr Maximum zero_constantCastcast)r"rrrrr7r$r&r'r=s   zLinearWithWarmUpLR.__init__cCs`||tj}||j|r8||j}|j|j|}n$||j|j ||j }|j|}|Sr() r;rrr rrrrr9rr7r*r&r&r'r-Ls  zLinearWithWarmUpLR.construct)rr.r&r&r$r'r :s r cs:eZdZdZd eeeeeedfdd ZddZZS) rzCosineWithWarmUpLR.?r)rrr num_cycleslr_endrcstt|td||}td|}||_||_||_t|tj |_ t|tj |_ t|tj |_ ||_ t|_t|_tj|_t|_tdtj |_t|_dSr6)rrrrrr>rrrrrr cosine_stepsr=rrr r8mathpimath_piCoscosr9r:r;)r"rrrr=r>rr?r$r&r'r[s     zCosineWithWarmUpLR.__init__c Cs||tj}||j|r8||j}|j|j|}nR||j|j}||j dd| |j |j d|}|j |j|j |}|S)r)r<rg@)r;rrr rrrr?rr9rDrBr=r>r"r+r,rZprogressr&r&r'r-ns $zCosineWithWarmUpLR.construct)r<rrr.r&r&r$r'rXs rcs:eZdZdZd eeeeeedfdd ZddZZS) rzCosineWithRestartsAndWarmUpLR.r<r)rrrr=r>rcstt|td||}td|}||_||_||_t|tj |_ t|tj |_ t|tj |_ ||_ t|_t|_tj|_t|_tdtj |_tdtj |_t|_dS)Nrrr)rrrrrr>rrrrrrr?r=rrr r8r@rArBrCrDr9rr:r;)r"rrrr=r>rr?r$r&r'rs"    z&CosineWithRestartsAndWarmUpLR.__init__c Cs||tj}||j|r8||j}|j|j|}nj||j|j}||j|r| |j dd| |j |j ||j}|j|j|j|}n|j S|S)r)r<r)r;rrr rrrr?rrr9rDrBr=r>rEr&r&r'r-s "z'CosineWithRestartsAndWarmUpLR.construct)r<rrr.r&r&r$r'r|s rcs:eZdZdZd eeeeeedfdd Zdd ZZS) rzPolynomialWithWarmUpLR.Hz>rr)rrrr>powerrcstt|td||}td|}||ksDtd|d|d||_||_||_||_t |t j |_ t |t j |_ t |t j |_t|_t|_dS)Nrzlr_end (z&) must be be smaller than initial lr ())rrrr ValueErrorrrr>rGrrrrr decay_stepsrrr r:r;)r"rrrr>rGrrJr$r&r'rs  zPolynomialWithWarmUpLR.__init__cCs||tj}||j|r8||j}|j|j|}nH|j|j}d||j|j}|||j |j}||j}|j|}|S)r)r) r;rrr rrrr>rJrG)r"r+r,rZlr_rangeZ pct_remainingdecayr&r&r'r-s    z PolynomialWithWarmUpLR.construct)rFrrr.r&r&r$r'rs rcs(eZdZdZfddZddZZS)r zLearningRateWiseLayer.cstt|||_||_dSN)rr rbase_lrlr_scale)r"rMrNr$r&r'rszLearningRateWiseLayer.__init__cCs||}|j|SrL)rMrN)r"r+lrr&r&r'r-s zLearningRateWiseLayer.constructr/r0r1r2rr-r5r&r&r$r'r s r cs*eZdZdZdfdd ZddZZS) r z6 Warmup-decay learning rate for Bert network. rFcstt|d|_|dkr.d|_t|||_t|||||_t||||_ t t |g t j|_t|_t t dg t j|_t|_||_dS)NFrTr)rr r warmup_flagr warmup_lrrdecay_lrrcosine_decay_lrrnparrayastyperrrrr oner:r; use_cosine)r"rend_learning_raterrJrGrYr$r&r'rs   zWarmUpDecayLR.__init__cCsd|js||}n ||}|jr\|||j|tj}| |}|j ||||}n|}|S)zdynamic learning rate) rYrSrTrQr;r rrrrRrX)r"r+rSZ is_warmuprRrOr&r&r'r-s   zWarmUpDecayLR.construct)rFrPr&r&r$r'r sr cs6eZdZdeeeeeedfdd ZddZZS)WarmUpStableDecayLRr)rrrdecay_last_stepsr>rcstt|td|}td||}||_t|tj|_||_ ||_ t|tj|_ t|tj|_ t|tj|_ t|_t|_t|_td|d|d|dS)Nrzuse WSD scheduler total: z , warmup: z , decay: )rr[rrr\rrrdecay_start_step_idxrr>rrrrrr Mulmulr:r;print)r"rrrr\r>rrJr$r&r'rs    zWarmUpStableDecayLR.__init__cCs||j|r*||j}|j|j|}nT|||jrpt|j|jtj|j }||j|t|jtj}nt|jtj}|Sr() r rrrr]rr>rrr\)r"r+r,rgradr&r&r'r-s zWarmUpStableDecayLR.construct)rr)r/r0r1r3r4rr-r5r&r&r$r'r[s  r[) r2r@numpyrU mindspore.opsrrZmindspore.common.dtypecommondtyper#mindspore.nn.learning_rate_schedulerrrrmindspore.common.tensorrmindformers.tools.registerrr __all__registerLRr r rrrr r r[r&r&r&r's0     # ( "  " pengcheng.mind-mindformer/mindformers/core/lr/build_lr.py000066400000000000000000000076361476650141400241660ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build LR Schedule API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .lr_schedule import ConstantWarmUpLR, CosineWithWarmUpLR, \ LinearWithWarmUpLR, CosineWithRestartsAndWarmUpLR, PolynomialWithWarmUpLR, WarmUpStableDecayLR, WarmUpCosineWithDecayLR def build_lr( config: dict = None, default_args: dict = None, module_type: str = 'lr', class_name: str = None, **kwargs): r"""Build lr For MindFormer. Instantiate the lr from MindFormerRegister's registry. Args: config (dict): The task lr's config. Default: None. default_args (dict): The default argument of lr API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'lr'. class_name (str): The class name of lr API. Default: None. Return: The function instance of lr API. Examples: >>> from mindformers import build_lr >>> lr_config = {'type': 'CosineDecayLR', 'max_lr': 0.001, ... 'min_lr': 0., 'decay_steps':10} >>> # 1) use config dict to build lr >>> lr_from_config = build_lr(lr_config) >>> # 2) use class name to build lr >>> lr_class_name = build_lr(class_name='CosineDecayLR', max_lr=0.001, min_lr=0., decay_steps=10) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.LR, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_lr(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.learning_rate_schedule): if module_name.startswith('__'): continue lr_schedule = getattr(nn.learning_rate_schedule, module_name) if inspect.isclass(lr_schedule): MindFormerRegister.register_cls( lr_schedule, MindFormerModuleType.LR) def register_mf_lr(): """ register MindFomrers builtin LR class. """ # adapt huggingface MindFormerRegister.register_cls( ConstantWarmUpLR, module_type=MindFormerModuleType.LR, alias="constant_with_warmup") MindFormerRegister.register_cls( LinearWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="linear") MindFormerRegister.register_cls( CosineWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="cosine") MindFormerRegister.register_cls( CosineWithRestartsAndWarmUpLR, module_type=MindFormerModuleType.LR, alias="cosine_with_restarts") MindFormerRegister.register_cls( PolynomialWithWarmUpLR, module_type=MindFormerModuleType.LR, alias="polynomial") ############################### build for WSD ############################## MindFormerRegister.register_cls( WarmUpStableDecayLR, module_type=MindFormerModuleType.LR, alias="warmup_stable_decay") MindFormerRegister.register_cls( WarmUpCosineWithDecayLR, module_type=MindFormerModuleType.LR, alias="warmup_consin_decay") register_ms_lr() register_mf_lr() pengcheng.mind-mindformer/mindformers/core/lr/lr_schedule.py000066400000000000000000000357061476650141400246620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/PanGu-%CE%B1/utils.py # https://github.com/huggingface/transformers/blob/main/src/transformers/optimization.py # ============================================================================ """Self-Define LR Schedule.""" import math import numpy as np from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.nn.learning_rate_schedule import LearningRateSchedule, PolynomialDecayLR, WarmUpLR, CosineDecayLR from mindspore.common.tensor import Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'LearningRateWiseLayer', 'WarmUpDecayLR', 'ConstantWarmUpLR', 'LinearWithWarmUpLR', 'CosineWithWarmUpLR', 'CosineWithRestartsAndWarmUpLR', 'PolynomialWithWarmUpLR'] @MindFormerRegister.register(MindFormerModuleType.LR) class ConstantWarmUpLR(LearningRateSchedule): """ConstantWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, warmup_lr_init: float = 0., **kwargs): super(ConstantWarmUpLR, self).__init__() warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.greater = P.Greater() self.total_steps = kwargs.get('total_steps', None) def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.one_constant learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LinearWithWarmUpLR(LearningRateSchedule): """LinearWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, warmup_lr_init: float = 0.): super(LinearWithWarmUpLR, self).__init__() linear_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.linear_steps = Tensor(linear_steps, mstype.float32) self.greater = P.Greater() self.max = P.Maximum() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: percent = self.max(self.zero_constant, (self.total_steps - global_step) / self.linear_steps) learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithWarmUpLR(LearningRateSchedule): """CosineWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpCosineWithDecayLR(LearningRateSchedule): """WarmUpCosineWithDecayLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, num_cycles: float = 0.5, lr_middle: float = 0., lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpCosineWithDecayLR, self).__init__() decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) cosine_steps = max(1, total_steps - warmup_steps - decay_last_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_middle = lr_middle self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.cast = P.Cast() print(f"Use ConSin WSD scheduler total: {total_steps}, warmup: {warmup_steps}, cosin: {cosine_steps}, decay: {decay_last_steps}") print(f"learning_rate: {learning_rate}, lr_middle: {lr_middle}, lr_end： {lr_end}") def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(self.decay_start_step_idx, global_step): progress = (global_step - self.warmup_steps) / self.cosine_steps percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * self.num_cycles * 2.0 * progress))) learning_rate = self.lr_middle + (self.learning_rate - self.lr_middle) * percent else: grad = Tensor((self.lr_end - self.lr_middle), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.lr_middle, mstype.float32) return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class CosineWithRestartsAndWarmUpLR(LearningRateSchedule): """CosineWithRestartsAndWarmUpLR.""" def __init__(self, learning_rate: float, total_steps: int, warmup_steps: int, num_cycles: float = 0.5, lr_end: float = 0., warmup_lr_init: float = 0.): super(CosineWithRestartsAndWarmUpLR, self).__init__() cosine_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = warmup_lr_init self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.cosine_steps = Tensor(cosine_steps, mstype.float32) self.num_cycles = num_cycles self.greater = P.Greater() self.max = P.Maximum() self.math_pi = math.pi self.cos = P.Cos() self.zero_constant = Tensor(0.0, mstype.float32) self.one_constant = Tensor(1.0, mstype.float32) self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: progress = (global_step - self.warmup_steps) / self.cosine_steps if self.greater(self.one_constant, progress): percent = self.max( self.zero_constant, 0.5 * (1.0 + self.cos(self.math_pi * ((self.num_cycles * progress) % self.one_constant)))) learning_rate = self.lr_end + (self.learning_rate - self.lr_end) * percent else: return self.zero_constant return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class PolynomialWithWarmUpLR(LearningRateSchedule): """PolynomialWithWarmUpLR.""" def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, lr_end: float = 1e-7, power: float = 1.0, warmup_lr_init: float = 0.): super(PolynomialWithWarmUpLR, self).__init__() decay_steps = max(1, total_steps - warmup_steps) warmup_steps = max(1, warmup_steps) if not learning_rate > lr_end: raise ValueError(f"lr_end ({lr_end}) must be be smaller than initial lr ({learning_rate})") self.learning_rate = learning_rate self.warmup_lr_init = warmup_lr_init self.lr_end = lr_end self.power = power self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.decay_steps = Tensor(decay_steps, mstype.float32) self.greater = P.Greater() self.cast = P.Cast() def construct(self, global_step): """compute current step lr.""" global_step = self.cast(global_step, mstype.float32) if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent else: lr_range = self.learning_rate - self.lr_end pct_remaining = 1 - (global_step - self.warmup_steps) / self.decay_steps decay = lr_range * pct_remaining ** self.power + self.lr_end percent = decay / self.learning_rate learning_rate = self.learning_rate * percent return learning_rate @MindFormerRegister.register(MindFormerModuleType.LR) class LearningRateWiseLayer(LearningRateSchedule): """LearningRateWiseLayer.""" def __init__(self, base_lr, lr_scale): super(LearningRateWiseLayer, self).__init__() self.base_lr = base_lr self.lr_scale = lr_scale def construct(self, global_step): lr = self.base_lr(global_step) return self.lr_scale * lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpDecayLR(LearningRateSchedule): """ Warmup-decay learning rate for Bert network. """ def __init__(self, learning_rate, end_learning_rate, warmup_steps, decay_steps, power=1.0, use_cosine=False): super(WarmUpDecayLR, self).__init__() self.warmup_flag = False if warmup_steps > 0: self.warmup_flag = True self.warmup_lr = WarmUpLR(learning_rate, warmup_steps) self.decay_lr = PolynomialDecayLR(learning_rate, end_learning_rate, decay_steps, power) self.cosine_decay_lr = CosineDecayLR(end_learning_rate, learning_rate, decay_steps) self.warmup_steps = Tensor(np.array([warmup_steps]).astype(np.float32)) self.greater = P.Greater() self.one = Tensor(np.array([1.0]).astype(np.float32)) self.cast = P.Cast() self.use_cosine = use_cosine def construct(self, global_step): """dynamic learning rate""" if not self.use_cosine: decay_lr = self.decay_lr(global_step) else: decay_lr = self.cosine_decay_lr(global_step) if self.warmup_flag: is_warmup = self.cast(self.greater(self.warmup_steps, global_step), mstype.float32) warmup_lr = self.warmup_lr(global_step) lr = (self.one - is_warmup) * decay_lr + is_warmup * warmup_lr else: lr = decay_lr return lr @MindFormerRegister.register(MindFormerModuleType.LR) class WarmUpStableDecayLR(LearningRateSchedule): def __init__(self, learning_rate: float, warmup_steps: int, total_steps: int, decay_last_steps: int, lr_end: float = 0., warmup_lr_init: float = 0.): super(WarmUpStableDecayLR, self).__init__() warmup_steps = max(1, warmup_steps) decay_steps = max(1, total_steps - decay_last_steps) self.decay_last_steps = decay_last_steps self.decay_start_step_idx = Tensor(decay_steps, mstype.float32) self.learning_rate = learning_rate self.lr_end = lr_end self.warmup_lr_init = Tensor(warmup_lr_init, mstype.float32) self.total_steps = Tensor(total_steps, mstype.float32) self.warmup_steps = Tensor(warmup_steps, mstype.float32) self.greater = P.Greater() self.mul = P.Mul() self.cast = P.Cast() print(f"use WSD scheduler total: {total_steps}, warmup: {warmup_steps}, decay: {decay_last_steps}") def construct(self, global_step): """compute current step lr.""" if self.greater(self.warmup_steps, global_step): percent = global_step / self.warmup_steps learning_rate = self.warmup_lr_init + self.learning_rate * percent elif self.greater(global_step, self.decay_start_step_idx): grad = Tensor((self.lr_end - self.learning_rate), mstype.float32) / self.decay_last_steps learning_rate = (global_step - self.decay_start_step_idx) * grad + Tensor(self.learning_rate, mstype.float32) else: learning_rate = Tensor(self.learning_rate, mstype.float32) return learning_rate pengcheng.mind-mindformer/mindformers/core/metric/000077500000000000000000000000001476650141400226525ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/metric/.ipynb_checkpoints/000077500000000000000000000000001476650141400264435ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/metric/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000014401476650141400327000ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Metric.""" from .build_metric import build_metric from .metric import * __all__ = [] __all__.extend(metric.__all__) pengcheng.mind-mindformer/mindformers/core/metric/.ipynb_checkpoints/build_metric-checkpoint.py000066400000000000000000000051731476650141400336120ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Metric API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_metric( config: dict = None, default_args: dict = None, module_type: str = 'metric', class_name: str = None, **kwargs): r"""Build metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.METRIC, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_mt(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.metrics): if module_name.startswith('__'): continue ms_metric = getattr(nn.metrics, module_name) if inspect.isclass(ms_metric): MindFormerRegister.register_cls( ms_metric, MindFormerModuleType.METRIC) register_ms_mt() pengcheng.mind-mindformer/mindformers/core/metric/.ipynb_checkpoints/metric-checkpoint.py000066400000000000000000001056111476650141400324310ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """MindFormer Self-Define Metric.""" import os import sys import re import collections import json import math import string import shutil import six import jieba import numpy as np from rouge_chinese import Rouge from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction import mindspore.nn as nn import mindspore as ms from mindspore.ops import operations as P from mindspore.communication import get_group_size, get_rank from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models import BasicTokenizer from mindformers.core.loss import CrossEntropyLoss from ...dataset.labels import cluener_labels __all__ = ['EntityScore', 'SQuADMetric', 'PerplexityMetric', 'ADGENMetric', 'PromptAccMetric', 'EmF1Metric'] @MindFormerRegister.register(MindFormerModuleType.METRIC) class EntityScore(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(EntityScore, self).__init__() self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.id2label = {label_id: label for label, label_id in self.label2id.items()} self.clear() def clear(self): "Initialization." self.origins = [] self.founds = [] self.rights = [] def update(self, *inputs): """Update results for every batch""" batch_logits = inputs[0].asnumpy() batch_label_ids = inputs[1].asnumpy() batch_pred_ids = np.argmax(batch_logits, axis=2).tolist() pred_paths = [[self.id2label[id_] for id_ in pred_ids] for pred_ids in batch_pred_ids] label_paths = [[self.id2label[id_] for id_ in label_ids] for label_ids in batch_label_ids] for label_path, pre_path in zip(label_paths, pred_paths): label_entities = self.get_entities_bios(label_path) pred_entities = self.get_entities_bios(pre_path) self.origins.extend(label_entities) self.founds.extend(pred_entities) self.rights.extend([pred_entity for pred_entity in pred_entities if pred_entity in label_entities]) def eval(self): """Compute final results.""" class_info = {} origin_counter = collections.Counter([x[0] for x in self.origins]) found_counter = collections.Counter([x[0] for x in self.founds]) right_counter = collections.Counter([x[0] for x in self.rights]) for type_, count in origin_counter.items(): origin = count found = found_counter.get(type_, 0) right = right_counter.get(type_, 0) recall, precision, f1 = self.compute(origin, found, right) class_info[type_] = {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)} origin = len(self.origins) found = len(self.founds) right = len(self.rights) recall, precision, f1 = self.compute(origin, found, right) return {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)}, class_info def compute(self, origin, found, right): """Compute f1, precision and recall.""" recall = 0 if origin == 0 else (right / origin) precision = 0 if found == 0 else (right / found) f1 = 0. if recall + precision == 0 else (2 * precision * recall) / (precision + recall) return recall, precision, f1 def get_entities_bios(self, seq): """Get entities from sequence.""" chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks @MindFormerRegister.register(MindFormerModuleType.METRIC) class SQuADMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self, dataset_dir, n_best_size=20, max_answer_len=30, do_lower_case=True, temp_file_dir="./squad_temp"): self.outputs = [] self.temp_file_dir = temp_file_dir temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") self.all_examples = self._load_temp_data(temp_examples_file) self.all_features = self._load_temp_data(temp_features_file) self.dev_file_path = os.path.join(dataset_dir, "dev-v1.1.json") self.basic_tokenizer = BasicTokenizer(do_lower_case) self.n_best_size = n_best_size self.max_answer_len = max_answer_len def clear(self): """Clearing the internal evaluation result.""" return def update(self, *inputs): """Update results for every batch""" ids = inputs[0].asnumpy() start = inputs[1].asnumpy() end = inputs[2].asnumpy() batch_size = len(ids) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) for i in range(batch_size): unique_id = int(ids[i]) start_logits = [float(x) for x in start[i].flat] end_logits = [float(x) for x in end[i].flat] self.outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) def eval(self): """Compute final result""" predictions = self._get_predictions() with open(self.dev_file_path) as ds: dataset_json = json.load(ds) dataset = dataset_json['data'] f1 = exact_match = total = 0 for article in dataset: for paragraph in article['paragraphs']: for qa in paragraph['qas']: total += 1 if qa['id'] not in predictions: message = 'Unanswered question ' + qa['id'] + \ ' will receive score 0.' print(message, file=sys.stderr) continue ground_truths = list(map(lambda x: x['text'], qa['answers'])) if not ground_truths: continue prediction = predictions[qa['id']] exact_match += self._metric_max_over_ground_truths( self._exact_match_score, prediction, ground_truths) f1 += self._metric_max_over_ground_truths( self._f1_score, prediction, ground_truths) exact_match = 100.0 * exact_match / total f1 = 100.0 * f1 / total self._remove_temp_data() return {'exact_match': exact_match, 'f1': f1} def _remove_temp_data(self): shutil.rmtree(self.temp_file_dir) def _load_temp_data(self, temp_file_path): with open(temp_file_path, "r", encoding="utf-8") as f: data = [] for line in f.readlines(): data.append(json.loads(line.strip())) return data def _normalize_answer(self, s): """Lower text and remove punctuation, articles and extra whitespace.""" def remove_articles(text): return re.sub(r'\b(a|an|the)\b', ' ', text) def white_space_fix(text): return ' '.join(text.split()) def remove_punc(text): exclude = set(string.punctuation) return ''.join(ch for ch in text if ch not in exclude) def lower(text): return text.lower() return white_space_fix(remove_articles(remove_punc(lower(s)))) def _f1_score(self, prediction, ground_truth): """calculate f1 score""" prediction_tokens = self._normalize_answer(prediction).split() ground_truth_tokens = self._normalize_answer(ground_truth).split() common = collections.Counter(prediction_tokens) & collections.Counter(ground_truth_tokens) num_same = sum(common.values()) if num_same == 0: return 0 precision = 1.0 * num_same / len(prediction_tokens) recall = 1.0 * num_same / len(ground_truth_tokens) f1 = (2 * precision * recall) / (precision + recall) return f1 def _exact_match_score(self, prediction, ground_truth): return self._normalize_answer(prediction) == self._normalize_answer(ground_truth) def _metric_max_over_ground_truths(self, metric_fn, prediction, ground_truths): scores_for_ground_truths = [] for ground_truth in ground_truths: score = metric_fn(prediction, ground_truth) scores_for_ground_truths.append(score) return max(scores_for_ground_truths) def _get_predictions(self): """Get final predictions""" example_index_to_features = collections.defaultdict(list) for feature in self.all_features: example_index_to_features[feature["example_index"]].append(feature) unique_id_to_result = {} for result in self.outputs: unique_id_to_result[result.unique_id] = result all_predictions = collections.OrderedDict() for (example_index, example) in enumerate(self.all_examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result) nbest = self._get_nbest(prelim_predictions, features, example) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) nbest_json = [] for (i, entry) in enumerate(nbest): output = collections.OrderedDict() output["text"] = entry.text output["probability"] = probs[i] output["start_logit"] = entry.start_logit output["end_logit"] = entry.end_logit nbest_json.append(output) assert len(nbest_json) >= 1 all_predictions[example["qas_id"]] = nbest_json[0]["text"] return all_predictions def _get_prelim_predictions(self, features, unique_id_to_result): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature["unique_id"] not in unique_id_to_result: continue result = unique_id_to_result[feature["unique_id"]] start_indexes = self._get_best_indexes(result.start_logits) end_indexes = self._get_best_indexes(result.end_logits) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature["tokens"]): continue if end_index >= len(feature["tokens"]): continue if str(start_index) not in feature["token_to_orig_map"]: continue if str(end_index) not in feature["token_to_orig_map"]: continue if not feature["token_is_max_context"].get(str(start_index), False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > self.max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= self.n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature["tokens"][pred.start_index:(pred.end_index + 1)] orig_doc_start = feature["token_to_orig_map"][str(pred.start_index)] orig_doc_end = feature["token_to_orig_map"][str(pred.end_index)] orig_tokens = example["doc_tokens"][orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= self.n_best_size: break best_indexes.append(score[0]) return best_indexes @MindFormerRegister.register(MindFormerModuleType.METRIC) class PerplexityMetric(nn.Metric): """Compute the loss and PPL of each entity""" def __init__(self): super(PerplexityMetric, self).__init__() self.num_data = None self.total_loss = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.not_equal = P.NotEqual() self.sub = P.Sub() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_loss = 0.0 def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # # input_mask was added 1 in GPT2LMModel to avoid allgather issue in Mindspore1.10 # input_mask = self.sub(input_mask, 1) # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None avg_loss = float(self.total_loss / self.num_data) result = {"loss": avg_loss, "PPL": math.exp(avg_loss)} if self.pipeline_parallel: print("Average Loss and PPL Metric:", result) return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class ADGENMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(ADGENMetric, self).__init__() self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def clear(self): self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def update(self, *inputs): """Update results for every batch""" preds = inputs[0] # list[numpy] labels = inputs[1] # numpy if isinstance(preds, tuple): preds = preds[0] for pred, label in zip(preds, labels): print(f"pred is:\n {pred}\n", f"label is:\n {label}") hypothesis = list(jieba.cut(pred)) reference = list(jieba.cut(label)) rouge = Rouge() scores = rouge.get_scores(' '.join(hypothesis), ' '.join(reference)) result = scores[0] for k, v in result.items(): self.score_dict[k].append(round(v["f"] * 100, 4)) bleu_score = sentence_bleu([list(label)], list(pred), smoothing_function=SmoothingFunction().method3) self.score_dict["bleu-4"].append(round(bleu_score * 100, 4)) def eval(self): """Compute final result""" for k, v in self.score_dict.items(): self.score_dict[k] = float(np.mean(v)) print('metric: ADGENMetric\n' + f'rouge-1: {self.score_dict["rouge-1"]:.4f}\n' + f'rouge-2: {self.score_dict["rouge-2"]:.4f}\n' + f'rouge-l: {self.score_dict["rouge-l"]:.4f}\n' + f'bleu-4: {self.score_dict["bleu-4"]:.4f}') return self.score_dict @MindFormerRegister.register(MindFormerModuleType.METRIC) class PromptAccMetric(nn.Metric): r""" Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} """ def __init__(self): super(PromptAccMetric, self).__init__() self.num_data = None self.total_acc_num = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.last_card_id = 0 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.equal = P.Equal() self.softmax = P.Softmax() self.argmin = P.Argmin() self.sum = P.ReduceSum() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_acc_num = 0 def calculate_circle(self, *inputs): """The main calculate logic.""" logits, input_ids, input_mask, labels = inputs[0], inputs[1], inputs[2], inputs[3] batch_size, num_labels, seq_length, _ = logits.shape logits = self.reshape(logits, (batch_size*num_labels, seq_length, -1)) ppl_list = [] for index in range(batch_size*num_labels): sub_logits, sub_tokens, sub_mask_list = logits[index], input_ids[index], input_mask[index] sub_logits = sub_logits[:-1, ::] sub_tokens = sub_tokens[1:] sub_mask_list = sub_mask_list[1:] loss = self.loss(sub_logits, sub_tokens, sub_mask_list) loss = float(loss.asnumpy()) ppl_list.append(loss) # smaller, better ppl_ms = ms.Tensor(ppl_list, dtype=ms.float32) ppl_ms = self.reshape(ppl_ms, (batch_size, num_labels)) ppl_ms = self.cast(self.argmin(ppl_ms), ms.int32) label = self.reshape(labels, (-1,)) cur_acc_num = self.cast(self.equal(ppl_ms, label), ms.float16).sum().asnumpy() self.num_data += batch_size self.total_acc_num += cur_acc_num def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) self.calculate_circle(*inputs) if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: self.calculate_circle(*inputs) print("Current data num is {}, total acc num is {}, ACC is {}".format( self.num_data, self.total_acc_num, "%.3f" % (self.total_acc_num / self.num_data))) return def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None acc_rate = float(self.total_acc_num / self.num_data) result = {"Acc": acc_rate} print(f"Acc: {('%.3f' % result.get('Acc', 0))}, total_acc_num: {self.total_acc_num}, " f"total_num: {self.num_data}") return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class EmF1Metric(nn.Metric): """ Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. """ def __init__(self): super(EmF1Metric, self).__init__() self.gens = None self.labels = None self.metrics = None self.num_data = None def clear(self): """Clearing the internal evaluation result.""" self.gens = [] self.labels = [] self.metrics = { 'Em': 0.0, 'F1': 0.0 } self.num_data = 0 def update(self, *inputs): """Update results for every batch""" gen, label = inputs[0], inputs[1] for i in range(len(gen)): gen[i] = gen[i].strip() gen[i] = gen[i].split("\n")[0] print(f"pred is:\n {gen}\n", f"label is:\n {label}") self.gens.extend(gen) self.labels.extend(label) self.num_data += len(gen) result, current_count = self.evaluate_pairs(gen, label) print("The F1/Em of this example is: ", result) if self.num_data % 10 == 0: result, current_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, current_count: {current_count}") def eval(self): """Compute final result""" result, total_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, total_count: {total_count}") return result def mixed_segmentation(self, in_str, rm_punc=False): """cut input for calculating lcs""" in_str = str(in_str).lower().strip() segs_out = [] temp_str = "" sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] for char in in_str: if rm_punc and char in sp_char: continue if re.search(r'[\u4e00-\u9fa5]', char) or char in sp_char: if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) temp_str = "" segs_out.append(char) else: temp_str += char if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) return segs_out def remove_punctuation(self, in_str): """remove punctuations in inputs""" in_str = str(in_str).lower().strip() sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] out_segs = [] for char in in_str: if char in sp_char: continue else: out_segs.append(char) return ''.join(out_segs) def find_lcs(self, s1, s2): """calculate the length of lcs""" m = [[0 for i in range(len(s2) + 1)] for j in range(len(s1) + 1)] mmax = 0 p = 0 for i in range(len(s1)): for j in range(len(s2)): if s1[i] == s2[j]: m[i + 1][j + 1] = m[i][j] + 1 if m[i + 1][j + 1] > mmax: mmax = m[i + 1][j + 1] p = i + 1 return s1[p - mmax:p], mmax def calc_f1_score(self, answers, prediction): """calculate f1 score""" f1_scores = [] for ans in answers: ans_segs = self.mixed_segmentation(ans, rm_punc=True) prediction_segs = self.mixed_segmentation(prediction, rm_punc=True) _, lcs_len = self.find_lcs(ans_segs, prediction_segs) if lcs_len == 0: f1_scores.append(0) continue precision = 1.0 * lcs_len / len(prediction_segs) recall = 1.0 * lcs_len / len(ans_segs) f1 = (2 * precision * recall) / (precision + recall) f1_scores.append(f1) return max(f1_scores) def calc_em_score(self, answers, prediction): """calculate em score""" em = 0 for ans in answers: ans_ = self.remove_punctuation(ans) prediction_ = self.remove_punctuation(prediction) if ans_ == prediction_: em = 1 break return em def evaluate_pairs(self, pred_, ans_): """calculate metric""" f1 = 0 em = 0 total_count = 0 for (prediction, answer) in zip(pred_, ans_): total_count += 1 f1 += self.calc_f1_score([answer], prediction) em += self.calc_em_score([answer], prediction) if total_count > 0: f1_score = 100.0 * f1 / total_count em_score = 100.0 * em / total_count result = {'F1': f1_score, 'Em': em_score} else: print("total_count is zero") result = {} return result, total_count pengcheng.mind-mindformer/mindformers/core/metric/__init__.py000066400000000000000000000014401476650141400247620ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Metric.""" from .build_metric import build_metric from .metric import * __all__ = [] __all__.extend(metric.__all__) pengcheng.mind-mindformer/mindformers/core/metric/__pycache__/000077500000000000000000000000001476650141400250625ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/metric/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004511476650141400312520ustar00rootroot00000000000000a ?cf @s,dZddlmZddlTgZeejdS)zMindFormers Metric.) build_metric)*N)__doc__rmetric__all__extendrrK/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/__init__.pys pengcheng.mind-mindformer/mindformers/core/metric/__pycache__/build_metric.cpython-39.pyc000066400000000000000000000037551476650141400321670ustar00rootroot00000000000000a ?cf{ @sTdZddlZddlmZddlmZmZmZd eee e dddZ d d Z e dS) zBuild Metric API.N)nn)MindFormerRegisterMindFormerModuleTypeMindFormerConfigmetric)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)agBuild metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') N)r) isinstancedictrrget_instance_from_cfgrMETRIC get_instance)rrr r kwargsrO/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/build_metric.py build_metricsrcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z& register MindSpore builtin LR class. __N) dirrmetrics startswithgetattrinspectisclassr register_clsrr) module_nameZ ms_metricrrrregister_ms_mt8s   r)NNrN) __doc__r mindsporermindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-mindformer/mindformers/core/metric/__pycache__/metric.cpython-39.pyc000066400000000000000000000647731476650141400310170ustar00rootroot00000000000000a Qff@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl mZddlmZmZddlmZddlZddlmZddlmZmZddlmZmZddlm Z dd l!m"Z"d d l#m$Z$gd Z%e&ej'Gd ddej(Z)e&ej'Gdddej(Z*e&ej'Gdddej(Z+e&ej'Gdddej(Z,e&ej'Gdddej(Z-e&ej'Gdddej(Z.dS)zMindFormer Self-Define Metric.N)Rouge) sentence_bleuSmoothingFunction) operations)get_group_sizeget_rank)MindFormerRegisterMindFormerModuleType)BasicTokenizer)CrossEntropyLoss)cluener_labels) EntityScore SQuADMetricPerplexityMetric ADGENMetricPromptAccMetric EmF1MetriccsHeZdZdZfddZddZddZdd Zd d Zd d Z Z S)r9Compute the f1, precision and recall score of each entitycsDtt|ddttD|_dd|jD|_|dS)NcSsi|]\}}||qSr).0label_idlabelrrI/home/ma-user/work/mindformers-baichuan/mindformers/core/metric/metric.py 2z(EntityScore.__init__..cSsi|]\}}||qSrr)rrrrrrr3r) superr__init__ enumerater Zlabel2iditemsid2labelclearself __class__rrr0szEntityScore.__init__cCsg|_g|_g|_dS)zInitialization.N)originsfoundsrightsr"rrrr!6szEntityScore.clearc s|d}|d}tj|dd}fdd|D}fdd|D}t||D]N\}}||} jj| j fdd| DqXd S) Update results for every batchr)axiscsg|]}fdd|DqS)csg|]}j|qSrr rid_r"rr Br1EntityScore.update...r)rZpred_idsr"rrr0Brz&EntityScore.update..csg|]}fdd|DqS)csg|]}j|qSrr-r.r"rrr0Crr1r)rZ label_idsr"rrr0Crcsg|]}|vr|qSrr)rZ pred_entity)label_entitiesrrr0JrN) asnumpynpargmaxtolistzipget_entities_biosr&extendr'r() r#inputsZ batch_logitsZbatch_label_idsZbatch_pred_idsZ pred_pathsZ label_pathsZ label_pathZpre_pathZ pred_entitiesr)r2r#rupdate<s      zEntityScore.updatec Csi}tdd|jD}tdd|jD}tdd|jD}|D]Z\}}|}||d}||d} |||| \} } } t| dt| dt| dd||<qNt |j}t |j}t |j} |||| \} } } t| dt| dt| dd|fS)zCompute final results.cSsg|] }|dqSrrrxrrrr0Orz$EntityScore.eval..cSsg|] }|dqSr<rr=rrrr0PrcSsg|] }|dqSr<rr=rrrr0Qrr) precisionrecallf1) collectionsCounterr&r'r(rgetcomputeroundlen) r#Z class_infoZorigin_counterZ found_counterZ right_countertype_countoriginfoundrightrAr@rBrrrevalLs  $   zEntityScore.evalcCsV|dkr dn||}|dkr dn||}||dkr8dnd||||}|||fS)z!Compute f1, precision and recall.rr+r)r#rKrLrMrAr@rBrrrrF^s$zEntityScore.computecCs>g}gd}t|D]"\}}|drz|ddkr>||gd}||d<||d<|dd|d<||gd}|dr|ddkr||gd}||d<|dd|d<q|d r|ddkr|dd}||dkr||d<|t|dkr8||q|ddkr0||gd}q|S) zGet entities from sequence.)rPrPzS-r+rPr*-rzB-zI-)r startswithappendsplitrH)r#seqchunkschunkindxtagZ entity_typerrrr8es8           zEntityScore.get_entities_bios) __name__ __module__ __qualname____doc__rr!r;rNrFr8 __classcell__rrr$rr,s rc@seZdZdZd'ddZdd Zd d Zd d ZddZddZ ddZ ddZ ddZ ddZ ddZddZddZd d!Zd"d#Zd$d%Zd&S)(rrT ./squad_tempcCsjg|_||_tj|d}tj|d}|||_|||_tj|d|_t ||_ ||_ ||_ dS)Nztemp_examples.jsonztemp_features.jsonz dev-v1.1.json) outputs temp_file_dirospathjoin_load_temp_data all_examples all_features dev_file_pathr basic_tokenizer n_best_sizemax_answer_len)r# dataset_dirrlrmZ do_lower_casercZtemp_examples_fileZtemp_features_filerrrrs   zSQuADMetric.__init__cCsdS)(Clearing the internal evaluation result.Nrr"rrrr!szSQuADMetric.clearc Gs|d}|d}|d}t|}tdgd}t|D]N}t||}dd||jD} dd||jD} |j||| | dqDd S) r)rr*r+ RawResult) unique_id start_logits end_logitscSsg|] }t|qSrfloatr=rrrr0rz&SQuADMetric.update..cSsg|] }t|qSrrtr=rrrr0rN) r3rHrC namedtuplerangeintflatrbrS) r#r:idsstartend batch_sizerpirqrrrsrrrr;s      zSQuADMetric.updatec Cs$|}t|j"}t|}|d}Wdn1s:0Yd}}}|D]}|dD]} | dD]} |d7}| d|vrd| dd } t| tjd qltt d d | d } | sql|| d} || |j | | 7}|| |j | | 7}qlq`qTd||}d||}| ||dS)Compute final resultdataNrZ paragraphsZqasr*idzUnanswered question z will receive score 0.)filecSs|dS)Ntextrr>rrrrz"SQuADMetric.eval..answersY@) exact_matchrB)_get_predictionsopenrjjsonloadprintsysstderrlistmap_metric_max_over_ground_truths_exact_match_score _f1_score_remove_temp_data)r# predictionsds dataset_jsondatasetrBrtotalZarticleZ paragraphZqamessage ground_truths predictionrrrrNs:  &         zSQuADMetric.evalcCst|jdSN)shutilrmtreercr"rrrrszSQuADMetric._remove_temp_datacCsXt|ddd6}g}|D]}|t|qWdn1sJ0Y|S)Nrzutf-8)encoding)r readlinesrSrloadsstrip)r#Ztemp_file_pathfrlinerrrrgs  4zSQuADMetric._load_temp_datacCs4dd}dd}dd}dd}|||||S) zALower text and remove punctuation, articles and extra whitespace.cSstdd|S)Nz\b(a|an|the)\b )resubrrrrremove_articlessz6SQuADMetric._normalize_answer..remove_articlescSsd|S)Nr)rfrTrrrrwhite_space_fixsz6SQuADMetric._normalize_answer..white_space_fixcs"ttjdfdd|DS)Nc3s|]}|vr|VqdSrr)rchexcluderr rzESQuADMetric._normalize_answer..remove_punc..)setstring punctuationrfrrrr remove_puncs z2SQuADMetric._normalize_answer..remove_punccSs|Sr)lowerrrrrrsz,SQuADMetric._normalize_answer..lowerr)r#srrrrrrr_normalize_answers zSQuADMetric._normalize_answerc Cs||}||}t|t|@}t|}|dkrHdSd|t|}d|t|}d||||} | S)calculate f1 scorer?r+)rrTrCrDsumvaluesrH) r#r ground_truthZprediction_tokensZground_truth_tokenscommonZnum_samer@rArBrrrrs zSQuADMetric._f1_scorecCs||||kSr)r)r#rrrrrrszSQuADMetric._exact_match_scorecCs*g}|D]}|||}||qt|Sr)rSmax)r#Z metric_fnrrZscores_for_ground_truthsrscorerrrrs   z*SQuADMetric._metric_max_over_ground_truthscCs8tt}|jD]}||d|qi}|jD]}|||j<q2t}t|j D]\}}||}| ||} | | ||} g} d} | D]$} | | j | j | s| jr| } q|| }g}t| D]D\}} t}| j|d<|||d<| j |d<| j |d<||qt|dksJ|dd||d <qT|S) zGet final predictions example_indexNr probability start_logit end_logitr*rZqas_id)rC defaultdictrrirSrbrq OrderedDictrrh_get_prelim_predictions _get_nbestrrr_compute_softmaxrH)r#Zexample_index_to_featuresfeatureunique_id_to_resultresultZall_predictionsrexamplefeaturesprelim_predictionsnbestZ total_scoresZbest_non_null_entryentryprobsZ nbest_jsonr~outputrrrrs<           zSQuADMetric._get_predictionsc Cs&tdgd}g}t|D]\}}|d|vr2q||d}||j}||j} |D]} | D]} | t|dkrxqb| t|dkrqbt| |dvrqbt| |dvrqb|dt| dsqb| | krqb| | d} | |j krqb| ||| | |j| |j| dqbqZqt |d d d d }|S) zget prelim predictionsZPrelimPrediction) feature_index start_index end_indexrrrqtokenstoken_to_orig_mapZtoken_is_max_contextFr*cSs |j|jSr)rrrrrrrSrz5SQuADMetric._get_prelim_predictions..Tkeyreverse) rCrvr_get_best_indexesrrrsrHstrrErmrSsorted) r#rrZ_PrelimPredictionrrrrZ start_indexesZ end_indexesrrlengthrrrr'sV       z#SQuADMetric._get_prelim_predictionscCsRtdgd}i}g}|D]}t||jkr6q"||j}|jdkr|d|j|jd} |dt|j} |dt|j} |d| | d} d| } | d d } | d d } | } d| } d| }| | |}||vrqd ||<n d }d ||<| |||j|jdq|s<| |d dddt|dksNJ|S)zget nbest predictionsZNbestPrediction)rrrrrr*rZ doc_tokensrz ##rz##TemptyrO)rCrvrHrlrrrrrfreplacerrT_get_final_textrSrr)r#rrrZ_NbestPredictionZseen_predictionsrpredrZ tok_tokensZorig_doc_startZ orig_doc_endZ orig_tokenstok_text orig_textZ final_textrrrrXsJ         zSQuADMetric._get_nbestcCs||sgSd}|D]}|dus$||kr|}qg}d}|D]$}t||}||||7}q6g}|D]}|||qd|S)z,Compute softmax probability over raw logits.NrO)mathexprS)r#scores max_scorerZ exp_scoresZ total_sumr>rrrrrs   zSQuADMetric._compute_softmaxcCsdd}d|j|}||}|dkr0|S|t|d}||\}}||\} } t|t| krl|Si} t| D]\} } | | | <qzd}|| vr| |}||vr||}|dur|Sd}|| vr| |}||vr||}|dur|S|||d}|S)z;Project the tokenized prediction back to the original text.cSsPg}t}t|D](\}}|dkr&q||t|<||qd|}||fS)Nrr)rCrrrHrSrf)rZns_charsZ ns_to_s_mapr~cZns_textrrr _strip_spacess   z2SQuADMetric._get_final_text.._strip_spacesrrPr*N)rfrktokenizefindrHsix iteritems)r#Z pred_textrrrstart_positionZ end_positionZ orig_ns_textZorig_ns_to_s_mapZ tok_ns_textZtok_ns_to_s_mapZtok_s_to_ns_mapr~Z tok_indexZorig_start_positionZns_start_positionZorig_end_positionZns_end_positionZ output_textrrrrs:     zSQuADMetric._get_final_textcCsLtt|dddd}g}t|D]$\}}||jkr8qH||dq"|S)z"Get the n-best logits from a list.cSs|dS)Nr*rrrrrrrz/SQuADMetric._get_best_indexes..Trr)rrrlrS)r#logitsZindex_and_scoreZ best_indexesr~rrrrrs zSQuADMetric._get_best_indexesN)r_r`Tra)rZr[r\r]rr!r;rNrrgrrrrrrrrrrrrrrrs&   (114rcs8eZdZdZfddZddZddZdd ZZS) rz'Compute the loss and PPL of each entitycstt|d|_d|_t|_td|_ |j dk|_ d|_ d|_ t |_t |_t |_t |_|j rt|_ t|_ |j |j }|j |}||j dk|_td|_td|_|jdv|_dSNpipeline_stagesr*r parallel_mode full_batch)semi_auto_parallel auto_parallel)rrrnum_data total_lossr lossmsget_auto_parallel_contextrpipeline_parallelrank_id device_numPCastcastReshapereshapeNotEqual not_equalSubrrr is_last_stagerrrr#Zper_stage_device_numstage_idr$rrrs*          zPerplexityMetric.__init__cCsd|_d|_dS)rorrON)rrr"rrrr!szPerplexityMetric.clearcGs|jrr|jsdS|jr&tjjddd|d}t|}|j|7_|j d7_ |jrtj|j d|j dn0|d}t|}|j|7_|j d7_ dS) r)N data_parallelFrrrr*Trrr) rr rrcontextset_auto_parallel_contextrur3rrrr)r#r:rrrrr;s$    zPerplexityMetric.updatecCsT|jr|jsdSt|j|j}|t|d}|jrP|j|jdkrPt d||S)rN)rZPPLr*zAverage Loss and PPL Metric:) rr rurrrrrrr)r#Zavg_lossrrrrrN,s  zPerplexityMetric.eval rZr[r\r]rr!r;rNr^rrr$rrs  *rcs8eZdZdZfddZddZddZdd ZZS) rrcs"tt|ggggd|_dSN)rouge-1rouge-2rouge-lbleu-4)rrr score_dictr"r$rrr;s zADGENMetric.__init__cCsggggd|_dSr)rr"rrrr!Ds zADGENMetric.clearcGs|d}|d}t|tr"|d}t||D]\}}td|dd|tt|}tt|}t}|d |d |} | d} | D]&\} } |j |  t | ddd qtt|gt|tjd } |j d  t | dd q,d S) r)rr* pred is:  label is: rrdr?)Zsmoothing_functionrN) isinstancetupler7rrjiebacutrZ get_scoresrfrrrSrGrrZmethod3)r#r:predslabelsrr hypothesisZ referenceZrougerrkv bleu_scorerrrr;Ls"   zADGENMetric.updatecCs|jD]\}}tt||j|<q tdd|jdddd|jdddd|jd ddd |jd d|jS) rzmetric: ADGENMetric z rouge-1: rz.4frz rouge-2: rz rouge-l: rz bleu-4: r)rrrur4meanr)r#r#r$rrrrNbszADGENMetric.evalrrrr$rr7s  rcs@eZdZdZfddZddZddZdd Zd d ZZ S) ru Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} cstt|d|_d|_t|_td|_ |j dk|_ d|_ d|_ d|_ t|_t|_t|_t|_t|_t|_|j rt|_ t|_ |j |j }|j |}||j dk|_td|_td|_|jdv|_ dSr)!rrrr total_acc_numr rrrrrZ last_card_idrrrrrrrEqualequalSoftmaxsoftmaxArgminargmin ReduceSumrrrr rrrr r$rrrs0            zPromptAccMetric.__init__cCsd|_d|_dS)rorN)rr'r"rrrr!szPromptAccMetric.clearcGsH|d|d|d|df\}}}}|j\}}}} |||||df}g} t||D]p} || || || } } }| ddddf} | dd} |dd}|| | |}t|}| |qXtj| tj d}||||f}| | |tj }||d}| | ||tj}|j|7_|j|7_dS) zThe main calculate logic.rr*r+r rPN)dtype)rP)shaperrwrrur3rSrTensorfloat32rr-int32r)float16rrr')r#r:r input_ids input_maskr!r}Z num_labels seq_length_Zppl_listindexZ sub_logitsZ sub_tokensZ sub_mask_listrZppl_msrZ cur_acc_numrrrcalculate_circles&$     z PromptAccMetric.calculate_circlecGs||jrL|jsdS|jr&tjjddd|j||jrVtj|jd|jdn |j|t d |j |j d|j |j dS) r)Nr Fr Trz6Current data num is {}, total acc num is {}, ACC is {}%.3f) rr rrrrr:rrrformatrr')r#r:rrrr;s   zPromptAccMetric.updatecCsV|jr|jsdSt|j|j}d|i}tdd|ddd|jd|j|S)rNZAcczAcc: r;rz, total_acc_num: z , total_num: )rr rur'rrrE)r#Zacc_raterrrrrNs zPromptAccMetric.eval) rZr[r\r]rr!r:r;rNr^rrr$rrns  rcsjeZdZdZfddZddZddZdd Zdd d Zd dZ ddZ ddZ ddZ ddZ ZS)ru Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. cs*tt|d|_d|_d|_d|_dSr)rrrgensr!metricsrr"r$rrrs zEmF1Metric.__init__cCs"g|_g|_ddd|_d|_dS)rorO)EmF1rN)r=r!r>rr"rrrr!s zEmF1Metric.clearcGs|d|d}}tt|D]*}||||<||dd||<qtd|dd||j||j||jt|7_| ||\}}td||jddkr| |j|j\}}td| d dd | d dd |d S)r)rr*rrrzThe F1/Em of this example is: F1 score: r@ , Em score: r?z, current_count: N) rwrHrrTrr=r9r!revaluate_pairsrE)r#r:genrr~r current_countrrrr;s    zEmF1Metric.updatecCsB||j|j\}}td|ddd|ddd||S)rrBr@rrCr?z, total_count: )rDr=r!rrE)r#r total_countrrrrNs*zEmF1Metric.evalFcCst|}g}d}gd}|D]^}|r6||vr6q$td|sJ||vrz|dkrntt|}||d}| |q$||7}q$|dkrtt|}|||S)zcut input for calculating lcsr!rQ:r8*^/\~`+=u，u。u：u？u！u“u”u；u’u《u》u……·u、u「u」u（u）u－u～u『u』z[\u4e00-\u9fa5]) rrrrsearchrrrr9rS)r#in_strrm_puncZsegs_outZtemp_strsp_charcharssrrrmixed_segmentations$     zEmF1Metric.mixed_segmentationcCsFt|}gd}g}|D]}||vr0q q ||q d|S)zremove punctuations in inputsrHr)rrrrSrf)r#rTrVZout_segsrWrrrremove_punctuation5s zEmF1Metric.remove_punctuationcsfddtt|dD}d}d}tt|D]z}ttD]h}|||krB|||d||d|d<||d|d|krB||d|d}|d}qBq2|||||fS)zcalculate the length of lcscs&g|]}ddttdDqS)cSsg|]}dqSr<r)rr~rrrr0Erz2EmF1Metric.find_lcs...r*rwrH)rjs2rrr0Erz'EmF1Metric.find_lcs..r*rr[)r#s1r^mmmaxpr~r\rr]rfind_lcsCs  zEmF1Metric.find_lcsc Csg}|D]}|j|dd}|j|dd}|||\}}|dkrL|dqd|t|} d|t|} d| | | | } || qt|S)rT)rUrrr+)rYrcrSrHr) r#rrZ f1_scoresansZans_segsZprediction_segsr8Zlcs_lenr@rArBrrr calc_f1_scoreQs  zEmF1Metric.calc_f1_scorecCs6d}|D](}||}||}||krd}q2q|S)zcalculate em scorerr*)rZ)r#rremrdans_Z prediction_rrr calc_em_scoreas  zEmF1Metric.calc_em_scorec Csd}d}d}t||D]4\}}|d7}|||g|7}|||g|7}q|dkrxd||}d||} || d} n tdi} | |fS)zcalculate metricrr*r)r@r?ztotal_count is zero)r7rerhr) r#Zpred_rgrBrfrGrZanswerf1_scoreZem_scorerrrrrDls   zEmF1Metric.evaluate_pairs)F)rZr[r\r]rr!r;rNrYrZrcrerhrDr^rrr$rrs    r)/r]rdrrrCrrrrrrnumpyr4Z rouge_chineserZnltk.translate.bleu_scorerr mindspore.nnnn mindsporer mindspore.opsrrmindspore.communicationrrmindformers.tools.registerrr mindformers.modelsr Zmindformers.core.lossr Zdataset.labelsr __all__registerMETRICMetricrrrrrrrrrrsH       Y Z W 6 p pengcheng.mind-mindformer/mindformers/core/metric/build_metric.py000066400000000000000000000051731476650141400256740ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Metric API.""" import inspect from mindspore import nn from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_metric( config: dict = None, default_args: dict = None, module_type: str = 'metric', class_name: str = None, **kwargs): r"""Build metric For MindFormer. Instantiate the metric from MindFormerRegister's registry. Args: config (dict, list): The task metric's config. Default: None. default_args (dict): The default argument of metric API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'metric'. class_name (str): The class name of metric API. Default: None. Return: The function instance of metric API. Examples: >>> from mindformers import build_metric >>> metric_config = {'type': 'Accuracy', 'eval_type': 'classification'} >>> # 1) use config dict to build metric >>> metric_from_config = build_metric(metric_config) >>> # 2) use class name to build metric >>> metric_class_name = build_metric(class_name='Accuracy', eval_type='classification') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.METRIC, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_mt(): """ register MindSpore builtin LR class. """ for module_name in dir(nn.metrics): if module_name.startswith('__'): continue ms_metric = getattr(nn.metrics, module_name) if inspect.isclass(ms_metric): MindFormerRegister.register_cls( ms_metric, MindFormerModuleType.METRIC) register_ms_mt() pengcheng.mind-mindformer/mindformers/core/metric/metric.py000066400000000000000000001056651476650141400245240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """MindFormer Self-Define Metric.""" import os import sys import re import collections import json import math import string import shutil import six import jieba import numpy as np from rouge_chinese import Rouge from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction import mindspore.nn as nn import mindspore as ms from mindspore.ops import operations as P from mindspore.communication import get_group_size, get_rank from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models import BasicTokenizer from mindformers.core.loss import CrossEntropyLoss from ...dataset.labels import cluener_labels __all__ = ['EntityScore', 'SQuADMetric', 'PerplexityMetric', 'ADGENMetric', 'PromptAccMetric', 'EmF1Metric'] @MindFormerRegister.register(MindFormerModuleType.METRIC) class EntityScore(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(EntityScore, self).__init__() self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.id2label = {label_id: label for label, label_id in self.label2id.items()} self.clear() def clear(self): "Initialization." self.origins = [] self.founds = [] self.rights = [] def update(self, *inputs): """Update results for every batch""" batch_logits = inputs[0].asnumpy() batch_label_ids = inputs[1].asnumpy() batch_pred_ids = np.argmax(batch_logits, axis=2).tolist() pred_paths = [[self.id2label[id_] for id_ in pred_ids] for pred_ids in batch_pred_ids] label_paths = [[self.id2label[id_] for id_ in label_ids] for label_ids in batch_label_ids] for label_path, pre_path in zip(label_paths, pred_paths): label_entities = self.get_entities_bios(label_path) pred_entities = self.get_entities_bios(pre_path) self.origins.extend(label_entities) self.founds.extend(pred_entities) self.rights.extend([pred_entity for pred_entity in pred_entities if pred_entity in label_entities]) def eval(self): """Compute final results.""" class_info = {} origin_counter = collections.Counter([x[0] for x in self.origins]) found_counter = collections.Counter([x[0] for x in self.founds]) right_counter = collections.Counter([x[0] for x in self.rights]) for type_, count in origin_counter.items(): origin = count found = found_counter.get(type_, 0) right = right_counter.get(type_, 0) recall, precision, f1 = self.compute(origin, found, right) class_info[type_] = {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)} origin = len(self.origins) found = len(self.founds) right = len(self.rights) recall, precision, f1 = self.compute(origin, found, right) return {"precision": round(precision, 4), 'recall': round(recall, 4), 'f1': round(f1, 4)}, class_info def compute(self, origin, found, right): """Compute f1, precision and recall.""" recall = 0 if origin == 0 else (right / origin) precision = 0 if found == 0 else (right / found) f1 = 0. if recall + precision == 0 else (2 * precision * recall) / (precision + recall) return recall, precision, f1 def get_entities_bios(self, seq): """Get entities from sequence.""" chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks @MindFormerRegister.register(MindFormerModuleType.METRIC) class SQuADMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self, dataset_dir, n_best_size=20, max_answer_len=30, do_lower_case=True, temp_file_dir="./squad_temp"): self.outputs = [] self.temp_file_dir = temp_file_dir temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") self.all_examples = self._load_temp_data(temp_examples_file) self.all_features = self._load_temp_data(temp_features_file) self.dev_file_path = os.path.join(dataset_dir, "dev-v1.1.json") self.basic_tokenizer = BasicTokenizer(do_lower_case) self.n_best_size = n_best_size self.max_answer_len = max_answer_len def clear(self): """Clearing the internal evaluation result.""" return def update(self, *inputs): """Update results for every batch""" ids = inputs[0].asnumpy() start = inputs[1].asnumpy() end = inputs[2].asnumpy() batch_size = len(ids) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) for i in range(batch_size): unique_id = int(ids[i]) start_logits = [float(x) for x in start[i].flat] end_logits = [float(x) for x in end[i].flat] self.outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) def eval(self): """Compute final result""" predictions = self._get_predictions() with open(self.dev_file_path) as ds: dataset_json = json.load(ds) dataset = dataset_json['data'] f1 = exact_match = total = 0 for article in dataset: for paragraph in article['paragraphs']: for qa in paragraph['qas']: total += 1 if qa['id'] not in predictions: message = 'Unanswered question ' + qa['id'] + \ ' will receive score 0.' print(message, file=sys.stderr) continue ground_truths = list(map(lambda x: x['text'], qa['answers'])) if not ground_truths: continue prediction = predictions[qa['id']] exact_match += self._metric_max_over_ground_truths( self._exact_match_score, prediction, ground_truths) f1 += self._metric_max_over_ground_truths( self._f1_score, prediction, ground_truths) exact_match = 100.0 * exact_match / total f1 = 100.0 * f1 / total self._remove_temp_data() return {'exact_match': exact_match, 'f1': f1} def _remove_temp_data(self): shutil.rmtree(self.temp_file_dir) def _load_temp_data(self, temp_file_path): with open(temp_file_path, "r", encoding="utf-8") as f: data = [] for line in f.readlines(): data.append(json.loads(line.strip())) return data def _normalize_answer(self, s): """Lower text and remove punctuation, articles and extra whitespace.""" def remove_articles(text): return re.sub(r'\b(a|an|the)\b', ' ', text) def white_space_fix(text): return ' '.join(text.split()) def remove_punc(text): exclude = set(string.punctuation) return ''.join(ch for ch in text if ch not in exclude) def lower(text): return text.lower() return white_space_fix(remove_articles(remove_punc(lower(s)))) def _f1_score(self, prediction, ground_truth): """calculate f1 score""" prediction_tokens = self._normalize_answer(prediction).split() ground_truth_tokens = self._normalize_answer(ground_truth).split() common = collections.Counter(prediction_tokens) & collections.Counter(ground_truth_tokens) num_same = sum(common.values()) if num_same == 0: return 0 precision = 1.0 * num_same / len(prediction_tokens) recall = 1.0 * num_same / len(ground_truth_tokens) f1 = (2 * precision * recall) / (precision + recall) return f1 def _exact_match_score(self, prediction, ground_truth): return self._normalize_answer(prediction) == self._normalize_answer(ground_truth) def _metric_max_over_ground_truths(self, metric_fn, prediction, ground_truths): scores_for_ground_truths = [] for ground_truth in ground_truths: score = metric_fn(prediction, ground_truth) scores_for_ground_truths.append(score) return max(scores_for_ground_truths) def _get_predictions(self): """Get final predictions""" example_index_to_features = collections.defaultdict(list) for feature in self.all_features: example_index_to_features[feature["example_index"]].append(feature) unique_id_to_result = {} for result in self.outputs: unique_id_to_result[result.unique_id] = result all_predictions = collections.OrderedDict() for (example_index, example) in enumerate(self.all_examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result) nbest = self._get_nbest(prelim_predictions, features, example) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) nbest_json = [] for (i, entry) in enumerate(nbest): output = collections.OrderedDict() output["text"] = entry.text output["probability"] = probs[i] output["start_logit"] = entry.start_logit output["end_logit"] = entry.end_logit nbest_json.append(output) assert len(nbest_json) >= 1 all_predictions[example["qas_id"]] = nbest_json[0]["text"] return all_predictions def _get_prelim_predictions(self, features, unique_id_to_result): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature["unique_id"] not in unique_id_to_result: continue result = unique_id_to_result[feature["unique_id"]] start_indexes = self._get_best_indexes(result.start_logits) end_indexes = self._get_best_indexes(result.end_logits) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature["tokens"]): continue if end_index >= len(feature["tokens"]): continue if str(start_index) not in feature["token_to_orig_map"]: continue if str(end_index) not in feature["token_to_orig_map"]: continue if not feature["token_is_max_context"].get(str(start_index), False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > self.max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= self.n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature["tokens"][pred.start_index:(pred.end_index + 1)] orig_doc_start = feature["token_to_orig_map"][str(pred.start_index)] orig_doc_end = feature["token_to_orig_map"][str(pred.end_index)] orig_tokens = example["doc_tokens"][orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= self.n_best_size: break best_indexes.append(score[0]) return best_indexes @MindFormerRegister.register(MindFormerModuleType.METRIC) class PerplexityMetric(nn.Metric): """Compute the loss and PPL of each entity""" def __init__(self): super(PerplexityMetric, self).__init__() self.num_data = None self.total_loss = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.not_equal = P.NotEqual() self.sub = P.Sub() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_loss = 0.0 def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # # input_mask was added 1 in GPT2LMModel to avoid allgather issue in Mindspore1.10 # input_mask = self.sub(input_mask, 1) # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: # logits, labels, input_mask = inputs[0], inputs[1], inputs[2] # batch_size, seq_length, _ = logits.shape # logits = self.reshape(logits[::, :-1, ::], (batch_size * (seq_length - 1), -1)) # labels = self.reshape(labels[::, 1:], (-1,)) # input_mask = self.reshape(input_mask[::, 1:], (-1,)) # loss = self.loss(logits, labels, input_mask) loss = inputs[0] loss = float(loss.asnumpy()) self.total_loss += loss self.num_data += 1 def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None avg_loss = float(self.total_loss / self.num_data) result = {"loss": avg_loss, "PPL": math.exp(avg_loss)} if self.pipeline_parallel and (self.rank_id == (self.device_num - 1)): print("Average Loss and PPL Metric:", result) return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class ADGENMetric(nn.Metric): """Compute the f1, precision and recall score of each entity""" def __init__(self): super(ADGENMetric, self).__init__() self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def clear(self): self.score_dict = { "rouge-1": [], "rouge-2": [], "rouge-l": [], "bleu-4": [] } def update(self, *inputs): """Update results for every batch""" preds = inputs[0] # list[numpy] labels = inputs[1] # numpy if isinstance(preds, tuple): preds = preds[0] for pred, label in zip(preds, labels): print(f"pred is:\n {pred}\n", f"label is:\n {label}") hypothesis = list(jieba.cut(pred)) reference = list(jieba.cut(label)) rouge = Rouge() scores = rouge.get_scores(' '.join(hypothesis), ' '.join(reference)) result = scores[0] for k, v in result.items(): self.score_dict[k].append(round(v["f"] * 100, 4)) bleu_score = sentence_bleu([list(label)], list(pred), smoothing_function=SmoothingFunction().method3) self.score_dict["bleu-4"].append(round(bleu_score * 100, 4)) def eval(self): """Compute final result""" for k, v in self.score_dict.items(): self.score_dict[k] = float(np.mean(v)) print('metric: ADGENMetric\n' + f'rouge-1: {self.score_dict["rouge-1"]:.4f}\n' + f'rouge-2: {self.score_dict["rouge-2"]:.4f}\n' + f'rouge-l: {self.score_dict["rouge-l"]:.4f}\n' + f'bleu-4: {self.score_dict["bleu-4"]:.4f}') return self.score_dict @MindFormerRegister.register(MindFormerModuleType.METRIC) class PromptAccMetric(nn.Metric): r""" Computes the prompt acc of each entity. The prompt acc is the accuracy of text classification base on building prompt. The accurate index is the index of the prompt which has the minimum perplexity. 1. Build the prompt for this metric is described as follows: 这是关于**体育**的文章：$passage 这是关于**文化**的文章：$passage 2. Computes perplexity of each generated context based on prompt. Perplexity is a measurement about how well a probability distribution or a model predicts a sample. A low perplexity indicates the model can predict the sample well. The function is shown as follows: .. math:: PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}} Where :math:`w` represents words in corpus. 3. Compute classification result by choosing the index of the prompt which has the minimum perplexity. 4. Count the number of correctly classified and the total number of samples and compute the acc as follows: .. math:: \text{accuracy} =\frac{\text{correct_sample_nums}}{\text{total_sample_nums}} """ def __init__(self): super(PromptAccMetric, self).__init__() self.num_data = None self.total_acc_num = None self.loss = CrossEntropyLoss() self.pipeline_stages = ms.get_auto_parallel_context('pipeline_stages') self.pipeline_parallel = self.pipeline_stages > 1 self.last_card_id = 0 self.rank_id = 0 self.device_num = 1 self.cast = P.Cast() self.reshape = P.Reshape() self.equal = P.Equal() self.softmax = P.Softmax() self.argmin = P.Argmin() self.sum = P.ReduceSum() if self.pipeline_parallel: self.rank_id = get_rank() self.device_num = get_group_size() per_stage_device_num = self.device_num // self.pipeline_stages stage_id = self.rank_id // per_stage_device_num self.is_last_stage = (stage_id == self.pipeline_stages - 1) self.parallel_mode = ms.get_auto_parallel_context("parallel_mode") self.full_batch = ms.get_auto_parallel_context("full_batch") self.auto_parallel = self.parallel_mode in ['semi_auto_parallel', 'auto_parallel'] def clear(self): """Clearing the internal evaluation result.""" self.num_data = 0 self.total_acc_num = 0 def calculate_circle(self, *inputs): """The main calculate logic.""" logits, input_ids, input_mask, labels = inputs[0], inputs[1], inputs[2], inputs[3] batch_size, num_labels, seq_length, _ = logits.shape logits = self.reshape(logits, (batch_size*num_labels, seq_length, -1)) ppl_list = [] for index in range(batch_size*num_labels): sub_logits, sub_tokens, sub_mask_list = logits[index], input_ids[index], input_mask[index] sub_logits = sub_logits[:-1, ::] sub_tokens = sub_tokens[1:] sub_mask_list = sub_mask_list[1:] loss = self.loss(sub_logits, sub_tokens, sub_mask_list) loss = float(loss.asnumpy()) ppl_list.append(loss) # smaller, better ppl_ms = ms.Tensor(ppl_list, dtype=ms.float32) ppl_ms = self.reshape(ppl_ms, (batch_size, num_labels)) ppl_ms = self.cast(self.argmin(ppl_ms), ms.int32) label = self.reshape(labels, (-1,)) cur_acc_num = self.cast(self.equal(ppl_ms, label), ms.float16).sum().asnumpy() self.num_data += batch_size self.total_acc_num += cur_acc_num def update(self, *inputs): """Update results for every batch""" if self.pipeline_parallel: if not self.is_last_stage: return if self.auto_parallel: ms.context.set_auto_parallel_context(parallel_mode='data_parallel', full_batch=False) self.calculate_circle(*inputs) if self.auto_parallel: ms.set_auto_parallel_context(parallel_mode=self.parallel_mode, full_batch=True, pipeline_stages=self.pipeline_stages) else: self.calculate_circle(*inputs) print("Current data num is {}, total acc num is {}, ACC is {}".format( self.num_data, self.total_acc_num, "%.3f" % (self.total_acc_num / self.num_data))) return def eval(self): """Compute final result""" if self.pipeline_parallel and not self.is_last_stage: return None acc_rate = float(self.total_acc_num / self.num_data) result = {"Acc": acc_rate} print(f"Acc: {('%.3f' % result.get('Acc', 0))}, total_acc_num: {self.total_acc_num}, " f"total_num: {self.num_data}") return result @MindFormerRegister.register(MindFormerModuleType.METRIC) class EmF1Metric(nn.Metric): """ Compute the Em/F1 scores of examples. Em score is the prediction exact matches the labels except the punctuations. For example, the question is "河南的省会是哪里？" and the label is "郑州市", when prediction is "郑州市", Em score is 100; when prediction is "郑州市。", Em score is 100; when prediction is "郑州", Em score is 0. F1 score is calculated as 2*precision*recall/(precision+recall), the precision and recall are calculated as precision = lcs_length/len(prediction_segment), recall = lcs_length/len(label_segment), lcs_length is the length of the longest common subsequence. """ def __init__(self): super(EmF1Metric, self).__init__() self.gens = None self.labels = None self.metrics = None self.num_data = None def clear(self): """Clearing the internal evaluation result.""" self.gens = [] self.labels = [] self.metrics = { 'Em': 0.0, 'F1': 0.0 } self.num_data = 0 def update(self, *inputs): """Update results for every batch""" gen, label = inputs[0], inputs[1] for i in range(len(gen)): gen[i] = gen[i].strip() gen[i] = gen[i].split("\n")[0] print(f"pred is:\n {gen}\n", f"label is:\n {label}") self.gens.extend(gen) self.labels.extend(label) self.num_data += len(gen) result, current_count = self.evaluate_pairs(gen, label) print("The F1/Em of this example is: ", result) if self.num_data % 10 == 0: result, current_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, current_count: {current_count}") def eval(self): """Compute final result""" result, total_count = self.evaluate_pairs(self.gens, self.labels) print(f"F1 score: {result.get('F1', 0)}, Em score: {result.get('Em', 0)}, total_count: {total_count}") return result def mixed_segmentation(self, in_str, rm_punc=False): """cut input for calculating lcs""" in_str = str(in_str).lower().strip() segs_out = [] temp_str = "" sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] for char in in_str: if rm_punc and char in sp_char: continue if re.search(r'[\u4e00-\u9fa5]', char) or char in sp_char: if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) temp_str = "" segs_out.append(char) else: temp_str += char if temp_str != "": ss = list(jieba.cut(temp_str)) segs_out.extend(ss) return segs_out def remove_punctuation(self, in_str): """remove punctuations in inputs""" in_str = str(in_str).lower().strip() sp_char = ['-', ':', '_', '*', '^', '/', '\\', '~', '`', '+', '=', '，', '。', '：', '？', '！', '“', '”', '；', '’', '《', '》', '……', '·', '、', '「', '」', '（', '）', '－', '～', '『', '』'] out_segs = [] for char in in_str: if char in sp_char: continue else: out_segs.append(char) return ''.join(out_segs) def find_lcs(self, s1, s2): """calculate the length of lcs""" m = [[0 for i in range(len(s2) + 1)] for j in range(len(s1) + 1)] mmax = 0 p = 0 for i in range(len(s1)): for j in range(len(s2)): if s1[i] == s2[j]: m[i + 1][j + 1] = m[i][j] + 1 if m[i + 1][j + 1] > mmax: mmax = m[i + 1][j + 1] p = i + 1 return s1[p - mmax:p], mmax def calc_f1_score(self, answers, prediction): """calculate f1 score""" f1_scores = [] for ans in answers: ans_segs = self.mixed_segmentation(ans, rm_punc=True) prediction_segs = self.mixed_segmentation(prediction, rm_punc=True) _, lcs_len = self.find_lcs(ans_segs, prediction_segs) if lcs_len == 0: f1_scores.append(0) continue precision = 1.0 * lcs_len / len(prediction_segs) recall = 1.0 * lcs_len / len(ans_segs) f1 = (2 * precision * recall) / (precision + recall) f1_scores.append(f1) return max(f1_scores) def calc_em_score(self, answers, prediction): """calculate em score""" em = 0 for ans in answers: ans_ = self.remove_punctuation(ans) prediction_ = self.remove_punctuation(prediction) if ans_ == prediction_: em = 1 break return em def evaluate_pairs(self, pred_, ans_): """calculate metric""" f1 = 0 em = 0 total_count = 0 for (prediction, answer) in zip(pred_, ans_): total_count += 1 f1 += self.calc_f1_score([answer], prediction) em += self.calc_em_score([answer], prediction) if total_count > 0: f1_score = 100.0 * f1 / total_count em_score = 100.0 * em / total_count result = {'F1': f1_score, 'Em': em_score} else: print("total_count is zero") result = {} return result, total_count pengcheng.mind-mindformer/mindformers/core/optim/000077500000000000000000000000001476650141400225175ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/optim/.ipynb_checkpoints/000077500000000000000000000000001476650141400263105ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/optim/.ipynb_checkpoints/optim-checkpoint.py000066400000000000000000000551541476650141400321510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """FusedAdamWeightDecay, a customized Adam for offloading.""" import numpy as np from mindspore import context, nn from mindspore.common import dtype as mstype from mindspore.ops import operations as P from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.common.initializer import initializer from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter, ParameterTuple # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel from mindspore.nn.optim.optimizer import Optimizer from mindformers.version_control import fix_optim_global_step_sig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['FusedAdamWeightDecay', 'FP32StateAdamWeightDecay'] _adam_opt = C.MultitypeFuncGraph("adam_opt") _scaler_one = Tensor(1, mstype.int32) op_mul = P.Mul() @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update_with_global_norm(opt, global_norm, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True if optim_filter: if decay_flags: next_param = opt(param, m, v, lr, beta1, beta2, eps, weight_decay, P.Cast()(gradient, mstype.float16), global_norm) else: next_param = opt(param, m, v, lr, beta1, beta2, eps, 0.0, P.Cast()(gradient, mstype.float16), global_norm) return F.depend(success, next_param) return success @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update(opt, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True op_cast = P.Cast() if optim_filter: if decay_flags: opt(param, m, v, lr, beta1, beta2, eps, weight_decay, op_cast(gradient, F.dtype(param))) else: opt(param, m, v, lr, beta1, beta2, eps, 0.0, op_cast(gradient, F.dtype(param))) return success @_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _update_run_op(beta1_power, beta2_power, beta1, beta2, eps, lr, weight_decay, param, \ m, v, gradient, decay_flag, optim_filter): """ Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. """ if optim_filter: # op_mul = P.Mul(), defined output op_square = P.Square() op_sqrt = P.Sqrt() op_cast = P.Cast() op_reshape = P.Reshape() op_shape = P.Shape() param_fp32 = op_cast(param, mstype.float32) m_fp32 = op_cast(m, mstype.float32) v_fp32 = op_cast(v, mstype.float32) gradient_fp32 = op_cast(gradient, mstype.float32) next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta1, gradient_fp32) next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta2, op_square(gradient_fp32)) regulate_m = next_m / (_scaler_one - beta1_power) regulate_v = next_v / (_scaler_one - beta2_power) update = regulate_m / (eps + op_sqrt(regulate_v)) if decay_flag: update = op_mul(weight_decay, param_fp32) + update update_with_lr = op_mul(lr, update) next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32)) next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param)))) next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m)))) next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v)))) return op_cast(next_param, F.dtype(param)) return gradient def _check_param_value(beta1, beta2, eps, prim_name): """Check the type of inputs.""" validator.check_value_type("beta1", beta1, [float], prim_name) validator.check_value_type("beta2", beta2, [float], prim_name) validator.check_value_type("eps", eps, [float], prim_name) validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name) validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name) validator.check_positive_float(eps, "eps", prim_name) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FusedAdamWeightDecay(Optimizer): """ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0, offload=False): super(FusedAdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix="adam_m", init='zeros') self.moments2 = self.clone_state(prefix="adam_v", init='zeros') self.opt = P.AdamWeightDecay() if offload: self.opt.add_prim_attr("primitive_target", "CPU") self.version_flag = fix_optim_global_step_sig() def construct(self, gradients): """construct with gradients""" lr = self.get_lr() if self.version_flag: self.assignadd(self.global_step, self.global_step_increase_tensor) if self.is_group: if self.is_group_lr: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps), lr, self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr), self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr, self.weight_decay), self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) if self.use_parallel: self.broadcast_params(optim_result) return optim_result def clone_state(self, prefix, init, forced_dtype=mstype.float32, is_follow=False): r""" Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: param_init = init if init is None: param_init = old_param.init cur_dtype = forced_dtype if is_follow: cur_dtype = old_param.dtype new_state = Parameter(initializer(param_init, shape=old_param.shape, dtype=cur_dtype)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.is_param_ps = old_param.is_param_ps new_state.init_in_server = old_param.init_in_server new_state.cache_enable = old_param.cache_enable new_state.requires_aggr = old_param.requires_aggr if old_param.cache_shape: new_state.cache_shape = old_param.cache_shape new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FP32StateAdamWeightDecay(nn.AdamWeightDecay): r""" This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(nn.AdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix='adam_m', init='zeros') self.moments2 = self.clone_state(prefix='adam_v', init='zeros') self.fused_opt = P.AdamWeightDecay() if context.get_context("device_target") == "CPU": self.use_fused_opt = True else: self.use_fused_opt = False def clone_state(self, prefix, init): r""" parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: new_state = Parameter(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.set_data(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) pengcheng.mind-mindformer/mindformers/core/optim/__init__.py000066400000000000000000000014361476650141400246340ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Optimizer.""" from .build_optim import build_optim from .optim import * __all__ = [] __all__.extend(optim.__all__) pengcheng.mind-mindformer/mindformers/core/optim/__pycache__/000077500000000000000000000000001476650141400247275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/core/optim/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004511476650141400311170ustar00rootroot00000000000000a ?cf@s,dZddlmZddlTgZeejdS)zMindFormers Optimizer.) build_optim)*N)__doc__roptim__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/__init__.pys pengcheng.mind-mindformer/mindformers/core/optim/__pycache__/build_optim.cpython-39.pyc000066400000000000000000000056221476650141400316740ustar00rootroot00000000000000a ?cf@sdZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZmZddlmZmZdeeeed d d Zd dZddZeedS)zBuild Optimizer API.N)nn) AdaFactorAdamWeightDecaySGDAdagradAdam)build_lr)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)FusedAdamWeightDecayFP32StateAdamWeightDecay optimizer)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurxt|jtrx|jjdurhtdt|j}||_tj|t j |dStj ||fi|S)aLBuild optim For MindFormer. Instantiate the optim from MindFormerRegister's registry. Args: config (dict): The task optim's config. Default: None. default_args (dict): The default argument of optim API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'optim'. class_name (str): The class name of optim API. Default: None. Return: The function instance of optim API. Examples: >>> from mindformers import build_optim >>> optim_config = {'type': 'AdamWeightDecay', 'weight_decay':0.05} >>> # 1) use config dict to build optim >>> optim_from_config = build_optim(optim_config) >>> # 2) use class name to build optim >>> optim_class_name = build_optim(class_name='AdamWeightDecay', weight_decay=0.05) Nzfoptimizer's learning rate must be LearningRateSchedule type, but the type is not specified, it is None)r) isinstancedictr learning_ratetype ValueErrorrr get_instance_from_cfgr OPTIMIZER get_instance)rrrrkwargs lr_schedulerM/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/build_optim.py build_optims  r cCsDttjD]4}|drq ttj|}t|r t|t j q dS)z- register MindSpore builtin optimizer class. __N) dirroptim startswithgetattrinspectisclassr register_clsr r) module_namer#rrrregister_ms_optimAs   r*cCstjttjddtjttjddtjttjddtjttjddtjttjddtjt tjddtjt tjddd S) z/ register MindFormers builtin optimizer class. Zadamw)ralias adafactorsgdadamZadagradZ fused_adamwZ fp32_adamwN) r r(rr rrrrrr rrrrrregister_mf_optimLs*r/)NNrN)__doc__r& mindsporerZmindspore.nn.optimrrrrrZmindformers.core.lrrmindformers.tools.registerr r r r#r rrstrr r*r/rrrrs    ' pengcheng.mind-mindformer/mindformers/core/optim/__pycache__/optim.cpython-39.pyc000066400000000000000000000450501476650141400305140ustar00rootroot00000000000000a ?cflZ@sdZddlZddlmZmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZmZzdd lmZdd lmZWn*eyddlmZddlmZYn0dd lmZddlm Z ddl!m"Z"m#Z#ddgZ$e %dZ&edej'Z(e )Z*e&+ddddddddddddd ddZ,e&+dddddddddddd ddZ-e&+ddddddddddddd ddZ.ddZ/e"+e#j0GdddeZ1e"+e#j0Gd ddej2Z3dS)!z7FusedAdamWeightDecay, a customized Adam for offloading.N)contextnn)dtype) operations) composite) functional) initializer)Tensor) ParameterParameterTuple) Validator)Rel) Optimizer)fix_optim_global_step_sig)MindFormerRegisterMindFormerModuleTypeFusedAdamWeightDecayFP32StateAdamWeightDecayZadam_optFunctionr Boolc  Csjd} | rf| r4|||| |||||t| tj| }n&|||| ||||dt| tj| }t| |S| Sz4 Update parameters by FusedAdamWeightDecay. T)PCastmstypefloat16Fdepend)opt global_normbeta1beta2epslr weight_decayparammvgradient decay_flags optim_filtersuccess next_paramr.G/home/ma-user/work/mindformers-baichuan/mindformers/core/optim/optim.py_fused_update_with_global_norm,s r0c Csbd} t} | r^| r:|||||||||| | t| n$||||||||d| | t| | Sr)rrrr)rr!r"r#r$r%r&r'r(r)r*r+r,op_castr.r.r/ _fused_update?s&$r2c  Csx| rtt} t}t}t}t}||tj}||tj}|| tj}|| tj}t||t|t dtj||}t||t|t dtj|| |}|t |}|t |}||||}| rt|||}t||}|||||}t |t |||t |}t |t |||t |}t |t | ||t | }||t |S| S)aT Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. )?)rSquareSqrtrReshapeShaperfloat32op_mulrtuple_to_array _scaler_onerassignr) beta1_power beta2_powerr!r"r#r$r%r&r'r(r) decay_flagr+ op_squareop_sqrtr1 op_reshapeop_shape param_fp32m_fp32v_fp32 gradient_fp32next_mnext_vZ regulate_mZ regulate_vupdateupdate_with_lrr-r.r.r/_update_run_opPs@          rLcCsttd|tg|td|tg|td|tg|t|ddtjd|t|ddtjd|t|d|dS)zCheck the type of inputs.r!r"r#rr3N) validatorcheck_value_typefloatcheck_float_ranger INC_NEITHERcheck_positive_float)r!r"r# prim_namer.r.r/_check_param_values rTcs:eZdZdZdfdd Zd d Zejdfd d ZZ S)ra_ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) MbP??+?ư>rFcstt||||t||||jtt|gtj |_ tt|gtj |_ tt|gtj |_ |j ddd|_|j ddd|_t|_|r|jddt|_dS)Nadam_mzerosprefixinitadam_vprimitive_targetCPU)superr__init__rTcls_namer nparrayastyper8r!r"r# clone_statemoments1moments2rAdamWeightDecayr add_prim_attrr version_flag)selfparams learning_rater!r"r#r%offload __class__r.r/rbs zFusedAdamWeightDecay.__init__c Cs|}|jr||j|j|jr|jrh|t t |j |j |j |j||j|j|j|j||j|j }q|t t |j |j |j |j||j|j|j|j||j|j}n<|t t |j |j |j |j||j|j|j|j||j|j}|jr|||S)zconstruct with gradients)get_lrrl assignadd global_stepglobal_step_increase_tensoris_group is_group_lr map_reverserpartial _adam_optrr!r"r#r% _parametersrhrir*r+ use_parallelbroadcast_params)rm gradientsr$ optim_resultr.r.r/ constructs8     zFusedAdamWeightDecay.constructc Cs|j}g}|D]}|}|dur$|j}|} |r2|j} tt||j| d} |j| _t|jdrn|jj | n | g|j_ d| _ |j | _ |j | _ |j| _|j| _|jr|j| _|d| j| _| | qt|S)a7 Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. Nshaper cloned_objF.) parametersr]rr rr param_infoclonehasattrrappendis_init is_param_psinit_in_server cache_enable requires_aggr cache_shapenamer ) rmr\r]Z forced_dtypeZ is_followparameter_tuplenew old_param param_init cur_dtype new_stater.r.r/rg s0     z FusedAdamWeightDecay.clone_state)rUrVrWrXrF) __name__ __module__ __qualname____doc__rbrrr8rg __classcell__r.r.rqr/rs Hcs*eZdZdZd fdd Zd d ZZS) ra# This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) rUrVrWrXrcsttj||||t||||jtt|g tj |_ tt|g tj |_ tt|g tj |_ |jddd|_|jddd|_t|_tddkrd|_nd|_dS) NrYrZr[r^ device_targetr`TF)rarrjrbrTrcr rdrerfr8r!r"r#rgrhrir fused_optr get_context use_fused_opt)rmrnror!r"r#r%rqr.r/rbs z!FP32StateAdamWeightDecay.__init__cCs|j}g}|D]}tt||jtjd}|j|_t|jdrP|jj |n |g|j_ d|_ | t||jtjd|d|j |_ | |qt|S)z parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method rrFr)rr rrrr8rrrrrrset_datarr )rmr\r]rrrrr.r.r/rgs    z$FP32StateAdamWeightDecay.clone_state)rUrVrWrXr)rrrrrbrgrr.r.rqr/r/se)4rnumpyrd mindsporerrmindspore.commonrr mindspore.opsrrrCrrmindspore.common.initializerrmindspore.common.tensorr mindspore.common.parameterr r Zmindspore._checkparamr rMr ImportError _checkparammindspore.nn.optim.optimizerrmindformers.version_controlrmindformers.tools.registerrr__all__MultitypeFuncGraphr{int32r;Mulr9registerr0r2rLrT OPTIMIZERrrjrr.r.r.r/sP                :  pengcheng.mind-mindformer/mindformers/core/optim/build_optim.py000066400000000000000000000102171476650141400254010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Optimizer API.""" import inspect from mindspore import nn from mindspore.nn.optim import AdaFactor, AdamWeightDecay, SGD, Adagrad, Adam from mindformers.core.lr import build_lr from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .optim import FusedAdamWeightDecay, FP32StateAdamWeightDecay def build_optim( config: dict = None, default_args: dict = None, module_type: str = 'optimizer', class_name: str = None, **kwargs): r"""Build optim For MindFormer. Instantiate the optim from MindFormerRegister's registry. Args: config (dict): The task optim's config. Default: None. default_args (dict): The default argument of optim API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'optim'. class_name (str): The class name of optim API. Default: None. Return: The function instance of optim API. Examples: >>> from mindformers import build_optim >>> optim_config = {'type': 'AdamWeightDecay', 'weight_decay':0.05} >>> # 1) use config dict to build optim >>> optim_from_config = build_optim(optim_config) >>> # 2) use class name to build optim >>> optim_class_name = build_optim(class_name='AdamWeightDecay', weight_decay=0.05) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.learning_rate is not None and isinstance(config.learning_rate, dict): if config.learning_rate.type is None: raise ValueError("optimizer's learning rate must be LearningRateSchedule type, " "but the type is not specified, it is None") lr_schedule = build_lr(config.learning_rate) config.learning_rate = lr_schedule return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.OPTIMIZER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_optim(): """ register MindSpore builtin optimizer class. """ for module_name in dir(nn.optim): if module_name.startswith('__'): continue optim = getattr(nn.optim, module_name) if inspect.isclass(optim): MindFormerRegister.register_cls( optim, MindFormerModuleType.OPTIMIZER) def register_mf_optim(): """ register MindFormers builtin optimizer class. """ # adapt huggingface MindFormerRegister.register_cls( AdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="adamw") MindFormerRegister.register_cls( AdaFactor, module_type=MindFormerModuleType.OPTIMIZER, alias="adafactor") MindFormerRegister.register_cls( SGD, module_type=MindFormerModuleType.OPTIMIZER, alias="sgd") MindFormerRegister.register_cls( Adam, module_type=MindFormerModuleType.OPTIMIZER, alias="adam") MindFormerRegister.register_cls( Adagrad, module_type=MindFormerModuleType.OPTIMIZER, alias="adagrad") MindFormerRegister.register_cls( FusedAdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="fused_adamw") MindFormerRegister.register_cls( FP32StateAdamWeightDecay, module_type=MindFormerModuleType.OPTIMIZER, alias="fp32_adamw") register_ms_optim() register_mf_optim() pengcheng.mind-mindformer/mindformers/core/optim/optim.py000066400000000000000000000551541476650141400242330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """FusedAdamWeightDecay, a customized Adam for offloading.""" import numpy as np from mindspore import context, nn from mindspore.common import dtype as mstype from mindspore.ops import operations as P from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.common.initializer import initializer from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter, ParameterTuple # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator as validator from mindspore._checkparam import Rel except ImportError: import mindspore._checkparam as validator import mindspore._checkparam as Rel from mindspore.nn.optim.optimizer import Optimizer from mindformers.version_control import fix_optim_global_step_sig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['FusedAdamWeightDecay', 'FP32StateAdamWeightDecay'] _adam_opt = C.MultitypeFuncGraph("adam_opt") _scaler_one = Tensor(1, mstype.int32) op_mul = P.Mul() @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update_with_global_norm(opt, global_norm, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True if optim_filter: if decay_flags: next_param = opt(param, m, v, lr, beta1, beta2, eps, weight_decay, P.Cast()(gradient, mstype.float16), global_norm) else: next_param = opt(param, m, v, lr, beta1, beta2, eps, 0.0, P.Cast()(gradient, mstype.float16), global_norm) return F.depend(success, next_param) return success @_adam_opt.register("Function", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _fused_update(opt, beta1, beta2, eps, lr, weight_decay, param, m, v, gradient, decay_flags, optim_filter): """ Update parameters by FusedAdamWeightDecay. """ success = True op_cast = P.Cast() if optim_filter: if decay_flags: opt(param, m, v, lr, beta1, beta2, eps, weight_decay, op_cast(gradient, F.dtype(param))) else: opt(param, m, v, lr, beta1, beta2, eps, 0.0, op_cast(gradient, F.dtype(param))) return success @_adam_opt.register("Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Tensor", "Bool", "Bool") def _update_run_op(beta1_power, beta2_power, beta1, beta2, eps, lr, weight_decay, param, \ m, v, gradient, decay_flag, optim_filter): """ Update parameters. Args: beta1 (Tensor): The exponential decay rate for the 1st moment estimations. Should be in range (0.0, 1.0). beta2 (Tensor): The exponential decay rate for the 2nd moment estimations. Should be in range (0.0, 1.0). eps (Tensor): Term added to the denominator to improve numerical stability. Should be greater than 0. lr (Tensor): Learning rate. weight_decay (Number): Weight decay. Should be equal to or greater than 0. param (Tensor): Parameters. m (Tensor): m value of parameters. v (Tensor): v value of parameters. gradient (Tensor): Gradient of parameters. decay_flag (bool): Applies weight decay or not. optim_filter (bool): Applies parameter update or not. Returns: Tensor, the new value of v after updating. """ if optim_filter: # op_mul = P.Mul(), defined output op_square = P.Square() op_sqrt = P.Sqrt() op_cast = P.Cast() op_reshape = P.Reshape() op_shape = P.Shape() param_fp32 = op_cast(param, mstype.float32) m_fp32 = op_cast(m, mstype.float32) v_fp32 = op_cast(v, mstype.float32) gradient_fp32 = op_cast(gradient, mstype.float32) next_m = op_mul(beta1, m_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta1, gradient_fp32) next_v = op_mul(beta2, v_fp32) + op_mul(op_cast(F.tuple_to_array((1.0,)), mstype.float32) - beta2, op_square(gradient_fp32)) regulate_m = next_m / (_scaler_one - beta1_power) regulate_v = next_v / (_scaler_one - beta2_power) update = regulate_m / (eps + op_sqrt(regulate_v)) if decay_flag: update = op_mul(weight_decay, param_fp32) + update update_with_lr = op_mul(lr, update) next_param = param_fp32 - op_reshape(update_with_lr, op_shape(param_fp32)) next_param = F.depend(next_param, F.assign(param, op_cast(next_param, F.dtype(param)))) next_param = F.depend(next_param, F.assign(m, op_cast(next_m, F.dtype(m)))) next_param = F.depend(next_param, F.assign(v, op_cast(next_v, F.dtype(v)))) return op_cast(next_param, F.dtype(param)) return gradient def _check_param_value(beta1, beta2, eps, prim_name): """Check the type of inputs.""" validator.check_value_type("beta1", beta1, [float], prim_name) validator.check_value_type("beta2", beta2, [float], prim_name) validator.check_value_type("eps", eps, [float], prim_name) validator.check_float_range(beta1, 0.0, 1.0, Rel.INC_NEITHER, "beta1", prim_name) validator.check_float_range(beta2, 0.0, 1.0, Rel.INC_NEITHER, "beta2", prim_name) validator.check_positive_float(eps, "eps", prim_name) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FusedAdamWeightDecay(Optimizer): """ Implements the Adam algorithm to fix the weight decay. It is a complete operator, not a combination of other ops. Note: When separating parameter groups, the weight decay in each group will be applied on the parameters if the weight decay is positive. When not separating parameter groups, the `weight_decay` in the API will be applied on the parameters without 'beta' or 'gamma' in their names if `weight_decay` is positive. To improve parameter groups performance, the customized order of parameters can be supported. Args: params (Union[list[Parameter], list[dict]]): When the `params` is a list of `Parameter` which will be updated, the element in `params` must be class `Parameter`. When the `params` is a list of `dict`, the "params", "lr", "weight_decay" and "order_params" are the keys can be parsed. - params: Required. The value must be a list of `Parameter`. - lr: Optional. If "lr" is in the keys, the value of the corresponding learning rate will be used. If not, the `learning_rate` in the API will be used. - weight_decay: Optional. If "weight_decay" is in the keys, the value of the corresponding weight decay will be used. If not, the `weight_decay` in the API will be used. - order_params: Optional. If "order_params" is in the keys, the value must be the order of parameters and the order will be followed in the optimizer. There are no other keys in the `dict` and the parameters which in the 'order_params' must be in one of group parameters. learning_rate (Union[float, Tensor, Iterable, LearningRateSchedule]): A value or a graph for the learning rate. When the learning_rate is an Iterable or a Tensor in a 1D dimension, use the dynamic learning rate, then the i-th step will take the i-th value as the learning rate. When the learning_rate is LearningRateSchedule, use dynamic learning rate, the i-th learning rate will be calculated during the process of training according to the formula of LearningRateSchedule. When the learning_rate is a float or a Tensor in a zero dimension, use fixed learning rate. Other cases are not supported. The float learning rate must be equal to or greater than 0. If the type of `learning_rate` is int, it will be converted to float. Default: 1e-3. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (float): Weight decay (L2 penalty). It must be equal to or greater than 0. Default: 0.0. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Supported Platforms: ``CPU`` Examples: >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FusedAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FusedAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0, offload=False): super(FusedAdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix="adam_m", init='zeros') self.moments2 = self.clone_state(prefix="adam_v", init='zeros') self.opt = P.AdamWeightDecay() if offload: self.opt.add_prim_attr("primitive_target", "CPU") self.version_flag = fix_optim_global_step_sig() def construct(self, gradients): """construct with gradients""" lr = self.get_lr() if self.version_flag: self.assignadd(self.global_step, self.global_step_increase_tensor) if self.is_group: if self.is_group_lr: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps), lr, self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr), self.weight_decay, self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) else: optim_result = self.map_reverse(F.partial(_adam_opt, self.opt, self.beta1, self.beta2, self.eps, lr, self.weight_decay), self._parameters, self.moments1, self.moments2, gradients, self.decay_flags, self.optim_filter) if self.use_parallel: self.broadcast_params(optim_result) return optim_result def clone_state(self, prefix, init, forced_dtype=mstype.float32, is_follow=False): r""" Clone the parameters parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method forced_dtype: mstype. The except the dtype to be cloned. If is_follow is True, forced_dtype will be ignored. Default: mstype.float32 is_follow: bool. Is clone the parameters with the original dtype. If is_follow is True, the forced_dtype argument will be ignored. Default: False. """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: param_init = init if init is None: param_init = old_param.init cur_dtype = forced_dtype if is_follow: cur_dtype = old_param.dtype new_state = Parameter(initializer(param_init, shape=old_param.shape, dtype=cur_dtype)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.is_param_ps = old_param.is_param_ps new_state.init_in_server = old_param.init_in_server new_state.cache_enable = old_param.cache_enable new_state.requires_aggr = old_param.requires_aggr if old_param.cache_shape: new_state.cache_shape = old_param.cache_shape new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class FP32StateAdamWeightDecay(nn.AdamWeightDecay): r""" This class is almost same with the mindspore's AdamWeightDecay implements, the only difference is the optimizer's state will be always initialized with float32, where the original AdamWeightDecay will initialize the optimizer's state with float16, if the parameters are initialized with fp16. This setting will avoid overflow in training big model using fp16. Args: params (Union[list[Parameter], list[dict]]): Must be list of `Parameter` or list of `dict`. When the `params` is a list of `dict`, the string "params", "lr", "weight_decay", and "order_params" are the keys can be parsed. - params: Required. Parameters in current group. The value must be a list of `Parameter`. - lr: Optional. If "lr" in the keys, the value of corresponding learning rate will be used. If not, the `learning_rate` in optimizer will be used. Fixed and dynamic learning rate are supported. - weight_decay: Optional. If "weight_decay" in the keys, the value of corresponding weight decay will be used. If not, the `weight_decay` in the optimizer will be used. It should be noted that weight decay can be a constant value or a Cell. It is a Cell only when dynamic weight decay is applied. Dynamic weight decay is similar to dynamic learning rate, users need to customize a weight decay schedule only with global step as input, and during training, the optimizer calls the instance of WeightDecaySchedule to get the weight decay value of current step. - order_params: Optional. When parameters is grouped, this usually is used to maintain the order of parameters that appeared in the network to improve performance. The value should be parameters whose order will be followed in optimizer. If `order_params` in the keys, other keys will be ignored and the element of 'order_params' must be in one group of `params`. learning_rate (Union[float, int, Tensor, Iterable, LearningRateSchedule]): Default: 1e-3. - float: The fixed learning rate value. Must be equal to or greater than 0. - int: The fixed learning rate value. Must be equal to or greater than 0. It will be converted to float. - Tensor: Its value should be a scalar or a 1-D vector. For scalar, fixed learning rate will be applied. For vector, learning rate is dynamic, then the i-th step will take the i-th value as the learning rate. - Iterable: Learning rate is dynamic. The i-th step will take the i-th value as the learning rate. - LearningRateSchedule: Learning rate is dynamic. During training, the optimizer calls the instance of LearningRateSchedule with step as the input to get the learning rate of current step. beta1 (float): The exponential decay rate for the 1st moment estimations. Default: 0.9. Should be in range (0.0, 1.0). beta2 (float): The exponential decay rate for the 2nd moment estimations. Default: 0.999. Should be in range (0.0, 1.0). eps (float): Term added to the denominator to improve numerical stability. Default: 1e-6. Should be greater than 0. weight_decay (Union[float, int, Cell]): Weight decay (L2 penalty). Default: 0.0. - float: The fixed weight decay value. Must be equal to or greater than 0. - int: The fixed weight decay value. Must be equal to or greater than 0. It will be converted to float. - Cell: Weight decay is dynamic. During training, the optimizer calls the instance of the Cell with step as the input to get the weight decay value of current step. Inputs: - **gradients** (tuple[Tensor]) - The gradients of `params`, the shape is the same as `params`. Outputs: tuple[bool], all elements are True. Raises: TypeError: If `learning_rate` is not one of int, float, Tensor, Iterable, LearningRateSchedule. TypeError: If element of `parameters` is neither Parameter nor dict. TypeError: If `beta1`, `beta2` or `eps` is not a float. TypeError: If `weight_decay` is neither float nor int. ValueError: If `eps` is less than or equal to 0. ValueError: If `beta1`, `beta2` is not in range (0.0, 1.0). ValueError: If `weight_decay` is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> from mindformers.core.optim import FP32StateAdamWeightDecay >>> >>> net = Net() >>> #1) All parameters use the same learning rate and weight decay >>> optim = FP32StateAdamWeightDecay(params=net.trainable_params()) >>> >>> #2) Use parameter groups and set different values >>> conv_params = list(filter(lambda x: 'conv' in x.name, net.trainable_params())) >>> no_conv_params = list(filter(lambda x: 'conv' not in x.name, net.trainable_params())) >>> group_params = [{'params': conv_params, 'weight_decay': 0.01}, ... {'params': no_conv_params, 'lr': 0.01}, ... {'order_params': net.trainable_params()}] >>> optim = FP32StateAdamWeightDecay(group_params, learning_rate=0.1, weight_decay=0.0) >>> # The conv_params's parameters will use default learning rate of 0.1 and weight decay of 0.01. >>> # The no_conv_params's parameters will use learning rate of 0.01 and default weight decay of 0.0. >>> # The final parameters order in which the optimizer will be followed is the value of 'order_params'. >>> >>> loss = nn.SoftmaxCrossEntropyWithLogits() >>> model = ms.Model(net, loss_fn=loss, optimizer=optim) """ def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(nn.AdamWeightDecay, self).__init__(learning_rate, params, weight_decay) _check_param_value(beta1, beta2, eps, self.cls_name) self.beta1 = Tensor(np.array([beta1]).astype(np.float32)) self.beta2 = Tensor(np.array([beta2]).astype(np.float32)) self.eps = Tensor(np.array([eps]).astype(np.float32)) self.moments1 = self.clone_state(prefix='adam_m', init='zeros') self.moments2 = self.clone_state(prefix='adam_v', init='zeros') self.fused_opt = P.AdamWeightDecay() if context.get_context("device_target") == "CPU": self.use_fused_opt = True else: self.use_fused_opt = False def clone_state(self, prefix, init): r""" parameter_tuple: ParameterTuple. The parameters of the network prefix: str. The prefix name of the parameters init: str. The initialization method """ parameter_tuple = self.parameters new = [] for old_param in parameter_tuple: new_state = Parameter(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.param_info = old_param.param_info.clone() if hasattr(old_param.param_info, "cloned_obj"): old_param.param_info.cloned_obj.append(new_state) else: old_param.param_info.cloned_obj = [new_state] new_state.is_init = False new_state.set_data(initializer(init, shape=old_param.shape, dtype=mstype.float32)) new_state.name = prefix + '.' + new_state.name new.append(new_state) return ParameterTuple(new) pengcheng.mind-mindformer/mindformers/core/parallel_config.py000066400000000000000000000047231476650141400250700ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Parallel Config Init.""" from mindformers.modules.transformer.moe import default_moe_config, MoEConfig from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.tools.logger import logger default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) def build_parallel_config(config): """Build context config.""" if config.moe_config: if not isinstance(config.moe_config, MoEConfig): logger.info("initial moe_config from dict: %s", config.moe_config) config.moe_config = MoEConfig(**config.moe_config) else: config.moe_config = default_moe_config if config.recompute_config: if not isinstance(config.recompute_config, TransformerRecomputeConfig): logger.info("initial recompute_config from dict: %s", config.recompute_config) config.recompute_config = TransformerRecomputeConfig(**config.recompute_config) else: config.recompute_config = default_recompute_config if config.parallel_config: if not isinstance(config.parallel_config, TransformerOpParallelConfig): logger.info("initial parallel_config from dict: %s", config.parallel_config) if config.parallel_config.pipeline_stage > 1: logger.info("pipeline_stage = %s > 1, vocab_emd_dp will be reset to False.", config.parallel_config.pipeline_stage) config.parallel_config.vocab_emb_dp = False config.parallel_config = TransformerOpParallelConfig(recompute=config.recompute_config, **config.parallel_config) else: config.parallel_config = default_parallel_config pengcheng.mind-mindformer/mindformers/dataset/000077500000000000000000000000001476650141400220645ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/.ipynb_checkpoints/000077500000000000000000000000001476650141400256555ustar00rootroot00000000000000causal_language_model_dataset-checkpoint.py000066400000000000000000000247021476650141400363020ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/.ipynb_checkpoints# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.models.build_tokenizer import build_tokenizer from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(input_ids, eos_token_id, dis, rank_id: int = 0): """ Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset """ rank = int(rank_id) input_ids = input_ids[rank*dis: (rank + 1)*dis] seq_length = input_ids.shape[1] - 1 # Initialize position_ids and attention_mask batch_input_ids = input_ids batch_position_ids = np.ones((dis, seq_length)) batch_attention_mask = np.ones((dis, seq_length, seq_length)) # Loop through batches for bs_i in range(len(input_ids)): # Get normal position_ids and attention_mask local_ids = input_ids[bs_i] batch_attention_mask[bs_i] = np.tril(np.ones(shape=(seq_length, seq_length))) batch_position_ids[bs_i] = np.arange(seq_length) # Find the index of eos_index = batch_position_ids[bs_i, local_ids[:-1] == eos_token_id].astype(np.int32) prev_index = 0 for i in range(eos_index.size): # Reset position_ids and attention_mask considering index = eos_index[i] batch_attention_mask[bs_i, (index + 1):, :(index + 1)] = 0 batch_position_ids[bs_i, (index + 1):] -= (index + 1 - prev_index) prev_index = index + 1 return batch_input_ids, batch_position_ids, batch_attention_mask @MindFormerRegister.register(MindFormerModuleType.DATASET) class CausalLanguageModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Causal Language Model Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) type_cast_op = C.TypeCast(mstype.int32) if dataset_config.eos_reset: if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) map_func = lambda input_ids: get_input_data_batch_slice_map(input_ids, eos_token_id=dataset_config.eos_token_id, rank_id=rank_id, dis=dis) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) for input_arg in dataset_config.output_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) else: dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if (re.findall(mind_compile, file) and not file.endswith(".db")) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-mindformer/mindformers/dataset/__init__.py000066400000000000000000000047641476650141400242100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Dataset.""" from .dataloader import * from .mask import * from .transforms import * from .sampler import * from .build_dataset import build_dataset from .dataloader.build_dataloader import build_dataset_loader from .mask.build_mask import build_mask from .sampler.build_sampler import build_sampler from .transforms.build_transforms import build_transforms from .base_dataset import BaseDataset from .mim_dataset import MIMDataset from .img_cls_dataset import ImageCLSDataset from .contrastive_language_image_pretrain_dataset import ContrastiveLanguageImagePretrainDataset from .zero_shot_image_classification_dataset import ZeroShotImageClassificationDataset from .token_classification_dataset import TokenClassificationDataset from .mask_language_model_dataset import MaskLanguageModelDataset from .translation_dataset import TranslationDataset from .image_text_retrieval_dataset import ImageToTextRetrievalDataset from .text_classification_dataset import TextClassificationDataset from .question_answering_dataset import QuestionAnsweringDataset from .causal_language_model_dataset import CausalLanguageModelDataset from .reward_model_dataset import RewardModelDataset from .utils import check_dataset_config from .keyword_gen_dataset import KeyWordGenDataset __all__ = ['MIMDataset', 'ImageCLSDataset', 'BaseDataset', 'check_dataset_config', 'MaskLanguageModelDataset', 'ContrastiveLanguageImagePretrainDataset', 'ZeroShotImageClassificationDataset', 'ImageToTextRetrievalDataset', 'TextClassificationDataset', 'TokenClassificationDataset', 'QuestionAnsweringDataset', 'CausalLanguageModelDataset', 'KeyWordGenDataset', 'RewardModelDataset'] __all__.extend(dataloader.__all__) __all__.extend(mask.__all__) __all__.extend(transforms.__all__) __all__.extend(sampler.__all__) __all__.extend(translation_dataset.__all__) pengcheng.mind-mindformer/mindformers/dataset/__pycache__/000077500000000000000000000000001476650141400242745ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__/__init__.cpython-39.pyc000066400000000000000000000034011476650141400304620ustar00rootroot00000000000000a ?cf @s\dZddlTddlTddlTddlTddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+gdZ,e,-ej,e,-ej,e,-ej,e,-ej,e,-ej,dS)zMindFormers Dataset.)*) build_dataset)build_dataset_loader) build_mask) build_sampler)build_transforms) BaseDataset) MIMDataset)ImageCLSDataset)'ContrastiveLanguageImagePretrainDataset)"ZeroShotImageClassificationDataset)TokenClassificationDataset)MaskLanguageModelDataset)TranslationDataset)ImageToTextRetrievalDataset)TextClassificationDataset)QuestionAnsweringDataset)CausalLanguageModelDataset)RewardModelDataset)check_dataset_config)KeyWordGenDataset)r r rrrr r rrr rrrrN).__doc__Z dataloadermask transformssamplerrZdataloader.build_dataloaderrZmask.build_maskrZsampler.build_samplerrZtransforms.build_transformsrZ base_datasetrZ mim_datasetr Zimg_cls_datasetr Z+contrastive_language_image_pretrain_datasetr Z&zero_shot_image_classification_datasetr Ztoken_classification_datasetr Zmask_language_model_datasetrZtranslation_datasetrZimage_text_retrieval_datasetrZtext_classification_datasetrZquestion_answering_datasetrZcausal_language_model_datasetrZreward_model_datasetrutilsrZkeyword_gen_datasetr__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/dataset/__init__.pys<                        pengcheng.mind-mindformer/mindformers/dataset/__pycache__/base_dataset.cpython-39.pyc000066400000000000000000000041671476650141400313540ustar00rootroot00000000000000a ?cf @s2dZddlZddlZddlmZGdddZdS) Base Dataset.Nc@sPeZdZdZdedddZeddZedd Zed d Z ed d Z dS) BaseDatasetrNdataset_configcCs ||_dS)Nr)selfrrK/home/ma-user/work/mindformers-baichuan/mindformers/dataset/base_dataset.py__init__szBaseDataset.__init__cCs|durtdtj|jtj|jtj|j|j r|j rNt dt j |jddt j|jd|_tjjd|jdtj|jdS)zInit dataset config.Nzdataset_config cannot be None!znMindSpore's AutoTune is enabled, so Profile cannot be enabled,now Profile's flag is True, please set to False!T)exist_okZautotune)filepath_prefix) ValueErrordsconfigset_seedseedset_prefetch_size prefetch_sizeset_numa_enable numa_enableZ auto_tuneprofileEnvironmentErrorosmakedirsr pathjoinset_enable_autotuneset_autotune_intervalZautotune_per_step)clsrrrrinit_dataset_configszBaseDataset.init_dataset_configcCs|rd}d}||fS)z3Check device num and rank id in auto parallel mode.N)_is_semi_full_batch)rrank_id device_numrrr_check_device_rank_for_parallel/sz+BaseDataset._check_device_rank_for_parallelcCstjddvotjdS)N parallel_mode)semi_auto_parallel auto_parallel full_batch)mscontextget_auto_parallel_contextrrrrr7s zBaseDataset._is_semi_full_batchcCstjdtjjjkS)Nr#)r'r(r) ParallelMode DATA_PARALLELr*rrr_is_data_parallel<szBaseDataset._is_data_parallel)N) __name__ __module__ __qualname____doc__dictr classmethodrr"rr-rrrrrs   r)r1r mindsporer'mindspore.datasetdatasetr rrrrrs pengcheng.mind-mindformer/mindformers/dataset/__pycache__/build_dataset.cpython-39.pyc000066400000000000000000000034771476650141400315440ustar00rootroot00000000000000a ?cf @s2dZddlmZmZmZdeeeedddZdS) zBuild Dataset API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigNdataset)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aCBuild dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) N)r) isinstancedictrrget_instance_from_cfgrDATASET get_instance)rrrr kwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/dataset/build_dataset.py build_datasetsr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrscausal_language_model_dataset.cpython-39.pyc000066400000000000000000000160471476650141400346560ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__a of*@sdZddlZddlZddlZddlZddlmmZ ddl m m m ZddlmZmZddlmZddlmZddlmZddlmZdd lmZded d d ZeejGd ddeZ dS)zCausal Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)build_tokenizer)get_dataset_map)build_dataset_loader) BaseDataset)rank_idc Cs"t|}||||d|}|jdd}|}t||f}t|||f}tt|D]} || } ttj||fd|| <t||| <|| | dd|kftj } d} t| j D]V} | | }d|| |ddd|df<|| |ddf|d| 8<|d} qq\|||fS)a Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset r)shapeNr) intr nponesrangelentrilarangeastypeint32size) input_ids eos_token_iddisr rank seq_lengthZbatch_input_idsZbatch_position_idsZbatch_attention_maskZbs_iZ local_idsZ eos_index prev_indexiindexr\/home/ma-user/work/mindformers-baichuan/mindformers/dataset/causal_language_model_dataset.pyget_input_data_batch_slice_maps$ $ r!c@sDeZdZdZd edddZeddZedd Zed d Z dS) CausalLanguageModelDatasetaCausal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) N)dataset_configcstdttddttdd}t|||\}_ |_ j j dkr~j j dkr~| }n |}ttj}jrN|s|rdjn0j|j|dkrtd jd |d |jjjjd }fd d}t||jjd}|jjd}jD]}t|||d}q6nD|jjjjjd}|jjd}jD]}t|||d}q||j}|S)Nz)Now Create Causal Language Model Dataset.RANK_ID0 RANK_SIZE1 MindDatasetTFRecordDatasetrz batch size z' should be a multiple of device number z-. You should change the args: per_batch_size.)drop_remainderoutput_columnscst|jdS)N)rr r)r!r)rr#rr rr sz4CausalLanguageModelDataset.__new__.. input_columnsr+)columns)r/)r*r+num_parallel_workers) rinfor osgetenvcopydeepcopyinit_dataset_config_check_device_rank_for_parallelr device_num data_loadertype_process_raw_text_data_process_mindrecord_dataCTypeCastmstyperZ eos_reset_is_semi_full_batch_is_data_parallel batch_size ValueErrorbatchr*r/rr+projectr1repeat)clsr#r9dataset type_cast_opmap_func input_argrr,r __new__`sb              z"CausalLanguageModelDataset.__new__cs:|j}t||jfdd}t|||j|jd}|S)zPreprocess data for gpt2 modelcs$|}|dddd}|dS)N max_lengthTF)paddingrN truncationadd_special_tokensr)tolistget) input_datarrN tokenizerrr rKs  z?CausalLanguageModelDataset._prepare_for_model..map_funcr.)rVrrNrr/)rHrIr#tokenizer_configrKrrUr _prepare_for_modelsz-CausalLanguageModelDataset._prepare_for_modelcCs6|jd}t|j||j|jdd}|||}|S)zProcess the text data dataset_dir)rY num_shardsshard_id default_args)r:poprr9r rX)rHr#rYrIrrr r<s  z1CausalLanguageModelDataset._process_raw_text_datac Cs:g}td}|jjr|jd}tj|rt|D]H\}}}|D]8}t ||r`| drj| drF| tj ||qFq8| qt ||s| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdtd|jt|j||j|j|jdd } |jr6| |j} td |jd | S) zProcess the mindrecord data mindrecordrYz.dbz .tfrecordz>data_loader must contain dataset_dir or dataset_files,but get .zbuild dataset with : ) dataset_filesrZr[ columns_listr\zskip z samples)recompiler:rYr^r3pathisdirwalkfindallendswithappendjoinsortra isinstancelisttuplerDprintrr9r r/consumed_samplesskip) rHr#raZ mind_compiledata_dirr_ffilerIrrr r=s>        z3CausalLanguageModelDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictrM classmethodrXr<r=rrrr r"Fs9  r")r)!r{r3r5rcnumpyrZmindspore.common.dtypecommondtyper@Z)mindspore.dataset.transforms.c_transformsrI transforms c_transformsr>mindformers.tools.registerrrmindformers.tools.loggerrZ"mindformers.models.build_tokenizerrmindformers.version_controlr dataloaderr base_datasetr r r!registerDATASETr"rrrr s     ( contrastive_language_image_pretrain_dataset.cpython-39.pyc000066400000000000000000000067361476650141400376410ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__a ?cf@sdZddlZddlmZddlmZddlmZddlm Z ddl m Z d d l m Z d d lmZd d lmZmZeejGd dde ZdS)z,Contrastive Language Image Pretrain Dataset.N)get_dataset_map)build_dataset_loader)build_transforms) build_sampler) BaseDataset)logger)build_tokenizer)MindFormerRegisterMindFormerModuleTypec@s eZdZdZdedddZdS)'ContrastiveLanguageImagePretrainDataseta^ Contrastive Language Image Pretrain Dataset API. output image and text columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ContrastiveLanguageImagePretrainTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_pretrain_flickr8k.yaml") >>> config = MindFormerConfig(config_path) Note: Put flickr8k dataset to ./checkpoint_download The detailed data setting could refer to ./configs/clip/clip.md >>> config.train_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.train_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[4.99690473e-001, 6.74871564e-001, ... 3.68304640e-001, 2.36918822e-001], [7.91658998e-001, 7.62462139e-001, ... -2.01033935e-001, -1.13443382e-001], ... [-5.98575652e-001, -6.12795711e-001, ... 1.47755420e+000, 1.46333420e+000], [-3.85274649e-001, -6.27015769e-001, ... 1.42067397e+000, 1.43489408e+000], [-7.97656536e-001, -1.01095748e+000, ... 9.37191546e-001, 9.08751369e-001]]]]), Tensor(shape=[1, 77], dtype=Int32, value= [[49406, 1237, 18250 ... 0, 0, 0]])] N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j }t|j d|id}t |j}|d ur||}|d urt||d |j|jd }|d urt||d |j|jd }|j|j|j|jd }||j}|S)z new methodz7Now Create Contrastive Language Image Pretrain Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerNimage) input_columnsnum_parallel_workerspython_multiprocessingtext)drop_remainderr)r infoinit_dataset_configintosgetenvr data_loaderr transformsr rtext_transformsrsampler use_samplerrrrbatch batch_sizerrepeat) clsrrank_id device_numdatasetr#rr$r%r.j/home/ma-user/work/mindformers-baichuan/mindformers/dataset/contrastive_language_image_pretrain_dataset.py__new__?sB        z/ContrastiveLanguageImagePretrainDataset.__new__)N)__name__ __module__ __qualname____doc__dictr0r.r.r.r/r s#r )r4r mindformers.version_controlr dataloaderrr#rr%r base_datasetrtoolsr Zmodels.build_tokenizerr Ztools.registerr r registerDATASETr r.r.r.r/s        image_text_retrieval_dataset.cpython-39.pyc000066400000000000000000000035671476650141400345710ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__a ?cf` @sdZddlZddlmmZddlmZddl mm m Z ddl mZmZddlmZddlmZddlmZddl mZeejGd d d eZdS) zImage-text Retrieval Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map) BaseDataset)build_transformsc@s eZdZdZdedddZdS)ImageToTextRetrievalDatasetzBImage-text Retrieval Dataset for filip fine-tuning and evaluation.N)dataset_configcCstd||ttdd}ttdd}tj|jj |jj ||d}t |j }|durzt |||j|jddgd}ttj}t ||ddgd }|dd g}|j|j|jd }|S) Nz(Now Create Image-text Retrieval Dataset.RANK_ID0 RANK_SIZE1)shuffle num_shardsshard_idimage)num_parallel_workerspython_multiprocessing input_columnsoutput_columns)rrtoken)drop_remainder)rinfoinit_dataset_configintosgetenvds MindDataset data_loader dataset_dirrr transformsrrrCTypeCastmstypefloat32projectbatch batch_sizer)clsr rank_id device_numdatasetr" type_cast_opr/[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/image_text_retrieval_dataset.py__new__s0     z#ImageToTextRetrievalDataset.__new__)N)__name__ __module__ __qualname____doc__dictr1r/r/r/r0r sr )r5rZmindspore.common.dtypecommondtyper%mindspore.datasetr-rZ)mindspore.dataset.transforms.c_transformsr" c_transformsr#mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr base_datasetrrregisterDATASETr r/r/r/r0s      pengcheng.mind-mindformer/mindformers/dataset/__pycache__/img_cls_dataset.cpython-39.pyc000066400000000000000000000066721476650141400320620ustar00rootroot00000000000000a ?cf@sdZddlZddlmmmZddlmm Z ddl m Z m Z ddlmZddlmZddlmZddlmZdd lmZdd lmZe e jGd d d eZdS) zImage Classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS)ImageCLSDataseta Image Classification Dataset API. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import ImageCLSDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['image_classification']['vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/image_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='ImageCLSDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = ImageCLSDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cs*td||ttdd}ttdd}t|j||dd}t|j }t |j }t tj}|durz||}|durt|||jd|j|jd }t|||jd |jd }|j|j|j|jd }|js |jdur tfd di|j}t|||j|j|jd}|j|jd}||j}|S)Nz(Now Create Image Classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_argsr) input_columnsnum_parallel_workerspython_multiprocessingr)rr)drop_remainderr class_nameMixup)routput_columnsr)columns)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr samplerCTypeCastmstypeint32 use_samplerrrrrbatch batch_sizerdo_evalmixup_oprprojectrepeat) clsr rank_id device_numdatasetr"r#Z type_cast_opr,r3N/home/ma-user/work/mindformers-baichuan/mindformers/dataset/img_cls_dataset.py__new__:sF         zImageCLSDataset.__new__)N)__name__ __module__ __qualname____doc__dictr5r3r3r3r4r sr )r9rZ)mindspore.dataset.transforms.c_transformsr2r" c_transformsr$Zmindspore.common.dtypecommondtyper&mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrrr#r base_datasetr registerDATASETr r3r3r3r4s       pengcheng.mind-mindformer/mindformers/dataset/__pycache__/keyword_gen_dataset.cpython-39.pyc000066400000000000000000000227211476650141400327530ustar00rootroot00000000000000a ?cf ;@sdZddlZddlZddlmmZddlmm m Z ddl Z ddlmZddlmZddlmZddlmZddlmZddlmZmZdd lmZeejGd d d eZdS) zKeyword Generation Dataset.N) BaseTokenizer) BaseDataset)build_dataset_loader)build_tokenizer)logger)MindFormerModuleTypeMindFormerRegister)get_dataset_mapc@seZdZdZdedddZeddZedd Zed d Z ed d Z eddZ eddZ eddZ edddZedddZed ddZdS)!KeyWordGenDatasetaKeyword generation dataset. Examples: >>> from mindformers.dataset.dataloader.adgen_dataloader import ADGenDataLoader >>> from mindformers.dataset import build_dataset >>> from mindformers import MindFormerConfig >>> cfg = MindFormerConfig("./configs/glm/run_glm_6b_finetune.yaml") >>> dataset = build_dataset(cfg.eval_dataset_task) >>> for item in dataset.create_dict_iterator(): >>> print(item) >>> break N)dataset_configcCstd||ttdd}ttdd}|||\}}||_||_t |j t rf|j |_ n t |j |_ |j |_ |j|_|j|_|j|j|_|jj|_|jj|_|jjdkr||}n ||}|j|j|j|jd}||j}ttj}|j D]}t!|||d}q|S) Nz&Now Create Keyword Generation Dataset.RANK_ID0 RANK_SIZE1 MindDataset)drop_remaindernum_parallel_workers) input_columns)"rinfoinit_dataset_configintosgetenv_check_device_rank_for_parallelrank_id device_num isinstance tokenizerrrignore_pad_token_for_lossmax_source_lengthmax_target_lengthmax_seq_length data_loaderphaseversiontype_process_raw_text_data_process_mindrecord_databatch batch_sizerrrepeatCTypeCastmstypeint32rr )clsr rrdataset type_cast_op input_argr3R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/keyword_gen_dataset.py__new__/s8             zKeyWordGenDataset.__new__c s|j}|jr|jnd}td||dkr@|jddg}|jn|jgd}|jddg}ddg}fd d }fd d } |d krt||||d}|j |d}|dkrt|| ||d}|j |d}|S)z/Maps the tokenizer on the source and the outputz1Start tokenize on the dataset using tokenizer: %s input_idslabels)r8r9 position_idsattention_maskpromptanswercs ||SNr3r<r=)train_dataset_functionr3r4train_dataset_funcfsz.train_dataset_funccs ||Sr>r3r?)eval_dataset_functionr3r4eval_dataset_funcisz;KeyWordGenDataset._tokenizer_map..eval_dataset_functrain)routput_columns)columnseval) r#r$rrtrain_dataset_functionv2eval_dataset_functionv2r@rBr project) r/r0tokenizer_configr#r$Ztrain_output_columnsrZeval_output_columnsrArCr3)rBr@r4_tokenizer_mapQs:     z KeyWordGenDataset._tokenizer_mapcCs8|jd}t|j||j|jdd}|||j}|S)zProcess the text data dataset_dir)rM num_shardsshard_id default_args)r"poprrrrLr)r/r rMr0r3r3r4r&zs z(KeyWordGenDataset._process_raw_text_datac Cst|}g}|jjr|jd}tj|rvt|D]2\}}}|D]"}| drF| tj ||qFq8| q| dr|}n:|jj r|jj }t|ttfrt|}ntd|jdtd|jt|j||j|j|jdd}|S)zProcess the mindrecord datarMz .mindrecordz>data_loader must contain dataset_dir or dataset_files,but get .z&Using args %s to instance the dataset.) dataset_filesrNrO columns_listrP)copydeepcopyr"rMrRrpathisdirwalkendswithappendjoinsortrTrlisttuple ValueErrorrrrrrr) r/r rTdata_dirr_ffiler0r3r3r4r's8        z*KeyWordGenDataset._process_mindrecord_datac s4||}}jj|dd}jj|dd}t|jdkrV|djd}t|jdkrz|djd}j||}|jj}|d}dg|||dd}j t|} |jj g| }|jj g| d}j rfdd|D} t |} t |} ||| | fS) generates train datasetFtextadd_special_tokensr6Nr7cs g|]}|jjkr|ndqSrkr pad_token_id.0lr/r3r4 z.)tolistrencodelenrr build_inputs_with_special_tokensindex bos_token_idr!rnrcreate_position_idsnparray get_masks) r/r<r= prompt_ids answer_idsr8context_lengthZ mask_positionlabelpad_lenr:r;r3rrr4r@s&z(KeyWordGenDataset.train_dataset_functionc s8jjd}||}}d}j||}jj|djd}jj|djd}t|jdkr|djd}t|jdkr|djd}t|}||jjg}jjg||ddjjg} |t|} |jjg| }| jjg| d} j r0fdd| D} || fS) rgr6NTrirj max_lengthFr7cs g|]}|jjkr|ndqSrlrmrorrr3r4rsrtz>KeyWordGenDataset.train_dataset_functionv2..) rr rur build_promptrvrw eos_token_idrnr) r/r<r=r!historyrrrr8r9rr3rrr4rHs&$ z*KeyWordGenDataset.train_dataset_functionv2cCs||}}d}|j||}t||jdkrH|d|jd}t||jdkrl|d|jd}|jj|d|jd}|jj|d|jd}|jt|}||jjg|}|jt|}||jjg|}||fS)generates eval datasetNr6Tr)rurrrwrr rvrn)r/r<r=rr8rrr3r3r4rIsz)KeyWordGenDataset.eval_dataset_functionv2cCs||}}t||jdkr6|d|jd}t||jdkrZ|d|jd}|jj|dd}|jj|dd}|jt|}||jjg|}|jt|}||jjg|}||fS)rr7NTrh)rurwrr rrvrn)r/r<r=r8rrr3r3r4rBsz'KeyWordGenDataset.eval_dataset_functioncCs|jd}|tj|tjd}t||}t|dddf}ttj||ftjd}|D]}d|ddd|f<q^t| tj }| tj}t |d}|S)zgenerate mask from input idrshapedtypeNrr6) rr|onesr.equalargwheretrilfloat32 logical_notastypebool_ expand_dims)r/r8rz seq_lengthmaskcontext_lengthsr;rr3r3r4r~ s    zKeyWordGenDataset.get_masksTc s|jd|durdg}|tjtjd}t||}t|dddf}|rtjtjd}t|D]\} } || || d<qjfdd|D} tj | dd  } tj || fdd }n:tjtjd}t|D] \} } || s|| || d<q|S) z6generate position ids from input id and mask positionsrNFrrrc s:g|]2}ttj|tjdtj|tjddfqS)rr6)r| concatenatezerosint64arange)rprrr3r4rs/s z6KeyWordGenDataset.get_position_ids..)axis) rr|rr.rrrr enumeratestacksqueeze) r/r8mask_positions use_gmasksrzZposition_encoding_2drrr:irZblock_position_idsr3rr4get_position_idss(   z"KeyWordGenDataset.get_position_idsc CsX|jd}|}|tj|tjd}t||}t|dddf}|j|||d}|S)z#generate position ids from input idrrNr)rr)rr|rr.rrr) r/r8Zgmask_token_idrseqsrrrr:r3r3r4r{<s  z%KeyWordGenDataset.create_position_ids)N)r)NrT)r)__name__ __module__ __qualname____doc__dictr5 classmethodrLr&r'r@rHrIrBr~rr{r3r3r3r4r s0 " (        r ) rrVrZmindspore.common.dtypecommonrr-Z)mindspore.dataset.transforms.c_transformsr0 transforms c_transformsr+numpyr|!mindformers.models.base_tokenizerrZ mindformers.dataset.base_datasetrZmindformers.dataset.dataloaderrZ"mindformers.models.build_tokenizerrmindformers.tools.loggerrmindformers.tools.registerrrmindformers.version_controlr registerDATASETr r3r3r3r4s       pengcheng.mind-mindformer/mindformers/dataset/__pycache__/labels.cpython-39.pyc000066400000000000000000000301311476650141400301650ustar00rootroot00000000000000a ?cf@?@sTdZgdZgdZgdZddgZddgZddgZgdZeeeeeeed Zd S) zlabels of imagenet(ZtenchZgoldfishzgreat white sharkz tiger sharkZ hammerheadz electric rayZstingrayZcockhenZostrichZ bramblingZ goldfinchz house finchZjuncozindigo buntingZrobinZbulbulZjayZmagpieZ chickadeez water ouzelZkitez bald eagleZvulturezgreat grey owlzEuropean fire salamanderz common newtZeftzspotted salamanderZaxolotlZbullfrogz tree frogz tailed frogZ loggerheadzleatherback turtlez mud turtleZterrapinz box turtlez banded geckoz common iguanazAmerican chameleonZwhiptailZagamazfrilled lizardzalligator lizardz Gila monsterz green lizardzAfrican chameleonz Komodo dragonzAfrican crocodilezAmerican alligatorZ triceratopsz thunder snakezringneck snakez hognose snakez green snakez king snakez garter snakez water snakez vine snakez night snakezboa constrictorz rock pythonz Indian cobraz green mambaz sea snakez horned viperZ diamondbackZ sidewinder trilobiteZ harvestmanscorpionzblack and gold garden spiderz barn spiderz garden spiderz black widowZ tarantulaz wolf spidertickZ centipedez black grouseZ ptarmiganz ruffed grousezprairie chickenZpeacockZquailZ partridgez African greyZmacawzsulphur-crested cockatooZlorikeetZcoucalz bee eaterZhornbillZ hummingbirdZjacamarZtoucanZdrakezred-breasted merganserZgoosez black swanZtuskerZechidnaplatypusZwallabyZkoalaZwombatZ jellyfishz sea anemonez brain coralZflatwormZnematodeZconchZsnailslugzsea slugZchitonzchambered nautiluszDungeness crabz rock crabz fiddler crabz king crabzAmerican lobsterz spiny lobstercrayfishz hermit crabZisopodz white storkz black storkZ spoonbillflamingozlittle blue heronzAmerican egretZbitterncraneZlimpkinzEuropean gallinulez American cootZbustardzruddy turnstonezred-backed sandpiperZredshankZ dowitcherZ oystercatcherZpelicanz king penguinZ albatrossz grey whalez killer whaleZdugongzsea lionZ ChihuahuazJapanese spanielz Maltese dogZPekinesezShih-TzuzBlenheim spanielZpapillonz toy terrierzRhodesian ridgebackz Afghan houndZbassetZbeagleZ bloodhoundZbluetickzblack-and-tan coonhoundz Walker houndzEnglish foxhoundZredboneZborzoizIrish wolfhoundzItalian greyhoundZwhippetz Ibizan houndzNorwegian elkhoundZ otterhoundZSalukizScottish deerhoundZ WeimaranerzStaffordshire bullterrierzAmerican Staffordshire terrierzBedlington terrierzBorder terrierzKerry blue terrierz Irish terrierzNorfolk terrierzNorwich terrierzYorkshire terrierzwire-haired fox terrierzLakeland terrierzSealyham terrierZAiredaleZcairnzAustralian terrierzDandie Dinmontz Boston bullzminiature schnauzerzgiant schnauzerzstandard schnauzerzScotch terrierzTibetan terrierz silky terrierzsoft-coated wheaten terrierzWest Highland white terrierZLhasazflat-coated retrieverzcurly-coated retrieverzgolden retrieverzLabrador retrieverzChesapeake Bay retrieverzGerman short-haired pointerZvizslazEnglish setterz Irish setterz Gordon setterzBrittany spanielZclumberzEnglish springerzWelsh springer spanielzcocker spanielzSussex spanielzIrish water spanielZkuvaszZ schipperkeZ groenendaelZmalinoisZbriardZkelpieZkomondorzOld English sheepdogzShetland sheepdogZcolliez Border colliezBouvier des FlandresZ RottweilerzGerman shepherdZDobermanzminiature pinscherzGreater Swiss Mountain dogzBernese mountain dogZ AppenzellerZ EntleBucherboxerz bull mastiffzTibetan mastiffzFrench bulldogz Great Danez Saint Bernardz Eskimo dogZmalamutezSiberian husky dalmatianZ affenpinscherZbasenjiZpugZLeonbergZ NewfoundlandzGreat PyreneesZSamoyedZ PomeranianZchowZkeeshondzBrabancon griffonZPembrokeZCardiganz toy poodlezminiature poodlezstandard poodlezMexican hairlessz timber wolfz white wolfzred wolfZcoyoteZdingoZdholezAfrican hunting dogZhyenazred foxzkit foxz Arctic foxzgrey foxZtabbyz tiger catz Persian catz Siamese catz Egyptian catZcougarZlynxZleopardz snow leopardZjaguarZlionZtigerZcheetahz brown bearzAmerican black bearzice bearz sloth bearZmongooseZmeerkatz tiger beetleZladybugz ground beetlezlong-horned beetlez leaf beetlez dung beetlezrhinoceros beetleZweevilZflyZbeeantZ grasshopperZcricketz walking stickZ cockroachZmantisZcicadaZ leafhopperZlacewing dragonflyZ damselflyZadmiralZringletZmonarchzcabbage butterflyzsulphur butterflyZlycaenidstarfishz sea urchinz sea cucumberz wood rabbithareZAngoraZhamsterZ porcupinez fox squirrelZmarmotbeaverz guinea pigZsorrelZzebraZhogz wild boarZwarthogZ hippopotamusoxz water buffaloZbisonZramZbighornZibexZ hartebeestZimpalaZgazellez Arabian camelllamaZweaselZminkZpolecatzblack-footed ferretZotterZskunkZbadgerZ armadillozthree-toed slothZ orangutanZgorillaZ chimpanzeeZgibbonZsiamangZguenonZpatasZbaboonZmacaqueZlangurZcolobuszproboscis monkeyZmarmosetZcapuchinz howler monkeyZtitiz spider monkeyzsquirrel monkeyzMadagascar catZindrizIndian elephantzAfrican elephantz lesser pandaz giant pandaZ barracoutaZeelZcohoz rock beautyz anemone fishZsturgeonZgarZlionfishZpufferZabacusZabayaz academic gown accordionzacoustic guitarzaircraft carrierZairlinerZairshipZaltarZ ambulanceZ amphibianz analog clockZapiaryZapronZashcanz assault rifleZbackpackZbakeryz balance beamZballoonZ ballpointzBand AidZbanjoZ bannisterZbarbellz barber chairZ barbershopZbarnZ barometerbarrelZbarrowZbaseballZ basketballZbassinetZbassoonz bathing capz bath towelZbathtubz beach wagonZbeaconZbeakerZbearskinz beer bottlez beer glassz bell cotebibzbicycle-built-for-twoZbikiniZbinderZ binocularsZ birdhouseZ boathouseZbobsledzbolo tieZbonnetZbookcaseZbookshopZ bottlecapZbowzbow tieZbrassZ brassiereZ breakwaterZ breastplateZbroombucketZbucklezbulletproof vestz bullet trainz butcher shopcabZcaldronZcandlecannonZcanoez can openerZcardiganz car mirrorZcarouselzcarpenter's kitZcartonz car wheelz cash machineZcassettezcassette playerZcastleZ catamaranz CD playerZcellozcellular telephonechainzchainlink fencez chain mailz chain sawZchestZ chiffonierZchimez china cabinetzChristmas stockingZchurchZcinemaZcleaverzcliff dwellingZcloakZclogzcocktail shakerz coffee mugZ coffeepotZcoilzcombination lockzcomputer keyboardZ confectioneryzcontainer shipZ convertibleZ corkscrewZcornetz cowboy bootz cowboy hatZcradler z crash helmetcrateZcribz Crock Potz croquet ballZcrutchZcuirassZdamZdeskzdesktop computerzdial telephoneZdiaperz digital clockz digital watchz dining tableZdishragZ dishwasherz disk brakeZdockZdogsledZdomeZdoormatzdrilling platformZdrumZ drumstickZdumbbellz Dutch ovenz electric fanzelectric guitarzelectric locomotivezentertainment centerZenvelopezespresso makerz face powderz feather boafileZfireboatz fire enginez fire screenZflagpoleZflutez folding chairzfootball helmetZforkliftZfountainz fountain penz four-posterz freight carz French hornz frying panzfur coatz garbage truckZgasmaskzgas pumpZgobletzgo-kartz golf ballZgolfcartZgondolaZgongZgownz grand pianoZ greenhouseZgrillez grocery storeZ guillotinez hair slidez hair sprayz half trackZhammerZhamperz hand blowerzhand-held computerZ handkerchiefz hard discZ harmonicaZharpZ harvesterZhatchetZholsterz home theaterZ honeycombhookZ hoopskirtzhorizontal barz horse cartZ hourglassZiPodZironzjack-o'-lanternZjeanZjeepZjerseyz jigsaw puzzleZ jinrikishaZjoystickZkimonozknee padknotzlab coatZladleZ lampshadelaptopz lawn mowerzlens capz letter openerlibraryZlifeboatZlighterZ limousineZlinerZlipstickZLoaferZlotionZ loudspeakerZloupeZ lumbermillzmagnetic compassZmailbagZmailboxmaillotr z manhole coverZmaracaZmarimbamaskZ matchstickZmaypoleZmazez measuring cupzmedicine chestZmegalithZ microphoneZ microwavezmilitary uniformzmilk canZminibusZ miniskirtZminivanZmissileZmittenz mixing bowlz mobile homezModel TZmodemZ monasterymonitorZmopedZmortarZ mortarboardZmosquez mosquito netz motor scooterz mountain bikez mountain tentZmouseZ mousetrapz moving vanZmuzzleZnailz neck braceZnecklaceZnipplenotebookZobeliskZoboeZocarinaZodometerz oil filterZorganZ oscilloscopeZ overskirtZoxcartz oxygen maskpacketpaddleZ paddlewheelZpadlockZ paintbrushZpajamaZpalaceZpanpipez paper towelZ parachutez parallel barsz park benchz parking meterz passenger carZpatioz pay-phoneZpedestalz pencil boxzpencil sharpenerZperfumez Petri dishZ photocopierpickZ pickelhaubez picket fenceZpickupZpierz piggy bankz pill bottleZpillowzping-pong ballZpinwheelZpirateZpitcherplaneZ planetariumz plastic bagz plate rackplowZplungerzPolaroid camerapolez police vanZponchoz pool tablez pop bottlepotzpotter's wheelz power drillz prayer rugprinterZprisonZ projectileZ projectorZpuckz punching bagZpurseZquillZquiltZracerZracketZradiatorZradiozradio telescopez rain barrelzrecreational vehicleZreelz reflex cameraZ refrigeratorzremote control restaurantrevolverZriflez rocking chairZ rotisseriez rubber eraserz rugby ballrulez running shoesafez safety pinZ saltshakerZsandalZsarongsaxZscabbardscalez school busschoonerZ scoreboardscreenZscrewZ screwdriverz seat beltzsewing machineZshieldz shoe shopZshojizshopping basketz shopping cartZshovelz shower capzshower curtainskizski maskz sleeping bagz slide rulez sliding doorslotZsnorkelZ snowmobileZsnowplowzsoap dispenserz soccer ballsockz solar dishZsombreroz soup bowlz space barz space heaterz space shuttleZspatulaZ speedboatz spider webZspindlez sports carZ spotlightstagezsteam locomotivezsteel arch bridgez steel drumZ stethoscopeZstolez stone wallZ stopwatchZstoveZstrainerZ streetcarZ stretcherz studio couchZstupaZ submarineZsuitZsundialZsunglassZ sunglassesZ sunscreenzsuspension bridgeZswabZ sweatshirtzswimming trunksZswingswitchZsyringez table lampZtankz tape playerteapotZteddyZ televisionz tennis ballZthatchztheater curtainZthimbleZthresherZthronez tile roofZtoasterz tobacco shopz toilet seattorchz totem polez tow truckZtoyshopZtractorz trailer truckZtrayz trench coatZtricycleZtrimaranZtripodztriumphal archZ trolleybusZtromboneZtubZ turnstileztypewriter keyboardumbrellaZunicycleZuprightZvacuumZvaseZvaultZvelvetzvending machineZvestmentZviaductZviolinZ volleyballz waffle ironz wall clockZwalletZwardrobeZwarplaneZ washbasinZwasherz water bottlez water jugz water towerz whiskey jugZwhistleZwigz window screenz window shadez Windsor tiez wine bottleZwingwokz wooden spoonZwoolz worm fenceZwreckZyawlZyurtzweb sitez comic bookzcrossword puzzlez street signz traffic lightz book jacketmenuZplateZ guacamoleZconsommezhot potZtriflez ice creamz ice lollyz French loafZbagelZpretzelZ cheeseburgerZhotdogz mashed potatoz head cabbageZbroccoliZ cauliflowerZzucchinizspaghetti squashz acorn squashzbutternut squashZcucumber artichokez bell pepperZcardoonZmushroomz Granny Smith strawberryZorangeZlemonfigZ pineappleZbananaZ jackfruitz custard appleZ pomegranateZhayZ carbonarazchocolate sauceZdoughz meat loafpizzaZpotpieZburritozred wineZespressocupZeggnogZalpZbubbleZcliffz coral reefZgeyserZlakesideZ promontoryZsandbarZseashoreZvalleyZvolcanoZ ballplayerZgroomz scuba diverZrapeseedZdaisyzyellow lady's slipperZcornZacornZhipZbuckeyez coral fungusZagaricZ gyromitraZ stinkhornZ earthstarzhen-of-the-woodsZboleteZearz toilet tissue)Oz B-addresszB-bookz B-companyzB-gamez B-governmentzB-moviezB-namezB-organizationz B-positionzB-scenez I-addresszI-bookz I-companyzI-gamez I-governmentzI-moviezI-namezI-organizationz I-positionzI-scenez S-addresszS-bookz S-companyzS-gamez S-governmentzS-moviezS-namezS-organizationz S-positionzS-scene)Z contradiction entailmentZneutralZ unacceptableZ acceptablenegativepositive)ZworldZsportsbusinesszsci/tech)imagenetZcluenerZmnliZcolaZsst2ZimdbZag_newsN) __doc__Zimagenet_labelscluener_labelsZ mnli_labelsZ cola_labelsZ sst2_labelsZ imdb_labelsZag_news_labelslabelsrLrLE/home/ma-user/work/mindformers-baichuan/mindformers/dataset/labels.pys Npengcheng.mind-mindformer/mindformers/dataset/__pycache__/mask_language_model_dataset.cpython-39.pyc000066400000000000000000000075621476650141400344220ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlmmZddlmm m Z ddl m Z mZddlmZddlmZddlmZddlmZe ejGd d d eZdS) zMasked Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)MaskLanguageModelDataseta Bert pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MaskLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['fill_mask']['bert_tiny_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/bert.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MaskLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MaskLanguageModelDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstdttdd}ttdd}|||||\}}t|}|j j dksn|j j dksnt dg}|j j r|j d }tj|rt|D]<\}}}|D],} | d s| d r|tj|| qq|n|d s|d r|}n>|j jr*|j j}t|ttfrdata_loader must contain dataset_dir or dataset_files,but get .) dataset_files num_shardsshard_id) default_args)drop_remainderoutput_columnsnum_parallel_workers)columns) input_columns)'rinfointosgetenvinit_dataset_config_check_device_rank_for_parallelcopydeepcopy data_loadertypeNotImplementedErrorrpoppathisdirwalkendswithappendjoinsortr isinstancelisttuple ValueErrorrbatch batch_sizerrrprojectrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdata_dirr_ffiledataset type_cast_op input_argrFZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask_language_model_dataset.py__new__7sX              z MaskLanguageModelDataset.__new__)N)__name__ __module__ __qualname____doc__dictrHrFrFrFrGr sr )rLrr"Zmindspore.common.dtypecommondtyper9Z)mindspore.dataset.transforms.c_transformsrC transforms c_transformsr7mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetrregisterDATASETr rFrFrFrGs     pengcheng.mind-mindformer/mindformers/dataset/__pycache__/mim_dataset.cpython-39.pyc000066400000000000000000000062401476650141400312160ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZeejGd d d eZdS)zMasked Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) build_mask)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS) MIMDatasetap Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j }t |j }|durx||}|dur|jD]}t||||j|jd}q|durt|||j|j|j|jd }|j|jd }|j|j|j|jd }||j}|S) Nz)Now Create Masked Image Modeling Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args) input_columnsnum_parallel_workerspython_multiprocessing)routput_columnsrr)columns)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr mask_policyr sampler use_samplerrrrrrprojectbatch batch_sizerrepeat) clsr rank_id device_numdatasetr!maskr#columnr/J/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mim_dataset.py__new__8sB         zMIMDataset.__new__)N)__name__ __module__ __qualname____doc__dictr1r/r/r/r0r sr )r5rmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrr-rr!r r#r base_datasetr registerDATASETr r/r/r/r0s        pengcheng.mind-mindformer/mindformers/dataset/__pycache__/question_answering_dataset.cpython-39.pyc000066400000000000000000000065641476650141400343710ustar00rootroot00000000000000a ?cf@sdZddlZddlmmZddlmmm Z ddl m Z m Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZe e jGd d d eZdS)zQuestion Answering Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_tokenizer) build_sampler)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)QuestionAnsweringDataseta Question Answering Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import QuestionAnsweringDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['question_answering']['qa_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/question_answering.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='QuestionAnsweringDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = QuestionAnsweringDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j}t|j j |j j ||j j |j j |j||d}t|j}|dur||}|j|j|j|j|jd}||j}ttj}|jD]}t|||d }q|S) z new methodz&Now Create Question Answering Dataset.RANK_ID0 RANK_SIZE1) class_name dataset_dir tokenizerstage column_namesnum_parallel_workers num_shardsshard_idN)drop_remainderoutput_columnsr) input_columns)rinfoinit_dataset_configintosgetenvrrr data_loadertyperrrrr sampler use_samplerbatch batch_sizerrrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdatasetr$ type_cast_op input_argr3Y/home/ma-user/work/mindformers-baichuan/mindformers/dataset/question_answering_dataset.py__new__9s6        z QuestionAnsweringDataset.__new__)N)__name__ __module__ __qualname____doc__dictr5r3r3r3r4r sr )r9r Zmindspore.common.dtypecommondtyper+Z)mindspore.dataset.transforms.c_transformsr0 transforms c_transformsr)mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlrZmodels.build_tokenizerrr$r dataloaderr base_datasetr registerDATASETr r3r3r3r4s       pengcheng.mind-mindformer/mindformers/dataset/__pycache__/reward_model_dataset.cpython-39.pyc000066400000000000000000000151511476650141400331010ustar00rootroot00000000000000a ?cfr(@sdZddlZddlZddlZddlZddlmmZ ddl m m m ZddlmZmZddlmZddlmZddlmZddlmZdd lmZdeeed d d ZeejGdddeZ dS)zCausal Image Modeling Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_tokenizer)build_dataset_loader) BaseDataset)rankdispad_idc Cst|}||||d|}||||d|}||||d|}||||d|}||||d|}||||d|}t||f} t||f} t||f}tj| | kdd}| || ||fS)z^ Generate position_id and attention_mask according to input_ids considering eos reset r)axis)intnp concatenatesum) chosen_input_idschosen_attention_maskrejected_input_idsrejected_attention_mask position_id loss_maskend_indr r r input_idsattention_maskrS/home/ma-user/work/mindformers-baichuan/mindformers/dataset/reward_model_dataset.pyget_input_data_batch_slice_mapsrc@sDeZdZdZd edddZeddZedd Zed d Z dS) RewardModelDatasetazCausal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import RewardModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='RewardModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = RewardModelDataset(config.train_dataset_task.dataset_config) N)dataset_configcstdtd|ttddttdd}t|}||td| |\}|_ ||_ |j j dkr|j j d kr||}n ||}td |s|rd |jn0|j||j|d krtd |jd |dtd|j|j|j|jd}|jdur$|jndtdfdd}t|||j|jd}|j|jd}ttj}ttj}t|d|d}t|d|d}t|d|d}t|d|d}t|d|d}| |j }|S)Nz Now Create Reward Model Dataset.r RANK_ID0 RANK_SIZE1z!Now Create Reward Model Dataset1. MindDatasetTFRecordDatasetz"Now Create Reward Model Dataset1.5rz batch size z' should be a multiple of device number z-. You should change the args: per_batch_size.z"Now Create Reward Model Dataset1.6)drop_remainderoutput_columnsr z!Now Create Reward Model Dataset2.c st|||||||d S)N)r r )r)rrrrrrrr r rank_idrros  z,RewardModelDataset.__new__.. input_columnsr()columnsr)r- operationsrrrr)!rinfoprintrosgetenvcopydeepcopyinit_dataset_config_check_device_rank_for_parallelr* device_num data_loadertype_process_raw_text_data_process_mindrecord_data_is_semi_full_batch_is_data_parallel batch_size ValueErrorbatchr'r- pad_token_idrr(projectCTypeCastmstypeint32float16repeat)clsr r8datasetmap_func type_cast_opZtype_cast_op_floatrr)r__new__Ms\                zRewardModelDataset.__new__cs:|j}t||jfdd}t|||j|jd}|S)zPreprocess data for gpt2 modelcs$|}|dddd}|dS)N max_lengthTF)paddingrO truncationadd_special_tokensr)tolistget) input_datarrO tokenizerrrrLs  z7RewardModelDataset._prepare_for_model..map_funcr,)rWrrOrr-)rJrKr tokenizer_configrLrrVr_prepare_for_modelsz%RewardModelDataset._prepare_for_modelcCs6|jd}t|j||j|jdd}|||}|S)zProcess the text data dataset_dir)rZ num_shardsshard_id default_args)r9poprr8r*rY)rJr rZrKrrrr;s  z)RewardModelDataset._process_raw_text_datac Csg}td}|jjr|jd}tj|rt|D]>\}}}|D].}t ||s`| drF| tj ||qFq8| qt ||s| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdt|j||j|j|jdd} | S)zProcess the mindrecord dataz mindrecord0*$rZz .tfrecordz>data_loader must contain dataset_dir or dataset_files,but get .) dataset_filesr[r\ columns_listr])recompiler9rZr_r2pathisdirwalkfindallendswithappendjoinsortra isinstancelisttupler@rr8r*r-) rJr ra mind_compiledata_dirr_ffilerKrrrr<s6      z+RewardModelDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictrN classmethodrYr;r<rrrrr3s9  r)rrr )!ryr2r4rcnumpyrZmindspore.common.dtypecommondtyperFZ)mindspore.dataset.transforms.c_transformsrK transforms c_transformsrDmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlrZ"mindformers.models.build_tokenizerr dataloaderr base_datasetr rrregisterDATASETrrrrrs$       pengcheng.mind-mindformer/mindformers/dataset/__pycache__/text_classification_dataset.cpython-39.pyc000066400000000000000000000070221476650141400344720ustar00rootroot00000000000000a ?cf*@sdZddlZddlZddlmmZddlmm m Z ddl m Z mZddlmZddlmZddlmZddlmZe ejGd d d eZdS) zText Classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDatasetc@s eZdZdZdedddZdS)TextClassificationDataseta Text Classification dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TextClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_classification']['txtcls_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/text_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TextClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TextClassificationDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cs`td||ttdd}ttdd}t|}d|jvr|jj rg}|jj }tj |rt |D]<\}}}|D],} | ds| dr|tj || qqrq| ds| dr||n t|jj}|jd t|j|||d d } | j|j|j|j|jd } | j|jd } | |j} ttj} |jD]} t| | | d} qF| S)Nz'Now Create Text Classification Dataset.RANK_ID0 RANK_SIZE1Z data_filesz .mindrecordz .tfrecord dataset_dir) dataset_files num_shardsshard_id) default_args)drop_remainderoutput_columnsnum_parallel_workers)columns) operations input_columns) rinfoinit_dataset_configintosgetenvcopydeepcopy data_loaderrpathisdirwalkendswithappendjoinlistrpoprbatch batch_sizerrrprojectrepeatCTypeCastmstypeint32r) clsr rank_id device_numrdata_dirr_ffiledataset type_cast_op input_argr=Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/text_classification_dataset.py__new__6sF           z!TextClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr?r=r=r=r>r sr )rCrrZmindspore.common.dtypecommondtyper0Z)mindspore.dataset.transforms.c_transformsr: transforms c_transformsr.mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetrregisterDATASETr r=r=r=r>s     token_classification_dataset.cpython-39.pyc000066400000000000000000000065411476650141400345540ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__a ?cf@sdZddlZddlmZmZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZeejGd ddeZdS)zToken classification Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader)build_tokenizer)build_transforms) build_sampler) BaseDatasetc@s eZdZdZdedddZdS)TokenClassificationDataseta Token classification Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TokenClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['token_classification']['tokcls_bert_base_chinese'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/token_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TokenClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TokenClassificationDataset(config.train_dataset_task.dataset_config) N)dataset_configc Cstd||ttdd}ttdd}t|j||dd}t|j }t |j d|id}t |j }t |j}|d ur||}|d urt||j||j|j|jd }|d urt||jd ||j|jd }|j|j|j|jd }||j}|S)z new methodz(Now Create Token classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerN) input_columns operationsoutput_columnsnum_parallel_workerspython_multiprocessingr)rrrr)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr rr text_transformslabel_transformsr sampler use_samplerrrrrrbatch batch_sizerrepeat) clsrrank_id device_numdatasetrr#r$r%r.[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/token_classification_dataset.py__new__8sH        z"TokenClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr0r.r.r.r/r sr )r4r mindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderrZmodels.build_tokenizerr transformsr r%r base_datasetr registerDATASETr r.r.r.r/s        pengcheng.mind-mindformer/mindformers/dataset/__pycache__/translation_dataset.cpython-39.pyc000066400000000000000000000125031476650141400327710ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlZddlmmZddl m m m Z ddlmZmZddlmZddlmZddlmZddlmZd d lmZd gZeejGd d d eZdS) z T5 Dataset.N)MindFormerRegisterMindFormerModuleType)logger)get_dataset_map)build_dataset_loader) BaseDataset) AutoTokenizerTranslationDatasetc@sDeZdZdZd edddZeddZedd Zed d Z dS) r afTranslation dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TranslationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['translation']['t5_small'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/t5.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TranslationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TranslationDataset(config.train_dataset_task.dataset_config) N)dataset_configcCstd|||jjdkr,||}n ||}|j|j|j |j d}| |j }t tj}|jD]}t|||d}qj|S)NzNow Create Translation Dataset. MindDataset)drop_remaindernum_parallel_workers) operations input_columns)rinfoinit_dataset_config data_loadertype_process_raw_text_data_process_mindrecord_databatch batch_sizerrrepeatCTypeCastmstypeint32rr)clsr dataset type_cast_op input_argr#R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/translation_dataset.py__new__:s        zTranslationDataset.__new__csht|j|j|j|jtd|fdd}t||ddggdd}|j gdd}|S) z/Maps the tokenizer on the source and the outputz1Start tokenize on the dataset using tokenizer: %scs|}t|tr|}|ddd}|}t|trH|}|ddd}t|dtj}t|dtj}t|dtj}|||fS)N max_lengthT)paddingr& truncation input_idsattention_mask)tolist isinstancebytesdecodenparrayrfloat32)srctgtoutputZ tgt_outputr)r*labelsprefixsrc_max_lengthtgt_max_length tokenizerr#r$pad_max_functionTs  z;TranslationDataset._tokenizer_map..pad_max_functionsourcetarget)r)r*r5)routput_columns)columns) r from_pretrainedrr7r8r9rrrproject)rr Ztokenizer_configr;r#r6r$_tokenizer_mapKs  z!TranslationDataset._tokenizer_mapcCsTttdd}ttdd}|jd}t|j|||dd}|||j}|S)zProcess the text dataRANK_ID0 RANK_SIZE1 dataset_dir)rG num_shardsshard_id default_args)intosgetenvrpoprrBr:)rr rank_id device_numrGr r#r#r$rjs z)TranslationDataset._process_raw_text_datac Cs ttdd}ttdd}t|}g}|jjr|jd}tj |rt |D]2\}}}|D]"} | drf| tj || qfqX|q| dr|}n:|jjr|jj}t|ttfrt|}ntd|jdtd |jt|j||||jd d } | S) zProcess the mindrecord datarCrDrErFrGz .mindrecordz>data_loader must contain dataset_dir or dataset_files,but get .z&Using args %s to instance the dataset.) dataset_filesrHrI columns_listrJ)rLrMrNcopydeepcopyrrGrOpathisdirwalkendswithappendjoinsortrSr,listtuple ValueErrorrrrr) rr rPrQrSdata_dirr_ffiler r#r#r$rws:        z+TranslationDataset._process_mindrecord_data)N) __name__ __module__ __qualname____doc__dictr% classmethodrBrrr#r#r#r$r s  )rirMrUnumpyr/Zmindspore.common.dtypecommondtyperZ)mindspore.dataset.transforms.c_transformsr transforms c_transformsrmindformers.tools.registerrrmindformers.tools.loggerrmindformers.version_controlr dataloaderr base_datasetr auto_classr __all__registerDATASETr r#r#r#r$s      pengcheng.mind-mindformer/mindformers/dataset/__pycache__/utils.cpython-39.pyc000066400000000000000000000014041476650141400300640ustar00rootroot00000000000000a ?cf@sdZddZdS)zDataset Utils.cCs|jdurpd|j_|j|j_|j|j_|j|j_|j|j_|j|j_|jj|j_|jj rf|jj |jj _ |j|j _ |j durd|j _|j|j _|j|j _|j|j _|j|j _|j|j _|jj|j _|j |j_ dS)zCheck dataset config.NFT) train_datasetdo_evalseed auto_tunefilepath_prefixautotune_per_stepprofile runner_config batch_sizemixup_op num_classesZtrain_dataset_taskdataset_config eval_datasetZeval_dataset_task)configrD/home/ma-user/work/mindformers-baichuan/mindformers/dataset/utils.pycheck_dataset_configs(               rN)__doc__rrrrrszero_shot_image_classification_dataset.cpython-39.pyc000066400000000000000000000073511476650141400366120ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/__pycache__a ?cf@sdZddlZddlmZddlmZddlmZddlm Z dd l m Z dd l m Z mZdd lmZe ejGd d d e ZdS)z'Zero Shot Image Classification Dataset.N)get_dataset_map)build_dataset_loader)build_transforms) BaseDataset)logger)MindFormerRegisterMindFormerModuleType)build_tokenizerc@s eZdZdZdedddZdS)"ZeroShotImageClassificationDataseta Zero Shot Image Classification Dataset API. output image, text, and label columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ZeroShotImageClassificationTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml") >>> config = MindFormerConfig(config_path) Note: Put cifar100 dataset to ./ The detailed data setting could refer to ./configs/clip/clip.md >>> config.eval_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.eval_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], ... [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000]]]]), Tensor(shape=[1, 100, 77], dtype=Int32, value= [[[49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], ... [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0]]]), Tensor(shape=[1], dtype=Int32, value= [49])] N)dataset_configcCstd||ttdd}ttdd}t|j||dd}t|j }t |j }t|j d|id}|d urt |d ||j|jd }|d urt |d ||j|jd }|j|j|j|jd }||j}|S)z New methodz2Now Create Zero Shot Image Classification Dataset.RANK_ID0 RANK_SIZE1) num_shardsshard_id) default_args tokenizerNimage) input_columns operationsnum_parallel_workerspython_multiprocessingtext)drop_remainderr)rinfoinit_dataset_configintosgetenvr data_loaderr transformsr rtext_transformsrrrbatch batch_sizerrepeat)clsr rank_id device_numdatasetr#rr$r,e/home/ma-user/work/mindformers-baichuan/mindformers/dataset/zero_shot_image_classification_dataset.py__new__Fs@      z*ZeroShotImageClassificationDataset.__new__)N)__name__ __module__ __qualname____doc__dictr.r,r,r,r-r s+r )r2r mindformers.version_controlr dataloaderrr#r base_datasetrtoolsrZtools.registerr r Zmodels.build_tokenizerr registerDATASETr r,r,r,r-s       pengcheng.mind-mindformer/mindformers/dataset/base_dataset.py000066400000000000000000000050161476650141400250570ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Dataset.""" import os import mindspore as ms import mindspore.dataset as ds class BaseDataset: """Base Dataset.""" def __init__(self, dataset_config: dict = None): self.dataset_config = dataset_config @classmethod def init_dataset_config(cls, dataset_config): """Init dataset config.""" if dataset_config is None: raise ValueError("dataset_config cannot be None!") ds.config.set_seed(dataset_config.seed) ds.config.set_prefetch_size(dataset_config.prefetch_size) ds.config.set_numa_enable(dataset_config.numa_enable) if dataset_config.auto_tune: if dataset_config.profile: raise EnvironmentError( "MindSpore's AutoTune is enabled, so Profile cannot be enabled," "now Profile's flag is True, please set to False!") os.makedirs(dataset_config.filepath_prefix, exist_ok=True) dataset_config.filepath_prefix = os.path.join(dataset_config.filepath_prefix, "autotune") ds.config.set_enable_autotune(True, filepath_prefix=dataset_config.filepath_prefix) ds.config.set_autotune_interval(dataset_config.autotune_per_step) @classmethod def _check_device_rank_for_parallel(cls, rank_id, device_num): """Check device num and rank id in auto parallel mode.""" if cls._is_semi_full_batch(): rank_id = None device_num = None return rank_id, device_num @classmethod def _is_semi_full_batch(cls): return ((ms.context.get_auto_parallel_context("parallel_mode") in ['semi_auto_parallel', 'auto_parallel']) and ms.context.get_auto_parallel_context("full_batch")) @classmethod def _is_data_parallel(cls): return ms.context.get_auto_parallel_context("parallel_mode") == ms.context.ParallelMode.DATA_PARALLEL pengcheng.mind-mindformer/mindformers/dataset/build_dataset.py000066400000000000000000000046561476650141400252550ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Dataset API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset( config: dict = None, default_args: dict = None, module_type: str = 'dataset', class_name: str = None, **kwargs): r"""Build dataset For MindFormer. Instantiate the dataset from MindFormerRegister's registry. Args: config (dict): The task dataset's config. Default: None. default_args (dict): The default argument of dataset API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset'. class_name (str): The class name of dataset API. Default: None. Return: The function instance of dataset API. Examples: >>> from mindformers import build_dataset >>> from mindformers.dataset import check_dataset_config >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_class_name = build_dataset(class_name='ImageCLSDataset', dataset_config=config.train_dataset_task) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/dataset/causal_language_model_dataset.py000066400000000000000000000252731476650141400304470ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.models.build_tokenizer import build_tokenizer from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(input_ids, eos_token_id, dis, rank_id: int = 0): """ Generate position_id and attention_mask according to input_ids considering eos reset Args: input_ids: the input token ids eos_token_id: the id for dis: the slice value for each rank rank_id: the current rank id Returns: batch_input_ids: the input token ids batch_position_ids: the position ids cosidering eos reset batch_attention_mask: the attention mask considering eos reset """ rank = int(rank_id) input_ids = input_ids[rank*dis: (rank + 1)*dis] seq_length = input_ids.shape[1] - 1 # Initialize position_ids and attention_mask batch_input_ids = input_ids batch_position_ids = np.ones((dis, seq_length)) batch_attention_mask = np.ones((dis, seq_length, seq_length)) # Loop through batches for bs_i in range(len(input_ids)): # Get normal position_ids and attention_mask local_ids = input_ids[bs_i] batch_attention_mask[bs_i] = np.tril(np.ones(shape=(seq_length, seq_length))) batch_position_ids[bs_i] = np.arange(seq_length) # Find the index of eos_index = batch_position_ids[bs_i, local_ids[:-1] == eos_token_id].astype(np.int32) prev_index = 0 for i in range(eos_index.size): # Reset position_ids and attention_mask considering index = eos_index[i] batch_attention_mask[bs_i, (index + 1):, :(index + 1)] = 0 batch_position_ids[bs_i, (index + 1):] -= (index + 1 - prev_index) prev_index = index + 1 return batch_input_ids, batch_position_ids, batch_attention_mask @MindFormerRegister.register(MindFormerModuleType.DATASET) class CausalLanguageModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import CausalLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='CausalLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = CausalLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Causal Language Model Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) type_cast_op = C.TypeCast(mstype.int32) if dataset_config.eos_reset: if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) map_func = lambda input_ids: get_input_data_batch_slice_map(input_ids, eos_token_id=dataset_config.eos_token_id, rank_id=rank_id, dis=dis) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) for input_arg in dataset_config.output_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) else: dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if (re.findall(mind_compile, file) and not file.endswith(".db")) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") print(f"build dataset with : {dataset_config.data_loader}") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) if dataset_config.consumed_samples: dataset = dataset.skip(dataset_config.consumed_samples) print(f"skip {dataset_config.consumed_samples} samples") return dataset pengcheng.mind-mindformer/mindformers/dataset/contrastive_language_image_pretrain_dataset.py000066400000000000000000000112171476650141400334170ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Dataset.""" import os from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset from ..tools import logger from ..models.build_tokenizer import build_tokenizer from ..tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET) class ContrastiveLanguageImagePretrainDataset(BaseDataset): r""" Contrastive Language Image Pretrain Dataset API. output image and text columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ContrastiveLanguageImagePretrainTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_pretrain_flickr8k.yaml") >>> config = MindFormerConfig(config_path) Note: Put flickr8k dataset to ./checkpoint_download The detailed data setting could refer to ./configs/clip/clip.md >>> config.train_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.train_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[4.99690473e-001, 6.74871564e-001, ... 3.68304640e-001, 2.36918822e-001], [7.91658998e-001, 7.62462139e-001, ... -2.01033935e-001, -1.13443382e-001], ... [-5.98575652e-001, -6.12795711e-001, ... 1.47755420e+000, 1.46333420e+000], [-3.85274649e-001, -6.27015769e-001, ... 1.42067397e+000, 1.43489408e+000], [-7.97656536e-001, -1.01095748e+000, ... 9.37191546e-001, 9.08751369e-001]]]]), Tensor(shape=[1, 77], dtype=Int32, value= [[49406, 1237, 18250 ... 0, 0, 0]])] """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Contrastive Language Image Pretrain Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: dataset = get_dataset_map(dataset, transforms, input_columns="image", num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if text_transforms is not None: dataset = get_dataset_map(dataset, text_transforms, input_columns="text", num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-mindformer/mindformers/dataset/dataloader/000077500000000000000000000000001476650141400241645ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__init__.py000066400000000000000000000024031476650141400262740ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers DataLoader.""" from .build_dataloader import build_dataset_loader from .flickr8k_dataloader import Flickr8kDataLoader from .multi_image_cap_dataloader import MultiImgCapDataLoader from .cifar100_dataloader import Cifar100DataLoader from .wmt16_dataloader import WMT16DataLoader from .cluener_dataloader import CLUENERDataLoader from .squad_dataloader import SQuADDataLoader from .adgen_dataloader import ADGenDataLoader __all__ = ['Flickr8kDataLoader', 'Cifar100DataLoader', 'WMT16DataLoader', 'CLUENERDataLoader', 'SQuADDataLoader', 'ADGenDataLoader', 'MultiImgCapDataLoader'] pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/000077500000000000000000000000001476650141400263745ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/__init__.cpython-39.pyc000066400000000000000000000013621476650141400325660ustar00rootroot00000000000000a ?cf@spdZddlmZddlmZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZgd Zd S) zMindFormers DataLoader.)build_dataset_loader)Flickr8kDataLoader)MultiImgCapDataLoader)Cifar100DataLoader)WMT16DataLoader)CLUENERDataLoader)SQuADDataLoader)ADGenDataLoader)rrrrrr rN)__doc__Zbuild_dataloaderrZflickr8k_dataloaderrZmulti_image_cap_dataloaderrZcifar100_dataloaderrZwmt16_dataloaderrZcluener_dataloaderrZsquad_dataloaderrZadgen_dataloaderr __all__r r R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/__init__.pys        pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/adgen_dataloader.cpython-39.pyc000066400000000000000000000077601476650141400342750ustar00rootroot00000000000000a ?cf`@shdZddlZddlZddlmZddlmZddlmZm Z e ej GdddZ Gdd d Z dS) zADGen DataLoaderN)GeneratorDataset)logger)MindFormerModuleTypeMindFormerRegisterc@seZdZdZddZdS)ADGenDataLoaderzADGen Dataloaderc Kstj|st|d|dvr*tdddg}t|ttfs\tdt|dt ||D] }t|t s`tdt|q`t|ttfrt |d krtd t|dt || d d t |||}d |d|d} t | | dd t||fd|i|S)a ADGen Dataloader API. Args: dataset_dir: The directory to ADGen dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 phase: The supported key words are in ["train", "dev"] Return: A GeneratorDataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import ADGenDataLoader >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> data_loader = ADGenDataLoader("./ADGen/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break  is not existed.)trainevalz!phase should be in train or eval.promptanswerzJcolumn_names should be a tuple or a list of string with length 2, but got z , length z8the item type of column_names should be string, but got zLorigin_columns should be a tuple or a list of string with length 2, but got NoneNz[DATASET] shuffle status is z , phase is .versionshuffle)ospathisfile ValueError isinstancetuplelist TypeErrortypelenstrpop ADGenDatasetrinfor) cls dataset_dirphaserorigin_columnskwargs column_namesnameZ adgen_datasetrr&Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/adgen_dataloader.py__new__s8       zADGenDataLoader.__new__N)__name__ __module__ __qualname____doc__r(r&r&r&r'rsrc@s*eZdZdZd ddZddZddZd S) rz ADGen Datasetrc Cstj|st|d||_||_i}g}g}|d|_|d|_t|jZ}|D]0}| t ||j| t ||jqV|||j<|||j<Wdn1s0Y||_ dS)aE ADGen Dataset Args: dataset_dir (str): The directory to ADGen dataset. phase (str): The supported key words are in ["train", "dev"] Return: A iterable dataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, phase. rrN) rrrrr r! prompt_columnresponse_columnopenappendjsonloadsexamples) selfr r"r!r4Z content_list summary_listfpliner&r&r'__init__Zs      (zADGenDataset.__init__cCst|j|jS)zGet the size of dataset)rr4r.)r5r&r&r'__len__{szADGenDataset.__len__cCs*|j|j||j|j|}}||fS)zReturn input data for model)r4r.r/)r5ir r r&r&r' __getitem__s"zADGenDataset.__getitem__N)r)r)r*r+r,r9r:r<r&r&r&r'rWs !r)r,r2rmindspore.datasetrmindformers.tools.loggerrmindformers.tools.registerrrregisterDATASET_LOADERrrr&r&r&r's   =pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/build_dataloader.cpython-39.pyc000066400000000000000000000043341476650141400343100ustar00rootroot00000000000000a ?cfr @sTdZddlZddlmZddlmZmZmZd e e e e dddZ d d Z e dS) zBuild DataLoader API.N)dataset)MindFormerRegisterMindFormerModuleTypeMindFormerConfigdataset_loader)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a!Build dataset loader For MindFormer. Instantiate the dataset loader from MindFormerRegister's registry. Args: config (dict): The task dataset loader's config. Default: None. default_args (dict): The default argument of dataset loader API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset loader'. class_name (str): The class name of dataset loader API. Default: None. Return: The function instance of dataset loader API. Examples: >>> from mindformers.dataset.dataloader import build_dataset_loader >>> dataset_loader_config = {'type': 'ImageFolderDataset', 'dataset_dir': './imagenet/train'} >>> # 1) use config dict to build dataset loader >>> dataset_loader_from_config = build_dataset_loader(dataset_loader_config) >>> # 2) use class name to build dataset loader >>> dataset_loader_class_name = build_dataset_loader( ... class_name='ImageFolderDataset', dataset_dir='./imagenet/train') N)r) isinstancedictrrget_instance_from_cfgrDATASET_LOADER get_instance)rrr r kwargsrZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/build_dataloader.pybuild_dataset_loadersrcCs@ttD]2}|drqtt|}t|rt|tj qdS)z2 register MindSpore builtin dataset loader class. __N) dirds startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_dataset_loader9s    r)NNrN) __doc__r mindsporerrmindformers.tools.registerrrrr strrrrrrrs  " cifar100_dataloader.cpython-39.pyc000066400000000000000000000141141476650141400344340ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__a ?cf@s|dZddlZddlZddlmZmZmZmZddlZ ddl m Z ddl m Z mZe ejGdddZGd d d ZdS) zCifar100 DataLoader.N)OptionalUnionListTuple)GeneratorDataset)MindFormerRegisterMindFormerModuleTypec@sVeZdZdZgdZd eeeeee efeeee ee eedd d Z dS) Cifar100DataLoaderz2Cifar100 Dataloader with class name as text columnimagetextlabelNtrainTFThis is a photo of {}.) dataset_dir column_namesstage fine_labelshufflehypothesis_templatec Kstj|st|d|dur(|j}t|ttfsHtdt |t |dkrftdt |t t |D](}t||t srtdt ||qr| ddt||||} t| ||d} t| d | j| S) a Cifar100 Dataloader API. Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python" column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 3. stage (Optional[str]): The supported key words are in ["train","test", "all"] fine_label (Optional[bool]): True for 100 classes, False for 10 classes. shuffle (Optional[bool]): Shuffle the samples. hypothesis_template (Optional[str]): Prompt template for class label. Return: A GeneratorDataset for Cifar100 dataset Raises: ValueError: Error input for dataset_dir and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Cifar100DataLoader >>> dataloader = Cifar100DataLoader("./cifar-100-python") >>> dataloader = dataloader.batch(1) >>> for item in dataloader: >>> print(item) >>> break [Tensor(shape=[1, 32, 32, 3], dtype=UInt8, value= [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], ... [195, 205, 193], [212, 224, 204], [182, 194, 167]]]]]), Tensor(shape=[1, 100], dtype=String, value= [['This is a photo of apple.', 'This is a photo of aquarium_fish.', 'This is a photo of baby.', 'This is a photo of bear.', ... 'This is a photo of woman.', 'This is a photo of worm.']]), Tensor(shape=[1], dtype=Int32, value= [19])]  is not existed.NzJcolumn_names should be a tuple or a list of string with length 3, but got rz0the length of column_names should be 3, but got z8the item type of column_names should be string, but got None)r label_names)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenrangestrpopCifar100DataSetrsetattrr) clsrrrrrrkwargsindexZcifar100_datasetcifar100_dataloaderr.]/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/cifar100_dataloader.py__new__s,,    zCifar100DataLoader.__new__)NrTFr) __name__ __module__ __qualname____doc__rr&rrrrboolr0r.r.r.r/r s r c@sBeZdZdZd eeeeeeedddZdd Zd d Z d S)r(z Cifar100 dataSet with class namerTr)rrrrcCsftj|st|d|dvr*tdtj|d}t|d}tj|dd}Wdn1sf0Y|r~|d }n|d }tt |D]} ||  || <q||_ g} tt |D]} | | || q|d vrtj|d } t| d}tj|dd} Wdn1s 0Y|r:| d } n| d} | d}|jd}||dddf}|dddd}||_| |_| |_|dvr8tj|d}t|d}tj|dd}Wdn1s0Y|r|d }n|d}|d}|jd}||dddf}|dddd}||_||_| |_|dkrbt||g|_| ||_| |_dS)a" Cifar100 Dataset Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python". stage (Optional[str]): The supported key words are in ["train", "test", "all"]. fine_label (Optional[bool]): True for 100 classes, False for 10 classes. hypothesis_template (Optional[str]): Prompt template for class label. Return: A iterable dataset for Cifar100 dataset. Raises: ValueError: Unsupported values for stage. r)rtestallz>unsupported stage, stage should be in ["train", "test", "all"]metarbbytes)encodingNsfine_label_namesscoarse_label_names)rr7rs fine_labelss coarse_labelssdatarr )r6r7r6r7)rrrrjoinopenpickleloadr%r$decoderappendformatshapereshape transposer rr np row_stack)selfrrrrZ meta_filefoZ meta_dictrr, hypothesis train_fileZ train_dictZ train_labelZ train_imageZ train_num test_fileZ test_dictZ test_labelZ test_imageZtest_numr.r.r/__init__es`  ,   .    .    zCifar100DataSet.__init__cCs|j||j|j|fS)getitemr )rKitemr.r.r/ __getitem__szCifar100DataSet.__getitem__cCs |jjdS)r$r)r rF)rKr.r.r/__len__szCifar100DataSet.__len__N)rTr) r1r2r3r4r&rr5rPrSrTr.r.r.r/r(cs  Pr()r4rrAtypingrrrrnumpyrImindspore.datasetrZtools.registerrr registerDATASET_LOADERr r(r.r.r.r/s  Hcluener_dataloader.cpython-39.pyc000066400000000000000000000140611476650141400345650ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__a ?cf1@sdZddlZddlZddlmZmZmZmZddlm Z ddl m Z m Z ddl mZe e jGd d d ZGd d d ZdS) zCLUENER DataLoader.N)OptionalUnionListTuple)GeneratorDataset)MindFormerRegisterMindFormerModuleType)cluener_labelsc@sDeZdZdZddgZd eeeeee efeedddZ dS) CLUENERDataLoaderzCLUENER Dataloadertextlabel_idNtrain) dataset_dir column_namesstagecKstj|st|d|dur(|j}t|ttfsHtdt |t |dkrftdt |t|dt rt|dt stdt |dd t |d| d dt ||}t||S) u CLUENER Dataloader API. Args: dataset_dir (str): The directory to cluener dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for cluener dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import CLUENERDataLoader >>> data_loader = CLUENERDataLoader("./cluener/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[], dtype=String, value= '虚幻引擎3动作游戏《黑光》新作公布'), Tensor(shape=[49], dtype=Int64, value= [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 15, 0, 0, 0, 0, 0, 5, 15, 15, 15, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])]  is not existed.NzJcolumn_names should be a tuple or a list of string with length 2, but got r z0the length of column_names should be 2, but got rz8the item type of column_names should be string, but got z and None)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpopCLUENERDataSetr)clsrrrkwargsZcluener_datasetr&\/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/cluener_dataloader.py__new__s*      zCLUENERDataLoader.__new__)Nr) __name__ __module__ __qualname____doc__rr!rrrrr(r&r&r&r'r sr c@sZeZdZdZdddZddZddZd d Zd d Zd dZ ddZ ddZ ddZ dS)r#zCLUENER DataSetrcCstj|st|d||_ddttD|_g|_g|_ |dkrbtj |d}| |nL|dkrtj |d}| |n*|dkrtj |d }| |ntd d S) aS CLUENER Dataset Args: dataset_dir (str): The directory to cluener dataset. stage (str): The supported key words are in ["train", "dev", "test"] Return: A iterable dataset for cluener dataset Raises: ValueError: Error input for dataset_dir, stage. rcSsi|]\}}||qSr&r&).0rlabelr&r&r' iz+CLUENERDataSet.__init__..rz train.jsondevzdev.jsontestz test.jsonzunsupported stage.N)rrrrr enumerater label2idtexts label_idsjoin_get_train_examples_get_dev_examples_get_test_examples)selfrrtrain_data_path dev_data_pathtest_data_pathr&r&r'__init__Ws     zCLUENERDataSet.__init__cCs|j|}|j|}||fSN)r5r6)r;itemr rr&r&r' __getitem__|s  zCLUENERDataSet.__getitem__cCs t|jSr@)r r5)r;r&r&r'__len__szCLUENERDataSet.__len__cCs|||S)zGet train examples._create_examples _read_json)r;r<r&r&r'r8sz"CLUENERDataSet._get_train_examplescCs|||S)zGet dev examples.rD)r;r=r&r&r'r9sz CLUENERDataSet._get_dev_examplescCs|||S)zGet test examples.rD)r;r>r&r&r'r:sz!CLUENERDataSet._get_test_examplescCs\t|ddd:}g}|D]}|t|q|WdS1sN0YdS)z!Reads a tab separated value file.rzutf-8)encodingN)open readlinesappendjsonloadsstrip)r; input_fileflinesliner&r&r'rFs  zCLUENERDataSet._read_jsonc Cs|dD]\}}|D]|\}}|D]n}|d|dkrNd|||d<q(d|||d<d|||d<t|dd|dD]}d|||<qq(qq |S)zGenerate labelr.rrzS-zB-zI-)itemsrange) r;rRr.lwords_indicesindexjr&r&r'generate_labelszCLUENERDataSet.generate_labelcCsn|D]d}|d}dgt|}d|vr2|||}g}|D]}||j|q:|j||j|qdS)zCreate Example.r Or.N)r r[rKr4r5r6)r;rQrRr r.rZ token_labelr&r&r'rEs  zCLUENERDataSet._create_examplesN)r) r)r*r+r,r?rBrCr8r9r:rFr[rEr&r&r&r'r#Us %r#)r,rrLtypingrrrrmindspore.datasetrZtools.registerrr labelsr registerDATASET_LOADERr r#r&r&r&r's   9flickr8k_dataloader.cpython-39.pyc000066400000000000000000000136201476650141400346450ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__a ?cf@sdZddlZddlmZddlmZmZmZmZddl m Z ddl m Z ddl mZmZeejGd d d ZGd d d ZdS) zFlickr8k DataLoader.N) defaultdict)OptionalUnionListTuple)GeneratorDataset) load_image)MindFormerRegisterMindFormerModuleTypec@sDeZdZdZddgZd eeeeee efeedddZ dS) Flickr8kDataLoaderzFlicker8k DataloaderimagetextNtrain) dataset_dir column_namesstagecKstj|d}tj|dd}tj|s8t|dtj|sRt|d|dur`|j}t|ttfst dt |t |dkrtdt |t|dt rt|d t std t |dd t |d | d dt|||}t||fi|S) a Flicker8k Dataloader API. Args: dataset_dir (str): The directory. Flickr_8k.devImages.txt, and Flickr8k.token.txt column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Flickr8kDataLoader >>> data_loader = Flickr8kDataLoader("./Flickr8k") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[1, 378, 500, 3], dtype=UInt8, value= [[[[ 89, 46, 55], [ 89, 62, 55], [ 83, 59, 49], ... [161, 152, 135], [162, 156, 142], [159, 162, 145]]]]), Tensor(shape=[1, 5], dtype=String, value= [['A woman and child sitting on a bench beside a wooden bear carving .', 'A woman and little girl are sitting on a wooden bench beside a wooden bear .', 'A woman holds a little girl next to a large wooden carving of a bear .', 'A woman with a young girl pose with bear statues in front of a store .', 'A young girl being held by an older woman wearing bluejeans sitting on the lap of a bear statue near the entrance to a wooden building .']])] Z Flickr8k_textZFlickr8k_Dataset is not existed.NzJcolumn_names should be a tuple or a list of string with length 2, but got z0the length of column_names should be 2, but got rz8the item type of column_names should be string, but got z and None)ospathjoinisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpopFlickr8kDataSetr)clsrrrkwargsannotation_dirZflick8k_datasetr)]/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/flickr8k_dataloader.py__new__s2*       zFlickr8kDataLoader.__new__)Nr) __name__ __module__ __qualname____doc__rr#rrrrr+r)r)r)r*r sr c@s*eZdZdZd ddZddZddZd S) r%zFlickr8k DataSetrcCstj|st|dtj|s4t|d||_|dkrtj|d}t|ddd}|}Wdn1s0Yn|dkrtj|d }t|ddd}|}Wdn1s0Yn~|d kr4tj|d }t|ddd}|}Wdn1s(0Yn(|d krTd dt |D}ntdtj|d} t| ddd}|} Wdn1s0Yt t } | D]:} | dd} | |vr| dd}| |  |q||_| |_dS)a Flickr8k Dataset Args: dataset_dir (str): The directory to images annotation_dir (str): The directory to Flickr_8k.trainImages.txt, Flickr_8k.testImages.txt, Flickr_8k.devImages.txt, and Flickr8k.token.txt stage (str): The supported key words are in ["train", "test", "dev", "all"] Return: A iterable dataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, annotation_dir, and stage. rrzFlickr_8k.trainImages.txtrzutf-8)encodingNtestzFlickr_8k.testImages.txtdevzFlickr_8k.devImages.txtallcSsg|]}|dr|qS)z.jpg)endswith).0filer)r)r* z,Flickr8kDataSet.__init__..zunsupported stage.zFlickr8k.token.txt#r )rrrrrropenread splitlineslistdirrrsplitappend image_names dataset_dict)selfrr(rZ train_filer7rCZ test_fileZdev_fileannotation_fileZannotation_listrDline image_name image_annor)r)r*__init__fs>  ,, . , zFlickr8kDataSet.__init__cCs4|j|}tj|j|}t|}|j|}||fSN)rCrrrrr rD)rEitemrH image_pathr rIr)r)r* __getitem__s   zFlickr8kDataSet.__getitem__cCs t|jSrK)r"rC)rEr)r)r*__len__szFlickr8kDataSet.__len__N)r)r,r-r.r/rJrNrOr)r)r)r*r%ds 7r%)r/r collectionsrtypingrrrrmindspore.datasetrZtools.image_toolsr Ztools.registerr r registerDATASET_LOADERr r%r)r)r)r*s    Imulti_image_cap_dataloader.cpython-39.pyc000066400000000000000000000117461476650141400362560ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__a ?cf@sdZddlZddlmZddlmZmZmZmZddl Z ddl Z ddl m Z ddl mZddlmZmZeejGdd d ZGd d d ZdS) zMultiImageCapPairs DataLoader.N) OrderedDict)OptionalUnionListTuple)GeneratorDataset)Image)MindFormerRegisterMindFormerModuleTypec @s\eZdZdZddgZd eeeeeeeeee efeeee ee ddd Z dS) MultiImgCapDataLoaderz!Multiple Image-Caption DataloaderimagetextNtrainF) dataset_dirannotation_files image_dirs column_namesstage repeat_imagesshufflec Kst|t|krtdt|D]@\} } tj||| || <tj|| s t|| dq t|D]@\} } tj||| || <tj|| sjt|| dqj|dur|j}| ddt ||||} |dkrt | |fd|i|St | ||dS)a2 MultiImgCapDataLoader Dataloader API. Args: dataset_dir (str): The directory which is the parent dir of these datasets. annotation_files list[str]: the list of files contains annotations. image_dirs list[str]: the list of dir contains images. (one-to-one matching to ' annotation_files') column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. shuffle (Optional[bool]): whether to shuffle the dataset. Return: A GeneratorDataset for loading multiple image-caption datasets Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. z=the number of image_dirs should be equal to annotation_files!z is not existed.NNonerr)r) len ValueError enumerateospathjoinisfileisdir_default_column_namespopMultiImgCapDataSetr) clsrrrrrrrkwargsi_Zmulticap_datasetr&d/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/multi_image_cap_dataloader.py__new__!s@ zMultiImgCapDataLoader.__new__)NrFF) __name__ __module__ __qualname____doc__rstrrrrrboolr(r&r&r&r'r sr c@s2eZdZdZd ddZddZdd Zd d Zd S)r!aMultiImgCapDataSet API. Args: image_dirs (str): The directory which contains images. annotation_files list[str]: the list of files contains annotations. stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. Return: A Dataset for loading multiple image-caption datasets rFc CsDg|_|dvr8t|D]p\}}t|ddd}t|}Wdn1sN0Y|D]} tj||| d| d<q\|j|q|dkr@i|_ i|_ |rg}d} t|jD]r\} } g|j | <t| dD]R\}} |j |  | | |j | <|rt | }|d| i| || d 7} qq|r@||_ntd dS) N)revalrzutf-8)encodingr r/rcaptionzunsupported stage.) annotationropenjsonloadrrrextendZtxt2imgZimg2txtappendcopydeepcopyupdater)selfrrrrr$annotation_filefileZnew_annotationZnew_annZtxt_idZimg_idannr2tempr&r&r'__init__ns: (        zMultiImgCapDataSet.__init__cCs*|j|}t|dd}||dfS)Nr RGBr2)r4rr5convert)r=indexr@r r&r&r' __getitem__s zMultiImgCapDataSet.__getitem__cCs t|jS)N)rr4)r=r&r&r'__len__szMultiImgCapDataSet.__len__cCs0|||j|}}t|d|d|ddS)z~display item Args: index (int): index Returns: out (OrderedDict): item info r r2r)r?r2r )r4r)r=rEsampler@r&r&r' display_items zMultiImgCapDataSet.display_itemN)rF)r)r*r+r,rBrFrGrIr&r&r&r'r!`s %r!)r,r collectionsrtypingrrrrr6r:mindspore.datasetrPILrmindformers.tools.registerr r registerDATASET_LOADERr r!r&r&r&r's    Cpengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/squad_dataloader.cpython-39.pyc000066400000000000000000000336531476650141400343340ustar00rootroot00000000000000a ?cfG]@sdZddlZddlZddlZddlmZddlmZmZe ej GdddZ GdddZ d d Z d d Zd dZddZddZGdddZGdddZdS)zSQuAD DataLoaderN)GeneratorDataset)MindFormerRegisterMindFormerModuleTypec@s"eZdZdZgdZd dd ZdS) SQuADDataLoaderzSQuAD Dataloader) input_ids input_mask token_type_idZstart_positionsZ end_positions unique_idNtrain@c Kstj|st|d|dvr*td|dur8|j}t|ttfsXtdt |t |dkrvtdt ||D] } t| t sztdt | qz| d dt ||||||} t| |fi|S) a SQuAD Dataloader API. Args: dataset_dir: The directory to SQuAD dataset. tokenizer: a tokenizer for text processing. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 stage: The supported key words are in ["train", "dev"] max_question_len: The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len: Maximum sequence length. doc_stride: When splitting up a long document into chunks, how much stride to take between chunks. Return: A GeneratorDataset for SQuAD dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import SQuADDataLoader >>> from mindformers.models import BertTokenizer >>> bert_tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data_loader = SQuADDataLoader("./squad/", bert_tokenizer) >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[24, 384], dtype=Int32, value= [[ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], ... [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], ... [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], ... [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0]]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [1000000029, 1000000020, 1000000023, 1000000012, 1000000024, 1000000004, 1000000006, 1000000003, 1000000017, 1000000022, 1000000028, 1000000007, 1000000005, 1000000027, 1000000014, 1000000015, 1000000002, 1000000025, 1000000011, 1000000008, 1000000021, 1000000010, 1000000019, 1000000016])]  is not existed.)r devz stage should be in train or dev.NzJcolumn_names should be a tuple or a list of string with length 7, but got z0the length of column_names should be 6, but got z8the item type of column_names should be string, but got None)ospathisdir ValueError_default_column_names isinstancetuplelist TypeErrortypelenstrpop SQuADDatasetr) cls dataset_dir tokenizer column_namesstagemax_question_len max_seq_len doc_stridekwargsnameZ squad_datasetr*Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/squad_dataloader.py__new__s0A      zSQuADDataLoader.__new__)Nr r r r )__name__ __module__ __qualname____doc__rr,r*r*r*r+rs rc@sJeZdZdZdddZd d Zd d Zd dZddZddZ ddZ dS)rz SQuAD Datasetr r r r ./squad_tempc Cstj|st|d||_||_||_||_||_|dkrbd|_ tj |d}| ||_ n2|dkrd|_ tj |d} | | |_ ntdt|j |j|j|j|j|j |_|dkr||d S) a3 SQuAd Dataset Args: dataset_dir (str): The directory to SQuAd dataset. tokenizer (Tokenizer): A tokenizer for text processing. stage (str): The supported key words are in ["train", "dev"] max_question_len (int): The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len (int): Maximum sequence length. doc_stride (int): When splitting up a long document into chunks, how much stride to take between chunks. temp_file_dir (str): Save temporary files for SQuAD dataset. Return: A iterable dataset for SQuAd dataset Raises: ValueError: Error input for dataset_dir, stage. rr Tztrain-v1.1.jsonrFz dev-v1.1.jsonzunsupported stage.N)rrrrr!r"r%r&r' is_trainingjoin_get_train_examplesexamples_get_dev_examplesconvert_examples_to_featuresinput_features _save_eval_examples_and_features) selfr!r"r$r%r&r' temp_file_dirtrain_data_path dev_data_pathr*r*r+__init__}s,  zSQuADDataset.__init__c Cs>|j|}|j}|j}|j}|j}|j}|j}||||||fS)zReturn input data for model)r8rrrstart_position end_positionr ) r:itemfeaturerrrr?r@r r*r*r+ __getitem__s zSQuADDataset.__getitem__cCs t|jS)zGet the size of dataset)rr8)r:r*r*r+__len__szSQuADDataset.__len__cCs ||S)zGet train examples._read_squad_examples)r:r<r*r*r+r4sz SQuADDataset._get_train_examplescCs ||S)zGet dev examples.rE)r:r=r*r*r+r6szSQuADDataset._get_dev_examplescCstj|ddtj|d}tj|d}t|ddd2}|jD]}|t|j dq@Wd n1sp0Yt|ddd2}|j D]}|t|j dqWd n1s0Yd S) z)Save examples and features for evaluationT)exist_okztemp_examples.jsonztemp_features.jsonwzutf-8)encoding N) rmakedirsrr3openr5writejsondumps__dict__r8)r:r;temp_examples_filetemp_features_filefexamplerBr*r*r+r9s 6 z-SQuADDataset._save_eval_examples_and_featuresc Cst|d}t|d}Wdn1s.0Yg}|D]}|d}|dD]}|d}|dD]~} | d} | d } d} d} g}| d d }|s|jr| d d }|d} |d} n| d }t| | || | |||d}||qhqTq@|S)z3Read a SQuAD json file into a list of SquadExample.rdataNtitle paragraphscontextqasidquestion is_impossibleFanswersrtextZ answer_start)qas_id question_text context_text answer_textstart_position_characterrWr]r^)rLrNloadgetr2 SquadExampleappend)r:input_file_pathreader input_datar5entryrW paragraphrbqar`rardrcr^r]answerrTr*r*r+rFs@ ,      z!SQuADDataset._read_squad_examplesN)r r r r r1) r-r.r/r0r>rCrDr4r6r9rFr*r*r*r+r{s 2 rc#Csg}d}t|D]\}} || j} t| |kr>| d|} ggg} } } t| jD]>\}}| t| ||}|D]}| || |q|qXd\}}|r| jrd}d}|r| js| | j}| jt| jdkr| | jdd}n t| d}t || ||| j \}}|t| d}t || |}t|D]n\}}gg}}ii}}|d|d| D]}|||dqn|d|dt |j D]P}|j|}| ||t|<t|||}||t|<|| ||dq|d|d||}dgt|}t||krV|d|d|dq&t|||t| | j|\} }!t|||||||||| |!| jd }"||"|d7}q8q|S) zConvert examples to featuresiʚ;r)NNz[CLS]z[SEP] r example_indexdoc_span_indextokenstoken_to_orig_maptoken_is_max_contextrrrr?r@r]) enumeratetokenizerar doc_tokensrhr]r?r@_improve_answer_spanrc_get_doc_spansrangelengthstart_check_is_max_contextconvert_tokens_to_ids_get_positions InputFeatures)#r5r"r&r%r'r2r8r rtrTZ query_tokensZtok_to_orig_indexZorig_to_tok_indexall_doc_tokensitoken sub_tokensZ sub_tokentok_start_positiontok_end_positionmax_tokens_for_doc doc_spansrudoc_spanrvrrwrxZsplit_token_indexZis_max_contextrrr?r@rBr*r*r+r7s                          r7c Csd\}}|rl|sl|j}|j|jd} d} ||kr:|| ks>d} | rLd\}}n |d} ||| }||| }|r||r|d\}}||fS)z#Get start position and end position)rprprqFTrrrr) rrrZquery_tokens_lengthr]r2r?r@Z doc_startZdoc_endZ out_of_spanZ doc_offsetr*r*r+r`s"   rcCsttdddg}g}d}|t|krpt||}||kr<|}||||d||t|kr`qp|t||7}q|S)z Get doc spanZDocSpanrrrr) collections namedtuplerrhmin)r'rrZ_DocSpanr start_offsetrr*r*r+r}ys  r}c Cspd||}t||dD]H}t||ddD]2}d|||d}||kr2||fSq2q||fS)zFReturns tokenized answer spans that better match the annotated answer. rqrp)r3rzr~) r"r{Z input_startZ input_endZorig_answer_textZtok_answer_text new_startnew_endZ text_spanr*r*r+r|sr|c Csd}d}t|D]l\}}|j|jd}||jkr4q||kr>q||j}||} t|| d|j} |dust| |kr| }|}q||kS)z:Check if this is the 'max context' doc span for the token.Nrqg{Gz?)ryrrr) rZcur_span_indexposition best_scoreZbest_span_indexZ span_indexrendZnum_left_contextZnum_right_contextscorer*r*r+rs  rc@s"eZdZdZdddZddZdS) rgaS A single training/test example for the Squad dataset, as loaded from disk. Args: qas_id: The example's unique identifier question_text: The question string context_text: The context string answer_text: The answer string start_position_character: The character position of the start of the answer title: The title of the example answers: None by default, this is used during evaluation. Holds answers as well as their start positions. is_impossible: False by default, set to True if the example has no possible answer. NFc Cs||_||_||_||_||_||_||_d\|_|_g} g} d} |jD]J} | | r\d} n$| rl| | n| d| 7<d} | t | dqH| |_ | |_ |dur|s| ||_| t|t |dt | d|_dS)NrTrpFrq)r`rarbrcrWr]r^r?r@_is_whitespacerhrr{char_to_word_offsetr) r:r`rarbrcrdrWr^r]r{rZprev_is_whitespacecr*r*r+r>s4       zSquadExample.__init__cCs4|dks,|dks,|dks,|dks,t|dkr0dSdS)z%Check whether character is whitespacer  rJi/ TF)ord)r:rr*r*r+r s,zSquadExample._is_whitespace)NF)r-r.r/r0r>rr*r*r*r+rgs  .rgc@seZdZdZdddZdS)rz!A single set of features of data.rpFc CsL||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ dS)Nrs) r:r rtrurvrwrxrrrr?r@r]r*r*r+r>s zInputFeatures.__init__N)rprpF)r-r.r/r0r>r*r*r*r+rs  r)r0rrNrmindspore.datasetrmindformers.tools.registerrrregisterDATASET_LOADERrrr7rr}r|rrgrr*r*r*r+s   ab#%Bpengcheng.mind-mindformer/mindformers/dataset/dataloader/__pycache__/wmt16_dataloader.cpython-39.pyc000066400000000000000000000072421476650141400341700ustar00rootroot00000000000000a ?cf@shdZddlZddlmZddlmZddlmZmZe ej GdddZ d d Z Gd d d Z dS) zWMT16 DataLoader.N)GeneratorDataset)logger)MindFormerRegisterMindFormerModuleTypec@s"eZdZdZddgZd dd ZdS) WMT16DataLoaderzWMT16 DataloadersourcetargetNtrainrcCstj|st|d|dur6|j}td|j|rZt|tt fsZt dt |t |dkrxtdt |t|dt rt|dt std t |dd t |dt||}t||||d S) uZ WMT16 Dataloader API Args: dataset_dir: the directory to dataset column_names: the output column names, a tuple or a list of string with length 2 stage: the supported `option` are in ["train"、"test"、"del"、"all"] Return: a GeneratorDataset for WMT16 dataset  is not existed.NzZThe column_names to the WMT16DataLoader is None, so assign it with default_column_names %szJcolumn_names should be a tuple or a list of string with length 2, but got z0the length of column_names should be 2, but got rr z8the item type of column_names should be string, but got z and ) num_shardsshard_id)ospathisdir ValueError_default_column_namesrinfo isinstancetuplelist TypeErrortypelenstr WMT16DataSetr)cls dataset_dir column_namesstagerrdatasetr#Z/home/ma-user/work/mindformers-baichuan/mindformers/dataset/dataloader/wmt16_dataloader.py__new__s.     zWMT16DataLoader.__new__)Nr r r)__name__ __module__ __qualname____doc__rr%r#r#r#r$rsrcCsPt|4}g}|D]}|}|r||qWdn1sB0Y|S)z&Read the text files and return a list.N)openstripappend) train_filefpdataliner#r#r$ read_textAs *r1c@s*eZdZdZd ddZddZddZd S) rz WMT16 DataSetr csztjstd|_||_t}fdd}tdddkrZ||||<ndD]}||||<q^||_ dS) u WMT16DataSet Dataset Args: dataset_dir: the directory to wmt16 dataset stage: the supported key word are in ["train"、"test"、"del"、"all"] Return: an iterable dataset for wmt16 dataset r csJtj|d}t|}tj|d}t|}tt||}|S)Nz.sourcez.target)rrjoinr1rzip) cur_stage source_pathsrcZ target_sourcetgtZ src_tgt_pairrr#r$read_and_add_to_stageas z4WMT16DataSet.__init__..read_and_add_to_stagez,Start to read the raw data from the disk %s.r!all)r devtestN) rrrrrr!dictrr dataset_dict)selfrr!r>r9itemr#r8r$__init__Os   zWMT16DataSet.__init__cCs|j|j|SN)r>r!)r?r@r#r#r$ __getitem__rszWMT16DataSet.__getitem__cCst|j|jSrB)rr>r!)r?r#r#r$__len__uszWMT16DataSet.__len__N)r )r&r'r(r)rArCrDr#r#r#r$rLs #r)r)rmindspore.datasetrmindformers.toolsrZtools.registerrrregisterDATASET_LOADERrr1rr#r#r#r$s   ( pengcheng.mind-mindformer/mindformers/dataset/dataloader/adgen_dataloader.py000066400000000000000000000115401476650141400277750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ADGen DataLoader""" import json import os from mindspore.dataset import GeneratorDataset from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerModuleType, MindFormerRegister @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class ADGenDataLoader: """ADGen Dataloader""" def __new__(cls, dataset_dir, phase, shuffle, origin_columns, **kwargs): r""" ADGen Dataloader API. Args: dataset_dir: The directory to ADGen dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 phase: The supported key words are in ["train", "dev"] Return: A GeneratorDataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import ADGenDataLoader >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> data_loader = ADGenDataLoader("./ADGen/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break """ if not os.path.isfile(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if phase not in ["train", "eval"]: raise ValueError(f"phase should be in train or eval.") column_names = ["prompt", "answer"] # verify dataset column names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}, length {len(column_names)}") for name in column_names: if not isinstance(name, str): raise ValueError(f"the item type of column_names should be string," f" but got {type(name)}") # verify origin column names if not isinstance(origin_columns, (tuple, list)) or len(origin_columns) != 2: raise TypeError(f"origin_columns should be a tuple or a list" f" of string with length 2, but got {type(origin_columns)}, length {len(origin_columns)}") kwargs.pop("None", None) adgen_dataset = ADGenDataset(dataset_dir, origin_columns, phase) info = f"[DATASET] shuffle status is {shuffle}, phase is {phase}." logger.info(info) kwargs.pop("version", None) return GeneratorDataset(adgen_dataset, column_names, shuffle=shuffle, **kwargs) class ADGenDataset: """ADGen Dataset""" def __init__(self, dataset_dir, origin_columns, phase="train"): r""" ADGen Dataset Args: dataset_dir (str): The directory to ADGen dataset. phase (str): The supported key words are in ["train", "dev"] Return: A iterable dataset for ADGen dataset Raises: ValueError: Error input for dataset_dir, phase. """ if not os.path.isfile(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.phase = phase examples = {} content_list = [] summary_list = [] self.prompt_column = origin_columns[0] self.response_column = origin_columns[1] with open(self.dataset_dir) as fp: for line in fp: content_list.append(json.loads(line)[self.prompt_column]) summary_list.append(json.loads(line)[self.response_column]) examples[self.prompt_column] = content_list examples[self.response_column] = summary_list self.examples = examples def __len__(self): """Get the size of dataset""" return len(self.examples[self.prompt_column]) def __getitem__(self, i): """Return input data for model""" prompt, answer = self.examples[self.prompt_column][i], self.examples[self.response_column][i] return prompt, answer pengcheng.mind-mindformer/mindformers/dataset/dataloader/build_dataloader.py000066400000000000000000000055621476650141400300250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build DataLoader API.""" import inspect from mindspore import dataset as ds from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_dataset_loader( config: dict = None, default_args: dict = None, module_type: str = 'dataset_loader', class_name: str = None, **kwargs): r"""Build dataset loader For MindFormer. Instantiate the dataset loader from MindFormerRegister's registry. Args: config (dict): The task dataset loader's config. Default: None. default_args (dict): The default argument of dataset loader API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'dataset loader'. class_name (str): The class name of dataset loader API. Default: None. Return: The function instance of dataset loader API. Examples: >>> from mindformers.dataset.dataloader import build_dataset_loader >>> dataset_loader_config = {'type': 'ImageFolderDataset', 'dataset_dir': './imagenet/train'} >>> # 1) use config dict to build dataset loader >>> dataset_loader_from_config = build_dataset_loader(dataset_loader_config) >>> # 2) use class name to build dataset loader >>> dataset_loader_class_name = build_dataset_loader( ... class_name='ImageFolderDataset', dataset_dir='./imagenet/train') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET_LOADER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_dataset_loader(): """ register MindSpore builtin dataset loader class. """ for module_name in dir(ds): if module_name.startswith('__'): continue dataset = getattr(ds, module_name) if inspect.isclass(dataset): MindFormerRegister.register_cls( dataset, MindFormerModuleType.DATASET_LOADER) register_ms_dataset_loader() pengcheng.mind-mindformer/mindformers/dataset/dataloader/cifar100_dataloader.py000066400000000000000000000173251476650141400302330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cifar100 DataLoader.""" import os import pickle from typing import Optional, Union, List, Tuple import numpy as np from mindspore.dataset import GeneratorDataset from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class Cifar100DataLoader: """Cifar100 Dataloader with class name as text column""" _default_column_names = ["image", "text", "label"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", fine_label: Optional[bool] = True, shuffle: Optional[bool] = False, hypothesis_template: Optional[str] = "This is a photo of {}.", **kwargs): r""" Cifar100 Dataloader API. Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python" column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 3. stage (Optional[str]): The supported key words are in ["train","test", "all"] fine_label (Optional[bool]): True for 100 classes, False for 10 classes. shuffle (Optional[bool]): Shuffle the samples. hypothesis_template (Optional[str]): Prompt template for class label. Return: A GeneratorDataset for Cifar100 dataset Raises: ValueError: Error input for dataset_dir and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Cifar100DataLoader >>> dataloader = Cifar100DataLoader("./cifar-100-python") >>> dataloader = dataloader.batch(1) >>> for item in dataloader: >>> print(item) >>> break [Tensor(shape=[1, 32, 32, 3], dtype=UInt8, value= [[[[255, 255, 255], [255, 255, 255], [255, 255, 255], ... [195, 205, 193], [212, 224, 204], [182, 194, 167]]]]]), Tensor(shape=[1, 100], dtype=String, value= [['This is a photo of apple.', 'This is a photo of aquarium_fish.', 'This is a photo of baby.', 'This is a photo of bear.', ... 'This is a photo of woman.', 'This is a photo of worm.']]), Tensor(shape=[1], dtype=Int32, value= [19])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 3, but got {type(column_names)}") if len(column_names) != 3: raise ValueError(f"the length of column_names should be 3," f" but got {len(column_names)}") for index in range(len(column_names)): if not isinstance(column_names[index], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[index])}") kwargs.pop("None", None) cifar100_dataset = Cifar100DataSet(dataset_dir, stage, fine_label, hypothesis_template) cifar100_dataloader = GeneratorDataset(cifar100_dataset, column_names, shuffle=shuffle) setattr(cifar100_dataloader, "label_names", cifar100_dataset.label_names) return cifar100_dataloader class Cifar100DataSet: """Cifar100 dataSet with class name""" def __init__(self, dataset_dir: str, stage: Optional[str] = "train", fine_label: Optional[bool] = True, hypothesis_template: Optional[str] = "This is a photo of {}."): r""" Cifar100 Dataset Args: dataset_dir (str): The dataset directory, such as "/home/desktop/cifar-100-python". stage (Optional[str]): The supported key words are in ["train", "test", "all"]. fine_label (Optional[bool]): True for 100 classes, False for 10 classes. hypothesis_template (Optional[str]): Prompt template for class label. Return: A iterable dataset for Cifar100 dataset. Raises: ValueError: Unsupported values for stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if stage not in ["train", "test", "all"]: raise ValueError("unsupported stage," " stage should be in [\"train\", \"test\", \"all\"]") meta_file = os.path.join(dataset_dir, "meta") with open(meta_file, 'rb') as fo: meta_dict = pickle.load(fo, encoding="bytes") if fine_label: label_names = meta_dict[b'fine_label_names'] else: label_names = meta_dict[b'coarse_label_names'] for index in range(len(label_names)): label_names[index] = label_names[index].decode() self.label_names = label_names hypothesis = [] for index in range(len(label_names)): hypothesis.append(hypothesis_template.format(label_names[index])) if stage in ["train", "all"]: train_file = os.path.join(dataset_dir, "train") with open(train_file, 'rb') as fo: train_dict = pickle.load(fo, encoding="bytes") if fine_label: train_label = train_dict[b"fine_labels"] else: train_label = train_dict[b"coarse_labels"] train_image = train_dict[b"data"] train_num = train_image.shape[0] train_image = train_image.reshape((train_num, 3, 32, 32)) train_image = train_image.transpose(0, 2, 3, 1) self.image = train_image self.label = train_label self.text = hypothesis if stage in ["test", "all"]: test_file = os.path.join(dataset_dir, "test") with open(test_file, 'rb') as fo: test_dict = pickle.load(fo, encoding="bytes") if fine_label: test_label = test_dict[b"fine_labels"] else: test_label = test_dict[b"coarse_labels"] test_image = test_dict[b"data"] test_num = test_image.shape[0] test_image = test_image.reshape((test_num, 3, 32, 32)) test_image = test_image.transpose(0, 2, 3, 1) self.image = test_image self.label = test_label self.text = hypothesis if stage == "all": self.image = np.row_stack([train_image, test_image]) self.label = train_label + test_label self.text = hypothesis def __getitem__(self, item): """getitem""" return self.image[item], self.text, self.label[item] def __len__(self): """len""" return self.image.shape[0] pengcheng.mind-mindformer/mindformers/dataset/dataloader/cluener_dataloader.py000066400000000000000000000154611476650141400303620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://gitee.com/mindspore/models/blob/master/official/nlp/Bert/src/generate_mindrecord/generate_cluener_mindrecord.py # ============================================================================ """CLUENER DataLoader.""" import os import json from typing import Optional, Union, List, Tuple from mindspore.dataset import GeneratorDataset from ...tools.register import MindFormerRegister, MindFormerModuleType from ..labels import cluener_labels @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class CLUENERDataLoader: """CLUENER Dataloader""" _default_column_names = ["text", "label_id"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", **kwargs): r""" CLUENER Dataloader API. Args: dataset_dir (str): The directory to cluener dataset. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for cluener dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import CLUENERDataLoader >>> data_loader = CLUENERDataLoader("./cluener/") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[], dtype=String, value= '虚幻引擎3动作游戏《黑光》新作公布'), Tensor(shape=[49], dtype=Int64, value= [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 15, 15, 0, 0, 0, 0, 0, 5, 15, 15, 15, 15, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") kwargs.pop("None", None) cluener_dataset = CLUENERDataSet(dataset_dir, stage) return GeneratorDataset(cluener_dataset, column_names) class CLUENERDataSet: """CLUENER DataSet""" def __init__(self, dataset_dir, stage="train"): r""" CLUENER Dataset Args: dataset_dir (str): The directory to cluener dataset. stage (str): The supported key words are in ["train", "dev", "test"] Return: A iterable dataset for cluener dataset Raises: ValueError: Error input for dataset_dir, stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.label2id = {label: label_id for label_id, label in enumerate(cluener_labels)} self.texts = [] self.label_ids = [] if stage == "train": train_data_path = os.path.join(dataset_dir, "train.json") self._get_train_examples(train_data_path) elif stage == "dev": dev_data_path = os.path.join(dataset_dir, "dev.json") self._get_dev_examples(dev_data_path) elif stage == "test": test_data_path = os.path.join(dataset_dir, "test.json") self._get_test_examples(test_data_path) else: raise ValueError("unsupported stage.") def __getitem__(self, item): text = self.texts[item] label_id = self.label_ids[item] return text, label_id def __len__(self): return len(self.texts) def _get_train_examples(self, train_data_path): """Get train examples.""" return self._create_examples(self._read_json(train_data_path)) def _get_dev_examples(self, dev_data_path): """Get dev examples.""" return self._create_examples(self._read_json(dev_data_path)) def _get_test_examples(self, test_data_path): """Get test examples.""" return self._create_examples(self._read_json(test_data_path)) def _read_json(self, input_file): """Reads a tab separated value file.""" with open(input_file, "r", encoding="utf-8") as f: lines = [] for line in f.readlines(): lines.append(json.loads(line.strip())) return lines def generate_label(self, line, label): """Generate label""" for l, words in line['label'].items(): for _, indices in words.items(): for index in indices: if index[0] == index[1]: label[index[0]] = 'S-' + l else: label[index[0]] = 'B-' + l label[index[1]] = 'I-' + l for j in range(index[0] + 1, index[1]): label[j] = 'I-' + l return label def _create_examples(self, lines): """Create Example.""" for line in lines: text = line['text'] label = ['O'] * len(text) if 'label' in line: label = self.generate_label(line, label) label_id = [] for token_label in label: label_id.append(self.label2id[token_label]) self.texts.append(text) self.label_ids.append(label_id) pengcheng.mind-mindformer/mindformers/dataset/dataloader/flickr8k_dataloader.py000066400000000000000000000156731476650141400304470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Flickr8k DataLoader.""" import os from collections import defaultdict from typing import Optional, Union, List, Tuple from mindspore.dataset import GeneratorDataset from ...tools.image_tools import load_image from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class Flickr8kDataLoader: """Flicker8k Dataloader""" _default_column_names = ["image", "text"] def __new__(cls, dataset_dir: str, column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", **kwargs): r""" Flicker8k Dataloader API. Args: dataset_dir (str): The directory. Flickr_8k.devImages.txt, and Flickr8k.token.txt column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "test", "del", "all"] Return: A GeneratorDataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import Flickr8kDataLoader >>> data_loader = Flickr8kDataLoader("./Flickr8k") >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[1, 378, 500, 3], dtype=UInt8, value= [[[[ 89, 46, 55], [ 89, 62, 55], [ 83, 59, 49], ... [161, 152, 135], [162, 156, 142], [159, 162, 145]]]]), Tensor(shape=[1, 5], dtype=String, value= [['A woman and child sitting on a bench beside a wooden bear carving .', 'A woman and little girl are sitting on a wooden bench beside a wooden bear .', 'A woman holds a little girl next to a large wooden carving of a bear .', 'A woman with a young girl pose with bear statues in front of a store .', 'A young girl being held by an older woman wearing bluejeans sitting on the lap of a bear statue near the entrance to a wooden building .']])] """ annotation_dir = os.path.join(dataset_dir, "Flickr8k_text") dataset_dir = os.path.join(dataset_dir, "Flickr8k_Dataset", "Flickr8k_Dataset") if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if not os.path.isdir(annotation_dir): raise ValueError(f"{annotation_dir} is not existed.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") kwargs.pop("None", None) flick8k_dataset = Flickr8kDataSet(dataset_dir, annotation_dir, stage) return GeneratorDataset(flick8k_dataset, column_names, **kwargs) class Flickr8kDataSet: """Flickr8k DataSet""" def __init__(self, dataset_dir, annotation_dir, stage="train"): r""" Flickr8k Dataset Args: dataset_dir (str): The directory to images annotation_dir (str): The directory to Flickr_8k.trainImages.txt, Flickr_8k.testImages.txt, Flickr_8k.devImages.txt, and Flickr8k.token.txt stage (str): The supported key words are in ["train", "test", "dev", "all"] Return: A iterable dataset for Flickr8k dataset Raises: ValueError: Error input for dataset_dir, annotation_dir, and stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if not os.path.isdir(annotation_dir): raise ValueError(f"{annotation_dir} is not existed.") self.dataset_dir = dataset_dir if stage == "train": train_file = os.path.join(annotation_dir, "Flickr_8k.trainImages.txt") with open(train_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "test": test_file = os.path.join(annotation_dir, "Flickr_8k.testImages.txt") with open(test_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "dev": dev_file = os.path.join(annotation_dir, "Flickr_8k.devImages.txt") with open(dev_file, 'r', encoding='utf-8') as file: image_names = file.read().splitlines() elif stage == "all": image_names = [file for file in os.listdir(dataset_dir) if file.endswith(".jpg")] else: raise ValueError("unsupported stage.") annotation_file = os.path.join(annotation_dir, "Flickr8k.token.txt") with open(annotation_file, 'r', encoding='utf-8') as file: annotation_list = file.read().splitlines() dataset_dict = defaultdict(list) for line in annotation_list: image_name = line.split("#")[0] if image_name in image_names: image_anno = line.split("\t")[-1] dataset_dict[image_name].append(image_anno) self.image_names = image_names self.dataset_dict = dataset_dict def __getitem__(self, item): image_name = self.image_names[item] image_path = os.path.join(self.dataset_dir, image_name) image = load_image(image_path) image_anno = self.dataset_dict[image_name] return image, image_anno def __len__(self): return len(self.image_names) pengcheng.mind-mindformer/mindformers/dataset/dataloader/multi_image_cap_dataloader.py000066400000000000000000000150211476650141400320340ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MultiImageCapPairs DataLoader.""" import os from collections import OrderedDict from typing import Optional, Union, List, Tuple import json import copy from mindspore.dataset import GeneratorDataset from PIL import Image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class MultiImgCapDataLoader: """Multiple Image-Caption Dataloader""" _default_column_names = ["image", "text"] def __new__(cls, dataset_dir: str, annotation_files: List[str], image_dirs: List[str], column_names: Optional[Union[List[str], Tuple[str]]] = None, stage: Optional[str] = "train", repeat_images: Optional[bool] = False, shuffle: Optional[bool] = False, **kwargs): r""" MultiImgCapDataLoader Dataloader API. Args: dataset_dir (str): The directory which is the parent dir of these datasets. annotation_files list[str]: the list of files contains annotations. image_dirs list[str]: the list of dir contains images. (one-to-one matching to ' annotation_files') column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 2 stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. shuffle (Optional[bool]): whether to shuffle the dataset. Return: A GeneratorDataset for loading multiple image-caption datasets Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. """ if len(image_dirs) != len(annotation_files): raise ValueError( "the number of image_dirs should be equal to annotation_files!" ) for i, _ in enumerate(annotation_files): annotation_files[i] = os.path.join(dataset_dir, annotation_files[i]) if not os.path.isfile(annotation_files[i]): raise ValueError(f"{annotation_files[i]} is not existed.") for i, _ in enumerate(image_dirs): image_dirs[i] = os.path.join(dataset_dir, image_dirs[i]) if not os.path.isdir(image_dirs[i]): raise ValueError(f"{image_dirs[i]} is not existed.") if column_names is None: column_names = cls._default_column_names kwargs.pop("None", None) multicap_dataset = MultiImgCapDataSet(image_dirs, annotation_files, stage, repeat_images) if stage == 'train': return GeneratorDataset(multicap_dataset, column_names, shuffle=shuffle, **kwargs) return GeneratorDataset(multicap_dataset, column_names, shuffle=shuffle) class MultiImgCapDataSet: """MultiImgCapDataSet API. Args: image_dirs (str): The directory which contains images. annotation_files list[str]: the list of files contains annotations. stage (Optional[str]): The supported key words are in ["train", "eval"] repeat_images (Optional[bool]): whether repeat image when it has multiple corresponding captions. Return: A Dataset for loading multiple image-caption datasets """ def __init__(self, image_dirs, annotation_files, stage="train", repeat_images=False): self.annotation = [] if stage in ("train", "eval"): for i, annotation_file in enumerate(annotation_files): with open(annotation_file, 'r', encoding='utf-8') as file: new_annotation = json.load(file) for new_ann in new_annotation: new_ann["image"] = os.path.join(image_dirs[i], new_ann["image"]) self.annotation.extend(new_annotation) if stage == "eval": self.txt2img = {} self.img2txt = {} if repeat_images: new_annotation = [] txt_id = 0 for img_id, ann in enumerate(self.annotation): self.img2txt[img_id] = [] for i, caption in enumerate(ann["caption"]): self.img2txt[img_id].append(txt_id) self.txt2img[txt_id] = img_id if repeat_images: temp = copy.deepcopy(ann) temp.update({"caption": caption}) new_annotation.append(temp) txt_id += 1 if repeat_images: self.annotation = new_annotation else: raise ValueError("unsupported stage.") def __getitem__(self, index): ann = self.annotation[index] image = Image.open(ann["image"]).convert("RGB") return image, ann["caption"] def __len__(self): return len(self.annotation) def display_item(self, index): """display item Args: index (int): index Returns: out (OrderedDict): item info """ sample, ann = self[index], self.annotation[index] return OrderedDict({ "file": ann["image"], "caption": ann["caption"], "image": sample[0], }) pengcheng.mind-mindformer/mindformers/dataset/dataloader/squad_dataloader.py000066400000000000000000000565071476650141400300500ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SQuAD DataLoader""" import os import json import collections from mindspore.dataset import GeneratorDataset from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class SQuADDataLoader: """SQuAD Dataloader""" _default_column_names = ["input_ids", "input_mask", "token_type_id", "start_positions", "end_positions", "unique_id"] def __new__(cls, dataset_dir, tokenizer, column_names=None, stage="train", max_question_len=64, max_seq_len=384, doc_stride=128, **kwargs): r""" SQuAD Dataloader API. Args: dataset_dir: The directory to SQuAD dataset. tokenizer: a tokenizer for text processing. column_names (Optional[Union[List[str], Tuple[str]]]): The output column names, a tuple or a list of string with length 6 stage: The supported key words are in ["train", "dev"] max_question_len: The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len: Maximum sequence length. doc_stride: When splitting up a long document into chunks, how much stride to take between chunks. Return: A GeneratorDataset for SQuAD dataset Raises: ValueError: Error input for dataset_dir, and column_names. TypeError: Type error for column_names. Examples: >>> from mindformers import SQuADDataLoader >>> from mindformers.models import BertTokenizer >>> bert_tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data_loader = SQuADDataLoader("./squad/", bert_tokenizer) >>> data_loader = data_loader.batch(1) >>> for item in data_loader: >>> print(item) >>> break [Tensor(shape=[24, 384], dtype=Int32, value= [[ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], ... [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0], [ 101, 2054, 3589 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], ... [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0], [1, 1, 1 ... 0, 0, 0]]), Tensor(shape=[24, 384], dtype=Int32, value= [[0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], ... [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0], [0, 0, 0 ... 0, 0, 0]]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24, 24 ]), Tensor(shape=[24], dtype=Int32, value= [1000000029, 1000000020, 1000000023, 1000000012, 1000000024, 1000000004, 1000000006, 1000000003, 1000000017, 1000000022, 1000000028, 1000000007, 1000000005, 1000000027, 1000000014, 1000000015, 1000000002, 1000000025, 1000000011, 1000000008, 1000000021, 1000000010, 1000000019, 1000000016])] """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if stage not in ["train", "dev"]: raise ValueError(f"stage should be in train or dev.") if column_names is None: column_names = cls._default_column_names if not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 7, but got {type(column_names)}") if len(column_names) != 6: raise ValueError(f"the length of column_names should be 6," f" but got {len(column_names)}") for name in column_names: if not isinstance(name, str): raise ValueError(f"the item type of column_names should be string," f" but got {type(name)}") kwargs.pop("None", None) squad_dataset = SQuADDataset(dataset_dir, tokenizer, stage, max_question_len, max_seq_len, doc_stride) return GeneratorDataset(squad_dataset, column_names, **kwargs) class SQuADDataset: """SQuAD Dataset""" def __init__(self, dataset_dir, tokenizer, stage="train", max_question_len=64, max_seq_len=384, doc_stride=128, temp_file_dir="./squad_temp"): r""" SQuAd Dataset Args: dataset_dir (str): The directory to SQuAd dataset. tokenizer (Tokenizer): A tokenizer for text processing. stage (str): The supported key words are in ["train", "dev"] max_question_len (int): The maximum number of tokens for the question, Questions longer than this will be truncated to this length. max_seq_len (int): Maximum sequence length. doc_stride (int): When splitting up a long document into chunks, how much stride to take between chunks. temp_file_dir (str): Save temporary files for SQuAD dataset. Return: A iterable dataset for SQuAd dataset Raises: ValueError: Error input for dataset_dir, stage. """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.tokenizer = tokenizer self.max_question_len = max_question_len self.max_seq_len = max_seq_len self.doc_stride = doc_stride if stage == "train": self.is_training = True train_data_path = os.path.join(dataset_dir, "train-v1.1.json") self.examples = self._get_train_examples(train_data_path) elif stage == "dev": self.is_training = False dev_data_path = os.path.join(dataset_dir, "dev-v1.1.json") self.examples = self._get_dev_examples(dev_data_path) else: raise ValueError("unsupported stage.") self.input_features = convert_examples_to_features(self.examples, self.tokenizer, self.max_seq_len, self.max_question_len, self.doc_stride, self.is_training) if stage == "dev": self._save_eval_examples_and_features(temp_file_dir) def __getitem__(self, item): """Return input data for model""" feature = self.input_features[item] input_ids = feature.input_ids input_mask = feature.input_mask token_type_id = feature.token_type_id start_position = feature.start_position end_position = feature.end_position unique_id = feature.unique_id return input_ids, input_mask, token_type_id, start_position, end_position, unique_id def __len__(self): """Get the size of dataset""" return len(self.input_features) def _get_train_examples(self, train_data_path): """Get train examples.""" return self._read_squad_examples(train_data_path) def _get_dev_examples(self, dev_data_path): """Get dev examples.""" return self._read_squad_examples(dev_data_path) def _save_eval_examples_and_features(self, temp_file_dir): """Save examples and features for evaluation""" os.makedirs(temp_file_dir, exist_ok=True) temp_examples_file = os.path.join(temp_file_dir, "temp_examples.json") temp_features_file = os.path.join(temp_file_dir, "temp_features.json") with open(temp_examples_file, 'w', encoding='utf-8') as f: for example in self.examples: f.write(json.dumps(example.__dict__) + '\n') with open(temp_features_file, 'w', encoding='utf-8') as f: for feature in self.input_features: f.write(json.dumps(feature.__dict__) + '\n') def _read_squad_examples(self, input_file_path): """Read a SQuAD json file into a list of SquadExample.""" with open(input_file_path, "r") as reader: input_data = json.load(reader)["data"] examples = [] for entry in input_data: title = entry["title"] for paragraph in entry["paragraphs"]: context_text = paragraph["context"] for qa in paragraph["qas"]: qas_id = qa["id"] question_text = qa["question"] start_position_character = None answer_text = None answers = [] is_impossible = qa.get("is_impossible", False) if not is_impossible: if self.is_training: answer = qa["answers"][0] answer_text = answer["text"] start_position_character = answer["answer_start"] else: answers = qa["answers"] example = SquadExample( qas_id=qas_id, question_text=question_text, context_text=context_text, answer_text=answer_text, start_position_character=start_position_character, title=title, is_impossible=is_impossible, answers=answers, ) examples.append(example) return examples def convert_examples_to_features(examples, tokenizer, max_seq_len, max_question_len, doc_stride, is_training): """Convert examples to features""" input_features = [] unique_id = 1000000000 for (example_index, example) in enumerate(examples): query_tokens = tokenizer.tokenize(example.question_text) if len(query_tokens) > max_question_len: query_tokens = query_tokens[0: max_question_len] tok_to_orig_index, orig_to_tok_index, all_doc_tokens = [], [], [] for (i, token) in enumerate(example.doc_tokens): orig_to_tok_index.append(len(all_doc_tokens)) sub_tokens = tokenizer.tokenize(token) for sub_token in sub_tokens: tok_to_orig_index.append(i) all_doc_tokens.append(sub_token) tok_start_position, tok_end_position = None, None if is_training and example.is_impossible: tok_start_position = -1 tok_end_position = -1 if is_training and not example.is_impossible: tok_start_position = orig_to_tok_index[example.start_position] if example.end_position < len(example.doc_tokens) - 1: tok_end_position = orig_to_tok_index[example.end_position + 1] - 1 else: tok_end_position = len(all_doc_tokens) - 1 (tok_start_position, tok_end_position) = _improve_answer_span(tokenizer, all_doc_tokens, tok_start_position, tok_end_position, example.answer_text) # The -3 accounts for [CLS], [SEP] and [SEP] max_tokens_for_doc = max_seq_len - len(query_tokens) - 3 # We can have documents that are longer than the maximum sequence length. # To deal with this we do a sliding window approach, where we take chunks # of the up to our max length with a stride of `doc_stride`. doc_spans = _get_doc_spans(doc_stride, all_doc_tokens, max_tokens_for_doc) for (doc_span_index, doc_span) in enumerate(doc_spans): tokens, token_type_id = [], [] token_to_orig_map, token_is_max_context = {}, {} tokens.append("[CLS]") token_type_id.append(0) for token in query_tokens: tokens.append(token) token_type_id.append(0) tokens.append("[SEP]") token_type_id.append(0) for i in range(doc_span.length): split_token_index = doc_span.start + i token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index] is_max_context = _check_is_max_context(doc_spans, doc_span_index, split_token_index) token_is_max_context[len(tokens)] = is_max_context tokens.append(all_doc_tokens[split_token_index]) token_type_id.append(1) tokens.append("[SEP]") token_type_id.append(1) input_ids = tokenizer.convert_tokens_to_ids(tokens) # The mask has 1 for real tokens and 0 for padding tokens. Only real # tokens are attended to. input_mask = [1] * len(input_ids) # Zero-pad up to the sequence length. while len(input_ids) < max_seq_len: input_ids.append(0) input_mask.append(0) token_type_id.append(0) start_position, end_position = _get_positions(doc_span, tok_start_position, tok_end_position, len(query_tokens), example.is_impossible, is_training) feature = InputFeatures( unique_id=unique_id, example_index=example_index, doc_span_index=doc_span_index, tokens=tokens, token_to_orig_map=token_to_orig_map, token_is_max_context=token_is_max_context, input_ids=input_ids, input_mask=input_mask, token_type_id=token_type_id, start_position=start_position, end_position=end_position, is_impossible=example.is_impossible) input_features.append(feature) unique_id += 1 return input_features def _get_positions(doc_span, tok_start_position, tok_end_position, query_tokens_length, is_impossible, is_training): """Get start position and end position""" start_position, end_position = -1, -1 if is_training and not is_impossible: # For training, if our document chunk does not contain an annotation # we throw it out, since there is nothing to predict. doc_start = doc_span.start doc_end = doc_span.start + doc_span.length - 1 out_of_span = False if not (tok_start_position >= doc_start and tok_end_position <= doc_end): out_of_span = True if out_of_span: start_position, end_position = 0, 0 else: doc_offset = query_tokens_length + 2 start_position = tok_start_position - doc_start + doc_offset end_position = tok_end_position - doc_start + doc_offset if is_training and is_impossible: start_position, end_position = 0, 0 return start_position, end_position def _get_doc_spans(doc_stride, all_doc_tokens, max_tokens_for_doc): """Get doc span""" _DocSpan = collections.namedtuple("DocSpan", ["start", "length"]) doc_spans = [] start_offset = 0 while start_offset < len(all_doc_tokens): length = len(all_doc_tokens) - start_offset if length > max_tokens_for_doc: length = max_tokens_for_doc doc_spans.append(_DocSpan(start=start_offset, length=length)) if start_offset + length == len(all_doc_tokens): break start_offset += min(length, doc_stride) return doc_spans def _improve_answer_span(tokenizer, doc_tokens, input_start, input_end, orig_answer_text): """Returns tokenized answer spans that better match the annotated answer.""" # The SQuAD annotations are character based. We first project them to # whitespace-tokenized words. But then after WordPiece tokenization, we can # often find a "better match". For example: # # Question: What year was John Smith born? # Context: The leader was John Smith (1895-1943). # Answer: 1895 # # The original whitespace-tokenized answer will be "(1895-1943).". However # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match # the exact answer, 1895. # # However, this is not always possible. Consider the following: # # Question: What country is the top exporter of electornics? # Context: The Japanese electronics industry is the lagest in the world. # Answer: Japan # # In this case, the annotator chose "Japan" as a character sub-span of # the word "Japanese". Since our WordPiece tokenizer does not split # "Japanese", we just use "Japanese" as the annotation. This is fairly rare # in SQuAD, but does happen. tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text)) for new_start in range(input_start, input_end + 1): for new_end in range(input_end, new_start - 1, -1): text_span = " ".join(doc_tokens[new_start:(new_end + 1)]) if text_span == tok_answer_text: return (new_start, new_end) return (input_start, input_end) def _check_is_max_context(doc_spans, cur_span_index, position): """Check if this is the 'max context' doc span for the token.""" # Because of the sliding window approach taken to scoring documents, a single # token can appear in multiple documents. E.g. # Doc: the man went to the store and bought a gallon of milk # Span A: the man went to the # Span B: to the store and bought # Span C: and bought a gallon of # ... # # Now the word 'bought' will have two scores from spans B and C. We only # want to consider the score with "maximum context", which we define as # the *minimum* of its left and right context (the *sum* of left and # right context will always be the same, of course). # # In the example the maximum context for 'bought' would be span C since # it has 1 left context and 3 right context, while span B has 4 left context # and 0 right context. best_score = None best_span_index = None for (span_index, doc_span) in enumerate(doc_spans): end = doc_span.start + doc_span.length - 1 if position < doc_span.start: continue if position > end: continue num_left_context = position - doc_span.start num_right_context = end - position score = min(num_left_context, num_right_context) + 0.01 * doc_span.length if best_score is None or score > best_score: best_score = score best_span_index = span_index return cur_span_index == best_span_index class SquadExample: """ A single training/test example for the Squad dataset, as loaded from disk. Args: qas_id: The example's unique identifier question_text: The question string context_text: The context string answer_text: The answer string start_position_character: The character position of the start of the answer title: The title of the example answers: None by default, this is used during evaluation. Holds answers as well as their start positions. is_impossible: False by default, set to True if the example has no possible answer. """ def __init__(self, qas_id, question_text, context_text, answer_text, start_position_character, title, answers=None, is_impossible=False): self.qas_id = qas_id self.question_text = question_text self.context_text = context_text self.answer_text = answer_text self.title = title self.is_impossible = is_impossible self.answers = answers self.start_position, self.end_position = 0, 0 doc_tokens = [] char_to_word_offset = [] prev_is_whitespace = True # Split on whitespace so that different tokens may be attributed to their original position. for c in self.context_text: if self._is_whitespace(c): prev_is_whitespace = True else: if prev_is_whitespace: doc_tokens.append(c) else: doc_tokens[-1] += c prev_is_whitespace = False char_to_word_offset.append(len(doc_tokens) - 1) self.doc_tokens = doc_tokens self.char_to_word_offset = char_to_word_offset # Start and end positions only has a value during evaluation. if start_position_character is not None and not is_impossible: self.start_position = char_to_word_offset[start_position_character] self.end_position = char_to_word_offset[ min(start_position_character + len(answer_text) - 1, len(char_to_word_offset) - 1) ] def _is_whitespace(self, c): """Check whether character is whitespace""" if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F: return True return False class InputFeatures: """A single set of features of data.""" def __init__(self, unique_id, example_index, doc_span_index, tokens, token_to_orig_map, token_is_max_context, input_ids, input_mask, token_type_id, start_position=-1, end_position=-1, is_impossible=False): self.unique_id = unique_id self.example_index = example_index self.doc_span_index = doc_span_index self.tokens = tokens self.token_to_orig_map = token_to_orig_map self.token_is_max_context = token_is_max_context self.input_ids = input_ids self.input_mask = input_mask self.token_type_id = token_type_id self.start_position = start_position self.end_position = end_position self.is_impossible = is_impossible pengcheng.mind-mindformer/mindformers/dataset/dataloader/wmt16_dataloader.py000066400000000000000000000104101476650141400276700ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """WMT16 DataLoader.""" import os from mindspore.dataset import GeneratorDataset from mindformers.tools import logger from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class WMT16DataLoader: """WMT16 Dataloader""" _default_column_names = ["source", "target"] def __new__(cls, dataset_dir, column_names=None, stage="train", num_shards=1, shard_id=0): """ WMT16 Dataloader API Args: dataset_dir: the directory to dataset column_names: the output column names, a tuple or a list of string with length 2 stage: the supported `option` are in ["train"、"test"、"del"、"all"] Return: a GeneratorDataset for WMT16 dataset """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") if column_names is None: column_names = cls._default_column_names logger.info("The column_names to the WMT16DataLoader is None, so assign it with default_column_names %s", cls._default_column_names) if column_names and not isinstance(column_names, (tuple, list)): raise TypeError(f"column_names should be a tuple or a list" f" of string with length 2, but got {type(column_names)}") if len(column_names) != 2: raise ValueError(f"the length of column_names should be 2," f" but got {len(column_names)}") if not isinstance(column_names[0], str) or not isinstance(column_names[1], str): raise ValueError(f"the item type of column_names should be string," f" but got {type(column_names[0])} and {type(column_names[1])}") dataset = WMT16DataSet(dataset_dir, stage) return GeneratorDataset(dataset, column_names, num_shards=num_shards, shard_id=shard_id) def read_text(train_file): """Read the text files and return a list.""" with open(train_file) as fp: data = [] for line in fp: line = line.strip() if line: data.append(line) return data class WMT16DataSet: """WMT16 DataSet""" def __init__(self, dataset_dir, stage="train"): """ WMT16DataSet Dataset Args: dataset_dir: the directory to wmt16 dataset stage: the supported key word are in ["train"、"test"、"del"、"all"] Return: an iterable dataset for wmt16 dataset """ if not os.path.isdir(dataset_dir): raise ValueError(f"{dataset_dir} is not existed.") self.dataset_dir = dataset_dir self.stage = stage dataset_dict = dict() def read_and_add_to_stage(cur_stage): source_path = os.path.join(dataset_dir, f'{cur_stage}.source') src = read_text(source_path) target_source = os.path.join(dataset_dir, f'{cur_stage}.target') tgt = read_text(target_source) src_tgt_pair = list(zip(src, tgt)) return src_tgt_pair logger.info("Start to read the raw data from the disk %s.", dataset_dir) if 'stage' != 'all': dataset_dict[stage] = read_and_add_to_stage(stage) else: for item in ['train', 'dev', 'test']: dataset_dict[stage] = read_and_add_to_stage(item) self.dataset_dict = dataset_dict def __getitem__(self, item): return self.dataset_dict[self.stage][item] def __len__(self): return len(self.dataset_dict[self.stage]) pengcheng.mind-mindformer/mindformers/dataset/image_text_retrieval_dataset.py000066400000000000000000000051401476650141400303460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-text Retrieval Dataset.""" import os import mindspore.common.dtype as mstype import mindspore.dataset as ds import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .base_dataset import BaseDataset from .transforms import build_transforms @MindFormerRegister.register(MindFormerModuleType.DATASET) class ImageToTextRetrievalDataset(BaseDataset): """Image-text Retrieval Dataset for filip fine-tuning and evaluation.""" def __new__(cls, dataset_config: dict = None): logger.info("Now Create Image-text Retrieval Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = ds.MindDataset(dataset_config.data_loader.dataset_dir, shuffle=dataset_config.data_loader.shuffle, num_shards=device_num, shard_id=rank_id) transforms = build_transforms(dataset_config.transforms) if transforms is not None: dataset = get_dataset_map(dataset, transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing, input_columns="image", output_columns=['image']) type_cast_op = C.TypeCast(mstype.float32) dataset = get_dataset_map(dataset, type_cast_op, input_columns="image", output_columns=['image']) dataset = dataset.project(["image", "token"]) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder) return dataset pengcheng.mind-mindformer/mindformers/dataset/img_cls_dataset.py000066400000000000000000000112541476650141400255630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Dataset.""" import os import mindspore.dataset.transforms.c_transforms as C import mindspore.common.dtype as mstype from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class ImageCLSDataset(BaseDataset): """ Image Classification Dataset API. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import ImageCLSDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['image_classification']['vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/image_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='ImageCLSDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = ImageCLSDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Image Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) sampler = build_sampler(dataset_config.sampler) type_cast_op = C.TypeCast(mstype.int32) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: dataset = get_dataset_map(dataset, transforms, input_columns=dataset_config.input_columns[0], num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = get_dataset_map(dataset, type_cast_op, input_columns=dataset_config.input_columns[1], num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) if not dataset_config.do_eval and dataset_config.mixup_op is not None: mixup_op = build_transforms(class_name="Mixup", **dataset_config.mixup_op) dataset = get_dataset_map(dataset, mixup_op, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-mindformer/mindformers/dataset/keyword_gen_dataset.py000066400000000000000000000354141476650141400264670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Keyword Generation Dataset.""" import copy import os import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C import numpy as np from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.dataset.base_dataset import BaseDataset from mindformers.dataset.dataloader import build_dataset_loader from mindformers.models.build_tokenizer import build_tokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from mindformers.version_control import get_dataset_map @MindFormerRegister.register(MindFormerModuleType.DATASET) class KeyWordGenDataset(BaseDataset): """Keyword generation dataset. Examples: >>> from mindformers.dataset.dataloader.adgen_dataloader import ADGenDataLoader >>> from mindformers.dataset import build_dataset >>> from mindformers import MindFormerConfig >>> cfg = MindFormerConfig("./configs/glm/run_glm_6b_finetune.yaml") >>> dataset = build_dataset(cfg.eval_dataset_task) >>> for item in dataset.create_dict_iterator(): >>> print(item) >>> break """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Keyword Generation Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if isinstance(dataset_config.tokenizer, BaseTokenizer): cls.tokenizer = dataset_config.tokenizer else: cls.tokenizer = build_tokenizer(dataset_config.tokenizer) cls.ignore_pad_token_for_loss = dataset_config.ignore_pad_token_for_loss cls.max_source_length = dataset_config.max_source_length cls.max_target_length = dataset_config.max_target_length cls.max_seq_length = cls.max_source_length + cls.max_target_length cls.phase = dataset_config.data_loader.phase cls.version = dataset_config.data_loader.version if dataset_config.data_loader.type != 'MindDataset': dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset @classmethod def _tokenizer_map(cls, dataset, tokenizer_config): """Maps the tokenizer on the source and the output""" phase = cls.phase version = cls.version if cls.version else 1 logger.info("Start tokenize on the dataset using tokenizer: %s", tokenizer_config) if version == 2: train_dataset_function = cls.train_dataset_functionv2 train_output_columns = ["input_ids", "labels"] eval_dataset_function = cls.eval_dataset_functionv2 else: train_dataset_function = cls.train_dataset_function train_output_columns = ["input_ids", "labels", "position_ids", "attention_mask"] eval_dataset_function = cls.eval_dataset_function input_columns = ["prompt", "answer"] eval_output_columns = ["input_ids", "labels"] # Avoid to_json error when summary monitor is opened def train_dataset_func(prompt, answer): return train_dataset_function(prompt, answer) def eval_dataset_func(prompt, answer): return eval_dataset_function(prompt, answer) if phase == "train": dataset = get_dataset_map(dataset, train_dataset_func, input_columns=input_columns, output_columns=train_output_columns) dataset = dataset.project(columns=train_output_columns) if phase == "eval": dataset = get_dataset_map(dataset, eval_dataset_func, input_columns=input_columns, output_columns=eval_output_columns) dataset = dataset.project(columns=eval_output_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._tokenizer_map(dataset, dataset_config.tokenizer) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_config = copy.deepcopy(dataset_config) dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") logger.info("Using args %s to instance the dataset.", dataset_config.data_loader) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset @classmethod def train_dataset_function(cls, prompt, answer): """generates train dataset""" prompt, answer = prompt.tolist(), answer.tolist() prompt_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=False) answer_ids = cls.tokenizer.encode(text=answer, add_special_tokens=False) if len(prompt_ids) > cls.max_source_length - 1: prompt_ids = prompt_ids[: cls.max_source_length - 1] if len(answer_ids) > cls.max_target_length - 2: answer_ids = answer_ids[: cls.max_target_length - 2] input_ids = cls.tokenizer.build_inputs_with_special_tokens(prompt_ids, answer_ids) context_length = input_ids.index(cls.tokenizer.bos_token_id) mask_position = context_length - 1 label = [-100] * context_length + input_ids[mask_position + 2:] # +1 for logits shift pad_len = cls.max_seq_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len label = label + [cls.tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if cls.ignore_pad_token_for_loss: label = [(l if l != cls.tokenizer.pad_token_id else -100) for l in label] position_ids = cls.create_position_ids(np.array(input_ids)) attention_mask = cls.get_masks(np.array(input_ids)) return input_ids, label, position_ids, attention_mask @classmethod def train_dataset_functionv2(cls, prompt, answer): """generates train dataset""" max_seq_length = cls.max_source_length + cls.max_target_length + 1 prompt, answer = prompt.tolist(), answer.tolist() history = None prompt = cls.tokenizer.build_prompt(prompt, history) prompt_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True, max_length=cls.max_source_length) answer_ids = cls.tokenizer.encode(text=answer, add_special_tokens=False, max_length=cls.max_target_length) if len(prompt_ids) > cls.max_source_length - 1: prompt_ids = prompt_ids[: cls.max_source_length - 1] if len(answer_ids) > cls.max_target_length - 2: answer_ids = answer_ids[: cls.max_target_length - 2] context_length = len(prompt_ids) input_ids = prompt_ids + answer_ids + [cls.tokenizer.eos_token_id] labels = [cls.tokenizer.pad_token_id] * context_length + answer_ids[1:] + [cls.tokenizer.eos_token_id] pad_len = max_seq_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len labels = labels + [cls.tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if cls.ignore_pad_token_for_loss: labels = [(l if l != cls.tokenizer.pad_token_id else -100) for l in labels] return input_ids, labels @classmethod def eval_dataset_functionv2(cls, prompt, answer): """generates eval dataset""" prompt, answer = prompt.tolist(), answer.tolist() history = None prompt = cls.tokenizer.build_prompt(prompt, history) if len(prompt) > cls.max_source_length - 1: prompt = prompt[: cls.max_source_length - 1] if len(answer) > cls.max_target_length - 1: answer = answer[: cls.max_target_length - 1] input_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True, max_length=cls.max_source_length) label = cls.tokenizer.encode(text=answer, add_special_tokens=True, max_length=cls.max_target_length) pad_len = cls.max_source_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len pad_len = cls.max_target_length - len(label) label = label + [cls.tokenizer.pad_token_id] * pad_len return input_ids, label @classmethod def eval_dataset_function(cls, prompt, answer): """generates eval dataset""" prompt, answer = prompt.tolist(), answer.tolist() if len(prompt) > cls.max_source_length - 2: prompt = prompt[: cls.max_source_length - 2] if len(answer) > cls.max_target_length - 2: answer = answer[: cls.max_target_length - 2] input_ids = cls.tokenizer.encode(text=prompt, add_special_tokens=True) label = cls.tokenizer.encode(text=answer, add_special_tokens=True) pad_len = cls.max_source_length - len(input_ids) input_ids = input_ids + [cls.tokenizer.pad_token_id] * pad_len pad_len = cls.max_target_length - len(label) label = label + [cls.tokenizer.pad_token_id] * pad_len return input_ids, label @classmethod def get_masks(cls, input_ids, bos_token_id=130004): """generate mask from input id""" seq_length = input_ids.shape[0] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] attention_mask = np.tril(np.ones((seq_length, seq_length), dtype=np.float32)) for context_length in context_lengths: attention_mask[:, :context_length] = 1 attention_mask = np.logical_not(attention_mask.astype(np.bool_)) attention_mask = attention_mask.astype(np.float32) attention_mask = np.expand_dims(attention_mask, 0) return attention_mask @classmethod def get_position_ids(cls, input_ids, mask_positions, use_gmasks=None, bos_token_id=130004, position_encoding_2d=True): """generate position ids from input id and mask positions""" seq_length = input_ids.shape[0] if use_gmasks is None: use_gmasks = [False] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] if position_encoding_2d: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): position_ids[context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, dtype=np.int64), np.arange(seq_length - context_length, dtype=np.int64) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0).squeeze() position_ids = np.stack((position_ids, block_position_ids), axis=0) else: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids @classmethod def create_position_ids(cls, input_ids, gmask_token_id=130001): """generate position ids from input id""" seq_length = input_ids.shape[0] seqs = input_ids # 要求input_ids中, 每行有且仅有一个gMASK use_gmasks = gmask_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(seqs, use_gmasks) mask_positions = np.argwhere(mask)[:, -1] position_ids = cls.get_position_ids(input_ids, mask_positions=mask_positions, use_gmasks=use_gmasks) return position_ids pengcheng.mind-mindformer/mindformers/dataset/labels.py000066400000000000000000000375001476650141400237050ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # This file was refer to project: # https://huggingface.co/WinKawaks/vit-tiny-patch16-224/blob/main/config.json # ============================================================================ """labels of imagenet""" imagenet_labels = [ "tench", "goldfish", "great white shark", "tiger shark", "hammerhead", "electric ray", "stingray", "cock", "hen", "ostrich", "brambling", "goldfinch", "house finch", "junco", "indigo bunting", "robin", "bulbul", "jay", "magpie", "chickadee", "water ouzel", "kite", "bald eagle", "vulture", "great grey owl", "European fire salamander", "common newt", "eft", "spotted salamander", "axolotl", "bullfrog", "tree frog", "tailed frog", "loggerhead", "leatherback turtle", "mud turtle", "terrapin", "box turtle", "banded gecko", "common iguana", "American chameleon", "whiptail", "agama", "frilled lizard", "alligator lizard", "Gila monster", "green lizard", "African chameleon", "Komodo dragon", "African crocodile", "American alligator", "triceratops", "thunder snake", "ringneck snake", "hognose snake", "green snake", "king snake", "garter snake", "water snake", "vine snake", "night snake", "boa constrictor", "rock python", "Indian cobra", "green mamba", "sea snake", "horned viper", "diamondback", "sidewinder", "trilobite", "harvestman", "scorpion", "black and gold garden spider", "barn spider", "garden spider", "black widow", "tarantula", "wolf spider", "tick", "centipede", "black grouse", "ptarmigan", "ruffed grouse", "prairie chicken", "peacock", "quail", "partridge", "African grey", "macaw", "sulphur-crested cockatoo", "lorikeet", "coucal", "bee eater", "hornbill", "hummingbird", "jacamar", "toucan", "drake", "red-breasted merganser", "goose", "black swan", "tusker", "echidna", "platypus", "wallaby", "koala", "wombat", "jellyfish", "sea anemone", "brain coral", "flatworm", "nematode", "conch", "snail", "slug", "sea slug", "chiton", "chambered nautilus", "Dungeness crab", "rock crab", "fiddler crab", "king crab", "American lobster", "spiny lobster", "crayfish", "hermit crab", "isopod", "white stork", "black stork", "spoonbill", "flamingo", "little blue heron", "American egret", "bittern", "crane", "limpkin", "European gallinule", "American coot", "bustard", "ruddy turnstone", "red-backed sandpiper", "redshank", "dowitcher", "oystercatcher", "pelican", "king penguin", "albatross", "grey whale", "killer whale", "dugong", "sea lion", "Chihuahua", "Japanese spaniel", "Maltese dog", "Pekinese", "Shih-Tzu", "Blenheim spaniel", "papillon", "toy terrier", "Rhodesian ridgeback", "Afghan hound", "basset", "beagle", "bloodhound", "bluetick", "black-and-tan coonhound", "Walker hound", "English foxhound", "redbone", "borzoi", "Irish wolfhound", "Italian greyhound", "whippet", "Ibizan hound", "Norwegian elkhound", "otterhound", "Saluki", "Scottish deerhound", "Weimaraner", "Staffordshire bullterrier", "American Staffordshire terrier", "Bedlington terrier", "Border terrier", "Kerry blue terrier", "Irish terrier", "Norfolk terrier", "Norwich terrier", "Yorkshire terrier", "wire-haired fox terrier", "Lakeland terrier", "Sealyham terrier", "Airedale", "cairn", "Australian terrier", "Dandie Dinmont", "Boston bull", "miniature schnauzer", "giant schnauzer", "standard schnauzer", "Scotch terrier", "Tibetan terrier", "silky terrier", "soft-coated wheaten terrier", "West Highland white terrier", "Lhasa", "flat-coated retriever", "curly-coated retriever", "golden retriever", "Labrador retriever", "Chesapeake Bay retriever", "German short-haired pointer", "vizsla", "English setter", "Irish setter", "Gordon setter", "Brittany spaniel", "clumber", "English springer", "Welsh springer spaniel", "cocker spaniel", "Sussex spaniel", "Irish water spaniel", "kuvasz", "schipperke", "groenendael", "malinois", "briard", "kelpie", "komondor", "Old English sheepdog", "Shetland sheepdog", "collie", "Border collie", "Bouvier des Flandres", "Rottweiler", "German shepherd", "Doberman", "miniature pinscher", "Greater Swiss Mountain dog", "Bernese mountain dog", "Appenzeller", "EntleBucher", "boxer", "bull mastiff", "Tibetan mastiff", "French bulldog", "Great Dane", "Saint Bernard", "Eskimo dog", "malamute", "Siberian husky", "dalmatian", "affenpinscher", "basenji", "pug", "Leonberg", "Newfoundland", "Great Pyrenees", "Samoyed", "Pomeranian", "chow", "keeshond", "Brabancon griffon", "Pembroke", "Cardigan", "toy poodle", "miniature poodle", "standard poodle", "Mexican hairless", "timber wolf", "white wolf", "red wolf", "coyote", "dingo", "dhole", "African hunting dog", "hyena", "red fox", "kit fox", "Arctic fox", "grey fox", "tabby", "tiger cat", "Persian cat", "Siamese cat", "Egyptian cat", "cougar", "lynx", "leopard", "snow leopard", "jaguar", "lion", "tiger", "cheetah", "brown bear", "American black bear", "ice bear", "sloth bear", "mongoose", "meerkat", "tiger beetle", "ladybug", "ground beetle", "long-horned beetle", "leaf beetle", "dung beetle", "rhinoceros beetle", "weevil", "fly", "bee", "ant", "grasshopper", "cricket", "walking stick", "cockroach", "mantis", "cicada", "leafhopper", "lacewing", "dragonfly", "damselfly", "admiral", "ringlet", "monarch", "cabbage butterfly", "sulphur butterfly", "lycaenid", "starfish", "sea urchin", "sea cucumber", "wood rabbit", "hare", "Angora", "hamster", "porcupine", "fox squirrel", "marmot", "beaver", "guinea pig", "sorrel", "zebra", "hog", "wild boar", "warthog", "hippopotamus", "ox", "water buffalo", "bison", "ram", "bighorn", "ibex", "hartebeest", "impala", "gazelle", "Arabian camel", "llama", "weasel", "mink", "polecat", "black-footed ferret", "otter", "skunk", "badger", "armadillo", "three-toed sloth", "orangutan", "gorilla", "chimpanzee", "gibbon", "siamang", "guenon", "patas", "baboon", "macaque", "langur", "colobus", "proboscis monkey", "marmoset", "capuchin", "howler monkey", "titi", "spider monkey", "squirrel monkey", "Madagascar cat", "indri", "Indian elephant", "African elephant", "lesser panda", "giant panda", "barracouta", "eel", "coho", "rock beauty", "anemone fish", "sturgeon", "gar", "lionfish", "puffer", "abacus", "abaya", "academic gown", "accordion", "acoustic guitar", "aircraft carrier", "airliner", "airship", "altar", "ambulance", "amphibian", "analog clock", "apiary", "apron", "ashcan", "assault rifle", "backpack", "bakery", "balance beam", "balloon", "ballpoint", "Band Aid", "banjo", "bannister", "barbell", "barber chair", "barbershop", "barn", "barometer", "barrel", "barrow", "baseball", "basketball", "bassinet", "bassoon", "bathing cap", "bath towel", "bathtub", "beach wagon", "beacon", "beaker", "bearskin", "beer bottle", "beer glass", "bell cote", "bib", "bicycle-built-for-two", "bikini", "binder", "binoculars", "birdhouse", "boathouse", "bobsled", "bolo tie", "bonnet", "bookcase", "bookshop", "bottlecap", "bow", "bow tie", "brass", "brassiere", "breakwater", "breastplate", "broom", "bucket", "buckle", "bulletproof vest", "bullet train", "butcher shop", "cab", "caldron", "candle", "cannon", "canoe", "can opener", "cardigan", "car mirror", "carousel", "carpenter's kit", "carton", "car wheel", "cash machine", "cassette", "cassette player", "castle", "catamaran", "CD player", "cello", "cellular telephone", "chain", "chainlink fence", "chain mail", "chain saw", "chest", "chiffonier", "chime", "china cabinet", "Christmas stocking", "church", "cinema", "cleaver", "cliff dwelling", "cloak", "clog", "cocktail shaker", "coffee mug", "coffeepot", "coil", "combination lock", "computer keyboard", "confectionery", "container ship", "convertible", "corkscrew", "cornet", "cowboy boot", "cowboy hat", "cradle", "crane", "crash helmet", "crate", "crib", "Crock Pot", "croquet ball", "crutch", "cuirass", "dam", "desk", "desktop computer", "dial telephone", "diaper", "digital clock", "digital watch", "dining table", "dishrag", "dishwasher", "disk brake", "dock", "dogsled", "dome", "doormat", "drilling platform", "drum", "drumstick", "dumbbell", "Dutch oven", "electric fan", "electric guitar", "electric locomotive", "entertainment center", "envelope", "espresso maker", "face powder", "feather boa", "file", "fireboat", "fire engine", "fire screen", "flagpole", "flute", "folding chair", "football helmet", "forklift", "fountain", "fountain pen", "four-poster", "freight car", "French horn", "frying pan", "fur coat", "garbage truck", "gasmask", "gas pump", "goblet", "go-kart", "golf ball", "golfcart", "gondola", "gong", "gown", "grand piano", "greenhouse", "grille", "grocery store", "guillotine", "hair slide", "hair spray", "half track", "hammer", "hamper", "hand blower", "hand-held computer", "handkerchief", "hard disc", "harmonica", "harp", "harvester", "hatchet", "holster", "home theater", "honeycomb", "hook", "hoopskirt", "horizontal bar", "horse cart", "hourglass", "iPod", "iron", "jack-o'-lantern", "jean", "jeep", "jersey", "jigsaw puzzle", "jinrikisha", "joystick", "kimono", "knee pad", "knot", "lab coat", "ladle", "lampshade", "laptop", "lawn mower", "lens cap", "letter opener", "library", "lifeboat", "lighter", "limousine", "liner", "lipstick", "Loafer", "lotion", "loudspeaker", "loupe", "lumbermill", "magnetic compass", "mailbag", "mailbox", "maillot", "maillot", "manhole cover", "maraca", "marimba", "mask", "matchstick", "maypole", "maze", "measuring cup", "medicine chest", "megalith", "microphone", "microwave", "military uniform", "milk can", "minibus", "miniskirt", "minivan", "missile", "mitten", "mixing bowl", "mobile home", "Model T", "modem", "monastery", "monitor", "moped", "mortar", "mortarboard", "mosque", "mosquito net", "motor scooter", "mountain bike", "mountain tent", "mouse", "mousetrap", "moving van", "muzzle", "nail", "neck brace", "necklace", "nipple", "notebook", "obelisk", "oboe", "ocarina", "odometer", "oil filter", "organ", "oscilloscope", "overskirt", "oxcart", "oxygen mask", "packet", "paddle", "paddlewheel", "padlock", "paintbrush", "pajama", "palace", "panpipe", "paper towel", "parachute", "parallel bars", "park bench", "parking meter", "passenger car", "patio", "pay-phone", "pedestal", "pencil box", "pencil sharpener", "perfume", "Petri dish", "photocopier", "pick", "pickelhaube", "picket fence", "pickup", "pier", "piggy bank", "pill bottle", "pillow", "ping-pong ball", "pinwheel", "pirate", "pitcher", "plane", "planetarium", "plastic bag", "plate rack", "plow", "plunger", "Polaroid camera", "pole", "police van", "poncho", "pool table", "pop bottle", "pot", "potter's wheel", "power drill", "prayer rug", "printer", "prison", "projectile", "projector", "puck", "punching bag", "purse", "quill", "quilt", "racer", "racket", "radiator", "radio", "radio telescope", "rain barrel", "recreational vehicle", "reel", "reflex camera", "refrigerator", "remote control", "restaurant", "revolver", "rifle", "rocking chair", "rotisserie", "rubber eraser", "rugby ball", "rule", "running shoe", "safe", "safety pin", "saltshaker", "sandal", "sarong", "sax", "scabbard", "scale", "school bus", "schooner", "scoreboard", "screen", "screw", "screwdriver", "seat belt", "sewing machine", "shield", "shoe shop", "shoji", "shopping basket", "shopping cart", "shovel", "shower cap", "shower curtain", "ski", "ski mask", "sleeping bag", "slide rule", "sliding door", "slot", "snorkel", "snowmobile", "snowplow", "soap dispenser", "soccer ball", "sock", "solar dish", "sombrero", "soup bowl", "space bar", "space heater", "space shuttle", "spatula", "speedboat", "spider web", "spindle", "sports car", "spotlight", "stage", "steam locomotive", "steel arch bridge", "steel drum", "stethoscope", "stole", "stone wall", "stopwatch", "stove", "strainer", "streetcar", "stretcher", "studio couch", "stupa", "submarine", "suit", "sundial", "sunglass", "sunglasses", "sunscreen", "suspension bridge", "swab", "sweatshirt", "swimming trunks", "swing", "switch", "syringe", "table lamp", "tank", "tape player", "teapot", "teddy", "television", "tennis ball", "thatch", "theater curtain", "thimble", "thresher", "throne", "tile roof", "toaster", "tobacco shop", "toilet seat", "torch", "totem pole", "tow truck", "toyshop", "tractor", "trailer truck", "tray", "trench coat", "tricycle", "trimaran", "tripod", "triumphal arch", "trolleybus", "trombone", "tub", "turnstile", "typewriter keyboard", "umbrella", "unicycle", "upright", "vacuum", "vase", "vault", "velvet", "vending machine", "vestment", "viaduct", "violin", "volleyball", "waffle iron", "wall clock", "wallet", "wardrobe", "warplane", "washbasin", "washer", "water bottle", "water jug", "water tower", "whiskey jug", "whistle", "wig", "window screen", "window shade", "Windsor tie", "wine bottle", "wing", "wok", "wooden spoon", "wool", "worm fence", "wreck", "yawl", "yurt", "web site", "comic book", "crossword puzzle", "street sign", "traffic light", "book jacket", "menu", "plate", "guacamole", "consomme", "hot pot", "trifle", "ice cream", "ice lolly", "French loaf", "bagel", "pretzel", "cheeseburger", "hotdog", "mashed potato", "head cabbage", "broccoli", "cauliflower", "zucchini", "spaghetti squash", "acorn squash", "butternut squash", "cucumber", "artichoke", "bell pepper", "cardoon", "mushroom", "Granny Smith", "strawberry", "orange", "lemon", "fig", "pineapple", "banana", "jackfruit", "custard apple", "pomegranate", "hay", "carbonara", "chocolate sauce", "dough", "meat loaf", "pizza", "potpie", "burrito", "red wine", "espresso", "cup", "eggnog", "alp", "bubble", "cliff", "coral reef", "geyser", "lakeside", "promontory", "sandbar", "seashore", "valley", "volcano", "ballplayer", "groom", "scuba diver", "rapeseed", "daisy", "yellow lady's slipper", "corn", "acorn", "hip", "buckeye", "coral fungus", "agaric", "gyromitra", "stinkhorn", "earthstar", "hen-of-the-woods", "bolete", "ear", "toilet tissue", ] """labels of cluener""" cluener_labels = ["O", "B-address", "B-book", "B-company", "B-game", "B-government", "B-movie", "B-name", "B-organization", "B-position", "B-scene", "I-address", "I-book", "I-company", "I-game", "I-government", "I-movie", "I-name", "I-organization", "I-position", "I-scene", "S-address", "S-book", "S-company", "S-game", "S-government", "S-movie", "S-name", "S-organization", "S-position", "S-scene"] """labels of mnli""" mnli_labels = ["contradiction", "entailment", "neutral"] """labels of cola""" cola_labels = ["unacceptable", "acceptable"] """labels of sst2""" sst2_labels = ["negative", "positive"] """labels of imdb""" imdb_labels = ["negative", "positive"] """labels of ag news""" ag_news_labels = ["world", "sports", "business", "sci/tech"] # labels dict labels = {"imagenet": imagenet_labels, "cluener": cluener_labels, "mnli": mnli_labels, "cola": cola_labels, "sst2": sst2_labels, "imdb": imdb_labels, "ag_news": ag_news_labels} pengcheng.mind-mindformer/mindformers/dataset/mask/000077500000000000000000000000001476650141400230175ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/mask/__init__.py000066400000000000000000000014501476650141400251300ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Mask API.""" from .build_mask import build_mask from .vision_mask import * __all__ = [] __all__.extend(vision_mask.__all__) pengcheng.mind-mindformer/mindformers/dataset/mask/__pycache__/000077500000000000000000000000001476650141400252275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/mask/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004571476650141400314250ustar00rootroot00000000000000a ?cf(@s,dZddlmZddlTgZeejdS)zMindFormers Mask API.) build_mask)*N)__doc__rZ vision_mask__all__extendrrL/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/__init__.pys pengcheng.mind-mindformer/mindformers/dataset/mask/__pycache__/build_mask.cpython-39.pyc000066400000000000000000000030471476650141400317760ustar00rootroot00000000000000a ?cf@s2dZddlmZmZmZdeeeedddZdS) zBuild Mask API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigN mask_policy)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a'Build mask For MindFormer. Instantiate the mask from MindFormerRegister's registry. Args: config (dict): The task mask's config. Default: None. default_args (dict): The default argument of mask API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'mask'. class_name (str): The class name of mask API. Default: None. Return: The function instance of mask API. Examples: >>> from mindformers import build_mask >>> mask_config = {'type': 'SimMask', 'input_size': 224} >>> # 1) use config dict to build mask >>> mask_from_config = build_mask(mask_config) >>> # 2) use class name to build mask >>> mask_class_name = build_mask(class_name='SimMask', input_size=224) N)r) isinstancedictrrget_instance_from_cfgr MASK_POLICY get_instance)rrrr kwargsrN/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/build_mask.py build_masksr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrspengcheng.mind-mindformer/mindformers/dataset/mask/__pycache__/vision_mask.cpython-39.pyc000066400000000000000000000057031476650141400322070ustar00rootroot00000000000000a ?cf@spdZddlZddlmZddlmZmZddgZe ej Gdddej Z e ej Gdddej Z dS) zSelf-Define Vision Mask Policy.N) py_transforms)MindFormerRegisterMindFormerModuleTypeSimMaskMaeMaskcs2eZdZdZd fdd Zdd Zd d ZZS) rzSimMIM Mask Policy. 333333?cstt|||_||_||_||_|j|jdkrNtd|jd|j|j|jdkrvtd|jd|j|j|j|_|j|j|_ |jd|_ t t |j |j|_dS)Nrz;input_size % mask_patch_size must be 0, but get input_size z and mask_patch_size zFmask_patch_size % model_patch_size must be 0, but get mask_patch_size z and model_patch_size )superr__init__ input_sizemask_patch_sizemodel_patch_size mask_ratio ValueError rand_sizescale token_countintnpceil mask_count)selfrrrr __class__O/home/ma-user/work/mindformers-baichuan/mindformers/dataset/mask/vision_mask.pyr s&  zSimMask.__init__cCshtj|jd|j}tj|jtjd}d||<||j|jf}|j |j ddj |j dd}||fS)N)dtyperaxis) rrandom permutationrrzerosint32reshaperrepeatr)rimgZmask_idxmaskrrr__call__0s zSimMask.__call__cCsdS)NzMask generator for simmin arch.rrrrr__repr__9szSimMask.__repr__)rrr r __name__ __module__ __qualname____doc__r r+r- __classcell__rrrrrs cs2eZdZdZd fdd ZddZd d ZZS) rzMAE Mask Policy.rr ?csRtt|d|kr"dks,ntd||d|_td||j|_dS)Nrr zLmasking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.r )r rr r num_patchesrkeep_num)rr patch_sizerrrrr @s zMaeMask.__init__cCs|tjtjj|jfdddtj}tj|ddtj}t|jftj}d|d|j<|d|j}||||f}|S)N)sizerr!) rargsortr#uniformr5astyper&onesr6)rZimgsZ rand_indicesZ ids_restorer*Z unmask_indexoutrrrr+Is zMaeMask.__call__cCsdS)NzMask generator for mae arch.rr,rrrr-SszMaeMask.__repr__)rr r4r.rrrrr=s  )r2numpyrZmindspore.dataset.transformsrmindformers.tools.registerrr__all__register MASK_POLICYPyTensorOperationrrrrrrs  # pengcheng.mind-mindformer/mindformers/dataset/mask/build_mask.py000066400000000000000000000042241476650141400255050ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Mask API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_mask( config: dict = None, default_args: dict = None, module_type: str = 'mask_policy', class_name: str = None, **kwargs): r"""Build mask For MindFormer. Instantiate the mask from MindFormerRegister's registry. Args: config (dict): The task mask's config. Default: None. default_args (dict): The default argument of mask API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'mask'. class_name (str): The class name of mask API. Default: None. Return: The function instance of mask API. Examples: >>> from mindformers import build_mask >>> mask_config = {'type': 'SimMask', 'input_size': 224} >>> # 1) use config dict to build mask >>> mask_from_config = build_mask(mask_config) >>> # 2) use class name to build mask >>> mask_class_name = build_mask(class_name='SimMask', input_size=224) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.MASK_POLICY, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/dataset/mask/vision_mask.py000066400000000000000000000072601476650141400257200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/adapter-hub/adapter-transformers/blob/master/examples/pytorch/image-pretraining/run_mim.py # ============================================================================ """Self-Define Vision Mask Policy.""" import numpy as np from mindspore.dataset.transforms import py_transforms from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SimMask', 'MaeMask'] @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class SimMask(py_transforms.PyTensorOperation): """SimMIM Mask Policy.""" def __init__(self, input_size=192, mask_patch_size=32, model_patch_size=4, mask_ratio=0.6): super(SimMask, self).__init__() self.input_size = input_size self.mask_patch_size = mask_patch_size self.model_patch_size = model_patch_size self.mask_ratio = mask_ratio if self.input_size % self.mask_patch_size != 0: raise ValueError(f"input_size % mask_patch_size must be 0, but get input_size {self.input_size} and " f"mask_patch_size {self.mask_patch_size}") if self.mask_patch_size % self.model_patch_size != 0: raise ValueError(f"mask_patch_size % model_patch_size must be 0, but get mask_patch_size " f"{self.mask_patch_size} and model_patch_size {self.model_patch_size}") self.rand_size = self.input_size // self.mask_patch_size self.scale = self.mask_patch_size // self.model_patch_size self.token_count = self.rand_size ** 2 self.mask_count = int(np.ceil(self.token_count * self.mask_ratio)) def __call__(self, img): mask_idx = np.random.permutation(self.token_count)[:self.mask_count] mask = np.zeros(self.token_count, dtype=np.int32) mask[mask_idx] = 1 mask = mask.reshape((self.rand_size, self.rand_size)) mask = mask.repeat(self.scale, axis=0).repeat(self.scale, axis=1) return img, mask def __repr__(self): return "Mask generator for simmin arch." @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class MaeMask(py_transforms.PyTensorOperation): """MAE Mask Policy.""" def __init__(self, input_size=192, patch_size=4, mask_ratio=0.75): super(MaeMask, self).__init__() if not 0 < mask_ratio < 1: raise ValueError('masking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.') # seq_length self.num_patches = (input_size // patch_size) ** 2 # seq masked number self.keep_num = int((1 - mask_ratio) * self.num_patches) def __call__(self, imgs): rand_indices = np.argsort( np.random.uniform(size=(self.num_patches,)), axis=0).astype(np.int32) ids_restore = np.argsort(rand_indices, axis=0).astype(np.int32) mask = np.ones((self.num_patches,)).astype(np.int32) mask[:self.keep_num] = 0 unmask_index = rand_indices[:self.keep_num] out = (imgs, mask, ids_restore, unmask_index,) return out def __repr__(self): return "Mask generator for mae arch." pengcheng.mind-mindformer/mindformers/dataset/mask_language_model_dataset.py000066400000000000000000000122171476650141400301240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Dataset.""" import os import copy import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MaskLanguageModelDataset(BaseDataset): """ Bert pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MaskLanguageModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['fill_mask']['bert_tiny_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/bert.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MaskLanguageModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MaskLanguageModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) cls.init_dataset_config(dataset_config) rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config = copy.deepcopy(dataset_config) if not (dataset_config.data_loader.type == 'MindDataset' or dataset_config.data_loader.type == 'TFRecordDataset'): raise NotImplementedError("Now, Causal Language Modeling Dataset only supports " "MindSpore's MindDataset and TFRecordDataset two data loading modes") dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord") or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord") or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id}) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-mindformer/mindformers/dataset/mim_dataset.py000066400000000000000000000106201476650141400247240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Dataset.""" import os from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .mask import build_mask from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class MIMDataset(BaseDataset): """ Masked Image Modeling Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import MIMDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['masked_image_modeling']['mae_vit_base_p16'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/mae.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='MIMDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = MIMDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Masked Image Modeling Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) mask = build_mask(dataset_config.mask_policy) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if transforms is not None: for column in dataset_config.input_columns: dataset = get_dataset_map(dataset, transforms, input_columns=column, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if mask is not None: dataset = get_dataset_map(dataset, mask, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.project(columns=dataset_config.output_columns) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-mindformer/mindformers/dataset/question_answering_dataset.py000066400000000000000000000102111476650141400300620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Question Answering Dataset.""" import os import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from ..models.build_tokenizer import build_tokenizer from .sampler import build_sampler from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class QuestionAnsweringDataset(BaseDataset): """ Question Answering Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import QuestionAnsweringDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['question_answering']['qa_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/question_answering.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='QuestionAnsweringDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = QuestionAnsweringDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Question Answering Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) tokenizer = build_tokenizer(dataset_config.tokenizer) dataset = build_dataset_loader( class_name=dataset_config.data_loader.type, dataset_dir=dataset_config.data_loader.dataset_dir, tokenizer=tokenizer, stage=dataset_config.data_loader.stage, column_names=dataset_config.data_loader.column_names, num_parallel_workers=dataset_config.num_parallel_workers, num_shards=device_num, shard_id=rank_id ) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-mindformer/mindformers/dataset/reward_model_dataset.py000066400000000000000000000241621476650141400266140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Dataset.""" import os import copy import re import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from mindformers.models.build_tokenizer import build_tokenizer from .dataloader import build_dataset_loader from .base_dataset import BaseDataset def get_input_data_batch_slice_map(chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind, rank: int = 0, dis: int = 1, pad_id: int = 2): """ Generate position_id and attention_mask according to input_ids considering eos reset """ rank = int(rank) chosen_input_ids = chosen_input_ids[rank*dis: (rank + 1)*dis] rejected_input_ids = rejected_input_ids[rank*dis: (rank + 1)*dis] chosen_attention_mask = chosen_attention_mask[rank*dis: (rank + 1)*dis] rejected_attention_mask = rejected_attention_mask[rank*dis: (rank + 1)*dis] position_id = position_id[rank*dis: (rank + 1)*dis] loss_mask = loss_mask[rank*dis: (rank + 1)*dis] input_ids = np.concatenate((chosen_input_ids, rejected_input_ids)) attention_mask = np.concatenate((chosen_attention_mask, rejected_attention_mask)) position_id = np.concatenate((position_id, position_id)) end_ind = np.sum(input_ids != pad_id, axis=1) return input_ids, position_id, attention_mask, loss_mask, end_ind @MindFormerRegister.register(MindFormerModuleType.DATASET) class RewardModelDataset(BaseDataset): """Causal Language Model pretrain dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import RewardModelDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_generation']['gpt2'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/dev/docs/model_cards/gpt2.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='RewardModelDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = RewardModelDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Reward Model Dataset.") print("dataset_config", dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) cls.init_dataset_config(dataset_config) logger.info("Now Create Reward Model Dataset1.") rank_id, device_num = cls._check_device_rank_for_parallel(rank_id, device_num) dataset_config.rank_id = rank_id dataset_config.device_num = device_num if dataset_config.data_loader.type != "MindDataset" and \ dataset_config.data_loader.type != "TFRecordDataset": dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) logger.info("Now Create Reward Model Dataset1.5") # type_cast_op = C.TypeCast(mstype.int32) if cls._is_semi_full_batch() or cls._is_data_parallel(): rank_id = 0 dis = dataset_config.batch_size else: # Each card slice a small batch from the full batch dis = dataset_config.batch_size // device_num if dataset_config.batch_size % device_num != 0: raise ValueError( f"batch size {dataset_config.batch_size} should be a multiple of device number {device_num}." " You should change the args: per_batch_size.") logger.info("Now Create Reward Model Dataset1.6") dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns) pad_id = dataset_config.pad_token_id if dataset_config.pad_token_id is not None else 2 logger.info("Now Create Reward Model Dataset2.") map_func = (lambda chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind: \ get_input_data_batch_slice_map(chosen_input_ids, chosen_attention_mask, rejected_input_ids, rejected_attention_mask, position_id, loss_mask, end_ind, rank_id, dis=dis, pad_id=pad_id)) dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.output_columns) dataset = dataset.project(columns=dataset_config.output_columns) type_cast_op = C.TypeCast(mstype.int32) type_cast_op_float = C.TypeCast(mstype.float16) dataset = get_dataset_map(dataset, input_columns="input_ids", operations=type_cast_op) dataset = get_dataset_map(dataset, input_columns="position_id", operations=type_cast_op) dataset = get_dataset_map(dataset, input_columns="attention_mask", operations=type_cast_op_float) dataset = get_dataset_map(dataset, input_columns="loss_mask", operations=type_cast_op_float) dataset = get_dataset_map(dataset, input_columns="end_ind", operations=type_cast_op) dataset = dataset.repeat(dataset_config.repeat) return dataset @classmethod def _prepare_for_model(cls, dataset, dataset_config): """Preprocess data for gpt2 model""" tokenizer_config = dataset_config.tokenizer tokenizer = build_tokenizer(tokenizer_config) max_length = tokenizer_config.max_length def map_func(input_data): input_data = input_data.tolist() input_ids = tokenizer(input_data, padding='max_length', max_length=max_length, truncation=True, add_special_tokens=False) return input_ids.get('input_ids') dataset = get_dataset_map(dataset, map_func, input_columns=dataset_config.input_columns, output_columns=dataset_config.input_columns) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id}) dataset = cls._prepare_for_model(dataset, dataset_config) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" dataset_files = [] mind_compile = re.compile("mindrecord0*$") if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if re.findall(mind_compile, file) or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if re.findall(mind_compile, data_dir) or data_dir.endswith(".tfrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': dataset_config.device_num, 'shard_id': dataset_config.rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-mindformer/mindformers/dataset/sampler/000077500000000000000000000000001476650141400235275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/sampler/__init__.py000066400000000000000000000013621476650141400256420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Sampler API.""" from .build_sampler import build_sampler __all__ = [] pengcheng.mind-mindformer/mindformers/dataset/sampler/__pycache__/000077500000000000000000000000001476650141400257375ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/sampler/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004061476650141400321270ustar00rootroot00000000000000a ?cf@sdZddlmZgZdS)zMindFormers Sampler API.) build_samplerN)__doc__r__all__rrO/home/ma-user/work/mindformers-baichuan/mindformers/dataset/sampler/__init__.pys pengcheng.mind-mindformer/mindformers/dataset/sampler/__pycache__/build_sampler.cpython-39.pyc000066400000000000000000000040211476650141400332070ustar00rootroot00000000000000a ?cf @sTdZddlZddlmZddlmZmZmZd e e e e dddZ d d Z e dS) zBuild Sampler API.N)samplers)MindFormerRegisterMindFormerModuleTypeMindFormerConfigdataset_sampler)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aiBuild sampler For MindFormer. Instantiate the sampler from MindFormerRegister's registry. Args: config (dict): The task sampler's config. Default: None. default_args (dict): The default argument of sampler API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'sampler'. class_name (str): The class name of sampler API. Default: None. Return: The function instance of sampler API. Examples: >>> from mindformers import build_sampler >>> sampler_config = {'type': 'RandomSampler', 'replacement': False} >>> # 1) use config dict to build sampler >>> sampler_from_config = build_sampler(sampler_config) >>> # 2) use class name to build sampler >>> sampler_class_name = build_sampler(class_name='RandomSampler', replacement=False) N)r) isinstancedictrrget_instance_from_cfgrDATASET_SAMPLER get_instance)rrr r kwargsrT/home/ma-user/work/mindformers-baichuan/mindformers/dataset/sampler/build_sampler.py build_samplersrcCs@ttD]2}|drqtt|}t|rt|tj qdS)z. register MindSpore builtin transforms class. __N) dirsp startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_samplers8s     r)NNrN) __doc__rmindspore.datasetrrmindformers.tools.registerrrrr strrrrrrrs  ! pengcheng.mind-mindformer/mindformers/dataset/sampler/build_sampler.py000066400000000000000000000052371476650141400267320ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Sampler API.""" import inspect from mindspore.dataset import samplers as sp from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_sampler( config: dict = None, default_args: dict = None, module_type: str = 'dataset_sampler', class_name: str = None, **kwargs): r"""Build sampler For MindFormer. Instantiate the sampler from MindFormerRegister's registry. Args: config (dict): The task sampler's config. Default: None. default_args (dict): The default argument of sampler API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'sampler'. class_name (str): The class name of sampler API. Default: None. Return: The function instance of sampler API. Examples: >>> from mindformers import build_sampler >>> sampler_config = {'type': 'RandomSampler', 'replacement': False} >>> # 1) use config dict to build sampler >>> sampler_from_config = build_sampler(sampler_config) >>> # 2) use class name to build sampler >>> sampler_class_name = build_sampler(class_name='RandomSampler', replacement=False) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.DATASET_SAMPLER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_samplers(): """ register MindSpore builtin transforms class. """ for module_name in dir(sp): if module_name.startswith('__'): continue samplers = getattr(sp, module_name) if inspect.isclass(samplers): MindFormerRegister.register_cls(samplers, MindFormerModuleType.DATASET_SAMPLER) register_ms_samplers() pengcheng.mind-mindformer/mindformers/dataset/text_classification_dataset.py000066400000000000000000000110521476650141400302010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Dataset.""" import os import copy import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class TextClassificationDataset(BaseDataset): """ Text Classification dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TextClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['text_classification']['txtcls_bert_base_uncased'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/text_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TextClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TextClassificationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Text Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) if "data_files" not in dataset_config.data_loader \ and dataset_config.data_loader.dataset_dir: dataset_files = [] data_dir = dataset_config.data_loader.dataset_dir if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord") or file.endswith(".tfrecord"): dataset_files.append(os.path.join(r, file)) else: if data_dir.endswith(".mindrecord") or data_dir.endswith(".tfrecord"): dataset_files.append(data_dir) else: dataset_files = list(dataset_config.data_loader.dataset_files) dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id}) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, output_columns=dataset_config.input_columns, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.project(columns=dataset_config.input_columns) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, operations=type_cast_op, input_columns=input_arg) return dataset pengcheng.mind-mindformer/mindformers/dataset/token_classification_dataset.py000066400000000000000000000113271476650141400303420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token classification Dataset.""" import os from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from ..models.build_tokenizer import build_tokenizer from .transforms import build_transforms from .sampler import build_sampler from .base_dataset import BaseDataset @MindFormerRegister.register(MindFormerModuleType.DATASET) class TokenClassificationDataset(BaseDataset): """ Token classification Dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TokenClassificationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['token_classification']['tokcls_bert_base_chinese'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/task_cards/token_classification.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TokenClassificationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TokenClassificationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): """new method""" logger.info("Now Create Token classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) label_transforms = build_transforms(dataset_config.label_transforms) sampler = build_sampler(dataset_config.sampler) if sampler is not None: dataset = dataset.use_sampler(sampler) if text_transforms is not None: dataset = get_dataset_map(dataset, input_columns=dataset_config.input_columns, operations=text_transforms, output_columns=dataset_config.output_columns, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if label_transforms is not None: dataset = get_dataset_map(dataset, input_columns=dataset_config.input_columns[1], operations=label_transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-mindformer/mindformers/dataset/transforms/000077500000000000000000000000001476650141400242625ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/transforms/__init__.py000066400000000000000000000022501476650141400263720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Transforms API.""" from .build_transforms import build_transforms from .vision_transforms import * from .text_transforms import * from .mixup import Mixup from .auto_augment import rand_augment_transform, auto_augment_transform, augment_and_mix_transform from .random_erasing import RandomErasing __all__ = ['Mixup', 'rand_augment_transform', 'auto_augment_transform', 'augment_and_mix_transform', 'RandomErasing'] __all__.extend(vision_transforms.__all__) __all__.extend(text_transforms.__all__) pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/000077500000000000000000000000001476650141400264725ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011461476650141400326640ustar00rootroot00000000000000a ?cf@spdZddlmZddlTddlTddlmZddlmZmZm Z ddl m Z gdZ e ej e ej dS) zMindFormers Transforms API.)build_transforms)*)Mixup)rand_augment_transformauto_augment_transformaugment_and_mix_transform) RandomErasing)rrrrrN)__doc__rZvision_transformsZtext_transformsZmixuprZ auto_augmentrrrZrandom_erasingr__all__extendr r R/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/__init__.pys    pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/auto_augment.cpython-39.pyc000066400000000000000000000627011476650141400336210ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlZddlZddlZddlmZmZm Z ddl m Z ddl m Z mZeddejddd DZd Zd Zed ed ZejejfZdd ddZddZddZddZGdddZGdddZeZ ddde j!e j"e j#e j$e j%e j&e j'e j(e j)e j(e j)e j(e j)e j(e j)e j*e j*e j+e j+e j,e j,dZ-eZ.e.j/e.j0e.j1e.j2e.j3e.j3e.j3e.j4e.j4e.j5e.j6e.j6e.j7e.j7e.j8e.j8e.j9e.j9e.j:e.j;e.je.j?dZ@GdddZAddZBdd ZCd!d"ZDd#d$ZEdDd&d'ZFGd(d)d)ZGe HejId*d+ZJgd,ZKgd-ZLd.d/d/d0d0d1d1d1d2d2d2d2d2ddd3ZMdEd4d5ZNdFd7d8ZOGd9d:d:e ZPe HejId;d<ZQgd=ZRdGd>d?ZSGd@dAdAZTe HejIdBdCZUdS)Hz2AutoAugment, RandAugment, and AugMix for MindSporeN)ImageImageOps ImageEnhance)PyTensorOperation)MindFormerRegisterMindFormerModuleTypecCsg|] }t|qS)int).0xrrV/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/auto_augment.py r .)rrg$@)translate_constimg_mean)nearestlinearcubiccCsF|dkrtj}n.|dkr(tj}n|dkr.RGB)Lri)rangemodelenpoint)r;r<rgrbr]lutr!rrfr solarize_adds  zPolicyHelper.solarize_addcKs|dkr |n t||}|S)z posterize of auto_augment policy)r posterize)r;r<Z bits_to_keepr]r!rrr rrszPolicyHelper.posterizecKst||S)zcontrast of auto_augment policy)rContrastenhancer;r<r=r]rrr contrastszPolicyHelper.contrastcKst||S)zcolor of auto_augment policy)rColorrtrurrr colorszPolicyHelper.colorcKst||S)z!brightness of auto_augment policy)r Brightnessrtrurrr brightnessszPolicyHelper.brightnesscKst||S)z sharpness of auto_augment policy)r Sharpnessrtrurrr sharpnessszPolicyHelper.sharpnessN)r)__name__ __module__ __qualname____doc__r>r?rBrCrFrGrJr^r_r`rarprrrvrxrzr|rrrr r4Ls$ r4c@sxeZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZddZddZdS) LevelHelperz Level helper cCstdkr| S|S)zWith 50% prob, negate the value?)r()r;vrrr _randomly_negateszLevelHelper._randomly_negatecCs||td}|fS)z&_randomly_negate of auto_augment levelg>@r _MAX_LEVELr;level_rrr rotate_level_to_argszLevelHelper.rotate_level_to_argcCs|tddfS)z*enhance_level_to_arg of auto_augment levelg?皙?)rrrrr enhance_level_to_argsz LevelHelper.enhance_level_to_argcCsd||td}|fS)z5enhance_increasing_level_to_arg of auto_augment level?g?rrrrr enhance_increasing_level_to_argsz+LevelHelper.enhance_increasing_level_to_argcCs||td}|fS)z(shear_level_to_arg of auto_augment level333333?rrrrr shear_level_to_argszLevelHelper.shear_level_to_argcCs$|d}||tt|}|fS)z0translate_abs_level_to_arg of auto_augment levelr)rrfloat)r;rhparamsrrrr translate_abs_level_to_argsz&LevelHelper.translate_abs_level_to_argcCs$|dd}||t|}|fS)z0translate_rel_level_to_arg of auto_augment level translate_pctg?)r+rr)r;rrrrrr translate_rel_level_to_args z&LevelHelper.translate_rel_level_to_argcCst|tdfS)z,posterize_level_to_arg of auto_augment levelr rrrrr posterize_level_to_argsz"LevelHelper.posterize_level_to_argcCsd|||dfS)z7posterize_increasing_level_to_arg of auto_augment levelrr)rr;rrrrr !posterize_increasing_level_to_argsz-LevelHelper.posterize_increasing_level_to_argcCst|tddfS)z5posterize_original_level_to_arg of auto_augment levelrrrrrr posterize_original_level_to_argsz+LevelHelper.posterize_original_level_to_argcCst|tdfS)z+solarize_level_to_arg of auto_augment levelrhrrrrr solarize_level_to_argsz!LevelHelper.solarize_level_to_argcCsd|||dfS)z6solarize_increasing_level_to_arg of auto_augment levelrhr)rrrrr solarize_increasing_level_to_argsz,LevelHelper.solarize_increasing_level_to_argcCst|tdfS)z/solarize_add_level_to_arg of auto_augment levelnrrrrr solarize_add_level_to_argsz%LevelHelper.solarize_add_level_to_argN)r}r~rrrrrrrrrrrrrrrrrrr rsr) AutoContrastEqualizeInvertRotate PosterizePosterizeIncreasingPosterizeOriginalSolarizeSolarizeIncreasing SolarizeAddrwColorIncreasingrsContrastIncreasingryBrightnessIncreasingr{SharpnessIncreasingShearXShearYZ TranslateXZ TranslateY TranslateXRel TranslateYRelc@s"eZdZdZdddZddZdS) AugmentOpr NcCsx|pt}t||_t||_||_||_||_t d|vrHt |dnt d|vrZ|dnt d|_ |jdd|_dS)Nrr-)r/r# magnitude_stdr)_HPARAMS_DEFAULT NAME_TO_OPaug_fn LEVEL_TO_ARGlevel_fnprob magnitudecopyrdictr'_FILL_RANDOM_INTERPOLATIONr,r+r)r;namerrrrrr __init__Fs    zAugmentOp.__init__cCs|jdkrt|jkr|S|j}|jr\|jtdkrDtd|}n|jdkr\t||j}ttt d|}|j dur| ||j }nt }|j |g|Ri|jS)z apply augmentrinfrN)rr(rrruniformgaussrdrmaxrrr'rr,)r;r<rZ level_argsrrr __call__Ys  zAugmentOp.__call__)rrNr}r~rrrrrrrr rCs rcsddgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gdd.gd/d0gg}fd1d2|D}|S)3auto_augment_policy_v0r皙?r5rrrrw皙? r333333?rrwrr5rrrqrrrrrrrrrrrrw皙?rrrrqrrrqrrrrrrrwrr5rrrrrrrrrrrrrrrrwrrrrr)rrrrrrrrqrwrrrrrrrrrrrrrrrrIrrrrrrrrwrrrwrrrrrrrrrrIrqrrrrrrrrrrrr)rrrrrrrrr5rwrrrrr1csg|]}fdd|DqS)csg|]}t|diqSrrr rUrrr r rz5auto_augment_policy_v0...rr sprrr r rz*auto_augment_policy_v0..rrpolicypcrrr rks8rcsddgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gdd.gd/d0gg}fd1d2|D}|S)3auto_augment_policy_v0rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr)rrrrrrrcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz6auto_augment_policy_v0r...rrrrr r rz+auto_augment_policy_v0r..rrrrr rs8rcsddgddgddgddgd d gd d gd dgddgddgddgd dgddgddgddgddgd dgddgd d!gd"d#gddgd d gddgddgddgddgg}fd$d%|D}|S)&auto_augment_policy_original)rrrqrrrrrr1rrr1rr)rrr)rrrrrrrrrrrqrrrrrr)rrr1rrrrrrrrq)rrrrwrrrrrrrrrrIrrrrrrrqrrsrrqrwrrrwrrqrrrr{rrrrrqrrr1rcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz;auto_augment_policy_original...rrrrr r rz0auto_augment_policy_original..rrrrr rs8rcsddgddgddgddgd d gd d gd dgddgddgddgd dgddgddgddgddgd dgddgd d!gd"d#gddgd d gddgddgddgddgg}fd$d%|D}|S)&auto_augment_policy_originalr)rrrqr r r rr)rrr)rrrrrr r rr)rrr1rrrrrrrrrrrrrrrrrrrrcsg|]}fdd|DqS)csg|]}t|diqSrrrrrr r rz..rrrrr r rz1auto_augment_policy_originalr..rrrrr rs8rv0cCsb|pt}|dkrt|}nD|dkr,t|}n2|dkr>t|}n |dkrPt|}ntd||S)auto_augment_policyoriginalZ originalrr Zv0rzUnknown AA policy )rrrrr ValueError)rrr!rrr r!s    r!c@s eZdZdZddZddZdS) AutoAugmentcCs ||_dSN)r)r;rrrr rszAutoAugment.__init__cCs"t|j}|D] }||}q|S)zapply autoaugment)r(r)r)r;r<Z sub_policyoprrr r s  zAutoAugment.__call__Nrrrrr r$sr$cCs|d}|d}|dd}|D]N}td|}t|dkr"|dd\}}|dkrh|dt|q"td q"tt||d S) a_ Create a AutoAugment transform :param config_str: String defining configuration of auto augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the AutoAugment policy (one of 'v0', 'v0r', 'original', 'originalr'). The remaining sections, not order specific determine 'mstd' - float std deviation of magnitude noise applied Ex 'original-mstd0.5' results in AutoAugment with original policy, magnitude_std 0.5 :param hparams: Other hparams (kwargs) for the AutoAugmentation scheme :return: A MindSpore compatible Transform -rr5N(\d.*)rmstdrz"Unknown AutoAugment config sectionr)splitrerm setdefaultrr#r$r!) config_strrconfigZ policy_namerWcskeyvalrrr auto_augment_transforms     r2)rrrrrrrrwrsryr{rrrr)rrrrrrrrrrrrrrrrrrg?g{Gzt?)rrrrrrwr{rrrrsryrrrcsF|pt}|dkrtd|tfdd|D}|t|}|S)_select_rand_weightsrz2only one set of weights currently, get weight_idx csg|] }|qSrr)r kZ rand_weightsrr r prz(_select_rand_weights..)_RAND_TRANSFORMSr#_RAND_CHOICE_WEIGHTS_0npsum) weight_idx transformsprobsrr5r r3jsr3rcs$pt|pt}fdd|DS)rand_augment_opscsg|]}t|ddqS)rrrrrr rrrrr r ysz$rand_augment_ops..)rr6rrr;rr@r r=us  r=cs*eZdZdZdfdd ZddZZS) RandAugmentrNcs$tt|||_||_||_dSr%)superrBrops num_layerschoice_weights)r;rDrErF __class__rr rszRandAugment.__init__cCs6tjj|j|j|jdu|jd}|D] }||}q$|S)N)replacep)r8r(r)rDrErF)r;r<rDr&rrr rs  zRandAugment.__call__)rN)r}r~rrrr __classcell__rrrGr rB}srBc Cst}d}d}t}|d}|ddkr8td|d|dd}|D]}td|}t|dkrH|dd\} } | d kr|d t| qH| d krt| rt }qH| d krt | }qH| d krt | }qH| dkrt | }qHtdqHt |||d} |durdnt |} t | || dS)a Create a RandAugment transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude of rand augment 'n' - integer num layers (number of transform ops selected per image) 'w' - integer probabiliy weight index (index of a set of weights to influence choice of op) 'mstd' - float std deviation of magnitude noise applied 'inc' - integer (bool), use augmentations that increase in severity with magnitude (default: 0) Ex 'rand-m9-n3-mstd0.5' results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5 'rand-mstd1-w0' results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2 :param hparams: Other hparams (kwargs) for the RandAugmentation scheme :return: A MindSpore compatible Transform rNr'rrandzconfig[0] is not rand, get r5r(r)rincmnwz"Unknown RandAugment config sectionrA)rF)rr6r*r#r+rmr,rbool_RAND_INCREASING_TRANSFORMSr r=r3rB) r-rrrEr:r;r.rWr/r0r1Zra_opsrFrrr rand_augment_transforms6         rS) rrrrrrrrrrrrrcs$pt|pt}fdd|DS) augmix_opscsg|]}t|ddqS)rr>rr?r@rr r szaugmix_ops..)r_AUGMIX_TRANSFORMSrArr@r rTs  rTc@s:eZdZdZdddZdd Zd d Zd d ZddZdS) AugMixAugmentz AugMix Transform Adapted and improved from impl here From paper: 'AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty - https://arxiv.org/abs/1912.02781 rrFcCs"||_||_||_||_||_dSr%)rDalphawidthdepthblended)r;rDrXrYrZr[rrr rs zAugMixAugment.__init__cCs\||}d}g}|dddD]"}||}|d|9}||qtj|dddtjdS)z&_calc_blended_weights of AugMixAugmentrNrWr5dtype)appendr8arrayfloat32)r;wsrNZcumpZrwsrPrXrrr _calc_blended_weightss  z#AugMixAugment._calc_blended_weightsc Csx|}|||}|D]Z}|jdkr,|jn tjdd}tjj|j|dd}|} |D] } | | } qVt || |}q|S)z_apply_blended of AugMixAugmentrr5rTrI) rrbrZr8r(randintr)rDrblend) r;r<mixing_weightsrNZimg_origrarPrZrDimg_augr&rrr _apply_blendeds  zAugMixAugment._apply_blendedc Cs|jd|jdt|f}tj|tjd}|D]d}|jdkrF|jn tjdd}tjj |j |dd}|} |D] } | | } qp||tj | tjd7}q2tj |dd|dt |tj}t |||S) z_apply_basic of AugMixAugmentrr5r\rTrcgo@)out)r8rmgetbandsr8zerosr`rZr(rdr)rDasarrayclipr fromarrayastypeuint8re) r;r<rfrN img_shapemixedmwrZrDrgr&rrr _apply_basics zAugMixAugment._apply_basiccCs\ttj|jg|j}ttj|j|j}|jrJ||||}n| |||}|S)zAugMixAugment apply) r8r`r( dirichletrXrYbetar[rhrt)r;r<rfrNrrrrr rs zAugMixAugment.__call__N)rrrWF) r}r~rrrrbrhrtrrrrr rVs   rVc Cs d}d}d}d}d}td|d<|d}|dd krHtd |d|d d }|D]}td |} t| dkrX| d d\} } | dkr|dt| qX| dkrt| }qX| dkrt| }qX| dkrt| }qX| dkrt| }qX| dkrt| }qXtdqXt||d} t | ||||dS)a Create AugMix MindSpore transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude (severity) of augmentation mix (default: 3) 'w' - integer width of augmentation chain (default: 3) 'd' - integer depth of augmentation chain (-1 is random [1, 3], default: -1) 'b' - integer (bool), blend each branch of chain into end result without a final blend, less CPU (default: 0) 'mstd' - float std deviation of magnitude noise applied (default: 0) Ex 'augmix-m5-w4-d2' results in AugMix with severity 5, chain width 4, chain depth 2 :param hparams: Other hparams (kwargs) for the Augmentation transforms :return: A MindSpore compatible Transform rrWrFrrr'rZaugmixzconfig[0] is not augmix, get r5Nr(rr)rNrPrXrUrVzUnknown AugMix config section)rr)rXrYrZr[) rr*r#r+rmr,r rQrTrV) r-rrrYrZrXr[r.rWr/r0r1rDrrr augment_and_mix_transform#s:             rw)r N)rN)rNN)rNN)VrrKr(r+numpyr8PILrrrZ#mindspore.dataset.vision.transformsrmindformers.tools.registerrrr' __version__r*r2rrrrrrrr*r"r.r3r4rlhrrrrrrrrrrrrrphr^r`r_rJrrrarprxrvrzr|r>r?rFrGrBrCrrrrrrr!r$register TRANSFORMSr2r6rRr7r3r=rBrSrUrTrVrwrrrr s "  hS("#!"      3 G pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/build_transforms.cpython-39.pyc000066400000000000000000000044321476650141400345030ustar00rootroot00000000000000a ?cf$ @sddZddlZddlmZddlmZddlmZm Z m Z d e e ge e e ddd Zd d ZedS) zBuild Transforms API.N) transforms)vision)MindFormerRegisterMindFormerModuleTypeMindFormerConfigr)config default_args module_type class_namec Ks|dur|durdS|durt|tr>t|ts>tfi|}|}t|ts^tj|tj|dSg}|D]}t|tj}||qf|Stj ||fi|S)a~Build transform For MindFormer. Instantiate the transform from MindFormerRegister's registry. Args: config (dict, list): The task transform's config. Default: None. default_args (dict): The default argument of transform API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'transform'. class_name (str): The class name of transform API. Default: None. Return: The function instance of transform API. Examples: >>> from mindformers import build_transforms >>> transform_config = [{'type': 'Decode'}, {'type': 'Resize', 'size': 256}] >>> # 1) use config dict to build transform >>> transform_from_config = build_transforms(transform_config) >>> # 2) use class name to build transform >>> transform_class_name = build_transforms(class_name='Decode') N)r) isinstancedictrlistrget_instance_from_cfgr TRANSFORMSappend get_instance) rrr r kwargsZcfg_transformsr transformZ transform_oprZ/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/build_transforms.pybuild_transformss$  rcCsntttjttjD]P}|dr(qttj|drDttj|dn ttj|}t|rt |t j qdS)z,register MindSpore builtin transforms class.__N) setdirvsrtf startswithgetattrinspectisclassr register_clsrr) module_namerrrrregister_ms_transformsAs    r")NNrN)__doc__rmindspore.datasetrrrrmindformers.tools.registerrrrr r strrr"rrrrs   )pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/mixup.cpython-39.pyc000066400000000000000000000205621476650141400322720ustar00rootroot00000000000000a ?cf,@sdZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd d Zdd dZdddZdddZdddZeejGddde ZdS)zMixup and Cutmix for MindSporeN)Tensordtype)ops)PyTensorOperation)logger)MindFormerRegisterMindFormerModuleType?cCs0|d}t||}tj|||tjd}|S)zone hot to label)a_mina_maxr)reshapenpeyeclipfloat32)x num_classeson_value off_valuerO/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/mixup.pyone_hots rcCsR||}d||}t||||d}ttj|dd|||d}|||d|S) mixup_targetr )rrraxis)rrflip)targetrlam smoothingrry1y2rrrr!s  rcCstd|}|dd\}}t||t||}}t||t||} } tjjd| || |d} tjjd| || |d} t| |dd|} t| |dd|}t| |dd|}t| |dd|}| |||fS)a Standard CutMix bounding-box Generates a random square bbox based on lambda value. This impl includes support for enforcing a border margin as percent of bbox dimensions. Args: img_shape (tuple): Image shape as tuple lam (float): Cutmix lambda value margin (float): Percentage of bbox dimension to enforce as margin (reduce amount of box outside image) count (int): Number of bbox to generate Nrsize)rsqrtintrandomrandintr) img_shaper margincountratioimg_himg_wcut_hcut_wZmargin_yZmargin_xcycxylyhxlxhrrr rand_bbox*s r;c Cst|dkrtd|dd\}}tjjt||dt||d|d}tjjt||dt||d|d}tjjd|||d}tjjd|||d}||} ||} || || fS)a Min-Max CutMix bounding-box Inspired by Darknet cutmix impl, generates a random rectangular bbox based on min/max percent values applied to each dimension of the input image. Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max. Args: img_shape (tuple): Image shape as tuple minmax (tuple or list): Min and max bbox ratios (as percent of image size) count (int): Number of bbox to generate r(zlen(minmax) is not 2. r%Nrr$r&)len ValueErrorrr+r,r*) r-minmaxr/r1r2r3r4r7r9yuxurrrrand_bbox_minmaxBs **rATc Cs~|dur t|||d\}}}}nt|||d\}}}}|sB|durn||||} d| t|d|d}||||f|fS)z0 Generate bbox and apply lambda correction. N)r/r r%r )rAr;float) r-r ratio_minmax correct_lamr/r7r?r9r@Z bbox_arearrrcutmix_bbox_and_lamZs rEc sReZdZdZdfd d Zd d ZddZddZddZddZ ddZ Z S)Mixupas Mixup/Cutmix that applies different params to each element or whole batch Args: mixup_alpha (float): mixup alpha value, mixup is active if > 0. cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0. cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None. prob (float): probability of applying mixup or cutmix per batch or element switch_prob (float): probability of switching to cutmix instead of mixup when both are active mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element) correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders label_smoothing (float): apply label smoothing to the mixed target tensor num_classes (int): number of classes for target r r N?batchT皙?c sttt|||_||_||_|jdurFt|jdkr@tdd|_||_||_ ||_ | |_ ||_ ||_ d|_dS)Nr(zlen(cutmix_minmax) is not 2.r T)superrF__init__ mixup_alpha cutmix_alpha cutmix_minmaxr<r=mix_prob switch_problabel_smoothingrmoderD mixup_enabled) selfrMrNrOprobrQrSrDrRr __class__rrrLws zMixup.__init__c Cs tj|tjd}tj|tjd}|jr|jdkr|jdkrtj ||j k}t |tjj |j|j|dtjj |j|j|d}n\|jdkrtjj |j|j|d}n:|jdkrtj|tjd}tjj |j|j|d}nt dt tj ||jk|tj|}||fS)_params_per_elemrr r&ROne of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.)ronesrzerosboolrTrMrNr+randrQwherebetar=rPastype)rU batch_sizer use_cutmixlam_mixrrrrYs$  $zMixup._params_per_elemcCsd}d}|jrtj|jkr|jdkrl|jdkrltj|jk}|rXtj|j|jntj|j|j}nH|jdkrtj|j|j}n*|jdkrd}tj|j|j}nt dt |}||fS)_params_per_batchr Fr TrZ) rTrr+r^rPrMrNrQr`r=rB)rUr rcrdrrrres   zMixup._params_per_batchc Cst|}||\}}|}t|D]}||d}||}|dkr&||rt||j||j|jd\\} } } } }||dd| | | | f||dd| | | | f<|||<q&|||||d|||<q&t t |t j ddS) _mix_elemr$r rCrDNr) r<rYclonerangerEshaperOrDP ExpandDimsrmstyper rUrrbZ lam_batchrcZx_origijr r7r8r9r:rrrrfs  8 "zMixup._mix_elemc Csjt|}||d\}}|}t|dD]}||d}||}|dkr.||rt||j||j|jd\\} } } } }||dd| | | | f||dd| | | | f<||dd| | | | f||dd| | | | f<|||<q.|||||d|||<|||||d|||<q.t ||dddf}t t |t jddS) _mix_pairr(r$r rgNr r)r<rYrhrirErjrOrDr concatenaterkrlrrmrrnrrrrqs$ 88  "zMixup._mix_pairc Cs|\}}|dkrdS|rt|j||j|jd\\}}}}}tj|dddddd||||f|dddd||||f<n&tj|ddd|}||9}||7}|S) _mix_batchr rgrrN)rerErjrOrDrr) rUrr rcr7r8r9r:Z x_flippedrrrrss HzMixup._mix_batchcCst|ddkr>t|dkr4|dd}tdn td|jdkrT||}n |jd krj||}n ||}t||j ||j }| t j | t j fS) z Mixup applyr(rr$Nr zBatch size is odd. When using mixup, batch size should be even.The last data in batch has been dropped to use mixip.you can set "drop_remainder" true in dataset config manually.zTBatch size is 1.If error occurs, please set "drop_remainder" true in dataset config.elempair)r<rwarningerrorrSrfrqrsrrrRrarr)rUrrr rrr__call__s         zMixup.__call__) r r Nr rGrHTrIrJ) __name__ __module__ __qualname____doc__rLrYrerfrqrsrx __classcell__rrrWrrFgsrF)r r )r r )r N)N)NTN)r|numpyr mindsporerrrmrrkZ#mindspore.dataset.vision.transformsrmindformers.tools.loggerrmindformers.tools.registerrr rrr;rArEregister TRANSFORMSrFrrrr s         pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/random_erasing.cpython-39.pyc000066400000000000000000000074201476650141400341160ustar00rootroot00000000000000a ?cf@sfdZddlZddlZddlZddlmZddlmZm Z ej fddZ e e j GdddeZdS) z%Random Erasing (Cutout) for MindSporeN)PyTensorOperation)MindFormerRegisterMindFormerModuleTypecCsZ|rtjj|d|}n<|r>tjj|dddfd|}ntj|dddf|d}|S) _get_pixels)sizerdtype)nprandomnormalastypezeros) per_pixel rand_color patch_sizer funcrX/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/random_erasing.pyrs  rc s2eZdZdZdfd d Zd d ZddZZS) RandomErasingaY Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https://arxiv.org/pdf/1708.04896.pdf This variant of RandomErasing is intended to be applied to either a batch or single image tensor after it has been normalized by dataset mean and std. Args: probability: Probability that the Random Erasing operation will be performed. min_area: Minimum percentage of erased area wrt input image area. max_area: Maximum percentage of erased area wrt input image area. min_aspect: Minimum aspect ratio of erased area. mode: pixel color mode, one of 'const', 'rand', or 'pixel' 'const' - erase block is constant color of 0 for all channels 'rand' - erase block is same per-channel random (normal) color 'pixel' - erase block is per-pixel random (normal) color max_count: maximum number of erasing blocks per image, area per box is scaled by count. per-image count is randomly chosen between 1 and this value. ?{Gz?UUUUUU?333333?Nconstrrc stt|||_||_||_|p*d|}t|t|f|_||_ |pN||_ | |_ | }d|_ d|_|dkr|d|_ n$|dkrd|_n|r|dkrtddS)NrFrandTpixelrz mode is const)superr__init__ probabilitymin_areamax_areamathloglog_aspect_ratio min_count max_count num_splitslowerrr ValueError) selfrr r!Z min_aspectZ max_aspectmoder%r&r' __class__rrr4s$   zRandomErasing.__init__c Cst|jkrn||}|j|jkr,|jnt|j|j}t|D]}tdD]}t|j|j||} t tj|j } t t t | | } t t t | | } | |krP| |krPtd|| } td|| }t|j|j|| | f|d|dd| | | ||| f<qDqPqD|S)_erase rrN)r rr%r&randintrangeuniformr r!r"expr$introundsqrtrrr)r*imgchanimg_himg_wr areacount_ target_area aspect_ratiohwtopleftrrrr.Js*  $zRandomErasing._erasec Cst|jdkr*|j|g|j|jR}n\t|}|j\}}}}|jdkrV||jnd}t||D] }|||||||j||<qd|S)zRandomErasing applyrr)lenshaper.r r zeros_liker'r1) r*xoutput batch_sizer8r9r: batch_startirrr__call__as zRandomErasing.__call__) rrrrNrrNr)__name__ __module__ __qualname____doc__rr.rM __classcell__rrr,rrs r)rQr"r numpyr Z#mindspore.dataset.vision.transformsrmindformers.tools.registerrrfloat32rregister TRANSFORMSrrrrr s  pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__/text_transforms.cpython-39.pyc000066400000000000000000000116741476650141400343760ustar00rootroot00000000000000a ?cfW@sdZddlZddlZddlmZmZgdZeej GdddZ eej Gdd d Z eej Gd d d Z eej Gd d d Z eej GdddZdS)zText Transforms.N)MindFormerRegisterMindFormerModuleType)RandomChoiceTokenizerForwardTokenizerForwardTokenizeWithLabel LabelPaddingCaptionTransformc@s"eZdZdZd ddZddZd S) rzRandom Choice Tokenizer ForwardM max_lengthcCs||_||_||_||_dSN)r padding tokenizer random_seed)selfrr rrrY/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/text_transforms.py__init__sz%RandomChoiceTokenizerForward.__init__cCsd|}tj|jtjt|}|jt||t rH|| ddn|||j |j dd}|S)Nutf-8ignorer r input_ids) tolistnprandomseedrchoicelenr isinstancebytesdecoder r)rtextZ text_listindextoken_idrrr__call__%s z%RandomChoiceTokenizerForward.__call__N)r r r __name__ __module__ __qualname____doc__rr%rrrrrs rc@s"eZdZdZd ddZddZdS) rzTokenizer Forwardr r cCs||_||_||_dSr rr rrrr rrrrr5szTokenizerForward.__init__cCsht|tjr|n|}tt|D]&}t||tr$||dd||<q$|j||j |j dd}|S) call methodrrrr) rrndarrayrrangerr r!rr r)rr"ir$rrrr%:szTokenizerForward.__call__N)r r r&rrrrr2s rc@s"eZdZdZd ddZddZdS) rzTokenizer With Labelr cCs||_||_||_dSr r+r,rrrrHszTokenizeWithLabel.__init__cGs|d}|d}t|tjr$|n|}t|tr>|dd}|j||j|jd}tj |dtj d}tj |dtj d}tj |d tj d}||||fS) r-rrrrrdtypetoken_type_idsattention_mask) rrr.rr r!rr rarrayint32)rinputsr"label_idoutputrr5r6rrrr%Ms  zTokenizeWithLabel.__call__N)r1r r&rrrrrEs rc@s"eZdZdZdddZddZdS) rz Label PaddingrcCs||_||_dSr )r padding_value)rr r<rrrr`szLabelPadding.__init__cCst||jdkr$|d|jd}g}||jtt|D]}|||q@t||jkrp||jqTtj|tjd}|S)r-r2Nr3)rr appendr<r/rr7r8)rr:Z pad_label_idr0rrrr%ds zLabelPadding.__call__N)rr&rrrrr]s rc@s*eZdZdZddd Zd d Zd d ZdS)r z\ Caption Transform, preprocess captions and tokenize it, align with torch impl. 2 r r Tc Cs4||_||_||_||_||_||_||_||_dSr )rprompt max_wordsr rr truncationadd_special_tokens) rrrArBr rrrCrDrrrr}szCaptionTransform.__init__cCsH|jdkr:g}t|jdD]}||||q|S||}|S)Nr2r)ndimr/shaper= pre_caption)rcaptionZ caption_listr0rrrrr%s  zCaptionTransform.__call__cCst|}|j|}tdd|}tdd|}|d}|d}|d}t||j krrd |d|j }|j ||j |j |j|jd}tj|dtjd}|S) z Caption preprocessing removes any punctuationmarks except commas, tailing spaces and transform sentence into lower case. z([.!\"()*#:;~]) z\s{2,} N)r rrCrDrr3)strrAresublowerrstripstripsplitrrBjoinrr rrCrDrr7r8)rrHZ caption_wordsr;rrrrrGs.     zCaptionTransform.pre_captionN)r>r?r@r r TT)r'r(r)r*rr%rGrrrrr ws   r )r*rLnumpyrZtools.registerrr__all__register TRANSFORMSrrrrr rrrrs     vision_transforms.cpython-39.pyc000066400000000000000000000261231476650141400346350ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/dataset/transforms/__pycache__a ?cf/@sBdZddlZddlmZddlmZddlZddl m Z ddl m Z m Z gdZe je je je je jdZGd d d ZGd d d ZGd ddZGdddZGdddZGdddZe e jGdddejjZe e jGdddejjZe e jGdddejj Z e e jGdddejj!Z!dS)z% transform methods for vision models N)Image)vision)Inter)MindFormerRegisterMindFormerModuleType) BatchResize BCHW2BHWC BatchPILizeBatchNormalizeBatchCenterCrop BatchToTensorRandomCropDecodeResizeRandomResizedCropResizeRandomHorizontalFlip)nearest antialiaslinearcubicbicubicc@seZdZdZddZdS)ra Transform a batch of image from CHW to HWC. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the channel should be (bz, c, h, w) or (c, h, w). for list, the item should be PIL.Image or numpy.array (c, h, w). Return: transformed image batch: for numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for PIL.Image input, it is returned directly. cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrZ| ddddSt|jdkrv| dddSt dt|jt|t j r|St d t |d d S) zthe call functioncsg|] }|qSr.0itemselfr[/home/ma-user/work/mindformers-baichuan/mindformers/dataset/transforms/vision_transforms.py >z&BCHW2BHWC.__call__..r2the rank of image_batch should be 3 or 4, but got the type  of image_batch is unsupported.N) isinstancemsTensorasnumpylistnpndarraylenshape transpose ValueErrorr TypeErrortyper image_batchrrr__call__8s    zBCHW2BHWC.__call__N__name__ __module__ __qualname____doc__r5rrrrr*s rc@s"eZdZdZdddZddZdS) rz Resize a batch of image to the given shape. Args: image_resolution (int): the target size. interpolation (str): interpolate method, default is 'cubic'. rcCs t||_t||j|_dSN) INTERPOLATIONget interpolationrrresize)rimage_resolutionr>rrr__init__Ts zBatchResize.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) a The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: resized image batch: for numpy or tensor input, return a numpy array; for PIL.Image input, it returns PIL.Image. csg|]}|qSr)r?rrrrrirz(BatchResize.__call__..rcs$g|]}|tjddfqSr;)r?r+newaxisrrrrrlsr!r#r$r%N)r&r'r(r)r*r+r,r-r. row_stackr?r0rr1r2r3rrrr5Xs"      zBatchResize.__call__N)rr7r8r9r:rAr5rrrrrKs rc@s eZdZdZddZddZdS)r zv CenterCrop a batch of image to the given shape. Args: image_resolution (int): the target size. cCst||_dSr;)r CenterCropcrop)rr@rrrrAszBatchCenterCrop.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) ab The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: center cropped image batch: for numpy or tensor input, return a numpy array, the shape is (bz, image_resolution, image_resolution, c) or (image_resolution, image_resolution, c); for PIL.Image input, it is returned with shape (image_resolution, image_resolution). csg|]}|qSr)rFrrrrrrz,BatchCenterCrop.__call__..rcs$g|]}|tjddfqSr;)rFr+rBrrrrrsr!r#r$r%N)r&r'r(r)r*r+r,r-r.rCrFr0rr1r2r3rrrr5s"      zBatchCenterCrop.__call__NrDrrrrr wsr c@s eZdZdZddZddZdS)r z9Transform a batch of image to tensor and scale to (0, 1).cCstjj|_dSr;)r'datasetrToTensortotensorrrrrrAszBatchToTensor.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrbt fdd|DSt|jdkrz |St dt|jt|t j r |St dt|dd S) a1 The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image or numpy.array. Returns: return a tensor or a list of tensor. csg|]}|qSr)rIrrrrrrz*BatchToTensor.__call__..rcs$g|]}|tjddfqSr;)rIr+rBrrrrrsr!r#r$r%N)r&r'r(r)r*r+r,r-r.rCrIr0rr1r2r3rrrr5s"      zBatchToTensor.__call__NrDrrrrr sr c@s"eZdZdZd ddZddZd S) r zNormalize a batch of image.g3<4'?gwg M?gy{ ?gB91?gwt.?g U?FcCstj|||d|_dS)N)meanstdis_hwc)r Normalize normalize)rrLrMrNrrrrAszBatchNormalize.__init__cst|tjr|}t|tr0fdd|DSt|tjrt|jdkrT |St|jdkrzt fdd|DSt dt|jt dt |dd S) a The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be numpy.array. Returns: return a tensor or a list of tensor. csg|]}|qSr)rPrrrrrrz+BatchNormalize.__call__..r!rcs$g|]}|tjddfqSr;)rPr+rBrrrrrsr#r$r%N)r&r'r(r)r*r+r,r-r.rPrCr0r1r2r3rrrr5s    zBatchNormalize.__call__N)rJrKFrDrrrrr s  r c@seZdZdZddZdS)r z-transform a batch of image to PIL.Image list.cCst|tjr|St|tr<|D]}t|tjstdq|St|tjrP|}t|tjrt |j dkrxdd|DSt |j dkrt | tj Stdt |j tddS) a The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image. Returns: return a tensor or a list of tensor. z_unsupported type in list, when the image_batch is a list, the item in list should be PIL.Image.rcSsg|]}t|tjqSr)r fromarrayastyper+uint8rrrrr rz(BatchPILize.__call__..r!r#zunsupported input type.N)r&rr*r1r'r(r)r+r,r-r.rQrRrSr0)rr4rrrrr5s$      zBatchPILize.__call__Nr6rrrrr sr cs"eZdZdZdfdd ZZS) r !wrapper of RandomCropDecodeResizeg{Gz?g?g?gUUUUUU?r csB||_||_||_t||_||_tt| ||||j|dSr;) sizescaleratior<r=r> max_attemptssuperr rArrXrYrZr>r[ __class__rrrAs  zRandomCropDecodeResize.__init__)rUrVrrWr7r8r9r:rA __classcell__rrr^rr sr cs"eZdZdZdfdd ZZS) rrTrUrVrrWcsB||_||_||_t||_||_tt| ||||j|dSr;) rXrYrZr<r=r>r[r\rrAr]r^rrrA%s  zRandomResizedCrop.__init__)rUrVrrWr`rrr^rr!srcs"eZdZdZdfdd ZZS)rzwrapper of Resizercs0||_t||_d|_tt|||jdS)NF)rXr<r=r>randomr\rrA)rrXr>r^rrrA3s zResize.__init__)rr`rrr^rr/srcs"eZdZdZdfdd ZZS)rzwrapper of RandomHorizontalFlip?cs||_tt||dSr;)probr\rrA)rrdr^rrrA>szRandomHorizontalFlip.__init__)rcr`rrr^rr:sr)"r:numpyr+PILrmindspore.datasetr mindsporer'Zmindspore.dataset.vision.utilsrmindformers.tools.registerrr__all__NEAREST ANTIALIASLINEARPILCUBICBICUBICr<rrr r r r register TRANSFORMS transformsr rrrrrrrs6   !,,#&'    pengcheng.mind-mindformer/mindformers/dataset/transforms/auto_augment.py000066400000000000000000001012011476650141400273170ustar00rootroot00000000000000# AutoAugment, RandAugment, and AugMix for MindSpore # # This code implements the searched ImageNet policies with various tweaks and improvements and # does not include any of the search code. # # AA and RA Implementation adapted from: # https://github.com/tensorflow/tpu/blob/master/models/official/efficientnet/autoaugment.py, Apache 2.0 # # AugMix adapted from: # https://github.com/google-research/augmix, Apache 2.0 # # Hacked together by / Copyright 2020 Ross Wightman # # has been modified to adapt mindspore and mindformers # ============================================================================ """AutoAugment, RandAugment, and AugMix for MindSpore""" import math import random import re import numpy as np import PIL from PIL import Image, ImageOps, ImageEnhance from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.register import MindFormerRegister, MindFormerModuleType _PIL_VER = tuple([int(x) for x in PIL.__version__.split('.')[:2]]) _FILL = (128, 128, 128) # This signifies the max integer that the controller RNN could predict for the # augmentation scheme. _MAX_LEVEL = 10. _HPARAMS_DEFAULT = dict( translate_const=250, img_mean=_FILL, ) _RANDOM_INTERPOLATION = (Image.BILINEAR, Image.BICUBIC) INTERPOLATION = {'nearest': 0, 'linear': 2, 'cubic': 3} def pil_interp(method): """Interpolation method selection""" if method.lower() == 'bicubic': func = Image.BICUBIC elif method.lower() == 'hamming': func = Image.HAMMING elif method.lower() == 'lanczos': func = Image.LANCZOS else: func = Image.BILINEAR return func def _interpolation(kwargs): """_interpolation of auto_augment""" interpolation = kwargs.pop('resample', 'linear') interpolation = random.choice(interpolation) \ if isinstance(interpolation, (list, tuple)) else interpolation return INTERPOLATION.get(interpolation) def _check_args_tf(kwargs): """_check_args_tf of auto_augment""" if 'fillcolor' in kwargs and _PIL_VER < (5, 0): kwargs.pop('fillcolor') kwargs['resample'] = _interpolation(kwargs) class PolicyHelper: """ auto_augment Policy helper """ def shear_x(self, img, factor, **kwargs): """shear_x of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, factor, 0, 0, 1, 0), **kwargs) def shear_y(self, img, factor, **kwargs): """shear_y of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, factor, 1, 0), **kwargs) def translate_x_rel(self, img, pct, **kwargs): """translate_x_rel of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, pct * img.size[0], 0, 1, 0), **kwargs) def translate_y_rel(self, img, pct, **kwargs): """translate_y_rel of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pct * img.size[1]), **kwargs) def translate_x_abs(self, img, pixels, **kwargs): """translate_x_abs of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, pixels, 0, 1, 0), **kwargs) def translate_y_abs(self, img, pixels, **kwargs): """translate_y_abs of auto_augment policy""" _check_args_tf(kwargs) return img.transform(img.size, Image.AFFINE, (1, 0, 0, 0, 1, pixels), **kwargs) def rotate(self, img, degrees, **kwargs): """rotate of auto_augment policy""" _check_args_tf(kwargs) if _PIL_VER >= (5, 2): func = img.rotate(degrees, **kwargs) elif _PIL_VER >= (5, 0): post_trans = (0, 0) rot_center = (img.size[0] / 2.0, img.size[1] / 2.0) angle = -math.radians(degrees) cos_val = round(math.cos(angle), 15) sin_val = round(math.sin(angle), 15) sin_val_neg = round(-math.sin(angle), 15) matrix = [cos_val, sin_val, 0.0, sin_val_neg, cos_val, 0.0] x = -rot_center[0] - post_trans[0] y = -rot_center[1] - post_trans[1] (a, b, c, d, e, f) = matrix matrix[2] = a * x + b * y + c matrix[5] = d * x + e * y + f matrix[2] += rot_center[0] matrix[5] += rot_center[1] func = img.transform(img.size, Image.AFFINE, matrix, **kwargs) else: func = img.rotate(degrees, resample=kwargs['resample']) return func def auto_contrast(self, img, **__): """auto_contrast of auto_augment policy""" return ImageOps.autocontrast(img) def invert(self, img, **__): """invert of auto_augment policy""" return ImageOps.invert(img) def equalize(self, img, **__): """equalize of auto_augment policy""" return ImageOps.equalize(img) def solarize(self, img, thresh, **__): """solarize of auto_augment policy""" return ImageOps.solarize(img, thresh) def solarize_add(self, img, add, thresh=128, **__): """solarize_add of auto_augment policy""" lut = [(min(255, i + add) if i < thresh else i) for i in range(256)] if img.mode == "RGB" and len(lut) == 256: lut = lut + lut + lut func = img.point(lut) if img.mode in ("L", "RGB") else img return func def posterize(self, img, bits_to_keep, **__): """posterize of auto_augment policy""" func = img if bits_to_keep >= 8 else ImageOps.posterize(img, bits_to_keep) return func def contrast(self, img, factor, **__): """contrast of auto_augment policy""" return ImageEnhance.Contrast(img).enhance(factor) def color(self, img, factor, **__): """color of auto_augment policy""" return ImageEnhance.Color(img).enhance(factor) def brightness(self, img, factor, **__): """brightness of auto_augment policy""" return ImageEnhance.Brightness(img).enhance(factor) def sharpness(self, img, factor, **__): """sharpness of auto_augment policy""" return ImageEnhance.Sharpness(img).enhance(factor) class LevelHelper: """Level helper """ def _randomly_negate(self, v): """With 50% prob, negate the value""" return -v if random.random() > 0.5 else v def rotate_level_to_arg(self, level, _): """_randomly_negate of auto_augment level""" # range [-30, 30] level = self._randomly_negate((level / _MAX_LEVEL) * 30.) return (level,) def enhance_level_to_arg(self, level, _): """enhance_level_to_arg of auto_augment level""" # range [0.1, 1.9] return ((level / _MAX_LEVEL) * 1.8 + 0.1,) def enhance_increasing_level_to_arg(self, level, _): """enhance_increasing_level_to_arg of auto_augment level""" # the 'no change' level is 1.0, moving away from that towards 0. or 2.0 increases the enhancement blend # range [0.1, 1.9] level = 1.0 + self._randomly_negate((level / _MAX_LEVEL) * .9) return (level,) def shear_level_to_arg(self, level, _): """shear_level_to_arg of auto_augment level""" # range [-0.3, 0.3] level = self._randomly_negate((level / _MAX_LEVEL) * 0.3) return (level,) def translate_abs_level_to_arg(self, level, hparams): """translate_abs_level_to_arg of auto_augment level""" translate_const = hparams['translate_const'] level = self._randomly_negate((level / _MAX_LEVEL) * float(translate_const)) return (level,) def translate_rel_level_to_arg(self, level, hparams): """translate_rel_level_to_arg of auto_augment level""" # default range [-0.45, 0.45] translate_pct = hparams.get('translate_pct', 0.45) level = self._randomly_negate((level / _MAX_LEVEL) * translate_pct) return (level,) def posterize_level_to_arg(self, level, _): """posterize_level_to_arg of auto_augment level""" # As per Tensorflow TPU EfficientNet impl # range [0, 4], 'keep 0 up to 4 MSB of original image' # intensity/severity of augmentation decreases with level return (int((level / _MAX_LEVEL) * 4),) def posterize_increasing_level_to_arg(self, level, hparams): """posterize_increasing_level_to_arg of auto_augment level""" # As per Tensorflow models research and UDA impl # range [4, 0], 'keep 4 down to 0 MSB of original image', # intensity/severity of augmentation increases with level return (4 - self.posterize_level_to_arg(level, hparams)[0],) def posterize_original_level_to_arg(self, level, _): """posterize_original_level_to_arg of auto_augment level""" # As per original AutoAugment paper description # range [4, 8], 'keep 4 up to 8 MSB of image' # intensity/severity of augmentation decreases with level return (int((level / _MAX_LEVEL) * 4) + 4,) def solarize_level_to_arg(self, level, _): """solarize_level_to_arg of auto_augment level""" # range [0, 256] # intensity/severity of augmentation decreases with return (int((level / _MAX_LEVEL) * 256),) def solarize_increasing_level_to_arg(self, level, hparams): """solarize_increasing_level_to_arg of auto_augment level""" # range [0, 256] # intensity/severity of augmentation increases with level return (256 - self.solarize_level_to_arg(level, hparams)[0],) def solarize_add_level_to_arg(self, level, _): """solarize_add_level_to_arg of auto_augment level""" # range [0, 110] return (int((level / _MAX_LEVEL) * 110),) lh = LevelHelper() LEVEL_TO_ARG = { 'AutoContrast': None, 'Equalize': None, 'Invert': None, 'Rotate': lh.rotate_level_to_arg, # There are several variations of the posterize level scaling in various Tensorflow/Google repositories/papers 'Posterize': lh.posterize_level_to_arg, 'PosterizeIncreasing': lh.posterize_increasing_level_to_arg, 'PosterizeOriginal': lh.posterize_original_level_to_arg, 'Solarize': lh.solarize_level_to_arg, 'SolarizeIncreasing': lh.solarize_increasing_level_to_arg, 'SolarizeAdd': lh.solarize_add_level_to_arg, 'Color': lh.enhance_level_to_arg, 'ColorIncreasing': lh.enhance_increasing_level_to_arg, 'Contrast': lh.enhance_level_to_arg, 'ContrastIncreasing': lh.enhance_increasing_level_to_arg, 'Brightness': lh.enhance_level_to_arg, 'BrightnessIncreasing': lh.enhance_increasing_level_to_arg, 'Sharpness': lh.enhance_level_to_arg, 'SharpnessIncreasing': lh.enhance_increasing_level_to_arg, 'ShearX': lh.shear_level_to_arg, 'ShearY': lh.shear_level_to_arg, 'TranslateX': lh.translate_abs_level_to_arg, 'TranslateY': lh.translate_abs_level_to_arg, 'TranslateXRel': lh.translate_rel_level_to_arg, 'TranslateYRel': lh.translate_rel_level_to_arg, } ph = PolicyHelper() NAME_TO_OP = { 'AutoContrast': ph.auto_contrast, 'Equalize': ph.equalize, 'Invert': ph.invert, 'Rotate': ph.rotate, 'Posterize': ph.posterize, 'PosterizeIncreasing': ph.posterize, 'PosterizeOriginal': ph.posterize, 'Solarize': ph.solarize, 'SolarizeIncreasing': ph.solarize, 'SolarizeAdd': ph.solarize_add, 'Color': ph.color, 'ColorIncreasing': ph.color, 'Contrast': ph.contrast, 'ContrastIncreasing': ph.contrast, 'Brightness': ph.brightness, 'BrightnessIncreasing': ph.brightness, 'Sharpness': ph.sharpness, 'SharpnessIncreasing': ph.sharpness, 'ShearX': ph.shear_x, 'ShearY': ph.shear_y, 'TranslateX': ph.translate_x_abs, 'TranslateY': ph.translate_y_abs, 'TranslateXRel': ph.translate_x_rel, 'TranslateYRel': ph.translate_y_rel, } class AugmentOp: """AugmentOp""" def __init__(self, name, prob=0.5, magnitude=10, hparams=None): hparams = hparams or _HPARAMS_DEFAULT self.aug_fn = NAME_TO_OP[name] self.level_fn = LEVEL_TO_ARG[name] self.prob = prob self.magnitude = magnitude self.hparams = hparams.copy() self.kwargs = dict( fillcolor=tuple(hparams['img_mean']) if 'img_mean' in hparams else _FILL, resample=hparams['interpolation'] if 'interpolation' in hparams else _RANDOM_INTERPOLATION, ) # If magnitude_std is > 0, we introduce some randomness # in the usually fixed policy and sample magnitude from a normal distribution # with mean `magnitude` and std-dev of `magnitude_std`. # NOTE This is my own hack, being tested, not in papers or reference impls. # If magnitude_std is inf, we sample magnitude from a uniform distribution self.magnitude_std = self.hparams.get('magnitude_std', 0) def __call__(self, img): """apply augment""" if self.prob < 1.0 and random.random() > self.prob: return img magnitude = self.magnitude if self.magnitude_std: if self.magnitude_std == float('inf'): magnitude = random.uniform(0, magnitude) elif self.magnitude_std > 0: magnitude = random.gauss(magnitude, self.magnitude_std) magnitude = min(_MAX_LEVEL, max(0, magnitude)) # clip to valid range if self.level_fn is not None: level_args = self.level_fn(magnitude, self.hparams) else: level_args = tuple() return self.aug_fn(img, *level_args, **self.kwargs) def auto_augment_policy_v0(hparams): """auto_augment_policy_v0""" # ImageNet v0 policy from TPU EfficientNet impl, cannot find a paper reference. policy = [ [('Equalize', 0.8, 1), ('ShearY', 0.8, 4)], [('Color', 0.4, 9), ('Equalize', 0.6, 3)], [('Color', 0.4, 1), ('Rotate', 0.6, 8)], [('Solarize', 0.8, 3), ('Equalize', 0.4, 7)], [('Solarize', 0.4, 2), ('Solarize', 0.6, 2)], [('Color', 0.2, 0), ('Equalize', 0.8, 8)], [('Equalize', 0.4, 8), ('SolarizeAdd', 0.8, 3)], [('ShearX', 0.2, 9), ('Rotate', 0.6, 8)], [('Color', 0.6, 1), ('Equalize', 1.0, 2)], [('Invert', 0.4, 9), ('Rotate', 0.6, 0)], [('Equalize', 1.0, 9), ('ShearY', 0.6, 3)], [('Color', 0.4, 7), ('Equalize', 0.6, 0)], [('Posterize', 0.4, 6), ('AutoContrast', 0.4, 7)], [('Solarize', 0.6, 8), ('Color', 0.6, 9)], [('Solarize', 0.2, 4), ('Rotate', 0.8, 9)], [('Rotate', 1.0, 7), ('TranslateYRel', 0.8, 9)], [('ShearX', 0.0, 0), ('Solarize', 0.8, 4)], [('ShearY', 0.8, 0), ('Color', 0.6, 4)], [('Color', 1.0, 0), ('Rotate', 0.6, 2)], [('Equalize', 0.8, 4), ('Equalize', 0.0, 8)], [('Equalize', 1.0, 4), ('AutoContrast', 0.6, 2)], [('ShearY', 0.4, 7), ('SolarizeAdd', 0.6, 7)], [('Posterize', 0.8, 2), ('Solarize', 0.6, 10)], # This results in black image with Tpu posterize [('Solarize', 0.6, 8), ('Equalize', 0.6, 1)], [('Color', 0.8, 6), ('Rotate', 0.4, 5)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_v0r(hparams): """auto_augment_policy_v0r""" # ImageNet v0 policy from TPU EfficientNet impl, with variation of Posterize used # in Google research implementation (number of bits discarded increases with magnitude) policy = [ [('Equalize', 0.8, 1), ('ShearY', 0.8, 4)], [('Color', 0.4, 9), ('Equalize', 0.6, 3)], [('Color', 0.4, 1), ('Rotate', 0.6, 8)], [('Solarize', 0.8, 3), ('Equalize', 0.4, 7)], [('Solarize', 0.4, 2), ('Solarize', 0.6, 2)], [('Color', 0.2, 0), ('Equalize', 0.8, 8)], [('Equalize', 0.4, 8), ('SolarizeAdd', 0.8, 3)], [('ShearX', 0.2, 9), ('Rotate', 0.6, 8)], [('Color', 0.6, 1), ('Equalize', 1.0, 2)], [('Invert', 0.4, 9), ('Rotate', 0.6, 0)], [('Equalize', 1.0, 9), ('ShearY', 0.6, 3)], [('Color', 0.4, 7), ('Equalize', 0.6, 0)], [('PosterizeIncreasing', 0.4, 6), ('AutoContrast', 0.4, 7)], [('Solarize', 0.6, 8), ('Color', 0.6, 9)], [('Solarize', 0.2, 4), ('Rotate', 0.8, 9)], [('Rotate', 1.0, 7), ('TranslateYRel', 0.8, 9)], [('ShearX', 0.0, 0), ('Solarize', 0.8, 4)], [('ShearY', 0.8, 0), ('Color', 0.6, 4)], [('Color', 1.0, 0), ('Rotate', 0.6, 2)], [('Equalize', 0.8, 4), ('Equalize', 0.0, 8)], [('Equalize', 1.0, 4), ('AutoContrast', 0.6, 2)], [('ShearY', 0.4, 7), ('SolarizeAdd', 0.6, 7)], [('PosterizeIncreasing', 0.8, 2), ('Solarize', 0.6, 10)], [('Solarize', 0.6, 8), ('Equalize', 0.6, 1)], [('Color', 0.8, 6), ('Rotate', 0.4, 5)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_original(hparams): """auto_augment_policy_original""" policy = [ [('PosterizeOriginal', 0.4, 8), ('Rotate', 0.6, 9)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], [('PosterizeOriginal', 0.6, 7), ('PosterizeOriginal', 0.6, 6)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Equalize', 0.4, 4), ('Rotate', 0.8, 8)], [('Solarize', 0.6, 3), ('Equalize', 0.6, 7)], [('PosterizeOriginal', 0.8, 5), ('Equalize', 1.0, 2)], [('Rotate', 0.2, 3), ('Solarize', 0.6, 8)], [('Equalize', 0.6, 8), ('PosterizeOriginal', 0.4, 6)], [('Rotate', 0.8, 8), ('Color', 0.4, 0)], [('Rotate', 0.4, 9), ('Equalize', 0.6, 2)], [('Equalize', 0.0, 7), ('Equalize', 0.8, 8)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Rotate', 0.8, 8), ('Color', 1.0, 2)], [('Color', 0.8, 8), ('Solarize', 0.8, 7)], [('Sharpness', 0.4, 7), ('Invert', 0.6, 8)], [('ShearX', 0.6, 5), ('Equalize', 1.0, 9)], [('Color', 0.4, 0), ('Equalize', 0.6, 3)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy_originalr(hparams): """auto_augment_policy_originalr""" # ImageNet policy from https://arxiv.org/abs/1805.09501 with research posterize variation policy = [ [('PosterizeIncreasing', 0.4, 8), ('Rotate', 0.6, 9)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], [('PosterizeIncreasing', 0.6, 7), ('PosterizeIncreasing', 0.6, 6)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Equalize', 0.4, 4), ('Rotate', 0.8, 8)], [('Solarize', 0.6, 3), ('Equalize', 0.6, 7)], [('PosterizeIncreasing', 0.8, 5), ('Equalize', 1.0, 2)], [('Rotate', 0.2, 3), ('Solarize', 0.6, 8)], [('Equalize', 0.6, 8), ('PosterizeIncreasing', 0.4, 6)], [('Rotate', 0.8, 8), ('Color', 0.4, 0)], [('Rotate', 0.4, 9), ('Equalize', 0.6, 2)], [('Equalize', 0.0, 7), ('Equalize', 0.8, 8)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Rotate', 0.8, 8), ('Color', 1.0, 2)], [('Color', 0.8, 8), ('Solarize', 0.8, 7)], [('Sharpness', 0.4, 7), ('Invert', 0.6, 8)], [('ShearX', 0.6, 5), ('Equalize', 1.0, 9)], [('Color', 0.4, 0), ('Equalize', 0.6, 3)], [('Equalize', 0.4, 7), ('Solarize', 0.2, 4)], [('Solarize', 0.6, 5), ('AutoContrast', 0.6, 5)], [('Invert', 0.6, 4), ('Equalize', 1.0, 8)], [('Color', 0.6, 4), ('Contrast', 1.0, 8)], [('Equalize', 0.8, 8), ('Equalize', 0.6, 3)], ] pc = [[AugmentOp(*a, hparams=hparams) for a in sp] for sp in policy] return pc def auto_augment_policy(name='v0', hparams=None): """auto_augment_policy""" hparams = hparams or _HPARAMS_DEFAULT if name == 'original': func = auto_augment_policy_original(hparams) elif name == 'originalr': func = auto_augment_policy_originalr(hparams) elif name == 'v0': func = auto_augment_policy_v0(hparams) elif name == 'v0r': func = auto_augment_policy_v0r(hparams) else: raise ValueError(f'Unknown AA policy {name}') return func class AutoAugment: """AutoAugment""" def __init__(self, policy): self.policy = policy def __call__(self, img): """apply autoaugment""" sub_policy = random.choice(self.policy) for op in sub_policy: img = op(img) return img @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def auto_augment_transform(config_str, hparams): """ Create a AutoAugment transform :param config_str: String defining configuration of auto augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the AutoAugment policy (one of 'v0', 'v0r', 'original', 'originalr'). The remaining sections, not order specific determine 'mstd' - float std deviation of magnitude noise applied Ex 'original-mstd0.5' results in AutoAugment with original policy, magnitude_std 0.5 :param hparams: Other hparams (kwargs) for the AutoAugmentation scheme :return: A MindSpore compatible Transform """ config = config_str.split('-') policy_name = config[0] config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) else: raise ValueError('Unknown AutoAugment config section') return AutoAugment(auto_augment_policy(policy_name, hparams=hparams)) _RAND_TRANSFORMS = [ 'AutoContrast', 'Equalize', 'Invert', 'Rotate', 'Posterize', 'Solarize', 'SolarizeAdd', 'Color', 'Contrast', 'Brightness', 'Sharpness', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', # 'Cutout' # NOTE I've implement this as random erasing separately ] _RAND_INCREASING_TRANSFORMS = [ 'AutoContrast', 'Equalize', 'Invert', 'Rotate', 'PosterizeIncreasing', 'SolarizeIncreasing', 'SolarizeAdd', 'ColorIncreasing', 'ContrastIncreasing', 'BrightnessIncreasing', 'SharpnessIncreasing', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', # 'Cutout' # NOTE I've implement this as random erasing separately ] # These experimental weights are based loosely on the relative improvements mentioned in paper. # They may not result in increased performance, but could likely be tuned to so. _RAND_CHOICE_WEIGHTS_0 = { 'Rotate': 0.3, 'ShearX': 0.2, 'ShearY': 0.2, 'TranslateXRel': 0.1, 'TranslateYRel': 0.1, 'Color': .025, 'Sharpness': 0.025, 'AutoContrast': 0.025, 'Solarize': .005, 'SolarizeAdd': .005, 'Contrast': .005, 'Brightness': .005, 'Equalize': .005, 'Posterize': 0, 'Invert': 0, } def _select_rand_weights(weight_idx=0, transforms=None): """_select_rand_weights""" transforms = transforms or _RAND_TRANSFORMS if weight_idx != 0: raise ValueError(f"only one set of weights currently, get weight_idx {weight_idx}") rand_weights = _RAND_CHOICE_WEIGHTS_0 probs = [rand_weights[k] for k in transforms] probs /= np.sum(probs) return probs def rand_augment_ops(magnitude=10, hparams=None, transforms=None): """rand_augment_ops""" hparams = hparams or _HPARAMS_DEFAULT transforms = transforms or _RAND_TRANSFORMS return [AugmentOp( name, prob=0.5, magnitude=magnitude, hparams=hparams) for name in transforms] class RandAugment(PyTensorOperation): """RandAugment""" def __init__(self, ops, num_layers=2, choice_weights=None): super(RandAugment, self).__init__() self.ops = ops self.num_layers = num_layers self.choice_weights = choice_weights def __call__(self, img): # no replacement when using weighted choice ops = np.random.choice( self.ops, self.num_layers, replace=self.choice_weights is None, p=self.choice_weights) for op in ops: img = op(img) return img @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def rand_augment_transform(config_str, hparams): """ Create a RandAugment transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude of rand augment 'n' - integer num layers (number of transform ops selected per image) 'w' - integer probabiliy weight index (index of a set of weights to influence choice of op) 'mstd' - float std deviation of magnitude noise applied 'inc' - integer (bool), use augmentations that increase in severity with magnitude (default: 0) Ex 'rand-m9-n3-mstd0.5' results in RandAugment with magnitude 9, num_layers 3, magnitude_std 0.5 'rand-mstd1-w0' results in magnitude_std 1.0, weights 0, default magnitude of 10 and num_layers 2 :param hparams: Other hparams (kwargs) for the RandAugmentation scheme :return: A MindSpore compatible Transform """ magnitude = _MAX_LEVEL # default to _MAX_LEVEL for magnitude (currently 10) num_layers = 2 # default to 2 ops per image weight_idx = None # default to no probability weights for op choice transforms = _RAND_TRANSFORMS config = config_str.split('-') if config[0] != 'rand': raise ValueError(f"config[0] is not rand, get {config[0]}") # [rand, m9, mstd0.5, inc1] config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) elif key == 'inc': if bool(val): transforms = _RAND_INCREASING_TRANSFORMS elif key == 'm': magnitude = int(val) elif key == 'n': num_layers = int(val) elif key == 'w': weight_idx = int(val) else: raise ValueError('Unknown RandAugment config section') ra_ops = rand_augment_ops(magnitude=magnitude, hparams=hparams, transforms=transforms) choice_weights = None if weight_idx is None else _select_rand_weights(weight_idx) return RandAugment(ra_ops, num_layers, choice_weights=choice_weights) _AUGMIX_TRANSFORMS = [ 'AutoContrast', 'ColorIncreasing', # not in paper 'ContrastIncreasing', # not in paper 'BrightnessIncreasing', # not in paper 'SharpnessIncreasing', # not in paper 'Equalize', 'Rotate', 'PosterizeIncreasing', 'SolarizeIncreasing', 'ShearX', 'ShearY', 'TranslateXRel', 'TranslateYRel', ] def augmix_ops(magnitude=10, hparams=None, transforms=None): """augmix_ops""" hparams = hparams or _HPARAMS_DEFAULT transforms = transforms or _AUGMIX_TRANSFORMS return [AugmentOp( name, prob=1.0, magnitude=magnitude, hparams=hparams) for name in transforms] class AugMixAugment: """ AugMix Transform Adapted and improved from impl here From paper: 'AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty - https://arxiv.org/abs/1912.02781 """ def __init__(self, ops, alpha=1., width=3, depth=-1, blended=False): self.ops = ops self.alpha = alpha self.width = width self.depth = depth self.blended = blended # blended mode is faster but not well tested def _calc_blended_weights(self, ws, m): """_calc_blended_weights of AugMixAugment""" ws = ws * m cump = 1. rws = [] for w in ws[::-1]: alpha = w / cump cump *= (1 - alpha) rws.append(alpha) return np.array(rws[::-1], dtype=np.float32) def _apply_blended(self, img, mixing_weights, m): """_apply_blended of AugMixAugment""" # This is my first crack and implementing a slightly faster mixed augmentation. Instead # of accumulating the mix for each chain in a Numpy array and then blending with original, # it recomputes the blending coefficients and applies one PIL image blend per chain. # TODO the results appear in the right ballpark but they differ by more than rounding. img_orig = img.copy() ws = self._calc_blended_weights(mixing_weights, m) for w in ws: depth = self.depth if self.depth > 0 else np.random.randint(1, 4) ops = np.random.choice(self.ops, depth, replace=True) img_aug = img_orig # no ops are in-place, deep copy not necessary for op in ops: img_aug = op(img_aug) img = Image.blend(img, img_aug, w) return img def _apply_basic(self, img, mixing_weights, m): """_apply_basic of AugMixAugment""" # This is a literal adaptation of the paper/official implementation without normalizations and # PIL <-> Numpy conversions between every op. It is still quite CPU compute heavy compared to the # typical augmentation transforms, could use a GPU / Kornia implementation. img_shape = img.size[0], img.size[1], len(img.getbands()) mixed = np.zeros(img_shape, dtype=np.float32) for mw in mixing_weights: depth = self.depth if self.depth > 0 else np.random.randint(1, 4) ops = np.random.choice(self.ops, depth, replace=True) img_aug = img # no ops are in-place, deep copy not necessary for op in ops: img_aug = op(img_aug) mixed += mw * np.asarray(img_aug, dtype=np.float32) np.clip(mixed, 0, 255., out=mixed) mixed = Image.fromarray(mixed.astype(np.uint8)) return Image.blend(img, mixed, m) def __call__(self, img): """AugMixAugment apply""" mixing_weights = np.float32(np.random.dirichlet([self.alpha] * self.width)) m = np.float32(np.random.beta(self.alpha, self.alpha)) if self.blended: mixed = self._apply_blended(img, mixing_weights, m) else: mixed = self._apply_basic(img, mixing_weights, m) return mixed @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) def augment_and_mix_transform(config_str, hparams): """ Create AugMix MindSpore transform :param config_str: String defining configuration of random augmentation. Consists of multiple sections separated by dashes ('-'). The first section defines the specific variant of rand augment (currently only 'rand'). The remaining sections, not order specific determine 'm' - integer magnitude (severity) of augmentation mix (default: 3) 'w' - integer width of augmentation chain (default: 3) 'd' - integer depth of augmentation chain (-1 is random [1, 3], default: -1) 'b' - integer (bool), blend each branch of chain into end result without a final blend, less CPU (default: 0) 'mstd' - float std deviation of magnitude noise applied (default: 0) Ex 'augmix-m5-w4-d2' results in AugMix with severity 5, chain width 4, chain depth 2 :param hparams: Other hparams (kwargs) for the Augmentation transforms :return: A MindSpore compatible Transform """ magnitude = 3 width = 3 depth = -1 alpha = 1. blended = False hparams['magnitude_std'] = float('inf') config = config_str.split('-') if config[0] != 'augmix': raise ValueError(f"config[0] is not augmix, get {config[0]}") config = config[1:] for c in config: cs = re.split(r'(\d.*)', c) if len(cs) >= 2: key, val = cs[:2] if key == 'mstd': # noise param injected via hparams for now hparams.setdefault('magnitude_std', float(val)) elif key == 'm': magnitude = int(val) elif key == 'w': width = int(val) elif key == 'd': depth = int(val) elif key == 'a': alpha = float(val) elif key == 'b': blended = bool(val) else: raise ValueError('Unknown AugMix config section') ops = augmix_ops(magnitude=magnitude, hparams=hparams) return AugMixAugment(ops, alpha=alpha, width=width, depth=depth, blended=blended) pengcheng.mind-mindformer/mindformers/dataset/transforms/build_transforms.py000066400000000000000000000064441476650141400302210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Transforms API.""" import inspect from mindspore.dataset import transforms as tf from mindspore.dataset import vision as vs from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_transforms( config: [dict, list] = None, default_args: dict = None, module_type: str = 'transforms', class_name: str = None, **kwargs): r"""Build transform For MindFormer. Instantiate the transform from MindFormerRegister's registry. Args: config (dict, list): The task transform's config. Default: None. default_args (dict): The default argument of transform API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'transform'. class_name (str): The class name of transform API. Default: None. Return: The function instance of transform API. Examples: >>> from mindformers import build_transforms >>> transform_config = [{'type': 'Decode'}, {'type': 'Resize', 'size': 256}] >>> # 1) use config dict to build transform >>> transform_from_config = build_transforms(transform_config) >>> # 2) use class name to build transform >>> transform_class_name = build_transforms(class_name='Decode') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) cfg_transforms = config if not isinstance(cfg_transforms, list): return MindFormerRegister.get_instance_from_cfg( cfg_transforms, MindFormerModuleType.TRANSFORMS, default_args=default_args) transforms = [] for transform in cfg_transforms: transform_op = MindFormerRegister.get_instance_from_cfg( transform, MindFormerModuleType.TRANSFORMS) transforms.append(transform_op) return transforms return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_transforms(): """register MindSpore builtin transforms class.""" for module_name in set(dir(vs.transforms) + dir(tf.transforms)): if module_name.startswith('__'): continue transforms = getattr(vs.transforms, module_name, None) \ if getattr(vs.transforms, module_name, None)\ else getattr(tf.transforms, module_name) if inspect.isclass(transforms): MindFormerRegister.register_cls(transforms, MindFormerModuleType.TRANSFORMS) register_ms_transforms() pengcheng.mind-mindformer/mindformers/dataset/transforms/mixup.py000066400000000000000000000263651476650141400260120ustar00rootroot00000000000000# Mixup and Cutmix for MindSpore # # Papers: # mixup: Beyond Empirical Risk Minimization (https://arxiv.org/abs/1710.09412) # # CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features (https://arxiv.org/abs/1905.04899) # # Code Reference: # CutMix: https://github.com/clovaai/CutMix-PyTorch, Apache 2.0 # # Hacked together by / Copyright 2020 Ross Wightman # ============================================================================ """Mixup and Cutmix for MindSpore""" import numpy as np from mindspore import Tensor from mindspore import dtype as mstype from mindspore import ops as P from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType def one_hot(x, num_classes, on_value=1., off_value=0.): """one hot to label""" x = x.reshape(-1) x = np.eye(num_classes)[x] x = np.clip(x, a_min=off_value, a_max=on_value, dtype=np.float32) return x def mixup_target(target, num_classes, lam=1., smoothing=0.0): """mixup_target""" off_value = smoothing / num_classes on_value = 1. - smoothing + off_value y1 = one_hot(target, num_classes, on_value=on_value, off_value=off_value) y2 = one_hot(np.flip(target, axis=0), num_classes, on_value=on_value, off_value=off_value) return y1 * lam + y2 * (1. - lam) def rand_bbox(img_shape, lam, margin=0., count=None): """ Standard CutMix bounding-box Generates a random square bbox based on lambda value. This impl includes support for enforcing a border margin as percent of bbox dimensions. Args: img_shape (tuple): Image shape as tuple lam (float): Cutmix lambda value margin (float): Percentage of bbox dimension to enforce as margin (reduce amount of box outside image) count (int): Number of bbox to generate """ ratio = np.sqrt(1 - lam) img_h, img_w = img_shape[-2:] cut_h, cut_w = int(img_h * ratio), int(img_w * ratio) margin_y, margin_x = int(margin * cut_h), int(margin * cut_w) cy = np.random.randint(0 + margin_y, img_h - margin_y, size=count) cx = np.random.randint(0 + margin_x, img_w - margin_x, size=count) yl = np.clip(cy - cut_h // 2, 0, img_h) yh = np.clip(cy + cut_h // 2, 0, img_h) xl = np.clip(cx - cut_w // 2, 0, img_w) xh = np.clip(cx + cut_w // 2, 0, img_w) return yl, yh, xl, xh def rand_bbox_minmax(img_shape, minmax, count=None): """ Min-Max CutMix bounding-box Inspired by Darknet cutmix impl, generates a random rectangular bbox based on min/max percent values applied to each dimension of the input image. Typical defaults for minmax are usually in the .2-.3 for min and .8-.9 range for max. Args: img_shape (tuple): Image shape as tuple minmax (tuple or list): Min and max bbox ratios (as percent of image size) count (int): Number of bbox to generate """ if len(minmax) != 2: raise ValueError(f"len(minmax) is not 2. ") img_h, img_w = img_shape[-2:] cut_h = np.random.randint(int(img_h * minmax[0]), int(img_h * minmax[1]), size=count) cut_w = np.random.randint(int(img_w * minmax[0]), int(img_w * minmax[1]), size=count) yl = np.random.randint(0, img_h - cut_h, size=count) xl = np.random.randint(0, img_w - cut_w, size=count) yu = yl + cut_h xu = xl + cut_w return yl, yu, xl, xu def cutmix_bbox_and_lam(img_shape, lam, ratio_minmax=None, correct_lam=True, count=None): """ Generate bbox and apply lambda correction. """ if ratio_minmax is not None: yl, yu, xl, xu = rand_bbox_minmax(img_shape, ratio_minmax, count=count) else: yl, yu, xl, xu = rand_bbox(img_shape, lam, count=count) if correct_lam or ratio_minmax is not None: bbox_area = (yu - yl) * (xu - xl) lam = 1. - bbox_area / float(img_shape[-2] * img_shape[-1]) return (yl, yu, xl, xu), lam @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class Mixup(PyTensorOperation): """ Mixup/Cutmix that applies different params to each element or whole batch Args: mixup_alpha (float): mixup alpha value, mixup is active if > 0. cutmix_alpha (float): cutmix alpha value, cutmix is active if > 0. cutmix_minmax (List[float]): cutmix min/max image ratio, cutmix is active and uses this vs alpha if not None. prob (float): probability of applying mixup or cutmix per batch or element switch_prob (float): probability of switching to cutmix instead of mixup when both are active mode (str): how to apply mixup/cutmix params (per 'batch', 'pair' (pair of elements), 'elem' (element) correct_lam (bool): apply lambda correction when cutmix bbox clipped by image borders label_smoothing (float): apply label smoothing to the mixed target tensor num_classes (int): number of classes for target """ def __init__(self, mixup_alpha=1., cutmix_alpha=0., cutmix_minmax=None, prob=1.0, switch_prob=0.5, mode='batch', correct_lam=True, label_smoothing=0.1, num_classes=1000): super(Mixup, self).__init__() self.mixup_alpha = mixup_alpha self.cutmix_alpha = cutmix_alpha self.cutmix_minmax = cutmix_minmax if self.cutmix_minmax is not None: if len(self.cutmix_minmax) != 2: raise ValueError(f"len(cutmix_minmax) is not 2.") # force cutmix alpha == 1.0 when minmax active to keep logic simple & safe self.cutmix_alpha = 1.0 self.mix_prob = prob self.switch_prob = switch_prob self.label_smoothing = label_smoothing self.num_classes = num_classes self.mode = mode self.correct_lam = correct_lam # correct lambda based on clipped area for cutmix self.mixup_enabled = True # set to false to disable mixing (intended tp be set by train loop) def _params_per_elem(self, batch_size): """_params_per_elem""" lam = np.ones(batch_size, dtype=np.float32) use_cutmix = np.zeros(batch_size, dtype=np.bool) if self.mixup_enabled: if self.mixup_alpha > 0. and self.cutmix_alpha > 0.: use_cutmix = np.random.rand(batch_size) < self.switch_prob lam_mix = np.where( use_cutmix, np.random.beta(self.cutmix_alpha, self.cutmix_alpha, size=batch_size), np.random.beta(self.mixup_alpha, self.mixup_alpha, size=batch_size)) elif self.mixup_alpha > 0.: lam_mix = np.random.beta(self.mixup_alpha, self.mixup_alpha, size=batch_size) elif self.cutmix_alpha > 0.: use_cutmix = np.ones(batch_size, dtype=np.bool) lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha, size=batch_size) else: raise ValueError("One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.") lam = np.where(np.random.rand(batch_size) < self.mix_prob, lam_mix.astype(np.float32), lam) return lam, use_cutmix def _params_per_batch(self): """_params_per_batch""" lam = 1. use_cutmix = False if self.mixup_enabled and np.random.rand() < self.mix_prob: if self.mixup_alpha > 0. and self.cutmix_alpha > 0.: use_cutmix = np.random.rand() < self.switch_prob lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha) if use_cutmix else \ np.random.beta(self.mixup_alpha, self.mixup_alpha) elif self.mixup_alpha > 0.: lam_mix = np.random.beta(self.mixup_alpha, self.mixup_alpha) elif self.cutmix_alpha > 0.: use_cutmix = True lam_mix = np.random.beta(self.cutmix_alpha, self.cutmix_alpha) else: raise ValueError("One of mixup_alpha > 0., cutmix_alpha > 0., cutmix_minmax not None should be true.") lam = float(lam_mix) return lam, use_cutmix def _mix_elem(self, x): """_mix_elem""" batch_size = len(x) lam_batch, use_cutmix = self._params_per_elem(batch_size) x_orig = x.clone() # need to keep an unmodified original for mixing source for i in range(batch_size): j = batch_size - i - 1 lam = lam_batch[i] if lam != 1.: if use_cutmix[i]: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x[i].shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[i][:, yl:yh, xl:xh] = x_orig[j][:, yl:yh, xl:xh] lam_batch[i] = lam else: x[i] = x[i] * lam + x_orig[j] * (1 - lam) return P.ExpandDims()(Tensor(lam_batch, dtype=mstype.float32), 1) def _mix_pair(self, x): """_mix_pair""" batch_size = len(x) lam_batch, use_cutmix = self._params_per_elem(batch_size // 2) x_orig = x.clone() # need to keep an unmodified original for mixing source for i in range(batch_size // 2): j = batch_size - i - 1 lam = lam_batch[i] if lam != 1.: if use_cutmix[i]: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x[i].shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[i][:, yl:yh, xl:xh] = x_orig[j][:, yl:yh, xl:xh] x[j][:, yl:yh, xl:xh] = x_orig[i][:, yl:yh, xl:xh] lam_batch[i] = lam else: x[i] = x[i] * lam + x_orig[j] * (1 - lam) x[j] = x[j] * lam + x_orig[i] * (1 - lam) lam_batch = np.concatenate((lam_batch, lam_batch[::-1])) return P.ExpandDims()(Tensor(lam_batch, dtype=mstype.float32), 1) def _mix_batch(self, x): """_mix_batch""" lam, use_cutmix = self._params_per_batch() if lam == 1.: return 1. if use_cutmix: (yl, yh, xl, xh), lam = cutmix_bbox_and_lam( x.shape, lam, ratio_minmax=self.cutmix_minmax, correct_lam=self.correct_lam) x[:, :, yl:yh, xl:xh] = np.flip(x, axis=0)[:, :, yl:yh, xl:xh] else: x_flipped = np.flip(x, axis=0) * (1. - lam) x *= lam x += x_flipped return lam def __call__(self, x, target): """Mixup apply""" # the same to image, label if len(x) % 2 != 0: if len(x) > 1: x = x[:-1] logger.warning('Batch size is odd. When using mixup, batch size should be even.' 'The last data in batch has been dropped to use mixip.' 'you can set "drop_remainder" true in dataset config manually.') else: logger.error('Batch size is 1.' 'If error occurs, please set "drop_remainder" true in dataset config.') if self.mode == 'elem': lam = self._mix_elem(x) elif self.mode == 'pair': lam = self._mix_pair(x) else: lam = self._mix_batch(x) target = mixup_target(target, self.num_classes, lam, self.label_smoothing) return x.astype(np.float32), target.astype(np.float32) pengcheng.mind-mindformer/mindformers/dataset/transforms/random_erasing.py000066400000000000000000000113561476650141400276320ustar00rootroot00000000000000# Random Erasing (Cutout) for MindSpore # # Originally inspired by impl at https://github.com/zhunzhong07/Random-Erasing, Apache 2.0 # Copyright Zhun Zhong & Liang Zheng # # Hacked together by / Copyright 2020 Ross Wightman # # has been modified to adapt mindspore and mindformers # ============================================================================ """Random Erasing (Cutout) for MindSpore""" import math import random import numpy as np from mindspore.dataset.vision.transforms import PyTensorOperation from mindformers.tools.register import MindFormerRegister, MindFormerModuleType def _get_pixels(per_pixel, rand_color, patch_size, dtype=np.float32): """_get_pixels""" if per_pixel: func = np.random.normal(size=patch_size).astype(dtype) elif rand_color: func = np.random.normal(size=(patch_size[0], 1, 1)).astype(dtype) else: func = np.zeros((patch_size[0], 1, 1), dtype=dtype) return func @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomErasing(PyTensorOperation): """ Randomly selects a rectangle region in an image and erases its pixels. 'Random Erasing Data Augmentation' by Zhong et al. See https://arxiv.org/pdf/1708.04896.pdf This variant of RandomErasing is intended to be applied to either a batch or single image tensor after it has been normalized by dataset mean and std. Args: probability: Probability that the Random Erasing operation will be performed. min_area: Minimum percentage of erased area wrt input image area. max_area: Maximum percentage of erased area wrt input image area. min_aspect: Minimum aspect ratio of erased area. mode: pixel color mode, one of 'const', 'rand', or 'pixel' 'const' - erase block is constant color of 0 for all channels 'rand' - erase block is same per-channel random (normal) color 'pixel' - erase block is per-pixel random (normal) color max_count: maximum number of erasing blocks per image, area per box is scaled by count. per-image count is randomly chosen between 1 and this value. """ def __init__(self, probability=0.5, min_area=0.02, max_area=1 / 3, min_aspect=0.3, max_aspect=None, mode='const', min_count=1, max_count=None, num_splits=0): super(RandomErasing, self).__init__() self.probability = probability self.min_area = min_area self.max_area = max_area max_aspect = max_aspect or 1 / min_aspect self.log_aspect_ratio = (math.log(min_aspect), math.log(max_aspect)) self.min_count = min_count self.max_count = max_count or min_count self.num_splits = num_splits mode = mode.lower() self.rand_color = False self.per_pixel = False if mode == 'rand': self.rand_color = True # per block random normal elif mode == 'pixel': self.per_pixel = True # per pixel random normal else: if mode and mode == 'const': raise ValueError("mode is const") def _erase(self, img, chan, img_h, img_w, dtype): """_erase""" if random.random() > self.probability: pass else: area = img_h * img_w count = self.min_count if self.min_count == self.max_count else \ random.randint(self.min_count, self.max_count) for _ in range(count): for _ in range(10): target_area = random.uniform(self.min_area, self.max_area) * area / count aspect_ratio = math.exp(random.uniform(*self.log_aspect_ratio)) h = int(round(math.sqrt(target_area * aspect_ratio))) w = int(round(math.sqrt(target_area / aspect_ratio))) if w < img_w and h < img_h: top = random.randint(0, img_h - h) left = random.randint(0, img_w - w) img[:, top:top + h, left:left + w] = _get_pixels( self.per_pixel, self.rand_color, (chan, h, w), dtype=dtype) break return img def __call__(self, x): """RandomErasing apply""" if len(x.shape) == 3: output = self._erase(x, *x.shape, x.dtype) else: output = np.zeros_like(x) batch_size, chan, img_h, img_w = x.shape # skip first slice of batch if num_splits is set (for clean portion of samples) batch_start = batch_size // self.num_splits if self.num_splits > 1 else 0 for i in range(batch_start, batch_size): output[i] = self._erase(x[i], chan, img_h, img_w, x.dtype) return output pengcheng.mind-mindformer/mindformers/dataset/transforms/text_transforms.py000066400000000000000000000141271476650141400301030ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Transforms.""" import re import numpy as np from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'RandomChoiceTokenizerForward', 'TokenizerForward', 'TokenizeWithLabel', 'LabelPadding', 'CaptionTransform' ] @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomChoiceTokenizerForward: """Random Choice Tokenizer Forward""" def __init__(self, tokenizer, max_length=77, padding="max_length", random_seed=2022): self.max_length = max_length self.padding = padding self.tokenizer = tokenizer self.random_seed = random_seed def __call__(self, text): text_list = text.tolist() np.random.seed(self.random_seed) index = np.random.choice(len(text_list)) token_id = self.tokenizer( text_list[index].decode("utf-8", "ignore") if isinstance(text_list[index], bytes) else text_list[index], max_length=self.max_length, padding=self.padding )["input_ids"] return token_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TokenizerForward: """Tokenizer Forward""" def __init__(self, tokenizer, max_length=77, padding="max_length"): self.tokenizer = tokenizer self.max_length = max_length self.padding = padding def __call__(self, text): """call method""" text = text.tolist() if isinstance(text, np.ndarray) else text for i in range(len(text)): if isinstance(text[i], bytes): text[i] = text[i].decode("utf-8", "ignore") token_id = self.tokenizer( text, max_length=self.max_length, padding=self.padding)["input_ids"] return token_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TokenizeWithLabel: """Tokenizer With Label""" def __init__(self, tokenizer, max_length=128, padding="max_length"): self.tokenizer = tokenizer self.max_length = max_length self.padding = padding def __call__(self, *inputs): """call method""" text = inputs[0] label_id = inputs[1] text = text.tolist() if isinstance(text, np.ndarray) else text if isinstance(text, bytes): text = text.decode("utf-8", "ignore") output = self.tokenizer(text, max_length=self.max_length, padding=self.padding) input_ids = np.array(output["input_ids"], dtype=np.int32) token_type_ids = np.array(output["token_type_ids"], dtype=np.int32) attention_mask = np.array(output["attention_mask"], dtype=np.int32) return input_ids, token_type_ids, attention_mask, label_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class LabelPadding: """Label Padding""" def __init__(self, max_length, padding_value=0): self.max_length = max_length self.padding_value = padding_value def __call__(self, label_id): """call method""" if len(label_id) > self.max_length - 1: label_id = label_id[:(self.max_length - 1)] pad_label_id = [] # For CLS token pad_label_id.append(self.padding_value) for i in range(len(label_id)): pad_label_id.append(label_id[i]) while len(pad_label_id) < self.max_length: pad_label_id.append(self.padding_value) pad_label_id = np.array(pad_label_id, dtype=np.int32) return pad_label_id @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class CaptionTransform: """ Caption Transform, preprocess captions and tokenize it, align with torch impl. """ def __init__(self, tokenizer, prompt="", max_words=50, max_length=32, padding="max_length", random_seed=2022, truncation=True, add_special_tokens=True): self.tokenizer = tokenizer self.prompt = prompt self.max_words = max_words self.max_length = max_length self.padding = padding self.random_seed = random_seed self.truncation = truncation self.add_special_tokens = add_special_tokens def __call__(self, caption): if caption.ndim == 1: caption_list = [] for i in range(caption.shape[0]): caption_list.append(self.pre_caption(caption[i])) return caption_list input_ids = self.pre_caption(caption) return input_ids def pre_caption(self, caption): """ Caption preprocessing removes any punctuationmarks except commas, tailing spaces and transform sentence into lower case. """ caption = str(caption) caption = self.prompt + caption caption = re.sub( r"([.!\"()*#:;~])", " ", caption.lower(), ) caption = re.sub( r"\s{2,}", " ", caption, ) caption = caption.rstrip("\n") caption = caption.strip(" ") # truncate caption caption_words = caption.split(" ") if len(caption_words) > self.max_words: caption = " ".join(caption_words[: self.max_words]) output = self.tokenizer(caption, max_length=self.max_length, padding=self.padding, truncation=self.truncation, add_special_tokens=self.add_special_tokens) input_ids = np.array(output["input_ids"], dtype=np.int32) return input_ids pengcheng.mind-mindformer/mindformers/dataset/transforms/vision_transforms.py000066400000000000000000000277541476650141400304400ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ transform methods for vision models """ import numpy as np from PIL import Image from mindspore.dataset import vision import mindspore as ms from mindspore.dataset.vision.utils import Inter from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = [ 'BatchResize', 'BCHW2BHWC', 'BatchPILize', 'BatchNormalize', 'BatchCenterCrop', 'BatchToTensor', 'RandomCropDecodeResize', 'RandomResizedCrop', 'Resize', 'RandomHorizontalFlip' ] INTERPOLATION = {'nearest': Inter.NEAREST, 'antialias': Inter.ANTIALIAS, 'linear': Inter.LINEAR, 'cubic': Inter.PILCUBIC, 'bicubic': Inter.BICUBIC} class BCHW2BHWC: """ Transform a batch of image from CHW to HWC. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the channel should be (bz, c, h, w) or (c, h, w). for list, the item should be PIL.Image or numpy.array (c, h, w). Return: transformed image batch: for numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for PIL.Image input, it is returned directly. """ def __call__(self, image_batch): """the call function""" if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return image_batch.transpose(0, 2, 3, 1) if len(image_batch.shape) == 3: return image_batch.transpose(1, 2, 0) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return image_batch raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchResize: """ Resize a batch of image to the given shape. Args: image_resolution (int): the target size. interpolation (str): interpolate method, default is 'cubic'. """ def __init__(self, image_resolution, interpolation='cubic'): self.interpolation = INTERPOLATION.get(interpolation) self.resize = vision.Resize(image_resolution, self.interpolation) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: resized image batch: for numpy or tensor input, return a numpy array; for PIL.Image input, it returns PIL.Image. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.resize(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.resize(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.resize(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.resize(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchCenterCrop: """ CenterCrop a batch of image to the given shape. Args: image_resolution (int): the target size. """ def __init__(self, image_resolution): self.crop = vision.CenterCrop(image_resolution) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the shape should be (bz, h, w, c) or (h, w, c). for list, the item should be PIL.Image or numpy.array (h, w, c). Returns: center cropped image batch: for numpy or tensor input, return a numpy array, the shape is (bz, image_resolution, image_resolution, c) or (image_resolution, image_resolution, c); for PIL.Image input, it is returned with shape (image_resolution, image_resolution). """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.crop(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.crop(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.crop(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.crop(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchToTensor: """Transform a batch of image to tensor and scale to (0, 1).""" def __init__(self): self.totensor = ms.dataset.vision.ToTensor() def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, PIL.Image, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image or numpy.array. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.totensor(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return np.row_stack([self.totensor(item)[np.newaxis, :] for item in image_batch]) if len(image_batch.shape) == 3: return self.totensor(image_batch) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") if isinstance(image_batch, Image.Image): return self.totensor(image_batch) raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchNormalize: """Normalize a batch of image.""" def __init__( self, mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711), is_hwc=False ): self.normalize = vision.Normalize(mean=mean, std=std, is_hwc=is_hwc) def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be numpy.array. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, list): return [self.normalize(item) for item in image_batch] if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 3: return self.normalize(image_batch) if len(image_batch.shape) == 4: return np.row_stack([self.normalize(item)[np.newaxis, :] for item in image_batch]) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") raise TypeError(f"the type {type(image_batch)} of image_batch is unsupported.") class BatchPILize: """transform a batch of image to PIL.Image list.""" def __call__(self, image_batch): """ The forward process. Args: image_batch (tensor, numpy.array, list): for tensor or numpy input, the rank should be 4 or 3. for list, the item should be PIL.Image. Returns: return a tensor or a list of tensor. """ if isinstance(image_batch, Image.Image): return image_batch if isinstance(image_batch, list): for item in image_batch: if not isinstance(item, Image.Image): raise TypeError("unsupported type in list," " when the image_batch is a list," " the item in list should be PIL.Image.") return image_batch if isinstance(image_batch, ms.Tensor): image_batch = image_batch.asnumpy() if isinstance(image_batch, np.ndarray): if len(image_batch.shape) == 4: return [Image.fromarray(item.astype(np.uint8)) for item in image_batch] if len(image_batch.shape) == 3: return Image.fromarray(image_batch.astype(np.uint8)) raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(image_batch.shape)}") raise ValueError("unsupported input type.") @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomCropDecodeResize(vision.transforms.RandomCropDecodeResize): """wrapper of RandomCropDecodeResize""" def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation='cubic', max_attempts=10): self.size = size self.scale = scale self.ratio = ratio self.interpolation = INTERPOLATION.get(interpolation) self.max_attempts = max_attempts super(RandomCropDecodeResize, self).__init__(size, scale, ratio, self.interpolation, max_attempts) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomResizedCrop(vision.transforms.RandomResizedCrop): """wrapper of RandomCropDecodeResize""" def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation='cubic', max_attempts=10): self.size = size self.scale = scale self.ratio = ratio self.interpolation = INTERPOLATION.get(interpolation) self.max_attempts = max_attempts super(RandomResizedCrop, self).__init__(size, scale, ratio, self.interpolation, max_attempts) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class Resize(vision.transforms.Resize): """wrapper of Resize""" def __init__(self, size, interpolation='cubic'): self.size = size self.interpolation = INTERPOLATION.get(interpolation) self.random = False super(Resize, self).__init__(size, self.interpolation) @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class RandomHorizontalFlip(vision.transforms.RandomHorizontalFlip): """wrapper of RandomHorizontalFlip""" def __init__(self, prob=0.5): self.prob = prob super(RandomHorizontalFlip, self).__init__(prob) pengcheng.mind-mindformer/mindformers/dataset/translation_dataset.py000066400000000000000000000162121476650141400265030ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Dataset.""" import os import copy import numpy as np import mindspore.common.dtype as mstype import mindspore.dataset.transforms.c_transforms as C from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .base_dataset import BaseDataset from ..auto_class import AutoTokenizer __all__ = ['TranslationDataset'] @MindFormerRegister.register(MindFormerModuleType.DATASET) class TranslationDataset(BaseDataset): """Translation dataset. Examples: >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers import MindFormerBook >>> from mindformers.dataset import TranslationDataset >>> from mindformers.dataset import build_dataset, check_dataset_config >>> config_dict_list = MindFormerBook.get_trainer_support_task_list() >>> config_path = config_dict_list['translation']['t5_small'] >>> # Initialize a MindFormerConfig instance with a specific config file of yaml. >>> config = MindFormerConfig(config_path) >>> config.train_dataset.data_loader.dataset_dir = "The required task dataset path" Note: The detailed data setting could refer to https://gitee.com/mindspore/mindformers/blob/r0.3/docs/model_cards/t5.md >>> check_dataset_config(config) >>> # 1) use config dict to build dataset >>> dataset_from_config = build_dataset(config.train_dataset_task) >>> # 2) use class name to build dataset >>> dataset_from_name = build_dataset(class_name='TranslationDataset', ... dataset_config=config.train_dataset_task.dataset_config) >>> # 3) use class to build dataset >>> dataset_from_class = TranslationDataset(config.train_dataset_task.dataset_config) """ def __new__(cls, dataset_config: dict = None): logger.info("Now Create Translation Dataset.") cls.init_dataset_config(dataset_config) if dataset_config.data_loader.type != 'MindDataset': dataset = cls._process_raw_text_data(dataset_config) else: dataset = cls._process_mindrecord_data(dataset_config) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) type_cast_op = C.TypeCast(mstype.int32) for input_arg in dataset_config.input_columns: dataset = get_dataset_map(dataset, operations=type_cast_op, input_columns=input_arg) return dataset @classmethod def _tokenizer_map(cls, dataset, tokenizer_config): """Maps the tokenizer on the source and the output""" tokenizer = AutoTokenizer.from_pretrained(tokenizer_config.type) prefix = tokenizer_config.prefix src_max_length = tokenizer_config.src_max_length tgt_max_length = tokenizer_config.tgt_max_length logger.info("Start tokenize on the dataset using tokenizer: %s", tokenizer_config) def pad_max_function(src, tgt): src = src.tolist() if isinstance(src, bytes): src = src.decode() output = tokenizer(prefix + src, padding='max_length', max_length=src_max_length, truncation=True) tgt = tgt.tolist() if isinstance(tgt, bytes): tgt = tgt.decode() tgt_output = tokenizer(tgt, padding='max_length', max_length=tgt_max_length, truncation=True) input_ids = np.array(output['input_ids'], np.int32) attention_mask = np.array(output['attention_mask'], np.float32) labels = np.array(tgt_output['input_ids'], np.int32) return input_ids, attention_mask, labels dataset = get_dataset_map(dataset, pad_max_function, input_columns=['source', 'target'], output_columns=['input_ids', 'attention_mask', 'labels']) dataset = dataset.project(columns=['input_ids', 'attention_mask', 'labels']) return dataset @classmethod def _process_raw_text_data(cls, dataset_config): """Process the text data""" rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_dir = dataset_config.data_loader.pop("dataset_dir") dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_dir': dataset_dir, 'num_shards': device_num, 'shard_id': rank_id}) dataset = cls._tokenizer_map(dataset, dataset_config.tokenizer) return dataset @classmethod def _process_mindrecord_data(cls, dataset_config): """Process the mindrecord data""" rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset_config = copy.deepcopy(dataset_config) dataset_files = [] if dataset_config.data_loader.dataset_dir: data_dir = dataset_config.data_loader.pop("dataset_dir") if os.path.isdir(data_dir): for r, _, f in os.walk(data_dir): for file in f: if file.endswith(".mindrecord"): dataset_files.append(os.path.join(r, file)) dataset_files.sort() else: if data_dir.endswith(".mindrecord"): dataset_files = data_dir elif dataset_config.data_loader.dataset_files: dataset_files = dataset_config.data_loader.dataset_files if isinstance(dataset_files, (list, tuple)): dataset_files = list(dataset_files) else: raise ValueError(f"data_loader must contain dataset_dir or dataset_files," f"but get {dataset_config.data_loader}.") logger.info("Using args %s to instance the dataset.", dataset_config.data_loader) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'dataset_files': dataset_files, 'num_shards': device_num, 'shard_id': rank_id, 'columns_list': dataset_config.input_columns}) return dataset pengcheng.mind-mindformer/mindformers/dataset/utils.py000066400000000000000000000036351476650141400236050ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Dataset Utils.""" def check_dataset_config(config): """Check dataset config.""" if config.train_dataset is not None: config.train_dataset.do_eval = False config.train_dataset.seed = config.seed config.train_dataset.auto_tune = config.auto_tune config.train_dataset.filepath_prefix = config.filepath_prefix config.train_dataset.autotune_per_step = config.autotune_per_step config.train_dataset.profile = config.profile config.train_dataset.batch_size = config.runner_config.batch_size if config.train_dataset.mixup_op: config.train_dataset.mixup_op.num_classes = config.runner_config.num_classes config.train_dataset_task.dataset_config = config.train_dataset if config.eval_dataset is not None: config.eval_dataset.do_eval = True config.eval_dataset.seed = config.seed config.eval_dataset.auto_tune = config.auto_tune config.eval_dataset.filepath_prefix = config.filepath_prefix config.eval_dataset.autotune_per_step = config.autotune_per_step config.eval_dataset.profile = config.profile config.eval_dataset.batch_size = config.runner_config.batch_size config.eval_dataset_task.dataset_config = config.eval_dataset pengcheng.mind-mindformer/mindformers/dataset/zero_shot_image_classification_dataset.py000066400000000000000000000116601476650141400324000ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Zero Shot Image Classification Dataset.""" import os from mindformers.version_control import get_dataset_map from .dataloader import build_dataset_loader from .transforms import build_transforms from .base_dataset import BaseDataset from ..tools import logger from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models.build_tokenizer import build_tokenizer @MindFormerRegister.register(MindFormerModuleType.DATASET) class ZeroShotImageClassificationDataset(BaseDataset): r""" Zero Shot Image Classification Dataset API. output image, text, and label columns Args: dataset_config (dict): Config for dataset. Returns: A dataset for ZeroShotImageClassificationTrainer. Examples: >>> import os >>> from mindformers import MindFormerBook, MindFormerConfig, build_dataset >>> project_path = MindFormerBook.get_project_path() >>> config_path = os.path.join(project_path, "configs", "clip", >>> "run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml") >>> config = MindFormerConfig(config_path) Note: Put cifar100 dataset to ./ The detailed data setting could refer to ./configs/clip/clip.md >>> config.eval_dataset_task.dataset_config.batch_size = 1 >>> dataset = build_dataset(config.eval_dataset_task) >>> for item in dataset: >>> print(item) >>> break [Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], [1.11282456e+000, 1.11282456e+000, ... 1.47778523e+000, 1.47778523e+000], ... [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000], [1.97748125e-001, 1.97748125e-001, ... 1.12205243e+000, 1.12205243e+000]]]]), Tensor(shape=[1, 100, 77], dtype=Int32, value= [[[49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], ... [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0], [49406, 320, 1674 ... 0, 0, 0]]]), Tensor(shape=[1], dtype=Int32, value= [49])] """ def __new__(cls, dataset_config: dict = None): """New method""" logger.info("Now Create Zero Shot Image Classification Dataset.") cls.init_dataset_config(dataset_config) rank_id = int(os.getenv("RANK_ID", "0")) device_num = int(os.getenv("RANK_SIZE", "1")) dataset = build_dataset_loader( dataset_config.data_loader, default_args={'num_shards': device_num, 'shard_id': rank_id}) transforms = build_transforms(dataset_config.transforms) tokenizer = build_tokenizer(dataset_config.tokenizer) text_transforms = build_transforms(dataset_config.text_transforms, default_args={"tokenizer": tokenizer}) if transforms is not None: dataset = get_dataset_map(dataset, input_columns="image", operations=transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) if text_transforms is not None: dataset = get_dataset_map(dataset, input_columns="text", operations=text_transforms, num_parallel_workers=dataset_config.num_parallel_workers, python_multiprocessing=dataset_config.python_multiprocessing) dataset = dataset.batch(dataset_config.batch_size, drop_remainder=dataset_config.drop_remainder, num_parallel_workers=dataset_config.num_parallel_workers) dataset = dataset.repeat(dataset_config.repeat) return dataset pengcheng.mind-mindformer/mindformers/generation/000077500000000000000000000000001476650141400225725ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/generation/__init__.py000066400000000000000000000017301476650141400247040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Mindformers generation.""" from .generation_config import * from .logits_process import * from .streamers import * from .text_generator import * __all__ = [] __all__.extend(generation_config.__all__) __all__.extend(logits_process.__all__) __all__.extend(streamers.__all__) __all__.extend(text_generator.__all__) pengcheng.mind-mindformer/mindformers/generation/__pycache__/000077500000000000000000000000001476650141400250025ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/generation/__pycache__/__init__.cpython-39.pyc000066400000000000000000000006101476650141400311670ustar00rootroot00000000000000a ?cf@s\dZddlTddlTddlTddlTgZeejeejeejeejdS)z Mindformers generation.)*N)__doc__Zgeneration_configZlogits_processZ streamersZtext_generator__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/generation/__init__.pys   pengcheng.mind-mindformer/mindformers/generation/__pycache__/generation_config.cpython-39.pyc000066400000000000000000000164161476650141400331230ustar00rootroot00000000000000a ?cf @sLdZddlZddlmZmZddlmZddlmZdgZ GdddZ dS)zgeneration config.N)AnyDict) BaseConfig)loggerGenerationConfigc@sreZdZdZddZedddZeeee fddd Z ee dd d d Z d dZ eee fdddZdS)ra Class that holds a configuration for a generation task. Arg: > Parameters that control the length of the output max_length (`int`, *optional*, defaults to 20): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (`int`, *optional*): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. > Parameters that control the generation strategy used do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. use_past (`bool`, *optional*, defaults to `False`): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. > Parameters for manipulation of the model output logits temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. top_k (`int`, *optional*, defaults to 50): The number of highest probability vocabulary tokens to keep for top-k-filtering. top_p (`float`, *optional*, defaults to 1.0): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. encoder_repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. renormalize_logits (`bool`, *optional*, defaults to `False`): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. > Special tokens that can be used at generation time pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*): The id of the *beginning-of-sequence* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. > Wild card generation_kwargs: Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass. c Ks>|dd|_|d|j|_|dd|_|dd|_|dd|_|dd |_|d d |_|d d |_|d d |_|dd |_ |dd|_ |dd|_ |dd|_ |dd|_ |dd|_|js:|D]T\}}zt|||Wqty6}z td||||WYd}~qd}~00qdS)NZmax_decode_length max_lengthmax_new_tokens do_sampleFuse_past temperatureg?top_k2top_prepetition_penaltyencoder_repetition_penaltyrenormalize_logits pad_token_id bos_token_id eos_token_id_from_model_configz!Can't set %s with value %s for %s)poprr r r r r rrrrrrrritemssetattrAttributeErrorrerror)selfkwargskeyvalueerrr!S/home/ma-user/work/mindformers-baichuan/mindformers/generation/generation_config.py__init__Qs,zGenerationConfig.__init__)returncCs t|jS)N)str__dict__)rr!r!r"__str__vszGenerationConfig.__str__) config_dictcKsN|dd}|fii||}|jfi|}td||rJ||fS|S)a Instantiates a [`GenerationConfig`] from a Python dictionary of parameters. Args: config_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the configuration object. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the configuration object. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. return_unused_kwargsFzGenerate config %s)rupdaterdebug)clsr(rr)config unused_kwargsr!r!r" from_dictys  zGenerationConfig.from_dict) model_configr$cCs(|}|dd|j|ddd}|S)a Instantiates a [`GenerationConfig`] from a [`PretrainedConfig`]. This function is useful to convert legacy [`BaseConfig`] objects, which may contain generation parameters, into a stand-alone [`GenerationConfig`]. Args: model_config (`PretrainedConfig`): The model config that will be used to instantiate the generation config. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. rNFT)r)r)to_dictrr/)r,r0r(r-r!r!r"from_model_configs z"GenerationConfig.from_model_configc sPg|D](\}}t||r t||||q fdd|D}|S)a Updates attributes of this class instance with attributes from `kwargs` if they match existing atributtes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. csi|]\}}|vr||qSr!r!).0rr to_remover!r" z+GenerationConfig.update..)rhasattrrappend)rrrrr.r!r4r"r*s    zGenerationConfig.updatecCst|j}|S)zto dict convert function.)copydeepcopyr&)routputr!r!r"r1s zGenerationConfig.to_dictN)__name__ __module__ __qualname____doc__r#r%r' classmethodrrr/rr2r*r1r!r!r!r"rs7%) r@r:typingrrZmindformers.models.base_configrmindformers.toolsr__all__rr!r!r!r"s   pengcheng.mind-mindformer/mindformers/generation/__pycache__/logits_process.cpython-39.pyc000066400000000000000000000201731476650141400324750ustar00rootroot00000000000000a ?cf"@sdZddlZddlZddlmZmZmZgdZGdddZ Gdd d Z Gd d d e Z Gd d d e Z Gddde ZGddde ZGddde ZGddde e ZdS)z Logits Processor for generation.N) log_softmaxsoftmaxtopk)LogitsProcessor LogitsWarperLogitsProcessorListc@seZdZdZddZdS)rzSAbstract base class for all logit processors that can be applied during generation.cCst|jddS)z#Torch method for processing logits.H is an abstract class. Only classes inheriting this class can be called.NNotImplementedError __class__self input_idsscoresrP/home/ma-user/work/mindformers-baichuan/mindformers/generation/logits_process.py__call__s zLogitsProcessor.__call__N__name__ __module__ __qualname____doc__rrrrrrsrc@seZdZdZddZdS)rznAbstract base class for all logit warpers that can be applied during generation with multinomial sampling.cCst|jddS)z Torch method for warping logits.r Nr r rrrr&s zLogitsWarper.__call__Nrrrrrr"src@seZdZdZddZdS)ra This class can be used to create a list of [`LogitsProcessor`] or [`LogitsWarper`] to subsequently process a `scores` input tensor. This class inherits from list and adds a specific *__call__* method to apply each [`LogitsProcessor`] or [`LogitsWarper`] to the inputs. c s|D]}t|jj}t|dkr|tfddt|ddDshtdt|d|j d|||fi}q|||}q|S)Nc3s|]}|vVqdSNr).0argkwargsrr 8z/LogitsProcessorList.__call__..z,Make sure that all the required parameters: z for z$ are passed to the logits processor.) inspect signaturer parameterslenalllistkeys ValueErrorr )rrrr processor function_argsrrrr4s & zLogitsProcessorList.__call__Nrrrrrr-src@s&eZdZdZedddZddZdS)TemperatureLogitsWarperz [`LogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`): The value used to module the logits distribution.  temperaturecCs(t|}|dkrtd|||_dS)Nrz:`temperature` has to be a strictly positive float, but is )floatr(r-)rr-rrr__init__Ls z TemperatureLogitsWarper.__init__cCs||j}|Srr,r rrrrUs z TemperatureLogitsWarper.__call__Nrrrrr.r/rrrrrr+Cs r+c@s&eZdZdZedddZddZdS) RepetitionPenaltyLogitsProcessora# [`LogitsProcessor`] enforcing an exponential penalty on repeated sequences. Args: repetition_penalty (`float`): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. )penaltycCs(t|}|dkrtd|||_dS)Nrz6`penalty` has to be a strictly positive float, but is )r.r(r2)rr2rrrr/ds z)RepetitionPenaltyLogitsProcessor.__init__cCsXtj||dd}|dk}|}|||j||<|||j||<tj|||dd|S)Nraxisr)nptake_along_axisr2put_along_axis)rrrscoreZnegative_indexZpositive_indexrrrrmsz)RepetitionPenaltyLogitsProcessor.__call__Nr0rrrrr1Zs  r1c@s.eZdZdZd eeeedddZdd Zd S) TopPLogitsWarpera  [`LogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`): If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. filter_value (`float`, *optional*, defaults to `-50000`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. candicate_token_num (`int`, *optional*, defaults to 200): Number of candicate tokens to calculate top_p. this can avoid sorting a huge seq, save time to speed up generation. 0 and < 1, but is z>`min_tokens_to_keep` has to be a non-negative integer, but is )r.r( isinstanceintr<r=r>r?)rr<r=r>r?rrrr/s zTopPLogitsWarper.__init__cCst||j\}}t|}tj|dd}||jk}tjtj|jddfd tj |dddfgdd}d|dd|j f<t | tj }tj |||dd|j||<|S)Nr3rr)shape.)rr?rr5cumsumr< concatenateonesrCastypebool_r> ones_liker7r=)rrrZcandicate_logitsZcandicate_indicesZcumulative_probsZsorted_indices_to_keepindices_to_removerrrrs ,  zTopPLogitsWarper.__call__N)r:rr;)rrrrr.rAr/rrrrrr9zs r9c@s4eZdZdZd eeedddZejddd Z d S) TopKLogitsWarpera [`LogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): The number of highest probability vocabulary tokens to keep for top-k-filtering. filter_value (`float`, *optional*, defaults to `-float("Inf")`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. r:r)top_kr=r>cCs:t|tr|dkr td|t|||_t||_dS)Nrz6`top_k` has to be a strictly positive integer, but is )r@rAr(maxrLr.r=)rrLr=r>rrrr/s  zTopKLogitsWarper.__init__)rcCs6t|j|jd}|t||ddk}|j||<|S)NrBr).rBN)minrLrCrr=)rrrrLrJrrrrs zTopKLogitsWarper.__call__N)r:r) rrrrrAr.r/r5ndarrayrrrrrrKs  rKc@seZdZdZddZdS)LogitNormalizationa [`LogitsWarper`] and [`LogitsProcessor`] for normalizing the scores using log-softmax. It's important to normalize the scores during beam search, after applying the logits processors or warpers, since the search algorithm used in this library doesn't do it (it only does it before, but they may need re-normalization) but it still supposes that the scores are normalized when comparing the hypotheses. cCst|dd}|S)NrBr3)rr rrrrs zLogitNormalization.__call__NrrrrrrPsrP)rr!numpyr5utilsrrr__all__rrr&rr+r1r9rKrPrrrrs   8pengcheng.mind-mindformer/mindformers/generation/__pycache__/streamers.cpython-39.pyc000066400000000000000000000212431476650141400314420ustar00rootroot00000000000000a ?cf,@sjdZddlmZddlmZddlZddlmZgdZ GdddZ Gd d d e Z Gd d d e Z dS) zStreamers for text generation.)Queue)OptionalN) BaseTokenizer) BaseStreamer TextStreamerTextIteratorStreamerc@s eZdZdZddZddZdS)rzG Base class from which `.generate()` streamers should inherit. cCs tdS)z;Function that is called by `.generate()` to push new tokensNNotImplementedErrorselfvaluer K/home/ma-user/work/mindformers-baichuan/mindformers/generation/streamers.pyputszBaseStreamer.putcCs tdS)zHFunction that is called by `.generate()` to signal the end of generationNrr r r rend#szBaseStreamer.endN)__name__ __module__ __qualname____doc__rrr r r rrsrc@sZeZdZdZdeeeedddZdd Zd d Z d d Z de edddZ ddZ dS)ra- Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` NFT) tokenizer skip_promptskip_special_tokenscKs:||_||_||_||_d|_g|_d|_d|_d|_dS)NFrT) rrr decode_kwargs batch_stream token_cache text_cache print_lennext_tokens_are_prompt)r rrrrr r r__init__KszTextStreamer.__init__cCst|tr|j|t|tjr*|}t|trt|dkrbt|dtrb|j sd|_ d|_ qt|dkrt|dtr|d}|j rd|_ g|_d|_ |j |nt d|jr|jrd|_dS|j r|jj||jfi|j}n|jj|j|jfi|j}||}||dS)zm Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. rTrFzJTextStreamer only supports int, or 1 ~ 2 dim numpy.ndarray/list as inputs.N) isinstanceintrappendnpndarraytolistlistlenrrrextend ValueErrorrrr batch_decoderrdecodeget_printable_texton_finalized_text)r r textprintable_textr r rr\s2      zTextStreamer.putcCs|jr |S|dr0||jd}g|_d|_nb|rh|t|drh||jd}|jt|7_n*||j|dd}|jt|7_|S)z/Get printable text when a new element comes in. Nr r!)rendswithrr_is_chinese_charordr)rfindr r0r1r r rr.s zTextStreamer.get_printable_textcCsf|js8|jr8|jj|j|jfi|j}||jd}nd}|j|ddd|_g|_d|_ d|_dS)z;Flushes any remaining cache and prints a newline to stdout.NrT) stream_endr) rrrr-rrrr/rrr9r r rrs zTextStreamer.endr0r:cCsj|jrP|js||_n|r0ddt|j|D|_td|jd|sFdnddnt|d|s^dndddS)zNPrints the new text to stdout. If the stream is ending, also prints a newline.cSsg|]\}}||qSr r ).0ijr r r z2TextStreamer.on_finalized_text.. TrN)flushr)rrzipprintr r0r:r r rr/s zTextStreamer.on_finalized_textcCsd|krdksnd|kr*dksnd|kr@dksnd|krVdksnd |krld ksnd |krd ksnd |krdksnd|krdkrnndSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr )r cpr r rr6sD  zTextStreamer._is_chinese_char)NFT)F)rrrrrrboolr rr.rstrr/r6r r r rr(s# ) rcsVeZdZdZdeeeeedfdd Zde eddd Z d d Z d d Z Z S)ra Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight,' ``` NF)rrtimeoutc s.tj||fi|t|_d|_||_dSN)superr r text_queue stop_signalrI)r rrrIr __class__r rr szTextIteratorStreamer.__init__r;cCs2|r|jj||jd|r.|jj|j|jddS)z\Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.rIN)rLrrIrMrEr r rr/sz&TextIteratorStreamer.on_finalized_textcCs|SrJr rr r r__iter__szTextIteratorStreamer.__iter__cCs$|jj|jd}||jkr t|S)NrP)rLgetrIrM StopIterationr r r r__next__s zTextIteratorStreamer.__next__)NFN)F)rrrrrrrGfloatr rHr/rQrT __classcell__r r rNrrs/ r) rqueuertypingrnumpyr%!mindformers.models.base_tokenizerr__all__rrrr r r rs   (pengcheng.mind-mindformer/mindformers/generation/__pycache__/text_generator.cpython-39.pyc000066400000000000000000000420311476650141400324650ustar00rootroot00000000000000a ?cfk@sdZddlZddlZddlmZmZmZddlZddl m Z ddl m Z ddlmmZddlmZddlmZdd lmZdd lmZmZmZmZmZmZddlmZdd l m!Z!d d l"m#Z#dgZ$GdddZ%dS)z For text generation N)OptionalListUnion) functional) operations)Tensor) BaseStreamer)GenerationConfig)LogitNormalizationLogitsProcessorList RepetitionPenaltyLogitsProcessorTemperatureLogitsWarperTopKLogitsWarperTopPLogitsWarper)softmax)loggerGeneratorMixinc@seZdZdZddZddZedddZed d d Zd&ddZ e e e dddZ e e dddZe dddZddZd'ddZed ddZd(e e e e e ed d!d"Zd)e eeeeeefe e e e e ee ed#d$d%Zd S)*rzGenerator For the nlp modelscCsdS)N)selfrrP/home/ma-user/work/mindformers-baichuan/mindformers/generation/text_generator.py__init__-szGeneratorMixin.__init__cKs tddS)a prepare inputs for generation. A model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()` Raises: RuntimeError: Not implemented in model but call `.generate()` zeA model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()`.N) RuntimeError)r input_idskwargsrrrprepare_inputs_for_generation1s z,GeneratorMixin.prepare_inputs_for_generation) model_kwargscCsdS)z update model kwargs before generate. If your model needs to update model kwargs before generate, implement this method in your model, else do nothing. Nr)rrrrrr#update_model_kwargs_before_generate@sz2GeneratorMixin.update_model_kwargs_before_generate) model_inputscCs|d}t|tr|}g}t|D]8\}}t|||jd}|||||dq(tj |tj d}t|t j |d<dS)zr?Z processorsrrr_get_logits_processoras   z$GeneratorMixin._get_logits_processor) default_list custom_listcCsv|s|S|D]Z}|D]P}t|t|urd}td|dt|d|d|d|d|d|d qq |||S) z.merge custom processor list with default list.zlogits processorz A custom z of type z with values zS has been passed to `.generate()`, but it has already been created with the values z. z has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of zO consider passing them as arguments to `.generate()` instead of using a custom .)type ValueErrorextend)rrFrGdefaultcustomZ object_typerrrrCss& z$GeneratorMixin._merge_processor_list)r>cCst}|jdur*|jdkr*|t|jd}|jdurV|jdkrV|t|j|d|jdur~|jdkr~|t|j|d|jdur|t |S)z This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. Nr@r r)top_kmin_tokens_to_keep)top_prOT) r temperaturer'rrNrrPrrDr )rr>ZwarpersrOrrr_get_logits_warpers  z!GeneratorMixin._get_logits_warpercCsp|jd}t|tj}|t|tj|}t||jj f}t d|jt |}d|dddf<||||fS)z#generate the inputs for the decoderrz-Decoder: pad the origin inputs into shape: %sr N) r&rr+float32Zencoder_forwardr*r(zerosconfigmax_decode_lengthrdebug zeros_like)rr input_mask batch_size encoder_maskencoder_output target_maskrrr!_prepare_model_inputs_for_decoders    z0GeneratorMixin._prepare_model_inputs_for_decoderrcCsX|jj|jd}|dkr6td|jd|jjdtj|dd|ffdd|fd}|S) z#pad the input_ids to the max_lengthr0rzorigin_inputs size is z1, you shouldincrease the seq_length of the model rH)rrconstant)constant_values)rU seq_lengthr&rJr(pad)r origin_inputs pad_token_idZ pad_lengthrrrr_pad_inputs_using_max_lengths  z+GeneratorMixin._pad_inputs_using_max_lengthcCs|jr\|jddt|tj|d<tdgtj|d<t|gtj|d<|fi|}d|_nZ|jdd|||t|tj|d<tdgtj|d<t|gtj|d<|fi|}|S)z$model forward for incremental infer.T)is_first_iterationZinput_positionF init_resetbatch_valid_length)rfadd_flags_recursiverr+r*bool_r/)rrr,valid_length_each_exampleresrrr_incremental_infers$   z!GeneratorMixin._incremental_infer)r>r? logits_warperstreamerc% st}t}|jdur d|_|dur2|||jd} |jj} td|jgt| D](}  t t || |jkdq^t tdt } |jdur|j| |_|j|jjkrtd|j|jj|jj|_td|j| s"| |jkr"td| d |jd |j||jd } td |j| jt | }tjdD]} d|| d| f<q\d}d}| r|j|jjkr|jj|_td |j|| |\}}} }ddt| Ddg| }|| ||jrd|_d}t }t|}td|t || kr8t}| jdfddt| D}td| rt| tj}|d|||t|tjd}nH||d<|j| fi|}|jr|j}|j ||d}n|fi|}t|}t}|jj!st"|t#r |dn|}t"|tr$|$}t %|d|jdf}|r\|jdt&|kr\||}|| |}|| |}t 't (|jd| df}n0|\}}t"|tr|$}t"|tr|$}t|}t}ddt| D}t| D]} || rq|j)r,t*|| } t j+j,t&|| | d}!nt -|| }!|| |!}"|"| | | f<|durj|"g|| <| rt.d|| | <| t.d7<d|| | d<|| |!|j/ksЈ| |jkrd|| <qq|dur| dkr||dn ||t|}td||||||qg}#t| D],} |# | | dt.| f0t jqDtd|#|dur|1t |}$t|}t2d||$|$||#S)a. Text generation given the model and origin inputs Inputs: origin_inputs(list): The prompt for generation, should be a list of ids. generation_config(GenerationConfig): The controlling config for text generation. streamer: Streamer object that will be used to stream the generated sequences. model_kwargs: dict of model input kwargs, when be passed to model construct during the generation forward. Returns: outputs: the ids for the generated text NrzThe input shape is: %sr z%Get the valid for each example is: %szOmax_length %s can not exceeds model seq_length %s, set max_length = seq_length.zmax length is: %szthe input_ids length z exceeds the max length config zE.check your inputs and set max_length larger than your inputs length.)rcrdz,pad the origin inputs from %s into shape: %szmax decode length is: %scSsg|]}dqSr r.0_rrr 3z+GeneratorMixin._forward..FTzforward prepare time: %s scs g|]}|d|qSrpr)rrr.rarkrrrtGszvalidate length: %s)rattention_maskZencoder_outputsZdecoder_input_idsZdecoder_attention_maskr,)rr,rkr0cSsg|]}gqSrrrqrrrrtru)pzKforward time: %s s; sample time: %s s; update time: %s s; total count: %s szThe output is: %szJtotal time: %s s; generated tokens: %s tokens; generate speed: %s tokens/s)3timerdputr&rUis_encoder_decoderrrWranger'r(maxargwherer)max_new_tokens max_lengthrawarningrJrerXrVr^ruse_pastrfsumrr+r*rSrrmZis_sample_accelerationr"tupler#r1lentilearange do_samplerrandomchoiceargmaxr% eos_token_idastypeendinfo)%rrcr>r?rnror total_timeZ prepare_timerZr{r.Zinput_ids_lengthrrYr\r[r]Z is_finishedZneed_gather_logitsZ origin_lenZ forward_timer,inputsrlrZ sample_timer9probsZp_argsZ update_time target_listp_norm target_indextarget output_idsZ generate_lenrrvr_forwards6                                                zGeneratorMixin._forward)rr>r?roseedc Ks|j}|dt|dt|d}|dur8dn|}tj||durT|nt}|durnt |j }t |}|jfi|}|jsd|_d|_td||j||d}||} |jf|||| |d|} ||d k| S) u Generate the words according to the given the input ids. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, top_k=3, do_sample=True)`. Args: input_ids(List(str), List(List(str))): The token id list or a batch of token id list. When input a batch of token id list, the length of each token id list should be same. generation_config (`GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default config from the model configuration will be used. Please note that unspecified parameters will inherit [`GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. streamer: The streamer that generator uses. seed: Random seed used in sample. kwargs: Specific parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Supported `generate_config` keywords can be checked in [`GenerationConfig`]'s documentation. Mainly used Keywords are shown below: max_length(int): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (int): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. pad_token_id(int): The pad token id. If set None, it follows the setting in the configureation in the model. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default None. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> t5 = T5ForConditionalGeneration.from_pretrained("t5_small") >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" >>> words = tokenizer(words, max_length=21, padding='max_length')['input_ids'] >>> output = t5.generate(words, do_sample=True) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top p sampling >>> output = t5.generate(words, do_sample=True, top_p=0.4) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top k sampling. >>> output = t5.generate(words, do_sample=True, top_k=10, top_p=1) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) Este comist de stat ale stateului membre nai uzusepa şi ONU Returns: A list of the generated token ids Fr0Nrr@zGeneration Config is: %sr=)rcr>r?rnrotrain)phase set_trainr(r)r1r&rrr r from_model_configrUcopydeepcopyupdaterrPrNrrWrErRr) rrr>r?rorr origin_phaserrnrrrrgenerates@S       zGeneratorMixin.generate)NF)r)NNN)NNNN)__name__ __module__ __qualname____doc__rrdictrr/r<r rr rErCrRr^rermrrrrr%rrrrrr*sJ     e)&rrrytypingrrrnumpyr( mindspore.opsrr6rr3mindspore.common.dtypecommonr r+mindspore.common.tensorrZ mindformers.generation.streamersrr>r logits_processr r r rrr streamersutilsrtoolsr__all__rrrrrs          pengcheng.mind-mindformer/mindformers/generation/__pycache__/utils.cpython-39.pyc000066400000000000000000000030641476650141400305760ustar00rootroot00000000000000a ?cfn @s.dZddlZd ddZd ddZd d d ZdS)zutils for text generation.NcCstj||dd}|jdkr,d|t|<nt|s:d}||}t|}tjdd*tj||dd}t|}Wdn1s0Y||}|S)z}numpy implemented log softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.pyTaxiskeepdimsrignore)divideN)npamaxndimisfiniteexperrstatesumlog)xrx_maxtmpexp_tmpsoutrG/home/ma-user/work/mindformers-baichuan/mindformers/generation/utils.py log_softmaxs   (rcCs2tj||dd}t||}|tj||ddS)zynumpy implemented softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.pyTr)rrr r )rrr exp_x_shiftedrrrsoftmax*srTcCs|j||kr|j|d}|r4tj| ||d}ntj|||d}tj|t||d}tj|||d}|r|rtj| |dn tj||d}tj|||d}tj|||d}||fS)znumpy implemented topk sample.)r)shaper argpartitiontakearangetake_along_axisargsort)rtop_krlargestsortZ topk_indexZ topk_data sort_indexrrrtopk2s r&)N)N)rTT)__doc__numpyrrrr&rrrrs  pengcheng.mind-mindformer/mindformers/generation/generation_config.py000066400000000000000000000203551476650141400266310ustar00rootroot00000000000000# Copyright 2022 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generation config.""" import copy from typing import Any, Dict from mindformers.models.base_config import BaseConfig from mindformers.tools import logger __all__ = ["GenerationConfig"] class GenerationConfig: """Class that holds a configuration for a generation task. Arg: > Parameters that control the length of the output max_length (`int`, *optional*, defaults to 20): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (`int`, *optional*): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. > Parameters that control the generation strategy used do_sample (`bool`, *optional*, defaults to `False`): Whether or not to use sampling ; use greedy decoding otherwise. use_past (`bool`, *optional*, defaults to `False`): Whether or not the model should use the past last key/values attentions (if applicable to the model) to speed up decoding. > Parameters for manipulation of the model output logits temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. top_k (`int`, *optional*, defaults to 50): The number of highest probability vocabulary tokens to keep for top-k-filtering. top_p (`float`, *optional*, defaults to 1.0): If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. encoder_repetition_penalty (`float`, *optional*, defaults to 1.0): The parameter for encoder_repetition_penalty. An exponential penalty on sequences that are not in the original input. 1.0 means no penalty. renormalize_logits (`bool`, *optional*, defaults to `False`): Whether to renormalize the logits after applying all the logits processors or warpers (including the custom ones). It's highly recommended to set this flag to `True` as the search algorithms suppose the score logits are normalized but some logit processors or warpers break the normalization. > Special tokens that can be used at generation time pad_token_id (`int`, *optional*): The id of the *padding* token. bos_token_id (`int`, *optional*): The id of the *beginning-of-sequence* token. eos_token_id (`Union[int, List[int]]`, *optional*): The id of the *end-of-sequence* token. Optionally, use a list to set multiple *end-of-sequence* tokens. > Wild card generation_kwargs: Additional generation kwargs will be forwarded to the `generate` function of the model. Kwargs that are not present in `generate`'s signature will be used in the model forward pass. """ def __init__(self, **kwargs): # max generate length self.max_length = kwargs.pop("max_decode_length", 20) self.max_length = kwargs.pop("max_length", self.max_length) self.max_new_tokens = kwargs.pop("max_new_tokens", None) # do sample or not self.do_sample = kwargs.pop("do_sample", False) # incremental infer self.use_past = kwargs.pop("use_past", False) # logits processors self.temperature = kwargs.pop("temperature", 1.0) self.top_k = kwargs.pop("top_k", 50) self.top_p = kwargs.pop("top_p", 1.0) self.repetition_penalty = kwargs.pop("repetition_penalty", 1.0) self.encoder_repetition_penalty = kwargs.pop("encoder_repetition_penalty", 1.0) self.renormalize_logits = kwargs.pop("renormalize_logits", False) # Special tokens that can be used at generation time self.pad_token_id = kwargs.pop("pad_token_id", None) self.bos_token_id = kwargs.pop("bos_token_id", None) self.eos_token_id = kwargs.pop("eos_token_id", None) # interface. self._from_model_config = kwargs.pop("_from_model_config", False) # Additional attributes without default values if not self._from_model_config: # we don't want to copy values from the model config # if we're initializing a `GenerationConfig` from a # model's default configuration file for key, value in kwargs.items(): try: setattr(self, key, value) except AttributeError as err: logger.error("Can't set %s with value %s for %s", key, value, self) raise err def __str__(self) -> str: return str(self.__dict__) @classmethod def from_dict(cls, config_dict: Dict[str, Any], **kwargs): """ Instantiates a [`GenerationConfig`] from a Python dictionary of parameters. Args: config_dict (`Dict[str, Any]`): Dictionary that will be used to instantiate the configuration object. kwargs (`Dict[str, Any]`): Additional parameters from which to initialize the configuration object. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. """ return_unused_kwargs = kwargs.pop("return_unused_kwargs", False) config = cls(**{**config_dict, **kwargs}) unused_kwargs = config.update(**kwargs) logger.debug("Generate config %s", config) if return_unused_kwargs: return config, unused_kwargs return config @classmethod def from_model_config(cls, model_config: BaseConfig) -> "GenerationConfig": """ Instantiates a [`GenerationConfig`] from a [`PretrainedConfig`]. This function is useful to convert legacy [`BaseConfig`] objects, which may contain generation parameters, into a stand-alone [`GenerationConfig`]. Args: model_config (`PretrainedConfig`): The model config that will be used to instantiate the generation config. Returns: [`GenerationConfig`]: The configuration object instantiated from those parameters. """ config_dict = model_config.to_dict() config_dict.pop("_from_model_config", None) config = cls.from_dict(config_dict, return_unused_kwargs=False, _from_model_config=True) return config def update(self, **kwargs): """ Updates attributes of this class instance with attributes from `kwargs` if they match existing atributtes, returning all the unused kwargs. Args: kwargs (`Dict[str, Any]`): Dictionary of attributes to tentatively update this class. Returns: `Dict[str, Any]`: Dictionary containing all the key-value pairs that were not used to update the instance. """ to_remove = [] for key, value in kwargs.items(): if hasattr(self, key): setattr(self, key, value) to_remove.append(key) unused_kwargs = {key: value for key, value in kwargs.items() if key not in to_remove} return unused_kwargs def to_dict(self) -> Dict[str, Any]: """to dict convert function.""" output = copy.deepcopy(self.__dict__) return output pengcheng.mind-mindformer/mindformers/generation/logits_process.py000066400000000000000000000213011476650141400262000ustar00rootroot00000000000000# Copyright 2020 The HuggingFace Inc. team # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Logits Processor for generation.""" import inspect import numpy as np from .utils import log_softmax, softmax, topk __all__ = ["LogitsProcessor", "LogitsWarper", "LogitsProcessorList"] class LogitsProcessor: """Abstract base class for all logit processors that can be applied during generation.""" def __call__(self, input_ids, scores): """Torch method for processing logits.""" raise NotImplementedError( f"{self.__class__} is an abstract class. Only classes inheriting this class can be called." ) class LogitsWarper: """Abstract base class for all logit warpers that can be applied during generation with multinomial sampling.""" def __call__(self, input_ids, scores): """Torch method for warping logits.""" raise NotImplementedError( f"{self.__class__} is an abstract class. Only classes inheriting this class can be called." ) class LogitsProcessorList(list): """ This class can be used to create a list of [`LogitsProcessor`] or [`LogitsWarper`] to subsequently process a `scores` input tensor. This class inherits from list and adds a specific *__call__* method to apply each [`LogitsProcessor`] or [`LogitsWarper`] to the inputs. """ def __call__(self, input_ids, scores, **kwargs): for processor in self: function_args = inspect.signature(processor.__call__).parameters if len(function_args) > 2: if not all(arg in kwargs for arg in list(function_args.keys())[2:]): raise ValueError( f"Make sure that all the required parameters: {list(function_args.keys())} for " f"{processor.__class__} are passed to the logits processor." ) scores = processor(input_ids, scores, **kwargs) else: scores = processor(input_ids, scores) return scores class TemperatureLogitsWarper(LogitsWarper): r""" [`LogitsWarper`] for temperature (exponential scaling output probability distribution). Args: temperature (`float`): The value used to module the logits distribution. """ def __init__(self, temperature: float): temperature = float(temperature) if temperature <= 0: raise ValueError( f"`temperature` has to be a strictly positive float, but is {temperature}" ) self.temperature = temperature def __call__(self, input_ids, scores): scores = scores / self.temperature return scores class RepetitionPenaltyLogitsProcessor(LogitsProcessor): r""" [`LogitsProcessor`] enforcing an exponential penalty on repeated sequences. Args: repetition_penalty (`float`): The parameter for repetition penalty. 1.0 means no penalty. See [this paper](https://arxiv.org/pdf/1909.05858.pdf) for more details. """ def __init__(self, penalty: float): penalty = float(penalty) if penalty <= 0: raise ValueError( f"`penalty` has to be a strictly positive float, but is {penalty}" ) self.penalty = penalty def __call__(self, input_ids, scores): score = np.take_along_axis(scores, input_ids, axis=1) # if score < 0 then repetition penalty has to be multiplied to reduce the previous token probability negative_index = score < 0 positive_index = ~negative_index score[negative_index] = score[negative_index] * self.penalty score[positive_index] = score[positive_index] / self.penalty np.put_along_axis(scores, input_ids, score, axis=1) return scores class TopPLogitsWarper(LogitsWarper): """ [`LogitsWarper`] that performs top-p, i.e. restricting to top tokens summing to prob_cut_off <= prob_cut_off. Args: top_p (`float`): If set to < 1, only the smallest set of most probable tokens with probabilities that add up to `top_p` or higher are kept for generation. filter_value (`float`, *optional*, defaults to `-50000`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. candicate_token_num (`int`, *optional*, defaults to 200): Number of candicate tokens to calculate top_p. this can avoid sorting a huge seq, save time to speed up generation. """ def __init__(self, top_p: float, filter_value: float = -50000, min_tokens_to_keep: int = 1, candicate_token_num: int = 200): top_p = float(top_p) if top_p < 0 or top_p > 1.0: raise ValueError(f"`top_p` has to be a float > 0 and < 1, but is {top_p}") if not isinstance(min_tokens_to_keep, int) or (min_tokens_to_keep < 0): raise ValueError( f"`min_tokens_to_keep` has to be a non-negative integer, but is {min_tokens_to_keep}" ) self.top_p = top_p self.filter_value = float(filter_value) self.min_tokens_to_keep = min_tokens_to_keep self.candicate_token_num = candicate_token_num def __call__(self, input_ids, scores): candicate_logits, candicate_indices = topk(scores, self.candicate_token_num) cumulative_probs = softmax(candicate_logits) cumulative_probs = np.cumsum(cumulative_probs, axis=-1) # Remove tokens with cumulative top_p above the threshold (token with 0 are kept) sorted_indices_to_keep = cumulative_probs < self.top_p # add the last token that exceed top_p sorted_indices_to_keep = np.concatenate( [np.ones(shape=(scores.shape[0], 1)).astype(np.bool_), sorted_indices_to_keep[..., :-1]], axis=-1 ) # Keep at least min_tokens_to_keep sorted_indices_to_keep[..., :self.min_tokens_to_keep] = 1 # set remove indices, filter negative value indices_to_remove = np.ones_like(scores).astype(np.bool_) np.put_along_axis( indices_to_remove, candicate_indices, ~sorted_indices_to_keep, axis=-1 ) scores[indices_to_remove] = self.filter_value return scores class TopKLogitsWarper(LogitsWarper): r""" [`LogitsWarper`] that performs top-k, i.e. restricting to the k highest probability elements. Args: top_k (`int`): The number of highest probability vocabulary tokens to keep for top-k-filtering. filter_value (`float`, *optional*, defaults to `-float("Inf")`): All filtered values will be set to this float value. min_tokens_to_keep (`int`, *optional*, defaults to 1): Minimum number of tokens that cannot be filtered. """ def __init__(self, top_k: int, filter_value: float = -50000, min_tokens_to_keep: int = 1): if not isinstance(top_k, int) or top_k <= 0: raise ValueError( f"`top_k` has to be a strictly positive integer, but is {top_k}" ) self.top_k = max(top_k, min_tokens_to_keep) self.filter_value = float(filter_value) def __call__(self, input_ids, scores: np.ndarray): top_k = min(self.top_k, scores.shape[-1]) # Safety check # Remove all tokens with a probability less than the last token of the top-k indices_to_remove = scores < topk(scores, top_k)[0][..., -1, None] scores[indices_to_remove] = self.filter_value return scores class LogitNormalization(LogitsProcessor, LogitsWarper): r""" [`LogitsWarper`] and [`LogitsProcessor`] for normalizing the scores using log-softmax. It's important to normalize the scores during beam search, after applying the logits processors or warpers, since the search algorithm used in this library doesn't do it (it only does it before, but they may need re-normalization) but it still supposes that the scores are normalized when comparing the hypotheses. """ def __call__(self, input_ids, scores): scores = log_softmax(scores, axis=-1) return scores pengcheng.mind-mindformer/mindformers/generation/streamers.py000066400000000000000000000262101476650141400251520ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/generation/streamers.py # ============================================================================ """Streamers for text generation.""" from queue import Queue from typing import Optional import numpy as np from mindformers.models.base_tokenizer import BaseTokenizer __all__ = ['BaseStreamer', 'TextStreamer', 'TextIteratorStreamer'] class BaseStreamer: """ Base class from which `.generate()` streamers should inherit. """ def put(self, value): """Function that is called by `.generate()` to push new tokens""" raise NotImplementedError() def end(self): """Function that is called by `.generate()` to signal the end of generation""" raise NotImplementedError() class TextStreamer(BaseStreamer): """ Simple text streamer that prints the token(s) to stdout as soon as entire words are formed. The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextStreamer(tok) >>> # Despite returning the usual output, the streamer will also print the generated text to stdout. >>> _ = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> 'An increasing sequence: one, two, three, four, five, six, seven, eight, nine, ten, eleven,' ``` """ def __init__(self, tokenizer: Optional[BaseTokenizer] = None, skip_prompt: bool = False, skip_special_tokens: bool = True, **decode_kwargs): self.tokenizer = tokenizer self.skip_prompt = skip_prompt self.skip_special_tokens = skip_special_tokens self.decode_kwargs = decode_kwargs # variables used in the streaming process self.batch_stream = False self.token_cache = [] self.text_cache = "" self.print_len = 0 self.next_tokens_are_prompt = True def put(self, value): """ Receives tokens, decodes them, and prints them to stdout as soon as they form entire words. """ if isinstance(value, int): self.token_cache.append(value) if isinstance(value, np.ndarray): value = value.tolist() if isinstance(value, list): if len(value) > 1 and isinstance(value[0], list): # switch from single mode to batch mode if not self.batch_stream: self.batch_stream = True self.text_cache = "" else: # batch that equals 1 if len(value) == 1 and isinstance(value[0], list): value = value[0] # switch from batch mode to single mode if self.batch_stream: self.batch_stream = False self.token_cache = [] self.print_len = 0 self.token_cache.extend(value) else: raise ValueError("TextStreamer only supports int, or 1 ~ 2 dim numpy.ndarray/list as inputs.") if self.skip_prompt and self.next_tokens_are_prompt: self.next_tokens_are_prompt = False return # Add the new token to the cache and decodes the entire thing. if self.batch_stream: text = self.tokenizer.batch_decode(value, self.skip_special_tokens, **self.decode_kwargs) else: text = self.tokenizer.decode(self.token_cache, self.skip_special_tokens, **self.decode_kwargs) printable_text = self.get_printable_text(text) self.on_finalized_text(printable_text) def get_printable_text(self, text): """Get printable text when a new element comes in.""" # for batch streamer, we directly return the text if self.batch_stream: return text # After the symbol for a new line, we flush the cache. if text.endswith("\n"): printable_text = text[self.print_len :] self.token_cache = [] self.print_len = 0 # If the last token is a CJK character, we print the characters. elif text and self._is_chinese_char(ord(text[-1])): printable_text = text[self.print_len :] self.print_len += len(printable_text) # Otherwise, prints until the last space char (simple heuristic to avoid printing incomplete words, # which may change with the subsequent token -- there are probably smarter ways to do this!) else: printable_text = text[self.print_len : text.rfind(" ") + 1] self.print_len += len(printable_text) return printable_text def end(self): """Flushes any remaining cache and prints a newline to stdout.""" # Flush the cache, if it exists if not self.batch_stream and self.token_cache: text = self.tokenizer.decode(self.token_cache, self.skip_special_tokens, **self.decode_kwargs) printable_text = text[self.print_len :] else: printable_text = "" self.on_finalized_text(printable_text, stream_end=True) # always reset values when stream ends self.next_tokens_are_prompt = True self.token_cache = [] self.text_cache = "" self.print_len = 0 def on_finalized_text(self, text: str, stream_end: bool = False): """Prints the new text to stdout. If the stream is ending, also prints a newline.""" if self.batch_stream: if not self.text_cache: self.text_cache = text elif text: self.text_cache = [i + j for i, j in zip(self.text_cache, text)] print(f'\r{self.text_cache}', flush=True, end="" if not stream_end else None) else: print(text, flush=True, end="" if not stream_end else None) def _is_chinese_char(self, cp): """Checks whether CP is the codepoint of a CJK character.""" # This defines a "chinese character" as anything in the CJK Unicode block: # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) # # Note that the CJK Unicode block is NOT all Japanese and Korean characters, # despite its name. The modern Korean Hangul alphabet is a different block, # as is Japanese Hiragana and Katakana. Those alphabets are used to write # space-separated words, so they are not treated specially and handled # like the all of the other languages. if ( (0x4E00 <= cp <= 0x9FFF) or (0x3400 <= cp <= 0x4DBF) or (0x20000 <= cp <= 0x2A6DF) or (0x2A700 <= cp <= 0x2B73F) or (0x2B740 <= cp <= 0x2B81F) or (0x2B820 <= cp <= 0x2CEAF) or (0xF900 <= cp <= 0xFAFF) or (0x2F800 <= cp <= 0x2FA1F) ): return True return False class TextIteratorStreamer(TextStreamer): """ Streamer that stores print-ready text in a queue, to be used by a downstream application as an iterator. This is useful for applications that benefit from accessing the generated text in a non-blocking way (e.g. in an interactive Gradio demo). The API for the streamer classes is still under development and may change in the future. Parameters: tokenizer (`BaseTokenizer`): The tokenized used to decode the tokens. skip_prompt (`bool`, *optional*, defaults to `False`): Whether to skip the prompt to `.generate()` or not. Useful e.g. for chatbots. timeout (`float`, *optional*): The timeout for the text queue. If `None`, the queue will block indefinitely. Useful to handle exceptions in `.generate()`, when it is called in a separate thread. decode_kwargs (`dict`, *optional*): Additional keyword arguments to pass to the tokenizer's `decode` method. Examples: ```python >>> from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextIteratorStreamer >>> from threading import Thread >>> tok = GPT2Tokenizer.from_pretrained("gpt2") >>> model = GPT2LMHeadModel.from_pretrained("gpt2") >>> inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) >>> streamer = TextIteratorStreamer(tok) >>> # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. >>> generation_kwargs = dict(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) >>> thread = Thread(target=model.generate, kwargs=generation_kwargs) >>> thread.start() >>> generated_text = "" >>> for new_text in streamer: ... generated_text += new_text >>> generated_text 'An increasing sequence: one, two, three, four, five, six, seven, eight,' ``` """ def __init__(self, tokenizer: Optional[BaseTokenizer] = None, skip_prompt: bool = False, timeout: Optional[float] = None, **decode_kwargs): super().__init__(tokenizer, skip_prompt, **decode_kwargs) self.text_queue = Queue() self.stop_signal = None self.timeout = timeout def on_finalized_text(self, text: str, stream_end: bool = False): """Put the new text in the queue. If the stream is ending, also put a stop signal in the queue.""" if text: self.text_queue.put(text, timeout=self.timeout) if stream_end: self.text_queue.put(self.stop_signal, timeout=self.timeout) def __iter__(self): return self def __next__(self): value = self.text_queue.get(timeout=self.timeout) if value == self.stop_signal: raise StopIteration() return value pengcheng.mind-mindformer/mindformers/generation/text_generator.py000066400000000000000000000656001476650141400262050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ For text generation """ import copy import time from typing import Optional, List, Union import numpy as np from mindspore.ops import functional as F from mindspore.ops import operations as P import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindformers.generation.streamers import BaseStreamer from .generation_config import GenerationConfig from .logits_process import (LogitNormalization, LogitsProcessorList, RepetitionPenaltyLogitsProcessor, TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper) from .streamers import BaseStreamer from .utils import softmax from ..tools import logger __all__ = ["GeneratorMixin"] class GeneratorMixin: """Generator For the nlp models""" def __init__(self): pass # pylint: disable=W0613 def prepare_inputs_for_generation(self, input_ids, **kwargs): """ prepare inputs for generation. A model class needs to define a `prepare_inputs_for_generation` method in order to use `.generate()` Raises: RuntimeError: Not implemented in model but call `.generate()` """ raise RuntimeError( "A model class needs to define a `prepare_inputs_for_generation`" " method in order to use `.generate()`." ) # pylint: disable=W0613 def update_model_kwargs_before_generate(self, input_ids, model_kwargs: dict): """ update model kwargs before generate. If your model needs to update model kwargs before generate, implement this method in your model, else do nothing. """ return def slice_incremental_inputs(self, model_inputs: dict, current_index): """used for non-first iterations, slice the inputs to length 1.""" input_ids = model_inputs.pop("input_ids") if isinstance(input_ids, Tensor): input_ids = input_ids.asnumpy() inputs_tmp = [] for i, index_value in enumerate(current_index): current_index_tmp = ( int(index_value) - i * input_ids.shape[1] ) # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp : current_index_tmp + 1]) inputs_tmp = np.array(inputs_tmp, dtype=np.int32) model_inputs["input_ids"] = Tensor(inputs_tmp, mstype.int32) def process_logits(self, logits, current_index=None, keep_all=False): """Process the logits""" logits = logits.reshape(-1, logits.shape[-1]) if not keep_all and current_index is not None: index = current_index.view(-1,) logits = P.Gather()(logits, index, 0) outputs = P.LogSoftmax(-1)(logits) outputs = F.tensor_pow(np.e, outputs) return outputs def _get_logits_processor(self, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList]): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. """ # instantiate processors list processors = LogitsProcessorList() if generation_config.repetition_penalty is not None and generation_config.repetition_penalty != 1.0: processors.append(RepetitionPenaltyLogitsProcessor(penalty=generation_config.repetition_penalty)) processors = self._merge_processor_list(processors, logits_processor) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: processors.append(LogitNormalization()) return processors def _merge_processor_list(self, default_list: LogitsProcessorList, custom_list: LogitsProcessorList): """merge custom processor list with default list.""" if not custom_list: return default_list for default in default_list: for custom in custom_list: if type(custom) is type(default): object_type = "logits processor" raise ValueError( f"A custom {object_type} of type {type(custom)} with values {custom} has been passed to" f" `.generate()`, but it has already been created with the values {default}." f" {default} has been created by passing the corresponding arguments to generate or" f" by the model's config default values. If you just want to change the default values" f" of {object_type} consider passing them as arguments to `.generate()`" f" instead of using a custom {object_type}." ) default_list.extend(custom_list) return default_list def _get_logits_warper(self, generation_config: GenerationConfig): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. """ # instantiate warpers list warpers = LogitsProcessorList() # all samplers can be found in `generation_utils_samplers.py` if generation_config.temperature is not None and generation_config.temperature != 1.0: warpers.append(TemperatureLogitsWarper(generation_config.temperature)) min_tokens_to_keep = 1 if generation_config.top_k is not None and generation_config.top_k != 0: warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep)) if generation_config.top_p is not None and generation_config.top_p < 1.0: warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep)) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: warpers.append(LogitNormalization()) return warpers def _prepare_model_inputs_for_decoder(self, input_ids, input_mask): """generate the inputs for the decoder""" batch_size = input_ids.shape[0] encoder_mask = Tensor(input_mask, mstype.float32) encoder_output = self.encoder_forward( Tensor(input_ids, mstype.int32), encoder_mask ) input_ids = np.zeros((batch_size, self.config.max_decode_length)) logger.debug("Decoder: pad the origin inputs into shape: %s", input_ids.shape) target_mask = np.zeros_like(input_ids) target_mask[:, 0] = 1 # As the decoder is generating from [START] token return encoder_output, encoder_mask, input_ids, target_mask def _pad_inputs_using_max_length(self, origin_inputs, pad_token_id=0): """pad the input_ids to the max_length""" pad_length = self.config.seq_length - origin_inputs.shape[-1] if pad_length < 0: raise ValueError( f"origin_inputs size is {origin_inputs.shape}, you should" f"increase the seq_length of the model {self.config.seq_length}." ) # Pad original inputs to model_origin_max_length input_ids = np.pad( origin_inputs, ((0, 0), (0, pad_length)), "constant", constant_values=(0, pad_token_id), ) return input_ids def _incremental_infer(self, model_inputs: dict, current_index, valid_length_each_example): """model forward for incremental infer.""" # Claim the first graph if self.is_first_iteration: self.add_flags_recursive(is_first_iteration=True) model_inputs["input_position"] = Tensor(current_index, mstype.int32) model_inputs["init_reset"] = Tensor([False], mstype.bool_) # init_reset (1,) bool False model_inputs["batch_valid_length"] = Tensor([valid_length_each_example], mstype.int32) # pylint: disable=E1102 res = self( **model_inputs, ) # first iter done, go to other iters self.is_first_iteration = False else: self.add_flags_recursive(is_first_iteration=False) # slice model inputs for incremental infer self.slice_incremental_inputs(model_inputs, current_index) model_inputs["input_position"] = Tensor(current_index, mstype.int32) model_inputs["init_reset"] = Tensor([True], mstype.bool_) # init_reset (1,) bool True model_inputs["batch_valid_length"] = Tensor([valid_length_each_example], mstype.int32) # pylint: disable=E1102 res = self( **model_inputs, ) return res def _forward(self, origin_inputs, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList] = None, logits_warper: Optional[LogitsProcessorList] = None, streamer: BaseStreamer = None, **model_kwargs): """ Text generation given the model and origin inputs Inputs: origin_inputs(list): The prompt for generation, should be a list of ids. generation_config(GenerationConfig): The controlling config for text generation. streamer: Streamer object that will be used to stream the generated sequences. model_kwargs: dict of model input kwargs, when be passed to model construct during the generation forward. Returns: outputs: the ids for the generated text """ total_time = time.time() prepare_time = time.time() if generation_config.pad_token_id is None: generation_config.pad_token_id = 0 if streamer is not None: streamer.put(origin_inputs) batch_size = origin_inputs.shape[0] is_encoder_decoder = self.config.is_encoder_decoder logger.debug("The input shape is: %s", origin_inputs.shape) valid_length_each_example = [] for i in range(batch_size): # As the nonzero returns the index and we need length valid_length_each_example.append( np.max(np.argwhere(origin_inputs[i] != generation_config.pad_token_id)) + 1 ) valid_length_each_example = np.array(valid_length_each_example) logger.debug("Get the valid for each example is: %s", valid_length_each_example) # Prepare `max_length` depending on other stopping criteria. input_ids_length = np.max(valid_length_each_example) if generation_config.max_new_tokens is not None: generation_config.max_length = generation_config.max_new_tokens + input_ids_length if generation_config.max_length > self.config.seq_length: logger.warning("max_length %s can not exceeds model seq_length %s, set max_length = seq_length.", generation_config.max_length, self.config.seq_length) generation_config.max_length = self.config.seq_length logger.debug("max length is: %s", generation_config.max_length) if not is_encoder_decoder and input_ids_length >= generation_config.max_length: raise ValueError( f"the input_ids length {input_ids_length} exceeds the max length config {generation_config.max_length}." f"check your inputs and set max_length larger than your inputs length." ) input_ids = self._pad_inputs_using_max_length( origin_inputs=origin_inputs, pad_token_id=generation_config.pad_token_id ) logger.debug( "pad the origin inputs from %s into shape: %s", origin_inputs.shape, input_ids.shape, ) input_mask = np.zeros_like(input_ids) for i in range(valid_length_each_example.shape[0]): input_mask[i, :valid_length_each_example[i]] = 1 encoder_output = None encoder_mask = None if is_encoder_decoder: if generation_config.max_length > self.config.max_decode_length: generation_config.max_length = self.config.max_decode_length logger.debug("max decode length is: %s", generation_config.max_length) # When do encoder and decoder prediction, the encoder can be cached # to speed up the inference ( encoder_output, encoder_mask, input_ids, target_mask, ) = self._prepare_model_inputs_for_decoder(input_ids, input_mask) valid_length_each_example = [1 for _ in range(batch_size)] # A single loop generates one token, loop until reaching target # model_origin_max_length or generating eod token is_finished = [False] * batch_size # update model kwargs once, before go into generate loop. self.update_model_kwargs_before_generate(input_ids, model_kwargs) # setup is_first_iteration flag for incremental infer if generation_config.use_past: self.is_first_iteration = True need_gather_logits = True origin_len = np.sum(valid_length_each_example) prepare_time = time.time() - prepare_time logger.debug("forward prepare time: %s s", prepare_time) while np.sum(is_finished) != batch_size: forward_time = time.time() seq_length = input_ids.shape[1] current_index = [ valid_length_each_example[i] - 1 + i * seq_length for i in range(batch_size) ] logger.debug("validate length: %s", valid_length_each_example) if is_encoder_decoder: inputs = Tensor(input_ids, mstype.int32) # pylint: disable=E1102 res = self( input_ids=None, attention_mask=encoder_mask, encoder_outputs=encoder_output, decoder_input_ids=inputs, decoder_attention_mask=Tensor(target_mask, mstype.float32), ) else: model_kwargs["current_index"] = current_index # model prepare input dict model_inputs = self.prepare_inputs_for_generation( # pylint: disable=E1111 input_ids, **model_kwargs ) # incremental generate if generation_config.use_past: # when first iteration, gather last logits; others keep all logits. need_gather_logits = self.is_first_iteration # incremental generate res = self._incremental_infer( model_inputs=model_inputs, current_index=current_index, valid_length_each_example=valid_length_each_example, ) # auto-aggressive generate else: res = self(**model_inputs) # pylint: disable=E1102 forward_time = time.time() - forward_time sample_time = time.time() # post process logits; skip this phase if post process is done in graph if not self.config.is_sample_acceleration: # convert to numpy for post process logits = res[0] if isinstance(res, tuple) else res if isinstance(logits, Tensor): logits = logits.asnumpy() logits = np.reshape(logits, (-1, logits.shape[-1])) # need gather last seq logits using current_index # compare length to determine if need gather; if not, gather should be done in model construct if need_gather_logits and logits.shape[0] > len(current_index): logits = logits[current_index] # post process logits, without changing logits shape and order probs = logits_processor(input_ids, logits) probs = logits_warper(input_ids, probs) p_args = np.tile(np.arange(logits.shape[-1]), (batch_size, 1)) else: probs, p_args = res if isinstance(probs, Tensor): probs = probs.asnumpy() if isinstance(p_args, Tensor): p_args = p_args.asnumpy() sample_time = time.time() - sample_time update_time = time.time() # Random select a token as final output for this round target_list = [[] for _ in range(batch_size)] for i in range(batch_size): if is_finished[i]: continue if generation_config.do_sample: # multinomial sample p_norm = softmax(probs[i]) target_index = np.random.choice(len(probs[i]), p=p_norm) else: # greedy target_index = np.argmax(probs[i]) # get target token id target = p_args[i][target_index] input_ids[i, valid_length_each_example[i]] = target if streamer is not None: # assign target element target_list[i] = [target] if is_encoder_decoder: target_mask[i][valid_length_each_example[i]] = int(1) valid_length_each_example[i] += int(1) input_mask[i][valid_length_each_example[i] - 1] = 1 # Stop judgment if p_args[i][target_index] == generation_config.eos_token_id \ or valid_length_each_example[i] == generation_config.max_length: is_finished[i] = True continue if streamer is not None: if batch_size == 1: streamer.put(target_list[0]) else: streamer.put(target_list) update_time = time.time() - update_time logger.debug("forward time: %s s; sample time: %s s; update time: %s s; total count: %s s", forward_time, sample_time, update_time, forward_time + sample_time + update_time) # Return valid outputs out of padded outputs output_ids = [] for i in range(batch_size): output_ids.append( input_ids[i, : int(valid_length_each_example[i])].astype(np.int32) ) logger.debug("The output is: %s", output_ids) if streamer is not None: streamer.end() generate_len = np.sum(valid_length_each_example) - origin_len total_time = time.time() - total_time logger.info("total time: %s s; generated tokens: %s tokens; generate speed: %s tokens/s", total_time, generate_len, generate_len / total_time) return output_ids def generate(self, input_ids: Optional[Union[List[int], List[List[int]]]], generation_config: Optional[GenerationConfig] = None, logits_processor: Optional[LogitsProcessorList] = None, streamer: Optional[BaseStreamer] = None, seed: Optional[int] = None, **kwargs): """ Generate the words according to the given the input ids. Most generation-controlling parameters are set in `generation_config` which, if not passed, will be set to the model's default generation configuration. You can override any `generation_config` by passing the corresponding parameters to generate(), e.g. `.generate(inputs, top_k=3, do_sample=True)`. Args: input_ids(List(str), List(List(str))): The token id list or a batch of token id list. When input a batch of token id list, the length of each token id list should be same. generation_config (`GenerationConfig`, *optional*): The generation configuration to be used as base parametrization for the generation call. `**kwargs` passed to generate matching the attributes of `generation_config` will override them. If `generation_config` is not provided, the default config from the model configuration will be used. Please note that unspecified parameters will inherit [`GenerationConfig`]'s default values, whose documentation should be checked to parameterize generation. logits_processor (`LogitsProcessorList`, *optional*): Custom logits processors that complement the default logits processors built from arguments and generation config. If a logit processor is passed that is already created with the arguments or a generation config an error is thrown. This feature is intended for advanced users. streamer: The streamer that generator uses. seed: Random seed used in sample. kwargs: Specific parametrization of `generate_config` and/or additional model-specific kwargs that will be forwarded to the `forward` function of the model. Supported `generate_config` keywords can be checked in [`GenerationConfig`]'s documentation. Mainly used Keywords are shown below: max_length(int): The maximum length the generated tokens can have. Corresponds to the length of the input prompt + `max_new_tokens`. Its effect is overridden by `max_new_tokens`, if also set. max_new_tokens (int): The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. pad_token_id(int): The pad token id. If set None, it follows the setting in the configureation in the model. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default None. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> t5 = T5ForConditionalGeneration.from_pretrained("t5_small") >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" >>> words = tokenizer(words, max_length=21, padding='max_length')['input_ids'] >>> output = t5.generate(words, do_sample=True) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top p sampling >>> output = t5.generate(words, do_sample=True, top_p=0.4) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) eful ONU declară că nu există o soluţie militară în Siria >>> # Enable the top k sampling. >>> output = t5.generate(words, do_sample=True, top_k=10, top_p=1) >>> output = tokenizer.decode(output[0], skip_special_tokens=True) >>> print(output) Este comist de stat ale stateului membre nai uzusepa şi ONU Returns: A list of the generated token ids """ origin_phase = self.phase self.set_train(False) input_ids = np.array(input_ids).reshape(-1, np.shape(input_ids)[-1]) seed = 0 if seed is None else seed np.random.seed(seed) logits_processor = logits_processor if logits_processor is not None else LogitsProcessorList() # Handle `generation_config` and kwargs that might update it # priority: `generation_config` argument > `model.generation_config` (default config) if generation_config is None: # legacy: users may modify the model configuration to control generation # model attribute accordingly, if it was created from the model config generation_config = GenerationConfig.from_model_config(self.config) generation_config = copy.deepcopy(generation_config) model_kwargs = generation_config.update( **kwargs ) # All unused kwargs must be model kwargs if not generation_config.do_sample: generation_config.top_p = 1.0 generation_config.top_k = 0 logger.debug("Generation Config is: %s", generation_config) logits_processor = self._get_logits_processor( generation_config=generation_config, logits_processor=logits_processor, ) logits_warper = self._get_logits_warper(generation_config) output_ids = self._forward( origin_inputs=input_ids, generation_config=generation_config, logits_processor=logits_processor, logits_warper=logits_warper, streamer=streamer, **model_kwargs, ) # set to original phase self.set_train(origin_phase == "train") return output_ids pengcheng.mind-mindformer/mindformers/generation/utils.py000066400000000000000000000045561476650141400243160ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """utils for text generation.""" import numpy as np def log_softmax(x, axis=None): """numpy implemented log softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.py""" x_max = np.amax(x, axis=axis, keepdims=True) if x_max.ndim > 0: x_max[~np.isfinite(x_max)] = 0 elif not np.isfinite(x_max): x_max = 0 tmp = x - x_max exp_tmp = np.exp(tmp) # suppress warnings about log of zero with np.errstate(divide="ignore"): s = np.sum(exp_tmp, axis=axis, keepdims=True) out = np.log(s) out = tmp - out return out def softmax(x, axis=None): """numpy implemented softmax function. refers to https://github.com/scipy/scipy/blob/v1.11.1/scipy/special/_logsumexp.py""" x_max = np.amax(x, axis=axis, keepdims=True) exp_x_shifted = np.exp(x - x_max) return exp_x_shifted / np.sum(exp_x_shifted, axis=axis, keepdims=True) def topk(x, top_k, axis=-1, largest=True, sort=True): """numpy implemented topk sample.""" # safety check if x.shape[axis] < top_k: top_k = x.shape[axis] - 1 if largest: topk_index = np.argpartition(-x, top_k, axis=axis) else: topk_index = np.argpartition(x, top_k, axis=axis) topk_index = np.take(topk_index, np.arange(top_k), axis=axis) topk_data = np.take_along_axis(x, topk_index, axis=axis) if sort: sort_index = ( np.argsort(-topk_data, axis=axis) if largest else np.argsort(topk_data, axis=axis) ) topk_data = np.take_along_axis(topk_data, sort_index, axis=axis) topk_index = np.take_along_axis(topk_index, sort_index, axis=axis) return topk_data, topk_index pengcheng.mind-mindformer/mindformers/inference/000077500000000000000000000000001476650141400223755ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/inference/__init__.py000066400000000000000000000015111476650141400245040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """infer module""" from .infer_config import InferConfig from .infer_task import InferTask from .pipeline import get_mslite_pipeline __all__ = ['InferConfig', 'InferTask'] pengcheng.mind-mindformer/mindformers/inference/context.py000066400000000000000000000025301476650141400244330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """context of mslite apis.""" from mindspore_lite.context import Context from mindformers.inference.infer_config import InferConfig def build_ascend_context(config: InferConfig): """build context for ascend.""" context = Context() context.ascend.rank_id = config.rank_id context.ascend.device_id = config.device_id context.ascend.provider = "ge" context.target = [config.target] return context def build_context(config: InferConfig): """build context for mslite.""" if config.target == "Ascend": return build_ascend_context(config) raise KeyError(f"Supported data type keywords include: " f"[Ascend], but get {config.target}") pengcheng.mind-mindformer/mindformers/inference/infer_config.py000066400000000000000000000041251476650141400254010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite config.""" from mindspore_lite import ModelType from mindformers.models import BaseConfig def convert_lite_model_type(model_type: str): """ convert str to ModelType :param model_type: str :return: ModelType """ if model_type == "mindir": return ModelType.MINDIR if model_type == "mindir_lite": return ModelType.MINDIR_LITE raise KeyError(f"Supported data type keywords include: " f"[mindir, mindir_lite], but get {model_type}") class InferConfig(BaseConfig): """ Inference config class. """ def __init__(self, prefill_model_path: str = "", increment_model_path: str = "", model_type: str = "mindir", model_name: str = "common", infer_seq_length: int = 1024, target: str = "Ascend", device_id: int = 0, rank_id: int = 0, ge_config_path: str = "", **kwargs): super(InferConfig, self).__init__(**kwargs) self.prefill_model_path = prefill_model_path self.increment_model_path = increment_model_path self.model_type = convert_lite_model_type(model_type) self.model_name = model_name self.seq_length = infer_seq_length self.target = target self.device_id = device_id self.rank_id = rank_id self.config_path = ge_config_path pengcheng.mind-mindformer/mindformers/inference/infer_task.py000066400000000000000000000035111476650141400250740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Infer task class""" from .infer_config import InferConfig from .infers.base_infer import BaseInfer from .infers.text_generator_infer import TextGeneratorInfer class InferTask: """ Infer Task Factory. """ task_mapping = { "text_generation": TextGeneratorInfer, } @classmethod def get_infer_task(cls, task_type: str, task_config: InferConfig, **kwargs) -> BaseInfer: """ Get a infer task obj. by task_type. Args: task_type (str): task name task_config (InferConfig): config of infer task. Returns: BaseInfer. infer task obj. """ return InferTask.task_mapping[task_type](task_config, **kwargs) @classmethod def check_task_valid(cls, task_type: str) -> bool: """ check whether task type is valid or not. Args: task_type: str. task name Returns: bool """ return task_type in InferTask.support_list() @classmethod def support_list(cls): """ get infer task support list. Returns: List[str] """ return InferTask.task_mapping.keys() pengcheng.mind-mindformer/mindformers/inference/infers/000077500000000000000000000000001476650141400236635ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/inference/infers/__init__.py000066400000000000000000000013001476650141400257660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """inference tasks.""" __all__ = [] pengcheng.mind-mindformer/mindformers/inference/infers/base_infer.py000066400000000000000000000063731476650141400263430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """base infer class""" import abc from typing import Optional from mindspore_lite import Model, ModelGroup, ModelGroupFlag from mindformers.models import BaseTokenizer, BaseImageProcessor from mindformers.inference.infer_config import InferConfig from mindformers.inference.context import build_context class BaseInfer(metaclass=abc.ABCMeta): """ BaseInfer. """ def __init__(self, config: InferConfig = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None): if config is None: config = InferConfig() self.config = config self.model_type = config.model_type self.model_name = config.model_name self.seq_length = config.seq_length self.config_path = config.config_path self.context = build_context(config=self.config) self.tokenizer = tokenizer self.image_processor = image_processor self.full_model = None self.cache_model = None if config.prefill_model_path and config.increment_model_path: self.full_model, self.cache_model = self._load_increment_models( config.prefill_model_path, config.increment_model_path ) else: self.full_model = self._load_model(config.prefill_model_path) def _load_model(self, model_path): """ load single model from model path. """ model = Model() model.build_from_file(model_path, model_type=self.model_type, context=self.context, config_path=self.config_path) return model def _load_increment_models(self, full_model_path, cache_model_path): """ load kv cache models. """ full_model = Model() cache_model = Model() model_group = ModelGroup(ModelGroupFlag.SHARE_WEIGHT) model_group.add_model([full_model, cache_model]) full_model.build_from_file(full_model_path, self.model_type, self.context, self.config_path) cache_model.build_from_file(cache_model_path, self.model_type, self.context, self.config_path) return full_model, cache_model def __call__(self, inputs, **kwargs): """call infer process.""" return self.infer(inputs, **kwargs) @abc.abstractmethod def infer(self, inputs, **kwargs): """infer interface.""" @abc.abstractmethod def preprocess(self, input_data, **kwargs): """preprocess interface.""" @abc.abstractmethod def postprocess(self, predict_data, **kwargs): """postprocess interface.""" pengcheng.mind-mindformer/mindformers/inference/infers/text_generator_infer.py000066400000000000000000000536301476650141400304610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Generator Infer.""" import abc import time from typing import Union, List, Optional import numpy as np from mindspore_lite import Model from mindformers.tools.logger import logger from mindformers.models import BaseTokenizer from mindformers.generation import GenerationConfig, LogitsProcessorList from mindformers.generation.logits_process import RepetitionPenaltyLogitsProcessor, LogitNormalization, \ TemperatureLogitsWarper, TopKLogitsWarper, TopPLogitsWarper from mindformers.generation.streamers import BaseStreamer from mindformers.generation.utils import softmax from .base_infer import BaseInfer class BaseInputsOfInfer: """ BaseInputsOfInfer interface. """ @abc.abstractmethod def get_inputs(self, model: Model, **kwargs): pass def get_lite_tensor_list(self, inputs, model): input_list = [] for item in inputs: if item is None: continue input_list.append(item) lite_inputs = model.get_inputs() for input_np, tensor in zip(input_list, lite_inputs): tensor.set_data_from_numpy(input_np) return lite_inputs class CommonInputsOfInfer(BaseInputsOfInfer): """ common infer inputs of llm models. """ # pylint: disable=W0221 def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): if not is_first_iteration: inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) input_ids = np.array(inputs_tmp, dtype=np.int32) inputs = [input_ids, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs class GLMInputsOfInfer(BaseInputsOfInfer): """ glm infer inputs. """ def get_masks_np(self, input_ids, tokenizer: BaseTokenizer): batch_size, seq_length = input_ids.shape context_lengths = [list(seq).index(tokenizer.bos_token_id) for seq in input_ids] attention_mask = np.tril(np.ones((batch_size, seq_length, seq_length))) for i, context_length in enumerate(context_lengths): attention_mask[i, :, :context_length] = 1 attention_mask = np.expand_dims(attention_mask, axis=1) attention_mask = np.array(attention_mask < 0.5, np.bool_) return attention_mask def get_position_ids_np(self, input_ids, mask_positions, tokenizer: BaseTokenizer, use_gmasks=None, position_encoding_2d=True): """Get position ids from input_ids and mask_positions with numpy""" batch_size, seq_length = input_ids.shape if use_gmasks is None: use_gmasks = [False] * batch_size context_lengths = [list(seq).index(tokenizer.bos_token_id) for seq in input_ids] if position_encoding_2d: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): position_ids[i, context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, np.int32), np.arange(seq_length - context_length, dtype=np.int32) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0) position_ids = np.stack((position_ids, block_position_ids), axis=1) else: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids_np(self, input_ids, tokenizer, position_encoding_2d=True): """Get position ids from input_ids with numpy""" mask, gmask = tokenizer.mask_token_id, tokenizer.gmask_token_id seqs = list(input_ids) mask_positions, use_gmasks = [], [] for seq in seqs: mask_token = gmask if gmask in seq else mask use_gmask = mask_token == gmask mask_positions.append(list(seq).index(mask_token)) use_gmasks.append(use_gmask) position_ids = self.get_position_ids_np(input_ids, mask_positions, tokenizer, use_gmasks=None, position_encoding_2d=position_encoding_2d) return position_ids # pylint: disable=W0221 def get_inputs(self, model: Model, input_ids=None, current_index=None, valid_length=None, init_reset=None, tokenizer=None, position_encoding_2d=True, is_first_iteration=True, **kwargs): attention_mask_ = self.get_masks_np(input_ids, tokenizer).astype(np.int32) position_ids_ = self.create_position_ids_np(input_ids, tokenizer, position_encoding_2d).astype(np.int32) if not is_first_iteration: attention_mask = [] position_ids = [] inputs_tmp = [] for i in range(len(current_index)): current_index_tmp = int(current_index[i]) - i * input_ids.shape[1] # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp:current_index_tmp + 1]) position_ids.append(position_ids_[i][:, current_index_tmp:current_index_tmp + 1]) attention_mask.append(attention_mask_[i][:, current_index_tmp:current_index_tmp + 1, :]) input_ids = np.array(inputs_tmp, np.int32) position_ids = np.array(position_ids, np.int32) attention_mask = np.array(attention_mask, np.int32) else: attention_mask = attention_mask_ position_ids = position_ids_ inputs = [input_ids, position_ids, attention_mask, current_index, init_reset, valid_length] lite_inputs = self.get_lite_tensor_list(inputs, model) return lite_inputs class InputOfInfer: """ Input of llm model. """ MAPPING = { "bloom": CommonInputsOfInfer, "llama": CommonInputsOfInfer, "glm2": CommonInputsOfInfer, "glm": GLMInputsOfInfer, "common": CommonInputsOfInfer } @classmethod def get_inputs(cls, model_name: str, model, **kwargs): """ Get input tensor list of mslite. Args: model_name: str, model name. model: mindspore_lite.Model. mode obj of mslite. Returns: tensor list of mslite. """ name = "" if model_name not in InputOfInfer.MAPPING: for k in InputOfInfer.MAPPING: if model_name.startswith(k): name = k break if not name: logger.warning("Model name not in support maps.Common input format will be used to do inference.") name = "common" else: name = model_name return InputOfInfer.MAPPING[name]().get_inputs(model, **kwargs) class TextGeneratorInfer(BaseInfer): """ Text generator infer implement class. """ # pylint: disable=W0221 def infer(self, inputs: Union[str, List[str]], do_sample: bool = False, top_k: int = 1, top_p: float = 1.0, temperature: float = 1.0, repetition_penalty: float = 1.0, eos_token_id: int = 2, pad_token_id: int = 0, max_length: int = 256, is_sample_acceleration: bool = False, add_special_tokens: bool = False, streamer: Optional[BaseStreamer] = None, **kwargs): """ text generator inference api Args: inputs(List(str), List(List(str))): The token id list or a list of token id list. do_sample(bool): Whether to do sampling on the candidate ids. If set True it will be enabled, and set it to be False to disable the sampling, equivalent to topk 1. If set None, it follows the setting in the configureation in the model. Default None. top_k(int): Determine the topK numbers token id as candidate. This should be a positive number. If set None, it follows the setting in the configureation in the model. Default 1. top_p(float): The accumulation probability of the candidate token ids below the top_p will be select as the condaite ids. The valid value of top_p is between (0, 1]. If the value is larger than 1, top_K algorithm will be enabled. If set None, it follows the setting in the configureation in the model. Default 1. temperature (`float`, *optional*, defaults to 1.0): The value used to modulate the next token probabilities. eos_token_id(int): The end of sentence token id. If set None, it follows the setting in the configureation in the model. Default 2. pad_token_id(int): The padding of sentence token id. If set None, it follows the setting in the configureation in the model. Default 0. repetition_penalty(float): The penalty factor of the frequency that generated words. The If set 1, the repetition_penalty will not be enabled. If set None, it follows the setting in the configureation in the model. Default 1. max_length: The maximum length of the generated words. If set None, it follows the setting in the configureation in the model. Default 256. is_sample_acceleration: The postprocess are processing in model. Default False. add_special_tokens: Add special tokens for preprocess. streamer: The streamer that generator uses. Returns: outputs of model infer """ input_ids = self.preprocess(inputs, add_special_tokens) output_ids = self.generate(input_ids, do_sample, top_k, top_p, temperature, repetition_penalty, eos_token_id, pad_token_id, max_length, is_sample_acceleration, streamer, **kwargs) outputs = self.postprocess(output_ids) return outputs # pylint: disable=W0613 def preprocess(self, input_data, add_special_tokens=False, **kwargs): """preprocess.""" tokens = self.tokenizer(input_data, add_special_tokens=add_special_tokens) input_ids = tokens["input_ids"] input_list = [] if isinstance(input_data, str): input_list.append(input_ids) else: input_list = input_ids return input_list # pylint: disable=W0613 def postprocess(self, predict_data, **kwargs): """postprocess.""" outputs = self.tokenizer.decode(predict_data, skip_special_tokens=True) return outputs def _get_logits_processor(self, generation_config: GenerationConfig, logits_processor: Optional[LogitsProcessorList]): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsProcessor`] instances used to modify the scores of the language model head. """ # instantiate processors list processors = LogitsProcessorList() if generation_config.repetition_penalty is not None and generation_config.repetition_penalty != 1.0: processors.append(RepetitionPenaltyLogitsProcessor(penalty=generation_config.repetition_penalty)) processors = self._merge_processor_list(processors, logits_processor) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: processors.append(LogitNormalization()) return processors def _merge_processor_list(self, default_list: LogitsProcessorList, custom_list: LogitsProcessorList): """merge custom processor list with default list.""" if not custom_list: return default_list for default in default_list: for custom in custom_list: if type(custom) is type(default): object_type = "logits processor" raise ValueError( f"A custom {object_type} of type {type(custom)} with values {custom} has been passed to" f" `.generate()`, but it has already been created with the values {default}." f" {default} has been created by passing the corresponding arguments to generate or" f" by the model's config default values. If you just want to change the default values" f" of {object_type} consider passing them as arguments to `.generate()`" f" instead of using a custom {object_type}." ) default_list.extend(custom_list) return default_list def _get_logits_warper(self, generation_config: GenerationConfig): """ This class returns a [`LogitsProcessorList`] list object that contains all relevant [`LogitsWarper`] instances used for multinomial sampling. """ # instantiate warpers list warpers = LogitsProcessorList() # all samplers can be found in `generation_utils_samplers.py` if generation_config.temperature is not None and generation_config.temperature != 1.0: warpers.append(TemperatureLogitsWarper(generation_config.temperature)) min_tokens_to_keep = 1 if generation_config.top_k is not None and generation_config.top_k != 0: warpers.append(TopKLogitsWarper(top_k=generation_config.top_k, min_tokens_to_keep=min_tokens_to_keep)) if generation_config.top_p is not None and generation_config.top_p < 1.0: warpers.append(TopPLogitsWarper(top_p=generation_config.top_p, min_tokens_to_keep=min_tokens_to_keep)) # `LogitNormalization` should always be the last logit processor, when present if generation_config.renormalize_logits is True: warpers.append(LogitNormalization()) return warpers def generate(self, input_ids, do_sample, top_k, top_p, temperature, repetition_penalty, eos_token_id, pad_token_id, max_length, is_sample_acceleration, streamer, **kwargs): """token generator.""" sampler_dict = {"do_sample": do_sample, "top_k": top_k, "top_p": top_p, "temperature": temperature, "repetition_penalty": repetition_penalty, "max_length": max_length, **kwargs} generation_config = GenerationConfig(**sampler_dict) if not generation_config.do_sample: generation_config.top_p = 1.0 generation_config.top_k = 0 logits_processor = LogitsProcessorList() logits_processor = self._get_logits_processor( generation_config=generation_config, logits_processor=logits_processor, ) logits_warper = self._get_logits_warper(generation_config) if streamer: streamer.put(np.array(input_ids[0])) batch_size = len(input_ids) valid_length = [] for i in range(batch_size): # As the nonzero returns the index and we need length valid_length.append(np.max(np.argwhere(np.array(input_ids[i]) != pad_token_id)) + 1) valid_length = np.array(valid_length, np.int32) target_length = self.seq_length if max_length > self.seq_length else max_length # pad original input ids to seq_length pad_length = self.seq_length - valid_length pad_input_ids = np.array([ np.pad(input_ids[i], (0, pad_length[i]), 'constant', constant_values=(pad_token_id)) for i in range(len(input_ids)) ], np.int32) input_ids = pad_input_ids # setup is_first_iteration flag for incremental infer is_first_iteration = True is_finished = [False] * batch_size use_past = self.full_model and self.cache_model while np.sum(is_finished) != batch_size: start_time = time.time() seq_length = input_ids.shape[1] current_index = [valid_length[i] - 1 + i * seq_length for i in range(batch_size)] current_index = np.array(current_index, np.int32) logger.debug("validate length: %s", valid_length) if use_past: outputs = self._inc_infer(input_ids, current_index, valid_length, is_first_iteration, **kwargs) else: outputs = self._full_infer(input_ids, current_index, is_sample_acceleration, **kwargs) if not is_sample_acceleration: logits = outputs[0].get_data_to_numpy() vocab_size = logits.shape[-1] if len(logits.shape) < 3: logits = logits.reshape(batch_size, -1, vocab_size) # gather logits if is_first_iteration: logits = np.array([logits[i][current_index[i] - i * seq_length, :] for i in range(batch_size)]) else: logits = np.array([logits[i][0, :] for i in range(batch_size)]) logits = logits.reshape(-1, vocab_size) log_probs = logits_processor(input_ids, logits) p = logits_warper(input_ids, log_probs) p_args = np.tile(np.arange(logits.shape[-1]), (batch_size, 1)) else: p = outputs[0].get_data_to_numpy().astype(np.int32) p_args = outputs[1].get_data_to_numpy() # Random select a token as final output for this round for i in range(batch_size): if is_finished[i]: continue # target_index = np.random.choice(len(p[i]), p=p[i]) if generation_config.do_sample: # multinomial sample p_norm = softmax(p[i]) target_index = np.random.choice(len(p[i]), p=p_norm) else: # greedy target_index = np.argmax(p[i]) # Stop judgment when length exceeds target_length. if valid_length[i] >= target_length: is_finished[i] = True continue # add next token to input_ids. target = p_args[i][target_index] input_ids[i, valid_length[i]] = target if streamer: streamer.put(np.asarray([target])) valid_length[i] += int(1) # Stop judgment when output is EOS token, with the output # is appended to input_ids and streamer. if target == eos_token_id: is_finished[i] = True continue is_first_iteration = not use_past print(f"one token takes {time.time() - start_time} s") # Return valid outputs out of padded outputs output_ids = [] for i in range(batch_size): output_ids.append(input_ids[i, : int(valid_length[i])].astype(np.int32)) logger.debug("The output is: %s", output_ids) if streamer: streamer.end() return output_ids def _inc_infer(self, input_ids, current_index, valid_length, is_first_iteration, **kwargs): """kvcache infer""" if is_first_iteration: init_reset = np.array([False]) lite_inputs = self.get_predict_inputs(self.full_model, input_ids, current_index, valid_length, init_reset, is_first_iteration, **kwargs) outputs = self.full_model.predict(lite_inputs) else: init_reset = np.array([True]) lite_inputs = self.get_predict_inputs(self.cache_model, input_ids, current_index, valid_length, init_reset, is_first_iteration, **kwargs) outputs = self.cache_model.predict(lite_inputs) return outputs def _full_infer(self, input_ids, current_index, is_npu_acceleration, **kwargs): """infer""" # get inputs if is_npu_acceleration: lite_inputs = self.get_predict_inputs(self.full_model, input_ids, current_index, **kwargs) else: lite_inputs = self.get_predict_inputs(self.full_model, input_ids, **kwargs) # do infer outputs = self.full_model.predict(lite_inputs) return outputs def get_predict_inputs(self, mode: Model, input_ids, current_index=None, valid_length=None, init_reset=None, is_first_iteration=True, **kwargs): """Get inputs of llm model for mslite.""" return InputOfInfer.get_inputs(self.model_name, mode, input_ids=input_ids, current_index=current_index, valid_length=valid_length, init_reset=init_reset, tokenizer=self.tokenizer, is_first_iteration=is_first_iteration, **kwargs) pengcheng.mind-mindformer/mindformers/inference/pipeline.py000066400000000000000000000236151476650141400245630ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite infer pipeline.""" import glob import os.path from typing import Optional, Union, Tuple from mindformers.models import BaseTokenizer, BaseImageProcessor, BaseAudioProcessor, \ build_tokenizer, build_processor from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerConfig from .infer_task import InferTask from .infers.base_infer import BaseInfer from .infer_config import InferConfig SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() def is_model_name(model): if not isinstance(model, str): return False if os.path.exists(model): return False return True def is_tuple(model): return isinstance(model, tuple) def is_path(model): return isinstance(model, str) and os.path.isfile(model) def is_dir(model): return isinstance(model, str) and os.path.isdir(model) # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_dir(task, model, tokenizer, image_processor, audio_processor, ge_config_path=None, **kwargs) -> BaseInfer: """ Support user project contain certain files. (*.yaml/config.ini/tokenizer.model/graph1.mindir graph2.mindir) Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (str): The model dir. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if not os.path.exists(model): raise FileNotFoundError(f"{model} is not found.") proj_dir = os.path.realpath(model) # check model run config run_config = glob.glob(proj_dir + "/*.yaml") if len(run_config) > 1: raise ValueError(f"{model} contains multi config files: {run_config}.") config = MindFormerConfig(run_config[0]) if config.infer is None: raise ValueError(f"There is no infer config in {run_config[0]}.") if tokenizer is None: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and hasattr(config.processor, 'image_processor'): image_processor = build_processor(config.processor.image_processor) # check ge config.ini if ge_config_path is None: ge_config_path = os.path.join(proj_dir, "config.ini") if not os.path.exists(ge_config_path): raise ValueError(f"There is ge config.ini file in {model}.") config.infer.config_path = ge_config_path # graph.mindir graphs = glob.glob(proj_dir + "/*.mindir") if len(graphs) > 2: raise ValueError(f"MindIR files in {model} must equal to 1 or 2. But get {len(graphs)} files.") if len(graphs) == 2: if "prefill" in graphs[0]: config.infer.prefill_model_path = graphs[0] config.infer.increment_model_path = graphs[1] else: config.infer.prefill_model_path = graphs[1] config.infer.increment_model_path = graphs[0] else: config.infer.prefill_model_path = graphs[0] # set device id and rank id if "device_id" in kwargs: config.infer.device_id = kwargs["device_id"] if "rank_id" in kwargs: config.infer.rank_id = kwargs["rank_id"] # add model name config.infer.model_name = config.trainer.model_name infer_config = InferConfig( **config.infer ) task_pipeline = InferTask.get_infer_task(task, infer_config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_model_name(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) -> BaseInfer: """ Support get infer pipeline from model name. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (str): The model name. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if model not in SUPPORT_MODEL_NAMES: raise KeyError( f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}.") pipeline_config = MindFormerConfig(SUPPORT_PIPELINES.get(task).get(model)) if pipeline_config.infer is None: raise ValueError(f"There is no infer config in {model} config.") if image_processor is None and hasattr(pipeline_config.processor, 'image_processor'): image_processor = build_processor(pipeline_config.processor.image_processor) if audio_processor is None and hasattr(pipeline_config.processor, 'audio_processor'): audio_processor = build_processor(pipeline_config.processor.audio_processor) if tokenizer is None: tokenizer = build_tokenizer(pipeline_config.processor.tokenizer) pipeline_config.infer.model_name = model # set device id and rank id if "device_id" in kwargs: pipeline_config.infer.device_id = kwargs["device_id"] if "rank_id" in kwargs: pipeline_config.infer.rank_id = kwargs["rank_id"] infer_config = InferConfig( **pipeline_config.infer ) task_pipeline = InferTask.get_infer_task(task, infer_config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline # align with pipeline() # pylint: disable=W0613 def get_infer_pipeline_from_model_files(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) -> BaseInfer: """ Support get infer pipeline from model files. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Union[str, Tuple[str, str]]): The model paths of tuple(model_path, model_path). tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if isinstance(model, str): full_model_path = model cache_model_path = "" else: full_model_path, cache_model_path = model full_model_path = os.path.realpath(full_model_path) if not os.path.exists(full_model_path): raise FileNotFoundError(f"{full_model_path} is not found.") if cache_model_path: cache_model_path = os.path.realpath(cache_model_path) if not os.path.exists(cache_model_path): raise FileNotFoundError(f"{cache_model_path} is not found.") config = InferConfig( prefill_model_path=full_model_path, increment_model_path=cache_model_path, ge_config_path=ge_config_path, **kwargs ) task_pipeline = InferTask.get_infer_task(task, config, tokenizer=tokenizer, image_processor=image_processor) return task_pipeline STRATEGY_MAP = { is_model_name: get_infer_pipeline_from_model_name, is_tuple: get_infer_pipeline_from_model_files, is_path: get_infer_pipeline_from_model_files, is_dir: get_infer_pipeline_from_dir, } def get_mslite_pipeline(task: str = None, model: Optional[Union[str, BaseInfer, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, ge_config_path: str = None, **kwargs) -> BaseInfer: """ Get mslite infer pipeline. Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Union[str, Tuple[str, str]]): The model name. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. ge_config_path: ge config file path Returns: BaseInfer. """ if not InferTask.check_task_valid(task): raise KeyError(f"{task} is not supported by pipeline. please select" f" a task from {InferTask.support_list()}.") # init infer_pipeline = None for k, v in STRATEGY_MAP.items(): if k(model): infer_pipeline = v(task, model, tokenizer, image_processor, audio_processor, ge_config_path, **kwargs) if not infer_pipeline: raise ValueError(f"model must be a dir or a mindir file path or a tuple of file path, but get {model}.") return infer_pipeline pengcheng.mind-mindformer/mindformers/mindformer_book.py000066400000000000000000001724601476650141400241770ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository. """ import os import copy from collections import OrderedDict from mindformers.tools import logger def print_dict(input_dict): """ Print dict function for show support method of MindFormerBook or other BaseClasses. Args: input_dict (dict): the dict to be printed. """ if isinstance(input_dict, dict): for key, val in input_dict.items(): if isinstance(val, dict): logger.info("%s:", key) print_dict(val) else: logger.info(" %s : %s", key, val) else: raise TypeError(f"{type(input_dict)} is unspoorted by print_dict") def print_path_or_list(input_path_or_list): """ Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. """ if isinstance(input_path_or_list, (str, list)): logger.info(" %s", input_path_or_list) logger.info("-------------------------------------") else: raise TypeError(f"{type(input_path_or_list)} is unsupported by print_path_or_list") class MindFormerBook: """ MindFormerBook class, which contains the lists of models, pipelines, tasks, and default settings in MindFormer repository When adding a new Model or assemble in this project, the following constants list and dict need adding. Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> >>> # 1) Fill the following constant list and dict in this class >>> # 2) Overwrite the support_list when define a new Model or Pipeline. >>> @MindFormerRegister.register(MindFormerModuleType.MODELS) ... class ViTForImageClassification(BaseModel): ... _support_list = MindFormerBook.get_model_support_list()['vit'] >>> # 3) Then you can use auto class and from pretrain to init an instance. >>> vit_model = AutoModel.from_pretrained('vit_base_p16') """ _PROJECT_PATH = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) _DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER = os.getenv("CHECKPOINT_DOWNLOAD_FOLDER", os.path.join('.', 'checkpoint_download')) _DEFAULT_CHECKPOINT_SAVE_FOLDER = os.getenv("CHECKPOINT_SAVE_FOLDER", os.path.join('.', 'checkpoint_save')) _TRAINER_SUPPORT_TASKS_LIST = OrderedDict([ ("general", OrderedDict([ ("common", os.path.join( _PROJECT_PATH, "configs/general/run_general_task.yaml"))]) ), ("masked_image_modeling", OrderedDict([ ("mae_vit_base_p16", os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml"))]) ), ("image_classification", OrderedDict([ ("vit_base_p16", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ("mindspore/vit_base_p16", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("mindspore/swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml"))]) ), ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")), ("bert_tiny_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml"))]) ), ("contrastive_language_image_pretrain", OrderedDict([ ("clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml")), ("blip2_stage1_vit_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_qformer_pretrain.yaml")), ("blip2_stage2_vit_g_baichuan_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b.yaml")), ("blip2_stage2_vit_g_llama_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b.yaml")), ("mindspore/clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml")), ("clip_vit_b_16", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_pretrain_flickr8k.yaml")), ("clip_vit_l_14", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_pretrain_flickr8k.yaml")), ("clip_vit_l_14@336", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_pretrain_flickr8k.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml"))]) ), ("image_to_text_retrieval", OrderedDict([ ("blip2_stage1_vit_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml"))]) ), ("zero_shot_image_classification", OrderedDict([ ("clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ("mindspore/clip_vit_b_32", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_b_16", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_l_14", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml")), ("clip_vit_l_14@336", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml")), ('blip2_stage1_classification', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml"))]) ), ("image_to_text_generation", OrderedDict([ ("itt_blip2_stage2_vit_g_baichuan_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml")), ("itt_blip2_stage2_vit_g_llama_7b", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/clip/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml"))]) ), ("translation", OrderedDict([ ("t5_small", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")), ("t5_tiny", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_tiny_on_wmt16.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml"))]) ), ("text_classification", OrderedDict([ ("txtcls_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")), ("txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("mindspore/txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("gpt2_txtcls", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_txtcls.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml"))]) ), ("token_classification", OrderedDict([ ("tokcls_bert_base_chinese", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")), ("tokcls_bert_base_chinese_cluener", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml"))]) ), ("question_answering", OrderedDict([ ("qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("qa_bert_base_uncased_squad", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("mindspore/qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml"))]) ), ("text_generation", OrderedDict([ ("gpt2", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")), ("gpt2_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_lora.yaml")), ("gpt2_13b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_13b.yaml")), ("gpt2_52b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_52b.yaml")), ("gpt2_xl", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl.yaml")), ("gpt2_xl_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl_lora.yaml")), ("llama_7b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b.yaml")), ("llama_13b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_13b.yaml")), ("llama_65b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_65b.yaml")), ("llama2_7b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_7b.yaml")), ("llama2_13b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_13b.yaml")), ("llama2_70b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_70b.yaml")), ("llama_7b_lora", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b_lora.yaml")), ("pangualpha_2_6b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_2_6b.yaml")), ("pangualpha_13b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_13b.yaml")), ("glm_6b", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_finetune.yaml")), ("glm_6b_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_infer.yaml")), ("glm_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora.yaml")), ("glm_6b_lora_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora_infer.yaml")), ("glm2_6b", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b.yaml")), ("glm2_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b_lora.yaml")), ("bloom_560m", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_560m.yaml")), ("bloom_7.1b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_7.1b.yaml")), ("bloom_65b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_65b.yaml")), ("bloom_176b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_176b.yaml")), ("baichuan_7b", os.path.join( _PROJECT_PATH, "research/baichuan/run_baichuan_7b.yaml")), ("baichuan2_7b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_7b.yaml")), ("baichuan2_13b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_13b.yaml")), ("ziya_13b", os.path.join( _PROJECT_PATH, "research/ziya/run_ziya_13b.yaml")), ("internlm_7b", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b.yaml")), ("internlm_7b_lora", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b_lora.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml"))]) ), ("segment_anything", OrderedDict([ ("sam_vit_b", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-b.yaml")), ("sam_vit_l", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-l.yaml")), ("sam_vit_h", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml"))]) ) ]) _PIPELINE_SUPPORT_TASK_LIST = OrderedDict([ ('zero_shot_image_classification', OrderedDict([ ('clip_vit_b_16', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_16_zero_shot_image_classification_cifar100.yaml")), ('mindspore/clip_vit_b_32', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_b_32', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_l_14', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14_zero_shot_image_classification_cifar100.yaml")), ('clip_vit_l_14@336', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_l_14@336_zero_shot_image_classification_cifar100.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml")), ('blip2_stage1_classification', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_zero_shot_image_classification_cifar100.yaml")) ])), ('image_to_text_generation', OrderedDict([ ('itt_blip2_stage2_vit_g_baichuan_7b', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_baichuan_7b_image_to_text_generation.yaml")), ('itt_blip2_stage2_vit_g_llama_7b', os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage2_vit_g_llama_7b_image_to_text_generation.yaml")) ])), ('masked_image_modeling', OrderedDict([ ('mae_vit_base_p16', os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/mae/run_mae_vit_base_p16_224_800ep.yaml")) ])), ('image_classification', OrderedDict([ ('vit_base_p16', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ('mindspore/vit_base_p16', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")), ("mindspore/swin_base_p4w7", os.path.join( _PROJECT_PATH, "configs/swin/run_swin_base_p4w7_224_100ep.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/vit/run_vit_base_p16_224_100ep.yaml")) ])), ('translation', OrderedDict([ ('t5_small', os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")), ('common', os.path.join( _PROJECT_PATH, "configs/t5/run_t5_small_on_wmt16.yaml")) ])), ("fill_mask", OrderedDict([ ("bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_base_uncased.yaml")), ("bert_tiny_uncased", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/bert/run_bert_tiny_uncased.yaml")) ])), ("text_classification", OrderedDict([ ("txtcls_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")), ("txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("mindspore/txtcls_bert_base_uncased_mnli", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased_mnli.yaml")), ("gpt2_txtcls", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_txtcls.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/txtcls/run_txtcls_bert_base_uncased.yaml")) ])), ("token_classification", OrderedDict([ ("tokcls_bert_base_chinese", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")), ("tokcls_bert_base_chinese_cluener", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese_cluener.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/tokcls/run_tokcls_bert_base_chinese.yaml")) ])), ("question_answering", OrderedDict([ ("qa_bert_base_uncased", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/qa/run_qa_bert_base_uncased.yaml")) ])), ("text_generation", OrderedDict([ ("gpt2", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")), ("gpt2_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_lora.yaml")), ("gpt2_xl", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl.yaml")), ("gpt2_xl_lora", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_xl_lora.yaml")), ("gpt2_13b", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2_13b.yaml")), ("llama_7b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b.yaml")), ("llama_13b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_13b.yaml")), ("llama_65b", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_65b.yaml")), ("llama2_7b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_7b.yaml")), ("llama2_13b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_13b.yaml")), ("llama2_70b", os.path.join( _PROJECT_PATH, "configs/llama2/run_llama2_70b.yaml")), ("llama_7b_lora", os.path.join( _PROJECT_PATH, "configs/llama/run_llama_7b_lora.yaml")), ("pangualpha_2_6b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_2_6b.yaml")), ("pangualpha_13b", os.path.join( _PROJECT_PATH, "configs/pangualpha/run_pangualpha_13b.yaml")), ("glm_6b", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_finetune.yaml")), ("glm_6b_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_infer.yaml")), ("glm_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora.yaml")), ("glm_6b_lora_chat", os.path.join( _PROJECT_PATH, "configs/glm/run_glm_6b_lora_infer.yaml")), ("glm2_6b", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b.yaml")), ("glm2_6b_lora", os.path.join( _PROJECT_PATH, "configs/glm2/run_glm2_6b_lora.yaml")), ("bloom_560m", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_560m.yaml")), ("bloom_7.1b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_7.1b.yaml")), ("bloom_65b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_65b.yaml")), ("bloom_176b", os.path.join( _PROJECT_PATH, "configs/bloom/run_bloom_176b.yaml")), ("baichuan_7b", os.path.join( _PROJECT_PATH, "research/baichuan/run_baichuan_7b.yaml")), ("baichuan2_7b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_7b.yaml")), ("baichuan2_13b", os.path.join( _PROJECT_PATH, "research/baichuan2/run_baichuan2_13b.yaml")), ("ziya_13b", os.path.join( _PROJECT_PATH, "research/ziya/run_ziya_13b.yaml")), ("internlm_7b", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b.yaml")), ("internlm_7b_lora", os.path.join( _PROJECT_PATH, "research/internlm/run_internlm_7b_lora.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/gpt2/run_gpt2.yaml")) ])), ("image_to_text_retrieval", OrderedDict([ ("blip2_stage1_vig_g", os.path.join( _PROJECT_PATH, "configs/blip2/run_blip2_stage1_vit_g_retrieval_flickr30k.yaml")) ])), ("segment_anything", OrderedDict([ ("sam_vit_b", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-b.yaml")), ("sam_vit_l", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-l.yaml")), ("sam_vit_h", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")), ("common", os.path.join( _PROJECT_PATH, "configs/sam/run_sam_vit-h.yaml")) ])) ]) _CONFIG_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vig_g', 'blip2_stage1_classification', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b', 'itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b', ]), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('filip', [ 'filip_vit_l_14', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora', 'baichuan_7b', 'baichuan2_7b', 'baichuan2_13b', 'ziya_13b', 'internlm_7b', 'internlm_7b_lora', ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('glm', [ 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat' ]), ('glm2', [ 'glm2_6b', 'glm2_6b_lora', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _MODEL_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', OrderedDict([ ('1-stg', ['blip2_stage1_vit_g', 'blip2_stage1_classification']), ('2-stg', ['blip2_stage1_pretrained', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b']) ])), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'mae_vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('mt5', [ 't5_pegasus_base', ]), ('filip', [ 'filip_vit_l_14', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora' ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('glm', [ 'glm_6b', 'glm_6b_chat', 'glm_6b_lora', 'glm_6b_lora_chat' ]), ('glm2', [ 'glm2_6b', 'glm2_6b_lora' ]), ('internlm', [ 'internlm_7b', 'internlm_7b_lora', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _PROCESSOR_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vit_g', 'blip2_stage1_classification', 'blip2_stage2_vit_g_baichuan_7b', 'blip2_stage2_vit_g_llama_7b' ]), ('itt', OrderedDict([ ('blip2', ['itt_blip2_stage2_vit_g_baichuan_7b', 'itt_blip2_stage2_vit_g_llama_7b']) ])), ('mae', [ 'mae_vit_base_p16', ]), ('vit', [ 'vit_base_p16', 'mindspore/vit_base_p16', 'vit_g_p16' ]), ('swin', [ 'swin_base_p4w7', 'mindspore/swin_base_p4w7', ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('t5', [ 't5_small', ]), ('gpt2', [ 'gpt2', 'gpt2_lora', 'gpt2_xl', 'gpt2_xl_lora', 'gpt2_13b' ]), ('llama', [ 'llama', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('glm', [ 'glm', ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('sam', [ 'sam_vit_b', 'sam_vit_l', 'sam_vit_h' ]) ]) _TOKENIZER_SUPPORT_LIST = OrderedDict([ ('clip', [ 'clip_vit_b_32', 'clip_vit_b_16', 'clip_vit_l_14', 'clip_vit_l_14@336', 'mindspore/clip_vit_b_32' ]), ('blip2', [ 'blip2_stage1_vit_g', 'blip2_stage1_classification' ]), ('bert', [ 'bert_base_uncased', 'bert_tiny_uncased', ]), ('tokcls', OrderedDict([ ('bert', ['tokcls_bert_base_chinese', 'tokcls_bert_base_chinese_cluener', 'mindspore/tokcls_bert_base_chinese_cluener']) ])), ('txtcls', OrderedDict([ ('bert', ['txtcls_bert_base_uncased', 'txtcls_bert_base_uncased_mnli', 'mindspore/txtcls_bert_base_uncased_mnli']), ('gpt2', ['gpt2_txtcls']) ])), ('qa', OrderedDict([ ('bert', ['qa_bert_base_uncased', 'qa_bert_base_uncased_squad']) ])), ('t5', [ 't5_small', ]), ('gpt2', [ 'gpt2', ]), ('llama', [ 'llama_7b', 'llama_13b', 'llama_65b', 'llama2_7b', 'llama2_13b', 'llama2_70b', 'llama_7b_lora', 'baichuan_7b', 'ziya_13b' ]), ('llama2', [ 'llama2_7b', 'llama2_13b', 'llama2_70b', ]), ('pangualpha', [ 'pangualpha_2_6b', 'pangualpha_13b' ]), ('glm', [ 'glm_6b', ]), ('glm2', [ 'glm2_6b' ]), ('bloom', [ 'bloom_560m', 'bloom_7.1b', 'bloom_65b', 'bloom_176b', ]), ('internlm', [ 'internlm_7b', 'internlm_7b_lora', ]), ('baichuan2', [ 'baichuan2_7b', 'baichuan2_13b', ]), ]) _MODEL_CONFIG_TO_NAME = OrderedDict([ ]) _MODEL_CKPT_URL_LIST = OrderedDict([ ('clip_vit_b_32', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/clip/clip_vit_b_32.ckpt' ]), ('mindspore/clip_vit_b_32', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/clip_vit_b_32.ckpt']), ('clip_vit_b_16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_b_16.ckpt' ]), ('clip_vit_l_14', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14.ckpt' ]), ('clip_vit_l_14@336', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14%40336.ckpt' ]), ('mae_vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/mae/mae_vit_base_p16.ckpt' ]), ('vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/vit/vit_base_p16.ckpt' ]), ('vit_g_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/vit_g_p16.ckpt' ]), ('blip2_stage1_classification', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage1_pretrained', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage2_vit_g_baichuan_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('itt_blip2_stage2_vit_g_baichuan_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('blip2_stage2_vit_g_llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('itt_blip2_stage2_vit_g_llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_pretrained.ckpt' ]), ('mindspore/vit_base_p16', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/vit_base_p16/file/vit_base_p16.ckpt']), ('swin_base_p4w7', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/swin/swin_base_p4w7.ckpt' ]), ('mindspore/swin_base_p4w7', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/swin_base_p4w7/file/swin_base_p4w7.ckpt']), ('t5_small', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/t5/mindspore_model.ckpt' ]), ('bert_tiny_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.ckpt' ]), ('bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.ckpt' ]), ('tokcls_bert_base_chinese', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/' 'token_classification/tokcls_bert_base_chinese.ckpt' ]), ('tokcls_bert_base_chinese_cluener', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/' 'tokcls_bert_base_chinese_cluener.ckpt' ]), ('mindspore/tokcls_bert_base_chinese_cluener', ['https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/' 'file/tokcls_bert_base_chinese_cluener.ckpt']), ('txtcls_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.ckpt' ]), ('txtcls_bert_base_uncased_mnli', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/' 'txtcls_bert_base_uncased_mnli.ckpt' ]), ('mindspore/txtcls_bert_base_uncased_mnli', ['https://xihe.mindspore.cn/' 'api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/txtcls_bert_base_uncased_mnli.ckpt' ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.ckpt' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/' 'qa_bert_base_uncased_squad.ckpt' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2.ckpt' ]), ('gpt2_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_lora.ckpt' ]), ('gpt2_xl', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_xl.ckpt' ]), ('gpt2_xl_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_xl_lora.ckpt' ]), ('gpt2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2_13b.ckpt' ]), ('pangualpha_2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/pangualpha_2_6b.ckpt' ]), ('pangualpha_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/pangualpha_13b.ckpt' ]), ('glm_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b.ckpt' ]), ('glm_6b_chat', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b.ckpt' ]), ('glm_6b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b_lora.ckpt' ]), ('glm_6b_lora_chat', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/glm_6b_lora.ckpt' ]), ('glm2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/glm2_6b.ckpt' ]), ('glm2_6b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/glm2_6b_lora.ckpt' ]), ('llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_7b.ckpt' ]), ('llama_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_13b.ckpt' ]), ('llama2_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/llama2_7b.ckpt' ]), ('llama2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/llama2-13b-fp16.ckpt' ]), ('llama_7b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/open_llama_7b_lora.ckpt' ]), ('bloom_560m', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/bloom_560m.ckpt' ]), ('bloom_7.1b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/bloom_7.1b.ckpt' ]), ('sam_vit_b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_b_01ec64.ckpt' ]), ('sam_vit_l', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_l_0b3195.ckpt' ]), ('sam_vit_h', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/sam/sam_vit_h_4b8939.ckpt' ]) ]) _MODEL_CONFIG_URL_LIST = OrderedDict([ ('clip_vit_b_32', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/clip/clip_vit_b_32.yaml' ]), ('clip_vit_b_16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_b_16.yaml' ]), ('clip_vit_l_14', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14.yaml' ]), ('clip_vit_l_14@336', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/clip/clip_vit_l_14%40336.yaml' ]), ('blip2_stage1_vit_g', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_stage1_vit_g.yaml' ]), ('blip2_stage1_classification', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/blip2_stage1_classification.yaml' ]), ('mae_vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/mae/mae_vit_base_p16.yaml' ]), ('vit_base_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/vit/vit_base_p16.yaml' ]), ('vit_g_p16', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/blip2/vit_g_p16.yaml' ]), ('swin_base_p4w7', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/swin/swin_base_p4w7.yaml' ]), ('t5_small', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/t5/mindspore_model.yaml' ]), ('bert_tiny_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_tiny/bert_tiny_uncased.yaml' ]), ('bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/bert_base/bert_base_uncased.yaml' ]), ('tokcls_bert_base_chinese', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/tokcls_bert_base_chinese.yaml' ]), ('tokcls_bert_base_chinese_cluener', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/token_classification/' 'tokcls_bert_base_chinese_cluener.yaml' ]), ('txtcls_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/txtcls_bert_base_uncased.yaml' ]), ('txtcls_bert_base_uncased_mnli', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/txtcls/' 'txtcls_bert_base_uncased_mnli.yaml' ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/qa_bert_base_uncased.yaml' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/bert/downstream_tasks/qa/' 'qa_bert_base_uncased_squad.yaml' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/gpt2.yaml' ]) ]) _PIPELINE_INPUT_DATA_LIST = OrderedDict([ ('zero_shot_image_classification', "https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") ]) _TOKENIZER_SUPPORT_URL_LIST = OrderedDict([ ('clip_vit_b_32', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('mindspore/clip_vit_b_32', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/clip_vit_b_32/file/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_b_16', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_l_14', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('clip_vit_l_14@336', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" ]), ('t5_small', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/t5/spiece.model" ]), ('bert_base_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/vocab.txt" ]), ('bert_tiny_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/vocab.txt" ]), ('tokcls_bert_base_chinese', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_chinese/vocab.txt" ]), ('tokcls_bert_base_chinese_cluener', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_chinese/vocab.txt" ]), ('mindspore/tokcls_bert_base_chinese_cluener', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/tokcls_bert_base_chineses_cluener/file/vocab.txt" ]), ('txtcls_bert_base_uncased', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_english/vocab.txt" ]), ('txtcls_bert_base_uncased_mnli', [ "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" "XFormer_for_mindspore/bert/bert_base_english/vocab.txt" ]), ('mindspore/txtcls_bert_base_uncased_mnli', [ "https://xihe.mindspore.cn/api/v1/repo/model/MindSpore/txtcls_bert_base_uncased_mnli/file/vocab.txt" ]), ('qa_bert_base_uncased', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/' 'XFormer_for_mindspore/bert/bert_base_english/vocab.txt' ]), ('qa_bert_base_uncased_squad', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/' 'XFormer_for_mindspore/bert/bert_base_english/vocab.txt' ]), ('gpt2', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_xl', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_xl_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('gpt2_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/vocab.json', 'https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/gpt2/merges.txt' ]), ('pangualpha_2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/vocab.model' ]), ('pangualpha_13b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/pangualpha/vocab.model' ]), ('glm_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm/ice_text.model' ]), ('glm2_6b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/glm2/tokenizer.model' ]), ('llama_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/tokenizer.model' ]), ('llama_7b_lora', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/XFormer_for_mindspore/llama/tokenizer.model' ]), ('llama2_7b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/llama2/tokenizer.model' ]), ('bloom_560m', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_7.1b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_65b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ('bloom_176b', ['https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com' '/MindFormers/bloom/tokenizer.json' ]), ]) _TOKENIZER_NAME_TO_PROCESSOR = OrderedDict([ ('ChatGLMTokenizer', 'GLMProcessor'), ('ChatGLM2Tokenizer', 'GLMProcessor'), ('CLIPTokenizer', 'CLIPProcessor'), ('BertTokenizer', 'BertProcessor'), ('T5Tokenizer', 'T5Processor'), ('LlamaTokenizer', 'LlamaProcessor'), ('GPT2Tokenizer', 'GPT2Processor'), ('PanguAlphaTokenizer', 'PanguAlphaProcessor'), ('BloomTokenizer', 'BloomProcessor'), ('InternLMTokenizer', 'LlamaProcessor'), ('Baichuan2Tokenizer', 'LlamaProcessor') ]) @classmethod def show_trainer_support_model_list(cls, task=None): """show_trainer_support_model_list""" all_list = copy.deepcopy(cls._TRAINER_SUPPORT_TASKS_LIST) all_list.pop("general") for key, val in all_list.items(): val.pop("common") temp_list = [] for model_name, _ in val.items(): temp_list.append(model_name) all_list[key] = temp_list if task: if task in all_list.keys(): all_list = all_list[task] logger.info("Trainer support model list for %s task is: ", str(task)) print_path_or_list(all_list) else: raise KeyError("unsupported task") else: logger.info("Trainer support model list of MindFormer is: ") print_dict(all_list) @classmethod def show_pipeline_support_model_list(cls, task=None): """show_pipeline_support_model_list""" all_list = copy.deepcopy(cls._PIPELINE_SUPPORT_TASK_LIST) for key, val in all_list.items(): val.pop("common") temp_list = [] for model_name, _ in val.items(): temp_list.append(model_name) all_list[key] = temp_list if task: if task in all_list.keys(): all_list = all_list[task] logger.info("Pipeline support model list for %s task is: ", str(task)) print_path_or_list(all_list) else: raise KeyError("unsupported task") else: logger.info("Pipeline support model list of MindFormer is: ") print_dict(all_list) @classmethod def show_tokenizer_name_to_processor(cls): """show_tokenizer_name_to_processor function""" logger.info("_TRAINER_SUPPORT_TASKS_LIST of MindFormer is: ") print_dict(cls._TOKENIZER_NAME_TO_PROCESSOR) @classmethod def get_tokenizer_name_to_processor(cls): """get_tokenizer_name_to_processor function""" return cls._TOKENIZER_NAME_TO_PROCESSOR @classmethod def show_trainer_support_task_list(cls): """show_trainer_support_task_list function""" logger.info("_TRAINER_SUPPORT_TASKS_LIST of MindFormer is: ") print_dict(cls._TRAINER_SUPPORT_TASKS_LIST) @classmethod def show_pipeline_support_input_data_list(cls): """show_pipeline_support_input_data_list function""" logger.info("_PIPELINE_INPUT_DATA_LIST of MindFormer is: ") print_dict(cls._PIPELINE_INPUT_DATA_LIST) @classmethod def get_pipeline_support_input_data_list(cls): """get_pipeline_support_input_data_list function""" return cls._PIPELINE_INPUT_DATA_LIST @classmethod def get_trainer_support_task_list(cls): """get_trainer_support_task_list function""" return cls._TRAINER_SUPPORT_TASKS_LIST @classmethod def show_pipeline_support_task_list(cls): """show_pipeline_support_task_list function""" logger.info("_PIPELINE_SUPPORT_TASK_LIST of MindFormer is: ") print_dict(cls._PIPELINE_SUPPORT_TASK_LIST) @classmethod def get_pipeline_support_task_list(cls): """get_pipeline_support_task_list function""" return cls._PIPELINE_SUPPORT_TASK_LIST @classmethod def show_model_config_to_name(cls): """show_model_config_to_name function""" print_dict(cls._MODEL_CONFIG_TO_NAME) @classmethod def get_model_config_to_name(cls): """show_model_config_to_name function""" return cls._MODEL_CONFIG_TO_NAME @classmethod def set_model_config_to_name(cls, model_config, model_name): """ set_model_config_to_name function Args: model_config (str): the name of model config model_name (str): the name of model """ cls._MODEL_CONFIG_TO_NAME.update({ model_config: model_name }) @classmethod def show_model_config_url_list(cls): """show_model_config_url_list function""" logger.info("_MODEL_CONFIG_URL_LIST of MindFormer is: ") print_dict(cls._MODEL_CONFIG_URL_LIST) @classmethod def get_model_config_url_list(cls): """get_model_config_url_list function""" return cls._MODEL_CONFIG_URL_LIST @classmethod def show_project_path(cls): """show_project_path function""" logger.info("_PROJECT_PATH of MindFormer is: ") print_path_or_list(cls._PROJECT_PATH) @classmethod def get_project_path(cls): """get_project_path function""" return cls._PROJECT_PATH @classmethod def get_xihe_checkpoint_download_folder(cls): """get xihe's mindspore checkpoint download folder.""" return os.path.join(cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER, 'mindspore') @classmethod def show_default_checkpoint_download_folder(cls): """show_default_checkpoint_download_folder function""" logger.info("_DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER of MindFormer is: ") print_path_or_list(cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER) @classmethod def get_default_checkpoint_download_folder(cls): """get_default_checkpoint_download_folder function""" return cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER @classmethod def set_default_checkpoint_download_folder(cls, download_folder): """ set_default_checkpoint_download_folder function Args: download_folder (str): the path of default checkpoint download folder """ if not os.path.isdir(download_folder): raise TypeError(f"{download_folder} should be a directory.") cls._DEFAULT_CHECKPOINT_DOWNLOAD_FOLDER = download_folder @classmethod def get_default_checkpoint_save_folder(cls): """get_default_checkpoint_save_folder function""" return cls._DEFAULT_CHECKPOINT_SAVE_FOLDER @classmethod def set_default_checkpoint_save_folder(cls, save_folder): """ set_default_checkpoint_save_folder function Args: save_folder (str): the path of default checkpoint save folder """ if not os.path.isdir(save_folder): raise TypeError(f"{save_folder} should be a directory.") cls._DEFAULT_CHECKPOINT_SAVE_FOLDER = save_folder @classmethod def show_default_checkpoint_save_folder(cls): """show_default_checkpoint_save_folder function""" logger.info("_DEFAULT_CHECKPOINT_SAVE_FOLDER of MindFormer is: ") print_path_or_list(cls._DEFAULT_CHECKPOINT_SAVE_FOLDER) @classmethod def show_config_support_list(cls): """show_config_support_list function""" logger.info("CONFIG_SUPPORT_LIST of MindFormer is: ") print_dict(cls._CONFIG_SUPPORT_LIST) @classmethod def get_config_support_list(cls): """get_config_support_list function""" return cls._CONFIG_SUPPORT_LIST @classmethod def show_model_support_list(cls): """show_model_support_list function""" logger.info("MODEL_SUPPORT_LIST of MindFormer is: ") print_dict(cls._MODEL_SUPPORT_LIST) @classmethod def get_model_support_list(cls): """get_model_support_list function""" return cls._MODEL_SUPPORT_LIST @classmethod def show_processor_support_list(cls): """show_processor_support_list function""" logger.info("PROCESSOR_SUPPORT_LIST of MindFormer is: ") print_dict(cls._PROCESSOR_SUPPORT_LIST) @classmethod def get_processor_support_list(cls): """get_processor_support_list function""" return cls._PROCESSOR_SUPPORT_LIST @classmethod def get_tokenizer_support_list(cls): """get_tokenizer_support_list function""" return cls._TOKENIZER_SUPPORT_LIST @classmethod def show_tokenizer_support_list(cls): """show_tokenizer_support_list function""" print_dict(cls._TOKENIZER_SUPPORT_LIST) @classmethod def get_tokenizer_url_support_list(cls): """get_tokenizer_url_support_list function""" return cls._TOKENIZER_SUPPORT_URL_LIST @classmethod def show_tokenizer_url_support_list(cls): """show_tokenizer_url_support_list function""" print_dict(cls._TOKENIZER_SUPPORT_URL_LIST) @classmethod def get_model_name_support_list(cls): """get_model_name_support_list""" support_model_name = [] for task_name in cls._TRAINER_SUPPORT_TASKS_LIST.keys(): support_model_name.extend(cls._TRAINER_SUPPORT_TASKS_LIST.get(task_name).keys()) return set(support_model_name) @classmethod def show_model_ckpt_url_list(cls): """show_model_ckpt_url_list function""" logger.info("MODEL_CKPT_URL_LIST of MindFormer is: ") print_dict(cls._MODEL_CKPT_URL_LIST) @classmethod def get_model_ckpt_url_list(cls): """get_model_ckpt_url_list function""" return cls._MODEL_CKPT_URL_LIST pengcheng.mind-mindformer/mindformers/models/000077500000000000000000000000001476650141400217225ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/.ipynb_checkpoints/000077500000000000000000000000001476650141400255135ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000036231476650141400317550ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .bert import * #from .mae import * #from .vit import * #from .swin import * #from .blip2 import * #from .clip import * #from .t5 import * #from .gpt2 import * #from .glm import * #from .glm2 import * from .llama import * #from .pangualpha import * #from .filip import * #from .bloom import * #from .sam import * from .base_tokenizer import * from .base_config import BaseConfig from .base_model import BaseModel from .base_processor import BaseProcessor, BaseImageProcessor, BaseAudioProcessor from .build_tokenizer import build_tokenizer from .build_processor import build_processor from .build_model import build_model_config, build_head, \ build_model, build_encoder __all__ = ['BaseConfig', 'BaseModel', 'BaseProcessor', 'BaseImageProcessor', 'BaseAudioProcessor'] #__all__.extend(blip2.__all__) #__all__.extend(mae.__all__) #__all__.extend(vit.__all__) #__all__.extend(swin.__all__) #__all__.extend(clip.__all__) #__all__.extend(t5.__all__) #__all__.extend(gpt2.__all__) #__all__.extend(glm.__all__) #__all__.extend(glm2.__all__) #__all__.extend(pangualpha.__all__) #__all__.extend(filip.__all__) #__all__.extend(bloom.__all__) __all__.extend(bert.__all__) __all__.extend(llama.__all__) __all__.extend(base_tokenizer.__all__) pengcheng.mind-mindformer/mindformers/models/.ipynb_checkpoints/base_config-checkpoint.py000066400000000000000000000243531476650141400324600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseConfig class, which is all model configs' base class """ import os import shutil import yaml from ..mindformer_book import MindFormerBook from ..mindformer_book import print_path_or_list from ..tools import logger from ..tools.register.config import MindFormerConfig from ..models.build_config import build_model_config class BaseConfig(dict): """ Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super(BaseConfig, self).__init__() self.update(kwargs) def __getattr__(self, key): if key not in self: return None return self[key] def __setattr__(self, key, value): self[key] = value def __delattr__(self, key): del self[key] def to_dict(self): """ for yaml dump, transform from Config to a strict dict class """ return_dict = {} for key, val in self.items(): if isinstance(val, BaseConfig): val = val.to_dict() return_dict[key] = val return return_dict @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif yaml_name_or_path not in cls._support_list: raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) save_path = os.path.join(save_directory, save_name + ".yaml") parsed_config, removed_list = self._inverse_parse_config() wraped_config = self._wrap_config(parsed_config) for key, val in removed_list: self[key] = val self.remove_type() meraged_dict = {} if os.path.exists(save_path): with open(save_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(save_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("config saved successfully!") def remove_type(self): """remove type caused by save’""" if isinstance(self, BaseConfig): self.pop("type") for key, val in self.items(): if isinstance(val, BaseConfig): val.pop("type") self.update({key: val}) def inverse_parse_config(self): """inverse_parse_config""" val, _ = self._inverse_parse_config() return val def _inverse_parse_config(self): """ Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. """ self.update({"type": self.__class__.__name__}) removed_list = [] for key, val in self.items(): if isinstance(val, BaseConfig): val = val.inverse_parse_config() elif not isinstance(val, (str, int, float, bool)): removed_list.append((key, val)) continue self.update({key: val}) for key, _ in removed_list: self.pop(key) return self, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/models/__init__.py000066400000000000000000000036231476650141400240370ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .bert import * #from .mae import * #from .vit import * #from .swin import * #from .blip2 import * #from .clip import * #from .t5 import * #from .gpt2 import * #from .glm import * #from .glm2 import * from .llama import * #from .pangualpha import * #from .filip import * #from .bloom import * #from .sam import * from .base_tokenizer import * from .base_config import BaseConfig from .base_model import BaseModel from .base_processor import BaseProcessor, BaseImageProcessor, BaseAudioProcessor from .build_tokenizer import build_tokenizer from .build_processor import build_processor from .build_model import build_model_config, build_head, \ build_model, build_encoder __all__ = ['BaseConfig', 'BaseModel', 'BaseProcessor', 'BaseImageProcessor', 'BaseAudioProcessor'] #__all__.extend(blip2.__all__) #__all__.extend(mae.__all__) #__all__.extend(vit.__all__) #__all__.extend(swin.__all__) #__all__.extend(clip.__all__) #__all__.extend(t5.__all__) #__all__.extend(gpt2.__all__) #__all__.extend(glm.__all__) #__all__.extend(glm2.__all__) #__all__.extend(pangualpha.__all__) #__all__.extend(filip.__all__) #__all__.extend(bloom.__all__) __all__.extend(bert.__all__) __all__.extend(llama.__all__) __all__.extend(base_tokenizer.__all__) pengcheng.mind-mindformer/mindformers/models/__pycache__/000077500000000000000000000000001476650141400241325ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/__pycache__/__init__.cpython-39.pyc000066400000000000000000000013351476650141400303240ustar00rootroot00000000000000a ?cf@sdZddlTddlTddlTddlmZddlmZddlm Z m Z m Z ddl m Z ddl m Z ddlmZmZmZmZgd Zeejeejeejd S) z models init)*) BaseConfig) BaseModel) BaseProcessorBaseImageProcessorBaseAudioProcessor)build_tokenizer)build_processor)build_model_config build_head build_model build_encoder)rrrrrN)__doc__ZbertllamaZbase_tokenizer base_configr base_modelrZbase_processorrrrrr r r r r __all__extendrrF/home/ma-user/work/mindformers-baichuan/mindformers/models/__init__.pys       pengcheng.mind-mindformer/mindformers/models/__pycache__/base_config.cpython-39.pyc000066400000000000000000000203131476650141400310210ustar00rootroot00000000000000a ?cf(@sldZddlZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z Gd d d e ZdS) z: BaseConfig class, which is all model configs' base class N)MindFormerBook)print_path_or_list)logger)MindFormerConfig)build_model_configcseZdZdZgZdZdZfddZddZdd Z d d Z d d Z e ddZ d ddZddZddZddZddZe ddZe ddZZS)! BaseConfiga Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) rc stt|||dSN)superr__init__update)selfkwargs __class__I/home/ma-user/work/mindformers-baichuan/mindformers/models/base_config.pyr =szBaseConfig.__init__cCs||vr dS||Sr rrkeyrrr __getattr__AszBaseConfig.__getattr__cCs |||<dSr r)rrvaluerrr __setattr__FszBaseConfig.__setattr__cCs ||=dSr rrrrr __delattr__IszBaseConfig.__delattr__cCs4i}|D]"\}}t|tr&|}|||<q |S)zU for yaml dump, transform from Config to a strict dict class )items isinstancerto_dict)r return_dictrvalrrrrLs   zBaseConfig.to_dictc Ks|dd}|dur|}t|ts6tdt|dtj|rr|dsZt |dt |}t d|n||j vrt |d|j dn|}|d r|d |j}tjt|d |j}ntjt|d |j}tj|s tj|d d tj||d}dd}tj|s||} tj| rrtj| rrt| |t d|ntd| t |}t|jj} tt| |jj j| S)a From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. pretrained_model_name_or_pathNz+yaml_name_or_path should be a str, but got ..yamlz) should be a .yaml file for model config.z.the content in %s is used for config building.zg is not a supported model type or a valid path to model config. supported model could be selected from mindspore/_Texist_okcSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget) model_namedefault_yaml_file model_dictrrrget_default_yaml_files  z9BaseConfig.from_pretrained..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get )!poprstr TypeErrortypeospathexistsendswith ValueErrorrrinfo _support_list startswithsplit _model_namejoinr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_foldermakedirsrealpathshutilcopyFileNotFoundErrorrmodel model_configset_model_config_to_nameidarch) clsZyaml_name_or_pathrr config_argsZ yaml_nameZcheckpoint_path yaml_filer.r,configrrrfrom_pretrainedXsR            zBaseConfig.from_pretrainedNmindspore_modelc CsT|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |ddtj ||d}| \}}| |}|D]\}}|||<q| i} tj|rt|d$} tj| tjd } Wdn1s0Y| | |t|d  } | t| Wdn1s40Y| td dS) z Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. Nz6save_directory and save_name should be a str, but got z and r Tr%r!r)Loaderwzconfig saved successfully!)r"get_default_checkpoint_save_folderrr0r1r2r3r4r5rAr=_inverse_parse_config _wrap_config remove_typeopenyamlloadreadrRcloser writedumprr8) rsave_directory save_name save_pathZ parsed_config removed_listZ wraped_configrrZ meraged_dict file_reader file_pointerrrrsave_pretraineds4         2  0zBaseConfig.save_pretrainedcCsLt|tr|d|D]*\}}t|tr|d|||iqdS)uremove type caused by save’r2N)rrr/rr )rrrrrrrWs     zBaseConfig.remove_typecCs|\}}|S)inverse_parse_config)rU)rrr$rrrrfs zBaseConfig.inverse_parse_configcCs|d|jjig}|D]L\}}t|tr:|}n"t|ttt t fs\| ||fq|||iq|D]\}}| |qp||fS)z Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. r2) r r__name__rrrrfr0intfloatboolappendr/)rrbrrr$rrrrUs    z BaseConfig._inverse_parse_configcCs>|dd}|dur(tt|d}d|d|idiS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. r+NrFr2)rGrJ)r/rget_model_config_to_namer*rIr)rrNr+rrrrVs zBaseConfig._wrap_configcCstd|jt|jdS)zshow support list methodzsupport list of %s is:N)rr8rgrr9rKrrrshow_support_listszBaseConfig.show_support_listcCs|jS)zget support list method)r9rmrrrget_support_list szBaseConfig.get_support_list)NrP)rg __module__ __qualname____doc__r9r?r<r rrrr classmethodrOrerWrfrUrVrnro __classcell__rrrrr s(   N '  r)rrr3rCrYmindformer_bookrrtoolsrZtools.register.configrZmodels.build_configrdictrrrrrs     pengcheng.mind-mindformer/mindformers/models/__pycache__/base_model.cpython-39.pyc000066400000000000000000000263001476650141400306560ustar00rootroot00000000000000a ?cfA@sdZddlmZddlZddlZddlZddlZddlmZddl m Z m Z ddl m Z ddlmZmZd d lmZd d lmZdd lmZdd lmZddlmZddlmZGdddeje ZdS)z BaseModel )OptionalN)nn)load_checkpointload_param_into_net)GeneratorMixin)MindFormerBookprint_path_or_list)build_model_config) BaseConfig)MindFormerConfig)download_with_progress_bar)logger) try_sync_filecseZdZdZgZdZdZedfdd ZddZ de e e d d d Z ddZ ddZddZee dddZeddZeddZZS) BaseModela[ The base model that contains the class method `from_pretained` and `save_pretrained`, any new model that should inherit the class. Note: GeneratorMixin provides the method `generate` that enable the generation for nlp models. Args: config(BaseConfig): The model configuration that inherits the `BaseConfig`. rr )configc s&tt|jfi|||_d|_dS)N)superr__init__r default_checkpoint_download_path)selfrkwargs __class__H/home/ma-user/work/mindformers-baichuan/mindformers/models/base_model.pyr5szBaseModel.__init__c Cs|j}|rt|ts(tdt|tj|r~t|}|}zt ||t d|Wnt yxt d|Yn0q||jvrt|d|jdq|}|dr|d|j}tjt|d|j}ntjt|d|j}tj|stj|d d tj||d }tj|sjt||j}t||}|sjt d d St|||_t dtj|t|}zt ||t d|Wn t yt d|Yn0n t dd S)a1 load checkpoint for models. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. z1checkpoint_name_or_path should be a str, but got zweights in %s are loadedzJthe given config and weights in %s are mismatched, and weights load failedzT is not a supported default model or a valid path to checkpoint, please select from . mindspore/_Texist_ok.ckptz@checkpoint download failed, and pretrained weights are unloaded.Nzstart to read the ckpt file: %szmodel built, but weights is unloaded, since the config has no checkpoint_name_or_path attribute or checkpoint_name_or_path is None.)checkpoint_name_or_path isinstancestr TypeErrortypeospathexistsrrrinfo RuntimeErrorerror _support_list ValueError startswithsplit _model_namejoinr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_foldermakedirsget_model_ckpt_url_listrrrgetsize) rrr#param ckpt_fileZcheckpoint_nameZ"default_checkpoint_download_folderurlsucceedrrrr:sh           zBaseModel.load_checkpointNmindspore_model)save_directory save_namec Cs|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |ddtj ||d}tj ||d}t |||j durtd ||j \}}||}|D]\}} | |j |<q||j i} tj|r6t|d $} tj| tjd } Wdn1s$0Y| | |t|d  } | t| Wdn1sr0Y| td dS)a  Save the model weight and configuration file. (only supports standalone mode, and distribute mode waits for developing) Args: save_directory(str): a directory to save the model weight and configuration. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. If set, the directory will be what is set. save_name(str): the name of saved files, including model weight and configuration file. Default mindspore_model. Examples: >>> import os >>> from mindformers import T5ForConditionalGeneration, MindFormerBook >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') >>> net.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'mindspore_model.ckpt'] Nz6save_directory and save_name should be a str, but got z and rTr r".yamlz"the model has no config attribute.r)Loaderwzmodel saved successfully!)r"get_default_checkpoint_save_folderr$r%r&r'r(r)r*r7r3mssave_checkpointrAttributeError_inverse_parse_config _wrap_config remove_typeopenyamlloadreadrCcloseupdatewritedumprr+) rr?r@checkpoint_pathZ config_path parsed_configZ remove_list wraped_configkeyval meraged_dict file_reader file_pointerrrrsave_pretraineds<         4  0zBaseModel.save_pretrainedcCsLt|tr|d|D]*\}}t|tr|d|||iqdS)uremove type caused by save’r'N)r$r popitemsrQ)rrrWrXrrrrKs     zBaseModel.remove_typecCsg}t|ts||fS|jj}|d|i|D]H\}}||\}}t|ttt t tfsn| ||fq4|||iq4|D]\}}| |q||fS)a Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. r') r$r r__name__rQr^rIr%intfloatboolappendr])rr removed_list class_namerWrXnew_valrrrrrIs    zBaseModel._inverse_parse_configcCs|jj}d|d|idiS)a Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. modelr') model_configarch)rr_to_dict)rr model_namerrrrJs zBaseModel._wrap_config)pretrained_model_name_or_dircKs|dd}|dur|}t|ts4tdt|tj|}tj|}|sx||j vrxt |d|j d|j d|r|st |d|rLdd t |D}d d t |D}|r|st d |dtj|||j}tj|||j} td || t|} | jjjfi|t| jj} | d | i|| } n4|} |dr| d|j} tjt| d|j}ntjt| d|j}tj|stj|ddtj|| d}dd}tj|s:|| }tj|r,tj|r,t ||td|nt d|t!|t|} | jjjfi|t| jj} | d |i|| } td| S)a. Instantiates a model by the pretrained_model_name_or_dir. It download the model weights if the user pass a model name, or load the weight from the given directory if given the path. (only support standalone mode, and distribute mode waits for developing!) Args: pretrained_model_name_or_dir (str): It supports the following two input types. If `pretrained_model_name_or_dir` is a supported model name, for example, `vit_base_p16` and `t5_small`, it will download the necessary files from the cloud. User can pass one from the support list by call `MindFormerBook.get_model_support_list()`. If `pretrained_model_name_or_dir` is a path to the local directory where there should have model weights ended with `.ckpt` and configuration file ended with `yaml`. pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Examples: >>> from mindformers import T5ForConditionalGeneration >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') Returns: A model, which inherited from BaseModel. pretrained_model_name_or_pathNz6pretrained_model_name_or_dir should be a str, but got z+ does not exist, or it is not supported by z. please select from rz is not a directory.cSsg|]}|dr|qS)rAendswith.0filerrr s z-BaseModel.from_pretrained..cSsg|]}|dr|qS)r"rnrprrrrs s zIthere is no yaml file for model config or ckpt file for model weights in z;config in %s and weights in %s are used for model building.r#rrrTr rAcSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget)rkdefault_yaml_file model_dictrrrget_default_yaml_fileDs  z8BaseModel.from_pretrained..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get zmodel built successfully!)"r]r$r%r&r'r(r)r*isdirr.r/r_listdirFileNotFoundErrorr3r5rr+r rgrhrQr r0r1r2rr4r6r7realpathshutilcopyr)clsrlrrmis_existis_dir yaml_list ckpt_list yaml_filer; config_argsrrgZpretrained_model_namerTrzrxrrrfrom_pretrainedsz              zBaseModel.from_pretrainedcCstd|jt|jdS)zshow_support_list methodzsupport list of %s is:N)rr+r_r r.rrrrshow_support_list\szBaseModel.show_support_listcCs|jS)zget_support_list method)r.rrrrget_support_listbszBaseModel.get_support_list)Nr>)r_ __module__ __qualname____doc__r.r5r2r rrrr%r\rKrIrJ classmethodrrr __classcell__rrrrr&s* I <  i r)rtypingrr(rrMrrFrmindspore.train.serializationrr generationrmindformer_bookrr build_configr base_configr Ztools.registerr Ztools.download_toolsrZ tools.loggerrZ tools.utilsrCellrrrrrs          pengcheng.mind-mindformer/mindformers/models/__pycache__/base_processor.cpython-39.pyc000066400000000000000000000252011476650141400315740ustar00rootroot00000000000000a ?cfO4@sdZddlZddlZddlZddlmZmZddlmZddlm Z ddl m Z dd l m Z Gd d d ZGd d d ZGdddZdS)z BaseProcessor N)print_path_or_listMindFormerBook)build_processor) BaseTokenizer)logger)MindFormerConfigc@s(eZdZdZddZddZddZdS) BaseImageProcessora BaseImageProcessor for all image preprocess. Examples: >>> from mindspore.dataset.vision.transforms import CenterCrop >>> from mindformers.models.base_processor import BaseImageProcessor >>> image_resolution = 224 >>> class MyImageProcessor(BaseImageProcessor): ... def __init__(self, image_resolution): ... super(MyImageProcessor, self).__init__(image_resolution=image_resolution) ... self.center_crop = CenterCrop(image_resolution) ... ... def preprocess(self, images, **kwargs): ... res = [] ... for image in images: ... image = self.center_crop(image) ... res.append(image) ... return res ... >>> my_image_processor = MyImageProcessor(image_resolution) >>> output = my_image_processor(image) cKsi|_|j|dSNconfigupdateselfkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/models/base_processor.py__init__6szBaseImageProcessor.__init__cKs|j|fi|Szforward process preprocess)rZ image_datarrrr__call__:szBaseImageProcessor.__call__cKs tddS)preprocess methodz=Each image processor must implement its own preprocess methodNNotImplementedError)rimagesrrrrr>szBaseImageProcessor.preprocessN__name__ __module__ __qualname____doc__rrrrrrrr sr c@s(eZdZdZddZddZddZdS) BaseAudioProcessora BaseAudioProcessor for all audio preprocess. Examples: >>> from mindspore.dataset.audio import AllpassBiquad >>> from mindformers.models.base_processor import BaseAudioProcessor >>> sample_rate = 44100 >>> central_freq = 200.0 >>> class MyAudioProcessor(BaseAudioProcessor): ... def __init__(self, audio_property): ... super(MyAudioProcessor, self).__init__(sample_rate=sample_rate, central_freq=central_freq ) ... self.all_pass_biquad = AllpassBiquad(44100, 200.0) ... ... def preprocess(self, audio_data, **kwargs): ... res = [] ... for audio in audio_data: ... audio = self.all_pass_biquad(audio) ... res.append(audio) ... return res ... >>> my_audio_processor = MyAudioProcessor(sample_rate, central_freq) >>> output = my_audio_processor(audio) cKsi|_|j|dSr r rrrrr[szBaseAudioProcessor.__init__cKs|j|fi|SrrrZ audio_datarrrrr_szBaseAudioProcessor.__call__cKs tddS)rz=Each audio processor must implement its own preprocess methodNrr#rrrrcszBaseAudioProcessor.preprocessNrrrrrr"Csr"c@sleZdZdZgZdZdZddZdddZdd d Z d d Z ddZ e ddZ e ddZe ddZdS) BaseProcessoraT Base processor Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_processor import BaseProcessor >>> class MyProcessor(BaseProcessor): ... _support_list = MindFormerBook.get_processor_support_list()['my_model'] ... ... def __init__(self, image_processor=None, audio_processor=None, tokenizer=None, return_tensors='ms'): ... super(MyProcessor, self).__init__( ... image_processor=image_processor, ... audio_processor=audio_processor, ... tokenizer=tokenizer, ... return_tensors=return_tensors) ... >>> myprocessor = MyProcessor(image_processor, audio_processor, tokenizer) >>> output = mynet(image, audio, text) rrcKsji|_|j||dd|_|dd|_|dd|_|dd|_|dd|_|dd|_dS)Nimage_processoraudio_processor tokenizer max_lengthpaddingFreturn_tensors) r rpopr%r&r'r(r)r*rrrrrs zBaseProcessor.__init__NcCsi}|durF|jrFt|jts4tdt|jd||}||d<|dur|jrt|jtsvtdt|jdt|tr|g}|j||j|j |j dd}||d<|S) z call functionNzFfeature_extractor should inherit from the BaseImageProcessor, but got .imagez;tokenizer should inherited from the BaseTokenizer, but got )r*r(r) input_idstext) r% isinstancer TypeErrortyper'rstrr*r(r))r image_input text_inputoutputZ image_output text_outputrrrrs.       zBaseProcessor.__call__mindspore_modelc Cs6|durt}t|tr$t|tsBtdt|dt|dtj|s\tj |dd| |j }| |}tj ||d}i}tj|rt|d$}tj|tjd }Wdn1s0Y|||t|d  }|t|Wdn1s0Y|td dS) z Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. Nz6save_directory and save_name should be a str, but got z and r,Texist_ok.yamlr)Loaderwzprocessor saved successfully!)r"get_default_checkpoint_save_folderr0r3r1r2ospathexistsmakedirs_inverse_parse_configr _wrap_configjoinopenyamlloadreadr=closerwritedumprinfo) rsave_directory save_name parsed_config wraped_config config_path meraged_dict file_reader file_pointerrrrsave_pretraineds.      2  0zBaseProcessor.save_pretrainedcCsd|jji}|D]~\}}t|trNd|jji}||j|||iqt|ttfrd|jji}||j |||iq|||iq|S)a Inverse parse config method, which builds yaml file content for feature extractor config. Args: Config (dict): a dict, which contains input parameters of feature extractor. Returns: A dict, which follows the yaml content. r2) __class__ritemsr0rr init_kwargsr r"r )rr rQkeyvalZparsed_sub_configrrrrDs      z#BaseProcessor._inverse_parse_configcCsd|iS)z Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (dict): a dict processed by _inverse_parse_config function. Returns: A dict for yaml.dump. processorr)rr rrrrEs zBaseProcessor._wrap_configc Ks|dd}|dur|}t|ts4tdt|tj|}|sl||jvrlt |d|j d|jd|rt d|t |}n|}|dr|d |j}tjt|d |j}ntjt|d |j}tj|stj|d d tj||d }dd} tj|sp| |} tj| rbtj| rbt| |t d|ntd| t |}t|j} t d| S)a From pretrain method, which instantiates a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. pretrained_model_name_or_pathNz+yaml_name_or_path should be a str, but got z+ does not exist, or it is not supported by z. please select from r,z,config in %s is used for processor building. mindspore/_Tr9r;cSs0d}tD]}||vr||}q,q|S)N)rget_trainer_support_task_listvaluesget) model_namedefault_yaml_file model_dictrrrget_default_yaml_file"s  z.get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get zprocessor built successfully!)r+r0r3r1r2r@rArB _support_list ValueErrorrrrNr startswithsplit _model_namerFr#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_folderrCrealpathshutilcopyFileNotFoundErrorrr]) clsyaml_name_or_pathrr^is_exist config_args yaml_namecheckpoint_path yaml_filerirgr]rrrfrom_pretrainedsT            zBaseProcessor.from_pretrainedcCstd|jt|jdS)show_support_listzsupport list of %s is:N)rrNrrrjrvrrrr~8szBaseProcessor.show_support_listcCs|jS)zget_support_list method)rjrrrrget_support_list>szBaseProcessor.get_support_list)NN)Nr8)rrr r!rjrprnrrrWrDrE classmethodr}r~rrrrrr$hs  "  L r$)r!r@rsrHmindformer_bookrrrbase_tokenizerrtoolsrtools.registerr r r"r$rrrrs    $%pengcheng.mind-mindformer/mindformers/models/__pycache__/base_tokenizer.cpython-39.pyc000066400000000000000000004426771476650141400316130ustar00rootroot00000000000000a ?cf8@sdZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z m Z ddl mZddlmZmZmZmZmZmZmZmZddlmZmZddlmZmZddlmZddl Z ddl!Z"ddl#Z$dd l%m&Z&dd l'm(Z(m)Z)m*Z*d d l+m+Z+d dl,m-Z-d dl.m/Z/d dl0m1Z1gdZ2dZ3dZ4e5dZ6e5dZ7e8Z9ee8Z:ee5Z;ee8e8fZdZ?dZ@dZAdZBeCdZDGddde8eZEGdddeEZFe dddGdd d ZGe Gd!d"d"ZHGd#d$d$eEZIGd%d&d&eZJGd'd(d(eZKGd)d*d*eZLGd+d,d,eEZMGd-d.d.ZNd/ZOd0ZPd1ZQd2d3ZRd4d5ZSd6d7ZTd8d9ZUd:d;ZVdd?ZXee8e8d@dAdBZYdCdDZZGdEdFdFZ[eReQGdGdHdHe[Z\eReQe(]e)j^GdIdJdJe\Z_dS)Kz+Base Tokenizer for the pretrained tokenizerN) dataclassfield)Enum)AnyDictList NamedTupleOptionalSequenceTupleUnion) OrderedDictUserDict)MappingSized)contextmanager)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_tokenizer)download_with_progress_bar) try_sync_file)MindFormerBook) BaseTokenizer TokenizerSpecialTokensMixinzspecial_tokens_map.jsontokenizer_config.jsongꌠ9Y>)Fg@xDzadded_tokens.jsonztokenizer.jsonztokenizer\.(.*)\.jsonc@seZdZdZeddZdS) ExplicitEnumzC Enum with more explicit error message for missing values. cCs(t|d|jdt|jdS)Nz is not a valid z, please select one of ) ValueError__name__list_value2member_map_keys)clsvaluer(L/home/ma-user/work/mindformers-baichuan/mindformers/models/base_tokenizer.py _missing_MszExplicitEnum._missing_N)r" __module__ __qualname____doc__ classmethodr*r(r(r(r)r Hsr c@seZdZdZdZdZdS) TensorTypez Possible values for the `return_tensors` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. npmsN)r"r+r,r-NUMPY MINDSPOREr(r(r(r)r/Tsr/T)frozeneqc@sdeZdZUdZeedZeed<dZe ed<dZ e ed<dZ e ed<dZ e ed <d d Z d d ZdS) AddedTokenz AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. )default_factorycontentF single_wordlstriprstripT normalizedcCs|jSN)__dict__selfr(r(r) __getstate__jszAddedToken.__getstate__cCs|jSr=)r8r?r(r(r)__str__mszAddedToken.__str__N)r"r+r,r-rstrr8__annotations__r9boolr:r;r<rArBr(r(r(r)r6]s     r6c@seZdZdZdS) EncodingFastz_This is dummy class because without the `tokenizers` library we don't have these objects anywayN)r"r+r,r-r(r(r(r)rFqsrFc@s eZdZdZdZdZdZdZdS)TruncationStrategyz Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. Z only_firstZ only_second longest_firstdo_not_truncateN)r"r+r,r- ONLY_FIRST ONLY_SECOND LONGEST_FIRSTDO_NOT_TRUNCATEr(r(r(r)rGvs rGc@s"eZdZUdZeed<eed<dS)CharSpanz Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. startendNr"r+r,r-intrDr(r(r(r)rNs rNc@s"eZdZUdZeed<eed<dS) TokenSpanz Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. rOrPNrQr(r(r(r)rSs rScs(eZdZdZde eee dd"d#Zd?e eee dd$d%Zd@e eee dd&d'ZdAe ee e d(d)d*ZdBe ee e d(d+d,Z dCe ee e ee!d-d.d/Z"dDe ee e#d(d0d1Z$dEe ee e e d2d3d4Z%dFe ee e e#d-d5d6Z&dGe ee e e d2d7d8Z'dHeeee fe d9d:d;Z(Z)S)I BatchEncodinga& Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. NF)dataencoding tensor_typeprepend_batch_axis n_sequencescsXt|t|tr|g}||_|dur@|dur@|r@|dj}||_|j||ddS)NrrWrX)super__init__ isinstancerF _encodingsrY _n_sequencesconvert_to_tensors)r@rUrVrWrXrY __class__r(r)r\s   zBatchEncoding.__init__returncCs|jS)z `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) )r_r?r(r(r)rYszBatchEncoding.n_sequencescCs |jduS)z `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. Nr^r?r(r(r)is_fastszBatchEncoding.is_fast)itemrdcsXttrjSjdur(jSttrLfddjDStddS)a If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. Ncsi|]}|j|qSr(rU).0keyrgr@r(r) z-BatchEncoding.__getitem__..zInvalid key. Only three types of key are available: (1) string, (2) integers for backend Encoding, and (3) slices for data subsetting.)r]rCrUr^slicer%KeyErrorr@rgr(rkr) __getitem__s     zBatchEncoding.__getitem__)rgcCs(z |j|WSty"tYn0dSr=)rUroAttributeErrorrpr(r(r) __getattr__s  zBatchEncoding.__getattr__cCs|j|jdS)N)rU encodingsrUr^r?r(r(r)rAszBatchEncoding.__getstate__cCs(d|vr|d|_d|vr$|d|_dS)NrUrtru)r@stater(r(r) __setstate__s zBatchEncoding.__setstate__cCs |jSr=)rUr%r?r(r(r)r%szBatchEncoding.keyscCs |jSr=)rUvaluesr?r(r(r)rxszBatchEncoding.valuescCs |jSr=)rUitemsr?r(r(r)ry szBatchEncoding.itemscCs|jS)z `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. rer?r(r(r)rtszBatchEncoding.encodingsr) batch_indexrdcCs|jstd|j|jS)a Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[str]`: The list of tokens at that index. zgtokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r!tokensr@rzr(r(r)r{s zBatchEncoding.tokenscCs|jstd|j|jS)a Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. zmsequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r! sequence_idsr|r(r(r)r}*s zBatchEncoding.sequence_idscCs$|jstdtdt||S)> Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). zfwords() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).z`BatchEncoding.words()` property is deprecated and should be replaced with the identical, but more self-explanatory `BatchEncoding.word_ids()` property.)r^r!warningswarn FutureWarningword_idsr|r(r(r)wordsBs zBatchEncoding.wordscCs|jstd|j|jS)r~ziword_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast` class).)r^r!rr|r(r(r)rZs zBatchEncoding.word_ids)batch_or_token_index token_indexrdcCsX|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j||S)a Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zGtoken_to_sequence() is not available when using Python based tokenizersNr)r^r! _batch_size_seq_lentoken_to_sequencer@rrrzr(r(r)rms  zBatchEncoding.token_to_sequencecCsX|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j||S)a  Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. zCtoken_to_word() is not available when using Python based tokenizersNr)r^r!rr token_to_wordrr(r(r)rs  zBatchEncoding.token_to_word)batch_or_word_index word_indexsequence_indexrdcCsn|jstd|dur|}nd}|}|dkr6|j|}|dkrH|j|}|j|||}|durjt|SdS)a- Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. zDword_to_tokens() is not available when using Python based tokenizersNr)r^r!rrword_to_tokensrS)r@rrrrzspanr(r(r)rs'  zBatchEncoding.word_to_tokenscCsH|jstd|dur|}nd}|}|j||}|durDt|SdS)a Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. zDtoken_to_chars() is not available when using Python based tokenizersNr)r^r!token_to_charsrN)r@rrrzZ span_indicesr(r(r)rszBatchEncoding.token_to_chars)batch_or_char_index char_indexrrdcCs6|jstd|dur|}nd}|}|j|||S)a Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token. zCchar_to_token() is not available when using Python based tokenizersNr)r^r! char_to_tokenr@rrrrzr(r(r)rs zBatchEncoding.char_to_tokencCs:|jstd|dur|}nd}|}t|j|||S)a/ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string zCword_to_chars() is not available when using Python based tokenizersNr)r^r!rN word_to_chars)r@rrrrzr(r(r)r?s$zBatchEncoding.word_to_charscCs6|jstd|dur|}nd}|}|j|||S)a@ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `List[int]`: Index or indices of the associated encoded token(s). zBchar_to_word() is not available when using Python based tokenizersNr)r^r! char_to_wordrr(r(r)rlszBatchEncoding.char_to_wordrZc s|dur |St|tst|}|tjkrBtj}tjdd}|}n d fdd dd}tj}|}|D]v\}}z$|rx||s||d} | ||<Wqjty} z0|d krt d | t d |d | WYd} ~ qjd} ~ 00qj|S)a Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. NcSs t|tjSr=)r]r1Tensorxr(r(r) is_ms_tensorsz6BatchEncoding.convert_to_tensors..is_ms_tensorcsnt|ttfr`t|dtttjfr`dd|D}tt|dkr`|dur`dd|Dtd}tj||dS)NrcSsg|] }t|qSr(lenrivalr(r(r) rmzGBatchEncoding.convert_to_tensors..as_tensor..rcSsg|]}t|qSr()r0asarrayrr(r(r)rrmdtype) r]r#tupler0ndarrayrsetobjectr)r'rZ value_lens as_tensorr(r)rs $z3BatchEncoding.convert_to_tensors..as_tensorcSs t|tjSr=)r]r0rrr(r(r)is_numpy_arraysz8BatchEncoding.convert_to_tensors..is_numpy_arrayroverflowing_tokenszUnable to create tensor returning overflowing tokens of different lengths. Please see if a fast version of this tokenizer is available to have this feature available.zUnable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`zY` in this case) have excessive nesting (inputs type `list` where type `int` is expected).)N) r]r/r3r1int32rr0ry Exceptionr!) r@rWrX tensor_dtyper is_tensorrrjr'tensorer(rr)r`sD    z BatchEncoding.convert_to_tensors)NNNFN)r)r)r)r)N)N)Nr)N)Nr)Nr)Nr)NF)*r"r+r,r-r rrCrr rFr r/rErRr\propertyrYrfrqrsrArwr%rxryrrtr{r}rrrrrSrrNrrrrr` __classcell__r(r(rar)rTsj  ''  5(  *  -(rTc@seZdZdZdZdZdZdS)PaddingStrategyz Possible values for the `padding` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. longest max_lengthZ do_not_padN)r"r+r,r-LONGEST MAX_LENGTH DO_NOT_PADr(r(r(r)rsrc@s`eZdZdZddZedddZeeeddd Zdd d Z dd dZ dddZ ddZ d S)Triez Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie cCs i|_dSr=rhr?r(r(r)r\sz Trie.__init__)wordcCsD|sdS|j}|D]$}||vr&||ni||<||}qd|d<dS)uy Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` Nrrh)r@rrefcharr(r(r)adds zTrie.addtextrdc Cst}dg}d}t|D]\}}|r.||kr.qt}d}|D]n\} } d| vr|j|||| |d\}} } }|| || d}qq@|| vr| |} | || <q@|| q@|j|||d}||kr||jvr|j|||<q|j |||d}| ||S)aY Will look for the words added to the trie within `text`. Output is the original string split along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` rFr)statescurrentrrOskipT)reset to_remover)rroffsets) r enumeraterry split_atom_3appendr split_atom_2rU split_atom_1cut_text) r@rrrrrZ current_charrrrO trie_pointerrPr(r(r)splits2!       z Trie.splitNcCs>|D]0\}}d|vrt|}||||q:q|S)atomic act of splitr)ryrr)r@rrrrOrrPr(r(r)r|s  zTrie.split_atom_1cCs|r i}n|D] }||=q|S)rr()r@rrrrOr(r(r)rs zTrie.split_atom_2c Cs|D]\}}||krqn"||kr8|d}|d} n|}|} |t|krT||nd} d|vrl|}|} |}| |vr|| }|d7}d|vr|}|} |}|t|krq||} qlq||| |fS)rrNr)ryr) r@rrrrOrZ lookstartZlooktrie_pointerZlookahead_indexrPZ next_charr(r(r)rs0   zTrie.split_atom_3cCsZ|t|g}d}|D]:}||kr4tdqn ||kr>q|||||}q|S)z cut thr text.rzbThere was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it anyway.)rrrerror)r@rrr{rOrPr(r(r)rsz Trie.cut_text)NNN)NNN)NNNNN) r"r+r,r-r\rCrrrrrrrr(r(r(r)rsk  %ra add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to encode the sequences with the special tokens relative to their model. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. a return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) a Class attributes (overridden by derived classes) - **vocab_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **max_model_input_sizes** (`Dict[str, Optional[int]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size. - **pretrained_init_configuration** (`Dict[str, Dict[str, Any]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`] method. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. model_input_names (`List[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they won't be split by the tokenization process. Will be associated to `self.additional_special_tokens` and `self.additional_special_tokens_ids`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. csfdd}|S)Ncs$|jdur|jndd|_|SNr)r-join)fndocstrr(r)docstring_decorators z/add_end_docstrings..docstring_decoratorr()rrr(rr)add_end_docstringss rcCsrt|ttfr dd|DSt|ttfr.cSsg|] }t|qSr(r)rior(r(r)rrmzto_py_obj..) r]dictrryr#rr1rasnumpytolistr0rnumber)objr(r(r)rs  rcCs&|dvr dSt|}|dkr"dSdS)z0Checks whether `char` is a whitespace character.)    TZsF) unicodedatacategoryrcatr(r(r)_is_whitespaces  rcCs(|dvr dSt|}|dr$dSdS)z-Checks whether `char` is a control character.)rrrFCT)rr startswithrr(r(r) _is_controls   rcCst|}d|krdksbnd|kr2dksbnd|krHdksbnd|kr^dkrfnnd St|}|d r~d Sd S) z1Checks whether `char` is a punctuation character.!/:@[`{~TPF)ordrrr)rcprr(r(r)_is_punctuationsZ  rcCs$|d}tt|t|Bt|BS)zcChecks whether the last character in text is one of a punctuation, control or whitespace character.rErrr)rZ last_charr(r(r)_is_end_of_wordsrcCs$|d}tt|t|Bt|BS)zdChecks whether the first character in text is one of a punctuation, control or whitespace character.rr)r first_charr(r(r)_is_start_of_wordsr) token_list new_tokencCs4t||}|t|kr$|||ks0|||dS)zm Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. N)bisect bisect_leftrinsert)rrZ insertion_idxr(r(r)!_insert_one_token_to_ordered_lists rcCs<t|ttfr&td|tdntt|ddS)z Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. z %sz%-------------------------------------z% is unsupported by print_path_or_listN)r]rCr#rinfo TypeErrortype)Zinput_path_or_listr(r(r)print_path_or_lists  r c @sveZdZdZgdZdOddZedddZdPee e e e ffed d d Z dQe e e e e e e ffeed ddZdRe e e e e feed ddZee dddZee dddZee dddZee dddZee dddZee dddZee dddZee e dd d!Zejd"dZejd#dZejd$dZejd%dZejd&dZejd'dZejd(dZejd)d!Zeeedd*d+Zeeedd,d-Zeeedd.d/Zeeedd0d1Zeeedd2d3Z eedd4d5Z!eeedd6d7Z"eeedd8d9Z#ee edd:d;Z$ejdd/Zejd?d1Ze jd@d3Z e"jdAd7Z"e#jdBd9Z#e$jdCd;Z$eee e e e e ffddDdEZ%eee e e e e e e e fffddFdGZ&ee e ddHdIZ'ee e e e fddJdKZ(ee eddLdMZ)dNS)Sra A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens. Args: bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. ) bos_token eos_token unk_token sep_token pad_token cls_token mask_tokenadditional_special_tokensTcKsd|_d|_d|_d|_d|_d|_d|_d|_g|_||_ | D]\}}|durVqD||j vrD|dkrt |t tfsJd|dtdd|DsJdt|||qDt |ttfrt|||qDtd|d t|qDdS) NrrzValue z is not a list or tuplecss|]}t|ttfVqdSr=r]rCr6ritr(r(r) sz.SpecialTokensMixin.__init__..z2One of the tokens is not a string or an AddedTokenzspecial token z- has to be either str or AddedToken but got: ) _bos_token _eos_token _unk_token _sep_token _pad_token _cls_token _mask_token_pad_token_type_id_additional_special_tokensverboserySPECIAL_TOKENS_ATTRIBUTESr]r#rallsetattrrCr6rr )r@r kwargsrjr'r(r(r)r\ s2  zSpecialTokensMixin.__init__rccCs|j|jddS)aJ Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the vocabulary. Add the missing ones to the vocabulary if needed. Return: `int`: The number of tokens added in the vocabulary during the operation. Tspecial_tokens) add_tokensall_special_tokens_extendedr?r(r(r)sanitize_special_tokens(s z*SpecialTokensMixin.sanitize_special_tokens)special_tokens_dictrdc CsP|sdSd}|D]4\}}||jvs8Jd|d|jrLtd|||dkrt|ttfrvtdd|DsJd|d |d |rt |||nRt ||}t |}g}|D]&} t | |vrt | |vr| | q||||_||j|d d 7}qt|t tfs*Jd |d |dt |||||j|gd d 7}q|S)aD Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens are carefully handled by the tokenizer (they are never split). - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the one specified in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2Model.from_pretrained("gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```rzKey z is not a special tokenz+Assigning %s to the %s key of the tokenizerrcss|]}t|ttfVqdSr=rrr(r(r)rvsz8SpecialTokensMixin.add_special_tokens..zTokens z for key z* should all be str or AddedToken instancesTr%Token z* should be a str or an AddedToken instance)ryr!r rrr]r#rr"r#getattrrrCrextendrr'r6) r@r*Z!replace_additional_special_tokens added_tokensrjr'rZadditional_special_tokens_setto_addtokenr(r(r)add_special_tokens4s@7      z%SpecialTokensMixin.add_special_tokensF new_tokensr&rdcCs*|sdSt|ttfs|g}|j||dS)a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way. Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Args: new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`): Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc. special_tokens (`bool`, *optional*, defaults to `False`): Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance). See details for `tokenizers.AddedToken` in HuggingFace tokenizers library. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```rr%)r]r#r _add_tokensr@r3r&r(r(r)r's )zSpecialTokensMixin.add_tokenscCstdSr=NotImplementedErrorr5r(r(r)r4szSpecialTokensMixin._add_tokenscCs(|jdur|jrtddSt|jS)ze `str`: Beginning of sentence token. Log an error if used while not having been set. Nz'Using bos_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.bos_tokencCs(|jdur|jrtddSt|jS)z_ `str`: End of sentence token. Log an error if used while not having been set. Nz'Using eos_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.eos_tokencCs(|jdur|jrtddSt|jS)zW `str`: Unknown token. Log an error if used while not having been set. Nz'Using unk_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.unk_tokencCs(|jdur|jrtddSt|jS)z `str`: Separation token, to separate context and query in an input sequence. Log an error if used while not having been set. Nz'Using sep_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.sep_tokencCs(|jdur|jrtddSt|jS)zW `str`: Padding token. Log an error if used while not having been set. Nz'Using pad_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.pad_tokencCs(|jdur|jrtddSt|jS)z `str`: Classification token, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. Nz'Using cls_token, but it is not set yet.)rr rrrCr?r(r(r)rs   zSpecialTokensMixin.cls_tokencCs(|jdur|jrtddSt|jS)z `str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while not having been set. Nz(Using mask_token, but it is not set yet.)rr rrrCr?r(r(r)r s   zSpecialTokensMixin.mask_tokencCs.|jdur|jrtddSdd|jDS)z `List[str]`: All the additional special tokens you may want to use. Log an error if used while not having been set. Nz7Using additional_special_tokens, but it is not set yet.cSsg|] }t|qSr(rCritokr(r(r)r rmz@SpecialTokensMixin.additional_special_tokens..)rr rrr?r(r(r)rs   z,SpecialTokensMixin.additional_special_tokenscCs ||_dSr=)rr@r'r(r(r)r "scCs ||_dSr=)rr;r(r(r)r &scCs ||_dSr=)rr;r(r(r)r *scCs ||_dSr=)rr;r(r(r)r.scCs ||_dSr=)rr;r(r(r)r2scCs ||_dSr=)rr;r(r(r)r6scCs ||_dSr=)rr;r(r(r)r:scCs ||_dSr=rr;r(r(r)r>scCs|jdurdS||jS)z `Optional[int]`: Id of the beginning of sentence token in the vocabulary. Returns `None` if the token has not been set. N)rconvert_tokens_to_idsr r?r(r(r) bos_token_idBs zSpecialTokensMixin.bos_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set. N)rr=r r?r(r(r) eos_token_idLs zSpecialTokensMixin.eos_token_idcCs|jdurdS||jS)z{ `Optional[int]`: Id of the unknown token in the vocabulary. Returns `None` if the token has not been set. N)rr=r r?r(r(r) unk_token_idVs zSpecialTokensMixin.unk_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the separation token in the vocabulary, to separate context and query in an input sequence. Returns `None` if the token has not been set. N)rr=rr?r(r(r) sep_token_id_s zSpecialTokensMixin.sep_token_idcCs|jdurdS||jS)z{ `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set. N)rr=rr?r(r(r) pad_token_idis zSpecialTokensMixin.pad_token_idcCs|jS)zH `int`: Id of the padding token type in the vocabulary. )rr?r(r(r)pad_token_type_idrsz$SpecialTokensMixin.pad_token_type_idcCs|jdurdS||jS)z `Optional[int]`: Id of the classification token in the vocabulary, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Returns `None` if the token has not been set. N)rr=rr?r(r(r) cls_token_idys zSpecialTokensMixin.cls_token_idcCs|jdurdS||jS)z `Optional[int]`: Id of the mask token in the vocabulary, used when training a model with masked-language modeling. Returns `None` if the token has not been set. N)rr=rr?r(r(r) mask_token_ids z SpecialTokensMixin.mask_token_idcCs ||jS)z `List[int]`: Ids of all the additional special tokens in the vocabulary. Log an error if used while not having been set. )r=rr?r(r(r)additional_special_tokens_idssz0SpecialTokensMixin.additional_special_tokens_idscCs|dur||nd|_dSr=)convert_ids_to_tokensrr;r(r(r)r>scCs|dur||nd|_dSr=)rGrr;r(r(r)r?scCs|dur||nd|_dSr=)rGrr;r(r(r)r@scCs|dur||nd|_dSr=)rGrr;r(r(r)rAscCs|dur||nd|_dSr=)rGrr;r(r(r)rBscCs|dur||nd|_dSr=)rGrr;r(r(r)rDscCs|dur||nd|_dSr=)rGrr;r(r(r)rEscsfdd|D_dS)Ncsg|]}|qSr()rG)rir'r?r(r)rrmzDSpecialTokensMixin.additional_special_tokens_ids..r<)r@rxr(r?r)rFscCsVi}|jD]F}t|d|}|r t|ttfrDt|dd|Dnt|||<q |S)a  `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. _css|]}t|VqdSr=r8)riZattr_value_subr(r(r)rrmz8SpecialTokensMixin.special_tokens_map..)r!r,r]r#rr rCr@set_attrattr attr_valuer(r(r)special_tokens_maps  z%SpecialTokensMixin.special_tokens_mapcCs.i}|jD]}t|d|}|r |||<q |S)a `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. rH)r!r,rIr(r(r)special_tokens_map_extendeds   z.SpecialTokensMixin.special_tokens_map_extendedcCsdd|jD}|S)z `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Convert tokens of `tokenizers.AddedToken` type to string. cSsg|] }t|qSr(r8)risr(r(r)rrmz9SpecialTokensMixin.all_special_tokens..)r()r@all_toksr(r(r)all_special_tokenssz%SpecialTokensMixin.all_special_tokenscCsJg}|j}|D]$}|t|ttfr.t|n|g}qtt|}|S)a1 `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. )rNrxr]r#rr fromkeys)r@rPrJrLr(r(r)r(s  "z.SpecialTokensMixin.all_special_tokens_extendedcCs|j}||}|S)zy `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. )rQr=)r@rPall_idsr(r(r)all_special_idss z"SpecialTokensMixin.all_special_idsN)T)T)F)F)*r"r+r,r-r!r\rRr)rrCr r6r1rrEr'r4rr r r rrrrrsetterr r>r?r@rArBrCrDrErFrMrNrQr(rTr(r(r(r)rs   _ 1$                             $. rcseZdZUdZiZeeefed<iZeeeeeffed<iZ eeeee ffed<iZ eee e fed<dZe eed<gZeeed<gd Zeeed <d Zeed <d Zeed <dZdZdZfddZee dddZee dddZeje dddZeje dddZedddZedddZe dddZeee fdd d!Ze ed"d#d$Z!e d%d&Z"e dd'd(Z#e d)d*Z$e d+d,Z%de eeed/d0d1Z&dee ee'ed2d3d4Z(dee ee)eed6d7d8Z*e+e,d9d:de-e.e/e0fe e-e.e/e0fe)e-e)ee1fe-e)ee2fe e e e e-ee3fee d< d=d>Z4de)e d?d@dAZ5ddBdCZ6e+e,e7de-e.e/ee.ee/fe e-e.e/ee.ee/fe-e.e/ee.ee/fe e-e.e/ee.ee/fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dDdEdFZ9de-e.e/ee.ee/fe e-e.e/ee.ee/fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dGdHdIZ:e+e,e7de-e.e/e0fe e-e.e/e0fe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dGdJdKZ;dd;e1je+e,e7de-ee.ee?ee/ee@ee0eeAfe)e-e)ee1fe-e)ee2fe e e e)e e e e-ee3fe e)e e)e)e)e)e)e)e8dOdPdQZBd;e1jsz%BaseTokenizer.max_len_single_sentencecCs|j|jddS)zi `int`: The maximum combined length of a pair of sentences that can be fed to the model. Trqrsr?r(r(r)max_len_sentences_pairEsz$BaseTokenizer.max_len_sentences_paircCsL||j|jddkr@|jr@|jdds4tdd|jd<ntddS)NFrqruzXSetting 'max_len_single_sentence' is now deprecated. This value is automatically set up.Trertr rogetrwarningr!r;r(r(r)ruLs cCsL||j|jddkr@|jr@|jdds4tdd|jd<ntddS)NTrqrvFzWSetting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.rwr;r(r(r)rvZs )rdcCs ||_dS)z%Sets processor class as an attribute.N)rm)r@rdr(r(r)_set_processor_classfsz"BaseTokenizer._set_processor_classcCsN|jjd|jd|jd|jd|jd|jd|jd|jd|j d S) Nz(name_or_path='z', vocab_size=z, model_max_length=z , is_fast=z, padding_side='z', truncation_side='z', special_tokens=z, clean_up_tokenization_spaces=)) rbr"rc vocab_sizererfrarbrNrhr?r(r(r)__repr__js zBaseTokenizer.__repr__cCs tdSr=r6r?r(r(r)__len__sszBaseTokenizer.__len__cCs tdS)a Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `Dict[str, int]`: The vocabulary. Nr6r?r(r(r) get_vocabvs zBaseTokenizer.get_vocab)rccKs|dd}|dur|}tj|}tj|}|s\||jvr\t|d|jd|jd|rr|srt|dt}d}i}|t vr| |\}} | |\}}| }d} tj|rddt |D} t| d krtd || | r$tj|| d } td | t| }| |\}}||\} } d | vrV| d dd}| d }n td| |||| |s|j}td||tfd|i|}|}|rtd|S)ax Instantiates a tokenizer by the name_or_path. User can get the name using `get_support_list` of any tokenizer, it will download the necessary files from the cloud. or pass a directory where contains the vocabulary file and tokenizers yaml configuration file. Args: name_or_path (str): It supports the following two input types: If the name_or_path is a supported tokenizer name, for example, `clip_vit_b_32` and `t5_small`, it will download the necessary files from the cloud. User can select one from the support list by call `MindFormerBook.show_tokenizer_support_list()`. If name_or_path is a path to the local directory where there should have vocaburary files and configuration file ended with `yaml`. The vocaburary file needed by the tokenizer is determined by `.vocab_files_names`. pretrained_model_name_or_path (Optional[str]): Equal to "name_or_path", if "pretrained_model_name_or_path" is set, "name_or_path" is useless. Examples: >>> from mindformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> res = tokenizer.encode("hello world!") >>> print(res) [21820, 296, 55, 1] Returns: A instanced tokenizer. pretrained_model_name_or_pathNz+ does not exist, or it is not supported by z. please select from .z is not a directory.cSsg|]}|dr|qS).yaml)endswith)rifiler(r(r)rrmz1BaseTokenizer.from_pretrained..rzWThere should be only one yaml file under the directory %s, but followings are found: %srz;config in the yaml file %s are used for tokenizer building.rtokenizer_classzWCan't find the tokenizer_config.json in the file_dict. The content of file_dict is : %sz0build tokenizer class name is: %s using args %s. class_namezuSpecial tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.)rlospathexistsisdir _support_listr!r"rrget_tokenizer_url_support_list_download_using_name$_get_class_name_and_args_form_configlistdirrrryrrr*read_files_according_specific_by_tokenizerupdaterr))r&rcr$ris_existis_dirrZ loaded_kwargsconfig cache_pathZ yaml_list yaml_file vocab_dict file_dict tokenizerr.r(r(r)from_pretrainedsf            zBaseTokenizer.from_pretrainedc CsT|}|dr>|d|j}tjt|d|j}ntjt |d|j}tj |svtj |ddtj||d}dd}tj |s||}tj |rtj |rt ||td |ntd |t|}|D]N}|dd } tj|| } tj | s:td || t|| t| qt|} | |fS) z9Given the supported model name, download it from the urls mindspore/rHTexist_okrcSs0d}tD]}||vr||}q,q|Sr)rZget_trainer_support_task_listrxrx) model_namedefault_yaml_fileZ model_dictr(r(r)get_default_yaml_files  zABaseTokenizer._download_using_name..get_default_yaml_filez"default yaml config in %s is used.z0default yaml file path must be correct, but get rz)Download the vocab from the url %s to %s.)rr _model_namerrrrZ#get_xihe_checkpoint_download_folder _model_type&get_default_checkpoint_download_folderrmakedirsrealpathshutilrirrFileNotFoundErrorrrrr) r&rcZtokenizer_namerrrrtokenizer_need_filesurl_filelocal_file_name file_pathrr(r(r)rs:         z"BaseTokenizer._download_using_namec Cs|s>> from mindformers import T5Tokenizer, MindFormerBook >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> tokenizer.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'spiece.model'] NTrr)rrz2format should be one of [`yaml`, `json`], but got rrr rr)Loaderrrwrr)indentz3file_format should be one of [json, yaml], but got )rZ"get_default_checkpoint_save_folderrrr!rirjrkrbr"rrrrrrrreadrZget_tokenizer_name_to_processorr]ryr6r8dumpTOKENIZER_CONFIG_NAMErsave_vocabulary)r@rrrZdefault_directoryr$ merged_dictr file_readerZprocessor_nameZ token_keyZ token_valueZtokenizer_config_pathfpr(r(r)save_pretrainedBsD     $     .   2zBaseTokenizer.save_pretrained)rfilename_prefixrdcCstdS)aO Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. Nr6)r@rrr(r(r)rszBaseTokenizer.save_vocabularyFrrrr1rdcKstdS)a Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. Nr6)r@rrrr1r$r(r(r)tokenizeszBaseTokenizer.tokenizezI **kwargs: Passed along to the `.tokenize()` method. zp Returns: `List[int]`, `ms.Tensor`, or `np.ndarray`: The tokenized ids of the text. T) r text_pairr1padding truncationrstridereturn_tensorsrdc Ks*|j|f|||||||d| } | dS)aC Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `List[str]` or `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). )rr1rrrrrr\) encode_plus) r@rrr1rrrrrr$encoded_inputsr(r(r)encodes%  zBaseTokenizer.encoderrrdcCstdSr=r6)r@rrr(r(r)rtsz'BaseTokenizer.num_special_tokens_to_addc Ks|dd}|dd}|durZ|durZ|durZ|rV|jddsLtdd|jd<d }|dur|r|rvtd t|durtj } ntj } n|dur|dur|r|dur|dus|dus|dkrtd |durtd tj } n$t |tst|} nt |tr|} ntj } |durD|dkrD|r:td tt |} nT|dur|dur|durjt j} n&t |t st |} nt |t r|} nt j} |durB| tj kr|jtkr|r|jddstdd|jd<tj } n|j}| t jkrB|jtkr<|r4|jdds*tdd|jd<t j} n|j}| tj krj|jrb|jdkrjtd| t jkr| tj kr|dur|dur||dkrtd|d|d| | ||fS)z Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) and behaviors. truncation_strategyrIZpad_to_max_lengthFNz#Truncation-not-explicitly-activatedatTruncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.TrHaThe `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).z`max_length` is ignored when `padding`=`True` and there is no truncation strategy. To pad to max length, use `padding='max_length'`.zLThough `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.aThe `truncation_strategy` argument is deprecated and will be removed in a future version, use `truncation=True` to truncate examples to a max length. You can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input size of the model (e.g. 512 for Bert). If you have pairs of inputs, you can give a specific truncation strategy selected among `truncation='only_first'` (will only truncate the first sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence in the pairs).zAsking-to-pad-to-max_lengthzAsking to pad to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no padding.z Asking-to-truncate-to-max_lengthzAsking to truncate to max_length but no maximum length is provided and the model has no predefined maximum length. Default to no truncation.rzAsking to pad but the tokenizer does not have a padding token. Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`.zATruncation and padding are both activated but truncation length (z+) is not a multiple of pad_to_multiple_of (z).)rlrorxrryrrrrrrr]rrGrLrMre LARGE_INTEGERrrBr!) r@rrrpad_to_multiple_ofr r$Zold_truncation_strategyZold_pad_to_max_lengthpadding_strategyrr(r(r)"_get_padding_truncation_strategiess                       z0BaseTokenizer._get_padding_truncation_strategies)rr text_targettext_pair_targetr1rrrris_split_into_wordsrrreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthr rdcKs||||| | | | | ||||||d}|||durF|durFtd|durr|js\||jf||d|}|dur||jf||d|}||dur|S|dur|S|d|d<|S)a  Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). )r1rrrrrrrrrrrrrr Nz3You need to specify either `text` or `text_target`.)rrr\labels)rr!rp_switch_to_input_mode _call_one_switch_to_target_mode)r@rrrrr1rrrrrrrrrrrrrr r$Z all_kwargsrtZtarget_encodingsr(r(r)__call__h sB2  zBaseTokenizer.__call__)rrr1rrrrrrrrrrrrrr rdcKs6dd}||std|dur0||s0td|rZt|ttfoV|oVt|dttf}nt|ttf}|rt|trtd|durt|t|krtdt|dt|d |durtt||n|}|jf|||||||| | | | | ||||d |S|j f||||||||| | | | | ||||d |S) zused by self.__call__cSslt|trdSt|ttfrh|s$dSt|dtr6dSt|dttfrd|d pbt|ddtSdSdS)NTrF)r]rCr#r)rr(r(r)_is_valid_text_input s z5BaseTokenizer._call_one.._is_valid_text_inputztext input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized examples).Nrzdwhen tokenizing batches of text, `text_pair` must be a list or tuple with the same length as `text`.zbatch length of `text`: z- does not match batch length of `text_pair`: r)batch_text_or_text_pairsr1rrrrrrrrrrrrrr )rrr1rrrrrrrrrrrrrr ) r!r]r#rrCrrzipbatch_encode_plusr)r@rrr1rrrrrrrrrrrrrr r$r is_batchedrr(r(r)r s&  zBaseTokenizer._call_onecKsX|jf|||| |d|\}}}}|jf||||||||| | | | | ||||d|S)a Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `List[str]` or `List[int]` (the latter only for not-fast tokenizers)): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). rrrrr )rrr1rrrrrrrrrrrrrr )r _encode_plus)r@rrr1rrrrrrrrrrrrrr r$rrr(r(r)r. s<+ zBaseTokenizer.encode_plusrrr1rrrrrrrrrrrrrr rdcKstdS)zused by self.encode_plusNr6)r@rrr1rrrrrrrrrrrrrr r$r(r(r)rw szBaseTokenizer._encode_plus)rr1rrrrrrrrrrrrrr rdcKsV|jf|||||d|\}}}}|jf||||||||| | | | | |||d|S)a Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs ( `List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). r)rr1rrrrrrrrrrrrrr )r_batch_encode_plus)r@rr1rrrrrrrrrrrrrr r$rrr(r(r)r s:/ zBaseTokenizer.batch_encode_plusrr1rrrrrrrrrrrrrr rdcKstdS)zused by self.batch_encode_plusNr6)r@rr1rrrrrrrrrrrrrr r$r(r(r)r sz BaseTokenizer._batch_encode_plus)rrrrrrr rdcs|jjdr<|jdds|d ur8dn|}n If the `encoded_inputs` passed are dictionary of numpy arrays, or Mindspore tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ( [`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str, List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `List[int]` you can have tensors (numpy arrays or Mindspore tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. ZFastzAsking-to-pad-a-fast-tokenizerFzYou're using a z tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.Trcs i|]fddDqS)csg|] }|qSr(r()riexamplerjr(r)rP rmz0BaseTokenizer.pad...r()ri)rrr)rlP rmz%BaseTokenizer.pad..zRYou should supply an encoding or a list of encodings to this method that includes z, but you provided Nr^r1r0ztype of z unknown: z7. Should be one of a python, mindspore or numpy object.)rrr )rrrrrWc3s|]}t|kVqdSr=r)rir) batch_sizer(r)r sz$BaseTokenizer.pad..zLSome items in the output dictionary have a different batch size than others.css|]}t|VqdSr=r)riinputsr(r(r)r rmcsi|]\}}||qSr(r(r)ir(r)rl rm)#rbr"rrorxrZwarning_advicer]r#rrr%r_r!rrRr1rr0rr ryrr_padrTrr"rxrrmaxrranger)r@rrrrrrr required_input first_elementrgrjr'rrH batch_outputsroutputsr()rrrr)pad sK        zBaseTokenizer.pad) token_ids_0 token_ids_1rdcCs2|durt|dgSdgt|dgt|S)a Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. Nrrrr@rrr(r(r)$create_token_type_ids_from_sequences sz2BaseTokenizer.create_token_type_ids_from_sequencescCs|dur |S||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens and this method should be overridden in a subclass. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The model input with special tokens. Nr(rr(r(r) build_inputs_with_special_tokens sz.BaseTokenizer.build_inputs_with_special_tokens)idspair_idsr1rrrrrrrrrrrr rXrdcKs |jf|||||d|\}}}}t|du}t|}|rDt|nd}| rX|sXtd| rv|tjkrv|durvtd| durd|jv} | durd|jv} i}|||r|j|dnd}g}|tjkr|r||kr|j ||||||d \}}}| r ||d <|||d <|r,| ||}| ||}n8|r:||n|}dgt||r^dgt|ng}||d <| rz||d<| r|r| |||d <ndgt||d <| |d |||tjks| r|j|||j|| d}|rt|d |d<t|| |d}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. rNrzAsking to return token_type_ids while setting add_special_tokens to False results in an undefined behavior. Please set add_special_tokens to True or set return_token_type_ids to None.zNot possible to return overflowing tokens for pair of sequences with the `longest_first`. Please select another truncation strategy than `longest_first`, for instance `only_second` or `only_first`.r]r^rq)rnum_tokens_to_removerrrZnum_truncated_tokensr\special_tokens_mask)rrrrlengthrZ)rrErr!rGrLr_rtrMtruncate_sequencesrrget_special_tokens_mask&_eventual_warn_about_too_long_sequencerrrr'rT)r@rrr1rrrrrrrrrrrr rXr$rrrrZlen_idsZ len_pair_idsr total_lenrsequencer]rr(r(r)prepare_for_model s&      &zBaseTokenizer.prepare_for_modelrH)rrrrrrdc Cs|dkr||gfSt|ts$t|}g}|tjksH|tjkr|durt||krtt|||}|jdkr|d|}||d}n:|jdkr|| d}|d| }ntd|jdn:d|dt|d }|tjkr|d |d }t |n|tjkrt d tjj t |D]} |dusHt|t|kr|jdkrb|dd }n,|jdkr||dd}ntdt |jnF|jdkr|dd }n,|jdkr|dd}ntdt |jq(n|tjkr|durt||krvtt|||}|jdkr<|| d}|d| }n8|jdkrb|d|}||d}ntdt |jnt d|t|||||fS)a Truncates a sequence pair in-place following the strategy. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (`int`, *optional*, defaults to 0): Number of tokens to remove using the truncation strategy. truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): The strategy to follow for truncation. Can be: - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). stride (`int`, *optional*, defaults to 0): If set to a positive number, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. Returns: `Tuple[List[int], List[int], List[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair of sequences (or a batch of pairs) is provided. rNrgr`zinvalid truncation strategy: z, use 'left' or 'right'.zWe need to remove z; to truncate the input but the first sequence has a length z. z/Please select another truncation strategy than z0, for instance 'longest_first' or 'only_second'.zBe aware, overflowing tokens are not returned for the setting you have chosen, i.e. sequence pairs with the %s truncation strategy. So the returned list will always be empty even if some tokens have been removed.rrzinvalid truncation strategy:zWe need to remove %s to truncate the input but the second sequence has a length %s. Please select another truncation strategy than %s, for instance 'longest_first' or 'only_first'.)r]rGrJrLrminrbr!rrryr'rrCrK) r@rrrrrr window_len error_msgrHr(r(r)r T sz-                  z BaseTokenizer.truncate_sequences)rrrrrrdc Cs|durd|jv}||jd}|tjkr2t|}|dur^|dur^||dkr^||d|}|tjkort||k}|rd|vrdgt||d<|r|t|}|jdkr&|r|ddg||d<d|vr|d|jg||d<d|vr |ddg||d<||jg|||jd<n|jdkr|rNdg||d|d<d|vrp|jg||d|d<d|vrdg||d|d<|jg||||jd<ntd t |j|S) a? Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) Nr^rrr`r]rrgzInvalid padding strategy:) r_rrrrrarCrBr!rC) r@rrrrrrZneeds_to_be_padded differencer(r(r)r sB            zBaseTokenizer._padr{rdcCstdS)aT Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. Args: tokens (`List[str]`): The token to join in a string. Returns: `str`: The joined tokens. Nr6r@r{r(r(r)convert_tokens_to_string s z&BaseTokenizer.convert_tokens_to_stringz np.ndarrayz ms.Tensor) sequencesskip_special_tokensrhrdc sfdd|DS)a\ Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. cs,g|]$}|r$j|fdndqS))rrhr)decode)riseqrhr$r@rr(r)r= sz.BaseTokenizer.batch_decode..r()r@rrrhr$r(rr) batch_decode% szBaseTokenizer.batch_decode token_idsrrhrdcKsdt|}t|dtrHg}|D]&}|jf|||d|}||qn|jf|||d|}|S)a Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. r)rrrh)rr]r#_decoder)r@rrrhr$outputrgZnew_strsr(r(r)rG s(zBaseTokenizer.decodecKstdSr=r6)r@rrrhr$r(r(r)rv szBaseTokenizer._decoderralready_has_special_tokensrdcs0|r |dusJd|jfdd|D}|S) Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. Na You cannot use ``already_has_special_tokens=False`` with this tokenizer. Please use a slow (full python) tokenizer to activate this argument. Or set `return_special_tokens_mask=True` when calling the encoding method to get the special tokens mask in any tokenizer. csg|]}|vrdndqS)rrr()rir0rTr(r)r rmz9BaseTokenizer.get_special_tokens_mask..r#)r@rrr!rr(r#r)r  s z%BaseTokenizer.get_special_tokens_mask) out_stringrdcCsX|ddddddddd d d d d ddddddd}|S)a Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: out_string (`str`): The text to clean up. Returns: `str`: The cleaned-up string. z .rz ??z !!z ,,z ' 'z n'tzn'tz 'mz'mz 'sz'sz 'vez'vez 'rez're)replace)r$r(r(r)clean_up_tokenization s*   z#BaseTokenizer.clean_up_tokenization)rrr cCsJ|durFt||jkrF|rF|jdds %s). Running this sequence through the model will result in indexing errorsT)rrerorxrry)r@rrr r(r(r)r  s  z4BaseTokenizer._eventual_warn_about_too_long_sequencecCsdS)zs Private method to put the tokenizer in input mode (when it has different modes for input/outputs) Nr(r?r(r(r)r sz#BaseTokenizer._switch_to_input_modecCsdS)zt Private method to put the tokenizer in target mode (when it has different modes for input/outputs) Nr(r?r(r(r)r sz$BaseTokenizer._switch_to_target_modeccs0td|d|_dVd|_|dS)z Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. a`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your labels by using the argument `text_target` of the regular `__call__` method (either in the same call as your input texts if you use the same keyword arguments, or in a separate call.TNF)rrrrprr?r(r(r)as_target_tokenizer sz!BaseTokenizer.as_target_tokenizerr) src_texts tgt_textsrmax_target_lengthrrrrdc Ksd} t| t|dd|dd|dur6|j}||fd||||d|} |dur^| S|durj|}|,||fd||||d|} Wdn1s0Y| d| d <| S) aa Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`List[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return PyTorch `ms.Tensor` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. a `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. Zsrc_langNZtgt_langT)r1rrrr)r1rrrrr\r)rrrrlrer+) r@r,r-rr.rrrr$Zformatted_warning model_inputsrr(r(r)prepare_seq2seq_batch sFG     $ z#BaseTokenizer.prepare_seq2seq_batch)N)Nrr)N)NF)NTFNNrN)F)FNNNT)NNNNTFNNrFNNNNFFFFT)NTFNNrFNNNNFFFFT)NTFNNrFNNNNFFFFT)TFNNrFNNNNFFFFT)TNNNNT)N)N)NTFNNrNNNNFFFTF)NrrHr)FN)FN)FN)NF)NNNrNT)Yr"r+r,r-rVrrCrDrWrXrrYr rRrZr[rr_rarbZslow_tokenizer_classrrr\rrurvrUrzr}r~rr.rrrrrrr rrErrENCODE_KWARGS_DOCSTRINGr TextInputPreTokenizedInput EncodedInputrrGr/rrtr'ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRINGrTrrrrrMr TextInputPairPreTokenizedInputPairEncodedInputPairrrrrrrr rrrrrrr  staticmethodr*r rrrr+r0rr(r(rar)rs    &    R 3    @    )    Z   n   K     U '   .         z M % 2   "  rcsfeZdZdZgZfddZeedddZee dddZ e e e fdd d Z d d ZdCeee eefee dddZddZdDee dddZdEeee eee dddZddZee ee fee ee fdddZd d!Zd"d#Zddejejdd$d ddddd d d d dfeee e!feeee e!feeeee e eee eee e"feeeeeeeeee#d%d&d'Z$dejejdd$d ddddd d d d dfeeeee%ee ee&ee!ee'feeeee e eee eee e"feeeeeeeeee#d(d)d*Z(e)e*e+dejejdd$ddddd d d df eee&e,ee dffeeeee e ee ee eeeeeeeee#d+d,d-Z-e e,e e e e.ffd.d/d0Z/dFeeeeee d1fd2d3 Z0dGee ee feee ee fd4d5d6Z1e e d7d8d9Z2ee e dd:d;Z3dHee eee d<d=d>Z4e5d?d@Z6e5dAdBZ7Z8S)Ira Base class for all slow tokenizers. Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). c s6tjfi|i|_i|_g|_t|_d|_dSNF)r[r\added_tokens_encoderadded_tokens_decoderunique_no_split_tokensr tokens_trie_decode_use_source_tokenizer)r@r$rar(r)r\ts zTokenizer.__init__rccCsdSr:r(r?r(r(r)rfszTokenizer.is_fastcCstdS)zP `int`: Size of the base vocabulary (without the added tokens). Nr6r?r(r(r)r|szTokenizer.vocab_sizecCs|jS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. )r;r?r(r(r)get_added_vocabszTokenizer.get_added_vocabcCs|jt|jS)zD Size of the full vocabulary with the added tokens. )r|rr;r?r(r(r)r~szTokenizer.__len__Fr2cspdd|D}g}|D]}t|ts>td|dt|d|sZtdrZjrZ|}|jkr|jkr||vr| |j rt d|qfdd t |D}d d |D}j|j||r"t|d krtj|d nttjt|_n:t|d krBtj|d nttjt|_jt|S) a Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens (`List[str]`or `List[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```cSsg|] }t|qSr(r8r9r(r(r)rrmz)Tokenizer._add_tokens..r+z is not a string but a r do_lower_casezAdding %s to the vocabularycsi|]\}}|t|qSr(r)rirr:r?r(r)rlrmz)Tokenizer._add_tokens..cSsi|]\}}||qSr(r(rr(r(r)rlrmrr)r]rCrr hasattrrAlowerr r=rr rrrryr;rr<rrr=sortedrunion _create_trie)r@r3r&Z tokens_to_addr0Zadded_tok_encoderZadded_tok_decoderr(r?r)r4s:     zTokenizer._add_tokenscCsNt}|D]8}t|dr8|jr8||jvr8||q ||q ||_dS)NrA)rrBrArQrrCr>)r@r=trier0r(r(r)rFs  zTokenizer._create_triercCs g}g}t|||r|ndS)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. N)rr)r@rrrrr(r(r)rtsz#Tokenizer.num_special_tokens_to_addNTrcKsdd|jD}|j|fi|\}}|r6td|t|dr|jrdd|j|jD}dd|d d }t |d d |}t |j}|j |} t| D]\} } | |vr|| d } | dkr| | dnd } | t| dkr| | dnd }t| tr@| jr |r || | d<| jrj| rj| | | d<q|rV|| | d<| r| | | d<qg}| D]6} | sqt| |vr|| n||| qt|S)a" Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens. Args: text (`str`): The sequence to be encoded. **kwargs (additional keyword arguments): Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: `List[str]`: The list of tokens. cSs i|]}t|trt||qSr()r]r6rCrr(r(r)rlsz&Tokenizer.tokenize..z$Keyword arguments %s not recognized.rAcSsg|]}t|qSr()reescape)riZs_tokr(r(r)rsz&Tokenizer.tokenize..(|z)|z(.+?)cSs|dp|dS)Nrr)groupsrC)mr(r(r)rmz$Tokenizer.tokenize..Nrr)r(prepare_for_tokenizationrryrBrAr=rQrrHsubrr>rrrxrr]r6r;r:rr- _tokenize)r@rrrr1r$r(Zescaped_special_tokspatternZno_split_tokenr{rr0Z tok_extendedrgr`Ztokenized_textr(r(r)rsH         zTokenizer.tokenizecKstdS)a Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. Nr6r@rr$r(r(r)rQCszTokenizer._tokenizercCsB|dur dSt|tr ||Sg}|D]}|||q(|S)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. N)r]rC#_convert_token_to_id_with_added_vocr)r@r{rr0r(r(r)r=Ls   zTokenizer.convert_tokens_to_idscCs*|dur dS||jvr |j|S||Sr=)r;_convert_token_to_idr@r0r(r(r)rTbs   z-Tokenizer._convert_token_to_id_with_added_voccCstdSr=r6rVr(r(r)rUjszTokenizer._convert_token_to_idrrc sffdd}|rtd||}|dur4||nd}j||||j|j||| | d| | | |||dS)Ncst|tr&j|fi}|St|ttfr||r|t|dtr|rrttjfdd|D}|S|St|ttfr|rt|dtr|Srt d|dt d|ddS)Nrc3s$|]}j|fddiVqdSrTNrrr$r@r(r)rrmz@Tokenizer._encode_plus..get_input_ids..zInput z] is not valid. Should be a string or a list/tuple of strings when `is_split_into_words=True`.zW is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers. r]rCrr=r#r itertoolschainrRr!rr{rr$r@r(r) get_input_idss&        z-Tokenizer._encode_plus..get_input_idsa return_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast. More information on available tokenizers at https://github.com/huggingface/transformers/pull/2674T)rr1rrrrrrrXrrrrrr )r7rr')r@rrr1rrrrrrrrrrrrrr r$r_ first_ids second_idsr(r^r)rms2zTokenizer._encode_plusrc sfdd}|rtdg}|D]r}t|ttfsB|d}}n*rdt|dttfsd|d}}n|\}}||}|dur||nd}|||fq$j|||||||| | | | || |d}t|S)Ncst|tr&j|fi}|St|ttfr||r|t|dtr|rrttjfdd|D}|S|St|ttfr|rt|dtr|St ddS)Nrc3s$|]}j|fddiVqdSrWrXrrYr(r)rrmzFTokenizer._batch_encode_plus..get_input_ids..z\Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.rZr]r^r(r)r_s      z3Tokenizer._batch_encode_plus..get_input_idszreturn_offset_mapping is not available when using Python tokenizers. To use this feature, change your tokenizer to one deriving from transformers.PreTrainedTokenizerFast.r) r1rrrrrrrrrrrr )r7r]r#rr_batch_prepare_for_modelrT)r@rr1rrrrrrrrrrrrrr r$r_r\Zids_or_pair_idsrrr`rarr(r^r)rs@  zTokenizer._batch_encode_plus)batch_ids_pairsr1rrrrrrrrrrrr rdcCsi}|D]h\}}|j|||tjj|j||dd| | | | dd|d}|D]&\}}||vr`g||<|||qHq|j||j||| d}t||d}|S)a Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs NF)r1rrrrrrrrrrrrXr )rrrrr)rrrr'ryrrrT)r@rcr1rrrrrrrrrrrr rr`rarrjr'r(r(r)rb sB  z"Tokenizer._batch_prepare_for_modelrcKs||fS)a Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. kwargs: Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. r(rSr(r(r)rOLsz"Tokenizer.prepare_for_tokenizationr csD|r&|durtdtj||ddSdg|r6t|ndt|S)r"NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.T)rrr!r)r!r[r r)r@rrr!rar(r)r dsz!Tokenizer.get_special_tokens_mask)rrrdcCst|trB||jvr|j|S||jkr8td|d||Sg}|D]b}t|}|rf||jvrfqJ||jvr||j|qJ||jkrtd|d|||qJ|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). z The token id zb is out of the size of vocabulary, please check your tokenizer and corresponding vocabulary files.)r]rRr<r| IndexError_convert_id_to_tokenrTr)r@rrr{indexr(r(r)rGs&        zTokenizer.convert_ids_to_tokens)rfrdcCstdSr=r6)r@rfr(r(r)reszTokenizer._convert_id_to_tokencCs d|S)Nr)rrr(r(r)rsz"Tokenizer.convert_tokens_to_stringrc Ks|dd|_|dd}|j||d}g}g}|D]L} |rH| |jvrHq4| |jvrv|rj|||g}|| q4|| q4|r||||rd|} n d|} |dur|n|j}|r| | } | S| S)NZuse_source_tokenizerFspaces_between_special_tokens)rrr) rlr?rGrTr;rrrrhr*) r@rrrhr$rgZfiltered_tokensZ sub_textsZcurrent_sub_textr0r clean_textr(r(r)rs6       zTokenizer._decodecCstd|jt|jdS)zshow_support_list methodzsupport list of %s is:N)rrr"r rr&r(r(r)show_support_listszTokenizer.show_support_listcCs|jS)zget_support_list method)rrir(r(r)get_support_listszTokenizer.get_support_list)F)F)NT)NF)F)FN)9r"r+r,r-rr\rrErfrRr|rrCr@r~r rr6r4rFrtr2r rrQr=rTrUrrrGrMr3r4r/rTrr6r7r8rrr1r5r rbrrOr rGrerrr.rjrkrr(r(rar)rds:   $A   H (  U SA   & - r)`r-rirrHrrrrrr[ dataclassesrrenumrtypingrrrrr r r r collectionsr rcollections.abcrr contextlibrrnumpyr0rr1mindformers.toolsrmindformers.tools.registerrrrrZtools.download_toolsrZ tools.utilsrmindformer_bookr__all__ZSPECIAL_TOKEN_FILE_NAMErrRrnrrCr2r3r4r6r7r8ZSPECIAL_TOKENS_MAP_FILEZADDED_TOKENS_FILEZTOKENIZER_CONFIG_FILEZFULL_TOKENIZER_FILEcompileZ_re_tokenizer_filer r/r6rFrGrNrSrTrrr1r5ZINIT_TOKENIZER_DOCSTRINGrrrrrrrrr rrregister TOKENIZERrr(r(r(r)s (             H h56E   s pengcheng.mind-mindformer/mindformers/models/__pycache__/build_config.cpython-39.pyc000066400000000000000000000033461476650141400312150ustar00rootroot00000000000000a ?cfF @s2dZddlmZmZmZdeeeedddZdS) z build model config modules )MindFormerRegisterMindFormerModuleTypeMindFormerConfigNconfig)r default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurTt|j|_|jdurjt|j|_|jdurt|j|_tj|t j |dStj ||fi|S)afBuild model config For MindFormer. Instantiate the model config from MindFormerRegister's registry. Args: config (dict): The task model config's config. Default: None. default_args (dict): The default argument of model config API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'config'. class_name (str): The class name of model config API. Default: None. Return: The function instance of model config API. Examples: >>> from mindformers import build_model_config >>> model_config = {'type': 'ViTConfig'} >>> # 1) use config dict to build model >>> model_config_from_config = build_model_config(model_config) >>> # 2) use class name to build model >>> model_config_class_name = build_model_config(class_name='ViTConfig') N)r) isinstancedictrZ text_configbuild_model_configZ vision_configZ head_configrget_instance_from_cfgrCONFIG get_instance)rrrrkwargsrJ/home/ma-user/work/mindformers-baichuan/mindformers/models/build_config.pyr s      r )NNrN)__doc__mindformers.tools.registerrrrr strr rrrrspengcheng.mind-mindformer/mindformers/models/__pycache__/build_model.cpython-39.pyc000066400000000000000000000050251476650141400310440ustar00rootroot00000000000000a ?cf@sjdZddlmZmZmZddlmZdeeeeddd Z deeeedd d Z deeeedddZ dS)zBuild Model API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_model_configNmodels)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|durJi}t|j|d}|dur|dur|D]\}}|||qpi}|d|tj |j t j |dSdStj ||fi|S)aBuild model For MindFormer. Instantiate the model from MindFormerRegister's registry. Args: config (dict): The task model's config. Default: None. default_args (dict): The default argument of model API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'model'. class_name (str): The class name of model API. Default: None. Return: The function instance of model API. Examples: >>> from mindformers import build_model >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> model_from_config = build_model(config.model) Nr r) isinstancedictrr model_configitems __setattr__ setdefaultrget_instance_from_cfgarchrMODELS get_instance)rr r r kwargsrkeyvaluerI/home/ma-user/work/mindformers-baichuan/mindformers/models/build_model.py build_models&  rencoderc Ksz|dur|durdS|durf|}t|tss(  - pengcheng.mind-mindformer/mindformers/models/__pycache__/build_processor.cpython-39.pyc000066400000000000000000000032341476650141400317630ustar00rootroot00000000000000a ?cf @s>dZddlmZmZmZddlmZd eeeeddd ZdS) zBuild Processor API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_tokenizerN processor)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jdurTt|j|_|jdurrt|jfi||_tj|t j |dStj ||fi|S)a Build processor For MindFormer. Instantiate the processor from MindFormerRegister's registry. Args: config (dict): The task processor's config. Default: None. default_args (dict): The default argument of processor API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'processor'. class_name (str): The class name of processor API. Default: None. Return: The function instance of processor API. Examples: >>> from mindformers import build_processor >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> processor_from_config = build_processor(config.processor) N)r ) isinstancedictrimage_processorbuild_processor tokenizerrrget_instance_from_cfgr PROCESSOR get_instance)rr r r kwargsrM/home/ma-user/work/mindformers-baichuan/mindformers/models/build_processor.pyrs   r)NNrN) __doc__mindformers.tools.registerrrrrr strrrrrrs pengcheng.mind-mindformer/mindformers/models/__pycache__/build_tokenizer.cpython-39.pyc000066400000000000000000000046631476650141400317650ustar00rootroot00000000000000a ?cf@sFdZddlmZmZmZddlmZddZd eee e d d d Z dS) zBuild Tokenizer API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)MindFormerBookc Ksd|vr dS|d}tjd|d}|js,dS|}|dd}t}||vrz|rz|j|d\}}||||n>| d} | | vr| } n | d } |j | d} || dS) zNAnd the vocab file path to the config if there is not vocab file in the config vocab_fileNtype tokenizer) module_type class_namelib_path) name_or_pathcheckpoint_name_or_pathr) rget_clsvocab_files_namespoprget_tokenizer_url_support_listkeys*read_files_according_specific_by_tokenizerupdategetget_support_listcache_vocab_files) configkwargsr Z dynamic_classr pathZremote_tokenizer_support_listread_vocab_file_dictread_tokenizer_file_dictrZ support_namerrM/home/ma-user/work/mindformers-baichuan/mindformers/models/build_tokenizer.pycheck_and_add_vocab_file_paths(          r Nr )r default_argsr r cKst|dur|durdS|dur`t|tr>t|ts>tfi|}t|fi|tj|tj|dStj||fi|S)aBuild tokenizer For MindFormer. Instantiate the tokenizer from MindFormerRegister's registry. Args: config (dict): The task tokenizer's config. Default: None. default_args (dict): The default argument of tokenizer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'tokenizer'. class_name (str): The class name of tokenizer API. Default: None. Return: The function instance of tokenizer API. Examples: >>> from mindformers import build_tokenizer >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> tokenizer_from_config = build_tokenizer(config.processor.tokenizer) N)r!) isinstancedictrr rget_instance_from_cfgr TOKENIZER get_instance)rr!r r rrrrbuild_tokenizer/sr')NNr N) __doc__mindformers.tools.registerrrrmindformer_bookrr r#strr'rrrrs pengcheng.mind-mindformer/mindformers/models/__pycache__/utils.cpython-39.pyc000066400000000000000000000014671476650141400277330ustar00rootroot00000000000000a ?cf@s>dZddlmmZddlmZd edddZd d Z dS) zCheck Model Input Config.Nget_cell_reusefloat16ms_typecCs>t|tjr|S|dkrtjS|dkr,tjStd|dS)z*Convert the string type to MindSpore type.rfloat32zBSupported data type keywords include: [float16, float32], but get N) isinstancemstypeFloatrrKeyErrorrr C/home/ma-user/work/mindformers-baichuan/mindformers/models/utils.pyconvert_mstypes rcCstS)Nrr r r r cell_reusesr)r) __doc__Zmindspore.common.dtypecommondtyper version_controlrstrrrr r r rs  pengcheng.mind-mindformer/mindformers/models/base_config.py000066400000000000000000000243531476650141400245420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseConfig class, which is all model configs' base class """ import os import shutil import yaml from ..mindformer_book import MindFormerBook from ..mindformer_book import print_path_or_list from ..tools import logger from ..tools.register.config import MindFormerConfig from ..models.build_config import build_model_config class BaseConfig(dict): """ Base Config for all models' config Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_config import BaseConfig >>> class MyConfig(BaseConfig): ... _support_list = MindFormerBook.get_config_support_list()['my_model'] ... ... def __init__(self, ... data_size: int = 32, ... net_size: list = [1, 2, 3], ... loss_type: str = "MyLoss", ... checkpoint_name_or_path: str = '', ... **kwargs): ... self.data_size = data_size ... self.net_size = net_size ... loss_type = loss_type ... self.checkpoint_name_or_path = checkpoint_name_or_path ... super(MyConfig, self).__init__(**kwargs) ... >>> mynet = MyModel(MyConfig) >>> output = mynet(input) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super(BaseConfig, self).__init__() self.update(kwargs) def __getattr__(self, key): if key not in self: return None return self[key] def __setattr__(self, key, value): self[key] = value def __delattr__(self, key): del self[key] def to_dict(self): """ for yaml dump, transform from Config to a strict dict class """ return_dict = {} for key, val in self.items(): if isinstance(val, BaseConfig): val = val.to_dict() return_dict[key] = val return return_dict @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a config by yaml name or path. Args: yaml_name_or_path (str): A supported model name or a path to model config (.yaml), the supported model name could be selected from AutoConfig.show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A model config, which inherited from BaseConfig. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}.") if os.path.exists(yaml_name_or_path): if not yaml_name_or_path.endswith(".yaml"): raise ValueError(f"{yaml_name_or_path} should be a .yaml file for model" " config.") config_args = MindFormerConfig(yaml_name_or_path) logger.info("the content in %s is used for" " config building.", yaml_name_or_path) elif yaml_name_or_path not in cls._support_list: raise ValueError(f"{yaml_name_or_path} is not a supported" f" model type or a valid path to model config." f" supported model could be selected from {cls._support_list}.") else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) config = build_model_config(config_args.model.model_config) MindFormerBook.set_model_config_to_name(id(config), config_args.model.arch.type) return config def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) save_path = os.path.join(save_directory, save_name + ".yaml") parsed_config, removed_list = self._inverse_parse_config() wraped_config = self._wrap_config(parsed_config) for key, val in removed_list: self[key] = val self.remove_type() meraged_dict = {} if os.path.exists(save_path): with open(save_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(save_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("config saved successfully!") def remove_type(self): """remove type caused by save’""" if isinstance(self, BaseConfig): self.pop("type") for key, val in self.items(): if isinstance(val, BaseConfig): val.pop("type") self.update({key: val}) def inverse_parse_config(self): """inverse_parse_config""" val, _ = self._inverse_parse_config() return val def _inverse_parse_config(self): """ Inverse parse config method, which builds yaml file content for model config. Returns: A model config, which follows the yaml content. """ self.update({"type": self.__class__.__name__}) removed_list = [] for key, val in self.items(): if isinstance(val, BaseConfig): val = val.inverse_parse_config() elif not isinstance(val, (str, int, float, bool)): removed_list.append((key, val)) continue self.update({key: val}) for key, _ in removed_list: self.pop(key) return self, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.pop("model_name", None) if model_name is None: model_name = MindFormerBook.get_model_config_to_name().get(id(config), None) return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def show_support_list(cls): """show support list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get support list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/models/base_model.py000066400000000000000000000407301476650141400243720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseModel """ from typing import Optional import os import shutil import yaml import mindspore as ms from mindspore import nn from mindspore.train.serialization import load_checkpoint, load_param_into_net from ..generation import GeneratorMixin from ..mindformer_book import MindFormerBook, print_path_or_list from .build_config import build_model_config from .base_config import BaseConfig from ..tools.register import MindFormerConfig from ..tools.download_tools import download_with_progress_bar from ..tools.logger import logger from ..tools.utils import try_sync_file class BaseModel(nn.Cell, GeneratorMixin): """ The base model that contains the class method `from_pretained` and `save_pretrained`, any new model that should inherit the class. Note: GeneratorMixin provides the method `generate` that enable the generation for nlp models. Args: config(BaseConfig): The model configuration that inherits the `BaseConfig`. """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, config: BaseConfig, **kwargs): super(BaseModel, self).__init__(**kwargs) self.config = config self.default_checkpoint_download_path = None def load_checkpoint(self, config): """ load checkpoint for models. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path if checkpoint_name_or_path: if not isinstance(checkpoint_name_or_path, str): raise TypeError(f"checkpoint_name_or_path should be a str," f" but got {type(checkpoint_name_or_path)}") if os.path.exists(checkpoint_name_or_path): param = load_checkpoint(checkpoint_name_or_path) ckpt_file = checkpoint_name_or_path try: load_param_into_net(self, param) logger.info("weights in %s are loaded", ckpt_file) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) elif checkpoint_name_or_path not in self._support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {self._support_list}.") else: checkpoint_name = checkpoint_name_or_path if checkpoint_name_or_path.startswith('mindspore'): # Adaptation the name of checkpoint at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" checkpoint_name = checkpoint_name_or_path.split('/')[self._model_name] default_checkpoint_download_folder = os.path.join( MindFormerBook.get_xihe_checkpoint_download_folder(), checkpoint_name.split('_')[self._model_type]) else: # Default the name of checkpoint, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[self._model_type]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][self._model_type] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) self.default_checkpoint_download_path = ckpt_file logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) param = load_checkpoint(ckpt_file) try: load_param_into_net(self, param) logger.info("weights in %s are loaded", ckpt_file) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) else: logger.info("model built, but weights is unloaded, since the config has no" " checkpoint_name_or_path attribute or" " checkpoint_name_or_path is None.") def save_pretrained(self, save_directory: Optional[str] = None, save_name: str = "mindspore_model"): """ Save the model weight and configuration file. (only supports standalone mode, and distribute mode waits for developing) Args: save_directory(str): a directory to save the model weight and configuration. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. If set, the directory will be what is set. save_name(str): the name of saved files, including model weight and configuration file. Default mindspore_model. Examples: >>> import os >>> from mindformers import T5ForConditionalGeneration, MindFormerBook >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') >>> net.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'mindspore_model.ckpt'] """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) checkpoint_path = os.path.join(save_directory, save_name + '.ckpt') config_path = os.path.join(save_directory, save_name + '.yaml') ms.save_checkpoint(self, checkpoint_path) if self.config is None: # A model should have "config" attribute for model save. raise AttributeError("the model has no config attribute.") parsed_config, remove_list = self._inverse_parse_config(self.config) wraped_config = self._wrap_config(parsed_config) for key, val in remove_list: self.config[key] = val self.remove_type(self.config) meraged_dict = {} if os.path.exists(config_path): with open(config_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(config_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("model saved successfully!") def remove_type(self, config): """remove type caused by save’""" if isinstance(config, BaseConfig): config.pop("type") for key, val in config.items(): if isinstance(val, BaseConfig): val.pop("type") config.update({key: val}) def _inverse_parse_config(self, config): """ Inverse parse config method, which builds yaml file content for model config. Args: config (BaseConfig): a model config inherited from BaseConfig. Returns: A model config, which follows the yaml content. """ removed_list = [] if not isinstance(config, BaseConfig): return config, removed_list class_name = config.__class__.__name__ config.update({"type": class_name}) for key, val in config.items(): new_val, _ = self._inverse_parse_config(val) if not isinstance(new_val, (str, int, float, bool, BaseConfig)): removed_list.append((key, new_val)) continue config.update({key: new_val}) for key, _ in removed_list: config.pop(key) return config, removed_list def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (BaseConfig): a config processed by _inverse_parse_config function. Returns: A (config) dict for yaml.dump. """ model_name = self.__class__.__name__ return {"model": {"model_config": config.to_dict(), "arch": {"type": model_name}}} @classmethod def from_pretrained(cls, pretrained_model_name_or_dir: str, **kwargs): """ Instantiates a model by the pretrained_model_name_or_dir. It download the model weights if the user pass a model name, or load the weight from the given directory if given the path. (only support standalone mode, and distribute mode waits for developing!) Args: pretrained_model_name_or_dir (str): It supports the following two input types. If `pretrained_model_name_or_dir` is a supported model name, for example, `vit_base_p16` and `t5_small`, it will download the necessary files from the cloud. User can pass one from the support list by call `MindFormerBook.get_model_support_list()`. If `pretrained_model_name_or_dir` is a path to the local directory where there should have model weights ended with `.ckpt` and configuration file ended with `yaml`. pretrained_model_name_or_path (Optional[str]): Equal to "pretrained_model_name_or_dir", if "pretrained_model_name_or_path" is set, "pretrained_model_name_or_dir" is useless. Examples: >>> from mindformers import T5ForConditionalGeneration >>> net = T5ForConditionalGeneration.from_pretrained('t5_small') Returns: A model, which inherited from BaseModel. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: pretrained_model_name_or_dir = pretrained_model_name_or_path if not isinstance(pretrained_model_name_or_dir, str): raise TypeError(f"pretrained_model_name_or_dir should be a str," f" but got {type(pretrained_model_name_or_dir)}") is_exist = os.path.exists(pretrained_model_name_or_dir) is_dir = os.path.isdir(pretrained_model_name_or_dir) if not is_exist and (pretrained_model_name_or_dir not in cls._support_list): raise ValueError(f'{pretrained_model_name_or_dir} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist and not is_dir: raise ValueError(f"{pretrained_model_name_or_dir} is not a directory.") if is_dir: yaml_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".yaml")] ckpt_list = [file for file in os.listdir(pretrained_model_name_or_dir) if file.endswith(".ckpt")] if not yaml_list or not ckpt_list: raise FileNotFoundError(f"there is no yaml file for model config or ckpt file " f"for model weights in {pretrained_model_name_or_dir}.") yaml_file = os.path.join(pretrained_model_name_or_dir, yaml_list[cls._model_type]) ckpt_file = os.path.join(pretrained_model_name_or_dir, ckpt_list[cls._model_type]) logger.info("config in %s and weights in %s are used for " "model building.", yaml_file, ckpt_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update(**kwargs) config = build_model_config(config_args.model.model_config) config.update({"checkpoint_name_or_path": ckpt_file}) model = cls(config) else: pretrained_model_name = pretrained_model_name_or_dir if pretrained_model_name_or_dir.startswith('mindspore'): # Adaptation the name of pretrained model at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" pretrained_model_name = pretrained_model_name.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), pretrained_model_name.split('_')[cls._model_type]) else: # Default the name of pretrained model, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), pretrained_model_name.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, pretrained_model_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(pretrained_model_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') try_sync_file(yaml_file) config_args = MindFormerConfig(yaml_file) config_args.model.model_config.update(**kwargs) config = build_model_config(config_args.model.model_config) config.update({"checkpoint_name_or_path": pretrained_model_name_or_dir}) model = cls(config) logger.info("model built successfully!") return model @classmethod def show_support_list(cls): """show_support_list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/models/base_processor.py000066400000000000000000000321171476650141400253110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BaseProcessor """ import os import shutil import yaml from ..mindformer_book import print_path_or_list, MindFormerBook from .build_processor import build_processor from .base_tokenizer import BaseTokenizer from ..tools import logger from ..tools.register import MindFormerConfig class BaseImageProcessor: """ BaseImageProcessor for all image preprocess. Examples: >>> from mindspore.dataset.vision.transforms import CenterCrop >>> from mindformers.models.base_processor import BaseImageProcessor >>> image_resolution = 224 >>> class MyImageProcessor(BaseImageProcessor): ... def __init__(self, image_resolution): ... super(MyImageProcessor, self).__init__(image_resolution=image_resolution) ... self.center_crop = CenterCrop(image_resolution) ... ... def preprocess(self, images, **kwargs): ... res = [] ... for image in images: ... image = self.center_crop(image) ... res.append(image) ... return res ... >>> my_image_processor = MyImageProcessor(image_resolution) >>> output = my_image_processor(image) """ def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) def __call__(self, image_data, **kwargs): """forward process""" return self.preprocess(image_data, **kwargs) def preprocess(self, images, **kwargs): """preprocess method""" raise NotImplementedError("Each image processor must implement its own preprocess method") class BaseAudioProcessor: """ BaseAudioProcessor for all audio preprocess. Examples: >>> from mindspore.dataset.audio import AllpassBiquad >>> from mindformers.models.base_processor import BaseAudioProcessor >>> sample_rate = 44100 >>> central_freq = 200.0 >>> class MyAudioProcessor(BaseAudioProcessor): ... def __init__(self, audio_property): ... super(MyAudioProcessor, self).__init__(sample_rate=sample_rate, central_freq=central_freq ) ... self.all_pass_biquad = AllpassBiquad(44100, 200.0) ... ... def preprocess(self, audio_data, **kwargs): ... res = [] ... for audio in audio_data: ... audio = self.all_pass_biquad(audio) ... res.append(audio) ... return res ... >>> my_audio_processor = MyAudioProcessor(sample_rate, central_freq) >>> output = my_audio_processor(audio) """ def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) def __call__(self, audio_data, **kwargs): """forward process""" return self.preprocess(audio_data, **kwargs) def preprocess(self, audio_data, **kwargs): """preprocess method""" raise NotImplementedError("Each audio processor must implement its own preprocess method") class BaseProcessor: """ Base processor Examples: >>> from mindformers.mindformer_book import MindFormerBook >>> from mindformers.models.base_processor import BaseProcessor >>> class MyProcessor(BaseProcessor): ... _support_list = MindFormerBook.get_processor_support_list()['my_model'] ... ... def __init__(self, image_processor=None, audio_processor=None, tokenizer=None, return_tensors='ms'): ... super(MyProcessor, self).__init__( ... image_processor=image_processor, ... audio_processor=audio_processor, ... tokenizer=tokenizer, ... return_tensors=return_tensors) ... >>> myprocessor = MyProcessor(image_processor, audio_processor, tokenizer) >>> output = mynet(image, audio, text) """ _support_list = [] _model_type = 0 _model_name = 1 def __init__(self, **kwargs): self.config = {} self.config.update(kwargs) self.image_processor = kwargs.pop("image_processor", None) self.audio_processor = kwargs.pop("audio_processor", None) self.tokenizer = kwargs.pop("tokenizer", None) self.max_length = kwargs.pop("max_length", None) self.padding = kwargs.pop("padding", False) self.return_tensors = kwargs.pop("return_tensors", None) def __call__(self, image_input=None, text_input=None): """call function""" output = {} if image_input is not None and self.image_processor: if not isinstance(self.image_processor, BaseImageProcessor): raise TypeError(f"feature_extractor should inherit from the BaseImageProcessor," f" but got {type(self.image_processor)}.") image_output = self.image_processor(image_input) output['image'] = image_output if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output def save_pretrained(self, save_directory=None, save_name="mindspore_model"): """ Save_pretrained. Args: save_directory (str): a directory to save config yaml save_name (str): the name of save files. """ if save_directory is None: save_directory = MindFormerBook.get_default_checkpoint_save_folder() if not isinstance(save_directory, str) or not isinstance(save_name, str): raise TypeError(f"save_directory and save_name should be a str," f" but got {type(save_directory)} and {type(save_name)}.") if not os.path.exists(save_directory): os.makedirs(save_directory, exist_ok=True) parsed_config = self._inverse_parse_config(self.config) wraped_config = self._wrap_config(parsed_config) config_path = os.path.join(save_directory, save_name + '.yaml') meraged_dict = {} if os.path.exists(config_path): with open(config_path, 'r') as file_reader: meraged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) file_reader.close() meraged_dict.update(wraped_config) with open(config_path, 'w') as file_pointer: file_pointer.write(yaml.dump(meraged_dict)) file_pointer.close() logger.info("processor saved successfully!") def _inverse_parse_config(self, config): """ Inverse parse config method, which builds yaml file content for feature extractor config. Args: Config (dict): a dict, which contains input parameters of feature extractor. Returns: A dict, which follows the yaml content. """ parsed_config = {"type": self.__class__.__name__} for key, val in config.items(): if isinstance(val, BaseTokenizer): parsed_sub_config = {"type": val.__class__.__name__} parsed_sub_config.update(val.init_kwargs) parsed_config.update({key: parsed_sub_config}) elif isinstance(val, (BaseImageProcessor, BaseAudioProcessor)): parsed_sub_config = {"type": val.__class__.__name__} parsed_sub_config.update(val.config) parsed_config.update({key: parsed_sub_config}) else: parsed_config.update({key: val}) return parsed_config def _wrap_config(self, config): """ Wrap config function, which wraps a config to rebuild content of yaml file. Args: config (dict): a dict processed by _inverse_parse_config function. Returns: A dict for yaml.dump. """ return {"processor": config} @classmethod def from_pretrained(cls, yaml_name_or_path, **kwargs): """ From pretrain method, which instantiates a processor by yaml name or path. Args: yaml_name_or_path (str): A supported yaml name or a path to .yaml file, the supported model name could be selected from .show_support_list(). If yaml_name_or_path is model name, it supports model names beginning with mindspore or the model name itself, such as "mindspore/vit_base_p16" or "vit_base_p16". pretrained_model_name_or_path (Optional[str]): Equal to "yaml_name_or_path", if "pretrained_model_name_or_path" is set, "yaml_name_or_path" is useless. Returns: A processor which inherited from BaseProcessor. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: yaml_name_or_path = pretrained_model_name_or_path if not isinstance(yaml_name_or_path, str): raise TypeError(f"yaml_name_or_path should be a str," f" but got {type(yaml_name_or_path)}") is_exist = os.path.exists(yaml_name_or_path) if not is_exist and yaml_name_or_path not in cls._support_list: raise ValueError(f'{yaml_name_or_path} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist: logger.info("config in %s is used for processor" " building.", yaml_name_or_path) config_args = MindFormerConfig(yaml_name_or_path) else: yaml_name = yaml_name_or_path if yaml_name_or_path.startswith('mindspore'): # Adaptation the name of yaml at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/vit_base_p16" yaml_name = yaml_name_or_path.split('/')[cls._model_name] checkpoint_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), yaml_name.split('_')[cls._model_type]) else: # Default the name of yaml, # the relevant file will be downloaded from the Obs platform. # such as "vit_base_p16" checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), yaml_name_or_path.split('_')[cls._model_type]) if not os.path.exists(checkpoint_path): os.makedirs(checkpoint_path, exist_ok=True) yaml_file = os.path.join(checkpoint_path, yaml_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(yaml_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') config_args = MindFormerConfig(yaml_file) processor = build_processor(config_args.processor) logger.info("processor built successfully!") return processor @classmethod def show_support_list(cls): """show_support_list""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/models/base_tokenizer.py000066400000000000000000006074701476650141400253160ustar00rootroot00000000000000# coding=utf-8 # Copyright 2020 The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Tokenizer for the pretrained tokenizer""" import copy import os import re import json import shutil import warnings import unicodedata import bisect import itertools from dataclasses import dataclass, field from enum import Enum from typing import Any, Dict, List, NamedTuple, Optional, Sequence, Tuple, Union from collections import OrderedDict, UserDict from collections.abc import Mapping, Sized from contextlib import contextmanager import yaml import numpy as np import mindspore as ms from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_tokenizer import build_tokenizer from ..tools.download_tools import download_with_progress_bar from ..tools.utils import try_sync_file from ..mindformer_book import MindFormerBook __all__ = ['BaseTokenizer', 'Tokenizer', 'SpecialTokensMixin'] SPECIAL_TOKEN_FILE_NAME = 'special_tokens_map.json' TOKENIZER_CONFIG_NAME = 'tokenizer_config.json' VERY_LARGE_INTEGER = int(1e30) # This is used to set the max input length for a model with infinite size input LARGE_INTEGER = int(1e20) # This is used when we need something big but slightly smaller than VERY_LARGE_INTEGER # Define type aliases and NamedTuples TextInput = str PreTokenizedInput = List[str] EncodedInput = List[int] TextInputPair = Tuple[str, str] PreTokenizedInputPair = Tuple[List[str], List[str]] EncodedInputPair = Tuple[List[int], List[int]] # Slow tokenizers used to be saved in three separated files SPECIAL_TOKENS_MAP_FILE = "special_tokens_map.json" ADDED_TOKENS_FILE = "added_tokens.json" TOKENIZER_CONFIG_FILE = "tokenizer_config.json" # Fast tokenizers (provided by HuggingFace tokenizer's library) can be saved in a single file FULL_TOKENIZER_FILE = "tokenizer.json" _re_tokenizer_file = re.compile(r"tokenizer\.(.*)\.json") class ExplicitEnum(str, Enum): """ Enum with more explicit error message for missing values. """ @classmethod def _missing_(cls, value): raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class TensorType(ExplicitEnum): """ Possible values for the `return_tensors` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ NUMPY = "np" MINDSPORE = "ms" @dataclass(frozen=True, eq=True) class AddedToken: """ AddedToken represents a token to be added to a Tokenizer An AddedToken can have special options defining the way it should behave. """ content: str = field(default_factory=str) single_word: bool = False lstrip: bool = False rstrip: bool = False normalized: bool = True def __getstate__(self): return self.__dict__ def __str__(self): return self.content @dataclass class EncodingFast: """This is dummy class because without the `tokenizers` library we don't have these objects anyway""" class TruncationStrategy(ExplicitEnum): """ Possible values for the `truncation` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ ONLY_FIRST = "only_first" ONLY_SECOND = "only_second" LONGEST_FIRST = "longest_first" DO_NOT_TRUNCATE = "do_not_truncate" class CharSpan(NamedTuple): """ Character span in the original string. Args: start (`int`): Index of the first character in the original string. end (`int`): Index of the character following the last character in the original string. """ start: int end: int class TokenSpan(NamedTuple): """ Token span in an encoded string (list of tokens). Args: start (`int`): Index of the first token in the span. end (`int`): Index of the token following the last token in the span. """ start: int end: int class BatchEncoding(UserDict): """ Holds the output of the [`~tokenization_utils_base.PreTrainedTokenizerBase.__call__`], [`~tokenization_utils_base.PreTrainedTokenizerBase.encode_plus`] and [`~tokenization_utils_base.PreTrainedTokenizerBase.batch_encode_plus`] methods (tokens, attention_masks, etc). This class is derived from a python dictionary and can be used as a dictionary. In addition, this class exposes utility methods to map from word/character space to token space. Args: data (`dict`): Dictionary of lists/arrays/tensors returned by the `__call__`/`encode_plus`/`batch_encode_plus` methods ('input_ids', 'attention_mask', etc.). encoding (`tokenizers.Encoding` or `Sequence[tokenizers.Encoding]`, *optional*): If the tokenizer is a fast tokenizer which outputs additional information like mapping from word/character space to token space the `tokenizers.Encoding` instance or list of instance (for batches) hold this information. tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. prepend_batch_axis (`bool`, *optional*, defaults to `False`): Whether or not to add a batch axis when converting to tensors (see `tensor_type` above). n_sequences (`Optional[int]`, *optional*): You can give a tensor_type here to convert the lists of integers in Mindspore/Numpy Tensors at initialization. """ def __init__( self, data: Optional[Dict[str, Any]] = None, encoding: Optional[Union[EncodingFast, Sequence[EncodingFast]]] = None, tensor_type: Union[None, str, TensorType] = None, prepend_batch_axis: bool = False, n_sequences: Optional[int] = None, ): super().__init__(data) if isinstance(encoding, EncodingFast): encoding = [encoding] self._encodings = encoding if n_sequences is None and encoding is not None and encoding: n_sequences = encoding[0].n_sequences self._n_sequences = n_sequences self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis) @property def n_sequences(self) -> Optional[int]: """ `Optional[int]`: The number of sequences used to generate each sample from the batch encoded in this [`BatchEncoding`]. Currently can be one of `None` (unknown), `1` (a single sentence) or `2` (a pair of sentences) """ return self._n_sequences @property def is_fast(self) -> bool: """ `bool`: Indicate whether this [`BatchEncoding`] was generated from the result of a [`PreTrainedTokenizerFast`] or not. """ return self._encodings is not None def __getitem__(self, item: Union[int, str]) -> Union[Any, EncodingFast]: """ If the key is a string, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.). If the key is an integer, get the `tokenizers.Encoding` for batch item with index `key`. If the key is a slice, returns the value of the dict associated to `key` ('input_ids', 'attention_mask', etc.) with the constraint of slice. """ if isinstance(item, str): return self.data[item] if self._encodings is not None: return self._encodings[item] if isinstance(item, slice): return {key: self.data[key][item] for key in self.data.keys()} raise KeyError( "Invalid key. Only three types of key are available: " "(1) string, (2) integers for backend Encoding, and (3) slices for data subsetting." ) def __getattr__(self, item: str): try: return self.data[item] except KeyError: raise AttributeError def __getstate__(self): return {"data": self.data, "encodings": self._encodings} def __setstate__(self, state): if "data" in state: self.data = state["data"] if "encodings" in state: self._encodings = state["encodings"] def keys(self): return self.data.keys() def values(self): return self.data.values() def items(self): return self.data.items() # After this point: # Extended properties and methods only available for fast (Rust-based) tokenizers # provided by HuggingFace tokenizers library. @property def encodings(self) -> Optional[List[EncodingFast]]: """ `Optional[List[tokenizers.Encoding]]`: The list all encodings from the tokenization process. Returns `None` if the input was tokenized through Python (i.e., not a fast) tokenizer. """ return self._encodings def tokens(self, batch_index: int = 0) -> List[str]: """ Return the list of tokens (sub-parts of the input strings after word/subword splitting and before conversion to integer indices) at a given batch index (only works for the output of a fast tokenizer). Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[str]`: The list of tokens at that index. """ if not self._encodings: raise ValueError( "tokens() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].tokens def sequence_ids(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to the id of their original sentences: - `None` for special tokens added around or between sequences, - `0` for tokens corresponding to words in the first sequence, - `1` for tokens corresponding to words in the second sequence when a pair of sequences was jointly encoded. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the sequence id corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding sequence. """ if not self._encodings: raise ValueError( "sequence_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].sequence_ids def words(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). """ if not self._encodings: raise ValueError( "words() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) warnings.warn( "`BatchEncoding.words()` property is deprecated and should be replaced with the identical, " "but more self-explanatory `BatchEncoding.word_ids()` property.", FutureWarning, ) return self.word_ids(batch_index) def word_ids(self, batch_index: int = 0) -> List[Optional[int]]: """ Return a list mapping the tokens to their actual word in the initial sentence for a fast tokenizer. Args: batch_index (`int`, *optional*, defaults to 0): The index to access in the batch. Returns: `List[Optional[int]]`: A list indicating the word corresponding to each token. Special tokens added by the tokenizer are mapped to `None` and other tokens are mapped to the index of their corresponding word (several tokens will be mapped to the same word index if they are parts of that word). """ if not self._encodings: raise ValueError( "word_ids() is not available when using non-fast tokenizers (e.g. instance of a `XxxTokenizerFast`" " class)." ) return self._encodings[batch_index].word_ids def token_to_sequence(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int: """ Get the index of the sequence represented by the given token. In the general use case, this method returns `0` for a single sequence or the first sequence of a pair, and `1` for the second sequence of a pair Can be called as: - `self.token_to_sequence(token_index)` if batch size is 1 - `self.token_to_sequence(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. """ if not self._encodings: raise ValueError("token_to_sequence() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index if batch_index < 0: batch_index = self._batch_size + batch_index if token_index < 0: token_index = self._seq_len + token_index return self._encodings[batch_index].token_to_sequence(token_index) def token_to_word(self, batch_or_token_index: int, token_index: Optional[int] = None) -> int: """ Get the index of the word corresponding (i.e. comprising) to an encoded token in a sequence of the batch. Can be called as: - `self.token_to_word(token_index)` if batch size is 1 - `self.token_to_word(batch_index, token_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e., words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token in the sequence. Returns: `int`: Index of the word in the input sequence. """ if not self._encodings: raise ValueError("token_to_word() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index if batch_index < 0: batch_index = self._batch_size + batch_index if token_index < 0: token_index = self._seq_len + token_index return self._encodings[batch_index].token_to_word(token_index) def word_to_tokens( self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0 ) -> Optional[TokenSpan]: """ Get the encoded token span corresponding to a word in a sequence of the batch. Token spans are returned as a [`~tokenization_utils_base.TokenSpan`] with: - **start** -- Index of the first token. - **end** -- Index of the token following the last token. Can be called as: - `self.word_to_tokens(word_index, sequence_index: int = 0)` if batch size is 1 - `self.word_to_tokens(batch_index, word_index, sequence_index: int = 0)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprises one sequence, this can be the index of the word in the sequence. word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: ([`~tokenization_utils_base.TokenSpan`], *optional*): Span of tokens in the encoded sequence. Returns `None` if no tokens correspond to the word. This can happen especially when the token is a special token that has been used to format the tokenization. For example when we add a class token at the very beginning of the tokenization. """ if not self._encodings: raise ValueError("word_to_tokens() is not available when using Python based tokenizers") if word_index is not None: batch_index = batch_or_word_index else: batch_index = 0 word_index = batch_or_word_index if batch_index < 0: batch_index = self._batch_size + batch_index if word_index < 0: word_index = self._seq_len + word_index span = self._encodings[batch_index].word_to_tokens(word_index, sequence_index) return TokenSpan(*span) if span is not None else None def token_to_chars(self, batch_or_token_index: int, token_index: Optional[int] = None) -> CharSpan: """ Get the character span corresponding to an encoded token in a sequence of the batch. Character spans are returned as a [`~tokenization_utils_base.CharSpan`] with: - **start** -- Index of the first character in the original string associated to the token. - **end** -- Index of the character following the last character in the original string associated to the token. Can be called as: - `self.token_to_chars(token_index)` if batch size is 1 - `self.token_to_chars(batch_index, token_index)` if batch size is greater or equal to 1 Args: batch_or_token_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the token in the sequence. token_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the token or tokens in the sequence. Returns: [`~tokenization_utils_base.CharSpan`]: Span of characters in the original string, or None, if the token (e.g. , ) doesn't correspond to any chars in the origin string. """ if not self._encodings: raise ValueError("token_to_chars() is not available when using Python based tokenizers") if token_index is not None: batch_index = batch_or_token_index else: batch_index = 0 token_index = batch_or_token_index span_indices = self._encodings[batch_index].token_to_chars(token_index) return CharSpan(*span_indices) if span_indices is not None else None def char_to_token( self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0 ) -> int: """ Get the index of the token in the encoded output comprising a character in the original string for a sequence of the batch. Can be called as: - `self.char_to_token(char_index)` if batch size is 1 - `self.char_to_token(batch_index, char_index)` if batch size is greater or equal to 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int`: Index of the token. """ if not self._encodings: raise ValueError("char_to_token() is not available when using Python based tokenizers") if char_index is not None: batch_index = batch_or_char_index else: batch_index = 0 char_index = batch_or_char_index return self._encodings[batch_index].char_to_token(char_index, sequence_index) def word_to_chars( self, batch_or_word_index: int, word_index: Optional[int] = None, sequence_index: int = 0 ) -> CharSpan: """ Get the character span in the original string corresponding to given word in a sequence of the batch. Character spans are returned as a CharSpan NamedTuple with: - start: index of the first character in the original string - end: index of the character following the last character in the original string Can be called as: - `self.word_to_chars(word_index)` if batch size is 1 - `self.word_to_chars(batch_index, word_index)` if batch size is greater or equal to 1 Args: batch_or_word_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the word in the sequence word_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the word in the sequence. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided word index belongs to. Returns: `CharSpan` or `List[CharSpan]`: Span(s) of the associated character or characters in the string. CharSpan are NamedTuple with: - start: index of the first character associated to the token in the original string - end: index of the character following the last character associated to the token in the original string """ if not self._encodings: raise ValueError("word_to_chars() is not available when using Python based tokenizers") if word_index is not None: batch_index = batch_or_word_index else: batch_index = 0 word_index = batch_or_word_index return CharSpan(*(self._encodings[batch_index].word_to_chars(word_index, sequence_index))) def char_to_word(self, batch_or_char_index: int, char_index: Optional[int] = None, sequence_index: int = 0) -> int: """ Get the word in the original string corresponding to a character in the original string of a sequence of the batch. Can be called as: - `self.char_to_word(char_index)` if batch size is 1 - `self.char_to_word(batch_index, char_index)` if batch size is greater than 1 This method is particularly suited when the input sequences are provided as pre-tokenized sequences (i.e. words are defined by the user). In this case it allows to easily associate encoded tokens with provided tokenized words. Args: batch_or_char_index (`int`): Index of the sequence in the batch. If the batch only comprise one sequence, this can be the index of the character in the original string. char_index (`int`, *optional*): If a batch index is provided in *batch_or_token_index*, this can be the index of the character in the original string. sequence_index (`int`, *optional*, defaults to 0): If pair of sequences are encoded in the batch this can be used to specify which sequence in the pair (0 or 1) the provided character index belongs to. Returns: `int` or `List[int]`: Index or indices of the associated encoded token(s). """ if not self._encodings: raise ValueError("char_to_word() is not available when using Python based tokenizers") if char_index is not None: batch_index = batch_or_char_index else: batch_index = 0 char_index = batch_or_char_index return self._encodings[batch_index].char_to_word(char_index, sequence_index) def convert_to_tensors( self, tensor_type: Optional[Union[str, TensorType]] = None, prepend_batch_axis: bool = False ): """ Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. prepend_batch_axis (`int`, *optional*, defaults to `False`): Whether or not to add the batch dimension during the conversion. """ if tensor_type is None: return self # Convert to TensorType if not isinstance(tensor_type, TensorType): tensor_type = TensorType(tensor_type) # Get a function reference for the correct framework if tensor_type == TensorType.MINDSPORE: tensor_dtype = ms.int32 as_tensor = ms.Tensor def is_ms_tensor(x): return isinstance(x, ms.Tensor) is_tensor = is_ms_tensor else: def as_tensor(value, dtype=None): if isinstance(value, (list, tuple)) and isinstance(value[0], (list, tuple, np.ndarray)): value_lens = [len(val) for val in value] if len(set(value_lens)) > 1 and dtype is None: # we have a ragged list so handle explicitly value = as_tensor([np.asarray(val) for val in value], dtype=object) return np.asarray(value, dtype=dtype) def is_numpy_array(x): return isinstance(x, np.ndarray) tensor_dtype = np.int32 is_tensor = is_numpy_array # Do the tensor conversion in batch for key, value in self.items(): try: if prepend_batch_axis: # value = [value] pass if not is_tensor(value): tensor = as_tensor(value, dtype=tensor_dtype) # Removing this for now in favor of controlling the shape with `prepend_batch_axis` # # at-least2d # if tensor.ndim > 2: # tensor = tensor.squeeze(0) # elif tensor.ndim < 2: # tensor = tensor[None, :] self[key] = tensor except Exception as e: if key == "overflowing_tokens": raise ValueError( "Unable to create tensor returning overflowing tokens of different lengths. " "Please see if a fast version of this tokenizer is available to have this feature available." ) from e raise ValueError( "Unable to create tensor, you should probably activate truncation and/or padding with" " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your" f" features (`{key}` in this case) have excessive nesting (inputs type `list` where type `int` is" " expected)." ) from e return self class PaddingStrategy(ExplicitEnum): """ Possible values for the `padding` argument in [`PreTrainedTokenizerBase.__call__`]. Useful for tab-completion in an IDE. """ LONGEST = "longest" MAX_LENGTH = "max_length" DO_NOT_PAD = "do_not_pad" class Trie: """ Trie in Python. Creates a Trie out of a list of words. The trie is used to split on `added_tokens` in one pass Loose reference https://en.wikipedia.org/wiki/Trie """ def __init__(self): self.data = {} def add(self, word: str): """ Passes over every char (utf-8 char) on word and recursively adds it to the internal `data` trie representation. The special key `""` is used to represent termination. This function is idempotent, adding twice the same word will leave the trie unchanged Example: ```python >>> trie = Trie() >>> trie.add("Hello 友達") >>> trie.data {"H": {"e": {"l": {"l": {"o": {" ": {"友": {"達": {"": 1}}}}}}}}} >>> trie.add("Hello") >>> trie.data {"H": {"e": {"l": {"l": {"o": {"": 1, " ": {"友": {"達": {"": 1}}}}}}}}} ``` """ if not word: # Prevent empty string return ref = self.data for char in word: ref[char] = ref[char] if char in ref else {} ref = ref[char] ref[""] = 1 def split(self, text: str) -> List[str]: """ Will look for the words added to the trie within `text`. Output is the original string split along the boundaries of the words found. This trie will match the longest possible word first ! Example: ```python >>> trie = Trie() >>> trie.split("[CLS] This is a extra_id_100") ["[CLS] This is a extra_id_100"] >>> trie.add("[CLS]") >>> trie.add("extra_id_1") >>> trie.add("extra_id_100") >>> trie.split("[CLS] This is a extra_id_100") ["[CLS]", " This is a ", "extra_id_100"] ``` """ # indexes are counted left of the chars index. # "hello", index 0, is left of h, index 1 is between h and e. # index 5 is right of the "o". # States are going to capture every possible start (indexes as above) # as keys, and have as values, a pointer to the position in the trie # where we're at. This is a partial match for now. # This enables to keep track of multiple matches while we're iterating # the string # If the trie contains, "blowing", and "lower" and we encounter the # string "blower", we need to split into ["b", "lower"]. # This is where we need to keep track of multiple possible starts. states = OrderedDict() # This will contain every indices where we need # to cut. # We force to cut at offset 0 and len(text) (added later) offsets = [0] # This is used by the lookahead which needs to skip over # some text where the full match exceeded the place in the initial # for loop skip = 0 # Main loop, Giving this algorithm O(n) complexity for current, current_char in enumerate(text): if skip and current < skip: # Prevents the lookahead for matching twice # like extra_id_100 and id_100 continue # This will track every state # that stop matching, we need to stop tracking them. # If we look at "lowball", we're going to match "l" (add it to states), "o", "w", then # fail on "b", we need to remove 0 from the valid states. to_remove = set() # Whenever we found a match, we need to drop everything # this is a greedy algorithm, it will match on the first found token reset = False # In this case, we already have partial matches (But unfinished) for start, trie_pointer in states.items(): if "" in trie_pointer: # This is a final match, we need to reset and # store the results in `offsets`. # Lookahead to match longest first # Important in case of extra_id_1 vs extra_id_100 # Here we are also actively looking for other earlier partial # matches # "[CLS]", "L", we need to match CLS even if L is special states, start, end, skip = \ self.split_atom_3(states=states, current=current, text=text, start=start, skip=skip) offsets.append(start) offsets.append(end) reset = True break elif current_char in trie_pointer: # The current character being looked at has a match within the trie # update the pointer (it will be stored back into states later). trie_pointer = trie_pointer[current_char] # Storing back the new pointer into the states. # Partial matches got longer by one. states[start] = trie_pointer else: # The new character has not match in the trie, we need # to stop keeping track of this partial match. # We can't do it directly within the loop because of how # python iteration works to_remove.add(start) # Either clearing the full start (we found a real match) # Or clearing only the partial matches that didn't work. states = self.split_atom_2(reset=reset, to_remove=to_remove, states=states) # If this character is a starting character within the trie # start keeping track of this partial match. if current >= skip and current_char in self.data: states[current] = self.data[current_char] # We have a cut at the end with states. offsets = self.split_atom_1(states=states, text=text, offsets=offsets) return self.cut_text(text, offsets) def split_atom_1(self, states=None, text=None, offsets=None): """atomic act of split""" for start, trie_pointer in states.items(): if "" in trie_pointer: # This is a final match, we need to reset and # store the results in `offsets`. end = len(text) offsets.append(start) offsets.append(end) # Longest cut is always the one with lower start so the first # item so we need to break. break return offsets def split_atom_2(self, reset=None, to_remove=None, states=None): """atomic act of split""" if reset: states = {} else: for start in to_remove: del states[start] return states def split_atom_3(self, states=None, current=None, text=None, start=None, skip=None): """atomic act of split""" for lookstart, looktrie_pointer in states.items(): if lookstart > start: # This partial match is later, we can stop looking break elif lookstart < start: # This partial match is earlier, the trie pointer # was already updated, so index is + 1 lookahead_index = current + 1 end = current + 1 else: # Here lookstart == start and # looktrie_pointer == trie_pointer # It wasn't updated yet so indices are current ones lookahead_index = current end = current next_char = text[lookahead_index] if lookahead_index < len(text) else None if "" in looktrie_pointer: start = lookstart end = lookahead_index skip = lookahead_index while next_char in looktrie_pointer: looktrie_pointer = looktrie_pointer[next_char] lookahead_index += 1 if "" in looktrie_pointer: start = lookstart end = lookahead_index skip = lookahead_index if lookahead_index == len(text): # End of string break next_char = text[lookahead_index] return states, start, end, skip def cut_text(self, text, offsets): """cut thr text.""" # We have all the offsets now, we just need to do the actual splitting. # We need to eventually add the first part of the string and the eventual # last part. offsets.append(len(text)) tokens = [] start = 0 for end in offsets: if start > end: logger.error( "There was a bug in Trie algorithm in tokenization. Attempting to recover. Please report it" " anyway." ) continue elif start == end: # This might happen if there's a match at index 0 # we're also preventing zero-width cuts in case of two # consecutive matches continue tokens.append(text[start:end]) start = end return tokens ENCODE_KWARGS_DOCSTRING = r""" add_special_tokens (`bool`, *optional*, defaults to `True`): Whether or not to encode the sequences with the special tokens relative to their model. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). max_length (`int`, *optional*): Controls the maximum length to use by one of the truncation/padding parameters. If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. stride (`int`, *optional*, defaults to 0): If set to a number along with `max_length`, the overflowing tokens returned when `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence returned to provide some overlap between truncated and overflowing sequences. The value of this argument defines the number of overlapping tokens. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. Requires `padding` to be activated. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. """ ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING = r""" return_token_type_ids (`bool`, *optional*): Whether to return token type IDs. If left to the default, will return the token type IDs according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are token type IDs?](../glossary#token-type-ids) return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_overflowing_tokens (`bool`, *optional*, defaults to `False`): Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead of returning overflowing tokens. return_special_tokens_mask (`bool`, *optional*, defaults to `False`): Whether or not to return special tokens mask information. return_offsets_mapping (`bool`, *optional*, defaults to `False`): Whether or not to return `(char_start, char_end)` for each token. This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using Python's tokenizer, this method will raise `NotImplementedError`. return_length (`bool`, *optional*, defaults to `False`): Whether or not to return the lengths of the encoded inputs. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. **kwargs: passed to the `self.tokenize()` method Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. [What are input IDs?](../glossary#input-ids) - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or if *"token_type_ids"* is in `self.model_input_names`). [What are token type IDs?](../glossary#token-type-ids) - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names`). [What are attention masks?](../glossary#attention-mask) - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and `return_overflowing_tokens=True`). - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and `return_overflowing_tokens=True`). - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`). - **length** -- The length of the inputs (when `return_length=True`) """ INIT_TOKENIZER_DOCSTRING = r""" Class attributes (overridden by derived classes) - **vocab_files_names** (`Dict[str, str]`) -- A dictionary with, as keys, the `__init__` keyword name of each vocabulary file required by the model, and as associated values, the filename for saving the associated file (string). - **pretrained_vocab_files_map** (`Dict[str, Dict[str, str]]`) -- A dictionary of dictionaries, with the high-level keys being the `__init__` keyword name of each vocabulary file required by the model, the low-level being the `short-cut-names` of the pretrained models with, as associated values, the `url` to the associated pretrained vocabulary file. - **max_model_input_sizes** (`Dict[str, Optional[int]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, the maximum length of the sequence inputs of this model, or `None` if the model has no maximum input size. - **pretrained_init_configuration** (`Dict[str, Dict[str, Any]]`) -- A dictionary with, as keys, the `short-cut-names` of the pretrained models, and as associated values, a dictionary of specific arguments to pass to the `__init__` method of the tokenizer class for this pretrained model when loading the tokenizer with the [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`] method. - **model_input_names** (`List[str]`) -- A list of inputs expected in the forward pass of the model. - **padding_side** (`str`) -- The default value for the side on which the model should have padding applied. Should be `'right'` or `'left'`. - **truncation_side** (`str`) -- The default value for the side on which the model should have truncation applied. Should be `'right'` or `'left'`. Args: model_max_length (`int`, *optional*): The maximum length (in number of tokens) for the inputs to the transformer model. When the tokenizer is loaded with [`~tokenization_utils_base.PreTrainedTokenizerBase.from_pretrained`], this will be set to the value stored for the associated model in `max_model_input_sizes` (see above). If no value is provided, will default to VERY_LARGE_INTEGER (`int(1e30)`). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. truncation_side (`str`, *optional*): The side on which the model should have truncation applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. model_input_names (`List[string]`, *optional*): The list of inputs accepted by the forward pass of the model (like `"token_type_ids"` or `"attention_mask"`). Default value is picked from the class attribute of the same name. bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. Will be associated to `self.bos_token` and `self.bos_token_id`. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. Will be associated to `self.eos_token` and `self.eos_token_id`. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. Will be associated to `self.unk_token` and `self.unk_token_id`. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). Will be associated to `self.sep_token` and `self.sep_token_id`. pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. Will be associated to `self.pad_token` and `self.pad_token_id`. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). Will be associated to `self.cls_token` and `self.cls_token_id`. mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). Will be associated to `self.mask_token` and `self.mask_token_id`. additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. Add them here to ensure they won't be split by the tokenization process. Will be associated to `self.additional_special_tokens` and `self.additional_special_tokens_ids`. clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. """ def add_end_docstrings(*docstr): def docstring_decorator(fn): fn.__doc__ = (fn.__doc__ if fn.__doc__ is not None else "") + "".join(docstr) return fn return docstring_decorator def to_py_obj(obj): """ Convert a Mindspore tensor, Numpy array or python list to a python list. """ if isinstance(obj, (dict, UserDict)): return {k: to_py_obj(v) for k, v in obj.items()} if isinstance(obj, (list, tuple)): return [to_py_obj(o) for o in obj] if isinstance(obj, ms.Tensor): return obj.asnumpy().tolist() if isinstance(obj, (np.ndarray, np.number)): # tolist also works on 0d np arrays return obj.tolist() return obj def _is_whitespace(char): """Checks whether `char` is a whitespace character.""" # \t, \n, and \r are technically control characters but we treat them # as whitespace since they are generally considered as such. if char in (' ', '\t', '\n', '\r'): return True cat = unicodedata.category(char) if cat == "Zs": return True return False def _is_control(char): """Checks whether `char` is a control character.""" # These are technically control characters but we count them as whitespace # characters. if char in ('\t', '\n', '\r'): return False cat = unicodedata.category(char) if cat.startswith("C"): return True return False def _is_punctuation(char): """Checks whether `char` is a punctuation character.""" cp = ord(char) # We treat all non-letter/number ASCII as punctuation. # Characters such as "^", "$", and "`" are not in the Unicode # Punctuation class but we treat them as punctuation anyways, for # consistency. if (33 <= cp <= 47) or (58 <= cp <= 64) or (91 <= cp <= 96) or (123 <= cp <= 126): return True cat = unicodedata.category(char) if cat.startswith("P"): return True return False def _is_end_of_word(text): """Checks whether the last character in text is one of a punctuation, control or whitespace character.""" last_char = text[-1] return bool(_is_control(last_char) | _is_punctuation(last_char) | _is_whitespace(last_char)) def _is_start_of_word(text): """Checks whether the first character in text is one of a punctuation, control or whitespace character.""" first_char = text[0] return bool(_is_control(first_char) | _is_punctuation(first_char) | _is_whitespace(first_char)) def _insert_one_token_to_ordered_list(token_list: List[str], new_token: str): """ Inserts one token to an ordered list if it does not already exist. Note: token_list must be sorted. """ insertion_idx = bisect.bisect_left(token_list, new_token) # Checks if new_token is already in the ordered token_list if not (insertion_idx < len(token_list) and token_list[insertion_idx] == new_token): # new_token is in token_list, don't add token_list.insert(insertion_idx, new_token) def print_path_or_list(input_path_or_list): """ Print path or list function for show support method of MindFormerBook or other BaseClasses. Args: input_path_or_list (str, list): the path or list to be printed. """ if isinstance(input_path_or_list, (str, list)): logger.info(" %s", input_path_or_list) logger.info("-------------------------------------") else: raise TypeError(f"{type(input_path_or_list)} is unsupported by print_path_or_list") class SpecialTokensMixin: """ A mixin derived by [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`] to handle specific behaviors related to special tokens. In particular, this class hold the attributes which can be used to directly access these special tokens in a model-independent manner and allow to set and update the special tokens. Args: bos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the beginning of a sentence. eos_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the end of a sentence. unk_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing an out-of-vocabulary token. sep_token (`str` or `tokenizers.AddedToken`, *optional*): A special token separating two different sentences in the same input (used by BERT for instance). pad_token (`str` or `tokenizers.AddedToken`, *optional*): A special token used to make arrays of tokens the same size for batching purpose. Will then be ignored by attention mechanisms or loss computation. cls_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing the class of the input (used by BERT for instance). mask_token (`str` or `tokenizers.AddedToken`, *optional*): A special token representing a masked token (used by masked-language modeling pretraining objectives, like BERT). additional_special_tokens (tuple or list of `str` or `tokenizers.AddedToken`, *optional*): A tuple or a list of additional special tokens. """ SPECIAL_TOKENS_ATTRIBUTES = [ "bos_token", "eos_token", "unk_token", "sep_token", "pad_token", "cls_token", "mask_token", "additional_special_tokens", ] def __init__(self, verbose=True, **kwargs): self._bos_token = None self._eos_token = None self._unk_token = None self._sep_token = None self._pad_token = None self._cls_token = None self._mask_token = None self._pad_token_type_id = 0 self._additional_special_tokens = [] self.verbose = verbose # We directly set the hidden value to allow initialization with special tokens # which are not yet in the vocabulary. Necessary for serialization/de-serialization # TODO clean this up at some point (probably by switching to fast tokenizers) for key, value in kwargs.items(): if value is None: continue if key in self.SPECIAL_TOKENS_ATTRIBUTES: if key == "additional_special_tokens": assert isinstance(value, (list, tuple)), f"Value {value} is not a list or tuple" assert all( isinstance(t, (str, AddedToken)) for t in value ), "One of the tokens is not a string or an AddedToken" setattr(self, key, value) elif isinstance(value, (str, AddedToken)): setattr(self, key, value) else: raise TypeError(f"special token {key} has to be either str or AddedToken but got: {type(value)}") def sanitize_special_tokens(self) -> int: """ Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`, `tokenizer.cls_token`, etc.) are in the vocabulary. Add the missing ones to the vocabulary if needed. Return: `int`: The number of tokens added in the vocabulary during the operation. """ return self.add_tokens(self.all_special_tokens_extended, special_tokens=True) def add_special_tokens( self, special_tokens_dict: Dict[str, Union[str, AddedToken]], replace_additional_special_tokens=True ) -> int: """ Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the current vocabulary). Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Using `add_special_tokens` will ensure your special tokens can be used in several ways: - Special tokens are carefully handled by the tokenizer (they are never split). - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This makes it easy to develop model-agnostic training and fine-tuning scripts. When possible, special tokens are already registered for provided pretrained models (for instance [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be `''`). Args: special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`): Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`]. Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`): If `True`, the existing list of additional special tokens will be replaced by the one specified in `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as non-special tokens. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to add a new classification token to GPT-2 tokenizer = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2Model.from_pretrained("gpt2") special_tokens_dict = {"cls_token": ""} num_added_toks = tokenizer.add_special_tokens(special_tokens_dict) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) assert tokenizer.cls_token == "" ```""" if not special_tokens_dict: return 0 added_tokens = 0 for key, value in special_tokens_dict.items(): assert key in self.SPECIAL_TOKENS_ATTRIBUTES, f"Key {key} is not a special token" if self.verbose: logger.info("Assigning %s to the %s key of the tokenizer", value, key) if key == "additional_special_tokens": assert isinstance(value, (list, tuple)) and all( isinstance(t, (str, AddedToken)) for t in value ), f"Tokens {value} for key {key} should all be str or AddedToken instances" if replace_additional_special_tokens: setattr(self, key, value) else: # This is a copy of `self._additional_special_tokens` additional_special_tokens = getattr(self, key) additional_special_tokens_set = set(additional_special_tokens) to_add = [] for token in value: if str(token) not in additional_special_tokens_set and str(token) not in to_add: to_add.append(token) # update the property additional_special_tokens.extend(to_add) self.additional_special_tokens = additional_special_tokens added_tokens += self.add_tokens(value, special_tokens=True) else: assert isinstance( value, (str, AddedToken) ), f"Token {value} for key {key} should be a str or an AddedToken instance" setattr(self, key, value) added_tokens += self.add_tokens([value], special_tokens=True) return added_tokens def add_tokens( self, new_tokens: Union[str, AddedToken, List[Union[str, AddedToken]]], special_tokens: bool = False ) -> int: """ Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary and and will be isolated before the tokenization algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore not treated in the same way. Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix of the model so that its embedding matrix matches the tokenizer. In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method. Args: new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`): Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string token to let you personalize its behavior: whether this token should only match against a single word, whether this token should strip all potential whitespaces on the left side, whether this token should strip all potential whitespaces on the right side, etc. special_tokens (`bool`, *optional*, defaults to `False`): Can be used to specify if the token is a special token. This mostly change the normalization behavior (special tokens like CLS or [MASK] are usually not lower-cased for instance). See details for `tokenizers.AddedToken` in HuggingFace tokenizers library. Returns: `int`: Number of tokens added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```""" if not new_tokens: return 0 if not isinstance(new_tokens, (list, tuple)): new_tokens = [new_tokens] return self._add_tokens(new_tokens, special_tokens=special_tokens) def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int: raise NotImplementedError @property def bos_token(self) -> str: """ `str`: Beginning of sentence token. Log an error if used while not having been set. """ if self._bos_token is None: if self.verbose: logger.error("Using bos_token, but it is not set yet.") return None return str(self._bos_token) @property def eos_token(self) -> str: """ `str`: End of sentence token. Log an error if used while not having been set. """ if self._eos_token is None: if self.verbose: logger.error("Using eos_token, but it is not set yet.") return None return str(self._eos_token) @property def unk_token(self) -> str: """ `str`: Unknown token. Log an error if used while not having been set. """ if self._unk_token is None: if self.verbose: logger.error("Using unk_token, but it is not set yet.") return None return str(self._unk_token) @property def sep_token(self) -> str: """ `str`: Separation token, to separate context and query in an input sequence. Log an error if used while not having been set. """ if self._sep_token is None: if self.verbose: logger.error("Using sep_token, but it is not set yet.") return None return str(self._sep_token) @property def pad_token(self) -> str: """ `str`: Padding token. Log an error if used while not having been set. """ if self._pad_token is None: if self.verbose: logger.error("Using pad_token, but it is not set yet.") return None return str(self._pad_token) @property def cls_token(self) -> str: """ `str`: Classification token, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Log an error if used while not having been set. """ if self._cls_token is None: if self.verbose: logger.error("Using cls_token, but it is not set yet.") return None return str(self._cls_token) @property def mask_token(self) -> str: """ `str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while not having been set. """ if self._mask_token is None: if self.verbose: logger.error("Using mask_token, but it is not set yet.") return None return str(self._mask_token) @property def additional_special_tokens(self) -> List[str]: """ `List[str]`: All the additional special tokens you may want to use. Log an error if used while not having been set. """ if self._additional_special_tokens is None: if self.verbose: logger.error("Using additional_special_tokens, but it is not set yet.") return None return [str(tok) for tok in self._additional_special_tokens] @bos_token.setter def bos_token(self, value): self._bos_token = value @eos_token.setter def eos_token(self, value): self._eos_token = value @unk_token.setter def unk_token(self, value): self._unk_token = value @sep_token.setter def sep_token(self, value): self._sep_token = value @pad_token.setter def pad_token(self, value): self._pad_token = value @cls_token.setter def cls_token(self, value): self._cls_token = value @mask_token.setter def mask_token(self, value): self._mask_token = value @additional_special_tokens.setter def additional_special_tokens(self, value): self._additional_special_tokens = value @property def bos_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the beginning of sentence token in the vocabulary. Returns `None` if the token has not been set. """ if self._bos_token is None: return None return self.convert_tokens_to_ids(self.bos_token) @property def eos_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been set. """ if self._eos_token is None: return None return self.convert_tokens_to_ids(self.eos_token) @property def unk_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the unknown token in the vocabulary. Returns `None` if the token has not been set. """ if self._unk_token is None: return None return self.convert_tokens_to_ids(self.unk_token) @property def sep_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the separation token in the vocabulary, to separate context and query in an input sequence. Returns `None` if the token has not been set. """ if self._sep_token is None: return None return self.convert_tokens_to_ids(self.sep_token) @property def pad_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set. """ if self._pad_token is None: return None return self.convert_tokens_to_ids(self.pad_token) @property def pad_token_type_id(self) -> int: """ `int`: Id of the padding token type in the vocabulary. """ return self._pad_token_type_id @property def cls_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the classification token in the vocabulary, to extract a summary of an input sequence leveraging self-attention along the full depth of the model. Returns `None` if the token has not been set. """ if self._cls_token is None: return None return self.convert_tokens_to_ids(self.cls_token) @property def mask_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the mask token in the vocabulary, used when training a model with masked-language modeling. Returns `None` if the token has not been set. """ if self._mask_token is None: return None return self.convert_tokens_to_ids(self.mask_token) @property def additional_special_tokens_ids(self) -> List[int]: """ `List[int]`: Ids of all the additional special tokens in the vocabulary. Log an error if used while not having been set. """ return self.convert_tokens_to_ids(self.additional_special_tokens) @bos_token_id.setter def bos_token_id(self, value): self._bos_token = self.convert_ids_to_tokens(value) if value is not None else None @eos_token_id.setter def eos_token_id(self, value): self._eos_token = self.convert_ids_to_tokens(value) if value is not None else None @unk_token_id.setter def unk_token_id(self, value): self._unk_token = self.convert_ids_to_tokens(value) if value is not None else None @sep_token_id.setter def sep_token_id(self, value): self._sep_token = self.convert_ids_to_tokens(value) if value is not None else None @pad_token_id.setter def pad_token_id(self, value): self._pad_token = self.convert_ids_to_tokens(value) if value is not None else None @cls_token_id.setter def cls_token_id(self, value): self._cls_token = self.convert_ids_to_tokens(value) if value is not None else None @mask_token_id.setter def mask_token_id(self, value): self._mask_token = self.convert_ids_to_tokens(value) if value is not None else None @additional_special_tokens_ids.setter def additional_special_tokens_ids(self, values): self._additional_special_tokens = [self.convert_ids_to_tokens(value) for value in values] @property def special_tokens_map(self) -> Dict[str, Union[str, List[str]]]: """ `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Convert potential tokens of `tokenizers.AddedToken` type to string. """ set_attr = {} for attr in self.SPECIAL_TOKENS_ATTRIBUTES: attr_value = getattr(self, "_" + attr) if attr_value: set_attr[attr] = ( type(attr_value)(str(attr_value_sub) for attr_value_sub in attr_value) if isinstance(attr_value, (list, tuple)) else str(attr_value) ) return set_attr @property def special_tokens_map_extended(self) -> Dict[str, Union[str, AddedToken, List[Union[str, AddedToken]]]]: """ `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.). Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. """ set_attr = {} for attr in self.SPECIAL_TOKENS_ATTRIBUTES: attr_value = getattr(self, "_" + attr) if attr_value: set_attr[attr] = attr_value return set_attr @property def all_special_tokens(self) -> List[str]: """ `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Convert tokens of `tokenizers.AddedToken` type to string. """ all_toks = [str(s) for s in self.all_special_tokens_extended] return all_toks @property def all_special_tokens_extended(self) -> List[Union[str, AddedToken]]: """ `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes. Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how special tokens are tokenized. """ all_toks = [] set_attr = self.special_tokens_map_extended for attr_value in set_attr.values(): all_toks = all_toks + (list(attr_value) if isinstance(attr_value, (list, tuple)) else [attr_value]) all_toks = list(OrderedDict.fromkeys(all_toks)) return all_toks @property def all_special_ids(self) -> List[int]: """ `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes. """ all_toks = self.all_special_tokens all_ids = self.convert_tokens_to_ids(all_toks) return all_ids @add_end_docstrings(INIT_TOKENIZER_DOCSTRING) class BaseTokenizer(SpecialTokensMixin): """ Base class for [`PreTrainedTokenizer`] and [`PreTrainedTokenizerFast`]. Handles shared (mostly boiler plate) methods for those two classes. """ vocab_files_names: Dict[str, str] = {} pretrained_vocab_files_map: Dict[str, Dict[str, str]] = {} pretrained_init_configuration: Dict[str, Dict[str, Any]] = {} max_model_input_sizes: Dict[str, Optional[int]] = {} _auto_class: Optional[str] = None FILE_LIST: List[str] = [] # first name has to correspond to main model input name # to make sure `tokenizer.pad(...)` works correctly model_input_names: List[str] = ["input_ids", "token_type_ids", "attention_mask"] padding_side: str = "right" truncation_side: str = "right" slow_tokenizer_class = None _model_type = 0 _model_name = 1 def __init__(self, **kwargs): super().__init__(self) # inputs and kwargs for saving and re-loading (see ``from_pretrained`` and ``save_pretrained``) self.init_inputs = () self.init_kwargs = copy.deepcopy(kwargs) self.name_or_path = kwargs.pop("name_or_path", "") self._processor_class = kwargs.pop("processor_class", None) # For backward compatibility we fallback to set model_max_length from max_len if provided model_max_length = kwargs.pop("model_max_length", kwargs.pop("max_len", None)) self.model_max_length = model_max_length if model_max_length is not None else VERY_LARGE_INTEGER # Padding and truncation side are right by default and overridden in subclasses. If specified in the kwargs, it # is changed. self.padding_side = kwargs.pop("padding_side", self.padding_side) if self.padding_side not in ["right", "left"]: raise ValueError( f"Padding side should be selected between 'right' and 'left', current value: {self.padding_side}" ) self.truncation_side = kwargs.pop("truncation_side", self.truncation_side) if self.truncation_side not in ["right", "left"]: raise ValueError( f"Padding side should be selected between 'right' and 'left', current value: {self.truncation_side}" ) self.model_input_names = kwargs.pop("model_input_names", self.model_input_names) # By default, cleaning tokenization spaces for both fast and slow tokenizers self.clean_up_tokenization_spaces = kwargs.pop("clean_up_tokenization_spaces", True) self.deprecation_warnings = ( {} ) # Use to store when we have already noticed a deprecation warning (avoid overlogging). self._in_target_context_manager = False super().__init__(**kwargs) @property def max_len_single_sentence(self) -> int: """ `int`: The maximum length of a sentence that can be fed to the model. """ return self.model_max_length - self.num_special_tokens_to_add(pair=False) @property def max_len_sentences_pair(self) -> int: """ `int`: The maximum combined length of a pair of sentences that can be fed to the model. """ return self.model_max_length - self.num_special_tokens_to_add(pair=True) @max_len_single_sentence.setter def max_len_single_sentence(self, value) -> int: # For backward compatibility, allow to try to setup 'max_len_single_sentence'. if value == self.model_max_length - self.num_special_tokens_to_add(pair=False) and self.verbose: if not self.deprecation_warnings.get("max_len_single_sentence", False): logger.warning( "Setting 'max_len_single_sentence' is now deprecated. This value is automatically set up." ) self.deprecation_warnings["max_len_single_sentence"] = True else: raise ValueError( "Setting 'max_len_single_sentence' is now deprecated. This value is automatically set up." ) @max_len_sentences_pair.setter def max_len_sentences_pair(self, value) -> int: # For backward compatibility, allow to try to setup 'max_len_sentences_pair'. if value == self.model_max_length - self.num_special_tokens_to_add(pair=True) and self.verbose: if not self.deprecation_warnings.get("max_len_sentences_pair", False): logger.warning( "Setting 'max_len_sentences_pair' is now deprecated. This value is automatically set up." ) self.deprecation_warnings["max_len_sentences_pair"] = True else: raise ValueError("Setting 'max_len_sentences_pair' is now deprecated. This value is automatically set up.") def _set_processor_class(self, processor_class: str): """Sets processor class as an attribute.""" self._processor_class = processor_class def __repr__(self) -> str: return ( f"{self.__class__.__name__}(name_or_path='{self.name_or_path}'," f" vocab_size={self.vocab_size}, model_max_length={self.model_max_length}, is_fast={self.is_fast}," f" padding_side='{self.padding_side}', truncation_side='{self.truncation_side}'," f" special_tokens={self.special_tokens_map_extended}, " f"clean_up_tokenization_spaces={self.clean_up_tokenization_spaces})" ) def __len__(self) -> int: raise NotImplementedError() def get_vocab(self) -> Dict[str, int]: """ Returns the vocabulary as a dictionary of token to index. `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the vocab. Returns: `Dict[str, int]`: The vocabulary. """ raise NotImplementedError() @classmethod def from_pretrained(cls, name_or_path: str, **kwargs): """ Instantiates a tokenizer by the name_or_path. User can get the name using `get_support_list` of any tokenizer, it will download the necessary files from the cloud. or pass a directory where contains the vocabulary file and tokenizers yaml configuration file. Args: name_or_path (str): It supports the following two input types: If the name_or_path is a supported tokenizer name, for example, `clip_vit_b_32` and `t5_small`, it will download the necessary files from the cloud. User can select one from the support list by call `MindFormerBook.show_tokenizer_support_list()`. If name_or_path is a path to the local directory where there should have vocaburary files and configuration file ended with `yaml`. The vocaburary file needed by the tokenizer is determined by `.vocab_files_names`. pretrained_model_name_or_path (Optional[str]): Equal to "name_or_path", if "pretrained_model_name_or_path" is set, "name_or_path" is useless. Examples: >>> from mindformers import T5Tokenizer >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> res = tokenizer.encode("hello world!") >>> print(res) [21820, 296, 55, 1] Returns: A instanced tokenizer. """ pretrained_model_name_or_path = kwargs.pop("pretrained_model_name_or_path", None) if pretrained_model_name_or_path is not None: name_or_path = pretrained_model_name_or_path is_exist = os.path.exists(name_or_path) is_dir = os.path.isdir(name_or_path) if not is_exist and (name_or_path not in cls._support_list): raise ValueError(f'{name_or_path} does not exist,' f' or it is not supported by {cls.__name__}. ' f'please select from {cls._support_list}.') if is_exist and not is_dir: raise ValueError(f"{name_or_path} is not a directory.") kwargs = dict() class_name = None loaded_kwargs = {} if name_or_path in MindFormerBook.get_tokenizer_url_support_list(): config, cache_path = cls._download_using_name(name_or_path) class_name, loaded_kwargs = cls._get_class_name_and_args_form_config(config) name_or_path = cache_path yaml_list = None if os.path.isdir(name_or_path): yaml_list = [file for file in os.listdir(name_or_path) if file.endswith(".yaml")] if len(yaml_list) > 1: logger.warning("There should be only one yaml file under the directory %s, " "but followings are found: %s", name_or_path, yaml_list) if yaml_list: yaml_file = os.path.join(name_or_path, yaml_list[0]) logger.info("config in the yaml file %s are used for tokenizer building.", yaml_file) config = MindFormerConfig(yaml_file) class_name, loaded_kwargs = cls._get_class_name_and_args_form_config(config) vocab_dict, file_dict = cls.read_files_according_specific_by_tokenizer(name_or_path) if 'tokenizer_config.json' in file_dict: class_name = file_dict['tokenizer_config.json'].pop('tokenizer_class', None) loaded_kwargs = file_dict['tokenizer_config.json'] else: logger.warning("Can't find the tokenizer_config.json in the file_dict. " "The content of file_dict is : %s", file_dict) kwargs.update(loaded_kwargs) kwargs.update(vocab_dict) if not class_name: class_name = cls.__name__ logger.info("build tokenizer class name is: %s using args %s.", class_name, kwargs) tokenizer = build_tokenizer(class_name=class_name, **kwargs) # Check all our special tokens are registered as "no split" token (we don't cut them) and are in the vocab added_tokens = tokenizer.sanitize_special_tokens() if added_tokens: logger.warning( "Special tokens have been added in the vocabulary, make sure the associated word embeddings are" " fine-tuned or trained." ) return tokenizer @classmethod def _download_using_name(cls, name_or_path): """Given the supported model name, download it from the urls""" tokenizer_name = name_or_path if name_or_path.startswith('mindspore'): # Adaptation the name of tokenizer at the beginning of mindspore, # the relevant file will be downloaded from the Xihe platform. # such as "mindspore/clip_vit_b_32" tokenizer_name = name_or_path.split('/')[cls._model_name] cache_path = os.path.join(MindFormerBook.get_xihe_checkpoint_download_folder(), tokenizer_name.split('_')[cls._model_type]) else: # Default the name of tokenizer, # the relevant file will be downloaded from the Obs platform. # such as "clip_vit_b_32" cache_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), name_or_path.split('_')[cls._model_type]) if not os.path.exists(cache_path): os.makedirs(cache_path, exist_ok=True) yaml_file = os.path.join(cache_path, tokenizer_name + ".yaml") def get_default_yaml_file(model_name): default_yaml_file = "" for model_dict in MindFormerBook.get_trainer_support_task_list().values(): if model_name in model_dict: default_yaml_file = model_dict.get(model_name) break return default_yaml_file if not os.path.exists(yaml_file): default_yaml_file = get_default_yaml_file(tokenizer_name) if os.path.realpath(default_yaml_file) and os.path.exists(default_yaml_file): shutil.copy(default_yaml_file, yaml_file) logger.info("default yaml config in %s is used.", yaml_file) else: raise FileNotFoundError(f'default yaml file path must be correct, but get {default_yaml_file}') # some tokenizers rely on more than one file, e.g gpt2 tokenizer_need_files = MindFormerBook.get_tokenizer_url_support_list()[name_or_path] for url_file in tokenizer_need_files: local_file_name = url_file.split('/')[-1] file_path = os.path.join(cache_path, local_file_name) if not os.path.exists(file_path): logger.info("Download the vocab from the url %s to %s.", url_file, file_path) download_with_progress_bar(url_file, file_path) try_sync_file(file_path) config = MindFormerConfig(yaml_file) return config, cache_path @classmethod def cache_vocab_files(cls, name_or_path, cache_path=None): """Cache the vocab files to the default dir""" if not cache_path: cache_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), name_or_path.split("_")[cls._model_type]) if not os.path.exists(cache_path): os.makedirs(cache_path, exist_ok=True) # some tokenizers rely on more than one file, e.g gpt2 tokenizer_need_files = MindFormerBook.get_tokenizer_url_support_list()[name_or_path] for url_file in tokenizer_need_files: local_file_name = url_file.split('/')[-1] file_path = os.path.join(cache_path, local_file_name) if not os.path.exists(file_path): logger.info("Download the yaml from the url %s to %s.", url_file, file_path) download_with_progress_bar(url_file, file_path) try_sync_file(file_path) read_vocab_file_dict, _ = cls.read_files_according_specific_by_tokenizer(cache_path) return read_vocab_file_dict @classmethod def _get_class_name_and_args_form_config(cls, config): """Lookup the yaml files under the name_or_path""" class_name = None tokenizer_args = {} if config and 'processor' in config and 'tokenizer' in config['processor'] \ and config.processor.tokenizer and 'type' in config.processor.tokenizer: tokenizer_args = config['processor']['tokenizer'] class_name = tokenizer_args.pop('type', None) else: logger.info("There is no matched format config['processor']['tokenizer'] in config %s", config) return class_name, tokenizer_args @classmethod def read_files_according_specific_by_tokenizer(cls, name_or_path): """Read the file path specific by the class variable in the tokenizer""" read_vocab_file_dict = {} read_tokenizer_file_dict = {} for k, name in cls.vocab_files_names.items(): if isinstance(name, str): path = os.path.join(name_or_path, name) if os.path.isfile(path): read_vocab_file_dict[k] = path # To support tokenizer like clip that has two types for vocab files. elif isinstance(name, list): for sub_name in name: path = os.path.join(name_or_path, sub_name) if os.path.isfile(path): read_vocab_file_dict[k] = path for item in cls.FILE_LIST: path = os.path.join(name_or_path, item) if os.path.isfile(path): read_tokenizer_file_dict[item] = json.load(open(path, 'r')) return read_vocab_file_dict, read_tokenizer_file_dict def save_pretrained(self, save_directory: Optional[str] = None, save_name: str = "mindspore_model", file_format: str = 'yaml'): """ Save the tokenizer by writing the `save_name`.yaml and vocaburary files those are determinied by `.vocab_files_names` to the disk. The kwargs passed to initialize the tokenizer will be saved. Args: save_directory(str): The output file directory. If None, the directory will be `./checkpoint_save`, which can be obtained by the `MindFormerBook.get_default_checkpoint_save_folder()`. Default None. save_name(str): The file name of the saved files. Default mindspore_model. file_format(str): Support json or yaml. Default yaml. Examples: >>> from mindformers import T5Tokenizer, MindFormerBook >>> tokenizer = T5Tokenizer.from_pretrained("t5_small") >>> tokenizer.save_pretrained() >>> output_path = MindFormerBook.get_default_checkpoint_save_folder() >>> print(os.listdir(output_path)) ['mindspore_model.yaml', 'spiece.model'] """ default_directory = MindFormerBook.get_default_checkpoint_save_folder() if save_directory is None: save_directory = default_directory os.makedirs(save_directory, exist_ok=True) if save_name is None: save_name = "mindspore_model" if file_format not in ('yaml', 'json'): raise ValueError(f"format should be one of [`yaml`, `json`], but got {file_format}.") kwargs = copy.deepcopy(self.init_kwargs) # Start to save the kwargs for the tokenizer if file_format == 'yaml': kwargs['type'] = self.__class__.__name__ merged_dict = dict() yaml_file = os.path.join(save_directory, save_name + '.yaml') if os.path.exists(yaml_file): with open(yaml_file, 'r') as file_reader: merged_dict = yaml.load(file_reader.read(), Loader=yaml.Loader) if merged_dict is None: merged_dict = dict() processor_name = MindFormerBook.get_tokenizer_name_to_processor()[kwargs['type']] merged_dict['processor'] = {"type": processor_name} if isinstance(kwargs, dict): for token_key, token_value in kwargs.items(): if isinstance(token_value, AddedToken): kwargs[token_key] = token_value.content merged_dict['processor']['tokenizer'] = kwargs with open(yaml_file, 'w') as file_reader: yaml.dump(merged_dict, file_reader) elif file_format == 'json': kwargs["tokenizer_class"] = self.__class__.__name__ tokenizer_config_path = os.path.join(save_directory, TOKENIZER_CONFIG_NAME) with open(tokenizer_config_path, 'w') as fp: json.dump(kwargs, fp, indent=4) else: raise ValueError(f"file_format should be one of [json, yaml], but got {file_format}.") self.save_vocabulary(save_directory) def save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]: """ Save only the vocabulary of the tokenizer (vocabulary + added tokens). This method won't save the configuration and special token mappings of the tokenizer. Use [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ raise NotImplementedError def tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]: """ Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`. Args: text (`str`): The sequence to be encoded. pair (`str`, *optional*): A second sequence to be encoded with the first. add_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to add the special tokens associated with the corresponding model. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific encode method. See details in [`~PreTrainedTokenizerBase.__call__`] Returns: `List[str]`: The list of tokens. """ raise NotImplementedError @add_end_docstrings( ENCODE_KWARGS_DOCSTRING, """ **kwargs: Passed along to the `.tokenize()` method. """, """ Returns: `List[int]`, `ms.Tensor`, or `np.ndarray`: The tokenized ids of the text. """, ) def encode( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, return_tensors: Optional[Union[str, TensorType]] = None, **kwargs, ) -> List[int]: """ Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary. Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`. Args: text (`str`, `List[str]` or `List[int]`): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). """ encoded_inputs = self.encode_plus( text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, return_tensors=return_tensors, **kwargs, ) return encoded_inputs["input_ids"] def num_special_tokens_to_add(self, pair: bool = False) -> int: raise NotImplementedError def _get_padding_truncation_strategies( self, padding=False, truncation=None, max_length=None, pad_to_multiple_of=None, verbose=True, **kwargs ): """ Find the correct padding/truncation strategy with backward compatibility for old arguments (truncation_strategy and pad_to_max_length) and behaviors. """ old_truncation_strategy = kwargs.pop("truncation_strategy", "do_not_truncate") old_pad_to_max_length = kwargs.pop("pad_to_max_length", False) # Backward compatibility for previous behavior, maybe we should deprecate it: # If you only set max_length, it activates truncation for max_length if max_length is not None and padding is False and truncation is None: if verbose: if not self.deprecation_warnings.get("Truncation-not-explicitly-activated", False): logger.warning( "Truncation was not explicitly activated but `max_length` is provided a specific value, please" " use `truncation=True` to explicitly truncate examples to max length. Defaulting to" " 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the" " tokenizer you can select this strategy more precisely by providing a specific strategy to" " `truncation`." ) self.deprecation_warnings["Truncation-not-explicitly-activated"] = True truncation = "longest_first" # Get padding strategy if padding is False and old_pad_to_max_length: if verbose: warnings.warn( "The `pad_to_max_length` argument is deprecated and will be removed in a future version, " "use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or " "use `padding='max_length'` to pad to a max length. In this case, you can give a specific " "length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the " "maximal input size of the model (e.g. 512 for Bert).", FutureWarning, ) if max_length is None: padding_strategy = PaddingStrategy.LONGEST else: padding_strategy = PaddingStrategy.MAX_LENGTH elif padding is not False: if padding is True: if verbose: if max_length is not None and ( truncation is None or truncation is False or truncation == "do_not_truncate" ): warnings.warn( "`max_length` is ignored when `padding`=`True` and there is no truncation strategy. " "To pad to max length, use `padding='max_length'`." ) if old_pad_to_max_length is not False: warnings.warn("Though `pad_to_max_length` = `True`, it is ignored because `padding`=`True`.") padding_strategy = PaddingStrategy.LONGEST # Default to pad to the longest sequence in the batch elif not isinstance(padding, PaddingStrategy): padding_strategy = PaddingStrategy(padding) elif isinstance(padding, PaddingStrategy): padding_strategy = padding else: padding_strategy = PaddingStrategy.DO_NOT_PAD # Get truncation strategy if truncation is None and old_truncation_strategy != "do_not_truncate": if verbose: warnings.warn( "The `truncation_strategy` argument is deprecated and will be removed in a future version, use" " `truncation=True` to truncate examples to a max length. You can give a specific length with" " `max_length` (e.g. `max_length=45`) or leave max_length to None to truncate to the maximal input" " size of the model (e.g. 512 for Bert). If you have pairs of inputs, you can give a specific" " truncation strategy selected among `truncation='only_first'` (will only truncate the first" " sentence in the pairs) `truncation='only_second'` (will only truncate the second sentence in the" " pairs) or `truncation='longest_first'` (will iteratively remove tokens from the longest sentence" " in the pairs).", FutureWarning, ) truncation_strategy = TruncationStrategy(old_truncation_strategy) elif truncation is not False and truncation is not None: if truncation is True: truncation_strategy = ( TruncationStrategy.LONGEST_FIRST ) # Default to truncate the longest sequences in pairs of inputs elif not isinstance(truncation, TruncationStrategy): truncation_strategy = TruncationStrategy(truncation) elif isinstance(truncation, TruncationStrategy): truncation_strategy = truncation else: truncation_strategy = TruncationStrategy.DO_NOT_TRUNCATE # Set max length if needed if max_length is None: if padding_strategy == PaddingStrategy.MAX_LENGTH: if self.model_max_length > LARGE_INTEGER: if verbose: if not self.deprecation_warnings.get("Asking-to-pad-to-max_length", False): logger.warning( "Asking to pad to max_length but no maximum length is provided and the model has no" " predefined maximum length. Default to no padding." ) self.deprecation_warnings["Asking-to-pad-to-max_length"] = True padding_strategy = PaddingStrategy.DO_NOT_PAD else: max_length = self.model_max_length if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE: if self.model_max_length > LARGE_INTEGER: if verbose: if not self.deprecation_warnings.get("Asking-to-truncate-to-max_length", False): logger.warning( "Asking to truncate to max_length but no maximum length is provided and the model has" " no predefined maximum length. Default to no truncation." ) self.deprecation_warnings["Asking-to-truncate-to-max_length"] = True truncation_strategy = TruncationStrategy.DO_NOT_TRUNCATE else: max_length = self.model_max_length # Test if we have a padding token if padding_strategy != PaddingStrategy.DO_NOT_PAD and (not self.pad_token or self.pad_token_id < 0): raise ValueError( "Asking to pad but the tokenizer does not have a padding token. " "Please select a token to use as `pad_token` `(tokenizer.pad_token = tokenizer.eos_token e.g.)` " "or add a new pad token via `tokenizer.add_special_tokens({'pad_token': '[PAD]'})`." ) # Check that we will truncate to a multiple of pad_to_multiple_of if both are provided if ( truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and padding_strategy != PaddingStrategy.DO_NOT_PAD and pad_to_multiple_of is not None and max_length is not None and (max_length % pad_to_multiple_of != 0) ): raise ValueError( "Truncation and padding are both activated but " f"truncation length ({max_length}) is not a multiple of pad_to_multiple_of ({pad_to_multiple_of})." ) return padding_strategy, truncation_strategy, max_length, kwargs @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def __call__( self, text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None, text_pair: Optional[Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]] = None, text_target: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None, text_pair_target: Optional[ Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] ] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of sequences. Args: text (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*): The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). """ # To avoid duplicating all_kwargs = { "add_special_tokens": add_special_tokens, "padding": padding, "truncation": truncation, "max_length": max_length, "stride": stride, "is_split_into_words": is_split_into_words, "pad_to_multiple_of": pad_to_multiple_of, "return_tensors": return_tensors, "return_token_type_ids": return_token_type_ids, "return_attention_mask": return_attention_mask, "return_overflowing_tokens": return_overflowing_tokens, "return_special_tokens_mask": return_special_tokens_mask, "return_offsets_mapping": return_offsets_mapping, "return_length": return_length, "verbose": verbose, } all_kwargs.update(kwargs) if text is None and text_target is None: raise ValueError("You need to specify either `text` or `text_target`.") if text is not None: # The context manager will send the inputs as normal texts and not text_target, but we shouldn't change the # input mode in this case. if not self._in_target_context_manager: self._switch_to_input_mode() encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs) if text_target is not None: self._switch_to_target_mode() target_encodings = self._call_one(text=text_target, text_pair=text_pair_target, **all_kwargs) # Leave back tokenizer in input mode self._switch_to_input_mode() if text_target is None: return encodings if text is None: return target_encodings encodings["labels"] = target_encodings["input_ids"] return encodings def _call_one( self, text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]], text_pair: Optional[Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.__call__""" # Input type checking for clearer error def _is_valid_text_input(t): if isinstance(t, str): # Strings are fine return True if isinstance(t, (list, tuple)): # List are fine as long as they are... if not t: # ... empty return True if isinstance(t[0], str): # ... list of strings return True if isinstance(t[0], (list, tuple)): # ... list with an empty list or with a list of strings return not t[0] or isinstance(t[0][0], str) return False return False if not _is_valid_text_input(text): raise ValueError( "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) " "or `List[List[str]]` (batch of pretokenized examples)." ) if text_pair is not None and not _is_valid_text_input(text_pair): raise ValueError( "text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) " "or `List[List[str]]` (batch of pretokenized examples)." ) if is_split_into_words: is_batched = isinstance(text, (list, tuple)) and text and isinstance(text[0], (list, tuple)) else: is_batched = isinstance(text, (list, tuple)) if is_batched: if isinstance(text_pair, str): raise TypeError( "when tokenizing batches of text, `text_pair` must be a list or tuple with the same length as" " `text`." ) if text_pair is not None and len(text) != len(text_pair): raise ValueError( f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:" f" {len(text_pair)}." ) batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text return self.batch_encode_plus( batch_text_or_text_pairs=batch_text_or_text_pairs, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) return self.encode_plus( text=text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding=padding, truncation=truncation, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Tokenize and prepare for the model a sequence or a pair of sequences. This method is deprecated, `__call__` should be used instead. Args: text (`str`, `List[str]` or `List[int]` (the latter only for not-fast tokenizers)): The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). text_pair (`str`, `List[str]` or `List[int]`, *optional*): Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids` method). """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) return self._encode_plus( text=text, text_pair=text_pair, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) def _encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.encode_plus""" raise NotImplementedError @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """ Tokenize and prepare for the model a list of sequences or a list of pairs of sequences. This method is deprecated, `__call__` should be used instead. Args: batch_text_or_text_pairs ( `List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`): Batch of sequences or pair of sequences to be encoded. This can be a list of string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see details in `encode_plus`). """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) return self._batch_encode_plus( batch_text_or_text_pairs=batch_text_or_text_pairs, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, is_split_into_words=is_split_into_words, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, return_token_type_ids=return_token_type_ids, return_attention_mask=return_attention_mask, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_offsets_mapping=return_offsets_mapping, return_length=return_length, verbose=verbose, **kwargs, ) def _batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: """used by self.batch_encode_plus""" raise NotImplementedError def pad( self, encoded_inputs: Union[ BatchEncoding, List[BatchEncoding], Dict[str, EncodedInput], Dict[str, List[EncodedInput]], List[Dict[str, EncodedInput]], ], padding: Union[bool, str, PaddingStrategy] = True, max_length: Optional[int] = None, pad_to_multiple_of: Optional[int] = None, return_attention_mask: Optional[bool] = None, return_tensors: Optional[Union[str, TensorType]] = None, verbose: bool = True, ) -> BatchEncoding: """ Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length in the batch. Padding side (left/right) padding token ids are defined at the tokenizer level (with `self.padding_side`, `self.pad_token_id` and `self.pad_token_type_id`). Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding. If the `encoded_inputs` passed are dictionary of numpy arrays, or Mindspore tensors, the result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of PyTorch tensors, you will lose the specific device of your tensors however. Args: encoded_inputs ( [`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`): Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str, List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader collate function. Instead of `List[int]` you can have tensors (numpy arrays or Mindspore tensors), see the note above for the return type. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the returned sequences (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific tokenizer's default, defined by the `return_outputs` attribute. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return Numpy `ms.Tensor` objects. verbose (`bool`, *optional*, defaults to `True`): Whether or not to print more information and warnings. """ if self.__class__.__name__.endswith("Fast"): if not self.deprecation_warnings.get("Asking-to-pad-a-fast-tokenizer", False): logger.warning_advice( f"You're using a {self.__class__.__name__} tokenizer. Please note that with a fast tokenizer," " using the `__call__` method is faster than using a method to encode the text followed by a call" " to the `pad` method to get a padded encoding." ) self.deprecation_warnings["Asking-to-pad-a-fast-tokenizer"] = True # If we have a list of dicts, let's convert it in a dict of lists # We do this to allow using this method as a collate_fn function in PyTorch Dataloader if isinstance(encoded_inputs, (list, tuple)) and isinstance(encoded_inputs[0], Mapping): encoded_inputs = {key: [example[key] for example in encoded_inputs] for key in encoded_inputs[0].keys()} # The model's main input name, usually `input_ids`, has be passed for padding if self.model_input_names[0] not in encoded_inputs: raise ValueError( "You should supply an encoding or a list of encodings to this method " f"that includes {self.model_input_names[0]}, but you provided {list(encoded_inputs.keys())}" ) required_input = encoded_inputs[self.model_input_names[0]] if required_input is None or (isinstance(required_input, Sized) and not required_input): if return_attention_mask: encoded_inputs["attention_mask"] = [] return encoded_inputs # If we have PyTorch/TF/NumPy tensors/arrays as inputs, we cast them as python objects # and rebuild them afterwards if no return_tensors is specified # Note that we lose the specific device the tensor may be on for PyTorch first_element = required_input[0] if isinstance(first_element, (list, tuple)): # first_element might be an empty list/tuple in some edge cases so we grab the first non empty element. for item in required_input: if item: first_element = item[0] break # At this state, if `first_element` is still a list/tuple, it's an empty one so there is nothing to do. if not isinstance(first_element, (int, list, tuple)): if isinstance(first_element, ms.Tensor): return_tensors = "ms" if return_tensors is None else return_tensors elif isinstance(first_element, np.ndarray): return_tensors = "np" if return_tensors is None else return_tensors else: raise ValueError( f"type of {first_element} unknown: {type(first_element)}. " "Should be one of a python, mindspore or numpy object." ) for key, value in encoded_inputs.items(): encoded_inputs[key] = to_py_obj(value) # Convert padding_strategy in PaddingStrategy padding_strategy, _, max_length, _ = self._get_padding_truncation_strategies( padding=padding, max_length=max_length, verbose=verbose ) required_input = encoded_inputs[self.model_input_names[0]] if required_input and not isinstance(required_input[0], (list, tuple)): encoded_inputs = self._pad( encoded_inputs, max_length=max_length, padding_strategy=padding_strategy, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) return BatchEncoding(encoded_inputs, tensor_type=return_tensors) batch_size = len(required_input) assert all( len(v) == batch_size for v in encoded_inputs.values() ), "Some items in the output dictionary have a different batch size than others." if padding_strategy == PaddingStrategy.LONGEST: max_length = max(len(inputs) for inputs in required_input) padding_strategy = PaddingStrategy.MAX_LENGTH batch_outputs = {} for i in range(batch_size): inputs = {k: v[i] for k, v in encoded_inputs.items()} outputs = self._pad( inputs, max_length=max_length, padding_strategy=padding_strategy, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) for key, value in outputs.items(): if key not in batch_outputs: batch_outputs[key] = [] batch_outputs[key].append(value) return BatchEncoding(batch_outputs, tensor_type=return_tensors) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create the token type IDs corresponding to the sequences passed. [What are token type IDs?](../glossary#token-type-ids) Should be overridden in a subclass if the model has a special way of building those. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The token type ids. """ if token_ids_1 is None: return len(token_ids_0) * [0] return [0] * len(token_ids_0) + [1] * len(token_ids_1) def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. This implementation does not add special tokens and this method should be overridden in a subclass. Args: token_ids_0 (`List[int]`): The first tokenized sequence. token_ids_1 (`List[int]`, *optional*): The second tokenized sequence. Returns: `List[int]`: The model input with special tokens. """ if token_ids_1 is None: return token_ids_0 return token_ids_0 + token_ids_1 @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def prepare_for_model( self, ids: List[int], pair_ids: Optional[List[int]] = None, add_special_tokens: bool = True, padding: Union[bool, str, PaddingStrategy] = False, truncation: Union[bool, str, TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_length: bool = False, verbose: bool = True, prepend_batch_axis: bool = False, **kwargs, ) -> BatchEncoding: """ Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids* different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return overflowing tokens. Such a combination of arguments will raise an error. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. """ # Backward compatibility for 'truncation_strategy', 'pad_to_max_length' padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies( padding=padding, truncation=truncation, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, verbose=verbose, **kwargs, ) pair = bool(pair_ids is not None) len_ids = len(ids) len_pair_ids = len(pair_ids) if pair else 0 if return_token_type_ids and not add_special_tokens: raise ValueError( "Asking to return token_type_ids while setting add_special_tokens to False " "results in an undefined behavior. Please set add_special_tokens to True or " "set return_token_type_ids to None." ) if ( return_overflowing_tokens and truncation_strategy == TruncationStrategy.LONGEST_FIRST and pair_ids is not None ): raise ValueError( "Not possible to return overflowing tokens for pair of sequences with the " "`longest_first`. Please select another truncation strategy than `longest_first`, " "for instance `only_second` or `only_first`." ) # Load from model defaults if return_token_type_ids is None: return_token_type_ids = "token_type_ids" in self.model_input_names if return_attention_mask is None: return_attention_mask = "attention_mask" in self.model_input_names encoded_inputs = {} # Compute the total size of the returned encodings total_len = len_ids + len_pair_ids + (self.num_special_tokens_to_add(pair=pair) if add_special_tokens else 0) # Truncation: Handle max sequence length overflowing_tokens = [] if truncation_strategy != TruncationStrategy.DO_NOT_TRUNCATE and max_length and total_len > max_length: ids, pair_ids, overflowing_tokens = self.truncate_sequences( ids, pair_ids=pair_ids, num_tokens_to_remove=total_len - max_length, truncation_strategy=truncation_strategy, stride=stride, ) if return_overflowing_tokens: encoded_inputs["overflowing_tokens"] = overflowing_tokens encoded_inputs["num_truncated_tokens"] = total_len - max_length # Add special tokens if add_special_tokens: sequence = self.build_inputs_with_special_tokens(ids, pair_ids) token_type_ids = self.create_token_type_ids_from_sequences(ids, pair_ids) else: sequence = ids + pair_ids if pair else ids token_type_ids = [0] * len(ids) + ([0] * len(pair_ids) if pair else []) # Build output dictionary encoded_inputs["input_ids"] = sequence if return_token_type_ids: encoded_inputs["token_type_ids"] = token_type_ids if return_special_tokens_mask: if add_special_tokens: encoded_inputs["special_tokens_mask"] = self.get_special_tokens_mask(ids, pair_ids) else: encoded_inputs["special_tokens_mask"] = [0] * len(sequence) # Check lengths self._eventual_warn_about_too_long_sequence(encoded_inputs["input_ids"], max_length, verbose) # Padding if padding_strategy != PaddingStrategy.DO_NOT_PAD or return_attention_mask: encoded_inputs = self.pad( encoded_inputs, max_length=max_length, padding=padding_strategy.value, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) if return_length: encoded_inputs["length"] = len(encoded_inputs["input_ids"]) batch_outputs = BatchEncoding( encoded_inputs, tensor_type=return_tensors, prepend_batch_axis=prepend_batch_axis ) return batch_outputs def truncate_sequences( self, ids: List[int], pair_ids: Optional[List[int]] = None, num_tokens_to_remove: int = 0, truncation_strategy: Union[str, TruncationStrategy] = "longest_first", stride: int = 0, ) -> Tuple[List[int], List[int], List[int]]: """ Truncates a sequence pair in-place following the strategy. Args: ids (`List[int]`): Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. pair_ids (`List[int]`, *optional*): Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize` and `convert_tokens_to_ids` methods. num_tokens_to_remove (`int`, *optional*, defaults to 0): Number of tokens to remove using the truncation strategy. truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`): The strategy to follow for truncation. Can be: - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). stride (`int`, *optional*, defaults to 0): If set to a positive number, the overflowing tokens returned will contain some tokens from the main sequence returned. The value of this argument defines the number of additional tokens. Returns: `Tuple[List[int], List[int], List[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair of sequences (or a batch of pairs) is provided. """ if num_tokens_to_remove <= 0: return ids, pair_ids, [] if not isinstance(truncation_strategy, TruncationStrategy): truncation_strategy = TruncationStrategy(truncation_strategy) overflowing_tokens = [] if truncation_strategy == TruncationStrategy.ONLY_FIRST or ( truncation_strategy == TruncationStrategy.LONGEST_FIRST and pair_ids is None ): if len(ids) > num_tokens_to_remove: window_len = min(len(ids), stride + num_tokens_to_remove) if self.truncation_side == "left": overflowing_tokens = ids[:window_len] ids = ids[num_tokens_to_remove:] elif self.truncation_side == "right": overflowing_tokens = ids[-window_len:] ids = ids[:-num_tokens_to_remove] else: raise ValueError(f"invalid truncation strategy: {self.truncation_side}, use 'left' or 'right'.") else: error_msg = ( f"We need to remove {num_tokens_to_remove} to truncate the input " f"but the first sequence has a length {len(ids)}. " ) if truncation_strategy == TruncationStrategy.ONLY_FIRST: error_msg = ( error_msg + "Please select another truncation strategy than " f"{truncation_strategy}, for instance 'longest_first' or 'only_second'." ) logger.error(error_msg) elif truncation_strategy == TruncationStrategy.LONGEST_FIRST: logger.warning( "Be aware, overflowing tokens are not returned for the setting you have chosen," " i.e. sequence pairs with the %s " "truncation strategy. So the returned list will always be empty even if some " "tokens have been removed.", TruncationStrategy.LONGEST_FIRST.value ) for _ in range(num_tokens_to_remove): if pair_ids is None or len(ids) > len(pair_ids): if self.truncation_side == "right": ids = ids[:-1] elif self.truncation_side == "left": ids = ids[1:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) else: if self.truncation_side == "right": pair_ids = pair_ids[:-1] elif self.truncation_side == "left": pair_ids = pair_ids[1:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) elif truncation_strategy == TruncationStrategy.ONLY_SECOND and pair_ids is not None: if len(pair_ids) > num_tokens_to_remove: window_len = min(len(pair_ids), stride + num_tokens_to_remove) if self.truncation_side == "right": overflowing_tokens = pair_ids[-window_len:] pair_ids = pair_ids[:-num_tokens_to_remove] elif self.truncation_side == "left": overflowing_tokens = pair_ids[:window_len] pair_ids = pair_ids[num_tokens_to_remove:] else: raise ValueError("invalid truncation strategy:" + str(self.truncation_side)) else: logger.error( "We need to remove %s to truncate the input but the second sequence has a length %s. " "Please select another truncation strategy than %s, for instance 'longest_first' or 'only_first'.", num_tokens_to_remove, len(pair_ids), truncation_strategy ) return (ids, pair_ids, overflowing_tokens) def _pad( self, encoded_inputs: Union[Dict[str, EncodedInput], BatchEncoding], max_length: Optional[int] = None, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, pad_to_multiple_of: Optional[int] = None, return_attention_mask: Optional[bool] = None, ) -> dict: """ Pad encoded inputs (on left/right and up to predefined length or max length in the batch) Args: encoded_inputs: Dictionary of tokenized inputs (`List[int]`) or batch of tokenized inputs (`List[List[int]]`). max_length: maximum length of the returned list and optionally padding length (see below). Will truncate by taking into account the special tokens. padding_strategy: PaddingStrategy to use for padding. - PaddingStrategy.LONGEST Pad to the longest sequence in the batch - PaddingStrategy.MAX_LENGTH: Pad to the max length (default) - PaddingStrategy.DO_NOT_PAD: Do not pad The tokenizer padding sides are defined in self.padding_side: - 'left': pads on the left of the sequences - 'right': pads on the right of the sequences pad_to_multiple_of: (optional) Integer if set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Core on NVIDIA hardware with compute capability `>= 7.5` (Volta). return_attention_mask: (optional) Set to False to avoid returning attention mask (default: set to model specifics) """ # Load from model defaults if return_attention_mask is None: return_attention_mask = "attention_mask" in self.model_input_names required_input = encoded_inputs[self.model_input_names[0]] if padding_strategy == PaddingStrategy.LONGEST: max_length = len(required_input) if max_length is not None and pad_to_multiple_of is not None and (max_length % pad_to_multiple_of != 0): max_length = ((max_length // pad_to_multiple_of) + 1) * pad_to_multiple_of needs_to_be_padded = padding_strategy != PaddingStrategy.DO_NOT_PAD and len(required_input) != max_length # Initialize attention mask if not present. if return_attention_mask and "attention_mask" not in encoded_inputs: encoded_inputs["attention_mask"] = [1] * len(required_input) if needs_to_be_padded: difference = max_length - len(required_input) if self.padding_side == "right": if return_attention_mask: encoded_inputs["attention_mask"] = encoded_inputs["attention_mask"] + [0] * difference if "token_type_ids" in encoded_inputs: encoded_inputs["token_type_ids"] = ( encoded_inputs["token_type_ids"] + [self.pad_token_type_id] * difference ) if "special_tokens_mask" in encoded_inputs: encoded_inputs["special_tokens_mask"] = encoded_inputs["special_tokens_mask"] + [1] * difference encoded_inputs[self.model_input_names[0]] = required_input + [self.pad_token_id] * difference elif self.padding_side == "left": if return_attention_mask: encoded_inputs["attention_mask"] = [0] * difference + encoded_inputs["attention_mask"] if "token_type_ids" in encoded_inputs: encoded_inputs["token_type_ids"] = [self.pad_token_type_id] * difference + encoded_inputs[ "token_type_ids" ] if "special_tokens_mask" in encoded_inputs: encoded_inputs["special_tokens_mask"] = [1] * difference + encoded_inputs["special_tokens_mask"] encoded_inputs[self.model_input_names[0]] = [self.pad_token_id] * difference + required_input else: raise ValueError("Invalid padding strategy:" + str(self.padding_side)) return encoded_inputs def convert_tokens_to_string(self, tokens: List[str]) -> str: """ Converts a sequence of tokens in a single string. The most simple way to do it is `" ".join(tokens)` but we often want to remove sub-word tokenization artifacts at the same time. Args: tokens (`List[str]`): The token to join in a string. Returns: `str`: The joined tokens. """ raise NotImplementedError def batch_decode( self, sequences: Union[List[int], List[List[int]], "np.ndarray", "ms.Tensor"], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> List[str]: """ Convert a list of lists of token ids into a list of strings by calling decode. Args: sequences (`Union[List[int], List[List[int]], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `List[str]`: The list of decoded sentences. """ return [ self.decode( seq, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) if seq else "" for seq in sequences ] def decode( self, token_ids: Union[int, List[int], "np.ndarray", "ms.Tensor"], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: """ Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special tokens and clean up tokenization spaces. Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`. Args: token_ids (`Union[int, List[int], np.ndarray, ms.Tensor]`): List of tokenized input ids. Can be obtained using the `__call__` method. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. clean_up_tokenization_spaces (`bool`, *optional*): Whether or not to clean up the tokenization spaces. If `None`, will default to `self.clean_up_tokenization_spaces`. kwargs (additional keyword arguments, *optional*): Will be passed to the underlying model specific decode method. Returns: `str`: The decoded sentence. """ # Convert inputs to python lists token_ids = to_py_obj(token_ids) if isinstance(token_ids[0], list): output = [] for item in token_ids: new_strs = self._decode( token_ids=item, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) output.append(new_strs) else: output = self._decode( token_ids=token_ids, skip_special_tokens=skip_special_tokens, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs) return output def _decode( self, token_ids: Union[int, List[int]], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: raise NotImplementedError def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ assert already_has_special_tokens and token_ids_1 is None, ( "You cannot use ``already_has_special_tokens=False`` with this tokenizer. " "Please use a slow (full python) tokenizer to activate this argument. " "Or set `return_special_tokens_mask=True` when calling the encoding method " "to get the special tokens mask in any tokenizer. " ) all_special_ids = self.all_special_ids # cache the property special_tokens_mask = [1 if token in all_special_ids else 0 for token in token_ids_0] return special_tokens_mask @staticmethod def clean_up_tokenization(out_string: str) -> str: """ Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms. Args: out_string (`str`): The text to clean up. Returns: `str`: The cleaned-up string. """ out_string = ( out_string.replace(" .", ".") .replace(" ?", "?") .replace(" !", "!") .replace(" ,", ",") .replace(" ' ", "'") .replace(" n't", "n't") .replace(" 'm", "'m") .replace(" 's", "'s") .replace(" 've", "'ve") .replace(" 're", "'re") ) return out_string def _eventual_warn_about_too_long_sequence(self, ids: List[int], max_length: Optional[int], verbose: bool): """ Depending on the input and internal state we might trigger a warning about a sequence that is too long for its corresponding model Args: ids (`List[str]`): The ids produced by the tokenization max_length (`int`, *optional*): The max_length desired (does not trigger a warning if it is set) verbose (`bool`): Whether or not to print more information and warnings. """ if max_length is None and len(ids) > self.model_max_length and verbose: if not self.deprecation_warnings.get("sequence-length-is-longer-than-the-specified-maximum", False): logger.warning( "Token indices sequence length is longer than the specified maximum sequence length " "for this model (%s > %s). Running this sequence through the model " "will result in indexing errors", len(ids), self.model_max_length ) self.deprecation_warnings["sequence-length-is-longer-than-the-specified-maximum"] = True def _switch_to_input_mode(self): """ Private method to put the tokenizer in input mode (when it has different modes for input/outputs) """ def _switch_to_target_mode(self): """ Private method to put the tokenizer in target mode (when it has different modes for input/outputs) """ @contextmanager def as_target_tokenizer(self): """ Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to sequence-to-sequence models that need a slightly different processing for the labels. """ warnings.warn( "`as_target_tokenizer` is deprecated and will be removed in v5 of Transformers. You can tokenize your " "labels by using the argument `text_target` of the regular `__call__` method (either in the same call as " "your input texts if you use the same keyword arguments, or in a separate call." ) self._switch_to_target_mode() self._in_target_context_manager = True yield self._in_target_context_manager = False self._switch_to_input_mode() def prepare_seq2seq_batch( self, src_texts: List[str], tgt_texts: Optional[List[str]] = None, max_length: Optional[int] = None, max_target_length: Optional[int] = None, padding: str = "longest", return_tensors: str = None, truncation: bool = True, **kwargs, ) -> BatchEncoding: """ Prepare model inputs for translation. For best performance, translate one sentence at a time. Arguments: src_texts (`List[str]`): List of documents to summarize or source language texts. tgt_texts (`list`, *optional*): List of summaries or target language texts. max_length (`int`, *optional*): Controls the maximum length for encoder inputs (documents to summarize or source language texts) If left unset or set to `None`, this will use the predefined model maximum length if a maximum length is required by one of the truncation/padding parameters. If the model has no specific maximum input length (like XLNet) truncation/padding to a maximum length will be deactivated. max_target_length (`int`, *optional*): Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set to `None`, this will use the max_length value. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`): Activates and controls padding. Accepts the following values: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'np'`: Return Numpy `np.ndarray` objects. - `'ms'`: Return PyTorch `ms.Tensor` objects. truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`): Activates and controls truncation. Accepts the following values: - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will truncate token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a batch of pairs) is provided. - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. This will only truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided. - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater than the model maximum admissible input size). **kwargs: Additional keyword arguments passed along to `self.__call__`. Return: [`BatchEncoding`]: A [`BatchEncoding`] with the following fields: - **input_ids** -- List of token ids to be fed to the encoder. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model. - **labels** -- List of token ids for tgt_texts. The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed. Otherwise, input_ids, attention_mask will be the only keys. """ # docstyle-ignore formatted_warning = """ `prepare_seq2seq_batch` is deprecated and will be removed in version 5 of HuggingFace Transformers. Use the regular `__call__` method to prepare your inputs and targets. Here is a short example: model_inputs = tokenizer(src_texts, text_target=tgt_texts, ...) If you either need to use different keyword arguments for the source and target texts, you should do two calls like this: model_inputs = tokenizer(src_texts, ...) labels = tokenizer(text_target=tgt_texts, ...) model_inputs["labels"] = labels["input_ids"] See the documentation of your specific tokenizer for more details on the specific arguments to the tokenizer of choice. For a more complete example, see the implementation of `prepare_seq2seq_batch`. """ warnings.warn(formatted_warning, FutureWarning) # mBART-specific kwargs that should be ignored by other models. kwargs.pop("src_lang", None) kwargs.pop("tgt_lang", None) if max_length is None: max_length = self.model_max_length model_inputs = self( src_texts, add_special_tokens=True, return_tensors=return_tensors, max_length=max_length, padding=padding, truncation=truncation, **kwargs, ) if tgt_texts is None: return model_inputs # Process tgt_texts if max_target_length is None: max_target_length = max_length with self.as_target_tokenizer(): labels = self( tgt_texts, add_special_tokens=True, return_tensors=return_tensors, padding=padding, max_length=max_target_length, truncation=truncation, **kwargs, ) model_inputs["labels"] = labels["input_ids"] return model_inputs @add_end_docstrings(INIT_TOKENIZER_DOCSTRING) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class Tokenizer(BaseTokenizer): """ Base class for all slow tokenizers. Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handle all the shared methods for tokenization and special tokens as well as methods downloading/caching/loading pretrained tokenizers as well as adding tokens to the vocabulary. This class also contain the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). """ _support_list = [] def __init__(self, **kwargs): super().__init__(**kwargs) # Added tokens - We store this for both slow and fast tokenizers # until the serialization of Fast tokenizers is updated self.added_tokens_encoder: Dict[str, int] = {} self.added_tokens_decoder: Dict[int, str] = {} self.unique_no_split_tokens: List[str] = [] self.tokens_trie = Trie() self._decode_use_source_tokenizer = False @property def is_fast(self) -> bool: return False @property def vocab_size(self) -> int: """ `int`: Size of the base vocabulary (without the added tokens). """ raise NotImplementedError def get_added_vocab(self) -> Dict[str, int]: """ Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. """ return self.added_tokens_encoder def __len__(self): """ Size of the full vocabulary with the added tokens. """ return self.vocab_size + len(self.added_tokens_encoder) def _add_tokens(self, new_tokens: Union[List[str], List[AddedToken]], special_tokens: bool = False) -> int: """ Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to it with indices starting from length of the current vocabulary. Args: new_tokens (`List[str]`or `List[tokenizers.AddedToken]`): Token(s) to add in vocabulary. A token is only added if it's not already in the vocabulary (tested by checking if the tokenizer assign the index of the `unk_token` to them). special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the tokens should be added as special tokens. Returns: `int`: The number of tokens actually added to the vocabulary. Examples: ```python # Let's see how to increase the vocabulary of Bert model and tokenizer tokenizer = BertTokenizer.from_pretrained("bert-base-uncased") model = BertModel.from_pretrained("bert-base-uncased") num_added_toks = tokenizer.add_tokens(["new_tok1", "my_new-tok2"]) print("We have added", num_added_toks, "tokens") # Note: resize_token_embeddings expects to receive the full size of the new vocabulary, i.e. the length of the tokenizer. model.resize_token_embeddings(len(tokenizer)) ```""" new_tokens = [str(tok) for tok in new_tokens] tokens_to_add = [] for token in new_tokens: if not isinstance(token, str): raise TypeError(f"Token {token} is not a string but a {type(token)}.") if not special_tokens and hasattr(self, "do_lower_case") and self.do_lower_case: token = token.lower() if ( token != self.unk_token and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token) and token not in tokens_to_add ): tokens_to_add.append(token) if self.verbose: logger.info("Adding %s to the vocabulary", token) added_tok_encoder = {tok: len(self) + i for i, tok in enumerate(tokens_to_add)} added_tok_decoder = {v: k for k, v in added_tok_encoder.items()} self.added_tokens_encoder.update(added_tok_encoder) self.added_tokens_decoder.update(added_tok_decoder) # Make sure we don't split on any special tokens (even they were already in the vocab before e.g. for Albert) if special_tokens: if len(new_tokens) == 1: _insert_one_token_to_ordered_list(self.unique_no_split_tokens, new_tokens[0]) else: self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(new_tokens))) else: # Or on the newly added tokens if len(tokens_to_add) == 1: _insert_one_token_to_ordered_list(self.unique_no_split_tokens, tokens_to_add[0]) else: self.unique_no_split_tokens = sorted(set(self.unique_no_split_tokens).union(set(tokens_to_add))) self._create_trie(self.unique_no_split_tokens) return len(tokens_to_add) def _create_trie(self, unique_no_split_tokens): trie = Trie() for token in unique_no_split_tokens: if hasattr(self, "do_lower_case") and self.do_lower_case and token not in self.all_special_tokens: trie.add(token.lower()) else: trie.add(token) self.tokens_trie = trie def num_special_tokens_to_add(self, pair: bool = False) -> int: """ Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. """ token_ids_0 = [] token_ids_1 = [] return len(self.build_inputs_with_special_tokens(token_ids_0, token_ids_1 if pair else None)) def tokenize( self, text: TextInput, pair: Optional[str] = None, add_special_tokens: bool = True, **kwargs ) -> List[str]: """ Converts a string in a sequence of tokens, using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Takes care of added tokens. Args: text (`str`): The sequence to be encoded. **kwargs (additional keyword arguments): Passed along to the model-specific `prepare_for_tokenization` preprocessing method. Returns: `List[str]`: The list of tokens. """ # Simple mapping string => AddedToken for special tokens with specific tokenization behaviors all_special_tokens_extended = { str(t): t for t in self.all_special_tokens_extended if isinstance(t, AddedToken) } text, kwargs = self.prepare_for_tokenization(text, **kwargs) if kwargs: logger.warning("Keyword arguments %s not recognized.", kwargs) # TODO: should this be in the base class? if hasattr(self, "do_lower_case") and self.do_lower_case: # convert non-special tokens to lowercase escaped_special_toks = [ re.escape(s_tok) for s_tok in (self.unique_no_split_tokens + self.all_special_tokens) ] pattern = r"(" + r"|".join(escaped_special_toks) + r")|" + r"(.+?)" text = re.sub(pattern, lambda m: m.groups()[0] or m.groups()[1].lower(), text) no_split_token = set(self.unique_no_split_tokens) tokens = self.tokens_trie.split(text) # ["This is something", "", " else"] for i, token in enumerate(tokens): if token in no_split_token: tok_extended = all_special_tokens_extended.get(token, None) left = tokens[i - 1] if i > 0 else None right = tokens[i + 1] if i < len(tokens) - 1 else None if isinstance(tok_extended, AddedToken): if tok_extended.rstrip and right: # A bit counter-intuitive but we strip the left of the string # since tok_extended.rstrip means the special token is eating all white spaces on its right tokens[i + 1] = right.lstrip() # Strip white spaces on the left if tok_extended.lstrip and left: tokens[i - 1] = left.rstrip() # Opposite here else: # We strip left and right by default if right: tokens[i + 1] = right.lstrip() if left: tokens[i - 1] = left.rstrip() # ["This is something", "", "else"] tokenized_text = [] for token in tokens: # Need to skip eventual empty (fully stripped) tokens if not token: continue if token in no_split_token: tokenized_text.append(token) else: tokenized_text.extend(self._tokenize(token)) # ["This", " is", " something", "", "else"] return tokenized_text def _tokenize(self, text, **kwargs): """ Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces). Do NOT take care of added tokens. """ raise NotImplementedError def convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ if tokens is None: return None if isinstance(tokens, str): return self._convert_token_to_id_with_added_voc(tokens) ids = [] for token in tokens: ids.append(self._convert_token_to_id_with_added_voc(token)) return ids def _convert_token_to_id_with_added_voc(self, token): if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self._convert_token_to_id(token) def _convert_token_to_id(self, token): raise NotImplementedError def _encode_plus( self, text: Union[TextInput, PreTokenizedInput, EncodedInput], text_pair: Optional[Union[TextInput, PreTokenizedInput, EncodedInput]] = None, add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: def get_input_ids(text): if isinstance(text, str): tokens = self.tokenize(text, **kwargs) return self.convert_tokens_to_ids(tokens) if isinstance(text, (list, tuple)) and text and isinstance(text[0], str): if is_split_into_words: tokens = list( itertools.chain(*(self.tokenize(t, is_split_into_words=True, **kwargs) for t in text)) ) return self.convert_tokens_to_ids(tokens) return self.convert_tokens_to_ids(text) if isinstance(text, (list, tuple)) and text and isinstance(text[0], int): return text if is_split_into_words: raise ValueError( f"Input {text} is not valid. Should be a string or a list/tuple of strings when" " `is_split_into_words=True`." ) raise ValueError( f"Input {text} is not valid. Should be a string, a list/tuple of strings or a list/tuple of" " integers." ) if return_offsets_mapping: raise NotImplementedError( "return_offset_mapping is not available when using Python tokenizers. " "To use this feature, change your tokenizer to one deriving from " "transformers.PreTrainedTokenizerFast. " "More information on available tokenizers at " "https://github.com/huggingface/transformers/pull/2674" ) first_ids = get_input_ids(text) second_ids = get_input_ids(text_pair) if text_pair is not None else None return self.prepare_for_model( first_ids, pair_ids=second_ids, add_special_tokens=add_special_tokens, padding=padding_strategy.value, truncation=truncation_strategy.value, max_length=max_length, stride=stride, pad_to_multiple_of=pad_to_multiple_of, return_tensors=return_tensors, prepend_batch_axis=True, return_attention_mask=return_attention_mask, return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, verbose=verbose, ) def _batch_encode_plus( self, batch_text_or_text_pairs: Union[ List[TextInput], List[TextInputPair], List[PreTokenizedInput], List[PreTokenizedInputPair], List[EncodedInput], List[EncodedInputPair], ], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[Union[str, TensorType]] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs, ) -> BatchEncoding: def get_input_ids(text): if isinstance(text, str): tokens = self.tokenize(text, **kwargs) return self.convert_tokens_to_ids(tokens) if isinstance(text, (list, tuple)) and text and isinstance(text[0], str): if is_split_into_words: tokens = list( itertools.chain(*(self.tokenize(t, is_split_into_words=True, **kwargs) for t in text)) ) return self.convert_tokens_to_ids(tokens) return self.convert_tokens_to_ids(text) if isinstance(text, (list, tuple)) and text and isinstance(text[0], int): return text raise ValueError( "Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers." ) if return_offsets_mapping: raise NotImplementedError( "return_offset_mapping is not available when using Python tokenizers. " "To use this feature, change your tokenizer to one deriving from " "transformers.PreTrainedTokenizerFast." ) input_ids = [] for ids_or_pair_ids in batch_text_or_text_pairs: if not isinstance(ids_or_pair_ids, (list, tuple)): ids, pair_ids = ids_or_pair_ids, None elif is_split_into_words and not isinstance(ids_or_pair_ids[0], (list, tuple)): ids, pair_ids = ids_or_pair_ids, None else: ids, pair_ids = ids_or_pair_ids first_ids = get_input_ids(ids) second_ids = get_input_ids(pair_ids) if pair_ids is not None else None input_ids.append((first_ids, second_ids)) batch_outputs = self._batch_prepare_for_model( input_ids, add_special_tokens=add_special_tokens, padding_strategy=padding_strategy, truncation_strategy=truncation_strategy, max_length=max_length, stride=stride, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, return_tensors=return_tensors, verbose=verbose, ) return BatchEncoding(batch_outputs) @add_end_docstrings(ENCODE_KWARGS_DOCSTRING, ENCODE_PLUS_ADDITIONAL_KWARGS_DOCSTRING) def _batch_prepare_for_model( self, batch_ids_pairs: List[Union[PreTokenizedInputPair, Tuple[List[int], None]]], add_special_tokens: bool = True, padding_strategy: PaddingStrategy = PaddingStrategy.DO_NOT_PAD, truncation_strategy: TruncationStrategy = TruncationStrategy.DO_NOT_TRUNCATE, max_length: Optional[int] = None, stride: int = 0, pad_to_multiple_of: Optional[int] = None, return_tensors: Optional[str] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_length: bool = False, verbose: bool = True, ) -> BatchEncoding: """ Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It adds special tokens, truncates sequences if overflowing while taking into account the special tokens and manages a moving window (with user defined stride) for overflowing tokens Args: batch_ids_pairs: list of tokenized input ids or input ids pairs """ batch_outputs = {} for first_ids, second_ids in batch_ids_pairs: outputs = self.prepare_for_model( first_ids, second_ids, add_special_tokens=add_special_tokens, padding=PaddingStrategy.DO_NOT_PAD.value, # we pad in batch afterward truncation=truncation_strategy.value, max_length=max_length, stride=stride, pad_to_multiple_of=None, # we pad in batch afterward return_attention_mask=False, # we pad in batch afterward return_token_type_ids=return_token_type_ids, return_overflowing_tokens=return_overflowing_tokens, return_special_tokens_mask=return_special_tokens_mask, return_length=return_length, return_tensors=None, # We convert the whole batch to tensors at the end prepend_batch_axis=False, verbose=verbose, ) for key, value in outputs.items(): if key not in batch_outputs: batch_outputs[key] = [] batch_outputs[key].append(value) batch_outputs = self.pad( batch_outputs, padding=padding_strategy.value, max_length=max_length, pad_to_multiple_of=pad_to_multiple_of, return_attention_mask=return_attention_mask, ) batch_outputs = BatchEncoding(batch_outputs, tensor_type=return_tensors) return batch_outputs def prepare_for_tokenization( self, text: str, **kwargs ) -> Tuple[str, Dict[str, Any]]: """ Performs any necessary transformations before tokenization. This method should pop the arguments from kwargs and return the remaining `kwargs` as well. We test the `kwargs` at the end of the encoding process to be sure all the arguments have been used. Args: text (`str`): The text to prepare. is_split_into_words (`bool`, *optional*, defaults to `False`): Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace) which it will tokenize. This is useful for NER or token classification. kwargs: Keyword arguments to use for the tokenization. Returns: `Tuple[str, Dict[str, Any]]`: The prepared text and the unused kwargs. """ return (text, kwargs) def get_special_tokens_mask( self, token_ids_0: List, token_ids_1: Optional[List] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of ids of the first sequence. token_ids_1 (`List[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: if token_ids_1 is not None: raise ValueError( "You should not supply a second sequence if the provided sequence of " "ids is already formatted with special tokens for the model." ) return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) return [0] * ((len(token_ids_1) if token_ids_1 else 0) + len(token_ids_0)) def convert_ids_to_tokens( self, ids: Union[int, List[int]], skip_special_tokens: bool = False ) -> Union[str, List[str]]: """ Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). """ if isinstance(ids, int): if ids in self.added_tokens_decoder: return self.added_tokens_decoder[ids] if ids >= self.vocab_size: raise IndexError(f"The token id {ids} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") return self._convert_id_to_token(ids) tokens = [] for index in ids: index = int(index) if skip_special_tokens and index in self.all_special_ids: continue if index in self.added_tokens_decoder: tokens.append(self.added_tokens_decoder[index]) else: if index >= self.vocab_size: raise IndexError( f"The token id {index} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") tokens.append(self._convert_id_to_token(index)) return tokens def _convert_id_to_token(self, index: int) -> str: raise NotImplementedError def convert_tokens_to_string(self, tokens: List[str]) -> str: return " ".join(tokens) def _decode( self, token_ids: List[int], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs, ) -> str: self._decode_use_source_tokenizer = kwargs.pop("use_source_tokenizer", False) spaces_between_special_tokens = kwargs.pop("spaces_between_special_tokens", False) filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens) # To avoid mixing byte-level and unicode for byte-level BPT # we need to build string separately for added tokens and byte-level tokens # cf. https://github.com/huggingface/transformers/issues/1133 sub_texts = [] current_sub_text = [] for token in filtered_tokens: if skip_special_tokens and token in self.all_special_ids: continue if token in self.added_tokens_encoder: if current_sub_text: sub_texts.append(self.convert_tokens_to_string(current_sub_text)) current_sub_text = [] sub_texts.append(token) else: current_sub_text.append(token) if current_sub_text: sub_texts.append(self.convert_tokens_to_string(current_sub_text)) if spaces_between_special_tokens: text = " ".join(sub_texts) else: text = "".join(sub_texts) clean_up_tokenization_spaces = ( clean_up_tokenization_spaces if clean_up_tokenization_spaces is not None else self.clean_up_tokenization_spaces ) if clean_up_tokenization_spaces: clean_text = self.clean_up_tokenization(text) return clean_text return text @classmethod def show_support_list(cls): """show_support_list method""" logger.info("support list of %s is:", cls.__name__) print_path_or_list(cls._support_list) @classmethod def get_support_list(cls): """get_support_list method""" return cls._support_list pengcheng.mind-mindformer/mindformers/models/bert/000077500000000000000000000000001476650141400226565ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/bert/__init__.py000066400000000000000000000021101476650141400247610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert""" from .bert_tokenizer import BertTokenizer, BasicTokenizer from .bert_config import BertConfig from .bert import ( BertForPreTraining, BertModel, BertForTokenClassification, BertForMultipleChoice, BertForQuestionAnswering) from .bert_processor import BertProcessor __all__ = [] __all__.extend(bert_tokenizer.__all__) __all__.extend(bert.__all__) __all__.extend(bert_processor.__all__) pengcheng.mind-mindformer/mindformers/models/bert/__pycache__/000077500000000000000000000000001476650141400250665ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/bert/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011561476650141400312610ustar00rootroot00000000000000a ?cfH@stdZddlmZmZddlmZddlmZmZm Z m Z m Z ddl m Z gZeejeejee jdS)zThe export function for bert) BertTokenizerBasicTokenizer) BertConfig)BertForPreTraining BertModelBertForTokenClassificationBertForMultipleChoiceBertForQuestionAnswering) BertProcessorN)__doc__Zbert_tokenizerrrZ bert_configrbertrrrrr Zbert_processorr __all__extendrrK/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/__init__.pys    pengcheng.mind-mindformer/mindformers/models/bert/__pycache__/bert.cpython-39.pyc000066400000000000000000000557111476650141400304640ustar00rootroot00000000000000a ?cf@sdZddlZddlZddlmmZddlm Z ddl m Z ddl m m ZddlmZmZddlmZmZddl mZddlmZddlmZmZddlmZdd lmZmZdd l m!Z!dd l"m#Z#dd l$m%Z%gd Z&e'ej(Gddde!Z)e'ej(Gddde!Z*Gddde j+Z,e'ej(Gddde!Z-e'ej(Gddde!Z.Gddde j+Z/Gddde j+Z0Gddde j+Z1Gddde j+Z2Gd d!d!e j+Z3Gd"d#d#e j+Z4dS)$z BERT Model.N)Tensor Parameter)TruncatedNormal initializer) operations) LayerNorm) TransformerVocabEmbedding)default_moe_config)MindFormerRegisterMindFormerModuleType) BaseModel)MindFormerBook) BertConfig)r BertModelBertForPreTrainingBertForTokenClassificationBertForMultipleChoiceBertForQuestionAnsweringcs@eZdZdZeddZeffdd Zd ddZ Z S) ru Bert with dense layer for name entity recoginition task. Args: config (BertConfig): The config of BertForTokenClassification. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForTokenClassification, BertTokenizer >>> model = BertForTokenClassification.from_pretrained('tokcls_bert_base_chinese') >>> tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') >>> data = tokenizer(["我在杭州华为工作。"], return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[[ 0.0886748 -0.23066735 -0.03969013 ... 0.07333283 -0.02968273 0.06125224] [-0.03115104 -0.47599065 0.01928361 ... 0.22821501 0.42415133 0.14563856] [ 0.05570185 -0.36168078 0.00884905 ... -0.15357454 -0.33119604 -0.09889631] ... [ 0.01028904 -0.20403272 -0.1117363 ... -0.03601318 0.034153 0.12492958] [-0.00420664 -0.1786235 -0.11332294 ... -0.07119732 0.02407441 0.14775723] [-0.04674841 -0.1608555 -0.0781534 ... -0.06064402 0.0439388 0.20419256]]] tokclsbertcsxtt|||j|_t||j|j|_t d|j |_ t |j |j|j|_t|_t|_||dSN)superr__init__ num_labelsr is_traininguse_one_hot_embeddingsrnnDropouthidden_dropout_probdropoutDense hidden_sizeto_float compute_dtype classifierCrossEntropyLosscross_entropy_lossPReshapereshapeload_checkpointselfconfig __class__G/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert.pyrLs  z#BertForTokenClassification.__init__Nc Csx|j|||d}|d}||}||}|jr^||d|jf}||d}|||}n|durl|}n||f}|S)Get Training Loss or Logits input_ids input_masktoken_type_idsrr9Nrr!r&trainingr+rr() r.r6r7r8 label_ids bert_outputssequence_outputlogitsoutputr2r2r3 constructVs   z$BertForTokenClassification.construct)N __name__ __module__ __qualname____doc__rget_model_support_list _support_listrrrB __classcell__r2r2r0r3r&s" rcsFeZdZdZedZeffdd Zd ddZ d dd Z Z S) ra Provide bert pre-training loss through network. Args: config (BertConfig): The config of BertForPreTraining. is_training (bool): Specifies whether to use the training mode. use_one_hot_embeddings (bool): Specifies whether to use one-hot for embeddings. Default: False. Returns: Tensor, the loss of the network. Examples: >>> from mindspore import Tensor >>> import mindspore.common.dtype as mstype >>> from mindformers import BertForPreTraining, BertTokenizer >>> model = BertForPreTraining.from_pretrained('bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') >>> data = tokenizer(["Paris is the [MASK] of France."], ... return_tensors='ms', max_length=128, padding="max_length") >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> masked_lm_positions = Tensor([[4]], mstype.int32) >>> next_sentence_labels = Tensor([[1]], mstype.int32) >>> masked_lm_weights = Tensor([[1]], mstype.int32) >>> masked_lm_ids = Tensor([[3007]], mstype.int32) >>> output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> print(output) [0.6706] rcstt|||j|_t||j|j|_t||_t ||_ t ||_ t |_|jjjdk|_t d|_||dS)Nrrr2)rrrrrrrGetMaskedLMOutputmlmlossGetNextSentenceOutputnsplossBertLosslossr)Castcastparallel_config moe_config expert_numuse_moeAddshardaddr,r-r0r2r3rs    zBertForPreTraining.__init__Nc Csjd}|jr"||||\}}}}n||||\}}}||||} ||} |js`||| | fS| | |fS)zconnect backbone and heads.r)rXrrNrPr<) r.r6r7 token_type_idmasked_lm_positionsmoe_lossr? pooled_outputembedding_tableprediction_scoresseq_relationship_scorer2r2r3 bert_forwards    zBertForPreTraining.bert_forwardc Cs^|js|||||S|||||\}} } ||| |||} |jrP|| | } || tjS)zGet pre-training loss)r<rcrRrXr[rTmstypefloat32) r.r6r7r\next_sentence_labelsr] masked_lm_idsmasked_lm_weightsrarbr^ total_lossr2r2r3rBs  zBertForPreTraining.construct)N)NNNN) rDrErFrGrrHrIrrrcrBrJr2r2r0r3rjs  rcs*eZdZdZdfdd ZddZZS) ra2 Bidirectional Encoder Representations from Transformers. Args: config (Class): Configuration for BertModel. is_training (bool): True for training mode. False for eval mode. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. NFcstt||sd|_d|_t||_|j|_|j|_|j|_ d|_ t |j dsZt |j _|j j}|j jjdk|_t|j|jtd|j|jgtjd|j jd|_d|j|j g}t||j ||jd|j||j|jd |_t|j|j|j |j|j|j!|j|j d ||j"|j#|j$|j%|j|jd |_&t'(|_)|j*|_*t+|j"d |_,t'-.d |_/t'j0dd|_1t2j3|j|jdt4|j5d6|j"|_7t'8|_9dS)NrVrtruncatedNormaldtype vocab_sizeembedding_size param_initrUr9T)rpembedding_shapeuse_relative_positionsuse_token_typetoken_type_vocab_sizermax_position_embeddings dropout_probr)r# batch_sizeffn_hidden_sizesrc_seq_lengthtgt_seq_length num_headsencoder_layersrUdecoder_layersrVparam_init_typelayernorm_compute_typesoftmax_compute_typepost_layernorm_residualattention_dropout_ratehidden_dropout_rate)dst_type)rrraxistanh) activation weight_init):rrrr attention_probs_dropout_prob CreateAttentionMaskFromInputMaskget_attention_maskr#num_hidden_layersrpr8hasattrrUr rVrWrXr rorrdreembedding_dp_mp_configword_embedding seq_lengthEmbeddingPostprocessorrstype_vocab_sizervembedding_postprocessorrrxintermediate_sizenum_attention_headsr%layernorm_dtype softmax_dtyper bert_encoderr)rSrTrm SaturateCastcast_compute_type StridedSlicerZsliceSqueeze squeeze_1rr"rinitializer_ranger$densePrintprint)r.r/rrrVZoutput_embedding_shaper0r2r3rs|      zBertModel.__init__cCs||\}}|||}t||j}||}||||}|d} t|d} | | d| d|j fd} | | } | | } | | |j} d}|jr|d}| | ||fS| | |fS)z8Bidirectional Encoder Representations from Transformers.r)rrrrrr9)rrr)rSrmrrrShaperr#rrrTrX)r.r6r8r7word_embeddingsZembedding_tablesZembedding_outputattention_maskencoder_outputr?rxsequence_slice first_tokenr_r^r2r2r3rBs(      zBertModel.construct)NFFrDrErFrGrrBrJr2r2r0r3rs  Drcs@eZdZdZeddZeffdd Zd ddZ Z S) ra Bert with dense layer for txt classification task. Args: config (BertConfig): The config of BertForMultipleChoice. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForMultipleChoice, BertTokenizer >>> model = BertForMultipleChoice.from_pretrained('txtcls_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('txtcls_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[0.05121157 0.3273056 0.31614417]] txtclsrcstt|||j|_t||j|j|_t|j |_ t j |j |j|j dd|j|_t d|j|_t |_t|_t|_|j|_|j|_||dS)NTrhas_biasr)rrrrrrrrrrrrr"r#r$r%r&rrwr!r'r(r)rSrTr*r+rmr,r-r0r2r3rEs     zBertForMultipleChoice.__init__Nc Cs||||\}}}||}||}|jrZ||d|jf}||d}|||}n"|durh|}n||d}||f}|S)r4r9r:Nr;) r.r6r7r\r=_r_r@rAr2r2r3rBTs    zBertForMultipleChoice.construct)NrCr2r2r0r3r,srcs@eZdZdZeddZeffdd Zd ddZ Z S) ra Bert with dense layer for question answering task. Args: config (BertConfig): The config of BertModel. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForQuestionAnswering, BertTokenizer >>> model = BertForQuestionAnswering.from_pretrained('qa_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=384, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) (Tensor(shape=[1, 384], dtype=Float32, value= [[ 5.30713797e-02, 1.08105361e-01, 1.55855455e-02 ... -3.73562761e-02, -1.89271923e-02, -2.99206823e-02]]), Tensor(shape=[1, 384], dtype=Float32, value= [[ 2.39873882e-02, 2.35675126e-01, 1.52210891e-01 ... -2.11214088e-02, 4.84664962e-02, -4.52579707e-02]])) qarcsXtt||t||j|j|_t|j d |j |_ | |tjddd|_dS)Nr9) output_numr)rrrrrrrrr"r#r$r% qa_outputsr,r)Splitsplitr-r0r2r3rs  z!BertForQuestionAnswering.__init__NcCs|j|||d}|d}||} || \} } | d} | d} |jrt|jdkrb|d}t|jdkrz|d}| jd} t|d| }t|d| }t j | d} | | |}| | |}||d}|}n || | f}|S)r4r5rr9r) ignore_indexr) rrrsqueezer<lenshapeops clip_by_valuerr')r.r6r7r\start_position end_position unique_idr>r?r@ start_logits end_logitsZ ignored_indexZloss_fctZ start_lossZend_lossrirAr2r2r3rBs*           z"BertForQuestionAnswering.construct)NNNrCr2r2r0r3rhsrcs*eZdZdZd fdd Zdd ZZS) rab Postprocessors apply positional and token type embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. embedding_shape (list): [batch_size, seq_length, embedding_size], the shape of each embedding vector. use_token_type (bool): Specifies whether to use token type embeddings. Default: False. token_type_vocab_size (int): Size of token type vocab. Default: 16. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 512. dropout_prob (float): The dropout probability. Default: 0.1. F皙?c stt|||_||_||_||_t||td||gt j d|j j d|_ d|_t|_tdt j |_tdt j |_t|_t|_t||_td| |_t|_||_ t!"d|_#|j\} } } t|j|td|j|gt j d|j j d|_$t%|f&t j |_'|j'"|j j(ddfftt)*| d | +t)j,|_-t."|j j(ddf|j j(ddff|_/t!"d|_#dS) Nrkrlrnr:?rr)rrr9)0rrrrtrurrvr rrdrerUrtoken_type_embeddingZ shape_flatr)OneHotone_hotron_value off_valueMatMul array_mulr*r+tuplerrrr!GathergatherrsrrZrfull_position_embeddingrr$ layernorm data_parallelnparangeastypeint32 position_idsrYr[) r.r/rprrrsrtrurrvrwrseqr0r2r3rsT        zEmbeddingPostprocessor.__init__c Cs|}|jr$||\}}|||}|jsxt|}t|j}||jd|d|dfd}||\} }||| }| |}| |}|S)zMPostprocessors apply positional and token type embeddings to word embeddings.)rrrrr) rtrr[rsFrrrrrr!) r.r8rrAZtoken_type_embeddingsrrZshape_positionrZposition_embeddingsr2r2r3rBs      z EmbeddingPostprocessor.construct)FFrFrrrr2r2r0r3rs/rcs.eZdZdZejffdd ZddZZS)ra Performs a safe saturating cast. This operation applies proper clamping before casting to prevent the danger that the value will overflow or underflow. Args: src_type (:class:`mindspore.dtype`): The type of the elements of the input tensor. Default: mstype.float32. dst_type (:class:`mindspore.dtype`): The type of the elements of the output tensor. Default: mstype.float32. csdtt|t|}tt|j|_ tt|j |_ t |_t |_t |_||_dSN)rrrrddtype_to_nptypefloatrfinfomintensor_min_typemaxtensor_max_typer)Minimummin_opMaximummax_oprSrTr)r.rnp_typer0r2r3rs    zSaturateCast.__init__cCs*|||j}|||j}|||jSr)rrrrrTr)r.xoutr2r2r3rBszSaturateCast.construct) rDrErFrGrdrerrBrJr2r2r0r3rs rcs(eZdZdZfddZddZZS)rz Create attention mask according to input mask. Args: config (Class): Configuration for BertForPreTraining. csJtt|d|_t|_t|_t j |j jddff|_dSr)rrrr7r)rSrTr*r+ mindsporerTilerZrUrtiler-r0r2r3rs   z)CreateAttentionMaskFromInputMask.__init__cCs@t|d}|||dd|ftj}||d|df}|S)Nrr9)rrrTr+rdfloat16r)r.r7rrr2r2r3rB!sz*CreateAttentionMaskFromInputMask.constructrr2r2r0r3rs rcs(eZdZdZfddZddZZS)rQz Provide bert pre-training loss. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, total loss. csDtt||j|_t|jjdfddf|_ t dt j |_ t dt j |_t|jjff|_t|jjdff|_t|jjff|_td|_t|_d|_t|jjff|_t|_td|_td|_t |jjdf|jjdff|_!t |jjf|jjff|_"dS) Nrr2rrj)rLr:)r2rLrK)#rrQrror)rrZrUronehotrrdrerr ReduceSum reduce_sum reduce_sum1 ReduceMean reduce_mean reduce_mean2r*r+last_idxNegnegrSrTRealDivdivrYr[Mulmulmul2r-r0r2r3r2s"  $zBertLoss.__init__cCs|||j}||||jtj}|||j|j|j}| | | |||j} | | || d} | |d|tdtj} || | } |||j} || d|j|j}| | | |||j} || |j}|| |}|S)z"Defines the computation performed.r2)gh㈵>r)r+rrTrdrerrorrrrrrrrtuple_to_arrayrrr[)r.rarbrgrhrfr=Z label_weightsZone_hot_labelsZper_example_loss numerator denominatorZmasked_lm_losslabelsZnext_sentence_lossrir2r2r3rBEs   zBertLoss.constructrr2r2r0r3rQ's rQcs(eZdZdZfddZddZZS)rMz Get masked lm output. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, masked lm output. cshtt||j|_t|_t|_ |j dt |j }t j|j|j||jd|j|_|jjr|jj |jjdfdfn"|jj |jjdfd|jjfft|jf|j|_|j |jjdffttd|j|_tjdd|_|j |jjdfdft jdd |_ |j j |jjdffd |_!d |_"d|jf|_#t$|_%|j|_|j&|_&dS) N)rrL)rrrrzeroT) transpose_br9r)r9rr:)'rrMrr#widthr)r*r+rrrZrrrr" hidden_actr$r%rrU vocab_emb_dpmatmulrmodel_parallelrrrrro output_biasr LogSoftmax log_softmaxshape_flat_offsetsrshape_flat_sequence_tensorrSrTrmr.r/rr0r2r3rhsH      zGetMaskedLMOutput.__init__c Cst|}tt|d}|||d|j}||||j}|||j}| ||d}| ||j }| ||j }| |}| |}|||} | | |j} | |j} || } | S)zGet output log_probsrr)r)rrr make_ranger+r rr rrTr%rrrrmrr ) r. input_tensorZoutput_weights positions input_shaperngZ flat_offsetsZ flat_positionZflat_sequence_tensorr@ log_probsr2r2r3rBs      zGetMaskedLMOutput.constructrr2r2r0r3rM]s %rMcs(eZdZdZfddZddZZS)rOz Get next sentence output. Args: config (BertConfig): The config of Bert. Returns: Tensor, next sentence output. cstt|t|_|j|jjdfft |j }t j |j d|dd|j|_|jjrz|jj|jjdfdfn&|jj|jj|jjf|jjdff|j|_t|_dS)NrrTrr)rrOrr)r r rZrUrrrrr"r#r$r%rrrrrmrSrTr r0r2r3rs     zGetNextSentenceOutput.__init__cCs&||}|||j}||}|Sr)rrTrmr )r.rr@log_probr2r2r3rBs  zGetNextSentenceOutput.constructrr2r2r0r3rOs rO)5rGnumpyrrZmindspore.common.dtypecommonrmrd mindspore.nnr mindspore.opsrZmindspore.ops.functional functionalrrrmindspore.common.initializerrrrr)mindformers.modules.layersrmindformers.modules.transformerrr #mindformers.modules.transformer.moer mindformers.tools.registerr r Zmindformers.models.base_modelr mindformers.mindformer_bookrZ#mindformers.models.bert.bert_configr__all__registerMODELSrrCellrrrrrrrQrMrOr2r2r2r3s@         C Xi ; @P6Fpengcheng.mind-mindformer/mindformers/models/bert/__pycache__/bert_config.cpython-39.pyc000066400000000000000000000053651476650141400320110ustar00rootroot00000000000000a ?cf'@stdZddlmmZddlmZmZddlm Z m Z ddl m Z ddl mZd gZe e jGd d d e ZdS) zBert Config API.N)default_transformer_configdefault_moe_config)MindFormerRegisterMindFormerModuleType) BaseConfig)MindFormerBook BertConfigcseZdZdZedZeeddeeddeedddee e ee e e e e e e e ee e e e e e e eeeee eeee dfdd Z Z S)r z8 BERT config class which defines the model size berttokclstxtclsqaF皙?:w geluTr{Gz?float32float16default) model_typeuse_one_hot_embeddings num_labelsassessment_method dropout_prob batch_size seq_length vocab_size hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_actpost_layernorm_residualhidden_dropout_probattention_probs_dropout_probmax_position_embeddingstype_vocab_sizeinitializer_rangeuse_relative_positionsdtypelayernorm_dtype softmax_dtype compute_dtypeuse_pastparallel_configcheckpoint_name_or_path moe_config is_trainingc s"tt|jfi|||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ | |_| |_||_||_||_||_||_||_||_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_||_||_|dkrt n||_!|dkrt"n||_#||_$dS)Nrr)%superr __init__rrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0mstyperrr1r2r3r4r5r7rr6rr8r9)selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9kwargs __class__N/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_config.pyr;#s<zBertConfig.__init__)r FrrrrrrrrrrrTrrrrrFrrrrFrrrT)__name__ __module__ __qualname____doc__r get_config_support_list _support_listextendstrboolintfloatr; __classcell__rArAr?rBr s )rFZmindspore.common.dtypecommonr1r<Z+mindformers.modules.transformer.transformerrrmindformers.tools.registerrr base_configrmindformer_bookr __all__registerCONFIGr rArArArBs   pengcheng.mind-mindformer/mindformers/models/bert/__pycache__/bert_processor.cpython-39.pyc000066400000000000000000000032611476650141400325540ustar00rootroot00000000000000a ?cf @s^dZddlmZddlmZddlmZddlmZm Z dgZ e e j Gd ddeZ d S) z BertProcessor )MindFormerBook) BaseTokenizer) BaseProcessor)MindFormerRegisterMindFormerModuleType BertProcessorcsdeZdZdZedZeeddeeddd fd d Zdd d Z Z S)r zU Bert processor, consists of a tokenizer (BaseTokenizer) for text input. berttokclstxtclsN max_lengthmscstt|j||||ddS)N) tokenizerrpaddingreturn_tensors)superr __init__)selfrrrr __class__Q/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_processor.pyr$s  zBertProcessor.__init__cCsli}|durh|jrht|jts4tdt|jdt|trD|g}|j||j|j|jdd}||d<|S)z call functionNz;tokenizer should inherited from the BaseTokenizer, but got .)rrr input_idstext) r isinstancer TypeErrortypestrrrr)rZ text_inputZ image_inputoutputZ text_outputrrr__call__-s     zBertProcessor.__call__)Nr rr)NN) __name__ __module__ __qualname____doc__rget_processor_support_list _support_listextendrr" __classcell__rrrrr s  N)r&mindformers.mindformer_bookrbase_tokenizerrbase_processorrZtools.registerrr__all__register PROCESSORr rrrrs    pengcheng.mind-mindformer/mindformers/models/bert/__pycache__/bert_tokenizer.cpython-39.pyc000066400000000000000000000443711476650141400325560ustar00rootroot00000000000000a ?cfZ@sdZddlZddlZddlZddlmZmZddlmZddl m Z m Z ddl m Z ddlmZd d gZd d iZd dZddZddZddZddZe e jGdd d e ZGdd d ZGdddZdS)zThe bert tokenizerN)ListOptional)logger)MindFormerRegisterMindFormerModuleType) Tokenizer)MindFormerBook BertTokenizerBasicTokenizer vocab_filez vocab.txtcCs&|dvr dSt|}|dkr"dSdS)z0Checks whether `char` is a whitespace character.)    TZZsF) unicodedatacategorycharcatrQ/home/ma-user/work/mindformers-baichuan/mindformers/models/bert/bert_tokenizer.py_is_whitespace s  rcCs(|dvr dSt|}|dr$dSdS)z-Checks whether `char` is a control character.)rrrFCT)rr startswithrrrr _is_control,s   rcCst|}d|krdksbnd|kr2dksbnd|krHdksbnd|kr^dkrfnnd St|}|d r~d Sd S) z1Checks whether `char` is a punctuation character.!/:@[`{~TPF)ordrrr)rcprrrr_is_punctuation8sZ  r'cCsft}t|ddd}|}Wdn1s40Yt|D]\}}|d}|||<qF|S)z*Loads a vocabulary file into a dictionary.rutf-8encodingNr) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokenrrr load_vocabGs&  r7cCs|}|sgS|}|S)z@Runs basic whitespace cleaning and splitting on a piece of text.)stripsplit)textr4rrrwhitespace_tokenizeRs r;c sHeZdZdZeZddgZgdZe dZ e e dde e dde e ddd-fdd Z eddZeddZddZddZddZddZdd Zd.eeeeeeed!d"d#Zd/eeeeeeeed$fd%d& Zd0eeeeeeed!d'd(Zd1d)d*Zd+d,ZZS)2r a Construct a BERT tokenizer. Based on WordPiece. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): File containing the vocabulary. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. do_basic_tokenize (`bool`, *optional*, defaults to `True`): Whether or not to do basic tokenization before WordPiece. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` unk_token (`str`, *optional*, defaults to `"[UNK]"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `"[PAD]"`): The token used for padding, for example when batching sequences of different lengths. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. is_tokenize_char (`bool`, *optional*, defaults to `False`): Whether or not to tokenize characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). ztokenizer_config.jsonzspecial_tokens_map.json) input_idsZtoken_type_idsattention_maskbertZtokclsZtxtclsqaTN[UNK][SEP][PAD][CLS][MASK]Fc  stjf|||||||| | | d | tj|sDtd|dt||_t dd|j D|_ ||_ |rt ||| | d|_t|j|jd|_g|_| |_dS)N) do_lower_casedo_basic_tokenize never_split unk_token sep_token pad_token cls_token mask_tokentokenize_chinese_chars strip_accentsz&Can't find a vocabulary file at path 'z'. To load the vocabulary from a Google pretrained model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`cSsg|]\}}||fqSrr).0tokidsrrr z*BertTokenizer.__init__..)rErGrMrN)r2rH)super__init__ospathisfile ValueErrorr7 vocab_dictr,r-items ids_to_tokensrFr basic_tokenizerWordpieceTokenizerrHwordpiece_tokenizer mask_indexis_tokenize_char)selfr rErFrGrHrIrJrKrLrMrNrakwargs __class__rrrUs>    zBertTokenizer.__init__cCs|jjSN)r]rErbrrrrEszBertTokenizer.do_lower_casecCs t|jSrf)lenrZrgrrr vocab_sizeszBertTokenizer.vocab_sizecCst|jfi|jSrf)dictrZZadded_tokens_encoderrgrrr get_vocabszBertTokenizer.get_vocabcKsg}|jrF|D]4}|jr |}||jvr6||q||jqnL|jr||D],}||jj vrr||qV||j |7}qVn |j |}|Srf) rarElowerrZappendrHrF_process_mask_tokensr]rGr_tokenize)rbr:rc split_tokens characterr6rrr _tokenizes     zBertTokenizer._tokenizecCs|j||j|jS)z0Converts a token (str) in an id using the vocab.)rZgetrH)rbr6rrr_convert_token_to_idsz"BertTokenizer._convert_token_to_idcCs|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r\rsrH)rbr5rrr_convert_id_to_tokensz"BertTokenizer._convert_id_to_tokencCsd|dd}|S)z:Converts a sequence of tokens (string) in a single string.r z ##)joinreplacer8)rbr4Z out_stringrrrconvert_tokens_to_stringsz&BertTokenizer.convert_tokens_to_string) token_ids_0 token_ids_1returncCs@|dur|jg||jgS|jg}|jg}|||||S)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. N) cls_token_id sep_token_id)rbrzr{clsseprrr build_inputs_with_special_tokenss z.BertTokenizer.build_inputs_with_special_tokens)rzr{already_has_special_tokensr|csf|rtj||ddS|durLdgdgt|dgdgt|dgSdgdgt|dgS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rzr{rNr)rTget_special_tokens_maskrh)rbrzr{rrdrrrs.z%BertTokenizer.get_special_tokens_maskcCsV|jg}|jg}|dur.t|||dgSt|||dgt||dgS)a Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). Nrr)r~r}rh)rbrzr{rrrrr$create_token_type_ids_from_sequences's z2BertTokenizer.create_token_type_ids_from_sequencescCsd}tj|r4tj||r$|dndtd}n|r@|dnd|}t|ddd^}t|jdd d D]6\}}||krt d ||}| |d |d 7}qnWdn1s0Y|S)Nr-rvr wr)r*cSs|dS)Nrr)kvrrrMrSz/BertTokenizer.save_vocabulary..)keyzsSaving vocabulary to %s: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!rr) rVrWisdirrwVOCAB_FILES_NAMESr.sortedrZr[rwarningwrite)rbZsave_directoryZfilename_prefixr5r writerr6 token_indexrrrsave_vocabularyDs" (zBertTokenizer.save_vocabularycsg}j|vrj|vrb|j}|j|d||j||tjd}q|j|fddt|D_njj|j d}|S)zprocess mask tokens in textNcsg|]\}}|jkr|qSr) _mask_token)rOindxrgrrrRbrSz6BertTokenizer._process_mask_tokens..)rG) rr5extendr]rormrhr0r`Zall_special_tokens)rbr:Z text_tokenizerrrgrrnXs    z"BertTokenizer._process_mask_tokens) TTNr@rArBrCrDTNF)N)NF)N)N) __name__ __module__ __qualname____doc__rZvocab_files_namesZ FILE_LISTZmodel_input_namesr Zget_tokenizer_support_listZ _support_listrZget_config_support_listrUpropertyrErirkrrrtruryrintrrboolrrrrn __classcell__rrrdrr [sb+ 1       c@sNeZdZdZdddZdddZdd Zdd d Zd d ZddZ ddZ dS)r a Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). TNcCs,|dur g}||_t||_||_||_dSrf)rEsetrGrMrN)rbrErGrMrNrrrrU|s  zBasicTokenizer.__init__cCs|r|jt|n|j}||}|jr4||}t|}g}|D]R}||vr|jrt|}|j dur| |}n|j r| |}| | ||qDtd |}|S)a Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. Fr )rGunionr _clean_textrM_tokenize_chinese_charsr;rErlrN_run_strip_accentsr_run_split_on_puncrw)rbr:rG orig_tokensrpr6 output_tokensrrrros"      zBasicTokenizer.tokenizecCsBtd|}g}|D]"}t|}|dkr,q||qd|S)z$Strips accents from a piece of text.NFDZMnrv)r normalizerrmrw)rbr:outputrrrrrrs   z!BasicTokenizer._run_strip_accentscCs|dur||vr|gSt|}d}d}g}|t|kr||}t|rX||gd}n |rf|gd}|d||d7}q*dd|DS) z&Splits punctuation on a piece of text.NrTFrcSsg|]}d|qS)rv)rw)rOrrrrrRrSz5BasicTokenizer._run_split_on_punc..)listrhr'rm)rbr:rGcharsiZstart_new_wordrrrrrrs"    z!BasicTokenizer._run_split_on_punccCsTg}|D]@}t|}||r>|d|||dq||qd|S)z)Adds whitespace around any CJK character.r rv)r%_is_chinese_charrmrwrbr:rrr&rrrrs     z&BasicTokenizer._tokenize_chinese_charscCsd|krdksnd|kr*dksnd|kr@dksnd|krVdksnd |krld ksnd |krd ksnd |krdksnd|krdkrnndSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rbr&rrrrsD  zBasicTokenizer._is_chinese_charcCsXg}|D]D}t|}|dks|dkst|r.qt|rB|dq||qd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir rv)r%rrrmrwrrrrrs  zBasicTokenizer._clean_text)TNTN)N)N) rrrrrUrorrrrrrrrrr hs  %  c@s"eZdZdZdddZddZdS) r^zRuns WordPiece tokenization.dcCs||_||_||_dSrf)rZrHmax_input_chars_per_word)rbr2rHrrrrrUszWordpieceTokenizer.__init__c Csg}t|D]}t|}t||jkr4||jq d}d}g}|t|krt|}d} ||krd|||} |dkrd| } | |jvr| } q|d8}qX| durd}q|| |}q@|r||jq ||q |S)a Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. FrNrvz##rT) r;rrhrrmrHrwrZr) rbr:rr6rZis_badstart sub_tokensendZ cur_substrsubstrrrrros:       zWordpieceTokenizer.tokenizeN)r)rrrrrUrorrrrr^s r^)rr,rVrtypingrrmindformers.toolsrmindformers.tools.registerrrZ!mindformers.models.base_tokenizerrmindformer_bookr __all__rrrr'r7r;register TOKENIZERr r r^rrrrs,       pengcheng.mind-mindformer/mindformers/models/bert/bert.py000066400000000000000000001006441476650141400241710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://gitee.com/mindspore/models/blob/master/official/nlp/Bert/src/bert_model.py # ============================================================================ """BERT Model.""" import numpy as np import mindspore import mindspore.common.dtype as mstype import mindspore.nn as nn import mindspore.ops as ops import mindspore.ops.functional as F from mindspore import Tensor, Parameter from mindspore.common.initializer import TruncatedNormal, initializer from mindspore.ops import operations as P from mindformers.modules.layers import LayerNorm from mindformers.modules.transformer import Transformer, VocabEmbedding from mindformers.modules.transformer.moe import default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.models.bert.bert_config import BertConfig __all__ = ['BertConfig', 'BertModel', 'BertForPreTraining', 'BertForTokenClassification', 'BertForMultipleChoice', 'BertForQuestionAnswering'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForTokenClassification(BaseModel): """ Bert with dense layer for name entity recoginition task. Args: config (BertConfig): The config of BertForTokenClassification. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForTokenClassification, BertTokenizer >>> model = BertForTokenClassification.from_pretrained('tokcls_bert_base_chinese') >>> tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') >>> data = tokenizer(["我在杭州华为工作。"], return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[[ 0.0886748 -0.23066735 -0.03969013 ... 0.07333283 -0.02968273 0.06125224] [-0.03115104 -0.47599065 0.01928361 ... 0.22821501 0.42415133 0.14563856] [ 0.05570185 -0.36168078 0.00884905 ... -0.15357454 -0.33119604 -0.09889631] ... [ 0.01028904 -0.20403272 -0.1117363 ... -0.03601318 0.034153 0.12492958] [-0.00420664 -0.1786235 -0.11332294 ... -0.07119732 0.02407441 0.14775723] [-0.04674841 -0.1608555 -0.0781534 ... -0.06064402 0.0439388 0.20419256]]] """ _support_list = MindFormerBook.get_model_support_list()['tokcls']['bert'] def __init__(self, config=BertConfig()): super(BertForTokenClassification, self).__init__(config) self.num_labels = config.num_labels self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.dropout = nn.Dropout(1 - config.hidden_dropout_prob) self.classifier = nn.Dense(config.hidden_size, config.num_labels).to_float(config.compute_dtype) self.cross_entropy_loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.load_checkpoint(config) def construct(self, input_ids, input_mask, token_type_ids, label_ids=None): """Get Training Loss or Logits""" bert_outputs = self.bert(input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_ids) sequence_output = bert_outputs[0] sequence_output = self.dropout(sequence_output) logits = self.classifier(sequence_output) if self.training: logits = self.reshape(logits, (-1, self.num_labels)) label_ids = self.reshape(label_ids, (-1,)) output = self.cross_entropy_loss(logits, label_ids) else: if label_ids is None: output = logits else: output = (logits, label_ids) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForPreTraining(BaseModel): """ Provide bert pre-training loss through network. Args: config (BertConfig): The config of BertForPreTraining. is_training (bool): Specifies whether to use the training mode. use_one_hot_embeddings (bool): Specifies whether to use one-hot for embeddings. Default: False. Returns: Tensor, the loss of the network. Examples: >>> from mindspore import Tensor >>> import mindspore.common.dtype as mstype >>> from mindformers import BertForPreTraining, BertTokenizer >>> model = BertForPreTraining.from_pretrained('bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') >>> data = tokenizer(["Paris is the [MASK] of France."], ... return_tensors='ms', max_length=128, padding="max_length") >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> masked_lm_positions = Tensor([[4]], mstype.int32) >>> next_sentence_labels = Tensor([[1]], mstype.int32) >>> masked_lm_weights = Tensor([[1]], mstype.int32) >>> masked_lm_ids = Tensor([[3007]], mstype.int32) >>> output = model(input_ids, attention_mask, token_type_ids, next_sentence_labels, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> print(output) [0.6706] """ _support_list = MindFormerBook.get_model_support_list()['bert'] def __init__(self, config=BertConfig()): super(BertForPreTraining, self).__init__(config) self.is_training = config.is_training # self.bert = BertScore(config, config.is_training, config.use_one_hot_embeddings) self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.mlmloss = GetMaskedLMOutput(config) self.nsploss = GetNextSentenceOutput(config) self.loss = BertLoss(config) self.cast = P.Cast() self.use_moe = (config.parallel_config.moe_config.expert_num > 1) self.add = P.Add().shard(((1,), ())) self.load_checkpoint(config) def bert_forward(self, input_ids, input_mask, token_type_id, masked_lm_positions=None): """connect backbone and heads.""" moe_loss = 0 if self.use_moe: sequence_output, pooled_output, embedding_table, moe_loss = \ self.bert(input_ids, token_type_id, input_mask) else: sequence_output, pooled_output, embedding_table = \ self.bert(input_ids, token_type_id, input_mask) prediction_scores = self.mlmloss(sequence_output, embedding_table, masked_lm_positions) seq_relationship_score = self.nsploss(pooled_output) if not self.training: return sequence_output, pooled_output, prediction_scores, seq_relationship_score return prediction_scores, seq_relationship_score, moe_loss def construct(self, input_ids, input_mask, token_type_id, next_sentence_labels=None, masked_lm_positions=None, masked_lm_ids=None, masked_lm_weights=None): """Get pre-training loss""" if not self.training: return self.bert_forward(input_ids, input_mask, token_type_id, masked_lm_positions) prediction_scores, seq_relationship_score, moe_loss = \ self.bert_forward(input_ids, input_mask, token_type_id, masked_lm_positions) total_loss = self.loss(prediction_scores, seq_relationship_score, masked_lm_ids, masked_lm_weights, next_sentence_labels) if self.use_moe: total_loss = self.add(total_loss, moe_loss) return self.cast(total_loss, mstype.float32) class BertModel(nn.Cell): """ Bidirectional Encoder Representations from Transformers. Args: config (Class): Configuration for BertModel. is_training (bool): True for training mode. False for eval mode. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. """ def __init__(self, config=None, is_training=False, use_one_hot_embeddings=False): super(BertModel, self).__init__() if not is_training: config.hidden_dropout_prob = 0.0 config.attention_probs_dropout_prob = 0.0 self.get_attention_mask = CreateAttentionMaskFromInputMask(config) self.hidden_size = config.hidden_size self.num_hidden_layers = config.num_hidden_layers self.embedding_size = config.hidden_size self.token_type_ids = None if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.use_moe = (config.parallel_config.moe_config.expert_num > 1) self.word_embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, param_init=initializer("truncatedNormal", [config.vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) output_embedding_shape = [-1, config.seq_length, self.embedding_size] self.embedding_postprocessor = EmbeddingPostprocessor(config, embedding_size=self.embedding_size, embedding_shape=output_embedding_shape, use_relative_positions=config.use_relative_positions, use_token_type=True, token_type_vocab_size=config.type_vocab_size, use_one_hot_embeddings=use_one_hot_embeddings, max_position_embeddings=config.max_position_embeddings, dropout_prob=config.hidden_dropout_prob) self.bert_encoder = Transformer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.intermediate_size, src_seq_length=config.seq_length, tgt_seq_length=config.seq_length, num_heads=config.num_attention_heads, encoder_layers=config.num_hidden_layers, parallel_config=config.parallel_config, decoder_layers=0, moe_config=moe_config, param_init_type=config.compute_dtype, layernorm_compute_type=config.layernorm_dtype, softmax_compute_type=config.softmax_dtype, post_layernorm_residual=config.post_layernorm_residual, attention_dropout_rate=config.attention_probs_dropout_prob, hidden_dropout_rate=config.hidden_dropout_prob,) self.cast = P.Cast() self.dtype = config.dtype self.cast_compute_type = SaturateCast(dst_type=config.compute_dtype) self.slice = P.StridedSlice().shard(((1, 1, 1),)) self.squeeze_1 = P.Squeeze(axis=1) self.dense = nn.Dense(self.hidden_size, self.hidden_size, activation="tanh", weight_init=TruncatedNormal(config.initializer_range)).to_float(config.compute_dtype) self.print = P.Print() def construct(self, input_ids, token_type_ids, input_mask): """Bidirectional Encoder Representations from Transformers.""" # embedding word_embeddings, embedding_tables = self.word_embedding(input_ids) embedding_output = self.embedding_postprocessor(token_type_ids, word_embeddings) # attention mask [batch_size, seq_length, seq_length](4, 1, 128) -> (4, 128, 128) input_mask = P.Cast()(input_mask, self.dtype) attention_mask = self.get_attention_mask(input_mask) # bert encoder encoder_output = self.bert_encoder(self.cast_compute_type(embedding_output), attention_mask) sequence_output = encoder_output[0] # pooler batch_size = P.Shape()(input_ids)[0] sequence_slice = self.slice(sequence_output, (0, 0, 0), (batch_size, 1, self.hidden_size), (1, 1, 1)) first_token = self.squeeze_1(sequence_slice) pooled_output = self.dense(first_token) pooled_output = self.cast(pooled_output, self.dtype) moe_loss = 0 if self.use_moe: moe_loss = encoder_output[-1] return sequence_output, pooled_output, embedding_tables, moe_loss return sequence_output, pooled_output, embedding_tables @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForMultipleChoice(BaseModel): """ Bert with dense layer for txt classification task. Args: config (BertConfig): The config of BertForMultipleChoice. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForMultipleChoice, BertTokenizer >>> model = BertForMultipleChoice.from_pretrained('txtcls_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('txtcls_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=128, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) [[0.05121157 0.3273056 0.31614417]] """ _support_list = MindFormerBook.get_model_support_list()['txtcls']['bert'] def __init__(self, config=BertConfig()): super(BertForMultipleChoice, self).__init__(config) self.num_labels = config.num_labels self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.weight_init = TruncatedNormal(config.initializer_range) self.classifier = nn.Dense(config.hidden_size, self.num_labels, weight_init=self.weight_init, has_bias=True).to_float(config.compute_dtype) self.dropout = nn.Dropout(1 - config.dropout_prob) self.cross_entropy_loss = nn.CrossEntropyLoss() self.cast = P.Cast() self.reshape = P.Reshape() self.dtype = config.dtype self.compute_dtype = config.compute_dtype self.load_checkpoint(config) def construct(self, input_ids, input_mask, token_type_id, label_ids=None): """Get Training Loss or Logits""" _, pooled_output, _ = self.bert(input_ids, token_type_id, input_mask) pooled_output = self.dropout(pooled_output) logits = self.classifier(pooled_output) if self.training: logits = self.reshape(logits, (-1, self.num_labels)) label_ids = self.reshape(label_ids, (-1,)) output = self.cross_entropy_loss(logits, label_ids) else: if label_ids is None: output = logits else: label_ids = self.reshape(label_ids, (-1,)) output = (logits, label_ids) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class BertForQuestionAnswering(BaseModel): """ Bert with dense layer for question answering task. Args: config (BertConfig): The config of BertModel. Returns: Tensor, loss, logits. Examples: >>> from mindformers import BertForQuestionAnswering, BertTokenizer >>> model = BertForQuestionAnswering.from_pretrained('qa_bert_base_uncased') >>> tokenizer = BertTokenizer.from_pretrained('qa_bert_base_uncased') >>> data = tokenizer(["The new rights are nice enough-Everyone really likes the newest benefits "], ... return_tensors='ms', max_length=384, padding='max_length') >>> input_ids = data['input_ids'] >>> attention_mask = data['attention_mask'] >>> token_type_ids = data['token_type_ids'] >>> output = model(input_ids, attention_mask, token_type_ids) >>> print(output) (Tensor(shape=[1, 384], dtype=Float32, value= [[ 5.30713797e-02, 1.08105361e-01, 1.55855455e-02 ... -3.73562761e-02, -1.89271923e-02, -2.99206823e-02]]), Tensor(shape=[1, 384], dtype=Float32, value= [[ 2.39873882e-02, 2.35675126e-01, 1.52210891e-01 ... -2.11214088e-02, 4.84664962e-02, -4.52579707e-02]])) """ _support_list = MindFormerBook.get_model_support_list()['qa']['bert'] def __init__(self, config=BertConfig()): super(BertForQuestionAnswering, self).__init__(config) self.bert = BertModel(config, config.is_training, config.use_one_hot_embeddings) self.qa_outputs = nn.Dense(config.hidden_size, 2).to_float(config.compute_dtype) self.load_checkpoint(config) self.split = P.Split(output_num=2, axis=-1) def construct(self, input_ids, input_mask, token_type_id, start_position=None, end_position=None, unique_id=None): """Get Training Loss or Logits""" bert_outputs = self.bert(input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_id) sequence_output = bert_outputs[0] logits = self.qa_outputs(sequence_output) start_logits, end_logits = self.split(logits) start_logits = start_logits.squeeze(-1) end_logits = end_logits.squeeze(-1) if self.training: if len(start_position.shape) > 1: start_position = start_position.squeeze(-1) if len(end_position.shape) > 1: end_position = end_position.squeeze(-1) ignored_index = start_logits.shape[1] start_position = ops.clip_by_value(start_position, 0, ignored_index) end_position = ops.clip_by_value(end_position, 0, ignored_index) loss_fct = nn.CrossEntropyLoss(ignore_index=ignored_index) start_loss = loss_fct(start_logits, start_position) end_loss = loss_fct(end_logits, end_position) total_loss = (start_loss + end_loss) / 2 output = total_loss else: output = (unique_id, start_logits, end_logits) return output class EmbeddingPostprocessor(nn.Cell): """ Postprocessors apply positional and token type embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. embedding_shape (list): [batch_size, seq_length, embedding_size], the shape of each embedding vector. use_token_type (bool): Specifies whether to use token type embeddings. Default: False. token_type_vocab_size (int): Size of token type vocab. Default: 16. use_one_hot_embeddings (bool): Specifies whether to use one hot encoding form. Default: False. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 512. dropout_prob (float): The dropout probability. Default: 0.1. """ def __init__(self, config, embedding_size, embedding_shape, use_relative_positions=False, use_token_type=False, token_type_vocab_size=16, use_one_hot_embeddings=False, max_position_embeddings=512, dropout_prob=0.1): super(EmbeddingPostprocessor, self).__init__() self.use_token_type = use_token_type self.token_type_vocab_size = token_type_vocab_size self.use_one_hot_embeddings = use_one_hot_embeddings self.max_position_embeddings = max_position_embeddings self.token_type_embedding = VocabEmbedding(vocab_size=token_type_vocab_size, embedding_size=embedding_size, param_init=initializer("truncatedNormal", [token_type_vocab_size, embedding_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) self.shape_flat = (-1,) self.one_hot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.1, mstype.float32) self.array_mul = P.MatMul() self.reshape = P.Reshape() self.shape = tuple(embedding_shape) self.dropout = nn.Dropout(1 - dropout_prob) self.gather = P.Gather() self.use_relative_positions = use_relative_positions self.slice = P.StridedSlice().shard(((1, 1),)) _, seq, _ = self.shape self.full_position_embedding = VocabEmbedding(vocab_size=config.max_position_embeddings, embedding_size=embedding_size, param_init=initializer("truncatedNormal", [config.max_position_embeddings, embedding_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) self.layernorm = LayerNorm((embedding_size,)).to_float(mstype.float32) self.layernorm.shard(((config.parallel_config.data_parallel, 1, 1),)) self.position_ids = Tensor(np.arange(seq).reshape(-1, seq).astype(np.int32)) self.add = P.Add().shard( ((config.parallel_config.data_parallel, 1, 1), (config.parallel_config.data_parallel, 1, 1))) self.slice = P.StridedSlice().shard(((1, 1),)) def construct(self, token_type_ids, word_embeddings): """Postprocessors apply positional and token type embeddings to word embeddings.""" output = word_embeddings if self.use_token_type: token_type_embeddings, _ = self.token_type_embedding(token_type_ids) output = self.add(output, token_type_embeddings) if not self.use_relative_positions: shape = F.shape(output) shape_position = F.shape(self.position_ids) position_ids = self.slice(self.position_ids, (0, 0), (shape_position[0], shape[1]), (1, 1)) position_embeddings, _ = self.full_position_embedding(position_ids) output = self.add(output, position_embeddings) output = self.layernorm(output) output = self.dropout(output) return output class SaturateCast(nn.Cell): """ Performs a safe saturating cast. This operation applies proper clamping before casting to prevent the danger that the value will overflow or underflow. Args: src_type (:class:`mindspore.dtype`): The type of the elements of the input tensor. Default: mstype.float32. dst_type (:class:`mindspore.dtype`): The type of the elements of the output tensor. Default: mstype.float32. """ def __init__(self, dst_type=mstype.float32): super(SaturateCast, self).__init__() np_type = mstype.dtype_to_nptype(dst_type) self.tensor_min_type = float(np.finfo(np_type).min) self.tensor_max_type = float(np.finfo(np_type).max) self.min_op = P.Minimum() self.max_op = P.Maximum() self.cast = P.Cast() self.dst_type = dst_type def construct(self, x): out = self.max_op(x, self.tensor_min_type) out = self.min_op(out, self.tensor_max_type) return self.cast(out, self.dst_type) class CreateAttentionMaskFromInputMask(nn.Cell): """ Create attention mask according to input mask. Args: config (Class): Configuration for BertForPreTraining. """ def __init__(self, config): super(CreateAttentionMaskFromInputMask, self).__init__() self.input_mask = None self.cast = P.Cast() self.reshape = P.Reshape() self.tile = mindspore.ops.Tile().shard(((config.parallel_config.data_parallel, 1, 1),)) def construct(self, input_mask): seq_length = F.shape(input_mask)[1] attention_mask = self.cast(self.reshape(input_mask, (-1, 1, seq_length)), mstype.float16) attention_mask = self.tile(attention_mask, (1, seq_length, 1)) return attention_mask class BertLoss(nn.Cell): """ Provide bert pre-training loss. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, total loss. """ def __init__(self, config): super(BertLoss, self).__init__() self.vocab_size = config.vocab_size self.onehot = P.OneHot().shard(((config.parallel_config.data_parallel, 1), (), ())) self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum().shard(((config.parallel_config.data_parallel,),)) self.reduce_sum1 = P.ReduceSum().shard(((config.parallel_config.data_parallel, 1),)) self.reduce_mean = P.ReduceMean().shard(((config.parallel_config.data_parallel,),)) self.reduce_mean2 = P.ReduceMean().shard(((1,),)) self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg().shard(((config.parallel_config.data_parallel,),)) self.cast = P.Cast() self.div = P.RealDiv().shard(((), (1,))) self.add = P.Add().shard(((1,), ())) self.mul = P.Mul().shard(((config.parallel_config.data_parallel, 1), (config.parallel_config.data_parallel, 1))) self.mul2 = P.Mul().shard(((config.parallel_config.data_parallel,), (config.parallel_config.data_parallel,))) def construct(self, prediction_scores, seq_relationship_score, masked_lm_ids, masked_lm_weights, next_sentence_labels): """Defines the computation performed.""" label_ids = self.reshape(masked_lm_ids, self.last_idx) label_weights = self.cast(self.reshape(masked_lm_weights, self.last_idx), mstype.float32) one_hot_labels = self.onehot(label_ids, self.vocab_size, self.on_value, self.off_value) per_example_loss = self.neg(self.reduce_sum1(self.mul(prediction_scores, one_hot_labels), self.last_idx)) numerator = self.reduce_sum(self.mul2(label_weights, per_example_loss), ()) denominator = self.reduce_sum(label_weights, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) masked_lm_loss = self.div(numerator, denominator) # next_sentence_loss labels = self.reshape(next_sentence_labels, self.last_idx) one_hot_labels = self.onehot(labels, 2, self.on_value, self.off_value) per_example_loss = self.neg(self.reduce_sum1( self.mul(one_hot_labels, seq_relationship_score), self.last_idx)) next_sentence_loss = self.reduce_mean2(per_example_loss, self.last_idx) # total_loss total_loss = self.add(masked_lm_loss, next_sentence_loss) return total_loss class GetMaskedLMOutput(nn.Cell): """ Get masked lm output. Args: config (BertConfig): The config of BertForPreTraining. Returns: Tensor, masked lm output. """ def __init__(self, config): super(GetMaskedLMOutput, self).__init__() self.width = config.hidden_size self.reshape = P.Reshape() self.gather = P.Gather() self.gather.shard(((1, 1), (1,))) weight_init = TruncatedNormal(config.initializer_range) self.dense = nn.Dense(self.width, config.hidden_size, weight_init=weight_init, activation=config.hidden_act).to_float(config.compute_dtype) if config.parallel_config.vocab_emb_dp: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) else: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), ( 1, config.parallel_config.model_parallel))) self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.compute_dtype) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.output_bias = Parameter( initializer( 'zero', config.vocab_size)) self.matmul = P.MatMul(transpose_b=True) self.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) self.log_softmax = nn.LogSoftmax(axis=-1) self.log_softmax.log_softmax.shard(((config.parallel_config.data_parallel, 1),)) self.shape_flat_offsets = (-1, 1) self.last_idx = (-1,) self.shape_flat_sequence_tensor = (-1, self.width) self.cast = P.Cast() self.compute_dtype = config.compute_dtype self.dtype = config.dtype def construct(self, input_tensor, output_weights, positions): """Get output log_probs""" input_shape = P.Shape()(input_tensor) rng = F.tuple_to_array(F.make_range(input_shape[0])) flat_offsets = self.reshape(rng * input_shape[1], self.shape_flat_offsets) flat_position = self.reshape(positions + flat_offsets, self.last_idx) flat_sequence_tensor = self.reshape(input_tensor, self.shape_flat_sequence_tensor) input_tensor = self.gather(flat_sequence_tensor, flat_position, 0) input_tensor = self.cast(input_tensor, self.compute_dtype) output_weights = self.cast(output_weights, self.compute_dtype) input_tensor = self.dense(input_tensor) input_tensor = self.layernorm(input_tensor) logits = self.matmul(input_tensor, output_weights) logits = self.cast(logits, self.dtype) logits = logits + self.output_bias log_probs = self.log_softmax(logits) return log_probs class GetNextSentenceOutput(nn.Cell): """ Get next sentence output. Args: config (BertConfig): The config of Bert. Returns: Tensor, next sentence output. """ def __init__(self, config): super(GetNextSentenceOutput, self).__init__() self.log_softmax = P.LogSoftmax() self.log_softmax.shard(((config.parallel_config.data_parallel, 1),)) weight_init = TruncatedNormal(config.initializer_range) self.dense = nn.Dense(config.hidden_size, 2, weight_init=weight_init, has_bias=True).to_float(config.compute_dtype) if config.parallel_config.vocab_emb_dp: self.dense.matmul.shard(((config.parallel_config.data_parallel, 1), (1, 1))) else: self.dense.matmul.shard(((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (config.parallel_config.model_parallel, 1))) self.dtype = config.dtype self.cast = P.Cast() def construct(self, input_tensor): logits = self.dense(input_tensor) logits = self.cast(logits, self.dtype) log_prob = self.log_softmax(logits) return log_prob pengcheng.mind-mindformer/mindformers/models/bert/bert_config.py000066400000000000000000000110471476650141400255140ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bert Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['BertConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BertConfig(BaseConfig): """ BERT config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['bert'] _support_list.extend(MindFormerBook.get_config_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['txtcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['qa']['bert']) def __init__(self, model_type: str = "bert", use_one_hot_embeddings: bool = False, num_labels: int = 1, assessment_method: str = "", dropout_prob: float = 0.1, batch_size: int = 16, seq_length: int = 128, vocab_size: int = 30522, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, hidden_act: str = "gelu", post_layernorm_residual: bool = True, hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, max_position_embeddings: int = 128, type_vocab_size: int = 2, initializer_range: float = 0.02, use_relative_positions: bool = False, dtype: str = "float32", layernorm_dtype: str = "float32", softmax_dtype: str = "float32", compute_dtype: str = "float16", use_past: bool = False, parallel_config: str = "default", checkpoint_name_or_path: str = "", moe_config: str = "default", is_training: bool = True, **kwargs): super(BertConfig, self).__init__(**kwargs) self.model_type = model_type self.use_one_hot_embeddings = use_one_hot_embeddings self.num_labels = num_labels self.assessment_method = assessment_method self.dropout_prob = dropout_prob self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.intermediate_size = intermediate_size self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.max_position_embeddings = max_position_embeddings self.type_vocab_size = type_vocab_size self.initializer_range = initializer_range self.use_relative_positions = use_relative_positions self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.layernorm_dtype = mstype.float32 if layernorm_dtype == "float32" else mstype.float16 self.softmax_dtype = mstype.float32 if softmax_dtype == "float32" else mstype.float16 self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.use_past = use_past self.checkpoint_name_or_path = checkpoint_name_or_path self.parallel_config = default_transformer_config if parallel_config == "default" else parallel_config self.moe_config = default_moe_config if moe_config == "default" else moe_config self.is_training = is_training pengcheng.mind-mindformer/mindformers/models/bert/bert_processor.py000066400000000000000000000046721476650141400262740ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BertProcessor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['BertProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class BertProcessor(BaseProcessor): """ Bert processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['bert'] _support_list.extend(MindFormerBook.get_processor_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_processor_support_list()['txtcls']['bert']) def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(BertProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-mindformer/mindformers/models/bert/bert_tokenizer.py000066400000000000000000000552021476650141400262620ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The bert tokenizer""" import collections import os import unicodedata from typing import List, Optional from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from ...mindformer_book import MindFormerBook __all__ = ['BertTokenizer', 'BasicTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "vocab.txt"} def _is_whitespace(char): """Checks whether `char` is a whitespace character.""" # \t, \n, and \r are technically control characters but we treat them # as whitespace since they are generally considered as such. if char in (' ', '\t', '\n', '\r'): return True cat = unicodedata.category(char) if cat == "Zs": return True return False def _is_control(char): """Checks whether `char` is a control character.""" # These are technically control characters but we count them as whitespace # characters. if char in ('\t', '\n', '\r'): return False cat = unicodedata.category(char) if cat.startswith("C"): return True return False def _is_punctuation(char): """Checks whether `char` is a punctuation character.""" cp = ord(char) # We treat all non-letter/number ASCII as punctuation. # Characters such as "^", "$", and "`" are not in the Unicode # Punctuation class but we treat them as punctuation anyways, for # consistency. if 33 <= cp <= 47 or 58 <= cp <= 64 or 91 <= cp <= 96 or 123 <= cp <= 126: return True cat = unicodedata.category(char) if cat.startswith("P"): return True return False def load_vocab(vocab_file): """Loads a vocabulary file into a dictionary.""" vocab = collections.OrderedDict() with open(vocab_file, "r", encoding="utf-8") as reader: tokens = reader.readlines() for index, token in enumerate(tokens): token = token.rstrip("\n") vocab[token] = index return vocab def whitespace_tokenize(text): """Runs basic whitespace cleaning and splitting on a piece of text.""" text = text.strip() if not text: return [] tokens = text.split() return tokens @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class BertTokenizer(Tokenizer): r""" Construct a BERT tokenizer. Based on WordPiece. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): File containing the vocabulary. do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. do_basic_tokenize (`bool`, *optional*, defaults to `True`): Whether or not to do basic tokenization before WordPiece. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` unk_token (`str`, *optional*, defaults to `"[UNK]"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. sep_token (`str`, *optional*, defaults to `"[SEP]"`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `"[PAD]"`): The token used for padding, for example when batching sequences of different lengths. cls_token (`str`, *optional*, defaults to `"[CLS]"`): The classifier token which is used when doing sequence classification (classification of the whole sequence instead of per-token classification). It is the first token of the sequence when built with special tokens. mask_token (`str`, *optional*, defaults to `"[MASK]"`): The token used for masking values. This is the token used when training this model with masked language modeling. This is the token which the model will try to predict. tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. is_tokenize_char (`bool`, *optional*, defaults to `False`): Whether or not to tokenize characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json', 'special_tokens_map.json'] model_input_names = ["input_ids", "token_type_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['bert'] _support_list.extend(MindFormerBook.get_config_support_list()['tokcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['txtcls']['bert']) _support_list.extend(MindFormerBook.get_config_support_list()['qa']['bert']) def __init__( self, vocab_file, do_lower_case=True, do_basic_tokenize=True, never_split=None, unk_token="[UNK]", sep_token="[SEP]", pad_token="[PAD]", cls_token="[CLS]", mask_token="[MASK]", tokenize_chinese_chars=True, strip_accents=None, is_tokenize_char=False, **kwargs, ): super().__init__( do_lower_case=do_lower_case, do_basic_tokenize=do_basic_tokenize, never_split=never_split, unk_token=unk_token, sep_token=sep_token, pad_token=pad_token, cls_token=cls_token, mask_token=mask_token, tokenize_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, **kwargs, ) if not os.path.isfile(vocab_file): raise ValueError( f"Can't find a vocabulary file at path '{vocab_file}'. To load the vocabulary from a Google pretrained" " model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`" ) self.vocab_dict = load_vocab(vocab_file) self.ids_to_tokens = collections.OrderedDict([(ids, tok) for tok, ids in self.vocab_dict.items()]) self.do_basic_tokenize = do_basic_tokenize if do_basic_tokenize: self.basic_tokenizer = BasicTokenizer( do_lower_case=do_lower_case, never_split=never_split, tokenize_chinese_chars=tokenize_chinese_chars, strip_accents=strip_accents, ) self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab_dict, unk_token=self.unk_token) self.mask_index = [] self.is_tokenize_char = is_tokenize_char @property def do_lower_case(self): return self.basic_tokenizer.do_lower_case @property def vocab_size(self): return len(self.vocab_dict) def get_vocab(self): return dict(self.vocab_dict, **self.added_tokens_encoder) def _tokenize(self, text, **kwargs): split_tokens = [] if self.is_tokenize_char: for character in text: if self.do_lower_case: character = character.lower() if character in self.vocab_dict: split_tokens.append(character) else: split_tokens.append(self.unk_token) else: if self.do_basic_tokenize: for token in self._process_mask_tokens(text): # If the token is part of the never_split set if token in self.basic_tokenizer.never_split: split_tokens.append(token) else: split_tokens += self.wordpiece_tokenizer.tokenize(token) else: split_tokens = self.wordpiece_tokenizer.tokenize(text) return split_tokens def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.vocab_dict.get(token, self.vocab_dict.get(self.unk_token)) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.ids_to_tokens.get(index, self.unk_token) def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" out_string = " ".join(tokens).replace(" ##", "").strip() return out_string def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ if token_ids_1 is None: return [self.cls_token_id] + token_ids_0 + [self.sep_token_id] cls = [self.cls_token_id] sep = [self.sep_token_id] return cls + token_ids_0 + sep + token_ids_1 + sep def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) if token_ids_1 is not None: return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1] return [1] + ([0] * len(token_ids_0)) + [1] def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ sep = [self.sep_token_id] cls = [self.cls_token_id] if token_ids_1 is None: return len(cls + token_ids_0 + sep) * [0] return len(cls + token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] def save_vocabulary(self, save_directory, filename_prefix=None): index = 0 if os.path.isdir(save_directory): vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) else: vocab_file = (filename_prefix + "-" if filename_prefix else "") + save_directory with open(vocab_file, "w", encoding="utf-8") as writer: for token, token_index in sorted(self.vocab_dict.items(), key=lambda kv: kv[1]): if index != token_index: logger.warning( "Saving vocabulary to %s: vocabulary indices are not consecutive." " Please check that the vocabulary is not corrupted!", vocab_file ) index = token_index writer.write(token + "\n") index += 1 return vocab_file def _process_mask_tokens(self, text): """process mask tokens in text""" text_tokenize = [] if self._mask_token in text: while self._mask_token in text: ind = text.index(self._mask_token) text_tokenize.extend(self.basic_tokenizer.tokenize(text[:ind])) text_tokenize.append(self._mask_token) text = text[ind + len(self._mask_token):] text_tokenize.extend(self.basic_tokenizer.tokenize(text)) self.mask_index = [ind for ind, x in enumerate(text_tokenize) if x == self._mask_token] else: text_tokenize = self.basic_tokenizer.tokenize(text, never_split=self.all_special_tokens) return text_tokenize class BasicTokenizer: """ Constructs a BasicTokenizer that will run basic tokenization (punctuation splitting, lower casing, etc.). Args: do_lower_case (`bool`, *optional*, defaults to `True`): Whether or not to lowercase the input when tokenizing. never_split (`Iterable`, *optional*): Collection of tokens which will never be split during tokenization. Only has an effect when `do_basic_tokenize=True` tokenize_chinese_chars (`bool`, *optional*, defaults to `True`): Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (see this [issue](https://github.com/huggingface/transformers/issues/328)). strip_accents (`bool`, *optional*): Whether or not to strip all accents. If this option is not specified, then it will be determined by the value for `lowercase` (as in the original BERT). """ def __init__(self, do_lower_case=True, never_split=None, tokenize_chinese_chars=True, strip_accents=None): if never_split is None: never_split = [] self.do_lower_case = do_lower_case self.never_split = set(never_split) self.tokenize_chinese_chars = tokenize_chinese_chars self.strip_accents = strip_accents def tokenize(self, text, never_split=None): """ Basic Tokenization of a piece of text. Split on "white spaces" only, for sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. """ # union() returns a new set by concatenating the two sets. never_split = self.never_split.union(set(never_split)) if never_split else self.never_split text = self._clean_text(text) # This was added on November 1st, 2018 for the multilingual and Chinese # models. This is also applied to the English models now, but it doesn't # matter since the English models were not trained on any Chinese data # and generally don't have any Chinese data in them (there are Chinese # characters in the vocabulary because Wikipedia does have some Chinese # words in the English Wikipedia.). if self.tokenize_chinese_chars: text = self._tokenize_chinese_chars(text) orig_tokens = whitespace_tokenize(text) split_tokens = [] for token in orig_tokens: if token not in never_split: if self.do_lower_case: token = token.lower() if self.strip_accents is not False: token = self._run_strip_accents(token) elif self.strip_accents: token = self._run_strip_accents(token) split_tokens.extend(self._run_split_on_punc(token, never_split)) output_tokens = whitespace_tokenize(" ".join(split_tokens)) return output_tokens def _run_strip_accents(self, text): """Strips accents from a piece of text.""" text = unicodedata.normalize("NFD", text) output = [] for char in text: cat = unicodedata.category(char) if cat == "Mn": continue output.append(char) return "".join(output) def _run_split_on_punc(self, text, never_split=None): """Splits punctuation on a piece of text.""" if never_split is not None and text in never_split: return [text] chars = list(text) i = 0 start_new_word = True output = [] while i < len(chars): char = chars[i] if _is_punctuation(char): output.append([char]) start_new_word = True else: if start_new_word: output.append([]) start_new_word = False output[-1].append(char) i += 1 return ["".join(x) for x in output] def _tokenize_chinese_chars(self, text): """Adds whitespace around any CJK character.""" output = [] for char in text: cp = ord(char) if self._is_chinese_char(cp): output.append(" ") output.append(char) output.append(" ") else: output.append(char) return "".join(output) def _is_chinese_char(self, cp): """Checks whether CP is the codepoint of a CJK character.""" # This defines a "chinese character" as anything in the CJK Unicode block: # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block) # # Note that the CJK Unicode block is NOT all Japanese and Korean characters, # despite its name. The modern Korean Hangul alphabet is a different block, # as is Japanese Hiragana and Katakana. Those alphabets are used to write # space-separated words, so they are not treated specially and handled # like the all of the other languages. if ( (0x4E00 <= cp <= 0x9FFF) or (0x3400 <= cp <= 0x4DBF) or (0x20000 <= cp <= 0x2A6DF) or (0x2A700 <= cp <= 0x2B73F) or (0x2B740 <= cp <= 0x2B81F) or (0x2B820 <= cp <= 0x2CEAF) or (0xF900 <= cp <= 0xFAFF) or (0x2F800 <= cp <= 0x2FA1F) ): return True return False def _clean_text(self, text): """Performs invalid character removal and whitespace cleanup on text.""" output = [] for char in text: cp = ord(char) if cp == 0 or cp == 0xFFFD or _is_control(char): continue if _is_whitespace(char): output.append(" ") else: output.append(char) return "".join(output) class WordpieceTokenizer: """Runs WordPiece tokenization.""" def __init__(self, vocab, unk_token, max_input_chars_per_word=100): self.vocab_dict = vocab self.unk_token = unk_token self.max_input_chars_per_word = max_input_chars_per_word def tokenize(self, text): """ Tokenizes a piece of text into its word pieces. This uses a greedy longest-match-first algorithm to perform tokenization using the given vocabulary. For example, `input = "unaffable"` wil return as output `["un", "##aff", "##able"]`. Args: text: A single token or whitespace separated tokens. This should have already been passed through *BasicTokenizer*. Returns: A list of wordpiece tokens. """ output_tokens = [] for token in whitespace_tokenize(text): chars = list(token) if len(chars) > self.max_input_chars_per_word: output_tokens.append(self.unk_token) continue is_bad = False start = 0 sub_tokens = [] while start < len(chars): end = len(chars) cur_substr = None while start < end: substr = "".join(chars[start:end]) if start > 0: substr = "##" + substr if substr in self.vocab_dict: cur_substr = substr break end -= 1 if cur_substr is None: is_bad = True break sub_tokens.append(cur_substr) start = end if is_bad: output_tokens.append(self.unk_token) else: output_tokens.extend(sub_tokens) return output_tokens pengcheng.mind-mindformer/mindformers/models/bert/convert_weight.py000066400000000000000000000206761476650141400262720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: if tgt not in weight_dict: if "LayerNorm.gamma" in tgt: tgt = tgt.replace("gamma", "weight") if "LayerNorm.beta" in tgt: tgt = tgt.replace("beta", "bias") value = weight_dict[tgt].numpy() if 'output.dense.weight' in tgt or 'intermediate.dense.weight' in tgt: value = np.transpose(value, [1, 0]) print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list def split_torch_attention(state): s = list(state.keys()) for name in s: if name.endswith('attn.c_attn.weight') or name.endswith('attn.c_attn.bias'): value = state.pop(name) q, k, v = np.split(value.numpy(), 3, -1) state[name + '.q'] = torch.tensor(q, dtype=value.dtype) state[name + '.k'] = torch.tensor(k) state[name + '.v'] = torch.tensor(v) if __name__ == '__main__': parser = argparse.ArgumentParser(description="BERT convert script") parser.add_argument('--layers', type=int, default=1, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') ms_name = [ "bert.bert_encoder.encoder.blocks.{}.attention.dense1.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense1.bias", "bert.bert_encoder.encoder.blocks.{}.attention.dense2.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense2.bias", "bert.bert_encoder.encoder.blocks.{}.attention.dense3.weight", "bert.bert_encoder.encoder.blocks.{}.attention.dense3.bias", "bert.bert_encoder.encoder.blocks.{}.attention.projection.weight", "bert.bert_encoder.encoder.blocks.{}.attention.projection.bias", "bert.bert_encoder.encoder.blocks.{}.layernorm2.gamma", "bert.bert_encoder.encoder.blocks.{}.layernorm2.beta", "bert.bert_encoder.encoder.blocks.{}.output.mapping.weight", "bert.bert_encoder.encoder.blocks.{}.output.mapping.bias", "bert.bert_encoder.encoder.blocks.{}.output.projection.weight", "bert.bert_encoder.encoder.blocks.{}.output.projection.bias", "bert.bert_encoder.encoder.blocks.{}.layernorm1.gamma", "bert.bert_encoder.encoder.blocks.{}.layernorm1.beta", ] torch_name = [ "bert.encoder.layer.{}.attention.self.query.weight", "bert.encoder.layer.{}.attention.self.query.bias", "bert.encoder.layer.{}.attention.self.key.weight", "bert.encoder.layer.{}.attention.self.key.bias", "bert.encoder.layer.{}.attention.self.value.weight", "bert.encoder.layer.{}.attention.self.value.bias", "bert.encoder.layer.{}.attention.output.dense.weight", "bert.encoder.layer.{}.attention.output.dense.bias", "bert.encoder.layer.{}.attention.output.LayerNorm.gamma", "bert.encoder.layer.{}.attention.output.LayerNorm.beta", "bert.encoder.layer.{}.intermediate.dense.weight", "bert.encoder.layer.{}.intermediate.dense.bias", "bert.encoder.layer.{}.output.dense.weight", "bert.encoder.layer.{}.output.dense.bias", "bert.encoder.layer.{}.output.LayerNorm.gamma", "bert.encoder.layer.{}.output.LayerNorm.beta", ] addition_mindspore = [ "bert.word_embedding.embedding_table", "bert.embedding_postprocessor.full_position_embedding.embedding_table", "bert.embedding_postprocessor.token_type_embedding.embedding_table", "bert.embedding_postprocessor.layernorm.gamma", "bert.embedding_postprocessor.layernorm.beta", "bert.dense.weight", "bert.dense.bias", "bert.mlmloss.dense.weight", "bert.mlmloss.dense.bias", "bert.mlmloss.layernorm.gamma", "bert.mlmloss.layernorm.beta", "bert.mlmloss.vocab_dense.weight", ] addition_torch = [ "bert.embeddings.word_embeddings.weight", "bert.embeddings.position_embeddings.weight", "bert.embeddings.token_type_embeddings.weight", "bert.embeddings.LayerNorm.gamma", "bert.embeddings.LayerNorm.beta", "bert.pooler.dense.weight", "bert.pooler.dense.bias", "cls.predictions.transform.dense.weight", "cls.predictions.transform.dense.bias", "cls.predictions.transform.LayerNorm.gamma", "cls.predictions.transform.LayerNorm.beta", "cls.predictions.decoder.weight" ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) split_torch_attention(state_dict) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-mindformer/mindformers/models/blip2/000077500000000000000000000000001476650141400227325ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/blip2/__init__.py000066400000000000000000000021431476650141400250430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """blip2 init""" from .blip2_config import Blip2Config from .blip2_llama import Blip2Llama, Blip2ImageToTextGeneration from .blip2_qformer import Blip2Qformer, Blip2Classifier from .blip2_processor import Blip2ImageProcessor, Blip2Processor from .qformer import BertLMHeadModel __all__ = ['BertLMHeadModel', 'Blip2Config', 'Blip2Classifier', 'Blip2Llama', 'Blip2ImageToTextGeneration', 'Blip2Qformer', 'Blip2ImageProcessor', 'Blip2Processor'] pengcheng.mind-mindformer/mindformers/models/blip2/blip2.py000066400000000000000000000171031476650141400243160ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """ BLIP2 Base Model, contains Blip2Base, ViTModelForBlip2, as well as itm computing procedures. """ import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore.common.initializer import initializer, Normal from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.models.blip2.qformer import BertLMHeadModel from mindformers.models.vit.vit import ViTModel, ViTConfig from mindformers.modules.layers import LayerNorm class ViTModelForBlip2(ViTModel): """ ViTModel For Blip2 Models, loading a pretrained weight. forward will return the penultimate output. """ _support_list = MindFormerBook.get_config_support_list()['vit'] def __init__(self, config: ViTConfig): super(ViTModelForBlip2, self).__init__(config) self.load_checkpoint(config) def construct(self, image): return self.construct_without_pool(image) class Blip2Base(BaseModel): """ BLIP2 BaseModel, all BLIP2 models inherit this class. """ _support_list = MindFormerBook.get_model_support_list()['blip2'] def init_qformer(self): """ Init qformer for blip2 model Raises: ValueError: qformer config wrong Returns: qformer, query_tokens """ qformer_config = self.config.qformer_config qformer_config.parallel_config = self.config.parallel_config qformer = BertLMHeadModel(qformer_config) if qformer is None: raise ValueError("qformer configuration is wrong. \ please check 'qformer_config' is set in Blip2Config") query_tokens = ms.Parameter(initializer( Normal(mean=0.0, sigma=qformer_config.initializer_range), [1, qformer_config.query_length, qformer_config.hidden_size])) return qformer, query_tokens def init_vision_encoder(self): """ init vision encoder for blip2 model Raises: ValueError: vit config wrong Returns: visual_encoder, ln_vision """ vision_config = self.config.vision_config if vision_config is not None: vision_config.parallel_config = self.config.parallel_config visual_encoder = ViTModelForBlip2(vision_config) if visual_encoder is None: raise ValueError("visual_encoder configuration is wrong. \ please check 'vision_config' is set in Blip2Config") for block in visual_encoder.blocks: mapping = block.output.mapping if mapping.activation_flag and isinstance(mapping.activation, nn.GELU): mapping.activation = nn.GELU(approximate=False) ln_vision = LayerNorm(visual_encoder.config.embed_dim) return visual_encoder, ln_vision def append_text_outputs(self, text_input, text_feats, text_ids): """append_text_outputs Args: text_input: text_input, can be: 1) 2-dimension: 1 image 1 caption 2) 3-dimension: 1 image multiple captions text_feats: output to append text_ids: output to append """ if text_input.ndim == 3: for input_ids in text_input: text_embed, _ = self.forward_text(input_ids) text_feat = ops.L2Normalize(axis=1, epsilon=1e-12)(self.text_proj(text_embed[:, 0, :])) text_feats.append(text_feat) text_ids.append(input_ids) else: text_embed, _ = self.forward_text(text_input) text_feat = ops.L2Normalize(axis=1, epsilon=1e-12)(self.text_proj(text_embed[:, 0, :])) text_feats.append(text_feat) text_ids.append(text_input) def append_image_outputs(self, image_input, image_feats, vit_outputs): """append_image_outputs Args: image_input: image_input, can be: 1) 4-dimension: 1 caption 1 image 2) 5-dimension: 1 caption multiple images image_feats: output to append vit_outputs: output to append """ if image_input.ndim == 5: for image in image_input: image_embed, vit_output = self.forward_image(image) image_embed = self.vision_proj(image_embed) image_feat = ops.L2Normalize(axis=-1, epsilon=1e-12)(image_embed) image_feats.append(image_feat) vit_outputs.append(vit_output) else: image_embed, vit_output = self.forward_image(image_input) image_embed = self.vision_proj(image_embed) image_feat = ops.L2Normalize(axis=-1, epsilon=1e-12)(image_embed) image_feats.append(image_feat) vit_outputs.append(vit_output) def prepare_inputs_for_itm_eval(self, dataloader): """ prepare inputs of BLIP-2 for image_to_text_reatrieval task. Args: dataloader (GeneratorDataset): image-caption pair dataloader Returns: image_feats, text_feats, vit_outputs, text_ids, text_atts """ image_feats = [] text_feats = [] vit_outputs = [] text_ids = [] for image_input, text_input in dataloader: self.append_text_outputs(text_input, text_feats, text_ids) self.append_image_outputs(image_input, image_feats, vit_outputs) image_feats = ops.concat(image_feats, axis=0) text_feats = ops.concat(text_feats, axis=0) vit_outputs = ops.concat(vit_outputs, axis=0) text_ids = ops.concat(text_ids, axis=0) return ( image_feats, text_feats, vit_outputs, text_ids, ) def compute_extra_itm(self, extra_args, i, k_test, topk_idx, i2t=True): """ compute extra_itm score, for those model have its own itm computing method, like blip-2. Args: extra_args (tuple): required args for this method. i (int): index in the sim_matrix k_test (int): k_test number topk_idx (Tensor): topk_idx i2t (bool, optional): is image-to-text or text-to-image matching. Returns: _type_: extra itm score """ vit_outputs, text_ids = extra_args if i2t: image_inputs = ms.numpy.tile(vit_outputs[i], (k_test, 1, 1)) score = self.compute_itm( image_inputs=image_inputs, text_ids=text_ids[topk_idx], vit_computed=True ).astype(ms.float32) else: image_inputs = vit_outputs[topk_idx] score = self.compute_itm( image_inputs=image_inputs, text_ids=ms.numpy.tile(text_ids[i], (k_test, 1)), vit_computed=True).astype(ms.float32) return score pengcheng.mind-mindformer/mindformers/models/blip2/blip2_config.py000066400000000000000000000145261476650141400256510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Blip2 Config API""" from typing import Optional import mindspore.common.dtype as mstype from mindformers.models.blip2.qformer_config import QFormerConfig from mindformers.models.llama import LlamaConfig from mindformers.models.vit import ViTConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_config import BaseConfig from mindformers.mindformer_book import MindFormerBook from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig __all__ = ['Blip2Config'] default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class Blip2Config(BaseConfig): r""" Config For BLIP2 Module Args: model_type (Optional[int]): model type for blip2 model, default is 'blip2'. batch_size (Optional[int]): batch size for the model freeze_vision (Optional[bool]): whether to freeze vit weights, default is True. freeze_text (Optional[bool]): whether to freeze LLM weights, default is True. max_txt_len (Optional[int]): max text length for llama model. prompt (Optional[str]): prompt for llama model. prompt_length (Optional[int]): prompt length for llama model. checkpoint_name_or_path (Optional[str]): checkpoint path or name. dtype (Optional[str]): layer digital type, default is "float32". compute_dtype (Optional[str]): Linear layer compute dtype, default is "float16". layernorm_compute_type (Optional[str]): layernorm compute dtype, default is "float32". softmax_compute_type (Optional[str]):softmax compute dtype, default is "float32". vision_config (Optional[ViTConfig]): config for ViTModel. qformer_config (Optional[QFormerConfig]): config for qformer. text_config (Optional[LlamaConfig]): config for LLM model, like llama. parallel_config (TransformerOpParallelConfig): parallel_config. is_training (Optional[bool]): whether the model is in training state. Examples: >>> from mindformers import Blip2Config >>> Blip2Config() {'model_type': 'blip2', 'batch_size': 8, 'freeze_vision': True, 'freeze_text': True, 'max_txt_len': 32, 'checkpoint_name_or_path': None, 'prompt': '', 'prompt_length': 0 ....} """ _support_list = MindFormerBook.get_config_support_list()['blip2'] def __init__(self, model_type: str = "blip2", batch_size: int = 8, freeze_vision: bool = True, freeze_text: bool = True, max_txt_len: int = 32, prompt: bool = False, prompt_length: int = 0, checkpoint_name_or_path: str = None, dtype: str = "float32", compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", vision_config: Optional[ViTConfig] = None, qformer_config: Optional[QFormerConfig] = None, text_config: Optional[LlamaConfig] = None, parallel_config: TransformerOpParallelConfig = default_parallel_config, is_training: bool = True, **kwargs): super(Blip2Config, self).__init__(**kwargs) self.model_type = model_type self.batch_size = batch_size self.freeze_vision = freeze_vision self.freeze_text = freeze_text self.max_txt_len = max_txt_len self.checkpoint_name_or_path = checkpoint_name_or_path self.prompt = prompt self.prompt_length = prompt_length self.parallel_config = parallel_config self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.is_training = is_training self.vision_config = vision_config if vision_config is not None else ViTConfig() self.qformer_config = QFormerConfig(**qformer_config) if qformer_config is not None else QFormerConfig() self.text_config = text_config # pass configs to submodule config self.qformer_config.encoder_width = self.vision_config.embed_dim self.qformer_config.parallel_config = parallel_config self.qformer_config.compute_dtype = self.compute_dtype self.qformer_config.layernorm_dtype = self.layernorm_compute_type self.qformer_config.softmax_dtype = self.softmax_compute_type self.qformer_config.dtype = self.dtype self.vision_config.parallel_config = parallel_config self.vision_config.compute_dtype = self.compute_dtype self.vision_config.layernorm_compute_type = self.layernorm_compute_type self.vision_config.softmax_compute_type = self.softmax_compute_type self.vision_config.dtype = self.dtype # first stage is without text config if self.text_config is not None: self.text_config.parallel_config = parallel_config self.text_config.compute_dtype = self.compute_dtype self.text_config.layernorm_compute_type = self.layernorm_compute_type self.text_config.softmax_compute_type = self.softmax_compute_type self.text_config.dtype = self.dtype pengcheng.mind-mindformer/mindformers/models/blip2/blip2_llama.py000066400000000000000000000410571476650141400254710ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """Blip2 second stage pretrain and infer with Llama model""" import mindspore as ms import mindspore.common.dtype as mstype from mindspore import ops, Tensor from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.models.blip2.blip2 import Blip2Base from mindformers.models.blip2.blip2_config import Blip2Config from mindformers.models.llama import LlamaConfig from mindformers.models.llama.llama import LlamaForCausalLM, LlamaModel from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['Blip2Llama', 'Blip2ImageToTextGeneration'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaModelForBlip2(LlamaModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`], it inherits from LlamaModel and is used to adapt blip2. Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoder layer """ def __init__(self, config): super().__init__(config) # pylint: disable=W0221 def construct(self, input_embeddings: Tensor, input_attention_masks: Tensor, input_position=None, init_reset=True, batch_valid_length=None): bs, seq_len, _ = input_embeddings.shape if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): batch_valid_length = self.ones((bs, 1), mstype.int32) if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) mask = self.get_attention_mask(input_attention_masks) # mask: [bs, seq, seq] else: cur_pos = batch_valid_length - 1 valid_length = self.reshape(cur_pos, (-1, 1, 1)) freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), self.swap_mask) mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) h = input_embeddings # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForBlip2(LlamaForCausalLM): r""" Provide llama inference loss or logits through network for blip2. Args: config (LlamaConfig): The config of llama model.` Inputs: input_embeddings(Tensor): the embeddings of inputs, shape :math:`(batch, seq\_length, hidden\_size)` input_ids(Tensor): the input ids of inputs, shape: math:`(batch, seq\_length)` label_ids(Tensor): Labels for computing loss. Tensor of shape :math:`(batch, seq\_length)` attention_mask(Tensor): attention_mask, used by model.construct Returns: Tensor, the loss of the network. Examples: >>> from mindformers.models.llama import LlamaConfig >>> from mindformers.models.blip2.blip2_llama import LlamaForBlip2 >>> config = LlamaConfig(batch_size=1) >>> network = LlamaForBlip2(config=config) """ def __init__(self, config: LlamaConfig = None): checkpoint_name_or_path = config.checkpoint_name_or_path config.checkpoint_name_or_path = "" super().__init__(config) self.model = LlamaModelForBlip2(config) self.config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) self.concat_2d = P.Concat(axis=1) self.concat_3d = P.Concat(axis=1) self.not_equal = P.NotEqual() self.ones = P.Ones() self.cast = P.Cast() self.is_first_iteration = True # pylint: disable=W0221 def construct(self, input_embeddings=None, input_ids=None, labels=None, input_position=None, attention_mask=None, init_reset=True, batch_valid_length=None): """LlamaForBlip2 forward.""" if input_embeddings is None and input_ids is not None: # for incremental infer input_embeddings = self.model.tok_embeddings(input_ids) batch_size, seq_length, _ = input_embeddings.shape output = self.model(input_embeddings=input_embeddings, input_attention_masks=attention_mask, input_position=input_position, init_reset=init_reset, batch_valid_length=batch_valid_length) logits = self.lm_head(output) logits = self.cast(logits, mstype.float32) if labels is None: # inference logits = self.reshape(logits, (batch_size, seq_length, -1)) return logits, attention_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) attention_mask = self.reshape(attention_mask, (-1,)) input_mask = self.mul(attention_mask, label_mask) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss def prepare_inputs_for_generation(self, input_ids, **kwargs): if self.is_first_iteration or not self.config.use_past: image_embeddings = kwargs.pop("image_embeds") image_embeddings_atts = self.ones(image_embeddings.shape[:-1], mstype.float32) image_embeddings_length = image_embeddings.shape[1] text_input_ids = Tensor(input_ids[:, image_embeddings_length:], mstype.int32) text_embeddings = self.model.tok_embeddings(text_input_ids) text_embeddings = self.cast(text_embeddings, mstype.float32) text_embeddings_atts = self.cast(self.not_equal(text_input_ids, self.pad_token_id), mstype.float32) llama_inputs_embeds = self.concat_3d([image_embeddings, text_embeddings]) llama_inputs_attention_mask = self.concat_2d([image_embeddings_atts, text_embeddings_atts]) return { "input_ids": Tensor(input_ids, mstype.int32), "input_embeddings": llama_inputs_embeds, "attention_mask": llama_inputs_attention_mask } return { "input_ids": Tensor(input_ids, mstype.int32), "input_embeddings": None, "attention_mask": None } @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Llama(Blip2Base): """ BLIP2 Llama model """ _support_list = MindFormerBook.get_model_support_list()['blip2']['2-stg'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2Llama, self).__init__(config, **kwargs) self.config = config if config is not None else Blip2Config() self.visual_encoder, self.ln_vision = self.init_vision_encoder() if config.freeze_vision: logger.info("freeze vision encoder") for param in self.visual_encoder.trainable_params(): param.requires_grad = False self.visual_encoder.set_train(False) self.qformer, self.query_tokens = self.init_qformer() self.qformer.cls = None self.qformer.bert.embeddings.word_embeddings = None self.qformer.bert.embeddings.position_embeddings = None for layer in self.qformer.bert.encoder.layer: layer.output = None layer.intermediate = None if config.text_config.seq_length < config.max_txt_len + config.qformer_config.query_length: raise ValueError( "seq_length should be greater than sum of max_text_len and num_query_token %d, but got %d" % (config.max_txt_len + config.qformer_config.query_length, config.text_config.seq_length)) self.llama_model = LlamaForBlip2(config.text_config) if config.freeze_text: logger.info("freeze llm model") for param in self.llama_model.trainable_params(): param.requires_grad = False self.llama_model.set_train(False) dp = config.parallel_config.data_parallel self.llama_proj = Linear(in_channels=self.config.qformer_config.hidden_size, out_channels=self.config.text_config.hidden_size, param_init_type=config.dtype, compute_dtype=config.compute_dtype) if config.checkpoint_name_or_path: logger.info( "load blip2 first stage pretrained model for vision model and qformer, checkpoint_name_or_path: " "%s. pretrained llm model: %s", config.checkpoint_name_or_path, config.text_config.checkpoint_name_or_path) self.load_checkpoint(config) else: if config.vision_config.checkpoint_name_or_path: vision_checkpoint = config.vision_config.checkpoint_name_or_path else: vision_checkpoint = 'not configured' if config.text_config.checkpoint_name_or_path: text_checkpoint = config.text_config.checkpoint_name_or_path else: text_checkpoint = 'not configured' if config.qformer_config.checkpoint_name_or_path: qformer_checkpoint = config.qformer_config.checkpoint_name_or_path else: qformer_checkpoint = 'not configured' logger.info("training blip2 second stage, pretrained vision model: %s, pretrained llm model: %s, " "pretrained qformer: %s", vision_checkpoint, text_checkpoint, qformer_checkpoint) self.eos_token_id = config.text_config.eos_token_id self.pad_token_id = config.text_config.pad_token_id self.ignore_token_id = config.text_config.ignore_token_id self.max_txt_len = config.max_txt_len self.prompt = config.prompt self.prompt_length = config.prompt_length self.broadcast_to = P.BroadcastTo((dp * self.config.batch_size, self.config.qformer_config.query_length, self.config.qformer_config.hidden_size)).shard(((1, 1, 1),)) self.fill = P.Fill().shard(((dp, 1),)) self.masked_fill = P.MaskedFill().shard(((dp, 1), ())) self.ones = P.Ones().shard(((dp, 1),)) self.concat_2d = P.Concat(axis=1).shard(((dp, 1), (dp, 1))) self.concat_3d = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.not_equal = P.NotEqual().shard(((dp, 1), ())) self.slice = P.StridedSlice().shard(((dp, 1),)) self.cast = P.Cast() # pylint: disable=W0221 def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor): """Blip2Llama forward.""" projected_qformer_output = self.forward_qformer_and_proj(image) projected_qformer_output_atts = self.ones(projected_qformer_output.shape[:-1], mstype.float32) batch_size, seq_length = text_input_ids.shape tokens = self.slice(text_input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) text_label_ids = self.slice(text_input_ids, (0, 1), (batch_size, seq_length), (1, 1)) targets = ops.masked_fill(text_label_ids, text_label_ids == self.pad_token_id, self.ignore_token_id) if self.prompt: prompt_label = self.ignore_token_id * self.ones((batch_size, self.prompt_length), mstype.int32) targets = self.concat_2d([prompt_label, targets[:, self.prompt_length:]]) empty_targets = self.fill(mstype.int32, projected_qformer_output_atts.shape, self.ignore_token_id) targets = self.concat_2d([empty_targets, targets]) # [batch_size, 1, query_size + max_txt_length] # [batch_size, max_txt_length, llama_hidden_size] text_inputs_embeds = self.llama_model.model.tok_embeddings(tokens) text_inputs_embeds = self.cast(text_inputs_embeds, mstype.float32) text_inputs_atts = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) # [batch_size, query_size + max_txt_length, llama_hidden_size] llama_inputs_embeds = self.concat_3d([projected_qformer_output, text_inputs_embeds]) # [batch_size, query_size + max_txt_length] llama_inputs_attention_mask = self.concat_2d([projected_qformer_output_atts, text_inputs_atts]) loss = self.llama_model( input_embeddings=llama_inputs_embeds, labels=targets, attention_mask=llama_inputs_attention_mask ) return loss def forward_qformer_and_proj(self, image: ms.Tensor): """forward the image tensor to the qformer, then project the output to adapt the dimension""" image_embeds = self.visual_encoder(image) image_embeds = self.ln_vision(image_embeds) # [batch_size, vit_seq_length, vit_encoder_hidden_width] image_atts = self.ones(image_embeds.shape[:-1], mstype.float32) # [batch_size, vit_seq_length] query_tokens = self.broadcast_to(self.query_tokens) # [batch_size, query_size, qformer_hidden_size] query_output = self.qformer.bert(query_embeds=query_tokens, encoder_hidden_states=image_embeds, encoder_attention_mask=image_atts, use_cache=True, return_dict=True) # [batch_size, query_size, qformer_hidden_size] -> [batch_size, query_size, llama_hidden_size] return self.llama_proj(query_output[0]) @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2ImageToTextGeneration(Blip2Llama): """ Blip2ImageToTextGeneration rely on Blip2Llama, used for image to text genearation. Usage: >>> from mindformers import AutoModel >>> model_type = 'itt_blip2_stage2_vit_g_llama_7b' >>> model = AutoModel.from_pretrained(model_type) """ _support_list = MindFormerBook.get_model_support_list()['itt']['blip2'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2ImageToTextGeneration, self).__init__(config, **kwargs) self.llama_model.set_train(False) self.one_prefix = ops.Ones() def generate_text_for_image(self, image: ms.Tensor, prompt_input_ids: ms.Tensor): batch_size = image.shape[0] prefix_ones = self.one_prefix((batch_size, self.config.qformer_config.query_length), mstype.int32) text_input_ids = self.concat_2d([prefix_ones, prompt_input_ids]) projected_qformer_output = self.forward_qformer_and_proj(image) output_ids = self.llama_model.generate(input_ids=text_input_ids.asnumpy(), image_embeds=projected_qformer_output) return output_ids pengcheng.mind-mindformer/mindformers/models/blip2/blip2_processor.py000066400000000000000000000157521476650141400264250ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BLIP2Processor """ from typing import Optional, Union, List import numpy as np import PIL import PIL.Image import mindspore as ms from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.dataset.transforms.vision_transforms import ( BatchPILize, BatchResize, BatchToTensor, BatchNormalize ) from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class Blip2ImageProcessor(BaseImageProcessor): """ Blip2ImageProcessor. Args: image_resolution (int): The target size. Examples: >>> from mindformers import Blip2ImageProcessor >>> from mindformers.tools.image_tools import load_image >>> processor = Blip2ImageProcessor(image_size=224) >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> processor(image) Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.55868769e+00, -1.52949083e+00, ... -1.48569560e+00, -1.48569560e+00], [-1.54408932e+00, -1.52949083e+00, ... -1.50029397e+00, -1.50029397e+00], [-1.52949083e+00, -1.52949083e+00, ... -1.50029397e+00, -1.50029397e+00], ... [-1.38067937e+00, -1.48021984e+00, ... -1.30957901e+00, -1.40911949e+00], [-1.46599972e+00, -1.43755960e+00, ... -1.48021984e+00, -1.43755960e+00], [-1.40911949e+00, -1.28113890e+00, ... -1.48021984e+00, -1.43755960e+00]]]]) """ def __init__(self, image_size: Optional[int] = 224, interpolation: Optional[str] = 'bicubic', mean=None, std=None, is_hwc=False): self.pilize = BatchPILize() super(Blip2ImageProcessor, self).__init__() if isinstance(image_size, int): image_size = (image_size,) * 2 self.resize = BatchResize(image_size, interpolation=interpolation) self.to_tensor = BatchToTensor() if mean is None: mean = (0.48145466, 0.4578275, 0.40821073) if std is None: std = (0.26862954, 0.26130258, 0.27577711) self.normalize = BatchNormalize(mean, std, is_hwc) def preprocess(self, images: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]], **kwargs): r""" Preprocess Required By Base Processor. Args: images (ms.Tensor, PIL.Image, numpy.array, List[PIL.Image]): A batch of images. Return: A 4-rank tensor for a batch of images. """ images = self.pilize(images) images = self.resize(images) images = self.to_tensor(images) images = self.normalize(images) kwargs.pop("other", None) if isinstance(images, list): return ms.Tensor(np.row_stack([np.expand_dims(item, axis=0) for item in images])) if len(images.shape) == 4: return ms.Tensor(images) return ms.Tensor(np.expand_dims(images, axis=0)) def _bhwc_check(self, image_batch: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]]): r"""Bhwc_check""" if isinstance(image_batch, np.ndarray): if image_batch.shape[-1] == 3: return True if isinstance(image_batch, ms.Tensor): if image_batch.asnumpy().shape[-1] == 3: return True if isinstance(image_batch, (list, PIL.Image.Image)): return True return False @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class Blip2Processor(BaseProcessor): r"""Blip2 Processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. Args: image_processor (BaseImageProcessor): Used for process image data. tokenizer (BaseTokenizer): Used for process text data. max_length (Optional[int]): The length of text tokens. padding (Optional[str]): The padding strategy of tokenizer, [None, "max_length"]. return_tensors (Optional[str]): The type of returned tensors for tokenizer, [None, "ms"]. Examples: >>> from mindformers import Blip2Processor >>> from mindformers.tools.image_tools import load_image >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> text = ["a boy", "a girl"] >>> Blip2Processor.show_support_list() INFO - support list of Blip2Processor is: INFO - ['blip2_stage1_vit_g', 'blip2_stage1_classification'] INFO - ------------------------------------- >>> processor = Blip2Processor.from_pretrained('blip2_stage1_vit_g') INFO - processor built successfully! >>> processor(image, text) {'image': Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.55868769e+00, -1.52949083e+00, -1.55868769e+00 ... -1.48569560e+00, -1.48569560e+00], [-1.54408932e+00, -1.52949083e+00, -1.54408932e+00 ... -1.50029397e+00, -1.50029397e+00], [-1.52949083e+00, -1.52949083e+00, -1.52949083e+00 ... -1.50029397e+00, -1.50029397e+00], ... [-1.38067937e+00, -1.48021984e+00, -1.38067937e+00 ... -1.30957901e+00, -1.40911949e+00], [-1.46599972e+00, -1.43755960e+00, -1.26691878e+00 ... -1.48021984e+00, -1.43755960e+00], [-1.40911949e+00, -1.28113890e+00, -1.30957901e+00 ... -1.48021984e+00, -1.43755960e+00] ]]]), 'text': Tensor(shape=[2, 32], dtype=Int32, value= [[ 101, 1037, 2879 ... 0, 0, 0], [ 101, 1037, 2611 ... 0, 0, 0]])} """ _support_list = MindFormerBook.get_processor_support_list()['blip2'] def __init__(self, image_processor, tokenizer, max_length=32, padding='max_length', return_tensors='ms'): super(Blip2Processor, self).__init__( image_processor=image_processor, tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors) pengcheng.mind-mindformer/mindformers/models/blip2/blip2_qformer.py000066400000000000000000000531171476650141400260560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/salesforce/LAVIS/tree/main/lavis/models/blip2_models # ============================================================================ """ Blip2 Qformer, link to ViT. the main model for image-text pretraining. """ import os import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore import Tensor import mindspore.numpy as np from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.modules.layers import Linear from mindformers.models.blip2.blip2 import Blip2Base from mindformers.models.blip2.blip2_config import Blip2Config from mindformers.models.blip2.qformer import CrossEntropyLoss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger def locate_card(rank_id: Tensor, group_size: int = 1): """ locate card number for ops.BroadCast, for graph mode. """ for i in range(0, group_size): if i == rank_id: return i return 0 def choose_idx_with_prob(weight: Tensor, batch_size: int = 1): """ choose idx depend on probability, replace torch.multinomial """ weight_acc = ops.cumsum(weight, -1) rand_x = np.rand([1], dtype=weight_acc.dtype) * weight_acc[-1] idx = np.argmax(weight_acc > rand_x) rank_id = idx // batch_size data_id = idx % batch_size return rank_id, data_id @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Qformer(Blip2Base): """ BLIP2 first-stage model with Q-former and ViT. Usage: >>> from mindformers.models.blip2 import Blip2Qformer >>> model = Blip2Qformer.from_pretrained("blip2_stage1_vit_g") """ _support_list = MindFormerBook.get_model_support_list()['blip2']['1-stg'] def __init__(self, config: Blip2Config, **kwargs): super(Blip2Qformer, self).__init__(config, **kwargs) self.config = config if config is not None else Blip2Config() self.group_size = int(os.getenv('RANK_SIZE', '1')) self.rank = int(os.getenv('RANK_ID', '0')) self.visual_encoder, self.ln_vision = self.init_vision_encoder() if config.freeze_vision: for _, cell in self.visual_encoder.cells_and_names(): params = cell.get_parameters() for param in params: param.requires_grad = False self.visual_encoder.set_train(False) logger.info("freeze vision encoder") self.qformer, self.query_tokens = self.init_qformer() qformer_config = self.config.qformer_config # note special token added: bos_token -> [DEC] self.qformer.resize_token_embeddings(qformer_config.vocab_size + 1) params = self.qformer.get_parameters() # modify layer names for param in params: if "_query" in param.name: key_orig = param.name.replace("_query", "") param.set_data(self.qformer.parameters_dict().get(key_orig)) # parallel settings if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.vision_proj = Linear(in_channels=qformer_config.hidden_size, out_channels=qformer_config.head_embed_dim, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.vision_proj.shard(strategy_matmul=((dp, mp), (1, mp))) self.text_proj = Linear(in_channels=qformer_config.hidden_size, out_channels=qformer_config.head_embed_dim, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.text_proj.shard(strategy_matmul=((dp, mp), (1, mp))) self.itm_head = Linear(in_channels=qformer_config.hidden_size, out_channels=2, param_init_type=config.dtype, compute_dtype=config.compute_dtype) self.itm_head.shard(strategy_matmul=((dp, mp), (1, mp))) self.matmul = P.BatchMatMul() self.matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.concat = ops.concat self.expand_dims = ops.expand_dims self.transpose = ops.transpose self.zeros = ops.zeros self.ones = ops.ones self.linspace = ops.linspace self.floor = ops.floor self.softmax = nn.Softmax(axis=1) self.softmax.softmax.shard(((dp, mp, 1),)) self.eye = ops.eye self.masked_fill = ops.masked_fill self.stack = ops.stack self.broadcast_to = ops.broadcast_to self.tile = P.Tile() self.tile.shard(((dp, mp, 1, 1),)) self.normalize = ops.L2Normalize(axis=-1, epsilon=1e-12) self.temp = ms.Parameter( Tensor(0.07, dtype=config.compute_dtype), requires_grad=True) self.max_txt_len = config.max_txt_len self.bos_token_id = qformer_config.bos_token_id self.pad_token_id = qformer_config.pad_token_id if self.group_size > 1: self.all_gather = ops.AllGather() self.broadcast = ops.Broadcast self.not_equal = P.NotEqual() self.cast = P.Cast() self.itc_loss = CrossEntropyLoss(label_smoothing=0.1) self.itm_loss = CrossEntropyLoss() def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor, return_tuple: bool = False): """ forwarding image and text, compute itc, itm and lm losses. """ image_feats, image_embeds, past_key_values = self.forward_image( image, use_cache=True) image_feats = self.normalize(self.vision_proj(image_feats)) text_embeds, text_attention_mask = self.forward_text(text_input_ids) text_feat = self.normalize(self.text_proj(text_embeds[:, 0, :])) image_feats = self.cast(image_feats, mstype.float16) text_feat = self.cast(text_feat, mstype.float16) ### ============== Image-text Contrastive ===================### # if/else branch: distribute setting if self.group_size > 1: # [batch_size*num_gpu, num_query_tokens, embed_dim] image_feats_all = self.all_gather(image_feats) # [batch_size*num_gpu, embed_dim] text_feat_all = self.all_gather(text_feat) else: image_feats_all = image_feats text_feat_all = text_feat batch_size = image.shape[0] sim_q2t = [] for i in range(self.group_size): text_feat_part = text_feat_all[i * batch_size: (i + 1) * batch_size] sim_temp = self.matmul(self.expand_dims(image_feats, 1), self.expand_dims( self.expand_dims(text_feat_part, -1), 0)).squeeze(-1) sim_q2t.append(sim_temp.max(-1)) # query-text similarity: [batch_size, batch_size*num_gpu] sim_q2t = self.concat(sim_q2t, axis=1) # image-text similarity: aggregate across all query tokens sim_i2t = sim_q2t / self.temp sim_t2q = [] # align with ops.matmul, x1 -> [batch_size, batch_size, 1, embed_dim] text_feat = self.tile(self.expand_dims( self.expand_dims(text_feat, 1), 1), (1, batch_size, 1, 1)) for i in range(self.group_size): image_feats_part = image_feats_all[i * batch_size: (i + 1) * batch_size] # align with ops.matmul, x2 -> [batch_size, batch_size, embed_dim, num_query_tokens] image_feats_part = self.tile(self.expand_dims( self.transpose(image_feats_part, (0, 2, 1)), 0), (batch_size, 1, 1, 1)) # compute similarity same as ops.matmul sim_temp = self.matmul(text_feat, image_feats_part).squeeze(2) sim_t2q.append(sim_temp.max(-1)) # text-query similarity: [batch_size, batch_size*num_gpu] sim_t2q = self.concat(sim_t2q, axis=1) # text-image similarity: aggregate across all query tokens sim_t2i = sim_t2q / self.temp targets = self.floor(self.linspace(ms.Tensor(self.rank * batch_size, mstype.float32), ms.Tensor(self.rank * batch_size + batch_size - 1, mstype.float32), batch_size)).astype(mstype.int32) sim_i2t = self.cast(sim_i2t, mstype.float32) sim_t2i = self.cast(sim_t2i, mstype.float32) loss_itc = (self.itc_loss(sim_i2t, targets) + self.itc_loss(sim_t2i, targets)) / 2 # ============== Image-text Matching =================== # mask text-image similarity as weights weights_t2i, weights_i2t = self.fill_masked_weight(sim_t2i, sim_i2t, batch_size) # choose negative image/text for each text/image image_embeds_neg, text_ids_neg = self.choose_negative_targets(weights_t2i, weights_i2t, batch_size, image_embeds, text_input_ids) text_ids_all = self.concat( [text_input_ids, text_input_ids, text_ids_neg], axis=0) # pos, pos, neg image_embeds_all = self.concat( [image_embeds, image_embeds_neg, image_embeds], axis=0) # pos, neg, pos vl_embeddings = self.forward_text_and_image( image_embeds_all, text_ids_all, vit_computed=True) vl_output = self.itm_head(vl_embeddings) logits = vl_output.mean(axis=1) itm_labels = self.concat( [self.ones(batch_size, mstype.int32), self.zeros(2 * batch_size, mstype.int32)], axis=0 ) loss_itm = self.itm_loss(logits, itm_labels) # ================= Image Captioning ======================== decoder_input_ids = text_input_ids.copy().astype(mstype.float32) decoder_input_ids[:, 0] = self.bos_token_id decoder_input_ids = decoder_input_ids.astype(mstype.int32) labels = decoder_input_ids.masked_fill( decoder_input_ids == self.pad_token_id, -100 ) query_tokens = self.broadcast_to(self.query_tokens, (image_embeds.shape[0], -1, -1)) query_atts = self.ones(query_tokens.shape[:-1], mstype.float32) attention_mask = self.concat( [query_atts, text_attention_mask.astype(mstype.float32)], axis=1) lm_output = self.qformer( decoder_input_ids, attention_mask=attention_mask, past_key_values=past_key_values, return_dict=True, labels=labels, ) loss_lm = lm_output[0] loss = loss_itc + loss_itm + loss_lm if return_tuple: return ( loss, loss_itc, loss_itm, loss_lm ) return loss def forward_image(self, image, use_cache=False): """ forawrd image through vit and the bert model. Args: image (Tensor): input image use_cache (bool, optional): whether to return past_key_values. Returns: hidden_states, image_embeds_frozen, past_key_values (optional) """ image_embeds_frozen = self.ln_vision(self.visual_encoder(image)) image_atts = self.ones( image_embeds_frozen.shape[:-1], mstype.float32) query_tokens = self.broadcast_to( self.query_tokens, (image_embeds_frozen.shape[0], -1, -1)) query_output = self.qformer.bert( query_embeds=query_tokens, encoder_hidden_states=image_embeds_frozen, encoder_attention_mask=image_atts, use_cache=use_cache, ) if use_cache: return query_output[0], image_embeds_frozen, query_output[1] return query_output[0], image_embeds_frozen def forward_text(self, text_input_ids): """ forawrd text_ids through the bert model. Args: text_input_ids (Tensor): input text_ids Returns: text embeddings and mask """ attention_mask = self.cast(self.not_equal( text_input_ids, self.pad_token_id), mstype.float32) text_output = self.qformer.bert( text_input_ids, attention_mask=attention_mask, return_dict=True ) # text embeddings and mask return text_output[0], attention_mask def forward_text_and_image(self, image_inputs, text_ids, vit_computed=False): """ forward text and image at the same time to the bert model. Args: image_inputs(Tensor): input image or image embeds (computed) text_input_ids (Tensor): input text_ids vit_computed (bool, optional): whether image embeds is computed Returns: multimodal embeddings """ if not vit_computed: image_embeds_frozen = self.ln_vision( self.visual_encoder(image_inputs)) else: image_embeds_frozen = image_inputs text_atts = self.cast(self.not_equal( text_ids, self.pad_token_id), mstype.float32) image_atts = self.ones( image_embeds_frozen.shape[:-1], mstype.float32) query_tokens = self.broadcast_to( self.query_tokens, (image_embeds_frozen.shape[0], -1, -1)) query_atts = self.ones(query_tokens.shape[:-1], mstype.float32) attention_mask = self.concat([query_atts, text_atts], axis=1) output_itm = self.qformer.bert( text_ids, query_embeds=query_tokens, attention_mask=attention_mask, encoder_hidden_states=image_embeds_frozen, encoder_attention_mask=image_atts, return_dict=True ) # multimodal embeddings multimodal_embeds = output_itm[0][:, : query_tokens.shape[1], :] return multimodal_embeds def fill_masked_weight(self, sim_t2i, sim_i2t, batch_size): """return masked weights based on similarity Args: sim_t2i (Tensor): text-to-image similarity sim_i2t (Tensor): image-to-text similarity batch_size (int): current batch size """ weights_t2i = self.softmax(sim_t2i) + 1e-4 diag_fill_mask_t2i = self.eye(weights_t2i.shape[0], batch_size, mstype.bool_) filled_weights_t2i = self.masked_fill( weights_t2i[:, self.rank * batch_size: self.rank * batch_size + batch_size], diag_fill_mask_t2i, 0) weights_t2i[:, self.rank * batch_size: self.rank * batch_size + batch_size] = filled_weights_t2i weights_i2t = self.softmax(sim_i2t) + 1e-4 diag_fill_mask_i2t = self.eye(weights_i2t.shape[0], batch_size, mstype.bool_) filled_weights_i2t = self.masked_fill( weights_i2t[:, self.rank * batch_size: self.rank * batch_size + batch_size], diag_fill_mask_i2t, 0) weights_i2t[:, self.rank * batch_size: self.rank * batch_size + batch_size] = filled_weights_i2t return weights_t2i, weights_i2t def choose_negative_targets(self, weights_t2i, weights_i2t, batch_size, image_embeds, text_input_ids): """choose negative targets for each image/text. Args: weights_t2i (Tensor): masked text-to-image weights weights_i2t (Tensor): masked image-to-text weights batch_size (int): current batch size image_embeds (Tensor): image embeddings text_input_ids (Tensor): text ids Returns: image_embeds_neg (Tensor): negative image_embeds text_ids_neg (Tensor): negative text ids """ # select a negative image for each text image_embeds_neg = [] for i in range(batch_size): rank_id, data_id = choose_idx_with_prob(weights_t2i[i], batch_size) if self.group_size > 1: card_id = locate_card(rank_id, self.group_size) image_embeds_temp = self.broadcast(card_id)((image_embeds,))[0] else: image_embeds_temp = image_embeds image_embeds_neg.append(image_embeds_temp[data_id]) image_embeds_neg = self.stack(image_embeds_neg, axis=0) # select a negative text for each image text_ids_neg = [] for i in range(batch_size): rank_id, data_id = choose_idx_with_prob(weights_i2t[i], batch_size) if self.group_size > 1: card_id = locate_card(rank_id, self.group_size) text_input_ids_temp = self.broadcast( card_id)((text_input_ids,))[0] else: text_input_ids_temp = text_input_ids text_ids_neg.append(text_input_ids_temp[data_id]) text_ids_neg = self.stack(text_ids_neg, axis=0) return image_embeds_neg, text_ids_neg def compute_itm(self, image_inputs, text_ids, vit_computed=False): """ compute image-text matching scores for the model. Args: image_inputs (Tensor): input image or image embeds (computed) text_ids (Tensor): input text_ids vit_computed (bool, optional): whether image embeds is computed Returns: itm_logit """ vl_embeddings = self.forward_text_and_image( image_inputs, text_ids, vit_computed) itm_logit = self.itm_head(vl_embeddings) itm_logit = itm_logit[:, :, 1].mean(axis=1) return itm_logit def get_image_feature(self, image, output_past_keys=False): """extract image feature""" forward_image_outputs = self.forward_image(image, output_past_keys) image_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.vision_proj(forward_image_outputs[0])) return image_features def get_text_feature(self, input_ids): """extract text feature""" forward_text_outputs = self.forward_text(input_ids) text_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.text_proj(forward_text_outputs[0])) return text_features def extract_features(self, samples, mode="multimodal"): """ extract feature as well as embeds by given mode, Args: samples (tuple of Tensors): image/text input mode (str): [image, text, multimodal] """ image = samples.get("image") text_ids = samples.get("text_input") # assert mode is one of "image", "text", "multimodal" assert mode in [ "image", "text", "multimodal", ], "mode must be one of 'image', 'text', 'multimodal'" # initialize output image_embeds, text_embeds, multimodal_embeds = None, None, None image_features, text_features = None, None if mode == "image": assert ( image is not None ), "Image is not provided for mode 'image' or 'multimodal'" # return query features forward_image_outputs = self.forward_image(image, use_cache=False) image_embeds = forward_image_outputs[0] image_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.vision_proj(image_embeds)) elif mode == "text": assert text_ids is not None, "text input is None for mode 'text' or 'multimodal'" forward_text_outputs = self.forward_text(text_ids) text_embeds = forward_text_outputs[0] text_features = ops.L2Normalize( axis=-1, epsilon=1e-12)(self.text_proj(text_embeds)) elif mode == "multimodal": # return multimodal query features multimodal_embeds = self.forward_text_and_image( image, text_ids, vit_computed=False) return (image_embeds, image_features, text_embeds, text_features, multimodal_embeds) @MindFormerRegister.register(MindFormerModuleType.MODELS) class Blip2Classifier(Blip2Qformer): """ Blip2Classifier rely on Blip2Qformer, used for zero-shot classification. Usage: >>> from mindformers import AutoModel >>> model_type = 'blip2_stage1_classification' >>> model = AutoModel.from_pretrained(model_type) """ def __init__(self, config: Blip2Config, **kwargs): super(Blip2Classifier, self).__init__(config, **kwargs) self.load_checkpoint(config) def construct(self, image: ms.Tensor, text_input_ids: ms.Tensor, return_tuple: bool = False): image_features = self.get_image_feature(image)[:, 0] text_features = self.get_text_feature(text_input_ids)[:, 0] sims = ops.matmul(image_features, text_features.T) / self.temp return sims, sims.T # no label as input (compare to CLIP) pengcheng.mind-mindformer/mindformers/models/blip2/convert_weight.py000066400000000000000000000126171476650141400263420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert checkpoint from salesforce.""" from collections import OrderedDict import argparse import mindspore as ms import mindspore.ops as P from mindspore import Tensor import torch def convert_weight(torch_path="blip2_pretrain.pth", mindspore_path="blip2_pretrain.ckpt"): r"""Convert Weight Convert blip2_qformer weights from pytorch to mindspore, pytorch (CPU) required. Args: torch_path: The torch path to blip2_qformer. mindspore_path: The save path for blip2_qformer. Returns: the converted mindspore_model_weight for Blip2Qformer class. Notice: when loading, warning of: ""2 parameters 'visual_encoder.fc_norm.gamma', 'visual_encoder.fc_norm.beta' ara not loaded. " is normal, because bilp2 utilizes the penultimate layer output of the eva_vit. """ pt_params = torch.load(torch_path, map_location='cpu') if not isinstance(pt_params, OrderedDict): if isinstance(pt_params, dict) and 'model' in pt_params.keys(): pt_params = pt_params['model'] else: raise ValueError("wrong torch state_dict format when loading {},\ please check.".format(torch_path)) name_pt2ms = { "cls_token": "cls_tokens", "attn.proj": "attention.projection", "attn.q_bias": "attention.dense1.bias", "attn.v_bias": "attention.dense3.bias", "norm1.weight": "layernorm1.gamma", "norm1.bias": "layernorm1.beta", "norm2.weight": "layernorm2.gamma", "norm2.bias": "layernorm2.beta", "fc_norm.weight": "fc_norm.gamma", "fc_norm.bias": "fc_norm.beta", "ln_vision.weight": "ln_vision.gamma", "ln_vision.bias": "ln_vision.beta", "mlp.fc2.weight": "output.mapping.weight", "mlp.fc1.bias": "output.mapping.bias", "mlp.fc1.weight": "output.projection.weight", "mlp.fc2.bias": "output.projection.bias", "LayerNorm.": "layernorm.", "layernorm.weight": "layernorm.gamma", "layernorm.bias": "layernorm.beta", "embeddings.weight": "embeddings.embedding_table", "self": "self_att", } ms_param_dict = [] for pt_name, pt_tensor in pt_params.items(): # initial name assign ms_name = pt_name # extract data numpy_value = pt_params[pt_name].detach().numpy() data = Tensor.from_numpy(numpy_value) # split qkv weights if "qkv.weight" in pt_name: length = pt_tensor.shape[0] // 3 ms_name1 = pt_name.replace("attn.qkv", "attention.dense1") ms_name2 = pt_name.replace("attn.qkv", "attention.dense2") ms_name3 = pt_name.replace("attn.qkv", "attention.dense3") ms_param_dict.append({"name": ms_name1, "data": data[:length]}) ms_param_dict.append({"name": ms_name2, "data": data[length:length*2]}) ms_param_dict.append({"name": ms_name3, "data": data[length*2:length*3]}) print("rename {} to {}, {} and {}".format(pt_name, ms_name1, ms_name2, ms_name3)) else: # Rename for replace_from, replace_to in name_pt2ms.items(): ms_name = ms_name.replace(replace_from, replace_to) ms_name = ms_name.replace("Qformer.", "qformer.") if ms_name.endswith("output.mapping.weight") or \ ms_name.endswith("output.projection.weight") or \ ms_name.endswith("attention.projection.weight"): data = data.T ms_param_dict.append({"name": ms_name, "data": data}) # when loading each query-bias, append a zero value key-bias. if ms_name.endswith("attention.dense1.bias"): ms_param_dict.append({ "name": ms_name.replace("dense1", "dense2"), "data": P.zeros_like(data) }) if ms_name != pt_name: print(f"rename {pt_name} to {ms_name}") print(f"\n----------------- convert {torch_path} to {mindspore_path} Finished! -----------------\n") ms.save_checkpoint(ms_param_dict, mindspore_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="blip2 weight convert script") parser.add_argument("--torch_path", type=str, default="blip2_pretrain.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="blip2_pretrain.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-mindformer/mindformers/models/blip2/qformer.py000066400000000000000000002061451476650141400247670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """qformer implementation.""" import math from collections import OrderedDict from typing import Optional import os import mindspore.common.dtype as mstype import mindspore.numpy as np import mindspore.ops as P from mindspore.train.serialization import load_checkpoint from mindspore import nn, Parameter, Tensor from mindspore.common.initializer import initializer, Normal, Zero from mindspore.nn import LossBase from mindformers.mindformer_book import MindFormerBook from mindformers.tools.download_tools import download_with_progress_bar from mindformers.tools.logger import logger from mindformers.tools.utils import try_sync_file from mindformers.models.base_model import BaseModel from mindformers.models.bert.bert_config import BertConfig from mindformers.models.blip2.qformer_config import QFormerConfig from mindformers.modules.layers import Dropout, LayerNorm, Linear class CrossEntropyLoss(LossBase): """Label smoothing align with torch implementation.""" def __init__(self, weight=None, ignore_index=-100, reduction='mean', label_smoothing=0.0): super().__init__(reduction) self.weight = weight self.ignore_index = ignore_index self.reduction = reduction self.label_smoothing = label_smoothing self.log_softmax = nn.LogSoftmax(axis=-1) self.cross_entropy = nn.CrossEntropyLoss( reduction=self.reduction, label_smoothing=0.0) @staticmethod def reduce_loss(loss, reduction='mean'): """reduce_loss. Args: loss (float): loss reduction (str, optional): reduction method. Defaults to 'mean'. Returns: reduced_loss: reduced_loss """ if reduction == 'mean': return loss.mean() if reduction == 'sum': return loss.sum() return loss def construct(self, logits, labels): loss_fct = self.cross_entropy(logits, labels) log_preds = self.log_softmax(logits) loss = self.reduce_loss(-log_preds.sum(axis=-1), reduction=self.reduction) n_class = logits.shape[-1] return (loss / n_class) * self.label_smoothing + loss_fct * (1 - self.label_smoothing) ACT2CLS = { "gelu": nn.GELU, "gelu_fast": nn.FastGelu, "relu": nn.ReLU, "relu6": nn.ReLU6, "sigmoid": nn.Sigmoid, "tanh": nn.Tanh, } class ClassInstanter(OrderedDict): """ClassInstanter for OrderedDict func-mapping input. Args: OrderedDict : function mapping. """ def __getitem__(self, key): content = super().__getitem__(key) cls, kwargs = content if isinstance(content, tuple) else (content, {}) return cls(**kwargs) ACT2FN = ClassInstanter(ACT2CLS) def recursive_apply(module: nn.Cell, function_call): """apply cetain function to a nn.Cell module, recursively. Args: module (nn.Cell): model input. fn (function): function call """ for submodule in module.cells(): recursive_apply(submodule, function_call) function_call(module) class BertEmbeddings(nn.Cell): """forward the embeddings from word and position embeddings.""" def __init__(self, config): super().__init__() self.word_embeddings = nn.Embedding( config.vocab_size, config.hidden_size, padding_idx=config.pad_token_id ) self.position_embeddings = nn.Embedding( config.max_position_embeddings, config.hidden_size ) # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load # any TensorFlow checkpoint file self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps) self.layernorm.shard(((config.parallel_config.data_parallel, 1, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) self.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.concat = P.Concat(axis=1) # position_ids (1, len position emb) is contiguous in memory and exported when serialized self.expand_x = P.BroadcastTo((1, config.max_position_embeddings)) position_embeds = Tensor( [i for i in range(config.max_position_embeddings)], dtype=mstype.int32) self.position_ids = Parameter( self.expand_x(position_embeds), requires_grad=False ) self.position_embedding_type = getattr( config, "position_embedding_type", "absolute" ) def construct(self, input_ids=None, position_ids=None, query_embeds=None, past_key_values_length=0): """forward the embeddings from word and position embeddings.""" if input_ids is not None: seq_length = input_ids.shape[1] else: seq_length = 0 if position_ids is None: position_ids = self.position_ids[:, past_key_values_length: seq_length + past_key_values_length].copy() if input_ids is not None: embeddings = self.word_embeddings(input_ids) if self.position_embedding_type == "absolute": position_embeddings = self.position_embeddings(position_ids) embeddings = embeddings + position_embeddings if query_embeds is not None: embeddings = self.concat((query_embeds, embeddings)) else: embeddings = query_embeds # [bz, query_size, qformer_hidden_size] embeddings = self.layernorm(embeddings) embeddings = self.dropout(embeddings) return embeddings class BertSelfAttention(nn.Cell): """ BertSelfAttention """ def __init__(self, config, is_cross_attention): super().__init__() self.config = config self.dtype = config.dtype self.softmax_dtype = config.softmax_dtype self.compute_dtype = config.compute_dtype if config.hidden_size % config.num_attention_heads != 0 and not hasattr(config, "embedding_size"): raise ValueError( "The hidden size (%d) is not a multiple of the number of attention " "heads (%d)" % (config.hidden_size, config.num_attention_heads) ) self.num_attention_heads = config.num_attention_heads self.attention_head_size = int( config.hidden_size / config.num_attention_heads) self.all_head_size = self.num_attention_heads * self.attention_head_size if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.query = Linear(in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.query.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) if is_cross_attention: self.key = Linear( in_channels=config.encoder_width, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.value = Linear( in_channels=config.encoder_width, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) else: self.key = Linear( in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.value = Linear( in_channels=config.hidden_size, out_channels=self.all_head_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.key.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.value.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.dropout = Dropout(1. - config.attention_probs_dropout_prob) self.position_embedding_type = getattr( config, "position_embedding_type", "absolute") if self.position_embedding_type == "relative_key" \ or self.position_embedding_type == "relative_key_query": self.max_position_embeddings = config.max_position_embeddings self.distance_embedding = nn.Embedding(2 * config.max_position_embeddings - 1, self.attention_head_size) self.save_attention = False self.einsum = P.Einsum("bhld,lrd->bhlr") self.einsum2 = P.Einsum("bhrd,lrd->bhlr") self.divider = math.sqrt(self.attention_head_size) self.cast = P.Cast() self.concat = P.Concat(axis=2) self.batch_matmul = P.BatchMatMul().shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.transpose = P.Transpose().shard(((1, 1, 1, 1),)) def transpose_for_scores(self, x: Tensor) -> Tensor: """ transpose input for scores output. Args: x (Tensor): input Returns: Tensor: output """ new_x_shape = x.shape[:-1] + ( self.num_attention_heads, self.attention_head_size, ) x = x.view(*new_x_shape) return self.transpose(x, (0, 2, 1, 3)) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, ): """ BertSelfAttention forwarding """ # If this is instantiated as a cross-attention module, the keys # and values come from an encoder; the attention mask needs to be # such that the encoder's padding tokens are not attended to. is_cross_attention = encoder_hidden_states is not None hidden_states = self.cast(hidden_states, self.compute_dtype) if is_cross_attention: # [batch_size, vit_seq_length, encoder_hidden_width] encoder_hidden_states = self.cast( encoder_hidden_states, self.compute_dtype) # [batch_size, num_attention_heads, vit_seq_length, attention_head_size] key_layer = self.transpose_for_scores( self.key(encoder_hidden_states)) # [batch_size, num_attention_heads, vit_seq_length, attention_head_size] value_layer = self.transpose_for_scores( self.value(encoder_hidden_states)) attention_mask = encoder_attention_mask elif past_key_value is not None: key_layer = self.transpose_for_scores(self.key(hidden_states)) value_layer = self.transpose_for_scores(self.value(hidden_states)) key_layer = self.concat([past_key_value[0], key_layer]) value_layer = self.concat([past_key_value[1], value_layer]) else: # [batch_size, num_attention_heads, query_size, attention_head_size] key_layer = self.transpose_for_scores(self.key(hidden_states)) # [batch_size, num_attention_heads, query_size, attention_head_size] value_layer = self.transpose_for_scores(self.value(hidden_states)) # [batch_size, query_size, qformer_hidden_size] mixed_query_layer = self.query(hidden_states) # [batch_size, num_attention_heads, query_size, attention_head_size] query_layer = self.transpose_for_scores(mixed_query_layer) past_key_value = (key_layer, value_layer) # Take the dot product between "query" and "key" to get the raw attention scores. # key_layer.transpose(tmp_shape) [batch_size, num_attention_heads, attention_head_size, query_size] # query_layer: [batch_size, num_attention_heads, query_size, attention_head_size] trans_key_layer = self.transpose(key_layer, (0, 1, 3, 2)) attention_scores = self.batch_matmul(query_layer, trans_key_layer) if ( self.position_embedding_type == "relative_key" or self.position_embedding_type == "relative_key_query" ): seq_length = hidden_states.shape[1] position_ids_l = Tensor( [i for i in range(seq_length)], dtype=mstype.int32).view(-1, 1) position_ids_r = Tensor( [i for i in range(seq_length)], dtype=mstype.int32).view(1, -1) distance = position_ids_l - position_ids_r positional_embedding = self.distance_embedding( distance + self.max_position_embeddings - 1 ) if self.position_embedding_type == "relative_key": relative_position_scores = self.einsum( query_layer, positional_embedding) attention_scores = attention_scores + relative_position_scores elif self.position_embedding_type == "relative_key_query": relative_position_scores_query = self.einsum( query_layer, positional_embedding) relative_position_scores_key = self.einsum2( key_layer, positional_embedding) attention_scores = ( attention_scores + relative_position_scores_query + relative_position_scores_key ) attention_scores /= self.divider if attention_mask is not None: # Apply the attention mask is (precomputed for all layers in BertModel forward() function) attention_scores = attention_scores + attention_mask # Normalize the attention scores to probabilities. [batch_size, num_heads, query_size, query_size] attention_scores = self.cast(attention_scores, self.softmax_dtype) attention_probs = self.softmax(attention_scores) # This is actually dropping out entire tokens to attend to, which might # seem a bit unusual, but is taken from the original Transformer paper. attention_probs_dropped = self.dropout(attention_probs) # Mask heads if we want to if head_mask is not None: attention_probs_dropped = attention_probs_dropped * head_mask attention_probs_dropped = self.cast( attention_probs_dropped, self.compute_dtype) context_layer = self.batch_matmul(attention_probs_dropped, value_layer) # [batch_size, num_heads, query_size, attention_head_size] context_layer = self.transpose(context_layer, (0, 2, 1, 3)).copy() # [batch_size, query_size, all_head_size] new_context_layer_shape = context_layer.shape[:-2] + ( self.all_head_size,) context_layer = context_layer.view(*new_context_layer_shape) outputs = ( (context_layer, attention_probs) if output_attentions else (context_layer,) ) outputs = outputs + (past_key_value,) return outputs class BertSelfOutput(nn.Cell): """ BertSelfOutput """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dtype = config.dtype self.dense = Linear( in_channels=config.hidden_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps).shard(((dp, mp, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) self.cast = P.Cast() def construct(self, hidden_states, input_tensor): hidden_states = self.dense(hidden_states) hidden_states = self.cast(hidden_states, self.dtype) hidden_states = self.dropout(hidden_states) hidden_states = self.layernorm(hidden_states + input_tensor) return hidden_states class BertAttention(nn.Cell): """ BertAttention """ def __init__(self, config, is_cross_attention=False): super().__init__() self.self_att = BertSelfAttention(config, is_cross_attention) self.output = BertSelfOutput(config) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, ): """ hidden_states: [batch_size, query_size, qformer_hidden_size] attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, vit_hidden_size] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ # self_outputs.shape ([batch_size, query_size, qformer_hidden_size], # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) self_outputs = self.self_att( hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions, ) attention_output = self.output(self_outputs[0], hidden_states) # add attentions if we output them outputs = (attention_output,) + self_outputs[1:] return outputs class BertIntermediate(nn.Cell): """ BertIntermediate """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dense = Linear( in_channels=config.hidden_size, out_channels=config.intermediate_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) if isinstance(config.hidden_act, str): self.intermediate_act_fn = ACT2FN[config.hidden_act] if isinstance(self.intermediate_act_fn, nn.GELU): self.transform_act_fn = nn.GELU(approximate=False) else: self.intermediate_act_fn = config.hidden_act def construct(self, hidden_states): hidden_states = self.dense(hidden_states) hidden_states = self.intermediate_act_fn(hidden_states) return hidden_states class BertOutput(nn.Cell): """ BertOutput """ def __init__(self, config): super().__init__() if config.parallel_config: dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel else: dp = mp = 1 self.dense = Linear( in_channels=config.intermediate_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.dense.shard(strategy_matmul=((dp, mp), (1, mp))) self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) self.dropout = Dropout(1. - config.hidden_dropout_prob) def construct(self, hidden_states, input_tensor): hidden_states = self.dense(hidden_states) hidden_states = self.dropout(hidden_states) hidden_states = self.layernorm(hidden_states + input_tensor) return hidden_states class BertLayer(nn.Cell): """ BertLayer """ def __init__(self, config, layer_num): super().__init__() self.config = config self.chunk_size_feed_forward = config.chunk_size_feed_forward self.seq_len_dim = 1 self.attention = BertAttention(config) self.layer_num = layer_num if self.config.add_cross_attention and layer_num % self.config.cross_attention_freq == 0: self.crossattention = BertAttention(config, is_cross_attention=self.config.add_cross_attention) self.has_cross_attention = True else: self.has_cross_attention = False self.intermediate = BertIntermediate(config) self.output = BertOutput(config) self.intermediate_query = BertIntermediate(config) self.output_query = BertOutput(config) self.concat = P.Concat(axis=1) self.concat_seq = P.Concat(axis=self.seq_len_dim) def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_value=None, output_attentions=False, query_length=0, ): """ hidden_states: [batch_size, query_size, qformer_hidden_size] attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, vit_hidden_size ] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ # decoder uni-directional self-attention cached key/values tuple is at positions 1,2 self_attn_past_key_value = ( past_key_value[:2] if past_key_value is not None else None ) self_attention_outputs = self.attention( hidden_states, attention_mask, head_mask, None, None, self_attn_past_key_value, output_attentions, ) # [batch_size, query_size, qformer_hidden_size] attention_output = self_attention_outputs[0] # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) outputs = self_attention_outputs[1:-1] present_key_value = self_attention_outputs[-1] if query_length > 0: # [batch_size, query_size, qformer_hidden_size] query_attention_output = attention_output[:, :query_length, :] if self.has_cross_attention: assert encoder_hidden_states is not None, \ "encoder_hidden_states must be given for cross-attention layers" cross_attention_outputs = self.crossattention( query_attention_output, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, None, output_attentions, ) # [batch_size, query_size, qformer_hidden_size] query_attention_output = cross_attention_outputs[0] # add cross attentions if we output attention weights outputs = (outputs + cross_attention_outputs[1:-1]) # [batch_size, query_size, qformer_hidden_size] layer_output = self.apply_chunking_to_forward(self.feed_forward_chunk_query, query_attention_output) if attention_output.shape[1] > query_length: layer_output_text = self.apply_chunking_to_forward(self.feed_forward_chunk, attention_output[:, query_length:, :]) layer_output = self.concat([layer_output, layer_output_text]) else: layer_output = self.apply_chunking_to_forward( self.feed_forward_chunk, attention_output) outputs = (layer_output,) + outputs outputs = outputs + (present_key_value,) return outputs def feed_forward_chunk(self, attention_output): """ apply feed_forward with chunks """ intermediate_output = self.intermediate(attention_output) layer_output = self.output(intermediate_output, attention_output) return layer_output def feed_forward_chunk_query(self, attention_output): """ apply feed_forward with chunks (query) """ intermediate_output = self.intermediate_query(attention_output) layer_output = self.output_query(intermediate_output, attention_output) return layer_output def apply_chunking_to_forward(self, forward_fn, *input_tensors): """ apply chunking to forward computation """ assert input_tensors, f"{input_tensors} has to be a tuple/list of tensors" if self.chunk_size_feed_forward > 0: tensor_shape = input_tensors[0].shape[self.seq_len_dim] for input_tensor in input_tensors: if input_tensor.shape[self.seq_len_dim] != tensor_shape: raise ValueError( f"All input tenors have to be of the same shape: {tensor_shape}, " f"found shape {input_tensor.shape[self.seq_len_dim]}" ) if input_tensors[0].shape[self.seq_len_dim] % self.chunk_size_feed_forward != 0: raise ValueError( f"The dimension to be chunked {input_tensors[0].shape[self.seq_len_dim]} " f"has to be a multiple of the chunk size {self.chunk_size_feed_forward}" ) num_chunks = input_tensors[0].shape[self.seq_len_dim] // self.chunk_size_feed_forward # chunk input tensor into tuples input_tensors_chunks = tuple(input_tensor.chunk(num_chunks, dim=self.seq_len_dim) for input_tensor in input_tensors) # apply forward fn to every tuple output_chunks = tuple(forward_fn(*input_tensors_chunk) for input_tensors_chunk in zip(*input_tensors_chunks)) # concatenate output at same dimension return self.concat_seq(output_chunks) return forward_fn(*input_tensors) class BertEncoder(nn.Cell): """ BertEncoder """ def __init__(self, config): super(BertEncoder, self).__init__() self.config = config self.layer = nn.CellList( [BertLayer(config, i) for i in range(self.config.num_hidden_layers)] ) self.num_hidden_layers = [ i for i in range(self.config.num_hidden_layers)] def construct( self, hidden_states, attention_mask=None, head_mask=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=False, output_hidden_states=False, return_dict=True, query_length=0, ): """ attention_mask: [batch_size, 1, 1, query_size] encoder_hidden_states: [batch_size, vit_seq_length, encoder_hidden_width] encoder_attention_mask: [batch_size, 1, 1, vit_seq_length] """ all_hidden_states = () if output_hidden_states else None all_self_attentions = () if output_attentions else None all_cross_attentions = ( () if output_attentions and self.config.add_cross_attention else None ) next_decoder_cache = () if use_cache else None for i in self.num_hidden_layers: layer_module = self.layer[i] if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states,) layer_head_mask = head_mask[i] if head_mask is not None else None past_key_value = past_key_values[i] if past_key_values is not None else None # layer_outputs shape ([batch_size, query_size, qformer_hidden_size], # ([batch_size, num_head, query_size, head_size], [batch_size, num_head, query_size, head_size])) layer_outputs = layer_module( hidden_states, attention_mask, layer_head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions, query_length, ) hidden_states = layer_outputs[0] if use_cache: next_decoder_cache += (layer_outputs[-1],) if output_attentions: all_self_attentions = all_self_attentions + (layer_outputs[1],) all_cross_attentions = all_cross_attentions + \ (layer_outputs[2],) if output_hidden_states: all_hidden_states = all_hidden_states + (hidden_states,) value_list = [hidden_states, next_decoder_cache, all_hidden_states, all_self_attentions, all_cross_attentions] if not return_dict: return tuple(v for v in value_list if v is not None) return tuple(value_list) class BertPredictionHeadTransform(nn.Cell): """ BertPredictionHeadTransform """ def __init__(self, config): super(BertPredictionHeadTransform, self).__init__() self.dense = Linear( in_channels=config.hidden_size, out_channels=config.hidden_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype ) if isinstance(config.hidden_act, str): self.transform_act_fn = ACT2FN[config.hidden_act] if isinstance(self.transform_act_fn, nn.GELU): self.transform_act_fn = nn.GELU(approximate=False) else: self.transform_act_fn = config.hidden_act self.layernorm = LayerNorm( (config.hidden_size,), eps=config.layer_norm_eps) def construct(self, hidden_states): hidden_states = self.dense(hidden_states) hidden_states = self.transform_act_fn(hidden_states) hidden_states = self.layernorm(hidden_states) return hidden_states class BertLMPredictionHead(nn.Cell): """ BertLMPredictionHead """ def __init__(self, config): super(BertLMPredictionHead, self).__init__() self.transform = BertPredictionHeadTransform(config) # The output weights are the same as the input embeddings, but there is # an output-only bias for each token. self.decoder = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, compute_dtype=config.compute_dtype, param_init_type=config.dtype) self.bias = Parameter(initializer( Zero(), config.vocab_size, dtype=config.dtype)) # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings` self.decoder.bias = self.bias def construct(self, hidden_states): hidden_states = self.transform(hidden_states) hidden_states = self.decoder(hidden_states) return hidden_states class BertOnlyMLMHead(nn.Cell): """ BertOnlyMLMHead """ def __init__(self, config): super(BertOnlyMLMHead, self).__init__() self.predictions = BertLMPredictionHead(config) def construct(self, sequence_output): prediction_scores = self.predictions(sequence_output) return prediction_scores class BertPreTrainedModel(BaseModel, nn.Cell): """ An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. """ def __init__(self, config: BertConfig, **kwargs): super(BertPreTrainedModel, self).__init__(config) if not isinstance(config, BertConfig): raise ValueError( f"Parameter config in `{self.__class__.__name__}(config)` should be an instance of class " "`BaseConfig`. To create a model from a pretrained model use " f"`model = {self.__class__.__name__}.from_pretrained(PRETRAINED_MODEL_NAME)`" ) # Save config and origin of the pretrained weights if given in model self.config = config self.name_or_path = config.name_or_path self.warnings_issued = {} self.ones_like = P.OnesLike() self.zeros_like = P.ZerosLike() self.expand_dims = P.ExpandDims().shard(((1, 1, 1, 1, 1),)) def init_weights(self): """ init_weights recursively for BertPreTrainedModel. """ # Initialize weights, recursively. recursive_apply(self, self._init_weights) def _init_weights(self, module): """Initialize the weights""" if isinstance(module, nn.Embedding): # Slightly different from the TF version which uses truncated_normal for initialization normed = initializer(Normal(sigma=self.config.initializer_range, mean=0.0), shape=module.embedding_table.data.shape) module.embedding_table.set_data(normed) elif isinstance(module, Linear): normed = initializer(Normal(sigma=self.config.initializer_range, mean=0.0), shape=module.weight.data.shape, dtype=self.config.compute_dtype) module.weight.set_data(normed) if module.bias is not None: module.bias.set_data(self.zeros_like(module.bias.value())) elif isinstance(module, LayerNorm): module.gamma.set_data(self.ones_like(module.gamma.value())) module.beta.set_data(self.zeros_like(module.beta.value())) @classmethod def from_config(cls, config, **kwargs): """ All context managers that the model should be initialized under go here. """ model = cls(config, **kwargs) return model def get_input_embeddings(self) -> nn.Cell: """ get input embeddings. """ raise NotImplementedError def set_input_embeddings(self, value: nn.Cell): """ set input embeddings. """ raise NotImplementedError def get_output_embeddings(self) -> nn.Cell: """ get_output_embeddings """ raise NotImplementedError def set_output_embeddings(self, value): """ set_output_embeddings """ raise NotImplementedError def tie_weights(self): """ Tie the weights between the input embeddings and the output embeddings. """ if getattr(self.config, "tie_word_embeddings", True): output_embeddings = self.get_output_embeddings() if output_embeddings is not None: self.tie_or_clone_weights( output_embeddings, self.get_input_embeddings()) for module in self.cells(): if hasattr(module, "tie_weights"): module.tie_weights() def tie_or_clone_weights(self, output_embeddings: Linear, input_embeddings: nn.Embedding): """Tie or clone module weights depending of whether we are using TorchScript or not""" output_embeddings.weight.assign_value(input_embeddings.embedding_table.value()) if getattr(output_embeddings, "bias", None) is not None: paddings = [ [0] * 2 for _ in range(output_embeddings.bias.value().ndim)] paddings[-1][-1] = output_embeddings.weight.shape[0] - \ output_embeddings.bias.shape[0] pad_op = P.Pad(paddings=tuple(paddings)) output_embeddings.bias.assign_value( pad_op(output_embeddings.bias.value()) ) if hasattr(output_embeddings, "out_channels") and hasattr(input_embeddings, "vocab_size"): output_embeddings.out_channels = input_embeddings.vocab_size def resize_token_embeddings(self, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ Resizes input token embeddings matrix of the model if `new_num_tokens != config.vocab_size`. """ model_embeds = self.resize_model_embeds(new_num_tokens) if new_num_tokens is None or model_embeds is None: return model_embeds # Update base model and current model config self.config.vocab_size = new_num_tokens self.vocab_size = new_num_tokens # Tie weights again if needed self.tie_weights() return model_embeds def resize_model_embeds(self, new_num_tokens): """ resize input embeddings and output_embeddings. """ old_embeddings = self.get_input_embeddings() new_embeddings = self._get_resized_embeddings( old_embeddings, new_num_tokens) self.set_input_embeddings(new_embeddings) # if word embeddings are not tied, make sure that lm head is resized as well if self.get_output_embeddings() is not None and not self.config.tie_word_embeddings: old_lm_head = self.get_output_embeddings() new_lm_head = self._get_resized_lm_head( old_lm_head, new_num_tokens) self.set_output_embeddings(new_lm_head) return self.get_input_embeddings() def _get_resized_embeddings(self, old_embeddings: nn.Embedding, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ internal function, get new embeddings with inited weights. """ if new_num_tokens is None: return old_embeddings old_num_tokens, old_embedding_dim = old_embeddings.embedding_table.shape if old_num_tokens == new_num_tokens: return old_embeddings if not isinstance(old_embeddings, nn.Embedding): raise TypeError( f"Old embeddings are of type {type(old_embeddings)}, which is not an instance of {nn.Embedding}. You" " should either use a different resize function or make sure that `old_embeddings` are an instance of" f" {nn.Embedding}." ) # Build new embeddings new_embeddings = nn.Embedding(new_num_tokens, old_embedding_dim) new_embeddings.embedding_table.name = old_embeddings.embedding_table.name # initialize all new embeddings (in particular added tokens) self._init_weights(new_embeddings) # Copy token embeddings from the previous weights # numbers of tokens to copy remain_num = min(old_num_tokens, new_num_tokens) new_embeddings.embedding_table.data[:remain_num, :] = old_embeddings.embedding_table.data[:remain_num, :] return new_embeddings def _get_resized_lm_head(self, old_lm_head: Linear, new_num_tokens: Optional[int] = None, transposed: Optional[bool] = False) -> Linear: """ Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized """ if new_num_tokens is None: return old_lm_head old_num_tokens, old_lm_head_dim = ( old_lm_head.weight.shape if not transposed else old_lm_head.weight.T.shape ) if old_num_tokens == new_num_tokens: return old_lm_head if not isinstance(old_lm_head, Linear): raise TypeError( f"Old language model head is of type {type(old_lm_head)}, which is not an instance of {Linear}. You" " should either use a different resize function or make sure that `old_lm_head` are an instance of" f" {Linear}." ) # Build new lm head new_lm_head_shape = (old_lm_head_dim, new_num_tokens) if not transposed else ( new_num_tokens, old_lm_head_dim) has_new_lm_head_bias = old_lm_head.bias is not None new_lm_head = Linear(in_channels=new_lm_head_shape[0], out_channels=new_lm_head_shape[1], has_bias=has_new_lm_head_bias, compute_dtype=self.config.compute_dtype, param_init_type=self.config.dtype) # initialize new lm head (in particular added tokens) self._init_weights(new_lm_head) num_tokens_to_copy = min(old_num_tokens, new_num_tokens) # Copy old lm head weights to new lm head if not transposed: new_lm_head.weight.data[:num_tokens_to_copy, :] = old_lm_head.weight.data[:num_tokens_to_copy, :] else: new_lm_head.weight.data[:, :num_tokens_to_copy] = old_lm_head.weight.data[:, :num_tokens_to_copy] # Copy bias weights to new lm head if has_new_lm_head_bias: new_lm_head.bias.data[:num_tokens_to_copy] = old_lm_head.bias.data[:num_tokens_to_copy] return new_lm_head def invert_attention_mask(self, encoder_attention_mask): """ Invert an attention mask (e.g., switches 0. and 1.). """ encoder_extended_attention_mask = encoder_attention_mask for _ in range(4 - encoder_attention_mask.ndim): encoder_extended_attention_mask = self.expand_dims( encoder_extended_attention_mask, 1) encoder_extended_attention_mask = ( 1.0 - encoder_extended_attention_mask) return encoder_extended_attention_mask def get_head_mask(self, head_mask: Optional[Tensor], num_hidden_layers: int, is_attention_chunked: bool = False) -> Tensor: """ Prepare the head mask if needed. """ if head_mask is not None: head_mask = self._convert_head_mask_to_5d(head_mask) if is_attention_chunked is True: head_mask = self.expand_dims(head_mask, -1) else: head_mask = [None] * num_hidden_layers return head_mask def _convert_head_mask_to_5d(self, head_mask): """-> [num_hidden_layers x batch x num_heads x seq_length x seq_length]""" if head_mask.dim() == 1: head_mask = self.expand_dims(self.expand_dims(head_mask, 0), 0) head_mask = self.expand_dims(self.expand_dims(head_mask, -1), -1) head_mask = self.broadcast_to(head_mask) elif head_mask.dim() == 2: head_mask = self.expand_dims(self.expand_dims(head_mask, 1), -1) # We can specify head_mask for each layer head_mask = self.expand_dims(head_mask, -1) assert head_mask.dim( ) == 5, f"head_mask.dim != 5, instead {head_mask.dim()}" return head_mask class BertModel(BertPreTrainedModel): """ The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in `Attention is all you need `__ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and :obj:`add_cross_attention` set to :obj:`True`; an :obj:`encoder_hidden_states` is then expected as an input to the forward pass. """ def __init__(self, config): super(BertModel, self).__init__(config) self.config = config self.num_hidden_layers = self.config.num_hidden_layers self.output_attentions = self.config.output_attentions self.output_hidden_states = self.config.output_hidden_states self.use_return_dict = self.config.use_return_dict self.query_length = self.config.query_length self.embeddings = BertEmbeddings(config) self.encoder = BertEncoder(config) self.ones = P.Ones().shard(((self.config.parallel_config.data_parallel, 1),)) self.zeros = P.Zeros() self.broadcast_to = P.BroadcastTo( (self.num_hidden_layers, -1, -1, -1, -1)) self.concat_one = P.Concat(axis=1) self.concat_minus_one = P.Concat(axis=-1) self.init_weights() def get_input_embeddings(self) -> nn.Cell: return self.embeddings.word_embeddings def set_input_embeddings(self, value): if not isinstance(value, nn.Embedding): raise ValueError( "expect new input_embeddings to be of type: ", nn.Embedding) self.embeddings.word_embeddings = value def get_output_embeddings(self) -> nn.Cell: return None # no output embeddings def set_output_embeddings(self, value): return # no output embeddings def tie_weights(self): return # no output embeddings for tie def get_extended_attention_mask( self, attention_mask, input_shape, is_decoder, has_query=False, ): """ Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (:obj:`mindspore.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (:obj:`Tuple[int]`): The shape of the input to the model. is_decoder (:obj:`Bool`): has_query (:obj:`Bool`): Returns: :obj:`mindspore.Tensor` The extended attention mask, with the same dtype as :obj:`attention_mask.dtype`. """ # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length] # ourselves in which case we just need to make it broadcastable to all heads. if attention_mask.ndim == 3: extended_attention_mask = attention_mask[:, None, :, :] elif attention_mask.ndim == 2: # Provided a padding mask of dimensions [batch_size, seq_length] # - if the model is a decoder, apply a causal mask in addition to the padding mask # - if the model is an encoder, make the mask broadcastable to # [batch_size, num_heads, seq_length, seq_length] if is_decoder: batch_size, seq_length = input_shape seq_ids = Tensor( [i for i in range(seq_length)], dtype=mstype.int32) causal_mask = ( np.tile(seq_ids[None, None, :], (batch_size, seq_length, 1)) <= seq_ids[None, :, None] ) # add a prefix ones mask to the causal mask # causal and attention masks must have same type with pytorch version < 1.3 causal_mask = causal_mask.astype(attention_mask.dtype) if causal_mask.shape[1] < attention_mask.shape[1]: prefix_seq_len = attention_mask.shape[1] - \ causal_mask.shape[1] if has_query: # UniLM style attention mask causal_mask = self.concat_one( [self.zeros((batch_size, prefix_seq_len, seq_length), causal_mask.dtype), causal_mask]) causal_mask = self.concat_minus_one( [self.ones((batch_size, causal_mask.shape[1], prefix_seq_len), causal_mask.dtype), causal_mask]) extended_attention_mask = ( causal_mask[:, None, :, :] * attention_mask[:, None, None, :]) else: extended_attention_mask = attention_mask[:, None, None, :] else: raise ValueError( "Wrong shape for input_ids (shape {}) or attention_mask (shape {})".format( input_shape, attention_mask.shape ) ) # Since attention_mask is 1.0 for positions we want to attend and 0.0 for # masked positions, this operation will create a tensor which is 0.0 for # positions we want to attend and -10000.0 for masked positions. # Since we are adding it to the raw scores before the softmax, this is # effectively the same as removing these entirely. extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0 return extended_attention_mask def construct( self, input_ids=None, attention_mask=None, position_ids=None, head_mask=None, query_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, past_key_values=None, use_cache=None, output_attentions=None, output_hidden_states=None, return_dict=None, is_decoder=False, ): r""" encoder_hidden_states (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`, `optional`): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (:obj:`torch.FloatTensor` of shape :obj:`(batch_size, sequence_length)`, `optional`): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (:obj:`tuple(tuple(torch.FloatTensor))` of length :obj:`config.n_layers` with each tuple having 4 tensors of shape :obj:`(batch_size, num_heads, sequence_length - 1, embed_size_per_head)`): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape :obj:`(batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape :obj:`(batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). """ output_attentions = ( output_attentions if output_attentions is not None else self.output_attentions ) output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.output_hidden_states ) return_dict = ( return_dict if return_dict is not None else self.use_return_dict ) if input_ids is None: assert query_embeds is not None, "You have to specify query_embeds when input_ids is None" # past_key_values_length past_key_values_length = ( past_key_values[0][0].shape[2] - self.query_length if past_key_values is not None else 0 ) query_length = query_embeds.shape[1] if query_embeds is not None else 0 # [bz, query_size/seq_length, qformer_hidden_size] dp embedding_output = self.embeddings( input_ids=input_ids, position_ids=position_ids, query_embeds=query_embeds, past_key_values_length=past_key_values_length, ) # [bz, query_size/seq_length] dp input_shape = embedding_output.shape[:-1] batch_size, seq_length = input_shape if attention_mask is None: attention_mask = self.ones( (batch_size, seq_length + past_key_values_length), mstype.float32 ) # [bz, seq_length] # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length] # ourselves in which case we just need to make it broadcastable to all heads. if is_decoder: extended_attention_mask = self.get_extended_attention_mask( attention_mask, input_ids.shape, is_decoder, has_query=(query_embeds is not None), ) else: extended_attention_mask = self.get_extended_attention_mask( attention_mask, input_shape, is_decoder, False ) # If a 2D or 3D attention mask is provided for the cross-attention # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length] if encoder_hidden_states is not None: if isinstance(encoder_hidden_states, list): encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states[0].shape else: encoder_batch_size, encoder_sequence_length, _ = encoder_hidden_states.shape encoder_hidden_shape = ( encoder_batch_size, encoder_sequence_length) if isinstance(encoder_attention_mask, list): encoder_extended_attention_mask = [ self.invert_attention_mask(mask) for mask in encoder_attention_mask] elif encoder_attention_mask is None: encoder_attention_mask = self.ones(encoder_hidden_shape) encoder_extended_attention_mask = self.invert_attention_mask( encoder_attention_mask) else: encoder_extended_attention_mask = self.invert_attention_mask( encoder_attention_mask) else: encoder_extended_attention_mask = None # Prepare head mask if needed # 1.0 in head_mask indicate we keep the head # attention_probs has shape bsz x n_heads x N x N # input head_mask has shape [num_heads] or [num_hidden_layers x num_heads] # and head_mask is converted to shape [num_hidden_layers x batch x num_heads x seq_length x seq_length] head_mask = self.get_head_mask(head_mask, self.num_hidden_layers) encoder_outputs = self.encoder( embedding_output, attention_mask=extended_attention_mask, head_mask=head_mask, encoder_hidden_states=encoder_hidden_states, encoder_attention_mask=encoder_extended_attention_mask, past_key_values=past_key_values, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, query_length=query_length, ) sequence_output = encoder_outputs[0] if not return_dict: return (sequence_output,) + encoder_outputs[1:] return ( sequence_output, encoder_outputs[1], encoder_outputs[2], encoder_outputs[3], encoder_outputs[4], ) class BertLMHeadModel(BertPreTrainedModel): """ BertLMHeadModel, the main model for Qformer Args: config (QFormerConfig): config for qformer, see qformer_config.py. Raises: ValueError: config type Error. Returns: a BertLMHeadModel instance. """ def __init__(self, config: QFormerConfig): super(BertLMHeadModel, self).__init__(config) if not isinstance(config, QFormerConfig): raise ValueError( f"Parameter config in `{self.__class__.__name__}(config)` " "should be an instance of class `QFormerConfig`. " "To create a model from a pretrained model use " f"`model = {self.__class__.__name__}.from_pretrained(PRETRAINED_MODEL_NAME)`" ) self.config = config self.bert = BertModel(config) self.cls = BertOnlyMLMHead(config) self.init_weights() if self.config.checkpoint_name_or_path: self.load_checkpoint(config) # for lm_loss reduction - GRAPH_MODE self.reduction = config.loss_reduction self.loss = CrossEntropyLoss( reduction=self.reduction, label_smoothing=0.1) self.vocab_size = self.config.vocab_size def convert_bert_model_params(self, bert_model_params: OrderedDict): """ convert params from BertModel in MindFormers, some param names are altered. """ dict_mapping = {'layer.': 'blocks.', 'encoder.': 'bert_encoder.encoder.', 'self_att.query.': 'dense1.', 'self_att.key.': 'dense2.', 'self_att.value.': 'dense3.', 'attention.output.dense.': 'attention.projection.', 'attention.output.layernorm.gamma': 'layernorm2.gamma', 'attention.output.layernorm.beta': 'layernorm2.beta', 'intermediate.dense.weight': 'output.mapping.weight', 'intermediate.dense.bias': 'output.mapping.bias', 'output.dense.weight': 'output.projection.weight', 'output.dense.bias': 'output.projection.bias', 'output.layernorm.gamma': 'layernorm1.gamma', 'output.layernorm.beta': 'layernorm1.beta', 'embeddings.position_embeddings.embedding_table': \ 'embedding_postprocessor.full_position_embedding.embedding_table', 'embeddings.layernorm.gamma': 'embedding_postprocessor.layernorm.gamma', 'embeddings.layernorm.beta': 'embedding_postprocessor.layernorm.beta', 'embeddings.word_embeddings.embedding_table': 'word_embedding.embedding_table', 'cls.predictions.transform.dense': 'bert.mlmloss.dense', 'cls.predictions.transform.layernorm': 'bert.mlmloss.layernorm', 'cls.predictions.decoder': 'bert.mlmloss.vocab_dense'} param_dict = self.parameters_dict() for name, data in param_dict.items(): new_name = name for replace_from, replace_to in dict_mapping.items(): new_name = new_name.replace(replace_from, replace_to) if new_name not in bert_model_params.keys(): logger.warning("%s not loaded.", name) continue new_data = bert_model_params[new_name] if name.endswith("intermediate.dense.weight") or name.endswith("output.dense.weight"): new_data = new_data.T data.assign_value(new_data) def load_checkpoint(self, config: QFormerConfig): """ load checkpoint for BertLMHeadModel. (we can use the param for BertModel on obs, but we need to alter the names of some param) Args: config (ModelConfig): QFormerConfig instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path # direct loading, no param conversion if not config.convert_param_from_bert: super(BertLMHeadModel, self).load_checkpoint(config) # convert params and load from BertModel else: # the relevant file will be downloaded from the Obs platform. if not os.path.exists(checkpoint_name_or_path): support_list = MindFormerBook.get_model_support_list()['bert'] if checkpoint_name_or_path not in support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {support_list}.") checkpoint_name = checkpoint_name_or_path default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[0]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][0] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) self.default_checkpoint_download_path = ckpt_file logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) else: ckpt_file = checkpoint_name_or_path param = load_checkpoint(ckpt_file) self.convert_bert_model_params(param) logger.info("weights in %s are loaded", ckpt_file) def get_input_embeddings(self) -> nn.Cell: return self.bert.get_input_embeddings() def set_input_embeddings(self, value): return self.bert.set_input_embeddings(value) def get_output_embeddings(self) -> nn.Cell: return self.cls.predictions.decoder def set_output_embeddings(self, value): self.cls.predictions.decoder = value def resize_token_embeddings(self, new_num_tokens: Optional[int] = None) -> nn.Embedding: """ resize token embeddings, inherit from super class. """ old_num_tokens = self.get_input_embeddings().embedding_table.shape[0] logger.info("resize_token_embeddings from %d to %d.", old_num_tokens, new_num_tokens) super(BertLMHeadModel, self).resize_token_embeddings(new_num_tokens) def tie_weights(self): """ tie encoder and decoder weights, inherit from super class. """ logger.info("weights tied.") super(BertLMHeadModel, self).tie_weights() def construct( self, input_ids=None, attention_mask=None, position_ids=None, head_mask=None, query_embeds=None, encoder_hidden_states=None, encoder_attention_mask=None, labels=None, past_key_values=None, use_cache=True, output_attentions=None, output_hidden_states=None, return_dict=None, return_logits=False, is_decoder=True ): r""" encoder_hidden_states `Tensor` of shape : (batch_size, sequence_length, hidden_size)` Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask `Tensor` of shape : (batch_size, sequence_length)`, `optional`) Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in ``[0, 1]``: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels `Tensor(mstype.int32)` of shape : (batch_size, sequence_length)`, `optional`) Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in ``[-100, 0, ..., config.vocab_size]`` (see ``input_ids`` docstring) Tokens with indices set to ``-100`` are ignored (masked), the loss is only computed for the tokens with labels n ``[0, ..., config.vocab_size]``, past_key_values (:obj:`tuple(tuple(Tensor(mstype.float)))` of length: `config.n_layers` with each tuple having 4 tensors of shape (batch_size, num_heads, sequence_length - 1, embed_size_per_head)), Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If :obj:`past_key_values` are used, the user can optionally input only the last :obj:`decoder_input_ids` (those that don't have their past key value states given to this model) of shape (batch_size, 1)` instead of all :obj:`decoder_input_ids` of shape (batch_size, sequence_length)`. use_cache (:obj:`bool`, `optional`): If set to :obj:`True`, :obj:`past_key_values` key value states are returned and can be used to speed up decoding (see :obj:`past_key_values`). Returns: Example:: >>> from mindformers import BertTokenizer, Blip2Config, BertLMHeadModel >>> tokenizer = BertTokenizer.from_pretrained('bert-base-cased') >>> config = Blip2Config.from_pretrained("blip2_stage1_vit_g") >>> model = BertLMHeadModel(config=config) >>> inputs = tokenizer("Hello, my dog is cute") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits """ return_dict = ( return_dict if return_dict is not None else self.config.use_return_dict ) if labels is not None: use_cache = False if past_key_values is not None: query_embeds = None outputs = self.bert( input_ids, attention_mask=attention_mask, position_ids=position_ids, head_mask=head_mask, query_embeds=query_embeds, encoder_hidden_states=encoder_hidden_states, encoder_attention_mask=encoder_attention_mask, past_key_values=past_key_values, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, is_decoder=is_decoder, ) sequence_output = outputs[0] if query_embeds is not None: sequence_output = outputs[0][:, query_embeds.shape[1]:, :] prediction_scores = self.cls(sequence_output) if return_logits: return prediction_scores[:, :-1, :].copy() lm_loss = None if labels is not None: # we are doing next-token prediction; shift prediction scores and input ids by one shifted_prediction_scores = prediction_scores[:, :-1, :].copy() labels = labels[:, 1:].copy() lm_loss = self.loss( shifted_prediction_scores.view(-1, self.vocab_size), labels.view(-1), ) if self.reduction == "none": lm_loss = lm_loss.view(prediction_scores.size(0), -1).sum(1) if not return_dict: output = (prediction_scores,) + outputs[1:] return ((lm_loss,) + output) if lm_loss is not None else output return ( lm_loss, prediction_scores, outputs[1], outputs[2], outputs[3], outputs[4], ) pengcheng.mind-mindformer/mindformers/models/blip2/qformer_config.py000066400000000000000000000121631476650141400263070ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Qformer Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.bert import BertConfig __all__ = ['QFormerConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class QFormerConfig(BertConfig): """ Qformer config class which defines the model size """ def __init__(self, model_type: str = "Qformer", num_hidden_layers: int = 12, num_attention_heads: int = 12, query_length: int = 32, vocab_size: int = 30523, hidden_size: int = 768, encoder_width: int = 1408, head_embed_dim: int = 256, bos_token_id: int = 30522, sep_token_id: int = 102, pad_token_id: int = 0, max_position_embeddings: int = 512, layer_norm_eps: float = 1.e-12, hidden_dropout_prob: float = 0.1, attention_probs_dropout_prob: float = 0.1, chunk_size_feed_forward: int = 0, cross_attention_freq: int = 2, intermediate_size: int = 3072, initializer_range: float = 0.02, hidden_act: str = "gelu", dtype: str = "float32", layernorm_dtype: str = "float32", softmax_dtype: str = "float32", compute_dtype: str = "float16", add_cross_attention: bool = True, use_relative_positions: bool = False, tie_word_embeddings: bool = True, output_attentions: bool = False, output_hidden_states: bool = False, use_return_dict: bool = False, convert_param_from_bert: bool = False, parallel_config: str = "default", moe_config: str = "default", **kwargs): super(QFormerConfig, self).__init__(**kwargs) self.model_type = model_type self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.query_length = query_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.encoder_width = encoder_width self.head_embed_dim = head_embed_dim self.bos_token_id = bos_token_id self.sep_token_id = sep_token_id self.pad_token_id = pad_token_id self.max_position_embeddings = max_position_embeddings self.layer_norm_eps = layer_norm_eps self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.chunk_size_feed_forward = chunk_size_feed_forward self.cross_attention_freq = cross_attention_freq self.intermediate_size = intermediate_size self.initializer_range = initializer_range self.hidden_act = hidden_act self.dtype = mstype.float32 if dtype == "float32" else mstype.float16 self.layernorm_dtype = mstype.float32 if layernorm_dtype == "float32" else mstype.float16 self.softmax_dtype = mstype.float32 if softmax_dtype == "float32" else mstype.float16 self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.add_cross_attention = add_cross_attention self.use_relative_positions = use_relative_positions self.tie_word_embeddings = tie_word_embeddings self.output_attentions = output_attentions self.output_hidden_states = output_hidden_states self.use_return_dict = use_return_dict self.convert_param_from_bert = convert_param_from_bert self.parallel_config = default_transformer_config if parallel_config == "default" \ else parallel_config self.moe_config = default_moe_config if moe_config == "default" else moe_config self.chunk_size_feed_forward = kwargs.pop("chunk_size_feed_forward", 0) self.position_embedding_type = "absolute" if not use_relative_positions else "relative" # Whether input and output word embeddings should be tied. self.tie_word_embeddings = kwargs.pop( "tie_word_embeddings", True ) self.loss_reduction = kwargs.pop("loss_reduction", "mean") pengcheng.mind-mindformer/mindformers/models/bloom/000077500000000000000000000000001476650141400230325ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/bloom/__init__.py000066400000000000000000000020461476650141400251450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bloom""" from .bloom_config import BloomConfig from .bloom import BloomModel, BloomLMHeadModel from .bloom_tokenizer import BloomTokenizer from .bloom_processor import BloomProcessor from .bloom_reward import * __all__ = ['BloomConfig', 'BloomModel', 'BloomLMHeadModel', 'BloomTokenizer', 'BloomProcessor'] __all__.extend(bloom_reward.__all__) pengcheng.mind-mindformer/mindformers/models/bloom/bloom.py000066400000000000000000000377301476650141400245260ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom model""" import copy import os import numpy as np import mindspore.common.dtype as mstype from mindspore import nn from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindformers.modules.transformer import VocabEmbedding from mindformers.modules.layers import LayerNorm, AlibiTensor from mindformers.core.loss import CrossEntropyLoss from mindformers.models.base_model import BaseModel from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from .layers import BloomBlocks, CausalMask from .bloom_config import BloomConfig from ..utils import convert_mstype, cell_reuse def jit_inference_with_condition(): """allow jit inference""" def decorator(func): if os.getenv("JIT_INFERENCE", "NOT_FOUND") == "NOT_FOUND": return func from mindspore import jit, JitConfig dec = jit(jit_config=JitConfig(jit_level="O2")) return dec(func) return decorator class BloomEmbeddingLayer(nn.Cell): """The Embedding Layer of Bloom network.""" def __init__(self, config=None): super(BloomEmbeddingLayer, self).__init__(auto_prefix=False) parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer("normal", [vocab_size, config.hidden_size], dtype=config.embedding_init_type), parallel_config=config.parallel_config.embedding_dp_mp_config) self.norm = LayerNorm((config.hidden_size,)).shard(((1, 1, 1), (1,), (1,))) def construct(self, input_ids): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) embedding = self.norm(word_embedding) embedding = embedding.astype(mstype.float16) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers, use_select_recompute=False): """ Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. use_select_recompute(bool) - Indicates whether to use the select recompute mode. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not use_select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not use_select_recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class BloomModel(nn.Cell): """ The backbone of Bloom network Args: config(BloomConfig): The config of network Inputs: input_ids(Tensor): The tokenized inputs with datatype int32 input_mask(Tensor): The mask indicating whether each position is a valid input Returns: output_state(Tensor): The output logit of backbone embedding_table(Tensor): The embedding table for the vocabulary """ def __init__(self, config): super(BloomModel, self).__init__() self.embedding = BloomEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.make_causal_attention = CausalMask(seq_length=config.seq_length, parallel_config=config.parallel_config.dp_mp_config) self.build_alibi_tensor = AlibiTensor(seq_length=config.seq_length, num_heads=config.num_heads, parallel_config=config.parallel_config) self.blocks = BloomBlocks(hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_layers=config.num_layers, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, lambda_func=set_parallel_configure_for_layer, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, use_past=config.use_past, use_seq_parallel=config.use_seq_parallel, use_select_recompute=config.use_select_recompute, parallel_config=config.parallel_config).blocks self.num_layers = config.num_layers self.ln_f = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.ln_f.set_comm_fusion(2) else: self.ln_f.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.ln_f.shard(((config.parallel_config.data_parallel, 1, 1),)) self.ln_f.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.use_past = config.use_past self.dtype = convert_mstype(config.param_init_type) self.mul_init_reset = P.Mul().shard( ((config.parallel_config.data_parallel, config.parallel_config.model_parallel, 1, 1), (1,))) def construct(self, input_ids, input_mask, init_reset=True, batch_valid_length=None): """Bloom model""" input_embedding, embedding_table = self.embedding(input_ids) hidden_states = input_embedding hidden_states_shape = hidden_states.shape hidden_states = hidden_states.reshape((-1, hidden_states_shape[-1])) causal_mask = self.make_causal_attention(input_mask) alibi_tensor = self.build_alibi_tensor(input_mask, hidden_states.dtype) if self.use_past: init_reset = self.mul_init_reset(self.blocks[0].key_past, init_reset.astype(self.dtype)) for i in range(self.num_layers): hidden_states, _ = self.blocks[i](hidden_states, alibi_tensor, causal_mask, init_reset, batch_valid_length) hidden_states = hidden_states.reshape(hidden_states_shape) output_state = self.ln_f(hidden_states) return output_state, embedding_table class BloomHead(nn.Cell): """Head for Bloom to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type="float16", parallel_config=None): super(BloomHead, self).__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of BloomHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of BloomHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = convert_mstype(compute_type) def construct(self, state, embedding_table): ori_dtype = state.dtype state = state.reshape((-1, self.hidden_size)) logits = self.matmul(state.astype(self.dtype), embedding_table.astype(self.dtype)) return logits.astype(ori_dtype) @MindFormerRegister.register(MindFormerModuleType.MODELS) class BloomLMHeadModel(BaseModel): """ Provide bloom training loss or logits through network. Args: config (BloomConfig): The config of BloomModel. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['bloom'] @cell_reuse() def __init__(self, config=None): config = config if config is not None else BloomConfig() super(BloomLMHeadModel, self).__init__(config, auto_prefix=False) self.use_past = self.config.use_past self.is_sample_acceleration = self.config.is_sample_acceleration if self.use_past: self.input_mask_all_ones = Tensor( np.ones((self.config.batch_size, self.config.seq_length), np.float32), mstype.float32) if self.is_sample_acceleration: self.p_all_ones = Tensor(np.ones((self.config.batch_size, 1), np.float32), mstype.float32) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.gt = P.Greater().shard(((parallel_config.data_parallel, 1), ())) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.abs = P.Abs().shard(((parallel_config.data_parallel, 1),)) self.transformer = BloomModel(self.config) self.head = BloomHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.transformer.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of Bloom Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of Bloom Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 @jit_inference_with_condition() def construct(self, input_ids, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, labels=None, init_reset=True, batch_valid_length=None): """ construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_position(Tensor): current position, used by model.predict. Default None. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. labels(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ batch_size, seq_length = input_ids.shape if self.training: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) else: tokens = input_ids input_mask = self.not_equal(tokens, self.eos_token_id).astype(mstype.float32) \ if not self.use_past else self.input_mask_all_ones loss_mask = self.mul(input_mask, self.gt(tokens, 0).astype(mstype.float32)) tokens = self.abs(tokens) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.transformer(tokens, input_mask, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: if self.is_sample_acceleration: return self.get_top_token_id(logits, current_index=input_position) return logits, tokens, input_mask labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = labels.reshape((-1,)) loss_mask = loss_mask.reshape((-1,)) loss = self.loss(logits, labels, loss_mask) return loss def get_top_token_id(self, logits, current_index=None): """get_top_token_id""" logits = logits.reshape(-1, logits.shape[-1]) if self.use_past and not self.is_first_iteration: logits = logits elif current_index is not None: index = current_index.view(-1,) logits = P.Gather()(logits, index, 0) probabilities = P.Softmax(-1)(logits) top_token_id = P.Argmax(-1)(probabilities) top_token_id = top_token_id.view(-1, 1) return self.p_all_ones, top_token_id pengcheng.mind-mindformer/mindformers/models/bloom/bloom_config.py000066400000000000000000000111321476650141400260370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom Config API""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.utils import convert_mstype from mindformers.models.base_config import BaseConfig from mindformers.mindformer_book import MindFormerBook __all__ = ['BloomConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class BloomConfig(BaseConfig): """ Bloom config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['bloom'] def __init__(self, embedding_dropout_prob: float = 0.0, batch_size: int = None, seq_length: int = 1024, vocab_size: int = 250880, hidden_size: int = 64, num_layers: int = 2, num_heads: int = 8, expand_ratio: int = 4, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, unk_token_id: int = 0, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 3, param_init_type: str = "float32", embedding_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", hidden_act: str = 'gelu', parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = "", moe_config: MoEConfig = default_moe_config, use_past: bool = False, use_seq_parallel: bool = False, use_select_recompute: bool = False, repetition_penalty: int = 1, max_decode_length: int = 1024, top_k: int = 5, top_p: int = 1, do_sample: bool = True, is_sample_acceleration: bool = False, **kwargs): super().__init__(**kwargs) self.embedding_dropout_prob = embedding_dropout_prob self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.expand_ratio = expand_ratio self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.param_init_type = convert_mstype(param_init_type) self.embedding_init_type = convert_mstype(embedding_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.moe_config = moe_config self.use_past = use_past self.unk_token_id = unk_token_id self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.hidden_act = hidden_act self.use_seq_parallel = use_seq_parallel self.use_select_recompute = use_select_recompute self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.is_sample_acceleration = is_sample_acceleration if self.batch_size is None: self.use_past = False # currently require batch_size = 1 self.is_sample_acceleration = False # currently require batch_size = 1 pengcheng.mind-mindformer/mindformers/models/bloom/bloom_processor.py000066400000000000000000000044741476650141400266240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ BloomProcessor """ from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['BloomProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class BloomProcessor(BaseProcessor): """ Bloom processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['bloom'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(BloomProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-mindformer/mindformers/models/bloom/bloom_reward.py000066400000000000000000000107631476650141400260670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """bloom reward model""" import copy import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.layers import Linear from mindformers.core.loss import CompareLoss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.tools.logger import logger from .bloom_config import BloomConfig from .bloom import BloomModel __all__ = ['BloomRewardModel', 'VHead'] class VHead(nn.Cell): r"""Head for Bloom to get the logits of each token in the vocab.""" def __init__(self, config=None): super().__init__() dp = config.parallel_config.data_parallel mp = 1 self.vhead = Linear(in_channels=config.hidden_size, out_channels=1, has_bias=False).to_float(mstype.float16) self.vhead.shard(strategy_matmul=((dp, mp), (mp, 1))) self.vhead.pipeline_stage = config.parallel_config.pipeline_stage - 1 def construct(self, output_states): """ construct function for vhead """ return self.vhead(output_states) @MindFormerRegister.register(MindFormerModuleType.MODELS) class BloomRewardModel(BaseModel): r""" Provide bloom reward model training loss or logits through network. Args: config (BloomConfig): The config of BloomModel. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config=None): config = config if config is not None else BloomConfig() super(BloomRewardModel, self).__init__(config, auto_prefix=False) mp = config.parallel_config.model_parallel self.eos_token_id = self.config.eos_token_id self.seq_length = config.seq_length self.eos_token = self.config.eos_token parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.transformer = BloomModel(self.config) self.vhead = VHead(self.config) if parallel_config.pipeline_stage > 1: self.vhead.pipeline_stage = parallel_config.pipeline_stage - 1 self.transformer.embedding.word_embedding.embedding_table.add_pipeline_stage(self.vhead.pipeline_stage) vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of Bloom Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of Bloom Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CompareLoss(config=loss_parallel_config) self.load_checkpoint(config) def construct(self, input_ids, position_id=None, attention_mask=None, loss_mask=None, end_ind=None): """ construct function for reward model """ _ = position_id if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id).astype(mstype.float32) output_states, _ = self.transformer(input_ids, attention_mask) # [bs, seq, hidden_size] logits = self.vhead(output_states) # [bs, seq, 1] logits = logits.squeeze(-1) logits = F.reshape(logits, (-1, self.seq_length)) loss, chosen_end_scores, reject_end_scores = self.loss(logits, loss_mask, end_ind) if self.training: return loss return loss, chosen_end_scores, reject_end_scores pengcheng.mind-mindformer/mindformers/models/bloom/bloom_tokenizer.py000066400000000000000000000247401476650141400266150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Bloom Tokenzier """ import json import os from functools import lru_cache from typing import List, Optional import regex as re from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.logger import logger from mindformers.mindformer_book import MindFormerBook __all__ = ['BloomTokenizer'] VOCAB_FILES_NAMES = {'vocab_file': 'tokenizer.json'} @lru_cache() def bytes_to_unicode(): """ bytes to unicode """ bs = list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1)) cs = bs[:] n = 0 for b in range(2 ** 8): if b not in bs: bs.append(b) cs.append(2 ** 8 + n) n += 1 cs = [chr(i) for i in cs] return dict(zip(bs, cs)) def get_pairs(word): """ Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). """ pairs = set() prev_char = word[0] for char in word[1:]: pairs.add((prev_char, char)) prev_char = char return pairs @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class BloomTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Args: vocab_file(str): The vocabulary file path. unk_token(str): The token that represents the unknown. Default "<|unk|>". bos_token(str): The token that represents the begin-of-sentence. Default "<|s|>"". eos_token(str): The token that represents the end-of-sentence. Default "<|/s|>". pad_token(str): The token that represents the pad. Default "<|pad|>". add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import BloomTokenizer >>> tokenizer = BloomTokenizer.from_pretrained("bloom_560m") >>> res = tokenizer("Hello world", add_special_tokens=False) >>> print(res) {'input_ids': [59414, 8876], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] model_input_names = ["input_ids", "token_type_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['bloom'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", add_prefix_space=False, add_bos_token=False, add_eos_token=False, **kwargs): super(BloomTokenizer, self).__init__( unk_token=unk_token, bos_token=bos_token, eos_token=eos_token, pad_token=pad_token, **kwargs ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token with open(vocab_file, 'r', encoding="utf-8") as vocab_handle: self.encoder = json.load(vocab_handle)["model"]["vocab"] self.decoder = {v: k for k, v in self.encoder.items()} with open(vocab_file, 'r', encoding="utf-8") as vocab_handle: bpe_merges = json.load(vocab_handle)["model"]["merges"] self.the_unk_token = unk_token bpe_merges = [tuple(merge.split()) for merge in bpe_merges] self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") self.add_prefix_space = add_prefix_space self.cache = {} self._unk_token_id = 0 self._bos_token_id = 1 self._eos_token_id = 2 self._pad_token_id = 3 def bpe(self, token): """ bpe encode """ if token in self.cache: return self.cache[token] word = tuple(token) pairs = get_pairs(token) if not pairs: return token while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf"))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) except ValueError: new_word.extend(word[i:]) break else: new_word.extend(word[i:j]) i = j if word[i] == first and i + 1 < len(word) and word[i + 1] == second: new_word.append(first + second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break else: pairs = get_pairs(word) word = " ".join(word) self.cache[token] = word return word def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Tokenize a string using bpe encode. """ text = self.prepare_for_tokenization(text, is_pretokenized=False) bpe_tokens = [] for token in re.findall(self.pat, text): token = "".join( self.byte_encoder[b] for b in token.encode("utf-8") ) bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" ")) return bpe_tokens def _convert_token_to_id(self, token): """ the index of the tokens in the vocabulary. """ return self.encoder.get(token, self.encoder.get(self.the_unk_token)) def _convert_id_to_token(self, index): """ return the origin bpe tokens according to ids """ return self.decoder.get(index) def _convert_tokens_to_string(self, tokens): """ return a string according to the list of tokens""" text = "".join(tokens) text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors='ignore') return text def convert_tokens_to_string(self, tokens): """Convert the tokens to the string""" return self._convert_tokens_to_string(tokens) def prepare_for_tokenization(self, text, **kwargs): """ whether to add a whitespace in the front of text """ add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space) is_pretokenized = kwargs.pop("is_pretokenized", False) if is_pretokenized or add_prefix_space: text = " " + text return text def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(output_file_path, "w", encoding="utf-8") as f: f.write(json.dumps(self.encoder)) return output_file_path @property def vocab_size(self): """Get the vocab size of the """ return len(self.decoder) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): """Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): return dict(self.encoder, **self.added_tokens_encoder) pengcheng.mind-mindformer/mindformers/models/bloom/convert_weight.py000066400000000000000000000145771476650141400264510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from huggingface""" import os import re import argparse import torch import mindspore from mindspore import Tensor, Parameter def layer_name_mapping(key): """Convert huggingface PP weights mapping in MindSpore. return: split, new_name """ prefix = '' if 'transformer' in key: prefix = 'transformer.' key = key.replace('transformer.', '') # Handle first and last layers layer_rename_map = { "word_embeddings.weight": "embedding.word_embedding.embedding_table", "word_embeddings_layernorm.weight": "embedding.norm.gamma", "word_embeddings_layernorm.bias": "embedding.norm.beta", "ln_f.weight": "ln_f.gamma", "ln_f.bias": "ln_f.beta", "input_layernorm.weight": "layernorm1.gamma", "input_layernorm.bias": "layernorm1.beta", "self_attention.query_key_value.weight": "attention.dense{}.weight", "self_attention.query_key_value.bias": "attention.dense{}.bias", "self_attention.dense.weight": "attention.projection.weight", "self_attention.dense.bias": "attention.projection.bias", "post_attention_layernorm.weight": "layernorm2.gamma", "post_attention_layernorm.bias": "layernorm2.beta", "mlp.dense_h_to_4h.weight": "output.mapping.weight", "mlp.dense_h_to_4h.bias": "output.mapping.bias", "mlp.dense_4h_to_h.weight": "output.projection.weight", "mlp.dense_4h_to_h.bias": "output.projection.bias", "lm_head.weight": "head.weight", "lm_head.bias": "head.bias", } split = False if key in layer_rename_map: return split, prefix + layer_rename_map[key] # Handle transformer blocks match = re.match(r'^\w*\.(\d+)\.(\w+\.\w+\.\w+|\w+\.\w+)$', key) layer_number = int(match.group(1)) text = match.group(2) if "self_attention.query_key_value" in key: split = True return split, f"{prefix}blocks.{layer_number}." + layer_rename_map[text] def hf_to_ms(hf_weights, args, ms_dtype=mindspore.float32, for_save=False): """Convert hf layers to ms.""" ms_params = {} for k, v in hf_weights.items(): print(k, v.shape, v.dtype) split, new_name = layer_name_mapping(k) if split: if 'weight' in new_name: v = v.reshape(args.n_head, 3, args.hidden_size // args.n_head, v.shape[-1]) v_list = v.tensor_split(3, dim=1) for i in range(1, 4): tmp_name = new_name.format(i) print(v_list[i-1].shape) tmp_tensor = Tensor(v_list[i-1].reshape(-1, v_list[i-1].shape[-1]).float().detach().numpy(), ms_dtype) ms_params[tmp_name] = Parameter(tmp_tensor, name=tmp_name) else: v = v.reshape(args.n_head, 3, args.hidden_size // args.n_head) v_list = v.tensor_split(3, dim=1) for i in range(1, 4): tmp_name = new_name.format(i) print(v_list[i-1].shape) tmp_tensor = Tensor(v_list[i-1].reshape(-1).float().detach().numpy(), ms_dtype) ms_params[tmp_name] = Parameter(tmp_tensor, name=tmp_name) else: if ('projection' in new_name or 'mapping' in new_name) and 'weight' in new_name: new_tensor = Tensor(v.transpose(0, 1).float().detach().numpy(), ms_dtype) else: new_tensor = Tensor(v.float().detach().numpy(), ms_dtype) ms_params[new_name] = Parameter(new_tensor, name=new_name) if for_save: return [{'name': k, 'data': v} for k, v in ms_params.items()] return ms_params def process_hf_shard_files(file_list, args, save_dir=None, combine=False, ms_dtype=mindspore.float32): ''' torch ckpt files loop''' if save_dir and not os.path.exists(save_dir): os.makedirs(save_dir, exist_ok=True) combine_params = [] file = None for file in file_list: pt_states = torch.load(file, map_location='cpu') ms_params = hf_to_ms(pt_states, args, ms_dtype, True) if combine: combine_params.extend(ms_params) else: save_file = save_dir + '/' + file.split('/')[-1] if save_dir else file + '.ckpt' mindspore.save_checkpoint(ms_params, save_file) del pt_states del ms_params if combine: path = save_dir + '/' + 'combine.ckpt' if save_dir else \ '/'.join(file.split('/')[:-1]) + 'combine.ckpt' mindspore.save_checkpoint(combine_params, path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="Bloom convert script") parser.add_argument('--n_head', type=int, default=32, required=True, help="The number of head of the model to be converted.") parser.add_argument('--hidden_size', type=int, default=4096, required=True, help="The number of hidden size of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The input torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default=None, help="The output mindspore checkpoint path.") config = parser.parse_args() # convert hf ckpt to ms process_hf_shard_files(file_list=[config.torch_path], args=config, combine=True, save_dir=config.mindspore_path) pengcheng.mind-mindformer/mindformers/models/bloom/layers.py000066400000000000000000000735751476650141400247240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Bloom custom layers""" import math import numpy as np from mindspore.common.tensor import Tensor from mindspore import nn, ops, Parameter import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindformers.tools.logger import logger as mindformer_logger from mindformers.modules import AttentionMask from mindformers.modules.transformer.op_parallel_config import default_dpmp_config from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.transformer import MultiHeadAttention,\ TransformerEncoderLayer, TransformerEncoder from mindformers.modules.transformer.transformer import default_transformer_config, _get_lambda_func class BloomAttention(MultiHeadAttention): r"""The implementation of Bloom attention.""" def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, use_seq_parallel=False, use_select_recompute=False, parallel_config=default_dpmp_config): super(BloomAttention, self).__init__(batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate, attention_dropout_rate, compute_dtype, softmax_compute_type, param_init_type, use_past, parallel_config) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.inv_norm_factor = Tensor([1.0 / math.sqrt(self.size_per_head)]) self.beta = Tensor([1.0]) else: self.add_alibi = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.mul_alibi = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1,))) self.mul_alibi1 = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1,))) # [B, 1, S, S] + [B, H, S, S] self.add_atten_score = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] * [B, 1, S, 1] self.mul_key = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1, 1, 1, 1))) # [B, H, S, D] * [B, 1, S, 1] self.mul_value = P.Mul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (1, 1, 1, 1))) # [B, H, S, D] * [B, H, S, D] self.add_key = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] * [B, H, S, D] self.add_value = P.Add().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.inv_norm_factor = Tensor([1.0 / math.sqrt(self.size_per_head)]) self.beta = Tensor([1.0]) self.cast = P.Cast().shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.batch_matmul_trans_b = P.BatchMatMul(transpose_b=True).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) if use_select_recompute: mindformer_logger.info("Using select recompute mode!") self.cast.recompute() self.batch_matmul.recompute() self.batch_matmul_trans_b.recompute() self.sub.recompute() self.mul_alibi1.recompute() self.add.recompute() self.add_alibi.recompute() self.prob_dropout.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() if use_seq_parallel: mindformer_logger.info("Using seq parallel mode!") self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) # pylint: disable=arguments-differ def construct(self, query_tensor, key_tensor, value_tensor, alibi_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the BloomAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = query_tensor.shape batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = query_tensor.dtype query_tensor = query_tensor.astype(self.dtype) key_tensor = key_tensor.astype(self.dtype) value_tensor = value_tensor.astype(self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( query.reshape((batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # return query, query # the returned shape is [bs, num_heads, size_per_head, seq_length] key = self.transpose( key.reshape((batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( value.reshape((batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and attention_mask.ndim == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = (self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_key(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_value(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: current_index = batch_valid_length - 1 current_index = current_index.reshape((-1, 1, 1)) current_mask = (self.equal(self.range, current_index)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_key(key, self.expand_dims(current_mask, 3)) current_value = self.mul_value(value, self.expand_dims(current_mask, 3)) # Concat the previous saved state and current state key = self.add_key(key_past, current_key) value = self.add_value(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, alibi_tensor, attention_mask, batch_valid_length) # Output output = self.projection(attention) output = self.dropout(output) output = output.reshape(ori_shape) output = output.astype(ori_dtype) return output, layer_present def _softmax(self, attention_scores): """ :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ attention_probs = self.softmax(attention_scores) return attention_probs def _attn(self, query, key, value, alibi_tensor, attention_mask, batch_valid_length): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix alibi_tensor: the alibi matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] ori_dtype = query.dtype score = self.batch_matmul_trans_b(query.astype(self.dtype), key.astype(self.dtype)) score = self.add_alibi( self.mul_alibi1(score, self.inv_norm_factor.astype(ori_dtype)), self.mul_alibi(alibi_tensor, self.beta.astype(ori_dtype)) ) attention_scores = self.cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token # bs, 1 -> bs, 1, 1 batch_valid_length = batch_valid_length - 1 index = batch_valid_length.reshape((-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = (self.tensor_le(self.range, index)).astype(mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( Tensor((1.0,)).astype(attention_scores.dtype), attention_mask.astype(attention_scores.dtype)) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add_atten_score(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs.astype(self.dtype), value.astype(self.dtype)) weighted_values = weighted_values.astype(self.softmax_dtype) attention_merge = self._merge_heads(weighted_values) return attention_merge class BloomBlock(TransformerEncoderLayer): r"""A block of Bloom model.""" def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, use_seq_parallel=False, use_select_recompute=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(BloomBlock, self).__init__(batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate, hidden_dropout_rate, post_layernorm_residual, layernorm_compute_type, softmax_compute_type, param_init_type, hidden_act, use_past, moe_config, parallel_config) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.use_past = use_past attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = BloomAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, parallel_config=attention_parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.use_past = use_past attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = BloomAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, parallel_config=attention_parallel_config) if self.use_past: size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, seq_length, size_per_head) self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") # [B, H, S, D] self.assign_key_past = P.Assign().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # [B, H, S, D] self.assign_value_past = P.Assign().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) if use_seq_parallel: self.add.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1), (parallel_config.data_parallel*parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel*parallel_config.model_parallel, 1),)) if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) # pylint: disable=arguments-differ def construct(self, x, alibi_tensor, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = input_x.astype(self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_key_past(self.key_past, init_reset) key_reset = self.key_past self.assign_value_past(self.value_past, init_reset) value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, alibi_tensor, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = output_x.astype(self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_key_past(self.key_past, key_present) key_update = self.key_past self.assign_value_past(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = ops.depend(mlp_logit, value_update) mlp_logit = ops.depend(mlp_logit, key_update) output = self.add(x, mlp_logit) if self.use_moe: return output, layer_present, aux_loss return output, layer_present class BloomBlocks(TransformerEncoder): r"""All blocks of Bloom model.""" def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, use_seq_parallel=False, use_select_recompute=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(BloomBlocks, self).__init__(batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate, hidden_dropout_rate, hidden_act, post_layernorm_residual, layernorm_compute_type, softmax_compute_type, param_init_type, lambda_func, offset, use_past, moe_config, parallel_config) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = BloomBlock(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, moe_config=moe_config, parallel_config=config_to_layer) if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config, use_select_recompute=use_select_recompute) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = BloomBlock(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, use_seq_parallel=use_seq_parallel, use_select_recompute=use_select_recompute, moe_config=moe_config, parallel_config=config_to_layer) if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config, use_select_recompute=use_select_recompute) self.blocks.append(block) # pylint: disable=arguments-differ def construct(self, hidden_states, alibi_tensor, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, alibi_tensor, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class CausalMask(AttentionMask): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). """ def __init__(self, seq_length, parallel_config=default_dpmp_config): super(CausalMask, self).__init__(seq_length, parallel_config) self.seq_length = seq_length self.parallel_config = parallel_config def construct(self, input_mask): """Forward process of the CausalMask""" input_mask = self.not_equal(input_mask, 0).astype(mstype.float32) input_shape = input_mask.shape shape_right = (input_shape[0], 1, input_shape[1]) # Mask the padded inputs attention_mask = input_mask.reshape(shape_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask pengcheng.mind-mindformer/mindformers/models/build_config.py000066400000000000000000000051061476650141400247220ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ build model config modules """ from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_model_config( config: dict = None, default_args: dict = None, module_type: str = 'config', class_name: str = None, **kwargs): r"""Build model config For MindFormer. Instantiate the model config from MindFormerRegister's registry. Args: config (dict): The task model config's config. Default: None. default_args (dict): The default argument of model config API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'config'. class_name (str): The class name of model config API. Default: None. Return: The function instance of model config API. Examples: >>> from mindformers import build_model_config >>> model_config = {'type': 'ViTConfig'} >>> # 1) use config dict to build model >>> model_config_from_config = build_model_config(model_config) >>> # 2) use class name to build model >>> model_config_class_name = build_model_config(class_name='ViTConfig') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.text_config is not None: config.text_config = build_model_config(config.text_config) if config.vision_config is not None: config.vision_config = build_model_config(config.vision_config) if config.head_config is not None: config.head_config = build_model_config(config.head_config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.CONFIG, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/models/build_model.py000066400000000000000000000102021476650141400245460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Model API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_config import build_model_config def build_model( config: dict = None, default_args: dict = None, module_type: str = 'models', class_name: str = None, **kwargs): r"""Build model For MindFormer. Instantiate the model from MindFormerRegister's registry. Args: config (dict): The task model's config. Default: None. default_args (dict): The default argument of model API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'model'. class_name (str): The class name of model API. Default: None. Return: The function instance of model API. Examples: >>> from mindformers import build_model >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> model_from_config = build_model(config.model) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if default_args is None: default_args = {} model_config = build_model_config(config.model_config, default_args=default_args) if model_config is not None: if default_args is not None: for key, value in default_args.items(): model_config.__setattr__(key, value) default_args = {} default_args.setdefault('config', model_config) return MindFormerRegister.get_instance_from_cfg( config.arch, MindFormerModuleType.MODELS, default_args=default_args) return None return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def build_encoder( config: dict = None, default_args: dict = None, module_type: str = 'encoder', class_name: str = None, **kwargs): """Build encoder API.""" if config is None and class_name is None: return None if config is not None: cfg_encoders = config if not isinstance(cfg_encoders, list): return MindFormerRegister.get_instance_from_cfg( cfg_encoders, MindFormerModuleType.ENCODER, default_args=default_args) encoders = [] for encoder in cfg_encoders: encoder_op = MindFormerRegister.get_instance_from_cfg( encoder, MindFormerModuleType.ENCODER) encoders.append(encoder_op) return encoders return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def build_head( config: dict = None, default_args: dict = None, module_type: str = 'head', class_name: str = None, **kwargs): """Build head API.""" if config is None and class_name is None: return None if config is not None: cfg_heads = config if not isinstance(cfg_heads, list): return MindFormerRegister.get_instance_from_cfg( cfg_heads, MindFormerModuleType.HEAD, default_args=default_args) heads = [] for head in cfg_heads: head_op = MindFormerRegister.get_instance_from_cfg( head, MindFormerModuleType.HEAD) heads.append(head_op) return heads return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/models/build_processor.py000066400000000000000000000046351476650141400255020ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Processor API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .build_tokenizer import build_tokenizer def build_processor( config: dict = None, default_args: dict = None, module_type: str = 'processor', class_name: str = None, **kwargs): r"""Build processor For MindFormer. Instantiate the processor from MindFormerRegister's registry. Args: config (dict): The task processor's config. Default: None. default_args (dict): The default argument of processor API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'processor'. class_name (str): The class name of processor API. Default: None. Return: The function instance of processor API. Examples: >>> from mindformers import build_processor >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/vit/run_vit_base_p16_224_100ep.yaml') >>> processor_from_config = build_processor(config.processor) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.image_processor is not None: config.image_processor = build_processor(config.image_processor) if config.tokenizer is not None: config.tokenizer = build_tokenizer(config.tokenizer, **kwargs) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.PROCESSOR, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/models/build_tokenizer.py000066400000000000000000000070201476650141400254640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Tokenizer API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from ..mindformer_book import MindFormerBook def check_and_add_vocab_file_path(config, **kwargs): """And the vocab file path to the config if there is not vocab file in the config""" if 'vocab_file' in config: return class_name = config['type'] dynamic_class = MindFormerRegister.get_cls(module_type='tokenizer', class_name=class_name) # If the tokenizer does not require the vocab_file, just stop if not dynamic_class.vocab_files_names: return name_or_path = class_name path = kwargs.pop('lib_path', None) remote_tokenizer_support_list = MindFormerBook.get_tokenizer_url_support_list().keys() if name_or_path not in remote_tokenizer_support_list and path: read_vocab_file_dict, read_tokenizer_file_dict = \ dynamic_class.read_files_according_specific_by_tokenizer(name_or_path=path) config.update(read_vocab_file_dict) config.update(read_tokenizer_file_dict) else: checkpoint_name_or_path = config.get("checkpoint_name_or_path") if checkpoint_name_or_path in dynamic_class.get_support_list(): support_name = checkpoint_name_or_path else: support_name = dynamic_class.get_support_list()[0] vocab_file = dynamic_class.cache_vocab_files(name_or_path=support_name) config.update(vocab_file) def build_tokenizer( config: dict = None, default_args: dict = None, module_type: str = 'tokenizer', class_name: str = None, **kwargs): r"""Build tokenizer For MindFormer. Instantiate the tokenizer from MindFormerRegister's registry. Args: config (dict): The task tokenizer's config. Default: None. default_args (dict): The default argument of tokenizer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'tokenizer'. class_name (str): The class name of tokenizer API. Default: None. Return: The function instance of tokenizer API. Examples: >>> from mindformers import build_tokenizer >>> from mindformers.tools.register import MindFormerConfig >>> config = MindFormerConfig('configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yaml') >>> tokenizer_from_config = build_tokenizer(config.processor.tokenizer) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) check_and_add_vocab_file_path(config, **kwargs) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.TOKENIZER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/models/clip/000077500000000000000000000000001476650141400226515ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/clip/__init__.py000066400000000000000000000020061476650141400247600ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """clip init""" from .clip_config import CLIPTextConfig, CLIPVisionConfig, CLIPConfig from .clip import CLIPModel from .clip_tokenizer import CLIPTokenizer from .clip_processor import CLIPProcessor, CLIPImageProcessor __all__ = ['CLIPModel', 'CLIPConfig', 'CLIPVisionConfig', 'CLIPImageProcessor', 'CLIPTextConfig', 'CLIPTokenizer', 'CLIPProcessor'] pengcheng.mind-mindformer/mindformers/models/clip/clip.py000066400000000000000000000261311476650141400241550ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPModel """ from typing import Optional, Union import numpy as np import mindspore as ms from mindspore import nn from mindspore.ops import functional as F from mindspore.common.initializer import Normal, initializer from mindspore import Parameter, Tensor import mindspore.ops as ops from mindformers.version_control import get_norm from ...mindformer_book import MindFormerBook from ..base_model import BaseModel from .clip_modules import VisionTransformer, Transformer, LayerNorm from .clip_config import CLIPConfig from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.MODELS) class CLIPModel(BaseModel): r"""CLIPModel. The supported model name could be selected from CLIPModel.show_support_list(). Args: config (CLIPConfig): The config of clip model, which could be obtained by CLIPConfig class. Examples: >>> from mindformers import CLIPModel >>> CLIPModel.show_support_list() INFO - support list of CLIPModel is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> type(model) """ _support_list = MindFormerBook.get_model_support_list()['clip'] def __init__(self, config: CLIPConfig): super(CLIPModel, self).__init__(config) self.dtype = self.get_dtype(config.dtype) self.cross_entropy = nn.SoftmaxCrossEntropyWithLogits(reduction="mean", sparse=True) self.max_position_embeddings = config.text_config.max_position_embeddings self.visual = VisionTransformer( input_resolution=config.vision_config.image_size, patch_size=config.vision_config.patch_size, width=config.vision_config.hidden_size, layers=config.vision_config.num_hidden_layers, heads=config.vision_config.num_attention_heads, output_dim=config.projection_dim, dtype=self.dtype ) self.transformer = Transformer( width=config.text_config.hidden_size, layers=config.text_config.num_hidden_layers, heads=config.text_config.num_attention_heads, dtype=self.dtype, attn_mask=self.build_attention_mask() ) self.token_embedding = \ nn.Embedding(config.text_config.vocab_size, config.text_config.hidden_size, embedding_table=Normal(mean=0.0, sigma=0.02)) self.positional_embedding = Parameter(initializer( Normal(mean=0.0, sigma=0.01), [config.text_config.max_position_embeddings, config.text_config.hidden_size])) self.ln_final = LayerNorm([config.text_config.hidden_size]) self.text_projection = Parameter(initializer( Normal(mean=0.0, sigma=config.text_config.hidden_size ** -0.5), [config.text_config.hidden_size, config.projection_dim], ms.float32)) self.logit_scale = Parameter(Tensor(np.log(1 / 0.07)).astype(ms.float32)) self.exp = ops.Exp() self.norm = get_norm() self.load_checkpoint(config) def get_dtype(self, dtype: str): """Get_dtype""" if dtype == "float16": return ms.float16 if dtype == "float32": return ms.float32 raise TypeError("unsupported data type.") def construct(self, image: ms.Tensor, text: ms.Tensor, label: Optional[Union[ms.Tensor, np.ndarray]] = None, input_ids: Optional[ms.Tensor] = None, pixel_values: Optional[ms.Tensor] = None): r"""Construct Args: image (Tensor): A image tensor processed by image_processor. text (Tensor): A text id tensor processed by tokenizer. input_ids (Optional[ms.Tensor]): Equal to "text", if "input_ids" is set, "text" is useless. pixel_values (Optional[ms.Tensor]): Equal to "image", if "pixel_values" is set, "image" is useless. label (Optional[Union[ms.Tensor, np.ndarray]]): The classification label. Returns: if not self.trainining: if label is None: logits_per_image: Similarity between image and text. logits_per_text: Similarity between text and image. else: logits_per_image: Similarity between image and text. label: The classification label. else: loss: Constructive language image pretraining loss. Examples: >>> import numpy as np >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_image_batch = np.random.random((5, 3, 578, 213)) >>> fake_text_batch = ["a boy", "a girl", "a women", "a men"] >>> model(**processor(fake_image_batch, fake_text_batch)) (Tensor(shape=[5, 4], dtype=Float32, value= [[2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001], [2.26102257e+001, 2.29256859e+001, 2.40180817e+001, 2.30393028e+001], [2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001], [2.26109924e+001, 2.29261818e+001, 2.40193062e+001, 2.30404854e+001], [2.26097965e+001, 2.29247952e+001, 2.40179482e+001, 2.30396290e+001]]), Tensor(shape=[4, 5], dtype=Float32, value= ...)) """ if pixel_values is not None: image = pixel_values if input_ids is not None: text = input_ids if len(text.shape) == 3: text = text[0].squeeze() image_features = self.get_image_features(image) text_features = self. get_text_features(text) image_features = image_features / self.norm(image_features, dim=1, keepdim=True) text_features = text_features / self.norm(text_features, dim=1, keepdim=True) logit_scale = self.exp(self.logit_scale) if not self.training: if label is None: logits_per_image = ops.matmul(logit_scale * image_features, text_features.T) logits_per_text = logits_per_image.T return logits_per_image, logits_per_text logits_per_image = ops.matmul(logit_scale * image_features, text_features.T) return logits_per_image, label logits = ops.matmul(logit_scale * image_features, text_features.T) batch_size, _ = F.shape(logits) labels = ms.Tensor(np.arange(batch_size)) images_loss = self.cross_entropy(logits, labels) texts_loss = self.cross_entropy(logits.T, labels) loss = (images_loss + texts_loss) / 2 return loss def build_attention_mask(self): """Build_attention_mask""" mask = np.ones((self.max_position_embeddings, self.max_position_embeddings)) mask = np.triu(mask * float("-inf"), k=1) return Tensor(mask).astype(self.dtype) def get_image_features(self, image: ms.Tensor, pixel_values: Optional[ms.Tensor] = None): r"""Get_image_features Args: image (ms.Tensor): A image tensor processed by image_processor. pixel_values (Optional[ms.Tensor]): Equal to "image", if "pixel_values" is set, "image" is useless. Returns: Image feature. Examples: >>> import numpy as np >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_image_batch = np.random.random((5, 3, 578, 213)) >>> model.get_image_features(processor.image_processor(fake_image_batch)) Tensor(shape=[5, 512], dtype=Float32, value= [[-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001], [-1.50103331e-001, -2.63622820e-001, -5.65623760e-001 ... -2.93337226e-001], [-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001], [-1.49712294e-001, -2.64100820e-001, -5.65740824e-001 ... -2.93599486e-001], [-1.50102973e-001, -2.63687313e-001, -5.65953791e-001 ... -2.93511450e-001]]) """ if pixel_values is not None: image = pixel_values image = image.astype(self.dtype) return self.visual(image) def get_text_features(self, text: ms.Tensor, input_ids: Optional[ms.Tensor] = None): r"""Get_text_features Args: text (ms.Tensor): A text id tensor processed by tokenizer. input_ids (Optional[ms.Tensor]): Equal to "text", if "input_ids" is set, "text" is useless. Returns: Text feature. Examples: >>> from mindformers import CLIPModel, CLIPProcessor >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> model = CLIPModel.from_pretrained('clip_vit_b_32') >>> fake_text_batch = ["a boy", "a girl", "a women", "a men"] >>> text = processor.tokenizer( ... fake_text_batch, max_length=77, padding="max_length", return_tensors="ms" ... )["input_ids"] >>> model.get_text_features(text) Tensor(shape=[4, 512], dtype=Float32, value= [[6.03631809e-002, 1.79528534e-001, ... -2.23753393e-001, 1.42413378e-002], [1.28974199e-001, 7.46373609e-002, ... -3.68579805e-001, 1.53980583e-001], [9.89909172e-002, 2.01410800e-002, ... -2.54495114e-001, 7.68117979e-002], [3.16975415e-002, 2.26992741e-001, ... -5.22942394e-002, 1.98922127e-001]]) """ if input_ids is not None: text = input_ids text_ = self.token_embedding(text).astype(self.dtype) text_ = ops.Add()(text_, self.positional_embedding).astype(self.dtype) text_ = text_.transpose(1, 0, 2) text_ = self.transformer(text_) text_ = text_.transpose(1, 0, 2) text_ = self.ln_final(text_).astype(ms.float32) text_ = ops.MatMul()( text_[ms.numpy.arange(text_.shape[0]), text.argmax(-1)], self.text_projection) return text_ pengcheng.mind-mindformer/mindformers/models/clip/clip_config.py000066400000000000000000000246571476650141400255150ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPConfig class, which consists of CLIPTextConfig and CLIPVisionConfig All configs here are inherited from BaseConfig """ from typing import Optional from ..base_config import BaseConfig from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook from ...tools import logger @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPTextConfig(BaseConfig): r""" Config For CLIP Text Module Args: vocab_size (Optional[int]): Vocabulary size of the CLIP text model. hidden_size (Optional[int]): The dims of text features. intermediate_size (Optional[int]): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (Optional[int]): The number of transformer layers in text encoder. num_attention_heads (Optional[int]): Number of attention heads for each attention layer in the Transformer encoder. max_position_embeddings (Optional[int]): The maximum sequence length that this model might ever be used with. hidden_act (Optional[str]): The non-linear activation function (function or string) in the encoder and pooler. Only "quick_gelu" supported currently. attention_dropout (Optional[float]): The dropout ratio for the attention probabilities. dropout (Optional[float]): The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. initializer_range (Optional[float]): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (Optional[float]): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Examples: >>> from mindformers import CLIPTextConfig >>> CLIPTextConfig(hidden_size=256, vocab_size=40000) {'vocab_size': 40000, 'hidden_size': 256, 'intermediate_size': 2048, 'num_hidden_layers': 12, 'num_attention_heads': 8, 'max_position_embeddings': 77, 'hidden_act': 'quick_gelu', 'attention_dropout': 0.0, 'initializer_range': 0.02, 'initializer_factor': 1.0} """ def __init__(self, vocab_size: Optional[int] = 49408, hidden_size: Optional[int] = 512, intermediate_size: Optional[int] = 2048, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 8, max_position_embeddings: Optional[int] = 77, hidden_act: Optional[str] = "quick_gelu", attention_dropout: Optional[float] = 0.0, drop_out: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs): super(CLIPTextConfig, self).__init__(**kwargs) self.vocab_size = vocab_size self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.max_position_embeddings = max_position_embeddings self.hidden_act = hidden_act self.attention_dropout = attention_dropout self.dropout = drop_out self.initializer_range = initializer_range self.initializer_factor = initializer_factor @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPVisionConfig(BaseConfig): r""" Config For CLIP Vision Module Args: hidden_size (Optional[int]): Dimensionality of the encoder layers and the pooler layer. intermediate_size (Optional[int]): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. num_hidden_layers (Optional[int]): Number of hidden layers in the Transformer encoder. num_attention_heads (Optional[int]): Number of attention heads for each attention layer in the Transformer encoder. image_size (Optional[int]): The size (resolution) of each image. patch_size (Optional[int]): The size (resolution) of each patch. hidden_act (Optional[str]): The non-linear activation function (function or string) in the encoder and pooler. Only "quick_gelu" supported currently. dropout (Optional[float]): The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler. attention_dropout (Optional[float]): The dropout ratio for the attention probabilities. initializer_range (Optional[float]): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. initializer_factor (Optional[float]): A factor for initializing all weight matrices (should be kept to 1, used internally for initialization testing). Examples: >>> from mindformers import CLIPVisionConfig >>> CLIPVisionConfig(hidden_size=512, image_size=256) {'hidden_size': 512, 'intermediate_size': 3072, 'num_hidden_layers': 12, 'num_attention_heads': 16, 'image_size': 256, 'patch_size': 32, 'hidden_act': 'quick_gelu', 'dropout': 0.0, 'attention_dropout': 0.0, 'initializer_range': 0.02, 'initializer_factor': 1.0} """ def __init__(self, hidden_size: Optional[int] = 768, intermediate_size: Optional[int] = 3072, num_hidden_layers: Optional[int] = 12, num_attention_heads: Optional[int] = 12, image_size: Optional[int] = 224, patch_size: Optional[int] = 32, hidden_act: Optional[str] = "quick_gelu", dropout: Optional[float] = 0.0, attention_dropout: Optional[float] = 0.0, initializer_range: Optional[float] = 0.02, initializer_factor: Optional[float] = 1.0, **kwargs): super(CLIPVisionConfig, self).__init__(**kwargs) self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.image_size = image_size self.patch_size = patch_size self.hidden_act = hidden_act self.dropout = dropout self.attention_dropout = attention_dropout self.initializer_range = initializer_range self.initializer_factor = initializer_factor @MindFormerRegister.register(MindFormerModuleType.CONFIG) class CLIPConfig(BaseConfig): r""" Config For CLIP Model Args: text_config (Optional[CLIPTextConfig]): The config of text transformer. vision_config (Optional[CLIPVisionConfig]): The config of vision transformer. projection_dim (Optional[int]): The dims of projected featrues. logit_scale_init_value (Optional[float]): The initial value of the *logit_scale* parameter. checkpoint_name_or_path (Optional[str]): The path of checkpoint(.ckpt) or a support model name in CLIPConfig.show_support_list() dtype (Optional[str]): The type of tensors in model, ["float16", "float32"]. Raises: TypeError: If the type of text_config is not CLIPTextConfig or the type of vision_config is not CLIPVisionConfig Examples: >>> from mindformers import CLIPConfig >>> CLIPConfig.show_support_list() INFO - support list of CLIPConfig is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> config = CLIPConfig.from_pretrained('clip_vit_b_32') >>> config {'text_config': {'hidden_size': 512, 'vocab_size': 49408, 'max_position_embeddings': 77, 'num_hidden_layers': 12}, 'vision_config': {'hidden_size': 768, 'image_size': 224, 'patch_size': 32, 'num_hidden_layers': 12}, 'projection_dim': 512, 'ratio': 64, 'checkpoint_name_or_path': 'clip_vit_b_32', 'dtype': 'float16'} >>> config.save_pretrained(save_directory="./", save_name="clip_config") INFO - config saved successfully! """ _support_list = MindFormerBook.get_config_support_list()['clip'] def __init__(self, text_config: Optional[CLIPTextConfig] = None, vision_config: Optional[CLIPVisionConfig] = None, projection_dim: Optional[int] = 512, logit_scale_init_value: Optional[float] = 2.6592, checkpoint_name_or_path: Optional[str] = "", dtype: Optional[str] = "float16", **kwargs): if text_config is None: text_config = CLIPTextConfig() logger.info("text_config is None. Initializing the CLIPTextConfig with default values.") elif isinstance(text_config, CLIPTextConfig): pass else: raise TypeError(f"text_config should be a " f"CLIpTextConfig class, but got {type(CLIPTextConfig)}") if vision_config is None: vision_config = CLIPVisionConfig() logger.info("vision_config is None." " Initializing the CLIPTextConfig with default values.") elif isinstance(vision_config, CLIPVisionConfig): pass else: raise TypeError("text_config should be a CLIPVisionConfig" f" class, but got {type(CLIPVisionConfig)}") super(CLIPConfig, self).__init__(**kwargs) self.text_config = text_config self.vision_config = vision_config self.projection_dim = projection_dim self.logit_scale_init_value = logit_scale_init_value self.checkpoint_name_or_path = checkpoint_name_or_path self.dtype = dtype pengcheng.mind-mindformer/mindformers/models/clip/clip_modules.py000066400000000000000000000236131476650141400257070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/openai/CLIP/blob/main/clip/clip.py # ============================================================================ """ Modules of CLIPModel, including MultiheadAttention，VisionTransformer, QuickGELU, ResidualAttentionBlock, Transformer """ from collections import OrderedDict from typing import Optional import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore import dtype as mstype from mindspore.ops import operations as P from mindspore import Parameter, Tensor from mindspore.common.initializer import Normal class LayerNorm(nn.LayerNorm): r"""Implementation That Supports Fp16 Inputs But Fp32 Gains Biases. Args: x (ms.Tensor): Input tensor. The detailed function could refer to mindspore.nn.LayerNorm. Return: y (ms.Tensor): Normalized tensor. """ def construct(self, x: ms.Tensor): """construct""" y = super().construct(P.Cast()(x, ms.float32)) y = P.Cast()(y, x.dtype) return y class MultiheadAttention(nn.Cell): r"""MultiheadAttention, With Layers As Input For Initialization Args: d_model (int): The feature dimension n_head (int): The number of attention heads layers (int): The number of transformers, used for weight initialization dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, d_model: int, n_head: int, layers: int, dtype: mstype): super(MultiheadAttention, self).__init__() self.num_heads = n_head self.head_dim = d_model // n_head self.softmax = nn.Softmax(-1) self.transpose = ops.Transpose() self.scaling = self.head_dim ** -0.5 proj_std = (d_model ** -0.5) * ((2 * layers) ** -0.5) attn_std = d_model ** -0.5 self.out_proj = nn.Dense(d_model, d_model, weight_init=Normal(mean=0.0, sigma=proj_std)).to_float(dtype) self.in_proj = nn.Dense(d_model, 3 * d_model, weight_init=Normal(mean=0.0, sigma=attn_std)).to_float(dtype) def construct(self, query: ms.Tensor, attn_mask: Optional[ms.Tensor] = None): r"""Construct Args: query (ms.Tensor): query of attention. attn_mask (Optional[ms.Tensor]): attention mask. Returns: attn_output (ms.Tensor): attention output. """ len_tgt, batch_size, width = query.shape qkv = self.in_proj(query).view(len_tgt, batch_size, 3, width).transpose((2, 0, 1, 3)) att_q = qkv[0:1] att_q = ops.Squeeze(0)(att_q) att_q = att_q * self.scaling att_q = att_q.view(len_tgt, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) att_k = qkv[1:2] att_k = ops.Squeeze(0)(att_k) att_k = att_k.view(-1, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) att_v = qkv[2:3] att_v = ops.Squeeze(0)(att_v) att_v = att_v.view(-1, batch_size * self.num_heads, self.head_dim).transpose((1, 0, 2)) if attn_mask is not None: attn_output_weights = attn_mask + ops.matmul(att_q, att_k.transpose((0, 2, 1))) else: attn_output_weights = ops.matmul(att_q, att_k.transpose((0, 2, 1))) attn_output_weights = self.softmax(attn_output_weights) attn_output = ops.matmul(attn_output_weights, att_v) attn_output = self.transpose(attn_output, (1, 0, 2)) attn_output = attn_output.view(len_tgt, batch_size, width) attn_output = self.out_proj(attn_output) return attn_output class VisionTransformer(nn.Cell): r"""VisionTransformer Of CLIPModel Args: input_resolution (int): The image size of input. patch_size (int): The patch size of vision transformer. width (int): The dimension of vision transformer. layers (int): The number of layers of vision transformer. heads (int): The number of attention heads. output_dim (int): The output dimension of vision transformer. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, input_resolution: int, patch_size: int, width: int, layers: int, heads: int, output_dim: int, dtype: mstype): super(VisionTransformer, self).__init__() self.conv1 = \ nn.Conv2d( in_channels=3, out_channels=width, kernel_size=patch_size, stride=patch_size, has_bias=False).to_float(dtype) scale = width ** -0.5 self.class_embedding = \ Parameter(scale * Tensor(np.random.normal(0, 1, size=(width))).astype(ms.float32)) self.positional_embedding = \ Parameter(scale * Tensor( np.random.normal(0, 1, size=( (input_resolution // patch_size) ** 2 + 1, width))).astype(ms.float32)) self.ln_pre = LayerNorm([width], epsilon=1e-5) self.transformer = Transformer(width, layers, heads, dtype) self.ln_post = LayerNorm([width], epsilon=1e-5) self.proj = \ Parameter(scale * Tensor(np.random.normal(0, 1, size=(width, output_dim))).astype(ms.float32)) self.cat = ops.Concat(1) self.tile = ops.Tile() self.slice = P.StridedSlice() self.dtype = dtype def construct(self, input_x: ms.Tensor): r"""Construct Args: input_x (ms.Tensor): Input tensor. Returns: input_x (ms.Tensor): Output tensor. """ input_x = self.conv1(input_x) input_x = input_x.reshape(input_x.shape[0], input_x.shape[1], -1) input_x = input_x.transpose(0, 2, 1) class_embedding = self.tile(self.class_embedding, (input_x.shape[0]+1, 1, 1)).astype(self.dtype) input_x = self.cat([ self.slice(class_embedding, (0, 0, 0), (-1, class_embedding.shape[1], class_embedding.shape[2]), (1, 1, 1)), input_x ]) input_x = ops.Add()(input_x, self.positional_embedding) input_x = self.ln_pre(input_x) input_x = input_x.transpose(1, 0, 2) input_x = self.transformer(input_x) input_x = input_x.transpose(1, 0, 2) input_x = self.ln_post(input_x[:, 0, :]) input_x = ops.matmul(input_x, self.proj) return input_x class QuickGELU(nn.Cell): r"""QuickGELU of CLIP""" def __init__(self, ratio: Optional[int] = 1.702): super(QuickGELU, self).__init__() self.ratio = ratio self.sigmoid = nn.Sigmoid() def construct(self, input_x: ms.Tensor): '''construct''' return input_x * self.sigmoid(self.ratio * input_x) class ResidualAttentionBlock(nn.Cell): r""" ResidualAttentionBlock of CLIP Args: d_model (int): The dimension of features. n_head (int): The number of attention heads. layers (int): The number of transformer layers for weight initialization. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. attn_mask (Optional[ms.Tensor]): attention mask. """ def __init__(self, d_model: int, n_head: int, layers: int, dtype: mstype, attn_mask: Optional[ms.Tensor] = None): super(ResidualAttentionBlock, self).__init__() proj_std = (d_model ** -0.5) * ((2 * layers) ** -0.5) fc_std = (2 * d_model) ** -0.5 self.dtype = dtype self.attn = MultiheadAttention(d_model, n_head, layers, dtype) self.ln_1 = LayerNorm([d_model], epsilon=1e-5) self.mlp = nn.SequentialCell(OrderedDict([ ("c_fc", nn.Dense(d_model, d_model * 4, weight_init=Normal(mean=0.0, sigma=fc_std)).to_float(dtype)), ("gelu", QuickGELU()), ("c_proj", nn.Dense(d_model * 4, d_model, weight_init=Normal(mean=0.0, sigma=proj_std)).to_float(dtype)) ])) self.ln_2 = LayerNorm([d_model], epsilon=1e-5) self.attn_mask = attn_mask def construct(self, input_x: ms.Tensor): r"""Construct""" input_x = ops.Add()(input_x, self.attention(self.ln_1(input_x))) input_x = ops.Add()(input_x, self.mlp(self.ln_2(input_x))) return input_x def attention(self, input_x: ms.Tensor): r"""Attention""" return self.attn(input_x, self.attn_mask) class Transformer(nn.Cell): r""" Text Transformer of CLIP Args: width (int): The dimension of input features. layers (int): The number of transformer layers. heads (int): The number of attention heads. attn_mask (ms.Tensor): Attention mask. dtype (mstype): The type of calculation, [mstype.float32, mstype.float16]. """ def __init__(self, width, layers, heads, dtype, attn_mask=None): super(Transformer, self).__init__() self.width = width self.layers = layers self.resblocks = nn.SequentialCell( *[ResidualAttentionBlock(width, heads, layers, dtype, attn_mask) for _ in range(layers)] ) def construct(self, input_x): r"""Construct""" return self.resblocks(input_x) pengcheng.mind-mindformer/mindformers/models/clip/clip_processor.py000066400000000000000000000154001476650141400262510ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ CLIPProcessor """ from typing import Optional, Union, List import numpy as np import PIL import mindspore as ms from mindformers.dataset import ( BCHW2BHWC, BatchResize, BatchToTensor, BatchNormalize, BatchCenterCrop, BatchPILize ) from mindformers.mindformer_book import MindFormerBook from ..base_processor import BaseImageProcessor from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class CLIPImageProcessor(BaseImageProcessor): """ CLIPImageProcessor. Args: image_resolution (int): The target size. Examples: >>> from mindformers import CLIPImageProcessor >>> from mindformers.tools.image_tools import load_image >>> processor = CLIPImageProcessor(image_resolution=256) >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> processor(image) Tensor(shape=[1, 3, 256, 256], dtype=Float32, value= [[[[-1.52949083e+000, -1.52949083e+000, ... -1.48569560e+000, -1.50029397e+000], [-1.52949083e+000, -1.52949083e+000, ... -1.50029397e+000, -1.50029397e+000], [-1.51489246e+000, -1.51489246e+000, ... -1.48569560e+000, -1.48569560e+000], ... ... [8.66091192e-001, 8.80311251e-001, ... -1.36645925e+000, -1.45177972e+000], [8.09210956e-001, 8.23431015e-001, ... -1.29535890e+000, -1.43755960e+000], [7.09670484e-001, 7.94990897e-001, ... -1.26691878e+000, -1.42333949e+000]]]]) """ def __init__(self, image_resolution: Optional[int] = 224): super(CLIPImageProcessor, self).__init__( image_resolution=image_resolution) self.bchw2bhwc = BCHW2BHWC() self.batch_pilizer = BatchPILize() self.batch_resizer = BatchResize(image_resolution) self.batch_crop = BatchCenterCrop(image_resolution) self.batch_totensor = BatchToTensor() self.batch_normalizer = BatchNormalize() def preprocess(self, images: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]], **kwargs): r""" Preprocess Required By Base Processor. Args: images (ms.Tensor, PIL.Image, numpy.array, List[PIL.Image]): A batch of images. Return: A 4-rank tensor for a batch of images. """ if not self._bhwc_check(images): images = self.bchw2bhwc(images) images = self.batch_pilizer(images) images = self.batch_resizer(images) images = self.batch_crop(images) images = self.batch_totensor(images) images = self.batch_normalizer(images) kwargs.pop("other", None) if isinstance(images, list): return ms.Tensor(np.row_stack([np.expand_dims(item, axis=0) for item in images])) if len(images.shape) == 4: return ms.Tensor(images) return ms.Tensor(np.expand_dims(images, axis=0)) def _bhwc_check(self, image_batch: Union[ms.Tensor, PIL.Image.Image, np.ndarray, List[PIL.Image.Image]]): r"""Bhwc_check""" if isinstance(image_batch, np.ndarray): if image_batch.shape[-1] == 3: return True if isinstance(image_batch, ms.Tensor): if image_batch.asnumpy().shape[-1] == 3: return True if isinstance(image_batch, (list, PIL.Image.Image)): return True return False @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class CLIPProcessor(BaseProcessor): r"""CLIP Processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. Args: image_processor (BaseImageProcessor): Used for process image data. tokenizer (BaseTokenizer): Used for process text data. max_length (Optional[int]): The length of text tokens. padding (Optional[str]): The padding strategy of tokenizer, [None, "max_length"]. return_tensors (Optional[str]): The type of returned tensors for tokenizer, [None, "ms"]. Examples: >>> from mindformers import CLIPProcessor >>> from mindformers.tools.image_tools import load_image >>> image = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> text = ["a boy", "a girl"] >>> CLIPProcessor.show_support_list() INFO - support list of CLIP Processor is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> processor = CLIPProcessor.from_pretrained('clip_vit_b_32') >>> processor(image, text) {'image': Tensor(shape=[1, 3, 224, 224], dtype=Float32, value= [[[[-1.52949083e+000, -1.52949083e+000,... -1.48569560e+000, -1.50029397e+000], [-1.52949083e+000, -1.52949083e+000, ... -1.50029397e+000, -1.50029397e+000], [-1.50029397e+000, -1.50029397e+000 ... -1.48569560e+000, -1.50029397e+000], ... [8.23431015e-001, 8.80311251e-001, ... -1.33801913e+000, -1.43755960e+000], [7.80770779e-001, 8.37651074e-001, ... -1.23847866e+000, -1.39489937e+000], [6.10130012e-001, 7.66550720e-001, ... -1.19581854e+000, -1.38067937e+000]]]]), 'text': Tensor(shape=[2, 77], dtype=Int32, value= [[49406, 320, 1876 ... 0, 0, 0], [49406, 320, 1611 ... 0, 0, 0]])} """ _support_list = MindFormerBook.get_processor_support_list()['clip'] def __init__(self, image_processor, tokenizer, max_length=77, padding='max_length', return_tensors='ms'): super(CLIPProcessor, self).__init__( image_processor=image_processor, tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors) pengcheng.mind-mindformer/mindformers/models/clip/clip_tokenizer.py000066400000000000000000000275051476650141400262550ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ This is a temporary version of clip tokenizer """ import gzip import html import os import shutil from functools import lru_cache from typing import List, Optional import ftfy import regex as re from mindformers.tools import logger from mindformers.tools.utils import try_sync_file from ...mindformer_book import MindFormerBook from ...tools.register import MindFormerRegister, MindFormerModuleType from ...tools.download_tools import download_with_progress_bar from ..base_tokenizer import Tokenizer __all__ = ['CLIPTokenizer'] VOCAB_FILES_NAMES = {'vocab_file': ['vocab.txt', 'bpe_simple_vocab_16e6.txt.gz']} @lru_cache() def default_bpe(): r"""Bpe path""" path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip', "bpe_simple_vocab_16e6.txt.gz") if not os.path.exists(path): url = "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/" \ "XFormer_for_mindspore/clip/bpe_simple_vocab_16e6.txt.gz" download_with_progress_bar(url, path) try_sync_file(path) return path def get_pairs(input_wd): r"""Get_pairs""" output = set() prev_char = input_wd[0] for char in input_wd[1:]: output.add((prev_char, char)) prev_char = char return output @lru_cache() def bytes_to_unicode(): r"""Bytes_to_unicode""" input_bt = list(range(ord("!"), ord("~")+1))\ +list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1)) output_cd = input_bt[:] num = 0 for item in range(2**8): if item not in input_bt: input_bt.append(item) output_cd.append(2**8+num) num += 1 output_cd = [chr(item) for item in output_cd] return dict(zip(input_bt, output_cd)) def whitespace_clean(input_text): r"""Whitespace clean""" input_text = re.sub(r'\s+', ' ', input_text) input_text = input_text.strip() return input_text def basic_clean(input_text): r"""Basic_clean""" input_text = ftfy.fix_text(input_text) input_text = html.unescape(html.unescape(input_text)) return input_text.strip() class TempTokenizer: r"""Simple Tokenizer""" def __init__(self, merges, vocab, flag_dict): self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} self.bpe_ranks = dict(zip(merges, range(len(merges)))) self.flag_dict = flag_dict self.encoder = dict(zip(vocab, range(len(vocab)))) self.decoder = {v: k for k, v in self.encoder.items()} def tokenize_alg(self, input_tk): r"""Bpe""" if input_tk in self.flag_dict: return self.flag_dict[input_tk] word = tuple(input_tk[:-1]) + (input_tk[-1] + '',) pairs = get_pairs(word) if not pairs: return input_tk+'' while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf'))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) new_word.extend(word[i:j]) i = j except ValueError: new_word.extend(word[i:]) break if word[i] == first and i < len(word)-1 and word[i+1] == second: new_word.append(first+second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break pairs = get_pairs(word) word = ' '.join(word) self.flag_dict[input_tk] = word return word def decode(self, input_ids): r"""Decode""" output_text = ''.join([self.decoder[input_id] for input_id in input_ids]) output_text = bytearray([self.byte_decoder[c] for c in output_text]).decode('utf-8', errors="replace").replace('', ' ') return output_text def encode(self, content): r"""Encode""" output_ids = [] content = whitespace_clean(basic_clean(content)).lower() for token in re.findall(self.pat, content): token = ''.join(self.byte_encoder[b] for b in token.encode('utf-8')) output_ids.extend(self.encoder[bpe_token] for bpe_token in self.tokenize_alg(token).split(' ')) print("res is:", output_ids) return output_ids @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class CLIPTokenizer(Tokenizer): r""" CLIP Tokenizer Args: vocab_file(str): The vocabulary file path. eos_token(str): The token that represents the end-of-sentence. Default "<|endoftext|>". bos_token(str): The token that represents the begin-of-sentence. Default "<|startoftext|>"". pad_token(str): The token that represents the pad. Default "<|endoftext|>". unk_token(str): The token that represents the unknown. Default "<|endoftext|>". add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import CLIPTokenizer >>> CLIPTokenizer.show_support_list() INFO - support list of CLIPTokenizer is: INFO - ['clip_vit_b_32'] INFO - ------------------------------------- >>> tokenizer = CLIPTokenizer.from_pretrained('clip_vit_b_32') >>> tokenizer("a boy") {'input_ids': [49406, 320, 1876, 49407], 'attention_mask': [1, 1, 1, 1]} """ model_input_names = ["input_ids", "attention_mask"] vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] '''clip tokenizer''' _support_list = MindFormerBook.get_tokenizer_support_list()['clip'] def __init__(self, vocab_file, eos_token="<|endoftext|>", bos_token="<|startoftext|>", pad_token="<|endoftext|>", unk_token="<|endoftext|>", add_bos_token=True, add_eos_token=True ): super(CLIPTokenizer, self).__init__( eos_token=eos_token, bos_token=bos_token, pad_token=pad_token, unk_token=unk_token ) self.path = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token merges = self._read_merge_files(vocab_file) vocab = list(bytes_to_unicode().values()) vocab = vocab + [v + '' for v in vocab] for merge in merges: vocab.append(''.join(merge)) vocab.extend([bos_token, eos_token]) flag_dict = {bos_token: bos_token, eos_token: eos_token} self.tool = TempTokenizer(merges, vocab, flag_dict) self.pat = re.compile(r"""<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re| 've|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+""", re.IGNORECASE) @staticmethod def _read_merge_files(text_path, start_pos=1, end_pos=49152-256-2+1): r"""Read the merge files""" with gzip.open(text_path) as fp: data = fp.read() merges = data.decode("utf-8").split('\n') merges = merges[start_pos: end_pos] new_list = [] for item in merges: new_list.append(tuple(item.split())) return new_list def _tokenize(self, text, **kwargs): r"""Tokenize""" output_ids = [] content = whitespace_clean(basic_clean(text)).lower() for token in re.findall(self.pat, content): token = ''.join(self.tool.byte_encoder[b] for b in token.encode('utf-8')) output_ids.extend(self.tool.tokenize_alg(token).split(' ')) return output_ids def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r""" Insert the special tokens to the input_ids. Currently, we support token_ids_0 is a list of ids. """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def save_vocabulary(self, save_directory, filename_prefix=None): r"""Save_vocabulary""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"][0]) shutil.copy(self.path, output_file_path) return output_file_path def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): r"""Tokenizer the input_text""" if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _convert_token_to_id(self, token): r"""Convert_token_to_id""" return self.tool.encoder[token] def _convert_id_to_token(self, index): r"""Convert_id_to_token""" return self.tool.decoder[index] @property def vocab_size(self): r"""Get the vocab size""" return len(self.tool.encoder) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab pengcheng.mind-mindformer/mindformers/models/clip/convert_weight.py000066400000000000000000000056551476650141400262650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert checkpoint from torch/huggingface """ import argparse import torch from mindspore import Tensor, Parameter import mindspore as ms def convert_weight(torch_path="ViT-B-32.pt", mindspore_path="clip_vit_b_32.ckpt"): r"""Convert Weight Convert clip_vit_b_32 weights from pytorch to mindspore pytorch and GPU required. Args: torch_path: The path to ViT-B-32.pt. mindspore_path： The save path for clip_vit_b_32.ckpt. """ param_dict = torch.load(torch_path) new_dict = [] for name, param in param_dict.items(): if "ln_pre.weight" in name or "ln_1.weight" in name or\ "ln_2.weight" in name or "ln_post.weight" in name or\ "ln_final.weight" in name: new_name = name.replace('weight', 'gamma') elif "ln_pre.bias" in name or "ln_1.bias" in name or\ "ln_2.bias" in name or "ln_post.bias" in name or\ "ln_final.bias" in name: new_name = name.replace('bias', 'beta') elif "in_proj_weight" in name: new_name = name.replace('in_proj_weight', 'in_proj.weight') elif "in_proj_bias" in name: new_name = name.replace('in_proj_bias', 'in_proj.bias') elif "token_embedding.weight" in name: new_name = name.replace('token_embedding.weight', 'token_embedding.embedding_table') else: new_name = name new_dict.append({ "name": new_name, "data": Parameter(Tensor(param.detach().numpy()), name=new_name) }) ms.save_checkpoint(new_dict, mindspore_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="clip weight convert script") parser.add_argument("--torch_path", type=str, default="ViT-B-32.pt", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="clip_vit_b_32.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-mindformer/mindformers/models/filip/000077500000000000000000000000001476650141400230255ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/filip/__init__.py000066400000000000000000000015461476650141400251440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """filip init""" from .filip_config import FilipTextConfig, FilipVisionConfig, FilipConfig from .filip import FilipModel __all__ = ['FilipModel', 'FilipConfig', 'FilipVisionConfig', 'FilipTextConfig'] pengcheng.mind-mindformer/mindformers/models/filip/filip.py000066400000000000000000000344111476650141400245050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ FilipModel """ import math import os import numpy as np import mindspore as ms import mindspore.ops as ops from mindspore.ops import functional as F from mindspore import nn, Parameter, Tensor from mindspore.common import dtype as mstype from mindspore.communication.management import get_group_size, get_rank, create_group from mindspore.nn.wrap.grad_reducer import DistributedGradReducer from mindspore.ops.primitive import constexpr from ...mindformer_book import MindFormerBook from ..base_model import BaseModel from .filip_modules import VisualTransformer, TextTransformer from ...tools.register import MindFormerRegister, MindFormerModuleType @constexpr def int_num(num): return int(num) class SoftCrossEntropyLoss(nn.Cell): """ SoftCrossEntropyLoss. """ def __init__(self, smooth=0.1): super(SoftCrossEntropyLoss, self).__init__() self.smooth = smooth self.expand_op = ops.ExpandDims() self.concat_op = ops.Concat(1) self.ones = ops.Ones() self.on_value = Tensor(1.0, ms.float32) self.off_value = Tensor(0.0, ms.float32) self.logsoftmax = nn.LogSoftmax() self.onehot = ops.OneHot() def construct(self, logit, target): target_onehot = self.onehot(target, logit.shape[1], self.on_value, self.off_value) target_onehot = (1 - self.smooth) * target_onehot + self.smooth / logit.shape[1] log_probs = self.logsoftmax(logit) loss = (- target_onehot * log_probs).mean(0).sum() return loss class FilipImgProcess(nn.Cell): """ FilipImgProcess, norm the input image features. """ def __init__(self): super(FilipImgProcess, self).__init__() self.image_norm = nn.Norm(axis=-1, keep_dims=True) def construct(self, image_features): image_features = image_features[:, 1:, :] image_features = image_features / self.image_norm(image_features) return image_features class FilipTextProcess(nn.Cell): """ FilipTextProcess, norm the input text features and get text padding mask. """ def __init__(self): super(FilipTextProcess, self).__init__() self.text_norm = nn.Norm(axis=-1, keep_dims=True) self.expand_dims = ops.ExpandDims() def construct(self, text_features, text): text_features = text_features / self.text_norm(text_features) text_pad_mask = text > 0 text_pad_mask_expand = self.expand_dims(text_pad_mask, -1) text_features = text_features * text_pad_mask_expand return text_features, text_pad_mask class FilipTopKFeature(nn.Cell): """ FilipTopKFeature, used in filip loss. """ def __init__(self, top_token_percent, local_group_size): super(FilipTopKFeature, self).__init__() self.percent_i, self.percent_t = top_token_percent self.local_group_size = local_group_size self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: rank_id = get_rank() sub_group = rank_id // local_group_size group = "{}-{}".format(sub_group * local_group_size, (sub_group + 1) * local_group_size) rank_ids = list(range(sub_group * local_group_size, (sub_group + 1) * local_group_size)) create_group(group, rank_ids) self.local_gather = ops.AllGather(group) self.matmul = ops.MatMul(transpose_b=True) self.topk = ops.TopK(sorted=True) self.expand_dim = ops.ExpandDims() def construct(self, image_feature, text_feature, text_pad_mask): """construct of FilipTopKFeature""" if self.rank_size > 1: text_local_gather = self.local_gather(text_feature) image_local_gather = self.local_gather(image_feature) else: text_local_gather = text_feature image_local_gather = image_feature # get image features batch_size, n_token1, feat_dim = image_feature.shape output = self.matmul(image_feature.reshape((-1, feat_dim)), text_local_gather.reshape((-1, feat_dim))) token_rep = output.reshape((batch_size, n_token1, -1)).max(2) max_token_idx = self.topk(token_rep, int_num(n_token1 * self.percent_i))[1] max_token_idx = ops.stop_gradient(max_token_idx) bs_index = nn.Range(batch_size)() bs_index = self.expand_dim(bs_index, 1) image_feature = image_feature[bs_index, max_token_idx] # get text features batch_size, n_token1, feat_dim = text_feature.shape output = self.matmul(text_feature.reshape((-1, feat_dim)), image_local_gather.reshape((-1, feat_dim))) token_rep = output.reshape((batch_size, n_token1, -1)).max(2) max_token_idx = self.topk(token_rep, int_num(n_token1 * self.percent_i))[1] max_token_idx = ops.stop_gradient(max_token_idx) bs_index = nn.Range(batch_size)() bs_index = self.expand_dim(bs_index, 1) text_feature = text_feature[bs_index, max_token_idx] text_pad_mask = text_pad_mask[bs_index, max_token_idx] return image_feature, text_feature, text_pad_mask class FilipLogit(nn.Cell): """ FilipLogit, compute logit. """ def __init__(self, use_mask_flag=False): super(FilipLogit, self).__init__() self.use_mask_flag = use_mask_flag self.matmul = ops.MatMul(transpose_b=True) logit_value = math.exp(3.8665097) self.logit_scale = Parameter(Tensor(np.log(logit_value), dtype=ms.float32)) self.exp = ops.Exp() self.cast = ops.Cast() def construct(self, rep1, rep2, mask=None): """construct of FilipLogit""" logit_scale = self.exp(self.logit_scale) batch_size1, n_token1, feat_dim = rep1.shape _, n_token2, _ = rep2.shape rep1 = rep1.reshape((-1, feat_dim)) rep2 = rep2.reshape((-1, feat_dim)) out = self.matmul(rep1, rep2) out = out.reshape((batch_size1, n_token1, -1, n_token2)).max(3) if self.use_mask_flag: out = out.sum(1) mask = self.cast(mask, ms.float32) mask_sum = mask.sum(axis=1, keepdims=True).clip(min=1.0, max=None) logits = out / mask_sum else: logits = out.mean(1) logits = logit_scale * logits logits = logits.clip(min=-100, max=100) return logits class FilipGather(nn.Cell): """ FilipGather, input image features, text features, and text padding mask, return logits. """ def __init__(self, top_token_percent, local_group_size): super(FilipGather, self).__init__() self.top_k_feature = FilipTopKFeature(top_token_percent, local_group_size) self.get_img_logits = FilipLogit() self.get_txt_logits = FilipLogit(True) self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: self.all_gather = ops.AllGather() def construct(self, image_features, text_features, text_pad_mask): """ construct of FilipGather """ if self.rank_size > 1: image_gather = self.all_gather(image_features) text_gather = self.all_gather(text_features) else: image_gather = image_features text_gather = text_features logits_per_image = self.get_img_logits(image_features, text_gather) logits_per_text = self.get_txt_logits(text_features, image_gather, text_pad_mask) return logits_per_image, logits_per_text class FilipLossCell(nn.Cell): """ FilipLossCell. """ def __init__(self, label_smooth=0.1, local_group_size=1, top_token_percent=(1, 1), ): super(FilipLossCell, self).__init__() self.local_group_size = local_group_size self.top_token_percent = top_token_percent self.process_img_features = FilipImgProcess() self.process_text_features = FilipTextProcess() self.rank_size = int(os.getenv("RANK_SIZE", '1')) if self.rank_size > 1: self.rank = get_rank() else: self.rank = 0 self.two = Tensor(2.0, dtype=mstype.float32) # self.all_gather = ops.AllGather() self.equal = ops.Equal() self.cast = ops.Cast() self.logsoftmax = nn.LogSoftmax() self.loss = SoftCrossEntropyLoss(label_smooth) self.gather_and_compute = FilipGather(top_token_percent, local_group_size) def construct(self, image_features, text_features, text): """construct of FilipLossCell""" image_features = self.process_img_features(image_features) text_features, text_pad_mask = self.process_text_features(text_features, text) batch_size = image_features.shape[0] logits_per_image, logits_per_text = self.gather_and_compute( image_features, text_features, text_pad_mask ) target = nn.Range(self.rank * batch_size, (self.rank + 1) * batch_size)() target = F.cast(target, ms.int32) loss = (self.loss(logits_per_image, target) + self.loss(logits_per_text, target)) / self.two return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class FilipModel(BaseModel): """ FilipModel. The supported model name could be selected from FilipModel.show_support_list(). Args: config (FilipConfig): the config of filip model. """ _support_list = MindFormerBook.get_model_support_list()['filip'] def __init__(self, config, is_training=True): super(FilipModel, self).__init__(config) self.image_encoder = VisualTransformer(config=config) self.text_encoder = TextTransformer(config=config) self.image_norm = nn.Norm(axis=-1, keep_dims=True) self.text_norm = nn.Norm(axis=-1, keep_dims=True) self.is_training = is_training self.loss = FilipLossCell() self.cast = ops.Cast() self.load_checkpoint(config) def get_image_feature(self, image): image_features = self.image_encoder(image) return image_features def get_text_feature(self, text): text_features = self.text_encoder(text) return text_features def construct(self, image, text): image_features = self.image_encoder(image) text_features = self.text_encoder(text) if not self.is_training: return image_features, text_features total_loss = self.loss(image_features, text_features, text) total_loss = self.cast(total_loss, mstype.float32) return total_loss GRADIENT_CLIP_TYPE = 1 GRADIENT_CLIP_VALUE = 1.0 clip_grad = ops.MultitypeFuncGraph("clip_grad") @clip_grad.register("Number", "Number", "Tensor") def _clip_grad(clip_type, clip_value, grad): """ Clip gradients. Inputs: clip_type (int): The way to clip, 0 for 'value', 1 for 'norm'. clip_value (float): Specifies how much to clip. grad (tuple[Tensor]): Gradients. Outputs: tuple[Tensor], clipped gradients. """ if clip_type not in (0, 1): return grad dt = F.dtype(grad) if clip_type == 0: new_grad = ops.clip_by_value(grad, F.cast(F.tuple_to_array((-clip_value,)), dt), F.cast(F.tuple_to_array((clip_value,)), dt)) else: new_grad = nn.ClipByNorm()(grad, F.cast(F.tuple_to_array((clip_value,)), dt)) return new_grad grad_scale = ops.MultitypeFuncGraph("grad_scale") reciprocal = ops.Reciprocal() @grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): """Construct the trainer of Bert.""" return grad * reciprocal(scale) _grad_overflow = ops.MultitypeFuncGraph("_grad_overflow") grad_overflow = ops.FloatStatus() @_grad_overflow.register("Tensor") def _tensor_grad_overflow(grad): """Construct the trainer of Bert.""" return grad_overflow(grad) @MindFormerRegister.register(MindFormerModuleType.WRAPPER, alias="FilipTrainOneStepWithLossScaleWrapper") class FilipTrainOneStepWithLossScaleCell(nn.TrainOneStepWithLossScaleCell): """ FilipTrainOneStepWithLossScaleCell wrapper """ def __init__(self, network, optimizer, scale_update_cell=None): super(FilipTrainOneStepWithLossScaleCell, self).__init__(network, optimizer, scale_update_cell) self.cast = ops.Cast() self.degree = 1 if self.reducer_flag: self.degree = get_group_size() self.grad_reducer = DistributedGradReducer(optimizer.parameters, False, self.degree) self.loss_scale = None self.loss_scaling_manager = scale_update_cell if scale_update_cell: self.loss_scale = Parameter(Tensor(scale_update_cell.get_loss_scale(), dtype=mstype.float32)) def construct(self, image, text, sens=None): """ construct of FilipTrainOneStepWithLossScaleCell """ weights = self.weights loss = self.network(image, text) if sens is None: scaling_sens = self.loss_scale else: scaling_sens = sens status, scaling_sens = self.start_overflow_check(loss, scaling_sens) grads = self.grad(self.network, weights)(image, text, self.cast(scaling_sens, mstype.float32)) # apply grad reducer on grads grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads) grads = self.hyper_map(F.partial(clip_grad, GRADIENT_CLIP_TYPE, GRADIENT_CLIP_VALUE), grads) cond = self.get_overflow_status(status, grads) overflow = cond if sens is None: overflow = self.loss_scaling_manager(self.loss_scale, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) else: print(">>>>overflow") return loss, cond, scaling_sens pengcheng.mind-mindformer/mindformers/models/filip/filip_config.py000066400000000000000000000075501476650141400260360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ FilipConfig class, which consists FilipTextConfig and FilipVisionConfig All configs here are inherited from BaseConfig """ from ..base_config import BaseConfig from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook from ...tools import logger @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipTextConfig(BaseConfig): """Config for filip text module""" def __init__(self, hidden_size=768, vocab_size=21128, max_position_embeddings=32, num_hidden_layers=12, **kwargs): super(FilipTextConfig, self).__init__(hidden_size=hidden_size, vocab_size=vocab_size, max_position_embeddings=max_position_embeddings, num_hidden_layers=num_hidden_layers, **kwargs) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipVisionConfig(BaseConfig): """Config for filip vision module""" def __init__(self, hidden_size=1024, image_size=224, patch_size=14, num_hidden_layers=24, **kwargs): super(FilipVisionConfig, self).__init__(hidden_size=hidden_size, image_size=image_size, patch_size=patch_size, num_hidden_layers=num_hidden_layers, **kwargs) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class FilipConfig(BaseConfig): """Config for filip model""" _support_list = MindFormerBook.get_model_support_list()['filip'] def __init__(self, text_config=None, vision_config=None, projection_dim=256, ratio=64, checkpoint_name_or_path="", dtype="float16", **kwargs): if text_config is None: text_config = FilipTextConfig() logger.info("text_config is None. " "Initializing the FilipTextConfig with default values.") elif isinstance(text_config, FilipTextConfig): pass else: raise TypeError(f"text_config should be a " f"FiLipTextConfig class, but got {type(FilipTextConfig)}") if vision_config is None: vision_config = FilipVisionConfig() logger.info("vision_config is None. " "Initializing the FilipTextConfig with default values.") elif isinstance(vision_config, FilipVisionConfig): pass else: raise TypeError("text_config should be a FilipVisionConfig" f"class, but got {type(FilipVisionConfig)}") super(FilipConfig, self).__init__(text_config=text_config, vision_config=vision_config, projection_dim=projection_dim, ratio=ratio, checkpoint_name_or_path=checkpoint_name_or_path, dtype=dtype, **kwargs) pengcheng.mind-mindformer/mindformers/models/filip/filip_modules.py000066400000000000000000000257211476650141400262410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/openai/CLIP/blob/main/clip/clip.py # ============================================================================ """ Modules of FilipModel, including image encoder model, text encoder model, and token learner """ import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops from mindspore import Parameter, Tensor from mindspore.ops.primitive import constexpr from mindspore.common.initializer import TruncatedNormal, initializer from mindformers.modules.transformer import Transformer from mindformers.modules import TransformerOpParallelConfig, TransformerRecomputeConfig class QuickGELU(nn.Cell): """QuickGELU of Filip""" def __init__(self): super(QuickGELU, self).__init__() self.ratio = 1.702 self.sigmoid = nn.Sigmoid() def construct(self, x): return x * self.sigmoid(self.ratio * x) class VisualTransformer(nn.Cell): """VisualTransformer of Filip""" def __init__(self, config): super(VisualTransformer, self).__init__() self.batch_size = config.batch_size self.single_bs = self.batch_size == 1 self.input_resolution = config.vision_config.image_size self.patch_size = config.vision_config.patch_size self.width = config.vision_config.hidden_size self.layers = config.vision_config.num_hidden_layers self.num_tokens_side = self.input_resolution // self.patch_size self.output_dim = config.projection_dim self.heads = config.vision_config.num_heads if not self.heads: self.heads = self.width // config.ratio else: self.heads = int(self.heads) self.token_learner_flag = config.vision_config.token_learner self.output_dim = config.projection_dim self.conv1 = nn.Conv2d(3, self.width, self.patch_size, self.patch_size) scale = self.width ** -0.5 src_seq_length = (self.input_resolution // self.patch_size) ** 2 + 1 self.class_embedding = Parameter(scale * Tensor(np.random.normal(0, 1, size=(self.width)).astype(np.float32))) self.positional_embedding = Parameter(scale * Tensor( np.random.normal(size=(src_seq_length, self.width)).astype(np.float32))) self.ln_pre = nn.LayerNorm([self.width]) parallel_config = TransformerOpParallelConfig() if config.recompute: recompute_config = TransformerRecomputeConfig(recompute=False) parallel_config = TransformerOpParallelConfig(recompute=recompute_config) self.transformer = Transformer(hidden_size=self.width, batch_size=self.batch_size, num_heads=self.heads, ffn_hidden_size=self.width * 4, encoder_layers=self.layers, attention_dropout_rate=0.0, hidden_dropout_rate=0.0, decoder_layers=0, # hidden_act=QuickGELU, hidden_act='fast_gelu', layernorm_compute_type=ms.float16, softmax_compute_type=ms.float16, param_init_type=ms.float16, src_seq_length=src_seq_length, tgt_seq_length=src_seq_length, parallel_config=parallel_config) self.ln_post = nn.LayerNorm([self.width]) self.proj = Parameter(scale * Tensor(np.random.normal(0, 1, size=(self.width, self.output_dim)).astype(np.float32))) self.cat = ops.Concat(1) self.tile = ops.Tile() self.ones = ops.OnesLike() self.expand_dims = ops.ExpandDims() if self.token_learner_flag: self.token_learner = TokenLearner(in_channels=self.output_dim, num_tokens=config.vision_config.num_tokens, num_groups=config.vision_config.num_token_groups, dropout_rate=config.vision_config.token_learner_dropout) def construct(self, x): """construct of VisualTransformer""" x = self.conv1(x) x = x.reshape(x.shape[0], x.shape[1], -1) x = x.transpose(0, 2, 1) if self.single_bs: class_embedding = self.expand_dims(self.class_embedding, 0) class_embedding = self.expand_dims(class_embedding, 0) else: class_embedding = self.tile(self.class_embedding, (x.shape[0], 1, 1)) x = self.cat((class_embedding, x)) x = x + self.positional_embedding x = self.ln_pre(x) x_shape = x.shape mask = ops.fill(ops.DType()(x), (x_shape[0], x_shape[1], x_shape[1]), 1) x = self.transformer(x, mask)[0] x = self.ln_post(x) x = ops.matmul(x, self.proj) if self.token_learner_flag: cls_token, x = x[:, 0, :], x[:, 1:, :] x = x.reshape(-1, self.num_tokens_side, self.num_tokens_side, self.output_dim) x = self.token_learner(x) cls_token = self.expand_dims(cls_token, 1).astype(x.dtype) x = self.cat((cls_token, x)) return x @constexpr def build_attntion_mask(batch_size, context_length): mask = np.tril(np.full((batch_size, context_length, context_length), 1).astype(np.float32), 0) mask = Tensor(mask) return mask class TextTransformer(nn.Cell): """TextTransformer of Filip""" def __init__(self, config): super(TextTransformer, self).__init__() self.batch_size = config.batch_size self.context_length = config.text_config.max_position_embeddins self.vocab_size = config.text_config.vocab_size self.width = config.text_config.hidden_size self.heads = config.text_config.num_heads if not self.heads: self.heads = self.width // config.ratio else: self.heads = int(self.heads) self.layers = config.text_config.num_hidden_layers self.output_dim = config.projection_dim self.embedding_table = Parameter(initializer(TruncatedNormal(0.02), [self.vocab_size, self.width])) self.gather = ops.Gather() self.reshape = ops.Reshape() self.not_equal = ops.NotEqual() self.cast = ops.Cast() self.positional_embedding = Parameter(initializer(TruncatedNormal(0.01), [self.context_length, self.width])) self.ln_final = nn.LayerNorm([self.width]) self.text_projection = Parameter(Tensor(np.random.normal(0, self.width ** -0.5, size=( self.width, self.output_dim)).astype(np.float32))) parallel_config = TransformerOpParallelConfig() if config.recompute: recompute_config = TransformerRecomputeConfig(recompute=False) parallel_config = TransformerOpParallelConfig(recompute=recompute_config) self.transformer_layer = Transformer(hidden_size=self.width, batch_size=self.batch_size, num_heads=self.heads, ffn_hidden_size=self.width * 4, encoder_layers=self.layers, attention_dropout_rate=0.0, hidden_dropout_rate=0.0, decoder_layers=0, # hidden_act=QuickGELU, hidden_act='fast_gelu', layernorm_compute_type=ms.float16, softmax_compute_type=ms.float16, param_init_type=ms.float16, src_seq_length=self.context_length, tgt_seq_length=self.context_length, parallel_config=parallel_config) def construct(self, text): """construct of TextTransformer""" bsz, ctx_len = text.shape flatten_id = text.flatten() gather_result = self.gather(self.embedding_table, flatten_id, 0) x = self.reshape(gather_result, (bsz, ctx_len, -1)) x = x + self.positional_embedding mask_tensor = build_attntion_mask(bsz, self.context_length) x = self.transformer_layer(x, mask_tensor)[0] x = self.ln_final(x) x = ops.matmul(x, self.text_projection) return x class TokenLearner(nn.Cell): """TokenLearner of Filip""" def __init__(self, in_channels, num_tokens, num_groups=8, dropout_rate=0.): super(TokenLearner, self).__init__() self.in_channels = in_channels self.num_tokens = num_tokens self.num_groups = num_groups self.norm = nn.LayerNorm([self.in_channels]) self.attntion_maps = nn.SequentialCell([ nn.Conv2d(self.in_channels, self.in_channels, kernel_size=1, group=self.num_groups), nn.Conv2d(self.in_channels, self.num_tokens, kernel_size=1) ]) self.feat_conv = nn.Conv2d( self.in_channels, self.in_channels, 1, group=self.num_groups ) self.dropout = nn.Dropout(dropout_rate) self.softmax = nn.Softmax() self.matmul = ops.BatchMatMul() def construct(self, x): """construct of TokenLearner""" feature_shape = x.shape selected = x selected = self.norm(selected) selected = selected.transpose((0, 3, 1, 2)) selected = self.attntion_maps(selected) selected = selected.transpose((0, 2, 3, 1)) selected = selected.reshape((feature_shape[0], feature_shape[1] * feature_shape[2], -1)) selected = selected.transpose((0, 2, 1)) selected = self.softmax(selected) feat = x feat = feat.transpose((0, 3, 1, 2)) feat = self.feat_conv(feat) feat = feat.transpose((0, 2, 3, 1)) feat = feat.reshape((feature_shape[0], feature_shape[1] * feature_shape[2], -1)) outputs = self.matmul(selected, feat) outputs = self.dropout(outputs) return outputs pengcheng.mind-mindformer/mindformers/models/glm/000077500000000000000000000000001476650141400225015ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/glm/__init__.py000066400000000000000000000021211476650141400246060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for glm""" from .chatglm_6b_tokenizer import ChatGLMTokenizer from .glm import (GLMChatModel, GLMChatModelWithLora, GLMForPreTraining, GLMForPreTrainingWithLora) from .glm_config import GLMConfig from .glm_processor import GLMProcessor __all__ = [] __all__.extend(chatglm_6b_tokenizer.__all__) __all__.extend(glm.__all__) __all__.extend(glm_config.__all__) __all__.extend(glm_processor.__all__) pengcheng.mind-mindformer/mindformers/models/glm/attention.py000066400000000000000000000474331476650141400250730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """attention modules.""" import numpy as np from mindspore import Tensor from mindspore import dtype as mstype from mindspore import nn, ops from mindspore.ops import functional as F from mindspore.ops import operations as P from mindformers.modules.transformer import OpParallelConfig from mindformers.modules.layers import Linear from mindformers.version_control import get_dropout default_dpmp_config = OpParallelConfig() def divide(numerator, denominator): """Ensure that numerator is divisible by the denominator and return the division value.""" assert numerator % denominator == 0, f'{numerator} is not divisible by {denominator}' return numerator // denominator class RotaryEmbedding(nn.Cell): """ Rotary embedding layer Args: dim (int): Hidden layer dimension number. base (int, optional): Rotary embedding base. Default: 10000. params_dtype (ms.dtype, optional): Default: mstype.float32. compute_dtype (ms.dtype, optional): Default: mstype.float16. max_seq_len (int): Max sequence length. parallel_config (optional): Operator parallel strategy. Default: `OpParallelConfig()`. """ def __init__(self, dim, base=10000, params_dtype=mstype.float32, compute_dtype=mstype.float16, max_seq_len=512, parallel_config=None): super(RotaryEmbedding, self).__init__() if not parallel_config: parallel_config = default_dpmp_config self.max_seq_len = max_seq_len self.params_dtype = params_dtype self.compute_dtype = compute_dtype inv_freq = 1. / (base ** (np.arange(0, dim, 2).astype(np.float32) / dim)) inv_freq = inv_freq.astype(np.float16) t_range = np.arange(max_seq_len, dtype=inv_freq.dtype) freqs = np.outer(t_range, inv_freq) emb = np.concatenate((freqs, freqs), axis=-1) self.emb = emb # cos_cached: [max_seq_len, 1, dim] self.cos_cached = np.expand_dims(np.cos(emb), 1) self.sin_cached = np.expand_dims(np.sin(emb), 1) self.cos_cached = Tensor(self.cos_cached, params_dtype) self.sin_cached = Tensor(self.sin_cached, params_dtype) self.stride_slice = ops.StridedSlice() self.mul = ops.Mul() self.add = ops.Add() self.squeeze = ops.Squeeze(axis=1).shard(((1, 1, 1),)) self.concat = P.Concat(axis=-1).shard(((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, 1, 1, 1))) def rotate_half(self, x): """rotate half""" x1 = x[:, :, :, : x.shape[-1] // 2] x2 = x[:, :, :, x.shape[-1] // 2:] return self.concat((-x2, x1)) def apply_rotary_pos_emb_index(self, q, k, position_id): """ apply rotary pos emb index Inputs: q, k (Tensor): Querry and key. position_id (Tensor): Used to identify each token's position in the list of tokens. returns: q, k (Tensor): Embedded querry and key layer. """ # position_id: [bs, seq_len] # cos, sin: [max_seq_len, 1, hidden_size] -> [max_seq_len, hidden_size] # -> [seq_len, bs, hidden_size] -> [seq_len, bs, 1, hidden_size] q, k = ops.cast(q, self.compute_dtype), ops.cast(k, self.compute_dtype) cos = ops.expand_dims(self.squeeze(self.cos_cached)[position_id], 2) sin = ops.expand_dims(self.squeeze(self.sin_cached)[position_id], 2) cos, sin = ops.cast(cos, q.dtype), ops.cast(sin, q.dtype) # q, k: [bs, seq_len, size_per_head, hidden_size] q = self.add(self.mul(q, cos), self.mul(self.rotate_half(q), sin)) k = self.add(self.mul(k, cos), self.mul(self.rotate_half(k), sin)) return q, k def construct(self, q, k, position_id): return self.apply_rotary_pos_emb_index(q, k, position_id) def split_tensor_along_last_dim(tensor, num_partitions): """ Split a tensor along its last dimension. Used in construct function. Arguments: tensor (Tensor): Input tensor. num_partitions (int): Number of partitions to split the tensor. """ # Get the size and dimension. last_dim = tensor.ndim - 1 # Split. tensor_list = ops.Split(axis=last_dim, output_num=num_partitions)(tensor) return tensor_list class RotaryEmbeddingFP32SoftmaxSelfAttention(nn.Cell): """ Attention layer with rotary embedding and softmax Args: hidden_size (int): Hidden layer size. batch_size (int): Batch size. num_attention_heads (int): Number of attention heads. parallel_config: Operator parallel strategy. attention_dropout_prob (float, [0, 1.0]): Attention layer dropout probability. output_dropout_prob (float, [0, 1.0]): Output dropout probability. layer_id (int): Layer id. max_seq_len (int): Max sequence length. hidden_size_per_attention_head (ms.dtype, optional): Default: None. position_encoding_2d (ms.dtype, optional): Use 2d position encoding or not. Default: True. bias (bool, optional): Use bias or not. Default: True. params_dtype (ms.dtype, optional): Parameter data type. Default: mstype.float32. softmax_dtype (ms.dtype, optional): Calculate softmax data type. Default: mstype.float32. compute_dtype (ms.dtype, optional): Other compute data type. Default: mstype.float16. use_past (bool, optional): Use infer cache or not. Default: False. """ def __init__( self, hidden_size, batch_size, num_attention_heads, parallel_config, attention_dropout_prob, output_dropout_prob, layer_id, max_seq_len=512, hidden_size_per_attention_head=None, position_encoding_2d=True, bias=True, params_dtype=mstype.float32, softmax_dtype=mstype.float32, compute_dtype=mstype.float16, use_past=False, ): super(RotaryEmbeddingFP32SoftmaxSelfAttention, self).__init__() self.params_dtype = params_dtype self.softmax_dtype = softmax_dtype self.compute_dtype = compute_dtype self.layer_id = layer_id # Per attention head values. self.hidden_size = hidden_size if hidden_size_per_attention_head: self.hidden_size_per_attention_head = hidden_size_per_attention_head else: self.hidden_size_per_attention_head = divide(hidden_size, num_attention_heads) self.num_attention_heads = num_attention_heads self.inner_hidden_size = (num_attention_heads * self.hidden_size_per_attention_head) self.use_past = use_past self.is_first_iteration = True self.position_encoding_2d = position_encoding_2d self.rotary_emb = RotaryEmbedding( self.hidden_size // (self.num_attention_heads * 2) if self.position_encoding_2d else self.hidden_size // self.num_attention_heads, base=10000, params_dtype=params_dtype, # need to use float32 for proper ops.cos/ops.sin compute_dtype=compute_dtype, max_seq_len=max_seq_len, parallel_config=parallel_config, ) # Strided linear layer. self.query_key_value = Linear( hidden_size, 3 * self.inner_hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype ) self.query_key_value.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,)) ) self.attention_dropout = get_dropout(attention_dropout_prob) self.dense = Linear( self.inner_hidden_size, hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype ) self.dense.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, 1), (1,))) self.output_dropout = get_dropout(output_dropout_prob) self.matmul = P.BatchMatMul() self.matmul.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.reduce_max = P.ReduceMax(keep_dims=False) self.split = P.Split(axis=-1, output_num=2) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.concat_query = P.Concat(axis=3).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1)) ) self.concat_key = P.Concat(axis=3).shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1)) ) if self.use_past: # operators used for state reuse seq_range = np.arange(max_seq_len).reshape(1, 1, -1) # [1, 1, config.seq_length] self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) # [bs, 1, config.seq_len] self.seq_length = max_seq_len self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.TensorAdd().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.concat_query = P.Concat(axis=3).shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.concat_key = P.Concat(axis=3).shard(((1, 1, 1, 1), (1, 1, 1, 1))) def attention_fn( self, query_layer, key_layer, value_layer, attention_mask, layer_id, scaling_attention_score=True, ): """ calculate attention function """ # seqlen, batch, head, hidden_size query_key_layer_scaling_coeff = F.cast(layer_id + 1, self.compute_dtype) if scaling_attention_score: query_layer_dtype = query_layer.dtype query_layer = F.cast(query_layer, self.compute_dtype) sqrt_value = F.sqrt(F.cast(self.hidden_size_per_attention_head, self.compute_dtype)) query_layer = query_layer / (sqrt_value * query_key_layer_scaling_coeff) query_layer = F.cast(query_layer, query_layer_dtype) # =================================== # Raw attention scores. [b, np, s, s] # =================================== # [b, np, sq, hn] * [b * np, hn, sk] matmul_result = self.matmul( query_layer.swapaxes(1, 2), F.transpose(key_layer, (0, 2, 3, 1)) ) # record original score dtype attention_scores_dtype = matmul_result.dtype # change view to [b, np, sq, sk] attention_scores = matmul_result attention_mask = F.cast(attention_mask, mstype.bool_) attention_scores = attention_scores.masked_fill(attention_mask, -10000.0) attention_scores = F.cast(attention_scores, self.softmax_dtype) attention_scores = attention_scores * query_key_layer_scaling_coeff # softmax under `softmax_dtype` mode attention_probs = self.softmax(attention_scores) # cast back to original score dtype attention_probs = F.cast(attention_probs, attention_scores_dtype) if self.training: attention_probs = self.attention_dropout(attention_probs) # ========================= # Context layer. [sq, b, hp] # ========================= # value_layer -> context layer. # [sk, b, np, hn] --> [b, np, sq, hn] # matmul: [b * np, sq, sk] x [b * np, sk, hn] --> [b * np, sq, hn] context_layer = self.matmul(attention_probs, value_layer) context_layer = F.cast(context_layer, self.compute_dtype) context_layer = self._merge_heads(context_layer) return context_layer def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (x_shape[0], x_shape[1], x_shape[-2] * x_shape[-1]) x_merge = F.reshape(x, new_shape) return x_merge def attention_forward(self, hidden_states, mask, position_ids, layer_id, key_past=None, value_past=None, batch_valid_length=None): """ attention forward Input: hidden_states (Tensor): Hidden layer states. mask (Tensor): Same as `attention_mask`, used when batching sequences together. position_ids (Tensor): Used to identify each token's position in the list of tokens. layer_id (int): Layer id. key_past (Tensor, optional): Default: None. value_past (Tensor, optional): Default: None. batch_valid_length (bool, optional): Default: None. return: output (Tensor): Attention output. layer_present (Tensor): Layer present, used for infer cache. """ mixed_raw_layer = self.query_key_value(hidden_states) mixed_raw_layer = F.cast(mixed_raw_layer, self.compute_dtype) # [b, seq, (heads * 3 * hidden_size_per_head)] --> [b, seq, heads, 3 * hidden_size_per_head] new_tensor_shape = mixed_raw_layer.shape[:-1] + ( self.num_attention_heads, 3 * self.hidden_size_per_attention_head, ) mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape) # [b, seq, heads, hidden_size_per_head] (query_layer, key_layer, value_layer) = split_tensor_along_last_dim(mixed_raw_layer, 3) if self.position_encoding_2d: q1, q2 = self.split(query_layer) k1, k2 = self.split(key_layer) position_ids, block_position_ids = position_ids[:, 0, :], \ position_ids[:, 1, :] q1, k1 = self.rotary_emb(q1, k1, position_ids) q2, k2 = self.rotary_emb(q2, k2, block_position_ids) query_layer = self.concat_query((q1, q2)) key_layer = self.concat_query((k1, k2)) else: # apply rotary embed on q, k: [bs, seq, num_heads, hidden_size] # position_ids: bs, 2, seq_length query_layer, key_layer = self.rotary_emb(query_layer, key_layer, position_ids) # key and value for current token(s) # [bs, num_heads, hidden_size, seq_len] value_layer = F.transpose(value_layer, (0, 2, 1, 3)) key_present = key_layer value_present = value_layer if self.use_past: # reshape key_present = F.transpose(key_present, (0, 2, 3, 1)) value_present = F.transpose(value_present, (0, 1, 3, 2)) # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.params_dtype) # [bs, 1, seq_len] # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key_present, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value_present, self.expand_dims(valid_length_vector, 2)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index # key_past: [batch_size, num_heads, size_per_head, seq_length] valid_length = batch_valid_length - 1 valid_length = F.reshape(valid_length, (-1, 1, 1)) # [bs, 1, 1] # self.range: [bs, 1, config.seq_len] valid_length_vector = F.cast(self.equal(valid_length, self.range), self.params_dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key_present, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value_present, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) # Concat the previous saved state and current state key_present = self.add(key_past, current_key) # [batch_size, num_heads, size_per_head, seq_length] value_present = self.add(value_past, current_value) # update k v for attention # [batch_size, num_heads, size_per_head, seq_length] -> [bs, num_heads, hidden_size, seq_len] key_layer = F.transpose(key_present, (0, 3, 1, 2)) # [batch_size, num_heads, size_per_head, seq_length] -> [bs, num_heads, seq_len, hidden_size] value_layer = F.transpose(value_present, (0, 1, 3, 2)) layer_present = (key_present, value_present) # [batch_size, num_heads, size_per_head, seq_length] -> [seq_len, bs, num_heads, hidden_size] query_layer = F.cast(query_layer, self.compute_dtype) key_layer = F.cast(key_layer, self.compute_dtype) value_layer = F.cast(value_layer, self.compute_dtype) context_layer = self.attention_fn(query_layer, key_layer, value_layer, mask, layer_id, True) output = self.dense(context_layer) output = F.cast(output, self.params_dtype) if self.training: output = self.output_dropout(output) return output, layer_present def construct(self, hidden_states, mask, position_ids, layer_id, key_past=None, value_past=None, batch_valid_length=None): return self.attention_forward(hidden_states, mask, position_ids, layer_id, key_past, value_past, batch_valid_length) pengcheng.mind-mindformer/mindformers/models/glm/chatglm_6b_tokenizer.py000066400000000000000000000405571476650141400271660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Tokenization classes for ChatGLM.""" import os from typing import List, Optional, Union import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType EncodedInput = List[int] __all__ = ['ChatGLMTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "ice_text.model"} class TextTokenizer: """Base text tokenizer.""" def __init__(self, model_path): self.sp = spm.SentencePieceProcessor() self.sp.Load(model_path) self.num_tokens = self.sp.vocab_size() def convert_token_to_id(self, token): return self.sp.PieceToId(token) def convert_id_to_token(self, idx): return self.sp.IdToPiece(idx) def tokenize(self, text): return self.sp.EncodeAsPieces(text) def decode(self, ids: List[int]): return self.sp.DecodeIds(ids) def __len__(self): return self.num_tokens class SPTokenizer: """Tokenizer process special tokens.""" def __init__( self, vocab_file, num_image_tokens=20000, max_blank_length=80, byte_fallback=True, ): assert vocab_file is not None self.vocab_file = vocab_file self.num_image_tokens = num_image_tokens self.special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "", "", "", "", ""] self.max_blank_length = max_blank_length self.byte_fallback = byte_fallback self.text_tokenizer = TextTokenizer(vocab_file) @staticmethod def get_blank_token(length: int): assert length >= 2 return f"<|blank_{length}|>" @staticmethod def get_tab_token(): return f"<|tab|>" @property def num_text_tokens(self): return self.text_tokenizer.num_tokens @property def num_tokens(self): return self.num_image_tokens + self.num_text_tokens @staticmethod def _encode_whitespaces(text: str, max_len: int = 80): text = text.replace("\t", SPTokenizer.get_tab_token()) for i in range(max_len, 1, -1): text = text.replace(" " * i, SPTokenizer.get_blank_token(i)) return text def _preprocess(self, text: str, linebreak=True, whitespaces=True): if linebreak: text = text.replace("\n", "") if whitespaces: text = self._encode_whitespaces(text, max_len=self.max_blank_length) return text def encode(self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True) -> List[int]: """ Encode text to token id. Args: text: Text to encode. linebreak: Whether to encode newline (\n) in text. whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. add_dummy_prefix: Whether to add dummy blank space in the beginning. """ text = self._preprocess(text, linebreak, whitespaces) if not add_dummy_prefix: text = "" + text tmp = self.text_tokenizer.encode(text) tokens = [x + self.num_image_tokens for x in tmp] return tokens if add_dummy_prefix else tokens[2:] def decode(self, text_ids: List[int]) -> str: """Decode id to text.""" ids = [int(id) - self.num_image_tokens for id in text_ids] ids = [id for id in ids if id >= 0] text = self.text_tokenizer.decode(ids) text = text.replace("", "\n") text = text.replace(SPTokenizer.get_tab_token(), "\t") for i in range(2, self.max_blank_length + 1): text = text.replace(self.get_blank_token(i), " " * i) return text def tokenize(self, text: str, linebreak=True, whitespaces=True, add_dummy_prefix=True) -> List[str]: """ Encode text to id. Args: text: Text to encode. linebreak: Whether to encode newline (\n) in text. whitespaces: Whether to encode multiple whitespaces or tab in text, useful for source code encoding. add_dummy_prefix: Whether to add dummy blank space in the beginning. """ text = self._preprocess(text, linebreak, whitespaces) if not add_dummy_prefix: text = "" + text tokens = self.text_tokenizer.tokenize(text) return tokens if add_dummy_prefix else tokens[2:] def __getitem__(self, x: Union[int, str]): if isinstance(x, int): if x < self.num_image_tokens: return "".format(x) return self.text_tokenizer.convert_id_to_token(x - self.num_image_tokens) if isinstance(x, str): if x.startswith("") and x[7:-1].isdigit(): return int(x[7:-1]) return self.text_tokenizer.convert_token_to_id(x) + self.num_image_tokens raise ValueError("The key should be str or int.") @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class ChatGLMTokenizer(Tokenizer): """ Construct a ChatGLM tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file(str): The vocabulary file path. do_lower_case(bool): Lower input text. Default False. remove_space(str): The merge file path. bos_token(str): The token that represents the begin-of-sentence. Default ''. eos_token(str): The token that represents the end-of-sentence. Default ''. end_token(str): The token that represents the end-of-sentence. Default ''. mask_token(str): The token that represents the special mask. Default '[MASK]', gmask_token(str): The token that represents the special mask. Default '[gMASK]', pad_token(str): The token that represents the pad. Default "". unk_token(str): The token that represents the unknown. Default ''. add_prefix_space(bool): whether to add a whitespace in the front of text. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import AutoTokenizer >>> tokenize = AutoTokenizer.from_pretrained('glm_6b') >>> tokenize("你好") {'input_ids': [5, 74874, 130001, 130004], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} >>> from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer >>> tokenizer = ChatGLMTokenizer('ice_text.model') >>> prompts_list = ["晚上睡不着应该怎么办"] >>> token_id = tokenizer(prompts) >>> input_ids = token_id['input_ids'] >>> print(input_ids) [[74747, 83400, 64213, 66846, 130001, 130004]] >>> response = tokenizer.decode(input_ids) >>> print(response) ['晚上睡不着应该怎么办'] Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "token_type_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['glm'] def __init__( self, vocab_file, do_lower_case=False, remove_space=False, bos_token='', eos_token='', end_token='', mask_token='[MASK]', gmask_token='[gMASK]', pad_token="", unk_token="", num_image_tokens=0, **kwargs ) -> None: super().__init__( do_lower_case=do_lower_case, remove_space=remove_space, bos_token=bos_token, eos_token=eos_token, end_token=end_token, mask_token=mask_token, gmask_token=gmask_token, pad_token=pad_token, unk_token=unk_token, num_image_tokens=num_image_tokens, **kwargs ) self.do_lower_case = do_lower_case self.remove_space = remove_space self.vocab_file = vocab_file self._bos_token = bos_token self._eos_token = eos_token self._end_token = end_token self._mask_token = mask_token self._gmask_token = gmask_token self.sp_tokenizer = SPTokenizer(vocab_file, num_image_tokens=num_image_tokens) self.added_tokens_encoder = {} self.added_tokens_decoder = {} @property def gmask_token_id(self) -> Optional[int]: if self._gmask_token is None: return None return self.convert_tokens_to_ids(self._gmask_token) @property def end_token_id(self) -> Optional[int]: """ `Optional[int]`: Id of the end of context token in the vocabulary. Returns `None` if the token has not been set. """ if self._end_token is None: return None return self.convert_tokens_to_ids(self._end_token) @property def vocab_size(self): """ Returns vocab size """ return self.sp_tokenizer.num_tokens def get_vocab(self): """ Returns vocab as a dict """ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def preprocess_text(self, inputs): """Preprocess text.""" if self.remove_space: outputs = " ".join(inputs.strip().split()) else: outputs = inputs if self.do_lower_case: outputs = outputs.lower() return outputs def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): """ Returns a tokenized string. """ return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Returns a tokenized string. """ text = self.preprocess_text(text) seq = self.sp_tokenizer.tokenize(text) return seq def _decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs): """ Decode id to text. """ # unused in this tokenizer. _, _ = skip_special_tokens, kwargs if isinstance(token_ids, int): token_ids = [token_ids] if self.pad_token_id in token_ids: # remove pad token_ids = list(filter(self.pad_token_id.__ne__, token_ids)) for token_id in token_ids: if token_id not in self.added_tokens_decoder and token_id >= self.vocab_size: raise IndexError(f"The token id {token_id} is out of the size of vocabulary, please check " f"your tokenizer and corresponding vocabulary files.") return self.sp_tokenizer.decode(token_ids) # pylint:disable=arguments-differ def convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ return self._convert_tokens_to_ids(tokens) def _convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]: """ Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. """ if tokens is None: return None if isinstance(tokens, str): return self._convert_token_to_id_with_added_voc(tokens) ids = [] for token in tokens: ids.append(self._convert_token_to_id_with_added_voc(token)) return ids def _convert_token_to_id_with_added_voc(self, token): if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self.sp_tokenizer[token] def _convert_token_to_id(self, token): """copy from _convert_token_to_id_with_added_voc""" if token is None: return None if token in self.added_tokens_encoder: return self.added_tokens_encoder[token] return self.sp_tokenizer[token] def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.sp_tokenizer[index] def build_inputs_with_special_tokens(self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None): """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ gmask_id = self.sp_tokenizer[self._gmask_token] eos_id = self.sp_tokenizer[self._eos_token] token_ids_0 = token_ids_0 + [gmask_id, self.sp_tokenizer[self._bos_token]] if token_ids_1 is not None: token_ids_0 = token_ids_0 + token_ids_1 + [eos_id] return token_ids_0 def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(self.vocab_file, 'rb') as fin: proto_str = fin.read() with open(vocab_file, "wb") as writer: writer.write(proto_str) return vocab_file def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ output = [0] * (len(token_ids_0) + 1 + 1) if token_ids_1 is not None: output += [1] * (len(token_ids_1) + 1) return output pengcheng.mind-mindformer/mindformers/models/glm/convert_weight.py000066400000000000000000000042601476650141400261040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import mindspore as ms import torch as pt def convert_pt_to_ms(pt_pth_path, ms_ckpt_path): """ Convert pytorch mpdel file to MindSpore model file. """ pt_param = pt.load(pt_pth_path) print('parameter convert....') ms_param = [] for k, v in pt_param.items(): if "word_embeddings.weight" in k: k = k.replace("weight", "embedding_table") if "post_attention_layernorm" in k or "input_layernorm" in k or "final_layernorm" in k: k = k.replace("weight", "gamma") k = k.replace("bias", "beta") if "input_layernorm" in k: v = v.half() ms_param.append({"name": k, "data": ms.Tensor(v.numpy())}) print('saving ms ckpt....') ms.save_checkpoint(ms_param, ms_ckpt_path) if __name__ == '__main__': parser = argparse.ArgumentParser(description="ChatGLM6B weight convert script") parser.add_argument("--pt_ckpt_path", type=str, required=True, default="None", help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=False, default="ms_glm_6b.ckpt", help='The output mindspore checkpoint path.') opt = parser.parse_args() convert_pt_to_ms(opt.pt_ckpt_path, opt.ms_ckpt_path) print(f"Convert finished, the output is saved to {opt.ms_ckpt_path}") pengcheng.mind-mindformer/mindformers/models/glm/glm.py000066400000000000000000000557131476650141400236450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GLM model.""" import os import numpy as np import mindspore as ms from mindspore import dtype as mstype from mindspore import nn, ops, Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.common.initializer import initializer from mindspore.common.parameter import Parameter from mindformers.mindformer_book import MindFormerBook from mindformers.modules.transformer import VocabEmbedding, EmbeddingOpParallelConfig, OpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.layers import LayerNorm from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.tuners.lora_adapter import LoraAdapter from mindformers.version_control import get_dropout from .glm_config import GLMConfig from .layers import DeepNormWithGLULayer from ..base_model import BaseModel # Get MS backend: 0 vm 1 GE is_ge = os.getenv('MS_ENABLE_GE') if is_ge == '1': jit_level = "O3" else: jit_level = "O1" default_dpmp_config = OpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() __all__ = ['GLMForPreTraining', 'GLMChatModel', 'GLMForPreTrainingWithLora', 'GLMChatModelWithLora'] def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class ProcessLogits(nn.Cell): r"""Process logits into probability distribution.""" def __init__(self, use_past=False): super(ProcessLogits, self).__init__() self.e = ms.Tensor(np.e) self.gather = P.Gather() self.logsoftmax = P.LogSoftmax() self.reshape = P.Reshape() self.use_past = use_past def construct(self, logits, current_index=None, is_first_iteration=False): logits = logits.reshape(-1, logits.shape[-1]) if self.use_past and not is_first_iteration: logits = logits elif current_index is not None: index = current_index.view(-1,) logits = self.gather(logits, index, 0) outputs = self.logsoftmax(logits) outputs = F.tensor_pow(self.e, outputs) return outputs class GLMModel(nn.Cell): """ The backbone of GLM network Args: config (GLMConfig): The config of network. op_parallel_config (optional): Operator parallel strategy. Default: `OpParallelConfig()`. embed_parallel_config (optional): Operator parallel strategy. Default: `EmbeddingOpParallelConfig()`. """ def __init__(self, config, op_parallel_config=default_dpmp_config, embed_parallel_config=default_embedding_parallel_config): super(GLMModel, self).__init__() # recording parameters self.num_layers = config.num_layers self.hidden_size = config.hidden_size self.num_heads = config.num_heads self.seq_length = config.seq_length self.use_past = config.use_past layernorm = LayerNorm if config.parallel_config: op_parallel_config = config.parallel_config # create embedding parameters self.embedding_dropout = get_dropout(config.embedding_dropout_prob) embed_parallel_config.data_parallel = op_parallel_config.data_parallel embed_parallel_config.model_parallel = op_parallel_config.model_parallel embed_parallel_config.vocab_emb_dp = False self.word_embeddings = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embed_parallel_config) self.word_embeddings.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.matmul = ops.MatMul().shard(((1, 1), (1, embed_parallel_config.model_parallel))) self.transpose = ops.Transpose().shard(((embed_parallel_config.model_parallel, 1),)) def get_layer(layer_id): return DeepNormWithGLULayer( self.num_layers, self.hidden_size, self.num_heads, config.batch_size, config.attention_dropout_rate, config.hidden_dropout_rate, config.layernorm_epsilon, layer_id, max_seq_len=self.seq_length, inner_hidden_size=config.inner_hidden_size, hidden_size_per_attention_head=config.hidden_size_per_attention_head, layernorm_order=config.layernorm_order, layernorm=layernorm, use_bias=True, activation_func=config.activation_func, position_encoding_2d=config.position_encoding_2d, params_dtype=config.param_init_type, layernorm_dtype=config.layernorm_compute_type, softmax_dtype=config.softmax_compute_type, compute_dtype=config.compute_dtype, use_past=self.use_past, parallel_config=op_parallel_config, ) self.layers = nn.CellList() for i in range(config.num_layers): layer = get_layer(i+1) set_parallel_configure_for_layer(layer, layer_id=i, layers=config.num_layers, offset=0, parallel_config=op_parallel_config) self.layers.append(layer) # Final layer norm before output. self.use_final_layernorm = config.use_final_layernorm if config.use_final_layernorm: self.final_layernorm = layernorm(config.hidden_size, eps=config.layernorm_epsilon) self.final_layernorm.shard(((op_parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, position_ids, attention_mask, init_reset=True, batch_valid_length=None): """ Get output logits Inputs: input_ids (Tensor): The tokenized inputs with dtype int32. input_mask (Tensor): The mask indicating whether each position is a valid input. position_ids (Tensor): Used to identify each token's position in the list of tokens. attention_mask (Tensor): Used when batching sequences together. init_reset (bool, optional): Default: True. batch_valid_length (Tensor, optional): Default: None. Returns: logits (Tensor): The output logit of backbone. table (Tensor): The embedding table for the vocabulary. """ if attention_mask is None: attention_mask = ops.ones((1, 1), mstype.int32) hidden_states, table = self.word_embeddings(input_ids) hidden_states = self.embedding_dropout(hidden_states) for i in range(self.num_layers): layer_ret = self.layers[i](hidden_states, attention_mask, position_ids, init_reset, batch_valid_length) if isinstance(layer_ret, tuple): layer_ret = layer_ret[0] hidden_states = layer_ret # Final layer norm. if self.use_final_layernorm: logits = self.final_layernorm(hidden_states) else: logits = hidden_states return logits, table class GLMHead(nn.Cell): r"""Head for GLM to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, param_init_type=mstype.float32, compute_dtype=mstype.float16, embed_parallel_config=None): super(GLMHead, self).__init__() self.param_init_type = param_init_type self.compute_dtype = compute_dtype self.weight = Parameter(initializer("normal", [vocab_size, hidden_size], compute_dtype), name="weight") self.transpose = ops.Transpose().shard(((embed_parallel_config.model_parallel, 1),)) self.matmul = ops.MatMul(transpose_b=True).shard( ((embed_parallel_config.data_parallel, 1), (embed_parallel_config.model_parallel, 1))) def construct(self, state, embedding_table=None): """Get vocab probs""" state = F.reshape(state, (-1, F.shape(state)[-1])) state = ops.cast(state, self.compute_dtype) if embedding_table is None: embedding_table = self.weight embedding_table = self.cast(embedding_table, self.compute_dtype) logits_parallel = self.matmul(state, embedding_table) return logits_parallel @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMForPreTraining(BaseModel): r""" Provide glm training loss or logits through network. Args: config (GLMConfig): The config of GLMModel. """ _support_list = MindFormerBook.get_model_support_list()['glm'] def __init__(self, config: GLMConfig): super(GLMForPreTraining, self).__init__(config) self.config = config self.position_encoding_2d = config.position_encoding_2d self.transformer = GLMModel(config) self.lm_head = GLMHead( hidden_size=config.hidden_size, vocab_size=config.vocab_size, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, embed_parallel_config=config.parallel_config) self.stridedslice = ops.StridedSlice().shard(((1, 1),)) self.loss = CrossEntropyLoss(parallel_config=config.parallel_config, eps_const=3.4e-38) self.gmask = config.gmask_token_id self.bos_token_id = config.bos_token_id self.ones = P.Ones() self.not_equal = P.NotEqual() self.gather = P.Gather() self.use_past = config.use_past self.is_first_iteration = True self.ignore_index = config.ignore_index self.load_checkpoint(config) def get_masks_np(self, input_ids): """get attention mask using numpy.""" batch_size, seq_length = input_ids.shape context_lengths = [list(seq).index(self.config.bos_token_id) for seq in input_ids] attention_mask = np.tril(np.ones((batch_size, seq_length, seq_length))) for i, context_length in enumerate(context_lengths): attention_mask[i, :, :context_length] = 1 attention_mask = np.expand_dims(attention_mask, axis=1) attention_mask = np.array(attention_mask < 0.5, np.bool_) return attention_mask def get_position_ids_np(self, input_ids, mask_positions, use_gmasks=None): """Get position ids from input_ids and mask_positions with numpy""" batch_size, seq_length = input_ids.shape if use_gmasks is None: use_gmasks = [False] * batch_size context_lengths = [list(seq).index(self.config.bos_token_id) for seq in input_ids] if self.config.position_encoding_2d: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): position_ids[i, context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, np.int32), np.arange(seq_length - context_length, dtype=np.int32) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0) position_ids = np.stack((position_ids, block_position_ids), axis=1) else: position_ids = np.repeat(np.expand_dims(np.arange(seq_length), 0), batch_size, axis=0) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids_np(self, input_ids): """Get position ids from input_ids with numpy""" mask, gmask = self.config.mask_token_id, self.config.gmask_token_id seqs = list(input_ids) mask_positions, use_gmasks = [], [] for seq in seqs: mask_token = gmask if gmask in seq else mask use_gmask = mask_token == gmask mask_positions.append(list(seq).index(mask_token)) use_gmasks.append(use_gmask) position_ids = self.get_position_ids_np(input_ids, mask_positions, use_gmasks=None) return position_ids def update_model_kwargs_before_generate(self, input_ids, model_kwargs: dict): """update glm kwargs before generate.""" # for GLM `attention_mask` and `position_ids` generation attention_mask = self.get_masks_np(input_ids) position_ids = self.create_position_ids_np(input_ids) # update in model kwargs model_kwargs["attention_mask"] = attention_mask model_kwargs["position_ids"] = position_ids def prepare_inputs_for_generation(self, input_ids, **kwargs): """prepare inputs for generation.""" attention_mask = kwargs.get("attention_mask", None) position_ids = kwargs.get("position_ids", None) input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "attention_mask": Tensor(attention_mask, mstype.int32), "position_ids": Tensor(position_ids, mstype.int32), "input_position": input_position } def slice_incremental_inputs(self, model_inputs: dict, current_index): """used for non-first iterations, slice the inputs to length 1.""" input_ids = model_inputs.pop("input_ids") attention_mask = model_inputs.pop("attention_mask") position_ids = model_inputs.pop("position_ids") if isinstance(input_ids, Tensor): input_ids = input_ids.asnumpy() if isinstance(attention_mask, Tensor): attention_mask = attention_mask.asnumpy() if isinstance(position_ids, Tensor): position_ids = position_ids.asnumpy() inputs_tmp = [] position_ids_tmp = [] attention_mask_tmp = [] for i, index_value in enumerate(current_index): current_index_tmp = ( int(index_value) - i * input_ids.shape[1] ) # multibatch # use numpy to slice array to avoid complie ascend slice op inputs_tmp.append(input_ids[i][current_index_tmp : current_index_tmp + 1]) position_ids_tmp.append(position_ids[i][..., current_index_tmp:current_index_tmp + 1]) attention_mask_tmp.append(attention_mask[i][:, current_index_tmp:current_index_tmp + 1, :]) inputs_tmp = np.array(inputs_tmp, dtype=np.int32) position_ids_tmp = np.array(position_ids_tmp, dtype=np.int32) attention_mask_tmp = np.array(attention_mask_tmp, dtype=np.int32) model_inputs["input_ids"] = Tensor(inputs_tmp, mstype.int32) model_inputs["position_ids"] = Tensor(position_ids_tmp, mstype.int32) model_inputs["attention_mask"] = Tensor(attention_mask_tmp, mstype.int32) # pylint: disable=W0613 def construct(self, input_ids, labels=None, position_ids=None, attention_mask=None, input_position=None, input_embeds=None, init_reset=True, batch_valid_length=None): """ Extract logits and calculate loss Inputs: input_ids (Tensor): the tokenized inputs with dtype int32. labels (Tensor): the indices of input sequence tokens in the vocabulary. position_ids (Tensor): used to identify each token's position in the list of tokens. attention_mask (Tensor): used when batching sequences together. input_position(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset (bool, optional): Default: True. batch_valid_length(Tensor, optional): Default: None. Returns: Training phase: loss: Training loss. Other phase: logits (Tensor): The output logit of backbone. """ batch_size, seq_length = input_ids.shape if self.training: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length), (1, 1)) else: tokens = input_ids output_states, _ = self.transformer(tokens, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.lm_head(output_states) logits_shape = logits.shape if not self.training: logits = logits.reshape((-1, logits_shape[-1])) # only gather in auto-aggressive generate or first iteration if (not self.use_past or self.is_first_iteration) and input_position is not None: logits = self.gather(logits, input_position, 0) return (logits,) labels = labels.reshape((-1,)) logits = logits.reshape((-1, logits_shape[-1])) input_mask = self.not_equal(labels, self.ignore_index).astype(logits.dtype) input_mask = input_mask.reshape((-1,)) loss = self.loss(logits, labels, input_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMChatModel(GLMForPreTraining): r""" Provide glm chat capability through network. Args: config (GLMConfig): The config of GLMModel. Returns: Tensor, the probability distribution of network loss. """ _support_list = MindFormerBook.get_model_support_list()['glm'] def __init__(self, config: GLMConfig): super(GLMChatModel, self).__init__(config) self.e = ms.Tensor(np.e, dtype=mstype.float32) self.pow = P.Pow() self.topk = P.TopK(sorted=True) self.cumsum = P.CumSum() self.sum = P.ReduceSum(keep_dims=False) self.vocab_size = config.vocab_size self.batch_size = config.batch_size self.frequency_list = ms.Tensor([[0 for _ in range(self.vocab_size)]]) self.post_logits = ProcessLogits(use_past=config.use_past) # seems not supported yet. # self.top_p = config.top_p self.top_p = 1 self.top_k = config.top_k self.repetition_penalty = config.repetition_penalty self.is_first_iteration = False self.is_sample_acceleration = config.is_sample_acceleration def sample(self, log_probs): """Convert the log_probs to probability""" if self.repetition_penalty != 1: log_probs = log_probs - self.frequency_list * self.repetition_penalty - \ (self.frequency_list > 0) * self.repetition_penalty # Process sample in graph to accelerate generate logits = self.pow(self.e, log_probs) # If top_p is less than 1.0, use top_p sampling # seems not supported yet. if self.top_p < 1.0: sorted_logits, index = self.topk(logits, 5000) cumsum_logits = self.cumsum(sorted_logits, 1) top_p_num = self.sum((cumsum_logits < self.top_p).astype(mstype.int32), -1) + 1 top_p_num = int(top_p_num) # Get the corresponding probs and indices probs = sorted_logits[:, :top_p_num] p_args = index[:, :top_p_num] p = probs / self.sum(probs, -1, keepdim=True) # if top_p is set to 1.0, use top_k sampling else: probs, p_args = self.topk(logits, self.top_k) p = probs return p, p_args # pylint:disable=arguments-differ,W0613 def construct(self, input_ids, position_ids=None, attention_mask=None, input_position=None, labels=None, input_embeds=None, init_reset=True, batch_valid_length=None): """Get probs and p_args""" # model forward output_states, _ = self.transformer(input_ids, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.lm_head(output_states) if not self.is_sample_acceleration: return (logits,) # logit post process log_probs = self.post_logits(logits, input_position, self.is_first_iteration) # logit sort and sample probs, p_args = self.sample(log_probs) return probs, p_args @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMForPreTrainingWithLora(GLMForPreTraining): """GLM Model for pretraining with LoRA Args: config (GLMConfig): The config of network. """ def __init__(self, config: GLMConfig = None, **kwargs): _ = kwargs super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) @MindFormerRegister.register(MindFormerModuleType.MODELS) class GLMChatModelWithLora(GLMChatModel): """GLM Model for pretraining with LoRA Args: config (GLMConfig): The config of network. """ def __init__(self, config: GLMConfig = None, **kwargs): _ = kwargs ckpt_cfg = config.checkpoint_name_or_path config.checkpoint_name_or_path = None super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) config.checkpoint_name_or_path = ckpt_cfg self.load_checkpoint(config) pengcheng.mind-mindformer/mindformers/models/glm/glm_config.py000066400000000000000000000124601476650141400251620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GLM config""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig, OpParallelConfig, EmbeddingOpParallelConfig, default_embedding_parallel_config from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook default_dpmp_config = OpParallelConfig() __all__ = ['GLMConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class GLMConfig(BaseConfig): """ GLM config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['glm'] def __init__(self, batch_size: int = 1, vocab_size: int = 130528, hidden_size: int = 4096, num_layers: int = 28, num_heads: int = 32, inner_hidden_size: int = 16384, seq_length: int = 512, embedding_dropout_prob: float = 0.0, attention_dropout_rate: float = 0.0, hidden_dropout_rate: float = 0.0, hidden_size_per_attention_head: bool = None, layernorm_order: str = "post", layernorm_epsilon: float = 1.0e-5, use_final_layernorm: bool = True, op_parallel_config: OpParallelConfig = default_dpmp_config, embed_parallel_config: EmbeddingOpParallelConfig = default_embedding_parallel_config, parallel_config: TransformerOpParallelConfig = default_transformer_config, moe_config: MoEConfig = default_moe_config, use_past: bool = False, activation_func: str = 'GELU', position_encoding_2d: bool = True, param_init_type: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", bos_token_id: int = 130004, eos_token_id: int = 130005, mask_token_id: int = 130000, gmask_token_id: int = 130001, pad_token_id: int = 3, is_enhanced_encoder: bool = True, is_sample_acceleration: bool = False, checkpoint_name_or_path: str = "", max_decode_length: int = 2048, top_k: int = 1, top_p: float = 1, repetition_penalty: float = 1.0, do_sample: bool = True, ignore_index: int = -100, **kwargs): super().__init__(**kwargs) self.batch_size = batch_size self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.hidden_size_per_attention_head = hidden_size_per_attention_head self.layernorm_order = layernorm_order self.layernorm_epsilon = layernorm_epsilon self.use_final_layernorm = use_final_layernorm self.op_parallel_config = op_parallel_config self.embed_parallel_config = embed_parallel_config self.moe_config = moe_config self.use_past = use_past self.parallel_config = parallel_config self.activation_func = activation_func self.inner_hidden_size = inner_hidden_size self.position_encoding_2d = position_encoding_2d self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.mask_token_id = mask_token_id self.gmask_token_id = gmask_token_id self.pad_token_id = pad_token_id self.max_decode_length = max_decode_length self.seq_length = seq_length self.is_enhanced_encoder = is_enhanced_encoder self.is_sample_acceleration = is_sample_acceleration self.checkpoint_name_or_path = checkpoint_name_or_path self.top_k = top_k self.top_p = top_p self.repetition_penalty = repetition_penalty self.do_sample = do_sample self.ignore_index = ignore_index pengcheng.mind-mindformer/mindformers/models/glm/glm_processor.py000066400000000000000000000055251476650141400257400ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ GLMProcessor """ import re from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['GLMProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class GLMProcessor(BaseProcessor): """ GLM processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['glm'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(GLMProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output def process_response(response): """ process the response of chat glm. """ response = response.strip() response = response.replace("[[训练时间]]", "2023年") punkts = [ [",", "，"], ["!", "！"], [":", "："], [";", "；"], [r"\?", "？"], ] for item in punkts: response = re.sub(r"([\u4e00-\u9fff])%s" % item[0], r"\1%s" % item[1], response) response = re.sub(r"%s([\u4e00-\u9fff])" % item[0], r"%s\1" % item[1], response) return response pengcheng.mind-mindformer/mindformers/models/glm/layers.py000066400000000000000000000314751476650141400243640ustar00rootroot00000000000000"""base transformer layer.""" import numpy as np from mindspore import Parameter, Tensor from mindspore import dtype as mstype from mindspore import nn from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import ops from mindformers.modules.layers import LayerNorm, Linear from mindformers.version_control import get_dropout from mindformers.models.glm.attention import RotaryEmbeddingFP32SoftmaxSelfAttention class GEGLU(nn.Cell): """GEGLU activation""" def __init__(self, parallel_config): super(GEGLU, self).__init__() self.split = ops.Split(output_num=2, axis=-1) self.activation_fn = P.GeLU() self.parallel_config = parallel_config def construct(self, x): x1, x2 = self.split(x) return x1 * self.activation_fn(x2) class MLPWithGEGLU(nn.Cell): """MLP layer with GEGLU""" def __init__(self, hidden_size, output_dropout_prob, inner_hidden_size=None, layer_id=None, bias=True, activation_func='GELU', params_dtype=mstype.float32, compute_dtype=mstype.float16, parallel_config=None): super(MLPWithGEGLU, self).__init__() self.layer_id = layer_id # Project to 4h. self.hidden_size = hidden_size if inner_hidden_size is None: inner_hidden_size = 4 * hidden_size self.inner_hidden_size = inner_hidden_size if activation_func == 'GEGLU': self.activation_func = GEGLU(parallel_config) h_to_4h_output_channel = 2 * self.inner_hidden_size elif activation_func == 'GELU': self.activation_func = P.GeLU() self.activation_func.shard(((parallel_config.data_parallel, 1, parallel_config.model_parallel),)) h_to_4h_output_channel = self.inner_hidden_size self.dense_h_to_4h = Linear( self.hidden_size, h_to_4h_output_channel, has_bias=bias, transpose_b=True, param_init_type=params_dtype, compute_dtype=compute_dtype, ) self.dense_h_to_4h.shard( strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Project back to h. self.dense_4h_to_h = Linear( self.inner_hidden_size, self.hidden_size, has_bias=bias, param_init_type=params_dtype, compute_dtype=compute_dtype, ) self.dense_4h_to_h.shard( strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (1, parallel_config.model_parallel)), strategy_bias=((parallel_config.data_parallel, 1), (1,)) ) self.dropout = get_dropout(output_dropout_prob) self.dropout.dropout.shard(((parallel_config.data_parallel, parallel_config.model_parallel),)) def mlp_forward(self, hidden_states): """mlp forward.""" intermediate_parallel = self.dense_h_to_4h(hidden_states) intermediate_parallel = self.activation_func(intermediate_parallel) output = self.dense_4h_to_h(intermediate_parallel) return output def construct(self, hidden_states): output = self.mlp_forward(hidden_states) if self.training: output = self.dropout(output) return output class DeepNormWithGLULayer(nn.Cell): """ GLM base layer Args: num_layers (int): Number of layers. hidden_size (int): Hidden layer size. num_attention_heads (int): Number of attention heads. batch_size (int): Batch size. attention_dropout_prob (float, [0, 1.0]): Attention layer dropout probability. output_dropout_prob (float, [0, 1.0]): Output dropout probability. layernorm_epsilon (float): Layernorm epsilon. layer_id (int): Layer id. max_seq_len (int): Max sequence length. inner_hidden_size (optional): Inner hidden layer size. Default: None. hidden_size_per_attention_head (optional): Default: None. layernorm_order (str, optional): Which order to use layernorm. Default: 'pre'. layernorm (optional): Layernorm function. Default: LayerNorm. use_bias (bool, optional): Use bias or not. Default: True. activation_func (str, optional, 'GEGLU'/'GELU'): Choose activation function. Default: 'GEGLU'. position_encoding_2d (bool, optional): Use 2d position encoding or not. Default: True. params_dtype (ms.dtype, optional): Parameter data type. Default: mstype.float32. layernorm_dtype (ms.dtype, optional): Calculate layernorm data type. Default: mstype.float32. softmax_dtype (ms.dtype, optional): Calculate softmax data type. Default: mstype.float32. compute_dtype (ms.dtype, optional): Other compute data type. Default: mstype.float16. parallel_config (optional): Operator parallel strategy, Default: None. use_past (bool, optional): Use infer cache or not. Default: False. """ def __init__(self, num_layers, hidden_size, num_attention_heads, batch_size, attention_dropout_prob, output_dropout_prob, layernorm_epsilon, layer_id, max_seq_len=512, inner_hidden_size=None, hidden_size_per_attention_head=None, layernorm_order='pre', layernorm=LayerNorm, use_bias=True, activation_func='GEGLU', position_encoding_2d=True, params_dtype=mstype.float32, layernorm_dtype=mstype.float32, softmax_dtype=mstype.float32, compute_dtype=mstype.float16, parallel_config=None, use_past=False): super(DeepNormWithGLULayer, self).__init__() # Set output layer initialization if not provided. self.num_layers = num_layers self.hidden_size = hidden_size self.layer_id = layer_id if inner_hidden_size is None: inner_hidden_size = 4 * hidden_size * 2 // 3 self.inner_hidden_size = inner_hidden_size self.position_encoding_2d = position_encoding_2d self.layernorm_order = layernorm_order self.use_past = use_past self.params_dtype = params_dtype self.layernorm_dtype = layernorm_dtype self.softmax_dtype = softmax_dtype self.compute_dtype = compute_dtype # Layernorm on the input data. self.input_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.input_layernorm.shard(((parallel_config.data_parallel, 1, 1),)) self.input_layernorm.set_comm_fusion(parallel_config.gradient_aggregation_group) # Self attention. self.attention = RotaryEmbeddingFP32SoftmaxSelfAttention( hidden_size, batch_size, num_attention_heads, parallel_config, attention_dropout_prob, output_dropout_prob, layer_id, hidden_size_per_attention_head=hidden_size_per_attention_head, position_encoding_2d=self.position_encoding_2d, bias=use_bias, params_dtype=params_dtype, softmax_dtype=softmax_dtype, compute_dtype=compute_dtype, max_seq_len=max_seq_len, use_past=use_past, ) # Layernorm on the input data. self.post_attention_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.post_attention_layernorm.shard(((parallel_config.data_parallel, 1, 1),)) self.post_attention_layernorm.set_comm_fusion(parallel_config.gradient_aggregation_group) if self.layernorm_order == 'sandwich': self.third_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) self.fourth_layernorm = layernorm(hidden_size, param_init_type=layernorm_dtype, eps=layernorm_epsilon) # MLP self.mlp = MLPWithGEGLU( hidden_size, output_dropout_prob, inner_hidden_size=inner_hidden_size, bias=use_bias, layer_id=layer_id, activation_func=activation_func, params_dtype=params_dtype, parallel_config=parallel_config ) self.key_past = None self.value_past = None if use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_attention_heads) self.kv_shape = (batch_size, num_attention_heads, size_per_head, max_seq_len) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.kv_shape), self.params_dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.kv_shape), self.params_dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), ())) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.mul = P.Mul() self.mul.shard(((parallel_config.data_parallel, 1, 1), ())) self.mul_4 = P.Mul() self.mul_4.shard(((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel,))) def layer_forward(self, hidden_states, mask, position_ids, init_reset=True, batch_valid_length=None): """ hidden_states: [seq_len, batch, hidden_size] with (1, dp, 1) mask: [(1, 1), seq_len, seq_len] Inputs: hidden_states (Tensor): Hidden layer output. mask (Tensor): Used when batching sequences together. position_ids (Tensor): Used to identify each token's position in the list of tokens. init_reset (bool, optional): Default: True. batch_valid_length (Tensor, optional): Default: None. Return: output (Tensor): Layer output. """ # Layer norm at the beginning of the transformer layer. attention_input = self.input_layernorm(hidden_states) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul_4(self.key_past, F.cast(init_reset, self.params_dtype))) value_reset = self.assign(self.value_past, self.mul_4(self.value_past, F.cast(init_reset, self.params_dtype))) # add dependency for desired execution order attention_input = F.depend(attention_input, key_reset) attention_input = F.depend(attention_input, value_reset) attention_output, layer_present = self.attention(attention_input, mask, position_ids, self.layer_id, self.key_past, self.value_past, batch_valid_length) # Residual connection. alpha = (2 * self.num_layers) ** 0.5 hidden_states = self.mul(attention_input, alpha) + attention_output mlp_input = self.post_attention_layernorm(hidden_states) # MLP. mlp_output = self.mlp(mlp_input) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_output = F.depend(mlp_output, value_update) mlp_output = F.depend(mlp_output, key_update) # Second residual connection. output = mlp_input * alpha + mlp_output return output def construct(self, hidden_states, mask, position_ids, init_reset=True, batch_valid_length=None): ''' hidden_states: [batch, seq_len, hidden_size] mask: [(1, 1), seq_len, seq_len] ''' return self.layer_forward(hidden_states, mask, position_ids, init_reset, batch_valid_length) pengcheng.mind-mindformer/mindformers/models/glm2/000077500000000000000000000000001476650141400225635ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/glm2/__init__.py000066400000000000000000000016051476650141400246760ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for ChatGLM2""" from .glm2_config import * from .glm2_tokenizer import * from .glm2 import * __all__ = [] __all__.extend(glm2_config.__all__) __all__.extend(glm2_tokenizer.__all__) __all__.extend(glm2.__all__) pengcheng.mind-mindformer/mindformers/models/glm2/glm2.py000066400000000000000000000237231476650141400240050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 model.""" import mindspore.ops as ops import mindspore.common.dtype as mstype import mindspore.ops.operations as P from mindspore import Tensor, nn import numpy as np from mindformers.mindformer_book import MindFormerBook from mindformers.modules import VocabEmbedding, EmbeddingOpParallelConfig from mindformers.modules.layers import Linear from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from mindformers.core.loss import CrossEntropyLoss from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.tuners.lora_adapter import LoraAdapter from mindformers.version_control import get_tril from ..base_model import BaseModel from .glm2_config import ChatGLM2Config from .glm2_modules import precompute_rotary_emb_cache from .glm2_transformer import ChatGLM2Transformer __all__ = ['ChatGLM2ForConditionalGeneration', 'ChatGLM2WithLora', 'ChatGLM2Model'] class ChatGLM2Model(nn.Cell): r""" The backbone of ChatGLM2 network Args: config (GLMConfig): The config of network. """ def __init__(self, config: ChatGLM2Config, **kwargs): super(ChatGLM2Model, self).__init__(config, **kwargs) self.num_layers = config.num_layers self.multi_query_group_num = config.multi_query_group_num self.kv_channels = config.kv_channels self.seq_length = config.seq_length self.compute_dtype = config.compute_dtype self.use_past = config.use_past self.is_first_iteration = True # vocab embedding embed_parallel_config = EmbeddingOpParallelConfig() embed_parallel_config.data_parallel = config.parallel_config.data_parallel embed_parallel_config.model_parallel = config.parallel_config.model_parallel embed_parallel_config.vocab_emb_dp = False self.embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embed_parallel_config) self.embedding.set_comm_fusion(config.parallel_config.gradient_aggregation_group) # rotary embedding rotary_dim = ( config.hidden_size // config.num_attention_heads if config.kv_channels is None else config.kv_channels ) self.rotary_pos_emb = precompute_rotary_emb_cache( seq_len=self.seq_length, dim=rotary_dim // 2 ) self.rotary_pos_emb = Tensor(self.rotary_pos_emb, config.compute_dtype) self.encoder = ChatGLM2Transformer(config) self.output_layer = Linear(config.hidden_size, config.vocab_size, has_bias=False, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype) self.output_layer.shard(strategy_matmul=((config.parallel_config.data_parallel, 1), (config.parallel_config.model_parallel, 1))) self.output_layer.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.tril = get_tril() self.ones = P.Ones() self.less = P.Less() self.gather = P.Gather() self.expand_dims = P.ExpandDims() self.reshape = P.Reshape() self.mul = P.Mul() self.tile = ops.Tile() low_triangle = np.tril(np.ones((1, self.seq_length, self.seq_length))) self.low_triangle = Tensor(low_triangle, mstype.int32) def get_masks(self, batch_size, padding_mask=None, input_position=None): """Get attention mask.""" # [1, seq_length, seq_length] -> [batch_size, seq_length, seq_length] low_triangle = self.tile(self.low_triangle, (batch_size, 1, 1)) if padding_mask is not None: low_triangle = self.mul(low_triangle, self.expand_dims(padding_mask, 1)) if self.use_past and padding_mask is not None: low_triangle -= self.expand_dims(padding_mask, -1) - 1 attention_mask = self.less(low_triangle, 0.5) if self.use_past and not self.is_first_iteration: # [bs, 1, seq_len] for incremental infer attention_mask = self.gather(attention_mask.view(-1, self.seq_length), input_position, 0) # [bs, 1, seq_len, seq_len] for normal, [bs, 1, 1, seq_len] for incremental infer attention_mask = self.reshape(attention_mask, (batch_size, 1, -1, self.seq_length)) return attention_mask def construct(self, input_ids, position_ids=None, attention_mask=None, full_attention_mask=None, inputs_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): """ChatGLM2 model.""" _ = position_ids batch_size, _ = input_ids.shape if inputs_embeds is None: inputs_embeds, _ = self.embedding(input_ids) # (bs, seq_len, hs) if full_attention_mask is None: # (bs, 1, seq_len, seq_len) full_attention_mask = self.get_masks(batch_size, attention_mask, input_position) # (sen length, kv_channels // 4, 2) rotary_pos_emb = self.rotary_pos_emb if self.use_past and not self.is_first_iteration and batch_valid_length is not None: # only take [bs, 1, kv_channels // 4, 2] batch_gather_position = batch_valid_length.view(-1, 1) - 1 # [bs, seq_len=1] rotary_pos_emb = self.gather(rotary_pos_emb, batch_gather_position, 0) # Run encoder. hidden_states = self.encoder( inputs_embeds, full_attention_mask, rotary_pos_emb=rotary_pos_emb, init_reset=init_reset, batch_valid_length=batch_valid_length) return hidden_states @MindFormerRegister.register(MindFormerModuleType.MODELS) class ChatGLM2ForConditionalGeneration(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GLMConfig): The config of ChatGLM2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['glm2'] def __init__(self, config: ChatGLM2Config, **kwargs): super(ChatGLM2ForConditionalGeneration, self).__init__(config, **kwargs) self.max_seq_len = config.max_length self.transformer = ChatGLM2Model(config=config) self.cast = P.Cast() self.gather = P.Gather() self.is_first_iteration = True self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) self.gmask = config.gmask_token_id self.bos_token_id = config.bos_token_id self.use_past = config.use_past self.is_first_iteration = True self.not_equal = P.NotEqual() self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): """prepare inputs for generation.""" input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "input_position": input_position } def construct(self, input_ids=None, labels=None, position_ids=None, attention_mask=None, inputs_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): """ChatGLM2 for conditional generation model.""" # input_ids: (bs, seq_len) # position_ids: (bs, seq_len) # attention_mask: (bs, seq_len) hidden_states = self.transformer( input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask, inputs_embeds=inputs_embeds, input_position=input_position, init_reset=init_reset, batch_valid_length=batch_valid_length, ) lm_logits = self.transformer.output_layer(hidden_states) if labels is not None: logits = self.cast(lm_logits, mstype.float32) logits_shape = logits.shape labels = labels.reshape((-1,)) logits = logits.reshape((-1, logits_shape[-1])) input_mask = self.not_equal(labels, -100).astype(logits.dtype) input_mask = input_mask.reshape((-1,)) loss = self.loss(logits, labels, input_mask) return loss lm_logits = lm_logits.reshape((-1, lm_logits.shape[-1])) if (not self.use_past or self.is_first_iteration) and input_position is not None: lm_logits = self.gather(lm_logits, input_position, 0) return (lm_logits,) @MindFormerRegister.register(MindFormerModuleType.MODELS) class ChatGLM2WithLora(ChatGLM2ForConditionalGeneration): """ChatGLM2 Model for pretraining with LoRA Args: config (ChatGLM2Config): The config of network. """ def __init__(self, config: ChatGLM2Config = None, **kwargs): _ = kwargs ckpt_cfg = config.checkpoint_name_or_path config.checkpoint_name_or_path = None super().__init__(config) # get Pet tuning model. config.pet_config.reg_rules = r'.*query_key_value*' self.transformer = LoraAdapter.get_pet_model(self.transformer, config.pet_config) if ckpt_cfg: config.checkpoint_name_or_path = ckpt_cfg self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) pengcheng.mind-mindformer/mindformers/models/glm2/glm2_config.py000066400000000000000000000105171476650141400253270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 config""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.modules.transformer.transformer import default_transformer_config from ..base_config import BaseConfig from ..utils import convert_mstype from ...mindformer_book import MindFormerBook __all__ = ['ChatGLM2Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ChatGLM2Config(BaseConfig): """ ChatGLM2 model config class. """ _support_list = MindFormerBook.get_config_support_list()['glm2'] def __init__(self, batch_size=1, # only for incremental infer num_layers=28, padded_vocab_size=65024, hidden_size=4096, ffn_hidden_size=13696, kv_channels=128, num_attention_heads=32, seq_length=2048, hidden_dropout=0.0, attention_dropout=0.0, layernorm_epsilon=1e-5, rmsnorm=True, apply_residual_connection_post_layernorm=False, post_layer_norm=True, add_bias_linear=False, add_qkv_bias=True, bias_dropout_fusion=True, multi_query_attention=True, multi_query_group_num=2, apply_query_key_layer_scaling=True, attention_softmax_in_fp32=True, fp32_residual_connection=False, quantization_bit=0, pre_seq_len=None, prefix_projection=False, param_init_type: str = "float16", compute_dtype: str = "float16", layernorm_compute_type: str = "float32", use_past=False, eos_token_id=2, pad_token_id=0, repetition_penalty=1.0, parallel_config=default_transformer_config, **kwargs): super().__init__(**kwargs) self.batch_size = batch_size self.num_layers = num_layers self.vocab_size = padded_vocab_size self.padded_vocab_size = padded_vocab_size self.hidden_size = hidden_size self.ffn_hidden_size = ffn_hidden_size self.kv_channels = kv_channels self.num_attention_heads = num_attention_heads self.seq_length = seq_length self.hidden_dropout = hidden_dropout self.attention_dropout = attention_dropout self.layernorm_epsilon = layernorm_epsilon self.rmsnorm = rmsnorm self.apply_residual_connection_post_layernorm = apply_residual_connection_post_layernorm self.post_layer_norm = post_layer_norm self.add_bias_linear = add_bias_linear self.add_qkv_bias = add_qkv_bias self.bias_dropout_fusion = bias_dropout_fusion self.multi_query_attention = multi_query_attention self.multi_query_group_num = multi_query_group_num self.apply_query_key_layer_scaling = apply_query_key_layer_scaling self.attention_softmax_in_fp32 = attention_softmax_in_fp32 self.fp32_residual_connection = fp32_residual_connection self.quantization_bit = quantization_bit self.pre_seq_len = pre_seq_len self.prefix_projection = prefix_projection self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.use_past = use_past self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.repetition_penalty = repetition_penalty self.parallel_config = parallel_config pengcheng.mind-mindformer/mindformers/models/glm2/glm2_modules.py000066400000000000000000000143621476650141400255340ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Modules.""" import numpy as np import mindspore.common.dtype as mstype import mindspore.ops.operations as P from mindspore import nn, Parameter, Tensor from mindspore.common.initializer import initializer from mindformers.modules.layers import Linear from .glm2_config import ChatGLM2Config def precompute_rotary_emb_cache(seq_len: int, dim: int, dtype=np.float32, base: int = 10000): """pre compute rotary emb cache.""" # $\Theta = {\theta_i = 10000^{\frac{2(i-1)}{d}}, i \in [1, 2, ..., \frac{d}{2}]}$ theta = 1.0 / (base ** (np.arange(0, dim, 2, dtype=dtype) / dim)) # Create position indexes `[0, 1, ..., seq_len - 1]` seq_idx = np.arange(seq_len, dtype=dtype) # Calculate the product of position index and $\theta_i$ idx_theta = np.outer(seq_idx, theta).astype(np.float32) cache = np.stack((np.cos(idx_theta), np.sin(idx_theta)), axis=-1).astype(dtype) return cache class ChatGLM2RMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, param_init_type=mstype.float32): super(ChatGLM2RMSNorm, self).__init__() self.eps = Tensor(float(eps), dtype=param_init_type) self.weight = Parameter(initializer('ones', (dim,), dtype=param_init_type)) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() def _norm(self, x): # shard:(dp, 1, 1) norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" output = self._norm(x) output = self.mul2(output, self.weight) return output def shard(self, strategy): """Parallel strategy configuratiuon interface.""" self.square.shard(strategy) self.mean.shard(strategy) self.rsqrt.shard(strategy) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], strategy[0])) self.mul2.shard((strategy[0], (1,))) class ChatGLM2SiLU(nn.Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super(ChatGLM2SiLU, self).__init__() self.sigmoid = P.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) class ChatGLM2SwiGLU(nn.Cell): """SwiGLU activation function.""" def __init__(self): super(ChatGLM2SwiGLU, self).__init__() self.split = P.Split(axis=-1, output_num=2) self.silu = ChatGLM2SiLU() self.mul = P.Mul() def construct(self, x): x0, x1 = self.split(x) return self.mul(self.silu(x0), x1) def shard(self, strategy): self.split.shard(strategy) self.silu.shard(strategy) self.mul.shard((strategy[0], strategy[0])) class ChatGLM2MLP(nn.Cell): """MLP. MLP will take the input with h hidden state, project it to 4*h hidden dimension, perform nonlinear transformation, and project the state back into h hidden dimension. """ def __init__(self, config: ChatGLM2Config): super(ChatGLM2MLP, self).__init__() self.add_bias = config.add_bias_linear self.dense_h_to_4h = Linear( config.hidden_size, config.ffn_hidden_size * 2, has_bias=self.add_bias, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, ) self.dense_h_to_4h.shard( strategy_matmul=((config.parallel_config.data_parallel, 1), (config.parallel_config.model_parallel, 1)), strategy_bias=((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (config.parallel_config.model_parallel,))) self.activation_func = ChatGLM2SwiGLU() # shard need to be checked. self.activation_func.shard(((config.parallel_config.data_parallel, 1, 1),)) # Project back to h. self.dense_4h_to_h = Linear( config.ffn_hidden_size, config.hidden_size, has_bias=self.add_bias, param_init_type=config.param_init_type, compute_dtype=config.compute_dtype, ) self.dense_4h_to_h.shard( strategy_matmul=((config.parallel_config.data_parallel, config.parallel_config.model_parallel), (1, config.parallel_config.model_parallel)), strategy_bias=((config.parallel_config.data_parallel, 1), (1,))) def construct(self, hidden_states): # [bs, seq_len, 4 * hidden_size] intermediate_parallel = self.dense_h_to_4h(hidden_states) intermediate_parallel = self.activation_func(intermediate_parallel) # [bs, seq_len, hidden_size] output = self.dense_4h_to_h(intermediate_parallel) return output pengcheng.mind-mindformer/mindformers/models/glm2/glm2_tokenizer.py000066400000000000000000000251021476650141400260700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Tokenizer.""" import os from typing import List, Optional, Union from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer from mindformers.tools.register import MindFormerModuleType, MindFormerRegister from sentencepiece import SentencePieceProcessor __all__ = ['ChatGLM2Tokenizer'] class SPTokenizer: """Tokenizer process special tokens.""" def __init__(self, model_path: str): # reload tokenizer assert os.path.isfile(model_path), model_path self.sp_model = SentencePieceProcessor(model_file=model_path) # BOS / EOS token IDs self.n_words: int = self.sp_model.vocab_size() self.bos_id: int = self.sp_model.bos_id() self.eos_id: int = self.sp_model.eos_id() self.pad_id: int = self.sp_model.unk_id() assert self.sp_model.vocab_size() == self.sp_model.get_piece_size() special_tokens = ["[MASK]", "[gMASK]", "[sMASK]", "sop", "eop"] self.special_tokens = {} self.index_special_tokens = {} for token in special_tokens: self.special_tokens[token] = self.n_words self.index_special_tokens[self.n_words] = token self.n_words += 1 def tokenize(self, s: str, pair=None, add_special_tokens=True, **kwargs): # unused in this tokenizer. _, _, _ = pair, add_special_tokens, kwargs return self.sp_model.EncodeAsPieces(s) def encode(self, s: str, bos: bool = False, eos: bool = False) -> List[int]: assert isinstance(s, str) t = self.sp_model.encode(s) if bos: t = [self.bos_id] + t if eos: t = t + [self.eos_id] return t def decode(self, t: List[int], skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs) -> str: # unused in this tokenizer. _, _, _ = skip_special_tokens, clean_up_tokenization_spaces, kwargs return self.sp_model.decode(t) def decode_tokens(self, tokens: List[str]) -> str: text = self.sp_model.DecodePieces(tokens) return text def convert_token_to_id(self, token): """ Converts a token (str) in an id using the vocab. """ if token in self.special_tokens: return self.special_tokens[token] return self.sp_model.PieceToId(token) def convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index in self.index_special_tokens or index in [self.eos_id, self.bos_id, self.pad_id] or index < 0: return "" return self.sp_model.IdToPiece(index) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class ChatGLM2Tokenizer(Tokenizer): """ Construct a ChatGLM2 tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file(str): The vocabulary file path. padding_side(str): ["left", "right"] Lower input text. Default False. **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import AutoTokenizer >>> tokenize = AutoTokenizer.from_pretrained('glm2_6b') >>> tokenize("你好") {'input_ids': [64790, 64792, 36474, 54591], 'attention_mask': [1, 1, 1, 1]} >>> from mindformers import ChatGLM2Tokenizer >>> tokenizer = ChatGLM2Tokenizer('tokenizer.model') >>> prompts = ["晚上睡不着应该怎么办"] >>> token_id = tokenizer(prompts) >>> input_ids = token_id['input_ids'] >>> print(input_ids) [[64790, 64792, 30910, 32820, 54266, 31876, 35153]] >>> response = tokenizer.decode(input_ids) >>> print(response) ['晚上睡不着应该怎么办'] Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = {"vocab_file": "tokenizer.model"} model_input_names = ["input_ids", "attention_mask", "position_ids"] _support_list = MindFormerBook.get_tokenizer_support_list()['glm2'] def __init__(self, vocab_file, bos_token='', eos_token='', end_token='', mask_token='[MASK]', gmask_token='[gMASK]', pad_token='', unk_token='', **kwargs): super().__init__(bos_token=bos_token, eos_token=eos_token, end_token=end_token, mask_token=mask_token, gmask_token=gmask_token, pad_token=pad_token, unk_token=unk_token, **kwargs) self.name = "ChatGLM2Tokenizer" self.vocab_file = vocab_file self.tokenizer = SPTokenizer(vocab_file) self.special_tokens = { "": self.tokenizer.bos_id, "": self.tokenizer.eos_id, "": self.tokenizer.pad_id } self._bos_token = bos_token self._eos_token = eos_token self._end_token = end_token self._mask_token = mask_token self._gmask_token = gmask_token def get_command(self, token): if token in self.special_tokens: return self.special_tokens[token] assert token in self.tokenizer.special_tokens, f"{token} is not a special token for {self.name}" return self.tokenizer.special_tokens[token] @property def pad_token_id(self): return self.get_command("") @property def eos_token_id(self): return self.get_command("") @property def vocab_size(self): return self.tokenizer.n_words def get_vocab(self): """ Returns vocab as a dict """ vocab = {self._convert_id_to_token(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def build_prompt(self, query, history=None): if history is None: history = [] prompt = "" for i, (old_query, response) in enumerate(history): prompt += "[Round {}]\n\n问：{}\n\n答：{}\n\n".format( i + 1, old_query, response) prompt += "[Round {}]\n\n问：{}\n\n答：".format(len(history) + 1, query) return prompt def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): """ Returns a tokenized string. """ return self._tokenize(text) def _tokenize(self, text, **kwargs): return self.tokenizer.tokenize(text) def _convert_token_to_id(self, token): return self.tokenizer.convert_token_to_id(token) def convert_tokens_to_ids(self, tokens: List[str]) -> list: """ Converts tokens to ids using the vocab. """ ids = [] for token in tokens: ids.append(self.tokenizer.convert_token_to_id(token)) return ids def _decode(self, token_ids: Union[int, List[int]], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = None, **kwargs) -> str: _, _ = skip_special_tokens, clean_up_tokenization_spaces tokens = [] for token_id in token_ids: tokens.append(self._convert_id_to_token(token_id)) return self.convert_tokens_to_string(tokens) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index >= self.vocab_size: raise IndexError( f"The token id {index} is out of the size of vocabulary, please check your tokenizer " f"and corresponding vocabulary files.") return self.tokenizer.convert_id_to_token(index) def convert_tokens_to_string(self, tokens: List[str]) -> str: return self.tokenizer.decode_tokens(tokens) def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. filename_prefix (`str`, *optional*): An optional prefix to add to the named of the saved files. Returns: `Tuple(str)`: Paths to the files saved. """ if os.path.isdir(save_directory): vocab_file = os.path.join( save_directory, self.vocab_files_names["vocab_file"] ) else: vocab_file = save_directory with open(self.vocab_file, 'rb') as fin: proto_str = fin.read() with open(vocab_file, "wb") as writer: writer.write(proto_str) return (vocab_file,) def get_prefix_tokens(self): prefix_tokens = [self.get_command("[gMASK]"), self.get_command("sop")] return prefix_tokens def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BERT sequence has the following format: - single sequence: `[CLS] X [SEP]` - pair of sequences: `[CLS] A [SEP] B [SEP]` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ prefix_tokens = self.get_prefix_tokens() token_ids_0 = prefix_tokens + token_ids_0 if token_ids_1 is not None: token_ids_0 = token_ids_0 + token_ids_1 + [self.get_command("")] return token_ids_0 pengcheng.mind-mindformer/mindformers/models/glm2/glm2_transformer.py000066400000000000000000000654221476650141400264310ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ChatGLM2 Transformer.""" import math import numpy as np import mindspore.ops.functional as F import mindspore.ops.operations as P from mindspore import Parameter, Tensor, nn, ops from mindspore import dtype as mstype from mindformers.modules import LayerNorm from mindformers.modules.layers import Linear from mindformers.version_control import get_dropout from .glm2_config import ChatGLM2Config from .glm2_modules import ChatGLM2MLP, ChatGLM2RMSNorm class CoreAttention(nn.Cell): """ChatGLM2 core attention.""" def __init__(self, config: ChatGLM2Config, layer_number): super(CoreAttention, self).__init__() self.apply_query_key_layer_scaling = config.apply_query_key_layer_scaling self.attention_softmax_in_fp32 = config.attention_softmax_in_fp32 if self.apply_query_key_layer_scaling: self.attention_softmax_in_fp32 = True self.layer_number = max(1, layer_number) projection_size = config.kv_channels * config.num_attention_heads self.hidden_size_per_partition = projection_size self.hidden_size_per_attention_head = projection_size // config.num_attention_heads self.num_attention_heads_per_partition = config.num_attention_heads coeff = None self.norm_factor = math.sqrt(self.hidden_size_per_attention_head) if self.apply_query_key_layer_scaling: coeff = self.layer_number self.norm_factor *= coeff self.coeff = coeff # Strided linear layer. self.attention_dropout = get_dropout(config.attention_dropout) parallel_config = config.parallel_config self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.batch_matmul_q_k.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.softmax = nn.Softmax(axis=-1) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.compute_dtype = config.compute_dtype def construct(self, query_layer, key_layer, value_layer, attention_mask): """ calculate attention function """ # query_layer [b, heads, seq, hidden_size_per_head] # key_layer [b, heads, seq, hidden_size_per_head] # value_layer # [bs, heads, seq_len, hidden_size_per_head] # seqlen, batch, head, hidden_size if self.apply_query_key_layer_scaling: query_layer = query_layer / self.norm_factor # =================================== # Raw attention scores. [b, heads, s, s] # =================================== # [b, heads, seq_q, hidden_size_per_head] × [b, heads, seq_k, hidden_size_per_head]^T -> [b, heads, seq_q, seq_k] matmul_result = self.batch_matmul_q_k(query_layer, key_layer) # record original score dtype attention_scores_dtype = matmul_result.dtype # [b, heads, seq, seq] attention_scores = matmul_result if self.coeff is not None: attention_scores = attention_scores * self.coeff if attention_mask is None and attention_scores.shape[2] == attention_scores.shape[3]: attention_mask = ops.ones((attention_scores.shape[0], 1, attention_scores.shape[2], attention_scores.shape[3]), dtype=mstype.bool_) attention_mask.tril() attention_mask = ~attention_mask if attention_mask is not None: attention_scores = attention_scores.masked_fill(attention_mask, -10000) if self.attention_softmax_in_fp32: attention_scores = F.cast(attention_scores, mstype.float32) attention_probs = self.softmax(attention_scores) attention_probs = F.cast(attention_probs, attention_scores_dtype) attention_probs = self.attention_dropout(attention_probs) # [bs, heads, seq_q, seq_k] x [bs, heads, seq_v, hidden_size_per_head] -> [b, heads, seq_q, hidden_size_per_head] context_layer = self.batch_matmul(attention_probs, value_layer) context_layer = F.cast(context_layer, self.compute_dtype) context_layer = self._merge_heads(context_layer) return context_layer def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose(x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = x.shape new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge class ChatGLM2SelfAttention(nn.Cell): """ChatGLM2 self-attention.""" def __init__(self, config: ChatGLM2Config, layer_number): super(ChatGLM2SelfAttention, self).__init__() self.layer_number = max(1, layer_number) self.projection_size = config.kv_channels * config.num_attention_heads # Per attention head and per partition values. self.hidden_size_per_attention_head = self.projection_size // config.num_attention_heads self.num_attention_heads_per_partition = config.num_attention_heads self.params_dtype = config.param_init_type self.compute_dtype = config.compute_dtype self.batch_size = config.batch_size self.seq_length = config.seq_length self.multi_query_attention = config.multi_query_attention self.qkv_hidden_size = 3 * self.projection_size if self.multi_query_attention: self.num_multi_query_groups_per_partition = config.multi_query_group_num self.qkv_hidden_size = ( self.projection_size + 2 * self.hidden_size_per_attention_head * config.multi_query_group_num) parallel_config = config.parallel_config self.query_key_value = Linear(config.hidden_size, self.qkv_hidden_size, has_bias=config.add_bias_linear or config.add_qkv_bias, param_init_type=self.params_dtype, compute_dtype=self.compute_dtype) self.query_key_value.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,)) ) self.core_attention = CoreAttention(config, self.layer_number) self.dense = Linear(self.projection_size, config.hidden_size, has_bias=config.add_bias_linear, param_init_type=self.params_dtype, compute_dtype=self.compute_dtype) self.dense.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, 1), (1,))) self.reshape = P.Reshape() self.stack = P.Stack(axis=-1) self.gather = P.Gather() self.index_0 = Tensor(0) self.index_1 = Tensor(1) self.mul = P.Mul() self.sub = P.Sub() self.add = P.Add() self.concat = P.Concat(axis=-1) self.split_3 = P.Split(axis=-1, output_num=3) self.transpose = P.Transpose() self.use_past = config.use_past if self.use_past: self.is_first_iteration = True seq_range = np.arange(self.seq_length).reshape(1, 1, -1) self.range = Tensor( np.tile(seq_range, (self.batch_size, 1, 1)), mstype.int32) self.less = P.Less() self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.tile = P.Tile().shard(((1, 1, 1, 1),)) def apply_rotary_pos_emb(self, x: Tensor, rope_cache: Tensor) -> Tensor: """apply rotary position embedding to q,k.""" # x: [b, heads, seq, hidden_size_per_head] bs, num_heads, seq_len, _ = x.shape # 1, 32，4, 128 # rope_cache: first (seq_len, kv_channels//4, 2), other (1, kv_channels//4, 2) rot_dim = rope_cache.shape[-2] * 2 # kv_channels // 2 x, x_pass = x[..., :rot_dim], x[..., rot_dim:] # ms not support variable sizes # truncate to support variable sizes # rope_cache = rope_cache[:sq] # [bs, nh, sq, kv_channels//4, 2] xshaped = self.reshape(x, (bs, num_heads, seq_len, rot_dim // 2, 2)) # [bs, 1, sq, kv_channels//4, 2] rope_cache = self.reshape(rope_cache, (-1, 1, seq_len, xshaped.shape[3], 2)) xshaped_0, xshaped_1 = ops.split(xshaped, 1, -1) rope_cache_0, rope_cache_1 = ops.split(rope_cache, 1, -1) x_out1 = self.sub(self.mul(xshaped_0, rope_cache_0), self.mul(xshaped_1, rope_cache_1)) x_out2 = self.add(self.mul(xshaped_1, rope_cache_0), self.mul(xshaped_0, rope_cache_1)) x_out = self.stack((x_out1, x_out2)) x_out = self.reshape(x_out, (x_out.shape[0], x_out.shape[1], x_out.shape[2], -1)) # [bs, sq, nh, hidden_size_per_head] return self.concat((x_out, x_pass)) def construct(self, hidden_states, attention_mask, rotary_pos_emb, key_past=None, value_past=None, batch_valid_length=None): """Forward process of self-attention.""" # hidden_states: [bs, seq_len, hidden_size] # attention_mask (bs, 1, seq_len, seq_len) # rotary_pos_emb: first: (sen length, kv_channels//4, 2)， after:(1, kv_channels//4, 2] # [bs, seq_len, qkv_hidden_size] mixed_raw_layer = self.query_key_value(hidden_states) # not compatible with ms below 2.0 if self.multi_query_attention: (query_layer, key_layer, value_layer) = mixed_raw_layer.split( [self.num_attention_heads_per_partition * self.hidden_size_per_attention_head, self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head, self.num_multi_query_groups_per_partition * self.hidden_size_per_attention_head, ], axis=-1, ) # [bs, seq_len, nh, hidden_size_per_attention_head] -> [bs, nh, seq_len, hidden_size_per_attention_head] query_layer = query_layer.view( query_layer.shape[:-1] + (self.num_attention_heads_per_partition, self.hidden_size_per_attention_head) ) query_layer = self.transpose(query_layer, (0, 2, 1, 3)) # [bs, seq_len, multi_query_groups, hidden_size_per_attention_head] # -> [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] key_layer = key_layer.view( key_layer.shape[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head) ) key_layer = self.transpose(key_layer, (0, 2, 1, 3)) # [bs, seq_len, multi_query_groups, hidden_size_per_attention_head] # -> [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] value_layer = value_layer.view( value_layer.shape[:-1] + (self.num_multi_query_groups_per_partition, self.hidden_size_per_attention_head) ) value_layer = self.transpose(value_layer, (0, 2, 1, 3)) else: # [b, seq, (heads * 3 * hidden_size_per_head)] --> [b, seq, heads, 3 * hidden_size_per_head] new_tensor_shape = mixed_raw_layer.shape[:-1] + ( self.num_attention_heads_per_partition, 3 * self.hidden_size_per_attention_head, ) mixed_raw_layer = mixed_raw_layer.view(*new_tensor_shape) # [b, seq, heads, hidden_size_per_head] (query_layer, key_layer, value_layer) = self.split_3(mixed_raw_layer) # [b, seq, heads, hidden_size_per_head] -> [bs, num_heads, seq_len, hidden_size_per_head] query_layer = self.transpose(query_layer, (0, 2, 1, 3)) key_layer = self.transpose(key_layer, (0, 2, 1, 3)) value_layer = self.transpose(value_layer, (0, 2, 1, 3)) # rotary_pos_emb: first: (seq_length, kv_channels//4, 2)， after:(1, kv_channels//4, 2) if rotary_pos_emb is not None: # [b, heads, seq, hidden_size_per_head] query_layer = self.apply_rotary_pos_emb(query_layer, rotary_pos_emb) # [bs, multi_query_groups, seq_len, hidden_size_per_attention_head] key_layer = self.apply_rotary_pos_emb(key_layer, rotary_pos_emb) # key and value for current token(s) # [bs, heads, seq_len, hidden_size_per_head] key_present = key_layer value_present = value_layer if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.params_dtype) # [bs, 1, seq_len] # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key_present, self.expand_dims(valid_length_vector, 3)) value_present = self.mul1(value_present, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, multi_query_groups, 1, size_per_head) # the shape of value is (bs, multi_query_groups, 1, size_per_head) else: # Get the current token position index # key_past: [batch_size, multi_query_groups, seq_length, size_per_head] valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) # [bs, 1, 1] # self.range: [bs, 1, config.seq_len] valid_length_vector = F.cast(self.equal(valid_length, self.range), self.params_dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key_present, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) current_value = self.mul1(self.tile(value_present, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state # [batch_size, multi_query_groups, seq_length, size_per_head] key_present = self.add(key_past, current_key) value_present = self.add(value_past, current_value) # update k v for attention # [batch_size, multi_query_groups, seq_length, size_per_head] key_layer = key_present # [batch_size, multi_query_groups, seq_length, size_per_head] value_layer = value_present layer_present = (key_present, value_present) # tile k,v to num_heads if self.multi_query_attention: bs, heads, _, hs_ph = key_layer.shape key_layer = key_layer.view((bs, heads, -1)) key_layer = key_layer.unsqueeze(2) key_layer = key_layer.tile( (1, 1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, 1)) # [b, heads, seq, hidden_size_per_head] key_layer = key_layer.view((bs, self.num_attention_heads_per_partition, -1, hs_ph)) value_layer = value_layer.view((bs, heads, -1)) value_layer = value_layer.unsqueeze(2) value_layer = value_layer.tile( (1, 1, self.num_attention_heads_per_partition // self.num_multi_query_groups_per_partition, 1)) # [b, heads, seq, hidden_size_per_head] value_layer = value_layer.view((bs, self.num_attention_heads_per_partition, -1, hs_ph)) context_layer = self.core_attention(query_layer, key_layer, value_layer, attention_mask) # # ================= # # Output. [bs, seq_len, hidden_size] # # ================= output = self.dense(context_layer) return output, layer_present class ChatGLM2Block(nn.Cell): """A single transformer layer. Transformer layer takes input with size [s, b, h] and returns an output of the same size. """ def __init__(self, config: ChatGLM2Config, layer_number: int): super(ChatGLM2Block, self).__init__() self.layer_number = layer_number self.apply_residual_connection_post_layernorm = config.apply_residual_connection_post_layernorm self.fp32_residual_connection = config.fp32_residual_connection self.use_past = config.use_past self.params_dtype = config.param_init_type self.layernorm_dtype = config.layernorm_compute_type self.compute_dtype = config.compute_dtype self.seq_length = config.seq_length layer_norm_func = ChatGLM2RMSNorm if config.rmsnorm else LayerNorm # Layernorm on the input data. self.input_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=self.layernorm_dtype) self.input_layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) # Self attention. self.self_attention = ChatGLM2SelfAttention(config, layer_number) self.hidden_dropout = config.hidden_dropout # Layernorm on the attention output self.post_attention_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=self.layernorm_dtype) # self.post_attention_layernorm.shard() # MLP self.mlp = ChatGLM2MLP(config) self.dropout = get_dropout(self.hidden_dropout) self.dropout.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.cast = P.Cast() self.key_past = None self.value_past = None if self.use_past: size_per_head = config.hidden_size // config.num_attention_heads kv_num_partition = config.num_attention_heads if config.multi_query_attention: kv_num_partition = config.multi_query_group_num kv_shape = (config.batch_size, kv_num_partition, self.seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter( Tensor(np.zeros(shape=kv_shape), self.params_dtype), name="key_past") self.value_past = Parameter( Tensor(np.zeros(shape=kv_shape), self.params_dtype), name="value_past") self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, hidden_states, attention_mask, rotary_pos_emb, init_reset=True, batch_valid_length=None): """Forward process of the transformer layer.""" # hidden_states: [bs, seq_len, hidden_size] # attention_mask first: (bs, 1, seq_len, seq_len), after: (bs, 1, 1, seq_len) # rotary_pos_emb: first: (seq_len, kv_channels//4, 2)， after: (1, kv_channels//4, 2) # Layer norm at the beginning of the transformer layer. hidden_states = self.cast(hidden_states, self.layernorm_dtype) layernorm_output = self.input_layernorm(hidden_states) # fp32 -> fp16 layernorm_output = self.cast(layernorm_output, self.compute_dtype) key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul( self.key_past, F.cast(init_reset, self.params_dtype))) value_reset = self.assign(self.value_past, self.mul( self.value_past, F.cast(init_reset, self.params_dtype))) # add dependency for desired execution order layernorm_output = F.depend(layernorm_output, key_reset) layernorm_output = F.depend(layernorm_output, value_reset) # Self attention. attention_output, layer_present = self.self_attention( layernorm_output, attention_mask, rotary_pos_emb, self.key_past, self.value_past, batch_valid_length ) # Residual connection. # False on default. if self.apply_residual_connection_post_layernorm: residual = layernorm_output else: residual = hidden_states layernorm_input = self.dropout(attention_output) layernorm_input = residual + layernorm_input # Layer norm post the self attention. layernorm_output = self.post_attention_layernorm(layernorm_input) layernorm_output = self.cast(layernorm_output, self.compute_dtype) # MLP. mlp_output = self.mlp(layernorm_output) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_output = F.depend(mlp_output, value_update) mlp_output = F.depend(mlp_output, key_update) # Second residual connection. # False on default. if self.apply_residual_connection_post_layernorm: residual = layernorm_output else: residual = layernorm_input output = self.dropout(mlp_output) output = residual + output return output def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class ChatGLM2Transformer(nn.Cell): """Transformer class.""" def __init__(self, config: ChatGLM2Config): super(ChatGLM2Transformer, self).__init__() self.post_layer_norm = config.post_layer_norm self.compute_dtype = config.compute_dtype # Number of layers. self.num_layers = config.num_layers # Transformer layers. def build_layer(layer_number): return ChatGLM2Block(config, layer_number) self.layers = nn.CellList() for i in range(self.num_layers): layer = build_layer(i + 1) set_parallel_configure_for_layer(layer, layer_id=i, layers=self.num_layers, offset=0, parallel_config=config.parallel_config) self.layers.append(layer) if self.post_layer_norm: layer_norm_func = ChatGLM2RMSNorm if config.rmsnorm else LayerNorm # Final layer norm before output. self.final_layernorm = layer_norm_func(config.hidden_size, eps=config.layernorm_epsilon, param_init_type=config.layernorm_compute_type) # self.final_layernorm.shard() self.final_layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) def construct(self, hidden_states, attention_mask, rotary_pos_emb, init_reset=True, batch_valid_length=None): """Forward process of the transformer.""" # hidden_states (bs, seq_len, hs) # attention_mask (bs, 1, seq_len, seq_len) # rotary_pos_emb: first: (sen length, kv_channels//2, 2)， after:[1, kv_channels // 2, 2] for layer in self.layers: hidden_states = layer( hidden_states, attention_mask, rotary_pos_emb, init_reset=init_reset, batch_valid_length=batch_valid_length, ) # Final layer norm. if self.post_layer_norm: hidden_states = self.final_layernorm(hidden_states) hidden_states = self.cast(hidden_states, self.compute_dtype) return hidden_states pengcheng.mind-mindformer/mindformers/models/gpt2/000077500000000000000000000000001476650141400225765ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/gpt2/.ipynb_checkpoints/000077500000000000000000000000001476650141400263675ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/gpt2/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000020121476650141400326200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for gpt""" from .gpt2_config import GPT2Config from .gpt2 import GPT2Model, GPT2LMHeadModel, GPT2ForSequenceClassification from .gpt2_tokenizer import GPT2Tokenizer from .gpt2_processor import GPT2Processor __all__ = ['GPT2Config', 'GPT2Model', 'GPT2LMHeadModel', 'GPT2ForSequenceClassification', 'GPT2Tokenizer', 'GPT2Processor'] pengcheng.mind-mindformer/mindformers/models/gpt2/.ipynb_checkpoints/gpt2-checkpoint.py000066400000000000000000000627671476650141400317640ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.layers import LayerNorm, Dropout, Linear from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.transformer import AttentionMask, VocabEmbedding from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.tools.logger import logger #from mindformers.pet import LoraAdapter, PetAdapter from .gpt2_config import GPT2Config from .gpt_modules import GPTTransformerDecoderLayer __all__ = ['GPT2LMHeadModel', 'GPT2WithLora', 'GPT2ForSequenceClassification', 'GPT2Model', 'GPTHead'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2LMHeadModel(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): config = config if config is not None else GPT2Config() super(GPT2LMHeadModel, self).__init__(config, auto_prefix=True) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(config) self.head = GPTHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.add = P.Add().shard(((parallel_config.data_parallel, 1), ())) self.tile = P.Tile() self.is_first_iteration = True self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, attention_mask=None, input_embeds=None, labels=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. labels (Tensor): the labels of inputs with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. input_position (Tensor): the position ids of inputs (at incremental reasoning mode) which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. position_ids (Tensor): the position ids of inputs which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. inputs_embeds (Tensor): the embedding of inputs with data type float32/float16, Tensor of shape :math:`(batch, seq\_length, hidden_size) init_reset (bool): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. batch_valid_length (Tensor): Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id) batch_size, seq_length = input_ids.shape attention_mask = self.cast(attention_mask, mstype.float32) loss_mask = attention_mask if not self.training: tokens = input_ids else: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.stridedslice(attention_mask, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.get_attention_mask(attention_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.backbone(tokens, attention_mask, input_position, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: logits = self.reshape(logits, (batch_size, seq_length, -1)) # makes cast effective to avoid allgather issue in Mindspore1.10 loss_mask = self.add(loss_mask, 1) return logits, tokens, loss_mask loss_mask = self.stridedslice(loss_mask, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.reshape(labels, (-1,)) loss_mask = self.reshape(loss_mask, (-1,)) loss = self.loss(logits, labels, loss_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2ForSequenceClassification(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): self.config = config if config is not None else GPT2Config() super(GPT2ForSequenceClassification, self).__init__(self.config, auto_prefix=True) self.eos_token = self.config.eos_token self.seq_length = self.config.seq_length self.num_labels = self.config.num_labels self.hidden_size = self.config.hidden_size parallel_config = self.config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.get_attention_mask = AttentionMask(seq_length=self.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(self.config) self.score = Linear(in_channels=self.hidden_size, out_channels=self.num_labels, has_bias=False, compute_dtype=self.config.compute_dtype) self.score.shard(strategy_matmul=((dp, 1), (1, 1))) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) vocab_size = self.config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.reduce_sum = P.ReduceSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp,), (dp,))) self.sub = P.Sub().shard(((1,), ())) self.gather = P.Gather().shard(((1, 1), (1,))) # pylint: disable=W0613 def construct(self, input_ids, attention_mask, labels=None, input_embeds=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for GPT2 Text Classification Model Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position. labels (Tensor): the labels of corresponding input sequences. input_embeds(Tensor): Reserved param, not used. input_position(Tensor): Reserved param, not used. position_ids(Tensor): Reserved param, not used. init_reset(Tensor): Reserved param, not used. batch_valid_length(Tensor): Reserved param, not used. Returns: (logits, labels) (Tensor, Tensor) or logits (Tensor) or loss (mstype.float32): in train mode, return loss; in eval mode, return logits and loss; in predict mode, return logits. """ attention_mask = self.cast(attention_mask, mstype.float32) attention_mask_lower_triangle = self.get_attention_mask(attention_mask) output_states, _ = self.backbone(input_ids, attention_mask_lower_triangle) output_states = self.reshape(output_states, (-1, self.hidden_size)) logits = self.score(output_states) # get the last logit of each sequence last_indices = self.sub(self.reduce_sum(attention_mask, -1), 1) batch_size = attention_mask.shape[0] indices_increments = Tensor(np.arange(0, self.seq_length * batch_size, self.seq_length)) last_indices = self.cast(self.add(last_indices, indices_increments), mstype.int32) pooled_logits = self.gather(logits, last_indices, 0) if labels is not None: labels = self.reshape(labels, (-1,)) if self.training: output = self.loss(pooled_logits, labels) else: if labels is not None: output = (pooled_logits, labels) else: output = pooled_logits return output class GPTEmbeddingLayer(nn.Cell): r"""The Embedding Layer of GPT-2 network.""" def __init__(self, config: GPT2Config = None): super(GPTEmbeddingLayer, self).__init__() parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer('normal', [vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=parallel_config.embedding_dp_mp_config) new_parallel_config = copy.deepcopy(parallel_config) new_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer('normal', [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=new_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dropout = Dropout(1 - config.embedding_dropout_prob) self.dropout.shard(((parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, input_position): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) position_embedding, _ = self.position_embedding(input_position) embedding = self.add(word_embedding, position_embedding) embedding = self.dropout(embedding) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class GPT2Model(nn.Cell): """ The backbone of GPT network Args: config(GPT2Config): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 input_mask: the mask indicating whether each position is a valid input Returns: output_state: Tensor, the output logit of backbone present_layer: Tensor, the current feature map embedding_table: Tensor, the embedding table for the vocabulary """ def __init__(self, config): super(GPT2Model, self).__init__() self.config = config self.embedding = GPTEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.blocks = nn.CellList() for i in range(config.num_layers): block = GPTTransformerDecoderLayer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config.dp_mp_config, moe_config=moe_config, use_past=config.use_past ) set_parallel_configure_for_layer( block, layer_id=i, layers=config.num_layers, offset=0, parallel_config=config.parallel_config) self.blocks.append(block) self.cast = P.Cast() self.tile = P.Tile().shard(((config.parallel_config.data_parallel,),)) self.dtype = mstype.float16 self.num_layers = config.num_layers self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.is_first_iteration = True def construct(self, input_ids, attention_mask, input_position=None, init_reset=True, batch_valid_length=None): """GPT model""" batch_size, seq_length = F.shape(input_ids) # When input_position is None, the phase is train mode. When the phase is train mode and is the first iteration # of incremental reasoning, it goes into the following logic. if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when the phase is not train and incremental reasoning is not the first iteration, it goes into the # following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) input_embedding, embedding_table = self.embedding(input_ids, input_position) hidden_states = self.cast(input_embedding, self.dtype) hidden_shape = F.shape(hidden_states) hidden_states = F.reshape(hidden_states, (-1, hidden_shape[-1])) for i in range(self.num_layers): hidden_states = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) output_state = self.layernorm(hidden_states) return output_state, embedding_table class GPTHead(nn.Cell): r"""Head for GPT to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type=mstype.float16, parallel_config=None): super().__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of GPTHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPTHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embedding_table): logits = self.matmul(self.cast(state, self.dtype), self.cast(embedding_table, self.dtype)) return logits @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): """ GPT2LMHeadModel with LoRA parameter-efficient tuning algorithm Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config: GPT2Config = None): checkpoint_name_or_path = config.pop("checkpoint_name_or_path") super().__init__(config) config.pet_config.reg_rules = r'.*dense1.*|.*dense3.*' self.backbone = LoraAdapter.get_pet_model(self.backbone, config.pet_config) config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) class CrossEntropyCalculationWithMask(nn.Cell): """ Cross Entropy loss """ def __init__(self, is_training=None, num_labels=None): super(CrossEntropyCalculationWithMask, self).__init__() self.onehot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum() self.reduce_mean = P.ReduceMean() self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg() self.cast = P.Cast() self.is_training = is_training self.num_labels = num_labels self.log_softmax = P.LogSoftmax(axis=-1) def construct(self, logits, label_ids, input_mask=None): """ Calculate loss Args: logits (Tensor): the probability distribution over vocabulary. label_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_mask (Tensor): input sentences padding mask, where 0 indicates padding position. Returns: return_value (Tensor, mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ # logits [batch * (seq_length-1), vocab_size] label_ids [batch, seq_length-1] logits = self.log_softmax(logits) if self.is_training: label_ids = self.reshape(label_ids, self.last_idx) # label_ids [batch * (seq_length-1)] one_hot_labels = self.onehot(label_ids, self.num_labels, self.on_value, self.off_value) # [batch * (seq_length-1), vocab_size] per_example_loss = self.neg( self.reduce_sum(one_hot_labels * logits, self.last_idx)) # [batch * (seq_length-1)] # for PPL calculation in evaluation if input_mask is not None: input_mask = self.cast(self.reshape(input_mask, self.last_idx), mstype.float32) # [batch * (seq_length-1)] valid_loss_sum = self.reduce_sum(input_mask * per_example_loss, ()) valid_element_sum = self.reduce_sum(input_mask, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) loss = valid_loss_sum / valid_element_sum else: loss = self.reduce_mean(per_example_loss, self.last_idx) # a number return_value = self.cast(loss, mstype.float32) else: return_value = logits * 1.0 # [batch * (seq_length-1), vocab_size] return return_value pengcheng.mind-mindformer/mindformers/models/gpt2/__init__.py000066400000000000000000000020121476650141400247020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for gpt""" from .gpt2_config import GPT2Config from .gpt2 import GPT2Model, GPT2LMHeadModel, GPT2ForSequenceClassification from .gpt2_tokenizer import GPT2Tokenizer from .gpt2_processor import GPT2Processor __all__ = ['GPT2Config', 'GPT2Model', 'GPT2LMHeadModel', 'GPT2ForSequenceClassification', 'GPT2Tokenizer', 'GPT2Processor'] pengcheng.mind-mindformer/mindformers/models/gpt2/convert_weight.py000066400000000000000000000160541476650141400262050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def print_dict(input_dict): """ Print the keys and values of input dict Args: input_dict(dict): input dict with key and value. Returns: None """ for k, v in input_dict.items(): print(f"Param: {k} with shape {v}") def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: value = weight_dict[tgt].numpy() # split the attention layer for q, k, v if 'c_attn.weight' in tgt: print("tgt:", tgt) value = np.transpose(value, [1, 0]) print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list def split_torch_attention(state): s = list(state.keys()) for name in s: if name.endswith('attn.c_attn.weight') or name.endswith('attn.c_attn.bias'): value = state.pop(name) q, k, v = np.split(value.numpy(), 3, -1) state[name + '.q'] = torch.tensor(q, dtype=value.dtype) state[name + '.k'] = torch.tensor(k) state[name + '.v'] = torch.tensor(v) if __name__ == '__main__': parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--layers', type=int, default=12, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') print_dict(state_dict) ms_name = [ "backbone.blocks.{}.layernorm1.gamma", "backbone.blocks.{}.layernorm1.beta", "backbone.blocks.{}.layernorm2.gamma", "backbone.blocks.{}.layernorm2.beta", "backbone.blocks.{}.attention.projection.weight", "backbone.blocks.{}.attention.projection.bias", "backbone.blocks.{}.attention.dense1.weight", "backbone.blocks.{}.attention.dense1.bias", "backbone.blocks.{}.attention.dense2.weight", "backbone.blocks.{}.attention.dense2.bias", "backbone.blocks.{}.attention.dense3.weight", "backbone.blocks.{}.attention.dense3.bias", "backbone.blocks.{}.output.mapping.weight", "backbone.blocks.{}.output.mapping.bias", "backbone.blocks.{}.output.projection.weight", "backbone.blocks.{}.output.projection.bias", ] torch_name = [ "h.{}.ln_1.weight", "h.{}.ln_1.bias", "h.{}.ln_2.weight", "h.{}.ln_2.bias", "h.{}.attn.c_proj.weight", "h.{}.attn.c_proj.bias", "h.{}.attn.c_attn.weight.q", "h.{}.attn.c_attn.bias.q", "h.{}.attn.c_attn.weight.k", "h.{}.attn.c_attn.bias.k", "h.{}.attn.c_attn.weight.v", "h.{}.attn.c_attn.bias.v", "h.{}.mlp.c_fc.weight", "h.{}.mlp.c_fc.bias", "h.{}.mlp.c_proj.weight", "h.{}.mlp.c_proj.bias" ] addition_mindspore = [ "backbone.layernorm.gamma", "backbone.layernorm.beta", "backbone.embedding.word_embedding.embedding_table", "backbone.embedding.position_embedding.embedding_table", ] addition_torch = [ "ln_f.weight", "ln_f.bias", "wte.weight", "wpe.weight", ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) split_torch_attention(state_dict) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-mindformer/mindformers/models/gpt2/gpt2.py000066400000000000000000000627671476650141400240460ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.layers import LayerNorm, Dropout, Linear from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.transformer import AttentionMask, VocabEmbedding from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.mindformer_book import MindFormerBook from mindformers.tools.logger import logger #from mindformers.pet import LoraAdapter, PetAdapter from .gpt2_config import GPT2Config from .gpt_modules import GPTTransformerDecoderLayer __all__ = ['GPT2LMHeadModel', 'GPT2WithLora', 'GPT2ForSequenceClassification', 'GPT2Model', 'GPTHead'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2LMHeadModel(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): config = config if config is not None else GPT2Config() super(GPT2LMHeadModel, self).__init__(config, auto_prefix=True) self.eos_token_id = self.config.eos_token_id parallel_config = self.config.parallel_config self.stridedslice = P.StridedSlice().shard(((parallel_config.data_parallel, 1),)) self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(config) self.head = GPTHead(hidden_size=config.hidden_size, vocab_size=config.vocab_size, parallel_config=self.config.parallel_config) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) mp = config.parallel_config.model_parallel vocab_size = config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.add = P.Add().shard(((parallel_config.data_parallel, 1), ())) self.tile = P.Tile() self.is_first_iteration = True self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, attention_mask=None, input_embeds=None, labels=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for Language Modeling Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. labels (Tensor): the labels of inputs with data type int64/int32, Tensor of shape :math:`(batch, seq\_length)`. input_position (Tensor): the position ids of inputs (at incremental reasoning mode) which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. position_ids (Tensor): the position ids of inputs which is an increasing sequence with data type int64/int32, Tensor :math:`(bacth, seq\_length)`. inputs_embeds (Tensor): the embedding of inputs with data type float32/float16, Tensor of shape :math:`(batch, seq\_length, hidden_size) init_reset (bool): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. batch_valid_length (Tensor): Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Returns: logits (Tensor) or loss (mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ if attention_mask is None: attention_mask = self.not_equal(input_ids, self.eos_token_id) batch_size, seq_length = input_ids.shape attention_mask = self.cast(attention_mask, mstype.float32) loss_mask = attention_mask if not self.training: tokens = input_ids else: tokens = self.stridedslice(input_ids, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.stridedslice(attention_mask, (0, 0), (batch_size, seq_length - 1), (1, 1)) attention_mask = self.get_attention_mask(attention_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) # [batch_size, seq_length, vocab_size] output_states, embedding_table = self.backbone(tokens, attention_mask, input_position, init_reset, batch_valid_length) logits = self.head(output_states, embedding_table) if not self.training: logits = self.reshape(logits, (batch_size, seq_length, -1)) # makes cast effective to avoid allgather issue in Mindspore1.10 loss_mask = self.add(loss_mask, 1) return logits, tokens, loss_mask loss_mask = self.stridedslice(loss_mask, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.stridedslice(input_ids, (0, 1), (batch_size, seq_length), (1, 1)) labels = self.reshape(labels, (-1,)) loss_mask = self.reshape(loss_mask, (-1,)) loss = self.loss(logits, labels, loss_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2ForSequenceClassification(BaseModel): r""" Provide gpt training loss or logits through network. Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ _support_list = MindFormerBook.get_model_support_list()['gpt2'] def __init__(self, config: GPT2Config = None): self.config = config if config is not None else GPT2Config() super(GPT2ForSequenceClassification, self).__init__(self.config, auto_prefix=True) self.eos_token = self.config.eos_token self.seq_length = self.config.seq_length self.num_labels = self.config.num_labels self.hidden_size = self.config.hidden_size parallel_config = self.config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.get_attention_mask = AttentionMask(seq_length=self.seq_length, parallel_config=parallel_config.dp_mp_config) self.backbone = GPT2Model(self.config) self.score = Linear(in_channels=self.hidden_size, out_channels=self.num_labels, has_bias=False, compute_dtype=self.config.compute_dtype) self.score.shard(strategy_matmul=((dp, 1), (1, 1))) if parallel_config.pipeline_stage > 1: self.head.pipeline_stage = parallel_config.pipeline_stage - 1 self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) vocab_size = self.config.vocab_size loss_parallel_config = copy.deepcopy(parallel_config) if vocab_size % mp != 0: logger.warning("The vocab size of GPT Loss is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPT Loss will be changed: mp = 1") loss_parallel_config.model_parallel = 1 self.loss = nn.CrossEntropyLoss() self.reshape = P.Reshape() self.cast = P.Cast() self.load_checkpoint(config) self.reduce_sum = P.ReduceSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp,), (dp,))) self.sub = P.Sub().shard(((1,), ())) self.gather = P.Gather().shard(((1, 1), (1,))) # pylint: disable=W0613 def construct(self, input_ids, attention_mask, labels=None, input_embeds=None, input_position=None, position_ids=None, init_reset=True, batch_valid_length=None): r""" construct function for GPT2 Text Classification Model Args: input_ids (Tensor): the indices of input sequence tokens in the vocabulary. attention_mask (Tensor): input sentences padding mask, where 0 indicates padding position. labels (Tensor): the labels of corresponding input sequences. input_embeds(Tensor): Reserved param, not used. input_position(Tensor): Reserved param, not used. position_ids(Tensor): Reserved param, not used. init_reset(Tensor): Reserved param, not used. batch_valid_length(Tensor): Reserved param, not used. Returns: (logits, labels) (Tensor, Tensor) or logits (Tensor) or loss (mstype.float32): in train mode, return loss; in eval mode, return logits and loss; in predict mode, return logits. """ attention_mask = self.cast(attention_mask, mstype.float32) attention_mask_lower_triangle = self.get_attention_mask(attention_mask) output_states, _ = self.backbone(input_ids, attention_mask_lower_triangle) output_states = self.reshape(output_states, (-1, self.hidden_size)) logits = self.score(output_states) # get the last logit of each sequence last_indices = self.sub(self.reduce_sum(attention_mask, -1), 1) batch_size = attention_mask.shape[0] indices_increments = Tensor(np.arange(0, self.seq_length * batch_size, self.seq_length)) last_indices = self.cast(self.add(last_indices, indices_increments), mstype.int32) pooled_logits = self.gather(logits, last_indices, 0) if labels is not None: labels = self.reshape(labels, (-1,)) if self.training: output = self.loss(pooled_logits, labels) else: if labels is not None: output = (pooled_logits, labels) else: output = pooled_logits return output class GPTEmbeddingLayer(nn.Cell): r"""The Embedding Layer of GPT-2 network.""" def __init__(self, config: GPT2Config = None): super(GPTEmbeddingLayer, self).__init__() parallel_config = copy.deepcopy(config.parallel_config) embedding_mp = config.parallel_config.embedding_dp_mp_config.model_parallel vocab_size = config.vocab_size if vocab_size % embedding_mp != 0: logger.warning("The vocab size of embedding layer is: %s, it is not divide by model_parallel: %s", vocab_size, embedding_mp) logger.warning("Now, model_parallel will be changed: mp = 1") parallel_config.embedding_dp_mp_config.model_parallel = 1 self.word_embedding = VocabEmbedding(vocab_size=vocab_size, embedding_size=config.hidden_size, param_init=initializer('normal', [vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=parallel_config.embedding_dp_mp_config) new_parallel_config = copy.deepcopy(parallel_config) new_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer('normal', [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=new_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dropout = Dropout(1 - config.embedding_dropout_prob) self.dropout.shard(((parallel_config.data_parallel, 1, 1),)) def construct(self, input_ids, input_position): """The forward compute of Embedding Layer.""" word_embedding, word_table = self.word_embedding(input_ids) position_embedding, _ = self.position_embedding(input_position) embedding = self.add(word_embedding, position_embedding) embedding = self.dropout(embedding) return embedding, word_table def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ pp_dis = max(int((layers + 1) / parallel_config.pipeline_stage), 1) pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class GPT2Model(nn.Cell): """ The backbone of GPT network Args: config(GPT2Config): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 input_mask: the mask indicating whether each position is a valid input Returns: output_state: Tensor, the output logit of backbone present_layer: Tensor, the current feature map embedding_table: Tensor, the embedding table for the vocabulary """ def __init__(self, config): super(GPT2Model, self).__init__() self.config = config self.embedding = GPTEmbeddingLayer(config) self.embedding.pipeline_stage = 0 self.layernorm = LayerNorm((config.hidden_size,)).to_float(config.layernorm_compute_type) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 if not hasattr(config.parallel_config, "moe_config"): config.parallel_config.moe_config = default_moe_config moe_config = config.parallel_config.moe_config self.blocks = nn.CellList() for i in range(config.num_layers): block = GPTTransformerDecoderLayer( hidden_size=config.hidden_size, batch_size=config.batch_size, ffn_hidden_size=config.hidden_size * config.expand_ratio, seq_length=config.seq_length, num_heads=config.num_heads, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config.dp_mp_config, moe_config=moe_config, use_past=config.use_past ) set_parallel_configure_for_layer( block, layer_id=i, layers=config.num_layers, offset=0, parallel_config=config.parallel_config) self.blocks.append(block) self.cast = P.Cast() self.tile = P.Tile().shard(((config.parallel_config.data_parallel,),)) self.dtype = mstype.float16 self.num_layers = config.num_layers self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.is_first_iteration = True def construct(self, input_ids, attention_mask, input_position=None, init_reset=True, batch_valid_length=None): """GPT model""" batch_size, seq_length = F.shape(input_ids) # When input_position is None, the phase is train mode. When the phase is train mode and is the first iteration # of incremental reasoning, it goes into the following logic. if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when the phase is not train and incremental reasoning is not the first iteration, it goes into the # following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) input_embedding, embedding_table = self.embedding(input_ids, input_position) hidden_states = self.cast(input_embedding, self.dtype) hidden_shape = F.shape(hidden_states) hidden_states = F.reshape(hidden_states, (-1, hidden_shape[-1])) for i in range(self.num_layers): hidden_states = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) output_state = self.layernorm(hidden_states) return output_state, embedding_table class GPTHead(nn.Cell): r"""Head for GPT to get the logits of each token in the vocab.""" def __init__(self, hidden_size, vocab_size, compute_type=mstype.float16, parallel_config=None): super().__init__() copied_parallel_config = copy.deepcopy(parallel_config) mp = copied_parallel_config.model_parallel if vocab_size % mp != 0: logger.warning("The vocab size of GPTHead MatMul is: %s, it is not divide by model_parallel: %s", vocab_size, mp) logger.warning("Now, the model_parallel num of GPTHead MatMul will be changed: mp = 1") copied_parallel_config.model_parallel = 1 if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False if copied_parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((copied_parallel_config.data_parallel, 1), ( copied_parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embedding_table): logits = self.matmul(self.cast(state, self.dtype), self.cast(embedding_table, self.dtype)) return logits @MindFormerRegister.register(MindFormerModuleType.MODELS) class GPT2WithLora(GPT2LMHeadModel): """ GPT2LMHeadModel with LoRA parameter-efficient tuning algorithm Args: config (GPT2Config): The config of Gpt2Model. Returns: Tensor, the loss or logits of the network. """ def __init__(self, config: GPT2Config = None): checkpoint_name_or_path = config.pop("checkpoint_name_or_path") super().__init__(config) config.pet_config.reg_rules = r'.*dense1.*|.*dense3.*' self.backbone = LoraAdapter.get_pet_model(self.backbone, config.pet_config) config.checkpoint_name_or_path = checkpoint_name_or_path self.load_checkpoint(config) # freeze pretrained model PetAdapter.freeze_pretrained_model(self, config.pet_config.pet_type) class CrossEntropyCalculationWithMask(nn.Cell): """ Cross Entropy loss """ def __init__(self, is_training=None, num_labels=None): super(CrossEntropyCalculationWithMask, self).__init__() self.onehot = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_sum = P.ReduceSum() self.reduce_mean = P.ReduceMean() self.reshape = P.Reshape() self.last_idx = (-1,) self.neg = P.Neg() self.cast = P.Cast() self.is_training = is_training self.num_labels = num_labels self.log_softmax = P.LogSoftmax(axis=-1) def construct(self, logits, label_ids, input_mask=None): """ Calculate loss Args: logits (Tensor): the probability distribution over vocabulary. label_ids (Tensor): the indices of input sequence tokens in the vocabulary. input_mask (Tensor): input sentences padding mask, where 0 indicates padding position. Returns: return_value (Tensor, mstype.float32): if is_training is False, directly return the logits, otherwise, return the computed loss. """ # logits [batch * (seq_length-1), vocab_size] label_ids [batch, seq_length-1] logits = self.log_softmax(logits) if self.is_training: label_ids = self.reshape(label_ids, self.last_idx) # label_ids [batch * (seq_length-1)] one_hot_labels = self.onehot(label_ids, self.num_labels, self.on_value, self.off_value) # [batch * (seq_length-1), vocab_size] per_example_loss = self.neg( self.reduce_sum(one_hot_labels * logits, self.last_idx)) # [batch * (seq_length-1)] # for PPL calculation in evaluation if input_mask is not None: input_mask = self.cast(self.reshape(input_mask, self.last_idx), mstype.float32) # [batch * (seq_length-1)] valid_loss_sum = self.reduce_sum(input_mask * per_example_loss, ()) valid_element_sum = self.reduce_sum(input_mask, ()) + self.cast(F.tuple_to_array((1e-5,)), mstype.float32) loss = valid_loss_sum / valid_element_sum else: loss = self.reduce_mean(per_example_loss, self.last_idx) # a number return_value = self.cast(loss, mstype.float32) else: return_value = logits * 1.0 # [batch * (seq_length-1), vocab_size] return return_value pengcheng.mind-mindformer/mindformers/models/gpt2/gpt2_config.py000066400000000000000000000103241476650141400253510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Gpt Config API.""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['GPT2Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class GPT2Config(BaseConfig): """ Gpt config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['gpt2'] def __init__(self, batch_size: int = 1, eos_token_id: int = 50256, pad_token_id: int = 50256, bos_token_id: int = 50256, unk_token_id: int = 50256, seq_length: int = 1024, vocab_size: int = 50257, hidden_size: int = 768, num_layers: int = 12, num_heads: int = 12, num_labels: int = 2, expand_ratio: int = 4, embedding_dropout_prob: float = 0.1, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, param_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", compute_dtype: str = "float16", hidden_act: str = 'gelu', use_past: bool = False, post_layernorm_residual: bool = False, offset: int = 0, parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = "", moe_config: MoEConfig = default_moe_config, repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs): super(GPT2Config, self).__init__(**kwargs) self.batch_size = batch_size self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.bos_token_id = bos_token_id self.unk_token_id = unk_token_id self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.num_labels = num_labels self.expand_ratio = expand_ratio self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.hidden_act = hidden_act self.use_past = use_past self.post_layernorm_residual = post_layernorm_residual self.offset = offset self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.moe_config = moe_config self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample pengcheng.mind-mindformer/mindformers/models/gpt2/gpt2_processor.py000066400000000000000000000044121476650141400261240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ GPT2Processor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['GPT2Processor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class GPT2Processor(BaseProcessor): """ GPT2 processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['gpt2'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(GPT2Processor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-mindformer/mindformers/models/gpt2/gpt2_tokenizer.py000066400000000000000000000367771476650141400261420ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT2 Tokenizer""" import json from functools import lru_cache from typing import List, Optional import os import regex as re from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools import logger from ...mindformer_book import MindFormerBook __all__ = ['GPT2Tokenizer'] @lru_cache() def bytes_to_unicode(): """ Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. """ bs = ( list(range(ord("!"), ord("~") + 1)) + list(range(ord("¡"), ord("¬") + 1)) + list(range(ord("®"), ord("ÿ") + 1)) ) cs = bs[:] n = 0 for b in range(2**8): if b not in bs: bs.append(b) cs.append(2**8 + n) n += 1 cs = [chr(n) for n in cs] return dict(zip(bs, cs)) def get_pairs(word): """ Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). """ pairs = set() prev_char = word[0] for char in word[1:]: pairs.add((prev_char, char)) prev_char = char return pairs VOCAB_FILES_NAMES = { "vocab_file": "vocab.json", "merges_file": "merges.txt", } @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class GPT2Tokenizer(Tokenizer): """ Construct a GPT-2 tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer has been trained to treat spaces like parts of the tokens (a bit like sentencepiece) so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from mindformers import GPT2Tokenizer >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> tokenizer("Hello world", add_bos_token=False, add_eos_token=False)["input_ids"] [15496, 995] >>> tokenizer(" Hello world", add_bos_token=False, add_eos_token=False)["input_ids"] [18435, 995] ``` You can get around that behavior by passing `add_prefix_space=True` when instantiating this tokenizer or when you call it on some text, but since the model was not pretrained this way, it might yield a decrease in performance. When used with `is_split_into_words=True`, this tokenizer will add a space before each word (even the first one). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `<|endoftext|>`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `<|endoftext|>`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `<|endoftext|>`): The end of sequence token. add_prefix_space (`bool`, *optional*, defaults to `False`): Whether or not to add an initial space to the input. This allows to treat the leading word just as any other word. (GPT2 tokenizer detect beginning of words by the preceding space). add_bos_token(`bool`, *optional*, defaults to `True`): Whether or not to add the bos_token_id to the left of the input. add_eos_token(`bool`, *optional*, defaults to `True`): Whether or not to add the eos_token_id to the right of the input. Examples: >>> from mindformers import GPT2Tokenizer >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> res = tokenizer("Hello world") >>> print(res) {'input_ids': [50256, 15496, 995, 50256], 'token_type_ids': [0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1]} >>> tokenizer = GPT2Tokenizer.from_pretrained("gpt2") >>> res = tokenizer("Hello world", add_special_tokens=False) >>> print(res) {'input_ids': [15496, 995], 'token_type_ids': [0, 0], 'attention_mask': [1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "token_type_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['gpt2'] def __init__( self, vocab_file, merges_file, errors="replace", unk_token="<|endoftext|>", bos_token="<|endoftext|>", eos_token="<|endoftext|>", pad_token="<|endoftext|>", add_prefix_space=False, add_bos_token=False, add_eos_token=False, **kwargs, ): bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( errors=errors, unk_token=unk_token, bos_token=bos_token, eos_token=eos_token, pad_token=pad_token, add_prefix_space=add_prefix_space, **kwargs, ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token with open(vocab_file, encoding="utf-8") as vocab_handle: self.encoder = json.load(vocab_handle) self.decoder = {v: k for k, v in self.encoder.items()} self.errors = errors # how to handle errors in decoding self.byte_encoder = bytes_to_unicode() self.byte_decoder = {v: k for k, v in self.byte_encoder.items()} with open(merges_file, encoding="utf-8") as merges_handle: bpe_merges = merges_handle.read().split("\n")[1:-1] bpe_merges = [tuple(merge.split()) for merge in bpe_merges] self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges)))) self.cache = {} self.add_prefix_space = add_prefix_space # Should have added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""") @property def vocab_size(self): return len(self.encoder) def get_vocab(self): return dict(self.encoder, **self.added_tokens_encoder) def bpe(self, token): """bpe encode """ if token in self.cache: return self.cache[token] word = tuple(token) pairs = get_pairs(word) if not pairs: return token while True: bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float("inf"))) if bigram not in self.bpe_ranks: break first, second = bigram new_word = [] i = 0 while i < len(word): try: j = word.index(first, i) except ValueError: new_word.extend(word[i:]) break else: new_word.extend(word[i:j]) i = j if word[i] == first and i < len(word) - 1 and word[i + 1] == second: new_word.append(first + second) i += 2 else: new_word.append(word[i]) i += 1 new_word = tuple(new_word) word = new_word if len(word) == 1: break else: pairs = get_pairs(word) word = " ".join(word) self.cache[token] = word return word def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): """Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) if not self.add_bos_token: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=False ) if token_ids_1 is None: return [1] + ([0] * len(token_ids_0)) return [1] + ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) def _tokenize(self, text, **kwargs): """Tokenize a string.""" bpe_tokens = [] for token in re.findall(self.pat, text): token = "".join( self.byte_encoder[b] for b in token.encode("utf-8") ) # Maps all our bytes to unicode strings, avoiding control tokens of the BPE (spaces in our case) bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(" ")) return bpe_tokens def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.encoder.get(token, self.encoder.get(self.unk_token)) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" return self.decoder.get(index) def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" text = "".join(tokens) text = bytearray([self.byte_decoder[c] for c in text]).decode("utf-8", errors=self.errors) return text def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) merge_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["merges_file"] ) with open(vocab_file, "w", encoding="utf-8") as f: # you can set sort_keys=True, then the keys of vocabulary dict is sorted, # which is different from original vocabulary file. f.write(json.dumps(self.encoder, indent=2, ensure_ascii=False) + "\n") index = 0 with open(merge_file, "w", encoding="utf-8") as writer: writer.write("#version: 0.2\n") for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]): if index != token_index: logger.warning( "Saving vocabulary to %s: BPE merge indices are not consecutive." " Please check that the tokenizer is not corrupted!", merge_file ) index = token_index writer.write(" ".join(bpe_tokens) + "\n") index += 1 return vocab_file, merge_file def prepare_for_tokenization(self, text, **kwargs): add_prefix_space = kwargs.pop("add_prefix_space", self.add_prefix_space) is_split_into_words = kwargs.pop("is_split_into_words", False) if is_split_into_words or add_prefix_space: text = " " + text return (text, kwargs) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-mindformer/mindformers/models/gpt2/gpt_modules.py000066400000000000000000000253621476650141400255020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """GPT modules.""" import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.modules.transformer.moe import default_moe_config from mindformers.modules.transformer import TransformerEncoderLayer from mindformers.modules.transformer.op_parallel_config import default_dpmp_config class GPTTransformerDecoderLayer(TransformerEncoderLayer): r"""GPT Transformer Decoder Layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` """ def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(GPTTransformerDecoderLayer, self).__init__( batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, hidden_act=hidden_act, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config ) def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, aux_loss return output pengcheng.mind-mindformer/mindformers/models/llama/000077500000000000000000000000001476650141400230105ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/000077500000000000000000000000001476650141400266015ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000017321476650141400330420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for llama""" from .llama_config import LlamaConfig from .llama import LlamaForCausalLM, LlamaModel from .llama_tokenizer import LlamaTokenizer from .llama_processor import LlamaProcessor __all__ = ['LlamaConfig', 'LlamaModel', 'LlamaForCausalLM', 'LlamaTokenizer', 'LlamaProcessor'] pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/llama-checkpoint.py000066400000000000000000000456311476650141400323770ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA models' APIs.""" import math import numpy as np import mindspore.common.dtype as mstype try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import Tensor, nn from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation try: # pylint: disable=W0611 from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindspore.common.initializer import Normal from mindspore import lazy_inline from mindformers.core.loss.loss import CrossEntropyLoss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear from mindformers.modules.transformer.op_parallel_config import _check_config from mindformers.modules.transformer.transformer import AttentionMask from mindformers.tools.register.register import MindFormerModuleType, MindFormerRegister #from mindformers.pet.tuners.pet_adapter import PetAdapter #from mindformers.pet.tuners.lora_adapter import LoraAdapter from .llama_config import LlamaConfig from .llama_layer import LlamaEmbedding, LlamaRMSNorm, precompute_freqs_cis from .llama_transformer import LLamaDecodeLayer from ..utils import cell_reuse from ...tools.logger import logger __all__ = ['LlamaModel', 'LlamaForCausalLM'] def layer_compute_dtype(layer, layer_id, offset, parallel_config, n_layers, select_recompute=False): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. """ pp_dis = max(int((n_layers + 1) / parallel_config.pipeline_stage), 1) if isinstance(offset, list): if len(offset) != parallel_config.pipeline_stage: raise ValueError(f"The length of `offset` {len(offset)} do not match " "`pipeline stage` {parallel_config.pipeline_stage}.") i = min(layer_id // pp_dis, parallel_config.pipeline_stage - 1) offset_layer = offset[i] elif isinstance(offset, int): offset_layer = offset else: raise TypeError(f"`offset` must be `int` of list of `int`, but got {type(offset)}.") if parallel_config.pipeline_stage == 4: pp_id_list = [10, 11, 11, 8] pp_id = 0 tmp = layer_id while tmp >= pp_id_list[pp_id]: tmp -= pp_id_list[pp_id] pp_id += 1 else: pp_id = min((layer_id + offset_layer) // pp_dis, parallel_config.pipeline_stage - 1) layer.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((n_layers + 1) / parallel_config.gradient_aggregation_group), 1) layer.set_comm_fusion(3 + layer_id) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not select_recompute: layer.recompute() else: if parallel_config.recompute.recompute and not select_recompute: layer.recompute( recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) #layer.attention.flash_attention.flash_attention.recompute(mode=False) class LlamaModel(BaseModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer """ _support_list = MindFormerBook.get_model_support_list()['llama'] def __init__(self, config: LlamaConfig = None): super().__init__(config, auto_prefix=True) _check_config(config.parallel_config) if config.batch_size or config.use_past: Validator.check_positive_int(config.batch_size) self.dtype = config.compute_dtype self.num_layers = config.num_layers self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.is_first_iteration = True self.use_past = config.use_past self.use_flash_attention = config.use_flash_attention and FLASHATTENTION_VALID if self.use_flash_attention: logger.info("Enable flash attention.") elif config.use_flash_attention: logger.info("Current MindSpore do not support flash attention.") self.freqs_cos, self.freqs_sin, self.swap_mask = precompute_freqs_cis( config.hidden_size // config.num_heads, config.seq_length, dtype=config.rotary_dtype, pretrain_seqlen=config.pretrain_seqlen, extend_method=config.extend_method) self.get_attention_mask = AttentionMask( config.seq_length, parallel_config=config.parallel_config.dp_mp_config).to_float(config.compute_dtype) self.multiply_data = Tensor([-10000.0], dtype=config.compute_dtype) self.one = Tensor([1.0], dtype=config.compute_dtype) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile() self.mul_mask = P.Mul() self.sub = P.Sub() self.expand_dims = P.ExpandDims() self.not_equal = P.NotEqual() self.gather = P.Gather() self.tok_embeddings = LlamaEmbedding( config.vocab_size, config.hidden_size, param_init_type=config.param_init_type) self.embed_scale = Tensor([config.embed_scale], dtype=config.compute_dtype) self.mul_embed = P.Mul() self.layers = nn.CellList() for layer_id in range(config.num_layers): layer = LLamaDecodeLayer(config.batch_size, config.seq_length, layer_id, config.num_layers, dim=config.hidden_size, n_heads=config.num_heads, multiple_of=config.multiple_of, n_kv_heads=config.n_kv_heads, ffn_dim_multiplier=config.ffn_dim_multiplier, mla_multiplier=config.mla_multiplier, norm_eps=config.rms_norm_eps, compute_dtype=config.compute_dtype, layernorm_compute_dtype=config.layernorm_compute_type, softmax_compute_dtype=config.softmax_compute_type, rotary_dtype=config.rotary_dtype, mup_width_base=config.mup_width_base, param_init_type=config.param_init_type, use_past=config.use_past, use_flash_attention=config.use_flash_attention, compute_in_2d=config.compute_in_2d, use_past_shard=config.use_past_shard, depth_scale=config.depth_scale, init_std=config.init_std, parallel_config=config.parallel_config) layer_compute_dtype(layer, layer_id, config.offset, config.parallel_config, config.num_layers, select_recompute=config.parallel_config.recompute.select_recompute) self.layers.append(layer) self.norm_out = LlamaRMSNorm(config.hidden_size, config.rms_norm_eps, compute_type=config.layernorm_compute_type) dp = config.parallel_config.data_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.tok_embeddings.pipeline_stage = 0 if config.parallel_config.pipeline_stage > 1: self.norm_out.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.tok_embeddings.set_comm_fusion(2) self.norm_out.set_comm_fusion(config.num_layers + 3) self.tok_embeddings.shard(config.parallel_config) self.tile.shard(((1, 1, 1, 1), ())) self.sub.shard(((1,), (dp, 1, 1))) self.mul_mask.shard(((dp, 1, 1, 1), (1,))) self.mul_embed.shard(((dp, 1, 1), (1,))) self.expand_dims.shard(((dp, 1, 1),)) self.not_equal.shard(((dp, 1), ())) self.gather.shard(((dp, 1), (1,))) if config.compute_in_2d: self.norm_out.shard((dp, 1)) else: self.norm_out.shard((dp, 1, 1)) if self.use_past: seq_range = np.arange(config.seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (config.batch_size, 1, 1)), mstype.int32) self.gather_past = P.Gather() self.expand_dims = P.ExpandDims() self.le_past = P.LessEqual() ones = np.ones(shape=(config.seq_length, config.seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) self.equal = P.Equal().shard(((dp, 1), ())) self.cumsum = P.CumSum(reverse=True).shard(((dp, 1),)) self.pow = P.Pow().shard(((), (dp, 1))) self.pow_up_value = 2.0 self.pow_down_value = 0.5 self.mask_reshape = P.Reshape() self.mask_mul = P.BatchMatMul().shard( ((dp, 1, 1), (dp, 1, 1))) self.up_threshold = 1.2 self.down_threshold = 0.8 self.greater = P.Greater().shard(((dp, 1, 1), ())) self.less = P.Less().shard(((dp, 1, 1), ())) self.mask_multiply = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mask_expand = P.ExpandDims().shard(((1, 1),)) self.mask_multiply2 = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) # pylint: disable=W0613 def construct(self, tokens: Tensor, input_position=None, init_reset=True, batch_valid_length=None): """Forward of llama model.""" # preprocess bs, seq_len = tokens.shape if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), self.dtype) seq_index = self.cast(self.equal(tokens, self.eos_token_id), mstype.float32) seq_mask_rev = self.cumsum(seq_index, 1) seq_mask_exp_up = self.mask_reshape(self.pow(self.pow_up_value, seq_mask_rev), (bs, seq_len, 1)) seq_mask_exp_down = self.mask_reshape(self.pow(self.pow_down_value, seq_mask_rev), (bs, 1, seq_len)) full_mask = self.mask_mul(seq_mask_exp_up, seq_mask_exp_down) full_mask_small = self.cast(self.less(full_mask, self.up_threshold), mstype.float32) full_mask_great = self.cast(self.greater(full_mask, self.down_threshold), mstype.float32) mask = self.mask_multiply(full_mask_small, full_mask_great) lower_triangle = self.mask_expand(self.lower_triangle_mask, 0) mask = self.mask_multiply2(full_mask, lower_triangle) # mask: [bs, seq, seq] # else: # cur_pos = batch_valid_length - 1 # valid_length = self.reshape(cur_pos, (-1, 1, 1)) # freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), # self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), # self.swap_mask) # mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) # tokens: [bs, seq/1] h = self.tok_embeddings(tokens) h = self.mul_embed(h, self.embed_scale) # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForCausalLM(BaseModel): r""" Provide llama training loss or logits through network. Args: config (LlamaConfig): The config of llama model. Inputs: input_ids(Tensor): the tokenized inputs with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. labels(Tensor): the tokenized labels with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. input_position(Tensor): current position, used by model.predict. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: Tensor, the loss or logits of the network. Examples: >>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) """ _support_list = MindFormerBook.get_model_support_list()['llama'] #@lazy_inline def __init__(self, config: LlamaConfig = None): super(LlamaForCausalLM, self).__init__(config, auto_prefix=True) _check_config(config.parallel_config) self.ignore_token_id = config.ignore_token_id self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.reshape = P.Reshape() self.cast = P.Cast() self.slice = P.StridedSlice() self.not_equal = P.NotEqual() self.mul = P.Mul() self.add = P.Add() self.model = LlamaModel(config=config) sigma = config.init_std if config.mup_width_base > 0: ratio = config.hidden_size / config.mup_width_base sigma = sigma / ratio self.lm_head = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, has_bias=False, compute_dtype=config.compute_dtype, param_init_type=config.param_init_type, weight_init=Normal(sigma=sigma, mean=0)) # meta default: xavier_normal self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.slice.shard(((dp, 1),)) self.not_equal.shard(((dp, 1), ())) self.mul.shard(((dp, 1), (dp, 1))) self.add.shard(((dp, 1), ())) if config.parallel_config.vocab_emb_dp: self.lm_head.shard(strategy_matmul=((dp, 1), (1, 1))) else: self.lm_head.shard(strategy_matmul=((dp, 1), (mp, 1))) if config.parallel_config.pipeline_stage > 1: self.lm_head.pipeline_stage = config.parallel_config.pipeline_stage - 1 #self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, labels=None, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, init_reset=True, batch_valid_length=None): """LlamaForCausalLM forward.""" bsz, seqlen = input_ids.shape #if self.training: # tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) #else: # tokens = input_ids tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) output = self.model(tokens, input_position, init_reset, batch_valid_length) logits = self.lm_head(output) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if labels is None: labels = self.slice(input_ids, (0, 1), (bsz, seqlen), (1, 1)) else: if labels.ndim > 1: if self.training: labels = self.slice(labels, (0, 1), (bsz, seqlen), (1, 1)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) input_mask = self.mul(input_mask, label_mask) logits = self.cast(logits, mstype.float32) #if not self.training: # logits = self.reshape(logits, (bsz, seqlen, -1)) # # makes cast effective to avoid allgather issue in Mindspore1.10 # input_mask = self.add(input_mask, 1) # return logits, tokens, input_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/llama_config-checkpoint.py000066400000000000000000000114721476650141400337200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Llama Config API.""" from typing import Optional from mindformers.modules.transformer.transformer import default_transformer_config, TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['LlamaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class LlamaConfig(BaseConfig): """ LLaMA config class which defines the model size. """ _support_list = MindFormerBook.get_config_support_list()['llama'] def __init__(self, batch_size: int = 1, seq_length: int = 2048, hidden_size: int = 4096, num_layers: int = 32, num_heads: int = 32, vocab_size: int = 32000, # defined later by tokenizer multiple_of: int = 256, # make SwiGLU hidden layer size multiple of large power of 2 n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, rms_norm_eps: float = 1e-5, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 32000, ignore_token_id: int = -100, compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", rotary_dtype: str = "float32", param_init_type: str = "float16", parallel_config: TransformerOpParallelConfig = default_transformer_config, use_past: bool = False, pretrain_seqlen: int = 2048, extend_method: str = "None", compute_in_2d: bool = False, use_flash_attention: bool = False, offset: int = 0, use_past_shard: bool = False, checkpoint_name_or_path: str = "", repetition_penalty: float = 1.0, max_decode_length: int = 1024, final_logit_softcapping: float = 30.0, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, mup_width_base: int = -1, embed_scale: float = 1.0, # depth_scale: float = 1.0, init_std: float = 0.01, **kwargs): super(LlamaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.multiple_of = multiple_of self.n_kv_heads = n_kv_heads self.ffn_dim_multiplier = ffn_dim_multiplier self.mla_multiplier = mla_multiplier self.rms_norm_eps = rms_norm_eps self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.rotary_dtype = convert_mstype(rotary_dtype) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.ignore_token_id = ignore_token_id self.use_past = use_past self.pretrain_seqlen = pretrain_seqlen self.extend_method = extend_method self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention self.offset = offset self.use_past_shard = use_past_shard self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.mup_width_base = mup_width_base self.embed_scale = embed_scale pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/llama_layer-checkpoint.py000066400000000000000000000364631476650141400335760ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA Model Layers' APIs.""" from enum import Enum import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore import nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell from mindspore.common.initializer import Normal try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.common.initializer import initializer from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import Linear, _check_input_dtype, _args_type_validator_check, _valid_value_checks from mindformers.tools.logger import _LogActionOnce class SeqExtendMethod(Enum): """Stores the acceptable string identifiers for seq length extend method""" PI = "PI" NTK = "NTK" NONE = "None" class LlamaSiLU(Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super().__init__() self.sigmoid = nn.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) def get_swap_mask(head_dim): """Swap matrix""" zero_block = np.zeros((head_dim // 2, head_dim // 2), dtype=np.float32) id_block = np.identity(head_dim // 2, dtype=np.float32) return np.block([[zero_block, id_block], [-id_block, zero_block]]) def precompute_freqs_cis( dim: int, end: int, theta: float = 10000.0, dtype=mstype.float32, pretrain_seqlen=2048, extend_method=SeqExtendMethod.NONE.value): """ Precompute of freqs and mask for rotary embedding. """ ratio = 1. if extend_method != SeqExtendMethod.NONE.value and end > pretrain_seqlen: ratio = end / pretrain_seqlen if extend_method == SeqExtendMethod.NTK.value: theta *= ratio freqs_base = np.arange(0, dim, 2)[: (dim // 2)].astype(np.float32) # (head_dim // 2, ) freqs = 1.0 / (theta ** (freqs_base / dim)) # (head_dim // 2, ) if extend_method == SeqExtendMethod.PI.value: t = np.arange(0, end / ratio, 1 / ratio).astype(np.float32) else: t = np.arange(0, end, 1).astype(np.float32) # type: ignore # (seq_len,) freqs = np.outer(t, freqs) # type: ignore (seq_len, head_dim // 2) emb = np.concatenate((freqs, freqs), axis=-1) freqs_cos = np.cos(emb) # (seq_len, head_dim) freqs_sin = np.sin(emb) # (seq_len, head_dim) freqs_cos = Tensor(freqs_cos, dtype=dtype) freqs_sin = Tensor(freqs_sin, dtype=dtype) swap_mask = get_swap_mask(dim) swap_mask = Tensor(swap_mask, dtype=dtype) return freqs_cos, freqs_sin, swap_mask class LlamaRotaryEmbedding(Cell): r""" Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, head_dim=128, compute_dtype=mstype.float32): super().__init__(auto_prefix=False) self.head_dim = head_dim self.dtype = compute_dtype self.add = P.Add() self.bmm_swap = P.BatchMatMul() self.mul = P.Mul() self.cast = P.Cast() def rotate_half(self, x, swap_mask): # [bs, n_head/n_kv_head, seq/1, head_dim], [head_dim, head_dim] x = self.bmm_swap(x, swap_mask) return x def construct(self, xq: Tensor, xk: Tensor, freqs_cis): """Forward of rotary position embedding.""" original_type = xq.dtype xq = self.cast(xq, self.dtype) xk = self.cast(xk, self.dtype) # xq, xk: [bs, n_head/n_kv_head, seq/1, head_dim] freqs_cos, freqs_sin, swap_mask = freqs_cis xq_out = self.add(self.mul(xq, freqs_cos), self.mul(self.rotate_half(xq, swap_mask), freqs_sin)) xk_out = self.add(self.mul(xk, freqs_cos), self.mul(self.rotate_half(xk, swap_mask), freqs_sin)) xq_out = self.cast(xq_out, original_type) xk_out = self.cast(xk_out, original_type) return xq_out, xk_out def shard(self, strategy_in): self.add.shard((strategy_in, strategy_in)) self.bmm_swap.shard((strategy_in, (1, 1))) self.mul.shard((strategy_in, (strategy_in[0], 1, 1, 1))) class LlamaEmbedding(Cell): """ Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). """ @_LogActionOnce(m_logger=logger, key='Embedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_table_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int) def __init__(self, vocab_table_size, embedding_size, param_init_type=mstype.float32, param_init='normal'): super().__init__() self.vocab_table_size = vocab_table_size self.embedding_size = embedding_size self.embedding_weight = Parameter( initializer(param_init, [self.vocab_table_size, self.embedding_size], dtype=param_init_type), name='embedding_weight', parallel_optimizer=False) self.gather = P.Gather() def construct(self, input_ids): """Forward of vocab embedding.""" _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32, mstype.int64], self.cls_name) output = self.gather(self.embedding_weight, input_ids, 0) return output def shard(self, parallel_config): """sharding for embedding""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if parallel_config.vocab_emb_dp: self.gather.shard(((1, 1), (dp, 1))) logger.info(f"Using {dp} data parallel for the embedding lookup.") else: if self.vocab_table_size % mp != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_table_size} must be a " f"multiple of parallel_config.model_parallel {mp}.") self.gather.shard(((mp, 1), (dp, 1))) logger.info(f"Using {dp} data parallel and {mp} " f"model parallel for the embedding lookup.") class LlamaRMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, compute_type=mstype.float32): super(LlamaRMSNorm, self).__init__() self.eps = eps self.weight = Parameter(initializer('ones', (dim,), dtype=mstype.float32), parallel_optimizer=False) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() self.cast = P.Cast() self.compute_type = compute_type def _norm(self, x): norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) x = self.cast(x, mstype.float16) norm_factor = self.cast(norm_factor, mstype.float16) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" original_type = x.dtype x = self.cast(x, self.compute_type) output = self._norm(x) output = self.cast(output, mstype.float16) weight = self.cast(self.weight, mstype.float16) output = self.mul2(output, weight) output = self.cast(output, original_type) return output def shard(self, strategy_in): """Parallel strategy configuratiuon interface.""" self.square.shard((strategy_in,)) self.mean.shard((strategy_in,)) self.rsqrt.shard((strategy_in,)) self.add.shard((strategy_in, ())) self.mul.shard((strategy_in, strategy_in)) self.mul2.shard((strategy_in, (1,))) class LlamaFeedForward(Cell): r""" LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(dim=Validator.check_positive_int, hidden_dim=Validator.check_positive_int, multiple_of=Validator.check_positive_int, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward")) def __init__(self, dim, hidden_dim, multiple_of, hidden_act=LlamaSiLU, ffn_dim_multiplier=None, mup_width_base=-1, compute_dtype=mstype.float16, param_init_type=mstype.float32): super().__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if ffn_dim_multiplier is not None: hidden_dim = int((ffn_dim_multiplier + 0.01) * hidden_dim) hidden_dim = int(2 * hidden_dim / 3) hidden_dim = multiple_of * \ ((hidden_dim + multiple_of - 1) // multiple_of) self.dtype = compute_dtype self.hidden_act = hidden_act self.dim = dim self.hidden_dim = hidden_dim self.mul = P.Mul() self.cast = P.Cast() sigma = 0.01 if mup_width_base > 0: ratio = dim / mup_width_base sigma = sigma / math.sqrt(ratio) self.w1 = Linear(in_channels=dim, out_channels=hidden_dim, activation=hidden_act, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w2 = Linear(in_channels=hidden_dim, out_channels=dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w3 = Linear(in_channels=dim, out_channels=hidden_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, self.dtype) # [bs, seq, hidden_dim] or [bs * seq, hidden_dim] gate = self.w1(x) # dp,1 -> dp, mp hidden = self.w3(x) # dp,1 -> dp, mp hidden = self.mul(hidden, gate) # dp,mp -> dp, mp output = self.w2(hidden) # dp,mp -> dp, 1 return output def shard(self, parallel_config): """sharding for feedforward""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if self.hidden_dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_dim' must be a multiple of the" "num of model parallel, but got the hidden_dim is {} and the num of model " "parallel is {}.".format(self.hidden_dim, mp)) if self.dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'dim' must be a multiple of the num of " "model parallel, but got the dim is {} and the num of model parallel is {}." .format(self.dim, mp)) self.w1.shard(((dp, 1), (mp, 1)), strategy_activation=((dp, mp),)) self.w1.activation.shard(((dp, mp),)) self.w2.shard(((dp, mp), (1, mp))) self.w3.shard(((dp, 1), (mp, 1))) self.mul.shard(((dp, mp), (dp, mp))) pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints/llama_tokenizer-checkpoint.py000066400000000000000000000263521476650141400344700ustar00rootroot00000000000000# coding=utf-8 # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA tokenizer APIs.""" import os from shutil import copyfile from typing import Any, Dict, List, Optional import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class LlamaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['llama'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text, **kwargs): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output llama_transformer-checkpoint.py000066400000000000000000001334451476650141400347430ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/llama/.ipynb_checkpoints# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA transformer Layer's APIs.""" from typing import Tuple, Optional import math import numpy as np try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import nn, ops import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.common.initializer import Normal try: from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindformers.models.llama.llama_layer import LlamaFeedForward, LlamaRMSNorm, LlamaRotaryEmbedding from mindformers.modules.layers import _check_input_dtype, Linear from mindformers.modules.transformer import TransformerOpParallelConfig class LLamaAttention(nn.Cell): r""" This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, n_kv_heads: Optional[int] = None, mup_width_base=-1, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.n_rep = self.n_head // self.n_kv_head self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) if self.n_kv_head % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of " "'parallel_config.model_parallel', but got the n_kv_head is {} " "and the parallel_config.model_parallel is {}." .format(self.n_kv_head, parallel_config.model_parallel)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() sigma = 0.01 if mup_width_base > 0: ratio = self.hidden_size / mup_width_base sigma = sigma / math.sqrt(ratio) self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wq = Linear(self.hidden_size, self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wk = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wv = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.wq.shard(((dp, 1), (mp, 1))) self.wk.shard(((dp, 1), (mp, 1))) self.wv.shard(((dp, 1), (mp, 1))) self.wo.shard(((dp, mp), (1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.wq(x), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.wk(x), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.wv(x), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] key = self._repeat_kv(key, self.n_rep) value = self._repeat_kv(value, self.n_rep) # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _repeat_kv(self, x, rep): if rep == 1: return x bs, n_kv_head, seqlen, head_dim = x.shape x = self.reshape(x, (bs * n_kv_head, 1, seqlen, head_dim)) x = self.tile_kv(x, (1, rep, 1, 1)) x = self.reshape(x, (bs, n_kv_head * rep, seqlen, head_dim)) return x def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaMLAAttention(nn.Cell): def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, mla_multiplier: int = 4, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.lora_size = self.head_dim * mla_multiplier self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.q_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.q_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.q_proj_down.shard(((dp, 1), (mp, 1))) self.q_proj_up.shard(((dp, 1), (mp, 1))) self.k_proj_down.shard(((dp, 1), (mp, 1))) self.k_proj_up.shard(((dp, 1), (mp, 1))) self.v_proj_down.shard(((dp, 1), (mp, 1))) self.v_proj_up.shard(((dp, 1), (mp, 1))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.q_proj_up(self.q_proj_down(x)), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.k_proj_up(self.k_proj_down(x)), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.v_proj_up(self.v_proj_down(x)), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaDecodeLayer(nn.Cell): r""" Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, layer_id, dim: int = 512, n_heads: int = 8, multiple_of: int = 256, n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, norm_eps: float = 1e-5, compute_dtype=mstype.float16, layernorm_compute_dtype=mstype.float32, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, mup_width_base=-1, parallel_config=TransformerOpParallelConfig()): super().__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size self.compute_in_2d = compute_in_2d self.seq_length = seq_length self.layer_id = layer_id self.hidden_size = dim self.n_head = n_heads self.head_dim = self.hidden_size // self.n_head self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.dtype = compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.key_past = None self.value_past = None self.reshape = P.Reshape() self.add = P.Add() self.attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) if mla_multiplier is None: self.attention = LLamaAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, n_kv_heads=n_kv_heads, mup_width_base=mup_width_base, compute_dtype=compute_dtype, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) else: self.attention = LLamaMLAAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, compute_dtype=compute_dtype, mla_multiplier=mla_multiplier, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) self.feed_forward = LlamaFeedForward(dim=self.hidden_size, hidden_dim=4 * self.hidden_size, multiple_of=multiple_of, ffn_dim_multiplier=ffn_dim_multiplier, mup_width_base=mup_width_base, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.feed_forward.shard(parallel_config) if self.compute_in_2d: self.add.shard(((dp, 1), (dp, 1))) self.attention_norm.shard((dp, 1)) self.ffn_norm.shard((dp, 1)) else: self.add.shard(((dp, 1, 1), (dp, 1, 1))) self.attention_norm.shard((dp, 1, 1)) self.ffn_norm.shard((dp, 1, 1)) self.feed_forward.mul.shard(((dp, 1, mp), (dp, 1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: if self.compute_in_2d: self.add.shard(((dp * mp, 1), (dp * mp, 1))) self.attention_norm.shard((dp * mp, 1)) self.ffn_norm.shard((dp * mp, 1)) else: self.add.shard(((dp, mp, 1), (dp, mp, 1))) self.attention_norm.shard((dp, mp, 1)) self.ffn_norm.shard((dp, mp, 1)) self.feed_forward.w2.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if self.use_past: kv_shape = (batch_size, self.n_kv_head, seq_length, self.head_dim) self.key_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="value_past") self.ones = P.Ones() self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.assign_past = P.Assign().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.mul_past.shard(((dp, mp, 1, 1), (1,))) self.assign_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) def construct(self, x, freqs_cis, mask=None, init_reset=True, batch_valid_length=None): """ Forward of transformer block. """ self._check_input(x, freqs_cis, mask, init_reset, batch_valid_length) # [bs, seq/1, hidden_dim] (first) [bs * seq/1, hidden_dim] (others) if self.compute_in_2d and x.ndim != 2: x = self.reshape(x, (-1, x.shape[-1])) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] input_x = self.attention_norm(x) key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_past(self.key_past, self.mul_past(self.key_past, self.cast(init_reset, self.dtype))) self.assign_past(self.value_past, self.mul_past(self.value_past, self.cast(init_reset, self.dtype))) key_reset = self.key_past value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] h, layer_present = self.attention(input_x, freqs_cis, mask, self.key_past, self.value_past, batch_valid_length) h = self.add(x, h) ffn_norm = self.ffn_norm(h) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.feed_forward(ffn_norm) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_past(self.key_past, key_present) self.assign_past(self.value_past, value_present) key_update = self.key_past value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order ffn_out = ops.depend(ffn_out, value_update) ffn_out = ops.depend(ffn_out, key_update) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] out = self.add(h, ffn_out) return out, layer_present def _check_input(self, x, freqs_cis, mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype( x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name) freqs_cos, freqs_sin, swap_mask = freqs_cis _check_input_dtype(freqs_cos.dtype, "freqs_cos", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(freqs_sin.dtype, "freqs_sin", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(swap_mask.dtype, "swap_mask", [mstype.float32, mstype.float16], self.cls_name) if mask is not None: _check_input_dtype(mask.dtype, "input_mask", [mstype.float32, mstype.float16], self.cls_name) if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): bs = freqs_cis[0].shape[0] batch_valid_length = self.ones((bs, 1), mstype.int32) _check_input_dtype(init_reset.dtype, "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(batch_valid_length.dtype, "batch_valid_length", [mstype.int32], self.cls_name) return True pengcheng.mind-mindformer/mindformers/models/llama/__init__.py000066400000000000000000000017321476650141400251240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for llama""" from .llama_config import LlamaConfig from .llama import LlamaForCausalLM, LlamaModel from .llama_tokenizer import LlamaTokenizer from .llama_processor import LlamaProcessor __all__ = ['LlamaConfig', 'LlamaModel', 'LlamaForCausalLM', 'LlamaTokenizer', 'LlamaProcessor'] pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/000077500000000000000000000000001476650141400252205ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/__init__.cpython-39.pyc000066400000000000000000000007421476650141400314130ustar00rootroot00000000000000a ?cf@sDdZddlmZddlmZmZddlmZddlm Z gdZ dS)zThe export function for llama) LlamaConfig)LlamaForCausalLM LlamaModel)LlamaTokenizer)LlamaProcessor)rrrrrN) __doc__Z llama_configrllamarrZllama_tokenizerrZllama_processorr__all__r r L/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/__init__.pys    pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama.cpython-39.pyc000066400000000000000000000251551476650141400307470ustar00rootroot00000000000000a rfB@sdZddlZddlZddlmmZzddlm Z Wne yTddlm Z Yn0ddl m Z mZddlmZddlmZddlmZmZzddlmZd ZWne yd ZYn0dd lmZdd l mZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddgZ8d"ddZ9Gd dde"Z:e+;e*j<Gd!dde"Z=dS)#zLLaMA models' APIs.N) Validator)Tensornn) ParallelMode) operations)_get_parallel_mode_is_sharding_propagation)FlashAttentionTF)Normal) lazy_inline)CrossEntropyLoss)MindFormerBook) BaseModel)Linear) _check_config) AttentionMask)MindFormerModuleTypeMindFormerRegister) LlamaConfig)LlamaEmbedding LlamaRMSNormprecompute_freqs_cis)LLamaDecodeLayer) cell_reuse)logger LlamaModelLlamaForCausalLMc Cstt|d|jd}t|trbt||jkrDtdt|dt|||jd}||}n$t|trr|}ntdt |dt||||jd} | |_tt|d|j d} t|j t r|j r|s| n|j j r|s|j |j j ddS)a Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. rzThe length of `offset` z@ do not match `pipeline stage` {parallel_config.pipeline_stage}.z1`offset` must be `int` of list of `int`, but got .)recompute_slice_activationN)maxintpipeline_stage isinstancelistlen ValueErrormin TypeErrortypegradient_aggregation_group recomputeboolr!) layerlayer_idoffsetparallel_configZn_layersselect_recomputepp_disiZ offset_layerpp_iddisr8I/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama.pylayer_compute_dtype8s&        r:csDeZdZdZedZd edfdd Zd e dd d Z Z S) raA Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer llamaNconfigcstj|ddt|j|js&|jr2t|j|j|_ |j |_ |j |_ d|_ |j|_|j o`t|_ |j rvtdn|j rtdt|j|j|j|j|j|jd\|_|_|_t|j|jjd|j|_tdg|jd|_ td g|jd|_!t"#|_$t"%|_&t"'|_(t")|_*t"+|_,t"-|_.t"/|_0t"1|_2t3|j4|j|j5d |_6t|j7g|jd|_7t")|_8t9:|_;t<|j D]}t=|j|j||j |j|j|j>|j?|j@|jA|jB|j|jC|jD|j|jE|j5|j|j |jF|jG|jH|jI|jd }tJ|||jK|j|j |jjLjMd |j;N|qtO|j|jB|jCd |_P|jjQ}tRtSjTfvr\tUsHd|j6_V|jjVdkr|jjVd|jP_V|j6W|j|j(Wd|j,Wd|ddff|j*W|dddfdf|j8W|ddfdf|j.W|ddff|j0W|dfdf|j2W|dfdf|jFr6|jPW|dfn|jPW|ddf|jrtXY|j$ddd}ttX(||jddftZj[|_>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) r;Nr<c stt|j|ddt|j|j|_|j|_t|_ t |_ t |_ t|_t|_t|_t|d|_|j}|jdkr|j|j}||}t|j|jd|j|jt|ddd|_t|jd|_ |jj!}|jj"}t#t$j%fvrt&s|j '|d ff|j'|d fd f|j'|d f|d ff|j'|d fd f|jj(rh|jj'|d fd fd n|jj'|d f|d ffd |jj)d kr|jj)d |j_)|*|dS) NTr>r<rF)sigmamean) in_channels out_channelshas_biasrJrD weight_initrCrr8rr)strategy_matmul)+rUrrVrr2ignore_token_idrZrirjrkrlrm StridedSliceslicervrwrpmulAddaddrmodelrRrLr^rrzrJrDr lm_headr lossrmodel_parallelrrrrr vocab_emb_dpr$load_checkpoint)rr=rratiormprr8r9rVsH            zLlamaForCausalLM.__init__cKsdt|tjiS)N input_ids)rrr)rrkwargsr8r8r9prepare_inputs_for_generation;s z.LlamaForCausalLM.prepare_inputs_for_generationTc Cs|j\} } ||d| | dfd} || |||} || } ||| |jtj}|durr||d| | fd}nH|j dkr|j r||d| | fd}||||j tj}| ||}|| tj} | j dkr| | d| jdf} | |d}| |d}|| ||}|S) zLlamaForCausalLM forward.)rrrrN)rrrrT)rT)rrrrrmrwrZrfloat32ndimtrainingrrrkr)rrlabelsr position_idsattention_maskZ input_embedsrrbszseqlenrrlogitsrZ label_maskrr8r8r9rAs&       zLlamaForCausalLM.construct)N)NNNNNTN) rrrrr rrr rrVrrrr8r8rr9rs *)F)>rmathnumpyrZmindspore.common.dtypecommonr@rZmindspore._checkparamr ImportError _checkparam mindsporerrZmindspore.contextr mindspore.opsrrimindspore.parallel._utilsrrZ"mindspore.nn.layer.flash_attentionr r\mindspore.common.initializerr r Zmindformers.core.loss.lossr mindformers.mindformer_bookr mindformers.models.base_modelrmindformers.modules.layersr2mindformers.modules.transformer.op_parallel_configrZ+mindformers.modules.transformer.transformerrZ#mindformers.tools.register.registerrr llama_configrZ llama_layerrrrZllama_transformerrutilsrZ tools.loggerr__all__r:rregisterMODELSrr8r8r8r9sH                   % pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama_config.cpython-39.pyc000066400000000000000000000055461476650141400322760ustar00rootroot00000000000000a &nfH@szdZddlmZddlmZmZddlmZmZddl m Z ddl m Z dd l mZd gZeejGd d d e Zd S) zLlama Config API.)Optional)default_transformer_configTransformerOpParallelConfig)MindFormerRegisterMindFormerModuleType)convert_mstype) BaseConfig)MindFormerBook LlamaConfigc(seZdZdZedZdddddddd d d d dd dd d dddd edddddddddddddddddf&eeeeeeee ee ee ee eeeee e e e e e e ee e e ee e e eee e ee e e d&fdd ZZS)r z: LLaMA config class which defines the model size. llamaii i}Ngh㈵>rifloat16float32FNonerg?iTg{Gz?)& batch_size seq_length hidden_size num_layers num_heads vocab_size multiple_of n_kv_headsffn_dim_multipliermla_multiplier rms_norm_eps bos_token_id eos_token_id pad_token_idignore_token_id compute_dtypelayernorm_compute_typesoftmax_compute_type rotary_dtypeparam_init_typeparallel_configuse_pastpretrain_seqlen extend_method compute_in_2duse_flash_attentionoffsetuse_past_shardcheckpoint_name_or_pathrepetition_penaltymax_decode_lengthtop_ktop_p do_samplemup_width_base embed_scale depth_scaleinit_stdc'( stt|jfi|'||_||_||_||_||_||_||_ ||_ | |_ | |_ | |_ t||_t||_t||_t||_t||_||_||_| |_| |_||_||_||_||_||_||_||_||_||_ ||_!||_"| |_#|!|_$|"|_%|#|_&|$|_'|%|_(|&|_)dS)N)*superr __init__rrrrrrrrrr r!rr*r'r(r)r&r+r3r"r#r$r%r,r-r.r/r0r1r2r4r5r6r7r8r9r:r;r<)(selfrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<kwargs __class__P/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_config.pyr>$sN(     zLlamaConfig.__init__)__name__ __module__ __qualname____doc__r get_config_support_list _support_listrintrfloatstrrboolr> __classcell__rCrCrArDr s N)rHtypingrZ+mindformers.modules.transformer.transformerrrmindformers.tools.registerrrutilsr base_configr mindformer_bookr __all__registerCONFIGr rCrCrCrDs     pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama_layer.cpython-39.pyc000066400000000000000000000313421476650141400321360ustar00rootroot00000000000000a rfV=@sdZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m mZddlmZddlmZdd lmZdd lmZzdd lmZWneyddlmZYn0dd l mZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*GdddeZ+GdddeZ,ddZ-dej.de+j/j0fe1e1e2dddZ3GdddeZ4Gdd d eZ5Gd!d"d"e jZ6Gd#d$d$eZ7dS)%zLLaMA Model Layers' APIs.)EnumN)Tensor) Parameter)nn) operations) functional)Cell)Normal) Validator)log) initializer)_get_parallel_mode) ParallelMode)Linear_check_input_dtype_args_type_validator_check_valid_value_checks)_LogActionOncec@seZdZdZdZdZdZdS)SeqExtendMethodzEStores the acceptable string identifiers for seq length extend methodPINTKNoneN)__name__ __module__ __qualname____doc__rrNONErrO/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_layer.pyr+srcs0eZdZdZfddZddZddZZS) LlamaSiLUz A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). cs"tt|_t|_dSN)super__init__rSigmoidsigmoidPMulmul)self __class__rrr"<s  zLlamaSiLU.__init__cCs*|jj||j|d|dfdS)Nr)r$shardr')r(strategyrrrr+AszLlamaSiLU.shardcCs||||Sr )r'r$)r(xrrr constructEszLlamaSiLU.construct)rrrrr"r+r. __classcell__rrr)rr2s rcCsHtj|d|dftjd}tj|dtjd}t||g| |ggS)z Swap matrixdtype)npzerosfloat32identityblock)head_dimZ zero_blockZid_blockrrr get_swap_maskIsr9g@i)dimendthetacCsd}|tjjkr ||kr ||}|tjjkr4||9}td|dd|dtj}d|||}|tjjkrtd||d|tj} ntd|dtj} t | |}tj ||fdd} t | } t | } t | |d} t | |d} t|} t | |d} | | | fS) z< Precompute of freqs and mask for rotary embedding. g?rr0N)axisr1)rrvaluerr3arangeastyper5router concatenatecossinrr9)r:r;r<r2pretrain_seqlen extend_methodratioZ freqs_basefreqstZemb freqs_cos freqs_sin swap_maskrrrprecompute_freqs_cisPs&  "        rOcsHeZdZdZdejffdd ZddZeeddd Z d d Z Z S) LlamaRotaryEmbeddinga Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. csFtjdd||_||_t|_t|_t |_ t |_ dS)NF) auto_prefix) r!r"r8r2r%Addadd BatchMatMulbmm_swapr&r'Castcast)r(r8 compute_dtyper)rrr"s   zLlamaRotaryEmbedding.__init__cCs|||}|Sr )rV)r(r-rNrrr rotate_halfs z LlamaRotaryEmbedding.rotate_half)xqxkc Cs|j}|||j}|||j}|\}}}|||||||||}|||||||||} |||}|| |} || fS)z%Forward of rotary position embedding.)r2rXrTr'rZ) r(r[r\ freqs_cis original_typerLrMrNZxq_outZxk_outrrrr.s   zLlamaRotaryEmbedding.constructcCs@|j||f|j|df|j||ddddffdS)Nr=r=rr=)rTr+rVr'r(Z strategy_inrrrr+szLlamaRotaryEmbedding.shard) rrrrmstyper5r"rZrr.r+r/rrr)rrPrs  rPcs`eZdZdZeedeejfvde e j e j de j dffdd Zdd Zd d ZZS) LlamaEmbeddinga Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). Embeddingm_loggerkey no_warning)vocab_table_sizeembedding_sizenormalcsFt||_||_tt||j|jg|dddd|_t|_ dS)Nr1embedding_weightT)nameparallel_optimizer) r!r"rhrirr rkr%Gathergather)r(rhriparam_init_type param_initr)rrr"s zLlamaEmbedding.__init__cCs2tt|dtjtjg|j||j|d}|S)zForward of vocab embedding. input_idsr) rFr2raint32int64cls_namerork)r(rroutputrrrr.szLlamaEmbedding.constructcCs|j}|j}|jr:|jd|dfftd|dnV|j|dkr`td|jd|d|j|df|dfftd|d |d d S) zsharding for embeddingr_r=zUsing z( data parallel for the embedding lookup.rz The vocab size of the embedding z6 must be a multiple of parallel_config.model_parallel .z data parallel and z) model parallel for the embedding lookup.N) data_parallelmodel_parallel vocab_emb_dpror+loggerinforh ValueErrorr(parallel_configdpmprrrr+s  zLlamaEmbedding.shard)rrrrrr|r r STAND_ALONErr check_positive_intrar5r"r.r+r/rrr)rrbs  rbcs@eZdZdZdejffdd ZddZddZd d Z Z S) LlamaRMSNorma A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. gư>cstt|||_ttd|ftjddd|_t |_ t j dd|_ t |_t |_t |_t |_t |_||_dS)Nonesr1F)rmT) keep_dims)r!rr"epsrr rar5weightr%Squaresquare ReduceMeanmeanrSrTRsqrtrsqrtr&r'mul2rWrX compute_type)r(r:rrr)rrr"s      zLlamaRMSNorm.__init__cCsV||}||d}|||j}||}||tj}||tj}|||S)Nr>) rrrTrrrXrafloat16r')r(r- norm_factorrrr_norms   zLlamaRMSNorm._normcCsX|j}|||j}||}||tj}||jtj}|||}|||}|S)zForward of RMSNorm.)r2rXrrrarrr)r(r-r^rwrrrrr.s   zLlamaRMSNorm.constructcCs^|j|f|j|f|j|f|j|df|j||f|j|dfdS)z+Parallel strategy configuratiuon interface.r)r=N)rr+rrrTr'rr`rrrr+s zLlamaRMSNorm.shard) rrrrrar5r"rr.r+r/rrr)rrs     rc seZdZdZeedeejfvde e j e j e j e e je jgde e je jgddeddde je jffdd Zd d Zd d ZZS)LlamaFeedForwardaE LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. FeedForwardrd)r: hidden_dim multiple_ofrYrpNr>{Gz?c s"t|dus(t|ts8t|tjs8td|d|durPt|d|}td|d}|||d|}||_ ||_ ||_ ||_ t |_t |_|} |dkr||} | t| } t|||d|| t| dd d |_t||d|| t| dd d |_t||d|| t| dd d |_dS) NzNFor FeedForward cell, the hidden_act should str type or nn.Cell type, but got rxrr0r=rF)sigmar) in_channels out_channels activationhas_biasrYrp weight_init)rrrrYrpr)r!r" isinstancestr issubclassrr TypeErrorintr2 hidden_actr:rr%r&r'rWrXmathsqrtrr w1w2w3) r(r:rrrffn_dim_multipliermup_width_baseinit_stdrYrprrIr)rrr",sV       zLlamaFeedForward.__init__cCsZtt|dtjtjg|j|||j}||}| |}| ||}| |}|S)z"Forward process of the FeedForwardr-) rrsr2rar5rrvrXrrr'r)r(r-gatehiddenrwrrrr.ls    zLlamaFeedForward.constructcCs|j}|j}|j|dkr,td|j||j|dkrLtd|j||jj|df|dff||ffd|jj||ff|j ||fd|ff|j |df|dff|j ||f||ffdS)zsharding for feedforwardrzFor 'FeedForward', the class variable 'hidden_dim' must be a multiple of thenum of model parallel, but got the hidden_dim is {} and the num of model parallel is {}.zFor 'FeedForward', the class variable 'dim' must be a multiple of the num of model parallel, but got the dim is {} and the num of model parallel is {}.r=)strategy_activationN) ryrzrr~formatr:rr+rrrr'rrrrr+ws"zLlamaFeedForward.shard)rrrrrr|r rrrr rrrar5rrr"r.r+r/rrr)rrs.    7 r)8renumrrnumpyr3mindspore.common.tensorrmindspore.common.parameterr mindsporermindspore.common.dtypecommonr2ra mindspore.opsrr%rrsmindspore.nn.cellrmindspore.common.initializerr mindspore._checkparamr ImportError _checkparamr r|r mindspore.parallel._utilsr mindspore.contextrmindformers.modules.layersrrrrmindformers.tools.loggerrrrr9r5rr@rfloatrOrPrbrrrrrrsJ                "589pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama_processor.cpython-39.pyc000066400000000000000000000031611476650141400330370ustar00rootroot00000000000000a ?cf @s^dZddlmZddlmZddlmZddlmZm Z dgZ e e j Gd ddeZ d S) z LlamaProcessor )MindFormerBook) BaseTokenizer) BaseProcessor)MindFormerRegisterMindFormerModuleTypeLlamaProcessorcs8eZdZdZedZd fdd Zd d d ZZ S) r zV Llama processor, consists of a tokenizer (BaseTokenizer) for text input. llamaN max_lengthmscstt|j||||ddS)N) tokenizerr paddingreturn_tensors)superr __init__)selfrr rr __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_processor.pyr"s  zLlamaProcessor.__init__cCsli}|durh|jrht|jts4tdt|jdt|trD|g}|j||j|j|jdd}||d<|S)z call functionNz;tokenizer should inherited from the BaseTokenizer, but got .)rr r input_idstext) r isinstancer TypeErrortypestrrr r)r text_input image_inputoutput text_outputrrr__call__+s     zLlamaProcessor.__call__)Nr r r )NN) __name__ __module__ __qualname____doc__rget_processor_support_list _support_listrr# __classcell__rrrrr s   N)r'mindformers.mindformer_bookrbase_tokenizerrbase_processorrZtools.registerrr__all__register PROCESSORr rrrrs    pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama_tokenizer.cpython-39.pyc000066400000000000000000000226041476650141400330350ustar00rootroot00000000000000a rf,@sdZddlZddlmZddlmZmZmZmZddl Z ddl m Z ddl mZddlmZmZddlmZmZd gZd d iZeejGd d d eZdS) zLLaMA tokenizer APIs.N)copyfile)AnyDictListOptional)logger)MindFormerBook) Tokenizer AddedToken)MindFormerRegisterMindFormerModuleTypeLlamaTokenizer vocab_fileztokenizer.modelcseZdZdZeZddgZdgZe dZ d*e e e efd fdd ZddZddZeddZddZddZddZddZddZd+d d!Zd,d"d#Zd-eee eeeeed$fd%d& Zd.eee eeeed'd(d)ZZ S)/r aj Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. input_idsattention_maskztokenizer_config.jsonllamaNTF)sp_model_kwargsc s|dur in||_t|tr*t|dddn|}t|trFt|dddn|}t|trbt|dddn|}t|tr~t|dddn|}tjf|||||j| d| ||_||_||_t j fi|j|_ |j |dS)NF)lstriprstrip) bos_token eos_token unk_token pad_tokenrclean_up_tokenization_spaces) r isinstancestrr super__init__r add_bos_token add_eos_tokenspmSentencePieceProcessorsp_modelLoad) selfrrrrrrr"r#rkwargs __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_tokenizer.pyr!Qs(  zLlamaTokenizer.__init__cCs|j}d|d<|S)Nr&)__dict__copy)r(stater,r,r- __getstate__rs zLlamaTokenizer.__getstate__cCs,||_tjfi|j|_|j|jdS)N)r.r$r%rr&r'r)r(dr,r,r- __setstate__wszLlamaTokenizer.__setstate__cCs |jS)zReturns vocab size)r&Zget_piece_sizer(r,r,r- vocab_size|szLlamaTokenizer.vocab_sizecs(fddtjD}|j|S)zReturns vocab as a dictcsi|]}||qSr,)convert_ids_to_tokens).0ir4r,r- z,LlamaTokenizer.get_vocab..)ranger5updateadded_tokens_encoder)r(vocabr,r4r- get_vocabs zLlamaTokenizer.get_vocabcKs|jj|tdS)zReturns a tokenized string.)out_type)r&encoder)r(textr)r,r,r- _tokenizeszLlamaTokenizer._tokenizecCs |j|S)z0Converts a token (str) in an id using the vocab.)r&Z piece_to_id)r(tokenr,r,r-_convert_token_to_idsz#LlamaTokenizer._convert_token_to_idcCs|j|}|S)z=Converts an index (integer) in a token (str) using the vocab.)r&Z IdToPiece)r(indexrDr,r,r-_convert_id_to_tokens z#LlamaTokenizer._convert_id_to_tokencCs|g}d}d}t|D]R\}}||jvrX|s:|dkr:|d7}||j||7}d}g}q||d}q||j|7}|S)z:Converts a sequence of tokens (string) in a single string.Fr T) enumerateall_special_tokensr&decodeappend)r(tokensZcurrent_sub_tokens out_stringZprev_is_specialr8rDr,r,r-convert_tokens_to_strings   z'LlamaTokenizer.convert_tokens_to_stringcCstj|std|dStj||r0|dndtd}tj|jtj|krttj |jrtt |j|nLtj |jst |d$}|j }||Wdn1s0Y|S)a Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. z*Vocabulary path (%s) should be a directoryN-rHrwb)ospathisdirrerrorjoinVOCAB_FILES_NAMESabspathrisfileropenr&Zserialized_model_protowrite)r(save_directoryfilename_prefixZout_vocab_filefiZcontent_spiece_modelr,r,r-save_vocabularys  (  (zLlamaTokenizer.save_vocabularycCsL|jr|jgng}|jr |jgng}|||}|durH||||}|S)z5Insert the special tokens to the input_ids. CurrentlyN)r" bos_token_idr# eos_token_idr( token_ids_0 token_ids_1rarboutputr,r,r- build_inputs_with_special_tokenss  z/LlamaTokenizer.build_inputs_with_special_tokens)rdrealready_has_special_tokensreturncs|rtj||ddS|jr"dgng}|jr2dgng}|durT|dgt||S|dgt|||dgt||S)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rdrerhNr)r get_special_tokens_maskr"r#len)r(rdrerhrarbr*r,r-rks(  z&LlamaTokenizer.get_special_tokens_mask)rdrericCs`|jr|jgng}|jr |jgng}dgt|||}|dur\|dgt|||7}|S)a Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). rNrj)r"rar#rbrlrcr,r,r-$create_token_type_ids_from_sequencess z3LlamaTokenizer.create_token_type_ids_from_sequences)rrrrNTFF)N)N)NF)N)!__name__ __module__ __qualname____doc__rXvocab_files_namesmodel_input_names FILE_LISTrget_tokenizer_support_list _support_listrrrrr!r1r3propertyr5r?rCrErGrPr`rgrintboolrkrm __classcell__r,r,r*r-r 'sP# !    ))rqrSshutilrtypingrrrrZ sentencepiecer$mindformers.toolsrmindformers.mindformer_bookr!mindformers.models.base_tokenizerr r mindformers.tools.registerr r __all__rXregister TOKENIZERr r,r,r,r-s    pengcheng.mind-mindformer/mindformers/models/llama/__pycache__/llama_transformer.cpython-39.pyc000066400000000000000000000613011476650141400333620ustar00rootroot00000000000000a זnf@sXdZddlmZmZddlZddlZzddlmZWne yRddlm ZYn0ddl m Z m Z ddlmmZddlmZddlmZddlmZdd lmZdd lmZmZdd lmZzdd l m!Z!d Z"Wne ydZ"Yn0ddl#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+Gddde j,Z-Gddde j,Z.Gddde j,Z/dS)zLLaMA transformer Layer's APIs.)TupleOptionalN) Validator)nnops)Tensor) Parameter) ParallelMode) operations)_get_parallel_mode_is_sharding_propagation)Normal)FlashAttentionTF)LlamaFeedForward LlamaRMSNormLlamaRotaryEmbedding)_check_input_dtypeLinear)TransformerOpParallelConfigcseZdZdZdddddejejejejddddefeee edfd d Z de e e e fd d d Z ddZddZddZddZZS)LLamaAttentiona This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). N{Gz?F)dimn_heads n_kv_headsc st||_||_||_|||_|dur2|n||_|j|j|_||_| |_ d|_ | |_ ||_ | ojt |_|j|jdkrtd|j|j|j|jdkrtd|j|jtdt|j|d|_t|_t|_t|_t|_tjdd|_t|_ t!|_"t#$%| |_&t'|_(t'|_)t*|_+|}|dkrn|j|}|t|}t,|j| |_-t.|j|jd|t/|dd | d |_0t.|j|jd|t/|dd | d |_1t.|j|j|jd|t/|dd | d |_2t.|j|j|jd|t/|dd | d |_3|j4}|j}t5t6j7fvr4t8s|j9|d |d ff|j9||d d ff|j9||d d f||d d ff|j9||d d f||d d ff|j 9||d d fd f|j"9|d d d f||d d ff|j&j&9||d d ff|j+9||d d d ff|j-9||d d f|j19|d f|d ff|j29|d f|d ff|j39|d f|d ff|j09||fd |ff|j:r|j r|j0j9||fd |ff||d ffd|j;j<r|j-;|j+;|j;|j ;|j";|j);|j&j&;|j;|jrt=|j||d| t>j?kd|_@t5t6j7fvrLt8s|j@9||d d f||d d f||d d f|d d fd f|j;j<r|j@;|j rtAB|d d d}ttAC||d d ft>jD|_EtF9|d d ff|_Gt!9|d d d f|d d d ff|_HtI9|d d f|d d ff|_JtK9|d d f|d d ff|_Lt9|d d d f|d d d ff|_M|r|jH9||d d f||d d ff|jM9||d d f|d d d ffdS)NTrFor 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple of 'n_head', but got the hidden_size is {} and the n_head is {}.zFor 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of 'parallel_config.model_parallel', but got the n_kv_head is {} and the parallel_config.model_parallel is {}.?dtype transpose_bF)sigmamean) in_channels out_channelshas_bias compute_dtype weight_initparam_init_type)r'r(r)r*out_strategy_matmuldpmpnext_block_numhigh_precisionr)Nsuper__init__ seq_length hidden_sizen_headhead_dim n_kv_headn_repr softmax_dtypeis_first_iterationuse_past compute_in_2dFLASHATTENTION_VALIDuse_flash_attention ValueErrorformatmodel_parallelrmathsqrtinv_norm_factorPReshapereshape Transpose transposemerger_head_transpose BatchMatMul batch_matmulbatch_matmul_q_kMulmulAddaddrSoftmaxto_floatsoftmaxCastcast cast_attnTiletile_kvrapply_rotary_embrr wowqwkwv data_parallelr r AUTO_PARALLELr sharduse_seq_parallel recomputeselect_recomputermstypefloat32flash_attentionnparangetileint32range ExpandDims expand_dimsadd_pastEqualequalLesslessmul_past)self batch_sizer6rrrmup_width_baseinit_stdr(softmax_compute_dtype rotary_dtyper*r>rAr?use_past_shardparallel_configr#ratior0r1 seq_range __class__r,U/home/ma-user/work/mindformers-baichuan/mindformers/models/llama/llama_transformer.pyr5is                       &          4  $  $ zLLamaAttention.__init__x freqs_ciscCs8|j}||d|jdf}||||j}||||j} ||||j} ||d||j|j |j f}|| d||j|j |j f} || d||j|j |j f} | |d}| | d} | | d} | || |\}} | } | } |jr|jrP||j|ddd|j} || || d} || || d} nr|d}||d}||j||j} || || d}|| || d}|||} |||} | } | } | | f}|| |j} || |j} |jr ||| | |}||}n||| | |}||}|||}||fSz)Forward process of the MultiHeadAttentionrrr+r+r)rr+r+)r rJshaperYr_r`ra!_get_seq_length_under_incrementalr6r8r9r:rLr]r>r=rvroviewastyperwrqrtrr _repeat_kvr;rArj _merge_heads_attnr^rxrrmaskkey_past value_pastbatch_valid_length ori_dtypequerykeyvalue key_present value_presentvalid_length_vector valid_length current_key current_value layer_present attentionoutputr,r,r constructs^         zLLamaAttention.constructcCsb|dkr |S|j\}}}}||||d||f}||d|ddf}|||||||f}|S)Nr+)rrJr\)rxrrepbsr:seqlenr9r,r,rr4szLLamaAttention._repeat_kvcCs|jr|jsdS|SzxReturn the length of the tensor. For the incremental prediction, the seq length for the input is 1. r+r>r=rxlengthr,r,rr=s z0LLamaAttention._get_seq_length_under_incrementalcCsP||d}|j}|jr.d|d|df}n|d|ddf}|||}|Sz convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output rrrr+rMrr?rJrxrx_shape new_shapex_merger,r,rrEs  zLLamaAttention._merge_headsc Cs^|||}|||j}|||}||||j}||||j |}| |}|Sau Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores rPrRrGrTrWrZr<rOrYr r rxrrrrscoreattention_probsweighted_valuesattention_merger,r,rr\s   zLLamaAttention._attn)NNNN)__name__ __module__ __qualname____doc__rhfloat16ririntrr5rrrrrrr __classcell__r,r,rrr-s8> B rc s|eZdZdddejejejejddddef eeedfdd Zde e e e fd d d Z d d Z ddZ ddZZS)LLamaMLAAttentionrrF)rrmla_multipliercst||_||_||_|||_|j||_||_||_d|_ | |_ | |_ | oVt |_ |j|jdkr~td|j|jtdt|j|d|_t|_t|_t|_t|_tjdd|_t|_t|_ t!"#||_$t%|_&t%|_'t(|_)t*|j||_+t,|j|jd|| d|_-t,|j|jd|| d|_.t,|j|jd|| d|_/t,|j|jd|| d|_0t,|j|jd|| d|_1t,|j|jd|| d|_2t,|j|jd|| d|_3|j4}|j5}t6t7j8fvrt9s|j:|d |d ff|j:||d d ff|j:||d d f||d d ff|j:||d d f||d d ff|j:||d d fd f|j :|d d d f||d d ff|j$j$:||d d ff|j):||d d d ff|j+:||d d f|j-:|d f|d ff|j.:|d f|d ff|j/:|d f|d ff|j0:|d f|d ff|j1:|d f|d ff|j2:|d f|d ff|j;r|j r|j3j:||fd |ff||d ffd |j|j||d|t?j@kd |_At6t7j8fvr2t9sf|jA:||d d f||d d f||d d f|d d fd f|jr?r@rArBrCrrErFrGrHrIrJrKrLrMrNrOrPrQrRrSrTrrUrVrWrXrYrZr[r\rr]r q_proj_down q_proj_up k_proj_down k_proj_up v_proj_down v_proj_upr^rbrDr r rcr rdrerfrgrrhrirjrkrlrmrnrorprqrrrsrtrurvrw)rxryr6rrrr(r|r}r*r>rAr?r~rr0r1rrr,rr5xs                &          4  $  $ zLLamaMLAAttention.__init__NrcCs.|j}||d|jdf}|||||j}|||||j} ||| ||j} ||d| |j |j |j f}|| d| |j |j |j f} || d| |j |j |j f} ||d}|| d} || d} ||| |\}} | } | } |jr|jrb||j|ddd|j} || || d} || || d} nr|d}||d}||j||j} || || d}|| || d}|||} |||} | } | } | | f}|jr||| | |}||}n||| | |}||}|||}||fSr)r rJrrYrrrrrrrr6r8r9rLr]r>r=rvrorrrwrqrtrrrArjrrr^rr,r,rrsZ         zLLamaMLAAttention.constructcCs|jr|jsdS|Srrrr,r,rrCs z3LLamaMLAAttention._get_seq_length_under_incrementalcCsP||d}|j}|jr.d|d|df}n|d|ddf}|||}|Srrrr,r,rrKs  zLLamaMLAAttention._merge_headsc Cs^|||}|||j}|||}||||j}||||j |}| |}|Srrrr,r,rrbs   zLLamaMLAAttention._attn)NNNN)rrrrhrrirrr5rrrrrrrr,r,rrrws0 @rcseZdZdZdddddddejejejejejdddddd d efeeee ee ee ee d fd d Z dddZ ddZ ZS)LLamaDecodeLayera Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). rrNgh㈵>Frrr)rr multiple_ofrffn_dim_multiplierrnorm_epscs.t|s|rt|||_||_||_||_||_||_ |j|j |_ |durZ|n||_ | |_ d|_ ||_||_d|_d|_t|t|gtjd|_t|_t|_t|_t|j| | d|_t|j| | d|_ | durt!|||||||| ||||||||d|_"n&t#||||| | ||||||||d|_"t$|jd|j|| ||| |d|_%|j&}|j'}t(t)j*fvrt+sZ|j%,||jr|j,|df|dff|j,|df|j ,|df|j,|dfd fnt|j,|ddf|ddff|j,|ddf|j ,|ddf|j%j-,|d|f|d|ff|j,|ddfd f|j.rR|j rR|jr|j,||df||dff|j,||df|j ,||df|j,||dfd fnV|j,||df||dff|j,||df|j ,||df|j,||dfd f|j%j/j,||fd|ff||dffd |jr*||j ||j f}t0tt12||j d d |_t0tt12||j d d |_t3|_4t,|dddfd f|_5t6,|dddf|dddff|_7|r*|j5,||ddfd f|j7,||ddf||ddffdS)NTr) compute_type)ryr6rrrrzr{r(r|r}r*r>rAr?r~r)ryr6rrr(rr|r}r*r>rAr?r~rr)r hidden_dimrrrzr{r(r*r+)r+r-r)namer)8r4r5rcheck_positive_intryr?r6layer_idr7r8r9r:r r=r>rrrrErFrhr scale_depthrHrQ depth_mulrIrJrSrTrattention_normffn_normrrrr feed_forwardrbrDr r rcr rdrRrew2rrkzerosOnesonesrwAssign assign_past)rxryr6r layer_numrrrrrrrr(layernorm_compute_dtyper|r}r*r>rAr?r~rz depth_scaler{rr0r1Zkv_shaperr,rr5s          ( $zLLamaDecodeLayer.__init__Tc Cs|||||||jr8|jdkr8||d|jdf}||}d}d}|jr|jr||j | |j | ||j ||j | |j | ||j |j }|j }t||}t||}|||||j |j |\} } || |j} ||| } || } || } d} d}|jr`| \}}||j |||j ||j }|j } t||}t| |} t| | } t| |} || |j} || | }|| fS)z Forward of transformer block. rrN) _check_inputr?ndimrJrrr>r=rrrwrYr rrdependrrrrTrr)rxrrr init_resetrinput_x key_reset value_resethrrZffn_out value_update key_updaterroutr,r,rr=sF  ""            zLLamaDecodeLayer.constructc Cst|jdtjtjg|j|\}}}t|jdtjtjg|jt|jdtjtjg|jt|jdtjtjg|j|durt|jdtjtjg|j|jr t|tst|gtj }t|ts|dj d} | | dftj }t|jd tj g|jt|jd tj g|jd S) z Check inputsr freqs_cos freqs_sin swap_maskN input_maskrr+rrT) rr rhrircls_namer> isinstancerbool_rrrn) rxrrrrrrrrrr,r,rrps$   zLLamaDecodeLayer._check_input)NTN)rrrrrhrrirrrfloatr5rrrr,r,rrr}sB? 3r)0rtypingrrrEnumpyrkmindspore._checkparamr ImportError _checkparam mindsporerrmindspore.common.dtypecommonr rhmindspore.common.tensorrmindspore.common.parameterrmindspore.contextr mindspore.opsr rHmindspore.parallel._utilsr r mindspore.common.initializerr "mindspore.nn.layer.flash_attentionrr@Z$mindformers.models.llama.llama_layerrrrmindformers.modules.layersrrmindformers.modules.transformerrCellrrrr,r,r,rs<          Lpengcheng.mind-mindformer/mindformers/models/llama/convert_weight.py000066400000000000000000000155271476650141400264230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Convert llama weight. Support huggingface format and Meta format. """ import os import json import argparse from pathlib import Path import numpy as np import mindspore as ms from mindspore import ops def convert_meta_torch_ckpt(ckpt_dir, output_name, dtype=ms.float16): """Support convert meta weight splited.""" print(f"Trying to convert pytorch checkpoint in '{ckpt_dir}'.", flush=True) try: from torch import load except: raise ImportError(f"Failed to load pytorch checkpoint. Please make sure pytorch is available.") dic = { 'tok_embeddings.weight': 1, 'norm.weight': None, 'output.weight': 0, 'attention.wq.weight': 0, 'attention.wk.weight': 0, 'attention.wv.weight': 0, 'attention.wo.weight': 1, 'feed_forward.w1.weight': 0, 'feed_forward.w2.weight': 1, 'feed_forward.w3.weight': 0, 'attention_norm.weight': None, 'ffn_norm.weight': None, 'rope.freqs': None, } ckpt_paths = sorted(Path(ckpt_dir).glob("*.pth")) if not ckpt_paths: print(f"Do not find pytorch checkpoint in '{ckpt_dir}'.", flush=True) return False with open(Path(ckpt_dir) / "params.json", "r") as f: model_args = json.loads(f.read()) n_heads = model_args["n_heads"] dim = model_args["dim"] def permute(w): return w.view(n_heads, dim // n_heads // 2, 2, dim).transpose(1, 2).reshape(dim, dim) checkpoints = [] for i in range(len(ckpt_paths)): checkpoints.append(load(ckpt_paths[i], map_location="cpu")) ckpt_list = [] for name in checkpoints[0].keys(): for k, v in dic.items(): if k in name: if v is not None: value = np.concatenate( [checkpoints[i][name].numpy() for i in range(len(checkpoints))], v) else: value = checkpoints[0][name].numpy() if name == 'norm.weight': name = 'norm_out.weight' if name == 'output.weight': name = 'lm_head.weight' else: name = 'model.' + name if 'rope.freqs' in name: continue if 'wq' in name or 'wk' in name: value = permute(value) print(f'\rprocessing parameter: {name} {value.shape} ', end='', flush=True) ckpt_list.append({'name': name, 'data': ms.Tensor(value, dtype=dtype)}) ckpt_file = os.path.join(ckpt_dir, output_name) ms.save_checkpoint(ckpt_list, ckpt_file) print(f"\rConvert pytorch checkpoint finished, the mindspore checkpoint is saved in '{ckpt_file}'.", flush=True) return True def read_json(path): with open(path, "r") as f: return json.load(f) def name_replace(name: str): """replace hf param name to ms.""" name = name.replace('embed_tokens.weight', 'tok_embeddings.embedding_weight') name = name.replace('.self_attn.q_proj.', '.attention.wq.') name = name.replace('.self_attn.k_proj.', '.attention.wk.') name = name.replace('.self_attn.v_proj.', '.attention.wv.') name = name.replace('.self_attn.o_proj.', '.attention.wo.') name = name.replace('.mlp.gate_proj.', '.feed_forward.w1.') name = name.replace('.mlp.down_proj.', '.feed_forward.w2.') name = name.replace('.mlp.up_proj.', '.feed_forward.w3.') name = name.replace('.input_layernorm.', '.attention_norm.') name = name.replace('.post_attention_layernorm.', '.ffn_norm.') name = name.replace('.norm.', '.norm_out.') return name def convert_hf_ckpt(ckpt_dir, output_name, dtype=ms.float16): """convert hf weight to ms.""" print(f"Trying to convert huggingface checkpoint in '{ckpt_dir}'.", flush=True) try: from transformers import LlamaForCausalLM except: raise ImportError(f"Failed to load huggingface checkpoint. Please make sure transformers is available.") try: model_hf = LlamaForCausalLM.from_pretrained(ckpt_dir) # pylint: disable=W0703 except Exception as e: print(f"Do not find huggingface checkpoint in '{ckpt_dir}', Error {e.message}.", flush=True) return False ckpt_list = [] for name, value in model_hf.named_parameters(): name = name_replace(name) if name == 'norm.weight': name = 'norm_out.weight' if name[:7] == 'layers.': name = name[7:] value = value.detach().numpy() print(f'\rprocessing parameter: {name} {value.shape} ', end='', flush=True) ckpt_list.append({'name': name, 'data': ms.Tensor(value, dtype=dtype)}) ckpt_file = os.path.join(ckpt_dir, output_name) ms.save_checkpoint(ckpt_list, os.path.join(ckpt_file)) print(f"\rConvert huggingface checkpoint finished, the mindspore checkpoint is saved in '{ckpt_file}'.", flush=True) return True def convert_to_new_ckpt(ckpt_path, config_path): """convert previous ckpt to new ckpt""" load_path = ckpt_path.split('.ckpt')[0] save_path = load_path + "_hf" params = ms.load_checkpoint(load_path.split('.ckpt')[0] + '.ckpt') with open(config_path, "r") as f: model_args = json.loads(f.read()) n_heads = model_args["n_heads"] dim = model_args["dim"] def permute(w): return ops.transpose(w.reshape(n_heads, dim // n_heads // 2, 2, dim), (0, 2, 1, 3)).reshape(dim, dim) ckpt_list = [] for name in params.keys(): value = params[name].value() if '.wq' in name or '.wk' in name: value = permute(value) ckpt_list.append({'name': name, 'data': value}) print("\r", name, value.shape, end=" ") ms.save_checkpoint(ckpt_list, save_path) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument('--torch_ckpt_dir', default='./llama_model/llama-13b-hf/') parser.add_argument('--mindspore_ckpt_path', default='transform.ckpt') parser.add_argument('--pre_ckpt_path', default=None) parser.add_argument('--config_path', default=None) args = parser.parse_args() if args.pre_ckpt_path is not None and args.config_path is not None: convert_to_new_ckpt(args.pre_ckpt_path, args.config_path) else: convert_hf_ckpt(ckpt_dir=args.torch_ckpt_dir, output_name=args.mindspore_ckpt_path) pengcheng.mind-mindformer/mindformers/models/llama/llama.py000066400000000000000000000472641476650141400244650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA models' APIs.""" import math import numpy as np import mindspore.common.dtype as mstype try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import Tensor, nn from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation try: # pylint: disable=W0611 from mindspore.nn.layer.flash_attention import FlashAttention FLASHATTENTION_VALID = True except ImportError: FLASHATTENTION_VALID = False from mindspore.common.initializer import Normal from mindspore import lazy_inline from mindformers.core.loss.loss import CrossEntropyLoss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear from mindformers.modules.transformer.op_parallel_config import _check_config from mindformers.modules.transformer.transformer import AttentionMask from mindformers.tools.register.register import MindFormerModuleType, MindFormerRegister #from mindformers.pet.tuners.pet_adapter import PetAdapter #from mindformers.pet.tuners.lora_adapter import LoraAdapter from .llama_config import LlamaConfig from .llama_layer import LlamaEmbedding, LlamaRMSNorm, precompute_freqs_cis from .llama_transformer import LLamaDecodeLayer from ..utils import cell_reuse from ...tools.logger import logger __all__ = ['LlamaModel', 'LlamaForCausalLM'] def layer_compute_dtype(layer, layer_id, offset, parallel_config, n_layers, select_recompute=False): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: layer(Cell) - Represents the transformer block parallel_config(dict) - Parallel Config layer_id(int) - Means the layer index for the current module, counts from zero. offset(Union[int, List[int]]) - Means the layer_index needs a offset, if there are other modules in the net. n_layers(int) - The total layers used for the model. """ pp_dis = max(int((n_layers + 1) / parallel_config.pipeline_stage), 1) if isinstance(offset, list): if len(offset) != parallel_config.pipeline_stage: raise ValueError(f"The length of `offset` {len(offset)} do not match " "`pipeline stage` {parallel_config.pipeline_stage}.") i = min(layer_id // pp_dis, parallel_config.pipeline_stage - 1) offset_layer = offset[i] elif isinstance(offset, int): offset_layer = offset else: raise TypeError(f"`offset` must be `int` of list of `int`, but got {type(offset)}.") if parallel_config.pipeline_stage == 4: pp_id_list = [9, 11, 11, 9] pp_id = 0 tmp = layer_id while tmp >= pp_id_list[pp_id]: tmp -= pp_id_list[pp_id] pp_id += 1 else: pp_id = min((layer_id + offset_layer) // pp_dis, parallel_config.pipeline_stage - 1) layer.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int((n_layers + 1) / parallel_config.gradient_aggregation_group), 1) layer.set_comm_fusion(3 + layer_id) if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not select_recompute: layer.recompute() else: if parallel_config.recompute.recompute and not select_recompute: layer.recompute( recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) #layer.attention.flash_attention.flash_attention.recompute(mode=False) class LlamaModel(BaseModel): r""" Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`LlamaDecoderLayer`] Args: config(LlamaConfig): the config of network Inputs: input_ids: the tokenized inputs with datatype int32 Returns: output: Tensor, the output of llama decoderlayer """ _support_list = MindFormerBook.get_model_support_list()['llama'] def __init__(self, config: LlamaConfig = None): super().__init__(config, auto_prefix=True) _check_config(config.parallel_config) if config.batch_size or config.use_past: Validator.check_positive_int(config.batch_size) self.dtype = config.compute_dtype self.num_layers = config.num_layers self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.is_first_iteration = True self.use_past = config.use_past self.use_flash_attention = config.use_flash_attention and FLASHATTENTION_VALID if self.use_flash_attention: logger.info("Enable flash attention.") elif config.use_flash_attention: logger.info("Current MindSpore do not support flash attention.") self.freqs_cos, self.freqs_sin, self.swap_mask = precompute_freqs_cis( config.hidden_size // config.num_heads, config.seq_length, dtype=config.rotary_dtype, pretrain_seqlen=config.pretrain_seqlen, extend_method=config.extend_method) self.get_attention_mask = AttentionMask( config.seq_length, parallel_config=config.parallel_config.dp_mp_config).to_float(config.compute_dtype) self.multiply_data = Tensor([-10000.0], dtype=config.compute_dtype) self.one = Tensor([1.0], dtype=config.compute_dtype) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile() self.mul_mask = P.Mul() self.sub = P.Sub() self.expand_dims = P.ExpandDims() self.not_equal = P.NotEqual() self.gather = P.Gather() self.tok_embeddings = LlamaEmbedding( config.vocab_size, config.hidden_size, param_init_type=config.param_init_type) self.embed_scale = Tensor([config.embed_scale], dtype=config.compute_dtype) self.mul_embed = P.Mul() self.layers = nn.CellList() for layer_id in range(config.num_layers): layer = LLamaDecodeLayer(config.batch_size, config.seq_length, layer_id, config.num_layers, dim=config.hidden_size, n_heads=config.num_heads, multiple_of=config.multiple_of, n_kv_heads=config.n_kv_heads, ffn_dim_multiplier=config.ffn_dim_multiplier, mla_multiplier=config.mla_multiplier, norm_eps=config.rms_norm_eps, compute_dtype=config.compute_dtype, layernorm_compute_dtype=config.layernorm_compute_type, softmax_compute_dtype=config.softmax_compute_type, rotary_dtype=config.rotary_dtype, mup_width_base=config.mup_width_base, param_init_type=config.param_init_type, use_past=config.use_past, use_flash_attention=config.use_flash_attention, compute_in_2d=config.compute_in_2d, use_past_shard=config.use_past_shard, depth_scale=config.depth_scale, init_std=config.init_std, parallel_config=config.parallel_config) layer_compute_dtype(layer, layer_id, config.offset, config.parallel_config, config.num_layers, select_recompute=config.parallel_config.recompute.select_recompute) self.layers.append(layer) self.norm_out = LlamaRMSNorm(config.hidden_size, config.rms_norm_eps, compute_type=config.layernorm_compute_type) dp = config.parallel_config.data_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.tok_embeddings.pipeline_stage = 0 if config.parallel_config.pipeline_stage > 1: self.norm_out.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.tok_embeddings.set_comm_fusion(2) self.norm_out.set_comm_fusion(config.num_layers + 3) self.tok_embeddings.shard(config.parallel_config) self.tile.shard(((1, 1, 1, 1), ())) self.sub.shard(((1,), (dp, 1, 1))) self.mul_mask.shard(((dp, 1, 1, 1), (1,))) self.mul_embed.shard(((dp, 1, 1), (1,))) self.expand_dims.shard(((dp, 1, 1),)) self.not_equal.shard(((dp, 1), ())) self.gather.shard(((dp, 1), (1,))) if config.compute_in_2d: self.norm_out.shard((dp, 1)) else: self.norm_out.shard((dp, 1, 1)) if self.use_past: seq_range = np.arange(config.seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (config.batch_size, 1, 1)), mstype.int32) self.gather_past = P.Gather() self.expand_dims = P.ExpandDims() self.le_past = P.LessEqual() ones = np.ones(shape=(config.seq_length, config.seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) self.equal = P.Equal().shard(((dp, 1), ())) self.cumsum = P.CumSum(reverse=True).shard(((dp, 1),)) self.pow = P.Pow().shard(((), (dp, 1))) self.pow_up_value = 2.0 self.pow_down_value = 0.5 self.mask_reshape = P.Reshape() self.mask_mul = P.BatchMatMul().shard( ((dp, 1, 1), (dp, 1, 1))) self.up_threshold = 1.2 self.down_threshold = 0.8 self.greater = P.Greater().shard(((dp, 1, 1), ())) self.less = P.Less().shard(((dp, 1, 1), ())) self.mask_multiply = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mask_expand = P.ExpandDims().shard(((1, 1),)) self.mask_multiply2 = P.Mul().shard(((dp, 1, 1), (1, 1, 1))) # pylint: disable=W0613 def construct(self, tokens: Tensor, input_position=None, init_reset=True, batch_valid_length=None): """Forward of llama model.""" # preprocess bs, seq_len = tokens.shape if self.is_first_iteration: freqs_cis = (self.tile(self.reshape(self.freqs_cos, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.tile(self.reshape(self.freqs_sin, (1, 1, seq_len, -1)), (bs, 1, 1, 1)), self.swap_mask) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), self.dtype) seq_index = self.cast(self.equal(tokens, self.eos_token_id), mstype.float32) seq_mask_rev = self.cumsum(seq_index, 1) seq_mask_exp_up = self.mask_reshape(self.pow(self.pow_up_value, seq_mask_rev), (bs, seq_len, 1)) seq_mask_exp_down = self.mask_reshape(self.pow(self.pow_down_value, seq_mask_rev), (bs, 1, seq_len)) full_mask = self.mask_mul(seq_mask_exp_up, seq_mask_exp_down) full_mask_small = self.cast(self.less(full_mask, self.up_threshold), mstype.float32) full_mask_great = self.cast(self.greater(full_mask, self.down_threshold), mstype.float32) mask = self.mask_multiply(full_mask_small, full_mask_great) lower_triangle = self.mask_expand(self.lower_triangle_mask, 0) mask = self.mask_multiply2(full_mask, lower_triangle) # mask: [bs, seq, seq] # else: # cur_pos = batch_valid_length - 1 # valid_length = self.reshape(cur_pos, (-1, 1, 1)) # freqs_cis = (self.reshape(self.gather_past(self.freqs_cos, cur_pos, 0), (bs, 1, seq_len, -1)), # self.reshape(self.gather_past(self.freqs_sin, cur_pos, 0), (bs, 1, seq_len, -1)), # self.swap_mask) # mask = self.cast(self.le_past(self.range, valid_length), self.dtype) # mask: [bs, 1, 1] mask = self.sub(self.one, self.cast(mask, self.dtype)) if not self.use_flash_attention: mask = self.expand_dims(mask, 1) mask = self.mul_mask(mask, self.multiply_data) # tokens: [bs, seq/1] h = self.tok_embeddings(tokens) h = self.mul_embed(h, self.embed_scale) # h: [bs, seq/1, hidden_dim] for i in range(self.num_layers): h, _ = self.layers[i](h, freqs_cis, mask, init_reset=init_reset, batch_valid_length=batch_valid_length) output = self.norm_out(h) return output @MindFormerRegister.register(MindFormerModuleType.MODELS) class LlamaForCausalLM(BaseModel): r""" Provide llama training loss or logits through network. Args: config (LlamaConfig): The config of llama model. Inputs: input_ids(Tensor): the tokenized inputs with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. labels(Tensor): the tokenized labels with datatype int32, Tensor of shape :math:`(batch, seq\_length)`. input_position(Tensor): current position, used by model.predict. position_ids(Tensor): Reserved param, not used. attention_mask(Tensor): Reserved param, not used. input_embeds(Tensor): Reserved param, not used. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: Tensor, the loss or logits of the network. Examples: >>> from mindformers.models.llama import LlamaConfig, LlamaForCausalLM >>> config = LlamaConfig(batch_size=2) >>> network = LlamaForCausalLM(config=config) """ _support_list = MindFormerBook.get_model_support_list()['llama'] #@lazy_inline def __init__(self, config: LlamaConfig = None): super(LlamaForCausalLM, self).__init__(config, auto_prefix=True) _check_config(config.parallel_config) self.ignore_token_id = config.ignore_token_id self.pad_token_id = config.pad_token_id self.eos_token_id = config.eos_token_id self.reshape = P.Reshape() self.cast = P.Cast() self.slice = P.StridedSlice() self.not_equal = P.NotEqual() self.mul = P.Mul() self.add = P.Add() self.model = LlamaModel(config=config) self.attn_logit_softcapping = Tensor(config.final_logit_softcapping, dtype=mstype.float32) self.inv_attn_logit_softcapping = Tensor(1.0 / config.final_logit_softcapping, dtype=mstype.float32) self.tanh = P.Tanh() self.mul_logits = P.Mul() sigma = config.init_std if config.mup_width_base > 0: ratio = config.hidden_size / config.mup_width_base sigma = sigma / ratio self.lm_head = Linear(in_channels=config.hidden_size, out_channels=config.vocab_size, has_bias=False, compute_dtype=config.compute_dtype, param_init_type=config.param_init_type, weight_init=Normal(sigma=sigma, mean=0)) # meta default: xavier_normal self.loss = CrossEntropyLoss(parallel_config=config.parallel_config) dp = config.parallel_config.data_parallel mp = config.parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.slice.shard(((dp, 1),)) self.not_equal.shard(((dp, 1), ())) self.mul.shard(((dp, 1), (dp, 1))) self.add.shard(((dp, 1), ())) # if config.parallel_config.vocab_emb_dp: # self.lm_head.shard(strategy_matmul=((dp, 1), (1, 1))) # else: # self.lm_head.shard(strategy_matmul=((dp, 1), (mp, 1))) self.lm_head.shard(((dp * mp, 1), (1, 1)), out_strategy_matmul=((dp * mp, 1),)) self.mul_logits.shard(((dp, mp, 1), ())) self.tanh.shard(((dp, mp, 1),)) if config.parallel_config.pipeline_stage > 1: self.lm_head.pipeline_stage = config.parallel_config.pipeline_stage - 1 #self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): return { "input_ids": Tensor(input_ids, mstype.int32) } # pylint: disable=W0613 def construct(self, input_ids, labels=None, input_position=None, position_ids=None, attention_mask=None, input_embeds=None, init_reset=True, batch_valid_length=None): """LlamaForCausalLM forward.""" bsz, seqlen = input_ids.shape #if self.training: # tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) #else: # tokens = input_ids tokens = self.slice(input_ids, (0, 0), (bsz, seqlen - 1), (1, 1)) output = self.model(tokens, input_position, init_reset, batch_valid_length) logits = self.lm_head(output) input_mask = self.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if labels is None: labels = self.slice(input_ids, (0, 1), (bsz, seqlen), (1, 1)) else: if labels.ndim > 1: if self.training: labels = self.slice(labels, (0, 1), (bsz, seqlen), (1, 1)) label_mask = self.cast(self.not_equal(labels, self.ignore_token_id), mstype.float32) input_mask = self.mul(input_mask, label_mask) logits = self.cast(logits, mstype.float32) ################################################################# logits = self.mul_logits(logits, self.inv_attn_logit_softcapping) logits = self.tanh(logits) logits = self.mul_logits(logits, self.attn_logit_softcapping) ################################################################# #if not self.training: # logits = self.reshape(logits, (bsz, seqlen, -1)) # # makes cast effective to avoid allgather issue in Mindspore1.10 # input_mask = self.add(input_mask, 1) # return logits, tokens, input_mask if logits.ndim > 2: logits = self.reshape(logits, (-1, logits.shape[-1])) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss pengcheng.mind-mindformer/mindformers/models/llama/llama_config.py000066400000000000000000000116771476650141400260110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Llama Config API.""" from typing import Optional from mindformers.modules.transformer.transformer import default_transformer_config, TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['LlamaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class LlamaConfig(BaseConfig): """ LLaMA config class which defines the model size. """ _support_list = MindFormerBook.get_config_support_list()['llama'] def __init__(self, batch_size: int = 1, seq_length: int = 2048, hidden_size: int = 4096, num_layers: int = 32, num_heads: int = 32, vocab_size: int = 32000, # defined later by tokenizer multiple_of: int = 256, # make SwiGLU hidden layer size multiple of large power of 2 n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, rms_norm_eps: float = 1e-5, bos_token_id: int = 1, eos_token_id: int = 2, pad_token_id: int = 32000, ignore_token_id: int = -100, compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", rotary_dtype: str = "float32", param_init_type: str = "float16", parallel_config: TransformerOpParallelConfig = default_transformer_config, use_past: bool = False, pretrain_seqlen: int = 2048, extend_method: str = "None", compute_in_2d: bool = False, use_flash_attention: bool = False, offset: int = 0, use_past_shard: bool = False, checkpoint_name_or_path: str = "", repetition_penalty: float = 1.0, max_decode_length: int = 1024, final_logit_softcapping: float = 30.0, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, mup_width_base: int = -1, embed_scale: float = 1.0, depth_scale: float = 1.0, init_std: float = 0.01, **kwargs): super(LlamaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.multiple_of = multiple_of self.n_kv_heads = n_kv_heads self.ffn_dim_multiplier = ffn_dim_multiplier self.mla_multiplier = mla_multiplier self.rms_norm_eps = rms_norm_eps self.param_init_type = convert_mstype(param_init_type) self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.rotary_dtype = convert_mstype(rotary_dtype) self.compute_dtype = convert_mstype(compute_dtype) self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path self.bos_token_id = bos_token_id self.eos_token_id = eos_token_id self.pad_token_id = pad_token_id self.ignore_token_id = ignore_token_id self.use_past = use_past self.pretrain_seqlen = pretrain_seqlen self.extend_method = extend_method self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention self.offset = offset self.use_past_shard = use_past_shard self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample self.mup_width_base = mup_width_base self.embed_scale = embed_scale self.depth_scale = depth_scale self.init_std = init_std self.final_logit_softcapping = final_logit_softcapping pengcheng.mind-mindformer/mindformers/models/llama/llama_layer.py000066400000000000000000000365261476650141400256600ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA Model Layers' APIs.""" from enum import Enum import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore import nn import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell from mindspore.common.initializer import Normal try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.common.initializer import initializer from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import Linear, _check_input_dtype, _args_type_validator_check, _valid_value_checks from mindformers.tools.logger import _LogActionOnce class SeqExtendMethod(Enum): """Stores the acceptable string identifiers for seq length extend method""" PI = "PI" NTK = "NTK" NONE = "None" class LlamaSiLU(Cell): r""" A self-defined SwiGlu. Inputs: - **x** (Tensor) - Tensor. Outputs: Tensor. x = x * sigmod(x). """ def __init__(self): super().__init__() self.sigmoid = nn.Sigmoid() self.mul = P.Mul() def shard(self, strategy): self.sigmoid.sigmoid.shard(strategy) self.mul.shard((strategy[0], strategy[0])) def construct(self, x): return self.mul(x, self.sigmoid(x)) def get_swap_mask(head_dim): """Swap matrix""" zero_block = np.zeros((head_dim // 2, head_dim // 2), dtype=np.float32) id_block = np.identity(head_dim // 2, dtype=np.float32) return np.block([[zero_block, id_block], [-id_block, zero_block]]) def precompute_freqs_cis( dim: int, end: int, theta: float = 10000.0, dtype=mstype.float32, pretrain_seqlen=2048, extend_method=SeqExtendMethod.NONE.value): """ Precompute of freqs and mask for rotary embedding. """ ratio = 1. if extend_method != SeqExtendMethod.NONE.value and end > pretrain_seqlen: ratio = end / pretrain_seqlen if extend_method == SeqExtendMethod.NTK.value: theta *= ratio freqs_base = np.arange(0, dim, 2)[: (dim // 2)].astype(np.float32) # (head_dim // 2, ) freqs = 1.0 / (theta ** (freqs_base / dim)) # (head_dim // 2, ) if extend_method == SeqExtendMethod.PI.value: t = np.arange(0, end / ratio, 1 / ratio).astype(np.float32) else: t = np.arange(0, end, 1).astype(np.float32) # type: ignore # (seq_len,) freqs = np.outer(t, freqs) # type: ignore (seq_len, head_dim // 2) emb = np.concatenate((freqs, freqs), axis=-1) freqs_cos = np.cos(emb) # (seq_len, head_dim) freqs_sin = np.sin(emb) # (seq_len, head_dim) freqs_cos = Tensor(freqs_cos, dtype=dtype) freqs_sin = Tensor(freqs_sin, dtype=dtype) swap_mask = get_swap_mask(dim) swap_mask = Tensor(swap_mask, dtype=dtype) return freqs_cos, freqs_sin, swap_mask class LlamaRotaryEmbedding(Cell): r""" Rotary Position Embedding. Args: - **head_dim** (int): The dim of multi head attention. - **compute_dtype** (mstype): The compute type, default mstype.float16. - **parallel_config** (dict): - Parallel Config. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, head_dim=128, compute_dtype=mstype.float32): super().__init__(auto_prefix=False) self.head_dim = head_dim self.dtype = compute_dtype self.add = P.Add() self.bmm_swap = P.BatchMatMul() self.mul = P.Mul() self.cast = P.Cast() def rotate_half(self, x, swap_mask): # [bs, n_head/n_kv_head, seq/1, head_dim], [head_dim, head_dim] x = self.bmm_swap(x, swap_mask) return x def construct(self, xq: Tensor, xk: Tensor, freqs_cis): """Forward of rotary position embedding.""" original_type = xq.dtype xq = self.cast(xq, self.dtype) xk = self.cast(xk, self.dtype) # xq, xk: [bs, n_head/n_kv_head, seq/1, head_dim] freqs_cos, freqs_sin, swap_mask = freqs_cis xq_out = self.add(self.mul(xq, freqs_cos), self.mul(self.rotate_half(xq, swap_mask), freqs_sin)) xk_out = self.add(self.mul(xk, freqs_cos), self.mul(self.rotate_half(xk, swap_mask), freqs_sin)) xq_out = self.cast(xq_out, original_type) xk_out = self.cast(xk_out, original_type) return xq_out, xk_out def shard(self, strategy_in): self.add.shard((strategy_in, strategy_in)) self.bmm_swap.shard((strategy_in, (1, 1))) self.mul.shard((strategy_in, (strategy_in[0], 1, 1, 1))) class LlamaEmbedding(Cell): """ Embedding Layer. Args: - **vocab_size** (int): Size of the dictionary of embeddings. - **embedding_size** (int): The size of each embedding vector. - **param_init_type** (mstype): The param init type, default mstype.float32. - **parallel_config** (TransformerOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. - **param_init** (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). """ @_LogActionOnce(m_logger=logger, key='Embedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_table_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int) def __init__(self, vocab_table_size, embedding_size, param_init_type=mstype.float32, param_init='normal'): super().__init__() self.vocab_table_size = vocab_table_size self.embedding_size = embedding_size self.embedding_weight = Parameter( initializer(param_init, [self.vocab_table_size, self.embedding_size], dtype=param_init_type), name='embedding_weight', parallel_optimizer=True) self.gather = P.Gather() def construct(self, input_ids): """Forward of vocab embedding.""" _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32, mstype.int64], self.cls_name) output = self.gather(self.embedding_weight, input_ids, 0) return output def shard(self, parallel_config): """sharding for embedding""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if parallel_config.vocab_emb_dp: self.gather.shard(((1, 1), (dp, 1))) logger.info(f"Using {dp} data parallel for the embedding lookup.") else: if self.vocab_table_size % mp != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_table_size} must be a " f"multiple of parallel_config.model_parallel {mp}.") self.gather.shard(((mp, 1), (dp, 1))) logger.info(f"Using {dp} data parallel and {mp} " f"model parallel for the embedding lookup.") class LlamaRMSNorm(nn.Cell): r""" A self-defined RMSNorm operation using reduce mean. Args: dim (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. compute_type: The compute type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, dim, eps=1e-6, compute_type=mstype.float32): super(LlamaRMSNorm, self).__init__() self.eps = eps self.weight = Parameter(initializer('ones', (dim,), dtype=mstype.float32), parallel_optimizer=False) self.square = P.Square() self.mean = P.ReduceMean(keep_dims=True) self.add = P.Add() self.rsqrt = P.Rsqrt() self.mul = P.Mul() self.mul2 = P.Mul() self.cast = P.Cast() self.compute_type = compute_type def _norm(self, x): norm_factor = self.square(x) norm_factor = self.mean(norm_factor, -1) norm_factor = self.add(norm_factor, self.eps) norm_factor = self.rsqrt(norm_factor) x = self.cast(x, mstype.float16) norm_factor = self.cast(norm_factor, mstype.float16) return self.mul(x, norm_factor) def construct(self, x): """Forward of RMSNorm.""" original_type = x.dtype x = self.cast(x, self.compute_type) output = self._norm(x) output = self.cast(output, mstype.float16) weight = self.cast(self.weight, mstype.float16) output = self.mul2(output, weight) output = self.cast(output, original_type) return output def shard(self, strategy_in): """Parallel strategy configuratiuon interface.""" self.square.shard((strategy_in,)) self.mean.shard((strategy_in,)) self.rsqrt.shard((strategy_in,)) self.add.shard((strategy_in, ())) self.mul.shard((strategy_in, strategy_in)) self.mul2.shard((strategy_in, (1,))) class LlamaFeedForward(Cell): r""" LLaMA FeedForward. .. math:: (xW_1 * xW_3)W_2 Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: ValueError: `hidden_dim` is not a multiple of the model parallel way. ValueError: `dim` is not a multiple of the model parallel way. """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(dim=Validator.check_positive_int, hidden_dim=Validator.check_positive_int, multiple_of=Validator.check_positive_int, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward")) def __init__(self, dim, hidden_dim, multiple_of, hidden_act=LlamaSiLU, ffn_dim_multiplier=None, mup_width_base=-1, init_std=0.01, compute_dtype=mstype.float16, param_init_type=mstype.float32): super().__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if ffn_dim_multiplier is not None: hidden_dim = int((ffn_dim_multiplier + 0.01) * hidden_dim) hidden_dim = int(2 * hidden_dim / 3) hidden_dim = multiple_of * \ ((hidden_dim + multiple_of - 1) // multiple_of) self.dtype = compute_dtype self.hidden_act = hidden_act self.dim = dim self.hidden_dim = hidden_dim self.mul = P.Mul() self.cast = P.Cast() sigma = init_std if mup_width_base > 0: ratio = dim / mup_width_base sigma = sigma / math.sqrt(ratio) self.w1 = Linear(in_channels=dim, out_channels=hidden_dim, activation=hidden_act, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w2 = Linear(in_channels=hidden_dim, out_channels=dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) self.w3 = Linear(in_channels=dim, out_channels=hidden_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type, weight_init=Normal(sigma=sigma, mean=0)) def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, self.dtype) # [bs, seq, hidden_dim] or [bs * seq, hidden_dim] gate = self.w1(x) # dp,1 -> dp, mp hidden = self.w3(x) # dp,1 -> dp, mp hidden = self.mul(hidden, gate) # dp,mp -> dp, mp output = self.w2(hidden) # dp,mp -> dp, 1 return output def shard(self, parallel_config): """sharding for feedforward""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel if self.hidden_dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_dim' must be a multiple of the" "num of model parallel, but got the hidden_dim is {} and the num of model " "parallel is {}.".format(self.hidden_dim, mp)) if self.dim % mp != 0: raise ValueError("For 'FeedForward', the class variable 'dim' must be a multiple of the num of " "model parallel, but got the dim is {} and the num of model parallel is {}." .format(self.dim, mp)) self.w1.shard(((dp, 1), (mp, 1)), strategy_activation=((dp, mp),)) self.w1.activation.shard(((dp, mp),)) self.w2.shard(((dp, mp), (1, mp))) self.w3.shard(((dp, 1), (mp, 1))) self.mul.shard(((dp, mp), (dp, mp))) pengcheng.mind-mindformer/mindformers/models/llama/llama_processor.py000066400000000000000000000044201476650141400265470ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ LlamaProcessor """ from mindformers.mindformer_book import MindFormerBook from ..base_tokenizer import BaseTokenizer from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class LlamaProcessor(BaseProcessor): """ Llama processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['llama'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(LlamaProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-mindformer/mindformers/models/llama/llama_tokenizer.py000066400000000000000000000263521476650141400265520ustar00rootroot00000000000000# coding=utf-8 # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA tokenizer APIs.""" import os from shutil import copyfile from typing import Any, Dict, List, Optional import sentencepiece as spm from mindformers.tools import logger from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import Tokenizer, AddedToken from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['LlamaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class LlamaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Tokenizer of llama will default add bos at the beginning of tokens and add eos token on the tail of tokens. Args: model_path(str): The spiece.model file path. add_bos(bool): The flag defines whether add bos token, Default True. eos_token(str): The token that represents the end-of-sentence. Default "". unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". sp_model_kwargs(str): Other kwargs for sp_model`. add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "True" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "True" clean_up_tokenization_spaces (bool): Whether or not the model should cleanup the spaces that were added when splitting the input text during the tokenization process. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import LlamaTokenizer >>> tokenizer = LlamaTokenizer.from_pretrained("llama_7b") >>> res = tokenizer("hello world") >>> print(res) {'input_ids': [1, 22172, 3186, 2]} >>> res = tokenizer("hello world", padding='max_length', max_length=10) >>> print(res) {'input_ids': [1, 22172, 3186, 2, 0, 0, 0, 0, 0, 0]} >>> res = tokenizer("hello world", return_tensors='ms') >>> print(res) {'input_ids': Tensor(shape=[3], dtype=Int32, value= [1, 22172, 3186, 2])} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['llama'] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token="", sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text, **kwargs): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix=None): """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-mindformer/mindformers/models/llama/llama_transformer.py000066400000000000000000001566641476650141400271140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LLaMA transformer Layer's APIs.""" from typing import Tuple, Optional import math import numpy as np try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import nn, ops import mindspore.common.dtype as mstype from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.context import ParallelMode from mindspore.ops import operations as P from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.common.initializer import Normal from flash_attention.flash_attention_impl import get_flash_attention from mindformers.models.llama.llama_layer import LlamaFeedForward, LlamaRMSNorm, LlamaRotaryEmbedding from mindformers.modules.layers import _check_input_dtype, Linear from mindformers.modules.transformer import TransformerOpParallelConfig HEAD_DIM_MAX_LIMIT = 256 class FlashAttention(nn.Cell): """Flash Attention Layer. This function contains the flash attention primitives used in FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. An interface for calling flashattention operation. 2. Two configuration parameters for enabling local block sparse of flashattention. Args: head_dim(int): The hidden size of input. dropout_rate(float): The dropout rate of the attention score. Default 0.0. prev_block_num(int): A integer to define the number of blocks to look ahead for local block sparse attention. Default 65536. next_block_num(int): A integer to define the number of blocks to look behind for local block sparse attention. Default 65536. tiling_stgy_name(str): A str to define tiling strategy of flash attention. dp(int): data parallel. Default 1. mp(int): model parallel. Default 1. have_attention_mask_batch(bool): indicates whether attention_mask contains the batch dimension. Default True alibi(bool): This parameter indicates whether the flashattention supports the Alibi. Default: False Inputs: - **query** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **key** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **value** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, head_dim]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): A matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, head_dim] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from acctransformer.flash_attention.nn.layer.flash_attention import FlashAttention >>> from mindspore import Tensor >>> model = FlashAttention(head_dim=128, ... dropout_rate=0.1, ... prev_block_num=7, ... next_block_num=0 ... ) >>> query = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> key = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> value = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.triu(np.ones((1, 128, 128)), k=1), mstype.float16) >>> output = model(query, key, value, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ def __init__(self, head_dim, dropout_rate=0.0, prev_block_num=65536, next_block_num=65536, tiling_stgy_name="sparse", dp=1, mp=1, have_attention_mask_batch=True, alibi=False ): super(FlashAttention, self).__init__() if head_dim > HEAD_DIM_MAX_LIMIT: raise ValueError("head_dim too large, reduce head_dim and try again.") if alibi: raise ValueError("alibi not supported in the current version.") self.flash_attention = get_flash_attention( prev_block_num=prev_block_num, next_block_num=next_block_num, tiling_stgy_name=tiling_stgy_name ) self.flash_attention.add_prim_attr("primitive_target", "Ascend") self.scale_factor = Tensor([1. / math.sqrt(math.sqrt(head_dim))], dtype=mstype.float16) self.scale_mul = ops.Mul().shard(((dp, mp, 1, 1), (1,))) self.dropout_rate = dropout_rate self.enable_dropout = self.dropout_rate > 0.0 self.have_attention_mask_batch = have_attention_mask_batch self.alibi = alibi if self.enable_dropout: self.keep_prob = Tensor(1 - self.dropout_rate, dtype=mstype.float16) self.fill_v2 = ops.FillV2().shard(((dp, mp, 1, 1), ())) self.tensor_one = Tensor(1.0, mstype.float16) self.drop_gen_mask = ops.DropoutGenMask() self.do_dropout = ops.DropoutDoMask().shard(((dp, mp, 1, 1),)) self.depend = ops.Depend() def shard(self, in_strategy=None, out_strategy=None): """Distributed configuration of FlashAttention :param in_strategy: Describe the split strategy of operator input. Default: None. :param out_strategy: Describe the split strategy of operator output, it is only for certain operators, such as MatMul. Default: None. :return: """ if in_strategy is None: # default: dp=1, mp=1, construct inputs only contain q, k, v in_strategy = ( (1, 1, 1, 1), (1, 1, 1, 1), (1, 1, 1, 1), ) self.flash_attention.shard(in_strategy) dp = in_strategy[0][0] mp = in_strategy[0][1] self.flash_attention.add_prim_attr("dev_matrix_shape", [dp, mp, 1, 1]) inputs_tensor_map = [ [3, 2, 1, 0], [3, 2, 1, 0], [3, 2, 1, 0], ] if self.have_attention_mask_batch: inputs_tensor_map.append([3, 1, 0]) else: inputs_tensor_map.append([-1, 1, 0]) input_empty_args_num = 2 # dropout_mask if self.enable_dropout: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) if self.alibi: input_empty_args_num -= 1 inputs_tensor_map.append([3, 2, 1, 0]) self.flash_attention.add_prim_attr("inputs_tensor_map", inputs_tensor_map) self.flash_attention.add_prim_attr("outputs_tensor_map", [ [3, 2, 1, 0], # O [3, 2, 1], # L ]) self.flash_attention.add_prim_attr("as_loss_divisor", 0) self.flash_attention.add_prim_attr("empty_mirror_ops", input_empty_args_num) def construct(self, q, k, v, attn_mask=None, alibi_mask=None): """FlashAttention forward :param q: [bsz, head_num, seq_len, head_dim] :param k: [bsz, head_num, seq_len, head_dim] :param v: [bsz, head_num, seq_len, head_dim] :param attn_mask: [1 or bsz, seq_len, seq_len], if not None :param alibi_mask: [bsz, head_num, 1, seq_len], if not None :return: o [bsz, head_num, seq_len, head_dim] """ q = self.scale_mul(q, self.scale_factor) k = self.scale_mul(k, self.scale_factor) bsz, head_num, seq_len, head_dim = q.shape _, k_head_num, _, _ = k.shape _, v_head_num, _, _ = v.shape if head_num != k_head_num or head_num != v_head_num: raise ValueError( "the head_num of query, key and value must be the same, " "If different head_num are used, users need to change themselves to be same by tile.") if self.enable_dropout: drop_mask_bits = self.drop_gen_mask((bsz, head_num, seq_len, seq_len), self.keep_prob) tensor_shape = Tensor((bsz, head_num, seq_len, seq_len), mstype.int32) ones = self.fill_v2(tensor_shape, self.tensor_one) ones = self.depend(ones, q) drop_mask = self.do_dropout(ones, drop_mask_bits, self.keep_prob) else: drop_mask = None o, _ = self.flash_attention(q, k, v, attn_mask, drop_mask, alibi_mask) return o class LLamaAttention(nn.Cell): r""" This is an implementation of multihead attention in LLaMA. Args: - **batch_size** (int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. - **src_seq_length** (int): The sequence length of the query vector. - **tgt_seq_length** (int): The sequence length of the key and value vector. - **dim** (int): The hidden size of the input. - **head_dim** (int): The dim of head. - **n_heads** (int): The number of the heads. - **compute_dtype** (dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. - **softmax_compute_type** (dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. - **param_init_type** (dtype.Number): The parameter initialization type of the module. Default mstype. float32. Should be mstype.float32 or mstype.float16. - **use_past** (bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. - **parallel_config** (OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - The input tokens with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, head_dim, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, head_dim). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, n_kv_heads: Optional[int] = None, mup_width_base=-1, init_std=0.01, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, attn_logit_softcapping=50.0, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.n_rep = self.n_head // self.n_kv_head self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) if self.n_kv_head % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'n_kv_head' must be a multiple of " "'parallel_config.model_parallel', but got the n_kv_head is {} " "and the parallel_config.model_parallel is {}." .format(self.n_kv_head, parallel_config.model_parallel)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() sigma = init_std if mup_width_base > 0: ratio = self.hidden_size / mup_width_base sigma = sigma / math.sqrt(ratio) self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wq = Linear(self.hidden_size, self.hidden_size, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wk = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) self.wv = Linear(self.hidden_size, self.n_kv_head * self.head_dim, has_bias=False, compute_dtype=compute_dtype, weight_init=Normal(sigma=sigma, mean=0), param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) # self.tanh.shard(((dp, mp, 1, 1),)) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.wq.shard(((dp, 1), (mp, 1))) self.wk.shard(((dp, 1), (mp, 1))) self.wv.shard(((dp, 1), (mp, 1))) self.wo.shard(((dp, mp), (1, mp))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() # self.tanh.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(head_dim=self.head_dim, prev_block_num=65536, next_block_num=0, dp=parallel_config.data_parallel, mp=parallel_config.model_parallel) q_shard_stgy = (dp, mp, 1, 1) k_shard_stgy = (dp, mp, 1, 1) v_shard_stgy = (dp, mp, 1, 1) attn_mask_shard_stgy = (dp, 1, 1) in_stgy = (q_shard_stgy, k_shard_stgy, v_shard_stgy, attn_mask_shard_stgy) self.flash_attention.shard(in_stgy) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.wq(x), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.wk(x), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.wv(x), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_kv_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] key = self._repeat_kv(key, self.n_rep) value = self._repeat_kv(value, self.n_rep) # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _repeat_kv(self, x, rep): if rep == 1: return x bs, n_kv_head, seqlen, head_dim = x.shape x = self.reshape(x, (bs * n_kv_head, 1, seqlen, head_dim)) x = self.tile_kv(x, (1, rep, 1, 1)) x = self.reshape(x, (bs, n_kv_head * rep, seqlen, head_dim)) return x def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) score = self.cast_attn(score, self.softmax_dtype) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) ######################################################## # score = self.mul(score, self.inv_attn_logit_softcapping) # score = self.tanh(score) # score = self.mul(score, self.attn_logit_softcapping) ######################################################## score = self.add(mask, score) attention_probs = self.softmax(score) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaMLAAttention(nn.Cell): def __init__(self, batch_size, seq_length, dim: int = 512, n_heads: int = 8, mla_multiplier: int = 4, compute_dtype=mstype.float16, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, parallel_config=TransformerOpParallelConfig()): super().__init__() self.seq_length = seq_length self.hidden_size = dim self.n_head = n_heads self.head_dim = dim // n_heads self.lora_size = self.head_dim * mla_multiplier self.dtype = compute_dtype self.softmax_dtype = softmax_compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.use_flash_attention = use_flash_attention and FLASHATTENTION_VALID if self.hidden_size % self.n_head != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'n_head', but got the hidden_size is {} and the n_head is {}." .format(self.hidden_size, self.n_head)) self.inv_norm_factor = Tensor(1.0 / math.sqrt(self.head_dim), dtype=compute_dtype) self.reshape = P.Reshape() self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.batch_matmul = P.BatchMatMul() self.batch_matmul_q_k = P.BatchMatMul(transpose_b=True) self.mul = P.Mul() self.add = P.Add() self.softmax = nn.Softmax().to_float(softmax_compute_dtype) self.cast = P.Cast() self.cast_attn = P.Cast() self.tile_kv = P.Tile() self.apply_rotary_emb = LlamaRotaryEmbedding(self.head_dim, rotary_dtype) self.q_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.q_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_down = Linear(in_channels=self.hidden_size, out_channels=self.lora_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj_up = Linear(in_channels=self.lora_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.wo = Linear(in_channels=self.hidden_size, out_channels=self.hidden_size, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.transpose.shard(((dp, 1, mp, 1),)) self.merger_head_transpose.shard(((dp, mp, 1, 1),)) self.batch_matmul_q_k.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.batch_matmul.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul.shard(((dp, mp, 1, 1), ())) self.add.shard(((dp, 1, 1, 1), (dp, mp, 1, 1))) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.tile_kv.shard(((dp * mp, 1, 1, 1),)) self.apply_rotary_emb.shard((dp, mp, 1, 1)) self.q_proj_down.shard(((dp, 1), (mp, 1))) self.q_proj_up.shard(((dp, 1), (mp, 1))) self.k_proj_down.shard(((dp, 1), (mp, 1))) self.k_proj_up.shard(((dp, 1), (mp, 1))) self.v_proj_down.shard(((dp, 1), (mp, 1))) self.v_proj_up.shard(((dp, 1), (mp, 1))) if parallel_config.use_seq_parallel and self.is_first_iteration: self.wo.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if parallel_config.recompute.select_recompute: self.apply_rotary_emb.recompute() self.tile_kv.recompute() self.batch_matmul_q_k.recompute() self.mul.recompute() self.add.recompute() self.cast_attn.recompute() self.softmax.softmax.recompute() self.batch_matmul.recompute() if self.use_flash_attention: self.flash_attention = FlashAttention(self.head_dim, dp=dp, mp=mp, next_block_num=0, high_precision=(softmax_compute_dtype == mstype.float32)) if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.flash_attention.shard(((dp, mp, 1, 1), (dp, mp, 1, 1), (dp, mp, 1, 1), (dp, 1, 1), ())) if parallel_config.recompute.select_recompute: self.flash_attention.recompute() if self.use_past: # operators used for state reuse seq_range = np.arange(seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.add_past = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.equal = P.Equal().shard(((dp, 1, 1), (dp, 1, 1))) self.less = P.Less().shard(((dp, 1, 1), (dp, 1, 1))) self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.add_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.mul_past.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def construct(self, x: Tensor, freqs_cis: Tuple[Tensor, Tensor], mask=None, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" ori_dtype = x.dtype # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] x = self.reshape(x, (-1, x.shape[-1])) # [bs * seq/1, hidden_dim] query = self.cast(self.q_proj_up(self.q_proj_down(x)), self.dtype) # dp, 1 -> dp, mp key = self.cast(self.k_proj_up(self.k_proj_down(x)), self.dtype) # dp, 1 -> dp, mp value = self.cast(self.v_proj_up(self.v_proj_down(x)), self.dtype) # dp, 1 -> dp, mp query = self.reshape(query, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) key = self.reshape(key, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) value = self.reshape(value, (-1, self._get_seq_length_under_incremental(self.seq_length), self.n_head, self.head_dim)) # [bs, seq/1, n_head/n_kv_head, head_dim] query = self.transpose(query, (0, 2, 1, 3)) key = self.transpose(key, (0, 2, 1, 3)) value = self.transpose(value, (0, 2, 1, 3)) # [bs, n_head/n_kv_head, seq/1, head_dim] query, key = self.apply_rotary_emb(query, key, freqs_cis) # dp, mp, 1, 1 # kv cache: [bs, n_kv_head, 1, head_dim] -> [bs, n_kv_head, seq, head_dim] key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = ( self.less(self.range, batch_valid_length.view(-1, 1, 1))).astype(self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) value_present = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = (self.equal(self.range, valid_length)).astype(self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul_past(key, self.expand_dims(valid_length_vector, 3)) current_value = self.mul_past(value, self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add_past(key_past, current_key) value = self.add_past(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value layer_present = (key_present, value_present) # kv share: [bs, n_kv_head, seq, head_dim] -> [bs, n_head, seq, head_dim] # q, k, v: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim], [bs, n_head, seq, head_dim] if self.use_flash_attention: attention = self.flash_attention(query, key, value, mask) attention = self._merge_heads(attention) else: attention = self._attn(query, key, value, mask) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] output = self.wo(attention) # dp, mp -> dp, 1 / dp * mp, 1 output = self.cast(output, ori_dtype) return output, layer_present def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _merge_heads(self, x): """ convert a 4d input to a 2d or 3d output Inputs: x: input tensor Output: x_merge: the 2d output """ # [bs, n_head, seq/1, head_dim] x = self.merger_head_transpose(x, (0, 2, 1, 3)) # dp,mp,1,1 -> dp,1,mp,1 # [bs, seq/1, n_head, head_dim] x_shape = x.shape if self.compute_in_2d: # [bs * seq/1, hidden_dim] new_shape = (-1, x_shape[-2] * x_shape[-1]) else: # [bs, seq/1, hidden_dim] new_shape = (x_shape[0], x_shape[1], -1) x_merge = self.reshape(x, new_shape) return x_merge def _attn(self, query, key, value, mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix mask: the attention mask adder matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # q, k: [bs, n_head, seq/1, head_dim], [bs, n_head, seq, head_dim] score = self.batch_matmul_q_k(query, key) # score: [bs, n_head, seq/1, seq] score = self.mul(score, self.inv_norm_factor) score = self.add(mask, score) attention_probs = self.softmax(self.cast_attn(score, self.softmax_dtype)) # score, v: [bs, n_head, seq/1, seq], [bs, n_head, seq, head_dim] weighted_values = self.batch_matmul(self.cast(attention_probs, self.dtype), value) # [bs, n_head, seq/1, head_dim] attention_merge = self._merge_heads(weighted_values) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] return attention_merge class LLamaDecodeLayer(nn.Cell): r""" Transformer Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. seq_length(int): The input sequence length. layer_id(int): The layer id of current transformer block layer. dim(int): The hidden size of the input. num_heads(int): The number of the heads. multiple_of(int): The SwiGLU hidden layer size multiple of large power of 2. norm_eps (float): The epsilon value of the denominator. Default 1e-5. compute_dtype(dtype.Number): The computation type of the layer. Should be mstype.float32 or mstype.float16. Default mstype.float32. layernorm_compute_type(dtype.Number): The computation type of the norm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **freqs_cis** (Tuple) - The precompute freqs and mask for rotary position embedding used in attention. - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, head_dim, seq_length), (batch_size, num_heads, seq_length, head_dim)). """ def __init__(self, batch_size, seq_length, layer_id, layer_num, dim: int = 512, n_heads: int = 8, multiple_of: int = 256, n_kv_heads: Optional[int] = None, ffn_dim_multiplier: Optional[int] = None, mla_multiplier: Optional[int] = None, norm_eps: float = 1e-5, compute_dtype=mstype.float16, layernorm_compute_dtype=mstype.float32, softmax_compute_dtype=mstype.float32, rotary_dtype=mstype.float32, param_init_type=mstype.float32, use_past=False, use_flash_attention=False, compute_in_2d=False, use_past_shard=False, mup_width_base=-1, depth_scale=1.0, init_std=0.01, parallel_config=TransformerOpParallelConfig()): super().__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size self.compute_in_2d = compute_in_2d self.seq_length = seq_length self.layer_id = layer_id self.hidden_size = dim self.n_head = n_heads self.head_dim = self.hidden_size // self.n_head self.n_kv_head = n_heads if n_kv_heads is None else n_kv_heads self.dtype = compute_dtype self.is_first_iteration = True self.use_past = use_past self.compute_in_2d = compute_in_2d self.key_past = None self.value_past = None self.scale_depth = Tensor([depth_scale / math.sqrt(layer_num)], dtype=mstype.float16) self.depth_mul = P.Mul() self.reshape = P.Reshape() self.add = P.Add() self.attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.post_attention_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) self.post_ffn_norm = LlamaRMSNorm(self.hidden_size, norm_eps, compute_type=layernorm_compute_dtype) if mla_multiplier is None: self.attention = LLamaAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, n_kv_heads=n_kv_heads, mup_width_base=mup_width_base, init_std=init_std, compute_dtype=compute_dtype, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) else: self.attention = LLamaMLAAttention(batch_size=batch_size, seq_length=seq_length, dim=dim, n_heads=n_heads, compute_dtype=compute_dtype, mla_multiplier=mla_multiplier, softmax_compute_dtype=softmax_compute_dtype, rotary_dtype=rotary_dtype, param_init_type=param_init_type, use_past=use_past, use_flash_attention=use_flash_attention, compute_in_2d=compute_in_2d, use_past_shard=use_past_shard, parallel_config=parallel_config) self.feed_forward = LlamaFeedForward(dim=self.hidden_size, hidden_dim=4 * self.hidden_size, multiple_of=multiple_of, ffn_dim_multiplier=ffn_dim_multiplier, mup_width_base=mup_width_base, init_std=init_std, compute_dtype=compute_dtype, param_init_type=param_init_type) dp = parallel_config.data_parallel mp = parallel_config.model_parallel if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.feed_forward.shard(parallel_config) if self.compute_in_2d: self.add.shard(((dp, 1), (dp, 1))) self.attention_norm.shard((dp, 1)) self.ffn_norm.shard((dp, 1)) self.post_attention_norm.shard((dp, 1)) self.post_ffn_norm.shard((dp, 1)) self.depth_mul.shard(((dp, 1), (1,))) else: self.add.shard(((dp, 1, 1), (dp, 1, 1))) self.attention_norm.shard((dp, 1, 1)) self.ffn_norm.shard((dp, 1, 1)) self.post_attention_norm.shard((dp, 1, 1)) self.post_ffn_norm.shard((dp, 1, 1)) self.feed_forward.mul.shard(((dp, 1, mp), (dp, 1, mp))) self.depth_mul.shard(((dp, 1, 1), (1,))) if parallel_config.use_seq_parallel and self.is_first_iteration: if self.compute_in_2d: self.add.shard(((dp * mp, 1), (dp * mp, 1))) self.attention_norm.shard((dp * mp, 1)) self.ffn_norm.shard((dp * mp, 1)) self.post_attention_norm.shard((dp * mp, 1)) self.post_ffn_norm.shard((dp * mp, 1)) self.depth_mul.shard(((dp * mp, 1), (1,))) else: self.add.shard(((dp, mp, 1), (dp, mp, 1))) self.attention_norm.shard((dp, mp, 1)) self.ffn_norm.shard((dp, mp, 1)) self.post_attention_norm.shard((dp, mp, 1)) self.post_ffn_norm.shard((dp, mp, 1)) self.depth_mul.shard(((dp, mp, 1), (1,))) self.feed_forward.w2.shard(((dp, mp), (1, mp)), out_strategy_matmul=((dp * mp, 1),)) if self.use_past: kv_shape = (batch_size, self.n_kv_head, seq_length, self.head_dim) self.key_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(kv_shape), self.dtype), name="value_past") self.ones = P.Ones() self.mul_past = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.assign_past = P.Assign().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) if use_past_shard: self.mul_past.shard(((dp, mp, 1, 1), (1,))) self.assign_past.shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) def construct(self, x, freqs_cis, mask=None, init_reset=True, batch_valid_length=None): """ Forward of transformer block. """ self._check_input(x, freqs_cis, mask, init_reset, batch_valid_length) # [bs, seq/1, hidden_dim] (first) [bs * seq/1, hidden_dim] (others) if self.compute_in_2d and x.ndim != 2: x = self.reshape(x, (-1, x.shape[-1])) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] input_x = self.attention_norm(x) key_reset = None value_reset = None if self.use_past and self.is_first_iteration: # reset states, init_reset True for reuse and False for reset self.assign_past(self.key_past, self.mul_past(self.key_past, self.cast(init_reset, self.dtype))) self.assign_past(self.value_past, self.mul_past(self.value_past, self.cast(init_reset, self.dtype))) key_reset = self.key_past value_reset = self.value_past # add dependency for desired execution order input_x = ops.depend(input_x, key_reset) input_x = ops.depend(input_x, value_reset) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] h, layer_present = self.attention(input_x, freqs_cis, mask, self.key_past, self.value_past, batch_valid_length) h = self.post_attention_norm(h) h = self.add(x, h) ffn_norm = self.ffn_norm(h) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.feed_forward(ffn_norm) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign_past(self.key_past, key_present) self.assign_past(self.value_past, value_present) key_update = self.key_past value_update = self.value_past # add dependency for desired execution order key_update = ops.depend(key_update, key_reset) value_update = ops.depend(value_update, value_reset) # add dependency for desired execution order ffn_out = ops.depend(ffn_out, value_update) ffn_out = ops.depend(ffn_out, key_update) # [bs, seq/1, hidden_dim] or [bs * seq/1, hidden_dim] ffn_out = self.post_ffn_norm(ffn_out) out = self.add(h, ffn_out) return out, layer_present def _check_input(self, x, freqs_cis, mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype( x.dtype, "x", [mstype.float32, mstype.float16], self.cls_name) freqs_cos, freqs_sin, swap_mask = freqs_cis _check_input_dtype(freqs_cos.dtype, "freqs_cos", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(freqs_sin.dtype, "freqs_sin", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(swap_mask.dtype, "swap_mask", [mstype.float32, mstype.float16], self.cls_name) if mask is not None: _check_input_dtype(mask.dtype, "input_mask", [mstype.float32, mstype.float16], self.cls_name) if self.use_past: if not isinstance(init_reset, Tensor): init_reset = Tensor([init_reset], mstype.bool_) if not isinstance(batch_valid_length, Tensor): bs = freqs_cis[0].shape[0] batch_valid_length = self.ones((bs, 1), mstype.int32) _check_input_dtype(init_reset.dtype, "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(batch_valid_length.dtype, "batch_valid_length", [mstype.int32], self.cls_name) return True pengcheng.mind-mindformer/mindformers/models/mae/000077500000000000000000000000001476650141400224645ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/mae/__init__.py000066400000000000000000000015561476650141400246040ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Model API.""" from .mae import * from .mae_config import * from .mae_processor import * __all__ = [] __all__.extend(mae.__all__) __all__.extend(mae_config.__all__) __all__.extend(mae_processor.__all__) pengcheng.mind-mindformer/mindformers/models/mae/convert_weight.py000066400000000000000000000065301476650141400260710ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import torch import mindspore as ms def convert_weight(torch_pth_path="mae_pretrain_vit_base.pth", ms_ckpt_path="mae_pretrain_vit_base.ckpt"): """ convert mae_vit_base_p16 weights from pytorch to mindspore pytorch and GPU required. """ param_dict = torch.load(torch_pth_path, map_location=torch.device("cpu")) new_dict = [] for k, v in param_dict["model"].items(): if k in ("cls_token", "mask_token"): k += "s" if "head" not in k: k = "vit." + k if "norm" in k: if "fc_norm" not in k and "vit.norm" not in k and "decoder_norm" not in k: k = k.replace("norm", "layernorm") if "weight" in k: k = k.replace("weight", "gamma") elif "bias" in k: k = k.replace("bias", "beta") if "mlp" in k: k = k.replace("mlp", "output") if "fc1" in k: k = k.replace("fc1", "mapping") if "weight" in k: v = v.transpose(-1, 0) elif "fc2" in k: k = k.replace("fc2", "projection") if "weight" in k: v = v.transpose(-1, 0) if "attn" in k: k = k.replace("attn", "attention") if "proj" in k: k = k.replace("proj", "projection") if "weight" in k: v = v.transpose(-1, 0) if "qkv" not in k: new_dict.append({"name": k, "data": ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = data.shape[0] // 3 new_dict.append({"name": k.replace(".qkv", ".dense1"), "data": data[:length]}) new_dict.append({"name": k.replace(".qkv", ".dense2"), "data": data[length:length*2]}) new_dict.append({"name": k.replace(".qkv", ".dense3"), "data": data[length*2:]}) ms.save_checkpoint(new_dict, ms_ckpt_path) print("Weights conversion completes. ") if __name__ == "__main__": parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_pth_path", type=str, default="mae_pretrain_vit_base.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=True, default="mae_pretrain_vit_base.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_pth_path, opt.ms_ckpt_path) pengcheng.mind-mindformer/mindformers/models/mae/mae.py000066400000000000000000000430041476650141400236010ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/facebookresearch/mae # ============================================================================ """Mae Model.""" import numpy as np from mindspore import Tensor, Parameter, nn from mindspore import dtype as mstype from mindspore import ops as P import mindspore.common.initializer as weight_init from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.models.mae.mae_modules import Block, LayerNorm, Linear from mindformers.models.mae.mae_modules import PatchEmbed, Patchify, UnPatchify from mindformers.models.mae.mae_modules import get_2d_sincos_pos_embed from mindformers.core.loss import MSELoss from mindformers.models.mae.mae_config import ViTMAEConfig __all__ = ['ViTMAEModel', 'ViTMAEForPreTraining'] @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTMAEModel(BaseModel): """ Pretrain MAE Module. The supported model name could be selected from ViTMAEConfig.show_support_list(). Args: config (ViTMAEConfig): the config of Mae model. Examples: >>> # input model name >>> model_a = ViTMAEModel.from_pretrained('mae_vit_base_p16') >>> # input config >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('mae_vit_base_p16') >>> model_b = ViTMAEModel(config) """ _support_list = MindFormerBook.get_model_support_list()['mae'] def __init__(self, config=None): config = config if config else ViTMAEConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel # -------------------------------------------------------------------------- # MAE encoder specifics self.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, parallel_config=config.parallel_config) num_patches = self.patch_embed.num_patches seq_length = int((1 - config.mask_ratio) * num_patches) + 1 self.seq_length = seq_length self.num_masked = num_patches - seq_length + 1 self.cls_tokens = Parameter( weight_init.initializer(weight_init.Normal(sigma=config.initializer_range), (1, 1, config.embed_dim)), requires_grad=True) self.num_patches = num_patches self.pos_embed = Parameter( weight_init.initializer(weight_init.Zero(), (1, num_patches + 1, config.embed_dim)), name='pos_embed', requires_grad=True) # stochastic depth decay rule hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.depth)] parallel_config_args = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config self.blocks = nn.CellList([ Block(hidden_size=config.embed_dim, ffn_hidden_size=config.intermediate_size, seq_length=seq_length, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=hdr[i], layer_norm_eps=config.layer_norm_eps, qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=config.window_size, num_heads=config.num_heads, hidden_act=config.hidden_act, moe_config=config.moe_config, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config_args) for i in range(config.depth)]) self.norm = LayerNorm((config.embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) # -------------------------------------------------------------------------- self.stride_slice = P.StridedSlice().shard(((1, 1, 1),)) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.expand_dim = P.ExpandDims().shard(((dp, 1),)) self.tile = P.Tile().shard(((dp, 1, 1),)) self.gather = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.cat = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.init_weights() self.init_pos_emd() def init_pos_emd(self): """init values of pos_embed""" encoder_pos_emd = Tensor( get_2d_sincos_pos_embed(self.pos_embed.shape[-1], int(self.num_patches ** .5), cls_token=True), mstype.float32 ) encoder_pos_emd = P.ExpandDims()(encoder_pos_emd, 0) self.pos_embed = Parameter(encoder_pos_emd, name='sincos_pos_embedding', requires_grad=False) def init_weights(self): """ ViT weight initialization.""" for name, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) if name == "patch_embed.proj": cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def construct(self, image, unmask_index): """construct of VisionTransformerForMae Encoder""" # patch to encoder tokens and add positions tokens = self.patch_embed(image) encoder_pos_embedding = self.stride_slice( self.pos_embed, (0, 1, 0), (1, self.pos_embed.shape[1], self.pos_embed.shape[2]), (1, 1, 1)) tokens = self.add(tokens, encoder_pos_embedding) # get the unmasked tokens to be encoded unmask_index_ = self.expand_dim(unmask_index, -1) unmask_index = self.tile(unmask_index_, (1, 1, tokens.shape[2])) unmask_tokens = self.gather(tokens, 1, unmask_index) # cls_tokens add pos_embedding cls_pos_embedding = self.stride_slice( self.pos_embed, (0, 0, 0), (1, 1, self.pos_embed.shape[2]), (1, 1, 1)) batch_size = image.shape[0] if batch_size == 1: cls_tokens = self.cls_tokens else: cls_tokens = self.tile(self.cls_tokens, (batch_size, 1, 1)) cls_tokens = self.add(cls_tokens, cls_pos_embedding) # concat cls_tokens encoded_tokens = self.cat((cls_tokens, unmask_tokens)) # attend with vision transformer encoder_input_mask = P.Ones()((batch_size, self.seq_length, self.seq_length), mstype.float32) for block in self.blocks: encoded_tokens = block(encoded_tokens, encoder_input_mask) encoded_tokens = self.norm(encoded_tokens) return encoded_tokens @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTMAEForPreTraining(BaseModel): """ Pretrain MAE Module. The supported model name could be selected from ViTMAEConfig.show_support_list(). Args: config (ViTMAEConfig): the config of Mae model. Examples: >>> # input model name >>> model_a = ViTMAEForPreTraining.from_pretrained('mae_vit_base_p16') >>> # input config >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('mae_vit_base_p16') >>> model_b = ViTMAEForPreTraining(config) """ _support_list = MindFormerBook.get_model_support_list()['mae'] def __init__(self, config=None): config = config if config else ViTMAEConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.vit = ViTMAEModel(config) self.num_patches = num_patches = self.vit.patch_embed.num_patches self.num_masked = num_patches - self.vit.seq_length + 1 # -------------------------------------------------------------------------- # MAE decoder specifics self.decoder_embed = Linear( config.embed_dim, config.decoder_embed_dim, weight_init="xavier_uniform", compute_dtype=mstype.float16).to_float(mstype.float16) self.decoder_embed.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.mask_tokens = Parameter( weight_init.initializer(weight_init.Normal(sigma=config.initializer_range), (1, 1, config.decoder_embed_dim)), name='mask_tokens', requires_grad=True) self.decoder_pos_embed = Parameter( weight_init.initializer(weight_init.Zero(), (1, num_patches + 1, config.decoder_embed_dim)), name='pos_embedding', requires_grad=False) hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.decoder_depth)] self.decoder_blocks = nn.CellList([ Block(hidden_size=config.decoder_embed_dim, ffn_hidden_size=config.decoder_intermediate_size, seq_length=num_patches + 1, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, layer_norm_eps=config.layer_norm_eps, hidden_dropout_rate=hdr[i], qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=config.window_size, num_heads=config.decoder_num_heads, hidden_act=config.hidden_act, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config.dp_mp_config) for i in range(config.decoder_depth)]) self.decoder_norm = LayerNorm((config.decoder_embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) patch_dim = config.in_chans * config.patch_size ** 2 self.decoder_pred = Linear( config.decoder_embed_dim, patch_dim, weight_init="xavier_uniform", compute_dtype=mstype.float16).to_float(mstype.float16) self.decoder_pred.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) # -------------------------------------------------------------------------- self.patchify = Patchify(patch_size=config.patch_size, parallel_config=parallel_config) self.unpatchify = UnPatchify( patch_size=config.patch_size, seq_length=num_patches, parallel_config=parallel_config) self.stride_slice = P.StridedSlice().shard(((1, 1, 1),)) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.expand_dim = P.ExpandDims().shard(((dp, 1),)) self.tile = P.Tile().shard(((dp, 1, 1),)) self.gather = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.cat = P.Concat(axis=1).shard(((dp, 1, 1), (dp, 1, 1))) self.gather1 = P.GatherD().shard(((dp, 1), (dp, 1))) self.gather2 = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) self.mse_loss = MSELoss(config.norm_pixel_loss, parallel_config) self.images_summary = P.ImageSummary().shard(((dp, 1, 1, 1),)) self.init_weights() self.init_pos_emd() self.load_checkpoint(config) def init_pos_emd(self): """init values of pos_embed""" decoder_pos_embed = Tensor( get_2d_sincos_pos_embed(self.decoder_pos_embed.shape[-1], int(self.num_patches ** .5), cls_token=True), mstype.float32 ) decoder_pos_embed = P.ExpandDims()(decoder_pos_embed, 0) self.decoder_pos_embed = Parameter(decoder_pos_embed, name='decoder_pos_embed', requires_grad=False) def init_weights(self): """ ViT weight initialization.""" for _, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) def construct(self, image, mask, ids_restore, unmask_index): """construct of VisionTransformerForMae""" self.images_summary("input images", image) encoder_tokens = self.vit(image, unmask_index) unmask_tokens = self.decoder_embed(encoder_tokens) unmask_tokens = self.cast(unmask_tokens, mstype.float32) # mask tokens add the positions using the masked indices derived above batch_size = encoder_tokens.shape[0] mask_tokens = self.tile(self.mask_tokens, (batch_size, self.num_masked, 1)) # concat the masked tokens to the decoder tokens and attend with decoder img_tokens = self.stride_slice( unmask_tokens, (0, 1, 0), (unmask_tokens.shape[0], unmask_tokens.shape[1], unmask_tokens.shape[2]), (1, 1, 1)) full_tokens_ = self.cat((img_tokens, mask_tokens)) ids_restore_copy = ids_restore ids_restore_ = self.expand_dim(ids_restore_copy, -1) ids_restore_ = self.tile(ids_restore_, (1, 1, unmask_tokens.shape[2])) full_tokens_ = self.gather2(full_tokens_, 1, ids_restore_) cls_tokens = self.stride_slice( unmask_tokens, (0, 0, 0), (unmask_tokens.shape[0], 1, unmask_tokens.shape[2]), (1, 1, 1)) decoder_tokens = self.cat((cls_tokens, full_tokens_)) # add position embendding for decoder tokens decoder_tokens = self.add(decoder_tokens, self.decoder_pos_embed) # decoder attention_mask = Tensor(np.ones((batch_size, self.num_patches + 1, self.num_patches + 1)), mstype.float32) for block in self.decoder_blocks: decoder_tokens = block(decoder_tokens, attention_mask) # normalize decoder tokens decoder_tokens = self.decoder_norm(decoder_tokens) # splice out the mask tokens and project to pixel values pred = self.decoder_pred(decoder_tokens) pred = self.cast(pred, mstype.float32) pred = self.stride_slice(pred, (0, 1, 0), (pred.shape[0], pred.shape[1], pred.shape[2]), (1, 1, 1)) reconstruct_images = self.unpatchify(pred) self.images_summary("reconstruct image", reconstruct_images) if not self.training: return reconstruct_images patches = self.patchify(image) mask = self.gather1(mask, 1, ids_restore) mae_loss = self.mse_loss(pred, patches, mask) return mae_loss pengcheng.mind-mindformer/mindformers/models/mae/mae_config.py000066400000000000000000000114651476650141400251340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) __all__ = ['ViTMAEConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ViTMAEConfig(BaseConfig): """ Config for Mae model Examples: >>> # init a config with a model name >>> config_a = ViTMAEConfig.from_pretrained('mae_vit_base_p16') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'mae', 'run_mae_vit_base_p16_224_800ep.yaml') >>> config_b = ViTMAEConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = ViTMAEConfig( >>> patch_size=16, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['mae'] def __init__(self, mask_ratio: float = 0.75, image_size: int = 224, patch_size: int = 16, num_channels: int = 3, initializer_range: float = 0.02, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, qkv_bias: bool = True, hidden_act: str = "gelu", post_layernorm_residual: bool = False, layer_norm_eps: float = 1e-6, attention_probs_dropout_prob: float = 0.0, hidden_dropout_prob: float = 0.0, drop_path_rate: float = 0., decoder_hidden_size: int = 512, decoder_num_hidden_layers: int = 8, decoder_num_attention_heads: int = 16, decoder_intermediate_size: int = 2048, norm_pix_loss: bool = True, checkpoint_name_or_path: str = '', layernorm_compute_type: mstype = mstype.float32, softmax_compute_type: mstype = mstype.float32, param_init_type: mstype = mstype.float32, parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs): super().__init__(**kwargs) self.mask_ratio = mask_ratio self.image_size = image_size self.patch_size = patch_size self.in_chans = num_channels self.initializer_range = initializer_range self.embed_dim = hidden_size self.depth = num_hidden_layers self.num_heads = num_attention_heads self.intermediate_size = intermediate_size self.qkv_bias = qkv_bias self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.layer_norm_eps = layer_norm_eps self.attention_dropout_rate = attention_probs_dropout_prob self.drop_rate = hidden_dropout_prob self.drop_path_rate = drop_path_rate self.decoder_embed_dim = decoder_hidden_size self.decoder_depth = decoder_num_hidden_layers self.decoder_num_heads = decoder_num_attention_heads self.decoder_intermediate_size = decoder_intermediate_size self.norm_pixel_loss = norm_pix_loss self.checkpoint_name_or_path = checkpoint_name_or_path self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.parallel_config = parallel_config self.moe_config = moe_config pengcheng.mind-mindformer/mindformers/models/mae/mae_modules.py000066400000000000000000001066711476650141400253430ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Modules of ViTMAEForPreTraining, including Linear, Block, MLP, Attention, PatchEmbed, etc. """ import math import numpy as np from mindspore import nn, Parameter, Tensor, context, ParallelMode import mindspore.common.dtype as mstype from mindspore.ops import operations as P, constexpr from mindspore.ops import functional as F import mindspore.common.initializer as init from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.moe import default_moe_config, _check_moe_config from mindformers.modules.transformer.transformer import default_dpmp_config, _check_config @constexpr def gen_shape(x_shape, ndim): return (x_shape,) + (1,) * (ndim + 1) def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False): """ grid_size: int of the grid height and width return: pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token) """ grid_h = np.arange(grid_size, dtype=np.float32) grid_w = np.arange(grid_size, dtype=np.float32) grid = np.meshgrid(grid_w, grid_h) # here w goes first grid = np.stack(grid, axis=0) grid = grid.reshape([2, 1, grid_size, grid_size]) pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid) if cls_token: pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed], axis=0) return pos_embed def get_2d_sincos_pos_embed_from_grid(embed_dim, grid): assert embed_dim % 2 == 0 # use half of dimensions to encode grid_h emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2) emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2) emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D) return emb def get_1d_sincos_pos_embed_from_grid(embed_dim, pos): """ embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D) """ assert embed_dim % 2 == 0 omega = np.arange(embed_dim // 2, dtype=np.float32) omega /= embed_dim / 2. omega = 1. / 10000 ** omega # (D/2,) pos = pos.reshape(-1) # (M,) out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product emb_sin = np.sin(out) # (M, D/2) emb_cos = np.cos(out) # (M, D/2) emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D) return emb class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 r""" A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-6, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Identity(nn.Cell): def construct(self, x): return x class RelativePositionBias(nn.Cell): """relative position bias""" def __init__(self, window_size, num_heads): super(RelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3 self.relative_position_bias_table = Parameter( init.initializer( init.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = \ np.zeros(((window_size[0] * window_size[1] + 1),) * 2, dtype=int) relative_position_index[1:, 1:] = np.sum(relative_coords, axis=-1) # Wh*Ww, Wh*Ww relative_position_index[0, 0:] = self.num_relative_distance - 3 relative_position_index[0:, 0] = self.num_relative_distance - 2 relative_position_index[0, 0] = self.num_relative_distance - 1 relative_position_index = Tensor(relative_position_index, mstype.int32) self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1] + 1, self.window_size[0] * self.window_size[1] + 1, -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) return relative_position_bias class Linear(layers.Linear): # pylint: disable=W0212 r""" Linear function for RingMo. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of layer""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Dropout(layers.Dropout): # pylint: disable=W0212 r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob): super(DropPath, self).__init__() self.drop = Dropout(keep_prob=1 - drop_prob) self.mask = Tensor(np.ones(1,), dtype=mstype.float32) self.tile = P.Tile() self.mul = P.Mul() def construct(self, x): if not self.training: return x mask = self.tile(self.mask, (x.shape[0],) + (1,) * (x.ndim-1)) out = self.drop(mask) out = self.mul(out, x) return out def shard(self, strategy): self.mul.shard(strategy) class Block(nn.Cell): """Block of ringmo""" def __init__(self, hidden_size, ffn_hidden_size, num_heads, seq_length, drop_rate=0., attention_dropout_rate=0., hidden_dropout_rate=0., layer_norm_eps=1e-6, qkv_bias=True, window_size=None, post_layernorm_residual=False, init_values=None, weight_init='XavierUniform', layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(Block, self).__init__() _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.seq_length = seq_length self.hidden_size = hidden_size self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) parallel_config_args = parallel_config.dpmp if self.use_moe else parallel_config self.attention = Attention(src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, window_size=window_size, num_heads=num_heads, weight_init=weight_init, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, qkv_bias=qkv_bias, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=parallel_config_args) self.output = MLP(hidden_size=hidden_size, dropout_rate=drop_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, weight_init=weight_init, hidden_act=hidden_act, use_dropout=False, parallel_config=parallel_config) if init_values is not None: self.gamma_1 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma1", requires_grad=True) self.gamma_2 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma2", requires_grad=True) else: self.gamma_1, self.gamma_2 = None, None self.mul_gamma = P.Mul().shard(((parallel_config.data_parallel, 1), (1,))) self.drop_path = DropPath(hidden_dropout_rate) self.drop_path.shard(((parallel_config.data_parallel, 1),)) self.drop_path3d = DropPath(hidden_dropout_rate) self.drop_path3d.shard(((parallel_config.data_parallel, 1, 1),)) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.reshape = P.Reshape() def construct(self, x, input_mask, rel_pos_bias=None): """construct of Block""" x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) attention = self.attention( input_x, input_x, input_x, input_mask, rel_pos_bias) if self.gamma_1 is not None: attention = self.mul_gamma(attention, self.gamma_1) if len(x_shape) == 3: attention = P.Reshape()(attention, x_shape) attention = self.drop_path3d(attention) attention = P.Reshape()(attention, (-1, x_shape[-1])) else: attention = self.drop_path(attention) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) if self.gamma_2 is not None: mlp_logit = self.mul_gamma(mlp_logit, self.gamma_2) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) mlp_logit = self.drop_path3d(mlp_logit) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: mlp_logit = self.drop_path(mlp_logit) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) return output class MLP(FeedForward): r"""MLP for ring-mo.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='XavierUniform', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(MLP, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class Attention(nn.Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following """ def __init__(self, src_seq_length, tgt_seq_length, hidden_size, num_heads, window_size=None, hidden_dropout_rate=0., attention_dropout_rate=0., qkv_bias=True, weight_init='XavierUniform', compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(Attention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] _check_config(parallel_config) self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - hidden_dropout_rate) self.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = Dropout(1 - attention_dropout_rate) self.prob_dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) if window_size: self.relative_position_bias = RelativePositionBias(window_size, num_heads) else: self.window_size = None self.relative_position_bias_table = None self.relative_position_index = None self.relative_position_bias = None self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1))) self.expand_dims_rpb = P.ExpandDims().shard(((1, 1, 1),)) self.add_rpb = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1, 1))) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, rel_pos_bias=None): """construct of attention""" ori_shape = F.shape(query_tensor) batch_size = F.shape(attention_mask)[0] query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask, rel_pos_bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) return output def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, rel_pos_bias): r"""Get the weighted score along the seq_length.""" # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # add window cross module if self.relative_position_bias is not None: relative_position_bias = self.expand_dims_rpb(self.relative_position_bias(), 0) attention_scores = self.add_rpb(attention_scores, relative_position_bias) if rel_pos_bias is not None: attention_scores = self.add_3d(attention_scores, rel_pos_bias) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge def _merge_heads(self, x): """ convert a 4d input to a 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor class PatchEmbed(nn.Cell): """Construct the embeddings from patch, position embeddings.""" def __init__(self, img_size=224, patch_size=16, in_features=3, out_features=768, norm_layer=False, parallel_config=None): super(PatchEmbed, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 img_size = (img_size, img_size) patch_size = (patch_size, patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.proj = nn.Conv2d( in_channels=in_features, out_channels=out_features, kernel_size=patch_size, stride=patch_size, weight_init=init.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.proj.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.proj.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm self.norm = LayerNorm((out_features,), eps=1e-6).shard(((dp, 1, 1),)) if norm_layer else Identity() def construct(self, x): """construct""" # True x: bs False x: bs * dp x = self.proj(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) return x def get_kernel_size(patch_size): """ input: 2^i & i <= 5 | 14 output: a list of kernel size """ x = None y = None z = None ans = False for i in range(1, patch_size + 1): if patch_size % i == 0: x = i mul_y_z = patch_size // i for j in range(1, mul_y_z + 1): if mul_y_z % j == 0: y = j z = mul_y_z // j if x >= y >= z: ans = True break if ans: break if not ans: raise ValueError(patch_size) return [x, y, z] class Patchify(nn.Cell): """Patchify""" def __init__(self, patch_size, parallel_config=None): super(Patchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.patch_size = patch_size self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, img): p = self.patch_size bs, channels, h, w = img.shape x = self.reshape(img, (bs, channels, h // p, p, w // p, p)) x = self.transpose(x, (0, 2, 4, 3, 5, 1)) patches = self.reshape(x, (bs, (h // p) * (w // p), channels * p * p)) return patches class UnPatchify(nn.Cell): """UnPatchify""" def __init__(self, patch_size, seq_length, parallel_config=None): super(UnPatchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.p = patch_size self.h = self.w = int(seq_length ** .5) assert self.h * self.w == seq_length self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): bs = x.shape[0] x = self.reshape(x, (bs, self.h, self.w, self.p, self.p, 3)) x = self.transpose(x, (0, 5, 1, 3, 2, 4)) images = self.reshape(x, (bs, 3, self.h * self.p, self.w * self.p)) return images pengcheng.mind-mindformer/mindformers/models/mae/mae_processor.py000066400000000000000000000137271476650141400257110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ ViTMAEProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision.transforms import ToTensor, Normalize from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['ViTMAEProcessor', 'ViTMAEImageProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTMAEImageProcessor(BaseImageProcessor): """ ViTMAEImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224, patch_size=16, mask_ratio=0.75): super(ViTMAEImageProcessor, self).__init__(image_resolution=size) self.resize = Resize((size, size), interpolation='cubic') self.to_tensor = ToTensor() self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) if not 0 < mask_ratio < 1: raise ValueError('masking ratio must be kept between 0 and 1, but get mask_ratio {mask_ratio}.') # seq_length self.num_patches = (size // patch_size) ** 2 # seq masked number self.keep_num = int((1 - mask_ratio) * self.num_patches) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] ids_restores = [] masks = [] unmask_indexes = [] for image in images: image = self.resize(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) rand_indices = np.argsort( np.random.uniform(size=(self.num_patches,)), axis=0).astype(np.int32) ids_restore = np.argsort(rand_indices, axis=0).astype(np.int32) ids_restores.append(ids_restore) mask = np.ones((self.num_patches,)).astype(np.int32) mask[:self.keep_num] = 0 masks.append(mask) unmask_index = rand_indices[:self.keep_num] unmask_indexes.append(unmask_index) return Tensor(res), Tensor(masks), Tensor(ids_restores), Tensor(unmask_indexes) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTMAEProcessor(BaseProcessor): """ ViTMAEProcessor, consists of a feature extractor (BaseFeatureEXtractor) for image input. Examples: >>> from mindformers import MindFormerBook >>> from mindformers.models import ViTMAEProcessor >>> yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", ... "mae", "model_config", "mae_vit_base_p16.yaml") >>> # build ViTMAEProcessor from pretrained >>> pro_a = ViTMAEProcessor.from_pretrained('mae_vit_base_p16') >>> # build ViTMAEProcessor from config >>> pro_b = ViTMAEProcessor.from_pretrained(yaml_path) """ _support_list = MindFormerBook.get_processor_support_list()['mae'] def __init__(self, image_processor=None, return_tensors='ms'): super(ViTMAEProcessor, self).__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-mindformer/mindformers/models/pangualpha/000077500000000000000000000000001476650141400240425ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/pangualpha/__init__.py000066400000000000000000000017761476650141400261660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for PanGu Alpha""" from .pangualpha import * from .pangualpha_config import * from .pangualpha_tokenizer import * from .pangualpha_processor import * __all__ = [] __all__.extend(pangualpha.__all__) __all__.extend(pangualpha_config.__all__) __all__.extend(pangualpha_tokenizer.__all__) __all__.extend(pangualpha_processor.__all__) pengcheng.mind-mindformer/mindformers/models/pangualpha/pangualpha.py000066400000000000000000000673371476650141400265540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanGuAlpha model""" import copy import numpy as np import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Tensor from mindspore.common.initializer import initializer from mindspore.ops import operations as P from mindspore.ops import functional as F from mindformers.models.base_model import BaseModel from mindformers.core.loss import CrossEntropyLoss from mindformers.modules.layers import LayerNorm from mindformers.modules.transformer import VocabEmbedding, TransformerEncoder, TransformerEncoderLayer, \ AttentionMask, MoEConfig from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from .pangualpha_config import PanguAlphaConfig __all__ = ['PanguAlphaHeadModel', 'PanguAlphaModel', 'PanguAlphaPromptTextClassificationModel'] class EmbeddingLayer(nn.Cell): r"""Embedding layer of the PanGuAlpha Model""" def __init__(self, config): super(EmbeddingLayer, self).__init__() # Only for the pipeline mode, the embedding needs to be row sliced. self.word_embedding = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, param_init=initializer("normal", [config.vocab_size, config.hidden_size], dtype=mstype.float32), parallel_config=config.parallel_config.embedding_dp_mp_config) copied_parallel_config = copy.deepcopy(config.parallel_config) copied_parallel_config.vocab_emb_dp = True self.position_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer("normal", [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=copied_parallel_config.embedding_dp_mp_config) self.add = P.Add().shard( ((config.parallel_config.data_parallel, 1, 1), (config.parallel_config.data_parallel, 1, 1))) self.dropout = nn.Dropout(keep_prob=1-config.embedding_dropout_prob) self.dropout.dropout.shard(((config.parallel_config.data_parallel, 1, 1),)) self.is_first_iteration = True self.use_past = config.use_past self.batch_size = config.batch_size def construct(self, input_ids, input_position, batch_valid_length): r"""forward pass of the layer""" word_embedding, word_table = self.word_embedding(input_ids) if self.use_past and not self.is_first_iteration: _, seq_length = F.shape(input_ids) input_position = batch_valid_length.view(self.batch_size, seq_length) position_embedding, _ = self.position_embedding(input_position) embed = self.add(word_embedding, position_embedding) embed = self.dropout(embed) return embed, word_table def get_word_embedding_weight(self): r"""get word embedding weight""" return self.word_embedding.embedding_table class QueryLayer(TransformerEncoderLayer): r"""Query Layer at the final layer.""" def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, param_init_type=mstype.float32, hidden_act='fast_gelu', use_past=False, parallel_config=None, softmax_compute_type=mstype.float32): super(QueryLayer, self).__init__(batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, hidden_act=hidden_act, use_past=use_past, parallel_config=parallel_config.dp_mp_config, softmax_compute_type=softmax_compute_type) # pylint: disable=W0221 def construct(self, x, query_vector, input_mask, init_reset=True, batch_valid_length=None): r""" The forward process of the block. """ # [bs * seq_length, embedding_size] input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(query_vector, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) return output, layer_present class PanguAlphaHead(nn.Cell): """ Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the backbone embedding_table: the embedding table of the vocabulary Returns: logits: Tensor, the logits of the corresponding inputs """ def __init__(self, hidden_size, compute_type=mstype.float16, parallel_config=None): super(PanguAlphaHead, self).__init__() if parallel_config.vocab_emb_dp: self.matmul = P.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = P.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), ( parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_type self.cast = P.Cast() def construct(self, state, embed): r"""forward pass of the head""" state = P.Reshape()(state, (-1, self.hidden_size)) # output logits over vocabulary [bs*seq_length, vocab_size] logits = self.matmul(self.cast(state, self.dtype), self.cast(embed, self.dtype)) return logits def set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs a offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # Used for the pipeline's stages setting # As the final layer is not included here, so we need to manually add here. # original: if set two stages, layers on two stages will be [15, 16+1] # with 1 added, the layers on two stages will be [16, 15 +1] pp_dis = max(int((layers + 1)/ parallel_config.pipeline_stage), 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id logger.info("pipeline stage id is %s", pp_id) # Used for optimizer's fusion tag dis = max(int((layers + 1) / parallel_config.gradient_aggregation_group), 1) if parallel_config.pipeline_stage > 1: # we give the fusion in pipeline mode a fixed value, otherwise the performance may become worse. network.set_comm_fusion(2) else: network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: network.recompute(recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) class PanguAlphaModel(nn.Cell): r"""The base backbone of the PanGuAlpha model""" def __init__(self, config): super(PanguAlphaModel, self).__init__() self.is_pipeline = config.parallel_config.pipeline_stage > 1 self.embedding = EmbeddingLayer(config) self.config = config self.layernorm = LayerNorm((config.hidden_size,)).to_float(mstype.float32) if config.parallel_config.pipeline_stage > 1: self.layernorm.set_comm_fusion(2) else: self.layernorm.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.layernorm.pipeline_stage = config.parallel_config.pipeline_stage - 1 # Configure the shard configure of the Embedding layer self.embedding.pipeline_stage = 0 self.num_layers = config.num_layers if config.use_moe: moe_config = MoEConfig(expert_num=config.expert_num, num_experts_chosen=config.per_token_num_experts_chosen) else: moe_config = MoEConfig(expert_num=1) # The shard setting of Transformer is set within the class StackedTransformer self.blocks = TransformerEncoder(num_layers=config.num_layers - 1, batch_size=config.batch_size, hidden_size=config.hidden_size, ffn_hidden_size=config.ffn_hidden_size, num_heads=config.num_heads, seq_length=config.seq_length, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, lambda_func=set_parallel_configure_for_layer, hidden_act=config.hidden_act, param_init_type=config.param_init_type, use_past=config.use_past, parallel_config=config.parallel_config, moe_config=moe_config, softmax_compute_type=config.softmax_compute_type).blocks copied_parallel_config = copy.deepcopy(config.parallel_config) copied_parallel_config.vocab_emb_dp = True self.top_query_embedding = VocabEmbedding(vocab_size=config.seq_length, embedding_size=config.hidden_size, param_init=initializer("normal", [config.seq_length, config.hidden_size], dtype=mstype.float32), parallel_config=copied_parallel_config.embedding_dp_mp_config) self.top_query_embedding.pipeline_stage = config.parallel_config.pipeline_stage - 1 if config.parallel_config.pipeline_stage > 1: self.top_query_embedding.set_comm_fusion(2) else: self.top_query_embedding.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.top_query_layer = QueryLayer(batch_size=config.batch_size, hidden_size=config.hidden_size, ffn_hidden_size=config.ffn_hidden_size, num_heads=config.num_heads, seq_length=config.seq_length, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, hidden_act=config.hidden_act, param_init_type=config.param_init_type, use_past=config.use_past, softmax_compute_type=config.softmax_compute_type, parallel_config=config.parallel_config) if isinstance(config.parallel_config.recompute, bool): if config.parallel_config.recompute: self.top_query_layer.recompute() else: if config.parallel_config.recompute.recompute: self.top_query_layer.recompute(recompute_slice_activation= config.parallel_config.recompute.recompute_slice_activation) self.top_query_layer.set_comm_fusion(config.parallel_config.gradient_aggregation_group) self.top_query_layer.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.dtype = mstype.float16 self.use_past = config.use_past self.add = P.TensorAdd() def construct(self, input_ids, input_position, encoder_masks, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" embed, word_table = self.embedding(input_ids, input_position, batch_valid_length) hidden_state = P.Cast()(embed, self.dtype) hidden_shape = F.shape(hidden_state) hidden_state = F.reshape(hidden_state, (-1, hidden_shape[-1])) # the input of the incremental prediction is 3d if self.blocks is not None: for i in range(self.num_layers - 1): hidden_state, _ = self.blocks[i](hidden_state, encoder_masks, init_reset, batch_valid_length) if self.use_past and not self.is_first_iteration: input_position = self.add(input_position, F.expand_dims(batch_valid_length, -1)) if self.is_pipeline: top_query_hidden_states, _ = self.top_query_embedding(input_position) top_query_hidden_states = self.reshape_to_2d(top_query_hidden_states) encoder_output, _ = self.top_query_layer(hidden_state, top_query_hidden_states, encoder_masks, init_reset, batch_valid_length) encoder_output = self.layernorm(encoder_output) else: hidden_state = self.reshape_to_2d(hidden_state) encoder_output = self.layernorm(hidden_state) encoder_output = P.Cast()(encoder_output, self.dtype) top_query_hidden_states, _ = self.top_query_embedding(input_position) top_query_hidden_states = self.reshape_to_2d(top_query_hidden_states) encoder_output, _ = self.top_query_layer(encoder_output, top_query_hidden_states, encoder_masks, init_reset, batch_valid_length) return encoder_output, word_table def reshape_to_2d(self, x): r"""reshape nd tensor to 2d, if n <= 2, keep original shape.""" shape = F.shape(x) if len(shape) <= 2: return x x = F.reshape(x, (-1, shape[-1])) return x @MindFormerRegister.register(MindFormerModuleType.MODELS) class PanguAlphaHeadModel(BaseModel): """ The PanguAlpha network consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network Inputs: input_ids: the tokenized inputs input_position(Tensor): current position, used by model.predict. attention_mask(Tensor): input sentences padding mask, where 0 indicates padding position. position_ids(Tensor): used to identify each token's position in the list of tokens. input_embeds(Tensor): reserved param, not used. labels(Tensor): the labels of corresponding input sequences. init_reset(bool, optional): A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Default True. batch_valid_length(Tensor): the past calculated the index with datatype int32, used for incremental prediction. Tensor of shape :math:`(batch_size,)`. Default None. Returns: logits: Tensor: the logits of the corresponding inputs with shape (batch_size, seq_length, vocab_size) Examples: >>> # input model name, load model and weights >>> model_a = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('pangualpha_2_6b') >>> model_b = PanguAlphaHeadModel(config) """ _support_list = MindFormerBook.get_model_support_list()['pangualpha'] def __init__(self, config: PanguAlphaConfig = None): config = config if config is not None else PanguAlphaConfig() super(PanguAlphaHeadModel, self).__init__(config) logger.info(config.parallel_config) # Network head to get logits over vocabulary copied_parallel_config = copy.deepcopy(config.parallel_config) logger.info(copied_parallel_config) if copied_parallel_config.pipeline_stage > 1: copied_parallel_config.vocab_emb_dp = False dp = config.parallel_config.data_parallel self.pad_token_id = config.pad_token_id self.get_attention_mask = AttentionMask(seq_length=config.seq_length, parallel_config=config.parallel_config.dp_mp_config) self.head = PanguAlphaHead(hidden_size=config.hidden_size, parallel_config=copied_parallel_config) self.head.pipeline_stage = config.parallel_config.pipeline_stage - 1 self.backbone = PanguAlphaModel(config) self.backbone.embedding.word_embedding.embedding_table.add_pipeline_stage(self.head.pipeline_stage) loss_parallel_config = copy.deepcopy(config.parallel_config) self.loss = CrossEntropyLoss(parallel_config=loss_parallel_config) self.reshape = P.Reshape() self.cast = P.Cast() self.tile = P.Tile().shard(((dp,),)) self.slice = P.StridedSlice().shard(((dp, 1),)) self.slice2 = P.StridedSlice().shard(((dp, 1, 1),)) self.not_equal = P.NotEqual().shard(((dp, 1), ())) self.input_position = Tensor(np.arange(config.seq_length), mstype.int32) self.expand = P.ExpandDims() self.add = P.Add().shard(((dp, 1), ())) self.gather = P.Gather() self.use_past = config.use_past self.is_first_iteration = True self.position_ids = Tensor(np.arange(config.seq_length), mstype.int32) self.all_ones_attention_mask = P.Ones()((1, 1, 1), mstype.float32) self.load_checkpoint(config) def prepare_inputs_for_generation(self, input_ids, **kwargs): input_position = kwargs.get("current_index", None) if input_position is not None: input_position = Tensor(input_position, mstype.int32) return { "input_ids": Tensor(input_ids, mstype.int32), "input_position": input_position } # pylint: disable=W0613 def construct(self, input_ids, input_position=None, attention_mask=None, position_ids=None, input_embeds=None, labels=None, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" batch_size, seq_length = input_ids.shape if self.training: seq_length = seq_length - 1 tokens = self.slice(input_ids, (0, 0), (batch_size, seq_length), (1, 1)) input_mask = F.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) input_position = self.slice(input_position, (0, 0), (batch_size, seq_length), (1, 1)) attention_mask = self.cast(attention_mask, mstype.float32) attention_mask = self.slice2(attention_mask, (0, 0, 0), (batch_size, seq_length, seq_length), (1, 1, 1)) else: tokens = input_ids input_mask = F.cast(self.not_equal(tokens, self.pad_token_id), mstype.float32) if attention_mask is None: attention_mask = self.get_attention_mask(input_mask) if not self.is_first_iteration: attention_mask = self.tile(self.all_ones_attention_mask, (batch_size, 1, 1)) if input_position is None or self.is_first_iteration: if batch_size == 1: input_position = F.reshape(self.position_ids, (1, seq_length)) else: input_position = self.tile(self.position_ids, (batch_size, 1)) # when incremental reasoning is not the first iteration, it goes into the following logic. else: bias = Tensor(np.arange(batch_size) * self.config.seq_length, mstype.int32) input_position = F.sub(input_position, bias) input_position = F.reshape(input_position, (batch_size, 1)) # [batch_size, seq_length, vocab_size] output_states, word_table = self.backbone(tokens, input_position, attention_mask, init_reset, batch_valid_length) logits = self.head(output_states, word_table) if not self.training: logits = logits.reshape((batch_size, seq_length, -1)) logits = logits.reshape((-1, logits.shape[-1])) if (not self.use_past or self.is_first_iteration) and input_position is not None: logits = self.gather(logits, input_position, 0) return logits, tokens, input_mask labels = self.slice(input_ids, (0, 1), (batch_size, seq_length + 1), (1, 1)) labels = self.reshape(labels, (-1,)) input_mask = self.reshape(input_mask, (-1,)) loss = self.loss(logits, labels, input_mask) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class PanguAlphaPromptTextClassificationModel(PanguAlphaHeadModel): """ The PanguAlpha network for prompt text classification consisting of two parts the backbone and the head Args: config(PanguAlphaConfig): the config of network Inputs: input_ids: the tokenized inputs, shape is [num_labels, seq_length] labels: the index of the true label, shape is [1,] attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, shape is [num_labels, seq_length] Returns: logits: Tensor: corresponding outputs for calculating metrics Examples: >>> # input model name, load model and weights >>> model_a = PanguAlphaHeadModel.from_pretrained('pangualpha_2_6b') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('pangualpha_2_6b') >>> model_b = PanguAlphaPromptTextClassificationModel(config) """ def __init__(self, config: PanguAlphaConfig = None): config = config if config is not None else PanguAlphaConfig() super().__init__(config) self.num_labels = config.num_labels # pylint: disable=arguments-differ def construct(self, input_ids=None, labels=None, attention_mask=None, position_ids=None, input_embeds=None, input_position=None, init_reset=True, batch_valid_length=None): r"""forward pass of the model""" if self.training: raise ValueError("PanguAlphaPromptTextClassificationModel just supports evaluate mode, " "please set run_mode to eval") if input_ids is None and input_embeds is None: raise ValueError("input_ids and input_embeds can not be all None") batch_size, num_labels_mul_seq_length = input_ids.shape seq_length = num_labels_mul_seq_length // self.num_labels input_ids = self.reshape(input_ids, (-1, seq_length)) input_mask = self.cast(self.reshape(attention_mask, (-1, seq_length)), mstype.float32) attention_mask = self.cast(self.not_equal(input_ids, self.pad_token_id), mstype.float32) attention_mask = self.get_attention_mask(attention_mask) if position_ids is None: position_ids = F.tuple_to_array(F.make_range(seq_length)) position_ids = self.expand(position_ids, 0) position_ids = self.tile(position_ids, (self.num_labels * batch_size, 1)) logits, vocab_table = self.backbone(input_ids, position_ids, attention_mask, init_reset, batch_valid_length) logits = self.head(logits, vocab_table) logits = self.reshape(logits, (batch_size, self.num_labels, seq_length, -1)) return logits, input_ids, input_mask, labels pengcheng.mind-mindformer/mindformers/models/pangualpha/pangualpha_config.py000066400000000000000000000104241476650141400300620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanGuAlpha Config API.""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config from mindformers.modules.transformer.transformer import default_moe_config from mindformers.modules.transformer.transformer import TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['PanguAlphaConfig'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class PanguAlphaConfig(BaseConfig): """ PanGuAlpha config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['pangualpha'] def __init__(self, batch_size: int = 1, seq_length: int = 1024, vocab_size: int = 40000, hidden_size: int = 2560, ffn_hidden_size: int = 2560 * 4, num_layers: int = 32, num_heads: int = 32, pad_token_id: int = 6, eos_token_id: int = 8, post_layernorm_residual: bool = False, param_init_type: str = 'float32', compute_dtype: str = 'float16', softmax_compute_type: str = 'float16', embedding_dropout_prob: float = 0.1, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, hidden_act: str = 'fast_gelu', use_past: bool = False, parallel_config: TransformerOpParallelConfig = default_transformer_config, moe_config: MoEConfig = default_moe_config, use_moe: bool = False, expert_num: int = 1, per_token_num_experts_chosen: int = 1, checkpoint_name_or_path: str = '', repetition_penalty: float = 1.0, max_decode_length: int = 1024, top_k: int = 5, top_p: float = 1.0, do_sample: bool = True, **kwargs): super(PanguAlphaConfig, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.ffn_hidden_size = ffn_hidden_size self.num_layers = num_layers self.num_heads = num_heads self.pad_token_id = pad_token_id self.eos_token_id = eos_token_id # Use post-layernorm or pre-layernrom, default:pre-layernorm self.post_layernorm_residual = post_layernorm_residual self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.embedding_dropout_prob = embedding_dropout_prob self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.hidden_act = hidden_act self.use_past = use_past self.parallel_config = parallel_config self.moe_config = moe_config self.use_moe = bool(use_moe) self.expert_num = expert_num self.per_token_num_experts_chosen = per_token_num_experts_chosen self.checkpoint_name_or_path = checkpoint_name_or_path self.repetition_penalty = repetition_penalty self.max_decode_length = max_decode_length self.top_k = top_k self.top_p = top_p self.do_sample = do_sample pengcheng.mind-mindformer/mindformers/models/pangualpha/pangualpha_processor.py000066400000000000000000000045321476650141400306370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ PanguAlphaProcessor """ from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_tokenizer import BaseTokenizer from mindformers.models.base_processor import BaseProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['PanguAlphaProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class PanguAlphaProcessor(BaseProcessor): """ PanguAlpha processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['pangualpha'] def __init__(self, tokenizer=None, max_length=128, padding='max_length', return_tensors='ms'): super(PanguAlphaProcessor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) def __call__(self, text_input=None, image_input=None): """call function""" output = {} if text_input is not None and self.tokenizer: if not isinstance(self.tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output return output pengcheng.mind-mindformer/mindformers/models/pangualpha/pangualpha_tokenizer.py000066400000000000000000000175451476650141400306420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """PanguAlpha Tokenzier""" import os from typing import List, Optional import jieba import sentencepiece as spm from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from ...mindformer_book import MindFormerBook __all__ = ['PanguAlphaTokenizer'] VOCAB_FILES_NAMES = {"vocab_file": "vocab.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class PanguAlphaTokenizer(Tokenizer): r""" Tokenize the input string and convert them into the ids. The tokenizer use the sentence piece internally. Args: vocab_file(str): The vocabulary file path. eos_token(str): The token that represents the end-of-sentence. Default "". bos_token(str): The token that represents the begin-of-sentence. Default """. unk_token(str): The token that represents the unknown. Default "". pad_token(str): The token that represents the pad. Default "". add_bos_token(bool): Whether or not to add the bos_token_id to the left of the input. Default "False" add_eos_token(bool): Whether or not to add the eos_token_id to the right of the input. Default "False" **kwargs: Other kwargs that will be passed into the base class of the `Tokenizer`. Examples: >>> from mindformers import PanguAlphaTokenizer >>> tokenizer = PanguAlphaTokenizer.from_pretrained("pangualpha_2_6b") >>> res = tokenizer("你好，今天天气不错。") >>> print(res) {'input_ids': [5772, 10, 465, 235, 464, 1123, 12], \ 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0], \ 'attention_mask': [1, 1, 1, 1, 1, 1, 1]} Outputs: A dict contains the processed ids, attention_mask that specific by the member `MODEL_INPUT_NAME` of the subclass. """ vocab_files_names = VOCAB_FILES_NAMES model_input_names = ["input_ids", "attention_mask"] FILE_LIST = ['tokenizer_config.json'] _support_list = MindFormerBook.get_tokenizer_support_list()['pangualpha'] def __init__(self, vocab_file, eos_token='', bos_token='', unk_token='', pad_token='', add_bos_token=False, add_eos_token=False, **kwargs): super(PanguAlphaTokenizer, self).__init__( eos_token=eos_token, bos_token=bos_token, unk_token=unk_token, pad_token=pad_token, **kwargs ) self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.encoder = {} self.sp = spm.SentencePieceProcessor(model_file=vocab_file) for i in range(self.sp.get_piece_size()): self.encoder[self.sp.id_to_piece(i)] = i self.translator = str.maketrans(" \n", "\u2582\u2583") def tokenize(self, text, pair=None, add_special_tokens=True, **kwargs): if not isinstance(text, str): raise ValueError("Text should be type str, but found type", type(text)) return self._tokenize(text) def _tokenize(self, text, **kwargs): """ Tokenize a string using bpe encode. """ seg_list = [x.translate(self.translator) for x in jieba.cut(text, cut_all=False)] return seg_list def convert_tokens_to_ids(self, tokens): """ the index of the tokens in the vocabulary. """ if isinstance(tokens, str): return self._convert_token_to_id(tokens) if isinstance(tokens, list): tokens = " ".join(tokens) return self.sp.encode(tokens) return None def _convert_token_to_id(self, token): """ the index of the tokens in the vocabulary. """ return self.sp.piece_to_id(token) def _convert_id_to_token(self, index): """ return the origin bpe tokens according to ids """ text = self.sp.decode(index) text = text.replace(' ', '').replace('\u2582', ' ').replace('\u2583', '\n') return text def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" text = "".join(tokens) return text def process_tokens(self, text): r"""replace special tokens with space and \n""" text = text.replace(' ', '').replace('\u2582', ' ').replace('\u2583', '\n') return text def pangu_encode(self, text): """pangu encode""" res = self.tokenize(text) return res def pangu_decode(self, tokens): """pangu decode""" text = self.sp.decode(tokens) return self.process_tokens(text) def save_vocabulary(self, save_directory, filename_prefix=None): """write the word to the files""" if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None output_file_path = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"]) with open(output_file_path, 'w', encoding="utf8") as fp: for k in self.encoder: fp.write(k + '\n') return output_file_path @property def vocab_size(self): return len(self.encoder) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): r"""Insert the special tokens to the input_ids. Currently""" bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab pengcheng.mind-mindformer/mindformers/models/sam/000077500000000000000000000000001476650141400225025ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/sam/__init__.py000066400000000000000000000022341476650141400246140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .sam import Sam from .sam_image_encoder import SAMImageEncoder from .sam_prompt_encoder import SAMPromptEncoder from .sam_mask_decoder import SAMMaskDecoder from .sam_config import SAMConfig, ImageEncoderConfig from .sam_processor import SAMProcessor, SAMImageProcessor from .sam_utils import * __all__ = ["Sam", "SAMImageEncoder", "SAMPromptEncoder", "SAMMaskDecoder", "SAMConfig", "ImageEncoderConfig", 'SAMProcessor', "SAMImageProcessor"] __all__.extend(sam_utils.__all__) pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/000077500000000000000000000000001476650141400247125ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/__init__.cpython-39.pyc000066400000000000000000000012171476650141400311030ustar00rootroot00000000000000a ?cf@stdZddlmZddlmZddlmZddlmZddl m Z m Z ddl m Z mZddlTgd Zeejd S) z models init)Sam)SAMImageEncoder)SAMPromptEncoder)SAMMaskDecoder) SAMConfigImageEncoderConfig) SAMProcessorSAMImageProcessor)*)rrrrrrrr N)__doc__samrZsam_image_encoderrZsam_prompt_encoderrZsam_mask_decoderrZ sam_configrrZ sam_processorrr Z sam_utils__all__extendrrJ/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/__init__.pys    pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam.cpython-39.pyc000066400000000000000000000105361476650141400301300ustar00rootroot00000000000000a ?cfO@sxdZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZe e jGdd d eZdS) z SAM Model)TupleN) build_model)MindFormerRegisterMindFormerModuleType)MindFormerBook) BaseModelc sxeZdZUdZdZeed<dZeed<e dZ ddfd d Z dd dZ ejejeedfejdddZZS)Samz SAM predicts object masks from an image and input prompts. If prompts are not known in advance, using Sam is recommended over calling the model directly. gmask_thresholdRGB image_formatsamN)returncs>t|t|j|_t|j|_t|j|_||dS)N) super__init__r image_encoderZ prompt_configprompt_encoderZdecoder_config mask_decoderload_checkpoint)selfconfig __class__E/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam.pyr$s     z Sam.__init__TFc Cs|dus|dusJd|dur0|dus0Jd|durB||}|j||||d\} } |j||j| | | d\} }|j| ||d}| s||jk}|| |fS)a Args: 'image': The image as a mindspore tensor in 3xHxW format, already transformed for input to the model. 'features': The features of an image, extracted by image encoder. 'input_size': (tuple(int, int)) The input size of the image after transformation, as (H, W). 'original_size': (tuple(int, int)) The original size of the image before transformation, as (H, W). 'point_coords': (ms.Tensor) Batched point prompts for this image, with shape BxNx2. Already transformed to the input frame of the model. 'point_labels': (ms.Tensor) Batched labels for point prompts, with shape BxN. 'boxes': (ms.Tensor) Batched box inputs, with shape Bx4. Already transformed to the input frame of the model. 'mask_inputs': (ms.Tensor) Batched mask inputs to the model, in the form Bx1xHxW. multimask_output (bool): Whether the model should predict multiple disambiguating masks, or return a single mask. return_logits (bool): Whether the mask return logits. Returns: 'masks': (ms.Tensor) Batched binary mask predictions, with shape BxCxHxW, where B is the number of input prompts, C is determined by multimask_output, and (H, W) is the original size of the image. 'iou_predictions': (ms.Tensor) The model's predictions of mask quality, in shape BxC. 'low_res_logits': (ms.Tensor) Low resolution logits with shape BxCxHxW, where H=W=256. Can be passed as mask input to subsequent iterations of prediction. Nz%image and feature can't be both None.z:point labels can't be None, when point coords is not None.) point_coords point_labelsboxes mask_inputs)Zimage_embeddingsZimage_peZsparse_prompt_embeddingsZdense_prompt_embeddingsmultimask_output) input_size original_size)rrrZ get_dense_pepostprocess_masksr )rimagefeaturesrr rrrrr return_logitsZsparse_embeddingsZdense_embeddings low_res_masksiou_predictionsmasksrrr construct,s6,     z Sam.construct.)r'rr r cCsRtj||jj|jjfddd}|dd|dd|df}tj||ddd}|S)zpostprocess masksbilinearT)sizemode align_corners.Nr)ops interpolaterimg_size)rr'rr rrrr!vszSam.postprocess_masks) NNNNNNNNTF)__name__ __module__ __qualname____doc__r float__annotations__r strrget_model_support_list _support_listrr(msTensorrintr! __classcell__rrrrrs(      Kr)r4typingr mindsporer: mindspore.opsr.Zmindformers.models.build_modelrmindformers.tools.registerrrmindformers.mindformer_bookrmindformers.models.base_modelrregisterMODELSrrrrrs      pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_config.cpython-39.pyc000066400000000000000000000117511476650141400314550ustar00rootroot00000000000000a ?cfN@sdZddlmZmZddlmmZddlm Z m Z ddl m Z m Z ddlmZddlmZddlmZmZe Ze ed ZeejGd d d eZeejGd d d eZeejGdddeZeejGdddeZdS)zSAM Config API.)UnionTupleN)TransformerOpParallelConfigTransformerRecomputeConfig) MoEConfigdefault_moe_config)MindFormerBook) BaseConfig)MindFormerRegisterMindFormerModuleType) recomputecs6eZdZdZedZdeddfdd ZZ S) SAMConfigz0 Configuration class for the SAM model. samN)checkpoint_name_or_pathreturnc s.tjfi|||_||_||_||_dS)N)super__init__ image_encoder prompt_configdecoder_configr)selfrrrrkwargs __class__L/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_config.pyr%s zSAMConfig.__init__)r) __name__ __module__ __qualname____doc__rget_config_support_list _support_liststrr __classcell__rrrrr s r cseZdZdZedZdddddddd d d d d d d d ddddeefe e e e e e e e e e e e e e e e e e e e eddfdd ZZS)ImageEncoderConfigzE Configuration class for the image encoder of the SAM model. ri T) rfloat16float32N)img_size patch_sizein_chans embed_dimdepth num_heads mlp_ratio out_chansqkv_bias use_abs_pos use_rel_posrel_pos_zero_init window_sizeglobal_attn_indexesr compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typeparallel_config moe_configrc stjfi|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_|dkrttjntj|_|dkrtjntj|_|dkrtjntj|_|dkrtjntj|_||_||_||_dSNr2)rrr3r4r5r6r7r8r9r:r;r<r=r>r?r@mstyper2r1rArBrCrDrErFr)rr3r4r5r6r7r8r9r:r;r<r=r>r?r@rrArBrCrDrErFrrrrr9s,zImageEncoderConfig.__init__)rrrr rr!r"default_parallel_configrintbooltupler#rrrr$rrrrr%2s^ r%c sleZdZdZedZddddddeefe e e e e fe e e e fe e e e ed d fd d ZZS) PromptEncoderConfigz@ Configuration for the prompt encoder of the SAM model. rr+)@rN)r&r&r'rr1N) prompt_embed_dimimage_embedding_sizeinput_image_size mask_in_chansrrArErFrc  sVtjfi| ||_||_||_||_|dkr8tjntj|_ ||_ ||_ ||_ dSrG) rrrOrPrQrRrHr2r1rArErFr) rrOrPrQrRrrArErFrrrrrqs zPromptEncoderConfig.__init__)rrrr rr!r"rIrrJrrr#rrrr$rrrrrMjs* rMcspeZdZdZedZdddddddddd d d d eefe e e e e e e e e e e e e e e d d fd d Z ZS)MaskDecoderConfigz> Configuration for the mask decoder of the SAM model. rr(r-r+ir/rr1r2N)num_multimask_outputs decoder_depthdecoder_embed_dimdecoder_mlp_dimdecoder_num_headstransformer_dimiou_head_depthiou_head_hidden_dimrrArBrCrDrErFrc stjfi|||_||_||_||_||_||_||_||_ | dkrPt j nt j |_ | dkrft j nt j |_| dkr|t j nt j |_| dkrt j nt j |_||_||_| |_dSrG)rrrTrUrVrWrXrYrZr[rHr2r1rArBrCrDrErFr)rrTrUrVrWrXrYrZr[rrArBrCrDrErFrrrrrs zMaskDecoderConfig.__init__)rrrr rr!r"rIrrJr#rrrr$rrrrrSsF rS)r typingrrZmindspore.common.dtypecommondtyperHmindformers.modules.transformerrr#mindformers.modules.transformer.moerrmindformers.mindformer_bookrZmindformers.models.base_configr mindformers.tools.registerr r default_recompute_configrIregisterCONFIGr r%rMrSrrrrs"     7  pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_image_encoder.cpython-39.pyc000066400000000000000000000277241476650141400330000ustar00rootroot00000000000000a ?cf3D @sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl mmZddlmZddlmZddlmZmZddlmZddlmZmZd d lmZmZeej Gd d d eZ!Gd ddej"Z#Gdddej"Z$dej%e&ee&e&feej%ee&e&ffdddZ'ej%e&ee&e&fee&e&fej%dddZ(e&e&ej%ej%dddZ)ej%ej%ej%ej%ee&e&fee&e&fej%dddZ*Gdddej"Z+dS) zSAM Image Encoder Model)OptionalTupleN) Parameter) operations)MindFormerRegisterMindFormerModuleType) BaseModel)Linear LayerNorm) LayerNorm2dMLPBlockcs:eZdZdZddfdd ZejejdddZZS) SAMImageEncoderz Image encoder N)returnc s`t||j|_|j|_|j|_|j|_|j|_|j|_|j|_|j |_ |j |_ |j |_ |j |_ |j |_ |j|_|j|_|j|_|j|_|j|_|j|_t|jtr|j}|j}n |j\}}||j}||j}||f|_|jdkr"|j||j|j}|j||j|j}||f|_nd|_t|j|jf|j|jf|j|jd|_d|_|j rttd|j|j|j|j|jftj|_t !|_"t#|jD]t}t$|j|j|j|j |j ||jvr|jnd|j|j|j|j|j|jf|j |j|j|j|jd} |j"%| qt &t j'|j|j dddt(|j t j'|j |j dd ddd t(|j |_)dS) Nrrr) kernel_sizestridein_chans embed_dimr )dim num_heads mlp_ratioqkv_bias use_rel_pos window_sizepad_size feat_size input_sizelayer_norm_eps compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typeF)rhas_biaspad)rpad_modepaddingr#)*super__init__img_size patch_sizerrdepthrrZ out_chansrrZ use_abs_posrrZglobal_attn_indexesrr r!r" isinstanceintrr PatchEmbed patch_embed pos_embedrPZerosmstypefloat32nnCellListblocksrangeBlockappendSequentialCellConv2dr neck) selfconfigimg_himg_wZfeat_hZfeat_wpad_hpad_wiblock __class__S/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_image_encoder.pyr)&s           zSAMImageEncoder.__init__xrcCsL||}|jdur||j}|jD] }||}q$||dddd}|S)z Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Encoded image tensor. Nrr$r )r0r1r8r> transpose)r?rLblkrIrIrJ constructs     zSAMImageEncoder.construct) __name__ __module__ __qualname____doc__r)msTensorrP __classcell__rIrIrGrJr!sYrc seZdZdZdddddddd ejejejejf eeee e ee eefe eefe e eefedd fd d Z e je jd ddZZS)r:z] Transformer blocks with support of window attention and residual propagation blocks g@TFrr)@rXNg-q=) rrrrrrrrrrrc stt|| d|_t|||||dkr.| n||f| | | |d |_t|f| d|_t|t||| |d|_ ||_ ||_ ||_ dS)N)epsr)rrrrrr r!r")Z embedding_dimZmlp_dimrr") r(r)r norm1 Attentionattnnorm2r r.mlprrr)r?rrrrrrrrrrrr r!r"rGrIrJr)s,   zBlock.__init__rKcCsp|}||}|jdkr(t||j|j}||}|jdkrPt||j|j|j}||}||||}|S)x Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. r) rZrwindow_partitionrr\window_unpartitionrr^r])r?rLZshortcutrIrIrJrPs    zBlock.construct)rQrRrSrTr4float16r5r.floatboolrrr)rUrVrPrWrIrIrGrJr:s6  'r:c sjeZdZdZddddejejejejfeeeee e eefddfdd Z e j e j d d d ZZS) r[zG Multi-head Attention block with relative position embeddings. TFN)rrrrrrc st||_||_||_||_||} | d|_t||d||| d|_t|||| d|_ ||_ |j r|duszJdt t d|dd| f|j|_t t d|dd| f|j|_t jd d |_t |_t jd d |_t |_t jdd |_dS) Ngr$) in_channels out_channelsr#rr")rfrgrr"zBInput size must be provided if using relative positional encoding.rMrr )axisT) transpose_b)r(r)rr r!rscaler qkvprojrrr2r3 rel_pos_h rel_pos_wSoftmaxsoftmax BatchMatMul batchmatmulbatchmatmul_trans_bCastcastUnstackunstack) r?rrrrrrr r!r"head_dimrGrIrJr)s>  &&  zAttention.__init__rKc Cs,|j\}}}}|j}|||||d|jdddddd}|||j}||d||j||d\}} } | ||j | } |j rt | ||j |j||f||f} || |j} || } || |j} || | }|||j||d}|ddddd}||||d}||}|||}|S)r_r$rhrMrr )shapedtyperlreshaperpermutervrrxrtrkradd_decomposed_rel_posrnror!rqrsviewrm) r?rLbhw_Zori_typerlqkvr\rIrIrJrPs$,&    zAttention.construct)rQrRrSrTr4rbr5r.rdrrr)rUrVrPrWrIrIrGrJr[s$,r[)rLrrrc Cs|j\}}}}|\}}|dks&|dkrJtjdd|fd|fdfd} | |}||||} } ||| ||| |||}|ddddddd |||} | S) a Partition the input tensor into non-overlapping windows with optional padding. Args: x (ms.Tensor): Input tensor with shape [B, H, W, C]. window_size (int): Window size. pad_size (tuple[int, int]): Padding size as (pad_h, pad_w). Returns: windows (ms.Tensor): Windows after partition with shape [B * num_windows, window_size, window_size, C]. rr)paddingsr r$rMrzrh)r{r2PadrrN) rLrrrrrcrCrDr%hpwpwindowsrIrIrJr`(s r`)rrpad_hwhwrc Cs|\}}|\}}||||}} |jd|| ||} || ||| |||d} | dddddd| || d} ||ks| |kr| ddd|d|ddf} | S) a Unpartition windows back into original sequences and remove padding if needed. Args: windows (ms.Tensor): Input windows with shape [B * num_windows, window_size, window_size, C]. window_size (int): Window size. pad_hw (Tuple[int, int]): Padded height and width (Hp, Wp). hw (Tuple[int, int]): Original height and width (H, W) before padding. Returns: x (ms.Tensor): Unpartitioned sequences with shape [B, H, W, C]. rrhr r$rMrzrN)r{rrN) rrrrrCrDrrrrrrLrIrIrJraBs  ra)q_sizek_sizerel_posrcCstdt||d}|jd|kr\tj|d|jddddd|dd}|d|}n|}t|dddft||d}t|dddft||d}|||dt||d}|| t j S) a_ Get relative positional embeddings based on the relative positions of query and key sizes. Args: q_size (int): Size of query q. k_size (int): Size of key k. rel_pos (ms.Tensor): Relative position embeddings (L, C). Returns: ms.Tensor: Extracted positional embeddings according to relative positions. rMr rrhlinear)sizemodeNg?) r.maxr{ops interpolater}rNnparangeastyper4int32)rrrZ max_rel_distZrel_pos_resizedZq_coordsZk_coordsZrelative_coordsrIrIrJ get_rel_pos^s$$r)r\rrnrorrrcCs|\}}|\}} t|||} t|| |} |j\} } }|| |||}t|| ddd| ||| jd}t|dddd| ddd| ||| jd}|dddd}|| |||| |dddddddddf|dddddddddf| |||| }|S)a  Calculate decomposed Relative Positional Embeddings from mvitv2 paper. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 Args: attn (ms.Tensor): Attention map. q (ms.Tensor): Query q in the attention layer with shape (B, q_h * q_w, C). rel_pos_h (ms.Tensor): Relative position embeddings (Lh, C) for the height axis. rel_pos_w (ms.Tensor): Relative position embeddings (Lw, C) for the width axis. q_size (Tuple[int, int]): Spatial sequence size of query q with (q_h, q_w). k_size (Tuple[int, int]): Spatial sequence size of key k with (k_h, k_w). Returns: ms.Tensor: Attention map with added relative positional embeddings. rrMr r$N)rr{r}rmatmulrNr)r\rrnrorrZq_hZq_wk_hk_wrhrwrrrZr_qZrel_hZrel_wrIrIrJrs"   (4  rcsbeZdZdZd eeefeeefeeeeefeeddfdd Zejejd d d Z Z S)r/z# Image to Patch Embedding. rrrrrr$N)rrr'rrrcs&ttj|||||dd|_dS)a Initialize the PatchEmbed layer. Args: kernel_size (Tuple[int, int]): Kernel size of the projection layer. stride (Tuple[int, int]): Stride of the projection layer. padding (Tuple[int, int, int, int]): Padding size of the projection layer. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. T)rrr'r#N)r(r)r6r=rm)r?rrr'rrrGrIrJr)s zPatchEmbed.__init__rKcCs||}|dddd}|S)z Forward pass of the PatchEmbed layer. Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Patch embeddings tensor. rrMr$r )rmrN)r?rLrIrIrJrPs zPatchEmbed.construct)rrrr$r) rQrRrSrTrr.r)rUrVrPrWrIrIrGrJr/s  r/)r),rTtypingrr mindsporerU mindspore.nnr6 mindspore.opsrZmindspore.numpynumpyrZmindspore.common.dtypecommonr|r4rrr2mindformers.tools.registerrrmindformers.models.base_modelrmindformers.modules.layersr r Z sam_layersr r registerMODELSrCellr:r[rVr.r`rarrr/rIrIrIrJsL       qCT    #  'pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_layers.cpython-39.pyc000066400000000000000000000046561476650141400315150ustar00rootroot00000000000000a ?cf @sdZddlmZddlZddlmZddlmm Z ddlm Z ddl m ZddlmZGdddejZGd d d ejZdS) z SAM Layers)TypeN) Parameter) operations)LinearcsTeZdZdZejejejfe e e ej ddfdd Z e je jdddZZS) MLPBlockz Multi-Layer Perceptron (MLP) block. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. N) embedding_dimmlp_dimactreturncs:tt||||d|_t||||d|_||_dS)N) in_channels out_channels compute_dtypeparam_init_type)super__init__rlin1lin2r )selfrrr r r __class__L/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_layers.pyr$s zMLPBlock.__init__xr cCs||||SN)rr r)rrrrr construct5szMLPBlock.construct)__name__ __module__ __qualname____doc__nnGELUmstypefloat16float32intrCellrmsTensorr __classcell__rrrrrs rcs@eZdZdZd eeddfdd Zejejddd Z Z S) LayerNorm2dz Layer Normalization for 2D data. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Normalized tensor. ư>N) num_channelsepsr csTttt|tj|_tt|tj|_ ||_ t |_ t |_dSr)rrrPOnesr'r$weightZerosbiasr-PowpowSqrtsqrt)rr,r-rrrrDs   zLayerNorm2d.__init__rcCsp|jddd}|||djddd}|||||j}|jddddf||jddddf}|S)NT) keep_dims)meanr4r6r-r0r2)rrusrrrrLs ,zLayerNorm2d.construct)r+) rrrrr%floatrr'r(rr)rrrrr*:s r*)rtypingr mindsporer' mindspore.nnr mindspore.common.dtypecommondtyper"r mindspore.opsrr.mindformers.modules.layersrr&rr*rrrrs      pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_mask_decoder.cpython-39.pyc000066400000000000000000000335771476650141400326420ustar00rootroot00000000000000a ?cfX@sdZddlZddlmZmZddlZddlmZddl m Z ddl m m Zddl mZddlmZmZddlmZddlmZmZdd lmZmZeejGd d d eZGd d d ejZ GdddejZ!GdddejZ"GdddejZ#dS)zSAM Mask Decoder ModelN)TupleType) operations)MindFormerRegisterMindFormerModuleType) BaseModel)Linear LayerNorm)MLPBlock LayerNorm2dcs|eZdZdZddfdd ZejejejejeeejejfdddZ ejejejejeejejfd d d Z Z S) SAMMaskDecoderzf A class to predict masks given image and prompt embeddings using a transformer architecture. N)returnc st||j_|j_|j_|j_|j_|j_|j_|j _ |j _ t j _ |j_|j_|j_|j_tjjjjj jjjjd _t dj_jd_t jj_t t jjjdddddtjd t jjdjddddd _t fdd tjD_tjj jjjjd _ t!j"d d _#t!j"dd _$t!%_&t!j'dd _(dS) N) depth embedding_dimmlp_dim num_headslayer_norm_eps compute_dtypelayernorm_compute_typesoftmax_compute_typeparam_init_typer T) kernel_sizestridehas_biasc s.g|]&}tjjjddjjdqS)rrr)MLPtransformer_dimrr).0iselfR/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_mask_decoder.py Xsz+SAMMaskDecoder.__init__..rraxis))super__init__r!Z decoder_depthZdecoder_embed_dimZdecoder_mlp_dimZdecoder_num_headsZnum_multimask_outputsZiou_head_depthZiou_head_hidden_dimrnnGELU activationrrrrTwoWayTransformer transformer Embedding iou_tokennum_mask_tokens mask_tokensSequentialCellConv2dTransposer output_upscalingCellListrangeoutput_hypernetworks_mlpsr iou_prediction_headPConcatconcat_0concat_1 ExpandDims expand_dimsStackstack)r%config __class__r$r'r,&sv       zSAMMaskDecoder.__init__)image_embeddingsimage_pesparse_prompt_embeddingsdense_prompt_embeddingsmultimask_outputrc Csd|j||||d\}}|r&tdd}n tdd}|dd|ddddf}|dd|f}||fS)a8 Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. )rHrIrJrKr Nr) predict_masksslice) r%rHrIrJrKrLmasksiou_predZ mask_slicer&r&r' constructis   zSAMMaskDecoder.construct)rHrIrJrKrc Cs|jd}||jj|jjg}t||d|ddf}t|rT| ||g}n|}| |jdd}||}| |jdd} |j\} } } } | || |\}}|dddddf}|dddd|j ddf}| ddd| | | | }||}g}t|j D],}||j||dd|ddfq ||}|j\} } } } t||| | | | | d| | }||}||fS)a Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. rNr r)shaper?r3embedding_tabler5ops broadcast_torBanyr@repeat_interleaver1r4 transposeviewr8r:appendr;rDmatmulr<)r%rHrIrJrKbs output_tokenstokenssrcZpos_srcbchwhsZ iou_token_outZmask_tokens_outZupscaled_embeddingZ hyper_in_listr#Zhyper_inrOrPr&r&r'rMs.    * & zSAMMaskDecoder.predict_masks) __name__ __module__ __qualname____doc__r,msTensorboolrrQrM __classcell__r&r&rFr'r sD $r c szeZdZdZejddejejejejfe e e e e ej e e ddfdd Z ejejejeejejfdd d ZZS) r0z Two-way transformer decoder that attends to an input image using queries whose positional embedding is supplied. r-q=N)rrrrr/attention_downsample_raterrc st||_||_||_||_||_| |_| |_| |_ t |_ t |D]6} |j t|||||| dk||j|j|j|j d qLt||||j|j|j|j d|_t|f|d|_dS)Nr) rrrr/roskip_first_layer_perrrrrdownsample_raterrrreps)r+r,rZ embeding_dimrZmpl_dimrrrrr-r9layersr:r[TwoWayAttentionBlock Attentionfinal_attn_token_to_imager norm_final_attn) r%rrrrr/rorrrrrr#rFr&r'r,sH    zTwoWayTransformer.__init__)image_embeddingrIpoint_embeddingrcCs|j\}}}}|||||ddd}|||||ddd}|}|} |jD]} | || ||d\}} qT||} | |} |j| | | d} || }||}|| fS)au Apply the two-way transformer. Args: image_embedding (ms.Tensor): Image to attend to. Should have shape B x embedding_dim x h x w for any h and w. image_pe (ms.Tensor): The positional encoding to add to the image. Must have the same shape as image_embedding. point_embedding (ms.Tensor): The embedding to add to the query points. Must have shape B x N_points x embedding_dim for any N_points. Returns: ms.Tensor: The processed point_embedding. ms.Tensor: The processed image_embedding. rrr )querieskeysquery_pekey_peqkv)rSreshapepermuterurxry)r%rzrIr{r]rbrcrdr|r}layerrrattn_outr&r&r'rQs$   zTwoWayTransformer.construct)rfrgrhrir-ReLUmstypefloat16float32intrCellfloatr,rjrkrrQrmr&r&rFr'r0s, 5r0c seZdZdZdejdddejejejejf e e e e ej e e e ddfdd Zejejejejeejejfd d d ZZS) rvz^ Two-way attention block consisting of self-attention and cross-attention layers. irFrnN)rrrr/rorprrc stt|||| | | d|_t|f|d|_t||||| | | d|_t|f|d|_t||||| d|_ t|f|d|_ t|f|d|_ t||||| | | d|_ ||_ dS)N)rrrrrsrqr)r+r,rw self_attnr norm1cross_attn_token_to_imagenorm2r mlpnorm3norm4cross_attn_image_to_tokenrp) r%rrrr/rorprrrrrrFr&r'r,+sF zTwoWayAttentionBlock.__init__)r|r}r~rrc Cs|jr|j|||d}n ||}|j|||d}||}||}||}||}|j|||d}||}||}||}||}||}||}||}|j|||d}||}||}||fS)a Apply the two-way attention block. Args: queries (ms.Tensor): Queries tensor. keys (ms.Tensor): Keys tensor. query_pe (ms.Tensor): Query positional encoding. key_pe (ms.Tensor): Key positional encoding. Returns: ms.Tensor: Processed queries tensor. ms.Tensor: Processed keys tensor. r) rprrrrrrrr) r%r|r}r~rrrrZmlp_outr&r&r'rQ[s(     zTwoWayAttentionBlock.construct)rfrgrhrir-rrrrrrrrlrr,rjrkrrQrmr&r&rFr'rv's21rvcseZdZdZdejejejejfeeeddfdd Ze j ee j ddd Z e j e j d d d Z e j e j e j e j d ddZ ZS)rwzu Attention layer that supports downscaling the embedding size after projection to queries, keys, and values. r N)rrrrrcst||_|||_||_||_||_||_||_|j|dksNJdt dt |j|j|_ t||j||d|_t||j||d|_t||j||d|_t|j|||d|_tjdd|_tjdd|_t|_t|_d S) a Initialize the Attention layer. Args: embedding_dim (int): The channel dimension of the embeddings. num_heads (int): The number of heads in the attention layer. downsample_rate (int): Downsample rate for the attention. rz$num_heads must divide embedding_dim.g? in_channels out_channelsrrrRr)T) transpose_bN)r+r,rZ internal_dimrrrrrrjrkmathsqrtinv_norm_factorrq_projk_projv_projout_projr=Softmaxsoftmax BatchMatMulbatchmatmul_trans_b batchmatmulMulmul)r%rrrrrrrrrFr&r'r,sD   zAttention.__init__)xrrcCs0|j\}}}||||||}|ddddS)a Separate the input tensor into multiple heads. Args: x (ms.Tensor): Input tensor. num_heads (int): Number of heads to separate into. Returns: ms.Tensor: Tensor with dimensions reshaped for separate heads. rrr r)rSrrY)r%rrranrbr&r&r'_separate_headss zAttention._separate_heads)rrcCs0|j\}}}}|dddd}|||||S)z Recombine the tensor with multiple heads into a single tensor. Args: x (ms.Tensor): Input tensor with separate heads. Returns: ms.Tensor: Recombined tensor. rrr r)rSrYr)r%rran_headsZn_tokensZ c_per_headr&r&r'_recombine_headss zAttention._recombine_heads)rrrrcCs|j}|||j}|||j}|||j}|||j}|||j}|||j}|||}| ||j }| ||j }| ||j|}||}|||}|S)a Apply the attention mechanism. Args: q (ms.Tensor): Queries tensor. k (ms.Tensor): Keys tensor. v (ms.Tensor): Values tensor. Returns: ms.Tensor: Output tensor after attention. )dtyperastyperrrrrrrrrrrrr)r%rrrori_typeattnoutr&r&r'rQs  zAttention.construct)rfrgrhrirrrrr,rjrkrrrQrmr&r&rFr'rws$3rwcsDeZdZdZdejejfeeeeeddfdd Z ddZ Z S) r z< Multi-Layer Perceptron (MLP) class definition. FN) input_dim hidden_dim output_dim num_layerssigmoid_outputrc sjt||_|g|d}tfddt|g|||gD|_||_t |_ t |_ dS)Nr cs g|]\}}t||dqS)r)r)r"rrrr&r'r(s z MLP.__init__..) r+r,rr-r9ziprurr=rreluSigmoidsigmoid) r%rrrrrrrrcrFrr'r,s   z MLP.__init__cCsLt|jD],\}}||jdkr.|||n||}q |jrH||}|S)z} Args: x (Tensor): Input tensor. Returns: Tensor: Output tensor of the MLP. r ) enumeraterurrrr)r%rr#rr&r&r'rQ%s & z MLP.construct) rfrgrhrirrrrrlr,rQrmr&r&rFr'r sr )$rirtypingrr mindsporerj mindspore.nnr- mindspore.opsrUZmindspore.common.dtypecommonrrrr=mindformers.tools.registerrrmindformers.models.base_modelrmindformers.modules.layersrr sam_layersr r registerMODELSr rr0rvrwr r&r&r&r's$     he~pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_processor.cpython-39.pyc000066400000000000000000000200761476650141400322270ustar00rootroot00000000000000a ?cfl!@sdZddlmZddlmZddlZddlmZddl Z ddl m Z ddl mZddlmZdd lmZdd lmZmZdd lmZmZeejGd d d eZGdddZeejGdddeZdS)z SAM Processor)deepcopy)TupleN)Image)vision)Inter) operations)MindFormerBook) BaseProcessorBaseImageProcessor)MindFormerRegisterMindFormerModuleTypecs*eZdZdZd fdd ZddZZS) SAMImageProcessora" Subclass of BaseImageProcessor implementing image preprocessing for SAM model. Args: img_size (int): The target size for image processing. mean (list): Mean values for image normalization. std (list): Standard deviation values for image normalization. g33333^@gR]@gRY@g(\2M@g(\L@gL@csJt||_t|ddd|_t|ddd|_t||_ dS)N) super__init__img_sizemsTensorview pixel_mean pixel_stdResizeLongestSide transform)selfrmeanstd __class__O/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_processor.pyr*s  zSAMImageProcessor.__init__c Ks|j|}|ddddddddddf}t|}|jdd}||j|j}|jdd\}}|j|}|j|}|j dkrt j ddd|fd|ffd}n$|j d krt j dd|fd|ffd}||}||fS) a3 Preprocess the input image for SAM model. Args: image (PIL.Image or numpy.array): Input image to be preprocessed. Returns: image (tensor): Preprocessed image as a tensor. input_size (tuple): Size of the input image after preprocessing. rrN)rr)paddings) r apply_image transposerrshaperrrndimPPad) rimageskwargs input_sizehwpadhpadwpadr"r"r# preprocess1s &     zSAMImageProcessor.preprocess)rrr)__name__ __module__ __qualname____doc__rr7 __classcell__r"r"r r#r s r c@seZdZdZeddddZejejdddZeje ed fejd d d Z eje ed fejd ddZ ejejdddZ eje ed fejd ddZ eje ed fejd ddZeeeee eefdddZdS)rz Resizes images to the longest side 'target_length', as well as provides methods for resizing coordinates and boxes. Provides methods for transforming both numpy array and batched mindspore tensors. N) target_lengthreturncCs ||_dS)z Initialize the ResizeLongestSide class. Args: target_length (int): Target length of the longest side. N)r=)rr=r"r"r#rXszResizeLongestSide.__init__)imager>cCsD||jd|jd|j}t|}tj|d}t||}|S)z Apply the resizing transformation to a single image. Args: image (np.ndarray): Input image as HxWxC numpy array. Returns: np.ndarray: Resized image as H'xW'xC numpy array. rr)size) get_preprocess_shaper+r=r fromarrayrResizenparray)rr? target_size pil_imageZ resize_opZ resized_imager"r"r#r)as   zResizeLongestSide.apply_image.)coords original_sizer>cCs^|\}}||d|d|j\}}t|t}|d|||d<|d|||d<|S)aS Apply the resizing transformation to coordinates. Args: coords (np.ndarray): Input coordinates as a numpy array of shape (N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized coordinates as a numpy array of shape (N, 2). rr.r.r)rAr=rastypefloatrrHrIZold_hZold_wnew_hnew_wr"r"r# apply_coordsqs zResizeLongestSide.apply_coords)boxesrIr>cCs"||ddd|}|ddS)aI Apply the resizing transformation to bounding boxes. Args: boxes (np.ndarray): Input boxes as a numpy array of shape (B, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized boxes as a numpy array of shape (B, 4). rr$r&)rQreshaperrRrIr"r"r# apply_boxess zResizeLongestSide.apply_boxescCs6||jd|jd|j}t|tj}||}|S)a Apply the resizing transformation to a batch of images. Args: image (np.ndarray): Input batched images as BxCxHxW numpy array. Returns: np.ndarray: Resized batched images as BxC'xH'xW' numpy array. r$r()rAr+r=rrCrBILINEAR)rr?rFZresize_r"r"r#apply_image_batchs z#ResizeLongestSide.apply_image_batchcCsX|\}}||d|d|j\}}t|}|d|||d<|d|||d<|S)ap Apply the resizing transformation to a batch of coordinates. Args: coords (np.ndarray): Input batched coordinates as numpy array of shape (B, N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched coordinates as numpy array of shape (B, N, 2). rrrJrK)rAr=rrNr"r"r#apply_coords_batchs z$ResizeLongestSide.apply_coords_batchcCs"||ddd|}|ddS)af Apply the resizing transformation to a batch of bounding boxes. Args: boxes (np.ndarray): Input batched boxes as numpy array of shape (B, N, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched boxes as numpy array of shape (B, N, 4). rr$r&)rXrSrTr"r"r#apply_boxes_batchs z#ResizeLongestSide.apply_boxes_batch)oldholdwlong_side_lengthr>cCsD|dt||}||||}}t|d}t|d}||fS)aX Compute the output size given input size and target long side length. Args: oldh (int): Original image height. oldw (int): Original image width. long_side_length (int): Target length of the longest side. Returns: Tuple[int, int]: Output size as (new_h, new_w). g?g?)maxint)rZr[r\scaleZnewhZnewwr"r"r#rAs   z&ResizeLongestSide.get_preprocess_shape)r8r9r:r;r^rrDndarrayr)rrQrUrWrXrY staticmethodrAr"r"r"r#rQs rcs.eZdZdZedZdfdd ZZS) SAMProcessorz Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. samNrcstj||ddS)N)image_processorreturn_tensors)rr)rrdrer r"r#rszSAMProcessor.__init__)Nr) r8r9r:r;rget_processor_support_list _support_listrr<r"r"r r#rbs rb)r;copyrtypingrnumpyrDPILr mindsporermindspore.datasetrZmindspore.dataset.visionr mindspore.opsrr-mindformers.mindformer_bookrZ!mindformers.models.base_processorr r mindformers.tools.registerr r register PROCESSORr rrbr"r"r"r#s"        1 pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_prompt_encoder.cpython-39.pyc000066400000000000000000000207751476650141400332360ustar00rootroot00000000000000a ?cfO)@sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl mmZddlmZddlmZddlmZmZddlmZddlmZd d lmZeejGd d d eZGd ddej Z!dS)zSAM Prompt Encoder)OptionalTupleN) Parameter) operations)MindFormerRegisterMindFormerModuleType)MindFormerBook) BaseModel) LayerNorm2dcseZdZdZedZddfdd Zej dddZ ej ej e ej d d d Z ej ej d d dZ ej ej dddZeej eej eej edddZeej eej eej eej eej ej fdddZZS)SAMPromptEncoderz: Encodes prompts for input to SAM's mask decoder. samN)returncsVt||j_|j_|j_|j_tj_ t jd_ d_ fddt j D}t|_tdj_djddjdf_ttjdjdddddtjd tjjdjddddtj tjjjddd _tdj_t_t_t_tjdd _ t!_"dS) Ncsg|]}tdjqS)r )nn Embedding embed_dim).0iselfT/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_prompt_encoder.py 2z-SAMPromptEncoder.__init__..r rT) kernel_sizestridehas_bias)rraxis)#super__init__Zprompt_embed_dimrinput_image_sizeimage_embedding_sizeZ mask_in_chansrGELU activationPositionEmbeddingRandompe_layerZnum_point_embeddingsrangeCellListpoint_embeddingsrnot_a_point_embedZmask_input_sizeSequentialCellConv2dr mask_downscaling no_mask_embedPZeroszerosOnesones ExpandDims expand_dimsConcatconcatTiletile)rconfigr+ __class__rrr"'s6       zSAMPromptEncoder.__init__cCs|||jdS)a Get the positional encoding used to encode point prompts, applied to a dense set of points the shape of the image encoding. Returns: ms.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w) r)r7r(r$rrrr get_dense_peHs zSAMPromptEncoder.get_dense_pe)pointslabelspadrc Cs|jd}|d}|j||j}|rp|||jjd|ddf}|||g}||| |dft j  g}|dk |dd}|t ||jdjt |jdj7}|dk |dd}|t ||jdjt |jdj7}|S)zEmbeds point prompts.r?r )shaper(forward_with_coordsr#r;r7r,embedding_tabler9r5mstypeint32reshapenpwherer+ zeros_like) rr@rArBbsZpoint_embeddingr,Zmask_negZmask_posrrr _embed_pointsSs  zSAMPromptEncoder._embed_points)boxesrcCsv|d}|ddd}|j||j}|dddddf|jdj7<|dddddf|jdj7<|S)zEmbeds box prompts.rCrDrNrr )rJr(rFr#r+rG)rrPcoordsZcorner_embeddingrrr _embed_boxesfs &&zSAMPromptEncoder._embed_boxes)masksrcCs||}|S)zEmbeds mask inputs.)r/)rrTZmask_embeddingrrr _embed_masksos zSAMPromptEncoder._embed_masks)r@rPrTrcCs:|dur|jdS|dur$|jdS|dur6|jdSdS)z^ Gets the batch size of the output given the batch size of the input prompts. Nrr )rE)rr@rPrTrrr_get_batch_sizets   z SAMPromptEncoder._get_batch_size) point_coords point_labelsrP mask_inputsrc Cs||||}t|d|jf}|dur@|j|||dud}|}|durV||}|}|durj||} n0t|j j dddd|d|j d|j df} || fS)aD Args: point_coords (ms.Tensor or none): Point coordinates to embed. point_labels (ms.Tensor or none): Point labels to embed. boxes (ms.Tensor or none): Boxes to embed. mask_inputs (ms.Tensor or none): Masks to embed. Returns: Tuple[ms.Tensor, ms.Tensor]: Sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined by the number of input points and boxes. Dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W). r N)rBrDr) rVrKemptyrrOrSrUops broadcast_tor0rGrJr$) rrWrXrPrYrNsparse_embeddingsr+Zbox_embeddingsdense_embeddingsrrr constructs  zSAMPromptEncoder.construct)__name__ __module__ __qualname____doc__rget_model_support_list _support_listr"msTensorr?boolrOrSrUrintrVrr_ __classcell__rrr=rr s  !   r cseZdZdZddejfeeeddfdd Z e j e j ddd Z e eefe j d d d Ze j e eefe j d ddZZS)r'z? Positional encoding using random spatial frequencies. @N) num_pos_featsscalercst||_|dus |dkr$d}|td|f|j}t|dd|_t|_ t |_ t |_ t|_tjdd|_tjdd|_dS) z Initialize the PositionEmbeddingRandom. Args: num_pos_feats (int): Number of positional encoding features. scale (float): Scale factor for the positional encoding Gaussian matrix. Ngg?r#positional_encoding_gaussian_matrix)namerDr)r!r" compute_dtyper1StandardNormalastyperrnCastcastr4r5SinsinCoscosr8r9Stackstack)rrlrmrpZpe_gaussian_matrixr=rrr"s     z PositionEmbeddingRandom.__init__)rRrcCs\d|d}|j}t||j|j}|||}dtj|}| | || |gS)z Positionally encode points that are normalized to [0,1]. Args: coords (ms.Tensor): Coordinates normalized to [0, 1]. Returns: ms.Tensor: Positional encoding for the given coordinates. rr ) dtyper[matmulrrrprnrtrKpir9rvrx)rrRori_typerrr _pe_encodings  z$PositionEmbeddingRandom._pe_encoding)sizercCsl|\}}|||ftj}|jddd}|jddd}||}||}||||g}|dddS)z Generate positional encoding for a grid of the specified size. Args: size (Tuple[int, int]): Grid size (H, W). Returns: ms.Tensor: Positional encoding for the grid. rrrCr r)r5rffloat32cumsumrrz transpose)rrhwgridZy_embedZx_embedperrrr_s z!PositionEmbeddingRandom.construct) coords_input image_sizercCsn|}|dddddf|d|dddddf<|dddddf|d|dddddf<||S)a> Positionally encode points that are not normalized to [0,1]. Args: coords_input (ms.Tensor): Coordinates not normalized to [0, 1]. image_size (Tuple[int, int]): Image size (H, W). Returns: ms.Tensor: Positional encoding for the given coordinates. Nrr )r)rrrrRrrrrFs 00z+PositionEmbeddingRandom.forward_with_coords)r`rarbrcrHfloat16rirfloatr"rfrgrrr_rFrjrrr=rr'sr')"rctypingrr mindsporerf mindspore.nnr mindspore.opsr[Zmindspore.numpynumpyrKZmindspore.common.dtypecommonr{rHrrr1mindformers.tools.registerrrmindformers.mindformer_bookrmindformers.models.base_modelr sam_layersr registerMODELSr Cellr'rrrrs          pengcheng.mind-mindformer/mindformers/models/sam/__pycache__/sam_utils.cpython-39.pyc000066400000000000000000000505211476650141400313460ustar00rootroot00000000000000a ?cfX@sTdZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z ddl Z ddlZgdZGdddZdAeje ee eeejd d d Zejejd ddZee e eddfdddZeje eeefdddZeeefejdddZeeefedddZejeeejdddZeejddd Zeeee ejd!d"d#Ze ed$feee e e ee efd%d&d'Zeje eejd(d)d*Z eje eejd+d,d-Z!eje eeeejd.d/d0Z"ejeee eje#fd1d2d3Z$eeefeeefd4d5d6Z%d7d8Z&d9d:Z'ejejd;dd?d@Z)dS)Bz SAM UtilsNdeepcopy)product)AnyDict Generator ItemsViewListTuple)MaskData area_from_rlebatch_iteratorbatched_mask_to_boxbox_xyxy_to_xywhbuild_all_layer_point_gridscalculate_stability_scorecoco_encode_rlegenerate_crop_boxesis_box_near_crop_edge mask_to_rleremove_small_regions rle_to_maskuncrop_boxes_xyxy uncrop_masks uncrop_pointsbox_areanmsc@seZdZdZddddZeeddddZedd d d Zeed d d Z e eefdddZ e j ddddZdddddZdS)r z A structure for storing masks and their related data in batched format. Implements basic filtering and concatenation. N)returncKs:|D]}t|ttjfsJdqtfi||_dS)z Initialize the MaskData. Args: **kwargs: Keyword arguments representing different mask-related data. z:MaskData only supports list, numpy arrays, and ms tensors.N)values isinstancelistnpndarraydict_stats)selfkwargsvr(K/home/ma-user/work/mindformers-baichuan/mindformers/models/sam/sam_utils.py__init__2s  zMaskData.__init__)keyitemrcCs&t|ttjfsJd||j|<dS)z Set an item in the MaskData. Args: key (str): The key of the item. item (Any): The item to be set. z*MaskData only supports list, numpy arrays.N)rr r!r"r$)r%r+r,r(r(r) __setitem__?s  zMaskData.__setitem__)r+rcCs |j|=dS)z| Delete an item from the MaskData. Args: key (str): The key of the item to be deleted. Nr$r%r+r(r(r) __delitem__LszMaskData.__delitem__cCs |j|S)z Get an item from the MaskData. Args: key (str): The key of the item to be retrieved. Returns: Any: The retrieved item. r.r/r(r(r) __getitem__Us zMaskData.__getitem__cCs |jS)z Get the items stored in the MaskData. Returns: ItemsView[str, Any]: A view of the items in the MaskData. )r$items)r%r(r(r)r2aszMaskData.items)keeprcs|jD]\}dur&d|j|<q ttjrB|j|<q ttrvjtjkrvfddtD|j|<q ttrfddD|j|<q t d|dt dq dS)z Filter the MaskData based on a boolean mask. Args: keep (np.ndarray): Boolean mask indicating which items to keep. Ncsg|]\}}|r|qSr(r().0ia)r3r(r) wz#MaskData.filter..csg|] }|qSr(r(r4r5)r'r(r)r7yr8 MaskData key  has an unsupported type .) r$r2rr!r"r dtypebool enumerate TypeErrortype)r%r3kr()r3r'r)filterjs   zMaskData.filter) new_statsrcCs|D]\}}||jvs(|j|dur8t||j|<qt|tjrdtj|j||gdd|j|<qt|tr|j|t||j|<qtd|dt |dqdS)z Concatenate new data from another MaskData instance. Args: new_stats (MaskData): The MaskData instance to concatenate. Nraxisr:r;r<) r2r$rrr!r" concatenater r@rA)r%rDrBr'r(r(r)cat}s   z MaskData.cat)__name__ __module__ __qualname____doc__r*strrr-r0r1rr2r!r"rCrHr(r(r(r)r ,s     r 4@)boxescrop_boxorig_boxatolrcCstj|tjd}tj|tjd}t||tj}tj|t|dddf|jddf|ddd}tj|t|dddf|jddf|ddd}t||}tj |ddS) a  Filter boxes at the edge of a crop, but not at the edge of the original image. Args: boxes (np.ndarray): Bounding boxes ndarray of shape (N, 4) in format (x1, y1, x2, y2). crop_box (List[int]): Crop box coordinates in format (x1, y1, x2, y2). orig_box (List[int]): Original image box coordinates in format (x1, y1, x2, y2). atol (float): Absolute tolerance for box comparison. Returns: np.ndarray: Boolean ndarray indicating whether boxes are near the crop edge. r=NrgT)rRrtol equal_nanrE) r!arrayfloat32rastypeisclosetileshape logical_andany)rOrPrQrRZnear_crop_edgeZnear_image_edger(r(r)rs"  r)box_xyxyrcCs4t|}|d|d|d<|d|d|d<|S)a6 Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, w, h) format. Args: box_xyxy (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x1, y1, x2, y2) format. Returns: box_xywh (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x, y, w, h) format. rrTr)r_Zbox_xywhr(r(r)rs r) batch_sizerc'srrtfddDs"Jdtdttddk}t|D]fddDVqRdS)a Generate batches of data from input arrays. Args: batch_size (int): Size of each batch. *args: Variable number of input arrays to iterate over. Yields: List[Any]: A batch of data containing corresponding elements from each input array. c3s"|]}t|tdkVqdS)rN)len)r4r6)argsr(r) sz!batch_iterator..z8Batched iteration must have inputs of all the same size.rcs$g|]}|dqSrTr()r4arg)brbr(r)r7r8z"batch_iterator..N)allrcintrange)rbrd n_batchesr()rdrhrbr)r s  ( r )tensorrc Cs4|j\}}}|ddd|jdd}t|ddddftj|ddddftj}tt|}g}t |D]}||dddf|kdf}t tj dg|j d|dtj ||g|j dg}|dd|dd} ||dfdkrgndg} | | |||g| dq|S)a+ Encode masks to an uncompressed RLE format, in the format expected by pycoco tools. Args: tensor (np.ndarray): Binary mask tensor of shape (B, H, W) where B is the batch size. Returns: List[Dict[str, Any]]: List of dictionaries representing RLE encoded masks. rr`rTNrS)sizecounts)r\ transposereshaper! bitwise_xorrYint32stacknonzerorkrGrWr=extendtolistappend) rmrhhwdiffZchange_indicesoutr5cur_idxsZbtw_idxsrpr(r(r)rs$ < r)rlercCsh|d\}}tj||td}d}d}|dD]$}|||||<||7}|dN}q.|||}|S)z Compute a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: np.ndarray: Binary mask as a NumPy array. rorSrFrpT)r!emptyr>rrrq)rrzr{maskidxparitycountr(r(r)rs    rcCst|ddddS)z Calculate the area of a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: int: Calculated area of the mask. rprTNr`)sum)rr(r(r)r s r )masksmask_thresholdthreshold_offsetrcCsP|||kjdtjdjdtjd}|||kjdtjdjdtjd}||S)a Calculate the stability score for a batch of masks. The stability score is the Intersection over Union (IoU) between the binary masks obtained by thresholding the predicted mask logits at high and low values. Args: masks (np.ndarray): Predicted mask logits tensor of shape (B, H, W). mask_threshold (float): Threshold value for binary mask creation. threshold_offset (float): Offset to adjust the threshold. Returns: np.ndarray: Stability scores for each mask in the batch. rnrS)rr!rX)rrr intersectionsunionsr(r(r)rs  r) n_per_sidercCstdd|}t|d||}t|dddf|df}t|dddfd|f}tj||gdddd}|S)z Generate a 2D grid of points evenly spaced in the [0, 1] x [0, 1] range. Args: n_per_side (int): Number of points per side of the grid. Returns: np.ndarray: 2D grid of points as a NumPy array. rTr`NrnrE)r!linspacer[rurr)roffsetZpoints_one_sideZpoints_xZpoints_ypointsr(r(r)build_point_grid<s r)rn_layersscale_per_layerrcCs8g}t|dD]"}t|||}|t|q|S)aj Generate point grids for all crop layers. Args: n_per_side (int): Number of points per side of the original grid. n_layers (int): Number of layers in the crop hierarchy. scale_per_layer (int): Scaling factor applied per layer. Returns: List[np.ndarray]: List of point grids for each crop layer as NumPy arrays. rT)rkrjryr)rrrZpoints_by_layerr5n_pointsr(r(r)rNs r.)im_sizer overlap_ratiorcs gg}}|\}}t||}|dd||g|ddd}t|D]} d| d} t||d| ||| ||| fddt| D} fddt| D} t| | D]@\} }| |t| |t||g}|||| dqqH||fS) a Generate a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: im_size (Tuple[int, ...]): Size of the original image in (height, width). n_layers (int): Number of layers in the crop hierarchy. overlap_ratio (float): Overlap ratio for calculating crop box size. Returns: Tuple[List[List[int]], List[int]]: Tuple containing a list of crop boxes and a list of layer indices for each crop box. rcSstt||d||S)NrT)rjmathceil)Zorig_lenZn_cropsoverlapr(r(r)crop_len{sz%generate_crop_boxes..crop_lenr`rTcsg|]}t|qSr(rjr9)crop_wrr(r)r7r8z'generate_crop_boxes..csg|]}t|qSr(rr9)crop_hrr(r)r7r8)minryrkrjr)rrr crop_boxes layer_idxsZim_hZim_wZ short_siderZi_layerZn_crops_per_sideZ crop_box_x0Z crop_box_y0x0y0boxr()rrrr)rcs$         r)rOrPrcCsH|\}}}}tj||||ggtjd}t|jdkr@t|d}||S)ar Uncrop the coordinates of bounding boxes from a cropped region to the original image. Args: boxes (np.ndarray): Bounding box coordinates in XYXY format (x0, y0, x1, y1). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped bounding box coordinates in XYXY format. rSrarTr!rWrtrcr\ expand_dims)rOrPrr_rr(r(r)rs  r)rrPrcCsD|\}}}}tj||ggtjd}t|jdkr.Frcsg|]}|vr|qSr(r(r9) fill_labelsr(r)r7r8T) rYr!uint8cv2connectedComponentsWithStatsr?rkrjargmaxisin) rrrZ correct_holesZ working_maskn_labelsregionsstatsrsizesZ small_regionsr()rrr)rs   r)uncompressed_rlercCs<ddlm}|d\}}||||}|dd|d<|S)z Encode uncompressed RLE (Run-Length Encoding) to COCO format RLE. Args: uncompressed_rle (Dict[str, Any]): Uncompressed RLE dictionary. Returns: Dict[str, Any]: Encoded RLE in COCO format. r)rrorpzutf-8)Z pycocotoolsrZ frPyObjectsdecode)rZ mask_utilsrzr{rr(r(r)rs  rcCst|jdkr4tjg|jdddRtjdS|j}|dd\}}t|dkrr|d|jd|jd}n t|d}tj| tj dd}|t |dddf}tj|dd}||| tj }tj |dd}tj| tj dd}|t |dddf} tj| dd} | || tj } tj | dd} | | k tj||k tjB tj } tj| || |gdd} | tj| dd} t|dkr| g|dddR} n| d} | tjS) z Convert masks to bounding boxes in XYXY format. Return [0,0,0,0] for an empty mask. Args: masks (np.ndarray): Binary masks of shape C1xC2x...xHxW. Returns: np.ndarray: Bounding boxes in XYXY format of shape C1xC2x...x4. rNrSr`rnrE)r!prodr\zerosrtrcrrrmaxrYrXaranger>rru)rr\rzr{ in_heightZin_height_coordsZ bottom_edgesZ top_edgesin_widthZin_width_coordsZ right_edgesZ left_edgesZ empty_filterr}r(r(r)rs8 $  rcCs@|dddf|dddf|dddf|dddfS)a Compute the area of a set of bounding boxes. Args: boxes (np.ndarray): Bounding boxes for which the area will be computed. Expected shape is [N, 4], where N is the number of boxes. Each box is specified by its coordinates (x1, y1, x2, y2) in the format with ``0 <= x1 < x2`` and ``0 <= y1 < y2``. Returns: np.ndarray: The area for each box in an ndarray of shape [N]. Nr`rrarTr()rOr(r(r)r7s r)box1box2c Cst|}t|}t|ddtjddf|ddddf}t|ddtjddf|ddddf}||}td|}|dddddf|dddddf}||ddtjf||}|S)a Calculate the Intersection over Union (IoU) between two sets of bounding boxes. Args: box1 (np.ndarray): NumPy array representing the first set of bounding boxes in format [x_min, y_min, x_max, y_max]. box2 (np.ndarray): NumPy array representing the second set of bounding boxes in the same format. Returns: iou (np.ndarray): NumPy array containing the IoU values for each pair of boxes. Example: >>> box1 = np.array([[0, 0, 10, 10], [5, 5, 15, 15]]) >>> box2 = np.array([[5, 5, 15, 15], [8, 8, 18, 18]]) >>> iou = box_iou(box1, box2) >>> print(iou) [[0.11111111 0. ] [1. 0.23076923]] Nr`rrT)rr!maximumnewaxisminimum) rrarea1Zarea2ltrbZwhinteriour(r(r)box_iouFs00 ,r)rOscores iou_thresholdc Cs|}g}|jdkrx|d}||dddf}|||jdkrHqx|dd}||}t||}||d|k}q t|}|S)a Apply Non-Maximum Suppression (NMS) algorithm to filter a set of bounding boxes, eliminating overlapping boxes based on the Intersection over Union (IoU) threshold. Args: boxes (np.ndarray): NumPy array containing bounding box coordinates, with each row representing a box [x_min, y_min, x_max, y_max]. scores (np.ndarray): NumPy array containing scores for each corresponding bounding box. iou_threshold (float): IoU (Intersection over Union) threshold to determine if boxes overlap. Returns: keep (np.ndarray): NumPy array containing indices of the retained bounding boxes. Example: >>> boxes = np.array([[0, 0, 10, 10], [5, 5, 15, 15], [8, 8, 18, 18]]) >>> scores = np.array([0.9, 0.75, 0.85]) >>> iou_threshold = 0.5 >>> keep = nms(boxes, scores, iou_threshold) >>> print(keep) [0 2] rrnNrT)argsortroryrr!rW) rOrridxsr3Zmax_score_indexZ max_score_boxZ other_boxesZiousr(r(r)rfs      r)rN)*rLrcopyr itertoolsrtypingrrrrr r rnumpyr!__all__r r"rjfloatrrr rMrrr rrrrrrrr>rrrrrrr(r(r(r)s`   f #%     .  4 pengcheng.mind-mindformer/mindformers/models/sam/conver_weight.py000066400000000000000000000065021476650141400257220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import mindspore as ms import torch def convert_weight_sam(pth_file="sam_vit_b_01ec64.pth", ckpt_file=None): """ convert sam weights from pytorch to mindspore. """ ckpt = torch.load(pth_file, map_location=torch.device('cpu')) if 'model_state' in ckpt: state_dict = ckpt['model_state'] elif 'module' in ckpt: state_dict = ckpt['module'] elif 'model' in ckpt: state_dict = ckpt['model'] else: state_dict = ckpt if not ckpt_file: ms_ckpt_path = pth_file.split('.')[0] + '.ckpt' else: ms_ckpt_path = ckpt_file ms_ckpt = [] for k, v in state_dict.items(): new_k = k if new_k.startswith('norm.'): new_k = new_k.replace('norm.', 'fc_norm.') if 'norm' in new_k: if 'weight' in new_k: new_k = new_k.replace('weight', 'gamma') if 'bias' in k: new_k = new_k.replace('bias', 'beta') if 'prompt_encoder' in new_k: if 'embed' in new_k and 'weight' in new_k: new_k = new_k.replace('weight', 'embedding_table') if 'mask_decoder' in new_k: if 'iou_token' in new_k or 'mask_tokens' in new_k: new_k = new_k.replace('weight', 'embedding_table') ms_ckpt.append({'name': new_k, 'data': ms.Tensor(v.numpy())}) ms.save_checkpoint(ms_ckpt, ms_ckpt_path) print("Convert finished!") def show_info(file_path="sam_vit_b_01ec64.pth"): """ show info of pth or ckpt """ if '.ckpt' in file_path: state_dict = ms.load_checkpoint(file_path) elif '.pth' in file_path: state_dict = torch.load(file_path, map_location=torch.device('cpu')) if 'model_state' in state_dict: state_dict = state_dict['model_state'] elif 'module' in state_dict: state_dict = state_dict['module'] elif 'model' in state_dict: state_dict = state_dict['model'] for k, v in state_dict.items(): info = k + ' ' + str(v.shape) + ' ' + str(v.dtype) print(info) if __name__ == '__main__': parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_path", type=str, default="path/to/sam_vit_b_01ec64.pth.pth", help="The torch checkpoint path.") parser.add_argument("--ckpt_path", type=str, default="path/to/sam_vit_b_01ec64.pth.ckpt", help="The torch checkpoint path.") opt = parser.parse_args() convert_weight_sam(opt.torch_path, opt.ckpt_path) pengcheng.mind-mindformer/mindformers/models/sam/sam.py000066400000000000000000000135171476650141400236430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Model""" from typing import Tuple import mindspore as ms import mindspore.ops as ops from mindformers.models.build_model import build_model from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel @MindFormerRegister.register(MindFormerModuleType.MODELS) class Sam(BaseModel): """ SAM predicts object masks from an image and input prompts. If prompts are not known in advance, using Sam is recommended over calling the model directly. """ mask_threshold: float = 0.0 image_format: str = "RGB" _support_list = MindFormerBook.get_model_support_list()['sam'] def __init__(self, config) -> None: super().__init__(config) self.image_encoder = build_model(config.image_encoder) self.prompt_encoder = build_model(config.prompt_config) self.mask_decoder = build_model(config.decoder_config) self.load_checkpoint(config) def construct(self, image=None, features=None, input_size=None, original_size=None, point_coords=None, point_labels=None, boxes=None, mask_inputs=None, multimask_output=True, return_logits=False): """ Args: 'image': The image as a mindspore tensor in 3xHxW format, already transformed for input to the model. 'features': The features of an image, extracted by image encoder. 'input_size': (tuple(int, int)) The input size of the image after transformation, as (H, W). 'original_size': (tuple(int, int)) The original size of the image before transformation, as (H, W). 'point_coords': (ms.Tensor) Batched point prompts for this image, with shape BxNx2. Already transformed to the input frame of the model. 'point_labels': (ms.Tensor) Batched labels for point prompts, with shape BxN. 'boxes': (ms.Tensor) Batched box inputs, with shape Bx4. Already transformed to the input frame of the model. 'mask_inputs': (ms.Tensor) Batched mask inputs to the model, in the form Bx1xHxW. multimask_output (bool): Whether the model should predict multiple disambiguating masks, or return a single mask. return_logits (bool): Whether the mask return logits. Returns: 'masks': (ms.Tensor) Batched binary mask predictions, with shape BxCxHxW, where B is the number of input prompts, C is determined by multimask_output, and (H, W) is the original size of the image. 'iou_predictions': (ms.Tensor) The model's predictions of mask quality, in shape BxC. 'low_res_logits': (ms.Tensor) Low resolution logits with shape BxCxHxW, where H=W=256. Can be passed as mask input to subsequent iterations of prediction. """ assert image is not None or features is not None, \ "image and feature can't be both None." if point_coords is not None: assert point_labels is not None, \ "point labels can't be None, when point coords is not None." if features is None: features = self.image_encoder(image) sparse_embeddings, dense_embeddings = self.prompt_encoder(point_coords=point_coords, point_labels=point_labels, boxes=boxes, mask_inputs=mask_inputs) low_res_masks, iou_predictions = self.mask_decoder(image_embeddings=features, image_pe=self.prompt_encoder.get_dense_pe(), sparse_prompt_embeddings=sparse_embeddings, dense_prompt_embeddings=dense_embeddings, multimask_output=multimask_output) masks = self.postprocess_masks(low_res_masks, input_size=input_size, original_size=original_size) if not return_logits: masks = masks > self.mask_threshold return masks, low_res_masks, iou_predictions def postprocess_masks(self, masks: ms.Tensor, input_size: ms.Tensor, original_size: Tuple[int, ...]) -> ms.Tensor: """postprocess masks""" masks = ops.interpolate( masks, size=(self.image_encoder.img_size, self.image_encoder.img_size), mode="bilinear", align_corners=True ) masks = masks[..., : input_size[0], : input_size[1]] masks = ops.interpolate(masks, size=original_size, mode="bilinear", align_corners=True) return masks pengcheng.mind-mindformer/mindformers/models/sam/sam_config.py000066400000000000000000000171161476650141400251670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Config API.""" from typing import Union, Tuple import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class SAMConfig(BaseConfig): """ Configuration class for the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, image_encoder, prompt_config, decoder_config, checkpoint_name_or_path: str = '', **kwargs) -> None: super().__init__(**kwargs) self.image_encoder = image_encoder self.prompt_config = prompt_config self.decoder_config = decoder_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ImageEncoderConfig(BaseConfig): """ Configuration class for the image encoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, img_size: int = 1024, patch_size: int = 16, in_chans: int = 3, embed_dim: int = 768, depth: int = 12, num_heads: int = 12, mlp_ratio: int = 4, out_chans: int = 256, qkv_bias: bool = True, use_abs_pos: bool = True, use_rel_pos: bool = True, # 仅支持默认False rel_pos_zero_init: bool = True, window_size: int = 14, global_attn_indexes: tuple = (2, 5, 8, 11), checkpoint_name_or_path: str = '', compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", param_init_type: str = "float32", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.img_size = img_size self.patch_size = patch_size self.in_chans = in_chans self.embed_dim = embed_dim self.depth = depth self.num_heads = num_heads self.mlp_ratio = mlp_ratio self.out_chans = out_chans self.qkv_bias = qkv_bias self.use_abs_pos = use_abs_pos self.use_rel_pos = use_rel_pos # 仅支持默认False self.rel_pos_zero_init = rel_pos_zero_init self.window_size = window_size self.global_attn_indexes = global_attn_indexes self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.param_init_type = mstype.float32 if param_init_type == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class PromptEncoderConfig(BaseConfig): """ Configuration for the prompt encoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, prompt_embed_dim: int = 256, image_embedding_size: Union[int, Tuple[int]] = (64, 64), input_image_size: Union[int, Tuple[int]] = (1024, 1024), mask_in_chans: int = 16, checkpoint_name_or_path: str = '', compute_dtype: str = "float16", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.prompt_embed_dim = prompt_embed_dim self.image_embedding_size = image_embedding_size self.input_image_size = input_image_size self.mask_in_chans = mask_in_chans self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path @MindFormerRegister.register(MindFormerModuleType.CONFIG) class MaskDecoderConfig(BaseConfig): """ Configuration for the mask decoder of the SAM model. """ _support_list = MindFormerBook.get_config_support_list()['sam'] def __init__(self, num_multimask_outputs: int = 3, decoder_depth: int = 2, decoder_embed_dim: int = 256, decoder_mlp_dim: int = 2048, decoder_num_heads: int = 8, transformer_dim: int = 256, iou_head_depth: int = 3, iou_head_hidden_dim: int = 256, checkpoint_name_or_path: str = '', compute_dtype: str = "float16", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", param_init_type: str = "float32", parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs) -> None: super().__init__(**kwargs) self.num_multimask_outputs = num_multimask_outputs # transformer: self.decoder_depth = decoder_depth self.decoder_embed_dim = decoder_embed_dim self.decoder_mlp_dim = decoder_mlp_dim self.decoder_num_heads = decoder_num_heads self.transformer_dim = transformer_dim self.iou_head_depth = iou_head_depth self.iou_head_hidden_dim = iou_head_hidden_dim self.compute_dtype = mstype.float32 if compute_dtype == "float32" else mstype.float16 self.layernorm_compute_type = mstype.float32 if layernorm_compute_type == "float32" else mstype.float16 self.softmax_compute_type = mstype.float32 if softmax_compute_type == "float32" else mstype.float16 self.param_init_type = mstype.float32 if param_init_type == "float32" else mstype.float16 self.parallel_config = parallel_config self.moe_config = moe_config self.checkpoint_name_or_path = checkpoint_name_or_path pengcheng.mind-mindformer/mindformers/models/sam/sam_image_encoder.py000066400000000000000000000420631476650141400265020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Image Encoder Model""" from typing import Optional, Tuple import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.numpy as np import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear, LayerNorm from .sam_layers import LayerNorm2d, MLPBlock # This class and its supporting functions below lightly adapted from the ViTDet backbone available at: https://github.com/facebookresearch/detectron2/blob/main/detectron2/modeling/backbone/vit.py # noqa @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMImageEncoder(BaseModel): """ Image encoder """ def __init__(self, config) -> None: super().__init__(config) self.img_size = config.img_size self.patch_size = config.patch_size self.in_chans = config.in_chans self.embed_dim = config.embed_dim self.depth = config.depth self.num_heads = config.num_heads self.mlp_ratio = config.mlp_ratio self.out_chans = config.out_chans self.qkv_bias = config.qkv_bias self.layer_norm_eps = config.layer_norm_eps self.use_abs_pos = config.use_abs_pos self.use_rel_pos = config.use_rel_pos self.window_size = config.window_size self.global_attn_indexes = config.global_attn_indexes self.compute_dtype = config.compute_dtype self.layernorm_compute_type = config.layernorm_compute_type self.softmax_compute_type = config.softmax_compute_type self.param_init_type = config.param_init_type if isinstance(self.img_size, int): img_h = self.img_size img_w = self.img_size else: img_h, img_w = self.img_size feat_h = img_h // self.patch_size feat_w = img_w // self.patch_size self.feat_size = (feat_h, feat_w) if self.window_size > 0: pad_h = (self.window_size - feat_h % self.window_size) % self.window_size pad_w = (self.window_size - feat_w % self.window_size) % self.window_size self.pad_size = (pad_h, pad_w) else: self.pad_size = (0, 0) self.patch_embed = PatchEmbed( kernel_size=(self.patch_size, self.patch_size), stride=(self.patch_size, self.patch_size), in_chans=self.in_chans, embed_dim=self.embed_dim, ) self.pos_embed: Optional[Parameter] = None if self.use_abs_pos: # Initialize absolute positional embedding with pretrain image size. self.pos_embed = Parameter(P.Zeros()((1, self.img_size // self.patch_size, self.img_size // self.patch_size, self.embed_dim), mstype.float32)) self.blocks = nn.CellList() for i in range(self.depth): block = Block( dim=self.embed_dim, num_heads=self.num_heads, mlp_ratio=self.mlp_ratio, qkv_bias=self.qkv_bias, use_rel_pos=self.use_rel_pos, window_size=self.window_size if i not in self.global_attn_indexes else 0, pad_size=self.pad_size, feat_size=self.feat_size, input_size=(self.img_size // self.patch_size, self.img_size // self.patch_size), layer_norm_eps=self.layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.blocks.append(block) self.neck = nn.SequentialCell( nn.Conv2d( self.embed_dim, self.out_chans, kernel_size=1, has_bias=False ), LayerNorm2d(self.out_chans), nn.Conv2d( self.out_chans, self.out_chans, kernel_size=3, pad_mode='pad', padding=1, has_bias=False, ), LayerNorm2d(self.out_chans), ) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Encoded image tensor. """ x = self.patch_embed(x) if self.pos_embed is not None: x = x + self.pos_embed for blk in self.blocks: x = blk(x) x = self.neck(x.transpose(0, 3, 1, 2)) return x class Block(nn.Cell): """ Transformer blocks with support of window attention and residual propagation blocks """ def __init__(self, dim: int, num_heads: int, mlp_ratio: float = 4.0, qkv_bias: bool = True, use_rel_pos: bool = False, window_size: int = 0, pad_size: Tuple[int, int] = (0, 0), feat_size: Tuple[int, int] = (64, 64), input_size: Optional[Tuple[int, int]] = None, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.norm1 = LayerNorm((dim), eps=layer_norm_eps) self.attn = Attention( dim, num_heads=num_heads, qkv_bias=qkv_bias, use_rel_pos=use_rel_pos, input_size=input_size if window_size == 0 else (window_size, window_size), compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type ) self.norm2 = LayerNorm((dim,), eps=layer_norm_eps) self.mlp = MLPBlock(embedding_dim=dim, mlp_dim=int(dim * mlp_ratio), compute_dtype=compute_dtype, param_init_type=param_init_type) self.window_size = window_size self.pad_size = pad_size self.feat_size = feat_size def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ shortcut = x x = self.norm1(x) # Window partition if self.window_size > 0: x = window_partition(x, self.window_size, self.pad_size) x = self.attn(x) # Reverse window partition if self.window_size > 0: x = window_unpartition(x, self.window_size, self.pad_size, self.feat_size) x = shortcut + x x = x + self.mlp(self.norm2(x)) return x class Attention(nn.Cell): """ Multi-head Attention block with relative position embeddings. """ def __init__(self, dim: int, num_heads: int = 8, qkv_bias: bool = True, use_rel_pos: bool = False, input_size: Optional[Tuple[int, int]] = None, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.num_heads = num_heads head_dim = dim // num_heads self.scale = head_dim**-0.5 self.qkv = Linear(in_channels=dim, out_channels=dim * 3, has_bias=qkv_bias, compute_dtype=compute_dtype, param_init_type=param_init_type) self.proj = Linear(in_channels=dim, out_channels=dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.use_rel_pos = use_rel_pos if self.use_rel_pos: assert ( input_size is not None ), "Input size must be provided if using relative positional encoding." # initialize relative positional embeddings self.rel_pos_h = Parameter(P.Zeros()((2 * input_size[0] - 1, head_dim), self.compute_dtype)) self.rel_pos_w = Parameter(P.Zeros()((2 * input_size[1] - 1, head_dim), self.compute_dtype)) self.softmax = P.Softmax(axis=-1) self.batchmatmul = P.BatchMatMul() self.batchmatmul_trans_b = P.BatchMatMul(transpose_b=True) self.cast = P.Cast() self.unstack = P.Unstack(axis=0) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ b, h, w, _ = x.shape ori_type = x.dtype # qkv with shape (3, B, nHead, H * W, C) qkv = self.qkv(x).reshape(b, h * w, 3, self.num_heads, -1).permute(2, 0, 3, 1, 4) qkv = self.cast(qkv, self.compute_dtype) # q, k, v with shape (B * nHead, H * W, C) q, k, v = self.unstack(qkv.reshape(3, b * self.num_heads, h * w, -1)) attn = self.batchmatmul_trans_b((q * self.scale), k) if self.use_rel_pos: attn = add_decomposed_rel_pos(attn, q, self.rel_pos_h, self.rel_pos_w, (h, w), (h, w)) attn = self.cast(attn, self.softmax_compute_type) attn = self.softmax(attn) attn = self.cast(attn, self.compute_dtype) x = self.batchmatmul(attn, v) x = x.view(b, self.num_heads, h, w, -1) x = x.permute(0, 2, 3, 1, 4) x = x.reshape(b, h, w, -1) x = self.proj(x) x = self.cast(x, ori_type) return x def window_partition(x: ms.Tensor, window_size: int, pad_size: Tuple[int, int] = 0) -> Tuple[ms.Tensor, Tuple[int, int]]: """ Partition the input tensor into non-overlapping windows with optional padding. Args: x (ms.Tensor): Input tensor with shape [B, H, W, C]. window_size (int): Window size. pad_size (tuple[int, int]): Padding size as (pad_h, pad_w). Returns: windows (ms.Tensor): Windows after partition with shape [B * num_windows, window_size, window_size, C]. """ b, h, w, c = x.shape pad_h, pad_w = pad_size if pad_h > 0 or pad_w > 0: pad = P.Pad(paddings=((0, 0), (0, pad_h), (0, pad_w), (0, 0))) x = pad(x) hp, wp = h + pad_h, w + pad_w x = x.view(b, hp // window_size, window_size, wp // window_size, window_size, c) windows = x.transpose(0, 1, 3, 2, 4, 5).view(-1, window_size, window_size, c) return windows def window_unpartition(windows: ms.Tensor, window_size: int, pad_hw: Tuple[int, int], hw: Tuple[int, int]) -> ms.Tensor: """ Unpartition windows back into original sequences and remove padding if needed. Args: windows (ms.Tensor): Input windows with shape [B * num_windows, window_size, window_size, C]. window_size (int): Window size. pad_hw (Tuple[int, int]): Padded height and width (Hp, Wp). hw (Tuple[int, int]): Original height and width (H, W) before padding. Returns: x (ms.Tensor): Unpartitioned sequences with shape [B, H, W, C]. """ pad_h, pad_w = pad_hw h, w = hw hp, wp = h + pad_h, w + pad_w b = windows.shape[0] // (hp * wp // window_size // window_size) x = windows.view(b, hp // window_size, wp // window_size, window_size, window_size, -1) x = x.transpose(0, 1, 3, 2, 4, 5).view(b, hp, wp, -1) if hp > h or wp > w: x = x[:, :h, :w, :] return x def get_rel_pos(q_size: int, k_size: int, rel_pos: ms.Tensor) -> ms.Tensor: """ Get relative positional embeddings based on the relative positions of query and key sizes. Args: q_size (int): Size of query q. k_size (int): Size of key k. rel_pos (ms.Tensor): Relative position embeddings (L, C). Returns: ms.Tensor: Extracted positional embeddings according to relative positions. """ max_rel_dist = int(2 * max(q_size, k_size) - 1) # Interpolate rel pos if needed. if rel_pos.shape[0] != max_rel_dist: # Interpolate rel pos. rel_pos_resized = ops.interpolate( rel_pos.reshape(1, rel_pos.shape[0], -1).transpose(0, 2, 1), size=max_rel_dist, mode='linear', ) rel_pos_resized = rel_pos_resized.reshape(-1, max_rel_dist) else: rel_pos_resized = rel_pos # Scale the coords with short length if shapes for q and k are different. q_coords = np.arange(q_size)[:, None] * max(k_size / q_size, 1.0) k_coords = np.arange(k_size)[None, :] * max(q_size / k_size, 1.0) relative_coords = (q_coords - k_coords) + (k_size - 1) * max(q_size / k_size, 1.0) return rel_pos_resized[relative_coords.astype(mstype.int32)] def add_decomposed_rel_pos(attn: ms.Tensor, q: ms.Tensor, rel_pos_h: ms.Tensor, rel_pos_w: ms.Tensor, q_size: Tuple[int, int], k_size: Tuple[int, int]) -> ms.Tensor: """ Calculate decomposed Relative Positional Embeddings from mvitv2 paper. https://github.com/facebookresearch/mvit/blob/19786631e330df9f3622e5402b4a419a263a2c80/mvit/models/attention.py # noqa B950 Args: attn (ms.Tensor): Attention map. q (ms.Tensor): Query q in the attention layer with shape (B, q_h * q_w, C). rel_pos_h (ms.Tensor): Relative position embeddings (Lh, C) for the height axis. rel_pos_w (ms.Tensor): Relative position embeddings (Lw, C) for the width axis. q_size (Tuple[int, int]): Spatial sequence size of query q with (q_h, q_w). k_size (Tuple[int, int]): Spatial sequence size of key k with (k_h, k_w). Returns: ms.Tensor: Attention map with added relative positional embeddings. """ q_h, q_w = q_size k_h, k_w = k_size rh = get_rel_pos(q_h, k_h, rel_pos_h) rw = get_rel_pos(q_w, k_w, rel_pos_w) b, _, dim = q.shape r_q = q.reshape(b, q_h, q_w, dim) rel_h = ops.matmul(r_q, rh.transpose(0, 2, 1)).reshape(b, q_h, q_w, rh.shape[1]) rel_w = ops.matmul(r_q.transpose(0, 2, 1, 3), rw.transpose(0, 2, 1)).reshape(b, q_h, q_w, rw.shape[1]) rel_w = rel_w.transpose(0, 2, 1, 3) attn = (attn.view(b, q_h, q_w, k_h, k_w) + \ rel_h[:, :, :, :, None] + \ rel_w[:, :, :, None, :]).view(b, q_h * q_w, k_h * k_w) return attn class PatchEmbed(nn.Cell): """ Image to Patch Embedding. """ def __init__(self, kernel_size: Tuple[int, int] = (16, 16), stride: Tuple[int, int] = (16, 16), padding: Tuple[int, int, int, int] = (0, 0, 0, 0), in_chans: int = 3, embed_dim: int = 768) -> None: """ Initialize the PatchEmbed layer. Args: kernel_size (Tuple[int, int]): Kernel size of the projection layer. stride (Tuple[int, int]): Stride of the projection layer. padding (Tuple[int, int, int, int]): Padding size of the projection layer. in_chans (int): Number of input image channels. embed_dim (int): Patch embedding dimension. """ super().__init__() self.proj = nn.Conv2d(in_chans, embed_dim, kernel_size=kernel_size, stride=stride, padding=padding, has_bias=True) def construct(self, x: ms.Tensor) -> ms.Tensor: """ Forward pass of the PatchEmbed layer. Args: x (ms.Tensor): Input image tensor. Returns: ms.Tensor: Patch embeddings tensor. """ x = self.proj(x) # B C H W -> B H W C x = x.transpose(0, 2, 3, 1) return x pengcheng.mind-mindformer/mindformers/models/sam/sam_layers.py000066400000000000000000000056001476650141400252140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Layers""" from typing import Type import mindspore as ms import mindspore.nn as nn import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.modules.layers import Linear class MLPBlock(nn.Cell): """ Multi-Layer Perceptron (MLP) block. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Output tensor. """ def __init__(self, embedding_dim: int, mlp_dim: int, act: Type[nn.Cell] = nn.GELU, compute_dtype=mstype.float16, param_init_type=mstype.float32) -> None: super().__init__() self.lin1 = Linear(in_channels=embedding_dim, out_channels=mlp_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.lin2 = Linear(in_channels=mlp_dim, out_channels=embedding_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.act = act() def construct(self, x: ms.Tensor) -> ms.Tensor: return self.lin2(self.act(self.lin1(x))) # From https://github.com/facebookresearch/detectron2/blob/main/detectron2/layers/batch_norm.py # noqa # Itself from https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 # noqa class LayerNorm2d(nn.Cell): """ Layer Normalization for 2D data. Args: x (ms.Tensor): Input tensor. Returns: ms.Tensor: Normalized tensor. """ def __init__(self, num_channels: int, eps: float = 1e-6) -> None: super().__init__() self.weight = Parameter(P.Ones()(num_channels, ms.float32)) self.bias = Parameter(P.Zeros()(num_channels, ms.float32)) self.eps = eps self.pow = P.Pow() self.sqrt = P.Sqrt() def construct(self, x: ms.Tensor) -> ms.Tensor: u = x.mean(1, keep_dims=True) s = self.pow(x - u, 2).mean(1, keep_dims=True) x = (x - u) / self.sqrt(s + self.eps) x = self.weight[:, None, None] * x + self.bias[:, None, None] return x pengcheng.mind-mindformer/mindformers/models/sam/sam_mask_decoder.py000066400000000000000000000542571476650141400263510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Mask Decoder Model""" import math from typing import Tuple, Type import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.modules.layers import Linear, LayerNorm from .sam_layers import MLPBlock, LayerNorm2d @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMMaskDecoder(BaseModel): """ A class to predict masks given image and prompt embeddings using a transformer architecture. """ def __init__(self, config) -> None: super().__init__(config) self.transformer_dim = config.transformer_dim self.decoder_depth = config.decoder_depth self.decoder_embed_dim = config.decoder_embed_dim self.decoder_mlp_dim = config.decoder_mlp_dim self.decoder_num_heads = config.decoder_num_heads self.num_multimask_outputs = config.num_multimask_outputs self.iou_head_depth = config.iou_head_depth self.iou_head_hidden_dim = config.iou_head_hidden_dim self.layer_norm_eps = config.layer_norm_eps self.activation = nn.GELU self.compute_dtype = config.compute_dtype self.layernorm_compute_type = config.layernorm_compute_type self.softmax_compute_type = config.softmax_compute_type self.param_init_type = config.param_init_type self.transformer = TwoWayTransformer( depth=self.decoder_depth, embedding_dim=self.decoder_embed_dim, mlp_dim=self.decoder_mlp_dim, num_heads=self.decoder_num_heads, layer_norm_eps=self.layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.iou_token = nn.Embedding(1, self.transformer_dim) self.num_mask_tokens = self.num_multimask_outputs + 1 self.mask_tokens = nn.Embedding(self.num_mask_tokens, self.transformer_dim) self.output_upscaling = nn.SequentialCell( nn.Conv2dTranspose(self.transformer_dim, self.transformer_dim // 4, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.transformer_dim // 4), self.activation(), nn.Conv2dTranspose(self.transformer_dim // 4, self.transformer_dim // 8, kernel_size=2, stride=2, has_bias=True), self.activation(), ) self.output_hypernetworks_mlps = nn.CellList( [ MLP(self.transformer_dim, self.transformer_dim, self.transformer_dim // 8, 3, compute_dtype=self.compute_dtype, param_init_type=self.param_init_type) for i in range(self.num_mask_tokens) ] ) self.iou_prediction_head = MLP( self.transformer_dim, self.iou_head_hidden_dim, self.num_mask_tokens, self.iou_head_depth, compute_dtype=self.compute_dtype, param_init_type=self.param_init_type ) self.concat_0 = P.Concat(axis=0) self.concat_1 = P.Concat(axis=1) self.expand_dims = P.ExpandDims() self.stack = P.Stack(axis=1) def construct(self, image_embeddings: ms.Tensor, image_pe: ms.Tensor, sparse_prompt_embeddings: ms.Tensor, dense_prompt_embeddings: ms.Tensor, multimask_output: bool) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. multimask_output (bool): Whether to return multiple masks or a single mask. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. """ masks, iou_pred = self.predict_masks( image_embeddings=image_embeddings, image_pe=image_pe, sparse_prompt_embeddings=sparse_prompt_embeddings, dense_prompt_embeddings=dense_prompt_embeddings, ) # Select the correct mask or masks for output if multimask_output: mask_slice = slice(1, None) else: mask_slice = slice(0, 1) masks = masks[:, mask_slice, :, :] iou_pred = iou_pred[:, mask_slice] # Prepare output return masks, iou_pred def predict_masks(self, image_embeddings: ms.Tensor, image_pe: ms.Tensor, sparse_prompt_embeddings: ms.Tensor, dense_prompt_embeddings: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: image_embeddings (ms.Tensor): Embeddings from the image encoder. image_pe (ms.Tensor): Positional encoding with the shape of image_embeddings. sparse_prompt_embeddings (ms.Tensor): Embeddings of the points and boxes. dense_prompt_embeddings (ms.Tensor): Embeddings of the mask inputs. Returns: Tuple[ms.Tensor, ms.Tensor]: Batched predicted masks and batched predictions of mask quality. """ bs = sparse_prompt_embeddings.shape[0] # Concatenate output tokens output_tokens = self.concat_0([self.iou_token.embedding_table, self.mask_tokens.embedding_table]) output_tokens = ops.broadcast_to(self.expand_dims(output_tokens, 0), (bs, -1, -1)) if ops.any(sparse_prompt_embeddings): tokens = self.concat_1([output_tokens, sparse_prompt_embeddings]) else: tokens = output_tokens # Expand per-image data in batch direction to be per-mask src = image_embeddings.repeat_interleave(tokens.shape[0], 0) src = src + dense_prompt_embeddings pos_src = image_pe.repeat_interleave(tokens.shape[0], 0) b, c, h, w = src.shape # Run the transformer hs, src = self.transformer(src, pos_src, tokens) iou_token_out = hs[:, 0, :] mask_tokens_out = hs[:, 1 : (1 + self.num_mask_tokens), :] # Upscale mask embeddings and predict masks using the mask tokens src = src.transpose(0, 2, 1).view(b, c, h, w) upscaled_embedding = self.output_upscaling(src) hyper_in_list = [] for i in range(self.num_mask_tokens): hyper_in_list.append(self.output_hypernetworks_mlps[i](mask_tokens_out[:, i, :])) hyper_in = self.stack(hyper_in_list) b, c, h, w = upscaled_embedding.shape masks = ops.matmul(hyper_in, upscaled_embedding.view(b, c, h * w)).view(b, -1, h, w) # Generate mask quality predictions iou_pred = self.iou_prediction_head(iou_token_out) return masks, iou_pred class TwoWayTransformer(nn.Cell): """ Two-way transformer decoder that attends to an input image using queries whose positional embedding is supplied. """ def __init__(self, depth: int, embedding_dim: int, num_heads: int, mlp_dim: int, activation: Type[nn.Cell] = nn.ReLU, attention_downsample_rate: int = 2, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.depth = depth self.embeding_dim = embedding_dim self.num_heads = num_heads self.mpl_dim = mlp_dim self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.layers = nn.CellList() for i in range(depth): self.layers.append( TwoWayAttentionBlock( embedding_dim=embedding_dim, num_heads=num_heads, mlp_dim=mlp_dim, activation=activation, attention_downsample_rate=attention_downsample_rate, skip_first_layer_pe=(i == 0), layer_norm_eps=layer_norm_eps, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) ) self.final_attn_token_to_image = Attention( embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=self.compute_dtype, layernorm_compute_type=self.layernorm_compute_type, softmax_compute_type=self.softmax_compute_type, param_init_type=self.param_init_type ) self.norm_final_attn = LayerNorm((embedding_dim,), eps=layer_norm_eps) def construct(self, image_embedding: ms.Tensor, image_pe: ms.Tensor, point_embedding: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Apply the two-way transformer. Args: image_embedding (ms.Tensor): Image to attend to. Should have shape B x embedding_dim x h x w for any h and w. image_pe (ms.Tensor): The positional encoding to add to the image. Must have the same shape as image_embedding. point_embedding (ms.Tensor): The embedding to add to the query points. Must have shape B x N_points x embedding_dim for any N_points. Returns: ms.Tensor: The processed point_embedding. ms.Tensor: The processed image_embedding. """ # BxCxHxW -> BxHWxC == B x N_image_tokens x C bs, c, h, w = image_embedding.shape image_embedding = image_embedding.reshape(bs, c, h*w).permute(0, 2, 1) image_pe = image_pe.reshape(bs, c, h*w).permute(0, 2, 1) # Prepare queries queries = point_embedding keys = image_embedding # Apply transformer blocks and final layernorm for layer in self.layers: queries, keys = layer( queries=queries, keys=keys, query_pe=point_embedding, key_pe=image_pe, ) # Apply the final attention layer from the points to the image q = queries + point_embedding k = keys + image_pe attn_out = self.final_attn_token_to_image(q=q, k=k, v=keys) queries = queries + attn_out queries = self.norm_final_attn(queries) return queries, keys class TwoWayAttentionBlock(nn.Cell): """ Two-way attention block consisting of self-attention and cross-attention layers. """ def __init__(self, embedding_dim: int, num_heads: int, mlp_dim: int = 2048, activation: Type[nn.Cell] = nn.ReLU, attention_downsample_rate: int = 2, skip_first_layer_pe: bool = False, layer_norm_eps: float = 1.e-12, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: super().__init__() self.self_attn = Attention(embedding_dim, num_heads, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.norm1 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.cross_attn_token_to_image = Attention(embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.norm2 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.mlp = MLPBlock(embedding_dim, mlp_dim, activation, compute_dtype=compute_dtype, param_init_type=param_init_type) self.norm3 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.norm4 = LayerNorm((embedding_dim,), eps=layer_norm_eps) self.cross_attn_image_to_token = Attention(embedding_dim, num_heads, downsample_rate=attention_downsample_rate, compute_dtype=compute_dtype, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type) self.skip_first_layer_pe = skip_first_layer_pe def construct(self, queries: ms.Tensor, keys: ms.Tensor, query_pe: ms.Tensor, key_pe: ms.Tensor) -> Tuple[ms.Tensor, ms.Tensor]: """ Apply the two-way attention block. Args: queries (ms.Tensor): Queries tensor. keys (ms.Tensor): Keys tensor. query_pe (ms.Tensor): Query positional encoding. key_pe (ms.Tensor): Key positional encoding. Returns: ms.Tensor: Processed queries tensor. ms.Tensor: Processed keys tensor. """ # Self attention block if self.skip_first_layer_pe: queries = self.self_attn(q=queries, k=queries, v=queries) else: q = queries + query_pe attn_out = self.self_attn(q=q, k=q, v=queries) queries = queries + attn_out queries = self.norm1(queries) # Cross attention block, tokens attending to image embedding q = queries + query_pe k = keys + key_pe attn_out = self.cross_attn_token_to_image(q=q, k=k, v=keys) queries = queries + attn_out queries = self.norm2(queries) # MLP block mlp_out = self.mlp(queries) queries = queries + mlp_out queries = self.norm3(queries) # Cross attention block, image embedding attending to tokens q = queries + query_pe k = keys + key_pe attn_out = self.cross_attn_image_to_token(q=k, k=q, v=queries) keys = keys + attn_out keys = self.norm4(keys) return queries, keys class Attention(nn.Cell): """ Attention layer that supports downscaling the embedding size after projection to queries, keys, and values. """ def __init__(self, embedding_dim: int, num_heads: int, downsample_rate: int = 1, compute_dtype=mstype.float16, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32) -> None: """ Initialize the Attention layer. Args: embedding_dim (int): The channel dimension of the embeddings. num_heads (int): The number of heads in the attention layer. downsample_rate (int): Downsample rate for the attention. """ super().__init__() self.embedding_dim = embedding_dim self.internal_dim = embedding_dim // downsample_rate self.num_heads = num_heads self.compute_dtype = compute_dtype self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type assert self.internal_dim % num_heads == 0, "num_heads must divide embedding_dim." self.inv_norm_factor = ms.Tensor(1.0 / math.sqrt(self.internal_dim // self.num_heads)) self.q_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.k_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.v_proj = Linear(in_channels=embedding_dim, out_channels=self.internal_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.out_proj = Linear(in_channels=self.internal_dim, out_channels=embedding_dim, compute_dtype=compute_dtype, param_init_type=param_init_type) self.softmax = P.Softmax(axis=-1) self.batchmatmul_trans_b = P.BatchMatMul(transpose_b=True) self.batchmatmul = P.BatchMatMul() self.mul = P.Mul() def _separate_heads(self, x: ms.Tensor, num_heads: int) -> ms.Tensor: """ Separate the input tensor into multiple heads. Args: x (ms.Tensor): Input tensor. num_heads (int): Number of heads to separate into. Returns: ms.Tensor: Tensor with dimensions reshaped for separate heads. """ b, n, c = x.shape x = x.reshape(b, n, num_heads, c // num_heads) return x.transpose(0, 2, 1, 3) def _recombine_heads(self, x: ms.Tensor) -> ms.Tensor: """ Recombine the tensor with multiple heads into a single tensor. Args: x (ms.Tensor): Input tensor with separate heads. Returns: ms.Tensor: Recombined tensor. """ b, n_heads, n_tokens, c_per_head = x.shape x = x.transpose(0, 2, 1, 3) return x.reshape(b, n_tokens, n_heads * c_per_head) def construct(self, q: ms.Tensor, k: ms.Tensor, v: ms.Tensor) -> ms.Tensor: """ Apply the attention mechanism. Args: q (ms.Tensor): Queries tensor. k (ms.Tensor): Keys tensor. v (ms.Tensor): Values tensor. Returns: ms.Tensor: Output tensor after attention. """ ori_type = k.dtype # Input projections q = self.q_proj(q).astype(self.compute_dtype) k = self.k_proj(k).astype(self.compute_dtype) v = self.v_proj(v).astype(self.compute_dtype) # Separate into heads q = self._separate_heads(q, self.num_heads) k = self._separate_heads(k, self.num_heads) v = self._separate_heads(v, self.num_heads) # Attention attn = self.batchmatmul_trans_b(q, k) attn = self.mul(attn, self.inv_norm_factor) attn = self.softmax(attn.astype(self.softmax_compute_type)) # Get output out = self.batchmatmul(attn.astype(self.compute_dtype), v) out = self._recombine_heads(out) out = self.out_proj(out).astype(ori_type) return out # Lightly adapted from # https://github.com/facebookresearch/MaskFormer/blob/main/mask_former/modeling/transformer/transformer_predictor.py # noqa class MLP(nn.Cell): """ Multi-Layer Perceptron (MLP) class definition. """ def __init__(self, input_dim: int, hidden_dim: int, output_dim: int, num_layers: int, sigmoid_output: bool = False, compute_dtype=mstype.float16, param_init_type=mstype.float32) -> None: super().__init__() self.num_layers = num_layers h = [hidden_dim] * (num_layers - 1) self.layers = nn.CellList([ Linear(in_channels=n, out_channels=k, compute_dtype=compute_dtype, param_init_type=param_init_type) for n, k in zip([input_dim] + h, h + [output_dim]) ]) self.sigmoid_output = sigmoid_output self.relu = P.ReLU() self.sigmoid = P.Sigmoid() def construct(self, x): """ Args: x (Tensor): Input tensor. Returns: Tensor: Output tensor of the MLP. """ for i, layer in enumerate(self.layers): x = self.relu(layer(x)) if i < self.num_layers - 1 else layer(x) if self.sigmoid_output: x = self.sigmoid(x) return x pengcheng.mind-mindformer/mindformers/models/sam/sam_processor.py000066400000000000000000000205541476650141400257410ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Processor""" from copy import deepcopy from typing import Tuple import numpy as np from PIL import Image import mindspore as ms from mindspore.dataset import vision from mindspore.dataset.vision import Inter from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SAMImageProcessor(BaseImageProcessor): """ Subclass of BaseImageProcessor implementing image preprocessing for SAM model. Args: img_size (int): The target size for image processing. mean (list): Mean values for image normalization. std (list): Standard deviation values for image normalization. """ def __init__(self, img_size=1024, mean=(123.675, 116.28, 103.53), std=(58.395, 57.12, 57.375)): super().__init__() self.img_size = img_size self.pixel_mean = ms.Tensor(mean).view(-1, 1, 1) self.pixel_std = ms.Tensor(std).view(-1, 1, 1) self.transform = ResizeLongestSide(img_size) def preprocess(self, images, **kwargs): """ Preprocess the input image for SAM model. Args: image (PIL.Image or numpy.array): Input image to be preprocessed. Returns: image (tensor): Preprocessed image as a tensor. input_size (tuple): Size of the input image after preprocessing. """ images = self.transform.apply_image(images) images = images.transpose(2, 0, 1)[None, :, :, :] images = ms.Tensor(images) input_size = images.shape[-2:] # Normalize colors images = (images - self.pixel_mean) / self.pixel_std # Pad h, w = images.shape[-2:] padh = self.img_size - h padw = self.img_size - w if images.ndim == 4: pad = P.Pad(paddings=((0, 0), (0, 0), (0, padh), (0, padw))) elif images.ndim == 3: pad = P.Pad(paddings=((0, 0), (0, padh), (0, padw))) images = pad(images) return images, input_size class ResizeLongestSide: """ Resizes images to the longest side 'target_length', as well as provides methods for resizing coordinates and boxes. Provides methods for transforming both numpy array and batched mindspore tensors. """ def __init__(self, target_length: int) -> None: """ Initialize the ResizeLongestSide class. Args: target_length (int): Target length of the longest side. """ self.target_length = target_length def apply_image(self, image: np.ndarray) -> np.ndarray: """ Apply the resizing transformation to a single image. Args: image (np.ndarray): Input image as HxWxC numpy array. Returns: np.ndarray: Resized image as H'xW'xC numpy array. """ target_size = self.get_preprocess_shape(image.shape[0], image.shape[1], self.target_length) pil_image = Image.fromarray(image) resize_op = vision.Resize(size=target_size) resized_image = np.array(resize_op(pil_image)) return resized_image def apply_coords(self, coords: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to coordinates. Args: coords (np.ndarray): Input coordinates as a numpy array of shape (N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized coordinates as a numpy array of shape (N, 2). """ old_h, old_w = original_size new_h, new_w = self.get_preprocess_shape( original_size[0], original_size[1], self.target_length ) coords = deepcopy(coords).astype(float) coords[..., 0] = coords[..., 0] * (new_w / old_w) coords[..., 1] = coords[..., 1] * (new_h / old_h) return coords def apply_boxes(self, boxes: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to bounding boxes. Args: boxes (np.ndarray): Input boxes as a numpy array of shape (B, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized boxes as a numpy array of shape (B, 4). """ boxes = self.apply_coords(boxes.reshape(-1, 2, 2), original_size) return boxes.reshape(-1, 4) def apply_image_batch(self, image: np.ndarray) -> np.ndarray: """ Apply the resizing transformation to a batch of images. Args: image (np.ndarray): Input batched images as BxCxHxW numpy array. Returns: np.ndarray: Resized batched images as BxC'xH'xW' numpy array. """ # Expects an image in BCHW format. May not exactly match apply_image. target_size = self.get_preprocess_shape(image.shape[2], image.shape[3], self.target_length) resize_ = vision.Resize(target_size, Inter.BILINEAR) image = resize_(image) return image def apply_coords_batch(self, coords: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to a batch of coordinates. Args: coords (np.ndarray): Input batched coordinates as numpy array of shape (B, N, 2). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched coordinates as numpy array of shape (B, N, 2). """ old_h, old_w = original_size new_h, new_w = self.get_preprocess_shape( original_size[0], original_size[1], self.target_length ) coords = deepcopy(coords) coords[..., 0] = coords[..., 0] * (new_w / old_w) coords[..., 1] = coords[..., 1] * (new_h / old_h) return coords def apply_boxes_batch(self, boxes: np.ndarray, original_size: Tuple[int, ...]) -> np.ndarray: """ Apply the resizing transformation to a batch of bounding boxes. Args: boxes (np.ndarray): Input batched boxes as numpy array of shape (B, N, 4). original_size (Tuple[int, int]): Original image size as (H, W). Returns: np.ndarray: Resized batched boxes as numpy array of shape (B, N, 4). """ boxes = self.apply_coords_batch(boxes.reshape(-1, 2, 2), original_size) return boxes.reshape(-1, 4) @staticmethod def get_preprocess_shape(oldh: int, oldw: int, long_side_length: int) -> Tuple[int, int]: """ Compute the output size given input size and target long side length. Args: oldh (int): Original image height. oldw (int): Original image width. long_side_length (int): Target length of the longest side. Returns: Tuple[int, int]: Output size as (new_h, new_w). """ scale = long_side_length * 1.0 / max(oldh, oldw) newh, neww = oldh * scale, oldw * scale neww = int(neww + 0.5) newh = int(newh + 0.5) return (newh, neww) @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SAMProcessor(BaseProcessor): """ Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['sam'] def __init__(self, image_processor=None, return_tensors='ms'): super().__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-mindformer/mindformers/models/sam/sam_prompt_encoder.py000066400000000000000000000245171476650141400267450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Prompt Encoder""" from typing import Optional, Tuple import mindspore as ms import mindspore.nn as nn import mindspore.ops as ops import mindspore.numpy as np import mindspore.common.dtype as mstype from mindspore import Parameter from mindspore.ops import operations as P from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from .sam_layers import LayerNorm2d @MindFormerRegister.register(MindFormerModuleType.MODELS) class SAMPromptEncoder(BaseModel): """ Encodes prompts for input to SAM's mask decoder. """ _support_list = MindFormerBook.get_model_support_list()['sam'] def __init__(self, config) -> None: super().__init__(config) self.embed_dim = config.prompt_embed_dim self.input_image_size = config.input_image_size self.image_embedding_size = config.image_embedding_size self.mask_in_chans = config.mask_in_chans self.activation = nn.GELU self.pe_layer = PositionEmbeddingRandom(self.embed_dim // 2) self.num_point_embeddings = 4 # pos/neg point + 2 box corners point_embeddings = [nn.Embedding(1, self.embed_dim) for i in range(self.num_point_embeddings)] self.point_embeddings = nn.CellList(point_embeddings) self.not_a_point_embed = nn.Embedding(1, self.embed_dim) self.mask_input_size = (4 * self.image_embedding_size[0], 4 * self.image_embedding_size[1]) self.mask_downscaling = nn.SequentialCell( nn.Conv2d(1, self.mask_in_chans // 4, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.mask_in_chans // 4), self.activation(), nn.Conv2d(self.mask_in_chans // 4, self.mask_in_chans, kernel_size=2, stride=2, has_bias=True), LayerNorm2d(self.mask_in_chans), self.activation(), nn.Conv2d(self.mask_in_chans, self.embed_dim, kernel_size=1, has_bias=True), ) self.no_mask_embed = nn.Embedding(1, self.embed_dim) self.zeros = P.Zeros() self.ones = P.Ones() self.expand_dims = P.ExpandDims() self.concat = P.Concat(axis=1) self.tile = P.Tile() def get_dense_pe(self) -> ms.Tensor: """ Get the positional encoding used to encode point prompts, applied to a dense set of points the shape of the image encoding. Returns: ms.Tensor: Positional encoding with shape 1x(embed_dim)x(embedding_h)x(embedding_w) """ return self.expand_dims(self.pe_layer(self.image_embedding_size), 0) def _embed_points(self, points: ms.Tensor, labels: ms.Tensor, pad: bool) -> ms.Tensor: """Embeds point prompts.""" bs = points.shape[0] points = points + 0.5 # Shift to center of pixel point_embedding = self.pe_layer.forward_with_coords(points, self.input_image_size) if pad: not_a_point_embed = self.tile(self.expand_dims(self.not_a_point_embed.embedding_table, 0), (bs, 1, 1)) point_embedding = self.concat([point_embedding, not_a_point_embed]) labels = self.concat([labels, -self.ones((bs, 1), mstype.int32)]) mask_neg = (labels == 0).reshape(bs, -1, 1) point_embedding += np.where(mask_neg, self.point_embeddings[0].embedding_table,\ np.zeros_like(self.point_embeddings[0].embedding_table)) mask_pos = (labels == 1).reshape(bs, -1, 1) point_embedding += np.where(mask_pos, self.point_embeddings[1].embedding_table,\ np.zeros_like(self.point_embeddings[1].embedding_table)) return point_embedding def _embed_boxes(self, boxes: ms.Tensor) -> ms.Tensor: """Embeds box prompts.""" boxes = boxes + 0.5 # Shift to center of pixel coords = boxes.reshape(-1, 2, 2) corner_embedding = self.pe_layer.forward_with_coords(coords, self.input_image_size) corner_embedding[:, 0, :] += self.point_embeddings[2].embedding_table corner_embedding[:, 1, :] += self.point_embeddings[3].embedding_table return corner_embedding def _embed_masks(self, masks: ms.Tensor) -> ms.Tensor: """Embeds mask inputs.""" mask_embedding = self.mask_downscaling(masks) return mask_embedding def _get_batch_size(self, points: Optional[ms.Tensor], boxes: Optional[ms.Tensor], masks: Optional[ms.Tensor]) -> int: """ Gets the batch size of the output given the batch size of the input prompts. """ if points is not None: return points.shape[0] if boxes is not None: return boxes.shape[0] if masks is not None: return masks.shape[0] return 1 def construct(self, point_coords: Optional[ms.Tensor], point_labels: Optional[ms.Tensor], boxes: Optional[ms.Tensor], mask_inputs: Optional[ms.Tensor]) -> Tuple[ms.Tensor, ms.Tensor]: """ Args: point_coords (ms.Tensor or none): Point coordinates to embed. point_labels (ms.Tensor or none): Point labels to embed. boxes (ms.Tensor or none): Boxes to embed. mask_inputs (ms.Tensor or none): Masks to embed. Returns: Tuple[ms.Tensor, ms.Tensor]: Sparse embeddings for the points and boxes, with shape BxNx(embed_dim), where N is determined by the number of input points and boxes. Dense embeddings for the masks, in the shape Bx(embed_dim)x(embed_H)x(embed_W). """ bs = self._get_batch_size(point_coords, boxes, mask_inputs) sparse_embeddings = np.empty((bs, 1, self.embed_dim)) # sparse_embeddings = None if point_coords is not None: point_embeddings = self._embed_points(point_coords, point_labels, pad=(boxes is None)) sparse_embeddings = point_embeddings if boxes is not None: box_embeddings = self._embed_boxes(boxes) sparse_embeddings = box_embeddings if mask_inputs is not None: dense_embeddings = self._embed_masks(mask_inputs) else: dense_embeddings = ops.broadcast_to(self.no_mask_embed.embedding_table.reshape(1, -1, 1, 1),\ (bs, -1, self.image_embedding_size[0], self.image_embedding_size[1])) return sparse_embeddings, dense_embeddings class PositionEmbeddingRandom(nn.Cell): """ Positional encoding using random spatial frequencies. """ def __init__(self, num_pos_feats: int = 64, scale: Optional[float] = None, compute_dtype=mstype.float16) -> None: """ Initialize the PositionEmbeddingRandom. Args: num_pos_feats (int): Number of positional encoding features. scale (float): Scale factor for the positional encoding Gaussian matrix. """ super().__init__() self.compute_dtype = compute_dtype if scale is None or scale <= 0.0: scale = 1.0 pe_gaussian_matrix = scale * P.StandardNormal()((2, num_pos_feats)).astype(self.compute_dtype) self.positional_encoding_gaussian_matrix = Parameter(pe_gaussian_matrix,\ name="positional_encoding_gaussian_matrix") self.cast = P.Cast() self.ones = P.Ones() self.sin = P.Sin() self.cos = P.Cos() self.concat = P.Concat(axis=-1) self.stack = P.Stack(axis=-1) def _pe_encoding(self, coords: ms.Tensor) -> ms.Tensor: """ Positionally encode points that are normalized to [0,1]. Args: coords (ms.Tensor): Coordinates normalized to [0, 1]. Returns: ms.Tensor: Positional encoding for the given coordinates. """ # assuming coords are in [0, 1]^2 square and have d_1 x ... x d_n x 2 shape coords = 2 * coords - 1 ori_type = coords.dtype # here must use ops.matmul, because 'x', 'y' have different dimension coords = ops.matmul(coords.astype(self.compute_dtype), self.positional_encoding_gaussian_matrix) coords = self.cast(coords, ori_type) coords = 2 * np.pi * coords # outputs d_1 x ... x d_n x C shape return self.concat([self.sin(coords), self.cos(coords)]) def construct(self, size: Tuple[int, int]) -> ms.Tensor: """ Generate positional encoding for a grid of the specified size. Args: size (Tuple[int, int]): Grid size (H, W). Returns: ms.Tensor: Positional encoding for the grid. """ h, w = size grid = self.ones((h, w), ms.float32) y_embed = grid.cumsum(axis=0) - 0.5 x_embed = grid.cumsum(axis=1) - 0.5 y_embed = y_embed / h x_embed = x_embed / w pe = self._pe_encoding(self.stack([x_embed, y_embed])) return pe.transpose(2, 0, 1) # C x H x W def forward_with_coords(self, coords_input: ms.Tensor, image_size: Tuple[int, int]) -> ms.Tensor: """ Positionally encode points that are not normalized to [0,1]. Args: coords_input (ms.Tensor): Coordinates not normalized to [0, 1]. image_size (Tuple[int, int]): Image size (H, W). Returns: ms.Tensor: Positional encoding for the given coordinates. """ coords = coords_input coords[:, :, 0] = coords[:, :, 0] / image_size[1] coords[:, :, 1] = coords[:, :, 1] / image_size[0] return self._pe_encoding(coords) pengcheng.mind-mindformer/mindformers/models/sam/sam_utils.py000066400000000000000000000543451476650141400250670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """SAM Utils""" import math from copy import deepcopy from itertools import product from typing import Any, Dict, Generator, ItemsView, List, Tuple import cv2 import numpy as np __all__ = [ "MaskData", "area_from_rle", "batch_iterator", "batched_mask_to_box", "box_xyxy_to_xywh", "build_all_layer_point_grids", "calculate_stability_score", "coco_encode_rle", "generate_crop_boxes", "is_box_near_crop_edge", "mask_to_rle", "remove_small_regions", "rle_to_mask", "uncrop_boxes_xyxy", "uncrop_masks", "uncrop_points", "box_area", "nms" ] class MaskData: """ A structure for storing masks and their related data in batched format. Implements basic filtering and concatenation. """ def __init__(self, **kwargs) -> None: """ Initialize the MaskData. Args: **kwargs: Keyword arguments representing different mask-related data. """ for v in kwargs.values(): assert isinstance( v, (list, np.ndarray) ), "MaskData only supports list, numpy arrays, and ms tensors." self._stats = dict(**kwargs) def __setitem__(self, key: str, item: Any) -> None: """ Set an item in the MaskData. Args: key (str): The key of the item. item (Any): The item to be set. """ assert isinstance( item, (list, np.ndarray) ), "MaskData only supports list, numpy arrays." self._stats[key] = item def __delitem__(self, key: str) -> None: """ Delete an item from the MaskData. Args: key (str): The key of the item to be deleted. """ del self._stats[key] def __getitem__(self, key: str) -> Any: """ Get an item from the MaskData. Args: key (str): The key of the item to be retrieved. Returns: Any: The retrieved item. """ return self._stats[key] def items(self) -> ItemsView[str, Any]: """ Get the items stored in the MaskData. Returns: ItemsView[str, Any]: A view of the items in the MaskData. """ return self._stats.items() def filter(self, keep: np.ndarray) -> None: """ Filter the MaskData based on a boolean mask. Args: keep (np.ndarray): Boolean mask indicating which items to keep. """ for k, v in self._stats.items(): if v is None: self._stats[k] = None elif isinstance(v, np.ndarray): self._stats[k] = v[keep] elif isinstance(v, list) and keep.dtype == np.bool: self._stats[k] = [a for i, a in enumerate(v) if keep[i]] elif isinstance(v, list): self._stats[k] = [v[i] for i in keep] else: raise TypeError(f"MaskData key {k} has an unsupported type {type(v)}.") def cat(self, new_stats: "MaskData") -> None: """ Concatenate new data from another MaskData instance. Args: new_stats (MaskData): The MaskData instance to concatenate. """ for k, v in new_stats.items(): if k not in self._stats or self._stats[k] is None: self._stats[k] = deepcopy(v) elif isinstance(v, np.ndarray): self._stats[k] = np.concatenate([self._stats[k], v], axis=0) elif isinstance(v, list): self._stats[k] = self._stats[k] + deepcopy(v) else: raise TypeError(f"MaskData key {k} has an unsupported type {type(v)}.") def is_box_near_crop_edge(boxes: np.ndarray, crop_box: List[int], orig_box: List[int], atol: float = 20.0) -> np.ndarray: """ Filter boxes at the edge of a crop, but not at the edge of the original image. Args: boxes (np.ndarray): Bounding boxes ndarray of shape (N, 4) in format (x1, y1, x2, y2). crop_box (List[int]): Crop box coordinates in format (x1, y1, x2, y2). orig_box (List[int]): Original image box coordinates in format (x1, y1, x2, y2). atol (float): Absolute tolerance for box comparison. Returns: np.ndarray: Boolean ndarray indicating whether boxes are near the crop edge. """ crop_box = np.array(crop_box, dtype=np.float32) orig_box = np.array(orig_box, dtype=np.float32) boxes = uncrop_boxes_xyxy(boxes, crop_box).astype(np.float32) near_crop_edge = np.isclose(boxes, np.tile(crop_box[None, :], (boxes.shape[0], 1)), atol=atol, rtol=0., equal_nan=True) near_image_edge = np.isclose(boxes, np.tile(orig_box[None, :], (boxes.shape[0], 1)), atol=atol, rtol=0., equal_nan=True) near_crop_edge = np.logical_and(near_crop_edge, ~near_image_edge) return np.any(near_crop_edge, axis=1) def box_xyxy_to_xywh(box_xyxy: np.ndarray) -> np.ndarray: """ Convert bounding box coordinates from (x1, y1, x2, y2) format to (x, y, w, h) format. Args: box_xyxy (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x1, y1, x2, y2) format. Returns: box_xywh (np.ndarray): Bounding boxes tensor of shape (N, 4) in (x, y, w, h) format. """ box_xywh = deepcopy(box_xyxy) box_xywh[2] = box_xywh[2] - box_xywh[0] box_xywh[3] = box_xywh[3] - box_xywh[1] return box_xywh def batch_iterator(batch_size: int, *args) -> Generator[List[Any], None, None]: """ Generate batches of data from input arrays. Args: batch_size (int): Size of each batch. *args: Variable number of input arrays to iterate over. Yields: List[Any]: A batch of data containing corresponding elements from each input array. """ assert args and all( len(a) == len(args[0]) for a in args ), "Batched iteration must have inputs of all the same size." n_batches = len(args[0]) // batch_size + int(len(args[0]) % batch_size != 0) for b in range(n_batches): yield [arg[b * batch_size : (b + 1) * batch_size] for arg in args] def mask_to_rle(tensor: np.ndarray) -> List[Dict[str, Any]]: """ Encode masks to an uncompressed RLE format, in the format expected by pycoco tools. Args: tensor (np.ndarray): Binary mask tensor of shape (B, H, W) where B is the batch size. Returns: List[Dict[str, Any]]: List of dictionaries representing RLE encoded masks. """ # Put in fortran order and flatten h,w b, h, w = tensor.shape tensor = tensor.transpose(0, 2, 1).reshape(tensor.shape[0], -1) # Compute change indices diff = np.bitwise_xor(tensor[:, 1:].astype(np.int32), tensor[:, :-1].astype(np.int32)) change_indices = np.stack(np.nonzero(diff)).transpose() # Encode run length out = [] for i in range(b): cur_idxs = change_indices[change_indices[:, 0] == i, 1] cur_idxs = np.concatenate( [ np.array([0], dtype=cur_idxs.dtype), cur_idxs + 1, np.array([h * w], dtype=cur_idxs.dtype) ] ) btw_idxs = cur_idxs[1:] - cur_idxs[:-1] counts = [] if tensor[i, 0] == 0 else [0] counts.extend(btw_idxs.tolist()) out.append({"size": [h, w], "counts": counts}) return out def rle_to_mask(rle: Dict[str, Any]) -> np.ndarray: """ Compute a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: np.ndarray: Binary mask as a NumPy array. """ h, w = rle["size"] mask = np.empty(h * w, dtype=bool) idx = 0 parity = False for count in rle["counts"]: mask[idx : idx + count] = parity idx += count parity ^= True mask = mask.reshape(w, h) return mask.transpose() # Put in C order def area_from_rle(rle: Dict[str, Any]) -> int: """ Calculate the area of a binary mask from an uncompressed RLE. Args: rle (Dict[str, Any]): Dictionary containing RLE encoded mask information. Returns: int: Calculated area of the mask. """ return sum(rle["counts"][1::2]) def calculate_stability_score(masks: np.ndarray, mask_threshold: float, threshold_offset: float) -> np.ndarray: """ Calculate the stability score for a batch of masks. The stability score is the Intersection over Union (IoU) between the binary masks obtained by thresholding the predicted mask logits at high and low values. Args: masks (np.ndarray): Predicted mask logits tensor of shape (B, H, W). mask_threshold (float): Threshold value for binary mask creation. threshold_offset (float): Offset to adjust the threshold. Returns: np.ndarray: Stability scores for each mask in the batch. """ # One mask is always contained inside the other. # Save memory by preventing unnecessary cast to torch.int64 intersections = ( (masks > (mask_threshold + threshold_offset)) .sum(-1, dtype=np.float32) .sum(-1, dtype=np.float32) ) unions = ( (masks > (mask_threshold - threshold_offset)) .sum(-1, dtype=np.float32) .sum(-1, dtype=np.float32) ) return intersections / unions def build_point_grid(n_per_side: int) -> np.ndarray: """ Generate a 2D grid of points evenly spaced in the [0, 1] x [0, 1] range. Args: n_per_side (int): Number of points per side of the grid. Returns: np.ndarray: 2D grid of points as a NumPy array. """ offset = 1 / (2 * n_per_side) points_one_side = np.linspace(offset, 1 - offset, n_per_side) points_x = np.tile(points_one_side[None, :], (n_per_side, 1)) points_y = np.tile(points_one_side[:, None], (1, n_per_side)) points = np.stack([points_x, points_y], axis=-1).reshape(-1, 2) return points def build_all_layer_point_grids(n_per_side: int, n_layers: int, scale_per_layer: int) -> List[np.ndarray]: """ Generate point grids for all crop layers. Args: n_per_side (int): Number of points per side of the original grid. n_layers (int): Number of layers in the crop hierarchy. scale_per_layer (int): Scaling factor applied per layer. Returns: List[np.ndarray]: List of point grids for each crop layer as NumPy arrays. """ points_by_layer = [] for i in range(n_layers + 1): n_points = int(n_per_side / (scale_per_layer**i)) points_by_layer.append(build_point_grid(n_points)) return points_by_layer def generate_crop_boxes(im_size: Tuple[int, ...], n_layers: int, overlap_ratio: float) -> Tuple[List[List[int]], List[int]]: """ Generate a list of crop boxes of different sizes. Each layer has (2**i)**2 boxes for the ith layer. Args: im_size (Tuple[int, ...]): Size of the original image in (height, width). n_layers (int): Number of layers in the crop hierarchy. overlap_ratio (float): Overlap ratio for calculating crop box size. Returns: Tuple[List[List[int]], List[int]]: Tuple containing a list of crop boxes and a list of layer indices for each crop box. """ crop_boxes, layer_idxs = [], [] im_h, im_w = im_size short_side = min(im_h, im_w) # Original image crop_boxes.append([0, 0, im_w, im_h]) layer_idxs.append(0) def crop_len(orig_len, n_crops, overlap): return int(math.ceil((overlap * (n_crops - 1) + orig_len) / n_crops)) for i_layer in range(n_layers): n_crops_per_side = 2 ** (i_layer + 1) overlap = int(overlap_ratio * short_side * (2 / n_crops_per_side)) crop_w = crop_len(im_w, n_crops_per_side, overlap) crop_h = crop_len(im_h, n_crops_per_side, overlap) crop_box_x0 = [int((crop_w - overlap) * i) for i in range(n_crops_per_side)] crop_box_y0 = [int((crop_h - overlap) * i) for i in range(n_crops_per_side)] # Crops in XYWH format for x0, y0 in product(crop_box_x0, crop_box_y0): box = [x0, y0, min(x0 + crop_w, im_w), min(y0 + crop_h, im_h)] crop_boxes.append(box) layer_idxs.append(i_layer + 1) return crop_boxes, layer_idxs def uncrop_boxes_xyxy(boxes: np.ndarray, crop_box: List[int]) -> np.ndarray: """ Uncrop the coordinates of bounding boxes from a cropped region to the original image. Args: boxes (np.ndarray): Bounding box coordinates in XYXY format (x0, y0, x1, y1). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped bounding box coordinates in XYXY format. """ x0, y0, _, _ = crop_box offset = np.array([[x0, y0, x0, y0]], dtype=np.int32) # Check if boxes has a channel dimension if len(boxes.shape) == 3: offset = np.expand_dims(offset, 1) return boxes + offset def uncrop_points(points: np.ndarray, crop_box: List[int]) -> np.ndarray: """ Uncrop the coordinates of points from a cropped region to the original image. Args: points (np.ndarray): Point coordinates in XY format (x, y). crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). Returns: np.ndarray: Uncropped point coordinates in XY format. """ x0, y0, _, _ = crop_box offset = np.array([[x0, y0]], dtype=np.int32) # Check if points has a channel dimension if len(points.shape) == 3: offset = np.expand_dims(offset, 1) return points + offset def uncrop_masks(masks: np.ndarray, crop_box: List[int], orig_h: int, orig_w: int) -> np.ndarray: """ Uncrop binary masks from a cropped region to the original image size. Args: masks (np.ndarray): Binary masks ndarray. crop_box (List[int]): Cropped region's bounding box in XYXY format (x0, y0, x1, y1). orig_h (int): Original height of the image. orig_w (int): Original width of the image. Returns: np.ndarray: Uncropped binary masks ndarray. """ x0, y0, x1, y1 = crop_box if x0 == 0 and y0 == 0 and x1 == orig_w and y1 == orig_h: return masks # Coordinate transform masks pad_x, pad_y = orig_w - (x1 - x0), orig_h - (y1 - y0) # Calculate pad widths pad_width = [(0, 0), (y0, pad_y - y0), (x0, pad_x - x0)] # Pad the masks return np.pad(masks, pad_width, mode='constant', constant_values=0) def remove_small_regions(mask: np.ndarray, area_thresh: float, mode: str) -> Tuple[np.ndarray, bool]: """ Removes small disconnected regions or holes in a binary mask. Args: mask (np.ndarray): Binary mask. area_thresh (float): Threshold area for removing small regions. mode (str): Either "holes" or "islands" indicating the type of regions to remove. Returns: Tuple[np.ndarray, bool]: A tuple containing the modified mask and a boolean indicating if the mask was modified. """ assert mode in ["holes", "islands"] correct_holes = mode == "holes" working_mask = (correct_holes ^ mask).astype(np.uint8) n_labels, regions, stats, _ = cv2.connectedComponentsWithStats(working_mask, 8) sizes = stats[:, -1][1:] # Row 0 is background label small_regions = [i + 1 for i, s in enumerate(sizes) if s < area_thresh] if not small_regions: return mask, False fill_labels = [0] + small_regions if not correct_holes: fill_labels = [i for i in range(n_labels) if i not in fill_labels] # If every region is below threshold, keep largest if not fill_labels: fill_labels = [int(np.argmax(sizes)) + 1] mask = np.isin(regions, fill_labels) return mask, True def coco_encode_rle(uncompressed_rle: Dict[str, Any]) -> Dict[str, Any]: """ Encode uncompressed RLE (Run-Length Encoding) to COCO format RLE. Args: uncompressed_rle (Dict[str, Any]): Uncompressed RLE dictionary. Returns: Dict[str, Any]: Encoded RLE in COCO format. """ from pycocotools import mask as mask_utils # type: ignore h, w = uncompressed_rle["size"] rle = mask_utils.frPyObjects(uncompressed_rle, h, w) rle["counts"] = rle["counts"].decode("utf-8") # Necessary to serialize with json return rle def batched_mask_to_box(masks): """ Convert masks to bounding boxes in XYXY format. Return [0,0,0,0] for an empty mask. Args: masks (np.ndarray): Binary masks of shape C1xC2x...xHxW. Returns: np.ndarray: Bounding boxes in XYXY format of shape C1xC2x...x4. """ # NumPy doesn't raise an error on empty inputs, so check for empty masks here if np.prod(masks.shape) == 0: return np.zeros((*masks.shape[:-2], 4), dtype=np.int32) # Normalize shape to CxHxW shape = masks.shape h, w = shape[-2:] if len(shape) > 2: masks = masks.reshape(-1, masks.shape[-2], masks.shape[-1]) else: masks = np.expand_dims(masks, 0) # Get top and bottom edges in_height = np.max(masks.astype(np.float32), axis=-1) in_height_coords = in_height * np.arange(h)[None, :] bottom_edges = np.max(in_height_coords, axis=-1) in_height_coords = in_height_coords + h * (~in_height.astype(np.bool)) top_edges = np.min(in_height_coords, axis=-1) # Get left and right edges in_width = np.max(masks.astype(np.float32), axis=-2) in_width_coords = in_width * np.arange(w)[None, :] right_edges = np.max(in_width_coords, axis=-1) in_width_coords = in_width_coords + w * (~in_width.astype(np.bool)) left_edges = np.min(in_width_coords, axis=-1) # If the mask is empty, the right edge will be to the left of the left edge. # Replace these boxes with [0, 0, 0, 0] empty_filter = ((right_edges < left_edges).astype(np.int32) |\ (bottom_edges < top_edges).astype(np.int32)).astype(np.bool) out = np.stack([left_edges, top_edges, right_edges, bottom_edges], axis=-1) out = out * np.expand_dims(~empty_filter, axis=-1) # Return to original shape if len(shape) > 2: out = out.reshape((*shape[:-2], 4)) else: out = out[0] return out.astype(np.int32) def box_area(boxes): """ Compute the area of a set of bounding boxes. Args: boxes (np.ndarray): Bounding boxes for which the area will be computed. Expected shape is [N, 4], where N is the number of boxes. Each box is specified by its coordinates (x1, y1, x2, y2) in the format with ``0 <= x1 < x2`` and ``0 <= y1 < y2``. Returns: np.ndarray: The area for each box in an ndarray of shape [N]. """ return (boxes[:, 2] - boxes[:, 0]) * (boxes[:, 3] - boxes[:, 1]) def box_iou(box1: np.ndarray, box2: np.ndarray): """ Calculate the Intersection over Union (IoU) between two sets of bounding boxes. Args: box1 (np.ndarray): NumPy array representing the first set of bounding boxes in format [x_min, y_min, x_max, y_max]. box2 (np.ndarray): NumPy array representing the second set of bounding boxes in the same format. Returns: iou (np.ndarray): NumPy array containing the IoU values for each pair of boxes. Example: >>> box1 = np.array([[0, 0, 10, 10], [5, 5, 15, 15]]) >>> box2 = np.array([[5, 5, 15, 15], [8, 8, 18, 18]]) >>> iou = box_iou(box1, box2) >>> print(iou) [[0.11111111 0. ] [1. 0.23076923]] """ area1 = box_area(box1) area2 = box_area(box2) lt = np.maximum(box1[:, np.newaxis, :2], box2[:, :2]) rb = np.minimum(box1[:, np.newaxis, 2:], box2[:, 2:]) wh = rb - lt wh = np.maximum(0, wh) inter = wh[:, :, 0] * wh[:, :, 1] iou = inter / (area1[:, np.newaxis] + area2 - inter) return iou def nms(boxes: np.ndarray, scores: np.ndarray, iou_threshold: float): """ Apply Non-Maximum Suppression (NMS) algorithm to filter a set of bounding boxes, eliminating overlapping boxes based on the Intersection over Union (IoU) threshold. Args: boxes (np.ndarray): NumPy array containing bounding box coordinates, with each row representing a box [x_min, y_min, x_max, y_max]. scores (np.ndarray): NumPy array containing scores for each corresponding bounding box. iou_threshold (float): IoU (Intersection over Union) threshold to determine if boxes overlap. Returns: keep (np.ndarray): NumPy array containing indices of the retained bounding boxes. Example: >>> boxes = np.array([[0, 0, 10, 10], [5, 5, 15, 15], [8, 8, 18, 18]]) >>> scores = np.array([0.9, 0.75, 0.85]) >>> iou_threshold = 0.5 >>> keep = nms(boxes, scores, iou_threshold) >>> print(keep) [0 2] """ idxs = scores.argsort() keep = [] while idxs.size > 0: max_score_index = idxs[-1] max_score_box = boxes[max_score_index][None, :] keep.append(max_score_index) if idxs.size == 1: break idxs = idxs[:-1] other_boxes = boxes[idxs] ious = box_iou(max_score_box, other_boxes) idxs = idxs[ious[0] <= iou_threshold] keep = np.array(keep) return keep pengcheng.mind-mindformer/mindformers/models/swin/000077500000000000000000000000001476650141400227025ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/swin/__init__.py000066400000000000000000000015651476650141400250220ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Swin Model API.""" from .swin import * from .swin_config import * from .swin_processor import * __all__ = [] __all__.extend(swin.__all__) __all__.extend(swin_config.__all__) __all__.extend(swin_processor.__all__) pengcheng.mind-mindformer/mindformers/models/swin/convert_weight.py000066400000000000000000000115251476650141400263070ustar00rootroot00000000000000"""Convert checkpoint from torch/MicroSoft""" import os import argparse import mindspore as ms import torch from mindformers.tools.utils import str2bool def convert_pretrained_weight(pth_file="swin_base_patch4_window7_224.pth", ms_ckpt_path="swin_base_p4w7.ckpt"): """ convert swin_base_p4w7 weights from pytorch to mindspore pytorch required. """ ms_ckpt = [] state_dict = torch.load(pth_file, map_location=torch.device('cpu')) for k, v in state_dict['model'].items(): if 'decoder.0' in k: k = k.replace('decoder.0', 'decoder') if 'patch_embed.' in k: k = k.replace('proj', 'projection') if 'relative_position' in k: k = k.replace('relative_position', 'relative_position_bias.relative_position') if 'norm' in k: if '.weight' in k: k = k.replace('.weight', '.gamma') if '.bias' in k: k = k.replace('.bias', '.beta') if '.mlp' in k: if '.fc1' in k: k = k.replace('.fc1', '.mapping') if "weight" in k: v = v.transpose(-1, 0) if '.fc2' in k: k = k.replace('.fc2', '.projection') if "weight" in k: v = v.transpose(-1, 0) if '.qkv' not in k: ms_ckpt.append({'name': k, 'data': ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = len(data) ms_ckpt.append({'name': k.replace('.qkv', '.q'), 'data': data[:length // 3]}) ms_ckpt.append({'name': k.replace('.qkv', '.k'), 'data': data[length // 3:length // 3 * 2]}) ms_ckpt.append({'name': k.replace('.qkv', '.v'), 'data': data[length // 3 * 2:]}) if not os.path.exists(ms_ckpt_path): try: ms.save_checkpoint(ms_ckpt, ms_ckpt_path) except: raise RuntimeError(f'Save checkpoint to {ms_ckpt_path} failed, please checkout the path.') def convert_finetuned_weight(pth_file="swin_base_patch4_window7_224.pth", ms_ckpt_path="swin_base_p4w7.ckpt"): """ convert swin_base_p4w7 weights from pytorch to mindspore pytorch required. """ ms_ckpt = [] state_dict = torch.load(pth_file, map_location=torch.device('cpu')) for k, v in state_dict['model'].items(): if 'head' in k: ms_ckpt.append({'name': k, 'data': ms.Tensor(v.numpy())}) continue if 'patch_embed.' in k: k = k.replace('proj', 'projection') if 'relative_position' in k: k = k.replace('relative_position', 'relative_position_bias.relative_position') if 'norm' in k: if '.weight' in k: k = k.replace('.weight', '.gamma') if '.bias' in k: k = k.replace('.bias', '.beta') if '.mlp' in k: if '.fc1' in k: k = k.replace('.fc1', '.mapping') if "weight" in k: v = v.transpose(-1, 0) if '.fc2' in k: k = k.replace('.fc2', '.projection') if "weight" in k: v = v.transpose(-1, 0) if '.qkv' not in k: ms_ckpt.append({'name': 'encoder.' + k, 'data': ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = len(data) ms_ckpt.append({'name': 'encoder.' + k.replace('.qkv', '.q'), 'data': data[:length // 3]}) ms_ckpt.append( {'name': 'encoder.' + k.replace('.qkv', '.k'), 'data': data[length // 3:length // 3 * 2]}) ms_ckpt.append({'name': 'encoder.' + k.replace('.qkv', '.v'), 'data': data[length // 3 * 2:]}) if not os.path.exists(ms_ckpt_path): try: ms.save_checkpoint(ms_ckpt, ms_ckpt_path) except: raise RuntimeError(f'Save checkpoint to {ms_ckpt_path} failed, please checkout the path.') if __name__ == '__main__': parser = argparse.ArgumentParser(description="swin weight convert script") parser.add_argument("--torch_path", type=str, default="swin_base_patch4_window7_224.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="swin_base_p4w7.ckpt", help="The output mindspore checkpoint path.") parser.add_argument("--is_pretrain", type=str2bool, default=False, help="whether converting pretrained weights.") opt = parser.parse_args() if opt.is_pretrain: convert_pretrained_weight(opt.torch_path, opt.mindspore_path) else: convert_finetuned_weight(opt.torch_path, opt.mindspore_path) pengcheng.mind-mindformer/mindformers/models/swin/swin.py000066400000000000000000000373131476650141400242430ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/Swin-Transformer # ============================================================================ """Swin Model.""" import os import numpy as np from scipy import interpolate from mindspore import nn from mindspore import Tensor from mindspore import Parameter from mindspore import dtype as mstype import mindspore.ops.operations as P import mindspore.common.initializer as weight_init_ from mindspore.train.serialization import load_checkpoint from mindspore.train.serialization import load_param_into_net from mindformers.tools.logger import logger from mindformers.tools.download_tools import download_with_progress_bar from mindformers.core.loss import build_loss from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_model import BaseModel from mindformers.models.base_config import BaseConfig from mindformers.models.swin.swin_config import SwinConfig from mindformers.models.swin.swin_modules import Linear from mindformers.models.swin.swin_modules import LayerNorm from mindformers.models.swin.swin_modules import Dropout from mindformers.models.swin.swin_modules import SwinPatchEmbeddings from mindformers.models.swin.swin_modules import SwinPatchMerging from mindformers.models.swin.swin_modules import SwinStage from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.tools.utils import try_sync_file __all__ = ['SwinForImageClassification', 'SwinModel'] @MindFormerRegister.register(MindFormerModuleType.ENCODER) class SwinBaseModel(BaseModel): """ An abstract class to handle weights initialization and save weights decay grouping. """ def init_weights(self): """ Swin weight initialization, original timm impl (for reproducibility) """ for _, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init_.initializer( weight_init_.TruncatedNormal(sigma=0.02), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init_.initializer(weight_init_.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init_.initializer(weight_init_.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init_.initializer(weight_init_.Zero(), cell.beta.shape, cell.beta.dtype)) @staticmethod def no_weight_decay(): return {'absolute_pos_embed'} @staticmethod def no_weight_decay_keywords(): return {'relative_position_bias_table'} @MindFormerRegister.register(MindFormerModuleType.ENCODER) class SwinModel(SwinBaseModel): """ Swin Transformer. The supported model name could be selected from SwinForImageClassification.show_support_list(). Args: config (SwinConfig): the config of Swin model. """ def __init__(self, config: BaseConfig = None): if config is None: config = SwinConfig() super(SwinModel, self).__init__(config) dp = config.parallel_config.data_parallel self.parallel_config = config.parallel_config self.use_moe = config.moe_config.expert_num > 1 self.num_labels = config.num_labels self.num_layers = len(config.depths) self.embed_dim = config.embed_dim self.ape = config.use_absolute_embeddings self.num_channels = config.num_channels self.patch_size = config.patch_size self.patch_norm = config.patch_norm self.num_features = int(config.embed_dim * 2 ** (self.num_layers - 1)) self.mlp_ratio = config.mlp_ratio self.cast = P.Cast() # split image into non-overlapping patches self.patch_embed = SwinPatchEmbeddings(config) num_patches = self.patch_embed.num_patches self.num_patches = num_patches patches_resolution = self.patch_embed.grid_size self.patches_resolution = patches_resolution # absolute position embedding if self.ape: self.absolute_pos_embed = Parameter( Tensor(np.zeros((1, num_patches, config.embed_dim)), dtype=mstype.float32), name="ape") self.pos_drop = Dropout(keep_prob=1.0 - config.hidden_dropout_prob) self.pos_drop.shard(((dp, 1, 1),)) # stochastic depth dpr = list(np.linspace(0, config.drop_path_rate, sum(config.depths))) # stochastic depth decay rule if self.use_moe: parallel_config_args = config.parallel_config.moe_parallel_config else: parallel_config_args = config.parallel_config.dp_mp_config # build layers self.layers = nn.CellList() self.final_seq = num_patches # downsample seq_length for i_layer in range(self.num_layers): layer = SwinStage( config=config, dim=int(config.embed_dim * 2 ** i_layer), input_resolution=(patches_resolution[0] // (2 ** i_layer), patches_resolution[1] // (2 ** i_layer)), depth=config.depths[i_layer], num_heads=config.num_heads[i_layer], drop_path=dpr[sum(config.depths[:i_layer]):sum(config.depths[:i_layer + 1])], norm_layer=LayerNorm, downsample=SwinPatchMerging if (i_layer < self.num_layers - 1) else None, parallel_config=parallel_config_args) # downsample seq_length if i_layer < self.num_layers - 1: self.final_seq = self.final_seq // 4 self.layers.append(layer) self.norm = LayerNorm([self.num_features,], eps=config.layer_norm_eps).shard(((dp, 1, 1),)) self.transpose = P.Transpose().shard(((dp, 1, 1),)) self.avgpool = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.init_weights() def construct(self, x): x = self.patch_embed(x) if self.ape: x = x + self.absolute_pos_embed x = self.pos_drop(x) for layer in self.layers: x = layer(x) x = self.norm(x) # B L C x = self.avgpool(self.transpose(x, (0, 2, 1)), 2) # B C 1 return x @MindFormerRegister.register(MindFormerModuleType.MODELS) class SwinForImageClassification(SwinBaseModel): """ Swin Transformer Model. The supported model name could be selected from SwinForImageClassification.show_support_list(). Args: config (SwinConfig): the config of Swin model. Examples: >>> # input model name, load model and weights >>> model_a = SwinForImageClassification.from_pretrained('swin_base_p4w7') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('swin_base_p4w7') >>> model_b = SwinForImageClassification(config) """ _support_list = MindFormerBook.get_model_support_list()['swin'] def __init__(self, config: BaseConfig = None): if config is None: config = SwinConfig() super(SwinForImageClassification, self).__init__(config) self.encoder = SwinModel(config) parallel_config = config.parallel_config dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.head = Linear( self.encoder.num_features, self.encoder.num_labels, weight_init=weight_init_.TruncatedNormal(sigma=2e-5), compute_dtype=mstype.float32).to_float(mstype.float32) self.head.shard(strategy_bias=((dp, mp), (mp,)), strategy_matmul=((dp, 1), (mp, 1))) self.loss = build_loss(class_name=config.loss_type) self.load_checkpoint(config) def construct(self, image, target=None): x = self.encoder(image) out = self.head(x) if not self.training: return out, target loss = self.loss(out, target) return loss def load_checkpoint(self, config): """ load checkpoint for SwinForImageClassification. Args: config (ModelConfig): a model config instance, which could have attribute "checkpoint_name_or_path (str)". set checkpoint_name_or_path to a supported model name or a path to checkpoint, to load model weights. """ checkpoint_name_or_path = config.checkpoint_name_or_path if checkpoint_name_or_path: if not isinstance(checkpoint_name_or_path, str): raise TypeError(f"checkpoint_name_or_path should be a str," f" but got {type(checkpoint_name_or_path)}") if os.path.exists(checkpoint_name_or_path): param = load_checkpoint(checkpoint_name_or_path) ckpt_file = checkpoint_name_or_path param = self.remap_pretrained_keys_swin(param) try: load_param_into_net(self, param) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) logger.info("weights in %s are loaded", ckpt_file) elif checkpoint_name_or_path not in self._support_list: raise ValueError(f"{checkpoint_name_or_path} is not a supported default model" f" or a valid path to checkpoint," f" please select from {self._support_list}.") else: default_checkpoint_download_folder = os.path.join( MindFormerBook.get_default_checkpoint_download_folder(), checkpoint_name_or_path.split("_")[0]) if not os.path.exists(default_checkpoint_download_folder): os.makedirs(default_checkpoint_download_folder, exist_ok=True) ckpt_file = os.path.join(default_checkpoint_download_folder, checkpoint_name_or_path + ".ckpt") if not os.path.exists(ckpt_file): url = MindFormerBook.get_model_ckpt_url_list()[checkpoint_name_or_path][0] succeed = download_with_progress_bar(url, ckpt_file) if not succeed: logger.info("checkpoint download failed, and pretrained weights are unloaded.") return try_sync_file(ckpt_file) logger.info("start to read the ckpt file: %s", os.path.getsize(ckpt_file)) param = load_checkpoint(ckpt_file) try: load_param_into_net(self, param) except RuntimeError: logger.error("the given config and weights in %s are" " mismatched, and weights load failed", ckpt_file) logger.info("weights in %s are loaded", ckpt_file) else: logger.info("model built, but weights is unloaded, since the config has no" " checkpoint_name_or_path attribute or" " checkpoint_name_or_path is None.") def remap_pretrained_keys_swin(self, checkpoint_model): # This class was refer to project: # https://github.com/microsoft/unilm/blob/master/beit/run_class_finetuning.py """remap pretrained keys swin""" # Geometric interpolation when pre-trained patch size mismatch with fine-tuned patch size state_dict = self.parameters_dict() all_keys = list(checkpoint_model.keys()) for key in all_keys: if "relative_position_bias_table" in key and "adam" not in key: relative_position_bias_table_pretrained = checkpoint_model[key] relative_position_bias_table_current = state_dict[key] l1, n_h1 = relative_position_bias_table_pretrained.shape l2, n_h2 = relative_position_bias_table_current.shape if n_h1 != n_h2: logger.info("Error in loading %s, passing......", key) else: if l1 != l2: logger.info("%s: Interpolate relative_position_bias_table using geo.", key) src_size = int(l1 ** 0.5) dst_size = int(l2 ** 0.5) def geometric_progression(a, r, n): return a * (1.0 - r ** n) / (1.0 - r) left, right = 1.01, 1.5 while right - left > 1e-6: q = (left + right) / 2.0 gp = geometric_progression(1, q, src_size // 2) if gp > dst_size // 2: right = q else: left = q dis = [] cur = 1 for i in range(src_size // 2): dis.append(cur) cur += q ** (i + 1) r_ids = [-_ for _ in reversed(dis)] x = r_ids + [0] + dis y = r_ids + [0] + dis t = dst_size // 2.0 dx = np.arange(-t, t + 0.1, 1.0) dy = np.arange(-t, t + 0.1, 1.0) str_x = str(x) str_dx = str(dx) logger.info("Original positions = %s", str_x) logger.info("Target positions = %s", str_dx) all_rel_pos_bias = [] for i in range(n_h1): z = relative_position_bias_table_pretrained[:, i].view( src_size, src_size).asnumpy().astype(np.float32) f_cubic = interpolate.interp2d(x, y, z, kind='cubic') all_rel_pos_bias.append(Tensor(f_cubic(dx, dy), mstype.float32).view(-1, 1)) new_rel_pos_bias = P.Concat(axis=-1)(tuple(all_rel_pos_bias)) new_rel_pos_bias = Parameter(new_rel_pos_bias) checkpoint_model[key] = new_rel_pos_bias # delete relative_coords_table and attn_mask, since we always re-init it checkpoint_model = self._del_unused_keys(checkpoint_model, "relative_position_index") checkpoint_model = self._del_unused_keys(checkpoint_model, "relative_coords_table") checkpoint_model = self._del_unused_keys(checkpoint_model, "attn_mask") return checkpoint_model @staticmethod def _del_unused_keys(checkpoint_model, key_name): unwanted_key = [k for k in checkpoint_model.keys() if key_name in k] for k in unwanted_key: del checkpoint_model[k] return checkpoint_model pengcheng.mind-mindformer/mindformers/models/swin/swin_config.py000066400000000000000000000135421476650141400255660ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writingsoftware # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Swin Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SwinConfig'] default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class SwinConfig(BaseConfig): """ Swin config class which defines the model size Args: image_size: The input image size, Default 224. patch_size: patch size, Default 4. num_channels: channels of input images, Default 3. embed_dim: embedding dimension, Default 128. depths: number of transformer blocks for each swin layer, Default (2, 2, 18, 2). num_heads: number of attention heads for each swin layer, Default (4, 8, 16, 32). window_size: window size for swin, Default 7. shift_size: window shift size, Default 0. mlp_ratio: ffn_hidden_size = mlp_ratio * embed_dim, Default 4. qkv_bias: has transformer qkv bias or not, Default True. hidden_dropout_prob: drop rate of MLP, Default 0. attention_probs_dropout_prob: drop rate of Attention, Default 0. drop_path_rate: drop path rate of transformer blocks, Default 0.1. use_absolute_embeddings: if using absolute position embedding, Default False. patch_norm: use norm in SwinPatchEmbeddings, Default True. hidden_act: activation of MLP, Default "gelu". weight_init: weight initialize type, Default "normal". num_labels: number of labels in downstream tasks, Default 1000. loss_type: loss type, Default "SoftTargetCrossEntropy". param_init_type:, Default mstype.float32. moe_config:, Default default_moe_config. parallel_config:, Default default_parallel_config. checkpoint_name_or_path:, Default "swin_base_p4w7". **kwargs Examples: >>> # init a config with a model name >>> config_a = SwinConfig.from_pretrained('swin_base_p4w7') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'swin', 'run_swin_base_p4w7_224_100ep.yaml') >>> config_b = SwinConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = SwinConfig( >>> patch_size=4, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['swin'] def __init__(self, image_size: int = 224, patch_size: int = 4, num_channels: int = 3, embed_dim: int = 128, depths: list = (2, 2, 18, 2), num_heads: list = (4, 8, 16, 32), window_size: int = 7, shift_size: int = 0, mlp_ratio: float = 4., qkv_bias: bool = True, layer_norm_eps: float = 1e-5, hidden_dropout_prob: float = 0., attention_probs_dropout_prob: float = 0., drop_path_rate: float = 0.1, use_absolute_embeddings: bool = False, patch_norm: bool = True, hidden_act: str = 'gelu', weight_init: str = 'normal', num_labels: int = 1000, loss_type: str = "SoftTargetCrossEntropy", param_init_type: mstype = mstype.float32, moe_config: MoEConfig = default_moe_config, parallel_config: TransformerOpParallelConfig = default_parallel_config, checkpoint_name_or_path: str = '', **kwargs): """Swin Base Config""" self.image_size = image_size self.patch_size = patch_size self.num_channels = num_channels self.embed_dim = embed_dim self.depths = depths self.num_heads = num_heads self.window_size = window_size self.shift_size = shift_size self.mlp_ratio = mlp_ratio self.qkv_bias = qkv_bias self.layer_norm_eps = layer_norm_eps self.hidden_dropout_prob = hidden_dropout_prob self.attention_probs_dropout_prob = attention_probs_dropout_prob self.drop_path_rate = drop_path_rate self.use_absolute_embeddings = use_absolute_embeddings self.patch_norm = patch_norm self.hidden_act = hidden_act self.weight_init = weight_init self.num_labels = num_labels self.loss_type = loss_type self.param_init_type = param_init_type self.moe_config = moe_config self.parallel_config = parallel_config self.checkpoint_name_or_path = checkpoint_name_or_path super(SwinConfig, self).__init__(**kwargs) pengcheng.mind-mindformer/mindformers/models/swin/swin_modules.py000066400000000000000000000746751476650141400260070ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/Swin-Transformer # ============================================================================ """Swin Transformer API.""" import collections.abc from itertools import repeat import numpy as np from mindspore import nn from mindspore import numpy from mindspore import context from mindspore import Tensor from mindspore import Parameter from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.primitive import constexpr import mindspore.common.dtype as mstype import mindspore.common.initializer as weight_init_ from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.op_parallel_config import default_dpmp_config from mindformers.models.base_model import BaseModel # utils def _ntuple(n): def parse(x): if isinstance(x, collections.abc.Iterable): return x return tuple(repeat(x, n)) return parse to_2tuple = _ntuple(2) @constexpr def gen_shape(x_shape, ndim): """generate shape for x.""" return (x_shape,) + (1,) * (ndim + 1) class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 """ A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Linear(layers.Linear): # pylint: disable=W0212 """ Linear function for Swin. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of linear.""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Identity(nn.Cell): """No construct for x.""" def __init__(self, auto_prefix=True): super(Identity, self).__init__(auto_prefix=auto_prefix) def construct(self, x): """No construct.""" return x class Dropout(layers.Dropout): # pylint: disable=W0212 """ A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob, ndim=1, parallel_config=None): # pylint: disable=W0613 super(DropPath, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.drop = Dropout(keep_prob=1 - drop_prob) self.drop.shard(((1, 1, 1),)) shape = (1,) + (1,) * (ndim + 1) self.ndim = ndim self.mask = Tensor(np.ones(shape), dtype=mstype.float32) self.tile = P.Tile().shard(((1, 1, 1),)) self.mul = P.Mul().shard(((dp, 1, 1),)) def construct(self, x): """Dropout.""" if not self.training: return x shape = gen_shape(x.shape[0], self.ndim) mask = self.tile(self.mask, shape) out = self.drop(mask) out = self.mul(out, x) return out class SwinPatchEmbeddings(BaseModel): """Construct the embeddings from patch, position embeddings.""" def __init__(self, config): super(SwinPatchEmbeddings, self).__init__(config) if config.parallel_config: dp = config.parallel_config.data_parallel else: dp = 1 img_size = (config.image_size, config.image_size) patch_size = (config.patch_size, config.patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.projection = nn.Conv2d(in_channels=config.num_channels, out_channels=config.embed_dim, kernel_size=patch_size[0], stride=patch_size[0], weight_init=weight_init_.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.projection.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.projection.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm if config.patch_norm: self.norm = LayerNorm((config.embed_dim,), eps=config.layer_norm_eps).shard(((dp, 1, 1),)) else: self.norm = Identity() def construct(self, x): """construct""" # True x: bs False x: bs * dp x = self.projection(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) return x class SwinStage(BaseModel): """ Swin Basic Layer """ def __init__(self, config, dim, input_resolution, depth, num_heads, drop_path, norm_layer=LayerNorm, downsample=None, parallel_config=default_dpmp_config): super(SwinStage, self).__init__(config) self.dim = dim self.input_resolution = input_resolution self.depth = depth # build blocks self.blocks = nn.CellList([ SwinTransformerBlock(config=config, dim=dim, input_resolution=input_resolution, num_heads=num_heads, shift_size=0 if (i % 2 == 0) else config.window_size // 2, drop_path=drop_path[i] if isinstance(drop_path, list) else drop_path, norm_layer=norm_layer, parallel_config=parallel_config) for i in range(depth)]) # patch merging layer if downsample is not None: self.downsample = downsample(dim, input_resolution, config.weight_init, norm_layer=norm_layer, parallel_config=parallel_config) else: self.downsample = None def construct(self, x): """construct""" for blk in self.blocks: x = blk(x) if self.downsample is not None: x = self.downsample(x) return x def extra_repr(self) -> str: return f"dim={self.dim}, input_resolution={self.input_resolution}, depth={self.depth}" class SwinTransformerBlock(BaseModel): """ Swin Transformer Block. Args: dim (int): Number of input channels. input_resolution (tuple[int]): Input resolution. num_heads (int): Number of attention heads. shift_size (int): Shift size for SW-MSA. drop_path (float, optional): Stochastic depth rate. Default: 0.0 norm_layer (nn.Cell, optional): Normalization layer. Default: nn.LayerNorm/LayerNorm """ def __init__(self, config, dim, input_resolution, num_heads, shift_size, drop_path, norm_layer=LayerNorm, parallel_config=default_dpmp_config): super(SwinTransformerBlock, self).__init__(config) self.use_moe = (config.moe_config.expert_num > 1) if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.dim = dim self.input_resolution = input_resolution self.num_heads = num_heads self.window_size = config.window_size self.shift_size = shift_size self.mlp_ratio = config.mlp_ratio self.reshape = P.Reshape() if min(self.input_resolution) <= self.window_size: # if window size is larger than input resolution, we don't partition windows self.shift_size = 0 self.window_size = min(self.input_resolution) self.norm1 = norm_layer((dim,), eps=config.layer_norm_eps) self.norm1.shard(((dp, 1, 1),)) self.attn = SwinAttention(dim, window_size=to_2tuple(self.window_size), num_heads=num_heads, qkv_bias=config.qkv_bias, attn_drop=config.attention_probs_dropout_prob, proj_drop=config.hidden_dropout_prob) self.drop_path = DropPath(drop_path, ndim=1, parallel_config=parallel_config) \ if drop_path > 0. else Identity() self.norm2 = norm_layer((dim,), eps=config.layer_norm_eps) self.norm2.shard(((dp, 1, 1),)) self.mlp = SwinIntermediate(hidden_size=dim, ffn_hidden_size=int(dim * config.mlp_ratio), dropout_rate=config.hidden_dropout_prob, weight_init=config.weight_init, hidden_act=config.hidden_act, parallel_config=parallel_config) if self.shift_size > 0: # calculate attention mask for SW-MSA in_h, in_w = self.input_resolution img_mask = np.zeros((1, in_h, in_w, 1)) # 1 H W 1 h_slices = (slice(0, -self.window_size), slice(-self.window_size, -self.shift_size), slice(-self.shift_size, None)) w_slices = (slice(0, -self.window_size), slice(-self.window_size, -self.shift_size), slice(-self.shift_size, None)) cnt = 0 for h in h_slices: for w in w_slices: img_mask[:, h, w, :] = cnt cnt += 1 # img_mask: [1, 56, 56, 1] window_size: 7 mask_windows = window_partition(img_mask, self.window_size) # nW, window_size, window_size, 1 mask_windows = mask_windows.reshape(-1, self.window_size * self.window_size) # mask: [64, 49, 49] attn_mask = mask_windows[:, np.newaxis] - mask_windows[:, :, np.newaxis] attn_mask = Tensor(np.where(attn_mask == 0, 0., -100.), dtype=mstype.float32) self.attn_mask = Parameter(attn_mask, requires_grad=False, name="attention_mask") self.roll_pos = Roll(self.shift_size, parallel_config=parallel_config) self.roll_neg = Roll(-self.shift_size, parallel_config=parallel_config) else: self.attn_mask = None self.dtype = P.DType() self.add_3d = P.Add().shard(((dp, 1, 1), (dp, 1, 1))) self.window_partition = WindowPartition(self.window_size) self.window_reverse = WindowReverse() def construct(self, x): """construct function""" h, w = self.input_resolution b, _, c = x.shape ori_type = self.dtype(x) shortcut = x x = self.norm1(x) x = self.reshape(x, (b, h, w, c)) # cyclic shift if self.shift_size > 0: shifted_x = self.roll_neg(x) else: shifted_x = x # partition windows x_windows = self.window_partition(shifted_x) # nW*B, window_size, window_size, C x_windows = self.reshape(x_windows, (-1, self.window_size * self.window_size, c)) # nW*B, window_size*window_size, C # W-MSA/SW-MSA attn_windows = self.attn(x_windows, mask=self.attn_mask) # nW*B, window_size*window_size, C # merge windows attn_windows = self.reshape(attn_windows, (-1, self.window_size, self.window_size, c)) shifted_x = self.window_reverse(attn_windows, self.window_size, h, w) # B H' W' C # reverse cyclic shift if self.shift_size > 0: x = self.roll_pos(shifted_x) else: x = shifted_x x = self.reshape(x, (b, h * w, c)) # FFN x = self.drop_path(x) x = self.cast(x, ori_type) x = self.add_3d(shortcut, x) x_tmp = self.norm2(x) x_tmp = self.mlp(x_tmp) x_tmp = self.drop_path(x_tmp) output = self.add_3d(x, x_tmp) output = self.cast(output, ori_type) return output def extra_repr(self) -> str: return f"dim={self.dim}, input_resolution={self.input_resolution}, num_heads={self.num_heads}, " \ f"window_size={self.window_size}, shift_size={self.shift_size}, mlp_ratio={self.mlp_ratio}" class SwinPatchMerging(nn.Cell): """ Patch Merging Layer. Args: input_resolution (tuple[int]): Resolution of input feature. dim (int): Number of input channels. norm_layer (nn.Module, optional): Normalization layer. Default: nn.LayerNorm/LayerNorm """ def __init__(self, dim, input_resolution, weight_init='normal', norm_layer=LayerNorm, parallel_config=default_dpmp_config): super(SwinPatchMerging, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.input_resolution = input_resolution self.dim = dim[0] if isinstance(dim, tuple) and len(dim) == 1 else dim # Default False self.reduction = Linear(in_channels=4 * dim, out_channels=2 * dim, has_bias=False, weight_init=weight_init).to_float(mstype.float16) self.reduction.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.norm = norm_layer([dim * 4,], eps=1e-5) self.norm.shard(((dp, 1, 1),)) self.h, self.w = self.input_resolution self.h_2, self.w_2 = self.h // 2, self.w // 2 self.h2w2 = int(self.h * self.w // 4) self.dim_mul_4 = int(dim * 4) self.cast = P.Cast() self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): """ x: B, H*W, C """ b = x.shape[0] x = self.reshape(x, (b, self.h_2, 2, self.w_2, 2, self.dim)) x = self.transpose(x, (0, 1, 3, 4, 2, 5)) x = self.reshape(x, (b, self.h2w2, self.dim_mul_4)) x = self.norm(x) x = self.cast(x, mstype.float16) x = self.reduction(x) x = self.cast(x, mstype.float32) return x def extra_repr(self) -> str: return f"input_resolution={self.input_resolution}, dim={self.dim}" class SwinAttention(nn.Cell): r""" Window based multi-head self attention (W-MSA) Cell with relative position bias. It supports both of shifted and non-shifted window. Args: dim (int): Number of input channels. window_size (tuple[int]): The height and width of the window. num_heads (int): Number of attention heads. qkv_bias (bool, optional): If True, add a learnable bias to query, key, value. Default: True qk_scale (float | None, optional): Override default qk scale of head_dim ** -0.5 if set attn_drop (float, optional): Dropout ratio of attention weight. Default: 0.0 proj_drop (float, optional): Dropout ratio of output. Default: 0.0 """ def __init__(self, dim, window_size, num_heads, qkv_bias=True, qk_scale=None, attn_drop=0., proj_drop=0., compute_dtype=mstype.float16, param_init_type=mstype.float32, softmax_compute_type=mstype.float32, parallel_config=default_dpmp_config): super(SwinAttention, self).__init__() if isinstance(dim, tuple) and len(dim) == 1: dim = dim[0] if parallel_config: dp = parallel_config.data_parallel mp = parallel_config.model_parallel else: dp = mp = 1 self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dim = dim self.window_size = window_size # Wh, Ww self.num_heads = num_heads head_dim = dim // num_heads self.scale_factor = Tensor(qk_scale or head_dim ** -0.5, mstype.float32) self.relative_position_bias = SwinRelativePositionBias(self.window_size, num_heads) # get pair-wise relative position index for each token inside the window self.q = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.q.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.k = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.k.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.v = Linear( in_channels=dim, out_channels=dim, has_bias=qkv_bias, param_init_type=param_init_type).to_float(compute_dtype) self.v.shard(strategy_matmul=((dp, 1), (mp, 1)), strategy_bias=((dp, mp), (mp,))) self.attn_drop = Dropout(keep_prob=1.0 - attn_drop) self.proj = Linear( in_channels=dim, out_channels=dim, has_bias=True, param_init_type=param_init_type).to_float(compute_dtype) self.proj.shard(strategy_bias=((dp, 1), (1,)), strategy_matmul=((dp, mp), (mp, 1))) self.proj_drop = Dropout(keep_prob=1.0 - proj_drop) self.proj_drop.shard(((dp, mp, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((dp, mp, 1, 1),)) self.cast = P.Cast() self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) self.matmul = P.BatchMatMul().shard(((dp, mp, 1, 1), (dp, mp, 1, 1))) self.real_div = P.RealDiv().shard(((dp, mp, 1, 1), ())) self.sub = P.Sub().shard(((1,), (dp, 1, 1, 1))) self.mul = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.add_4d = P.Add().shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.add_5d = P.Add().shard(((dp, 1, 1, 1, 1), (dp, 1, 1, 1, 1))) self.dtype = P.DType() self.shape = P.Shape() self.compute_type = compute_dtype self.softmax_dtype = softmax_compute_type def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ attention_probs = self.softmax(attention_scores) return attention_probs def construct(self, x, mask=None): """ Args: x: input features with shape of (num_windows*B, N, C) mask: (0/-inf) mask with shape of (num_windows, Wh*Ww, Wh*Ww) or None """ b, seq, c = x.shape ori_type = self.dtype(x) x = self.cast(x, self.compute_type) q = self.q(x) k = self.k(x) v = self.v(x) q = self.transpose(self.reshape(q, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 1, 3)) k = self.transpose(self.reshape(k, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 3, 1)) v = self.transpose(self.reshape(v, (b, seq, self.num_heads, c // self.num_heads)), (0, 2, 1, 3)) factor = self.cast(self.scale_factor, self.compute_type) q = self.mul(q, factor) attn = self.matmul(q, k) attn = self.cast(attn, ori_type) attn = self.add_4d(attn, self.relative_position_bias()) if mask is not None: nw, ws2, _ = mask.shape # mask: [64, 49, 49] ==> [1, 64, 1, 49, 49] mask = self.reshape(mask, (1, nw, 1, ws2, ws2)) attn = self.reshape(attn, (b // nw, nw, self.num_heads, seq, seq)) attn = self.add_5d(attn, mask) attn = self.reshape(attn, (-1, self.num_heads, seq, seq)) attn = self._softmax(attn) else: attn = self._softmax(attn) attn = self.cast(attn, self.compute_type) attn = self.attn_drop(attn) x = self.matmul(attn, v) x = self.transpose(x, (0, 2, 1, 3)) x = self.reshape(x, (b, seq, c)) x = self.proj(x) x = self.proj_drop(x) return x def extra_repr(self) -> str: return f'dim={self.dim}, window_size={self.window_size}, num_heads={self.num_heads}' class SwinIntermediate(FeedForward): """MLP for Swin Transformer Blocks.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='normal', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(SwinIntermediate, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class SwinRelativePositionBias(nn.Cell): """relative position bias for swin""" def __init__(self, window_size, num_heads): super(SwinRelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) self.relative_position_bias_table = Parameter( weight_init_.initializer( weight_init_.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = Tensor(np.sum(relative_coords, axis=-1), mstype.int32) # Wh*Ww, Wh*Ww self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.expand_dim = P.ExpandDims() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1], self.window_size[0] * self.window_size[1], -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) relative_position_bias = self.expand_dim(relative_position_bias, 0) return relative_position_bias def window_partition(x, window_size): """ Args: x: (B, H, W, C) window_size (int): window size Returns: windows: (num_windows*B, window_size, window_size, C) """ b, h, w, c = x.shape x = np.reshape(x, (b, h // window_size, window_size, w // window_size, window_size, c)) windows = x.transpose(0, 1, 3, 2, 4, 5).reshape(-1, window_size, window_size, c) return windows class Roll(nn.Cell): """Roll Cell""" def __init__(self, shift_size, shift_axis=(1, 2), parallel_config=default_dpmp_config): # pylint: disable=W0613 super(Roll, self).__init__() if parallel_config: dp = parallel_config.data_parallel # pylint: disable=W0612 else: dp = 1 # pylint: disable=W0612 self.shift_size = to_2tuple(shift_size) self.shift_axis = shift_axis def construct(self, x): x = numpy.roll(x, self.shift_size, self.shift_axis) return x class WindowPartition(nn.Cell): """WindowPartitionConstruct Cell""" def __init__(self, window_size, parallel_config=None): super(WindowPartition, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) self.window_size = window_size def construct(self, x): """ Args: x: (B, H, W, C) Returns: windows: (num_windows*B, window_size, window_size, C) """ b, h, w, c = x.shape x = self.reshape(x, (b, h // self.window_size, self.window_size, w // self.window_size, self.window_size, c)) x = self.transpose(x, (0, 1, 3, 2, 4, 5)) x = self.reshape(x, (b * h * w // (self.window_size ** 2), self.window_size, self.window_size, c)) return x class WindowReverse(nn.Cell): """WindowReverseConstruct Cell""" def __init__(self, parallel_config=None): super(WindowReverse, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1),)) def construct(self, windows, window_size, h, w): """ Args: windows: (num_windows*B, window_size, window_size, C) window_size (int): Window size h (int): Height of image w (int): Width of image Returns: x: (B, H, W, C) """ b = windows.shape[0] // (h * w // window_size // window_size) x = self.reshape(windows, (b, h // window_size, w // window_size, window_size, window_size, -1)) x = self.transpose(x, (0, 1, 3, 2, 4, 5)) x = self.reshape(x, (b, h, w, -1)) return x pengcheng.mind-mindformer/mindformers/models/swin/swin_processor.py000066400000000000000000000122141476650141400263330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ SwinProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision.transforms import CenterCrop, ToTensor, Normalize, Rescale from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['SwinProcessor', 'SwinImageProcessor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SwinImageProcessor(BaseImageProcessor): """ SwinImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224): super(SwinImageProcessor, self).__init__(image_resolution=size) self.resize = Resize(256, interpolation='cubic') self.center_crop = CenterCrop(size) self.to_tensor = ToTensor() self.rescale = Rescale(1.0 / 255.0, 0.0) self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] for image in images: image = self.resize(image) image = self.center_crop(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) return Tensor(res) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class SwinProcessor(BaseProcessor): """ Swin processor, consists of a feature extractor (BaseFeatureEXtractor) for image input. Examples: >>> from mindformers import MindFormerBook >>> from mindformers.models import SwinProcessor >>> yaml_path = os.path.join(MindFormerBook.get_project_path(), "configs", ... "swin", "run_swin_base_p4w7_224_100ep.yaml") >>> # build SwinProcessor from pretrained >>> pro_a = SwinProcessor.from_pretrained('swin_base_p4w7') >>> # build SwinProcessor from config >>> pro_b = SwinProcessor.from_pretrained(yaml_path) """ _support_list = MindFormerBook.get_processor_support_list()['swin'] def __init__(self, image_processor=None, return_tensors='ms'): super(SwinProcessor, self).__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-mindformer/mindformers/models/t5/000077500000000000000000000000001476650141400222525ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/t5/__init__.py000066400000000000000000000021171476650141400243640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """t5 init""" from . import t5, t5_config from .mt5 import MT5ForConditionalGeneration from .t5 import T5ForConditionalGeneration from .t5_config import T5Config from .t5_processor import T5Processor from .t5_tokenizer import T5PegasusTokenizer, T5Tokenizer __all__ = [] __all__.extend(t5.__all__) __all__.extend(mt5.__all__) __all__.extend(t5_config.__all__) __all__.extend(t5_tokenizer.__all__) __all__.extend(t5_processor.__all__) pengcheng.mind-mindformer/mindformers/models/t5/convert_weight.py000066400000000000000000000176151476650141400256650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/huggingface""" import argparse import numpy as np import torch from mindspore import save_checkpoint, Tensor def generate_params_dict(total_layers, mindspore_params_per_layer, torch_params_per_layer, mindspore_additional_params, torch_additional_params): """ Generate the total parameter mapping of mindspore and pytorch. Args: total_layers(int): The total layers of the net. mindspore_params_per_layer(list): The list of params per layer for the net of mindspore. torch_params_per_layer(list): The list of params per layer for the net of pytorch. mindspore_additional_params(list): The list of params outside the layer for the net of mindspore torch_additional_params(list): The list of params outside the layer for the net of pytorch. Returns: A list of tuple. The first element is the parameter name of mindspore, the another is the parameter name of pytorch. """ mapped_params = list(zip(mindspore_params_per_layer, torch_params_per_layer)) ms_extend_param_list = [] torch_extend_param_list = [] for i in range(total_layers): for ms_para, torch_para in mapped_params: src = ms_para.format(i) tgt = torch_para.format(i) ms_extend_param_list.append(src) torch_extend_param_list.append(tgt) mapped_params = list(zip(mindspore_additional_params, torch_additional_params)) for ms_para, torch_para in mapped_params: ms_extend_param_list.append(ms_para) torch_extend_param_list.append(torch_para) return list(zip(ms_extend_param_list, torch_extend_param_list)) def get_converted_ckpt(mapped_params, weight_dict): """ Print the keys of the loaded checkpoint Args: mapped_params(dict): The loaded checkpoint. The key is parameter name and value is the numpy array. weight_dict(dict): The loaded pytorch checkpoint. Returns: None """ new_ckpt_list = [] # Currently, the ms_extend_param the torch_extend_param is the full parameters. for src, tgt in mapped_params: value = weight_dict[tgt].numpy() is_transpose = "" if '.o.' in tgt or '.wi.' in tgt or '.wo.' in tgt: value = np.transpose(value, [1, 0]) is_transpose = " transposed" print(f"Mapping table Mindspore:{src:<30} \t Torch:{tgt:<30} with shape {value.shape}" + f"---{is_transpose}") new_ckpt_list.append({"data": Tensor(value), "name": src}) return new_ckpt_list if __name__ == '__main__': parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--layers', type=int, default=1, help="The number of layers of the model to be converted.") parser.add_argument("--torch_path", type=str, default=None, required=True, help="The torch checkpoint path.") parser.add_argument("--mindspore_path", type=str, required=True, default="The output mindspore checkpoint path.", help="Use device nums, default is 128.") opt = parser.parse_args() state_dict = torch.load(opt.torch_path, map_location='cpu') ms_name = [ "t5_model.tfm_encoder.blocks.{}.layernorm1.gamma", "t5_model.tfm_encoder.blocks.{}.layernorm2.gamma", "t5_model.tfm_encoder.blocks.{}.attention.dense1.weight", "t5_model.tfm_encoder.blocks.{}.attention.dense2.weight", "t5_model.tfm_encoder.blocks.{}.attention.dense3.weight", "t5_model.tfm_encoder.blocks.{}.attention.projection.weight", "t5_model.tfm_encoder.blocks.{}.output.mapping.weight", "t5_model.tfm_encoder.blocks.{}.output.projection.weight", "t5_model.tfm_decoder.blocks.{}.layernorm1.gamma", "t5_model.tfm_decoder.blocks.{}.cross_attention_layernorm.gamma", "t5_model.tfm_decoder.blocks.{}.layernorm2.gamma", "t5_model.tfm_decoder.blocks.{}.attention.dense1.weight", "t5_model.tfm_decoder.blocks.{}.attention.dense2.weight", "t5_model.tfm_decoder.blocks.{}.attention.dense3.weight", "t5_model.tfm_decoder.blocks.{}.attention.projection.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense1.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense2.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.dense3.weight", "t5_model.tfm_decoder.blocks.{}.cross_attention.projection.weight", "t5_model.tfm_decoder.blocks.{}.output.mapping.weight", "t5_model.tfm_decoder.blocks.{}.output.projection.weight", ] torch_name = [ "encoder.block.{}.layer.0.layer_norm.weight", "encoder.block.{}.layer.1.layer_norm.weight", "encoder.block.{}.layer.0.SelfAttention.q.weight", "encoder.block.{}.layer.0.SelfAttention.k.weight", "encoder.block.{}.layer.0.SelfAttention.v.weight", "encoder.block.{}.layer.0.SelfAttention.o.weight", "encoder.block.{}.layer.1.DenseReluDense.wi.weight", "encoder.block.{}.layer.1.DenseReluDense.wo.weight", "decoder.block.{}.layer.0.layer_norm.weight", "decoder.block.{}.layer.1.layer_norm.weight", "decoder.block.{}.layer.2.layer_norm.weight", "decoder.block.{}.layer.0.SelfAttention.q.weight", "decoder.block.{}.layer.0.SelfAttention.k.weight", "decoder.block.{}.layer.0.SelfAttention.v.weight", "decoder.block.{}.layer.0.SelfAttention.o.weight", "decoder.block.{}.layer.1.EncDecAttention.q.weight", "decoder.block.{}.layer.1.EncDecAttention.k.weight", "decoder.block.{}.layer.1.EncDecAttention.v.weight", "decoder.block.{}.layer.1.EncDecAttention.o.weight", "decoder.block.{}.layer.2.DenseReluDense.wi.weight", "decoder.block.{}.layer.2.DenseReluDense.wo.weight", ] addition_mindspore = [ "t5_model.encoder_layernorm.gamma", "t5_model.decoder_layernorm.gamma", "t5_model.tfm_embedding_lookup.embedding_table", "t5_model.tfm_encoder.blocks.0.attention.bias_generator.embeddings_table", "t5_model.tfm_decoder.blocks.0.attention.bias_generator.embeddings_table", ] addition_torch = [ "encoder.final_layer_norm.weight", "decoder.final_layer_norm.weight", "shared.weight", "encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight", "decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight", ] mapped_param = generate_params_dict(total_layers=opt.layers, mindspore_params_per_layer=ms_name, torch_params_per_layer=torch_name, mindspore_additional_params=addition_mindspore, torch_additional_params=addition_torch) new_ckpt = get_converted_ckpt(mapped_param, state_dict) save_checkpoint(new_ckpt, opt.mindspore_path) print(f"Convert finished, the output is saved to {opt.mindspore_path}") pengcheng.mind-mindformer/mindformers/models/t5/mt5.py000066400000000000000000000253241476650141400233370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """mt5 model.""" import mindspore.common.dtype as mstype from mindspore import nn, ops from mindspore.common.tensor import Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindformers.core.loss import CrossEntropyLoss from mindformers.modules import FeedForward from mindformers.modules.layers import Linear from mindformers.modules.transformer.transformer import ( OpParallelConfig, default_dpmp_config, ) from ...mindformer_book import MindFormerBook from ...tools.register import MindFormerModuleType, MindFormerRegister from ..base_model import BaseModel from .t5 import T5Config, T5Model __all__ = ["MT5ForConditionalGeneration"] class T5FeedFowardGatedGelu(FeedForward): """ T5 feedfoward cell with gated-gelu as hidden act """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act="gelu", expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(T5FeedFowardGatedGelu, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=expert_num, expert_group_size=expert_group_size, param_init_type=param_init_type, parallel_config=parallel_config, ) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = int(parallel_config.data_parallel / ep) del self.mapping # replace origin mapping with 2 mappings self.mapping_0 = Linear( in_channels=hidden_size, out_channels=ffn_hidden_size, has_bias=False, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.mapping_0.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),), ) self.mapping_1 = Linear( in_channels=hidden_size, out_channels=ffn_hidden_size, has_bias=False, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.mapping_1.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), ) self.projection = Linear( in_channels=ffn_hidden_size, out_channels=hidden_size, has_bias=False, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type, ) self.projection.shard( strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,)) ) self.mul = ops.Mul() self.mul.shard(in_strategy=((dp, 1), (dp, 1))) def construct(self, x): """construct of T5 FeedFoward-GatedGelu""" x = self.cast(x, mstype.float16) hidden_gelu = self.mapping_0(x) hidden_linear = self.mapping_1(x) hidden_states = self.mul(hidden_gelu, hidden_linear) if len(F.shape(hidden_states)) == 3: hidden_states = self.dropout_3d(hidden_states) elif len(F.shape(hidden_states)) == 2: hidden_states = self.dropout(hidden_states) hidden_states = self.projection(hidden_states) return hidden_states class MT5Head(nn.Cell): """MT5 model Head. No parameter share between embedding tale and head liner. """ def __init__(self, in_channels, out_channels, compute_dtype=mstype.float16, parallel_config=default_dpmp_config): super(MT5Head, self).__init__() dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.linear = Linear( in_channels=in_channels, out_channels=out_channels, has_bias=False ) self.linear.shard( strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),), ) self.dtype = compute_dtype self.cast = ops.Cast() def construct(self, state, emb_table): # to fit t5 head usage _ = emb_table # output logits over vocabulary [bs, seq_length, vocab_size] return self.linear(self.cast(state, self.dtype)) class MT5Model(T5Model): """MT5 Model, a variant of T5 model. Main changes are: use gated-gelu feedforward, no share parameters between embedding tale and head liner. here is the paper: Args: config(T5Config) : The network config of mt5. """ def __init__(self, config): super(MT5Model, self).__init__(config) for block in self.tfm_encoder.blocks: block.output = T5FeedFowardGatedGelu( hidden_size=config.hidden_size, dropout_rate=config.hidden_dropout_rate, ffn_hidden_size=config.d_ff, hidden_act=config.hidden_act, parallel_config=default_dpmp_config, ) for block in self.tfm_decoder.blocks: block.output = T5FeedFowardGatedGelu( hidden_size=config.hidden_size, dropout_rate=config.hidden_dropout_rate, ffn_hidden_size=config.d_ff, hidden_act=config.hidden_act, parallel_config=default_dpmp_config, ) # no parameter sharing between embedding and classifier layer self.projection = MT5Head( in_channels=config.hidden_size, out_channels=config.vocab_size, parallel_config=default_dpmp_config, ) @MindFormerRegister.register(MindFormerModuleType.MODELS) class MT5ForConditionalGeneration(BaseModel): """ An MT5 model with the loss added. Args: config(T5Config) : The network config of mt5. """ _support_list = MindFormerBook.get_model_support_list()["mt5"] def __init__(self, config: T5Config): super(MT5ForConditionalGeneration, self).__init__(config) parallel_config = config.parallel_config self.mt5_model = MT5Model(config=config) self.vocab_size = config.vocab_size self.loss = CrossEntropyLoss( parallel_config=OpParallelConfig( data_parallel=parallel_config.data_parallel, model_parallel=parallel_config.model_parallel, ) ) self.cast = ops.Cast() self.shape = ops.Shape() # The value of start and end should get from the tokenizer start_token_id = config.start_token_id eos_token_id = config.eos_token_id self.start_token_id = Tensor(((start_token_id,),), dtype=mstype.int32) # [CLS] self.eos_token_id = Tensor(((eos_token_id,),), dtype=mstype.int32) # [SEP] self.concat = P.Concat(axis=1) self.tile = P.Tile() self.reshape = ops.Reshape() # disable the bias for param in self.trainable_params(): if "bias" in param.name or "beta" in param.name and "relative" not in param.name: param.requires_grad = False self.load_checkpoint(config) def _add_start_to_inputs(self, target_ids): """concat the start id to the decoder inputs""" start_token = self.tile(self.start_token_id, (F.shape(target_ids)[0], 1)) decoder_inputs = self.concat((start_token, target_ids)) return decoder_inputs def _add_eos_to_inputs(self, target_ids): """concat the eos id to the end of the decoder inputs""" eos_token = self.tile(self.eos_token_id, (F.shape(target_ids)[0], 1)) inputs_with_eos = self.concat((target_ids, eos_token)) return inputs_with_eos def encoder_forward(self, source_ids, source_mask): """Execute the encoder forward process""" return self.mt5_model.encoder_forward(source_ids, source_mask) def construct(self, input_ids, attention_mask, labels=None, decoder_input_ids=None, decoder_attention_mask=None, memory_mask=None, encoder_outputs=None): """t5_model network with loss.""" if decoder_attention_mask is None: decoder_attention_mask = F.cast(labels != 0, mstype.float32) if decoder_input_ids is None: decoder_input_ids = labels # replace start token using self.start_token_id decoder_input_ids = self._add_start_to_inputs(decoder_input_ids[:, 1:]) # shape: [bs, seq_len, hidden_size] logits = self.mt5_model( input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, memory_mask, encoder_cache=encoder_outputs, ) total_loss = None if labels is not None: # on training label_ids = ops.Reshape()(decoder_input_ids[:, 1:], (-1,)) label_weights = ops.Reshape()(decoder_attention_mask[:, 1:], (-1,)) logits = logits[:, :-1] # logits keep shape same with label_ids total_loss = self.loss( self.reshape(logits, (-1, self.vocab_size)), label_ids, self.cast(label_weights, mstype.float32), ) # reshape: [bs, seq_len, hidden_size] -> [bs*seq, hidden_size] logits = self.reshape(logits, (-1, self.vocab_size)) if self.training: return total_loss return logits pengcheng.mind-mindformer/mindformers/models/t5/t5.py000066400000000000000000002602641476650141400231660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 model.""" import math import numpy as np import mindspore.ops as ops import mindspore.nn as nn from mindspore import context from mindspore.ops.primitive import constexpr from mindspore.ops import functional as F from mindspore.ops import operations as P import mindspore.numpy from mindspore.context import ParallelMode from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter import mindspore.common.dtype as mstype from mindspore.common.initializer import TruncatedNormal, initializer from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules.layers import Linear, _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.moe import MoE, _check_moe_config from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ default_dpmp_config, \ EmbeddingOpParallelConfig, OpParallelConfig from mindformers.core.loss import CrossEntropyLoss from mindformers.modules import VocabEmbedding from .t5_config import T5Config from ..base_model import BaseModel from ...tools import logger from ...tools.register import MindFormerRegister, MindFormerModuleType from ...mindformer_book import MindFormerBook __all__ = ['T5ForConditionalGeneration'] class LayerNorm(nn.Cell): """ T5 layer norm nn.Cell """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32): super(LayerNorm, self).__init__() if param_init_type not in [mstype.float32, mstype.float16]: raise TypeError("The type of parameter 'param_init_type' should in [float32, float16], " "but got the type : {}.".format(type(param_init_type))) self.gamma = Parameter(initializer('ones', normalized_shape, param_init_type), name="gamma", parallel_optimizer=False) self.mean = P.ReduceMean(keep_dims=True) self.square = P.Square() self.sqrt = P.Sqrt() self.sub1 = P.Sub() self.sub2 = P.Sub() self.add = P.Add() self.eps = eps self.mul = P.Mul() self.add2 = P.Add() self.real_div = P.RealDiv() def construct(self, x): r""" x : batch x seq_length x hidden_size """ variance = self.mean(self.square(x), -1) variance_eps = self.sqrt(self.add(variance, self.eps)) output = self.real_div(x, variance_eps) output = self.mul(output, self.gamma) return output def shard(self, strategy): r""" Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. """ self.mean.shard(strategy) self.square.shard(strategy) self.sqrt.shard(strategy) self.sub1.shard((strategy[0], strategy[0])) self.sub2.shard((strategy[0], strategy[0])) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], (1,))) self.add2.shard((strategy[0], (1,))) self.real_div.shard((strategy[0], strategy[0])) return self class T5FeedFoward(nn.Cell): """ T5 feedfoward cell with relu as hidden act """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(T5FeedFoward, self).__init__() mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = int(parallel_config.data_parallel / ep) if ffn_hidden_size % mp != 0: raise ValueError("For 'T5FeedFoward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'T5FeedFoward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'T5FeedFoward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, has_bias=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, has_bias=False, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.dropout = nn.Dropout(1 - dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d = nn.Dropout(1 - dropout_rate) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d = nn.Dropout(1 - dropout_rate) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """The forward function of FFN""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class RelaPosMatrixGenerator(nn.Cell): """ The relative position index generator. The result of the cell should be feed into the bias embedding table. """ def __init__(self, max_relative_position, log_relative_distance): super(RelaPosMatrixGenerator, self).__init__() self._max_relative_position = max_relative_position self._min_relative_position = -max_relative_position self.tile = P.Tile() self.range_mat = P.Reshape() self.sub = P.Sub() self.expanddims = P.ExpandDims() self.cast = P.Cast() self.log_relative_distance = log_relative_distance def construct(self, relative_position, bidirectional=True, num_buckets=32): """The forward of the bias position""" relative_bucket = 0 if bidirectional: num_buckets = num_buckets // 2 relative_bucket = relative_bucket + (relative_position > 0).astype(mstype.int32) * num_buckets relative_position = P.Abs()(relative_position) else: relative_position = -P.Minimum()(relative_position, P.ZerosLike()(relative_position)) max_exact = num_buckets // 2 is_small = relative_position < max_exact relative_position_if_large = max_exact + (P.Log()(relative_position.astype(mstype.float32) / max_exact) / self.log_relative_distance * (num_buckets - max_exact)) relative_position_if_large = relative_position_if_large.astype(mstype.int32) relative_position_if_large = P.Minimum()(relative_position_if_large, mindspore.numpy.full_like(relative_position_if_large, num_buckets - 1)) relative_bucket += mindspore.numpy.where(is_small, relative_position, relative_position_if_large) return relative_bucket class RelaPosEmbeddingsGenerator(nn.Cell): """The relative position embedding generator.""" def __init__(self, depth, max_relative_position, initializer_range, is_decoder): super(RelaPosEmbeddingsGenerator, self).__init__() self.depth = depth self.vocab_size = max_relative_position self.embeddings_table = Parameter(initializer(TruncatedNormal(initializer_range), [self.vocab_size, self.depth])) self.reshape = P.Reshape() self.one_hot = nn.OneHot(depth=self.vocab_size) self.shape = P.Shape() self.gather = P.Gather() self.matmul = P.BatchMatMul() self.relative_attention_num_buckets = 32 self.relative_attention_max_distance = 128 self.is_decoder = is_decoder num_buckets = self.relative_attention_num_buckets max_exact = self.relative_attention_num_buckets // 2 if not self.is_decoder: max_exact = max_exact // 2 num_buckets //= 2 self.log_relative_distance = math.log(self.relative_attention_max_distance / max_exact) self.relative_position_matrix = RelaPosMatrixGenerator(max_relative_position=max_relative_position, log_relative_distance=self.log_relative_distance) def construct(self, query_length, key_length): """The forward function""" context_position = mindspore.numpy.arange(query_length, dtype=mstype.int32).expand_dims(-1) memory_position = mindspore.numpy.arange(key_length, dtype=mstype.int32).expand_dims(0) relative_position = memory_position - context_position relative_position_bucket = self.relative_position_matrix( relative_position, bidirectional=(not self.is_decoder), num_buckets=self.relative_attention_num_buckets) embeddings = self.gather(self.embeddings_table, relative_position_bucket, 0) embeddings = embeddings.transpose((2, 0, 1)).expand_dims(0) return embeddings class T5MultiHeadAttention(nn.Cell): """ T5 multi head attention """ def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, kv_size=64, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, has_relative_bias=False, is_decoder=False, is_cross_atten=False, parallel_config=default_dpmp_config): super(T5MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size self.has_relative_bias = has_relative_bias if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'T5MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'T5MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) if self.is_parallel_mode and batch_size % parallel_config.data_parallel != 0: raise ValueError("For 'T5MultiHeadAttention', the class variable 'batch_size' must be a multiple of " "'parallel_config.data_parallel', but got the batch_size is {} " "and the parallel_config.data_parallel is {}." .format(batch_size, parallel_config.data_parallel)) self.is_first_iteration = True self.inner_dim = num_heads * kv_size # Output layer self.projection = Linear(in_channels=self.inner_dim, out_channels=hidden_size, transpose_b=False, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = kv_size self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.use_past = use_past self.dropout = nn.Dropout(1 - hidden_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = nn.Dropout(1 - attention_dropout_rate) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, self.inner_dim, has_bias=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type self.is_decoder = is_decoder self.has_relative_bias = has_relative_bias self.is_cross_atten = is_cross_atten self.cross_bias = None if self.has_relative_bias: if not self.is_cross_atten: self.bias_generator = RelaPosEmbeddingsGenerator(depth=num_heads, max_relative_position=32, initializer_range=0.02, is_decoder=self.is_decoder) else: self.cross_bias = Parameter(initializer("zero", [1, self.src_seq_length, self.tgt_seq_length]), name='cross_attention_bias', parallel_optimizer=False) if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, bias=None, key_past=None, value_past=None, batch_valid_length=None): """forward function for attention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) query_tensor, key_tensor, value_tensor, batch_size, ori_shape = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor, attention_mask) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = self.reducesum(F.cast(self.not_equal(self.slice(key_past, (0, 0, 0, 0), (F.shape(key_tensor)[0], 1, 1, self.src_seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) valid_length = F.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention, bias = self._attn(query, key, value, attention_mask, bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present, bias def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor, attention_mask): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor, F.shape(attention_mask)[0], query_shape def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, bias): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) if bias is None and self.has_relative_bias: if not self.is_cross_atten: bias = self.bias_generator(F.shape(score)[-1], F.shape(score)[-1]) else: bias = P.ExpandDims()(self.cross_bias, 0) score = self.add(score, bias) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge, bias class TransformerEncoderLayer(nn.Cell): """ Transformer Encoder Layer """ def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', use_past=False, moe_config=default_moe_config, has_bias=False, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}.".format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.batch_size = batch_size self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.attention = T5MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, kv_size=kv_size, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, is_decoder=False, has_relative_bias=has_bias, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = T5FeedFoward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_heads) self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask, bias, init_reset=True, batch_valid_length=None): """Forward function of the EncoderLayer""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present, bias = self.attention(input_x, input_x, input_x, input_mask, bias, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present, bias def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(nn.Cell): """ The Transformer Decoder Layer """ def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', has_bias=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = T5MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, kv_size=kv_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, is_decoder=True, has_relative_bias=has_bias, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) # Cross attention with the output of encoder as memory tensor self.cross_attention = T5MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, kv_size=kv_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, is_decoder=True, is_cross_atten=True, has_relative_bias=has_bias, param_init_type=param_init_type, parallel_config=parallel_config.dpmp if self.use_moe else parallel_config) self.cross_attention_layernorm = LayerNorm((hidden_size,), eps=layer_norm_epsilon).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = T5FeedFoward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = int(hidden_size / num_heads) self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, self_bias=None, encoder_attention_bias=None, init_reset=True, batch_valid_length=None): """The forward function of the decoder layer""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset key_reset = self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) value_reset = self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present, self_bias = self.attention(input_x, input_x, input_x, decoder_mask, self_bias, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_out, cross_layer_present, encoder_attention_bias = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, encoder_attention_bias, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_out) else: x = self.add(x, cross_attn_out) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step key_update = self.assign(self.key_past, key_present) value_update = self.assign(self.value_past, value_present) # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present, self_bias, encoder_attention_bias def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(int(layers / parallel_config.pipeline_stage), 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(int(layers / parallel_config.gradient_aggregation_group), 1) network.set_comm_fusion(int((layer_id + offset) / dis) + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute: network.recompute() else: if parallel_config.recompute.recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(nn.Cell): """The TransformerEncoder Cell""" def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', layer_norm_epsilon=1e-6, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, kv_size=kv_size, num_heads=num_heads, hidden_act=hidden_act, has_bias=(i == 0), layer_norm_epsilon=layer_norm_epsilon, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """The forward process of the encoder""" present_layer = () attention_bias = None if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present, attention_bias = self.blocks[i](hidden_states, attention_mask, attention_bias, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(nn.Cell): """The TransformerDecoder cell""" def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, kv_size=64, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, layer_norm_epsilon=1e-6, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, kv_size=kv_size, use_past=use_past, has_bias=(i == 0), param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, layer_norm_epsilon=layer_norm_epsilon, moe_config=moe_config, parallel_config=parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): pass elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """For forward process of the decoder""" present_layer = () self_bias = None encoder_decoder_bias = None if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present, self_bias, encoder_decoder_bias = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, self_bias, encoder_decoder_bias, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer def position_encoding(length, depth, min_timescale=1, max_timescale=1e4): """ Create Tensor of sinusoids of different frequencies. Args: length (int): Length of the Tensor to create, i.e. Number of steps. depth (int): Hidden size. min_timescale (float): Default: 1. max_timescale (float): Default: 10000. Returns: Tensor of shape (length, depth) """ depth = depth // 2 positions = np.arange(length, dtype=np.float32) log_timescale_increment = (np.log(max_timescale / min_timescale) / (depth - 1)) inv_timescales = min_timescale * np.exp(np.arange(depth, dtype=np.float32) * -log_timescale_increment) scaled_time = np.expand_dims(positions, 1) * np.expand_dims(inv_timescales, 0) x = np.concatenate([np.sin(scaled_time), np.cos(scaled_time)], axis=1) return x class EmbeddingPostprocessor(nn.Cell): """ Postprocessors apply positional embeddings to word embeddings. Args: embedding_size (int): The size of each embedding vector. initializer_range (float): Initialization value of TruncatedNormal. Default: 0.02. max_position_embeddings (int): Maximum length of sequences used in this model. Default: 128. dropout_prob (float): The dropout probability. Default: 0.1. """ def __init__(self, embedding_size, max_position_embeddings=128, dropout_prob=0.1): super(EmbeddingPostprocessor, self).__init__() self.scores_mul = Tensor([math.sqrt(float(embedding_size))], dtype=mstype.float32) self.multiply = ops.Mul() self.add = ops.Add() self.dropout = nn.Dropout(1 - dropout_prob, dtype=mstype.float32) self.expand_dims = ops.ExpandDims() self.position_embedding_table = Tensor(position_encoding(max_position_embeddings, embedding_size), mstype.float32) self.shape = ops.Shape() self.slice = ops.StridedSlice().shard(((1, 1),)) def construct(self, word_embeddings): """Postprocessors apply positional embeddings to word embeddings.""" output = self.multiply(word_embeddings, self.scores_mul) output = self.dropout(output) return output class CastWrapper(nn.Cell): """ Cast wrapper. """ def __init__(self, dst_type=mstype.float32): super(CastWrapper, self).__init__() self.cast = ops.Cast() self.dst_type = dst_type def construct(self, x): return self.cast(x, self.dst_type) class CreateAttentionMaskFromInputMask(nn.Cell): """ Create attention mask according to input mask. Args: config (:class:`TransformerConfig`): Configuration for Transformer. """ def __init__(self, parallel_config): super(CreateAttentionMaskFromInputMask, self).__init__() self.reshape = ops.Reshape() self.shape = ops.Shape() self.batch_matmul = ops.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) def construct(self, input_mask): """Create attention mask according to input mask.""" input_shape = self.shape(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) input_mask = F.cast(input_mask, mstype.float32) mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.batch_matmul(mask_left, mask_right) return attention_mask @constexpr def convert_np_to_tensor_encoder(seq_length): ones = np.ones(shape=(seq_length, seq_length)) return Tensor(np.tril(ones), dtype=mstype.float32) class T5Head(nn.Cell): """ Head to get the logits of each token in the vocab Args: config(): the config of network Inputs: state: the output of the backbone embedding_table: the embedding table of the vocabulary Returns: logits: Tensor, the logits of the corresponding inputs """ def __init__(self, hidden_size, compute_dtype=mstype.float16, parallel_config=None): super(T5Head, self).__init__() if parallel_config.vocab_emb_dp: self.matmul = ops.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), (1, 1))) else: self.matmul = ops.MatMul(transpose_b=True).shard(((parallel_config.data_parallel, 1), ( parallel_config.model_parallel, 1))) self.hidden_size = hidden_size self.dtype = compute_dtype self.cast = ops.Cast() def construct(self, state, embed): state = ops.Reshape()(state, (-1, self.hidden_size)) # output logits over vocabulary [bs*seq_length, vocab_size] logits = self.matmul(self.cast(state, self.dtype), self.cast(embed, self.dtype)) return logits class T5Model(BaseModel): """ T5Model with encoder and decoder. Args: config (Class): Configuration for T5Model. """ def __init__(self, config): super(T5Model, self).__init__(config) self.batch_size = config.batch_size self.hidden_size = config.hidden_size self.max_decode_length = config.max_decode_length self.scale_output = config.scale_output embedding_config = EmbeddingOpParallelConfig(data_parallel=config.parallel_config.data_parallel, model_parallel=config.parallel_config.model_parallel) self.tfm_embedding_lookup = VocabEmbedding(vocab_size=config.vocab_size, embedding_size=config.hidden_size, parallel_config=embedding_config) self.tfm_embedding_postprocessor_for_encoder = EmbeddingPostprocessor(embedding_size= config.hidden_size, max_position_embeddings= config.max_position_embeddings, dropout_prob= config.embedding_dropout_prob) self.tfm_embedding_postprocessor_for_decoder = EmbeddingPostprocessor( embedding_size=config.hidden_size, max_position_embeddings=config.max_position_embeddings, dropout_prob=config.embedding_dropout_prob) self.tfm_encoder = TransformerEncoder( batch_size=self.batch_size, hidden_size=config.hidden_size, num_heads=config.num_heads, num_layers=config.num_layers, seq_length=config.seq_length, ffn_hidden_size=config.d_ff, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, layer_norm_epsilon=config.layer_norm_epsilon, kv_size=config.kv_size, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, post_layernorm_residual=config.post_layernorm_residual, offset=config.offset, use_past=config.use_past, moe_config=config.moe_config) self.tfm_decoder = TransformerDecoder( batch_size=self.batch_size, hidden_size=config.hidden_size, src_seq_length=config.seq_length, tgt_seq_length=config.max_decode_length, num_heads=config.num_heads, ffn_hidden_size=config.d_ff, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=config.hidden_dropout_rate, layer_norm_epsilon=config.layer_norm_epsilon, kv_size=config.kv_size, num_layers=config.num_decoder_layers if config.num_decoder_layers else config.num_layers, hidden_act=config.hidden_act, param_init_type=config.param_init_type, layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, post_layernorm_residual=config.post_layernorm_residual, offset=config.offset, use_past=config.use_past, moe_config=config.moe_config) self.projection = T5Head(config.hidden_size, compute_dtype=mstype.float16, parallel_config=config.parallel_config) self.cast = ops.Cast() self.dtype = config.dtype self.cast_compute_type = CastWrapper(dst_type=config.compute_dtype) self.expand = ops.ExpandDims() self.multiply = ops.Mul() self.shape = ops.Shape() self.encoder_layernorm = LayerNorm(normalized_shape=(config.hidden_size,), eps=config.layer_norm_epsilon).to_float(mstype.float32) self.decoder_layernorm = LayerNorm(normalized_shape=(config.hidden_size,), eps=config.layer_norm_epsilon).to_float(mstype.float32) self.encoder_layernorm.shard(((config.parallel_config.data_parallel, 1),)) self.decoder_layernorm.shard(((config.parallel_config.data_parallel, 1),)) self._create_attention_mask_from_input_mask = CreateAttentionMaskFromInputMask(config.parallel_config) self.ones_like = P.OnesLike() def construct(self, source_ids=None, source_mask=None, target_ids=None, target_mask=None, memory_mask=None, encoder_cache=None): """T5Model with encoder and decoder.""" if source_mask is None and source_ids is not None: source_mask = self.ones_like(source_ids) source_mask = self._create_attention_mask_from_input_mask(source_mask) if source_ids is not None: encoder_output = self.encoder_forward(source_ids, source_mask) else: encoder_output = encoder_cache if target_ids is None: return encoder_output # process target sentence tgt_embedding_output, embedding_table = self.tfm_embedding_lookup(target_ids) # attention mask [batch_size, seq_length, seq_length] tgt_length = self.shape(target_ids)[1] if memory_mask is None: memory_mask = self.create_memory_mask(source_mask, target_mask) if len(ops.shape(target_mask)) == 2: future_mask = convert_np_to_tensor_encoder(tgt_length) tgt_attention_mask = self._create_attention_mask_from_input_mask(target_mask) tgt_attention_mask = self.multiply(tgt_attention_mask, self.expand(future_mask, 0)) else: tgt_attention_mask = target_mask # transformer decoder decoder_output, _ = self.tfm_decoder(self.cast_compute_type(tgt_embedding_output), self.cast_compute_type(tgt_attention_mask), encoder_output, memory_mask) decoder_output = self.decoder_layernorm(decoder_output) if self.scale_output: decoder_output = decoder_output * (self.hidden_size ** -0.5) # calculate logits and log_probs log_probs = self.projection(decoder_output, embedding_table) return log_probs def encoder_forward(self, source_ids, source_mask): """Execute the forward process""" # process source sentence src_embedding_output, _ = self.tfm_embedding_lookup(source_ids) # attention mask [batch_size, seq_length, seq_length] if len(F.shape(source_mask)) == 2: enc_attention_mask = self._create_attention_mask_from_input_mask(source_mask) else: enc_attention_mask = source_mask # transformer encoder encoder_output, _ = self.tfm_encoder(self.cast_compute_type(src_embedding_output), self.cast_compute_type(enc_attention_mask)) encoder_output = self.encoder_layernorm(encoder_output) return encoder_output def create_memory_mask(self, source_mask, target_mask): memory_mask = P.Ones()((F.shape(source_mask)[0], F.shape(target_mask)[-1], F.shape(source_mask)[-1]), mstype.float32) memory_mask = memory_mask * F.expand_dims(source_mask, 1) memory_mask = memory_mask * F.expand_dims(target_mask, 2) return memory_mask @MindFormerRegister.register(MindFormerModuleType.MODELS) class T5ForConditionalGeneration(BaseModel): """ A T5 model with the loss added. Args: config(T5Config) : The network of the transformer. Examples: >>> from mindformers import T5ForConditionalGeneration, T5Tokenizer >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> tokenizer = T5Tokenizer.from_pretrained('t5_small') >>> src_output = tokenizer(["hello world"], padding='max_length', max_length=model.config.seq_length, ... return_tensors='ms') >>> model_input = tokenizer(["So happy to see you!"], padding='max_length', ... max_length=model.config.max_decode_length, ... return_tensors='ms')["input_ids"] >>> input_ids = src_output['input_ids'] >>> attention_mask = src_output['attention_mask'] >>> output = model(input_ids, attention_mask, model_input) >>> print(output) [5.64458] """ _support_list = MindFormerBook.get_model_support_list()['t5'] def __init__(self, config: T5Config): super(T5ForConditionalGeneration, self).__init__(config) parallel_config = config.parallel_config self.t5_model = T5Model(config=config) self.loss = CrossEntropyLoss(parallel_config=OpParallelConfig(data_parallel=parallel_config.data_parallel, model_parallel=parallel_config.model_parallel)) self.cast = ops.Cast() self.shape = ops.Shape() # The value of start and end should get from the tokenizer self.start_token = Tensor(np.zeros((1, 1)).astype(np.int32)) self.eod_token = Tensor(np.zeros((1, 1)).astype(np.int32)) self.concat = P.Concat(axis=1) self.tile = P.Tile() # disable the bias for param in self.trainable_params(): if ('bias' in param.name or 'beta' in param.name) and 'relative' not in param.name: param.requires_grad = False self.set_train(True) self.load_checkpoint(config) def _add_start_to_inputs(self, target_ids): """concat the start id to the decoder inputs""" start_token = self.tile(self.start_token, (F.shape(target_ids)[0], 1)) decoder_inputs = self.concat((start_token, target_ids)) return decoder_inputs def _add_eos_to_inputs(self, target_ids): """concat the eos id to the end of the decoder inputs""" eod_token = self.tile(self.eod_token, (F.shape(target_ids)[0], 1)) inputs_with_eos = self.concat((target_ids, eod_token)) return inputs_with_eos def encoder_forward(self, source_ids, source_mask): """Execute the encoder forward process""" return self.t5_model.encoder_forward(source_ids, source_mask) def construct(self, input_ids, attention_mask, labels=None, decoder_input_ids=None, decoder_attention_mask=None, memory_mask=None, encoder_outputs=None, return_loss=False): """t5_model network with loss.""" if decoder_attention_mask is None: decoder_attention_mask = F.cast(labels != 0, mstype.float32) if decoder_input_ids is None: decoder_input_ids = self._add_start_to_inputs(labels[:, :-1]) logits = self.t5_model(input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, memory_mask, encoder_cache=encoder_outputs) total_loss = None if labels is not None: label_ids = ops.Reshape()(labels, (-1,)) label_weights = ops.Reshape()(decoder_attention_mask, (-1,)) total_loss = self.loss(logits, label_ids, self.cast(label_weights, mstype.float32)) if self.training: return total_loss if return_loss: return total_loss return logits pengcheng.mind-mindformer/mindformers/models/t5/t5_config.py000066400000000000000000000124141476650141400245030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Configuration""" from mindformers.modules.transformer.moe import MoEConfig from mindformers.modules.transformer.transformer import default_transformer_config, default_moe_config, \ TransformerOpParallelConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from ..utils import convert_mstype from ..base_config import BaseConfig from ...mindformer_book import MindFormerBook __all__ = ['T5Config'] @MindFormerRegister.register(MindFormerModuleType.CONFIG) class T5Config(BaseConfig): """ T5 config class which defines the model size """ _support_list = MindFormerBook.get_config_support_list()['t5'] def __init__(self, vocab_size: int = 32128, hidden_size: int = 512, d_kv: int = 64, d_ff: int = 2048, num_layers: int = 6, num_decoder_layers: int = None, num_heads: int = 8, relative_attention_num_buckets: int = 32, hidden_dropout_rate: float = 0.1, attention_dropout_rate: float = 0.1, embedding_dropout_prob: float = 0.1, layer_norm_epsilon: float = 1e-6, initializer_factor: float = 1.0, is_encoder_decoder: bool = True, use_cache: bool = True, pad_token_id: int = 0, start_token_id: int = 0, eos_token_id: int = 1, batch_size: int = 1, seq_length: int = 1024, max_position_embeddings: int = 1024, initializer_range: float = 0.02, max_decode_length: int = 128, length_penalty_weight: float = 1.0, compute_dtype: str = "float32", has_relative_bias: bool = True, scale_output: bool = True, parallel_config: TransformerOpParallelConfig = default_transformer_config, checkpoint_name_or_path: str = None, top_p: float = 0.95, top_k: int = 1, repetition_penalty: float = 1.0, max_length: int = 20, do_sample: bool = False, param_init_type: str = "float32", layernorm_compute_type: str = "float32", softmax_compute_type: str = "float32", hidden_act: str = 'relu', post_layernorm_residual: bool = False, offset: int = 0, use_past: bool = False, moe_config: MoEConfig = default_moe_config, **kwargs): super(T5Config, self).__init__(**kwargs) self.batch_size = batch_size self.seq_length = seq_length self.vocab_size = vocab_size self.hidden_size = hidden_size self.num_layers = num_layers self.num_heads = num_heads self.d_ff = d_ff self.hidden_act = hidden_act self.kv_size = d_kv self.hidden_dropout_rate = hidden_dropout_rate self.attention_dropout_rate = attention_dropout_rate self.embedding_dropout_prob = embedding_dropout_prob self.initializer_factor = initializer_factor self.max_position_embeddings = max_position_embeddings self.initializer_range = initializer_range self.max_decode_length = max_decode_length self.length_penalty_weight = length_penalty_weight self.has_relative_bias = has_relative_bias self.scale_output = scale_output self.parallel_config = parallel_config self.num_decoder_layers = num_decoder_layers self.relative_attention_num_buckets = relative_attention_num_buckets self.layer_norm_epsilon = layer_norm_epsilon self.use_cache = use_cache self.checkpoint_name_or_path = checkpoint_name_or_path self.pad_token_id = pad_token_id self.top_p = top_p self.top_k = top_k self.repetition_penalty = repetition_penalty self.max_length = max_length self.start_token_id = start_token_id self.eos_token_id = eos_token_id self.is_encoder_decoder = is_encoder_decoder self.do_sample = do_sample self.layernorm_compute_type = convert_mstype(layernorm_compute_type) self.softmax_compute_type = convert_mstype(softmax_compute_type) self.compute_dtype = convert_mstype(compute_dtype) self.use_past = use_past self.post_layernorm_residual = post_layernorm_residual self.offset = offset self.moe_config = moe_config self.param_init_type = convert_mstype(param_init_type) pengcheng.mind-mindformer/mindformers/models/t5/t5_processor.py000066400000000000000000000055511476650141400252610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ T5Processor """ from mindformers.models.base_tokenizer import Tokenizer from mindformers.mindformer_book import MindFormerBook from ..base_processor import BaseProcessor from ...tools.register import MindFormerRegister, MindFormerModuleType __all__ = ['T5Processor'] @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class T5Processor(BaseProcessor): """ T5 processor, consists of a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['t5'] def __init__(self, tokenizer=None, max_length=77, tgt_max_length=128, padding='max_length', return_tensors='ms'): super(T5Processor, self).__init__( tokenizer=tokenizer, max_length=max_length, padding=padding, return_tensors=return_tensors ) self.tgt_max_length = tgt_max_length def __call__(self, text_input=None, text_pair=None): """call function""" output = {} if not self.tokenizer: raise ValueError(f"For {self.__name__}, the `tokenizer` should not be None.") if not isinstance(self.tokenizer, Tokenizer): raise TypeError(f"tokenizer should inherited from the BaseTokenizer," f" but got {type(self.tokenizer)}.") if text_input: # Format the input into a batch if isinstance(text_input, str): text_input = [text_input] text_output = self.tokenizer(text_input, return_tensors=self.return_tensors, max_length=self.max_length, padding=self.padding)["input_ids"] output['text'] = text_output if text_pair: # Format the input into a batch if isinstance(text_pair, str): text_input = [text_pair] text_output = self.tokenizer(text_pair, return_tensors=self.return_tensors, max_length=self.tgt_max_length, padding=self.padding)["input_ids"] output['tgt_output'] = text_output return output pengcheng.mind-mindformer/mindformers/models/t5/t5_tokenizer.py000066400000000000000000000376331476650141400252620ustar00rootroot00000000000000# coding=utf-8 # Copyright 2018 T5 Authors and HuggingFace Inc. team. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """T5 Tokenzier""" import os import re import warnings from shutil import copyfile from typing import Any, Dict, List, Optional import jieba import sentencepiece as spm from mindformers.tools import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_tokenizer import Tokenizer from mindformers.models.bert import BertTokenizer from ...mindformer_book import MindFormerBook __all__ = ["T5Tokenizer", "T5PegasusTokenizer"] VOCAB_FILES_NAMES = {"vocab_file": "spiece.model"} @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class T5Tokenizer(Tokenizer): """ Construct a T5 tokenizer. Based on [SentencePiece](https://github.com/google/sentencepiece). This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): [SentencePiece](https://github.com/google/sentencepiece) file (generally has a *.spm* extension) that contains the vocabulary necessary to instantiate a tokenizer. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. extra_ids (`int`, *optional*, defaults to 100): Add a number of extra ids added to the vocabulary for use as sentinels. These tokens are accessible as "" where "{%d}" is a number between 0 and extra_ids-1. These tokens can be retrieved by calling get_sentinel_tokens method and token ids can be by calling get_sentinel_token_ids method additional_special_tokens (`List[str]`, *optional*): Additional special tokens used by the tokenizer. sp_model_kwargs (`dict`, *optional*): Will be passed to the `SentencePieceProcessor.__init__()` method. The [Python wrapper for SentencePiece](https://github.com/google/sentencepiece/tree/master/python) can be used, among other things, to set: - `enable_sampling`: Enable subword regularization. - `nbest_size`: Sampling parameters for unigram. Invalid for BPE-Dropout. - `nbest_size = {0,1}`: No sampling is performed. - `nbest_size > 1`: samples from the nbest_size results. - `nbest_size < 0`: assuming that nbest_size is infinite and samples from the all hypothesis (lattice) using forward-filtering-and-backward-sampling algorithm. - `alpha`: Smoothing parameter for unigram sampling, and dropout probability of merge operations for BPE-dropout. Attributes: sp_model (`SentencePieceProcessor`): The *SentencePiece* processor that is used for every conversion (string, tokens and IDs). """ vocab_files_names = VOCAB_FILES_NAMES FILE_LIST = ['tokenizer_config.json'] model_input_names = ["input_ids", "attention_mask"] _support_list = MindFormerBook.get_tokenizer_support_list()['t5'] def __init__( self, vocab_file, eos_token="", unk_token="", pad_token="", extra_ids=100, additional_special_tokens=None, sp_model_kwargs: Optional[Dict[str, Any]] = None, **kwargs, ) -> None: # Add extra_ids to the special token list if extra_ids > 0 and additional_special_tokens is None: additional_special_tokens = [f"" for i in range(extra_ids)] elif extra_ids > 0 and additional_special_tokens is not None: # Check that we have the right number of extra_id special tokens extra_tokens = len(set(filter(lambda x: bool("extra_id" in str(x)), additional_special_tokens))) if extra_tokens != extra_ids: raise ValueError( f"Both extra_ids ({extra_ids}) and additional_special_tokens ({additional_special_tokens}) are" " provided to T5Tokenizer. In this case the additional_special_tokens must include the extra_ids" " tokens" ) self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs super().__init__( eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, extra_ids=extra_ids, additional_special_tokens=additional_special_tokens, sp_model_kwargs=self.sp_model_kwargs, **kwargs, ) self.vocab_file = vocab_file self._extra_ids = extra_ids self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) @staticmethod def _eventually_correct_t5_max_length(pretrained_model_name_or_path, max_model_length, init_max_model_length): """eventually_correct_t5_max_length""" if pretrained_model_name_or_path in T5Tokenizer.max_model_input_sizes: deprecated_max_model_length = T5Tokenizer.max_model_input_sizes[pretrained_model_name_or_path] if init_max_model_length is not None and init_max_model_length != max_model_length: return init_max_model_length if init_max_model_length is None: warnings.warn( "This tokenizer was incorrectly instantiated with a model max length of" f" {deprecated_max_model_length} which will be corrected in Transformers v5.\nFor now, this" " behavior is kept to avoid breaking backwards compatibility when padding/encoding with" " `truncation is True`.\n- Be aware that you SHOULD NOT rely on" f" {pretrained_model_name_or_path} automatically truncating your input to" f" {deprecated_max_model_length} when padding/encoding.\n- If you want to encode/pad to sequences" f" longer than {deprecated_max_model_length} you can either instantiate this tokenizer with" " `model_max_length` or pass `max_length` when encoding/padding.\n- To avoid this warning, please" " instantiate this tokenizer with `model_max_length` set to your preferred value.", FutureWarning, ) return max_model_length @property def vocab_size(self): return self.sp_model.get_piece_size() + self._extra_ids def get_vocab(self): vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) # normal case: some special tokens if token_ids_1 is None: return ([0] * len(token_ids_0)) + [1] return ([0] * len(token_ids_0)) + [1] + ([0] * len(token_ids_1)) + [1] def get_sentinel_tokens(self): return list( set(filter(lambda x: bool(re.search(r"", x)) is not None, self.additional_special_tokens)) ) def get_sentinel_token_ids(self): return [self._convert_token_to_id(token) for token in self.get_sentinel_tokens()] def _add_eos_if_not_present(self, token_ids: List[int]) -> List[int]: """Do not add eos again if user already added it.""" if token_ids and token_ids[-1] == self.eos_token_id: warnings.warn( f"This sequence already has {self.eos_token}. In future versions this behavior may lead to duplicated" " eos tokens being added." ) return token_ids return token_ids + [self.eos_token_id] def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Create a mask from the two sequences passed to be used in a sequence-pair classification task. T5 does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of zeros. """ eos = [self.eos_token_id] if token_ids_1 is None: return len(token_ids_0 + eos) * [0] return len(token_ids_0 + eos + token_ids_1 + eos) * [0] def build_inputs_with_special_tokens( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A sequence has the following format: - single sequence: `X ` - pair of sequences: `A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. """ token_ids_0 = self._add_eos_if_not_present(token_ids_0) if token_ids_1 is None: return token_ids_0 token_ids_1 = self._add_eos_if_not_present(token_ids_1) return token_ids_0 + token_ids_1 def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d # for backward compatibility if not hasattr(self, "sp_model_kwargs"): self.sp_model_kwargs = {} self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) def _tokenize(self, text: str, **kwargs) -> List[str]: """Take as input a string and return a list of strings (tokens) for words/sub-words""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" if token.startswith("", token) num = int(match.group(1)) return self.vocab_size - num - 1 return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" if index < self.sp_model.get_piece_size(): token = self.sp_model.IdToPiece(index) else: token = f"" return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for token in tokens: # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string.strip() def save_vocabulary(self, save_directory, filename_prefix=None): if not os.path.isdir(save_directory): logger.error("Vocabulary path (%s) should be a directory", save_directory) return None out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return out_vocab_file @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class T5PegasusTokenizer(BertTokenizer): """Tokenizer for T5 Pegasus, which is suitable for Chinese dataset. Based on word granularity, if the word does not appear in the vocabulary file, then call BERT native Tokenizer. Args: vocab_file(str): Path to vocab file. pre_tokenizer(function): Default using jieba to cut words """ def __init__(self, vocab_file, *args, pre_tokenizer=lambda x: jieba.cut(x, HMM=False), **kwargs): super(T5PegasusTokenizer, self).__init__(vocab_file, *args, **kwargs) self.pre_tokenizer = pre_tokenizer def _tokenize(self, text, **kwargs): split_tokens = [] for word in self.pre_tokenizer(text): if word in self.vocab_dict: split_tokens.append(word) else: split_tokens.extend(super()._tokenize(word)) return split_tokens def convert_tokens_to_string(self, tokens: list): """Convert the tokens to the string""" return " ".join(tokens) def _decode(self, token_ids, skip_special_tokens=False, clean_up_tokenization_spaces=None, **kwargs): ids = self.convert_ids_to_tokens( token_ids, skip_special_tokens=skip_special_tokens ) return self.convert_tokens_to_string(ids) pengcheng.mind-mindformer/mindformers/models/utils.py000066400000000000000000000023241476650141400234350ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Check Model Input Config.""" import mindspore.common.dtype as mstype from ..version_control import get_cell_reuse def convert_mstype(ms_type: str = "float16"): """Convert the string type to MindSpore type.""" if isinstance(ms_type, mstype.Float): return ms_type if ms_type == "float16": return mstype.float16 if ms_type == "float32": return mstype.float32 raise KeyError(f"Supported data type keywords include: " f"[float16, float32], but get {ms_type}") def cell_reuse(): return get_cell_reuse() pengcheng.mind-mindformer/mindformers/models/vit/000077500000000000000000000000001476650141400225245ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/models/vit/__init__.py000066400000000000000000000016641476650141400246440ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Vit Model API.""" from .vit import ViTForImageClassification, ViTModel from .vit_config import ViTConfig from .vit_processor import ViTProcessor, ViTImageProcessor __all__ = ['ViTForImageClassification', 'ViTModel', 'ViTConfig', 'ViTProcessor', 'ViTImageProcessor'] pengcheng.mind-mindformer/mindformers/models/vit/convert_weight.py000066400000000000000000000064441476650141400261350ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Convert checkpoint from torch/facebook""" import argparse import torch import mindspore as ms def convert_weight(torch_pth_path="mae_pretrain_vit_base.pth", ms_ckpt_path="mae_pretrain_vit_base.ckpt"): """ convert mae_vit_base_p16 weights from pytorch to mindspore pytorch and GPU required. """ param_dict = torch.load(torch_pth_path, map_location=torch.device("cpu")) new_dict = [] for k, v in param_dict["model"].items(): if k in ("cls_token", "mask_token"): k += "s" if "head" not in k: k = "vit." + k if "norm" in k: if "fc_norm" not in k: k = k.replace("norm", "layernorm") if "weight" in k: k = k.replace("weight", "gamma") elif "bias" in k: k = k.replace("bias", "beta") if "mlp" in k: k = k.replace("mlp", "output") if "fc1" in k: k = k.replace("fc1", "mapping") if "weight" in k: v = v.transpose(-1, 0) elif "fc2" in k: k = k.replace("fc2", "projection") if "weight" in k: v = v.transpose(-1, 0) if "attn" in k: k = k.replace("attn", "attention") if "proj" in k: k = k.replace("proj", "projection") if "weight" in k: v = v.transpose(-1, 0) if "qkv" not in k: new_dict.append({"name": k, "data": ms.Tensor(v.numpy())}) else: data = ms.Tensor(v.numpy()) length = data.shape[0] // 3 new_dict.append({"name": k.replace(".qkv", ".dense1"), "data": data[:length]}) new_dict.append({"name": k.replace(".qkv", ".dense2"), "data": data[length:length*2]}) new_dict.append({"name": k.replace(".qkv", ".dense3"), "data": data[length*2:]}) ms.save_checkpoint(new_dict, ms_ckpt_path) print("Weights conversion completes. ") if __name__ == "__main__": parser = argparse.ArgumentParser(description="mae vit weight convert script") parser.add_argument("--torch_pth_path", type=str, default="mae_pretrain_vit_base.pth", required=True, help="The torch checkpoint path.") parser.add_argument("--ms_ckpt_path", type=str, required=True, default="mae_pretrain_vit_base.ckpt", help="The output mindspore checkpoint path.") opt = parser.parse_args() convert_weight(opt.torch_pth_path, opt.ms_ckpt_path) pengcheng.mind-mindformer/mindformers/models/vit/vit.py000066400000000000000000000316741476650141400237130ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/facebookresearch/mae # ============================================================================ """ViT Model.""" import math import numpy as np from mindspore import load_param_into_net, Parameter, nn from mindspore import ops as P from mindspore import dtype as mstype import mindspore.common.initializer as weight_init from mindformers.mindformer_book import MindFormerBook from mindformers.core.loss import build_loss from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.base_model import BaseModel from mindformers.models.vit.vit_modules import Block, LayerNorm, Linear, Dropout, PixelShuffle from mindformers.models.vit.vit_modules import PatchEmbed from mindformers.models.vit.vit_config import ViTConfig @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTModel(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTModel.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTModel(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.use_moe = (config.moe_config.expert_num > 1) parallel_config = config.parallel_config dp = parallel_config.data_parallel self.global_pool = config.use_mean_pooling self.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, parallel_config=parallel_config) self.cls_tokens = Parameter( weight_init.initializer(weight_init.TruncatedNormal(sigma=config.initializer_range), (1, 1, config.embed_dim)), requires_grad=True) num_patches = self.patch_embed.num_patches seq_length = num_patches + 1 self.seq_length = seq_length self.num_patches = num_patches self.num_masked = num_patches - seq_length + 1 self.pos_embed = Parameter( weight_init.initializer(weight_init.TruncatedNormal(sigma=config.initializer_range), (1, seq_length, config.embed_dim)), requires_grad=True) # stochastic depth decay rule hdr = [x.item() for x in np.linspace(0, config.drop_path_rate, config.depth)] parallel_config_args = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config self.blocks = nn.CellList([ Block(hidden_size=config.embed_dim, ffn_hidden_size=config.intermediate_size, seq_length=seq_length, drop_rate=config.drop_rate, attention_dropout_rate=config.attention_dropout_rate, hidden_dropout_rate=hdr[i], layer_norm_eps=config.layer_norm_eps, qkv_bias=config.qkv_bias, init_values=config.init_values, weight_init='XavierUniform', layernorm_compute_type=config.layernorm_compute_type, softmax_compute_type=config.softmax_compute_type, window_size=None, num_heads=config.num_heads, hidden_act=config.hidden_act, post_layernorm_residual=config.post_layernorm_residual, param_init_type=config.param_init_type, parallel_config=parallel_config_args) for i in range(config.depth)]) self.add = P.Add().shard(((dp, 1, 1), (1, 1, 1))) self.cast = P.Cast() self.tile = P.Tile().shard(((dp, 1, 1),)) self.cat = P.Concat(axis=1) self.fc_norm = LayerNorm((config.embed_dim,), eps=1e-6).shard(((dp, 1, 1),)) self.reduce_mean = P.ReduceMean().shard(((dp, 1, 1),)) self.dropout = Dropout(keep_prob=(1. - config.drop_rate)) self.dropout.shard(((dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((dp, 1, 1),)) self.init_weights_vit() self.fix_init_weight() def fix_init_weight(self): """fix init weight""" def rescale(param, layer_id): values = param.data / (math.sqrt(2.0 * layer_id)) param.set_data(values) for layer_id, block in enumerate(self.blocks): if self.use_moe: rescale(block.attention.projection.weight, layer_id + 1) rescale(block.output.ffn.projection.weight, layer_id + 1) else: rescale(block.attention.projection.weight, layer_id + 1) rescale(block.output.projection.weight, layer_id + 1) def init_weights_vit(self): """init weights vit ViT weight initialization, original timm impl (for reproducibility) """ for name, cell in self.cells_and_names(): if isinstance(cell, Linear): cell.weight.set_data(weight_init.initializer( weight_init.TruncatedNormal(sigma=self.config.initializer_range), cell.weight.shape, cell.weight.dtype)) if isinstance(cell, Linear) and cell.bias is not None: cell.bias.set_data(weight_init.initializer(weight_init.Zero(), cell.bias.shape, cell.bias.dtype)) elif isinstance(cell, (LayerNorm, nn.LayerNorm)): cell.gamma.set_data(weight_init.initializer(weight_init.One(), cell.gamma.shape, cell.gamma.dtype)) cell.beta.set_data(weight_init.initializer(weight_init.Zero(), cell.beta.shape, cell.beta.dtype)) if name == "patch_embed.proj": cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def no_weight_decay(self): return {'pos_embed', 'cls_tokens'} def load_pretrained(self, params_dict): return load_param_into_net(self, params_dict) def construct_without_pool(self, image, mask=None): """construct of vit without pool""" tokens = self.patch_embed(image, mask) batch_size = image.shape[0] cls_tokens = self.tile(self.cls_tokens, (batch_size, 1, 1)) tokens = self.cat((cls_tokens, tokens)) if self.pos_embed is not None: tokens = self.add(tokens, self.pos_embed) x = self.dropout(tokens) encoder_input_mask = P.Ones()((batch_size, self.seq_length, self.seq_length), mstype.float32) for block in self.blocks: x = block(x, encoder_input_mask) return x def construct(self, image): """construct of vit""" x = self.construct_without_pool(image) b, s, c = x.shape if self.global_pool: x = self.stride_slice( x, (0, 1, 0), (b, s, c), (1, 1, 1) ) x = self.reduce_mean(x, 1) out = self.fc_norm(x) else: out = self.stride_slice( x, (0, 0, 0), (b, 1, c), (1, 1, 1) ) return out @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTForImageClassification(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTForImageClassification.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTForImageClassification(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.vit = ViTModel(config) self.head = Linear( config.embed_dim, config.num_classes, weight_init=weight_init.TruncatedNormal(sigma=2e-5), compute_dtype=mstype.float32).to_float(mstype.float32) self.loss = build_loss(class_name=config.loss_type) self.load_checkpoint(config) def construct(self, image, target=None): """construct of vit""" out = self.vit(image) out = self.head(out) if not self.training: return out, target loss = self.loss(out, target) return loss @MindFormerRegister.register(MindFormerModuleType.MODELS) class ViTForMaskedImageModeling(BaseModel): """ Vision Transformer with support for patch or hybrid CNN input stage. The supported model name could be selected from ViTConfig.show_support_list(). Args: config (ViTConfig): the config of Vit model. Examples: >>> # input model name, load model and weights >>> model_a = ViTForMaskedImageModeling.from_pretrained('vit_base_p16') >>> # input config, load model without weights >>> from mindformers import AutoConfig >>> config = AutoConfig.from_pretrained('vit_base_p16') >>> model_b = ViTForMaskedImageModeling(config) """ _support_list = MindFormerBook.get_model_support_list()['vit'] def __init__(self, config=None): config = config if config else ViTConfig() super().__init__(config) self.vit = ViTModel(config) self.vit.patch_embed = PatchEmbed(img_size=config.image_size, patch_size=config.patch_size, in_features=config.in_chans, out_features=config.embed_dim, use_mask=True, parallel_config=config.parallel_config) self.decoder = nn.CellList( nn.Conv2d( in_channels=config.embed_dim, out_channels=config.encoder_stride ** 2 * config.in_chans, kernel_size=1, ), PixelShuffle(config.encoder_stride), ) self.transpose = P.Transpose() self.reshape = P.Reshape() self.expand_dims = P.ExpandDims() self.l1_loss = nn.L1Loss(reduction='none') # Initialize weights and apply final processing self.init_weights_vit() def init_weights_vit(self): for _, cell in self.cells_and_names(): if isinstance(cell, nn.Conv2d): cell.weight.set_data(weight_init.initializer(weight_init.XavierUniform(), cell.weight.shape, cell.weight.dtype)) def construct(self, image, mask=None): """construct of vit for MIM""" x = self.vit.construct_without_pool(image) b, s, c = x.shape height = width = math.floor(s ** 0.5) x = self.reshape(self.transpose(x, (0, 2, 1)), (b, c, height, width)) reconstruct_images = self.decoder(x) if not self.training: return reconstruct_images size = self.config.image_size // self.config.patch_size mask = self.reshape(mask, (-1, size, size)) mask = P.repeat_elements(mask, self.config.patch_size, 1) mask = P.repeat_elements(mask, self.config.patch_size, 2) mask = self.expand_dims(mask, 1) reconstruction_loss = self.l1_loss(image, reconstruct_images) masked_im_loss = (reconstruction_loss * mask).sum() / (mask.sum() + 1e-5) / self.config.in_chans return masked_im_loss pengcheng.mind-mindformer/mindformers/models/vit/vit_config.py000066400000000000000000000110041476650141400252210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Mae Config API.""" import mindspore.common.dtype as mstype from mindformers.modules.transformer import TransformerOpParallelConfig, TransformerRecomputeConfig from mindformers.modules.transformer.moe import MoEConfig, default_moe_config from mindformers.mindformer_book import MindFormerBook from mindformers.models.base_config import BaseConfig from mindformers.tools.register import MindFormerRegister, MindFormerModuleType default_recompute_config = TransformerRecomputeConfig() default_parallel_config = TransformerOpParallelConfig(recompute=default_recompute_config) @MindFormerRegister.register(MindFormerModuleType.CONFIG) class ViTConfig(BaseConfig): """ Config for ViT model Examples: >>> # init a config with a model name >>> config_a = ViTConfig.from_pretrained('vit_base_p16') >>> # init a config with a config path >>> import os >>> from mindformers.mindformer_book import MindFormerBook >>> config_path = os.path.join(MindFormerBook.get_project_path(), >>> 'configs', 'vit', 'run_vit_base_p16_224_100ep.yaml') >>> config_b = ViTConfig.from_pretrained(config_path) >>> # init a config with args >>> config_c = ViTConfig( >>> patch_size=16, >>> in_chans=3, >>> ... >>> ) """ _support_list = MindFormerBook.get_config_support_list()['vit'] def __init__(self, image_size: int = 224, patch_size: int = 16, num_channels: int = 3, initializer_range: float = 0.02, hidden_size: int = 768, num_hidden_layers: int = 12, num_attention_heads: int = 12, intermediate_size: int = 3072, qkv_bias: bool = True, hidden_act: str = "gelu", post_layernorm_residual: bool = False, layer_norm_eps: float = 1e-12, attention_probs_dropout_prob: float = 0.0, hidden_dropout_prob: float = 0.0, drop_path_rate: float = 0.1, use_mean_pooling: bool = True, num_labels: int = 1000, loss_type: str = "SoftTargetCrossEntropy", encoder_stride: int = 16, checkpoint_name_or_path: str = '', layernorm_compute_type: mstype = mstype.float32, softmax_compute_type: mstype = mstype.float32, param_init_type: mstype = mstype.float32, parallel_config: TransformerOpParallelConfig = default_parallel_config, moe_config: MoEConfig = default_moe_config, **kwargs): super().__init__(**kwargs) self.image_size = image_size self.patch_size = patch_size self.in_chans = num_channels self.initializer_range = initializer_range self.embed_dim = hidden_size self.depth = num_hidden_layers self.num_heads = num_attention_heads self.intermediate_size = intermediate_size self.qkv_bias = qkv_bias self.hidden_act = hidden_act self.post_layernorm_residual = post_layernorm_residual self.layer_norm_eps = layer_norm_eps self.attention_dropout_rate = attention_probs_dropout_prob self.drop_rate = hidden_dropout_prob self.drop_path_rate = drop_path_rate self.num_classes = num_labels self.use_mean_pooling = use_mean_pooling self.loss_type = loss_type self.encoder_stride = encoder_stride self.checkpoint_name_or_path = checkpoint_name_or_path self.layernorm_compute_type = layernorm_compute_type self.softmax_compute_type = softmax_compute_type self.param_init_type = param_init_type self.parallel_config = parallel_config self.moe_config = moe_config pengcheng.mind-mindformer/mindformers/models/vit/vit_modules.py000066400000000000000000001111521476650141400254310ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Modules of ViTForImageClassification, including Linear, Block, MLP, Attention, PatchEmbed, etc. """ import math import numpy as np from mindspore import nn, Parameter, Tensor, context, ParallelMode import mindspore.common.dtype as mstype from mindspore.ops import operations as P, constexpr from mindspore.ops import functional as F import mindspore.common.initializer as init from mindspore.parallel._utils import _get_parallel_mode from mindformers.modules import layers, FeedForward from mindformers.modules.transformer.moe import default_moe_config, _check_moe_config from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _check_config @constexpr def gen_shape(x_shape, ndim): return (x_shape,) + (1,) * (ndim + 1) def get_2d_sincos_pos_embed(embed_dim, grid_size, cls_token=False): """ grid_size: int of the grid height and width return: pos_embed: [grid_size*grid_size, embed_dim] or [1+grid_size*grid_size, embed_dim] (w/ or w/o cls_token) """ grid_h = np.arange(grid_size, dtype=np.float32) grid_w = np.arange(grid_size, dtype=np.float32) grid = np.meshgrid(grid_w, grid_h) # here w goes first grid = np.stack(grid, axis=0) grid = grid.reshape([2, 1, grid_size, grid_size]) pos_embed = get_2d_sincos_pos_embed_from_grid(embed_dim, grid) if cls_token: pos_embed = np.concatenate([np.zeros([1, embed_dim]), pos_embed], axis=0) return pos_embed def get_2d_sincos_pos_embed_from_grid(embed_dim, grid): assert embed_dim % 2 == 0 # use half of dimensions to encode grid_h emb_h = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[0]) # (H*W, D/2) emb_w = get_1d_sincos_pos_embed_from_grid(embed_dim // 2, grid[1]) # (H*W, D/2) emb = np.concatenate([emb_h, emb_w], axis=1) # (H*W, D) return emb def get_1d_sincos_pos_embed_from_grid(embed_dim, pos): """ embed_dim: output dimension for each position pos: a list of positions to be encoded: size (M,) out: (M, D) """ assert embed_dim % 2 == 0 omega = np.arange(embed_dim // 2, dtype=np.float32) omega /= embed_dim / 2. omega = 1. / 10000 ** omega # (D/2,) pos = pos.reshape(-1) # (M,) out = np.einsum('m,d->md', pos, omega) # (M, D/2), outer product emb_sin = np.sin(out) # (M, D/2) emb_cos = np.cos(out) # (M, D/2) emb = np.concatenate([emb_sin, emb_cos], axis=1) # (M, D) return emb class LayerNorm(layers.LayerNorm): # pylint: disable=W0212 r""" A self-defined layer norm operation using reduce sum and reduce mean. """ def __init__(self, normalized_shape, eps=1e-6, param_init_type=mstype.float32): super(LayerNorm, self).__init__( normalized_shape, eps=eps, param_init_type=param_init_type) class Identity(nn.Cell): def construct(self, x): return x class RelativePositionBias(nn.Cell): """relative position bias""" def __init__(self, window_size, num_heads): super(RelativePositionBias, self).__init__() self.window_size = window_size # cls to token & token to cls & cls to cls self.num_relative_distance = (2 * window_size[0] - 1) * (2 * window_size[1] - 1) + 3 self.relative_position_bias_table = Parameter( init.initializer( init.TruncatedNormal(sigma=.02), (self.num_relative_distance, num_heads)), name='relative_position_bias_table') # get pair-wise relative position index for each token inside the window coords_h = Tensor(np.arange(window_size[0]), mstype.int32) coords_w = Tensor(np.arange(window_size[1]), mstype.int32) coords = P.Stack(axis=0)(P.Meshgrid(indexing='ij')((coords_h, coords_w))) # 2, Wh, Ww coords_flatten = P.Flatten()(coords) # 2, Wh*Ww relative_coords = coords_flatten[:, :, None] - coords_flatten[:, None, :] # 2, Wh*Ww, Wh*Ww relative_coords = P.Transpose()(relative_coords, (1, 2, 0)).asnumpy() # Wh*Ww, Wh*Ww, 2 relative_coords[:, :, 0] += window_size[0] - 1 relative_coords[:, :, 1] += window_size[1] - 1 relative_coords[:, :, 0] *= 2 * window_size[1] - 1 relative_position_index = \ np.zeros(((window_size[0] * window_size[1] + 1),) * 2, dtype=int) relative_position_index[1:, 1:] = np.sum(relative_coords, axis=-1) # Wh*Ww, Wh*Ww relative_position_index[0, 0:] = self.num_relative_distance - 3 relative_position_index[0:, 0] = self.num_relative_distance - 2 relative_position_index[0, 0] = self.num_relative_distance - 1 relative_position_index = Tensor(relative_position_index, mstype.int32) self.relative_position_index = Parameter( relative_position_index, requires_grad=False, name="relative_position_index") self.reshape = P.Reshape() self.transpose = P.Transpose() self.gather = P.Gather() def construct(self): relative_position_index = self.relative_position_index.view(-1) relative_position_bias = self.gather(self.relative_position_bias_table, relative_position_index, 0) relative_position_bias = self.reshape( relative_position_bias, (self.window_size[0] * self.window_size[1] + 1, self.window_size[0] * self.window_size[1] + 1, -1)) relative_position_bias = self.transpose(relative_position_bias, (2, 0, 1)) return relative_position_bias class Linear(layers.Linear): # pylint: disable=W0212 r""" Linear function for RingMo. """ def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, activation_compute_type=mstype.float16, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__( in_channels, out_channels, weight_init=weight_init, bias_init=bias_init, has_bias=has_bias, activation=activation, transpose_b=transpose_b, expert_num=expert_num, outer_batch=outer_batch, param_init_type=param_init_type, compute_dtype=compute_dtype) self.activation_compute_type = activation_compute_type def construct(self, x): """construct of layer""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) weight = self.cast(self.weight, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(self.cast(x, self.activation_compute_type)) x = self.cast(x, self.dtype) output = P.Reshape()(x, out_shape) return output class Dropout(layers.Dropout): # pylint: disable=W0212 r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for context training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__(keep_prob=keep_prob, dtype=dtype) class DropPath(nn.Cell): """Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).""" def __init__(self, drop_prob): super(DropPath, self).__init__() self.drop = Dropout(keep_prob=1 - drop_prob) self.mask = Tensor(np.ones(1,), dtype=mstype.float32) self.tile = P.Tile() self.mul = P.Mul() def construct(self, x): if not self.training: return x mask = self.tile(self.mask, (x.shape[0],) + (1,) * (x.ndim-1)) out = self.drop(mask) out = self.mul(out, x) return out def shard(self, strategy): self.mul.shard(strategy) class Block(nn.Cell): """Block of ringmo""" def __init__(self, hidden_size, ffn_hidden_size, num_heads, seq_length, drop_rate=0., attention_dropout_rate=0., hidden_dropout_rate=0., layer_norm_eps=1e-6, qkv_bias=True, window_size=None, post_layernorm_residual=False, init_values=None, weight_init='XavierUniform', layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(Block, self).__init__() _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.seq_length = seq_length self.hidden_size = hidden_size self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None self.layernorm1 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,), eps=layer_norm_eps).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) parallel_config_args = parallel_config.dpmp if self.use_moe else parallel_config self.attention = Attention(src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, window_size=window_size, num_heads=num_heads, weight_init=weight_init, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, qkv_bias=qkv_bias, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=parallel_config_args) self.output = MLP(hidden_size=hidden_size, dropout_rate=drop_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, weight_init=weight_init, hidden_act=hidden_act, use_dropout=False, parallel_config=parallel_config) if init_values is not None: self.gamma_1 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma1", requires_grad=True) self.gamma_2 = Parameter( Tensor(init_values * np.ones((hidden_size,)), mstype.float32), name="gamma2", requires_grad=True) else: self.gamma_1, self.gamma_2 = None, None self.mul_gamma = P.Mul().shard(((parallel_config.data_parallel, 1), (1,))) self.drop_path = DropPath(hidden_dropout_rate) self.drop_path.shard(((parallel_config.data_parallel, 1),)) self.drop_path3d = DropPath(hidden_dropout_rate) self.drop_path3d.shard(((parallel_config.data_parallel, 1, 1),)) self.mul = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.reshape = P.Reshape() def construct(self, x, input_mask, rel_pos_bias=None): """construct of Block""" x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) attention = self.attention( input_x, input_x, input_x, input_mask, rel_pos_bias) if self.gamma_1 is not None: attention = self.mul_gamma(attention, self.gamma_1) if len(x_shape) == 3: attention = P.Reshape()(attention, x_shape) attention = self.drop_path3d(attention) attention = P.Reshape()(attention, (-1, x_shape[-1])) else: attention = self.drop_path(attention) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) mlp_logit = self.output(output_x) if self.gamma_2 is not None: mlp_logit = self.mul_gamma(mlp_logit, self.gamma_2) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) mlp_logit = self.drop_path3d(mlp_logit) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: mlp_logit = self.drop_path(mlp_logit) if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) return output class MLP(FeedForward): r"""MLP for ring-mo.""" def __init__(self, hidden_size, ffn_hidden_size=None, out_features=None, dropout_rate=0., hidden_act='gelu', weight_init='XavierUniform', use_dropout=True, param_init_type=mstype.float32, parallel_config=default_dpmp_config): ffn_hidden_size = ffn_hidden_size or hidden_size super(MLP, self).__init__( hidden_size, ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, param_init_type=mstype.float32, parallel_config=parallel_config) mp = parallel_config.model_parallel dp = parallel_config.data_parallel out_features = out_features or hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=hidden_size, out_channels=ffn_hidden_size, activation=hidden_act, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=ffn_hidden_size, out_channels=out_features, transpose_b=False, outer_batch=dp, weight_init=weight_init, param_init_type=param_init_type) self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - dropout_rate) self.dropout.shard(((dp, 1),)) self.dropout_3d = Dropout(1 - dropout_rate) self.dropout_3d.shard(((dp, 1, 1),)) self.use_dropout = use_dropout def construct(self, x): """construct of mlp""" x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) if self.use_dropout: if len(F.shape(hidden)) == 3: hidden = self.dropout_3d(hidden) elif len(F.shape(hidden)) == 2: hidden = self.dropout(hidden) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) return output class Attention(nn.Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following """ def __init__(self, src_seq_length, tgt_seq_length, hidden_size, num_heads, window_size=None, hidden_dropout_rate=0., attention_dropout_rate=0., qkv_bias=True, weight_init='XavierUniform', compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(Attention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] _check_config(parallel_config) self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.dropout = Dropout(1 - hidden_dropout_rate) self.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout = Dropout(1 - attention_dropout_rate) self.prob_dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, has_bias=qkv_bias, weight_init=weight_init, param_init_type=param_init_type).to_float(compute_dtype) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) if window_size: self.relative_position_bias = RelativePositionBias(window_size, num_heads) else: self.window_size = None self.relative_position_bias_table = None self.relative_position_index = None self.relative_position_bias = None self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1))) self.expand_dims_rpb = P.ExpandDims().shard(((1, 1, 1),)) self.add_rpb = P.Add().shard(((parallel_config.data_parallel, 1, 1, 1), (1, 1, 1, 1))) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, rel_pos_bias=None): """construct of attention""" ori_shape = F.shape(query_tensor) batch_size = F.shape(attention_mask)[0] query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, -1, self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask, rel_pos_bias) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) return output def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( F.reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = F.reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask, rel_pos_bias): r"""Get the weighted score along the seq_length.""" # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_dtype) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(score)), P.Cast()(attention_mask, P.DType()(score))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, score) # add window cross module if self.relative_position_bias is not None: relative_position_bias = self.expand_dims_rpb(self.relative_position_bias(), 0) attention_scores = self.add_rpb(attention_scores, relative_position_bias) if rel_pos_bias is not None: attention_scores = self.add_3d(attention_scores, rel_pos_bias) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = P.Cast()(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge def _merge_heads(self, x): """ convert a 4d input to a 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor class PatchEmbed(nn.Cell): """Construct the embeddings from patch, position embeddings.""" def __init__(self, img_size=224, patch_size=16, in_features=3, out_features=768, norm_layer=False, use_mask=False, parallel_config=None): super(PatchEmbed, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 img_size = (img_size, img_size) patch_size = (patch_size, patch_size) self.grid_size = (img_size[0] // patch_size[0], img_size[1] // patch_size[1]) self.num_patches = self.grid_size[0] * self.grid_size[1] self.proj = nn.Conv2d( in_channels=in_features, out_channels=out_features, kernel_size=patch_size, stride=patch_size, weight_init=init.TruncatedNormal(sigma=0.02), has_bias=True, pad_mode='pad') self.proj.conv2d.shard(((dp, 1, 1, 1), (1, 1, 1, 1))) self.proj.bias_add.shard(((dp, 1, 1, 1), (1,))) self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1),)) # usually not use norm self.norm = LayerNorm((out_features,), eps=1e-6).shard(((dp, 1, 1),)) if norm_layer else Identity() self.mask_token = Parameter(init.initializer(init.Zero(), (1, 1, out_features)), requires_grad=True) if use_mask else None self.tile = P.Tile() self.mul = P.Mul() self.add = P.Add() def construct(self, x, mask=None): """construct""" # True x: bs False x: bs * dp x = self.proj(x) b, c, h, w = x.shape x = self.reshape(x, (b, c, h * w)) x = self.transpose(x, (0, 2, 1)) x = self.norm(x) if mask is not None: mask_tokens = self.tile(self.mask_token, (b, self.num_patches, 1)) x = self.mul(x, (1.0 - mask)) x = self.add(x, self.mul(mask_tokens, mask)) return x def get_kernel_size(patch_size): """ input: 2^i & i <= 5 | 14 output: a list of kernel size """ x = None y = None z = None ans = False for i in range(1, patch_size + 1): if patch_size % i == 0: x = i mul_y_z = patch_size // i for j in range(1, mul_y_z + 1): if mul_y_z % j == 0: y = j z = mul_y_z // j if x >= y >= z: ans = True break if ans: break if not ans: raise ValueError(patch_size) return [x, y, z] class Patchify(nn.Cell): """Patchify""" def __init__(self, patch_size, parallel_config=None): super(Patchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.patch_size = patch_size self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, img): p = self.patch_size bs, channels, h, w = img.shape x = self.reshape(img, (bs, channels, h // p, p, w // p, p)) x = self.transpose(x, (0, 2, 4, 1, 3, 5)) patches = self.reshape(x, (bs, (h // p) * (w // p), channels * p * p)) return patches class UnPatchify(nn.Cell): """UnPatchify""" def __init__(self, patch_size, seq_length, parallel_config=None): super(UnPatchify, self).__init__() if parallel_config: dp = parallel_config.data_parallel else: dp = 1 self.p = patch_size self.h = self.w = int(seq_length ** .5) assert self.h * self.w == seq_length self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, 1, 1, 1, 1),)) def construct(self, x): bs = x.shape[0] x = self.reshape(x, (bs, self.h, self.w, self.p, self.p, 3)) x = self.transpose(x, (0, 5, 1, 3, 2, 4)) images = self.reshape(x, (bs, 3, self.h * self.p, self.w * self.p)) return images class PixelShuffle(nn.Cell): """PixelShuffle""" def __init__(self, upscale_factor): super().__init__() self.upscale_factor = upscale_factor self.reshape = P.Reshape() self.transpose = P.Transpose() def construct(self, x): b, _, h, w = x.shape x = self.reshape(x, (b, -1, self.upscale_factor, self.upscale_factor, h, w)) x = self.transpose(x, (0, 1, 4, 2, 5, 3)) x = self.reshape(x, (b, -1, h * self.upscale_factor, w * self.upscale_factor)) return x pengcheng.mind-mindformer/mindformers/models/vit/vit_processor.py000066400000000000000000000111201476650141400257720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ ViTProcessor """ import numpy as np from PIL import Image from mindspore import Tensor from mindspore.dataset.vision import CenterCrop, ToTensor, Normalize, Rescale from mindformers.mindformer_book import MindFormerBook from mindformers.dataset import Resize from mindformers.dataset.base_dataset import BaseDataset from mindformers.models.base_processor import BaseProcessor, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTImageProcessor(BaseImageProcessor): """ ViTImageProcessor. Args: image_resolution (int): the target size. """ def __init__(self, size=224): super().__init__(image_resolution=size) self.resize = Resize(256, interpolation='cubic') self.center_crop = CenterCrop(size) self.to_tensor = ToTensor() self.rescale = Rescale(1.0 / 255.0, 0.0) self.normalize = Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_hwc=False) def preprocess(self, images, **kwargs): """ Preprocess required by base processor. Args: images (tensor, PIL.Image, numpy.array, list): a batch of images. Return: A 4-rank tensor for a batch of images. """ images = self._format_inputs(images) res = [] for image in images: image = self.resize(image) image = self.center_crop(image) image = self.to_tensor(image) image = self.normalize(image) res.append(image) return Tensor(res) def _format_inputs(self, inputs): """ Transform image classification inputs into (bz, h, w, c) or (h, w, c) numpy array. Args: inputs (tensor, numpy.array, PIL.Image, list, BaseDataset): for numpy or tensor input, the channel could be (bz, c, h, w), (c, h, w) or (bz, h, w, c), (h, w, c); for list, the item could be PIL.Image, numpy.array, Tensor; for BaseDataset, return without any operations. Return: transformed images: for PIL.Image, numpy or tensor input, return a numpy array, the channel is (bz, h, w, c) or (h, w, c); for list, return a numpy array for each element; for BaseDataset, it is returned directly. """ if not isinstance(inputs, (list, Image.Image, Tensor, np.ndarray, BaseDataset)): raise TypeError("input type is not Tensor, numpy, Image, list of Image or MindFormer BaseDataset") if isinstance(inputs, list): return [self._format_inputs(item) for item in inputs] if isinstance(inputs, Image.Image): inputs = np.array(inputs) if isinstance(inputs, Tensor): inputs = inputs.asnumpy() if isinstance(inputs, np.ndarray): if len(inputs.shape) == 3: inputs = np.expand_dims(inputs, 0) inputs = self._chw2hwc(inputs) elif len(inputs.shape) == 4: inputs = self._chw2hwc(inputs) else: raise ValueError(f"the rank of image_batch should be 3 or 4," f" but got {len(inputs.shape)}") return inputs @staticmethod def _chw2hwc(inputs): if inputs.shape[-1] != 3: inputs = inputs.transpose(0, 2, 3, 1) return inputs @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class ViTProcessor(BaseProcessor): """ Vit processor, consists of a feature extractor (BaseFeatureEXtractor) for image input, and a tokenizer (BaseTokenizer) for text input. """ _support_list = MindFormerBook.get_processor_support_list()['vit'] def __init__(self, image_processor=None, return_tensors='ms'): super().__init__( image_processor=image_processor, return_tensors=return_tensors ) pengcheng.mind-mindformer/mindformers/modules/000077500000000000000000000000001476650141400221075ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/__init__.py000066400000000000000000000016421476650141400242230ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Transformers API.""" from .transformer import * from .layers import * from .local_block_sparse_attention import * __all__ = [] __all__.extend(transformer.__all__) __all__.extend(layers.__all__) __all__.extend(local_block_sparse_attention.__all__) pengcheng.mind-mindformer/mindformers/modules/__pycache__/000077500000000000000000000000001476650141400243175ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005471476650141400305150ustar00rootroot00000000000000a ?cf@sHdZddlTddlTddlTgZeejeejeejdS)zMindFormers Transformers API.)*N)__doc__ transformerlayersZlocal_block_sparse_attention__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/modules/__init__.pys  pengcheng.mind-mindformer/mindformers/modules/__pycache__/activation.cpython-39.pyc000066400000000000000000001512401476650141400311140ustar00rootroot00000000000000a ?cf@sdZddlmZddlZddlmZddlm Z ddl m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZgdZGdddeZGdddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%GdddeZ&Gd d!d!eZ'Gd"d#d#eZ(Gd$d%d%eZ)Gd&d'd'eZ*Gd(d)d)eZ+Gd*d+d+eZ,Gd,d-d-eZ-Gd.d/d/eZ.Gd0d1d1eZ/Gd2d3d3eZ0Gd4d5d5eZ1Gd6d7d7eZ2Gd8d9d9eZ3Gd:d;d;eZ4Gdd?d?eZ6Gd@dAdAeZ7GdBdCdCeZ8GdDdEdEeZ9GdFdGdGeZ:GdHdIdIeZ;e e"e!e#e%e&e(e*e+e,e-e.e/e$e0e1e2e'e3e4e5e7e8e9e:dJZ>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-2.0, -1.0, 1.0, 2.0]), mindspore.float32) >>> celu = nn.CELU() >>> output = celu(x) >>> print(output) [-0.86466473 -0.63212055 1. 2. ] ?cs tt|tj|d|_dS)zInitialize CELU.)alphaN)superr*__init__PCeLUceluselfr/ __class__I/home/ma-user/work/mindformers-baichuan/mindformers/modules/activation.pyr1nsz CELU.__init__cCs ||SN)r4r6xr9r9r: constructsszCELU.construct)r.__name__ __module__ __qualname____doc__r1r> __classcell__r9r9r7r:r*As,r*cs*eZdZdZdfdd ZddZZS)ra: Softmin activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, and the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmin is defined as: .. math:: \text{softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_{j=0}^{n-1}\exp(-x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmin operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`. -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - Tensor for computing Softmin functions with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range [0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in the range [-x.ndim, x.ndim). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmin = nn.Softmin() >>> output = softmin(x) >>> print(output) [0.2341 0.636 0.0862 0.01165 0.03168 ] cstt|||_dS)zInitialize Softmin.N)r0rr1axisr6rFr7r9r:r1szSoftmin.__init__cCstj||jSr;)rfunctionsoftminrFr<r9r9r:r>szSoftmin.construct)rEr?r9r9r7r:rws-rcs8eZdZdZfddZeeddZddZZ S)ra Softmax function applied to 2D features data. Applies `Softmax` to each location :math:`(c, h, w)` with an input Tensor of shape :math:`(C, H, W)` . Inputs: - **x** (Tensor) - Tensor of shape :math:`(N, C_{in}, H_{in}, W_{in})` or :math:`(C_{in}, H_{in}, W_{in})`. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `data_format` is neither 'NCHW' nor 'CHW'. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.2]], [[0.3, 0.4]], [[0.6, 0.5]]]]), mindspore.float32) >>> softmax2d = nn.Softmax2d() >>> output = softmax2d(x) >>> print(output) [[[[0.25838965 0.28001308]] [[0.31559783 0.34200877]] [[0.42601252 0.37797815]]]] cs*tt|tjdd|_t|_dS)zInitialize Softmax2d.)rFN)r0rr1r2rsoftmaxShapeshaper6r7r9r:r1szSoftmax2d.__init__cCs*t|}|dvr&td|d|ddS)N)For 'z/', the in_shape must have 3 or 4 dims, but got .)len ValueError)rMcls_namedimr9r9r:_check_input_dimszSoftmax2d._check_input_dimcCs"||}|||j||Sr;)rMrWrUrK)r6r=x_shaper9r9r:r>s zSoftmax2d.construct) r@rArBrCr1 staticmethodrrWr>rDr9r9r7r:rs   rcs*eZdZdZdfdd ZddZZS)ra Softmax activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmax is defined as: .. math:: \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_{j=0}^{n-1}\exp(x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmax operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of Softmax with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmax = nn.Softmax() >>> output = softmax(x) >>> print(output) [0.03168 0.01166 0.0861 0.636 0.2341 ] rEcstt|t||_dS)zInitialize Softmax.N)r0rr1r2rKrGr7r9r:r1szSoftmax.__init__cCs ||Sr;)rKr<r9r9r:r>szSoftmax.construct)rEr?r9r9r7r:rs-rcs*eZdZdZdfdd ZddZZS)ra: Applies the LogSoftmax function to n-dimensional input tensor. The input is transformed by the Softmax function and then by the log function to lie in range[-inf,0). Logsoftmax is defined as: .. math:: \text{logsoftmax}(x_i) = \log \left(\frac{\exp(x_i)}{\sum_{j=0}^{n-1} \exp(x_j)}\right), Args: axis (int): The axis to apply LogSoftmax operation, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of LogSoftmax, with float16 or float32 data type. Outputs: Tensor, which has the same type and shape as `x` with output values in the range[-inf,0). Raises: TypeError: If `axis` is not an int. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is not in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> log_softmax = nn.LogSoftmax() >>> output = log_softmax(x) >>> print(output) [[-5.00672150e+00 -6.72150636e-03 -1.20067215e+01] [-7.00091219e+00 -1.40009127e+01 -9.12250078e-04]] rEcstt|t||_dS)zInitialize LogSoftmax.N)r0rr1r2 log_softmaxrGr7r9r:r1AszLogSoftmax.__init__cCs ||Sr;)rZr<r9r9r:r>FszLogSoftmax.construct)rEr?r9r9r7r:rs(rcs*eZdZdZdfdd ZddZZS)r%ah Exponential Linear Unit activation function. Applies the exponential linear unit function element-wise. The activation function is defined as: .. math:: E_{i} = \begin{cases} x_i, &\text{if } x_i \geq 0; \cr \alpha * (\exp(x_i) - 1), &\text{otherwise.} \end{cases} where :math:`x_i` represents the element of the input and :math:`\alpha` represents the `alpha` parameter. ELU Activation Function Graph: .. image:: images/ELU.png :align: center Args: alpha (float): The alpha value of ELU, the data type is float. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of ELU is a Tensor of any dimension with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `alpha` is not equal to 1.0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float32) >>> elu = nn.ELU() >>> result = elu(x) >>> print(result) [-0.63212055 -0.86466473 0. 2. 1.] r.cstt|t||_dS)zInitialize ELU.N)r0r%r1r2Eluelur5r7r9r:r1{sz ELU.__init__cCs ||Sr;)r\r<r9r9r:r>sz ELU.construct)r.r?r9r9r7r:r%Js0r%cs(eZdZdZfddZddZZS)ra4 Rectified Linear Unit activation function. .. math:: \text{ReLU}(x) = (x)^+ = \max(0, x), It returns element-wise :math:`\max(0, x)`. Specially, the neurons with the negative output will be suppressed and the active neurons will stay the same. ReLU Activation Function Graph: .. image:: images/ReLU.png :align: center Inputs: - **x** (Tensor) - The input of ReLU is a Tensor of any dimension. The data type is `number `_ . Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is not a number. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> relu = nn.ReLU() >>> output = relu(x) >>> print(output) [0. 2. 0. 2. 0.] cstt|t|_dS)zInitialize ReLU.N)r0rr1r2relurNr7r9r:r1sz ReLU.__init__cCs ||Sr;)r]r<r9r9r:r>szReLU.constructr?r9r9r7r:rs' rcs(eZdZdZfddZddZZS)ra Compute ReLU6 activation function. ReLU6 is similar to ReLU with a upper limit of 6, which if the inputs are greater than 6, the outputs will be suppressed to 6. It computes element-wise as .. math:: Y = \min(\max(0, x), 6). The input is a Tensor of any valid shape. ReLU6 Activation Function Graph: .. image:: images/ReLU6.png :align: center Inputs: - **x** (Tensor) - The input of ReLU6 with data type of float16 or float32. Outputs: Tensor, which has the same type as `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> relu6 = nn.ReLU6() >>> output = relu6(x) >>> print(output) [0. 0. 0. 2. 1.] cstt|t|_dS)zInitialize ReLU6.N)r0rr1r2relu6rNr7r9r:r1szReLU6.__init__cCs ||Sr;)r^r<r9r9r:r>szReLU6.constructr?r9r9r7r:rs) rcs*eZdZdZdfdd ZddZZS)r"a3 Leaky ReLU activation function. The activation function is defined as: .. math:: \text{leaky_relu}(x) = \begin{cases}x, &\text{if } x \geq 0; \cr {\alpha} * x, &\text{otherwise.}\end{cases} where :math:`\alpha` represents the `alpha` parameter. For more details, see `Rectifier Nonlinearities Improve Neural Network Acoustic Models `_. LeakyReLU Activation Function Graph: .. image:: images/LeakyReLU.png :align: center Args: alpha (Union[int, float]): Slope of the activation function at x < 0. Default: ``0.2`` . Inputs: - **x** (Tensor) - The input of LeakyReLU is a Tensor of any dimension. Outputs: Tensor, has the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> leaky_relu = nn.LeakyReLU() >>> output = leaky_relu(x) >>> print(output) [[-0.2 4. -1.6] [ 2. -1. 9. ]] 皙?cs`tt|td|ttg|jt |_ t |_ ||_ t|_|j dkr\t|_dS)zInitialize LeakyReLU.r/N)r0r"r1 validatorcheck_value_typefloatintrUr2 GreaterEqual greater_equalMulmulr/Maximum select_opMinimumr5r7r9r:r1s    zLeakyReLU.__init__cCs2tt|jt|}||||}|Sr;)r2CastFscalar_to_tensorr/DTyperj)r6r=Z alpha_arrayoutr9r9r:r>"szLeakyReLU.construct)r_r?r9r9r7r:r"s. r"cs*eZdZdZdfdd ZddZZS) raR Randomized Leaky ReLU activation function. The activation function is defined as: .. math:: \text{RReLU}(x_{ji}) = \begin{cases}x_{ji}, &\text{if } x_{ji} \geq 0; \cr {\alpha_{ji}} * x_{ji}, &\text{otherwise.}\end{cases} where :math:`\alpha_{ji}` ~ :math:`U(l, u)`, :math:`l \le u`. Applies the RReLU function elementally, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolution Network `_ . Args: lower (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 8`` . upper (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 3`` . Inputs: - **x** (Tensor) - The input of RReLU is a Tensor of any dimension. Outputs: Tensor, after RReLU, has the same type and shape as the `x`. Raises: TypeError: If `lower` is not a float or an int. TypeError: If `upper` is not a float or an int. TypeError: If `x` is not a Tensor. TypeError: If `x` is not a Tensor of mindspore.float16 or mindpore.float32. ValueError: If `lower` is greater than upper. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0], [2.0, 0]]), mindspore.float32) >>> r_relu = nn.RReLU() >>> output = r_relu(x) >>> print(output) [[-0.31465699 4. ] [ 2. 0. ]] ?UUUUUU?cstt|td|ttg|jtd|ttg|j||kr`td|jd|d|dt |t j d|_ t |t j d|_ t|_dS)NupperlowerzFor zP, the value of 'upper' must be greater than or equal to 'lower', but got upper: z , lower: z. r)r0rr1rarbrcrdrUrTr mstypefloat32rtrsr2Signsign)r6rtrsr7r9r:r1Xs  zRReLU.__init__c Cst|tstdt|d|j}|j}||}|dd}|dd}t ||j |j  |}||d}||} | |} | S)zRReLU construct.z1For 'rrelu', the input must be a Tensor, but got rRNrrE) isinstancer TypeErrortyperMrrxclipruniformrtrsastype) r6r=sizer sign_matrixnegative_filterpositive_filtermask negative_mask total_maskrpr9r9r:r>cs     zRReLU.construct)rqrrr?r9r9r7r:r(s/ rcs(eZdZdZfddZddZZS)ra Activation function SeLU (Scaled exponential Linear Unit). SeLU Activation Function Graph: .. image:: images/SeLU.png :align: center Refer to :func:`mindspore.ops.selu` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> selu = nn.SeLU() >>> output = selu(input_x) >>> print(output) [[-1.1113307 4.202804 -1.7575096] [ 2.101402 -1.7462534 9.456309 ]] cstt|t|_dS)zInitialize SeLUN)r0rr1r2selurNr7r9r:r1sz SeLU.__init__cCs ||Sr;)rr6input_xr9r9r:r>szSeLU.constructr?r9r9r7r:rss rc@seZdZdZddZdS)ra Sigmoid Linear Unit activation function. Applies the silu linear unit function element-wise. .. math:: \text{SiLU}(x) = x * \sigma(x), where :math:`x_i` is input, :math:`\sigma(x)` is Sigmoid function. .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, SiLU Activation Function Graph: .. image:: images/SiLU.png :align: center Inputs: - **x** (Tensor) - Input with the data type float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> silu = nn.SiLU() >>> output = silu(x) >>> print(output) [-0.269 1.762 -0.1423 1.762 -0.269] cCs tj|Sr;)rrHsilur<r9r9r:r>szSiLU.constructNr@rArBrCr>r9r9r9r:rs+rcs(eZdZdZfddZddZZS)rae Applies the Tanh function element-wise, returns a new tensor with the hyperbolic tangent of the elements of input, The input is a Tensor with any valid shape. Tanh function is defined as: .. math:: tanh(x_i) = \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = \frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Tanh Activation Function Graph: .. image:: images/Tanh.png :align: center Inputs: - **x** (Tensor) - Tensor of any dimension, input with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), mindspore.float16) >>> tanh = nn.Tanh() >>> output = tanh(x) >>> print(output) [0.7617 0.964 0.995 0.964 0.7617] cstt|t|_dS)zInitialize Tanh.N)r0rr1r2tanhrNr7r9r:r1sz Tanh.__init__cCs ||Sr;)rr<r9r9r:r>szTanh.constructr?r9r9r7r:rs' rc@seZdZdZddZdS)ra Tanhshrink activation function. The tanhshrink function is evaluated by element and returns a new tensor. Tanh function is defined as: .. math:: tanhshrink(x_i) =x_i- \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = x_i-\frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Inputs: - **x** (Tensor) - Tensor of any dimension. Outputs: Tensor, with the same shape as the `x`. Raises: TypeError: If `x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), ms.float16) >>> tanhshrink = nn.Tanhshrink() >>> output = tanhshrink(x) >>> print(output) [0.2383 1.036 2.004 1.036 0.2383] cCs t|Sr;)rm tanhshrinkr<r9r9r:r>szTanhshrink.constructNrr9r9r9r:rs$rcs*eZdZdZdfdd ZddZZS) ra Applies the Hardtanh function element-wise. The activation function is defined as: .. math:: \text{Hardtanh}(x) = \begin{cases} 1, & \text{ if } x > 1; \\ -1, & \text{ if } x < -1; \\ x, & \text{ otherwise. } \end{cases} Linear region range :math:`[-1, 1]` can be adjusted using `min_val` and `max_val`. Hardtanh Activation Function Graph: .. image:: images/Hardtanh.png :align: center Note: On Ascend, data type of float16 might lead to accidental accuracy problem. Args: min_val (Union[int, float]): Minimum value of the linear region range. Default: ``-1.0`` . max_val (Union[int, float]): Maximum value of the linear region range. Default: ``1.0`` . Inputs: - **x** (Tensor) - Input Tensor with data type of float16 or float32. On CPU and Ascend support dimension 0-7D. On GPU support dimension 0-4D. Outputs: Tensor, with the same dtype and shape as `x`. Raises: TypeError: If `x` is not a Tensor. TypeError: If dtype of `x` is neither float16 nor float32. TypeError: If dtype of `min_val` is neither float nor int. TypeError: If dtype of `max_val` is neither float nor int. ValueError: If `min_val` is not less than `max_val`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hardtanh = nn.Hardtanh(min_val=-1.0, max_val=1.0) >>> output = hardtanh(x) >>> print(output) [-1. -1. 0. 1. 1.] r.csBtt|||_||_|j|jkr>td|jd|jdS)zInitialize Hardtanh.z:For Hardtanh, min_val should be less than max_val,but got z and N)r0rr1min_valmax_valrT)r6rrr7r9r:r1Us zHardtanh.__init__cCst||j|jSr;)rmhardtanhrrr<r9r9r:r>^szHardtanh.construct)rr.r?r9r9r7r:r s4 rcs*eZdZdZdfdd ZddZZS)ra Gaussian error linear unit activation function. Applies GELU function to each element of the input. The input is a Tensor with any valid shape. GELU is defined as: .. math:: GELU(x_i) = x_i*P(X < x_i), where :math:`P` is the cumulative distribution function of standard Gaussian distribution and :math:`x_i` is the element of the input. GELU Activation Function Graph: .. image:: images/GELU.png :align: center Args: approximate (bool): Whether to enable approximation. Default: ``True`` . If `approximate` is ``True``, The gaussian error linear activation is: :math:`0.5 * x * (1 + tanh(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))` else, it is: :math:`x * P(X <= x) = 0.5 * x * (1 + erf(x / \sqrt(2)))`, where P(X) ~ N(0, 1). Inputs: - **x** (Tensor) - The input of GELU with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> gelu = nn.GELU() >>> output = gelu(x) >>> print(output) [[-1.5880802e-01 3.9999299e+00 -3.1077917e-21] [ 1.9545976e+00 -2.2918017e-07 9.0000000e+00]] >>> gelu = nn.GELU(approximate=False) >>> # CPU not support "approximate=False", using "approximate=True" instead >>> output = gelu(x) >>> print(output) [[-1.5865526e-01 3.9998732e+00 -0.0000000e+00] [ 1.9544997e+00 -1.4901161e-06 9.0000000e+00]] Tcsxtt|t|d|j||_|jr6t|_ n>t |_ t |_ tdtj|_tdtj|_tdtj|_dS)zInitialize GELU. approximate?r.g@N)r0rr1ra check_boolrUrr2GeLUgeluErferfSqrtsqrtr rurvconst0const1const2)r6rr7r9r:r1s   z GELU.__init__c CsT|jr||S|t|j|jt|j|j|||t|j |jSr;) rrrmcastrrrrrrr<r9r9r:r>s   zGELU.construct)Tr?r9r9r7r:rbs=rcs(eZdZdZfddZddZZS)ra< Fast Gaussian error linear unit activation function. Applies FastGelu function to each element of the input. The input is a Tensor with any valid shape. FastGelu is defined as: .. math:: FastGelu(x_i) = \frac {x_i} {1 + \exp(-1.702 * \left| x_i \right|)} * \exp(0.851 * (x_i - \left| x_i \right|)) where :math:`x_i` is the element of the input. FastGelu Activation Function Graph: .. image:: images/FastGelu.png :align: center Inputs: - **x** (Tensor) - The input of FastGelu with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> fast_gelu = nn.FastGelu() >>> output = fast_gelu(x) >>> print(output) [[-1.5418735e-01 3.9921875e+00 -9.7473649e-06] [ 1.9375000e+00 -1.0052517e-03 8.9824219e+00]] cstt|t|_dS)zInitialize FastGelu.N)r0rr1r2FastGeLU fast_gelurNr7r9r:r1szFastGelu.__init__cCs ||Sr;)rr<r9r9r:r>szFastGelu.constructr?r9r9r7r:rs+ rcs(eZdZdZfddZddZZS)ral Sigmoid activation function. Applies sigmoid-type activation element-wise. Sigmoid function is defined as: .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, where :math:`x_i` is the element of the input. Sigmoid Activation Function Graph: .. image:: images/Sigmoid.png :align: center Inputs: - **input_x** (Tensor) - Tensor of any dimension, the data type is float16, float32, float64, complex64 or complex128. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If dtype of `input_x` is not float16, float32, float64, complex64 or complex128. TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> sigmoid = nn.Sigmoid() >>> output = sigmoid(x) >>> print(output) [0.2688 0.11914 0.5 0.881 0.7305 ] cstt|t|_dS)zInitialize Sigmoid.N)r0rr1r2sigmoidrNr7r9r:r1szSigmoid.__init__cCs ||Sr;)rr<r9r9r:r>szSigmoid.constructr?r9r9r7r:rs+ rcs(eZdZdZfddZddZZS)raj Softsign activation function. Softsign Activation Function Graph: .. image:: images/Softsign.png :align: center Refer to :func:`mindspore.ops.softsign` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([0, -1, 2, 30, -30]), mindspore.float32) >>> softsign = nn.Softsign() >>> output = softsign(x) >>> print(output) [ 0. -0.5 0.6666667 0.9677419 -0.9677419] cstt|t|_dS)zInitialize Softsign.N)r0rr1r2softsignrNr7r9r:r18szSoftsign.__init__cCs ||Sr;)rr<r9r9r:r>=szSoftsign.constructr?r9r9r7r:rs rcs4eZdZdZeddd fdd Zdd ZZS) r a PReLU activation function. Applies the PReLU function element-wise. PReLU is defined as: .. math:: PReLU(x_i)= \max(0, x_i) + w * \min(0, x_i), where :math:`x_i` is an element of an channel of the input. Here :math:`w` is a learnable parameter with a default initial value 0.25. Parameter :math:`w` has dimensionality of the argument channel. If called without argument channel, a single parameter :math:`w` will be shared across all channels. PReLU Activation Function Graph: .. image:: images/PReLU.png :align: center Args: channel (int): The elements number of parameter :math:`w`. It could be an int, and the value is 1 or the channels number of input tensor `x`. Default: ``1`` . w (Union[float, list, Tensor]): The initial value of parameter. It could be a float, a float list or a tensor has the same dtype as the input tensor `x`. Default: ``0.25`` . Inputs: - **x** (Tensor) - The input of PReLU with data type of float16 or float32. The shape is :math:`(N, *)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same dtype and shape as the `x`. Raises: TypeError: If `channel` is not an int. TypeError: If `w` is not one of a float, a float list, a float Tensor. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If the `x` is a 0-D or 1-D Tensor on Ascend. ValueError: If `channel` is less than 1. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.6], [0.9, 0.9]]]]), mindspore.float32) >>> prelu = nn.PReLU() >>> output = prelu(x) >>> print(output) [[[[0.1 0.6] [0.9 0.9]]]] )attrsr`?cstt|t|d|jt|ttj fr\tj |ftj d}| |t |t j d}nt|trt||krtd|jdt|d|d|D],}t|ttj fstd|jd|dqt |t j d}nt|t rR|jt jt j fvrtd|jd|jdt|jd ks0|jd |krptd|jd |jd |dntd|jd t|jdt|dd|_t|_dS)zInitialize PReLU.channelrrQzg', the length of 'w' must be equal to the 'channel' when the 'w' is a list, but got the length of 'w': z, the 'channel': rRzE', all elements in 'w' must be float when the 'w' is a list, but got zQ', the dtype of 'w' must be float16 or float32 when the 'w' is a tensor, but got r`rz', the dimension of 'w' must be 1, and the elements number should be equal to the 'channel' when the 'w' is a tensor, but got 'w' shape z, the 'channel' z:', the 'w' only supported float, list and tensor, but got a)nameN)r0r r1racheck_positive_intrUryrcnprvemptyfillr rulistrSrTrfloat16rMrzr{r@rwr2prelu)r6rrtmpir7r9r:r1|sF               zPReLU.__init__cCs||t|j|jSr;)rrmrrrr<r9r9r:r>szPReLU.construct)r`r)r@rArBrCrr1r>rDr9r9r7r:r As: r cs(eZdZdZfddZddZZS)r$a Applies hswish-type activation element-wise. The input is a Tensor with any valid shape. Hard swish is defined as: .. math:: \text{hswish}(x_{i}) = x_{i} * \frac{ReLU6(x_{i} + 3)}{6}, HSwish Activation Function Graph: .. image:: images/HSwish.png :align: center Inputs: - **x** (Tensor) - The input of HSwish, data type must be float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hswish = nn.HSwish() >>> result = hswish(x) >>> print(result) [-0.3333 -0.3333 0. 1.667 0.6665] cstt|t|_dS)zInitialize HSwish.N)r0r$r1r2hswishrNr7r9r:r1szHSwish.__init__cCs ||Sr;)rr<r9r9r:r>szHSwish.constructr?r9r9r7r:r$s% r$cs(eZdZdZfddZddZZS)r#a Hard sigmoid activation function. Calculates the output according to the input elements. Hard sigmoid is defined as: .. math:: \text{hsigmoid}(x_{i}) = \max(0, \min(1, \frac{x_{i} + 3}{6})), HSigmoid Activation Function Graph: .. image:: images/HSigmoid.png :align: center Inputs: - **input_x** (Tensor) - The input of HSigmoid. Tensor of any dimension. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hsigmoid = nn.HSigmoid() >>> result = hsigmoid(x) >>> print(result) [0.3333 0.1666 0.5 0.8335 0.6665] cstt|t|_dS)zInitialize HSigmoid.N)r0r#r1r2hsigmoidrNr7r9r:r1szHSigmoid.__init__cCs ||Sr;)rrr9r9r:r>szHSigmoid.constructr?r9r9r7r:r#s$ r#cs(eZdZdZfddZddZZS)r&aN Applies logsigmoid activation element-wise. The input is a Tensor with any valid shape. Logsigmoid is defined as: .. math:: \text{logsigmoid}(x_{i}) = \log(\frac{1}{1 + \exp(-x_i)}), where :math:`x_{i}` is the element of the input. LogSigmoid Activation Function Graph: .. image:: images/LogSigmoid.png :align: center Inputs: - **x** (Tensor) - The input of LogSigmoid with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> net = nn.LogSigmoid() >>> x = Tensor(np.array([1.0, 2.0, 3.0]), mindspore.float32) >>> output = net(x) >>> print(output) [-0.31326166 -0.12692806 -0.04858734] csDtt|t|_t|_t|_ t |_ t |_ dS)zInitialize LogSigmoid.N)r0r&r1r2rgrhExpexpAddadd ReciprocalrecLoglogrNr7r9r:r1&s     zLogSigmoid.__init__cCs:||d}||}||d}||}||}|S)NrEr`)rhrrrr)r6r neg_input exp_neg_inputexp_neg_input_1rec_exp_neg_input_1retr9r9r:r>/s      zLogSigmoid.constructr?r9r9r7r:r&s' r&cs*eZdZdZd fdd Zdd ZZS) r'a; Local Response Normalization. Refer to :func:`mindspore.ops.lrn` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[[[0.1], [0.2]], ... [[0.3], [0.4]]]]), mindspore.float32) >>> output = nn.LRN()(input_x) >>> print(output) [[[[0.09534626] [0.1825742 ]] [[0.2860388 ] [0.3651484 ]]]] r.rACROSS_CHANNELScs&tt|t||||||_dS)zInitialize LRN.N)r0r'r1NN_OPSlrn_op)r6 depth_radiusbiasr/beta norm_regionr7r9r:r1Osz LRN.__init__cCs ||Sr;)rrr9r9r:r>Tsz LRN.construct)rr.r.rrr?r9r9r7r:r'8sr'cs*eZdZdZdfdd ZddZZS)r(a Applies the SoftShrink function element-wise. .. math:: \text{SoftShrink}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} SoftShrink Activation Function Graph: .. image:: images/Softshrink.png :align: center Args: lambd (float): the :math:`\lambda` must be no less than zero for the SoftShrink formulation. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of SoftShrink with data type of float16 or float32. Any number of additional dimensions. Outputs: Tensor, has the same shape and data type as `input_x`. Raises: TypeError: If lambd is not a float. TypeError: If input_x is not a Tensor. TypeError: If dtype of input_x is neither float16 nor float32. ValueError: If lambd is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5297, 0.7871, 1.1754], [ 0.7836, 0.6218, -1.1542]]), mindspore.float16) >>> softshrink = nn.SoftShrink() >>> output = softshrink(input_x) >>> print(output) [[ 0.02979 0.287 0.676 ] [ 0.2837 0.1216 -0.6543 ]] rcstt|t||_dSr;)r0r(r1r2 softshrinkr6lambdr7r9r:r1szSoftShrink.__init__cCs||}|Sr;)r)r6routputr9r9r:r>s zSoftShrink.construct)rr?r9r9r7r:r(Xs0r(cs*eZdZdZdfdd ZddZZS)r)a* Hard Shrink activation function. Calculates the output according to the input elements. The formula is defined as follows: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} HShrink Activation Function Graph: .. image:: images/HShrink.png :align: center Args: lambd (float): The threshold :math:`\lambda` defined by the Hard Shrink formula. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of Hard Shrink with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `lambd` is not a float. TypeError: If dtype of `input_x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5, 1, 2.0], [0.0533,0.0776,-2.1233]]), mindspore.float32) >>> hshrink = nn.HShrink() >>> output = hshrink(input_x) >>> print(output) [[ 0. 1. 2. ] [ 0. 0. -2.1233]] rcstt|t||_dSr;)r0r)r1r2hshrinkrr7r9r:r1szHShrink.__init__cCs ||Sr;)rrr9r9r:r>szHShrink.construct)rr?r9r9r7r:r)s.r)cs(eZdZdZfddZddZZS)r+a4Thresholds each element of the input Tensor. The formula is defined as follows: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold (Union[int, float]): The value to threshold at. value (Union[int, float]): The value to replace with when element is less than threshold. Inputs: - **input_x** (Tensor) - The input of Threshold with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `threshold` is not a float or an int. TypeError: If `value` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> m = nn.Threshold(0.1, 20) >>> inputs = Tensor([0.1, 0.2, 0.3], mindspore.float32) >>> outputs = m(inputs) >>> print(outputs) [ 20.0 0.2 0.3] cstt|||_||_dS)zInitialize Threshold.N)r0r+r1 thresholdvalue)r6rrr7r9r:r1szThreshold.__init__cCst||j|jSr;)rmrrrr9r9r:r>szThreshold.constructr?r9r9r7r:r+s& r+cs(eZdZdZfddZddZZS)r,a Computes MISH(A Self Regularized Non-Monotonic Neural Activation Function) of input tensors element-wise. Refer to :func:`mindspore.ops.mish` for more details. Mish Activation Function Graph: .. image:: images/Mish.png :align: center Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> mish = nn.Mish() >>> output = mish(x) >>> print(output) [[-3.03401530e-01 3.99741292e+00 -2.68321624e-03] [ 1.94395900e+00 -3.35762873e-02 9.00000000e+00]] cstdt|_dS)zInitialize Mish.r,N)r0r1rr,mishrNr7r9r:r1s z Mish.__init__cCs ||Sr;)rrr9r9r:r>szMish.constructr?r9r9r7r:r,s r,cs*eZdZdZdfdd ZddZZS)r-a The gated linear unit function. .. math:: {GLU}(a, b)= a \otimes \sigma(b) where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Here :math:`\sigma` is the sigmoid function, and :math:`\otimes` is the Hadamard product. Args: axis (int): the axis to split the input. Default: ``-1`` , the last axis in `x`. Inputs: - **x** (Tensor) - :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions. Outputs: Tensor, the same dtype as the `x`, with the shape :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> m = ms.nn.GLU() >>> input = ms.Tensor([[0.1,0.2,0.3,0.4],[0.5,0.6,0.7,0.8]]) >>> output = m(input) >>> print(output) [[0.05744425 0.11973753] [0.33409387 0.41398472]] rEcs0td||_tj|dd|_t|_dS)zInitialize GLU.r-)rF output_numN)r0r1rVr2SplitspiltrrrGr7r9r:r1>s z GLU.__init__cCs ||\}}||}||Sr;)rr)r6r=x1x2r9r9r:r>Es z GLU.construct)rEr?r9r9r7r:r-s r-)rIrK softmax2d logsoftmaxr]r^rrelurrrrrrr\rrr leakyrelurr logsigmoidrrrrcCsT|rd|dnd}|dur dS|tvrJt|dttd|dt|S)a Gets the activation function. Args: name (str): The name of the activation function. prim_name (Union[str, None]): The name of primitive. Default: ``None`` . Returns: Function, the activation function. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore.nn as nn >>> sigmoid = nn.get_activation('sigmoid') >>> print(sigmoid) Sigmoid<> rQz', theTheNz 'name' must be in z , but got rR) _activationKeyErrorrkeys)r prim_name msg_prefixr9r9r:r!hs "r!)N)>rC __future__rnumpyr mindsporerramindspore._extendsrmindspore.commonrrumindspore.common.parameterrmindspore.common.tensorr mindspore.opsr rmr r2Zmindspore.ops.operationsr rmindspore.nn.cellr rmindspore.ops.primitiver__all__r*rrrrr%rrr"rrrrrrrrrrr r$r#r&r'r(r)r+r,r-rr!r9r9r9r:s            !67372:13@K#01)BS55"`/.: :71#/pengcheng.mind-mindformer/mindformers/modules/__pycache__/layers.cpython-39.pyc000066400000000000000000001040071476650141400302510ustar00rootroot00000000000000a ?cfE@s0dZddlmZddlmZmZddlZddlZddlZ ddl m Z m Z ddl mZddlmZmZddlmmZddlmZdd lmZdd lmZdd lmZdd lmZ dd l!m"Z"zddl#m$Z$Wne%yddl#m&Z$Yn0ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3gdZ4ddZ5ddZ6ddZ7GdddZ8e"ddZ9e"dd Z:e"d!d"Z;e"d#d$ZGd)d*d*e jZ?Gd+d,d,eZ@Gd-d.d.eZAGd/d0d0e jZBGd1d2d2e jZCGd3d4d4e jZDd5d6ZEd7ejFfd8d9ZGdS):zw The basic layer of the Transformer Networks. This is an experimental interface that is subject to change or deletion. )absolute_import)wrapspartialN)nncontext) Parameter) initializerTensor)_get_graph_seed)cell_attr_register)Cell) functional) operations) constexpr) Validator)_get_parallel_mode_is_sharding_propagation) ParallelMode)logger)get_activation)default_dpmp_configOpParallelConfigMoEParallelConfig)FixedSparseAttentionDropout LayerNormLinearcsfdd}|S)z)Check whether input data type is correct.cs8tjijtfdd}|S)Ncslj|i|}|j}dvr&dd|vr6|d}|D]\}}|vr>|||q>|i|S)Nkwargs)bind argumentsitems)argsr bound_values argument_dictnamevalue bound_typesfuncsigE/home/ma-user/work/mindformers-baichuan/mindformers/modules/layers.pywrapper?sz?_args_type_validator_check..type_check..wrapper)inspect signature bind_partialrr)r(r, type_args type_kwargsr&r+ type_check;s  z._args_type_validator_check..type_checkr*)r1r2r3r*r0r+_args_type_validator_check8sr4csfdd}|S)Ncsttjd}||t|SN) valid_types prim_name)rrcheck_type_nametyper%r$Z partial_check class_nametypesr*r+validator_check_funcUsz0_valid_type_checks..validator_check_funcr*r=r<r>r*r;r+_valid_type_checksSsr@csfdd}|S)Ncsttjd}|||Sr5)rrr8r:r;r*r+r>`sz1_valid_value_checks..validator_check_funcr*r?r*r;r+_valid_value_checks^srAc@s@eZdZdZeddZeddZeddZedd Zd S) _LayerInputCheckzI A input check class for the inputs of the transformer model. cCsTt|ts|g}d}|D]}t||krd}q|sPt|d|d|d|dS)aX Check the input shape's length is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_len: the expected length of the shape. :return: FT z shape length must be one of z dimension, but got shape ) isinstancelistlen ValueError) input_shape param_name func_name target_lenmatcheditemr*r*r+check_shape_lengthns  z#_LayerInputCheck.check_shape_lengthcCst|dts|g}t|tr&t|}t|||dd|Dd}|D]}||krHd}q^qH|st|d|d|d|dS) aE Check the input shape's is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_shape: the expected shape. :return: rcSsg|] }t|qSr*)rF).0rMr*r*r+ z6_LayerInputCheck.check_shape_equal..FTrCz shape must be one of ,but got )rDrEtuplerBrNrG)rHrIrJ target_shaperLrMr*r*r+check_shape_equals"    z"_LayerInputCheck.check_shape_equalc Cs8|||kr4t|d|d|d|d|| dS)z< Check whether the input_shape[dim] is equal to target valuerCz at  shape must be rRT)rGrHdimrIcls_nameZ target_valuer*r*r+check_shape_value_on_axiss  z*_LayerInputCheck.check_shape_value_on_axisc Cs|}|\}}t|trt|}t|||t|t|dg|d|krvtd|d|d|d|dd|d |d d krtd|d |d |d d|d t|d kr|d |d krtd|d |d |d|d d|d dS)z Check the input shape's is equal to the expected shape, the value on 0-th is viewed as batch, and the batch size will not be checked. zFor z, the last dimension of rVz,but got the last dimension z in .rz, the first dimension of z2 shape greater than 0,but got the first dimension z shape should be divisible by z, but got the first dimension T)rDrSrErBrNrFrG)rHrIrJrTlengthhiddenr*r*r+check_shape_equal_without_batchs8       z0_LayerInputCheck.check_shape_equal_without_batchN) __name__ __module__ __qualname____doc__ staticmethodrNrUrZrar*r*r*r+rBis   rBcCs^|sB|r"t|d|d|d|sZt|d|d|dn|sZt|d|ddS)z6 If the past is True, check whether the inputs is NonerCz must be z), if use_pat is False, but found a tensorz, if use_pat is False.z# must be tensor, if use_pat is TrueT) TypeError)use_pastrIrJ default_value is_tensor is_defaultr*r*r+_check_past_none_input_nonesrlcCst||||dSN)rr8) input_dtyperI allow_dtypesrYr*r*r+_check_input_dtypesrpcCst||||dSrm)rBrN)rHrIrJrKr*r*r+_check_input_shapesrqcCst||||dSrm)rBrUrHrIrJrTr*r*r+_check_shape_equalsrscCst|||||dSrm)rBrZrWr*r*r+_check_input_shape_valuesrtcCst||||dSrm)rBrarrr*r*r+ _check_shape_equal_without_batchsrucs@eZdZdZdejffdd ZddZddZd d Z Z S) rzd A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for parallel training. g?cstt||dks|dkr,td|td|tj|j t d|t g|j ||_ t ddv|_|jrtdd\}}||_||_||_t|_tj|j|jd |_t|_t|_n t||_dS) Nrr[zBdropout probability should be a number in range (0, 1], but got {}dtype keep_prob device_target)Ascenddropout)Seed0Seed1)superr__init__rGformatrcheck_subclassmstype number_typerYcheck_value_typefloatrwr get_context is_ascendr seed0seed1rvPShape get_shapeDropoutGenMaskdropout_gen_mask DropoutDoMaskdropout_do_maskCastcastrz)selfrwrvrr __class__r*r+r~s*   zDropout.__init__cCsn|js |S|jdkr|S|js0||\}}|S||}t|}||j|}|||}| |||S)zA Input: a tensor Returns: a tensor r[) trainingrwrrzrrDTyperrr)rxout_shapervrwoutputr*r*r+ constructs    zDropout.constructcCs d|jS)Nz keep_prob={})rrw)rr*r*r+ extend_reprszDropout.extend_reprcCs0|jr |j||j|n |j|dSrm)rrshardrrzrstrategyr*r*r+rs z Dropout.shard) rbrcrdrerfloat32r~rrr __classcell__r*r*rr+rs rcs:eZdZdZdejdffdd ZddZdd ZZ S) ra A self-defined layer norm operation using reduce sum and reduce mean Args: normalized_shape (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. gh㈵>Fcstt||tjtjfvr0tdt|||_ |j sNt jdd|d|_ t t d||ddd|_t t d||d dd|_t jd d |_t |_t |_t |_t |_t |_||_t |_t |_t |_ dS) Nz\The type of parameter 'param_init_type' should in [float32, float16], but got the type : {}.r\)begin_norm_axisbegin_params_axisepsilononesgammaF)r$parallel_optimizerzerosbetaT) keep_dims)!r}rr~rrfloat16rgrr9is_self_definedr layer_normrrrr ReduceMeanmeanSquaresquareSqrtsqrtSubsub1sub2AddaddepsMulmuladd2RealDivreal_div)rnormalized_shaperparam_init_typerrr*r+r~3s6       zLayerNorm.__init__cCs|jrj||d}|||}|||d}||||j}|||}|| ||j |j }n| ||j |j \}}}|S)z8 x : batch x seq_length x hidden_size r\) rrrrrrrrrrrrr)rrrdiffvarianceZ variance_epsrrr*r*r+rMs   zLayerNorm.constructcCs|jr|j||j||j||j|d|df|j|d|df|j|ddf|j|ddf|j |ddf|j |d|dfn|j |dddf|S)a3 Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. Examples: >>> import mindspore >>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10)) >>> net.shard(((10, 2, 1),)) rr*r[) rrrrrrrrrrrrrr*r*r+r\s   zLayerNorm.shard) rbrcrdrerrr~rrrr*r*rr+r$src seZdZdZeeejejejejejeje e j e j gde e j e j gdddddddddde j e j f fdd Z d d Zdd d ZZS)raU The dense connected layer. Once the parallel mode is enabled, the input shape should be 3-D tensor. Applies dense connected layer for the input. This layer implements the operation as: .. math:: \text{outputs} = \text{activation}(\text{X} * \text{kernel} + \text{bias}), where :math:`X` is the input tensors, :math:`\text{activation}` is the activation function passed as the activation argument (if passed in), :math:`\text{kernel}` is a weight matrix with the same data type as the :math:`X` created by the layer, and :math:`\text{bias}` is a bias vector with the same data type as the :math:`X` created by the layer (only if has_bias is True). Args: in_channels (int): The number of channels in the input space. out_channels (int): The number of channels in the output space. weight_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable weight_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'normal'. bias_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable bias_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'zeros'. has_bias (bool): Specifies whether the layer uses a bias vector. Default: True. activation (str): activate function applied to the output of the fully connected layer, eg. 'ReLU'. Default: None. expert_num (int): The number of experts used in this Linear. Here, for the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. outer_batch (int): The replication number of experts. The replication is effective only when MoE is applied. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. compute_dtype (dtype.Number): The computation type. Default: mstype.float16 Inputs: - **x** (Tensor) - Tensor of shape :math:`(*, in\_channels)`. The `in_channels` in `Args` should be equal to :math:`in\_channels` in `Inputs`. Outputs: Tensor of shape :math:`(*, out\_channels)`. Raises: TypeError: If `in_channels` or `out_channels` is not an int. TypeError: If `has_bias` is not a bool. TypeError: If `activation` is not one of str, Cell, Primitive, None. ValueError: If length of shape of `weight_init` is not equal to 2 or shape[0] of `weight_init` is not equal to `out_channels` or shape[1] of `weight_init` is not equal to `in_channels`. ValueError: If length of shape of `bias_init` is not equal to 1 or shape[0] of `bias_init` is not equal to `out_channels`. Supported Platforms: ``Ascend`` ``GPU`` ) in_channels out_channelshas_bias transpose_b expert_num outer_batchr compute_dtypenormalrTNr[c sFtt|||_||_t|tsH|dusHt|tj sHt d|dt|t r|j dksx|j d|ksx|j d|krtd|r||gn||g} ||_| |_| |_||_|jdkrd|_tt||jg| | dd |_tj|d |_n*d |_tt|| | dd |_tj|d |_ttjfvo6t o6|jdu|_|jdurZ|jdurZtd d|_ ||_!|j!rt|t r|j dks|j d|krtd |jrtt|d|jd|g| dd |_ ntt||g| dd |_ d |j _"t#|_$||_%t&|r ||_'nt|tr t(|n||_'|j'du|_)| |_*t+|_,dS)NzIFor Linear cell, the activation should str type or nn.Cell type, but got r]r^rr[zSThe shape of parameter 'weight_init' is error, please check shape of 'weight_init'.Tweight)r$)rFzD'expert_group_size' should be configured as an integer in MoEConfig.zOThe shape of parameter 'bias_init' is error, please check shape of 'bias_init'.bias)-r}rr~rrrDstr issubclassrr rgr ndimrrGrrexpert_group_sizer expert_flagrrrr BatchMatMulmatmulMatMulrr AUTO_PARALLELruse_expert_group_sizerrrrbias_addact_namecallable activationractivation_flagrvrr)rrr weight_init bias_initrrrrrrrr weight_shaperr*r+r~sf"  (     zLinear.__init__cCst|dd|jf}t|d|jf}|jrz|jdur^t|d|j|j|jf}nt||j |jd|jf}t |}| |j |j }| ||j }|||}|jr||| |j|j }|jr||}t ||}t||}|S)z%Forward process, x should be a tensorNr\T)rrrReshaperrrrrrFrvrrrrrrrr)rr out_shape ori_dtyperrr*r*r+rs"     zLinear.constructcCs|jj||d|jr"|j||jrt|jtr|jdkr^|j j |d|dfn|jdkr|j j |ddf|j j ||j j |ddf|j j||j j|n,|jdkrtdnt|j |j|n|jrt|j trt|j dr|r|d}t|d krHt|d|d d }n2t|d krrt|d|d |d d }ntd|j |ntd|j|S)a Set the shard for the linear. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy_matmul (tuple): The strategy for the matmul. Should be the same shape as the inputs. strategy_bias (tuple): The strategy for the bias_add. Should be the same shape as the inputs. strategy_activation (tuple): The strategy for the strategy_activation. Should be the same shape as the inputs. out_strategy_matmul (tuple): The out strategy for the matmul. Should be the same shape as the inputs. ) in_strategy out_strategy leakyrelur logsigmoidr* logsoftmaxzWThe 'LogSoftmax' function is not supported in semi auto parallel or auto parallel mode.activation_shardr^r[) data_parallelmodel_parallel)rexpert_parallelra'The user-defined activation function currently only supports the case where the input policy is 2 or 4, so that relevant policies can be extracted from it.To avoid this error, you need to add the function of extracting 'ParallelConfig' or 'OpParallelConfig' for the incoming strategy_activation zThe user passed the custom defined activation function %s. If the user want to enable shard for the activation cell, the user should set the shard for each primitives in the cell.)rrrrrrDrrlowerr select_oprexprreclogrGgetattrr hasattrrFrrrrwarning)rstrategy_matmul strategy_biasstrategy_activationout_strategy_matmulZ shard_tupleparallel_configr*r*r+r sB  z Linear.shard)NNN)rbrcrdrer r4rcheck_positive_int check_boolrArrrr~rrrr*r*rr+r{s82   <rc sjeZdZdZeejejejejejejeegdddde ffdd Z ddZ d d Z d d Z ZS) ra Fixed Sparse Attention Layer. This function contains the sparse attention primitives used in Sparse Transformers (see paper) `Generating Long Sequences with Sparse Transformers `_. Specifically, it includes the following: 1. A faster implementation of normal attention (the upper triangle is not computed, and many operations are fused). 2. An implementation of "strided" and "fixed" attention, as in the Sparse Transformers paper. Args: batch_size(int): Number of input batch size. num_heads(int): Number of attention heads. size_per_head(int): An integer determining embedding size of each attention head, only supports 64, 128 for now. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines the size of such blocks, Block X Block. Only supports 64 for now. seq_length(int): length of input sequence, only supports 1024 for now. Default 1024. num_different_global_patterns(int): An integer determining the number of different global attentions layouts. While global attention can be fixed by which block/s are representative of any local window, since there are multi-heads, each head can use a different global representative, only supports 4 for now. Default 4. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch size, sequence length, Embedding Size]): Sequence of queries to query the context. - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp32`, :class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, seq_length, hidden_size] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import FixedSparseAttention >>> from mindspore import Tensor >>> model = FixedSparseAttention(batch_size=2, ... num_heads=8, ... size_per_head=64, ... block_size=64) >>> q = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> k = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> v = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1024, 1024)), mstype.float32) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 1024, 512) ) batch_size num_heads size_per_head block_size seq_lengthnum_different_global_patternsrrc stt||j|j}} || dkr>td|d| d||dkrbtd|d|jd||_||_|||_||_ ||_ |||_ ||_ |d|_ t|_t|d| dff|_t|dddf|dddff|_t|dddfd f|_td gtjd |_||_d d g} |jdkr} t!|j D],}| d|dddkrd| | |f<qԐqd| } | |jdd|j ddf}t|d}tj"|tj#ddddddddf|jdd}||j|j d|jdddf}t|tj|_$tt%| tj|_&tt'(|j |_)t*|j|j | |ddf| |ddfd|ddf|dddff|_+t,|| dddddf|| dddddf|| ddff|_-t.d|dddff|_/t|dddfdf|_0t|dddff|_1t|dddff|_2t|| ddddff|_3t|| ddff|_4t5| |ddfdf|_6t7|ddff|_8dS)NrzThe number of heads z6 must be a multiple of parallel_config.model_parallel r]zThe batch_size z5 must be a multiple of parallel_config.data_parallel rr[r[r[r[grv@rzaFor 'FixedSparseAttention', the class variable 'seq_length' must be 1024, but got the value : {}.z_For 'FixedSparseAttention', the class variable 'block_size' must be 64, but got the value : {}.zpFor 'FixedSparseAttention', the class variable 'num_different_global_patterns' must be 4, but got the value : {}zhFor 'FixedSparseAttention', the class variable 'size_per_head' only supports {}, but got the value : {}.gir^rr[axisrr*)9r}rr~rrrGrr hidden_sizerr block_numr global_sizerrreshape Transposer transposer batch_matmulrmultiplyr rr multiply_datarrnprrrangerepeatnewaxis global_masktrillocal_mask_trianglemathr scale_factor MatmulDDS matmul_dds DSDMatmul matmul_dsdrrmul1 transpose1 transpose2 transpose3 transpose4rdiv StridedSliceslice1)rrrrrrrrdpmpZsize_per_head_listZ local_onesZglobal_mask_originalijZglobal_mask_fxrrr*r+r~s          $     2$   :zFixedSparseAttention.__init__c Cstt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j g|jtt|d|j|j|j|jgtt|dt j t j g|j| |||\}}}| |\}}||t|jt|}||t|jt|}|||||\}}||||} || d} t| d|j|j|jf} || d} t| d|j|j|jf} | S)zForward processqkvattention_mask)rr[r rr^r\rr^r[r )rsrrrYrrrrprvrrr_transpose_inputs_generate_attention_maskr)rrr!r#r'rrrr() rr0r1r2r3 local_maskr local_prob global_prob attentionattention_merger*r*r+rsB  zFixedSparseAttention.constructcCsn||d|j|jf}||d|jddf|j|j|jfd}||d|jf}t|}|d|jd|jf}|d|j|jdf}|||}|||}|||}t |j d} | || }| t tdtjt |tj} || |j} || d} || |j|j|j|jf} || |j|jdd|j|jddf} || d}|j}||fS) zd generate global attention mask and local attention mask from origin attention mask r\rr[r)g?r5r r )rrr+rrrrrr ExpandDimsrrrrrtuple_to_arrayrrr$rr%r&r)rr3 input_maskrHZlocal_shape_rightZlocal_shape_leftZlocal_mask_leftZlocal_mask_rightZlocal_attention_maskZlower_triangleZlocal_multiplied_outZ local_adderZlocal_mask_originalZ local_mask_fxr8rr*r*r+r7s>       z-FixedSparseAttention._generate_attention_maskc Cs|||dd|j|jddfd}|||dd|j|jddfd}|||dd|j|jddfd}|||fS)z4 do reshape and transpose to inputs r\r r )rr^r r[)rrrr)rr0r1r2r*r*r+r6s*z&FixedSparseAttention._transpose_inputs)rbrcrdrer4rrr@rrr~rr7r6rr*r*rr+rBs >  I #rcs,eZdZdZeffdd ZddZZS) AlibiTensor Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) c stt||j}||_||_tt| t j |_ t |dff|_t |ddff|_t |dff|_t |dfdf|_t |df|dff|_t d|ddff|_dtt|}tjddt|d  tj d}tjdd|tjd}t||}||krtjddtd|d  tj d} t|||} tjddd| dtjd} tj |t| | gdd}t|dddft j |_!dS) Nr[r)r[r[r^r rrr )"r}r@r~rrrr rrrr minus_onerr=r expand_2d expand_3dCumSumcumsumrrrr mul_sloperfloorlog2arrayarangeint32powermin concatenateslopes) rrrrr,closest_power_of_2basepowersrP extra_basenum_remaining_heads extra_powersrr*r+r~<s.&   zAlibiTensor.__init__cCsV||d}|||j}|||}||d}||j|}||d|}|S) Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) r\r[r^) rFrrBrrCrGrPrDastype)rr3rv arange_tensoralibir*r*r+r[s   zAlibiTensor.construct)rbrcrdrerr~rrr*r*rr+r@)sr@cs6eZdZdZfddZejfddZddZZ S) AlibiTensorV2rAc stt|||_||_tt| tj |_ t |_ t |_t |_t |_t |_t |_t |_t |_t |_t |_dtt|}tjddt|d  tj d}tj dd|tj!d}t"||}||krhtjddtd|d  tj d}t#|||}tj ddd|dtj!d} tj$|t"|| gdd}t|dddddftj |_%dS)Nr^r rr[rr )&r}r[r~rrr rrrrrBrr=rCrDrErFradd_2dadd_3drrrGrrrrmul_maskrrHrIrJrKrLrMrNrOrP) rrrrQrRrSrPrTrUrVrr*r+r~|s4          &   zAlibiTensorV2.__init__c Cs|j\}}||d}|ddddf }|||}||d}||d }|||}||d}||j|}| || ||d|df} | | | ||dd|f} | |S)rWr\Nr[rr^r[) rrFr\rCrr]rDrGrPr^rrX) rr3rvbsZseqlenrYZmax_posdiagrZZ alibi_maskr*r*r+rs      zAlibiTensorV2.constructcCs|j}|j}|j|dff|j|ddff|j|dff|j|df|dff|j|ddf|ddff|j|df|dff|j d|dddff|j |ddff|j ||ddf|dddffdS)z+Parallel strategy configuratiuon interface.r[)r[r[r[r[N) rrrCrrDrFr\r]rrGrr^)rrr,r-r*r*r+rszAlibiTensorV2.shard) rbrcrdrer~rrrrrr*r*rr+r[is "r[cCs\dd}t|r||Sdtt|}||td|dddd||S)z calculate slopes of alibi tensorcs6ddt|d  fddt|DS)Nr^r csg|]}|qSr*r*)rOr.ratiostartr*r+rPrQzG_get_interleave.._get_interleave_power_of_2..)rrIr)nr*rbr+_get_interleave_power_of_2sz3_get_interleave.._get_interleave_power_of_2r^rN)rrI is_integerrH_get_interleave)rerfrQr*r*r+rhs rhmscCs|dvsJd|dt|}tt|dd}t||d}tt|dd}t|||df}t|d}tt|dd}t|d}||}||}t|d}|dkr|St||dS) zbuild alibi tensor)rriz%return tensors must be 'np' or 'ms', z not support.r[rr_rr)rhr expand_dimsrKtilerarr )seq_lenrZreturn_tensorsrvrPZposition_pointrarZr*r*r+build_alibi_tensor_v2s"    rm)Hre __future__r functoolsrrr-rnumpyr mindsporerrmindspore.common.parameterrmindspore.common.initializerrr mindspore.common.dtypecommonrvrmindspore.common.seedr Zmindspore._extendsr mindspore.nn.cellr mindspore.opsr rrrmindspore.ops.primitivermindspore._checkparamr ImportError _checkparammindspore.parallel._utilsrrmindspore.contextrmindformers.tools.loggerrZmindformers.modules.activationr2mindformers.modules.transformer.op_parallel_configrrr__all__r4r@rArBrlrprqrsrtrurrrrr@r[rhrrmr*r*r*r+sf              V      ;WHh@Wlocal_block_sparse_attention.cpython-39.pyc000066400000000000000000000256601476650141400346100ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/__pycache__a ?cfN8@sdZddlmZddlmZmZddlZddlZddlZ ddl Z ddl m Z ddlmZddl mZddl mZddlmZzddlmZWneyddlmZYn0dd lmZmZd gZd adada d d Z!ddZ"ddZ#GdddZ$ddZ%Gdd d e j&Z'dS)z! A Local Block Sparse Attention. )absolute_import)wrapspartialN)Tensordtype)cell_attr_register) Validator)default_dpmp_configOpParallelConfigLocalBlockSparseAttentionFcsfdd}|S)z)Check whether input data type is correct.cs8tjijtfdd}|S)Ncslj|i|}|j}dvr&dd|vr6|d}|D]\}}|vr>|||q>|i|S)Nkwargs)bind argumentsitems)argsr bound_values argument_dictnamevalue bound_typesfuncsig[/home/ma-user/work/mindformers-baichuan/mindformers/modules/local_block_sparse_attention.pywrapper4sz?_args_type_validator_check..type_check..wrapper)inspect signature bind_partialrr)rr type_args type_kwargsrr type_check0s  z._args_type_validator_check..type_checkr)r!r"r#rr r_args_type_validator_check-sr$csfdd}|S)zWtypes should be a list of types, this function check if the type is in the valid dtypescsttjd}||t|SN) valid_types prim_name)rr check_type_nametyperr partial_check class_nametypesrrvalidator_check_funcJs z0_valid_type_checks..validator_check_funcrr.r-r/rr,r_valid_type_checksHsr1csfdd}|S)z\the value should be a list of types, this function check if the value is in the valid dtypescsttjd}|||Sr%)rr r(r*r,rrr/Ws z1_valid_value_checks..validator_check_funcrr0rr,r_valid_value_checksUsr2c@s(eZdZdZddZddZddZdS) InitGatherIndexa% A self-defined function to init the index of gather operation Args: seq_len(int): Length of input sequence. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference block sparse attention in FlashAttention cCs>||_||_||_|j|j|_|j|j|_|jd|_dS)N)seq_len local_size block_size group_numlocal_block_numprev_block_num)selfr5r6r7rrr__init__ms zInitGatherIndex.__init__cCst|j|j}t|jD]`}td||j|j}td||j|j|j|j}t|||||j|d|j<q|S)Nrr4) nponesr8r6rangemaxr:r7aranger;indexistartendrrrgen_kv_gather_indexus z#InitGatherIndex.gen_kv_gather_indexcCstd|j|jf}t|jD]h}td||j|j}td||j|j |j}t |||dd||j|d|jddf<q|S)Nr4r) r=r>r5r6r?r8r@r:r7r9rArBrrrgen_mask_gatherd_indexs z&InitGatherIndex.gen_mask_gatherd_indexN)__name__ __module__ __qualname____doc__r<rGrHrrrrr3bs  r3cCs0t|||}t|tjat|tjadS)N)r3rrGmstypeint32kv_indexrH mask_index)r5r6r7 gather_indexrrrinit_gather_indexs rRc sjeZdZdZeeejejejejee j e j gde e gddde j effdd Zd ddZZS) r a Local Block Sparse Attention Layer. This function contains the sliding window attention primitives used in Longformer (see paper) `Longformer: The Long-Document Transformer ` and block sparse attention primitives used in Longformer FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. A faster implementation of normal attention (the entire area of the upper triangle and part of the area of the lower triangle are not computed, and many operations are fused). 2. An implementation of "local" and "block" sparse attention, as in the Longformer and FlashAttention. Args: seq_len(int): Length of input sequence. size_per_head(int): The hidden size of the input. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference sliding window attention in FlashAttention. dropout_rate(float): The dropout rate of the attention score. Default 0.1. softmax_compute_type(mstype.Number): The type of softmax computation module. Default mstype.float16. Should be mstype.float32 or mstype.float16. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, size_per_head] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import LocalBlockSparseAttention >>> from mindspore import Tensor >>> model = LocalBlockSparseAttention(seq_len=4096, ... size_per_head=128, ... local_size=1024, ... block_size=128) >>> q = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> k = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> v = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 4096, 4096)), mstype.float16) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) )r5 size_per_headr6r7softmax_compute_typeparallel_configg?c srtt|||dkr.td|d|||dkrNtd|d|||krjtd|d|||krtd|d ||dks|d krtd |tst|||d attt||_ ||_ ||_ ||_ ||_ |j}|j} t|| d d fd f|_tjjd d|| d d d f|| d d d ff|_t|d d fdf|_t|d d ff|_t|d d d d f|| d d d ff|_t|| d d d ff|_|j dkrt j!d |j d|_"|j"j#|| d d d fftj|| d d d f|| d d d ff|_$tdg|j d|_%t&|_'t(|| d d fdf|_)t*|d d f|d d ff|_+dS)Nrz) keep_probgr),superr r< ValueErrorgather_index_initedrRrmathsqrt scale_factorr6r7 dropout_raterT data_parallelmodel_parallelPRealDivshardreal_divmsops BatchMatMulqk_bmmMul attn_mask_mul ExpandDimsattn_mask_expand_dimsAdd attn_mask_addSoftmaxsoftmaxnnDropoutattention_dropoutdropoutpv_bmm multiply_dataReshapereshapeGathergather1GatherDgather2) r;r5rSr6r7r_rTrUdpmp __class__rrr<s       z"LocalBlockSparseAttention.__init__NcCs|j\}}}}t|jt|} ||| }||| }||||||j|j|f}||t d}||||d|j |f}||t d}||||d|j |f}| ||} t| } t| |j } |durFt t|ddf} ||d| }|t||j |j} || d} || |dd|j|j f} || | } || }t|| }|jdkrt||}|||}||||||f}|S)zForward processNr4rW)shaperbCastr^DTypereryr7r{rOr6rirTtilerPr}rkrwrmrorqr_rtrv)r;qkvattention_maskbszhead_numr5head_dimfactorscore ori_dtypeZ mask_index_adderprobsoutputrrr constructsB           z#LocalBlockSparseAttention.construct)N)rIrJrKrLrr$r check_positive_intr2rMfloat16float32r1r r r<r __classcell__rrrrr s&: A)(rL __future__r functoolsrrrr\numpyr= mindsporerf mindspore.nnrr mindspore.opsrgrbrrrMmindspore._extendsrZmindspore._checkparamr ImportError _checkparam2mindformers.modules.transformer.op_parallel_configr r __all__r[rOrPr$r1r2r3rRCellr rrrrs4         (pengcheng.mind-mindformer/mindformers/modules/activation.py000066400000000000000000001422701476650141400246300ustar00rootroot00000000000000# Copyright 2020-2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """activation""" from __future__ import absolute_import import numpy as np from mindspore import _checkparam as validator from mindspore._extends import cell_attr_register from mindspore.common import dtype as mstype from mindspore.common.parameter import Parameter from mindspore.common.tensor import Tensor from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.operations import nn_ops as NN_OPS from mindspore.nn.cell import Cell from mindspore import ops from mindspore.ops.primitive import _primexpr __all__ = ['Softmin', 'Softmax', 'Softmax2d', 'LogSoftmax', 'ReLU', 'ReLU6', 'RReLU', 'SeLU', 'SiLU', 'Tanh', 'Tanhshrink', 'Hardtanh', 'GELU', 'FastGelu', 'Sigmoid', 'Softsign', 'PReLU', 'get_activation', 'LeakyReLU', 'HSigmoid', 'HSwish', 'ELU', 'LogSigmoid', 'LRN', 'SoftShrink', 'HShrink', 'CELU', 'Threshold', 'Mish', 'GLU' ] class CELU(Cell): r""" Continuously differentiable exponential linear units activation function. Applies the continuously differentiable exponential linear units function element-wise. .. math:: \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) CELU Activation Function Graph: .. image:: images/CELU.png :align: center Args: alpha (float): The :math:`\alpha` value for the Celu formulation. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of CELU. The required dtype is float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. ValueError: If `alpha` has the value of 0. TypeError: If `x` is not a Tensor. TypeError: If the dtype of 'input_x' is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-2.0, -1.0, 1.0, 2.0]), mindspore.float32) >>> celu = nn.CELU() >>> output = celu(x) >>> print(output) [-0.86466473 -0.63212055 1. 2. ] """ def __init__(self, alpha=1.0): """Initialize CELU.""" super(CELU, self).__init__() self.celu = P.CeLU(alpha=alpha) def construct(self, x): return self.celu(x) class Softmin(Cell): r""" Softmin activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, and the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmin is defined as: .. math:: \text{softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_{j=0}^{n-1}\exp(-x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmin operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`. -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - Tensor for computing Softmin functions with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range [0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in the range [-x.ndim, x.ndim). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmin = nn.Softmin() >>> output = softmin(x) >>> print(output) [0.2341 0.636 0.0862 0.01165 0.03168 ] """ def __init__(self, axis=-1): """Initialize Softmin.""" super(Softmin, self).__init__() self.axis = axis def construct(self, x): return ops.function.softmin(x, self.axis) class Softmax2d(Cell): r""" Softmax function applied to 2D features data. Applies `Softmax` to each location :math:`(c, h, w)` with an input Tensor of shape :math:`(C, H, W)` . Inputs: - **x** (Tensor) - Tensor of shape :math:`(N, C_{in}, H_{in}, W_{in})` or :math:`(C_{in}, H_{in}, W_{in})`. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `data_format` is neither 'NCHW' nor 'CHW'. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.2]], [[0.3, 0.4]], [[0.6, 0.5]]]]), mindspore.float32) >>> softmax2d = nn.Softmax2d() >>> output = softmax2d(x) >>> print(output) [[[[0.25838965 0.28001308]] [[0.31559783 0.34200877]] [[0.42601252 0.37797815]]]] """ def __init__(self): """Initialize Softmax2d.""" super(Softmax2d, self).__init__() self.softmax = P.Softmax(axis=-3) self.shape = P.Shape() @staticmethod @_primexpr def _check_input_dim(shape, cls_name): dim = len(shape) if dim not in (3, 4): raise ValueError(f"For '{cls_name}', the in_shape must have 3 or 4 dims, but got {dim}.") def construct(self, x): x_shape = self.shape(x) self._check_input_dim(x_shape, self.cls_name) return self.softmax(x) class Softmax(Cell): r""" Softmax activation function, which is a two-category function :class:`mindspore.nn.Sigmoid` in the promotion of multi-classification, the purpose is to show the results of multi-classification in the form of probability. Calculate the value of the exponential function for the elements of the input Tensor on the `axis`, and then normalized to lie in range [0, 1] and sum up to 1. Softmax is defined as: .. math:: \text{softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_{j=0}^{n-1}\exp(x_j)}, where :math:`x_{i}` is the :math:`i`-th slice in the given dimension of the input Tensor. Args: axis (Union[int, tuple[int]]): The axis to apply Softmax operation, if the dimension of input `x` is x.ndim, the range of axis is `[-x.ndim, x.ndim)`, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of Softmax with data type of float16 or float32. Outputs: Tensor, which has the same type and shape as `x` with values in the range[0,1]. Raises: TypeError: If `axis` is neither an int nor a tuple. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is a tuple whose length is less than 1. ValueError: If `axis` is a tuple whose elements are not all in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> # axis = -1(default), and the sum of return value is 1.0. >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> softmax = nn.Softmax() >>> output = softmax(x) >>> print(output) [0.03168 0.01166 0.0861 0.636 0.2341 ] """ def __init__(self, axis=-1): """Initialize Softmax.""" super(Softmax, self).__init__() self.softmax = P.Softmax(axis) def construct(self, x): return self.softmax(x) class LogSoftmax(Cell): r""" Applies the LogSoftmax function to n-dimensional input tensor. The input is transformed by the Softmax function and then by the log function to lie in range[-inf,0). Logsoftmax is defined as: .. math:: \text{logsoftmax}(x_i) = \log \left(\frac{\exp(x_i)}{\sum_{j=0}^{n-1} \exp(x_j)}\right), Args: axis (int): The axis to apply LogSoftmax operation, -1 means the last dimension. Default: ``-1`` . Inputs: - **x** (Tensor) - The input of LogSoftmax, with float16 or float32 data type. Outputs: Tensor, which has the same type and shape as `x` with output values in the range[-inf,0). Raises: TypeError: If `axis` is not an int. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `axis` is not in range [-len(x), len(x)). Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> log_softmax = nn.LogSoftmax() >>> output = log_softmax(x) >>> print(output) [[-5.00672150e+00 -6.72150636e-03 -1.20067215e+01] [-7.00091219e+00 -1.40009127e+01 -9.12250078e-04]] """ def __init__(self, axis=-1): """Initialize LogSoftmax.""" super(LogSoftmax, self).__init__() self.log_softmax = P.LogSoftmax(axis) def construct(self, x): return self.log_softmax(x) class ELU(Cell): r""" Exponential Linear Unit activation function. Applies the exponential linear unit function element-wise. The activation function is defined as: .. math:: E_{i} = \begin{cases} x_i, &\text{if } x_i \geq 0; \cr \alpha * (\exp(x_i) - 1), &\text{otherwise.} \end{cases} where :math:`x_i` represents the element of the input and :math:`\alpha` represents the `alpha` parameter. ELU Activation Function Graph: .. image:: images/ELU.png :align: center Args: alpha (float): The alpha value of ELU, the data type is float. Default: ``1.0`` . Inputs: - **x** (Tensor) - The input of ELU is a Tensor of any dimension with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If `alpha` is not equal to 1.0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float32) >>> elu = nn.ELU() >>> result = elu(x) >>> print(result) [-0.63212055 -0.86466473 0. 2. 1.] """ def __init__(self, alpha=1.0): """Initialize ELU.""" super(ELU, self).__init__() self.elu = P.Elu(alpha) def construct(self, x): return self.elu(x) class ReLU(Cell): r""" Rectified Linear Unit activation function. .. math:: \text{ReLU}(x) = (x)^+ = \max(0, x), It returns element-wise :math:`\max(0, x)`. Specially, the neurons with the negative output will be suppressed and the active neurons will stay the same. ReLU Activation Function Graph: .. image:: images/ReLU.png :align: center Inputs: - **x** (Tensor) - The input of ReLU is a Tensor of any dimension. The data type is `number `_ . Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is not a number. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> relu = nn.ReLU() >>> output = relu(x) >>> print(output) [0. 2. 0. 2. 0.] """ def __init__(self): """Initialize ReLU.""" super(ReLU, self).__init__() self.relu = P.ReLU() def construct(self, x): return self.relu(x) class ReLU6(Cell): r""" Compute ReLU6 activation function. ReLU6 is similar to ReLU with a upper limit of 6, which if the inputs are greater than 6, the outputs will be suppressed to 6. It computes element-wise as .. math:: Y = \min(\max(0, x), 6). The input is a Tensor of any valid shape. ReLU6 Activation Function Graph: .. image:: images/ReLU6.png :align: center Inputs: - **x** (Tensor) - The input of ReLU6 with data type of float16 or float32. Outputs: Tensor, which has the same type as `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> relu6 = nn.ReLU6() >>> output = relu6(x) >>> print(output) [0. 0. 0. 2. 1.] """ def __init__(self): """Initialize ReLU6.""" super(ReLU6, self).__init__() self.relu6 = P.ReLU6() def construct(self, x): return self.relu6(x) class LeakyReLU(Cell): r""" Leaky ReLU activation function. The activation function is defined as: .. math:: \text{leaky_relu}(x) = \begin{cases}x, &\text{if } x \geq 0; \cr {\alpha} * x, &\text{otherwise.}\end{cases} where :math:`\alpha` represents the `alpha` parameter. For more details, see `Rectifier Nonlinearities Improve Neural Network Acoustic Models `_. LeakyReLU Activation Function Graph: .. image:: images/LeakyReLU.png :align: center Args: alpha (Union[int, float]): Slope of the activation function at x < 0. Default: ``0.2`` . Inputs: - **x** (Tensor) - The input of LeakyReLU is a Tensor of any dimension. Outputs: Tensor, has the same type and shape as the `x`. Raises: TypeError: If `alpha` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> leaky_relu = nn.LeakyReLU() >>> output = leaky_relu(x) >>> print(output) [[-0.2 4. -1.6] [ 2. -1. 9. ]] """ def __init__(self, alpha=0.2): """Initialize LeakyReLU.""" super(LeakyReLU, self).__init__() validator.check_value_type('alpha', alpha, [float, int], self.cls_name) self.greater_equal = P.GreaterEqual() self.mul = P.Mul() self.alpha = alpha self.select_op = P.Maximum() if self.alpha > 1: self.select_op = P.Minimum() def construct(self, x): alpha_array = P.Cast()(F.scalar_to_tensor(self.alpha), P.DType()(x)) out = self.select_op(alpha_array * x, x) return out class RReLU(Cell): r""" Randomized Leaky ReLU activation function. The activation function is defined as: .. math:: \text{RReLU}(x_{ji}) = \begin{cases}x_{ji}, &\text{if } x_{ji} \geq 0; \cr {\alpha_{ji}} * x_{ji}, &\text{otherwise.}\end{cases} where :math:`\alpha_{ji}` ~ :math:`U(l, u)`, :math:`l \le u`. Applies the RReLU function elementally, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolution Network `_ . Args: lower (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 8`` . upper (Union[int, float]): Slope of the activation function at x < 0. Default: ``1 / 3`` . Inputs: - **x** (Tensor) - The input of RReLU is a Tensor of any dimension. Outputs: Tensor, after RReLU, has the same type and shape as the `x`. Raises: TypeError: If `lower` is not a float or an int. TypeError: If `upper` is not a float or an int. TypeError: If `x` is not a Tensor. TypeError: If `x` is not a Tensor of mindspore.float16 or mindpore.float32. ValueError: If `lower` is greater than upper. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0], [2.0, 0]]), mindspore.float32) >>> r_relu = nn.RReLU() >>> output = r_relu(x) >>> print(output) [[-0.31465699 4. ] [ 2. 0. ]] """ def __init__(self, lower=1 / 8, upper=1 / 3): super(RReLU, self).__init__() validator.check_value_type('upper', upper, [float, int], self.cls_name) validator.check_value_type('lower', lower, [float, int], self.cls_name) if lower > upper: raise ValueError(f"For {self.cls_name}, the value of 'upper' must be greater than or equal to 'lower', " f"but got upper: {upper}, lower: {lower}. ") self.lower = Tensor(lower, dtype=mstype.float32) self.upper = Tensor(upper, dtype=mstype.float32) self.sign = P.Sign() def construct(self, x): """RReLU construct.""" if not isinstance(x, Tensor): raise TypeError(f"For 'rrelu', the input must be a Tensor, but got {type(x)}.") size = x.shape dtype = x.dtype sign_matrix = self.sign(x) negative_filter = sign_matrix.clip(None, 0) positive_filter = sign_matrix.clip(0, None) mask = ops.uniform(size, self.lower, self.upper).astype(dtype) negative_mask = negative_filter * mask * -1 total_mask = negative_mask + positive_filter out = total_mask * x return out class SeLU(Cell): r""" Activation function SeLU (Scaled exponential Linear Unit). SeLU Activation Function Graph: .. image:: images/SeLU.png :align: center Refer to :func:`mindspore.ops.selu` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> selu = nn.SeLU() >>> output = selu(input_x) >>> print(output) [[-1.1113307 4.202804 -1.7575096] [ 2.101402 -1.7462534 9.456309 ]] """ def __init__(self): """Initialize SeLU""" super(SeLU, self).__init__() self.selu = P.SeLU() def construct(self, input_x): return self.selu(input_x) class SiLU(Cell): r""" Sigmoid Linear Unit activation function. Applies the silu linear unit function element-wise. .. math:: \text{SiLU}(x) = x * \sigma(x), where :math:`x_i` is input, :math:`\sigma(x)` is Sigmoid function. .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, SiLU Activation Function Graph: .. image:: images/SiLU.png :align: center Inputs: - **x** (Tensor) - Input with the data type float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, 2, -3, 2, -1]), mindspore.float16) >>> silu = nn.SiLU() >>> output = silu(x) >>> print(output) [-0.269 1.762 -0.1423 1.762 -0.269] """ def construct(self, x): return ops.function.silu(x) class Tanh(Cell): r""" Applies the Tanh function element-wise, returns a new tensor with the hyperbolic tangent of the elements of input, The input is a Tensor with any valid shape. Tanh function is defined as: .. math:: tanh(x_i) = \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = \frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Tanh Activation Function Graph: .. image:: images/Tanh.png :align: center Inputs: - **x** (Tensor) - Tensor of any dimension, input with data type of float16 or float32. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), mindspore.float16) >>> tanh = nn.Tanh() >>> output = tanh(x) >>> print(output) [0.7617 0.964 0.995 0.964 0.7617] """ def __init__(self): """Initialize Tanh.""" super(Tanh, self).__init__() self.tanh = P.Tanh() def construct(self, x): return self.tanh(x) class Tanhshrink(Cell): r""" Tanhshrink activation function. The tanhshrink function is evaluated by element and returns a new tensor. Tanh function is defined as: .. math:: tanhshrink(x_i) =x_i- \frac{\exp(x_i) - \exp(-x_i)}{\exp(x_i) + \exp(-x_i)} = x_i-\frac{\exp(2x_i) - 1}{\exp(2x_i) + 1}, where :math:`x_i` is an element of the input Tensor. Inputs: - **x** (Tensor) - Tensor of any dimension. Outputs: Tensor, with the same shape as the `x`. Raises: TypeError: If `x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([1, 2, 3, 2, 1]), ms.float16) >>> tanhshrink = nn.Tanhshrink() >>> output = tanhshrink(x) >>> print(output) [0.2383 1.036 2.004 1.036 0.2383] """ def construct(self, x): return F.tanhshrink(x) class Hardtanh(Cell): r""" Applies the Hardtanh function element-wise. The activation function is defined as: .. math:: \text{Hardtanh}(x) = \begin{cases} 1, & \text{ if } x > 1; \\ -1, & \text{ if } x < -1; \\ x, & \text{ otherwise. } \end{cases} Linear region range :math:`[-1, 1]` can be adjusted using `min_val` and `max_val`. Hardtanh Activation Function Graph: .. image:: images/Hardtanh.png :align: center Note: On Ascend, data type of float16 might lead to accidental accuracy problem. Args: min_val (Union[int, float]): Minimum value of the linear region range. Default: ``-1.0`` . max_val (Union[int, float]): Maximum value of the linear region range. Default: ``1.0`` . Inputs: - **x** (Tensor) - Input Tensor with data type of float16 or float32. On CPU and Ascend support dimension 0-7D. On GPU support dimension 0-4D. Outputs: Tensor, with the same dtype and shape as `x`. Raises: TypeError: If `x` is not a Tensor. TypeError: If dtype of `x` is neither float16 nor float32. TypeError: If dtype of `min_val` is neither float nor int. TypeError: If dtype of `max_val` is neither float nor int. ValueError: If `min_val` is not less than `max_val`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hardtanh = nn.Hardtanh(min_val=-1.0, max_val=1.0) >>> output = hardtanh(x) >>> print(output) [-1. -1. 0. 1. 1.] """ def __init__(self, min_val=-1.0, max_val=1.0): """Initialize Hardtanh.""" super(Hardtanh, self).__init__() self.min_val = min_val self.max_val = max_val if self.min_val >= self.max_val: raise ValueError(f"For Hardtanh, min_val should be less than max_val," f"but got {self.min_val} and {self.max_val}") def construct(self, x): return F.hardtanh(x, self.min_val, self.max_val) class GELU(Cell): r""" Gaussian error linear unit activation function. Applies GELU function to each element of the input. The input is a Tensor with any valid shape. GELU is defined as: .. math:: GELU(x_i) = x_i*P(X < x_i), where :math:`P` is the cumulative distribution function of standard Gaussian distribution and :math:`x_i` is the element of the input. GELU Activation Function Graph: .. image:: images/GELU.png :align: center Args: approximate (bool): Whether to enable approximation. Default: ``True`` . If `approximate` is ``True``, The gaussian error linear activation is: :math:`0.5 * x * (1 + tanh(\sqrt(2 / \pi) * (x + 0.044715 * x^3)))` else, it is: :math:`x * P(X <= x) = 0.5 * x * (1 + erf(x / \sqrt(2)))`, where P(X) ~ N(0, 1). Inputs: - **x** (Tensor) - The input of GELU with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> gelu = nn.GELU() >>> output = gelu(x) >>> print(output) [[-1.5880802e-01 3.9999299e+00 -3.1077917e-21] [ 1.9545976e+00 -2.2918017e-07 9.0000000e+00]] >>> gelu = nn.GELU(approximate=False) >>> # CPU not support "approximate=False", using "approximate=True" instead >>> output = gelu(x) >>> print(output) [[-1.5865526e-01 3.9998732e+00 -0.0000000e+00] [ 1.9544997e+00 -1.4901161e-06 9.0000000e+00]] """ def __init__(self, approximate=True): """Initialize GELU.""" super(GELU, self).__init__() validator.check_bool(approximate, 'approximate', self.cls_name) self.approximate = approximate if self.approximate: self.gelu = P.GeLU() else: self.erf = P.Erf() self.sqrt = P.Sqrt() self.const0 = Tensor(0.5, mstype.float32) self.const1 = Tensor(1.0, mstype.float32) self.const2 = Tensor(2.0, mstype.float32) def construct(self, x): if self.approximate: return self.gelu(x) return x * F.cast(self.const0, x.dtype) * (F.cast(self.const1, x.dtype) + \ self.erf(x / self.sqrt(F.cast(self.const2, x.dtype)))) class FastGelu(Cell): r""" Fast Gaussian error linear unit activation function. Applies FastGelu function to each element of the input. The input is a Tensor with any valid shape. FastGelu is defined as: .. math:: FastGelu(x_i) = \frac {x_i} {1 + \exp(-1.702 * \left| x_i \right|)} * \exp(0.851 * (x_i - \left| x_i \right|)) where :math:`x_i` is the element of the input. FastGelu Activation Function Graph: .. image:: images/FastGelu.png :align: center Inputs: - **x** (Tensor) - The input of FastGelu with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> fast_gelu = nn.FastGelu() >>> output = fast_gelu(x) >>> print(output) [[-1.5418735e-01 3.9921875e+00 -9.7473649e-06] [ 1.9375000e+00 -1.0052517e-03 8.9824219e+00]] """ def __init__(self): """Initialize FastGelu.""" super(FastGelu, self).__init__() self.fast_gelu = P.FastGeLU() def construct(self, x): return self.fast_gelu(x) class Sigmoid(Cell): r""" Sigmoid activation function. Applies sigmoid-type activation element-wise. Sigmoid function is defined as: .. math:: \text{sigmoid}(x_i) = \frac{1}{1 + \exp(-x_i)}, where :math:`x_i` is the element of the input. Sigmoid Activation Function Graph: .. image:: images/Sigmoid.png :align: center Inputs: - **input_x** (Tensor) - Tensor of any dimension, the data type is float16, float32, float64, complex64 or complex128. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If dtype of `input_x` is not float16, float32, float64, complex64 or complex128. TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> sigmoid = nn.Sigmoid() >>> output = sigmoid(x) >>> print(output) [0.2688 0.11914 0.5 0.881 0.7305 ] """ def __init__(self): """Initialize Sigmoid.""" super(Sigmoid, self).__init__() self.sigmoid = P.Sigmoid() def construct(self, x): return self.sigmoid(x) class Softsign(Cell): r""" Softsign activation function. Softsign Activation Function Graph: .. image:: images/Softsign.png :align: center Refer to :func:`mindspore.ops.softsign` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([0, -1, 2, 30, -30]), mindspore.float32) >>> softsign = nn.Softsign() >>> output = softsign(x) >>> print(output) [ 0. -0.5 0.6666667 0.9677419 -0.9677419] """ def __init__(self): """Initialize Softsign.""" super(Softsign, self).__init__() self.softsign = P.Softsign() def construct(self, x): return self.softsign(x) class PReLU(Cell): r""" PReLU activation function. Applies the PReLU function element-wise. PReLU is defined as: .. math:: PReLU(x_i)= \max(0, x_i) + w * \min(0, x_i), where :math:`x_i` is an element of an channel of the input. Here :math:`w` is a learnable parameter with a default initial value 0.25. Parameter :math:`w` has dimensionality of the argument channel. If called without argument channel, a single parameter :math:`w` will be shared across all channels. PReLU Activation Function Graph: .. image:: images/PReLU.png :align: center Args: channel (int): The elements number of parameter :math:`w`. It could be an int, and the value is 1 or the channels number of input tensor `x`. Default: ``1`` . w (Union[float, list, Tensor]): The initial value of parameter. It could be a float, a float list or a tensor has the same dtype as the input tensor `x`. Default: ``0.25`` . Inputs: - **x** (Tensor) - The input of PReLU with data type of float16 or float32. The shape is :math:`(N, *)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same dtype and shape as the `x`. Raises: TypeError: If `channel` is not an int. TypeError: If `w` is not one of a float, a float list, a float Tensor. TypeError: If dtype of `x` is neither float16 nor float32. ValueError: If the `x` is a 0-D or 1-D Tensor on Ascend. ValueError: If `channel` is less than 1. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[[[0.1, 0.6], [0.9, 0.9]]]]), mindspore.float32) >>> prelu = nn.PReLU() >>> output = prelu(x) >>> print(output) [[[[0.1 0.6] [0.9 0.9]]]] """ @cell_attr_register(attrs="") def __init__(self, channel=1, w=0.25): """Initialize PReLU.""" super(PReLU, self).__init__() validator.check_positive_int(channel, 'channel', self.cls_name) if isinstance(w, (float, np.float32)): tmp = np.empty((channel,), dtype=np.float32) tmp.fill(w) w = Tensor(tmp, dtype=mstype.float32) elif isinstance(w, list): if len(w) != channel: raise ValueError(f"For '{self.cls_name}', the length of 'w' must be equal to the 'channel' when " f"the 'w' is a list, but got the length of 'w': {len(w)}, the 'channel': {channel}.") for i in w: if not isinstance(i, (float, np.float32)): raise ValueError(f"For '{self.cls_name}', all elements in 'w' must be " f"float when the 'w' is a list, but got {i}.") w = Tensor(w, dtype=mstype.float32) elif isinstance(w, Tensor): if w.dtype not in (mstype.float16, mstype.float32): raise ValueError(f"For '{self.cls_name}', the dtype of 'w' must be float16 or " f"float32 when the 'w' is a tensor, but got {w.dtype}.") if len(w.shape) != 1 or w.shape[0] != channel: raise ValueError(f"For '{self.cls_name}', the dimension of 'w' must be 1, and the elements number " f"should be equal to the 'channel' when the 'w' is a tensor, " f"but got 'w' shape {w.shape}, the 'channel' {channel}.") else: raise TypeError(f"For '{self.cls_name}', the 'w' only supported float, list and tensor, " f"but got {type(w).__name__}.") self.w = Parameter(w, name='a') self.prelu = P.PReLU() def construct(self, x): return self.prelu(x, F.cast(self.w, x.dtype)) class HSwish(Cell): r""" Applies hswish-type activation element-wise. The input is a Tensor with any valid shape. Hard swish is defined as: .. math:: \text{hswish}(x_{i}) = x_{i} * \frac{ReLU6(x_{i} + 3)}{6}, HSwish Activation Function Graph: .. image:: images/HSwish.png :align: center Inputs: - **x** (Tensor) - The input of HSwish, data type must be float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hswish = nn.HSwish() >>> result = hswish(x) >>> print(result) [-0.3333 -0.3333 0. 1.667 0.6665] """ def __init__(self): """Initialize HSwish.""" super(HSwish, self).__init__() self.hswish = P.HSwish() def construct(self, x): return self.hswish(x) class HSigmoid(Cell): r""" Hard sigmoid activation function. Calculates the output according to the input elements. Hard sigmoid is defined as: .. math:: \text{hsigmoid}(x_{i}) = \max(0, \min(1, \frac{x_{i} + 3}{6})), HSigmoid Activation Function Graph: .. image:: images/HSigmoid.png :align: center Inputs: - **input_x** (Tensor) - The input of HSigmoid. Tensor of any dimension. Outputs: Tensor, with the same type and shape as the `input_x`. Raises: TypeError: If `input_x` is not a Tensor. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([-1, -2, 0, 2, 1]), mindspore.float16) >>> hsigmoid = nn.HSigmoid() >>> result = hsigmoid(x) >>> print(result) [0.3333 0.1666 0.5 0.8335 0.6665] """ def __init__(self): """Initialize HSigmoid.""" super(HSigmoid, self).__init__() self.hsigmoid = P.HSigmoid() def construct(self, input_x): return self.hsigmoid(input_x) class LogSigmoid(Cell): r""" Applies logsigmoid activation element-wise. The input is a Tensor with any valid shape. Logsigmoid is defined as: .. math:: \text{logsigmoid}(x_{i}) = \log(\frac{1}{1 + \exp(-x_i)}), where :math:`x_{i}` is the element of the input. LogSigmoid Activation Function Graph: .. image:: images/LogSigmoid.png :align: center Inputs: - **x** (Tensor) - The input of LogSigmoid with data type of float16 or float32. The shape is :math:`(N,*)` where :math:`*` means, any number of additional dimensions. Outputs: Tensor, with the same type and shape as the `x`. Raises: TypeError: If dtype of `x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> net = nn.LogSigmoid() >>> x = Tensor(np.array([1.0, 2.0, 3.0]), mindspore.float32) >>> output = net(x) >>> print(output) [-0.31326166 -0.12692806 -0.04858734] """ def __init__(self): """Initialize LogSigmoid.""" super(LogSigmoid, self).__init__() self.mul = P.Mul() self.exp = P.Exp() self.add = P.Add() self.rec = P.Reciprocal() self.log = P.Log() def construct(self, input_x): neg_input = self.mul(input_x, -1) exp_neg_input = self.exp(neg_input) exp_neg_input_1 = self.add(exp_neg_input, 1) rec_exp_neg_input_1 = self.rec(exp_neg_input_1) ret = self.log(rec_exp_neg_input_1) return ret class LRN(Cell): r""" Local Response Normalization. Refer to :func:`mindspore.ops.lrn` for more details. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[[[0.1], [0.2]], ... [[0.3], [0.4]]]]), mindspore.float32) >>> output = nn.LRN()(input_x) >>> print(output) [[[[0.09534626] [0.1825742 ]] [[0.2860388 ] [0.3651484 ]]]] """ def __init__(self, depth_radius=5, bias=1.0, alpha=1.0, beta=0.5, norm_region="ACROSS_CHANNELS"): """Initialize LRN.""" super(LRN, self).__init__() self.lrn_op = NN_OPS.LRN(depth_radius, bias, alpha, beta, norm_region) def construct(self, input_x): return self.lrn_op(input_x) class SoftShrink(Cell): r""" Applies the SoftShrink function element-wise. .. math:: \text{SoftShrink}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} SoftShrink Activation Function Graph: .. image:: images/Softshrink.png :align: center Args: lambd (float): the :math:`\lambda` must be no less than zero for the SoftShrink formulation. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of SoftShrink with data type of float16 or float32. Any number of additional dimensions. Outputs: Tensor, has the same shape and data type as `input_x`. Raises: TypeError: If lambd is not a float. TypeError: If input_x is not a Tensor. TypeError: If dtype of input_x is neither float16 nor float32. ValueError: If lambd is less than 0. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5297, 0.7871, 1.1754], [ 0.7836, 0.6218, -1.1542]]), mindspore.float16) >>> softshrink = nn.SoftShrink() >>> output = softshrink(input_x) >>> print(output) [[ 0.02979 0.287 0.676 ] [ 0.2837 0.1216 -0.6543 ]] """ def __init__(self, lambd=0.5): super(SoftShrink, self).__init__() self.softshrink = P.SoftShrink(lambd) def construct(self, input_x): output = self.softshrink(input_x) return output class HShrink(Cell): r""" Hard Shrink activation function. Calculates the output according to the input elements. The formula is defined as follows: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} HShrink Activation Function Graph: .. image:: images/HShrink.png :align: center Args: lambd (float): The threshold :math:`\lambda` defined by the Hard Shrink formula. Default: ``0.5`` . Inputs: - **input_x** (Tensor) - The input of Hard Shrink with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `lambd` is not a float. TypeError: If dtype of `input_x` is neither float16 nor float32. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> input_x = Tensor(np.array([[ 0.5, 1, 2.0], [0.0533,0.0776,-2.1233]]), mindspore.float32) >>> hshrink = nn.HShrink() >>> output = hshrink(input_x) >>> print(output) [[ 0. 1. 2. ] [ 0. 0. -2.1233]] """ def __init__(self, lambd=0.5): super(HShrink, self).__init__() self.hshrink = P.HShrink(lambd) def construct(self, input_x): return self.hshrink(input_x) class Threshold(Cell): r"""Thresholds each element of the input Tensor. The formula is defined as follows: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold (Union[int, float]): The value to threshold at. value (Union[int, float]): The value to replace with when element is less than threshold. Inputs: - **input_x** (Tensor) - The input of Threshold with data type of float16 or float32. Outputs: Tensor, the same shape and data type as the input. Raises: TypeError: If `threshold` is not a float or an int. TypeError: If `value` is not a float or an int. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> m = nn.Threshold(0.1, 20) >>> inputs = Tensor([0.1, 0.2, 0.3], mindspore.float32) >>> outputs = m(inputs) >>> print(outputs) [ 20.0 0.2 0.3] """ def __init__(self, threshold, value): """Initialize Threshold.""" super(Threshold, self).__init__() self.threshold = threshold self.value = value def construct(self, input_x): return F.threshold(input_x, self.threshold, self.value) class Mish(Cell): r""" Computes MISH(A Self Regularized Non-Monotonic Neural Activation Function) of input tensors element-wise. Refer to :func:`mindspore.ops.mish` for more details. Mish Activation Function Graph: .. image:: images/Mish.png :align: center Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore >>> from mindspore import Tensor, nn >>> import numpy as np >>> x = Tensor(np.array([[-1.0, 4.0, -8.0], [2.0, -5.0, 9.0]]), mindspore.float32) >>> mish = nn.Mish() >>> output = mish(x) >>> print(output) [[-3.03401530e-01 3.99741292e+00 -2.68321624e-03] [ 1.94395900e+00 -3.35762873e-02 9.00000000e+00]] """ def __init__(self): """Initialize Mish.""" super().__init__("Mish") self.mish = NN_OPS.Mish() def construct(self, input_x): return self.mish(input_x) class GLU(Cell): r""" The gated linear unit function. .. math:: {GLU}(a, b)= a \otimes \sigma(b) where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Here :math:`\sigma` is the sigmoid function, and :math:`\otimes` is the Hadamard product. Args: axis (int): the axis to split the input. Default: ``-1`` , the last axis in `x`. Inputs: - **x** (Tensor) - :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions. Outputs: Tensor, the same dtype as the `x`, with the shape :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2`. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore as ms >>> m = ms.nn.GLU() >>> input = ms.Tensor([[0.1,0.2,0.3,0.4],[0.5,0.6,0.7,0.8]]) >>> output = m(input) >>> print(output) [[0.05744425 0.11973753] [0.33409387 0.41398472]] """ def __init__(self, axis=-1): """Initialize GLU.""" super().__init__("GLU") self.dim = axis self.spilt = P.Split(axis=axis, output_num=2) self.sigmoid = P.Sigmoid() def construct(self, x): x1, x2 = self.spilt(x) x2 = self.sigmoid(x2) return x1 * x2 _activation = { 'softmin': Softmin, 'softmax': Softmax, 'softmax2d': Softmax2d, 'logsoftmax': LogSoftmax, 'relu': ReLU, 'relu6': ReLU6, 'rrelu': RReLU, 'silu': SiLU, 'tanh': Tanh, 'tanhshrink': Tanhshrink, 'hardtanh': Hardtanh, 'gelu': GELU, 'fast_gelu': FastGelu, 'elu': ELU, 'sigmoid': Sigmoid, 'softsign': Softsign, 'prelu': PReLU, 'leakyrelu': LeakyReLU, 'hswish': HSwish, 'hsigmoid': HSigmoid, 'logsigmoid': LogSigmoid, 'softshrink': SoftShrink, 'hshrink': HShrink, 'threshold': Threshold, 'mish': Mish, } def get_activation(name, prim_name=None): """ Gets the activation function. Args: name (str): The name of the activation function. prim_name (Union[str, None]): The name of primitive. Default: ``None`` . Returns: Function, the activation function. Supported Platforms: ``Ascend`` ``GPU`` ``CPU`` Examples: >>> import mindspore.nn as nn >>> sigmoid = nn.get_activation('sigmoid') >>> print(sigmoid) Sigmoid<> """ msg_prefix = f"For '{prim_name}', the" if prim_name else "The" if name is None: return None if name not in _activation: raise KeyError(f"{msg_prefix} 'name' must be in {list(_activation.keys())}, but got {name}.") return _activation[name]() pengcheng.mind-mindformer/mindformers/modules/layers.py000066400000000000000000001371051476650141400237670ustar00rootroot00000000000000# Copyright 2021 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ The basic layer of the Transformer Networks. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from functools import wraps, partial import inspect import math import numpy as np from mindspore import nn, context from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer, Tensor import mindspore.common.dtype as mstype from mindspore.common.seed import _get_graph_seed from mindspore._extends import cell_attr_register from mindspore.nn.cell import Cell from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore.ops.primitive import constexpr # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindspore.context import ParallelMode from mindformers.tools.logger import logger from mindformers.modules.activation import get_activation from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, OpParallelConfig, MoEParallelConfig __all__ = [ "FixedSparseAttention", "Dropout", "LayerNorm", "Linear" ] def _args_type_validator_check(*type_args, **type_kwargs): """Check whether input data type is correct.""" def type_check(func): sig = inspect.signature(func) bound_types = sig.bind_partial(*type_args, **type_kwargs).arguments @wraps(func) def wrapper(*args, **kwargs): nonlocal bound_types bound_values = sig.bind(*args, **kwargs) argument_dict = bound_values.arguments if "kwargs" in bound_types: bound_types = bound_types["kwargs"] if "kwargs" in argument_dict: argument_dict = argument_dict["kwargs"] for name, value in argument_dict.items(): if name in bound_types: bound_types[name](value, name) return func(*args, **kwargs) return wrapper return type_check def _valid_type_checks(types, class_name): # types should be a list of types, this function check if the type is in the valid dtypes def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, type(value)) return validator_check_func def _valid_value_checks(types, class_name): # the value should be a list of types, this function check if the value is in the valid dtypes def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, value) return validator_check_func class _LayerInputCheck: """ A input check class for the inputs of the transformer model. """ @staticmethod def check_shape_length(input_shape, param_name, func_name, target_len): """ Check the input shape's length is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_len: the expected length of the shape. :return: """ if not isinstance(target_len, list): target_len = [target_len] matched = False for item in target_len: if len(input_shape) == item: matched = True if not matched: raise ValueError(f"{func_name} {param_name} shape length must be one of {target_len} dimension, " f"but got shape {input_shape}") return True @staticmethod def check_shape_equal(input_shape, param_name, func_name, target_shape): """ Check the input shape's is equal to the expected shape :param input_shape(list): a list of the tensor shapes. :param param_name(str): the name of the checked parameter. :param func_name(str): the name of the function. :param target_shape: the expected shape. :return: """ if not isinstance(target_shape[0], list): target_shape = [target_shape] if isinstance(input_shape, tuple): input_shape = list(input_shape) _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, [len(item) for item in target_shape]) matched = False for item in target_shape: if item == input_shape: matched = True break if not matched: raise ValueError(f"{func_name} {param_name} shape must be one of {target_shape}," f"but got {input_shape}") return True @staticmethod def check_shape_value_on_axis(input_shape, dim, param_name, cls_name, target_value): """ Check whether the input_shape[dim] is equal to target value""" if input_shape[dim] != target_value: raise ValueError(f"{cls_name} {param_name} at {dim} shape must be {target_value}," f"but got {input_shape[dim]}") return True @staticmethod def check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape): """ Check the input shape's is equal to the expected shape, the value on 0-th is viewed as batch, and the batch size will not be checked. """ target_shape = target_shape length, hidden = target_shape if isinstance(input_shape, tuple): input_shape = list(input_shape) _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, [len(target_shape), len(target_shape) + 1]) if input_shape[-1] != hidden: raise ValueError(f"For {func_name}, the last dimension of {param_name} shape must be {hidden}," f"but got the last dimension {input_shape[-1]} in {input_shape}.") if input_shape[0] == 0: raise ValueError(f"For {func_name}, the first dimension of {param_name} shape greater than 0," f"but got the first dimension {input_shape[0]} in {input_shape}.") if len(input_shape) == 2 and input_shape[0] % length != 0: raise ValueError(f"For {func_name}, the first dimension of {param_name} shape should be divisible " f"by {length}, " f"but got the first dimension {input_shape[0]} in {input_shape}.") return True @constexpr def _check_past_none_input_none(use_past, param_name, func_name, default_value, is_tensor, is_default): """ If the past is True, check whether the inputs is None""" if not use_past: if is_tensor: raise TypeError(f"{func_name} {param_name} must be {default_value}, if use_pat is False, but found " f"a tensor") if not is_default: raise TypeError(f"{func_name} {param_name} must be {default_value}, if use_pat is False.") else: if not is_tensor: raise TypeError(f"{func_name} {param_name} must be tensor, if use_pat is True") return True @constexpr def _check_input_dtype(input_dtype, param_name, allow_dtypes, cls_name): Validator.check_type_name(param_name, input_dtype, allow_dtypes, cls_name) @constexpr def _check_input_shape(input_shape, param_name, func_name, target_len): # check the input length _LayerInputCheck.check_shape_length(input_shape, param_name, func_name, target_len) @constexpr def _check_shape_equal(input_shape, param_name, func_name, target_shape): # check the input length _LayerInputCheck.check_shape_equal(input_shape, param_name, func_name, target_shape) @constexpr def _check_input_shape_value(input_shape, dim, param_name, cls_name, target_value): _LayerInputCheck.check_shape_value_on_axis(input_shape, dim, param_name, cls_name, target_value) @constexpr def _check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape): _LayerInputCheck.check_shape_equal_without_batch(input_shape, param_name, func_name, target_shape) class Dropout(nn.Cell): r""" A Dropout Implements with P.DropoutGenMask and P.DropoutDoMask for parallel training. """ def __init__(self, keep_prob=0.5, dtype=mstype.float32): super(Dropout, self).__init__() if keep_prob <= 0 or keep_prob > 1: raise ValueError( "dropout probability should be a number in range (0, 1], but got {}".format( keep_prob)) Validator.check_subclass("dtype", dtype, mstype.number_type, self.cls_name) Validator.check_value_type('keep_prob', keep_prob, [float], self.cls_name) self.keep_prob = keep_prob self.is_ascend = context.get_context('device_target') in ["Ascend"] if self.is_ascend: seed0, seed1 = _get_graph_seed(0, "dropout") self.seed0 = seed0 self.seed1 = seed1 self.dtype = dtype self.get_shape = P.Shape() self.dropout_gen_mask = P.DropoutGenMask(Seed0=self.seed0, Seed1=self.seed1) self.dropout_do_mask = P.DropoutDoMask() self.cast = P.Cast() else: self.dropout = P.Dropout(keep_prob) def construct(self, x): r""" Input: a tensor Returns: a tensor """ if not self.training: return x if self.keep_prob == 1: return x if not self.is_ascend: out, _ = self.dropout(x) return out shape = self.get_shape(x) dtype = P.DType()(x) keep_prob = self.cast(self.keep_prob, dtype) output = self.dropout_gen_mask(shape, keep_prob) return self.dropout_do_mask(x, output, keep_prob) def extend_repr(self): return 'keep_prob={}'.format(self.keep_prob) def shard(self, strategy): if self.is_ascend: self.dropout_gen_mask.shard(strategy) self.dropout_do_mask.shard(strategy) else: self.dropout.shard(strategy) class LayerNorm(Cell): r""" A self-defined layer norm operation using reduce sum and reduce mean Args: normalized_shape (tuple): The shape of the input tensor eps (float): The epsilon value of the denominator. Default 1e-5. param_init_type: The param init type. Inputs: - **x** (Tensor) - Tensor of shape :math:`(batch, seq\_length, hidden\_size)`. Outputs: Tensor of shape :math:`(batch, seq_length, hidden_size)`. """ def __init__(self, normalized_shape, eps=1e-5, param_init_type=mstype.float32, is_self_defined=False): super(LayerNorm, self).__init__() if param_init_type not in [mstype.float32, mstype.float16]: raise TypeError("The type of parameter 'param_init_type' should in [float32, float16], " "but got the type : {}.".format(type(param_init_type))) # Since the mindspore 1.10 version, the layernorm has been changed to P.LayerNorm self.is_self_defined = is_self_defined if not self.is_self_defined: self.layer_norm = P.LayerNorm(begin_norm_axis=-1, begin_params_axis=-1, epsilon=eps) self.gamma = Parameter(initializer('ones', normalized_shape, param_init_type), name="gamma", parallel_optimizer=False) self.beta = Parameter(initializer('zeros', normalized_shape, param_init_type), name="beta", parallel_optimizer=False) self.mean = P.ReduceMean(keep_dims=True) self.square = P.Square() self.sqrt = P.Sqrt() self.sub1 = P.Sub() self.sub2 = P.Sub() self.add = P.Add() self.eps = eps self.mul = P.Mul() self.add2 = P.Add() self.real_div = P.RealDiv() def construct(self, x): r""" x : batch x seq_length x hidden_size """ if self.is_self_defined: mean = self.mean(x, -1) diff = self.sub1(x, mean) variance = self.mean(self.square(diff), -1) variance_eps = self.sqrt(self.add(variance, self.eps)) output = self.real_div(diff, variance_eps) output = self.add2(self.mul(output, self.gamma), self.beta) else: output, _, _ = self.layer_norm(x, self.gamma, self.beta) return output def shard(self, strategy): r""" Set the shard for the layer norm. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy (tuple): The strategy for the dropout. Should be the same shape as the inputs. Examples: >>> import mindspore >>> net = mindformers.modules.transformer.LayerNorm(normalized_shape=(1024, 10)) >>> net.shard(((10, 2, 1),)) """ if self.is_self_defined: self.mean.shard(strategy) self.square.shard(strategy) self.sqrt.shard(strategy) self.sub1.shard((strategy[0], strategy[0])) self.sub2.shard((strategy[0], strategy[0])) self.add.shard((strategy[0], ())) self.mul.shard((strategy[0], (1,))) self.add2.shard((strategy[0], (1,))) self.real_div.shard((strategy[0], strategy[0])) else: self.layer_norm.shard((strategy[0], (1,), (1,))) return self class Linear(Cell): r""" The dense connected layer. Once the parallel mode is enabled, the input shape should be 3-D tensor. Applies dense connected layer for the input. This layer implements the operation as: .. math:: \text{outputs} = \text{activation}(\text{X} * \text{kernel} + \text{bias}), where :math:`X` is the input tensors, :math:`\text{activation}` is the activation function passed as the activation argument (if passed in), :math:`\text{kernel}` is a weight matrix with the same data type as the :math:`X` created by the layer, and :math:`\text{bias}` is a bias vector with the same data type as the :math:`X` created by the layer (only if has_bias is True). Args: in_channels (int): The number of channels in the input space. out_channels (int): The number of channels in the output space. weight_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable weight_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'normal'. bias_init (Union[Tensor, str, Initializer, numbers.Number]): The trainable bias_init parameter. The dtype is same as `x`. The values of str refer to the function `initializer`. Default: 'zeros'. has_bias (bool): Specifies whether the layer uses a bias vector. Default: True. activation (str): activate function applied to the output of the fully connected layer, eg. 'ReLU'. Default: None. expert_num (int): The number of experts used in this Linear. Here, for the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. outer_batch (int): The replication number of experts. The replication is effective only when MoE is applied. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. compute_dtype (dtype.Number): The computation type. Default: mstype.float16 Inputs: - **x** (Tensor) - Tensor of shape :math:`(*, in\_channels)`. The `in_channels` in `Args` should be equal to :math:`in\_channels` in `Inputs`. Outputs: Tensor of shape :math:`(*, out\_channels)`. Raises: TypeError: If `in_channels` or `out_channels` is not an int. TypeError: If `has_bias` is not a bool. TypeError: If `activation` is not one of str, Cell, Primitive, None. ValueError: If length of shape of `weight_init` is not equal to 2 or shape[0] of `weight_init` is not equal to `out_channels` or shape[1] of `weight_init` is not equal to `in_channels`. ValueError: If length of shape of `bias_init` is not equal to 1 or shape[0] of `bias_init` is not equal to `out_channels`. Supported Platforms: ``Ascend`` ``GPU`` """ @cell_attr_register @_args_type_validator_check(in_channels=Validator.check_positive_int, out_channels=Validator.check_positive_int, has_bias=Validator.check_bool, transpose_b=Validator.check_bool, expert_num=Validator.check_positive_int, outer_batch=Validator.check_positive_int, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Linear"), compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "Linear")) def __init__(self, in_channels, out_channels, weight_init='normal', bias_init='zeros', has_bias=True, activation=None, transpose_b=True, expert_num=1, outer_batch=1, expert_group_size=None, param_init_type=mstype.float32, compute_dtype=mstype.float16): super(Linear, self).__init__() self.in_channels = in_channels self.out_channels = out_channels if not (isinstance(activation, str) or activation is None or issubclass(activation, nn.Cell)): raise TypeError(f"For Linear cell, the activation should str type or nn.Cell type, but got {activation}.") if isinstance(weight_init, Tensor) and (weight_init.ndim != 2 or weight_init.shape[0] != out_channels or weight_init.shape[1] != in_channels): raise ValueError("The shape of parameter 'weight_init' is error, please check shape of 'weight_init'.") weight_shape = [out_channels, in_channels] if transpose_b else [in_channels, out_channels] self.expert_num = expert_num self.outer_batch = outer_batch self.expert_group_size = expert_group_size self.transpose_b = transpose_b if self.expert_num > 1: self.expert_flag = True self.weight = Parameter(initializer(weight_init, [self.expert_num] + weight_shape, param_init_type), name="weight") self.matmul = P.BatchMatMul(transpose_b=transpose_b) else: self.expert_flag = False self.weight = Parameter(initializer(weight_init, weight_shape, param_init_type), name="weight") self.matmul = P.MatMul(transpose_b=transpose_b) self.use_expert_group_size = _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) \ and not _is_sharding_propagation() and self.expert_flag is True if self.use_expert_group_size is True and self.expert_group_size is None: raise ValueError("'expert_group_size' should be configured as an integer in MoEConfig.") self.bias = None self.has_bias = has_bias if self.has_bias: if isinstance(bias_init, Tensor) and (bias_init.ndim != 1 or bias_init.shape[0] != out_channels): raise ValueError("The shape of parameter 'bias_init' is error, please check shape of 'bias_init'.") if self.expert_flag: self.bias = Parameter(initializer(bias_init, [1, self.expert_num, 1, out_channels], param_init_type), name="bias") else: self.bias = Parameter(initializer(bias_init, [out_channels], param_init_type), name="bias") self.bias.parallel_optimizer = False self.bias_add = P.Add() self.act_name = activation if callable(activation): self.activation = activation() else: self.activation = get_activation(activation) if isinstance(activation, str) else activation self.activation_flag = self.activation is not None self.dtype = compute_dtype self.cast = P.Cast() def construct(self, x): """Forward process, x should be a tensor""" out_shape = P.Shape()(x)[:-1] + (self.out_channels,) x = P.Reshape()(x, (-1, self.in_channels)) if self.expert_flag: if self.use_expert_group_size is True: x = P.Reshape()(x, (-1, self.expert_num, self.expert_group_size, self.in_channels)) else: x = P.Reshape()(x, (self.outer_batch, self.expert_num, -1, self.in_channels)) ori_dtype = F.dtype(x) weight = self.cast(self.weight, self.dtype) x = self.cast(x, self.dtype) x = self.matmul(x, weight) if self.has_bias: x = self.bias_add(x, self.cast(self.bias, self.dtype)) if self.activation_flag: x = self.activation(x) x = F.cast(x, ori_dtype) output = P.Reshape()(x, out_shape) return output def shard(self, strategy_matmul, strategy_bias=None, strategy_activation=None, out_strategy_matmul=None): r""" Set the shard for the linear. the strategy size should be equal to the inputs. Note: It is valid only in semi auto parallel or auto parallel mode. In other parallel modes, strategies set here will be ignored. Args: strategy_matmul (tuple): The strategy for the matmul. Should be the same shape as the inputs. strategy_bias (tuple): The strategy for the bias_add. Should be the same shape as the inputs. strategy_activation (tuple): The strategy for the strategy_activation. Should be the same shape as the inputs. out_strategy_matmul (tuple): The out strategy for the matmul. Should be the same shape as the inputs. """ self.matmul.shard(in_strategy=strategy_matmul, out_strategy=out_strategy_matmul) if self.has_bias: self.bias_add.shard(strategy_bias) if self.activation_flag and isinstance(self.act_name, str): # some operations has many primitives, need to manually set the shard if self.act_name.lower() == "leakyrelu": self.activation.select_op.shard((strategy_activation[0], strategy_activation[0])) elif self.act_name.lower() == "logsigmoid": self.activation.mul.shard((strategy_activation[0], ())) self.activation.exp.shard(strategy_activation) self.activation.add.shard((strategy_activation[0], ())) self.activation.rec.shard(strategy_activation) self.activation.log.shard(strategy_activation) elif self.act_name.lower() == "logsoftmax": raise ValueError("The 'LogSoftmax' function is not supported in semi auto parallel " "or auto parallel mode.") else: getattr(self.activation, self.act_name).shard(strategy_activation) elif self.activation_flag and isinstance(self.activation, Cell): if hasattr(self.activation, 'activation_shard') and strategy_activation: shard_tuple = strategy_activation[0] if len(shard_tuple) == 2: parallel_config = OpParallelConfig(data_parallel=shard_tuple[0], model_parallel=shard_tuple[1]) elif len(shard_tuple) == 4: parallel_config = MoEParallelConfig(data_parallel=shard_tuple[0], expert_parallel=shard_tuple[1], model_parallel=shard_tuple[2]) else: raise ValueError("The user-defined activation function currently only supports the case where the " "input policy is 2 or 4, so that relevant policies can be extracted from it." "To avoid this error, you need to add the function of extracting " "'ParallelConfig' or 'OpParallelConfig' for the incoming strategy_activation ") self.activation.activation_shard(parallel_config) else: logger.warning("The user passed the custom defined activation function %s. " "If the user want to enable shard for the activation cell, " "the user should set the shard for each primitives in the cell.", self.activation_flag) return self class FixedSparseAttention(nn.Cell): """ Fixed Sparse Attention Layer. This function contains the sparse attention primitives used in Sparse Transformers (see paper) `Generating Long Sequences with Sparse Transformers `_. Specifically, it includes the following: 1. A faster implementation of normal attention (the upper triangle is not computed, and many operations are fused). 2. An implementation of "strided" and "fixed" attention, as in the Sparse Transformers paper. Args: batch_size(int): Number of input batch size. num_heads(int): Number of attention heads. size_per_head(int): An integer determining embedding size of each attention head, only supports 64, 128 for now. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices. In which this parameter defines the size of such blocks, Block X Block. Only supports 64 for now. seq_length(int): length of input sequence, only supports 1024 for now. Default 1024. num_different_global_patterns(int): An integer determining the number of different global attentions layouts. While global attention can be fixed by which block/s are representative of any local window, since there are multi-heads, each head can use a different global representative, only supports 4 for now. Default 4. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, seq_length, hidden_size]): Sequence of queries to query the context. - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch size, sequence length, Embedding Size]): Sequence of queries to query the context. - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp32`, :class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, seq_length, hidden_size] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import FixedSparseAttention >>> from mindspore import Tensor >>> model = FixedSparseAttention(batch_size=2, ... num_heads=8, ... size_per_head=64, ... block_size=64) >>> q = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> k = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> v = Tensor(np.ones((2, 1024, 8*64)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1024, 1024)), mstype.float32) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 1024, 512) """ @_args_type_validator_check(batch_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, size_per_head=Validator.check_positive_int, block_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_different_global_patterns=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "FixedSparseAttention")) def __init__(self, batch_size, num_heads, size_per_head, block_size, seq_length=1024, num_different_global_patterns=4, parallel_config=default_dpmp_config): super(FixedSparseAttention, self).__init__() dp, mp = parallel_config.data_parallel, parallel_config.model_parallel if num_heads % mp != 0: raise ValueError(f"The number of heads {num_heads} must be a " f"multiple of parallel_config.model_parallel {mp}.") if batch_size % dp != 0: raise ValueError(f"The batch_size {batch_size} must be a " f"multiple of parallel_config.data_parallel {parallel_config.data_parallel}.") self.seq_length = seq_length self.batch_size = batch_size self.hidden_size = size_per_head * num_heads self.num_heads = num_heads self.block_size = block_size self.block_num = seq_length // block_size self.size_per_head = size_per_head self.global_size = seq_length // 4 self.reshape = P.Reshape() self.transpose = P.Transpose().shard(((dp, 1, mp, 1),)) self.batch_matmul = P.BatchMatMul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.multiply = P.Mul().shard(((dp, 1, 1, 1), (1, 1, 1))) self.multiply_data = Tensor([-10000.0], dtype=mstype.float32) self.parallel_config = parallel_config size_per_head_list = [64, 128] if self.seq_length != 1024: raise ValueError("For 'FixedSparseAttention', the class variable 'seq_length' must be 1024, " "but got the value : {}.".format(seq_length)) if self.block_size != 64: raise ValueError("For 'FixedSparseAttention', the class variable 'block_size' must be 64, " "but got the value : {}.".format(block_size)) if num_different_global_patterns != 4: raise ValueError("For 'FixedSparseAttention', the class variable 'num_different_global_patterns' " "must be 4, but got the value : {}".format(num_different_global_patterns)) if self.size_per_head not in size_per_head_list: raise ValueError("For 'FixedSparseAttention', the class variable 'size_per_head' only supports {}, " "but got the value : {}.".format(size_per_head_list, self.size_per_head)) local_ones = np.ones((self.block_size, self.block_size), dtype=np.float16) global_mask_original = np.ones((self.seq_length, self.global_size), dtype=np.float16) for i in range(self.seq_length): for j in range(self.global_size): if i // 16 >= (j // 16 + 1) * 4: global_mask_original[i, j] = 0.0 global_mask_original = -10000 * global_mask_original global_mask_fx = global_mask_original.reshape((self.seq_length // 16, 16, self.global_size // 16, 16)) global_mask = np.transpose(global_mask_fx, (2, 0, 1, 3)) global_mask = np.repeat(global_mask[np.newaxis, :, :, :, :], self.batch_size, axis=0) global_mask = global_mask.reshape((self.batch_size * self.global_size // 16, self.seq_length // 16, 16, 16)) self.global_mask = Tensor(global_mask, mstype.float32) self.local_mask_triangle = Tensor(np.tril(local_ones), mstype.float32) self.scale_factor = Tensor((math.sqrt(self.size_per_head))) self.matmul_dds = P.MatmulDDS(self.batch_size, self.num_heads).shard(((mp, dp, 1, 1), (mp, dp, 1, 1), (1, dp, 1, 1), (dp, 1, 1, 1))) self.matmul_dsd = P.DSDMatmul().shard(((dp, mp, 1, 1, 1, 1, 1), (dp, mp, 1, 1, 1, 1, 1), (dp, mp, 1, 1))) self.sub1 = P.Sub().shard(((1,), (dp, 1, 1, 1))) self.mul1 = P.Mul().shard(((dp, 1, 1, 1), (1,))) self.transpose1 = P.Transpose().shard(((dp, 1, 1, 1),)) self.transpose2 = P.Transpose().shard(((dp, 1, 1, 1),)) self.transpose3 = P.Transpose().shard(((dp, mp, 1, 1, 1, 1),)) self.transpose4 = P.Transpose().shard(((dp, mp, 1, 1),)) self.div = P.RealDiv().shard(((mp, dp, 1, 1), ())) self.slice1 = P.StridedSlice().shard(((dp, 1, 1),)) def construct(self, q, k, v, attention_mask): """Forward process""" _check_shape_equal(F.shape(q), "q", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(q), "q", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(k), "k", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(k), "k", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(v), "v", self.cls_name, [self.batch_size, self.seq_length, self.hidden_size]) _check_input_dtype(F.dtype(v), "v", [mstype.float16], self.cls_name) _check_shape_equal(F.shape(attention_mask), "attention_mask", self.cls_name, [self.batch_size, self.seq_length, self.seq_length]) _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) q, k, v = self._transpose_inputs(q, k, v) local_mask, global_mask = self._generate_attention_mask(attention_mask) q = self.div(q, F.cast(self.scale_factor, F.dtype(q))) k = self.div(k, F.cast(self.scale_factor, F.dtype(k))) local_prob, global_prob = self.matmul_dds(q, k, local_mask, global_mask) attention = self.matmul_dsd(local_prob, global_prob, v) attention_merge = self.transpose3(attention, (0, 1, 3, 4, 2, 5)) attention_merge = F.reshape( attention_merge, (-1, self.num_heads, self.seq_length, self.size_per_head)) attention_merge = self.transpose4(attention_merge, (0, 2, 1, 3)) attention_merge = F.reshape( attention_merge, (-1, self.seq_length, self.size_per_head * self.num_heads)) return attention_merge def _generate_attention_mask(self, attention_mask): """ generate global attention mask and local attention mask from origin attention mask """ attention_mask = self.reshape(attention_mask, (-1, self.seq_length, self.seq_length)) input_mask = self.slice1(attention_mask, (0, self.seq_length - 1, 0), (self.batch_size, self.seq_length, self.seq_length), (1, 1, 1)) input_mask = self.reshape(input_mask, (-1, self.seq_length)) input_shape = P.Shape()(input_mask) # bs, seq_length # bs, block_num, 1, block_size local_shape_right = (input_shape[0], self.block_num, 1, self.block_size) # bs, block_num, block_size, 1 local_shape_left = (input_shape[0], self.block_num, self.block_size, 1) local_mask_left = self.reshape(input_mask, local_shape_left) local_mask_right = self.reshape(input_mask, local_shape_right) # bs, block_num, block_size, block_size local_attention_mask = self.batch_matmul(local_mask_left, local_mask_right) lower_triangle = P.ExpandDims()(self.local_mask_triangle, 0) local_attention_mask = self.multiply(local_attention_mask, lower_triangle) local_multiplied_out = self.sub1(P.Cast()(F.tuple_to_array((1.0,)), mstype.float32), P.Cast()(local_attention_mask, mstype.float32)) local_adder = self.mul1(local_multiplied_out, self.multiply_data) local_mask_original = self.transpose1(local_adder, (0, 2, 1, 3)) local_mask_original = self.reshape( local_mask_original, (self.batch_size * self.block_size, self.block_num * self.block_size)) local_mask_fx = self.reshape( local_mask_original, (self.batch_size * self.block_size // 16, 16, self.block_num * self.block_size // 16, 16)) local_mask = self.transpose2(local_mask_fx, (2, 0, 1, 3)) global_mask = self.global_mask return local_mask, global_mask def _transpose_inputs(self, q, k, v): """ do reshape and transpose to inputs """ q = self.transpose( self.reshape( q, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (2, 0, 1, 3)) k = self.transpose( self.reshape( k, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (2, 0, 1, 3)) v = self.transpose( self.reshape( v, (-1, 16, self.num_heads * self.size_per_head // 16, 16)), (0, 2, 3, 1)) return q, k, v class AlibiTensor(nn.Cell): """ Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) """ def __init__(self, seq_length, num_heads, parallel_config=default_dpmp_config): super(AlibiTensor, self).__init__() dp = parallel_config.data_parallel self.seq_length = seq_length self.num_heads = num_heads self.minus_one = Tensor(-np.ones(seq_length), mstype.float32) self.expand_2d = P.ExpandDims().shard(((dp, 1),)) self.expand_3d = P.ExpandDims().shard(((dp, 1, 1),)) self.cumsum = P.CumSum().shard(((dp, 1),)) self.add = P.Add().shard(((dp, 1), (1,))) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul_slope = P.Mul().shard(((1, 1), (dp, 1, 1))) # build slopes closest_power_of_2 = 2 ** math.floor(math.log2(num_heads)) base = np.array(2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), dtype=np.float32) powers = np.arange(1, 1 + closest_power_of_2, dtype=np.int32) slopes = np.power(base, powers) if closest_power_of_2 != num_heads: extra_base = np.array( 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), dtype=np.float32 ) num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2) extra_powers = np.arange(1, 1 + 2 * num_remaining_heads, 2, dtype=np.int32) slopes = np.concatenate([slopes, np.power(extra_base, extra_powers)], axis=0) self.slopes = Tensor(slopes[:, None], mstype.float32) # (num_heads, 1) def construct(self, attention_mask, dtype): """ Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) """ arange_tensor = self.cumsum(attention_mask, -1) # (batch_size, seq_len) arange_tensor = self.add(arange_tensor, self.minus_one) # (batch_size, seq_len) arange_tensor = self.mul(arange_tensor, attention_mask) # (batch_size, seq_len) arange_tensor = self.expand_2d(arange_tensor, 1) # (batch_size, 1, seq_len) alibi = self.mul_slope(self.slopes, arange_tensor) # (batch_size, num_heads, seq_len) alibi = self.expand_3d(alibi, 2).astype(dtype) # (batch_size, num_heads, 1, seq_len) return alibi class AlibiTensorV2(nn.Cell): """ Link to paper: https://arxiv.org/abs/2108.12409 Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation: with l being a tensor, and a fixed value `softmax(l+a) = softmax(l)`. Based on https://github.com/ofirpress/attention_with_linear_biases/blob/a35aaca144e0eb6b789dfcb46784c4b8e31b7983/fairseq/models/transformer.py#L742 Args: seq_length(int) - length of sequence num_heads(int) - number of heads Inputs: attention_mask(Tensor) - Token-wise attention mask, this should be of shape (batch_size, max_seq_len). dtype(mstype) - dtype of the output tensor Returns: alibi(Tensor), ailibi tensor shaped (batch_size, num_heads, 1, max_seq_len) """ def __init__(self, seq_length, num_heads): super(AlibiTensorV2, self).__init__() self.seq_length = seq_length self.num_heads = num_heads self.minus_one = Tensor(-np.ones(seq_length), mstype.float32) self.expand_2d = P.ExpandDims() self.expand_3d = P.ExpandDims() self.cumsum = P.CumSum() self.add_2d = P.Add() self.add_3d = P.Add() self.mul = P.Mul() self.mul_slope = P.Mul() self.transpose = P.Transpose() self.reshape = P.Reshape() self.mul_mask = P.Mul() # build slopes closest_power_of_2 = 2 ** math.floor(math.log2(num_heads)) base = np.array(2 ** (-(2 ** -(math.log2(closest_power_of_2) - 3))), dtype=np.float32) powers = np.arange(1, 1 + closest_power_of_2, dtype=np.int32) slopes = np.power(base, powers) if closest_power_of_2 != num_heads: extra_base = np.array( 2 ** (-(2 ** -(math.log2(2 * closest_power_of_2) - 3))), dtype=np.float32 ) num_remaining_heads = min(closest_power_of_2, num_heads - closest_power_of_2) extra_powers = np.arange(1, 1 + 2 * num_remaining_heads, 2, dtype=np.int32) slopes = np.concatenate([slopes, np.power(extra_base, extra_powers)], axis=0) self.slopes = Tensor(slopes[None, :, None, None], mstype.float32) # (num_heads, 1) def construct(self, attention_mask, dtype=mstype.float32): """ Note: alibi will added to the attention bias that will be applied to the query, key product of attention therefore alibi will have to be of shape (batch_size, num_heads, query_length, key_length) """ bs, seqlen = attention_mask.shape arange_tensor = self.cumsum(attention_mask, -1) # (batch_size, seq_len) max_pos = -arange_tensor[:, -1:] # (batch_size, 1) arange_tensor = self.add_2d(arange_tensor, max_pos) # (batch_size, seq_len) arange_tensor = self.expand_2d(arange_tensor, 1) # (batch_size, 1, seq_len) diag = -self.transpose(arange_tensor, (0, 2, 1)) # (batch_size, seq_len, 1) arange_tensor = self.add_3d(arange_tensor, diag) # (batch_size, seq_len, seq_len) arange_tensor = self.expand_3d(arange_tensor, 1) # (batch_size, 1, seq_len, seq_len) alibi = self.mul_slope(self.slopes, arange_tensor) # (batch_size, num_heads, seq_len, seq_len) alibi_mask = self.mul_mask(alibi, self.reshape(attention_mask, (bs, 1, seqlen, 1))) # (batch_size, num_heads, seq_len, seq_len) alibi_mask = self.mul_mask(alibi_mask, self.reshape(attention_mask, (bs, 1, 1, seqlen))) # (batch_size, num_heads, seq_len, seq_len) return alibi_mask.astype(dtype) def shard(self, parallel_config): """Parallel strategy configuratiuon interface.""" dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.expand_2d.shard(((dp, 1),)) self.expand_3d.shard(((dp, 1, 1),)) self.cumsum.shard(((dp, 1),)) self.add_2d.shard(((dp, 1), (dp, 1))) self.add_3d.shard(((dp, 1, 1), (dp, 1, 1))) self.mul.shard(((dp, 1), (dp, 1))) self.mul_slope.shard(((1, 1, 1, 1), (dp, 1, 1, 1))) self.transpose.shard(((dp, 1, 1),)) self.mul_mask.shard(((dp, mp, 1, 1), (dp, 1, 1, 1))) def _get_interleave(n): """calculate slopes of alibi tensor""" def _get_interleave_power_of_2(n): start = (2 ** (-2 ** -(math.log2(n) - 3))) ratio = start return [start * ratio ** i for i in range(n)] if math.log2(n).is_integer(): return _get_interleave_power_of_2(n) closest_power_of_2 = 2 ** math.floor(math.log2(n)) return _get_interleave_power_of_2(closest_power_of_2) + \ _get_interleave(2 * closest_power_of_2)[0::2][:n - closest_power_of_2] def build_alibi_tensor_v2(seq_len, num_heads, return_tensors='ms', dtype=mstype.float32): """build alibi tensor""" assert return_tensors in ['np', 'ms'],\ f"return tensors must be 'np' or 'ms', {return_tensors} not support." slopes = _get_interleave(num_heads) slopes = np.expand_dims(np.expand_dims(slopes, 1), 1) position_point = np.arange(seq_len) - seq_len + 1 position_point = np.expand_dims(np.expand_dims(position_point, 0), 0) position_point = np.tile(position_point, (num_heads, seq_len, 1)) diag = np.diag(position_point[0]) diag = np.expand_dims(np.expand_dims(diag, 0), 0) diag = np.transpose(diag, (0, 2, 1)) position_point = position_point - diag alibi = slopes * position_point alibi = np.expand_dims(alibi, 0) if return_tensors == 'np': return alibi return Tensor(alibi, dtype=dtype) pengcheng.mind-mindformer/mindformers/modules/local_block_sparse_attention.py000066400000000000000000000341161476650141400303740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ A Local Block Sparse Attention. """ from __future__ import absolute_import from functools import wraps, partial import inspect import math import numpy as np import mindspore as ms import mindspore.nn as nn import mindspore.ops as P from mindspore import Tensor from mindspore import dtype as mstype from mindspore._extends import cell_attr_register try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, OpParallelConfig __all__ = ["LocalBlockSparseAttention"] gather_index_inited = False kv_index = None mask_index = None def _args_type_validator_check(*type_args, **type_kwargs): """Check whether input data type is correct.""" def type_check(func): sig = inspect.signature(func) bound_types = sig.bind_partial(*type_args, **type_kwargs).arguments @wraps(func) def wrapper(*args, **kwargs): nonlocal bound_types bound_values = sig.bind(*args, **kwargs) argument_dict = bound_values.arguments if "kwargs" in bound_types: bound_types = bound_types["kwargs"] if "kwargs" in argument_dict: argument_dict = argument_dict["kwargs"] for name, value in argument_dict.items(): if name in bound_types: bound_types[name](value, name) return func(*args, **kwargs) return wrapper return type_check def _valid_type_checks(types, class_name): """types should be a list of types, this function check if the type is in the valid dtypes""" def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, type(value)) return validator_check_func def _valid_value_checks(types, class_name): """the value should be a list of types, this function check if the value is in the valid dtypes""" def validator_check_func(value, name): # The args of Validator.check_type_name is (arg_name, arg_type, valid_types, prim_name) # as the input of _args_type_validator_check is fixed, so we need to manually change the input order partial_check = partial(Validator.check_type_name, valid_types=types, prim_name=class_name) return partial_check(name, value) return validator_check_func class InitGatherIndex: """ A self-defined function to init the index of gather operation Args: seq_len(int): Length of input sequence. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference block sparse attention in FlashAttention """ def __init__(self, seq_len, local_size, block_size): self.seq_len = seq_len self.local_size = local_size self.block_size = block_size self.group_num = self.seq_len // self.block_size self.local_block_num = self.local_size // self.block_size self.prev_block_num = self.local_block_num - 1 def gen_kv_gather_index(self): index = np.ones((self.group_num * self.local_size)) for i in range(self.group_num): start = max(0, i - self.prev_block_num) * self.block_size end = (max(0, i - self.prev_block_num) + self.local_size // self.block_size) * self.block_size index[i * self.local_size:(i + 1) * self.local_size] = np.arange( start, end) return index def gen_mask_gatherd_index(self): index = np.ones((1, self.seq_len, self.local_size)) for i in range(self.group_num): start = max(0, i - self.prev_block_num) * self.block_size end = (max(0, i - self.prev_block_num) + self.local_block_num) * self.block_size index[:, i * self.block_size:(i + 1) * self.block_size, :] = np.arange(start, end) return index def init_gather_index(seq_len, local_size, block_size): gather_index = InitGatherIndex(seq_len, local_size, block_size) global kv_index kv_index = Tensor(gather_index.gen_kv_gather_index(), mstype.int32) global mask_index mask_index = Tensor(gather_index.gen_mask_gatherd_index(), mstype.int32) class LocalBlockSparseAttention(nn.Cell): """ Local Block Sparse Attention Layer. This function contains the sliding window attention primitives used in Longformer (see paper) `Longformer: The Long-Document Transformer ` and block sparse attention primitives used in Longformer FlashAttention (see paper) `FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ` Specifically, it includes the following: 1. A faster implementation of normal attention (the entire area of the upper triangle and part of the area of the lower triangle are not computed, and many operations are fused). 2. An implementation of "local" and "block" sparse attention, as in the Longformer and FlashAttention. Args: seq_len(int): Length of input sequence. size_per_head(int): The hidden size of the input. local_size(int): An integer determining the local size. Current implementation of sparse self-attention is based on local sparse matrices, reference sliding window attention in Longformer. block_size(int): An integer determining the block size. Current implementation of sparse self-attention is based on blocked sparse matrices, reference sliding window attention in FlashAttention. dropout_rate(float): The dropout rate of the attention score. Default 0.1. softmax_compute_type(mstype.Number): The type of softmax computation module. Default mstype.float16. Should be mstype.float32 or mstype.float16. parallel_config(OpParallelConfig): The config of parallel setting, see `OpParallelConfig`. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **q** (Tensor) - Tensor query (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **k** (Tensor) - Tensor key (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **v** (Tensor) - Tensor value (:class:`mstype.fp16` [batch_size, head_num, seq_length, size_per_head]) - **attention_mask** (Tensor) - Float Tensor the mask of (:class:`mstype.fp16` [batch_size, seq_length, seq_length]): Lower triangular matrix to pass masked information. Outputs: A Tensor. The output of the attention with shape [batch_size, head_num, seq_length, size_per_head] Supported Platforms: ``Ascend`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules import LocalBlockSparseAttention >>> from mindspore import Tensor >>> model = LocalBlockSparseAttention(seq_len=4096, ... size_per_head=128, ... local_size=1024, ... block_size=128) >>> q = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> k = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> v = Tensor(np.ones((2, 16, 4096, 128)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 4096, 4096)), mstype.float16) >>> output = model(q, k, v, attention_mask) >>> print(output.shape) (2, 16, 4096, 128) """ @cell_attr_register @_args_type_validator_check( seq_len=Validator.check_positive_int, size_per_head=Validator.check_positive_int, local_size=Validator.check_positive_int, block_size=Validator.check_positive_int, softmax_compute_type=_valid_value_checks( [mstype.float16, mstype.float32], "LocalBlockSparseAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "LocalBlockSparseAttention")) def __init__(self, seq_len, size_per_head, local_size, block_size, dropout_rate=0.1, softmax_compute_type=mstype.float16, parallel_config=default_dpmp_config): super(LocalBlockSparseAttention, self).__init__() if seq_len % block_size != 0: raise ValueError( f"block_size must be divisible by seq_len, " f"but got block_size {block_size} is not divisible by seq_len {seq_len}" ) if local_size % block_size != 0: raise ValueError( f"block_size must be divisible by local_size, " f"but got block_size {block_size} is not divisible by local_size {local_size}" ) if local_size >= seq_len: raise ValueError( f"local_size must be less than seq_len, " f"but got local_size {local_size} is not less than seq_len {seq_len}" ) if block_size > local_size: raise ValueError( f"block_size must be less than (or equal to) local_size, " f"but got block_size {local_size} is not less than (or equal to) local_size {local_size}" ) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError( f"dropout probability should be a number in range [0, 1), but got {dropout_rate}" ) global gather_index_inited if not gather_index_inited: init_gather_index(seq_len, local_size, block_size) gather_index_inited = True self.scale_factor = Tensor(math.sqrt(math.sqrt(size_per_head))) self.local_size = local_size self.block_size = block_size self.dropout_rate = dropout_rate self.softmax_compute_type = softmax_compute_type dp = parallel_config.data_parallel mp = parallel_config.model_parallel self.real_div = P.RealDiv().shard(((dp, mp, 1, 1), ())) self.qk_bmm = ms.ops.BatchMatMul(transpose_b=True).shard( ((dp, mp, 1, 1, 1), (dp, mp, 1, 1, 1))) self.attn_mask_mul = P.Mul().shard(((dp, 1, 1), (1,))) self.attn_mask_expand_dims = P.ExpandDims().shard(((dp, 1, 1),)) self.attn_mask_add = P.Add().shard( ((dp, 1, 1, 1, 1), (dp, mp, 1, 1, 1))) self.softmax = P.Softmax().shard(((dp, mp, 1, 1, 1),)) if self.dropout_rate > 1e-5: self.attention_dropout = nn.Dropout(keep_prob=1 - self.dropout_rate) self.attention_dropout.dropout.shard(((dp, mp, 1, 1, 1),)) self.pv_bmm = ms.ops.BatchMatMul().shard( ((dp, mp, 1, 1, 1), (dp, mp, 1, 1, 1))) self.multiply_data = Tensor([-10000.0], dtype=self.softmax_compute_type) self.reshape = P.Reshape() self.gather1 = P.Gather().shard(((dp, mp, 1, 1), (1,))) self.gather2 = P.GatherD().shard(((dp, 1, 1), (dp, 1, 1))) def construct(self, q, k, v, attention_mask=None): """Forward process""" bsz, head_num, seq_len, head_dim = q.shape factor = P.Cast()(self.scale_factor, P.DType()(q)) q = self.real_div(q, factor) k = self.real_div(k, factor) q = self.reshape(q, (bsz, head_num, seq_len // self.block_size, self.block_size, head_dim)) # kv: [bsz, head_num, seq_len, head_dim] -> [bsz, head_num, seq_len // block_size, local_size, head_dim] k = self.gather1(k, kv_index, 2) k = self.reshape(k, (bsz, head_num, -1, self.local_size, head_dim)) v = self.gather1(v, kv_index, 2) v = self.reshape(v, (bsz, head_num, -1, self.local_size, head_dim)) # q * k.T score = self.qk_bmm(q, k) ori_dtype = P.DType()(score) score = P.Cast()(score, self.softmax_compute_type) # softmax if attention_mask is not None: mask_index_ = P.tile(mask_index, (bsz, 1, 1)) attention_mask = self.gather2( attention_mask, 2, mask_index_) # [bsz, seq_len, local_size] adder = self.attn_mask_mul( P.Cast()(attention_mask, self.softmax_compute_type), self.multiply_data) adder = self.attn_mask_expand_dims(adder, 1) adder = self.reshape( adder, (bsz, 1, -1, self.block_size, self.local_size)) score = self.attn_mask_add(adder, score) probs = self.softmax(score) probs = P.Cast()(probs, ori_dtype) if self.dropout_rate > 1e-5: probs = self.attention_dropout(probs) # p * v output = self.pv_bmm(probs, v) output = self.reshape(output, (bsz, head_num, seq_len, head_dim)) return output pengcheng.mind-mindformer/mindformers/modules/transformer/000077500000000000000000000000001476650141400244515ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/transformer/.ipynb_checkpoints/000077500000000000000000000000001476650141400302425ustar00rootroot00000000000000transformer-checkpoint.py000066400000000000000000005737651476650141400352530ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/transformer/.ipynb_checkpoints# Copyright 2021-2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Transformer Networks. This is interface that is subject to change or deletion. """ from __future__ import absolute_import import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer from mindspore import nn from mindspore import context import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import LayerNorm, Linear, \ _args_type_validator_check, _valid_type_checks, _valid_value_checks, \ _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _PipeLineConfig, OpParallelConfig, \ _Config, _check_config, MoEParallelConfig from mindformers.modules.transformer.moe import default_moe_config, MoE, _check_moe_config from mindformers.version_control import get_dropout from mindformers.tools.logger import _LogActionOnce __all__ = [ "AttentionMask", "AttentionMaskHF", "VocabEmbedding", "MultiHeadAttention", "FeedForward", "TransformerEncoder", "TransformerDecoder", "TransformerEncoderLayer", "TransformerDecoderLayer", "Transformer", "TransformerOpParallelConfig", "EmbeddingOpParallelConfig", "TransformerRecomputeConfig"] class EmbeddingOpParallelConfig(_Config): r""" The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False, vocab_emb_dp=True): self._dp_mp_config = OpParallelConfig(data_parallel=data_parallel, use_seq_parallel=use_seq_parallel, model_parallel=model_parallel, select_recompute=select_recompute) Validator.check_bool(vocab_emb_dp, "vocab_emb_dp") self.vocab_emb_dp = vocab_emb_dp self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._dp_mp_config.data_parallel = value @property def model_parallel(self): return self._dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._dp_mp_config.model_parallel = value @property def vocab_emb_dp(self): return self._vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): Validator.check_bool(value, "vocab_emb_dp") self._vocab_emb_dp = value @property def dp_mp_config(self): return self._dp_mp_config class TransformerRecomputeConfig(_Config): r""" TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) """ def __init__(self, recompute=False, select_recompute=False, parallel_optimizer_comm_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): Validator.check_bool(recompute, "recompute") Validator.check_bool(parallel_optimizer_comm_recompute, "parallel_optimizer_comm_recompute") Validator.check_bool(mp_comm_recompute, "mp_comm_recompute") Validator.check_bool(select_recompute, "select_recompute") Validator.check_bool(recompute_slice_activation, "recompute_slice_activation") self._recompute = recompute self._select_recompute = select_recompute self._parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self._mp_comm_recompute = mp_comm_recompute self._recompute_slice_activation = recompute_slice_activation @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): Validator.check_bool(value, "recompute") self._recompute = value @property def select_recompute(self): return self._select_recompute @select_recompute.setter def select_recompute(self, value): Validator.check_bool(value, "select_recompute") self._select_recompute = value @property def parallel_optimizer_comm_recompute(self): return self._parallel_optimizer_comm_recompute @parallel_optimizer_comm_recompute.setter def parallel_optimizer_comm_recompute(self, value): Validator.check_bool(value, "parallel_optimizer_comm_recompute") self._parallel_optimizer_comm_recompute = value @property def mp_comm_recompute(self): return self._mp_comm_recompute @mp_comm_recompute.setter def mp_comm_recompute(self, value): Validator.check_bool(value, "mp_comm_recompute") self._mp_comm_recompute = value @property def recompute_slice_activation(self): return self._recompute_slice_activation @recompute_slice_activation.setter def recompute_slice_activation(self, value): Validator.check_bool(value, "recompute_slice_activation") self._recompute_slice_activation = value default_transformer_recompute_config = TransformerRecomputeConfig() class TransformerOpParallelConfig(_Config): r""" TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel. Note: Except the recompute argument, other arguments will **not** be effective when the user doesn't set auto_parallel_context to `SEMI_AUTO_PARALLEL` or `AUTO_PARALLEL`. The micro_batch_num must be greater than or equal to pipeline_stage when training. The data_parallel\*model_parallel \*pipeline_stage must be equal or less equal to the device. When setting the pipeline stage and optimizer_shard, the config will overwrite the auto_parallel_context. When given the 8 devices and the data_parallel is 1 and model_parallel is 1, the calculation will be repeated on each device. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. recompute (Union[TransformerRecomputeConfig, bool]): The configuration of recomputation for the transformer block. Default: An instance of TransformerRecomputeConfig with default values. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_num=1, recompute=default_transformer_recompute_config, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): self.recompute = recompute self.select_recompute = recompute.select_recompute self.use_seq_parallel = use_seq_parallel self.optimizer_shard = optimizer_shard self.gradient_aggregation_group = gradient_aggregation_group self._embed_dp_mp_config = EmbeddingOpParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, vocab_emb_dp=vocab_emb_dp, use_seq_parallel=use_seq_parallel, select_recompute=recompute.select_recompute) self._pp_config = _PipeLineConfig(pipeline_stage=pipeline_stage, micro_batch_num=micro_batch_num) self._moe_config = MoEParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, select_recompute=recompute.select_recompute, expert_parallel=expert_parallel, use_seq_parallel=use_seq_parallel) @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): if not isinstance(value, TransformerRecomputeConfig) and not isinstance(value, bool): raise TypeError(f"recompute must be a TransformerRecomputeConfig/bool, but got {type(value).__name__}.") if isinstance(value, bool): logger.warning(f"TransformerRecomputeConfig is recommended as the recompute configuration type.") self._recompute = value @property def vocab_emb_dp(self): return self._embed_dp_mp_config.vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): self._embed_dp_mp_config.vocab_emb_dp = value @property def gradient_aggregation_group(self): return self._gradient_aggregation_group @gradient_aggregation_group.setter def gradient_aggregation_group(self, value): Validator.check_positive_int(value, "gradient_aggregation_group") self._gradient_aggregation_group = value @property def micro_batch_num(self): return self._pp_config.micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): self._pp_config.micro_batch_num = value @property def model_parallel(self): return self._embed_dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._embed_dp_mp_config.model_parallel = value self._moe_config.model_parallel = value @property def data_parallel(self): return self._embed_dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._embed_dp_mp_config.data_parallel = value self._moe_config.data_parallel = value @property def expert_parallel(self): return self._moe_config.expert_parallel @expert_parallel.setter def expert_parallel(self, value): self._moe_config.expert_parallel = value @property def pipeline_stage(self): return self._pp_config.pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): self._pp_config.pipeline_stage = value @property def optimizer_shard(self): return self._optimizer_shard @optimizer_shard.setter def optimizer_shard(self, value): Validator.check_bool(value, "optimizer_shard") self._optimizer_shard = value context.set_auto_parallel_context(enable_parallel_optimizer=value) @property def embedding_dp_mp_config(self): return self._embed_dp_mp_config @property def dp_mp_config(self): return self._embed_dp_mp_config.dp_mp_config @property def moe_parallel_config(self): return self._moe_config default_transformer_config = TransformerOpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() class FeedForward(Cell): r""" The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, dropout_rate=Validator.check_non_negative_float, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "FeedForward")) def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(FeedForward, self).__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.cast = P.Cast() else: _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class AttentionMask(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMask', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMask")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMask, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMask""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(self.not_equal(input_mask, 0), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class AttentionMaskHF(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMaskHF', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMaskHF")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMaskHF, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMaskHF""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(P.OnesLike()(input_mask), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class VocabEmbedding(Cell): """ The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) """ @_LogActionOnce(m_logger=logger, key='VocabEmbedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int, parallel_config=_valid_type_checks([EmbeddingOpParallelConfig], "VocabEmbedding")) def __init__(self, vocab_size, embedding_size, parallel_config=default_embedding_parallel_config, param_init='normal'): super(VocabEmbedding, self).__init__() _check_config(parallel_config) self.vocab_size = vocab_size self.embedding_size = embedding_size self.embedding_table = Parameter(initializer(param_init, [self.vocab_size, self.embedding_size]), name='embedding_table', parallel_optimizer=False) if parallel_config.vocab_emb_dp: self.gather = P.Gather().shard(((1, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel for the embedding lookup.") else: if self.vocab_size % parallel_config.model_parallel != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_size} must be a " f"multiple of parallel_config.model_parallel {parallel_config.model_parallel}.") self.gather = P.Gather().shard(((parallel_config.model_parallel, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel and {parallel_config.model_parallel} " f"model parallel for the embedding lookup.") def construct(self, input_ids): _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32], self.cls_name) output = self.gather(self.embedding_table, input_ids, 0) return output, self.embedding_table.value() class MultiHeadAttention(Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) """ @_LogActionOnce(m_logger=logger, key='MultiHeadAttention', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "MultiHeadAttention"), use_past=Validator.check_bool) def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, parallel_config=default_dpmp_config): super(MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dp = parallel_config.data_parallel self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if batch_size: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul() self.real_div = P.RealDiv() self.sub = P.Sub() self.mul = P.Mul() self.add = P.Add() # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims() # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) else: _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.select_recompute: # _attn中涉及的关键算子使用重计算逻辑，通常配合序列并行使用，会有较好的性能提升效果 self.batch_matmul.recompute() self.sub.recompute() self.add.recompute() self.merger_head_transpose.recompute() self.softmax_reshape.recompute() self.prob_dropout.recompute() self.softmax_cast.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = F.shape(query_tensor) batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present def _get_batch_size_from_query(self, query): r"""Get the batch size from query tensor""" # For the incremental prediction, the seq length for the input is 1. if len(F.shape(query)) == 2 and ((self.use_past and self.is_first_iteration) or (not self.use_past)): return F.shape(query)[0] // self.src_seq_length return F.shape(query)[0] def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(key_past), "key_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_past), "value_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( self.softmax_reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = self.softmax_reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) attention_scores = self.softmax_cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(attention_scores)), P.Cast()(attention_mask, P.DType()(attention_scores))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.softmax_cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge class TransformerEncoderLayer(Cell): r""" Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerEncoderLayer"), use_past=Validator.check_bool) def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) if input_mask is not None: _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(Cell): r""" Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerDecoderLayer"), use_past=Validator.check_bool) def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) config_to_attention = parallel_config.dpmp if self.use_moe else parallel_config if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add() self.add_3d = P.Add() self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, decoder_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_output, cross_layer_present = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_output) else: x = self.add(x, cross_attn_output) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(layers // parallel_config.pipeline_stage, 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(layers // parallel_config.gradient_aggregation_group, 1) network.set_comm_fusion((layer_id + offset) // dis + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not parallel_config.select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not parallel_config.recompute.select_recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(Cell): r""" Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerEncoder"), use_past=Validator.check_bool) def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_config(parallel_config) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(Cell): r""" Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerDecoder"), use_past=Validator.check_bool) def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() _check_moe_config(moe_config, parallel_config) _check_config(parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class Transformer(Cell): r""" Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='Transformer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, encoder_layers=Validator.check_positive_int, decoder_layers=Validator.check_non_negative_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "Transformer"), use_past=Validator.check_bool) def __init__(self, hidden_size, batch_size, ffn_hidden_size, src_seq_length, tgt_seq_length, encoder_layers=3, decoder_layers=3, num_heads=2, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(Transformer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, encoder_inputs, encoder_masks, decoder_inputs=None, decoder_masks=None, memory_mask=None, init_reset=True, batch_valid_length=None): """process process""" encoder_output = None output = None encoder_layer_present = None decoder_layer_present = None accum_loss = self.aux_loss if self.encoder is not None: if self.use_moe: encoder_output, encoder_layer_present, encoder_aux_loss = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, encoder_aux_loss) else: encoder_output, encoder_layer_present = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) output = encoder_output if self.decoder is not None: # decoder mask should be created outside of the model if self.use_moe: decoder_output, decoder_layer_present, decoder_aux_loss = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, decoder_aux_loss) else: decoder_output, decoder_layer_present = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) output = decoder_output if self.use_moe: return output, encoder_layer_present, decoder_layer_present, accum_loss return output, encoder_layer_present, decoder_layer_present pengcheng.mind-mindformer/mindformers/modules/transformer/__init__.py000066400000000000000000000024671476650141400265730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ NOTE: Transformer Networks. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from .transformer import AttentionMask, VocabEmbedding, MultiHeadAttention, \ FeedForward, TransformerEncoder, TransformerDecoder, TransformerEncoderLayer, TransformerDecoderLayer, \ Transformer, TransformerOpParallelConfig, EmbeddingOpParallelConfig, TransformerRecomputeConfig from .transformer import * from .moe import MoEConfig from .op_parallel_config import OpParallelConfig __all__ = [] __all__.extend(transformer.__all__) __all__.extend(op_parallel_config.__all__) __all__.extend(moe.__all__) pengcheng.mind-mindformer/mindformers/modules/transformer/__pycache__/000077500000000000000000000000001476650141400266615ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/transformer/__pycache__/__init__.cpython-39.pyc000066400000000000000000000016241476650141400330540ustar00rootroot00000000000000a ?cf7@sdZddlmZddlmZmZmZmZmZm Z m Z m Z m Z m Z mZmZddlTddlmZddlmZgZeejeejeejdS) zn NOTE: Transformer Networks. This is an experimental interface that is subject to change or deletion. )absolute_import) AttentionMaskVocabEmbeddingMultiHeadAttention FeedForwardTransformerEncoderTransformerDecoderTransformerEncoderLayerTransformerDecoderLayer TransformerTransformerOpParallelConfigEmbeddingOpParallelConfigTransformerRecomputeConfig)*) MoEConfig)OpParallelConfigN)__doc__ __future__r transformerrrrrrr r r r r rrZmoerZop_parallel_configr__all__extendrrS/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/__init__.pys 8    pengcheng.mind-mindformer/mindformers/modules/transformer/__pycache__/moe.cpython-39.pyc000066400000000000000000000500571476650141400321010ustar00rootroot00000000000000a ?cf@sTdZddlmZddlmZddlZddlZddlmZddl m m Z ddl mmZzddlmZWneyddlmZYn0ddlmZddlmZdd lmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&dgZ'GdddZ(e(Z)dddZ*eddZ+GdddeZ,GdddeZ-GdddeZ.dS)zs Note: Mixture of Expert (MoE) structure. This is an experimental interface that is subject to change or deletion. )absolute_import)divisionN)Tensor) Validator) operations) functional) constexpr)Cell)Dense) ParallelMode)_get_parallel_mode_is_sharding_propagation)default_moeparallel_config MoEConfigc@seZdZdZd dd ZdS) raz The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import MoEConfig >>> moe_config = MoEConfig(expert_num=4, capacity_factor=5.0, aux_loss_factor=0.05, num_experts_chosen=1, ... expert_group_size=64, group_wise_a2a=True, comp_comm_parallel=False, ... comp_comm_parallel_degree=2) 皙?皙?NFc Cst|dt|dt|dt|dt|dt|dt|d|durht|d|d krtd |d |d krtd |d ||krtd |d ||_||_||_||_||_ ||_ ||_ ||_ dS)N expert_numcapacity_factoraux_loss_factornum_experts_chosengroup_wise_a2acomp_comm_parallelcomp_comm_parallel_degreeexpert_group_size?z@'capacity_factor' must be equal to or greater than 1.0, but got .z1'aux_loss_factor' must be less than 1.0, but got zC'num_experts_chosen' must not be larger than 'expert_num', but got ) rcheck_positive_intcheck_positive_float check_bool ValueErrorrrrrrrrr) selfrrrrrrrrr#N/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/moe.py__init__Ns:           zMoEConfig.__init__)rrrrNFFr)__name__ __module__ __qualname____doc__r%r#r#r#r$r-s c Cst|ts tdt|jd|jdk}|dur6dS|j|jdkrxtdt|jdt|jd |jd |jd t }||jdkrtd |d |jd|j |jdkrtd|j d |jd|j |j |krtd|j d|j d|ddS)z4 check if MoE with right configuration. z7'moe_config' must be an instance of MoEConfig, but got rrFNrz$When using MoE, the 'expert_num' in z2 must be a multiple of 'expert_parallel' value in z , but got z for 'expert_num' and z for 'expert_parallel'.z device_num: z( must be a multiple of expert_parallel: zdata parallel: z when using MoE.z"The product of the data parallel: z and model parallel: z! should be less than device_num: ) isinstancer TypeErrortyper&rexpert_parallelr!Dget_group_size data_parallelmodel_parallel) moe_configparallel_configuse_moe device_numr#r#r$_check_moe_configos8        r6cCst||||S)N)mathceil)ktokens_per_groupr expert_dimr#r#r$calculate_expert_capacitysr<csDeZdZdZdejeeffdd ZddZ ddZ d d Z Z S) MoEa The mixture of experts (MoE) implementation. The implementation includes a router and a FeedForward layer. The router dispatches tokens to experts in FeedForward, then FeedForward does computation, and the final output is obtained by multiplying FeedForward's output and router's combine weight. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. Default: gelu. param_init_type (dtype.Number): The parameter initialization type. Can be dtype.float32 or dtype.float16. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(MoEParallelConfig): The parallel config for MoE, see `MoEParallelConfig`. Default `default_moeparallel_config`, an instance of `MoEParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size]`. geluc spttttjfvrtr|_|j_ |j _ |j _ |j _ |j _ |j_|j_|j_|j_|j_|j_|j_jdkrjjksd_ddlm}|||||j j ||d_t_t_t !jdff_"t !jddff_#t !djddff_$t !ddjdff_%t&!jddfjddff_'t&!jddfjddff_(t)_*t+||dd|d_,t-_.t/d!t0fd d t1jD_2t/d !djddfdjddff_3tj4d jd !djddff_5t6!jdddff_7t6!djddff_8t6!jdddff_9t6!djjdff_:t6!jdjdff_;n|_|j_ |j _ |j _ |j _ |j_|j_|j_|j_|j_|j_|j_jdkrDjjksJd_ddlm}|||||j ||d _t_t_t !jdff_"t !jddff_#t !djddff_$t !ddjdff_%t&!jddfjddff_'t&!jddfjddff_(t)!d_*t+||dd|d_,t-_.t/d!t0fdd t1jD_2t/d !djddfdjddff_3tj4d jd !djddff_5t6!jdddff_7t6!djddff_8t6!jdddff_9t6!djjdff_:t6!jdjdff_;dS)NrFr) FeedForward) hidden_sizeffn_hidden_size dropout_rate hidden_actrrparam_init_typer3T)d_modelr2routing_policytrainingr3c3s|]}jdddfVqdSrNdp.0_r"r#r$ zMoE.__init__..r)axis output_num)r@rArBrCrrDr3r#r#c3s|]}jdddfVqdSrIrJrLrOr#r$rPrQ)<superr=r%r r AUTO_PARALLELr r@rr;rrrrr0dp_grouprKr-epr1mprrrZmindformers.modules.transformerr?ffnPReshapereshapeShapeshape Transposeshardtranspose_2dimtranspose_3dimtranspose_4dimtranspose_4dim_dp BatchMatMulbatch_mm batch_mm2MulmulRouterrouterCastcastConcattuplerangeconcat concat_dpSplitsplit StridedSlice stride_slicestride_slice_dpstride_slice_epstride_slice_dp_mpstride_slice_ep_mp) r"r@rArBrCrDr2r3r? __class__rOr$r%s   $$  **$"   $$ **$z MoE.__init__cCsd}|jr||jdkr(|j||j}|dkrd||7}||d|j|j||jfd}|||f}||d|j|j||jfd}||d|j|j||jfd}| |d|j|j||jfd}| ||j|j||jf}| |}| ||j|j||jf}|jr||d|j|j||jfd}||d|j|j||jfd}||d|j|j||jfd}|dkr||8}||d|j|j||jfd}| |d}|S)z$ Computing the FFN. rrrrrrrrr)rrHrr) rrYrxr;rWr@rsrzr{ryr]rZrd)r" expert_inputcapacitypad_size pad_tensor expert_outputr#r#r$ ffn_infer sf   z MoE.ffn_inferc Csd}||jdkrV|j||j}||7}||d|j|j||jfd}|||f}||j}g}||D]}|||}||qn| |} |dkr||8}| | d|j|j|j|fd} | S)zB Split and overlap FFN compute and communication. rr~r) rrxr;rWr@rsrurappendrrrw) r"rrrrZ sub_capacity output_listZsub_expert_inputZsub_expert_outputrr#r#r$ffn_parallel_inferGs,    zMoE.ffn_parallel_inferc Cst|}||d|jf}||}|d|j}|||j||jf}t|j||j|j}| |\}}}| |d}|||j||j|f}| |t |}| ||} || |j|j|j|f} || |j|j|j|f} || d} || |j||j|jf} || d} |jr2|| |} n || |} || |j|j|j|f} |||j||j|f}| |d}| |t | }|| |} | | d} || |d|df} || |} ||j|}| |fS)forward processr)rrr)rr)rrrrHr)Fr_r]r@rWr<rrr;rlrcrndtypergrbrerrrrhrjr) r" input_tensor input_shapeZ bs_and_dmodelr:expert_capacitydispatch_tensorcombine_tensoraux_lossrrZcombined_outputr#r#r$ constructdsV             z MoE.construct) r&r'r(r)mstypefloat32default_moe_configrr%rrr __classcell__r#r#r|r$r=sc:r=cs*eZdZdZdfdd ZddZZS) rka A router backbone used to calculate logits of each token, which should be cascaded by router implementations mapping tokens to experts. when moe_config.num_experts_chosen = 1, use top1 routing; when moe_config.num_experts_chosen > 1, use topk routing Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). routing_policy: The policy of mapping tokens to experts. Default: topkRouter training (bool): The value indicating whether is in training phase. parallel_config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim)`. NTcstt||j}||_|j|_|j|_|j|_||_ ||_ d|_ d|_ t tjd|j d|j |f|_t|j|jdd|_|jj|dfdft|_t|_|j durt||||d|_n||_ttjfvrt s|j|ddf|ffdS)Ng{Gz?rF) in_channels out_channelshas_bias)rr)rEr2rGr3)!rUrkr%r0rErr;rrrGrF noisy_policyZ noisy_epsilonrnprandomuniformnoiser densematmulrar[rirjrmrn TopkRouterrlr r rVr )r"rEr2rFrGr3rKr|r#r$r%s,$    zRouter.__init__cCs@||tj}|jdkr,|jr,|||j}||}||S)NZjitter) rnrrrrGrjrrrl)r"r router_logitsr#r#r$rs  zRouter.construct)NTN)r&r'r(r)r%rrr#r#r|r$rks  rkcs:eZdZdZd fdd ZddZdd Zd d ZZS) ra A router implementation which maps each tokens to the topk expert. Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). training (bool): The value indicating whether is in training phase. config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(1)`. TNcstt|ttjfvrRtrR|j}||_|j |_ |j |_ ||_ ||_ d|_t|_t|_t|_tjdd|_tjddd|_|j|_t|_t|_t|_tdt j!|_"tdt j!|_#tj$dd|_%tj$dd|_&tj$dd|_'t(|_)t(|_*t(|_+t(|_,t(|_-t(|_.t(|_/t(0|ddf|ddff|_1t(0|dddf|dddff|_2t3|_4t5|_6t5|_7t8|_9t8|_:t8|_;t8|_|_?tj@d d |_AtB|_CtjDdd|_EtjDd d|_FtjDd d|_GtH|_ItH|_Jt8|_Ktdt j!|_Ln2|j}||_|j |_ |j |_ ||_ ||_ d|_t|_t|_t|_tjdd0|ddff|_tjddd0|ddff|_|j|_t0|ddfd d f|_t0|ddfd d f|_t0|dddfd d f|_tdt j!|_"tdt j!|_#tj$dd0|ddff|_%tj$dd0|ddff|_&tj$dd0|dff|_'t(0|df|dff|_)t(0d |_*t(0d |_+t(0|ddf|ddff|_,t(0|ddf|ddff|_-t(0|df|dff|_.t(0|df|dff|_/t(0|ddf|ddff|_1t(0|dddf|dddff|_2t30|dddfd f|_4t50|ddf|ddff|_6t50|dddf|dddff|_7t80|ddf|ddff|_9t80|ddfd f|_:t80|dddf|dddff|_;t80|df|dff|_0d |ddff|_?tj@d d 0|ddff|_AtB0|ddfd f|_CtjDdd0|ddff|_EtjDd d0|ddff|_FtjDd d0|dddff|_GtH0|dff|_ItH0|ddff|_Jt80d |_Ktdt j!|_LdS) Nr)rRF)rR keep_dimsr)rrT) exclusiver#rT)MrUrr%r r rVr r0rErr;rrGrWrr[rmrnr\r]r^r_SoftmaxsoftmaxArgMaxWithValueargmaxrOneHotonehotonehot2onehot3rrron_value off_value ReduceMean reduce_mean reduce_mean2 reduce_mean3rirjmul2mul3mul4mul5mul6mul7ramul8mul9NotEqual not_equalRealDivdiv1div2Addaddadd1add2add3add4SubsubCumSumcumsumLessless ReduceSum reduce_sumreduce_sum_keepreduce_sum_keep2 ExpandDimsexpandexpand2 add_scala init_loss)r"rEr2rGr3rKr|r#r$r%s              $                   $ $ $zTopkRouter.__init__c Cs||}||d|df}||}|d|j}t|j||j|j}|||j||jf}d}d}|j}d} d} ||} t |jD]<} | | | |j |} | | \} }|| |j|j |j}|| ||| dd}|||||}||||| | }|d|d|d|df\}}}}|||}|||}|| ||d} |||}|||d|| |j|j |j}|||d|||t j!||j |j}|"| |} q|#|#| dd}|$| |%|d} |&| d}|| |fS) rrrg& .>rrrHr)'r_r]rWr<rrr;rrrqrrrrrrrrrr_auxiliary_loss_maskout_overflowed_tokensrrrrrrrrrrnrint32rrrrr)r"rZrouter_logits_shape logits_shaper:rZaccum_expert_maskZaccum_expert_gatelossZ mask_countZaccum_combine_tensor router_probexpert_chosen_indexZ expert_index expert_gate expert_maskZrouter_prob_normaloutputexpert_mask_flatposition_in_expertrZcombine_tensor_sumrr#r#r$r^sR     $    zTopkRouter.constructcCsJ||d}||d}|||}||}||||j|j}|S)z2 Computing the load balance loss. r)rrrjrrrr;)r"rrZ density_1Zdensity_1_proxyrr#r#r$rs     zTopkRouter._auxiliary_lossc Csl||d}|dkr |||}|||}|||}|||}||d} ||| }||| |f} | S)zJ Keeping only the tokens that fit within expert_capacity. rrr)rrrrrrr) r"rrrlast_numrrrZ less_resultrrr#r#r$rs        z%TopkRouter._maskout_overflowed_tokens)TN) r&r'r(r)r%rrrrr#r#r|r$rsm9 r)NN)/r) __future__rrr7numpyrmindspore.common.tensorrmindspore.common.dtypecommonrr"mindspore.communication.management communication managementr.mindspore._checkparamr ImportError _checkparam mindspore.opsrr[rrmindspore.ops.primitivermindspore.nn.cellr Zmindspore.nn.layerr mindspore.contextr mindspore.parallel._utilsr r 2mindformers.modules.transformer.op_parallel_configr__all__rrr6r<r=rkrr#r#r#r$s>           ?  @op_parallel_config.cpython-39.pyc000066400000000000000000000162341476650141400350600ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/modules/transformer/__pycache__a ?cf!!@sdZddlmZzddlmZWney>ddlmZYn0ddlmZddl m m Z ddl mZddlmZddlmZd gZGd d d ZGd d d eZGdd d eZGdddeZeZeZddZdS)zx Parallel Config for the Parallel Training This is an experimental interface that is subject to change and/or deletion. )absolute_import) ValidatorN)context) ParallelMode)_get_parallel_mode)logOpParallelConfigc@seZdZdZddZdS)_Configz A basic class of the configurecCs0d}|jD]\}}d||}||7}q|S)Nz[ParallelConfig] z{}:{} )__dict__itemsformat)selfinfokvZvar_infor]/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/op_parallel_config.py__str__'s   z_Config.__str__N)__name__ __module__ __qualname____doc__rrrrrr $sr c@steZdZdZdddZeddZejddZed d Zejd d Zed d Z e jdd Z eddZ dS)MoEParallelConfiga MoEParallelConfig for MoE structure, which includes setting data parallel, model parallel and expert parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 expert_parallel (int): The expert parallel way. Default: 1 Supported Platforms: ``Ascend`` FcCsdt|dt|dt|dt|dt|dt||||d|_||_||_||_dS)N data_parallelmodel_parallelexpert_paralleluse_seq_parallelselect_recompute)rrrr)rcheck_positive_int check_boolr_dpmprrr)r rrrrrrrr__init__;s     zMoEParallelConfig.__init__cCs|jjSN)r!rr rrrrJszMoEParallelConfig.data_parallelcCst|d||j_dSNr)rrr!rr valuerrrrNs cCs|jjSr#)r!rr$rrrrSsz MoEParallelConfig.model_parallelcCst|d||j_dSNr)rrr!rr&rrrrWs cCs|jSr#)_expert_parallelr$rrrr\sz!MoEParallelConfig.expert_parallelcCst|d||_dS)Nr)rrr)r&rrrr`s cCs|jS)z Get the configuration for dpmp )r!r$rrrdpmpeszMoEParallelConfig.dpmpN)rrrFF) rrrrr"propertyrsetterrrr*rrrrr/s$        rc@sNeZdZdZd ddZeddZejddZed d Zejd d Zd S)ra OpParallelConfig for the setting data parallel and model parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules import OpParallelConfig >>> config=OpParallelConfig(data_parallel=1, model_parallel=1) rFcCsLt|dt|dt|dt|d||_||_||_||_dS)Nrrrr)rrr rrrr)r rrrrrrrr"zs    zOpParallelConfig.__init__cCs|jSr#)_data_parallelr$rrrrszOpParallelConfig.data_parallelcCst|d||_dSr%)rrr-r&rrrrs cCs|jSr#)_model_parallelr$rrrrszOpParallelConfig.model_parallelcCst|d||_dSr()rrr.r&rrrrs N)rrFF) rrrrr"r+rr,rrrrrrks    c@sNeZdZdZd ddZeddZejddZedd Zejd d Zd S) _PipeLineConfiga PPConfig for the setting data parallel, model parallel Args: pipeline_stage (int): The number of the pipeline stages. Default: 1 micro_batch_num (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> config=_PipeLineConfig(pipeline_stage=1, micro_batch_num=1) rcCs(t|dt|d||_||_dS)Npipeline_stagemicro_batch_num)rrr0r1)r r0r1rrrr"s  z_PipeLineConfig.__init__cCs|jSr#)_pipeline_stager$rrrr0sz_PipeLineConfig.pipeline_stagecCs"t|d||_tj|ddS)Nr0)pipeline_stages)rrr2rset_auto_parallel_contextr&rrrr0s cCs|jSr#)_micro_batch_numr$rrrr1sz_PipeLineConfig.micro_batch_numcCst|d||_dS)Nr1)rrr5r&rrrr1s N)rr) rrrrr"r+r0r,r1rrrrr/s    r/c Cstd}t|dr6||jkr6td|d|jdttjtjfv}|sPdSt }td}|j |j ||krtd|j d |j d |d |d t|d r|r||j krtd|d|j ddS)z7 Check if micro_batch_num >= pipeline_stage r3r0zThe pipeline stage z= in auto_parallel_context is not equal to the pipeline_stage z in the config.Nenable_parallel_optimizerz!The product of the data parallel z, model parallel z pipeline stages z should be less than device_num .optimizer_shardzThe optimizer shard z> in auto_parallel_context is not equal to the optimizer_shard zS in the OpParallelConfig. Please check the optimizer_shard to make them consistent.)rget_auto_parallel_contexthasattrr0 ValueErrorrrSEMI_AUTO_PARALLEL AUTO_PARALLELDget_group_sizerrr8loggerwarning)configr0is_auto_parallel device_numr8rrr _check_configs0     rE)r __future__rmindspore._checkparamr ImportError _checkparam mindsporer"mindspore.communication.management communication managementr>mindspore.contextrmindspore.parallel._utilsrrr@__all__r rrr/default_dpmp_configZdefault_moeparallel_configrErrrrs&       <,)pengcheng.mind-mindformer/mindformers/modules/transformer/__pycache__/transformer.cpython-39.pyc000066400000000000000000003323621476650141400336650ustar00rootroot00000000000000a ?cf@sNdZddlmZddlZddlZddlmZddlm Z ddl m Z ddl m Z ddl mZddlmmZdd lmZdd lmZdd lmZzdd lmZWneyddlmZYn0dd l mZddl m!Z!ddl"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8ddl9m:Z:gdZ;Gddde0ZGddde0Z?e?Z@e<ZAGdddeZBGdddeZCGd d!d!eZDGd"d#d#eZEGd$d%d%eZFGd&d'd'eZGGd(d)d)eZHd2d*d+ZIGd,d-d-eZJGd.d/d/eZKGd0d1d1eZLdS)3zZ Note: Transformer Networks. This is interface that is subject to change or deletion. )absolute_importN)Tensor) Parameter) initializer)nn)context) operations) functional)Cell) Validator)log)_get_parallel_mode) ParallelMode) LayerNormLinear_args_type_validator_check_valid_type_checks_valid_value_checks_check_past_none_input_none_check_input_dtype)default_dpmp_config_PipeLineConfigOpParallelConfig_Config _check_configMoEParallelConfig)default_moe_configMoE_check_moe_config) get_dropout)_LogActionOnce) AttentionMaskAttentionMaskHFVocabEmbeddingMultiHeadAttention FeedForwardTransformerEncoderTransformerDecoderTransformerEncoderLayerTransformerDecoderLayer TransformerTransformerOpParallelConfigEmbeddingOpParallelConfigTransformerRecomputeConfigc@steZdZdZdddZeddZejd dZed d Zejd d Zed dZ e jddZ eddZ dS)r,a The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) FTcCs4t||||d|_t|d||_||_||_dS)N) data_paralleluse_seq_parallelmodel_parallelselect_recompute vocab_emb_dp)r _dp_mp_configr check_boolr3r0r2)selfr/r1r0r2r3r7V/home/ma-user/work/mindformers-baichuan/mindformers/modules/transformer/transformer.py__init__Ys z"EmbeddingOpParallelConfig.__init__cCs|jjSNr4r/r6r7r7r8r/esz'EmbeddingOpParallelConfig.data_parallelcCs ||j_dSr:r;r6valuer7r7r8r/iscCs|jjSr:r4r1r<r7r7r8r1msz(EmbeddingOpParallelConfig.model_parallelcCs ||j_dSr:r?r=r7r7r8r1qscCs|jSr:) _vocab_emb_dpr<r7r7r8r3usz&EmbeddingOpParallelConfig.vocab_emb_dpcCst|d||_dS)Nr3)r r5r@r=r7r7r8r3ys cCs|jSr:)r4r<r7r7r8 dp_mp_config~sz&EmbeddingOpParallelConfig.dp_mp_configN)r.r.FFT) __name__ __module__ __qualname____doc__r9propertyr/setterr1r3rAr7r7r7r8r,Bs&       r,c@seZdZdZdddZeddZejddZed d Zejd d Zed d Z e jdd Z eddZ e jddZ eddZ e jddZ dS)r-a TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) FTcCs^t|dt|dt|dt|dt|d||_||_||_||_||_dS)N recompute!parallel_optimizer_comm_recomputemp_comm_recomputer2recompute_slice_activation)r r5 _recompute_select_recompute"_parallel_optimizer_comm_recompute_mp_comm_recompute_recompute_slice_activation)r6rHr2rIrJrKr7r7r8r9s     z#TransformerRecomputeConfig.__init__cCs|jSr:rLr<r7r7r8rHsz$TransformerRecomputeConfig.recomputecCst|d||_dS)NrH)r r5rLr=r7r7r8rHs cCs|jSr:)rMr<r7r7r8r2sz+TransformerRecomputeConfig.select_recomputecCst|d||_dS)Nr2)r r5rMr=r7r7r8r2s cCs|jSr:)rNr<r7r7r8rIsz>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) r.FTc Cs^||_|j|_||_||_| |_t||| ||jd|_t||d|_t |||j||d|_ dS)N)r/r1r3r0r2)pipeline_stagemicro_batch_num)r/r1r2expert_parallelr0) rHr2r0optimizer_shardgradient_aggregation_groupr,_embed_dp_mp_configr _pp_configr _moe_config) r6r/r1rUrSrTrHr0rVrWr3r7r7r8r9s z$TransformerOpParallelConfig.__init__cCs|jSr:rQr<r7r7r8rHsz%TransformerOpParallelConfig.recomputecCsHt|ts*t|ts*tdt|jdt|tr>td||_dS)Nz=recompute must be a TransformerRecomputeConfig/bool, but got .zNTransformerRecomputeConfig is recommended as the recompute configuration type.) isinstancer-bool TypeErrortyperBloggerwarningrLr=r7r7r8rHs   cCs|jjSr:rXr3r<r7r7r8r3sz(TransformerOpParallelConfig.vocab_emb_dpcCs ||j_dSr:rbr=r7r7r8r3 scCs|jSr:)_gradient_aggregation_groupr<r7r7r8rW$sz6TransformerOpParallelConfig.gradient_aggregation_groupcCst|d||_dS)NrW)r check_positive_intrcr=r7r7r8rW(s cCs|jjSr:rYrTr<r7r7r8rT-sz+TransformerOpParallelConfig.micro_batch_numcCs ||j_dSr:rer=r7r7r8rT1scCs|jjSr:)rXr1r<r7r7r8r15sz*TransformerOpParallelConfig.model_parallelcCs||j_||j_dSr:)rXr1rZr=r7r7r8r19scCs|jjSr:)rXr/r<r7r7r8r/>sz)TransformerOpParallelConfig.data_parallelcCs||j_||j_dSr:)rXr/rZr=r7r7r8r/BscCs|jjSr:rZrUr<r7r7r8rUGsz+TransformerOpParallelConfig.expert_parallelcCs ||j_dSr:rfr=r7r7r8rUKscCs|jjSr:rYrSr<r7r7r8rSOsz*TransformerOpParallelConfig.pipeline_stagecCs ||j_dSr:rgr=r7r7r8rSSscCs|jSr:)_optimizer_shardr<r7r7r8rVWsz+TransformerOpParallelConfig.optimizer_shardcCs"t|d||_tj|ddS)NrV)enable_parallel_optimizer)r r5rhrset_auto_parallel_contextr=r7r7r8rV[s cCs|jSr:)rXr<r7r7r8embedding_dp_mp_configasz2TransformerOpParallelConfig.embedding_dp_mp_configcCs|jjSr:)rXrAr<r7r7r8rAesz(TransformerOpParallelConfig.dp_mp_configcCs|jSr:)rZr<r7r7r8moe_parallel_configisz/TransformerOpParallelConfig.moe_parallel_configN)rBrCrDrE$default_transformer_recompute_configr9rFrHrGr3rWrTr1r/rUrSrVrkrArlr7r7r7r8r+s^%                      r+c s~eZdZdZeedeejfvde e j e j e j e ejejgdeeegdddddejeffdd Zd d ZZS) r%a/ The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) m_loggerkey no_warning) hidden_sizeffn_hidden_size dropout_rateparam_init_typeparallel_configgelur.Nc  stt||dus,t|ts>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] rn seq_lengthrvcstt|||_t|jdfdf|_t |_ t |jddf|jddff|_ t d|_tj||fd}tt|tj|_t|jddfdf|_dSNr.r7r.r.rr.r.r.)rr!r9rrNotEqualrr/ not_equalReshapereshape BatchMatMulmul ExpandDims expand_dimnponesrtrilrrlower_triangle_maskMulmultiplyr6rrvrrr7r8r9s zAttentionMask.__init__c Cstt|dtjtjg|jt| |dtj}t |}|dd|df}|d}| ||}| ||}| ||}| |jd}|||}|S)z$Forward process of the AttentionMask input_maskrr.r)rrrrrrrrrrShaperrrrr r6r input_shapeZ shape_right shape_left mask_leftZ mask_rightattention_maskZlower_traiangler7r7r8rs    zAttentionMask.constructrBrCrDrEr r`r rrrr rdrrrr9rrr7r7rr8r!hs'   r!csZeZdZdZeedeejfvde e j e e gddeffdd ZddZZS)r"a Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] rnrcstt|||_t|jdfdf|_t |_ t |jddf|jddff|_ t d|_tj||fd}tt|tj|_t|jddfdf|_dSr)rr"r9rrrrr/rrrrrrrrrrrrrrrrrrr7r8r9s zAttentionMaskHF.__init__c Cstt|dtjtjg|jtt |tj}t |}|dd|df}|d}| ||}| ||}| ||}| |jd}|||}|S)z&Forward process of the AttentionMaskHFrrr.r)rrrrrrrrrOnesLikerrrrrrrr7r7r8rs    zAttentionMaskHF.constructrr7r7rr8r"s&   r"cs`eZdZdZeedeejfvde e j e j e e gddedffdd ZddZZS) r#a The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) rn) vocab_sizeembedding_sizervnormalcstt|t|||_||_tt||j|jgddd|_|j rvt d|j dff|_td|j dnf|j|jdkrtd |jd |jd t |jdf|j dff|_td|j d |jd dS)Nembedding_tableF)namerrr.zUsing z( data parallel for the embedding lookup.rz The vocab size of the embedding z6 must be a multiple of parallel_config.model_parallel r[z data parallel and z) model parallel for the embedding lookup.)rr#r9rrrrrrr3rGatherrr/gatherr`infor1r)r6rrrv param_initrr7r8r94s    zVocabEmbedding.__init__cCs8tt|dtjg|j||j|d}||jfS)N input_idsr) rrrrint32rrrr>)r6rrr7r7r8rMszVocabEmbedding.construct)rBrCrDrEr r`r rrrr rdrr,!default_embedding_parallel_configr9rrr7r7rr8r#s6  r#cseZdZdZeedeejfvde e j e j e j e j e j e j e ejejgde ejejgde ejejgdeegde jd ddejejejdeffdd Zdd d Zd d Zd dZdddZddZddZddZddZZS)r$a This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) rn) rr num_headssrc_seq_lengthtgt_seq_lengthattention_dropout_ratehidden_dropout_rate compute_dtypesoftmax_compute_typerurvuse_past皙?Fc sZ tt|tddv|_| j|_tt j t j fv|_ |rHt |tt j fvrt| ||_||_||_||_|dks|dkrtd||dks|dkrtd|||dkrtd|||| jdkrtd|| jd |_t||d || d |_|jj| jdfd f| j| jf| jdffd d |jj_t|_t|_ t!|_"||_#||j#|_$tj%dd|_&tj%dd|_'t(dg| d|_)t*|_+t,|_-t.|_/t0|_1t2|_3t(t45t45|j$|_6| |_7t8||_9t8||_:t;<=| |_>t;<=| |_?t@|_At!|_BtC|_Dt|||| d|_Et|||| d|_Ft|||| d|_G||_H| |_I|j7rtJK|"ddd} t(tJL| |ddftMjN|_O||_Pt(tJQtJjR|jP|jPfdtMjN|_StTd|_UtVd|_WtXd|_YtCd|_DtZd|_[t2d|_3t\d|_]t.d|_^t_d|_Lt`d|_at0d|_bn\t| ||_||_||_||_|dks|dkrtd||dks|dkrtd|||dkrtd|||| jdkr:td|| jd |_t||d || d |_|jj| jdfd f| j| jf| jdffd d |jj_t| jd| jdff|_t| j| jddff|_ t!|_"||_#||j#|_$tj%dd|_&tj%dd|_'t(dg| d|_)t*| j| jddf| j| jddff|_+t,| j| jddfdf|_-t.d | jdddff|_/t0| jdddfd f|_1t2| jdddf| j| jddff|_3t(t45t45|j$|_6| |_7t8||_9t8||_:|j9j9| jdff|j:j9| j| jddfft;<=| |_>|j>j>| j| jddfft;<=| |_?|j?j>| j| jdfft@|_At!|_BtC| jddff|_Dt|||| d|_E|jEj| jdf| jdff| j| jf| jffdt|||| d|_F|jFj| jdf| jdff| j| jf| jffdt|||| d|_G|jGj| jdf| jdff| j| jf| jffd||_H| |_I|j7rtJK|"ddd} t(tJL| |ddftMjN|_O||_Pt(tJQtJjR|jP|jPfdtMjN|_StTd|_UtVd|_WtXd|_YtCd|_DtZd|_[t2d|_3t\d|_]t.d|_^t_d|_Lt`d|_at0d|_b| jcr|j9j9| j| jdff|jj| j| jdfd f| j| jf| jdff| j| jdffd| jd rV|j+e|j/e|j3e|j e|jBe|j:e|jAe|j>j>e|j?edS)N device_target)Ascendrr.zuFor 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be in range [0, 1.0), but got the value : {}.zxFor 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be in range [0, 1.0), but got the value : {}.zFor 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple of 'num_heads', but got the hidden_size is {} and the num_heads is {}.zFor 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of 'parallel_config.model_parallel', but got the num_heads is {} and the parallel_config.model_parallel is {}.TF)rxryr{rrur)rrr)axisrg)r)rrurr.r.r.r.rr7)r)rrrr)rr7r7rrrZout_strategy_matmul)frr$r9r get_context _is_ascendr/rr rSEMI_AUTO_PARALLELrZis_parallel_moder rdrrrrr batch_sizerrr1is_first_iterationrrrrrr Transpose transposemerger_head_transposerrn_head size_per_headConcatZconcat_kZconcat_vr multiply_datar batch_matmulRealDivreal_divSubsubrrAddaddmathsqrt scale_factorrrr prob_dropoutrSoftmaxto_floatsoftmax softmax_3dr softmax_castsoftmax_reshaper expand_dimsdense1dense2dense3r softmax_dtyperarangetilerrrangerrrr StridedSliceslicerr ReduceSum reducesum LessEqual tensor_leEqualequalsub1TileLesslessmul1r0r2rH)r6rrrrrrrrrrrurrvZ seq_rangerr7r8r9s                  $            $         zMultiHeadAttention.__init__Nc Cs||||||||t|}||} ||||\}}}t|} t||j}t||j}t||j}||} ||} | |} | t | | | |j |j|jfd} | t | | | |j|j|jfd} | t | | | |j|j|jfd} |dur2tt|dkr2||d}| }| }|jrH|jrt||j|ddd|j}|| ||d}|| ||d}n|d}| |d}t|||j|j}||| ddd|jf||d}||| dd|jdf||d}|||} |||} | }| }t |j|j|jddf}||f}|| | | |}||}| |}t ||}t|| }||fS) z)Forward process of the MultiHeadAttentionrrr.r)rrrr.Nrr.rrrr.r.)! _check_inputsrr_get_batch_size_from_query_convert_to_2d_tensorrrrr r rr!_get_seq_length_under_incrementalrrrrrrrrrrviewrrr rrr_attnrr)r6 query_tensor key_tensor value_tensorrkey_past value_pastbatch_valid_length ori_shaper ori_dtypequeryrpr> key_present value_presentZvalid_length_vector valid_lengthZ current_key current_value layer_present attentionrr7r7r8rs          "         zMultiHeadAttention.constructcCsFtt|dkr8|jr|js$|js8t|d|jSt|dS)z$Get the batch size from query tensorrr)rrrrrr)r6r,r7r7r8r3s$z-MultiHeadAttention._get_batch_size_from_querycCs|jr|jsdS|S)zxReturn the length of the tensor. For the incremental prediction, the seq length for the input is 1. r.)rr)r6lengthr7r7r8r!:s z4MultiHeadAttention._get_seq_length_under_incrementalcCsRtt|dtjtjg|jtt|dtjtjg|jtt|dtjtjg|j|durtt|dtjtjg|jt|t}t|t} t|t} |du} |du} |du} t |j d|jd|| t |j d|jd| | t |j d|jd| | |j rNtt|dtjg|jtt|dtjg|jtt|dtj g|jd S) Check inputsr$r%r&Nrr'r(r)T rrrrrrrr\rrrr)r6r$r%r&rr'r(r)Z key_is_tensorZvalue_is_tensorbatch_valid_length_is_tensorZkey_is_defaultZvalue_is_defaultbatch_is_defaultr7r7r8rBs6   z MultiHeadAttention._check_inputscCsdt|}t|d|df}t|}t|d|df}t|}t|d|df}|||fS)z"convert a nd tensor to a 2d tensorr)rrr)r6r$r%r& query_shape key_shape value_shaper7r7r8r ^s   z(MultiHeadAttention._convert_to_2d_tensorcCs<||d}t|}d|d|df}|||}|S)z convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output rr)rrrr)r6rx_shape new_shapeZx_merger7r7r8 _merge_headsis   zMultiHeadAttention._merge_headsc Cs^|jr|jtjks|js$||}n6t|}||||dd|df}|||}|S)z For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. rr) rr rrrrrrr)r6attention_scoresattention_probsrr7r7r8_softmaxzs   zMultiHeadAttention._softmaxcCszt|jt|}|||}|||}|||}t|}|||j}|dur@|jr|j s| t | ||dt |ddd|jfddtjd} |t | tjd} t | d} t ||j| tj}||d}|tt d t|t|t|} || |j} || |}||} || |} || } || |}| |}|S) ay Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores N)rrrrrr.r)r.rrrr)g?)!rrrDTyperrrr rrrrrrrrrrrrrrrrrrtuple_to_arrayrrrrArr>)r6r,rpr>rfactorscorer+r? current_indexindexZ multiplu_outZadderr@Zweighted_valuesZattention_merger7r7r8r#sH              zMultiHeadAttention._attn)NNN)NNN)rBrCrDrEr r`r rrrr rdrrrrrrrr5rr9rrr!rr r>rAr#rr7r7rr8r$SsZ~    | U  r$cseZdZdZeedeejfvde e j e j e j e j e j e j e j eejejgdeejejgdeejejgdeeegde j d dddejejejddeef fdd Zdd d Zd dZZS)r(a Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) rn) rrrrsrrrpost_layernorm_residuallayernorm_compute_typerrurvrrFrwc stt||s| r t|||_ttjfvrt |||j dkr^t d ||j ||j dkr~t d ||j ||j dkrt d ||j t |||jdk|_| |_||_||_t|f| |_t|f| |_|jr|jn|}t|||||||| | | |d |_|jr<|js|jj?j |j!|j dfdf|j!|j f|j dff|j!|j dffd|jj@j@ |j!|j dffntAd|jBddS)NrzFor 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the 'parallel_config.model_parallel', but got the num_heads is {} and parallel_config.model_parallel is {}.zFor 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config. model_parallel is {}.zFor 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} and parallel_config. model_parallel is {}.r.) rrrrrrrrrrurrvrrrtrsrur moe_configrv)rrrtrsrurrvrrrr'rr(rrrrrrThe B only support sharding propagation or semi-auto parallel mode now.)Crr(r9r rdrr rrrr1rrrr|use_moerrrrrr layernorm1 layernorm2dpmpr$r2rrr%rHrrrr/radd_3drrrr'r(rrrrrrr9r:rrrzerosrr rrAssignassignr0rHr2 layer_normrr RuntimeErrorr)r6rrrrsrrrrrHrIrrurrrKrvZattention_parallel_configrrr7r8r96s\"      $       $      z TransformerEncoderLayer.__init__NTc Cs|||||t|}t|d|df}|jr:|}n ||}t||j}d}d}|jr| |j | |j t||j|j }| |j | |j t||j|j }t ||}t ||}||||||j |j |\} } |jr||| }n ||| }||} t| |j} d} |jr:|| \} } n || } d}d}|jr| \}}| |j ||j }| |j ||j }t ||}t ||}t | |} t | |} t|dkr:t| |} t| |} t||}|jr,|| | }t|d|df}||}t||}n ||| }n8|jrZ|| | }||}n ||| }t||}|jr|| | fS|| fSforward processrNr) _check_inputrrrrHrQrrrrWr'rr(dependr2rrRrPrrrrrT)r6rr init_resetr)r<input_x key_reset value_resetr2r1output_xaux_loss mlp_logit value_update key_updater-r.rr7r7r8r st  ""                  z!TransformerEncoderLayer.constructc Cstt|dtjtjg|j|durDtt|dtjtjg|jt|t}|du}t|t}|du}t |j d|jd||t |j d|jd|||j rtt|dtj g|jdS)r4rNrTr^r)r5) r6rrr^r)init_reset_is_tensorinit_reset_is_defaultr6r7r7r7r8r\^s   z$TransformerEncoderLayer._check_input)NTNrBrCrDrEr r`r rrrr rdrr5rrrrrrrrrr9rr\rr7r7rr8r(sNp    B Ur(cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j eejejgdeejejgdeejejgdeeegde j d ddddejejejdeef fdd Zdd d Zd dZZS)r)a Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) rn) rrrrsrrrrrHrIrrurvrrFrwc sztt|t|||jdk|_|jr0|jn|}|s<| rFt|t t j fvrt |||j dkr~td||j ||j dkrtd||j ||j dkrtd||j | rtd|jd||_| |_| |_||_||_| |_||_t|f| |_t|f| |_t|||||||| | | |d |_t|||||||| | | |d |_t|f| |_|jrt|||| |||d |_ nt!||||| |d |_ | |_"t#$|_%t#$|_&t'j(|_)d|_*d|_+|jrvt#,-d |_.t#/-d |_0t#1-d |_2||}||||f|_3||||f|_4t5t6t7j8|j3d|j)dd|_*t5t6t7j8|j4d|j)dd|_+t#9-d|_:t#;-d|_nt t j fvrdt |||j dkrtd||j ||j dkrtd||j ||j dkr(td||j | r@td|jd||_| |_| |_||_||_| |_||_t|f| |_|j-|j?dfft|f| |_|j-|j?dfft|||||||| | | |d |_t|||||||| | | |d |_t|f| |_|j-|j?dff|jr>t|||| |||d |_ nt!||||| |d |_ | |_"t#$-|j?df|j?dff|_%t#$-|j?ddf|j?ddff|_&t'j(|_)d|_*d|_+|jr~t#,-d |_.t#/-d |_0t#1-d |_2||}||||f|_3||||f|_4t5t6t7j8|j3d|j)dd|_*t5t6t7j8|j4d|j)dd|_+t#9-d|_:t#;-d|_|j@rv|j%-|j?|j df|j?|j dff|j-|j?|j dff|j-|j?|j dff|jAjBr|jjCA|jsv|j jDj-|j?|j dfdf|j?|j f|j dff|j?|j dffd|j jEjE-|j?|j dffntFd|jddS)Nr.rzFor 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by 'parallel_config.model_parallel', but got the num_heads is {} and parallel_config.model_parallel is {}.zFor 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config.model_parallel is {}.zFor 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} and parallel_config.model_parallel is {}.rNz does not support use_past=True.) rrrrrrrrrrrurv) rrrrrrrrrrrurvrJ)rrrtrsrrurvrrrr'rLr(rrMrrrrO)Grr)r9rr|rPrSr rdr rrrr1rrrrrrrrrrrrrQrRr$r2cross_attentioncross_attention_layernormrrr%rHrrrrTrrrr'r(rrrrrrrr9r:rrrrUrr rrrVrWr/r0rHr2rXrrrY)r6rrrsrrrrrrrHrrIrrurrKrvZconfig_to_attentionrrr7r8r9s#              $      z TransformerDecoderLayer.__init__NTc Cs|||||||t|}t|d|df}||}t||j}d} d} |jr||j | |j t||j|j } ||j | |j t||j|j } t || }t || }| |||||j |j |\} } |jr||| } n ||| } d}|durz|| }t||j}t||j}||||||j |j |\}}| |7} |jrn|||} n || |} || }t||j}d}|jr||\}}n ||}d}d}|jr| \}}||j ||j }||j ||j }t || }t || }t ||}t ||}t|dkrt||}t||}t| |} |jrv|||}n || |}n.|jr|||}n || |}t||}|jr|| |fS|| fSrZ)r\rrrrQrrrrWr'rr(r]r2rHrrkrjrRrPrrrrrT)r6Z hidden_statsZ decoder_maskencoder_output memory_maskr^r)Z hidden_shaper_r`rar2r1rZ middle_outputZcross_attn_outputZcross_layer_presentrbrcrdrerfr-r.rr7r7r8rs  ""               z!TransformerDecoderLayer.constructc Cs tt|dtjtjg|j|durDtt|dtjtjg|j|durjtt|dtjtjg|j|durtt|dtjtjg|jt|t}|du}t|t} |du} t |j d|jd||t |j d|jd| | |j rtt|dtj g|jtt|dtj g|jdS) r4 hidden_statesNrrlrmTr^r)) rrrrrrrr\rrrbool_r) r6rnrrlrmr^r)rgrhr6r7r7r7r8r\+ s6    z$TransformerDecoderLayer._check_input)NNTNrir7r7rr8r)rsVZ    m ar)csfdd}|S)a A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. c sr}||jkr(td|d|jt||jd}t||||jd}||_t||jd}||||dt|jtr|jr|j s|n0|jjr|jj s|jj }|j||jj |jj ddS)a Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. zlayers z$ must be larger than pipeline stage r.)rIrJrKN) rSrmaxminrWset_comm_fusionr\rHr]r2rIrJrK) networklayer_idoffsetrvlayersZpp_disZpp_iddisZparalel_op_comm_compute total_layerr7r8!_set_parallel_configure_for_layerP s$     z;_get_lambda_func.._set_parallel_configure_for_layerr7)ryrzr7rxr8_get_lambda_funcG s $r{cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jdddddejejejdddeef fd d Zdd d ZZS)r&a! Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) rn)rrrrsr num_layersrurrrHrIrrurvrrrwFNrcstt|t|t|||jdk|_|s4|r>t||jrJ|j n|j }t t j fvrt|_tdtj|_||_t|_t|D]R}t||||||| | || | | |||d}|st}||||||d|j|qnt t j fvrtd|_tdtj|_td||_t|_t|D]V}t||||||| | || | | |||d}|sjt}||||||d|j|q4nt d|j!ddS) Nr.)rrrrsrrrrIrrrrHrurrKrvrtrvrurvr7r7For parallel mode, sharding propagation is recommended, you can use it by setting 'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, search_mode="sharding_propagation")' and 'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'rNrO)"rr&r9rrr|rPr rdrlrAr rrrrrrrrrcr|rCellListblocksrr(r{appendrr`rarYr)r6rr|rrrsrrrrrrHrIrru lambda_funcrurrKrvconfig_to_layeriblockrr7r8r9 s'         zTransformerEncoder.__init__Tc Csd}|jrZ|j}t|jD]4}|j|||||\}}} ||f}||| }q|||fSt|jD]&}|j|||||\}}||f}qd||fSr[r7rPrcrr|rr) r6rnrr^r) present_layer accum_lossrpresentrcr7r7r8ra s(      zTransformerEncoder.construct)TNrBrCrDrEr r`r rrrr rdcheck_non_negative_intrr5rrrrrr+rdefault_transformer_configr9rrr7r7rr8r&w sR{    Zr&cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jddddejejejddddeef fd d Zdd d ZZS)r'aS Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) rn)rrrrsrr|rrurrrHrIrrurvrrFrwNrcstt|t||t||jdk|_|s4|r>t||jrJ|j n|j }t t j fvrt|_tdtj|_||_t|_t|D]T}t||||||| || | ||| | ||d}|st}||||||d|j|qnt t j fvrtd|_tdtj|_td||_t|_t|D]X}t||||||| || | ||| | ||d}|snt}||||||d|j|q6nt d|j!ddS) Nr.r})rrrrsrrrrrrIrrrrurHrKrvr~rrrNrO)"rr'r9rrr|rPr rdrlrAr rrrrrrrrrcr|rrrrr)r{rrr`rarYr)r6r|rrrrsrrrrrrHrIrrurrrrurKrvrrrrr7r8r9 s)         zTransformerDecoder.__init__Tc Csd}|jr^|j}t|jD]8} |j| ||||||\}} } || f}||| }q|||fSt|jD]*} |j| ||||||\}} || f}qh||fSrr) r6rnrrlrmr^r)rrrrrcr7r7r8rU s0      zTransformerDecoder.construct)NNTNrr7r7rr8r'y sXf    `r'cseZdZdZeedeejfvde e j e j e j e j e j e j e j e j e j e jeejejgdeejejgdeejejgdeegde jddddddddejejejd deeffd d Zdd dZZS)r*a# Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) rn)rrrrsrencoder_layersdecoder_layersrrrrHrIrrurvrrrrrwFNcstt||s|r t|ttjfvrlt|||_ ||_ ||_ ||_ ||_ |dkrj|krnntd|d|d|dkr|dkr|rtd|jd|st||d}t|||jdk|_t|_td tj|_|dkr"t||||||| | | | || |||||d |_nd|_d|_|dkrt|||||||| | | | | |||||||d |_nxttjfvrt|||_ ||_ ||_ ||_ ||_ |dkr|krnntd|d|d|dkr|dkr|rtd|jdt d |st||d}t|||jdk|_t!d |_td tj|_|dkrt||||||| | | | || |||||d |_nd|_d|_|dkrt|||||||| | | | | |||||||d |_nt"d|jddS)Nrz(Transformer doest support encoder layer z and decoderlayer z, please use TransformerDecoderrNz9 with encoder and decoder does not support use_past=True.rxr.r})r|rrrrsrrrrrrIrrHrurrrKrv)r|rrrrsrrrrrrrHrIrrrrururKrvrrrO)#rr*r9r rdr rrrrrrrrrrrr{rr|rPrrrrrrrcr&encoderdecoderr'r`rarrY)r6rrrrsrrrrrrrrrHrIrrurrrKrvrr7r8r9 s'                   zTransformer.__init__TcCsd}d} d} d} |j} |jdurb|jrJ|||||\}} } || | } n|||||\}} |} |jdur|jr|||||||\}} }|| |} n|||||||\}} |} |jr| | | | fS| | | fS)zprocess processN)rcrrPrr)r6Zencoder_inputsZ encoder_masksZdecoder_inputsZ decoder_masksrmr^r)rlrZencoder_layer_presentZdecoder_layer_presentrZencoder_aux_lossZdecoder_outputZdecoder_aux_lossr7r7r8r s@      zTransformer.construct)NNNTNrr7r7rr8r*s s\|     r*)N)MrE __future__rrnumpyrmindspore.common.tensorrmindspore.common.parameterrmindspore.common.initializerr mindsporerrZmindspore.common.dtypecommonrr mindspore.opsrrr rmindspore.nn.cellr Zmindspore._checkparamr ImportError _checkparamr r`mindspore.parallel._utilsr mindspore.contextrZmindformers.modules.layersrrrrrrr2mindformers.modules.transformer.op_parallel_configrrrrrrZ#mindformers.modules.transformer.moerrrZmindformers.version_controlrmindformers.tools.loggerr __all__r,r-rmr+rrr%r!r"r#r$r(r)r{r&r'r*r7r7r7r8sr             $   ARwKJVv0X 0{pengcheng.mind-mindformer/mindformers/modules/transformer/moe.py000066400000000000000000001132131476650141400256040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Mixture of Expert (MoE) structure. This is an experimental interface that is subject to change or deletion. """ from __future__ import absolute_import from __future__ import division import math import numpy as np from mindspore.common.tensor import Tensor import mindspore.common.dtype as mstype import mindspore.communication.management as D # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.ops.primitive import constexpr from mindspore.nn.cell import Cell from mindspore.nn.layer import Dense from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode, _is_sharding_propagation from mindformers.modules.transformer.op_parallel_config import default_moeparallel_config __all__ = [ "MoEConfig"] class MoEConfig: r""" The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import MoEConfig >>> moe_config = MoEConfig(expert_num=4, capacity_factor=5.0, aux_loss_factor=0.05, num_experts_chosen=1, ... expert_group_size=64, group_wise_a2a=True, comp_comm_parallel=False, ... comp_comm_parallel_degree=2) """ def __init__(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): Validator.check_positive_int(expert_num, "expert_num") Validator.check_positive_float(capacity_factor, "capacity_factor") Validator.check_positive_float(aux_loss_factor, "aux_loss_factor") Validator.check_positive_int(num_experts_chosen, "num_experts_chosen") Validator.check_bool(group_wise_a2a, "group_wise_a2a") Validator.check_bool(comp_comm_parallel, "comp_comm_parallel") Validator.check_positive_int(comp_comm_parallel_degree, "comp_comm_parallel_degree") if expert_group_size is not None: Validator.check_positive_int(expert_group_size, "expert_group_size") if capacity_factor < 1.0: raise ValueError(f"'capacity_factor' must be equal to or greater than 1.0, " f"but got {capacity_factor}.") if aux_loss_factor >= 1.0: raise ValueError(f"'aux_loss_factor' must be less than 1.0, " f"but got {aux_loss_factor}.") if num_experts_chosen > expert_num: raise ValueError(f"'num_experts_chosen' must not be larger than 'expert_num', " f"but got {num_experts_chosen}.") self.expert_num = expert_num self.capacity_factor = capacity_factor self.aux_loss_factor = aux_loss_factor self.num_experts_chosen = num_experts_chosen self.expert_group_size = expert_group_size self.group_wise_a2a = group_wise_a2a self.comp_comm_parallel = comp_comm_parallel self.comp_comm_parallel_degree = comp_comm_parallel_degree default_moe_config = MoEConfig() def _check_moe_config(moe_config=None, parallel_config=None): """ check if MoE with right configuration. """ if not isinstance(moe_config, MoEConfig): raise TypeError(f"'moe_config' must be an instance of MoEConfig, but got {type(moe_config).__name__}.") use_moe = (moe_config.expert_num > 1) if use_moe is False: return if moe_config.expert_num % parallel_config.expert_parallel != 0: raise ValueError(f"When using MoE, the 'expert_num' in {type(moe_config).__name__} must be a multiple " f"of 'expert_parallel' value in {type(parallel_config).__name__}, but got " f"{moe_config.expert_num} for 'expert_num' and {parallel_config.expert_parallel} for " f"'expert_parallel'.") device_num = D.get_group_size() if device_num % parallel_config.expert_parallel != 0: raise ValueError(f"device_num: {device_num} must be a multiple of expert_parallel: " f"{parallel_config.expert_parallel}.") if parallel_config.data_parallel % parallel_config.expert_parallel != 0: raise ValueError(f"data parallel: {parallel_config.data_parallel} must be a multiple of " f"expert_parallel: {parallel_config.expert_parallel} when using MoE.") if parallel_config.data_parallel * parallel_config.model_parallel > device_num: raise ValueError(f"The product of the data parallel: {parallel_config.data_parallel} and " f"model parallel: {parallel_config.model_parallel} " f"should be less than device_num: {device_num}.") @constexpr def calculate_expert_capacity(k, tokens_per_group, capacity_factor, expert_dim): return math.ceil(k * tokens_per_group * capacity_factor / expert_dim) class MoE(Cell): """ The mixture of experts (MoE) implementation. The implementation includes a router and a FeedForward layer. The router dispatches tokens to experts in FeedForward, then FeedForward does computation, and the final output is obtained by multiplying FeedForward's output and router's combine weight. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. Default: gelu. param_init_type (dtype.Number): The parameter initialization type. Can be dtype.float32 or dtype.float16. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(MoEParallelConfig): The parallel config for MoE, see `MoEParallelConfig`. Default `default_moeparallel_config`, an instance of `MoEParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size]`. """ def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', param_init_type=mstype.float32, moe_config=default_moe_config, parallel_config=default_moeparallel_config): super(MoE, self).__init__() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): self.hidden_size = hidden_size self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.aux_loss_factor = moe_config.aux_loss_factor self.num_experts_chosen = moe_config.num_experts_chosen self.expert_group_size = moe_config.expert_group_size self.dp_group = parallel_config.data_parallel self.dp = parallel_config.data_parallel self.ep = parallel_config.expert_parallel self.mp = parallel_config.model_parallel self.comp_comm_parallel = moe_config.comp_comm_parallel self.comp_comm_parallel_degree = moe_config.comp_comm_parallel_degree self.group_wise_a2a = moe_config.group_wise_a2a if not (self.mp > 1 and self.dp == self.ep): self.group_wise_a2a = False from mindformers.modules.transformer import FeedForward self.ffn = FeedForward(hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=self.expert_dim, expert_group_size=self.expert_group_size, param_init_type=param_init_type, parallel_config=parallel_config) self.reshape = P.Reshape() self.shape = P.Shape() self.transpose_2dim = P.Transpose().shard(((self.dp, 1),)) self.transpose_3dim = P.Transpose().shard(((self.dp, 1, 1),)) self.transpose_4dim = P.Transpose().shard(((1, self.dp, 1, 1),)) self.transpose_4dim_dp = P.Transpose().shard(((1, 1, self.dp, 1),)) self.batch_mm = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.batch_mm2 = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.mul = P.Mul() self.router = Router(d_model=hidden_size, moe_config=moe_config, routing_policy=None, training=True, parallel_config=parallel_config) self.cast = P.Cast() self.concat = P.Concat(3).shard(tuple((self.dp, 1, 1, 1) for _ in range(self.comp_comm_parallel_degree))) self.concat_dp = P.Concat(2).shard(((1, self.dp, 1, 1), (1, self.dp, 1, 1))) self.split = P.Split(axis=2, output_num=self.comp_comm_parallel_degree).shard(((1, self.dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((self.dp, 1, 1, 1),)) self.stride_slice_dp = P.StridedSlice().shard(((1, self.dp, 1, 1),)) self.stride_slice_ep = P.StridedSlice().shard(((self.ep, 1, 1, 1),)) self.stride_slice_dp_mp = P.StridedSlice().shard(((1, self.dp, self.mp, 1),)) self.stride_slice_ep_mp = P.StridedSlice().shard(((self.ep, 1, self.mp, 1),)) else: self.hidden_size = hidden_size self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.aux_loss_factor = moe_config.aux_loss_factor self.num_experts_chosen = moe_config.num_experts_chosen self.dp_group = parallel_config.data_parallel self.dp = parallel_config.data_parallel self.ep = parallel_config.expert_parallel self.mp = parallel_config.model_parallel self.comp_comm_parallel = moe_config.comp_comm_parallel self.comp_comm_parallel_degree = moe_config.comp_comm_parallel_degree self.group_wise_a2a = moe_config.group_wise_a2a if not (self.mp > 1 and self.dp == self.ep): self.group_wise_a2a = False from mindformers.modules.transformer import FeedForward self.ffn = FeedForward(hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, dropout_rate=dropout_rate, hidden_act=hidden_act, expert_num=self.expert_dim, param_init_type=param_init_type, parallel_config=parallel_config) self.reshape = P.Reshape() self.shape = P.Shape() self.transpose_2dim = P.Transpose().shard(((self.dp, 1),)) self.transpose_3dim = P.Transpose().shard(((self.dp, 1, 1),)) self.transpose_4dim = P.Transpose().shard(((1, self.dp, 1, 1),)) self.transpose_4dim_dp = P.Transpose().shard(((1, 1, self.dp, 1),)) self.batch_mm = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.batch_mm2 = P.BatchMatMul().shard(((self.dp, 1, 1), (self.dp, 1, 1))) self.mul = P.Mul().shard(((), ())) self.router = Router(d_model=hidden_size, moe_config=moe_config, routing_policy=None, training=True, parallel_config=parallel_config) self.cast = P.Cast() self.concat = P.Concat(3).shard(tuple((self.dp, 1, 1, 1) for _ in range(self.comp_comm_parallel_degree))) self.concat_dp = P.Concat(2).shard(((1, self.dp, 1, 1), (1, self.dp, 1, 1))) self.split = P.Split(axis=2, output_num=self.comp_comm_parallel_degree).shard(((1, self.dp, 1, 1),)) self.stride_slice = P.StridedSlice().shard(((self.dp, 1, 1, 1),)) self.stride_slice_dp = P.StridedSlice().shard(((1, self.dp, 1, 1),)) self.stride_slice_ep = P.StridedSlice().shard(((self.ep, 1, 1, 1),)) self.stride_slice_dp_mp = P.StridedSlice().shard(((1, self.dp, self.mp, 1),)) self.stride_slice_ep_mp = P.StridedSlice().shard(((self.ep, 1, self.mp, 1),)) def ffn_infer(self, expert_input, capacity): """ Computing the FFN. """ pad_size = 0 if self.group_wise_a2a: # If capacity can't div by mp, pad for mp shard. if capacity%self.mp != 0: pad_size = self.mp-(capacity%self.mp) if pad_size != 0: capacity += pad_size pad_tensor = self.stride_slice_dp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, pad_size, self.hidden_size), (1, 1, 1, 1)) expert_input = self.concat_dp((expert_input, pad_tensor)) # capacity shard by mp expert_input = self.stride_slice_dp_mp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # group-wise alltoall expert_input = self.stride_slice_ep_mp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # allgather expert_input = self.stride_slice_ep(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) expert_input = self.reshape(expert_input, (self.expert_dim * self.dp_group * capacity, self.hidden_size)) # expert_output's shape: (self.expert_dim, self.dp_group*expert_capacity, self.hidden_size) expert_output = self.ffn(expert_input) expert_output = self.reshape(expert_output, (self.expert_dim, self.dp_group, capacity, self.hidden_size)) if self.group_wise_a2a: # capacity shard by mp expert_output = self.stride_slice_ep_mp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # group-wise alltoall expert_output = self.stride_slice_dp_mp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # allgather expert_output = self.stride_slice_dp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # Slice capacity back to org shape. if pad_size != 0: capacity -= pad_size expert_output = self.stride_slice_dp(expert_output, (0, 0, 0, 0), (self.expert_dim, self.dp_group, capacity, self.hidden_size), (1, 1, 1, 1)) # expert_output's shape: (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity) expert_output = self.transpose_4dim(expert_output, (1, 3, 0, 2)) return expert_output def ffn_parallel_infer(self, expert_input, capacity): """ Split and overlap FFN compute and communication. """ # Pad capacity for comp_comm_parallel_degree split. pad_size = 0 if capacity%self.comp_comm_parallel_degree != 0: pad_size = self.comp_comm_parallel_degree-(capacity%self.comp_comm_parallel_degree) capacity += pad_size pad_tensor = self.stride_slice_dp(expert_input, (0, 0, 0, 0), (self.expert_dim, self.dp_group, pad_size, self.hidden_size), (1, 1, 1, 1)) expert_input = self.concat_dp((expert_input, pad_tensor)) sub_capacity = capacity // self.comp_comm_parallel_degree output_list = [] for sub_expert_input in self.split(expert_input): sub_expert_output = self.ffn_infer(sub_expert_input, sub_capacity) output_list.append(sub_expert_output) expert_output = self.concat(output_list) # Slice capacity back to org shape. if pad_size != 0: capacity -= pad_size expert_output = self.stride_slice(expert_output, (0, 0, 0, 0), (self.dp_group, self.hidden_size, self.expert_dim, capacity), (1, 1, 1, 1)) return expert_output def construct(self, input_tensor): """forward process""" input_shape = F.shape(input_tensor) input_tensor = self.reshape(input_tensor, (-1, self.hidden_size)) bs_and_dmodel = self.shape(input_tensor) tokens_per_group = bs_and_dmodel[0] // self.dp_group input_tensor = self.reshape(input_tensor, (self.dp_group, tokens_per_group, self.hidden_size)) expert_capacity = calculate_expert_capacity(self.num_experts_chosen, tokens_per_group, self.capacity_factor, self.expert_dim) # dispatch_tensor's shape: (self.dp_group, tokens_per_group, self.expert_dim, expert_capacity) # combine_tensor's shape: (self.dp_group, tokens_per_group, self.expert_dim, expert_capacity) dispatch_tensor, combine_tensor, aux_loss = self.router(input_tensor) # after transpose, input_tensor's shape: (self.dp_group, self.hidden_size, tokens_per_group) input_tensor = self.transpose_3dim(input_tensor, (0, 2, 1)) dispatch_tensor = self.reshape(dispatch_tensor, (self.dp_group, tokens_per_group, self.expert_dim * expert_capacity)) dispatch_tensor = self.cast(dispatch_tensor, F.dtype(input_tensor)) # expert_input's shape: (self.dp_group, self.hidden_size, self.expert_dim * expert_capacity) expert_input = self.batch_mm(input_tensor, dispatch_tensor) expert_input = self.reshape(expert_input, (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity)) # The following four ops are to implement transpose(expert_input, (2, 0, 3, 1)), for that a single transpose # has bad performance expert_input = self.reshape(expert_input, (self.dp_group * self.hidden_size, self.expert_dim * expert_capacity)) expert_input = self.transpose_2dim(expert_input, (1, 0)) expert_input = self.reshape(expert_input, (self.expert_dim, expert_capacity, self.dp_group, self.hidden_size)) # expert_input's shape: (self.expert_dim, self.dp_group, expert_capacity, self.hidden_size) expert_input = self.transpose_4dim_dp(expert_input, (0, 2, 1, 3)) # expert_output's shape: (self.dp_group, self.hidden_size, self.expert_dim, expert_capacity) if self.comp_comm_parallel: expert_output = self.ffn_parallel_infer(expert_input, expert_capacity) else: expert_output = self.ffn_infer(expert_input, expert_capacity) expert_output = self.reshape(expert_output, (self.dp_group, self.hidden_size, self.expert_dim * expert_capacity)) combine_tensor = self.reshape(combine_tensor, (self.dp_group, tokens_per_group, self.expert_dim * expert_capacity)) # combine_tensor's shape: (self.dp_group, self.expert_dim*expert_capacity, tokens_per_group) combine_tensor = self.transpose_3dim(combine_tensor, (0, 2, 1)) combine_tensor = self.cast(combine_tensor, F.dtype(expert_output)) # combined_output's shape: (self.dp_group, self.hidden_size, tokens_per_group) combined_output = self.batch_mm2(expert_output, combine_tensor) # combined_output's shape: (self.dp_group, tokens_per_group, self.hidden_size) combined_output = self.transpose_3dim(combined_output, (0, 2, 1)) combined_output = self.reshape(combined_output, (bs_and_dmodel[0], bs_and_dmodel[1])) combined_output = self.reshape(combined_output, input_shape) aux_loss = self.mul(self.aux_loss_factor, aux_loss) return combined_output, aux_loss class Router(Cell): r""" A router backbone used to calculate logits of each token, which should be cascaded by router implementations mapping tokens to experts. when moe_config.num_experts_chosen = 1, use top1 routing; when moe_config.num_experts_chosen > 1, use topk routing Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). routing_policy: The policy of mapping tokens to experts. Default: topkRouter training (bool): The value indicating whether is in training phase. parallel_config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim)`. """ def __init__(self, d_model, moe_config, routing_policy=None, training=True, parallel_config=None): super(Router, self).__init__() dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.num_experts_chosen = moe_config.num_experts_chosen self.training = training self.routing_policy = routing_policy self.noisy_policy = None # candidate: ["jitter", "rsample", "None"] self.noisy_epsilon = 1e-2 self.noise = Tensor(np.random.uniform(1 - self.noisy_epsilon, 1 + self.noisy_epsilon, (d_model,))) self.dense = Dense(in_channels=self.d_model, out_channels=self.expert_dim, has_bias=False) self.dense.matmul.shard(((dp, 1), (1, 1))) self.mul = P.Mul() self.cast = P.Cast() if self.routing_policy is None: self.router = TopkRouter(d_model=d_model, moe_config=moe_config, training=training, parallel_config=parallel_config) else: self.router = routing_policy if not (_get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation()): self.mul.shard(((dp, 1, 1), (dp,))) def construct(self, input_tensor): input_tensor = self.cast(input_tensor, mstype.float32) if self.noisy_policy == "jitter" and self.training: # Here, we temporarily implement the multiplicative jitter this way, # for the lack of UniforReal parallel operator. input_tensor = self.mul(input_tensor, self.noise) router_logits = self.dense(input_tensor) return self.router(router_logits) class TopkRouter(Cell): r""" A router implementation which maps each tokens to the topk expert. Args: d_model (int): The hidden size of each token. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). training (bool): The value indicating whether is in training phase. config: The parallel-related configuration. Inputs: - **input_tensor** (Tensor) - Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, hidden\_size)`. Outputs: Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(expert\_parallel, tokens\_per\_device, expert\_dim, expert\_capacity)`, Tensor of shape :math:`(1)`. """ def __init__(self, d_model, moe_config, training=True, parallel_config=None): super(TopkRouter, self).__init__() if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,) and _is_sharding_propagation(): dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.training = training self.dp_group = dp self.noisy_policy = None self.cast = P.Cast() self.reshape = P.Reshape() self.shape = P.Shape() self.softmax = P.Softmax(axis=-1) self.argmax = P.ArgMaxWithValue(axis=-1, keep_dims=False) self.num_experts_chosen = moe_config.num_experts_chosen self.onehot = P.OneHot() self.onehot2 = P.OneHot() self.onehot3 = P.OneHot() self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_mean = P.ReduceMean(keep_dims=False) self.reduce_mean2 = P.ReduceMean(keep_dims=False) self.reduce_mean3 = P.ReduceMean(keep_dims=False) self.mul = P.Mul() self.mul2 = P.Mul() self.mul3 = P.Mul() self.mul4 = P.Mul() self.mul5 = P.Mul() self.mul6 = P.Mul() self.mul7 = P.Mul() self.mul8 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul9 = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.not_equal = P.NotEqual() self.div1 = P.RealDiv() self.div2 = P.RealDiv() self.add = P.Add() self.add1 = P.Add() self.add2 = P.Add() self.add3 = P.Add() self.add4 = P.Add() self.sub = P.Sub() self.cumsum = P.CumSum(exclusive=True) self.less = P.Less() self.reduce_sum = P.ReduceSum(keep_dims=False) self.reduce_sum_keep = P.ReduceSum(keep_dims=True) self.reduce_sum_keep2 = P.ReduceSum(keep_dims=True) self.expand = P.ExpandDims() self.expand2 = P.ExpandDims() self.add_scala = P.Add() self.init_loss = Tensor(0.0, mstype.float32) else: dp = parallel_config.data_parallel self.d_model = d_model self.expert_dim = moe_config.expert_num self.capacity_factor = moe_config.capacity_factor self.training = training self.dp_group = dp self.noisy_policy = None self.cast = P.Cast() self.reshape = P.Reshape() self.shape = P.Shape() self.softmax = P.Softmax(axis=-1).shard(((dp, 1, 1,),)) self.argmax = P.ArgMaxWithValue(axis=-1, keep_dims=False).shard(((dp, 1, 1),)) self.num_experts_chosen = moe_config.num_experts_chosen self.onehot = P.OneHot().shard(((dp, 1, 1), (), ())) self.onehot2 = P.OneHot().shard(((dp, 1, 1), (), ())) self.onehot3 = P.OneHot().shard(((dp, 1, 1, 1), (), ())) self.on_value = Tensor(1.0, mstype.float32) self.off_value = Tensor(0.0, mstype.float32) self.reduce_mean = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_mean2 = P.ReduceMean(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_mean3 = P.ReduceMean(keep_dims=False).shard(((dp, 1),)) self.mul = P.Mul().shard(((dp, 1), (dp, 1))) self.mul2 = P.Mul().shard(((), ())) self.mul3 = P.Mul().shard(((), ())) self.mul4 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul5 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul6 = P.Mul().shard(((dp, 1), (dp, 1))) self.mul7 = P.Mul().shard(((dp, 1), (dp, 1))) self.mul8 = P.Mul().shard(((dp, 1, 1), (dp, 1, 1))) self.mul9 = P.Mul().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.not_equal = P.NotEqual().shard(((dp, 1, 1, 1), ())) self.div1 = P.RealDiv().shard(((dp, 1, 1), (dp, 1, 1))) self.div2 = P.RealDiv().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.add = P.Add().shard(((dp, 1, 1), (dp, 1, 1))) self.add1 = P.Add().shard(((dp, 1, 1), ())) self.add2 = P.Add().shard(((dp, 1, 1, 1), (dp, 1, 1, 1))) self.add3 = P.Add().shard(((dp, 1), (dp, 1))) self.add4 = P.Add().shard(((dp, 1, 1, 1), ())) self.sub = P.Sub().shard(((), (dp, 1, 1))) self.cumsum = P.CumSum(exclusive=True).shard(((dp, 1, 1),)) self.less = P.Less().shard(((dp, 1, 1), ())) self.reduce_sum = P.ReduceSum(keep_dims=False).shard(((dp, 1, 1),)) self.reduce_sum_keep = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1),)) self.reduce_sum_keep2 = P.ReduceSum(keep_dims=True).shard(((dp, 1, 1, 1),)) self.expand = P.ExpandDims().shard(((dp, 1),)) self.expand2 = P.ExpandDims().shard(((dp, 1, 1),)) self.add_scala = P.Add().shard(((), ())) self.init_loss = Tensor(0.0, mstype.float32) def construct(self, router_logits): """forward process""" router_logits_shape = self.shape(router_logits) router_logits = self.reshape(router_logits, (-1, router_logits_shape[-1])) logits_shape = self.shape(router_logits) tokens_per_group = logits_shape[0] // self.dp_group expert_capacity = calculate_expert_capacity(self.num_experts_chosen, tokens_per_group, self.capacity_factor, self.expert_dim) router_logits = self.reshape(router_logits, (self.dp_group, tokens_per_group, self.expert_dim)) accum_expert_mask = 0 accum_expert_gate = 0 loss = self.init_loss mask_count = 0 accum_combine_tensor = 0 # Probabilities for each token of what expert is should be sent to router_prob = self.softmax(router_logits) for expert_chosen_index in range(self.num_experts_chosen): # for each token, set the router_prob of the selected experts to zero router_prob = self.mul4(router_prob, self.sub(self.on_value, accum_expert_mask)) # shape is : (dp_group, tokens_per_group) expert_index, expert_gate = self.argmax(router_prob) # expert_mask's shape: (dp_group, tokens_per_group, self.expert_dim) expert_mask = self.onehot(expert_index, self.expert_dim, self.on_value, self.off_value) # renormalize the rest prob to be of sum 1 router_prob_normal = self.div1(router_prob, self.add1(self.reduce_sum_keep(router_prob, -1), 1e-9)) # the balance loss is computed at each routing step loss = self.add_scala(loss, self._auxiliary_loss(expert_mask, router_prob_normal)) output = self._maskout_overflowed_tokens(expert_mask, expert_capacity, expert_gate, mask_count, expert_chosen_index) expert_mask, expert_gate, expert_mask_flat, position_in_expert = output[0], output[1], output[2], output[3] accum_expert_mask = self.add(accum_expert_mask, expert_mask) accum_expert_gate = self.add3(accum_expert_gate, expert_gate) mask_count = self.add(mask_count, self.reduce_sum_keep(expert_mask, 1)) # combine_tensor's shape: (dp_group, tokens_per_group) combine_tensor = self.mul7(expert_gate, expert_mask_flat) # combine_tensor's shape: (dp_group, tokens_per_group, self.expert_dim) combine_tensor = self.mul8(self.expand(combine_tensor, -1), self.onehot2(expert_index, self.expert_dim, self.on_value, self.off_value)) # combine_tensor's shape: (dp_group, tokens_per_group, self.expert_dim, self.expert_capacity) combine_tensor = self.mul9(self.expand2(combine_tensor, -1), self.onehot3(self.cast(position_in_expert, mstype.int32), expert_capacity, self.on_value, self.off_value)) accum_combine_tensor = self.add2(accum_combine_tensor, combine_tensor) # expert weights normalization combine_tensor_sum = self.reduce_sum_keep2(self.reduce_sum_keep2(accum_combine_tensor, -1), -2) accum_combine_tensor = self.div2(accum_combine_tensor, self.add4(combine_tensor_sum, 1e-9)) # dispatch_tensor is of boolean type. Here, using NotEqual instead of Cast, for that 'Cast to bool' has # bad performance dispatch_tensor = self.not_equal(accum_combine_tensor, 0.0) return dispatch_tensor, accum_combine_tensor, loss def _auxiliary_loss(self, expert_mask, router_prob): """ Computing the load balance loss. """ # density_1's shape: (dp_group, self.expert_dim) density_1 = self.reduce_mean(expert_mask, 1) # density_1_proxy's shape: (dp_group, self.expert_dim) density_1_proxy = self.reduce_mean2(router_prob, 1) loss = self.mul(density_1, density_1_proxy) loss = self.reduce_mean3(loss) loss = self.mul3(self.mul2(loss, self.expert_dim), self.expert_dim) return loss def _maskout_overflowed_tokens(self, expert_mask, expert_capacity, expert_gate, last_num, expert_chosen_index): """ Keeping only the tokens that fit within expert_capacity. """ cumsum = self.cumsum(expert_mask, 1) if expert_chosen_index > 0: cumsum = self.add(cumsum, last_num) # position_in_expert's shape: (dp_group, tokens_per_group, self.expert_dim) position_in_expert = self.mul4(cumsum, expert_mask) less_result = self.less(position_in_expert, expert_capacity) # expert_mask's shape: (dp_group, tokens_per_group, self.expert_dim) expert_mask = self.mul5(less_result, expert_mask) # expert_mask_flat's shape: (dp_group, tokens_per_group) expert_mask_flat = self.reduce_sum(expert_mask, -1) # Mask out the experts that have overflowed the expert_capacity. # expert_gate's shape: (dp_group, tokens_per_group) expert_gate = self.mul6(expert_gate, expert_mask_flat) output = (expert_mask, expert_gate, expert_mask_flat, position_in_expert) return output pengcheng.mind-mindformer/mindformers/modules/transformer/op_parallel_config.py000066400000000000000000000204411476650141400306430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Parallel Config for the Parallel Training This is an experimental interface that is subject to change and/or deletion. """ from __future__ import absolute_import # MindSpore 2.0 has changed the APIs of _checkparam, the following try except is for compatibility try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import context import mindspore.communication.management as D from mindspore.context import ParallelMode from mindspore.parallel._utils import _get_parallel_mode from mindspore import log as logger __all__ = [ "OpParallelConfig" ] class _Config: r""" A basic class of the configure""" def __str__(self): info = "[ParallelConfig]" + '\n' for k, v in self.__dict__.items(): var_info = "{}:{}\n".format(k, v) info += var_info return info class MoEParallelConfig(_Config): r""" MoEParallelConfig for MoE structure, which includes setting data parallel, model parallel and expert parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 expert_parallel (int): The expert parallel way. Default: 1 Supported Platforms: ``Ascend`` """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, use_seq_parallel=False, select_recompute=False): Validator.check_positive_int(data_parallel, "data_parallel") Validator.check_positive_int(model_parallel, "model_parallel") Validator.check_positive_int(expert_parallel, "expert_parallel") Validator.check_bool(use_seq_parallel, "use_seq_parallel") Validator.check_bool(select_recompute, "select_recompute") self._dpmp = OpParallelConfig(data_parallel=data_parallel, model_parallel=model_parallel, use_seq_parallel=use_seq_parallel, select_recompute=select_recompute) self.expert_parallel = expert_parallel self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dpmp.data_parallel @data_parallel.setter def data_parallel(self, value): Validator.check_positive_int(value, "data_parallel") self._dpmp.data_parallel = value @property def model_parallel(self): return self._dpmp.model_parallel @model_parallel.setter def model_parallel(self, value): Validator.check_positive_int(value, "model_parallel") self._dpmp.model_parallel = value @property def expert_parallel(self): return self._expert_parallel @expert_parallel.setter def expert_parallel(self, value): Validator.check_positive_int(value, "expert_parallel") self._expert_parallel = value @property def dpmp(self): """ Get the configuration for dpmp """ return self._dpmp class OpParallelConfig(_Config): r""" OpParallelConfig for the setting data parallel and model parallel. Args: data_parallel (int): The data parallel way. Default: 1 model_parallel (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules import OpParallelConfig >>> config=OpParallelConfig(data_parallel=1, model_parallel=1) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False): Validator.check_positive_int(data_parallel, "data_parallel") Validator.check_positive_int(model_parallel, "model_parallel") Validator.check_bool(use_seq_parallel, "use_seq_parallel") Validator.check_bool(select_recompute, "select_recompute") self.data_parallel = data_parallel self.model_parallel = model_parallel self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._data_parallel @data_parallel.setter def data_parallel(self, value): Validator.check_positive_int(value, "data_parallel") self._data_parallel = value @property def model_parallel(self): return self._model_parallel @model_parallel.setter def model_parallel(self, value): Validator.check_positive_int(value, "model_parallel") self._model_parallel = value class _PipeLineConfig(_Config): r""" PPConfig for the setting data parallel, model parallel Args: pipeline_stage (int): The number of the pipeline stages. Default: 1 micro_batch_num (int): The model parallel way. Default: 1 Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> config=_PipeLineConfig(pipeline_stage=1, micro_batch_num=1) """ def __init__(self, pipeline_stage=1, micro_batch_num=1): Validator.check_positive_int(pipeline_stage, "pipeline_stage") Validator.check_positive_int(micro_batch_num, "micro_batch_num") self.pipeline_stage = pipeline_stage self.micro_batch_num = micro_batch_num @property def pipeline_stage(self): return self._pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): Validator.check_positive_int(value, "pipeline_stage") self._pipeline_stage = value context.set_auto_parallel_context(pipeline_stages=value) @property def micro_batch_num(self): return self._micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): Validator.check_positive_int(value, "micro_batch_num") self._micro_batch_num = value # In case the user doesn't pass a config as args. default_dpmp_config = OpParallelConfig() default_moeparallel_config = MoEParallelConfig() def _check_config(config): """ Check if micro_batch_num >= pipeline_stage """ # the config pipeline_stage is same with context.pipeline_stage pipeline_stage = context.get_auto_parallel_context("pipeline_stages") if hasattr(config, 'pipeline_stage') and pipeline_stage != config.pipeline_stage: raise ValueError( f"The pipeline stage {pipeline_stage} in auto_parallel_context is not equal to the pipeline_stage " f"{config.pipeline_stage}" f" in the config.") # make sure the following is in auto parallel mode is_auto_parallel = _get_parallel_mode() in (ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if not is_auto_parallel: return device_num = D.get_group_size() optimizer_shard = context.get_auto_parallel_context("enable_parallel_optimizer") if config.data_parallel * config.model_parallel * pipeline_stage > device_num: raise ValueError(f"The product of the data parallel {config.data_parallel}, " f"model parallel {config.model_parallel} " f"pipeline stages {pipeline_stage} " f"should be less than device_num {device_num}.") # the config optimizer_shard is same with context.optimizer_shard if hasattr(config, "optimizer_shard") and optimizer_shard and optimizer_shard != config.optimizer_shard: logger.warning(f"The optimizer shard {optimizer_shard} in auto_parallel_context is not equal to the" f" optimizer_shard {config.optimizer_shard} in the OpParallelConfig. Please check the " f"optimizer_shard to make them consistent.") pengcheng.mind-mindformer/mindformers/modules/transformer/transformer.py000066400000000000000000005737651476650141400274140ustar00rootroot00000000000000# Copyright 2021-2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Transformer Networks. This is interface that is subject to change or deletion. """ from __future__ import absolute_import import math import numpy as np from mindspore.common.tensor import Tensor from mindspore.common.parameter import Parameter from mindspore.common.initializer import initializer from mindspore import nn from mindspore import context import mindspore.common.dtype as mstype from mindspore.ops import operations as P from mindspore.ops import functional as F from mindspore.nn.cell import Cell try: from mindspore._checkparam import Validator except ImportError: import mindspore._checkparam as Validator from mindspore import log as logger from mindspore.parallel._utils import _get_parallel_mode from mindspore.context import ParallelMode from mindformers.modules.layers import LayerNorm, Linear, \ _args_type_validator_check, _valid_type_checks, _valid_value_checks, \ _check_past_none_input_none, _check_input_dtype from mindformers.modules.transformer.op_parallel_config import default_dpmp_config, _PipeLineConfig, OpParallelConfig, \ _Config, _check_config, MoEParallelConfig from mindformers.modules.transformer.moe import default_moe_config, MoE, _check_moe_config from mindformers.version_control import get_dropout from mindformers.tools.logger import _LogActionOnce __all__ = [ "AttentionMask", "AttentionMaskHF", "VocabEmbedding", "MultiHeadAttention", "FeedForward", "TransformerEncoder", "TransformerDecoder", "TransformerEncoderLayer", "TransformerDecoderLayer", "Transformer", "TransformerOpParallelConfig", "EmbeddingOpParallelConfig", "TransformerRecomputeConfig"] class EmbeddingOpParallelConfig(_Config): r""" The parallel config of :class:`VocabEmbedding` for the setting data parallel or model parallel for the embedding table. Args: data_parallel(int): The data parallel way. The input data will be sliced into n parts for embedding layer according to this value. Default: 1. model_parallel(int): The model parallel way. The embedding table parameters will be sliced at 0-th axis according to the model parallel way. Default: 1. vocab_emb_dp(bool): Shard embedding in model parallel or data parallel. If True, the embedding lookup will be a data parallel style training and model_parallel value will be ignored. If false, the embedding table will be sharded into n parts at the 0-th dimension row slice of the embedding table, where the n is the model parallel way determined by this parameter. Default: True Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import EmbeddingOpParallelConfig >>> config=EmbeddingOpParallelConfig(data_parallel=1, model_parallel=1, vocab_emb_dp=True) """ def __init__(self, data_parallel=1, model_parallel=1, use_seq_parallel=False, select_recompute=False, vocab_emb_dp=True): self._dp_mp_config = OpParallelConfig(data_parallel=data_parallel, use_seq_parallel=use_seq_parallel, model_parallel=model_parallel, select_recompute=select_recompute) Validator.check_bool(vocab_emb_dp, "vocab_emb_dp") self.vocab_emb_dp = vocab_emb_dp self.use_seq_parallel = use_seq_parallel self.select_recompute = select_recompute @property def data_parallel(self): return self._dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._dp_mp_config.data_parallel = value @property def model_parallel(self): return self._dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._dp_mp_config.model_parallel = value @property def vocab_emb_dp(self): return self._vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): Validator.check_bool(value, "vocab_emb_dp") self._vocab_emb_dp = value @property def dp_mp_config(self): return self._dp_mp_config class TransformerRecomputeConfig(_Config): r""" TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) """ def __init__(self, recompute=False, select_recompute=False, parallel_optimizer_comm_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): Validator.check_bool(recompute, "recompute") Validator.check_bool(parallel_optimizer_comm_recompute, "parallel_optimizer_comm_recompute") Validator.check_bool(mp_comm_recompute, "mp_comm_recompute") Validator.check_bool(select_recompute, "select_recompute") Validator.check_bool(recompute_slice_activation, "recompute_slice_activation") self._recompute = recompute self._select_recompute = select_recompute self._parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self._mp_comm_recompute = mp_comm_recompute self._recompute_slice_activation = recompute_slice_activation @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): Validator.check_bool(value, "recompute") self._recompute = value @property def select_recompute(self): return self._select_recompute @select_recompute.setter def select_recompute(self, value): Validator.check_bool(value, "select_recompute") self._select_recompute = value @property def parallel_optimizer_comm_recompute(self): return self._parallel_optimizer_comm_recompute @parallel_optimizer_comm_recompute.setter def parallel_optimizer_comm_recompute(self, value): Validator.check_bool(value, "parallel_optimizer_comm_recompute") self._parallel_optimizer_comm_recompute = value @property def mp_comm_recompute(self): return self._mp_comm_recompute @mp_comm_recompute.setter def mp_comm_recompute(self, value): Validator.check_bool(value, "mp_comm_recompute") self._mp_comm_recompute = value @property def recompute_slice_activation(self): return self._recompute_slice_activation @recompute_slice_activation.setter def recompute_slice_activation(self, value): Validator.check_bool(value, "recompute_slice_activation") self._recompute_slice_activation = value default_transformer_recompute_config = TransformerRecomputeConfig() class TransformerOpParallelConfig(_Config): r""" TransformerOpParallelConfig for setting parallel configuration, such as the data parallel and model parallel. Note: Except the recompute argument, other arguments will **not** be effective when the user doesn't set auto_parallel_context to `SEMI_AUTO_PARALLEL` or `AUTO_PARALLEL`. The micro_batch_num must be greater than or equal to pipeline_stage when training. The data_parallel\*model_parallel \*pipeline_stage must be equal or less equal to the device. When setting the pipeline stage and optimizer_shard, the config will overwrite the auto_parallel_context. When given the 8 devices and the data_parallel is 1 and model_parallel is 1, the calculation will be repeated on each device. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. recompute (Union[TransformerRecomputeConfig, bool]): The configuration of recomputation for the transformer block. Default: An instance of TransformerRecomputeConfig with default values. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> from mindformers.modules.transformer import TransformerRecomputeConfig >>> recompute_config=TransformerRecomputeConfig(recompute=True, parallel_optimizer_comm_recompute=True, \ ... mp_comm_recompute=True, recompute_slice_activation=True) >>> config=TransformerOpParallelConfig(data_parallel=1, model_parallel=1, recompute=recompute_config) """ def __init__(self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_num=1, recompute=default_transformer_recompute_config, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): self.recompute = recompute self.select_recompute = recompute.select_recompute self.use_seq_parallel = use_seq_parallel self.optimizer_shard = optimizer_shard self.gradient_aggregation_group = gradient_aggregation_group self._embed_dp_mp_config = EmbeddingOpParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, vocab_emb_dp=vocab_emb_dp, use_seq_parallel=use_seq_parallel, select_recompute=recompute.select_recompute) self._pp_config = _PipeLineConfig(pipeline_stage=pipeline_stage, micro_batch_num=micro_batch_num) self._moe_config = MoEParallelConfig( data_parallel=data_parallel, model_parallel=model_parallel, select_recompute=recompute.select_recompute, expert_parallel=expert_parallel, use_seq_parallel=use_seq_parallel) @property def recompute(self): return self._recompute @recompute.setter def recompute(self, value): if not isinstance(value, TransformerRecomputeConfig) and not isinstance(value, bool): raise TypeError(f"recompute must be a TransformerRecomputeConfig/bool, but got {type(value).__name__}.") if isinstance(value, bool): logger.warning(f"TransformerRecomputeConfig is recommended as the recompute configuration type.") self._recompute = value @property def vocab_emb_dp(self): return self._embed_dp_mp_config.vocab_emb_dp @vocab_emb_dp.setter def vocab_emb_dp(self, value): self._embed_dp_mp_config.vocab_emb_dp = value @property def gradient_aggregation_group(self): return self._gradient_aggregation_group @gradient_aggregation_group.setter def gradient_aggregation_group(self, value): Validator.check_positive_int(value, "gradient_aggregation_group") self._gradient_aggregation_group = value @property def micro_batch_num(self): return self._pp_config.micro_batch_num @micro_batch_num.setter def micro_batch_num(self, value): self._pp_config.micro_batch_num = value @property def model_parallel(self): return self._embed_dp_mp_config.model_parallel @model_parallel.setter def model_parallel(self, value): self._embed_dp_mp_config.model_parallel = value self._moe_config.model_parallel = value @property def data_parallel(self): return self._embed_dp_mp_config.data_parallel @data_parallel.setter def data_parallel(self, value): self._embed_dp_mp_config.data_parallel = value self._moe_config.data_parallel = value @property def expert_parallel(self): return self._moe_config.expert_parallel @expert_parallel.setter def expert_parallel(self, value): self._moe_config.expert_parallel = value @property def pipeline_stage(self): return self._pp_config.pipeline_stage @pipeline_stage.setter def pipeline_stage(self, value): self._pp_config.pipeline_stage = value @property def optimizer_shard(self): return self._optimizer_shard @optimizer_shard.setter def optimizer_shard(self, value): Validator.check_bool(value, "optimizer_shard") self._optimizer_shard = value context.set_auto_parallel_context(enable_parallel_optimizer=value) @property def embedding_dp_mp_config(self): return self._embed_dp_mp_config @property def dp_mp_config(self): return self._embed_dp_mp_config.dp_mp_config @property def moe_parallel_config(self): return self._moe_config default_transformer_config = TransformerOpParallelConfig() default_embedding_parallel_config = EmbeddingOpParallelConfig() class FeedForward(Cell): r""" The multilayer perceptron with two linear layers with dropout applied at final output. The first linear will project the input dimension from hidden_size to ffn_hidden_size. The second linear will project the dimension from ffn_hidden_size to hidden_size. The first linear is sharded on the relative dimension, and the second linear is sharded on the output dimension. The overview process can be: .. math:: Dropout((xW_1+b_1)W_2 + b_2) where the :math:`W_1, W_2, b_1` and :math:`b_2` are trainable parameters. Args: hidden_size (int): The dimension of the inputs. ffn_hidden_size (int): The intermediate hidden size. dropout_rate (float): The dropout rate for the second linear's output. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see examples. Default: gelu. expert_num (int): The number of experts used in Linear. For the case expert_num > 1, BatchMatMul is used and the first dimension in BatchMatMul indicate expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. param_init_type (dtype.Number): The parameter initialization type. Should be mstype.float32 or mstype.float16. Default: mstype.float32. parallel_config (OpParallelConfig, MoEParallelConfig): The config of parallel setting, see `OpParallelConfig` or `MoEParallelConfig`. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - should be `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Float tensor. Outputs: Tensor, the output of this layer after mapping. The shape is `[batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]`. Raises: TypeError: `hidden_act` is not a string or nn.Cell. TypeError: `parallel_config` is not a subclass of OpParallelConfig. ValueError: `ffn_hidden_size` is not a multiple of the model parallel way. ValueError: `hidden_size` is not a multiple of the model parallel way. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import FeedForward >>> from mindspore import dtype as mstype >>> from mindspore import Tensor, nn >>> import mindspore.ops as ops >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 2 using custom hidden activation >>> class MyActivationNoShard(nn.Cell): ... def __init__(self): ... super(MyActivationNoShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationNoShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) >>> # Example 3 using custom hidden activation with activation_shard >>> # If user wantss to run on the SEMI/AUTO parallel mode, the custom activation must provide >>> # a class function named activation_shard. It accepts the argument parallel_config (OpParallelConfig, >>> # MoEParallelConfig) and set the shard for the primitives used in the construct. >>> class MyActivationWithShard(nn.Cell): ... def __init__(self): ... super(MyActivationWithShard, self).__init__() ... self.add = ops.Add() ... def construct(self, x): ... return self.add(x, 0.1) ... def activation_shard(self, parallel_config): ... self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) >>> >>> model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, ... hidden_act=MyActivationWithShard) >>> tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> output = model(tensor) >>> print(output.shape) (2, 20, 15) """ @_LogActionOnce(m_logger=logger, key='FeedForward', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, dropout_rate=Validator.check_non_negative_float, param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "FeedForward"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "FeedForward")) def __init__(self, hidden_size, ffn_hidden_size, dropout_rate, hidden_act='gelu', expert_num=1, expert_group_size=None, param_init_type=mstype.float32, parallel_config=default_dpmp_config): super(FeedForward, self).__init__() if hidden_act is None or not (isinstance(hidden_act, str) or issubclass(hidden_act, nn.Cell)): raise TypeError(f"For FeedForward cell, the hidden_act should str type or nn.Cell type, " f"but got {hidden_act}.") if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.cast = P.Cast() else: _check_config(parallel_config) mp = parallel_config.model_parallel if expert_num > 1: ep = parallel_config.expert_parallel else: ep = 1 # ffn use less dp than other ops when use_moe, due to there are ops use dp and ep. dp = parallel_config.data_parallel // ep if ffn_hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'ffn_hidden_size' must be a multiple of the" "num of model parallel, but got the ffn_hidden_size is {} and the num of model " "parallel is {}.".format(ffn_hidden_size, mp)) if hidden_size % mp != 0: raise ValueError("For 'FeedForward', the class variable 'hidden_size' must be a multiple of the num of " "model parallel, but got the hidden_size is {} and the num of model parallel is {}." .format(hidden_size, mp)) if dropout_rate < 0 or dropout_rate >= 1: raise ValueError("For 'FeedForward', the class variable 'dropout_rate' must be in the range [0, 1.0), " "but got the value : {}.".format(dropout_rate)) input_size = hidden_size output_size = ffn_hidden_size # Project to ffn_hidden_size self.mapping = Linear(in_channels=input_size, out_channels=output_size, activation=hidden_act, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.mapping.shard(strategy_matmul=((dp, ep, 1, 1), (ep, 1, mp)), strategy_bias=((dp, ep, 1, mp), (1, ep, 1, mp)), strategy_activation=((dp, ep, 1, mp),)) else: self.mapping.shard(strategy_matmul=((dp, 1), (1, mp)), strategy_bias=((dp, mp), (mp,)), strategy_activation=((dp, mp),)) # Project back to hidden_size self.projection = Linear(in_channels=output_size, out_channels=input_size, transpose_b=False, expert_num=expert_num, expert_group_size=expert_group_size, outer_batch=dp, param_init_type=param_init_type) if expert_num > 1: self.projection.shard(strategy_matmul=((dp, ep, 1, mp), (ep, mp, 1)), strategy_bias=((dp, ep, 1, 1), (1, ep, 1, 1))) else: self.projection.shard(strategy_matmul=((dp, mp), (mp, 1)), strategy_bias=((dp, 1), (1,))) self.projection.bias.parallel_optimizer = False self.dropout = get_dropout(dropout_rate) self.dropout_3d = get_dropout(dropout_rate) self.dropout_4d = get_dropout(dropout_rate) self.dropout.dropout.shard(((dp, 1),)) self.dropout_3d.dropout.shard(((dp, 1, 1),)) self.dropout_4d.dropout.shard(((dp, ep, 1, 1),)) self.cast = P.Cast() def construct(self, x): """Forward process of the FeedForward""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) x = self.cast(x, mstype.float16) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] hidden = self.mapping(x) output = self.projection(hidden) # returned shape is [bs, seq_length, ffn_hidden_size] or [bs * seq_length, ffn_hidden_size] if len(F.shape(output)) == 3: output = self.dropout_3d(output) elif len(F.shape(output)) == 2: output = self.dropout(output) else: output = self.dropout_4d(output) return output class AttentionMask(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length) with 1 and 0, where 1 indicates the current position is a valid token, otherwise not. Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMask >>> from mindspore import Tensor >>> mask = AttentionMask(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [0. 0. 0. 0]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMask', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMask")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMask, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMask""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(self.not_equal(input_mask, 0), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class AttentionMaskHF(Cell): r""" Get the Lower triangular matrix from the input mask. The input mask is a 2D tensor (batch_size, seq_length). Args: seq_length(int): The sequence length of the input tensor. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **input_mask** (Tensor) - The mask indicating whether each position is a valid input with (batch_size, seq_length). Outputs: Tensor. The attention mask matrix with shape (batch_size, seq_length, seq_length). Raises: TypeError: `seq_length` is not an integer. ValueError: `seq_length` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import AttentionMaskHF >>> from mindspore import Tensor >>> mask = AttentionMaskHF(seq_length=4) >>> mask_array = np.array([[1, 1, 1, 0]], np.float32) >>> inputs = Tensor(mask_array) >>> res = mask(inputs) >>> print(res) [[[1. 0. 0. 0] [1. 1. 0. 0] [1. 1. 1. 0] [1. 1. 1. 1]]] """ @_LogActionOnce(m_logger=logger, key='AttentionMaskHF', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(seq_length=Validator.check_positive_int, parallel_config=_valid_type_checks([OpParallelConfig], "AttentionMaskHF")) def __init__(self, seq_length, parallel_config=default_dpmp_config): super(AttentionMaskHF, self).__init__() self.seq_length = seq_length self.not_equal = P.NotEqual().shard(((parallel_config.data_parallel, 1), ())) self.reshape = P.Reshape() self.mul = P.BatchMatMul().shard( ((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.expand_dim = P.ExpandDims().shard(((1, 1),)) ones = np.ones(shape=(seq_length, seq_length)) # Default lower triangle mask matrix self.lower_triangle_mask = Tensor(np.tril(ones), mstype.float32) self.multiply = P.Mul().shard(((parallel_config.data_parallel, 1, 1), (1, 1, 1))) def construct(self, input_mask): """Forward process of the AttentionMaskHF""" _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) input_mask = P.Cast()(P.OnesLike()(input_mask), mstype.float16) input_shape = P.Shape()(input_mask) shape_right = (input_shape[0], 1, input_shape[1]) shape_left = input_shape + (1,) # Mask the padded inputs mask_left = self.reshape(input_mask, shape_left) mask_right = self.reshape(input_mask, shape_right) attention_mask = self.mul(mask_left, mask_right) lower_traiangle = self.expand_dim(self.lower_triangle_mask, 0) # the returned shape is [bs, seq_length, seq_length] attention_mask = self.multiply( attention_mask, lower_traiangle) return attention_mask class VocabEmbedding(Cell): """ The embedding lookup table from the 0-th dim of the parameter table. When the parallel_config.vocab_emb_dp is True and in the `AUTO_PARALLEL` mode, the embedding lookup will be trained by the data parallel way, as the parameters will be repeated on each device. If false, the embedding table will be sharded into n parts at the 0-th dimension of the embedding table, where the n is the model parallel way determined by `parallel_config.model_parallel` (EmbeddingOpParallelConfig). Note: When `AUTO_PARALLEL` or `SEMI_AUTO_PARALLEL` mode is enabled, this layer support only 2-d dimension inputs, as the shard is designed for 2d inputs. Args: vocab_size (int): Size of the dictionary of embeddings. embedding_size (int): The size of each embedding vector. parallel_config (EmbeddingOpParallelConfig): The parallel config of network. Default `default_embedding_parallel_config`, an instance of `EmbeddingOpParallelConfig` with default args. param_init (Union[Tensor, str, Initializer, numbers.Number]): Initializer for the embedding_table. Refer to class `initializer` for the values of string when a string is specified. Default: 'normal'. Inputs: - **input_ids** (Tensor) - The tokenized inputs with datatype int32 with shape (batch_size, seq_length) Outputs: Tuple, a tuple contains (`output`, `embedding_table`) - **output** (Tensor) - The embedding vector for the input with shape (batch_size, seq_length, embedding_size). - **embedding_table** (Tensor) - The embedding table with shape (vocab_size, embedding_size). Raises: ValueError: If the parallel_config.vocab_emb_dp is True, the vocab size is not a multiple of parallel_config.model_parallel ValueError: `vocab_size` is not a positive value. ValueError: `embedding_size` is not a positive value. TypeError: `parallel_config` is not a subclass of OpParallelConfig. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import VocabEmbedding >>> from mindspore import Tensor >>> from mindspore import dtype as mstype >>> model = VocabEmbedding(vocab_size=30, embedding_size=30) >>> tensor = Tensor(np.ones((20, 15)), mstype.int32) >>> output, table = model(tensor) >>> print(output.shape) (20, 15, 30) >>> print(table.shape) (30, 30) """ @_LogActionOnce(m_logger=logger, key='VocabEmbedding', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(vocab_size=Validator.check_positive_int, embedding_size=Validator.check_positive_int, parallel_config=_valid_type_checks([EmbeddingOpParallelConfig], "VocabEmbedding")) def __init__(self, vocab_size, embedding_size, parallel_config=default_embedding_parallel_config, param_init='normal'): super(VocabEmbedding, self).__init__() _check_config(parallel_config) self.vocab_size = vocab_size self.embedding_size = embedding_size self.embedding_table = Parameter(initializer(param_init, [self.vocab_size, self.embedding_size]), name='embedding_table', parallel_optimizer=False) if parallel_config.vocab_emb_dp: self.gather = P.Gather().shard(((1, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel for the embedding lookup.") else: if self.vocab_size % parallel_config.model_parallel != 0: raise ValueError(f"The vocab size of the embedding {self.vocab_size} must be a " f"multiple of parallel_config.model_parallel {parallel_config.model_parallel}.") self.gather = P.Gather().shard(((parallel_config.model_parallel, 1), (parallel_config.data_parallel, 1))) logger.info(f"Using {parallel_config.data_parallel} data parallel and {parallel_config.model_parallel} " f"model parallel for the embedding lookup.") def construct(self, input_ids): _check_input_dtype(F.dtype(input_ids), "input_ids", [mstype.int32], self.cls_name) output = self.gather(self.embedding_table, input_ids, 0) return output, self.embedding_table.value() class MultiHeadAttention(Cell): r""" This is an implementation of multihead attention in the paper `Attention is all you need `_. Given the query vector with source length, and the key and value vector with target length, the attention will be performed as the following .. math:: MultiHeadAttention(query, key, vector) = Concat(head_1, \dots, head_h)W^O where :math:`head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)`. The default is with a bias. if query, key and value tensor is same, then it will be self attention. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The sequence length of the query vector. tgt_seq_length(int): The sequence length of the key and value vector. hidden_size(int): The hidden size of the input. num_heads(int): The number of the heads. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. compute_dtype(dtype.Number): The computation type of dense. Default mstype.float16. Should be mstype.float32 or mstype.float16. softmax_compute_type(dtype.Number): The type of softmax computation module. Default mstype.float32. Should be mstype.float32 or mstype.float16. param_init_type(dtype.Number): The parameter initialization type of the module. Default mstype.float32. Should be mstype.float32 or mstype.float16. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. parallel_config(OpParallelConfig): The parallel configure. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **query_tensor** (Tensor) - The query vector with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **key_tensor** (Tensor) - The key vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **value_tensor** (Tensor) - The value vector with shape (batch_size, tgt_seq_length, hidden_size) or (batch_size * tgt_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, must be (batch_size, 1, hidden_size) - **attention_mask** (Tensor) - If the use_past is False or is_first_iteration=True, the attention mask matrix should ba (batch_size, src_seq_length, tgt_seq_length), or None. None means there will be no mask in softmax computation. Otherwise, the mask must be (batch_size, 1, tgt_seq_length) - **key_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, size_per_head, tgt_seq_length). The past calculated key vector. Used for incremental prediction when the use_past is True. Default None. - **value_past** (Tensor) - Float16 tensor with shape (batch_size, num_heads, tgt_seq_length, size_per_head). The past calculated value vector. Used for incremental prediction when the use_past is True. Default None. - **batch_valid_length** (Tensor) - Int32 tensor with shape (batch_size,) the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - Tensor, the float tensor of the output of the layer with shape (batch_size, src_seq_length, hidden_size) or (batch_size * src_seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindformers.modules.transformer import MultiHeadAttention >>> from mindspore import dtype as mstype >>> from mindspore import Tensor >>> model = MultiHeadAttention(batch_size=None, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3) >>> from_tensor = Tensor(np.ones((2, 20, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 20, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> # We need to prepare the memory parameters for saving key and value states firstly. >>> model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, ... num_heads=3, use_past=True) >>> key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), mstype.float16) >>> value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), mstype.float16) >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> # Set is_first_iteration=True to generate the full memory states >>> model.add_flags_recursive(is_first_iteration=True) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 20, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) >>> from_tensor = Tensor(np.ones((2, 1, 15)), mstype.float32) >>> to_tensor = Tensor(np.ones((2, 1, 15)), mstype.float16) >>> attention_mask = Tensor(np.ones((2, 1, 20)), mstype.float16) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the >>> # full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> attn_out, past = model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, ... batch_valid_length) >>> print(attn_out.shape) (2, 1, 15) >>> print(past[0].shape) (2, 3, 5, 20) >>> print(past[1].shape) (2, 3, 20, 5) """ @_LogActionOnce(m_logger=logger, key='MultiHeadAttention', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, compute_dtype=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "MultiHeadAttention"), parallel_config=_valid_type_checks([OpParallelConfig], "MultiHeadAttention"), use_past=Validator.check_bool) def __init__(self, batch_size, src_seq_length, tgt_seq_length, hidden_size, num_heads, hidden_dropout_rate=0.1, attention_dropout_rate=0.1, compute_dtype=mstype.float16, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, use_past=False, parallel_config=default_dpmp_config): super(MultiHeadAttention, self).__init__() self._is_ascend = context.get_context('device_target') in ["Ascend"] self.dp = parallel_config.data_parallel self.is_parallel_mode = _get_parallel_mode() in ( ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL) if batch_size: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose() self.merger_head_transpose = P.Transpose() self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul() self.real_div = P.RealDiv() self.sub = P.Sub() self.mul = P.Mul() self.add = P.Add() # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims() # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) else: _check_config(parallel_config) self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.hidden_size = hidden_size self.batch_size = batch_size if hidden_dropout_rate < 0 or hidden_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(hidden_dropout_rate)) if attention_dropout_rate < 0 or attention_dropout_rate >= 1: raise ValueError("For 'MultiHeadAttention', the class variable 'attention_dropout_rate' must be " "in range [0, 1.0), but got the value : {}.".format(attention_dropout_rate)) if hidden_size % num_heads != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'hidden_size' must be a multiple " "of 'num_heads', but got the hidden_size is {} and the num_heads is {}." .format(hidden_size, num_heads)) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'MultiHeadAttention', the class variable 'num_heads' must be a multiple of " "'parallel_config.model_parallel', but got the num_heads is {} " "and the parallel_config.model_parallel is {}." .format(num_heads, parallel_config.model_parallel)) self.is_first_iteration = True # Output layer self.projection = Linear(in_channels=hidden_size, out_channels=hidden_size, transpose_b=False, compute_dtype=compute_dtype, param_init_type=param_init_type) self.projection.shard(strategy_bias=((parallel_config.data_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1))) self.projection.bias.parallel_optimizer = False self.transpose = P.Transpose().shard( ((parallel_config.data_parallel, 1, parallel_config.model_parallel, 1),)) self.merger_head_transpose = P.Transpose().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.reshape = P.Reshape() self.n_head = num_heads # embedding size per head self.size_per_head = hidden_size // self.n_head self.concat_k = P.Concat(axis=3) self.concat_v = P.Concat(axis=2) self.multiply_data = Tensor([ -10000.0, ], dtype=softmax_compute_type) self.batch_matmul = P.BatchMatMul().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) self.real_div = P.RealDiv().shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1), ())) self.sub = P.Sub().shard( ((1,), (parallel_config.data_parallel, 1, 1, 1))) self.mul = P.Mul().shard( ((parallel_config.data_parallel, 1, 1, 1), (1,))) self.add = P.Add().shard( ((parallel_config.data_parallel, 1, 1, 1), (parallel_config.data_parallel, parallel_config.model_parallel, 1, 1))) # Normalize factor for attention, sqrt(dk) as widely used self.scale_factor = Tensor(math.sqrt(math.sqrt(self.size_per_head))) self.use_past = use_past self.dropout = get_dropout(hidden_dropout_rate) self.prob_dropout = get_dropout(attention_dropout_rate) self.dropout.dropout.shard(((parallel_config.data_parallel, 1),)) self.prob_dropout.dropout.shard( ((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax = nn.Softmax().to_float(softmax_compute_type) self.softmax.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1, 1),)) self.softmax_3d = nn.Softmax().to_float(softmax_compute_type) self.softmax_3d.softmax.shard(((parallel_config.data_parallel, parallel_config.model_parallel, 1),)) self.softmax_cast = P.Cast() self.softmax_reshape = P.Reshape() self.expand_dims = P.ExpandDims().shard(((parallel_config.data_parallel, 1, 1),)) # Query self.dense1 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense1.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Key self.dense2 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense2.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) # Value self.dense3 = Linear(hidden_size, hidden_size, compute_dtype=compute_dtype, param_init_type=param_init_type) self.dense3.shard(strategy_matmul=((parallel_config.data_parallel, 1), (parallel_config.model_parallel, 1)), strategy_bias=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel,))) self.dtype = compute_dtype self.softmax_dtype = softmax_compute_type if self.use_past: # operators used for state reuse seq_range = np.arange(src_seq_length).reshape(1, 1, -1) self.range = Tensor(np.tile(seq_range, (batch_size, 1, 1)), mstype.int32) self.seq_length = src_seq_length self.attention_mask = Tensor(np.tril(np.ones(shape=(self.seq_length, self.seq_length))), mstype.int32) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.expand_dims = P.ExpandDims().shard(((1, 1, 1),)) self.tensor_le = P.LessEqual().shard(((1, 1, 1), (1, 1, 1))) self.add = P.Add().shard(((1, 1, 1, 1), (1, 1, 1, 1))) self.equal = P.Equal().shard(((1, 1, 1), (1, 1, 1))) self.sub1 = P.Sub().shard(((1,), ())) self.tile = P.Tile().shard(((1, 1, 1, 1),)) self.less = P.Less().shard(((1, 1, 1), (1, 1, 1))) self.mul1 = P.Mul().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.dropout.dropout.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.select_recompute: # _attn中涉及的关键算子使用重计算逻辑，通常配合序列并行使用，会有较好的性能提升效果 self.batch_matmul.recompute() self.sub.recompute() self.add.recompute() self.merger_head_transpose.recompute() self.softmax_reshape.recompute() self.prob_dropout.recompute() self.softmax_cast.recompute() self.softmax.softmax.recompute() self.softmax_3d.recompute() def construct(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): """Forward process of the MultiHeadAttention""" self._check_inputs(query_tensor, key_tensor, value_tensor, attention_mask, key_past, value_past, batch_valid_length) ori_shape = F.shape(query_tensor) batch_size = self._get_batch_size_from_query(query_tensor) query_tensor, key_tensor, value_tensor = self._convert_to_2d_tensor(query_tensor, key_tensor, value_tensor) ori_dtype = F.dtype(query_tensor) query_tensor = F.cast(query_tensor, self.dtype) key_tensor = F.cast(key_tensor, self.dtype) value_tensor = F.cast(value_tensor, self.dtype) # multi head attention: query, key, value are derived from the same inputs query = self.dense1(query_tensor) key = self.dense2(key_tensor) value = self.dense3(value_tensor) # the returned shape is [bs, num_heads, seq_length, size_per_head] query = self.transpose( F.reshape( query, (batch_size, self._get_seq_length_under_incremental(self.src_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # the returned shape is [bs, size_per_head, seq_length, num_heads] key = self.transpose( F.reshape( key, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 3, 1)) # the returned shape is [bs, num_heads, seq_length, size_per_head] value = self.transpose( F.reshape( value, (batch_size, self._get_seq_length_under_incremental(self.tgt_seq_length), self.n_head, self.size_per_head)), (0, 2, 1, 3)) # support input shape is [bs, seq, seq] or [bs, heads, seq, seq] if attention_mask is not None and len(F.shape(attention_mask)) == 3: # expand attention mask from [bs, seq, seq] -> [bs, 1, seq, seq] attention_mask = self.expand_dims(attention_mask, 1) # key and value for current token(s) key_present = key value_present = value if self.use_past: # The first graph with the input size of (bs, seq_length) if self.is_first_iteration: # Get the valid input length without padding valid_length_vector = F.cast(self.less(self.range, batch_valid_length.view(-1, 1, 1)), self.dtype) # Cover the key and value numbers corresponding to the padding position key_present = self.mul1(key, self.expand_dims(valid_length_vector, 2)) value_present = self.mul1(value, self.expand_dims(valid_length_vector, 3)) # The second graph with the inpus size of (bs, 1) # the shape of query is (bs, num_heads, 1, size_per_head) # the shape of key is (bs, num_heads, size_per_head, 1) # the shape of value is (bs, num_heads, 1, size_per_head) else: # Get the current token position index valid_length = batch_valid_length - 1 valid_length = self.reshape(valid_length, (-1, 1, 1)) valid_length_vector = F.cast(self.equal(valid_length, self.range), self.dtype) # Pad the key and value to seq_length with only the position index not zero current_key = self.mul1(self.tile(key, (1, 1, 1, self.seq_length)), self.expand_dims(valid_length_vector, 2)) current_value = self.mul1(self.tile(value, (1, 1, self.seq_length, 1)), self.expand_dims(valid_length_vector, 3)) # Concat the previous saved state and current state key = self.add(key_past, current_key) value = self.add(value_past, current_value) # Update key_present and value_present for state update key_present = key value_present = value attention_mask = F.reshape(self.attention_mask, (self.seq_length, self.seq_length, 1, 1)) layer_present = (key_present, value_present) # multi head attention considering attention mask # the return shape is [bs * seq_length, hidden_size] attention = self._attn(query, key, value, attention_mask) # Output output = self.projection(attention) output = self.dropout(output) output = F.reshape(output, ori_shape) output = F.cast(output, ori_dtype) return output, layer_present def _get_batch_size_from_query(self, query): r"""Get the batch size from query tensor""" # For the incremental prediction, the seq length for the input is 1. if len(F.shape(query)) == 2 and ((self.use_past and self.is_first_iteration) or (not self.use_past)): return F.shape(query)[0] // self.src_seq_length return F.shape(query)[0] def _get_seq_length_under_incremental(self, length): r"""Return the length of the tensor. For the incremental prediction, the seq length for the input is 1. """ if self.use_past and not self.is_first_iteration: return 1 return length def _check_inputs(self, query_tensor, key_tensor, value_tensor, attention_mask, key_past=None, value_past=None, batch_valid_length=None): r"""Check inputs""" _check_input_dtype(F.dtype(query_tensor), "query_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(key_tensor), "key_tensor", [mstype.float32, mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_tensor), "value_tensor", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) key_is_tensor = isinstance(key_past, Tensor) value_is_tensor = isinstance(value_past, Tensor) batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) key_is_default = key_past is None value_is_default = value_past is None batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "key_past", self.cls_name, None, key_is_tensor, key_is_default) _check_past_none_input_none(self.use_past, "value_past", self.cls_name, None, value_is_tensor, value_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(key_past), "key_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(value_past), "value_past", [mstype.float16], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _convert_to_2d_tensor(self, query_tensor, key_tensor, value_tensor): """convert a nd tensor to a 2d tensor""" query_shape = F.shape(query_tensor) query_tensor = F.reshape(query_tensor, (-1, query_shape[-1])) key_shape = F.shape(key_tensor) key_tensor = F.reshape(key_tensor, (-1, key_shape[-1])) value_shape = F.shape(value_tensor) value_tensor = F.reshape(value_tensor, (-1, value_shape[-1])) return query_tensor, key_tensor, value_tensor def _merge_heads(self, x): """ convert a 4d input to a 2d output Inputs: x: input tensor Output: x_merge: the 2d output """ x = self.merger_head_transpose( x, (0, 2, 1, 3)) # bs, seq_length, head, size_per_head x_shape = P.Shape()(x) new_shape = (-1, x_shape[-2] * x_shape[-1]) x_merge = self.reshape(x, new_shape) return x_merge def _softmax(self, attention_scores): """ For the consideration of the performance, do softmax according to different situations :param attention_scores: a 3d tensor before softmax :return: the attention scores. """ if self._is_ascend and self.softmax_dtype == mstype.float16 or not self._is_ascend: attention_probs = self.softmax(attention_scores) else: shape = F.shape(attention_scores) # attention probs attention_probs = self.softmax_3d( self.softmax_reshape(attention_scores, (shape[0], -1, shape[-1]))) attention_probs = self.softmax_reshape(attention_probs, shape) return attention_probs def _attn(self, query, key, value, attention_mask): """ Get the weighted score along the seq_length Inputs: query: the query matrix key: the key matrix value: the value matrix attention_mask: the attention mask matrix with shape (batch_size, 1, seq_length, seq_length) Outputs: weighted_values: Tensor, the weighted sum scores """ # Normalize query and key before MatMul, default off # Attention score [bs, num_heads, seq_length, seq_length] factor = P.Cast()(self.scale_factor, P.DType()(query)) query = self.real_div(query, factor) key = self.real_div(key, factor) score = self.batch_matmul(query, key) ori_dtype = P.DType()(score) attention_scores = self.softmax_cast(score, self.softmax_dtype) # for input size of (bs, 1) namely the second graph, # the shape of attention_mask matrix should be (bs, 1, 1, seq_length) if attention_mask is not None: if self.use_past and not self.is_first_iteration: # Calculate the current total token current_index = self.reducesum(F.cast(self.not_equal(self.slice(key, (0, 0, 0, 0), (F.shape(query)[0], 1, 1, self.seq_length), (1, 1, 1, 1)), 0), mstype.float32), (1, 2, 3)) # Get the precise position index index = self.sub1(F.cast(current_index, mstype.int32), 1) index = F.reshape(index, (-1, 1, 1)) # Calculate the attention_mask matrix via the position index attention_mask = F.cast(self.tensor_le(self.range, index), mstype.int32) attention_mask = self.expand_dims(attention_mask, 2) # Minus 10000 for the position where masked to exclude them from softmax multiplu_out = self.sub( P.Cast()(F.tuple_to_array((1.0,)), P.DType()(attention_scores)), P.Cast()(attention_mask, P.DType()(attention_scores))) adder = self.mul(multiplu_out, self.multiply_data) attention_scores = self.add(adder, attention_scores) # attention probs attention_probs = self._softmax(attention_scores) attention_probs = self.softmax_cast(attention_probs, ori_dtype) attention_probs = self.prob_dropout(attention_probs) # Weighted sum output [bs, num_heads, seq_length, size_per_head] weighted_values = self.batch_matmul(attention_probs, value) attention_merge = self._merge_heads(weighted_values) return attention_merge class TransformerEncoderLayer(Cell): r""" Transformer Encoder Layer. This is an implementation of the single layer of the transformer encoder layer, including multihead attention and feedward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. seq_length(int): The input sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **x** (Tensor) - Float Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size] - **input_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`). - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size) - **layer_present** (Tuple) - A tuple of the Tensor of the projected key and value vector with ((batch_size, num_heads, size_per_head, seq_length), (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoderLayer >>> from mindspore import Tensor >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0].shape) (2, 2, 4, 16) >>> print(past[1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerEncoderLayer"), use_past=Validator.check_bool) def __init__(self, batch_size, hidden_size, ffn_hidden_size, num_heads, seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', use_past=False, moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerEncoderLayer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) self.batch_size = batch_size if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'num_heads' must be divisibled by the " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'hidden_size' must be divisibled by " "the 'parallel_config.model_parallel', but got the hidden_size is {} and parallel_config." " model_parallel is {}.".format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerEncoderLayer', the class variable 'ffn_hidden_size' must be divisibled " "by the 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config. model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.use_past = use_past self.seq_length = seq_length self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) attention_parallel_config = parallel_config.dpmp if self.use_moe else parallel_config self.attention = MultiHeadAttention(batch_size=batch_size, src_seq_length=seq_length, tgt_seq_length=seq_length, hidden_size=hidden_size, num_heads=num_heads, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, use_past=use_past, parallel_config=attention_parallel_config) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, seq_length) self.value_shape = (batch_size, num_heads, seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, x, input_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(x, input_mask, init_reset, batch_valid_length) x_shape = F.shape(x) x = F.reshape(x, (-1, x_shape[-1])) if self.post_layernorm_residual: input_x = x else: input_x = self.layernorm1(x) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, input_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(x, attention) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(x_shape) == 3: output_x = P.Reshape()(output_x, x_shape) mlp_logit = P.Reshape()(mlp_logit, x_shape) x = P.Reshape()(x, x_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) output = F.reshape(output, (-1, x_shape[-1])) output = self.layernorm1(output) output = F.reshape(output, x_shape) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) output = self.layernorm1(output) else: output = self.add(x, mlp_logit) output = F.reshape(output, x_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, x, input_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(x), "x", [mstype.float32, mstype.float16], self.cls_name) if input_mask is not None: _check_input_dtype(F.dtype(input_mask), "input_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True class TransformerDecoderLayer(Cell): r""" Transformer Decoder Layer. This is an implementation of the single layer of the transformer decoder layer, including self-attention, cross attention and feedward layer. When the encoder_output is None, the cross attention will not be effective. Args: hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. num_heads(int): The number of the heads. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(OpParallelConfig, MoEParallelConfig): The parallel configure. When MoE is applied, MoEParallelConfig is effective, otherwise OpParallelConfig is effective. Default `default_dpmp_config`, an instance of `OpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, tgt_seq_length, hidden_size] or [batch_size * tgt_seq_length, hidden_size]. - **decoder_mask** (Tensor) - The attention mask for decoder with shape [batch_size, src_seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, seq_length, hidden_size] or [batch * seq_length, hidden_size]. - **layer_present** (Tuple) - A tuple, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoderLayer >>> from mindspore import Tensor >>> model = TransformerDecoderLayer(batch_size=2, hidden_size=64, ffn_hidden_size=64, num_heads=2, ... src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(past[0].shape) (2, 2, 32, 10) >>> print(past[1].shape) (2, 2, 10, 32) >>> print(past[2].shape) (2, 2, 32, 20) >>> print(past[3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoderLayer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoderLayer"), parallel_config=_valid_type_checks([OpParallelConfig, MoEParallelConfig], "TransformerDecoderLayer"), use_past=Validator.check_bool) def __init__(self, hidden_size, ffn_hidden_size, num_heads, batch_size, src_seq_length, tgt_seq_length, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, use_past=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', moe_config=default_moe_config, parallel_config=default_dpmp_config): super(TransformerDecoderLayer, self).__init__() _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) config_to_attention = parallel_config.dpmp if self.use_moe else parallel_config if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add() self.add_3d = P.Add() self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) if num_heads % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'num_heads' must be divisibled by " "'parallel_config.model_parallel', but got the num_heads is {} and " "parallel_config.model_parallel is {}.".format(num_heads, parallel_config.model_parallel)) if hidden_size % parallel_config.model_parallel != 0: raise ValueError( "For 'TransformerDecoderLayer', the class variable 'hidden_size' must be divisibled by " "'parallel_config.model_parallel', but got the hidden_size is {} and " "parallel_config.model_parallel is {}." .format(hidden_size, parallel_config.model_parallel)) if ffn_hidden_size % parallel_config.model_parallel != 0: raise ValueError("For 'TransformerDecoderLayer', the class variable 'ffn_hidden_size' must be " "divisibled by 'parallel_config.model_parallel', but got the ffn_hidden_size is {} " "and parallel_config.model_parallel is {}." .format(ffn_hidden_size, parallel_config.model_parallel)) if use_past: raise ValueError(f"The {self.cls_name} does not support use_past=True.") self.batch_size = batch_size self.use_past = use_past self.softmax_compute_type = softmax_compute_type self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past self.hidden_size = hidden_size self.layernorm1 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm1.shard(((parallel_config.data_parallel, 1),)) self.layernorm2 = LayerNorm((hidden_size,)).to_float(layernorm_compute_type) self.layernorm2.shard(((parallel_config.data_parallel, 1),)) self.attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=tgt_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, use_past=use_past, softmax_compute_type=softmax_compute_type, param_init_type=param_init_type, parallel_config=config_to_attention) # Cross attention with the output of encoder as memory tensor self.cross_attention = MultiHeadAttention(hidden_size=hidden_size, num_heads=num_heads, batch_size=batch_size, src_seq_length=tgt_seq_length, tgt_seq_length=src_seq_length, hidden_dropout_rate=hidden_dropout_rate, attention_dropout_rate=attention_dropout_rate, softmax_compute_type=softmax_compute_type, use_past=use_past, param_init_type=param_init_type, parallel_config=config_to_attention) self.cross_attention_layernorm = LayerNorm((hidden_size,)).to_float( layernorm_compute_type) self.cross_attention_layernorm.shard(((parallel_config.data_parallel, 1),)) if self.use_moe: self.output = MoE(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, param_init_type=param_init_type, hidden_act=hidden_act, moe_config=moe_config, parallel_config=parallel_config) else: # Feed Forward Network, FFN self.output = FeedForward(hidden_size=hidden_size, dropout_rate=hidden_dropout_rate, ffn_hidden_size=ffn_hidden_size, hidden_act=hidden_act, param_init_type=param_init_type, parallel_config=parallel_config) self.post_layernorm_residual = post_layernorm_residual self.add = P.Add().shard(((parallel_config.data_parallel, 1), (parallel_config.data_parallel, 1))) self.add_3d = P.Add().shard(((parallel_config.data_parallel, 1, 1), (parallel_config.data_parallel, 1, 1))) self.dtype = mstype.float16 self.key_past = None self.value_past = None if self.use_past: # operator used for state reuse self.reducesum = P.ReduceSum().shard(((1, 1, 1, 1),)) self.not_equal = P.NotEqual().shard(((1, 1, 1, 1), ())) self.slice = P.StridedSlice().shard(((1, 1, 1, 1),)) size_per_head = hidden_size // num_heads self.key_shape = (batch_size, num_heads, size_per_head, tgt_seq_length) self.value_shape = (batch_size, num_heads, tgt_seq_length, size_per_head) # parameters saving key and value states self.key_past = Parameter(Tensor(np.zeros(shape=self.key_shape), self.dtype), name="key_past") self.value_past = Parameter(Tensor(np.zeros(shape=self.value_shape), self.dtype), name="value_past") self.tile = P.Tile().shard(((1, 1),)) self.mul = P.Mul().shard(((1, 1, 1, 1), (1,))) self.assign = P.Assign().shard(((1, 1, 1, 1), (1, 1, 1, 1))) if parallel_config.use_seq_parallel: self.add.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1), (parallel_config.data_parallel * parallel_config.model_parallel, 1))) self.layernorm1.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.layernorm2.shard(((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) if parallel_config.recompute.select_recompute: # 此处会消耗较大内存，开启后会损失一部分计算性能 self.layernorm2.layer_norm.recompute() if not self.use_moe: self.output.projection.shard( strategy_bias=((parallel_config.data_parallel * parallel_config.model_parallel, 1), (1,)), strategy_matmul=((parallel_config.data_parallel, parallel_config.model_parallel), (parallel_config.model_parallel, 1)), out_strategy_matmul=((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) self.output.dropout.dropout.shard( ((parallel_config.data_parallel * parallel_config.model_parallel, 1),)) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_stats, decoder_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" self._check_input(hidden_stats, decoder_mask, encoder_output, memory_mask, init_reset, batch_valid_length) # the returned shape is [bs, seq_length, embedding_size] or [bs * seq_length, embedding_size] hidden_shape = F.shape(hidden_stats) hidden_stats = F.reshape(hidden_stats, (-1, hidden_shape[-1])) input_x = self.layernorm1(hidden_stats) input_x = F.cast(input_x, self.dtype) # indicate whether reset saved states key_reset = None value_reset = None if self.use_past: # reset states, init_reset True for reuse and False for reset self.assign(self.key_past, self.mul(self.key_past, F.cast(init_reset, self.dtype))) key_reset = self.key_past self.assign(self.value_past, self.mul(self.value_past, F.cast(init_reset, self.dtype))) value_reset = self.value_past # add dependency for desired execution order input_x = F.depend(input_x, key_reset) input_x = F.depend(input_x, value_reset) attention, layer_present = self.attention(input_x, input_x, input_x, decoder_mask, self.key_past, self.value_past, batch_valid_length) # For post-layernorm the inputs for residual path are output of self-attention and output of layernorm if self.post_layernorm_residual: x = self.add(input_x, attention) # For pre-layernorm the inputs for residual path are output of self-attention and input of this layer else: x = self.add(hidden_stats, attention) middle_output = None if encoder_output is not None: middle_output = self.cross_attention_layernorm(x) middle_output = F.cast(middle_output, self.dtype) encoder_output = F.cast(encoder_output, self.dtype) cross_attn_output, cross_layer_present = self.cross_attention(middle_output, encoder_output, encoder_output, memory_mask, self.key_past, self.value_past, batch_valid_length) layer_present += cross_layer_present if self.post_layernorm_residual: x = self.add(middle_output, cross_attn_output) else: x = self.add(x, cross_attn_output) output_x = self.layernorm2(x) output_x = F.cast(output_x, self.dtype) aux_loss = None if self.use_moe: mlp_logit, aux_loss = self.output(output_x) else: mlp_logit = self.output(output_x) value_update = None key_update = None if self.use_past: # current key and value key_present, value_present = layer_present # update key and value calculated this step self.assign(self.key_past, key_present) key_update = self.key_past self.assign(self.value_past, value_present) value_update = self.value_past # add dependency for desired execution order key_update = F.depend(key_update, key_reset) value_update = F.depend(value_update, value_reset) # add dependency for desired execution order mlp_logit = F.depend(mlp_logit, value_update) mlp_logit = F.depend(mlp_logit, key_update) # if shape is 3d, we reshape the inputs of the add if len(hidden_shape) == 3: output_x = P.Reshape()(output_x, hidden_shape) mlp_logit = P.Reshape()(mlp_logit, hidden_shape) x = P.Reshape()(x, hidden_shape) if self.post_layernorm_residual: output = self.add_3d(output_x, mlp_logit) else: output = self.add_3d(x, mlp_logit) else: if self.post_layernorm_residual: output = self.add(output_x, mlp_logit) else: output = self.add(x, mlp_logit) output = F.reshape(output, hidden_shape) if self.use_moe: return output, layer_present, aux_loss return output, layer_present def _check_input(self, hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length): r"""Check inputs""" _check_input_dtype(F.dtype(hidden_states), "hidden_states", [mstype.float32, mstype.float16], self.cls_name) if attention_mask is not None: _check_input_dtype(F.dtype(attention_mask), "attention_mask", [mstype.float32, mstype.float16], self.cls_name) if encoder_output is not None: _check_input_dtype(F.dtype(encoder_output), "encoder_output", [mstype.float32, mstype.float16], self.cls_name) if memory_mask is not None: _check_input_dtype(F.dtype(memory_mask), "memory_mask", [mstype.float32, mstype.float16], self.cls_name) init_reset_is_tensor = isinstance(init_reset, Tensor) init_reset_is_default = init_reset is True batch_valid_length_is_tensor = isinstance(batch_valid_length, Tensor) batch_is_default = batch_valid_length is None _check_past_none_input_none(self.use_past, "init_reset", self.cls_name, True, init_reset_is_tensor, init_reset_is_default) _check_past_none_input_none(self.use_past, "batch_valid_length", self.cls_name, None, batch_valid_length_is_tensor, batch_is_default) if self.use_past: _check_input_dtype(F.dtype(init_reset), "init_reset", [mstype.bool_], self.cls_name) _check_input_dtype(F.dtype(batch_valid_length), "batch_valid_length", [mstype.int32], self.cls_name) return True def _get_lambda_func(total_layer=None): r""" A wrapper function of specifying pipeline stage and gradient aggregation fusion. If the total layer is not None, for example, set in the transformer model, the pipeline stage setting function will be `(layer_id + 0) // (total_layers / parallel_config.pipeline_stage)` for the encoder and, `(layer_id + offset) // (total_layers / parallel_config.pipeline_stage)` for the decoder, where `offset` is the layers in the encoder. """ def _set_parallel_configure_for_layer(network, layer_id, offset, parallel_config, layers): r""" Default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Args: network(Cell) - Represents the transformer block layer_id(int) - Means the layer index for the current module, counts from zero. offset(int) - Means the layer_index needs an offset, if there are other modules in the net. layers(int) - The total layers used for the model. """ # override the layers if total_layer: layers = total_layer # Used for the pipeline's stages setting if layers < parallel_config.pipeline_stage: raise ValueError(f"layers {layers} must be larger than pipeline stage {parallel_config.pipeline_stage}") pp_dis = max(layers // parallel_config.pipeline_stage, 1) # the pipeline stage must be in [0, parallel_config.pipeline_stage - 1] pp_id = min((layer_id + offset) // pp_dis, parallel_config.pipeline_stage - 1) network.pipeline_stage = pp_id # Used for optimizer's fusion tag dis = max(layers // parallel_config.gradient_aggregation_group, 1) network.set_comm_fusion((layer_id + offset) // dis + 1) # Used for enabling recomputation of the block if isinstance(parallel_config.recompute, bool): if parallel_config.recompute and not parallel_config.select_recompute: network.recompute() else: if parallel_config.recompute.recompute and not parallel_config.recompute.select_recompute: paralel_op_comm_compute = parallel_config.recompute.parallel_optimizer_comm_recompute network.recompute(parallel_optimizer_comm_recompute=paralel_op_comm_compute, mp_comm_recompute=parallel_config.recompute.mp_comm_recompute, recompute_slice_activation=parallel_config.recompute.recompute_slice_activation) return _set_parallel_configure_for_layer class TransformerEncoder(Cell): r""" Transformer Encoder module with multi-layer stacked of `TransformerEncoderLayer`, including multihead self attention and feedforward layer. Args: batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. num_layers(int): The layers of the `TransformerEncoderLayer` hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. seq_length(int): The seq_length of the input tensor. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default: 0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default: mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default: mstype.float32. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. offset(int): The initial layer index for the `encoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. use_past(bool): Use the past state to compute, used for incremental prediction. For example, if we have two words and want to generate the ten more words. We just need to compute the two words' state only once, and generate the next word one by one. When use_past is True, there are two steps to run the prediction. In the first step, set the is_first_iteration to be True by `model.add_flags_recursive(is_first_iteration=True)`, and pass the full inputs. Then, set the is_first_iteration to be False by `model.add_flags_recursive(is_first_iteration=False)`. At this moment, pass the single step's input tensor, and loop it. Default: False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_states** (Tensor) - Tensor, shape should be [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], if the use_past is False or is_first_iteration=True. Otherwise, should be [batch_size, 1, hidden_size]. - **attention_mask** (Tensor) - Float Tensor, If the use_past is False or is_first_iteration=True, the attention mask matrix should ba [batch_size, seq_length, seq_length], or None. None means there will be no mask in softmax computation. Otherwise, should be [batch_size, 1, hidden_size] - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The float tensor of the output of the layer with shape (batch_size, seq_length, hidden_size) or (batch_size * seq_length, hidden_size), if the use_past is False or is_first_iteration=True. Otherwise, it will be (batch_size, 1, hidden_size). - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple contains the Tensor the projected key and value vector with shape ((batch_size, num_heads, size_per_head, seq_length), and (batch_size, num_heads, seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerEncoder >>> from mindspore import Tensor >>> model = TransformerEncoder(batch_size=2, num_layers=2, hidden_size=8, ffn_hidden_size=64, ... seq_length=16, num_heads=2) >>> encoder_input_value = Tensor(np.ones((2, 16, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 16, 16)), mstype.float16) >>> output, past = model(encoder_input_value, encoder_input_mask) >>> print(output.shape) (2, 16, 8) >>> print(len(past)) 2 >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> # When use use_past=True, it includes two steps to implement the incremental prediction. >>> # Step 1: set is_first_iteration=True, and input the full sequence length's state. >>> batch_valid_length = Tensor(np.ones((2,)), mstype.int32) >>> init_reset = Tensor([True], mstype.bool_) >>> # Set is_first_iteration=True to generate the full memory states >>> model = TransformerEncoder(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, ... num_heads=2, num_layers=2, use_past=True) >>> model.add_flags_recursive(is_first_iteration=True) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 16, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) >>> encoder_input_value = Tensor(np.ones((2, 1, 8)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 1, 16)), mstype.float16) >>> init_reset = Tensor([False], mstype.bool_) >>> # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than >>> # the full sequence. >>> model.add_flags_recursive(is_first_iteration=False) >>> hidden, past = model(encoder_input_value, encoder_input_mask, init_reset, batch_valid_length) >>> print(hidden.shape) (2, 1, 8) >>> print(past[0][0].shape) (2, 2, 4, 16) >>> print(past[0][1].shape) (2, 2, 16, 4) """ @_LogActionOnce(m_logger=logger, key='TransformerEncoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerEncoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerEncoder"), use_past=Validator.check_bool) def __init__(self, batch_size, num_layers, hidden_size, ffn_hidden_size, seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, offset=0, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerEncoder, self).__init__() _check_config(parallel_config) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerEncoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, seq_length=seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, num_heads=num_heads, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, use_past=use_past, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class TransformerDecoder(Cell): r""" Transformer Decoder module with multi-layer stacked of `TransformerDecoderLayer`, including multihead self attention, cross attention and feedforward layer. Args: num_layers(int): The layers of the `TransformerDecoderLayer`. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. hidden_size(int): The hidden size of the input. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The input source sequence length. tgt_seq_length(int): The input target sequence length. num_heads(int): The number of the heads. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be mstype.float32 or mstype.float16. Default mstype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be mstype.float32 or mstype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be mstype.float32 or mstype.float16. Default mstype.float32. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. lambda_func(function): A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // (layers / pipeline_stage)`. Default: None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. offset(int): The initial layer index for the `decoder`. Used for setting the fusion id and stage id, to not overlap with the encoder layer. Default 0. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **hidden_stats** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size] - **attention_mask** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention. - **encoder_output** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. Note this args can not be passed by None when the net is in outermost layer. Default None. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The user can also pass None. None means there will be no mask in softmax computation in cross attention. Default None. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `layer_present`) - **output** (Tensor) - The output logit of this layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size] - **layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head), and of the projected key and value vector in cross attention with shape (batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import TransformerDecoder >>> from mindspore import Tensor >>> model = TransformerDecoder(batch_size=2, num_layers=1, hidden_size=64, ffn_hidden_size=64, ... num_heads=2, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, past = model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(past)) 1 >>> print(past[0][0].shape) (2, 2, 32, 10) >>> print(past[0][1].shape) (2, 2, 10, 32) >>> print(past[0][2].shape) (2, 2, 32, 20) >>> print(past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='TransformerDecoder', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, num_layers=Validator.check_positive_int, tgt_seq_length=Validator.check_positive_int, offset=Validator.check_non_negative_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "TransformerDecoder"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "TransformerDecoder"), use_past=Validator.check_bool) def __init__(self, num_layers, batch_size, hidden_size, ffn_hidden_size, src_seq_length, tgt_seq_length, num_heads, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, hidden_act='gelu', lambda_func=None, use_past=False, offset=0, moe_config=default_moe_config, parallel_config=default_transformer_config): super(TransformerDecoder, self).__init__() _check_moe_config(moe_config, parallel_config) _check_config(parallel_config) self.use_moe = (moe_config.expert_num > 1) if batch_size or use_past: Validator.check_positive_int(batch_size) config_to_layer = parallel_config.moe_parallel_config if self.use_moe else parallel_config.dp_mp_config if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") self.num_layers = num_layers self.blocks = nn.CellList() for i in range(num_layers): block = TransformerDecoderLayer(hidden_size=hidden_size, batch_size=batch_size, ffn_hidden_size=ffn_hidden_size, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, num_heads=num_heads, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, hidden_act=hidden_act, use_past=use_past, param_init_type=param_init_type, post_layernorm_residual=post_layernorm_residual, moe_config=moe_config, parallel_config=config_to_layer) # If the user doesn't pass the fusion function, use the default one if not lambda_func: lambda_func = _get_lambda_func() lambda_func(block, layer_id=i, layers=num_layers, offset=offset, parallel_config=parallel_config) self.blocks.append(block) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, hidden_states, attention_mask, encoder_output=None, memory_mask=None, init_reset=True, batch_valid_length=None): """forward process""" present_layer = () if self.use_moe: accum_loss = self.aux_loss for i in range(self.num_layers): hidden_states, present, aux_loss = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) accum_loss = self.add(accum_loss, aux_loss) return hidden_states, present_layer, accum_loss # Loop through each self-attention layer for i in range(self.num_layers): hidden_states, present = self.blocks[i](hidden_states, attention_mask, encoder_output, memory_mask, init_reset, batch_valid_length) present_layer = present_layer + (present,) return hidden_states, present_layer class Transformer(Cell): r""" Transformer module including encoder and decoder. The difference with the original implements is the module use the residual addition before the layer normalization. And the default hidden act is `gelu`. The details can be found in `Attention is all you need `_. Note: This is an experimental interface that is subject to change or deletion. Args: hidden_size(int): The hidden size of the input. batch_size(int): The batch size of the input tensor when do increnmental prediction. Should be a positive value. When do training or prediction, the argument will not work and the user can just pass None to the argument. ffn_hidden_size(int): The hidden size of bottleneck in the feedforward layer. src_seq_length(int): The seq_length of the encoder's input tensor. tgt_seq_length(int): The seq_length of the decoder's input tensor. encoder_layers(int): The layers of the `TransformerEncoderLayer`. Default 3. decoder_layers(int): The layers of the `TransformerDecoderLayer`. Default 3. num_heads(int): The number of the heads. Default: 2. attention_dropout_rate(float): The dropout rate of the attention scores. Default:0.1. hidden_dropout_rate(float): The dropout rate of the final output of the layer. Default:0.1. hidden_act (str, nn.Cell): The activation of the internal feedforward layer. Supports 'relu', 'relu6', 'tanh', 'gelu', 'fast_gelu', 'elu', 'sigmoid', 'prelu', 'leakyrelu', 'hswish', 'hsigmoid', 'logsigmoid' and so on. User can provide custom activition to the argument. If user wants to run the net in the parallel mode, the custom activation must also provide the `activation_shard` function. Please see the examples of the class:`mindformers.modules.transformer.FeedForward`. Default: gelu. post_layernorm_residual(bool): Do residuals adds before the layernorm. Default False. layernorm_compute_type(dtype.Number): The computation type of the layernorm. Should be dtype.float32 or dtype.float16. Default dtype.float32. softmax_compute_type(dtype.Number): The computation type of the softmax in the attention. Should be dtype.float32 or dtype.float16. Default mstype.float32. param_init_type(dtype.Number): The parameter initialization type of the module. Should be dtype.float32 or dtype.float16. Default dtype.float32. lambda_func: A function can determine the fusion index, pipeline stages and recompute attribute. If the user wants to determine the pipeline stage and gradient aggregation fusion, the user can pass a function that accepts `network`, `layer_id`, `offset`, `parallel_config`, `layers`. The `network(Cell)` represents the transformer block, `layer_id(int)` means the layer index for the current module, counts from zero, `offset(int)` means the layer_index needs an offset, if there are other modules in the net. The default setting for the pipeline is: `(layer_id + offset) // ((encoder_layers + decoder_layers) / pipeline_stage)`. Default None. use_past(bool): Use the past state to compute, used for incremental prediction. Default False. moe_config(MoEConfig): The configuration of MoE (Mixture of Expert). Default is an instance of MoEConfig with default values. Please see `MoEConfig`. parallel_config(TransformerOpParallelConfig): The parallel configure. Default `default_transformer_config`, an instance of `TransformerOpParallelConfig` with default args. Inputs: - **encoder_inputs** (Tensor) - The input tensor with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size]. - **encoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the encoder module. - **decoder_inputs** (Tensor) - The output of the encoder with shape [batch_size, seq_length, hidden_size] or [batch_size * seq_length, hidden_size], this should be none if the decoder layer is 0. - **decoder_masks** (Tensor) - The attention mask for decoder with shape [batch_size, seq_length, seq_length] or None. None means there will be no mask in softmax computation in self attention of the decoder module. - **memory_mask** (Tensor) - The memory mask of the cross attention with shape [batch, tgt_seq_length, src_seq_length] where tgt_seq_length is the length of the decoder. The output of the encoder with shape [batch_size, seq_length, hidden_size], this should be none if the decoder layer is 0 or the user wants no mask. - **init_reset** (Tensor) - A bool tensor with shape [1], used to clear the past key parameter and past value parameter used in the incremental prediction. Only valid when use_past is True. Default True. - **batch_valid_length** (Tensor) - Int32 tensor with shape [batch_size] the past calculated the index. Used for incremental prediction when the use_past is True. Default None. Outputs: Tuple, a tuple contains(`output`, `encoder_layer_present`, `decoder_layer_present`, `accum_loss`) - **output** (Tensor) - If there is only encoder, the output logit of the encoder layer. The shape is [batch, src_seq_length, hidden_size] or [batch * src_seq_length, hidden_size], if there are encoder and decoders, the output is from the decoder layer. The shape is [batch, tgt_seq_length, hidden_size] or [batch * tgt_seq_length, hidden_size]. - **encoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). - **decoder_layer_present** (Tuple) - A tuple with size of num_layers, where each tuple is the tensor of the projected key and value vector in self attention with shape ((batch_size, num_heads, size_per_head, tgt_seq_length), (batch_size, num_heads, tgt_seq_length, size_per_head)), and the projected key and value vector in cross attention with shape ((batch_size, num_heads, size_per_head, src_seq_length), (batch_size, num_heads, src_seq_length, size_per_head)). If the decoder is not set, the returned value will be None. - **accum_loss** (Tensor) - A Tensor indicates an auxiliary loss to minimize the mean square of the data part routed to each expert, and only returned if the number of experts is greater than 1. Supported Platforms: ``Ascend`` ``GPU`` Examples: >>> import numpy as np >>> from mindspore import dtype as mstype >>> from mindformers.modules.transformer import Transformer >>> from mindspore import Tensor >>> model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, ... ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) >>> encoder_input_value = Tensor(np.ones((2, 20, 64)), mstype.float32) >>> encoder_input_mask = Tensor(np.ones((2, 20, 20)), mstype.float16) >>> decoder_input_value = Tensor(np.ones((2, 10, 64)), mstype.float32) >>> decoder_input_mask = Tensor(np.ones((2, 10, 10)), mstype.float16) >>> memory_mask = Tensor(np.ones((2, 10, 20)), mstype.float16) >>> output, en_past, de_past = model(encoder_input_value, encoder_input_mask, decoder_input_value, ... decoder_input_mask, memory_mask) >>> print(output.shape) (2, 10, 64) >>> print(len(en_past)) 1 >>> print(len(de_past)) 2 >>> print(en_past[0][0].shape) (2, 2, 32, 20) >>> print(en_past[0][1].shape) (2, 2, 20, 32) >>> print(de_past[0][0].shape) (2, 2, 32, 10) >>> print(de_past[0][1].shape) (2, 2, 10, 32) >>> print(de_past[0][2].shape) (2, 2, 32, 20) >>> print(de_past[0][3].shape) (2, 2, 20, 32) """ @_LogActionOnce(m_logger=logger, key='Transformer', no_warning=_get_parallel_mode() in (ParallelMode.STAND_ALONE,)) @_args_type_validator_check(hidden_size=Validator.check_positive_int, num_heads=Validator.check_positive_int, ffn_hidden_size=Validator.check_positive_int, src_seq_length=Validator.check_positive_int, encoder_layers=Validator.check_positive_int, decoder_layers=Validator.check_non_negative_int, tgt_seq_length=Validator.check_positive_int, attention_dropout_rate=Validator.check_non_negative_float, hidden_dropout_rate=Validator.check_non_negative_float, post_layernorm_residual=Validator.check_bool, layernorm_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), softmax_compute_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), param_init_type=_valid_value_checks([mstype.float32, mstype.float16], "Transformer"), parallel_config=_valid_type_checks([TransformerOpParallelConfig], "Transformer"), use_past=Validator.check_bool) def __init__(self, hidden_size, batch_size, ffn_hidden_size, src_seq_length, tgt_seq_length, encoder_layers=3, decoder_layers=3, num_heads=2, attention_dropout_rate=0.1, hidden_dropout_rate=0.1, hidden_act='gelu', post_layernorm_residual=False, layernorm_compute_type=mstype.float32, softmax_compute_type=mstype.float32, param_init_type=mstype.float32, lambda_func=None, use_past=False, moe_config=default_moe_config, parallel_config=default_transformer_config): super(Transformer, self).__init__() if batch_size or use_past: Validator.check_positive_int(batch_size) if _get_parallel_mode() in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add() self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) elif _get_parallel_mode() not in (ParallelMode.AUTO_PARALLEL,): _check_config(parallel_config) self.batch_size = batch_size self.hidden_size = hidden_size self.src_seq_length = src_seq_length self.tgt_seq_length = tgt_seq_length self.use_past = use_past if encoder_layers <= 0 < decoder_layers: raise ValueError(f"Transformer doest support encoder layer {encoder_layers} and decoder" f"layer {decoder_layers}, please use TransformerDecoder") if encoder_layers > 0 and decoder_layers > 0 and use_past: raise ValueError(f"The {self.cls_name} with encoder and decoder does not support use_past=True.") logger.warning("For parallel mode, sharding propagation is recommended, you can use it by setting " "'set_auto_parallel_context(parallel_mode=ParallelMode.AUTO_PARALLEL, " "search_mode=\"sharding_propagation\")' and " "'set_algo_parameters(elementwise_op_strategy_follow=False, fully_use_devices=False)'") # The shard setting of Transformer is set within the TransformerEncoderLayer if not lambda_func: lambda_func = _get_lambda_func(total_layer=encoder_layers + decoder_layers) _check_moe_config(moe_config, parallel_config) self.use_moe = (moe_config.expert_num > 1) self.add = P.Add().shard(((), ())) self.aux_loss = Tensor(0.0, mstype.float32) if encoder_layers > 0: self.encoder = TransformerEncoder(num_layers=encoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, seq_length=src_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, post_layernorm_residual=post_layernorm_residual, param_init_type=param_init_type, lambda_func=lambda_func, use_past=use_past, moe_config=moe_config, parallel_config=parallel_config) else: self.encoder = None # Offset is needed as the encoder has consumed some flags. # so the decoder need to increase the flags based on the encoder layer self.decoder = None if decoder_layers > 0: self.decoder = TransformerDecoder(num_layers=decoder_layers, batch_size=batch_size, hidden_size=hidden_size, ffn_hidden_size=ffn_hidden_size, num_heads=num_heads, src_seq_length=src_seq_length, tgt_seq_length=tgt_seq_length, attention_dropout_rate=attention_dropout_rate, hidden_dropout_rate=hidden_dropout_rate, hidden_act=hidden_act, post_layernorm_residual=post_layernorm_residual, layernorm_compute_type=layernorm_compute_type, softmax_compute_type=softmax_compute_type, lambda_func=lambda_func, use_past=use_past, param_init_type=param_init_type, offset=encoder_layers, moe_config=moe_config, parallel_config=parallel_config) else: raise RuntimeError(f"The {self.cls_name} only support sharding propagation or " f"semi-auto parallel mode now.") def construct(self, encoder_inputs, encoder_masks, decoder_inputs=None, decoder_masks=None, memory_mask=None, init_reset=True, batch_valid_length=None): """process process""" encoder_output = None output = None encoder_layer_present = None decoder_layer_present = None accum_loss = self.aux_loss if self.encoder is not None: if self.use_moe: encoder_output, encoder_layer_present, encoder_aux_loss = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, encoder_aux_loss) else: encoder_output, encoder_layer_present = self.encoder(encoder_inputs, encoder_masks, init_reset, batch_valid_length) output = encoder_output if self.decoder is not None: # decoder mask should be created outside of the model if self.use_moe: decoder_output, decoder_layer_present, decoder_aux_loss = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) accum_loss = self.add(accum_loss, decoder_aux_loss) else: decoder_output, decoder_layer_present = self.decoder(decoder_inputs, decoder_masks, encoder_output, memory_mask, init_reset, batch_valid_length) output = decoder_output if self.use_moe: return output, encoder_layer_present, decoder_layer_present, accum_loss return output, encoder_layer_present, decoder_layer_present pengcheng.mind-mindformer/mindformers/pet/000077500000000000000000000000001476650141400212275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pet/__init__.py000066400000000000000000000014361476650141400233440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """models init""" from .pet_config import PetConfig from .tuners import * __all__ = ['PetConfig'] __all__.extend(tuners.__all__) pengcheng.mind-mindformer/mindformers/pet/constants.py000066400000000000000000000021731476650141400236200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Constants used by pet modules. """ import enum class PetType(str, enum.Enum): r""" The enum of pet methods. """ LORA = "lora" ADAPTER = "adapter" ADALORA = "low_rank_adapter" PREFIX_TUNING = "prefixtuning" class BaseModelInitType(str, enum.Enum): r""" The enum of base model initial. """ MODEL_NAME = "MODEL_NAME" MODEL_DIR = "MODEL_DIR" MODEL_CONFIG = "MODEL_CONFIG" MODEL_INSTANCE = "MODEL_INSTANCE" pengcheng.mind-mindformer/mindformers/pet/pet_config.py000066400000000000000000000040551476650141400237220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: The config module of Parameter Efficient Tuning module. """ from mindformers.models.base_config import BaseConfig from mindformers.models.utils import convert_mstype from mindformers.pet.constants import PetType __all__ = ['LoraConfig'] class PetConfig(BaseConfig): """ Pet model config class which defines the tuning algorithm and pretrained for tuning. """ def __init__(self, pet_type=PetType.LORA, base_mode=None, **kwargs): super().__init__(**kwargs) self.pet_type = pet_type self.base_mode = base_mode class LoraConfig(PetConfig): """ Lora tuning algorithm config. """ def __init__(self, lora_rank: int = 8, lora_alpha: int = 16, lora_dropout: float = 0.01, lora_a_init: str = 'normal', lora_b_init: str = 'zero', param_init_type: str = 'float16', compute_dtype: str = 'float16', **kwargs): super().__init__(pet_type=PetType.LORA, **kwargs) self.lora_rank = lora_rank self.lora_alpha = lora_alpha self.lora_dropout = lora_dropout self.lora_a_init = lora_a_init self.lora_b_init = lora_b_init self.param_init_type = convert_mstype(param_init_type) self.compute_dtype = convert_mstype(compute_dtype) pengcheng.mind-mindformer/mindformers/pet/tuners/000077500000000000000000000000001476650141400225475ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pet/tuners/.ipynb_checkpoints/000077500000000000000000000000001476650141400263405ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pet/tuners/.ipynb_checkpoints/__init__-checkpoint.py000066400000000000000000000016731476650141400326050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert_pet.""" from .adalora_adapter import * from .ada_adapter import * from .lora_adapter import * from .prefix_tuning_adapter import * from .pet_adapter import * __all__ = ['AdaLoraAdapter', 'AdaAdapter', 'LoraAdapter', 'PetAdapter', 'PrefixTuningAdapter'] pengcheng.mind-mindformer/mindformers/pet/tuners/.ipynb_checkpoints/ada_adapter-checkpoint.py000066400000000000000000000022301476650141400332610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/pdf/1902.00751.pdf """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaAdapter(PetAdapter): r""" AdaAdapter is the adapter to modify the pretrained model, which uses adapter tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-mindformer/mindformers/pet/tuners/.ipynb_checkpoints/adalora_adapter-checkpoint.py000066400000000000000000000022521476650141400341430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.04647 """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaLoraAdapter(PetAdapter): r""" AdaLoraAdapter is the adapter to modify the pretrained model, which uses adapter lora tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-mindformer/mindformers/pet/tuners/.ipynb_checkpoints/lora_adapter-checkpoint.py000066400000000000000000000132121476650141400334730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.09685 """ import re from mindspore import nn from tk.delta.lora import LoRADense from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from .pet_adapter import PetAdapter from ..pet_config import PetConfig from ..utils import re_match_list def recursive_replace_dense_cell(net, config): """default replace all dense.""" # pylint: disable=W0212 for name, cell in net._cells.items(): if cell: # add white list spaces. if re_match_list(name, config.exclude_layers): continue if re.match(config.reg_rules, name): if not isinstance(cell, nn.Dense) and not isinstance(cell, Linear): continue in_channels = cell.in_channels out_channels = cell.out_channels dest_cell = LoRADense(in_channels=in_channels, out_channels=out_channels, lora_rank=config.lora_rank, lora_alpha=config.lora_alpha, lora_dropout=config.lora_dropout, has_bias=cell.has_bias, activation=cell.activation) # load weight of oriangal layers. dest_cell.matmul = cell.matmul dest_cell.weight = cell.weight if cell.has_bias: dest_cell.bias = cell.bias dest_cell.bias_add = cell.bias_add # Shard strategies now only support loaded by linear layers. if isinstance(cell, Linear): strategy_matmul = cell.matmul.in_strategy dest_cell.lora_dropout.dropout.shard((strategy_matmul[0],)) if cell.transpose_b: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (1, strategy_matmul[1][1]))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (strategy_matmul[1][0], 1))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), (strategy_matmul[0][0], strategy_matmul[1][0]))) else: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (strategy_matmul[1][0], 1))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (1, strategy_matmul[1][1]))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), (strategy_matmul[0][0], strategy_matmul[1][1]))) # pylint: disable=W0212 net._cells[name] = dest_cell else: recursive_replace_dense_cell(cell, config) return net class LoraAdapter(PetAdapter): r""" LoraAdapter is the adapter to modify the pretrained model, which uses lora tuning algorithm. Args: model (BaseModel): The base pretrained model of mindformers. pet_config (PetConfig): The configurition of the Pet model. Return: model (BaseModel): The model replace the linear layer with lora dense layer. Examples: 1.modify certain task of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> class LlamaForCausalLMWithLora(LlamaForCausalLM): >>> def __init__(self, config: LlamaConfig = None, pet=None): >>> super().__init__(config) >>> # get Pet tuning model. >>> self.pet = pet >>> self.pet.pet_config.reg_rules = r'.*wq|.*wv' >>> self.model = LoraAdapter.get_pet_model(self.model, self.pet.pet_config) >>> # freeze pretrained model >>> PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) 2.modify certain model of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> from mindformers.model.llama import LlamaModel >>> from mindformers.pet.pet_config import LoraConfig >>> llama_model = LlamaModel() >>> pet_config = LoraConfig() >>> llama_pet_model = LoraAdapter.get_pet_model(llama_model, pet_config) """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): model = model if model else PetAdapter.get_pretrained_model(config) if config.reg_rules is None: logger.warning("Lora Adapter use default replace rules: \'.*dense*|*linear*\'") config.reg_rules = r'.*dense*|.*linear*' model = recursive_replace_dense_cell(model, config) return model pengcheng.mind-mindformer/mindformers/pet/tuners/__init__.py000066400000000000000000000016731476650141400246670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """The export function for bert_pet.""" from .adalora_adapter import * from .ada_adapter import * from .lora_adapter import * from .prefix_tuning_adapter import * from .pet_adapter import * __all__ = ['AdaLoraAdapter', 'AdaAdapter', 'LoraAdapter', 'PetAdapter', 'PrefixTuningAdapter'] pengcheng.mind-mindformer/mindformers/pet/tuners/ada_adapter.py000066400000000000000000000022301476650141400253430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/pdf/1902.00751.pdf """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaAdapter(PetAdapter): r""" AdaAdapter is the adapter to modify the pretrained model, which uses adapter tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-mindformer/mindformers/pet/tuners/adalora_adapter.py000066400000000000000000000022521476650141400262250ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.04647 """ from mindspore import nn from .pet_adapter import PetAdapter from ..pet_config import PetConfig class AdaLoraAdapter(PetAdapter): r""" AdaLoraAdapter is the adapter to modify the pretrained model, which uses adapter lora tuning algorithm. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-mindformer/mindformers/pet/tuners/lora_adapter.py000066400000000000000000000132121476650141400255550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://arxiv.org/abs/2106.09685 """ import re from mindspore import nn from tk.delta.lora import LoRADense from mindformers.modules.layers import Linear from mindformers.tools.logger import logger from .pet_adapter import PetAdapter from ..pet_config import PetConfig from ..utils import re_match_list def recursive_replace_dense_cell(net, config): """default replace all dense.""" # pylint: disable=W0212 for name, cell in net._cells.items(): if cell: # add white list spaces. if re_match_list(name, config.exclude_layers): continue if re.match(config.reg_rules, name): if not isinstance(cell, nn.Dense) and not isinstance(cell, Linear): continue in_channels = cell.in_channels out_channels = cell.out_channels dest_cell = LoRADense(in_channels=in_channels, out_channels=out_channels, lora_rank=config.lora_rank, lora_alpha=config.lora_alpha, lora_dropout=config.lora_dropout, has_bias=cell.has_bias, activation=cell.activation) # load weight of oriangal layers. dest_cell.matmul = cell.matmul dest_cell.weight = cell.weight if cell.has_bias: dest_cell.bias = cell.bias dest_cell.bias_add = cell.bias_add # Shard strategies now only support loaded by linear layers. if isinstance(cell, Linear): strategy_matmul = cell.matmul.in_strategy dest_cell.lora_dropout.dropout.shard((strategy_matmul[0],)) if cell.transpose_b: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (1, strategy_matmul[1][1]))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (strategy_matmul[1][0], 1))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][0]), (strategy_matmul[0][0], strategy_matmul[1][0]))) else: dest_cell.lora_a_matmul.shard((strategy_matmul[0], (strategy_matmul[1][0], 1))) dest_cell.lora_b_matmul.shard(((strategy_matmul[0][0], 1), (1, strategy_matmul[1][1]))) dest_cell.mul.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), ())) dest_cell.add.shard(((strategy_matmul[0][0], strategy_matmul[1][1]), (strategy_matmul[0][0], strategy_matmul[1][1]))) # pylint: disable=W0212 net._cells[name] = dest_cell else: recursive_replace_dense_cell(cell, config) return net class LoraAdapter(PetAdapter): r""" LoraAdapter is the adapter to modify the pretrained model, which uses lora tuning algorithm. Args: model (BaseModel): The base pretrained model of mindformers. pet_config (PetConfig): The configurition of the Pet model. Return: model (BaseModel): The model replace the linear layer with lora dense layer. Examples: 1.modify certain task of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> class LlamaForCausalLMWithLora(LlamaForCausalLM): >>> def __init__(self, config: LlamaConfig = None, pet=None): >>> super().__init__(config) >>> # get Pet tuning model. >>> self.pet = pet >>> self.pet.pet_config.reg_rules = r'.*wq|.*wv' >>> self.model = LoraAdapter.get_pet_model(self.model, self.pet.pet_config) >>> # freeze pretrained model >>> PetAdapter.freeze_pretrained_model(self, self.pet.pet_type) 2.modify certain model of llama >>> from mindformers.pet.tuners.lora_adapter import LoraAdapter >>> from mindformers.model.llama import LlamaModel >>> from mindformers.pet.pet_config import LoraConfig >>> llama_model = LlamaModel() >>> pet_config = LoraConfig() >>> llama_pet_model = LoraAdapter.get_pet_model(llama_model, pet_config) """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): model = model if model else PetAdapter.get_pretrained_model(config) if config.reg_rules is None: logger.warning("Lora Adapter use default replace rules: \'.*dense*|*linear*\'") config.reg_rules = r'.*dense*|.*linear*' model = recursive_replace_dense_cell(model, config) return model pengcheng.mind-mindformer/mindformers/pet/tuners/pet_adapter.py000066400000000000000000000054041476650141400254140ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: PET Adapter is the base adapter class for Parameter Efficient Tuning of MindFormers. """ import os from mindspore import nn from tk.graph.freeze_utils import freeze_delta from mindformers.auto_class import AutoModel from mindformers.mindformer_book import MindFormerBook from ..pet_config import PetConfig from ..constants import BaseModelInitType, PetType class PetAdapter: r""" PetAdapter is the base class of adapter to modify the pretrained model. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): """Add efficient tuning parameters to ptm.""" raise NotImplementedError("should implemented by the certain tuning algorithm.") @classmethod def get_pretrained_model(cls, config): """ Get pretrained model from config. """ init_type = config.base_model.init_type if init_type == BaseModelInitType.MODEL_CONFIG: config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'clip', 'model_config', config.base_model.model_config) base_model = AutoModel.from_config(config_path) if config.base_model.model_ckpt is None: raise ValueError("init base from model config, the model ckpt cannot be None") elif init_type == BaseModelInitType.MODEL_NAME: base_model = AutoModel.from_pretrained(config.base_model_name) elif init_type == BaseModelInitType.MODEL_DIR: checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), config.base_model_dir_name) base_model = AutoModel.from_pretrained(checkpoint_dir) else: raise ValueError("type of init base model must in BaseModelInitType") return base_model @classmethod def freeze_pretrained_model(cls, model, pet_type: PetType): """ Freeze the parameters of ptm which no update in the tuning process. Notes: Refer to mindpet api. """ freeze_delta(model, pet_type) pengcheng.mind-mindformer/mindformers/pet/tuners/prefix_tuning_adapter.py000066400000000000000000000022151476650141400275020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: Low-Rank Adapter algrithm for mindformers' pretrained model. Reference: https://aclanthology.org/2021.acl-long.353.pdf """ from mindspore import nn from mindformers.pet.tuners.pet_adapter import PetAdapter from mindformers.pet.pet_config import PetConfig class PrefixTuningAdapter(PetAdapter): r""" Prefix tuning implement. """ @classmethod def get_pet_model(cls, model: nn.Cell = None, config: PetConfig = None): return super().get_pet_model(model, config) pengcheng.mind-mindformer/mindformers/pet/utils.py000066400000000000000000000021101476650141400227330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Note: functions. """ import re from typing import Optional, List def re_match_list(mstr, patterns: Optional[List[str]] = None): """string regular match""" if patterns is None: return False for pattern in patterns: if not isinstance(pattern, str): raise TypeError(f"List item '{pattern}' is not a string.") if re.match(pattern, mstr): return True return False pengcheng.mind-mindformer/mindformers/pipeline/000077500000000000000000000000001476650141400222445ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__init__.py000066400000000000000000000041111476650141400243520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Pipeline API.""" from .pipeline import pipeline from .build_pipeline import build_pipeline from .base_pipeline import BasePipeline from .image_classification_pipeline import ImageClassificationPipeline from .zero_shot_image_classification_pipeline import ZeroShotImageClassificationPipeline from .image_to_text_generation_pipeline import ImageToTextGenerationPipeline from .translation_pipeline import TranslationPipeline from .fill_mask_pipeline import FillMaskPipeline from .text_classification_pipeline import TextClassificationPipeline from .token_classification_pipeline import TokenClassificationPipeline from .question_answering_pipeline import QuestionAnsweringPipeline from .text_generation_pipeline import TextGenerationPipeline from .masked_image_modeling_pipeline import MaskedImageModelingPipeline from .segment_anything_pipeline import SegmentAnythingPipeline __all__ = ['ZeroShotImageClassificationPipeline', 'ImageClassificationPipeline', 'ImageToTextGenerationPipeline', 'pipeline', 'BasePipeline'] __all__.extend(translation_pipeline.__all__) __all__.extend(fill_mask_pipeline.__all__) __all__.extend(text_classification_pipeline.__all__) __all__.extend(token_classification_pipeline.__all__) __all__.extend(question_answering_pipeline.__all__) __all__.extend(text_generation_pipeline.__all__) __all__.extend(masked_image_modeling_pipeline.__all__) __all__.extend(image_to_text_generation_pipeline.__all__) pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/000077500000000000000000000000001476650141400244545ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/__init__.cpython-39.pyc000066400000000000000000000024761476650141400306550ustar00rootroot00000000000000a ?cfI@sdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZdd lmZdd lmZddlmZddlmZgdZee jee jeejeejeejeejeejee jdS)zMindFormers Pipeline API.)pipeline)build_pipeline) BasePipeline)ImageClassificationPipeline)#ZeroShotImageClassificationPipeline)ImageToTextGenerationPipeline)TranslationPipeline)FillMaskPipeline)TextClassificationPipeline)TokenClassificationPipeline)QuestionAnsweringPipeline)TextGenerationPipeline)MaskedImageModelingPipeline)SegmentAnythingPipeline)rrrrrN)__doc__rr base_pipelinerZimage_classification_pipelinerZ'zero_shot_image_classification_pipelinerZ!image_to_text_generation_pipelinerZtranslation_pipelinerZfill_mask_pipeliner Ztext_classification_pipeliner Ztoken_classification_pipeliner Zquestion_answering_pipeliner Ztext_generation_pipeliner Zmasked_image_modeling_pipelinerZsegment_anything_pipeliner__all__extendrrH/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/__init__.pys.                     pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/base_pipeline.cpython-39.pyc000066400000000000000000000166671476650141400317240ustar00rootroot00000000000000a ?cf3$@sdZddlmZmZddlmZmZddlZddl m Z ddl m Z m Z ddl mZmZmZmZddlmZmZdd lmZdd lmZd d lmZd d lmZmZmZGdddeZdS) BasePipeline)ABCabstractmethod)OptionalUnionN)tqdm)TensorModel)GeneratorDatasetVisionBaseDataset SourceDatasetMappableDataset) BatchDataset RepeatDataset)logger) print_dict) AutoModel) BaseModel BaseTokenizerBaseImageProcessorcseZdZdZiZdeeeefe e e e dfdd Z dee eeeejfe edddZed d Zeeeejefeeed d d Zeeefeeed ddZeeeeejefdddZeeeeejefdddZeeeeejefdddZeddZZS)ra Base Pipeline For All Task Pipelines Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): The tokenizer of model, it could be None if the model do not need tokenizer. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. N)model tokenizerimage_processorc stt|||_t|tr6||jvr6t||_ n:t|t rH||_ n(t|t r\|j |_ nt dt|d||_||_|jfi|\|_|_|_d|_|dd|_dS)NzGmodel should be str or inherited from BaseModel or Model, but got type .r batch_size)superr__init__r isinstancestr _support_listrfrom_pretrainednetworkrr predict_network TypeErrortyperr_sanitize_parameters_preprocess_params_forward_params_postprocess_params call_countpop _batch_size)selfrrrkwargs __class__M/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/base_pipeline.pyr4s     zBasePipeline.__init__)inputsrc Ks<|jfi|\}}}i|j|}i|j|}i|j|}t|ttttf}t|t }|j d7_ |j dkr|st d|dur|j durd}n|j }|dkr|sd}t d|rt dt|ttfs||}g} t|D]} | || |||qn(|r(|||||} n|||||} | S)aZCall Method Args: inputs (Union[GeneratorDataset, list, str, etc]): The inputs of pipeline, the type of inputs depends on task. batch_size (Optional[int]): The batch size for a GeneratorDataset input, for other types of inputs, the batch size would be set to 1 by default. zYou seem to be using the pipeline sequentially for numerous samples. In order to maximize efficiency please set input a mindspore.dataset.GeneratorDataset.Nz.batch_size is set to 1 for non-dataset inputs.zdataset is processing.)r&r'r(r)rr r r r listr*rinfor,rrbatchrcreate_dict_iteratorextend run_single run_multi) r-r3rr.preprocess_paramsforward_paramspostprocess_paramsZ is_datasetis_listoutputsitemsr1r1r2__call__Js@          zBasePipeline.__call__cKs tddS)zSanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. Raises: NotImplementedError: If the method is not implemented. z$_sanitize_parameters not implementedNNotImplementedError)r-Zpipeline_parametersr1r1r2r&s z!BasePipeline._sanitize_parameters)r3r=r>r?cCs:|j|fi|}|j|fi|}|j|fi|}|S)aRun Single method This function is used to run a single forward process for task. Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. ) preprocessforward postprocess)r-r3r=r>r? model_inputs model_outputsrAr1r1r2r;szBasePipeline.run_singlec Cs(g}|D]}||||||q|S)aRun Multiple Method This function is used to run a list input for task. Args: inputs (Union[list, tuple, iterator]): The iterable input for pipeline. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. )r:r;)r-r3r=r>r?rAitemr1r1r2r<s  zBasePipeline.run_multi)r3cKs tddS)aDThe Preprocess For Task Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. Raises: NotImplementedError: If the method is not implemented. zpreprocess not implemented.NrD)r-r3r=r1r1r2rFs zBasePipeline.preprocess)rIcKs tddS)aWThe Forward Process of Model Args: model_inputs (Union[dict, str, etc]): The output of preprocess, the type of model_inputs depends on task. forward_params (dict): The parameter dict for model forward. Raises: NotImplementedError: If the method is not implemented. zforward not implemented.NrD)r-rIr>r1r1r2rGs zBasePipeline.forward)rJcKs tddS)aZThe Postprocess of Task Args: model_outputs (Union[dict, str, etc]): The output of model forward, the type of model_outputs depends on task. postprocess_params (dict): The parameter dict for post process. Raises: NotImplementedError: If the method is not implemented. zpostprocess not implemented.NrD)r-rJr?r1r1r2rHs zBasePipeline.postprocesscCstd|jt|jdS)show_support_listzsupport list of %s is:N)rr7__name__rr )clsr1r1r2rLszBasePipeline.show_support_list)NN)N) rM __module__ __qualname____doc__r rrrr rrrrr r6rnparrayintrCrr&dictr;tupler<rFrGrH classmethodrL __classcell__r1r1r/r2r%s@  5    ) rQabcrrtypingrrnumpyrRr mindsporerr mindspore.datasetr r r r Z!mindspore.dataset.engine.datasetsrrmindformers.toolsrmindformers.mindformer_bookr auto_classrmodelsrrrrr1r1r1r2s    pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/build_pipeline.cpython-39.pyc000066400000000000000000000032741476650141400320770ustar00rootroot00000000000000a ?cf) @s2dZddlmZmZmZdeeeedddZdS) zBuild Pipeline API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfigNpipeline)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)aBuild Pipeline API. Instantiate the task pipeline from MindFormerRegister's registry. Args: config (dict): The task pipeline's config. default_args (dict): The default args of pipeline. module_type (str): The module type of MindFormerModuleType. Default: 'pipline'. class_name (str): The class name of task pipeline. Returns: The task pipeline instance by config. Examples: >>> from mindformers import build_pipeline >>> pipeline_config = {'type': 'zero_shot_image_classification', 'model': 'clip_vit_b_32', 'candidate_labels': ["sunflower", "tree", "dog", "cat", "toy"], 'hypothesis_template': "This is a photo of {}."} >>> classifier = build_pipeline(pipeline_config) >>> type(classifier) N)r) isinstancedictrrget_instance_from_cfgrPIPELINE get_instance)rrrr kwargsrN/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/build_pipeline.pybuild_pipelinesr)NNrN)__doc__mindformers.tools.registerrrrr strrrrrrspengcheng.mind-mindformer/mindformers/pipeline/__pycache__/fill_mask_pipeline.cpython-39.pyc000066400000000000000000000114771476650141400327450ustar00rootroot00000000000000a ?cf.@sdZddlZddlZddlZddlmZmZmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZmZd gZejejd d Gdd d eZdS)TranslationPipelineN)opsTensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel TokenizerFillMaskPipeline fill_mask)aliascsReZdZdZedZdZdfdd ZddZ d d Z d d Z d dZ Z S)rz Pipeline for mask fill Args: model: a pretrained model (str or BaseModel) in _supproted_list. tokenizer : a tokenizer (None or Tokenizer) for text processing bertZfillmaskNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdd|_tj||fi|dS)Nz@tokenizer should be inherited from PretrainedTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__rr input_textsuper__init__)selfmodelrkwargsr!R/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/fill_mask_pipeline.pyr%+s$      zFillMaskPipeline.__init__c std|vrtd|jjdi}gd}i}|D]}||vr0||||<q0gdfdd|D}|||fS)z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)r-r.r/r0r1r2paddingcsi|]\}}|vr||qSr*r*).0kvZpreprocess_key_namer*r+ Rz9FillMaskPipeline._sanitize_parameters..)r r!r"getitems)r&pipeline_parameterspostprocess_paramsforward_key_nameforward_kwargsitempreprocess_paramsr*r7r+_sanitize_parametersBsz%FillMaskPipeline._sanitize_parameterscKst|tr*|d}t|tjr*|}t|tr8|nd|_|dd}|dd}|j |f||dd|}t }||dd ||d d ||d d |t|j j d d S) ax Preprocess of mask fill Args: inputs (url, PIL.Image, tensor, numpy): the image to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed image. textrr2r3ms)r2r3return_tensors input_idsrattention_masktoken_type_ids)rG input_mask token_type_idmasked_lm_positions) rdict mindsporerasnumpytolistrr#poprr ExpandDims mask_index)r&inputsrAr2r3 expand_dimsr*r*r+ preprocessUs$         zFillMaskPipeline.preprocesscKs$|dd|jfi|}d|iS)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. NoneN output_ids)rQnetwork)r& model_inputsforward_paramsrXr*r*r+forwardqs zFillMaskPipeline.forwardc Ks|dd}g}tj|dd}t|D]V\}}|jjt|gdd}|dd}||||f||d |j d |d|_ q*|d |j i|gS) z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results rXr )axisT)skip_special_tokens r)scoretokenZ token_strz[MASK]sequence) rOnpargmax enumeraterdecodeintreplaceappendr#) r& model_outputsr=outputsZ tokens_dictZ max_tokensindZtokenidrbr*r*r+ postprocesss  zFillMaskPipeline.postprocess)N)r" __module__ __qualname____doc__r get_model_support_listr return_namer%rBrVr\rn __classcell__r*r*r)r+rs )rqos.pathrnumpyrdrNrrr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsrr__all__registerPIPELINErr*r*r*r+s  image_classification_pipeline.cpython-39.pyc000066400000000000000000000144331476650141400350550ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cf@sdZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZd dlmZejej ddGdddeZ!dS)z"Image Classification Pipeline API.)OptionalUnionN)Image) operations)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor) load_image)MindFormerRegisterMindFormerModuleType)labels) BasePipelineimage_classification)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ImageClassificationPipelineafPipeline for image classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import ImageClassificationPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageClassificationFoPipeline requires for a image_processor.F)moder) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rsuper__init__ set_train)selfrrkwargs __class__]/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/image_classification_pipeline.pyr"Ds"       z$ImageClassificationPipeline.__init__cKs:i}i}ddg}|D]}||vr||||<q|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. top_kcandidate_labels)get)r$pipeline_parameterspreprocess_paramspostprocess_paramsZ post_listitemr(r(r)_sanitize_parameters\sz0ImageClassificationPipeline._sanitize_parameters)inputscKs6t|tr|d}t|tr$t|}||}d|iS)zThe Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. imageimage_processed)rdictrr r)r$r2r.r4r(r(r) preprocessls   z&ImageClassificationPipeline.preprocess) model_inputscKs:|dd|d}||d}t|}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr4rprobs)popnetworkPSoftmaxasnumpy)r$r7forward_paramsr4Zlogits_per_imager9r(r(r)forwards  z#ImageClassificationPipeline.forwardc Ks|dd}|dd}|d}g}t|tr:t|}n"t|trJ|}ntdt||durltd|D]L}tt ||d d d } |dur| dt |t |} | d d | Dqp|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. r*r+imagenetr9zRThe candidate_labels should be dataset name (str) or custom labels (list) but got NzZThe custom candidate_labels is None or the input dataset labels name is not supported yet.cSs |d S)Nrr()xr(r(r)z9ImageClassificationPipeline.postprocess..)keycSsg|]\}}||dqS))scorelabelr().0Z score_itemrHr(r(r) sz;ImageClassificationPipeline.postprocess..) r:rrrr,listr rsortedzipminlenappend) r$ model_outputsr/r*r+scoresoutputsZ inputs_labelsrGZ sorted_resr(r(r) postprocesss*       z'ImageClassificationPipeline.postprocess)N)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r"r1rrnpndarrayr6r5r@rT __classcell__r(r(r&r)r!sr)"rXtypingrrnumpyr[PILr mindspore.opsrr< mindsporerrmindformers.auto_classrr mindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsr mindformers.tools.registerrrZmindformers.dataset.labelsr base_pipelinerregisterPIPELINErr(r(r(r)s      image_to_text_generation_pipeline.cpython-39.pyc000066400000000000000000000123651476650141400357650ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cfY@sdZddlmZmZddlZddlmZddlm Z m Z ddl m Z ddlmZmZddlmZdd lmZmZdd lmZdd lmZmZd d lmZdgZejejddGdddeZ dS)z"Image Classification Pipeline API.)OptionalUnionN)Image)TensorModel) operations) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor load_image)MindFormerRegisterMindFormerModuleType) BasePipelineImageToTextGenerationPipelineimage_to_text_generation)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ra'Pipeline for image to text generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt d| dd|_tj|jdd f||d |dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zH is not supported by ImageToTextGenerationPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageToTextGenerationPipeline requires for a image_processor.hypothesis_templatez{}F)mode)r tokenizer) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rpoprsuper__init__ set_train)selfrrrkwargs __class__a/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/image_to_text_generation_pipeline.pyr%3s$       z&ImageToTextGenerationPipeline.__init__c Ksi}i}i}dg}gd}gd}|D]}||vr&||||<q&|D]}||vrF||||<qF|D]}||vrf||||<qf|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. top_k)r max_lengthpadding)r-top_p do_sample eos_token_idrepetition_penaltyr.seed)get) r'pipeline_parameterspreprocess_paramspostprocess_paramsforward_params post_listpre_listZ forward_listitemr+r+r,_sanitize_parametersLs z2ImageToTextGenerationPipeline._sanitize_parameters)inputsc Kst|tr|d}t|tr$t|}t|trxz.r. r/rNrmsF)r.r/return_tensorsadd_special_tokens input_idsr)image_processedprompt_input_ids) rdictrrlistrr#rshaperlenP ExpandDims) r'r>r7rHr.r/rZ prompt_listZprompt_processedrIr+r+r, preprocessgs2       z(ImageToTextGenerationPipeline.preprocess) model_inputscKs&|d}|d}|j||}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. rHrI output_ids)networkZgenerate_text_for_image)r'rQr9rHrIZoutput_ids_per_imager+r+r,forwardsz%ImageToTextGenerationPipeline.forwardcKs|d}|jj|dd}|S)NrRT)skip_special_tokens)rdecode)r' model_outputsr8rRoutputsr+r+r, postprocesssz)ImageToTextGenerationPipeline.postprocess)NN)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r%r=rrnpndarrayrPrJrTrY __classcell__r+r+r)r,r"s ))!r]typingrrnumpyr`PILr mindsporerr mindspore.opsrrNmindformers.auto_classrr mindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsrmindformers.tools.registerrr base_pipeliner__all__registerPIPELINErr+r+r+r,s     masked_image_modeling_pipeline.cpython-39.pyc000066400000000000000000000125261476650141400352050ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cfu@sdZddlZddlmZmZddlZddlmZddl m Z m Z ddl m Z mZddlmZddlmZmZdd lmZdd lmZmZd d lmZd gZddlmZejejddGdd d eZ dS)z#Masked Image Modeling Pipeline API.N)OptionalUnion)Image)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor) load_image)MindFormerRegisterMindFormerModuleType) BasePipelineMaskedImageModelingPipeline)LOCAL_DEFAULT_PATHmasked_image_modeling)aliascseZdZdZedZdee e e fe e dfdd ZddZee ejeejfd d d Zed d dZddZZS)ra Pipeline for masked image modeling Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import MaskedImageModelingPipeline >>> from mindformers import ViTMAEImageProcessor >>> processor = ViTMAEImageProcessor(size=224) >>> reconstructor = MaskedImageModelingPipeline( ... model='mae_vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> reconstructor(np.uint8(np.random.random((5, 3, 255, 255)))) rN)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z;MaskedImageModelingPipeline requires for a image_processor.F)moder) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyper ValueErrorr rsuper__init__ set_train)selfrrkwargs __class__^/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/masked_image_modeling_pipeline.pyr"As"       z$MaskedImageModelingPipeline.__init__cKs6i}i}g}|D]}||vr||||<q|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. )get)r$pipeline_parameterspreprocess_paramspostprocess_params post_listitemr(r(r)_sanitize_parametersYsz0MaskedImageModelingPipeline._sanitize_parameters)inputscKs6t|tr|d}t|tr$t|}||}d|iS)aThe Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be reconstructed. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. imageimage_processed)rdictrr r)r$r1r,r3r(r(r) preprocessis   z&MaskedImageModelingPipeline.preprocess) model_inputscKs.|dd|d}|j|}|}d|iS)zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr3 images_array)popnetworkasnumpy)r$r6forward_paramsr3r8r(r(r)forward|s   z#MaskedImageModelingPipeline.forwardc Ks|d}g}t|D]\}}|d}t|jjj}t|jjj}|||}|d}|tj }t |} t j td|d} | | || | dq|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: classification results. r8)rrrZ output_imagez.jpg)infodata) enumerate transposenparrayr normalizemeanstdastypeuint8r fromarrayospathjoinrsaveappend) r$ model_outputsr-r8outputsiZ image_arrayrFrGZreconstruct_image image_pathr(r(r) postprocesss      z'MaskedImageModelingPipeline.postprocess)N)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysrrrr rrr r"r0rrrCndarrayr5r4r=rT __classcell__r(r(r&r)r$s)!rXrKtypingrrnumpyrCPILr mindsporerrmindformers.auto_classrrmindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.image_toolsr mindformers.tools.registerr r base_pipeliner__all__Z tools.utilsrregisterPIPELINErr(r(r(r)s     pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/pipeline.cpython-39.pyc000066400000000000000000000075211476650141400307170ustar00rootroot00000000000000a ?cf @sdZddlmZddlmZmZmZddlmZddl m Z m Z m Z m Z mZmZmZddlmZddlmZdd lmZeZeZGd d d eZdeeeee eeeeffeeeeeeedddZddZd S)z pipeline )Enum)OptionalUnionTuple)Model) build_modelbuild_tokenizerbuild_processor BaseModel BaseTokenizerBaseImageProcessorBaseAudioProcessor)MindFormerBook)MindFormerConfig)build_pipelinec@seZdZdZdZdS)BackendmsZmsliteN)__name__ __module__ __qualname__MSMS_LITErrH/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/pipeline.pyr"srNr)taskmodel tokenizerimage_processoraudio_processorbackendc Ksl|tjjkr2ddlm}||||||fi|}n6|tjjkrXt|||||fi|}ntd|d|S)aPipeline for downstream tasks Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Optional[Union[str, BaseModel]]): The model used for task. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. backend(str): The inference backend. Default "ms", now support ["ms", "mslite"]. Return: A task pipeline. Raises: KeyError: If the task or model is not supported. Examples: >>> from mindformers import pipeline >>> from mindformers.tools.image_tools import load_image >>> classifier = pipeline("zero_shot_image_classification", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] r)get_mslite_pipelinezThe inference backend "z@" is not supported,please select a backend from ["ms", "mslite"])rrvalueZmindformers.inferencer!rget_ms_pipeline ValueError) rrrrrr kwargsr! task_pipelinerrrpipeline's'   r'c Ks|tvr$t|dtdt|trV|tvrLtdtd|d|}d}nd}tt||}|jj d|i|durt |j}|durt |j drt|j j}|durt |j d rt|j j}|durt|j j}tf|||||d |}|S) zget mindspore infer pipeline.z9 is not supported by pipeline. please select a task from .zmodel must be in z& when model's type is string, but get Ncommoncheckpoint_name_or_pathrr) class_namerrrr)SUPPORT_PIPELINESkeysKeyError isinstancestrSUPPORT_MODEL_NAMESrgetr model_configupdaterhasattr processorr rrrrr) rrrrrr% model_nameZpipeline_configr&rrrr#Ys>       r#)NNNNNr)__doc__enumrtypingrrr mindsporermindformers.modelsrrr r r r r mindformers.mindformer_bookrmindformers.tools.registerrrget_pipeline_support_task_listr,get_model_name_support_listr1rr0r'r#rrrrs2  $      2question_answering_pipeline.cpython-39.pyc000066400000000000000000000277541476650141400346560ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cf.H@sdZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddlmZmZddlmZmZdd lmZd d lmZdd lmZmZdd lmZmZdgZejej ddGdddeZ!dS)QuestionAnsweringPipelineN)TensorModel)dtype)BasicTokenizer)convert_examples_to_features SquadExample) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizerquestion_answering)aliascseZdZdZedZdfd d Zd d Z d dZ ddZ ddZ ddZ ddZddZddZddZddZZS) ragPipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import QuestionAnsweringPipeline >>> from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig >>> input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] >>> tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') >>> qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') >>> model = BertForQuestionAnswering(qa_squad_config) >>> qa_pipeline = QuestionAnsweringPipeline(task='question_answering', ... model=model, ... tokenizer=tokenizer) >>> results = qa_pipeline(input_data) >>> print(results) [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] r@r c  slt|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdd|_d|_tdd|_||krt d |d |d |d kr t d |d |d kr"t d||| d<|| d<|| d<|| d<|| d<|| d<tj||fi| dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.T) do_lower_casez`doc_stride` (z ) is larger than `max_seq_len` ()r z$top_k parameter should be >= 1 (got z-max_answer_len parameter should be >= 1 (got doc_stridemax_question_len max_seq_lentop_k n_best_sizemax_answer_len) isinstancestr _support_listospathisdirr from_pretrained tokenizerr r TypeErrortype ValueError __class____name__rrfeaturesexamplesrbasic_tokenizersuper__init__) selfmodelr*rrrr r!r"kwargsr.[/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/question_answering_pipeline.pyr4Cs@         z"QuestionAnsweringPipeline.__init__c sbd|vrtd|jjdgdgdfdd|D}fdd|D}|i|fS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)rrr)r r!r"csi|]\}}|vr||qSr9r9.0kv)preprocess_key_namer9r: uzBQuestionAnsweringPipeline._sanitize_parameters..csi|]\}}|vr||qSr9r9r<)postprocess_key_namer9r:rAvrB)r-r.r/items)r5pipeline_parameterspreprocess_paramspostprocess_paramsr9)rCr@r:_sanitize_parameterslsz.QuestionAnsweringPipeline._sanitize_parameterscKst|tstdd|vr"td|d\}}|}|}td||ddd}t|g|j|d|d|ddd }||_|g|_ |S) a# Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output return_tensors ("ms"): the type of returned tensors Return: processed text. z!The type of inputs should be str.-zHThe inputs should contain context and question separeated by '-' syntax.NrrrF)r1r*rrr is_training) r#r$r-splitstripr rr*r0r1)r5inputsrF context_text question_textZ squad_exampler0r9r9r: preprocesszs&  z$QuestionAnsweringPipeline.preprocessc Ks|jdg}|D]}t|jtjdt|jtjdt|jtjdt|j tjdt|j tjdt|j tjdd}|j fi|\}}}t dgd}t|d} dd|dD} dd|dD} ||| | | dq|S) z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. Fr) input_ids input_mask token_type_idstart_position end_position unique_id RawResult)rV start_logits end_logitscSsg|] }t|qSr9floatr=xr9r9r: rBz5QuestionAnsweringPipeline.forward..cSsg|] }t|qSr9rZr\r9r9r:r^rB)r6 set_trainrrQmstypeint32 expand_dimsrRrSrTrUrVnetwork collections namedtupleintappend) r5 model_inputsforward_params model_outputsfeatureidsstartendrWrVrXrYr9r9r:forwards&    z!QuestionAnsweringPipeline.forwardcKs.|d}|d}|d}tt}|jD]}||j|q(i}|D]} | || j<qFt|jD]\} } || } | | |||} | | | | |}g}d}|D]$}||j |j |s|j r|}q||}g}t|D]F\}}i}|j |d<t||d|d<|j|d<|j|d <||qq`|d|S) z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results r r!r"Ntextscorermrn)rd defaultdictlistr0 example_indexrgrV enumerater1_get_prelim_predictions _get_nbest start_logit end_logitrp_compute_softmaxroundrmrn)r5rjrGr r!r"example_index_to_featuresrkunique_id_to_resultresultruexampler0prelim_predictionsnbest total_scoresbest_non_null_entryentryprobsresultsioutputr9r9r: postprocesss@        z%QuestionAnsweringPipeline.postprocessc Cstdgd}g}t|D]\}}|j|vr0q||j} || j|} || j|} | D]} | D]} | t|jkrvqb| t|jkrqb| |j vrqb| |j vrqb|j | dsqb| | krqb| | d}||krqb| ||| | | j| | j| dqbqZqt |dddd}|S) zget prelim predictionsPrelimPrediction) feature_index start_index end_indexryrzFr cSs |j|jS)N)ryrzr]r9r9r:rBzCQuestionAnsweringPipeline._get_prelim_predictions..Tkeyreverse)rdrervrV_get_best_indexesrXrYlentokenstoken_to_orig_maptoken_is_max_contextgetrgsorted)r5r0r~r"r!_PrelimPredictionrrrkr start_indexes end_indexesrrlengthr9r9r:rwsV      z1QuestionAnsweringPipeline._get_prelim_predictionsc Cs`tdgd}i}g}|D]}t||kr4q0||j} |jdkr| j|j|jd} | j|j} | j|j} |j| | d} d | }| dd}| dd}| }d | }d | }| ||}||j| | \}}||vrqd ||<nd}d ||<d \}}||||j|j||dq|sJ||d d d d t|dks\J|S)zget nbest predictionsNbestPrediction)rpryrzrmrnrr  z ##z##T)rempty)rpryrz)rdrerrrrrr doc_tokensjoinreplacerLrK_get_final_text_get_answer_indexrNrgryrz)r5rr0rr!_NbestPredictionseen_predictionsrpredrk tok_tokensorig_doc_start orig_doc_end orig_tokenstok_text orig_text final_textchar_start_indexchar_end_indexr9r9r:rx$sV             z$QuestionAnsweringPipeline._get_nbestc CsTd}t|dD]8\}}||kr&|}||kr:|t|}|t|d7}q||fS)Nrrr )rvrKr) r5rNrr char_indexrwordrrr9r9r:r[s z+QuestionAnsweringPipeline._get_answer_indexcCs||sgSd}|D]}|dus$||kr|}qg}d}|D]$}t||}||||7}q6g}|D]}|||qd|S)z,Compute softmax probability over raw logits.Nr)mathexprg)r5scores max_scorerr exp_scores total_sumr]rr9r9r:r{fs   z*QuestionAnsweringPipeline._compute_softmaxcCsdd}d|j|}||}|dkr0|S|t|d}||\}}||\} } t|t| krl|Si} t| D]\} } | | | <qzd}|| vr| |}||vr||}|dur|Sd}|| vr| |}||vr||}|dur|S|||d}|S)z;Project the tokenized prediction back to the original text.cSsPg}t}t|D](\}}|dkr&q||t|<||qd|}||fS)Nrr)rd OrderedDictrvrrgr)rpns_chars ns_to_s_maprcns_textr9r9r: _strip_spaces~s   z@QuestionAnsweringPipeline._get_final_text.._strip_spacesrrr N)rr2tokenizefindrsix iteritems)r5 pred_textrrrrTrU orig_ns_textorig_ns_to_s_map tok_ns_texttok_ns_to_s_maptok_s_to_ns_mapr tok_indexorig_start_positionns_start_positionorig_end_positionns_end_position output_textr9r9r:r|s:     z)QuestionAnsweringPipeline._get_final_textcCsJtt|dddd}g}t|D]"\}}||kr6qF||dq"|S)z"Get the n-best logits from a list.cSs|dS)Nr r9rr9r9r:rrBz=QuestionAnsweringPipeline._get_best_indexes..Trr)rrvrg)r5logitsr!index_and_score best_indexesrrrr9r9r:rsz+QuestionAnsweringPipeline._get_best_indexes)rrrr rr)r/ __module__ __qualname____doc__r get_pipeline_support_task_listkeysr%r4rHrProrrwrxrr{rr __classcell__r9r9r8r:r$s)' 127 3)"ros.pathr&rdrr mindsporerrmindspore.commonrr`mindformers.modelsrZ#dataset.dataloader.squad_dataloaderrr auto_classr r mindformer_bookr base_pipelinerZtools.registerrrmodelsrr__all__registerPIPELINErr9r9r9r:s     pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/segment_anything_pipeline.cpython-39.pyc000066400000000000000000000363011476650141400343400ustar00rootroot00000000000000a ?cfi@s dZddlmZmZddlZddlZddlZddlm Z ddlm Z m Z ddl m Z mZddlmZddlmZmZdd lmZmZdd lmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)d d l*m+Z+d gZ,ej-ej.ddGdd d e+Z/dS)z"Image Classification Pipeline API.)OptionalUnionN)ops)TensorModel) AutoProcessor AutoModel)MindFormerBook) BaseModelBaseImageProcessor)MindFormerRegisterMindFormerModuleType)MaskData area_from_rlebatch_iteratorbatched_mask_to_boxbox_xyxy_to_xywhbuild_all_layer_point_gridscalculate_stability_scorecoco_encode_rlegenerate_crop_boxesis_box_near_crop_edge mask_to_rleremove_small_regions rle_to_maskuncrop_boxes_xyxy uncrop_masks uncrop_pointsbox_areanms) BasePipelineSegmentAnythingPipelinesegment_anything)aliasc seZdZdZedZd-ee e e fe e dfdd ZddZd d Zd d Zd dZd.e ddddZd/ddZd0e dddZddddZddZd1ddd(d)Zeeeed*d+d,ZZS)2r"a[Pipeline for image segment Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import SegmentAnythingPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] r#N)modelimage_processorc st|trh||jvrR|dur(t|j}t|tsFtdt|dt |}nt |d|jdt|t t fstdt|d|durt dt j|jddfd|i||jfi|tjd d |_|dS) NzEimage_processor should be inherited from BaseImageProcessor, but got .zI is not supported by ImageClassificationForPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ImageClassificationFoPipeline requires for a image_processor.Fmoder&axis) isinstancestr _support_listrfrom_pretrainedr&r TypeErrortyper ValueErrorr rsuper__init__ set_train reset_configrStackstack reset_image)selfr%r&kwargs __class__Y/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/segment_anything_pipeline.pyr5Ws(        z SegmentAnythingPipeline.__init__c Ksddi}ddi}ddi}dg}|D]}||vr"||||<q"d|d<gd}|D]"}||vrR||||<d|d<qRgd}|D]}||vr||||<qdg} | D]}||vr||||<q|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. seg_imageFr7) points_per_sidepoints_per_batchpred_iou_threshstability_score_threshstability_score_offsetbox_nms_thresh crop_n_layerscrop_nms_threshcrop_overlap_ratiocrop_n_points_downscale_factor point_gridsmin_mask_region_area output_modeT)rAmultimask_output return_logits)get) r;pipeline_parameterspreprocess_paramsforward_paramspostprocess_paramsZpreprocess_list_0itemZpreprocess_list_1 forward_list post_listr?r?r@_sanitize_parametersts, z,SegmentAnythingPipeline._sanitize_parameterscKsi}|d}|s|dd}|durX||\}}}||d<d|d<||d<||d<n:|jsfJd|j}d|d<|j|d<|j|d<|j|d<|dd} |d d} |d d} |d d} |j|| | | | d \} } } } | |d <| |d<| |d <| |d<n|dd}|dus Jdt|trDt |}t |t j }|j dd} |dd}|rt|jfi|t| |j|j\}}| |d<g|d<g|d<g|d<g|d<g|d<g|d<g|d<g|d<t||D]\}}|\}}}}|||||ddf}|j dd}||t|ddddf}|j||}|j||dd\} } }}|d|j|d|j|d|j|d||d||d| |d| q|S)zThe Preprocess For Task Args: inputs (dict): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. rAimageNfeatures original_size input_sizez*image must be set when image not in inputspointslabelsboxesmasks)r\ point_coords point_labelsr`rarbrc mask_inputsz5An image must be supported, when 'seg_image' is True.r7FZ image_list features_listoriginal_size_listinput_size_listcrop_boxes_listpoints_for_image_listpoint_coords_listpoint_labels_listr*T)r\rb multi_seg)rQpreprocess_image is_image_setr\r[r]preprocess_promptsr-r.cv2imreadcvtColor COLOR_BGR2RGBshaper7rrHrJzip set_imagenparrayrLappend)r;inputsrS model_inputsrArZ input_imager\r]rbrcr`ra orig_sizer7 crop_boxesZ layer_idxscrop_boxlayer_idx0y0x1y1Z crop_imageZcrop_image_sizeZ points_scalepoints_for_image_r?r?r@ preprocesss                  z"SegmentAnythingPipeline.preprocessc! sB|dd}|dd}|d}|st|j|d|d|d|d |d |d |d |d ||d \}}}|||dS|d } | \} } |d} |d} |d}|d}t}t|dD]D\}}| |}||}| |}|||d|}|d|}t}t|j|||D]j\}}}|j||||||dd\}}}|}|}|}t|d|jd|jdf|d|j |jdddd}~|j dkr|d |j k}| |t |d!|jj |j|d"<|jdkr|d"|jk}| ||d!|jj k|d!<t|d!|d <t|d dd| | g}|sB| |t|d!| | |d!<t|d!|d#<|d!=||q|d }|d }t|||jd$} | | t|d |d <t|d%|d%<tfd&d'tt|d#D|d(<||qt|dkr>|d }d)t|d(}t|||jd$} | | |S)*zThe Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. rOTrPFrArZr[r]r\rbrcr`rd) rZr[r]r\rbrcr`rdrOrP)ra low_res_masksiou_predictionsrjrkrlrirfrhrg)r[r]r\rbrcrOrPr*)r*r rr+)ra iou_predsr^rrastability_scorerles iou_thresholdr^csg|]}qSr?r?).0rrr?r@ ez3SegmentAnythingPipeline.forward..r?)rQr%r enumeraterrCasnumpyreshaperurepeatrDfilterrZmask_thresholdrFrErrallrrcatrrIrrrxryrangelenr)!r;r|rTrOrPrArarrr~Zorig_hZorig_wrjrkrlridatarr[rbrcrr]r\Z data_cropZpoints_for_image_batchZpoint_coords_batchZpoint_labels_batch data_batch keep_mask input_boxes input_scores keep_by_nmsr?rr@forwards             (  zSegmentAnythingPipeline.forwardc Ksr|d}|sh|djd}|dkrF|D]\}}|d||<q*n|D]\}}|||<qN|S|jdkr|||jt|j|j}|jdkrdd|dD|d <n.|jd krd d|dD|d <n |d|d <g}t t |d D]|}|d |t |d|t |d | |d ||d| g|d|t |d| d} || q|S)zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. rArarr coco_rlecSsg|] }t|qSr?)rrrler?r?r@rrz7SegmentAnythingPipeline.postprocess..rZ segmentations binary_maskcSsg|] }t|qSr?)rrr?r?r@rrr`rr^rr) segmentationareabboxZ predicted_iourbrr)ruitemsrrMpostprocess_small_regionsmaxrGrIrNrrrrtolistrVrz) r; model_outputsrUrAbskvZ curr_annsidxannr?r?r@ postprocessss@        z#SegmentAnythingPipeline.postprocessRGB) image_formatreturncCs~t|tr"t|}t|tj}|dvs:Jd|d||jjkrX|ddddf}|jdd}| |\}}|||fS)a Calculates the image embeddings for the provided image, allowing masks to be predicted with the 'predict' method. Arguments: image (np.ndarray): The image for calculating masks. Expects an image in HWC uint8 format, with pixel values in [0, 255]. image_format (str): The color format of the image, in ['RGB', 'BGR']. )rZBGRz+image_format must be in ['RGB', 'BGR'], is r'.Nr*re) r-r.rqrrrsrtnetworkrrur&)r;rZrr\r}r]r?r?r@rns     z(SegmentAnythingPipeline.preprocess_imageFcCs|dur|j}|dkr&|dddf}|dkr|durLtj|jdtjd}|jj||}tj|dd}tj|dd}|r| ddd}| }nJ|dkr|durtj|jd|jdftjd}|jj ||}nt dt |t jd}t |t jd}|durf|j}|dkr4|jj||}tj|dd}n$|dkrP|jj||}nt d t |t jd}|durtj|dd}t |t jd}||||fS) zpreprocess promptsNr rer)dtyper+zpoints's ndim < 3.zboxes's ndim must be 1 or 2.)ndimrxonesruint32r& transformZ apply_coords expand_dims transposeZapply_coords_batchr3rmsfloat32Z apply_boxesZapply_boxes_batch)r;r\rbrcr`rarmrr?r?r@rpsB      z*SegmentAnythingPipeline.preprocess_prompts)rcCs>||||\}}}|j||_||_||_d|_dS)NT)r:rnrZ image_encoderr[r\r]ro)r;rZrr}r\r]r?r?r@rws z!SegmentAnythingPipeline.set_image)rcCsd|_d|_d|_d|_dS)zResets the currently set image.FN)ror[r\r]r;r?r?r@r:sz#SegmentAnythingPipeline.reset_imagecCs"|jstd|jdusJ|jS)z Returns the image embeddings for the currently set image, with shape 1xCxHxW, where C is the embedding dimension and (H,W) are the embedding spatial dimension of SAM (typically C=256, H=W=64). zCAn image must be set with .set_image(...) to generate an embedding.N)ro RuntimeErrorr[rr?r?r@get_image_embeddings z+SegmentAnythingPipeline.get_image_embedding @)\(?ffffff?rffffff?r?r rcKsJ|p|dd|_|p |dd|_|p2|dd|_|pD|dd|_|pV|dd |_|ph|d d |_|pz|d d |_|p|dd |_| p|dd|_ | p|dd|_ | p|dd|_ | p|dd |_ | p|dd|_ |jdu|j duksJd|jdur(t|j|j|j |_ |j dvsFJd|j ddS)z reset configrBrrCrrrErrFrrGrrHrrIrJrrKr rLNrMrNrz>Exactly one of points_per_side or point_grid must be provided.)rZuncompressed_rlerzUnknown output_mode r')rQrBrCrDrErFrGrHrIrJrKrLrMrNr)r;rBrCrDrErFrGrHrIrJrKrLrMrNr<r?r?r@r7s:   z$SegmentAnythingPipeline.reset_config) mask_datamin_area nms_threshrcCs|ds |Sg}g}|dD]b}t|}t||dd\}}| } t||dd\}}| oZ| } |tj|dd|t| qtj|dd} t| } | tj } t |} t | | |d}|D]F}||dkrtj| |dd}t |d|d|<| ||d |<q| ||S) z Removes small disconnected regions and holes in masks, then reruns box NMS to remove any new duplicates. Edits mask_data in place. Requires open-cv as a dependency. rZholesr(Zislandsrr+rrr`)rrrzrxrfloat concatenaterastyperryrrr)r;rrrZ new_masksscoresrmaskchanged unchangedrar`rrrZi_mask mask_tensorr?r?r@rCs0       z1SegmentAnythingPipeline.postprocess_small_regions)N)r)NNNNF)r) rrrrrrrrrr Nrr)__name__ __module__ __qualname____doc__r get_pipeline_support_task_listkeysr/rr.r rrr r5rYrrrrnrprwr:rr7rintrr __classcell__r?r?r=r@r"4s\&ax9  1  0)0rtypingrrrqnumpyrx mindsporerrrrmindformers.auto_classrrmindformers.mindformer_bookr mindformers.modelsr r mindformers.tools.registerr r Zmindformers.models.samrrrrrrrrrrrrrrrrrr base_pipeliner!__all__registerPIPELINEr"r?r?r?r@s  P text_classification_pipeline.cpython-39.pyc000066400000000000000000000212721476650141400347560ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cf(@sdZddlZddlZddlmZmZmZddl m Z m Z ddl m Z ddlmZdd lmZmZdd lmZmZdd lmZdgZejejd d GdddeZdS)TextClassificationPipelineN)opsTensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizer)labelstext_classification)aliascsbeZdZdZedZdfdd ZddZ dd Z d d Z d d Z ddZ ddZZS)raPipeline for text classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TextClassificationPipeline >>> from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') >>> txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') >>> model = BertForMultipleChoice(txtcls_mnli_config) >>> txtcls_pipeline = TextClassificationPipeline(task='text_classification', ... model=model, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = txtcls_pipeline(input_data, top_k=1) >>> print(results) [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] rNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__rrsuper__init__)selfmodelrkwargsr\/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/text_classification_pipeline.pyr"@s"      z#TextClassificationPipeline.__init__c sd|vrtd|jjdi}gd}i}|D]}||vr0||||<q0gdfdd|D}d|vr|d|d<d |vr|d |d <t|d std |d d t|||fS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_size)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)r*r+r,r-r.r/paddingcsi|]\}}|vr||qSr'r').0kvpreprocess_key_namer'r( fzCTextClassificationPipeline._sanitize_parameters..r*datasetzThe dataset does not support z, but only support )rrr getitemsrkeys)r#pipeline_parameterspostprocess_paramsforward_key_nameforward_kwargsitempreprocess_paramsr'r4r(_sanitize_parametersVs&z/TextClassificationPipeline._sanitize_parametersc Cst|d}t|d}t|d}dd|D}dd|D}dgt|dgt|d}|dg|t|}||dd}t|}|dg|t|}dg|dg||} || |fS)a  process of two sentences relationship classification Args: inputs_zero (str): the first sentence inputs_one (str): the second sentence Return: processed inputs, mask, token_type about two sentences input_idscSsg|]}|dkr|qSrr'r1xr'r'r( r7z=TextClassificationPipeline.inputs_process..cSsg|]}|dkr|qSrDr'rEr'r'r(rGr7rr N)lenlistasnumpy) r# inputs_zero inputs_oneZ len_inputsZinputs_zero_inputZinputs_one_input token_typeinputsZlen_inputs_maskmaskr'r'r(inputs_processqs  z)TextClassificationPipeline.inputs_processcKsht|tstdt}|jjdkrb|j|fddi|}|d}|d}||d||ddSd |vrrtd |d }| d d }| d d }|j|df||dd|} |j|df||dd|} | | | \}} } i} t t j|t jd| d<t t j| t jd| d<t t j| t jd| d<t}|| dd|| dd|| ddddS)ae Preprocess of text classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. zInputs type must be str gpt2_txtclsreturn_tensorsmsrCattention_maskr)rCrT-z+two texts of text pair should be split by -r/r0)r/r0rRr )dtypetoken_type_idsN)rC input_mask token_type_id label_ids)rrrr ExpandDimsr$ model_namersplitpoprPr from_numpynparrayint32)r#rNrA expand_dimstokensrCrTr/r0rKrLrOrMZ inputs_finalr'r'r( preprocesssH        z%TextClassificationPipeline.preprocesscKs |dd|jfi|}|S)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. NoneN)r_network)r# model_inputsforward_params output_idsr'r'r(forwards z"TextClassificationPipeline.forwardcCs0tj|ddd}t||}||jdddS)NT)axiskeepdims)ramaxexpsum)r#outputsmaxesZ shifted_expr'r'r(softmaxsz"TextClassificationPipeline.softmaxc s|dd}|dd}|r6ddtt|DnddttdD|d}|}||}fd d t|D}|jd d d d|dur|d|}|gS)z Postprocess Args: model_outputs (dict): outputs of forward process. Return: Classification results r*Nr8cSsi|]\}}||qSr'r'r1idlabelr'r'r(r6r7z:TextClassificationPipeline.postprocess..cSsi|]\}}||qSr'r'rvr'r'r(r6r7mnlircs"g|]\}}||dqS))rxscore)r@)r1irzid2labelr'r(rGsz:TextClassificationPipeline.postprocess..cSs|dS)Nrzr')rFr'r'r(r7z8TextClassificationPipeline.postprocess..T)keyreverse)r_ enumeraterr9rJrusort)r# model_outputsr=r*r8rsscoresZ dict_scoresr'r|r( postprocesss     z&TextClassificationPipeline.postprocess)N)r __module__ __qualname____doc__r get_pipeline_support_task_listr;rr"rBrPrfrlrur __classcell__r'r'r&r(rs-)ros.pathrnumpyra mindsporerrr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsrrdataset.labelsr__all__registerPIPELINErr'r'r'r(s   pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/text_generation_pipeline.cpython-39.pyc000066400000000000000000000116261476650141400341770ustar00rootroot00000000000000a ?cf@sdZddlZddlmZmZddlZddlmZmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdgZd d ZejejddGdddeZdS)TextGenerationPipelineN)UnionOptional)TensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel BaseTokenizercCs&g}|D]}|t|q|S)N)appendr get_model_support_listget)Zsupport_model_list support_listZ support_modelrX/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/text_generation_pipeline.py_setup_support_list srtext_generation)aliascseZdZdZeddgZdZdeee e fe e dfdd Z d d Zeeeefd d d ZedddZedddZZS)raPipeline for Text Generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TextGenerationPipeline >>> text_generate = TextGenerationPipeline("gpt2") >>> output = text_generate("I love Beijing, because ") gpt2glmrN)model tokenizerc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrainedrr r TypeErrortype ValueError __class____name__rrsuper__init__)selfrrkwargsr)rrr,>s"      zTextGenerationPipeline.__init__cKsfddg}i}|D]}||vr||||<qi}gd}i}|D]}||vr@||||<q@|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. keysadd_special_tokens)top_ktop_p do_sample eos_token_idrepetition_penalty max_lengthseed)r)r-pipeline_parameterspreprocess_keyspreprocess_paramsitempostprocess_paramsforward_key_nameforward_kwargsrrr_sanitize_parametersVsz+TextGenerationPipeline._sanitize_parameters)inputscKsv|dd}t|trZ|dd}d}|r6|d|n|}||}t|tjrZ|}|j|d|dd}d|iS) zThe Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. r1Tr0Ntext src_language)return_tensorsr1 input_ids)rrdict mindsporerasnumpytolistr)r-rAr;r1r0default_src_language_name feature_namerErrr preprocessks     z!TextGenerationPipeline.preprocess) model_inputscKs0|dd|d}|jj|fi|}d|iS)zThe Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNrE output_ids)popnetworkgenerate)r-rMforward_paramsrErOrrrforwards zTextGenerationPipeline.forward) model_outputscKs$|jj|ddd}|jd|igS)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. rOT)skip_special_tokens_text)rdecode return_name)r-rUr=outputsrrr postprocesss z"TextGenerationPipeline.postprocess)N)r* __module__ __qualname____doc__rr!rYrr rrrrr,r@rFrrLrTr[ __classcell__rrr/rr's  )r^os.pathr"typingrrrGrr auto_classrr mindformer_bookr base_pipeliner Ztools.registerr rmodelsrr__all__rregisterPIPELINErrrrrs  token_classification_pipeline.cpython-39.pyc000066400000000000000000000205431476650141400351120ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cf'@sdZddlZddlZddlmZmZddlm Z m Z ddl m Z ddl mZdd lmZmZdd lmZmZdgZejejd d Gd ddeZdS)TokenClassificationPipelineN)opsModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel Tokenizertoken_classification)aliascsZeZdZdZedZdfdd ZddZ dd Z d d Z d d Z ddZ ZS)ruPipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TokenClassificationPipeline >>> from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig >>> from mindformers.dataset.labels import cluener_labels >>> id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] >>> tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') >>> ner_dense_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') >>> model = BertForTokenClassification(ner_dense_cluener_config) >>> tokcls_pipeline = TokenClassificationPipeline(task='token_classification', ... model=model, ... id2label=id2label, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = tokcls_pipeline(input_data) >>> print(results) [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] rNc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjd|durt |jjd||_d|_tj||fi|dS) Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.z6 requires for a dict which maps label id to label str.) isinstancestr _support_listospathisdirrfrom_pretrained tokenizerrr TypeErrortype ValueError __class____name__r rid2label input_textsuper__init__)selfmodelr rkwargsr]/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/token_classification_pipeline.pyr#Ds*      z$TokenClassificationPipeline.__init__c s`d|vrtd|jjddhddgfdd|D}fd d|D|ifS) z=sanitize parameters for preprocess, forward, and postprocess. batch_sizezThe z? does not support batch inference, please remove the batch_sizer max_lengthpaddingcsi|]\}}|vr||qSr(r(.0kv)preprocess_key_namer(r) jzDTokenClassificationPipeline._sanitize_parameters..csi|]\}}|vr||qSr(r(r-)postprocess_paramsr(r)r2kr3)rrritems)r$pipeline_parameterspreprocess_paramsr()r4r1r)_sanitize_parametersasz0TokenClassificationPipeline._sanitize_parameterscKs|t|tstd||_|dd}|dd}|j|f||dd|}t}||dd||d d||d dd S) af Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. zInputs type must be strr+r,ms)r+r,return_tensors input_idsrattention_masktoken_type_ids)r< input_maskr>)rrrr!poprr ExpandDims)r$inputsr7r+r, expand_dimsr(r(r) preprocessos       z&TokenClassificationPipeline.preprocesscKs$|jd|jfi|}d|iS)z Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. Flogits)r% set_trainnetwork)r$ model_inputsforward_paramsrEr(r(r)forwards z#TokenClassificationPipeline.forwardc sT|d}tj|ddd}t||}||jddd}tj|dd}tj|dd}fdd|D} d d|D} g} t| | D]\} } g}| }|D]}i}|d |d <|d |d <|d|d<t| |d |dd |dd |d |d<t |dd|d<j |d |dd |d<| |q| |q| S)z Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results rET)axiskeepdimsr)rLcs&g|]}fdd|ddDqS)csg|]}j|qSr()r )r.id_r$r(r) r3zFTokenClassificationPipeline.postprocess...r rKr()r.pred_idsrOr(r)rPr3z;TokenClassificationPipeline.postprocess..cSsg|]}|ddqS)r rKr()r. best_scoresr(r(r)rPr3rZ entity_groupr startendscoreword) asnumpynpmaxexpsumargmaxtolistzipget_entities_biosroundr!append)r$ model_outputsr4rEmaxes shifted_expprobsbatch_pred_idsZbatch_best_scores pred_pathsrRZ total_resultZ pred_path best_scoreZ single_result pred_entities pred_entityZ entity_resultr(rOr) postprocesss2       z'TokenClassificationPipeline.postprocesscCs>g}gd}t|D]"\}}|drz|ddkr>||gd}||d<||d<|dd|d<||gd}|dr|ddkr||gd}||d<|dd|d<q|d r|ddkr|dd}||dkr||d<|t|dkr8||q|ddkr0||gd}q|S) a_Gets entities from sequence. note: BIOS Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chunk_start, chunk_end). Example: # >>> seq = ['B-PER', 'I-PER', 'O', 'S-LOC'] # >>> get_entity_bios(seq) [['PER', 0, 1], ['LOC', 3, 3]] )rKrKrKzS-rrKr -rzB-zI-) enumerate startswithrbsplitlen)r$seqchunkschunkindxtag entity_typer(r(r)r`s8            z-TokenClassificationPipeline.get_entities_bios)N)r __module__ __qualname____doc__rget_pipeline_support_task_listkeysrr#r8rDrJrlr` __classcell__r(r(r'r)rs!()rzos.pathrnumpyrY mindsporerr auto_classrrmindformer_bookr base_pipeliner Ztools.registerr r modelsr r__all__registerPIPELINErr(r(r(r)s  pengcheng.mind-mindformer/mindformers/pipeline/__pycache__/translation_pipeline.cpython-39.pyc000066400000000000000000000111011476650141400333220ustar00rootroot00000000000000a ?cf|@sdZddlZddlmZmZddlZddlmZmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdgZejejd d GdddeZdS)TranslationPipelineN)UnionOptional)TensorModel) AutoProcessor AutoModel)MindFormerBook) BasePipeline)MindFormerRegisterMindFormerModuleType) BaseModel BaseTokenizer translation)aliascseZdZdZedZdZdee e e fe e dfdd Zdd Zee eefd d d Zed ddZedddZZS)raPipeline for Translation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TranslationPipeline >>> translator = TranslationPipeline("t5_small") >>> output = translator("abc") t5rN)model tokenizerc st|tr~||jvs tj|r^|dur4t|j}t |}t|t s~t dt |dn t |d|jjd|jdt|ttfst dt |d|durt |jjdtj||fi|dS)Nz:tokenizer should be inherited from BaseTokenizer, but got .z is not supported by z,please selected from z@model should be inherited from BaseModel or Model, but got type z requires for a tokenizer.) isinstancestr _support_listospathisdirrfrom_pretrainedrr r TypeErrortype ValueError __class____name__rrsuper__init__)selfrrkwargsr!T/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/translation_pipeline.pyr$7s"      zTranslationPipeline.__init__cKsddg}i}|D]}||vr||||<qi}gd}i}|D]}||vr>||||<q>|||fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. keys)top_ktop_p do_sample eos_token_idrepetition_penalty max_length)get)r%pipeline_parametersZpreprocess_keyspreprocess_paramsitempostprocess_paramsZforward_key_nameZforward_kwargsr(r(r)_sanitize_parametersOsz(TranslationPipeline._sanitize_parameters)inputscKsht|trN|dd}d}|r*|d|n|}||}t|tjrN|}|j|ddd}d|iS)zThe Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. r*NtextZ src_language)return_tensors input_ids)rdictr1 mindsporerasnumpytolistr)r%r7r3r*Zdefault_src_language_name feature_namer:r(r(r) preprocessds    zTranslationPipeline.preprocess) model_inputscKs0|dd|d}|jj|fi|}d|iS)zThe Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. NoneNr: output_ids)popnetworkgenerate)r%rAforward_paramsr:rCr(r(r)forwardzs zTranslationPipeline.forward) model_outputscKs$|jj|ddd}|jd|igS)zPostprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. rCT)skip_special_tokens_text)rdecode return_name)r%rIr5outputsr(r(r) postprocesss zTranslationPipeline.postprocess)N)r" __module__ __qualname____doc__r get_model_support_listrrMrrrrrrr$r6r;rr@rHrO __classcell__r(r(r'r)r s  )rRos.pathrtypingrrr<rr auto_classrr mindformer_bookr base_pipeliner Ztools.registerr rmodelsrr__all__registerPIPELINErr(r(r(r)s  zero_shot_image_classification_pipeline.cpython-39.pyc000066400000000000000000000167221476650141400371540ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/pipeline/__pycache__a ?cf#@sdZddlmZmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZddlmZmZdd lmZmZd d lmZd d lmZejejddGdddeZdS)#ZeroShotImageClassificationPipeline)UnionOptional)Model) operations)MindFormerBook) load_image) BaseModelBaseImageProcessor Tokenizer)MindFormerRegisterMindFormerModuleType) AutoProcessor AutoModel) BasePipeline) BaseTokenizerzero_shot_image_classification)aliascseZdZdZedZdee e e fe e e edfdd ZddZed d d Zed d dZedddZZS)raPipeline For Zero Shot Image Classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer for text processing. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model, tokenizer, and image_processor's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.tools.image_tools import load_image >>> from mindformers.pipeline import ZeroShotImageClassificationPipeline >>> classifier = ZeroShotImageClassificationPipeline( ... model='clip_vit_b_32', ... candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], ... hypothesis_template="This is a photo of {}." ... ) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] rN)model tokenizerimage_processorc st|tr||jvr|dur(t|j}t|tsFtdt|d|durZt|j }t|t sxtdt|dt |}nt |d|jdt|t tfstdt|d|durt d|durt dtj|||fi|dS)NzEimage_processor should be inherited from BaseImageProcessor, but got .z@tokenizer should be inherited from PretrainedTokenizer, but got zN is not supported by ZeroShotImageClassificationPipeline,please selected from z@model should be inherited from BaseModel or Model, but got type z=ZeroShotImageClassificationPipeline requires for a tokenizer.zCZeroShotImageClassificationPipeline requires for a image_processor.) isinstancestr _support_listrfrom_pretrainedrr TypeErrortyperr r ValueErrorr rsuper__init__)selfrrrkwargs __class__g/home/ma-user/work/mindformers-baichuan/mindformers/pipeline/zero_shot_image_classification_pipeline.pyr"Bs2          z,ZeroShotImageClassificationPipeline.__init__cKsPi}i}gd}|D]}||vr||||<qd|vrF|d|d<|i|fS)z~Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. )candidate_labelshypothesis_template max_lengthpaddingreturn_tensorstop_k)get)r#pipeline_parameterspreprocess_paramspostprocess_paramsZpre_listitemr'r'r(_sanitize_parameterscsz8ZeroShotImageClassificationPipeline._sanitize_parameters)inputsc  s|dd}|dd|dd}|dd}|dd }|durLtd dur\td t|trn|d }t|trt|}||}fd d|D}|j||||dd} || |dS)aPreprocess of ZeroShotImageClassificationPipeline Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. candidate_labels (List[str]): The candidate labels for classification. hypothesis_template (Optional[str]): Prompt for text input. max_length (Optional[int]): Max length of tokenizer's output padding (Optional[Union[False, "max_length"]]): Padding for max_length return_tensors (Optional["ms"]): The type of returned tensors Return: Processed data. Raises: ValueError: If candidate_labels or hypothesis_template is None. r)Nr*za picture of {}.r+Mr,r-mszTcandidate_labels are supposed for ZeroShotImageClassificationPipeline, but got None.zVhypothesis_template is supposed for ZeroShotImageClassificationPipeline, but got None.imagecsg|]}|qSr')format).0Zcandidate_labelr*r'r( szBZeroShotImageClassificationPipeline.preprocess..)r+r,r- input_ids)image_processedr=r))popr rdictrrrr) r#r5r1r)r+r,r-r> sentencesr=r'r;r( preprocessvs6         z.ZeroShotImageClassificationPipeline.preprocess) model_inputscKsJ|dd|d}|d}|||\}}t|}||ddS)zForward process Args: model_inputs (dict): Outputs of preprocess. Return: Probs dict. NoneNr>r=r))probsr))r?networkPSoftmaxasnumpy)r#rCforward_paramsr>r=logits_per_image_rEr'r'r(forwards z+ZeroShotImageClassificationPipeline.forward) model_outputsc Ksv|dd}|d}|d}g}|D]L}tt||ddd}|dur\|dt|t|}|dd |Dq$|S) zPostprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result Return: Classification results. r.Nr)rEcSs |d S)Nrr')xr'r'r(zAZeroShotImageClassificationPipeline.postprocess..)keycSsg|]\}}||dqS))scorelabelr')r: score_itemrTr'r'r(r<szCZeroShotImageClassificationPipeline.postprocess..)r?sortedzipminlenappend) r#rNr2r.labelsscoresoutputsrS sorted_resr'r'r( postprocesss   z/ZeroShotImageClassificationPipeline.postprocess)NN)__name__ __module__ __qualname____doc__rget_pipeline_support_task_listkeysrrrr rrrr r"r4r@rBrMr_ __classcell__r'r'r%r(rs!+N)rctypingrr mindsporer mindspore.opsrrGmindformers.mindformer_bookrmindformers.tools.image_toolsrmindformers.modelsr r r mindformers.tools.registerr r mindformers.auto_classrr base_pipelinermodelsrregisterPIPELINErr'r'r'r(s      pengcheng.mind-mindformer/mindformers/pipeline/base_pipeline.py000066400000000000000000000220631476650141400254200ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/base.py # ============================================================================ """BasePipeline""" from abc import ABC, abstractmethod from typing import Optional, Union import numpy as np from tqdm import tqdm from mindspore import Tensor, Model from mindspore.dataset import ( GeneratorDataset, VisionBaseDataset, SourceDataset, MappableDataset ) from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.tools import logger from mindformers.mindformer_book import print_dict from ..auto_class import AutoModel from ..models import BaseModel, BaseTokenizer, BaseImageProcessor class BasePipeline(ABC): r""" Base Pipeline For All Task Pipelines Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): The tokenizer of model, it could be None if the model do not need tokenizer. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. """ _support_list = {} def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): super(BasePipeline, self).__init__() self.model = model if isinstance(model, str) and model in self._support_list: self.network = AutoModel.from_pretrained(model) elif isinstance(model, BaseModel): self.network = model elif isinstance(model, Model): self.network = model.predict_network else: raise TypeError(f"model should be str or inherited from BaseModel or Model, but got type {type(model)}.") self.tokenizer = tokenizer self.image_processor = image_processor self._preprocess_params, self._forward_params, \ self._postprocess_params = self._sanitize_parameters(**kwargs) self.call_count = 0 self._batch_size = kwargs.pop("batch_size", None) def __call__(self, inputs: Union[GeneratorDataset, list, str, Tensor, np.array], batch_size: Optional[int] = None, **kwargs): r"""Call Method Args: inputs (Union[GeneratorDataset, list, str, etc]): The inputs of pipeline, the type of inputs depends on task. batch_size (Optional[int]): The batch size for a GeneratorDataset input, for other types of inputs, the batch size would be set to 1 by default. """ preprocess_params, forward_params, postprocess_params = self._sanitize_parameters(**kwargs) preprocess_params = {**self._preprocess_params, **preprocess_params} forward_params = {**self._forward_params, **forward_params} postprocess_params = {**self._postprocess_params, **postprocess_params} is_dataset = isinstance(inputs, ( GeneratorDataset, VisionBaseDataset, MappableDataset, SourceDataset)) is_list = isinstance(inputs, list) self.call_count += 1 if self.call_count == 20 and not is_dataset: logger.info("You seem to be using the pipeline sequentially for" " numerous samples. In order to maximize efficiency" " please set input a mindspore.dataset.GeneratorDataset.") if batch_size is None: if self._batch_size is None: batch_size = 1 else: batch_size = self._batch_size if batch_size > 1 and not is_dataset: batch_size = 1 logger.info("batch_size is set to 1 for non-dataset inputs.") if is_dataset: logger.info("dataset is processing.") if not isinstance(inputs, (BatchDataset, RepeatDataset)): inputs = inputs.batch(batch_size) outputs = [] for items in tqdm(inputs.create_dict_iterator()): outputs.extend(self.run_single(items, preprocess_params, forward_params, postprocess_params)) elif is_list: outputs = self.run_multi(inputs, preprocess_params, forward_params, postprocess_params) else: outputs = self.run_single(inputs, preprocess_params, forward_params, postprocess_params) return outputs @abstractmethod def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("_sanitize_parameters not implemented") def run_single(self, inputs: Union[dict, str, np.array, Tensor], preprocess_params: dict, forward_params: dict, postprocess_params: dict): r"""Run Single method This function is used to run a single forward process for task. Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. """ model_inputs = self.preprocess(inputs, **preprocess_params) model_outputs = self.forward(model_inputs, **forward_params) outputs = self.postprocess(model_outputs, **postprocess_params) return outputs def run_multi(self, inputs: Union[list, tuple], preprocess_params: dict, forward_params: dict, postprocess_params: dict): r"""Run Multiple Method This function is used to run a list input for task. Args: inputs (Union[list, tuple, iterator]): The iterable input for pipeline. preprocess_params (dict): The parameter dict for preprocess. forward_params (dict): The parameter dict for model forward process. postprocess_params (dict): The parameter dict for postprocess. """ outputs = [] for item in inputs: outputs.extend(self.run_single(item, preprocess_params, forward_params, postprocess_params)) return outputs @abstractmethod def preprocess(self, inputs: Union[dict, str, np.array, Tensor], **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[dict, str, etc]): The inputs of pipeline, the type of inputs depends on task. preprocess_params (dict): The parameter dict for preprocess. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("preprocess not implemented.") @abstractmethod def forward(self, model_inputs: Union[dict, str, np.array, Tensor], **forward_params): r"""The Forward Process of Model Args: model_inputs (Union[dict, str, etc]): The output of preprocess, the type of model_inputs depends on task. forward_params (dict): The parameter dict for model forward. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("forward not implemented.") @abstractmethod def postprocess(self, model_outputs: Union[dict, str, np.array, Tensor], **postprocess_params): r"""The Postprocess of Task Args: model_outputs (Union[dict, str, etc]): The output of model forward, the type of model_outputs depends on task. postprocess_params (dict): The parameter dict for post process. Raises: NotImplementedError: If the method is not implemented. """ raise NotImplementedError("postprocess not implemented.") @classmethod def show_support_list(cls): """show_support_list""" logger.info("support list of %s is:", cls.__name__) print_dict(cls._support_list) pengcheng.mind-mindformer/mindformers/pipeline/build_pipeline.py000066400000000000000000000044511476650141400256060ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Pipeline API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig def build_pipeline( config: dict = None, default_args: dict = None, module_type: str = 'pipeline', class_name: str = None, **kwargs): r"""Build Pipeline API. Instantiate the task pipeline from MindFormerRegister's registry. Args: config (dict): The task pipeline's config. default_args (dict): The default args of pipeline. module_type (str): The module type of MindFormerModuleType. Default: 'pipline'. class_name (str): The class name of task pipeline. Returns: The task pipeline instance by config. Examples: >>> from mindformers import build_pipeline >>> pipeline_config = {'type': 'zero_shot_image_classification', 'model': 'clip_vit_b_32', 'candidate_labels': ["sunflower", "tree", "dog", "cat", "toy"], 'hypothesis_template': "This is a photo of {}."} >>> classifier = build_pipeline(pipeline_config) >>> type(classifier) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.PIPELINE, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) pengcheng.mind-mindformer/mindformers/pipeline/fill_mask_pipeline.py000066400000000000000000000140561476650141400264520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TranslationPipeline""" import os.path # from tkinter import _Padding import numpy as np import mindspore from mindspore import ops, Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['FillMaskPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="fill_mask") class FillMaskPipeline(BasePipeline): """ Pipeline for mask fill Args: model: a pretrained model (str or BaseModel) in _supproted_list. tokenizer : a tokenizer (None or Tokenizer) for text processing """ _support_list = MindFormerBook.get_model_support_list()['bert'] return_name = 'fillmask' def __init__(self, model, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" PretrainedTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") self.input_text = "" super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) preprocess_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of mask fill Args: inputs (url, PIL.Image, tensor, numpy): the image to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed image. """ if isinstance(inputs, dict): inputs = inputs['text'] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() self.input_text = inputs if isinstance(inputs, str) else "" max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs = self.tokenizer(inputs, max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs["input_ids"], 0), "input_mask": expand_dims(inputs["attention_mask"], 0), "token_type_id": expand_dims(inputs["token_type_ids"], 0), "masked_lm_positions": expand_dims(Tensor(self.tokenizer.mask_index), 0)} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ forward_params.pop("None", None) output_ids = self.network(**model_inputs) return {"output_ids": output_ids} def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ outputs = model_outputs["output_ids"][-2].asnumpy() tokens_dict = [] max_tokens = np.argmax(outputs, axis=1) for ind, tokenid in enumerate(max_tokens): token = self.tokenizer.decode([int(tokenid),], skip_special_tokens=True) token = token.replace(' ', '') tokens_dict.append({'score': outputs[ind, tokenid], 'token': tokenid, 'token_str': token}) self.input_text = self.input_text.replace('[MASK]', token, 1) tokens_dict.append({'sequence': self.input_text}) return [tokens_dict,] pengcheng.mind-mindformer/mindformers/pipeline/image_classification_pipeline.py000066400000000000000000000162301476650141400306420ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import numpy as np from PIL import Image from mindspore.ops import operations as P from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.dataset.labels import labels from .base_pipeline import BasePipeline @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="image_classification") class ImageClassificationPipeline(BasePipeline): r"""Pipeline for image classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import ImageClassificationPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['image_classification'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageClassificationFoPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} post_list = ["top_k", "candidate_labels"] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, {}, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) return {"image_processed": image_processed} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] logits_per_image = self.network(image_processed)[0] probs = P.Softmax()(logits_per_image).asnumpy() return {"probs": probs} def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. """ top_k = postprocess_params.pop("top_k", 3) candidate_labels = postprocess_params.pop("candidate_labels", 'imagenet') scores = model_outputs['probs'] outputs = [] if isinstance(candidate_labels, str): inputs_labels = labels.get(candidate_labels) elif isinstance(candidate_labels, list): inputs_labels = candidate_labels else: raise ValueError(f"The candidate_labels should be dataset name (str) or custom labels (list)" f" but got {type(candidate_labels)}") if inputs_labels is None: raise ValueError(f"The custom candidate_labels is None or " f"the input dataset labels name is not supported yet.") for score in scores: sorted_res = sorted(zip(score, inputs_labels), key=lambda x: -x[0]) if top_k is not None: sorted_res = sorted_res[:min(top_k, len(inputs_labels))] outputs.append([{"score": score_item, "label": label} for score_item, label in sorted_res]) return outputs pengcheng.mind-mindformer/mindformers/pipeline/image_to_text_generation_pipeline.py000066400000000000000000000155311476650141400315530ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import numpy as np from PIL import Image from mindspore import Tensor, Model from mindspore.ops import operations as P from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .base_pipeline import BasePipeline __all__ = ['ImageToTextGenerationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="image_to_text_generation") class ImageToTextGenerationPipeline(BasePipeline): r"""Pipeline for image to text generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. """ _support_list = MindFormerBook.get_pipeline_support_task_list()['image_to_text_generation'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageToTextGenerationPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageToTextGenerationPipeline" " requires for a image_processor.") self.hypothesis_template = kwargs.pop("hypothesis_template", "{}") super().__init__(model.set_train(mode=False), image_processor=image_processor, tokenizer=tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} forward_params = {} post_list = ["top_k"] pre_list = ["hypothesis_template", "max_length", "padding"] forward_list = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'seed'] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) for item in pre_list: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) for item in forward_list: if item in pipeline_parameters: forward_params[item] = pipeline_parameters.get(item) return preprocess_params, forward_params, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) if isinstance(inputs, list): inputs = [load_image(item) for item in inputs] image_processed = self.image_processor(inputs) max_length = preprocess_params.pop("max_length", 32) padding = preprocess_params.pop("padding", "max_length") hypothesis_template = preprocess_params.pop("hypothesis_template", None) if hypothesis_template is not None: self.hypothesis_template = hypothesis_template prompt_list = [self.hypothesis_template] * image_processed.shape[0] prompt_processed = self.tokenizer(prompt_list, max_length=max_length, padding=padding, return_tensors="ms", add_special_tokens=False, **preprocess_params) prompt_input_ids = prompt_processed["input_ids"] if len(prompt_input_ids.shape) == 1: prompt_input_ids = P.ExpandDims()(prompt_input_ids, 0) return {"image_processed": image_processed, "prompt_input_ids": prompt_input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ image_processed = model_inputs["image_processed"] prompt_input_ids = model_inputs["prompt_input_ids"] output_ids_per_image = self.network.generate_text_for_image(image_processed, prompt_input_ids) return {"output_ids": output_ids_per_image} def postprocess(self, model_outputs, **postprocess_params): output_ids = model_outputs["output_ids"] outputs = self.tokenizer.decode(output_ids, skip_special_tokens=True) return outputs pengcheng.mind-mindformer/mindformers/pipeline/masked_image_modeling_pipeline.py000066400000000000000000000145651476650141400310020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Pipeline API.""" import os from typing import Optional, Union import numpy as np from PIL import Image from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .base_pipeline import BasePipeline __all__ = ['MaskedImageModelingPipeline'] from ..tools.utils import LOCAL_DEFAULT_PATH @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="masked_image_modeling") class MaskedImageModelingPipeline(BasePipeline): r"""Pipeline for masked image modeling Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import MaskedImageModelingPipeline >>> from mindformers import ViTMAEImageProcessor >>> processor = ViTMAEImageProcessor(size=224) >>> reconstructor = MaskedImageModelingPipeline( ... model='mae_vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> reconstructor(np.uint8(np.random.random((5, 3, 255, 255)))) """ _support_list = MindFormerBook.get_pipeline_support_task_list()['masked_image_modeling'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("MaskedImageModelingPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} post_list = [] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, {}, postprocess_params def preprocess(self, inputs: (Union[str, Image.Image, Tensor, np.ndarray]), **preprocess_params): r"""The Preprocess For Task Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be reconstructed. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) return {"image_processed": image_processed} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] images_array = self.network(*image_processed) images_array = images_array.asnumpy() return {"images_array": images_array} def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: classification results. """ images_array = model_outputs['images_array'] outputs = [] for i, image_array in enumerate(images_array): image_array = image_array.transpose((1, 2, 0)) mean = np.array(self.image_processor.normalize.mean) std = np.array(self.image_processor.normalize.std) image_array = image_array * std + mean image_array = image_array * 255 image_array = image_array.astype(np.uint8) reconstruct_image = Image.fromarray(image_array) image_path = os.path.join(LOCAL_DEFAULT_PATH, f'output_image{i}.jpg') reconstruct_image.save(image_path) outputs.append({"info": image_path, "data": reconstruct_image}) return outputs pengcheng.mind-mindformer/mindformers/pipeline/pipeline.py000066400000000000000000000122351476650141400244260ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ pipeline """ from enum import Enum from typing import Optional, Union, Tuple from mindspore import Model from mindformers.models import build_model, build_tokenizer, build_processor, \ BaseModel, BaseTokenizer, BaseImageProcessor, BaseAudioProcessor from mindformers.mindformer_book import MindFormerBook from mindformers.tools.register import MindFormerConfig from .build_pipeline import build_pipeline SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() class Backend(Enum): MS = "ms" MS_LITE = "mslite" def pipeline( task: str = None, model: Optional[Union[str, BaseModel, Model, Tuple[str, str]]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, backend: str = "ms", **kwargs): r"""Pipeline for downstream tasks Args: task (str): The supported task could be selected from MindFormerBook.show_pipeline_support_task_list(). model (Optional[Union[str, BaseModel]]): The model used for task. tokenizer (Optional[BaseTokenizer]): The tokenizer of the model. image_processor (Optional[BaseImageProcessor]): The image processor of the model. audio_processor (Optional[BaseAudioProcessor]): The audio processor of the model. backend(str): The inference backend. Default "ms", now support ["ms", "mslite"]. Return: A task pipeline. Raises: KeyError: If the task or model is not supported. Examples: >>> from mindformers import pipeline >>> from mindformers.tools.image_tools import load_image >>> classifier = pipeline("zero_shot_image_classification", candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] """ if backend == Backend.MS_LITE.value: from mindformers.inference import get_mslite_pipeline task_pipeline = get_mslite_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs) elif backend == Backend.MS.value: task_pipeline = get_ms_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs) else: raise ValueError(f"The inference backend \"{backend}\" is not supported," f"please select a backend from [\"ms\", \"mslite\"]") return task_pipeline def get_ms_pipeline(task, model, tokenizer, image_processor, audio_processor, **kwargs): """get mindspore infer pipeline.""" if task not in SUPPORT_PIPELINES.keys(): raise KeyError(f"{task} is not supported by pipeline. please select" f" a task from {SUPPORT_PIPELINES.keys()}.") if isinstance(model, str): if model not in SUPPORT_MODEL_NAMES: raise KeyError( f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}.") model_name = model model = None else: model_name = "common" pipeline_config = MindFormerConfig(SUPPORT_PIPELINES.get(task).get(model_name)) pipeline_config.model.model_config.update({"checkpoint_name_or_path": model_name}) if model is None: model = build_model(pipeline_config.model) if image_processor is None and hasattr(pipeline_config.processor, 'image_processor'): image_processor = build_processor(pipeline_config.processor.image_processor) if audio_processor is None and hasattr(pipeline_config.processor, 'audio_processor'): audio_processor = build_processor(pipeline_config.processor.audio_processor) if tokenizer is None: tokenizer = build_tokenizer(pipeline_config.processor.tokenizer) task_pipeline = build_pipeline(class_name=task, model=model, image_processor=image_processor, audio_processor=audio_processor, tokenizer=tokenizer, **kwargs) return task_pipeline pengcheng.mind-mindformer/mindformers/pipeline/question_answering_pipeline.py000066400000000000000000000440561476650141400304400ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """QuestionAnsweringPipeline""" import os.path import collections import math import six from mindspore import Tensor, Model from mindspore.common import dtype as mstype from mindformers.models import BasicTokenizer from ..dataset.dataloader.squad_dataloader import convert_examples_to_features, SquadExample from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['QuestionAnsweringPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="question_answering") class QuestionAnsweringPipeline(BasePipeline): r"""Pipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import QuestionAnsweringPipeline >>> from mindformers import AutoTokenizer, BertForQuestionAnswering, AutoConfig >>> input_data = ["My name is Wolfgang and I live in Berlin - Where do I live?"] >>> tokenizer = AutoTokenizer.from_pretrained('qa_bert_base_uncased_squad') >>> qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') >>> model = BertForQuestionAnswering(qa_squad_config) >>> qa_pipeline = QuestionAnsweringPipeline(task='question_answering', ... model=model, ... tokenizer=tokenizer) >>> results = qa_pipeline(input_data) >>> print(results) [{'text': 'Berlin', 'score': 0.9941, 'start': 34, 'end': 40}] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['question_answering'].keys() def __init__(self, model, tokenizer, doc_stride=128, max_question_len=64, max_seq_len=384, top_k=1, n_best_size=20, max_answer_len=30, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") self.features = None self.examples = None self.basic_tokenizer = BasicTokenizer(do_lower_case=True) if doc_stride > max_seq_len: raise ValueError(f"`doc_stride` ({doc_stride}) is larger than `max_seq_len` ({max_seq_len})") if top_k < 1: raise ValueError(f"top_k parameter should be >= 1 (got {top_k})") if max_answer_len < 1: raise ValueError(f"max_answer_len parameter should be >= 1 (got {max_answer_len}") kwargs["doc_stride"] = doc_stride kwargs["max_question_len"] = max_question_len kwargs["max_seq_len"] = max_seq_len kwargs["top_k"] = top_k kwargs["n_best_size"] = n_best_size kwargs["max_answer_len"] = max_answer_len super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") preprocess_key_name = ['doc_stride', 'max_question_len', 'max_seq_len'] postprocess_key_name = ['top_k', 'n_best_size', 'max_answer_len'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} postprocess_params = {k: v for k, v in pipeline_parameters.items() if k in postprocess_key_name} return preprocess_params, {}, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("The type of inputs should be str.") if '-' not in inputs: raise ValueError("The inputs should contain context and question separeated by '-' syntax.") context_text, question_text = inputs.split("-") context_text = context_text.strip() question_text = question_text.strip() squad_example = SquadExample(None, question_text, context_text, None, None, None) features = convert_examples_to_features( examples=[squad_example], tokenizer=self.tokenizer, max_seq_len=preprocess_params['max_seq_len'], max_question_len=preprocess_params['max_question_len'], doc_stride=preprocess_params['doc_stride'], is_training=False ) self.features = features self.examples = [squad_example] return features def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ self.model.set_train(False) model_outputs = [] for feature in model_inputs: model_inputs = {"input_ids": Tensor(feature.input_ids, mstype.int32).expand_dims(0), "input_mask": Tensor(feature.input_mask, mstype.int32).expand_dims(0), "token_type_id": Tensor(feature.token_type_id, mstype.int32).expand_dims(0), "start_position": Tensor(feature.start_position, mstype.int32).expand_dims(0), "end_position": Tensor(feature.end_position, mstype.int32).expand_dims(0), "unique_id": Tensor(feature.unique_id, mstype.int32).expand_dims(0)} ids, start, end = self.network(**model_inputs) RawResult = collections.namedtuple("RawResult", ["unique_id", "start_logits", "end_logits"]) unique_id = int(ids[0]) start_logits = [float(x) for x in start[0]] end_logits = [float(x) for x in end[0]] model_outputs.append(RawResult(unique_id=unique_id, start_logits=start_logits, end_logits=end_logits)) return model_outputs def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ top_k = postprocess_params['top_k'] n_best_size = postprocess_params['n_best_size'] max_answer_len = postprocess_params['max_answer_len'] example_index_to_features = collections.defaultdict(list) for feature in self.features: example_index_to_features[feature.example_index].append(feature) unique_id_to_result = {} for result in model_outputs: unique_id_to_result[result.unique_id] = result for (example_index, example) in enumerate(self.examples): features = example_index_to_features[example_index] prelim_predictions = self._get_prelim_predictions(features, unique_id_to_result, max_answer_len, n_best_size) nbest = self._get_nbest(prelim_predictions, features, example, n_best_size) total_scores = [] best_non_null_entry = None for entry in nbest: total_scores.append(entry.start_logit + entry.end_logit) if not best_non_null_entry: if entry.text: best_non_null_entry = entry probs = self._compute_softmax(total_scores) results = [] for (i, entry) in enumerate(nbest): output = {} output["text"] = entry.text output["score"] = round(probs[i], 4) output["start"] = entry.start output["end"] = entry.end results.append(output) return results[:top_k] def _get_prelim_predictions(self, features, unique_id_to_result, max_answer_len, n_best_size): """get prelim predictions""" _PrelimPrediction = collections.namedtuple( "PrelimPrediction", ["feature_index", "start_index", "end_index", "start_logit", "end_logit"]) prelim_predictions = [] # keep track of the minimum score of null start+end of position 0 for (feature_index, feature) in enumerate(features): if feature.unique_id not in unique_id_to_result: continue result = unique_id_to_result[feature.unique_id] start_indexes = self._get_best_indexes(result.start_logits, n_best_size) end_indexes = self._get_best_indexes(result.end_logits, n_best_size) # if we could have irrelevant answers, get the min score of irrelevant for start_index in start_indexes: for end_index in end_indexes: # We could hypothetically create invalid predictions, e.g., predict # that the start of the span is in the question. We throw out all # invalid predictions. if start_index >= len(feature.tokens): continue if end_index >= len(feature.tokens): continue if start_index not in feature.token_to_orig_map: continue if end_index not in feature.token_to_orig_map: continue if not feature.token_is_max_context.get(start_index, False): continue if end_index < start_index: continue length = end_index - start_index + 1 if length > max_answer_len: continue prelim_predictions.append( _PrelimPrediction( feature_index=feature_index, start_index=start_index, end_index=end_index, start_logit=result.start_logits[start_index], end_logit=result.end_logits[end_index])) prelim_predictions = sorted( prelim_predictions, key=lambda x: (x.start_logit + x.end_logit), reverse=True) return prelim_predictions def _get_nbest(self, prelim_predictions, features, example, n_best_size): """get nbest predictions""" _NbestPrediction = collections.namedtuple( "NbestPrediction", ["text", "start_logit", "end_logit", "start", "end"]) seen_predictions = {} nbest = [] for pred in prelim_predictions: if len(nbest) >= n_best_size: break feature = features[pred.feature_index] if pred.start_index > 0: # this is a non-null prediction tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)] orig_doc_start = feature.token_to_orig_map[pred.start_index] orig_doc_end = feature.token_to_orig_map[pred.end_index] orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)] tok_text = " ".join(tok_tokens) # De-tokenize WordPieces that have been split off. tok_text = tok_text.replace(" ##", "") tok_text = tok_text.replace("##", "") # Clean whitespace tok_text = tok_text.strip() tok_text = " ".join(tok_text.split()) orig_text = " ".join(orig_tokens) final_text = self._get_final_text(tok_text, orig_text) char_start_index, char_end_index = self._get_answer_index(example.context_text, orig_doc_start, orig_doc_end) if final_text in seen_predictions: continue seen_predictions[final_text] = True else: final_text = "" seen_predictions[final_text] = True char_start_index, char_end_index = -1, -1 nbest.append( _NbestPrediction( text=final_text, start_logit=pred.start_logit, end_logit=pred.end_logit, start=char_start_index, end=char_end_index)) # In very rare edge cases we could have no valid predictions. So we # just create a nonce prediction in this case to avoid failure. if not nbest: nbest.append(_NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0)) assert len(nbest) >= 1 return nbest def _get_answer_index(self, context_text, orig_doc_start, orig_doc_end): char_index = 0 for i, word in enumerate(context_text.split(" ")): if i == orig_doc_start: char_start_index = char_index if i == orig_doc_end: char_end_index = char_index + len(word) char_index += len(word) + 1 return char_start_index, char_end_index def _compute_softmax(self, scores): """Compute softmax probability over raw logits.""" if not scores: return [] max_score = None for score in scores: if max_score is None or score > max_score: max_score = score exp_scores = [] total_sum = 0.0 for score in scores: x = math.exp(score - max_score) exp_scores.append(x) total_sum += x probs = [] for score in exp_scores: probs.append(score / total_sum) return probs def _get_final_text(self, pred_text, orig_text): """Project the tokenized prediction back to the original text.""" def _strip_spaces(text): ns_chars = [] ns_to_s_map = collections.OrderedDict() for (i, c) in enumerate(text): if c == " ": continue ns_to_s_map[len(ns_chars)] = i ns_chars.append(c) ns_text = "".join(ns_chars) return (ns_text, ns_to_s_map) tok_text = " ".join(self.basic_tokenizer.tokenize(orig_text)) start_position = tok_text.find(pred_text) if start_position == -1: return orig_text end_position = start_position + len(pred_text) - 1 (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text) (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text) if len(orig_ns_text) != len(tok_ns_text): return orig_text tok_s_to_ns_map = {} for (i, tok_index) in six.iteritems(tok_ns_to_s_map): tok_s_to_ns_map[tok_index] = i orig_start_position = None if start_position in tok_s_to_ns_map: ns_start_position = tok_s_to_ns_map[start_position] if ns_start_position in orig_ns_to_s_map: orig_start_position = orig_ns_to_s_map[ns_start_position] if orig_start_position is None: return orig_text orig_end_position = None if end_position in tok_s_to_ns_map: ns_end_position = tok_s_to_ns_map[end_position] if ns_end_position in orig_ns_to_s_map: orig_end_position = orig_ns_to_s_map[ns_end_position] if orig_end_position is None: return orig_text output_text = orig_text[orig_start_position:(orig_end_position + 1)] return output_text def _get_best_indexes(self, logits, n_best_size): """Get the n-best logits from a list.""" index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True) best_indexes = [] for (i, score) in enumerate(index_and_score): if i >= n_best_size: break best_indexes.append(score[0]) return best_indexes pengcheng.mind-mindformer/mindformers/pipeline/segment_anything_pipeline.py000066400000000000000000000646111476650141400300560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Pipeline API.""" from typing import Optional, Union import cv2 import numpy as np import mindspore as ms from mindspore import ops from mindspore import Tensor, Model from mindformers.auto_class import AutoProcessor, AutoModel from mindformers.mindformer_book import MindFormerBook from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.models.sam import ( MaskData, area_from_rle, batch_iterator, batched_mask_to_box, box_xyxy_to_xywh, build_all_layer_point_grids, calculate_stability_score, coco_encode_rle, generate_crop_boxes, is_box_near_crop_edge, mask_to_rle, remove_small_regions, rle_to_mask, uncrop_boxes_xyxy, uncrop_masks, uncrop_points, box_area, nms ) from .base_pipeline import BasePipeline __all__ = ['SegmentAnythingPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="segment_anything") class SegmentAnythingPipeline(BasePipeline): r"""Pipeline for image segment Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> import numpy as np >>> from mindformers.pipeline import SegmentAnythingPipeline >>> from mindformers import ViTImageProcessor >>> processor = ViTImageProcessor(size=224) >>> classifier = ImageClassificationPipeline( ... model='vit_base_p16', ... image_processor=processor, ... top_k=5 ... ) >>> classifier(np.uint8(np.random.random((5, 3, 255, 255)))) [[{'score': 0.0016654134, 'label': 'matchstick'}, {'score': 0.0015071577, 'label': 'theater curtain'}, {'score': 0.0014839625, 'label': 'ocarina'}, {'score': 0.0014319294, 'label': 'abaya'}, {'score': 0.0014109017, 'label': 'bottlecap'}], ..., {'score': 0.0014109018, 'label': 'bottlecap'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['segment_anything'].keys() def __init__(self, model: Union[str, BaseModel, Model], image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ImageClassificationForPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if image_processor is None: raise ValueError("ImageClassificationFoPipeline" " requires for a image_processor.") super().__init__(model.set_train(mode=False), image_processor=image_processor, **kwargs) self.reset_config(**kwargs) self.stack = ops.Stack(axis=-1) self.reset_image() def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {"seg_image": False} forward_params = {"seg_image": False} postprocess_params = {"seg_image": False} preprocess_list_0 = ["seg_image"] for item in preprocess_list_0: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) preprocess_params['reset_config'] = False preprocess_list_1 = ["points_per_side", "points_per_batch", "pred_iou_thresh", "stability_score_thresh", "stability_score_offset", "box_nms_thresh", "crop_n_layers", "crop_nms_thresh", "crop_overlap_ratio", "crop_n_points_downscale_factor", "point_grids", "min_mask_region_area", "output_mode"] for item in preprocess_list_1: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) preprocess_params['reset_config'] = True forward_list = ["seg_image", "multimask_output", "return_logits"] for item in forward_list: if item in pipeline_parameters: forward_params[item] = pipeline_parameters.get(item) post_list = ["seg_image"] for item in post_list: if item in pipeline_parameters: postprocess_params[item] = pipeline_parameters.get(item) return preprocess_params, forward_params, postprocess_params def preprocess(self, inputs, **preprocess_params): r"""The Preprocess For Task Args: inputs (dict): The image to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed image. """ model_inputs = {} seg_image = preprocess_params["seg_image"] if not seg_image: image = inputs.get("image", None) if image is not None: input_image, original_size, input_size = self.preprocess_image(image) model_inputs["image"] = input_image model_inputs["features"] = None model_inputs["original_size"] = original_size model_inputs["input_size"] = input_size else: assert self.is_image_set, "image must be set when image not in inputs" original_size = self.original_size model_inputs["image"] = None model_inputs["features"] = self.features model_inputs["original_size"] = self.original_size model_inputs["input_size"] = self.input_size point_coords = inputs.get("points", None) point_labels = inputs.get("labels", None) boxes = inputs.get("boxes", None) masks = inputs.get("masks", None) point_coords, point_labels, boxes, masks = self.preprocess_prompts( original_size=original_size, point_coords=point_coords, point_labels=point_labels, boxes=boxes, masks=masks ) model_inputs["point_coords"] = point_coords model_inputs["point_labels"] = point_labels model_inputs["boxes"] = boxes model_inputs["mask_inputs"] = masks else: image = inputs.get("image", None) assert image is not None, "An image must be supported, when 'seg_image' is True." if isinstance(image, str): image = cv2.imread(image) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) orig_size = image.shape[:2] reset_config = preprocess_params.get("reset_config", False) if reset_config: self.reset_config(**preprocess_params) crop_boxes, layer_idxs = generate_crop_boxes( orig_size, self.crop_n_layers, self.crop_overlap_ratio ) model_inputs["original_size"] = orig_size model_inputs["image_list"] = [] model_inputs["features_list"] = [] model_inputs["original_size_list"] = [] model_inputs["input_size_list"] = [] model_inputs["crop_boxes_list"] = [] model_inputs["points_for_image_list"] = [] model_inputs["point_coords_list"] = [] model_inputs["point_labels_list"] = [] for crop_box, layer_id in zip(crop_boxes, layer_idxs): x0, y0, x1, y1 = crop_box crop_image = image[y0:y1, x0:x1, :] crop_image_size = crop_image.shape[:2] self.set_image(crop_image) points_scale = np.array(crop_image_size)[None, ::-1] points_for_image = self.point_grids[layer_id] * points_scale point_coords, point_labels, _, _ = self.preprocess_prompts( original_size=crop_image_size, point_coords=points_for_image, multi_seg=True ) model_inputs["features_list"].append(self.features) model_inputs["original_size_list"].append(self.original_size) model_inputs["input_size_list"].append(self.input_size) model_inputs["crop_boxes_list"].append(crop_box) model_inputs["points_for_image_list"].append(points_for_image) model_inputs["point_coords_list"].append(point_coords) model_inputs["point_labels_list"].append(point_labels) return model_inputs def forward(self, model_inputs, **forward_params): r"""The Forward Process of Model Args: model_inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ multimask_output = forward_params.get("multimask_output", True) return_logits = forward_params.get("return_logits", False) seg_image = forward_params["seg_image"] if not seg_image: masks, low_res_masks, iou_predictions = self.model( image=model_inputs["image"], features=model_inputs["features"], input_size=model_inputs["input_size"], original_size=model_inputs["original_size"], point_coords=model_inputs["point_coords"], point_labels=model_inputs["point_labels"], boxes=model_inputs["boxes"], mask_inputs=model_inputs["mask_inputs"], multimask_output=multimask_output, return_logits=return_logits ) return {"masks": masks, "low_res_masks": low_res_masks, "iou_predictions": iou_predictions} orig_size = model_inputs["original_size"] orig_h, orig_w = orig_size points_for_image_list = model_inputs["points_for_image_list"] point_coords_list = model_inputs["point_coords_list"] point_labels_list = model_inputs["point_labels_list"] crop_boxes_list = model_inputs["crop_boxes_list"] data = MaskData() for layer_id, features in enumerate(model_inputs["features_list"]): point_coords = point_coords_list[layer_id] point_labels = point_labels_list[layer_id] points_for_image = points_for_image_list[layer_id] crop_box = crop_boxes_list[layer_id] input_size = model_inputs["input_size_list"][layer_id] original_size = model_inputs["original_size_list"][layer_id] # Generate masks for this crop in batches data_crop = MaskData() for (points_for_image_batch, point_coords_batch, point_labels_batch) in \ batch_iterator(self.points_per_batch, points_for_image, point_coords, point_labels): masks, low_res_masks, iou_predictions = self.model( features=features, input_size=input_size, original_size=original_size, point_coords=point_coords_batch, point_labels=point_labels_batch, multimask_output=multimask_output, return_logits=True ) masks = masks.asnumpy() low_res_masks = low_res_masks.asnumpy() iou_predictions = iou_predictions.asnumpy() data_batch = MaskData( masks=masks.reshape((-1, masks.shape[-2], masks.shape[-1])), iou_preds=iou_predictions.reshape((-1,)), points=points_for_image_batch.repeat(masks.shape[1], axis=0) ) del masks # Filter by predicted IoU if self.pred_iou_thresh > 0.0: keep_mask = data_batch["iou_preds"] > self.pred_iou_thresh data_batch.filter(keep_mask) # Calculate stability score data_batch["stability_score"] = calculate_stability_score( data_batch["masks"], self.model.mask_threshold, self.stability_score_offset ) if self.stability_score_thresh > 0.0: keep_mask = data_batch["stability_score"] >= self.stability_score_thresh data_batch.filter(keep_mask) # Threshold masks and calculate boxes data_batch["masks"] = data_batch["masks"] > self.model.mask_threshold data_batch["boxes"] = batched_mask_to_box(data_batch["masks"]) # Filter boxes that touch crop boundaries keep_mask = ~is_box_near_crop_edge(data_batch["boxes"], crop_box, [0, 0, orig_w, orig_h]) if not keep_mask.all(): data_batch.filter(keep_mask) # Compress to RLE data_batch["masks"] = uncrop_masks(data_batch["masks"], crop_box, orig_h, orig_w) data_batch["rles"] = mask_to_rle(data_batch["masks"]) del data_batch["masks"] data_crop.cat(data_batch) # Remove duplicates within this crop. input_boxes = data_crop["boxes"] input_scores = data_crop["iou_preds"] keep_by_nms = nms(input_boxes, input_scores, iou_threshold=self.crop_nms_thresh) data_crop.filter(keep_by_nms) # Return to the original image frame data_crop["boxes"] = uncrop_boxes_xyxy(data_crop["boxes"], crop_box) data_crop["points"] = uncrop_points(data_crop["points"], crop_box) data_crop["crop_boxes"] = np.array([crop_box for _ in range(len(data_crop["rles"]))]) data.cat(data_crop) # Remove duplicate masks between crops if len(crop_boxes_list) > 1: # Prefer masks from smaller crops input_boxes = data["boxes"] input_scores = 1.0 / box_area(data["crop_boxes"]) keep_by_nms = nms(input_boxes, input_scores, iou_threshold=self.crop_nms_thresh) data.filter(keep_by_nms) return data def postprocess(self, model_outputs, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result. Return: classification results. """ seg_image = postprocess_params["seg_image"] if not seg_image: bs = model_outputs["masks"].shape[0] if bs == 1: for k, v in model_outputs.items(): model_outputs[k] = v[0].asnumpy() else: for k, v in model_outputs.items(): model_outputs[k] = v.asnumpy() return model_outputs # Filter small disconnected regions and holes in masks if self.min_mask_region_area > 0: model_outputs = self.postprocess_small_regions( model_outputs, self.min_mask_region_area, max(self.box_nms_thresh, self.crop_nms_thresh), ) # Encode masks if self.output_mode == "coco_rle": model_outputs["segmentations"] = [coco_encode_rle(rle) for rle in model_outputs["rles"]] elif self.output_mode == "binary_mask": model_outputs["segmentations"] = [rle_to_mask(rle) for rle in model_outputs["rles"]] else: model_outputs["segmentations"] = model_outputs["rles"] # Write mask records curr_anns = [] for idx in range(len(model_outputs["segmentations"])): ann = { "segmentation": model_outputs["segmentations"][idx], "area": area_from_rle(model_outputs["rles"][idx]), "bbox": box_xyxy_to_xywh(model_outputs["boxes"][idx]).tolist(), "predicted_iou": model_outputs["iou_preds"][idx].item(), "point_coords": [model_outputs["points"][idx].tolist()], "stability_score": model_outputs["stability_score"][idx].item(), "crop_box": box_xyxy_to_xywh(model_outputs["crop_boxes"][idx]).tolist(), } curr_anns.append(ann) return curr_anns def preprocess_image(self, image, image_format: str = "RGB") -> None: """ Calculates the image embeddings for the provided image, allowing masks to be predicted with the 'predict' method. Arguments: image (np.ndarray): The image for calculating masks. Expects an image in HWC uint8 format, with pixel values in [0, 255]. image_format (str): The color format of the image, in ['RGB', 'BGR']. """ if isinstance(image, str): image = cv2.imread(image) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) assert image_format in [ "RGB", "BGR", ], f"image_format must be in ['RGB', 'BGR'], is {image_format}." if image_format != self.network.image_format: image = image[..., ::-1] # Transform the image to the form expected by the model original_size = image.shape[:2] # h, w input_image, input_size = self.image_processor(image) return input_image, original_size, input_size def preprocess_prompts(self, original_size, point_coords=None, point_labels=None, boxes=None, masks=None, multi_seg=False): """preprocess prompts""" if point_coords is not None: ndim = point_coords.ndim if ndim == 1: point_coords = point_coords[None, :] if ndim == 2: if point_labels is None: point_labels = np.ones(point_coords.shape[0], dtype=np.int32) point_coords = self.image_processor.transform.apply_coords(point_coords, original_size) point_coords = np.expand_dims(point_coords, axis=0) point_labels = np.expand_dims(point_labels, axis=0) if multi_seg: point_coords = point_coords.transpose(1, 0, 2) point_labels = point_labels.transpose() elif ndim == 3: if point_labels is None: point_labels = np.ones((point_coords.shape[0], point_coords.shape[1]), dtype=np.int32) point_coords = self.image_processor.transform.apply_coords_batch(point_coords, original_size) else: raise ValueError("points's ndim < 3.") point_coords = Tensor(point_coords, dtype=ms.float32) point_labels = Tensor(point_labels, dtype=ms.int32) if boxes is not None: ndim = boxes.ndim if ndim == 1: boxes = self.image_processor.transform.apply_boxes(boxes, original_size) boxes = np.expand_dims(boxes, axis=0) elif ndim == 2: boxes = self.image_processor.transform.apply_boxes_batch(boxes, original_size) else: raise ValueError("boxes's ndim must be 1 or 2.") boxes = Tensor(boxes, dtype=ms.float32) if masks is not None: masks = np.expand_dims(masks, axis=0) masks = Tensor(masks, dtype=ms.float32) return point_coords, point_labels, boxes, masks def set_image(self, image, image_format: str = "RGB"): self.reset_image() input_image, original_size, input_size = self.preprocess_image(image, image_format) self.features = self.network.image_encoder(input_image) self.original_size = original_size # h, w self.input_size = input_size # h, w self.is_image_set = True def reset_image(self) -> None: """Resets the currently set image.""" self.is_image_set = False self.features = None self.original_size = None self.input_size = None def get_image_embedding(self): """ Returns the image embeddings for the currently set image, with shape 1xCxHxW, where C is the embedding dimension and (H,W) are the embedding spatial dimension of SAM (typically C=256, H=W=64). """ if not self.is_image_set: raise RuntimeError( "An image must be set with .set_image(...) to generate an embedding." ) assert self.features is not None return self.features def reset_config(self, points_per_side=32, points_per_batch=64, pred_iou_thresh=0.88, stability_score_thresh=0.95, stability_score_offset=1.0, box_nms_thresh=0.7, crop_n_layers=0, crop_nms_thresh=0.7, crop_overlap_ratio=0.3413, crop_n_points_downscale_factor=1, point_grids=None, min_mask_region_area=0, output_mode="binary_mask", **kwargs) -> None: """reset config""" self.points_per_side = points_per_side or kwargs.get('points_per_side', 32) self.points_per_batch = points_per_batch or kwargs.get('points_per_batch', 64) self.pred_iou_thresh = pred_iou_thresh or kwargs.get('points_per_batch', 0.88) self.stability_score_thresh = stability_score_thresh or kwargs.get('stability_score_thresh', 0.95) self.stability_score_offset = stability_score_offset or kwargs.get('stability_score_offset', 1.0) self.box_nms_thresh = box_nms_thresh or kwargs.get('box_nms_thresh', 0.7) self.crop_n_layers = crop_n_layers or kwargs.get('crop_n_layers', 0) self.crop_nms_thresh = crop_nms_thresh or kwargs.get('crop_nms_thresh', 0.7) self.crop_overlap_ratio = crop_overlap_ratio or kwargs.get('crop_overlap_ratio', 0.3413) self.crop_n_points_downscale_factor = crop_n_points_downscale_factor or\ kwargs.get('crop_n_points_downscale_factor', 1) self.point_grids = point_grids or kwargs.get('point_grids', None) self.min_mask_region_area = min_mask_region_area or kwargs.get('min_mask_region_area', 0) self.output_mode = output_mode or kwargs.get('output_mode', "binary_mask") assert (self.points_per_side is None) != ( self.point_grids is None ), "Exactly one of points_per_side or point_grid must be provided." if self.points_per_side is not None: self.point_grids = build_all_layer_point_grids( self.points_per_side, self.crop_n_layers, self.crop_n_points_downscale_factor ) assert self.output_mode in [ "binary_mask", "uncompressed_rle", "coco_rle", ], f"Unknown output_mode {self.output_mode}." def postprocess_small_regions(self, mask_data: MaskData, min_area: int, nms_thresh: float) -> MaskData: """ Removes small disconnected regions and holes in masks, then reruns box NMS to remove any new duplicates. Edits mask_data in place. Requires open-cv as a dependency. """ if not mask_data["rles"]: return mask_data # Filter small disconnected regions and holes new_masks = [] scores = [] for rle in mask_data["rles"]: mask = rle_to_mask(rle) mask, changed = remove_small_regions(mask, min_area, mode="holes") unchanged = not changed mask, changed = remove_small_regions(mask, min_area, mode="islands") unchanged = unchanged and not changed new_masks.append(np.expand_dims(mask, axis=0)) # Give score=0 to changed masks and score=1 to unchanged masks # so NMS will prefer ones that didn't need postprocessing scores.append(float(unchanged)) # Recalculate boxes and remove any new duplicates masks = np.concatenate(new_masks, axis=0) boxes = batched_mask_to_box(masks) input_boxes = boxes.astype(np.float32) input_scores = np.array(scores) keep_by_nms = nms(input_boxes, input_scores, iou_threshold=nms_thresh) # Only recalculate RLEs for masks that have changed for i_mask in keep_by_nms: if scores[i_mask] == 0.0: mask_tensor = np.expand_dims(masks[i_mask], axis=0) mask_data["rles"][i_mask] = mask_to_rle(mask_tensor)[0] mask_data["boxes"][i_mask] = boxes[i_mask] # update res directly mask_data.filter(keep_by_nms) return mask_data pengcheng.mind-mindformer/mindformers/pipeline/text_classification_pipeline.py000066400000000000000000000242671476650141400305550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TextClassificationPipeline""" import os.path import numpy as np from mindspore import ops, Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer from ..dataset.labels import labels __all__ = ['TextClassificationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="text_classification") class TextClassificationPipeline(BasePipeline): """Pipeline for text classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TextClassificationPipeline >>> from mindformers import AutoTokenizer, BertForMultipleChoice, AutoConfig >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> tokenizer = AutoTokenizer.from_pretrained('txtcls_bert_base_uncased_mnli') >>> txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') >>> model = BertForMultipleChoice(txtcls_mnli_config) >>> txtcls_pipeline = TextClassificationPipeline(task='text_classification', ... model=model, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = txtcls_pipeline(input_data, top_k=1) >>> print(results) [[{'label': 'neutral', 'score': 0.9714198708534241}], [{'label': 'contradiction', 'score': 0.9967639446258545}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['text_classification'].keys() def __init__(self, model, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) preprocess_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} if "top_k" in pipeline_parameters: postprocess_params["top_k"] = pipeline_parameters.get("top_k") if "dataset" in pipeline_parameters: postprocess_params["dataset"] = pipeline_parameters.get("dataset") if not labels.get(postprocess_params["dataset"]): raise ValueError(f"The dataset does not support {postprocess_params['dataset']}, " f"but only support {labels.keys()}") return preprocess_params, forward_kwargs, postprocess_params def inputs_process(self, inputs_zero, inputs_one): """ process of two sentences relationship classification Args: inputs_zero (str): the first sentence inputs_one (str): the second sentence Return: processed inputs, mask, token_type about two sentences """ len_inputs = len(inputs_zero["input_ids"]) inputs_zero_input = list(inputs_zero["input_ids"].asnumpy()) inputs_one_input = list(inputs_one["input_ids"].asnumpy()) inputs_zero_input = [x for x in inputs_zero_input if x != 0] inputs_one_input = [x for x in inputs_one_input if x != 0] token_type = [0] * len(inputs_zero_input) + [1] * (len(inputs_one_input) - 1) token_type = token_type + [0] * (len_inputs - len(token_type)) inputs = inputs_zero_input + inputs_one_input[1:] len_inputs_mask = len(inputs) inputs = inputs + [0] * (len_inputs - len(inputs)) mask = [1] * len_inputs_mask + [0] * (len_inputs - len_inputs_mask) return inputs, mask, token_type def preprocess(self, inputs, **preprocess_params): """ Preprocess of text classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("Inputs type must be str") expand_dims = ops.ExpandDims() if self.model.model_name == "gpt2_txtcls": tokens = self.tokenizer(inputs, return_tensors="ms", **preprocess_params) input_ids = tokens["input_ids"] attention_mask = tokens["attention_mask"] return {"input_ids": expand_dims(input_ids, 0), "attention_mask": expand_dims(attention_mask, 0)} if '-' not in inputs: raise ValueError("two texts of text pair should be split by -") inputs = inputs.split('-') max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs_zero = self.tokenizer(inputs[0], max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) inputs_one = self.tokenizer(inputs[1], max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) inputs, mask, token_type = self.inputs_process(inputs_zero, inputs_one) inputs_final = {} inputs_final["input_ids"] = Tensor.from_numpy(np.array(inputs, dtype=np.int32)) inputs_final["attention_mask"] = Tensor.from_numpy(np.array(mask, dtype=np.int32)) inputs_final["token_type_ids"] = Tensor.from_numpy(np.array(token_type, dtype=np.int32)) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs_final["input_ids"], 0), "input_mask": expand_dims(inputs_final["attention_mask"], 0), "token_type_id": expand_dims(inputs_final["token_type_ids"], 0), "label_ids": None} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ forward_params.pop("None", None) output_ids = self.network(**model_inputs) return output_ids def softmax(self, outputs): maxes = np.max(outputs, axis=-1, keepdims=True) shifted_exp = np.exp(outputs - maxes) return shifted_exp / shifted_exp.sum(axis=-1, keepdims=True) def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: Classification results """ top_k = postprocess_params.pop("top_k", None) dataset = postprocess_params.pop("dataset", None) if dataset: id2label = {id: label for id, label in enumerate(labels.get(dataset))} else: id2label = {id: label for id, label in enumerate(labels.get("mnli"))} outputs = model_outputs[0] outputs = outputs.asnumpy() scores = self.softmax(outputs) dict_scores = [ {"label": id2label[i], "score": score.item()} for i, score in enumerate(scores) ] dict_scores.sort(key=lambda x: x["score"], reverse=True) if top_k is not None: dict_scores = dict_scores[:top_k] return [dict_scores] pengcheng.mind-mindformer/mindformers/pipeline/text_generation_pipeline.py000066400000000000000000000140331476650141400277030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TextGenerationPipeline""" import os.path from typing import Union, Optional import mindspore from mindspore import Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, BaseTokenizer __all__ = ['TextGenerationPipeline'] def _setup_support_list(support_model_list): support_list = [] for support_model in support_model_list: support_list.append(MindFormerBook.get_model_support_list().get(support_model)) return support_list @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="text_generation") class TextGenerationPipeline(BasePipeline): r"""Pipeline for Text Generation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TextGenerationPipeline >>> text_generate = TextGenerationPipeline("gpt2") >>> output = text_generate("I love Beijing, because ") """ _support_list = _setup_support_list(["gpt2", "glm"]) return_name = 'text_generation' def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_keys = ['keys', 'add_special_tokens'] preprocess_params = {} for item in preprocess_keys: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length', 'seed'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs: Union[str, dict, Tensor], **preprocess_params): r"""The Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. """ add_special_tokens = preprocess_params.get('add_special_tokens', True) if isinstance(inputs, dict): keys = preprocess_params.get('keys', None) default_src_language_name = 'text' feature_name = keys.get('src_language', default_src_language_name) if keys else default_src_language_name inputs = inputs[feature_name] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() input_ids = self.tokenizer(inputs, return_tensors=None, add_special_tokens=add_special_tokens)["input_ids"] return {"input_ids": input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) input_ids = model_inputs["input_ids"] output_ids = self.network.generate(input_ids, **forward_params) return {"output_ids": output_ids} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. """ outputs = self.tokenizer.decode(model_outputs["output_ids"], skip_special_tokens=True) return [{self.return_name + '_text': outputs}] pengcheng.mind-mindformer/mindformers/pipeline/token_classification_pipeline.py000066400000000000000000000236161476650141400307060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/lonePatient/daguan_2019_rank9/blob/master/pydatagrand/train/ner_utils.py # ============================================================================ """TokenClassificationPipeline""" import os.path import numpy as np from mindspore import ops, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, Tokenizer __all__ = ['TokenClassificationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="token_classification") class TokenClassificationPipeline(BasePipeline): r"""Pipeline for token classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer : a tokenizer (None or Tokenizer) for text processing. id2label : a dict which maps label id to label str. Raises: TypeError: If input model and image_processor's types are not corrected. ValueError: If the input model is not in support list. Examples: >>> from mindformers.pipeline import TokenClassificationPipeline >>> from mindformers import AutoTokenizer, BertForTokenClassification, AutoConfig >>> from mindformers.dataset.labels import cluener_labels >>> id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。"] >>> tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') >>> ner_dense_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') >>> model = BertForTokenClassification(ner_dense_cluener_config) >>> tokcls_pipeline = TokenClassificationPipeline(task='token_classification', ... model=model, ... id2label=id2label, ... tokenizer=tokenizer, ... max_length=model.config.seq_length, ... padding="max_length") >>> results = tokcls_pipeline(input_data) >>> print(results) [[{'entity_group': 'address', 'start': 6, 'end': 8, 'score': 0.52329, 'word': '日内瓦'}, {'entity_group': 'name', 'start': 12, 'end': 25, 'score': 0.83922, 'word': 'freresoltramar'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['token_classification'].keys() def __init__(self, model, id2label, tokenizer=None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") if id2label is None: raise ValueError(f"{self.__class__.__name__}" " requires for a dict which maps label id to label str.") self.id2label = id2label self.input_text = "" super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): """sanitize parameters for preprocess, forward, and postprocess.""" if 'batch_size' in pipeline_parameters: raise ValueError(f"The {self.__class__.__name__} does not support batch inference, please remove the " f"batch_size") postprocess_params = {'id2label'} preprocess_key_name = ['max_length', 'padding'] preprocess_params = {k: v for k, v in pipeline_parameters.items() if k in preprocess_key_name} postprocess_params = {k: v for k, v in pipeline_parameters.items() if k in postprocess_params} return preprocess_params, {}, postprocess_params def preprocess(self, inputs, **preprocess_params): """ Preprocess of token classification Args: inputs (str): the str to be classified. max_length (int): max length of tokenizer's output padding (False / "max_length"): padding for max_length return_tensors ("ms"): the type of returned tensors Return: processed text. """ if not isinstance(inputs, str): raise ValueError("Inputs type must be str") self.input_text = inputs max_length = preprocess_params.pop("max_length", 128) padding = preprocess_params.pop("padding", "max_length") inputs = self.tokenizer(inputs, max_length=max_length, padding=padding, return_tensors="ms", **preprocess_params) expand_dims = ops.ExpandDims() return {"input_ids": expand_dims(inputs["input_ids"], 0), "input_mask": expand_dims(inputs["attention_mask"], 0), "token_type_ids": expand_dims(inputs["token_type_ids"], 0)} def forward(self, model_inputs, **forward_params): """ Forward process Args: model_inputs (dict): outputs of preprocess. Return: probs dict. """ self.model.set_train(False) logits = self.network(**model_inputs) return {"logits": logits} def postprocess(self, model_outputs, **postprocess_params): """ Postprocess Args: model_outputs (dict): outputs of forward process. Return: The generated results """ logits = model_outputs["logits"].asnumpy() maxes = np.max(logits, axis=-1, keepdims=True) shifted_exp = np.exp(logits - maxes) probs = shifted_exp / shifted_exp.sum(axis=-1, keepdims=True) batch_pred_ids = np.argmax(probs, axis=2).tolist() batch_best_scores = np.max(probs, axis=2).tolist() # remove CLS and SEP token pred_paths = [[self.id2label[id_] for id_ in pred_ids[1:-1]] for pred_ids in batch_pred_ids] best_scores = [best_scores[1:-1] for best_scores in batch_best_scores] total_result = [] for pred_path, best_score in zip(pred_paths, best_scores): single_result = [] pred_entities = self.get_entities_bios(pred_path) for pred_entity in pred_entities: entity_result = {} entity_result["entity_group"] = pred_entity[0] entity_result["start"] = pred_entity[1] entity_result["end"] = pred_entity[2] entity_result["score"] = sum(best_score[entity_result["start"]:entity_result["end"] + 1]) / \ (entity_result["end"] + 1 - entity_result["start"]) entity_result["score"] = round(entity_result["score"], 5) entity_result["word"] = self.input_text[entity_result["start"]:entity_result["end"] + 1] single_result.append(entity_result) total_result.append(single_result) return total_result def get_entities_bios(self, seq): """Gets entities from sequence. note: BIOS Args: seq (list): sequence of labels. Returns: list: list of (chunk_type, chunk_start, chunk_end). Example: # >>> seq = ['B-PER', 'I-PER', 'O', 'S-LOC'] # >>> get_entity_bios(seq) [['PER', 0, 1], ['LOC', 3, 3]] """ chunks = [] chunk = [-1, -1, -1] for indx, tag in enumerate(seq): if tag.startswith("S-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[2] = indx chunk[0] = tag.split('-')[1] chunks.append(chunk) chunk = [-1, -1, -1] if tag.startswith("B-"): if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] chunk[1] = indx chunk[0] = tag.split('-')[1] elif tag.startswith('I-') and chunk[1] != -1: entity_type = tag.split('-')[1] if entity_type == chunk[0]: chunk[2] = indx if indx == len(seq) - 1: chunks.append(chunk) else: if chunk[2] != -1: chunks.append(chunk) chunk = [-1, -1, -1] return chunks pengcheng.mind-mindformer/mindformers/pipeline/translation_pipeline.py000066400000000000000000000131741476650141400270470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """TranslationPipeline""" import os.path from typing import Union, Optional import mindspore from mindspore import Tensor, Model from ..auto_class import AutoProcessor, AutoModel from ..mindformer_book import MindFormerBook from .base_pipeline import BasePipeline from ..tools.register import MindFormerRegister, MindFormerModuleType from ..models import BaseModel, BaseTokenizer __all__ = ['TranslationPipeline'] @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="translation") class TranslationPipeline(BasePipeline): r"""Pipeline for Translation Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer (None or Tokenizer) for text processing. Raises: TypeError: If input model and tokenizer's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.pipeline import TranslationPipeline >>> translator = TranslationPipeline("t5_small") >>> output = translator("abc") """ _support_list = MindFormerBook.get_model_support_list()['t5'] return_name = 'translation' def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, **kwargs): if isinstance(model, str): if model in self._support_list or os.path.isdir(model): if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer model = AutoModel.from_pretrained(model) if not isinstance(tokenizer, BaseTokenizer): raise TypeError(f"tokenizer should be inherited from" f" BaseTokenizer, but got {type(tokenizer)}.") else: raise ValueError(f"{model} is not supported by {self.__class__.__name__}," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError(f"{self.__class__.__name__}" " requires for a tokenizer.") super().__init__(model, tokenizer, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_keys = ['keys'] preprocess_params = {} for item in preprocess_keys: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) postprocess_params = {} forward_key_name = ['top_k', 'top_p', 'do_sample', 'eos_token_id', 'repetition_penalty', 'max_length'] forward_kwargs = {} for item in forward_key_name: if item in pipeline_parameters: forward_kwargs[item] = pipeline_parameters.get(item) return preprocess_params, forward_kwargs, postprocess_params def preprocess(self, inputs: Union[str, dict, Tensor], **preprocess_params): r"""The Preprocess For Translation Args: inputs (Union[str, dict, Tensor]): The text to be classified. preprocess_params (dict): The parameter dict for preprocess. Return: Processed text. """ if isinstance(inputs, dict): keys = preprocess_params.get('keys', None) default_src_language_name = 'text' feature_name = keys.get('src_language', default_src_language_name) if keys else default_src_language_name inputs = inputs[feature_name] if isinstance(inputs, mindspore.Tensor): inputs = inputs.asnumpy().tolist() input_ids = self.tokenizer(inputs, return_tensors=None)["input_ids"] return {"input_ids": input_ids} def forward(self, model_inputs: dict, **forward_params): r"""The Forward Process of Model Args: inputs (dict): The output of preprocess. forward_params (dict): The parameter dict for model forward. """ forward_params.pop("None", None) input_ids = model_inputs["input_ids"] output_ids = self.network.generate(input_ids, **forward_params) return {"output_ids": output_ids} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. Return: translation results. """ outputs = self.tokenizer.decode(model_outputs["output_ids"], skip_special_tokens=True) return [{self.return_name + '_text': outputs}] pengcheng.mind-mindformer/mindformers/pipeline/zero_shot_image_classification_pipeline.py000066400000000000000000000216751476650141400327470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ZeroShotImageClassificationPipeline""" from typing import Union, Optional from mindspore import Model from mindspore.ops import operations as P from mindformers.mindformer_book import MindFormerBook from mindformers.tools.image_tools import load_image from mindformers.models import BaseModel, BaseImageProcessor, Tokenizer from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.auto_class import AutoProcessor, AutoModel from .base_pipeline import BasePipeline from ..models import BaseTokenizer @MindFormerRegister.register(MindFormerModuleType.PIPELINE, alias="zero_shot_image_classification") class ZeroShotImageClassificationPipeline(BasePipeline): r"""Pipeline For Zero Shot Image Classification Args: model (Union[str, BaseModel]): The model used to perform task, the input could be a supported model name, or a model instance inherited from BaseModel. tokenizer (Optional[BaseTokenizer]): A tokenizer for text processing. image_processor (Optional[BaseImageProcessor]): The image_processor of model, it could be None if the model do not need image_processor. Raises: TypeError: If input model, tokenizer, and image_processor's types are not corrected. ValueError: if the input model is not in support list. Examples: >>> from mindformers.tools.image_tools import load_image >>> from mindformers.pipeline import ZeroShotImageClassificationPipeline >>> classifier = ZeroShotImageClassificationPipeline( ... model='clip_vit_b_32', ... candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], ... hypothesis_template="This is a photo of {}." ... ) >>> img = load_image("https://ascend-repo-modelzoo.obs.cn-east-2." ... "myhuaweicloud.com/XFormer_for_mindspore/clip/sunflower.png") >>> classifier(img) [[{'score': 0.99995565, 'label': 'sunflower'}, {'score': 2.5318595e-05, 'label': 'toy'}, {'score': 9.903885e-06, 'label': 'dog'}, {'score': 6.75336e-06, 'label': 'tree'}, {'score': 2.396818e-06, 'label': 'cat'}]] """ _support_list = MindFormerBook.get_pipeline_support_task_list()['zero_shot_image_classification'].keys() def __init__(self, model: Union[str, BaseModel, Model], tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): if isinstance(model, str): if model in self._support_list: if image_processor is None: image_processor = AutoProcessor.from_pretrained(model).image_processor if not isinstance(image_processor, BaseImageProcessor): raise TypeError(f"image_processor should be inherited from" f" BaseImageProcessor, but got {type(image_processor)}.") if tokenizer is None: tokenizer = AutoProcessor.from_pretrained(model).tokenizer if not isinstance(tokenizer, Tokenizer): raise TypeError(f"tokenizer should be inherited from" f" PretrainedTokenizer, but got {type(tokenizer)}.") model = AutoModel.from_pretrained(model) else: raise ValueError(f"{model} is not supported by ZeroShotImageClassificationPipeline," f"please selected from {self._support_list}.") if not isinstance(model, (BaseModel, Model)): raise TypeError(f"model should be inherited from BaseModel or Model, but got type {type(model)}.") if tokenizer is None: raise ValueError("ZeroShotImageClassificationPipeline" " requires for a tokenizer.") if image_processor is None: raise ValueError("ZeroShotImageClassificationPipeline" " requires for a image_processor.") super().__init__(model, tokenizer, image_processor, **kwargs) def _sanitize_parameters(self, **pipeline_parameters): r"""Sanitize Parameters Args: pipeline_parameters (Optional[dict]): The parameter dict to be parsed. """ preprocess_params = {} postprocess_params = {} pre_list = ["candidate_labels", "hypothesis_template", "max_length", "padding", "return_tensors"] for item in pre_list: if item in pipeline_parameters: preprocess_params[item] = pipeline_parameters.get(item) if "top_k" in pipeline_parameters: postprocess_params["top_k"] = pipeline_parameters.get("top_k") return preprocess_params, {}, postprocess_params def preprocess(self, inputs: dict, **preprocess_params): r"""Preprocess of ZeroShotImageClassificationPipeline Args: inputs (Union[url, PIL.Image, tensor, numpy]): The image to be classified. candidate_labels (List[str]): The candidate labels for classification. hypothesis_template (Optional[str]): Prompt for text input. max_length (Optional[int]): Max length of tokenizer's output padding (Optional[Union[False, "max_length"]]): Padding for max_length return_tensors (Optional["ms"]): The type of returned tensors Return: Processed data. Raises: ValueError: If candidate_labels or hypothesis_template is None. """ candidate_labels = preprocess_params.pop("candidate_labels", None) hypothesis_template = preprocess_params.pop("hypothesis_template", "a picture of {}.") max_length = preprocess_params.pop("max_length", 77) padding = preprocess_params.pop("padding", "max_length") return_tensors = preprocess_params.pop("return_tensors", "ms") if candidate_labels is None: raise ValueError("candidate_labels are supposed for" " ZeroShotImageClassificationPipeline, but got None.") if hypothesis_template is None: raise ValueError("hypothesis_template is supposed for" " ZeroShotImageClassificationPipeline, but got None.") if isinstance(inputs, dict): inputs = inputs['image'] if isinstance(inputs, str): inputs = load_image(inputs) image_processed = self.image_processor(inputs) sentences = [hypothesis_template.format(candidate_label) for candidate_label in candidate_labels] input_ids = self.tokenizer(sentences, max_length=max_length, padding=padding, return_tensors=return_tensors)["input_ids"] return {"image_processed": image_processed, "input_ids": input_ids, "candidate_labels": candidate_labels} def forward(self, model_inputs: dict, **forward_params): r"""Forward process Args: model_inputs (dict): Outputs of preprocess. Return: Probs dict. """ forward_params.pop("None", None) image_processed = model_inputs["image_processed"] input_ids = model_inputs["input_ids"] logits_per_image, _ = self.network(image_processed, input_ids) probs = P.Softmax()(logits_per_image).asnumpy() return {"probs": probs, "candidate_labels": model_inputs["candidate_labels"]} def postprocess(self, model_outputs: dict, **postprocess_params): r"""Postprocess Args: model_outputs (dict): Outputs of forward process. top_k (int): Return top_k probs of result Return: Classification results. """ top_k = postprocess_params.pop("top_k", None) labels = model_outputs['candidate_labels'] scores = model_outputs['probs'] outputs = [] for score in scores: sorted_res = sorted(zip(score, labels), key=lambda x: -x[0]) if top_k is not None: sorted_res = sorted_res[:min(top_k, len(labels))] outputs.append([{"score": score_item, "label": label} for score_item, label in sorted_res]) return outputs pengcheng.mind-mindformer/mindformers/tools/000077500000000000000000000000001476650141400215775ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/.ipynb_checkpoints/000077500000000000000000000000001476650141400253705ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/.ipynb_checkpoints/logger-checkpoint.py000066400000000000000000000444021476650141400313520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LOGGER Module""" import logging import logging.config import logging.handlers import os import stat import sys from functools import wraps from tempfile import TemporaryFile from typing import Dict, List, Tuple, Union from mindformers.tools.utils import get_num_nodes_devices, get_rank_info, \ convert_nodes_devices_input, generate_rank_list,\ check_in_modelarts, check_list, LOCAL_DEFAULT_PATH,\ get_output_root_path logger_list = [] LEVEL = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') MODELARTS_LOG_FILE_DIR = '/cache/ma-user-work/' LOCAL_DEFAULT_LOG_FILE_DIR = os.path.join( os.getenv("LOCAL_DEFAULT_PATH", LOCAL_DEFAULT_PATH), 'log') RANK_DIR_FORMATTER = 'rank_{}/' DEFAULT_FILEHANDLER_FORMAT = '[%(levelname)s] %(asctime)s ' \ '[%(pathname)s:%(lineno)d] %(funcName)s: %(message)s' DEFAULT_STDOUT_FORMAT = '%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)s' DEFAULT_REDIRECT_FILE_NAME = 'mindspore.log' def judge_redirect(rank_id: int, rank_size: int, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None): """Determine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. """ is_redirect = True if rank_size > 1 and (redirect_nodes is not None or redirect_devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) redirect_nodes = convert_nodes_devices_input(redirect_nodes, num_nodes) redirect_devices = convert_nodes_devices_input(redirect_devices, num_devices) check_list('nodes', redirect_nodes, num_nodes) check_list('devices', redirect_devices, num_devices) rank_list = generate_rank_list(redirect_nodes, redirect_devices) if rank_id not in rank_list: is_redirect = False return is_redirect class StreamRedirector: """Stream Re-director for Log.""" def __init__(self, source_stream, target_stream): """Redirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. """ super(StreamRedirector, self).__init__() self.source_stream = source_stream self.target_stream = target_stream self.save_source_stream_fd = os.dup(self.source_stream.fileno()) def __call__(self, func): @wraps(func) def wrapper(*args, **kwargs): self.start() func(*args, **kwargs) self.stop() return wrapper def __enter__(self): self.start() def __exit__(self, exc_type, exc_val, exc_tb): self.stop() def start(self): """start.""" self.source_stream.flush() os.dup2(self.target_stream.fileno(), self.source_stream.fileno()) def stop(self): """stop.""" self.source_stream.flush() os.dup2(self.save_source_stream_fd, self.source_stream.fileno()) self.target_stream.flush() class AiLogFastStreamRedirect2File(StreamRedirector): """AiLogFastStreamRedirect2File.""" def __init__(self, source_stream=None, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None, **kwargs): """Redirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. """ rank_id, rank_size = get_rank_info() self.is_redirect = judge_redirect(rank_id=rank_id, rank_size=rank_size, redirect_nodes=redirect_nodes, redirect_devices=redirect_devices) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', '') if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') if append_rank_dir: rank_str = RANK_DIR_FORMATTER.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) if not file_name: file_name = DEFAULT_REDIRECT_FILE_NAME self.file_path = os.path.join(file_save_dir, file_name) self.file_save_dir = os.path.dirname(self.file_path) if source_stream is None: source_stream = sys.stderr target_stream = TemporaryFile(mode='w+') super(AiLogFastStreamRedirect2File, self).__init__(source_stream=source_stream, target_stream=target_stream) def start(self): if self.is_redirect: super(AiLogFastStreamRedirect2File, self).start() def stop(self): if self.is_redirect: self.target_stream.flush() if not os.path.exists(self.file_save_dir): os.makedirs(self.file_save_dir, exist_ok=True) self.target_stream.seek(0, 0) flags = os.O_WRONLY | os.O_CREAT modes = stat.S_IWUSR | stat.S_IRUSR with os.fdopen(os.open(self.file_path, flags, modes), 'w') as fp: for line in self.target_stream: fp.write(line) super(AiLogFastStreamRedirect2File, self).stop() self.target_stream.close() def judge_stdout(rank_id: int, rank_size: int, is_output: bool, nodes: Union[List, Tuple, Dict[str, int], None] = None, devices: Union[List, Tuple, Dict[str, int], None] = None) -> bool: """Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. """ if is_output and rank_size > 1 and (nodes is not None or devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) stdout_nodes = convert_nodes_devices_input(nodes, num_nodes) stdout_devices = convert_nodes_devices_input(devices, num_devices) check_list('nodes', stdout_nodes, num_nodes) check_list('devices', stdout_devices, num_devices) rank_list = generate_rank_list(stdout_nodes, stdout_devices) if rank_id not in rank_list: is_output = False return is_output def validate_nodes_devices_input(var_name: str, var): """Check the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None """ if not (var is None or isinstance(var, (list, tuple, dict))): raise TypeError('The value of {} can be None or a value of type tuple, ' 'list, or dict.'.format(var_name)) if isinstance(var, (list, tuple)): for item in var: if not isinstance(item, int): raise TypeError('The elements of a variable of type list or ' 'tuple must be of type int.') def validate_level(var_name: str, var): """Verify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None """ if not isinstance(var, str): raise TypeError('The format of {} must be of type str.'.format(var_name)) if var not in LEVEL: raise ValueError('{}={} needs to be in {}'.format(var_name, var, LEVEL)) def validate_std_input_format(to_std: bool, stdout_nodes: Union[List, Tuple, None], stdout_devices: Union[List, Tuple, None], stdout_level: str): """Validate the input about stdout of the get_logger function.""" if not isinstance(to_std, bool): raise TypeError('The format of the to_std must be of type bool.') validate_nodes_devices_input('stdout_nodes', stdout_nodes) validate_nodes_devices_input('stdout_devices', stdout_devices) validate_level('stdout_level', stdout_level) def validate_file_input_format(file_level: Union[List[str], Tuple[str]], file_save_dir: str, append_rank_dir: str, file_name: Union[List[str], Tuple[str]]): """Validate the input about file of the get_logger function.""" if not isinstance(file_level, (tuple, list)): raise TypeError('The value of file_level should be list or a tuple.') for level in file_level: validate_level('level in file_level', level) if not len(file_level) == len(file_name): raise ValueError('The length of file_level and file_name should be equal.') if not isinstance(file_save_dir, str): raise TypeError('The value of file_save_dir should be a value of type str.') if not isinstance(append_rank_dir, bool): raise TypeError('The value of append_rank_dir should be a value of type bool.') if not isinstance(file_name, (tuple, list)): raise TypeError('The value of file_name should be list or a tuple.') for name in file_name: if not isinstance(name, str): raise TypeError('The value of name in file_name should be a value of type str.') def _convert_level(level: str) -> int: """Convert the format of the log to logging level. Args: level (str): User log level. Returns: level (str): Logging level. """ level_convert = { 'DEBUG': logging.DEBUG, 'INFO': logging.INFO, 'WARNING': logging.WARNING, 'ERROR': logging.ERROR, 'CRITICAL': logging.CRITICAL } level = level_convert.get(level, logging.INFO) return level def get_logger(logger_name: str = 'mindformers', **kwargs) -> logging.Logger: """Get the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. """ mf_logger = logging.getLogger(logger_name) if logger_name in logger_list: return mf_logger to_std = kwargs.get('to_std', True) stdout_nodes = kwargs.get('stdout_nodes', None) rank_id, rank_size = get_rank_info() def get_stdout_devices(): if os.getenv("STDOUT_DEVICES"): devices = os.getenv("STDOUT_DEVICES") if devices.startswith(("(", "[")) and devices.endswith((")", "]")): devices = devices[1:-1] devices = tuple(map(lambda x: int(x.strip()), devices.split(","))) elif check_in_modelarts(): devices = kwargs.get('stdout_devices', (min(rank_size - 1, 7),)) else: devices = kwargs.get('stdout_devices', None) return devices stdout_devices = get_stdout_devices() stdout_level = kwargs.get('stdout_level', 'INFO') stdout_format = kwargs.get('stdout_format', '') file_level = kwargs.get('file_level', ('INFO', 'ERROR')) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', (f'info.log', 'error.log')) max_file_size = kwargs.get('max_file_size', 50) max_num_of_files = kwargs.get('max_num_of_files', 5) validate_std_input_format(to_std, stdout_nodes, stdout_devices, stdout_level) validate_file_input_format(file_level, file_save_dir, append_rank_dir, file_name) to_std = judge_stdout(rank_id=rank_id, rank_size=rank_size, is_output=to_std, nodes=stdout_nodes, devices=stdout_devices) if to_std: if not stdout_format: stdout_format = DEFAULT_STDOUT_FORMAT stream_handler = logging.StreamHandler(sys.stdout) stream_handler.setLevel(_convert_level(stdout_level)) stream_formatter = logging.Formatter(stdout_format) stream_handler.setFormatter(stream_formatter) mf_logger.addHandler(stream_handler) logging_level = [] for level in file_level: logging_level.append(_convert_level(level)) if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') rank_dir = RANK_DIR_FORMATTER if append_rank_dir: rank_str = rank_dir.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) file_path = [] for name in file_name: path = os.path.join(file_save_dir, name) path = os.path.realpath(path) base_dir = os.path.dirname(path) if not os.path.exists(base_dir): os.makedirs(base_dir, exist_ok=True) file_path.append(path) max_file_size = max_file_size * 1024 * 1024 file_formatter = logging.Formatter(DEFAULT_FILEHANDLER_FORMAT) for i, level in enumerate(file_level): file_handler = logging.handlers.RotatingFileHandler(filename=file_path[i], maxBytes=max_file_size, backupCount=max_num_of_files) file_handler.setLevel(level) file_handler.setFormatter(file_formatter) mf_logger.addHandler(file_handler) mf_logger.setLevel(_convert_level('INFO')) mf_logger.propagate = False logger_list.append(logger_name) return mf_logger class _LogActionOnce: """ A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. """ is_logged = dict() def __init__(self, m_logger, key, no_warning=False): self.logger = m_logger self.key = key self.no_warning = no_warning def __call__(self, func): def wrapper(*args, **kwargs): if not hasattr(self.logger, 'warning'): return func(*args, **kwargs) old_func = self.logger.warning if self.no_warning or self.key in _LogActionOnce.is_logged: self.logger.warning = lambda x: x else: _LogActionOnce.is_logged[self.key] = True res = func(*args, **kwargs) if hasattr(self.logger, 'warning'): self.logger.warning = old_func return res return wrapper logger = get_logger() pengcheng.mind-mindformer/mindformers/tools/.ipynb_checkpoints/utils-checkpoint.py000066400000000000000000000261511476650141400312340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Utils For Tools.""" import json import os from typing import Dict, List, Tuple, Union from multiprocessing import Process try: import fcntl except ImportError: fcntl = None import numpy as np from mindspore import Tensor from mindspore import context PARALLEL_MODE = {'DATA_PARALLEL': context.ParallelMode.DATA_PARALLEL, 'SEMI_AUTO_PARALLEL': context.ParallelMode.SEMI_AUTO_PARALLEL, 'AUTO_PARALLEL': context.ParallelMode.AUTO_PARALLEL, 'HYBRID_PARALLEL': context.ParallelMode.HYBRID_PARALLEL, 'STAND_ALONE': context.ParallelMode.STAND_ALONE, 0: context.ParallelMode.DATA_PARALLEL, 1: context.ParallelMode.SEMI_AUTO_PARALLEL, 2: context.ParallelMode.AUTO_PARALLEL, 3: context.ParallelMode.HYBRID_PARALLEL} MODE = {'PYNATIVE_MODE': context.PYNATIVE_MODE, 'GRAPH_MODE': context.GRAPH_MODE, 0: context.GRAPH_MODE, 1: context.PYNATIVE_MODE} SAVE_WORK_PATH = '/cache/ma-user-work/rank_{}' MA_OUTPUT_ROOT = '/cache/ma-user-work' DEBUG_INFO_PATH = '/cache/debug' PROFILE_INFO_PATH = '/cache/profile' PLOG_PATH = '/root/ascend/log' LOCAL_DEFAULT_PATH = os.getenv("LOCAL_DEFAULT_PATH", './output') LAST_TRANSFORM_LOCK_PATH = "/tmp/last_transform_done.lock" RANK_FINISH_PATH = "/cache/rank_*_finish.txt" _PROTOCOL = 'obs' _PROTOCOL_S3 = 's3' def check_in_modelarts(): """Check if the training is on modelarts. Returns: (bool): If it is True, it means ModelArts environment. """ # 'KUBERNETES_PORT' in os.environ or \ return 'MA_LOG_DIR' in os.environ or \ 'MA_JOB_DIR' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ or \ 'S3_ACCESS_KEY_ID' in os.environ or \ 'S3_SECRET_ACCESS_KEY' in os.environ or \ 'BATCH_GROUP_NAME' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ class Validator: """validator for checking input parameters""" @staticmethod def check_type(arg_value, arg_type): """Check int.""" if not isinstance(arg_value, arg_type): raise TypeError('{} should be {} type, but get {}'.format(arg_value, arg_type, type(arg_value))) @staticmethod def is_obs_url(url): """Check obs url.""" return url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://') def check_obs_url(url): """Check obs url.""" if not isinstance(url, str): raise TypeError('remote_save_url type should be a str, but get {}, ' 'please check your remote_save_url config'.format(type(url))) if not (url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://')): raise TypeError('remote_save_url should be start with obs:// or s3://, ' 'but get {}, please check your remote_save_url config'.format(url)) return True def check_list(var_name: str, list_var: Union[Tuple, List], num: int): """Checks the legitimacy of elements within a node or device list. Args: var_name (str): The Name of variable need to check. list_var (tuple or list): Variables in list format. num (int): The number of nodes or devices. Returns: None """ for value in list_var: if value >= num: raise ValueError('The index of the {} needs to be less than the number of nodes {}.'.format(var_name, num)) def format_path(path): """Check path.""" return os.path.realpath(path) def sync_trans(f): """Asynchronous transport decorator.""" try: def wrapper(*args, **kwargs): pro = Process(target=f, args=args, kwargs=kwargs) pro.start() return pro return wrapper except Exception as e: raise e def get_output_root_path(): """get default output path in local/AICC.""" if check_in_modelarts(): return MA_OUTPUT_ROOT return LOCAL_DEFAULT_PATH def get_output_subpath(sub_class, rank_id=0, append_rank=True): """get output store path for sub output class.""" Validator.check_type(sub_class, str) root_path = get_output_root_path() directory = os.path.join(root_path, sub_class) if append_rank: directory = os.path.join(directory, 'rank_{}'.format(rank_id)) return format_path(directory) def set_remote_save_url(remote_save_url): check_obs_url(remote_save_url) os.environ.setdefault('REMOTE_SAVE_URL', remote_save_url) def get_remote_save_url(): return os.environ.get('REMOTE_SAVE_URL', None) def get_net_outputs(params): """Get network outputs.""" if isinstance(params, (tuple, list)): if isinstance(params[0], Tensor) and isinstance(params[0].asnumpy(), np.ndarray): params = params[0] elif isinstance(params, Tensor) and isinstance(params.asnumpy(), np.ndarray): params = np.mean(params.asnumpy()) return params def get_rank_info() -> Tuple[int, int]: """Get rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. """ rank_id = int(os.getenv('RANK_ID', '0')) rank_size = int(os.getenv('RANK_SIZE', '1')) return rank_id, rank_size def get_num_nodes_devices(rank_size: int) -> Tuple[int, int]: """Derive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. """ if rank_size in (2, 4, 8): num_nodes = 1 num_devices = rank_size else: num_nodes = rank_size // 8 num_devices = 8 return num_nodes, num_devices class Const: """Const.""" def __setattr__(self, key, value): if key in self.__dict__: raise PermissionError('Can not change const {0}.'.format(key)) if not key.isupper(): raise ValueError('Const name {0} is not all uppercase.'.format(key)) self.__dict__[key] = value def generate_rank_list(stdout_nodes: Union[List, Tuple], stdout_devices: Union[List, Tuple]): """Generate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. """ rank_list = [] for node in stdout_nodes: for device in stdout_devices: rank_list.append(8 * node + device) return rank_list def convert_nodes_devices_input(var: Union[List, Tuple, Dict[str, int], None], num: int) -> Union[List, Tuple]: """Convert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. """ if var is None: var = tuple(range(num)) elif isinstance(var, dict): var = tuple(range(var['start'], var['end'])) return var def str2bool(b): """String convert to Bool.""" if b.lower() in ["false"]: output = False elif b.lower() in ["true"]: output = True else: raise Exception("Invalid Bool Value") return output def count_params(net): """Count number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params """ total_params = [np.prod(param.shape) for param in net.trainable_params()] return sum(total_params) // 1000000 def try_sync_file(file_name): """If the file is still downloading, we need to wait before the file finished downloading""" if fcntl: with open(file_name, 'r') as fp: fcntl.flock(fp.fileno(), fcntl.LOCK_EX) def is_version_le(current_version, base_version): """ return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1, base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) <= int(y) return True def is_version_ge(current_version, base_version): """ return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1， base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) >= int(y) return True def parse_value(value): """ parse value from command line. handles with int, float, bool, string, list and dict. """ def isint(x): try: a = float(x) b = int(a) except (TypeError, ValueError): return False else: return a == b def isfloat(x): try: float(x) except (TypeError, ValueError): return False else: return True def isbool(x): return x in ["True", "False"] def isjson(x): try: json.loads(x) except json.decoder.JSONDecodeError: return False else: return True if isint(value): return int(value) if isfloat(value): return float(value) if isbool(value): return value == "True" if isjson(value): return json.loads(value) return value pengcheng.mind-mindformer/mindformers/tools/__init__.py000066400000000000000000000020451476650141400237110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Tools.""" from .cloud_adapter import * from .register import * from .logger import logger, StreamRedirector, AiLogFastStreamRedirect2File from .utils import PARALLEL_MODE, MODE, DEBUG_INFO_PATH, \ check_in_modelarts, str2bool, count_params, get_output_root_path, get_output_subpath __all__ = ['logger'] __all__.extend(cloud_adapter.__all__) __all__.extend(register.__all__) pengcheng.mind-mindformer/mindformers/tools/__pycache__/000077500000000000000000000000001476650141400240075ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/__pycache__/__init__.cpython-39.pyc000066400000000000000000000011451476650141400302000ustar00rootroot00000000000000a >cf%@srdZddlTddlTddlmZmZmZddlmZmZm Z m Z m Z m Z m Z mZdgZeejeejdS)zMindFormers Tools.)*)loggerStreamRedirectorAiLogFastStreamRedirect2File) PARALLEL_MODEMODEDEBUG_INFO_PATHcheck_in_modelartsstr2bool count_paramsget_output_root_pathget_output_subpathrN)__doc__ cloud_adapterregisterrrrutilsrrrr r r r r __all__extendrrE/home/ma-user/work/mindformers-baichuan/mindformers/tools/__init__.pys( pengcheng.mind-mindformer/mindformers/tools/__pycache__/download_tools.cpython-39.pyc000066400000000000000000000050141476650141400314670ustar00rootroot00000000000000a >cf @sdZddlZddlZddlZddlZddlmZddlmZz ddlZWn e yhdZe dYn0e GdddZ d d d Z dS) Zdownload_toolsN)tqdm)loggerz{The library fcntl is not found. This may cause the reading file failed when call the from_pretrained for different process.c@seZdZdZdZdS) StatusCodeN)__name__ __module__ __qualname____doc__succeedr r K/home/ma-user/work/mindformers-baichuan/mindformers/tools/download_tools.pyr!src CsLttdd}|ddkrs    pengcheng.mind-mindformer/mindformers/tools/__pycache__/image_tools.cpython-39.pyc000066400000000000000000000022421476650141400307420ustar00rootroot00000000000000a >cf@s:dZddlZddlZddlZddlZddlZdddZdS)z Image tools Nc Cst|tr|ds|drztj|d|d}WnDttjjtjj fyv}zt d|d|WYd}~n d}~00t j |j}n(tj|rt j |}nt|dt|tjr|St|t j j std t j|}|d }|S) z load imagezhttps://zhttp://T)streamtimeoutzConnect error, please download .NzI is not a valid path. If URL, it must start with `http://` or `https://`.z]Input should be an url linking to an image, a local path, a Mindspore Tensor, or a PIL image.RGB) isinstancestr startswithrequestsget TimeoutErrorurllib3 exceptions MaxRetryError ProxyErrorConnectionErrorPILImageopenrawospathisfile ValueErrormsTensorImageOpsexif_transposeconvert)contentrresponseexcr"H/home/ma-user/work/mindformers-baichuan/mindformers/tools/image_tools.py load_images.   (    r$)r)__doc__rr r r mindsporerr$r"r"r"r#s pengcheng.mind-mindformer/mindformers/tools/__pycache__/logger.cpython-39.pyc000066400000000000000000000365521476650141400277320ustar00rootroot00000000000000a >cfI @sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z m Z mZmZddlmZmZmZmZmZmZmZmZgZdZdZejed ed Zd Z d Z!d Z"dZ#d-e$e$ee ee e%e$fdfee ee e%e$fdfdddZ&GdddZ'Gddde'Z(d.e$e$e)ee ee e%e$fdfee ee e%e$fdfe)dddZ*e%dddZ+e%dddZ,e)ee edfee edfe%ddd Z-ee e%ee%fe%e%ee e%ee%fd!d"d#Z.e%e$d$d%d&Z/d/e%ej0d(d)d*Z1Gd+d,d,Z2e1Z3dS)0z LOGGER ModuleNwraps) TemporaryFile)DictListTupleUnion)get_num_nodes_devices get_rank_infoconvert_nodes_devices_inputgenerate_rank_listcheck_in_modelarts check_listLOCAL_DEFAULT_PATHget_output_root_pathDEBUGINFOWARNINGERRORCRITICALz/cache/ma-user-work/rlogzrank_{}/zO[%(levelname)s] %(asctime)s [%(pathname)s:%(lineno)d] %(funcName)s: %(message)szM%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)sz mindspore.logrank_id rank_sizeredirect_nodesredirect_devicescCsnd}|dkrj|dus|durjt|\}}t||}t||}td||td||t||}||vrjd}|S)aDetermine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. TNnodesdevicesFr r rr )rrrr is_redirect num_nodes num_devices rank_listr%C/home/ma-user/work/mindformers-baichuan/mindformers/tools/logger.pyjudge_redirect-s      r'csHeZdZdZfddZddZddZdd Zd d Zd d Z Z S)StreamRedirectorzStream Re-director for Log.cs0tt|||_||_t|j|_dS)zRedirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. N) superr(__init__ source_stream target_streamosdupfilenosave_source_stream_fd)selfr+r, __class__r%r&r*OszStreamRedirector.__init__cstfdd}|S)Ncs"|i|dSN)startstop)argskwargsfuncr1r%r&wrapper_sz*StreamRedirector.__call__..wrapperrr1r:r;r%r9r&__call__]szStreamRedirector.__call__cCs |dSr4)r5r1r%r%r& __enter__gszStreamRedirector.__enter__cCs |dSr4)r6)r1exc_typeexc_valexc_tbr%r%r&__exit__jszStreamRedirector.__exit__cCs&|jt|j|jdS)zstart.N)r+flushr-dup2r,r/r>r%r%r&r5ms zStreamRedirector.startcCs,|jt|j|j|jdS)zstop.N)r+rDr-rEr0r/r,r>r%r%r&r6rs zStreamRedirector.stop) __name__ __module__ __qualname____doc__r*r=r?rCr5r6 __classcell__r%r%r2r&r(Ls  r(csjeZdZdZd eeeeee fdfeeeeee fdfdfdd Z fddZ fdd Z Z S) AiLogFastStreamRedirect2FilezAiLogFastStreamRedirect2File.N)rrc  st\}}t||||d|_|dd}|dd}|dd} |sTtjtd}|rpt |} tj|| }| sxt } tj|| |_ tj |j |_ |durtj}td d } tt|j|| d dS) aRedirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. r file_save_dirappend_rank_dirT file_namerNzw+)mode)r+r,)r r'r!getr-pathjoinrRANK_DIR_FORMATTERformatDEFAULT_REDIRECT_FILE_NAME file_pathdirnamerLsysstderrrr)rKr*) r1r+rrr8rrrLrNrOrank_strr,r2r%r&r*{s,      z%AiLogFastStreamRedirect2File.__init__cs|jrtt|dSr4)r!r)rKr5r>r2r%r&r5sz"AiLogFastStreamRedirect2File.startcs|jr|jtj|js.tj|jdd|jddtj tj B}t j t j B}tt|j||d&}|jD]}||qtWdn1s0Ytt||jdS)NTexist_okrw)r!r,rDr-rRexistsrLmakedirsseekO_WRONLYO_CREATstatS_IWUSRS_IRUSRfdopenopenrWwriter)rKr6close)r1flagsmodesfpliner2r%r&r6s    *z!AiLogFastStreamRedirect2File.stop)NNN)rFrGrHrIrrrrstrintr*r5r6rJr%r%r2r&rKys6 rK)rr is_outputrrreturnc Csn|rj|dkrj|dus|durjt|\}}t||}t||}td||td||t||} || vrjd}|S)a'Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. rNrrFr ) rrrqrrr"r# stdout_nodesstdout_devicesr$r%r%r& judge_stdouts      ru)var_namecCsT|dus&t|tttfs&td|t|ttfrP|D]}t|ts8tdq8dS)zCheck the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None NzDThe value of {} can be None or a value of type tuple, list, or dict.zEThe elements of a variable of type list or tuple must be of type int.) isinstancelisttupledict TypeErrorrUrp)rvvaritemr%r%r&validate_nodes_devices_inputs  r~cCs6t|tstd||tvr2td||tdS)zVerify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None z%The format of {} must be of type str.z{}={} needs to be in {}N)rwror{rULEVEL ValueError)rvr|r%r%r&validate_levels rto_stdrsrt stdout_levelcCs4t|tstdtd|td|td|dS)z;Validate the input about stdout of the get_logger function.z.The format of the to_std must be of type bool.rsrtrN)rwboolr{r~rrr%r%r&validate_std_input_formats    r) file_levelrLrNrOcCst|ttfstd|D]}td|qt|t|ksBtdt|tsTtdt|tsftdt|ttfs|td|D]}t|tstdqdS) z9Validate the input about file of the get_logger function.z2The value of file_level should be list or a tuple.zlevel in file_levelz7The length of file_level and file_name should be equal.z9The value of file_save_dir should be a value of type str.z\}}tjj||||d}| |||||q| t dd|_ t||S)aGet the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. rTrsNcs~tdrNtd}|dr4|dr4|dd}ttdd|d}n,trnd t dd f}n d d}|S) NZSTDOUT_DEVICES)([))]rcSs t|Sr4)rpstripxr%r%r&qz8get_logger..get_stdout_devices..,rt) r-getenv startswithendswithrymapsplitr rQmin)rr8rr%r&get_stdout_devicesls    z&get_logger..get_stdout_devicesrr stdout_formatrMr)rrrLrNrOinfo.log error.log max_file_size2max_num_of_files)rrrqrrrr\i)filenamemaxBytes backupCountF)rr)!r getLogger logger_listrQr rrruDEFAULT_STDOUT_FORMAT StreamHandlerrYstdoutsetLevelr Formatter setFormatter addHandlerappendr-rRrSrrTrUrealpathrXr_r`DEFAULT_FILEHANDLER_FORMAT enumeratehandlersRotatingFileHandler propagate)rr8Z mf_loggerrrsrrrtrrrrLrNrOrrstream_handlerZstream_formatterZ logging_levelrZrank_dirr[rWrrRbase_dirZfile_formatteri file_handlerr%rr& get_loggerAs~"                          rc@s(eZdZdZeZdddZddZdS) _LogActionOncez A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. FcCs||_||_||_dSr4)loggerkey no_warning)r1Zm_loggerrrr%r%r&r*sz_LogActionOnce.__init__csfdd}|S)Ncsttjds|i|Sjj}js4jtjvrBddj_n dtjj<|i|}tjdrp|j_|S)NwarningcSs|Sr4r%rr%r%r&rrz:_LogActionOnce.__call__..wrapper..T)hasattrrrrrr is_logged)r7r8Zold_funcresr9r%r&r;s   z(_LogActionOnce.__call__..wrapperr%r<r%r9r&r=sz_LogActionOnce.__call__N)F)rFrGrHrIrzrr*r=r%r%r%r&rs r)NN)NN)r)4rIrZlogging.configlogging.handlersr-rdrY functoolsrtempfilertypingrrrrmindformers.tools.utilsr r r r r rrrrrZMODELARTS_LOG_FILE_DIRrRrSrZLOCAL_DEFAULT_LOG_FILE_DIRrTrrrVrpror'r(rKrrur~rrrrLoggerrrrr%r%r%r&sb  (  -N !  z"pengcheng.mind-mindformer/mindformers/tools/__pycache__/transform_ckpt.cpython-39.pyc000066400000000000000000000035411476650141400314770ustar00rootroot00000000000000a ?cf @sFdZddlZddlZddlZdddZedkrBeZej dddd ej d dd d ej d de d dej dde ddej dde dde Z ee j Z ee jZe jZe jZe jZeje jsJe jdede edeedeedeedeedeeeee eeddS)ztransform ckptNcCs|sdStj|s"J|dtj|r2|Stj|r|rZtj|d|d}ntj|d}tj|r~t|t|||SdS)zMerge strategy if strategy path is dir Args: startegy_path (str): The path of stategy. rank_id (int): The rank id of device. Returns: None or strategy path N not found!Zmerged_ckpt_strategy_z.ckptzmerged_ckpt_strategy.ckpt) ospathexistsisfileisdirjoinremovemsmerge_pipeline_strategys)Z startegy_pathrank_idZ merge_pathr K/home/ma-user/work/mindformers-baichuan/mindformers/tools/transform_ckpt.py get_strategys      r__main__z--src_ckpt_strategyzpath of src ckpt strategy)defaulthelpz--dst_ckpt_strategyzpath of dst ckpt strategyz--src_ckpt_dirzpath of src ckpt)rtyperz--dst_ckpt_dirzpath where to save dst ckptz--prefix checkpoint_z prefix of transformed checkpointrzsrc_ckpt_strategy: zdst_ckpt_strategy: zsrc_ckpt_dir: zdst_ckpt_dir: zprefix: z......Start transform......z......Transform succeed!......)N)__doc__rargparse mindsporer r__name__ArgumentParserparser add_argumentstr parse_argsargssrc_ckpt_strategydst_ckpt_strategyZ src_ckpt_dirZ dst_ckpt_dirprefixrrprinttransform_checkpointsr r r rsX   pengcheng.mind-mindformer/mindformers/tools/__pycache__/utils.cpython-39.pyc000066400000000000000000000252471476650141400276120ustar00rootroot00000000000000a ?cf;, @s*dZddlZddlZddlmZmZmZmZddlm Z z ddl Z Wne yZdZ Yn0ddl Z ddlmZddlmZejjejjejjejjejjejjejjejjejjd ZejejejejdZd Zd Zd Zd Zd Ze ddZ!dZ"dZ#dZ$ddZ%GdddZ&ddZ'e(eeefe)dddZ*ddZ+ddZ,d d!Z-dEd#d$Z.d%d&Z/d'd(Z0d)d*Z1ee)e)fd+d,d-Z2e)ee)e)fd.d/d0Z3Gd1d2d2Z4eeefeeefd3d4d5Z5eeeee(e)fdfe)eeefd6d7d8Z6d9d:Z7d;d<Z8d=d>Z9d?d@Z:dAdBZ;dCdDZ.wrapperN) Exception)rArBerr@r sync_transzs  rEcCstr tStS)z&get default output path in local/AICC.)rMA_OUTPUT_ROOTrrrrrget_output_root_pathsrGTcCs@t|tt}tj||}|r8tj|d|}t|S)z+get output store path for sub output class.zrank_{}) rr"r0rGrr8joinrr:) sub_classrank_idZ append_rank root_path directoryrrrget_output_subpaths  rMcCst|tjd|dSNZREMOTE_SAVE_URL)r1rr setdefault)remote_save_urlrrrset_remote_save_urlsrQcCstjddSrN)rrgetrrrrget_remote_save_urlsrScCsft|ttfr:t|dtrbt|dtjrb|d}n(t|trbt|tjrbt|}|S)zGet network outputs.r)rtuplelistrasnumpynpndarraymean)paramsrrrget_net_outputss " r[)returncCs(ttdd}ttdd}||fS)zGet rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. RANK_ID0 RANK_SIZE1)intrgetenv)rJ rank_sizerrr get_rank_infosrd)rcr\cCs&|dvrd}|}n |d}d}||fS)zDerive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. )rrrfr)rc num_nodes num_devicesrrrget_num_nodes_devicess ric@seZdZdZddZdS)ConstzConst.cCs<||jvrtd||s.td|||j|<dS)NzCan not change const {0}.z$Const name {0} is not all uppercase.)__dict__PermissionErrorrisupperr5)selfkeyr6rrr __setattr__s  zConst.__setattr__N)r+r,r-r.rprrrrrjsrj) stdout_nodesstdout_devicescCs.g}|D] }|D]}|d||qq|S)alGenerate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. rf)append)rqrr rank_listnodedevicerrrgenerate_rank_lists rw)varr4r\cCs:|durtt|}n t|tr6tt|d|d}|S)aJConvert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. Nr>end)rTrangerdict)rxr4rrrconvert_nodes_devices_inputs  r|cCs0|dvrd}n|dvr$d}ntd|S)zString convert to Bool.)falseF)trueTzInvalid Bool Value)lowerrC)boutputrrrstr2bools   rcCsdd|D}t|dS)zCount number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params cSsg|]}t|jqSr)rWprodshape).0paramrrr z count_params..i@B)trainable_paramssum)net total_paramsrrr count_params srcCsDtr@t|d"}t|tjWdn1s60YdS)zVIf the file is still downloading, we need to wait before the file finished downloadingrN)fcntlopenflockfilenoLOCK_EX) file_namefprrr try_sync_files rcCsvd}||vs||vrtdt||||D]>\}}|r2|sLq2t|t|kr2t|t|kSq2dS)z return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. .zaThe version string will contain the `.`.For example, current_version 1.8.1, base_version: 1.11.0.Tr5zipsplitisdigitraZcurrent_version base_versionZversion_split_charxyrrr is_version_lesrcCsvd}||vs||vrtdt||||D]>\}}|r2|sLq2t|t|kr2t|t|kSq2dS)z return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. rucThe version string will contain the `.`.For example, current_version 1.8.1， base_version: 1.11.0.Trrrrr is_version_ge-srcCsfdd}dd}dd}dd}||r0t|S||r@t|S||rP|d kS||rbt|S|S) zj parse value from command line. handles with int, float, bool, string, list and dict. c Ss:zt|}t|}Wnttfy,YdS0||kSdS)NF)floatrarr5)rarrrrisintDs  zparse_value..isintc Ss.z t|Wnttfy$YdS0dSdSNFT)rrr5rrrrisfloatMs  zparse_value..isfloatcSs|dvS)N)TrueFalserrrrrisboolUszparse_value..isboolcSs0zt|Wntjjy&YdS0dSdSr)jsonloadsdecoderJSONDecodeErrorrrrrisjsonXs zparse_value..isjsonr)rarrr)r6rrrrrrr parse_value?s  r)rT)=r.rrtypingrrrrmultiprocessingrr ImportErrornumpyrW mindsporerr ParallelModer r r r r PARALLEL_MODErrMODEZSAVE_WORK_PATHrFDEBUG_INFO_PATHZPROFILE_INFO_PATHZ PLOG_PATHrbrZLAST_TRANSFORM_LOCK_PATHr&r'rrr1r0rar7r:rErGrMrQrSr[rdrirjrwr|rrrrrrrrrrsr              .  pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/000077500000000000000000000000001476650141400244055ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/.ipynb_checkpoints/000077500000000000000000000000001476650141400301765ustar00rootroot00000000000000cloud_adapter-checkpoint.py000066400000000000000000000227401476650141400354310ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/.ipynb_checkpoints# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cloud Adapter.""" import os import time from mindspore.train.callback import Callback from ..logger import logger from ..utils import check_obs_url, check_in_modelarts, \ Validator, sync_trans if check_in_modelarts(): import moxing as mox __all__ = ['Local2ObsMonitor', 'Obs2Local', 'mox_adapter'] class Local2ObsMonitor(Callback): """File saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, src_dir, target_dir, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last=True, retry=3, retry_time=5, log=logger): super(Local2ObsMonitor, self).__init__() self.src_dir = src_dir self.target_dir = target_dir self.step_upload_frequence = step_upload_frequence self.epoch_upload_frequence = epoch_upload_frequence self.keep_last = keep_last self.rank_id = int(os.getenv('RANK_ID', '0')) self.retry_time = retry_time self.retry = retry self.log = log self.cb_params = None self.pro = None self.on_modelarts = check_in_modelarts() if self.on_modelarts: check_obs_url(target_dir) def step_end(self, run_context): """upload files at the end of step.""" if not self.on_modelarts: return if self.step_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_step_num % self.step_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def epoch_end(self, run_context): """upload files at the end of epoch.""" if not self.on_modelarts: return if self.epoch_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_epoch_num % self.epoch_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload(True) print(f"finish uploading") def upload(self, last_epoch=False): """Upload Files to OBS.""" if self.pro: # wait for last upload done self.pro.join() if last_epoch: self.sync2obs_synchronize(self.src_dir, self.target_dir) else: self.pro = self.sync2obs(self.src_dir, self.target_dir) @sync_trans def sync2obs(self, src_dir, target_dir): """Asynchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) def sync2obs_synchronize(self, src_dir, target_dir): """synchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) class Obs2Local: """File saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, rank_id=0, retry=3, retry_time=5, log=logger): self.rank_id = int(rank_id) self.retry_time = retry_time self.retry = retry self.log = log def obs2local(self, obs_url, local_url, special_id=None): """Pull Obs Files to Local.""" obs_name = obs_url.split("/")[-1] mox_lock = os.path.join(local_url, "mox_copy_{}.lock".format(obs_name)) local_url = os.path.join(local_url, obs_name) if special_id is None: if self.rank_id % 8 == 0: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) else: Validator.check_type(special_id, int) if self.rank_id == special_id: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) return local_url def mox_adapter(src_dir, target_dir, retry=3, retry_time=5, log=logger): """File interaction with Moxing.""" success = False for i in range(retry + 1): start = time.time() try: mox.file.copy_parallel(src_url=src_dir, dst_url=target_dir, threads=0, is_processing=False) except (FileNotFoundError, RuntimeError) as e: log.info("%s, from %s download to %s failed, will retry(%d) again.", e, src_dir, target_dir, i) # sleep due to restriction of obs log.info("sleep time %d for waiting download file from obs.", retry_time) continue end = time.time() if Validator.is_obs_url(target_dir): if mox.file.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break else: if os.path.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break return success cloud_monitor-checkpoint.py000066400000000000000000000072621476650141400355020ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/.ipynb_checkpoints# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Aicc monitor.""" import os import traceback import time from mindformers.tools.logger import get_logger from .cloud_adapter import mox_adapter from ..utils import DEBUG_INFO_PATH, PROFILE_INFO_PATH, PLOG_PATH, LAST_TRANSFORM_LOCK_PATH,\ get_output_root_path, get_remote_save_url, RANK_FINISH_PATH def cloud_monitor(log=get_logger()): """Aicc monitor for main function.""" def decorator(run_func): def wrapper(*args, **kwargs): local_id = int(os.getenv('RANK_ID', '0')) try: result = run_func(*args, **kwargs) except BaseException as exc: # pylint: disable=W0703 error = traceback.format_exc() log.error(error) raise exc finally: _last_transform(local_id, log) return result return wrapper return decorator def _last_transform(local_id, log=get_logger()): """Transform file when progress ending or except.""" local_node_id = local_id % 8 finish_file = RANK_FINISH_PATH.replace("*", str(local_node_id)) os.mknod(finish_file) world_size = int(os.getenv("RANK_SIZE", '1')) if world_size > 8: world_size = 8 if os.environ.get("SPECIAL_ID") and get_remote_save_url(): target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=get_remote_save_url(), log=log) else: if local_id % 8 == 0 and get_remote_save_url(): for i in range(world_size): finish_file = RANK_FINISH_PATH.replace("*", str(i)) while not os.path.exists(finish_file): time.sleep(1) target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=target_dir, log=log) if local_id % 8 == 0 and get_remote_save_url(): target_dir = get_remote_save_url() task_dir = os.path.join(target_dir, 'ascend-log') node = 'node_{}'.format(local_id) if os.path.exists(PLOG_PATH): mox_adapter(src_dir=PLOG_PATH, target_dir=os.path.join(task_dir, 'plog', node), log=log) if os.path.exists(DEBUG_INFO_PATH): mox_adapter(src_dir=DEBUG_INFO_PATH, target_dir=os.path.join(target_dir, 'debug_info', node), log=log) if os.path.exists(PROFILE_INFO_PATH): mox_adapter(src_dir=PROFILE_INFO_PATH, target_dir=os.path.join(target_dir, 'profile'), log=log) os.mknod(LAST_TRANSFORM_LOCK_PATH) elif get_remote_save_url(): log.info("Wait for the first card to complete the file and send it back to OBS: %s.", get_remote_save_url()) while True: if os.path.exists(LAST_TRANSFORM_LOCK_PATH): log.info("All files have been sent back to the OBS: %s," "and the process exits normally.", get_remote_save_url()) break def upload_log(): """Upload log according to runtime for FM.""" local_id = int(os.getenv('RANK_ID', '0')) _last_transform(local_id) pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/__init__.py000066400000000000000000000015121476650141400265150ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Cloud Adapter API.""" from .cloud_adapter import * from .cloud_monitor import cloud_monitor __all__ = ['cloud_monitor'] __all__.extend(cloud_adapter.__all__) pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/__pycache__/000077500000000000000000000000001476650141400266155ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/__pycache__/__init__.cpython-39.pyc000066400000000000000000000005131476650141400330040ustar00rootroot00000000000000a ?cfJ@s.dZddlTddlmZdgZeejdS)zMindFormers Cloud Adapter API.)*) cloud_monitorrN)__doc__ cloud_adapterr__all__extendrrS/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/__init__.pys pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/__pycache__/cloud_adapter.cpython-39.pyc000066400000000000000000000145041476650141400340600ustar00rootroot00000000000000a ?cfJ!@sdZddlZddlZddlmZddlmZddlmZmZm Z m Z erRddl Z gdZ Gdd d eZGd d d Zd d efddZdS)zCloud Adapter.N)Callback)logger) check_obs_urlcheck_in_modelarts Validator sync_trans)Local2ObsMonitor Obs2Local mox_adaptercsZeZdZdZdddddefeedfdd Zd d Zd d ZddZ e ddZ Z S)r aGFile saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. dT)step_upload_frequenceepoch_upload_frequencec svtt|||_||_||_||_||_tt dd|_ ||_ ||_ ||_d|_d|_t|_|jrrt|dS)NRANK_ID0)superr __init__src_dir target_dirrr keep_lastintosgetenvrank_id retry_timeretrylog cb_paramspror on_modelartsr) selfrrrrrrrr __class__X/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/cloud_adapter.pyr3s zLocal2ObsMonitor.__init__cCsX|js dS|jdkrdS||_|jj|jdkrTt|jrT|j d| dS)z upload files at the end of step.Nr#Starting upload output file to obs!) r"r original_argsr cur_step_numrlistdirrrinfouploadr# run_contextr&r&r'step_endLs   zLocal2ObsMonitor.step_endcCsX|js dS|jdkrdS||_|jj|jdkrTt|jrT|j d| dS)z!upload files at the end of epoch.Nrr() r"rr)r cur_epoch_numrr+rrr,r-r.r&r&r' epoch_endWs   zLocal2ObsMonitor.epoch_endcCs&|jr|j||j|j|_dS)zUpload Files to OBS.N)r!joinsync2obsrr)r#r&r&r'r-bs zLocal2ObsMonitor.uploadcCst|}|D]}d|vrtj||d|j}tj||d|j}|jrltj|rltjj |ddt |||j |j |j q|jddkrt tj||tj|||j |j |j qdS)zAsynchronous transfer to OBS. checkpointrank_T) recursiverN)rr+pathr3rrmoxfileexistsremover rrr)r#rrZ sub_dir_listsub_dirZ scr_ckpt_dirZtarget_ckpt_dirr&r&r'r4is   zLocal2ObsMonitor.sync2obs) __name__ __module__ __qualname____doc__rrrr0r2r-rr4 __classcell__r&r&r$r'r s   r c@s,eZdZdZdddefddZd dd ZdS) r aFile saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. rrrcCs t||_||_||_||_dS)N)rrrrr)r#rrrrr&r&r'rs zObs2Local.__init__NcCs(|dd}tj|d|}tj||}|dur|jddkrt|||j|j|j zt |Wqt y|Yq0n&|j dtj |rqtdqn|t|t|j|krt|||j|j|j zt |Wnt yYn0n,|j dtj |rq$tdq|S) zPull Obs Files to Local./r zmox_copy_{}.lockNr8rz>programming sleep for waiting download file from obs to local.)splitrr9r3formatrr rrrmknodIOErrorr,r<timesleepr check_typer)r#Zobs_urlZ local_urlZ special_idZobs_nameZmox_lockr&r&r' obs2locals4       zObs2Local.obs2local)N)r?r@rArBrrrMr&r&r&r'r |s r rrc Csd}t|dD]}t}ztjj||dddWnRttfy}z6|d|||||d|WYd}~qWYd}~n d}~00t} t |rtj |rd}|d || |qqt j |rd}|d || |qq|S) zFile interaction with Moxing.FrEr)Zsrc_urlZdst_urlthreadsZ is_processingz8%s, from %s download to %s failed, will retry(%d) again.z1sleep time %d for waiting download file from obs.NTz(Pull/Push file %s success, cost time: %f) rangerJr:r;Z copy_parallelFileNotFoundError RuntimeErrorr,rZ is_obs_urlr<rr9) rrrrrsuccessistarteendr&r&r'r s, "   r )rBrrJmindspore.train.callbackrrutilsrrrrmoxingr:__all__r r r r&r&r&r's  \6pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/__pycache__/cloud_monitor.cpython-39.pyc000066400000000000000000000044741476650141400341340ustar00rootroot00000000000000a ?cf @stdZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z efdd Zefd d Zd d ZdS)z Aicc monitor.N) get_logger) mox_adapter)DEBUG_INFO_PATHPROFILE_INFO_PATH PLOG_PATHLAST_TRANSFORM_LOCK_PATHget_output_root_pathget_remote_save_urlcsfdd}|S)zAicc monitor for main function.csfdd}|S)Nc s~ttdd}z\z|i|}Wn:ty^}z"t}||WYd}~n d}~00Wt|n t|0|S)NRANK_ID0)intosgetenv BaseException traceback format_excerror_last_transform)argskwargslocal_idresultexcr)logrun_funcX/home/ma-user/work/mindformers-baichuan/mindformers/tools/cloud_adapter/cloud_monitor.pywrappers z1cloud_monitor..decorator..wrapperr)rrr)rr decorators z cloud_monitor..decoratorr)rr!rr r cloud_monitors r"cCsBtjdr,tr,t}ttt|dn(|ddkrTtrTt}tt||d|ddkrtrt}tj|d}d|}tj t rtt tj|d||dtj t rtt tj|d||dtj t rtt tj|d |dt tn:tr>|d ttj tr|d tq>qd S) z.Transform file when progress ending or except.Z SPECIAL_ID)src_dir target_dirrrz ascend-logznode_{}Zplog debug_infoprofilezIWait for the first card to complete the file and send it back to OBS: %s.zLAll files have been sent back to the OBS: %s,and the process exits normally.N)renvirongetr rr pathjoinformatexistsrrrmknodr info)rrr$Ztask_dirnoderrrr.s8     rcCsttdd}t|dS)z'Upload log according to runtime for FM.r r N)rrrr)rrrr upload_logMsr1)__doc__rrmindformers.tools.loggerr cloud_adapterrutilsrrrr r r r"rr1rrrrs   pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/cloud_adapter.py000066400000000000000000000205121476650141400275650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Cloud Adapter.""" import os import time from mindspore.train.callback import Callback from ..logger import logger from ..utils import check_obs_url, check_in_modelarts, \ Validator, sync_trans if check_in_modelarts(): import moxing as mox __all__ = ['Local2ObsMonitor', 'Obs2Local', 'mox_adapter'] class Local2ObsMonitor(Callback): """File saved from local system to OBS server. Args: src_dir (str): Local system path, which means path of AI computing center platform. target_dir (str): OBS path starting with S3 or obs is used to save files. rank_id (int): the device's contents will be saved according to the actual rank_id. Default: None, means only the contents of the first device of each node are saved. upload_frequence (int): How often files are saved in AI computing center platform. Default: -1. keep_last (bool): Check whether files in the OBS are consistent with AI computing center platform. Default: True, means old file will be removed. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, src_dir, target_dir, step_upload_frequence: int = 100, epoch_upload_frequence: int = -1, keep_last=True, retry=3, retry_time=5, log=logger): super(Local2ObsMonitor, self).__init__() self.src_dir = src_dir self.target_dir = target_dir self.step_upload_frequence = step_upload_frequence self.epoch_upload_frequence = epoch_upload_frequence self.keep_last = keep_last self.rank_id = int(os.getenv('RANK_ID', '0')) self.retry_time = retry_time self.retry = retry self.log = log self.cb_params = None self.pro = None self.on_modelarts = check_in_modelarts() if self.on_modelarts: check_obs_url(target_dir) def step_end(self, run_context): """upload files at the end of step.""" if not self.on_modelarts: return if self.step_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_step_num % self.step_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def epoch_end(self, run_context): """upload files at the end of epoch.""" if not self.on_modelarts: return if self.epoch_upload_frequence <= 0: return self.cb_params = run_context.original_args() if self.cb_params.cur_epoch_num % self.epoch_upload_frequence == 0 and os.listdir(self.src_dir): self.log.info("Starting upload output file to obs!") self.upload() def upload(self): """Upload Files to OBS.""" if self.pro: # wait for last upload done self.pro.join() self.pro = self.sync2obs(self.src_dir, self.target_dir) @sync_trans def sync2obs(self, src_dir, target_dir): """Asynchronous transfer to OBS.""" sub_dir_list = os.listdir(src_dir) for sub_dir in sub_dir_list: if 'checkpoint' in sub_dir: # for checkpoint, all processes upload its own checkpoint scr_ckpt_dir = os.path.join(src_dir, sub_dir, f"rank_{self.rank_id}") target_ckpt_dir = os.path.join(target_dir, sub_dir, f"rank_{self.rank_id}") if self.keep_last and mox.file.exists(target_ckpt_dir): mox.file.remove(target_ckpt_dir, recursive=True) mox_adapter(scr_ckpt_dir, target_ckpt_dir, self.retry, self.retry_time, self.log) elif self.rank_id % 8 == 0: # for other outputs, one rank upload all mox_adapter(os.path.join(src_dir, sub_dir), os.path.join(target_dir, sub_dir), self.retry, self.retry_time, self.log) class Obs2Local: """File saved from OBS server to local system of AI computing center platform. Args: rank_id (int): The obs's contents will be upload according to the actual rank_id value. Default: 0, means stored only one OBS file each node. retry (int): The number of attempts to save again if the first attempt fails. Default: 3, will be try three times. retry_time: The time of resaving the previously dormant program, after each attempt fails. Default: 5, will sleep five seconds. log (logger): Use the log system to print information. Default: logging class for Python. """ def __init__(self, rank_id=0, retry=3, retry_time=5, log=logger): self.rank_id = int(rank_id) self.retry_time = retry_time self.retry = retry self.log = log def obs2local(self, obs_url, local_url, special_id=None): """Pull Obs Files to Local.""" obs_name = obs_url.split("/")[-1] mox_lock = os.path.join(local_url, "mox_copy_{}.lock".format(obs_name)) local_url = os.path.join(local_url, obs_name) if special_id is None: if self.rank_id % 8 == 0: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) else: Validator.check_type(special_id, int) if self.rank_id == special_id: mox_adapter(obs_url, local_url, self.retry, self.retry_time, self.log) try: os.mknod(mox_lock) except IOError: pass else: self.log.info("programming sleep for waiting download file from obs to local.") while True: if os.path.exists(mox_lock): break time.sleep(1) return local_url def mox_adapter(src_dir, target_dir, retry=3, retry_time=5, log=logger): """File interaction with Moxing.""" success = False for i in range(retry + 1): start = time.time() try: mox.file.copy_parallel(src_url=src_dir, dst_url=target_dir, threads=0, is_processing=False) except (FileNotFoundError, RuntimeError) as e: log.info("%s, from %s download to %s failed, will retry(%d) again.", e, src_dir, target_dir, i) # sleep due to restriction of obs log.info("sleep time %d for waiting download file from obs.", retry_time) continue end = time.time() if Validator.is_obs_url(target_dir): if mox.file.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break else: if os.path.exists(target_dir): success = True log.info("Pull/Push file %s success, cost time: %f", target_dir, end - start) break return success pengcheng.mind-mindformer/mindformers/tools/cloud_adapter/cloud_monitor.py000066400000000000000000000072621476650141400276430ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Aicc monitor.""" import os import traceback import time from mindformers.tools.logger import get_logger from .cloud_adapter import mox_adapter from ..utils import DEBUG_INFO_PATH, PROFILE_INFO_PATH, PLOG_PATH, LAST_TRANSFORM_LOCK_PATH,\ get_output_root_path, get_remote_save_url, RANK_FINISH_PATH def cloud_monitor(log=get_logger()): """Aicc monitor for main function.""" def decorator(run_func): def wrapper(*args, **kwargs): local_id = int(os.getenv('RANK_ID', '0')) try: result = run_func(*args, **kwargs) except BaseException as exc: # pylint: disable=W0703 error = traceback.format_exc() log.error(error) raise exc finally: _last_transform(local_id, log) return result return wrapper return decorator def _last_transform(local_id, log=get_logger()): """Transform file when progress ending or except.""" local_node_id = local_id % 8 finish_file = RANK_FINISH_PATH.replace("*", str(local_node_id)) os.mknod(finish_file) world_size = int(os.getenv("RANK_SIZE", '1')) if world_size > 8: world_size = 8 if os.environ.get("SPECIAL_ID") and get_remote_save_url(): target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=get_remote_save_url(), log=log) else: if local_id % 8 == 0 and get_remote_save_url(): for i in range(world_size): finish_file = RANK_FINISH_PATH.replace("*", str(i)) while not os.path.exists(finish_file): time.sleep(1) target_dir = get_remote_save_url() mox_adapter(src_dir=get_output_root_path(), target_dir=target_dir, log=log) if local_id % 8 == 0 and get_remote_save_url(): target_dir = get_remote_save_url() task_dir = os.path.join(target_dir, 'ascend-log') node = 'node_{}'.format(local_id) if os.path.exists(PLOG_PATH): mox_adapter(src_dir=PLOG_PATH, target_dir=os.path.join(task_dir, 'plog', node), log=log) if os.path.exists(DEBUG_INFO_PATH): mox_adapter(src_dir=DEBUG_INFO_PATH, target_dir=os.path.join(target_dir, 'debug_info', node), log=log) if os.path.exists(PROFILE_INFO_PATH): mox_adapter(src_dir=PROFILE_INFO_PATH, target_dir=os.path.join(target_dir, 'profile'), log=log) os.mknod(LAST_TRANSFORM_LOCK_PATH) elif get_remote_save_url(): log.info("Wait for the first card to complete the file and send it back to OBS: %s.", get_remote_save_url()) while True: if os.path.exists(LAST_TRANSFORM_LOCK_PATH): log.info("All files have been sent back to the OBS: %s," "and the process exits normally.", get_remote_save_url()) break def upload_log(): """Upload log according to runtime for FM.""" local_id = int(os.getenv('RANK_ID', '0')) _last_transform(local_id) pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/000077500000000000000000000000001476650141400254715ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/bloom/000077500000000000000000000000001476650141400266015ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/bloom/make_mindrecord.py000066400000000000000000000112321476650141400322750ustar00rootroot00000000000000# Copyright 2020 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate mindrecord script""" import os import argparse import collections import json import numpy as np from tqdm import tqdm from mindspore.mindrecord import FileWriter from mindformers import AutoTokenizer class AlpacaDatasetMaker: """ AlpacaDatasetMaker """ def __init__(self, input_dataset_file, output_dataset_file, seq_length, for_finetune): self.input_dataset_file = input_dataset_file self.output_dataset_file = output_dataset_file self.seq_length = seq_length self.for_finetune = for_finetune self.tokenizer = AutoTokenizer.from_pretrained("bloom_560m") self.eos_token_id = self.tokenizer.eos_token_id self.pad_token_id = self.tokenizer.pad_token_id self.prompt_with_input = ( "Below is an instruction that describes a task, paired with an input that provides further context. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n" ) self.prompt_without_input = ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:\n" ) def make(self, num_of_prompts): """make mindrecord""" writer = FileWriter(self.output_dataset_file, 1) writer.add_schema( {"input_ids": {"type": "int64", "shape": [-1]}}, 'lm-schema') with open(self.input_dataset_file) as ds: dataset = json.load(ds) num_of_prompts = min(num_of_prompts, len(dataset)) if num_of_prompts > 0 else len(dataset) for i in tqdm(range(num_of_prompts)): prompt = dataset[i] prompt_ids = self.make_prompt_ids(prompt) features = collections.OrderedDict() features["input_ids"] = np.asarray(prompt_ids) writer.write_raw_data([features]) writer.commit() def make_prompt_ids(self, prompt): """make prompt dict into ids""" if 'input' in prompt: prompt_text = self.prompt_with_input.format_map(prompt) else: prompt_text = self.prompt_without_input.format_map(prompt) # question prompt_ids = self.tokenizer(prompt_text)["input_ids"] if self.for_finetune: prompt_ids = list(-np.array(prompt_ids, dtype=np.int32)) # answer output_text = prompt["output"] if output_text: output_ids = self.tokenizer(output_text)["input_ids"] else: output_ids = [] # Q+A ids = prompt_ids + output_ids if len(ids) > self.seq_length: ids = ids[:self.seq_length] self._add_eos_token_id(ids) self._add_pad_token_id(ids) return ids def _add_eos_token_id(self, ids): if len(ids) < self.seq_length: ids.append(self.eos_token_id) def _add_pad_token_id(self, ids): if len(ids) < self.seq_length: # pad eos instead of pad ids += [self.eos_token_id] * (self.seq_length - len(ids)) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--input_dataset_file", type=str, default="/home/work/czh/data/alpaca_data.json") parser.add_argument("--output_path", type=str, default="/home/work/czh/data/alpaca_2049/") parser.add_argument("--seq_length", type=int, default=2049) parser.add_argument("--N", type=int, default=-1) parser.add_argument("--for_finetune", type=int, default=1) args = parser.parse_args() if args.output_path and not os.path.exists(args.output_path): os.makedirs(args.output_path, exist_ok=True) args.output_dataset_file = os.path.join( args.output_path, "dataset.mindrecord") maker = AlpacaDatasetMaker( args.input_dataset_file, args.output_dataset_file, args.seq_length, args.for_finetune) maker.make(args.N) pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/glm/000077500000000000000000000000001476650141400262505ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/glm/adgen_dataset.py000066400000000000000000000241541476650141400314130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Process of ADGEN dataset.""" import argparse import json import numpy as np from tqdm import tqdm from mindspore.mindrecord import FileWriter from mindformers.models.glm.chatglm_6b_tokenizer import ChatGLMTokenizer from mindformers.tools.logger import logger prompt_column = "content" response_column = "summary" history_column = None prefix = "" def get_masks(input_ids, bos_token_id=130004): """Get attention mask.""" seq_length = input_ids.shape[0] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] attention_mask = np.tril(np.ones((seq_length, seq_length), dtype=np.float32)) for _, context_length in enumerate(context_lengths): attention_mask[:, :context_length] = 1 attention_mask = np.logical_not(attention_mask.astype(np.bool_)) attention_mask = attention_mask.astype(np.float32) attention_mask = np.expand_dims(attention_mask, 0) return attention_mask def get_position_ids(input_ids, mask_positions, use_gmasks=None, bos_token_id=130004, position_encoding_2d=True): """Get position ids.""" seq_length = input_ids.shape[0] if use_gmasks is None: use_gmasks = [False] mask = bos_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(input_ids, mask) # 要求input_ids中有且仅有一个bos_token_id context_lengths = np.argwhere(mask)[:, -1] if position_encoding_2d: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): position_ids[context_length:] = mask_positions[i] block_position_ids = [np.concatenate(( np.zeros(context_length, dtype=np.int64), np.arange(seq_length - context_length, dtype=np.int64) + 1 )) for context_length in context_lengths] block_position_ids = np.stack(block_position_ids, axis=0).squeeze() position_ids = np.stack((position_ids, block_position_ids), axis=0) else: position_ids = np.arange(seq_length, dtype=np.int64) for i, context_length in enumerate(context_lengths): if not use_gmasks[i]: position_ids[context_length:] = mask_positions[i] return position_ids def create_position_ids(input_ids, gmask_token_id=130001): """Get position ids.""" seq_length = input_ids.shape[0] seqs = input_ids # 要求input_ids中, 每行有且仅有一个gMASK use_gmasks = gmask_token_id * np.ones(shape=(seq_length), dtype=np.int32) mask = np.equal(seqs, use_gmasks) mask_positions = np.argwhere(mask)[:, -1] position_ids = get_position_ids(input_ids, mask_positions=mask_positions, use_gmasks=use_gmasks) return position_ids def preprocess_function(input_file, vocab_file, output_file, num_splits, max_source_length, max_target_length, mode='train', ignore_pad_token_for_loss=True): """ Process data file to mindrecord data file. Args: input_file (str): Original data set file. vocab_file (str): Vocab file path. output_file (str): Output MindRecord file. num_splits (str): The MindRecord file will be split into the number of partition. max_source_length (int): The max sequence length. max_target_length (int): The max label sequence length. mode (str): Dataset use to train or test. Default: 'train'. ignore_pad_token_for_loss (bool): Whether ignore pad token for loss, default: True. Returns: MindDataset object """ if input_file is None: raise ValueError("Please enter a valid dataset path") if vocab_file is None: raise ValueError("Please enter a valid vocab file path") max_seq_length = max_source_length + max_target_length tokenizer = ChatGLMTokenizer(vocab_file=vocab_file) writer = FileWriter(output_file, num_splits, overwrite=True) if mode == 'train': data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}, "position_ids": {"type": "int32", "shape": [2, max_seq_length]}, "attention_mask": {"type": "int32", "shape": [1, max_seq_length, max_seq_length]} } else: data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}} writer.add_schema(data_schema, "lm-schema") total_written = 0 file = open(input_file, 'r') lines = file.readlines() examples = {} content_list = [] summary_list = [] for line in lines: content_list.append(json.loads(line)[prompt_column]) summary_list.append(json.loads(line)[response_column]) examples[prompt_column] = content_list examples[response_column] = summary_list model_inputs = {} for i in tqdm(range(len(examples[prompt_column])), ascii=True, ncols=120): if examples[prompt_column][i] and examples[response_column][i]: query, answer = examples[prompt_column][i], examples[response_column][i] if history_column is None: prompt = query else: prompt = "" history = examples[history_column][i] for turn_idx, (old_query, response) in enumerate(history): prompt += "[Round {}]\n问：{}\n答：{}\n".format(turn_idx, old_query, response) prompt += "[Round {}]\n问：{}\n答：".format(len(history), query) prompt = prefix + prompt if mode == 'train': prompt_ids = tokenizer.encode(text=prompt, add_special_tokens=False) answer_ids = tokenizer.encode(text=answer, add_special_tokens=False) if len(prompt_ids) > max_source_length - 1: prompt_ids = prompt_ids[: max_source_length - 1] if len(answer_ids) > max_target_length - 2: answer_ids = answer_ids[: max_target_length - 2] input_ids = tokenizer.build_inputs_with_special_tokens(prompt_ids, answer_ids) context_length = input_ids.index(tokenizer.bos_token_id) mask_position = context_length - 1 label = [-100] * context_length + input_ids[mask_position + 2:] # +1 for logits shift pad_len = max_seq_length - len(input_ids) input_ids = input_ids + [tokenizer.pad_token_id] * pad_len label = label + [tokenizer.pad_token_id] * (pad_len + 1) # +1 for logits shift if ignore_pad_token_for_loss: label = [(l if l != tokenizer.pad_token_id else -100) for l in label] position_ids = create_position_ids(np.array(input_ids)) attention_mask = get_masks(np.array(input_ids)) model_inputs["position_ids"] = np.array(position_ids) model_inputs["attention_mask"] = np.array(attention_mask) else: if len(prompt) > max_source_length - 2: prompt = prompt[: max_source_length - 2] if len(answer) > max_target_length - 2: answer = answer[: max_target_length - 2] input_ids = tokenizer.encode(text=prompt, add_special_tokens=True) label = tokenizer.encode(text=answer, add_special_tokens=True) pad_len = max_source_length - len(input_ids) input_ids = input_ids + [tokenizer.pad_token_id] * pad_len model_inputs["input_ids"] = np.array(input_ids) model_inputs["labels"] = np.array(label) writer.write_raw_data([model_inputs]) total_written += 1 writer.commit() logger.info(f"Wrote {total_written} total instances") if __name__ == "__main__": parser = argparse.ArgumentParser(description="OPT convert script") parser.add_argument('--input_file', type=str, required=True, default='./AdvertiseGen/train.json', help="The number of layers of the model to be converted.") parser.add_argument("--vocab_file", type=str, default='ice_text.model', required=True, help="The torch checkpoint path.") parser.add_argument("--output_file", type=str, required=True, default="adgen_train.mindrecord", help="The output mindspore checkpoint path.") parser.add_argument("--max_source_length", type=int, default=64, help="The max input sequence length") parser.add_argument("--max_target_length", type=int, default=64, help="The max label sequence length") parser.add_argument("--mode", type=str, default="train", help="Process train or eval dataset.") opt = parser.parse_args() preprocess_function( input_file=opt.input_file, vocab_file=opt.vocab_file, output_file=opt.output_file, num_splits=1, max_source_length=opt.max_source_length, max_target_length=opt.max_target_length, mode=opt.mode, ignore_pad_token_for_loss=True ) pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/gpt2/000077500000000000000000000000001476650141400263455ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/gpt2/txtcls_dataset_to_mindrecord.py000066400000000000000000000130411476650141400346540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """convert dataset to mindrecord""" from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse import collections import numpy as np from mindspore.mindrecord import FileWriter from mindformers.tools.logger import logger from mindformers import AutoTokenizer, MindFormerBook def create_instance(ds_name, tokenizer, text, max_length): """A single sample instance for LM task.""" if ds_name == 'cola': _, label, _, sentence = text.strip().split("\t") elif ds_name == 'ag_news': label, title, description = text.strip().split("\",\"") sentence = title + ". " + description label = label.strip("\"") sentence = sentence.strip("\"") label = int(label) - 1 elif ds_name == 'imdb': sentence = text.strip()[:-9] label = text.strip()[-8:] sentence = sentence.strip("\"") label = 1 if label == 'positive' else 0 else: sentence, label = text.strip().split("\t") output = tokenizer(sentence, padding='max_length', max_length=max_length) output['labels'] = label return output def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(label) writer.write_raw_data([features]) return features def main(): dataset_support_list = ['cola', 'sst_2', 'ag_news', 'imdb'] tokenizer_support_list = list(MindFormerBook.get_tokenizer_url_support_list().keys()) parser = argparse.ArgumentParser() parser.add_argument("--dataset_name", type=str, required=True, help='Dataset name. Now only supports [cola, sst_2, ag_news, imdb]. ', choices=dataset_support_list) parser.add_argument("--tokenizer_type", type=str, default="gpt2", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ", choices=tokenizer_support_list) parser.add_argument("--data_columns", type=list, default=["input_ids", "labels", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") parser.add_argument("--input_file", type=str, required=True, help='Input raw text file. ') parser.add_argument("--output_file", type=str, required=True, help='Output MindRecord file. ') parser.add_argument("--max_length", type=int, default=1024, help='Maximum sequence length. ') parser.add_argument("--header", type=bool, default=True, help='Has header or not. ') args = parser.parse_args() tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) input_file = args.input_file logger.info("***** Reading from input files *****") logger.info("Input File: %s", input_file) output_file = args.output_file logger.info("***** Writing to output files *****") logger.info("Output File: %s", output_file) writer = FileWriter(output_file) data_schema = {"input_ids": {"type": "int64", "shape": [-1]}, "attention_mask": {"type": "int64", "shape": [-1]}, "labels": {"type": "int64", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema) total_written = 0 total_read = 0 logger.info("***** Reading from %s *****", input_file) with open(input_file, "r") as f: if args.header: f.readline() while True: line = f.readline() if not line: break total_read += 1 if total_read % 500 == 0: logger.info("%d ...", total_read) output = create_instance(args.dataset_name, tokenizer, line, args.max_length) features = write_instance_to_file(writer, instance=output) total_written += 1 if total_written <= 20: logger.info("***** Example *****") logger.info("input tokens: %s", tokenizer.decode(output["input_ids"])) logger.info("label: %s", output["labels"]) for feature_name in features.keys(): feature = features[feature_name] logger.info("%s: %s", feature_name, feature) writer.commit() logger.info("Wrote %d total instances", total_written) if __name__ == "__main__": main() pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/gpt2/wikitext2_data_process.py000066400000000000000000000204101476650141400333750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Data process for wikitext-2 Dataset The generated dataset has output columns: ["input_ids", "attention_mask", "labels"], and the data type of three columns is int64. Columns： input_ids: the tokenized inputs, Tensor of shape :math:`(batch, seq_length)`. attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, Tensor of shape :math:`(batch, seq_length)`. labels: same as input_ids, Tensor of shape :math:`(batch, seq_length)`. About wikitext-2 Dataset: The wikitext-2 language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia. As it is composed of full articles, the dataset is well suited for models that can take advantage of long term dependencies. You can unzip the dataset files into the following structure: .. code-block:: . └── wikitext-2 ├── wiki.test.tokens ├── wiki.train.tokens └── wiki.valid.tokens """ from __future__ import absolute_import from __future__ import division from __future__ import print_function import argparse import collections import logging import re import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer def wikitext_clean(string): """ string clean """ # contractions string = string.replace("s '", "s'") string = re.sub(r"/' [0-9]/", r"/'[0-9]/", string) # number separators string = string.replace(" @-@ ", "-") string = string.replace(" @,@ ", ",") string = string.replace(" @.@ ", ".") # punctuation string = string.replace(" : ", ": ") string = string.replace(" ; ", "; ") string = string.replace(" . ", ". ") string = string.replace(" .", ".") string = string.replace(" ! ", "! ") string = string.replace(" ? ", "? ") string = string.replace(" , ", ", ") # double brackets string = re.sub(r"\(\s*([^\)]*?)\s*\)", r"(\1)", string) string = re.sub(r"\[\s*([^\]]*?)\s*\]", r"[\1]", string) string = re.sub(r"{\s*([^}]*?)\s*}", r"{\1}", string) string = re.sub(r"\"\s*([^\"]*?)\s*\"", r'"\1"', string) string = re.sub(r"'\s*([^']*?)\s*'", r"'\1'", string) # miscellaneous string = string.replace("= = = =", "====") string = string.replace("= = =", "===") string = string.replace("= =", "==") string = string.replace(" " + chr(176) + " ", chr(176)) string = string.replace(" \n", "\n") string = string.replace("\n ", "\n") string = string.replace(" N ", " 1 ") string = string.replace(" 's", "'s") return string def preprocess_data(input_file): """ preprocess data """ dataset_valid = [] passage = [] count = 0 with open(input_file, 'r', encoding='utf-8') as f: for line in f: line = line.strip() if line: if line.startswith('=') and line.endswith('=') and passage: dataset_valid.append(passage) count += 1 passage = [] elif line.startswith('=') and line.endswith('='): continue else: passage.append(line) print('read {} file finished!\n total count = {}'.format(input_file, count)) res = [] for line in dataset_valid: text = "" for sentence in line: sentence = wikitext_clean(sentence) text = text + " " + sentence text = text.strip() res.append(text) return res def create_instance(tokenizer, sentence, ids, max_length=None): """A single sample instance for LM task.""" pair_ids = None if len(sentence) == 2: pair_ids = tokenizer.encode(sentence[1]) output = tokenizer.prepare_for_model(ids=ids, pair_ids=pair_ids, add_special_tokens=False, max_length=max_length, padding='max_length', truncate_direction="LEFT", return_overflowing_tokens=False, return_attention_mask=True) return output def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] labels = instance["input_ids"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(labels) writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="../wikitext-2/wiki.valid.tokens", help='Input raw text file. ') parser.add_argument("--output_file", type=str, default="../wikitext2_processed/wikitext-2.mindrecord", help='Output MindRecord file. ') parser.add_argument("--num_splits", type=int, default=1, help="The MindRecord file will be split into the number of partition. ") parser.add_argument("--max_length", type=int, default=1024, help="Maximum sequence length. ") parser.add_argument("--tokenizer_type", type=str, default="gpt2", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") args = parser.parse_args() tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) tokenizer.pad_token = tokenizer.eos_token tokenizer.add_tokens([]) input_file = args.input_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", input_file) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, "lm-schema") dataset_valid = preprocess_data(args.input_file) total_written = 0 logging.info("***** Reading from %s *****", input_file) text_total = "\n".join(dataset_valid) # the logic of \n is copied from modelzoo sentence = text_total.strip().split("\t") block_size = args.max_length total_ids = tokenizer.encode(sentence[0]) total_length = len(total_ids) total_length = (total_length // block_size) * block_size print("total_length", total_length) for i in range(total_length // block_size): ids = total_ids[block_size*i:block_size*(i+1)] output = create_instance(tokenizer, sentence, ids, args.max_length) write_instance_to_file(writer, instance=output) total_written += 1 writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) main() pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/llama/000077500000000000000000000000001476650141400265575ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/llama/alpaca_converter.py000066400000000000000000000037521476650141400324500ustar00rootroot00000000000000""" fastchat stanford alpaca data convert tools. """ import argparse import json import pathlib # Prompt from stanford alpaca's training script PROMPT_DICT = { "prompt_input": ( "Below is an instruction that describes a task, paired with an input that provides further context. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Input:\n{input}\n\n### Response:" ), "prompt_no_input": ( "Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{instruction}\n\n### Response:" ), } def main(args_param): data_path = pathlib.Path(args_param.data_path) with data_path.open() as f: data = json.load(f) prompt_input, prompt_no_input = ( PROMPT_DICT["prompt_input"], PROMPT_DICT["prompt_no_input"], ) sources = [ prompt_input.format_map(example) if example.get("input", "") != "" else prompt_no_input.format_map(example) for example in data ] targets = [example["output"] for example in data] new_data = [] cnt = 1 for s, t in zip(sources, targets): new_data.append( { "id": str(cnt), "conversations": [ { "from": "human", "value": s, }, { "from": "gpt", "value": t, }, ], } ) cnt += 1 json.dump(new_data, open(args_param.output_path, "w"), indent=2) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("--data_path", type=str, default="alpaca-data.json") parser.add_argument( "--output_path", type=str, default="alpaca-data-conversation.json" ) args = parser.parse_args() main(args) pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/llama/llama_preprocess.py000066400000000000000000000221161476650141400324660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ transform wikitext-2, wikitext-103, lambada, openwebtext dataset to mindrecord. """ import argparse import json import os import re import numpy as np # fschat >= 0.2.13 from fastchat.conversation import get_conv_template from mindspore.mindrecord import FileWriter from mindformers.models.llama.llama_tokenizer import LlamaTokenizer IGNORE_TOKEN_ID = -100 def chunks(lst, n): """ yield n sized chunks from list""" for i in range(0, len(lst), n): yield lst[i:i+n] def package_file(it, n): """ package multiple files""" stop = False while not stop: batch = [] for _ in range(n): try: batch.append(next(it)) except StopIteration: stop = True if not batch: break yield batch def clean_wikitext(string): """ cleaning wikitext dataset""" # contractions string = string.replace("s '", "s'") string = re.sub(r"/' [0-9]/", r"/'[0-9]/", string) # number separators string = string.replace(" @-@ ", "-") string = string.replace(" @,@ ", ",") string = string.replace(" @.@ ", ".") # punctuation string = string.replace(" : ", ": ") string = string.replace(" ; ", "; ") string = string.replace(" . ", ". ") string = string.replace(" ! ", "! ") string = string.replace(" ? ", "? ") string = string.replace(" , ", ", ") # double brackets string = re.sub(r"\(\s*([^\)]*?)\s*\)", r"(\1)", string) string = re.sub(r"\[\s*([^\]]*?)\s*\]", r"[\1]", string) string = re.sub(r"{\s*([^}]*?)\s*}", r"{\1}", string) string = re.sub(r"\"\s*([^\"]*?)\s*\"", r'"\1"', string) string = re.sub(r"'\s*([^']*?)\s*'", r"'\1'", string) # miscellaneous string = string.replace("= = = =", "====") string = string.replace("= = =", "===") string = string.replace("= =", "==") string = string.replace(" "+chr(176)+" ", chr(176)) string = string.replace(" \n", "\n") string = string.replace("\n ", "\n") string = string.replace(" N ", " 1 ") string = string.replace(" 's", "'s") return string def preprocess(sources, tokenizer, seq_length): """conversation preprocess.""" conv = get_conv_template("vicuna_v1.1").copy() roles = {"human": conv.roles[0], "gpt": conv.roles[1]} # Apply prompt templates conversations = [] for i, source in enumerate(sources): if roles[source[0]["from"]] != conv.roles[0]: # Skip the first one if it is not from human source = source[1:] conv.messages = [] for j, sentence in enumerate(source): role = roles[sentence["from"]] assert role == conv.roles[j % 2], f"{i}" conv.append_message(role, sentence["value"]) conversations.append(conv.get_prompt()) sep = conv.sep + conv.roles[1] + ": " # Tokenize conversations input_ids = [] targets = [] # attention_mask = [] for conversation in conversations: rounds = conversation.split(conv.sep2) ids = [tokenizer.bos_token_id] mask = [1] for i, rou in enumerate(rounds): if rou == "": break conv_out = tokenizer(rou) ids.extend(conv_out['input_ids'][1:]) mask.extend(conv_out['attention_mask'][1:]) d = {'input_ids': ids, 'attention_mask': mask} # pylint: disable=W0212 d = tokenizer._pad(d, max_length=seq_length, padding_strategy='max_length') input_ids.append(d['input_ids']) # attention_mask.append(d['attention_mask']) target = np.array(d['input_ids']) total_len = int(np.not_equal(target, tokenizer.pad_token_id).sum()) cur_len = 1 target[:cur_len] = IGNORE_TOKEN_ID for i, rou in enumerate(rounds): if rou == "": break parts = rou.split(sep) if len(parts) != 2: break parts[0] += sep round_len = len(tokenizer(rou)['input_ids']) - 1 instruction_len = len(tokenizer(parts[0])['input_ids']) - 3 target[cur_len : cur_len + instruction_len] = IGNORE_TOKEN_ID cur_len += round_len target[cur_len:] = IGNORE_TOKEN_ID if cur_len < seq_length: if cur_len != total_len: target[:] = IGNORE_TOKEN_ID targets.append(target.tolist()) input_ids = np.array(input_ids) targets = np.array(targets) return dict( input_ids=input_ids, labels=targets, ) class SupervisedDataset: """Dataset for supervised fine-tuning.""" def __init__(self, raw_data, tokenizer, seq_length): super(SupervisedDataset, self).__init__() sources = [example["conversations"] for example in raw_data] data_dict = preprocess(sources, tokenizer, seq_length) self.input_ids = data_dict["input_ids"] self.labels = data_dict["labels"] def __len__(self): return len(self.input_ids) def __getitem__(self, i): return dict( input_ids=self.input_ids[i], labels=self.labels[i] ) def tokenize_wiki(tokenizer, file_path, seq_length, repeat): """tokenize wikitext-2/wikitext-103 dataset""" content = [] with open(file_path, 'r', encoding='utf-8') as f: for para in clean_wikitext(f.read()).split("\n\n"): if para and para.strip().startswith('=') is False: content += tokenizer(para)['input_ids'] content_out = [] for _ in range(repeat): content_out.extend(content) content = content_out for chunk in chunks(content, seq_length): sample = {} if len(chunk) == seq_length: sample['input_ids'] = np.array(chunk, dtype=np.int32) yield sample def tokenize_qa(tokenizer, file_path, seq_length): raw_data = json.load(open(file_path, "r")) dataset_cls = SupervisedDataset(raw_data, tokenizer, seq_length) for i in range(len(dataset_cls)): yield dataset_cls[i] if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--dataset_type', type=str, default='wiki') parser.add_argument('--input_glob', type=str, default='/mnt/luolan/wikitext-2/wiki.train.tokens') parser.add_argument('--output_file', type=str, default='./dataset/wiki2048/wiki2048') parser.add_argument('--tokenizer', type=str, default='llama', choices=['llama']) parser.add_argument('--model_file', type=str, default='/mnt/luolan/llama/tokenizer.model') parser.add_argument('--file_partition', type=int, default=1) parser.add_argument('--repeat', type=int, default=1) parser.add_argument('--seq_length', type=int, default=2048) args = parser.parse_args() out_dir, out_file = os.path.split(os.path.abspath(args.output_file)) if not os.path.exists(out_dir): os.mkdir(out_dir) if args.dataset_type == 'wiki': schema = {'input_ids': {"type": "int32", "shape": [-1]},} elif args.dataset_type == 'qa': schema = {'input_ids': {"type": "int32", "shape": [-1]}, 'labels': {"type": "int32", "shape": [-1]}} writer = FileWriter(file_name=args.output_file, shard_num=args.file_partition) writer.add_schema(schema, args.dataset_type) writer.open_and_set_header() # Start to load tokenizer if not os.path.exists(args.model_file): raise FileNotFoundError(f"file {args.model_file} do not exists.") transforms_count = 0 word_tokenizer = LlamaTokenizer(vocab_file=args.model_file) if hasattr(word_tokenizer, 'add_bos_token'): word_tokenizer.add_bos_token = True if hasattr(word_tokenizer, 'add_eos_token'): word_tokenizer.add_eos_token = True if args.dataset_type == 'wiki': for x in tokenize_wiki(word_tokenizer, args.input_glob, args.seq_length + 1, args.repeat): transforms_count += 1 writer.write_raw_data([x]) print("Transformed {} records.".format(transforms_count)) elif args.dataset_type == 'qa': for x in tokenize_qa(word_tokenizer, args.input_glob, args.seq_length + 1): transforms_count += 1 writer.write_raw_data([x]) print("Transformed {} records.".format(transforms_count)) else: raise ValueError( "Not support dataset type: {}".format(args.dataset_type)) writer.commit() out_file = args.output_file if args.file_partition > 1: out_file += '0' print("Transform finished, output files refer: {}".format(out_file)) pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/llama/squad_data_process.py000066400000000000000000000136031476650141400330000ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Data process for SQuAD Dataset""" import argparse import json import collections import copy import logging import pathlib import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer IGNORE_TOKEN_ID = -100 def write_instance_to_file(writer, instance): """write the instance to file""" input_ids = instance["input_ids"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["labels"] = np.asarray(label) writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="./squad/train-v1.1.json", required=False, help='Input raw json file. ') parser.add_argument("--output_file", type=str, default="./squad/recode_train.mindrecord", required=False, help='Output MindRecord file. ') parser.add_argument("--mode", type=str, default="train", help='Set Data for train or eval.') parser.add_argument("--max_length", type=int, default=2048, help='Maximum sequence length. ') parser.add_argument("--tokenizer_type", type=str, default="llama_7b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") args = parser.parse_args() logging.info("***** Reading from input files *****") logging.info("Input File: %s", args.input_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) input_file = pathlib.Path(args.input_file) with input_file.open() as f: file = json.load(f) sources = [] targets = [] for data in file["data"]: for paragraph in data["paragraphs"]: passage = paragraph["context"] query = paragraph["qas"][0]["question"] answer = paragraph["qas"][0]["answers"][0]["text"] input_str = f"Read the passage and answer the question below.\n\n### Instruction:\n{passage}\n\n### Input:\n{query}\n\n### Response:" sources.append(input_str) targets.append(answer) logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(args.output_file, 1) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } writer.add_schema(data_schema, "lm-schema") total_written = 0 # for eval if args.mode == "eval": if hasattr(tokenizer, 'add_bos_token'): tokenizer.add_bos_token = True if hasattr(tokenizer, 'add_eos_token'): tokenizer.add_eos_token = False for prompt, answer in zip(sources, targets): total_written += 1 input_ids = tokenizer.encode(prompt, add_special_tokens=True) if len(input_ids) >= args.max_length: input_ids = input_ids[:args.max_length] else: input_ids = np.pad(input_ids, (0, args.max_length - len(input_ids)), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) input_ids = np.array(input_ids).reshape(1, -1) label_id = tokenizer.encode(answer, add_special_tokens=False) label_id = np.pad(label_id, (0, args.max_length - len(label_id)), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) label_id = np.array(label_id).reshape(1, -1) instance = {"input_ids": input_ids, "labels": label_id} write_instance_to_file(writer, instance=instance) # for train/finetune elif args.mode == "train": if hasattr(tokenizer, 'add_bos_token'): tokenizer.add_bos_token = True if hasattr(tokenizer, 'add_eos_token'): tokenizer.add_eos_token = True for prompt, answer in zip(sources, targets): total_written += 1 concated_qa = prompt + answer input_ids = tokenizer.encode(concated_qa, add_special_tokens=True) input_ids = np.array(input_ids) prompt_ids = tokenizer.encode(prompt, add_special_tokens=False) prompt_ids = np.array(prompt_ids) prompt_length = len(prompt_ids) concat_length = len(input_ids) pad_length = args.max_length + 1 - concat_length input_ids_new = np.pad(input_ids, (0, pad_length), 'constant', constant_values=(tokenizer.pad_token_id, tokenizer.pad_token_id)) label_id_new = copy.deepcopy(input_ids_new) label_id_new[:prompt_length] = IGNORE_TOKEN_ID label_id_new[-pad_length:] = IGNORE_TOKEN_ID instance = {"input_ids": input_ids_new, "labels": label_id_new} write_instance_to_file(writer, instance=instance) else: logging.error("No mode named %s, please set mode as train/eval.", args.mode) writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": logging.basicConfig(level=logging.INFO) main() pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/pangualpha/000077500000000000000000000000001476650141400276115ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/pangualpha/cmrc2018_data_process.py000066400000000000000000000252651476650141400341630ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Data process for CMRC2018 Dataset""" import re import json import argparse import collections import logging import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer np.random.seed(666) def gen_prompt(prompts_, n_shot, len_ori, max_length=1024): if n_shot == 0: res = "" else: ids = np.random.choice(len(prompts_), n_shot).tolist() res = "" for i in ids: if len(res) + len(prompts_[i]) < max_length - len_ori - 10: res = res + prompts_[i] return res def cut_sent(para): para = re.sub(r'([。！？\?])([^”’])', r"\1\n\2", para) para = re.sub(r'([。！？\?][”’])([^，。！？\?])', r'\1\n\2', para) para = para.rstrip() # del extra \n return para.split("\n") class Cmrc2018Dataset: """dataset process""" def __init__(self, cmrc2018_train_json, cmrc2018_dev_json): with open(cmrc2018_train_json, "r", encoding="utf-8") as f: self.train_data = json.load(f)["data"] with open(cmrc2018_dev_json, "r", encoding="utf-8") as f: self.dev_data = json.load(f)["data"] def gen_prompt_from_train(self, seg=False): """gen prompt from train dataset""" data_list = self.train_data index = 0 prompts_ = [] if seg: for data in data_list: context = data["paragraphs"][0]["context"] context_splits = cut_sent(context) qas = data["paragraphs"][0]["qas"] for qa in qas: index += 1 q = qa["question"] a = qa["answers"][0]["text"] for sent in context_splits: if a in sent: prompt = f"阅读文章：{sent}\n问：{q}\n答：{a}\n" prompts_.append(prompt) prompt_max_length = 80 prompts_ = [x for x in prompts_ if len(x) < prompt_max_length] else: for data in data_list: context = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas: index += 1 q = qa["question"] a = qa["answers"][0]["text"] prompt = f"阅读文章：{context}\n问：{q}\n答：{a}\n" prompts_.append(prompt) prompt_max_length = 400 prompts_ = [x for x in prompts_ if len(x) < prompt_max_length] return prompts_ def get_data_allshot(self, n_content=50, n_q=1, max_length=1024): """gen data for all shots""" prompts_seg = self.gen_prompt_from_train(seg=True) prompts_full = self.gen_prompt_from_train(seg=False) input_prompts = [] answers = [] data_list = self.dev_data for data in data_list[:n_content]: context_ = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas[:n_q]: q = qa["question"] a = qa["answers"][0]["text"] input_str0 = f"阅读文章：{context_}\n问：{q}\n答：" demo0 = "" demo1 = gen_prompt(prompts_full, n_shot=1, len_ori=len(input_str0), max_length=max_length) demo2 = gen_prompt(prompts_seg, n_shot=5, len_ori=len(input_str0), max_length=max_length) input_prompts.append([demo0 + input_str0, demo1 + input_str0, demo2 + input_str0]) answers.append(a) return input_prompts, answers def get_data(self, n_content=50, n_q=1, n_shot=0): """get finally data""" prompts_seg = self.gen_prompt_from_train(seg=True) prompts_full = self.gen_prompt_from_train(seg=False) input_prompts = [] answers = [] data_list = self.dev_data for data in data_list[:n_content]: context_ = data["paragraphs"][0]["context"] qas = data["paragraphs"][0]["qas"] for qa in qas[:n_q]: q = qa["question"] a = qa["answers"][0]["text"] input_str0 = f"阅读文章：{context_}\n问：{q}\n答：" if n_shot == 0: demo = "" elif n_shot == 1: demo = gen_prompt(prompts_full, n_shot=1, len_ori=len(input_str0)) else: demo = gen_prompt(prompts_seg, n_shot=n_shot, len_ori=len(input_str0)) input_prompts.append(demo + input_str0) answers.append(a) return input_prompts, answers def write_instance_to_file(writer, instance, need_del_keys): """write the instance to file""" input_ids = instance["input_ids"] attention_mask = instance["attention_mask"] label = instance["labels"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["attention_mask"] = np.asarray(attention_mask) features["labels"] = np.asarray(label) for need_del_key in need_del_keys: del features[need_del_key] writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--train_file", type=str, default="./cmrc2018_public/train.json", required=True, help='Input raw train file. ') parser.add_argument("--dev_file", type=str, default="./cmrc2018_public/dev.json", required=True, help='Input raw dev file. ') parser.add_argument("--output_file", type=str, default="./cmrc2018.mindrecord", required=True, help='Output MindRecord file. ') parser.add_argument("--num_splits", type=int, default=1, help='The MindRecord file will be split into the number of partition. ') parser.add_argument("--max_length", type=int, default=1024, help='Maximum sequence length. ') parser.add_argument("--n_shot", type=str, default="zero_shot", choices=['zero_shot', 'one_shot', 'five_shot'], help="N shot learning of cmrc2018. ") parser.add_argument("--tokenizer_type", type=str, default="pangualpha_2_6b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--n_content", type=int, default=500, help="How many contents are sampled. ") parser.add_argument("--n_q", type=int, default=100, help="How many questions per content as most are sampled. ") parser.add_argument("--mindrecord_schema", type=str, default="pangualpha_cmrc2018", help="The name of mindrecord_schema. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "labels"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. " "Please note that 'labels' does input model.construct, it only used at train.evaluate. ") args = parser.parse_args() dev_file = args.dev_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", dev_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) shot_dict = {'zero_shot': 0, 'one_shot': 1, 'five_shot': 2} dataset_cmrc2018 = Cmrc2018Dataset(cmrc2018_train_json=args.train_file, cmrc2018_dev_json=args.dev_file) prompts, answers = dataset_cmrc2018.get_data_allshot( n_content=args.n_content, n_q=args.n_q, max_length=args.max_length) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, args.mindrecord_schema) total_written = 0 for prompt, answer in zip(prompts, answers): prompt = prompt[shot_dict[args.n_shot]] input_ids = tokenizer.encode(prompt, add_special_tokens=False) # this logic refers the open source # https://gitee.com/foundation-models/tk-models/blob/master/models/pangu_alpha/cmrc2018/src/evaluate_main.py if len(input_ids) >= args.max_length - 4: truncated_length = (args.max_length // 100 - 1) * 100 input_ids = input_ids[-truncated_length:] attention_mask = [1] * len(input_ids) pad_len = args.max_length - len(input_ids) input_ids += [tokenizer.pad_token_id] * pad_len attention_mask += [0] * pad_len input_ids = np.array(input_ids).reshape(1, -1) attention_mask = np.array(attention_mask).reshape(1, -1) label_id = tokenizer.encode(answer, add_special_tokens=False) label_id = np.array(label_id).reshape(1, -1) instance = {"input_ids": input_ids, "attention_mask": attention_mask, "labels": label_id} features = write_instance_to_file(writer, instance=instance, need_del_keys=need_del_keys) total_written += 1 display_num = 20 # how many example to display if total_written <= display_num: logging.info("***** Example *****") logging.info("input tokens: %s", tokenizer.decode(instance["input_ids"])) logging.info("label tokens: %s", tokenizer.decode(instance["labels"])) for feature_name in features.keys(): feature = features[feature_name] logging.info("%s: %s", feature_name, feature) writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": # logging.basicConfig(level=logging.INFO) main() pengcheng.mind-mindformer/mindformers/tools/dataset_preprocess/pangualpha/tnews_data_process.py000066400000000000000000000246721476650141400340650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Data process for TNEWS Dataset. The generated dataset has output columns: ["input_ids", "labels", "attention_mask"], and the data type of three columns is int32. Columns： input_ids: the tokenized inputs, shape is [num_labels, seq_length] labels: the index of the true label, shape is [1,] attention_mask: the mask indicating whether each position is a valid input and is not the added prompt, shape is [num_labels, seq_length] About TNEWS Dataset: TNEWS is a 15-class(finance, technology, sports, etc.) short news text classification dataset. Each piece of data has three attributes, which are classification ID, classification name, and news string (only including the title). You can unzip the dataset files into the following structure: .. code-block:: . └── tnews_public ├── train,json ├── test.json ├── test1.0.json ├── dev.json ├── labels.json └── README.txt """ import itertools import json import argparse import collections import logging import numpy as np from mindspore.mindrecord import FileWriter from mindformers.auto_class import AutoTokenizer EN2ZH_LABEL_MAP = {"news_story": "故事", "news_culture": "文化", "news_entertainment": "娱乐", "news_sports": "体育", "news_finance": "财经", "news_house": "房产", "news_car": "汽车", "news_edu": "教育", "news_tech": "科技", "news_military": "军事", "news_travel": "旅行", "news_world": "世界", "news_stock": "股票", "news_agriculture": "农业", "news_game": "游戏"} NUM_LABELS = 4 def get_en2zh_label(label_path): """get label mapping""" en2zh_labels = {} print('loading label file ') # labels_ch.json is the results that translated from original labels with open(label_path, "r", encoding="utf-8") as f: lines = f.readlines() for line in lines: line_json = json.loads(line) if "label_desc_ch" in line_json: en_label = line_json['label_desc'] zh_label = line_json['label_desc_ch'] en2zh_labels[en_label] = zh_label else: break if en2zh_labels == {}: en2zh_labels = EN2ZH_LABEL_MAP return en2zh_labels def load_tnews_example_for_shot(data_path, en2zh_labels, num_sample=6, np_rng=None, max_len=150, prompt="这是关于{}的文章：{}"): """get prompt for n shot""" with open(data_path, 'r', encoding="utf-8") as fid: data = [json.loads(x) for x in fid.readlines()] if np_rng is None: np_rng = np.random.default_rng(0) # select sample with balanced labels label_key = lambda x: x[1] sample_groupbyed = itertools.groupby( sorted([(x, en2zh_labels[y['label_desc']]) for x, y in enumerate(data)], key=label_key), key=label_key) group_index = [np.array([z[0] for z in y]) for x, y in sample_groupbyed] for x in group_index: np_rng.shuffle(x) # in-place nums = (num_sample - 1) // len(group_index) + 1 group_index_concated = np.concatenate([x[:nums] for x in group_index]) np_rng.shuffle(group_index_concated) selected_index = group_index_concated[:num_sample] examples = [] for x in selected_index: sentence = data[x]['sentence'] example_formated = prompt.format(en2zh_labels[data[x]['label_desc']], sentence)[:max_len] examples.append(example_formated) ret = { 'zero_shot': '', 'one_shot': examples[0] + '\n', 'few_shot': ('\n'.join(examples)) + '\n', } return ret def get_data_generate(data_path, en2zh_labels, tokenizer, shot_examples, pad_token="", seq_length=1024, np_rng=None, task="one_shot", prompt="这是关于{}的文章：{}"): """gen data instances""" with open(data_path, 'r', encoding="utf-8") as fid: data = [json.loads(x) for x in fid.readlines()] label_list = sorted({en2zh_labels[x['label_desc']] for x in data}) id_to_label = dict(enumerate(label_list)) print('All test case num ', len(data)) example = shot_examples[task] pad_id = tokenizer.encoder[pad_token] for instance in data: true_label = en2zh_labels[instance['label_desc']] tmp0 = sorted(list(set(id_to_label.values()) - {true_label})) fake_label = [tmp0[x] for x in np_rng.permutation(len(tmp0))[:NUM_LABELS-1]] # [:3] instance_tf_label = [true_label] + fake_label instance_tf_label = [instance_tf_label[x] for x in np_rng.permutation(len(instance_tf_label))] # shuffle input_ids_list = [] mask_list = [] label_list = [] instance_res = {} for label_i in instance_tf_label: prompt_with_shot_example = "{}" + prompt tmp0 = tokenizer.tokenize(prompt_with_shot_example.format(example, label_i, "")) tmp1 = prompt_with_shot_example.format(example, label_i, instance['sentence']) input_ids = tokenizer.encode(tmp1)[:seq_length] mask = np.zeros(seq_length) mask[len(tmp0):len(input_ids)] = 1 input_ids = np.pad(input_ids, ((0, seq_length - len(input_ids)),), 'constant', constant_values=(0, pad_id)) input_ids_list.append(input_ids) mask_list.append(mask) label_list.append(label_i) true_label = [x for x, y in enumerate(label_list) if y == true_label][0] instance_res["input_ids"] = input_ids_list instance_res["labels"] = true_label instance_res["attention_mask"] = mask_list yield instance_res def write_instance_to_file(writer, instance, need_del_keys): """write the instance to file""" input_ids = instance["input_ids"] labels = instance["labels"] attention_mask = instance["attention_mask"] features = collections.OrderedDict() features["input_ids"] = np.asarray(input_ids) features["labels"] = np.asarray(labels) features["attention_mask"] = np.asarray(attention_mask) for need_del_key in need_del_keys: del features[need_del_key] writer.write_raw_data([features]) return features def main(): parser = argparse.ArgumentParser() parser.add_argument("--input_file", type=str, default="./tnews_public/dev.json", required=True, help="Input raw text file. ") parser.add_argument("--label_file", type=str, default="./tnews_public/labels.json", required=True, help="Input label file. ") parser.add_argument("--output_file", type=str, default="./tnews.mindrecord", required=True, help="Output MindRecord file which ends with '.mindrecord' ") parser.add_argument("--num_splits", type=int, default=1, help="The MindRecord file will be split into the number of partition. ") parser.add_argument("--max_length", type=int, default=1024, help="Maximum sequence length. ") parser.add_argument("--tokenizer_type", type=str, default="pangualpha_2_6b", help="Tokenizer type, can be set to any tokenizer " "if its relevant model supports prompt text classification. ") parser.add_argument("--mindrecord_schema", type=str, default="pangualpha_tnews", help="The name of mindrecord_schema. ") parser.add_argument("--data_columns", type=list, default=["input_ids", "labels", "attention_mask"], help="The data columns which should be saved in mindrecord. This can refer used yaml file. ") parser.add_argument("--n_shot", type=str, default="one_shot", choices=['zero_shot', 'one_shot', 'five_shot'], help="N shot learning of t-news. ") parser.add_argument("--prompt_text", type=str, default="这是关于{}的文章：{}", help="Input String format. ") args = parser.parse_args() np_rng = np.random.default_rng(seed=2) input_file = args.input_file logging.info("***** Reading from input files *****") logging.info("Input File: %s", input_file) tokenizer = AutoTokenizer.from_pretrained(args.tokenizer_type) en2zh_labels = get_en2zh_label(args.label_file) shot_to_example = load_tnews_example_for_shot(args.input_file, en2zh_labels, np_rng=np_rng, prompt=args.prompt_text) data_generate = get_data_generate(args.input_file, en2zh_labels=en2zh_labels, tokenizer=tokenizer, shot_examples=shot_to_example, seq_length=args.max_length, np_rng=np_rng, task=args.n_shot, prompt=args.prompt_text) output_file = args.output_file logging.info("***** Writing to output files *****") logging.info("Output File: %s", args.output_file) writer = FileWriter(output_file, args.num_splits) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]} } data_columns = args.data_columns need_del_keys = set(data_columns) - set(data_schema.keys()) for need_del_key in need_del_keys: del data_schema[need_del_key] writer.add_schema(data_schema, args.mindrecord_schema) total_written = 0 for instance in data_generate: write_instance_to_file(writer, instance=instance, need_del_keys=need_del_keys) total_written += 1 writer.commit() logging.info("Wrote %d total instances", total_written) if __name__ == "__main__": # logging.basicConfig(level=logging.INFO) main() pengcheng.mind-mindformer/mindformers/tools/download_tools.py000066400000000000000000000067421476650141400252110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ '''download_tools''' import time import os import requests import urllib3 from tqdm import tqdm from mindformers.tools.logger import logger try: import fcntl except ImportError: fcntl = None logger.warning("The library fcntl is not found. This may cause the reading file failed " "when call the from_pretrained for different process.") urllib3.disable_warnings() class StatusCode: '''StatusCode''' succeed = 200 def download_with_progress_bar(url, filepath, chunk_size=1024, timeout=4): """download_with_progress_bar""" local_id = int(os.getenv("RANK_ID", "0")) if local_id % 8 != 0: logger.info("Wait for the first card to download file. ") while True: if os.path.exists(filepath+".lock"): return True if not os.path.exists(filepath): os.makedirs(os.path.dirname(filepath), exist_ok=True) header = { "Accept-Encoding": "identity", "User-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:65.0) Gecko/20100101 Firefox/65.0' } start = time.time() try: response = requests.get(url, verify=False, stream=True, timeout=timeout) except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False content_size = response.headers.get('content-length') if content_size is None: response_json = response.json() download_url = response_json.get("data").get("download_url") if download_url: response = requests.get(download_url, verify=False, stream=True, timeout=timeout, headers=header) content_size = int(response.headers.get('content-length')) else: logger.error("Download url parsing failed from json file, please download %s to %s.", url, filepath) return False else: content_size = int(content_size) size = 0 if response.status_code == StatusCode.succeed: logger.info('Start download %s', filepath) with open(filepath, 'wb') as file: if fcntl: fcntl.flock(file.fileno(), fcntl.LOCK_EX) with tqdm(total=content_size, desc='Downloading', leave=True, ncols=100, unit='B', unit_scale=True) as pbar: for data in response.iter_content(chunk_size=chunk_size): file.write(data) size += len(data) pbar.update(1024) end = time.time() logger.info('Download completed!,times: %.2fs', (end - start)) if not os.path.exists(filepath+".lock"): os.mknod(filepath+".lock") return True logger.error("%s is unconnected!", url) return False pengcheng.mind-mindformer/mindformers/tools/download_tools_multithread.py000066400000000000000000000141431476650141400276050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ '''download_tools_multithread''' import os import time from threading import Thread, Lock import requests import urllib3 from tqdm import tqdm from logger import logger try: import fcntl except ImportError: fcntl = None logger.warning("The library fcntl is not found. This may cause the reading file failed " "when call the from_pretrained for different process.") urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) def download_chunk(url, start, end, file, pbar, lock, chunk_size=1024): '''Download a chunk of a file from the given URL and write it to the provided file object. Args: url (str): The URL to download from. start (int): The starting byte position of the chunk to download. end (int): The ending byte position of the chunk to download. file (file): The file object to write the downloaded chunk to. pbar (tqdm.tqdm): The progress bar object to update the download progress. lock (threading.Lock): The lock object to synchronize writing to the file. chunk_size (int, optional): The size of each chunk to download in bytes. Defaults to 1024. Returns: None. Raises: None. ''' headers = {'Range': 'bytes=%d-%d' % (start, end)} response = requests.get(url, headers=headers, verify=False, stream=True) if response.status_code == requests.codes.partial_content: with lock: file.seek(start) for data in response.iter_content(chunk_size=chunk_size): file.write(data) pbar.update(chunk_size) else: logger.error("%s is unconnected!", url) def download_with_progress_bar(url, filepath, num_threads=1, timeout=4): """Downloads a file from the given URL with multi-threading support, resuming from breakpoints, and displays a progress bar to show the progress of the download. Args: url (str): The URL to download the file from. filepath (str): The path where the downloaded file will be saved. num_threads (int, optional): The number of threads to use for the download. Default is 8. timeout (int, optional): The connection timeout in seconds. Default is 4 seconds. Returns: bool: Returns True if the download is successful; otherwise, False. """ if not os.path.exists(filepath): os.makedirs(os.path.dirname(filepath), exist_ok=True) start_time = time.time() try: response = requests.head(url, verify=False, stream=True, timeout=timeout) content_size = int(response.headers['content-length']) etag = response.headers.get('etag') except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False if content_size is None: response_json = response.json() download_url = response_json.get("data").get("download_url") if download_url: header = { "Accept-Encoding": "identity", "User-agent": 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:65.0) Gecko/20100101 Firefox/65.0' } try: response = requests.head(download_url, verify=False, stream=True, timeout=timeout, headers=header) content_size = int(response.headers.get('content-length')) etag = response.headers.get('etag') except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError, requests.exceptions.ConnectionError): logger.error("Connect error, please download %s to %s.", url, filepath) return False else: logger.error("Download url parsing failed from json file, please download %s to %s.", url, filepath) return False start = 0 if os.path.exists(filepath): # check file size and etag file_size = os.path.getsize(filepath) if file_size == content_size and etag == response.headers.get('etag'): logger.info('File already exists: %s', filepath) return True logger.info('File is not complete, resuming download: %s', filepath) start = file_size with open(filepath, 'ab') as file: if fcntl: fcntl.flock(file.fileno(), fcntl.LOCK_EX) chunk_size = (content_size - start) // num_threads with tqdm(total=content_size, initial=start, unit='B', unit_scale=True, desc=filepath.split('/')[-1]) as pbar: threads = [] for i in range(num_threads): if i < num_threads - 1: end = start + chunk_size - 1 else: end = content_size - 1 t = Thread(target=download_chunk, args=(url, start, end, file, pbar, Lock(), 1024)) threads.append(t) t.start() start += chunk_size for t in threads: t.join() # check downloaded file size and etag file_size = os.path.getsize(filepath) if file_size == content_size and etag == response.headers.get('etag'): end_time = time.time() logger.info('Download finished: %s times: %.2fs', filepath, (end_time - start_time)) return True logger.error('Download failed or interrupted: %s', filepath) return False pengcheng.mind-mindformer/mindformers/tools/export.py000066400000000000000000000200061476650141400234700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """export mindir.""" import argparse import glob import os.path import numpy as np import mindspore as ms import mindspore.common.dtype as mstype from mindformers import AutoModel from mindformers.tools.register import MindFormerConfig def get_glm_prefill_model_input(batch_size, seq_length): """get glm model input tuple.""" x = ms.Tensor(np.ones([batch_size, seq_length]).astype(np.int32)) position_ids = ms.Tensor(np.ones([batch_size, 2, seq_length]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, seq_length, seq_length]).astype(np.int32)) return x, position_ids, attention_mask def get_llm_common_prefill_model_input(batch_size, seq_length): """get llama model input tuple.""" x = ms.Tensor(np.ones([batch_size, seq_length]).astype(np.int32)) return x def get_bloom_inc_model_input(batch_size, seq_length, prefill): """get bloom kv cache model input tuple.""" if not prefill: seq_length = 1 init_reset = not prefill input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, input_position, None, None, None, None, init_reset, batch_valid_length def get_llama_inc_model_input(batch_size, seq_length, prefill): """get llama kv cache model input tuple.""" if not prefill: seq_length = 1 init_reset = not prefill input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, None, input_position, None, None, None, init_reset, batch_valid_length def get_glm_inc_model_input(batch_size, seq_length, prefill): """get glm kv cache model input tuple.""" if prefill: input_ids = ms.Tensor(np.ones((batch_size, seq_length)), mstype.int32) position_ids = ms.Tensor(np.ones([batch_size, 2, seq_length]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, seq_length, seq_length]).astype(np.int32)) else: input_ids = ms.Tensor(np.ones((batch_size, 1)), mstype.int32) position_ids = ms.Tensor(np.ones([batch_size, 2, 1]).astype(np.int32)) attention_mask = ms.Tensor(np.ones([batch_size, 1, 1, seq_length]).astype(np.int32)) input_position = ms.Tensor([127] * batch_size, mstype.int32) init_reset = not prefill init_reset = ms.Tensor([init_reset], mstype.bool_) batch_valid_length = ms.Tensor([128] * batch_size, mstype.int32) return input_ids, position_ids, attention_mask, input_position, None, None, init_reset, batch_valid_length PREFILL_MODEL_INPUT_MAP = { "bloom": get_llm_common_prefill_model_input, "llama": get_llm_common_prefill_model_input, "glm": get_glm_prefill_model_input, } INCREMENT_MODEL_INPUT_MAP = { "bloom": get_bloom_inc_model_input, "llama": get_llama_inc_model_input, "glm": get_glm_inc_model_input } def export_single_model(config, batch_size, model_type: str = 'MINDIR', model_dir=None): """ export no kvcache model. Args: config: config from yaml. batch_size(int): batch size model_type: model type """ model_name = config.trainer.model_name if model_dir: model = AutoModel.from_pretrained(model_dir) else: model = AutoModel.from_pretrained(model_name) model.set_train(False) inputs = () for item in PREFILL_MODEL_INPUT_MAP: if model_name.startswith(item): inputs = PREFILL_MODEL_INPUT_MAP[item](batch_size, config.infer.infer_seq_length) break if not inputs: raise NotImplementedError(f"model {model_name} not implemented.") suffix = model_type.lower() filename = config.infer.prefill_model_path.rstrip("." + suffix) model.add_flags_recursive(is_first_iteration=True) ms.export(model, *inputs, file_name=filename, file_format=model_type.upper()) def export_inc_model(config, batch_size, model_type: str = 'MINDIR', model_dir=None): """ export kvcache model. Args: config: config from yaml. batch_size(int): batch size model_type: model type """ model_name = config.trainer.model_name if model_dir: model = AutoModel.from_pretrained(model_dir) else: model = AutoModel.from_pretrained(model_name) model.set_train(False) func = None for item in INCREMENT_MODEL_INPUT_MAP: if model_name.startswith(item): func = INCREMENT_MODEL_INPUT_MAP[item] break if not func: raise NotImplementedError(f"model {model_name} not implemented.") suffix = model_type.lower() # export prefill filename = config.infer.prefill_model_path.rstrip("." + suffix) inputs = func(batch_size, config.infer.infer_seq_length, True) model.add_flags_recursive(is_first_iteration=True) ms.export(model, *inputs, file_name=filename, file_format=model_type.upper()) print("Prefill model exported.") # export inc filename = config.infer.increment_model_path.rstrip("." + suffix) inputs = func(batch_size, config.infer.infer_seq_length, False) model.add_flags_recursive(is_first_iteration=False) ms.export(model, *inputs, file_name=filename, file_format="MINDIR") print("Increment model exported.") def main(args_): """export mindir.""" ms.set_context(mode=ms.GRAPH_MODE, device_target="Ascend", device_id=args_.device_id) if args_.config_path is None and args_.model_dir is None: raise ValueError(f"config_path or model_dir must exist one.But get none.") if args_.config_path: config = MindFormerConfig(args_.config_path) model_dir = None else: model_dir = os.path.realpath(args_.model_dir) run_config = glob.glob(model_dir + "/*.yaml") if len(run_config) > 1: raise ValueError(f"{model_dir} contains multi config files: {run_config}.") config = MindFormerConfig(run_config[0]) if config.infer is None: raise KeyError(f"infer config not in {args_.config_path}.") if not config.infer.prefill_model_path: raise ValueError(f"prefill_model_path in {args_.config_path} is empty.") if not config.infer.increment_model_path: export_single_model(config, args_.batch_size, args_.model_type, model_dir) else: export_inc_model(config, args_.batch_size, args_.model_type, model_dir) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( '--config_path', default=None, type=str, help='YAML config files') parser.add_argument( '--model_dir', default=None, type=str, help='model config and ckpt file dir path') parser.add_argument( '--batch_size', default=1, type=int, help='batch size of model.') parser.add_argument( '--model_type', default="MINDIR", type=str, help='model type of exported model.') parser.add_argument( '--device_id', default=0, type=int, help='device id to export mindir.') args = parser.parse_args() main(args) pengcheng.mind-mindformer/mindformers/tools/hccl_tools.py000066400000000000000000000142521476650141400243060ustar00rootroot00000000000000# Copyright 2020 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate hccl config file script""" import json import os import socket import sys from argparse import ArgumentParser from typing import Dict, Any def parse_args(): """ parse args . Args: Returns: args. Examples: >>> parse_args() """ parser = ArgumentParser(description="mindspore distributed training launch " "helper utility that will generate hccl" " config file") parser.add_argument("--device_num", type=str, default="[0,8)", help="The number of the Ascend accelerators used. please note that the Ascend accelerators" "used must be continuous, such [0,4) means using four chips " "0，1，2，3; [0,1) means using chip 0; In the most Ascend system, " "the first four chips belong to one group, and the last four chips belong to another one." "Only full chips are allowed to cross-group such as [0,8), other cross-group such as [3,6)" "are prohibited.") parser.add_argument("--visible_devices", type=str, default="0,1,2,3,4,5,6,7", help="The visible devices according to the software system. " "Usually used in the virtual system or docker container " "that makes the device_id dismatch logic_id. --device_num uses logic_id. " "For example \"4,5,6,7\" means the system has 4 logic chips " "which are actually the last 4 chips in hardware " "while `--device_num` could only be set to \"[0, 4)\" instead of \"[4, 8)\"") parser.add_argument("--server_ip", type=str, default="", help="Set the server_ip manually, to avoid errors in auto detection.") args = parser.parse_args() return args def get_host_ip(): """ get host ip """ ip = None try: hostname = socket.gethostname() ip = socket.gethostbyname(hostname) except EOFError: pass return ip def main(): print("start", __file__) args = parse_args() # visible_devices visible_devices = args.visible_devices.split(',') print('visible_devices:{}'.format(visible_devices)) # server_id ip = get_host_ip() if args.server_ip: server_id = args.server_ip elif ip: server_id = ip else: raise ValueError("please input server ip!") print('server_id:{}'.format(server_id)) # device_num first_num = int(args.device_num[1]) last_num = int(args.device_num[3]) if first_num < 0 or last_num > 8: raise ValueError("device num {} must be in range [0,8] !".format(args.device_num)) if first_num > last_num: raise ValueError("First num {} of device num {} must less than last num {} !".format(first_num, args.device_num, last_num)) if first_num < 4 < last_num: if first_num == 0 and last_num == 8: pass else: raise ValueError("device num {} must be in the same group of [0,4] or [4,8] !".format(args.device_num)) device_num_list = list(range(first_num, last_num)) print("device_num_list:", device_num_list) assert len(visible_devices) >= len(device_num_list) # construct hccn_table device_ips: Dict[Any, Any] = {} try: for device_id in device_num_list: ret = os.popen("hccn_tool -i %d -ip -g" % device_id).readlines() device_ips[str(device_id)] = ret[0].split(":")[1].replace('\n', '') except IndexError: print("Failed to call hccn_tool, try to read /etc/hccn.conf instead") try: with open('/etc/hccn.conf', 'r') as fin: for hccn_item in fin.readlines(): if hccn_item.strip().startswith('address_'): device_id, device_ip = hccn_item.split('=') device_id = device_id.split('_')[1] device_ips[device_id] = device_ip.strip() except OSError: print("Failed to read /etc/hccn.conf") raise SystemError("Failed to find information for hccl") hccn_table = {'version': '1.0', 'server_count': '1', 'server_list': []} device_list = [] rank_id = 0 for instance_id in device_num_list: device_id = visible_devices[instance_id] device_ip = device_ips[device_id] device = {'device_id': device_id, 'device_ip': device_ip, 'rank_id': str(rank_id)} print('rank_id:{}, device_id:{}, device_ip:{}'.format(rank_id, device_id, device_ip)) rank_id += 1 device_list.append(device) hccn_table['server_list'].append({ 'server_id': server_id, 'device': device_list, 'host_nic_ip': 'reserve' }) hccn_table['status'] = 'completed' # save hccn_table to file table_path = os.getcwd() table_fn = os.path.join(table_path, 'hccl_{}p_{}_{}.json'.format(len(device_num_list), "".join(map(str, device_num_list)), server_id)) with open(table_fn, 'w') as table_fp: json.dump(hccn_table, table_fp, indent=4) sys.stdout.flush() print("Completed: hccl file was save in :", table_fn) if __name__ == "__main__": main() pengcheng.mind-mindformer/mindformers/tools/image_tools.py000066400000000000000000000036051476650141400244570ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ ''' Image tools ''' import os import requests import urllib3 import PIL import mindspore as ms def load_image(content, timeout=4): """load image""" if isinstance(content, str): if content.startswith("https://") or content.startswith("http://"): try: response = requests.get(content, stream=True, timeout=timeout) except (TimeoutError, urllib3.exceptions.MaxRetryError, requests.exceptions.ProxyError) as exc: raise ConnectionError(f"Connect error, please download {content}.") from exc content = PIL.Image.open(response.raw) elif os.path.isfile(content): content = PIL.Image.open(content) else: raise ValueError( f"{content} is not a valid path. If URL, it must start with `http://` or `https://`." ) if isinstance(content, ms.Tensor): return content if not isinstance(content, PIL.Image.Image): raise ValueError( "Input should be an url linking to an image," " a local path, a Mindspore Tensor, or a PIL image." ) content = PIL.ImageOps.exif_transpose(content) content = content.convert("RGB") return content pengcheng.mind-mindformer/mindformers/tools/logger.py000066400000000000000000000444021476650141400234340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """LOGGER Module""" import logging import logging.config import logging.handlers import os import stat import sys from functools import wraps from tempfile import TemporaryFile from typing import Dict, List, Tuple, Union from mindformers.tools.utils import get_num_nodes_devices, get_rank_info, \ convert_nodes_devices_input, generate_rank_list,\ check_in_modelarts, check_list, LOCAL_DEFAULT_PATH,\ get_output_root_path logger_list = [] LEVEL = ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL') MODELARTS_LOG_FILE_DIR = '/cache/ma-user-work/' LOCAL_DEFAULT_LOG_FILE_DIR = os.path.join( os.getenv("LOCAL_DEFAULT_PATH", LOCAL_DEFAULT_PATH), 'log') RANK_DIR_FORMATTER = 'rank_{}/' DEFAULT_FILEHANDLER_FORMAT = '[%(levelname)s] %(asctime)s ' \ '[%(pathname)s:%(lineno)d] %(funcName)s: %(message)s' DEFAULT_STDOUT_FORMAT = '%(asctime)s - %(name)s[%(filename)s:%(lineno)d] - %(levelname)s - %(message)s' DEFAULT_REDIRECT_FILE_NAME = 'mindspore.log' def judge_redirect(rank_id: int, rank_size: int, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None): """Determine if the stderr of this process needs to be redirected. Args: rank_id (int): Rank id. rank_size (int): Rank Size. redirect_nodes (list or tuple or dict or None): Node list. The nodes in the list will redirect stderr. redirect_devices (list or tuple or dict or None): Device list. The devices in the list will redirect stderr. Returns: prerequisite (bool): If true, stderr will redirect. """ is_redirect = True if rank_size > 1 and (redirect_nodes is not None or redirect_devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) redirect_nodes = convert_nodes_devices_input(redirect_nodes, num_nodes) redirect_devices = convert_nodes_devices_input(redirect_devices, num_devices) check_list('nodes', redirect_nodes, num_nodes) check_list('devices', redirect_devices, num_devices) rank_list = generate_rank_list(redirect_nodes, redirect_devices) if rank_id not in rank_list: is_redirect = False return is_redirect class StreamRedirector: """Stream Re-director for Log.""" def __init__(self, source_stream, target_stream): """Redirects the source stream to the target stream. Args: source_stream: Source stream. target_stream: Target stream. """ super(StreamRedirector, self).__init__() self.source_stream = source_stream self.target_stream = target_stream self.save_source_stream_fd = os.dup(self.source_stream.fileno()) def __call__(self, func): @wraps(func) def wrapper(*args, **kwargs): self.start() func(*args, **kwargs) self.stop() return wrapper def __enter__(self): self.start() def __exit__(self, exc_type, exc_val, exc_tb): self.stop() def start(self): """start.""" self.source_stream.flush() os.dup2(self.target_stream.fileno(), self.source_stream.fileno()) def stop(self): """stop.""" self.source_stream.flush() os.dup2(self.save_source_stream_fd, self.source_stream.fileno()) self.target_stream.flush() class AiLogFastStreamRedirect2File(StreamRedirector): """AiLogFastStreamRedirect2File.""" def __init__(self, source_stream=None, redirect_nodes: Union[List, Tuple, Dict[str, int], None] = None, redirect_devices: Union[List, Tuple, Dict[str, int], None] = None, **kwargs): """Redirect stream to file. Args: source_stream (file object or None): Streams that need to be redirected. Default: None, select stderr. redirect_nodes (list[int] or tuple[int] or optional): The computation nodes that will redirect stderr. Default is None: indicates that all nodes will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all redirect stderr. redirect_devices (list[int] or tuple[int] or optional): The computation devices that will redirect stderr. Default is None, indicates that all devices will redirect stderr. Eg [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all redirect stderr. kwargs (dict): File-related parameters. file_save_dir (str): The folder where the files that save redirected stream are saved. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (str): Redirect file name. """ rank_id, rank_size = get_rank_info() self.is_redirect = judge_redirect(rank_id=rank_id, rank_size=rank_size, redirect_nodes=redirect_nodes, redirect_devices=redirect_devices) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', '') if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') if append_rank_dir: rank_str = RANK_DIR_FORMATTER.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) if not file_name: file_name = DEFAULT_REDIRECT_FILE_NAME self.file_path = os.path.join(file_save_dir, file_name) self.file_save_dir = os.path.dirname(self.file_path) if source_stream is None: source_stream = sys.stderr target_stream = TemporaryFile(mode='w+') super(AiLogFastStreamRedirect2File, self).__init__(source_stream=source_stream, target_stream=target_stream) def start(self): if self.is_redirect: super(AiLogFastStreamRedirect2File, self).start() def stop(self): if self.is_redirect: self.target_stream.flush() if not os.path.exists(self.file_save_dir): os.makedirs(self.file_save_dir, exist_ok=True) self.target_stream.seek(0, 0) flags = os.O_WRONLY | os.O_CREAT modes = stat.S_IWUSR | stat.S_IRUSR with os.fdopen(os.open(self.file_path, flags, modes), 'w') as fp: for line in self.target_stream: fp.write(line) super(AiLogFastStreamRedirect2File, self).stop() self.target_stream.close() def judge_stdout(rank_id: int, rank_size: int, is_output: bool, nodes: Union[List, Tuple, Dict[str, int], None] = None, devices: Union[List, Tuple, Dict[str, int], None] = None) -> bool: """Determines if logs will be output. Args: rank_id (int): Rank id. rank_size (int): Rank size. is_output (int): If set to true, logs or others will be output. nodes (list or tuple or dict or None): Node list. The nodes in the list will output the log to stdout. devices (list or tuple or dict or None): Device list. The devices in the list or output the log to stdout. Returns: is_output (bool): If set to true, logs or others will be output or redirect. """ if is_output and rank_size > 1 and (nodes is not None or devices is not None): num_nodes, num_devices = get_num_nodes_devices(rank_size) stdout_nodes = convert_nodes_devices_input(nodes, num_nodes) stdout_devices = convert_nodes_devices_input(devices, num_devices) check_list('nodes', stdout_nodes, num_nodes) check_list('devices', stdout_devices, num_devices) rank_list = generate_rank_list(stdout_nodes, stdout_devices) if rank_id not in rank_list: is_output = False return is_output def validate_nodes_devices_input(var_name: str, var): """Check the list of nodes or devices. Args: var_name (str): Variable name. var: The name of the variable to be checked. Returns: None """ if not (var is None or isinstance(var, (list, tuple, dict))): raise TypeError('The value of {} can be None or a value of type tuple, ' 'list, or dict.'.format(var_name)) if isinstance(var, (list, tuple)): for item in var: if not isinstance(item, int): raise TypeError('The elements of a variable of type list or ' 'tuple must be of type int.') def validate_level(var_name: str, var): """Verify that the log level is correct. Args: var_name (str): Variable name. var: The name of variable to be checked. Returns: None """ if not isinstance(var, str): raise TypeError('The format of {} must be of type str.'.format(var_name)) if var not in LEVEL: raise ValueError('{}={} needs to be in {}'.format(var_name, var, LEVEL)) def validate_std_input_format(to_std: bool, stdout_nodes: Union[List, Tuple, None], stdout_devices: Union[List, Tuple, None], stdout_level: str): """Validate the input about stdout of the get_logger function.""" if not isinstance(to_std, bool): raise TypeError('The format of the to_std must be of type bool.') validate_nodes_devices_input('stdout_nodes', stdout_nodes) validate_nodes_devices_input('stdout_devices', stdout_devices) validate_level('stdout_level', stdout_level) def validate_file_input_format(file_level: Union[List[str], Tuple[str]], file_save_dir: str, append_rank_dir: str, file_name: Union[List[str], Tuple[str]]): """Validate the input about file of the get_logger function.""" if not isinstance(file_level, (tuple, list)): raise TypeError('The value of file_level should be list or a tuple.') for level in file_level: validate_level('level in file_level', level) if not len(file_level) == len(file_name): raise ValueError('The length of file_level and file_name should be equal.') if not isinstance(file_save_dir, str): raise TypeError('The value of file_save_dir should be a value of type str.') if not isinstance(append_rank_dir, bool): raise TypeError('The value of append_rank_dir should be a value of type bool.') if not isinstance(file_name, (tuple, list)): raise TypeError('The value of file_name should be list or a tuple.') for name in file_name: if not isinstance(name, str): raise TypeError('The value of name in file_name should be a value of type str.') def _convert_level(level: str) -> int: """Convert the format of the log to logging level. Args: level (str): User log level. Returns: level (str): Logging level. """ level_convert = { 'DEBUG': logging.DEBUG, 'INFO': logging.INFO, 'WARNING': logging.WARNING, 'ERROR': logging.ERROR, 'CRITICAL': logging.CRITICAL } level = level_convert.get(level, logging.INFO) return level def get_logger(logger_name: str = 'mindformers', **kwargs) -> logging.Logger: """Get the logger. Both computing centers and bare metal servers are available. Args: logger_name (str): Logger name. kwargs (dict): Other input. to_std (bool): If set to True, output the log to stdout. stdout_nodes (list[int] or tuple[int] or optional): The computation nodes that will output the log to stdout. default: None, indicates that all nodes will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that nodes 0, 1, 2, and 3 all output logs to stdout. stdout_devices (list[int] or tuple[int] or optional): The computation devices that will output the log to stdout. default: None, indicates that all devices will output logs to stdout. eg: [0, 1, 2, 3] or (0, 1, 2, 3): indicates that devices 0, 1, 2, and 3 all output logs to stdout. stdout_level (str): The level of the log output to stdout. If the type is str, the options are DEBUG, INFO, WARNING, ERROR, CRITICAL. stdout_format (str): Log format. file_level (list[str] or tuple[str]): The level of the log output to file. eg: ['INFO', 'ERROR'] Indicates that the logger will output logs above the level INFO and ERROR in the list to the corresponding file. The length of the list needs to be the same as the length of file_name. file_save_dir (str): The folder where the log files are stored. append_rank_dir (bool): Whether to add a folder with the format rank{}. file_name (list[str] or list[tuple]): Store a list of output file names. max_file_size (int): The maximum size of a single log file. Unit: MB. max_num_of_files (int): The maximum number of files to save. Returns: logger (logging.Logger): Logger. """ mf_logger = logging.getLogger(logger_name) if logger_name in logger_list: return mf_logger to_std = kwargs.get('to_std', True) stdout_nodes = kwargs.get('stdout_nodes', None) rank_id, rank_size = get_rank_info() def get_stdout_devices(): if os.getenv("STDOUT_DEVICES"): devices = os.getenv("STDOUT_DEVICES") if devices.startswith(("(", "[")) and devices.endswith((")", "]")): devices = devices[1:-1] devices = tuple(map(lambda x: int(x.strip()), devices.split(","))) elif check_in_modelarts(): devices = kwargs.get('stdout_devices', (min(rank_size - 1, 7),)) else: devices = kwargs.get('stdout_devices', None) return devices stdout_devices = get_stdout_devices() stdout_level = kwargs.get('stdout_level', 'INFO') stdout_format = kwargs.get('stdout_format', '') file_level = kwargs.get('file_level', ('INFO', 'ERROR')) file_save_dir = kwargs.get('file_save_dir', '') append_rank_dir = kwargs.get('append_rank_dir', True) file_name = kwargs.get('file_name', (f'info.log', 'error.log')) max_file_size = kwargs.get('max_file_size', 50) max_num_of_files = kwargs.get('max_num_of_files', 5) validate_std_input_format(to_std, stdout_nodes, stdout_devices, stdout_level) validate_file_input_format(file_level, file_save_dir, append_rank_dir, file_name) to_std = judge_stdout(rank_id=rank_id, rank_size=rank_size, is_output=to_std, nodes=stdout_nodes, devices=stdout_devices) if to_std: if not stdout_format: stdout_format = DEFAULT_STDOUT_FORMAT stream_handler = logging.StreamHandler(sys.stdout) stream_handler.setLevel(_convert_level(stdout_level)) stream_formatter = logging.Formatter(stdout_format) stream_handler.setFormatter(stream_formatter) mf_logger.addHandler(stream_handler) logging_level = [] for level in file_level: logging_level.append(_convert_level(level)) if not file_save_dir: file_save_dir = os.path.join(get_output_root_path(), 'log') rank_dir = RANK_DIR_FORMATTER if append_rank_dir: rank_str = rank_dir.format(rank_id) file_save_dir = os.path.join(file_save_dir, rank_str) file_path = [] for name in file_name: path = os.path.join(file_save_dir, name) path = os.path.realpath(path) base_dir = os.path.dirname(path) if not os.path.exists(base_dir): os.makedirs(base_dir, exist_ok=True) file_path.append(path) max_file_size = max_file_size * 1024 * 1024 file_formatter = logging.Formatter(DEFAULT_FILEHANDLER_FORMAT) for i, level in enumerate(file_level): file_handler = logging.handlers.RotatingFileHandler(filename=file_path[i], maxBytes=max_file_size, backupCount=max_num_of_files) file_handler.setLevel(level) file_handler.setFormatter(file_formatter) mf_logger.addHandler(file_handler) mf_logger.setLevel(_convert_level('INFO')) mf_logger.propagate = False logger_list.append(logger_name) return mf_logger class _LogActionOnce: """ A wrapper for modify the warning logging to an empty function. This is used when we want to only log once to avoid the repeated logging. Args: logger (logging): The logger object. """ is_logged = dict() def __init__(self, m_logger, key, no_warning=False): self.logger = m_logger self.key = key self.no_warning = no_warning def __call__(self, func): def wrapper(*args, **kwargs): if not hasattr(self.logger, 'warning'): return func(*args, **kwargs) old_func = self.logger.warning if self.no_warning or self.key in _LogActionOnce.is_logged: self.logger.warning = lambda x: x else: _LogActionOnce.is_logged[self.key] = True res = func(*args, **kwargs) if hasattr(self.logger, 'warning'): self.logger.warning = old_func return res return wrapper logger = get_logger() pengcheng.mind-mindformer/mindformers/tools/merge_hccl.py000066400000000000000000000044141476650141400242440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """generate hccl config file script""" import os import sys import json from argparse import ArgumentParser def parse_args(): """ parse args . Args: Returns: args. Examples: >>> parse_args() """ parser = ArgumentParser(description="Merge several hccl config json files" "of single server into one config file of the whole cluster") parser.add_argument("file_list", type=str, nargs="+", help="Hccl file lists") arg = parser.parse_args() return arg if __name__ == "__main__": args = parse_args() print(args.file_list) server_count = 0 json_list = [] for f_name in args.file_list: with open(f_name) as f: f_json = json.load(f) json_list.append(f_json) server_count += int(f_json['server_count']) hccl_table = {'version': '1.0', 'server_count': f'{server_count}', 'server_list': []} rank_id = 0 for j in json_list: server_list = j['server_list'] for server in server_list: for device in server['device']: device['rank_id'] = str(rank_id) rank_id += 1 hccl_table['server_list'].extend(server_list) hccl_table['status'] = 'completed' table_path = os.getcwd() table_name = os.path.join(table_path, 'hccl_{}s_{}p.json'.format(server_count, rank_id)) with open(table_name, 'w') as table_fp: json.dump(hccl_table, table_fp, indent=4) sys.stdout.flush() print("Completed: hccl file was save in :", table_name) pengcheng.mind-mindformer/mindformers/tools/register/000077500000000000000000000000001476650141400234235ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/register/__init__.py000066400000000000000000000016111476650141400255330ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Register Tools.""" from .config import MindFormerConfig, ActionDict from .register import MindFormerRegister, MindFormerModuleType __all__ = ['MindFormerRegister', 'MindFormerModuleType', 'ActionDict', 'MindFormerConfig'] pengcheng.mind-mindformer/mindformers/tools/register/__pycache__/000077500000000000000000000000001476650141400256335ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/tools/register/__pycache__/__init__.cpython-39.pyc000066400000000000000000000006241476650141400320250ustar00rootroot00000000000000a ?cf@s0dZddlmZmZddlmZmZgdZdS)zMindFormers Register Tools.)MindFormerConfig ActionDict)MindFormerRegisterMindFormerModuleType)rrrrN)__doc__configrrregisterrr__all__r r N/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/__init__.pyspengcheng.mind-mindformer/mindformers/tools/register/__pycache__/config.cpython-39.pyc000066400000000000000000000232211476650141400315310ustar00rootroot00000000000000a ?cf+@sdZddlZddlZddlZddlmZddlmZddlZdZGddde Z Gdd d eZ ej efd d Z dejefd d ZddZdS)z& Transformer-Config dict parse module N)Action) OrderedDictZ base_configcspeZdZdZfddZddZddZdd Zdd d Zd dZ e ddZ e dddZ e ddZ ZS)MindFormerConfiga A Config class is inherit from dict. Config class can parse arguments from a config file of yaml or a dict. Args: args (list) : config filenames kwargs (dict) : config dictionary list Example: test.yaml: a:1 >>> cfg = MindFormerConfig('./test.yaml') >>> cfg.a 1 >>> cfg = MindFormerConfig(**dict(a=1, b=dict(c=[0,1]))) >>> cfg.b {'c': [0, 1]} csptt|i}|D]6}t|tr|ds8|drt|}||q|dur`||t||dS)NyamlZyml) superr__init__ isinstancestrendswith _file2dictupdate _dict2config)selfargskwargscfg_dictargZraw_dict __class__L/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/config.pyr1s    zMindFormerConfig.__init__cCs||vr dS||S)zGet a object attr by its `key` Args: key (str) : the name of object attr. Returns: attr of object that name is `key` Nrrkeyrrr __getattr__Bs zMindFormerConfig.__getattr__cCs |||<dS)zSet a object value `key` with `value` Args: key (str) : The name of object attr. value : the `value` need to set to the target object attr. Nr)rrvaluerrr __setattr__PszMindFormerConfig.__setattr__cCs ||=dS)zkDelete a object attr by its `key`. Args: key (str) : The name of object attr. Nrrrrr __delattr__YszMindFormerConfig.__delattr__Nc Cs:t}|D]&}|t||t|||q|S)zDeep copy operation on arbitrary MindFormerConfig objects. Args: memo (dict) : Objects that already copied. Returns: MindFormerConfig : The deep copy of the given MindFormerConfig object. )rkeysrcopydeepcopyr)rmemoconfigrrrr __deepcopy__as  zMindFormerConfig.__deepcopy__c Csxi}|D]N\}}|}|d}|ddD]}||t||}q.|d}|||<q t||}t||dS)a:Merge options into config file. Args: options (dict): dict of configs to merge from Examples: >>> options = {'model.arch': 'simmim'} >>> cfg = MindFormerConfig(**dict(model=dict(backbone=dict(type='vit')))) >>> cfg.merge_from_dict(options) .N)itemssplit setdefaultr_merge_a_into_br ) roptionsZoption_cfg_dictfull_keyrdkey_listZsub_key merge_dictrrrmerge_from_dictos     z MindFormerConfig.merge_from_dictcCsN|}|D]8\}}t|tr@||vr@t|||||<q|||<q|S)a1Merge dict ``a`` into dict ``b`` Values in ``a`` will overwrite ``b`` Args: a (dict) : The source dict to be merged into b. b (dict) : The origin dict to be fetch keys from ``a``. Returns: dict: The modified dict of ``b`` using ``a``. )rr%rdictrr()abkvrrrr(s  z MindFormerConfig._merge_a_into_bc Cs|durtd|tj|}t|dd}t|tjd}Wdn1sR0Yt |vrtj |}| t }t |t r|n|g}t }|D]"}ttj||}||qt} |D]} | | qt|| } | }|S)zdConvert config file to dictionary. Args: filename (str) : config file. NzThis {} cannot be empty.zutf-8)encoding) yaml_loader) NameErrorformatospathrealpathopenordered_yaml_loadrZ FullLoader BASE_CONFIGdirnamepoprlistrr joinappendr/r r() filenamefilepathfprZcfg_dirZbase_filenamesZ cfg_dict_list base_filenameZ cfg_dict_itemZ base_cfg_dictcfgrrrr s2 ,      zMindFormerConfig._file2dictcCsXt|trT|D]@\}}t|trFt}t|||t||q||||<qdS)zConvert dictionary to config. Args: config : Config object dic (dict) : dictionary Returns: Exceptions: N)rr/r%r __setitem__r )r!dicrrZ sub_configrrrr s  zMindFormerConfig._dict2config)N)N)__name__ __module__ __qualname____doc__rrrrr"r. staticmethodr(r r __classcell__rrrrrs     $rc@s>eZdZdZeddZeddZeddZd d d ZdS) ActionDicta Argparse action to split an option into KEY=VALUE from on the first = and append to dictionary. List options can be passed as comma separated values. i.e. 'KEY=Val1,Val2,Val3' or with explicit brackets i.e. 'KEY=[Val1,Val2,Val3]'. cCsVz t|WStyYn0z t|WSty:Yn0|dvrR|jdkS|S)z9convert string val to int or float or bool or do nothing.)TRUEFALSErQ)int ValueErrorfloatupper)valrrr_parse_int_float_bools      z ActionDict._parse_int_float_boolcCs|d|dkr(|d|dks,Jt|}t|D]L\}}|d|}|dkr<|d|dkr<|d|dkr<|}qq<|S)zfind the position of next comma in the string. note: '(' and ')' or '[' and']' must appear in pairs or not exist. ()[]N,)countlen enumerate)Zval_strendidxcharprerrrfind_next_commas zActionDict.find_next_commacCs|ddd}d}|dr<|dr>> ActionDict._parse_value_iter('1,2,3') [1,2,3] >>> ActionDict._parse_value_iter('[1,2,3]') [1,2,3] >>> ActionDict._parse_value_iter('(1,2,3)') (1,2,3) >>> ActionDict._parse_value_iter('[1,[1,2],(1,2,3)') [1, [1, 2], (1, 2, 3)] z'" FrYrZTr$r[r\r]rN) stripreplace startswithr rPrXr_re_parse_value_iterrBtuple)rWis_tuplevaluesZ len_of_valZ comma_idxelerrrrls(    zActionDict._parse_value_iterNc Cs@i}|D]$}|jddd\}}||||<qt||j|dS)N=rh)maxsplit)r&rlsetattrdest) rparser namespacero option_stringr) key_valuerrrrr__call__1s zActionDict.__call__)N) rJrKrLrMrNrXrerlryrrrrrPs   0rPcs:Gddd|}fdd}|tjjj|t||S)zLoad Yaml File in Orderedly.c@s eZdZdS)z(ordered_yaml_load..OrderedLoaderNrJrKrLrrrr OrderedLoader<sr{cs||||SN)Zflatten_mappingZconstruct_pairs)loadernodeobject_pairs_hookrr_construct_mapping?s z-ordered_yaml_load.._construct_mapping)Zadd_constructorrresolver BaseResolverDEFAULT_MAPPING_TAGload)streamr5rr{rrrrr<9s r<cKs:Gddd|}dd}|||tj|||fi|S)z$Dump Dict to Yaml File in Orderedly.c@s eZdZdS)z(ordered_yaml_dump..OrderedDumperNrzrrrr OrderedDumperLsrcSs|tjjj|Sr|)Zrepresent_mappingrrrrr%)Zdumperdatarrr_dict_representerOsz,ordered_yaml_dump.._dict_representer)Zadd_representerrdump)rrZ yaml_dumperrrrrrrrordered_yaml_dumpIs rcCs&td}|jddtddd|S)zh Parse arguments from `yaml or yml` config file. Returns: object: argparse object. zTransformer Config.z-cz--configrgz(Enter the path of the model config file.)typedefaulthelp)argparseArgumentParser add_argumentr parse_args)rurrrrXs r)rMr8rrr collectionsrrr=r/rrPZ SafeLoaderr<Z SafeDumperrrrrrrs"  ;d  pengcheng.mind-mindformer/mindformers/tools/register/__pycache__/register.cpython-39.pyc000066400000000000000000000131121476650141400321060ustar00rootroot00000000000000a ?cf@s,dZddlZGdddZGdddZdS)z' Class Register Module For MindFormers.Nc@seZdZdZddZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdS)MindFormerModuleTypez%Class module type for vision pretraincCsdSNselfrrN/home/ma-user/work/mindformers-baichuan/mindformers/tools/register/register.py__init__szMindFormerModuleType.__init__trainerpipeline processor tokenizerdatasetZ mask_policyZdataset_loaderZdataset_sampler transformsencodermodelsmodules transformerZ base_layercoreheadlosslr optimizercontextcallbackwrappermetricconfigtoolsZfeature_extractorN)__name__ __module__ __qualname____doc__rZTRAINERZPIPELINEZ PROCESSORZ TOKENIZERZDATASETZ MASK_POLICYZDATASET_LOADERZDATASET_SAMPLERZ TRANSFORMSZENCODERZMODELSMODULESZ TRANSFORMERZ BASE_LAYERZCOREZHEADZLOSSLRZ OPTIMIZERCONTEXTZCALLBACKZWRAPPERZMETRICCONFIGTOOLSZFEATURE_EXTRACTORrrrrrs8rc@seZdZdZddZiZeejdfddZ eejdfddZ edd d Z edd d Z eejdfd dZ eejdfddZdS)MindFormerRegisterz+ Module class factory for ring-mo. cCsdSrrrrrrr=szMindFormerRegister.__init__Ncsfdd}|S)zRegister class into registry Args: module_type (ModuleType) : module type name, default ModuleType.GENERAL alias (str) : class alias Returns: wrapper cs>dur n|j}jvr,||ij<n|j|<|S)zRegister-Class with wrapper function. Args: register_class : class need to register Returns: wrapper of register_class Nrregistry)register_class class_namealiascls module_typerrrNs  z,MindFormerRegister.register..wrapperr)r.r/r-rrr,rregisterBs zMindFormerRegister.registercCs>|dur |n|j}||jvr,||i|j|<n||j||<|S)zRegister class with type name. Args: register_class : class need to register module_type : module type name, default ModuleType.COMMON alias : class name Nr()r.r*r/r-r+rrr register_cls`s  zMindFormerRegister.register_clscCs,|s||jvS||jvo&||j|v}|S)zDetermine whether class name is in the current type group. Args: module_type : Module type class_name : class name Returns: True/False )r)get)r.r/r+ registeredrrris_existps  zMindFormerRegister.is_existcCsD|||std|||s.td||j||}|S)zGet class Args: module_type : Module type class_name : class name Returns: register_class DCan't find class type {} class name {} in class registryz!Can't find class. class type = {})r4 ValueErrorformatr)r2)r.r/r+r*rrrget_clss zMindFormerRegister.get_clsc Cs,t|tstdt|d|vr4td||t|tsX|dusXtdt||}|dur|D]&\}}||vr|||qp|||<qp| d}t|t r| ||}n t |r|}ntdt|z|fi|WSty&} z"t| d|j| WYd} ~ n d} ~ 00dS)a%Get instance. Args: cfg (dict) : Config dict. It should at least contain the key "type". module_type : module type default_args (dict, optional) : Default initialization arguments. Returns: obj : The constructed object. z Cfg must be a Config, but got {}typezA`cfg` or `default_args` must contain the key "type",but got {} {}Nz-default_args must be a dict or Nonebut got {}r5{}: {}) isinstancedict TypeErrorr7r9KeyErrorcopyitems setdefaultpopstrr8inspectisclassr6 Exceptionr) r.cfgr/ default_argsargskvobj_typeobj_clserrrget_instance_from_cfgsD      z(MindFormerRegister.get_instance_from_cfgc Ks|durtdt|tr(|||}n t|r8|}ntdt|z|fi|WSty}z"t|d|j |WYd}~n d}~00dS)zGet instance. Args: module_type : module type class_name : class type Returns: object : The constructed object NzClass name cannot be None.zECan't find class type {} class name {} in class registry.r:) r6r;rCr8rDrEr7r9rFr)r.r/r+kwargsrMrNrrr get_instances   zMindFormerRegister.get_instance)N)N)rrr r!rr) classmethodrr&r0r1r4r8rOrQrrrrr'8s  .r')r!rDrr'rrrrs"pengcheng.mind-mindformer/mindformers/tools/register/config.py000066400000000000000000000254351476650141400252530ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Transformer-Config dict parse module """ import os import copy import argparse from argparse import Action from collections import OrderedDict import yaml BASE_CONFIG = 'base_config' class MindFormerConfig(dict): """ A Config class is inherit from dict. Config class can parse arguments from a config file of yaml or a dict. Args: args (list) : config filenames kwargs (dict) : config dictionary list Example: test.yaml: a:1 >>> cfg = MindFormerConfig('./test.yaml') >>> cfg.a 1 >>> cfg = MindFormerConfig(**dict(a=1, b=dict(c=[0,1]))) >>> cfg.b {'c': [0, 1]} """ def __init__(self, *args, **kwargs): super(MindFormerConfig, self).__init__() cfg_dict = {} # load from file for arg in args: if isinstance(arg, str): if arg.endswith('yaml') or arg.endswith('yml'): raw_dict = MindFormerConfig._file2dict(arg) cfg_dict.update(raw_dict) # load dictionary configs if kwargs is not None: cfg_dict.update(kwargs) MindFormerConfig._dict2config(self, cfg_dict) def __getattr__(self, key): """Get a object attr by its `key` Args: key (str) : the name of object attr. Returns: attr of object that name is `key` """ if key not in self: return None return self[key] def __setattr__(self, key, value): """Set a object value `key` with `value` Args: key (str) : The name of object attr. value : the `value` need to set to the target object attr. """ self[key] = value def __delattr__(self, key): """Delete a object attr by its `key`. Args: key (str) : The name of object attr. """ del self[key] def __deepcopy__(self, memo=None): """Deep copy operation on arbitrary MindFormerConfig objects. Args: memo (dict) : Objects that already copied. Returns: MindFormerConfig : The deep copy of the given MindFormerConfig object. """ config = MindFormerConfig() for key in self.keys(): config.__setattr__(copy.deepcopy(key, memo), copy.deepcopy(self.__getattr__(key), memo)) return config def merge_from_dict(self, options): """Merge options into config file. Args: options (dict): dict of configs to merge from Examples: >>> options = {'model.arch': 'simmim'} >>> cfg = MindFormerConfig(**dict(model=dict(backbone=dict(type='vit')))) >>> cfg.merge_from_dict(options) """ option_cfg_dict = {} for full_key, value in options.items(): d = option_cfg_dict key_list = full_key.split('.') for sub_key in key_list[:-1]: d.setdefault(sub_key, MindFormerConfig()) d = d[sub_key] sub_key = key_list[-1] d[sub_key] = value merge_dict = MindFormerConfig._merge_a_into_b(option_cfg_dict, self) MindFormerConfig._dict2config(self, merge_dict) @staticmethod def _merge_a_into_b(a, b): """Merge dict ``a`` into dict ``b`` Values in ``a`` will overwrite ``b`` Args: a (dict) : The source dict to be merged into b. b (dict) : The origin dict to be fetch keys from ``a``. Returns: dict: The modified dict of ``b`` using ``a``. """ b = b.copy() for k, v in a.items(): if isinstance(v, dict) and k in b: b[k] = MindFormerConfig._merge_a_into_b(v, b[k]) else: b[k] = v return b @staticmethod def _file2dict(filename=None): """Convert config file to dictionary. Args: filename (str) : config file. """ if filename is None: raise NameError('This {} cannot be empty.'.format(filename)) filepath = os.path.realpath(filename) with open(filepath, encoding='utf-8') as fp: cfg_dict = ordered_yaml_load(fp, yaml_loader=yaml.FullLoader) # Load base config file. if BASE_CONFIG in cfg_dict: cfg_dir = os.path.dirname(filename) base_filenames = cfg_dict.pop(BASE_CONFIG) base_filenames = base_filenames if isinstance( base_filenames, list) else [base_filenames] cfg_dict_list = list() for base_filename in base_filenames: cfg_dict_item = MindFormerConfig._file2dict( os.path.join(cfg_dir, base_filename)) cfg_dict_list.append(cfg_dict_item) base_cfg_dict = dict() for cfg in cfg_dict_list: base_cfg_dict.update(cfg) # Merge config base_cfg_dict = MindFormerConfig._merge_a_into_b(cfg_dict, base_cfg_dict) cfg_dict = base_cfg_dict return cfg_dict @staticmethod def _dict2config(config, dic): """Convert dictionary to config. Args: config : Config object dic (dict) : dictionary Returns: Exceptions: """ if isinstance(dic, dict): for key, value in dic.items(): if isinstance(value, dict): sub_config = MindFormerConfig() dict.__setitem__(config, key, sub_config) MindFormerConfig._dict2config(sub_config, value) else: config[key] = dic[key] class ActionDict(Action): """ Argparse action to split an option into KEY=VALUE from on the first = and append to dictionary. List options can be passed as comma separated values. i.e. 'KEY=Val1,Val2,Val3' or with explicit brackets i.e. 'KEY=[Val1,Val2,Val3]'. """ @staticmethod def _parse_int_float_bool(val): """convert string val to int or float or bool or do nothing.""" try: return int(val) except ValueError: pass try: return float(val) except ValueError: pass if val.upper() in ['TRUE', 'FALSE']: return val.upper == 'TRUE' return val @staticmethod def find_next_comma(val_str): """find the position of next comma in the string. note: '(' and ')' or '[' and']' must appear in pairs or not exist. """ assert (val_str.count('(') == val_str.count(')')) and \ (val_str.count('[') == val_str.count(']')) end = len(val_str) for idx, char in enumerate(val_str): pre = val_str[:idx] if ((char == ',') and (pre.count('(') == pre.count(')')) and (pre.count('[') == pre.count(']'))): end = idx break return end @staticmethod def _parse_value_iter(val): """Convert string format as list or tuple to python list object or tuple object. Args: val (str) : Value String Returns: list or tuple Examples: >>> ActionDict._parse_value_iter('1,2,3') [1,2,3] >>> ActionDict._parse_value_iter('[1,2,3]') [1,2,3] >>> ActionDict._parse_value_iter('(1,2,3)') (1,2,3) >>> ActionDict._parse_value_iter('[1,[1,2],(1,2,3)') [1, [1, 2], (1, 2, 3)] """ # strip ' and " and delete whitespace val = val.strip('\'\"').replace(" ", "") is_tuple = False if val.startswith('(') and val.endswith(')'): is_tuple = True # remove start '(' and end ')' val = val[1:-1] elif val.startswith('[') and val.endswith(']'): # remove start '[' and end ']' val = val[1:-1] elif ',' not in val: return ActionDict._parse_int_float_bool(val) values = [] len_of_val = len(val) while len_of_val > 0: comma_idx = ActionDict.find_next_comma(val) ele = ActionDict._parse_value_iter(val[:comma_idx]) values.append(ele) val = val[comma_idx + 1:] len_of_val = len(val) if is_tuple: values = tuple(values) return values def __call__(self, parser, namespace, values, option_string=None): options = {} for key_value in values: key, value = key_value.split('=', maxsplit=1) options[key] = self._parse_value_iter(value) setattr(namespace, self.dest, options) def ordered_yaml_load(stream, yaml_loader=yaml.SafeLoader, object_pairs_hook=OrderedDict): """Load Yaml File in Orderedly.""" class OrderedLoader(yaml_loader): pass def _construct_mapping(loader, node): loader.flatten_mapping(node) return object_pairs_hook(loader.construct_pairs(node)) OrderedLoader.add_constructor( yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, _construct_mapping) return yaml.load(stream, OrderedLoader) def ordered_yaml_dump(data, stream=None, yaml_dumper=yaml.SafeDumper, object_pairs_hook=OrderedDict, **kwargs): """Dump Dict to Yaml File in Orderedly.""" class OrderedDumper(yaml_dumper): pass def _dict_representer(dumper, data): return dumper.represent_mapping( yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, data.items()) OrderedDumper.add_representer(object_pairs_hook, _dict_representer) return yaml.dump(data, stream, OrderedDumper, **kwargs) def parse_args(): """ Parse arguments from `yaml or yml` config file. Returns: object: argparse object. """ parser = argparse.ArgumentParser("Transformer Config.") parser.add_argument('-c', '--config', type=str, default="", help='Enter the path of the model config file.') return parser.parse_args() pengcheng.mind-mindformer/mindformers/tools/register/register.py000066400000000000000000000156201476650141400256250ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project # https://gitee.com/mindspore/vision/blob/master/mindvision/engine/class_factory.py # ============================================================================ """ Class Register Module For MindFormers.""" import inspect class MindFormerModuleType: """Class module type for vision pretrain""" def __init__(self): pass TRAINER = 'trainer' PIPELINE = 'pipeline' PROCESSOR = 'processor' TOKENIZER = 'tokenizer' DATASET = 'dataset' MASK_POLICY = 'mask_policy' DATASET_LOADER = 'dataset_loader' DATASET_SAMPLER = 'dataset_sampler' TRANSFORMS = 'transforms' ENCODER = 'encoder' MODELS = 'models' MODULES = 'modules' TRANSFORMER = 'transformer' BASE_LAYER = 'base_layer' CORE = 'core' HEAD = 'head' LOSS = 'loss' LR = 'lr' OPTIMIZER = 'optimizer' CONTEXT = 'context' CALLBACK = 'callback' WRAPPER = 'wrapper' METRIC = 'metric' CONFIG = 'config' TOOLS = 'tools' FEATURE_EXTRACTOR = 'feature_extractor' class MindFormerRegister: """ Module class factory for ring-mo. """ def __init__(self): pass registry = {} @classmethod def register(cls, module_type=MindFormerModuleType.TOOLS, alias=None): """Register class into registry Args: module_type (ModuleType) : module type name, default ModuleType.GENERAL alias (str) : class alias Returns: wrapper """ def wrapper(register_class): """Register-Class with wrapper function. Args: register_class : class need to register Returns: wrapper of register_class """ class_name = alias if alias is not None else register_class.__name__ if module_type not in cls.registry: cls.registry[module_type] = {class_name: register_class} else: cls.registry[module_type][class_name] = register_class return register_class return wrapper @classmethod def register_cls(cls, register_class, module_type=MindFormerModuleType.TOOLS, alias=None): """Register class with type name. Args: register_class : class need to register module_type : module type name, default ModuleType.COMMON alias : class name """ class_name = alias if alias is not None else register_class.__name__ if module_type not in cls.registry: cls.registry[module_type] = {class_name: register_class} else: cls.registry[module_type][class_name] = register_class return register_class @classmethod def is_exist(cls, module_type, class_name=None): """Determine whether class name is in the current type group. Args: module_type : Module type class_name : class name Returns: True/False """ if not class_name: return module_type in cls.registry registered = module_type in cls.registry and class_name in cls.registry.get(module_type) return registered @classmethod def get_cls(cls, module_type, class_name=None): """Get class Args: module_type : Module type class_name : class name Returns: register_class """ if not cls.is_exist(module_type, class_name): raise ValueError("Can't find class type {} class name {} \ in class registry".format(module_type, class_name)) if not class_name: raise ValueError( "Can't find class. class type = {}".format(class_name)) register_class = cls.registry.get(module_type).get(class_name) return register_class @classmethod def get_instance_from_cfg(cls, cfg, module_type=MindFormerModuleType.TOOLS, default_args=None): """Get instance. Args: cfg (dict) : Config dict. It should at least contain the key "type". module_type : module type default_args (dict, optional) : Default initialization arguments. Returns: obj : The constructed object. """ if not isinstance(cfg, dict): raise TypeError( "Cfg must be a Config, but got {}".format(type(cfg)) ) if 'type' not in cfg: raise KeyError( '`cfg` or `default_args` must contain the key "type",' 'but got {}\n{}'.format(cfg, default_args) ) if not (isinstance(default_args, dict) or default_args is None): raise TypeError( 'default_args must be a dict or None' 'but got {}'.format(type(default_args))) args = cfg.copy() if default_args is not None: for k, v in default_args.items(): if k not in args: args.setdefault(k, v) else: args[k] = v obj_type = args.pop('type') if isinstance(obj_type, str): obj_cls = cls.get_cls(module_type, obj_type) elif inspect.isclass(obj_type): obj_cls = obj_type else: raise ValueError("Can't find class type {} class name {} \ in class registry".format(type, obj_type)) try: return obj_cls(**args) except Exception as e: raise type(e)('{}: {}'.format(obj_cls.__name__, e)) @classmethod def get_instance(cls, module_type=MindFormerModuleType.TOOLS, class_name=None, **kwargs): """Get instance. Args: module_type : module type class_name : class type Returns: object : The constructed object """ if class_name is None: raise ValueError("Class name cannot be None.") if isinstance(class_name, str): obj_cls = cls.get_cls(module_type, class_name) elif inspect.isclass(class_name): obj_cls = class_name else: raise ValueError("Can't find class type {} class name {} \ in class registry.".format(type, class_name)) try: return obj_cls(**kwargs) except Exception as e: raise type(e)('{}: {}'.format(obj_cls.__name__, e)) pengcheng.mind-mindformer/mindformers/tools/transform_ckpt.py000066400000000000000000000062221476650141400252070ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """transform ckpt""" import os import argparse import mindspore as ms def get_strategy(startegy_path, rank_id=None): """Merge strategy if strategy path is dir Args: startegy_path (str): The path of stategy. rank_id (int): The rank id of device. Returns: None or strategy path """ if not startegy_path: return None assert os.path.exists(startegy_path), f'{startegy_path} not found!' if os.path.isfile(startegy_path): return startegy_path if os.path.isdir(startegy_path): if rank_id: merge_path = os.path.join(startegy_path, f'merged_ckpt_strategy_{rank_id}.ckpt') else: merge_path = os.path.join(startegy_path, f'merged_ckpt_strategy.ckpt') if os.path.exists(merge_path): os.remove(merge_path) ms.merge_pipeline_strategys(startegy_path, merge_path) return merge_path return None if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('--src_ckpt_strategy', default="", help='path of src ckpt strategy') parser.add_argument('--dst_ckpt_strategy', default="", help='path of dst ckpt strategy') parser.add_argument('--src_ckpt_dir', default="", type=str, help='path of src ckpt') parser.add_argument('--dst_ckpt_dir', default="", type=str, help='path where to save dst ckpt') parser.add_argument('--prefix', default='checkpoint_', type=str, help='prefix of transformed checkpoint') args = parser.parse_args() src_ckpt_strategy = get_strategy(args.src_ckpt_strategy) dst_ckpt_strategy = get_strategy(args.dst_ckpt_strategy) src_ckpt_dir = args.src_ckpt_dir dst_ckpt_dir = args.dst_ckpt_dir prefix = args.prefix assert os.path.exists(args.src_ckpt_dir), f'{args.src_ckpt_dir} not found!' print(f"src_ckpt_strategy: {src_ckpt_strategy}") print(f"dst_ckpt_strategy: {dst_ckpt_strategy}") print(f"src_ckpt_dir: {src_ckpt_dir}") print(f"dst_ckpt_dir: {dst_ckpt_dir}") print(f"prefix: {prefix}") print("......Start transform......") ms.transform_checkpoints(src_ckpt_dir, dst_ckpt_dir, prefix, src_ckpt_strategy, dst_ckpt_strategy) print("......Transform succeed!......") pengcheng.mind-mindformer/mindformers/tools/utils.py000066400000000000000000000261511476650141400233160ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Utils For Tools.""" import json import os from typing import Dict, List, Tuple, Union from multiprocessing import Process try: import fcntl except ImportError: fcntl = None import numpy as np from mindspore import Tensor from mindspore import context PARALLEL_MODE = {'DATA_PARALLEL': context.ParallelMode.DATA_PARALLEL, 'SEMI_AUTO_PARALLEL': context.ParallelMode.SEMI_AUTO_PARALLEL, 'AUTO_PARALLEL': context.ParallelMode.AUTO_PARALLEL, 'HYBRID_PARALLEL': context.ParallelMode.HYBRID_PARALLEL, 'STAND_ALONE': context.ParallelMode.STAND_ALONE, 0: context.ParallelMode.DATA_PARALLEL, 1: context.ParallelMode.SEMI_AUTO_PARALLEL, 2: context.ParallelMode.AUTO_PARALLEL, 3: context.ParallelMode.HYBRID_PARALLEL} MODE = {'PYNATIVE_MODE': context.PYNATIVE_MODE, 'GRAPH_MODE': context.GRAPH_MODE, 0: context.GRAPH_MODE, 1: context.PYNATIVE_MODE} SAVE_WORK_PATH = '/cache/ma-user-work/rank_{}' MA_OUTPUT_ROOT = '/cache/ma-user-work' DEBUG_INFO_PATH = '/cache/debug' PROFILE_INFO_PATH = '/cache/profile' PLOG_PATH = '/root/ascend/log' LOCAL_DEFAULT_PATH = os.getenv("LOCAL_DEFAULT_PATH", './output') LAST_TRANSFORM_LOCK_PATH = "/tmp/last_transform_done.lock" RANK_FINISH_PATH = "/cache/rank_*_finish.txt" _PROTOCOL = 'obs' _PROTOCOL_S3 = 's3' def check_in_modelarts(): """Check if the training is on modelarts. Returns: (bool): If it is True, it means ModelArts environment. """ # 'KUBERNETES_PORT' in os.environ or \ return 'MA_LOG_DIR' in os.environ or \ 'MA_JOB_DIR' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ or \ 'S3_ACCESS_KEY_ID' in os.environ or \ 'S3_SECRET_ACCESS_KEY' in os.environ or \ 'BATCH_GROUP_NAME' in os.environ or \ 'MA_LOCAL_LOG_PATH' in os.environ class Validator: """validator for checking input parameters""" @staticmethod def check_type(arg_value, arg_type): """Check int.""" if not isinstance(arg_value, arg_type): raise TypeError('{} should be {} type, but get {}'.format(arg_value, arg_type, type(arg_value))) @staticmethod def is_obs_url(url): """Check obs url.""" return url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://') def check_obs_url(url): """Check obs url.""" if not isinstance(url, str): raise TypeError('remote_save_url type should be a str, but get {}, ' 'please check your remote_save_url config'.format(type(url))) if not (url.startswith(_PROTOCOL + '://') or url.startswith(_PROTOCOL_S3 + '://')): raise TypeError('remote_save_url should be start with obs:// or s3://, ' 'but get {}, please check your remote_save_url config'.format(url)) return True def check_list(var_name: str, list_var: Union[Tuple, List], num: int): """Checks the legitimacy of elements within a node or device list. Args: var_name (str): The Name of variable need to check. list_var (tuple or list): Variables in list format. num (int): The number of nodes or devices. Returns: None """ for value in list_var: if value >= num: raise ValueError('The index of the {} needs to be less than the number of nodes {}.'.format(var_name, num)) def format_path(path): """Check path.""" return os.path.realpath(path) def sync_trans(f): """Asynchronous transport decorator.""" try: def wrapper(*args, **kwargs): pro = Process(target=f, args=args, kwargs=kwargs) pro.start() return pro return wrapper except Exception as e: raise e def get_output_root_path(): """get default output path in local/AICC.""" if check_in_modelarts(): return MA_OUTPUT_ROOT return LOCAL_DEFAULT_PATH def get_output_subpath(sub_class, rank_id=0, append_rank=True): """get output store path for sub output class.""" Validator.check_type(sub_class, str) root_path = get_output_root_path() directory = os.path.join(root_path, sub_class) if append_rank: directory = os.path.join(directory, 'rank_{}'.format(rank_id)) return format_path(directory) def set_remote_save_url(remote_save_url): check_obs_url(remote_save_url) os.environ.setdefault('REMOTE_SAVE_URL', remote_save_url) def get_remote_save_url(): return os.environ.get('REMOTE_SAVE_URL', None) def get_net_outputs(params): """Get network outputs.""" if isinstance(params, (tuple, list)): if isinstance(params[0], Tensor) and isinstance(params[0].asnumpy(), np.ndarray): params = params[0] elif isinstance(params, Tensor) and isinstance(params.asnumpy(), np.ndarray): params = np.mean(params.asnumpy()) return params def get_rank_info() -> Tuple[int, int]: """Get rank_info from environment variables. Returns: rank_id (int): Rank id. rand_size (int): The number of rank. """ rank_id = int(os.getenv('RANK_ID', '0')) rank_size = int(os.getenv('RANK_SIZE', '1')) return rank_id, rank_size def get_num_nodes_devices(rank_size: int) -> Tuple[int, int]: """Derive the number of nodes and devices based on rank_size. Args: rank_size (int): rank size. Returns: num_nodes (int): number of nodes. num_devices (int): number of devices. """ if rank_size in (2, 4, 8): num_nodes = 1 num_devices = rank_size else: num_nodes = rank_size // 8 num_devices = 8 return num_nodes, num_devices class Const: """Const.""" def __setattr__(self, key, value): if key in self.__dict__: raise PermissionError('Can not change const {0}.'.format(key)) if not key.isupper(): raise ValueError('Const name {0} is not all uppercase.'.format(key)) self.__dict__[key] = value def generate_rank_list(stdout_nodes: Union[List, Tuple], stdout_devices: Union[List, Tuple]): """Generate a list of the ranks to output the log. Args: stdout_nodes (list or tuple): The compute nodes that will output the log to stdout. stdout_devices (list or tuple): The compute devices that will output the log to stdout. Returns: rank_list (list): A list of the ranks to output the log to stdout. """ rank_list = [] for node in stdout_nodes: for device in stdout_devices: rank_list.append(8 * node + device) return rank_list def convert_nodes_devices_input(var: Union[List, Tuple, Dict[str, int], None], num: int) -> Union[List, Tuple]: """Convert node and device inputs to list format. Args: var (list[int] or tuple[int] or dict[str, int] or optional): The variables that need to be converted to the format. num (str): The number of nodes or devices. Returns: var (list[int] or tuple[int]): A list of nodes or devices. """ if var is None: var = tuple(range(num)) elif isinstance(var, dict): var = tuple(range(var['start'], var['end'])) return var def str2bool(b): """String convert to Bool.""" if b.lower() in ["false"]: output = False elif b.lower() in ["true"]: output = True else: raise Exception("Invalid Bool Value") return output def count_params(net): """Count number of parameters in the network Args: net (mindspore.nn.Cell): Mindspore network instance Returns: total_params (int): Total number of trainable params """ total_params = [np.prod(param.shape) for param in net.trainable_params()] return sum(total_params) // 1000000 def try_sync_file(file_name): """If the file is still downloading, we need to wait before the file finished downloading""" if fcntl: with open(file_name, 'r') as fp: fcntl.flock(fp.fileno(), fcntl.LOCK_EX) def is_version_le(current_version, base_version): """ return current_version <= base_version. Check whether the current version is lower than or equal to the base version. For example: for current_version: 1.8.1, base_version: 1.11.0, it return True. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1, base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) <= int(y) return True def is_version_ge(current_version, base_version): """ return current_version >= base_version. Check whether the current version is higher than or equal to the base version. for current_version: 1.8.1, base_version: 1.11.0, it return False. """ version_split_char = '.' if version_split_char not in base_version or version_split_char not in current_version: raise ValueError("The version string will contain the `.`." "For example, current_version 1.8.1， base_version: 1.11.0.") for x, y in zip(current_version.split(version_split_char), base_version.split(version_split_char)): if not x.isdigit() or not y.isdigit(): continue if int(x) != int(y): return int(x) >= int(y) return True def parse_value(value): """ parse value from command line. handles with int, float, bool, string, list and dict. """ def isint(x): try: a = float(x) b = int(a) except (TypeError, ValueError): return False else: return a == b def isfloat(x): try: float(x) except (TypeError, ValueError): return False else: return True def isbool(x): return x in ["True", "False"] def isjson(x): try: json.loads(x) except json.decoder.JSONDecodeError: return False else: return True if isint(value): return int(value) if isfloat(value): return float(value) if isbool(value): return value == "True" if isjson(value): return json.loads(value) return value pengcheng.mind-mindformer/mindformers/trainer/000077500000000000000000000000001476650141400221035ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/.ipynb_checkpoints/000077500000000000000000000000001476650141400256745ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/.ipynb_checkpoints/base_trainer-checkpoint.py000066400000000000000000001266151476650141400330440ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Trainer.""" import math import os import shutil import time from pprint import pprint from functools import partial from typing import Optional, Union, List import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor, DatasetHelper from mindspore.train.model import Model from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.common import initializer as init from mindformers.mindformer_book import MindFormerBook from mindformers.core import build_lr, build_optim, build_callback, build_metric from mindformers.core.callback.callback import EvalCallBack from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, build_processor, build_tokenizer, \ BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.pipeline import pipeline from mindformers.wrapper import build_wrapper from mindformers.tools.register import MindFormerConfig from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.auto_class import AutoModel from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_runner_config, transform_and_load_checkpoint, load_resume_context_from_checkpoint, load_merged_checkpoint from .optimizer_grouped_parameters import get_optimizer_grouped_parameters, get_mup_optimizer_grouped_parameters from .utils import set_seed, check_train_data_loader_type, \ check_eval_data_loader_type, check_optimizer_and_lr_type, check_wrapper_config SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CONFIG_DIR = 'configs' def download_ckpt_from_obs(remote_url, local_url, rank): """ Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id """ import moxing as mox # print(f"begin download ckpt from {obs_ckpt_url}") # if not os.path.exists(os.path.join(cache_url, f"rank_{rank}")): # os.makedirs(os.path.join(cache_url, f"rank_{rank}"), exist_ok=True) # if not os.path.exists(cache_url): # os.makedirs(cache_url, exist_ok=True) time.sleep(rank % 4) try: mox.file.copy(src_url=remote_url, dst_url=local_url) except Exception as error: print(f"Error: {error}, download again...") time.sleep(10) mox.file.copy(src_url=remote_url, dst_url=local_url) print(f"Rank {rank} download ckpt {remote_url} succeed!", flush=True) class BaseTrainer: r"""Base Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, task: str = None, model_name: str = None): if model_name is None: model_name = "model name unspecified." if task is None: task = "task name unspecified." logger.info("Now Running Task is: %s, Model is: %s", task, model_name) self.model_name = model_name self.task = task self.config = None self.default_task_config = None self.train_dataset = None self.eval_dataset = None self.network = None self.optimizer = None self.image_processor = None self.audio_processor = None self.tokenizer = None self.callbacks = None self.eval_callbacks = None self.model_wrapper = None self.compute_metrics = None self.kwargs = None self.pipeline_task = None if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 to avoid dirs_exist_ok=False shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if task not in SUPPORT_TASKS.keys(): logger.warning("Input task name is not in the supported list or unspecified.") if task in SUPPORT_TASKS.keys() and model_name not in SUPPORT_TASKS.get(task).keys(): logger.warning("Input model name is not in the supported list or unspecified.") logger.warning("See the list of supported task and model name: %s", SUPPORT_TASKS) logger.warning("The default model config: %s will now be used for the %s task ", SUPPORT_TASKS.get(self.task).get("common"), task) self.model_name = "common" def set_config(self, config: Optional[Union[dict, str, ConfigArguments, TrainingArguments]] = None, is_full_config: bool = False): """Set the task config for task trainer.""" if config is not None: if is_full_config: self.config = config else: self.setup_task_config() self.config = self._merge_config(config) else: self.setup_task_config() self.config = self.default_task_config build_parallel_config(self.config) self._check_global_batch_size_for_auto_parallel() return self.config def setup_task_config(self): """Setup the default task config.""" task_config = None if self.task in SUPPORT_TASKS.keys() and self.model_name in SUPPORT_TASKS.get(self.task).keys(): task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if isinstance(task_config, MindFormerConfig): self.default_task_config = task_config else: logger.warning("If the default config arguments is not specified," "you must define the required model or wrapper, optimizer, and so on" "in the train or evaluate or predict attribute function.") def _check_global_batch_size_for_auto_parallel(self): """Check global batch size in auto parallel mode.""" batch_size = self.config.runner_config.batch_size dp = self.config.parallel_config.data_parallel micro_batch_num = self.config.parallel_config.micro_batch_num micro_batch_interleave_num = self.config.micro_batch_interleave_num parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") pp = self.get_pipeline_stages() if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: if full_batch: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is True, " "global batch size will be changed: " "global_batch_size = " "batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s * %s = %s).", pp, batch_size, dp, micro_batch_num, micro_batch_interleave_num, batch_size * dp * micro_batch_num * micro_batch_interleave_num) self.config.runner_config.batch_size = \ batch_size * dp * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, full batch is True," "so global batch size will be changed: " "global_batch_size = batch_size * data_parallel * micro_batch_interleave_num " "= %s * %s * %s = %s", parallel_mode, batch_size, dp, micro_batch_interleave_num, batch_size * dp * micro_batch_interleave_num) self.config.runner_config.batch_size = batch_size * dp * micro_batch_interleave_num else: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is False, " "batch size per card will be changed: " "per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s = %s).", pp, batch_size, micro_batch_num, micro_batch_interleave_num, batch_size * micro_batch_num * micro_batch_interleave_num) logger.info( "global_batch_size = per_batch_size * device_num " "= %s * %s = %s", batch_size * micro_batch_num * micro_batch_interleave_num, int(os.getenv('RANK_SIZE', '1')), batch_size * micro_batch_num * micro_batch_interleave_num * int(os.getenv('RANK_SIZE', '1'))) self.config.runner_config.batch_size = \ batch_size * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, batch size per card will not be changed: " "batch_size_per_card = %s", parallel_mode, batch_size) logger.info( "global_batch_size = batch_size_per_card * device_num " "= %s * %s = %s", batch_size, int(os.getenv('RANK_SIZE', '1')), batch_size * int(os.getenv('RANK_SIZE', '1'))) self.config.parallel_config.data_parallel = 1 self.config.parallel_config.model_parallel = 1 self.config.parallel_config.pipeline_stage = 1 self.config.parallel_config.optimizer_shard = False self.config.parallel_config.micro_batch_num = 1 logger.info("parallel_config will be change to default config: %s.", self.config.parallel_config) def _reset_wrapper_for_pipeline_parallel(self): """Reset wrapper when pipeline parallel.""" if self.config.runner_wrapper is not None: self.config.runner_wrapper.type = "MFPipelineWithLossScaleCell" \ if self.config.runner_wrapper.type != "MFPipelineWithLossScaleCell" else self.config.runner_wrapper.type self.config.runner_wrapper.micro_batch_num = self.config.parallel_config.micro_batch_num logger.warning( "When using the pipeline parallel mode, " "the MFPipelineWithLossScaleCell class is used by default.") else: logger.info( "When using the pipeline parallel mode, " "because the wrapper class is not specified, " "MindSpore's built-in PipelineCell is used by default") logger.info("PipelineWrapper under evaluate or predict mode will not take effect.") def _reset_dataset_batch_size(self): """Reset dataset batch size according to the global batch size of runner config.""" check_dataset_config(self.config) def _merge_config(self, args): """Merge config from default task config.""" if self.default_task_config is None: logger.warning("default task config is None, you will not be able to merge config parameters.") return args if isinstance(args, dict): self.default_task_config.merge_from_dict(args) elif isinstance(args, str): if not (os.path.realpath(args) and os.path.exists(args)): raise FileNotFoundError(f"config path must be exist, but get {args}.") if not args.endswith(('.yaml', '.yml')): raise ValueError(f"config file must be end with .yaml or .yml, but get {args}") self.default_task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, self.default_task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, self.default_task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, self.default_task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, self.default_task_config) self.default_task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): args.convert_args_to_mindformers_config(self.default_task_config) else: logger.warning( "The type of config parameter to merge is not supported, " "currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], " "but get %s", type(args)) return self.default_task_config def create_train_dataset(self, default_args: dict = None): """Create the train dataset for training.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() train_dataset = build_dataset(self.config.train_dataset_task, default_args=default_args) return train_dataset def create_eval_dataset(self, default_args: dict = None): """Create the eval dataset for evaluate.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() # reduce batch size on eval mode, for that micro_batch will not take effect on eval. parallel_mode = ms.get_auto_parallel_context("parallel_mode") if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.parallel_config.micro_batch_num self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.micro_batch_interleave_num eval_dataset = build_dataset(self.config.eval_dataset_task, default_args=default_args) return eval_dataset def create_network(self, default_args: dict = None): """Create the network for task trainer.""" logger.info(".........Build Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(network, micro_batch_interleave_num) return network, eval_network def create_pipeline_network(self, default_args: dict = None): """Create the network of pipeline parallel for task trainer.""" logger.info(".........Build Pipeline Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(network, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(network, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) return network, eval_network def create_image_processor(self, default_args: dict = None): """Create the image processor for predict.""" logger.info(".........Build Image Processor From Config..........") self.image_processor = build_processor( self.config.processor.image_processor, default_args=default_args) return self.image_processor def create_tokenizer(self, default_args: dict = None): """Create the tokenizer for task trainer.""" logger.info(".........Build Text Tokenizer From Config..........") self.tokenizer = build_tokenizer( self.config.processor.tokenizer, default_args=default_args) return self.tokenizer def create_optimizer_scheduler(self, network, layer_scale=False): """Create the optimizer for training.""" logger.info(".........Build Optimizer From Config..........") # learning rate scale for multi-nodes training learning_scale = self.config.lr_scale scale_factor = self.config.lr_scale_factor # build learning rate schedule lr_schedule = self.create_lr_scheduler(learning_scale, scale_factor) weight_decay = self.config.optimizer.weight_decay if self.config.optimizer.weight_decay else 0. layer_decay = self.config.layer_decay if self.config.layer_decay else 1.0 if self.config.model.model_config.mup_width_base is not None: ratio = self.config.model.model_config.hidden_size / self.config.model.model_config.mup_width_base group_params = get_mup_optimizer_grouped_parameters(network, weight_decay, ratio, self.config.lr_schedule) else: group_params = get_optimizer_grouped_parameters(network, weight_decay, lr_schedule, layer_scale=layer_scale, layer_decay=layer_decay) if lr_schedule is not None: self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: if self.config.optimizer.learning_rate is None: raise ValueError("learning_rate must be input") self.config.optimizer.learning_rate = self.learning_rate_scale( self.config.optimizer.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.optimizer.learning_rate self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params}) return self.optimizer def create_lr_scheduler(self, learning_scale: bool = False, scale_factor: int = 256): """Create the learning rate scheduler.""" logger.info(".........Build LR Schedule From Config..........") train_data_size = self.get_train_data_size() if self.config.lr_schedule: warmup_epochs = self.config.lr_schedule.pop("warmup_epochs", None) warmup_ratio = self.config.lr_schedule.pop("warmup_ratio", None) # WSD decay_ratio = self.config.lr_schedule.pop("decay_ratio", None) if not self.config.runner_config.sink_mode: total_steps = int(self.config.runner_config.epochs * train_data_size) else: total_steps = int(self.config.runner_config.epochs * self.config.runner_config.sink_size) if warmup_epochs is not None and warmup_ratio is not None: logger.warning("warmup_epochs and warmup_ratio are set simultaneously," "warmup_ratio takes precedence.") warmup_epochs = None if warmup_epochs is not None: logger.warning("warmup_epochs was set in lr_schedule," "it will multiply the data size to represent the warmup steps") self.config.lr_schedule.warmup_steps = int(warmup_epochs * train_data_size) self.config.lr_schedule.total_steps = total_steps \ if self.config.lr_schedule.total_steps is None or self.config.lr_schedule.total_steps == -1 \ else int(self.config.lr_schedule.total_steps) if warmup_ratio is not None: self.config.lr_schedule.warmup_steps = int(self.config.lr_schedule.total_steps * warmup_ratio) if decay_ratio is not None: #WSD self.config.lr_schedule.decay_last_steps = int(self.config.lr_schedule.total_steps * decay_ratio) self.config.lr_schedule.learning_rate = self.learning_rate_scale( self.config.lr_schedule.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.lr_schedule.learning_rate lr_schedule = build_lr(self.config.lr_schedule) return lr_schedule def create_model_wrapper(self, network, optimizer): """Create the model wrapper for training.""" logger.info(".........Build Model Wrapper for Train From Config..........") model_wrapper = build_wrapper(self.config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "parallel_config": self.config.parallel_config}) return model_wrapper def create_callbacks(self, default_args: dict = None): """Create the callback list for training.""" logger.info(".........Build Callbacks for Train From Config..........") self.callbacks = [] if self.config.profile: self.callbacks.append(self.config.profile_cb) self.callbacks.extend(build_callback(self.config.callbacks, default_args=default_args)) return self.callbacks def create_eval_callbacks(self, default_args: dict = None): """Create the eval callback list for training.""" logger.info(".........Build Callbacks for Evaluate From Config..........") self.eval_callbacks = [] self.eval_callbacks.extend(build_callback(self.config.eval_callbacks, default_args=default_args)) return self.eval_callbacks def create_metrics(self, metric_name: str = None): """Create Metrics For Evaluate or Fit.""" if metric_name is None: metric_name = self.model_name + "_metric" self.compute_metrics = {metric_name: build_metric(self.config.metric)} return self.compute_metrics def count_parameters(self): """Count network parameters number.""" if self.network is not None: logger.info("Network Parameters: %s M.", str(count_params(self.network))) elif self.model_wrapper is not None: logger.info("Network Parameters: %s M.", str(count_params(self.model_wrapper.network))) else: logger.warning("Network is None, parameters incalculable.") def set_seed(self, seed: int = None): """Set seed for training.""" if seed is None: if self.config.seed is None: raise ValueError("seed is not set in config, it is None.") set_seed(self.config.seed) else: set_seed(seed) def set_train_dataset(self, dataset): """Set the attribute of train dataset.""" if dataset is None: raise ValueError("Train dataset is None") self.train_dataset = dataset def set_eval_dataset(self, dataset): """Set the attribute of eval dataset .""" if dataset is None: raise ValueError("Eval dataset is None") self.eval_dataset = dataset def set_network(self, network, is_train: bool = True): """Set the attribute of network.""" if network is None: raise ValueError("network is None") if isinstance(network, (Cell, BaseModel)): network.set_train(is_train) self.network = network def set_model_wrapper(self, model_wrapper): """Set the attribute of model_wrapper.""" if model_wrapper is None: raise ValueError("model wrapper is None") self.model_wrapper = model_wrapper def get_train_data_size(self): """Get train dataset size.""" if self.train_dataset is None: raise NotImplementedError("train dataset is None") return self.train_dataset.get_dataset_size() def get_eval_data_size(self): """Get eval dataset size.""" if self.eval_dataset is None: raise NotImplementedError("train dataset is None") return self.eval_dataset.get_dataset_size() def get_pipeline_stages(self): """Get pipeline stages for task trainer.""" pipeline_stages = ms.get_auto_parallel_context("pipeline_stages") return pipeline_stages def learning_rate_scale(self, base_learning_rate: float = 0., scale_factor: Optional[Union[float, int]] = 256.): """Scale learning rate for training.""" if not isinstance(base_learning_rate, float): raise ValueError(f"learning rate must be float type, but get {type(base_learning_rate)}") if not isinstance(scale_factor, (float, int)): raise ValueError(f"scale_factor must be float or int type, but get {type(scale_factor)}") device_num = int(os.getenv("RANK_SIZE", "1")) per_device_batch_size = self.config.train_dataset.batch_size learning_rate = (base_learning_rate * device_num * per_device_batch_size) / scale_factor return learning_rate def training_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """Train or Fine-tune for BaseTrainer in MindFormers.""" self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if config.resume_training and config.load_checkpoint: logger.info(".............Start load resume context from checkpoint..................") rank = int(os.getenv("RANK_ID", "0")) if config.resume_merged_ckpt: logger.info(".............Start load merged checkpoint..................") obs_ckpt_url = os.path.join(config.load_checkpoint, config.restore_ckpt_name_prefix) local_url = f"/cache/checkpoint/{config.restore_ckpt_name_prefix}" config.load_checkpoint = local_url if rank%8==0: download_ckpt_from_obs(obs_ckpt_url, local_url, rank) with open('/cache/checkpoint/download_successed.txt', 'w')as fl: fl.write("") else: while not os.path.isfile('/cache/checkpoint/download_successed.txt'): time.sleep(3) else: download_rank = rank%config.parallel.optimizer_weight_shard_size if config.reuse_ckpt else rank obs_ckpt_url = os.path.join(config.load_checkpoint, f"rank_{download_rank}", f"{config.restore_ckpt_name_prefix.replace('*', str(download_rank))}") local_url = f"/cache/checkpoint/rank_{rank}.ckpt" download_ckpt_from_obs(obs_ckpt_url, local_url, rank) config.load_checkpoint = local_url load_resume_context_from_checkpoint(config) # build dataset logger.info(".........Build Dataset For Train..........") if dataset is None: dataset = self.create_train_dataset() self.set_train_dataset(dataset) check_runner_config(config, dataset) config.runner_config.epochs += config.runner_config.initial_epoch if config.runner_config.sink_mode: epoch_num = math.ceil((config.runner_config.epochs - config.runner_config.initial_epoch) * config.runner_config.sink_size / dataset.get_dataset_size()) # pylint: disable=W0212 dataset._dataset_helper = DatasetHelper(dataset, config.runner_config.sink_mode, config.runner_config.sink_size, epoch_num) # build network logger.info(".........Build Net For Train..........") eval_network = None if network is None and wrapper is None and \ self.model_wrapper is None and self.network is None: # If neither network nor wrapper exists, create a network if self.get_pipeline_stages() > 1: network, eval_network = self.create_pipeline_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) else: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and wrapper is None and self.network is not None: logger.info(".........Using The Existing Network For Train:: %s", self.network.__class__.__name__) network = self.network if network is not None: self.set_network(network, is_train=True) if eval_network is None: eval_network = network if wrapper is not None: self.set_model_wrapper(wrapper) self.count_parameters() # build optimizer logger.info(".........Build Optimizer For Train..........") if optimizer is None and wrapper is None and self.model_wrapper is None: optimizer = self.create_optimizer_scheduler(network, layer_scale=config.layer_scale) # build model wrapper if wrapper is None and self.model_wrapper is None: logger.info(".........Build Running Wrapper From Config For Train..........") wrapper = self.create_model_wrapper(network, optimizer) elif wrapper is None and self.model_wrapper is not None: logger.info(".........Using The Existing Model Wrapper: %s", self.model_wrapper.__class__.__name__) wrapper = self.model_wrapper # build callback logger.info(".........Build Callbacks For Train..........") if callbacks is None: callbacks = self.create_callbacks(default_args={ "learning_rate": optimizer.learning_rate if optimizer else wrapper.optimizer.learning_rate, "origin_epochs": config.runner_config.origin_epochs, "dataset_size": config.data_size, "micro_batch_interleave_num": config.micro_batch_interleave_num, "micro_batch_num": config.parallel_config.micro_batch_num, "initial_epoch": config.runner_config.initial_epoch, "global_batch_size": self.config.runner_config.batch_size, "device_num": int(os.getenv('RANK_SIZE', '1'))}) # define compute metrics for evaluate in training compute_metrics = None if config.do_eval: compute_metrics = self.create_metrics() # define Model and begin training logger.info(".........Starting Init Train Model..........") if wrapper is not None: model = Model(wrapper, metrics=compute_metrics, eval_network=eval_network) else: model = Model(network, optimizer=optimizer, metrics=compute_metrics, eval_network=eval_network) # resume checkpoint if config.load_checkpoint or config.only_save_strategy: if config.resume_training: logger.info(".............Start resume training from checkpoint..................") if config.resume_merged_ckpt: load_merged_checkpoint(config, model, network, dataset, optimizer=optimizer) else: transform_and_load_checkpoint(config, model, network, dataset, optimizer=optimizer) else: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path transform_and_load_checkpoint(config, model, network, dataset) # build evaluate in training if config.do_eval: logger.info(".........Build Evaluate in Training Callback..........") eval_dataset = kwargs.get('eval_dataset', None) if eval_dataset is None: eval_dataset = self.create_eval_dataset() eval_callback = EvalCallBack( partial( self._evaluate_in_training, model=model, eval_dataset=eval_dataset, ), step_interval=config.eval_step_interval if config.eval_step_interval else 100, epoch_interval=config.eval_epoch_interval if config.eval_epoch_interval else -1, ) callbacks.append(eval_callback) logger.info(".........Starting Training Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) logger.info(".........Model Compiling, Please Wait a Moment...........") model.train(config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size, initial_epoch=config.runner_config.initial_epoch) logger.info(".........Training Over!.............") def evaluate_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): """Evaluate for BaseTrainer in MindFormers.""" metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) logger.info("Create evaluate dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network eval_network = None if network is None and self.network is None: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and self.network is not None: logger.info(".........Using The Existing Network For Evaluate: %s", self.network.__class__.__name__) network = self.network self.set_network(network, is_train=False) if eval_network is None: eval_network = network self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = self.create_metrics(metric_name=metric_name) # build callback logger.info(".........Build Callbacks For Evaluate..........") if callbacks is None: callbacks = self.create_eval_callbacks() logger.info(".........Starting Init Evaluate Model..........") model = Model(network, metrics=compute_metrics, eval_network=eval_network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info(".........Starting Evaluate Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output = model.eval(dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode) logger.info('%s = %s', metric_name, str(output)) logger.info(".........Evaluate Over!.............") def predict_process(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, task: str = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseImageProcessor] = None, **kwargs): """Predict for BaseTrainer in MindFormers.""" if not self.pipeline_task: is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if task not in ["translation", "text_generation"]: raise SystemExit("Currently distributed predict only support translation and text_generation. " "Process exit!") if config.parallel_config.pipeline_stage > 1: raise SystemExit("Currently distributed predict dose not support pipeline parallel. " "Process exit!") # build network if network is None: network, _ = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) network.model_name = kwargs.get("model_name", None) self.count_parameters() if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and config.processor.image_processor: image_processor = build_processor(config.processor.image_processor) if audio_processor is None and config.processor.audio_processor: audio_processor = build_processor(config.processor.audio_processor) model = Model(network) if config.load_checkpoint or config.only_save_strategy: if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: seq_length = config.model.model_config.seq_length infer_data = Tensor(shape=(1, seq_length), dtype=ms.int32, init=init.One()) transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) else: transform_and_load_checkpoint(config, model, network, None, do_predict=True) self.pipeline_task = pipeline( task=task, model=network, tokenizer=tokenizer, image_processor=image_processor, audio_processor=audio_processor, **kwargs ) top_k = kwargs.pop("top_k", None) if top_k is None: if config.top_k is not None: top_k = config.top_k elif config.model and config.model.model_config.top_k: top_k = config.model.model_config.top_k else: top_k = 1 save_file = kwargs.pop("save_file", None) if save_file is None: if config.save_file is not None: save_file = config.save_file else: save_file = f"{task}_result.txt" if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output_results = self.pipeline_task(input_data, top_k=top_k) output_info = [] for one_output in output_results: if isinstance(one_output, dict) and "info" in one_output: output_info.append(one_output["info"]) else: output_info.append(one_output) with open(save_file, 'w') as file: file.write(str(output_info)) file.close() logger.info("output result is: %s", str(output_info)) logger.info("output result is saved at: %s", save_file) logger.info(".........Predict Over!.............") return output_results def _evaluate_in_training(self, model: Model, eval_dataset: BaseDataset): origin_phase = model.eval_network.phase model.eval_network.set_train(False) output = model.eval( eval_dataset, dataset_sink_mode=self.config.runner_config.sink_mode ) model.eval_network.set_train(origin_phase) return output optimizer_grouped_parameters-checkpoint.py000066400000000000000000000173541476650141400363200ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/.ipynb_checkpoints# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Grouped parameters of optimizer.""" import json from functools import partial from typing import Optional import copy from mindspore.nn import Cell from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindformers.models import BaseModel from mindformers.core.lr import LearningRateWiseLayer from mindformers.core import build_lr from mindformers.tools import logger from .utils import check_keywords_in_name def get_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, dynamic_lr_schedule: Optional[LearningRateSchedule] = None, layer_scale: bool = False, layer_decay: float = 1.0): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") skip_params = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip_params = model.no_weight_decay() logger.info('No weight decay: %s', skip_params) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) decay_parameters_names = [ param.name for param in model.trainable_params() if not (len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip_params) or check_keywords_in_name(param.name, skip_keywords)) ] get_layer_id_func = None scales_list = [] if dynamic_lr_schedule is not None: if layer_scale: if model.__class__.__name__ == 'SwinForImageClassification': depths = model.config.depths num_layers = sum(depths) get_layer_id_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) elif model.__class__.__name__ == 'ViTForImageClassification': num_layers = model.config.depth get_layer_id_func = partial(get_vit_layer, num_layers=num_layers + 2) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) else: logger.warning("Now only support VitModel and SwinModel," "if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.") layer_scale = False dynamic_lr_schedule = None else: logger.warning("dynamic_lr_schedule will be reset and invalid when layer_scale is False.") dynamic_lr_schedule = None parameter_group_names = {} parameter_group_vars = {} scale = 1. for param in model.trainable_params(): if param.name in decay_parameters_names: group_name = 'decay' else: group_name = 'no_decay' weight_decay = 0. if get_layer_id_func is not None: layer_id = get_layer_id_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) if scales_list is not None: scale = scales_list[layer_id] if group_name not in parameter_group_names: parameter_group_names[group_name] = { "weight_decay": weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": weight_decay, "params": [], } if isinstance(dynamic_lr_schedule, LearningRateSchedule): if layer_scale: parameter_group_vars[group_name]["lr"] = LearningRateWiseLayer(dynamic_lr_schedule, scale) else: parameter_group_vars[group_name]["lr"] = dynamic_lr_schedule parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_mup_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, mup_ratio: float = 0.0, lr_schedule_config = None): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") def find_in_name(name, key_list): for k in key_list: if name.find(k) >= 0: return True return False middle_weight_key_list = [ "attention.wo", "attention.wq", "attention.wk", "attention.wv", "feed_forward", "lm_head" ] parameter_group_vars = {} middle_weight_group = {"params": []} # mlp and attention for param in model.trainable_params(): if find_in_name(param.name, middle_weight_key_list): middle_weight_group["params"].append(param) middle_lr_config = copy.deepcopy(lr_schedule_config) middle_lr_config.learning_rate = middle_lr_config.learning_rate / mup_ratio middle_lr_config.lr_middle = middle_lr_config.lr_middle / mup_ratio middle_lr_config.lr_end = middle_lr_config.lr_end / mup_ratio middle_lr = build_lr(middle_lr_config) middle_weight_group["lr"] = middle_lr middle_weight_group["weight_decay"] = weight_decay parameter_group_vars["middle_output"] = middle_weight_group norm_key_list = [ "norm", "embedding" ] norm_weight_group = {"params": []} for param in model.trainable_params(): if find_in_name(param.name, norm_key_list): norm_weight_group["params"].append(param) norm_weight_group["weight_decay"] = 0.0 parameter_group_vars["norm_key"] = norm_weight_group return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[2]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num pengcheng.mind-mindformer/mindformers/trainer/.ipynb_checkpoints/trainer-checkpoint.py000066400000000000000000001504651476650141400320520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer API For Import.""" import os import shutil from collections import OrderedDict from typing import List, Optional, Union import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor from mindspore.common import set_seed from mindspore import load_checkpoint, load_param_into_net from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, build_dataset_loader, \ check_dataset_config, BaseDataset from mindformers.mindformer_book import MindFormerBook from mindformers.models import build_model, BaseModel, BaseImageProcessor, \ BaseTokenizer, BaseAudioProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.register.config import ordered_yaml_dump from .build_trainer import build_trainer from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_train_data_loader_type, check_eval_data_loader_type, \ check_optimizer_and_lr_type, check_wrapper_config, config2dict __all__ = ['Trainer'] SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CHECKPOINT_DIR = 'checkpoint' DEFAULT_CONFIG_DIR = 'configs' class Trainer: r""" Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) """ def __init__(self, args: Optional[Union[str, dict, ConfigArguments, TrainingArguments]] = None, task: Optional[str] = 'general', model: Optional[Union[str, Cell, BaseModel]] = None, model_name: Optional[Union[str]] = None, train_dataset: Optional[Union[str, BaseDataset]] = None, eval_dataset: Optional[Union[str, BaseDataset]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, optimizers: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, pet_method: Optional[str] = '', callbacks: Optional[Union[Callback, List[Callback]]] = None, eval_callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, save_config: bool = False, **kwargs): self.task = task self.model = model self.model_name = model_name self.pet_method = pet_method self.train_dataset = train_dataset self.eval_dataset = eval_dataset self.optimizers = optimizers self.wrapper = wrapper self.tokenizer = tokenizer self.image_processor = image_processor self.audio_processor = audio_processor self.callbacks = callbacks self.eval_callbacks = eval_callbacks self.compute_metrics = compute_metrics self.default_checkpoint_name_or_path = None self.configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) self.kwargs = kwargs if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if wrapper is not None: if model is not None: logger.warning( 'wrapper has existed, input model invalid, it should be include in wrapper.') if optimizers is not None: logger.warning( 'wrapper has existed, input optimizers invalid, it should be include in wrapper.') assert task in SUPPORT_TASKS.keys(), \ f"task name must be in {SUPPORT_TASKS.keys()}, but get {task}." if isinstance(model, str): model = model + '_{}'.format(pet_method) if pet_method else model assert model in SUPPORT_MODEL_NAMES, \ f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}." self.model_name = model self.model = None if isinstance(self.model, (Cell, BaseModel)): logger.info("The model instance has been entered, " "and the model will not be created from model_config") if pet_method: logger.warning("pet_method is not valid when a model instance is passed in." "Currently, only part of the model in MindFormers is supported." "Please pass in the model keyword") self.is_model_instance = True else: self.is_model_instance = False if self.is_model_instance and self.model_name is None: logger.warning( "Recognizing that a model instance is sent and model_name is None,") logger.warning( "it is recommended to select a model configuration that corresponds " "to the support of MindFormers based on the instance model and set model_name.") logger.warning( "Otherwise, they will default to a general configuration." "You are advised to pass instances such as optimizers, metric, tokenizer, and processor") self.model_name = 'common' task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if self.model_name == "common": if self.model is not None: task_config.trainer.model_name = self.model.__class__.__name__ if self.wrapper is not None: task_config.trainer.model_name = self.wrapper.network.__class__.__name__ if args is None: self.config = task_config else: if isinstance(args, dict): task_config.merge_from_dict(args) elif isinstance(args, str): assert os.path.realpath(args) and os.path.exists(args), \ f"config path must be exist, but get {args}." assert args.endswith(('.yaml', '.yml')), \ f"config file must be end with .yaml or .yml, but get {args}" task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, task_config) task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): logger.warning( "When using the TrainingArguments class, " "its arguments will override the default config configuration.") args.convert_args_to_mindformers_config(task_config) self.config = task_config if save_config: self.save_config_to_yaml(self.config) logger.info("save running config success of %s_new.", task_config.trainer.model_name.lower()) # check dataset config if isinstance(train_dataset, str): assert os.path.exists(train_dataset), \ f"train dataset path must be exist, but get {train_dataset}." self.config.train_dataset.data_loader.dataset_dir = train_dataset self.train_dataset = None if isinstance(eval_dataset, str): assert os.path.exists(eval_dataset), \ f"eval dataset path must be exist, but get {eval_dataset}." self.config.eval_dataset.data_loader.dataset_dir = eval_dataset self.eval_dataset = None if tokenizer is not None: if self.config.train_dataset is not None: self.config.train_dataset.tokenizer = tokenizer if self.config.eval_dataset is not None: self.config.eval_dataset.tokenizer = tokenizer check_dataset_config(self.config) # build parallel config self.rank_id = int(os.getenv("RANK_ID", "0")) self.device_num = int(os.getenv("RANK_SIZE", "1")) self.config.rank_id = self.rank_id self.config.device_num = self.device_num self.is_set_parallel_config = False self.is_set_moe_config = False self.is_set_recompute_config = False # set seed if self.config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(self.config.seed) np.random.seed(self.config.seed) # set output directory os.environ.setdefault("LOCAL_DEFAULT_PATH", self.config.output_dir) # build task trainer self.trainer = build_trainer(self.config.trainer) if self.trainer is None: raise ModuleNotFoundError("config must be contain 'trainer' key, but get None.") def train(self, train_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Train task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) """ if train_checkpoint is not None and \ not isinstance(train_checkpoint, (bool, str)): raise TypeError(f"resume_or_finetune_from_checkpoint must be one of [None, string, bool], " f"but get {train_checkpoint}") if train_checkpoint is False: train_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if train_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(train_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = train_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def finetune(self, finetune_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Finetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') """ if finetune_checkpoint is not None and \ not isinstance(finetune_checkpoint, (bool, str)): raise TypeError(f"load_checkpoint must be one of [None, string, bool], " f"but get {finetune_checkpoint}") if finetune_checkpoint is False: finetune_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if finetune_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(finetune_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = finetune_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be fine-tune are specified" "by the finetune_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def evaluate(self, eval_checkpoint: Optional[Union[str, bool]] = False, auto_trans_ckpt: Optional[bool] = None, **kwargs): r"""Evaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') """ if eval_checkpoint is not None and not isinstance(eval_checkpoint, (bool, str)): raise TypeError(f"eval_checkpoint must be one of [None, string, bool], " f"but get {eval_checkpoint}") if eval_checkpoint is False: eval_checkpoint = None if eval_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(eval_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = eval_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=False) self.trainer.evaluate( config=self.config, network=self.model, dataset=self.eval_dataset, callbacks=self.eval_callbacks, is_full_config=True, **kwargs) def predict(self, predict_checkpoint: Optional[Union[str, bool]] = None, auto_trans_ckpt: Optional[bool] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, **kwargs): r"""Predict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) """ if predict_checkpoint is not None and not isinstance(predict_checkpoint, (bool, str)): raise TypeError(f"predict_checkpoint must be one of [None, string, bool], " f"but get {predict_checkpoint}") if self.task not in SUPPORT_PIPELINES.keys(): raise NotImplementedError(f"The {self.task} not support predict, " f"now this tasks {SUPPORT_PIPELINES.keys()} is support predict.") if predict_checkpoint is False: predict_checkpoint = None if predict_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(predict_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = predict_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if input_data is None: input_data = build_dataset_loader(self.config.eval_dataset.data_loader) logger.info("dataset by config is used as input_data.") assert isinstance(input_data, (GeneratorDataset, BaseDataset, RepeatDataset, BatchDataset, Tensor, np.ndarray, Image, str, list)), \ "Input data's type must be one of [GeneratorDataset," \ " str, ms.Tensor, np.ndarray, PIL.Image.Image]" if self.is_model_instance: self.reset_model_instance(is_train=False) kwargs['model_name'] = self.model_name output_result = self.trainer.predict( config=self.config, input_data=input_data, network=self.model, image_processor=self.image_processor, audio_processor=self.audio_processor, tokenizer=self.tokenizer, is_full_config=True, **kwargs) return output_result def build_network(self, input_checkpoint: Optional[Union[str, bool]] = None, is_train: bool = True): """build network for trainer.""" if self.model is None and self.task != 'general': logger.info("...........Start Init Network..........") self.model = build_model(self.config.model) # set running mode if self.model is not None: self.model.set_train(is_train) else: logger.warning("network will be create in %s task trainer class.", self.task) if self.is_model_instance and input_checkpoint: self._load_model_checkpoint() def set_parallel_config( self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_interleave_num=1, micro_batch_num=1, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): r""" set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) """ self.config.parallel_config.data_parallel = data_parallel self.config.parallel_config.model_parallel = model_parallel self.config.parallel_config.expert_parallel = expert_parallel self.config.parallel_config.pipeline_stage = pipeline_stage self.config.parallel_config.use_seq_parallel = use_seq_parallel self.config.parallel_config.optimizer_shard = optimizer_shard self.config.parallel_config.micro_batch_num = micro_batch_num self.config.parallel_config.vocab_emb_dp = vocab_emb_dp self.config.parallel_config.gradient_aggregation_group = gradient_aggregation_group self.config.micro_batch_interleave_num = micro_batch_interleave_num self.is_set_parallel_config = True def set_recompute_config(self, recompute=False, parallel_optimizer_comm_recompute=False, select_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): r"""Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) """ self.config.recompute_config.recompute = recompute self.config.recompute_config.select_recompute = select_recompute self.config.recompute_config.parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self.config.recompute_config.mp_comm_recompute = mp_comm_recompute self.config.recompute_config.recompute_slice_activation = recompute_slice_activation self.is_set_recompute_config = True def set_moe_config(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): r"""The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) """ self.config.moe_config.expert_num = expert_num self.config.moe_config.capacity_factor = capacity_factor self.config.moe_config.aux_loss_factor = aux_loss_factor self.config.moe_config.num_experts_chosen = num_experts_chosen self.config.moe_config.expert_group_size = expert_group_size self.config.moe_config.group_wise_a2a = group_wise_a2a self.config.moe_config.comp_comm_parallel = comp_comm_parallel self.config.moe_config.comp_comm_parallel_degree = comp_comm_parallel_degree self.is_set_moe_config = True def reset_model_instance(self, is_train=True): """Reset model instance for new model config.""" if True not in [self.is_set_parallel_config, self.is_set_moe_config, self.is_set_recompute_config]: return if self.is_set_parallel_config: logger.info("The incoming model will be configured in parallel.") if self.is_set_recompute_config: logger.info("The incoming model will be configured in recompute.") if self.is_set_moe_config: logger.info("The incoming model will be configured in moe.") if not isinstance(self.model, BaseModel): raise NotImplementedError("Currently only the integrated model structure in MindFormers is supported.") build_parallel_config(self.config) model_config = self.model.config model_config.parallel_config = self.config.parallel_config model_config.moe_config = self.config.moe_config self.model.__init__(model_config) self.model.set_train(is_train) if not is_train: return network = self.model micro_batch_interleave_num = self.config.micro_batch_interleave_num if ms.get_auto_parallel_context("pipeline_stages") > 1: micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(self.model, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(self.model, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) else: if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(self.model, micro_batch_interleave_num) del self.model self.model = network def get_train_dataloader(self): """get train dataloader of mindspore.""" return build_dataset_loader(self.config.train_dataset.data_loader) def get_eval_dataloader(self): """get eval dataloader of mindspore.""" return build_dataset_loader(self.config.eval_dataset.data_loader) def get_last_checkpoint(self): """get last checkpoint for resuming or finetune.""" output_folder = self.config.output_dir checkpoint_dir = os.path.join( output_folder, DEFAULT_CHECKPOINT_DIR, 'rank_{}'.format(self.rank_id)) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) def save_config_to_yaml(self, config: dict = None): """save now config file to yaml file.""" if config is None: config = self.config model_name = self.config.trainer.model_name config_dict = _reset_config_for_save(config) config_dir = os.path.join( self.configs_directory, model_name.lower() + '_new') if not os.path.exists(config_dir): os.makedirs(config_dir, exist_ok=True) run_yaml_path = os.path.join(config_dir, 'run_{}.yaml'.format(model_name.lower())) _save_config_to_yaml(run_yaml_path, config_dict) def _load_model_checkpoint(self): """Load model checkpoint to network.""" checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if checkpoint_name_or_path is None: logger.warning("checkpoint_name_or_path is None, not load input checkpoint.") elif isinstance(checkpoint_name_or_path, str): is_exist_path = os.path.exists(checkpoint_name_or_path) is_checkpoint_name = checkpoint_name_or_path in SUPPORT_MODEL_NAMES if is_exist_path: logger.info("now input valid checkpoint path, it will load to network.") checkpoint_dict = load_checkpoint(checkpoint_name_or_path) not_load_params = load_param_into_net(self.model, checkpoint_dict) logger.info("not load parameters is: %s", str(not_load_params)) elif is_checkpoint_name: logger.info("now input valid checkpoint name, it will load to network.") if isinstance(self.model, (Cell, BaseModel)): self.model.load_checkpoint(self.config.model.model_config) else: logger.warning("model must be BaseModel or Cell type, but get %s", type(self.model)) else: logger.warning("input checkpoint args is invalid, " "it must be valid and real checkpoint path or a valid checkpoint name," "but get %s", checkpoint_name_or_path) else: raise TypeError(f"checkpoint_name_or_path type error, " f"it should be one of [None, str], " f"but get {type(checkpoint_name_or_path)}") def _check_checkpoint_config(self, checkpoint: Optional[Union[str, bool]] = None): """check checkpoint config.""" if checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = self.get_last_checkpoint() elif isinstance(checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = checkpoint else: if self.default_checkpoint_name_or_path is not None: self.config.model.model_config.checkpoint_name_or_path = self.default_checkpoint_name_or_path def _save_config_to_yaml(save_file_path: str = None, save_config: dict = None): r"""Save Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. """ if save_config is None: save_config = {} with open(save_file_path, 'w', encoding='utf-8') as file_pointer: file_pointer.write( ordered_yaml_dump( save_config, default_flow_style=False, allow_unicode=True, sort_keys=False)) def _reset_config_for_save(config: dict = None): r"""Reset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. """ if config is None: config = {} config = config.copy() config_dict = OrderedDict() if config.get('model') is not None: model_config = config2dict(config.pop('model')) config_dict.setdefault('model', model_config) if config.get('processor') is not None: processor_config = config2dict(config.pop('processor')) config_dict.setdefault('processor', processor_config) if config.get('train_dataset_task') is not None and config.get('train_dataset') is not None: train_dataset_config = config2dict(config.pop('train_dataset')) train_dataset_task_config = config2dict(config.pop('train_dataset_task')) config_dict.setdefault('train_dataset', train_dataset_config) config_dict.setdefault('train_dataset_task', train_dataset_task_config) if config.get('eval_dataset_task') is not None and config.get('eval_dataset') is not None: eval_dataset_config = config2dict(config.pop('eval_dataset')) eval_dataset_task_config = config2dict(config.pop('eval_dataset_task')) config_dict.setdefault('train_dataset', eval_dataset_config) config_dict.setdefault('train_dataset_task', eval_dataset_task_config) if config.get('context') is not None: context_config = config2dict(config.pop('context')) parallel_context_config = config2dict(config.pop('parallel')) moe_conifg = config2dict(config.pop('moe_config')) recompute_config = config2dict(config.pop('recompute_config')) parallel_config = config2dict(config.pop('parallel_config')) config_dict.setdefault('context', context_config) config_dict.setdefault('parallel', parallel_context_config) config_dict.setdefault('moe_conifg', moe_conifg) config_dict.setdefault('recompute_config', recompute_config) config_dict.setdefault('parallel_config', parallel_config) if config.get('runner_config') is not None: runner_config = config2dict(config.pop('runner_config')) config_dict.setdefault('runner_config', runner_config) if config.get('runner_wrapper') is not None: wrapper_config = config2dict(config.pop('runner_wrapper')) config_dict.setdefault('runner_wrapper', wrapper_config) if config.get('optimizer') is not None: optim_config = config2dict(config.pop('optimizer')) config_dict.setdefault('optimizer', optim_config) if config.get('lr_schedule') is not None: lr_config = config2dict(config.pop('lr_schedule')) config_dict.setdefault('lr_schedule', lr_config) if config.get('callbacks') is not None: cb_config = config2dict(config.pop('callbacks')) config_dict.setdefault('callbacks', cb_config) run_config = config2dict(config) for key, value in run_config.items(): config_dict.setdefault(key, value) return config_dict pengcheng.mind-mindformer/mindformers/trainer/.ipynb_checkpoints/utils-checkpoint.py000066400000000000000000000625201476650141400315400ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer Utils.""" import os import time import random from glob import glob from enum import Enum import numpy as np import mindspore as ms from mindspore.communication.management import get_rank, get_group_size from mindspore import context, load_checkpoint, load_param_into_net from mindspore import set_seed as ms_set_seed from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.utils import check_in_modelarts, get_output_root_path from mindformers.tools.transform_ckpt import get_strategy if check_in_modelarts(): import moxing as mox ELAPSE_TIME = 7200 class BaseEnum(str, Enum): """ Base Enum for MindFormers. """ @classmethod def _missing_(cls, value): """Enum with more explicit error message for missing values.""" raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class SaveIntervalStrategy(BaseEnum): """ Stores the acceptable string identifiers for save checkpoint monitor. """ NO = "no" STEPS = "steps" SECONDS = "seconds" class LRType(BaseEnum): """ Stores the acceptable string identifiers for learning rate schedule. """ # will be support item for future. LINEAR = "linear" COSINE = "cosine" COSINE_WITH_RESTARTS = "cosine_with_restarts" POLYNOMIAL = "polynomial" CONSTANT_WITH_WARMUP = "constant_with_warmup" class OptimizerType(BaseEnum): """ Stores the acceptable string identifiers for optimizers. """ # supported item for test, will be delete in the future. ADAMWEIGHTDECAY = 'AdamWeightDecay' # will be support item for future. ADAMW = "adamw" ADAM = "adam" SGD = "sgd" ADAGRAD = "adagrad" ADAFACTOR = "adafactor" class WrapperType(BaseEnum): """ Stores the acceptable string identifiers for training wrapper. """ # will be support item for future. MFWRAPPER = 'mf_wrapper' TRAINONESTEP = 'wrapper' TRAINONESTEPWITHLOSSSCALE = 'loss_scale_wrapper' def set_seed(seed: int = 0): """ Helper function for reproducible behavior to set the seed in `random`, `numpy`, `MindSpore`. Args: seed (`int`): The seed to set. """ random.seed(seed) np.random.seed(seed) ms_set_seed(seed) def check_keywords_in_name(name, keywords=()): """ Check keywords in name. """ isin = False for keyword in keywords: if keyword in name: isin = True return isin def check_runner_config(config, dataset): """ Check runner config. """ data_size = dataset.get_dataset_size() new_epochs = config.runner_config.epochs config.runner_config.origin_epochs = new_epochs if config.runner_config.initial_epoch is None: config.runner_config.initial_epoch = 0 if config.runner_config.sink_mode: if config.runner_config.sink_size != -1: if config.runner_config.sink_size <= 0: raise ValueError("per epoch size must be more than 0 or equal to -1, " f"but get {config.runner_config.sink_size}") if data_size < config.runner_config.sink_size: logger.warning("The data size %s (get from dataset.get_dataset_size()) is smaller " "than the sink_size %s (get from config.runner_config.sink_size), " "you should set the config.runner_config.sink_size to %s", data_size, config.runner_config.sink_size, data_size) config.runner_config.epochs = int((data_size / config.runner_config.sink_size) * new_epochs) else: config.runner_config.sink_size = data_size else: logger.warning("Sink mode is False, per epoch size is invalid, it will reset -1.") config.runner_config.sink_size = -1 config.data_size = data_size logger.info("Will be Training epochs:%d, sink_size:%d", config.runner_config.origin_epochs, config.runner_config.sink_size) logger.info("Create training dataset finish, dataset size:%d", data_size) def check_train_data_loader_type(new_config, old_config): """Check train data loader config type.""" if new_config.train_dataset is None: return None if new_config.train_dataset.get('data_loader') is None: return None train_data_loader_type = new_config.train_dataset.get('data_loader').get('type') if old_config.train_dataset is not None and train_data_loader_type is not None: default_train_data_loader_type = old_config.train_dataset.data_loader.type if train_data_loader_type != default_train_data_loader_type: logger.warning("train dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", train_data_loader_type) old_config.train_dataset.data_loader = {} return None def check_eval_data_loader_type(new_config, old_config): """Check eval data loader config type.""" if new_config.eval_dataset is None: return None if new_config.eval_dataset.get('data_loader') is None: return None eval_data_loader_type = new_config.eval_dataset.get('data_loader').get('type') if old_config.eval_dataset is not None and eval_data_loader_type is not None: default_eval_data_loader_type = old_config.eval_dataset.data_loader.type if eval_data_loader_type != default_eval_data_loader_type: logger.warning("eval dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", eval_data_loader_type) old_config.eval_dataset.data_loader = {} return None def check_optimizer_and_lr_type(new_config, old_config): """Check optimizer and lr schedule config type.""" optimizer_type = new_config.optimizer.get('type') if old_config.optimizer is not None and optimizer_type is not None: default_optimizer_type = old_config.optimizer.type if optimizer_type != default_optimizer_type: logger.warning( "optimizer type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually except (params).", optimizer_type) old_config.optimizer = {} if hasattr(new_config.optimizer, 'learning_rate'): lr_type = new_config.optimizer.learning_rate.get('type') if old_config.lr_schedule is not None and lr_type is not None: default_lr_type = old_config.lr_schedule.type if lr_type != default_lr_type: logger.warning( "lr schedule type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", lr_type) old_config.lr_schedule = None def check_wrapper_config(new_config, old_config): """Check wrapper config.""" wrapper_type = new_config.runner_wrapper.get('type') if old_config.runner_wrapper is not None and wrapper_type is not None: default_wrapper_type = old_config.runner_wrapper.type if wrapper_type != default_wrapper_type: logger.warning( "wrapper type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", wrapper_type) old_config.runner_wrapper = {} def config2dict(config): """MindFormerConfig Type Convert to Dict.""" if not isinstance(config, (dict, MindFormerConfig)): return config new_dict = {} for key, value in config.items(): if isinstance(value, MindFormerConfig): value = config2dict(value) new_dict.setdefault(key, value) return new_dict def load_distributed_checkpoint(config, specify_prefix=None): """Load Checkpoint in Parallel Mode.""" checkpoint_dir = config.load_checkpoint if os.path.isdir(checkpoint_dir): logger.info( "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/checkpoint/rank_{0-*}/**.ckpt") distribute_checkpoint_dir = os.path.join( checkpoint_dir, "rank_{}".format(int(os.getenv("RANK_ID", "0")))) distribute_checkpoint_path = get_last_checkpoint(distribute_checkpoint_dir) elif os.path.isfile(checkpoint_dir): logger.info("Your load_checkpoint is file, it will be load in network.") distribute_checkpoint_path = checkpoint_dir else: raise FileNotFoundError(f"{checkpoint_dir} is not found.") checkpoint_dict = load_checkpoint(distribute_checkpoint_path, specify_prefix=specify_prefix) logger.info("Distribute load is success.") return checkpoint_dict def load_resume_context_from_checkpoint(config): """resume training, load training info from checkpoint to config""" if not os.path.realpath(config.load_checkpoint) or \ not os.path.exists(config.load_checkpoint): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if os.path.isdir(config.load_checkpoint): resume_dict = load_distributed_checkpoint(config, ["loss_scale", "epoch_num"]) if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") else: resume_dict = load_checkpoint(config.load_checkpoint, specify_prefix=["loss_scale", "epoch_num"]) if "epoch_num" in resume_dict: config.runner_config.initial_epoch = int(resume_dict["epoch_num"]) else: config.runner_config.initial_epoch = 0 for callback in config.callbacks: if "type" in callback and callback["type"] == "CheckpointMointor": if config.runner_wrapper.scale_sense is not None and "loss_scale" in resume_dict: config.runner_wrapper.scale_sense.loss_scale_value = resume_dict["loss_scale"] break def transform_and_load_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not config.only_save_strategy and (not os.path.realpath(config.load_checkpoint) or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 1. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) if config.auto_trans_ckpt: # 2. get strategy dst_ckpt_strategy = get_dst_strategy(config) # 3. transform checkpoint if needed transform_ckpt(config, dst_ckpt_strategy=dst_ckpt_strategy) # 4. load ckpt load_ckpt(config, network, optimizer=optimizer) def load_merged_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not os.path.realpath(config.load_checkpoint or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") # 1. load merged ckpt load_ckpt(config, network, optimizer=optimizer) if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 2. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) def build_model(config, model, dataset, do_eval=False, do_predict=False): """build model, generate strategy file""" if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") if do_eval: model.infer_predict_layout(*next(dataset.create_tuple_iterator())) elif do_predict: model.infer_predict_layout(dataset) else: if config.runner_config.epochs > 1 and config.runner_config.sink_size == 1: raise ValueError(f"When distributed loads are sliced weights, it does not support" f"epochs = {config.runner_config.epochs} > 1 and " f"sink_size = {config.runner_config.sink_size} = 1," f"sink_size must be more than 1") logger.info(".........Building model.........") model.build(train_dataset=dataset, epoch=config.runner_config.epochs, sink_size=config.runner_config.sink_size) if config.only_save_strategy: raise SystemExit("Only_save_strategy is True, model.compile() finished, system exit! ") elif config.only_save_strategy: raise SystemExit("only_save_strategy is True, " "but stand_alone and data_parallel mode do not have strategy file, system exit! ") def get_dst_strategy(config): """get strategy""" rank_id = int(os.getenv('RANK_ID', '0')) world_size = int(os.getenv('RANK_SIZE', '1')) dst_strategy_path = None if world_size == 1: return dst_strategy_path if check_in_modelarts(): # local send all strategy file to obs obs_save_dir = os.path.join(config.remote_save_url, "strategy") mox.file.make_dirs(obs_save_dir) local_strategy_path = config.parallel.strategy_ckpt_save_file obs_strategy_path = os.path.join(obs_save_dir, os.path.basename(local_strategy_path)) mox.file.copy(local_strategy_path, obs_strategy_path) # obs send all strategy to each node logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if rank_id % 8 == 0: wait_collect_all_strategy(local_strategy_dir, world_size, obs_save_dir) else: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) merged_strategy_name = os.path.basename(merged_strategy_path) obs_merged_strategy_path = os.path.join(obs_save_dir, merged_strategy_name) mox.file.copy(merged_strategy_path, obs_merged_strategy_path) logger.info("Save %s to %s", merged_strategy_path, obs_merged_strategy_path) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_path else: logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if world_size <= 8: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_paths[0] else: logger.warning("Can't collecting all strategy, device num > 8!") config.auto_trans_ckpt = False dst_strategy_path = None return dst_strategy_path def transform_ckpt(config, dst_ckpt_strategy=None): """auto transform ckpt""" rank_id = get_rank() if config.use_parallel else 0 world_size = get_group_size() if config.use_parallel else 1 transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") os.makedirs(transformed_ckpt_dir, exist_ok=True) if rank_id % 8 == 0: logger.info(".........Transforming ckpt.........") src_ckpt_strategy = get_strategy(config.src_strategy_path_or_dir) logger.info("Src ckpt strategy: %s", src_ckpt_strategy) logger.info("Src ckpt: %s", config.load_checkpoint) logger.info("Dst ckpt strategy: %s", dst_ckpt_strategy) logger.info("Dst ckpt: %s", transformed_ckpt_dir) ms.transform_checkpoints(config.load_checkpoint, transformed_ckpt_dir, 'checkpoint_', src_ckpt_strategy, dst_ckpt_strategy) logger.info(".........Transform succeed!.........") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_{rank_id}.txt') f = open(transform_succeed_txt, 'w') f.close() if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") mox.file.make_dirs(transformed_ckpt_dir_obs) transform_succeed_txt_obs = os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{rank_id}.txt') mox.file.copy(transform_succeed_txt, transform_succeed_txt_obs) wait_transform(config, rank_id, world_size) if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") rank_transformed_ckpt_dir_obs = os.path.join(transformed_ckpt_dir_obs, f'rank_{rank_id}') rank_transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint", f'rank_{rank_id}') mox.file.copy_parallel(rank_transformed_ckpt_dir, rank_transformed_ckpt_dir_obs) print(f"Rank {rank_id}: Save {rank_transformed_ckpt_dir} to {rank_transformed_ckpt_dir_obs}") config.load_checkpoint = transformed_ckpt_dir def wait_transform(config, rank_id, world_size): """wait all node transform over""" print(f'Rank {rank_id}: Waiting transforming ckpt......') time_out = False if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") transform_succeed_txts_obs = [os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{int(i/8)*8}.txt') for i in range(0, world_size, 8)] total_num = len(transform_succeed_txts_obs) all_node_transform_succeed = False start_time = time.time() while not all_node_transform_succeed: node_transform_succeed_num = 0 for transform_succeed_txt in transform_succeed_txts_obs: if mox.file.exists(transform_succeed_txt): node_transform_succeed_num += 1 else: break if node_transform_succeed_num == total_num: all_node_transform_succeed = True else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Automatic weight transform is timed out!") time_out = True break time.sleep(5) else: transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_0.txt') start_time = time.time() while not os.path.exists(transform_succeed_txt): elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: time_out = True break time.sleep(5) print(f'Rank {rank_id}: Transform succeed!') if time_out: return False return True def wait_collect_all_strategy(strategy_dir, total_num, obs_strategy_dir=None): """wait all strategy collect over""" start_time = time.time() time_out = False while True: if obs_strategy_dir: obs_strategy_paths = mox.file.glob(os.path.join(obs_strategy_dir, "*.ckpt")) if len(obs_strategy_paths) < total_num: time.sleep(5) continue for obs_strategy_path in obs_strategy_paths: local_strategy_path = os.path.join(strategy_dir, os.path.basename(obs_strategy_path)) mox.file.copy(obs_strategy_path, local_strategy_path) local_strategy_paths = glob(os.path.join(strategy_dir, "*_rank_*.ckpt")) if len(local_strategy_paths) == total_num: break else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Collection of all strategy timed out!") time_out = True break time.sleep(5) if time_out: return False return True def load_ckpt(config, network, optimizer=None): """load checkpoint""" # load checkpoint params into dict logger.info(".............Start load checkpoint from checkpoint..................") if os.path.isdir(config.load_checkpoint): checkpoint_dict = load_distributed_checkpoint(config) else: checkpoint_dict = load_checkpoint(config.load_checkpoint) # load params into net not_load_network_params = load_param_into_net(network, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) if optimizer: not_load_optim_params = load_param_into_net(optimizer, checkpoint_dict) logger.info("Optimizer parameters are not loaded: %s", str(not_load_optim_params)) def get_last_checkpoint(checkpoint_dir): """get last checkpoint for resuming or finetune.""" if not os.path.isdir(checkpoint_dir): raise NotADirectoryError( f"{checkpoint_dir} is not a real directory," "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckpt") output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) pengcheng.mind-mindformer/mindformers/trainer/__init__.py000066400000000000000000000036761476650141400242300ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Trainer API.""" from .config_args import * from .image_classification import * from .masked_image_modeling import * from .masked_language_modeling import * from .general_task_trainer import * from .contrastive_language_image_pretrain import * from .image_to_text_retrieval import * from .image_to_text_generation import * from .translation import * from .text_classfication import * from .token_classification import * from .question_answering import * from .causal_language_modeling import * from .trainer import Trainer from .training_args import TrainingArguments from .base_trainer import BaseTrainer from .build_trainer import build_trainer __all__ = ['BaseTrainer', 'Trainer', 'TrainingArguments'] __all__.extend(config_args.__all__) __all__.extend(image_classification.__all__) __all__.extend(masked_image_modeling.__all__) __all__.extend(masked_language_modeling.__all__) __all__.extend(general_task_trainer.__all__) __all__.extend(contrastive_language_image_pretrain.__all__) __all__.extend(image_to_text_retrieval.__all__) __all__.extend(image_to_text_generation.__all__) __all__.extend(translation.__all__) __all__.extend(text_classfication.__all__) __all__.extend(token_classification.__all__) __all__.extend(question_answering.__all__) __all__.extend(causal_language_modeling.__all__) pengcheng.mind-mindformer/mindformers/trainer/__pycache__/000077500000000000000000000000001476650141400243135ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/__pycache__/__init__.cpython-39.pyc000066400000000000000000000020151476650141400305010ustar00rootroot00000000000000a >cf@sDdZddlTddlTddlTddlTddlTddlTddlTddlTddl Tddl Tddl Tddl Tddl TddlmZddlmZddlmZddlmZgdZeejeejeejeejeejeejeejeejee jee jee jee jee jdS) zMindFormers Trainer API.)*)Trainer)TrainingArguments) BaseTrainer) build_trainer)rrrN)__doc__ config_argsimage_classificationmasked_image_modelingZmasked_language_modelingZgeneral_task_trainer#contrastive_language_image_pretrainimage_to_text_retrievalimage_to_text_generation translationZtext_classficationtoken_classificationquestion_answeringZcausal_language_modelingtrainerrZ training_argsrZ base_trainerrr__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/trainer/__init__.pys>                pengcheng.mind-mindformer/mindformers/trainer/__pycache__/base_trainer.cpython-39.pyc000066400000000000000000000640771476650141400314200ustar00rootroot00000000000000a of@s4dZddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl Z ddlmZddlZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZm Z m!Z!dd l"m#Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;mZ>ddl?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHddlImJZJddlKmLZLmMZMmNZNddlOmPZPmQZQddlKmRZRmSZSmTZTmUZUmVZVe&WZXe&YZZe&[Z\e&]Z^e&_Z`d Zad!d"ZbGd#d$d$ZcdS)%z Base Trainer.N)pprint)partial)OptionalUnionList)Image)Tensor DatasetHelper)Model)Callback)GeneratorDataset)_VirtualDatasetCell)TrainOneStepCell OptimizerCell PipelineCellMicroBatchInterleaved) initializer)MindFormerBook)build_lr build_optimbuild_callback build_metric) EvalCallBack)build_parallel_config) build_datasetcheck_dataset_config BaseDataset) build_modelbuild_processorbuild_tokenizer BaseModel BaseTokenizerBaseImageProcessor)pipeline) build_wrapper)MindFormerConfig)logger) count_params) AutoModel)ConfigArguments)TrainingArguments)check_runner_configtransform_and_load_checkpoint#load_resume_context_from_checkpoint) get_optimizer_grouped_parameters$get_mup_optimizer_grouped_parameters)set_seedcheck_train_data_loader_typecheck_eval_data_loader_typecheck_optimizer_and_lr_typecheck_wrapper_configconfigsc Csddl}t|dz|jj||dWnNtyx}z6td|dtd|jj||dWYd}~n d}~00td|d |d d d dS) z Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id rN)src_urldst_urlzError: z, download again... zRank z download ckpt z succeed!T)flush)moxingtimesleepfilecopy Exceptionprint)Z remote_url local_urlrankmoxerrorrHK/home/ma-user/work/mindformers-baichuan/mindformers/trainer/base_trainer.pydownload_ckpt_from_obs>s &rJc @seZdZdZdVeedddZdWeeeee e fe ddd Z d d Z d d ZddZddZddZdXedddZdYedddZdZedddZd[edddZd\edddZd]eddd Zd^d!d"Zd_e ed$d%d&Zd'd(Zd`edd)d*Zdaedd+d,Zdbed-d.d/Zd0d1Zdced2d3d4Z d5d6Z!d7d8Z"dde d:d;d<Z#d=d>Z$d?d@Z%dAdBZ&dCdDZ'dee(eee(efdGdHdIZ)dfeeee*e e feee+e,feee-e.fee/ee0eee1e2e1fdJdKdLZ3dgeeee*e e feee+e,feee-e.feee1e2e1feeee4fdMdNdOZ5dheeee*e e feee.e6e7j8e9ee:feeee+e,fee;ee<ee<dPdQdRZ=e>e-dSdTdUZ?dS)i BaseTraineraBase Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N)task model_namec Cs|dur d}|durd}td||||_||_d|_d|_d|_d|_d|_d|_ d|_ d|_ d|_ d|_ d|_d|_d|_d|_d|_tjtjdts(tjdt}tjtjttr(tjtt}ztj||ddWn:ty&zt||Wnty Yn0Yn0|tvr@td|tvr|t |vrtdtd ttd t |j d |d |_dS) Nzmodel name unspecified.ztask name unspecified.z%Now Running Task is: %s, Model is: %s.T) dirs_exist_okzThe default model config: %s will now be used for the %s task common)!r'inforMrLconfigdefault_task_config train_dataset eval_datasetnetwork optimizerimage_processoraudio_processor tokenizer callbackseval_callbacks model_wrappercompute_metricskwargs pipeline_taskospathexistsjoinDEFAULT_CONFIG_DIRCURRENT_PROJECT_PATHshutilcopytree TypeErrorFileExistsError SUPPORT_TASKSkeyswarningget)selfrLrMZconfigs_directoryZmindformers_configs_directoryrHrHrI__init___sT  "  zBaseTrainer.__init__F)rRis_full_configcCsR|dur*|r||_q:||||_n||j|_t|j||jS)z%Set the task config for task trainer.N)rRsetup_task_config _merge_configrSr*_check_global_batch_size_for_auto_parallel)rorRrqrHrHrI set_configs zBaseTrainer.set_configcCs`d}|jtvr@|jt|jvr@tt|j|j}t|trR||_n t ddS)zSetup the default task config.NzIf the default config arguments is not specified,you must define the required model or wrapper, optimizer, and so onin the train or evaluate or predict attribute function.) rLrkrlrMrnr& isinstancerSr'rm)ro task_configrHrHrIrrs $ zBaseTrainer.setup_task_configc Cs|jjj}|jjj}|jjj}|jj}td}td}| }|dvrH|r|dkrt d||||||||||||||jj_| n.t d|||||||||||jj_n||dkrt d|||||||t d|||t td d |||t td d ||||jj_| n|t d ||t d |t td d |t td d d|jj_d|jj_d|jj_d |jj_d|jj_t d|jjdS)z.Check global batch size in auto parallel mode. parallel_mode full_batchsemi_auto_parallel auto_parallelr*zPipeline parallel was opened: pipeline_stages = %s, full batch is True, global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num = %s * %s * %s * %s = %s).zThe current parallel mode is %s, full batch is True,so global batch size will be changed: global_batch_size = batch_size * data_parallel * micro_batch_interleave_num = %s * %s * %s = %szPipeline parallel was opened: pipeline_stages = %s, full batch is False, batch size per card will be changed: per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num = %s * %s * %s = %s).z>global_batch_size = per_batch_size * device_num = %s * %s = %s RANK_SIZE1zbThe current parallel mode is %s, batch size per card will not be changed: batch_size_per_card = %szCglobal_batch_size = batch_size_per_card * device_num = %s * %s = %sFz5parallel_config will be change to default config: %s.N)rR runner_config batch_sizeparallel_config data_parallelmicro_batch_nummicro_batch_interleave_nummsget_auto_parallel_contextget_pipeline_stagesr'rQ$_reset_wrapper_for_pipeline_parallelintragetenvmodel_parallelpipeline_stageoptimizer_shard)rordprrrxrypprHrHrIrtsh                   z6BaseTrainer._check_global_batch_size_for_auto_parallelcCsb|jjdurJ|jjjdkrdn|jjj|jj_|jjj|jj_tdn tdtddS)z%Reset wrapper when pipeline parallel.NZMFPipelineWithLossScaleCellz`When using the pipeline parallel mode, the MFPipelineWithLossScaleCell class is used by default.zWhen using the pipeline parallel mode, because the wrapper class is not specified, MindSpore's built-in PipelineCell is used by defaultzDPipelineWrapper under evaluate or predict mode will not take effect.)rRrunner_wrappertyperrr'rmrQrorHrHrIrs  z0BaseTrainer._reset_wrapper_for_pipeline_parallelcCst|jdS)zMReset dataset batch size according to the global batch size of runner config.N)rrRrrHrHrI_reset_dataset_batch_sizesz%BaseTrainer._reset_dataset_batch_sizecCs(|jdurtd|St|tr0|j|nt|trtj |rRtj |sbt d|d| dszt d|t||_nt|trt|drt||jt|drt||jt|d rt||jt|d rt||j|j|jn*t|tr||jntd t||jS) z&Merge config from default task config.NzMdefault task config is None, you will not be able to merge config parameters.z#config path must be exist, but get rN)z.yamlz.ymlz4config file must be end with .yaml or .yml, but get rTrUrWrzThe type of config parameter to merge is not supported, currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], but get %s)rSr'rmrvdictmerge_from_dictstrrarbrealpathrcFileNotFoundErrorendswith ValueErrorr&r+hasattrr3r4r5r6__dict__r,"convert_args_to_mindformers_configr)roargsrHrHrIrss8                zBaseTrainer._merge_config default_argscCs&td|t|jj|d}|S)z&Create the train dataset for training.,.........Build Dataset From Config..........r)r'rQrrrRtrain_dataset_task)rorrTrHrHrIcreate_train_dataset)s z BaseTrainer.create_train_datasetcCsjtd|td}|dvrV|jjj|jjj |jj_|jjj|jj |jj_t |jj |d}|S)z%Create the eval dataset for evaluate.rrxrzr) r'rQrrrrRrUrrrrreval_dataset_task)rorrxrUrHrHrIcreate_eval_dataset0s  zBaseTrainer.create_eval_datasetcCsJtdt|jj|d}|}|jj}|dkrBtdt||}||fS)z$Create the network for task trainer.z,.........Build Network From Config..........rr*Nmicro_batch_interleave_num > 1, the double copy parallel feature is turned on.)r'rQrrRmodelrr)ror eval_networkrVrrHrHrIcreate_network>s   zBaseTrainer.create_networkcCsrtdt|jj|d}|}|jj}|jjj}|dkrVtdtt |||d}n t||d}t |}||fS)z9Create the network of pipeline parallel for task trainer.z5.........Build Pipeline Network From Config..........rr*r) micro_size) r'rQrrRrrrrrrr )rorrrVrrrHrHrIcreate_pipeline_networkIs     z#BaseTrainer.create_pipeline_networkcCs$tdt|jjj|d|_|jS)z'Create the image processor for predict.z4.........Build Image Processor From Config..........r)r'rQrrR processorrXrorrHrHrIcreate_image_processorYs   z"BaseTrainer.create_image_processorcCs$tdt|jjj|d|_|jS)z&Create the tokenizer for task trainer.z3.........Build Text Tokenizer From Config..........r)r'rQr rRrrZrrHrHrIcreate_tokenizer`s   zBaseTrainer.create_tokenizerc Cs"td|jj}|jj}|||}|jjjr:|jjjnd}|jjrN|jjnd}|jj j j dur|jj j j |jj j j }t ||||jj} nt|||||d} |durt|jj| |dd|_nZ|jjjdurtd|r|dur||jjj|n|jjj|jj_t|jjd | id|_|jS) z"Create the optimizer for training.z..........Build Optimizer From Config..........g?N) layer_scale layer_decay)params learning_raterzlearning_rate must be inputr)r'rQrRlr_scaleZlr_scale_factorcreate_lr_schedulerrW weight_decayrr model_configmup_width_base hidden_sizer1 lr_scheduler0rrrlearning_rate_scale) rorVrlearning_scale scale_factorrrrratio group_paramsrHrHrIcreate_optimizer_schedulergsR    z&BaseTrainer.create_optimizer_scheduler)rrc Cs~td|}|jjrn|jjdd}|jjdd}|jjdd}|jjjsjt|jjj |}nt|jjj |jjj }|dur|durt dd}|durt dt|||jj_ |jjj dus|jjj dkr|n t|jjj |jj_ |durt|jjj ||jj_ |dur|_t:|||||j6rtd |d!d }|d ur|?}t@tA|jB||d"|jCr|jCnd#|jDr|jDnd$d%}|E|td&tt ddd'd(kr@tF|td)|jG|jj|||jj|jj|jjd*td+d S),z2Train or Fine-tune for BaseTrainer in MindFormers.rqFzH.............Start load resume context from checkpoint..................RANK_ID0rank_*z/cache/checkpoint/rank_z.ckptz*.........Build Dataset For Train..........Nz&.........Build Net For Train..........r*r moe_configrz2.........Using The Existing Network For Train:: %sTrz,.........Build Optimizer For Train..........)rz>.........Build Running Wrapper From Config For Train..........z-.........Using The Existing Model Wrapper: %sz,.........Build Callbacks For Train..........r}r~)r origin_epochs dataset_sizerr initial_epochglobal_batch_sizerz,.........Starting Init Train Model..........metricsr)rWrrzD.............Start resume training from checkpoint..................)rWz6.........Build Evaluate in Training Callback..........rUrrUdr) step_intervalepoch_intervalz*.........Starting Training Model..........rz9.........Model Compiling, Please Wait a Moment...........)r[dataset_sink_moderrz$.........Training Over!.............)Hr_rnruresume_trainingload_checkpointr'rQrrarrbrdrestore_ckpt_name_prefixreplacerrJr/rrr-rrmathceilrrrrr _dataset_helperr]rVrrrrr __class____name__rrrrrrrrrWr data_sizerrrRrdo_evalrr only_save_strategyr.SUPPORT_MODEL_NAMESr)from_pretrained default_checkpoint_download_pathrrr_evaluate_in_trainingeval_step_intervaleval_epoch_intervalrrtrain)rorRrVrrWrr[r_rqrEZ obs_ckpt_urlrD epoch_numrr^rrU eval_callbackrHrHrItraining_process$s                                       zBaseTrainer.training_process)rRrVrr[r^c Ks|d}|dd}|||}td|dur<|}||td|d} |dur|jdur|j|j |j dd\}} n*|dur|jdurtd |jj j |j}|j |dd | dur|} |td |dur|j|d }td |dur|}tdt||| d} |js6|jrn|jtvrRt|jj|_tdt|| ||ddtdttddddkrt|| j|||jjd} td|t | tddS)z(Evaluate for BaseTrainer in MindFormers.rrqFz-.........Build Dataset For Evaluate..........Nz/Create evaluate dataset finish, dataset size:%drrz4.........Using The Existing Network For Evaluate: %srz5.........Build Compute Metrics For Evaluate..........rz/.........Build Callbacks For Evaluate..........z/.........Starting Init Evaluate Model..........rz=.............Start load checkpoint for eval..................T)r z*.........Starting Evaluate Model..........rrrr)r[rz%s = %sz$.........Evaluate Over!.............)!rnrur'rQrrrrVrrrrr rrrrr rr r r)rrr.rrarrevalrrr) rorRrVrr[r^r_rrqrroutputrHrHrIevaluate_processsZ               zBaseTrainer.evaluate_process)rR input_datarLrVrZrXrYcKs|js||dd} ||| }tjddvrT|dvr@td|jjdkrTtd|d urv|j |j|j d d \}} |j |dd |d d |_ | |d ur|jjrt|jj}|d ur|jjrt|jj}|d ur|jjrt|jj}t|} |js|jr`tjddvrN|jjj} td| ftjtd} t|| || ddnt|| |d ddtf|||||d||_| dd }|d ur|j!d ur|j!}n$|jr|jjj!r|jjj!}nd}| dd }|d ur|j"d ur|j"}n |d}t#t$%ddddkr t&||j||d}g}|D]6}t'|t(r`d|vr`|)|dn |)|q6t*|d}|+t,|Wd n1s0Y|-t./dt,|t./d|t./d|S)z'Predict for BaseTrainer in MindFormers.rqFrx)r{r|hybrid_parallel) translationtext_generationzYCurrently distributed predict only support translation and text_generation. Process exit!r*zOCurrently distributed predict dose not support pipeline parallel. Process exit!NrrrrM)shapedtypeinitT) do_predict)rLrrZrXrYtop_k save_filez _result.txtrrrr)r"rQwzoutput result is: %szoutput result is saved at: %sz#.........Predict Over!.............)0r`rnrurcontextr SystemExitrrrrrrMrrrZr rXrrYr rr rr seq_lengthrint32r Oner.r$rr"r#rrarrrvrropenwritercloser'rQ)rorRrrLrVrZrXrYr_rq_rr'Z infer_datar"r#Zoutput_results output_infoZ one_outputr@rHrHrIpredict_processs                    .  zBaseTrainer.predict_processrcCs8|jj}|jd|j||jjjd}|j||S)NF)r)rphaserrrRrr)rorrU origin_phaserrHrHrIrXs   z!BaseTrainer._evaluate_in_training)NN)NF)N)N)N)N)N)N)F)Fr)N)N)N)N)T)rr)NNNNNN)NNNNN)NNNNNNN)@r  __module__ __qualname____doc__rrprrrr+r,boolrurrrtrrrsrrrrrrrrrrrrrrr2rrrrrrrrrr&rr!rr rrr rrsetrrnpndarrayrlistr"r#r/r rrHrHrHrIrKVs6  A!  (+      A  drK)dr4rrargr>r functoolsrtypingrrrnumpyr7 PIL.Imager mindsporerrr mindspore.train.modelr mindspore.trainr mindspore.datasetr mindspore.nn.wrap.cell_wrapperr mindspore.nnrrrrrmindspore.commonrr mindformers.mindformer_bookrmindformers.corerrrrZ"mindformers.core.callback.callbackr mindformers.core.parallel_configrmindformers.datasetrrrmindformers.modelsrrr r!r"r#mindformers.pipeliner$mindformers.wrapperr%mindformers.tools.registerr&mindformers.tools.loggerr'mindformers.tools.utilsr(mindformers.auto_classr) config_argsr+ training_argsr,utilsr-r.r/Zoptimizer_grouped_parametersr0r1r2r3r4r5r6get_trainer_support_task_listrkget_model_name_support_listr get_pipeline_support_task_listZSUPPORT_PIPELINES$get_pipeline_support_input_data_listZSUPPORT_PIPELINE_INPUT_DATAget_project_pathrfrerJrKrHrHrHrIsT                         pengcheng.mind-mindformer/mindformers/trainer/__pycache__/build_trainer.cpython-39.pyc000066400000000000000000000054561476650141400316010ustar00rootroot00000000000000a >cfL@stdZddlmZmZmZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZd eeeeddd Zd d ZedS) zBuild Trainer API.)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) ImageClassificationTrainer"ZeroShotImageClassificationTrainerMaskedImageModelingTrainerMaskedLanguageModelingTrainerImageToTextRetrievalTrainerTranslationTrainerTokenClassificationTrainerTextClassificationTrainer'ContrastiveLanguageImagePretrainTrainerQuestionAnsweringTrainerGeneralTaskTrainerNtrainer)config default_args module_type class_namecKsd|dur|durdS|durPt|tr>t|ts>tfi|}tj|tj|dStj||fi|S)a|Build trainer API. Instantiate the task trainer from MindFormerRegister's registry. Args: config (dict): The task trainer's config. Default: None. default_args (dict): The default argument of trainer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'trainer'. class_name (str): The class name of task trainer API. Default: None. Return: The function instance of task trainer API. Examples: >>> from mindformers import build_trainer >>> trainer_config = {'type': 'image_classification', 'model_name': 'vit'} >>> # 1) use config dict to build trainer >>> cls_trainer_config = build_trainer(trainer_config) >>> # 2) use class name to build trainer >>> cls_trainer_class_name = build_trainer(class_name='image_classification', model_name='vit') N)r) isinstancedictrrget_instance_from_cfgrTRAINER get_instance)rrrrkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/trainer/build_trainer.py build_trainersrcCstjttjddtjttjddtjttjddtjttjddtjttjddtjt tjddtjt tjddtjt tjd dtjt tjd dtjt tjd dtjttjd dd S)z( register MindFomrers builtin LR class. image_classification)raliaszero_shot_image_classificationmasked_image_modeling fill_masktoken_classificationtext_classification#contrastive_language_image_pretrain translationquestion_answeringimage_to_text_retrievalgeneralN)r register_clsrrrrrr r r rr rr rrrrrregister_mf_trainer8sTr,)NNrN)__doc__mindformers.tools.registerrrrrrrr r r r r rrrrstrrr,rrrrs4 !.pengcheng.mind-mindformer/mindformers/trainer/__pycache__/config_args.cpython-39.pyc000066400000000000000000000232521476650141400312310ustar00rootroot00000000000000a >cfl5@sFdZddlZddlmZmZddlmZddlZddlm Z ddl m Z m Z gdZ eGdd d ZeGd d d eZeGd d d eZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZeGdddeZdS) zMindFormers' Config API.N)OptionalUnion) dataclass)CheckpointMointor)MindFormerRegisterMindFormerModuleType) BaseArgsConfig RunnerConfig DatasetConfigDataLoaderConfigConfigArguments ContextConfig CloudConfigCheckpointConfigParallelContextConfigOptimizerConfigLRConfig WrapperConfigc@seZdZdZgZddZdS)rzBase Argument config.cKsd|dur`|D]N\}}||jvs8Jd|jd||durBqt|trR|j}|||qdS)Nz+The Config Class support input argument is z , but get )items_support_kwargs isinstancer__dict__ __setattr__)selfkwargskeyvaluerJ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/config_args.py__init__$s  zBaseArgsConfig.__init__N)__name__ __module__ __qualname____doc__rrrrrrrsrcsZeZdZdZgdZddeeddddfee ee fe ee e d fd d Z Z S) r zContext Config.)modeprecompile_only device_target device_id save_graphssave_graphs_path enable_dumpauto_tune_modesave_dump_pathenable_reduce_precisionvariable_memory_max_sizeenable_profilingprofiling_optionsenable_auto_mixed_precisionenable_graph_kernelreserve_class_name_in_scope check_bpropmax_device_memoryprint_file_path enable_sparsemax_call_depthenv_config_pathgraph_kernel_flagssave_compile_cacheruntime_num_threadsload_compile_cachegrad_for_scalarpynative_synchronizemempool_block_sizerAscend DEVICE_ID0F.)r$r&r'r(r)c s&tt|jf|||||d|dS)N)r$r'r&r(r))superr r)rr$r&r'r(r)r __class__rrr>szContextConfig.__init__)r r!r"r#rintosgetenvrrstrboolr __classcell__rrrFrr 0s r csFeZdZdZgdZdeedddfeee dfdd Z Z S) rzParallel Context Config.) device_num global_rankgradients_meangradient_fp32_sync parallel_modeauto_parallel_search_mode search_modeparameter_broadcaststrategy_ckpt_load_filestrategy_ckpt_save_file full_batchenable_parallel_optimizerenable_alltoallall_reduce_fusion_configpipeline_stagesgrad_accumulation_stepparallel_optimizer_config comm_fusion STAND_ALONE RANK_SIZE1FrRrNrPc s"tt|jf|||d|dS)Nrc)rErr)rrRrNrPrrFrrrVs zParallelContextConfig.__init__) r r!r"r#rrHrIrJrKrLrrMrrrFrrJs rcs8eZdZdZgdZd eeeeedfdd ZZ S) rzCloud Config For ModelArts.)obs_path root_pathrank_idupload_frequence keep_lastretry retry_timeN/cacheFrdrerfrgrhc s&tt|jf|||||d|dS)Nrm)rErr)rrdrerfrgrhrrFrrrgszCloudConfig.__init__)NrkNrlF) r r!r"r#rrKrHrLrrMrrrFrr_src s<eZdZdZgdZdeeeeeeedfdd ZZS)r z'MindFormers' config when running model.) epochs batch_size sink_mode sink_size initial_epochhas_trained_epocheshas_trained_steps image_size num_classesrqN)rnrorprqrrrsrtc s*tt|jf|||||||d|dS)N)rnrorprqrrrtrs)rEr r) rrnrorprqrrrsrtrrFrrr~szRunnerConfig.__init__)NNNNNNN) r r!r"r#rrHrLrrMrrrFrr tsr c s@eZdZdZeejZd e e e e e e e dfdd Z Z S) rz$MindFormers' save checkpoint config. mindformersNrlTF)prefix directorysave_checkpoint_stepskeep_checkpoint_maxintegrated_save async_save saved_networkc s*tt|jf|||||||d|dS)N)rxryr~rzr{r|r})rErr) rrxryrzr{r|r}r~rrFrrrszCheckpointConfig.__init__)rwNrlrlTFN)r r!r"r#inspectgetfullargspecrargsrrKrHrLrrMrrrFrrs$ rcs0eZdZdZgdZdedfdd ZZS)rz+MindFormers' learning rate schedule config.)typemax_lrmin_lr decay_steps decay_ratepowerend_learning_rate warmup_stepsN)lr_typec sJ|dur,tjtj|d}|jt|jt t |j fd|i|dSN module_type class_namer) rget_clsrLRrextendrrrrErr)rrr lr_schedulerFrrrs zLRConfig.__init__)Nr r!r"r#rrKrrMrrrFrrsrcs>eZdZdZgdZdeeeee fdfdd Z Z S)rzMindFormers' optimizer config.) r learning_ratebeta1beta2epsepsilon weight_decay loss_scalemomentumN) optim_typerc sL|dur,tjtj|d}|jt|jt t |j f||d|dS)Nr)rr) rrr OPTIMIZERrrrrrrErr)rrrr optimizerrFrrrszOptimizerConfig.__init__)NN) r r!r"r#rrKrrrfloatrrMrrrFrrsrcs0eZdZdZgdZdedfdd ZZS)rzMindFormers' wrapper config.)rsens scale_senseN) wrapper_typec sJ|dur,tjtj|d}|jt|jt t |j fd|i|dSr) rrrWRAPPERrrrrrrErr)rrrwrapperrFrrrs zWrapperConfig.__init__)NrrrrFrrsrcs2eZdZdZgdZdeedfdd ZZS)r z MindFormers' data loader config.)r dataset_dir num_samplesnum_parallel_workersshufflesampler extensionsclass_indexing language_pairdecode num_shardsshard_idcachedecrypttaskusagetest_set valid_set padded_sample num_paddedN)dataloader_typerc sL|dur,tjtj|d}|jt|jt t |j f||d|dS)N)r)rr) rrrDATASET_LOADERrrrrrrEr r)rrrr dataloaderrFrrrszDataLoaderConfig.__init__)NNrrrrFrr sr c s|eZdZdZgdZd eeeefeee e feee e feee e fe e e eee e e fdfdd ZZS) r zMindFormers' dataset config.) data_loader input_columnsoutput_columns column_orderdrop_remainderrepeatrorur per_batch_mappython_multiprocessing max_rowsizeroffloadNTrl)rrrrrrroruc s,tt|jf||||||||d| dS)N)rrorurrrrr)rEr r) rrrrrrrrorurrFrrrszDatasetConfig.__init__)NNNNTrlNN)r r!r"r#rrrdictrrKlistrLrHtuplerrMrrrFrr s r cseZdZdZgdZd eeeeeeee e e fee e e fee e e fee e e fee e e fee e e fee e e fee e e fdfd d Z Z S) r zMindFormers' config arguments.) output_dirprofile auto_tunefilepath_prefixautotune_per_step train_dataset eval_datasetZpredict_dataset runner_configrrsave_checkpoint cloud_configseedrunner_wrapper./outputF ./autotune N)rrrrrrrrrrrrrrcs0tt|j||||||||| | | | | |ddS)N)rrrrrrrrrrrrrr)rEr r)rrrrrrrrrrrrrrrrFrrrs zConfigArguments.__init__)rFFrrNNNNNNNNN)r r!r"r#rrKrLrHrrrrrrMrrrFrr s2r )r#rItypingrr dataclassesrrmindformers.core.callbackrmindformers.tools.registerrr__all__rr rrr rrrrr r r rrrrs>  optimizer_grouped_parameters.cpython-39.pyc000066400000000000000000000120351476650141400346600ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/__pycache__a of@sdZddlZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl mZdd lmZdd lmZd d lmZdee eee eedddZdee eedddZddZddZdS)z Grouped parameters of optimizer.N)partial)Optional)Cell)LearningRateSchedule) BaseModel)LearningRateWiseLayer)build_lr)logger)check_keywords_in_nameF?)model weight_decaydynamic_lr_schedule layer_scale layer_decaycs^t|ttfs tdt|iit|drF|tdt|drd| tdfdd| D}d}g}|durR|rD|j j d kr|j j}t|} tt| d |d }tfd d tt| d D}n\|j j dkr0|j j} tt| d d}tfdd tt| d D}ntdd}d}ntdd}i} i} d} | D]} | j|vr|d}nd}d}|dur|| j}d||f}|dur||} || vr|gd| |<|gd| |<t|tr|rt|| | |d<n || |d<| |d| | |d| jqftj| d d}td|t| S)3Get grouped parameters of the network for training.(model type should be BaseModel, but get no_weight_decayzNo weight decay: %sno_weight_decay_keywordszNo weight decay keywords: %scsBg|]:}t|jdks|jds|jvst|js|jqS)r z.bias)lenshapenameendswithr ).0param) skip_keywords skip_params[/home/ma-user/work/mindformers-baichuan/mindformers/trainer/optimizer_grouped_parameters.py 1s    z4get_optimizer_grouped_parameters..NZSwinForImageClassification) num_layersdepthsc3s|]}|VqdSNrrirrr Az3get_optimizer_grouped_parameters..ZViTForImageClassification)r#c3s|]}|VqdSr%rr&r(rr r)Er*zsNow only support VitModel and SwinModel,if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.FzHdynamic_lr_schedule will be reset and invalid when layer_scale is False.r decayZno_decayr z layer_%d_%s)rparamslrr,)indentzParam groups = %s) isinstancerr TypeErrortypehasattrrr infortrainable_params __class____name__configr$sumrget_swin_layerlistreversedrangedepth get_vit_layerwarningrrrappendjsondumpsvalues)rrrrrZdecay_parameters_namesZget_layer_id_funcZ scales_listr$r#Zparameter_group_namesparameter_group_varsscaler group_namelayer_id param_groupsr)rrrr get_optimizer_grouped_parameters sx       $$              rI)rr mup_ratioc Cst|ttfs tdt|dd}gd}i}dgi}|D]}||j|rD|d|qDt |} | j || _ | j || _ t | } | |d<||d<||d<d d g} dgi} |D]}||j| r| d|qd | d<| |d <t |S) rrcSs"|D]}||dkrdSqdS)NrTF)find)rkey_listkrrr find_in_name~sz:get_mup_optimizer_grouped_parameters..find_in_name)z attention.woz attention.wqz attention.wkz attention.wv feed_forwardlm_headr,r-r middle_outputnorm embeddingr Znorm_key)r/rrr0r1r4rr@copydeepcopy learning_ratelr_endrr:rC) rrrJZlr_schedule_configrNZmiddle_weight_key_listrDZmiddle_weight_grouprZmiddle_lr_configZ middle_lrZ norm_key_listZnorm_weight_grouprrr $get_mup_optimizer_grouped_parametersvs4        rXcCsR|drd}n>|dr d}n.|drFt|dd}|d}n|d}|S)z get vit layer)Z cls_tokens mask_tokens pos_embedrzvit.patch_embedz vit.blocks.r"r )r startswithintsplit)rr# layer_numrGrrr r>s    r>cCs|dvrd}n|drd}nr|drt|dd}|dd}|dvrft|d |d }qt|d |t|}|d }n|d }|S) zget swin layer)zencoder.mask_tokenrzencoder.patch_embedzencoder.layersr[r") reductionrRNr )r\r]r^r8)rr#r$r_rGblock_idrrr r9s   r9)Nr NFr )Nr r N)__doc__rA functoolsrtypingrrT mindspore.nnr#mindspore.nn.learning_rate_schedulermindformers.modelsrZmindformers.core.lrrmindformers.corermindformers.toolsr utilsr floatboolrIrXr>r9rrrr s<          V 3pengcheng.mind-mindformer/mindformers/trainer/__pycache__/trainer.cpython-39.pyc000066400000000000000000001127661476650141400304250ustar00rootroot00000000000000a >cf5@sdZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl Zddl mZddlmZddl mZmZdd lmZmZmZmZmZdd lmZdd lmZdd lmZdd l m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7ddl8m8Z8ddl9m:Z:ddl;mZ>m?Z?m@Z@mAZAmBZBdgZCe+DZEe+FZGe+HZIe+JZKe+LZMdZNdZOGdddZPd$eQeRddd ZSd%eRd!d"d#ZTdS)&zTrainer API For Import.N) OrderedDict)ListOptionalUnion)Image)Tensor)set_seed)load_checkpointload_param_into_net)TrainOneStepCell OptimizerCell PipelineCellMicroBatchInterleaved)_VirtualDatasetCell)Callback)GeneratorDataset) BatchDataset RepeatDataset)build_parallel_config) build_datasetbuild_dataset_loadercheck_dataset_config BaseDataset)MindFormerBook) build_model BaseModelBaseImageProcessor BaseTokenizerBaseAudioProcessor)logger)MindFormerConfig)ordered_yaml_dump) build_trainer)ConfigArguments)TrainingArguments)check_train_data_loader_typecheck_eval_data_loader_typecheck_optimizer_and_lr_typecheck_wrapper_config config2dictTrainer checkpointconfigsc@seZdZdZd4eeeeee feeeeee e feeeeeee feeee fee eeeeeeeeeeeeeeefeeeeefeeeefedddZd5eeeefeeeeed d d Zd6eeeefeeeeed d dZd7eeeefeedddZd8eeeefeeeeeeejeeefdddZ d9eeeefedddZ!d:ddZ"d;ddZ#ded,d-d.Z)d/d0Z*d?eeeefd1d2d3Z+dS)@r,a Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) NgeneralF)argstaskmodel model_name train_dataset eval_dataset tokenizerimage_processoraudio_processor optimizerswrapper pet_method callbackseval_callbackscompute_metrics save_configc Ks>||_||_||_| |_||_||_| |_| |_||_||_ | |_ | |_ ||_ ||_ d|_tjdt|_||_tjtjdts tjdt}tjtjttr tjtt}ztj||ddWn:ty zt||WntyYn0Yn0| dur>|dur*td| dur>td|tvsfJdtd|dt|t r| r|d!| n|}|t"vsJd t"d |d||_d|_t|jt#t$frt%d | rtd d|_&nd |_&|j&r(|jdur(tdtdtdd|_t't(|j(|j}|jdkr|jdurf|jj)j*|j+_|jdur|jj,j)j*|j+_|dur||_-n t|t.r|/|nt|t r tj0|rtj|sJd|d|1dsJd|t'|}nt|t2r|t3|dr,t4||t3|drBt5||t3|drXt6||t3|drnt7|||/|j8n t|t9rtd|:|||_-|r|;|j-t%d|j+j<t|t rtj|sJd|d||j-jj=_>d|_t|t r@tj|s.Jd|d||j-jj=_>d|_|durz|j-jdurb||j-j_|j-jdurz||j-j_t?|j-t@tAdd|_Bt@tAdd |_C|jB|j-_B|jC|j-_Cd |_Dd |_Ed |_F|j-jGrtHjIJd!d"vrtK|j-jGtLjMG|j-jGtjNOd#|j-jPtQ|j-j+|_+|j+dur:tRd$dS)%N.T) dirs_exist_okzJwrapper has existed, input model invalid, it should be include in wrapper.zOwrapper has existed, input optimizers invalid, it should be include in wrapper.ztask name must be in z , but get z_{}zmodel must be in z& when model's type is string, but get zXThe model instance has been entered, and the model will not be created from model_configzpet_method is not valid when a model instance is passed in.Currently, only part of the model in MindFormers is supported.Please pass in the model keywordFzARecognizing that a model instance is sent and model_name is None,zit is recommended to select a model configuration that corresponds to the support of MindFormers based on the instance model and set model_name.zOtherwise, they will default to a general configuration.You are advised to pass instances such as optimizers, metric, tokenizer, and processorcommonz#config path must be exist, but get )z.yamlz.ymlz4config file must be end with .yaml or .yml, but get r5r6 optimizerrunner_wrapperzeWhen using the TrainingArguments class, its arguments will override the default config configuration.z&save running config success of %s_new.z*train dataset path must be exist, but get z)eval dataset path must be exist, but get RANK_ID0 RANK_SIZE1 parallel_mode)semi_auto_parallel auto_parallelLOCAL_DEFAULT_PATHz3config must be contain 'trainer' key, but get None.)Sr2r3r4r<r5r6r:r;r7r8r9r=r>r?default_checkpoint_name_or_pathospathjoinDEFAULT_CONFIG_DIRconfigs_directorykwargsexistsCURRENT_PROJECT_PATHshutilcopytree TypeErrorFileExistsErrorr warning SUPPORT_TASKSkeys isinstancestrformatSUPPORT_MODEL_NAMESr rinfois_model_instancer!get __class____name__trainernetworkconfigdictmerge_from_dictrealpathendswithr%hasattrr'r(r)r*__dict__r&"convert_args_to_mindformers_configsave_config_to_yamllower data_loader dataset_dirrintgetenvrank_id device_numis_set_parallel_configis_set_moe_configis_set_recompute_configseedmscontextget_auto_parallel_contextrnprandomenviron setdefault output_dirr$ModuleNotFoundError)selfr1r2r3r4r5r6r7r8r9r:r;r<r=r>r?r@rTrSmindformers_configs_directory task_configrF/home/ma-user/work/mindformers-baichuan/mindformers/trainer/trainer.py__init__s                                              zTrainer.__init__)train_checkpointresume_trainingauto_trans_ckptdo_evalc KsX|dur$t|ttfs$td||dur0d}|p:|jj}|r|jdurbtdt |jj |_|jdur~t d|j dd|j_|durd|jj j_||j_n:t|trd|jj j_||j_n|jj jj|_d|jj j_|dur||j_|dur||j_|jr|jdd|jjf|j|j |j|j|r<|jnd|j|jdd |dS) aPTrain task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) NzPresume_or_finetune_from_checkpoint must be one of [None, string, bool], but get F6do_eval is enabled, building eval_dataset from config.9if do_eval is true, eval_dataset must be input, the task  is not support eval now.Tis_trainrirhdatasetrDr6r;r=is_full_config)r^boolr_rYrirr6r rbreval_dataset_task ValueErrorr2r3 model_configcheckpoint_name_or_pathget_last_checkpointr rNrrrcreset_model_instancergtrainr5r:r;r=)rrrrrrTrrrrWsT&             z Trainer.train)finetune_checkpointrrrc Ks|dur$t|ttfs$td||dur0d}|p:|jj}|r|jdurbtdt |jj |_|jdur~t d|j dd|j_|durd|jj j_||j_nxt|trd|jj j_||j_nX|jj jj|_|r |jrtdn|jj jj|j_d|jj j_nd|j_d|jj j_|dur2||j_|durD||j_|jrX|jdd |jjf|j|j |j|j|r||jnd|j|jdd |dS) atFinetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') Nz=load_checkpoint must be one of [None, string, bool], but get FrrrTzWhen a model instance is identified,the weights that are currently proposed to be fine-tune are specifiedby the finetune_checkpoint argument or a model instance with the model weights already loaded is importedrr)r^rr_rYrirr6r rbrrrr2r3rrrr rNrcr[rrrrgrr5r:r;r=)rrrrrrTrrrfinetunesd+              zTrainer.finetune)eval_checkpointrcKs|dur$t|ttfs$td||dur0d}|durRd|jjj_||j_ njt|trrd|jjj_||j_ nJ|jjjj|_ |r|j rt dn|jjjj|j_ d|jjj_nd|j_ |dur||j_|j r|jdd|jjf|j|j|j|jdd|dS)aEvaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') Nz=eval_checkpoint must be one of [None, string, bool], but get FTWhen a model instance is identified,the weights that are currently proposed to be evaluated are specifiedby the eval_checkpoint argument or a model instance with the model weights already loaded is importedr)rirhrr=r)r^rr_rYrir3rrrr rNrcr r[rrrgevaluater6r>)rrrrTrrrr s@     zTrainer.evaluate)predict_checkpointr input_datac Ks|dur$t|ttfs$td||jtvrNtd|jdtd|durZd}|dur|d|jj j _ | |j_ njt|trd|jj j _ ||j_ nJ|jj j j |_|r|jrtdn|jj j j |j_ d|jj j _ nd|j_ |dur||j_|durt|jjj}td t|ttttttjtttf s@Jd |jrT|j dd |j!|d <|j"j#f|j||j |j$|j%|j&dd |}|S)azPredict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) Nz@predict_checkpoint must be one of [None, string, bool], but get zThe z% not support predict, now this tasks z is support predict.FTrz(dataset by config is used as input_data.z`Input data's type must be one of [GeneratorDataset, str, ms.Tensor, np.ndarray, PIL.Image.Image]rr4)rirrhr8r9r7r)'r^rr_rYr2SUPPORT_PIPELINESr]NotImplementedErrorrir3rrrr rNrcr r[rrr6rsrbrrrrrrndarrayrlistrr4rgpredictr8r9r7)rrrrrTZ output_resultrrrrOs`%           zTrainer.predictT)input_checkpointrcCsh|jdur,|jdkr,tdt|jj|_|jdurD|j|ntd|j|jrd|rd| dS)zbuild network for trainer.Nr/z'...........Start Init Network..........z0network will be create in %s task trainer class.) r3r2r rbrri set_trainr[rc_load_model_checkpoint)rrrrrr build_networks   zTrainer.build_networkr#c Csl||jj_||jj_||jj_||jj_||jj_||jj_||jj_| |jj_ | |jj_ ||j_ d|_ dS)an set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) TN) riparallel_config data_parallelmodel_parallelexpert_parallelpipeline_stageuse_seq_paralleloptimizer_shardmicro_batch_num vocab_emb_dpgradient_aggregation_groupmicro_batch_interleave_numry) rrrrrrrrrrrrrrset_parallel_configs         zTrainer.set_parallel_configcCs<||jj_||jj_||jj_||jj_||jj_d|_dS)a>Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) TN)rirecompute_config recomputeselect_recompute!parallel_optimizer_comm_recomputemp_comm_recomputerecompute_slice_activationr{)rrrrrrrrrset_recompute_configs      zTrainer.set_recompute_config皙?皙?c CsZ||jj_||jj_||jj_||jj_||jj_||jj_||jj_||jj_ d|_ dS)aThe configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) TN) ri moe_config expert_numcapacity_factoraux_loss_factornum_experts_chosenexpert_group_sizegroup_wise_a2acomp_comm_parallelcomp_comm_parallel_degreerz) rrrrrrrrrrrrset_moe_configs&        zTrainer.set_moe_configcCs8d|j|j|jfvrdS|jr(td|jr8td|jrHtdt|jts\tdt |j |jj }|j j |_ |j j |_ |j ||j||sdS|j}|j j}tddkr |j j j}|dkrtd tt|j||d }nt|j|d }t|}n |dkr*td t|j|}|`||_dS) z*Reset model instance for new model config.TNz2The incoming model will be configured in parallel.z3The incoming model will be configured in recompute.z-The incoming model will be configured in moe.zJCurrently only the integrated model structure in MindFormers is supported.pipeline_stagesr#zNmicro_batch_interleave_num > 1, the double copy parallel feature is turned on.) micro_size)ryrzr{r rbr^r3rrrrirrrrrr}rrrrr)rrrrhrrrrrr7sD                zTrainer.reset_model_instancecCst|jjjS)z"get train dataloader of mindspore.)rrir5rsrrrrget_train_dataloaderbszTrainer.get_train_dataloadercCst|jjjS)z!get eval dataloader of mindspore.)rrir6rsrrrrget_eval_dataloaderfszTrainer.get_eval_dataloadercsb|jj}tj|td|jddtD}|sos z/Trainer.get_last_checkpoint..Ncstjtj|S)N)rOrPgetmtimerQ)xcheckpoint_dirrrvz-Trainer.get_last_checkpoint..)key) rirrOrPrQDEFAULT_CHECKPOINT_DIRr`rwlistdirsorted)rZ output_folderoutput_checkpoint_pathrrrrjs zTrainer.get_last_checkpointricCsx|dur|j}|jjj}t|}tj|j|d}tj |sRtj |ddtj|d |}t ||dS)z"save now config file to yaml file.N_newT)exist_okz run_{}.yaml) rirgr4_reset_config_for_saverOrPrQrSrrrUmakedirsr`_save_config_to_yaml)rrir4 config_dictZ config_dirZ run_yaml_pathrrrrqys  zTrainer.save_config_to_yamlcCs|jjjj}|dur tdnt|trtj |}|t v}|rrt dt |}t|j|}t dt|q|rt dt|jttfr|j |jjjqtdt|jqtd|ntdt|dS) z!Load model checkpoint to network.Nz;checkpoint_name_or_path is None, not load input checkpoint.z9now input valid checkpoint path, it will load to network.znot load parameters is: %sz9now input valid checkpoint name, it will load to network.z0model must be BaseModel or Cell type, but get %szqinput checkpoint args is invalid, it must be valid and real checkpoint path or a valid checkpoint name,but get %szMcheckpoint_name_or_path type error, it should be one of [None, str], but get )rir3rrr r[r^r_rOrPrUrarbr r r rtyperY)rrZ is_exist_pathZis_checkpoint_namecheckpoint_dictZnot_load_paramsrrrrs,       zTrainer._load_model_checkpoint)r-cCsN|dur||jjj_n0t|tr2||jjj_n|jdurJ|j|jjj_dS)zcheck checkpoint config.TN)rrir3rrr^r_rN)rr-rrr_check_checkpoint_configs   z Trainer._check_checkpoint_config)Nr/NNNNNNNNNr0NNNF)FNNF)FNNF)FN)NNN)NT) r#r#r#r#r#r#FFrT)FFFTF)r#rrr#NFFr)T)N)N),rf __module__ __qualname____doc__rrr_rjr%r&r rrrrrr r rrsetrrrrrrrrrrrrrrrrrrrrrqrrrrrrr,=sj  2 S c"D  ^ ,  1 +)save_file_pathr@c CsT|dur i}t|ddd&}|t|ddddWdn1sF0YdS)zSave Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. Nwzutf-8)encodingFT)default_flow_style allow_unicode sort_keys)openwriter")rr@ file_pointerrrrrsrrcCs|dur i}|}t}|ddurBt|d}|d||ddurjt|d}|d||ddur|ddurt|d}t|d}|d||d||ddur|ddurt|d}t|d}|d||d||ddurt|d}t|d } t|d } t|d } t|d } |d||d | |d | |d | |d | |ddurt|d} |d| |ddurt|d}|d||ddurt|d}|d||ddurHt|d}|d||ddurrt|d}|d|t|}|D]\}}|||q|S)zReset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. Nr3 processortrain_dataset_taskr5rr6r~parallelrrr moe_conifg runner_configrErD lr_scheduler=)copyrrdr+popritems)rirrZprocessor_configZtrain_dataset_configZtrain_dataset_task_configZeval_dataset_configZeval_dataset_task_configcontext_configZparallel_context_configrrrrZwrapper_configZ optim_configZ lr_configZ cb_configZ run_configrvaluerrrrsd                 r)NN)N)UrrOrW collectionsrtypingrrrnumpyrZ PIL.Imager mindsporer}rmindspore.commonrr r mindspore.nnr r r rrmindspore.nn.wrap.cell_wrapperrZmindspore.trainrmindspore.datasetrZ!mindspore.dataset.engine.datasetsrr mindformers.core.parallel_configrmindformers.datasetrrrrmindformers.mindformer_bookrmindformers.modelsrrrrrmindformers.tools.loggerr mindformers.tools.registerr!Z!mindformers.tools.register.configr"r$ config_argsr% training_argsr&utilsr'r(r)r*r+__all__get_trainer_support_task_listr\get_model_name_support_listraget_pipeline_support_task_listr$get_pipeline_support_input_data_listSUPPORT_PIPELINE_INPUT_DATAget_project_pathrVrrRr,r_rjrrrrrrsV                    wpengcheng.mind-mindformer/mindformers/trainer/__pycache__/training_args.cpython-39.pyc000066400000000000000000000220041476650141400315710ustar00rootroot00000000000000a >cf9@s~dZddlZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z mZmZd d Zd d ZeGd ddZdS)z'Default Training Arguments for Trainer.N)OptionalUnion) dataclassfield)MindFormerConfig)logger)LRType OptimizerTypeSaveIntervalStrategycCs|durt|trdSdS)z,check task config for adapting hugging-face.NTF) isinstancer)Z check_configr L/home/ma-user/work/mindformers-baichuan/mindformers/trainer/training_args.py_check_task_configsrcCs|dur |S|S)z/check training arguments for adapt MindFormers.Nr ) ori_value new_valuer r r_check_training_args#src@seZdZUdZedddidZeed<edddidZe ed<eddd idZ e ed <eddd idZ e ed <eddd idZ e ed<edddidZ e ed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed<edddidZeed <eddd!idZeed"<eddd#idZeed$<eddd%idZeeefed&<eddd'idZeeefed(<ed)dd*idZeed+<eddd,idZ ee!efed-<eddd.idZ"e ed/<eddd0idZ#eed1<eddd2idZ$eed3<eddd4idZ%e&eed5<eddd6idZ'eed7<eddd8idZ(e&eed9<eddd:idZ)e&eed;<edddidZ+e ed?<eddd@idZ,e edA<edddBidZ-eedC<edddDidZ.eedE<ee/0dFdGZ1ee/0dHdIZ2ee/0dJdIZ3dKdLZ4e5dMdNZ6e5dOdPZ7e5dQdRZ8d`e9dSdTdUZ:dVdWZ;dXdYZd^d_Z?dS)aTrainingArgumentsz TrainingArguments is the subset of the arguments we use in our default config **which is relate to the training in MindSpore**. NhelpzQThe output directory where the model predictions and checkpoints will be written.)defaultmetadata output_dirzFThe use_parallel is used to enable distribute parallel of the network. use_parallelz7The profile is used to enable profiling of the network.profilezFThe only_save_strategy is used to enable save strategy of the network.only_save_strategyz9The sink_mode is used to enable data sink of the network. sink_modezUThe sink_size is used to enable data sink number per step for training or evaluation. sink_sizezCGlobal batch size per GPU/TPU core/CPU for training and evaluation. batch_sizez-Batch size per GPU/TPU core/CPU for training.per_device_train_batch_sizez3Batch size m per GPU/TPU core/CPU for evaluation.per_device_eval_batch_sizez$The initial learning rate for AdamW. learning_ratez(Weight decay for AdamW if we apply some. weight_decayzBeta1 for AdamW optimizer adam_beta1zBeta2 for AdamW optimizer adam_beta2zEpsilon for AdamW optimizer. adam_epsilonzMax gradient norm. max_grad_normz+Total number of training epochs to perform.num_train_epochszThe lr scheduler type to use.lr_scheduler_typezThe optimizer type to use.optimrz Linear warmup over warmup_steps. warmup_stepsz$The checkpoint save strategy to use. save_strategyz Whether to merge and save the split Tensor in the automatic parallel scenario. Integrated save function is only supported in automatic parallel scene, not supportedin manual parallel.integrated_savez&Save checkpoint every X updates steps. save_stepsz(Save checkpoint every X updates seconds. save_secondszxLimit the total amount of checkpoints. Deletes the older checkpoints in the output_dir. Default is unlimited checkpointssave_total_limitz:Random seed that will be set at the beginning of training.seedz*Random seed to be used with data samplers. data_seedzWhether to transform checkpoint according to parallel config. auto_trans_ckptzHWhether to load training context info, such as optimizer and epoch num. resume_trainingz.Whether to eval current model while Training. do_evalzDNum of step intervals between each eval, -1 means no step end eval. eval_step_intervalzKNum of epoch intervals between each eval, 1 means eval on every epoch end. eval_epoch_interval RANK_SIZE1 DEVICE_ID0RANK_IDcCsf|jdurtj|j|_|jdur2t|jj|_|jdurJt|jj|_|j durbt |j j|_ dS)N) rospath expanduserr'r valuer(r r*r selfr r r __post_init__s    zTrainingArguments.__post_init__cCs|jS)zget device num for training.) device_numr@r r rget_device_numsz TrainingArguments.get_device_numcCs|jS)zget device id for training.) device_idr@r r r get_device_idszTrainingArguments.get_device_idcCs|jS)zget rank id for training.)rank_idr@r r r get_rank_idszTrainingArguments.get_rank_id) task_configcCs|durt}||||||||||t|j|j|_t|j|j|_t|j |j |_ t|j |j |_ t|j |j |_ t|j |j |_ t|j |j|_ t|j|j|_t|j|j|_t|j|j|_t|j|j|_t|j|j|_dS)zOconvert training arguments to mindformer config type for adapting hugging-face.N)r_adapt_dataset_config_adapt_runner_config_adapt_lr_schedule_config_adapt_optimizer_config_adapt_save_checkpoint_configrrr/r0rrrload_checkpointr1r2r3r4r5r6rArIr r r"convert_args_to_mindformers_configsB     z4TrainingArguments.convert_args_to_mindformers_configcCs`t|jr t|jj|j|j_n tdt|jrRt|jj|j|j_|j_n tddS)zadapt dataset config.zThis task does not support training at the moment; it does not have a default training data configuration,so the per_device_train_batch_size setting will not take effectzThis task does not support evaluate at the moment;it does not have a default evaluate data configuration,so the per_device_eval_batch_size setting will not take effectN) r train_datasetrrrrwarning eval_datasetrrPr r rrJs    z'TrainingArguments._adapt_dataset_configcCsrt|jrnt|jj|j|j_t|jj|j|j_t|jj|j|j_t|jj|j|j_t|jj|j|j_dS)zadapt runner config.N) r runner_configrepochsr&rrZper_epoch_sizerrPr r rrKs       z&TrainingArguments._adapt_runner_configcCsnt|jrj|jjdur*t|jj|j|j_|jjdurJt|jj|j|j_|jjdurjt|jj|j|j_dS)zadapt lr schedule config.N)r lr_scheduletyperr'r r)rPr r rrLs       z+TrainingArguments._adapt_lr_schedule_configcCst|jr|jjdur*t|jj|j|j_|jjdurJt|jj|j|j_|jjdurjt|jj|j|j_|jjdurt|jj|j|j_|jj durt|jj |j |j_ |jj durt|jj |j |j_ dS)zadapt optimizer config.N) r optimizerrXrr(r r!beta1r"beta2r#epsr$rPr r rrMs2             z)TrainingArguments._adapt_optimizer_configcCs|jdurt|jtr|jdurt|jD]\}}t|tr*|ddkr*|jdkr`|j|q*|jdkrt|j|d|j|j|d<n*|jdkrt|j|d|j |j|d<|j r|j nd |j|d <t|j|d |j |j|d <q*dS) zadapt save checkpoint config.NrXCheckpointMointornostepssave_checkpoint_stepssecondssave_checkpoint_secondskeep_checkpoint_maxr+) callbacksr listr* enumeratedictpoprr,r-r.r+)rArIicallbackr r rrNs2       z/TrainingArguments._adapt_save_checkpoint_config)N)@__name__ __module__ __qualname____doc__rrstr__annotations__rboolrrrrrintrrr floatr!r"r#r$r%r&r'rr r(r r)r*r r+r,r-r.rr/r0r1r2r3r4r5r6r<getenvrCrErGrBpropertyrDrFrHrrQrJrKrLrMrNr r r rr*s        r)ror<typingrr dataclassesrrmindformers.tools.registerrmindformers.toolsrutilsr r r rrrr r r rs  pengcheng.mind-mindformer/mindformers/trainer/__pycache__/utils.cpython-39.pyc000066400000000000000000000400171476650141400301060ustar00rootroot00000000000000a >cfa@sdZddlZddlZddlZddlmZddlmZddlZddl Z ddl m Z m Z ddl mZmZmZddl mZddlmZdd lmZdd lmZmZdd lmZerddlZd ZGd ddeeZ Gddde Z!Gddde Z"Gddde Z#Gddde Z$d>e%dddZd?ddZ&ddZ'dd Z(d!d"Z)d#d$Z*d%d&Z+d'd(Z,d@d)d*Z-d+d,Z.dAd.d/Z/dBd0d1Z0d2d3Z1dCd4d5Z2d6d7Z3dDd8d9Z4dEd:d;Z5d}d|vr|ddkr|jjd urd|vr|d|jj_qqd S) z=resume training, load training info from checkpoint to config-The load_checkpoint must be correct, but get loss_scale epoch_numFWhen distributed loads are sliced weights, sink_mode must be set True.rirrNCheckpointMointorN)rkrlrealpathrexistsrsrmrvr@rDrrGrC callbacksr] scale_senseloss_scale_value)rJZ resume_dictcallbackrrr#load_resume_context_from_checkpoints&   rFcCs~|js2tj|jr"tj|js2td|jtddvrRt |||||d|j rlt |}t ||dt |||ddS)z load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt rw parallel_modesemi_auto_parallel auto_parallelhybrid_parallel)do_eval do_predict)dst_ckpt_strategy)rWN)only_save_strategyrkrlr|rr}rsrget_auto_parallel_context build_modelauto_trans_ckptget_dst_strategytransform_ckpt load_ckpt)rJmodelnetworkrKrWrrrrrrtransform_and_load_checkpoints  rcCstddvr|jjstd|r6|jt|nh|rF||nX|jjdkr||jj dkr|td|jjd|jj dt d|j ||jj|jj d |j rtd n|j rtd d S) z#build model, generate strategy filerrrzzGWhen distributed loads are sliced weights, it does not supportepochs = z > 1 and sink_size = z" = 1,sink_size must be more than 1z .........Building model.........)rOepochrEzCOnly_save_strategy is True, model.compile() finished, system exit! zkonly_save_strategy is True, but stand_alone and data_parallel mode do not have strategy file, system exit! N)rrr@rDrinfer_predict_layoutnextcreate_tuple_iteratorrArEr rIbuildr SystemExit)rJrrKrrrrrr8s*     rc Cshttdd}ttdd}d}|dkr0|Strtj|jd}tj ||j j }tj|tj |}tj ||tdtjtd}|d d krt|||n t||td ttj|d }||D]}td |qtdt||jjdkr|d d kr|tdt|} tj | } tj|| } tj | | td| | td| }nd}n|}ntdtjtd}|d krPt||td ttj|d }||D]}td |qtdt||jjdkrF|d d kr@tdt|} td| }nd}n|d }ntdd|_d}|S)z get strategyrgrh RANK_SIZE1Nrstrategyz%.........Collecting strategy.........rz'.........All strtegy as follow......... *_rank_*.ckptz strategy: %sz(.........Collecting %d strategy.........z".........Merging strategy.........z Save %s to %sz!.........Merging succeed.........z.Can't collecting all strategy, device num > 8!F)rGrkrpr rlrnremote_save_urlmoxfile make_dirsparallelstrategy_ckpt_save_filebasenamecopyr rIr wait_collect_all_strategyrsortlenparallel_configpipeline_stagerrFr) rJrank_id world_sizeZdst_strategy_pathZ obs_save_dirlocal_strategy_pathobs_strategy_pathZlocal_strategy_dirlocal_strategy_pathsZmerged_strategy_pathZmerged_strategy_nameZobs_merged_strategy_pathrrrrTsp                rc Cs|jr tnd}|jrtnd}tjtd}tj|dd|ddkr t dt |j }t d|t d |j t d |t d |t |j |d ||t d tj|d|d}t|d}|tr tj|jd}tj|tj|d|d} tj|| t|||trtj|jd}tj|d|} tjtdd|} tj| | td|d| d| ||_ dS)zauto transform ckptrrtransformed_checkpointT)exist_okrz#.........Transforming ckpt.........zSrc ckpt strategy: %sz Src ckpt: %szDst ckpt strategy: %sz Dst ckpt: %sZ checkpoint_z$.........Transform succeed!.........transform_succeed_rank_.txtwrank_Rank z: Save z to N) use_parallelrrrkrlrnr makedirsr rIrsrc_strategy_path_or_dirrmstransform_checkpointsopencloser rrrrrwait_transform copy_parallelprint) rJrrrtransformed_ckpt_dirZsrc_ckpt_strategytransform_succeed_txtftransformed_ckpt_dir_obsZtransform_succeed_txt_obsZrank_transformed_ckpt_dir_obsZrank_transformed_ckpt_dirrrrrsD         rc s@td|dd}trtj|jdfddtd|dD}t|}d}t}|sd}|D]} t j | r~|d 7}qdqqd||krd }qXt|} | t krt d d }qtd qXn^tjtd} tj| d } t}tj | s"t|} | t krd }q"td qtd|d|rsz"wait_transform..rrrTz(Automatic weight transform is timed out!ztransform_succeed_rank_0.txtz: Transform succeed!)rr rkrlrnrrangertimerrr} ELAPSE_TIMEr rFsleepr ) rJrrtime_outZtransform_succeed_txts_obs total_numZall_node_transform_succeed start_timeZnode_transform_succeed_numr elapsed_timerrrrrsJ         rc Cst}d}|rltjtj|d}t||kr>tdq |D](}tj|tj |}tj ||qBttj|d}t||krqq t|} | t krt dd}qtdq |rdSdS)zwait all strategy collect overFz*.ckptrrz%Collection of all strategy timed out!T)rrrrrkrlrnrrrrrr rF) strategy_dirrobs_strategy_dirrrZobs_strategy_pathsrrrrrrrrs,      rcCshtdtj|jr"t|}n t|j}t||}tdt||rdt||}tdt|dS)zload checkpointzD.............Start load checkpoint from checkpoint..................z%Network parameters are not loaded: %sz'Optimizer parameters are not loaded: %sN) r rIrkrlrmrrvrstr)rJrrWruZnot_load_network_paramsZnot_load_optim_paramsrrrrs     rcs\tjstdddtD}|s6dSt|fddd}tj|dS) z-get last checkpoint for resuming or finetune.z is not a real directory,When distributed loads are sliced weights,load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*},The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckptcSsg|]}|dr|qS)z.ckpt)endswith)r checkpointrrrr,s z'get_last_checkpoint..Ncstjtj|S)N)rkrlgetmtimern)xrtrr3z%get_last_checkpoint..)rfr>)rkrlrmNotADirectoryErrorlistdirsortedrn)rtZoutput_checkpoint_pathrrrrq$s  rq)r)r)N)NFF)FF)N)N)N)7rrkrr6renumrnumpyr7 mindsporer"mindspore.communication.managementrrrrrr r8mindformers.tools.loggerr mindformers.tools.registerr mindformers.tools.utilsr r Z mindformers.tools.transform_ckptrmoxingrrrrrr#r*r2rGr=rLrSrUr\r_rcrvrrrrrrrrrqrrrrsP              N )+  pengcheng.mind-mindformer/mindformers/trainer/base_trainer.py000066400000000000000000001266711476650141400251300ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Base Trainer.""" import math import os import shutil import time from pprint import pprint from functools import partial from typing import Optional, Union, List import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor, DatasetHelper from mindspore.train.model import Model from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.common import initializer as init from mindformers.mindformer_book import MindFormerBook from mindformers.core import build_lr, build_optim, build_callback, build_metric from mindformers.core.callback.callback import EvalCallBack from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, build_processor, build_tokenizer, \ BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.pipeline import pipeline from mindformers.wrapper import build_wrapper from mindformers.tools.register import MindFormerConfig from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.auto_class import AutoModel from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_runner_config, transform_and_load_checkpoint, load_resume_context_from_checkpoint, load_merged_checkpoint from .optimizer_grouped_parameters import get_optimizer_grouped_parameters, get_mup_optimizer_grouped_parameters from .utils import set_seed, check_train_data_loader_type, \ check_eval_data_loader_type, check_optimizer_and_lr_type, check_wrapper_config SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CONFIG_DIR = 'configs' def download_ckpt_from_obs(remote_url, local_url, rank): """ Download the dataset from the obs. src_data_url (Str): should be the dataset path in the obs tgt_data_path (Str): the local dataset path rank (Int): the current rank id """ import moxing as mox # print(f"begin download ckpt from {obs_ckpt_url}") # if not os.path.exists(os.path.join(cache_url, f"rank_{rank}")): # os.makedirs(os.path.join(cache_url, f"rank_{rank}"), exist_ok=True) # if not os.path.exists(cache_url): # os.makedirs(cache_url, exist_ok=True) time.sleep(rank % 4) try: mox.file.copy(src_url=remote_url, dst_url=local_url) except Exception as error: print(f"Error: {error}, download again...") time.sleep(10) mox.file.copy(src_url=remote_url, dst_url=local_url) print(f"Rank {rank} download ckpt {remote_url} succeed!", flush=True) class BaseTrainer: r"""Base Task Trainer. Args: task (str): The task name supported. model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, task: str = None, model_name: str = None): if model_name is None: model_name = "model name unspecified." if task is None: task = "task name unspecified." logger.info("Now Running Task is: %s, Model is: %s", task, model_name) self.model_name = model_name self.task = task self.config = None self.default_task_config = None self.train_dataset = None self.eval_dataset = None self.network = None self.optimizer = None self.image_processor = None self.audio_processor = None self.tokenizer = None self.callbacks = None self.eval_callbacks = None self.model_wrapper = None self.compute_metrics = None self.kwargs = None self.pipeline_task = None if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 to avoid dirs_exist_ok=False shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if task not in SUPPORT_TASKS.keys(): logger.warning("Input task name is not in the supported list or unspecified.") if task in SUPPORT_TASKS.keys() and model_name not in SUPPORT_TASKS.get(task).keys(): logger.warning("Input model name is not in the supported list or unspecified.") logger.warning("See the list of supported task and model name: %s", SUPPORT_TASKS) logger.warning("The default model config: %s will now be used for the %s task ", SUPPORT_TASKS.get(self.task).get("common"), task) self.model_name = "common" def set_config(self, config: Optional[Union[dict, str, ConfigArguments, TrainingArguments]] = None, is_full_config: bool = False): """Set the task config for task trainer.""" if config is not None: if is_full_config: self.config = config else: self.setup_task_config() self.config = self._merge_config(config) else: self.setup_task_config() self.config = self.default_task_config build_parallel_config(self.config) self._check_global_batch_size_for_auto_parallel() return self.config def setup_task_config(self): """Setup the default task config.""" task_config = None if self.task in SUPPORT_TASKS.keys() and self.model_name in SUPPORT_TASKS.get(self.task).keys(): task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if isinstance(task_config, MindFormerConfig): self.default_task_config = task_config else: logger.warning("If the default config arguments is not specified," "you must define the required model or wrapper, optimizer, and so on" "in the train or evaluate or predict attribute function.") def _check_global_batch_size_for_auto_parallel(self): """Check global batch size in auto parallel mode.""" batch_size = self.config.runner_config.batch_size dp = self.config.parallel_config.data_parallel micro_batch_num = self.config.parallel_config.micro_batch_num micro_batch_interleave_num = self.config.micro_batch_interleave_num parallel_mode = ms.get_auto_parallel_context("parallel_mode") full_batch = ms.get_auto_parallel_context("full_batch") pp = self.get_pipeline_stages() if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: if full_batch: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is True, " "global batch size will be changed: " "global_batch_size = " "batch_size * data_parallel * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s * %s = %s).", pp, batch_size, dp, micro_batch_num, micro_batch_interleave_num, batch_size * dp * micro_batch_num * micro_batch_interleave_num) self.config.runner_config.batch_size = \ batch_size * dp * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, full batch is True," "so global batch size will be changed: " "global_batch_size = batch_size * data_parallel * micro_batch_interleave_num " "= %s * %s * %s = %s", parallel_mode, batch_size, dp, micro_batch_interleave_num, batch_size * dp * micro_batch_interleave_num) self.config.runner_config.batch_size = batch_size * dp * micro_batch_interleave_num else: if pp > 1: logger.info("Pipeline parallel was opened: pipeline_stages = %s, full batch is False, " "batch size per card will be changed: " "per_batch_size = batch_size * micro_batch_num * micro_batch_interleave_num " "= %s * %s * %s = %s).", pp, batch_size, micro_batch_num, micro_batch_interleave_num, batch_size * micro_batch_num * micro_batch_interleave_num) logger.info( "global_batch_size = per_batch_size * device_num " "= %s * %s = %s", batch_size * micro_batch_num * micro_batch_interleave_num, int(os.getenv('RANK_SIZE', '1')), batch_size * micro_batch_num * micro_batch_interleave_num * int(os.getenv('RANK_SIZE', '1'))) self.config.runner_config.batch_size = \ batch_size * micro_batch_num * micro_batch_interleave_num self._reset_wrapper_for_pipeline_parallel() else: logger.info("The current parallel mode is %s, batch size per card will not be changed: " "batch_size_per_card = %s", parallel_mode, batch_size) logger.info( "global_batch_size = batch_size_per_card * device_num " "= %s * %s = %s", batch_size, int(os.getenv('RANK_SIZE', '1')), batch_size * int(os.getenv('RANK_SIZE', '1'))) self.config.parallel_config.data_parallel = 1 self.config.parallel_config.model_parallel = 1 self.config.parallel_config.pipeline_stage = 1 self.config.parallel_config.optimizer_shard = False self.config.parallel_config.micro_batch_num = 1 logger.info("parallel_config will be change to default config: %s.", self.config.parallel_config) def _reset_wrapper_for_pipeline_parallel(self): """Reset wrapper when pipeline parallel.""" if self.config.runner_wrapper is not None: self.config.runner_wrapper.type = "MFPipelineWithLossScaleCell" \ if self.config.runner_wrapper.type != "MFPipelineWithLossScaleCell" else self.config.runner_wrapper.type self.config.runner_wrapper.micro_batch_num = self.config.parallel_config.micro_batch_num logger.warning( "When using the pipeline parallel mode, " "the MFPipelineWithLossScaleCell class is used by default.") else: logger.info( "When using the pipeline parallel mode, " "because the wrapper class is not specified, " "MindSpore's built-in PipelineCell is used by default") logger.info("PipelineWrapper under evaluate or predict mode will not take effect.") def _reset_dataset_batch_size(self): """Reset dataset batch size according to the global batch size of runner config.""" check_dataset_config(self.config) def _merge_config(self, args): """Merge config from default task config.""" if self.default_task_config is None: logger.warning("default task config is None, you will not be able to merge config parameters.") return args if isinstance(args, dict): self.default_task_config.merge_from_dict(args) elif isinstance(args, str): if not (os.path.realpath(args) and os.path.exists(args)): raise FileNotFoundError(f"config path must be exist, but get {args}.") if not args.endswith(('.yaml', '.yml')): raise ValueError(f"config file must be end with .yaml or .yml, but get {args}") self.default_task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, self.default_task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, self.default_task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, self.default_task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, self.default_task_config) self.default_task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): args.convert_args_to_mindformers_config(self.default_task_config) else: logger.warning( "The type of config parameter to merge is not supported, " "currently supported types are [dict, config_path(str), ConfigArguments, TrainingArguments], " "but get %s", type(args)) return self.default_task_config def create_train_dataset(self, default_args: dict = None): """Create the train dataset for training.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() train_dataset = build_dataset(self.config.train_dataset_task, default_args=default_args) return train_dataset def create_eval_dataset(self, default_args: dict = None): """Create the eval dataset for evaluate.""" logger.info(".........Build Dataset From Config..........") self._reset_dataset_batch_size() # reduce batch size on eval mode, for that micro_batch will not take effect on eval. parallel_mode = ms.get_auto_parallel_context("parallel_mode") if parallel_mode in ["semi_auto_parallel", "auto_parallel"]: self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.parallel_config.micro_batch_num self.config.eval_dataset.batch_size = \ self.config.eval_dataset.batch_size // self.config.micro_batch_interleave_num eval_dataset = build_dataset(self.config.eval_dataset_task, default_args=default_args) return eval_dataset def create_network(self, default_args: dict = None): """Create the network for task trainer.""" logger.info(".........Build Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(network, micro_batch_interleave_num) return network, eval_network def create_pipeline_network(self, default_args: dict = None): """Create the network of pipeline parallel for task trainer.""" logger.info(".........Build Pipeline Network From Config..........") eval_network = build_model(self.config.model, default_args=default_args) network = eval_network micro_batch_interleave_num = self.config.micro_batch_interleave_num micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(network, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(network, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) return network, eval_network def create_image_processor(self, default_args: dict = None): """Create the image processor for predict.""" logger.info(".........Build Image Processor From Config..........") self.image_processor = build_processor( self.config.processor.image_processor, default_args=default_args) return self.image_processor def create_tokenizer(self, default_args: dict = None): """Create the tokenizer for task trainer.""" logger.info(".........Build Text Tokenizer From Config..........") self.tokenizer = build_tokenizer( self.config.processor.tokenizer, default_args=default_args) return self.tokenizer def create_optimizer_scheduler(self, network, layer_scale=False): """Create the optimizer for training.""" logger.info(".........Build Optimizer From Config..........") # learning rate scale for multi-nodes training learning_scale = self.config.lr_scale scale_factor = self.config.lr_scale_factor # build learning rate schedule lr_schedule = self.create_lr_scheduler(learning_scale, scale_factor) weight_decay = self.config.optimizer.weight_decay if self.config.optimizer.weight_decay else 0. layer_decay = self.config.layer_decay if self.config.layer_decay else 1.0 if self.config.model.model_config.mup_width_base is not None: ratio = self.config.model.model_config.hidden_size / self.config.model.model_config.mup_width_base group_params = get_mup_optimizer_grouped_parameters(network, weight_decay, ratio, self.config.lr_schedule) else: group_params = get_optimizer_grouped_parameters(network, weight_decay, lr_schedule, layer_scale=layer_scale, layer_decay=layer_decay) if lr_schedule is not None: self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: if self.config.optimizer.learning_rate is None: raise ValueError("learning_rate must be input") self.config.optimizer.learning_rate = self.learning_rate_scale( self.config.optimizer.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.optimizer.learning_rate self.optimizer = build_optim( self.config.optimizer, default_args={"params": group_params}) return self.optimizer def create_lr_scheduler(self, learning_scale: bool = False, scale_factor: int = 256): """Create the learning rate scheduler.""" logger.info(".........Build LR Schedule From Config..........") train_data_size = self.get_train_data_size() if self.config.lr_schedule: warmup_epochs = self.config.lr_schedule.pop("warmup_epochs", None) warmup_ratio = self.config.lr_schedule.pop("warmup_ratio", None) # WSD decay_ratio = self.config.lr_schedule.pop("decay_ratio", None) if not self.config.runner_config.sink_mode: total_steps = int(self.config.runner_config.epochs * train_data_size) else: total_steps = int(self.config.runner_config.epochs * self.config.runner_config.sink_size) if warmup_epochs is not None and warmup_ratio is not None: logger.warning("warmup_epochs and warmup_ratio are set simultaneously," "warmup_ratio takes precedence.") warmup_epochs = None if warmup_epochs is not None: logger.warning("warmup_epochs was set in lr_schedule," "it will multiply the data size to represent the warmup steps") self.config.lr_schedule.warmup_steps = int(warmup_epochs * train_data_size) self.config.lr_schedule.total_steps = total_steps \ if self.config.lr_schedule.total_steps is None or self.config.lr_schedule.total_steps == -1 \ else int(self.config.lr_schedule.total_steps) if warmup_ratio is not None: self.config.lr_schedule.warmup_steps = int(self.config.lr_schedule.total_steps * warmup_ratio) if decay_ratio is not None: #WSD self.config.lr_schedule.decay_last_steps = int(self.config.lr_schedule.total_steps * decay_ratio) self.config.lr_schedule.learning_rate = self.learning_rate_scale( self.config.lr_schedule.learning_rate, scale_factor) \ if learning_scale and scale_factor is not None else self.config.lr_schedule.learning_rate lr_schedule = build_lr(self.config.lr_schedule) return lr_schedule def create_model_wrapper(self, network, optimizer): """Create the model wrapper for training.""" logger.info(".........Build Model Wrapper for Train From Config..........") model_wrapper = build_wrapper(self.config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "parallel_config": self.config.parallel_config}) return model_wrapper def create_callbacks(self, default_args: dict = None): """Create the callback list for training.""" logger.info(".........Build Callbacks for Train From Config..........") self.callbacks = [] if self.config.profile: self.callbacks.append(self.config.profile_cb) self.callbacks.extend(build_callback(self.config.callbacks, default_args=default_args)) return self.callbacks def create_eval_callbacks(self, default_args: dict = None): """Create the eval callback list for training.""" logger.info(".........Build Callbacks for Evaluate From Config..........") self.eval_callbacks = [] self.eval_callbacks.extend(build_callback(self.config.eval_callbacks, default_args=default_args)) return self.eval_callbacks def create_metrics(self, metric_name: str = None): """Create Metrics For Evaluate or Fit.""" if metric_name is None: metric_name = self.model_name + "_metric" self.compute_metrics = {metric_name: build_metric(self.config.metric)} return self.compute_metrics def count_parameters(self): """Count network parameters number.""" if self.network is not None: logger.info("Network Parameters: %s M.", str(count_params(self.network))) elif self.model_wrapper is not None: logger.info("Network Parameters: %s M.", str(count_params(self.model_wrapper.network))) else: logger.warning("Network is None, parameters incalculable.") def set_seed(self, seed: int = None): """Set seed for training.""" if seed is None: if self.config.seed is None: raise ValueError("seed is not set in config, it is None.") set_seed(self.config.seed) else: set_seed(seed) def set_train_dataset(self, dataset): """Set the attribute of train dataset.""" if dataset is None: raise ValueError("Train dataset is None") self.train_dataset = dataset def set_eval_dataset(self, dataset): """Set the attribute of eval dataset .""" if dataset is None: raise ValueError("Eval dataset is None") self.eval_dataset = dataset def set_network(self, network, is_train: bool = True): """Set the attribute of network.""" if network is None: raise ValueError("network is None") if isinstance(network, (Cell, BaseModel)): network.set_train(is_train) self.network = network def set_model_wrapper(self, model_wrapper): """Set the attribute of model_wrapper.""" if model_wrapper is None: raise ValueError("model wrapper is None") self.model_wrapper = model_wrapper def get_train_data_size(self): """Get train dataset size.""" if self.train_dataset is None: raise NotImplementedError("train dataset is None") return self.train_dataset.get_dataset_size() def get_eval_data_size(self): """Get eval dataset size.""" if self.eval_dataset is None: raise NotImplementedError("train dataset is None") return self.eval_dataset.get_dataset_size() def get_pipeline_stages(self): """Get pipeline stages for task trainer.""" pipeline_stages = ms.get_auto_parallel_context("pipeline_stages") return pipeline_stages def learning_rate_scale(self, base_learning_rate: float = 0., scale_factor: Optional[Union[float, int]] = 256.): """Scale learning rate for training.""" if not isinstance(base_learning_rate, float): raise ValueError(f"learning rate must be float type, but get {type(base_learning_rate)}") if not isinstance(scale_factor, (float, int)): raise ValueError(f"scale_factor must be float or int type, but get {type(scale_factor)}") device_num = int(os.getenv("RANK_SIZE", "1")) per_device_batch_size = self.config.train_dataset.batch_size learning_rate = (base_learning_rate * device_num * per_device_batch_size) / scale_factor return learning_rate def training_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """Train or Fine-tune for BaseTrainer in MindFormers.""" self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if config.resume_training and config.load_checkpoint: logger.info(".............Start load resume context from checkpoint..................") rank = int(os.getenv("RANK_ID", "0")) if config.resume_merged_ckpt: logger.info(".............Start load merged checkpoint..................") obs_ckpt_url = os.path.join(config.load_checkpoint, config.restore_ckpt_name_prefix) local_url = f"/cache/checkpoint/{config.restore_ckpt_name_prefix}" config.load_checkpoint = local_url if rank%8==0: download_ckpt_from_obs(obs_ckpt_url, local_url, rank) with open('/cache/checkpoint/download_successed.txt', 'w')as fl: fl.write("") else: while not os.path.isfile('/cache/checkpoint/download_successed.txt'): time.sleep(3) else: download_rank = rank%config.parallel.optimizer_weight_shard_size if config.reuse_ckpt else rank #download_rank = rank % 256 obs_ckpt_url = os.path.join(config.load_checkpoint, f"rank_{download_rank}", f"{config.restore_ckpt_name_prefix.replace('*', str(download_rank))}") local_url = f"/cache/checkpoint/rank_{rank}.ckpt" download_ckpt_from_obs(obs_ckpt_url, local_url, rank) config.load_checkpoint = local_url load_resume_context_from_checkpoint(config) # build dataset logger.info(".........Build Dataset For Train..........") if dataset is None: dataset = self.create_train_dataset() self.set_train_dataset(dataset) check_runner_config(config, dataset) config.runner_config.epochs += config.runner_config.initial_epoch if config.runner_config.sink_mode: epoch_num = math.ceil((config.runner_config.epochs - config.runner_config.initial_epoch) * config.runner_config.sink_size / dataset.get_dataset_size()) # pylint: disable=W0212 dataset._dataset_helper = DatasetHelper(dataset, config.runner_config.sink_mode, config.runner_config.sink_size, epoch_num) # build network logger.info(".........Build Net For Train..........") eval_network = None if network is None and wrapper is None and \ self.model_wrapper is None and self.network is None: # If neither network nor wrapper exists, create a network if self.get_pipeline_stages() > 1: network, eval_network = self.create_pipeline_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) else: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and wrapper is None and self.network is not None: logger.info(".........Using The Existing Network For Train:: %s", self.network.__class__.__name__) network = self.network if network is not None: self.set_network(network, is_train=True) if eval_network is None: eval_network = network if wrapper is not None: self.set_model_wrapper(wrapper) self.count_parameters() # build optimizer logger.info(".........Build Optimizer For Train..........") if optimizer is None and wrapper is None and self.model_wrapper is None: optimizer = self.create_optimizer_scheduler(network, layer_scale=config.layer_scale) # build model wrapper if wrapper is None and self.model_wrapper is None: logger.info(".........Build Running Wrapper From Config For Train..........") wrapper = self.create_model_wrapper(network, optimizer) elif wrapper is None and self.model_wrapper is not None: logger.info(".........Using The Existing Model Wrapper: %s", self.model_wrapper.__class__.__name__) wrapper = self.model_wrapper # build callback logger.info(".........Build Callbacks For Train..........") if callbacks is None: callbacks = self.create_callbacks(default_args={ "learning_rate": optimizer.learning_rate if optimizer else wrapper.optimizer.learning_rate, "origin_epochs": config.runner_config.origin_epochs, "dataset_size": config.data_size, "micro_batch_interleave_num": config.micro_batch_interleave_num, "micro_batch_num": config.parallel_config.micro_batch_num, "initial_epoch": config.runner_config.initial_epoch, "global_batch_size": self.config.runner_config.batch_size, "device_num": int(os.getenv('RANK_SIZE', '1'))}) # define compute metrics for evaluate in training compute_metrics = None if config.do_eval: compute_metrics = self.create_metrics() # define Model and begin training logger.info(".........Starting Init Train Model..........") if wrapper is not None: model = Model(wrapper, metrics=compute_metrics, eval_network=eval_network) else: model = Model(network, optimizer=optimizer, metrics=compute_metrics, eval_network=eval_network) # resume checkpoint if config.load_checkpoint or config.only_save_strategy: if config.resume_training: logger.info(".............Start resume training from checkpoint..................") if config.resume_merged_ckpt: load_merged_checkpoint(config, model, network, dataset, optimizer=optimizer) else: transform_and_load_checkpoint(config, model, network, dataset, optimizer=optimizer) else: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path transform_and_load_checkpoint(config, model, network, dataset) # build evaluate in training if config.do_eval: logger.info(".........Build Evaluate in Training Callback..........") eval_dataset = kwargs.get('eval_dataset', None) if eval_dataset is None: eval_dataset = self.create_eval_dataset() eval_callback = EvalCallBack( partial( self._evaluate_in_training, model=model, eval_dataset=eval_dataset, ), step_interval=config.eval_step_interval if config.eval_step_interval else 100, epoch_interval=config.eval_epoch_interval if config.eval_epoch_interval else -1, ) callbacks.append(eval_callback) logger.info(".........Starting Training Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) logger.info(".........Model Compiling, Please Wait a Moment...........") model.train(config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size, initial_epoch=config.runner_config.initial_epoch) logger.info(".........Training Over!.............") def evaluate_process( self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): """Evaluate for BaseTrainer in MindFormers.""" metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) logger.info("Create evaluate dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network eval_network = None if network is None and self.network is None: network, eval_network = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) elif network is None and self.network is not None: logger.info(".........Using The Existing Network For Evaluate: %s", self.network.__class__.__name__) network = self.network self.set_network(network, is_train=False) if eval_network is None: eval_network = network self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = self.create_metrics(metric_name=metric_name) # build callback logger.info(".........Build Callbacks For Evaluate..........") if callbacks is None: callbacks = self.create_eval_callbacks() logger.info(".........Starting Init Evaluate Model..........") model = Model(network, metrics=compute_metrics, eval_network=eval_network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info(".........Starting Evaluate Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output = model.eval(dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode) logger.info('%s = %s', metric_name, str(output)) logger.info(".........Evaluate Over!.............") def predict_process(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, task: str = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseImageProcessor] = None, **kwargs): """Predict for BaseTrainer in MindFormers.""" if not self.pipeline_task: is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if task not in ["translation", "text_generation"]: raise SystemExit("Currently distributed predict only support translation and text_generation. " "Process exit!") if config.parallel_config.pipeline_stage > 1: raise SystemExit("Currently distributed predict dose not support pipeline parallel. " "Process exit!") # build network if network is None: network, _ = self.create_network( default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) network.model_name = kwargs.get("model_name", None) self.count_parameters() if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) if image_processor is None and config.processor.image_processor: image_processor = build_processor(config.processor.image_processor) if audio_processor is None and config.processor.audio_processor: audio_processor = build_processor(config.processor.audio_processor) model = Model(network) if config.load_checkpoint or config.only_save_strategy: if ms.context.get_auto_parallel_context('parallel_mode') in \ ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: seq_length = config.model.model_config.seq_length infer_data = Tensor(shape=(1, seq_length), dtype=ms.int32, init=init.One()) transform_and_load_checkpoint(config, model, network, infer_data, do_predict=True) else: transform_and_load_checkpoint(config, model, network, None, do_predict=True) self.pipeline_task = pipeline( task=task, model=network, tokenizer=tokenizer, image_processor=image_processor, audio_processor=audio_processor, **kwargs ) top_k = kwargs.pop("top_k", None) if top_k is None: if config.top_k is not None: top_k = config.top_k elif config.model and config.model.model_config.top_k: top_k = config.model.model_config.top_k else: top_k = 1 save_file = kwargs.pop("save_file", None) if save_file is None: if config.save_file is not None: save_file = config.save_file else: save_file = f"{task}_result.txt" if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) output_results = self.pipeline_task(input_data, top_k=top_k) output_info = [] for one_output in output_results: if isinstance(one_output, dict) and "info" in one_output: output_info.append(one_output["info"]) else: output_info.append(one_output) with open(save_file, 'w') as file: file.write(str(output_info)) file.close() logger.info("output result is: %s", str(output_info)) logger.info("output result is saved at: %s", save_file) logger.info(".........Predict Over!.............") return output_results def _evaluate_in_training(self, model: Model, eval_dataset: BaseDataset): origin_phase = model.eval_network.phase model.eval_network.set_train(False) output = model.eval( eval_dataset, dataset_sink_mode=self.config.runner_config.sink_mode ) model.eval_network.set_train(origin_phase) return output pengcheng.mind-mindformer/mindformers/trainer/build_trainer.py000066400000000000000000000105141476650141400253010ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Trainer API.""" from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from . import ImageClassificationTrainer, ZeroShotImageClassificationTrainer, \ MaskedImageModelingTrainer, MaskedLanguageModelingTrainer, ImageToTextRetrievalTrainer, \ TranslationTrainer, TokenClassificationTrainer, TextClassificationTrainer, \ ContrastiveLanguageImagePretrainTrainer, QuestionAnsweringTrainer, GeneralTaskTrainer def build_trainer( config: dict = None, default_args: dict = None, module_type: str = 'trainer', class_name: str = None, **kwargs): r"""Build trainer API. Instantiate the task trainer from MindFormerRegister's registry. Args: config (dict): The task trainer's config. Default: None. default_args (dict): The default argument of trainer API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'trainer'. class_name (str): The class name of task trainer API. Default: None. Return: The function instance of task trainer API. Examples: >>> from mindformers import build_trainer >>> trainer_config = {'type': 'image_classification', 'model_name': 'vit'} >>> # 1) use config dict to build trainer >>> cls_trainer_config = build_trainer(trainer_config) >>> # 2) use class name to build trainer >>> cls_trainer_class_name = build_trainer(class_name='image_classification', model_name='vit') """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.TRAINER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_mf_trainer(): """ register MindFomrers builtin LR class. """ # adapt huggingface MindFormerRegister.register_cls( ImageClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="image_classification") MindFormerRegister.register_cls( ZeroShotImageClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="zero_shot_image_classification") MindFormerRegister.register_cls( MaskedImageModelingTrainer, module_type=MindFormerModuleType.TRAINER, alias="masked_image_modeling") MindFormerRegister.register_cls( MaskedLanguageModelingTrainer, module_type=MindFormerModuleType.TRAINER, alias="fill_mask") MindFormerRegister.register_cls( TokenClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="token_classification") MindFormerRegister.register_cls( TextClassificationTrainer, module_type=MindFormerModuleType.TRAINER, alias="text_classification") MindFormerRegister.register_cls( ContrastiveLanguageImagePretrainTrainer, module_type=MindFormerModuleType.TRAINER, alias="contrastive_language_image_pretrain") MindFormerRegister.register_cls( TranslationTrainer, module_type=MindFormerModuleType.TRAINER, alias="translation") MindFormerRegister.register_cls( QuestionAnsweringTrainer, module_type=MindFormerModuleType.TRAINER, alias="question_answering") MindFormerRegister.register_cls( ImageToTextRetrievalTrainer, module_type=MindFormerModuleType.TRAINER, alias="image_to_text_retrieval") MindFormerRegister.register_cls( GeneralTaskTrainer, module_type=MindFormerModuleType.TRAINER, alias="general") register_mf_trainer() pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/000077500000000000000000000000001476650141400270745ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/__init__.py000066400000000000000000000014651476650141400312130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .causal_language_modeling import CausalLanguageModelingTrainer __all__ = ['CausalLanguageModelingTrainer'] pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/__pycache__/000077500000000000000000000000001476650141400313045ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005211476650141400354130ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/__pycache__a ?cf5@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)CausalLanguageModelingTrainerrN)__doc__causal_language_modelingr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/causal_language_modeling/__init__.pys causal_language_modeling.cpython-39.pyc000066400000000000000000000264721476650141400406620ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/__pycache__a ?cfl<@sBdZddlZddlZddlZddlmZmZmZddlmZddl Z ddl m Z ddl mZddlmZmZmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddgZ/e&0Z1e2ej3Gddde,Z4dS)zCausal Image Modeling Trainer.N)OptionalListUnion)pprint)Model)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizerbuild_tokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) build_metric) AutoModel)MindFormerBook)ConfigArguments)TrainingArguments) BaseTrainer)transform_and_load_checkpoint ADGENMetric EmF1Metricc s,eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zdd d Zdeeee e e feeeeefeee e feedddZZS)CausalLanguageModelingTraineraCausalLanguageModelingTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import CausalLanguageModelingTrainer >>> gen_trainer = CausalLanguageModelingTrainer(model_name="gpt2") >>> gen_trainer.train() >>> res = gen_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntext_generation)superr__init__)selfr __class__p/home/ma-user/work/mindformers-baichuan/mindformers/trainer/causal_language_modeling/causal_language_modeling.pyr";sz&CausalLanguageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for CausalLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r(r)r-r*r+r,N)training_process)r#r(r)r*r+r,r-kwargsr&r&r'train>s%z#CausalLanguageModelingTrainer.trainr(r)r*r-compute_metricscKsX|jj}|d||tvr8|j|f|||d|n|jf|||||d|dS)aEvaluate task for CausalLanguageModeling Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. metric_name)r)r*r2r1N)metrictype setdefaultGENERATE_METRIC_NAMESgenerate_evaluateevaluate_process)r#r(r)r*r-r2r/r3r&r&r'evaluatels* z&CausalLanguageModelingTrainer.evaluatec% Ks|d}|dd}|||}t|jjj} |dkr\| r\|jjdkr\t d|d|j_t d|durv| }| ||dur|j |j |jd d \} }|j|dd |t d |durt|j}|t d |dur|jjrt|jj}t dt||d} |js,|jrd|jtvrHt|jj|_t dt|| ||ddt dtt !ddddkrt"||jjj#} |jjj$} |jjj%}|jjj&}d}d}|j'}|(}t)|*D]\}}|d+}|d+}g}t,|j-dD](}|.t/0t/1|||kdqt/2|}| r\|d|j3jjj}t44}| j5j6|| || |d}g}t,|j-dD]$}|.||t||dqt44}|||j-d}d}t,t7|D]} ||| j-d7}q|dkr||7}|||7}||d}!||d|!}"t d|dd|d||dd d!|dd"d#|||dd$d%||dd&t8j9t|"d'|j:|dd(}#|j:|dd(}$|;|#|$q|<t d)dS)*zYEvaluate the text generate task. Return metrics with Rouge-1, Rouge-2, Rouge-l and BLEU. r3is_full_configFrzSFor metric %s, it only supports batch size equals 1, so reset batch size to 1 here.z-.........Build Dataset For Evaluate..........N)parallel_config moe_config) default_args)is_trainz5.........Build Compute Metrics For Evaluate..........z/.........Build tokenizer For Evaluate..........z/.........Starting Init Evaluate Model..........) eval_networkz=.............Start load checkpoint for eval..................T)do_evalz*.........Starting Evaluate Model..........RANK_ID0rg-C6? input_idslabels) do_sample max_lengthtop_ptop_kzStep[/z ], cost time z.4fzs, zevery example cost time is z, zgenerate speed: z tokens/s, z avg speed: z tokens/s, remaining time: )seconds)skip_special_tokensz(...........Evaluate Over!...............)=get set_configboolmodel model_configmax_new_tokens runner_config batch_sizerinfocreate_eval_datasetset_eval_datasetcreate_networkr=r> set_networkcount_parametersrr4clear processor tokenizerrrload_checkpointonly_save_strategySUPPORT_MODEL_NAMESrfrom_pretrained default_checkpoint_download_pathrintosgetenvrrHrJrKmax_decode_length pad_token_idget_dataset_size enumeratecreate_dict_iteratorasnumpyrangeshapeappendnpmaxargwherearrayr(timepredict_networkgeneratelendatetime timedeltadecodeupdateeval)%r#r(r)r*r2r_r/r3r;Zenable_max_new_tokens_rRrHrJrKrIZtotal_tokens_num total_timeriZ len_datasetiinputsrFrGvalid_length_each_examplej start_timeoutputs output_idsend_timeZ avg_cost_timeZ tokens_num batch_indexZavg_timeZ remain_timeZpres_str labels_strr&r&r'r8s                     &  "   $ z/CausalLanguageModelingTrainer.generate_evaluate)r( input_datar)r_c Ks|dur|j}t|tttfs0tdt|t|trtj |rt |d(}g}|D]}| |qZWdn1s~0Y|}|j f||d||d|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. NzMInput data's type must be one of [str, list, GeneratorDataset], but got type rr )r(rtaskr)r_) r isinstancestrlistr ValueErrorr5rfpathisfileopenextendpredict_process) r#r(rr)r_r/fpinput_data_listliner&r&r'predicts( *z%CausalLanguageModelingTrainer.predict)N)NNNNNN)NNNNN)NNNN)NNNN)__name__ __module__ __qualname____doc__rr"rrdictrrrr r r r rr rrr0setr:r8rrr __classcell__r&r&r$r'r-sV  / 4 wr)5rrfrurytypingrrrrnumpyrq mindsporerZmindspore.trainr mindspore.nnrr r mindspore.datasetr mindformers.datasetr mindformers.modelsr rrmindformers.tools.loggerrmindformers.tools.registerrrrmindformers.corermindformers.auto_classrmindformers.mindformer_bookr config_argsr training_argsr base_trainerrutilsrr7get_model_name_support_listrbregisterTRAINERrr&r&r&r's2               pengcheng.mind-mindformer/mindformers/trainer/causal_language_modeling/causal_language_modeling.py000066400000000000000000000361541476650141400344500ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Causal Image Modeling Trainer.""" import os import time import datetime from typing import Optional, List, Union from pprint import pprint import numpy as np from mindspore import Model from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer, build_tokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from mindformers.core import build_metric from mindformers.auto_class import AutoModel from mindformers.mindformer_book import MindFormerBook from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer from ..utils import transform_and_load_checkpoint GENERATE_METRIC_NAMES = ['ADGENMetric', 'EmF1Metric'] SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() @MindFormerRegister.register(MindFormerModuleType.TRAINER) class CausalLanguageModelingTrainer(BaseTrainer): r"""CausalLanguageModelingTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import CausalLanguageModelingTrainer >>> gen_trainer = CausalLanguageModelingTrainer(model_name="gpt2") >>> gen_trainer.train() >>> res = gen_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(CausalLanguageModelingTrainer, self).__init__("text_generation", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for CausalLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for CausalLanguageModeling Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. """ metric_name = config.metric.type kwargs.setdefault("metric_name", metric_name) if metric_name in GENERATE_METRIC_NAMES: self.generate_evaluate( config, network=network, dataset=dataset, compute_metrics=compute_metrics, **kwargs) else: self.evaluate_process( config=config, network=network, dataset=dataset, callbacks=callbacks, compute_metrics=compute_metrics, **kwargs) def generate_evaluate(self, config, network=None, dataset=None, compute_metrics=None, tokenizer=None, **kwargs): r"""Evaluate the text generate task. Return metrics with Rouge-1, Rouge-2, Rouge-l and BLEU. """ metric_name = kwargs.get("metric_name") is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) enable_max_new_tokens = bool(config.model.model_config.max_new_tokens) # it does not support max_new_tokens as input parameter in text_generate, so reset batch_size to 1 when the # follow scenario happens if metric_name == "EmF1Metric" and enable_max_new_tokens and config.runner_config.batch_size != 1: logger.info("For metric %s, it only supports batch size equals 1, so reset batch size to 1 here.", metric_name) config.runner_config.batch_size = 1 # build dataset logger.info(".........Build Dataset For Evaluate..........") if dataset is None: dataset = self.create_eval_dataset() self.set_eval_dataset(dataset) # build network if network is None: _, network = self.create_network(default_args={"parallel_config": config.parallel_config, "moe_config": config.moe_config}) self.set_network(network, is_train=False) self.count_parameters() # build metric logger.info(".........Build Compute Metrics For Evaluate..........") if compute_metrics is None: compute_metrics = build_metric(config.metric) compute_metrics.clear() # build tokenizer logger.info(".........Build tokenizer For Evaluate..........") if tokenizer is None and config.processor.tokenizer: tokenizer = build_tokenizer(config.processor.tokenizer) logger.info(".........Starting Init Evaluate Model..........") model = Model(network, eval_network=network) if config.load_checkpoint or config.only_save_strategy: if config.load_checkpoint in SUPPORT_MODEL_NAMES: config.load_checkpoint = \ AutoModel.from_pretrained(config.load_checkpoint).default_checkpoint_download_path logger.info(".............Start load checkpoint for eval..................") transform_and_load_checkpoint(config, model, network, dataset, do_eval=True) logger.info('.........Starting Evaluate Model..........') if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) # generate config do_sample = config.model.model_config.do_sample top_p = config.model.model_config.top_p top_k = config.model.model_config.top_k max_length = config.model.model_config.max_decode_length total_tokens_num = 0 total_time = 0.0001 pad_token_id = tokenizer.pad_token_id len_dataset = dataset.get_dataset_size() for i, inputs in enumerate(dataset.create_dict_iterator()): input_ids = inputs['input_ids'].asnumpy() labels = inputs['labels'].asnumpy() valid_length_each_example = [] for j in range(input_ids.shape[0]): # As the nonzero returns the index and we need length valid_length_each_example.append(np.max(np.argwhere(input_ids[j] != pad_token_id)) + 1) valid_length_each_example = np.array(valid_length_each_example) if enable_max_new_tokens: # When we act as it, the batch_size is 1. it will be replaced when text_generator supports batch_size # inference quickly or text_generator supports max_new_tokens as the input parameter. max_length = valid_length_each_example[0] + self.config.model.model_config.max_new_tokens start_time = time.time() outputs = model.predict_network.generate(input_ids, do_sample=do_sample, max_length=max_length, top_p=top_p, top_k=top_k) output_ids = [] for j in range(input_ids.shape[0]): output_ids.append(outputs[j][int(valid_length_each_example[j]):]) end_time = time.time() avg_cost_time = (end_time - start_time) / input_ids.shape[0] tokens_num = 0 for batch_index in range(len(output_ids)): tokens_num += output_ids[batch_index].shape[0] if i != 0: total_tokens_num += tokens_num total_time += end_time - start_time # compute time remaining avg_time = total_time / (i + 1) remain_time = (len_dataset - i - 1) * avg_time logger.info(f"Step[{i+1}/{len_dataset}], cost time {end_time-start_time:.4f}s, "+ f"every example cost time is {avg_cost_time:.4f}, "+ f"generate speed: {tokens_num/(end_time-start_time):.4f} tokens/s, "+ f"avg speed: {total_tokens_num/total_time:.4f} tokens/s, " f"remaining time: {datetime.timedelta(seconds=int(remain_time))}") # decode input_id and label to string pres_str = tokenizer.decode(output_ids, skip_special_tokens=True) labels_str = tokenizer.decode(labels, skip_special_tokens=True) compute_metrics.update(pres_str, labels_str) compute_metrics.eval() logger.info('...........Evaluate Over!...............') def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list, GeneratorDataset]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list, GeneratorDataset)): raise ValueError("Input data's type must be one of " f"[str, list, GeneratorDataset], but got type {type(input_data)}") if isinstance(input_data, str) and os.path.isfile(input_data): with open(input_data, 'r') as fp: input_data_list = [] for line in fp: input_data_list.extend(line) input_data = input_data_list return self.predict_process(config=config, input_data=input_data, task='text_generation', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/config_args.py000066400000000000000000000325541476650141400247470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers' Config API.""" import os from typing import Optional, Union from dataclasses import dataclass import inspect from mindformers.core.callback import CheckpointMointor from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType __all__ = ['BaseArgsConfig', 'RunnerConfig', 'DatasetConfig', 'DataLoaderConfig', 'ConfigArguments', 'ContextConfig', 'CloudConfig', 'CheckpointConfig', 'ParallelContextConfig', 'OptimizerConfig', 'LRConfig', 'WrapperConfig'] @dataclass class BaseArgsConfig: """Base Argument config.""" _support_kwargs = [] def __init__(self, **kwargs): if kwargs is not None: for key, value in kwargs.items(): assert key in self._support_kwargs, \ f"The Config Class support input argument is {self._support_kwargs}, but get {key}" if value is None: continue if isinstance(value, BaseArgsConfig): value = value.__dict__ self.__setattr__(key, value) @dataclass class ContextConfig(BaseArgsConfig): """Context Config.""" _support_kwargs = [ 'mode', 'precompile_only', 'device_target', 'device_id', 'save_graphs', 'save_graphs_path', 'enable_dump', 'auto_tune_mode', 'save_dump_path', 'enable_reduce_precision', 'variable_memory_max_size', 'enable_profiling', 'profiling_options', 'enable_auto_mixed_precision', 'enable_graph_kernel', 'reserve_class_name_in_scope', 'check_bprop', 'max_device_memory', 'print_file_path', 'enable_sparse', 'max_call_depth', 'env_config_path', 'graph_kernel_flags', 'save_compile_cache', 'runtime_num_threads', 'load_compile_cache', 'grad_for_scalar', 'pynative_synchronize', 'mempool_block_size' ] def __init__(self, mode: Optional[Union[int, str]] = 0, device_target: str = "Ascend", device_id: int = int(os.getenv('DEVICE_ID', '0')), save_graphs: bool = False, save_graphs_path: str = ".", **kwargs): super(ContextConfig, self).__init__(mode=mode, device_id=device_id, device_target=device_target, save_graphs=save_graphs, save_graphs_path=save_graphs_path, **kwargs) @dataclass class ParallelContextConfig(BaseArgsConfig): """Parallel Context Config.""" _support_kwargs = [ 'device_num', 'global_rank', 'gradients_mean', 'gradient_fp32_sync', 'parallel_mode', 'auto_parallel_search_mode', 'search_mode', 'parameter_broadcast', 'strategy_ckpt_load_file', 'strategy_ckpt_save_file', 'full_batch', 'enable_parallel_optimizer', 'enable_alltoall', 'all_reduce_fusion_config', 'pipeline_stages', 'grad_accumulation_step', 'parallel_optimizer_config', 'comm_fusion' ] def __init__(self, parallel_mode: str = 'STAND_ALONE', device_num: int = int(os.getenv('RANK_SIZE', '1')), gradients_mean: bool = False, **kwargs): super(ParallelContextConfig, self).__init__(parallel_mode=parallel_mode, device_num=device_num, gradients_mean=gradients_mean, **kwargs) @dataclass class CloudConfig(BaseArgsConfig): """Cloud Config For ModelArts.""" _support_kwargs = [ 'obs_path', 'root_path', 'rank_id', 'upload_frequence', 'keep_last', 'retry', 'retry_time' ] def __init__(self, obs_path: str = None, root_path: str = '/cache', rank_id: int = None, upload_frequence: int = 1, keep_last: bool = False, **kwargs): super(CloudConfig, self).__init__(obs_path=obs_path, root_path=root_path, rank_id=rank_id, upload_frequence=upload_frequence, keep_last=keep_last, **kwargs) @dataclass class RunnerConfig(BaseArgsConfig): """MindFormers' config when running model.""" _support_kwargs = [ 'epochs', 'batch_size', 'sink_mode', 'sink_size', 'initial_epoch', 'has_trained_epoches', 'has_trained_steps', 'image_size', 'num_classes', 'sink_size', ] def __init__(self, epochs: int = None, batch_size: int = None, sink_mode: bool = None, sink_size: int = None, initial_epoch: int = None, has_trained_epoches: int = None, has_trained_steps: int = None, **kwargs): super(RunnerConfig, self).__init__(epochs=epochs, batch_size=batch_size, sink_mode=sink_mode, sink_size=sink_size, initial_epoch=initial_epoch, has_trained_steps=has_trained_steps, has_trained_epoches=has_trained_epoches, **kwargs) @dataclass class CheckpointConfig(BaseArgsConfig): """MindFormers' save checkpoint config.""" _support_kwargs = inspect.getfullargspec(CheckpointMointor).args def __init__(self, prefix: str = 'mindformers', directory: str = None, save_checkpoint_steps: int = 1, keep_checkpoint_max: int = 1, integrated_save: bool = True, async_save: bool = False, saved_network: bool = None, **kwargs): super(CheckpointConfig, self).__init__(prefix=prefix, directory=directory, saved_network=saved_network, save_checkpoint_steps=save_checkpoint_steps, keep_checkpoint_max=keep_checkpoint_max, integrated_save=integrated_save, async_save=async_save, **kwargs) @dataclass class LRConfig(BaseArgsConfig): """MindFormers' learning rate schedule config.""" _support_kwargs = [ 'type', 'max_lr', 'min_lr', 'decay_steps', 'decay_rate', 'power', 'end_learning_rate', 'warmup_steps' ] def __init__(self, lr_type: str = None, **kwargs): if lr_type is not None: lr_schedule = MindFormerRegister.get_cls( module_type=MindFormerModuleType.LR, class_name=lr_type) self._support_kwargs.extend(inspect.getfullargspec(lr_schedule).args) super(LRConfig, self).__init__(type=lr_type, **kwargs) @dataclass class OptimizerConfig(BaseArgsConfig): """MindFormers' optimizer config.""" _support_kwargs = [ 'type', 'learning_rate', 'beta1', 'beta2', 'eps', 'epsilon', 'weight_decay', 'loss_scale', 'momentum' ] def __init__(self, optim_type: str = None, learning_rate: Optional[Union[BaseArgsConfig, float]] = None, **kwargs): if optim_type is not None: optimizer = MindFormerRegister.get_cls( module_type=MindFormerModuleType.OPTIMIZER, class_name=optim_type) self._support_kwargs.extend(inspect.getfullargspec(optimizer).args) super(OptimizerConfig, self).__init__(type=optim_type, learning_rate=learning_rate, **kwargs) @dataclass class WrapperConfig(BaseArgsConfig): """MindFormers' wrapper config.""" _support_kwargs = [ 'type', 'sens', 'scale_sense' ] def __init__(self, wrapper_type: str = None, **kwargs): if wrapper_type is not None: wrapper = MindFormerRegister.get_cls( module_type=MindFormerModuleType.WRAPPER, class_name=wrapper_type) self._support_kwargs.extend(inspect.getfullargspec(wrapper).args) super(WrapperConfig, self).__init__(type=wrapper_type, **kwargs) @dataclass class DataLoaderConfig(BaseArgsConfig): """MindFormers' data loader config.""" _support_kwargs = [ 'type', 'dataset_dir', 'num_samples', 'num_parallel_workers', 'shuffle', 'sampler', 'extensions', 'class_indexing', 'language_pair', 'decode', 'num_shards', 'shard_id', 'cache', 'decrypt', 'task', 'usage', 'test_set', 'valid_set', 'padded_sample', 'num_padded' ] def __init__(self, dataloader_type: str = None, dataset_dir: str = None, **kwargs): if dataloader_type is not None: dataloader = MindFormerRegister.get_cls( MindFormerModuleType.DATASET_LOADER, class_name=dataloader_type) self._support_kwargs.extend(inspect.getfullargspec(dataloader).args) super(DataLoaderConfig, self).__init__(type=dataloader_type, dataset_dir=dataset_dir, **kwargs) @dataclass class DatasetConfig(BaseArgsConfig): """MindFormers' dataset config.""" _support_kwargs = [ 'data_loader', 'input_columns', 'output_columns', 'column_order', 'drop_remainder', 'repeat', 'batch_size', 'image_size', 'num_parallel_workers', 'per_batch_map', 'python_multiprocessing', 'max_rowsize', 'cache', 'offload' ] def __init__(self, data_loader: Optional[Union[dict, BaseArgsConfig]] = None, input_columns: Optional[Union[str, list]] = None, output_columns: Optional[Union[str, list]] = None, column_order: Optional[Union[str, list]] = None, drop_remainder: bool = True, repeat: int = 1, batch_size: int = None, image_size: Optional[Union[int, list, tuple]] = None, **kwargs): super(DatasetConfig, self).__init__(data_loader=data_loader, batch_size=batch_size, image_size=image_size, repeat=repeat, input_columns=input_columns, output_columns=output_columns, column_order=column_order, drop_remainder=drop_remainder, **kwargs) @dataclass class ConfigArguments(BaseArgsConfig): """MindFormers' config arguments.""" _support_kwargs = [ 'output_dir', 'profile', 'auto_tune', 'filepath_prefix', 'autotune_per_step', 'train_dataset', 'eval_dataset', 'predict_dataset', 'runner_config', 'optimizer', 'lr_schedule', 'save_checkpoint', 'cloud_config', 'seed', 'runner_wrapper' ] def __init__(self, output_dir: str = './output', profile: bool = False, auto_tune: bool = False, filepath_prefix: str = './autotune', autotune_per_step: int = 10, seed: int = None, train_dataset: Optional[Union[dict, BaseArgsConfig]] = None, eval_dataset: Optional[Union[dict, BaseArgsConfig]] = None, runner_config: Optional[Union[dict, BaseArgsConfig]] = None, optimizer: Optional[Union[dict, BaseArgsConfig]] = None, runner_wrapper: Optional[Union[dict, BaseArgsConfig]] = None, lr_schedule: Optional[Union[dict, BaseArgsConfig]] = None, save_checkpoint: Optional[Union[dict, BaseArgsConfig]] = None, cloud_config: Optional[Union[dict, BaseArgsConfig]] = None): super(ConfigArguments, self).__init__(output_dir=output_dir, profile=profile, auto_tune=auto_tune, seed=seed, filepath_prefix=filepath_prefix, autotune_per_step=autotune_per_step, train_dataset=train_dataset, eval_dataset=eval_dataset, runner_config=runner_config, optimizer=optimizer, runner_wrapper=runner_wrapper, lr_schedule=lr_schedule, save_checkpoint=save_checkpoint, cloud_config=cloud_config) pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain/000077500000000000000000000000001476650141400313555ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain/__init__.py000066400000000000000000000015421476650141400334700ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Trainer API.""" from .contrastive_language_image_pretrain import ContrastiveLanguageImagePretrainTrainer __all__ = ['ContrastiveLanguageImagePretrainTrainer'] pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain/__pycache__/000077500000000000000000000000001476650141400335655ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000006001476650141400376720ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain/__pycache__a ?cfb@sdZddlmZdgZdS)z0Contrastive Language Image Pretrain Trainer API.)'ContrastiveLanguageImagePretrainTrainerrN)__doc__#contrastive_language_image_pretrainr__all__rrk/home/ma-user/work/mindformers-baichuan/mindformers/trainer/contrastive_language_image_pretrain/__init__.pys contrastive_language_image_pretrain.cpython-39.pyc000066400000000000000000000101621476650141400454110ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain/__pycache__a ?cf@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)z,Contrastive Language Image Pretrain Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZd edfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zd d ZZS)'ContrastiveLanguageImagePretrainTraineraContrastive Language Image Pretrain Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ContrastiveLanguageImagePretrainTrainer >>> trainer = ContrastiveLanguageImagePretrainTrainer(model_name="clip_vit_b_b32") >>> trainer.train() N) model_namecstt|d|dS)N#contrastive_language_image_pretrain)superr__init__)selfr __class__/home/ma-user/work/mindformers-baichuan/mindformers/trainer/contrastive_language_image_pretrain/contrastive_language_image_pretrain.pyr0s z0ContrastiveLanguageImagePretrainTrainer.__init__)confignetworkdataset optimizerwrapper callbacksc Ks"|jf||||||d|dS)aTrain For Trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The optimizer used for training .Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. )rrr"rr r!N)training_process)rrrrr r!r"kwargsrrrtrain4sz-ContrastiveLanguageImagePretrainTrainer.traincKs tddS)NzGThe contrastive language image pretrain task does not support evaluate.NotImplementedErrorrr$rrrevaluateZsz0ContrastiveLanguageImagePretrainTrainer.evaluatecKs tddS)NzFThe contrastive language image pretrain task does not support predict.r&r(rrrpredict^sz/ContrastiveLanguageImagePretrainTrainer.predict)N)NNNNNN)__name__ __module__ __qualname____doc__strrrrdictrrrr r r rrrrrr%r)r* __classcell__rrrrr s$ &rN)r.typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr mindformers.tools.registerr r r config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        contrastive_language_image_pretrain.py000066400000000000000000000103201476650141400411160ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/contrastive_language_image_pretrain# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Contrastive Language Image Pretrain Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ContrastiveLanguageImagePretrainTrainer(BaseTrainer): r"""Contrastive Language Image Pretrain Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ContrastiveLanguageImagePretrainTrainer >>> trainer = ContrastiveLanguageImagePretrainTrainer(model_name="clip_vit_b_b32") >>> trainer.train() """ def __init__(self, model_name: str = None): super(ContrastiveLanguageImagePretrainTrainer, self).__init__( "contrastive_language_image_pretrain", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, optimizer: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train For Trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The optimizer used for training .Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, optimizer=optimizer, wrapper=wrapper, **kwargs) def evaluate(self, **kwargs): raise NotImplementedError( "The contrastive language image pretrain task does not support evaluate.") def predict(self, **kwargs): raise NotImplementedError( "The contrastive language image pretrain task does not support predict.") pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/000077500000000000000000000000001476650141400262665ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/__init__.py000066400000000000000000000014331476650141400304000ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """General Task Example For Trainer.""" from .general_task_trainer import GeneralTaskTrainer __all__ = ['GeneralTaskTrainer'] pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/__pycache__/000077500000000000000000000000001476650141400304765ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000004761476650141400346160ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/__pycache__a ?cf@sdZddlmZdgZdS)z!General Task Example For Trainer.)GeneralTaskTrainerrN)__doc__general_task_trainerr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/general_task_trainer/__init__.pys general_task_trainer.cpython-39.pyc000066400000000000000000000204631476650141400372400ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/__pycache__a ?cfT$@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)z!General Task Example For Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zd d ZZS)GeneralTaskTraineraK General Task Example For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.ones(1000, np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> train_dataset = train_dataset.batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> eval_dataset = eval_dataset.batch(batch_size=2) >>> general_task = GeneralTaskTrainer(model_name='common') >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> general_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> general_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ngeneral)superr__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/general_task_trainer/general_task_trainer.pyrHszGeneralTaskTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc KsJ|durtd|dur(|dur(td|jf||||||d|dS)aTrain task for General-Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If network or dataset not implemented. Nz+train dataset must be define, but get None.z+train network must be define, but get None.)rrr"rr r!)NotImplementedErrortraining_process)rrrrr r!r"kwargsrrrtrainKs%zGeneralTaskTrainer.train)rrrr"compute_metricscKs`|durtd|dur td|dur0tdd}|d||jf|||||d|dS)aEvaluate task for General-Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callback, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. Raises: NotImplementedError: If network or dataset or compute_metrics not implemented. Nz*eval dataset must be define, but get None.z*eval network must be define, but get None.z*eval metrics must be define, but get None.zGeneral Task Metrics metric_name)rrrr'r")r# setdefaultevaluate_process)rrrrr"r'r%r(rrrevaluate~s"" zGeneralTaskTrainer.evaluatecKs tddS)NzOThe general task does not support predict, please customize pipeline inference.)r#)rr%rrrpredictszGeneralTaskTrainer.predict)N)NNNNNN)NNNNN)__name__ __module__ __qualname____doc__strrrrdictrrrr r r rrrrrr&setr+r, __classcell__rrrrrs:' 4 5rN)r0typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr mindformers.tools.registerr r r config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        pengcheng.mind-mindformer/mindformers/trainer/general_task_trainer/general_task_trainer.py000066400000000000000000000221241476650141400330240ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """General Task Example For Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class GeneralTaskTrainer(BaseTrainer): r"""General Task Example For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.ones(1000, np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> train_dataset = train_dataset.batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']) >>> eval_dataset = eval_dataset.batch(batch_size=2) >>> general_task = GeneralTaskTrainer(model_name='common') >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> general_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> general_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(GeneralTaskTrainer, self).__init__("general", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for General-Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If network or dataset not implemented. """ if dataset is None: raise NotImplementedError("train dataset must be define, but get None.") if network is None and wrapper is None: raise NotImplementedError("train network must be define, but get None.") self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for General-Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callback, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. Raises: NotImplementedError: If network or dataset or compute_metrics not implemented. """ if dataset is None: raise NotImplementedError("eval dataset must be define, but get None.") if network is None: raise NotImplementedError("eval network must be define, but get None.") if compute_metrics is None: raise NotImplementedError("eval metrics must be define, but get None.") metric_name = "General Task Metrics" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, **kwargs): raise NotImplementedError( "The general task does not support predict, please customize pipeline inference.") pengcheng.mind-mindformer/mindformers/trainer/image_classification/000077500000000000000000000000001476650141400262405ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_classification/__init__.py000066400000000000000000000016401476650141400303520ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer API.""" from .image_classification import ImageClassificationTrainer from .zero_shot_image_classification import ZeroShotImageClassificationTrainer __all__ = ['ImageClassificationTrainer', 'ZeroShotImageClassificationTrainer'] pengcheng.mind-mindformer/mindformers/trainer/image_classification/__pycache__/000077500000000000000000000000001476650141400304505ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000006461476650141400345670ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_classification/__pycache__a ?cf@s(dZddlmZddlmZddgZdS)z!Image Classification Trainer API.)ImageClassificationTrainer)"ZeroShotImageClassificationTrainerrrN)__doc__image_classificationrzero_shot_image_classificationr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/__init__.pys  image_classification.cpython-39.pyc000066400000000000000000000244571476650141400371730ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_classification/__pycache__a ?cf-@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZddl m!Z!ddl"m#Z#ddl$m%Z%dgZ&ej'ej(ddGddde%Z)dS)zImage Classification Trainer.)OptionalListUnionN)Image)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell)Tensor) BaseDataset) BaseModelBaseImageProcessor)logger) load_image)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerImageClassificationTrainerimage_classification)aliasc s(eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd d d Zdeeee e e feeeejeeefeee e feed d dZZS)ra ImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import ImageClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.eye(1000)[i] for i in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> input_data = np.uint8(np.random.random((5, 3, 255, 255))) >>> #1) use config to train >>> cls_task = ImageClassificationTrainer(model_name='vit_base_p16') >>> cls_task.train(dataset=train_dataset) >>> cls_task.evaluate(dataset=eval_dataset) >>> cls_task.predict(input_data=input_data, top_k=5) >>> #2) use instance function to train >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> cls_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> cls_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) >>> image_processor = ViTImageProcessor(size=224) >>> cls_task.predict(input_data=input_data, image_processor=image_processor, top_k=5) N) model_namecstd|dS)Nr)super__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/image_classification.pyr]sz#ImageClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for ImageClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r#r$r(r%r&r'N)Ztraining_process)rr#r$r%r&r'r(kwargsr!r!r"train`s%z ImageClassificationTrainer.train)r#r$r%r(compute_metricscKs0d}|d||jf|||||d|dS)aEvaluate task for ImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Top1 Accuracy metric_name)r#r$r%r+r(N) setdefaultZevaluate_process)rr#r$r%r(r+r)r,r!r!r"evaluates z#ImageClassificationTrainer.evaluate)r# input_datar$image_processorcKstd|dur|j}t|ttjttt fs6t dg}t|trT| t |n(t|t rx|D]}| t |qbn|}|j f||d||d|S)apPredict task for ImageClassification Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. z/.........Build Input Data For Predict..........NzTInput data's type must be one of [str, ms.Tensor, np.ndarray, PIL.Image.Image, list]r)r#r/taskr$r0)rinfor/ isinstancer npndarrayrstrlist ValueErrorappendrZpredict_process)rr#r/r$r0r)Zbatch_input_data data_pathr!r!r"predicts(   z"ImageClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r6rrrdictrrrr r r rrr rrr*setr.r r4r5rr7rr; __classcell__r!r!rr"r(sL3 / +)*r?typingrrrnumpyr4Z PIL.ImagerZmindspore.trainrmindspore.datasetr mindspore.nnrr r mindsporer mindformers.datasetr mindformers.modelsr rmindformers.tools.loggerrZmindformers.tools.image_toolsrmindformers.tools.registerrrr config_argsr training_argsr base_trainerr__all__registerTRAINERrr!r!r!r"s$          zero_shot_image_classification.cpython-39.pyc000066400000000000000000000143521476650141400412600ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_classification/__pycache__a ?cf@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZd dlmZddlm Z ddl!m"Z"ddl#m$Z$dgZ%e&ej'Gddde$Z(dS)zImage Classification Trainer.)OptionalListUnionN)Image)Callback)Tensor)Cell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizerBaseImageProcessor)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_dataset_loader)ConfigArguments)TrainingArguments) BaseTrainer"ZeroShotImageClassificationTrainerc seZdZdZdedfdd ZddZdeee e e e feee efeeeefeeeeefeee efdd d Zdeee e e e feeeeejeeefeee efeeeed d d ZZS)raZeroShotImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ZeroShotImageClassificationTrainer >>> trainer = ZeroShotImageClassificationTrainer(model_name="clip_vit_b_32") >>> trainer.evaluate() >>> trainer.predict() N) model_namecstt|d|dS)Nzero_shot_image_classification)superr__init__)selfr __class__r/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_classification/zero_shot_image_classification.pyr8sz+ZeroShotImageClassificationTrainer.__init__cOs tddS)Nzs$            pengcheng.mind-mindformer/mindformers/trainer/image_classification/group_ic_params.py000066400000000000000000000110161476650141400317630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/SimMIM # ============================================================================ """group parameters for masked image modeling.""" import json from functools import partial from mindformers.core.lr import LearningRateWiseLayer from mindformers.tools import logger def get_group_parameters(config, model, base_lr): """get finetune param groups""" if config.model.arch.type == 'SwinForImageClassification': depths = config.model.model_config.depths num_layers = sum(depths) get_layer_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) elif config.model.arch.type == 'ViTForImageClassification': num_layers = config.model.model_config.num_hidden_layers get_layer_func = partial(get_vit_layer, num_layers=num_layers + 2) else: raise NotImplementedError layer_decay = config.runner_config.layer_decay scales = list(layer_decay ** i for i in reversed(range(num_layers + 2))) skip = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip = model.no_weight_decay() logger.info('No weight decay: %s', skip) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) parameter_group_names = {} parameter_group_vars = {} for param in model.trainable_params(): if len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip) or \ check_keywords_in_name(param.name, skip_keywords): group_name = "no_decay" this_weight_decay = 0. else: group_name = "decay" this_weight_decay = config.optimizer.weight_decay if get_layer_func is not None: layer_id = get_layer_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) else: layer_id = None if group_name not in parameter_group_names: if scales is not None: scale = scales[layer_id] else: scale = 1. parameter_group_names[group_name] = { "weight_decay": this_weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": this_weight_decay, "params": [], "lr": LearningRateWiseLayer(base_lr, scale), } parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[1]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def check_keywords_in_name(name, keywords=()): isin = False for keyword in keywords: if keyword in name: isin = True break return isin pengcheng.mind-mindformer/mindformers/trainer/image_classification/image_classification.py000066400000000000000000000267411476650141400327610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore import Tensor from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.image_tools import load_image from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ImageClassificationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="image_classification") class ImageClassificationTrainer(BaseTrainer): r""" ImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell, Accuracy >>> from mindformers.trainer import ImageClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTForImageClassification, ViTConfig, ViTImageProcessor >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self._label = [np.eye(1000)[i] for i in range(64)] ... ... def __getitem__(self, index): ... return self._data[index], self._label[index] ... ... def __len__(self): ... return len(self._data) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> eval_dataset = GeneratorDataset(source=MyDataLoader(), column_names=['image', 'label']).batch(batch_size=2) >>> input_data = np.uint8(np.random.random((5, 3, 255, 255))) >>> #1) use config to train >>> cls_task = ImageClassificationTrainer(model_name='vit_base_p16') >>> cls_task.train(dataset=train_dataset) >>> cls_task.evaluate(dataset=eval_dataset) >>> cls_task.predict(input_data=input_data, top_k=5) >>> #2) use instance function to train >>> vit_config = ViTConfig(batch_size=2) >>> network_with_loss = ViTForImageClassification(vit_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> cls_task.train(wrapper=wrapper, dataset=train_dataset) >>> compute_metrics = {"Accuracy": Accuracy(eval_type='classification')} >>> cls_task.evaluate(network=network_with_loss, dataset=eval_dataset, compute_metrics=compute_metrics) >>> image_processor = ViTImageProcessor(size=224) >>> cls_task.predict(input_data=input_data, image_processor=image_processor, top_k=5) """ def __init__(self, model_name: str = None): super().__init__('image_classification', model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for ImageClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for ImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ImageClassification Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (Tensor, np.ndarray, Image, str, list)): raise ValueError("Input data's type must be one of " "[str, ms.Tensor, np.ndarray, PIL.Image.Image, list]") batch_input_data = [] if isinstance(input_data, str): batch_input_data.append(load_image(input_data)) elif isinstance(input_data, list): for data_path in input_data: batch_input_data.append(load_image(data_path)) else: batch_input_data = input_data return self.predict_process(config=config, input_data=batch_input_data, task='image_classification', network=network, image_processor=image_processor, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/image_classification/zero_shot_image_classification.py000066400000000000000000000170251476650141400350500ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image Classification Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore.train import Callback from mindspore import Tensor from mindspore.nn import Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ...dataset.dataloader import build_dataset_loader from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ZeroShotImageClassificationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ZeroShotImageClassificationTrainer(BaseTrainer): r"""ZeroShotImageClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers import ZeroShotImageClassificationTrainer >>> trainer = ZeroShotImageClassificationTrainer(model_name="clip_vit_b_32") >>> trainer.evaluate() >>> trainer.predict() """ def __init__(self, model_name: str = None): super(ZeroShotImageClassificationTrainer, self).__init__("zero_shot_image_classification", model_name) def train(self, *args, **kwargs): raise NotImplementedError( "The ZeroShotImageClassification task does not support train.") def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for ZeroShotImageClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It supports CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It supports dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ZeroShotImageClassification Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None and config.input_data is not None: input_data = config.input_data if input_data is None: input_data = build_dataset_loader(config.eval_dataset.data_loader) candidate_labels = kwargs.pop("candidate_labels", None) if candidate_labels is None: if hasattr(input_data, "label_names"): candidate_labels = input_data.label_names else: candidate_labels = ["sunflower", "tree", "dog", "cat", "toy"] hypothesis_template = kwargs.pop("hypothesis_template", None) if hypothesis_template is None: if config.eval_dataset.data_loader.hypothesis_template is not None: hypothesis_template = config.eval_dataset.data_loader.hypothesis_template else: hypothesis_template = "{}" return self.predict_process(config=config, input_data=input_data, task='zero_shot_image_classification', network=network, tokenizer=tokenizer, image_processor=image_processor, candidate_labels=candidate_labels, hypothesis_template=hypothesis_template, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/000077500000000000000000000000001476650141400271465ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/__init__.py000066400000000000000000000014671476650141400312670ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image to text generation Trainer API.""" from .image_to_text_generation import ImageToTextGenerationTrainer __all__ = ['ImageToTextGenerationTrainer'] pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/__pycache__/000077500000000000000000000000001476650141400313565ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005241476650141400354700ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/__pycache__a ?cf7@sdZddlmZdgZdS)z%Image to text generation Trainer API.)ImageToTextGenerationTrainerrN)__doc__image_to_text_generationr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_generation/__init__.pys image_to_text_generation.cpython-39.pyc000066400000000000000000000106601476650141400407760ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/__pycache__a ?cf{@sdZddlmZmZddlZddlmZddlm Z ddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZmZd d lmZd dlmZd dlmZd dlmZdgZ e!ej"GdddeZ#dS)zImageToTextGeneration Trainer.)OptionalUnionN)Image)Tensor)Cell)GeneratorDataset) BaseModel BaseTokenizerBaseImageProcessor)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)build_dataset_loader)ConfigArguments)TrainingArguments) BaseTrainerImageToTextGenerationTrainerc seZdZdZd edfdd ZddZdd Zdee e e e e fee eeejeeefee eefeeeed d d ZZS)raImageToTextGenerationTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers.trainer import ImageToTextGenerationTrainer >>> trainer = ImageToTextGenerationTrainer(model_name="blip2_stage2_vit_g_llama_7b") >>> trainer.predict() N) model_namecstt|d|dS)Nimage_to_text_generation)superr__init__)selfr __class__p/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_generation/image_to_text_generation.pyr5sz%ImageToTextGenerationTrainer.__init__cOs tddS)Nz=The ImageToTextGenerationTrainer task does not support train.NotImplementedErrorrargskwargsrrrtrain8sz"ImageToTextGenerationTrainer.traincOs tddS)Nz@The ImageToTextGenerationTrainer task does not support evaluate.rr!rrrevaluate<sz%ImageToTextGenerationTrainer.evaluate)config input_datanetwork tokenizerimage_processorc Ks||}td|dur,|jdur,|j}|dur@t|jj}|dd}|durr|jj j durn|jj j }nd}|dd}|dur|jj j dur|jj j }nd}|dd} | dur|jjj dur|jjj } nd} |j f||d|||||| d |S) a;Predict task for ZeroShotImageToTextGenerationTrainer Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. z/.........Build Input Data For Predict..........N max_length paddinghypothesis_templater) r&r'taskr(r)r*r+r-r.) set_configr infor'r eval_dataset data_loaderpop processorr)r+r-r.predict_process) rr&r'r(r)r*r#r+r-r.rrrpredict@sB          z$ImageToTextGenerationTrainer.predict)N)NNNNN)__name__ __module__ __qualname____doc__strrr$r%rrdictrrrrrnpndarrayrlistrrr r r8 __classcell__rrrrr%s$ )$r<typingrrnumpyr?Z PIL.Imager mindsporer mindspore.nnrmindspore.datasetrmindformers.modelsrr r mindformers.tools.loggerr mindformers.tools.registerr r rZdataset.dataloaderr config_argsr training_argsr base_trainerr__all__registerTRAINERrrrrrs           pengcheng.mind-mindformer/mindformers/trainer/image_to_text_generation/image_to_text_generation.py000066400000000000000000000131731476650141400345700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ImageToTextGeneration Trainer.""" from typing import Optional, Union import numpy as np from PIL.Image import Image from mindspore import Tensor from mindspore.nn import Cell from mindspore.dataset import GeneratorDataset from mindformers.models import BaseModel, BaseTokenizer, BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ...dataset.dataloader import build_dataset_loader from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer __all__ = ['ImageToTextGenerationTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class ImageToTextGenerationTrainer(BaseTrainer): r"""ImageToTextGenerationTrainer Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. Examples: >>> from mindformers.trainer import ImageToTextGenerationTrainer >>> trainer = ImageToTextGenerationTrainer(model_name="blip2_stage2_vit_g_llama_7b") >>> trainer.predict() """ def __init__(self, model_name: str = None): super(ImageToTextGenerationTrainer, self).__init__("image_to_text_generation", model_name) def train(self, *args, **kwargs): raise NotImplementedError( "The ImageToTextGenerationTrainer task does not support train.") def evaluate(self, *args, **kwargs): raise NotImplementedError( "The ImageToTextGenerationTrainer task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for ZeroShotImageToTextGenerationTrainer Trainer. This function is used to predict the network. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. input_data (Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]]): The dataset. It supports real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): Used for text process. image_processor (Optional[BaseImageProcessor]): Used for image process. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None and config.input_data is not None: input_data = config.input_data if input_data is None: input_data = build_dataset_loader(config.eval_dataset.data_loader) max_length = kwargs.pop("max_length", None) if max_length is None: if config.processor.tokenizer.max_length is not None: max_length = config.processor.tokenizer.max_length else: max_length = 32 padding = kwargs.pop("padding", None) if padding is None: if config.processor.tokenizer.padding is not None: padding = config.processor.tokenizer.padding else: padding = "max_length" hypothesis_template = kwargs.pop("hypothesis_template", None) if hypothesis_template is None: if config.eval_dataset.data_loader.hypothesis_template is not None: hypothesis_template = config.eval_dataset.data_loader.hypothesis_template else: hypothesis_template = "" return self.predict_process(config=config, input_data=input_data, task='image_to_text_generation', network=network, tokenizer=tokenizer, image_processor=image_processor, max_length=max_length, padding=padding, hypothesis_template=hypothesis_template, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/000077500000000000000000000000001476650141400270105ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/__init__.py000066400000000000000000000014621476650141400311240ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer API.""" from .image_to_text_retrieval import ImageToTextRetrievalTrainer __all__ = ['ImageToTextRetrievalTrainer'] pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/__pycache__/000077500000000000000000000000001476650141400312205ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005201476650141400353260ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/__pycache__a ?cf2@sdZddlmZdgZdS)z$Image-to-text Retrieval Trainer API.)ImageToTextRetrievalTrainerrN)__doc__image_to_text_retrievalr__all__rr_/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/__init__.pys eval_utils.cpython-39.pyc000066400000000000000000000073211476650141400357440ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/__pycache__a ?cf@sRdZddlZddlmZddlmZddlmZddl m Z ddZ d d d Z dS)z&Image-to-text Retrieval Trainer Utils.N)Tensor)logger)topk)MultiImgCapDataLoaderc sT|}t|dr$|jdur$|jd}qt|tr@|j}|j|jfStdt ||j d|j df|j d|j dfks|J|j d}|j d}||krddt |D}||fS||kr|n||kr|n|}|fddt |D} g} t |dD]} | | g7} q|d} | |dg| 7} |krL| | fS| | fS) zextract_image_text_mapping from eval_dataloader. Args: eval_dataloader: evaluation dataloader score_i2t, score_t2i: two score matrix (I2T, T2I) Returns: img2txt, txt2img: ground truth image-text mapping childrenNrzexpect the eval dataset to be generate from MultiImgCapDataLoader, but is %s, will generate image-text mapping with accumulate indexes by default.cSsg|]}|qSr.0irra/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/eval_utils.py .z.extract_image_text_mapping..cs&g|]fddtDqS)csg|]}t|qSr)minr )biggerfactorkrr r 3rz9extract_image_text_mapping...)range)r rr)rr r 3r) hasattrr isinstancersourceimg2txttxt2imgrwarningtypeshaper) Zeval_dataloader score_i2t score_t2i dataloaderdatasetZ image_numZtext_numindsZsmallerZ smaller_indsZ bigger_indsr last_numrrr extract_image_text_mappings6    ,   r#FcCs"td|g}|d|d|dd}}}|}|}|D]>}t|j|j|jj|jt||j} | d} | | qDtj |dd}t |jd|jdfd} t |D]\} } | ddkrtd | d |jdd | jt| j|\}}|j}|j}|| | |f<|rt|}|j|| ||d d }| | |f|7<q|j}t |jd|jdfd}t |D]\} } | ddkrtd| d |jdd | jt| j|\}}|j}|j}||| |f<|rzt|}|j|| ||dd }|| |f|7<qz| |fS)a compute image-text matching scores, in matrix format. Args: network (BaseModel): network for evaluate ITM eval_inputs (tuple): inputs for evaluate itm scores. k_test (int, optional): k_test num, Defaults to 128. add_extra_itm_score (bool, optional): whether to add extra scores (model decides), Defaults to False. Returns: score_matrix_i2t, score_matrix_t2i: two score matrix (I2T, T2I) z"========= k_text num: %d =========rrN)axisgY2zI2T: /z - sims.shape: T)Zi2tzT2I: F)rinfoasnumpyprintrdtypeTnpmatmulmaxappendstackfull enumeraterr from_numpyZcompute_extra_itm)network eval_inputsk_testadd_extra_itm_scoreZ sims_matrixZ image_featsZ text_feats extra_argsZ image_featZsim_q2tZsim_i2tZscore_matrix_i2tr ZsimsZtopk_simZtopk_idxZ topk_idx_msscoreZscore_matrix_t2irrr compute_itm_scores=sR     "  "  r<)r$F) __doc__numpyr. mindsporermindformers.tools.loggerrZmindformers.generation.utilsrZ9mindformers.dataset.dataloader.multi_image_cap_dataloaderrr#r<rrrr s    &image_to_text_retrieval.cpython-39.pyc000066400000000000000000000204721476650141400405040ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/__pycache__a ?cf2@shdZddlZddlmZmZmZddlmZddlZddl Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4ddl5m&Z&ddl6m(Z(e.j7e/j8ddGddde&Z9dS) Image-to-text Retrieval Trainer.N)ListOptionalUnion)pprint)dtype)Model)DynamicLossScaleUpdateCell)TrainOneStepCell Optimizer)Callback) build_datasetcheck_dataset_config BaseDataset) build_model BaseModel)build_callback)build_lr) build_optim) build_wrapper) BaseTrainer)check_runner_config)logger) count_params)MindFormerRegisterMindFormerModuleType)compute_itm_scoresextract_image_text_mapping)ConfigArgumentsimage_to_text_retrieval)aliasc seZdZdZdedfdd Zdeeee feeee feeee fee ee eeeeefdddZdeeee feeee feeee feeeeefd d d Zd d ZZS)ImageToTextRetrievalTrainerrN) model_namecs"tt|d|||_d|_dS)Nr!)superr#__init__r$kwargs)selfr$ __class__n/home/ma-user/work/mindformers-baichuan/mindformers/trainer/image_to_text_retrieval/image_to_text_retrieval.pyr&1sz$ImageToTextRetrievalTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc sHtjdstdtj||_fddtdt|durRtj }t |td|durt j j jdd }|tj}td tt|td |durZtd tj}|} tt| } ttfd d| } | jjdd| id| ig} |dur6tj| |dd }n$jjsHJdtjd| id }|durg}jr||j| t!j"d|jid tdt#dddd} |durt$j%||| dd }n t&|t'r|}ntd|tdt(|}t)t*+dddd kr$t,|j-j.j/||j.j0j.j1d!dS)"zH Training task for ImageToTextRetrievalTrainer Trainer. z2.0zNCurrently, filip only support training under mindspore2.0, but with mindspore cs$j}|D]}||jvr dSq dS)NFT)no_decay_paramsname) param_namer3keyword)r-r+r, decay_filterJs  z7ImageToTextRetrievalTrainer.train..decay_filter .........Build Dataset..........N.........Build Net..........)parallel_config moe_config default_argsu网络参数量：%s M.z".........Build Optimizer..........z$.........Build LR Schedule..........cs | S)Nr+)x)r7r+r,lz3ImageToTextRetrievalTrainer.train..)params weight_decayrA order_params)rA learning_ratezlearning_rate must be inputrDz(.........Build Running Wrapper..........g0Ari)loss_scale_value scale_factor scale_window)r.r1Zscale_update_cellzINow not support this wrapper,it should be TrainOneStepCell type, but get z,.........Starting Init Train Model..........RANK_ID0r)r2dataset_sink_mode sink_size)2ms __version__ startswithNotImplementedErrorr'rinforr train_dataset_taskrrmodelr:r;to_floatmstypefloat16strrr lr_scheduletrainable_paramslistfilterr1rBrrDprofileappend profile_cbextendrr2r rrunner_wrapper isinstancer rintosgetenvrtrain runner_configepochs sink_moderL)r(r-r.r/r0r1r2r'rXrAZ decay_params other_params group_params update_cellrSr+)r-r7r,re6s                      z!ImageToTextRetrievalTrainer.train)r-r.r/r2cKs||_|dd}|||}tdt||durBt|j}td|td|dur~t |j |j |j ddd}| tj}td |j jjdur||j jtd tt|td |durg}|jr||j|t|jtd ttd dddkr(t|||}|j j!durH|j j!nd}|"d|}|j j#} |"d| } t$|||| d\} } t%|j | | \} } |&| | | | }t|td|S)zJ Evaluation task for ImageToTextRetrievalTrainer Trainer. is_full_configFr8Nz+Create eval dataset finish, dataset size:%dr9)r:r; is_trainingr<z%.........Loading Checkpoint..........zNetwork Parameters: %s M.z/.........Build Callbacks for Evaluate..........z).........Starting Evaling Model..........rHrIrJrk_testadd_extra_itm_score)rorpz$.........Evaluate Over!.............)'r'get set_configrrQrr eval_dataset_taskget_dataset_sizerrSr:r;rTrUrV model_configcheckpoint_name_or_pathload_checkpointrWrr\r]r^r_reval_callbacksrbrcrdrZprepare_inputs_for_itm_eval eval_datasetropoprprr_report_metrics)r(r-r.r/r2r'rlZ eval_inputsrorpZ score_i2tZ score_t2iimg2txttxt2img eval_resultr+r+r,evaluatesh                   z$ImageToTextRetrievalTrainer.evaluatec Csdd}t|jd}t|D]L\}}td|||t|ddd} td| || ||||<q dtt|d kdt|} dtt|d kdt|} dtt|d kdt|} t|jd}t|D]L\}}td |||t|ddd} td| || ||||<qdtt|d kdt|} dtt|d kdt|}dtt|d kdt|}| | | d }| ||d }||d}| | | d }| | | || |||||d }|S)a report metrics for image-text matching Args: scores_i2t: image-to-text similarity score matrix scores_t2i: text-to-image similarity score matrix img2txt: image-to-text ground truth mapping txt2img: text-to-image ground truth mapping Returns: eval_result: A dictionary containing r1, r5, r10 scores cSsft|tr t||kddSt|ts2Jdd}|D]&}t||kdd}||kr:|}q:|S)Nrz:img2txt or txt2img should be list[int] or list[list[int]]!g@xD)rarbnpwhererZ)ranks ground_truthrankitmpr+r+r,get_lowest_from_rankss zJImageToTextRetrievalTrainer._report_metrics..get_lowest_from_ranksrzimg2txt[{}]: {}Nzmost similarity:gY@r ztxt2img[{}]: {}r) Ztxt_r1Ztxt_r5Ztxt_r10Z txt_r_meanZimg_r1Zimg_r5Zimg_r10Z img_r_meanr_mean agg_metrics) rzerosshape enumerateprintformatargsortlenr)r(Z scores_i2tZ scores_t2ir|r}rrindexscoreindstr1Ztr5Ztr10Zir1Zir5Zir10Ztr_meanZir_meanrrr~r+r+r,r{sF   """ """  z+ImageToTextRetrievalTrainer._report_metrics)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__rWr&rrdictr rrr r r rrerr{ __classcell__r+r+r)r,r#.s6 j Rr#):rrctypingrrrrnumpyr mindsporerMrrUmindspore.train.modelrmindspore.nn.wrap.loss_scaler mindspore.nnr r Zmindspore.trainr mindformers.datasetr rrmindformers.modelsrrmindformers.core.callbackrZmindformers.core.lrrZmindformers.core.optimrmindformers.wrapperrZ mindformers.trainer.base_trainerrZmindformers.trainer.utilsrmindformers.tools.loggerrmindformers.tools.utilsrmindformers.tools.registerrrZ eval_utilsrr config_argsr base_trainerutilsregisterTRAINERr#r+r+r+r,s6                pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/eval_utils.py000066400000000000000000000117021476650141400315320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer Utils.""" import numpy as np from mindspore import Tensor from mindformers.tools.logger import logger from mindformers.generation.utils import topk from mindformers.dataset.dataloader.multi_image_cap_dataloader import MultiImgCapDataLoader def extract_image_text_mapping(eval_dataloader, score_i2t, score_t2i): """extract_image_text_mapping from eval_dataloader. Args: eval_dataloader: evaluation dataloader score_i2t, score_t2i: two score matrix (I2T, T2I) Returns: img2txt, txt2img: ground truth image-text mapping """ dataloader = eval_dataloader while hasattr(dataloader, "children") and dataloader.children is not None: dataloader = dataloader.children[0] if isinstance(dataloader, MultiImgCapDataLoader): dataset = dataloader.source return dataset.img2txt, dataset.txt2img logger.warning("expect the eval dataset to be generate \ from MultiImgCapDataLoader, but is %s, will generate \ image-text mapping with accumulate indexes by default.", type(dataloader)) assert (score_i2t.shape[0], score_i2t.shape[1]) == (score_t2i.shape[1], score_t2i.shape[0]) image_num = score_i2t.shape[0] text_num = score_t2i.shape[0] if image_num == text_num: inds = [i for i in range(image_num)] return inds, inds bigger = image_num if image_num > text_num else text_num smaller = text_num if bigger == image_num else image_num factor = bigger // smaller smaller_inds = [[min(i + k * factor, bigger) for i in range(factor)] for k in range(smaller)] bigger_inds = [] for i in range(smaller - 1): bigger_inds += [i] * factor last_num = bigger - (smaller - 1) * factor bigger_inds += [smaller - 1] * last_num if bigger == image_num: return bigger_inds, smaller_inds return smaller_inds, bigger_inds def compute_itm_scores(network, eval_inputs, k_test=128, add_extra_itm_score=False): """ compute image-text matching scores, in matrix format. Args: network (BaseModel): network for evaluate ITM eval_inputs (tuple): inputs for evaluate itm scores. k_test (int, optional): k_test num, Defaults to 128. add_extra_itm_score (bool, optional): whether to add extra scores (model decides), Defaults to False. Returns: score_matrix_i2t, score_matrix_t2i: two score matrix (I2T, T2I) """ logger.info("========= k_text num: %d =========", k_test) sims_matrix = [] image_feats, text_feats, extra_args = eval_inputs[0], eval_inputs[1], eval_inputs[2:] image_feats = image_feats.asnumpy() text_feats = text_feats.asnumpy() for image_feat in image_feats: print(image_feat.shape, image_feat.dtype, text_feats.T.shape, text_feats.dtype) sim_q2t = np.matmul(image_feat, text_feats.T) sim_i2t = sim_q2t.max(0) sims_matrix.append(sim_i2t) sims_matrix = np.stack(sims_matrix, axis=0) score_matrix_i2t = np.full( (image_feats.shape[0], text_feats.shape[0]), -100.0 ) for i, sims in enumerate(sims_matrix): if i % 50 == 0: print(f"I2T: {i}/{sims_matrix.shape[0]} - sims.shape: {sims.shape}") topk_sim, topk_idx = topk(sims.T, k_test) topk_sim = topk_sim.T topk_idx = topk_idx.T score_matrix_i2t[i, topk_idx] = topk_sim # .astype(mstype.float32) if add_extra_itm_score: topk_idx_ms = Tensor.from_numpy(topk_idx) score = network.compute_extra_itm(extra_args, i, k_test, topk_idx_ms, i2t=True) score_matrix_i2t[i, topk_idx] += score.asnumpy() sims_matrix = sims_matrix.T score_matrix_t2i = np.full( (text_feats.shape[0], image_feats.shape[0]), -100.0 ) for i, sims in enumerate(sims_matrix): if i % 50 == 0: print(f"T2I: {i}/{sims_matrix.shape[0]} - sims.shape: {sims.shape}") topk_sim, topk_idx = topk(sims.T, k_test) topk_sim = topk_sim.T topk_idx = topk_idx.T score_matrix_t2i[i, topk_idx] = topk_sim if add_extra_itm_score: topk_idx_ms = Tensor.from_numpy(topk_idx) score = network.compute_extra_itm(extra_args, i, k_test, topk_idx_ms, i2t=False) score_matrix_t2i[i, topk_idx] += score.asnumpy() return score_matrix_i2t, score_matrix_t2i pengcheng.mind-mindformer/mindformers/trainer/image_to_text_retrieval/image_to_text_retrieval.py000066400000000000000000000313421476650141400342720ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Image-to-text Retrieval Trainer.""" import os from typing import List, Optional, Union from pprint import pprint import numpy as np import mindspore as ms from mindspore import dtype as mstype from mindspore.train.model import Model from mindspore.nn.wrap.loss_scale import DynamicLossScaleUpdateCell from mindspore.nn import TrainOneStepCell, Optimizer from mindspore.train import Callback from mindformers.dataset import build_dataset, check_dataset_config, BaseDataset from mindformers.models import build_model, BaseModel from mindformers.core.callback import build_callback from mindformers.core.lr import build_lr from mindformers.core.optim import build_optim from mindformers.wrapper import build_wrapper from mindformers.trainer.base_trainer import BaseTrainer from mindformers.trainer.utils import check_runner_config from mindformers.tools.logger import logger from mindformers.tools.utils import count_params from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from .eval_utils import compute_itm_scores, extract_image_text_mapping from ..config_args import ConfigArguments from ..base_trainer import BaseTrainer from ..utils import check_runner_config @MindFormerRegister.register(MindFormerModuleType.TRAINER, alias="image_to_text_retrieval") class ImageToTextRetrievalTrainer(BaseTrainer): """Image-to-text Retrieval Trainer.""" def __init__(self, model_name: str = None): super(ImageToTextRetrievalTrainer, self).__init__("image_to_text_retrieval", model_name) self.model_name = model_name self.kwargs = None def train(self, config: Optional[Union[dict, ConfigArguments]] = None, network: Optional[Union[str, BaseModel]] = None, dataset: Optional[Union[str, BaseDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """ Training task for ImageToTextRetrievalTrainer Trainer. """ # check mindspore version # currently, filip only support training under mindspore2.0 if not ms.__version__.startswith('2.0'): raise NotImplementedError(f"Currently, filip only support training under mindspore2.0, " f"but with mindspore {ms.__version__}") self.kwargs = kwargs # no_weight_decay_params filter def decay_filter(param_name): no_decay_params = config.no_decay_params for keyword in no_decay_params: if keyword in param_name.name: return False return True # build dataset logger.info(".........Build Dataset..........") check_dataset_config(config) if dataset is None: dataset = build_dataset(config.train_dataset_task) check_runner_config(config, dataset) # build network logger.info(".........Build Net..........") if network is None: network = build_model(config.model, default_args={ "parallel_config": config.parallel_config, "moe_config": config.moe_config}) network = network.to_float(mstype.float16) logger.info("网络参数量：%s M.", str(count_params(network))) # build optimizer logger.info(".........Build Optimizer..........") if optimizer is None: # build learning rate schedule logger.info(".........Build LR Schedule..........") lr_schedule = build_lr(config.lr_schedule) params = network.trainable_params() decay_params = list(filter(decay_filter, params)) other_params = list(filter(lambda x: not decay_filter(x), params)) group_params = [{'params': decay_params, 'weight_decay': config.optimizer.weight_decay}, {'params': other_params}, {'order_params': params}] if lr_schedule is not None: optimizer = build_optim( config.optimizer, default_args={"params": group_params, "learning_rate": lr_schedule}) else: assert config.optimizer.learning_rate, "learning_rate must be input" optimizer = build_optim( config.optimizer, default_args={"params": group_params}) # build callback if callbacks is None: callbacks = [] if config.profile: callbacks.append(config.profile_cb) callbacks.extend(build_callback( config.callbacks, default_args={"learning_rate": optimizer.learning_rate})) # build runner wrapper logger.info(".........Build Running Wrapper..........") update_cell = DynamicLossScaleUpdateCell(loss_scale_value=2. ** 20, scale_factor=2, scale_window=1000) if wrapper is None: model = build_wrapper(config.runner_wrapper, default_args={"network": network, "optimizer": optimizer, "scale_update_cell": update_cell}) elif isinstance(wrapper, TrainOneStepCell): model = wrapper else: raise NotImplementedError(f"Now not support this wrapper," f"it should be TrainOneStepCell type, but get {wrapper}") # define Model and begin training logger.info(".........Starting Init Train Model..........") model = Model(model) if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) model.train( config.runner_config.epochs, dataset, callbacks=callbacks, dataset_sink_mode=config.runner_config.sink_mode, sink_size=config.runner_config.sink_size) def evaluate(self, config: Optional[Union[dict, ConfigArguments]] = None, network: Optional[Union[str, BaseModel]] = None, dataset: Optional[Union[str, BaseDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): """ Evaluation task for ImageToTextRetrievalTrainer Trainer. """ self.kwargs = kwargs is_full_config = kwargs.get("is_full_config", False) config = self.set_config(config, is_full_config) # build dataset logger.info(".........Build Dataset..........") check_dataset_config(config) if dataset is None: dataset = build_dataset(config.eval_dataset_task) logger.info("Create eval dataset finish, dataset size:%d", dataset.get_dataset_size()) # build network logger.info(".........Build Net..........") if network is None: network = build_model(config.model, default_args={ "parallel_config": config.parallel_config, "moe_config": config.moe_config, "is_training": False}) network = network.to_float(mstype.float16) # checkpoint logger.info(".........Loading Checkpoint..........") if config.model.model_config.checkpoint_name_or_path is not None: network.load_checkpoint(config.model.model_config) logger.info("Network Parameters: %s M.", str(count_params(network))) # build callback logger.info(".........Build Callbacks for Evaluate..........") if callbacks is None: callbacks = [] if config.profile: callbacks.append(config.profile_cb) callbacks.extend(build_callback(config.eval_callbacks)) logger.info(".........Starting Evaling Model..........") if int(os.getenv("RANK_ID", '0')) % 8 == 0: pprint(config) # prepare inputs for computing simliarity matrix eval_inputs = network.prepare_inputs_for_itm_eval(dataset) # k_test value, for topk k_test = config.eval_dataset.k_test if \ config.eval_dataset.k_test is not None else 128 # trainer arguments overrides top_k k_test = kwargs.pop('k_test', k_test) # whether adding additional itm score add_extra_itm_score = config.eval_dataset.add_extra_itm_score # trainer arguments overrides add_extra_itm_score add_extra_itm_score = kwargs.pop('add_extra_itm_score', add_extra_itm_score) # compute image-to-text/text-to-image similarity scores score_i2t, score_t2i = compute_itm_scores(network, eval_inputs, k_test=k_test, add_extra_itm_score=add_extra_itm_score) # ground-truth image-text mapping img2txt, txt2img = extract_image_text_mapping(config.eval_dataset, score_i2t, score_t2i) # report evaluation results eval_result = self._report_metrics( score_i2t, score_t2i, img2txt, txt2img, ) logger.info(eval_result) logger.info(".........Evaluate Over!.............") return eval_result def _report_metrics(self, scores_i2t, scores_t2i, img2txt, txt2img): """ report metrics for image-text matching Args: scores_i2t: image-to-text similarity score matrix scores_t2i: text-to-image similarity score matrix img2txt: image-to-text ground truth mapping txt2img: text-to-image ground truth mapping Returns: eval_result: A dictionary containing r1, r5, r10 scores """ def get_lowest_from_ranks(ranks, ground_truth): if isinstance(ground_truth, int): return np.where(ranks == ground_truth)[0][0] assert isinstance(ground_truth, list), "img2txt or txt2img should be list[int] or list[list[int]]!" rank = 1e20 for i in ground_truth: tmp = np.where(ranks == i)[0][0] if tmp < rank: rank = tmp return rank # Images->Text ranks = np.zeros(scores_i2t.shape[0]) for index, score in enumerate(scores_i2t): print("img2txt[{}]: {}".format(index, img2txt[index])) inds = np.argsort(score)[::-1] print("most similarity:", inds) # Score ranks[index] = get_lowest_from_ranks(inds, img2txt[index]) # 计算度量, 100 是百分比基数, # ranks为每张图ground-truth对应的 # text在similarity score中排位的最小值。 tr1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks) tr5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks) tr10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks) # Text->Images ranks = np.zeros(scores_t2i.shape[0]) for index, score in enumerate(scores_t2i): print("txt2img[{}]: {}".format(index, txt2img[index])) inds = np.argsort(score)[::-1] print("most similarity:", inds) ranks[index] = get_lowest_from_ranks(inds, txt2img[index]) # 此段逻辑同上注释，以text为基准。 ir1 = 100.0 * len(np.where(ranks < 1)[0]) / len(ranks) ir5 = 100.0 * len(np.where(ranks < 5)[0]) / len(ranks) ir10 = 100.0 * len(np.where(ranks < 10)[0]) / len(ranks) tr_mean = (tr1 + tr5 + tr10) / 3 ir_mean = (ir1 + ir5 + ir10) / 3 r_mean = (tr_mean + ir_mean) / 2 agg_metrics = (tr1 + tr5 + tr10) / 3 eval_result = { "txt_r1": tr1, "txt_r5": tr5, "txt_r10": tr10, "txt_r_mean": tr_mean, "img_r1": ir1, "img_r5": ir5, "img_r10": ir10, "img_r_mean": ir_mean, "r_mean": r_mean, "agg_metrics": agg_metrics, } return eval_result pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/000077500000000000000000000000001476650141400263675ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/__init__.py000066400000000000000000000014621476650141400305030ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from .masked_image_modeling_pretrain import MaskedImageModelingTrainer __all__ = ['MaskedImageModelingTrainer'] pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/__pycache__/000077500000000000000000000000001476650141400305775ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005161476650141400347120ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/__pycache__a ?cf2@sdZddlmZdgZdS)zMasked Image Modeling Trainer.)MaskedImageModelingTrainerrN)__doc__Zmasked_image_modeling_pretrainr__all__rr]/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_image_modeling/__init__.pys masked_image_modeling_pretrain.cpython-39.pyc000066400000000000000000000207631476650141400413510ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/__pycache__a ?cf$@sdZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZdd lmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&e'ej(Gddde Z)dS)zMasked Image Modeling Trainer.)OptionalListUnionN)Image)Tensor)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainer)BaseImageProcessor)logger) load_imagec seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeejeeefeee e feed d d ZZS)MaskedImageModelingTrainera MaskedImageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTMAEForPreTraining, ViTMAEConfig >>> class MyDataLoader: ... def __init__(self): ... self.image = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self.mask = [np.zeros((196,), np.int32) for _ in range(64)] ... self.ids_restore = [np.zeros((196,), np.int32) for _ in range(64)] ... self.unmask_index = [np.zeros((49,), np.int32) for _ in range(64)] ... def __getitem__(self, index): ... return self.image[index], self.mask[index], self.ids_restore[index], self.unmask_index[index] ... def __len__(self): ... return len(self.image) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), ... column_names=["image", "mask", "ids_restore", "unmask_index"]).batch(2) >>> #1) use config to train >>> mae_trainer = MaskedImageModelingTrainer(model_name='mae_vit_base_p16') >>> mae_trainer.train(dataset=train_dataset) >>> #2) use instance function to train >>> mae_config = ViTMAEConfig(batch_size=2) >>> network_with_loss = ViTMAEForPreTraining(mae_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> mae_trainer.train(wrapper=wrapper, optimizer=optimizer, dataset=train_dataset) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Nmasked_image_modeling)superr__init__)selfr __class__s/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_image_modeling/masked_image_modeling_pretrain.pyrNsz#MaskedImageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for MaskedImageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r!r"r&r#r$r%N)training_process)rr!r"r#r$r%r&kwargsrrr trainQs%z MaskedImageModelingTrainer.traincOs tddS)Nz7The MaskedImageModeling task does not support evaluate.)NotImplementedError)rargsr(rrr evaluatesz#MaskedImageModelingTrainer.evaluate)r! input_datar"image_processorcKstd|dur|j}t|ttjttt fs6t dg}t|trT| t |n(t|t rx|D]}| t |qbn|}|j f||d||d|S)aPredict task for MaskedImageModeling Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. z/.........Build Input Data For Predict..........NzTInput data's type must be one of [str, ms.Tensor, np.ndarray, PIL.Image.Image, list]r)r!r-taskr"r.)rinfor- isinstancernpndarrayrstrlist ValueErrorappendrpredict_process)rr!r-r"r.r(batch_input_data data_pathrrr predicts(   z"MaskedImageModelingTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r4rrrdictrrrr r r r rr rrr)r,rr2r3rr5rr; __classcell__rrrr r%s6' .r)*r?typingrrrnumpyr2Z PIL.Imager mindsporerZmindspore.trainr mindspore.nnrr r mindspore.datasetr mindformers.datasetr mindformers.modelsr mindformers.tools.registerrrrmindformers.trainer.config_argsrZ!mindformers.trainer.training_argsrZ mindformers.trainer.base_trainerrZ!mindformers.models.base_processorrmindformers.tools.loggerrZmindformers.tools.image_toolsrregisterTRAINERrrrrr s$             pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling/group_mim_parameters.py000066400000000000000000000042251476650141400331650ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/microsoft/SimMIM # ============================================================================ """group parameters for masked image modeling.""" from mindformers.tools import logger def get_group_parameters(config, model): """get pretrain param groups""" skip = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip = model.no_weight_decay() logger.info('No weight decay: %s', skip) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) has_decay = [] no_decay = [] has_decay_name = [] no_decay_name = [] for param in model.trainable_params(): if len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip) or \ check_keywords_in_name(param.name, skip_keywords): no_decay.append(param) no_decay_name.append(param.name) else: has_decay.append(param) has_decay_name.append(param.name) logger.info('No decay params: %s', no_decay_name) logger.info('Has decay params: %s', has_decay_name) return [{'params': has_decay, 'weight_decay': config.optimizer.weight_decay}, {'params': no_decay, 'weight_decay': 0.0}, {'order_params': model.trainable_params()}] def check_keywords_in_name(name, keywords=()): isin = False for keyword in keywords: if keyword in name: isin = True break return isin masked_image_modeling_pretrain.py000066400000000000000000000223731476650141400350610ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_image_modeling# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from typing import Optional, List, Union import numpy as np from PIL.Image import Image from mindspore import Tensor from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from mindformers.trainer.config_args import ConfigArguments from mindformers.trainer.training_args import TrainingArguments from mindformers.trainer.base_trainer import BaseTrainer from mindformers.models.base_processor import BaseImageProcessor from mindformers.tools.logger import logger from mindformers.tools.image_tools import load_image @MindFormerRegister.register(MindFormerModuleType.TRAINER) class MaskedImageModelingTrainer(BaseTrainer): r"""MaskedImageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import ViTMAEForPreTraining, ViTMAEConfig >>> class MyDataLoader: ... def __init__(self): ... self.image = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... self.mask = [np.zeros((196,), np.int32) for _ in range(64)] ... self.ids_restore = [np.zeros((196,), np.int32) for _ in range(64)] ... self.unmask_index = [np.zeros((49,), np.int32) for _ in range(64)] ... def __getitem__(self, index): ... return self.image[index], self.mask[index], self.ids_restore[index], self.unmask_index[index] ... def __len__(self): ... return len(self.image) >>> train_dataset = GeneratorDataset(source=MyDataLoader(), ... column_names=["image", "mask", "ids_restore", "unmask_index"]).batch(2) >>> #1) use config to train >>> mae_trainer = MaskedImageModelingTrainer(model_name='mae_vit_base_p16') >>> mae_trainer.train(dataset=train_dataset) >>> #2) use instance function to train >>> mae_config = ViTMAEConfig(batch_size=2) >>> network_with_loss = ViTMAEForPreTraining(mae_config) >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=network_with_loss.trainable_params()) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper = TrainOneStepWithLossScaleCell(network_with_loss, optimizer, scale_sense=loss_scale) >>> mae_trainer.train(wrapper=wrapper, optimizer=optimizer, dataset=train_dataset) Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(MaskedImageModelingTrainer, self).__init__("masked_image_modeling", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for MaskedImageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The MaskedImageModeling task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[Tensor, np.ndarray, Image, str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, image_processor: Optional[BaseImageProcessor] = None, **kwargs): r"""Predict task for MaskedImageModeling Trainer. This function is used to predict the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, tokenizer, image_processor, audio_processor. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (Tensor, np.ndarray, Image, str, list)): raise ValueError("Input data's type must be one of " "[str, ms.Tensor, np.ndarray, PIL.Image.Image, list]") batch_input_data = [] if isinstance(input_data, str): batch_input_data.append(load_image(input_data)) elif isinstance(input_data, list): for data_path in input_data: batch_input_data.append(load_image(data_path)) else: batch_input_data = input_data return self.predict_process(config=config, input_data=batch_input_data, task='masked_image_modeling', network=network, image_processor=image_processor, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling/000077500000000000000000000000001476650141400270705ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling/__init__.py000066400000000000000000000014761476650141400312110ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .masked_language_modeling_pretrain import MaskedLanguageModelingTrainer __all__ = ['MaskedLanguageModelingTrainer'] pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling/__pycache__/000077500000000000000000000000001476650141400313005ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005321476650141400354110ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling/__pycache__a ?cf>@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)MaskedLanguageModelingTrainerrN)__doc__Z!masked_language_modeling_pretrainr__all__rr`/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_language_modeling/__init__.pys masked_language_modeling_pretrain.cpython-39.pyc000066400000000000000000000157311476650141400425520ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling/__pycache__a ?cf@sdZddlmZmZmZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZddlmZmZmZd d lmZd d lmZd d lmZeejGd ddeZdS)zMasked Image Modeling Trainer.)OptionalListUnion)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizer)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeefeeee feed d d ZZS)MaskedLanguageModelingTrainera]MaskedLanguageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import MaskedLanguageModelingTrainer >>> def generator(): >>> data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) >>> input_mask = np.ones_like(data) >>> token_type_id = np.zeros_like(data) >>> next_sentence_lables = np.array([1]).astype(np.int32) >>> masked_lm_positions = np.array([1, 2]).astype(np.int32) >>> masked_lm_ids = np.array([1, 2]).astype(np.int32) >>> masked_lm_weights = np.ones_like(masked_lm_ids) >>> train_data = (data, input_mask, token_type_id, next_sentence_lables, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> for _ in range(512): ... yield train_data >>> dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", ... "next_sentence_labels", "masked_lm_positions", ... "masked_lm_ids", "masked_lm_weights"]) >>> dataset = dataset.batch(batch_size=16) >>> mlm_trainer = MaskedLanguageModelingTrainer(model_name="bert_tiny_uncased") >>> mlm_trainer.train(dataset=dataset) >>> res = mlm_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)N fill_mask)superr__init__)selfr __class__y/home/ma-user/work/mindformers-baichuan/mindformers/trainer/masked_language_modeling/masked_language_modeling_pretrain.pyr>sz&MaskedLanguageModelingTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for MaskedLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rrr#r r!r"N)training_process)rrrr r!r"r#kwargsrrrtrainAs%z#MaskedLanguageModelingTrainer.traincOs tddS)Nz:The MaskedLanguageModeling task does not support evaluate.)NotImplementedError)rargsr%rrrevaluateosz&MaskedLanguageModelingTrainer.evaluate)r input_datar tokenizerc Ks||}d|jj_|dur"|j}t|ttfsBtdt |d|jj_ |rX|j j n|jjj }|j f||d|||dd|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[str, BaseModel]]): The network for trainer. It support model name supported or BaseModel class. Supported model name can refer to model support list. For . Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. FNz;Input data's type must be one of [str, list], but got type r max_length)rr*taskrr+r-padding) set_configmodel model_config is_trainingr* isinstancestrlist ValueErrortype batch_sizer seq_lengthpredict_process)rrr*rr+r%r-rrrpredictss(   z%MaskedLanguageModelingTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r5rrrdictrrrrr r r rrrrr&r)r6r r< __classcell__rrrrr s6 .rN)r@typingrrrZmindspore.trainr mindspore.nnrrrmindspore.datasetr mindformers.datasetr mindformers.modelsr r mindformers.tools.registerr rr config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs       masked_language_modeling_pretrain.py000066400000000000000000000173401476650141400362610ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/masked_language_modeling# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Image Modeling Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class MaskedLanguageModelingTrainer(BaseTrainer): r"""MaskedLanguageModeling Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers import MaskedLanguageModelingTrainer >>> def generator(): >>> data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) >>> input_mask = np.ones_like(data) >>> token_type_id = np.zeros_like(data) >>> next_sentence_lables = np.array([1]).astype(np.int32) >>> masked_lm_positions = np.array([1, 2]).astype(np.int32) >>> masked_lm_ids = np.array([1, 2]).astype(np.int32) >>> masked_lm_weights = np.ones_like(masked_lm_ids) >>> train_data = (data, input_mask, token_type_id, next_sentence_lables, ... masked_lm_positions, masked_lm_ids, masked_lm_weights) >>> for _ in range(512): ... yield train_data >>> dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", ... "next_sentence_labels", "masked_lm_positions", ... "masked_lm_ids", "masked_lm_weights"]) >>> dataset = dataset.batch(batch_size=16) >>> mlm_trainer = MaskedLanguageModelingTrainer(model_name="bert_tiny_uncased") >>> mlm_trainer.train(dataset=dataset) >>> res = mlm_trainer.predict(input_data = "hello world [MASK]") Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(MaskedLanguageModelingTrainer, self).__init__("fill_mask", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for MaskedLanguageModeling Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The MaskedLanguageModeling task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[str, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[str, BaseModel]]): The network for trainer. It support model name supported or BaseModel class. Supported model name can refer to model support list. For . Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) config.model.model_config.is_training = False if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of " f"[str, list], but got type {type(input_data)}") config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length return self.predict_process(config=config, input_data=input_data, task='fill_mask', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", **kwargs) pengcheng.mind-mindformer/mindformers/trainer/optimizer_grouped_parameters.py000066400000000000000000000173541476650141400304610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Grouped parameters of optimizer.""" import json from functools import partial from typing import Optional import copy from mindspore.nn import Cell from mindspore.nn.learning_rate_schedule import LearningRateSchedule from mindformers.models import BaseModel from mindformers.core.lr import LearningRateWiseLayer from mindformers.core import build_lr from mindformers.tools import logger from .utils import check_keywords_in_name def get_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, dynamic_lr_schedule: Optional[LearningRateSchedule] = None, layer_scale: bool = False, layer_decay: float = 1.0): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") skip_params = {} skip_keywords = {} if hasattr(model, 'no_weight_decay'): skip_params = model.no_weight_decay() logger.info('No weight decay: %s', skip_params) if hasattr(model, 'no_weight_decay_keywords'): skip_keywords = model.no_weight_decay_keywords() logger.info('No weight decay keywords: %s', skip_keywords) decay_parameters_names = [ param.name for param in model.trainable_params() if not (len(param.shape) == 1 or param.name.endswith(".bias") or (param.name in skip_params) or check_keywords_in_name(param.name, skip_keywords)) ] get_layer_id_func = None scales_list = [] if dynamic_lr_schedule is not None: if layer_scale: if model.__class__.__name__ == 'SwinForImageClassification': depths = model.config.depths num_layers = sum(depths) get_layer_id_func = partial(get_swin_layer, num_layers=num_layers + 2, depths=depths) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) elif model.__class__.__name__ == 'ViTForImageClassification': num_layers = model.config.depth get_layer_id_func = partial(get_vit_layer, num_layers=num_layers + 2) scales_list = list(layer_decay ** i for i in reversed(range(num_layers + 2))) else: logger.warning("Now only support VitModel and SwinModel," "if use dynamic_lr_schedule and layer_scale, they will be reset and invalid.") layer_scale = False dynamic_lr_schedule = None else: logger.warning("dynamic_lr_schedule will be reset and invalid when layer_scale is False.") dynamic_lr_schedule = None parameter_group_names = {} parameter_group_vars = {} scale = 1. for param in model.trainable_params(): if param.name in decay_parameters_names: group_name = 'decay' else: group_name = 'no_decay' weight_decay = 0. if get_layer_id_func is not None: layer_id = get_layer_id_func(param.name) group_name = "layer_%d_%s" % (layer_id, group_name) if scales_list is not None: scale = scales_list[layer_id] if group_name not in parameter_group_names: parameter_group_names[group_name] = { "weight_decay": weight_decay, "params": [], } parameter_group_vars[group_name] = { "weight_decay": weight_decay, "params": [], } if isinstance(dynamic_lr_schedule, LearningRateSchedule): if layer_scale: parameter_group_vars[group_name]["lr"] = LearningRateWiseLayer(dynamic_lr_schedule, scale) else: parameter_group_vars[group_name]["lr"] = dynamic_lr_schedule parameter_group_vars[group_name]["params"].append(param) parameter_group_names[group_name]["params"].append(param.name) param_groups = json.dumps(parameter_group_names, indent=2) logger.info("Param groups = %s", param_groups) return list(parameter_group_vars.values()) def get_mup_optimizer_grouped_parameters(model: Optional[BaseModel] = None, weight_decay: float = 0.0, mup_ratio: float = 0.0, lr_schedule_config = None): """Get grouped parameters of the network for training.""" if not isinstance(model, (Cell, BaseModel)): raise TypeError(f"model type should be BaseModel, but get {type(model)}") def find_in_name(name, key_list): for k in key_list: if name.find(k) >= 0: return True return False middle_weight_key_list = [ "attention.wo", "attention.wq", "attention.wk", "attention.wv", "feed_forward", "lm_head" ] parameter_group_vars = {} middle_weight_group = {"params": []} # mlp and attention for param in model.trainable_params(): if find_in_name(param.name, middle_weight_key_list): middle_weight_group["params"].append(param) middle_lr_config = copy.deepcopy(lr_schedule_config) middle_lr_config.learning_rate = middle_lr_config.learning_rate / mup_ratio middle_lr_config.lr_middle = middle_lr_config.lr_middle / mup_ratio middle_lr_config.lr_end = middle_lr_config.lr_end / mup_ratio middle_lr = build_lr(middle_lr_config) middle_weight_group["lr"] = middle_lr middle_weight_group["weight_decay"] = weight_decay parameter_group_vars["middle_output"] = middle_weight_group norm_key_list = [ "norm", "embedding" ] norm_weight_group = {"params": []} for param in model.trainable_params(): if find_in_name(param.name, norm_key_list): norm_weight_group["params"].append(param) norm_weight_group["weight_decay"] = 0.0 parameter_group_vars["norm_key"] = norm_weight_group return list(parameter_group_vars.values()) def get_vit_layer(name, num_layers): """get vit layer""" if name.endswith(("cls_tokens", "mask_tokens", "pos_embed")): layer_num = 0 elif name.startswith("vit.patch_embed"): layer_num = 0 elif name.startswith("vit.blocks"): layer_id = int(name.split('.')[2]) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num def get_swin_layer(name, num_layers, depths): """get swin layer""" if name in ("encoder.mask_token",): layer_num = 0 elif name.startswith("encoder.patch_embed"): layer_num = 0 elif name.startswith("encoder.layers"): layer_id = int(name.split('.')[2]) block_id = name.split('.')[4] if block_id in ('reduction', 'norm'): layer_num = sum(depths[:layer_id + 1]) else: layer_id = sum(depths[:layer_id]) + int(block_id) layer_num = layer_id + 1 else: layer_num = num_layers - 1 return layer_num pengcheng.mind-mindformer/mindformers/trainer/question_answering/000077500000000000000000000000001476650141400260275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/question_answering/__init__.py000066400000000000000000000014371476650141400301450ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Question Answering Trainer.""" from .question_answering import QuestionAnsweringTrainer __all__ = ['QuestionAnsweringTrainer'] pengcheng.mind-mindformer/mindformers/trainer/question_answering/__pycache__/000077500000000000000000000000001476650141400302375ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/question_answering/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004721476650141400344320ustar00rootroot00000000000000a ?cf@sdZddlmZdgZdS)zQuestion Answering Trainer.)QuestionAnsweringTrainerrN)__doc__question_answeringr__all__rrZ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/question_answering/__init__.pys question_answering.cpython-39.pyc000066400000000000000000000177411476650141400365470ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/question_answering/__pycache__a ?cf$@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZdgZeej GdddeZ!dS)z Question Answer Trainer. )OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArgumentsQuestionAnsweringTrainerc s$eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd fd d Zdeeee e e feeeefeee e feed d dZZS)raQuestionAnswering Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForQuestionAnswering, BertConfig >>> config = MindFormerConfig("configs/qa/run_qa_bert_base_uncased.yaml") >>> #1) use config to train >>> cls_task = QuestionAnsweringTrainer(model_name='qa_bert_base_uncased') >>> cls_task.train(config=config) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Nquestion_answering)superr__init__)selfr __class__d/home/ma-user/work/mindformers-baichuan/mindformers/trainer/question_answering/question_answering.pyr7sz!QuestionAnsweringTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rr r$r!r"r#N)training_process)rrr r!r"r#r$kwargsrrrtrain:s%zQuestionAnsweringTrainer.train)rr r!r$compute_metricsc s2d}|d|tjf|||||d|dS)aEvaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z QA Metric metric_name)rr r!r(r$N) setdefaultrevaluate_process)rrr r!r$r(r&r)rrrevaluatehs z!QuestionAnsweringTrainer.evaluate)r input_datar tokenizercKstd|dur$|jr |j}nd}t|ttfs:tdt|tr`|D]}t|tsHtdqHd|jj_ |j f||d||d|S) ai Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz;My name is Wolfgang and I live in Berlin - Where do I live?z,Input data's type must be one of [str, list]z*The element of input data list must be strr)rr-taskr r.) r infor- isinstancestrlist ValueErrormodel model_config batch_sizepredict_process)rrr-r r.r&itemrrrpredicts(     z QuestionAnsweringTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r3rrrdictrrrr r r rrrrrr'setr,r4r r; __classcell__rrrrr"sL /+N)"r?typingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr base_trainerr config_argsr training_argsr__all__registerTRAINERrrrrrs        pengcheng.mind-mindformer/mindformers/trainer/question_answering/question_answering.py000066400000000000000000000223461476650141400323340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Question Answer Trainer. """ from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments __all__ = ['QuestionAnsweringTrainer'] @MindFormerRegister.register(MindFormerModuleType.TRAINER) class QuestionAnsweringTrainer(BaseTrainer): r"""QuestionAnswering Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import GeneralTaskTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForQuestionAnswering, BertConfig >>> config = MindFormerConfig("configs/qa/run_qa_bert_base_uncased.yaml") >>> #1) use config to train >>> cls_task = QuestionAnsweringTrainer(model_name='qa_bert_base_uncased') >>> cls_task.train(config=config) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(QuestionAnsweringTrainer, self).__init__("question_answering", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "QA Metric" kwargs.setdefault("metric_name", metric_name) super().evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ logger.info(".........Build Input Data For Predict..........") if input_data is None: if config.input_data: input_data = config.input_data else: input_data = "My name is Wolfgang and I live in Berlin - Where do I live?" if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # This is a known issue, you need to specify batch size equal to 1 when creating bert model. config.model.model_config.batch_size = 1 return self.predict_process(config=config, input_data=input_data, task='question_answering', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/text_classfication/000077500000000000000000000000001476650141400257715ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/text_classfication/__init__.py000066400000000000000000000014431476650141400301040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Trainer.""" from .text_classification import TextClassificationTrainer __all__ = ['TextClassificationTrainer'] pengcheng.mind-mindformer/mindformers/trainer/text_classfication/__pycache__/000077500000000000000000000000001476650141400302015ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/text_classfication/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004751476650141400343770ustar00rootroot00000000000000a ?cf#@sdZddlmZdgZdS)zText Classification Trainer.)TextClassificationTrainerrN)__doc__text_classificationr__all__rrZ/home/ma-user/work/mindformers-baichuan/mindformers/trainer/text_classfication/__init__.pys text_classification.cpython-39.pyc000066400000000000000000000204331476650141400366140ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/text_classfication/__pycache__a ?cf&@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZeejGdddeZ dS)zText Classification Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)ConfigArguments)TrainingArguments) BaseTrainerc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee eeeefeeeeefeeeefd d d Zdeeee e e feeeefeee eed d dZZS)TextClassificationTraineraTextClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TextClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForMultipleChoice, BertConfig >>> config = MindFormerConfig("configs/txtcls/run_txtcls_bert_base_uncased.yaml") >>> #1) use default config to train >>> txtcls_task = TextClassificationTrainer(model_name='bert_for_multiple_choice') >>> txtcls_task.train(config=config) >>> txtcls_task.evaluate(config=config) >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> res = txtcls_task.predict(input_data=input_data) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntext_classification)superr__init__)selfr __class__e/home/ma-user/work/mindformers-baichuan/mindformers/trainer/text_classfication/text_classification.pyr9sz"TextClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TextClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rr r$r!r"r#N)training_process)rrr r!r"r#r$kwargsrrrtrain<s%zTextClassificationTrainer.train)rr r!r$compute_metricscKs0d}|d||jf|||||d|dS)aEvaluate task for TextClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Top1 Accuracy metric_name)rr r!r(r$N) setdefaultevaluate_process)rrr r!r$r(r&r)rrrevaluatejs z"TextClassificationTrainer.evaluate)r input_datar tokenizerc Ks||}td|dur"|j}t|ttfs8tdt|tr^|D]}t|tsFtdqFd|jj _ |rt|j j n|jj j }|j f||d|||dd|S) ak Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz,Input data's type must be one of [str, list]z*The element of input data list must be strr max_length)rr-taskr r.r0padding) set_configr infor- isinstancestrlist ValueErrormodel model_config batch_sizer seq_lengthpredict_process)rrr-r r.r&itemr0rrrpredicts,      z!TextClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r6rrrdictrrrr r r rrrrrr'setr,r7r r? __classcell__rrrrr sL /  + rN)!rCtypingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr config_argsr training_argsr base_trainerrregisterTRAINERrrrrrs        pengcheng.mind-mindformer/mindformers/trainer/text_classfication/text_classification.py000066400000000000000000000232621476650141400324070ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Text Classification Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ..base_trainer import BaseTrainer @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TextClassificationTrainer(BaseTrainer): r"""TextClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TextClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForMultipleChoice, BertConfig >>> config = MindFormerConfig("configs/txtcls/run_txtcls_bert_base_uncased.yaml") >>> #1) use default config to train >>> txtcls_task = TextClassificationTrainer(model_name='bert_for_multiple_choice') >>> txtcls_task.train(config=config) >>> txtcls_task.evaluate(config=config) >>> input_data = ["The new rights are nice enough-Everyone really likes the newest benefits ", ... "i don't know um do you do a lot of camping-I know exactly."] >>> res = txtcls_task.predict(input_data=input_data) >>> #2) use instance function to train Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TextClassificationTrainer, self).__init__("text_classification", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TextClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TextClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Top1 Accuracy" kwargs.setdefault("metric_name", metric_name) self.evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # bert模型已知issue，由于bert模型在创建的时候需要batch_size参数， # 同时pipeline是一个样本一个样本进行处理，所以这里设定为1 config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length return self.predict_process(config=config, input_data=input_data, task='text_classification', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", **kwargs) pengcheng.mind-mindformer/mindformers/trainer/token_classification/000077500000000000000000000000001476650141400262765ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/token_classification/__init__.py000066400000000000000000000014471476650141400304150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token Classification Trainer.""" from .token_classification import TokenClassificationTrainer __all__ = ['TokenClassificationTrainer'] pengcheng.mind-mindformer/mindformers/trainer/token_classification/__pycache__/000077500000000000000000000000001476650141400305065ustar00rootroot00000000000000__init__.cpython-39.pyc000066400000000000000000000005021476650141400346140ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/token_classification/__pycache__a ?cf'@sdZddlmZdgZdS)zToken Classification Trainer.)TokenClassificationTrainerrN)__doc__token_classificationr__all__rr\/home/ma-user/work/mindformers-baichuan/mindformers/trainer/token_classification/__init__.pys token_classification.cpython-39.pyc000066400000000000000000000214661476650141400372640ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/token_classification/__pycache__a ?cf (@sdZddlmZmZmZddlmZddlmZddl m Z m Z m Z ddl mZddlmZmZddlmZdd lmZmZmZd d lmZd d lmZd d lmZddlmZe ej!GdddeZ"dS)zToken Classification Trainer.)OptionalListUnion)Callback)GeneratorDataset)TrainOneStepCell OptimizerCell) BaseDataset) BaseModel BaseTokenizer)logger)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArguments)cluener_labelsc s$eZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZdeeee e e feee e feeeefeeeeefeeeefd fd d Zdeeee e e feeeefeee e feed d dZZS)TokenClassificationTraineruTokenClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TokenClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForTokenClassification, BertConfig >>> config = MindFormerConfig("configs/tokcls/run_tokcls_bert_base_chinese.yaml") >>> #1) use config to train or evaluate or predict >>> tokcls_task = TokenClassificationTrainer(model_name='tokcls_bert_base_chinese') >>> tokcls_task.train(config=config) >>> tokcls_task.evaluate(config=config) >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。", "的时间会去玩玩星际2。"] >>> res = tokcls_task.predict(input_data=input_data) >>> print(res) [[{'entity_group': 'address', 'word': '日内瓦', 'start': 6, 'end': 9}], [{'entity_group': 'game', 'word': '星际2', 'start': 7, 'end': 10}]] >>> #2) use instance function to train or evaluate or predict Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)Ntoken_classification)superr__init__)selfr __class__h/home/ma-user/work/mindformers-baichuan/mindformers/trainer/token_classification/token_classification.pyr<sz#TokenClassificationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )r!r"r&r#r$r%N)training_process)rr!r"r#r$r%r&kwargsrrr train?s%z TokenClassificationTrainer.train)r!r"r#r&compute_metricsc s2d}|d|tjf|||||d|dS)aEvaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. z Entity Metric metric_name)r!r"r#r*r&N) setdefaultrevaluate_process)rr!r"r#r&r*r(r+rrr evaluatems z#TokenClassificationTrainer.evaluate)r! input_datar" tokenizerc Ks||}td|dur"|j}t|ttfs8tdt|tr^|D]}t|tsFtdqFd|jj _ |rt|j j n|jj j }ddt tD}|jf||d|||d |d |S) ai Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. z/.........Build Input Data For Predict..........Nz,Input data's type must be one of [str, list]z*The element of input data list must be strcSsi|]\}}||qSrr).0label_idlabelrrr z6TokenClassificationTrainer.predict..r max_length)r!r/taskr"r0r7paddingid2label) set_configr infor/ isinstancestrlist ValueErrormodel model_config batch_sizer! seq_length enumeraterpredict_process) rr!r/r"r0r(itemr7r:rrr predicts0      z"TokenClassificationTrainer.predict)N)NNNNNN)NNNNN)NNNN)__name__ __module__ __qualname____doc__r>rrrdictrrrr r r rrrrrr)setr.r?r rH __classcell__rrrr r sL /+rN)#rLtypingrrrZmindspore.trainrmindspore.datasetr mindspore.nnrrr mindformers.datasetr mindformers.modelsr r mindformers.tools.loggerr mindformers.tools.registerrrr base_trainerr config_argsr training_argsrdataset.labelsrregisterTRAINERrrrrr s         pengcheng.mind-mindformer/mindformers/trainer/token_classification/token_classification.py000066400000000000000000000240131476650141400330430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Token Classification Trainer.""" from typing import Optional, List, Union from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerRegister, \ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments from ...dataset.labels import cluener_labels @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TokenClassificationTrainer(BaseTrainer): r"""TokenClassification Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> from mindspore.nn import AdamWeightDecay, TrainOneStepCell >>> from mindformers.core.lr import build_lr >>> from mindformers.trainer import TokenClassificationTrainer >>> from mindformers.tools.register import MindFormerConfig >>> from mindformers.models import BertForTokenClassification, BertConfig >>> config = MindFormerConfig("configs/tokcls/run_tokcls_bert_base_chinese.yaml") >>> #1) use config to train or evaluate or predict >>> tokcls_task = TokenClassificationTrainer(model_name='tokcls_bert_base_chinese') >>> tokcls_task.train(config=config) >>> tokcls_task.evaluate(config=config) >>> input_data = ["表身刻有代表日内瓦钟表匠freresoltramare的“fo”字样。", "的时间会去玩玩星际2。"] >>> res = tokcls_task.predict(input_data=input_data) >>> print(res) [[{'entity_group': 'address', 'word': '日内瓦', 'start': 6, 'end': 9}], [{'entity_group': 'game', 'word': '星际2', 'start': 7, 'end': 10}]] >>> #2) use instance function to train or evaluate or predict Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TokenClassificationTrainer, self).__init__("token_classification", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for TokenClassification Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, **kwargs): r"""Evaluate task for TokenClassification Trainer. This function is used to evaluate the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, dataset, callbacks, compute_metrics. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The eval callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. """ metric_name = "Entity Metric" kwargs.setdefault("metric_name", metric_name) super().evaluate_process( config=config, network=network, dataset=dataset, compute_metrics=compute_metrics, callbacks=callbacks, **kwargs ) def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ config = self.set_config(config) logger.info(".........Build Input Data For Predict..........") if input_data is None: input_data = config.input_data if not isinstance(input_data, (str, list)): raise ValueError("Input data's type must be one of [str, list]") if isinstance(input_data, list): for item in input_data: if not isinstance(item, str): raise ValueError("The element of input data list must be str") # This is a known issue, you need to specify batch size equal to 1 when creating bert model. config.model.model_config.batch_size = 1 max_length = network.config.seq_length if network else config.model.model_config.seq_length id2label = {label_id: label for label_id, label in enumerate(cluener_labels)} return self.predict_process(config=config, input_data=input_data, task='token_classification', network=network, tokenizer=tokenizer, max_length=max_length, padding="max_length", id2label=id2label, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/trainer.py000066400000000000000000001504651476650141400241340ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer API For Import.""" import os import shutil from collections import OrderedDict from typing import List, Optional, Union import numpy as np from PIL.Image import Image import mindspore as ms from mindspore import Tensor from mindspore.common import set_seed from mindspore import load_checkpoint, load_param_into_net from mindspore.nn import TrainOneStepCell, Optimizer, Cell, \ PipelineCell, MicroBatchInterleaved from mindspore.nn.wrap.cell_wrapper import _VirtualDatasetCell from mindspore.train import Callback from mindspore.dataset import GeneratorDataset from mindspore.dataset.engine.datasets import BatchDataset, RepeatDataset from mindformers.core.parallel_config import build_parallel_config from mindformers.dataset import build_dataset, build_dataset_loader, \ check_dataset_config, BaseDataset from mindformers.mindformer_book import MindFormerBook from mindformers.models import build_model, BaseModel, BaseImageProcessor, \ BaseTokenizer, BaseAudioProcessor from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.register.config import ordered_yaml_dump from .build_trainer import build_trainer from .config_args import ConfigArguments from .training_args import TrainingArguments from .utils import check_train_data_loader_type, check_eval_data_loader_type, \ check_optimizer_and_lr_type, check_wrapper_config, config2dict __all__ = ['Trainer'] SUPPORT_TASKS = MindFormerBook().get_trainer_support_task_list() SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() SUPPORT_PIPELINES = MindFormerBook().get_pipeline_support_task_list() SUPPORT_PIPELINE_INPUT_DATA = MindFormerBook().get_pipeline_support_input_data_list() CURRENT_PROJECT_PATH = MindFormerBook().get_project_path() DEFAULT_CHECKPOINT_DIR = 'checkpoint' DEFAULT_CONFIG_DIR = 'configs' class Trainer: r""" Trainer package to train\evaluate\predict class. The trainer interface is used to quickly start training, evaluation and predict for integrated tasks. It also allows users to customize the model, optimizer, dataset, tokenizer, processor, train_one_step, callback, and metric. Args: args (Optional[Union[str, dict, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It support yaml path or config dict or ConfigArguments class. Default: None. task (str): The task name supported. Please refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: 'general'. model (Optional[Union[str, Cell, BaseModel]]): The network for trainer. It support model name supported or BaseModel or MindSpore Cell class. Supported model name can refer to https://gitee.com/mindspore/transformer#%E4%BB%8B%E7%BB%8D. Default: None. model_name (Optional[Union[str]]): The model name supported. Default: None. train_dataset (Optional[Union[str, BaseDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. eval_dataset (Optional[Union[str, BaseDataset]]): The evaluate dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for text preprocessing. It support BaseTokenizer class. Default: None. image_processor (Optional[BaseImageProcessor]): The processor for image preprocessing. It support BaseImageProcessor class. Default: None. audio_processor (Optional[BaseAudioProcessor]): The processor for audio preprocessing. It support BaseAudioProcessor class. Default: None. optimizers (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. eval_callbacks (Optional[Union[Callback, List[Callback]]]): The evaluate callback function. It support CallBack or CallBack List of MindSpore. Default: None. compute_metrics (Optional[Union[dict, set]]): The metric of evaluating. It support dict or set in MindSpore's Metric class. Default: None. save_config (bool): Save current the config of task. Default: False. Raises: KeyError: If 'task' or 'model' not in supported trainer. Examples: >>> from mindformers import Trainer >>> import numpy as np >>> from mindspore.dataset import GeneratorDataset >>> class MyDataLoader: ... def __init__(self): ... self._data = [np.zeros((3, 224, 224), np.float32) for _ in range(64)] ... ... def __getitem__(self, index): ... return self._data[index] ... ... def __len__(self): ... return len(self._data) >>> #1) input task name and model name to init trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train') >>> #2) input config to init trainer >>> from mindformers.trainer.config_args import ConfigArguments, OptimizerConfig, \ ... RunnerConfig, LRConfig, WrapperConfig >>> from mindspore.nn import AdamWeightDecay, WarmUpLR, \ ... DynamicLossScaleUpdateCell, TrainOneStepWithLossScaleCell >>> from mindspore.train.callback import LossMonitor >>> runner_config = RunnerConfig(epochs=10, batch_size=2, image_size=224) >>> lr_schedule_config = LRConfig(lr_type='WarmUpLR', learning_rate=0.001, warmup_steps=10) >>> optim_config = OptimizerConfig(optim_type='Adam', beta1=0.009, learning_rate=lr_schedule_config) >>> loss_scale = DynamicLossScaleUpdateCell(loss_scale_value=2**12, scale_factor=2, scale_window=1000) >>> wrapper_config = WrapperConfig(wrapper_type='TrainOneStepWithLossScaleCell', scale_sense=loss_scale) >>> dataset = GeneratorDataset(source=MyDataLoader(), column_names='image') >>> dataset = dataset.batch(batch_size=2) >>> config = ConfigArguments(seed=2022, runner_config=runner_config, ... optimizer=optim_config, runner_wrapper=wrapper_config) >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... args=config, train_dataset=dataset) >>> #3) input instance to init trainer >>> from mindformers.models import ViTForImageClassification >>> vit_model_with_loss = ViTForImageClassification() >>> lr_schedule = WarmUpLR(learning_rate=0.001, warmup_steps=100) >>> optimizer = AdamWeightDecay(beta1=0.009, beta2=0.999, ... learning_rate=lr_schedule, ... params=vit_model_with_loss.trainable_params()) >>> loss_cb = LossMonitor(per_print_times=2) >>> callbacks = [loss_cb] >>> task_trainer = Trainer(task='image_classification', ... model=vit_model_with_loss, ... args=config, ... optimizers=optimizer, ... train_dataset=dataset, ... callbacks=callbacks) """ def __init__(self, args: Optional[Union[str, dict, ConfigArguments, TrainingArguments]] = None, task: Optional[str] = 'general', model: Optional[Union[str, Cell, BaseModel]] = None, model_name: Optional[Union[str]] = None, train_dataset: Optional[Union[str, BaseDataset]] = None, eval_dataset: Optional[Union[str, BaseDataset]] = None, tokenizer: Optional[BaseTokenizer] = None, image_processor: Optional[BaseImageProcessor] = None, audio_processor: Optional[BaseAudioProcessor] = None, optimizers: Optional[Optimizer] = None, wrapper: Optional[TrainOneStepCell] = None, pet_method: Optional[str] = '', callbacks: Optional[Union[Callback, List[Callback]]] = None, eval_callbacks: Optional[Union[Callback, List[Callback]]] = None, compute_metrics: Optional[Union[dict, set]] = None, save_config: bool = False, **kwargs): self.task = task self.model = model self.model_name = model_name self.pet_method = pet_method self.train_dataset = train_dataset self.eval_dataset = eval_dataset self.optimizers = optimizers self.wrapper = wrapper self.tokenizer = tokenizer self.image_processor = image_processor self.audio_processor = audio_processor self.callbacks = callbacks self.eval_callbacks = eval_callbacks self.compute_metrics = compute_metrics self.default_checkpoint_name_or_path = None self.configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) self.kwargs = kwargs if not os.path.exists(os.path.join('.', DEFAULT_CONFIG_DIR)): configs_directory = os.path.join('.', DEFAULT_CONFIG_DIR) if os.path.exists(os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR)): mindformers_configs_directory = os.path.join(CURRENT_PROJECT_PATH, DEFAULT_CONFIG_DIR) # python 3.7 版本不支持dirs_exist_ok入参, python 3.8及以上版本支持 try: # adapt to python 3.8+ # pylint: disable=E1123 shutil.copytree(mindformers_configs_directory, configs_directory, dirs_exist_ok=True) except TypeError: try: # adapt to python 3.7 shutil.copytree(mindformers_configs_directory, configs_directory) except FileExistsError: pass if wrapper is not None: if model is not None: logger.warning( 'wrapper has existed, input model invalid, it should be include in wrapper.') if optimizers is not None: logger.warning( 'wrapper has existed, input optimizers invalid, it should be include in wrapper.') assert task in SUPPORT_TASKS.keys(), \ f"task name must be in {SUPPORT_TASKS.keys()}, but get {task}." if isinstance(model, str): model = model + '_{}'.format(pet_method) if pet_method else model assert model in SUPPORT_MODEL_NAMES, \ f"model must be in {SUPPORT_MODEL_NAMES} when model's type is string, but get {model}." self.model_name = model self.model = None if isinstance(self.model, (Cell, BaseModel)): logger.info("The model instance has been entered, " "and the model will not be created from model_config") if pet_method: logger.warning("pet_method is not valid when a model instance is passed in." "Currently, only part of the model in MindFormers is supported." "Please pass in the model keyword") self.is_model_instance = True else: self.is_model_instance = False if self.is_model_instance and self.model_name is None: logger.warning( "Recognizing that a model instance is sent and model_name is None,") logger.warning( "it is recommended to select a model configuration that corresponds " "to the support of MindFormers based on the instance model and set model_name.") logger.warning( "Otherwise, they will default to a general configuration." "You are advised to pass instances such as optimizers, metric, tokenizer, and processor") self.model_name = 'common' task_config = MindFormerConfig(SUPPORT_TASKS.get(self.task).get(self.model_name)) if self.model_name == "common": if self.model is not None: task_config.trainer.model_name = self.model.__class__.__name__ if self.wrapper is not None: task_config.trainer.model_name = self.wrapper.network.__class__.__name__ if args is None: self.config = task_config else: if isinstance(args, dict): task_config.merge_from_dict(args) elif isinstance(args, str): assert os.path.realpath(args) and os.path.exists(args), \ f"config path must be exist, but get {args}." assert args.endswith(('.yaml', '.yml')), \ f"config file must be end with .yaml or .yml, but get {args}" task_config = MindFormerConfig(args) elif isinstance(args, ConfigArguments): if hasattr(args, 'train_dataset'): check_train_data_loader_type(args, task_config) if hasattr(args, 'eval_dataset'): check_eval_data_loader_type(args, task_config) if hasattr(args, 'optimizer'): check_optimizer_and_lr_type(args, task_config) if hasattr(args, 'runner_wrapper'): check_wrapper_config(args, task_config) task_config.merge_from_dict(args.__dict__) elif isinstance(args, TrainingArguments): logger.warning( "When using the TrainingArguments class, " "its arguments will override the default config configuration.") args.convert_args_to_mindformers_config(task_config) self.config = task_config if save_config: self.save_config_to_yaml(self.config) logger.info("save running config success of %s_new.", task_config.trainer.model_name.lower()) # check dataset config if isinstance(train_dataset, str): assert os.path.exists(train_dataset), \ f"train dataset path must be exist, but get {train_dataset}." self.config.train_dataset.data_loader.dataset_dir = train_dataset self.train_dataset = None if isinstance(eval_dataset, str): assert os.path.exists(eval_dataset), \ f"eval dataset path must be exist, but get {eval_dataset}." self.config.eval_dataset.data_loader.dataset_dir = eval_dataset self.eval_dataset = None if tokenizer is not None: if self.config.train_dataset is not None: self.config.train_dataset.tokenizer = tokenizer if self.config.eval_dataset is not None: self.config.eval_dataset.tokenizer = tokenizer check_dataset_config(self.config) # build parallel config self.rank_id = int(os.getenv("RANK_ID", "0")) self.device_num = int(os.getenv("RANK_SIZE", "1")) self.config.rank_id = self.rank_id self.config.device_num = self.device_num self.is_set_parallel_config = False self.is_set_moe_config = False self.is_set_recompute_config = False # set seed if self.config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(self.config.seed) np.random.seed(self.config.seed) # set output directory os.environ.setdefault("LOCAL_DEFAULT_PATH", self.config.output_dir) # build task trainer self.trainer = build_trainer(self.config.trainer) if self.trainer is None: raise ModuleNotFoundError("config must be contain 'trainer' key, but get None.") def train(self, train_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Train task for Trainer. This function is used to train or fine-tune the network. Args: train_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It supports real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. resume_training (bool): Whether to perform resume training. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if resume_or_finetune_from_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/eval') >>> # 1) default train task to reproduce model. >>> task_trainer.train() >>> # 2) eval network when train task to reproduce model. >>> task_trainer.train(do_eval=True) >>> # 3) resume train task to auto load the last checkpoint, if training break after 10 epochs. >>> task_trainer.train(train_checkpoint=True, resume_training=True) >>> # 4) resume train task according to checkpoint path, if training break after 10 epochs. >>> task_trainer.train( ... resume_or_finetune_from_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... resume_training=True) """ if train_checkpoint is not None and \ not isinstance(train_checkpoint, (bool, str)): raise TypeError(f"resume_or_finetune_from_checkpoint must be one of [None, string, bool], " f"but get {train_checkpoint}") if train_checkpoint is False: train_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if train_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(train_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = train_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def finetune(self, finetune_checkpoint: Optional[Union[str, bool]] = False, resume_training: Optional[bool] = None, auto_trans_ckpt: Optional[bool] = None, do_eval: bool = False, **kwargs): r"""Finetune task for Trainer. This function is used to fine-tune the network. Args: finetune_checkpoint (Optional[Union[str, bool]]): Used to restore training or fine-tune the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. if resume_training is true, this checkpoint will be used to restore training of the network. Default: False. resume_training (bool): Whether to perform resume training. Default: False auto_trans_ckpt: auto transform checkpoint to load in distributed model do_eval (bool): Whether evaluations are performed during training. Default: False. Raises: TypeError: if load_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='text_generation', ... model='gpt2', ... pet_method='lora', ... train_dataset='./train', ... eval_dataset='./eval') >>> # 1) default finetune task. >>> task_trainer.finetune() >>> # 2) eval network when finetune model. >>> task_trainer.finetune(do_eval=True) >>> # 3) The last weight in the output directory is automatically loaded to resume training. >>> task_trainer.finetune(finetune_checkpoint=True, resume_training=True) >>> # 4) The last weight in the output directory is automatically loaded for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint=True) >>> # 5) Specify weights to fine-tune. >>> task_trainer.finetune(finetune_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') >>> # 6) Automatically load preset weights for fine-tuning. >>> task_trainer.finetune(finetune_checkpoint='gpt2') """ if finetune_checkpoint is not None and \ not isinstance(finetune_checkpoint, (bool, str)): raise TypeError(f"load_checkpoint must be one of [None, string, bool], " f"but get {finetune_checkpoint}") if finetune_checkpoint is False: finetune_checkpoint = None do_eval = do_eval or self.config.do_eval if do_eval: if self.eval_dataset is None: logger.info("do_eval is enabled, building eval_dataset from config.") self.eval_dataset = build_dataset(self.config.eval_dataset_task) if self.eval_dataset is None: raise ValueError(f"if do_eval is true, eval_dataset must be input, " f"the task {self.task} is not support eval now.") # open do_eval for trainer config self.config.do_eval = True if finetune_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(finetune_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = finetune_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be fine-tune are specified" "by the finetune_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None self.config.model.model_config.checkpoint_name_or_path = None if resume_training is not None: self.config.resume_training = resume_training if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=True) self.trainer.train( config=self.config, network=self.model, dataset=self.train_dataset, optimizer=self.optimizers, eval_dataset=self.eval_dataset if do_eval else None, wrapper=self.wrapper, callbacks=self.callbacks, is_full_config=True, **kwargs) def evaluate(self, eval_checkpoint: Optional[Union[str, bool]] = False, auto_trans_ckpt: Optional[bool] = None, **kwargs): r"""Evaluate task for Trainer. This function is used to evaluate the network. Args: eval_checkpoint (Optional[Union[str, bool]]): Used to evaluate the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model Raises: TypeError: if eval_checkpoint is not bool or str type. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... eval_dataset='data/imagenet/train') >>> # 1) default evaluate task to test model. >>> task_trainer.evaluate() >>> # 2) evaluate task to auto load the last checkpoint. >>> task_trainer.evaluate(eval_checkpoint=True) >>> # 3) evaluate task according to checkpoint path. >>> task_trainer.evaluate(eval_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt') """ if eval_checkpoint is not None and not isinstance(eval_checkpoint, (bool, str)): raise TypeError(f"eval_checkpoint must be one of [None, string, bool], " f"but get {eval_checkpoint}") if eval_checkpoint is False: eval_checkpoint = None if eval_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(eval_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = eval_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if self.is_model_instance: self.reset_model_instance(is_train=False) self.trainer.evaluate( config=self.config, network=self.model, dataset=self.eval_dataset, callbacks=self.eval_callbacks, is_full_config=True, **kwargs) def predict(self, predict_checkpoint: Optional[Union[str, bool]] = None, auto_trans_ckpt: Optional[bool] = None, input_data: Optional[Union[GeneratorDataset, Tensor, np.ndarray, Image, str, list]] = None, **kwargs): r"""Predict task for Trainer. This function is used to predict the network. Args: predict_checkpoint (Optional[Union[str, bool]]): Used to predict the weight of the network. It support real checkpoint path or valid model name of mindformers or bool value. if it's true, the last checkpoint file saved from the previous training round is automatically used. Default: False. auto_trans_ckpt: auto transform checkpoint to load in distributed model input_data (Optional[Union[Tensor, np.ndarray, Image, str, list]]): The predict data. Default: None. Return: predict result (dict). Raises: TypeError: if predict_checkpoint is not bool or str type. TypeError: if input_data is not Tensor or np.ndarray or Image or str or list. Examples: >>> from mindformers import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16') >>> input_data = "./sunflower.png" >>> # 1) predict task to auto load the last checkpoint. >>> task_trainer.predict(predict_checkpoint=True, input_data=input_data) >>> # 2) predict task according to checkpoint path. >>> task_trainer.predict(predict_checkpoint='./output/rank_0/checkpoint/mindformers.ckpt', ... input_data=input_data) >>> # 3) download and auto load the checkpoint on obs and predict. >>> task_trainer.predict(input_data=input_data) """ if predict_checkpoint is not None and not isinstance(predict_checkpoint, (bool, str)): raise TypeError(f"predict_checkpoint must be one of [None, string, bool], " f"but get {predict_checkpoint}") if self.task not in SUPPORT_PIPELINES.keys(): raise NotImplementedError(f"The {self.task} not support predict, " f"now this tasks {SUPPORT_PIPELINES.keys()} is support predict.") if predict_checkpoint is False: predict_checkpoint = None if predict_checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = self.get_last_checkpoint() elif isinstance(predict_checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = None self.config.load_checkpoint = predict_checkpoint else: self.default_checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if auto_trans_ckpt: if self.is_model_instance: logger.warning( "When a model instance is identified," "the weights that are currently proposed to be evaluated are specified" "by the eval_checkpoint argument or a model instance " "with the model weights already loaded is imported") else: self.config.load_checkpoint = self.config.model.model_config.checkpoint_name_or_path self.config.model.model_config.checkpoint_name_or_path = None else: self.config.load_checkpoint = None if auto_trans_ckpt is not None: self.config.auto_trans_ckpt = auto_trans_ckpt if input_data is None: input_data = build_dataset_loader(self.config.eval_dataset.data_loader) logger.info("dataset by config is used as input_data.") assert isinstance(input_data, (GeneratorDataset, BaseDataset, RepeatDataset, BatchDataset, Tensor, np.ndarray, Image, str, list)), \ "Input data's type must be one of [GeneratorDataset," \ " str, ms.Tensor, np.ndarray, PIL.Image.Image]" if self.is_model_instance: self.reset_model_instance(is_train=False) kwargs['model_name'] = self.model_name output_result = self.trainer.predict( config=self.config, input_data=input_data, network=self.model, image_processor=self.image_processor, audio_processor=self.audio_processor, tokenizer=self.tokenizer, is_full_config=True, **kwargs) return output_result def build_network(self, input_checkpoint: Optional[Union[str, bool]] = None, is_train: bool = True): """build network for trainer.""" if self.model is None and self.task != 'general': logger.info("...........Start Init Network..........") self.model = build_model(self.config.model) # set running mode if self.model is not None: self.model.set_train(is_train) else: logger.warning("network will be create in %s task trainer class.", self.task) if self.is_model_instance and input_checkpoint: self._load_model_checkpoint() def set_parallel_config( self, data_parallel=1, model_parallel=1, expert_parallel=1, pipeline_stage=1, micro_batch_interleave_num=1, micro_batch_num=1, use_seq_parallel=False, optimizer_shard=False, gradient_aggregation_group=4, vocab_emb_dp=True): r""" set_parallel_config for the setting global data parallel, model parallel and fusion group. The parallel configure setting for Trainer. Args: data_parallel (int): The data parallel way. The input data will be sliced into n parts for each layer according to the data parallel way. Default: 1. model_parallel (int): The model parallel way. The parameters of dense layers in MultiheadAttention and FeedForward layer will be sliced according to the model parallel way. Default: 1. expert_parallel (int): The expert parallel way. This is effective only when MoE (Mixture of Experts) is applied. This value specifies the number of partitions to split the experts into. pipeline_stage (int): The number of the pipeline stage. Should be a positive value. Default: 1. micro_batch_num (int): The micro size of the batches for the pipeline training. Default: 1. use_seq_parallel (bool): Whether to enable sequence parallel. Default False. optimizer_shard (bool): Whether to enable optimizer shard. Default False. gradient_aggregation_group (int): The fusion group size of the optimizer state sharding. Default: 4. vocab_emb_dp (bool): Shard embedding in model parallel or data parallel. Default: True. micro_batch_interleave_num (int): split num of batch size. Default: 1. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_parallel_config(data_parallel=2, model_parallel=2) """ self.config.parallel_config.data_parallel = data_parallel self.config.parallel_config.model_parallel = model_parallel self.config.parallel_config.expert_parallel = expert_parallel self.config.parallel_config.pipeline_stage = pipeline_stage self.config.parallel_config.use_seq_parallel = use_seq_parallel self.config.parallel_config.optimizer_shard = optimizer_shard self.config.parallel_config.micro_batch_num = micro_batch_num self.config.parallel_config.vocab_emb_dp = vocab_emb_dp self.config.parallel_config.gradient_aggregation_group = gradient_aggregation_group self.config.micro_batch_interleave_num = micro_batch_interleave_num self.is_set_parallel_config = True def set_recompute_config(self, recompute=False, parallel_optimizer_comm_recompute=False, select_recompute=False, mp_comm_recompute=True, recompute_slice_activation=False): r"""Set recompute config. TransformerRecomputeConfig for the setting recompute attributes for encoder/decoder layers. Args: recompute (bool): Enable recomputation of the transformer block or not. Default: False. select_recompute (bool): Only Enable recomputation of the attention layer or not. Default: False. parallel_optimizer_comm_recompute (bool): Specifies whether the communication operator allgathers introduced by optimizer shard are recomputed in auto parallel or semi auto parallel mode. Default: False. mp_comm_recompute (bool): Specifies whether the model parallel communication operators in the cell are recomputed in auto parallel or semi auto parallel mode. Default: True. recompute_slice_activation (bool): Slice the cell output which would remains in memory. Default: False. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_recompute_config(recompute=True) """ self.config.recompute_config.recompute = recompute self.config.recompute_config.select_recompute = select_recompute self.config.recompute_config.parallel_optimizer_comm_recompute = parallel_optimizer_comm_recompute self.config.recompute_config.mp_comm_recompute = mp_comm_recompute self.config.recompute_config.recompute_slice_activation = recompute_slice_activation self.is_set_recompute_config = True def set_moe_config(self, expert_num=1, capacity_factor=1.1, aux_loss_factor=0.05, num_experts_chosen=1, expert_group_size=None, group_wise_a2a=False, comp_comm_parallel=False, comp_comm_parallel_degree=2): r"""The configuration of MoE (Mixture of Expert). Args: expert_num (int): The number of experts employed. Default: 1 capacity_factor (float): The factor is used to indicate how much to expand expert capacity, which is >=1.0. Default: 1.1. aux_loss_factor (float): The factor is used to indicate how much the load balance loss (produced by the router) to be added to the entire model loss, which is < 1.0. Default: 0.05. num_experts_chosen (int): The number of experts is chosen by each token and it should not be larger than expert_num. Default: 1. expert_group_size (int): The number of tokens in each data parallel group. Default: None. This parameter is effective only when in AUTO_PARALLEL mode, and NOT SHARDING_PROPAGATION. group_wise_a2a (bool): Whether to enable group-wise alltoall communication, which can reduce communication time by converting part of inter communication into intra communication. Default: False. This parameter is effective only when model parallel > 1 and data_parallel equal to expert parallel. comp_comm_parallel (bool): Whether to enable ffn compute and communication parallel, which can reduce pure communicattion time by splitting and overlapping compute and communication. Default: False. comp_comm_parallel_degree (int): The split number of compute and communication. The larger the numbers, the more overlap there will be but will consume more memory. Default: 2. This parameter is effective only when comp_comm_parallel enable. Examples: >>> from mindformers.trainer import Trainer >>> task_trainer = Trainer(task='image_classification', ... model='vit_base_p16', ... train_dataset='data/imagenet/train', ... eval_dataset='data/imagenet/train') >>> task_trainer.set_moe_config(expert_num=2, capacity_factor=1.2, aux_loss_factor=0.001) """ self.config.moe_config.expert_num = expert_num self.config.moe_config.capacity_factor = capacity_factor self.config.moe_config.aux_loss_factor = aux_loss_factor self.config.moe_config.num_experts_chosen = num_experts_chosen self.config.moe_config.expert_group_size = expert_group_size self.config.moe_config.group_wise_a2a = group_wise_a2a self.config.moe_config.comp_comm_parallel = comp_comm_parallel self.config.moe_config.comp_comm_parallel_degree = comp_comm_parallel_degree self.is_set_moe_config = True def reset_model_instance(self, is_train=True): """Reset model instance for new model config.""" if True not in [self.is_set_parallel_config, self.is_set_moe_config, self.is_set_recompute_config]: return if self.is_set_parallel_config: logger.info("The incoming model will be configured in parallel.") if self.is_set_recompute_config: logger.info("The incoming model will be configured in recompute.") if self.is_set_moe_config: logger.info("The incoming model will be configured in moe.") if not isinstance(self.model, BaseModel): raise NotImplementedError("Currently only the integrated model structure in MindFormers is supported.") build_parallel_config(self.config) model_config = self.model.config model_config.parallel_config = self.config.parallel_config model_config.moe_config = self.config.moe_config self.model.__init__(model_config) self.model.set_train(is_train) if not is_train: return network = self.model micro_batch_interleave_num = self.config.micro_batch_interleave_num if ms.get_auto_parallel_context("pipeline_stages") > 1: micro_batch_num = self.config.parallel_config.micro_batch_num if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = PipelineCell(MicroBatchInterleaved(self.model, micro_batch_interleave_num), micro_size=micro_batch_num) else: network = PipelineCell(self.model, micro_size=micro_batch_num) network = _VirtualDatasetCell(network) else: if micro_batch_interleave_num > 1: logger.info("micro_batch_interleave_num > 1, the double copy parallel feature is turned on.") network = MicroBatchInterleaved(self.model, micro_batch_interleave_num) del self.model self.model = network def get_train_dataloader(self): """get train dataloader of mindspore.""" return build_dataset_loader(self.config.train_dataset.data_loader) def get_eval_dataloader(self): """get eval dataloader of mindspore.""" return build_dataset_loader(self.config.eval_dataset.data_loader) def get_last_checkpoint(self): """get last checkpoint for resuming or finetune.""" output_folder = self.config.output_dir checkpoint_dir = os.path.join( output_folder, DEFAULT_CHECKPOINT_DIR, 'rank_{}'.format(self.rank_id)) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) def save_config_to_yaml(self, config: dict = None): """save now config file to yaml file.""" if config is None: config = self.config model_name = self.config.trainer.model_name config_dict = _reset_config_for_save(config) config_dir = os.path.join( self.configs_directory, model_name.lower() + '_new') if not os.path.exists(config_dir): os.makedirs(config_dir, exist_ok=True) run_yaml_path = os.path.join(config_dir, 'run_{}.yaml'.format(model_name.lower())) _save_config_to_yaml(run_yaml_path, config_dict) def _load_model_checkpoint(self): """Load model checkpoint to network.""" checkpoint_name_or_path = self.config.model.model_config.checkpoint_name_or_path if checkpoint_name_or_path is None: logger.warning("checkpoint_name_or_path is None, not load input checkpoint.") elif isinstance(checkpoint_name_or_path, str): is_exist_path = os.path.exists(checkpoint_name_or_path) is_checkpoint_name = checkpoint_name_or_path in SUPPORT_MODEL_NAMES if is_exist_path: logger.info("now input valid checkpoint path, it will load to network.") checkpoint_dict = load_checkpoint(checkpoint_name_or_path) not_load_params = load_param_into_net(self.model, checkpoint_dict) logger.info("not load parameters is: %s", str(not_load_params)) elif is_checkpoint_name: logger.info("now input valid checkpoint name, it will load to network.") if isinstance(self.model, (Cell, BaseModel)): self.model.load_checkpoint(self.config.model.model_config) else: logger.warning("model must be BaseModel or Cell type, but get %s", type(self.model)) else: logger.warning("input checkpoint args is invalid, " "it must be valid and real checkpoint path or a valid checkpoint name," "but get %s", checkpoint_name_or_path) else: raise TypeError(f"checkpoint_name_or_path type error, " f"it should be one of [None, str], " f"but get {type(checkpoint_name_or_path)}") def _check_checkpoint_config(self, checkpoint: Optional[Union[str, bool]] = None): """check checkpoint config.""" if checkpoint is True: self.config.model.model_config.checkpoint_name_or_path = self.get_last_checkpoint() elif isinstance(checkpoint, str): self.config.model.model_config.checkpoint_name_or_path = checkpoint else: if self.default_checkpoint_name_or_path is not None: self.config.model.model_config.checkpoint_name_or_path = self.default_checkpoint_name_or_path def _save_config_to_yaml(save_file_path: str = None, save_config: dict = None): r"""Save Config to Yaml File. Args: save_file_path (str): The real path to save yaml file. Default: None. save_config (dict): The task config. Default: None. """ if save_config is None: save_config = {} with open(save_file_path, 'w', encoding='utf-8') as file_pointer: file_pointer.write( ordered_yaml_dump( save_config, default_flow_style=False, allow_unicode=True, sort_keys=False)) def _reset_config_for_save(config: dict = None): r"""Reset Config According to Yaml File Number. Args: config (dict): The task config. Default: None. model_name (str): The model name to save. Default: 'common'. """ if config is None: config = {} config = config.copy() config_dict = OrderedDict() if config.get('model') is not None: model_config = config2dict(config.pop('model')) config_dict.setdefault('model', model_config) if config.get('processor') is not None: processor_config = config2dict(config.pop('processor')) config_dict.setdefault('processor', processor_config) if config.get('train_dataset_task') is not None and config.get('train_dataset') is not None: train_dataset_config = config2dict(config.pop('train_dataset')) train_dataset_task_config = config2dict(config.pop('train_dataset_task')) config_dict.setdefault('train_dataset', train_dataset_config) config_dict.setdefault('train_dataset_task', train_dataset_task_config) if config.get('eval_dataset_task') is not None and config.get('eval_dataset') is not None: eval_dataset_config = config2dict(config.pop('eval_dataset')) eval_dataset_task_config = config2dict(config.pop('eval_dataset_task')) config_dict.setdefault('train_dataset', eval_dataset_config) config_dict.setdefault('train_dataset_task', eval_dataset_task_config) if config.get('context') is not None: context_config = config2dict(config.pop('context')) parallel_context_config = config2dict(config.pop('parallel')) moe_conifg = config2dict(config.pop('moe_config')) recompute_config = config2dict(config.pop('recompute_config')) parallel_config = config2dict(config.pop('parallel_config')) config_dict.setdefault('context', context_config) config_dict.setdefault('parallel', parallel_context_config) config_dict.setdefault('moe_conifg', moe_conifg) config_dict.setdefault('recompute_config', recompute_config) config_dict.setdefault('parallel_config', parallel_config) if config.get('runner_config') is not None: runner_config = config2dict(config.pop('runner_config')) config_dict.setdefault('runner_config', runner_config) if config.get('runner_wrapper') is not None: wrapper_config = config2dict(config.pop('runner_wrapper')) config_dict.setdefault('runner_wrapper', wrapper_config) if config.get('optimizer') is not None: optim_config = config2dict(config.pop('optimizer')) config_dict.setdefault('optimizer', optim_config) if config.get('lr_schedule') is not None: lr_config = config2dict(config.pop('lr_schedule')) config_dict.setdefault('lr_schedule', lr_config) if config.get('callbacks') is not None: cb_config = config2dict(config.pop('callbacks')) config_dict.setdefault('callbacks', cb_config) run_config = config2dict(config) for key, value in run_config.items(): config_dict.setdefault(key, value) return config_dict pengcheng.mind-mindformer/mindformers/trainer/training_args.py000066400000000000000000000344241476650141400253130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # This file was refer to project: # https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py # ============================================================================ """Default Training Arguments for Trainer.""" import os from typing import Optional, Union from dataclasses import dataclass, field from mindformers.tools.register import MindFormerConfig from mindformers.tools import logger from .utils import LRType, OptimizerType, SaveIntervalStrategy def _check_task_config(check_config): """check task config for adapting hugging-face.""" if check_config is not None and isinstance(check_config, MindFormerConfig): return True return False def _check_training_args(ori_value, new_value): """check training arguments for adapt MindFormers.""" if new_value is not None: return new_value return ori_value @dataclass class TrainingArguments: """ TrainingArguments is the subset of the arguments we use in our default config **which is relate to the training in MindSpore**. """ output_dir: str = field( default=None, metadata={"help": "The output directory where the model predictions and checkpoints will be written."}, ) use_parallel: bool = field( default=None, metadata={"help": "The use_parallel is used to enable distribute parallel of the network."} ) profile: bool = field( default=None, metadata={"help": "The profile is used to enable profiling of the network."} ) only_save_strategy: bool = field( default=None, metadata={"help": "The only_save_strategy is used to enable save strategy of the network."} ) sink_mode: bool = field( default=None, metadata={"help": "The sink_mode is used to enable data sink of the network."} ) sink_size: bool = field( default=None, metadata={"help": "The sink_size is used to enable data sink number per step for training or evaluation."} ) batch_size: int = field( default=None, metadata={"help": "Global batch size per GPU/TPU core/CPU for training and evaluation."} ) per_device_train_batch_size: int = field( default=None, metadata={"help": "Batch size per GPU/TPU core/CPU for training."} ) per_device_eval_batch_size: int = field( default=None, metadata={"help": "Batch size m per GPU/TPU core/CPU for evaluation."} ) learning_rate: float = field(default=None, metadata={"help": "The initial learning rate for AdamW."}) weight_decay: float = field(default=None, metadata={"help": "Weight decay for AdamW if we apply some."}) adam_beta1: float = field(default=None, metadata={"help": "Beta1 for AdamW optimizer"}) adam_beta2: float = field(default=None, metadata={"help": "Beta2 for AdamW optimizer"}) adam_epsilon: float = field(default=None, metadata={"help": "Epsilon for AdamW optimizer."}) max_grad_norm: float = field(default=None, metadata={"help": "Max gradient norm."}) num_train_epochs: float = field(default=None, metadata={"help": "Total number of training epochs to perform."}) lr_scheduler_type: Union[LRType, str] = field( default=None, metadata={"help": "The lr scheduler type to use."}, ) optim: Union[OptimizerType, str] = field( default=None, metadata={"help": "The optimizer type to use."}, ) warmup_steps: int = field(default=0, metadata={"help": "Linear warmup over warmup_steps."}) save_strategy: Union[SaveIntervalStrategy, str] = field( default=None, metadata={"help": "The checkpoint save strategy to use."}, ) integrated_save: bool = field( default=None, metadata={ "help": ( " Whether to merge and save the split Tensor in the automatic parallel scenario. " "Integrated save function is only supported in automatic parallel scene, not supported" "in manual parallel." ) } ) save_steps: int = field(default=None, metadata={"help": "Save checkpoint every X updates steps."}) save_seconds: int = field(default=None, metadata={"help": "Save checkpoint every X updates seconds."}) save_total_limit: Optional[int] = field( default=None, metadata={ "help": ( "Limit the total amount of checkpoints. " "Deletes the older checkpoints in the output_dir. Default is unlimited checkpoints" ) }, ) seed: int = field(default=None, metadata={"help": "Random seed that will be set at the beginning of training."}) data_seed: Optional[int] = field(default=None, metadata={"help": "Random seed to be used with data samplers."}) resume_from_checkpoint: Optional[str] = field( default=None, metadata={"help": "The path to a folder with a valid checkpoint for your model."}, ) auto_trans_ckpt: bool = field( default=None, metadata={"help": "Whether to transform checkpoint according to parallel config. "} ) resume_training: bool = field( default=None, metadata={"help": "Whether to load training context info, such as optimizer and epoch num. "} ) do_eval: bool = field( default=None, metadata={"help": "Whether to eval current model while Training. "} ) eval_step_interval: int = field( default=None, metadata={"help": "Num of step intervals between each eval, -1 means no step end eval. "} ) eval_epoch_interval: int = field( default=None, metadata={"help": "Num of epoch intervals between each eval, 1 means eval on every epoch end. "} ) device_num = int(os.getenv("RANK_SIZE", "1")) device_id = int(os.getenv("DEVICE_ID", "0")) rank_id = int(os.getenv("RANK_ID", "0")) def __post_init__(self): if self.output_dir is not None: self.output_dir = os.path.expanduser(self.output_dir) if self.lr_scheduler_type is not None: self.lr_scheduler_type = LRType(self.lr_scheduler_type).value if self.optim is not None: self.optim = OptimizerType(self.optim).value if self.save_strategy is not None: self.save_strategy = SaveIntervalStrategy(self.save_strategy).value @property def get_device_num(self): """get device num for training.""" return self.device_num @property def get_device_id(self): """get device id for training.""" return self.device_id @property def get_rank_id(self): """get rank id for training.""" return self.rank_id def convert_args_to_mindformers_config(self, task_config: MindFormerConfig = None): """convert training arguments to mindformer config type for adapting hugging-face.""" if task_config is None: task_config = MindFormerConfig() self._adapt_dataset_config(task_config) self._adapt_runner_config(task_config) self._adapt_lr_schedule_config(task_config) self._adapt_optimizer_config(task_config) self._adapt_save_checkpoint_config(task_config) task_config.output_dir = _check_training_args(task_config.output_dir, self.output_dir) task_config.seed = _check_training_args(task_config.seed, self.seed) task_config.data_seed = _check_training_args(task_config.data_seed, self.data_seed) task_config.profile = _check_training_args(task_config.profile, self.profile) task_config.only_save_strategy = _check_training_args(task_config.only_save_strategy, self.only_save_strategy) task_config.use_parallel = _check_training_args(task_config.use_parallel, self.use_parallel) task_config.load_checkpoint = _check_training_args( task_config.load_checkpoint, self.resume_from_checkpoint) task_config.auto_trans_ckpt = _check_training_args( task_config.auto_trans_ckpt, self.auto_trans_ckpt) task_config.resume_training = _check_training_args( task_config.resume_training, self.resume_training) task_config.do_eval = _check_training_args( task_config.do_eval, self.do_eval) task_config.eval_step_interval = _check_training_args( task_config.eval_step_interval, self.eval_step_interval) task_config.eval_epoch_interval = _check_training_args( task_config.eval_epoch_interval, self.eval_epoch_interval) def _adapt_dataset_config(self, task_config): """adapt dataset config.""" if _check_task_config(task_config.train_dataset): task_config.train_dataset.batch_size = _check_training_args( task_config.train_dataset.batch_size, self.per_device_train_batch_size) else: logger.warning( "This task does not support training at the moment; " "it does not have a default training data configuration," "so the per_device_train_batch_size setting will not take effect") if _check_task_config(task_config.eval_dataset): task_config.eval_dataset.batch_size = task_config.train_dataset.batch_size = _check_training_args( task_config.eval_dataset.batch_size, self.per_device_eval_batch_size) else: logger.warning( "This task does not support evaluate at the moment;" "it does not have a default evaluate data configuration," "so the per_device_eval_batch_size setting will not take effect") def _adapt_runner_config(self, task_config): """adapt runner config.""" if _check_task_config(task_config.runner_config): task_config.runner_config.epochs = _check_training_args( task_config.runner_config.epochs, self.num_train_epochs) task_config.runner_config.batch_size = _check_training_args( task_config.runner_config.batch_size, self.batch_size) task_config.runner_config.sink_size = _check_training_args( task_config.runner_config.sink_size, self.sink_size) task_config.runner_config.per_epoch_size = _check_training_args( task_config.runner_config.per_epoch_size, self.sink_size) task_config.runner_config.sink_mode = _check_training_args( task_config.runner_config.sink_mode, self.sink_mode) def _adapt_lr_schedule_config(self, task_config): """adapt lr schedule config.""" if _check_task_config(task_config.lr_schedule): if task_config.lr_schedule.type is not None: task_config.lr_schedule.type = _check_training_args( task_config.lr_schedule.type, self.lr_scheduler_type) if task_config.lr_schedule.learning_rate is not None: task_config.lr_schedule.learning_rate = _check_training_args( task_config.lr_schedule.learning_rate, self.learning_rate) if task_config.lr_schedule.warmup_steps is not None: task_config.lr_schedule.warmup_steps = _check_training_args( task_config.lr_schedule.warmup_steps, self.warmup_steps) def _adapt_optimizer_config(self, task_config): """adapt optimizer config.""" if _check_task_config(task_config.optimizer): if task_config.optimizer.type is not None: task_config.optimizer.type = _check_training_args( task_config.optimizer.type, self.optim) if task_config.optimizer.learning_rate is not None: task_config.optimizer.learning_rate = _check_training_args( task_config.optimizer.learning_rate, self.learning_rate) if task_config.optimizer.weight_decay is not None: task_config.optimizer.weight_decay = _check_training_args( task_config.optimizer.weight_decay, self.weight_decay) if task_config.optimizer.beta1 is not None: task_config.optimizer.beta1 = _check_training_args( task_config.optimizer.beta1, self.adam_beta1) if task_config.optimizer.beta2 is not None: task_config.optimizer.beta2 = _check_training_args( task_config.optimizer.beta2, self.adam_beta2) if task_config.optimizer.eps is not None: task_config.optimizer.eps = _check_training_args( task_config.optimizer.eps, self.adam_epsilon) def _adapt_save_checkpoint_config(self, task_config): """adapt save checkpoint config.""" if task_config.callbacks is not None and \ isinstance(task_config.callbacks, list) and \ self.save_strategy is not None: for i, callback in enumerate(task_config.callbacks): if isinstance(callback, dict) and callback['type'] == "CheckpointMointor": if self.save_strategy == 'no': task_config.callbacks.pop(i) continue if self.save_strategy == 'steps': task_config.callbacks[i]['save_checkpoint_steps'] = _check_training_args( task_config.callbacks[i]['save_checkpoint_steps'], self.save_steps) elif self.save_strategy == 'seconds': task_config.callbacks[i]['save_checkpoint_seconds'] = _check_training_args( task_config.callbacks[i]['save_checkpoint_seconds'], self.save_seconds) task_config.callbacks[i]['keep_checkpoint_max'] = self.save_total_limit \ if self.save_total_limit else 5 task_config.callbacks[i]['integrated_save'] = _check_training_args( task_config.callbacks[i]['integrated_save'], self.integrated_save) pengcheng.mind-mindformer/mindformers/trainer/translation/000077500000000000000000000000001476650141400244415ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/translation/__init__.py000066400000000000000000000014331476650141400265530ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Masked Language Modeling Trainer.""" from .translation_finetune import TranslationTrainer __all__ = ['TranslationTrainer'] pengcheng.mind-mindformer/mindformers/trainer/translation/__pycache__/000077500000000000000000000000001476650141400266515ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/translation/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004651476650141400330460ustar00rootroot00000000000000a ?cf@sdZddlmZdgZdS)z!Masked Language Modeling Trainer.)TranslationTrainerrN)__doc__Ztranslation_finetuner__all__rrS/home/ma-user/work/mindformers-baichuan/mindformers/trainer/translation/__init__.pys translation_finetune.cpython-39.pyc000066400000000000000000000143511476650141400354620ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/trainer/translation/__pycache__a ?cf@sdZddlZddlmZmZmZddlmZddl m Z m Z m Z ddl mZddlmZddlmZmZdd lmZmZmZd d lmZd d lmZd d lmZeejGdddeZ dS)zTranslation Modeling Trainer.N)OptionalListUnion)Callback)TrainOneStepCell OptimizerCell)GeneratorDataset) BaseDataset) BaseModel BaseTokenizer)MindFormerRegisterMindFormerModuleTypeMindFormerConfig) BaseTrainer)ConfigArguments)TrainingArgumentsc seZdZdZdedfdd Zdeeee e e feee e feeeefeeeeeeeeefdddZd d Zdeeee e e feeeeefeee e feed d d ZZS)TranslationTraineraTranslation Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers.trainer import TranslationTrainer >>> from mindformers import T5ForConditionalGeneration, TranslationTrainer >>> # follow the instruction in t5 section in the README.md and download wmt16 dataset. >>> # change the dataset_files path of configs/t5/wmt16_dataset.yaml >>> trans_trainer = TranslationTrainer(model_name='t5_small') >>> trans_trainer.train() >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> trans_trainer = TranslationTrainer(model_name="t5_small") >>> res = trans_trainer.predict(input_data="translate the English to Romanian: a good boy!", network=model) [{'translation_text': ['hello world']}] Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. N) model_namecstt|d|dS)N translation)superr__init__)selfr __class___/home/ma-user/work/mindformers-baichuan/mindformers/trainer/translation/translation_finetune.pyr3szTranslationTrainer.__init__)confignetworkdatasetwrapper optimizer callbacksc Ks"|jf||||||d|dS)aTrain task for Translation Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. )rrr#r r!r"N)training_process)rrrr r!r"r#kwargsrrrtrain6s$zTranslationTrainer.traincOs tddS)Nz/The Translation task does not support evaluate.)NotImplementedError)rargsr%rrrevaluatecszTranslationTrainer.evaluate)r input_datar tokenizerc Ks|dur|j}t|trjtj|rjt|d,}g}|D]}||q8|}Wdn1s`0Y|jf||d||d|S)a Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. Nrr)rr*taskrr+) r* isinstancestrospathisfileopenappendpredict_process) rrr*rr+r%fpZinput_data_listlinerrrpredictgs   "zTranslationTrainer.predict)N)NNNNNN)NNNN)__name__ __module__ __qualname____doc__r/rrrdictrrrrr r r rrrrr&r)listr r8 __classcell__rrrrr s6 -r)!r<os.pathr0typingrrrZmindspore.trainr mindspore.nnrrrmindspore.datasetr mindformers.datasetr mindformers.modelsr r mindformers.tools.registerr rr base_trainerr config_argsr training_argsrregisterTRAINERrrrrrs       pengcheng.mind-mindformer/mindformers/trainer/translation/translation_finetune.py000066400000000000000000000156451476650141400312610ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Translation Modeling Trainer.""" import os.path from typing import Optional, List, Union from mindspore.train import Callback from mindspore.nn import TrainOneStepCell, Optimizer, Cell from mindspore.dataset import GeneratorDataset from mindformers.dataset import BaseDataset from mindformers.models import BaseModel, BaseTokenizer from mindformers.tools.register import MindFormerRegister,\ MindFormerModuleType, MindFormerConfig from ..base_trainer import BaseTrainer from ..config_args import ConfigArguments from ..training_args import TrainingArguments @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TranslationTrainer(BaseTrainer): r"""Translation Task For Trainer. Args: model_name (str): The model name of Task-Trainer. Default: None Examples: >>> from mindformers.trainer import TranslationTrainer >>> from mindformers import T5ForConditionalGeneration, TranslationTrainer >>> # follow the instruction in t5 section in the README.md and download wmt16 dataset. >>> # change the dataset_files path of configs/t5/wmt16_dataset.yaml >>> trans_trainer = TranslationTrainer(model_name='t5_small') >>> trans_trainer.train() >>> model = T5ForConditionalGeneration.from_pretrained('t5_small') >>> trans_trainer = TranslationTrainer(model_name="t5_small") >>> res = trans_trainer.predict(input_data="translate the English to Romanian: a good boy!", network=model) [{'translation_text': ['hello world']}] Raises: NotImplementedError: If train method or evaluate method or predict method not implemented. """ def __init__(self, model_name: str = None): super(TranslationTrainer, self).__init__("translation", model_name) def train(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, network: Optional[Union[Cell, BaseModel]] = None, dataset: Optional[Union[BaseDataset, GeneratorDataset]] = None, wrapper: Optional[TrainOneStepCell] = None, optimizer: Optional[Optimizer] = None, callbacks: Optional[Union[Callback, List[Callback]]] = None, **kwargs): r"""Train task for Translation Trainer. This function is used to train or fine-tune the network. The trainer interface is used to quickly start training for general task. It also allows users to customize the network, optimizer, dataset, wrapper, callback. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. dataset (Optional[Union[BaseDataset, GeneratorDataset]]): The training dataset. It support real dataset path or BaseDateset class or MindSpore Dataset class. Default: None. optimizer (Optional[Optimizer]): The training network's optimizer. It support Optimizer class of MindSpore. Default: None. wrapper (Optional[TrainOneStepCell]): Wraps the `network` with the `optimizer`. It support TrainOneStepCell class of MindSpore. Default: None. callbacks (Optional[Union[Callback, List[Callback]]]): The training callback function. It support CallBack or CallBack List of MindSpore. Default: None. Raises: NotImplementedError: If wrapper not implemented. """ self.training_process( config=config, network=network, callbacks=callbacks, dataset=dataset, wrapper=wrapper, optimizer=optimizer, **kwargs) def evaluate(self, *args, **kwargs): raise NotImplementedError( "The Translation task does not support evaluate.") def predict(self, config: Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]] = None, input_data: Optional[Union[str, list, GeneratorDataset]] = None, network: Optional[Union[Cell, BaseModel]] = None, tokenizer: Optional[BaseTokenizer] = None, **kwargs): """ Executes the predict of the trainer. Args: config (Optional[Union[dict, MindFormerConfig, ConfigArguments, TrainingArguments]]): The task config which is used to configure the dataset, the hyper-parameter, optimizer, etc. It supports config dict or MindFormerConfig or TrainingArguments or ConfigArguments class. Default: None. input_data (Optional[Union[Tensor, str, list]]): The predict data. It supports 1) a text string to be translated, 1) a file name where each line is a text to be translated and 3) a generator dataset. Default: None. network (Optional[Union[Cell, BaseModel]]): The network for trainer. It supports model name or BaseModel or MindSpore Cell class. Default: None. tokenizer (Optional[BaseTokenizer]): The tokenizer for tokenizing the input text. Default: None. Returns: A list of prediction. """ if input_data is None: input_data = config.input_data if isinstance(input_data, str) and os.path.isfile(input_data): with open(input_data, 'r') as fp: input_data_list = [] for line in fp: input_data_list.append(line) input_data = input_data_list return self.predict_process(config=config, input_data=input_data, task='translation', network=network, tokenizer=tokenizer, **kwargs) pengcheng.mind-mindformer/mindformers/trainer/utils.py000066400000000000000000000651111476650141400236210ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Trainer Utils.""" import os import time import random from glob import glob from enum import Enum import numpy as np import mindspore as ms from mindspore.communication.management import get_rank, get_group_size from mindspore import context, load_checkpoint, load_param_into_net from mindspore import set_seed as ms_set_seed from mindformers.tools.logger import logger from mindformers.tools.register import MindFormerConfig from mindformers.tools.utils import check_in_modelarts, get_output_root_path from mindformers.tools.transform_ckpt import get_strategy if check_in_modelarts(): import moxing as mox ELAPSE_TIME = 7200 class BaseEnum(str, Enum): """ Base Enum for MindFormers. """ @classmethod def _missing_(cls, value): """Enum with more explicit error message for missing values.""" raise ValueError( f"{value} is not a valid {cls.__name__}, please select one of {list(cls._value2member_map_.keys())}" ) class SaveIntervalStrategy(BaseEnum): """ Stores the acceptable string identifiers for save checkpoint monitor. """ NO = "no" STEPS = "steps" SECONDS = "seconds" class LRType(BaseEnum): """ Stores the acceptable string identifiers for learning rate schedule. """ # will be support item for future. LINEAR = "linear" COSINE = "cosine" COSINE_WITH_RESTARTS = "cosine_with_restarts" POLYNOMIAL = "polynomial" CONSTANT_WITH_WARMUP = "constant_with_warmup" class OptimizerType(BaseEnum): """ Stores the acceptable string identifiers for optimizers. """ # supported item for test, will be delete in the future. ADAMWEIGHTDECAY = 'AdamWeightDecay' # will be support item for future. ADAMW = "adamw" ADAM = "adam" SGD = "sgd" ADAGRAD = "adagrad" ADAFACTOR = "adafactor" class WrapperType(BaseEnum): """ Stores the acceptable string identifiers for training wrapper. """ # will be support item for future. MFWRAPPER = 'mf_wrapper' TRAINONESTEP = 'wrapper' TRAINONESTEPWITHLOSSSCALE = 'loss_scale_wrapper' def set_seed(seed: int = 0): """ Helper function for reproducible behavior to set the seed in `random`, `numpy`, `MindSpore`. Args: seed (`int`): The seed to set. """ random.seed(seed) np.random.seed(seed) ms_set_seed(seed) def check_keywords_in_name(name, keywords=()): """ Check keywords in name. """ isin = False for keyword in keywords: if keyword in name: isin = True return isin def check_runner_config(config, dataset): """ Check runner config. """ data_size = dataset.get_dataset_size() new_epochs = config.runner_config.epochs config.runner_config.origin_epochs = new_epochs if config.runner_config.initial_epoch is None: config.runner_config.initial_epoch = 0 if config.runner_config.sink_mode: if config.runner_config.sink_size != -1: if config.runner_config.sink_size <= 0: raise ValueError("per epoch size must be more than 0 or equal to -1, " f"but get {config.runner_config.sink_size}") if data_size < config.runner_config.sink_size: logger.warning("The data size %s (get from dataset.get_dataset_size()) is smaller " "than the sink_size %s (get from config.runner_config.sink_size), " "you should set the config.runner_config.sink_size to %s", data_size, config.runner_config.sink_size, data_size) config.runner_config.epochs = int((data_size / config.runner_config.sink_size) * new_epochs) else: config.runner_config.sink_size = data_size else: logger.warning("Sink mode is False, per epoch size is invalid, it will reset -1.") config.runner_config.sink_size = -1 config.data_size = data_size logger.info("Will be Training epochs:%d, sink_size:%d", config.runner_config.origin_epochs, config.runner_config.sink_size) logger.info("Create training dataset finish, dataset size:%d", data_size) def check_train_data_loader_type(new_config, old_config): """Check train data loader config type.""" if new_config.train_dataset is None: return None if new_config.train_dataset.get('data_loader') is None: return None train_data_loader_type = new_config.train_dataset.get('data_loader').get('type') if old_config.train_dataset is not None and train_data_loader_type is not None: default_train_data_loader_type = old_config.train_dataset.data_loader.type if train_data_loader_type != default_train_data_loader_type: logger.warning("train dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", train_data_loader_type) old_config.train_dataset.data_loader = {} return None def check_eval_data_loader_type(new_config, old_config): """Check eval data loader config type.""" if new_config.eval_dataset is None: return None if new_config.eval_dataset.get('data_loader') is None: return None eval_data_loader_type = new_config.eval_dataset.get('data_loader').get('type') if old_config.eval_dataset is not None and eval_data_loader_type is not None: default_eval_data_loader_type = old_config.eval_dataset.data_loader.type if eval_data_loader_type != default_eval_data_loader_type: logger.warning("eval dataset's data_loader type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", eval_data_loader_type) old_config.eval_dataset.data_loader = {} return None def check_optimizer_and_lr_type(new_config, old_config): """Check optimizer and lr schedule config type.""" optimizer_type = new_config.optimizer.get('type') if old_config.optimizer is not None and optimizer_type is not None: default_optimizer_type = old_config.optimizer.type if optimizer_type != default_optimizer_type: logger.warning( "optimizer type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually except (params).", optimizer_type) old_config.optimizer = {} if hasattr(new_config.optimizer, 'learning_rate'): lr_type = new_config.optimizer.learning_rate.get('type') if old_config.lr_schedule is not None and lr_type is not None: default_lr_type = old_config.lr_schedule.type if lr_type != default_lr_type: logger.warning( "lr schedule type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", lr_type) old_config.lr_schedule = None def check_wrapper_config(new_config, old_config): """Check wrapper config.""" wrapper_type = new_config.runner_wrapper.get('type') if old_config.runner_wrapper is not None and wrapper_type is not None: default_wrapper_type = old_config.runner_wrapper.type if wrapper_type != default_wrapper_type: logger.warning( "wrapper type is changed to %s." "The default parameters will be cleared." "Please make sure to input the corresponding parameter values manually.", wrapper_type) old_config.runner_wrapper = {} def config2dict(config): """MindFormerConfig Type Convert to Dict.""" if not isinstance(config, (dict, MindFormerConfig)): return config new_dict = {} for key, value in config.items(): if isinstance(value, MindFormerConfig): value = config2dict(value) new_dict.setdefault(key, value) return new_dict def load_distributed_checkpoint(config, specify_prefix=None): """Load Checkpoint in Parallel Mode.""" checkpoint_dir = config.load_checkpoint if os.path.isdir(checkpoint_dir): logger.info( "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/checkpoint/rank_{0-*}/**.ckpt") distribute_checkpoint_dir = os.path.join( checkpoint_dir, "rank_{}".format(int(os.getenv("RANK_ID", "0")))) distribute_checkpoint_path = get_last_checkpoint(distribute_checkpoint_dir) elif os.path.isfile(checkpoint_dir): logger.info("Your load_checkpoint is file, it will be load in network.") distribute_checkpoint_path = checkpoint_dir else: raise FileNotFoundError(f"{checkpoint_dir} is not found.") checkpoint_dict = load_checkpoint(distribute_checkpoint_path, specify_prefix=specify_prefix) logger.info("Distribute load is success.") return checkpoint_dict def load_resume_context_from_checkpoint(config): """resume training, load training info from checkpoint to config""" if not os.path.realpath(config.load_checkpoint) or \ not os.path.exists(config.load_checkpoint): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if os.path.isdir(config.load_checkpoint): resume_dict = load_distributed_checkpoint(config, ["loss_scale", "epoch_num"]) if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") else: resume_dict = load_checkpoint(config.load_checkpoint, specify_prefix=["loss_scale", "epoch_num"]) if "epoch_num" in resume_dict: config.runner_config.initial_epoch = int(resume_dict["epoch_num"]) else: config.runner_config.initial_epoch = 0 for callback in config.callbacks: if "type" in callback and callback["type"] == "CheckpointMointor": if config.runner_wrapper.scale_sense is not None and "loss_scale" in resume_dict: config.runner_wrapper.scale_sense.loss_scale_value = resume_dict["loss_scale"] break def transform_and_load_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not config.only_save_strategy and (not os.path.realpath(config.load_checkpoint) or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 1. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) if config.auto_trans_ckpt: # 2. get strategy dst_ckpt_strategy = get_dst_strategy(config) # 3. transform checkpoint if needed transform_ckpt(config, dst_ckpt_strategy=dst_ckpt_strategy) # 4. load ckpt load_ckpt(config, network, optimizer=optimizer) def load_merged_checkpoint(config, model, network, dataset, optimizer=None, do_eval=False, do_predict=False): """ load checkpoint into net, transform checkpoint if transform is True 1. build net if parallel mode is auto_parallel 2. get strategy 3. transform checkpoint if need 4. load ckpt """ if not os.path.realpath(config.load_checkpoint or not os.path.exists(config.load_checkpoint)): raise FileNotFoundError(f"The load_checkpoint must be correct, " f"but get {config.load_checkpoint}") # 1. load merged ckpt load_ckpt(config, network, optimizer=optimizer) if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: # 2. build net if parallel mode is auto_parallel build_model(config, model, dataset, do_eval=do_eval, do_predict=do_predict) def build_model(config, model, dataset, do_eval=False, do_predict=False): """build model, generate strategy file""" if context.get_auto_parallel_context('parallel_mode') in ['semi_auto_parallel', 'auto_parallel', 'hybrid_parallel']: if not config.runner_config.sink_mode: raise ValueError("When distributed loads are sliced weights, sink_mode must be set True.") if do_eval: model.infer_predict_layout(*next(dataset.create_tuple_iterator())) elif do_predict: model.infer_predict_layout(dataset) else: if config.runner_config.epochs > 1 and config.runner_config.sink_size == 1: raise ValueError(f"When distributed loads are sliced weights, it does not support" f"epochs = {config.runner_config.epochs} > 1 and " f"sink_size = {config.runner_config.sink_size} = 1," f"sink_size must be more than 1") logger.info(".........Building model.........") model.build(train_dataset=dataset, epoch=config.runner_config.epochs, sink_size=config.runner_config.sink_size) if config.only_save_strategy: raise SystemExit("Only_save_strategy is True, model.compile() finished, system exit! ") elif config.only_save_strategy: raise SystemExit("only_save_strategy is True, " "but stand_alone and data_parallel mode do not have strategy file, system exit! ") def get_dst_strategy(config): """get strategy""" rank_id = int(os.getenv('RANK_ID', '0')) world_size = int(os.getenv('RANK_SIZE', '1')) dst_strategy_path = None if world_size == 1: return dst_strategy_path if check_in_modelarts(): # local send all strategy file to obs obs_save_dir = os.path.join(config.remote_save_url, "strategy") mox.file.make_dirs(obs_save_dir) local_strategy_path = config.parallel.strategy_ckpt_save_file obs_strategy_path = os.path.join(obs_save_dir, os.path.basename(local_strategy_path)) mox.file.copy(local_strategy_path, obs_strategy_path) # obs send all strategy to each node logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if rank_id % 8 == 0: wait_collect_all_strategy(local_strategy_dir, world_size, obs_save_dir) else: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) merged_strategy_name = os.path.basename(merged_strategy_path) obs_merged_strategy_path = os.path.join(obs_save_dir, merged_strategy_name) mox.file.copy(merged_strategy_path, obs_merged_strategy_path) logger.info("Save %s to %s", merged_strategy_path, obs_merged_strategy_path) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_path else: logger.info(".........Collecting strategy.........") local_strategy_dir = os.path.join(get_output_root_path(), "strategy") if world_size <= 8: wait_collect_all_strategy(local_strategy_dir, world_size) logger.info(".........All strtegy as follow.........") local_strategy_paths = glob(os.path.join(local_strategy_dir, "*_rank_*.ckpt")) local_strategy_paths.sort() for local_strategy_path in local_strategy_paths: logger.info("strategy: %s", local_strategy_path) logger.info(".........Collecting %d strategy.........", len(local_strategy_paths)) # merge strategy if pipeline_stage > 1 if config.parallel_config.pipeline_stage > 1: if rank_id % 8 == 0: logger.info(".........Merging strategy.........") merged_strategy_path = get_strategy(local_strategy_dir) logger.info(".........Merging succeed.........") dst_strategy_path = merged_strategy_path else: dst_strategy_path = None else: dst_strategy_path = local_strategy_paths[0] else: logger.warning("Can't collecting all strategy, device num > 8!") config.auto_trans_ckpt = False dst_strategy_path = None return dst_strategy_path def transform_ckpt(config, dst_ckpt_strategy=None): """auto transform ckpt""" rank_id = get_rank() if config.use_parallel else 0 world_size = get_group_size() if config.use_parallel else 1 transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") os.makedirs(transformed_ckpt_dir, exist_ok=True) if rank_id % 8 == 0: logger.info(".........Transforming ckpt.........") src_ckpt_strategy = get_strategy(config.src_strategy_path_or_dir) logger.info("Src ckpt strategy: %s", src_ckpt_strategy) logger.info("Src ckpt: %s", config.load_checkpoint) logger.info("Dst ckpt strategy: %s", dst_ckpt_strategy) logger.info("Dst ckpt: %s", transformed_ckpt_dir) ms.transform_checkpoints(config.load_checkpoint, transformed_ckpt_dir, 'checkpoint_', src_ckpt_strategy, dst_ckpt_strategy) logger.info(".........Transform succeed!.........") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_{rank_id}.txt') f = open(transform_succeed_txt, 'w') f.close() if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") mox.file.make_dirs(transformed_ckpt_dir_obs) transform_succeed_txt_obs = os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{rank_id}.txt') mox.file.copy(transform_succeed_txt, transform_succeed_txt_obs) wait_transform(config, rank_id, world_size) if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") rank_transformed_ckpt_dir_obs = os.path.join(transformed_ckpt_dir_obs, f'rank_{rank_id}') rank_transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint", f'rank_{rank_id}') mox.file.copy_parallel(rank_transformed_ckpt_dir, rank_transformed_ckpt_dir_obs) print(f"Rank {rank_id}: Save {rank_transformed_ckpt_dir} to {rank_transformed_ckpt_dir_obs}") config.load_checkpoint = transformed_ckpt_dir def wait_transform(config, rank_id, world_size): """wait all node transform over""" print(f'Rank {rank_id}: Waiting transforming ckpt......') time_out = False if check_in_modelarts(): transformed_ckpt_dir_obs = os.path.join(config.remote_save_url, "transformed_checkpoint") transform_succeed_txts_obs = [os.path.join(transformed_ckpt_dir_obs, f'transform_succeed_rank_{int(i/8)*8}.txt') for i in range(0, world_size, 8)] total_num = len(transform_succeed_txts_obs) all_node_transform_succeed = False start_time = time.time() while not all_node_transform_succeed: node_transform_succeed_num = 0 for transform_succeed_txt in transform_succeed_txts_obs: if mox.file.exists(transform_succeed_txt): node_transform_succeed_num += 1 else: break if node_transform_succeed_num == total_num: all_node_transform_succeed = True else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Automatic weight transform is timed out!") time_out = True break time.sleep(5) else: transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") transform_succeed_txt = os.path.join(transformed_ckpt_dir, f'transform_succeed_rank_0.txt') start_time = time.time() while not os.path.exists(transform_succeed_txt): elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: time_out = True break time.sleep(5) print(f'Rank {rank_id}: Transform succeed!') if time_out: return False return True def wait_collect_all_strategy(strategy_dir, total_num, obs_strategy_dir=None): """wait all strategy collect over""" start_time = time.time() time_out = False while True: if obs_strategy_dir: obs_strategy_paths = mox.file.glob(os.path.join(obs_strategy_dir, "*.ckpt")) if len(obs_strategy_paths) < total_num: time.sleep(5) continue for obs_strategy_path in obs_strategy_paths: local_strategy_path = os.path.join(strategy_dir, os.path.basename(obs_strategy_path)) mox.file.copy(obs_strategy_path, local_strategy_path) local_strategy_paths = glob(os.path.join(strategy_dir, "*_rank_*.ckpt")) if len(local_strategy_paths) == total_num: break else: elapsed_time = time.time() - start_time if elapsed_time >= ELAPSE_TIME: logger.warning("Collection of all strategy timed out!") time_out = True break time.sleep(5) if time_out: return False return True def load_ckpt(config, network, optimizer=None): """load checkpoint""" # load checkpoint params into dict logger.info(".............Start load checkpoint from checkpoint..................") if os.path.isdir(config.load_checkpoint): checkpoint_dict = load_distributed_checkpoint(config) else: checkpoint_dict = load_checkpoint(config.load_checkpoint) # taoht if config.reset_steps: logger.info(".............Reset checkpoint's global_step=0..................") from mindspore.common import Parameter from mindspore.common.tensor import Tensor import mindspore.common.dtype as mstype checkpoint_dict["global_step"] = Parameter(Tensor([0], dtype=mstype.int32), name="global_step") checkpoint_dict["current_iterator_step"] = Parameter(Tensor(0, dtype=mstype.int32), name="current_iterator_step") checkpoint_dict["step_num"] = Parameter(Tensor(0,dtype=mstype.int32), name="step_num") checkpoint_dict["epoch_num"] = Parameter(Tensor(0, dtype=mstype.int32), name="epoch_num") checkpoint_dict["last_overflow_iterator_step"] = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") if config.unload_adam: logger.info(".............unload checkpoint's adam and accu_grads parameters..................") parame_list = {} for k, v in checkpoint_dict.items(): if 'adam' in k or 'accu_grads' in k: # del checkpoint_dict[k] continue else: parame_list[k]=v checkpoint_dict = parame_list # load params into net not_load_network_params = load_param_into_net(network, checkpoint_dict) logger.info("Network parameters are not loaded: %s", str(not_load_network_params)) if optimizer: not_load_optim_params = load_param_into_net(optimizer, checkpoint_dict) logger.info("Optimizer parameters are not loaded: %s", str(not_load_optim_params)) def get_last_checkpoint(checkpoint_dir): """get last checkpoint for resuming or finetune.""" if not os.path.isdir(checkpoint_dir): raise NotADirectoryError( f"{checkpoint_dir} is not a real directory," "When distributed loads are sliced weights," "load_checkpoint should be a checkpoint directory containing the directory of rank_{0-*}," "The directory structure is as follows: **checkpoint_root_dir/rank_{0-*}/checkpoint/**.ckpt") output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] if not output_checkpoint_path: return None output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) return os.path.join(checkpoint_dir, output_checkpoint_path[-1]) pengcheng.mind-mindformer/mindformers/version_control.py000066400000000000000000000072651476650141400242500ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindSpore Version Control""" import os import mindspore as ms from mindspore import nn import mindspore.ops.operations as P import mindspore.ops.functional as F from .tools.utils import is_version_ge from .tools.logger import logger def get_cell_reuse(): """Cell reuse decorator.""" def decorator(func): if os.getenv("MS_DEV_CELL_REUSE", "0") == "0" or not is_version_ge(ms.__version__, "2.1.0"): return func logger.info("Enable cell use mode at %s.", func.__class__.__name__) from mindspore._extends import cell_attr_register return cell_attr_register()(func) return decorator def get_dropout(dropout_prob): if is_version_ge(ms.__version__, '1.11.0'): dropout = nn.Dropout(p=dropout_prob) else: dropout = nn.Dropout(keep_prob=1 - dropout_prob) return dropout def get_tril(): if is_version_ge(ms.__version__, '1.11.0'): tril = P.Tril() else: tril = nn.Tril() return tril def get_norm(): """return ops.norm""" # pylint: disable=C0103 # pylint: disable=W0622 def tensor_norm1(A, ord=None, dim=None, keepdim=False, dtype=None): return F.norm(A, ord=ord, dim=dim, keepdim=keepdim, dtype=dtype) # pylint: disable=C0103 # pylint: disable=W0622 def tensor_norm2(A, ord=2, dim=None, keepdim=False, dtype=None): if dtype is not None: logger.warning("The 'dtype' is not available when mindspore version < '1.11.0'") if not isinstance(ord, int): raise TypeError("The type of 'ord' should be int when mindspore version < '1.11.0'") return F.norm(A, dim, p=ord, keep_dims=keepdim) if is_version_ge(ms.__version__, '1.11.0'): return tensor_norm1 return tensor_norm2 def get_dataset_map(dataset, operations, input_columns=None, output_columns=None, num_parallel_workers=None, **kwargs): if is_version_ge(ms.__version__, "1.11.0"): return dataset.map(operations, input_columns=input_columns, output_columns=output_columns, num_parallel_workers=num_parallel_workers, **kwargs) return dataset.map(operations, input_columns=input_columns, output_columns=output_columns, column_order=output_columns, num_parallel_workers=num_parallel_workers, python_multiprocessing=kwargs.get('python_multiprocessing', False), max_rowsize=kwargs.get('max_rowsize', 16), cache=kwargs.get('cache', None), callbacks=kwargs.get('callbacks', None), offload=kwargs.get('offload', None)) def get_identity(): if is_version_ge(ms.__version__, "1.11.0"): return nn.Identity() return F.identity def fix_optim_global_step_sig(): # when the version of mindspore bigger than 2.2.0, it should update global step explicitly. return is_version_ge(ms.__version__, "2.2.0") pengcheng.mind-mindformer/mindformers/wrapper/000077500000000000000000000000001476650141400221175ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/wrapper/.ipynb_checkpoints/000077500000000000000000000000001476650141400257105ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/wrapper/.ipynb_checkpoints/build_wrapper-checkpoint.py000066400000000000000000000100251476650141400332440ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Wrapper.""" import inspect import mindspore as ms from mindspore import nn, Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .wrapper import MFTrainOneStepCell, MFPipelineWithLossScaleCell WRAPPERS_MINDFORMERS_DEFINED = ["MFTrainOneStepCell", "MFPipelineWithLossScaleCell"] def build_wrapper(config: dict = None, default_args: dict = None, module_type: str = 'wrapper', class_name: str = None, **kwargs): r"""Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.type not in WRAPPERS_MINDFORMERS_DEFINED: if default_args and "parallel_config" in default_args: del default_args["parallel_config"] if config.scale_sense is not None: if isinstance(config.scale_sense, dict) and config.scale_sense.type is not None: config.scale_sense = build_wrapper(config.scale_sense) elif isinstance(config.scale_sense, int): config.scale_sense = Tensor(config.scale_sense, ms.float32) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.WRAPPER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_wrap(): """ register MindSpore builtin wrapper class. """ for module_name in dir(nn.wrap): if module_name.startswith('__'): continue wrap = getattr(nn.wrap, module_name) if inspect.isclass(wrap): MindFormerRegister.register_cls( wrap, MindFormerModuleType.WRAPPER) def register_mf_wrapper(): """ register MindFormers builtin wrapper class. """ # Support built-in model wrapper of MindFormers. MindFormerRegister.register_cls( nn.wrap.TrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="wrapper") MindFormerRegister.register_cls( nn.wrap.TrainOneStepWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="loss_scale_wrapper") MindFormerRegister.register_cls( MFTrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="mf_wrapper") MindFormerRegister.register_cls( MFPipelineWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="pipeline_wrapper") register_ms_wrap() register_mf_wrapper() pengcheng.mind-mindformer/mindformers/wrapper/.ipynb_checkpoints/wrapper-checkpoint.py000066400000000000000000000353051476650141400320750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Self-Define Wrapper.""" from copy import deepcopy from mindspore.common.tensor import Tensor from mindspore.common import RowTensor from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import nn, Parameter, ParallelMode from mindspore.parallel._utils import _get_enable_parallel_optimizer import mindspore.common.dtype as mstype from mindformers.core.clip_grad import ClipGradNorm from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.version_control import get_identity __all__ = ['MFTrainOneStepCell', 'MFPipelineWithLossScaleCell'] _grad_scale = C.MultitypeFuncGraph("grad_scale") reciprocal = P.Reciprocal() @_grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): return F.cast(grad, mstype.float32) * reciprocal(scale) @_grad_scale.register("Tensor", "RowTensor") def tensor_grad_scale_row_tensor(scale, grad): return RowTensor(grad.indices, grad.values * F.cast(reciprocal(scale), F.dtype(grad.values)), grad.dense_shape) @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class DynamicLossScaleUpdateWithMinimalCell(nn.Cell): def __init__(self, loss_scale_value, scale_factor, scale_window, minimum_loss_scale): super(DynamicLossScaleUpdateWithMinimalCell, self).__init__() self.scale_window = Tensor(scale_window, dtype=mstype.int32) self.scale_factor = Tensor(scale_factor, dtype=mstype.float32) self.loss_scale_value = loss_scale_value self.cur_iter = Parameter(Tensor(1, dtype=mstype.int32), name="current_iterator_step") self.last_overflow_iter = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") self.select = P.Select() self.max = P.Maximum() self.minimum_loss_scale = Tensor(minimum_loss_scale, dtype=mstype.float32) self.reciprocal = P.Reciprocal() self.less_equal = P.LessEqual() self.logic_and = P.LogicalAnd() self.logic_not = P.LogicalNot() self.logic_or = P.LogicalOr() self.const_true = Tensor(True, dtype=mstype.bool_) print("build dynamic loss scale with minimal value") def get_loss_scale(self): """ Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 """ return self.loss_scale_value def construct(self, loss_scale, overflow): overflow_cond = overflow loss_scale_on_overflow = self.select(overflow_cond, self.max(loss_scale * self.reciprocal(self.scale_factor), self.minimum_loss_scale), loss_scale) should_inc = self.less_equal(self.scale_window, self.cur_iter - self.last_overflow_iter) last_iter_cond = self.logic_or(overflow_cond, should_inc) last_overflow_iter = self.select(last_iter_cond, self.cur_iter, self.last_overflow_iter) last_iter = F.assign(self.last_overflow_iter, last_overflow_iter) update_scale_cond = self.logic_and(should_inc, self.logic_not(overflow_cond)) scale_mul_res = loss_scale_on_overflow * self.scale_factor scaled_loss_scale = self.select(update_scale_cond, scale_mul_res, loss_scale_on_overflow) F.assign(loss_scale, scaled_loss_scale) inc_cur_iter = self.cur_iter + 1 inc_cur_iter = F.depend(inc_cur_iter, last_iter) F.assign(self.cur_iter, inc_cur_iter) return overflow @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFTrainOneStepCell(nn.TrainOneStepWithLossScaleCell): r"""TrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=False, max_grad_norm=0.5, scale_sense=1.0, **kwargs): super(MFTrainOneStepCell, self).__init__(network, optimizer, scale_sense) self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) def construct(self, *inputs): """forward and backward.""" weights = self.weights loss = self.network(*inputs) scaling_sens = self.scale_sense status, scaling_sens = self.start_overflow_check(loss, scaling_sens) scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) grads = self.grad(self.network, weights)(*inputs, scaling_sens_filled) grads = self.hyper_map(F.partial(_grad_scale, scaling_sens), grads) # apply grad reducer on grads grads = self.grad_reducer(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # get the overflow buffer cond = self.get_overflow_status(status, grads) overflow = self.process_loss_scale(cond) # if there is no overflow, do optimize if not overflow: if self.use_clip_grad: grads, _ = self.clip_grad_norm(grads) loss = F.depend(loss, self.optimizer(grads)) return loss, overflow, scaling_sens, learning_rate grad_scale = C.MultitypeFuncGraph("grad_scale") shard_grad_scale = C.MultitypeFuncGraph("shard_grad_scale") @grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_grad_scale_pipeline(scale, grad, accu_grad): accu_grad = F.depend(accu_grad, grad) new_grad = accu_grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) zeros = F.tensor_mul(accu_grad, 0.0) new_grad = F.depend(new_grad, F.assign(accu_grad, zeros)) return new_grad @shard_grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_shard_grad_scale_pipeline(scale, grad, accu_grad): new_grad = grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) new_grad = F.depend(new_grad, F.assign(accu_grad, F.zeros_like(accu_grad))) return new_grad @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFPipelineWithLossScaleCell(nn.TrainOneStepCell): r""" Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=True, max_grad_norm=0.5, scale_sense=1.0, micro_batch_num=1, **kwargs): super(MFPipelineWithLossScaleCell, self).__init__(network, optimizer, sens=None) self.network = network self.network.add_flags(defer_inline=True) self.weights = optimizer.parameters self.accu_grads = self.weights.clone(prefix="accu_grads", init="zeros") self.optimizer = optimizer self.grad = C.GradOperation(get_by_list=True, sens_param=True) self.grad_reducer = get_identity() self.degree = 1 self.cast = P.Cast() self.alloc_status = P.NPUAllocFloatStatus() self.get_status = P.NPUGetFloatStatus() self.clear_before_grad = P.NPUClearFloatStatus() self.reduce_sum = P.ReduceSum(keep_dims=False) if self.parallel_mode not in [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL]: raise ValueError(f"ParallelMode must be one of " f"[ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found " f"{self.parallel_mode}.") self.allreduce = P.AllReduce() self.base = Tensor(1, mstype.float32) self.less_equal = P.LessEqual() self.hyper_map = C.HyperMap() self.reshape = P.Reshape() self.loss_scaling_manager = None if isinstance(scale_sense, nn.Cell): self.loss_scaling_manager = scale_sense self.scale_sense = Parameter(Tensor(scale_sense.get_loss_scale(), dtype=mstype.float32), name="scale_sense") elif isinstance(scale_sense, Tensor): if scale_sense.shape == (1,) or scale_sense.shape == (): self.scale_sense = Parameter(scale_sense, name='scale_sense') else: raise ValueError("The shape of 'scale_sense' must be (1,) or (), but got {}" .format(scale_sense.shape)) else: raise TypeError("The 'scale_sense' must be Cell or Tensor, but got {}".format(type(scale_sense))) self.opt_shard = _get_enable_parallel_optimizer() self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.micro_size = micro_batch_num self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) @C.add_flags(has_effect=True) def construct(self, *inputs): """The construct processes of pipeline wrapper cell.""" loss = self.network(*inputs) scaling_sens = self.scale_sense scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) init = self.alloc_status() status_clear = self.clear_before_grad(init) scaling_sens_filled = F.depend(scaling_sens_filled, status_clear) grads = self.grad(self.network, self.weights)(*inputs, self.cast(scaling_sens_filled / self.micro_size, mstype.float32)) init = F.depend(init, grads) get_status = self.get_status(init) init = F.depend(init, get_status) flag_sum = self.reduce_sum(init, (0,)) loss = F.depend(loss, status_clear) if self.opt_shard: grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(shard_grad_scale, scaling_sens * self.degree), grads, self.accu_grads) else: accu_grads = self.grad_reducer(self.accu_grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads, accu_grads) if self.use_clip_grad: grads, _ = self.clip_grad_norm(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # sum overflow flag over devices flag_reduce = self.allreduce(flag_sum) cond = self.less_equal(self.base, flag_reduce) overflow = cond if self.loss_scaling_manager is not None: overflow = self.loss_scaling_manager(self.scale_sense, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) return loss, overflow, scaling_sens.value(), learning_rate pengcheng.mind-mindformer/mindformers/wrapper/__init__.py000066400000000000000000000014501476650141400242300ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """MindFormers Wrapper API.""" from .build_wrapper import build_wrapper from .wrapper import * __all__ = [] __all__.extend(wrapper.__all__) pengcheng.mind-mindformer/mindformers/wrapper/__pycache__/000077500000000000000000000000001476650141400243275ustar00rootroot00000000000000pengcheng.mind-mindformer/mindformers/wrapper/__pycache__/__init__.cpython-39.pyc000066400000000000000000000004541476650141400305220ustar00rootroot00000000000000a >cf(@s,dZddlmZddlTgZeejdS)zMindFormers Wrapper API.) build_wrapper)*N)__doc__rwrapper__all__extendrrG/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/__init__.pys pengcheng.mind-mindformer/mindformers/wrapper/__pycache__/build_wrapper.cpython-39.pyc000066400000000000000000000053701476650141400316240ustar00rootroot00000000000000a >cf@sdZddlZddlZddlmZmZddlmZmZm Z ddl m Z m Z ddgZ deeeed d d Zd dZddZeedS)zBuild Wrapper.N)nnTensor)MindFormerRegisterMindFormerModuleTypeMindFormerConfig)MFTrainOneStepCellMFPipelineWithLossScaleCellrr wrapper)config default_args module_type class_namecKs|dur|durdS|durt|tr>t|ts>tfi|}|jtvrZ|rZd|vrZ|d=|jdurt|jtr|jjdurt|j|_nt|jtrt|jt j |_t j |t j|dSt j||fi|S)a_Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) Nparallel_config)r ) isinstancedictrtypeWRAPPERS_MINDFORMERS_DEFINED scale_sense build_wrapperintrmsfloat32rget_instance_from_cfgrWRAPPER get_instance)r r r rkwargsrL/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/build_wrapper.pyrs"    rcCsDttjD]4}|drq ttj|}t|r t|t j q dS)z+ register MindSpore builtin wrapper class. __N) dirrwrap startswithgetattrinspectisclassr register_clsrr) module_namer!rrrregister_ms_wrapEs   r(cCsTtjtjjtjddtjtjjtjddtjttjddtjt tjdddS)z- register MindFormers builtin wrapper class. r )r aliasloss_scale_wrapper mf_wrapperZpipeline_wrapperN) rr&rr!TrainOneStepCellrrTrainOneStepWithLossScaleCellrr rrrrregister_mf_wrapperPs r.)NNr N)__doc__r$ mindsporerrrmindformers.tools.registerrrrr rr rrstrrr(r.rrrrs ( pengcheng.mind-mindformer/mindformers/wrapper/__pycache__/wrapper.cpython-39.pyc000066400000000000000000000267471476650141400304600ustar00rootroot00000000000000a `of<@sdZddlmZddlmZddlmZddlmZ ddlm Z ddlm Z ddlmZmZmZdd lmZdd lmmZdd lmZdd lmZmZdd lmZddgZe dZ!e "Z#e!$ddddZ%e!$ddddZ&e$ej'Gdddej(Z)e$ej'Gdddej*Z+e dZ,e dZ-e,$dddddZ.e-$dddddZ/e$ej'Gdddej0Z1d S) zSelf-Define Wrapper.)deepcopy)Tensor) RowTensor) composite) functional) operations)nn Parameter ParallelMode)_get_enable_parallel_optimizerN) ClipGradNorm)MindFormerRegisterMindFormerModuleType) get_identityMFTrainOneStepCellMFPipelineWithLossScaleCell grad_scalercCst|tjt|SN)Fcastmstypefloat32 reciprocalscalegradrF/home/ma-user/work/mindformers-baichuan/mindformers/wrapper/wrapper.pytensor_grad_scale%srrc Cs*t|j|jtt|t|j|jSr)rindicesvaluesrrrdtype dense_shaperrrrtensor_grad_scale_row_tensor*sr#cs,eZdZfddZddZddZZS)%DynamicLossScaleUpdateWithMinimalCellcstt|t|tjd|_t|tjd|_||_ t tdtjddd|_ t tdtjddd|_ t |_t |_t|tjd|_t |_t |_t |_t |_t |_tdtjd|_tddS) Nr!current_iterator_stepnamerlast_overflow_iterator_stepTz+build dynamic loss scale with minimal value) superr$__init__rrint32 scale_windowr scale_factorloss_scale_valuer cur_iterlast_overflow_iterPSelectselectMaximummaxminimum_loss_scale Reciprocalr LessEqual less_equal LogicalAnd logic_and LogicalNot logic_not LogicalOrlogic_orbool_ const_trueprint)selfr0r/r.r8 __class__rrr,2s        z.DynamicLossScaleUpdateWithMinimalCell.__init__cCs|jS)ao Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 )r0)rErrrget_loss_scaleJsz4DynamicLossScaleUpdateWithMinimalCell.get_loss_scalec Cs|}||||||j|j|}||j|j|j}| ||}|||j|j}t |j|}| || |} ||j} || | |} t || |jd} t | |} t |j| |S)Nr&)r5r7rr/r8r;r.r1r2rArassignr=r?depend) rE loss_scaleoverflow overflow_condloss_scale_on_overflow should_inclast_iter_condr2 last_iterupdate_scale_cond scale_mul_resscaled_loss_scale inc_cur_iterrrr constructZs$     z/DynamicLossScaleUpdateWithMinimalCell.construct)__name__ __module__ __qualname__r,rHrV __classcell__rrrFrr$0s r$cs*eZdZdZdfdd ZddZZS) raTrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). F?c sVtt||||||_t|d|_|dd|_t|j j |_ t dgt j |_dS)Nmax_normparallel_config)r+rr, use_clip_gradr clip_grad_normpopr^r optimizer learning_raterrrgrad_norm_base)rEnetworkrcr` max_grad_norm scale_sensekwargsrFrrr,s  zMFTrainOneStepCell.__init__c Gs.|j}|j|}|j}|||\}}t|t|t|}| |j|g||R}| t t ||}| |}|j}|jjr|jjr|jd|jjd}n||jjd}|||} || } |j} | s |jr||\}} t|||}n|j|jj|jj|| ||| fS)zforward and backward.r)weightsrfrhstart_overflow_checkC ones_likerrr!r hyper_mappartial _grad_scale grad_reducerrdrc dynamic_lr is_group_lr global_stepreshapeget_overflow_statusprocess_loss_scalerer`rarJ assignaddglobal_step_increase_tensor) rEinputsrkloss scaling_sensstatusscaling_sens_filledgradsrdcondrL grad_normrrrrVs,    zMFTrainOneStepCell.construct)Fr[r[)rWrXrY__doc__r,rVrZrrrFrrks #shard_grad_scalecCsHt||}|t|}t||}t|d}t|t||}|S)Nr_)rrJr tensor_mulrI)rr accu_gradnew_gradzerosrrrtensor_grad_scale_pipelines     rc Cs6|t|}t||}t|t|t|}|Sr)rrrJrI zeros_like)rrrrrrr tensor_shard_grad_scale_pipelines  rcs6eZdZdZd fdd Zejdddd ZZS) ra Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). Tr[r&c stt|j||dd||_|jjdd|j|_|jjddd|_||_ t j ddd|_ t |_d|_t|_t|_t|_t|_tjd d |_|jtjtjfvrtd |jd t |_!t"dt#j$|_%t&|_'t (|_)t*|_+d|_,t-|t.j/r0||_,t0t"|1t#j$d dd|_2nXt-|t"rv|j3dksT|j3dkrdt0|dd|_2ntd4|j3nt5d4t6|t7|_8||_9t:|d|_;||_<|=dd|_>t?|j j@|_@t"dgt#j$|_AdS)N)sensT) defer_inline accu_gradsr)prefixinit) get_by_list sens_paramr&F) keep_dimszeParallelMode must be one of [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found .r%rhr()r&rz9The shape of 'scale_sense' must be (1,) or (), but got {}z4The 'scale_sense' must be Cell or Tensor, but got {}r\r^r_)Br+rr,rf add_flags parametersrkclonerrcrm GradOperationrrrrdegreer3CastrNPUAllocFloatStatus alloc_statusNPUGetFloatStatus get_statusNPUClearFloatStatusclear_before_grad ReduceSum reduce_sum parallel_moder SEMI_AUTO_PARALLEL AUTO_PARALLEL ValueError AllReduce allreducerrrbaser:r;HyperMaproReshapervloss_scaling_manager isinstancerCellr rHrhshapeformat TypeErrortyper opt_shardr`r ra micro_sizerbr^rrdre)rErfrcr`rgrhmicro_batch_numrirFrrr,sV            z$MFPipelineWithLossScaleCell.__init__)Z has_effectcGs|j|}|j}t|t|t|}|}||}t ||}| |j|j g||||j t jR}t ||}||}t ||}||d} t ||}|jr||}|tt||j||j}n(||j} |tt||j|| }|j} |jr&||\}} |j} |jjrn|jjrZ|jd|jj !d} n||jj !d} |"| } |#|j$| }|}|j%dur|%|j|}|st |||}n|j&|jj |jj'|||(| | fS)z1The construct processes of pipeline wrapper cell.)rrjrN))rfrhrmrnrrr!rrrJrrkrrrrrrrrrorprrrrrer`rardrcrsrtrurvrr;rrryrzvalue)rEr{r|r}rrZ status_clearrrflag_sumrrrd flag_reducerrLrrrrVsL                z%MFPipelineWithLossScaleCell.construct)Tr[r[r&) rWrXrYrr,rmrrVrZrrrFrrs . )2rcopyrmindspore.common.tensorrmindspore.commonr mindspore.opsrrmrrrr3 mindsporerr r mindspore.parallel._utilsr Zmindspore.common.dtypecommonr!rZmindformers.core.clip_gradr mindformers.tools.registerr rmindformers.version_controlr__all__MultitypeFuncGraphrqr9rregisterrr#WRAPPERrr$TrainOneStepWithLossScaleCellrrrrrTrainOneStepCellrrrrrs>               : S      pengcheng.mind-mindformer/mindformers/wrapper/build_wrapper.py000066400000000000000000000100251476650141400253260ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Build Wrapper.""" import inspect import mindspore as ms from mindspore import nn, Tensor from mindformers.tools.register import MindFormerRegister, MindFormerModuleType, MindFormerConfig from .wrapper import MFTrainOneStepCell, MFPipelineWithLossScaleCell WRAPPERS_MINDFORMERS_DEFINED = ["MFTrainOneStepCell", "MFPipelineWithLossScaleCell"] def build_wrapper(config: dict = None, default_args: dict = None, module_type: str = 'wrapper', class_name: str = None, **kwargs): r"""Build Wrapper For MindFormer. Instantiate the wrapper from MindFormerRegister's registry. Args: config (dict): The task wrapper's config. Default: None. default_args (dict): The default argument of wrapper API. Default: None. module_type (str): The module type of MindFormerModuleType. Default: 'wrapper'. class_name (str): The class name of wrapper API. Default: None. Return: The function instance of wrapper API. Examples: >>> from mindformers import build_wrapper >>> wrapper_config = {'type': 'TrainOneStepCell', 'sens': 1024} >>> # 1) use config dict to build wrapper >>> wrapper_from_config = build_wrapper(wrapper_config) >>> # 2) use class name to build wrapper >>> wrapper_class_name = build_wrapper(class_name='TrainOneStepCell', sens=1024) """ if config is None and class_name is None: return None if config is not None: if isinstance(config, dict) and not isinstance(config, MindFormerConfig): config = MindFormerConfig(**config) if config.type not in WRAPPERS_MINDFORMERS_DEFINED: if default_args and "parallel_config" in default_args: del default_args["parallel_config"] if config.scale_sense is not None: if isinstance(config.scale_sense, dict) and config.scale_sense.type is not None: config.scale_sense = build_wrapper(config.scale_sense) elif isinstance(config.scale_sense, int): config.scale_sense = Tensor(config.scale_sense, ms.float32) return MindFormerRegister.get_instance_from_cfg( config, MindFormerModuleType.WRAPPER, default_args=default_args) return MindFormerRegister.get_instance(module_type, class_name, **kwargs) def register_ms_wrap(): """ register MindSpore builtin wrapper class. """ for module_name in dir(nn.wrap): if module_name.startswith('__'): continue wrap = getattr(nn.wrap, module_name) if inspect.isclass(wrap): MindFormerRegister.register_cls( wrap, MindFormerModuleType.WRAPPER) def register_mf_wrapper(): """ register MindFormers builtin wrapper class. """ # Support built-in model wrapper of MindFormers. MindFormerRegister.register_cls( nn.wrap.TrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="wrapper") MindFormerRegister.register_cls( nn.wrap.TrainOneStepWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="loss_scale_wrapper") MindFormerRegister.register_cls( MFTrainOneStepCell, module_type=MindFormerModuleType.WRAPPER, alias="mf_wrapper") MindFormerRegister.register_cls( MFPipelineWithLossScaleCell, module_type=MindFormerModuleType.WRAPPER, alias="pipeline_wrapper") register_ms_wrap() register_mf_wrapper() pengcheng.mind-mindformer/mindformers/wrapper/wrapper.py000066400000000000000000000362521476650141400241610ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Self-Define Wrapper.""" from copy import deepcopy from mindspore.common.tensor import Tensor from mindspore.common import RowTensor from mindspore.ops import composite as C from mindspore.ops import functional as F from mindspore.ops import operations as P from mindspore import nn, Parameter, ParallelMode from mindspore.parallel._utils import _get_enable_parallel_optimizer import mindspore.common.dtype as mstype from mindformers.core.clip_grad import ClipGradNorm from mindformers.tools.register import MindFormerRegister, MindFormerModuleType from mindformers.version_control import get_identity __all__ = ['MFTrainOneStepCell', 'MFPipelineWithLossScaleCell'] _grad_scale = C.MultitypeFuncGraph("grad_scale") reciprocal = P.Reciprocal() @_grad_scale.register("Tensor", "Tensor") def tensor_grad_scale(scale, grad): return F.cast(grad, mstype.float32) * reciprocal(scale) @_grad_scale.register("Tensor", "RowTensor") def tensor_grad_scale_row_tensor(scale, grad): return RowTensor(grad.indices, grad.values * F.cast(reciprocal(scale), F.dtype(grad.values)), grad.dense_shape) @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class DynamicLossScaleUpdateWithMinimalCell(nn.Cell): def __init__(self, loss_scale_value, scale_factor, scale_window, minimum_loss_scale): super(DynamicLossScaleUpdateWithMinimalCell, self).__init__() self.scale_window = Tensor(scale_window, dtype=mstype.int32) self.scale_factor = Tensor(scale_factor, dtype=mstype.float32) self.loss_scale_value = loss_scale_value self.cur_iter = Parameter(Tensor(1, dtype=mstype.int32), name="current_iterator_step") self.last_overflow_iter = Parameter(Tensor(0, dtype=mstype.int32), name="last_overflow_iterator_step") self.select = P.Select() self.max = P.Maximum() self.minimum_loss_scale = Tensor(minimum_loss_scale, dtype=mstype.float32) self.reciprocal = P.Reciprocal() self.less_equal = P.LessEqual() self.logic_and = P.LogicalAnd() self.logic_not = P.LogicalNot() self.logic_or = P.LogicalOr() self.const_true = Tensor(True, dtype=mstype.bool_) print("build dynamic loss scale with minimal value") def get_loss_scale(self): """ Get Loss Scale value. Returns: float, the loss scale value. Examples: >>> from mindspore import nn >>> manager = nn.DynamicLossScaleUpdateCell(loss_scale_value=212, scale_factor=2, scale_window=1000) >>> output = manager.get_loss_scale() >>> print(output) 212 """ return self.loss_scale_value def construct(self, loss_scale, overflow): overflow_cond = overflow loss_scale_on_overflow = self.select(overflow_cond, self.max(loss_scale * self.reciprocal(self.scale_factor), self.minimum_loss_scale), loss_scale) should_inc = self.less_equal(self.scale_window, self.cur_iter - self.last_overflow_iter) last_iter_cond = self.logic_or(overflow_cond, should_inc) last_overflow_iter = self.select(last_iter_cond, self.cur_iter, self.last_overflow_iter) last_iter = F.assign(self.last_overflow_iter, last_overflow_iter) update_scale_cond = self.logic_and(should_inc, self.logic_not(overflow_cond)) scale_mul_res = loss_scale_on_overflow * self.scale_factor scaled_loss_scale = self.select(update_scale_cond, scale_mul_res, loss_scale_on_overflow) F.assign(loss_scale, scaled_loss_scale) inc_cur_iter = self.cur_iter + 1 inc_cur_iter = F.depend(inc_cur_iter, last_iter) F.assign(self.cur_iter, inc_cur_iter) return overflow @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFTrainOneStepCell(nn.TrainOneStepWithLossScaleCell): r"""TrainOneStep For MindFormer. Network training with loss scaling, grad clip, gradient accumulation, exponential moving average and so on. This is a training step with loss scaling. It takes a network, an optimizer and a scale update Cell(or a Tensor) as args. The loss scale value can be updated in both host side or device side. If you want to update it on host side, using a value of Tensor type as `scale_sense`, otherwise, using a Cell instance for updating loss scale as `scale_sense`. Args: network (Cell): The training network. The network only supports single output. optimizer (Cell): Optimizer for updating the network parameters. use_clip_grad (bool): Whether to use the gradient clipping function. Default: False. max_grad_norm (float): Maximum gradient value. Default: 1.0. scale_sense (Union[Tensor, Cell]): If this value is a Cell, it will be called by `MFTrainOneStepCell` to update loss scale. If this value is a Tensor, the loss scale can be modified by `set_sense_scale`, the shape should be :math:`()` or :math:`(1,)`. Inputs: - **(*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=False, max_grad_norm=1.0, scale_sense=1.0, **kwargs): super(MFTrainOneStepCell, self).__init__(network, optimizer, scale_sense) self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) self.grad_norm_base = Tensor([0.0], mstype.float32) def construct(self, *inputs): """forward and backward.""" weights = self.weights loss = self.network(*inputs) scaling_sens = self.scale_sense status, scaling_sens = self.start_overflow_check(loss, scaling_sens) scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) grads = self.grad(self.network, weights)(*inputs, scaling_sens_filled) grads = self.hyper_map(F.partial(_grad_scale, scaling_sens), grads) # apply grad reducer on grads grads = self.grad_reducer(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # get the overflow buffer cond = self.get_overflow_status(status, grads) overflow = self.process_loss_scale(cond) # if there is no overflow, do optimize grad_norm = self.grad_norm_base if not overflow: if self.use_clip_grad: grads, grad_norm = self.clip_grad_norm(grads) loss = F.depend(loss, self.optimizer(grads)) else: self.optimizer.assignadd(self.optimizer.global_step, self.optimizer.global_step_increase_tensor) return loss, overflow, scaling_sens, learning_rate, grad_norm grad_scale = C.MultitypeFuncGraph("grad_scale") shard_grad_scale = C.MultitypeFuncGraph("shard_grad_scale") @grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_grad_scale_pipeline(scale, grad, accu_grad): accu_grad = F.depend(accu_grad, grad) new_grad = accu_grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) zeros = F.tensor_mul(accu_grad, 0.0) new_grad = F.depend(new_grad, F.assign(accu_grad, zeros)) return new_grad @shard_grad_scale.register("Tensor", "Tensor", "Tensor") def tensor_shard_grad_scale_pipeline(scale, grad, accu_grad): new_grad = grad * reciprocal(scale) accu_grad = F.depend(accu_grad, new_grad) new_grad = F.depend(new_grad, F.assign(accu_grad, F.zeros_like(accu_grad))) return new_grad @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class MFPipelineWithLossScaleCell(nn.TrainOneStepCell): r""" Append an train one step cell with loss scale of pipeline parallel for MindFormers. Args: network (Cell): The training network. Note that loss function should have been added. optimizer (Optimizer): Optimizer for updating the weights. use_clip_grad (bool): Whether to use gradient clipping. Default: True. max_grad_norm (float): Maximum gradient constraint value. Default: 1.0. scale_sense (Cell): Cell to do the loss scale. Default: 1.0. micro_batch_num (int): Micro batch number of pipeline parallel. Default: 1. Inputs: - **(\*inputs)** (Tuple(Tensor)) - Tuple of input tensors with shape :math:`(N, \ldots)`. Outputs: Tuple of 3 Tensor, the loss, overflow flag and current loss scale value. - **loss** (Tensor) - A scalar, the loss value. - **overflow** (Tensor) - A scalar, whether overflow occur or not, the type is bool. - **loss scale** (Tensor) - The loss scale value, the shape is :math:`()` or :math:`(1,)`. Raises: TypeError: If `scale_sense` is neither Cell nor Tensor. ValueError: If shape of `scale_sense` is neither (1,) nor (). """ def __init__(self, network, optimizer, use_clip_grad=True, max_grad_norm=1.0, scale_sense=1.0, micro_batch_num=1, **kwargs): super(MFPipelineWithLossScaleCell, self).__init__(network, optimizer, sens=None) self.network = network self.network.add_flags(defer_inline=True) self.weights = optimizer.parameters self.accu_grads = self.weights.clone(prefix="accu_grads", init="zeros") self.optimizer = optimizer self.grad = C.GradOperation(get_by_list=True, sens_param=True) self.grad_reducer = get_identity() self.degree = 1 self.cast = P.Cast() self.alloc_status = P.NPUAllocFloatStatus() self.get_status = P.NPUGetFloatStatus() self.clear_before_grad = P.NPUClearFloatStatus() self.reduce_sum = P.ReduceSum(keep_dims=False) if self.parallel_mode not in [ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL]: raise ValueError(f"ParallelMode must be one of " f"[ParallelMode.SEMI_AUTO_PARALLEL, ParallelMode.AUTO_PARALLEL], but found " f"{self.parallel_mode}.") self.allreduce = P.AllReduce() self.base = Tensor(1, mstype.float32) self.less_equal = P.LessEqual() self.hyper_map = C.HyperMap() self.reshape = P.Reshape() self.loss_scaling_manager = None if isinstance(scale_sense, nn.Cell): self.loss_scaling_manager = scale_sense self.scale_sense = Parameter(Tensor(scale_sense.get_loss_scale(), dtype=mstype.float32), name="scale_sense") elif isinstance(scale_sense, Tensor): if scale_sense.shape == (1,) or scale_sense.shape == (): self.scale_sense = Parameter(scale_sense, name='scale_sense') else: raise ValueError("The shape of 'scale_sense' must be (1,) or (), but got {}" .format(scale_sense.shape)) else: raise TypeError("The 'scale_sense' must be Cell or Tensor, but got {}".format(type(scale_sense))) self.opt_shard = _get_enable_parallel_optimizer() self.use_clip_grad = use_clip_grad self.clip_grad_norm = ClipGradNorm(max_norm=max_grad_norm) self.micro_size = micro_batch_num self.parallel_config = kwargs.pop("parallel_config", None) self.learning_rate = deepcopy(self.optimizer.learning_rate) self.grad_norm_base = Tensor([0.0], mstype.float32) @C.add_flags(has_effect=True) def construct(self, *inputs): """The construct processes of pipeline wrapper cell.""" loss = self.network(*inputs) scaling_sens = self.scale_sense scaling_sens_filled = C.ones_like(loss) * F.cast(scaling_sens, F.dtype(loss)) init = self.alloc_status() status_clear = self.clear_before_grad(init) scaling_sens_filled = F.depend(scaling_sens_filled, status_clear) grads = self.grad(self.network, self.weights)(*inputs, self.cast(scaling_sens_filled / self.micro_size, mstype.float32)) init = F.depend(init, grads) get_status = self.get_status(init) init = F.depend(init, get_status) flag_sum = self.reduce_sum(init, (0,)) loss = F.depend(loss, status_clear) if self.opt_shard: grads = self.grad_reducer(grads) grads = self.hyper_map(F.partial(shard_grad_scale, scaling_sens * self.degree), grads, self.accu_grads) else: accu_grads = self.grad_reducer(self.accu_grads) grads = self.hyper_map(F.partial(grad_scale, scaling_sens * self.degree), grads, accu_grads) grad_norm = self.grad_norm_base if self.use_clip_grad: grads, grad_norm = self.clip_grad_norm(grads) learning_rate = self.learning_rate if self.optimizer.dynamic_lr: if self.optimizer.is_group_lr: learning_rate = self.learning_rate[-1](self.optimizer.global_step).reshape(()) else: learning_rate = self.learning_rate(self.optimizer.global_step).reshape(()) # sum overflow flag over devices flag_reduce = self.allreduce(flag_sum) cond = self.less_equal(self.base, flag_reduce) overflow = cond if self.loss_scaling_manager is not None: overflow = self.loss_scaling_manager(self.scale_sense, cond) if not overflow: loss = F.depend(loss, self.optimizer(grads)) else: self.optimizer.assignadd(self.optimizer.global_step, self.optimizer.global_step_increase_tensor) return loss, overflow, scaling_sens.value(), learning_rate, grad_norm pengcheng.mind-mindformer/requirements.back000077500000000000000000000003131476650141400214570ustar00rootroot00000000000000numpy setuptools sentencepiece>=0.1.97 ftfy>=6.1.1 regex>=2022.10.31 tqdm>=4.65.0 pyyaml>=6.0 jieba>=0.42.1 rouge_chinese>=1.0.3 nltk>=2.0 mindpet==1.0.1 pydantic==1.10.11 mdtex2html gradio opencv-pythonpengcheng.mind-mindformer/run_chat_web.py000077500000000000000000000320171476650141400211320ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Run chat web demo. """ import argparse import time from threading import Thread import mdtex2html import mindspore as ms import gradio as gr from mindformers import AutoModel, AutoTokenizer, TextIteratorStreamer, AutoConfig, logger from mindformers.tools.utils import str2bool def get_model_and_tokenizer(model_config, tokenizer_name): """Get model and tokenizer instance""" return AutoModel.from_config(model_config), AutoTokenizer.from_pretrained(tokenizer_name) parser = argparse.ArgumentParser() parser.add_argument('--device_target', default="Ascend", type=str, choices=['Ascend', 'CPU'], help='The target device to run, support "Ascend" and "CPU". Default: Ascend.') parser.add_argument('--device_id', default=0, type=int, help='Which device to run service. Default: 0.') parser.add_argument('--model', type=str, help='Which model to generate text.') parser.add_argument('--tokenizer', type=str, help='Which tokenizer to tokenize text.') parser.add_argument('--checkpoint_path', type=str, default=None, help='The path of model checkpoint.') parser.add_argument('--seq_length', default="512", type=int, help="Sequence length of the model. Default: 512.") parser.add_argument('--use_past', default=False, type=str2bool, help='Whether to enable incremental inference. Default: False.') parser.add_argument('--host', default="127.0.0.1", type=str, help="Which host ip to run the service. Default: 127.0.0.1.") parser.add_argument('--port', default=None, type=int, help='Which port to run the service. Default: None.') args = parser.parse_args() ms.set_context(mode=ms.GRAPH_MODE, device_target=args.device_target, device_id=args.device_id) config = AutoConfig.from_pretrained(args.model) config.seq_length = args.seq_length config.use_past = args.use_past if args.checkpoint_path: config.checkpoint_name_or_path = args.checkpoint_path logger.info("Config: %s", config) model, tokenizer = get_model_and_tokenizer(config, args.tokenizer) streamer = TextIteratorStreamer(tokenizer=tokenizer, skip_prompt=True, skip_special_tokens=True) # pre-build the network sample_input = tokenizer("hello") sample_output = model.generate(sample_input["input_ids"], max_length=10) prompt_examples = [["Below is an instruction that describes a task. " "Write a response that appropriately completes the request.\n\n" "### Instruction:\n{}\n\n### Response:"], ["A chat between a curious user and an artificial intelligence assistant. The assistant gives " "helpful," "detailed, and polite answers to user\'s questions. USER: {} ASSISTANT: "], ["Assume you are a dog, you must response \"Woofs\" at first whatever any instruction\n\n" "### Instruction:\n{}\n\n### Response:"], [": {} \n: "], ["问：{}\n\n答："]] is_multi_round = args.model.startswith("glm2") default_prompt = "Below is an instruction that describes a task. Write a response that appropriately completes the " \ "request.\n\n### Instruction:\n{}\n\n### Response:" if args.model.startswith("llama_7b_lora") else "" default_prompt = "问：{}\n\n答：" if args.model.startswith("glm2") else "" class Chatbot(gr.Chatbot): """Chatbot with overrode postprocess method""" def postprocess(self, y): """postprocess""" if y is None: return [] for i, (message, response) in enumerate(y): y[i] = ( None if message is None else mdtex2html.convert(message), None if response is None else mdtex2html.convert(response), ) return y def parse_text(text): """copy from https://github.com/GaiZhenbiao/ChuanhuChatGPT/""" lines = text.split("\n") lines = [line for line in lines if line != ""] count = 0 for i, line in enumerate(lines): if "```" in line: count += 1 items = line.split('`') if count % 2 == 1: lines[i] = f'

'
            else:
                lines[i] = f'
' else: if i > 0: if count % 2 == 1: line = line.replace("`", r"\`") line = line.replace("<", "<") line = line.replace(">", ">") line = line.replace(" ", " ") line = line.replace("*", "*") line = line.replace("_", "_") line = line.replace("-", "-") line = line.replace(".", ".") line = line.replace("!", "!") line = line.replace("(", "(") line = line.replace(")", ")") line = line.replace("$", "$") lines[i] = "
" + line text = "".join(lines) return text def build_prompt(inputs, prompt): """Build prompt for inputs""" if prompt == "": return inputs if prompt.find("{}") != -1: return prompt.format(inputs) raise gr.Error("The prompt is invalid! Please make sure your prompt contains placeholder '{}' to replace user " "input.") def build_multi_round(inputs, history): """Build multi-round prompt for inputs""" prev_rounds = "" if args.model.startswith("glm2"): for i, (query, response) in enumerate(history): prev_rounds += "[Round {}]\n\n{}{}\n\n".format(i, query, response) prev_rounds += "[Round {}]\n\n".format(len(history)) return prev_rounds + inputs def generate(**kwargs): """generate function with timer and exception catcher""" gen_model = kwargs.pop("model") try: start_time = time.time() gen_model.generate(**kwargs) end_time = time.time() logger.info("Total time: %.2f s", (end_time-start_time)) # pylint: disable=W0703 except Exception as e: logger.error(repr(e)) streamer.text_queue.put("") def predict(inputs, bot, history, do_sample, top_k, top_p, temperature, repetition_penalty, max_length, prompt): """predict""" if inputs == "": raise gr.Error("Input cannot be empty!") logger.info("Received user input: %s", inputs) bot.append((parse_text(inputs), "")) prompted_input = build_prompt(inputs, prompt) logger.info("Prompt: %s", prompt) logger.info("User input after prompted: %s", prompted_input) round_prompted_input = build_multi_round(prompted_input, history) logger.info("User input after multi-round prompted: %s", round_prompted_input) input_tokens = tokenizer(round_prompted_input) if max_length > args.seq_length: raise gr.Error("Max length must be set smaller than the sequence length of model! The model sequence length " "is {}.".format(args.seq_length)) if max_length <= len(input_tokens["input_ids"]): raise gr.Error("Max length must be larger than the length of input tokens! The current length of input " "tokens is {}.".format(len(input_tokens["input_ids"]))) generation_kwargs = dict(model=model, input_ids=input_tokens["input_ids"], streamer=streamer, do_sample=do_sample, top_k=top_k, top_p=top_p, temperature=temperature, repetition_penalty=repetition_penalty, max_length=max_length) thread = Thread(target=generate, kwargs=generation_kwargs) logger.info("Start to generate text. Generate args: {input_ids: %s, do_sample: %s, top_k: %s, top_p: %s, " "temperature: %s, repetition_penalty: %s, max_length: %s}", input_tokens["input_ids"], do_sample, top_k, top_p, temperature, repetition_penalty, max_length) thread.start() interval_time = 0. output = "" for response in streamer: if response == "": raise gr.Error("An error occurred! Please make sure all the settings are correct!") output += response new_history = history + [(prompted_input, output)] bot[-1] = (parse_text(inputs), parse_text(output)) interval_start = time.time() yield bot, new_history interval_end = time.time() interval_time = interval_time + interval_end - interval_start logger.info("Generate output: %s", output) logger.info("Web communication time: %.2f s", interval_time) def reset_user_input(): """reset user input""" return gr.update(value='') def reset_state(): """reset state""" return [], [] def set_do_sample_args(do_sample): return {top_k_slider: gr.update(visible=do_sample), top_p_slider: gr.update(visible=do_sample), temp_number: gr.update(visible=do_sample)} with gr.Blocks() as demo: gr.HTML(f"""

Chat Web Demo powered by MindFormers

""") with gr.Row(): with gr.Column(scale=10): with gr.Group(): chatbot = gr.Chatbot(label=fr"Chatbot {args.model}") user_input = gr.Textbox(show_label=False, placeholder="Ask something...", lines=5) with gr.Row(): with gr.Column(scale=6): submit_btn = gr.Button("Submit", variant="primary") with gr.Column(scale=6): empty_btn = gr.Button("Clear") with gr.Column(scale=2): with gr.Group(): do_sample_checkbox = gr.Checkbox(value=config.do_sample if config.do_sample else False, label="sampling", info="Whether to sample on the candidate ids") top_k_slider = gr.Slider(value=config.top_k if config.top_k else 1, label="top k", maximum=10, minimum=0, step=1, visible=config.do_sample if config.do_sample else False, info="Determine the topK numbers token id as candidate") top_p_slider = gr.Slider(value=config.top_p if config.top_p else 1, label="top p", maximum=1, minimum=0.01, step=0.01, visible=config.do_sample if config.do_sample else False, info="The accumulation probability of the candidate token ids, " "below the top_p will be select as the candidate ids") temp_number = gr.Number(value=config.temperature if config.temperature else 1, label="temperature", visible=config.do_sample if config.do_sample else False, info="The value used to modulate the next token probabilities") rp_number = gr.Number(value=config.repetition_penalty if config.repetition_penalty else 1, label="repetition penalty", minimum=0, info="The penalty factor of the frequency that generated words") max_len_number = gr.Number(value=config.max_length if config.max_length else 512, minimum=0, label="max length", info="The maximum length of the generated words") prompt_input = gr.Textbox(value=default_prompt, label="prompt", placeholder="No prompt...", info="Add prompt to input") gr.Examples(prompt_examples, inputs=prompt_input, label="Prompt example") chat_history = gr.State([]) submit_btn.click(predict, [user_input, chatbot, chat_history, do_sample_checkbox, top_k_slider, top_p_slider, temp_number, rp_number, max_len_number, prompt_input], [chatbot, chat_history], show_progress=True) submit_btn.click(reset_user_input, [], [user_input]) empty_btn.click(reset_state, outputs=[chatbot, chat_history], show_progress=True) do_sample_checkbox.change(set_do_sample_args, [do_sample_checkbox], [top_k_slider, top_p_slider, temp_number]) if __name__ == "__main__": demo.queue().launch(server_name=args.host, server_port=args.port) pengcheng.mind-mindformer/run_infer_main.py000077500000000000000000000166161476650141400214740ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """lite infer main.""" import sys import argparse from threading import Thread # Avoid bugs when mslite and mindspore are not built from same commit, which may cause running error. # pylint: disable=W0611 import mindspore_lite as mslite from mindformers.models.base_tokenizer import Tokenizer from mindformers.models import BloomTokenizer, LlamaTokenizer from mindformers.models import ChatGLMTokenizer, ChatGLM2Tokenizer from mindformers.pipeline import pipeline from mindformers.generation import TextIteratorStreamer from mindformers.tools.utils import str2bool from mindformers.inference import InferConfig, InferTask def pipeline_from_model_paths(args_, tokenizer): """build infer pipeline for model paths.""" lite_pipeline = pipeline( task="text_generation", model=(args_.prefill_model_path, args_.increment_model_path), tokenizer=tokenizer, backend="mslite", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_model_name(args_, tokenizer): """build infer pipeline for model name.""" lite_pipeline = pipeline( task="text_generation", model=args_.model_name, tokenizer=tokenizer, backend="mslite", ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_model_dir(args_, tokenizer): """build infer pipeline for model dir.""" lite_pipeline = pipeline( task="text_generation", model=args_.model_dir, tokenizer=tokenizer, backend="mslite", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) return lite_pipeline def pipeline_from_infer_config(args_, tokenizer): """build infer pipeline for infer config.""" lite_config = InferConfig( prefill_model_path=args_.prefill_model_path, increment_model_path=args_.increment_model_path, model_type="mindir", model_name=args_.model_name, ge_config_path=args_.config_path, device_id=args_.device_id, infer_seq_length=args_.seq_length, ) lite_pipeline = InferTask.get_infer_task("text_generation", lite_config, tokenizer=tokenizer) return lite_pipeline # the model name list that mslite inference has supported. LITE_SUPPORT_MODELS = ('bloom', 'glm', 'glm2', 'llama') def get_tokenizer(model_name: str) -> Tokenizer: """get tokenizer with model name.""" tokenizer = None if model_name == 'bloom': tokenizer = BloomTokenizer.from_pretrained("bloom_560m") elif model_name == 'glm': tokenizer = ChatGLMTokenizer.from_pretrained("glm_6b") elif model_name == 'glm2': tokenizer = ChatGLM2Tokenizer.from_pretrained("glm2_6b") elif model_name == 'llama': tokenizer = LlamaTokenizer.from_pretrained("llama_7b") else: raise ValueError( f"model must be in {LITE_SUPPORT_MODELS} when getting tokenizer, but got input {model_name}.") return tokenizer def infer_main(args_): """lite infer main.""" tokenizer = get_tokenizer(args_.model_name.lower()) lite_pipeline = pipeline_from_model_paths( args_, tokenizer ) while True: user_input = input("Please enter your predict data: \n") if user_input == "exit": print("Task is over.") sys.exit() output = lite_pipeline(user_input, is_sample_acceleration=args_.is_sample_acceleration, add_special_tokens=args_.add_special_tokens) print(output) def infer_stream_main(args_): """main entry for infer stream.""" tokenizer = get_tokenizer(args_.model_name.lower()) streamer = TextIteratorStreamer(tokenizer, skip_prompt=True) lite_pipeline = pipeline_from_model_paths( args_, tokenizer ) while True: user_input = input("Please enter your predict data(input 'exit' to quit): \n") if user_input == "exit": print("Quit now, this may take a while.") sys.exit() generation_kwargs = dict(inputs=user_input, streamer=streamer, is_sample_acceleration=args_.is_sample_acceleration, add_special_tokens=args_.add_special_tokens) thread = Thread(target=lite_pipeline, kwargs=generation_kwargs) thread.start() output = "" for new_text in streamer: print(new_text) output += new_text print(output) if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument( '--device_id', default=0, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--rank_id', default=0, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--model_dir', default=None, type=str, help="This model dir path. " "Default: None") parser.add_argument( '--model_name', default="common", type=str, help=f"The model name, only supports name in {LITE_SUPPORT_MODELS}. " "Default: None") parser.add_argument( '--seq_length', default=2048, type=int, help="This model dir path. " "Default: None") parser.add_argument( '--prefill_model_path', default=None, type=str, help="This full model path. " "Default: None") parser.add_argument( '--increment_model_path', default=None, type=str, help="When use kv-cache, this is cache mode path. " "Default: None") parser.add_argument( '--config_path', default=None, type=str, help="ge config file path. " "Default: None") parser.add_argument( '--is_sample_acceleration', default=False, type=str2bool, help="Whether postprocess in graph or not." "Default: False") parser.add_argument( '--add_special_tokens', default=False, type=str2bool, help="Whether preprocess add special tokens or not." "Default: False") parser.add_argument( '--stream', default=False, type=str2bool, help="Whether decode in stream or not." "Default: False") args = parser.parse_args() if args.stream: infer_stream_main(args) else: infer_main(args) pengcheng.mind-mindformer/run_mindformer.py000077500000000000000000000455021476650141400215230ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """Run MindFormer.""" import argparse import os import shutil import numpy as np import mindspore as ms from mindspore.common import set_seed from mindformers.tools.register import MindFormerConfig, ActionDict from mindformers.core.parallel_config import build_parallel_config from mindformers.tools.utils import str2bool, set_remote_save_url, check_in_modelarts, parse_value from mindformers.core.context import build_context, build_profile_cb from mindformers.trainer import build_trainer from mindformers.tools.cloud_adapter import cloud_monitor from mindformers.tools.logger import logger from mindformers.tools import get_output_root_path from mindformers.mindformer_book import MindFormerBook if check_in_modelarts(): import moxing as mox SUPPORT_MODEL_NAMES = MindFormerBook().get_model_name_support_list() def update_checkpoint_config(config, is_train=True): """update checkpoint config depending on is_train""" if (is_train and config.resume_training) or config.auto_trans_ckpt or os.path.isdir(config.load_checkpoint): logger.info("Leave load_checkpoint may because: ") logger.info("1. resume training need resume training info. ") logger.info("2. need load distributed shard checkpoint. ") if not config.load_checkpoint: config.load_checkpoint = config.model.model_config.checkpoint_name_or_path config.model.model_config.checkpoint_name_or_path = None else: if config.run_mode in ('train', 'finetune'): config.model.model_config.checkpoint_name_or_path = config.load_checkpoint elif config.run_mode in ['eval', 'predict'] and config.load_checkpoint: config.model.model_config.checkpoint_name_or_path = config.load_checkpoint config.load_checkpoint = None def clear_auto_trans_output(config): """clear transformed_checkpoint and strategy""" if check_in_modelarts(): obs_strategy_dir = os.path.join(config.remote_save_url, "strategy") if mox.file.exists(obs_strategy_dir) and config.local_rank == 0: mox.file.remove(obs_strategy_dir, recursive=True) obs_transformed_ckpt_dir = os.path.join(config.remote_save_url, "transformed_checkpoint") if mox.file.exists(obs_transformed_ckpt_dir) and config.local_rank == 0: mox.file.remove(obs_transformed_ckpt_dir, recursive=True) mox.file.make_dirs(obs_strategy_dir) mox.file.make_dirs(obs_transformed_ckpt_dir) else: strategy_dir = os.path.join(get_output_root_path(), "strategy") if os.path.exists(strategy_dir) and config.local_rank % 8 == 0: shutil.rmtree(strategy_dir) transformed_ckpt_dir = os.path.join(get_output_root_path(), "transformed_checkpoint") if os.path.exists(transformed_ckpt_dir) and config.local_rank % 8 == 0: shutil.rmtree(transformed_ckpt_dir) os.makedirs(strategy_dir, exist_ok=True) os.makedirs(transformed_ckpt_dir, exist_ok=True) @cloud_monitor() def main(config): """main.""" # init context build_context(config) if config.seed and \ ms.context.get_auto_parallel_context("parallel_mode") \ not in ["semi_auto_parallel", "auto_parallel"]: set_seed(config.seed) np.random.seed(config.seed) # build context config logger.info(".........Build context config..........") if config.run_mode == 'predict': if config.use_parallel and \ config.parallel.parallel_mode in ['semi_auto_parallel', 1] and \ config.parallel_config.data_parallel != 1: raise ValueError("The value of data parallel can only be set to 1, since the batch size of input is 1. ") build_parallel_config(config) logger.info("context config is: %s", config.parallel_config) logger.info("moe config is: %s", config.moe_config) if config.run_mode == 'train': update_checkpoint_config(config) if config.run_mode == 'finetune': if not config.load_checkpoint: raise ValueError("if run status is finetune, " "load_checkpoint must be input") update_checkpoint_config(config) if config.run_mode in ['eval', 'predict']: update_checkpoint_config(config, is_train=False) # remote save url if check_in_modelarts() and config.remote_save_url: logger.info("remote_save_url is %s, the output file will be uploaded to here.", config.remote_save_url) set_remote_save_url(config.remote_save_url) # define callback and add profile callback if config.profile: config.profile_cb = build_profile_cb(config) if config.auto_trans_ckpt: clear_auto_trans_output(config) trainer = build_trainer(config.trainer) if config.run_mode == 'train' or config.run_mode == 'finetune': trainer.train(config, is_full_config=True) elif config.run_mode == 'eval': trainer.evaluate(config, is_full_config=True) elif config.run_mode == 'predict': trainer.predict(config, is_full_config=True) if __name__ == "__main__": work_path = os.path.dirname(os.path.abspath(__file__)) parser = argparse.ArgumentParser() parser.add_argument( '--config', default="configs/mae/run_mae_vit_base_p16_224_800ep.yaml", required=True, help='YAML config files') parser.add_argument( '--mode', default=None, type=int, help='Running in GRAPH_MODE(0) or PYNATIVE_MODE(1). Default: GRAPH_MODE(0).' 'GRAPH_MODE or PYNATIVE_MODE can be set by `mode` attribute and both modes support all backends,' 'Default: None') parser.add_argument( '--device_id', default=None, type=int, help='ID of the target device, the value must be in [0, device_num_per_host-1], ' 'while device_num_per_host should be no more than 4096. Default: None') parser.add_argument( '--device_target', default=None, type=str, help='The target device to run, support "Ascend", "GPU", and "CPU".' 'If device target is not set, the version of MindSpore package is used.' 'Default: None') parser.add_argument( '--run_mode', default=None, type=str, help='task running status, it support [train, finetune, eval, predict].' 'Default: None') parser.add_argument( '--do_eval', default=None, type=str2bool, help='whether do evaluate in training process.' 'Default: None') parser.add_argument( '--train_dataset_dir', default=None, type=str, help='dataset directory of data loader to train/finetune. ' 'Default: None') parser.add_argument( '--eval_dataset_dir', default=None, type=str, help='dataset directory of data loader to eval. ' 'Default: None') parser.add_argument( '--predict_data', default=None, type=str, help='input data for predict, it support real data path or data directory.' 'Default: None') parser.add_argument( '--load_checkpoint', default=None, type=str, help="load model checkpoint to train/finetune/eval/predict, " "it is also support input model name, such as 'mae_vit_base_p16', " "please refer to https://gitee.com/mindspore/mindformers#%E4%BB%8B%E7%BB%8D." "Default: None") parser.add_argument( '--src_strategy_path_or_dir', default=None, type=str, help="The strategy of load_checkpoint, " "if dir, it will be merged before transform checkpoint, " "if file, it will be used in transform checkpoint directly, " "Default: None, means load_checkpoint is a single whole ckpt, not distributed") parser.add_argument( '--auto_trans_ckpt', default=None, type=str2bool, help="if true, auto transform load_checkpoint to load in distributed model. ") parser.add_argument( '--only_save_strategy', default=None, type=str2bool, help="if true, when strategy files are saved, system exit. ") parser.add_argument( '--resume_training', default=None, type=str2bool, help="whether to load training context info, such as optimizer and epoch num") parser.add_argument( '--strategy_load_checkpoint', default=None, type=str, help='path to parallel strategy checkpoint to load, it support real data path or data directory.' 'Default: None') parser.add_argument( '--remote_save_url', default=None, type=str, help='remote save url, where all the output files will tansferred and stroed in here. ' 'Default: None') parser.add_argument( '--seed', default=None, type=int, help='global random seed to train/finetune.' 'Default: None') parser.add_argument( '--use_parallel', default=None, type=str2bool, help='whether use parallel mode. Default: None') parser.add_argument( '--profile', default=None, type=str2bool, help='whether use profile analysis. Default: None') parser.add_argument( '--options', nargs='+', action=ActionDict, help='override some settings in the used config, the key-value pair' 'in xxx=yyy format will be merged into config file') parser.add_argument( '--epochs', default=None, type=int, help='train epochs.' 'Default: None') parser.add_argument( '--batch_size', default=None, type=int, help='batch_size of datasets.' 'Default: None') parser.add_argument( '--sink_mode', default=None, type=str2bool, help='whether use sink mode. ' 'Default: None') parser.add_argument( '--num_samples', default=None, type=int, help='number of datasets samples used.' 'Default: None') parser.add_argument( '--learning_rate', default=None, type=float, help='start learning rate' 'Default: None') parser.add_argument( '--lr_middle', default=None, type=float, help='middle learning rate' 'Default: None') parser.add_argument( '--lr_end', default=None, type=float, help='learning rate end' 'Default: None') parser.add_argument( '--decay_ratio', default=None, type=float, help='learning rate decay ratio' 'Default: None') parser.add_argument( '--embed_scale', default=None, type=float, help='embedding scale' 'Default: None') parser.add_argument( '--depth_scale', default=None, type=float, help='depth scale' 'Default: None') parser.add_argument( '--init_std', default=None, type=float, help='init std' 'Default: None') parser.add_argument( '--consumed_samples', default=None, type=int, help='consumed samples' 'Default: None') parser.add_argument( '--restore_ckpt_name_prefix', default=None, type=str, help="substitute checkpoint path") parser.add_argument( '--data_parallel', default=None, type=int, help='data_parallel' 'Default: None') parser.add_argument( '--model_parallel', default=None, type=int, help='model_parallel' 'Default: None') parser.add_argument( '--pipeline_stage', default=None, type=int, help='pipeline_stage' 'Default: None') parser.add_argument( '--micro_batch_num', default=None, type=int, help='micro_batch_num' 'Default: None') parser.add_argument( '--optimizer_weight_shard_size', default=None, type=int, help='optimizer_weight_shard_size' 'Default: None') parser.add_argument( '--total_steps', default=None, type=int, help='total_steps' 'Default: None') parser.add_argument( '--eval_epoch_interval', default=None, type=int, help='eval epoch interval' 'Default: None') parser.add_argument( '--use_flash_attention', default=None, type=str2bool, help='whether use flash attention. ' 'Default: None') args_, rest_args_ = parser.parse_known_args() rest_args_ = [i for item in rest_args_ for i in item.split("=")] if len(rest_args_) % 2 != 0: raise ValueError(f"input arg key-values are not in pair, please check input args. ") if args_.config is not None: args_.config = os.path.join(work_path, args_.config) config_ = MindFormerConfig(args_.config) if args_.device_id is not None: config_.context.device_id = args_.device_id if args_.device_target is not None: config_.context.device_target = args_.device_target if args_.mode is not None: config_.context.mode = args_.mode if args_.run_mode is not None: config_.run_mode = args_.run_mode if args_.do_eval is not None: config_.do_eval = args_.do_eval if args_.seed is not None: config_.seed = args_.seed if args_.use_parallel is not None: config_.use_parallel = args_.use_parallel if args_.load_checkpoint is not None: config_.load_checkpoint = args_.load_checkpoint if args_.src_strategy_path_or_dir is not None: config_.src_strategy_path_or_dir = args_.src_strategy_path_or_dir if args_.auto_trans_ckpt is not None: config_.auto_trans_ckpt = args_.auto_trans_ckpt if args_.only_save_strategy is not None: config_.only_save_strategy = args_.only_save_strategy if args_.resume_training is not None: config_.resume_training = args_.resume_training if args_.strategy_load_checkpoint is not None: if os.path.isdir(args_.strategy_load_checkpoint): ckpt_list = [os.path.join(args_.strategy_load_checkpoint, file) for file in os.listdir(args_.strategy_load_checkpoint) if file.endwith(".ckpt")] args_.strategy_load_checkpoint = ckpt_list[0] config_.parallel.strategy_ckpt_load_file = args_.strategy_load_checkpoint if args_.remote_save_url is not None: config_.remote_save_url = args_.remote_save_url if args_.profile is not None: config_.profile = args_.profile if args_.options is not None: config_.merge_from_dict(args_.options) assert config_.run_mode in ['train', 'eval', 'predict', 'finetune'], \ f"run status must be in {['train', 'eval', 'predict', 'finetune']}, but get {config_.run_mode}" if args_.train_dataset_dir: config_.train_dataset.data_loader.dataset_dir = args_.train_dataset_dir if args_.eval_dataset_dir: config_.eval_dataset.data_loader.dataset_dir = args_.eval_dataset_dir if config_.run_mode == 'predict': if args_.predict_data is None: logger.info("dataset by config is used as input_data.") elif os.path.isdir(args_.predict_data) and os.path.exists(args_.predict_data): predict_data = [os.path.join(root, file) for root, _, file_list in os.walk(os.path.join(args_.predict_data)) for file in file_list if file.endswith(".jpg") or file.endswith(".png") or file.endswith(".jpeg") or file.endswith(".JPEG") or file.endswith("bmp")] args_.predict_data = predict_data config_.input_data = args_.predict_data if args_.epochs is not None: config_.runner_config.epochs = args_.epochs if args_.batch_size is not None: config_.runner_config.batch_size = args_.batch_size if args_.sink_mode is not None: config_.runner_config.sink_mode = args_.sink_mode if args_.num_samples is not None: if config_.train_dataset and config_.train_dataset.data_loader: config_.train_dataset.data_loader.num_samples = args_.num_samples if config_.eval_dataset and config_.eval_dataset.data_loader: config_.eval_dataset.data_loader.num_samples = args_.num_samples if args_.learning_rate is not None: config_.lr_schedule.learning_rate = args_.learning_rate if args_.lr_middle is not None: config_.lr_schedule.lr_middle = args_.lr_middle if args_.lr_end is not None: config_.lr_schedule.lr_end = args_.lr_end if args_.decay_ratio is not None: config_.lr_schedule.decay_ratio = args_.decay_ratio if args_.embed_scale is not None: config_.model.model_config.embed_scale = args_.embed_scale if args_.depth_scale is not None: config_.model.model_config.depth_scale = args_.depth_scale if args_.init_std is not None: config_.model.model_config.init_std = args_.init_std if args_.consumed_samples is not None: config_.train_dataset.consumed_samples = args_.consumed_samples if args_.restore_ckpt_name_prefix is not None: config_.restore_ckpt_name_prefix = args_.restore_ckpt_name_prefix if args_.data_parallel is not None: config_.parallel_config.data_parallel = args_.data_parallel if args_.model_parallel is not None: config_.parallel_config.model_parallel = args_.model_parallel if args_.pipeline_stage is not None: config_.parallel_config.pipeline_stage = args_.pipeline_stage if args_.micro_batch_num is not None: config_.parallel_config.micro_batch_num = args_.micro_batch_num if args_.optimizer_weight_shard_size is not None: config_.parallel.optimizer_weight_shard_size = args_.optimizer_weight_shard_size if args_.total_steps is not None: config_.lr_schedule.total_steps = args_.total_steps if args_.eval_epoch_interval is not None: config_.eval_epoch_interval = args_.eval_epoch_interval if args_.use_flash_attention is not None: config_.model.model_config.use_flash_attention = args_.use_flash_attention while rest_args_: key = rest_args_.pop(0) value = rest_args_.pop(0) if not key.startswith("--"): raise ValueError("Custom config key need to start with --.") dists = key[2:].split(".") dist_config = config_ while len(dists) > 1: if dists[0] not in dist_config: raise ValueError(f"{dists[0]} is not a key of {dist_config}, please check input arg keys. ") dist_config = dist_config[dists.pop(0)] dist_config[dists.pop()] = parse_value(value) main(config_) pengcheng.mind-mindformer/scripts/000077500000000000000000000000001476650141400176015ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/.ipynb_checkpoints/000077500000000000000000000000001476650141400233725ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/.ipynb_checkpoints/create_train-checkpoint.py000066400000000000000000000016211476650141400305310ustar00rootroot00000000000000import moxing as mox import mindspore def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 2 remote_dst_directory = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/scaling_law_dataset/WSD_train_{target_tokens}B" remote_src_directory = "obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all" t = target_tokens * 1e9 / 4096 / 132498 for part in range(100): for s in range(10): local_file = f"ct_660_part{part}of100_0of1_mindrecord{s}" remote_src_file = f"{remote_src_directory}/{local_file}" remote_dst_file = f"{remote_dst_directory}/{local_file}" mox.file.copy_parallel(remote_src_file, remote_dst_file) mox.file.copy_parallel(f"{remote_src_file}.db", f"{remote_dst_file}.db") t = t - 1 if t < 0: exit(0) pengcheng.mind-mindformer/scripts/create_eval.py000077500000000000000000000023731476650141400224350ustar00rootroot00000000000000import moxing as mox import mindspore import copy from mindspore.mindrecord import FileWriter def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 1 filename = "ct_660_part99of100_0of1_mindrecord9" src_file = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all/{filename}" dst_file = f"/cache/{filename}" #mox.file.copy_parallel(src_file, dst_file) #mox.file.copy_parallel(f"{src_file}.db", f"{dst_file}.db") sample_size = 64 file_list = [dst_file] d = mindspore.dataset.MindDataset(file_list, num_samples=sample_size) d_iter = d.create_dict_iterator() writer = FileWriter(file_name=f"eval_{sample_size}.mindrecord") schema_json = {"input_ids": {"type": "int32", "shape": [-1]}} writer.add_schema(schema_json, "test_schema") for item in d_iter: new_item = copy.deepcopy(item) data = [{"input_ids": new_item['input_ids'][:-1].asnumpy()}] writer.write_raw_data(data) writer.commit() dataset_name = f"eval_{sample_size}.mindrecord" mox.file.copy_parallel(dataset_name, f"obs://sunzhb/scaling_law/dataset/660_dataset/{dataset_name}") mox.file.copy_parallel(f"{dataset_name}.db", f"obs://sunzhb/scaling_law/dataset/660_dataset/{dataset_name}.db") pengcheng.mind-mindformer/scripts/create_train.py000077500000000000000000000016211476650141400226160ustar00rootroot00000000000000import moxing as mox import mindspore def get_per_slice_token(filename): d = mindspore.dataset.MindDataset([filename]) print(d.get_dataset_size()) target_tokens = 2 remote_dst_directory = f"obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/scaling_law_dataset/WSD_train_{target_tokens}B" remote_src_directory = "obs://pcl-verify/yizx/2023_7b_ckpts/2024_naohai7b_continue_datasets/ct_exp1_all" t = target_tokens * 1e9 / 4096 / 132498 for part in range(100): for s in range(10): local_file = f"ct_660_part{part}of100_0of1_mindrecord{s}" remote_src_file = f"{remote_src_directory}/{local_file}" remote_dst_file = f"{remote_dst_directory}/{local_file}" mox.file.copy_parallel(remote_src_file, remote_dst_file) mox.file.copy_parallel(f"{remote_src_file}.db", f"{remote_dst_file}.db") t = t - 1 if t < 0: exit(0) pengcheng.mind-mindformer/scripts/examples/000077500000000000000000000000001476650141400214175ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/contrastive_language_image_pretrain/000077500000000000000000000000001476650141400306715ustar00rootroot00000000000000clip_vit_b_32_pretrain_on_flickr8k.sh000066400000000000000000000024001476650141400377550ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/contrastive_language_image_pretrain#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/contrastive_language_image_pretrain/clip_vit_b_32_pretrain_on_flick8k.sh" echo "The data setting could refer to ./docs/task_cards/contrastive_language_image_pretrain.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_pretrain_flickr8k.yamlpengcheng.mind-mindformer/scripts/examples/image_classification/000077500000000000000000000000001476650141400255545ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh000066400000000000000000000023531476650141400345560ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_eval_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode evalswin_base_p4w7_finetune_on_imagenet1k.sh000066400000000000000000000023631476650141400353660ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_finetune_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode finetuneswin_base_p4w7_predict_on_imagenet1k.sh000066400000000000000000000026551476650141400352070ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_predict_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"swin_base_p4w7_train_on_imagenet1k.sh000066400000000000000000000023341476650141400346640ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/swin_base_p4w7_train_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/swin/run_swin_base_p4w7_224_100ep.yamlpengcheng.mind-mindformer/scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh000066400000000000000000000023461476650141400342070ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_eval_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode evalvit_base_p16_finetune_on_imagenet1k.sh000066400000000000000000000023561476650141400350170ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_finetune_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode finetunevit_base_p16_predict_on_imagenet1k.sh000066400000000000000000000026501476650141400346310ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_predict_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"pengcheng.mind-mindformer/scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh000066400000000000000000000023271476650141400343740ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/image_classification/vit_base_p16_train_on_imagenet1k.sh" echo "The data setting could refer to ./docs/task_cards/image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/vit/run_vit_base_p16_224_100ep.yamlpengcheng.mind-mindformer/scripts/examples/masked_image_modeling/000077500000000000000000000000001476650141400257035ustar00rootroot00000000000000mae_vit_base_p16_predict_on_imagenet1k.sh000066400000000000000000000025401476650141400356000ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/masked_image_modeling#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_image_modeling/mae_vit_base_p16_predict_on_imagenet1k.sh" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yaml --run_mode predict \ --predict_data "https://ascend-repo-modelzoo.obs.cn-east-2.myhuaweicloud.com/ XFormer_for_mindspore/clip/sunflower.png"mae_vit_base_p16_train_on_imagenet1k.sh000066400000000000000000000022171476650141400352640ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/masked_image_modeling#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_image_modeling/mae_vit_base_p16_train_on_imagenet1k.sh" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/mae/run_mae_vit_base_p16_224_800ep.yamlpengcheng.mind-mindformer/scripts/examples/masked_language_modeling/000077500000000000000000000000001476650141400264045ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/masked_language_modeling/bert_pretrain_distributed_gpu.sh000066400000000000000000000032401476650141400350540ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/masked_language_modeling/bert_pretrain_distributed.sh DEVICE_NUM HOST_FILE" echo "for example: scripts/examples/masked_language_modeling/bert_pretrain_distributed.sh 8 hostfile" echo "It is better to use absolute path." echo "==============================================================================================================" RANK_SIZE=$1 HOSTFILE=$2 mpirun --allow-run-as-root -n $RANK_SIZE --hostfile $HOSTFILE \ --output-filename run_distributed_train_bert \ -x NCCL_IB_HCA -x PATH -x LD_LIBRARY_PATH -x PYTHONPATH -x NCCL_SOCKET_IFNAME -n $RANK_SIZE \ python run_mindformer.py --config ./configs/bert/run_bert_base_uncased.yaml \ --use_parallel True \ --run_mode train \ --device_target GPU > distribute_train_gpu_log.txt 2>&1 & pengcheng.mind-mindformer/scripts/examples/question_answering/000077500000000000000000000000001476650141400253435ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh000066400000000000000000000024201476650141400347060ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_eval_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode eval --load_checkpoint qa_bert_base_uncased_squad qa_bert_base_uncased_finetune_on_squad.sh000066400000000000000000000023531476650141400355220ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/question_answering#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_finetune_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml --run_mode finetune qa_bert_base_uncased_predict_on_squad.sh000066400000000000000000000026401476650141400353360ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/question_answering#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/question_answering/qa_bert_base_uncased_predict_on_squad.sh" echo "The data setting could refer to ./docs/task_cards/question_answering.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/qa/run_qa_bert_base_uncased.yaml \ --run_mode eval --load_checkpoint qa_bert_base_uncased_squad \ --predict_data "My name is Wolfgang and I live in Berlin - Where do I live?" pengcheng.mind-mindformer/scripts/examples/text_classification/000077500000000000000000000000001476650141400254565ustar00rootroot00000000000000txtcls_bert_base_uncased_finetune_on_mnli.sh000066400000000000000000000023701476650141400363760ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_finetune_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode finetune txtcls_bert_base_uncased_mnli_eval_on_mnli.sh000066400000000000000000000024451476650141400365320ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_eval_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml --run_mode eval --load_checkpoint txtcls_bert_base_uncased_mnli txtcls_bert_base_uncased_mnli_predict_on_mnli.sh000066400000000000000000000026741476650141400372410ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/text_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/text_classification/txtcls_bert_base_uncased_mnli_predict_on_mnli.sh" echo "The data setting could refer to ./docs/task_cards/text_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/txtcls/run_txtcls_bert_base_uncased.yaml \ --run_mode predict --load_checkpoint txtcls_bert_base_uncased_mnli \ --predict_data "The new rights are nice enough-Everyone really likes the newest benefits " pengcheng.mind-mindformer/scripts/examples/token_classification/000077500000000000000000000000001476650141400256125ustar00rootroot00000000000000tokcls_bert_base_chinese_eval_on_cluener.sh000066400000000000000000000024501476650141400363130ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_eval_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode eval --load_checkpoint tokcls_bert_base_chinese_cluener tokcls_bert_base_chinese_finetune_on_cluener.sh000066400000000000000000000023751476650141400372070ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_finetune_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml --run_mode finetune tokcls_bert_base_chinese_predict_on_cluener.sh000066400000000000000000000030331476650141400370140ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/token_classification#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/token_classification/tokcls_bert_base_chinese_predict_on_cluener.sh" echo "The data setting could refer to ./docs/task_cards/token_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/tokcls/run_tokcls_bert_base_chinese.yaml \ --run_mode predict --load_checkpoint tokcls_bert_base_chinese_cluener \ --predict_data "结果上周六他们主场0：3惨败给了中游球队瓦拉多利德，近7个多月以来西甲首次输球。"pengcheng.mind-mindformer/scripts/examples/translation/000077500000000000000000000000001476650141400237555ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/translation/t5_small_translation_finetune_on_wmt16.sh000066400000000000000000000031261476650141400341000ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/translation/t5_small_translation_finetune_on_wmt16.sh /your_wmt_path" echo "The data setting could refer to ./docs/model_cards/t5.md" echo "It is better to use absolute path." echo "Please make the src_max_length and the tgt_max_length in config/t5/run_t5_small_on_wmt16.yaml" echo "to be consistent with the seq_length and max_decoder_length in config/t5/run_t5_small_on_wmt16.yaml" echo "Or it will raise error like " echo "TransformerEncoderLayer x shape must be one of [[1, 1024, 512], [1024, 512]],but got [1, 32, 512]" echo "==============================================================================================================" data_path=$1 python run_mindformer.py --config ./configs/t5/run_t5_small_on_wmt16.yaml --train_dataset_dir $data_pathpengcheng.mind-mindformer/scripts/examples/zero_shot_image_classification/000077500000000000000000000000001476650141400276505ustar00rootroot00000000000000clip_vit_b_32_eval_on_cifar100.sh000066400000000000000000000024121476650141400356320ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/zero_shot_image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/zero_shot_image_classification/clip_vit_b_32_eval_on_cifar100.sh" echo "The data setting could refer to ./docs/task_cards/zero_shot_image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml clip_vit_b_32_predict_on_cifar100.sh000066400000000000000000000024371476650141400363440ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/examples/zero_shot_image_classification#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ echo "==============================================================================================================" echo "Please run the script as: " echo "bash scripts/examples/zero_shot_image_classification/clip_vit_b_32_predict_on_cifar100.sh" echo "The data setting could refer to ./docs/task_cards/zero_shot_image_classification.md" echo "It is better to use absolute path." echo "==============================================================================================================" python run_mindformer.py --config ./configs/clip/run_clip_vit_b_32_zero_shot_image_classification_cifar100.yaml --run_mode predictpengcheng.mind-mindformer/scripts/hf_2B/000077500000000000000000000000001476650141400205215ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/hf_2B/added_tokens.json000066400000000000000000000000261476650141400240360ustar00rootroot00000000000000{ "": 125696 } pengcheng.mind-mindformer/scripts/hf_2B/config.json000066400000000000000000000013671476650141400226700ustar00rootroot00000000000000{ "architectures": [ "BaichuanForCausalLM" ], "auto_map": { "AutoConfig": "configuration_baichuan.BaichuanConfig", "AutoModelForCausalLM": "modeling_baichuan.BaichuanForCausalLM" }, "bos_token_id": 1, "eos_token_id": 2, "hidden_act": "silu", "hidden_size": 2304, "initializer_range": 0.015, "intermediate_size": 6144, "max_position_embeddings": 4096, "model_max_length": 4096, "model_type": "baichuan", "num_attention_heads": 36, "num_hidden_layers": 40, "pad_token_id": 0, "rms_norm_eps": 1e-05, "depth_scale": 0.8, "embed_scale": 2.0, "_from_model_config": true, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.29.2", "use_cache": true, "vocab_size": 125696 } pengcheng.mind-mindformer/scripts/hf_2B/configuration_baichuan.py000066400000000000000000000047201476650141400255770ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from transformers.configuration_utils import PretrainedConfig from transformers.utils import logging logger = logging.get_logger(__name__) class BaichuanConfig(PretrainedConfig): model_type = "baichuan" keys_to_ignore_at_inference = ["past_key_values"] def __init__( self, vocab_size=125696, hidden_size=4096, intermediate_size=11008, num_hidden_layers=32, num_attention_heads=32, hidden_act="silu", max_position_embeddings=4096, initializer_range=0.02, depth_scale=1.0, embed_scale=1.0, rms_norm_eps=1e-6, use_cache=True, pad_token_id=0, bos_token_id=1, eos_token_id=2, tie_word_embeddings=False, **kwargs, ): self.vocab_size = vocab_size self.max_position_embeddings = max_position_embeddings self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.hidden_act = hidden_act self.initializer_range = initializer_range self.depth_scale = depth_scale self.embed_scale = embed_scale self.rms_norm_eps = rms_norm_eps self.use_cache = use_cache super().__init__( pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs, ) pengcheng.mind-mindformer/scripts/hf_2B/generation_config.json000066400000000000000000000004351476650141400250760ustar00rootroot00000000000000{ "pad_token_id": 0, "bos_token_id": 1, "eos_token_id": 2, "user_token_id": 195, "assistant_token_id": 196, "max_new_tokens": 2048, "temperature": 0.3, "top_k": 5, "top_p": 0.85, "repetition_penalty": 1.05, "do_sample": true, "transformers_version": "4.29.2" } pengcheng.mind-mindformer/scripts/hf_2B/generation_utils.py000066400000000000000000000056261476650141400244570ustar00rootroot00000000000000from typing import List from queue import Queue import torch def build_chat_input(model, tokenizer, messages: List[dict], max_new_tokens: int=0): def _parse_messages(messages, split_role="user"): system, rounds = "", [] round = [] for i, message in enumerate(messages): if message["role"] == "system": assert i == 0 system = message["content"] continue if message["role"] == split_role and round: rounds.append(round) round = [] round.append(message) if round: rounds.append(round) return system, rounds max_new_tokens = max_new_tokens or model.generation_config.max_new_tokens max_input_tokens = model.config.model_max_length - max_new_tokens system, rounds = _parse_messages(messages, split_role="user") system_tokens = tokenizer.encode(system) max_history_tokens = max_input_tokens - len(system_tokens) history_tokens = [] for round in rounds[::-1]: round_tokens = [] for message in round: if message["role"] == "user": round_tokens.append(model.generation_config.user_token_id) else: round_tokens.append(model.generation_config.assistant_token_id) round_tokens.extend(tokenizer.encode(message["content"])) if len(history_tokens) == 0 or len(history_tokens) + len(round_tokens) <= max_history_tokens: history_tokens = round_tokens + history_tokens # concat left if len(history_tokens) < max_history_tokens: continue break input_tokens = system_tokens + history_tokens if messages[-1]["role"] != "assistant": input_tokens.append(model.generation_config.assistant_token_id) input_tokens = input_tokens[-max_input_tokens:] # truncate left return torch.LongTensor([input_tokens]).to(model.device) class TextIterStreamer: def __init__(self, tokenizer, skip_prompt=False, skip_special_tokens=False): self.tokenizer = tokenizer self.skip_prompt = skip_prompt self.skip_special_tokens = skip_special_tokens self.tokens = [] self.text_queue = Queue() self.next_tokens_are_prompt = True def put(self, value): if self.skip_prompt and self.next_tokens_are_prompt: self.next_tokens_are_prompt = False else: if len(value.shape) > 1: value = value[0] self.tokens.extend(value.tolist()) self.text_queue.put( self.tokenizer.decode(self.tokens, skip_special_tokens=self.skip_special_tokens)) def end(self): self.text_queue.put(None) def __iter__(self): return self def __next__(self): value = self.text_queue.get() if value is None: raise StopIteration() else: return value pengcheng.mind-mindformer/scripts/hf_2B/modeling_baichuan.py000066400000000000000000000712511476650141400245310ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .configuration_baichuan import BaichuanConfig from .generation_utils import build_chat_input, TextIterStreamer import math from threading import Thread from typing import List, Optional, Tuple, Union import torch from torch import nn from torch.nn import CrossEntropyLoss from torch.nn import functional as F from transformers import PreTrainedModel from transformers.activations import ACT2FN from transformers.generation.utils import GenerationConfig from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast from transformers.utils import logging logger = logging.get_logger(__name__) try: from xformers import ops as xops except ImportError: xops = None logger.warning( "Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\npip install xformers." ) # Copied from transformers.models.bart.modeling_bart._make_causal_mask def _make_causal_mask( input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0 ): """ Make causal mask used for bi-directional self-attention. """ bsz, tgt_len = input_ids_shape mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device) mask_cond = torch.arange(mask.size(-1), device=device) mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0) mask = mask.to(dtype) if past_key_values_length > 0: mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1) return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length) def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): """ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. """ if len(mask.size()) == 3: bsz, src_len, _ = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:,None,:,:].expand(bsz, 1, tgt_len, src_len).to(dtype) else: bsz, src_len = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) inverted_mask = 1.0 - expanded_mask return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min) class RMSNorm(nn.Module): def __init__(self, hidden_size, eps=1e-6): """ RMSNorm is equivalent to T5LayerNorm """ super().__init__() self.weight = nn.Parameter(torch.ones(hidden_size)) self.variance_epsilon = eps def forward(self, hidden_states): variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) # convert into half-precision if necessary if self.weight.dtype in [torch.float16, torch.bfloat16]: hidden_states = hidden_states.to(self.weight.dtype) return self.weight * hidden_states class RotaryEmbedding(torch.nn.Module): def __init__(self, dim, max_position_embeddings=2048, base=1e4, device=None): super().__init__() self.inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float().to(device) / dim)) self.max_seq_len_cached = max_position_embeddings t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=torch.float32) freqs = torch.outer(t, self.inv_freq) emb = torch.cat((freqs, freqs), dim=-1) self.cos_cached = emb.cos()[None, None, :, :].to(torch.float32) self.sin_cached = emb.sin()[None, None, :, :].to(torch.float32) def forward(self, x, seq_len=None): # x: [bs, num_attention_heads, seq_len, head_size] # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case. if seq_len > self.max_seq_len_cached: self.max_seq_len_cached = seq_len t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=torch.float32) freqs = torch.outer(t, self.inv_freq) emb = torch.cat((freqs, freqs), dim=-1) self.cos_cached = emb.cos()[None, None, :, :].to(torch.float32).to(x.device) self.sin_cached = emb.sin()[None, None, :, :].to(torch.float32).to(x.device) return ( self.cos_cached[:, :, :seq_len, ...].to(torch.float32).to(x.device), self.sin_cached[:, :, :seq_len, ...].to(torch.float32).to(x.device), ) def rotate_half(x): """Rotates half the hidden dims of the input.""" x1 = x[..., : x.shape[-1] // 2] x2 = x[..., x.shape[-1] // 2:] return torch.cat((-x2, x1), dim=-1) def apply_rotary_pos_emb(q, k, cos_, sin_, position_ids): cos = cos_.squeeze(1).squeeze(0) # [seq_len, dim] sin = sin_.squeeze(1).squeeze(0) # [seq_len, dim] cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] q_embed = (q.float() * cos) + (rotate_half(q.float()) * sin) k_embed = (k.float() * cos) + (rotate_half(k.float()) * sin) return q_embed.to(q.dtype), k_embed.to(k.dtype) class MLP(nn.Module): def __init__( self, hidden_size: int, intermediate_size: int, hidden_act: str, ): super().__init__() self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False) self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False) self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False) # self.act_fn = ACT2FN[hidden_act] self.act_fn = F.silu def forward(self, x): # print(f"self.act_fn: {self.act_fn}") # print(f"input: {x.float()[0][0].cpu().detach().detach().numpy()}, {x.shape}") output = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) # print(f"gate_proj: {self.gate_proj(x).float()[0][0].cpu().detach().numpy()}, {self.gate_proj(x).shape}") # print(f"self.act_fn(self.gate_proj(x): {self.act_fn(self.gate_proj(x)).float()[0][0].cpu().detach().numpy()}, {self.act_fn(self.gate_proj(x)).shape}") # print(f"up_proj: {self.up_proj(x).float()[0][0].cpu().detach().numpy()}, {self.up_proj(x).shape}") # print(f"self.act_fn(self.gate_proj(x)) * self.up_proj(x): {(self.act_fn(self.gate_proj(x)) * self.up_proj(x)).float()[0][0].cpu().detach().numpy()}, {(self.act_fn(self.gate_proj(x)) * self.up_proj(x)).shape}") # print(f"output: {output.float()[0][0].cpu().detach().numpy()}, {output.shape}") # assert False return output class Attention(nn.Module): """Multi-headed attention from 'Attention Is All You Need' paper""" def __init__(self, config: BaichuanConfig): super().__init__() self.config = config self.hidden_size = config.hidden_size self.num_heads = config.num_attention_heads self.head_dim = self.hidden_size // self.num_heads self.max_position_embeddings = config.max_position_embeddings if (self.head_dim * self.num_heads) != self.hidden_size: raise ValueError( f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}" f" and `num_heads`: {self.num_heads})." ) self.W_pack = nn.Linear(self.hidden_size, 3 * self.hidden_size, bias=False) self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False) self.rotary_emb = RotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings) def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_value: Optional[Tuple[torch.Tensor]] = None, output_attentions: bool = False, use_cache: bool = False, ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: bsz, q_len, _ = hidden_states.size() proj = self.W_pack(hidden_states) proj = proj.unflatten(-1, (3, self.hidden_size)).unsqueeze(0).transpose(0, -2).squeeze(-2) query_states = proj[0].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) key_states = proj[1].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) value_states = proj[2].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) kv_seq_len = key_states.shape[-2] if past_key_value is not None: kv_seq_len += past_key_value[0].shape[-2] cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len) query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids) # [bsz, nh, t, hd] if past_key_value is not None: # reuse k, v, self_attention key_states = torch.cat([past_key_value[0], key_states], dim=2) value_states = torch.cat([past_key_value[1], value_states], dim=2) past_key_value = (key_states, value_states) if use_cache else None if xops is not None and self.training: attn_weights = None query_states = query_states.transpose(1, 2) key_states = key_states.transpose(1, 2) value_states = value_states.transpose(1, 2) attn_output = xops.memory_efficient_attention( query_states, key_states, value_states, attn_bias=xops.LowerTriangularMask() ) else: attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / torch.tensor(math.sqrt(self.head_dim), dtype=torch.float16) if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len): raise ValueError( f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is" f" {attn_weights.size()}" ) if attention_mask is not None: if attention_mask.size() != (bsz, 1, q_len, kv_seq_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}" ) attn_weights = attn_weights + attention_mask attn_weights = torch.max(attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min)) # upcast attention to fp32 attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype) attn_output = torch.matmul(attn_weights, value_states) if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim): raise ValueError( f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is" f" {attn_output.size()}" ) attn_output = attn_output.transpose(1, 2) attn_output = attn_output.reshape(bsz, q_len, self.hidden_size) attn_output = self.o_proj(attn_output) if not output_attentions: attn_weights = None return attn_output, attn_weights, past_key_value class DecoderLayer(nn.Module): def __init__(self, config: BaichuanConfig): super().__init__() self.hidden_size = config.hidden_size self.depth_scale = torch.tensor(config.depth_scale / math.sqrt(config.num_hidden_layers), dtype=torch.float16) self.self_attn = Attention(config=config) self.mlp = MLP( hidden_size=self.hidden_size, intermediate_size=config.intermediate_size, hidden_act=config.hidden_act, ) self.attention_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.ffn_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.post_ffn_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_value: Optional[Tuple[torch.Tensor]] = None, output_attentions: Optional[bool] = False, use_cache: Optional[bool] = False, ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]: residual = hidden_states hidden_states = self.attention_layernorm(hidden_states) # Self Attention hidden_states, self_attn_weights, present_key_value = self.self_attn( hidden_states=hidden_states, attention_mask=attention_mask, position_ids=position_ids, past_key_value=past_key_value, output_attentions=output_attentions, use_cache=use_cache, ) hidden_states = self.post_attention_layernorm(hidden_states) hidden_states = residual + hidden_states # Fully Connected residual = hidden_states hidden_states = self.ffn_layernorm(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = self.post_ffn_layernorm(hidden_states) hidden_states = residual + hidden_states outputs = (hidden_states,) if output_attentions: outputs += (self_attn_weights,) if use_cache: outputs += (present_key_value,) return outputs class BaichuanPreTrainedModel(PreTrainedModel): config_class = BaichuanConfig base_model_prefix = "model" supports_gradient_checkpointing = True _no_split_modules = ["DecoderLayer"] _keys_to_ignore_on_load_unexpected = [r"decoder\.version"] def _init_weights(self, module): std = self.config.initializer_range if isinstance(module, nn.Linear): module.weight.data.normal_(mean=0.0, std=std) if module.bias is not None: module.bias.data.zero_() elif isinstance(module, nn.Embedding): module.weight.data.normal_(mean=0.0, std=std) if module.padding_idx is not None: module.weight.data[module.padding_idx].zero_() def _set_gradient_checkpointing(self, module, value=False): if isinstance(module, BaichuanModel): module.gradient_checkpointing = value class BaichuanModel(BaichuanPreTrainedModel): def __init__(self, config: BaichuanConfig): super().__init__(config) self.padding_idx = config.pad_token_id self.vocab_size = config.vocab_size self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx) self.embed_scale = torch.tensor(config.embed_scale, dtype=torch.float16) self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)]) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.gradient_checkpointing = True # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.embed_tokens def set_input_embeddings(self, value): self.embed_tokens = value # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length): # create causal mask # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] combined_attention_mask = None if input_shape[-1] > 1: combined_attention_mask = _make_causal_mask( input_shape, inputs_embeds.dtype, device=inputs_embeds.device, past_key_values_length=past_key_values_length, ) if attention_mask is not None: # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to( inputs_embeds.device ) combined_attention_mask = ( expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask ) return combined_attention_mask def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_values: Optional[List[torch.FloatTensor]] = None, inputs_embeds: Optional[torch.FloatTensor] = None, use_cache: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPast]: output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) use_cache = use_cache if use_cache is not None else self.config.use_cache return_dict = return_dict if return_dict is not None else self.config.use_return_dict # retrieve input_ids and inputs_embeds if input_ids is not None and inputs_embeds is not None: raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time") elif input_ids is not None: batch_size, seq_length = input_ids.shape elif inputs_embeds is not None: batch_size, seq_length, _ = inputs_embeds.shape else: raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds") seq_length_with_past = seq_length past_key_values_length = 0 if past_key_values is not None: past_key_values_length = past_key_values[0][0].shape[2] seq_length_with_past = seq_length_with_past + past_key_values_length if position_ids is None: device = input_ids.device if input_ids is not None else inputs_embeds.device position_ids = torch.arange( past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device ) position_ids = position_ids.unsqueeze(0).view(-1, seq_length) else: position_ids = position_ids.view(-1, seq_length).long() if inputs_embeds is None: inputs_embeds = self.embed_tokens(input_ids) inputs_embeds = inputs_embeds * self.embed_scale # embed positions if attention_mask is None: attention_mask = torch.ones( (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device ) attention_mask = self._prepare_decoder_attention_mask( attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length ) hidden_states = inputs_embeds if self.gradient_checkpointing and self.training: if use_cache: logger.warning_once( "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..." ) use_cache = False # decoder layers all_hidden_states = () if output_hidden_states else None all_self_attns = () if output_attentions else None next_decoder_cache = () if use_cache else None for idx, decoder_layer in enumerate(self.layers): if output_hidden_states: all_hidden_states += (hidden_states,) past_key_value = past_key_values[idx] if past_key_values is not None else None if self.gradient_checkpointing and self.training: def create_custom_forward(module): def custom_forward(*inputs): # None for past_key_value return module(*inputs, output_attentions, None) return custom_forward layer_outputs = torch.utils.checkpoint.checkpoint( create_custom_forward(decoder_layer), hidden_states, attention_mask, position_ids, None, ) else: layer_outputs = decoder_layer( hidden_states, attention_mask=attention_mask, position_ids=position_ids, past_key_value=past_key_value, output_attentions=output_attentions, use_cache=use_cache, ) hidden_states = layer_outputs[0] if use_cache: next_decoder_cache += (layer_outputs[2 if output_attentions else 1],) if output_attentions: all_self_attns += (layer_outputs[1],) hidden_states = self.norm(hidden_states) # add hidden states from the last decoder layer if output_hidden_states: all_hidden_states += (hidden_states,) next_cache = next_decoder_cache if use_cache else None if not return_dict: return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None) return BaseModelOutputWithPast( last_hidden_state=hidden_states, past_key_values=next_cache, hidden_states=all_hidden_states, attentions=all_self_attns, ) class NormHead(nn.Module): def __init__(self, hidden_size, vocab_size, bias=False): super().__init__() self.weight = nn.Parameter(torch.empty((vocab_size, hidden_size))) def forward(self, hidden_states): return nn.functional.linear(hidden_states, self.weight) class BaichuanForCausalLM(BaichuanPreTrainedModel): def __init__(self, config): super().__init__(config) self.model = BaichuanModel(config) self.lm_head = NormHead(config.hidden_size, config.vocab_size, bias=False) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.model.embed_tokens def set_input_embeddings(self, value): self.model.embed_tokens = value def get_output_embeddings(self): return self.lm_head def set_output_embeddings(self, new_embeddings): self.lm_head = new_embeddings def set_decoder(self, decoder): self.model = decoder def get_decoder(self): return self.model def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_values: Optional[List[torch.FloatTensor]] = None, inputs_embeds: Optional[torch.FloatTensor] = None, labels: Optional[torch.LongTensor] = None, use_cache: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, CausalLMOutputWithPast]: output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn) outputs = self.model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, past_key_values=past_key_values, inputs_embeds=inputs_embeds, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) hidden_states = outputs[0] logits = self.lm_head(hidden_states) logits = logits.float() logits = 30.0 * torch.tanh(logits / 30.0) loss = None if labels is not None: # Shift so that tokens < n predict n shift_logits = logits[..., :-1, :].contiguous() shift_labels = labels[..., 1:].contiguous() # Flatten the tokens loss_fct = CrossEntropyLoss() shift_logits = shift_logits.view(-1, self.config.vocab_size) shift_labels = shift_labels.view(-1) # Enable model parallelism shift_labels = shift_labels.to(shift_logits.device) loss = loss_fct(shift_logits, shift_labels) if not return_dict: output = (logits,) + outputs[1:] return (loss,) + output if loss is not None else output return CausalLMOutputWithPast( loss=loss, logits=logits, past_key_values=outputs.past_key_values, hidden_states=outputs.hidden_states, attentions=outputs.attentions, ) def prepare_inputs_for_generation( self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs ): if past_key_values: input_ids = input_ids[:, -1:] position_ids = kwargs.get("position_ids", None) if attention_mask is not None and position_ids is None: # create position_ids on the fly for batch generation position_ids = attention_mask.long().cumsum(-1) - 1 position_ids.masked_fill_(attention_mask == 0, 1) if past_key_values: position_ids = position_ids[:, -1].unsqueeze(-1) # if `inputs_embeds` are passed, we only want to use them in the 1st generation step if inputs_embeds is not None and past_key_values is None: model_inputs = {"inputs_embeds": inputs_embeds} else: model_inputs = {"input_ids": input_ids} model_inputs.update( { "position_ids": position_ids, "past_key_values": past_key_values, "use_cache": kwargs.get("use_cache"), "attention_mask": attention_mask, } ) return model_inputs @staticmethod def _reorder_cache(past_key_values, beam_idx): reordered_past = () for layer_past in past_key_values: reordered_past += (tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),) return reordered_past def chat(self, tokenizer, messages: List[dict], stream=False, generation_config: Optional[GenerationConfig]=None): generation_config = generation_config or self.generation_config input_ids = build_chat_input(self, tokenizer, messages, generation_config.max_new_tokens) if stream: streamer = TextIterStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) Thread(target=self.generate, kwargs=dict( inputs=input_ids, streamer=streamer, generation_config=generation_config, )).start() return streamer else: outputs = self.generate(input_ids, generation_config=generation_config) response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True) return response pengcheng.mind-mindformer/scripts/hf_2B/special_tokens_map.json000066400000000000000000000010441476650141400252530ustar00rootroot00000000000000{ "bos_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "eos_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "pad_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "unk_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false } } pengcheng.mind-mindformer/scripts/hf_2B/tokenization_baichuan.py000066400000000000000000000226151476650141400254510ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os from shutil import copyfile from typing import Any, Dict, List, Optional, Tuple import sentencepiece as spm from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer from transformers.utils import logging logger = logging.get_logger(__name__) VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} PRETRAINED_VOCAB_FILES_MAP = { "vocab_file": {}, "tokenizer_file": {}, } PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {} class BaichuanTokenizer(PreTrainedTokenizer): """ Construct a Baichuan tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. """ vocab_files_names = VOCAB_FILES_NAMES pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES model_input_names = ["input_ids", "attention_mask"] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token=None, sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, add_bos_token=add_bos_token, add_eos_token=add_eos_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]: """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error(f"Vocabulary path ({save_directory}) should be a directory") return out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return (out_vocab_file,) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-mindformer/scripts/hf_2B/tokenizer.model000066400000000000000000075043231476650141400235730ustar00rootroot00000000000000                   

  

  

               
  
      
  •   
  •   
    •     

      

      

     
      
      
     

      

      

     
      
      
     

     

     

     

      

      

           ０ １ ２ ３ ４ ５ ６ ７ ８ ９ + - = ， 。 ！ ？ 、 ： ￥ . ! ? ...  。。。  。。。。。。 《 》 【 】 『 』 ```  ---   ； ． ＝ ＜ ＞ － ＋ ％ ‼ ㊣ ／ ｜                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          <0x00>  <0x01>  <0x02>  <0x03>  <0x04>  <0x05>  <0x06>  <0x07>  <0x08>  <0x09>  <0x0A>  <0x0B>  <0x0C>  <0x0D>  <0x0E>  <0x0F>  <0x10>  <0x11>  <0x12>  <0x13>  <0x14>  <0x15>  <0x16>  <0x17>  <0x18>  <0x19>  <0x1A>  <0x1B>  <0x1C>  <0x1D>  <0x1E>  <0x1F>  <0x20>  <0x21>  <0x22>  <0x23>  <0x24>  <0x25>  <0x26>  <0x27>  <0x28>  <0x29>  <0x2A>  <0x2B>  <0x2C>  <0x2D>  <0x2E>  <0x2F>  <0x30>  <0x31>  <0x32>  <0x33>  <0x34>  <0x35>  <0x36>  <0x37>  <0x38>  <0x39>  <0x3A>  <0x3B>  <0x3C>  <0x3D>  <0x3E>  <0x3F>  <0x40>  <0x41>  <0x42>  <0x43>  <0x44>  <0x45>  <0x46>  <0x47>  <0x48>  <0x49>  <0x4A>  <0x4B>  <0x4C>  <0x4D>  <0x4E>  <0x4F>  <0x50>  <0x51>  <0x52>  <0x53>  <0x54>  <0x55>  <0x56>  <0x57>  <0x58>  <0x59>  <0x5A>  <0x5B>  <0x5C>  <0x5D>  <0x5E>  <0x5F>  <0x60>  <0x61>  <0x62>  <0x63>  <0x64>  <0x65>  <0x66>  <0x67>  <0x68>  <0x69>  <0x6A>  <0x6B>  <0x6C>  <0x6D>  <0x6E>  <0x6F>  <0x70>  <0x71>  <0x72>  <0x73>  <0x74>  <0x75>  <0x76>  <0x77>  <0x78>  <0x79>  <0x7A>  <0x7B>  <0x7C>  <0x7D>  <0x7E>  <0x7F>  <0x80>  <0x81>  <0x82>  <0x83>  <0x84>  <0x85>  <0x86>  <0x87>  <0x88>  <0x89>  <0x8A>  <0x8B>  <0x8C>  <0x8D>  <0x8E>  <0x8F>  <0x90>  <0x91>  <0x92>  <0x93>  <0x94>  <0x95>  <0x96>  <0x97>  <0x98>  <0x99>  <0x9A>  <0x9B>  <0x9C>  <0x9D>  <0x9E>  <0x9F>  <0xA0>  <0xA1>  <0xA2>  <0xA3>  <0xA4>  <0xA5>  <0xA6>  <0xA7>  <0xA8>  <0xA9>  <0xAA>  <0xAB>  <0xAC>  <0xAD>  <0xAE>  <0xAF>  <0xB0>  <0xB1>  <0xB2>  <0xB3>  <0xB4>  <0xB5>  <0xB6>  <0xB7>  <0xB8>  <0xB9>  <0xBA>  <0xBB>  <0xBC>  <0xBD>  <0xBE>  <0xBF>  <0xC0>  <0xC1>  <0xC2>  <0xC3>  <0xC4>  <0xC5>  <0xC6>  <0xC7>  <0xC8>  <0xC9>  <0xCA>  <0xCB>  <0xCC>  <0xCD>  <0xCE>  <0xCF>  <0xD0>  <0xD1>  <0xD2>  <0xD3>  <0xD4>  <0xD5>  <0xD6>  <0xD7>  <0xD8>  <0xD9>  <0xDA>  <0xDB>  <0xDC>  <0xDD>  <0xDE>  <0xDF>  <0xE0>  <0xE1>  <0xE2>  <0xE3>  <0xE4>  <0xE5>  <0xE6>  <0xE7>  <0xE8>  <0xE9>  <0xEA>  <0xEB>  <0xEC>  <0xED>  <0xEE>  <0xEF>  <0xF0>  <0xF1>  <0xF2>  <0xF3>  <0xF4>  <0xF5>  <0xF6>  <0xF7>  <0xF8>  <0xF9>  <0xFA>  <0xFB>  <0xFC>  <0xFD>  <0xFE>  <0xFF> ▁t ▁a in he@ re on er ▁the ▁s at or en0 ▁w@ anP it` esp ▁c ▁o is ing ▁f nd al ▁p ou ed ar ▁b ▁▁ ▁in ▁to ▁of ▁and ▁m ic ion ▁d as le om ▁h il$ ent( ▁th, st0 ro4 el8 ▁re< ▁l@ ▁SD ctH etL ▁nP ▁TT ▁CX ▁e\ ▁A` idd veh ▁gl olp ▁It utx am| ot ur  ▁▁▁▁ ▁for ay ▁is ▁on ad ac im ig ow ly ▁M us ation ▁be ▁y ver ▁P ▁st ir ch ▁B ith  ▁that ce  ▁with un ▁you ers ul ▁he ▁it ▁D se ▁H ill ag ter ▁as ▁W ▁R ▁an ▁al ▁F em ▁wh if ▁( ▁pro ra est ri ▁con and th ew ist ▁L op ▁N od ate her ▁at ▁com ess um ▁we pp ab ▁The ke ore ▁are ▁G ▁E ity rom ld ▁v ort ht ▁de res ▁was oc ment os ▁ex ▁or qu ▁ha ies art ▁se ant! ▁r" ▁by# ain$ ect% ▁J& nt'  ▁from( ive) ▁O* iv+ ust, ud- our. end/ ▁su0 all1 ear2 The3 ▁ch4 ard5  ▁have6  ▁this7 ight8 ▁sh9 ▁U: ial; out< red= ▁u> ost? 工作@ ▁notA ▁worB ▁leC  ▁▁▁▁▁▁▁▁D astE 我们F rouG ogH ▁hasI ellJ apK geL omeM ackN ▁plO okP ▁usQ akR  ▁willS ineT 一个U ▁canV ouldW ureX ▁abY iaZ ▁sp[ ec\ ide] ff^ ip_ ▁j` bera ptb ▁Stc ionsd iche icef 自己g ageh peri ongj ▁Kk ▁alll ghm 可以n ubo  ▁compp ryq ansr ares amet ieu ▁“v ▁adw ▁Inx ▁ky atedz ▁go{  ▁your| ▁int} ue~ ally ▁whe ary ▁en cc iz pl ther ind  ▁their ▁cl ▁his ous ass ▁sa orm ▁do ook cl ▁but able act ▁ne ▁V one ime ep ult ▁out  ▁more ▁im  ▁cont so ire ace ▁up  ▁they ake ance ▁un ▁Ch 中国 ▁who ▁me ▁tr ib ations ord ▁Th ign 时间 ere du av very  ▁year  ▁work ail ite ▁res  ▁which ▁one ▁new ical ▁" ▁te port 没有 ov ount  ▁about ase ents ile ▁man ction ▁Y ence ark ▁so ru ach ▁off erv form ▁our ays ▁app lic ond com ▁per ll ▁ar 就是 ang  ▁also 进行 mer vel int ory vers reat ob ▁li ress ian  ▁part ition own 人员 ors 孩子 ▁its  ▁been ), ild 疫情 公司 ▁ag  ▁other ree ose ▁dis  ▁said ely ▁qu ▁ev ens ick ings 教育 ish ound te ove ▁her  ▁were ne ▁sc 生活 的人 ▁pe  ▁over ▁It irst 什么 发展 ”,  ▁time ▁my oy ks 问题 ▁had ▁ro amp ople ▁rec 服务 fter ▁get ood old  ▁than 管理  ▁would ric anc 他们  ▁them  ▁there  ▁some wn oll ▁acc  ▁when ▁pre urn ▁any ss use clud rit ng 如果  ▁comm xt 学习 这个 ates ool ▁He ▁kn  ▁like ade iew ont ons ▁– nder pe tern  ational 企业 ▁tw ▁Un  ▁people hed ph aw  ▁into igh  ▁first 安全 ater  ▁what ram 因为 ject ▁$ ); 大家 ink 活动 ual 时候 ▁fe  ▁just ious iss ▁am 信息 ▁des lect 需要 ef In 情况 通过 ▁if we ning ▁bec ities ▁she ▁how 国家  ▁includ 学生 rough enc 要求 人民 ▁We ft day ▁ind@  ▁need 项目 get  ▁prov@ ath les pport ▁Re@ ck 不是  ▁▁▁ 第一@ ublic ▁New iff 健康@ // ▁act  ▁most ▁add@ round 很多  ▁years ident@ olog ▁Com ▁two 建设@  ▁after ps ▁& 可能@ 这样 人的 ert Th@ ise li hen ▁bet@  ▁through any  ▁under 社会@ ug ock ▁em  ▁play@ ward ▁no sh row@  ▁only ife ating 已经@ 大学 ork ty  ▁serv@ ific  ▁cons ▁inv ▁hel@ 学校 ▁bu 中心 ▁bl@  ▁stud als pect 来源@ ting ▁co yst 防控@ —— ics  自己的 ause@ ▁att 专业 所以 ▁imp@ ivers  ▁produ 都是 und@ ring io arch eth@ ▁reg 相关  ▁back ased@ ough  ▁This ,” oint@ ily ited uring rib@ ▁Pro 使用 oth ▁may@ 开始 chool ments ict@ velop  ▁well elf ful@ 重要 ek  ▁make erm@ iness 联系 ren ▁pr@ land ata alth ange@  ▁every ▁fl ts! ▁col@! 的一! 市场! ience" ▁Ar@" 也是"  ▁know" ▁sec# 研究@#  ▁these# ▁ph# 这些$ 还是@$  ▁where$  ▁look$  ▁help%  ▁could@% ▁inc% ener%  ▁many& riv@& ility& 网络& ex' ices@' 好的' 现在' 方式( ▁tra@( 技术( 单位( erson) ▁fin@) ys) gan) oss* 但是@* ▁pol* ures* ement+  ▁want@+ ▁rem+ ▁ass+ ash,  ▁diff@, 关注, 朋友, bs- ▁ac@- ystem- :“- aking. 同时@.  usiness. ▁rel. ues/  ▁exper@/ 微信/ ▁set/ 组织0 ract@0 ▁ent0 ull0 ▁now1 ower@1 一定1 ▁De1 ative2 老师@2 uch2 ible2 ular3 部分@3  ▁world3 ▁Wh3 ▁use4 到了@4 oun4 ner4 az5 up@5 非常5 所有5 ww6 air@6 ix6 个人6  ▁show7 pr@7 内容7 |　7 发现8 its@8 ▁{8 fore8 ax9 irect@9 不能9 ▁way9 世界: 知道@: 能力: ied: ▁ke; ▁dec@; St; ▁Sh;  ▁pres< meric@< 提供<  ▁count< 影响= 的是@= ank=  ▁start=  的时候> ▁own@> itt> ▁see> 招聘? ▁Se@? ah? ҉҉? 经济@ con@@  ▁very@ ▁Al@ insA ▁fam@A 一起A ▁artA ▁himB view@B manB ▁'B 发布C  ▁right@C ▁locC  ▁evenC asonD 以上@D ientD hesD  formationE ced@E gramE ▁YouE 选择F  ▁such@F  ▁instF  ▁againF  ▁highG iel@G ▁LeG  ▁developG ▁AndH the@H  ▁leadH 出现H 产品I hip@I retI chnI otherJ ale@J venJ  ▁specJ 医院K ince@K 全国K letK ▁monL ▁eff@L ▁endL  　|　L 为了M  ▁should@M 文化M 城市M 根据N ▁mem@N  ▁thenN ivesN 不要O ble@O 政府O  ▁beingO 设计P ines@P ▁guP vernP  ▁callQ ▁did@Q ▁carQ ucQ 其他R  iversity@R ▁smR WeR  ▁transS 对于@S 平台S penS 你的T ▁An@T 目前T  ▁longT 考试U atch@U  ▁publicU  ▁supportU  ▁becauseV 还有@V 报名V 一些V ohW ▁sur@W 开展W ollowW unityX ouse@X  ▁goodX  ▁madeX  ▁lastY 电话@Y  ▁AmericY 方面Y 作为Z 中的@Z 工程Z  ▁greatZ ution[ ▁ret@[  ▁team[ ▁gr[ ▁ins\ ▁med@\ to\ ital\ **] ▁Tr@] 主要]  疫情防控]  ▁char^ cept@^ ▁fil^  ▁much^ ets_ 大的@_ 交通_ ton_ Wh` ave@` uth` 不会` ▁eara 发生@a ▁bela amesa 环境b  ▁those@b ▁modb ologyb msc 以及@c agesc 部门c  ▁downd ces@d 规定d  ▁businessd ▁done  ▁before@e  ▁reque ▁defe ▁numf ▁br@f tenf  ▁findf  ▁markg oci@g lineg  ▁bestg ▁serh 系统@h  ▁differeh ▁vish istsi read@i ▁curi  ▁classi cessj ▁Sc@j ▁edj ▁crej ledk ▁min@k Itk 是一k  ▁doesl 更多@l 或者l 地区l weenm  ▁take@m ▁Clm 美国m illionn way@n 喜欢n athern ▁rao rent@o 责任o ……o 怎么p ern@p  ▁interp ▁avp  ▁readq  ▁while@q  ▁lifeq ▁Spq 了解r imes@r  ▁systemr ▁subr 来的s 生产@s egs roups ://t orn@t 有限t  ▁persont elsu alk@u 视频u  ▁hereu 知识v ▁‘@v ▁Forv ▁dayv ▁Indw uss@w  ▁foundw  ▁thinkw  ▁discx oot@x 了一x derx ,　y 做好@y  ▁informationy 注意y  ▁homez ▁—@z Thisz ▁Asz ize{ 如何@{ vent{ ▁ext{ 精神| ode@| ands| 进入|  ▁between} 那么@} ▁Ad} 成为} WS~  ▁report@~ ▁exp~ ute~ 关于  ▁conf@ 病毒  ▁used ered  ▁care ield@  ▁sign` ▁ref aj ▁str ann akes ross 新冠@  ▁child` 不同 第二 ars arm  ▁cour  ▁around  ▁cent@ ▁det` ral ▁Bl  ▁follow 表示 ually 教师  ▁program@ tt`  ▁three ever ional ▁But 能够 oad ▁|@ ina` 每天 ox gy ety  ▁provid 其中 ▁Fr@ att`  ▁design 身体 usic ▁som 标准 ▁sim  ▁both@ 北京` 一步 bers ier  ▁during ik  ▁including ▁Is@ ER`  ▁still ▁rep ▁fun ants 数据 ished 特别@ 之后` ness 价格 eng 这种 今天  ▁school ism@ 点击` 资源  ▁contin 起来 ▁Ph 其实 ▁che ▁val@ ▁*` mber 方法 完成 学院 ▁del 书记 ▁gl@ ality` ght ved 一次  ▁book 7 0▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁University 比较@ ▁run` ision  ▁each 重点 ▁inf ized 基础  ▁company@ 领导`  ▁import 期间 综合 lease arg ▁Z CWS@ au`  ▁techn 提升 ee ▁Pl  ▁health  ▁point 投资@  ▁build`  ▁same nce 行为 ural ired  ▁form ▁Col@ 通知` ple 过程 上的  ▁different ▁cr  ▁project 参加@ 这里` 病例 政策 ▁[ 提高  vernment ature  ▁data@ 及时` 实现 () ▁Q 群众  ▁game ined  ▁gener@ ering` 条件 ▁cap ▁sol 产业 岗位 ▁Ex 机构@ ▁adv` eb 最后  ▁prof  ▁open 年的 son ving@  ▁post` ID omen 关系  ▁direct ery 希望 保障@ ari` 计划  ▁month 现场 阅读 ▁iss  ▁Comm 卫生@ ▁She` ▁Con erest 自然 基本 容易 ared 免费@ ▁ann`  ▁week 直接 全面 of  ▁dist  ▁really ▁ob@ ruct` 而且 app 行业 作用  ▁invest ploy inal@ riend` 检测  ▁going by 家庭 推荐 看到 公众@ me`  ▁place 支持 ▁Te 按照 IN arn ccess@ ▁say`  ▁resp  ▁product  ▁organ 质量 cent 一样  ▁appro@ br`  有限公司 orth ▁law ▁top reen  ▁pass  ▁list@ outh` 坚持 公安 ▁If uro ▁mov 合作 ining@ ustom` 家长 国际 有效 ▁Res 加强  ▁result 落实@ ,“` ▁dep  ▁process 会议 风险 ody ▁cor 科技@  ▁Cent` 这一 ▁ty 处理 ember 课程  ▁says  ▁rese@  ▁writ`  ▁stat oard  ▁partic rist sc raph ense@ 积极` ump ission iving 一直  ▁since 历史  ▁family@ co` ways ator  ▁prot ▁la 文章 ley ocial@ 编辑` 不断 分享 代表  ▁hist 患者 科学 ▁cle@ 具有` 负责  ▁perform 有关 ▁sl  ▁against 作者 肺炎@ ▁US` ▁el Ch ron  ▁state  ▁second ," 这是@ 成绩` 获得 ON  ailable 事业  ▁expl 社区 结果@  ▁They` ▁leg 感染 :　 虽然  ▁real ▁All min@ be`  ▁main ▁sk 不仅 觉得 enn EN  ▁happ@ 帮助`  ▁number ▁lot ohn 小学 self 地方 ▁Mar@ )　` ▁pur 小时 只有 ially  ternational  ▁fact 培训@ ▁Be` ▁Jan  ▁rest ling ency ▁How 医疗 ▁big@ 解决` ior 就会 + $▁▁▁▁▁▁▁▁▁▁▁▁  ▁rece ▁est AR ves@ ases` ▁few 今年  ▁current less ▁pop  ▁allow pro@ 集团` ically ▁i 毕业  ▁oper 应该 我的  ▁next@ ▁pat`  ▁interest gg 甚至 uthor 治疗 ▁/ 一般@ 必须` ury  ▁using ration 不过 que une 手机@ 措施` 继续 新闻 努力  ▁poss nect 保护 教学@  ▁incre`  lection 实施 err  ▁const ording ead iver@ ird`  ▁plan ott ▁fr 建议  ours@ 法律`  ▁sing arent  ▁turn 农村 As aut  ▁services@ lex`  ▁hard oney  ▁success ged 有人 无法  ▁view@ ured` 来了  ▁cost 经验  ▁group  ▁never ▁Rep  ▁return@  ▁story` 造成  ▁John  ▁season oor  ▁change 电子 由于@ 只是` 确诊  ▁children 是否 itions ource 同学  ▁friend@ ▁Pr` uture uck ▁aff 销售 控制 真的 指导@  ▁water`  ▁record 水平 开发 ising ▁Med  ▁case www@ for` ▁Br AS  ▁research  ▁belie  ▁elect 干部  ▁able@ 执行` ified aff ived If  ▁proble ify ▁equ@  我们的` ider ocus 结合 年级 ▁Am 作品 第三@ ope`  ▁▁▁▁▁▁▁  ▁provide 简单 汽车 二维 ▁met 这么@ 决定` ▁bre 出了 严重 ister 业务 viron  ▁meet@  二维码`  ▁chang ourn inc 艺术  ▁today ▁beh aster@ ument`  ▁fund  ▁little urs  这样的 ▁got 最新 左右@  ▁within` 只要 都会  ▁film put  ▁days tain osp@ ii` aim 记者  ▁past uary  ▁less 委员 超过@  ▁expect` 的大  ▁might  ability 更加 ients 感觉 org@ 目标`  ▁visit 价值  ▁area ▁Fl  ▁Euro  ▁event 区域@ urity` ▁At 介绍 ateg ential ▁Gr 方案 til@  ▁York` ideo obal  ▁activ 都有 icy  ▁priv  ▁social@ ▁Su` ▁Pol By What  ▁employ An  ▁With  ▁present@  ▁across` 需求  ▁near  ▁special  ▁Pres ▁So ▁air ems@ 购买` 接受 不到 的话  ▁camp 公告 Pro ask@ gin` ane  ▁conc 接种 职业  ▁Christ 车辆  ▁proper@  ▁addition` 一点 over ele side 优秀 arge  ▁women@ 人才` 正在  ▁making ▁ask oid  ination 调整 范围@ orts` 正常 iven  ▁city  ▁World ▁exc  ▁members  ▁National@ 办公` atic ball  为什么 面试 ▁spe 专家 %,@ 空间` 即可  ▁profess 形成 ales ext  artment 国内@ ards` AP 政治 ample  ▁School ograph ▁Ass He@ ▁No` my  ▁That ike rict 疾病 ▁z wards@ 除了` 利用 来自 ev duct ▁Gu 然后 atter@  vironment` 确保 ienc 症状 小区 ▁job  ▁major  ▁State@ ▁par` 之前 达到  ▁pract ▁cho ▁ben  ▁educ ending@  ▁Inst` idd  ▁creat  ▁phot  ▁impro  ▁order 个月 ▁saf@ 别人` 查看  ▁City  ▁protect ready 有的 eter 幸福@ 经营` You 共同 iam  ▁along 武汉  ▁move ▁Min@ ention` ung  ▁Count eral  ▁term  ▁feat  ▁webs  ▁polit@ ▁App` 比如 IS  ▁young 不可 lish che 公共@  ▁econom` ians 妈妈  ▁development ▁mil 关键  ▁give 心理@  ▁supp` How 真正 完全 表现 经过  ▁called  一定要@ ably`  ▁focus wit ▁: ris 生态 veral 工资@  ▁orig` 交流 消息 ills 东西 ▁En 全球 机会@  ▁Europe` ▁Oct 直播 得到 一种  ▁left 幼儿 alf@  ▁become` lege gether  ▁together 志愿 严格  ▁charact  ▁future@ ners` ows 招生 机关 驾驶 保险 For  ▁News@ 产生` go 面积  ▁rele 女人 ials  ▁trad  ▁fore@ 来越` itor val ▁aut IC 违法 出来 ended@ pril` 不少 ▁Ed ances IT 隔离 阶段 ored rop0  ▁sure@  ▁historyP  越来越` ivelyp  ▁applic yle 加入 ▁far 有些 aces 来看 ): 推动 ↓↓ iting   ▁online0 多少@ bookP  ▁include`  ▁opportp  ▁commun erence 让人 全部 yl 思想 党员  ▁away resent ique rem   ▁until0 ▁let@  ▁connectP ▁Af` 随着p ▁Bro rap 属于  ▁news  ▁▁▁▁▁▁  ▁line ony 拥有 系列 ▁non  ▁based  ipp0  ▁series@ ▁optP anies` ▁likp 变化  ▁March 里的 也会  ▁mult 收入 AL itive 客户 英语 ey  ▁why0 任务@ 想要P 力量` ▁Xp ▁dev 主动 我国  ▁industry  ▁Aust ones  ▁January ”　  ▁several  ▁already 之一   ▁announ0 回复@  ▁condP  新冠肺炎`  ▁earlyp 集中 文明 ▁Off 身份 ▁Cal joy 调查  ▁site 网友  ▁States 以后  ▁net0  ▁South@ ExP  ▁times` ▁Abp ED  ▁author ene 　( 减少 ots  ▁understand 意识 ', 主义 ▁Aug  中华0 return@  ▁oftenP ▁Mc` 小编p 确定 ▁Im ds  ▁cult abor 建立 pos utions irl imate ources  欢迎0 ▁ve@  ividualP ster` 道路p ched  ▁food 状态 办理 接触 删除 ada  ▁short 人们 leg 制度  ▁following0 也有@ 最高P 传统` rawp AC 新的 ray 经常  ▁vers 符合 reet  ▁known rew  ▁create ued not0 稳定@  ▁humanP 操作` iftp aring 高考 就业 居民 广大 ger ▁dri ▁pot 员工  ▁five 儿童 版权0 oman@ 受到P ails` 团队p  ▁money  ▁looking 行政 以来 领域  ▁invol  ▁course  ▁live back  ▁compet 明显  ▁North0 而是@ 食品P 设备` idsp 公开  ▁Sept Com ole 高校 改革  ▁learn ▁occ  ▁April  ▁possible de  ulation0 Al@ rollP 地址` 采用p urther comm  ▁What raft  ▁chall hers 取得 登记 cri 明确  ▁ident  utes0  ▁media@ 高速P  ▁having` cemberp  ▁ever ▁jo ml  ▁June year 显示  ovember 证明 面对 ▁won isc   ▁percent0 广东@ 无论P 医学` REp 特色  ▁came 那些  ▁value 山东 ▁di 最终  ▁website 增长  ▁kind ▁Dep   ▁study0 打造@  ▁largeP elling` selp 当时 ape ethod 适合 提醒  ▁President  ▁others 应用  ▁later ▁key  过程中  避免0 正式@ 亿元P 丰富` 下来p  工作人员 ▁Feb 促进 方向  ▁account ▁war 模式  ▁treat aper 新增  ҈҉҉҉   ▁July0 联合@ ▁SecP  ▁Cont`  ▁holdp  ▁house  ▁deal There ▁lim  ▁among ief 观察 ▁ca Con 深入 劳动   ▁review0 相信@  ▁individualP 充分` Onp  ▁companies hel  ▁light  ▁function verage  ▁thought ▁htt 告诉 ctor ▁ris  ▁Serv  iday0 出的@ whP ▁bas` 成了p  ▁makes 医生 用户 urch ilt ▁mar  ▁benef 街道 金融 gr Ph  ▁#0 犯罪@ iodP most` 学历p 保证 iol  ▁field (" 并且 play ▁men  ▁technology ▁Har new ▁Ge  ▁dr0  ▁offer@ timeP  ▁strong`  ▁tookp  ▁didn ▁Em  ▁told But ▁@ 如此 形式 load fect  ▁larg  ▁publ  uff0 年轻@  ▁needsP ST` 快乐p  ▁night  是一个 针对 aps  ▁anal ogn  ▁Will  ▁enjoy  ▁room ▁Gl 统一  成长0 如有@  ▁comesP 感情`  的情况p  ▁thing  ▁mater 新型 做到 线上  ▁months ma  ▁mean 过去  ▁products 说明  aken0  ▁self@ rainP  的问题` 体验p 办法 是在 培养 unch 监督  ▁personal  ▁building 习惯 改变 长期 现代  uf0  ▁clear@  ▁bringP ips` earchp  ▁started ▁Dr ators 立即  ilities now 提前 主题 恢复 结束 idence  施工0  ▁control@  ▁WhenP 审核`  ▁videop 许多 认真 ▁Art cer 目的 ival ches Sh off 快速 To  ▁hig0 ained@  ▁ParkP reg` ▁Agp ournal ▁Me itle  ▁means  ▁Part 理解 分别  ▁least 只能 ler irm   ▁song0  ▁talk@ 当地P ides`  ▁sidep 实验  ▁enough 日本 ▁sum sw 中学 vert 游戏  ▁particular  ▁pain 就能  当然0 ana@ 资金P 本文` 农业p Ar  ▁December 开放 识别 ▁et  ▁November 传播 性的  ▁certain  ▁October  就可以   ▁Comp0 nov@ AMP  ▁final` ▁COp set ze ▁bo head ither ▁tem ▁doc  ▁done ▁War 年龄 ▁wr 体系0 ▁mom@ ▁taxP When` 对方p And  ▁though 先生  ▁quality  ▁ener ison  ▁vari 采取 规范! opy! ▁tre ! ament0!  ▁environment@! luP! 在线`! 依法p! ancial! illed!  ▁given! 智能!  ▁September!  ▁games! ▁One!  ▁appear!  ▁example" irc" 数学 " 并不0"  ▁claim@"  工作的P"  ▁County`"  ▁Healthp" 网上" 面的" crib" ▁enc" ▁deb"  ▁Center"  ▁private"  ▁seen" ▁ant# ▁vol# 公布 #  ▁doesn0# 结构@#  ▁critP#  ▁role`#  ▁initp# 福利# 每个# 青年# angu#  ▁comb# ▁var# 侵权# ▁tot# 一切$  ▁space$ 电视 $ 中医0$ 法院@$  ▁WestP$  ▁land`$  ▁happenp$ ounc$ New$ 电影$  ▁America$ 尤其$ ▁Bar$ ests$  ▁issues$ room% 体育% 遇到 %  ▁national0%  ▁once@% ibrP%  ▁type`%  一个人p% ▁ens% ▁fav% 皮肤%  很多人%  ▁share% 计算% 都不%  ▁doing%  ▁August& 后的&  ▁Afric &  ▁House0& 经历@& ▁ActP& 参考`& 困难p& ▁My& ▁ago& 一天& ('& igital& ▁<&  ▁Stud& ▁Par& 精彩' 营养' umb ' ruary0' ilar@' ▁audP' 意见`'  ▁redup' aur'  ▁press'  ▁stop' ▁Cor' 都在'  ▁content' minist'  ▁discuss' ▁low(  确诊病例( ▁fre ( 核心0(  ▁relations@(  ▁formerP(  ▁complet`( 享受p(  ▁super(  ▁require( oon( Re( ino( 男人( 支付( elt( right)  孩子的) ham ) 防护0)  ▁port@) 天气P) arget`) 程序p) 十分) ▁due)  ▁address)  ▁page) 压力)  ▁getting) 民族)  ▁track) 音乐*  ▁impact*  ▁works * 一年0* 商业@* 距离P* import`* 优势p*  ▁specific* 工业* gress* 民警* ▁Dav*  ▁death* 食物* ▁UK* aid+ Ad+  ibility +  ▁single0+ ▁six@+ 设施P+ 发挥`+ blp+  ▁Austral+ 长按+ anks+ ecut+ ▁pri+ ::+ ington+ 死亡+ 位置, tr,  ▁sound , ault0, ET@, 报考P, Un`, 成立p,  ▁went,  ▁check, itted,  ▁according, ▁Do,  ▁contact, 训练, nes, ▁yet-  ▁Brit- key - emic0- ▁buy@-  ▁responsP- 家人`-  ▁thirdp- 大量- ocr- ille- 近平- fort- 满足- rodu-  ▁begin- ades. 考虑. 身边 .  委员会0. 机制@. 核酸P.  习近平`.  的时间p. 费用.  ▁Mark. 真实. 引起. 重大. bum. 原则. elebr.  ▁represent/ urg/ 交易 / chie0/  ▁energy@/ 文字P/ ▁Sm`/ 认识p/ ondon/ ague/ 录取/ ();/ atur/ used/ ▁El/ 的高/ 日常0  ▁body0  ▁further 0 EC00 女性@0 都能P0  ▁whether`0 asingp0 市民0 事故0  ▁host0 ▁Qu0  如果你0 答案0  ▁beaut0 学会0 ▁bit1  ▁step1 党委 1 atest01  ▁These@1 特殊P1 cast`1  ▁footp1 iter1 下的1 US1 ▁typ1 河南1 领取1 湖北1 下午1 添加2 让你2  ▁international 2 https02  ▁However@2 linP2 整个`2  ▁continuep2 lete2 如下2  ▁period2  ▁signific2 fer2 aul2  ▁leaders2  ▁coll2 ouncil3 ▁By3  ▁true 3 aign03  ▁hours@3 ▁CheP3 增强`3 深圳p3 规模3 ▁cou3 add3 召开3 irt3 ends3  ▁range3 ociety3 日起4 ▁His4 adu 4 BC04 cy@4  幼儿园P4 合同`4  ▁Trumpp4 ▁Her4 开学4 ▁age4 icro4 技能4  ▁quick4 感谢4 累计4  ▁Russ5 oe5  ▁problem 5 http05  ▁added@5 降低P5 alt`5 ▁Regp5  ▁tell5 她的5  ▁parent5 提出5 各地5 ****5 看看5  ruction5 实践6  ▁Cour6 ▁upd 6  ▁February06  ▁staff@6  ▁globalP6  ▁travel`6 值得p6 ule6 生物6  ▁rights6  ▁events6 ama6  ▁performance6  ▁previous6 人数6 atural7 apt7  ▁half 7 数字07 场所@7 restP7 ecess`7 mitp7 智慧7  olution7 正确7  ▁particip7 掌握7 会有7 配合7  ▁heart7 ▁Pe8 制作8  的重要 8 消毒08  ▁member@8  eciallyP8 error`8  ▁actionp8 though8 样的8 高中8 agn8 atform8  ▁miss8 ried8 事情8  ▁mess9 ees9 地点 9 onse09 动车@9 clusP9 冠状`9 记录p9  ▁actually9 设置9 era9  ▁International9 土地9  ▁areas9 涉及9 声明9 ▁esp:  ▁wind:  ▁education : ▁Rec0: HE@: rugP: 一位`: ▁Perp: 完善: 测试: 年度: uk: aged: ▁Not:  ▁contrib: 另外: 运营;  ▁taking; ald ; 在看0; ▁cut@;  ▁breakP; 应急`; ▁Mrp; ▁mot; 今日;  ▁career;  ▁below; 用于; 整体; ▁Law; hern;  ▁question<  ▁meas< 多的 < 优惠0< coming@< hibP< 作业`<  ▁Afterp<  ▁mind< 近日< 本次< ▁hom< ett<  ▁reason< Cl< iences<  ▁security=  ▁famil= 乡村 = ogle0= ederal@= ▁OurP= 部署`= 理财p=  ▁easy= 预防=  ▁office= 心的= 监管= ▁hot= oph=  ▁sent= ▁sw> ▁lo> 酒店 > 回家0>  ▁fall@>  ▁presidentP>  ▁Associ`> ensivep> 日报> 时期> na> aching> 公里>  ▁watch>  请联系> ently>  ▁held? amed? eed ? 官方0? 程度@? 最近P? 总是`?  冠状病毒p? 情绪?  ▁Mich? 强化? ender?  ▁character?  ▁improve?  ▁results? 爱情? lev@  ▁soft@ str @  ▁countries0@ eal@@ 方便P@ 自身`@  ▁Directp@ 位于@ ops@ 语文@ ume@ 每日@  ▁shows@ 老人@ 最大@ kyA 提示A 浙江 A 的心0A ully@A  ▁HighPA  ▁forward`A agingpA 更是A  ▁LondonA  ▁closeA 垃圾A  ▁estabA  ▁collectA ependA  ▁featuresA 城区B iclesB 这次 B div0B 意义@B 共产PB 如今`B  毕业生pB itteB uxB 变得B  ▁recognB  ▁takenB OSB  也可以B ▁NetB 有了C gesC  ▁face C ores0C  ▁court@C 带来PC 证书`C entalpC WithC icultC selvesC 人体C  ▁violC  ▁significantC  ▁perfectC itesC 经典D  ▁existD uild D 转载0D hold@D  ▁popularPD 基地`D  ▁managementpD ulesD ▁waD  ▁packD 不好D  ▁textD accD gingD aughD 指挥E  ▁physE ided E  ▁India0E  ▁issue@E  ▁likelyPE come`E ugepE basedE  不知道E  ▁pointsE  ▁frontE IPE  ▁receivedE  ▁DepartmentE 在家E suF shipF  ▁political F 网站0F 上午@F ▁TVPF arter`F aelpF  ▁attackF  ▁methodF  ▁internF  ▁almostF  他们的F isticF isesF  ▁everythingF uationG  ▁achieG  ▁believe G  ▁common0G  ification@G 不管PG 合理`G  ▁ChinapG 防疫G  ▁policyG  ▁installG 成员G 年来G ifornG emptG  ▁playersG 合格H ▁DonH 首先 H 名单0H  ▁stay@H TrPH cont`H 战略pH flH ▁honH  itionalH faceH 到底H partH ▁AngH 全市H 成本I iresI  ▁behind I  ▁Just0I yd@I  ▁comingPI  ospital`I  ▁clospI 母亲I ▁DayI  办公室I 消防I  ceptionI 事件I cleI borI hingJ 往往J  ructure J 运输0J  ▁various@J 联网PJ  ▁entire`J 动物pJ  ▁currentlyJ  ▁toolJ 职工J  的工作J 永远J 各类J 儿子J 江苏J ▁hitK  是一种K ▁Int K pre0K  ▁sugg@K 后来PK 所在`K COpK  ▁objectK 报道K ADK 强调K 此次K ▁douK  ▁trainingK acedK atoryL  ▁MoreL 哪些 L 各项0L  ▁position@L ▁SchPL 预约`L  ▁offerspL 本科L 再次L iddleL itaryL 一线L 景区L idgeL ▁GodL 根本M ailyM  ▁created M  的事情0M  公安局@M  ▁relationshipPM 下方`M itutepM  ▁especiallyM uelM ▁SanM 扫码M akerM EOM  ▁productionM groundM  ▁probN  ▁priceN  ▁tour N ▁op0N 自治@N  iforniaPN 公路`N 自由pN  ▁modelN  ▁aboveN ▁MonN ledgeN  ▁streN  ▁wholeN  ▁credN 肯定N 不用O  ▁failO 维护 O 出行0O ▁Tra@O  ▁friendsPO 广告`O  ▁originalpO 几个O ooO 案件O  ▁agreO  ▁articleO  ▁replO ▁SerO  ▁livingO  ▁livesP 委会P 找到 P  ▁became0P 一名@P  进行了PP 引导`P ▁DispP  icationP ictionP 照片P  ▁versionP 自我P ranP ypeP syP 配图Q  ▁recentlyQ  ▁investig Q 养老0Q  agement@Q 党的PQ 应当`Q untpQ  ▁releaseQ 专项Q 大多Q cialQ ovedQ  ▁providesQ 校园Q  ▁campaignQ  ▁tryingR  核酸检测R  ▁walk R useum0R  ▁total@R  ▁concernPR ▁ep`R  ▁enginepR 待遇R  ▁pickR AllR 　“R  新型冠状病毒R  ▁roadR ▁barR  ▁policeR  ▁assistS  ▁boardS  ▁celebr S  anguage0S  ▁amount@S 沟通PS ▁Sen`S 下载pS 理论S  ▁financialS 公园S otoS 文件S  ▁workedS  ▁considerS  ▁seemS  ▁albumT 笔试T 句话 T 基层0T 千万@T  ▁includesPT iet`T ▁eypT 对象T  ▁elseT ▁domT ▁DesT  ▁latestT  ▁announcedT amsT  ▁ideaT ▁genU ▁FinU 三个 U VID0U ▁Of@U modPU  ▁similar`U 改善pU  ▁BlackU 输入U 房屋U  ▁selectU  ▁CollegeU 临床U 高的U 获取U 河北V 自动V  ▁file V oud0V ufact@V 美好PV ▁Inc`V 展示pV  ▁KingV  ▁purchV 内的V 市委V 大会V ▁ValV  ▁consumV 注册V mmW ContW  ▁associ W 伙伴0W ▁Air@W 学科PW  ▁reported`W ILpW 基金W ▁logW 感受W ijW 红色W  ▁changesW ambW 下降W 表达X ▁immX 上了 X 查询0X 经理@X  的地方PX  ▁risk`X  ▁decisionpX essionX 必要X 指出X  ▁casesX fordX araX  ▁propertyX ELX ▁morY 不了Y iple Y 的好0Y  ▁black@Y apanPY 四川`Y OTpY 不再Y  ▁successfulY  ▁howeverY  ▁opportunityY  ▁someoneY 美丽Y AtY zhY serZ  ▁fireZ  ▁everyone Z  ▁customers0Z  ▁minutes@Z ▁xPZ  ▁date`Z 创业pZ icaZ  ▁townZ  ▁targetZ 留言Z  ▁GroupZ  网络配图Z ##Z  的生活Z  ▁difficult[ 然而[ name [  ▁questions0[  ▁general@[ 打开P[  ▁respect`[ 近期p[ 搭配[  ▁platform[  ▁Germ[  ▁takes[ 呼吸[ 加快[  ▁anim[  ▁potential[ 还要\ 细胞\ ▁Dem \  ▁provided0\ 于是@\  ▁searchP\ wood`\  ▁uniquep\ ▁rad\ ▁opp\ ▁La\  ashington\  ▁Court\ ▁ess\ 的新\ ply\  ▁released] obile] ▁inj ]  ▁contract0] 明白@] 更新P] 切实`] 出生p]  ▁began]  ▁challeng]  ▁wanted] 轻松] 新区] 贯彻] roups] 制造] 不足^ 就像^ orks ^ vis0^  的方式@^  ▁YourP^ 品质`^ 绿色p^  ▁party^ oring^  ▁increase^  ▁throughout^  ▁needed^  ▁either^  ▁ways^ ▁veh^ 曾经_ So_ iful _ 父亲0_ 思维@_  事业单位P_ 某某`_ 最好p_ ida_ 全省_  ▁growth_ 晚上_ 才是_ ection_ eks_ 连续_  ▁required`  ▁Char` 子的 ` 还能0` ivil@` 广州P` ▁Hol``  ▁latep` 宝宝` 人群` 那个`  ▁systems`  ▁ground`  ▁California` this` 体检` arca 一旦a  ▁pers a 记得0a ▁via@a 人口Pa 可是`a ishingpa  ▁networka ▁Aca 不错a 数量a duca 迅速a esea 整理a ▁calb 它的b RO b  ▁recomm0b 下面@b ▁undPb text`b  ▁insppb lierb fficb ▁argb  ▁fightb  ▁victb  网络配图网络配图b 补贴b  最大的b ▁>c  ▁pictc  ▁dest c 给予0c  ▁partners@c  负责人Pc  ▁leading`c  ▁publishedpc  ▁soonc 比赛c  ologicalc 制定c 改造c ativesc /　c  ▁legalc iated hoodd 爱的 d ▁Pat0d ittee@d 流程Pd 女儿`d ▁talpd umnd 语言d 物质d  ▁necessd ▁redd 以为d 实行d  ▁subjectd ancede ordere ports e  ▁built0e  ▁code@e  ▁playedPe 山西`e ")pe 比例e rece 启动e  ▁printe 供应e 当前e overede urbe  ▁writingf  ▁completef 工具 f  ▁hope0f ▁isn@f mingPf 治理`f 总结pf  ▁outsidef 创造f youf ▁Upf attlef  ▁Fromf  ▁wondf  ▁professionalf  ▁computg actiong arning g ude0g 事项@g 突然Pg 里面`g zzpg  ▁anythingg 公示g  重要的g witterg lightg  ▁emerg 南京g  ▁ratherg ughth ▁Volh ocal h 此外0h  ▁asked@h ▁poPh 平方`h  acebookph  ▁linkh  ▁Generh ▁,h 协会h 处罚h Sph INGh avingh  了一个i  ▁girli 给你 i 竞争0i 更好@i ellowPi  在一起`i  ▁determpi 速度i  是什么i ▁ledi  ▁archi  ▁quitei oicei  ▁Streeti ▁badi  让我们j ▁figj  ▁director j END0j field@j ▁stePj 免疫`j 即使pj 现象j  ▁acceptj ▁sexj URj 教授j  ▁primj ▁votj  ▁moviej 反应k identsk 少年 k ▁neg0k  ▁playing@k  ▁ensurePk  如有侵权`k 院校pk  ▁Philk 本人k  ▁expressk 优先k ysisk 收到k  联系电话k optk  ▁celll ▁Accl 第四 l  ▁projects0l 阳光@l nyPl  ▁Hist`l  ▁coverpl 月份l 终于l  ▁describl rabl ;　l 动力l 脱贫l 就要l inksm 攻坚m  ▁prop m  互联网0m  ▁introdu@m ▁RedPm  ▁medical`m roidpm dom  ▁awardm  ▁cleanm 现实m 因素m  ▁pleasem ▁illm ablesm 事实n  ecutiven 运行 n ▁deg0n orge@n 示范Pn 软件`n 独立pn  ▁parkn 交警n  ▁tablen agern  ▁Davidn  ▁remainn  ▁regardn 就在n 案例o ▁Steo  原作者 o urance0o 升级@o 重视Po 农民`o  ▁userspo mbo ▁пo 举行o downo 优质o ▁abso  ▁patientso  ▁sizeo asticp ▁toup  ▁email p  ▁English0p 高度@p 相对Pp 同志`p  ▁masspp agsp 读书p 放在p  ▁documentp 湖南p relp 想到p стp  ▁favorq 承担q ached q PS0q ▁Sw@q  ▁accomPq ico`q 离开pq 预计q markq cingq 等等q  ▁Washingtonq ceedq 收费q izingq CCr 商品r mp r  ▁exam0r BA@r ▁damPr  ▁expected`r 成果pr  ▁plansr  ▁Fridayr  ▁Foundr viller  itutionr aintr  ▁historr 评价r  ▁Books 研发s uses s  ▁running0s  ▁integ@s  ▁EastPs oke`s  ▁whiteps  研究生s 时刻s 人类s  ▁Canadas 同样s  ▁designeds ivitys  ▁brands  ▁Councilt 粉丝t  ▁answ t 资讯0t 全力@t 鼓励Pt  ▁Sund`t Apppt 为主t  ▁circt  ▁Researcht hat  ▁Const  身份证t  ▁stylet  ▁weekst lau 复工u airs u 们的0u  ▁billion@u 平均Pu 愿意`u Onepu defu 小组u 投稿u secu 也要u 危险u rownu 尽量u ▁hopv irthv  ▁pret v  ▁European0v 顺利@v oomPv  ▁matter`v  ▁studentpv 春节v imsv 您的v 统计v 队伍v  是因为v 从而v  ▁naturalv 能源w 充满w Sc w  ▁included0w 一条@w rsPw No`w 岁的pw  ▁involvedw  ▁affectw  ▁administw wow umentsw classw 广场w 平时w  第一时间x 举办x ▁bi x 用的0x 一份@x  ▁JapanPx  ▁band`x ibrarypx 依然x  ▁userx  ▁goalx  最好的x 主席x  ▁cookx 老年x 都要x  ▁Republicy 课堂y ▁Now y  之间的0y 密切@y 学员Py den`y  ▁seemspy 取消y ":y  ▁moderny  ▁momenty  ▁Profy 详细y 截至y  ▁lossy 各位z vestz  ▁higher z  ▁books0z  ▁meeting@z ondayPz 自行`z 地产pz 还会z ariesz akersz 党史z  ▁ratez  ▁revez  生活中z  ▁Businessz ension{ vey{  ▁innov { resh0{ 去年@{  ▁groupsP{  ▁digital`{ 上市p{ My{ 女士{ Is{ 和国{ istry{ 血管{ 始终{ 几乎{ 特点| 日子| 铁路 | 不知0| ror@|  ▁TransP| 整治`|  ▁simplep| 应对| alls| ION| ref| 哈哈| iles|  ▁region|  ▁manufact|  ▁Dist}  ▁prior} cn }  ▁problems0}  ▁Techn@}  ▁influP}  ography`} 力的p}  ▁economic} 美的} 有着}  ▁lost} II} 生肖} 革命} 天的} 云南~ 安徽~  尤其是 ~ ▁Mor0~ 相应@~  ▁diseP~ 原创`~ Ourp~ ▁sus~  ▁receive~  ▁deep~  aturday~ 居住~ 高级~ iece~ 能量~ 商务 rect 每年   ▁base0  发展的@  ▁detailsP ▁Wed`  ▁Williamp pped  ▁cand  ▁probably 先进 错误  ▁Post 考核  ▁words 家的 贡献 确认 天津 有多 log(  ▁approach0  ▁path8  ▁broad@  ▁programsH  ▁FirstP roughtX  ▁worth`  ▁strih oppp iationx  的朋友 常见 练习 ▁web  ▁interview 重庆 )(  ▁Year  ▁terms  良好的Ȁ 重新Ѐ ▁sym؀ 申报 者的  ▁learning  ▁Review  可能会 心里 逐渐  ▁themselves icated estern( 执法0  ▁treatment8 col@ zyH 往期P  ▁optionsX 青春`  ▁Whileh 家里p  ▁suggestx 主持  ▁screen ;( ares 初中  ▁mother  ▁nation  ▁woman aches raelȁ  ▁averageЁ ements؁  ▁System Id ▁sat  ▁drug mon  不同的 具备 ▁Dec 贷款  ▁sense(  是不是0  ▁itself8 ength@ ))H erousP asonsX Se`  ▁declh  unitiesp operx anch  ▁defin  ▁https etimes  也就是 Pl  ▁sett  ▁regular 投入  ▁titleȂ  ▁storiesЂ 也能؂ ▁Pal 婚姻 资产 半年 生的  ▁written  ▁source 所谓  ▁application 他人( ▁obs0  ▁practice8 伤害@ 放弃H frP  ▁nothingX  ▁sales`  ▁drivh Abp 环节x ▁Out 动作 IM ▁der 中共 每次  ▁sold  的发展  新冠病毒 ercȃ  ▁struЃ  ▁simply؃  ▁round  ▁states 另一 周边  ▁longer  ▁prevent 植物 ening  ▁Program  ▁club( hu0 女孩8 Res@ 平安H 博士P 加上X  ▁Indian` ▁sawh boardp Namex  总书记 色的 集体  ▁activities  在这里 ▁jud  ▁attempt 态度 ustain ayerȄ  自治区Є 推广؄  ▁British erved ▁Are 期待  ▁Institute 这款  ▁safety abel 理念 ado( ▁son0 ▁Tex8 邮箱@  特别是H  共产党P 带着X ▁ut` 空气h 房子p  ▁requestx  ▁parents inary  ▁match 毕竟  戴口罩  ▁cannot 意思 即将 以前 发热ȅ ▁magЅ  ▁Australia؅ avy 福建  ▁knowledge  estival 反复 ago html  高质量 ▁Ob 紧急( After0  人民政府8 acing@  新时代H  ▁implementP  ▁threatX  ▁schools` islh  ▁additionalp 住房x 清楚 身上 word rench ▁в 优化 yp  大部分 asc 温暖Ȇ  ▁automІ 物业؆ 理想 一段 itect  ▁Sunday 相当  ▁Facebook 加大 双方 担心 多个( 发放0  ▁beautiful8 美元@ 在这H  ▁starP iberX oul` iringh wherep 慢慢x irus  版权归 ava  ▁Music  ▁located  ▁response  ▁extra 审查 确实  ▁imageȇ  ▁conductЇ ript؇ esc 坚决  ▁summer 技巧  ▁fast на ▁} 从事 illing ▁cy( ▁Cap0 多次8 还在@ ilsH ThatP  mercialX 第十` PPh 物资p 处于x  ▁color Col 开启  ▁inside ▁rev uesday 是不  mission  ▁culture  ▁AssociationȈ 英国Ј  ▁fans؈ var LE 监测  ▁leave  ▁mach  ▁statement 限制  ▁Director ao ▁hol( 之外0  ▁resources8 突出@ 男子H  ▁softwareP iiiiX 附近` ursdayh 思考p alsex ▁Ear  ▁pred 英雄 娱乐 机场 enge  ▁Some  ▁player  ▁Monday  ▁Paulȉ  criptionЉ oday؉  共和国  ications 状况 dom olar 声音  ▁related  ▁Educ  ▁families 村民( 香港0 上下8 ра@ netH ▁CrP 追求X  ▁pretty` AAh  ▁organizp  ▁anyonex 派出 inter unk 讲话  ▁distrib 公交 ▁box  ▁norm  ▁card 日至Ȋ  ▁encourЊ  ▁paper؊ ▁sen  ▁draw comes  ▁COVID  ▁center  ▁Public 防止 aly  ▁couple  ▁conditions( UT0 蒙古8  ▁comment@  ▁kidsH ▁panP  ▁collectionX 药物`  ▁languageh CHp inatedx 协调  ▁James 我市  ▁father  ▁stage uted arr  ▁prep  有什么 ctionsȋ erveЋ UN؋ 奋斗  ▁phone  ▁Google mas 很难 财政  ▁pric 干净  ▁military  ▁White( String0 二十8 成都@  ▁earnH  imatelyP  ▁surprX 蛋白` Listh  ampionsp APPx  ▁Island omb eren  ҈҉҉҉҈  ▁Cong 安装  ҈҉҉҉҈҉҉҉҈ ▁mid 通报 ViewȌ 收藏Ќ DA، 精准 有序  ҈҉҉҉҈҉҉҉҈҈҉҉  ▁organization  ▁serious  ▁Saturday ▁Ann ▁£  ▁largest ▁den( nesday0 西安8 通道@ ▁ModH 会员P 一体X 利益` 吸引h 兴趣p ▁widx 考察 ▁bed 完整 # ҉҉҈҉҉҉҈҉҉҉҈҈҉҉  ▁businesses itiz 律师  ▁ability  itation  ▁driveȍ 聚集Ѝ TV؍ iqu 覆盖 阳市  ▁Port ston gest 至少 imum  ▁bank pm( rote0  疫情防控工作8  ▁Jack@ 运势H dingP ▁BenX  ▁immed` EMh 学位p ▁plex  ▁developed 停车 ▁squ 衣服 ▁Ant 全体 ▁div  ▁ball 不如  ▁GreenȎ 相比Ў 百年؎ yond 来到  ▁store  ▁upon  ▁Science 前往  ▁Board  微信公众号 力度  ▁instead( azing0 排名8  ▁Africa@ ademH ubeP  来源于X  ▁Develop` 突破h 记忆p 警方x 温度  ▁Hall ▁fit 去了  教育部  ▁Best  ▁firm 绝对 过的  ▁collaborȏ  ▁sensЏ force؏  ▁Angel  ▁block  ▁sche ples 万人 irit 广西  ▁scient  ▁multiple ▁hor( ▁\0 ни8 排查@ 饮食H listP  ▁TexasX 渠道`  ▁blogh limp 检察x  ▁Service 境外  ▁gradu like  ▁levels 公务 idered  ▁Read hest  ▁progressȐ 反映А  ▁providingؐ 各级  并不是  ▁nearly  ▁ready  ▁quickly  ▁looks 解释 Im ▁Sim 退休( 路上0  ▁complex8 En@  ▁causeH  有一个P 眼睛X  lements` ▁Preh osingp  ▁deadx  消费者 唯一  给大家 分类 ▁ded  ▁hous ki ▁nov ▁Fil 令人ȑ  ▁materialБ criptؑ  ▁named 风格 wide using inese 节目  ▁Office olf  ▁growing  ▁skills( aves0 新疆8 每一@ 规则H  ▁DemocrP 最低X 调研`  ▁separh atalp Shex 强的  ▁prem 梦想 sych 形象  ▁teac  ▁teams 予以 原来 陕西Ȓ ▁matВ  ▁regulؒ 下了 ▁wel ▁Je  ▁standard urt  ▁rout Home  ▁himself  ▁Inter 坚定(  ▁plant0  ▁costs8 amily@  更多的H  ▁readingP  ▁MichaelX 主体` 哪里h bodyp  这就是x  ▁independ  ▁Israel  ▁capital 图文 大道  ▁feature  ▁Home band OL ▁Usȓ ▁NeГ udgetؓ 爸爸  ▁exact aby venue 多年 Bl  ▁federal  ▁goes 它们 背景( Get0  ▁employees8 地铁@ ▁oilH 多种P 使命X  ▁chance` ▁INh  ▁Generalp ▁licx ▁Dan 大队 时的 她们  社会主义 化的 支部 颜色 isions 本身Ȕ  ▁seekД % ҉҉҈҉҉҉҈҉҉҉҈҈҉҉҈ؔ  ▁trade 协议 ▁ver  ▁wide 女生  ▁cross 分数 ▁ord ounter pose( 污染0 lick8 广播@  ▁scienceH 义务P IVX  ▁race`  ▁Superh uyp Indx 一家 营业  ▁Mart 依据  unction 江西  人民共和国 see 谢谢 国务ȕ  ▁displayЕ 说的ؕ ka 回顾 长的 件事 不起 lying ▁Get 一生 ▁Ev ▁ten( men0  ▁subs8 nown@ ▁musH  ▁whoseP  中华人民共和国X 应聘`  ▁Tuesdayh 好了p 收获x oes rian  ↓↓↓ 贫困 工艺  ▁Rich 观点  ▁Great 探索 详情Ȗ rialЖ 财富ؖ 有所 ▁sil 检验 rast 上述  并没有  的同时 体温  机动车 Or(  ▁ideas0 摄影8 明星@  ▁maintainH  ▁decidedP 上涨X  出来的`  　/　h  ▁broughtp 诈骗x  ▁England ona 禁止 rated  大多数 ategy  ▁workers 违规  ▁official 开心ȗ  ▁communitiesЗ  ▁achieveؗ  ▁acqu 统筹 人在 通行 美食  ▁Thursday но 担当  ▁parts 形势(  ▁wasn0  ▁click8 lam@ ▁mixH  ▁WindP ▁DelX 扶贫` 感到h 配套p  aughterx ties ко  ▁effic  ▁wonder 内部 十年 ae 举报 ipment :(Ș 公益И 挑战ؘ  ▁Rober 结婚 有时 刺激  ▁rock  ▁served 不住  ▁respond  ▁neigh 的不(  ▁candid0  ▁Here8  ▁Sing@ ')H 有点P  ▁LifeX  ▁investment` encesh 复杂p  ▁opportunitiesx 变成 ▁boy 假期 融合  ▁saying 化学 懂得 做法 权益 一场ș houseЙ 评估ؙ  ▁streng  ▁considered 实力 多了  ▁demand iger  孩子们 ро  ▁Press 是个(  ▁word0  ▁wrote8 win@  ▁reachH  ▁attentionP 股份X  ▁Conf` 四个h hapsp 良好x 将会  都可以  ▁reports  ▁Services  ▁lower 印度 组成  ▁suff  ▁efforts itarȚ 职位К 熟悉ؚ 人大  ▁gave  ▁capt  ▁allows  ▁trust  ▁Gold ▁cam  ▁Church  ▁huge gen( bon0 或许8 考研@  ▁happyH 类型P eckX 休息`  也没有h  ▁Wednesdayp  ▁qualx 内心  ▁towards PA icip 助力 老板 iant bit 夫妻 分为ț 科研Л ios؛  ▁http 补充 世纪  ▁safe 体现 出租  每一个 ▁bal 加工 微博( 也许0 很快8  ▁govern@  ▁ArchH 下去P itcX  ▁contain` 食用h 神经p  ▁evidencex sequ 省委  ▁highly (　  ▁Flor  ▁gets 嫌疑 让我  ▁variety  中国共产党Ȝ BeМ cel؜ 素质 引发  ▁choose  ▁Twitter ache 本地 三年  ▁situation  ▁Commission  第一次( istan0 师范8  ▁message@ ▁BerH 称为P  ▁locationX ota` icineh morep ▁Magx  ▁Foundation  ▁enter met  ▁Company  ▁positive  的作用 berg  生活的  国务院 withȝ ancyН OP؝  ▁Award oyal za aled 司机 TS 血压  ▁usually 非法  ▁spent( 种植0 change8  ▁born@ 链接H  ▁expertP  ▁religX  ▁deliver`  ▁promoth  所有的p 随后x bre enty  ▁seven 出口 认定 扩大 很好 贵州 税务  ▁handsȞ 而言О ▁с؞  ▁traditional 准确  ▁Work ▁Mem 就不  ▁Thom  ▁speed  ▁necessary  还可以  ▁arriv(  ▁population0  ▁fair8  ▁wrong@ 文学H 效率P 专题X  ▁Education`  ▁clientsh 温馨p pperx iding  ▁shoot  ▁lack 角度 ▁hum  ▁thous 天下  ▁expand  ▁download 判断ȟ  ▁benefitsП lished؟ 我是  ▁century  ▁copy 流行 意味  ▁giving ocks 围绕 法规  ▁bott( While0 手术8  ▁earlier@  ▁givesH 会计P  ▁choiceX data` 辅导h jsp  ▁survx 理由  ▁customer They ▁Bel  ▁exhib ____ het ▁Ke 早上 大型Ƞ  ▁sellР 味道ؠ  ▁demon 好好 Man 很大 ▁Met 包含 chan date  ▁restaur 建成( ▁etc0  ▁increasing8  ▁nature@ 就有H  ▁interestingP hiX 动态`  ▁farmh ▁dogp 人物x 成分 奖励 mitted  ▁Franc 线路  ▁solution 随时  及以上 AB  ▁quarterȡ 看着С  ▁leaderء ▁Mac box 几年  ▁Over  ▁write 人力 加油  ▁hear  ▁eight 青岛(  ▁Road0 生长8 舒适@ ▁THEH  ▁climateP  ▁feltX  ▁cast` 两人h  ▁comfortp ▁femx fully  ▁excell ▁Leg  ▁touch just gl  ▁strateg  ▁wife  派出所 损失Ȣ  ▁morningТ 推出آ 传染  ▁French ya From 救援 采购 ▁aw 太阳  ▁Bank ancing(  ▁detail0 ▁sn8 cm@  ▁journalH liedP olicX CT` ▁bush questp  习近平总书记x  ▁solutions 日前 气温 风景 place 战争  每个人  ▁tick 女子  ▁saveȣ  ▁commitУ 放心أ istics 深度 iller  ▁credit 妻子 Post  ▁starting leep  ▁directly IG(  ▁avoid0  ▁section8 简介@ 十二H  ▁hospitalP ▁AddX ▁Mal` ▁Tomh Morep esterx bed 备考 财务 自主 PR 进口  一定的 难以 care  ▁vaccȤ 时尚Ф 百姓ؤ  ▁Museum  ▁creating  志愿者 讨论  ▁strugg  ▁effective ▁gas  公务员  ▁college 评论(  ▁individuals0 itch8 arily@ 人士H 手段P 拒绝X  ▁bill` innh >>p  ▁awarex  ▁died 名字  ▁launch  ▁mention undred  ▁sports 小的 ashion 时光 宣布ȥ 面临Х Valإ  ▁League ▁lif 杭州 发表 Read  ▁artist 天然 ▁PM  ▁construction 出去( 论坛0 iverse8  平方米@ 会上H  ressionP  ▁optionX  ▁disease` ▁ANh 健身p owersx  ▁characters  ▁helped 邀请 清洁 incip 运用  ▁places 成就 这位  朋友圈Ȧ  ▁HistoryЦ  ▁limitedئ  ▁feed ults 很有 Acc  ▁liter  ologies han porate  ▁convers 后台( ▁Bre0 硕士8 赚钱@ 循环H 提交P ·　X  ▁Class` 营销h  ▁beyondp 中小x urrent 曝光  的人生 进步 ▁Str  ▁Mary  ▁effort  ▁civil orage 实在ȧ  ▁registЧ 关心ا 司法  ▁fully 锻炼 刚刚  ▁particularly 尊重 上班  igation ▁Mus  ▁yourself(  ▁inform0 reme8  ▁materials@ 指南H ▁salP  ▁rollX 县委`  ▁dailyh Prep fox  同学们 icon uan ecial  ▁annual  的一个 ulture  ▁piece  ▁behav  ▁consultȨ  ▁launШ MAب 医保 NA  ▁Government ourney  ▁green 合法 趋势  ▁force edy 包装(  ▁Child0  ▁reflect8 Type@ anderH DeP 纷纷X  ▁transport` ▁ulth  ▁appointp  ▁usesx  ▁officials 广泛 不限 bt porary 隐患 书法 养殖 仍然 子女ȩ  ▁conceptЩ ▁misة  ▁degree  ▁birth  ▁Journal  ▁analysis 装修 ▁ap dd 告知  ▁consist  ▁Intern(  ▁Fore0  ▁ones8 ▁ri@ ▁sweH 在于P 热线X  ▁computer` 处置h elligp  ▁Clubx  ▁Star eat 缴费 ▁Lou lor 利于 odes  ▁society  ▁answer  ▁heardȪ SSЪ  ▁recommendت 住宅 ▁Def 旅行  ▁Most  ▁Play 适应 等待 纪念  ▁electric  ▁stream(  ▁commercial0  的影响8  ▁stra@  ▁CongressH 血液P 不良X  ▁dream`  ▁movingh 拍摄p erox yth 传承  的基础  ▁attract  ▁economy rief ste  ▁Chinese ca  ▁planningȫ 讲解Ы 把握ث  ▁moved rive 单元 异常  ▁subst  ▁beginning icate  日上午 扫描 指标( 先后0 上方8 一边@ 传递H 当天P 仪式X  ▁marketing` 客服h  ▁missionp 业主x rim 打击 蔬菜 生素  ▁vill 水果  ▁increased 不想 ta  ▁footballȬ 创建Ь iblyج 资本 移动 ▁har ▁Sam  ▁Development ▁Rel  ▁Florida  ▁Camp ▁fa annel( ▁Rob0 ros8 hens@  ▁stringH  ▁allowedP 瞬间X 药品` 第五h 职责p  ▁shotx Not rey ▁Nov  ▁mobile 适当 成熟 comp  ▁types 游客  版权归原作者ȭ 名的Э 初心ح 最佳 arden ▁II 缓解  ▁changed 不够 encies ▁CEO acity ▁ter( 名称0 itable8 超市@ 性格H 机械P hemX lear` 更有h ▁Milp 整改x 吸收 细节 ▁Cam 自信 展现  ▁Minister 承诺 深刻 but rieȮ 在此Ю 东北خ ><  可以说 ened  ▁wild  ▁gold 收益 irgin 周年  ▁items  ▁experienc( azine0 isode8  ▁challenges@  ▁categH 截止P 复习X  ▁Project`  ▁sharedh  ▁servep 临时x 面向  ▁clin  ▁goals ( !▁▁▁▁▁▁▁▁▁▁▁  让孩子 城乡  ▁remember  ▁continued 做了 餐饮ȯ 走进Я ▁Bigد 情感 fig 典型  ▁District  ▁numbers  ▁mount  ▁fresh isco  ▁amazing  ▁easily(  意味着0 通常8  ▁Rock@ ▁AMH  ▁slowP SEX ▁Sur` 也就h FFp ~~x azon 主管 ▁Sal 祝福 ▁Bay  ▁active  ▁desp 汇总 乡镇 ▁idȰ 面前а  ▁studiesذ umber  ▁election ▁er 作出 门诊  ▁organizations  有没有  ▁George 睡眠  ▁▁▁▁▁( 互动0  ▁wood8  ▁Times@ 全程H  ▁physicalP 怎样X no` iledh Lep printx  ▁sustain  房地产 及其  ▁speak 高效 igned icago vant 赶紧 izesȱ 上面б  ▁wantsر ▁Att 意外 ▁Top GB 的前  ▁Smith ken ying  ▁turned ▁Mex( 一路0 专门8  ▁grand@ ▁RetH  ▁favoriteP 人工X  ▁Christmas` 做的h 马上p 入口x 过了  ▁meaning  ▁Committee 寻找 gu  人民法院  issions 线下  ▁Open  ▁WomenȲ  ▁imagв 自觉ز 做出 海南 房产  ▁Carol  outhern  ▁biggest soft ords  ▁tools 鸡蛋(  ▁understanding0  ▁images8 回到@  ▁emotH ▁SyP 各个X rupt` ▁droh 将在p ▁USAx 外出  ▁finally 正是  ▁initi  ▁ahead 彻底  ▁established  ▁rules ▁Jew ▁…ȳ utyг  ▁carryس  ▁remains 布局 Qu  ▁Society term 发动  ▁Good  ▁Care 相互 life(  脱贫攻坚0  ▁Alex8  感染者@  ▁darkH atreP inallyX  ▁equipment`  ▁budgeth ▁Quep  ▁Professx  的通知  ▁legisl  ▁device mar  的情况下  ▁paid 上一  副书记 ecause  ervationȴ  ▁Peopleд 警察ش  ▁join  ▁foreign 车型 anta  人民医院  中小学  ▁Management  ▁correct 场景  ▁blood(  ▁challenge0 iers8  ▁effects@ 出版H 到的P 标志X 师生` onenth 填写p 成年x SC 管控 点赞  ▁tried  ▁cool ronic RA  ▁funding  ▁assess  ▁Mediaȵ TAе 定期ص 几天  ▁background resp 新鲜 Data  ▁sometimes edom 履行 同比  ▁owners( 元素0 ▁Los8 ▁tim@  ▁policH  ▁sendP  ▁sectorX 森林` IAh 军人p 黄金x  人民币 Don eared  ▁Plan  ▁interested 中考  ▁completely 抗疫 华为  ▁Everyȶ  ▁occurж mailض  ▁opening 养生 满意  指挥部 带领  ▁altern  ▁sale 否则  真正的  内蒙古( ait0  ▁partner8  ▁Bill@ etingH  ▁pullP  ▁FranceX  ▁floor`  andemich ouslyp 一致x 氛围 呈现 anned 病人 共享  ▁applications PC  ▁names 居家 紧张ȷ  ▁Riverз Intط ▁» ▁sun Mod 想象 SU  ▁artists  更好的  ▁spread 出院 原文( ▁Mat0 ji8 ria@ 心情H oundsP  ositionX  看起来` 苹果h 太多p 说话x 流动 top edia  ▁operations  ▁benefit  ampionship  ▁Under ▁fix ], ▁Moȸ  ▁vehicleи 局长ظ  ▁highest  的方法  ▁knew xx ▁Sl 大利 层次  ▁Miss igr  isation(  ▁insurance0  ▁citiz8 ▁UN@ 出发H  ▁continuesP 物品X 不在` 功效h ▁lovp 一是x  ▁cert iny 电脑 信用 付出  ▁African ien  ▁jobs  ▁dram ▁Subȹ arantй priع 回答 prof 振兴 足够 高铁  ▁smart 对此  ▁stock 罗斯 ▁max( 通告0 会在8  ▁critical@ FLH  ▁overallP FrX Over` wellh pointp  等方面x  ▁Policy lands 支撑 出门 look Te  ▁voice 纪委 erry ifyingȺ 航空к agramغ  ▁conv  嫌疑人 之中 首次 关节 apter 秩序 overy atab ▁pm( 独特0  ▁door8  ▁Other@ 一张H 战斗P  ▁HillX 解读`  ▁cancerh 造型p 十四x  ▁senior  ▁sites Do  ▁values  ▁surround  ▁hundred  ▁movement 热情  在这个  ▁fineȻ omaл 创作ػ state 似乎 允许 前的  ▁difference  ▁peace itely Go 无数  ▁produced(  ▁status0  ▁condition8 陪伴@ ▁WebH adyP 治愈X 直到` 流量h 试验p Mex  ▁mill ▁IS 也在  ▁Christian 用地  ▁followed ius  ▁audience 图书  ▁leadershipȼ  ▁sceneм 星期ؼ unte  ▁miles  ▁Community  ▁helps  ▁songs ▁Pet 气质  的原因  副主任 好看(  ▁obtain0  ▁residents8  大学生@ 观众H 物流P  ▁thinkingX 此时` 解放h 热点p  的人员x yan ▁lay IR  ▁apply 明天 也很 固定  vention  ▁passed 当中Ƚ ▁Madн  ▁Timeؽ LC Su  ▁spot  ▁existing 丈夫  ▁Scott  ▁Arts 认证 观看 ▁Hot( ▁du0 表面8 的那@ roudH  ▁precP  ▁professionX  ▁church`  levisionh gencyp 消化x 　, ▁die iately 诊断 ▁Bra  ▁activity 涉嫌 rd 省级  的孩子Ⱦ 营造о 央视ؾ uing 事务  ▁lands  ▁responsible 科目 ano 金额  standing iec  中国的(  不一样0 ▁cru8  ▁doll@ 防范H ulaP ▁SunX 环保` 吃饭h iarp 例如x iling  orthern 社保  ▁shall MS 就算  ▁Angeles  ▁middle 大概  ▁frequȿ 手续п ▁Hamؿ 聘用 痛苦  ▁Design 违反 尽快 信心 同步  ▁completed 是有 防治(  ▁purpose0 超级8 规律@  人力资源H 大众P  ▁strategyX  ▁vote`  ▁brotherh  ▁offeredp  ▁architectx 美术 护理  ▁extrem ▁ge  ▁offering works 辖区 写作 ▁Aut 看来 semb ini 你会 含有 脂肪  uration 路段 给我  ▁purchase  ▁double 引领 ▁bur( from0 iment8 同意@  ▁novelH 回来P ademicX  ▁Kore` agonh  带来的p  广东省x  ▁dang  ▁Social 纪律  ▁attend df 相同 同事  ▁cris ▁aim  日下午 ▁Hel  ▁Then 园区  丰富的 尽管 IF ped 才会  ▁Free 出售 东方  ▁load(  身体健康0  ▁reduce8 ▁sch@  ▁transferH  ▁weekendP 构建X 天天`  ▁Appleh 想法p ▁Cupx 甘肃  ▁virt orney 校长  ▁prices 所得  ▁refer 批准 ▁Sum  无症状 efore 教材  ▁entertain 刑事  ▁anti  ▁powerful  ▁photo  ▁advert 配置 样子 指数  ▁feet(  ▁compared0 Why8  ▁Central@ ▁semH 社交P pectsX 二是`  ▁granth  ▁Leadp 无人x 遵守 二手  ▁helping  ▁Union 这两 reate ▁Rem  ▁throw ▁reb ▁mer 其它 ер  ▁rates  ▁wall 平衡 ette Next  ▁conference 为你  ▁false  ▁note  ▁Micro( rang0 心中8  ▁basis@ 期的H  ▁trendP  ▁demonstrX ▁Sol` OMh 放松p  ▁joinedx  ▁trip  ▁Russia 而不 中间 BS 精选  ▁requirements го 信号 UL  ▁arrest 财产 订阅 购物 地位 等级 蓝色 PD inner  ▁Mont ynam zer(  ▁researc0  ▁journey8 有个@ 可见H  ▁talentP  维生素X  ▁charge` 山区h itudep  ▁happenedx "); Up  ▁lines adian  ▁Even 医护  ▁picture 贵的 发出 跟着  ▁Human ivery usband  ▁weight  ▁spirit  ▁paint aud imal  ▁advant  ▁train 总体 注重( ▁Tor0 ▁Pri8  ▁Econom@  ▁ShowH  ▁WeekP 辈子X ctors`  ▁althoughh  ▁exactlyp  ▁exceptx  ▁Series 道德 肌肉  ▁establish retary ruit ▁Jo 填报 转发  很容易 ulf 周岁  ▁sort Add lo 是最 type  ▁myself  ▁hour 低于  ▁Global ▁ST( 岁月0  新冠肺炎疫情8  ▁primary@  ▁VirginH  ▁ChicagoP 含量X ▁gun` 为准h alap 胜利x 董事 wise 落地  ▁structure 被告  版权归原作者所有  有很多 读者 ▁enh 快递  ▁fear 辽宁 Ed  ▁central 构成  新型冠状病毒感染 缴纳 解析 日期  ▁poll  ▁competition test( ▁Ins0  ▁immediately8 论文@ 洗手H  服务中心P 留下X 回应` ressedh pathp andax 分子  ▁pressure 乘坐 疾控 商家  ▁certainly uled 身材 includ ervice  ▁associated 照顾  ▁comments ▁pen ▁nom antly  ▁rent  ▁void  ▁alone  成为了 equ  公安机关( 至今0  ▁famous8  可能是@ 从来H  ▁EngineP  ▁greaterX Phone` 矛盾h  ▁Fundp 大小x 接到 road ▁www 早安 ($ 门口 witch  ▁posted  ▁scen 命运  ▁Technology ▁vs ▁que 职务 不出 inate ▁beg  ▁station 物理 搜索  都没有 asy( 疑似0 Now8  ▁killed@  ▁partnershipH ademyP nergyX  ▁supply` 依旧h xyp 小心x 深化  ▁records  ▁Robert  ▁opened  密切接触  俄罗斯 放入 aming  北京市  来看看 考点 账户 装备  ▁despite 聚焦 ▁Tur etic  ▁homes  ▁nice 演员 About  ▁retail(  ▁error0  ▁developing8 竟然@ 等奖H 停止P icksX 担任` orterh bolp 我就x  ▁manager estic 上升  ▁photos  clusive 仅供  ▁calls 群体 起了 医师  igration ▁Bi 当年  ▁models  ▁Brown  ▁Elect 我想 ▁cry 党组 到达 ▁rob 部位( 逐步0 isf8 eline@ 魅力H ▁graP 学术X  也不会`  ▁measureh anticp 决策x 疼痛  ▁previously  ▁Grand  ▁agreement  无论是 going 发送 igan  ▁claims  ▁daughter  ▁films 行驶  ▁produce van 成交 退役  ▁pandemic bert 室内 完美 道理  ▁temper(  ▁Family0 打印8  ▁Well@  pectiveH  ▁devicesP 贸易X orial` 离婚h Obp  ▁clientx  ▁coach  ▁thousands 清晰 纳入 登录  ▁Data  ▁appropri  ▁Germany  ▁context 背后 Dr 大幅 ocol 地理 定位 机器  ▁shown 类似 签订 itors LO 量的(  ▁push0 珍惜8  ▁Information@  ▁warmH ▁SkP  ▁letterX  ▁administration`  ▁incomeh ▁Letp  ▁Officx ▁il  ▁Frank  ▁propos 条例 党建 vere SP  ▁setting  问题的  ▁spend CA  的东西  实际上 ▁pet ▁Div  ▁launched 显得 sm  ▁reasons  ▁chem 设立 CE( rad0  ▁independent8 ~　@  ▁IranH TheseP  人民群众X  irection` imbh 过来p  ▁protectionx ENT 大赛 合适 gar  第一个 itchen  ▁incred  ▁views bour  ▁excellent pat 夏天 印象 ▁mur 季度 Can  ▁wait 知名  ▁passion ▁Rad  ▁straight  ▁opin( MP0  ▁pages8  ▁Team@  领导小组H  ▁ConstP ▁EUX ena` rapyh  ▁Centrep  ocationx SA 电池 编制 姐姐 祖国 节日 补助 ▁Av  安全生产  的感觉 OD  ▁Internet  怎么办 ▁Sil 保健  ▁Rights  学生的  现在的 空调 地块  出现了 roy(  ▁cities0 ▁Tre8 携带@  ▁lovedH urlP 性和X  ▁reality` ▁Deth  ▁fundsp  ▁presentedx  ▁entry  ▁executive  ▁creative ▁• 给了  ▁advice 常态  ▁rank itness 当你 ework  ▁estim writ  ▁struct 封闭 附件 ▁Jul  erences  ▁violence  ▁Form Americ 热爱( 龙江0  ▁environmental8 known@  ▁partiesH  情况下P 一批X 利润` 报警h  ▁writerp 团结x  ▁Festival ipl ▁и ita onents  ▁cars 肿瘤 休闲  ▁Step  ▁master da 危害 突发 lt ▁eth  ▁Please ▁Vis can 众多  高速公路 ipe  教育局( iance0 ▁Ir8 ato@ 号线H  ▁SecurityP  ▁culturalX 人都` 纳税h gnp 采访x 不动 到位 ▁OF ieces  风险地区 ▁fan  ▁truly Here 山市  来源于网络  和社会 friend  ▁chief  ▁raised 爆发 高等 姓名 oms 一级  ▁Russian  ▁Americans 组合( 每月0  ▁About8 ▁Wil@ 表演H  ▁apprecP 老公X 对比`  ▁feelingh Amp  ativelyx  ▁signed 实习 ias ▁van 台湾 通风  ▁Chief 转移 强烈 7 0网络配图网络配图网络配图网络配图 ba  ▁normal pack  ▁afford 错过  ▁connection 新华 电商 反而 大力  ▁score 路口( 看见0 ▁Bet8  ▁machine@ 开通H 欣赏P  近年来X  ▁poor`  ▁Privh  一个月p 彼此x 概念 站在 气象 上线  ▁experiences 审批 世间 td px  ▁Love ▁lab  ▁Mass  ▁dedicated  ▁guid  不得不 积累 CL  ▁skin  的一种  ▁capacity public  很好的( 推送0 花园8 至于@ 指定H  知识点P atersX allery` eduh 解除p  管理局x opping 民生 入学  ▁standards  ▁Down 打卡 Rep ella 周围 iminal  ▁guide 头发 也没 性能  ▁driving  ▁regarding 你是  ▁Power  ▁majority  ▁wish 和谐  ▁Mount(  别人的0  的要求8  ▁compl@  ▁adoptH ▁AtlP azzX  According`  ▁Listh  ▁basicp  ▁Lakex  ▁Enter  ▁patient  ▁Wood  ▁Town  ▁Party 官网 缺乏 部长  ▁decades 了吗  ▁update ideos  ▁eyes 好友  ▁psych 户籍  ▁Vict dule eful  联系方式 sk ario(  ▁proud0  违法行为8 ▁Iss@ 吉林H ternalP 现有X  ▁runs` 求职h  ▁rarep 行的x 尝试 NO 房间 一片 展开 aling 两年 工人 ▁Mer  ▁files 立刻 active 外的 eless то 产权 ▁о  的主要 学期 精品  ▁Head 聊天( ▁tit0 爱心8 大师@ uddH  ▁InvestP 基因X  ▁announce`  ▁microh  ▁Louisp  ▁Manyx 不多 ▁eye  ▁agency inem cul bar 伟大 thlet asket 失败  ▁husband  ▁vision 原价  ▁species  ▁facilities  ▁policies anging 城镇  abilities 格局 迎来 素材(  ▁definitely0 博物8 特征@ ▁escH  ▁mechanP 反馈X 监察`  条件的h 坚守p  ▁arenx ooth  atabase  ▁stuff 路线 电动  ▁disapp  ▁experienced 男女 联盟  ▁testing  actions 失去  ▁cash  ▁princip  ▁talking  世界上  ▁trial orrow lymp 修复 证据 ▁map(  ▁crisis0 德国8 年代@  山东省H  ▁TourP  ▁goneX  的故事` 薪资h 加速p 队员x 校区 ▁Mel ban  ▁focused  ▁pattern  uilding  ▁district  ▁Peter 爱你 的水  公开招聘  ▁Thomas 上课 ▁emb  ▁seeing 便利  ▁larger  ▁looked  ▁Water ▁ang  ▁figure 返校(  无症状感染者0  ▁proced8  ▁elements@ aughtH  ▁absolP  ▁initialX 权利` 军事h uriesp kgx  公司的  ▁exerc 简历 授权  ******** iler  党支部 常常 不开  ▁Earth ▁Why Car ▁sy 韩国  opyright  范围内  ▁Valley  ▁couldn 作文 cell % ▁▁▁▁▁▁▁▁▁▁  ▁becoming( 对外0  的学生8  ▁Johnson@ ”(H  ▁ReportP 葡萄X 辛苦` phoneh  ▁harmp  ▁Sincex  ▁essential apers  ▁generation  ▁owner ▁Gen 维持 人事 burg ▁Oh 漂亮 难度 抖音 osh riage 高端  ▁pros  ▁Asia  长时间  ▁experts 舒服  ▁Western aste(  ▁fant0  ▁gain8 思路@ 玻璃H 聪明P 保留X 工厂`  ▁translh 不见p 好像x  ▁perhaps DP  ▁null level 电梯 Mar ▁Er 认可 咳嗽  ▁hotel 之下  的机会  ▁unit craft  大家都  ▁editor  ▁damage  ▁north ML  ▁solid string 年底( 清理0  ▁blue8 mary@  ▁internetH ogueP  ▁showedX  ▁length` cherh ▁catp ▁Burx ▁End  有时候  给孩子 办事 十三 不敢 复产  ▁Medical  ▁sexual 他说 zen 地的  ▁crow  开发区  ▁eval 男性  ▁sources night  卫生健康  ▁described 共有 人心( 为何0  电视台8 可爱@ CompH  贯彻落实P  ▁appearedX 真是` ingerh 元的p 视觉x  ▁Long 部队 Some  ▁repe pid 是指 扎实  医学观察  emporary  entially 玉米  ▁analy  elligence  ▁committed  我们要 品种 des  ▁thanks 角色  ▁href 此前  ▁department(  ▁Also0 ▁See8  ▁Main@ thatH  ▁GermanP  ▁lawsX 网红` ▁Timh ppingp  ▁forcesx atin rc 车的 连接 }, 心脏 满满  ▁radio  ▁Awards  ▁Game 表情 修改  ▁winning 开工 便是 ▁Fre  ▁toward  ▁catch 感恩 车站  人员的 暂停( unning0 进展8 院长@  ▁reachedH  专业的P ▁NYX user`  ▁sporth  ▁stepsp 区别x ns  ▁south hab ▁Ill  ▁Jose  ▁brain  ▁markets 有趣 浪费 长沙 ”) 周末 ▁Jud reek  ▁glass  ▁concerns 不停 引进  段时间 爱好 三大 ен( 试点0 ffee8 奉献@ handH  ▁weatherP 基于X  ▁closed`  ▁Amazonh  ▁occasp  ▁accurx  ▁Martin  ▁hair  ▁citizens  ▁Paris 是对  ▁Special ▁Mur 新生  ▁rise  ▁caused  ▁Their lict 欧洲 atever  oration  市公安局  ▁mental  ▁Richard ▁Nor  ▁healthy  接下来 现金(  ▁Stat0 ▁Mot8 黑色@ ▁DenH  ▁neighborP RecX 两种`  ▁learnedh 破坏p 口腔x  ublished  ▁youth  请联系删除  ▁fashion  ▁finished 常委  ▁manufacture 家中 不上  工作中 的美 使得 cean  ▁kept HS isher ▁Mov omet %; 特朗  ▁driver 收集( value0 响应8  ▁storage@  ▁limitH 逻辑P 仔细X 一件` 板块h каp  ▁browx  ▁practices  ▁impl 预警 aren essage ls  ▁volunte  医疗机构  ▁Food  ▁Sports  ▁Windows  我们将 策略 enses reland  ▁prison  的行为  ▁trou  ▁faith ucky 回忆  年轻人(  ▁transform0 Form8  ▁approx@  ▁drinkH  ▁addingP  ▁AustralianX 兄弟`  ▁exploreh  ▁windowp  ▁technicalx 餐厅 ▁anc  生态环境 ▁Aff 步骤 西北 表明  ▁older  在中国  小伙伴 QQ pected 听到 ulty  高质量发展  的内容  ▁maybe obe  ▁Mexico  ▁Today DC 实用(  ▁extremely0  ▁strength8 ▁IT@  的学习H  ▁chairP  ▁meantX 究竟` 分布h 第六p ▁racx  ▁Arab  ▁delivery  ▁session  ▁Academy 三角 季节 ▁Spe  ▁Police 玫瑰 调节 均为 while 转型  ▁restrict 活力  ▁measures  特朗普 全身 权威 ola  没想到  ▁Oper(  ▁counter0  ▁television8 news@  ▁playsH  疫苗接种P  ▁noticeX  ▁female` 政协h 抗击p  ▁agreex ▁AP  为大家  学习教育 护士 ▁Jim ups 地下  ▁selected  黑龙江 财经 外国 月底 周期 ▁*/ ва 对应 iat ’,  ▁Indust  ▁becomes 进去  不需要(  ▁lock0  ▁officer8  ▁obvious@  ▁BackH 军队P 温柔X  atically` 对待h 政法p alledx  ▁Network 走向 rid 增值  ▁Energy asure  ▁surface Pr mes 辅助  ▁rich 法治  ▁numerous 卫健 用来  工程师 版本  老年人 ▁` AG 睡觉 ▁ban( pan0  ▁appears8 顾客@ 消失H attersP  ▁bringsX  ▁Canadian` hosth yerp 户型x 本来 西南 恋爱 郑州 osure xi uclear 咖啡  ▁infrast  ▁operating  ▁forms  河南省 span 医药  ▁spect 家乡  ▁Film irmed  ▁fram  而不是  有多少  ▁requires( illa0  ▁actions8 飞机@ 让他H inctP edsX 荣誉` 发票h abinp 守护x  ▁Lear  ▁thank %) 忘记  ▁Royal 请您  ▁traffic 拓展 och 陆续 或是 const ▁org press itage ili 你要  ▁heavy HO rick  ▁researchers ▁Jes( 一一0 XX8  还没有@  HoweverH ▁SafP  ▁LandX  ▁Virginia` ['h 康复p idenx ipping ▁thr 代谢  ▁Georg John prene 带动  ▁returned 负担 ▁cas kins  ▁easier 强大  ▁sched 头像  ▁Fire 出台  ▁estate nel  一次性 oses onav( 必然0 合肥8 rics@ 局部H  ▁classicP  inationsX 春天` ▁Whoh  ▁Fortp 口感x 海洋  ▁teaching  ▁street uit  ▁task ses  ▁finding  的能力  ▁facility  ▁Olymp  ▁Obama 每周 plate  ▁controll 全县 预测  ▁actual achel 财运 重症 fol ipped( 阳性0 ▁cop8  让自己@  ▁WallH  ▁battleP  ▁decisionsX 套餐` liesh okenp 监控x iques  ▁instit  ▁movies  unately  可以在 消除  ▁beat  reprene  长按识别  要注意 抓住  ▁ended  实验室 oria 一人  ▁sleep 说了  ▁shop  ▁girls bo abled  ▁memory(  ▁advance0 策划8 片区@ 看了H лиP ostonX 一周`  clusionh 长江p  ▁Libraryx 车主 ▁Joe  ▁Last 侵犯 ▁Two 心态 携手  ▁presence  ▁vehicles ▁amb ▁% ani 许可  小程序 印发 一代  ▁border icians ▁Bo  ▁contains  ▁noted 用人( 气的0  ▁investigation8 红包@  ▁CathH 氧化P 协助X 焦虑` 和平h 安置p  ▁appealx 就诊 简直 Last  ▁Justice  乡村振兴 点个 保存  ▁Carolina 野生  ▁core ▁gar 精心  研究院  下方二维码  ▁satisf 举措 lywood  ▁monitor  ▁methods ▁PC 较大 perty( ▁Lee0 注明8  ▁static@ 罚款H  ▁arrayP 市长X  ▁links` 征收h  ▁securep 我在x  ▁camera 抓好  ▁rule 重磅  ▁excited 之路 对你 色彩 file 高达  密切接触者 othing 都市  ▁plus  ▁university Posted 房价 分局  ▁graph 简称 Co 的主( 海外0  ▁direction8  ▁resident@  ▁minorH  ▁SandP  ▁surveyX  ▁Organ` Classh 便宜p  佩戴口罩x  ▁debut ]; 很少 体内  总经理  ▁multi 说法 Inst  ▁fourth  融媒体  ▁gift  ▁brief ython 后续 bur  ▁Chris  ▁changing 上去 备案  ▁advoc gov 灵活( 免责0  ▁managed8 Time@ acksH  ▁advantageP 零售X  ▁truth`  ▁instrh 济南p 融资x 差异 度的  ▁Ireland 厦门 昨天 andom  ▁factors ▁TO  的就是 ▁Som 账号 感动  博物馆  ▁nine  ▁wonderful  ▁scale  ▁Islam  经济社会 ▁Way  以上的 osis 那种(  ▁sharing0 东南8  ▁Hospital@  这也是H  ▁centreP 地球X  ▁placed`  ▁Communh  ▁Accordingp ratex  ▁Test  ▁mist 到来 发病  ▁fuel 别的 ▁Rev sembly  ▁supported 一句 后果 普遍  境外输入  inating  复工复产 下册 espite 号码 糖尿  发布的 index  ▁visitors( ";0  ▁musical8 并在@ 交叉H 职场P  ▁terrorX 专注`  ▁susph 出席p ▁Dix  主要是 eder Er 人家  企业的  onavirus 业绩 画面 inder 最美 原本 C <▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁appropriate agged  ▁finish  ▁During 本书  ▁eventually 释放 Sub  ▁leaving  了很多( 白色0  ▁infrastructure8 under@ 并非H  ▁relevantP 景观X  ▁showing` 桃花h 相处p  的肺炎x Of  ▁drop 还没 方位  ▁failed idden Des 转化  ▁visual  很大的  ▁guarant  青少年 uke code  ▁injury osen  ▁articles 前行  ▁academic ▁nav ▁Uk  ▁animals( 线索0 aven8 考证@ 养成H 补偿P alianX  ▁operation`  ▁knowsh  ▁commitmentp  推荐阅读x  ▁defense  朋友们 qq  ▁String  我们会 ▁Old ▁ir  市政府  ▁documents ui gers 当事 anche  ▁Color  给自己 烦恼  班主任 OC  ▁pieces acific 肌肤 玩家(  ▁Senate0 EP8  ▁manage@  ulturalH 十九P 慢性X …　`  ▁batterh  ▁proposedp 遇见x  提供了 ▁Its ora Top Che ading 惊喜 Part  ▁adapt ▁sea  ▁diagn ▁fol 为人  ▁Professor ▁Boy olit ▁ing 专科 金属 TC 害怕 善良(  ▁reli0 灵魂8 附属@ 责编H  ▁adventP  党中央X  ▁episode` howh  ▁elementp 途径x 维修  ▁Prot  工作者  ▁config  ▁Blue  ▁covered  ▁introduced  是非常 哈尔 家庄  liament  ▁landsc 用品 ears  ▁religious 这场  ▁influence pload  ▁ship 测量  规定的 著名( ga0 人社8 sub@ 发言H  据了解P ioneX ribute`  中国人民h  ▁Gamesp  ▁revealedx  有效的 诉讼  ▁Author  ▁predict ograp 积分 残疾  疫情期间  ▁housing 五年  ▁spending ▁Cy ws  ▁crime ▁д 购房 障碍  ▁protected 远离  ▁coast ▁spl urban(  ▁baby0  ▁bottom8 ón@ 对接H  的设计P 情形X 二级` ');h yep 无限x 清华 rehens ▁unc ▁wed 哪个  ▁welcome 同一 的中 有机 ль  计算机 一流 家属  ▁Real  ▁spring 向上  ▁instance ishes  ▁stress 讲述 重复 inding( ▁Whe0  ▁prefer8 从小@  电动车H  ▁AdministP 导师X 减肥` 接近h  ▁flowp 询问x  ▁advanced 无需 分泌 lines  方面的 ▁)  ▁issued  ▁Mike  ▁prepar cho  ▁justice  ▁slight usion  结束后 模拟 他在  ▁Young iency eler  ▁identify ▁Gal  ▁Penn( 新城0 我要8  ▁evening@ ▁HawH  ▁buildingsP ▁HenX  ▁pictures` таh  ▁excitingp  ▁heatx  ▁Middle 细菌 "" 的老  大数据 owl  ▁Beach 心灵 法国  ▁generally 纤维 粮食  ▁corporate 通讯 中有 ри aks 买房 起到  ▁Jones 春季 ▁Any(  ▁coverage0 充足8 west@ 信任H  ▁propertiesP 争取X 摩托` ▁Menh 记住p  ▁poolx ▁rat NE  ▁onto 过于  ▁happens FC doc First  ▁wouldn po 分配  ▁technologies 定制 iture  ▁weap TH FA  ▁separate  ▁View 从业 适用 Par( ▁nut0 食材8 男孩@ MayH  ▁importanceP  ▁communicationX long`  ▁thereforeh 定点p  ▁Onlinex 对其  ▁earth  ▁haven  现代化  发动机 公平 认知 模型  ▁freedom 百万 ▁Ep 这份 巩固  ▁classes 延长 ▁Ver  ▁starts 人间  iiiiiiii 天地  ▁emergency  ▁recorded  ▁explain ▁err 指示 PM$  ▁watching(  温馨提示,  ▁valid0 冠军4  ▁situ8 凌晨< (　)@  ▁jointD 难题H  acementL 出于P  ▁JeffT  糖尿病X  ▁alle\  ▁supporting`  ulationsd 一大h doorl  我们在p  ▁performedt  ▁Chairx  ▁column|  ▁Williams param inst 显著 巨大 档案  ▁incor ( !新型冠状病毒感染的肺炎 odies ▁hy  ▁selection 白天  ▁Federal  教师资格 itz arks  得到了  poration iner  ▁craft  那就是 关闭 一套 大连 bi 代理 ла News  ▁impress 子里  ▁hearing  ▁married 顾问 性质 短信 个性  ▁Aud  ▁command  ▁Microsoft ▁ign  ▁cold  地上$ 原料( verse, 国民0 run4  ▁lived8 而在<  ▁shut@  ▁flexD 我省H 生意L  有一种P 县城T 成效X ▁Tri\  ▁labor` 都很d 入境h ▁ecl kerp  简单的t  ▁occupx  不超过| 描述 树立 wen 随意 颜值  ▁adjust 理工 日的  ▁raise 长大 arts 适量  ▁Executive 将于 jj 科普 ▁Dev 市区 Just  ▁Wild 一只  的效果 验证 低头 ▁Lab  ▁univers ersey eech 创意 rant  ▁efficient 宿舍  ▁bath 名额 斗争 ▁_  爆料 iana iced 在外  ▁calling   ▁Live$  ▁plants( 不久,  ▁cells0 Every4 融入8  师范大学< 激发@  ▁determineD 妇女H eriaL 前期P  istanceT  ▁clearlyX 救治\  医护人员`  Previousd  ▁officersh  ▁whoml  ▁herop ▁Limt 签约x 星座| ▁Wal Photo free  ▁util  ▁Photo ▁Sup 前进  ▁Cast 总会  ▁depend  ▁stars 工商  ▁videos  ▁sets 民间 prise km  ▁sequ ▁dig 清洗 ansion  ▁critic 激励 Reg ▁Squ  function  相关部门 Ref 互相  常委会  ▁historical Value ju 主播  ▁Although  事实上   ▁forced anish 充电 政务  感受到   ▁restaurant$ 平常( 耐心,  ▁shares0 wan4 ancer8 医务<  ▁Like@  ▁removeD 笔记H  ▁EachL  ▁usefulP FileT  ▁mostlyX  ▁keeping\ 共计` ▁Jond 我校h  的第一l 那就p  ▁Standt  ▁tendx ▁Dou|  交通安全  ▁Japanese 姑娘  ▁missing  ▁listed  ▁professionals  ▁Children  ▁motor  ▁phil 精致 Pe 转让  ▁veter BI  ▁bringing 南方  ▁balance 暑假 应届 证券  ▁Trust onds 警惕 php  ▁politics 全新 话题 link ▁hab 人人  ▁fellow 小孩 停电  本平台  ▁secret  ▁Medicine   ▁construct ▁Fe  ▁item  ▁inspired  ▁discover  敏感$ 两次(  ▁behavior,  严重的0 部的4 ▁Pan8 总统< Text@ 酒精D ArtH  contentL 查处P ▁SarT 则是X  ▁Polit\  ▁serving` ▁eatd scripth  相关的l  未来的p  ▁icont ulatedx ▁Ro| 杂志 危机  ▁prepared  ▁criminal  ▁boost hent  ▁instruct  ▁cere  什么是  ▁worldwide 监委 赔偿  ▁updated  ▁Conference 癌症  党史学习教育 ▁Die  ▁units 爱国 后期  akistan 损伤 ixt 发行  ▁concert 古代  ▁listen  ▁teacher  喜欢的  你可以 ▁fif  ▁Card 家族  ▁Francisco 萝卜 ▁ice  ']  ▁responsibility  ▁Prov EST 班级   ▁teachers$ 麻烦(  ▁dynam, astern0 法定4 ▁Bob8  ▁regional<  ▁spons@  ploymentD  ▁CharlH 等你L  ▁winterP  ▁lotsT  ▁somewX yles\  ▁metal` namd ▁Refh  党组织l 小白p 最多t  relatedx BO|  ▁investors ulate gment  ▁по 病情 新建 暴雨  ▁arrang icious 将其  ▁shape iano 批评 ▁dry 小说  ▁stated Key 不懂 限时  ▁otherwise aylor 的外 Ass  ▁reviews 不忘 Please  ▁institutions  ▁ment 强制 疯狂 ▁vac  ▁highlight 电力 面上  江苏省  股份有限公司  各大 一口  ▁authorities 全日 ERS   ▁guys$ 民主( 招商, AI0 吃的4 ffect8  ▁studio< 高峰@  免责声明D  ▁smallerH  ▁BostonL  ▁StudiesP  ▁relationshipsT  ▁considX 这类\ 录用` aisd  ▁approximatelyh ligl fep  ▁Andrewt powerx onom|  ▁kitchen 纠纷 前后 ▁pod  ▁allowing  ▁Exper 严禁 发育 利率  ▁format  ▁agreed ▁fra 苏州 真题  ▁Niger  ▁panel 接着 uthors  ▁feels 加班 销量 });  ▁waiting 抑制  ▁Call  ▁Scot ··  ▁server EE rite 厨房 汇报 也可 影视  ▁dance  ▁island  Test rier  ▁input ingly 走了  LA$  ▁connected( State, 不易0  ▁package4  老师的8 不但< ▁fab@  ▁buttonD ansasH  有的人L utelyP  asketballT  ▁decadeX  ▁alternative\ 干货`  ▁consumersd 体会h  ▁Armyl ellerp 高温t 旅客x ▁Hor| 毫无 股票 做事  ▁burn  联系人  ▁furn ivered  ▁solar ▁Use ▁lit ▁Adv  ▁holiday 主办  ▁revenue 幸运 国外  的价格 asks  新能源  检察院 CS  ▁innovative  河北省 配备 zil 对话 竞赛  ▁approved stra  ▁charges Set icial dis  ▁speech TR 好处   ▁Writ  ▁plenty  真的是 留学  中国特色   发生了$ 十一( See,  ▁magn0 real4 EX8  也不是< ▁tro@ реD 帮扶H  ▁interestsL  ▁identityP ▁JunT  正确的X 秘密\ amm`  irationd ellyh 节奏l 实事p 返回t  ▁soundsx 年初| elines  ▁locations  ▁theme ▁© stein Us 进度  ▁на 感冒 开了 遗憾  ▁schedule  一定会  ▁jump  ▁mentioned  招聘岗位  阅读原文  Exception 冬季  ▁vent 舞蹈 已有 which ▁ton  ▁notes  ▁compar 所需 其次  ▁pair  ▁rapid  ▁hundreds  交通运输 ▁ID 回归  ▁coord 美女 装饰  ▁renew  不错的 生存 中药  ▁flight$  健康码( 捐赠, etime0  ▁proceed4  ▁illust8 订单< pret@ postD 本期H 点的L  ▁CharlesP 户口T 诗词X 答题\ 大脑` cksd  全日制h atil  ▁commissionp  ▁treet 牢记x  ▁confirmed| 那里 庆祝 证件  ▁thus  ▁Environment  犯罪嫌疑人  ▁detect irms  ▁dial 受伤 必备 清单  ▁breat  ▁Prem 门店  ▁doctor  ▁surg  ▁bird 拿到  ▁output 风水 翻译  之所以 ▁flu 太原  ▁museum rical  ▁leads bb 大大  forcement 健全 okes ▁guy ownt  ▁promote ▁Ken /**  ▁holds eries ▁CD oles$ ”;(  等方式, 当下0  American4 动员8 户外< gs@ streamD 夏季H  ▁discussionL 窗口P 督导T reesX ▁kid\ 时不` 期限d phash  是我们l 菩萨p  ▁Michigant  ▁departx 看完|  essions igen CP  ▁script 　| 西部  ▁advis 是我  ▁Phot  ▁affected  ▁apart 后面  ▁Columb 江南  ▁handle  ▁Person 住院 题目 日晚  ▁Digital  ▁fish  董事长 ▁Gra ки ▁Id  ▁clinical  ▁reference  ▁innovation 腾讯 Book  ▁kick 第七 三十 ▁ven  ▁featured size  一直在 ador  ▁Secretary 舞台 发的  ▁negot$  ▁industrial( 阴性, HR0  ensions4 这条8 真相< ▁Ma@  ▁distanceD ghanH  被告人L 不及P 优雅T  ▁seekingX 合集\  ▁Italy`  ▁processesd  ▁exchangeh 专属l 实地p datedt reditx  ▁wear| 用了  ▁optim 游泳  中国人 rum  有哪些 亲子  ▁campus 扩散  的女人 区的 ▁AND 神秘 英文 have  ▁massive  常态化 ▁AL 你就 传媒  ▁committee utor  ▁Season  ▁victory ▁Ari  ▁explained  ▁Night  ▁greatest 值班  ▁payment ura 有钱 放假  ▁opinion 批次 致力 名为 考场  ▁edition 家园  县公安局  亿美元$ raine( oted, 减轻0 精力4 ▁bul8 dy<  ▁Donald@  ▁MarketD  ▁ProductH 门槛L 不太P  ▁loseT  ▁witnessX enced\ 和你`  成年人d  大量的h pagel  ▁accidentp  ▁telet  ▁indicx  ▁village| 文艺  tenance Object ja  ▁Jackson ounsel ▁san  ▁protest 这时  出现在 ping ▁Ell  ▁putting DS  ▁Space 机遇 ▁Dom ▁flo  两个人 '; 本土 用心 储备 服装 百分 四大 ▁Van coh ▁Bas outs  ▁library El osoph hr 男生  ▁scientific 佩戴  ▁distribution  ▁Coll 少数  ▁Theatre  的精神$ 有力( zx, 夜间0 几十4  ▁cloud8 视为< 观念@ 带你D ▁pilH fsL athanP InterT 协同X 名师\  长按二维码`  养老金d  ▁Watchh  ▁fieldsl  ▁diversep  ▁testst ////x  自行车| 货币  ▁collaboration 要有 疑问 行程 水的  建筑面积 Out 资质 作风 年前 助于 轨道 按摩  ▁identified  ▁Lord  的关键 三是 整合 ▁bra  ▁Master  ▁Support  ▁Championship  ▁domestic  ▁tells 热门  ▁Force  优秀的  ▁assistance 中秋  ▁agencies Em ▁Max Since 均匀  党员干部   ▁rain  ▁engineering Gu 年纪  ▁Because  讲座$  rehensive(  卫健委,  ▁gather0  ▁circum4  法律法规8  ▁beauty< 滑动@ uableD atusH 更为L  ▁removP 收购T 身心X  融媒体中心\  ▁discovered` 监制d kesh 那样l ilitp  ▁Thankst  ▁fillx 早已|  ▁Instagram 冬天 Jan title 奶奶  ▁index  ▁cookies Mr  ▁coffee 解答  这个时候 uzz ride  ▁earned  ▁winner  ▁reform water 仅仅 但在  市场监管 武器 参观 壁纸  ▁combined 开车 jo  ▁familiar  ▁photograph  ▁Connect  ▁corner  ▁square 乘客  ▁Zeal  的相关 办学 血糖   ▁seemed 一日  ▁ideal she  ▁returns  Gl$  高血压(  的关系, icit0 一看4  ▁bound8 otic<  ▁delivered@ 精华D 专用H  图片来源L passP  ▁grewT 目录X  的一些\ alle` 风雨d  ▁categoryh mercel 总局p 全民t 定了x 弘扬|  ▁Coast 看出  ▁carbon ▁Tem 试卷 委托  ▁alleged  ▁adults 参赛  ▁Walk 初步 浪漫  ▁attacks  ▁willing  ▁shooting  基本上 授课 假日  人生的 较为  itutional ▁Hon  是为了  ▁Once  党委书记 ▁rom 和人 ▁fle 加拿  ▁comfortable  iversary  ▁fell  新媒体  ▁default  的考生  ▁Jewish  客运 着力  ▁Spring  ▁forest Image  某些$ 当日( 货车,  ▁dress0  疾控中心4 地说8 猪肉<  时间为@  ▁resolD TubeH  充分发挥L  微信号P 民办T  ▁wasteX 通信\ 打赢`  ▁registeredd 西方h  ▁Lightl  ▁Girlp 南山t  ▁whateverx ));| 通用  ▁resource  ▁programme  ▁stores 字体 不适 ulner  ▁conven  ▁consumer 家具 给他  ▁Guide  ▁gender  ▁Sciences 牛肉  ▁combination 礼物 CR ”:  ▁defend andon 选手 工会  的基础上 只会 ▁unf  湖北省 Follow 住宿 国有  ▁upcoming 飘飘 你在 一把  ▁internal NS   的专业  正能量 老婆 iy 复试   ▁ongoing$ 抢购(  fortunately,  实现了0 OK4  lication8  ▁educational< MM@  ▁SupremeD 过年H 的手L lightsP 攻击T 故意X profit\ 穿着` Yourd 过度h ▁Licl  ▁leaguep lockt ARTx osc|  石家庄  ▁Zealand  ▁accompl aser ▁Log  ▁asking ▁ur  注意事项 otal  ▁selling gypt 受理  ▁begins 威胁 raq  ▁crowd pet ello 早餐  ▁Disney 高三 rig law 更高 legal  ▁Republican 秘书  ▁Look istant  增值税  ▁Division 践行  ▁Southern ician  ▁Roman ented   ▁animal  当事人  ▁native  ▁democr 培育   ▁doubt$ 歌曲( 受害, 咱们0 火锅4  不仅仅8 即便< 处分@  ▁railD PerH aminL  ▁labelP 支队T mondX 重的\ osit`  ▁Danield 在大h posesl 区委p 话说t 定的x  传染病| ▁Pay ▁Ok  ▁upgr rome 核实  ▁readers adium king  你们的 高兴 仿佛  ▁Check 哪怕  ▁Brook Log  ▁cards onto Source 演出 Let  ▁honor ▁Mid  ▁Prime  ▁decide  等问题 生日 提到 sen  ▁candidates ▁da 等地 Sec  ▁Georgia  ▁awards 透露  美好的  uous  哈尔滨 院士  ▁favour  图书馆  ▁Set$  ▁guitar( 顺序, 就医0  ▁privacy4  新冠疫苗8 ▁Equ< 大约@ ▁armD /㎡H ssssL  ▁wealthP ushedT itiX ether\  的安全` 标注d 十八h IEl 尚未p 原件t  反反复x 任职|  ▁arts Are  ▁tech  合法权益 化工 ▁CA 成的 ingu  ▁candidate  中华民族 Trans CD 一律 evin wx 多人 音频  ▁Britain  ▁quant  ▁dollars  ▁conducted 信念  ▁brands ▁Far  摩托车 火灾  ▁marriage 一对  ▁Story  ossible  ▁historic Act cles  ournament mosp  ▁founded  各自 ocket  的日子 ▁Jr  ▁Foot   ▁applied$ 又有(  ▁Catholic, umer0  有关部门4  ▁guests8 征集< sole@  工作经验D  ▁professorH 北方L  相应的P  ▁updatesT  一方面X  ▁negative\  ▁Korea` 遭遇d 梳理h orall 获奖p Endt  常见的x 占比|  更好地 芯片 平凡  ▁narr  活动的  ▁sand  ▁Make lix tered  ▁Italian  ▁sustainable  ▁Second 租赁  ▁volume  不符合 ▁McC 暂时  上面的 失业  ▁millions  ▁teen parent ▁Gar 双手 作家  的过程中  准考证  咨询电话 督促  ▁draft 是由 ▁aid  ▁Jesus  ▁planned isters  ▁enjoyed  apore 塑料 aa 可在  浙江省  常务$  请及时(  上海市,  ▁Govern0  ▁holding4 在了8  ▁Page<  基础设施@ ▁eraD 全区H ampionL akedP  按规定T dbX  ▁filed\ 二次` 却是d  ▁debth  ▁chargedl  ▁servesp 公斤t 损害x 1 *▁▁▁▁▁▁▁▁▁▁▁▁▁▁| 不论  ▁talks ▁Mic 又是 ▁dim abeth 形态 文旅 每人 ▁IP Will 默默  的信息 orders  ▁cooper 请在  下一步 NN  ▁leaves ▁tow  ▁Pakistan 省份  重要的是 圆满  大家的  ▁snow  反反复复 楼市  ▁seat ih  ▁Through  ▁honest  变成了  ▁legislation ▁pun 从此   有利于  等工作 更大 外卖  ▁carried  大专$  ▁consequ(  ▁accounts, ▁м0 ▁Kar4 ▁RE8  ▁Android<  ▁shift@ 便民D  也就是说H  ▁PacificL 十五P irtyT oweredX  ▁actor\  ▁neighborhood` usterd  ▁athleth  ▁stickl  的历史p  ▁conversationt ▁Ltdx mg| 师傅  ▁bright case 真心  用人单位  ▁minimum 睡前 fast 一辆 美容  ▁entertainment  ▁route  respond  ▁database  我们可以 Eng aduate 得了 凭借 据悉  ▁Engineering 常规  ▁awarded  ▁forget 既然 若有 入选 ali ▁Pop  ▁originally 差距 不下 手工 个体 混合 Def  帮你 厉害  ▁Spanish ▁Cle  ▁aspects  制品$  ▁accommod( ua, 多多0  个小时4  ▁exercise8 是你<  中医药@  ▁sweetD umsH ▁KimL  的风险P  ▁LiberT renceX 成人\ GO`  ▁traditiond 清明h 投诉l  ▁conflictp  ▁nucleart ▁twex  ▁concent| ▁Lo 而已  ▁facing  ▁administr  ▁ride eper  ▁passeng 多元  看到了 我也 avor va 奥运  ▁Steve ▁rid  ▁drugs void  的规定 教程 reens  巨大的 公民  ▁android 冲击 只需  ▁Fair  ▁liber illy  ▁specifically  ▁minute DF  ▁guess  ▁Kingdom 由此  ▁tough  ▁Both   ▁manufacturing 复印  ▁Ohio ▁Ext 凡事  光明$ 称号(  ▁follows, ighter0  ▁emphas4 门票8 气血< 陌生@  什么时候D 神奇H  ▁postsL adedP 上有T  ▁newspX ▁tom\  微信群` 说过d  强大的h лоl  ▁creationp 块钱t 石油x  ▁entreprene|  ▁infect 亲自 ▁fat 虚假 kin  ▁confidence  ▁intended 航班 HT  ▁recip  ▁confident 高于  一段时间 tes noon 还不  ▁scholar Ge bridge  ▁filled idth  ▁employee  ▁transition  ▁enable nic cknow inton  ▁abuse  ▁virtual  ▁thoughts 月初  分数线 ▁pal  ▁concerned  ▁perspective obs  下列 截图  ▁risks  ▁detailed 试用   ▁belong$ inds( 脾气, 张家0  ▁recognized4 大桥8 apping<  ▁chemical@ 坚强D GenerH 假如L 表彰P  ▁dispT  ▁figuresX 鉴定\  有些人`  ▁theoryd 行情h ▁hydl lynp ####t 昆明x 多云|  ▁healthcare  等相关 谨慎  ▁fort oster 标题 的花 ▁lux  ▁expertise 牺牲 isk ATION  　·　 公寓  ▁encourage Rem 一颗  ▁chain 抓获  ▁elim  是中国 招标 要在  ▁owned 不变 ▁Hy ▁Win 追究 依靠 iest front 国庆 ifts  ▁motiv  ▁edge 万吨   ▁county 金钱 Sm  农产品  ▁matters  少女$ 西藏(  ▁Know, 宁波0  严格落实4 ennis8  ▁Video< 承认@  ▁removedD  的企业H  ▁participantsL acleP  ▁treesT thenX  ▁incident\  本公众号`  产业链d seyh 单词l  不管是p  ▁assetst  ▁Passx 在校| 等到 我不  上半年 强度  ▁scheduled  ▁Jersey ле  esterday rooms  ▁dangerous 存款 复制 中午 店铺 ▁($  再加上  ▁rooms udden car ads  ▁arrived 五一 拨打  ▁mort  ▁wine amil Comm  国家级 не  ▁perman rete 陷入  ▁Manager ▁Cov  ▁Democratic 坐在  转变  蛋白质  ▁wedding  ▁roles 十足   学校的$  ▁warn( adel,  的过程0 收取4  ▁employment8  非常的< 都没@ 职称D 境内H  ▁purposesL 怀疑P  志愿服务T  ▁apparentX  ▁signs\ 土壤`  ▁Iraqd  疫情的h Otherl 严肃p  ▁secondst 秋季x  ▁prote| ▁TH 作战  ▁tickets 决心  ▁Sign  ▁switch  ▁shopping 页面  ▁council  ▁appearance 入住  ▁Queen 物的  iveness 厚的  ▁expensive 一款 动画 纪录 全年 全会  ▁height 不宜  ▁Access  ▁worksh inson 你想  ▁decor  ▁Northern  的原则 ▁tun unting 服用 Stud 随便  加拿大  те 例外 elled  ▁fighting  不合格   十四五$  ▁loan( sl, pace0 情节4 进了8 原理<  ▁assum@ HPD yncH  交通事故L  ▁organisP 更换T  ▁contemporaryX  ▁herself\ 花开` ▁Lucd ▁rowh  ▁Crossl  报名时间p  ▁gardent  ▁householdx 均可|  ▁maximum  ▁vulner 不去 ifer  ▁improved 慰问 ▁Po 我说 ▁mut agues 牛奶  ▁profile 拘留 中山 Fl  这件事 isa  ▁divid suit  ▁walking Tags  ▁drivers  ▁victims acts  ▁caught anger  ▁temperature  ▁installed 房源  有关规定 ATE  投稿邮箱  驾驶员  ▁Biden  另一方面  来自于  毛泽  ▁Little  ▁license ▁AR  ▁virus   ▁typically$  中国特色社会主义(  了解到,  ▁courses0  ▁moments4 ▁з8  ▁failure< 本质@  ▁determinedD  ▁magazineH  ▁sisterL 肝脏P ikiT  ▁stormX  ▁severe\ ican`  ▁involved  ▁sympth  所谓的l 未经p 先锋t  ▁scientistsx  ▁recovery|  的基本  ▁advertising  ▁circumst  相当于  ▁discl  ▁Joseph  ▁channel  ▁extensive  ▁mand 不然 ▁NE  ▁techniques ▁adm  Service Dav inois esome  ▁statements 类的 OW 为此 赠送  提出了 Bar ▁Kh  健康的  ▁kill  ▁Creat  我觉得  ▁provider 客观 PE 得知 anges };  都知道   ▁comprehensive apes  ▁graduate shire  高新区  飞行$ 头的( yers, fin0 ▁nor4  致力于8 欢乐< 放下@ 判决D  ▁evaluH 体的L 实时P otesT rencyX  ▁ancient\ pir` 单身d 齐全h  ificationsl oustonp  ▁boughtt  和其他x 好运| Click  ▁positions 身高 Pol 宣讲  ▁Harris ▁ran ▁sav  ▁murder  ▁sides 辣椒  ▁Rest ME  ▁cart MT ▁cra  ▁symbol  ▁Staff erman  ▁featuring 售价  ▁reporting  不容易  ▁male  全世界  越来越多  驾驶证 大街 现状  ▁Hollywood  ▁incredible  意大利  请联系我们 医用 cohol  数据显示  也将 教你 大全 GA  ▁Broad  对手$  ▁Civil(  ▁assert, 单独0 style4 教练8 提取<  国家的@  学习的D 南部H 助理L 为您P 上传T 不对X 高分\  ▁debate` 务必d sumh 厘米l 骄傲p  ▁topict  ▁Brazilx 手指| 不止  应该是  ▁significantly 贵人 ▁Ent foot 凝聚 上学 豆腐 继承 体重  ▁urban ':  ▁functions  ▁surrounding  ▁scored 导演  ▁alongside 凭证 cious  ▁flat  地区的 过敏 东莞  毛泽东  ▁defined  izabeth  ▁Next 给人 生育  对自己  ▁acknow  也不能 ▁sam ordan  ▁strategic  ▁vaccine  的一切 频道 着你 此文  resents$  ▁relax( 肠道, 楼盘0 走访4 VD8  分公司< aped@ 座谈D 上门H openL  ▁equalP layT 估计X Many\ 电信`  ▁Taylord Mosth  ▁steell  篇文章p  ▁Irisht  ▁attorneyx  给我们| 不必  ▁emotional 去世  ▁slightly  ▁Matt  ▁tack  ▁believes g `▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ appy  ▁Cost found  ▁unless 沈阳 asters DT 牙齿  ▁directed ras  另一个  ▁Point  今天的  ▁execut 密码 渐渐 式的 依托 活着 地面  ▁Start  ▁param  ▁absolutely  ▁Privacy 畅通! estyle! Mark! rich !  ▁believed! 他是!  ▁spoke!  ▁intelligence! pty ! 居然$! 水分(! La,!  主管部门0!  ▁supports4! ▁wat8! 寒假<"  ▁topics"  ▁impressive" 平稳"  ▁festival" 头条" 化解" ▁Tal" table" 学费"  ▁remaining" ▁Son" AND" 荣获" Gr" 违纪"  holders" 毫不"  ▁enhance"  ▁underst" 孤独"  ▁Custom"  ▁successfully" trans# 见证#  ▁arrested#  ▁Daily # ▁Ha#  相结合# 微笑# ership# count #  ▁territ$# 失眠(# 羡慕,# 邓州0#  ▁atmosp4#  ▁judge8# 这就<#  一句话@#  ▁authorityD# ▁SeaH# umaL#  联系我们P#  ▁singerT#  ▁housesX#  第二天\# 脾胃`#  ▁flagd#  项目建设h# actoryl# 诚信p#  ▁ministert#  ▁Administrationx# 自在|# 可靠#  ▁receiving# 见到# 体质# During#  ▁audio#  ▁apps#  ▁resolution# 哥哥# bury#  ▁spaces#  ▁Stan#  市场的#  ▁Radio# 大气#  ▁participate# ▁MP#  ▁shock# 聚会#  ▁buying# 民众#  ▁repair#  让他们# 延伸#  ▁goods#  ▁Ford# Rel#  千万不要# arp# ▁Hom# 与其# ▁Sus#  ▁mode$  免疫力$  ▁formed$  ▁calcul $  ▁awareness$  必要的$ 现将$ 乐观$ greg $  ▁shel$$  ▁info($ orough,$  ▁flood0$  ▁standing4$  ▁twice8$ uling<$  ▁providers@$ ▁astD$ TERH$  ▁OfficerL$ iroP$ 下跌T$  ▁competitiveX$ 喝酒\$  ▁fold`$ ▁kmd$  ▁closerh$ 产量l$ 带头p$ воt$ 承受x$ MC|$ 实战$  ▁reject$ 被人$ 过多$ hetic$  各部门$  ▁Action$  ▁maintenance$  全方位$  产品的$  ▁destroy$ unc$  ▁broadcast$  ographic$ 南宁$ arters$ hus$ 学者$  ▁Circ$  ▁covers$ burgh$  ▁none$  被称为$ 帮忙$ 挖掘$ 立场$ ▁af$ item$  ▁planet$ ▁Op$ ucks$  新型冠状病毒肺炎$ rovers%  ▁Have% 高层%  ▁grown % 回报% 爷爷%  ▁explains% 月至% ECT %  退役军人$% 中级(%  福建省,%  ▁prepare0%  ▁Kong4%  ▁Cook8% 等于<% 激素@% 山水D%  詹飘飘H% 评选L%  ruptionP% ▁pedT% 导向X%  ▁YouTube\% aine`%  复印件d%  山西省h% ualityl% ▁Jacp% 小镇t% 关爱x% 姐妹|% 算是% 我县% 美味%  ▁Hard% 将来%  项目的%  ▁Country% 拆迁%  ▁beach%  ▁Colorado% ▁Ap% 一道% 致敬% 之处% Where% 分手%  提供的% child%  ▁ends%  詹飘飘詹飘飘%  ▁bigger% stone%  农业农村%  ▁aircraft% 文末% 年限%  的状态% ▁cro% bro% Any% ylvan% 车道% 反对& 日内& 同行&  疑似病例 & gener& 青海&  ▁wire& stract& ▁NFL & 若干$&  ▁landscape(&  ▁medium,& 领先0& 气候4& Jo8& 快来<&  ▁Full@& 著作D&  ▁iPhoneH& rawnL& 联动P& 清水T& IBX&  ▁stret\& 南昌`&  ▁factord&  ▁Villh&  ▁reducedl&  ▁Concp& 评审t& 阳县x&  怎么样|& 国产& 学子&  ▁Financial&  ▁adult& 亚洲& 车间& 排除& 有意&  高风险地区& bec&  ▁Term& 视野& ESS& 尺寸& IO& lets& 护肤&  ▁coronavirus&  ▁agric& aron&  ▁celebrate&  ▁friendly& 上级&  ▁Blog&  ▁Ministry&  ▁pregn&  ▁parking& 不大&  ▁manner&  如果是& 两天& roller&  ▁Again' 答疑'  ▁attended'  ▁afternoon '  ▁crew' 而来' 起点' ▁AT'  ▁surprise ' 股东$' ▁Ol('  ibilities,'  ▁increasingly0'  ▁frame4'  的形式8'  ▁platforms<'  ▁Follow@'  ▁plasticD'  ▁restaurantsH'  带来了L' 以外P'  ▁happeningT' infoX' Ne\' zona`'  不可能d' 抱怨h' rml' 持有p'  ▁rought' ▁mirx'  ▁considering|'  科技有限公司' Error'  ▁worry' 一块'  ▁speaking' ▁ce' 灾害' 阿里' 十大'  ▁convert'  ▁Spain' 4 -▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁'  ▁entirely' 饮料' Who' 新年' 以往' 浏览'  ▁Only' iary'  疫情影响'  ▁elig'  ▁Kenn'  ▁treated' vy'  ▁soul' othe' 战士' 警告' vin' 透明' 文物' 宇宙( fil(  ▁tips( 轻易 (  点击下方( 黄色(  ▁represents(  ▁agent( 就把 (  ▁Partners$( iors(( Ret,( makers0(  经济发展4(  ▁mail8( 搭建<( inent@( 功夫D( 绿化H( ▁flyL( 可怕P( 不行T( ▁naX(  ogether\(  个人防护`( lianced( iamih(  ▁Booksl(  ▁retirep(  ▁Songt(  今年的x(  好消息|(  ▁Disc( 接待(  owntown(  ▁Take( ▁Rod(  独特的(  有助于(  的同学(  违法犯罪(  ▁Singapore(  Another( 买卖( 镜头( 艰难(  ▁faster( mony( iginal( 生气(  ▁Control(  ▁extended(  ▁streets(  ▁challenging( ▁Mu( ersion(  ▁entered( 开设( 班子( ▁mal(  ▁relatively( 约定(  的产品( 这也( 安心) 之际) 边的)  ▁Phys ) AV)  ▁Mill) DR) 消耗)  ▁Four )  ▁deals$) wa() 调理,) 下一0) 实的4) cons8) 演讲<)  ▁west@) astsD)  ▁submitH)  ▁fascL) 多名P)  ▁ultimatelyT)  fficientX) 面议\) AF`) poold)  工作室h)  ▁vastl) letterp) 列车t)  ▁visitingx)  ▁Ryan|)  ▁Henry)  ▁decre)  ▁affordable)  ▁claimed) 公积) awa) world) 多数)  ▁Field)  ▁accepted) uts) 善于) 漫画)  的可能) 采集)  ▁Asian)  ▁hopes) 预期)  ▁suit)  就开始) 小米) 化妆)  ▁implementation) 四季)  ▁reput)  ▁components) Two) ▁sac) 主人)  ▁regulations)  ▁Labor)  广州市)  ▁compos* 体制*  ▁Hand*  ▁battery * 我家*  ▁Capital* 北部*  ▁initiative*  ▁register * ▁mit$* ko(*  ▁Perform,* ▁у0* 书写4* 商场8* 另行<* onym@* 实体D* flixH*  资格审查L*  ▁forgP* ilaT* alysisX* war\* 模板`*  ▁ourselvesd*  人大常委会h*  ▁networksl* ▁Darp* qlt* 大地x*  include|*  ▁Agency* 多家*  ▁chosen*  公共卫生* 协作* 说到* 干燥* 大陆* 其余* 县级*  ▁remote* away* input* PL* 两位* 保养* 省市* 本周* ▁AI*  ▁recording*  ▁menu*  ▁elected* 救助* 绘画*  ▁banks* 摄入*  ▁effectively*  达到了* 严峻* 支援*  ▁authors* endar* 安市+ 宁夏+  ▁taught+ PPT +  ▁elections+  ▁Turn+ 要把+  ▁smooth+ ти +  ▁Guard$+ ());(+ 前面,+ 教室0+ 指引4+  ▁stake8+  ▁trading<+ worth@+  tainmentD+  ▁closelyH+ 航天L+  食品安全P+ afeT+ 主编X+  ▁prosec\+  ▁evol`+ ▁Twd+  高度重视h+ 定义l+ Attp+ 李某t+ 中年x+  投资者|+  ▁examples+ ycl+ 与人+ 留在+  ▁adds+  ▁acting+ spe+  就已经+  ▁surgery+ ▁Rom+  ▁fundament+ 气息+  下来的+  一个个+  ylvania+ 出血+ 眼前+  ▁confirm+  这是一+ 三种+ ▁Ron+ 天空+ 分的+ 事宜+ ión+ 可谓+  ▁Kent+  最重要的+  ategory+ 复兴+ 诊疗+ erg+  有一天, fa, 说是, 装置 , 以免,  ▁Ukraine, ▁OR,  ▁founder, 均价 , ▁Fac$,  ▁guest(,  各项工作,, ira0,  ▁phase4, uting8,  ▁objects<, cd@, 度假D, ▁SkyH, 立足L,  ▁delayP, 外科T, folioX,  ▁vital\,  ▁busy`, Modeld, ▁bagh,  ▁Torontol,  ▁worstp,  重要讲话t, 生理x,  的发生|, Path, 真诚,  ▁expansion,  自然资源,  ▁iron,  公积金, ▁Cru,  ▁meetings, 密集, 一方,  在我们,  ▁sitting,  的力量,  ▁fantastic, 做人, 它是,  等症状,  novation,  ▁circumstances,  ▁Expl, 邮件,  的样子,  ▁valuable,  明显的,  ▁Illinois,  ▁Images, 依赖, 返乡,  ▁electronic,  ▁possibly,  要加强, Count,  ▁Histor- core-  ▁finds-  高校毕业生 - 代码-  ▁interpret-  多年的- 要想-  ▁worse -  告诉你$-  ▁pref(-  ▁practical,-  ▁legend0- author4-  ▁websites8- ▁Eth<-  湖南省@-  ▁TechD-  ▁kingH-  心理学L- 平米P-  的环境T- 第八X- icide\-  ▁guard`- 就被d-  研究所h-  ▁Bangl-  ▁Wherep- 验收t-  ▁missedx- 熟练|- 不妨-  ▁Area- ▁NC-  系统的-  完成了-  ▁widely- requ-  ▁desire- 期内- 不怕-  ▁accessible-  ▁Nick- Off-  ▁Corporation- 马克- 海鲜- 拆除- eters-  ▁diversity-  一样的-  ▁teach-  或者是- 受益-  经济社会发展- és-  ▁marg-  ▁Travel- 名校- nesota- 请求-  ▁anywhere-  igrants-  ▁stone.  的教育.  ▁tight. 电气 . 工委.  的世界. bal.  ▁nurs.  ▁paying .  ▁Vice$.  ▁reserved(. 人和,. 太平0. ▁;4. Ke8.  ▁faculty<. config@.  ▁SoftD. shipsH. 处处L.  ▁SantaP. 看似T.  AlthoughX.  ▁producer\. 东部`.  ▁distinctd.  ▁brokenh. 轻轻l.  ▁Magazinep. 承包t.  ▁contestx.  ▁settings|. 预算. Ste. 细致. 衔接.  义务教育. image.  ▁yards.  做什么. Vis.  ▁quiet. ▁·. 事儿. 治安. 五个. person.  ▁designs. 债务. 打破. 报价. 勇敢.  ▁deaths. boy.  ▁recommended.  防控措施.  ▁delight. 除此. base. 一杯.  ▁cheap. arian. 频繁. acking. UM/  ▁substant/  ▁occasion/  ▁enforcement / ▁Ur/ ▁von/ 点亮/  ▁telling/ ▁sad / 情怀$/  ▁therapy(/  自网络,/ 这段0/  所有人4/  ▁replace8/ orne ▁mob>  achelor>  解决方案 >  ▁dogs> etts> 激活> ▁Non>  所得税 >  ▁answers$>  ▁fixed(> 台风,> 与此0> ockey4> 拍卖8>  垃圾分类<>  ▁Line@> ▁synD>  ▁industriesH> WindL> oddP> navT> 步行X> 组建\> 县长`>  ▁intentd> ileyh>  ▁Pictl>  ▁yesterdayp>  ▁findingst>  ▁Summerx> una|> 会长>  ▁creates> ▁FOR> rog>  ▁capable>  ▁counsel> Hel>  ▁giant> 凤凰>  ▁embr>  ▁boat> 局面>  感兴趣>  俱乐部>  ▁Wilson> 方的>  的前提>  ▁Culture>  武汉市>  ▁inspiration>  ocument>  ▁totally> 中毒>  expected>  除此之外>  这个问题>  都不会>  ▁interior>  ▁illness> 细化>  ▁moves>  ▁external> 改进?  责任编辑? 涵盖?  ▁seasons ?  残疾人?  性价比? 未能?  ▁Entertainment? 课题 ?  ▁sample$?  ▁alcohol(?  ▁Forest,? 控股0? 退出4?  ▁facts8? ova}U unterU  供应商U  ▁strict U  ▁investments$U 筛查(U  先点击,U  ▁copyright0U  才知道4U akh8U 头痛t 能让t 生姜t wpt 意愿t goodt 从中t asantt Lifet поt 也都t oydt  ▁clothingt  文化的t  三个月t 学霸t 发射t 洞口t  副市长t  每日一u 雄安u offsu 产后 u  ▁noiseu 华南u junu 划分u 宝妈 u 主张$u  体现了(u encing,u ▁Nob0u 小小4u  请您先点击上面的8u 分散  ▁dollar@ 昙花B 的多D 走过F  ▁sectionsH 团购J  ▁requestedL  企业和N 运转P 离子R  ▁calendarT 车上V  等重点X ▁JosZ  届全国\ otype^ 备受` 招收b  月中旬d  了解更多f 散发h entlej SONl olkn irtsp  ▁speaksr 稿件t 室外v  ▁albumsx 不等z 门市|  ▁insert~ 业内  ▁peak 菩提  的其他 我还 olo 榜单 igs prot 仓库 见的  ▁Professional witzer  的管理  ▁Marvel iatric agic 交管 巡逻  与其他 慈善  message sche 好转  ▁publications James  ▁hosts ▁aer 题型 崛起  作为一个  ▁theater  ▁Organization  从业人员€ 大海Ā  工作地点ƀ  ▁extraordinaryȀ  ▁passwordʀ 湿气̀ Use΀  ▁VeterЀ  工作会议Ҁ 秋天Ԁ  ▁indicateր  ▁till؀ 万事ڀ 不应܀ 白羊ހ 书中 upload 代价 Cap  新中国  ▁loyal 架构 签证 多万  ▁Solutions 涨价  不断地  ▁workplace  ervations 堵塞  严格执行 adi  ▁Missouri 友们  注意力  有效期 nor  ▁oldest ln  ▁signature 阻止  公安部 Run  ▁objective  ▁harder  ▁operated 呕吐  ▁Estate  ▁Trib" 小姐$ ibl&  想要的(  ▁True*  ▁opposed, 丽的.  ▁Hind0 "/2  ▁absor4 yt6 淀粉8  ▁samples: amous< zo> 7 0习近平新时代中国特色社会主义思想@  ▁MatthewB ▁unaD  ▁associateF  按要求H vsJ ▁datL 担忧N  薪资待遇P 寒冷R 加热T  管理部门V 环比X 走势Z  只要你\ 对着^ 换届` 初期b 层层d 分工f 开除h ▁Halj  早上好l 媳妇n 不已p  ▁Centuryr  ▁dustt 摆放v  卫生健康委x  ▁Scottishz  ▁Howard|  ▁algorith~  ▁residence Hz 退款 喜悦 一小 更要  ▁refused  ▁comprom 评分  还有很多  ▁conscious  ▁Private  ▁entrance ▁Tai 预习 悲剧 clear  中小学生 Tim gate 总量 Local  ▁reven olitan 直至  Business 自杀 省内  ▁jazz Tech bound rior  全力以 对他  ▁squadā ablingƁ + $这样您就可以继续免费收到ȁ  ▁Utahʁ  ▁Electrić  ▁tied΁  ▁MoneyЁ  就是要ҁ 防晒ԁ 太过ց 跟我؁  男人的ځ  ▁shell܁ 群里ށ 时长  可以看到 溃疡  General 调动  的时代 朝鲜  市委书记  ▁Select  ▁poetry  ▁completion TOP  在农村 事发  的心态 %　  ationally 擅自 病变 电网  有一次 mate  Controller  ▁tone 人不  ▁piano  十九届  ▁intense  最后的  部分地区 电源 林业  ▁Archives  完整的" ▁FA$ 做饭&  ▁lifetime( hot* 而成, 隐瞒.  ▁demonstrate0  ▁Turk2  ▁participating4 做得6 die8  ▁evil: Vol< 倾向> 入园@ buB ▁PedD  ▁trackingF 妥善H  的空间J 放宽L aniaN  ▁essentiallyP  ssssssssR  ▁MarineT dersV  ▁orientX  ▁continuZ  村委会\ 为其^ 排版` 介入b  achmentd  ▁boldf 做过h 保定j ▁FLl  ▁firedn  信息的p oubler Richt ▁Bluv Alexx  ▁workshopz  ▁behaviour|  ▁speakers~ 更快 PH 诠释  ▁transactions 愉快  管委会 ▁Wi  ▁Auto 特价  的特点  witzerland 履职 幽默 ▁AC 偷偷  辽宁省 潜在 tical  ▁glob  ▁ministry times 标的 ля де (( 打好  ▁reporter  我知道  扫描下方二维码  riculum 工伤 疗效  ▁Hope 一处‚  可以选择Ă 正义Ƃ  ▁recruȂ  ▁addressedʂ 防线̂  电动自行车΂ ratingЂ ▁sty҂ 就很Ԃ  ▁Philippinesւ wall؂ ▁godڂ 贴心܂ 中原ނ TD  ▁Khan  ▁whilst  ▁electrical  和工作 一番  信息网  的习惯 PDF ▁Opt 暗示  防控制 干预  ▁prest  全力以赴  ▁interactive 港口  条规定 疲惫  ▁Standard 经开 利的  因为你  ▁regulation 执着  可爱的 双子  ▁removal ACT  相关规定  ▁unlikely omic  ▁races  ▁grounds" 一站$ 津贴&  和家长(  ▁fundra* 一金,  ▁distingu. 首选0 是无2  联合国4 新房6 眼泪8 买到:  ▁Performance< 金山>  部编版@ rockB  ▁politiciansD 但却F  岗位要求H  ——　J 把你L 的地N ▁SOP  ▁ReservedR 联络T  ▁viewersV  ▁CorpX 海口Z  活动时间\ 改为^ 日益` 同心b 气管d CONf 行车h  ▁Corej  请联系我们删除l  ▁mapsn  调整为p 必需r  ▁unvet 周日v zensx  不小心z  ▁involvement| 算法~ 新人 容量 ▁Ark  流行病学 这门 iem 助手 失信 实操  ▁legit 历经 全套 播出 再生 往年 ","  ▁font  技术的  ▁Brooklyn  ▁suddenly  ▁complaint  ▁upgrade aro 他还  ▁promotion  ▁ethnic 小于 AST  Property ▁OS DAY  ▁imper 再到 双向ƒ Centă AKƃ  研究中心ȃ 光伏ʃ  ▁empower̃ Super΃  ▁choosingЃ 按钮҃  ▁Marsԃ 设有փ 额外؃ 湿地ڃ dr܃ 扬州ރ inth 不顾  ▁Constitution 警务 叠加 调控  ▁module  的一部分  效果图 双鱼 ugg books  ▁Planning ,' 锅中 跨境  ▁query array  ▁fault 不做 座的 stud  ▁absolute 亮相  纪录片  ▁pill ▁Pur 美景 ▁sin 角落 八字 好人  ▁Sustain 西医" ▁Il$  ▁classroom&  人教版( oscow*  ▁jail,  ▁destroyed.  改革开放0 味的2  等部门4 志成6  之前的8  ▁penalty:  ▁Dutch< 王者>  最美的@ 商户B 口语D  ▁relationF 车身H ausJ 五十L  ▁UnitN  ▁quoteP  ▁BeingR 下旬T  ▁ChapterV 周知X  ▁activelyZ ▁Dur\  ▁Such^ 长久` 预售b Emaild  为了让f abuh  ▁mysterj  ▁Javal 高职n  ▁Indexp healthr  ▁sacrt 火热v  微生物x 主演z 是的|  已经在~ 教会 主力 aver  被执行人 可达  男朋友  刚开始  ▁spots ▁BY 端正 utive  ribution Micro what 又不  ▁devast  ▁Paper  法律责任 场的  ▁gathered  也不用 存储 enna  ▁Kentucky 设定 想过 ▁tap  ▁Dean  有一定的 Create  ▁licensed  背景下 人居 辞职„ ankaĄ 崩溃Ƅ  incipleȄ ▁LEʄ  izations̄ 救护΄  对孩子Є ▁tox҄ bottԄ  ▁Pierք 升本؄  ▁laptڄ  leveland܄  ▁Propertyބ  ▁gifts  ▁vert  ▁divided  ▁protests  ▁chicken " 侵犯了您的合法权益 那天 定价  得不到 Pal  小姐姐  ▁Stew  ▁fascinating 煤矿 rt  专项整治 irds  ▁seeks 出轨 estone 石化  文章了 活得  ▁User aria  隔离治疗  ——“ 供水  ▁armed minute сь GM 不准 ippi" 巨蟹$ 优越&  ▁pleasure( ▁Fle* ento, Action.  马克思主义0 olars2 haust4 妇科6  ▁Source8  ▁deck: frac< 带货> ▁hip@ 沉淀B  是没有D  ▁defensiveF 甲状H  ▁greatlyJ 测温L  breakingN  ▁CapeP 破产R  ▁rangingT  ▁clockV 太高X emplZ Rober\  etition^  每个人的` 理科b ▁NOd  ▁Jessf  ▁bankingh  在国内j  ▁voicesl 码头n 车内p 极致r  的道路t Linev  ▁roughlyx COVIDz  和服务| 首发~  ▁Norm  ▁Oklahoma ▁Tru  ▁lowest 提倡 录制  ▁estimates 鞋子 笔者  中共党员 相继 预告  受疫情影响 ▁Gov Result 油画  这些人  ▁whenever 附加  perties 旗舰  ▁defe  ▁exclus 进来  招聘人数 耶稣  才发现  公布的 开课  销售额  project  ▁anxiety  ▁Duke  Through…  ▁dangerą 时常ƅ 往下ȅ 自家ʅ 处在̅ 从容΅ ramaЅ otton҅ 闺蜜ԅ 一笔օ  ▁Leadership؅ )“څ 内地܅ ▁Wyޅ ▁BC  rection 秒杀 usher 繁荣  等信息  ▁teeth ▁В  ▁Roberts Take  selling 同等  ▁Working Arch  ▁Creative 纠正  以上学历 压实  很多人都  ▁Capitol anne 日后  关系的 amer 饺子  半导体  ▁Galaxy 日照  ▁conservative  ▁grants ▁lad 硬件 养猪 商铺" 莆田$ 互通& cos(  ▁concepts* ishers, ▁DJ.  ▁Thus0  ▁Fant2  ▁cameras4 错峰6  是一名8  ▁Nash: assion< 大批> orses@  ▁investingB  参加工作D eiF ▁EXH  ▁consultationJ  ▁CrimeL  ▁MelbourneN stanP  日凌晨R  ▁formationT  ▁SpotV  满满的X  ▁stayingZ 赶到\ 戏剧^ 眼神`  ▁currencyb  performd  ▁offensef 严谨h hlj 所能l  招聘公告n 歉意p 财物r  ▁anywayt  ▁employedv  ▁computersx  ▁characteristicsz 力争|  ▁assigned~ 之上  ▁pride 固醇  ▁pushing 蓝天 徐州  目的地  aration  ▁Maria  ▁protecting  不动产  的指导 值守 开口 VR 伴有 uten  ▁thin  转化为  ▁adequ ча  ▁Tele hy Query ▁Ta 享有  ▁occurs Love ▁Gab  ▁Hamilton  ▁loop 负债  看不到 用工† quisĆ  相关信息Ɔ  职业技术学院Ȇ 震惊ʆ  ▁meaningful̆  sectionΆ 你我І  ▁minim҆  ▁shipsԆ 相伴ֆ  ▁desper؆  ▁Castleچ  北京大学܆  ▁suicideކ  完全是免费订阅 封面 变形 苏联 TY live omm oli  ▁Queens Bel #　  党委副书记  ▁Trail  ▁consec  ▁Autom  ▁Steel ку  ▁workforce  ▁gang ▁Cho  ▁clim  ▁concluded  的市场  ▁Walker 邢台 Global  健康监测  ▁Argent ted  俗话说  大家好  ▁Empire mem  ▁Jews"  ▁temperatures$  ▁Bron&  ▁newsletter( 遏制*  ▁Viol, 空军.  ▁Region0 花了2 分校4  ▁moral6 楼层8  半小时: 我很<  ▁parliament>  ▁conversations@ NumberB fitD ILLF  ▁LevelH 唱歌J  医疗卫生L ▁$\N 注入P 入驻R  ▁gratefulT 在意V  ▁survivalX  一下子Z  ▁Tokyo\  ▁borrow^  ▁elsewhere` 扫黑b ▁Obsd  小学生f 堪称h  ▁Phillj  ▁conclusionl 恶性n 冷冻p Finr  ▁moodt  ▁Bloomv ▁McKx  ▁administrativez 南路|  ▁Dance~  ▁tackle  ▁sharp  人才培养  ▁defeat  条件下  ▁vacation olen 守法  的视频 ▁Try  面试成绩 文职 ▁Cow  ▁opinions 提成 区间 Video ▁Naz 防御 City  ▁Terms  ▁environments  ▁indicated 约束 ▁BE 华东 MD 一楼  International  都不是  ▁journalists 花椒 里程 起步‡  ▁Andyć 连锁Ƈ oustȇ 题材ʇ  ▁NASȦ urer· 花的Ї  ▁Golf҇ 外部ԇ  和大家և direct؇ 还款ڇ 出国܇  ▁leatherއ 友好  教师资格证 说着 quire sing  ▁Kennedy 被动 参会 核桃 前方 奋力 father 条款  ▁ratio 赶快  中国共产党成立  ▁prev 运作  预防控制 ussion  ▁depression ▁USD ▁tie Sam Str linary  ▁instruments 聆听  有网友  ▁anymore  ▁bonus  //////// 全天 7 0来源标注错误或侵犯了您的合法权益" ▁Ec$  已成为& ▁Od(  的项目*  ▁enjoying,  ▁Military.  如果说0 ounced2 piece4 精美6 ailand8  Download: ▁odd<  ▁naturally>  ▁Diff@  ▁functionalB 和大D 商贸F besH engerJ 保卫L ORDN 取决P 头皮R  ▁LanguageT  ▁containerV ennyX 解毒Z  ▁го\ ▁Sab^  甲状腺`  ▁documentationb 解题d 外语f techh  ▁homelessj  ▁routinel  ▁compatn 加分p  净利润r orict 愤怒v  ▁Effectx 函数z  目的是|  加快推进~ Tag lined  ▁Samsung 遭到 约为  奇特的  吉林省  ▁Justin ▁Bah Long quiry leans  和发展 沿线  ▁Updated  children emed client  ▁cheese  ▁Update  ▁mystery ▁Wel Av  ▁tale master  ▁obst 还得  ▁amongst  ▁earnings  ▁Come wang  ▁cancel  ▁incent  ▁evaluationˆ 得住Ĉ 期货ƈ  ▁titledȈ  ▁superiorʈ 收缩̈  郑州市Έ JackЈ semble҈ istingԈ 不懈ֈ  的美好؈ icutڈ  这不是܈ 鼻子ވ 入党  — 长达 grade 手的 育人 roph  issippi mers  ictionary 和他  ▁finger ogs  区政府 ▁П 要以  ▁coaching  ▁Jonathan 借鉴  ▁rally  ▁movements 廉洁  ▁specified  ▁Object ▁Gun eals  ▁forum  ▁Freedom  ▁mistake 最具 热量  在学校  积极参与  ▁neighb" 饮用$ Af&  ▁interaction( 脱离*  一家人, 宽松.  不少人0 ▁Sov2 FI4 ▁McG6 励志8 学堂: 八卦< ▁Sac> ▁Tel@  ▁afraidB  ▁AndreD letedF  不少于H 转向J  族自治L selectN  你喜欢P  ▁discriminationR 全长T ▁DakV  行为的X 这三Z 整齐\ 丰田^  ▁disrupt` 搞定b aposd ▁**f  ▁Girlsh = 6若有来源标注错误或侵犯了您的合法权益j 内科l 奖项n 山药p  ▁cockr perst  ▁popularityv aliax  请注明z  ▁Vancouver| ▁PRO~ CEO  的可能性  ▁creativity  各位朋友 率为 杀人 几百 " 传递更多信息之目的  ▁paintings paren  ▁centuries backs 阻碍 淄博 画家  下一个  行政部门 带上 try  ▁parks 共识 Three 讨厌  ▁Unfortunately  高等学校 ▁cm  criptions  ▁petition Make 葫芦  ▁lake 月生  高水平 此刻‰ avedĉ ateralƉ  ▁Computerȉ 久了ʉ 严防̉ 临汾Ή  伴随着Љ  ▁alter҉  ▁screensԉ 谈判։ 她是؉ ▁EPډ  ▁smile܉ 谈话މ  ▁Sale unate overs 年第 oni 专场 Phil  ▁respective  自己在  ▁credits 天秤 还未  ▁relating del  心理健康 Both ▁__  职业资格  ▁investigate 充实  Response  商学院 迷茫 Rob 眼光  ▁Content 任意  missions 余额 HD 师大  ▁Made 难忘 恒大"  ▁gene$  ▁enorm& layout( 限度* uct, 站位. 沿海0 一股2 burn4  工作时间6  ▁Woman8 旋转: 身的<  了一些>  ▁Having@ ewsB  ▁ChelD 作息F  ▁paddH 碰到J  ▁stabilityL 灿烂N  ▁throwsP 省考R  ▁CommentT dirV 侵害X  请放心关注Z är\ Del^ 座椅`  ▁operatesb 如需d 膝盖f 计较h 胡椒j  ▁Shell 欢喜n  ▁tonightp  点个在看r  ▁Employt  疫情防控阻击战v 民国x googlez 主导|  告诉我~ raf 煤炭  October 阿拉 而被  ▁chocolate ▁liv  lementary 编审  ▁abortion  集团公司  ▁Kath 亮的  面向社会  受害人 感慨 面子 耐劳  幸福的  ▁Path  ▁cust  ▁conver ▁gal 队长 ▁iOS  ”　“ 名词  ▁Bull 中和 ega 专职  ▁euro  ▁affairs  ▁demonstratedŠ 图中Ċ  ▁preventionƊ  ▁geneticȊ 了他ʊ  ▁Christopher̊ assertΊ 变动Њ 图像Ҋ 心思Ԋ SI֊ oting؊  建设工程ڊ  属于自己的܊ judފ  都非常  我们一起  ▁ties 挫折  ▁panels zb  ▁Secret izza 付费 athon 纠结  不断提升 果然 沙发 的两  取决于 勤奋  本文转载 之为  一会儿 一号 ▁OK 收听 一来 node bered ibbean 大纲  Location  普通高中  ▁setup  强烈的 因子  人体的" they$ ▁ax&  ▁coup(  明确了* school,  ▁uniform.  ▁desc0  人民代表大会2 ▁Han4 玩耍6  ▁proport8 拼命:  ▁fantasy< 发烧> ardo@  ▁encounterB  考试时间D 一丝F 四年H reachJ  ▁ArabiaL EvN  金融机构P  脑血管R 烹饪T  ▁VentV 饮水X 法学Z  很可能\  ▁breath^ 白菜` 必定b  准备好d ▁FCf 应在h  也需要j  ▁Initil hadn  ▁operatorsp  ▁Readingr 所长t 上榜v  ▁Interviewx  要做好z 古典|  ▁packages~  ▁entity 全是 爆竹 ▁Eve 质感 August  ▁trips  立刻就  ▁antib 竣工 ensing  ▁breaks 演绎  因为他  ▁Investment 过滤 一根 开朗  ▁smartphone GC  ▁stayed  ▁Less 废物 ▁FBI  疫情防控指挥部 bow  ▁Additionally ▁ABC  ▁sixth  最重要 石洞  Welcome 禁忌 民宿‹  ▁jurisċ  进一步加强Ƌ  ▁Antonioȋ clickʋ  贸市场̋  或其他΋ 女主Ћ  科学技术ҋ  单位的ԋ  ▁courtesy֋  外国语؋  ▁Greeceڋ 富裕܋  ▁Daysދ  outheast 发声 进攻 BB  文明城市  ropical  ▁salary 大同 UST 投档  ▁turb  ▁emails  ▁Published  你自己 Lab  圣诞节 检疫  一个是 rones  ▁preferred  ▁admin 无可 雨水  医院的  ▁rescue Access 暖心 卡拉 olph 给出  ▁preced ( !长按上方二维码识别关注 nan 观赏"  ▁suffer$ 契机& ▁sid( ACE*  迎来了,  ▁lyrics. 扮演0 isdom2 露出4 ▁CC6  ▁overcome8 ublin: 巴巴< 克兰> 宁县@ 网点B 留意D 合计F  ▁chronicH  ▁SelfJ  ▁subsequentL ovesN 小人P  ▁blindR  ▁enemyT 鲜明V  parencyX 伴你Z  ▁utility\ 剧院^ 后勤` Joinb  ▁intersd  ▁graphicsf  ▁emotionh  ▁accompanj 遗址l 个省n laimp urir  ▁Economicst 淡淡v 永恒x Macz 一顿| ▁К~ 寓意 活泼 abies  ▁shower 小红 落后  新增确诊病例  生活方式  ▁Catal  ▁Liverpool 庄村 口臭 脖子 "]  chnology oning  ▁admit  幸福感 next lad  就可以了  ▁consistently 构造  ▁Gulf 还可  ▁Bruce  的人们  ▁Plant  绝大多数 激光 售后 三轮 华北 rotŒ  冷空气Č  多年来ƌ  年开始Ȍ 犹如ʌ asm̌  基础知识Ό  ▁portrayЌ  就会被Ҍ 事后Ԍ  ▁junior֌ 不觉،  ▁Thompsonڌ  ▁soccer܌ 之年ތ ▁<< 滨海  ▁casual  ▁minds  ▁tours plex ▁Li  December 心疼 反正 Base  ▁invas  ▁weapon  胆固醇 走在  许多人 ▁Sri  化妆品  ▁grid 柠檬 韭菜 erer ▁Esp  这个人 status 发型 地处 wered  ▁Political  商品房  把自己的  岗位职责  ▁pour  ▁inaug"  ▁boxes$  政策的& yu(  interest*  ▁marijuana,  为一体.  六年级0 遴选2 排毒4  ▁Orleans6 ㎡,8 ▁Na: 惠民< rary> 合物@ InsB  ▁exitD 情商F  ▁resolveH 言论J madL  ▁secretaryN  ▁shortlyP  ▁swimmingR боT angoV effectX NAMEZ  实在是\  用自己的^  ▁asks`  文化旅游b Unitedd rinef  ▁Swedenh 山路j  心理咨询l 翡翠n 对该p  副院长r 上网t  ▁Blockv issax 头上z 之力| 向您~ 模糊  ▁Denver  自媒体  Facebook  不允许  发展和  Project 河流 经查 神仙 公顷 夜听  一般来说  ▁meters ques 田园 ,( 农药 ▁SD 商店  市人民政府  就知道  ▁warned 仙女  众志成  ▁holidays  ▁Classic  ▁Lane  ▁Must  之一的 OUR ados  ▁blank 筑牢 GLč 充值ƍ  ▁integrityȍ Legʍ 步入̍ 盘点΍ 节气Ѝ  每天都有分享ҍ uvԍ  ▁moon֍  各方面؍  ▁bunchڍ 使人܍ ixedލ 但也  ▁hunt nell 最强 房企  越来越多的  ▁Exam  ository  ▁unem  ▁Episode  ▁Together 问我 stock  ▁buyers 财神 yler  ▁mathem  原材料  English riven 只想 sect  吃苦耐劳 出汗  ▁Prop 拿下  ▁tort  ▁criticism  市场监督管理局  tsburgh 普洱  出现发热  ▁advertis  ▁meals"  orgeous$ 此后& 宝马( 搞笑* 在乎,  众志成城. 交汇0  ▁racial2  能力强4 合规6  illance8  ▁careful:  Support<  ▁workshops> Que@  一般的B  领导下D  ▁attitudeF  唯一的H  经营者J  RelatedL  典型的N  ▁SometimesP 超出R 丰收T BreV 大楼X 不分Z  分享给\  也比较^  ▁printing` 广阔b  ▁Olympicsd  ▁substancef ooph  安全管理j iratesl  ▁Legisln 五行p  个月的r  ▁screeningt  推荐理由v  做出了x 用法z  ▁tons| 伤口~  市面上  ▁agreements LL  ▁Visit  向社会 单一 家务  学生们 集聚  ▁para  ▁Album oda 淋巴 安区 丧失 iop ▁Mun  ▁Holl 比起 pin 透气  ▁hired 去除 kee  有一定 common ttp 特意  ▁Russell  住房公积金 stop  ▁coaches 情景  的作品Ž ▁EVĎ 保洁Ǝ 语法Ȏ qingʎ  甚至是̎ 增大Ύ 母猪Ў  ▁emotionsҎ  ▁corruptionԎ  想不到֎ 大国؎  ▁Priorڎ + $是出于传递更多信息之目的܎ Didގ 眼中 作了 天后 变革  ▁lets ▁hub 复学  ▁cancell write center  ▁exploration  都是在  ▁rarely  ▁Moscow  ▁Iron  ▁Switzerland  ▁artificial 手套 人次  ▁tries  年以来 绿地 kind 寂寞 yang lang  November  ▁minimal ▁MO  ▁Mind  血液循环  ▁Urban 将被  ▁Dick" 屏障$  熟悉的& Share( 万个* Nav,  ▁robust. prises0  ▁directory2  ▁donations4 ultane6  ▁romantic8 深处:  ▁argued<  ▁arguments> 三人@ 答应B 没事D 创立F  ▁periodsH  viouslyJ 经纪L  下面的N ▁unnP  ▁renownR 实质T 节能V  对我们X  ▁wirelessZ  的角度\ heart^ leting` 一波b jxd 决胜f 呵护h 饰演j 建档l 菏泽n mediap Hir 体能t  ▁Garyv 快手x  ▁configurationz 邯郸| 审议~  ▁elder  ▁towns 北海 无线 恐慌  在内的 些年  你的人 夸张 王子 界的 他会 allel  极大的  ▁legendary 时髦 凸显 Phot Mart  ▁Convention annah 法庭 更能 词语 北路 cure  ▁Orange  ▁rival angh  的话题 统编 ousand  女性的 oving Schď  这篇文章Ə  孩子在ȏ  ▁bagsʏ 午餐̏  ▁survivΏ ▁("Џ 事关ҏ  评论区ԏ  ▁instant֏ 说出؏ ▁sedڏ  二等奖܏ 艺人ޏ  类专业 期满  作为一名 Sing  太多的 酒后  ▁dual  成都市 umph 打了  ▁Leader wy 敬业 表格  ▁Domin  ▁Labour 少吃 抽奖  William Music  ▁provisions + $詹飘飘詹飘飘詹飘飘詹飘飘 归来 拥堵 ogg  ▁weird  ▁boards  ▁bowl ▁gym  稳定的  体温检测  ▁dozen 米的  ▁Pear" 图为$ 节省&  起来的( 两者* 但这, ▁ans.  ▁Beijing0 开创2  你会发现4 进取6 阳区8 蚂蚁: ▁une<  ▁encouraging> 跟你@  ▁somehowB rapeutD BUF  namespaceH 弥补J 意向L  ▁fiscalN 如其P 二楼R 热度T  ▁gatheringV  ▁BreakX  每个人都Z 转为\ 不愿^  ▁satellite`  很多的b Persond 一台f  ▁allegationsh 年薪j 尊敬l  ▁Theatern  ▁engineersp 马路r ▁mmt  ▁gottenv  ▁constructedx 船舶z 外来|  ▁mountains~ bet 晚餐  ▁independence  ▁diagnosis True  ▁Gram ▁pul  ▁worried School  各种各  ▁Sony 衣物  ▁Alan ▁coc  ▁Movies 恐怕  ▁Interest 将对 底层 急诊  ▁Adams  的重点 气体 浦东 ▁kit  ▁Parent  充分利用 航母 四是  ▁lesson  ▁hide PI 人身 国学 作案Đ  转载此文Ɛ 秋冬Ȑ Paramʐ ▁IV̐  ▁Coleΐ " ▁▁▁▁▁▁▁▁▁А  ventionsҐ givingԐ Brit֐  五险一金ؐ  ▁nonprofitڐ 论述ܐ 中专ސ ieve 听课 阵容 shore 尼亚 ndrome  ▁refugees town  ▁modified  ▁annually  ▁jury 增收 beat than ▁Fed  ▁establishment 矫正 合影 平行 面粉  目前的  ▁tremend 大爷 股市  ▁toler  ▁Brexit 一组 全线 Former 体力  大家可以 aco 队的 ▁Gas" LD$ San& SM(  记者从* ivals,  ▁Kings.  ▁veterans0 ethe2  都不能4 ASE6 acious8 红的:  ▁nights< 纺织>  ▁hybrid@  ▁MexicanB BasedD  centuryF rusH LearnJ 冲锋L  成立于N 当作P  ▁viewingR 瞄准T  交叉口V  杭州市X ▁NBCZ ▁tu\ ▁fu^  ▁reviewed` 人际b ▁Johd  的人才f 阵地h emetj 张文l 叶片n 叶子p  西安市r  ▁Palacet  ▁sequencev 监理x 是非z  ▁flexibility| 交友~ shot 此事 维权  ▁None 四级 走上  ▁listing  党支部书记 си  ▁applying  ▁Cleveland woman  ▁pollution ве  ▁acres 缘分  ▁inspire  处女座  将成为 常住 言语 terior  ▁Intelligence 弟子 学士 央行  ▁delivers  平方公里 仪器 看待  ishment  的生命 头晕  的男人‘ 加盟đ  ▁раƑ 我区ȑ  Rememberʑ  assment̑ 旅居Α  ▁cannabisБ 汕头ґ Liveԑ  ▁excellence֑  ▁immigrantsؑ 多项ڑ  ▁careersܑ 开着ޑ 城管 恶意  ▁exclud 再见  ▁duties  ▁muscle  ▁Nichol  ▁enabled glas  September  ▁humanity  本文部分  ▁про 好几 灵异  ▁Phoenix 助推 终究 上演 呼和  家人的 切勿  ▁adoption 饭后  ▁accounting  ▁Federation 烟草 西亚  是一位 防水 新品  ▁participated  ▁exhaust  ▁edited"  ▁globally$ ▁WH& 几何( herent*  招生考试, entry. 氨基0 教科2  ▁populations4 elect6 Care8 牵头:  ▁districts< ailing> Law@ ASHB oftD ONEF  ▁passengerH 欧盟J  ▁BroadwayL 合一N WHP  表现为R 丢失T _{V  较高的X  ▁folkZ Button\  ▁flour^  起来了` ▁Rocb 也别d  ▁begunf 大战h 顶尖j  这才是l  予以删除n ▁Bhp  ▁legsr “　t ashingv  ▁keysx  ▁Portlandz  ▁experiencing| ulu~  ▁recall  ▁casino  ▁Rain  ▁Steven  ▁Snow 细则 NBA 月饼  ▁agricultural  科学与 流失 购票  ▁Challenge  ▁crack 若不 enez 因果 送上 不着  了起来 凡是 深情 达人 Expl 四十 颁发  ▁accordance Pass 特效  ▁attempted  Following  ▁certificate  ▁nationwide 百科’ 损坏Ē  之后的ƒ  专升本Ȓ  ▁Chamberʒ  ▁renewable̒ ▁ENΒ  老旧小区В 篇章Ғ  的政治Ԓ sis֒ Methodؒ 贵阳ڒ 停放ܒ  ▁importantlyޒ Enter  ▁Reserve 牢固 感激 信访 元旦  ▁absence 不为 roduct  月出生  ▁Agreement 没错  等你来 吃完 窗户 感知 达标 误区 aha 感叹 快捷 月光 自驾 lims 大蒜  ▁designers  对中国  生活在  ▁Million  ▁reflects 里有  ▁Native  ▁Lawrence 醒来" 倾斜$ Bo& 终点( 并发*  重要讲话精神, annon. 拇指0  ▁riding2  ▁Race4 Sw6  ▁achievement8  ▁agriculture: 因而< ▁Gib>  的做法@  的存在B turnD  的一项F edlyH  ▁wavesJ  ▁graphicL  不断提高N ﹌﹌P  ▁ApproR  教师招聘T  城乡居民V outubeX  ▁activistsZ 油价\  ▁drives^ 重组`  未来会b 供稿d 品行f  目标的h RTj Fel tainsn 打包p  ▁stakeholdersr 老的t 签字v 华夏x 查获z iche|  ▁spectacular~ 便于 开的  重要指示 ▁Bus 金牌  ▁parameters  ▁departments  ▁Finally  ▁healing  ▁kilomet 计时 Lay 果实 剩下  准备工作  形成的  去医院 Sk  ▁stocks Bet  ▁Kate 区块  ▁Maine  ▁principle 收拾 acles 稳步  ▁estimate  ▁continuous avas  福利待遇  同一个 念佛  ▁locally“ 恶心ē  县委书记Ɠ 衡水ȓ Thoseʓ  ▁pricing̓  不过是Γ 无私Г  ▁overseasғ  ▁Machineԓ 假的֓  ▁scholarshipؓ 加持ړ  ▁damagedܓ 上车ޓ 诉求  值得一提 jin 绍兴  ▁Louisiana 汗水 查找 各区 不像 衬衫  ▁Regist  ▁radical alg  ceptions  ▁settled ▁Hug ▁Ott  ▁controversial  ▁Could 延安  ▁displays  intendo  ▁declined 做个  urricane  ▁Singh 教案  税务局 翻炒  负责同志  ▁inclusion  ▁Oscar  ▁founding  的思想"  ▁Anim$ 各方& Input( 随即*  ilation, 楼梯. 绿卡0 越高2  ▁Products4 坚实6 三国8  ▁versus: 滋养< ▁Vi> 总有@ 两会B  可以直接D tooF  privateH 岁时J ▁rugL capN ▁NapP 用车R  ▁garageT 报送V 融媒X  ▁merelyZ  爸爸妈妈\ ▁nil^  识别二维码`  了一种b  ▁mainstreamd 分行f  ▁Experienceh  ▁Joshj  ▁enginesl QUn 有声p  被誉为r  排行榜t  一直是v  ▁Charliex 全场z  ▁engineer|  ▁surrounded~  ▁wanting  日常生活中  ▁Studios Sl 海岸  微信关注 毛病  ▁ruled  elligent 奶粉 克斯  ▁promised  ▁Mand  ▁Against )” 业态  定点医院  ▁Jennifer  积极性  ▁Baltimore  ▁sophistic 一篇 特种  ▁brut  ▁Friends  年全国 rator 众所 ▁Om  ▁footage 走的  ▁Side  ▁objectives ▁cha”  ▁codesĔ  ▁ordinaryƔ exportȔ  ▁Mississippiʔ  ▁desert̔  经历过Δ  价值的Д 问答Ҕ  目前已Ԕ 番茄֔ моؔ  ▁emergedڔ 躺在ܔ motherޔ  天然气  ▁epic  ▁Temple  ▁echo  ▁virtually  ▁Solar 封信  ▁argue  国家卫健委 people  support  ▁Taiwan ▁Bor  了解一下 drop 命名  ▁Host  ▁Blood  ▁Gordon  ▁accuracy idget  ▁checks  的照片 单品 不算 学前 国土 千元  ▁fake  推出了  ▁desired  ▁conventional  认识到 菜品"  ▁judges$  ▁crown& charge( 小伙*  ▁Excell, jan.  核酸检测阴性0  ▁allegedly2  ▁Malaysia4 """6 pled8  ▁assembly: 读懂<  ▁bottle> 紧缺@ 明日B 以色D 初三F  我们是H  ▁QualityJ 他也L ▁SudN  ▁easternP 生机R  ▁financingT ernameV  ▁CaribbeanX 营收Z  的车辆\  ▁applies^  医疗保障` Greenb  ▁instructiond  ▁Surveyf  ▁correctlyh  Meanwhilej 小火l 占有n  ▁Jacobp  itativer urioust  Professv  高品质x  身上的z define|  有时需要我们~ 贯通  ▁aest  因为他们 官宣 cedes 运河 ▁sle  ▁combine  两个月 侥幸 首页  相关负责人 Only  ▁dict fish phony  南京市  ▁Athlet asty  ▁Disease 所示 众人  ▁internationally Return  ▁corpor 西湖 少量 痛风  已经成为 emen  ▁buck Jul  ▁Perhaps arma• 报备ĕ ▁hazƕ  ▁marineȕ 对了ʕ 等着̕  ▁prayerΕ ▁TyЕ  ▁mathҕ tvԕ 模仿֕  ▁nodeؕ 堆积ڕ  ▁frequentܕ CIޕ 真情  ▁scenario  新能源汽车 до  已经被 人是  ▁possession orious  ▁proposals 实务  ▁hack  espread 死的 有余  中医院  呼和浩特  的心情 加以  ▁headquarters  复杂的  ▁renowned spring  ▁emphasis  Factory 征地  整个人  ▁safely  Research  ▁produces 公元 来得 牙膏  性疾病 生源" ellers$  ▁hiring& andra( ▁Bec* PN,  ▁manip. ▁HT0  ▁execution2 古镇4 zg6 网民8  ▁routes:  的一面< uffer> 大哥@  ▁disabilityB  ▁BirthD 钻石F  ▁circleH 品味J 刚好L  ▁therapN  ▁AssociateP twR 不离T  让我们一起V 忽然X  文化和旅游Z  非常好\ ospel^  ▁churches`  ▁Indonesiab  报名表d  ▁culturesf  个百分h  ▁loudj  ▁ranksl  ▁spectrumn  ▁dozensp bulr  其他的t  研究员v  ▁puzzx  ▁Thirdz chi|  综合性~  是我国 ▁к  ▁Jazz  process ${  ▁promises 人与  ▁Strateg 冲动 天赋 万科 混乱 Sur 立法  ▁Everything  ▁subscription 可怜  ▁Very  ▁frequency Apple omed  ▁summary  ▁Patri 漏洞  ▁eggs 总分 Found  ▁packed 两条 学家 lett  ▁testim elli 老了– ▁SinĖ  ▁belovedƖ  专业知识Ȗ ▁URLʖ  往期推荐̖  AddressΖ 座位Ж  ▁ReleҖ 献血Ԗ 帝国֖ 性感ؖ ▁Aveږ 口岸ܖ  ▁weighޖ  ▁negotiations 廊坊  提升了  Austral 悲伤 复旦  ▁Nelson 无穷  疾病预防控制 火了 mal eness Show 一到  辅导员  ▁Ident bd 直达  inction Cam  emetery  ▁Graham  ▁Nort 偶然  ▁Mort Does  ▁render  　:　  小小的 统考 没钱  ▁Captain SR Media" 一审$  加油站& 奔驰(  然后再*  ▁hitting, hattan.  ▁Cars0  的意见2  ▁pregnancy4  管理人员6 ▁Cop8  信息技术: san< 哪儿>  ▁Summit@  ▁governanceB 散文D  医疗器械F  这一年H blogJ actorL  原来的N TemP  新征程R  ▁arbitT  ▁certificationV  ▁boneX 要做Z 不完\  context^ 推文`  新冠病毒疫苗b  ▁Sovietd  ▁Connecticutf URLh jingj  ▁Arenl  来自于网络n  ▁delightedp ▁Bior ▁Himt dalev 团圆x  ▁Wrightz xia| 匆匆~ 无声 择优  奖学金 痕迹 而上  ▁charging 事迹  ▁liability liver  ▁Senator  历史的  规范化 薄弱 宽容 下属  但如果 静静 礼仪 oded  ▁receives  ▁disorder  ▁literary  ▁consecutive USA  ▁affili zech  ▁Christians 高原 }{ 仅有  ▁keen  建设和  ▁Practice  ▁pregnant— IIIė 完工Ɨ  的速度ȗ  通知我们ʗ 夫人̗  进入了Η  ▁unlikeЗ 进门җ  ▁feminԗ 远方֗  ▁Successؗ 几点ڗ gieܗ  很快就ޗ  如今的  ▁Bath ixel  一直以来 习俗  ▁Campaign 廉政 Main  ▁fought 愉悦 司令 红线 高手 雷锋  ▁automatic  安全带 联赛  ▁deadline esla 珠宝 限行  ▁innoc wear  ropolitan 尽早 nc 友情  ▁Pant  ▁deter 美白 Europe 故乡 制药 般若"  政府的$ 大米& 六个(  ▁Associated*  ▁judgment,  积极配合.  ▁Palestinian0  ▁cake2  第二批4  毕业于6  没有了8 尽力: 车位<  ▁Baby>  ▁Brew@ OperB  ▁immuneD  ▁donationF 细心H aghJ 出色L 互助N stageP  ▁tennisR 会同T 树木V 沉浸X 华中Z 均有\ early^ 自学` няb 干事d ▁sif  万元的h  ▁referencesj 疑惑l 金刚n 最全p  ▁publishr  westernt 派遣v ▁Cryx 语录z  健康状况|  每天都~  ▁Stars 科创 rowing 使其 造价 锁定 请勿  ▁curriculum  ▁Buck  ▁Thailand  ▁Fame 童话  document ▁Rat  ▁Poland  严格的  ▁alarm  ▁Tree  ▁rend  ▁intervention  ▁drag ▁Lu Full  不可以 首届 下发  ▁Hans  具体的 brit 轻微  ▁bull  可以看出 才华  新闻发布会˜  ▁BeckĘ 避开Ƙ atileȘ  ▁classicalʘ 手上̘ 套装Θ  ▁liberalИ 人到Ҙ 区长Ԙ  开学时间֘ 入场ؘ 餐桌ژ  工作要求ܘ >'ޘ 出自  ▁Tower  不确定  资格复审 抵押  公布了 ▁Cos  表示歉意  ervices  ▁functionality 圈子 对口  ▁divers 蓝字 plant  天蝎座 小吃  以色列  ▁smoke ▁Pil 毫升  ▁enhanced  取得的  ▁reporters 赋能  ▁mechanism ▁ham 含义 扩展 房贷  考试成绩 erse  response 准时" 乌鲁$ 也让& 血栓(  ▁resil*  刑事责任, 民俗. 双重0 主观2  ▁Orig4 vity6  人员进行8 联邦:  ▁advocate< 名片> 两国@  它们的B 列为D  ecessaryF optionH usalemJ 型的L 高低N  ▁terribleP  ▁spokenR  平台上T  服务业V 售卖X 少儿Z  准备了\ 必修^  国家税务总局`  ▁Constructionb flyd ▁CTf  ▁mistakesh  街道办事处j  ▁aggressivel ▁Jenn 再度p  企事业单位r  个百分点t  往期精彩回顾v  鼎力支持x  ▁Muchz 星级| 年间~  专注于  版权声明 车牌 IGHT  有两个  ▁Anna 时要 encer 优美 ▁Ju 代言  ▁Deputy 有用  ▁tagged  坚持以 泉州 春秋  ▁plain hero acco Tok 服饰  的意义 mind  ▁possess 拯救  的知识  ▁rejected IDS  ▁не  他们是  气象台  优惠政策 module™ 市值ę ▁Rapƙ  ▁lobbyș  ▁drinksʙ 登上̙ jaminΙ  ▁restoreЙ 外套ҙ  pectivesԙ 一双֙ blingؙ 不难ڙ  告诉我们ܙ zjޙ elve 金银 ois  ▁Application arity  ▁nursing  ▁entrepreneurs  更高的  职业学院 器的 简洁 繁殖 巴西  数据库  ▁Princip  ▁prestigious ▁gre 奋勇 在上  ▁purchasing 本市 ▁Nik  让你的 fi  和支持  ▁Xbox 决战 ["  情况的  ▁simultane ▁Joy  专项行动 ▁Aqu 专员" Non$ 包头& false(  ▁rehab*  ▁rice, 抓实.  comfort0 ▁ft2  ▁disturb4  工程建设6  article8 结节: 火的< 上调>  ▁encount@  ▁TitleB 文科D  ▁suppliersF ▁MasH  乌鲁木J  志愿填报L  ▁MagnN 冒险P  ▁FacultyR 保密T 长治V iwX 诱发Z  ▁alike\ rous^  防护服`  ▁Dealb  ▁postingd Gettyf 良心h olyj  ▁Versionl  的方向n  ▁Strep provr Whicht  ▁Commissionerv  ▁citizenx  第一名z comed| 嘉兴~  ▁Impact  ▁wooden 注销 airy 结石  ▁rolling 耽误 滋味 只好  ▁Following 投身  ▁inspiring  ▁touchdown big  乌鲁木齐  ▁Carter 宝贵 将以 仅为  ▁Meeting  人大代表  网友们 酱油 sur  ▁formula 接地 党内 天堂  我们会第一时间 Robert omas  ▁flower  ▁Dubai 一笑š  在疫情Ě 仍有ƚ 基建Ț  是一家ʚ 实效̚  长三角Κ  ▁IndependentК ectedҚ  ▁mayorԚ 原油֚  工作报告ؚ  ▁auctionښ  ▁Stewartܚ 之情ޚ  ▁astron 两岸 下游 美团  acteria 摩擦 夯实  了一份 се  四年级 dc  还记得 Rad 终止  ▁tower  不用担心 为期  ▁blow 三位 ▁Gre ▁Inn  ▁aver  的课程  经开区  这样的人  ▁colleges  theless Util  ▁suggestions  ▁examine  最新消息  ▁addressing  的身体 ▁pin"  ▁Activ$  ▁vaccines&  ▁Fast(  ▁painted* 春晚,  ▁Companies. 豆瓣0  ▁keyboard2  ▁Better4  ▁partnerships6 下水8 草莓: ARD< 机体> ▁cow@  品牌的B 几种D  最新的F 清风H ▁);J 长征L  ▁BasedN  ▁newestP ScriptR  ▁AlberT  ▁MathV 一览X 牢牢Z ivo\ orized^  ▁Industrial`  的说法b 垂直d 改正f  ▁heartsh  ▁searj 7 0转载此文是出于传递更多信息之目的l  ▁circuitn  ▁Deepp  ▁exclusivelyr  ▁Createt  ▁Commonsv 录音x uruz  writing|  烟花爆竹~  ▁honored ANT 健脾 面貌  ▁Diam 受贿  ▁Fight  ▁yard 前线  ▁contents 卫视  改变了  ▁Highway Load ▁bio  ▁overwhelming  ▁courage  ▁Warren 玩法 ере header 生效 命中  大家在 吉利 ervous  ▁fastest 疏导  ▁Body 伤心 促销  ▁tribute  ▁Cand  ▁maxim› 也为ě  高风险ƛ  displayț  ▁skilledʛ 大使̛  ▁CommerceΛ uerЛ 在其қ hedralԛ 特岗֛  ▁Citiz؛  ▁Playerڛ  通讯员ܛ smallޛ nal ocur  ▁stops 这名 给您  Builder anches  社会各界  合伙人  ▁celebrating  没办法 除非 变速 大理 ourt  ▁gentle  ▁Hawaii  ▁connecting  ▁Coal  ▁threatened  ▁designated 白的  ▁loving  病毒的  给他们 逃避 ▁CNN  ▁Boys  并同时  ▁checking 茂名  ▁shed  ▁preserve 次的"  itement$  市场监管局&  国民党(  第一位* 江山, 更改.  的公告0 摸排2 相机4 ▁Ky6 祝愿8 糊涂:  ▁underlying< Wed> make@ 困惑B  请注意D  ▁abilitiesF ▁HitH 台阶J whiteL StyN  limitedP  公共服务R  ▁ContractT ChrisV 修建X 数额Z 不尽\ esides^  ▁wage`  了一场b  ▁acclaimd  ▁Strategyf  消毒液h Policej 早起l 开盘n 春运p  irminghamr  ▁Dublint 就来v  ▁sitsx BTz  ▁calm|  广大群众~  ▁exhibit hang  ▁Comb  ▁stem  ▁finest 莲花 直观 精装 Status 持久  ▁notable 探寻  ▁refers 皱纹 入群 不打  ▁impression  ▁enormous 故宫  相同的 无意  ▁lens 过大  ▁guns  ▁equality  ▁Think 斯特  ▁composition 品尝  ▁captain  很强的  公告如下 ör  交警大队œ  这些都是Ĝ ntilƜ 风口Ȝ  你是否ʜ esis̜ 上看Μ 歌手М 祭扫Ҝ 南县Ԝ  接下来的֜  恢复正常؜ 随地ڜ 份额ܜ  ictionsޜ  最高气温  称之为 平和  怎么做 顺畅 钢筋 ▁sew  ▁widespread  ▁Bachelor 战疫 本号 举动 再来  android  ▁serial 丝毫  ▁photographer 梦幻 电流  所说的 ▁obl 消灭  成立了  这段时间 rams 建房 神的  你可能 ▁Amy 考查 tons  生产建设  ▁implications 厂商"  ▁cited$  原因是&  的营养(  ▁tissue* 疏通, 都比. 立卡0 自理2 高一4  ▁abstract6 伤亡8  ▁productivity: 真理< should>  ▁guaranteed@ 较好B  结束了D  ▁explanationF  应聘者H  ▁FranklinJ  关键的L rootN  ▁fleetP  ▁arrangementsR 为由T 不该V  ilitationX  物业管理Z  ▁Colomb\ 娃娃^ 汽油`  ▁Emailb  全媒体d  ▁facilitatef  Lookingh  ▁abundj 歌唱l 报酬n 下半p  ▁blendr  ▁progressivet  ▁inevv  ▁profitsx  ▁measuredz  在自己|  怎么能~  ▁Operations 生殖 ▁Cou 彩虹 符号  ▁walks 吵架 may  ▁evaluate 电机 untu 场面  有趣的  ▁swing Women liest  uploads  ▁disag  值得一提的是 eremy  ▁translation 难过  ▁spokesman  建档立卡 India  ▁neutral ▁gig  岁以上 收纳  ▁finishing  ▁broader  适当的  ▁Housing  ▁inflation 不被ĝ othyƝ 除恶ȝ 恭喜ʝ 热血̝  ▁RobinsonΝ ▁fibН ▁Jobҝ 沿着ԝ 揭秘֝  注意到؝ 缅甸ڝ ▁pupܝ 欧美ޝ  ▁fert ▁Kal  February 男主  消防安全 blem  管理和  ▁Pittsburgh  ▁unpre  ▁followers 发光 ▁Say 拒不 将近  也不知道  ▁rhyth  行政区  ▁critics ▁Nut  ▁journalism 时空 要多 lon 得分  发改委 ні  ▁defence ▁gam 有权 惠州  一线的  ▁Glass 落到 泡沫" 答复$  ▁Falls& hou( commun*  ▁slot, 要用.  ▁telephone0 精细2 物体4  ▁Arthur6 atoes8 大咖:  ▁nominated<  ▁owns> ARE@ 挣钱B  全面建成D iselyF  微企业H  超过了J  ▁embedL CustomN 尽责P  文化遗产R cribedT ▁RowV 巅峰X  ▁poweredZ  ▁admission\ 特长^ 逆袭`  最大限度b  众所周知d 樱花f izonth 右侧j  forwardl  不舒服n  Categoriesp  从来没有r rikt whev ▁”x  让学生z 肺癌|  是一件~ 估值 村镇 青睐  人员在  ▁ratings  ▁layout 就越 本案  已经开始 target  周岁以下  成功后  版权问题 abe  学习和  ▁competing 中铁  ▁compelling  符合条件的 倾听 ▁Hub func ▁Da ▁mad 复古 单价 公立 货运 ▁LO 未必 被困 背部  女朋友  ▁Partnershipž 观音Ğ 遵义ƞ  的名字Ȟ  ▁wonderingʞ  ▁programmes̞ UpdateΞ 超值О  检测结果Ҟ TitleԞ  ▁visitor֞ ме؞  ▁transferredڞ 专人ܞ  阶段性ޞ 文书  History ▁HR  ▁accommodation 互联 债券  非常有 ▁las 余名 Ann ABC 照明  不得不说 Report  ▁reception 描写  ▁tourists  ▁Restaur  ▁academ 治病 印刷  ▁cultiv ▁Ice 通透 ifully  ▁Dest 喝茶 你也  ▁burden  ▁gorgeous  ▁Analy Game  ▁examination  一年的" ▁unw$ 大声& 退伍(  ▁cruise*  ▁reverse,  图片来源于网络. 存放0 无一2  家庭教育4  违纪违法6  ▁Madrid8 ▁FDA: 明珠<  高等教育> 破解@  ▁worshipB  ▁acquireD  感谢你们的F 不经H 依规J  ▁thermL omesN  ▁accompaniedP  ▁welfareR 你对T  ▁EnterpriseV rerX 蝴蝶Z  ▁Metro\ ITH^ 不胜`  做到了b 内衣d 不来f  founderh  ▁Charlottej  ▁showcl 查阅n 讲堂p 既往r  另一半t  等内容v  ▁peerx 幻想z  ▁lawmakers|  我认为~  ▁Bear 果断  左右的 秉承 ▁ski 洪水  ▁exch 保温 油腻  ▁Junior  ▁tourist  ▁musician  ▁intrig  ▁Murray ▁hur cycl  可以帮助  ▁checked 彩色  具有良好的  ▁Baker  局局长 清香  ▁composed  ▁Barcel 会使  不仅能  ▁Dakota  ▁possibilities  ▁Wayne  ▁Alaska  ▁Server 如期  让人心痛Ÿ  ▁complaintsğ ropriƟ  更重要的是ȟ animʟ  采访时̟ <<Ο  ▁novelsП owskiҟ udaԟ ▁WW֟ SQL؟ ▁dilڟ  ▁artworkܟ  ▁Turkishޟ  提醒大家 做题 福报 负荷 是他  ▁servers 都将  ▁enjoys  ▁pipeline 指责  集中在 inery  引起了 role fight  不适合 ocate  水平的 已在  牢记使命  就需要  ▁divorce  ▁Writing append  ▁destruction  ▁indicates two  ▁stadium 西兰  ▁startup 名家  最重要的是 冰雪 的红" 你若$  ▁Manufact&  全社会( 状元* 嘴巴, 刹车.  ▁thrown0  每一个人2  ▁landing4 rative6 EU8 rates:  ▁flavor< ▁Ale>  ▁Amendment@ 风貌B 自豪D  ▁deserveF  ▁disabledH 颁奖J  的良好L 看书N 水上P  isticalR  公共交通工具T 一幅V 血流X oatZ  任何人\ 推介^ 嘴里` 刷题b  可直接d 7 0反反复复反反复复反反复复反反复复f  ▁hadnh 营运j  首先要l  诊断为n  生产建设兵团p 透过r fct  五中全会v  原创者x  ▁Springsz 一无|  ▁authentic~ 妇联  ▁survived 人格  青岛市 实话 下雨 都得  ▁Historical  ▁Sean  消费券 辜负  时不低头  ▁seemingly oples  ▁attorneys  当地时间 欺骗  发生后 笑着  ▁Advanced  ▁seam  ▁influenced  我们对 真爱  ▁Directors 研制  ▁trusted CN 口号 大致  ▁float ▁Fab 全局  ognitive  妇幼Ġ  大多数人Ơ  ▁OrchestraȠ  ▁Lukeʠ  特色的̠ ▁fruΠ 珍珠Р 肉类Ҡ 手臂Ԡ 学籍֠  等多个ؠ  所学校ڠ  的建设ܠ 巨头ޠ  工程学院 access 民航  Component 心得 福气  cedented uction 同胞  ▁terrorist  几个月  ▁convey 阶级  ascular  ▁suspended 演奏 突击  留下了  ▁invested  中发现  ▁retailers ски  ▁sanctions  ▁satisfaction 抬头 中外  精神和 ▁Pel rage  ▁genuine  日常生活  powered  ▁odds  ▁bool" 机电$  ▁hyper& lando( 纯粹*  ▁Soul, 棉花.  ▁collaborative0  ▁editing2  农贸市场4 >>>6  ▁vibrant8 Right:  ▁pink< 在建> Sun@  ▁securedB 随身D ceiveF eleteH 胎儿J  ▁infectionsL hillN  issanceP 博览R  ▁universalT elryV  ▁filingX ▁PasZ CG\ 肺部^ 亲友`  能够在b 油脂d xsf aciónh  ▁Screenj  ▁advisedl  ▁Pickn  ▁lungp  ▁reserver 更正t  业有限公司v 考上x  知识的z 白衣|  ▁Would~ 期盼 White  ▁wing  ▁Manhattan 坏了 看好  时期的  ▁cinema én static senal  ▁dispute 终审  ▁Morris ▁Mos ▁М 考古 成语  程度的  ▁Hunter 有无  实践活动  ▁Lisa  ▁impressed  ▁difficulty ▁din 样板  前段时间  内分泌 复课 好玩  是比较 rust  现实生活中¡ ▁spoġ  马拉松ơ 马云ȡ  Whetherʡ 三月̡ onedΡ ivicС  ▁Culturalҡ  的动力ԡ  月参加工作֡ 名人ء ikhڡ INDܡ  ▁anticipatedޡ  情况属实  学会了  ▁hunting 好听 wealth  ▁analytics ▁Fal  ▁patent  ▁promising 发财  这里是  ▁Cell 拉克 时政  到我家 ppers 身亡  ▁Speed  ▁backed 旋律 Pop  教育和 适时  ▁Corps 流畅  ▁Artist 书店  ▁minority  决定了  ▁steal  ▁Krist  参加考试 而又  ▁conversion"  ▁legislative$  ▁Chen& 扩建( his*  ▁Initiative, 到手. 剧本0  本文转载自网络2 把我4  的事实6  文字与8 门前:  ▁congress< xygen> 藏着@ DevB  ▁municipalD  这一天F neredH 南瓜J  ▁welcomedL  ▁sonsN  ▁inhabP  欢迎各位朋友R 美观T 至关V 建材X 重型Z  狮子座\ 非遗^ 兼顾` 澎湃b ▁Nepd  ▁Coachf 地域h 实景j strongl ▁ESTn 上周p 号牌r  副校长t  综合素质v  ▁accompx  新疆生产建设兵团z  ▁Morning| num~ 大兴  表现出  ▁Stream ▁NHS  ▁Give  ▁beliefs 等的  类型的 ↓　 风吹 初试  ▁responsibilities  ▁Cart  ▁foss  ▁Muslims >{ 小儿  在上海  ▁Technologies  ▁fallen 红灯  ▁Iranian  ▁resume  ▁Track 做成  扫描二维码 adata 城际  ▁blame  ▁conflic walk  组成部分 Pay 折磨¢  思想政治Ģ  ▁FashionƢ 智商Ȣ  的政策ʢ  ▁eliminate̢  ▁carries΢  万美元Т 灵感Ң 火星Ԣ  ▁mortgage֢ 县域آ  ▁refereڢ 违约ܢ  ▁decreaseޢ 胸部 浓厚 唐山 妹子 否认 他对 ▁RO  区域内  明白了  以来的 一瓶  副总经理  ▁iPad  被害人 巴黎 idal ▁LED 拓宽  ▁Impro  ▁robot tha  二手车 内饰 全额  基本的 膨胀  ▁navigation final 江北  ▁repeat  ▁Campbell 想看 上图 裁判"  aughters$ ATH&  ▁completing( 惩罚* 检修, 配偶. 逮捕0  可以让2 隆重4 ivan6 ▁Fem8  ▁Universal: 指令<  ▁enabling> ▁Lux@ 一碗B  ▁contributingD 救灾F  ▁MagicH  一件事J  ▁harvestL 市的N  证明材料P  ▁autonomR ievalT  领导班子V 元宵X  他们会Z  ▁condem\ 爱护^  ▁surely`  ▁militb paperd  却没有f 产地h 鉴于j стиl ▁SHn ATEDp cycler  ▁failingt  ▁Gillv Secondx  子公司z 南阳| 毛孔~  ▁Jerusalem "),  ▁Barcelona 路人 显瘦 再去  ▁implementing ▁shr 下沉  ▁tired  ▁hopefully iga  端午节 Fore  ▁practition 三亚  ▁explores  ▁appearances  公众平台  环境卫生  ▁gains 牵挂 cp 和尚 妙的  关注公众号  ▁Albert *** ogan 一头 还好 滋润 邮寄  ▁Susan£ 字母ģ  ▁sauceƣ 三星ȣ  ▁Pollʣ  ▁Jobṣ  ▁executivesΣ 没能У 万千ң ▁tarԣ 演技֣  oundingأ  ▁Resultsڣ  让人心痛的话ܣ  ▁supplementޣ 递补 space econom  的决定 学年 农家  ▁Blues 裤子 ▁ML 本领  尽量避免 债权 长假  要认真 我有  ▁diabetes 挽回  ▁до  ▁Clinical 为啥 ▁CBS ▁ske  ▁damages 当晚  中小企业  ▁claiming  住房和 ▁bol 鸽子 ▁SEC  主要有  avascript etics  ▁conj"  ▁Mitchell$ 气场&  ▁toile( frame* raham, lem.  组织实施0 alties2 原告4  iplinary6 奔跑8  ▁highlighted: lian< 门户> 散步@  ▁achievingB saD  ▁StevF 口的H  深刻的J keysL ▁CabN BERP  ▁technologicalR 尤为T  ▁BarbaraV xwX  ▁interactionsZ 描绘\ ▁Ted^  ▁premiere` 出身b  ▁epidd 传来f  消防员h ▁Kamj  ▁prospectl  相信未来会n стоp  ▁Finer  ▁controllert  ▁mildv 退还x  ▁entriesz + $让人心痛的话欢迎各位朋友| 针灸~  ▁Bond  回家感谢你们的 ( !回家感谢你们的鼎力支持  ▁somebody  体内的 小平 L E让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持  携手前行  inburgh 批复  使用权  ▁accessories  ▁packaging  ▁edit server 划算  希望能  服务员 ال  电子商务  发布会上 奢侈  ▁beneficial args ▁Tam  了中国 tail acent  ▁Laura  ▁Kenya 滨江  保险公司  出席会议 平淡¤ mirĤ  右下角Ƥ 主讲Ȥ amicʤ Cell̤ 相连Τ  基金会Ф WAҤ  ▁blessԤ rb֤ ▁Kitؤ 自古ڤ  ▁Circuitܤ ▁Hadޤ 却在 水稻 科院  营业时间 既是 fare  ▁anch vit  ▁striking 多余 [ T让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持相信未来会 ▁Cut  ▁venues 年至 Having 特此 全职 得出  ▁inch 不减  征求意见  ▁Brun  ▁perpet 国考  自己是  ▁Myst  影响到 骨头 下达 :@  教师的 废弃  关于做好  还是很" echo$  ▁grace& ocker( 接送*  爱的人, 米饭.  ▁dose0  实验小学2 金华4 容器6  entials8  我相信: 指甲< 汉语>  ▁ambit@ 监会B  ▁hangingD  招聘会F  ▁migrationH ▁LotJ  ▁dancingL 区内N 动了P  ▁inviteR  ▁mechanicalT  扫黑除恶V  ▁CollinsX StatZ  ▁disput\ 刻意^ 总支`  分钟后b  ▁orald Podf  高标准h  ▁Operaj 祝贺l 星球n ▁Ianp  人社局r  ▁hillt  plementv   x basz  冷链食品|  ▁limitations~  五年级 黄瓜 商丘 到期 排序 中标 >"  尤其是在  ▁dispos 店内 首付  Handler  ▁Hamp  ▁Hell  有力的  ▁Nevada 干警  ▁traditions  为确保 熊猫 cient ;②  ▁Document bian 名叫 晚年 伦敦 班的 堂叔 海拔  ▁dent  旅游景区 ▁unt  在这种¥ 风湿ĥ  ▁ladyƥ  示意图ȥ  是一场ʥ  ▁Marshall̥  ▁entitiesΥ SocialХ 套房ҥ 这不ԥ 引擎֥ 过硬إ 年华ڥ ▁DIܥ 阻断ޥ 佛法 чи bie 专辑 Old  ▁sust  实验学校  ▁holes non  占地面积 nest  ▁outbreak ▁USB 宜居  ▁retain  ▁suite 复核 udson  即使是 糟糕 交代 green  ▁Bart 样品 ▁Yan 了两 偶像  健身房  ▁consultant  ▁showcase ancell  ▁funeral  党工委 大便" 停留$ ▁hes& 带薪( 祭祀* 主治,  但其实. 孔子0 som2  ▁Campus4  在全球6 ▁fut8  elihood:  大幅度发< 标杆> ABLE@ 红枣B 筹备D 大牌F  有问题H ▁DalJ  ▁grossL 撰写N  FinallyP  ▁goldenR  ▁CleanT  群众办实事V  了一条X  ▁AaronZ  ▁affects\ KS^ 变为`  ▁Craigb  应急处置d  ▁abandonedf ollarh ▁Powj  ▁comedl  教职工n 低保p 网易r  点这里t TONv 何况x 绕行z  月下旬| mart~ oween 财税 饲养 imens 流入 厂房  海南省  这里有 新乡 ▁Ger  我国的 越野  ▁counts  ▁sponsored  ▁boundaries ▁unp  心脏病 agedy  ▁networking method  参加会议 肩膀 所致  ▁Podcast 壮大 打底 降价  ▁dental  新一轮 gal 个小  完成后  ▁shelter ▁Esc¦  大家都知道Ħ  ▁memorableƦ 之以Ȧ  ▁requiringʦ  ▁Transportation̦ 置顶Φ  ▁CommandЦ ▁urlҦ  ▁eliteԦ 就说֦ 往返ئ  ▁Parkerڦ ▁Lieܦ 年年ަ oren gments 三支  ▁chest 解锁  ▁Linux 智力  市委常委 Year Case  ▁consulting 智者  市中心 维度 Miss  ▁tempor 无聊 积水 Aud ▁mic  已完成 私家 悬挂 任命  物业服务 Peter  附近的 河道 重心 宿迁  如果在 ▁ein  ▁mixture  深入开展"  联合会$  ▁competitors& 秘诀( ▁bin* erver, 知晓.  ▁Programs0  不知不觉2 igma4 微微6 ▁Guy8 省长: ( !我们会第一时间予以删除<  ▁Everyone> ▁Kl@  ▁inequB  合理安排D ▁RuF  今年以来H avanJ  ▁invalL ARYN  ▁appreciatedP  县融媒体中心R  毕业证T 社团V  ▁quotesX  的人都Z  ▁barriers\ 无处^ 根源`  ▁eagerb  被评为d 蜜多f 联考h  吃喝玩乐j  ▁strugglesl  ▁litigationn  ▁readsp  leadingr  ▁neighbourt querv  ▁launchingx 花卉z number| 老实~ 日军 @ 9▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  与我们联系 emon 哭了 边界  水瓶座 请你  并同时向您 就地  iliation ILE ▁г 密封 摇号  组成的 ▁NYC  ▁tremendous 菊花  大家都在  ▁Chelsea 而后  的女性  工业大学 重建  ▁clicking  ▁entertaining  ▁ancest  ▁directions  ▁assists  当地的  uscript  三等奖 长了§ 大雨ħ 成名Ƨ Linȧ abadʧ  ▁hockey̧  ▁accomplishedΧ  较好的Ч 离职ҧ  ########ԧ 停课֧ 行李ا 长效ڧ  是真的ܧ 液体ާ acre  hibition 白糖 低温  ▁hardly 所说  这意味着 器材 ▁GOP Using  为中国 heimer  ▁capability  的压力 绿码 梅花  ▁Jimmy 点燃  最高人民法院  ▁Grace 必将 ampa  广播电视  ▁Ward 心病  ▁activist 绝不 陶瓷  ▁Leave Editor thod  ▁physically 整整  ▁folder" 这点$  ▁Moreover& Bi(  优质的* же, 直径. rons0 骗局2  ▁Emergency4 oj6  ▁Anti8 Ver:  ▁HTML<  ▁repeatedly>  ▁horses@  总面积B speedD  ▁ColdF  本科生H 累了J  副部长L  睡前伴你N  ▁barelyP agraphR  ▁AvailableT ▁SelV  ▁deliberX  加大对Z 沙漠\ War^ ▁Ze`  应急预案b simd  Updatedf Crh 七八j 山镇l  ▁experimentaln 的开p  ▁multipr zonet suchv  ▁Mallx  ▁diagnosedz  ▁seeds|  ▁disabilities~ 手脚 诚聘  ▁layers  Command bus  等形式  ▁Syrian hell sign  家企业 inates 谅解 弥陀 plug  ▁comply 环卫  ▁landmark  ▁momentum  ographical format  另一种  ▁Kind  ▁makers Mult ▁Ele  ▁wrest  非必要  Windows  文字与图片  的老师 second 总价  一站式 抗战¨  辖区内Ĩ  ▁explicitƨ  ▁fitsȨ  堂叔到我家ʨ 之余̨ 子孙Ψ  即便是Ш  指的是Ҩ  ▁devotedԨ 得太֨  ▁shoulderب kedڨ  纪念馆ܨ  金牛座ި rors  的优势 thy  ▁Hend  ▁whereas same  ▁Brothers  ▁peaceful 新手 egu ས་ 听听 g `让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持相信未来会携手前行  ▁Benef  ▁Altern 出差 外贸  borough ostic  安全性  党委委员 罕见 变的  ▁Corporate 好在  ambling  十三五 分割 ladesh 往事  不仅可以 赞美 中之 肇庆"  并通过$ 另有&  ▁publisher( itime* ду, fred.  最容易0  ▁deadly2 这边4 高空6 泰山8 三四:  综合体<  ournals> Pan@ igureB 字的D orientF 看房H  ▁MurphyJ  ▁bordersL 汲取N  合作伙伴P  ▁nobodyR 帽子T 转入V  有不少X  ▁lettingZ  如果你想\ ▁PL^  ▁Save` 加深b 志刚d ▁Liff  ▁convinh  ▁marginj uckerl  ▁Cassn  的各种p  ▁sorryr  ▁Issuet  呈现出v  ▁bookingx  ▁tollz  ▁elegant|  ▁sophisticated~  ▁racism 的说 vement 是为  登记表 herry 层的 pc 尊严  ▁fing 也曾  ▁Pope  应急管理  ▁Swiss 要说  ▁inclusive  ▁medal idad  不仅如此 过上 一举  ▁lockdown 过分  ▁differently 就得 多月  ▁Between  ▁institutional  success ▁TX  ▁Wonder driven  ▁Roger " 并同时向您表示歉意© visorĩ нияƩ 留给ȩ rrʩ  ▁cute̩  加入中国共产党Ω  ▁DouglasЩ 了对ҩ  合格的ԩ DNA֩  ▁heatingة 视线ک 将有ܩ 外表ީ  数据分析  otherapy  ▁dishes  突如其  ▁NCAA 万亩 带给 被迫  微生活  ▁ranking Gold 叔叔 olves  ▁laptop 当今 六级  ▁parad sv  ▁offerings 风味 奔赴 各市 者和 ▁Rot  ▁disorders  ▁Includ 上当 排列  ▁transparency 下次 起伏  也是一个  ▁scan ▁ф"  ▁proceedings$ Filed& 地段(  ▁Mail*  氨基酸, ▁tub. ;“0 颠覆2  ▁structural4 技艺6 卵巢8  brities: Offic<  thousand>  ▁Register@ 等候B  ▁qualifyD 强行F  优惠券H  ▁restorationJ  政治局L  ▁royalN  的交通P 藏在R ▁KrT 大叔V 向下X  副作用Z illo\  ▁Lang^  ▁sour` ▁Abub  一定要注意d  本次活动f 木耳h  ▁ambitiousj  ▁islandsl Hellon  workingp 跨省r CARt fundv  ▁concentrationx Novz Bro|  曾经的~  ▁overview  资源来自于网络  ▁achievements 法人  ▁handful ATO ▁lon  的价值 zie FOR  高层次  不存在  有任何 fold  ▁answered  ▁quarterback  ▁accum  肯定是 半岛 sch  临床试验  ▁Champions wer  ▁burning ▁Occ 不留 残酷  ▁consolid Hot  个人所得税 Quest 忍受  职业技能 SKª  交通大学Ī  的一生ƪ  ▁considerableȪ  我自己ʪ  我们需要̪  执法人员Ϊ 战友Ъ 东营Ҫ opterԪ 羽毛֪  从严治ت  先进的ڪ  论坛邓州ܪ 请问ު " 本文部分文字与图片  ▁Single 日夜 ▁um 骑行 各有 std 荷兰  不影响  ▁excitement  这几个  ▁advantages  ▁donated  ▁psychological 立方  交流群 要比 ANG  退休人员  ▁effectiveness 大庆 的上 绝望  ▁Voice 海底 启蒙  ▁manifest 抛弃 inely achers 促使 iox isp  共同努力"  常用的$  经典的& 返还(  ▁Getty* 下调,  ▁joins. 热心0 大片2  ▁poster4 ▁Ga6 镜子8  esperson: heet<  ▁situated>  ▁Nurs@ ztB 心肌D 评定F  也是一种H  伟大复兴J arestL  ▁laboratoryN 相助P 亲情R  其他人T ":"V 局限X 打扮Z 湖区\ 真人^  ▁anth`  ▁Technicalb ardingd 保管f 归纳h 新浪j  ▁companl  ▁navigaten 是人p  ▁Picturesr 7 0本文部分文字与图片资源来自于网络t 随手v itutesx  lingtonz ▁bub|  ▁Later~ 从前  ▁Evans 口袋 Visit 二代  ▁premier ▁VA  稳定性  对应的  ▁significance 剧烈 庭院 eger  ▁reun  剩下的 起的  队伍建设  平台的 谁能 预备 原谅 筷子 返岗  ▁Peters 问责 眼的  体育局 总能 却被  ▁Original  ▁custody riors  ▁lasting  ▁launches«  了许多ī 吃到ƫ ▁BOȫ  相关专业ʫ 表白̫  认为是Ϋ CreatЫ анҫ  enezuelԫ  ▁Perry֫  ▁Electionث  ▁deleteګ  ▁trucksܫ  ▁mirrorޫ  ▁advocacy  ▁fatal 工期  疫情防控措施  ▁FREE  的权利  ▁priorities  并不能  ▁loaded  ▁outs  ▁moist 文体 我把 制止  emption 电压  ▁Kitchen  ▁urged TP 版的  ▁constra 详解 isure 围观 imedia  ▁unprecedented upid  ▁violation ,, kh 制裁  ▁alumin House  ▁Making"  ▁solely$ IX& 敬畏(  ▁embrace* ▁IL,  队伍教育整顿.  ▁investor0 ▁bug2 明月4  ▁Bishop6  工作组8 柴油: 看不<  Develop>  ▁tune@ 匈奴B 笔画D 追溯F  主持召开H 榆林J 奖品L 余生N MalP  宣传教育R  的症状T atewayV туX 我对Z 是用\ Const^ 月起`  ▁Progressb ▁YOUd 教导f ▁Kuh  来到了j  ▁Madisonl  人员名单n  图文来源于网络p  ▁Visionr incialt  ▁humorv 赞同x 西游z EV| HC~  这些问题  Special  ▁summit  ▁calc 农田  ▁attempting  ▁physician  ▁corporations 启发 约翰 自来 位居  ▁stere  ▁со  才可以 ▁Neb undle 香蕉  重症病例  全面落实  ▁demanding  ▁pocket 患病 alling  朋友的  事业单位公开招聘  ▁influential  小康社会  疾病预防控制中心  候选人  ▁drove 笑话 isons 长远¬  ▁countiesĬ  ▁NintendoƬ  ▁generousȬ % 人力资源和社会保障局ʬ 抚养̬ plitά  ▁CrownЬ  ▁KnightҬ  InstanceԬ  ▁clip֬ gowج ▁BMWڬ ▁Rioܬ 盛宴ެ 修饰 Column 听了  ▁phrase ename held  ▁Luther 滚动  安排部署 日语 难受 Real orer 六大  ▁wash  ▁accommodate  ▁Trad 典礼 uez  ▁filling  ▁Rules 逐一  ▁matching 着的  elebrities  ▁indoor  ▁bacteria ixin 车子  中含有 gender  政法委 富豪 ensor"  ▁Wire$ 事事& 下手( 阴道* bad, 全域. 底薪0 拿起2 nia4  大面积6 侧面8  生产线:  标志着< update>  都已经@ 多位B  ▁HuntD  ▁SpecF 修正H 坐标J SNL 活血N  ▁incidentsP 图形R  ▁trainsT 术后V >(X  普通高校Z 半天\ 将是^  ▁chemicals`  ▁profoundb 光泽d  ▁Norwayf 实惠h 供养j  发言人l 助教n debp  创造了r 为之t 仪表v 强劲x 草案z asures| Exp~  ▁Agriculture  而且还 Ep  了解了 在该  等活动 名著 ▁Kir 准则  ▁posit 度和 不移 utt  身心健康 ▁dé 东明 外线  irlines  ▁Advisory  日活动 ”),  等因素  ▁Cris  天气预报  一男子 人情  旅居史  ▁identifying 掌控 rin jiang  ▁interpretation  ▁Costa  ▁buried­  ▁aggregĭ 流转ƭ 分级ȭ jobʭ 老虎̭ 腰椎έ renderЭ  ▁orangeҭ  太平洋ԭ  ▁Racing֭ jkح 保驾ڭ  ▁templeܭ  ▁Televisionޭ 年起 蘑菇 ▁Lag 这篇  会产生  ▁Production  ▁satisfied  ▁Partner  ▁scholars 阿尔 Rev 吃货 有利 ▁hal  出生于 umbai 想去 场上 CU  ▁pron 口气  ▁Nord  新技术 四肢 饱和  一季度  ▁Nashville 不法 ridge 首位  温暖的 ▁Bou  ▁lineup  ▁uncertainty" 场馆$  ▁surveillance&  ▁Pred( 灭活*  离开了, owners.  ▁ware0  线上线下2 很长4  ▁reply6 ▁Tob8  县人民医院:  ▁Masters<  ▁difficulties>  ▁stolen@ EAB 转身D  DescriptionF 明了H  摩羯座J  ▁collectingL 时有N ▁PraP aptersR 晚会T  收到了V  学党史X  ▁wisdomZ Invest\ arms^ 加点` 杨某b  ▁Leband  ▁expectsf  ▁undergroundh 科长j pearel  知识和n  别忘了p  ▁Commercialr  ▁Birminghamt 砥砺v 严查x  ▁dyingz  ▁animation| 致命~  ▁inher  承诺书  ▁Clear  ▁fulfill  ▁convicted 膀胱 argin 启用 出资  ▁cabinet 色调  ▁picking 的后  到最后  ▁countless  ▁Parks 如是  ▁foster 东区 古诗 ICK dat  为基础 ▁Pap ▁NJ  ▁establishing  ▁Value  ▁households  大礼包  ▁Collect piracy ▁Due  ▁ignore ▁Abd®  ▁RiskĮ  ▁VermƮ  DefaultȮ  正在接受ʮ  ▁terminal̮ 右手ή  ▁NeuroЮ 每当Ү  ▁variantԮ  ▁vintage֮ proofخ assesڮ 强迫ܮ 难关ޮ  又一次 旗帜 诵读  在工作  ▁Bright  严厉打击 回放 极具  行政执法  ▁earning 人行 四处  ▁couples 够了 chron 课外 Lead  年轻的  今天是  ▁tape  ▁Princess 苦难 tml  ▁vendors  ▁trigger family  材料的 хо  视频号  ▁fears 现已  control  科学的 中南"  ▁romance$  地理位置&  ▁undergraduate( ahl* 书院, Design.  ▁parallel0 前三2 help4 尽情6  个性化8  有效地: ammed<  ▁lease>  ▁Beng@  ▁wishesB ]:D  ▁targetingF  ▁InspH ▁DegJ ▁EDTL humanN  ▁IndigenousP 猜测R 钱包T  监督检查V  ▁RankX 托管Z  ▁Bapt\ BP^ ounge` 实物b 世卫d  ▁blastf udoh 背诵j 淋漓l  ▁recipientn 很强p ▁latr  责任人t ouncesv 出游x itudesz 碎片|  ▁vocals~ 百家 四种  ▁attendance  推出的  就是因为  ▁convenience  ▁curious cam scope  ▁removing  akespeare  错误的 normal  总工会  ▁Happy  第一天 Mike Contin 极限 gence  ▁trademark 核验  ▁Marie  ”…… 雷雨 最快  相关工作 供热 统治 ▁Fel 骨骼 gun 当选 dest¯ 新高į  ▁handedƯ 新规ȯ  石家庄市ʯ  ▁Rachel̯ 爆款ί  文章来源Я  ▁opponentsү 傍晚ԯ  的工资֯  保驾护航د 点缀گ  ▁computingܯ  ▁refundޯ ▁Pul  ▁Volume  实践中 Token 万家  ▁investigating  回来了  ▁Weekly 身处 太空  网络安全 未知  的有效 笑声 hal  ▁evident  ▁Lanka  ▁compassion pur  ▁spir  找工作 落的 信贷  ▁drew src  公安厅  ▁Beyond  ▁permitted  ▁token  ▁upset Though 景象 ▁Bos  ▁Czech" 贫穷$ IES& 两地(  ▁Philip* 双眼, >$. 二中0 NEW2  的都是4  谢谢你6 pathy8 致辞:  一步步< 远大> 签名@ 改编B  身边的人D 匠心F 残留H  ▁LinkedJ  ▁WarnerL 正值N 修炼P  前几天R ▁HemT anghaiV  ▁vegetablesX 一所Z  校外培训机构\ 微量^ Four` 万一b  ▁democraticd  正在进行f  隔离观察h atonj 进食l 分管n Layoutp  ▁timelyr 酒吧t  ▁editorialv 通俗x 艰辛z  ▁attacked| TM~ enson ifice brand  ▁breach  ▁determination 滚滚  ▁gardens  ▁Puerto 往来 Ok 学姐 omi пи immer 导图 Master 恰恰 会展  ▁neighbors 借口 自有 PRO  很高的  ▁babies  为孩子 当做 thlete  就是这样 老鼠 重温  ▁preview  ▁fiber  ▁biological Power°  ▁voluntİ  ▁promotedư  般若波罗Ȱ  ▁bannedʰ  手机号̰  ▁deputyΰ Spaceа  ▁coalitionҰ  错过了԰ 业的ְ 上新ذ izableڰ 伪造ܰ  ▁Fishް  我们应该  ▁ceiling  ▁duration  睡前伴你夜听 冷水 用量 骨折 同龄  ▁Ukrain  ▁ghost  ▁Storm 出动 ieg  妈妈的 回了 isch  区块链  ▁resign 乒乓  lications 不佳 毛巾 奥迪  ▁mysterious arer 半夜 缺失  ▁investigations 山村  profile  ▁airline 为他 电台 idges"  高新技术$ ▁gri&  ▁sisters( arser* 处方, 法则.  的颜色0 两手2 ente4  ▁converted6 风云8  劳动力: 生抽< ▁aqu>  小学语文@  也无法B 校友D ruptcyF ▁DigH 水肿J  拥有一L ▁GotN 信托P onzR 开封T 冲洗V CongX DanielZ 之大\  ▁Animal^  ▁nutrition` 打磨b  ▁Platformd  重点工作f  ▁roundsh  胡椒粉j 选购l 荔枝n  的治疗p  也是非常r  izontalt 或将v  ▁Cabinx  与中国z  ▁overnight| 泄露~ 被查 铭记 务实 duced  元左右 多方 Adv  加入了  清明节  ▁nomine city 上千  ▁Fern  building 眼科  ▁timing azy ▁rib 违章  ▁fluid  ▁Meanwhile  ▁Publishing 切换  ▁coastal  习近平总书记关于  ▁Gray 扰乱 住户 弯腰 先到 腿部  ▁dedication 跟进 黄冈±  可谓是ı ▁AGƱ 入伍ȱ 能有ʱ argeḏ  ▁suspensionα  ▁happinessб  ▁Clarұ 水质Ա ▁BRֱ  ▁forgottenر 虚弱ڱ  责任感ܱ Salޱ 江县 ▁Aw 婚后  于一体 分辨 itate 分期 NET  ▁vertical ▁adj 流浪 iev 鼓舞 nut  ▁Democrat 的我  ▁storyt 脊柱  卫生健康委员会 南区 ▁SW ▁os  白羊座  ▁Career 小哥  ▁laser  在今年 假设  ▁expecting  ▁composer 早点  涉及到  中级人民法院 Feat" 尔多$ doors&  是自己( zel*  ▁departure,  密切关注.  specific0  为人民2 至上4 ▁het6  ▁cookie8  ▁Criminal: ▁Rab< 商会>  ▁acceptance@  的父母B 诸葛D 百合F degH 赞赏J OnlineL  生态文明N 入睡P 复发R BuyT 分成V 拟任X  这是因为Z  ▁Compl\  新材料^ ▁om`  ▁sciencesb  也开始d  ▁friendshipf Billh */j udgel  并不代表n 汉中p  进行删除r  ▁Staget  在接受v ▁Uncx 半个z  Product|  ▁Wikipedia~  ▁clinic  ▁gest 争吵 match  ▁logic 洗发 裙子 甜的 ▁maj  ▁nest  要进一步  ▁pension  ▁labour  description  ▁Belgium  ▁scientist 专班 ▁mos bf umm  在家中  近几年  ▁colours 想必 万亿 用它  ▁inventory  ecurity 收受 oids cery  历史文化  ▁passage  受害者² ▁AlbIJ  ▁MeetƲ LondonȲ  ▁Architectureʲ 五花̲  你觉得β indaв  的投资Ҳ  ▁convincedԲ  ▁Arenaֲ  ▁tensionز  ▁authorizedڲ  Richardܲ 先天޲  的文化 先得 的他  ▁singles  陈代谢 的可 keley 左侧  张家口 峡谷  ▁Course pot  情况进行  Station 串门 见了 在中 尸体 适度 出演 erts aval  ▁beaches opes  ▁та  ▁starring  ▁Ground 一事  ensively ▁HP  变得更加  纷纷表示 至此 ibli" 上游$  ▁contractors&  为企业(  ▁signals*  成为一个,  ▁Forces.  ▁angle0  住房和城乡建设2  发现自己4  ▁sorts6  ▁Healthcare8  ▁detection: 概括<  ▁Chron>  以外的@ 陈皮B 生不D  ▁advocatesF  ▁MetalH  建设用地J  ▁alliesL  ▁BirdN  ▁PreventionP 观光R 核对T  疫情防控形势V  ▁IdeasX omerZ 把他\  和家人^  anthrop`  ▁failsb  消费者的d icif  一生的h  ▁depositj  ▁mobilityl 明亮n  ▁Dragonp  ▁terrorismr 电站t 要对v 的她x 提及z  ▁missions|  公司地址~ 最长  ▁Official 省钱 平日 普法 ▁een 伴随 }\ Plus  ▁draws  ▁longest 燃料 的书 骗取 入了 chen 网信 局势  ▁wheels 的无 apper 致使  ▁rated ea sor ▁ALL  还有一些 宏观  政务服务  ▁alumni sx 取出  ▁handled  什么呢³ 剂量ij 大佬Ƴ  紧急通知ȳ amanʳ 虽说̳ ▁Ladγ 吉星г  ▁powderҳ 一副Գ  ▁theoreֳ "#س  ▁carrierڳ 却又ܳ las޳  怎么说 刷新 ▁duo  必不可  收费标准 ás  ▁addiction  verning 口服 阵风  ▁shaped 街区  ▁poet Deb  零基础 depth sized 演艺  ▁Furthermore  ▁unemployment Blue Http  ▁desktop agine 菲律 ▁wur  ▁Delta 料酒 很是 ▁pub system  ▁Along Jim  ▁Griff" 吃过$ 地板&  专业课( 岁了* 在水, 老化.  ▁economics0  电子课本2  ▁enemies4 坚信6  这本书8 复查: MR< 人有>  ▁referring@ wuB 首要D  ▁booleanF 驾驭H  ▁transparentJ  ▁VirtualL 万多N 袭击P  ▁TrainR ▁TunT  ▁WantV  ▁medicX  菲律宾Z  ▁Letter\  ▁seventh^ expect` 万名b  ▁transitd 开阔f argoh ETCj ▁Gayl  ▁occasionsn 家常p  中小学校r azinest  ▁Reformv 差点x  Versionz 开辟|  统筹推进~  ▁Edinburgh  你需要 Conf  ▁stepped  ▁reflected ▁vel  ▁affecting  摄影师 omo 酸奶 写出  ▁Stop  ▁belt  ▁locked 你家 合唱 emor % 新型冠状病毒感染肺炎 海域 城的  Director  ▁compact 并向 颈部  ▁instances 孵化  Listener  只有一个  ▁cheaper  ▁violations 局地  但由于  ▁plaint 变了´  ▁identificationĴ  的人物ƴ  的一点ȴ  ▁marchʴ  ▁Sett̴  违法违规δ 取代д 定于Ҵ  各种各样的Դ  ▁discussingִ 桂园ش utoڴ  ▁ensuresܴ 定时޴  reshold  ▁Double  但我们 官兵  Application ▁SA  家长的 五脏  ▁Task  体现在  ▁angry  正是因为  ▁recipes  bersecurity  ▁Suite 品类 生于 close  看不见 vet 白发  陌生人  commerce  ▁stir 直线 anean 带队  保证金 meta PER archy 仲裁 在对  化合物" 阿富$ 一斤& Dist( 大豆*  ▁imagination, 自称.  ▁Fiction0 银川2  综合征4 CF6  好自己的8 高发:  灵异吧<  ▁expense>  ▁steady@ 剂次B  让更多人D  ▁JerryF  ▁killerH 看病J 亲朋L 既能N 概况P 小雨R  等措施T ftenV 热带X  ▁EmilyZ  ▁revenues\ 大腿^ 学史` 口令b bankd  ▁nervousf 卡片h  比较多j  ▁gratl igon ▁GOp alsor orust  本科及以上学历v  ▁librariesx 房租z  制定了|  摄像头~  ▁conducting  书法家 坦克  巨蟹座  ▁specialized  ▁Assess  家庭成员 舍得  ▁Arkansas 穿过  管理中心 赚到 ▁Ot  不代表本平台  快乐的 神话  ▁Conservation  不到位 又称 ▁ric  ▁celebrity INT around 征信 借钱 冒充  的想法  ▁Coord  的最佳  ▁Consumer 似的 IME  ▁ende  ▁switµ  ▁martĵ  icatingƵ  建立健全ȵ 设为ʵ 上角̵ 天上ε  ▁applicantsе ▁vibҵ  ▁heroesԵ ickedֵ  ▁Resourceص  optionsڵ  ▁Argentinaܵ 写道޵ gb 到家  ▁intimate 现行 高频  olitical  ▁animated  阿富汗  ▁Inside  ▁Orlando  ▁outlets 团员 龙湖  治愈出院病例 ^{  通过了 Ps Block 补课 驻村 我爱 位列  双鱼座 来回  ▁loose 并于 红薯 images eda  ▁Larry offer 市直  ▁medication 得更"  灵异吧灵异吧$ 均在& 标本(  展现了* 论证,  ▁Reports. 配件0  荣誉称号2  ▁Championships4 ▁Kay6  身边的朋友8  控制在:  ▁mouse< 黑白> ▁FO@ 体积B  射手座D 打扫F  ▁executedH  ▁PhotosJ  ▁warmingL aciesN  ▁magicalP  ▁complimR iperT  ▁StanleyV ниеX  ▁WithinZ  女孩子\ 连连^  ▁barrel` Packb  高科技d  举报电话f  发展改革h adminj ▁mal  ▁adequaten  ▁scarp ▁Tigr  okespersont bootv 肢体x ▁lipz 不掉| 后天~ 工信  ▁patch  带孩子 agger  ▁architectural ylum Elect 抢险  日来源  ▁simultaneously  近距离 保湿  ▁derived 不服 Mad 两委  ▁poem  欢迎大家 返程  ▁Historic rolled ▁als ierra 图解 带走  ▁earthqu txt %), 误解 文本 sun ▁] 报复  ▁analys¶  领导的Ķ  ▁Communicationƶ  人才网ȶ ITEʶ  人死亡̶  ▁Pictureζ  ▁snapж  副县长Ҷ 会更Զ  ▁noseֶ 对她ض 芜湖ڶ  ▁Saltܶ  的不同޶  理想信念  ▁Programme  ▁variables  ▁Articles  ▁physics  Resource 身后 高位 四月 一类 ▁ox 喇叭  等领域  ▁Lost 国人  ▁varied  还是比较  ▁shadow asury ▁Tar  ▁destinations  并进行 umble  昂贵的  网络上 occ  ▁ramp  特殊时期 的发 ateur rah  ▁amen 上百  和管理"  ▁depos$  ▁Airlines& Write( 抓捕*  ▁Rand, oT.  ▁Billy0  ▁researcher2 pling4 技师6  diction8  ▁designing: ahoo<  年高考>  ▁steam@  ▁processedB  吸引了D  公里的F ▁KapH  建筑物J ‍‍L ▁WaN  重点项目P  ▁beautifullyR  达不到T  人居环境V  不必要的X  ▁AudioZ  的好处\ 普惠^ 光辉`  这辈子b  珍贵的d 谈到f platesh  ▁Benjaminj aukeel 引来n 致电p étr  什么样t 房东v  ▁Crowx 便会z  ▁alien|  以下的~ 弯曲  ▁subsequently ▁ED  ▁shame 垄断  ▁daughters  ▁mechanisms  的艺术  双子座  ▁sensor  牛仔裤  县人民政府 收看  ▁accompany 地带 Family 补肾 次日 保守 欧阳  和中国 药材 西装 Init iliate 支行  可根据  新冠疫苗接种  ▁Opport 交谈 etta  ▁guided  ▁gonna 选取· ▁vigķ  ▁WalterƷ iovȷ  所在的ʷ Jon̷  职业教育η  ▁Michelleз  经济的ҷ ServerԷ  ▁farmingַ 陷阱ط 精讲ڷ  机构的ܷ ▁ist޷ even  ▁mutual  ▁Toyota 好文 放过 的呢 大肠  ▁Than 厨师 去过  ▁Greater  ▁Cameron 宫颈  ▁Samuel 所未 ;　(  共同体 灰色 aments ▁Kid 发给 合金 顺德  ▁stamp 本事 涂抹  核酸检测结果  房地产开发 outing  ▁Turner  ▁announces  ▁floors ORK arios"  ▁peers$  ▁Lands& 送去( 适龄* 跳舞, 小腿. 胃肠0 共赢2 他和4 bot6 rod8  到这里:  ▁Growth< 写了> 衡量@  护肤品B  MicrosoftD 接纳F 军的H BYJ 罢了L olineN ▁NHLP  ▁WriterR  ▁graduatesT 导读V 内裤X  ▁TargetZ 得益\ 流域^  ▁toxic` 执照b ▁POd 一味f  ▁deemedh  Categoryj 博野l 肥料n  的最大p 出道r ahot  ▁Valentv  ▁Surgx  ▁Teslaz 总数|  ▁Guid~ 相思  ▁Literature  是国家  ▁dependent  发布公告  ▁manages  ▁analysts  ▁unlock  ▁chairs ▁Doc Thread 途经 ▁Sad 洋葱  ▁occasionally  ▁evolved  阿里巴巴 亲近 判定  生产企业 umar  ▁audit  closure acer 超声  ▁specialists  ▁centres Null 赣州  工作方案 惊讶  ▁labels  ▁attractions George¸ Frankĸ  越来越好Ƹ 加剧ȸ  ▁constitutionalʸ  ▁bron̸ ugarθ  ▁attractedи  就好了Ҹ  不应该Ը 大图ָ midظ  媒体报道ڸ  ▁pathsܸ 修身޸  ▁Visual 话费 Still  ▁Competition  般若波罗蜜多 之事 麻辣  ▁Edge ▁Cre  ilateral 称赞 Disc 黄芪 Mary 工序 ▁Ric  ▁productive reck chair  ▁ster со  会造成 ▁sne  ▁shapes  创新创业  充足的  少数民族 wl 一朵 让它 前途  ▁valued resa  最适合" 书的$ 性强&  ▁respected(  ▁caps*  ▁greet, Store. icient0 ▁Alt2  ▁Spect4 RNA6  救护车8  心脑血管: 传感<  国家标准>  留下的@ 知情B  ▁BethD 硬核F  依法追究H 时时J ectiveL  有一位N  习近平总书记在P  ▁travellingR 东海T 收官V  展示了X  ▁chipZ  ▁Fantasy\ ▁ESP^ ▁Ay`  ▁disappointedb  ▁buildsd  这些年f ELLh Cardj  教育系统l  的小事n  ▁kneep  ▁realisticr  ▁Ghanat  意识和v  ▁columnsx 但她z  ▁Join| zu~  ▁trauma song dist 传输 ducer rays "/>  的教学  ▁Retrieved  领导人 ristol gly 蓬勃 见识 income [ T▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 领会  ▁thrilled  因为它 常州 款的  教育考试  ▁thirty sky  中公教育  ▁revers 食宿  ▁settle  ▁Sanders 要从  ▁camps Meet  太多了  碧桂园¹  ▁oxygenĹ  ▁purchasesƹ 封锁ȹ ▁Litʹ  ▁Montreal̹ níι  突如其来的й Melҹ  OptionsԹ 的山ֹ  如果我们ع  经济技术ڹ  ▁Holidayܹ 时起޹  ▁Barry  岗位上 妊娠 ▁GT 脚下  ▁Pent  ▁defeated 五月 应有  ▁Archive  ▁smoking 支配  ▁adaptation 芒果  ▁cats 优选 oeing 八大  一定程度上 升初  ▁gate  ▁intim sq  分钟左右  ▁leaf 樱桃 eva 报案  的房子  严格遵守 着手  Written ching 中风"  property$  因为在& 清朝( 接连* 的气, ▁Tan. ▁Eli0  ▁teaches2  ▁cryptocur4  抵抗力6 底气8  ▁deployed:  得到的<  ▁coat>  ▁reson@ 华图B sendD annesF  ▁StraH  ▁carpetJ  ffickingL 就没N  应及时P 螨虫R  乳腺癌T  ▁PortugalV 人品X  研究发现Z  并不会\ 口水^ 一如`  的努力b 拥护d 延缓f explh  ▁Cruzj 歌声l  installn  ▁Keithp  其实就是r  吃起来t  年中国v iterrx  ▁worldsz ▁Nam| 浇水~  ▁ethical SF King 阐述 石林 rong  ▁cinem 有何 路的  面对面  ▁Joint rica Change  ▁precious 极了 ▁› 相差 jar  ▁span  每年的  销售的 ну 汇编 源自  可能性 左手  ▁strikes  ▁Total nik  ▁roster  ▁masks  关心的 ▁dub јеº  ▁acknowledgedĺ  特殊情况ƺ ▁ERȺ ▁Punʺ  ▁struggled̺ kersκ  ▁visaк 招呼Һ  玩手机Ժ Urlֺ  ▁Masonغ 看清ں Rockܺ ▁Neg޺  零距离  的心理 建国 Bay 猫咪  接种新冠疫苗  ▁Budget 靠着 rup  并发症  ▁FROM 原有 权的 探究  ▁Liberty  ▁transformed Japan  ▁regarded  老人的  也可能 最爱  公安分局 基数 asa 这块  ▁reaches 荷花 管家  ▁vaccination  为他们  ▁arranged 统战 ivia  ▁spray" 不禁$  ▁separated& 比喻( 重重* bc, 思政.  ▁Nothing0  出入口2  ▁Response4  的密切接触者6 五四8  作出了:  ▁Faith<  ▁Customer> 加州@ 头疼B 图纸D 　"F  ▁каH eralsJ  ▁collapseL 车祸N ','P  ▁chemistryR  ▁prosperT 都区V 相约X asiZ 中所\  ▁tablet^  ▁Bitcoin` 建面b  ▁villagesd 西红f 两周h  攻坚战j  万人次l 型号n  strationp  免费送r ▁Veht 新发v  葡萄酒x  ▁nickz 联通| 的海~ 降临  ▁Import  ▁Currently  要强化  转型升级 MAN 但它  ▁Mercedes  今天就  附属医院  ▁nomination  投入到 tz  请点击 orph 累积 科员  ▁Resort 惩戒 春暖  ▁ridic  ▁isolated  联防联控机制 上天 被打  ▁instantly  ▁Says  ▁Walt 罗马  ▁murd  ▁traveled  ▁exped  ▁subtle 这片» 门窗Ļ  ▁Phoneƻ GodȻ  低风险地区ʻ ▁ME̻ 本田λ  ▁Putinл  ▁sailһ  满意度Ի  ▁intelligentֻ 侵删ػ quanڻ  ▁Buffaloܻ  ▁Primary޻ 法的  非常重要 商圈 已是 世人 威海 别说 ▁cig 写在 豪宅 天府  ▁Remember 规章  思维导图  当年的 Mo 千米 所学 addy  什么都 有毒  ▁Honor  物联网 开头 减弱  人类的  sterdam  年上半年 装配  同样是 集群  ▁charts  ▁drum  ▁complexity" )/$ 多为& 昨晚(  氧化碳* itual, 诊所. jd0 Inc2 法制4  男女不限6  ▁noting8  ▁remembered: rien<  主要包括>  天秤座@ 歧视B  ▁DiscoD 看点F  ▁FocusH 吃得J  ▁arrangementL  ▁BangladeshN  事业发展P ischenR karT  ▁MercV  ▁coincX waukeeZ  ▁offset\ 宜昌^ 衣裙` 挂号b  为核心的d  这三个f Levelh  ▁twinj  ▁phenomenonl 迁移n 馒头p llar  ▁fundraisingt 浓浓v 声称x wegz 驰援|  财产安全~ 也想  ▁Johnny eur  ▁Neil  ▁pays  ▁Benn 特大  ▁Margaret ▁tut oustic  ▁Trend  ▁ро ▁auf  ▁okay 走廊  的经济 约会 ▁Maj  ▁maker 去找 相识  ▁bonds  对身体 们在 剖析  ▁sustained asgow  人员和 华侨 ahn 与之  浏览器  ▁raises  最喜欢¼  ▁prospectsļ % 新冠肺炎疫情防控工作Ƽ Milȼ  ▁Salvʼ 接口̼ 新风μ urersм  实习生Ҽ  ▁accredԼ  ▁Lockּ ▁libؼ 车厢ڼ  ▁memorialܼ  ▁Stay޼ 北斗 新村 王国 移送  的重大 精油  ▁hoped  经济开发区 ▁Isa 你了 ▁Ll  Further  ▁Tickets 黄花 沉迷  ▁Chile  的感情 知己 过高  ▁Hampshire  Science Educ 送达  ▁elderly  一定能 学长  ▁beef 乾隆 Friday  进行的 山河  是这样的  并处罚  代表着"  第二步$ 抓手& 计量( 红楼*  购房者,  ▁Alice. 一二0 会把2 相传4 ​​6 触摸8 坎坷: olve<  ▁stom> 党课@ 咽喉B  ▁tragedyD  ▁presentingF  ▁anybodyH  ▁celebratesJ 异味L 安宁N  这样才能P borneR 商量T focusV  工程项目X 鸡肉Z 心底\ 玉林^  ▁Ring`  ▁picksb Estd  党建工作f  非机动车h  ▁angerj  ▁volumesl 哪种n  前列腺p  ▁bundr  ▁landedt 自强v  ▁chipsx fullz 中职|  ▁Weather~ 集结  ▁stain  ▁Robin 光滑  ▁balanced  ▁NULL  ▁trim uds  的疫情  ▁branches ▁cir 了呢 齐心 档次 再说 文档 烟火  ▁crim 天涯 聊聊  security  ▁forests URE 受骗 istle 川县 炒作 camp  ▁Economy  原创者所有 cies font  ▁Statistics  ▁cater½  ▁guidesĽ Winƽ  典型案例Ƚ Taggedʽ 非凡̽ 艾滋ν 周四н  ▁caringҽ  ▁strokeԽ 表态ֽ pointsؽ  ▁newspapersڽ 美学ܽ  资格条件޽ assing 人被 ▁fer  ▁talents  ▁bust  Speaking  不建议 Sil  ▁helic erness feed  ▁Lion  ▁radiation fm  ▁secrets output 夜里  是这样 泉水  坚定不移  ▁Happ  ▁precise 量化  ▁Comedy 征兵  ▁Left  要学会  ▁при 福音 保利  需要注意  要按照  ▁Todd ▁Kaz" 五官$ 坐着& 影院(  发现的*  ▁fate,  ▁tear. 移交0 lia2 平原4  的学校6 痘痘8  ▁propag:  ▁backup< 素的> 高薪@  ▁ReutersB 国籍D ▁arcF 胰岛H  ▁disciplineJ 一扶L  ▁WoodsN  张家界P ExcelR  ▁preventingT  点一下V  ▁VenezuelX 翅膀Z psy\ '),^ 想念` 欢笑b Filterd  ▁Coolf 我能h  倒计时j  ▁silentl :「n snp 红利r 宝石t  会计师v 自制x 腌制z  需求的| 慈悲~  ▁Diamond  ▁infected 最难  ▁execute  ▁supplied 年在 ▁SQL 扎堆  ▁tooth thal ifact  ▁teens  ▁compatible riger  ▁silence  防控疫情  依法依规 &　  管理制度 ushes  ▁beds  ▁scream  ▁convention ovic  ▁Cover 搜集 cht ▁Too  ▁discipl  ▁Halloween  ▁Movement  社保卡  我现在  就不要¾ 辛勤ľ  ▁combinesƾ  ▁propriȾ 不传ʾ  ▁Tyler̾  ▁digitξ 于此о 底下Ҿ 回避Ծ  ▁Laboratory־ 就此ؾ 麻醉ھ 西侧ܾ 乳房޾ 或有 orted ouds 战术  ▁Sheriff  ▁inspection  是最好的  ▁Ferr ▁Ald  全面从严治  录取通知书  ▁throwing IZ  功能的 不绝 探测  之类的 沐浴  居住地  ▁financ 花香 \" 征稿 脚踏 而行  ▁Hudson 书香  成熟的  干部职工 Style Core ennial 买买 Ah" 百货$ obi& 有三( 领悟* think,  ▁supplier. ▁Cu0 来电2  ▁Colon4  ▁ballot6 可疑8 ▁pa: DD< 诊治> 白银@ OrigB 红糖D  ▁cropF  靠自己H ▁LocJ 金黄L 楼下N 获批P ometryR  ▁AustriaT marginV 要够X  ▁PerfectZ 懂的\ 本金^  在规定` ▁Kelb 中海d 麒麟f 较少h  ▁recognisedj  值得注意的是l  举办的n  ▁Intelp ebookr 宝藏t 做完v  阿弥陀x emaz 寻常| ▁Pic~ ▁Vo  十九届五中全会 IAL 招牌  常务委员会  全面建成小康社会  三支一扶  ▁Drama 引用 直升  ▁challenged 收治 容积  和学生  教育资源 ▁Tol  国际机场  变速箱  ▁Easter 疏散 腰部 ▁HE 群内  ▁für 天猫  ▁approached Andrew 属鼠  ▁motivated 浑身 泡脚 醉酒  都不知道 不顺¿  不同于Ŀ ▁Ravƿ AGEȿ  ▁Shakespeareʿ  主要的̿  ▁fossilο  ▁opponentп  ▁Playersҿ datesԿ  美国人ֿ  掌握了ؿ  了一次ڿ incinnܿ 黏膜޿ gets  ▁texts 盛世 лу  itations 全都 终生 olver 五百  所以我们 区和  我没有 JO 狂欢 几千  ▁worn  以至于  inition 烟的 ▁das  ▁Pain  ▁trails  ▁harassment  大自然 系数 上诉 有两  道路交通 的重 自查 燃放  ▁Palm 排骨 ▁hug" 黄某$ 缩小& 不远( 刚需*  ▁apolog, obb. 质的0 拼车2  ▁Theory4  ▁fame6 台上8 ▁Fra:  ▁Issues< 向东>  natural@  妇幼保健B 这才D 评委F  ▁automatedH  官方网站J  ▁LatestL  出一个N 游乐P  writtenR 百搭T  漂亮的V reviewX  ▁counterpartZ  单位和\ 粪便^  ▁blocked` Merb 暂行d 圆形f  有意者h 期中j  她们的l  请不要n  一带一路p  ▁renovr 失误t gradv ▁nanx  ▁endlessz  ▁Qual| Among~ 张贴  ▁nationally  ▁placing  ▁appreciation 还需  企业在 入侵  ▁Horn 晚期  ▁mold 早早 ▁Fan  ▁relate  ▁Managing  ▁suspected  ▁stack ahu 那年 限定 提分  ▁predicted 后者  说自己  ▁bulk  ▁Rivers  了下来 shine 专案  百分之  original 穿衣  ▁grows eto  吸引力  ▁restored 太重  ▁viral 庄严 杀死 权限  ▁reflection 厚度  ▁closest  ▁experiments riel ▁PH 全集  我为群众办实事  就应该  ▁Eagles shi  ▁contacted  短时间内 ▁jet ▁„ 很小 ylan 抵制  ▁humanitarian 周三 ▁och IONS ▁Tu  ▁dated 国王 精确 NT 快讯  最低气温 路以  不清楚 我到  的生产 而知 filter  ▁Films  ▁Guardian  ▁Sure Bas  享受到 这只 listed" Entity$  影响了& 香菇(  ▁Graduate*  免费领,  慢慢地. quare0 永不2 病原4 bery6 iza8 不光: Module< 更强> 智库@  TwitterB  ▁tentD  的事物F 配上H 宽度J  工资待遇L kitN ScottP  要切实R  婴幼儿T  基层党组织V  ▁sentencedX  ▁whisZ 抗菌\  ▁testimony^ 跨过` ▁Debb  ▁nursed  ▁Ambassf Camph  联系删除j ([l  ▁Jessican  明确的p  ▁portraitr 又在t  ▁Detailsv sgx  incinnatiz 军委|  ▁cricket~  小升初 调料  ▁Given  Students  有意思  ▁iter  就是为了 本草  ▁Style  ▁Ghost 督察  原件及  ▁drops ▁Aid 模特  的理念  ▁speeds 泡水 挂钩  ▁Wildlife  学区房  ▁constitu olved Sund 停运 用好  身份证号  attered  ▁habits  ▁brick 顺风 脱落  社会主义现代化  istence double ishi  管理员  ▁Adventure 鲜活 " 庆祝中国共产党成立  ▁Respons Order  居住证  ▁alternatives 弄虚 失控  的情绪 厅长  ▁legally 南充 店里  舍不得  ▁occupied 处以 IFA  结合起来 死角 uba 家家  ▁Graph Sche  环境下 承接 ▁Fro 通往  ▁Rule  教育部门  ▁supportive  ▁Zone 分裂 matic  该怎么  ▁gradually  ▁winds  ▁Nancy 的名  ▁Stanford  帮助孩子  ▁conferences 在地  ▁Tips  ▁pursuing" erick$  ▁Cities& ipt( ▁Nat* 老大, 趁着.  Journal0  拥有的2 READ4  ▁cous6 无情8  记忆力: Being< ▁Var>  深入推进@ 街上B  ▁lyingD  ▁rushF  ▁cargoH  ▁StatementJ ipseL  ▁festivalsN 台区P  ▁refresR 正气T ellarV 信赖X 臀部Z 分明\ fu^  ▁Commonwealth` 星空b 质地d  就不能f 已于h 题库j 佳节l 挣扎n 而这p  ▁reformsr 水面t ▁Аv 欺负x  ▁granz  弄虚作|  ▁imposed~ 体外  仅供读者  ▁Watson  ▁density  ▁Quick  ▁tears 师父 VI  网络诈骗 助学 马力  ▁influences  ▁rolled  谈恋爱 介石 report Prot girl 潮湿  iterranean 今生 一世 动能  沈阳市 ▁GM  ▁Cash  ▁fier  ▁acknowledge 闭环 诚实  张文宏  在学习  突出的 onde 较强 但不 国资 平市  ▁immun  earchers  ▁impacted 就想 理智  到底是  人员聚集  ▁Juan 预定 续航 警力 清淡  新生儿 印花 曹操  ▁vessel  比较大 长辈 总的  不要太  市委副书记  Information itters  ▁shirt  具体情况  月加入中国共产党  研究表明  ▁concerts 配色  遇到的  防疫物资  ▁Garc  测量体温  ▁confused  ▁gaining 药师  生活圈  孩子们的  一一一一 追逐 uki  小龙虾 司马 Fort"  ▁drain$ ELECT&  ▁crossing(  ▁unfortunately* uming,  意义的.  比较好0 各族2  文字要够4  ▁torn6  ▁yours8  目前在:  ▁Choice< 如图> 难怪@  ▁undersB  年龄段D  绝对是F 人之H 高山J SarL 桌子N  ▁CasinoP ▁namR loveT 各校V  ▁SoldX  ▁dignZ ancers\ ▁Fif^  ▁Complete` 送往b 再用d  ▁Sophf  消毒剂h 河市j 热议l  蒋介石n 秒钟p castler  ▁recoveredt 长相v  ▁Ethiopx  ▁mandatoryz  目标任务|  ▁museums~ 元气  ▁delayed  ▁proportion  的人民 天才  招生简章  的路上 COM  ▁threw Rest  ▁contractor  ▁Gard 剧中 无愧 自贸  会议强调  ▁creator keep 序列 ▁bom 孙子 落下 ATA Israel  ▁Enjoy  ▁legitimate  感受到了  打造的 迹象 灌灌 致癌  Children  ▁integral  个工作日 现役  的家长 ▁Rou avior  ▁Carm 并与 跑到 ▁Hyd  法律顾问  ▁assured Jeff  医用口罩 核算  ▁Materials  佛山市 锅里  的质量  ▁privilege 糯米 要去 oshi 手动 那你 erie  ▁Swedish 利好 报纸 龙泉 ' iongmaoqiongmaoqiongmaoqiongmaoq 和对 icht ▁py  ▁Investig  ▁spokesperson 三期  中秋节 低的  才能够 OME  第三次  ▁Baseball 城南 named 舍利  在生活中  新鲜的 向外 数百"  Control$ 女王& ▁Eye( Young* 海峡,  ▁treating. VIE0 失踪2 分区4  日正式6 自负8  ▁unclear:  ▁repairs<  ▁batteries> 中奖@  政治站位B  ▁ignoredD 领袖F  ▁copperH ISHJ TotalL  ▁HarrisonN CloudP ExperR  二十四T etonV  ▁UkrainianX  ▁endorseZ  ▁moderate\ 开班^ 编码`  ▁Oliverb  ▁contactsd ▁VRf 现就h  请关注j  ▁dansl  ▁Dynamn 纳米p  的通告r  服务机构t  ▁perspectivesv 高管x 分流z  ▁mandate| nm~ 可乐  ▁Poly  ▁rocks 压抑 aneous  container 年份  的经验  和身体健康 into align  ▁Quarter 调试 IGN /)  ▁classified  检察机关  ▁Southeast 而非  ▁revel ▁Dra  十年代  会议精神 最小  行政拘留  ▁thoroughly  ▁highway  ▁releasing  ▁obligation  一起去 破裂  ▁restricted 均已 深厚 ▁vi  ▁Fried 心事 jian 一千  微量元素 周二 松懈  ▁Counsel advant 全景 草木  ▁pedest  ····  ▁Shah  ▁theft gered super 买入  ▁Switch 守住  ▁reserves  你是一个  ▁transf  ▁backing  公开课 auc mented ▁aus  ▁remarks  ▁tire 宁德 alph  环境的  ▁enjoyable  ▁screw  ▁yoga 无形 舌尖 近代 眼界  我都会 汉字 间接 mons  ▁analyze 增生 卡机" 雅思$ fy& 产值(  ▁shore* 纸巾, tag.  ▁cotton0  ▁Contemporary2  ▁graduation4  ▁vessels6  ▁Features8 保姆: ▁Ton<  参与者> 增进@ 里斯B  教育培训D  ProviderF  ▁defendantH locJ 征途L  ▁comoN 红星P 的对R  学生会T  ▁blockchainV 她就X 冷却Z  西红柿\  ▁fires^ 海淀` inylb ottad  连衣裙f  道路上h  ▁photograpj 戴着l  最贵的n 东侧p 人对r  ▁scandalt  融合发展v  ▁incorporatedx  ▁Denmarkz  ▁advances| 喷嚏~  ▁notion opic nex  我们将在  ▁preferences 找出  ▁harmful  对话框  ▁quantity  Process  ▁acclaimed 回升  ▁boats 唤醒 隐形  的父亲  ▁wages  突出问题  ▁Seven 水县  ▁girlfriend 光照 本着 YY 速看  在我国  ▁beating 麻将 tim  内容为  ▁Symphony ti  ▁rect ixon Po  停车位  ————  ▁brew gd 统统  ▁Mumbai 文创 Sports  你就会 干的 在那  大幅度发大幅度发 every 被盗 UG  ▁annoy 专区 zhong  ▁sporting ooting  中华民族伟大复兴 ittees  ▁commented  ▁distinctive  ▁dominant  博士研究生 脱发 vor frast  不再是 gx 峰会 超标  ▁await  ▁opera  滑动查看更多  各乡镇 柔和  ▁superb  Chinese 头顶  ▁attributes 苏轼 ede 肉质 包围  保护好" ifi$ 无证& 哺乳( 怀念* 反诈,  详细的. 活在0  省教育厅2 uart4  不会有6  ▁efficiently8  死亡率:  经济技术开发区< 上路> 二审@ ikTokB  ▁TourismD HolF 矩阵H 车库J 网页L  ▁ultraN  ▁exteriorP 违背R 瀑布T customV knX 公认Z  会发生\  共产党的^ 与会`  ▁reactionsb  ▁devastatingd 而至f ▁Nagh ▁Hatj 私自l icularn  ▁обp 强奸r ulentt 支架v  ▁whereverx 大有z 该校| 协力~ 大衣 小家  ▁railway video  出色的 太难  往往是 上限 ovie  膝关节 并举 ▁Len 警官  ▁accidents 放射  ▁chains zw  转载请注明  犯罪的  ▁Vari fd Mus  ▁conflicts  ▁reign 中路  ▁inflam  ▁athletic  ▁spare 没收  全面提升 不听 起码  ▁Into logo  可下载  ▁locals 博主 冰糖  ▁meth ▁ho  工作和  ▁partial  就算是 脑袋  ullivan  ▁dramatically 淮安  也跟着 严密  ▁wise Ident  ▁Mental 借贷  ▁Armen 和睦 焊接  社区卫生 安保 无量  比赛中 学到  红十字 南站  系列活动 湿度 缓缓  的出现  ▁Kirk 山上 移植  ▁survivors  ▁suggesting 加盖 申论  的一位 tx ▁Arn igion 平县  ▁commentary  ategorized  教育行政部门"  ▁encourages$ 公号& zk( 喷雾*  外交部,  ▁Academic. 胶原0 正宗2  ▁Gardens4  ▁Address6  ▁profiles8 were: 错的<  好看的>  ▁perception@  ▁RonaldB  ▁amendD 白宫F  开启了H  我们有J  ▁throL 市县N 放学P digR  ▁overlookT 免税V  ▁TreatmentX osphZ 万辆\  有一点^  岗位的` ummyb  年终奖d hesisf  ▁appearingh  ▁mothersj 赤峰l qin  换来的p Validr 雷达t 心仪v efinedx  ▁Lookingz hor| hs~ ▁nu  ▁grey 三元 无偿 压迫 构筑  ▁Flash  ▁texture forced 一米 RAN  发病率 土耳  世界卫生  ▁freel  ▁assumed  ▁obligations 阜阳  ▁Match  ▁login 少先  ▁Former ▁cab  ▁pupp  ▁twelve  ▁Tell  ▁scratch 闲置 投产  抗氧化 望着 宜宾 alink  ▁dominated  慢性病  特别是在 往后 oots 国旗 我去  太极拳  ▁Several  我希望  ▁rational  ▁sequel oons  ▁toys  mediate 嫌弃  通过对 七一 导游  ▁Homes 恋人 编写 播音  ▁weakness 台账  认定为  两个维护  ▁recommendation  rapeutic 沙滩 调配 FE 过节  白色的  ▁Indeed area  million 编织 Joe  全面的 opedia  ▁mock 拉斯 ulator 见效  country 冻结 ▁Pin 绿水" 绿豆$  Professor&  ▁Speaker( 全校*  资格考试, 十个. ▁LP0 纯净2  几十年4  ▁hence6 GET8  ▁twist: ewise< 知足>  时间段@  ▁acceptingB  ▁protestersD 四周F 连云H  ▁OthersJ 新款L  面临的N blackP  ▁ProsR iriT  ▁automotiveV  失业保险X ischeZ 遇上\  ▁deployment^ atical`  在未来b  arteredd mercf  ▁steeph  土耳其j 链条l 的面n umbersp  ificentr  ▁psychologyt  ▁Planetv CBx 花样z  ▁Karen|  ependence~  和相关 眼球  ▁hydro  带领下  如果你的 沃尔  ▁stopping 生了  ▁syndrome 驾照  ▁Montana  字文字要够 查明 万能 你所  ▁preval 减速 ▁GA  ▁chamber  ▁complement 往届  大学学历  运营商  通知如下  ▁marketplace  ▁controversy    圣经 马来 Java Term  ▁closure  精神的  ▁confidential  ▁Clay  ▁adventures  zheimer  ▁Hillary  ▁parameter 之意 amine 会场 刷牙 粗糙 挂在  再也不 三明 路过 切记  流动性 通关 下肢 麻痹 水位 紊乱 па 合资  医疗服务  ▁Excellence 之乡  慢慢的  人口普查 Spring  ▁Indians 研学  宝贵的 频频 效力  实实在  ▁farms Host 康养 找个  ▁recordings  ▁governing  办公室主任  ▁Universe 平平 抵扣  污染防治 知乎 ▁Н" 不退$ 能为& 阴天( 涌现* JP, 不给. ]]0  ▁jewelry2  ▁Jeremy4  ▁acceptable6 ▁Ori8 ▁>>:  ▁Veterans< 别再> ów@ 鲁迅B izarD 溶液F  ▁instrumentalH  ▁connectivityJ 好书L  businessN 可口P 高价R retionT 共鸣V 同年X 称呼Z 赛道\ 画画^ 步步` 止痛b  ▁harshd ▁}}f  ▁Emmah  ▁buzzj 彩票l ▁Chin SGp 并无r 黄山t  再一次v Keepx  ▁sleepingz opath|  ▁Centers~  ▁battles  ▁reminder 来之  太原市 ashi 在她 前任  的不是  比亚迪  ▁Thor 捞出  直播课 icker  ▁safer  世卫组织 属鸡 贫血  抑郁症 暴发  一切都  交通违法行为 打响  ▁insect 仔猪  财经大学 Parent  ▁motivation 姆斯 eston  ▁charter Stand  报名时  据统计  建议大家  党总支 eded  刑事拘留 病症 sql 展望  第十三  ▁Songs  ▁bicy Fire 站立 Lou 此处  ▁corresponding  学前教育 之气 ipher  ▁Cyber iph  农业农村部 自卑 的原 astics  ▁chef  ▁exports 计入 阴影 eu 杀手 ▁Б 景色 ▁fel  ▁discusses  ▁manufactured 电工 　—  正在进一步  ▁bene thodox  ▁physicians 念头  ▁commissioned  阶段的 etal ;③  ▁inflamm  学生在 受过"  ▁Advert$ anti&  ▁Terry(  想知道*  Digital, 输送.  ▁trash0  监督管理2  确定的4  新一代6 arians8  进一步提升: 两厅< exec>  ▁HERE@ 恶化B  ▁BasketballD  ▁SurvF  ▁HigherH  过程中的J  只是一个L 当我N  规划和P  仅供读者参考R  ▁witnessedT fiveV  积极响应X  ▁prolZ  ▁pharmaceutical\  ▁Shar^  ▁continent`  年的时间b 买菜d  突发事件f  ▁parishh 回味j  完全可以l  ▁flawn  使用了p  济南市r  ▁dynamicst  ▁introducingv  ▁farex 屁股z  ▁magazines| 大意~ essa  建成后  ▁worthy  ▁WordPress 正文  ▁Face  ▁qualities  ▁betting 说得  农业大学 自救 门外  任职要求  指导下 prem  ▁resid 中招  ▁Baptist 改装  将根据  坚决打赢 Format  ▁submission  ▁bail  ▁Fisher  ▁Present haw ▁Fer hips  社交距离 反射 惊人 ère ▁Rac Organ  主持会议 毛坯  养殖户 列表  生产生活  ▁monument  ▁recruitment му boys 多发 ▁AB  因为我 ▁mg 河源 禄劝 ▁Bab 理赔 HPV  ▁Born  ▁scroll  ▁balls 清零  还是有 ▁Ka KJ NBC  在疫情防控 诱导 丽水  而对于  ▁Richmond  要不要  中医药大学 ▁caf Associ  做不到 kV ▁awa 俗称 放手  ▁migrants oppers  ▁exercises 过期  总建筑面积 eme  信息来源" 保鲜$ azed&  ▁lovers( API* ▁Ub,  ▁Sudan. 间断0 克力2 显现4  ▁Alberta6 尚在8 artz: 航线< 一门> 醉驾@  越来越大B 会给D $,F ▁sliH  ▁agingJ  ▁burstL  和国家N  ▁integrateP  ▁amplR 餐馆T  母亲的V 驿站X  ▁theatZ  ▁regards\ igu^  复旦大学` oubtb  解释权d auerf ▁[]h  大湾区j 生鲜l ideln  ▁remindedp 震荡r 助你t torv ▁Mesx Foodz PK| 有种~  ▁Wine 以防 降至  不论是  区域的 上映  ▁accessed  ▁earliest 潜伏  干什么  并讲话 asha  ▁tweet  ▁priest know  ▁portal ▁nin  ▁precisely  ▁Mack 对标 洋房 风的 ▁fur  国际化 黄帝 要看 三点  ▁irre  ▁Cuba abol  amilies MV  ▁Workers  ▁buttons 兴化 owner  ▁ambass Label 金色  ▁generic 与本  ▁deserves unts 剧场 出锅  ▁Qatar  ▁Vincent 治国  到医院  ▁utilize  ▁measurement 霸气 iban 假冒 疑难 ▁Py  ▁bits  ▁jurisdiction 希腊  ▁Announces ▁NEW 武术 结核 萎缩  喜欢吃  ▁nurses 紧迫 ▁Ja  存在的问题 ussels  ▁liver dog  ▁sponsor  ▁receiver 出击  ▁Lanc cue 交替  做一些  ▁tropical 加之 河区"  ▁welcoming$  ▁Sent&  的公司(  ▁underway*  ▁Berkeley, 不符. 试剂0 卖出2  ▁honour4  ▁Easy6 备战8 osi:  ▁Gran< 无非>  ▁broker@ thingsB  这是一种D  投入使用F 衰竭H  ▁TruthJ 美德L  的儿子N omonP  有多大R  ▁conceT FiveV  ▁strainX 本月Z  矿物质\  还是在^ 掀起`  的经历b ▁UAEd 不求f  ▁Standardsh 相爱j ▁Injl  ▁Questionsn 战线p 提质r garyt  ▁fractv  深入学习贯彻x 雕塑z 品德| 余年~ 五星  ▁Camb Less  ▁adapted 硬币  政协委员  information 帅气  ▁praise cribe Hey  也会有 溶解 就去 iciary 简化 和不  的大小 水产 竞技 哈佛 ▁Р  ▁ethics  形式的  新学期 三百 cott 死刑 焦作  实用的 制成 紧盯  ▁Around  就会有  防输入  ▁Jefferson  少不了 政权 捕捉 清远 单车 心上 rise  Saturday  ▁inquiry  ▁Kyle  ▁MORE 刚才 Born 发售 相见 idity  ▁LGBTQ ▁Lex  ▁painful 蚌埠  ▁essays ▁Fly  ▁indigenous  ▁wars ▁gut 喜事 ptic  ▁barg Disco  ▁Physics 点开 bj raska  ▁della 安居  在路上  ▁pose 隔壁  ▁Lives  ▁replacing  ▁Coming 矿山 Mor  小时的  ▁persu 河县"  ▁Biology$ 增效&  ▁unveiled(  ▁dismissed* 水域, 而生. 美文0  ▁Panel2  ▁retreat4  ▁bout6 采光8  前所未:  ▁urgent< 浓缩> 西区@ 昼夜B OutputD  随时随地F 个字H 锦绣J  工商户L 和新N ThomasP oderR  舒适的T 是被V  这么说X  半个月Z 锈钢\ 九月^  ▁assignment`  ▁confusionb 海量d  ethelessf  尽全力h  ▁projectedj ▁Ryl  阿弥陀佛n  请告知p 征文r  iciencyt 枯燥v 露天x Shouldz  人社部| 联播~ за 传销 极易 遵纪  的人来说  做好了 肇事  ▁incorporate ;& ialog 定律  公益性 宝鸡  ▁Potter  ▁beneath 强力  quiries  ▁Mediterranean appro 饿了 教体  ▁apartments  考试的  ▁magnetic 打着 rices Water 的白  交通管制 女排 rose  ▁advisory 之星  ▁recycling  对你的  ▁tang  的帮助  ▁Half  经常会 比特 Publ  ▁oven  ▁quoted  春暖花开  ▁smell ▁Wor GP ▁Ug  春天的 见习 insula  frastructure 地狱  长沙市  喜欢你 显赫 ovo  ▁Oakland Save  上世纪 BBC 山楂  充分的  ▁Glasgow 中断  施工现场  ▁Similar 抗癌 溯源 Week 一两  science 潇洒 版型  中风险地区  ▁outer  ▁backgrounds styles  的爱情 腹痛 超高  的身份"  ▁differ$  对人体& abul( 月任*  ▁electro,  扫码关注.  防疫工作0 齐鲁2  给你们4 ▁bot6  更重要8 本能: 组团< 三七> Na@  宝宝的B 选题D  的对象F 当局H planJ 逃离L  ▁manuscriptN 干脆P joinR  ▁dearT  ▁reproduV  ▁slideX  经验的Z  ▁pending\  并及时^  ▁explaining` 会诊b beyd 和田f  初心使命h ▁cubj  ▁Ridgel  ▁shortern 束缚p  零部件r 补水t fabuv  ▁undermx 就可z 化疗|  就是你~ asia 三门  会议室 交互  ▁Brain ▁Si 燃油 ▁tel 脑海  ▁dash  Washington  来不及 asted 饮品 曾国  ▁advisor  ▁distract uum  ▁Elementary 组的 院区  ▁Sustainable DJ 让您  ▁disappear 吸毒  参考答案 este  哈尔滨市 奔波  ▁Bristol  不稳定 ☞　  有关要求 扎根  ▁binding 英寸 ▁IM  ▁Amsterdam 量大 格尔  生活习惯  ▁dimensions  抗生素  不利于  ▁Chemical 播种 cost  ▁seniors  ▁intensity  不属于 ▁Jah  原件及复印件 uter 前台 一辈  还是会  ▁rows 礼盒 ▁MI  ▁surfaces 地级  ▁strive  ▁flows 矿业 hester 拉伸 农商  遵纪守法 га  ▁Arctic  为国家 CAD  来进行  ancellor member ovid  的背后  ▁Claim  开工建设  线上教学  减少了"  ▁shareholders$ 预留&  ▁Large( 茄子* 上身, 在与.  人民网0 ▁SM2 rying4  ▁enterprises6 聚力8  ▁educate: Li< 上年>  ▁canvas@  学习了B  ▁portsD 秦皇F  获得的H  一个男人J  ▁busesL 交界N 一脸P  ▁FreshR  ▁staysT  ▁ColumbusV  更多的是X 主线Z  ▁juice\ 港市^  企业复工复产`  ▁crusherb 没法d mysf  ▁theoriesh  获得感j 回首l  旅行社n  ▁comicsp 风暴r 通畅t 杀害v  ▁Strix  ▁honorsz 香的| 属兔~ 水晶 iasm 光线 轮回  观点和  ▁stupid 相较 remove Sum  监护人 Di Row ▁Sha OSS  ▁enact four  丰富多彩 学府 果树 bone boards 同类 nity 催化 喜剧  自己和 Canada  ▁Yellow  ▁arrives  我们能 　”  等一系列  和技术 门的  ▁habitat  ▁quit ometer ’” 劝导 ▁Pad anship  ▁Swift  便民信息  ▁hydrogen 放开 脑子  ▁stressed  ▁biology usters  ervatives  ▁presentations 未满  ▁jumped  国有企业  reational 万象  讲述了 祝你 解惑  ▁paragraph  ▁wealthy 戴上  ▁Circle 跟党  ▁tens 这事 icting  ▁placement 按压 新版  更多资讯 猪场 许昌 批示 阻挡 izard  ▁pulling 集资 自尊  ▁heaven keeper  万余元"  适量的$ 地里& 除夕(  ▁rivers* ▁sq,  设置了.  ▁modify0 ▁mes2 院内4 湛江6  ▁embarr8  ▁leverage:  facebook< Tex>  就能够@  ▁LegendB  effectiveD 地去F  ▁cheerH 易经J  ▁PsychologyL  ▁FinlandN  了一系列P 义乌R  ▁organisedT  知道了V  ▁beingsX  ▁catalogZ 鼻炎\ isha^ mel`  ▁Flightb 在内d 接力f  Activityh COLj 耳机l  ▁tagsn 优待p 滋生r  ▁Drawt 双脚v  ▁Routex 管辖z  成长的| Team~  ▁horm  ▁innings  ▁Maur 跟他  个体工商户  ▁rubber  ▁poison  ▁attribute 抽烟  小孩子 瓶颈  ▁NATO  ▁Edwards  积极开展  业务的  了一套 Talk 净土  ▁Batman 启示  ▁demonstrates  对他们  ▁boasts 军区 南沙 etown  ▁Rodrig  准确性 etters fun asting  ▁archive 那边 实处 大病  一般是  ▁Facts 善待 ▁Vel 摆在  ▁donate  ▁metres 工匠 网约 stat /㎡, 汇算  ▁examined  ▁override  ▁wrapped  关键时刻 大部 ▁Rug  内容来源  ▁assisted  及周边 事先 煮熟  的任务  ▁imaging  ▁securities 花海 Has  ▁Musical  ▁Beauty 北区  ▁Clinic 入院 usive 机票 engu  科主任  ▁Soon 一贯 uisine  ▁ward  这几天  ▁electronics  ▁treasure  ▁Rather 春夏 进阶" 地标$ ▁Ni&  ▁translated(  ▁wrap*  关于印发,  ▁automation. 你将0 ▁SUV2 绑定4 anga6 黄石8  government:  赢得了<  ▁revised> 滋补@ 装箱B  防护措施D  人与人F 德尔H  ▁dressedJ  ountainL neysN 甘草P  ▁MedicareR  的季节T  ▁sinkV  ▁escalX  ▁travelsZ  ▁Anniversary\ 送你^ 双腿`  他表示b 着眼d 中式f  发送至h 在小j NOTl 启事n 京津p  ▁sheetsr  ▁overdt 喜好v 上半x urredz  不只是|  ▁Projects~  就容易 ▁lap  framework  ▁logistics  ▁Rogers  ▁optional  节目中  ▁assemb 提名  推荐的 试着  ▁Restaurant ▁Ul ▁gi  ▁streams  ▁Securities 录入  可以根据  ▁Harvey 官微  ▁blues brew  ▁crossed  ▁Coalition  内容及 Jun 炫耀 ▁Mob avel  ▁tragic 听从 银保  ▁Associates  ▁Cohen  ▁mineral  ▁strings 钱币  其中包括 脸色  ▁Nathan  大连市  ▁embedded  有许多  ▁strip 售楼  ▁precision 或在 vec  ▁continuously 拿来 消极 员的 ▁NPR  落实好  战略合作 FFFF 这笔 ▁NAS 雕刻 的黑 IPO 王朝  可以从 量子 演示 替换 百强 unted ONT  工作经历  ▁Mans 往前 低碳 眼袋  宣传活动  灭活疫苗  ▁trace ▁dos storm 后代  向上的  ▁neighborhoods" spec$  了多少& 花朵(  世界上最*  对外开放,  的大型. 基准0 之所2  ▁ESPN4  法律规定6 问他8 税率: auth< 昂首>  ▁Pink@  ▁buyerB 安卓D urstF  ▁footprintH 班车J leanL bottomN 光芒P ORYR 辆车T  ▁makeupV  填报志愿X LatestZ UND\  派出所民警^ 珍藏`  ▁Kashb 神山d 县区f  theningh ▁CSj ARKl  但他们n 强基p  ▁tenderr ▁GRt  设备的v 李白x annersz Ray|  ▁pled~  ▁innocent 痛点  ▁calculated  ▁Feed 其在  ▁creatures Credit  ▁Grey  ▁inherent  不满意  先到先得 全心  ▁Rico  第一款 lab isen  ▁resolved  ▁favorites  ▁cluster  ▁Releases  如下图 饭菜  ▁Minor sett  面临着  ▁Lions  ▁lever 切断  任何一个  级别的  ▁spreading 李小  ▁wurde  ▁prints  拼多多  理想和 Office gom  高级中学 预订  ▁formerly Tor 收割 个个  ▁pets  ▁prisoners  彼此的  ▁Jama  ▁surprisingly  ▁congreg ▁Vik env ULT  ▁disclosure  ▁headlines  未成年 清净 年了 陈列 牵引  今天小编 色泽 公诉  ▁protective 盯着  获得成功 antry 管网 比重 下行  港澳台 cur  了一批  ▁corporation  ▁coin  举行的 amas  ▁Barack  的增长 ▁Aer  patience  ▁cognitive" 芙蓉$  ▁Fest& ▁е( 子弹* ockets, 酸甜. 访谈0  ▁Lawyers2  ▁fruits4  和其它6  ▁Tampa8 存量:  ▁Mario<  ▁hint> 西门@  ▁SweetB  ▁cleverD  ▁constitutionF 超大H  范围内的J  ▁ImperL antonN  templateP HERR  ▁perceivedT 集训V 写下X  ▁PhotographyZ 唐代\  ▁vine^ files`  居民身份证b AWd 冰冰f  存在着h Neverj  三轮车l  stationn 特区p  进出口r  ▁joket 她和v ###x  ▁trillionz 骨质| Hub~  为主要 MAX ochem  合肥市  ▁polls Their 明朝  领导和  instein  传感器 既要 oba 保质  ▁archae 礼貌  olesale  ▁modest 留守 East  ▁Shows  ▁quantum wei  年春季  ▁Thanksgiving Op  ▁Analytics adows 上前 学问  真实性负责 Task  ▁orientation  ▁prelim  ▁performers  ▁donors 4 -这样您就可以继续免费收到文章了  ▁из 住址 业余 了你 atial 日到 了大  税务机关  ▁aviation  保持着  ▁recept 撤离 Fun  ▁Border  有人在  主动向 降压 ▁Eld  　&　 破浪 观摩 三五 两家 传授  发生一起  ▁Init sha 监局  ▁Upon Engine  第一书记  是通过  跟大家  风景区  ▁processor  也是这样 chant 归属 Happy  ▁defending 何处 踊跃 解剖  回到了 ▁nit  亲朋好友" ski$  中西医& 一则(  在行动* 乘风, 精通.  heastern0  ▁chapters2 ▁Bud4 ielder6 令营8  ▁advise:  ▁formats< 果园>  ▁delays@ 氧气B 几分D incesF  ▁inhibH  ▁FloreJ  ▁infringL igueN 二的P  ▁RetailR  地方政府T 忙着V 肤色X 起火Z  中心的\ са^ 上进` 底蕴b  招聘条件d 省道f  居委会h  ▁conspiracyj  在自己的l 鉴别n 转自p 喂养r 美满t 小花v 升职x  ▁attractionz  ▁Horse|  ▁reliability~  高分到低  一个女人  ▁Employment  而来的 造就  的传统 征服 十里 ]);  ▁triple  ▁sensors 墙面  ▁inhabit ▁WE  ▁diver ▁Hab  ▁bullet 酸痛  ▁elabor 赶赴 boro  ▁reduces 诚意  有能力 ▁Kle  ▁chaos opard Record  精彩的  你以为 tage tic  该怎么办  ▁economies  就业创业  ▁sensit 终极  ▁generating  ▁Therapy " 生命安全和身体健康 机动  了出来 属牛  乒乓球  ▁amenities 再也  和我们  ▁pizza 物价 begin ograf  ▁presum  clusions 时限 相声 ▁piv GT 枣庄 店的  ▁poems celand ushi 勿扰  ▁dece  ▁arriving Human 惬意  ▁cord  ▁Jamie  require  ▁Mathem 暂不  ▁Kill  ▁partly  ▁notification 马桶  相对于  ▁commemor 太子 后备 权属  ▁Milwaukee" ioned$ 环绕& phis(  ▁Coffee* 段子, 又会.  ▁diamond0  ▁invasion2 three4  ▁Category6 双语8 这让: 取暖< 悠悠>  ▁Shaw@ 叮嘱B CVD SteveF 勤劳H hallJ  ▁eighthL  ▁ChampionN  第二届P 公筷R  这一生T pushV  ▁relativesX 想的Z  发展改革委\  东南亚^  独立的` 羽绒b  ▁temptd  疫情形势f 突变h ▁BLj  ▁Milanl 陈述n 封顶p  ▁colorfulr ERTt  ▁observev 简易x Mattz  ▁prescription|  ▁symbols~ 巧妙  结合实际  ▁caution  往往会  细胞的 因地  不良反应 密接 endif  加入我们 渐进  可靠性  ▁Hero 施策 细细  应有的 东站 理发 engine  ▁Queensland 不畏 学好 报表  ▁portable 本届 food 原地  积极的 sz 分支  ▁Delaware  Program  ▁Bass 腐蚀  ▁Entreprene  接受的  ▁Cincinnati ▁§  ▁consensus CLA 属虎 必看  ▁interviewed  ▁zones  ▁spell ▁Wes  ▁algorithm 起义  公司在 quet  conduct 产物 一万  youtube 下周  绿色通道 台州 isse  ▁Artists  或者其他 ши  ▁nutri 益生  ▁innovations ANS  ifiable  ▁observations 内有 电缆  ▁unions ▁wit 一枚  ▁Egyptian him usc 面条 还真  有自己的 年是  America  ▁commod 千克" 幽门$  我可以&  纪念币( ▁э* once, ▁asc.  体系建设0  ▁ladies2 inars4  ▁isolation6  ▁playoff8 血清: htm< ▁Kil> 沭阳@ 寄语B  委员会委员D  ▁scalF  ▁competitionsH  ▁MiniJ 插画L DMN  UnfortunatelyP 检出R  降低了T 病区V walX 属相Z  的一大\  ▁priorit^ 母婴` 将为b birdd  产品质量f  市场主体h elongj  我已经l 变现n 畜牧p 与您r 孤单t 肝癌v 发掘x 三餐z 有心| 密闭~ 主干  都比较  学习资料  ▁specialty  ▁Gift  截止时间 urring 队友 细分 周恩 时请 ▁Fu 电线  研究人员 送至 师德 效能 货源 …”  ▁pipe  第一时间删除  指导意见 买家 Mobile 剪辑  ▁Immigration ▁SEO 上手 病理 python Smith  ▁Motors  ▁pleasant CK 备课  ▁investigators 推理 ▁zu 遍布 Json 听过  作者简介  ▁McDonald arten  族自治县 影子 chin  ▁dirt 清代 您好 ulin  ▁mature  报名人数  肯定会 老兵 堡垒  ▁kingdom ▁Nom  ▁Mechan  ▁grades 的总  将持续  ▁grain 习题 Short ergy ▁pig  困难的 want  心灵的 用到 创伤  的那种 Mill 产妇 立交  ▁Beautiful 曾任 INS uler  和信息化 ulates" Park$  专业化&  ▁Holland(  ▁toilet* 慎重, 应付.  ▁describing0  客户的2 ▁tob4 再看6 遮挡8 几率:  ▁sovere< ▁Arc>  文明实践@  appropriB 拥挤D SemF andedH  ▁FloorJ  ▁colonialL  的动作N 实训P baiR 分批T 其间V 初始X 校内Z  南宁市\ 答卷^ 中期` 洽谈b 陆军d  族自治区f  世界卫生组织h 不一j yclel irablen  关节炎p 孝顺r 中型t  ▁inclv  economicx  在没有z  ▁Trek|  ▁Phase~  ▁Concert fox  单位和个人  ▁fitting  统计局 idi  ▁Phillips  相关人员  的朋友们  美术学院 未婚 食盐 older 购置  管控措施 洞察  Subscribe  要做到 精益 ▁Ple  月底前 Monday Works 搬运  ▁crops  实实在在 bably  一个大 agne 针织 ▁pel  的衣服  ▁diagram  ▁Monitor 施肥  password  的未来  日期间 终结 连夜 prom Word cloud ▁Kan Yet  ▁assass  就是不 ▁IBM  ▁schemes een 迷人 ▁ч 发音  ▁mounted  人力资源社会保障  通行证  ▁racist  ▁Honda  甚至还  ▁considers  ▁affair 世的 狭窄  ▁Prep 折腾  确定为  尽量减少  ▁rust  ▁ritual ipper 中建  ▁safegu  ▁measurements  ▁adjacent  双一流  加微信  代表团  ▁aesthetic  他自己  ▁amendment  ▁confirmation 干活"  ▁grocery$ 班长&  ▁dumpster( rea*  核酸检测阴性证明,  ▁enrolled. 磷酸0  ▁kicked2 保研4  ▁arena6 卖家8 导入: 公证< 求是> 无常@  locationB  环境中D 病菌F 油耗H ▁BotJ  的运动L 可选N 援鄂P 专心R 拖欠T 济宁V  ▁ConditionsX  ▁BrigZ  ▁identical\  ▁во^  ▁Strong` 制约b  ▁punchd 展会f ▁Mih  ▁Activej ICUl  ▁linearn 四方p  ▁Glenr Dailyt 桥镇v ▁  ▁essence@ 餐具B 重启D 增设F  进行全面H 出众J  第三个L tokenN  的身影P  一分钟R  是一款T  事业上V  ▁existedX 地摊Z  ▁contam\  ▁rape^  说什么` closedb  的平台d leenf  ▁Riceh  ▁fingersj 都想l ▁WARn 短缺p ▁Mozr 名利t ctuaryv ліx byez  疫苗的| 较小~  prising  爱国主义  人员密集  你就是  ▁Particip  ▁journals ucle  ▁origins  ▁mentor  ▁retro  ▁Tool  ▁Brazilian  ▁Danny  ▁charming 殴打  需提供 双人 iko  副会长  开始的 姓氏  ▁educated  ▁shield  了我们 edit 辨别  ▁threshold 进场 phe 写着 Cost strap  ▁unfair  还有人 anyon 祥和  住建局 失调 icons  ▁celebrities  ▁Harbor  ▁shortage  配套设施 vine  ▁kiss  ▁wellness online unit 报废 色系 消炎  多元化  疫情防控的  星期六  很有可能  ▁personalized  ▁Victorian 北侧 抵御 引资 Mag 包包 不散  情况和  境外输入病例 Luc  前些年 ▁IR  美丽乡村 间距  ▁Reed  ▁header 大美 Fund 人也  师范学院  ▁rings airo 奇葩  ▁Bros  ▁athlete 得意"  ▁Nigerian$  ▁conviction&  万亿元(  那一刻*  主题教育,  ervatory. 的神0 田间2  claimed4  ▁feeding6  支持和8  的母亲:  ▁Plans< Edit>  的一句话@ 白了B HelpD halfF  ▁pursuitH ambersJ 舒缓L 石榴N  ▁coinsP CollR ocoT  ▁SnapV 实拍X  ▁sendsZ  驾驶机动车\  ▁fighter^  免费的`  ▁earsb 公考d 夕阳f 牙龈h 全科j  为契机l  ▁biasn  奠定了p 狠抓r ▁tatt atinumv 再现x  简直是z  ▁desperate|  ▁Unlike~ 不合  有关的 Frame 聊城  ▁edges 为我  的一款 users  ▁accountability  右上角  ▁respondents 六十 中了 嫉妒 ulus  巧克力 睡后  说不定 冷漠  ▁Northwest  ▁performs  ▁Dame diff icts  禄劝禄劝  ▁Draft  ▁cement Tw Film  ▁associations Bra 点多  ▁dive  ▁barrier  给出了  ▁Catherine  ▁Wells  ▁cooling 听见 睡着  ▁evac  管理条例 xim  进行核酸检测 ▁Rud 安庆 qual 天鹅 薄荷 acker  交易所  ▁Ottawa 城里  企业所得税 itcher ▁dys 制宜  ▁Safe 噪音  ▁resist  Connection ▁Ye Equal 野外  ▁editors  ▁brush 品格 收录  ▁Lower 即时 NAS  年出生  的成功 向你 打扰  出具的 邂逅  学士学位  ▁chron 阳明 ATCH  ▁revis" ▁Haz$  ▁Hear&  ▁Bernard( wegian* 攻关,  ▁spotlight. 完后0  ▁renewed2  员工的4  ▁sacrifice6  ▁youngest8  ▁Wang:  ▁demol< 看懂>  ▁WITH@  ▁preliminaryB  集体经济D  ▁suitsF 刷屏H " 和新疆生产建设兵团J  ▁MineL →　N  ▁outlookP 蕴含R bageT  个项目V 缺口X  后台留言通知我们Z  在他的\  ▁greenhouse^  服务区`  ▁Popularb ▁lund formedf asseh  中心城区j  ▁longtimel ( !请立即后台留言通知我们n rhp 表述r  的关注t  定点医疗机构v OHx 乐于z 明知|  给人一种~  ▁Getting  行业协会 杂物 势头  ▁detected  ▁separation  fulness  ▁Late 该案 YS  ▁Perm  ▁penalties 润滑  这两种 global ' iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii  ▁thesis  无论怎么  ▁lean  ▁Nepal 屋顶  以为是  ▁touched 阔步  切实做好 ▁UC  没有一个  将通过  句话说 直言  ▁fool 再多 炎热 否定 化名  解除医学观察 沧桑 北美  购物中心 rowse 可行 粽子 调侃  ▁Simple  ▁distinguished  ▁Lakes Ill 之称 ▁:: 祈福  ▁bases Amazon  视频中 plot mile  我们都知道 历代  ▁blogs  Document 和她 手表  ▁Curt  ▁Install  是多少  相关政策  ▁Vermont  的魅力  也是很  ▁acute  我们已经 放进  ▁subsidiary  ▁ought endant 机能 擦拭 感应 寄托 本院 包吃  每个人都有 顺应" kan$ 偿还& 骚扰(  ▁decides* 大妈, 又被. 回馈0 xc2 男士4 顿时6  的老人8  身体状况: 看望<  全国人民>  ▁Polish@  ▁ComicsB  ▁vinylD ▁UIF  ▁exploredH  工作安排J 对流L  ▁coordinatorN  一键添加P  部部长R  小学数学T  ▁ChaseV 某一X 极度Z 离家\  ▁batch^ 华盛` 要将b  ▁Ranged 悠久f 焚烧h 气势j  ▁enthusiasml  尽量不要n  为什么会p  所有人都r  ▁compiledt 上火v  ▁semesterx ingoz kl| 宽敞~  运动会 救命  红色基因  ▁outreach  ▁slee  ▁refugee  ▁castle  不作为  ▁Ambassador  ▁Lect column 去掉 ancies 地道  不代表本平台立场  主城区 mu  ▁subscribe 仓储  的主题 抽查 个股 照亮  就成了 phant  ▁Lebanon 　@ 吃亏  获得者 久之 ymph 必有 ▁WA igg  的人数  ▁expend  不是很  高到低 七月 六年 lit 防守  曾国藩 动词  适合自己的  ▁plates 只见 母乳 None  ▁Jake 供需 Prom  千万人 万众  不锈钢  ▁Info  分析师  ▁stomach  父亲的 gomery Want 着重 );　 不畅 ▁fed 疏松 贴士  资格审核  不仅有  disciplinary 瘙痒  Earlier 绘制 富人  前不久  来源丨 初一 好物 IDE 油菜 panic 襄阳"  ▁champions$  ▁organize& 即刻(  ▁frozen*  ▁riders, 而过. 贵族0 总队2  ▁Crisis4  等服务6 夺取8  ▁patri:  ▁Inspect< 开窗> 松弛@  ▁proteinsB  ▁belongsD ▁DimF 心意H  ▁repliedJ 鸡汤L  ▁postpN  ▁curveP 首例R  ▁peoplesT 常德V  有序推进X gitZ ▁Lov\ ▁ANY^  　,　`  技术和b  ▁Peted  ▁Carlosf  ▁warrantyh  非物质j  其他地区l  ▁pressingn 公章p 文昌r 二者t  至关重要v acceptx  ▁pricedz  元以上|  会觉得~  ▁immers 各行 payers tn 差的 战机 儿的 诗意 同情 星光 cond ção 脱颖  博士后 design  Thursday  ▁Lucas  的方式进行 ▁bay  ▁fond  ▁proceeds  ▁bankruptcy  招聘计划  ▁cave 分化 漫长 ▁GP 帮我 zp  组委会 加装  ▁bottles xing 买车 扩招 融创  ▁unnecessary  ▁Cathedral rives  紫外线 粘贴  ▁Fellowship Sus 积蓄 分值 enti  ▁Batt 思议  rencies  ▁Representatives  社会组织  管理者  ▁stylish obj 被子  全面推进  ▁hospitality  ▁memor 外婆  ▁scrut  清晰的  需要在 apsed 机组  category orf risk  language 票据  ▁tactics  ▁gambling  基本医疗保险 绿灯 曾说  本期编辑  ▁deciding 劝阻  第一线 免去  ▁verify arium Mic" social$ ▁Ko&  城市建设(  connected*  正处于,  atories. 柳州0 标配2 片段4  ▁polar6  先进个人8 EMA:  外防输入< 约谈> 高潮@  ▁ArrayB  ▁tenureD  ▁DefenceF  ▁brutalH 答辩J 各科L  ▁BrooksN  ▁musclesP  ▁grandmotherR MassT takeV 提炼X  ▁tradedZ 票价\ 福田^  往期精选`  一本书b ectsd 心存f  类似的h brookj 全城l  ▁Fitzn 庭审p  建筑工程r  这件事情t Ilv  的时刻x  研讨会z 偏偏| techn~  ▁responding  客服微信 内脏 Leave 村落  的品牌 ▁Thr  ▁glory 我来 Jose 祛湿 　· 兑现 历时 解脱  ▁judicial 滑雪 ▁toy  的建议 过量  ▁appealing ▁vir ▁DR ischer ▁Syn  ▁recalled  始终坚持 spot 洁净  非常高 PART  ▁Mend float 之城 看重 Hard  ▁threatening 模具  generation  建面约  作品内容 暂缓  财政局  职业道德 造谣  ▁tube  电影院 ▁ACT  的个人  高考成绩  complete  疫情防控工作的 博会  投资人  ▁Gonz  用实际行动 $( ▁Hum 热文  ▁Linda 时应  California  Sometimes 倒是  ▁Medal  也越来越 书房  ▁stones 收回 Tunes  ▁understands 肠癌 丢了  ▁Brandon  ablished 贩卖 经办  科学历  ▁startups  综合成绩 折叠 ISS"  ▁touring$ 位数& 十堰(  妇幼保健院*  ▁charm,  neapolis.  ▁dispar0  ▁Generation2 草药4  就像是6 三楼8 中说:  Collection<  日起施行>  同比下降@  要保持B  学历要求D  ▁pointingF  各学校H 在哪J ▁DjL uskN  ▁aluminumP  多功能R cciT  ▁HockeyV LengthX  ▁invitationZ  ▁occupation\ 出栏^ 如遇`  东明圈b ▁Dold ociof  新陈代谢h  增强了j  ▁biodl  做的事n  职业学校p  ▁hiker )]t  ▁knifev  些什么x  ▁Baldz 各村|  versation~ 伸出  又一个 容忍  ▁Nebraska 滥用  必须在  ▁voter 阿胶 oug 太湖 气味 fill ___ arus  ▁compromise  ▁mamm  请务必  ▁Nicholas Opt  ▁encomp 血脉 轻型  ▁qualifying 双休  angular ifle anmar  代表的 空腹 同仁 peror  ▁blessed 税费 抗议  可以将  这都是  ▁refriger  ▁freely SCI ▁î  红楼梦 ▁cer  普通高等学校  活动期间 干咳 bao 巴基 专访 连线  ▁leak REF  ▁determining 古今 omore  ▁stating 百日  ometric  ▁storytelling 任性  第十四 橄榄 . '新型冠状病毒感染的肺炎疫情  如果不是  添加剂  ▁Thunder  的理解 三峡 ▁''  ▁frames 纠缠 脱单 九江 登场 umbled 身穿  那个时候 掌声  完整版 制动  ▁bones 何必 巴菲"  ▁surveys$  ▁travelers& 止咳( 镇长*  不停地, 比你. 一出0  ▁thrive2 Target4  ▁develops6 选址8  ▁Bott:  ▁nort< 能把>  ▁finale@  ▁SpiderB  这几年D  ▁civilianF  ▁DougH 修剪J 坚果L 可视N  ▁scenariosP 提案R  ▁IncreT 刑侦V  全面从严治党X repeZ 在手\  ▁wore^  弄虚作假` Henb  好口罩d  ▁Sierraf 之王h 错题j  最后一个l 不由n  ▁publishersp 的全r 埃及t 紫色v  Interestx 点和z  ▁intens|  iblings~ 测算  幽门螺  ▁vaccinated ONG  第十二  图片均 两只 精灵 高地 高档 oro  在这场  ▁Blake  ▁forming 也更 ▁ink  县委常委 天长  ▁Treasury ▁ré 层楼  ▁Hughes  提到的  ▁decreased Books 发车  ▁Degree 瑞士  博士生 ocking  ▁resilience 确立 hours  ▁breathing (){  线城市 uh 准入 HK  ▁evolving  后面的  ▁Arsenal  添加微信  招聘单位  ▁badly  ▁elementary Little  教育集团 契税  统一的  ▁Size activ  ▁demo  ▁Dennis 南侧  创新发展  ▁reck ▁mi chem  平均工资  在现场 了好 ols  ▁subt  为目的 hend 样式 pher  ▁floating (\  这些地方 rene 一说  你有没有  ▁installing  只是一 两不  或者说 ▁IC  等各种 随心  身份证号码" 吴某$ >　& 内侧(  ▁playoffs*  准备好了,  ▁visibility. 抄袭0  ▁dealt2 Sever4  重要作用6 以待8 转弯:  应届生< 温水> 周口@ 烦躁B  ▁premisesD  一般在F 烤肉H  ▁sharJ 汇集L  ▁partiallyN vtP latR  ▁anonymousT  ▁BringV ▁BBX  ▁rewardsZ 滨州\  今后的^ 出货` outerb  ▁utterd  ▁droughtf  ▁missileh  ▁Citizensj  ▁pupilsl  ▁intensiven Syp 带娃r  项目部t  发现问题v  ▁curex ▁ogz 两边| 展厅~  ▁costly 打败  ▁benchmark  ▁toss Early  都能够  ▁translate 尺度 与众 外资  ▁spotted  ulatory vd bat 七夕 ERR  ▁durable  ▁dess 免征 hanie  ▁imprison  ▁congr cert 英勇  的区别  ▁fabulous 他又 绿茶  马来西亚 ▁q  ▁cycling ▁Kon  ▁Metropolitan 最少  ▁incorrect  ▁embark oping  ного  硕士学位 衣柜 两级 德州  ▁sculpture  ▁tobacco  ▁conjunction Prior  集装箱  ▁outfit 制剂 短时  ▁Trip  ▁combining  Environment  民营企业  ▁compliment 外形  ▁tweeted 通勤  ▁wings 养育 取证 educ  等原因 板上  将迎来  思想和  ▁seasonal  但对于 MENT  ▁Saints 无尽  乘坐公共交通工具  ▁Aviation 订购 笑了 ▁Et  接触史  让自己的  ▁urge  ▁NEWS 退费 Tur"  initely$ 战国&  ▁Banks( 想吃* 迟迟, VM. 精髓0 三观2 补齐4  道路交通安全法6  社会治理8 ById:  据介绍< 长途> sson@  ▁bearsB  认真的D  公共卫生事件F 名录H  ▁witnessesJ  查看详情L ▁AbsN  各级各类P  招商引资R  ▁sellsT  ▁reboundsV  说实话X  ▁ShanghaiZ  会议上\  永远不会^ Bank` NYSEb  comfortabled 换乘f 查收h  本次招聘j  ▁correll  ▁Claren 霉素p 鱼的r Nickt }$v  ▁loadsx  工业园区z  ▁corrupt| 勇士~  ▁slip {"  膳食纤维  development lette  ▁Helen 转折  尔多斯  说的话  ▁Near 株洲  政法队伍教育整顿 人流 ▁eru  规划建设  和要求  ▁observation  ▁downloaded  ▁briefly OSE  ▁Gaza  集中隔离医学观察  European  特有的 加薪 打理  ▁Calif  ▁Bever 贴近  着自己的  ▁Yale  小学英语 庚子  ▁Grade  ▁offshore 安定  自来水  可以把 做大  ▁Really 封城  ▁measuring  ▁medieval  更多人 ▁aug  聊天记录  British uang  以及其他  ▁Desk 封建  鄂尔多斯  的真实  工作需要  ▁runner  ▁Besides  ▁independently 漫步 恋情  ▁alliance 不走 向南 ean  ▁Karl  ▁Globe ▁Ts  ▁melt  都应该  ▁Strategic  人事考试  ▁solving uate  所需的  ▁crypto ②　 ▁Ax  奋战在 供暖  ▁practitioners 色素 Fre ▁EPA" 编剧$  ▁Butler& 交接(  ▁fost*  ▁magnificent, 两点.  ▁LinkedIn0 oped2  istically4 冷藏6 我又8  放弃了: aca< 心语>  洗衣机@  ▁drawingsB зоD  淡淡的F 地形H 提琴J  爱情的L  文章阅读N  健康发展P  要根据R  和学习T 里克V  ▁emergeX 话语Z 宝典\ 补血^ 主打`  和朋友b  先生的d 号的f  商业银行h  ▁valleyj  ▁Cemeteryl Denn  ▁sentencesp  资格初审r 因其t 属猪v 多大x ocityz 恰好|  ▁disadvant~  ▁nowhere 属龙  ▁Chand 那一 紧致  ▁smartphones 轻盈  ▁Christianity  市人民医院  对孩子的  ▁wines 切的 ▁Tib  ▁deficit 军官  ▁licensing  ▁Regulation  的一名  北京时间 unge  ▁Assessment ▁Tab rained uid  起到了 ▁NSW  ▁Hopkins  ▁stabil  ▁Trailer SB  ▁reass pic  ▁hazard 芹菜 erek Far 一听  ▁rhythm  实验中学 一字 康熙  有需要  ▁plugin  ▁Parad 负重 老子  ▁strengths 干了  疫情防控要求 ▁MBA ernel inance  avorite abwe 茫茫  ▁arrange  党和国家 旅程 树林 千古 久而  制作的 就更 三里 uddy  example  ▁soldier 病因  ▁fled ги  ▁ranges 死者 穷人  几分钟 Ren ▁hem  中医医院 ré 麻木 争夺 正月  化管理" 领略$  而这些&  巴基斯坦( 开办* idate,  变得更. allow0 otypes2  也能够4 黄埔6 同款8  ▁critically:  ▁honestly<  提高政治站位>  ▁Colombia@ TravelB  ▁loadingD 儿科F 跌幅H  ▁grandfatherJ 指控L  orkshireN  跟党走P  ▁exemptR  ▁promptedT  智能手机V  都无法X  ▁hikingZ  ▁beta\  美好生活^ 党旗` deleteb 三伏d  Attributef 走来h  ▁Presidentialj 歌舞l  确实是n Satp 带到r ▁Sidt 大夫v  orientedx  完成的z ▁Cad| 不惜~ 雪花 happy 她还 imo 银河  详细了解 恰当 比对 umatic 奥斯  为自己的  ▁historically  ▁triumph 红豆  并提供 Hon 把控 grid 鲜艳  把孩子 听书 店长  ▁variations  ▁suited  独一无 ▁rap 总共  的热情 高二  支气管  为实现  退役军人事务 them isi ▁mph  ▁diesel  Community  和学校  ▁dealer eval  ▁accelerate 手足  也有人  不经意 ▁Pok  ▁molecular  ▁Leaders 转眼  ▁bachelor  ▁Gene Maybe 零件 osex ▁Mak 印尼 核定 毒株  切实保障  ▁flooding  ▁Capt  的优秀  ssssssssssssssss  ▁fence 和中  ▁deny mare club  javascript 服役  扎实推进  物业公司  ▁Users  ▁forcing 时事 价的  确定了  ▁outlet  ▁attendees  展现出 ▁obj  ▁asylum 鲜美" 领军$  精细化&  ▁subscribers( 你最* issues,  ▁Musk. 恩爱0 失联2 常说4 勘察6 navbar8  ▁predecess:  卡积分< ▁Inv>  ▁rankings@ UBB zdD  ▁citizenshipF  是因为你H 球迷J  ▁QuebecL  流行病学调查N ueseP  ▁StorageR 新政T  ▁BranchV  ▁organizingX 兔子Z ▁MB\ 心跳^ GI` 不平b menud  ▁Achief  ▁fortunateh 稀缺j  ▁gapsl  时间安排n 党风p teamr  帮助你t ▁CBDv  ▁complicationsx 巾帼z  和专业| 小巷~  ▁Anal mund  ▁notably  ▁boyfriend  ▁Launch umbing  ▁Violence 佛陀 ollo  是目前 收款 拉萨 死后 arded  新发展理念 ▁MIT 湘潭 喀什  营业收入 养颜  ▁Biography 卡通  ▁excessive  ▁Ruth  ▁educators  一口气 税额  不要再  周恩来 iments  ▁Journey 切除 exper thood  ▁prizes  他强调  ▁dirty  ▁sponsors  才不会  接种点  非常适合 ewater 鲜嫩 Body Cast 皇家 璀璨 妙招 不肯 接入  ▁absent 附中 涉事  ▁crystal  ▁demonstration 纵横 揭谛  ▁ISBN  ▁Bryan ACH arin  的应用  ▁wounded 客流  他指出  还不错 ysical 百花 口红  ▁locate  余万元 持股  ▁surgical ▁Ask 风电 的了  在日本  ▁Industries  ▁streak  ▁Eagle  ▁medications  玫瑰花" 年期$ 破损&  结构化( music*  ▁ре,  过多的.  element0  完整性2  imbabwe4  ▁Python6 KTV8  专案组: 的脸<  立刻就立刻就> 羊毛@  ▁NeighB  舍利子D  ▁PlayStationF  urchaseH  ▁CensusJ 过户L  所以我N  不可思议P ancelR rioT  资料图V 宣告X 得头Z ▁Dod\  ▁Soccer^  Department` 川普b 换个d 首家f 马鞍h 和高j  ▁Sleepl  ▁distributen 案发p 龙岩r  ixturest 见解v  的工作人员x pptz 作弊|  报名信息~ folk  ▁trio  ▁gray iji  依然是  ▁selecting 乐器 撒娇 roe  意思是 有事  的经典  Tuesday  建造师 LI 跟鞋 联手 iane  liances  ▁seating  测试卷  ▁halt tree harm ookie  ▁Celt  ▁holder  的土地  及以下 imir 填空 经贸  用户的  ▁Close  ▁accurately  ▁drone  ▁promotes  ▁consisting ▁gem ▁doi  瞄准农村 mith  平常不过  ▁Aven  告诉大家 玩的 ▁Opp 推拿  选择的  ▁Brief  ullying 一扫 仍是 ():  ▁argues 防空  ▁explosion Ta 祈祷 范文  ▁cleared 方程 ▁apt  ▁tuition shirt 华润  Working  公职人员  ▁adverse 泪水  会让你 画的  curement  ▁loyalty  ▁Additional 上衣  想办法  发生变化  ▁Liberal SARS  ▁от  ▁appointments" ▁Mut$  ▁Uber&  ▁rides(  手把手*  ▁comprised, 漳州. 为本0  ▁earthquake2 倘若4  ▁protagon6  教育学8  ▁initiated: lich< 瘦身>  ▁Factory@ 江门B 胃癌D  犯罪事实F  ▁PattH etoothJ 限额L  ificatesN  igatingP KimR 杰出T  ▁boomV  ▁intersectionX 来往Z 油炸\  三角形^  ▁chore` parkb 万户d  的妈妈f stenh ▁NAj 不相l ICSn 德育p 日程r  课堂教学t  平常人v  ▁traffickingx  莫过于z  ▁teenager| 益气~ ▁GPS 秘籍  ▁dimension  来之不易  ▁anytime  ▁Upper 先是  ▁distant 一动 人参 纲要  changing  订阅号  ▁mortality  ecution 极高  ▁indirect  ▁dimin 哮喘  ▁Finn  要继续  ▁disk 月上  奋勇前进  ▁Grove 多得 ▁ISO 邮票 Dom  大家庭  工作机制 港区  ▁Canyon  为指导  不具备  和企业 graph  电子凭证 减退  验证码 无缘  ▁Luck  ▁Mountains 立德  可持续发展  ▁occasional  服务于  紧紧围绕  非物质文化遗产  是不会 ▁Mix 息息  ▁Cabinet  ▁creators  ▁Chemistry 车速 武警 她也 道教  等进行 OO  高中生 勉强 灵芝 ▁Fat  ▁Heat 之初 河西  人民警察 纸上 公英  和自己  ▁burned 实干  ▁Idaho 无缝 ▁Д  ▁tanks  ▁connects  　~　  ▁median  夏令营"  请在后台留言$  Interface&  航空公司(  ▁bearing*  ▁gameplay,  的概念. 世代0  服务站2  付出了4 动机6  选择题8  ▁Uncategorized: 一向<  ▁aspir>  ▁Julia@  ▁PlacesB 花草D 最怕F  ▁lectureH  EverythingJ  个人账户L ▁LayN  ▁relaxedP  ▁FosterR  ▁troublesT 北站V 缅怀X sleepZ  按照规定\  ▁shifts^  ▁instructor`  更加的b  ▁breakthroughd  黑色的f  博士学位h Typesj  ▁remindsl арn  付出的p  新模式r ongot yiv 红火x менz  流行的|  angered~ 央企  ▁flee  垃圾桶  Members ☟☟  理想的 NP //　 对联 日历  队员们  ▁dairy inea  确诊患者 osity WI border  新西兰 海参  ▁neglig  务工人员  再平常不过  ▁Iceland 之作 Tree  ▁Mills 警报  Section  ▁arise 莫名  ▁Reyn ▁AF ▁FM  ▁corners 的王  ▁thriller  足不出  收益率  报名参加  的处理 丁香 心愿 泡泡  ▁Matter 大棚  ▁viable great 加水  ▁Fact  ▁Zero  在使用  ementia 张国  心中有 metic  中国科学院  绝大部分 粉色  可获得  现将有关事项 ▁Rah  ▁shocked 翻身  ▁kidn  ▁Collabor  和安全  明确规定  ▁rocket  年级上册  服务保障 many  不得超过 " 党的十九届五中全会  ▁containers ndum  十九大 拖延  ▁graduating 蜜蜂  ▁lane  好朋友  ▁sake" 蛋黄$ OUT&  比较高(  还会有* ▁[', ospace. 停工0 存活2  旗下的4  胰岛素6 有条8 将至: record<  快速发展> 区位@  ▁drunkB 这天D 明代F 中新H valsJ  ▁CinemaL 决议N  ▁discretionP OGR  ▁segmentsT  ▁JulieV PutX  著作权属Z  ▁detention\  为保障^ 出征`  ▁Teachingb  日白天d  研究决定f  注意了h 穿梭j asisl  ▁billionsn  ▁bronzep riever  不出来t  structionv  批发市场x  Insteadz  在武汉|  ▁Ship~  ▁Rosen  ▁Harper  习近平同志 ▁Nan 固体  学历学位  ▁Vacc  ▁obtaining  ▁tends  ▁figured  不仅要  ▁Chan  imensional 停产 帷幕 ▁sel  联系我们进行删除  ▁Moroc 试讲  ▁scary 赵丽 动人 同济 相邻  ▁Physical  ▁bedrooms  是需要  和使用 ▁McL  ▁Pere 会说 Window  ▁shine 属羊  预防和  中国政府  ▁Always 向好 大神  购买的  取得联系  在他们  积极主动 黄豆 公文 切割 端的 胸闷  境外输入确诊病例 ▁Beg  睡后收入  ▁dupl  ▁individ  一些人  ▁Abraham 反抗 丝绸 方形  ▁Returns irs  ▁hills 家用  一般情况下  ▁electron  殊不知 来做  太高了  蒲公英  ▁Tigers  ▁commerc  ▁uniqu  ▁optimal 大涨 整车 Arg ▁Rd  从来不 克拉 ishe  ▁sacred 梅州  ▁мо"  ▁functioning$ 破获& 明媚(  ▁Continue*  ▁Individual,  ▁fitted. 伪装0 土木2  ▁Preview4  看来都是6 ostalg8 ▁sie:  ologically<  ▁Request>  ▁Script@  青海省B achiD  信息系统F 全册H 意境J KWL 更具N 直击P  ▁NormanR  ▁ChapelT 不全V 面谈X  导致了Z  编辑丨\ 亦无^ 繁忙` 不异b 于心d 个税f  ▁appealsh 歌词j 获利l  疫情传播n  ▁affiliatep 吃药r  就这么t жиv  会引起x 成型z 清澈|  Wednesday~  好不好  体系和  represent  ▁Croat  because  ▁societies 香水  ▁continually  ▁Representative OLD uits ▁GDP  的整体  ▁Guys 签到  所高校  ▁discounts 干嘛 应试 安抚  ▁Previous 村级  讲故事 Doc  ▁leisure  咨询服务  大概率 ▁ber 教资  脂肪酸  服务器 oko  ▁Scar  ▁Holmes ▁$( 待人  市场化 两款  两人的 祷告  Function 顺丰  ▁Malt  主要负责人 ▁Sr  以下是  虽然是  ▁vendor  ▁collaborate 教职  现阶段  ▁queen 现身  的根本  毕业证书  高高的  执行力 ested  即可查看  ▁milestone  ▁affiliated 发性 ▁Mey  领导小组办公室  ▁Lauren  研究生学历  很多家长  ▁Used  大家都在看 ND 斗志 暴涨  ▁Until window 香甜 ès nn iosity  ulative  ▁variation 家政 大山"  深入贯彻落实$  ▁oversee&  全程佩戴口罩(  日晚上* 三室,  均为阴性. 说服0 Job2 抗美4 源源6 郑爽8  污水处理:  爱自己<  ▁Parish> 梦中@ 脑梗B 转正D  大家对F  玩游戏H  ▁gramJ  fictionL 三代N  ▁MedicaidP 恩施R ChiefT 点滴V  一次次X 退税Z  的办法\  点在看^ 认清`  ▁argsb 以致d 物种f 汤汁h ▁},j  ▁prospectivel 李文n  ▁Hurricanep  广大人民群众r  宣传片t  因为我们v  ▁impex  ▁recreationalz  交通工具|  ▁Prison~  大年初  ▁Profile 心生  ▁Sacram  ▁tablets  可不是  乡村旅游  之间的关系  address 橡胶  眼中的 Cho  ▁approaching 力求 得失  怎么看 ▁RES 九年 depend  atility  发现有 但你  ▁Schedule  游览时 aran 找准  天然的  ▁recruiting 县人  以后出生 idding  ▁fuels 重度 方舱  ▁lawn  ardless 把她  低下了 对策 ▁GMT  社交媒体  ▁modes  ▁cancelled  ▁contrad  ▁satisfy  ▁Vienna adians 畅销 isb  没必要 ungle  富有的人 军训 mount  资格证书 六月 肆虐 排便  不合理  全力做好 菠菜  可能的 ortium  微信关注博野  治安管理 parse 念念  的快乐  带回家  才会有 在售 差不  副总裁  都市报 半生  一个个美丽 肚脐  ▁fighters 和水 不慎 尾巴  ▁Resident"  场比赛$ ▁Seb& ocaust(  的一段* 费的, 二胎.  网约车0 辛巴2  益生菌4 eor6 ▁Tat8 地在:  停电时间<  ▁firmly>  想象的@ 起飞B 特质D ▁GamF 二战H  ▁healJ  ▁TeacherL  atabasesN 大额P  ▁traditionallyR 挤压T  ▁biteV 深耕X  ▁temporarilyZ 不长\ 久坐^ late` 问卷b 人意d  研发的f  旗舰店h  的一封信j  ▁Tournamentl  ▁mattn 特级p  ▁Warriorsr ISOt 人了v  传染性x 私密z  拿出来| Blog~  ▁intr orical 鱼肉  人们对 上万 ensen 刷单  表达能力 )" wiki  ▁encountered 巨额  工作队  出现问题 在读 ів  ▁Lith  应用于  也已经  行动起来  还有什么 料的  无疑是 ▁sla 庄镇 高大 连日 Given 引人  ▁crowds 膜炎  管理系统  病人的  对学生  ▁Something  journal 视察 卖淫 平板 桌上 Ken  大酒店  遇到了一个个美丽 ”— 有好 ▁Ng 外在  届中国  ▁Error  ▁baking 几位  ▁disciplines  ▁matched  ▁civilians  ▁се  周一至  公务员考试 ▁HBO  ▁handy 家门  就行了 毛衣  ▁Charg ▁Dad  ▁tunnel  ▁Welcome  ▁advers  个孩子  工作岗位 每到  ▁Below  imental Walk 开户  :　“  体会到 short 会很 ▁CDC  ▁convince ▁О 美方"  今年是$  ▁Roth& 手绘( aset*  ▁listings,  人公告.  ▁doses0  这对于2  特别行政区4 ▁Jar6 盐城8  喜爱的: ང་<  ▁casting> 红娘@  编辑部B  ▁monsterD  ▁tongF  ▁versatileH  管理有限公司J 秘方L 高危N  也都是P  参加面试R  也只能T  营养价值V  只有在X oceseZ  现在是\  …………^  ▁prohibited` 曾被b 将从d  ▁Risef 包子h граj  四个意识l  ▁Weddingn Hyp 天真r  客运站t 开出v  ▁consciousnessx  为重点z  Article|  Everyone~  ▁commands 时速  将自己的 老龄 食疗 昔阳 uded  你是不是  接受了  生活中再平常不过  ▁inevitable  ▁Begin  ▁illustrated 胸怀  我一直  作出的 赞助  做好准备  ▁Answ  ▁rook minded  ▁Muham  艺术类  Session ( !遇到了一个个美丽自然的  牢固树立  开学前  ▁unity  ▁Door 院子 arming 痔疮 来访 团聚 悟空 会儿  为主的  Charles  ▁vector 裁定  ▁marginal  因涉嫌  管理学院  ▁Markets 牛年  ▁revealing opez odia  ▁Volunte 亚马 FDA  东莞市  ▁probe 充沛  这款车  ▁punishment 栽培 ▁Bak 屠宰  久而久之  ▁backs  严重违纪违法  不定期 相聚 Mont 初衷 劳累 捷径 ▁Kin  专业技术人员 Marc  到时候  ▁midst  ▁accountable  ▁Surgery  我们希望  ▁voluntary 楼面 idy  ▁Bobby 多天 口诀"  的数量$  归原创者所有& 小车(  ▁disputes*  的理由, iker.  Perhaps0  做的事情2 读物4  ▁Recovery6 破碎8 当他:  因地制宜<  工程技术>  Configuration@ 刘海B 年少D  关注我F  ▁nurtH  ▁HeightsJ  ▁DylanL filmN  ▁namelyP 插入R shopT  有时还会V  ▁SociX 喝了Z izers\  ▁Territ^ 遗忘`  ▁provincialb  整理了d ▁Pacf Beyh  ▁thermalj  性生活l thsn 八十p  ▁metaphr  ▁statuet  这部分v 公用x 泰安z 甜美|  纪律审查~  ▁anime  不下来 until 打出  破血流 配音 ▁tv  安全头盔  的岗位  村村民  乘风破浪 大运 ▁UT utory 满怀  ▁compound  艾滋病 画卷  严重影响 火焰 tex  逆行者  自动驾驶  正式启动 禁令  义务教育阶段 Equals 牵手  ▁Vlad  ▁Squad 款项  知名度  拿到了  小女孩 Stay 不振 阳泉  ▁Electronic  ▁Certificate mult  将进一步 Spec  路交叉口  ▁Founder 听着 它能  手机号码 pal 运送 降解 心安  ▁Boeing  ▁scrap  ▁straw  ▁Nuclear 先把  仅仅是  司法局  ▁divide  现有的  外国人  inceton 并列 fw  社区卫生服务中心 立减  ▁congrat  ▁Agent 台山  可享受  也希望 违建  ▁Firm  ▁filming = 6▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁preference 降到 SELECT  高峰期 isive  管理规定  ▁strictly"  ▁Pages$ Mode& 专线(  ▁rivals*  ▁obstacles, 有色. 植株0  不能再2 数值4  强降雨6 时任8  ▁commitments:  ▁Gaming< Iran> 投案@  消失了B ▁LedD  各地区F  ▁incentivesH  小朋友们J  教育事业L  ▁companionN  ▁ComplexP ▁nodR 上加T 荆州V 后人X  返乡人员Z  微信公众平台\ 亲手^ 传出`  Accountb  ▁hidingd roleumf epsh  ▁Goingj 金的l  提高警惕n  有下列p  进入面试r  Interviewt  ▁harmonv Centerx  上班族z 横财| Flor~ 攀升 舌头 脏腑  这些看来都是 药业  ▁flagship 年末  ▁customized  完善的  ▁Ashley  ▁holders  ▁controlling  ▁saves 1 *这些看来都是生活中再平常不过  给别人  长按下方二维码  ▁Yemen 唯美 Reader 被判  中央军委 edar 录像 妥协  Australia 顺便 人脸 worthy 家风  还不够 Sep 而去 Bit  四个自信 植树  ▁Legacy 犯了  ▁Taliban  Currently 三千  ▁productions 年假 七年  ▁Adult  ▁lifted 光是  ▁enforce  ▁Conservative  和建议  ▁kilometers  习惯了  ▁Tiger  ▁revolutionary enery 一整  而且在  ▁chap 兰花 icion  笔记本 ▁hed  交通运输局  ▁distinction  ▁Gate anie ctrine  ▁basement  菜市场  米以上  小区的  去探寻  ▁separately 横向  指挥中心 从头  ▁straightforward 民用  健康成长  ▁Leonard  ▁contrary  工作内容  ▁hilar" OA$  就是我们& ( !人民代表大会常务委员会( 骑士* east,  ▁merchand. 道具0  家协会2  ▁Certified4  进行中6 全镇8 对的: 心目<  oubtedly>  ▁teenage@ 戒指B raceD  PacificF 言行H 属马J 父子L  学习方法N 开会P manuelR uberT 碰头V 交房X  ▁hopefulZ  的女儿\  Several^ 情愿`  判处有期徒刑b 信阳d  ▁mixingf  ▁specifyh 能屈j amptonl 内经n  electionp raitsr 援朝t  在当地v 的金x  ▁shippedz  ▁philanthrop|  ▁radi~  基础设施建设  直播带货  ▁Rams  首先是  在过去 jjjt  ▁protocols  在外面  ▁interventions 前夕 vari  才能在 对称 fields 宗亲 ▁Tea 徒弟 孕育  ▁recipients 常吃 ▁GB  ▁horn  的人口  ▁intend 斯科 arel  ▁Rent  ▁Crystal  停留在 先烈 客房 hm  活动现场  ▁indication  开玩笑 posium  ▁Thai  ▁Hours xue  足不出户 并以 网购  intendent  会议指出 建于 hyd  ▁Spencer 救人 间盘  沟通能力 河镇 dn  不动产登记 ringe  ▁Equity  patient 办好 泗县  这是我 业界  表达了  ▁Terror ilia  ▁practicing 嘲笑 undry  ▁Notice  脱颖而出 到店  针对性的  中央纪委 hong EG  受到影响  ▁slavery 八月 顽强  ▁illeg  重点人群 执政 Ma affe" 四面$  打底裤& "@(  进入到* 不言,  已全部. LR0  打喷嚏2  知道自己4 不息6 坦然8 ▁è:  ▁Memphis<  ▁declaration> uli@ 日出B capeD  ▁OpinF 周刊H  汽车站J 龙门L FacN augeP  不接受R  ▁radarT 操控V  简直就是X  我们将及时Z 卡点\  ▁tailored^ 超载` 世事b  Referencesd 这儿f  长大后h aintedj  有这么l 花呗n  是一项p 文人r  胶原蛋白t 假装v  ▁citingx  短期内z % 著作权属归原创者所有|  opening~  更需要  ▁approve Bur  ▁adjusted  ▁json  美术馆  ▁porn 合约 路况  各行各  ▁Multi  ▁charitable ICT  就在于  常住人口 东坡  ▁Horror 精度  ▁theaters  等功能  ▁merc  ▁Brady  ▁apple  ▁grave eca WC fty 名下  ▁Philosophy 年满 恐龙  合作的 auf  ▁inequality 徒步 周某 税款 seen 敦煌 ifiers avil  ▁Bennett  充分肯定 ▁т 深层 确需  ▁Bour 精密  ▁Joel 皮书 ▁pad  生态环境保护  落到实处 蒸发 types 个大  ▁вы dj docs  中心小学  ▁Nursing mag  ▁rewarding  ▁mapping  的女生 thread  ▁Ancient 苏宁  方向盘  ▁handles  ▁facial  ▁Harm  ▁Gender 大吉 药房  路上的 弥漫  ▁Fans  留下遗憾  交通违法  ▁Lloyd  ▁optical" 占领$  ▁soundtrack&  空间的(  ▁toug* esity, impse.  小贴士0 awei2  关注吧4  得益于6 翻倍8  ▁accessibility:  ▁freshman< 到场>  在这一@  招聘职位B 万万D 气和F 稳健H  ▁pairsJ  一小时L ycN  期间的P ▁AnaR  ▁DifferentT ▁BagV  ▁HinduX 划定Z  同时还\ Kevin^  ▁unlimited` playerb 唐朝d 桥头f  西游记h  精装修j wedl  这次的n 交际p osuresr  ▁Miningt 县的v  会增加x sharez 渭南| 特产~  哈哈哈  本人身份证 湿疹  奋斗目标 讲课 ipes 网传 iploma 团团 hash  电信网络诈骗 AH  家门口 小明  ▁Railway 邻里 rants their  ▁reviewing  ▁stainless 外汇 床位 olia  ▁orche 沿途  机动车驾驶证 一案  ▁Pride  ▁overl 号称 查实 桂花  多一点 sal  ▁assessed 化为 avier  ▁Miles  ▁therapeutic 票房  科学研究  ▁Infrastructure 功课 大一 before 不计  ▁Choose  在这样的  ▁shade 主营  ▁Status  ▁breakdown  容易被 ▁WHO 市中 视听  ▁Patriots inis Buffer ▁Hob  非常好的 ▁{}  卫生习惯 打压  ▁Trial 刻苦  着自己  ▁glasses 风力  华盛顿  个国家  ▁buffer  ▁Parents  服务热线  ▁migr blue  ▁terrorists 根基  普洱茶 私信 足迹  也是有" 现货$  ▁lover&  ▁fever(  ▁fights*  ▁boring, Land. 呼气0  粤港澳2  amation4 韵味6 values8 Listen: 李克<  ▁bell>  不小的@  ▁prayersB 券商D  容积率F  ▁YorkshireH  ▁DriverJ ▁SFL  桂林游览时N  ▁jacketP 产区R 讲的T  森林公园V ▁UPX enesZ  ▁withdrawal\  第三步^ Small`  详情请b 会是d  ▁Rangersf 在最h  进口冷链食品j 增幅l 争做n  ▁hoodp 开支r 但从t  backgroundv  ▁fictx  欢迎留言z  只要是|  环境保护~ 看你  乌克兰  呈阳性  ▁shifting 李佳 秸秆 adays  ▁cache 小王 规避  小白理财  引发的 alone  ▁formally uity  封闭管理 沸腾 except 高出  ▁Leeds  ▁darkness  ▁Broadcast  ▁genres 运算  ▁Applications 不加 中途 指尖 触碰  ▁Bulgar 服药 排泄  ▁то hey 肉眼  红色的  ▁iTunes 顽固 adding  ▁Behind  永远是  诸葛亮 随处  其中有  紧接着  ▁questioned 不实 chaft  ▁uncomfortable contin  ▁inaugural 沧州  就会出现 精气  高分到低分  ▁Ways  ▁parade 下乡 渲染 环路 Come 欧元  ▁tile KM  ▁Laun bind  重大项目  ▁Formula  为保证 偏方  综合整理  的自然 patch 根系  社会责任 fix 抢劫  发脾气  ▁Venezuela  抗美援朝  厦门市  喜欢的人"  ▁prosecutors$ ottage&  满意的( Econom* 非典,  ▁shelf. 提拔0  县委副书记2  ▁immigrant4  亚马逊6 手掌8  ▁spouse:  ▁varying<  ▁culmin> 守望@  ▁BrusselsB VIEWD  组织领导F  ▁reconcH 魔法J  ▁RoleL  ▁ChromeN 容纳P  里的风景R  项规定T ewideV avinX  ▁THISZ 女的\ 研修^ kward`  ▁Bloombergb 宣判d  ▁outlinedf 内存h 商人j  记得前些年l  ▁attributedn 护照p bachr  的正确t  ▁sellerv  ▁landlx 在做z  ▁extending| 互补~ Very 只剩 造假  ▁beaten 日月  现场报名  ▁applicant  他认为 宁愿  核心价值观  important  昂首阔步  有限的 低下 电磁  ▁Bradley 李嘉  省委书记  什么叫 amon  的小伙伴们  ▁strongest 下巴  正因为  都有一个  iblical  奇特的造型 掩盖  ▁Guest  的生长  高层次人才 uson 大厂  记得前些年我到  ▁united 韶关 Russia ▁EC  ▁Ranch 胶囊 阅卷 ▁Hur  ▁Bach 开来  就必须  由此可见  发展规划  ▁transfers  ▁Ferg 看图 全党 XT kees 与否  ▁specifications 化肥 东至 中雨 ▁Pom  年龄在 issan  ▁praised 　♪  ▁indie  平常人低 年会  ▁Teen ende  大家一起 淡化 红尘 串联  经验者优先  ressing 杨幂  深深的  ▁Minneapolis  到底是什么 voc ▁IF  第一课 时而" 建军$ 衣食&  ▁robots( 说他* 厚重,  检察官. kj0 亲身2  ▁Gates4 . '遇到了一个个美丽自然的石洞6 分娩8 weixin:  洞口都比< 车企> 可控@ 天文B  ▁foundationsD riciaF 产科H  发展有限公司J 水源L 必胜N 波澜P  ▁belongingR 泳池T 逃生V biaX  ▁hometownZ  virtual\ 吸入^  神经系统`  咨询热线b ▁Nind  爱国卫生f  ▁appeth 墙上j  发生以来l  ▁interruptn  Educationp 驾校r  鸡蛋价格t  洞口都比平常人低v  ▁Furnx 稳妥z  可能导致|  ▁Midd~ 该车 ployed  ▁cuisine 观影 尔斯 打架 )$ 抑菌 真假  roductive 雨天 suits  ▁Expert 还被 时效  ▁circles WR  祝大家 心肺  开幕式 params Mur  ▁ministers MBA  更何况 买单  国内的  现在已经  说明书 脸部 填充 属狗  见附件  ▁penet  ▁Montgomery 饼干  保护区  弯下了  空气质量  ▁acoustic 分红  ▁Newton  ▁Villa JS 的第 感人  压力大 anç  ▁Stein 中信 Know  ▁cartoon 窗外 Assert  aterial  告诉他 ' ******************************** Series  ▁Independence 杨柳  ▁Grammy ▁NGO  ▁booth ▁bip  Comment 一地  数据的  ▁cattle 二字  有温度 国网 haul  奇特的造型充满了  ▁flags 收敛 迫切 就用  ▁Venice 举例 离去  ▁rehabilitation 铸造" 区市$ 拎包& 有奖(  ▁Kumar* ▁Cox,  ▁retailer. msg0  老师微信2  全心全4  ▁diplomatic6 save8 家和:  停电范围< 置换>  资本市场@  ▁centsB 寡妇D  的用户F  ▁elevatedH MYJ 芬芳L  ▁reluctN  ▁landscapesP 华丽R 名誉T 清扫V  当然是X  昆明市Z  结果显示\  ▁intu^ 养分` 畜禽b  ▁airlinesd 主食f Karh ▁rodj  ▁burgl  ▁learn Graphp 肝炎r  高效的t 部落v 承德x  ▁contributorz  战斗力|  神奇的~ nu 上饶  ▁catast ці 心怀 规程 女星 轰炸 Carthy Entry  艺术品  keeping 切片  影响力的  ▁characterized 流出  ▁disposal  ▁matrix  ▁Gets ▁hij 有为  把这个 窒息 far 政委  ▁warehouse " 记得前些年我到石林 化市 到你  时间表 摧毁  ▁Equipment 小数 电竞  十分的 限期  Building  中国青年  ▁Frances 刘备 略有  我还是 缤纷 宣誓 事了 把钱  ▁relaxing  ▁Lodge  ▁queries 瑶浴 漫漫  ignment  ▁Pret  ▁newer  ▁interim  更适合  总承包 ▁Wag %~ 配料 ▁len  ▁Joshua  ▁Videos  ▁Consider  ▁individually 抢占 mos  ▁studios  ▁Rebec  精致的 Turn  parents 鉴赏 五彩 无畏 opped oys  需要注意的是 敏锐  ▁politically 率达 关切" feat$ 唯独& 女方( 单人* 多半, 肩负.  了一个道理0  民族团结2 % 新型冠状病毒肺炎疫情4 ambda6  ▁Funeral8  为单位:  ▁carriers< 颁布> 氨酸@ 在心B 马店D  学习成绩F  中国经济H  副秘书长J  可能有L  古诗词N  只能说P 体型R  的新闻T  福州市V  产业化X  ▁guitaristZ  苏州市\ ▁Cit^ 读完` 亲们b  网邓州d 就跟f 德华h tarj 着我l 力气n  ▁rehep  回来的r  ▁eternt 涉黑v 改版x 那位z sanwen| 暧昧~  ▁Plaza  根本没有 Brian 增量 Labels  中医认为  ▁horr 彩礼  年轻人的 “,  写作业  ▁Principal  电商平台 汕尾 Friend ▁Meg 山海  ▁Norwegian ▁Mis 西洋 口头  ▁suggestion  四六级 农牧  在这些  ▁motorcycle 染色 Senior Writer  征途上 源县 Tele  一如既往  ▁Cruise  ▁breed 疫区  artifact ▁Pos  研究报告  ▁brave 你来 紫砂 落叶 INGTON  ▁tales 谎言 丰县 bag 跳动 elia 纹身  ▁Colin  文章阅读网  ▁messaging ▁і 当当 ི་  广大党员干部 ор 分队  rington 祖先 精湛  ▁murdered 高架  ▁Vers  都低下了  ▁nominations 起初  ▁bump 安稳  始终保持  ▁Papers 无悔 ▁[[  心理咨询师 utable 誓言  计划的 FILE  ▁scared  阅读量" 龙华$  昂贵的头& izarre(  ▁Newcastle*  ▁treats,  ▁Sullivan.  ▁Welsh0  ▁illum2 辟谣4  富有的人物6 olving8 投保: 做强<  无论怎么显赫>  洗手液@ 反击B  这个地方D uablyF 共济H 在岗J 涂料L  ▁statisticalN rinaP  扎实做好R  ▁TouchT 赵某V 高高X 入味Z 寄生\ USE^ 树人` negb  都低下了昂贵的头d  农业生产f  ▁Wereh  ▁beatsj ▁Kral 美化n 背叛p  最贵的腰r 种族t  去探寻石洞v  弯下了最贵的腰x  必要时z  停课不停| handle~ jack wd  ▁disclose 融化 ▁är 克的 开考 years  价格为 死于  ▁readily  ▁notified O H詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘 肖战  ▁skip  ▁Thought  ▁Educational  法定节假日  ▁fancy  ▁Valentine 书本 庄园 痛经  ▁soap 契合  ▁builder  available  ▁meantime 逐年  Security  ▁puzzle ▁ore  ▁counting  ▁Jeffrey 年中  自觉遵守  ▁cruel 城东 好吗  ▁preservation 北风 寒冬 anto 克制  的收入  ▁Period 微风 KA Martin " 去探寻石洞里的风景  其中一个  ▁timeline 和华 万分  美国总统  度假区  学习交流 复活 抽象 unal 心想 用房  ▁Operation 春光 hz 总计 得不  ▁inbox  ▁regulated  ▁suppose  认真落实  主办单位  ▁attacking 的情 Sarah 打拼  的一条 章程 items  ▁calculate  ▁cooked 影城"  ▁Province$ 升温&  好不容易(  ▁Pearl* ▁Т, 个子.  王某某0  ▁profitable2  ▁Chuck4  ▁occurring6  倡议书8  等各类:  ▁beans< places>  ▁equation@  ▁ProtestB  efficientD  ▁SendF  ▁ContribH  只是在J  ▁DragL ailureN 它在P ▁uptR 重生T  ▁RalphV  ryptionX 上架Z 逛街\  还不如^ 分组`  Positionb 抽检d  ▁Renewf  脱贫攻坚战h 右边j Suppl 佣金n 后排p 春耕r 典范t primv 妥妥x  ▁Meadz 孕期| 辩护~  非常大  不传谣 云端 喜庆  面积约  ▁avoiding  通告如下  ▁prost  很多人的 洋溢 章节  的秘密 也对 jab  也因此 向阳 明细 1 *请在后台留言联系我们进行删除 还想 置身  王女士  ▁Diversity  ▁cryptocurrency 考官  涉及的  ▁Message 猪瘟  ▁viewer  立方米 枕头 ourses 患上 lar 开后  ▁Applied  处处长 fre  中老年 为宜 buy oval Lim aires 校考 houses ▁Maz  ▁sever holm  ▁Floyd 魔鬼  ▁spirits 体贴  搅拌均匀 拖拉  都是有  农业农村局 rivacy  桃花运  Present 调和 并处 街坊 coll 左边 名气  人人都  一直没有 出路  ▁Shore 弯路  ▁algorithms  的决心  带头人  ▁indicating 红花  instance  ▁Registration  ▁suspects  ▁shake HI 购车 振奋" 泰州$  都不敢& both(  ▁lightweight*  ▁taxi,  ▁grandchildren. 池塘0 爱我2 vised4 日产6 Intern8  部署要求:  ▁lined< 会和>  不断增强@  截止到B ▁CAND  ▁craftsF  的一份H 冬瓜J  现如今L 参展N 交所P  ▁coupledR  优化营商环境T  ▁introducesV 指向X  全面建设Z 白沙\  发挥了^ 轮廓` 起始b wbd 西站f  我母亲h  他们对j ( !和新疆生产建设兵团报告l 把关n  ▁Toursp  原有的r  ▁varietiest 储蓄v 本职x  的保护z 听完|  ▁bridges~ ▁Gaz  ▁financially 冰凉  ▁watches  ▁Knowledge  ▁substitute  ▁dignity  ▁Walking  ▁drums 检索  ▁electoral  ▁attitudes  ▁airports  一阵阵  追踪到  ▁feas  ▁firing  在感情 onc  ▁neat uced  ▁evaluated 默认  ▁Overall  和规划  新闻联播 坚韧 nings  在医院 北宋 + $请您先点击上面的蓝色字体 ▁Mul 变身 跑了 龙山  有关情况  ▁vitamin  累计确诊 大笑 流泪 iOS Soft  ▁Prix argest  的最后  ▁Herald  ▁este  的事件 新颖 点到  ▁rack  的危害  ▁FIFA  ▁quad pad  的智慧  课程内容 关乎 十月 毛囊 蠕动  ▁Pradesh lining  ▁Naval  ▁Database outed  ▁crafted  Website  为学生 ▁CIA  ▁sentiment 批量  ▁pepper  ▁inspir ▁BA 原生  听取了 人而  ▁Monte rov  ▁fraction 开庭"  试验区$  ▁attach&  umbling(  特别好*  但实际上,  ▁besides. 一滴0  时不时2 ▁Г4  只要有6  ▁washing8 公馆:  ▁rhet<  ▁Burg>  ▁comparable@  ▁AlfredB 香菜D MGF  并且在H 紧扣J ▁SuzL  ▁memoirN  关于加强P  危险的R  ▁statsT  ▁prosecutionV  放不下X  一直都是Z  ▁Marcus\  Christian^ 可供` 阻力b  我们现在d  ▁impliedf  ▁bookedh 献礼j  ▁Kashmirl añn  onsoredp  ▁botherr  ▁Reportert  ▁partneredv  ▁cybersecurityx  基本养老保险z  拥有了| 省会~ 树枝 较差 谨防 ▁Enc  相关内容  ▁padding :\ dp 谱写  现代农业 紧身  Template  对所有  ▁licenses 同舟 水流 足足  雷阵雨  资格证 repeat 我妈  ▁presidency  ▁alternate 低者  ▁census ín 主的 置于 前十  ▁structured 侵略  喜欢就  ▁openly  竞争对手 处长 教养  ▁Rebecca  得头破血流  ▁tensions 他却  ▁pole  各级各部门 oi ры сть  卫生部  生物学 字号 建模 Draw  他们都 属蛇  ▁Derby  ▁Location  ▁disclosed  今天我们 良性  ▁Florence  ▁promotional 闪光 uated jz 店面 毒性 Ber  ▁glimpse  ▁autonomous 范畴 ESCO  ▁deleted encia  ▁nicely  不少网友  老党员 gement  ▁sheer 千秋  ▁hier  ▁Session BJ 冷暖  小长假"  中心医院$ 王小&  ▁scholarships( ▁Vit*  ▁html,  ▁Audi.  resource0 呼应2 斑马4 城关6  也只有8 ablish: ▁Pi< 千金> '],@  ▁mobilB  微课堂D 航拍F oltH EXTJ  ▁QuantL  ▁WallaceN  风景和P  我发现R  ▁SimplyT 图表V  ▁hungryX  ▁logicalZ 梦见\ uka^ Stop`  很明显b  ▁announcingd %(f  ▁Innovh  minsterj  Containerl 家有n rezp 了的r ERVt ▁CPv  ▁sheepx  ▁Rushz  恶势力| 柔情~ 眼部 可知 奢华 苦恼 小班 下垂 淮北 ▁Daw  一点都不 photo  ▁Hungary  哈哈哈哈哈哈哈哈  ▁superhero  广告合作 残忍  ▁Hotels Texas зи 点名  容易出现  ▁enters  必须是  ▁predomin 敬意 内防  ▁monetary 科大 钻研 Smart  信息发布  微信平台  communications Left REL  ▁regret  食用油  ▁siblings lesh  看得见  ↑↑↑ Serial onut 语气 上台 一间  ▁upgrades  ▁decorated  ▁Bent vation 手持 Making 处事 bris  市委组织部  都在这里  ▁Garcia  ▁heated 找回 arett 门面 oka 召唤  ▁printer  ▁reasonably  不要在  完全没有  ▁listened  ▁charities  面向社会公开招聘 曙光  ▁adviser  ▁Nobel  那样的  olitics  ▁Mull 兴旺  最好是  ▁permanently  ▁Carn  ▁Modi  ▁Weekend 猕猴"  就不是$  的综合&  idation( 南湖*  ▁Eddie,  美好的一天. 标语0 apor2 向西4 Own6 所述8 不舍:  经济损失<  ▁Suff>  ▁Portuguese@ 爆棚B  ▁WingD  的职业F  大城市H enzaJ  ▁examinesL ▁RAMN  的性格P 搭档R 起源T 逆转V  研究会X  录取分数线Z 乐队\ 过好^  ▁Tools`  ▁declareb 戏曲d 硝酸f  ▁Ugandah 党群j  工伤保险l emian Commonp 出场r  ▁surpasst 自私v  学校和x  考生在z  公交线路|  ▁cloth~ 各式  ▁appliances 很低 石斛 迄今 推销  ▁Powell  电动汽车  ▁Schw  ▁politician  当成了  ▁Dawn ▁ACC anium 英才  ▁behaviors 之光  ▁Weight 告诫 陪护  ▁sacrific  关键是 ilers  ▁mice 人脉  ▁theoretical 克难  的语言  交叉感染 三项 ▁qui  ▁extern 播报 财源 华山  ▁Stef  application 妨碍 acey  ▁accus 往上  浙江大学  具体时间  第一中学 吕梁  的幸福 全书  治疗的 avia idas  ▁diameter 有责 深知 协和  在全市  如果你是  ▁Haven  ▁clinics 相近 揭示 ▁spy  通过网络 初次  ▁Julian 分许 界限  教育考试院 属猴 间隙 狐狸  免疫系统 讨好 拦截 迷信 选聘 ▁SS 不看  春节假期  ▁shoe  ▁preserved  洗干净 degree"  etitive$  ▁farmer& 了她(  ▁Chev*  龙头企业,  ▁boots.  的形象0  中学生2 转到4 子和6  低头弯腰8 redict: inv<  不错的选择> 后在@  的重要组成部分B codD  ▁filtersF  污染物H  ▁tunedJ 世家L  understandN btnP  ▁organizationalR releT  中之重V GrandX ptrZ  College\  ▁positioned^  ▁Workshop` 变迁b noted  ▁aboardf 以南h  职能部门j 的去l  抗日战争n  一点也不p  红十字会r  sessiont  ▁highlightingv  服务有限公司x  ▁plantedz  采取了|  ▁manually~ 我局  的角色 北上 rises  ▁lately  与健康  ▁Stuart  ▁enrich ке  ▁remed 口中 递交  ▁lacking  ▁Basic 白白  ▁Arabic deal ▁Eug built  准备的  ▁pond 压器  ▁Heaven 淮南  ▁Holdings ▁Dow  欢迎关注  中国人的  关键字 GD  澎湃新闻 生钱 itten  ▁dilig  ▁Newsletter 预示 pres 已然 传导 clipse  ▁credentials ENCE  ▁Gand ▁Cli  ▁accent  ognition 衍生 前辈  投资理财 大选 plane  实现美好 一见 了就  ▁similarly 科级  taining stown  日本的 ylon ORS  当下的  漫长的  ▁Imperial  黑名单 集市 穿的 ▁FR 有情  ▁optimistic amo  是一次  ▁cush  跨过这道 ▁anx  ▁searches 万年  意想不到  ▁fifty 势必 CNN  的话语"  ▁wound$ 便可&  ▁Carbon( 开播* ayan, 神圣. 可报0 嫁给2  政府工作报告4  说的是6  ▁Crypt8 潮汕: oner< 为她> ▁%}@  ▁FourthB  ▁excuseD  查看更多F rowserH 加密J 迷惑L  实施意见N  ▁holyP  智者懂得R 出不T  ▁definingV  ▁ForwardX 赔付Z  ▁recession\ 设想^ 集镇`  ▁woodsb 三层d  ▁livestf 对人h  名学生j 前者l 洋洋n  詹姆斯p 奇妙r ▁NSt 太行v 辛辣x  ▁coordinationz  建设工作|  各地的~  业务员  ▁ISIS Cur zn imag  ▁Appeals  副教授 ①　  ▁Productions 觉醒  在地上  ▁Occup  Abstract  为核心的党中央 数码  ▁finishes  ▁shocking olan 这在 两项 erred  ▁steering ▁Nad 躲避  ▁lengthy  ▁jumping  办公软件  数字经济  ▁Repair 共和 抢先  ▁predictions  以前的 WD Bundle puter  等行为 免冠  ▁adop 奶油  Overall jet 赏花  这两天  去年同期  干部群众  着一个  ▁artic  ▁Links 奈何  ▁Foods  ▁Chad 中东  ▁demanded  如果能 以身 Ult  与发展  是不能  的这个  ▁undertaken  ▁celebrations  比特币  ▁Diana  ▁forefront  日中午 很不  ▁Craw 器件 吸气 美妆 下架 名言  远大目标的  ▁hunger 真挚  自己的孩子  地铁站  ▁purple POST 虹桥 吸纳  驻马店  ▁pastor 反之  ▁Transfer 士官 立面  ▁participant 油烟  ▁mechanics idae  关键的门槛  才能跨过这道  ▁strengthening  library Lee  先进集体!  方能高" 了近# ayed$ ▁EL% 党纪& ( !才能跨过这道关键的门槛' 单日(  通行费) 无为* 而有+  ▁territories,  方能伸-  iovascular.  ▁broadband/  最近的0  的一致1  能低者2  ▁Manual3  ▁sweat4  Android5  能屈者6  也是如此7 吊销8  温州市9  ▁Peru: 复利; 都可<  齐心协力=  国家卫生健康委>  ▁intriguing? Header@ 油田A  羽毛球B 侮辱C  ▁spaciousD 离合E  的代表F endyG  实现美好理想和H  远大目标的征途上I 多岁J % 有时需要我们低头弯腰K 4 -实现美好理想和远大目标的征途上L % 有时需要我们昂首阔步M = 6在我们实现美好理想和远大目标的征途上N 乙肝O 风筝P  ▁TommyQ 霸王R  ReferenceS seminT kelU squareV prusW  绿色发展X ItsY creatZ 一寸[ 兵役\  ▁Diet]  ▁Religion^  真的太_ 农机`  ▁startera  ▁helicopterb 后可c  分左右d  ▁Familiese Hopf 宋代g ▁Nash 镇江i ▁jamj  ▁einek  ▁Coronavirusl 味精m  问题是n  ▁seafo 车门p 结晶q 离世r  ▁weekendss  ▁stylt 中也u 网吧v 的吗w  ▁nearestx  的费用y urablez 住所{ Send|  ▁exchanges} Sett~ 穿出  ▁Ster Need 蚊子  ▁recalls  ▁succ 并肩 大姐 ной rugg aque 收紧 警戒 宁区 imen  这样说 讲义 眼看  世界里 考题 ▁Yam  ▁fortune usa  说明了一个道理  ▁slots stick  资源的  ▁Luis  自己的生活 ▁CAR 排在  第十五 artet  arantine 字样 自习  ▁disappeared  引发了  ▁Indianapolis 诚挚 技法 肉的 AX 低至  这么多年 城建  ▁sooner  arently 校招  巴菲特 一模 特邀 珠江 itus 我没 纸币  ▁Lynch  老旧小区改造  ▁trapped illery  ▁Alexand 持平 ный 呼伦  ▁Ahmed  高质量发展的  　—　  ▁socio  ▁Myanmar ruck  ▁imported 素食 配电  ▁Emirates 沉积  全国的 rish  最终解释权  ▁succeeded 月入  不聚集  ▁wellbeing office  ▁healthier Sex  新世纪 畸形  ▁substances 橘子  刑事处罚  突发公共卫生事件  ▁Networks  ▁nerve 可观 ako ansom 限速  ▁Gospel 格兰 灰尘 anic 首饰 vip  ▁stance  ▁referendum  ▁Nova flu  ▁males  ▁binary  的认识  新的一年  本身就是 响起  ▁grief 处级 飙升  不信谣  较多的 预言  ylinder eward 案情 French  ▁Actor  ▁regulators  你知道吗  校园招聘  ▁assessments  ▁ambassador  ▁YOUR 背面 atican phabet  和诱惑  ▁costume ▁Koh  ▁bikes  println 隆平   赞同其   风险等级   ▁topped  Wal  八九  理解和 年货 ▁oh ▁Sor  ▁consistency fang  ▁cope  ▁masc oan  智能制造  Company 西县 bourg 安阳  就不知  的需要  ▁lighter  ▁Herm  找我!  过一个" inas# 跨界$  ▁Geneva%  ▁imagery&  ▁Drop'  的学员(  ▁kits) 六一*  ▁modeling+ 争执,  庆祝建党-  请及时联系. 追捧/  ▁progression0  ▁browse1  ▁Question2 处世3 在某4 大名5 崇拜6  ▁Writers7 当好8 赏析9  ▁clouds:  ▁Ultimate;  全面发展< 嘴唇=  ▁genius> ritis?  ▁circular@  CountryA 血症B  responsC  ▁administeredD  ▁rotationE 天花F  特别提醒G  ▁speciallyH MenuI  ▁IncomeJ  fortunateK  ▁AthensL 晶晶M 迷你N  志愿军O  ﹌﹌﹌﹌P andezQ loginR 读了S ▁MTT  ▁administratorU ouxV 当归W 哲理X : 3这些看来都是生活中再平常不过的小事Y  发展战略Z 立志[ gon\  ▁Charter] orp^  此时的_ 美联`  ▁provesa Shopb 生子c  却说明了一个道理d  ▁Preste 请假f  化学品g ceansh  ▁lakesi 收盘j 昏迷k 网盘l  养殖场m 开票n orieo 果蔬p >&q ▁gelr  ▁agreess pitt Beautu neathv STRw  就是这么x  无论如何y 大树z ilogy{  项目经理| 峰值} 芳华~ Would 但又  ▁machinery 空缺  但没有  ▁Bailey 沉重  Getting  ▁complain 中立  ▁endeav NR  ▁refreshing 生前  不知是 光阴 cx  有这样  我喜欢 Los  ▁nominee  电子邮箱  ▁Bookmark  ▁doubled  ▁committees  ▁Kend ▁Fri  ▁realm  ▁illustrations  ▁patience 时艰 Plan Adam HTML Names  ▁crore 国会 溪县 痊愈 私募 勋章  olecular  新高考 应变  有很大的  ▁verdict  即日起至 元起 ▁FAQ 瓜果  深厚的  文化艺术 换了  ▁oversight ▁Vac 切忌 挽救  ▁ninth  ▁Hack  Central  ▁commander 严惩  ▁Filip 棵树  Private  举行了  ▁Orthodox  自动放弃  多样的  ▁locks 篇幅 oyle  几乎没有  ▁Throughout etti 教务  教育孩子  //////////////// 火花 默契  ▁flip Stack 也得 Site  ▁Away attery  ▁Amanda  Without ▁MC  ▁envelop  看一看  参加活动 circ  会发现  在国际  ▁Client  ▁quarters  设为星标  ▁criticized  交警支队  ▁outrage  里面有什么  ▁literacy ▁RSS 一秒 六安 豆浆 通话  ▁Nazi 同伴  ▁flew  的操作 的古  ▁administrators  ▁Alpha 一厅 我将  市场需求  ▁realise usr 豁达  ▁tires 四五 ité 稀释  iratory  等项目  ▁ridiculous  手中的  成为一名  ▁Township  课后服务 WO  本科学历  一起来看看吧  ▁flooring Enc Yeah price   做生意   统编版   ▁basket   ▁epidemic   另外一个  ▁utilizing ше  　||　 盛开 ▁OH  ▁performer hui  内容包括  有的是  ▁Democracy  不法分子 笑脸  ▁flavors 七个 树叶 多重  ▁Claire 无敌   的生活方式! 载人"  报道称# sure$ apple% 我只& 举个' 二年(  太阳能)  ▁Tall*  ▁classification+  ▁mathematics, 迟到-  ▁lectures.  ▁midnight/ 你去0  ▁headline1 副业2  通过微信3 ( !疫情防控和经济社会发展4 小猪5  ▁Supply6  免费提供7 山顶8  最好不要9 捧着:  感谢您; ▁IRS<  法定代表=  ▁infectious>  给你的?  ▁JSON@  ouncillA  安全教育B  建设单位C  是我们的D GermE  ▁lendingF ioxideG  regularH 躺着I SixJ ▁SoxK 无边L 亚太M  ressiveN initeO 奥特P  应对新冠肺炎疫情Q  ▁blockingR 党籍S  endantsT rivedU 换来V ▁KosW ▁RafX iblesY  什么事Z 愿景[ 重疾\  ▁spac] scri^ xton_ 即止` 受限a 痴呆b Badc  的到来d Jace 组件f 一届g  主办方h  看一下i iottj  ▁anchork 所欲l  ▁evolvem 打下n  ▁Zimbabweo 阶层p  车辆的q Utilsr  ▁renovations 潜能t  深化改革u  ▁importsv 谈谈w  ▁exemplx 专卖y  呼吸困难z  ▁Benefits{  感谢您的| 本色} 即为~  一般人  全心全意  ▁deport  ▁spends  ▁ordering 综上 ▁Duc  ▁Motion 须在  甚至连  ▁Maps 估算 外人 自发  ▁delicate  artifactId  ▁arguably  ▁recognizes KEY  ▁taxpayers 盒子  出入境 二月  ▁Kick  ▁diagnostic ahan  ▁Waters Users  ▁Cher  事故发生 一想  ▁runners  ▁cher 断了 仍未  不住了  在我的 文史  ▁Cricket  ▁criminals ▁RS 大爱 表率  ▁utilized  不值得  ▁Poetry  相比于  到底有多  金融服务  以实际行动 吸附 ▁PT ▁jav 之战 打的 ▁Yu  此次活动 三中 ▁RT  整体的  经纪人  目前已经  ▁Guild orsche listen  ▁donor  ▁planes ▁Dh  ▁recommends  ▁imagined 警情 无视 聘请  ▁compounds Wood 泸州 DAQ  老师和 iating  ▁solic  监察调查 shaped  如何在  了不起 一圈  进行检查  ▁Tags 就将  工程院 批改  年收入 无色  相关要求  多年前 Foot Democr nom  ▁touching 超强 台词 兴起  更多的人  新冠病毒疫苗接种  ▁workout ▁shy 改建 oku jl  具体要求  面试时间  Government  ▁meditation  ▁recognise falls  ▁Memory  enaissance 香油 曲折  国务院办公厅 平顶 ’:  ▁sensitivity 我最  ▁Entrepreneur 走红  新中国成立  碰得头破血流  ▁unfold ▁env  年秋季 utils 胃口  部分文章 温情 阴虚 fx ▁Nar stad 抓拍  cannot   utenant   黑色素  卡车  高了  ▁noon  点左右  ▁amended 搬家  气象局  ▁Zach resses 插件  我们还 病史  本质上  不一致  ▁harness 投机  我们必须 yll >, 吃法   从现在!  防疫措施"  ▁railroad# 大巴$  ▁justify% 银壶&  ▁Lynn'  ▁contributes( 心和)  ▁touches* wheel+ aders,  原来是- 给的.  习近平在/  ▁merger0 李子1 量身2  ▁Canadians3 两面4  ▁Tribune5 ▁Aub6 Final7 师兄8 润肺9  事业单位招聘: 河口;  长大了< ▁ups=  ▁licence> cheng?  ▁Athletic@  ▁RefugeA 心地B  眼前的C PingD  不好意思E ▁MtF  environmentG  能看到H 元抢I 谈论J asmaK 家校L  ▁undergoM 曲靖N 山大O  一不小心P  ▁gripQ  前所未有的R 有爱S 雷电T 吃不U  ▁PalestiniansV USDW 起草X SummerY  ▁allergZ mil[  推进会\  account] 蓝图^  新时代文明实践_  镇政府`  appropriatea 楞严b  ▁freelancec 术语d  中西医结合e  Studentf 多点g totalh  ▁Yangi 果汁j  人民为中心k 谷歌l 高楼m ▁NIn  容不下o 长短p  ▁Revenueq Varir 追寻s 非要t  ▁filmmakeru 仍旧v  小时后w  的模式x  启动仪式y Celebrz  农作物{  分以上|  ▁exhibits} 换成~ 亿人 OND 之日 放生  手续费 来信 Prem  交通便利 一盘 下心  ▁insisted 以其 澄清 并购 ▁Jur  目前为止 dx 有缘 成才  ▁surroundings  ▁Illust  ▁irrit  即可购买  ▁founders  ▁stepping  不同意 ▁cod speak 挑剔 旅途  ▁Wiki  大力发展 venile  ▁Serge 扭转 饥饿  奥克兰  ▁soup Kat 迈出 ▁meg  数字货币 徘徊  为目标  上一句 也无 自首 瓷器 顺着 出厂 考评  京津冀 母校  ▁vivid  ▁filmed 吸取 talk  变得越来越 流调  ▁afterwards 深远 ovak 揭牌 考取  大会上 运城  ▁confusing ▁gru rapper 致以 + $习近平同志为核心的党中央 用在 紧凑 侦办  外包装 ximity  ▁referen 十佳 astian  运动的 utton  ▁Flag  ▁galleries 顺义 forget Gal  经验和  ▁pine  present  ▁prevented 易懂  ▁congressional  是全国 也算 入市 向来 )—  ▁Joan  ▁Hour  ▁Score  ▁advancing  ▁Southwest  ▁securing  Chicago  ▁Brend attr 养胃 十万  与管理 ▁OP  本身的 向北 eric  侥幸心理  ▁Earn  的是什么 周转  差异化  月开始 齐齐  ▁recreation  ▁teammates 锅底  ▁Function  如果想  中的一 耕耘  ▁Mann  ▁lesser  不自在  reading  ▁Coordinator  ▁forthcoming 爱与  会更加 ▁Tak  ▁wondered 而易  在工作中  旗袍  期为   ▁installations   低头的时候   ▁Taking 就拿  ▁Tamil 限购  都喜欢  正月初  看电视  营养物质 标明 ▁TD 盖上  ▁Eight  ▁comparing ▁Ord  ▁ensemble  技术创新  ▁Trinity 秉持 rology  夜叔! Height"  ▁rumors# 喷洒$  大部分人%  工业和信息化&  ▁altogether' 楼道(  ordable)  ▁reset*  社会上+  音乐学院,  在那里-  ▁Lopez.  ▁extensively/ 喉咙0  主题活动1  ▁lifelong2  就有了3  ▁Bears4 并为5 数千6  ▁Oracle7  上当受骗8  ▁Birthday9 china:  ▁Truck; 立项< 保罗= 而为>  目的在于?  ▁Infin@ 不力A  庞大的B  historyC 社工D  优异成绩E  UniversF  ▁needingG RyanH  参与到I  颈椎病J OsK  ▁poorlyL  ▁frustrationM moveN  工作单位O  主任医师P  registerQ  进洞口R  DetailsS  实事求是T 昆山U 五是V 煤气W 受灾X oquY  ▁InfluZ 养肝[  开学后\  ▁premie]  民政局^ assy_ 锅内`  逾期未a  ▁finesb 用最c  切实把d onoe 片刻f  ▁pilotsg  ▁intakeh 去皮i  ▁Notrej 多姿k 扭矩l  房地产市场m  ▁archivesn 序号o  希望你p  ▁Cyclq  老师在r % 奇特的造型充满了神秘s 断裂t  ▁corridu 尽显v 面板w  甚而被x Apy  就会被碰头z  ▁Engineer{  就会被碰头甚而被| 1 *就会被碰头甚而被碰得头破血流} 全村~ roc  ▁Ellen  ▁aiming  假如进洞口 " 假如进洞口时不低头 不漏  风景和奇特的  ▁conditioning  ▁Cann  ▁False  有时还会留下遗憾  ▁inning  风景和奇特的构造  日以来 ▁Roh  ▁fuck  ▁backdrop  好好的  的危险 ▁Hip  ▁Billboard 勃勃 之窗  贫困人口  ▁swift  这就是需要我们  ▁specializes  ▁teenagers opin  ▁awkward  ▁Libya Indust 功倍 endo 玉溪  ▁bitter  ▁Framework Scient 与他 hhhh 不作 agra  ▁outf  ▁sophomore 十条 Girl  本单位  ▁lasted 入校 + $这就是需要我们低头的时候  青春期  Explore 八方  是我的 三类 官僚 留观 留存 搬到  毕业的 Ten ▁cu 船员 一刀 翼翼  两个人的 九分  非常重要的 . '奇特的造型充满了神秘和诱惑 Low 懂事 行使  处理的 年仅  从小就  制度的  ▁Glenn 星辰  消化不良 安利 到校 到头 ▁Raw  银行的 半月 caster  ▁philosoph  假若不 濮阳  ▁implements 爱尔 过错 企图 让其 Jenn  年时间  ▁Hannah 淡定  抗病毒  就不知里面有什么  ▁remot vens 舒展  ▁Calendar  ▁Manufacturing 7 0就不知里面有什么风景和奇特的构造  主要领导 裁员  你若喜欢 actic 忻州 ▁dip 　#  发出的  数量有限 Indian 媒介  知道你  ▁Iraqi  ▁resistant 总值  怎么会  我们分享  假若不进去  performance 建华  ▁upgraded  关于开展  电信诈骗  ▁niche 牛津  ▁Ethics  ▁analyzed 晋中  ▁Pool  是你的 总台  觉得很  理论学习  飞扬  踏上  家教   ▁triggered  rable  ▁apprent  ▁moisture 雨雪  ▁solved  ▁panic  内容和  答案是 骨科 菠萝  ▁tweets  人际关系  备考资料  ▁Rescue 印章 shu  ▁STEM 得罪 小微   控制的!  交通运输部"  )——#  ▁pioneer$ 漂流% 黎明&  ▁informal' 说我(  在你的) 蒸汽* 冒着+  学英语, 转债- 再走. 多达/ 鼓楼0 摸底1 ▁Tag2  每天的3  报考条件4  把这些5  的战略6  市人大常委会7 婚恋8 植入9  ournaments: 南开; 外侧< iones=  ▁divisions> 色斑?  ▁Internal@ 手游A  ▁sketchB  ▁punkC ▁AVD  ▁homeownersE  教科书F  呼伦贝尔G  ▁kidneyH  了一段I  的唯一J  本科专业K  民主党L eatingM  不了的N  ▁meterO largeP  ▁improvesQ 百事R  抗击新冠肺炎疫情S 夜班T  李某某U  ▁IsaacV CrossW LINEX 但对Y  钟南山院士Z 为首[ OTO\  落实到位]  ▁catching^ 城堡_ 大河`  ▁Ethiopiaa  半功倍b  警示教育c 七天d  丝毫不e  合格后f  通过后g  ▁Encyclh 危急i 增高j 金沙k  ▁Sacramentol  ▁tenantsm 评议n 近年o ▁SLp 卡口q  不是为了r  ▁templatess  ▁Appealt  ▁AIDSu 发芽v  ▁supervisionw 帝王x  但是却y  ▁abundancez 地藏{ Width|  ▁constitute} sers~  ▁Clarke  ▁customs  封闭式 Eric  ▁Ellis  Stephen 再加  ▁Armed 俱全 太岁 ▁rab  养老院  的措施 流年 路网 ▁PHP  ▁metals 物料  ▁Feel  日报社  ▁destro  后发现  ▁undoubtedly 标段 缴存  个工作 偏差 ▁kne 小店  ▁permalink 川区  的合作  ▁biography  ▁Exhibition ▁zip  体育馆 拉到  各级党组织  ratulations  ▁autumn  很喜欢  以下罚款 水准 一网 率和  ▁sights  ▁Whats 日均  陪伴着你 闻名 ▁HS  ▁inhabitants Hu 番禺 遇难  信息平台 ід ieu  ▁Holocaust ▁Wu 上次  ▁shifted  实在太  没有办法 ▁Sof  ▁Munich  ▁Premium 冷的  ▁ceremon ▁{@ Cy INK  ▁Rugby  星期日  张某某  现有疑似病例 Van  但还是 城北 轻重  专项斗争 的茶 cas Close 通辽 阳春 ▁Jet  ▁Desert 一堂 上也  官方微博 rella 不忍  ▁sexually 卖了 在世  最后都是  直升机 IAN Ra 胳膊 那份 haus  ▁Recent  在国家  第一条 做实  ▁analyses 家村 ▁Oz 过着 ungen 与我 了然 戒烟 ▁Mol  ▁listeners 独具 micro  stylesheet 功底  ▁massage 率高 含金  ▁Teachers  医务工作者  ▁remod MIN empty  ▁rebuild 贴合 敏捷 Auto each nick 个县 astro 地向 评级   确诊为   ▁Consulting   袁隆平  回购  unct ▁Fo 闪耀 Nor azi  开展的  不在乎 固然  自我的  生活垃圾 举止  成绩单  ▁garbage 考入  听起来 谁的  ▁colored 技工  ▁marry   ▁unanim! urally" 捐献#  很多事情$ olesc%  报考者&  的氛围' Ha(  ▁conclude)  ▁Nicole* phan+  ▁Words, 在即- opa. 别看/ Due0  ▁proprietary1  ▁Pierre2 杂草3 公子4 大富5 驱逐6  channel7  ▁refresh8 stack9  ▁historian: 超多; vance< Price= 号为> 停业? 九龙@ 索取A  职位描述B  riticalC 直面D  ▁failuresE  DiscoverF  ▁CongressionalG 配有H  为广大I  ▁enrollmentJ 真好K  干净的L oticsM  在心里N  专业委员会O  睡不着P MemberQ  ▁CornerR  机关事业单位S  施工单位T dingsU  一般都是V ophobW 我从X  ▁guardsY  ▁metricsZ  ▁forty[  ▁Compar\  ▁Reynolds] ▁Ing^ 高尚_  ▁simulation` 不甘a 昭通b  ▁immensec 逍遥d  报名系统e  不完全f  ▁mindsetg 烘焙h  十几年i  ▁mantj  ▁avoidedk ▁MLBl Rebm 宣言n  ▁outlineo  ▁intentionsp  做出的q  ▁definesr 大盘s  ▁patiot 稳重u wentv  Chapterw  赵丽颖x immingy  闯红灯z 通达{  ▁Ultra|  ▁Americas} 腹胀~ CoV 共克 郭德  王者荣耀  ▁Repl 纯正 圣地  基本信息  ▁collision  可能出现  ▁crude 战队  ▁camping 这话  ▁McCarthy  无私奉献  交通广播 Radio  必需品  ▁colleague 回暖 湖人  ▁consume  世界一流  专业合作社 支柱  ▁satisfying 渣男  请广大 xiang 久久  水平和 渔业 考情 微软  ▁maximize 运会 像素  走进去 大鱼 男篮 牛逼  加快建设  Success 剥夺 omatic omp 小便 就看  ▁Beatles  工作人员的  ▁protections  博览会 抗拒 洗礼 对付  ▁chrom  ▁transgender  ▁counseling 墙壁 湿润  ▁painter 温差 饭碗 ▁Ce 南无  创业板 豪门 hart  ▁Forbes  也只是 在天  比别人 ▁Leo  ▁cleaner  规划设计  着我们  ▁Prec  ▁ambition 丰满  高等院校 Div  每天都在 emi 揭晓  ▁Calgary  工作任务  ▁assembled 疗程  他就是 again Things 想你  分分钟  ▁Journalism ")) olas Imm amel  班子成员 丸子  业内人士  饮食习惯 邀约 走着  总觉得 roots 因病 ▁Bir  ▁Palestine  ▁vulnerability  中国人民银行  ▁transplant  ▁Comic  为主题的 Grid 胃炎 (:  版权和其它  ▁terrain  法律的 茯苓 ▁TN anson 证的  年底前  fighters 而我 ▁kin 人要 参谋 除去  ▁sang 一等 ▁Vir  ▁Collections  ▁tricks DL LOG  ▁Lis   ▁captures  宠爱   王先生   为了避免  ▁vacuum 谋取  ▁wheat  ▁expressions 食谱 应征  申请表  ▁Reid 之类  的品质 dress  ▁kicks 忏悔 潜艇 选出 invest prop  ▁endors  省外!  ▁surviving" 部与#  就够了$  ▁Katie% Queen& ▁xml' 测绘(  中科院)  医学会*  市场价+ Inside,  中国人民解放军- 好礼. 辗转/  质量和0 kle1  internal2  ▁gravity3  ▁shoulders4 inez5  表面的6 怀着7 ENTS8  全流程9 又名: 尽在;  下一代< 毅然=  退休年龄> 制订?  ▁trainer@  ▁RodriguezA  调研员B  ▁yeahC  城市管理D ,「E  的日子里F kswG stepsH 明清I ettesJ  ▁welcomesK 退货L  李先生M 误会N  ▁SteamO 或多P  了一张Q  ▁DiesR  ▁TestingS  ▁defenderT  夏天的U  到朋友圈V  ▁rapperW  ▁femalesX  羽绒服Y 光电Z 滞留[ ▁VI\  有很好的]  纪委书记^ 颠倒_ 安检` 遥远a  个月内b  ▁minimizec  ▁brackd  和一个e  ▁cousinf ▁ecog  officialh 水务i  ▁jokesj 竟是k 干涉l  focusedm 一文n 出土o  ▁privatelyp  消息称q 骑车r  ▁sofas 午后t  五花肉u  ▁considerablyv 多是w rilx  ▁CLASSy  ▁Drewz  ▁summar{ 地价|  ▁nostalg}  最后成了~  ▁salad  都属于  ▁frank  有时间 螃蟹  Russian 键盘 ▁SK 白石  实施办法  ▁socially 快快 IGH  好心情 letion  ▁Param  强对流  ▁Ubuntu  ▁Communities 洛杉 ampire  services 眉毛 岭南  第三届  ▁orchestra 力学  ▁deriv 五六  一种是 性侵 悬赏  老太太  虽然没有  ▁Fleet 税前  ▁Write  电子信息 viol ▁pie fan  我不是  ▁Ther 作废  自己去  ▁Downtown  ▁indicators 定金  现实中 识字 liter  ▁oils  ▁emotionally  在校生 aky iaz  找一个  贡献力量 Active  ▁Arnold 辩论 谁都 始于  方式进行 uren 打动 日被  保健品 政审  ▁neur 循序  ▁behavioral  ▁descriptions 乱象 阿克  ▁luxurious wire  电子产品 WN 的内 沉稳 这本  的国际 umen 窗帘 年被  ▁bubble  ▁debris  这部剧  ▁Crew  ▁inviting  工作上  严肃处理 奠定 过早 aml pine  groupId  年轻时 ▁Lil Player device 修理 rice  ▁Decor 凯旋 易学  ▁quietly  ▁Bomb 苦苦  举办了  ▁consumed  在微信  先锋模范  日夜间  教学视频 提现 不讳 atta  值得我们  岁的时候 France 更显  长春市  ▁civic 孝感  以下有期徒刑  洛杉矶 ardi  ▁mall 量为  认真学习 Diff erton 构图 经商 超员 nis 猴子   ▁pian  上月   ▁Earl  红肿  PDATE  的几率  湿地公园 盖章  ▁Properties  类似于  ▁Discover Mah CPA  地下室  astically  半个小时 楼房 ')) awi  laration  总决赛  我们的生活 filled  uilt!  局副局长" 沙龙#  选调生$ 蚝油% 四海& ▁Va' 伸手(  ▁Chef) payer* 洗脸+  表示感谢, andel- ▁Tow.  但是如果/  ▁dressing0 报刊1  ▁frustrated2 ▁CM3  贵阳市4  刑事案件5 十余6  ▁methodology7  颜色的8  ▁opio9 王牌:  新发展格局; ffield<  otation= FX>  办主任?  的范围@  ▁knowledgeableA  重要意义B  ▁WyomingC  ▁awfulD  programE  ▁demographicF 巢湖G  ▁WinnH 班组I  风格的J ▁spaK  ▁eurosL {%M 化痰N  stanbulO  racticeP 户户Q 蜘蛛R  ▁stapS  邢台市T  长期的U  什么都不V  ▁finalsW  ▁faithfulX 荷叶Y 送货Z  这么做[ ondo\ 吃饱]  交易中心^  团队合作_ 附着`  因为她a  指定的b Modernc  ▁sidewd  小心翼翼e 首富f 灯火g 党性h ▁UTCi  请各位j  ▁protectsk Rangel  素材来源m  ▁medicinesn 拍打o 吉他p  知道的q 费等r ▁Lans  ▁googlet  如涉及作品内容u lossv  制度改革w  及时发现x  ▁Readyy  ▁enhancingz  请及时联系我们{  生命中| 过关} 虫害~  ▁Options  考试中 和县 毁灭  舞台上 差价 雪山 third  ▁rookie 崇尚  早发现 相逢 拉着 收银 们都 缺席 stairs 绝佳 ▁Gem  发热门诊就诊  电子邮件  ▁constraints  党建引领 还算 快点  应注意  深入学习  ▁deceased 停靠 谁是  ▁prescribed 富足 指望  ▁Female 时装 白茶 燕窝 Lic  的妻子  猜你喜欢  ▁Tibet  兰州市 一侧 化身 的张  经济圈  ▁redirect  劳动关系  ▁Eval  总是在  ▁silly WHO  本公告  管理体系  ▁severely 百病  ▁Vehicle  它可以 与日 illes  ▁Conduct fair  给予了  任何时候  ▁Apache  ▁injection  ▁fright  ▁technically EAR 平民  哪还有 ▁kg 玉石 eches 国画  ▁humble  生态保护  ▁liable  ▁sailing  但不能  ▁exceptions Die  可选择 你和  ▁бы 不时  版权属于  ▁rolls  ▁accessing 供述 遗嘱  ▁squee  ▁cens 性地 不理 管线 ▁WWE ombie  版权和其它问题 察觉 tab 退票  重要性  ▁poker  按照国家  报记者 encil 轰轰 照射 热潮  安全和 hole  ▁marking  ▁Legislature  　•　 BIM 认罪  学生返校 几句 ▁sou  年前的  凭什么  ▁lemon 中枢  让别人  ▁altered  共同的 倾诉 联想 Wild Could  ▁Angels éd  我们从  ▁spite  抖音号  新华网 农资  一直到 变换  ▁tops 痛心 或缺  ▁macro  ▁rises  ▁Hispanic  产业的  我就是 kk  ▁dough 种的  ▁Twin  ▁nail 转动  商务部  奢侈品 丽江 加倍 杠杆  ▁accordingly 提速  师范生  警告处分!  征求意见稿" Atl# 亲和$  atching%  国家重点&  工作部署' 学徒( 转至)  不了解*  ▁Behavior+ ▁CE, 大象- 改良. 尿液/  ▁invented0  ▁bloom1 林区2  ▁energ3 冰淇4  打破了5  ▁seized6  ▁Exception7  ▁Sara8 iba9 Mid: 关门; 春雨< 越过= kov> 悬崖? 盘中@  ▁UpdatesA 挺身B 料理C 隔着D energyE  ▁consequenceF  村里的G  ▁suckH 店员I WilJ  也应该K gieneL 加减M 药用N ▁AurO shanP 浪潮Q 衰退R feldS  引导学生T  唐山市U  教师节V  ▁FeatureW  ▁persistentX  ▁legislatureY modelsZ equal[ 伤人\ ▁jaw]  包含了^  ▁cant_ 村村`  ▁Gotha ▁Likb  法律服务c  ▁variantsd 落水e  了一位f  ▁losesg  事半功倍h  ▁sungi 的双j  教学楼k  ▁Solidl  ▁disappointingm  这样一来n  项目管理o  论坛邓州论坛邓州p  ▁стаq  ▁databasesr reacts  ▁Winst  ▁substantiallyu 护栏v 解说w ontinx 锡林y 无反z  战斗机{  进一步提高|  ▁dealers}  ▁hilarious~  ▁Johns 停在 ▁mud 善意  ▁pickup 正直 Effect  ▁kills 大放 顾之  ▁deterior 小板  ▁seas  ▁Enforcement 尽职 Pict  文件精神 安逸  ▁Running 所作  ▁synthetic  ▁Extra  ▁frustrating  不方便  ▁Publications  ▁Almost  ▁residency  ▁exotic  ▁Spotify  每天都会有  的女孩  ▁solidarity  otional  ▁reflecting  添加了  最早的 ▁LLP  interrupt  当然了  体温测量  ▁assumptions  活动轨迹  来得及 Cache 问问 inned  分享给身边的朋友  ▁prototype Sym  中心主任  ▁poses  ▁shutdown  个城市 Give 本题 泥土 稿费  不明白 病痛 auto dong  ▁Angela yx ▁DB  转移到  ▁transmit  以后的  有以下 博弈 aways 逝世  ▁relies  ▁practically  担当作为 用过  猕猴桃  ▁intric 阵阵 ocide  乘人员  武汉大学 供货  同时在 ▁я 杂交  小红书  一起来看 纯洁  第一期  是他们 erseys  ▁architects  激烈的 Living 个村 片的 aja  环境和  研究的  ▁temp 撰稿  ▁Gregory 着实 初二  可以使用 密密 喧嚣  只剩下 肝肾  也表示 三维 感触 娄底  连日来  ▁impose 三名  交警部门 口鼻 的要  ▁Hebrew ▁Saw levard  ▁responsive ixels 祖宗  ▁opposing 后会  ▁empire phot  ▁invites 支教  千万不能  扫描件  最后成了对不起 cache Jason  otechnology 赢家 Bal atz  惊人的 涨停  ▁Tagged Brien  ▁finances 后将  ▁dangers 八一 cology grand  的因素 赶往  十八大  说着就 irie  ▁arguing ém  划重点  明星在农村  有多少句我爱你  ▁Yahoo 请加  含有的!  古老的"  ▁Rece#  自由贸易$  ▁rushed% 碳水&  新发地' ENS(  ▁tongue) jah*  ▁Morrison+ : 3新型冠状病毒感染的肺炎疫情防控工作, 面团- 联名. hb/  工程施工0 odium1  营养成分2  根据自己的3  ▁boundary4  ▁Bark5  ▁pist6 wright7 拜年8  ▁forb9  综合治理:  专门的;  ▁slave<  ▁Introduction=  最后一次> 上行? 轻度@  普通型A  如有侵权请联系B 面目C  招聘求职D  被发现E  政府部门F  事业部G  星期一H  ▁MysteryI 无误J  ▁disruptionK  ▁capacL 冶金M 大三N  ▁intervenO 情境P  ▁comebQ 刊登R 尘埃S  ▁transcriptT  ▁metaU  ▁orbitV 盛大W  志愿者们X  ▁SiliconY  党风廉政Z  ▁Assistance[ '"\ 地为] 后又^  ▁trades_  ▁conform`  ▁Regulationsa  ▁repositoryb  日本人c  问题上d mande  ▁redesf 尽心g 燕子h Peti  郭德纲j 电瓶k  看电影l 夜市m  投递简历n  ▁Fundingo 升华p 将要q  Integerr 不乱s  ▁eligibilityt ▁Migu Dimv  ▁characteristicw  想象力x okiay estyz 切入{  形式主义|  ▁unfortunate} 安娜~  今日头条  ▁Advance 无名 气球  ▁pound  督导检查  专业技能 ▁Zoo  ▁fertil  ▁eliminated 憧憬  ▁Cord  全国文明城市 以北  要积极  由中国  之规定 要闻 hum  的情形 Ash  迈进了  ▁Andrea  脚踏实地  具有较强的 好心  ▁surprises  ▁directing ные  从业者  都市网 Mot rights  广西壮  进行消毒  脸上的 射击  ▁Princeton 正经  变压器  菜单栏  的时间内  ▁Heath  随身携带  ▁transported  生长发育  ▁necessity 总编 房车 牛排 ▁lod  区委书记  医疗费用 背影  ▁infant  的信心 观测 任免  而言之  ▁indicator done emp Route Ron  热水器  高跟鞋 南至 旧址 ▁Hi  ▁Bengal 无害  考试科目  ▁thriving mn 光学  username 斩获 sin kw 请教  ▁Reference zl  ▁thumb 占了 alla 真空  的电话 回想 性与 击败 艺考  ▁wand  国家和地区  依法对 患儿  保持在 侦破 拉动  市纪委监委 zerbai 售票 勾勒 召集  应急救援  识别下方二维码  让孩子们  ▁explicitly 仙游 关停  进行时  求职者 Prim  什么的  ▁merchandise 俗语  mentioned 只在 遇事 退回 ▁pap AMP comb  的那一刻 velt  能有效  新闻热线  也是在  leyball 磁场  的区域 代办  公共资源  ▁limiting  ▁uncle  ▁retained  你必须  也算是  改革发展 Angel  ▁retention  市疾控中心  ▁shelves 悄然 (* Techn  ▁Threat 我看 ifacts  农村地区  ▁Teams  ▁Emmy NU Nic  宣传资料 tu  ▁slower 后就  ▁debates  站起来  ▁Often це 字节  ▁fifteen! 宜春"  教体局#  ▁incons$ 不小% 可想& 胜过'  ▁breeding(  ▁comprises)  去年的* 河水+ ynote,  ▁plaintiff-  ographs.  unicipality/  ▁mush0 寺庙1 面色2 Tax3  ▁tumor4  该男子5 喊话6 ▁Ow7 宣称8 变相9 无不:  教职员工; etc< 水区=  Connell>  进行现场? eking@  攻坚克难A  市场监管部门B ℃;C  诺贝尔D 葱花E  写字楼F  我们不能G  最后都是没关系H 审稿I  都是为了J 打个K 能手L  的英雄M  ▁HairN  都不要O  每日一练P integQ  ▁depositsR  不断完善S  体制机制T abloU  ▁basicsV 粘膜W 清算X 杂质Y uintZ Along[  ufficient\ 欺诈]  自然资源局^ 心头_ 减负`  冰淇淋a  ▁violinb  有条件的c  ▁Bertd 的这e 教版f  的概率g 瓷砖h  growingi ihadj  ▁doubk  是人体l fengm atemn  Resultso  ▁professionallyp 西宁q mutr  生活会s rart  因此在u 进修v  ▁admitsw  网络平台x ▁Ludy guardz Greg{  三门峡| 动静}  ▁auth~  ▁Remove  你应该 清气 肆意 千亿 Way  ▁dire 设法  外科口罩  有多少句对不起 zip  ▁opener  有多少句没关系  ▁detained 也太  ▁Rica  ▁caut  ▁aired  ▁traits  ▁waited  临床医学  作用的  制造商 " 社会主义核心价值观 点位 七大 闪闪 ervoir 实业 aney sb  ▁Character  ▁academics 离校  ▁Shield 罗汉 叛逆 UTH 僵硬 压制  根本上 Impl  ▁Shadow 河中  我都会在这里 内含 铺设 限价  消防车  ▁Concept icky . '以习近平同志为核心的党中央 芳香 文员 Les  月以来 无知 讲好  ▁Butter  尊敬的  提交的  ▁posed  新浪微博 Place  ousands 光彩 学佛 院的  ▁Bund 照样 enges Empty 肿胀 重返  会选择 在海  ▁Raid  ▁Evolution  统一部署 诸如  ▁carp  ▁bure  ▁logging iej 顶部 Paper 行事  对这个  ▁Featured  规章制度  检察长  其他国家 揭开 isible 造福 foo  查看答案  ▁Barnes  广西壮族自治区 Hash  循序渐进  ▁Pine  太重感情 临夏 袜子  controlled habi  ▁slic Clear 一反  开发的 hao >(); uras  ▁Heavy  ▁switching  睡眠质量 一言 桃源 ogenic 沉着  对我们的  都有着  快来看看  李克强  ▁Yankees  工作开展 大唐 火山 性情  ▁drill ན་ 云浮  含有丰富的 拜访  本公司 xian  区教育局 刘晓  ▁updating  ▁enthusiasts  ▁Opening  优秀教师 有位  ▁countryside  中国共产党的  状态下 坐落 领队 (&  ▁Gabriel 折不  ▁Crist  教育基地 % 我都会在这里陪伴着你 东山 序幕  ▁gard  ▁admire 眼下 升值 木材  办公楼!  坚强的" 传闻# ▁Sah$  也十分% blems&  聚集性活动' 法门(  ▁Armstrong) ▁Won*  ▁affirm+ 一新,  不承担-  ▁membr.  市场价格/ 敷衍0 镇的1  ▁Renaissance2 oya3 价位4  连云港5 面相6  ▁overtime7 泄漏8 ▁bis9 浴室:  为客户; 说完<  ▁foam=  ▁Names>  有的时候?  Continue@  生长的A 表决B  ▁StevensC  ▁sincereD 声乐E 身子F  ▁RhodeG 平阴H bolicI  ▁clauseJ 海带K unnL  配备了M  ▁HoodN 尔夫O  也许是P  工作情况Q 诸佛R 真菌S )[T  ▁oliveU 刹那V  那个人W  ▁ConditionX  ▁LucyY  ▁DanishZ 摔倒[  ▁React\ 生根]  ▁diving^ 高雅_  可以先` umpera icatesb  ▁assumingc  ▁assistingd 流产e 自如f  的准备g 快看h 铅笔i 特训j ▁Pamk 欣慰l vectorm Suchn  ▁examso  ▁budgetsp  ▁nutsq  体育锻炼r  癌细胞s 指点t 喜报u  全体党员v  吃什么w  ▁Resolutionx  大概是y  ▁dioxidez elaide{  ▁fictional|  年下半年} 埔寨~ 会用 尼斯  哪怕是 rys  但现在  信息和 ▁Por  ▁Belt  ▁sich  ▁Anyone  thritis 吴亦  ▁horizon  未按规定 Cat Below 汇款  作品的  ▁disagree 职权 ruce  ▁realised  就会接 引流  综合行政执法 东城 ixian  很大程度上 北极 烈烈 ▁dot 海棠 锦旗 Font agar  部分内容  ▁Porter  ▁Aber  command  ▁Palmer 车管 ▁IoT 兼具 整形  落实情况  arettes 广元  ▁pants  ▁intends  ▁fisher  ▁resur  ▁patrol  ▁browsing unks 留住 晋城  ▁Groups  ▁wrestling ▁� 攻克 Genre  ▁Trends  ▁Description 大军  istries 盐水  万达广场  ▁pockets 之本  财政厅  ▁minerals 小鱼 迭代 Energy imated 抽出  任务的  到一个  ▁pier  replace  仪式感 王晓  第一百  ▁proximity 为国  ▁Gibson ISE Bio arta  开着玩笑  ▁centered  ▁Sustainability 机和 gc 上汽  笔试时间 止血  案件正在进一步  ▁relie quin  ▁negotiate  的伟大 robe ▁((  ▁candy  多少人 器具 已达 商用 运维 楼上  dimensional  一定不要  总的来说  ▁utilities  回到家  一起关注吧  ▁proxy  共克时艰 油的  ▁underneath 动动 ▁Pf aying ▁luc  ▁pressed  especially 群的  ▁posters 行测  ▁linking  ▁Thrones 要不  ▁lounge 信箱 hengh  ▁horrible 男方  ▁Posts  被感染 ayette  ▁Wake 洗涤  something  ▁arom 甜品 rators  ▁ви 佩服 Major  强基计划 时隔 ▁Zen 香料  灵活就业 名声  ▁Peninsula 日渐 之源  不管你  才能进去  ▁Jesse  ▁Makes!  ▁commence"  Original# 风华$  自己也% cross&  ▁pools' 南风(  低风险)  但是对于*  ▁probability+  ▁icons, 火力- 摸索.  的商业/ WW0  ▁Reality1 荧光2  ▁Pand3  ▁molecules4  ▁Cort5  这些东西6  ▁async7 战时8  我们所9 Import: 电解;  ▁warnings< 漫天=  ▁expans>  的女人吗? orig@  ▁NatalA ▁CFB  停课不停学C  ▁bannerD inkleE  固定资产F TryG wouldH 火烧I  答疑解惑J  常见问题K  为什么不L 磨损M 宁可N 缺血O  ▁liftingP  ▁instituteQ 兴隆R  DisplayS  ▁rushingT  ributesU ▁înV  ▁MariW 家国X  ▁hardestY 西至Z  小微企业[  一天的\ hind]  ▁smallest^ 逃逸_  厉害了` 车票a  ▁Gangb 又将c 开新d  三分之一e  ▁Lawsf 1 *中华人民共和国道路交通安全法g 平整h  娱乐场所i  ensationj . '分享给身边的朋友一起关注吧k doctorl 地把m  忽略了n  还不是o 辨识p  ▁clayq  Networkr  其次是s  星期五t  消化道u  ▁shitv 红了w  ▁patronx  ▁willingnessy  ▁gospelz  报名条件{  ▁Alumni|  上厕所}  城镇化~ 无故  因为疫情 零星 青椒  ▁Shell  工作经验者优先  上下班 大王 tax 酸性  Republic 安然 adequ eh 人手 赶上  有良好的  uckland  ▁amateur ▁cum 点对 果子 imi  转运至  ▁reper  entieth 想在 拆解 atra  会成为  ▁Posted  ▁Amazing 省直 翻了  我真的  黄帝内经 凉拌  一点的  企业发展  社区居民  ▁cryst  日报道  提前做好 iens DU 参军 一角 复检  饮用水 ▁Rut  你是一个太重感情 iotics 还很  ▁ugly  率达到 快要 regn clock  柬埔寨  ▁cease  ▁restart  ▁remotely 年到  最后说了  等专业 一堆 ▁Pub  ▁Leaf  此文出于  长的太高了 韧性 补习 Ell ▁ub 顾虑 上阵  留言区  自我介绍 加了  ▁twitter  Personal 登山 ▁(*  服务和 的四 ▁pdf  ▁Evil  ▁Patients  reference 侦察 的国 + $你是一个太重感情的女人吗  ▁salmon 腊月 光大  ▁drilling  优美的  长的高 开采  ▁Letters  ▁Singer 女装  后来的  ▁Hass 急于  也有很多  骨质疏松  国民经济 多处 ologic  无反顾 腊八  健康安全  的游戏 似锦  ▁chol 振动 疏忽  ▁Romania  你的心  ▁easiest 杜甫  ▁likelihood task  ▁crowded  ▁lamp Regist  ▁trump 文的 较长 Policy  动物园  ▁enthusiastic 贪污  深入贯彻 凉爽  ▁Bryant 基调 避孕  NASA  aku  三八   市政协   东京奥运会  理工学院  ▁tastes 请看  的皮肤 DN 冬至 已婚  ▁sometime 教书  面上的 ▁Hus  ▁accredited 的非 石材  ▁voltage 胶州 ▁NOW  ▁receipt  市内! нов"  病患者# 跟上$  毫无疑问% 献给& 兽医' 常有( 派驻)  ▁credited*  ▁injust+ Nature,  ▁stranger- 繁花. 沉香/ arse0  说着就低着头1  ▁Camera2  地质灾害3 ät4  ▁де5 ovan6 交织7  命名为8  ▁Specific9 会将:  ▁starred;  ▁guarantees<  ▁commerce= 壮观>  ▁aftermath? 低迷@ 深思A  ▁provingB 三条C  我不知道D 同道E BeyondF 出新G  ▁tortureH 堪比I  ▁weakenJ 助攻K 光影L  点击进入M 超车N  忘记了O  朝阳区P 吃点Q  ▁proudlyR 碳酸S  ▁incentiveT endorU  ▁ChristineV  拎包入住W  ▁sponsorshipX  重要通知Y 水印Z  ▁junk[  医保局\ 昂扬]  External^  ▁Counter_  的全部`  ▁Directorya Doubleb  ▁Reliefc 半价d ogglee  小白理财训练营f 行列g ▁Kunh oleri  ▁Sugarj  吴亦凡k 周密l ▁octm  扫码回复n  责任制o  ▁Terrp  ▁bindq  ▁Sandyr  ▁scrutinys  等方面的t ecraftu  ▁stiffv  ▁Startingw 推向x aaSy 曾是z  ▁tutorial{  过来的|  坚守岗位} 南门~ 重塑  的心灵 isle  尊重原创 iman 归根 法宝  ▁Dent  ▁Sector  ▁namespace  ▁Swan  串门的时候  总会开着玩笑  ▁accompanying 沿江 物件 pu 品位  ▁allocated  ▁craw  新东方 这支 agle  ructive 并用  意见建议 时分  可以去  喜欢夜叔  ▁cycles 水和  你没有  的电影  Question Notes 凉山 长白 鱼类  ▁tiles 变质  网友的  纪律审查和  ▁Simpson  ▁Marriage  第一医院 坏事  ▁Berg  ▁genuinely ▁DA  有兴趣 ▁Ign  自我防护 勤俭  独有的  二氧化碳  办案民警 捍卫  有较强的 ▁Rum  到好处 磨难  第一种 tre  ▁attain 加固 猝死  是由于 硫酸  的两个  面包车  本身就 甲醛  ▁ignor 丰厚  ▁disasters ▁Spa 本轮 不发  ▁respiratory  它就是 核准  课堂上 " 说着就低着头迈进了  ▁coloring 河边  ▁dont 古老  ▁informative 道上  息息相关  ▁Astron 勒索  葡萄糖 在逃 agment 请于  优先考虑  生活费  ▁chase 六合 远景  ▁Delivery 克尔 isbane 名胜 一念 oxy 个好  九年级  ▁paired  等其他  ▁hypothes 的就 ▁Cav 北至 LT ▁OUT  ▁Nutrition  背景图  中国移动  敬请谅解 宏伟 Tony 钉钉 Joseph 承租 之用 ۛ۟ rand Brown  反映了  ▁convinc 填补  分析和  ▁Grande  张先生  退伍军人 海绵  不间断   文职人员   几乎是  作曲  私下   组织和 作答  可能在  ▁Lights ( !说着就低着头迈进了门槛 安家 美金  小编微信  ▁sized 味蕾 itzer  师生员工  Originally  工作站  ▁schedules  ▁thoughtful  打招呼 ishops  董事会   会议要求!  谁家里" 抱团# 租售$  ▁prejud%  ▁regener& 萍乡'  ▁pressures(  他们也)  的音乐*  银行账户+ pict,  ▁Cere- 瑞丽.  成交量/ igator0  违规行为1  往往我母亲2  ▁Nine3  ▁sealed4 阿斯5  一共有6 红酒7  ▁statewide8 的很9 重回: bay; ▁SN< glass=  ▁Loss>  ▁beside? 煎熬@ heticsA BenzB 草坪C  一个问题D  ▁recycledE  我们来F  圆满完成G 德镇H ingeI  规模化J  必然就会K  往往我母亲就会接L 巡回M stedN  ▁gratitudeO  ▁discoveringP  和他的Q нийR  总会开着玩笑说S  基本条件T  抓紧时间U 有道V  的一家W  人民群众的X  高铁站Y  工信部Z  违反了[  Medical\ ▁Typ] 万例^ 弱势_ False`  ▁flowinga  ▁barnb  Channelc 东湖d 洒脱e 必读f  ▁bizarreg 画出h Anthi  ▁Cantj  主题党k geticl  ▁Chartm  ▁bracen  大专及以上学历o 弱的p 豪车q 石墨r 老伴s 落花t 拉黑u 龙岗v  副主任医师w  ▁Vaticanx  强降水y  一时间z  正所谓{ 减龄|  平凡的} throw~ Fast  Additional  在朋友圈  对其真实性负责  ▁Henderson  泉州市 宅家 开胃 水量  ▁Dayton  ▁speculation 前程  事业编制 马甲  中国人民大学  ▁Topics 溢价 降幅  ▁erect ▁*) 名牌  ▁Critical funded CCTV  自然而  ▁Specialist igger 超速 力行 ▁ash  应急管理局 Mex  从一个 素有  rolling 预缴  ▁Heroes  也别扭  ▁switched 炸弹 戒毒  应立即  ▁trademarks  日发布  ▁Display  人员流动 暖暖 触发  品行端正  病毒感染  因疫情 讲授  同学们的  程序员  ▁interference ogical  ▁classics  Storage  困难和  ▁technicians 月子  城市网邓州 百亿 dad 推测  一线城市 班次  表面上  长期以来 缝隙 招人 ▁umb  ▁therapies Cour  ▁Buch  ▁Duty  的脚步  艺术学院  宁波市  ▁Skills  ▁Bluetooth  ▁је Sir  ▁lively 快报  ▁Instit ( !往往我母亲就会接上一句 一曲  党的十九大  糖尿病患者 向后 牙刷 uffy  放假时间 缺氧  分享了 yson ▁Liz  太大的  职业生涯  ▁roller 冬奥  有两种 如来  收获了  再也不用  多样化  这次疫情 偏财 追责  ▁correspondence 领土 AVE 民心  ▁Protect  的各项 ▁Sci  ▁Jets  的指导下 oland  quartered ła 推特 ▁FE copy SY  的祝福  第二种  纪检监察机关 商超 麻麻 1 *这样您就可以继续免费收到最新  有相关 任教  ▁Related 不改  ▁downt  更不要  为避免  湿热  毕节  迷失   ▁prevents   ▁Guinea  人口的  件事情 隐蔽 斑点  ▁withd  一个好 块的 王一 Api  不会再 bassy  的一幕  ▁purely  等单位  学习能力 橙色 袋子 督办   ▁Travis!  ▁supplements"  ________________#  还是批评$  ▁couch%  可适当&  宫颈癌'  的趋势(  轰轰烈烈)  ▁Rural* urbs+  领导职务, ullah- 暴跌.  造成了小/ arcer0 议员1  幽门螺杆菌2 学段3  ▁Cyprus4 背着5 青铜6 瘦肉7  是否存在8 这台9  在一些: LOW; 大臣< hp=  ▁autism>  容不下你? 干劲@ complA  一切苦B  晒太阳C  保持安全D 虚构E ▁AREF  PublisherG 其一H  都觉得I  ▁timberJ  ▁magnitudeK 的来L 升起M ▁AjN 追梦O  妻子的P ializeQ 岳阳R  的吸收S  总会引来T  聘用资格U ImpV  ▁mediW  团队的X  视频会议Y 青云Z  ▁Humanities[ 严控\  ▁ме]  未经授权^ 有志_  恰到好处`  政策和a splitb  长的高了c 山人d 绑架e  ▁clashf 萨克g  ▁glamh  ▁salariesi 泌尿j 刮痧k  ▁comebackl 万方m ▁ramn 他把o 银杏p  toStringq 气色r Tours  ▁quarterlyt  ▁Buzzu 全能v  ificantw 夺冠x 问道y  ▁incomingz  ▁undertake{  率高达|  例确诊病例}  低着头走进去~  ▁Bullet  文明的 得名 " 哪还有低着头走进去  一阵阵笑声 五种 视界  企业名称  别人也跟着  换来的那种  ▁backyard  也不敢  进门时不低头  ▁Apply 1 *哪还有低着头走进去换来的那种  ▁Chat ▁Cer 拖鞋  我们知道 Events  假如堂叔到我家  应对疫情  ▁rooted  avilion  义无反顾 riages . '假如堂叔到我家进门时不低头  ▁coordinate 走得 ▁VP  有责任心  给出的  随着年龄  ▁playground  ▁thankful  小规模  ▁tunes 请以  别人也跟着不自在  与家人  地进行 ▁Cha 愈合  出台了  实施的 表皮 适中 冲泡  ▁prolong  ▁palm odo  文物保护  长的太高了也别扭 会见  应急指挥部  ▁rugby  ▁mentally  碰着头  为了保证  资料显示  ▁counted ▁RC  ▁amendments 演习  美国政府 7 0哪还有低着头走进去换来的那种欢笑 幕后  running 向着 处境 mac 胜任 VEL  不知是表扬 迪士  ▁phases " 总会引来一阵阵笑声  自己造成了小 扛起  ▁Folk  自己造成了小痛苦  icultural amy  ▁Rate  ▁submitting 惠及 一试 上山  不可或缺 理所  ▁Hunting  基本情况  ▁Advoc  必然就会碰着头 ▁uk  俺这门 hran 福特  ▁conclusions 辱骂  ▁Apps  发展新 变态 % 城市网邓州城市网邓州  养成良好的 工龄 肝胆  自然是  ▁defendants " 纪律审查和监察调查  ▁counterparts  是因为你长的高了 七星 虫草  isheries lynn ▁CB  做自己 病害  打开了  ▁Ferguson  ▁Fitness  国庆节  special  ▁verified 重任  因人而 王王 ▁LI 滨河  月上旬  ▁Monica 受访  积极推进  sequently 大到 ▁Bog 讽刺 xls   低着头才能进去  zhi  none  有空  根部  中国梦  ▁Expo  见证了  ▁clips  ▁relates  ▁Might 地市  交汇处  ▁Kerry  给大家推荐  俺这门容不下你 ▁Cob  ▁Religious Sport fac 军民 ▁lag pherd  JSON!  ▁offline"  springframework#  ▁adjustments$  市场营销%  ▁aligned&  主要用于' micron( 会所)  多岁的*  ▁masses+ avi, " 都得低着头才能进去- 名医.  安置房/ Figure0  在整个1  电话联系2  ▁shaping3 走近4 ▁Vic5  到了谁家里6  ▁Tennis7 lings8 风气9 mom:  ▁Jamaica; 亦是<  伊拉克=  元宵节>  ▁exhibited? 百岁@ ▁CVA  希望通过B  ▁mastersC ottestD щеE 破了F  ▁driedG 透了H 其所I cestJ ynastyK  虽然说L  ▁swordM  ▁relyingN  决定的O  ▁prosperityP  仍然是Q 湖州R  ▁hashS  有一句T 定额U 定为V 床头W  FederalX  专业人士Y 木兰Z did[  ▁researching\ 抽调]  ▁brass^ 明目_  张女士`  Matthewa  迪士尼b  ▁Bossc imeterd  新闻发布会上e Songf 典雅g 动着h  可以是i  下面就j  ▁Congok  李女士l 营利m  人来说n  ▁whito  ▁millennp  每天早上q  ▁Yeahr unions " 国务院联防联控机制t 方正u 考量v ysicsw 懒惰x  黑科技y 双节z 提振{  成员单位| 月球} 殿堂~  中老年人 ripp  ▁Mathematics  ▁qualifications (@ ér Georg 通办  标准为  工作时  甚至会 cock 病死  ▁hesitate  自己的人生  ▁neighbourhood  最初的 争先  和他们 esch ▁oak 卡斯 weets 1 *到了谁家里都得低着头才能进去  ▁leap 警车  ▁employs  寒暑假 短发  基本养老金 分段 名副 议论 hma  ▁trap 防腐  的成本 摊位 考务  ▁laundry 测定 plugin  ▁lawsuits  ▁ingredient 者在  ▁harmony  行政村  ▁searched 流淌  ▁gently  ▁Essay ▁VPN  越来越少  ▁branding % 堂叔到我家串门的时候  节日福利 云上 不强 不缺  向我们  ▁Guitar  成品油  ▁Adobe ▁Egg  了一句  inspired 散热  恶性肿瘤 光的  ▁coding  便利店 ▁Gru VAL  ▁maritime  ▁Melissa 紧跟  ▁sparked  ▁Brisbane 是天 招财  建筑设计 Panel  ▁Rolling 晓明  最后说了谢谢你  ▁packs ▁Kur 大开  ▁pile ۙۙ 塞尔 随访  危重症 启航 小李  ▁storms  ▁lingu  不确定性 可使 大跌 遍地  和完善  ▁Boot  从高分到低分 专享 冬日 滋事 威尔  居住在  意义上  ▁tournaments  便民服务 ▁Lac 带的  非常多  方方面  传播更多资讯  attering  时间里 urate  解决问题的 变量  我们先 ▁Mik  粤港澳大湾区 : 3这样您就可以继续免费收到最新文章了  飞行员  改造项目 批判 皮质  ▁personalities ▁Dry  所在社区  ▁Fortune nsic 滋阴  户口簿  对其进行  累计确诊病例   等相关专业  ogo   ▁Haiti   ▁accomplishments   又是一 ovich 狠狠  户口本 也叫  死工资 1 *最后说了谢谢你有多少句我爱你 onal reason  ▁rendered  的还是 yuan icals  ▁Rising  的距离  在做好  银保监会 部级 风机   ▁anonym! 能使"  耶和华# 花花$  ▁tribal% 心经& Ess' Recent(  charged) [:* estamp+  打造成, 愈发-  ▁sunny.  ▁alerts/  ▁violated0 Self1 ▁Sue2  各行各业3 农林4  ▁Advertising5  的粉丝6  绿水青山7 肥皂8  参与了9 字数: 围棋;  追踪到密切接触者< Jess=  不可避免> 早年? osl@  团支部A  新华社记者B  ▁knockedC 小三D  没那么E ISAF  下来了G  要求和H asionI  ▁SitesJ bergerK colmL OFM  即可获得N  这样的话O domainP  ▁ChangQ  ▁ChesterR  ▁VladimirS 肌瘤T  ▁LoganU  上一个V  和武汉W  之间的距离X 足了Y ▁MANZ Wi[ 你都\  activity]  责任的^  ▁Stephanie_  经营的`  ▁deera  ▁costumesb 梦里c 生津d 稍有e 绝缘f  满足了g 我才h 日从i 未尽j  扎实开展k 风寒l hdm  雄安新区n 药剂o  的事儿p ETFq  ▁Fearr 剪刀s 喜讯t  能让人u Broadv 专为w  ▁propagandax  生活网y  ▁Larsz fest{  中小学教师|  ▁Explorer}  措并举~  面膜膏  ▁elaborate 在被 所用 Dear  ▁coordinated  ▁bucket 大雪 HB  ▁singers  tingham  ▁generator  ▁Nevertheless 仅作 ▁cad  ▁Nice doing  是否会  ▁calories XXX  ▁aston 这群  ▁metabol  空气中  预防接种  ▁blessing  副区长  新冠病毒核酸检测 Dream 层级 奋发 音响  ▁Evan 地将 ▁ng  ▁Berry  ▁anthrop bishop (_  取消其 柏芝 投降  ▁Abbey  ▁Hence 抢抓  图片来自网络 和解 ijn  ▁Sexual  ▁transforming 渤海  ▁freight bps 身价  ▁Contest impl g `反反复复反反复复反反复复反反复复反反复复反反复复反反复复反反复复  ▁travelled  osevelt bial  顺利完成 ▁nel  请考生 孔雀  ▁Marathon  用手机  ▁remainder  基础教育 lickr 印记 考的 enberg 收缴  ▁certificates  累计报告确诊病例  ▁dementia  ▁rigorous  ▁systematic 理学 ambia  Settings Future  omedical  并根据  ▁trophy aru ▁sci ▁Fut  再也没有 尽头 消肿 频次 " 传播更多资讯之目的 为先 带入 年味 但其 首相 gang 产出  ▁Bend  ▁verb  ▁Drake ▁Sie  ▁Gear 运费  ▁dancers 组装  科科长  ▁Hoff  ▁Tanz  爱上了 拼团  ASHINGTON 原子 微波 警钟  落实到 ▁MPs  ▁Antar 按下  旅游业 Obs  因为有 不惧 世俗 Pa 晶莹 达州  ▁supporter  ▁ма  ▁rehears  带大家 Employ  ▁Rocky 兴业 华诞 course 受人   理财的   download   ▁cocktail   ▁Andrews   形势下 玩转  ▁Activity 路灯  ▁vegan req  ▁tutor  ▁sequences  ▁Officers  ▁Speech 晚了  第二名 atomy 怀抱  articles 回眸 好后 想行 这下   管理学! jn" 共振# 枝花$  ▁expose%  ▁Whatever& LG' 分院( 单单) 莲子* amente+  ▁Dhabi, Pres-  ▁wonders.  ▁bicycle/ 宛如0  ▁validation1  idering2  ▁Recomm3  左下角4  ▁submissions5 uxe6 Carl7  letters8 EEE9  含金量:  ▁damaging; 鸡腿<  ▁tolerance= 储能> 车载? 在不@  的一系列A 偏向B 分歧C  ▁TalentD 实则E ныхF  ▁MoroccoG PosH 外籍I 敲诈J  ▁ChangesK 担负L  ▁BankingM  ▁whistleN  的销售O 之物P  本文来源Q  看到这里R  ▁curiosityS DIYT ScreenU  ▁plantingV  ▁ThemeW heriffX  ▁CarrollY  asteredZ "));[ cience\ 元宝] 纹理^  越来越高_  受影响`  夫妻俩a 情话b 称作c 绵阳d  ▁Owene  ▁Derekf  交通警察g 初恋h  为群众i 崇高j  ▁nodesk  ▁Gilbertl  Canadianm 视镜n lvo  培养孩子p  接种新冠病毒疫苗q 脚趾r anyas  ▁resembt 黄昏u  为贯彻落实v  ▁uploadedw 百色x  开心的y 灵动z  生产基地{ 为一| 遗体}  这种方式~  社会工作  ▁prosecutor  ♩~　 的下 八路 鹤壁 草本 供销 腾飞 风顺 美股 新化  的时候就  的小朋友 艰巨  点点滴滴  ▁evangel  医疗救治 江阴  技术人员  ▁dialog  ▁phosph zc 中去  不正当  ▁discour ▁​  ▁Canon  icester mente  ♩~　♫  ▁recount  ▁Hilton 做梦 ▁cel 杯子  并做好 就上  清热解毒  进一步完善  ▁Engagement  ▁Azure  ▁pathway  ▁positively 康码 随缘  ▁biodiversity ogram  不开心 民意  ▁Duncan );( 瘫痪  医生的  不规范  意识形态 审视 押金 首创  生命财产安全  祖国的 无助 ZZ ▁LG  的口感 从我  一分钱 洁白  认真负责 ( !全面建设社会主义现代化 手感  的特殊  多一些  的居民 % 我们将在第一时间删除 萨斯  打开手机  ▁Quinn  而我们 用料  坚定信心 打折 dit  　♪　~　  乡村振兴战略 轴距  ▁Walmart debug  质量问题 京剧 探险  ▁Mist 平凉  组组长  ▁Mueller  开除党籍 Items  ▁confirms  却发现  因人而异  ▁Certain  　♪　~　♬ 后背  工作效率  ▁Kurt  ▁Latino 终将  杀菌和 wrap 历练 坚固  保证了  ▁carb 通货 illon  ▁sweeping  ▁rall monton 慷慨 ovies  ▁Bened  ▁telev 易于 毫克  受不了  想行识 ▁coh 军校  劳务派遣 }" )*  ▁disturbing 底盘  事业的  受想行识 数名 ▁Ful 异物 . '高高的堂叔到我家串门的时候   ▁explosive  致死  Var   严重后果  hit  这方面 全州  硝烟的  党和人民  ▁pale  ▁meanwhile 还给  当孩子  ▁Abuse  acterial 财报  在日常 Cond 天门  运输车辆  同龄人  奋斗的  profits   设计和!  最低工资" escope#  好吃的$ ulsion%  ▁Subject& 反转'  赶到现场(  ▁questioning) 首府* ▁upl+  ▁crews,  不及待-  ▁discharge.  他已经/  ▁indul0 乾坤1  ▁Trading2  失信被执行人3 关税4  ▁Lakers5 制片6 供认7 官场8 RP9 找人:  商业用途; ▁Og<  ▁bree=  紧急提醒>  ▁Dale?  ▁forums@ 武昌A 大白B 对症C 暴富D  ▁touchdownsE 河池F 正品G  截止日期H  ▁ComposI  ▁UnlessJ 蝙蝠K ooterL 出境M  召开的N  尚在医学观察O  ▁captivP  ▁circulationQ 牌照R  他没有S  时间是T forestU  ▁delegationV 单调W  两不误X insertY TFZ 园长[ 热销\ 飘扬]  可用于^  第四届_  难忘的` 沈丘a  ▁inflammationb  建立起c  ▁tradersd  ▁cellulare  ▁passportf 亿万g  ▁Windowh  ▁eliminatingi 竞买j  的市民k 湖泊l  ▁unstm  ▁Waldn  ▁housedo ▁ESp isersq 公费r  特岗教师s 雪球t  价格上涨u 有偿v 老少w 名城x  ourcingy 一包z ▁Bin{  ▁capturing| grav} ▁WI~  和经济  投资的 ▁Jin cu  能做到  无论你  要通过  ▁Load  迫不及待 esar 酿酒 堆放 酵母  十二生肖  ▁aggregate 一再  (　)　  Services OWN  最大化  的大门 筹划  管理与 选派 洗头  ▁Brent  ▁Multiple  ▁expectation ▁Ny 进军 kok  的最新 基站 毅力 请将 itably  持续推进  ▁Entry  有这样的  ▁Same  ▁angel Ali  ▁Hitler  ▁Recreation  ▁classrooms 八个 实例  ▁environmentally  适龄儿童  Similar 多做  ▁Traffic 原先 rove 胰腺  ▁Carr  综合执法  一大批  和法律 像个  的程度  ▁Files 短片 Again rians antha  就好像  ▁renewal  ▁instinct 充当  说明了 mode  ▁Instr Jes  等疾病  ▁Paint 出错  教师资格考试  实施细则 胃部  ▁Brett  学生家长 Michel 晒出 atable  ▁Gross  ▁Frederick apest  ▁Greens  第一人民医院  ▁shooter 名列 输液 ocy 榴莲  ▁Thing 港湾  所在单位 咀嚼 热衷  ▁python  ▁cares 解封 ��  ▁tonnes 体态 或少 分付  的形成  法定代表人  ▁Cards 仅剩 Photos  ▁foremost  ▁authentication 美美 ifa 飞沫 站着 rops  ▁abol  ▁bytes  ▁suspicious 朝夕 凹陷  未取得  前面的  有保障 提议  ▁metro  的场所  在世界 要先  ▁successes  最主要  毫不犹豫 万只 GMT ▁NZ   范围的  ANCE  联席  变色   也同样  ▁creature  劳动节  召开了  Reuters  房产证 心梗 按揭 他不 胞胎  这么大  ▁Piano  ▁Novel 山地  ▁nutrients  Recently 是她  ▁Muhammad LPR  三江!  ▁arising" 张大# 食指$ ▁gad%  required& 粉碎'  明确要求(  ▁scripts) 尽了*  国资委+  工作进行,  数据来源-  ▁tricky. alpha/ 发愁0 写给1 价钱2  ▁parse3 你没4 拍客5  心血管疾病6  治安管理处罚7 ARC8 民兵9 海事: 邵阳;  阿拉伯< 选中= 守土>  ▁bush? 坦言@ WowA  ▁wiringB  崭新的C  classesD  认为自己E  berriesF  ▁ecologicalG 灾区H  的公众号I  ▁libertyJ ICAK 顺手L  看看吧M  ▁AttackN  发生过O 流逝P  情况后Q  ▁CarolineR  每小时S  LocatedT  ▁ProcedU 电费V  ▁CharityW  ▁terrificX  ▁permissionsY rophicZ  多余的[ ▁lav\  ▁Wave] 加群^  在微博_ 重逢`  中共中央政治局a  所有权b  是一座c Boardd 八角e  接地气f  的手机g ▁Folh 同级i 咸阳j 江河k 重阳l  可想而知m 错时n 生怕o 高涨p killq 寒潮r  ▁bugss  abularyt  部分的u  每一步v iratew 照料x  ▁canceledy  人的生活z 吃着{ 譬如|  ▁slice}  amiliar~ ▁ONE 体征 吾尔  更方便  ▁biom  ▁varies mus  分会场  ▁concentrated abi  不及时 би  总结了 后被  ▁neighboring 算了  ▁assignments 虚心 一针  ▁expressing  中山大学 风扇 战绩 一品 Staff  ▁consultants 源泉  上热搜 路站 ▁${ 记分 颇有 做工  欢迎您  ▁shades  ▁concentrate  ertation  ▁inception 茉莉 sheet  接触过  严格控制  这样可以  给大家分享  ▁Opinion 文具  治理能力  ▁Midwest 原著 去向 噪声 无明 rosso  htaking  一旦发现  Technology  特别注意  ▁portions wild  ▁Lawyer  ▁Communist 共创  ▁са Anyone 方言  名副其实 蜕变  精彩推荐  有必要  ▁Chain  ▁potatoes  标志性  的场景  ▁Reagan  很适合  每天都会 主权 腰痛  有效性  行业发展  ▁lith  ▁troubled 编者 则会 坦诚  ▁excav 晋级 日零  中等职业学校  张柏芝 角质  ▁propose  ▁Burns  现有确诊病例 ▁Ker  ▁statutory 关口 顶层  ▁embraced COMM 明示 中来  ▁slides 协定  ▁Advisor  ▁invention  ▁distancing amped 认购  之以恒  ▁Hern 省心 ▁Lar PY  这个词  ▁Methodist  相较于 受阻 ▁orn 养犬 最先  小提琴 山林  ▁zoom  ▁dense  ▁demonstrations 包住 我了 嗓子 约有 ETH  ▁creep  有阵雨  央视财经 县市 年头 每隔  ▁Embassy 酱香  isy   ▁identifies  秋风  累累   真正做到  ▁drummer  ▁Views 飞翔  老干部 ▁DIY 科幻 螺旋  符合条件 ▁Syl  ▁Sach 布施  ▁Injury  六个月  ▁Scheme 只用  促进了 无事  ▁laying  补气! 信宜" 减灾#  ▁wreck$ 燕麦% 高贵&  帆风顺'  ivities(  ▁Adrian) 唐诗*  ▁Moving+  艺术的,  ▁worries-  是孩子.  开展工作/  ▁Points0  你一定要1  源于网络2  ▁Guidelines3 特地4 Enjoy5 GN6 小品7  ▁additions8  和世界9  极高的:  ▁fingerprint; 更深<  和方法=  及答案> MORE? ▁Das@  ▁WestminsterA șiB  ▁lanesC 朱元D  ▁CookiesE 闭幕F  本项目G  介绍一下H 统招I  而导致J  各高校K itanL  文章内容M 三合N ▁FYO  持之以恒P ▁MYQ 着想R  ▁uniquelyS ▁ЛT 已知U  白衣天使V  changedW jcX  电话号码Y  准确的Z 华府[ 相生\  ▁Elliott] 二〇^ athy_ % 观点和对其真实性负责` 部委a 重在b IPSc  会比较d 停下e TBf Feedg antish 利民i Storyj  ▁Engineersk 练字l  重中之重m  ▁infamousn  须提供o  住宿费p \\\\q 红河r 你你s 最能t  与大家u  款车型v Bronw  ▁Noahx 舒畅y Henryz  ▁Summary{ 西汉| zhang} 浮躁~  自己喜欢的 jb  和文化 但凡 园地  ▁Dust 悲观  ▁Sussex Tel  的自己  ▁Cornell :[ 笑语  为导向  ▁filmmakers ③　 igm  其行为  网信办 rium  ▁Doll 本营  ▁considerations adas  我们分享此文出于  ▁competent  ▁callback  ▁pork  ▁Fifth 朴素 Dir 粮油 尽致 柚子 红烧  处理好  认真贯彻落实  ▁visually 越发  的气质  platform Think pless  ▁Pitt FORM 探亲  周围的  每人每月  ▁ours  具有一定的  ▁photographers 结缘  ▁Wife  ▁beverage 慌张 童心  ▁verification  这首歌 之风 坏死  ▁concess 旺季 安神 bool  ▁hedge  ▁horizontal 缘故 金奖  的日常 趋于  ▁Offer opsis 武夷 老王 接诊  ▁plea  维吾尔 onies  更不能 一抹  ▁Rating  ▁Walsh  ▁Raymond ▁DIS 阶梯 这起 辩证 口径 chang 棚户 供图 atu  登记的  强对流天气  imation 中控 」　  在我们的 一月 再好 idian  ▁crying  请您放心关注 向我 青龙 yster  ▁seiz  的开始  免费下载 撞击 挑衅  感谢大家的 ercise  ▁towers  燃放烟花爆竹  不得以  ▁pause  Learning  姐妹们  ▁unle  ▁villain 尚有 Sky 三家  ▁desires 人称  ▁refined  ▁Producer  注明来源  ▁Dictionary  技术与  身份证件 固执  Example  ▁Nissan  ▁associates 哭泣  ▁Listen  ourneys   耐心等待   购买了   角度看   ▁Vern  吃肉 迎宾  ▁Problem 以最  ▁collectively 垂钓 畏惧  各自的  是世界  ▁professors  然后在  立即删除 卑微 лен  ▁secular  ▁Cliff ▁WR  新基建  teaching   ▁Electronics!  ▁Casey" Royal# 求购$ 过长% 在京& 结语'  ▁Municipal( 行善) 发黄* Helper+  防范意识,  ▁rhetoric- 趣事. ENG/  中小微企业0 极简1  就是说2 韶华3  中药材4  公务员录用5  ▁yacht6 在场7  ▁spine8  ▁pledge9 ugin: Study;  核心区<  而现在= 心声>  的色彩? L E▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁@  ▁ChiefsA  ▁MohammedB  ▁PetersburgC  ▁impeachmentD 身旁E 十点F 帐篷G 海的H RESI vidJ  自然资源和规划K abolicL  ▁verseM 接警N  PropertiesO 二层P 应激Q 埋怨R emadeS 聚众T 自立U GermanV 紧绷W 毗邻X  ▁garlicY 学了Z 千人[ ▁AW\ 摘要]  ulously^ 衬托_  事业单位工作人员` 昌县a  ▁virusesb cesterc  大大的d  ▁optimizatione  Floridaf  为做好g ▁:)h 光绪i ajij  ▁Istanbulk  建筑面积约l 惊叹m  第十六n 出头o 好用p 谦虚q geonr headss  我们不t 纤细u  Subjectv @ 9习近平新时代中国特色社会主义思想为指导w  扫黑除恶专项斗争x 七彩y mockz  违法犯罪行为{  赣州市| 想让} 可不~  ▁examining  都有哪些 地利  ▁Ride  ▁sellers 试探 Fred  ▁Cuban  ▁dividend icz ▁GE umi gio ▁np  就可能  生孩子 偏远 itars  ▁palace 敬老 连载 ▁Jed  ▁listener  ▁Vista  哪一个  养老服务  向大家  成就感  的地区 Stock  ▁excluded  ▁unrest 情歌  坚持下去 Az Too 医美 levant 欣喜  ▁Chin  是这个 才算 ointed achine idates NEWS 水温 蜀黍 心酸  Additionally 搜救 Num  ▁dubbed 顺心 xxx  ▁organizers 大贵 晚饭  riction 阔腿  等行业  成长为  Operation  Repository 地毯  ▁Encyclopedia  重点地区  ▁Solomon IFI  ▁resigned 多条 司长  ▁induct  ▁lenses  防溺水 events 也给 利尿 短裤  报名人员  还是非常  回头看  ▁Richardson 落细 田野 ▁pi 必背  ▁procurement  我开始 indy  ▁editions  ▁Canal  今天上午 indust 黑人  ▁Built  至少有 hari  ▁READ 轻薄 停滞  退役士兵  ▁scenery gex 将由  ▁Knights TN gone 团的  ▁drones 义词  党的建设  行政审批  ▁Apollo  如果孩子  ▁Ferrari 营口  积极探索  人才的  ▁panc 贝壳  资格的 听证  ▁Curtis  ▁undertaking 保费 糖果  与孩子  plementation ructor  ▁breathe 诗书  是一样的  ▁defended  　——　 文山 生字 ellen anding   ▁Ivan   ▁pipes   先锋模范作用   ▁prepares  证照  是一条 量产  ▁encounters 不以 aines 利器  ▁Agricultural 的头 侵袭  ▁Zoom  ▁combinations  部主任 晨报 放眼  ▁logged  德尔塔 GH 问了  editor! 文摘"  ▁pediatric#  建筑业$  老司机%  ▁UNESCO& 是啊' 本校(  ▁Healthy)  ▁Activities* horn+  ▁laughing, 一盒- 蜂王.  不构成/  的宣传0  能让你1  ▁Solution2 她会3  ▁happily4 野心5 养身6  ▁Recently7  ▁Fell8 逃跑9  ▁notify:  不见不散; 深挖<  婚姻登记= 紧邻>  ▁fellowship? Louis@  ▁dockA  ▁tendencyB  ▁ConstantC zhaD  外面的E  经过了F arasG  NothingH  白皮书I believJ  ▁placeholderK 放飞L  ▁SisterM  LibraryN  多万元O  中国工程院P 却有Q  辛苦了R NINGS  ▁ExpectT  必不可少的U 的画V  只知道W  ▁NikeX 中层Y ▁tilZ  广大党员[  ▁Heather\ 家在]  ▁Rochester^  ▁activated_  楼面价`  女生的a  进展情况b  的法律c 前世d 年产e  ▁Flatf  forgettableg Safh  ▁optimizei 除螨j  通知要求k  ▁Sensl  ▁Shirm  ▁gatesn 跟风o  优惠价p  基本功q  小区内r 骆驼s mott  ▁nachu  ▁Columnv 征兆w 福德x  ▁contemply  分之一z 抬起{ 新安| ACC} 并有~ 房的  ▁marvel  ▁Panama Iss 神农  锂电池  ▁Timothy 老爷  变化的  志刚哥  legraph 算命 &# 仰望  了一大  ▁conceptual  责任担当 quez  随处可见  ▁regulate  ▁nucle  ▁bureauc 自考 高明  release  ▁Barr  以上就是 LV 传言  ▁Punjab  ▁republic 极佳 癫痫  安排在 ▁mum 走好 金秋 如愿  ▁Cowboys 张小  取消了 项链  你现在 cin  实景图  齐齐哈尔 敲门 圆梦  ▁jointly 大招  结构性 ▁Ki itat 往生 昆仑  网站上  ▁overlooked 用书 胃病 中被 pta cano ighed 未曾  ▁bundle  帮助我们 余个  ▁volcan UA 自费  文化课 ери  副省长  Perform  ▁revenge  ▁obesity 出彩 ▁psy 意料 does odi six 扣押  ▁admissions rowth 一目  法学院 `,  olphins 不和 免试 Jud  ▁tribe  中心组  ▁Martha 进店 开源 ▁Fig  ▁sadd 仿真 重伤 Age  家里有 分层 次于 三甲  分钟的 帮他  ▁vibe  personal  ▁demonstrating 四人  的员工  存在于  ▁Polic  ▁balcon  更应该  政法大学 infect  ▁damn  云计算 ▁Zel 闻到  olutions  ▁culinary 写得  但是他们  ▁Sask  鲜明的 晚舟 索要 Hill 闪电 评判 Quick  来这里 侧重  ▁Sharon 人说   Official  配角   ▁endangered  手腕  ▁° 昆虫  ▁portraits  ▁identities 动工  ▁Commander 扭曲 Ever 年夜  ▁Madd 可为 进城  根据国家  研究和 航运  自然而然 电厂 ▁db  淋漓尽致  不放! ▁NH"  不合格的#  ▁lottery$  ▁contributors% cq& 完结' 得过( 大院)  ▁tariffs*  感染了+  ▁Visa, 率的-  ▁theatrical.  出问题/ 补钙0  ▁Especially1  各级党委2 细微3 庞大4 机型5 爱意6 cats7  ▁crashes8 Tab9 圆圆: 楷书; : 3我们分享此文出于传播更多资讯之目的< 磅礴=  和健康>  皮肤的? 皮的@ 海区A 怎能B  在一次C  村民们D 何种E  因为没有F 赚了G  综合评价H 还说I 重来J  保定市K nameseL  inologyM  ▁FacilityN  ▁SuperiorO ioletP  的时光Q  ▁WadeR 形体S 南非T  保护自己U ▁thyV 就让W 余家X  浓厚的Y  的情感Z  ▁smoothly[ UNT\  上的讲话]  ▁Winner^ 海水_ 禅师` 纳斯a  家长和b  有一颗c  ▁Proceedd  ▁championshipse NASDAQf  ▁Boulevardg 接听h  多措并举i 日元j )　(k  ▁betrayl  ▁Opportunitym  你也可以n 瑞典o  空白处p  ▁Troyq 听得r 养花s 柑橘t 沙雕u  聚集性疫情v  authorizedw 之夜x 特权y 跟她z 套票{ 年版|  所能及} 发自~ 神州  生态环境局 穿戴 堂食 Loc 回向  理事长  ▁acquisitions 武威 饰品 人已 大红  ▁Parts  ▁Identity ▁OFF  省委常委  ▁Perth  ▁Huawei 霸道  前一天  可怕的 亿的  ▁Martinez 芝士  相关情况 捆绑 搭乘  有一份  现实的 剑桥  十足的 osta ▁Bee  多样性 ▁UFC  北师大  ▁generates  中心城市 着他 报了 将与 莱斯  一下吧  ▁Understanding  传染病防治  ▁mines duce  ▁turt 耗材  了一件 玛丽  决战决胜  时发现 ▁Fla  安全事故 红茶 礼金  和精神 每户  一年四季  社会实践 Cook  contact  洛阳市 半身  奥斯卡  指导老师 dan ▁É 对新  单纯的  ▁stair  ▁Pharmaceutical  ▁foul  ▁Dental  外国语学校 其后  案件的  ▁bombs 只因  张老师  ▁Allah 值钱 都与  ▁Meghan  ▁Porsche ▁HTT  体现出 悬浮 空的  ▁Chancellor ▁lou  ▁Courtesy 富士 剧组 大悟  ▁Material ignt  一女子  ▁sticks  ▁Insider rivers  ▁Elections 稳固 Perm amples 接手 林某 ucus  打印准考证  ▁simpler  ▁Aboriginal  ▁observ  最大程度  ▁dissemin 结识 闲暇 全勤 得的 金星 树上 anye ▁Und  要充分 andro  等场所 kens 深渊  会遇到  的实际  ▁coupon  还能够 灵敏  全力保障  ▁portrayed  的高级  个工作日内  cussion 竞价  立秋   协调能力  班会  igi  辞退 吃瓜 客家  相对来说  ▁crashed  ▁precip 仙境 拌匀  ▁Sheffield  ▁kilometres  处理器 ▁Cel wi 带回  有帮助 之名 法语 ▁NW tails   ▁announcements!  ▁provinces"  管理法#  感觉自己$  ▁learners% deep&  毕业生就业'  是美国(  ▁Griffin) Commun*  砥砺前行+ 觉悟, DER-  Adapter.  ▁briefing/  少年的0 靶向1 气虚2  从源头3  ▁Moss4  将按照5  请提前6  atorial7  ▁relevance8  的灵魂9 ▁gly:  帮助他们; verter<  ▁giants=  ▁Wait>  ritional?  成交价@  ▁WheelA  政策措施B  还包括C 涨了D  深度融合E 鸡精F JDG ▁УH ursorI " 打赢疫情防控阻击战J  还存在K  ▁gaugeL  ▁wristM sfN LMO  ▁fatigueP 生生Q  朱元璋R 时就S ivierT ricanU  ▁absurdV  ▁CafeW IncludX 抗洪Y 需在Z  人才引进[ 格林\ boat] . '赞同其观点和对其真实性负责^  散发着_ 楼栋`  名额有限a  在全省b  Registerc  孟晚舟d  ▁Protocole 伸展f 朴实g  中山市h  ▁seamlessi 不解j  的大力支持k 出名l Deathm ignantn  ▁escapedo 住的p ▁CPUq  有很大r  重要提醒s  ▁Croatiat ▁MNu  坚实基础v  ▁opposew 削弱x 厂区y 来稿z ▁Lak{ GF|  ipeline} 常熟~  ▁vide  容易导致 拍了 ▁PE  有钱人 可期  ▁Louisville  记者在 ▁Rus  在线教育 倒闭  有一条 AAAA  ▁implies 液化  价格的  ▁choir  诉讼法  安全风险 ▁Kas  交流学习 driver  持续关注 端庄  ▁treaty  ▁insects Third 这与 dot oche  ▁Outdoor  代表性 rolet  ▁uncover  ▁Actually 绚丽  有那么  切实增强  ▁Alternative  ▁doubles  ▁relied 评比 论语  第十七 zhen  ▁advertisers  医用外科口罩  ▁adjustment  不相同 寒气  日消息  死亡人数  Available  工业园  ▁closet arez oyu  这其中  ▁vegetable 一汽 如若  ▁ancestors  都市圈  和力量  对企业  ▁statute 示例 ('#  和时间  ▁promptly ▁VIP  ▁cups  爱你的  ▁protagonist  仔细阅读  ▁reproductive 到岗  ▁Treasure 反响  ributed  的小区  ▁Billion 兼容 书包 Ro Rose 演变  自然科学  ▁victories ERY  ▁TODAY ▁CSS  次全国 ▁Cul 窝点 苦瓜  大拇指  ▁metric  深入了解  ▁detective 回乡  ▁Excel 悠闲 麦克  内容文字 起居  iencies  重点关注  左右滑动查看更多 pages 追赶 enary  矛盾纠纷  ▁homework  的了解 容颜  第二期  ▁recovering 和非 INA ▁Ran  ▁ст  据报道  生活质量 忽悠 nl 生化 grass 拍戏  你身边  建党百年  eteenth 二轮 创城 青蛙  ▁parliamentary 市域 Ol 牌子   重要指示精神   ▁illustration   ▁yields  ▁CI  FONT 乖乖 流向  ▁Cambodia vc 由来 设在  责任和 结对 得一  ▁Evidence 犹太 :　(  ▁felony  步行街  集团的  ▁Aware  我们党  并不意味着  南宋! 但要" 天桥# marine$ 之分%  主义的& random'  景德镇(  ▁wallet)  两个字*  ▁additionally+  航空航天,  ▁Mode-  提高到.  供认不讳/ 成龙0 摸清1 这根2 微观3  ▁Salvador4  没时间5  ▁eleven6  ▁marathon7  ▁pedestrian8 通通9 锅炉: " 中华人民共和国刑法; Single< 进货=  进行一次>  ▁Roosevelt?  ▁threads@  ▁promotionsA CBDB  成立的C ▁taD  为他人E  ▁pharmacyF 双双G  代理人H  ▁acceptsI 额头J 身在K  ▁FurnitureL  创业者M  ▁caregN 粗暴O wigP  因为是Q 嘴上R  的一半S  ▁CarsonT  ▁intactU  ▁kindnessV 仙人W 普京X скаY ▁vicZ  ▁Lithuan[  ▁vigil\ headed] MIT^ erpt_  тель`  小男孩a 绿的b 该院c  ▁comediand  军人的e  不同的是f vertedg 体量h 金字i  ▁risenj  ▁wholesalek Davel  ▁overturnm 飞跃n  ▁roundedo itoredp  ▁neglectq 转给r  ▁premises 收好t arisu 侵入v  ▁overtw ▁Bilx  ▁lacksy 告白z 花期{ 补录|  ▁blown}  物业费~  现代人  ▁Chronicle oulder 出卖  的天气  ▁credibility 求学  长大的  uations  ▁lymph Govern  ▁immunity 装的 车展 轻信 ▁USC 排斥  全媒体记者  ▁Kazakh  ▁axis 天数 就回  国家监委 Heart 桃子  ▁Connection 且在 刀具  ▁Jill clean 因缘 脉络 药学  ▁Jenkins 主页 伤残 以东  的画面  ▁Methods  干净整洁  ▁Dating  🏻🏻  ▁prediction  ▁Require  累计治愈出院病例 八年  八项规定 捕捞  产业集群 SEC ▁irr  长时间的  ▁hugely 大宗 盛夏  management  胃肠道  writers  ▁heels  ▁Christie  小助手  设计与  ▁suprem  ▁evaluating region HOW 情趣 ego logger ▁Hav venth 帅哥  ▁bored  connection  ▁successor RY 他从  是怎样 点了  茅台酒 ▁Jak 村子 ▁Rid 殡葬  ▁Louise  出不穷 淡然  接触到  供大家 ervlet  ▁Belarus 舒心 银耳 Made pick ×× 先用  抢救无效  ▁seminar 贬值 图网  自由基 rera 不孕 率领 行长 gae 掉的 好坏 entity 接班 此生  可查看 易燃  ▁drafted  ▁пре  ▁Gren  ▁analog  ▁Pentagon  ▁missiles 东县 indsay 这句  的姿势  人心的  自己做  全面加强 atche  ▁Measure 红润  ▁Dogs  ▁factories 私立 纵深 Delta  后方可  和美国  ▁automobile 反面  ▁Hil   大使馆   最常见的   ▁glac  衷心  即可获取  扫码添加 lg  说出来  ский 湘西  ▁planets  ▁attracting 到户 日为 避险 sort  身份信息  ▁todd  几年前  ▁Cream 教委  学历证书   ▁excerpt!  中指出" 女婿#  原创文章$  继续教育%  ▁Conn&  是国内' 橡皮(  关系到) ▁Gor*  ▁sounded+  ▁Infect,  ▁marker-  ▁boarding.  老师说/  若有侵权0  ▁Ballet1  可申请2 完好3  ▁reservation4  是多么5  改革创新6  方舱医院7  方法和8  银行业9  ▁correction:  ▁sexuality; Files< 烘干=  ▁urging>  ▁dawn?  ▁Shipping@ ChemA  ▁concludesB  的风格C  ▁BrightonD 人多E  浓浓的F 刑警G  文化馆H  基本都是I  工作了J 到账K 衡阳L RhM 大类N  ▁marginsO  实在是太P ▁ArgQ  ▁ConfederR  常通风S 就该T 建的U ▁GLV ▁LivW  ▁disappointmentX 本站Y  ▁ShotZ  ▁Claims[ Ari\  legiate]  ▁organs^  ▁astronaut_  ▁predecessor` iffea 青菜b ~~~~c  隐患排查d  ▁warnse  ▁riderf 桌面g ▁Nuh  ▁boxingi intersj CDCk  ▁patentsl  创建工作m  妈妈们n 人头o 兴安p naiq  ▁systemicr  ▁Effectss 不带t  ▁Pricesu  可能会有v 例行w  把握好x  ▁pharmacy 群防z  Researchers{  ▁notifications| 暴躁}  海淀区~ 嫩的  通过的  发表了  ▁anten  ▁tribes icism  ▁campuses  本文由  ▁queue " 工作领导小组办公室  初中生 ▁Reb  ▁turnover  希望能够 Cert 油气  ▁Beer  ▁Devil  chester  ▁gifted  一反三  碳中和  ▁Pione 定居 编造 飘香  ▁Mong  ▁annoying 无从  ▁Eyes  层出不穷  ▁aerial  ▁Otherwise 已成  微信支付  ▁fixtures  换句话说  ▁perfection 二驴 lag 骨髓  ▁hearings  不注意  多大的  等环节  ▁Adelaide 争论 逆境 大灯  问题导向  到来的 三方 入库 各镇 分量 信誉 ▁RF  ▁stairs  提醒广大  ▁Outside  ▁Governance 四位 日落 酰胺 Args 修护 到老  行动计划  ▁dorm  ▁Northeast  有关单位 美图 mun ▁(' 多肉 报料 椅子  ▁cleaned 紫金  可以达到  印发了 盲区 铜川 防暑 ;④ Hor lot ENSE  长效机制 之二 职员  棚户区 晋江  ▁Investigation rowave  ▁efficacy raz otyp  ▁dresses 忠实  千万人千万人  ▁Traditional  ▁Belf 脾虚 危及 一图  ▁logs  ellington 墨西  Australian  关注微信公众号  ▁Blair  学习强国 + $灵异吧灵异吧灵异吧灵异吧 Focus Season 柿子 Tre  ▁helm  ▁divine 每逢 Dead  民警在  洗发水 鲤鱼  ▁composite  军队文职 而你  几十年的  ▁invaluable onduct  ▁overwhelmed  小事也是这样  ▁sculptures  特定的 摆摊 起身 Emer  ▁Performing  ansing   隔离医学观察  新盘  朝廷  科比 CES 老爸  ▁Cult  ▁martial 大户 满脸  ▁gloss  绝不会 ▁Lav  ▁gamers 嘎嘎 Abs  迷人的  黑芝麻  ▁pathways  ▁feminist  ▁Azerbai  conscious   ellites!  ▁grabbed" ARCH#  重大的$ 涨跌% Mir& 外省'  ▁heights( 幼教) ▁RA*  ▁Pont+  ▁mate,  ▁incomp-  的正常.  ▁quarantine/  ▁songwriter0 也从1 花瓣2  报警称3  ▁trek4 费力5 ▁RV6 ▁XML7  倾向于8  整治工作9  ▁domains:  在公司;  直通车<  ▁Hook=  ▁Worldwide> 历任? 口镇@ 壹号A  给宝宝B 此举C  ▁KaneD  在城市E 风尚F 才行G  看这里H  自治区党委I  生产和J 汉服K FrancL  ▁MarxM  ▁AdventuresN 说不O listsP  ▁acquiringQ  ▁exploitationR 时辰S  ▁desperatelyT 斗罗U ▁auxV  ▁inabilityW RoadX 均由Y 暖气Z 余杭[ 摇头\ ▁Exc]  ▁hurd^ 连环_  ▁Cardinals`  ▁Marina  参加过b 申明c 用作d 就和e  以下为f  学位证g  食品有限公司h ▁UVi  ▁mountingj  ▁preservingk spentl  指导价m 奥数n 排污o 白头p  本人的q oyager  ▁reefs  ▁synct  ▁invisibleu  ▁reminisv  升职加薪w  ▁civilizationx 盎然y 学分z  ▁relent{  ▁invalid| 分担}  ▁Auckland~ 言之 命案 成像  证监会  网格员  ▁merit  都必须 个案  作为一种 霆锋 副总 点心 罪名  进行调整  的人在  ▁Hugh wife  ▁byte  ▁energetic  的大部分  ▁incorporating 悉尼 乙烯 散布  ▁surgeon  ▁Peterson 区县  党的十八大  ▁Boat  ▁debuted 我心  做好防护 uggest 上个 预估 鸭子 杜鹃  ▁Honey  ▁blamed лю  的本质  提前批  企业管理  ▁Skin  从这个  也不例外  ▁conting  ▁Ventures 点头 题为 我用  　"　 教堂  同学的 数控  后视镜 非诚 去处 朝向 作画 会比 Lock  很多朋友  我们都是 eworks  ▁NASCAR ▁vou 动图  坚定的  联系我  ▁crushing Ca 王宝  ▁Monroe  ▁Mercury  ▁Monster 广汽 ALE mor 乳酸  ▁phrases  cultural  才能让 问她 通州  恢复了  更新的 听话  因为这个  ▁hobby 正版 疱疹 导购 耳鸣  根据地 肩颈  人群中  ▁cemetery 他用 新旧  双十一 车头  ▁Jagu 安门 开到 infl  面对疫情  ▁bats  ▁Boris 免收 陵区  发展到  说到底 Der  already 黑马  ▁Labs 下入  是全球  科学与技术  身边事 osion 小块  情人节  已久的 性命  无异议  ▁bullying 霉菌  音乐会 叙述 пе  对社会 Sa  ▁Meyer 繁琐  条件和 谁知  或个人  篮板   ▁emphasized  ych   可以使  楚雄 汇率 ▁Adm  生肖兔 give  ▁flesh  ▁tenant  ▁Categories 两三 idos court 眼圈  ▁interval  ▁resear  ▁groundbreaking nas 借此  脂肪肝 团长  守信!  ▁closes" vm# 年老$ 超前% Club& regist' 列举(  ▁Kard)  ▁believing*  ▁transmitted+ 百里,  坚强领导下- 张扬.  犯罪团伙/ KK0 忍耐1 艾草2  期末考试3 妨害4  ▁Seth5  kswagen6  证明了7 三宝8  的组织9  ▁marble: 原版;  临床上<  Jennifer= 观望>  ▁Lagos?  ▁Megan@  可怕的是A DriveB UnlikeC  大队长D  ▁BrandsE  无任何F  ▁extensionsG 可将H 一勺I  第一季度J  回老家K  临床表现L  公共安全M  ▁SerbiaN  语言文学O  ▁YogaP  图中二维码Q 两口R 布鲁S  女明星T  南昌市U azerV  fielderW chiefX DestY 奥秘Z 据此[  的局面\  ▁scenic] 校门^  表彰大会_  ▁plag`  ▁wallpapera  ▁decorativeb 萧山c  好的人d 战区e  的那些f 玛拉g 溪镇h  ▁pleadedi  ▁Wirelessj  职位表k  ▁aforel  往下看m 越长n uffleo 走失p  betweenq bmr 铁道s  扬州市t 有几u  了一点v  饿了么w  ▁optedx  认识了y  人可以z  ▁tattoo{ 两张| 装扮}  ▁Dial~ LIC 调休 ▁jar 翻车 Cra  ▁Abdul fires widget MAC sys opal  专业人员  ▁Avengers  ▁cancellation 和省  ▁Dell  ▁steadily  并要求  ▁inadequ  苑小区 为难 复读  Pingback  aditional  interface  ▁minorities  production 营地 pd 海边  ▁upside  ▁Located Pers  ▁synchron 赛区  蓝色的  病毒传播 runner  经营管理  ▁Moses inski oise zym  ▁beam 有福 ▁ul 少于  就发了 lass  ▁pioneering ▁ate  ▁Norfolk  的最高 体面  护士长  无所谓 传真  在进行 环线  全国统一  改扩建  ▁absorb  最需要 Icon 心血 ▁""  ▁Package  兴趣爱好  ▁wounds  ▁mitigate 小手  ▁Kiss ▁Lem  Throughout 偏高 春日  artisan  ▁Eston  ▁Spons  能达到  专用发票  ampagne rob  上班时间  信息登记 Taking  ▁javax 暗访 沙特 送礼  有针对性 凝心 Simple  墨西哥 知青 orget  ▁Wrest  Parameter 站长  太难了 itas 的黄 黑夜 riad 软弱 再把  ▁mailing  ▁advancement  ▁Customers 寻衅  基础的  点个赞 Legal GG 郭某  主要为  ▁Lines  ▁Annie  ▁Colleges 定投 归还  问题和 marks 永州 到过  黑寡妇  重要论述  血小板  相关单位 巴彦  市委市政府 静谧 巴中 饭局 etz  在此期间 antom 他想  仅次于   ▁Murder   等着你   ▁scales   莆田市   分布在  ▁qualification 礼让  甚至在 backed 择校 偏见 ▁Aus  立德树人  武汉的  ▁warmth  ▁Beast 老头 ▁Sed  ▁appropriately 获评 ▁Dat  力所能及  自然会   ▁illegally!  ▁settlements" 足额#  主要负责同志$  知名的%  的建筑&  不见了' Chair( 品鉴) 亮眼* 元钱+  王老师,  ▁Legends-  责任心强.  ▁displaced/  没问题0 育才1  我的心2  ▁Pinterest3 按揉4  呼和浩特市5 orman6 姥姥7 车队8 炒菜9  ▁params:  ▁rendering; 开园< 齐聚=  举一反三> 素描? Movie@  ▁KennethA istasB 法案C 车轮D ▁SRE  机构和F 减排G 圆润H 已久I  ▁throatJ 同居K  使命感L  日出生M HallN  强制措施O  ▁stellarP 阀门Q  履行职责R 茶饮S 炎炎T  和别人U 心无V  的不良W PPOX  亿元的Y  ▁pasteZ tein[  仅代表\  周岁以上]  ▁consuming^  ▁Charleston_ 缓冲` starsa  ringingb  ▁snacksc 平庸d  ▁carvede  ▁Romancef  只能在g 云龙h  同志任i  如果我j ▁Ecok  ▁Carpl 就曾m  这是什么n  ▁shopperso  ▁breathtakingp  ▁Boomq  ▁inputsr ermats  ▁Marcot  ▁Cableu  给学生v yardsw 看淡x 凶手y  会根据z  清洁消毒{  ▁pitcher|  ▁Sisters}  谢霆锋~  ▁supposedly  儿童的 低估 湖畔 失落  扫一扫 + $我们将在第一时间删除内容 轮流 速递 ···  ▁Optim  ▁Patrol 损耗  ▁Announce 威力  系统性  ▁Elite  ▁Hamb bg 肾虚  为一体的 注释  ▁malware  ▁ambitions inde  ▁geographic  儿童节 斑斓 demand  还有一种 Until  ▁Riley  ▁artifacts 抓细  ▁Height  ▁sampling 车窗  ▁transc baby  ▁forgot  正式开始  ▁Matters 展播  ▁Distribution  也很难  短暂的 眉山  emiconduct  领先的 庆幸 也比 ▁IO  ▁merge  ▁Depression 缩影  当中的  ▁wholly 开机  mentation 幸好  ▁grinding  创造力  created furt jr 北纬 庄子  采访中 能用 软糯  经济增长  ▁norms  ▁lining 德里 远处 外围  游戏中 工学 Fox 发帖 eled ▁bod  入境人员  ▁problematic  acional  一整天 个头  ▁пере 六稳  你已经 GIS  ▁distress adh  免费发布  ▁clearance  ▁lenders Bon  食品药品 底的  一切都是  ▁interpre  干干净 液晶  了大量  依旧是  ▁Domestic  中国驻  服务工作  ▁favorable 领事  ▁monitors  ▁constructor Audio allic  而且是 木质 ▁wax  蒙古国  ▁thrilling 他被 使之 Lo  ▁cabinets 宫廷 惊险 中性 榜首 遗漏  主要以 三高  ▁incarcer 竞选  收入的 格里  才是最  ▁misconduct 姜片  集成电路 threat ethyl  你们是  ▁pilgr  ▁allocation 水里  ▁Wool  公办学校 交融 米兰 辛丑  及家长  年度汇算  ▁Ment ADA 站台  ▁Config 食管 地对 种草  ▁yearly shall Prince  ▁Randy ),　! Mock"  component#  什么事情$  ▁Animals%  ▁Burton&  ▁Patient'  ▁shallow( 在高) 靠前* ▁mim+ 原味, ▁ü-  好工作.  的科学/  的强大0 Deep1  ▁Hopefully2  的记忆3  的主要原因4 散打5 ба6 成一7  ▁timeless8  帮助企业9 上瘾: 主意;  不限于<  族自治州=  非诚勿扰> 也罢?  更不会@ 婷婷A 精进B 酸菜C  olutelyD  改造工程E  ▁JackieF &&G ▁EURH  到现场I 追随J  动不动K ▁BunL HarryM 走私N  ▁scatteredO 震慑P " 中华人民共和国国籍Q 议题R 过头S 多有T 混淆U  ▁illnessesV upiterW 以案X 失常Y eniZ mitt[ ariat\  熟练掌握]  ▁promin^  ▁commissioner_  初级中学`  ▁medalsa Hawb  本土病例c  ▁Namedd 越早e 　;f Wasg  安全意识h 及早i  ▁glancej 赌场k  ▁disprol  ▁overheadm  ▁ecosystemsn  ▁Fieldso 举手p  和教育q  ▁upholdr  ▁distinguishs Schemat  看出来u erersv  如果觉得w  都是一个x  发布通知y  ▁Nestz Es{ 延年| 春联} smith~  分享的  ▁fusion 三口 美感 操场  供电公司  ▁Profession 天山 线和 道家  看看看看  ▁UCLA 果肉  做准备  温馨提醒 一趟 排气  的大学生 why  是一个人  ▁Millenn 沥青 进站 兼备 AMA arette  人群的 憔悴  ▁guilt 不虚 刷卡 强壮  九分裤  你的生活 当即  ▁broadly  ▁resignation roads  寻衅滋事 乱了 嘲讽  ▁dread 熟知  浓郁的  干干净净 忧郁 成败  有几个 是想  ▁Durham  ▁declining  加强与 招工  就会发现 modern  irement  ▁clearing ▁pec  努力的 atted kinson 天性  一目了然 满面  ▁sadly 妆容  分钟内  ▁Roof  刚刚好 ▁på  应纳税  实用性  ▁writings  ▁advertisements  ▁Lenn  ▁Trent 瘟疫 梧桐 不乏  不要让  几年的  ▁Myers  高血脂 格调  很多网友 没啥 多变  borhood  ▁Decision  没多久  医疗保障局  ▁sensation  ▁Population Logger  莫斯科  跨境电商 匿名 商机 nels  ▁competitor 纪实  的伤害  ▁enduring 鲫鱼  这种人 强者 还对  ▁gods  ▁drown  dependency 瑕疵 盆栽 cro PF  可以多  违法所得 偏低  为什么呢 Async  ▁Lamb 回看 zheng  ▁Analyst 懂了 单个  安全稳定  防护用品 dirty 东门  文明办!  network! 小巧! 推移! 两类! ensely! 活的! 老天! 跳跃! ffff !  ▁threaten !  ▁anticipate !  RELATED ! 赚取 ! 星河! axis!  圈子里!  打赢这场!  inguished! arius! 这批! olin!  回家后! 树脂! midt!  ▁packing!  不要去!  财务自由! ervers!  ▁debug!  ▁Portal! 巴马!  钱生钱 !  社区的!!  结构的"! 度高#! ད་$! 忧伤%! 琢磨&!  第三批'!  ▁arrests(! enos)! ▁Lor*! 前瞻+! 口干,! NB-! 它会.!  ▁grasp/!  在规定时间内0! 拼接1!  ▁Similarly2! 要让3!  ▁Shopping4! ió5! 谁来6!  要严格落实7!  bootstrap8! 斯基9! 草地:! 机油;! icable!  突击队?!  在里面@!  itoriumA! igntyB! 卫士C!  现金流D!  ▁limitationE!  第二轮F!  operativeG!  ▁definitiveH!  ▁UtilI!  一览表J! 轴承K!  ▁unlawL!  ▁MirrorM!  ▁resilientN!  ▁motorsO! 托福P!  目标是Q!  点击下图R! 交纳S!  ▁sweepT!  ▁bestsellingU! ▁FitV!  ocativeW!  到哪里X!  时间长Y!  大自然的Z!  生肖鼠[!  众多的\! 不休]! Dar^! Disney_!  第一届`! elina!  红绿灯b!  人的一生c!  排名第d! COLORe!  ▁illustratef!  withoutg!  ▁readingsh!  ▁definitionsi!  保卫战j!  ▁catalystk!  听说过l! sensm!  认识的n! NERo! 无暇p! 天一q!  ▁Supermanr!  ▁originateds! 山庄t!  拒绝了u! 河东v! 不与w!  ▁Talksx!  althoughy! 闪烁z! ño{! 上司|!  改变命运}!  Featured~! 灌溉! Much!  ▁sued! 介意! 日和!  和建设! 七十! 致病!  同事们! 五菱! 寻人!  岁的小! ál! 迈进!  ▁grit! 洗碗! hagen!  新鲜事! ос!  ▁Icon!  ▁Tank! 光盘! 急着! 发泄!  Germany! 免除! 比我! since!  ▁simplicity!  点对点!  ▁endorsed!  工商银行! rub! Params!  清新的!  微信同步! ellery!  ▁Experts!  ▁inmates!  ▁stark! ▁Tah! 低压! zhu!  自己想要!  ▁inherit!  ▁Marl! jquery!  自古以来! 微创!  万元以上! 顺势! 学完!  ▁designation! 铜陵! ▁Sit! 如初! 点我!  ▁noble!  ▁Reader!  ▁complained! 是小! EY!  不是因为!  城际铁路!  选择在! Uk! jw! 药的!  ▁Dollar! ▁soy! ря! 江淮! 当面!  房子的!  ▁Trou! carbon! Son! 按月!  ▁alignment! 卫衣!  ▁Font! 不收! Virgin! series!  没有那么!  一个有! ixty! 停水! 灵璧!  县级以上! KO! 更低!  expensive!  ▁bathrooms! 阅历! ouns!  rimination! Wall! raul! 挥霍! 施救!  ▁Buddhist!  ▁holistic! 以求!  进行公示! 拘役! 文理! 瑞金! charg!  塑料袋! Bec!  爱国卫生运动!  女儿的!  让人们!  合格分数线!  ▁undergoing!  ▁Atlas! azione! 风俗! ryn! 不失! raise!  坚守在! undai!  不足以!  委员会主任! 米粉! ▁orb! 唱和! 火腿"  两个孩子"  ▁func"  ▁Designer"  ▁entrepreneurship" 响了" 爱岗"  不如意" 急剧"  ▁coconut " Apr "  不同程度的 "  ▁convincing " 全军 " 招待" 心力"  ▁Imagine" 爱奇" 防灾" 鼻腔"  ▁expands"  天左右"  的温度"  的时间里" ▁lid" 埋头" 阅览" 冰雹" 之子"  意义上的" ilib"  高中数学" 腐烂 "  生产总值!"  ▁chorus""  记者了解到#"  ▁WhatsApp$" )\%"  ▁Universities&" 肾病'" LAY("  幸福生活)" 审讯*"  ▁Ticket+"  Between," 下定-"  但是他." он/" ITED0"  ▁swept1" 误导2" 图标3" sle4"  ▁administer5"  ▁pars6"  点击阅读原文7" appa8"  应该如何9"  ▁Explore:" 交集;"  固定的<"  主任科员="  ▁groom>"  顺利进行?"  ▁Pure@" 安乐A"  ▁pursuantB" ▁VMC"  家公司D"  开除公职E"  免疫屏障F" PORTG"  开新局H"  开放时间I" 养血J"  一个字K" 屏蔽L"  保障工作M"  计划生育N" graO"  英语学习P" 自备Q" 腰带R" 学制S"  过敏性T"  ▁heterU" )}V"  社会稳定W"  就是对X"  ▁HollyY" aidenZ" 不复[" 同在\" 迎新]"  ▁dominate^"  ▁Chip_" ▁JO`"  的美国a"  养老机构b" leysc"  ▁Institutiond" 网贷e"  WASHINGTONf" vong"  自己的人h" 揭阳i"  ▁Kleinj" silk" 视网l"  就比较m" ▁WORn" 内向o" 归案p" 测验q"  ▁collaborationsr" 心境s" ▁Swet"  wikipediau" 交道v" 大哭w"  ▁Milex"  而这个y" ▁"/z"  准确把握{"  小知识|"  然后用}"  ▁Seoul~" 仰卧" 听众" 远超"  为阳性" 白马"  ▁Pars"  其实在" aidu" 子弟"  ▁Janet" 自此"  也一样"  ▁stakes" 名企"  新型肺炎"  同微信"  更喜欢" 落幕" amins"  贴心的"  广场舞"  Password" 下锅"  将达到"  acements" 母子"  更名为" 成佛"  已进入" 欺凌"  恨不得" 合照" ▁Tas" ▁Ara"  的文字"  胡椒粉和"  ▁username"  ▁remembers"  效果好"  标准和"  ▁Malta"  ▁Paradise" 劝说" 三房"  manager" CBA"  ▁taxpayer"  心脑血管疾病" 乌海"  就觉得"  ▁accidentally"  ▁displaying" anner" yun" Gov" 当头"  ▁fierce"  的行业" 荣华"  抗衰老" fn"  既可以"  产业园区"  的手段" 自贡"  ▁assure"  粉丝们" oors"  ▁abundant" ( !具有中华人民共和国国籍" Match" 吸管"  contract"  ▁legends" 高升"  分泌物" arde" astery"  ▁nephe"  ▁christ"  群防群"  召开会议"  ▁desirable"  老龄化" 鲜香"  ▁Greatest"  ▁те"  ▁analyzing" 伊始" 在各" zero" 受邀" 还挺"  户外活动"  ▁Sessions"  资料包"  确保安全"  ▁naked" 补缴" asso"  ▁inspirational"  感情的"  数字化转型"  可视化" ▁Era"  ▁fetch"  提取物"  portion"  徐州市"  科创板"  疫情爆发"  问题整改" 顺其"  ▁pledged"  ▁installment"  房间里" ▁lug" 操纵"  的新鲜"  ▁activation"  ▁herb"  新冠肺炎患者" itated"  ▁encrypt"  ▁correspondent"  方方面面"  人生中" ем" 国立"  ilingual# 思绪# 盛典# 肉肉#  物质的#  招生录取#  工程有限公司#  ▁Across#  ▁mentions# 低谷 # negie #  ▁Tehran # 底色 # 伤者 # 居士#  展开了#  ▁unwanted# 新娘#  ▁deduct#  ▁ideology# жа#  就意味着#  招聘网#  ▁roku#  ▁Pengu#  检验标志#  ▁descent#  ################# 养的# 痉挛#  ▁neurolog# 小幅# ▁Fix #  委书记!#  你永远"# iago## 办证$# Ask%#  ▁Grid&# 向党'# 就从(# rev)#  ▁onboard*# cler+# 在编,# 心如-#  但因为.#  前提下/# plash0# onial1#  ▁livestock2# 花儿3# 山歌4# ▁Lip5#  ▁learnt6# 公勺7# 的诗8# wyn9#  ▁Bren:#  ▁mathematical;# 张学<#  也成为=# icator>#  ▁numer?#  ▁Emperor@#  上下功夫A# 日新B# 一幕C# 极乐D#  正确答案E# 模考F#  ▁communicatingG# 打赏H# 跑路I#  以及对J#  ▁ferryK# iariesL# 呼声M# 都用N#  的计划O#  ▁collapsedP# 基石Q# 问你R# ievesS#  ▁swapT#  InstagramU#  叙利亚V# 花鸟W#  雌激素X#  ▁RomneyY# couldZ# 昔日[#  你一定\#  ▁punct]#  医疗卫生机构^#  ▁Khal_#  ▁customize`#  ▁Operatinga#  也随之b#  的网络c#  ▁delightfuld#  生肖鸡e# annif#  截止目前g#  聘用合同h# 是好i# ▁Pegj#  但是我们k# 张晓l#  ▁Credm#  ▁arbitrationn# yneso#  本土确诊病例p# 陵园q#  进行调查r#  ▁clarifys# 全美t# 过半u#  县人民法院v#  ▁revisionw# 这所x# ▁MVPy#  ▁Hashz# UTC{#  投资有限公司|#  应当在}#  目前正在~#  道路交通事故#  第二百#  ▁academy# 义工# 触及#  ▁Takes# 奥林#  都可能# 习作#  北京师范大学#  精准扶贫# utf#  为抓手#  这条路#  ▁Huff#  ▁businessman#  ▁discourse#  ▁Spark# 星人# 一词#  的希望# 走走# ▁cit# ▁dra#  fashion#  ▁monitored#  直接影响#  ▁educator# Brad#  履职尽责# chard# 渡过#  ▁quantities#  会因为#  的一般# 三世#  并表示# Birth#  ▁assurance# 减缓#  ▁JavaScript#  ▁negotiating# 义勇#  ▁Windsor# 太好# 与时# rint# 井冈#  签约仪式# 遗留#  党的领导#  ▁abused#  ▁impair#  ▁incumb# Past# ande#  ▁stub# 对党#  ▁hammer#  ▁invasive#  ▁rainfall# 德云#  贫困地区# omics# 受了#  或许是# 发来# 贵在# ön# 而定# 打仗# Tri#  ▁Zion#  ▁pursued# 给它#  ▁ample#  思想上#  等重要# 约合# umed# 满天#  ▁ruins#  operation# 快感#  利用率#  ▁reimb# 募集# 特困#  ▁Flex#  为推动# Within#  annotation# 小张# ,…# 兢兢#  负面情绪# ▁Vas# 中兴#  上大学# 郭勒# 飞速#  learning# 并存# wat#  可以有效#  ▁overhaul#  ▁Unity# 手艺# 打车#  中央八项规定#  ▁mins#  的大学#  ▁delegates# ▁DM# 却能# 条纹# 黄连#  如果要#  系统中#  quarter# ▁AZ# 手部# iq# 归于#  今天要#  ▁chick# 高点# shaw#  申报表# 反省$ 胡同$  ▁Diagn$  ▁slated$ 追回$  年公开招聘$ HH$ XL$ 课间$  ▁sunset $ 闷热 $  ▁livelihood $  新增病例 $  ▁tract $ apa$ eco$  Privacy$  ▁Rehab$ 农夫$  ▁tackles$  ▁Epic$  ▁Bills$ Bus$ inders$  ▁migrant$ ▁SB$ ▁TE$  人员伤亡$ bestos$  ▁nonetheless$ 并能$ Altern$  Reading $  ▁dictionary!$ 大好"$ Ast#$  停下来$$  是无法%$ pler&$ ▁Kor'$ ")]($  Western)$  中西部*$  资料来源+$ yz,$  ▁Male-$  ▁repay.$ 柏林/$ 调入0$ 耳鼻1$ 迎合2$ 年报3$  ▁fixture4$ ві5$ 不曾6$  ▁Bulgaria7$  秋冬季8$ 看守9$ umat:$  ▁punish;$  的水平<$ iotic=$ 监考>$ 偏爱?$  挺身而出@$ 疫病A$ 嵌入B$ 干旱C$ 中西D$ 新课E$ 帮手F$  生肖牛G$ тыH$  举报人I$  ▁feedsJ$ 命理K$  司法机关L$ lbM$ 有成N$ 情谊O$  中国古代P$  ▁modificationsQ$ 济源R$ 光了S$  从根本上T$  ▁guidingU$ 洗完V$ 而下W$  但随着X$  条第一款Y$  管理处Z$  精美的[$ 不讲\$ 天价]$ 输了^$  这一幕_$  实施了`$ Inva$  这种方法b$ dkc$  的业务d$  也是可以e$ 擦亮f$  ▁acidsg$ ▁Igh$  ▁redei$ 得天j$ 漯河k$ ▁Cirl$ Sorrym$ 千瓦n$ ▁Hodo$ 编排p$ ^^q$ YPEr$ ▁Kws$ Timest$  ▁Fuelu$ jgv$  ▁Employeesw$ 雍正x$  家中的y$  调查中z$  村集体{$  ▁encryption|$  ▁consisted}$  ▁Isle~$  ▁quicker$ 早前$  ▁ambul$ 小了$ 无辜$ atisf$ 全靠$ 赛场$ trl$ ▁Cay$ 并由$ flower$ 球场$ 弹力$ ▁McN$  常态化疫情防控$ 只做$  ▁stip$ CRE$  ▁accusations$ Arts$ props$  买卖合同$ 电阻$  ▁Overview$ 团建$ 不弃$  猪肉价格$ mobile$  不懂得$  要严格$  工商管理$ 卖的$  ▁voor$  ▁Archae$ 遗症$ 漂浮$  大力推进$ 他将$ 酿造$  ▁adolesc$ 之首$  又开始$ 自知$  自然村$  国家机关$  ▁Tales$ 劲儿$ år$  acceptable$ Market$ 给力$  对他的$  ▁Funds$ 传人$ 圈内$ 耗费$ 先将$ 巴克$ 特尔$ 电饭$  而成的$  思想的$ 件套$ 时针$  不相信$  一把手$  喜马拉$  ▁assumption$  ▁Harold$  最小的$ mini$ 五味$ 外墙$ Fall$ scape$ 杀了$  ▁Civic$  的造型$  采用的是$ Dialog$ 新京$  ▁Bernie$ 绳子$  ▁labeled$  ▁grounded$ 军工$ 危房$  涵盖了$  阔腿裤$  隆重举行$ 军方$  ▁Springfield$  ▁chess$ 碱性$  你怎么看$ 算计$  ographies$  ▁fulfilling$ 翻天$ 多加$ 派对$  ▁Geoff$  给对方$  一步一步$ 笼罩$ gust$ publ$ 舆情$  ▁wool$ 一卫$ 风向$ 单项$ 无须$  的微信$ 本部$ itic$ heres$  举个例子$ adia$ ▁Viv$  ▁slaves$ 震动$  就直接$ 心率$  ▁Vine$ 供血$  ▁senator$ 近百$  一时的%  ▁Dreams% anyahu%  ▁firearms% 容貌% 焯水%  我个人%  绝对不会%  的提升% 筹集 %  的主人 % 杨梅 %  就是这个 % onge % 险情%  时间节点% 减脂%  ▁leagues% 金句% 驻地%  打造出% Baby%  工业互联网%  ▁Sheet%  ▁Electrical%  还可能% пу%  总监制% ▁Xi%  如果你有% 义诊%  的创新%  ▁ally %  给自己的!% 之感"%  ▁riot#% 前端$% VERS%% insky&%  和田玉'%  ▁Pakistani(%  ▁practiced)%  ▁scheduling*%  研究成果+%  ▁Waste,% 课上-% 迟早.%  工作部/% 拟聘0%  交通枢纽1%  协议书2%  帮助下3% lated4%  undreds5%  有小雨6% 山下7% 查封8% вер9%  学科类:%  ▁anxious;%  ▁Accounting<%  招投标=%  测体温>% otech?% ETA@%  心肌梗A%  岁左右B%  reatmentC%  ▁BaldwinD% 也已E% 寄宿F% 原型G% ▁vomH%  ClimateI%  年春节J%  李嘉诚K%  ▁OutlookL%  ▁VoteM% 剪纸N% 梧州O%  帮助大家P%  resourcesQ%  表现在R%  的道理S% npT%  承载着U% ppeV% 肝气W%  ▁SequX% 伤痛Y% 锤炼Z%  身体不适[%  进行处理\% 年和]% 本性^% 串串_% 室友`%  喜马拉雅a%  ▁modificationb% trainc% ▁dwd% 热气e% 宽阔f% 摇篮g% ▁Wikh% arsityi%  ▁spacecraftj%  我们把k%  ▁Allegl% 潮州m%  ▁lipsn%  ▁flieso%  ▁passivep% 的真q%  ectionsr% 野菜s%  ▁Fortunatelyt% 颇为u%  逻辑思维v%  ▁congregationw% 将它x% 横幅y%  下面是z% Earth{% HN|% 中天}%  ▁bride~% 熟人% 笔墨% 延时%  ▁downs%  ▁cardiovascular% 文稿% ▁Lun%  ▁Someone% 劳模% pub%  ▁denial% ▁mol%  ▁inappropriate% Amy% 不正% 行星% 共用%  ▁carn% 感官%  副组长%  ▁scram% 白领%  来形容%  Netflix%  温柔的%  本以为% emer%  ▁года%  绿色的%  ▁reservations% 就别%  ▁Eventually%  ▁resem% 操心%  情节严重的%  的行动% 倍感% 温润% 滴水%  ▁disable%  ▁empathy%  ▁Dana%  发布之日起%  ▁fluct%  ▁Tale% ealous% 触动% Auth% ▁Bey% 宽广% ksh% 切身% 永生%  星期四%  ▁spatial% 微型%  示范基地%  ▁Alert%  ▁mainland% 除草%  每个孩子%  行政机关% 港元%  Internet%  ▁halls%  期待的% 白玉% bucks%  ▁Ruby%  ▁glorious%  寄生虫%  ▁plumbing%  每天坚持%  ▁Dinner%  届时将会% 乔治% 党务%  大本营%  ▁Countries%  ▁chlor%  ▁dwell%  ▁curated% 宽带%  传承红色基因% Jay%  ▁cort%  ▁supervisor%  ▁Jacksonville%  ▁Brid%  ▁Transform% 集锦%  results% 脱口%  基本工资% 放纵% 阿根%  ▁refurb% efe%  相关问题%  在这种情况下%  workers% 石桥%  相关阅读% INGS%  监管部门% 专硕% >>>>% 收货% 烧开% 寺院% 纵向%  在不同% 国六% 蓬松%  ▁spawn%  ▁intuitive%  儿子的%  第二针%  级景区% Connor% 勘查% 徐某% 架子%  ▁Knox% Jordan%  ▁seafood%  走过的% oppy% 耗时%  新产品&  ▁Peak& 经期&  ▁Guatem&  ▁fireplace&  ▁backpack&  ▁severity& 铜仁& 占道&  人脸识别 & 天主 &  救援队 &  是什么呢 &  精准施策 &  ▁Tunis&  ▁traject& 交响& ▁Liu&  副厅长&  建设高校&  ▁homosex&  Management& 忧愁& 送来& 云烟&  努力学习&  ▁adopting& Taylor& 上都&  也正是& 才艺& 店主& TVB &  ▁Shen!&  ▁encoding"& вы#& hesive$& ít%&  ▁Abstract&& Islam'& 尾声(& 巧合)& 换取*& 籍贯+&  ▁Transit,&  星期三-& 演义.&  ographics/&  官僚主义0&  ▁Airways1&  ▁pronounced2&  ▁envision3& 管用4& 卖房5&  怎么回事6& 丢弃7& 死去8& assis9&  ▁spanning:& 存栏;& 中交<&  ▁notebook=&  ▁endorsement>& 滑坡?& 金龙@&  一旦发生A& 褐色B& 当月C& ▁DinD&  ▁BreakfastE&  ▁answeringF& 呼唤G&  上下游H& 干线I& 越远J&  虽然在K& 事的L&  ▁meshM&  ▁nightmareN& UVO&  一经查实P& 旺财Q& 指纹R& 职级S& ulseT& 细雨U& ZEV& phiW&  第一大X&  ▁anglesY&  里面有Z& 游园[& 鲜肉\&  路小学]& 暂未^&  等什么_&  最新通知`& 灯笼a& 蚕丝b& 吓得c& 沮丧d&  竟然是e&  工程专业f&  ▁overlyg&  Startingh& 魔术i&  不文明j& 倒在k& 汛期l& edom& 日是n& 转业o& Mailp& ramesq&  скогоr&  ▁Kindles& 长发t& 之手u& 悲痛v& 版块w& 赛跑x&  报告显示y& 枝条z& ucked{&  ▁bombing|&  助学金}&  的不断~&  您的权益& ITION& daily& 东盟&  ▁polling&  ▁Opportunities&  个专业&  ▁Either&  Complete&  ▁unforgettable& 屋里&  精神文明&  中央纪委国家监委&  ▁inherited& 张艺& ▁az&  一个好的& 不耐& 别克& 永康& 萃取& Dark&  ▁sunlight& 加码& 爱和&  视为自动放弃&  万左右&  早知道&  ▁freeze&  必不可少&  实体经济&  等地的&  ▁imperial&  始终把&  ▁Animation&  坚持不懈& ▁bei&  个人简历& 接单&  ▁bypass&  第五届& orde& 生计&  领导者&  ▁Strip&  ▁atmospheric& 深沉&  不理解& hon&  这样的情况&  ▁Prosec& 积雪&  上了一& 掩饰&  ▁Wagner& strom& + $新型冠状病毒肺炎确诊病例&  ▁entrepreneurial& 长出&  让人觉得& 安顺& 白血&  ▁Courts& 不设&  ▁Treaty& 醒了& ▁BJP&  ▁recognizing& 专车& 沧海&  的物品&  ▁Needs&  ▁McCain& 源地&  身体素质&  ▁brains&  ▁Promise& GW& 任正&  卡是不&  verages&  是否需要& 卒中& 小黑&  ▁Membership& 猥亵&  ▁hackers&  所带来的& 享用& ▁vet& "_& 分餐& follow&  一般为&  ▁exemption& 较快&  ▁allev& 钱塘& ▁și&  对这些&  在疫情期间& imer&  城市更新& Option&  ▁className& 直呼&  一点一点& 最最&  的资金& ▁oun& ▁Bod&  ▁Tomorrow&  我当时&  清洗干净& 不老&  微信关注博野在线&  ▁Spotlight&  ▁exploit& Kit&  ▁legislators& 寝室&  ▁Abbott&  ▁reiter&  ▁exceeded& 绵绵&  语文下册& 信网& irteen&  有效期内& 赞叹' 骑着' 官司'  ▁dancer'  ▁Wash' ▁Git'  ▁souls'  有价值' ▁Gir'  反而是 ' Values ' 靓丽 '  ▁Goes ' olics '  ▁suburban'  虚假信息' zm' Roy'  ▁Devon'  十分钟'  protected' 六腑'  旁边的'  病虫害' ▁Agg'  需谨慎' ▁dir'  邯郸市'  这个世界上' 又叫'  Elizabeth' 充斥'  ▁avid ' 脊椎!'  使用时间"'  ▁Davies#'  ▁Tonight$'  潜意识%'  ▁tide&'  ▁props''  ▁TikTok(' 净水)' sym*' 校车+'  才开始,' 日式-' ▁Cec.' 炒股/' 制冷0' 呵呵1' 孕妈2'  和国际3' 一粒4'  义务教育学校5'  为方便6' 都已7' 人防8' 可可9'  ▁adorable:' 彼得;' fj<' ▁И='  对象的>' 坍塌?'  阅读理解@'  主办的A'  ▁NewmanB' 译文C' ▁EMD'  ▁lightlyE' 倒了F'  出了问题G'  BesidesH' StreetI'  排名第一J'  遇到困难K'  市场监管总局L'  统一到M' 节选N'  大讲堂O'  公益活动P' 阳虚Q'  让对方R'  中国大陆S'  使我们T'  ▁ActressU' 残联V' 用时W'  ▁LeslieX' 墙体Y'  海内外Z' 阳江['  ▁rifle\' 我军]'  连续三年^' 周炎_' 树下`'  ▁subjecteda' 太后b' 烧热c'  ▁Sebastiand'  避免了e' aspxf' ▁Cafg'  ▁Sharesh'  ▁selectionsi' 呆了j' 易地k' 色情l'  主要负责m' 代会n' 长腿o'  特别适合p'  esterolq' cookr'  Selecteds' Incret'  ▁imaginu' 菌群v' ▁Heyw'  ▁pullsx'  自然灾害y'  ▁tierz'  ▁Bollywood{' 阻塞|' zq}'  是人类~'  理事会' 抽签'  ▁Beverly' 明智'  ▁morph' 那我'  很多同学'  内防反弹' Later'  perature'  ▁Homeland' 愚蠢' Saint' ▁AA'  详细信息'  高速铁路'  ▁collector'  onsense'  ▁traces' 揭露'  餐饮服务'  房屋出租'  question' ▁BT' inel' equals'  ▁tones'  essages'  生活和'  检测机构' 场次'  不得已'  健康和' 城西' 福清'  Johnson'  ▁Giving' Ox' 七分'  星期二' 格栅' 财气'  这只是'  ▁Declaration'  ▁remedy' 完备' ▁ART'  ▁Asset'  ▁priests'  ▁battling'  平日里'  ▁hints' 来来'  打造了'  neither' 晶体' fen' tesy' ▁pse'  条的规定'  ▁chill'  scription'  ▁discoveries'  下一次'  为您提供'  都有自己的'  ▁Burke'  ▁markers' learn'  ▁Gandhi'  进行分析'  ▁Outstanding' 金币' ▁ion'  ▁downloads' 废水'  不至于'  学位证书' 相交' 红外' Trust'  总会有'  ▁congest'  的含量'  精彩内容'  ▁economically'  是否可以'  ▁philos'  埃里克' 迎难' 动辄' nar' 姻缘' ▁Rib'  ▁bloody'  ▁activism' 栋楼' neum'  轻松松'  没有硝烟的' Si' 相依'  ▁lush' ▁Iv'  博物院' ▁Wid'  国际学校' ANA'  ▁Surface'  ▁requesting' 在院'  ▁particle' 味甘'  事物的'  工作年限' amboo'  ▁broadcasting'  爱奇艺' 回访' 阴谋' rehend' 疯了'  直接接触'  ↓↓　'  采取措施'  或者有'  ▁imperative' 厌恶' 店镇' 透彻( inz(  所需要的( SET( anchez( iona(  ▁publicity(  ▁comprising( 之势(  ▁lion (  ▁Literary ( 管教 (  邀请了 ( 无用 ( 文凭(  ▁Reporting(  Created( ▁GD( ▁SAP( 变大( ▁Rw( 商报(  ▁pivotal( 尾部( Rule( owitz(  ▁Them(  ▁Goals( 壮阔(  怎样才能(  ▁curb( 经的( {　 ( 申诉!( 更佳"(  矿泉水#( 人来$( 王菲%(  ructuring&(  ▁catalogue'( 称其((  走访慰问)( ( !党的十九届五中全会精神*( 攀比+(  ▁Hayes,( 五谷-( 飞天.(  iplayer/( 危重0( 沙尘1( ▁Au2( 时会3( 特警4( 未被5( 网签6(  ▁singular7( 借着8( Asia9(  ▁Blvd:( ▁lam;( 梯队<( 投影=(  Progress>(  称得上?(  ▁underwater@(  抗疫一线A(  ▁AcceptB(  个名额C(  ▁cheapestD(  ▁stealingE(  教学质量F(  ▁GustG( cornH( 伤了I( ▁DyJ(  ▁brandedK(  inationalL( 稳中M(  使用方法N( bolaO(  ▁soreP( 的字Q( 赢了R( 之于S(  阿根廷T(  ▁ballotsU(  ▁realitiesV(  生肖虎W( handedX( 活出Y(  ▁spokeswomanZ( 玄关[(  实名制\(  我的朋友]( iste^( lain_(  ▁deserved`( ▁rena(  即可领取b( Lisac(  每天吃d(  切实加强e(  ▁Citizenf(  ▁Sonsg(  ▁suscepth( 文言i( 搞好j(  ▁encouragementk( 出家l(  在当时m(  ▁grapn(  ▁dragono(  ▁governedp(  颁发的q(  ▁Tanzaniar(  次全会s( 巴士t(  陌生的u( 特约v( 蜗牛w(  兄弟姐妹x(  ▁worryingy( eousz(  优秀的人{(  ▁albeit|(  ▁partnering}( 一招~( 犯错(  市场份额(  第二年(  隶属于( apply( 慈禧(  ▁naval(  ▁Vander( 六保(  ▁futures( 土豪( 制备(  ▁Mond( 惆怅(  Release(  ▁Edmonton(  大富大贵(  Christmas(  ▁unpredict( 高处(  八年级( 系的( 莲藕(  非洲猪瘟( 驳回(  淘口令( xl(  ▁exquis(  ▁Customs( 7 0微信关注博野在线微信关注博野在线( 体谅(  Language( 富强(  脱贫攻坚成果( 确有(  ▁Cork( terror(  ▁Marina( Neil(  Something(  ▁Remote(  electric( 英烈(  ▁wilderness( 火速(  ▁append(  凭借着( 分发(  开窗通风( inet(  全产业链( teenth(  ▁Vietnamese( 送出(  ▁Elder(  检验机构(  ▁bleeding( 石门(  ▁competed( 你得( 有啥(  ▁masterpiece( 他家(  ▁internship( QS( 大体(  及时向( 早睡(  安全防护(  ▁Twenty(  ▁Organisation( 定西( 祖父(  有多么(  可能会出现(  核心技术(  standard( 穿透( urry(  ▁torque(  ▁misleading( 致的( 根治( 懈怠(  ▁surveyed( 杀虫(  土地上(  李老师(  ▁compilation(  轻轻松松(  Financial( 鞭炮(  ▁deliberately( 独生(  大多是(  比例的( byte(  分析了( 如在( 滤镜(  等现象( 绿城( floor(  招聘程序( 休克( 悲哀( 拉近(  不起来(  ▁pave( 皆可( ostics( 纷呈(  提到了( 酥脆( 讲到(  ▁Exhib( 质检(  补助金( CPU(  发展和改革(  优惠活动(  ▁clue(  ▁snack( 化石( Spot(  ▁pear(  ▁Affordable(  让更多的人(  免疫功能( ▁ND)  通报了) 他能) 考区)  七年级)  ▁payroll) 我行) 主事)  公安交管)  ▁Essex )  ▁Panthers )  全力推进 )  ▁Patricia )  ▁velocity ) 造血)  照顾好)  财务管理)  ▁Fraser) 饵料) biz) 本是) anqu)  薪酬待遇)  ▁emergence) 领跑)  当地政府) 水墨)  ▁Raiders)  全球化)  ▁feather) 三省) ▁AWS) xh ) 起拍!)  任何事情") 则为#)  ▁crisp$) 多起%)  能吃苦耐劳&) 工农')  微信小程序() alin)) 成渝*) 球菌+) 使馆,) 多见-)  ▁Mood.)  ▁assessing/) 不死0) 大都1) got2) Prop3) ▁GRE4)  ▁Interactive5) 太久6) 我所7)  整治行动8)  ▁Shannon9)  ▁calculations:)  三个方面;)  ▁bureaucr<)  按实际=) 男神>) 百计?) 问到@) JewA) ️　B) hurstC)  有条件D) 百态E) 加价F) alineG) ▁EcuH) 主机I)  的成长J)  项目和K)  ▁FreemanL) 烟气M) 撒上N) oboO) asiveP)  ▁complimentaryQ)  尤其在R) shameS) 月休T) 可贵U) ▁CocV)  的结构W) änX)  ▁reasoningY)  ▁festiveZ)  更健康[)  抑制剂\)  ▁scam])  会展中心^)  ▁analytical_)  ▁burial`)  ▁distancesa)  ▁positioningb) 满是c) ynthd)  任何费用e)  ▁Hindif)  ▁Katherineg) Mrsh) ▁shai) 摄氏j)  ▁casinosk) 咽痛l) 那段m)  ▁silkn)  ▁bilateralo)  controllerp)  岁月的q) ▁btnr) liants)  ▁crediblet)  ▁Wideu) 乱放v) 哥们w)  做家务x)  版权属于原作者y) rossez)  ▁magnet{)  ▁appetite|)  拍摄的})  ▁Hyper~)  的观点)  吃东西)  ▁cables)  快速路)  我不想)  保持良好的)  ▁Nottingham) Inf)  中国银行)  工作中的)  ▁narratives)  国家卫生)  ▁Newsp) GRE) % 微信公众号微信公众号) bst)  ▁learns) 白皙)  ▁divor)  ▁Tong) thirds) XXXX)  ▁developmental) 高昂) 国药)  精气神)  ▁Kurd)  ▁Whole)  ▁melod)  ▁blade)  事实上事实上)  ▁mattress) 乐团) Front)  武汉加油)  时间长了)  ▁tasty) 佳绩) 太少) 画像) 相撞)  阳光下)  ▁Pastor) 尔德) 未见)  weather)  orgetown) 哈弗) 内幕) 岩石)  孙悟空) ismo)  怎么才能)  ▁darker) 乙醇)  和能力) ecca) Sound)  黄色的) tags)  ▁Been) 指战) 聚合)  图片来源网络) 天河) 观景)  陷入了)  ▁epit) 村居) ▁MLS) 和林)  ▁partition) Root)  遇到问题) 借用) 从零)  ▁Loan)  ▁accounted)  层次感)  显示屏)  了一家)  人到中年)  ▁Stockholm) 黑的)  ▁committing)  ▁Quotes)  ▁empowering) 有新) 王文)  极大地) omal) Track)  生产力)  阿克苏)  ▁Written)  Summary) Math) ▁Pep)  ▁Triple)  Americans)  ▁Menu)  ▁Manit) 伤寒)  上市的) 破案)  第一阶段)  ▁Philippine) ▁fog)  自然资源和规划局) 离别)  积累了) 上牌)  ▁Simmons)  ▁tackling)  一个孩子) 晾晒)  主流媒体)  ▁Packers)  情形之一的)  累计死亡病例)  凝心聚力)  城市发展) 福祉) 单科) 把好)  死亡的) 永定)  美联储)  的时间和)  教育引导*  ▁Liqu*  欢迎在* 年检* 窍门*  秋天的*  ▁therapist* 站的* 造车* 地貌 * 顶着 * 监视 * tering * Ut * 领馆*  都会被* Prime*  ▁Testament*  辛辛苦* antes*  ▁Ages*  案例分析* stead*  ▁ceramic*  ▁resolutions*  宣传工作* iatry* 调价* erala*  开放的* 是将* oxic*  ▁Figure * 倍的!*  自我保护"*  ▁YORK#* UID$* ▁AFP%*  ▁Extension&* money'*  ▁Hands(* 由市)* Biden**  quisition+*  一些小,* 琐事-*  如发现.*  ▁lasts/*  ▁wishing0*  心里的1*  ussions2* ▁Gri3* 这波4*  科技成果5* said6*  ▁Visitors7*  ▁Awareness8* 撑起9* 联谊:* 尿素;* 民居<*  有意义=* 软化>*  生动的?* 不愧@*  能得到A*  fficiencyB* 美洲C* 随风D*  的保证E*  管理服务F* 眼角G*  ▁enactedH*  延长至I*  有话说J* itoryK* showerL* CYM* 班线N* 任选O* 天道P* 者可Q*  遭到了R* utaS*  电子书T*  ▁COMPU*  ▁coolerV*  ▁bloggerW*  consoleX*  你真的Y*  ropractZ*  ▁Johannes[* 刘邦\* odore]*  pathetic^*  settings_* 一旁`*  等平台a*  辛辛苦苦b* 做着c* 加的d* 开场e* 螺丝f*  ▁respondsg*  沉浸在h* openedi*  ▁enlightj*  攀枝花k*  ▁Maritimel*  ▁hurricanem* 进京n* talo* 真切p*  精神状态q* TYPEr* ipegs*  ▁seminarst*  ▁neverthelessu* 胆囊v* 纵观w* bugx*  ▁Shiny*  ▁Sallyz* 动荡{* 猛烈|*  约定的}* 一拳~*  сько* ▁Gur* FB*  读书会* 开好* 石狮*  Install* 绿叶* 疙瘩* 罪犯* dal*  Patrick* 赞扬*  ▁Celebrity*  ▁paras*  ▁Birds*  ▁Diseases* 史诗* 简要*  上网课* ▁cov*  lymouth*  ▁leaked* 孙女*  ▁scra*  selected*  总体规划* 儿媳* 沦为*  肾功能* 当了* 退缩*  有效控制*  ▁concentrations* fts* PAR*  就是一种*  ▁Minutes*  ▁inquiries* junct* 杏花* 空港*  我们自己* 卡的*  ▁Brock* 就如*  很重要的* 街拍*  ▁Litigation*  大流行* ▁REL* orian*  区人民政府*  工程质量*  henghuo*  ▁thrill* 这四* EMS* 确权*  不要把*  黑社会*  ▁pixels* 不调* 但有* 使我*  社会科学* 优抚*  日以后出生*  告诉孩子*  非法集资* 公社*  ▁naming*  ▁hottest*  反而会*  以下几点*  ▁Sets*  赚钱的*  三十年*  不是一个* 王家* 眩晕* 拍下* ▁APP*  ▁galaxy*  有序开展*  ▁monsters* 丝瓜*  Background* ▁ISS*  ▁Parade*  ▁bitcoin*  ▁contacting* 土拍* 地名* 微商*  的控制* 伊斯* 用完*  ▁unified* 以西*  ▁testament* 游击*  多小时* hidden*  所以他们*  一人一*  ▁canal* 冷淡*  是属于* 出题* 慢行*  ▁hatred*  为代表的* 德基* 美了* 被封*  户籍地* lash* 李世*  走出去*  如有问题*  东方红*  ▁hygiene* 牛羊*  ighthouse* rylic* 虚伪*  如侵权*  ▁Finding* 老街+ 兼任+  公立医院+ 坠落+ 驶入+  ▁mirrors+ 厅级+  但是我+ xu+ 探访 + iii +  ▁triv + 桌椅 +  保质期 +  就是个+ 脚印+ 静好+  的单位+ itism+ 人会+ 天眼+  ▁refusing+ 通便+  中表示+  ▁Elvis+  高颜值+ 重担+ 前排+  ▁rescued+  实力的+  ▁stressful+  给人的+ 想说 +  美容院!+ 调取"+  戴好口罩#+  绩效考核$+  ximately%+ 张掖&+  等特殊'+  ▁perf(+  的那个)+  车管所*+  ▁marketers++ 有氧,+  完全不-+  ▁showcases.+ gold/+  enabled0+ 白雪1+  转载自2+  也要注意3+  节日期间4+ 洗浴5+ Cas6+  相信大家7+  进一步增强8+  ▁Joyce9+ submit:+ UPDATE;+  ▁Highland<+ 都对=+  ▁Solo>+  建设中?+  打游戏@+ 办税A+  也有了B+  代表作C+ 毛钱D+  难得的E+ 曾因F+ ▁SaxG+  设施设备H+  ▁curtainI+ 迷路J+ oshopK+  ▁THATL+ 风流M+ ▁EyN+  延伸阅读O+  心理疏导P+ 冬虫Q+  可以不R+ 祝您S+  技能培训T+  ▁fairyU+  广泛的V+  forcingW+  ▁destroyingX+  开展党史学习教育Y+ 李晓Z+ ▁RBI[+ 肝病\+ tri]+ 静心^+ ▁PDT_+  ▁tasting`+  ▁Formata+ ▁fueb+ 练兵c+ 薄的d+  ▁attachmente+ があf+ 等同g+  与众不同h+  ▁durabilityi+  ▁predominantlyj+  方向的k+ 心电l+  ▁signaturesm+  ▁collectorsn+ 危化o+ Studiop+  ▁emissionq+ womenr+ itchess+  设置的t+ 换气u+ 北门v+ 婚前w+  modulesx+ 走动y+  地级市z+  眼前一{+  说白了|+ 临朐}+ 受众~+ 栖息+  ▁discovers+ 开店+  有风险+  这个方法+ 竞得+  怎么可能+  ▁collaborated+ 的血+ 胃酸+  尤为重要+ utsch+  ▁determines+ 科协+  强有力的+ 外媒+  人们在+  ▁senses+ 各异+  ▁hostile+ 动感+  ▁condemned+  三年的+  ▁malicious+ 纽带+ 交办+  扫码查看+  市委宣传部+  ▁likewise+  手机的+ 有线+  经历的+  美女美女+ ℃~+  ▁tracked+ 临港+ 冷战+ 看护+ 邀您+ bright+ 占总+  所以他+  ▁damp+ 贪婪+  药监局+ 二本+  下心来+  下午茶+  要坚决+  几个小时+ ZKJ+ EDIT+  教育工作+ 多说+  可以做+ 半径+ 山湖+  交朋友+  检查和+ need+  ▁bounce+ 寒风+  艰苦奋斗+  中国疾控中心+  ▁shareholder+  ▁Malcolm+ 绷紧+ exc+ 师的+  ▁Sterling+ Gar+ lr+ 市容+ 迈向+ 奖章+  好习惯+  ▁slim+ 昵称+ 涡轮+  政治任务+ 六盘+ 能给+  必备的+  ▁bees+  ▁chim+  山景区+ ▁ol+ Rod+  到自己+ ilee+  建议你+ buffer+  ▁specializing+ USB+  出现过+  新冠病毒的+  enstein+  帮孩子+ Andy+  ▁cardiac+ 出任+ 未有+ eri+  抓落实+ Cover+  改革的+  Culture+  ▁speeches+ hess+ 盟友+  ▁Leicester+ 大明+ 秦岭+  应有尽+  火神山+  products+  签订的+  ▁Item+  卡积分绿卡+  ▁revival+ ▁"$+  也容易+  ▁gatherings+  Anthony+ 越走+  身边人+ umers+  ▁recruited+ 心胸+  ▁configure, Jones,  ▁Cases, 为己,  ▁fragile,  全体会议, PRE,  ▁warfare, 孙某, mant , . '全面建设社会主义现代化国家 ,  ▁incon ,  ▁Chevrolet , 美貌 , 追问, 夏草, ▁mig, ▁Few, 强硬,  ▁browsers, 巡航,  启动了, orthy,  ▁Clara, 山川,  ▁jurisdict,  ▁Alban, ALS, 学着,  最美好的, 语句,  经营性, 杀灭 ,  ▁seasoned!, 用笔", 阵营#, lake$, 深造%,  任职资格&,  ▁optimism',  ▁Parkinson(, 节后), 致远*,  ▁extraction+,  老一辈,, lane-, 牧场., ном/, Jean0, ossip1, 扫除2,  工业企业3,  ▁newcom4, 他与5,  营养素6,  源自网络7,  ▁Strat8,  ▁geographical9, 脚踝:,  科技股份有限公司;,  ▁incorporates<,  ▁reconstruction=, 分包>, 索赔?,  现在很多@, NightA, 账单B,  城市群C,  是自己的D, 疗愈E,  点击文末F, churchG,  总医院H,  若涉及I, 男的J, 肝火K,  ▁NewportL,  爱岗敬业M,  ▁caredN,  transformO, 汉堡P, 不净Q, 放出R,  这个小S, wicT,  创作的U,  积极参加V, 解难W,  解决的X,  服务等Y,  国家发展改革委Z, ajo[, 分寸\, 当心],  ▁framed^, 他为_, Ly`,  ▁colonya, Anneb,  ▁routinelyc, osod, LNGe,  ▁Randomf, 乎乎g,  YouTubeh,  支持下i, 直流j,  ▁Robertsonk,  ▁expeditionl,  ▁interpretedm, >_n,  ▁noticeso, ▁McDp,  共产主义q,  可分为r,  ▁reducts,  ▁notorioust, фиu, ▁Dwv,  规划师w, ▁Levx, sayy,  女演员z,  爷爷奶奶{, 更长|, 蓝光},  ▁diamonds~, 普拉, 生下,  ▁Launches,  ▁prevalent, 拍到, OCK,  从这里, 提早,  ▁facebook,  这一个, XY, 烟雾, 升至, 在市,  前两天,  的制作, ioni,  ▁staged, Simon, 误差, ulk, 鼠年, 正向, 象棋,  ▁Benedict, 交费,  starter,  的实力, 硬笔,  ▁Kenny, 嘱托, 总要,  ▁Pull,  给你带来,  青年教师,  ▁threatens, 过失, Remove,  ▁Areas,  ▁Trophy, 病患, ogene,  骨干教师,  ▁gateway, 押题, 稿酬, grown, 千方, 透支,  ▁Breaking,  ▁bang,  ▁fart, hai,  表现得,  ▁Practices, ▁lac,  ▁valuation, 入命, eno, ▁Nak,  ▁spear, 旺运,  浦东新区, 心系, FROM, ▁Laf, erset, ▁VAT, 平遥,  扫二维码,  ▁INTER,  违法违规行为, 谴责, 起跑, . '▁▁▁▁▁▁▁▁▁▁▁▁▁,  ▁integrating, 是把,  ▁activate,  ▁terrifying,  疫情输入,  ▁sanctuary, 减小, RF,  便宜的, Winter,  更多精彩内容, 值为, 道口, ophone, 誓词, alore,  部位的, wave, 风波, Hope,  放在第一位, ( !如有侵权请联系我们删除,  ▁famed, Roll,  ▁Payment,  保留了, 印证, 所为, ▁RB, ▁Buc,  ▁Libraries, 血的,  是一部, 温室, 迎春,  新增无症状感染者, 普陀, 该国, 别让, sta,  都是非常, 上过,  几个人,  路公交车,  Database, 水浒,  ▁immersive, Kn, 清查,  近两年,  啊啊啊啊,  ▁Wesley, ▁{{,  快递员, visors-  ▁Flying- 穿了-  ▁symbolic- 人就- 脑洞-  ▁Crawford-  的一线-  动脉硬化-  ▁segreg -  ▁Compliance - 跳绳 - iffany -  农商银行 - route- 日系-  个月后-  ▁insult- 望远- 碾压-  进行报名-  ▁northeast- 两千- nesty- 绿植-  中央政治局-  ▁alloc- 泡茶- nation-  海底捞-  内部的-  实现的- 公款 -  代表了!-  这个名字"- rences#- swick$-  ▁Rout%- ées&-  和政府'- TikTok(- 水体)-  ▁mosqu*- 作则+-  智慧的,-  ▁Greene-- 骨盆.-  ▁herbs/-  比较好的0-  ▁doubts1-  越来越多的人2-  未来发展3- 巴掌4-  ▁lazy5-  ▁monuments6- atri7-  ▁catches8- 二百9- 接续:- át;- 上是<-  ▁Laur=-  创造性>-  ▁quir?- 廉价@- smartA- ClinB-  有多种C- 茶园D-  冬虫夏草E- ▁akaF-  间盘突出G-  ▁GiantH-  ▁EarlierI-  ▁suburbsJ- 局的K-  上来看L- 道题M- AnnaN- ▁TapO-  ▁averagedP-  以内的Q-  汽车的R- ріS- 强弱T- 拉长U- seatV- 刊发W-  ▁HerbertX-  ▁lightningY- 古法Z- 天王[- 家境\- 且有]- Login^-  随着时间的_-  演唱会`-  ▁antiquea- 篮子b-  在面对c-  自由的d- anase-  招生专业f- ienneg- 武冈h- MODi-  ▁Discussionj-  ▁Certificationk- Linuxl- 折射m- 整容n-  同舟共济o-  熟练使用p-  现役军人q-  ▁mentoringr-  签订了s-  速度快t-  可以考虑u-  岁的孩子v-  ▁Nokiaw- 定格x-  ▁coordinatesy- 绞痛z- asser{-  ▁Created|-  城街道}-  ▁uncons~- iPad-  ▁advisors- 延误- 香辣-  巡视员-  把我们-  有意向- 秋招-  ▁podium- ▁př-  感染的-  第三代-  进校园- )))-  开放时- alez- 化了-  ▁Computing- Nob- 我叫- 江东-  Jonathan-  ▁Kardash- 量和-  存在问题- birth- 疲倦-  Episode-  Statement-  ▁Achievement- 只狼- 察看-  就是命令-  ▁Albany- 而无-  青年的- 人中-  摄图网-  的消费-  企业文化- щи- 围墙- 邮编-  Founded- 嘱咐-  与自己-  为了防止- 他曾- 新老-  高尔夫-  ▁valve-  门市价-  ▁acre-  ▁Device-  不包括-  克萨斯-  参保人员-  ▁sheriff- exit- 冤枉- 常在- igel-  深入人心- tools- 酒业-  ▁Sharma- 熟的-  research-  新媒体中心- 之举- 等功- scopy-  ▁WiFi-  ▁baked- 照耀-  ▁Olivia-  subscribe-  ▁Wellness- 相通-  ▁Belgian-  经调查-  职位要求- olute- 医嘱-  了一款-  请广大市民- 利落- 严守- 出水- 贴身-  统筹做好- 并按-  在很多- Tips-  ▁catering- 截然- 用火-  美好生活的-  认真做好- 奇的- 甄嬛-  playing- 或不-  ▁spam-  ▁indef-  ▁Driving-  同道合- Mate-  官方微信-  志愿服务活动- 配比- nyder- ogh-  ▁Depart-  rapeutics- 铁矿- pex-  已经很- 保底- 新书- ▁PU- ▁REC-  ▁injustice-  顺其自然-  ▁Horizon-  医疗物资- 中暑-  各地要- 打得- 陪着.  ▁Trustees.  不要轻易. 中大.  刘德华. 这幅.  彰显了.  自己的身体.  ▁inevitably.  又可以 .  门户网站 . thouse .  少先队 .  配合做好 .  Profile. 药方. 通气. loe.  ▁pest.  ▁undefined. 来宾. terns. 婚纱. ▁fet.  ▁downloading.  ▁Yard.  Definition.  founded. 较量.  分布式. wz. 便能. Jam . Chat!.  ▁Ecuador". 不争#.  的地位$. 点后%. xxxx&.  ▁Administrative'. 下令(. aunder).  ▁refuses*. ان+. たの,.  资源和-. 塔利..  Represent/.  成一个0.  ▁Mick1.  ▁flaws2. 所辖3.  研究室4.  ▁Vikings5.  拥有一个6.  本赛季7. ipur8.  招聘工作9.  ▁boutique:. 寂静;.  要进行<. fax=. Device>. 矿产?. 线性@. heaA.  ▁crustB.  ▁GrantsC.  ▁restraD.  ▁tangibleE.  万众一心F. 酵素G.  独一无二的H. IgI. 得其J.  在其中K. ▁ReeL. 此人M. 筋骨N.  有些事O.  玛拉沁P.  ▁proneQ.  自己的情绪R. 后退S. ▁LtT. 眼底U.  优秀作品V.  ▁EugeneW. orateX. ▁BleY.  ▁TerritoryZ. 坏的[. 多糖\.  同时对]. alli^. oser_. 先来`.  ▁Playinga.  金银花b.  也逐渐c. 沟村d. 王室e. 腹腔f.  资源共享g.  ▁cluesh. mci. 静态j. 华尔k. 激增l.  ▁rebellm. 就怕n. Middleo. 的的p.  ▁outlinesq. reedr. 张玉s.  ▁keywordst. 亦可u. 再三v. 找了w. 深山x. 龙舟y. 老死z.  无数人{.  erville|. 朦胧}. 内置~.  ▁Athletics.  ▁interdisciplinary. 向左.  这一步. Money.  alities.  任正非.  centric. 防滑. 怀里. 自述. 丛林. 方丈. 油盐.  一个小时.  ▁popul.  高三年级.  的人和.  ▁parenting.  司令员.  ▁LOVE. 雨中.  healthy. 石膏. unders.  ▁populated.  ▁cooperative. must. dh.  公安民警. 弱点. 预见. 张开. 满载.  综合考虑.  ▁potent.  ▁tomatoes.  任何责任.  治安管理处罚法.  ▁Nearly. 主业. 平的. 礼拜.  医学科.  ▁enjoyment. 保值.  凝聚力. 胚胎. 深海. 找你. 用具.  各环节. iani. 素颜. 省属.  引导孩子.  ▁Omar. 大枣.  依法追究刑事责任. 亮丽.  ▁prevalence. 后世.  ▁Jenny.  ▁eaten.  事件的.  战争中. tte.  ▁reopen. ugu. 菜肴. 开行.  是怎样的.  ▁Luxury. 黔南. 前置.  日子里.  竞争力的.  中国工程院院士. 解压.  Resources. 刻画. 唐三.  的胜利. 推免.  的特色. 上桌.  ▁Mang.  ▁histories. 突围.  ouncing. ABS.  防护口罩. 推崇. mma.  给了我. 外包.  沟通交流.  东南部.  也有一些. radio.  继续保持.  ▁penn. ▁Rey.  riminal.  指战员.  工作职责.  ▁Secondary. 意为.  ▁depicted.  自己的努力. 危难. Move.  ▁Subst. % 实现中华民族伟大复兴. ▁Kre. Marg. 并就.  company. shell.  ▁Classical.  ▁Processing. buster.  ▁Coleman.  ▁Employee. 消退. 而他.  ▁aster.  与实践.  观众的.  ▁Slovak/ 更像/ GPS/ rag/  战斗堡垒/ 格力/  的模样/  督导组/ Bean/  玛拉沁信息网 / 老歌 / 还为 / DIR /  ▁Till /  ▁floral/  ▁socket/ ▁uma/  ▁webinar/  ▁Insights/ 建功/  健康教育/ 直通/  南阳市/  开放式/ 读写/ ▁ecc/  ▁compositions/ gong/ ▁Gy/ 保育/ 上人/  ▁Speaking/  的理论 /  ▁Davidson!/ 补缺"/  ▁multimedia#/  报名后$/  ▁opioid%/ 抗旱&/  ▁goodness'/ 外伤(/ ▁RM)/  儿童医院*/  ▁Winston+/  ▁ballet,/  ▁что-/  碳水化合物./  分享给大家// 不善0/  应有尽有1/ 创下2/ 嘌呤3/  ▁polished4/  ▁authorization5/ 一战6/  vitational7/  后遗症8/ 理疗9/  成年人的:/  ▁spont;/ 山脉/  和创新?/  今天下午@/ 聘任A/ anskB/ 关羽C/ 末端D/ 油条E/  是否符合F/ 防卫G/ ▁RPGH/  全民健身I/  能力提升J/ itlesK/  ▁GrassL/  ▁PedroM/ 亲眼N/ 节俭O/  可以获得P/  每天都要Q/ feltR/  治疗方案S/  新时代的T/  ▁MintU/  食物的V/  只是为了W/  每天晚上X/  精彩回顾Y/  隔离点Z/  注册资本[/  你想要\/ wart]/  ▁Ware^/ 水彩_/ Lesson`/  ▁Usuallya/ utscheb/ 玲珑c/ 嫦娥d/  ▁minte/  ▁Editorialf/ 赵薇g/ ciah/  ▁glovesi/ 居室j/ 反感k/ 幼稚l/  agneticm/ 手可n/  ▁detailingo/ 匹克p/  一帆风顺q/  ▁hatsr/ shirtss/  ▁issuingt/ AFPu/  上海交通大学v/ ▁ouw/ ▁ERAx/  受欢迎的y/  ▁riskyz/  ▁honesty{/  里程碑|/ 善恶}/ ▁Bug~/ schema/  千千万/  来说是/ 教招/  第一单元/ reads/  ▁COMM/  ▁potato/  ▁Zhang/  angerous/ 朱德/  这说明/ street/  ▁mistaken/  影视剧/ 卡尔/ 渔民/  ▁canon/ 传动/ 浮动/  干事创业/  ▁Obviously/ 懒得/  视频来源/  criptor/  ▁Cherry/ 从重/  ▁Struct/ Answ/ anut/ odynam/  是时候/ anol/  届毕业生/  的姿态/  ▁strangers/  ▁collaborating/ 西山/  各级各类学校/  ▁conced/ 升降/ 变好/ along/ 急忙/ 甜甜/  ▁feared/  家长要/ Stan/  全面开展/  商务局/ Tell/ native/ 冲天/ 逃犯/  在爱情/ 打听/ 村委/ 校企/ Ven/ ▁IE/  ▁dumb/ 平区/  ▁plasma/ 严寒/ :// 俱进/  传承人/  提供任何/  ▁Richards/ 中还/ tool/ 庆典/ 现存/  中国教育/  从高到低/ 化作/ 杂粮/  与社会/ ▁Wol/ still/  明确提出/ 立春/  将依法/  设计了/ 好久/ ств/ 年均/  深深地/  也不错/  性问题/  学习时间/  ▁Diane/  ▁obsessed/  网联系/ 制服/ 波浪/  刘某某/  发朋友圈/  好好学习/  ▁waist/  ▁Administrator/ 冷库/ ▁PG/ ▁Ore/ ▁emp/ 无恙/  去面对/  没有被/ ▁MTV/  一个新的/  ▁Wellington/ 乡愁/  ▁Policies/  ▁postponed/  ▁overlooking/  及作者/  ▁wished/ 企鹅/  一手抓/ 最优/  发行的/ 万起/  ▁skiing/  的能量/ 公约/  ▁constitutes/  优秀传统文化/  廊坊市/  高素质/ 一例0 风靡0  和单位0  参与活动0 肖像0 豌豆0  ▁Pilot0 ▁lbs0  深刻认识0  孩子的成长 0 唠叨 0  ▁Cleaning 0 所见 0  在香港 0 去买0  信号灯0  我们再0 也说0  任何问题0  olescent0 臭味0 sonian0 放管0 incre0  ▁prolific0  在此之前0  ▁lang0 Pool0  安全生产工作0 PLE0  志同道合0  第十九0  ▁myriad 0  从今天!0  时左右"0  getElement#0  ▁keynote$0 老妈%0  装配式&0 capt'0 大黄(0 张力)0 ▁FOX*0 ▁SMS+0  身体条件,0  ▁relaxation-0  占据了.0  中央党校/0 旧改00  钢结构10  运营管理20 保供30 动向40  并结合50 包车60  话不多70  Standard80 不买90 对华:0 日复;0  二季度<0  ▁gasoline=0 养心>0 oha?0  ▁calculation@0  ▁OrganizationsA0 但仍B0 ▁KolC0  饮酒后D0  ▁PragueE0 采纳F0  更好的自己G0 书单H0 jenI0  ▁homemadeJ0 不许K0 梦境L0 IFEM0 轻症N0  课程的O0 poraP0  ▁MitchQ0  ▁exceptionallyR0 飞的S0 肩部T0  休息时间U0  提供服务V0 istonW0  ▁merchantX0  详见附件Y0  ▁northwestZ0 tong[0  小分队\0 ▁Cod]0 水煎^0  ▁barrels_0  所有人的`0 怀旧a0 饱含b0  之后再c0  社会治安d0  ▁utterlye0  多喝水f0  ▁spillg0 军团h0 小结i0  ▁hectj0  ▁indictk0  ▁engineeredl0 ilightm0 祛除n0 浇筑o0 ➤　p0 羞涩q0  ▁Evangelr0 paids0  有多少人t0  ▁scholarlyu0  主人公v0 布斯w0  严格要求x0 声声y0  embourgz0 某的{0  还不知道|0 接线}0  ▁erupt~0  ▁Finals0 reath0 主场0 化瘀0  好声音0 应以0  ▁privileged0  营养丰富0 Arthur0  分别在0 rame0  ▁aforementioned0  ▁ransom0 孤儿0 甘油0 首日0 欢声0 Liter0  社区工作者0  遇到过0 ▁Eff0 报社0 瓶子0  ▁themed0  ▁Betty0  iciaries0  森林防火0  用地面积0  的财富0  ▁flick0 导语0 汹涌0 艾叶0  ▁Pelosi0  学专业0  中到大0 修缮0  ▁grandparents0 呐喊0 金陵0  坐落于0 的苦0  ▁Arms0  是什么样的0  ▁satellites0  这里面0  各地各校0  ▁WWII0  ▁Manuel0  提高免疫力0 acters0  的一样0  ▁sphere0  涉嫌严重违纪违法0  ▁peripher0 坐姿0 lace0 thews0  是关键0  胡萝卜素0 谈及0 '])0  了一名0  songwriter0 cot0 ▁Hir0 朝气0  的高端0  ▁eBay0 一回0  doctoral0  ▁happier0 好学0 Da0  在身上0 memory0  ▁retrieve0 嘉定0  ▁pointer0  绝不是0  ▁Highlights0 涌泉0 erem0  ▁infinite0  在此基础上0 夺得0  有一部分0 满洲0 qt0  ▁prey0  省委组织部0  制定的0 午睡0 火炬0  体系的0  年轻的时候0 深切0  ▁José0  ▁Strange0 诸法0  的物质0  小伙子0  ▁compile0  ▁denying0  ▁forecasts0 ▁BAS0  也更加0 扬帆0  相对应0 fact0  摄入量0  北京的0  综艺节目0  打交道0  Changed0 语数0  毕业年级0 Images0 盆腔0  简单来说0 蔚来0  从玛拉沁信息网0  ▁americ0 想把0 lters0 融洽0  ▁compromised0  女人味0  带给你0  一路上1  这套书1 善举1  故意杀人1 肛门1  活力的1 荡荡1  本文内容1  特殊教育1 极强 1  不排除 1  ▁Keys 1  产品和 1  吉林市 1  大货车1 спо1 ▁HO1  水资源1  ▁wasted1 ▁SY1  ▁Celtic1 ifax1  ▁tissues1  对生活1  甚至还有1  盲目的1  空间啊1 Justin1  ▁registry1 offcn1  successful1  ▁Mold1 飞船 1  ▁eternal!1  ▁sticking"1 EF#1  最后一天$1 这辆%1  招生工作&1  ▁washed'1 化州(1  的意识)1  以下几个*1  有很多人+1 入股,1  始建于-1  的搭配.1  ▁pumps/1 实录01  即可进入11 Widget21 √　31 TX41 中对51 在同61  原作者所有71  ▁aspirations81 化验91 马尔:1  rospective;1 起航<1 ▁bud=1 是真>1  无数次?1 ajax@1  生命危险A1 很短B1  发达国家C1 浸润D1  ▁baptE1  ▁capitalismF1  ▁AustraliansG1 科教H1 电量I1  学习群J1  小可爱K1 进项L1  ▁kickingM1  ▁investigativeN1  能帮助O1  户籍所在地P1 又说Q1 itutedR1 audeS1 果农T1  ▁reproductionU1 讲了V1 ▁âW1 的酒X1  ▁PortsY1 比利Z1  ▁Designed[1 米路\1  优秀人才]1 田径^1  ▁Trevor_1  参与方式`1  签订劳动合同a1  lastingb1  物资储备c1 复议d1 astae1  ▁shortsf1  ▁Demong1 积攒h1 Rosi1  ▁wearsj1  ▁doctoralk1 缠绕l1  与企业m1  顾之忧n1  openhageno1 伸直p1 肿痛q1 宿州r1  新篇章s1  ▁Biket1 的李u1 坏人v1 摘下w1 精子x1  ▁philosophicaly1 冥想z1 朝天{1  从而导致|1 妄想}1 西昌~1 近乎1 碧玉1  监督管理局1  多云转1 PLUS1  默默地1  公安部门1 花落1  hentication1  第三名1  ▁Owner1  ▁impart1 ®,1  的宝宝1  专科学校1 点儿1 从上1  原生态1  补短板1  ▁Investors1  ▁proactive1  ▁negligence1  走下去1 ichi1 逆向1  ▁calcium1 整套1  报名登记表1  ▁spans1  ▁contracted1  沧州市1  党政机关1  ▁Terminal1 Town1  的训练1  可使用1  就不用1  比例为1 ▁Sou1 PPP1  ▁Russians1  更年期1  时间管理1  统一安排1  你还在1  ▁rope1 ANK1  农村人居环境1 挂碍1 bons1 孤立1 sth1  告诉她1 ▁MIN1 Kate1  ▁Baron1 让步1  人体内1  商业模式1 对本1 求精1  ▁ankle1  ▁Donna1 跳楼1  要提高1 Vector1 heid1  一定程度1 JC1  ▁Edit1 死神1 ▁Sz1  ▁stimulus1  ▁unbeliev1  ▁laughter1  practice1  ▁newborn1  arenthood1  不是你1 楷模1 正财1 被捕1  委员会办公室1 群聊1  claimer1  做的就是1  ▁Chambers1  ▁southwest1  ▁Copy1 Boston1  ▁Voices1  visible1 ▁Ф1 记账1  生活服务1  ▁entre1 苏打1  ▁spike1 ▁Ti1 启迪1 汤圆1  ▁specification1  做了一个1 舟山1 nton1 倒车1  ▁abnormal1 soever1  只有自己1  一直都1  ▁doctrine1 忠于1  公租房1 I B▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁1  ▁Rally1 铿锵1 遍了1  金融科技1  ▁Maintenance1 ▁nas1 umping1  ▁rejection1  ▁promo1 Lib1 alis1 自拍1 hin1  预警信号1  初中数学1 *,2 咬牙2  ▁Reply2 上期2  ▁Crusher2  ▁tokens2  ▁caption2  一百年2  工商联2  经常吃 2 ▁bee 2  专项计划 2  ▁Celebration 2  ▁thunder 2  ▁lakh2  ▁fabrics2 协办2 ▁MW2 中选2 勇往2 bee2 派位2  直接的2  ▁awaiting2   2 先辈2 可与2 防城2 iates2 Copy2  重点场所2 佛像2 平价 2  培养孩子的!2  ▁balancing"2  给我的#2  ▁justified$2 伦理%2 脚跟&2  事情的'2  ▁auch(2  年春季学期)2 hello*2  ▁sourced+2 机床,2 圈的-2  提供一个.2 跑道/2  ▁Cooperation02  testing12  ▁Powered22 运程32  Comments42 身着52 中转62  ▁trilogy72 说你82 分诊92 ▁ETF:2  ▁continuity;2  在其他<2  ▁journeys=2  临沂市>2  ▁Victory?2 ngth@2  主要从事A2 不自B2  东北地区C2  ▁pastaD2  ▁dealershipE2  前行的F2 乳头G2 诱人H2  参观了I2  ▁fathersJ2 ▁AmbK2 + $新型冠状病毒感染肺炎疫情L2 élM2 仙桃N2  二十年O2 女童P2  ▁SettingsQ2 heeR2  lessnessS2  ▁stoleT2 并被U2  民警立即V2  ▁BasinW2  给大家带来X2  从来不会Y2 vaccZ2  各专业[2  实际的\2  ▁Blanc]2  各项防控措施^2  ▁tasked_2  ▁religions`2  资产管理a2  ▁supplyingb2 皆空c2  ▁acceleratedd2  真是太e2  千方百计f2 上交g2 后半h2 始皇i2 面孔j2  ▁Domaink2 独处l2 ICALm2  数据中心n2 此项o2  这是我们p2  ▁Sophieq2 真皮r2  ▁climbeds2 藏品t2 隔夜u2  全家人v2 alamw2  ▁Elsex2  患者在y2 Aroundz2 espoke{2  ▁halfway|2  培训学校}2 优于~2 宇航2 水土2 saving2  ▁Growing2  olation2  爱你的人2 Impro2  ▁removes2 全院2  生物医药2 不屈2 凭着2 顺延2  不支持2  ▁defects2  ▁lengths2 生水2 ▁Nex2  药物治疗2  ской2  中国石油2  ▁Scene2 ▁MR2  ▁gluc2 无言2  的报道2  的材料2  压实责任2 碗中2  战场上2  ▁striker2 悠然2  ▁subsidies2  ▁obsc2  ▁Neither2 丢掉2 日日2 racuse2 cup2  间多云2  ется2 减压2 淮河2 诸暨2  ▁Carnegie2 从天2 命的2 都像2 三万2  ▁Pizza2 nst2  ▁contem2  ▁expenditure2 alion2  不停的2  国家主席2 养活2 links2  通过理财2 喜迎2 ▁SV2  口腔溃疡2  ▁dietary2 二环2 展演2  大专以上学历2 dream2 不问2 头等2 要与2  好奇心2  家长会2  对于一些2  ▁addict2 ▁Enh2  ▁pitched2 况且2 简史2  ▁Wrong2 转告2  ▁amazed2 向他2  思想认识2  umption2  范冰冰2  ▁arbitrary2  ▁nutritional2  在深圳2  永远的2  ▁showcasing2  itchens2  的考试2  ▁discontin2  工业化2 mask2  也很好2 龙的2  疫情扩散2  愉快的2 CHO2  理论上2  共和党2  ▁jersey2 HF2 ▁ze2 大考2 开国2  ▁homelessness2 老区2 ►　2 ATIONS2  ▁accommodations2 ▁});2 Wire2 ovel2  大学毕业2  常说的2  活动策划2  ▁suspicion2 你再2  African2 下同2  members2  ▁Goldman2  ▁exclusion2  ▁grams2 转机2 ▁PD2  下降了2 醒目3  引以为3 何为3 转转3  ▁panor3  ▁Leading3  能够让3 ughed3  ▁Poor3  ircraft 3  你的孩子 3 drive 3  相关法律法规 3 持证 3 柔性3 必考3 meter3  ▁senators3  ▁arrow3 老挝3 护卫3  不变的3  全天候3 警醒3 hered3 虐待3  负责任3  暂行办法3  浪漫的3  subject3  ▁lonely3 *)3 倒霉 3  地方的!3 chart"3 一元#3  李佳琦$3  版本的%3  ▁Dodge&3 ilo'3  所包含(3  少年儿童)3  时刻保持*3 螺纹+3  改变自己,3 tmp-3 陈晓.3 Insp/3  ▁WILL03 ):　13 卖给23 ▁rul33  ▁resorts43  团组织53  和旅游63 中日73  体制改革83  切实维护93  ▁Manila:3  ▁respects;3 千家<3 水乡=3 胡子>3 dar?3 safe@3 ChooseA3 ( !新冠肺炎疫情防控指挥部B3 通了C3  安全保障D3  教学经验E3  ▁rodeF3  什么样的人G3  ▁flavourH3  居民的I3  ▁ScientistsJ3  原标题K3  可以为L3 大势M3  高中阶段N3 EBO3 故居P3 失衡Q3  和省委R3  ▁stemsS3 AreaT3  ▁TorresU3 相貌V3 麦当W3 人以X3 讲坛Y3 喜乐Z3 Pref[3  ▁rage\3  Package]3 远了^3 祈求_3  职业能力`3  ▁midfieldera3 发炎b3  指示精神c3  ▁travellersd3  ▁conceivede3 下坡f3  是不可g3  表现的h3  ▁Itemsi3  ▁glutenj3 声誉k3 andil3 舞弊m3 花果n3 金石o3 ▁gyp3  ▁Hannq3  到目前为止r3  让群众s3  原创作品t3 Forceu3  ▁pertv3  降水量w3  面向全国x3  commenty3 城墙z3  ▁Integrated{3  ▁offence|3 AAA}3  ▁PART~3  ▁grassroots3 标兵3  家乡的3  ▁fixing3  urricanes3 ILD3  是一所3 惦记3 两卫3  错过的3  突出贡献3 多喝3  respect3 Comb3  健康运势3 月末3 棕色3 目镜3 ▁BU3  ▁Coin3  ▁shirts3 を见3 一拍3  Particip3 矿区3 CBS3 oge3  ▁elevation3  台湾地区3  ▁Winners3 类人3 Neill3  ▁orchestr3  中学校3 经脉3 ▁Eat3  ▁billing3  ▁dysfunction3  ▁intentionally3  这四个3  ▁jeans3  ▁blanket3  ▁eventual3 ▁BI3  ▁Armenia3  的外观3  干细胞3  可以让你3  ▁teamed3  选择性3 乐清3 新民3  或者在3 军营3 不稳3 所含3  ▁Darwin3  ▁empowerment3  ▁ال3  社交平台3  ▁gases3  ▁dessert3  ▁Forever3 巨人3  介绍说3  ▁murders3 小暑3  技能人才3  ▁Wear3 后才3 繁多3  行政管理3 折合3  无条件3 Mother3  折不扣3  清楚地3  发挥作用3  ▁Ratings3  应急演练3  ▁beverages3 顺水3  但也是3  ▁humour3  ▁Dodgers3 渔船3  ▁reluctant3 жно3 Db3 大头3  ▁spinal3  ▁similarities3  ▁volunteering3 太快3  手机上3  ▁Hang3  amental3  构成了3  了他的3  ▁crises3 眼霜3  代表性的3  持续更新3 Eth3  影像学3 usage3  ▁cops3  Callback3 医治3 WP3  ▁presidents3 石灰3  设计方案3 琐碎3 gem3  必修课3 occup3  的重要讲话3  ▁historians3 火势3 热力3 晓彤3 稻草3  这些事3 today3  ▁poised3 古琴3  ▁dull3 assets3  是怎么回事4  是否能4 强身4 电车4 示威4  Directory4  ▁Architects4 Fa4  已经有了4  报名材料 4  第二阶段 4  统筹安排 4  重点是 4 噩梦 4 cdn4  ▁для4 大成4 暗恋4  理解的4  脱贫致富4  ▁Plastic4 陕北4  带薪年假4  构成犯罪的4 绅士4 Fair4 也应4 travel4  ▁patrons4 jie4  在广州4  坚持把4  人民银行 4 年金!4  aternity"4  ▁Vert#4  土地使用权$4  肯德基%4  ▁handler&4 拭子'4  八路军(4  就只能)4 州长*4 flag+4 用以,4 一晚-4 IQ.4 一脚/4 有我04 ▁wip14  ▁ranch24  ▁guitars34 坐下44  ▁weddings54 大伙64 密的74 席卷84  ▁Chicken94  ▁uncovered:4  outheastern;4 动摇<4  ▁infringement=4 拐点>4 浪尖?4 发到@4  学院的A4  灭火器B4  ▁witchC4 故里D4  折优惠E4  情侣头像F4  重点任务G4  ▁NobleH4 历来I4  发出了J4  政府采购K4  ▁floodsL4  立即购买M4 背上N4 ▁BSO4  ▁geographyP4  临床研究Q4  不平衡R4  借款人S4  就会导致T4 semblU4  ▁BarrettV4 滁州W4 树的X4  小技巧Y4  ▁projectionsZ4  都是因为[4  ▁Rosa\4 黄鹤]4  的那样^4 pieces_4 糖分`4  主要是因为a4 眼花b4 十天c4 卷烟d4 如火e4 大手f4 崇德g4 棒的h4  系统和i4  政法委书记j4 有理k4  而是在l4 亚军m4 望去n4 田县o4 教辅p4  第十八q4  ▁germr4  ▁Evenings4  指导和t4  就地过年u4 enziev4  ▁prematurew4  经济运行x4 attiy4 囊肿z4  持续开展{4  ▁cough|4 原名}4  道德经~4  ▁queer4 minent4 去哪4 芥子4  国内疫情4 背包4 平坦4  健康中国4 腾腾4 扔掉4 领衔4 ▁KB4 Integ4 仇恨4  医保基金4  ▁WARRAN4  ▁Wealth4  daughter4  ▁Equality4  第一部4  服务质量4  Tickets4 标示4  路街道4 ▁AK4  公认的4  建议收藏4  感染风险4  ▁integer4  ▁Artificial4 防盗4  ▁hormone4  induced4 needed4 ATES4 jh4 谢绝4  我只是4  印象深刻4  Virtual4  ▁Default4 ovi4  的顺序4 领航4 adors4  ▁skept4 ilot4 光谷4  还不能4 所幸4  视频教程4  ▁blogging4  图片来自4  ▁coff4 耐用4  与时俱进4 拉法4  ▁Hampton4 凉的4 aeda4  ▁vest4  ▁skies4  ▁commodity4  ▁Bash4  完成任务4  城中村4  ▁Cinem4  ▁policym4  ▁się4  ▁Regular4 宋朝4  exports4 丹东4 更难4  纯手工4  传播途径4 杏仁4  粉丝福利4  ▁Marion4  ▁Harbour4  ▁Legislative4  的人文4  活动和4 lio4 直系4 省的4  就是我4  请直接4  现实生活中的4 Career4  ▁Cornwall4 维亚4 Boy4  所以要4  全面贯彻4  ▁fres4  ▁Nixon4  ▁satur4  ▁collects4  注意啦4  皮肤病4  ▁Demand4  ▁stab4 lif4  缴费年限4  在日常生活中4  的婚姻4 粉末4  系列的4  年的时候4 烈日4 释迦4 ści4 谷物4  充满着4  ▁Glad4  ▁Venture4  ▁Previously4 ▁fli4 Parser4  ▁aspiring4 老是4 硬度4 ritz4 ▁toe4  警务辅助4 水煮4 考了4  打疫苗4  ▁gesture5 BG5 跳出5  学习习惯5  eliness5 透视5  的一场5 缓和5 弟兄5  ▁Flow 5  ▁Strategies 5 桥区 5  交流平台 5 OLED 5  ▁pictured5 空虚5  为建设5  部门联合5 不入5 赠与5  延迟退休5  统筹协调5  ▁Gardner5  ▁embracing5 筹码5 诉说5  ▁mega5  图文来源5 不增5  ▁Quad5  etically5  ▁cancers5  ▁realizing 5 地势!5 ERROR"5  委员会副主任#5  冲锋在前$5  加班加点%5 KY&5 云集'5 文安(5 车里)5 ubb*5  拟录取+5  ▁physi,5  ▁Cotton-5 叫我.5  爱心人士/5  ▁Legion05  ▁affiliates15 丰盛25 换新35  家家户户45  创作者55  诠释了65 为爱75 显微85 拳头95  农村的:5  Articles;5 ▁FCC<5 油门=5 下线>5  ▁meteor?5  市中心医院@5 gasA5  ▁jointsB5 黑板C5 jeeD5 ▁ISTE5 opensF5  ForeignG5  的病人H5 无忌I5  ▁PatentJ5 腹肌K5 军舰L5 花粉M5 椰子N5 黑豆O5  水产品P5 雷霆Q5  南北通透R5 flightS5  疫情之下T5  ▁RustU5 严管V5  统一战线W5 潜心X5 inhoY5  无限的Z5 条约[5 公分\5 烫伤]5 tta^5 品读_5 小额`5  排名前a5  县委宣传部b5  常态化防控c5 avourd5  ▁Giane5 跳水f5  orderedg5 好感h5 润发i5 编导j5  ▁presumablyk5  进一步强化l5  ▁semifm5  ▁Kazakhstann5  ▁retainingo5 就走p5  ▁Signsq5  ▁abortr5 HMs5  ▁Wendyt5 Liberu5  科学与工程v5 谈心w5  ▁Crimx5 毛发y5 居多z5 石窟{5  对于我们|5 之多}5  的现实~5  切实提高5  ▁appealed5 权重5 Ba5 应为5 频发5  个人卫生5  锻炼身体5  ▁sovereignty5  赶赴现场5 台北5 啦啦5 Rom5 UTF5  ▁patches5  ▁validate5 这将5  本文为5 groups5  大剧院5  泡水喝5  重难点5  ▁Miguel5  即使在5  ▁princess5  ▁thickness5  离退休5  ▁adjustable5  坚定理想信念5 ingle5  不久前5  坏习惯5  ▁traction5 inian5  人与人之间5  及时的5 udeau5  ▁Hamas5  最关键5  理论和5  的违法行为5  ▁Important5 起于5 定力5 铸就5  coholic5  张照片5 Court5  ▁itiner5 衰弱5 breaks5  的方式来5  ▁Claud5 光灯5 极速5  东北部5 恶魔5 选修5 DVD5  必须有5  属鼠人5  锡林郭勒5 旧爱5  企业职工5  解决办法5 光华5 预检5 一课5  可行性5  帮我们5  功能障碍5  留守儿童5  ▁Accident5  ▁Weap5  rossover5 Gs5 铁军5  几十万5  ▁bipartisan5 些许5  ▁Shut5  ▁Collective5  六安市5 一封5  这种现象5 潜移5 لى5 通胀5  ▁renamed5  丝绸之路5 丹麦5 嘉峪5  ▁Powers5 盖子5 增至5  综合服务5 )|5  等诸多5  ▁Shane5  淄博市5 要好5  ▁über5 深爱5 青花5 做点5 图画5 ▁awe5  权威发布5  ▁dile5  设立了5 На5 AMS5  党工委书记5  ▁twentieth5  与应用5  ▁slowing5 老抽5 ETS5  ▁Steps5  两部分5  ▁shadows5 ("#5 菜鸟5  ▁configured5  踏踏实5  ▁skate5  ▁apprentices5  ▁governmental5 XML5 在山6 ▁Sak6  Pakistan6  ▁coating6  supported6 各家6  ▁Tucker6  ▁grands6 冷饮6  功能性 6  ▁Average 6 盲盒 6 定量 6 要走 6  日早上6  黑眼圈6 rais6 城投6 版纳6  ▁synthesis6  正式发布6  理论知识6  的勇气6  ourable6  ▁ladder6 压轴6  莫名其6  有害物质6  全员核酸检测6  食品安全国家标准6  ▁Christina6 kW6 单招 6  综合能力!6 世音"6 厚道#6 ▁QB$6 Guard%6 波兰&6  看起来很'6  负面影响(6  ▁Models)6  なたの*6  ▁Syracuse+6  规模的,6  收费公路-6  相比之下.6 在人/6 之父06 微粒16  高新技术企业26  ▁Cardiff36  ▁Titans46  正式上线56  ▁elephant66 Meta76  ▁Turb86  ▁starters96 违停:6 icus;6  ermanent<6  ▁predictable=6  ▁Azerbaijan>6 数是?6  ▁Mets@6  ▁TemplateA6 五蕴B6  ▁conservativesC6 らんD6 ▁BriE6  ▁WongF6 一环G6  ▁AmberH6 极力I6  很长时间J6 loopK6  提起公诉L6  ▁AlexandriaM6  ▁uncertaintiesN6  ▁appellO6  ▁restrictionP6 小金Q6  刺激性R6  不要错过S6 遮阳T6  有名的U6 以轻V6  ▁picnicW6  红色文化X6 马虎Y6  ▁outrightZ6  生命力[6  第二大\6  营造出]6  工资面议^6 武则_6  跟我说`6  ▁Platea6 家暴b6 餐后c6  打工人d6 ▁Were6  潜移默f6 advg6 ▁SMEh6  州市公安局i6  ▁Dependingj6  终于在k6 RECTl6 FDm6 odiedn6 VOo6 小龙p6  的一个重要q6  就是把r6 首款s6  ▁robberyt6 桃李u6 示意v6 modalw6  ▁irregularx6 而她y6  ▁freezingz6  陈先生{6 握手|6 镶嵌}6  为一个~6  ▁undermine6  多媒体6  ▁uncommon6 岸边6  ▁tactical6  无锡市6 rz6  准备好了吗6 dz6  ▁firefighters6 主卧6  售楼处6  ▁stretches6 照护6  ▁McCl6 诗经6  图文来自网络6  你不能6  社区工作人员6 带您6  ▁baseline6 销毁6  考生可6 之都6  西城区6  我和我的6  高质量的6  ▁Originally6  生肖蛇6 塑形6  换届选举6 多层6  月份的6 复盘6 elo6 lower6 宴会6  ▁Erin6 birds6 点上6 usions6  ▁shuttle6  没有必要6 负有6 aleigh6  ▁inserted6  ▁organisms6  还是挺6 空闲6  ▁pitching6 马某6 Pier6  视觉上6  ▁Chrom6  ▁inacc6 ▁ot6 mania6  general6 按期6 防病6 淹没6  ▁Poss6 武帝6  结婚了6  ▁authenticity6 ▁Dre6  ▁Cookie6 争当6  重点区域6  ▁societal6 年幼6  反过来6 爆了6  的分析6 列出6 绯闻6  不可抗6 国安6 芭蕾6 蜜桃6 Kind6 航行6  自己对6  ▁Belfast6 为新6  ▁Dining6  与美国6  ▁excel6  ▁membrane6  ▁synd6  无理由6  在历史6 irling6 一班6 网球6 RMB6 陇南6  了解详情6  努力奋斗6 不进6 长兴6 ▁UE6  ▁raid6  ▁Carrie6  ▁Verizon6 四楼6 Admin6  ▁pestic6  ▁intermediate6 乡土6 放映6  ▁Avoid6 elic6  ▁radius6 一棵6 将该6 oby6 益处6 锦鲤6 lead6 港股6  ▁Schne6  ▁Browns6 如画6 盛会6 医者6  ▁erad6 潜水6 精明6 匮乏7 现住7 弯道7 footer7 志强7 Conc7  ▁Genesis7  ▁Adapt7 Occ7 àn 7 鼻涕 7  容易引起 7 储物 7 Tal 7  ▁од7  ▁economist7 invent7 linked7  ▁Sloven7 做菜7 末尾7 顺滑7 flies7  ▁internally7  家庭经济7  告诉他们7 叙事7 INTER7 兵力7 听闻7  数据统计7  这世上7  ▁retiring 7  通货膨胀!7  ▁Publisher"7 侵蚀#7  ▁generosity$7 不需%7  学信网&7  传染源'7  过不去(7  ▁rentals)7  痛苦的*7 没用+7  集中的,7  ▁vacant-7  医科大学附属.7  的他们/7 伯特07  ▁Symposium17 魔力27 tok37  speaking47 ▁Gym57  ▁mansion67  ▁defenders77 utz87  ▁Firefox97 ATT:7  特别说明;7 byter<7  ▁dentist=7  ▁paperwork>7  ▁ceremonies?7  青春的@7 一觉A7 西街B7  れがあC7  警方提醒D7 raulicE7  ▁strainsF7 てごG7 砂糖H7 ikuI7  ▁AustrianJ7 ultanK7  过日子L7  会越来越M7  疫情当前N7  ▁asleepO7 ほらP7 AffQ7 cartR7  ▁StandingS7 经受T7 两句U7 劣势V7 大洋W7 过剩X7  都是从Y7 inlineZ7 也挺[7 同源\7 tti]7 Mom^7  综合实力_7  ▁contexts`7 ▁PSTa7  ▁orthb7  建议在c7 浦区d7 呼叫e7 蜡烛f7 ▁heyg7 asperh7 电价i7  成绩查询j7 锻造k7  的星座l7 Factm7 ▁Lazn7  ▁antibioticso7  为防止p7  ▁Pillq7 人缘r7  公安交警s7  ▁Tiert7  项目中u7  睡眠时间v7 帮您w7 玄武x7 POSy7 文玩z7 Zone{7  ▁Sounds|7 超长}7 遗失~7 社科7  困难群众7 萌芽7  不重要7 Browse7  ▁Sherman7  ▁Examples7 功劳7 祛斑7  ▁Medium7  向中国7  祝福语7 lectic7  builder7  てごらん7 ILITY7  起来吧7  ▁deliveries7 )),7  和行为7 收支7  ▁Whitney7 内膜7 推开7  就业的7 写照7 得当7 永安7 而起7  不太好7  十三届全国7  ▁cinematic7  为了方便7  ▁Dominican7  就一定7  纪念日7  不经意间7 扩容7  ▁Welfare7  ▁randomly7  ▁switches7 出事7 涵养7 ▁Ek7  开展以来7  ▁Position7 rika7 说要7 甲方7  れがあなたの7  让我们的7  ▁deaf7  ▁appla7 吾悦7  跟不上7 ▁mp7 洗车7 的门7  都是由7  ▁Cheap7 vable7  关心的问题7  ▁seekers7 边防7 辅食7 VAC7 良药7 鸡毛7  ▁Hers7  做到的7  ▁columnist7  ▁Judy7 哭着7  condition7 Ho7 有礼7  ▁Syndrome7 paced7 views7 HY7 amba7  ▁Chance7  ▁thereafter7  等业务7  贫困村7  ▁occurrence7 小林7  求职招聘7  ▁comfortably7  肝硬化7  利用职务7 owment7 一气7 挂着7  动人的7 哭闹7 插图7 签发7 接壤7  意见和建议7  stories7  ▁cocaine7  丰富了7 ▁Vot7  ▁reinforce7  周边的7 跨国7  ▁wheelchair7  sharing7 ▁Tut7  即将到来7  好好珍惜7  ▁laughed7  ▁multitude7  ▁seamlessly7  ▁attracts7 关村7 吊带7 赡养7  ▁helmet7  ▁Challenges7 涉疫7 屋内7  ▁Rare7  ▁alpha7 与月7  都会在7  ▁hazards7  ▁screaming7 亲吻8 两层8 ▁DS8 流的8  ▁termination8 请战8 ▁quo8 agging8 Capt8  主义者 8  浪费时间 8 查漏 8  影响着 8  舒适度 8  然后将8  ▁chooses8  进入考场8 背负8  支持你8  ▁atop8  第二季8  ▁router8  ▁Bangkok8 下山8  公共区域8  ▁introduct8  ▁astonishing8 amma8 beck8  工程的8  篇小说8  ▁inexpensive8  都是免费订阅 8  公务员录用体检!8  ▁scanning"8  副产品#8  强制性$8  如何才能%8 scold&8  ▁verbal'8 交大(8 约占)8  ▁servants*8  ▁Carey+8  健康生活,8  无数的-8  三室两厅.8  Congress/8  要高度重视08 西风18  神秘的28  generated38 ▁Lon48 黑头58 ods68  这三种78  ▁destructive88  抗疫情98  ▁hazardous:8  ▁Maya;8  各族人民<8 metry=8  ▁ruin>8  ▁preceded?8 职教@8  一切的A8 磨练B8  ▁blossC8  presidentD8  ▁CONTE8 僵尸F8  低收入G8 athaH8  ▁metropolitanI8 coolJ8  岁以后K8  ▁dellL8  ▁brakeM8  ▁MiltonN8  ▁AutomotiveO8 声中P8  大部分地区Q8  ▁payloadR8 RogS8 ′,T8 增产U8 等我V8  政府和W8 灭亡X8 正视Y8  是完全Z8  将出现[8 就已\8  万块钱]8 小黄^8  ▁Persian_8  无处不在`8 月季a8  日之前b8 挺拔c8  ▁Pushd8 胜地e8 rokesf8  生态文明建设g8 唱响h8  才有了i8 bralj8 ooksk8  ▁Disasterl8 打球m8  零容忍n8 想来o8  小哥哥p8  ottenhamq8 变幻r8 锦州s8  ▁Lifestylet8 着吃u8 逼迫v8 贪图w8  及应用x8  秦始皇y8  送给你z8 立意{8  ▁kernel|8  ▁bonuses}8  ▁southeast~8  Analysis8  不那么8 ynamic8  ▁sings8  成为中国8  中央电视台8  ▁Providence8 ELD8 armed8  粉红色8  房屋征收8 ▁sax8 抉择8  ▁Bronx8 富翁8  一直被8 ▁tin8 钟情8  疫情对8  ▁premiered8 Ax8 哈利8 步走8 面具8 出走8  ▁Favorite8 数为8 会后8 话剧8  ▁Spin8  ▁granite8  ▁Landscape8  ▁Documentary8 用钱8  报考人数8  getElementById8 机智8  明年初8  ▁oceans8  ▁skull8 越秀8 ranch8 中俄8 园艺8 跻身8 rano8  ▁Alcohol8 隐隐8  ▁Cairo8  ▁pianist8 ▁Lap8  ▁plots8  多公里8  ▁embry8 Sure8 ijk8  ▁Lightning8 美妙8  之外的8 Artist8  操作系统8  ▁unaware8  ▁Underground8 国债8 独有8  甚至有8  ▁laptops8 再审8 nerg8  ▁competit8 она8  每天都会有分享8  ▁Molecular8 乐意8  ▁Handbook8 ▁pas8  希望自己8  ▁HTTP8 中段8  极强的8  的这些8  放管服8  潜伏期8 uffs8 为大8  勇往直8 化和8 变局8 多用8 泛滥8 再有8  ▁Spiel8 换位8 ▁FI8  ▁reminiscent8  ▁debts8  正在接受医学观察8 长约8 焕发8 宜兴8 碎了8  defined8  临床症状8  ▁resides8 远近8 相册8 雇主8  协会会员8 ▁'/8 ▁Lok8 Almost8  聊城市8  恢复到8 爱因8 秀丽8 Circ8  Collections8  ▁whatsoever8 短语8 心智8 高能8  同志们8  农业银行8 mass8 寒湿8 派的8 bid8 打掉8 ▁io8  嘉峪关9  迎难而上9  progress9  成功了9 长乐9  刘女士9  新业态9  院校的9  urations9  ▁Respond 9 药效 9 送医 9  ▁reservoir 9 刊物 9  抗击疫情的9 隆回9 挥发9  服务体系9 奥利9  ▁nowadays9 录播9 andin9  ▁regulator9 ▁NT9 associ9  ▁recurring9  指导组9 继而9 肾上9 脚尖9 Delete9 伙食9 储藏 9  等有关!9 留念"9  审查起诉#9  史上最$9 earth%9  各位家长&9  ▁Dash'9 污垢(9 飘逸)9 anu*9  多渠道+9  呼吸系统,9 辨证-9  的评价.9  现代化的/9  ▁intricate09 五华19  我看到29  不会因为39  五脏六腑49 如水59  民政部69  这个过程79 亦复89  ▁Bulld99 丝丝:9 枫叶;9 OPPO<9  ▁Advice=9  ▁Jacobs>9  ▁Hidden?9 春招@9 CATA9 GoingB9 伊犁C9 古风D9  三分钟E9  技术支持F9  ▁shortagesG9  ▁surrenderH9  ▁RailroadI9 古朴J9 殖民K9 linearL9  想一想M9 在和N9 王府O9  香港特别行政区P9 安吉Q9  ▁HortR9 贵妃S9  ▁CallsT9  ▁RiversideU9 加紧V9 琉璃W9  ▁vaultX9 ioxidY9  ▁FlowersZ9  不同程度[9 Indeed\9 红领]9  马鞍山^9  物理学_9 better`9  ▁inadequatea9 罪恶b9 轰动c9 Damd9  紧张的e9  ▁Authorsf9  人生在世g9  ▁Merrh9 ▁Зi9 大卫j9 胸痛k9  故意伤害l9 queuem9  ▁HREFn9  ▁reuniono9 暖和p9 streq9  结构化面试r9  Williamss9  ▁avent9 hofu9 心烦v9 民歌w9 石河x9 科的y9 Terz9 极少{9  ▁Lounge|9 deen}9  ▁suffers~9  ▁Meta9 强强9 __(9  ▁Sell9 回望9  的小孩9 ▁exh9 体操9  ▁Bahrain9 爆出9 abl9  ▁clicks9 花都9  合格证9 质朴9  心理学家9 插座9  浏览全文9  被曝光9 应收9  大道与9 ▁IPO9  ▁sorted9 你可9 abc9 如一9  也是对9  报名地点9 之门9  会更好9  练习题9  解决实际9  ▁Mayo9  Transport9  ▁Netanyahu9 本品9  的感受9 inqu9 灯塔9  不公平9 ifty9  以此来9  ▁unsure9  ▁correlation9 亲生9 受体9 dm9  了一年9  ▁Node9 入微9  护目镜9  规划的9  ▁insufficient9 exe9 透析9  阜阳市9  有异议9  ▁Premiere9 此案9  小仙女9  停不下来9  ▁sunshine9  ▁storing9 者们9 Covid9 总站9 时被9  ▁Researchers9  ▁Eden9  应用程序9  所以大家9  ▁avec9  这种行为9 本该9  ▁plaintiffs9  ▁Cave9  质量安全9 将就9 雨量9 ▁lig9  ▁shark9  再进行9  ▁Inform9  returns9  ▁remake9 血浆9  和面试9 ospice9  ▁Tang9 ▁Gul9  ▁keyword9 SX9 shape9 GU9 改成9 环评9 ▁ICC9 主妇9 台县9  南京大学9 集美9  金山银9 之久9 罪行9  这座城市9 怪兽9  认可的9  ▁Flood9  ▁Manning9 凶猛9  我们会在9 元年9 ribly9 佳作9 液压9  给大家介绍9  estinal9  ▁smiling9 vn9  招聘教师9  培训中心9  ▁vegetation9 踏青9  ▁terrace9 看作9  接电话9 普工9  意识的9  自家的9 棚改: 涉恶:  战胜疫情:  双胞胎:  的口号: entric: 嚣张: 说她:  在上面:  ▁Northwestern : urdy :  ▁averaging :  通过电话 :  ▁Gerald : 抗压: 支线: 无毒: aye: 之谜:  ▁Sanchez:  人群聚集: emale:  ▁converting: ▁dar: ecd:  欢声笑语:  ▁needle: 舌苔:  总公司:  ▁pulse:  ▁owning:  ▁creations:  ▁Comes : 开奖!:  年休假": 枪支#: 散了$: 航海%:  宣传引导&:  十字路口':  ▁cozy(: 在下): Jos*: ctr+:  ▁investigator,: 奇幻-:  Finding.:  ▁judged/:  和地方0:  勇于担当1:  ▁Lore2: 仍需3: 强健4:  解除隔离5:  上的一6:  ▁workflow7: 徐志8: 还应9: ienced::  调查发现;:  ▁Gateway<: 肺气=:  主要集中在>:  受贿罪?: 他都@:  ▁cliffA:  alleledB:  ▁onsetC: 反倒D: 照见E:  看不懂F:  呼吸机G:  的道路上H:  isionalI:  生肖羊J: проK: ▁TucL:  是不是很M:  浓厚氛围N:  舌尖上的O: letesP: 忧虑Q:  四川大学R:  与本网联系S:  herenceT:  ▁slidingU: 演戏V:  奥特曼W: posalX: stonesY: 骑手Z:  数量的[:  ▁fridge\: 板的]:  一个家庭^: 博大_: 坚硬`:  人性的a:  加入到b: ▁FINc:  也是最d:  交通管理e:  ▁Arrowf: Approg:  ▁maturityh: NavBari:  ▁chasingj:  封闭施工k:  aggeringl: 塘镇m:  ▁confrontedn: 心悦o: 投行p: 法务q: encersr:  Samsungs:  都有了t:  ▁outdatedu:  母亲节v:  estonesw:  ▁Stonesx: ▁Sagy: 瓜子z: 画作{:  ▁jumps|:  具有重要意义}:  具体以~: 每亩: uni:  　…　:  为社会:  最新进展: 瞒报: IRST:  那是因为:  现在我们:  网络直播:  ▁adhere:  ▁useless:  ▁mercy: 凝固: 求救:  拖拉机:  科技学院:  ▁ling:  容易造成:  ▁refusal: 克星:  广泛关注: omical: >):  层面的: shang: 拿了: ECK:  ▁Purple:  斑马线:  ▁cohort: 成大: Submit:  ▁Doctors: 洗漱:  奥林匹克:  ▁births:  ▁Arlington: 关照:  自己所:  以上内容:  ▁orphan:  ▁Evaluation:  ▁decoration:  ▁Sample: 或未:  字节跳动:  扶贫开发:  ▁storyline:  raising:  招聘对象:  ▁Hammer:  男孩子:  ▁snake:  gencies:  ▁Rapids:  ▁monop: 维稳: 鸭蛋:  亿元人民币: 干练:  不怎么:  当成了喜欢:  ▁hype:  就是最好的:  ▁urges: 地黄: 逃脱:  绝对的:  ▁Cisco: ecl:  资料的:  ▁lend:  党组副书记:  不负韶华:  ▁projection: otti: ardon:  到达现场:  保持中立: 回合: 者为: 调养:  去看看: 红叶: 雄厚: LAND:  治疗方法:  ▁infants:  要主动: 欠缺:  urgical: 置疑:  惠州市:  司法解释: 心有:  ▁Designs: JF: 缩减:  ativity: 对在:  巩固拓展:  ▁Arrest:  转载请注明出处:  村党支部书记: RR: 磨砺: 三尺: 夜景: 他很:  本科毕业生:  ▁intra: 雨花:  digital:  服务意识:  ▁Looks:  aterials: 迪拜: 赘肉:  第二款:  ▁warmer: 龙凤:  活动内容:  公元前: 建言: 自拔:  经研究:  ▁insane:  ▁Regulatory: 清北;  ▁petrol; 有幸;  刘先生;  工资的; rared;  在某些;  厚厚的;  ▁Aston; 倒挂 ;  ▁Volunteer ;  好成绩 ;  希望的 ;  ▁Distinguished ;  最爱的;  努力工作;  贫困县;  ▁cherry;  那些事; 手握; 多汁; 过瘾;  ropolis; 展出;  战略性; Fresh; 商洛; ▁va;  加工厂; '>; 有车; 难的;  ▁Whis ; 火龙!; 哔哩";  自信心#;  inosaur$;  ▁cottage%; 下楼&;  filename';  ▁survivor(;  岁那年);  你点的*; 做客+; 写成,; 刑罚-;  创新的.;  如果想要/;  ▁badge0;  ▁corridor1; 挪用2; 黄海3;  ▁educating4;  ▁Surrey5;  ▁rewarded6;  ▁staple7; 宣化8; Kelly9;  ▁Ober:;  ▁Scout;; 叹息<; 兰卡=; 有味>; ▁č?; 快餐@; 护师A;  ▁pillsB;  感兴趣的C; 的五D;  ▁ноE;  人为本F;  ▁testifiedG;  新天地H;  essionalI;  年夜饭J;  ▁hindK;  ▁dissertationL;  ▁springsM;  ▁emphasizeN; 息肉O; 载客P; THERQ;  ▁MapleR;  ▁DoingS;  ▁copiedT; 摒弃U; ▁PGAV;  ▁PrayerW; 活了X;  ▁DoneY; ▁zumZ; 新会[; 简便\; ▁ip];  角度来看^; yssey_;  hearted`;  ▁Businessesa; 摄取b; themedc;  总是会d;  行业内e;  违约金f; opiag;  ▁Ladiesh; 闯关i;  然后把j; 肝经k; 桂圆l; mentalm; 我跟n; 空心o;  ▁prisonerp;  检查组q;  ▁Lessonsr; āns; 偶遇t;  ▁relianceu; 议院v;  culturew;  事业运x;  思维方式y;  ▁chefsz;  ▁nominees{;  ▁disproportion|;  但不限于}; 比比~;  ▁Disability;  ▁negotiation; 制图; 杂乱;  发展趋势;  国外的;  ▁hooked;  ▁umbrella; 最深; 交了;  和家庭; esthes;  ▁pushes; wt;  麦当劳;  文化程度;  向社会公布;  ▁Frankfurt; 抢修;  北京日报;  ▁Jared; 钢材; '];; EMENT;  买了一;  推动了; ▁Tip; 同为;  第二人民医院;  ▁Pierce;  ▁Chocolate; 惊慌; ambi;  培养学生;  ▁Anders;  ▁spinning; onso; 老牌;  感觉很;  ▁Moment; 恭敬;  ▁Constitutional;  历史性;  爱因斯坦;  ▁horrific;  ▁unlawful;  ▁adul; 开元;  ▁Blind;  ▁Organic; 得体; 要吃; cards;  ▁incidence; 避暑; ▁TB; 我与;  期限内; Roman; acerb; 冰冷; 却说; 暂无; 送一; 土建; loo;  管理等;  ▁reside;  ▁parental; 却很;  居住的;  制度和;  建立在;  ▁revelation; 跌倒; 领奖; 用上; ODO;  老人家; 耀眼;  一旦出现;  不要因为; 要领;  ▁digging; yg; 驱蚊;  一年后;  非常大的;  manship;  ▁Pfizer;  优良传统;  ▁referenced; 保税;  ▁virtue; blood;  ▁Brunswick;  details; ▁DAY; 松动;  非必要不; 选考;  也不想;  ▁lender;  禄劝禄劝禄劝禄劝; 掉落;  就无法;  电视机; 代为;  买不起;  那么大;  ▁advising; 酒泉; LES; 奠基; 欣欣; 虚荣;  风险的; 内搭;  含答案; lore;  嘉年华; arna;  ▁stripped; 着火;  ▁styling;  动画片;  ▁Label;  ▁Farmers;  热烈的;  padding< ▁Von<  素质教育< anting<  ▁infer<  ▁Includes< Bern<  视觉效果<  ▁balloon< 饭前 <  ▁imposing <  经济效益 < 跑去 < Steven < 侵占< 陆丰<  ▁finalists<  教师队伍<  aundering<  ▁checkout<  市场上的< urous< 文联<  参赛选手<  Customer< 攻读< ishu< 家禽<  互联互通<  孩子学习<  前公示<  ▁practitioner<  ▁Allan < 绕城!< 马蹄"<  商业街#< 螺蛳$< 中生%< 拉你&< 摘帽'<  县教育局(< 禄口)<  作品中*< agher+< 心了,< 说来-<  你有什么.<  的儿童/< 万字0<  安排了1<  ▁throne2<  ▁fundamentally3<  汕头市4<  ▁beers5< 硕果6<  ▁Drum7< 畅游8<  ▁engra9<  综合分析:< ▁Pix;< Sales<<  及要求=< 成药>< 四溢?<  学习方式@<  解决好A< zymeB< ravelC<  releasedD< 大肆E<  大健康F<  ▁HitsG<  ▁BoroughH< 迅猛I<  采访时表示J<  参考价K<  类药物L<  可免费M< 倒数N<  最喜欢的O< 坟墓P< 代金Q< 船上R<  是根据S< ieleT<  ▁sandwichU<  ▁preparationsV< 问诊W< ▁bouX<  转多云Y<  生肖马Z<  ▁Sandra[<  ▁Reason\< 是与]<  ▁reversed^<  建成通车_< amar`<  的年轻人a< 署名b<  ▁superstarc<  呼吸道症状d< 跳槽e< ▁apif<  不能够g< 后有h< 排球i< 王俊j<  体能测试k<  characterl<  ▁jurisdictionsm<  市领导n< otono<  ▁borep< KGq< hider<  是一门s< 就职t< 装卸u<  ”,　v< ahaww<  ▁Fraudx< 成千y< 龙城z<  ▁beneficiaries{< MU|< Sol}< 往常~<  在一线<  ▁fulfilled< 抗病< 目睹<  ▁contests<  已经不是< detail<  ▁municipalities< 同理<  ▁laps<  新冠肺炎病例<  director<  ▁cholesterol< ▁Wo< 支书< beta< 人设<  大力弘扬<  ▁headquartered< 长廊<  ▁redund<  resistant< ▁AFC<  ochemistry<  不下去< 烤箱< 壁垒< 结实< 全系<  ▁Become<  ▁Ernest<  ▁Classes< 腊肉< 砂石< 鸟类<  房产网< 四天< 管的<  子宫肌瘤<  国家发改委< 中院< 和李< 事变<  当他们< 居高<  ▁balcony< 国强<  潍坊市<  注意安全<  Courtesy<  简历投递<  ▁Mate<  ▁quot<  Founder< 旅馆<  公开发<  去追求<  零售价格<  Running<  ▁Koch<  ▁Flower< jjjj<  ▁cafe<  ▁Volkswagen<  ▁multinational< 双城< 胖子< 为题< 两头< 回信< 比拼< uga<  那么在< 无前< 的在<  ▁tense< % 现将有关事项公告如下<  ▁Hybrid< 谨记<  ▁Caval<  中国在<  ▁aliens< enne< ▁owe< Safe<  催化剂<  密密麻麻<  ▁Bathroom<  ▁prolonged<  你认为< ▁Sau<  国家队< 没过<  ▁imprisonment< ции<  中草药<  动力电池<  志愿服务队<  ▁fried<  ▁belonged<  需要进行<  废弃物< 轮椅<  双方的< 要靠< 不慌< 扣分<  精益求精<  ▁shelters<  的网友< 离异<  卫生室<  ▁conqu<  申请材料<  ▁duck<  ▁clerk<  ▁anticipation< 产的<  对相关< baum< Laura< 放缓< 昂贵<  过很多<  审查调查< secure<  危险废物<  uggling<  ▁Chapman< 金钟=  主题党日活动= 三区=  ▁behave=  ▁offenders=  ▁бу=  ▁Prophet=  ▁metadata= LAN= ▁Sic = ار = 宣教 = 有房 =  ▁Wings =  看过来=  第一家=  最值得=  华中科技大学=  ▁Controller= Jane= finals=  新增疑似病例= 难求=  第二十=  ▁neighbours=  ▁Laud=  ▁sovereign= ▁kay=  ▁atomic= 搞得= 某人= ▁BG=  大学习 = 驻足!=  亲和力"=  ▁stun#= placed$= 到我%=  包头市&=  时时刻'=  改变的(=  林业局)=  ▁focal*= 去往+= 多伦,= 大喊-=  优秀奖.=  市融媒体中心/=  ▁Apparently0=  ▁proceeding1= 代代2=  重金属3=  全域旅游4= 州委5= 积木6= 交会7= Cand8= 附带9=  美味的:=  ▁Conversation;= ég<= VIS==  无意中>=  这两年?=  collection@=  学期的A=  信息港B=  ▁BrittC=  性关系D=  有所不同E= 一村F= ADEG=  ▁SlowH=  ▁stretchingI=  ▁worthwhileJ= 奔向K=  ▁SilvaL=  ▁proposesM= 听的N= 外教O=  于一体的P=  我们看到Q=  JapaneseR= 大财S=  安排的T=  生肖猴U=  ▁LindsayV= 爱美W=  统编版语文X= 好比Y= 月累Z= 相克[= 闵行\=  出生在]= ibo^=  ▁Spart_=  collect`=  发挥好a=  最常见b= 特聘c=  ▁Rodgersd= ▁cave=  ▁Requf=  利益的g= phantsh=  肾上腺i=  ▁unintj=  ▁Guatemalak=  都有可能l=  级旅游景区m= muchn=  ▁hassleo= Potp=  对未来q= 日积r=  历史悠久s= 补给t=  节目的u=  或多或少v=  ▁Gonew=  延缓衰老x=  ▁conductory=  ▁Dressz= 阴霾{= stru|=  ▁Cedar}= ▁Jal~=  先进典型= icos=  ▁twins=  ▁Santiago=  ▁Migration= faces=  ▁exhausted=  住宅小区=  ▁Kingston=  ▁acceleration= Bry= 你把=  课程中=  ▁strang= 之又= 拨付= 舍弃=  马上就要=  下定决心=  ▁Franch= 小雪= 过重=  ▁maneu= Upon= arag= 行贿= 强求= 扑灭= AZ=  向他们=  ▁adap=  ▁autonomy= 两室=  ▁Alger=  ▁Origin= ifs=  ▁Conflict=  增加值= 到人=  ▁Mercy=  ▁urgency=  本职工作=  发送到=  这几种=  ▁Curry= Xml=  高性能=  ▁Mald=  ▁compliant=  Christopher=  ▁vocabulary= monary=  ▁rugged=  评论区留言=  ▁hasht= 常人=  来临之际= 看他=  消费品=  站出来= Tool=  不知所=  ▁haul=  生不灭=  中国市场=  各项措施= 木斯= Ball=  ▁dominance= etric= staff= offset=  ▁inspires=  可以促进= 像我=  高清壁纸= ucket= ▁IA= 应酬=  建设银行= {}=  生肖龙= ▁Kaw=  ▁sanction= 解开= ▁zur= 大钱=  日晚间=  珠三角= ctx=  ▁Muse=  ▁asthma= 丙烯=  电子烟= 收视= 限流=  ▁Catholics= 再接= Paris=  生活保障=  ▁consolidation=  出去玩=  ▁Youtube= 尖叫= 幼师=  ▁rebels=  代言人=  更安全=  予以公示=  征收土地=  时间久了= 严把=  ▁melody=  咨询师= chu=  房屋详情= 精简=  还表示= HIV=  婚姻的=  ▁Crus=  ▁Choir= 容城= 关头=  ▁Paulo= Ready=  ▁Echo=  ▁knees= 那句=  ▁marsh>  ustainable>  popular>  ▁prolifer> 出镜>  中学业>  就很难> 爽口>  的改变> ortal >  ▁candle > ▁nap >  ▁insured > 测序 > 煮沸>  ▁Anch>  ▁Famous>  ▁Probably> 凉水> 镇静> spoken>  ▁observers>  ▁fals> 绝了>  租赁住房> 自选>  本平台将>  ▁Buhari>  ▁Inspector>  德云社>  ▁miracle>  什么东西>  ▁Scale > 带去!>  王一博">  气候变化#> 陆地$>  ▁Hungarian%>  主要内容&>  ▁beast'> 邪淫(> olor)>  ▁rooft*>  ▁Battery+>  ▁unacceptable,>  ▁Roma-> 最受.> 亲民/> ја0>  高强度1>  ▁goodbye2> 当她3>  度过了4> olith5> 鲜血6>  得天独7> 壁画8>  高级人民法院9> 相接:> ▁Ace;> 鱼儿<>  家里人=> 大打>> 路至?>  的阳光@>  但是现在A>  ▁sleekB>  ▁GazetteC> 凝胶D> 大为E> 诺如F> SupG> 气囊H>  时强调I> beenJ>  ▁FalconK> 穿刺L> ▁×M> ▁ozN> % 人力资源和社会保障厅O>  希望在P> lawsQ> 伟业R> 武功S>  发展需要T>  ▁crushedU> 步履V> 要到W>  稳步推进X> 牛市Y>  ▁KnownZ>  ▁podcasts[>  ▁probation\> 无心]>  国家电网^>  执业药师_> 山羊`>  关注度a>  初学者b>  ▁Monumentc> 出海d> ▁MARe> 与年f> 喜怒g>  ▁chunkh>  活动中心i>  命运共同体j>  ▁prosek>  ▁Georgetownl>  并不多m> 看透n> 光环o>  曾多次p>  考试总成绩q> 坐拥r>  工作计划s>  比较大的t> Sandu> borgv>  ▁Snyderw>  老人们x>  核酸检测阳性y>  重要的作用z>  ▁sadness{> ▁bog|>  工作餐}> 练就~>  理所当然> 预览> 宴请> sheng> 不值> 麻疹> Begin> 莱芜>  ▁cosmetic> omez>  敲诈勒索>  民族大学> Topics> 发觉>  调查显示> andem>  ▁Allison>  ▁Plymouth> 湘江>  考试内容>  ▁neut>  特此公告> 懒人> 软骨>  增长了>  不是说> STEM> ▁Wii>  卫生健康局> 筑梦> 伤感>  静静地> 沙拉>  一定不能> 共生> 发霉> 口才> 负压>  增值税专用发票> ▁PER> 小妹> 珍爱> ▁[">  ▁Rash>  ▁Maxwell> Dele>  原因就是>  counter>  ▁defic>  ▁coral>  ▁genome> Crit>  ▁handsome> 劝返> 口渴>  ▁stimulate>  ▁instructors>  支付的>  向上滚动> 他所>  ▁sprint>  ▁telecommunications> [[> 杨紫> 橙子> 剧集>  独生子女> Sad>  万公里> Volume>  的联系>  妇产科>  已经成为了>  ▁compression> 抛物> 航道> 苗木>  但是由于>  ▁Churchill> FW>  做事情>  现场的>  ▁manipulation> 暗沉> ▁Ski>  ▁Frost>  ▁suites>  ▁insists> 革新>  国家战略>  idelity>  ▁obsession> aisy>  车流量>  与法治> 蓝莓> 饱受> BAR>  开始啦> 易错> 马丁>  多一份> tyard>  一一一一一一一一> 内蒙>  不得了>  生肖猪> iances> 小鸟> PCR>  utterstock> 月度> 牧师>  abstract>  ▁Monitoring> 乘务> Flo>  ▁Subscribe> 情深>  理解为>  技术员>  ▁Bulletin> 打完> 朱某> 上不> 薏米>  却不知道>  本地确诊病例>  冬天的>  绝不能? 交出?  就显得? Native? 救济?  会让人?  这是个?  localhost?  ▁Lancaster?  Experience ?  你不知道 ?  依次递补 ?  理财知识 ? xj ? ▁VAL? 产销?  时时刻刻? 彼岸? 落入?  ▁Winnipeg? 邮轮? Mu?  消化吸收? 构件? 抚恤?  第二条?  风景名胜?  积极向上? 友善?  整个过程?  essential?  ▁maternal? 只看 ?  划分为!? 重现"? eye#?  的象征$?  提供优质%?  ▁collar&? mL'?  了一段时间(?  检查站)? 首开*?  ▁Maggie+? 救了,? 曾有-?  罕见的.? Switch/?  校外培训0?  质量管理1?  ▁entirety2?  新疆维吾尔3?  ▁habitats4?  这部电影5? 将领6? 每每7? 五金8?  ▁weighing9? 奴隶:? 谈起;?  Commerce? 诚恳?? 赵本@?  的一件事A?  componentsB? 再将C?  体育运动D?  优秀共产党员E?  ▁EuropaF? 短线G?  ▁visualsH?  ▁humidityI? ▁CAPJ?  ▁wokeK? 二号L? 帐号M?  ▁PiratesN?  ▁staffingO?  ▁PrettyP?  balanceQ?  ▁KuwaitR?  另一位S? 皆有T? 豆类U? 买个V? 用眼W? 男童X? 盼望Y? 小肠Z?  把握住[?  ▁Option\? 回路]?  多层次^?  ▁ribbon_?  ▁Natalie`? Carola? 在南b? ▁enlc?  基因组d?  并取得e? 你很f? 虫子g?  了一把h?  当前的i? oquej? youngk? 接过l?  世界各地m? Civiln? 屠杀o? 苍白p? 落脚q? 个班r? 莱坞s?  价格在t?  但这些u?  用行动v?  ▁Meanw? 防风x?  中国画y? Monthz?  ▁improper{? 葱姜|? 木头}? Must~?  作者所有? 卖掉? 气泡? 还与?  ▁Invent?  资质的?  日积月累?  ▁Timber?  ▁revital?  老年痴呆? 出世?  语言文字?  ▁Hearing?  ▁translates? prep? 丽莎? Role?  ▁Example?  ▁distributor? 植被?  ▁envelope? 如因?  ▁weed? Bat? adj? 外貌? intech?  ▁prince? 他要? 方圆?  ▁Flynn? 出借?  买买买? urdue? 韧带? ebe? 年里? 金斯?  中国传统? 黄陂?  级以上? 熬制? 立业?  ▁irres?  ▁Resume?  ▁Telegraph? 淡泊? won?  以确保?  一起学习?  安全有序? 述职?  实体店?  很多人在?  ▁Auburn? temp?  ▁Zimmer? dollar?  ▁slate? 当家?  ▁restoring? Hong?  ▁fireworks? 镜片?  ▁majors?  ▁ignoring?  ▁renovated?  犯罪分子?  ▁modular? 绚烂?  诊疗方案? 荣幸? 敬佩? 车友?  货币政策?  ▁runway?  ▁servant? 代购? 哈酒? 酒的? ramid? lux? 四中? 回流? 东街? 乘法?  ▁photographic? 本就?  血液中?  语言表达? 安国? 罪的? 家装? 心眼? 现价? 醇厚?  的问候?  的第一步? кт?  很长一段时间?  无障碍?  ▁Salem? 王的? 预判? fg? 构思?  反映出? eros?  ▁exagger?  ▁Norton? 悦读?  投融资?  只不过是?  ▁expansive? 比尔?  的部位?  ▁Hutch?  ▁equitable? 无几? Bytes? 纲目?  仪式上?  loading? 打分? 江镇? Thus? 美颜? anan?  ▁dashboard? 就到@  等技术@  ▁Sacred@ 出息@ 钢丝@  最强的@  ▁Mell@  ▁municipality@  ▁representations@  ▁tailor @  ▁ventures @  ▁perceptions @  解除劳动合同 @  ▁Rings @  ▁Preston@ 嘴角@ toire@ 成全@  总而言之@  ▁parked@ 再创@ 稻田@  ▁Castro@  ▁merged@ 怨恨@  学习平台@  不会被@  ▁Witch@  ▁drift@  ▁airplane@ ▁BP@ Stars@ brahim @  作风建设!@  违法犯罪活动"@  ▁insulin#@ 乐山$@  切实做到%@  市面上的&@ 之急'@ 姊妹(@  ▁Improvement)@ 参战*@  事业编+@  ▁Planned,@ loaded-@ 不愁.@ 山口/@  为主体0@  ruitment1@  柳州市2@  ▁Numbers3@  ▁Copenhagen4@  服务商5@ ▁IMP6@  ▁Tate7@ rimp8@  十二条9@ 创文:@ 酱汁;@  ▁Trin<@  ▁Divine=@  ▁greens>@ iguous?@  ▁Featuring@@ 参演A@ 巨星B@ EasyC@  ▁ButtonD@ 爱了E@ BasicF@  比较少G@ ▁dugH@  ouncillorI@  ▁ImmunJ@ FINK@ 一至L@ 总署M@ NOWN@  成功率O@ eenthP@  ▁StrikeQ@  ▁MatthewsR@  开学第一课S@  ▁SelectionT@ 界定U@  项目位于V@  CongratulationsW@ zzaX@  投标人Y@  有关系Z@  ▁thanked[@  ▁reductions\@  ▁LIVE]@ feel^@  ▁palette_@  卡拉胶`@  项目为a@ 按键b@  ▁unpredictablec@  和组织d@  ▁Classicse@ 句句f@ uchig@ ▁Rewh@ 心律i@ 杀伤j@ 师长k@ 穿插l@ ▁Spym@  ▁procn@ 将他o@  ▁Regardlessp@ 心痛q@  教育家r@ 书目s@  的花朵t@ 翻阅u@ 就给v@  也因为w@  ▁sensiblex@ 中签y@ 鸳鸯z@ ▁Wet{@  ▁reboot|@ 群主}@ 赞誉~@  关于进一步加强@  或暗示@  接到报警@  ▁Molly@  心理素质@  转载目的在于@  ▁dispat@ score@ 午休@ 积电@  一年一@ ibe@  报告的@  ▁volatile@ 群岛@  研究显示@  ▁wast@ 学法@ xd@ 海湾@ 迈入@  ▁это@ 养牛@ elman@  旅游区@  团队协作@ 凝结@  ▁lungs@  ▁processors@  大大小@  ▁Said@  的一道@ inee@ metric@ agi@  ▁Indonesian@ 嗅觉@ ungs@ 楼宇@ 郊区@ iso@  ▁insulation@  乌鲁木齐市@ 薪水@ estly@ 评测@  才明白@ Africa@  ▁Tests@ IPT@  Executive@  ricting@  查漏补缺@  责任单位@  ▁Login@  ▁Ка@  学习中@ 白露@  比自己@  集团股份有限公司@  ▁intercept@ 腰酸@  承担责任@  的人是@  费用由@  ▁surreal@ 别提@ 辅料@  ▁Cardinal@  在这次@ 疤痕@  增添了@  只要我们@  Whatever@  印发的@ ▁POL@  女主角@  ▁Wish@  ▁mighty@  ▁iPod@ 对已@  ▁scare@  ▁microwave@ гу@ 一派@ 粗粮@ 乞丐@ 路由@  国家统计局@  上个月@ 他有@ 黄土@  ▁roofing@  是用来@ 如您@  ▁puzzles@  ▁stretched@ 心悸@  和使命@  强烈推荐@ 解冻@  ▁Guer@ 奔腾@ 游行@  Furthermore@ ftime@ 高举@ Gary@  ▁Erik@  的工具@  ▁validity@ 断奶@ 上将@ junit@  给你一个@ 失职@ 松鼠@  打麻将@ Iter@ cular@ 院落@  的成分@ dens@ brance@ 增压@  就回房间@  ▁reconciliation@ tops@  主营业务@  ▁ExceptA  理性的A  有机肥A 甘蔗A 重拳A  prisinglyA 有苦A  特点是A  的岁月A  ▁Calcul A  ▁syntax A  ▁coupons A 得来 A Joy A svgA  ▁microphoneA  经营场所A  ▁tomatoA 东经A  就业岗位A playedA RubA 而喻A 之歌A 最短A 京城A  自己的力量A 李家A lliA 在西A  创造出A  存在感A 体格 A  ▁Answer!A 乐乐"A  就如同#A 竹林$A 被罚%A  ▁commercially&A 劳作'A 尾号(A  高校的)A clair*A €™+A iya,A ▁Thu-A  ▁Investments.A 新药/A  等手段0A 保单1A 刺绣2A  ▁confined3A 从那4A  调整后5A 拿地6A  合格者7A  ▁Griffith8A 装机9A 户晓:A 松江;A  ▁violating    A 查出?A  让生活@A 周星AA  深入浅BA  ▁sprawCA uatingDA 宝玉EA  ▁charismFA NHGA  井冈山HA  ▁dareIA  ▁AboveJA  ▁synthesKA 杀猪LA  电视电话MA  ▁DartNA  请及时与我们联系OA 四驱PA 小到QA 方子RA 管住SA  ▁archaeologicalTA 雇佣UA ianneVA 幼升WA 汤匙XA  ▁adequatelyYA міZA 与前[A 海滩\A 网校]A  ▁altitude^A  的状况_A  ▁Approach`A ▁levaA  不生不灭bA 零下cA 鼠标dA ;B  赔偿责任?B  ▁unconscious@B  ▁compassionateAB  的使命BB ▁DESCB  监督电话DB 展馆EB  各行业FB 差额GB  ▁stripsHB 野味IB ▁ShiJB  ▁ShawnKB OmLB 马尾MB  淋巴结NB ▁lobOB  ▁bentPB  primaryQB  ▁vocalistRB 止损SB  分区分TB 环城UB  怎么了VB  抓住了WB 建平XB  保护法YB  努力实现ZB  不完全统计[B 姨妈\B 新县]B  ▁Lyon^B  菩提萨_B Lake`B  ▁consultancyaB AlanbB  经济区cB  总有一天dB  新增治愈出院病例eB 放得fB  一个女孩gB  ▁SensehB ▁vagiB  CapitaljB  万余人kB  属鸡人lB KImB 路虎nB  合作协议oB  ▁lyricpB 气愤qB  提质增效rB 改动sB irentB mathbbuB 柜台vB  ▁insurerswB 五项xB 稀少yB  经济带zB  镇党委{B  政策规定|B 熏陶}B  checked~B 月考B 焕然B  聪明的B  letcherB  ▁RhodesB 山体B  不仅在B  的最好B  产业结构B ferredB 莅临B  有三个B  的那个人B QCB 政局B  ▁aidsB  人力资源管理B 蕾丝B  中国铁路B  相关资料B 和张B  实质性B  生态系统B 车程B  ▁labsB 反驳B  千年的B  ▁mentorsB 段的B 防洪B  骨子里B  工作地址B 省事B 请点B  ▁templesB 惨烈B  largestB  ▁LilyB  ▁exquisiteB  微视频B  的乐趣B suppB  政策解读B 毁了B  特别喜欢B  ▁NobodyB ▁BeaB  新兴产业B  不按规定B  ▁aggressionB  ▁accumulatedB 使劲B 病灶B  掌握的B ocomB  经常在B  ▁blowingB 餐补B 花在B  继承人B ▁ACLB  销售人员B 病魔B 四项B vicB LargeB  毕竟是B ▁uitB 商界B ▁OleB TIESB  的身材B  ▁leaksB  ▁trajectoryB  自然界B 双减B 她对B  渐渐地B  免费培训B  人员密集场所B 闹钟B  AdvancedB  ▁traumaticB 元帅B 划片B 都以B 玩意B 笑笑B 省力B 小众B 小子B  可靠的B  ▁teachingsB 神兽B буB 争相B  可以进行B  ▁violateB  ▁underwentB  全面排查B anicalB ▁FenB  ▁HyundaiB heedB 一锅B 一应B CutB 操盘B  ▁dismantB  paralleledB 均是B 郁闷B 粘稠B ▁MüB  喻户晓B  菏泽市B porterB  ▁applesB 何以B  establishedB  的人格B aurusB  ▁phenomenalB  有部分B  长期使用B  专项整治行动B agentB ypseB  ▁SantosB 开示C  也可以在C  ▁NielC  ▁subwayC 唾液C  风景线C MexicoC PATHC TraceC 隐秘 C  ▁Kerala C  ▁Streets C " 房地产开发有限公司 C  ▁poets C 鹦鹉C  CollectC  有关事宜C  问题来了C  ▁homageC  ▁GermansC 急躁C  全国性C  最有效的C 管局C  看不起C  休闲娱乐C ▁NVC  模式的C  ▁HamburgC  ▁apparelC  ▁ChangingC  为有效C ▁Dot C  相信自己!C 二类"C 申购#C  爱尔兰$C  人以上%C Wait&C  ▁pleas'C  ▁Reasons(C 旁观)C 蓝牙*C  方法是+C 每家,C  ▁Proceedings-C  ▁picturesque.C  也有着/C ICH0C  Therefore1C  摄氏度2C 赶超3C oji4C  ▁furnish5C  会自动6C  长长的7C 害羞8C  如果出现9C 国宝:C 毛细;C MultiC 句型?C  为公司@C  ▁clustersAC  inementBC  ▁fascinatedCC  ▁SternDC  内容及图片均EC  结果的FC  还有哪些GC 低位HC  自然人IC  运动鞋JC 前去KC  还有就是LC ▁OrtMC  结婚后NC DoctorOC  ▁apparPC  ▁denomQC  WritingRC  ▁grammarSC  RetrievedTC 缺钱UC 脑筋VC  ▁EssentialWC 南下XC 思义YC  国家对ZC erns[C 步枪\C  增长的]C Accept^C  ▁advertisement_C 俯卧`C  散发出aC 痛哭bC  台积电cC  ▁DynamicsdC 境地eC embfC kregC  经验分享hC 展区iC  命运的jC 躲在kC ▁CWlC  观世音mC ICOnC  要结合oC  技术有限公司pC 一发qC 一架rC 大口sC 考编tC 二路uC 若无vC  喝一杯wC  资本主义xC 经文yC  司法部zC  ▁Types{C 访友|C  ▁Tenn}C  ▁splash~C  预示着C  动员大会C 就对C SeanC  ▁boroughC  ClassesC  尽量选择C  现场确认C  ▁pentC ~(C  非常简单C  是人生C  过往车辆C  ▁bunkC 但因C 寥寥C 气流C  处三年C ollahC  ▁betsC  关注后C  民族才能C  塔利班C 定州C 盆地C ALLYC  不得报考C  ▁shootingsC  ▁RelationshipC  职业资格证书C  ▁juvenileC ▁masC  获取更多C  ▁brochC panelC 弹出C  各学科C  ▁jetsC 史记C 朝南C  被别人C FatherC  ▁establishmentsC  商丘市C 音符C ▁NFTC  帮助学生C  osphereC 皆为C 能吃C 便携C  即是空C  ▁DualC  ▁relayC  ▁cocktailsC  IntroductionC  ▁feastC 被害C  ▁whaleC  ▁compulsC  拉法基C tmC  作品集C  肉眼可见C secretC  成功举办C  ▁TracyC  原定于C  ▁cubicC 娘家C  ▁neglectedC naireC 吉安C 寒凉C 有病C  成本的C  紫色的C 三环C 审定C sjC 千百C DriverC ACAC  主干道C  前提是C  从而达到C  ▁profitabilityC 短袖C 走入C 高调C ▁XXC  考试大纲C  ▁jungleC  ▁PurchaseC 史料C GERC ▁jsC 流言C 比方C 生发C 腥味C  第一轮C  疫情就是命令C 金针C  和老师C  ▁AgendaC 貔貅C 冰冻C  有重大C 严苛C  满足女C singleC 路程C  华尔街C  ▁absorbedC ▁GCC 是会C 海滨C  见义勇C  ▁BrasC ▁GETC 皮下C zonC 献策C  眼前一亮C  ▁embassyC  转变为C  ▁brakesC  ▁ShepherdC  都十分D  小姑娘D  ▁GoreD  ▁disgustD  ▁thereofD 跨度D  　)　D  ▁nailsD  白血病D  ▁Doors D  住房和城乡建设局 D  充电桩 D  ▁Aurora D  ▁Offers D 一九D  好日子D  联系了D  市场监督管理D  ▁InterviewsD 之说D ▁RisD  ▁rockyD 宣城D  最开始D  非全日制D WordsD  每一种D ObamaD  ▁FinnishD 不料D 三部D 农耕D  ▁photographed D 卫国!D  忻州市"D  ▁Drugs#D  又不失$D  ▁haunted%D pn&D 柴胡'D 母女(D 靠山)D  才是真正的*D  ▁alphabet+D  ▁Mustang,D  ▁unhappy-D  ▁negotiated.D  ▁illustrates/D 威廉0D 明火1D 温热2D  点击右上角3D 年卡4D  对于那些5D  ▁dataset6D  有明显7D 偷懒8D 附上9D  动植物:D  ▁suburb;D  ▁CannabisD  象征着?D advert@D  强调了AD  ▁BoostBD 车流CD  ▁noticeableDD 葛根ED 流血FD 烹调GD  在英国HD ducingID  ▁richestJD  ographersKD 二甲LD  大小事MD  自己喜欢ND 热恋OD 淡雅PD 订立QD  ▁employingRD 查清SD  博士生导师TD  ▁BowieUD  的疾病VD  专题会议WD  ▁peaksXD  提出申请YD  ▁smokZD 舅舅[D  滨海新区\D  ▁weaknesses]D  的认知^D 挥手_D ▁SIM`D  而这种aD 扑面bD 脱水cD  ▁excursdD  ▁certaintyeD 很差fD  的独特gD  ▁battinghD 结业iD  ParametersjD ▁DeekD  迄今为止lD leighmD  executenD  新局面oD 该村pD  设置为qD 热火rD  挖掘机sD  国家法律法规tD  ▁LutheranuD arethvD  ▁EbolawD  icularlyxD  就一直yD 同日zD  机械化{D  ▁Hull|D 取名}D 吞噬~D  ▁KathyD  ▁InstitutesD 瞩目D  将进行D  在别人D ByteD 加收D  ▁lecturerD  佳木斯D ’)D 交错D 反腐D 搜寻D  医疗资源D  文化传媒D  ▁JongD  ▁MeaningD  ▁WillieD 现为D 含糖D 普宁D CharlD  ▁gemsD  批准的D 花苞D optersD  ▁AdminD суD  ▁ratsD 施教D  行程卡D majorD  你想要的D  ▁negativelyD  ▁pertainingD uccD aiserD  国际合作D  ▁showersD JJD 顾及D ▁WITD 上好D 说课D  不放松D  额外的D garsD layerD  ▁highsD ynnD  服务水平D  ▁drasticallyD  显示出D  等多项D  当事人的D 盖茨D  经得起D BreakD  ▁expiredD  ▁compatibilityD 举起D 待定D 不透D 之功D 的正D ▁bruD  世界各国D FAULTD  教育体育局D 奶酪D 扶手D  ▁picsD 指着D  中央空调D  出院病例D 犬只D  在去年D ▁HepD 人无D  clamationD 不衰D  jectionD OFFD  ▁GretD ynthiaD 比分D  代金券D ▁KotD  不离不弃D  对话框回复D  加起来D  马斯克D itousD  的梦想D  那些事儿D 点钱D 网申D  色即是空D andumD  想起了D  镇人民政府D 留着D  西海岸D ▁ampD  相当的D CartD  ▁DumpD 纸张D  ▁upwardD  ▁benefitedD 门类D  公司和D 拍出D figureD  总有一D  顺风车D  ▁JoeyD  会议由D 工种D  猜你还D  看到一个D  ▁cracksD  ▁famouslyD ▁baD 枝头D  说起来D 拍的D 遥遥D  全体师生D SevenE  ▁gardeningE  并按照E CathE  发展中心E  今天我们就E  ▁depthsE 勤务E  ▁shootsE 背心 E  的地步 E ▁sys E  三伏天 E  boarding E tipE  根本无法E  一生中E  ▁MegaE 棋牌E 鸡翅E  和父母E 点是E  海口市E etchesE  ▁telecomE  ▁fundraiserE  ▁WitnessE 独角E  ▁fertilityE 尼尔E  ▁nastyE  ▁interrogE Coming E  我以为!E 太小"E  激发了#E  handler$E 志向%E  垃圾疯狂&E 裂缝'E _____(E 请示)E 黄牛*E  ▁poured+E  ▁По,E 傲慢-E 变美.E  税前扣除/E 节水0E 西双1E stri2E 悉心3E 溶性4E  带着孩子5E  ▁begg6E 压根7E 该局8E  密切相关9E  ▁initiate:E  可以向;E  老和尚E  元以下罚款?E 似水@E 味觉AE 通航BE  代理商CE  第四步DE  roducingEE  执行官FE  的客户GE  ▁BrunoHE  ▁HugoIE  ▁mythsJE 结冰KE  中长期LE  时间在ME  ▁miscNE 冷落OE ewoodPE  并没有什么QE  RuntimeRE  见义勇为SE  partialTE  ▁harborUE  会变得VE  的笑容WE  ▁percussionXE  ▁InputYE  觉得不错ZE 数次[E 全屋\E Niger]E  ▁elimination^E  中到大雨_E 增减`E 核销aE  ▁JahrbE  ▁VaccinecE  在校期间dE ammadeE 机理fE  ▁pixelgE 副本hE ringsiE  乃至无jE 和气kE 东华lE  ▁honemE ′~nE 吸食oE  沉浸式pE mineqE ColumbrE  ▁CoastalsE  ▁loweredtE  ▁ContinentaluE  称自己vE % 累计追踪到密切接触者wE 烧伤xE  ▁CoursesyE  ▁mitigationzE  也是中国{E  研究部署|E  ▁Hawaiian}E  医院进行~E 较轻E bandsE  NaturalE 伪劣E 车次E packedE  storageE 远比E 并报E  个乡镇E 毁掉E  资金的E  ▁smarterE  ▁ObservatoryE  才能真正E  ▁KarnE ▁TajE  初中英语E  representedE  生肖狗E ▁AchE MountE konE  亚健康E  安全工作E  ▁RoverE  ▁SafariE 因公E 脸型E  清楚了E ▁INDE  ▁obscureE  ▁spellingE  行驶至E  很快就会E  参与的E 明理E  老年人的E boostE  其他事项E  ▁JupiterE 愧疚E takingE LBE HandleE  CouncilE  ▁buildersE 直销E  我们家E  亦复如是E  ▁equalsE  ▁ManitobaE ▁TME ucingE  ▁jewelleryE  ▁councilsE  ▁optimizedE  信息服务E 查证E ikanE  防控要求E  ▁rentedE  哪一种E  亲子关系E  ▁laughsE 白术E heitE ▁BesE  assemblyE BrookE  道路运输E 起舞E  后勤保障E  销售收入E  ▁HollowE 雪糕E  就业服务E gebraE daqE  开通了E  ▁OrdersE  应该在E  的眼睛E  一群人E  国土空间E  ▁strivesE  ▁cigarettesE amazonE  也变得E 择业E  我感觉E  革命的E PennE  名单公布E 大年E  那时的E ▁jedE  ▁abusesE 下雪E  买不到E  ▁strollE  的研发E  这可是E MemoryE 每期E 心与E  设备和E rrrrE ScopeE ocracyE  iatricsE 改名E 代人E  ▁CreatingE  可能会导致E 富民E  开发者E PBE  岗位表E  新生活E  ▁DearE  ▁phenomenaE  十三届E  成立以来E BrandE pleaseE 全款E 郴州E  及时调整E 瑞安F  新变化F 进气F  也是个F GrantF breadF  JacksonF  ▁geometryF 得像F 空相 F ▁Ez F ۥۛ F hig F 枇杷 F  及时处理F  ▁RainbowF 松软F  哈佛大学F  ▁whalesF õesF  灵魂的F  月收入F  告知书F 大发F  销售价格F ▁QUF  还有些F otswF  周年大会F hertyF FlowF  distanceF  ▁openings F  Understanding!F 院所"F  他们将#F sets$F  ▁brokers%F 守候&F 河村'F  营养师(F  ▁weighs)F  Professional*F 钜惠+F  百年华诞,F  ▁deliberate-F  有什么区别.F 圣人/F 大汗0F  ▁memo1F 每名2F  最终的3F preme4F 黄酮5F  企业复工6F  毛主席农村7F  ▁Debut8F  最高人民法院关于9F  只能是:F  ▁Professionals;F 撤回F  ▁Combat?F  ▁Senators@F 污渍AF  ▁settlingBF 介休CF 抱歉DF DISEF  ↓↓↓↓FF  ▁wardrobeGF  ▁PasoHF  集中隔离点IF 单体JF PurKF  使用时LF  ▁JohnstonMF  哈萨克NF  ▁ExportOF  ▁homelandPF kieQF  大暴雨RF  五蕴皆空SF facingTF ildaUF 沉睡VF  周一至周五WF 发呆XF asketsYF 高通ZF  走的路[F 自燃\F Brazil]F  ocalypse^F 重叠_F  丰台区`F  五一假期aF VisionbF  ulatorscF 不紊dF 山洪eF 牛皮fF  以上为gF  ▁overseeshF 高人iF ▁(#jF  ▁endurekF  ▁DucklF  ▁observingmF 小狗nF  很少有人oF 学识pF 栗子qF  保障了rF  提供帮助sF 范儿tF  的关键时期uF 千山vF  都值得wF SOURxF eanoryF  依般若波罗蜜多zF  ▁succession{F  接到了|F 地表}F  ▁Schmidt~F  很高兴F  我们没有F 婚外F 环游F linerF geraldF  ▁handmadeF 怕冷F 标价F 至极F osaursF 变种F 远县F  在身边F  被依法F 单向F  VERSIONF  漳州市F 如荼F  大学排名F  ▁stintF  byterianF  依般若波罗蜜多故F  ▁supernaturalF ramerF istineF  ▁genocideF 深感F ▁GigF  世界第一F  请点这里F ▁PBSF  ▁RapidF :¥F 止于F 甄别F  EnglandF 归宿F  ▁turkeyF  建筑的F  的革命F  判决书F orporF 　*F  ▁professionalismF  ▁McKinF 丹田F 暴食F  不自觉F  聘用的F  ▁sulfF  identifiedF  新概念F 年就F  口罩的F KristF  医院就诊F  ▁rigidF 慈溪F  的好坏F 路基F  试用期满F  ▁sharplyF  危化品F  在国外F  您可以F  早安心语F  掉以轻F  指定地点F athamF  ▁gravelF  ▁insiderF KNF 五角F  在同一F 高超F  ▁headphonesF  ▁austF  ▁comparisonsF  原出处F  协同发展F 百余F ▁StaF uliarF 例为F  不完的F  ▁franklyF 切莫F  的水分F  unsignedF  ▁ContextF  ▁grappF  ▁assumesF  ▁supremeF eyedF  科研人员F  ▁GonzalezF 敦促F berraF  ▁garneredF  ▁hypothesisF 泊位F  ▁CrazyF  合同的F  中国国际F  这么大的F  ▁cakesF  ▁flamesF  ▁insistF 括号F  应根据F  简简单F  ▁thrustF  ▁NordicF  觉得好看F  ▁mosqueF 状的F ulsaF ▁CMSF  ▁loneF  青年志愿者F  建筑师F  有更多F 写好F  的一员F ▁ICEF  ▁ChennaiF 逗留F  十二星座F  ▁SpursG  ▁crushG  组织召开G 刘涛G 自愈G  笑着说G  排出体外G 牵着G 网店G  ▁Edgar G blank G 紧要 G Ford G  最基本的 G  最新资讯G  晋城市G 中指G 代步G 到货G  在一起的G  治疗效果G  emiconductorG 天窗G  风口浪尖G  要来了G 下文G  的重要举措G  TransformG bardG studyG  ▁stalG 英镑G  为全国 G csv!G Visual"G marked#G  规律的$G  明示或暗示%G  ▁crunch&G  ▁Armenian'G 再不(G  关键时期)G  三季度*G  沉重的+G phal,G illas-G 买一.G  ▁closures/G 延展0G  在校学生1G  工业区2G ▁ech3G  消化系统4G punk5G 礼服6G Lady7G  ▁researched8G  ▁volleyball9G 病历:G pound;G  ▁tumorsG 园内?G  文章的@G  上新台阶AG  ▁ProblemsBG 字幕CG 降雪DG 卖到EG  ▁plasticsFG 喜鹊GG  活下去HG ▁GapIG  食欲不振JG  JessicaKG  ▁paradiseLG  ▁questionableMG  空即是NG  ▁tightlyOG 佛说PG  营养不良QG  火车票RG  ▁MonacoSG 之法TG  答题卡UG 同名VG 体彩WG 茶的XG  ▁glowYG QueueZG  driving[G  ▁nationals\G  ▁Irving]G  ▁Acceler^G  联席会议_G  ▁teammate`G  各式各aG  联系微信bG  新发展阶段cG 散户dG 流氓eG  不知道大家fG 胃痛gG  表达对hG  省委党校iG  ▁DrinkjG 兽药kG 肥牛lG 虚寒mG  不断增加nG 一格oG  ▁BellepG 保全qG 选秀rG  手卫生sG ▁PREtG  ▁biblicaluG 肾气vG  不仅会wG htraxG 卖货yG 哈密zG  不参加{G  交通秩序|G  ▁Broncos}G 宗地~G PartyG  ▁shiningG 福鼎G  ▁cautiousG  ▁electrG  ▁neuralG 认准G 高热G  爱一个人G ▁XLG  也不好G  发展方向G  形成一个G  西方国家G asonicG  ▁SuperintendentG 焦急G  如需转载G 皮卡G  ▁MatrixG  ▁CopperG  ▁ReleasedG  ▁sexyG 俯瞰G 九州G  用清水G  ▁EtherG  ▁BoulderG 减产G ▁ZurG PsychG  CharacterG  ▁CatalogueG 乡下G 人造G 较弱G  躺在床上G 口区G 林肯G  注册会计师G bathG  国家一级G  ▁beanG 角逐G  请与我们联系G  ▁excludeG " 人力资源社会保障部G  ▁ResistanceG  PoliticsG 则在G ▁svG  视网膜G  ▁VernonG  ▁lackedG 古迹G  后顾之忧G  GenericG 尖锐G  个方面G BangG leaderG  本文系G  今年上半年G  ▁kindaG aqG coalG 充裕G 党恩G 水管G  上海的G ombiesG 偏离G  由学校G umbsG SweetG  ▁GarrettG 概述G  ▁restingG 偏好G 酷暑G  公共设施G 奖牌G 三文G 阐释G  简简单单G AntiG  ▁safeguardG 路子G 而从G JoshG 内敛G 顾名G 写到G  在网络G  很不错G  酒精含量G  ArgumentG  ▁🙂G  闭环管理G  ▁FactorG 马龙G ECTIONG 费率G ──G gelG  这么好G  人际交往G  ▁HolidaysG  全球的G  ▁PumpG  的人际G  ▁porchG 我一G  ▁ReadersG  ▁lithiumG  这种情况下G  ▁InstallationG 热巴G  大范围G  植物油G  不得转载G 宝山G 写信G 国标G 月在G ▁MauG  需注意G  ▁whitesG  ▁textileG 民进H 犀利H  大家要H  都离不开H  如果你觉得H  ractionsH  ▁BeginningH  ▁smilesH hubH 张伯 H 我这 H  女职工 H  ▁Improve H 可信 H  ▁EpsteinH  的相关规定H  ▁cooperateH  能够帮助H 捐助H  接班人H 悲惨H 报国H  ▁UnknownH  ▁HernandezH 米尔H BibliH  ▁comparesH  客服电话H  publishedH 侦探H 已满H 古树H  ▁ordinance H  ▁Apost!H  ▁onion"H  ▁landlord#H 关火$H  信息采集%H HU&H  加装电梯'H 无语(H 案号)H ▁XP*H ་བ+H 点火,H  去哪里-H  最佳的.H  白萝卜/H  ▁textures0H  ▁Hello1H 外滩2H ▁ú3H  大会堂4H  ▁vitamins5H  设区市6H  ▁Ministers7H 哪有8H Pen9H ▁zoo:H  ▁quiz;H  ▁surplusH  月经不调?H  ▁screened@H  分子的AH  咳嗽等症状BH  ▁depictsCH />DH 亳州EH 骨架FH  淮安市GH  项目负责人HH  ▁judgingIH DuJH 下期KH 全资LH 可按MH  不在于NH 格子OH 求生PH  陈某某QH 诡异RH adalSH  ▁handicTH ۛ۟ۡUH  等特点VH  最大功率WH  在她的XH  失败的YH  ▁deanZH  ▁loos[H  ▁Devils\H  ▁disreg]H "}^H  ▁daunting_H  properties`H  ▁EnterprisesaH % 境外输入无症状感染者bH 近来cH ARMdH DigeH  拟聘用fH  新京报gH  ▁lesbianhH 湖水iH  ▁JailjH 闭上kH staylH 凌云mH 出乎nH  ReferenoH jmpH 打入qH  平顶山rH  的产生sH KidstH  ▁KardashianuH ▁DUIvH  ▁combowH  ▁eveningsxH  无死角yH  提升自己zH 酒水{H 倒地|H isex}H Syn~H  第六届H  一经发现H  开发建设H  一夜之间H  ▁JudeH  内容的准确性H  ▁continentsH  ▁discomfortH  ▁inaugurationH  ▁DunnH  房屋租售H 脚底H  ▁inconvenH 稽查H  的经营H ledonH  生活上H  劳动模范H  ▁CorrespondH  ▁GeographicH 轻柔H ▁TayH  紧密结合H  ▁advisesH  ▁touredH 自已H 积淀H 之交H  不符合规定H veringH ilonH 杰克H  ▁pretendH  方设法H AlwaysH GoldenH  ▁nitrogenH  发生事故H sharedH ECDH 国货H  肚子里H  通过率H 江汉H  ▁reinforcedH :$H 盆底H 选一H riqueH 周到H ранH  ▁grillH  ▁technicianH 挂职H  如何让H  对美国H  ▁RavensH 如梦H ▁KabH 红木H 几下H 静电H tianH  管理人H  ▁orientedH  服务群众H ▁lbH  各有关部门H 平底H 理清H megH  试一试H  已经超过H CleanH  ▁inducedH 遥控H  ▁rebelH  ▁offseasonH  党总支书记H  面试官H 到访H  ▁WilliamsonH  党中央决策部署H  ▁endeavorH  ▁BroadcastingH  ▁PattersonH 首歌H ryslerH  ▁DeborH IronH ▁BUTH  庚子年H  项目名称H  的节奏H  竣工验收H itheH  ▁meticH  做好疫情防控工作H iliaryH  ▁ThinkingH 步道H  并获得H 贮存H  在下面H  ▁FightingH ▁chiH ▁deiH  本文内容及图片均H  范围为H  中华文化H  ▁multidH 随性H  新增确诊H  ▁intimidH 财会H 写意H  thoughtH 边疆H  ▁implantH oosH  共同富裕H ituresH 法和H 大观H 打成H  五星级H  ▁companionsH 张伟H  ▁WarwickH 严明I 诚心I  一切苦厄I  ▁weightsI izardsI 应力I  区委副书记I  ▁hatedI 样样I  人行道 I  ▁Eisen I  操作流程 I  操作工 I ▁Dul I  ▁mangaI  年龄要求I  政务处分I  生活用品I  ▁lucrativeI 毛笔I  日放假I  葡萄牙I  ▁reimburseI 肩上I RecommI 粗心I chetI 太低I  ▁FillI wesI  座城市I  参考文献I  ▁Alison I  ▁lineback!I 对立"I  ▁Forecast#I  ▁Buddh$I  ▁Venus%I  ▁reconstruct&I 前言'I 清河(I  走进了)I 忌讳*I 梨花+I  ▁flooded,I 欢呼-I  样板间.I  ▁Cuomo/I  ▁Franco0I 打死1I  文中观点2I  的人在一起3I 午饭4I 栏杆5I  也包括6I ▁Kom7I  增加到8I " 我都认真当成了喜欢9I 和王:I  能促进;I  ▁remixI ▁ICT?I 恒温@I  ▁CubsAI 强军BI  硝酸盐CI  不适症状DI  小编的EI  知识与FI ▁BIGGI ▁RasHI 独创II  他还是JI  educationKI  umberlandLI  ▁yarnMI 欠款NI  ▁pitchesOI 巡警PI  电子化QI  ▁HospitalsRI  如果不能SI  敬请关注TI  ▁ParkingUI  ▁consortiumVI 云中WI SeaXI  这根弦YI BottZI GEN[I ▁LCD\I  是世界上]I ár^I 海涛_I 艰险`I  考试报名aI  ▁NursebI  拉开了cI ▁suedI  每一份eI  办理相关fI  ▁mutationsgI  顾名思义hI  ▁UniqueiI 货品jI  整理发布kI  designedlI  ▁convictionsmI  ▁FernandonI 的量oI  ▁SurvepI 豆角qI  六盘水rI jpsI BultI  但只要uI  ▁eyebvI  ▁CamerwI ▁sabxI  系安全带yI AlbumzI 名次{I  空即是色|I  ▁surfing}I 安防~I 雨后I  扫码进入I icornI  ▁PreparI  国防部I  地球上I  具体内容I ▁NRI  不收取I 夏至I 蚊虫I  ▁SecureI  ▁RetirementI  更大贡献I 临摹I  县政协I зуI  工作实际I orisI  ▁unparalleledI 乐业I 领券I  ▁hackingI  ▁contractingI 别急I 字帖I readerI  ▁RequirementsI ▁TotI  实施诈骗I udingI 题干I  佩戴安全头盔I  ▁ProviderI 味儿I 浓烈I  ▁LeoneI 可做I 多国I 计税I  ocraticI 写过I  掉以轻心I  已连续I  我们用I 人寿I  ▁hempI  岗位要求的I  ▁AdjustI dimI  主动作为I  尽量少I mansI 超低I  的朋友圈I  ▁SharpI  ▁breachesI  ▁LeatherI  ▁backwardsI 掌心I EmbI  的组合I  ▁polymerI xfI iltersI  ▁incubI 滋滋I  的大量I 一书I  ▁metabolismI 船只I  酒驾醉驾I ouringI  ▁BetaI  ▁firearmI 律所I LewI seeingI  图文来源网络I  ▁EVERI 会以I 着装I  升级改造I  度一切苦厄I 能耗I 补办I  的黄金I  点击下方图片I  archiveI  ▁decoratingI 住行I ruleI UnionI  ▁trunkI ohydI  ▁forkI anzaI 当之I 薪金I TypI  腾讯视频I  ▁catastrophicI  有效果I ▁werI  ▁HonorsI 友人I  学员们I 黑发I  会有所I 一朝I  cellentI 睡得I  想了解I  ▁corpI 在后I  ▁BoltonI  ▁SomersetI 心机I 议程I  积极作用I icansI  区人民法院I  ▁staggeringI 倍增I  ▁RocketI  ▁fairnessI 破财I 缺水I  活动规则I  ▁bansJ  ▁workloadJ  ▁elasticJ  ▁currenciesJ 业障J 深陷J  那么你J landerJ  ▁ArchbishopJ 行书 J 预设 J  为人民服务 J  脑子里 J  也不过 J LOGOJ  ▁ExclusiveJ 演播J 稳岗J  ▁fillsJ  ▁SundaysJ 老李J DropJ ▁XiaJ  风险评估J  ▁KangJ  ▁ResultJ  周星驰J  ▁ductJ  ▁disguJ unerJ  ▁declaringJ 单抗J 加拉 J  也纷纷!J  ▁Facilities"J ysc#J ▁INC$J  无论是在%J  在这个过程中&J 媲美'J 放着(J middle)J 以示*J  却无法+J  ▁Pompe,J  ▁vague-J  ▁athletics.J Amb/J  ▁Consultant0J  与支持1J  本课程2J  ▁Flint3J ▁EB4J  ▁Pharmacy5J 实测6J 海尔7J 而对8J gins9J  薄弱环节:J 井然;J  ▁crossesJ xq?J 开销@J 避让AJ  的游客BJ  科技公司CJ  ▁literalDJ  工作是EJ  ▁ScotiaFJ 又多GJ наяHJ  ▁GlobIJ  ▁AppointJJ  ▁trailersKJ  为加强LJ 登机MJ  ▁wellsNJ  并提出OJ  色不异PJ 听党QJ 耳边RJ "{{SJ brainTJ  则需要UJ  方式和VJ 光亮WJ ▁PVXJ  融入到YJ 钱了ZJ  敌人的[J  热线电话\J  研究方向]J those^J  ▁nickname_J  ▁billionaire`J  一起玩aJ  为阴性bJ  干辣椒cJ  还有个dJ darkeJ  控制分数线fJ  ▁unitegJ  ▁suspensehJ  疫情蔓延iJ  ConsumerjJ ohankJ  表示自己lJ  LicensemJ 都难nJ  ▁multiplayeroJ 不垢pJ HomqJ  请联系小编rJ  ▁abusivesJ 反差tJ 心慌uJ EmpvJ  而那些wJ  首席执行官xJ  上的重要讲话精神yJ lczJ  平行志愿{J 造林|J 解密}J  的限制~J  老城区J 宽高J  指导员J  ▁unchangedJ 龙口J  ▁apologyJ  要加强对J idanJ ▁TueJ  ▁abruptJ  坚实的J  ▁complainingJ ▁yaJ  ▁DepotJ  职业院校J 往外J ▁RNJ  ▁CycleJ  在平时J axiesJ  ▁EmmanuelJ 推断J  买保险J  沃尔沃J onteJ  ▁CoverageJ ITSJ catchJ  ▁detrimJ  ▁decisiveJ  ▁LieutenantJ  石河子J GYJ  退休后J 甘南J  学习计划J 心神J  这首诗J 一袋J  ▁bellyJ  天时间J  通过各种J  ▁empoweredJ 总在J ýchJ  ▁despairJ 勉励J  ▁CITYJ  ▁shinyJ  中国女排J  ▁RwandaJ 粥样J msonJ 天际J LatJ  贵人的J  燃料电池J  ▁breadthJ 追剧J  ▁executingJ  ▁identifiableJ 清清J ▁hiJ  数字人民币J momentJ  ▁PreventJ  ▁publishesJ 水性J  值得关注J  其中重症病例J 四点J  南通市J  都在看J 领地J  YesterdayJ  上发表J cyclesJ 放了J 卖点J 洗钱J FriJ adenJ  专项治理J  ▁extractedJ  ▁ActingJ 抗争J  服务费J 切开J 法式J  金黄色J  的环境中J  ExpressionJ 有料J aktJ  诸法空相J  ▁DissJ  ▁dazzJ emyJ  了一项J  突破口J  支援湖北J EmilyJ odianJ  ▁continuationJ 猪蹄J  分区分级J  ImagineJ 追尾J JetJ 花式J  ▁PayPalJ  的药物J shenJ  ▁unsafeJ  适应性J 二三J SocketJ 栈道J  债权人J  ▁whoeverJ  ▁DetectiveJ 掌柜J ▁DobJ 赶来J  ▁TIMEJ HgJ 墨尔J 重整J  的反应J  也不可能K  色不异空K  ▁empirK  ▁SereK 一鸣K  纯天然K  相关知识K 落马K  小区居民K  ▁scent K  ▁Garage K " 提供任何明示或暗示 K 教的 K  目前有 K NiceK  ▁categorK 江水K 青青K  试点工作K 低落K 充盈K 连忙K  空不异K  在家门口K  ▁LithuaniaK 」「K 染发K ▁ivK  不折不扣K  ▁prejudiceK  ▁ProgressiveK BloodK  ▁metaphor K hew!K  的美味"K  睡着了#K  话不多说$K  安排好%K 三连&K 云雾'K  特惠价(K  信息公开)K  ▁settlers*K ▁Erd+K  服务项目,K Offer-K HOUT.K 睁开/K  ▁Brick0K  业务能力1K  盐城市2K 废旧3K 林生4K  名单公示5K 执导6K 社长7K  ▁conscience8K 淖尔9K  ▁Duchess:K  ▁killings;K  ▁facilitatingK Opin?K  ▁Residential@K 屋子AK  春节前BK ▁DewCK  ▁NGOsDK  ▁tubesEK + $本文内容及图片均来源网络FK  省委省政府GK  ▁occupyHK  ▁DeveloperIK  ▁huntersJK  ▁acknowledgesKK 再过LK  洋溢着MK  ▁HeartsNK 海燕OK 浮肿PK 透着QK  武则天RK  ▁AllowSK  ▁StarbucksTK 傻傻UK 路桥VK 小城WK  空不异色XK  ▁HawksYK 东亚ZK  再接再[K 涌入\K 使者]K  ▁flock^K 抗原_K  健康管理师`K policyaK 不倒bK 恰逢cK 如约dK  derabadeK  ▁ThreadfK  analysisgK  ▁precedenthK ttariK 中人jK 手下kK 非瘟lK  特训营mK 卫东nK  属兔人oK  绩效工资pK 津津qK WTrK 多看sK  时间及tK 多想uK 梵蒂vK 客气wK  ▁BreastxK  ▁DocumentsyK  家喻户晓zK peace{K ▁COM|K 欢欢}K  ▁Offices~K 集会K  关于调整K  ▁AmerK  知识竞赛K  qualifiedK  ▁stereotypesK 反派K 人权K 再做K  图文编辑K  是诸法空相K  ▁spiceK  ▁sandwicK 二要K ▁MukK  小规模纳税人K 晦涩K eccK  认真地K  都不一样K  大家一定要K  ▁OscarsK  构建新发展格局K  ▁ambientK  中央八项规定精神K 伤痕K  生产车间K 食客K  我心里K  工作流程K  自行承担K  ▁registeringK 预热K FantK 公报K 古道K  ▁neonK 等额K 荟萃K  过年的K  ▁fadeK  ▁CranK  ▁ChildhoodK ällK  决定对K  新闻记者K  ▁sleeveK 消食K homK  以身作则K  刑侦大队K  ▁JungK  出席会议并讲话K  SelectorK  靠谱的K  ▁restructuringK 酝酿K  了我的K RoundK  ▁LifetimeK 飞舞K 九天K 角膜K  应当按照K  ophobiaK  党的十八大以来K  ▁VintageK  ▁EmergingK  ▁interactingK 配对K  ▁bandwidthK + $转载目的在于传递更多信息K 也未K 张志K jayK 金城K  调研组K  ▁McConnellK 号角K oilK  环境整治K  生活物资K kbK 易感K  factoryK  ▁IntegrationK ohoK  ▁strengthenedK 由各K vertsK 露营K ▁USSK  ▁SurvivK 抬高K  记录的K  性格开朗K  ▁LampK  执勤民警K mbHK 反观K 药性K ▁GUK  我不能K atalogK  梵蒂冈K ownsK scarK ▁RheK  ▁wolfK ALTK  承担起K ▁AlfK  ▁sodiumK  ▁impairedK 博白K  西双版纳K 后经K 家伙K  交通管理部门K  ▁LicensedK  就很容易K  的案件K 大鹏K 年月K  为了能K  这些都K  工作能力K  CreatingK ANNK atenL uyenL alachL  ▁selectiveL 三重L  规定了L  ▁DevicesL 波特L  公交站L 展位 L  绝对不能 L  不垢不净 L ▁AMD L 热身 L  信息共享L 不俗L 平度L 少林L 风范L  如果把L  ▁TheresaL  综合改革L  甘情愿L  幼升小L 引爆L 搜狐L 镇雄L  植物园L  这个故事L LetterL  ▁advocatingL 尼克L 本级 L 理气!L Side"L  并不知道#L  根据实际情况$L  ▁receptor%L 德宏&L 评卷'L Zero(L  工作量)L osaic*L 人呢+L 汉口,L  ▁Colonel-L 同性.L 围着/L  全长约0L Leon1L 辉瑞2L bilt3L  ▁Missing4L  ▁judgement5L 获胜6L 阴雨7L  ▁resh8L 一指9L 菜谱:L 长制;L  最简单L  ▁relocated?L 减持@L  这么好的AL  思想和行动BL  ▁devilCL 由衷DL  ▁WolverEL  ▁KanyeFL  ▁comparativeGL  ▁obstacleHL 有名IL 风趣JL  她就是KL  近些年LL  ContractML ñaNL  ▁chromeOL CAPPL 这般QL  ▁RootRL 锦标SL 扫地TL  属虎人UL  的追求VL ▁segWL 先知XL  平安安YL  marketsZL 不齐[L  发展为\L  的商品]L  ▁comprise^L 内陆_L 无能`L zieraL  中职学校bL  本草纲目cL  perfectdL 陋习eL NOTEfL 三者gL  GeorgiahL ' ssssssssssssssssssssssssssssssssiL 邪恶jL sqrtkL WMlL  ▁PublishersmL GanL 包袱oL 害虫pL ctvqL  两手抓rL printfsL  的大事tL  检查结果uL  ▁CafévL 打发wL 蔬果xL  一直都在yL  ▁chiczL  ▁booster{L  危险化学品|L 苍蝇}L  一点儿~L 海宁L  以往的L  受访者L  ▁mankindL  用的是L  可能要L 审慎L 银联L  divisionL 剪裁L  发生交通事故L 脉搏L  革命先烈L  ▁curatorL shotsL 一岁L  ▁lobbyingL 长子L  ▁seatedL  经济高质量发展L 舰队L BandL ▁konL  ▁ConcordL 教官L  当天下午L  ▁unfamiliarL  电影节L  ▁LuckilyL  动物的L  ▁JumpL 盘活L  的影响力L  ▁osteL  ▁tenthL  副处长L  日确诊L ▁ViaL 香型L  资源优势L  ▁BiologicalL  开发有限公司L 围巾L  ▁favouritesL 抚顺L rityL 空降L SANL  维持在L  PoliticalL 宝安L  ▁BoothL  ▁enhancesL 可有L 无菌L  周岁的L obbiesL 分夺L eductL 新余L 落网L 调皮L  ▁shoresL  先生说L NikeL ▁tedL  你不是L  ▁chemotherapyL 两端L  信息中心L 爱党L 江夏L 节前L  人社厅L  底层逻辑L 现房L  天花板L  一应俱全L  大幅提升L  ▁passwordsL 元件L 酌情L 取缔L 要约L  产阶级L  元一斤L  的还有L  aduatesL  ▁culturallyL 写真L afaL  ▁GalaL  ▁PixelL 少走L  小学校L obilL edayL  ▁HelsL  ▁BullsL PickL  可以起到L  的教师L  王阳明L  广州日报L 阖家L  但同时L  纪念碑L  齐白石L ▁LumL  ▁CriticsL 微店L  ▁xmlnsL  ▁stewardL  ▁DetermL  ▁awakenL  这个消息L isiteL 五位L 生出L  统一组织L  研究者L  分析研判L 顶端L ppyL  乡亲们L  少先队员L  ▁SprintL  ▁aggravL  SingaporeL 令第M 齿轮M orumM  ▁cylinderM BUGM  金山银山M  MoreoverM 大把M  ▁sneakM  ▁Peoples M 商住 M  ▁capacities M 一码 M 问政 M  FeatureM 简短M  中出现M  专业人才M 过境M  孩子是M ▁cynM  立案调查M  在一定程度上M  astrousM  ▁SpeciesM 瑶族M 苦干M  年中考M 不累M ourmetM  天安门M  ▁mysteriesM  个人观点 M  ▁bargain!M  又到了"M  顺利通过#M oyan$M  ▁Mechanical%M 的马&M  参加笔试'M worker(M 极品)M  与他们*M  可有效+M 龙潭,M  ▁Kris-M  正常情况下.M 该项/M  基督徒0M  ▁tornado1M 你又2M 过快3M Room4M 粉笔5M  征集志愿6M  autions7M xz8M 开裂9M 身受:M ▁Sig;M  决定书M  也可能是?M 温哥@M  ▁flawedAM  保护的BM  ▁einerCM 机车DM  有一名EM  贵人相助FM  起源于GM  ▁SonicHM 下场IM 双层JM  ▁UltimatelyKM amacLM  次全国人口普查MM  ▁PrairieNM  如何做好OM  ▁AvivPM  急诊科QM  JusticeRM  ▁destinySM 联社TM 节制UM  使用手机VM  ▁AgentsWM  ▁adapterXM  成为一种YM  领取方式ZM 萌萌[M  ▁sworn\M 定兴]M 换上^M ▁Rex_M  厦门大学`M  可提供aM  调查结果bM  反腐败cM  在此次dM  公益性岗位eM  走到了fM 布尔gM 而当hM ieberiM  eworthyjM 冰川kM ▁HyplM  环卫工人mM  还是可以nM 一盏oM  在网络上pM OkayqM 梁山rM 胸前sM 讲台tM  等重大uM  县委组织部vM 河畔wM  基层治理xM  科技进步yM  ▁devotezM  ▁persecution{M 香肠|M  ▁prophe}M  ▁Preservation~M 晒干M 结伴M ovskyM 博爱M  从武汉M  的一定M  各个方面M  ▁beginningsM 久违M  诈骗罪M  平均水平M  的气息M  生态环境部M 小觑M 随迁M 多情M 政客M  住房城乡建设M  我们又M  日举行M AMESM  支配收入M . '人民群众生命安全和身体健康M 破除M  电话咨询M  ▁padsM  封闭式管理M enzM  再后来M  投资项目M  特级教师M  综合整治M  ▁jealousM  的人类M MarieM  VirginiaM 季后M  ▁outfitsM 倒塌M  仅供学习M  新疆维吾尔自治区M  ▁SteelersM ▁QurM  ▁drainageM  ▁OfficialsM  ▁repertoireM 二连M  足三里M enburgM geriesM  communityM 万块M  不出门M  ▁issuM  ▁navyM  ▁combustM clinM  也成为了M  ▁FighterM  ▁wonderfullyM  本地宝M urnameM 歌颂M lassesM  includesM  ▁remarkablyM  徐志摩M  团结带领M  ▁ConfederateM  ▁sufficientlyM  ▁upgradingM FitM errorsM 日趋M  ▁twistsM  最高人民检察院M 风控M BetterM  ▁menusM  producedM 皈依M  生活必需品M 二建M 傻子M  的必要M  医生说M  降血压M 雪中M ▁JDM 升旗M 两份M 喝点M  济宁市M ahrM  的人工M  的阶段M  感受一下M  ▁inspectionsM 兴国M 剔透M  制成的M OTAM  版权归属M  新世界M  想到了M DebugM 前脸M  平平安安M  ▁idealsM  ▁HoffmanM 兵器M  在古代M  identialM  reementsM 盘子M  明确表示M 得快M SCRM  有关问题M 丰盈M 放行M  跟自己M  省公安厅M 一聊M  课教师M peredM 并入M  相关材料M  道德与法治M 和在N 娜娜N  两个小时N OffsetN 寸步N  ▁assassinationN 儒家N  用什么N 来年N  出来后 N  跟我们 N Games N  ▁Rapt N  ▁shred N  AverageN  ▁royaltyN 征迁N 桑叶N  医保电子凭证N  不断发展N 不治N 焚香N  从目前N ActorN cticaN  ▁MythN 村内N  了一半N  新农村N  innamonN  主动性N  转型发展N 被撞 N 该剧!N 余地"N  坚实的基础#N  Consider$N  ▁Dorothy%N  ▁Speedway&N 变差'N  ▁forgiveness(N  金字塔)N  集团董事长*N 无阻+N 白嫩,N  讲解了-N  抓获犯罪嫌疑人.N 上说/N 宋词0N  ▁Attorneys1N notes2N omers3N 益阳4N 若能5N ▁$_6N 昌盛7N 王思8N  兢兢业9N  ▁Kimber:N  严重时;N  分夺秒N  ▁Phillip?N  Development@N alerAN  一件事情BN  ▁InvestorCN 连通DN  ▁AheadEN 啪啪FN StrongGN  ATIONALHN 诚邀IN  黄鹤楼JN 慵懒KN 其二LN  到就近MN 切碎NN 这颗ON  ▁HerzPN  ▁StructureQN  ▁TransformationRN  感谢大家的阅读SN  感染者的TN 倾情UN  市场秩序VN 你却WN  ▁weighedXN 经管YN  管理层ZN  身体健康和[N Nearly\N hton]N ourced^N 很远_N DES`N  TimeoutaN 汹汹bN afiacN  的报告dN ineneN  ▁mentalityfN 不苟gN 查分hN xiaoiN  ▁autobijN  ▁DaughterkN  功能区lN  音视频mN  特异性nN 东汉oN  没有得到pN  DurationqN  局限于rN  其中之一sN  ▁scooptN  以为自己uN  ▁indoorsvN 找寻wN 纹路xN IngyN  中国文化zN  那么简单{N  ▁Penny|N  ▁theology}N cmd~N fedN  努力去N  网友评论N  若干问题的N  ▁WeissN  ▁StraitN  ▁bespokeN 岛屿N 浦江N  ▁contraN  ▁onionsN 大展N 周杰N  InternalN –　N 大关N  比上年N  ▁spicesN 淡水N ▁KYN  ▁unauthorizedN 牵线N 章丘N  ▁chassisN  ▁introN 紧贴N  ▁militantsN  主治医师N 的陈N 黄梅N  旅游度假区N  ▁grainsN 乳胶N ▁szN AutomN adderN 士气N 活禽N 写生N + $现实生活中的小事也是这样N yjN 井下N 日光N 脊髓N  创新能力N  ▁LotsN 世外N 传送N  预检分诊N  ▁wiresN  ▁GuidesN  如实填写N 回调N  地理位置优越N 一要N  ObviouslyN  膀胱经N 居易N  ▁förN  新要求N  相比较N  识别图中二维码N 特许N ItalyN  imbledonN 十日N  ▁playfulN 筑起N  ▁CoupN  ▁POSTN 送餐N  如果发现N 分两N  ▁ReachN  ▁DiplomN 共治N 无理N pentN 永嘉N ylieN  由高到低N 有难N 触电N 请与N 课前N  一氧化碳N  批评教育N  ▁oppressionN 早教N  观看了N  如何处理N  ▁GavinN penameN  ▁gingerN  describeN 匆忙N 清纯N  ▁asymN ruguN  酒后驾驶N  ▁sodaN  ▁accreditationN 沥干N ▁neoN 防寒N  以人民为中心N  老祖宗N 新兵N 帖子N 痛的N engersN  ▁AlexaN  ▁WeberN 在原N 比的N  ▁barkN LordN 牛油N  战疫情N  局集团有限公司N  通常是N  念念不忘N 帮她N upleN  助学贷款N  岁月静好N  所学专业N  ▁BenchN  ▁VariousN 写话O 和各O  回家了O  更是一种O  ▁impressionsO 放的O LETO WatO ffitiO  ▁territorial O 抱有 O  老中医 O licia O ▁Agu O 千伏O rebO  创新型O  正常运行O  ▁quirkyO 就叫O  为满足O  究竟是O  工程造价O 匝道O 黄酒O ▁hayO ▁necO  ▁НаO akisO '));O  ▁lockerO 护送O 搬进 O 攀登!O ▁","O  预防措施#O  ▁Basil$O  ▁Stafford%O 为好&O  专家学者'O beaut(O  ▁dependence)O 尽享*O  ▁paved+O %%,O  ▁McCart-O  ▁colonies.O 皮炎/O 返贫0O  ▁persist1O lambda2O 始发3O 长此4O  原因的5O  因此而6O  一篇文章7O  社区矫正8O ricing9O  ▁Rabbi:O  ▁hunter;O )×O 丝路?O these@O 无异AO 老去BO  ▁brighterCO  isationsDO KREO 艳丽FO yinGO  争分夺秒HO :‘IO 向右JO  ▁DeerKO  ▁котоLO  ▁exceedingMO  ▁nostalgiaNO 地头OO  对学校PO  女性朋友QO 橱柜RO 止痒SO  的领导TO  从零开始UO eqVO 筹建WO 惋惜XO  公告发布YO  等级证书ZO 一栋[O  又来了\O vier]O  ▁Deborah^O  的节日_O  当之无愧`O  越来越近aO  很可能会bO  被网友cO  recateddO 古文eO 定罪fO 俄语gO ollahO AwiO  你的人生jO  积极推动kO  陪孩子lO RachelmO  ▁AutumnnO ▁TinoO  直肠癌pO 外研qO  的表情rO  自然就sO  ▁ammunitiontO 翻开uO 早饭vO 供奉wO GixO 王志yO  中国家zO  大部分的{O  自主招生|O  世界级}O 库尔~O  保障和O 古装O  ▁PavilionO 挥洒O VIDEOO  ▁centrO  从一开始O  ▁securelyO  ▁BangaloreO RamO  的好习惯O  镇党委书记O 释义O  出自己的O  ▁typingO  ▁extinctionO 售罄O  ▁trainersO 东升O  ▁understandableO ineesO 中将O  找上门O  服务人员O  ▁presentlyO 诗篇O  外来人员O 六七O  ▁antioxidO  ▁packetO WINO  预计在O  再接再厉O  ▁pencilO  ▁RevealsO  政策性O  restrialO  ▁authoredO  ▁eclecticO 惩处O 获益O  这笔钱O  平台和O  独立思考O  ……“O  和政治O  dependentO  作品展O  friendsO  在实际O 下辖O 星的O 而动O  孩子的学习O 日志O  ▁MarksO 体魄O  山东大学O  农村宅基地O  ▁reproduceO 联欢O  ▁HydroO  prototypeO 惊心O  ▁OmanO  只有这样O  没有想到O  提升自己的O 佳品O  游戏的O lausO  ▁ReichO 间谍O  再加入O  ▁BermO 场站O  也知道O  ▁depictingO  在任何O AIDSO  ▁tabsO linuxO  ▁ensuredO xmO ndraO  价格面议O  BeautifulO 吸睛O 灾情O  统编语文O  ▁frameworksO 办结O 灵山O 防骗O ▁IsnO 架上O 水滴O 海景O  ▁corpsO  ▁overseeingO  不能吃O 享年O 用语O  点击下面O  伊斯兰O orityO ONSO  温哥华O 返利O mologyO 摘自O UESO  ▁dependencyO  ▁MidlandsO  妥妥的O  ▁buysO 懒腰O futureO  ▁ValuesO 私营O  属地管理O  开心快乐O  有需要的O  ▁prayingO 点前O  面积为O RankO  贫困群众O  ▁CalvinO 不回P MiniP  工程大学P  ▁TrudeauP digitP  ▁heavierP 敞开P  事业运势P  sponsoredP  ▁Sunshine P 排卵 P  一份力量 P  ▁Rouge P  ▁Vehicles P  炎热的P 顶替P  有阵雨或P  纤维素P enicP RiverP 区级P 牛的P  ▁caregiversP agogueP ATERP  文化建设P  ▁HydeP  ▁marriagesP 扣缴P  亲戚朋友P 双色P  ▁logosP  毛利率 P  ▁Cause!P  月工资"P  细致的#P  有效措施$P  二十五%P  ▁summ&P 小院'P 稍稍(P 缓刑)P  全自动*P 西城+P  ▁Quint,P  ▁contested-P 胡某.P ▁Ivy/P  的重要内容0P ARA1P 等症2P 航站3P 孟子4P  去哪儿5P  都是很6P  gradation7P 存钱8P  大众网9P  ▁Academ:P 平谷;P  获取方式

    p isans?p  ▁Lemon@p  ▁copingAp  ▁RecruitmentBp 首尔Cp  莫名其妙Dp  后进行Ep  ViewControllerFp 监事Gp ▁CypHp 约上Ip 送别Jp  世界经济Kp  改造提升Lp  财政部门Mp 工行Np  湛江市Op INFOPp  ▁DenisQp  ▁DiaryRp 笔录Sp borsTp  ▁comprehendUp 意象Vp 肉食Wp 起好Xp inusYp  ▁APIsZp  ▁nephew[p  ▁Katrina\p  学生资助]p  ▁blessings^p  充斥着_p  在各个`p  下面这些ap  在娱乐圈bp  发展和改革委员会cp ▁Ebdp  ▁Slovakiaep  照片中fp 亏了gp  ▁Initiallyhp  ▁simulationsip qinjp  ▁accustomedkp  城镇居民lp  ▁Chestmp  ▁endeavorsnp 相辅op ▁NLpp  更新了qp  转化成rp  ▁Fullersp 加成tp  区域性up 都和vp Whitwp  ▁scalablexp  该患者yp  食物中毒zp  ▁Savings{p disco|p  ▁contrace}p 换算~p 烤鸭p  ▁metabolicp  的青春p  ▁hypocp 清静p 喝完p 说唱p  真诚的p  睡觉前p  大概就是p  图片来源于p  ▁sortingp  ▁poignantp ▁MONp  接受治疗p 带班p  的主角p LOOKp ▁lanp 道来p  真抓实干p  ▁Guidancep  assertEqualsp  Productionp  ▁ministriesp 图的p  模拟卷p  如一日p  化学反应p  directedp  ▁Disorderp ikedp  行政法规p 一档p 小草p ▁MUp  roundedp  一直坚持p  审核通过p Annotp 过客p  梅州市p  治国理政p 既定p  ▁Reggp  ▁wrappingp loup  我就不p  人生路上p 转行p ▁FTp  ▁паp Jeremyp  ▁Dominicp  方法的p  意义重大p あゆp 含氯p 平乐p 童趣p  都是可以p  ▁Platp Odp 也多p 新晋p  ▁coincidencep 年生p bilp  老奶奶p ▁LTEp  也可以用p 王大p alsap iancp  优异的成绩p  ▁Highlandsp  ▁fragrancep 前卫p mphp  报名中p  有关负责人p  ABILITYp  ▁growersp  经典老歌p  是所有p  已经完成p dockerp  ▁confidentialityp  他们还p  享受生活p  完全不同p  ▁Blankp  Warningp  roducedp  给人以p  ▁Graphicsp  Dictionaryp  每一名p 之最p 鱿鱼p  ▁Saturdaysp 纵然p 鸿运p  由国家p CELp  央视网p  的称号p  updatedp 戈壁p 无极p  承担着p  技师学院p  ▁herdp  ▁connectorp  等待着p  申请办理p Austinp  并处罚金人民币p 大坝p ritop  通用标准p  ▁enricp  ▁emailedp  ▁mutationp  ▁Procedurep 中游p 我司p  ▁pantp 走心p Noticeq  ▁curseq  ▁tappingq 受精q  教学内容q  ▁Coralq  ▁unterq  ▁daytimeq 泪流q ент q  应符合 q 变频 q 的被 q  铲屎官 q  ▁defensesq 固态q havenq 濒危q 玩了q 镀铬q  ▁Romanticq 暴利q 分班q  ▁Chamq 许愿q  西南地区q  ▁Nileq 兑付q 拥军q  LICENSEq 一城q 主旨q  眼看着 q  个人原因!q 医术"q  ▁turnaround#q  ▁Conservatives$q 的歌%q  在逃人员&q  总体来说'q  ▁encl(q 月龄)q 灾祸*q 质问+q  科医院,q  training-q  更有利于.q  ▁aide/q  ▁Roads0q 较短1q ▁ASE2q ▁Jol3q  ▁Shoes4q  ▁resemble5q 上肢6q  动车组7q  工作者的8q  rospect9q  ▁parole:q  仅有的;q  ▁Automationq 之长?q 增信@q  结束时Aq 奖杯Bq  垃圾的Cq PollDq  ▁poisoningEq 贴吧Fq PagesGq  阐述了Hq njIq  酒店的Jq  ▁beamsKq  全勤奖Lq  StrategyMq  ▁AbdullahNq 变性Oq 激昂Pq 责怪Qq  未发现Rq  ▁FlipSq ophersTq  元素的Uq  沈丘都市网Vq  ▁FischerWq  中心和Xq  产品设计Yq  全球疫情Zq  出行安全[q  ▁Extended\q  ▁Congressman]q 学而^q  都不够_q  就有可能`q  ▁Lumpaq 五千bq 住处cq  稳就业dq  更多精彩资讯eq 仙家fq abovegq  ▁upheldhq 童子iq doujq  比较高的kq  公共科目lq REDmq  调查报告nq  ▁layeredoq  ▁fashionablepq fraqq  不干净rq  书记员sq ▁Vijtq Unlessuq ▁IQvq Kathwq  区融媒体中心xq 乐享yq 报业zq 更远{q 芳草|q  归属于}q  的好奇~q  ▁Seeingq 密云q vehq  大药房q  都来了q auffq  communicationq 四轮q 氢能q agonalq 坦白q  认真对待q 重力q  汇报了q andanq  ▁Bladeq  需求和q ightedq  雷神山q sidedq WFq  公平的q  上一层楼q  ▁signageq 矿井q  最长的q Bridgeq  ▁foldedq fabq 很重q  户型图q  那我们q 共性q  ▁Codesq  ▁intoxq  是否能够q  Transactionq 两块q  ▁pensq 我再q 水生q  的是一个q otineq 置信q  身份证明q  ▁cathq  Outsideq  romagneticq  ▁Sociologyq 上扬q ▁ABOq  ▁qualitativeq  黔西南q  ▁Valeq 挽留q  录取的q ▁Segq 李玉q  ▁Sharkq  ▁medicinalq MONq  亚硝酸盐q visitq ividedq  ▁optimumq 校方q  ▁thirstq 扎心q 畅玩q abellaq  ▁treadq  ▁aquaticq 繁星q  教育活动q  ▁Gujaratq 情缘q  全都是q  学生和q  ▁rebellionq WGq  组建了q  ▁Throwq 白癜q 看中q  主动请缨q  ▁intellectq ▁Neoq  ▁vampireq  更别说q  的外表q  ▁Cakeq  ▁fatalitiesq oinq  meaningq  华为的q sportq answerq  ▁Debateq 多维q 晾干q  什么时候开始q 监审q  出台的q  李玫瑾q  该企业q ereyq  最起码q  ▁strokesq 一厨q Einq  和在看q  越走越q droq  新增了q  是他的q  ▁whoppingq 为上q  第三条q  ▁hubsq  ▁Yourselfq 开往q 智造q 亮剑q 检举q 甲环q 香精q  我也不q  精准的q  ▁Compareq  南开大学r  团队合作精神r  ▁Anglicr 爱民r  ▁overlapr  ▁Trinidadr ▁EFr  美国在r  ▁bashr  ▁serum r  ▁Adding r 妇人 r 耶鲁 r 雪梨 r ANYr DECr  可以减少r  ▁eclipser  ▁fuckingr daor 木板r 辣的r  咱们的r  ▁cappedr  ▁predictiver  米左右r mannedr  ▁poultryr  ▁Headsr 易俗r  了解下r ▁Yukr  准确地 r 中含!r 日间"r 闲着#r  电子码$r  心存侥幸%r 和美&r  工作会议精神'r 自责(r  认定的)r  ▁sandy*r  ▁tended+r  ▁Helsinki,r  ▁Considering-r 北辰.r  和实践/r Chart0r  ▁Worst1r ▁Coy2r ▁pir3r  耐心的4r  一夜暴富5r  ▁Amid6r  ▁Cage7r 用气8r ▁ov9r  微博热搜:r  ▁Ricky;r 麻雀r  ▁commute?r  但是从@r 一倍Ar 维斯Br 西斯Cr 责备Dr "},Er  ▁чеFr ▁AnsGr ▁gebHr  osteroneIr  ▁PhilharmonicJr 张三Kr  泥石流Lr  ▁SunnyMr 余项Nr 房门Or  回忆起Pr  重点企业Qr qhRr 成团Sr  岁之间Tr  ▁sitcomUr 付诸Vr  学不会Wr  的病毒Xr DominYr  ▁electronsZr  太好了[r 抒情\r 脸颊]r witz^r 泗洪_r  ▁tariff`r 赌王ar 香葱br  镇卫生院cr Structdr 真香er  全国政协fr  ▁VIIIgr  ▁blochr 酸钠ir 雷暴jr ITIONSkr vidialr  ▁zincmr  ▁sedimentnr  研究团队or 都给pr ouselqr  ▁Triberr 锐意sr  ▁akintr  ▁exileur 洋河vr  大兴区wr  也不用担心xr  国际上yr COMPzr  专家提醒{r  加快推动|r  ▁prevailing}r 五次~r 有其r 湿巾r  ▁bitesr mongr 宿主r helfr  吉林大学r  使用后r terminr 咸丰r 池州r 药企r  本案中r REAMr  智慧城市r 在省r 镇痛r  但是她r  道德品质r  国家卫生城市r dianr ▁Dixr  才有可能r  网络游戏r 月和r  西北风r othsr  潜移默化r  直接购买r  ▁Varietyr 跑腿r  不了了r  的明星r Urbanr  种植的r  习近平总书记对r  ▁Instantr 配以r lordr  ▁upstairsr  ▁humanitiesr riotr 飞飞r ▁WINr  可能就会r  原件和r  限时免费r  ▁Danieller 好啦r ñosr  战士们r  医保卡r kickr phaser  ▁hopelessr  ▁mechanicr 地步r  殡仪馆r 不渝r  啥时候r  将来的r  ▁HAVEr 归类r  自己不r  ▁differentiater  对工作r ’’r 镇上r  教学中r  用一个r addrr  ▁Clockr  ▁perennr 为辅r  气溶胶r ▁babr  最新动态r  顺利开展r  ▁polymr  活动等r  不断创新r  ▁penalr  ▁likingr 烤鱼r Stoner esselr  西藏自治区r  ▁botheredr 卸妆r 陈志r  英格兰r afkar lavr  未履行r  ▁syllr  ▁persuader 芦荟r  观自在r LOADr 量力r  十多年r  ▁Heinr ilepsyr  ▁Buddyr 彝族r  的精彩r  健康长寿r  涡轮增压r  ▁unjustr  ▁slaughterr  ▁demolishedr  ▁transformsr  ▁computer  ▁standpointr  ▁Alternativelyr  可以分为r  ▁gentr  ▁marginalizedr  自己没有r  ▁lentr 免于r  却依然r  参谋长r  内心深处r  Monitorr  coloredr  ▁Zuckerbergr 写法r  穿衣服r  服务号s  ▁placebos  ▁accusings bxs 减量s  李小璐s swires 车尾s  极有可能s EMAIL s 不换 s 四风 s 点餐 s Hig s  多云间s 贩毒s  事故现场s asuss  保持一致s Karens  研究中s ▁Rims  节日快乐s  比普通s 河谷s åns 是故s LOCs  不定时s 内政s  类食物s  无能为力s  生态修复 s  ▁butterfly!s  Reviews"s 腋下#s  发生的事情$s  ▁embarked%s 之神&s 清脆's  微粒贷(s crit)s  ▁Jorge*s  ▁materially+s  和城市,s  聚集地-s bh.s  默默的/s  pertension0s 轴线1s 迷恋2s  不断扩大3s 冷了4s 退烧5s  土地征收6s  ▁KIND7s  Everybody8s  ▁tolerate9s ▁Mum:s  每日一题;s  ▁Logos 和好?s 计件@s  的平均As 不报Bs 武力Cs  元罚款Ds  ▁StressEs  ▁swiftlyFs 她被Gs  在所有Hs  零售药店Is LewisJs  ▁BoxingKs  ▁MajestyLs  ▁quartetMs  ▁evacuationNs 临海Os  中国和Ps  小户型Qs айRs 旭辉Ss  不可缺少Ts  ▁diceUs 种下Vs zuiWs asakiXs  ▁ReevesYs  ▁influencersZs 贩子[s ivable\s 带量]s  老先生^s  专家团队_s  明显改善`s Andreas  ▁Mohammadbs 岑溪cs 隐性ds  内容丰富es izophfs 掩护gs  时尚感hs  有明显的is  ▁Perspectivejs 阴气ks  ▁possessedls 临安ms 前身ns  ▁Crossingos  ▁peanutps  的人身qs 乱的rs 骨的ss  具有一定ts  小型汽车us  美容养颜vs  课程体系ws  有点像xs  餐桌上ys ▁USEzs  ▁expresses{s 油箱|s cg}s 入眼~s  元不等s 接头s 浙大s  年国家s  ▁Penss  ▁textbooks  ▁consequentlys 是哪s  不一定是s  Israelis 二氢s 关中s 可从s  ▁interchanges  日结束s  全国各地的s 干涩s  女主播s 风度s ▁WVs  烦心事s  ▁conquers  男主角s  餐饮店s ▁Mugs  ▁bulbss ▁'',s  版权说明s  ▁informationals 和东s  ▁debits  ▁Caesars  ▁swellings iptss  疾控部门s  ▁Resils 亚裔s 非公s  集中医学观察s 之三s 正中s  总编辑s ▁cyls  everances  ▁Prospects 台海s  刚刚开始s 病种s  法院判决s Htmls  arnations  ▁Randalls 当过s 早报s  没有谁s  ▁terminalss  ▁generatorss  它有着s  这正是s 其乐s  在教育s  直接将s 笔下s  汇总表s 取穴s  ▁Essayss 小看s  第一件事s  ▁Winnings 微小s 行进s  仅作参考s  鼓励孩子s 任由s  自贸区s ▁huss  甲状腺结节s 权和s  ▁misinformations 外环s  ▁grooves Ghs brevs ▁Wans  ▁Nadus  ▁Sophias  ▁playwrights 清退s  要保证s  ▁planners  但事实上s  ▁Dions ▁idis ariouss krs  ▁Bards  ▁catchys 该市s  要选择s 又来s  从严从s 卷入s  考生和家长s }(s 到案s 母鸡s 获刑s 劳损s 征税s  深度好文s  ▁rollouts  巡察组s  以下的罚款s 生辉s  碳达峰s  考生们s 中旗s 这和s  学习资源s 台的s  payments  ▁paradoxs 重播s  直播课程s asthans  ▁Mixeds  ▁acknowledgings qds  马路上s  professionalt 厨艺t 聪慧t 范本t 养羊t 总说t 报批t felt  基本生活t  ▁smash t  ▁frontier t ▁omn t 晨曦 t 迷糊 t  只是因为t 巴赫t  统一考试t bsitest 溶剂t 脚上t ▁jat ▁Witt  鄂尔多斯市t  ▁cafest 大业t ▁PIt  第一章t  就开始了t Challt 婆媳t  entationt 气压t  可拨打 t  真的有!t  易懂的"t  ▁Careers#t 皮带$t  Questions%t  ▁virgin&t 屋面't 在三(t  未经许可)t exclud*t  ▁lumin+t  actively,t  ▁demographics-t ▁CJ.t  投集团/t  合作共赢0t  ▁Disabilities1t fé2t  密接者3t  尽量不4t lez5t  ▁Vanderbilt6t  子女的7t  走到哪里8t Float9t  ▁Loans:t 一查;t  主耶稣t  ▁Speakers?t  的瞬间@t  ▁linenAt 拉升Bt  国家政策Ct  edienceDt  ▁surpassedEt 党章Ft  留下来Gt  的种类Ht SequIt  衣食住行Jt  ▁HaleKt  ▁plagueLt  任何单位和个人Mt 罪被Nt  价值和Ot  更像是Pt  ▁burgerQt  ▁ClemsonRt  少量的St  南无阿弥陀佛Tt  ▁relievedUt  ▁KickstarterVt URIWt unkyXt 才好Yt 易爆Zt  都能看到[t rules\t  ▁Lexington]t  　–　^t  ▁Hammond_t 待用`t 一排at 两组bt  可作为ct  懂得了dt ermoet hawksft  全方位的gt  考试期间ht firedit  信息表jt  avelengthkt  也确实lt  薪资面议mt  ▁Barclnt 白粉ot 有近pt 殷切qt  让更多的rt 小鸡st 高州tt  教练员ut Halfvt  为了更好地wt  ▁Lordsxt  arationsyt Wangzt  ▁Cory{t 生来|t  低年级}t  ▁Brass~t 走一t 鲜味t  转折点t ▁CIOt ▁Hogt  原创内容t 两江t Oxfordt  ▁collaboratorst 不拔t ▁%>t 煮水t 鹌鹑t  年至今t  早睡早起t  雷锋精神t  惊呆了t  interprett  ▁Survivalt 个亿t hesiont  culosist 奇异t 爱着t  质量的t  提供更多t  ▁einemt  自己要t goerst  ▁Vitalt  明确指出t  暂停营业t  儿童青少年t  ▁Rankingst 从教t 使他t 和陈t 辟邪t  就好比t  ▁Successfult fortht  ▁Seahawkst  ▁mutuallyt 在北t 时需t  我还记得t  ▁arcadet  应对措施t  ▁TODOt  一年来t  ▁Shimt  ▁Proudt  ▁Toraht 场内t ▁UNCt  Beijingt 布拉t ▁evet  rollerst  ▁Possiblet  ▁kidnappedt 印制t  十几个t  让家长t  ▁Straightt 几万t  客流量t  ▁Cindyt 摩天t  上一次t  只提供t  满足感t  ▁Characterst 取胜t salet playst  省委副书记t  Trainingt  ▁permet  ▁Visitingt 珠峰t  本办法t inait ▁NAVt  ▁Pallt  ▁stickerst  金融市场t  ▁alleyt  ▁gravest  延期开学t  这个男人t  ▁jakot  ▁reconnectt  ▁noteworthyt  和维生素t  ▁Pipelinet PEGt  让宝宝t Hadt guant ndont  幸福美满t 直营t ▁reqt  及时进行t 生俱t 疏远t Jimmyt  ▁Kolkatat 小年t  党代会t  小动物t miat  一个月的t  ▁Attendt errickt  在群里t  还容易t 水系t  ▁Pharmact  ▁Buddhismt  ▁endpointt ,[t 匠人t  下班后t çõest enmentt scopict  更高效t  工作委员会u  ▁Beethovenu  ▁augmentedu 新开u  ▁rosesu  ▁Gloriau  ▁madnessu Piu zzyu  ▁domest u  综合行政执法局 u  ▁monastery u 扑救 u 炮弹 u ▁aceu  合同纠纷u ammersu  的时期u asiou  机动车驾驶人u 九十u 惨重u  可以用来u  看图写话u  ▁Personsu idavu  中国加油u  成为全球u  获得国家u  ▁clanu 商事u  一般都u  出水面 u  早已经!u Bottom"u endale#u  ▁echoed$u  ▁adolescent%u  Directed&u 成片'u  了一切(u  ▁Kirby)u  腰椎间盘突出*u кі+u 推车,u  不同的人-u iphone.u 贿赂/u  春夏秋冬0u heavy1u  ▁lump2u  ▁Infantry3u  的信念4u  耕地建房5u  ▁deploying6u 席位7u uen8u  地展现9u  这其实:u phil;u  零售价u 爱过?u 申办@u 粗细Au OTTBu  一个新Cu  anguardDu  ▁BrennanEu 小二Fu  关键点Gu  范围和Hu  ▁GoldbergIu 来过Ju  无一失Ku  美国海军Lu 仅此Mu  五台山Nu  最好在Ou enegPu  行动轨迹Qu 固始Ru  很正常Su rundTu  ▁CandidateUu  ▁concealedVu 做些Wu  引导和Xu  ▁sympatheticYu qxZu  决定性[u 爱惜\u  ▁Sham]u  四年级下册^u  ▁enzyme_u  ▁Minority`u 水镇au oehbu  ▁Expressioncu MEDdu ▁éseu  国内市场fu  出租车司机gu  ▁Filterhu 凡人iu 招惹ju 课余ku  在各种lu  风险隐患mu  ▁Lingnu  ▁Siemou 滕州pu  立足于qu  没过多久ru  自学考试su famoustu  ▁Tibetanuu 乔丹vu 血红wu 重拾xu  桃花源yu ciaszu inker{u  汉语言文学|u 绥化}u  小编就~u  追溯到u  客户经理u  ▁downsideu 智利u  ▁gastrou 术前u  Classicu rxu 有度u  规范性u Violu ubleu  明天白天u 刘少u  年代的u ▁ITVu  即可查询u  igatorsu  ▁cardboardu 项城u 抛光u  专属福利u  ▁Commentaryu  ▁heartbreakingu 顺遂u WORKu  ▁Docku  ▁accentsu  超强的u  ▁Gloryu  ▁Lebaneseu  ▁imaginativeu  不及防u  ▁shaftu  ▁inferioru 财星u 蹭蹭u lysu  expressu 未然u  的信号u  复习资料u  ▁Revelu  ▁resinu  ▁allegingu  谈天说地u  ▁boomingu  ▁milestonesu 热忱u  发展空间u  ▁methaneu  ▁sailorsu 儿时u ▁wieu  ▁tuningu  ▁knockingu 弹幕u 抒发u umabu ▁Ostu  ▁nutrientu 悦府u  ▁lineru  ▁Johannesburgu jitu  森林火灾u 辅以u  ▁harvestingu  行政复议u  ▁Cyrusu  ▁Vogueu  trainedu 百般u lmu 较前u  只要能u  要怎么u 剥离u 扁豆u 断电u  意外伤害u  还要注意u  ▁plannersu 着迷u  维护国家u  USINESSu  ▁elusiveu 宣泄u 荐读u  ▁enzymu 句号u 恪守u  取得实效u  坚定了u  Assistantu 消暑u  ▁Virtu  ▁Helenau 慕名u  allowedu  validateu  ▁undocumentedu 从心u 刘强u 多篇u 对您u  ▁Chopu  ▁Giulu 伺候u ▁MKu  以下条件u 回响u 附院u  目前已有u  负重前行u 担责u  财运运势u  那么我们u  ▁Mandelau karau Sleepu  ▁coutu 镌刻u  为促进u  吃火锅u  在过去的u  非常多的u  ▁sorrowu  ▁Kristenu 验方v  街道办v  ▁Moodyv  ▁Responsibilityv 临界v 龙眼v  从今年v  这道菜v  ▁Biggestv  清楚楚 v  著作权人 v  ▁mindful v  指挥部办公室 v 薪火 v  从来不是v  ▁painsv 胁迫v Sherv  消毒工作v  Constructv 九九v  孟加拉v  ▁septv  ▁Telecomv 巡游v 经纬v  生成的v  行动方案v  ▁Episcopalv  轰炸机v ▁Glev  微信群里v olate v  ▁seals!v  的安排"v  知情人士#v  ▁Reef$v  ▁cruelty%v  孩子会&v 无虑'v 玉龙(v 译为)v oprano*v  ▁Bonnie+v  也正在,v  五花八-v  人员对.v ▁FIL/v hex0v  两位数1v  每一件2v  的一件3v UAL4v ▁HK5v  号文件6v  ▁Thoughts7v 甘孜8v  志愿者的9v 黑客:v  ▁flank;v  低成本v  ▁Susp?v  ▁superf@v  ▁GeographyAv 惊奇Bv 有水Cv  ▁даDv  ▁mixesEv  ▁OfferingFv 东进Gv 结余Hv AboveIv  ▁KatyJv  ▁remindingKv 每秒Lv 德斯Mv ICEFNv  加快发展Ov  附属中学Pv  中央军委主席Qv  ▁HighlyRv 体弱Sv 栋梁Tv  日进行Uv  时间点Vv  长白山Wv  ▁SplitXv 宝坻Yv  ▁ruthZv  很方便[v ▁CAD\v  协调发展]v  招生单位^v  消防通道_v  ▁Bieber`v  ▁fountainav  ▁helicoptersbv 口译cv  下方的dv 单次ev  性行为fv  ▁Thankfullygv Tonhv Viniv  中使用jv  ▁tapeskv  的美景lv  的难度mv Miaminv JMov 云淡pv 传到qv Flagrv ▁ESGsv  DOCTYPEtv  ▁Futuresuv OBvv 妩媚wv 难懂xv  睡眠不足yv  同志的zv  医疗废物{v  ▁multicultural|v 悼念}v "">~v  录取分数v Holderv abbingv  物业服务企业v :”v EMBERv  降雨量v  可联系v  清清楚楚v  蔬菜水果v  ▁alcoholicv 早有v  的后果v  根据不同v latestv  官方微信公众号v  补偿安置v  餐饮企业v  ▁lavishv  ▁declaresv 自控v  即将来临v molv  机械设备v  对我来说v  ▁Roundupv 悬架v  ▁Acidv  ▁avantv  十七条v  地道的v  ▁lockingv  ▁conservev AMDv ▁glev  组织学生v 脚部v  请记得v  ▁jQueryv  ▁Unlimitedv  什么人v ▁Hutv 不学v 听歌v  坚决做到v 丹阳v 公信v 那条v kalv  五粮液v  广大青年v  ▁Holtv  ▁Balticv  已经达到v 博客v  约翰逊v  经审理v  ▁bulbv  ominiumv 抹布v maniv  烦请告知v  ▁loomingv 可享v 翻盘v  高效率v  俄罗斯的v routerv 双料v 套数v UTEv  人世间v  在全国范围内v  这东西v  危险因素v 用刀v 肾炎v  ▁Fijiv  ▁yeastv  ▁protestingv  的总体v  ▁prefersv 对齐v 相随v 缺一v  采取的v ▁Pagv ▁mugv suitev 白墙v PEPv  保护好自己v 刊载v 嘟嘟v ▁TAv  购买方式v  ▁Dispv  ▁puntv 追缴v  经济与v  重点大学v  长按二维码识别v 云鹏v 中火v  全面恢复v  通过视频v  ▁pubsv  ▁Marriedv illaryv  Submittedv 万件v  ▁revivev  ▁overwhelminglyv  ▁tacticv  ▁copyingv 南平v 酸碱v  业委会v  引导群众v  学生的学习v  了一圈v lostv  而不腻v  居民小区v  深度学习v  ▁Cronv  ▁Seasv  ▁dictatorsv  普惠性v  从来都是w  ▁Macronw  搭载了w  ▁Pharmaw  高中英语w  ▁Proofw ;<w 津液w 游子w 钉子 w  好东西 w  下雨天 w ayson w 守卫 w  ▁Abramsw 飞来w  保洁员w  县医院w  ▁Camdenw  ▁cultivatew  ▁astronautsw lzw 计价w  使自己w  经济特区w  ▁crackingw 周全w ▁hymw 杂谈w  守护者w quotew ampingw reich w  有下列情形之一的!w 降级"w  ▁reclaim#w  ▁tractor$w 双侧%w 海面&w 字头'w 丰硕(w 建业)w  种食物*w 党参+w  需要你,w ▁Edu-w  教学资源.w  低血糖/w  斤左右0w  ▁Archer1w 馅料2w  发展前景3w  在今后的4w  ▁faint5w  heading6w 施加7w 果报8w  的舞台9w WiFi:w  干部队伍;w  文章推荐w  ▁Personalized?w  三年级下册@w mAhAw heneBw  主体结构Cw 活成Dw  万无一失Ew  更多内容Fw  女孩的Gw  某一天Hw  紧急状态Iw  自主品牌Jw  ▁rhythmsKw  ▁PortsmouthLw  ▁festivitiesMw  不容缓Nw  UkraineOw 吵闹Pw 心丸Qw  归属感Rw  烟火气Sw  自定义Tw radiusUw 更替Vw 气喘Ww  说明你Xw 丽丽Yw 亲临Zw  的口罩[w STAT\w  ▁Agencies]w  ▁Increase^w 权衡_w 脉冲`w 过完aw ▁KCbw  已达到cw  ▁Carlsondw  ▁geneticallyew  ▁stimulatingfw 蹦床gw  这样一句话hw Pierreiw  ▁bodilyjw  踏踏实实kw  健全完善lw 底数mw jaxnw  加速度ow  第十三届pw  ▁frostqw 对阵rw  司法厅sw Battletw  ▁farewelluw 军长vw 宴席ww  卢卡斯xw  自己的心yw Safetyzw  证书的{w  ▁backbone|w 富力}w 水痘~w REAKw JTw 切尔w  ▁pumpingw  战斗堡垒作用w  Reportsw  ▁Kelleyw 凉鞋w 而立w  桥头堡w ballsw  Irelandw  的巨大w  不合格者w  ▁Funkw acialw laveryw  izophrenw  ▁allergiesw  得非常w ▁Show  ▁objectionw  延续了w  教育理念w 口音w  世界观w  ▁hourlyw 对决w  累计死亡w entonw  ▁homeownerw  Notificationw 只管w  老板娘w  那么好w ▁Zakw  勇敢的w  的依据w  音乐节w  根本任务w 终会w  发挥着w  蕴含着w kidsw 照常w 移风w  养殖业w 涂鸦w  的小白w  老朋友w ▁cmdw Mapperw  ▁Individualsw  ▁examinationsw 网的w idxw  getNamew Polyw ilinw  城市规划w  ▁surgeonsw  工程设计w  皮肤问题w  Supremew 功力w  不过来w  洗手间w IDENTw ulencew  ▁tigerw  ▁evaluationsw ódw 对它w 广为w 立竿w 简报w 游人w 谎称w assetw  ▁knotw  是一个很w  ▁Empowerw  ▁turbinew  ▁organisew 安放w 战中w 石的w  劳逸结合w  ▁parapw  经济社会发展的w huangw  可关注w 大举w 药水w 霉运w  的病例w 挺直w  为进一步加强w 波折w Luxw  检测出w  ▁Pokémonw  女子的w  立交桥w  视频课w  这一段w boldw  天气变化w  工程量w  和你一起w 暴击w 甲基w 面容w Analw  ▁Combinedw ▁fnw ▁INTw  ▁reinventw  ▁MacBookw Tyw  种植户w  ▁misusew 乐高w 他了w Statesw  ▁inlandw  ▁parserw  ▁warrantsw  手脚冰凉w turnedx  ▁Clausx  ▁tunnelsx  ▁Curriculumx rasex  如果能够x  ▁savvyx 一题x 东台x  主要针对 x 哈里 x  以此为 x  的增加 x ▁ost x  周围的人x  武夷山x bossx  切身利益x  为客户提供x  ▁slicesx  ▁somedayx  医疗费x uestx  关心关爱x  ▁aislex  的共同x  ▁degenerx 彼时x 自用x 讴歌x ▁Abex 上街x 跌宕 x 病了!x  像一个"x  笔下的#x  生活水平$x  立即组织%x  ▁specifics&x  ▁Colors'x  ▁revisions(x  白岩松)x  ▁Peer*x  ▁intrigued+x heat,x ▁Dud-x  公益诉讼.x  梦想着/x intern0x  Servlet1x  简单地2x  ▁Infl3x  Excellent4x 推算5x  平方米的6x 四舍7x 自取8x 镇定9x  任何形式:x  澳门特别行政区;x 免受x  好好地?x  ▁Desktop@x CouAx  开发和Bx WolfCx  奉献精神Dx  ▁PrelEx airobiFx 兜兜Gx 行踪Hx  出国留学Ix  宁德时代Jx gmKx  ▁pensionsLx  更有效Mx  ▁SageNx  ▁contingentOx 手势Px  致癌物Qx  随心所欲Rx quantSx  ▁wastingTx  ▁recklessUx 学龄Vx 投射Wx 板书Xx 母爱Yx 翻看Zx C <城市网邓州城市网邓州城市网邓州城市网邓州[x мы\x  人物的]x 注视^x  esthesia_x  ▁proliferation`x JKax 和有bx 课表cx 那次dx  需要一个ex  ▁Bartonfx 正轨gx 藏匿hx 半夏ix  ▁cultivatedjx 差评kx 某在lx  解释说mx  放入锅中nx prefixox  ▁attentpx 人所qx 每条rx  一次性使用sx 满地tx 停药ux  ▁popevx  ▁solitarywx 滞销xx 新星yx  被隔离zx ▁lar{x  新增死亡病例|x  克拉玛}x  看到这~x  签署了x  现代社会x 丑陋x 晴朗x Banx  也许会x  决定将x  有序的x boxesx enumx 打捞x  ставx  ▁Lauriex  Regardlessx 兰特x 函授x 得着x 或其x Bedx  铁路部门x  示范村x  三次会议x  ▁Componentx  ▁broadcastsx 后继x 园的x  的一片x  ▁Soldiersx 乳汁x 我先x ▁Tetx  ▁LEGOx 发家x 必究x ovyx  分配到x 余亩x 您在x 核减x 公婆x 熄灭x 满头x scrollx 鄱阳x  极好的x  目前来看x  ▁proclaimx  而作为x 喝到x  ▁Boutx  otswanax  ▁fisheriesx 包机x 月圆x  意外的x  大年初一x  平均气温x  我们身边x  提前预约x  中加入x 口苦x zzlex  引进人才x  ▁distractionx 维尔x  视频讲解x  ▁Wincx  ▁Newspaperx 还曾x DHx 显露x 暗中x originx  ▁crippx 卖场x  广告投放x  ▁unrex 新歌x Flexx ▁PETx  ▁Ottomanx  ▁energiesx Mix 一撇x  ▁Oncologyx abeticx  ▁Acquisitionx 发散x 参评x  她没有x  研究制定x 牛角x  取得的成绩x 布料x weedx  如何预防x finderx 尼古x 极好x  财富自由x ultonx 了下x 分摊x LGBTx  教师岗位x grox  做任何x  开好局x  一直处于x  宣讲会x  政务服务中心x 之士x 在村x 常青x 采食x  中成药x  分手后x Bootx  让人感到x  ▁Jenningsx 王道x  交汇点x  半身裙x  秦皇岛市x  安全员x  ▁snappedx 劳斯x 汇演x  即便如此x 早春y  小游戏y  最深的y  ▁Nguyeny  ▁preventivey  植物的y 展销y 酒庄y  与环境y Remote y *; y  但是也 y  投诉举报 y 学过 y 迷雾y  各岗位y  发生重大y  合同制y  所以就y  如果不想y athiy  报告制度y  ▁furnishingsy  不知所措y  的发展思想y  ▁doublingy  ▁contestantsy  出生后y  ▁clery jiaoyuy Copy prony ▁sar y  名列前茅!y  近段时间"y  ▁Naomi#y  第一附属医院$y  万多元%y  与自然&y  的三大'y  ▁sequencing(y 才可)y 精挑*y 共设+y 馆长,y  头像壁纸-y  Transfer.y  ▁Minimum/y  ▁breakout0y  先进工作者1y  ▁ranged2y 元和3y  实际操作4y  该岗位5y iator6y ussen7y 水岸8y 西药9y  了一波:y 伤官;y  一大波y 云霄?y 别致@y 尺码Ay 雾化By  听党话Cy  该不该Dy  这种感觉Ey BrexitFy  北京市房Gy  ▁SimsHy  ▁MarriottIy 奔走Jy 漩涡Ky 贴息Ly  一周内My ippleNy  ▁SharedOy 李宁Py 永辉Qy  ▁spousesRy 登顶Sy 神情Ty  能适应Uy  对项目Vy  通报称Wy  其他平台Xy  制作了Yy  适用法律Zy  ▁trustworthy[y OTC\y  ▁Byron]y  ▁introductory^y  ▁Shak_y  越野车`y  喜怒哀乐ay  特此通告by 周报cy  科技馆dy Daysey  ▁Robbiefy  放射性gy  ▁stubbornhy  ▁itineraryiy 分清jy 待发ky 心火ly  但不要my  的心声ny  ▁Wheelsoy  ▁motivepy iziqy  ▁pupilry  ▁Seminarysy ▁OTty  旅行史uy ▁Kyrvy Bellwy  中南大学xy MATyy  的人都知道zy YOU{y 视作|y  ……(}y  这个话题~y 上皮y DELy  ▁monety  ▁stunnedy 致谢y 玉兰y 苍南y  洗发皂y 无损y 烟民y "],y avoy 我便y 生蚝y  ▁panoramicy 扑克y 点子y  你就能y Ratey igary  ▁Costsy " 第七次全国人口普查y 跌破y  恶劣天气y testsy 长年y 劲爆y 服刑y veredy  ▁roofsy Lity  创新中心y  如果有人y  ▁cavesy 乌云y 酗酒y 出圈y 无痛y 杨振y  分辨率y  大放送y  ▁Chicky 战的y 阳痿y ктоy  粤康码y plainy  ▁villagersy 倔强y 煲汤y  无形中y " 国民经济和社会发展y  ▁Explorationy  挡不住y  融入了y 网名y 面授y  又或者y  艺术团y  卡机上的y vasivey  ▁Reveny  ▁insurery  白细胞y  通知为准y 世故y 他乡y 蒸蒸y  安全出行y  预约电话y  ▁alliedy  ▁BGCOLORy  ▁restsy 在有y 落落y  上半身y  染色体y 品相y 授信y  对象为y 栓塞y ICCy  张家港y  增强体质y 诱因y 说好y  ▁ripey  ▁nurturey Ruy 越小y манy  工学院y  春季学期y  ▁Middletony  ▁Cynthiay 排练y 配制y  舍小家y  ▁Coloringy  ▁Personallyy  覆盖了y 暖阳y ujey ▁iby  ▁communicatedy 水火y  几乎都是y  招聘简章y  regationy  具体操作y  ▁tappedy 就问y ▁jogy  交流合作y  ▁bucksy FYy 无房y  ▁Cancely  ▁Coachingy 亚于y 动听y  ▁burgly  ▁vapory  五千年y  工匠精神y  internationaly  ▁interviewingy 偏僻y  十五条y  萨克斯z  最新情况z  如果一个人z Reallyz 四分z 着色z  可以随时z 用小z lobz  暖心的 z  的结合 z  共产党人 z elsius z  ▁chromos z  谁都不z  阳光房z  可能影响z  ▁objectionsz  ▁arraysz  ▁Neurologz 逐个z iciaz 敌方z  的介绍z  田间地头z  ▁resonancez  traditionalz  李子柒z  胃黏膜z  企业提供z cjz 年迈z apis z  ▁Kosovo!z  ▁redeem"z  ▁elevate#z  ▁feasibility$z 光纤%z 天线&z 派人'z 翻新(z  规模最大的)z 五入*z 有变+z  ▁skew,z  ▁unpleasant-z 还行.z nite/z  立竿见影0z  ▁lateral1z itsu2z  ▁Killed3z  ▁Strugg4z owo5z ▁aan6z assium7z 干道8z ▁CDT9z ▁pac:z  ▁Illum;z 静待z  ▁therapists?z  健康问题@z  ▁lordAz 波音Bz 皇上Cz  将无法Dz IllustEz '　Fz 头号Gz 案子Hz lestIz TweetsJz  ▁reiteratedKz  在南京Lz  当一个Mz  自己有Nz  ▁JaguarOz  ▁SuitesPz  把它们Qz  各式各样的Rz  ▁allegesSz  ▁EthereumTz  ▁pedestriansUz 逾越Vz  ▁symptomWz  ▁coherentXz 拼读Yz 江海Zz ▁Mé[z  进程中\z 咸鱼]z 宜人^z dash_z  天以上`z  ▁resolvingaz  ▁stationedbz 双全cz  对其他dz  Generatorez 难堪fz IZEgz Corthz  ▁Erieiz  ▁unsignedjz  责任意识kz  ▁Ninjalz  ilibriummz  ▁imprintnz  ▁Regardingoz 入行pz  ▁wanderingqz  ▁coworkrz 心间sz  亲访友tz  ▁Probuz  ▁florvz  ▁discswz 过山xz  带动作用yz  的原因是zz  雷克萨斯{z ◐◑|z 胀痛}z  一起看看~z 无奇z  介绍的z  推荐表z  如此的z  ▁Sickz 老酒z 要来z  实验幼儿园z  ▁antibioticz  ▁Taraz plantsz  ▁chickensz 就打z iniaz  ▁missesz  尝不是z 到此z ▁Lalz  科研机构z ABOUTz  寸步难行z  ▁tossedz  ▁illicitz ullaz  ▁beltsz  ▁Nonethelessz  ▁constituencyz 乌黑z Lasz  理解了z  版权归原作者及z  幸运的z  谋发展z gottenz ninez 战火z Rogerz  inishedz 噢噢z 我曾z ▁takz  能力建设z  ▁hashtagz  成了一个z  ▁inaccuratez 压岁z  提名为z Allenz  ▁Elliotz  上门服务z  ocatingz  ▁Dumpsterz 想像z  一般会z  不敢相信z  信息显示z  ▁Sealz  ▁Victimsz  等不良z  如果可以z  市卫健委z  ▁numericalz  办理的z  商品的z  准备好的z  ▁witnessingz  经理人z  特别声明z 意在z  中写道z  胜利的z egenz  ▁Mastz  作用于z WAREz  考生本人z  ▁poppingz carsz  国家规定z  思政课z ▁Nemz  的新时代z  ▁Alexisz  行李箱z  实名登记z Coachz 古村z 脏乱z uningz  民族复兴z 圈粉z  人民调解z  ▁Pulitzerz  ▁circulatingz 外用z  展览馆z  当然不是z  检验标志电子凭证z  ▁Hardwarez  ▁liabilitiesz  ▁Parliamentaryz 银子z  ▁Niagaraz  扫码加入z  ▁lowsz 为非z 大江z 审校z ckez  摄影作品z  ▁Extremez  ▁keeperz 不免z  对一些z  院子里z  使用日期z  事发后z analyz 后门z  流行性z oirez  ▁Kinderz 发量z 谷子z  努力把z  辨识度z  ▁pastoralz  余人次z  等设施z lichen{ 大可{  为载体{  形象的{  请及时联系删除{ 均以{ 援疆{ 病程{ ▁Bid{ ldon { rews {  心理测试 {  社会影响 {  进行排查 {  ▁Mens{  ▁Knowing{  ▁altar{ 履历{ 爆香{  西北地区{  ▁Chic{ Nobody{ 题名{  降压药{  体温正常{ 小宝{  平均每{ glers{ 上场{ 中止{ 纸箱{  商业化{ Phones { 贺州!{ 缝合"{ 雨林#{  ▁discern${  ▁Expansion%{ 菜刀&{  欢迎您的'{ 平南({ 私教){  报考的*{  新员工+{  ▁theor,{  ▁Deadline-{  自我隔离.{ 介于/{  国庆假期0{  移风易俗1{  ▁scarce2{  学起来3{  给了他4{  ▁melodic5{  ▁collateral6{ 三两7{ 叫他8{  没想到的是9{  ▁stagn:{ 不前;{ 台独<{  学子们={  等区域>{ 立在?{  世纪的@{  各不相同A{  ▁necklaceB{ 省略C{  卡就是不D{ MoviesE{  ▁ObamacareF{ 排忧G{ 数年H{  思维和I{  波士顿J{  这可能是K{  MinisterL{ etiM{  自然科学基金N{  ▁sensingO{ 圣母P{ 家访Q{ ucesR{  ▁mundS{  aspberryT{ 某事U{  带动了V{  也不愿意W{  既然是X{ 出狱Y{ 留香Z{ ▁Bj[{ ▁IB\{  先锋队]{  十分重要^{  所在地的_{ 同框`{  时表示a{ ▁Noub{ 会商c{ 废气d{  ▁Alonee{  疫苗接种工作f{  ▁rainforestg{ 蒸馏h{  以下内容i{ 滑板j{ 硬的k{  ▁Puppl{ movingm{  商贸有限公司n{ 煽动o{ arthp{  就没了q{  ▁inspectorr{  ▁concessionss{ 简谱t{ 趋向u{  ▁deductionv{ 树牢w{  及时将x{  隐藏着y{  征集活动z{  请联系后台{{ 年时|{ 腿型}{  会继续~{  很舒服{  疫情发生后{  ▁unhealthy{ 产前{  :①　{  会逐渐{  esville{  ▁characterization{ 丽萍{  ▁longstanding{  耳熟能{  ▁sickness{ 手无{ 新店{ 见于{  单选题{  和重点{  对全市{  放下了{  ▁Bundes{ ogi{ ▁HB{  可再生能源{  不胜数{ kang{  ▁Remem{  ▁disadvantage{ 弹奏{ atsu{  如果一个{  ▁Lank{  ▁Wand{  ▁Cheese{  在疫情防控期间{ 购药{ 鸦片{  以上信息{  相关工作经验{  ▁Triangle{  这也是为什么{ 天台{ 狩猎{  的理想{ Phys{  精准防控{  ambién{  ▁separating{  ▁provocative{ 接吻{ 不二{ 临街{ 牙痛{  ▁flats{ 老张{ érie{ 沉沉{ 闭塞{  这段话{  关注订阅{  ▁Carpet{ 令我{  有太多{  黄晓明{  百度网盘{ 增城{  ▁COND{  ▁Djokovic{ 微妙{ /#{ 主板{ 多于{ 锦衣{  陈老师{  ▁asserted{ 轮换{  在企业{ 双赢{ 豁免{  张伯礼{  效果的{  类风湿{  boolean{  ellaneous{  系统学习{  ▁Omega{  少聚集{  有大量{  ▁Liberia{  ▁Theology{ nat{  不适宜{  在本次{  防汛抗旱{  ▁Canterbury{ 水洗{  这种事{  向上滑动{  秋季学期{ ▁ц{ 警觉{  专委会{  亲人的{  做错了{  ▁Burk{ 均需{ 议会{ 采编{  和妈妈{ 友爱{  始终没有{  ▁brilliantly{ 白虎{ ▁Roe{  有机结合{  同等条件下{ 实况{ 幕墙{ 手软{  耳熟能详{ 仁和{ 癌变{ 选的{  ▁paddle{  ▁Borders{ 自成{ bek{  在欧洲{  ▁geometric{ 中环{ 迎风| 口口| 早熟| 海归| 精辟| 老二|  刻不容缓|  最近几年|  深刻领会|  NSString |  ▁primitive | 长官 |  的调整 |  自己想 |  为人处世| 无惧|  仙游县|  脚手架| ▁ell|  ▁enhancements| 蒙城| 面食|  专业性| 再难|  不经过| Pret|  得到控制|  ▁expressly|  ▁electronically| 不忙|  人事处| wolf|  黑社会性质组织 |  ▁Pharmaceuticals!| 大字"| 新郑#|  投资建设$|  ▁Speak%| ▁tan&|  大大提高'|  武器装备(|  并主动)|  ▁Arist*|  ▁slick+|  ▁Meteor,|  ▁Franchise-| 却要.|  改善了/|  conference0|  污染的1| Intel2|  wrapper3|  ▁regimes4|  最主要的5|  ▁sidewalk6| Tu7| rf8| GEL9|  ▁advancements:| ynt;|  日新月<|  突发情况=| 买点>| 鼓起?|  也有可能@|  普通家庭A|  管理部B| 奥体C|  ▁beadsD|  ▁WorcesterE|  必须的F|  补习班G|  落实落细H| CyberI| 优厚J| 锂电K|  འི་L|  一无所M|  以满足N|  哲学家O|  动脉粥样硬化P|  ▁silhouQ| 两车R|  出一种S|  加班费T|  这一问题U|  也正是因为V| 之灾W| 木马X| 此日Y| 深有Z|  疫情信息[|  给我们的\| Behind]|  ▁recruits^| 李清_| inform`| 开篇a| 莫属b| PVCc|  在生产d|  的真正e|  ▁pedalf|  卫生健康部门g|  ▁portfoliosh| 东昌i|  ▁Involj|  ▁smiledk| 佛系l| 囤货m| 小路n| 相看o| ewaysp| atchesq|  瑞丽市r| 美剧s| U N▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁t| 万套u|  十堰市v|  郑州大学w| circlex|  ▁disparitiesy|  ▁Guangz|  ▁grocer{|  ▁vanity||  ▁Thousands}|  善良的~| ilio|  中考成绩| 仕途| 山头| 开悟| 放慢|  发布通告| gren| sales| "'|  Selection| 寨村| 由其| ▁',| ▁Kem|  ▁Steele|  ▁dumped|  ▁reopened| 姐弟| 捕鱼|  ▁Intent|  ▁youtube|  坚持以人民为中心| 境遇| 王伟|  地中海|  教育行业|  违法时间|  我们每个人| 大德|  ▁Stevenson|  全因子| Fill|  ▁specials| 主宰| 施压|  效果更好|  这张照片| Built|  ▁INTO|  ▁enquiries|  responsible|  本资料|  的思路|  一网打尽|  ▁ethn|  困难和问题|  ▁roasted|  ▁distributions|  不难发现|  电子注册|  课程标准| 上位| 易烊| 邮报|  消费的|  ▁pops| 功臣| 上床| 海里| maybe| 追忆|  发展成|  教室里|  理发店|  也出现了|  还有不少|  ▁Penguin|  ▁mileage|  Officials|  ▁perimeter| 景物| 猜想| Nat|  ▁Scots|  ▁youthful|  ▁synonymous|  Keeping|  消除安全隐患|  ▁ransomware| 水文| given| ocaly|  ▁Okay|  ▁Morton|  ▁upbeat|  ▁corrections| 蒜蓉| DOI|  全面了解| 多亩|  金色的|  ▁Pension|  ▁dispatch| 曹县|  软组织|  ▁Patch|  ▁inventor|  ▁Citizenship| 衣裳|  相匹配|  为了解决|  周边环境|  ogenesis| 分了| 护佑|  坚持问题导向|  ▁Pocket|  ▁accumulation| ▁Yas|  含有大量的|  dominal|  querque| ighs|  停电线路|  ▁Judith|  陪伴孩子|  据不完全统计|  ▁Janeiro|  的青年| 灵气| 配菜|  三个字|  的直接|  ▁Arabian| goal|  ▁Moments|  ▁Provide|  ▁indications|  部分文章推送时}  却一直}  家人们}  直播中} 下放} ORTS}  三十六} loved}  ▁Ethiopian} 高僧 } Corn }  住院治疗 }  ▁Mormon }  任何形式的 }  ▁informing}  ▁plac}  ▁nickel} 恍惚}  感谢信}  windows} 自达}  多一个}  少部分} liga} ungal}  ▁Luna} Models}  ▁stringent} 引出} 调出}  看到这个}  随着时间的推移} 地问 }  的真相!} uzzy"}  作业人员#}  西部计划$} guang%} 初赛&} 房型'} ochond(}  ▁reaff)}  ▁peripheral*}  ▁stimulation+} 契约,}  但如果你-} 抄底.}  纽约时报/} 吃透0} 怜悯1} stv2}  ▁wary3} ▁Ч4}  防晒霜5} acting6} eff7}  好运气8}  聪明人9} jected:} 如花;}  及时删除<} 长生=}  ▁innovate>} 歌剧?} 谋求@}  学到的A} 迟钝B}  关注了C}  ▁facilitatesD} 印有E} ousseF}  ▁CatalogG}  ▁unravelH}  ▁identifierI} 一叶J}  了大家K}  停放在L}  要么是M} AdobeN} 金森O} xingsP}  ▁diversifiedQ} 喷泉R}  无不良S}  甚至比T}  averageU} 享誉V} 白带W} metalX}  ▁stuntY} 手慢Z} 泡面[}  超高的\}  如果你们]}  ▁bere^}  好一点_}  学生证`}  校园内a} 沸水b}  在外地c}  就是从d}  扑面而来e} ▁Bacf} soleteg}  二级建造师h}  污水处理厂i}  后可以j} ientek} 一盆l} 寓言m}  ▁Noeln} 后市o} 杨晓p} 民安q}  四舍五入r}  ▁Editings} 连用t} 郑某u} PCAv}  免费通行w}  粉丝的x} 尚书y} Taiz}  连接线{}  隐藏在|}  有人认为}} 方舟~}  向世界}  研究机构} 则有} ▁dod}  fashioned}  ▁Genetics} 拍手} ▁GS} facts} 住着}  自主学习}  自主选择} mouse}  ▁penetration} 总和} 援引} 班里}  驱逐舰} gj} 本体}  总冠军} qz} 那人}  ▁SpaceX} 触感} сту}  ▁Direction} 可耻}  的景象}  进一步规范}  的小说}  就相当于} 将士}  下个月} INST}  upuncture} 拿证}  缓解了} combe} Muslim}  busters}  ▁gossip}  ▁Liberation}  你不会}  锦上添花} dragon}  pointer} 刷爆}  汽车有限公司}  ▁rehearsal}  的疫苗}  ▁amusing}  我这个}  落后的}  ▁waking} 认错} зе}  校团委}  平面设计}  这方面的}  Compare}  ▁Symbol}  提供便利}  ▁Punk}  ▁childcare}  ▁prominence} 首乌}  充电器}  证据证明} olding}  ▁deeds} 咫尺}  Commission}  供电所} 小猫} 灌木}  控股集团}  第四季度}  ▁heel}  ▁chaired} 依山}  白癜风}  房贷利率}  也很重要} Aside}  ausible}  ▁Definition}  ▁constraint}  ▁redevelopment}  的红色}  王王王王}  withstanding} 读研}  更重要的}  需要帮助}  人员信息}  公务员法}  ▁Pale}  微信朋友圈}  过程的}  高考生} 相冲} ▁OC}  ▁jihad} 工装}  的利益}  你的未来} parts}  ▁uranium} ↑　}  诞生了} yield}  ▁Españ}  ▁eater}  ▁neurological} 之高} 甘愿} 篡改} 他俩} 迎着}  比比皆是}  欢迎广大} 避难}  ▁dynasty} 三水} 垫付} 庇佑} 想得} 私聊}  万以上~  担使命~ 刚烈~  请家长~ elif~ 价廉~  外三元~  文件的~ Chen~  疫情发生 ~ 重创 ~  健康信息 ~  社会发展 ~ 日经 ~  ▁lace~  ▁busiest~  ▁galaxies~ 满腔~  初试成绩~  ▁scarc~  ▁resumed~  桥街道~  结婚证~ 故而~ 身形~  心理压力~  ▁Loud~  ▁beneficiary~ 玩水~ 长处~  ▁Philly~  ▁disconnect~ 早泄 ~ ▁ye!~  剩余的"~  备案表#~  我今天$~ ashire%~  ▁Fabric&~ 五日'~ 路西(~  utation)~  ▁Astronomy*~  ▁transcend+~ 平定,~ 菜场-~  ▁Kremlin.~  ▁stumbled/~  到北京0~  很严重1~  一路走来2~  ▁Vall3~  ▁wiped4~ czy5~  行程轨迹6~  ▁pneumonia7~ uu8~ orb9~  曾经说过:~  正常生活;~  较小的<~  高考数学=~  模式下>~  第二类?~  Boolean@~ 红码A~ HumB~ UriC~  ▁EdisonD~ 各司E~  ▁PietF~ 武林G~ 青色H~ 高利I~ 等非J~  支持力度K~  ▁LimaL~  远方的M~  ▁hurryN~  ▁jerseysO~ 长痘P~  杂交水稻Q~  ▁ExitR~ 国风S~ 无果T~  和努力U~  ▁rapedV~  ▁VolunteersW~ 福彩X~ 联军Y~  后疫情Z~  学科的[~  Arizona\~  ▁Implementation]~ ('/^~ ème_~ 无休`~ Cancera~  ▁softballb~ 本场c~  复杂性d~ aguse~ ▁SDKf~  ▁trapsg~  Missingh~  ▁resenti~  ▁spiderj~  微循环k~  来势汹汹l~ 提档m~ 着她n~  结果为o~  不懈努力p~  ▁Killingq~  ▁reactedr~  ▁Biblicals~  ▁brightnesst~ 阜新u~ Halv~  不对劲w~ CPIx~  ▁Listsy~  Identifierz~  管理机构{~ OST|~ uca}~  劳动保障~~  可爱的小~ 里说~  的解释~ 两场~  就业困难~  ▁TABLE~ 懦弱~ 铃木~ ▁NIH~  加班补助~  ▁togg~  ▁blunt~  ▁relic~  精神文明建设~  北部湾~  ▁Asked~  ▁majestic~ 舰艇~ 鹏飞~  民航局~  作出新的~  坚定信念~  ▁concise~  多方面~  ▁mornings~ 来吧~ 高歌~  开学季~ ▁FAA~  ▁recharge~  十二月~  考试前~ Pri~  对人的~  写下了~  司法鉴定~  培训的~  一丝不苟~ : 3▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁~ 出纳~ 闺女~ 农庄~  某公司~ ▁MAG~  相结合的方式~ 失恋~  获奖证书~  ▁dreaming~  ▁combustion~  促进学生~  ▁Perspectives~ 文静~  与众不同的~  ▁gestures~ 几人~ 辛酸~  分不清~  初中学校~ 五楼~ 拭目~  ▁condemn~ 北街~ 食道~ ▁DH~  ▁Augustine~  ▁hepatitis~  ▁Presentation~ 脑力~  集中力量~  ▁fist~ 博学~ 雪峰~  岁高龄~  ▁trait~ 人命~ 居于~ 生熟~  的光辉~  ▁Narendra~  ▁prosperous~  ▁giveaway~  ▁experimentation~ 以无~  中间商~  做核酸检测~  ▁Therap~  ▁Macedonia~  的那一天~ 多花~ 烂尾~ 睡衣~  ▁dude~  ▁Gupta~ 伤的~  ▁сво~  以下拘留~  ▁totaling~ 晓得~ 独行~ 结账~  进项税额~ 那只~ 游离~  第一反应~  那么多的~ 收了~  年全年~  发生碰撞~ 不配~  ▁Quiz~  ▁Watching~ asan~ ansen~ adores~  ▁Factors~  ▁Meadows~ 和交~ 遗弃~  请尽快~ 秀色~ 缓缴~  后备箱~  戴着口罩~  ▁focuss~ 动容~ 暴雪~ 李国~ 驼背~  我们今天~ 牧民  有人会  日新月异  提醒广大市民  ▁finite  ▁entrust 这把  ▁Brow 医养 熔喷  留有   的作者   说了算  初见  打消  以上图文  ▁teaser  ▁Weinstein 玉器  文学作品  ▁Levine 基坑  百分制 acha  ▁CNBC 办的  办公场所 欧亚 热敷 导线  技术研发  ▁overr 春花   阳光的!  隔离病房"  要达到# ituary$  ▁financed% 抽搐& 拉的'  刃有余( 正定) 进校* nuts+  教育督导,  ▁Cater-  ▁airing. YA/ 各处0  个多月1  温馨的2  ▁circumstance3 tf4 找来5  不适应6  造成伤害7 基点8 羞辱9  口中的: 神咒; 黑茶< ▁lol=  变成一个>  ▁Huss? umbles@  isexualA 举世B MPVC 王氏D  讲到了E  ▁addictiveF 涉外G  材料和H  ▁synthI 还用J  再出发K  能满足L  ▁DeliverM 蜂群N  ▁TempO RXP  建设规划Q  监控视频R  ▁avenuesS 主语T  是实现U 降费V  消费需求W  编辑的文章X  terrorismY 先看Z 趴在[  届高三\  适合你] ▁Rox^  ▁poorest_ 五点` peza  原产地b  了一会儿c  ▁Recapd  ▁Clearlye 巴尔f  转自网络g 急事h  土地管理法i  buquerquej 别有k  显著的l  学科带头人m  ▁pesticidesn  ▁Reddito 引力p 房地q  安心的r  宝宝们s  情节严重t  ractiveu  ▁Unifiedv  ▁rubbishw 精打x  补贴政策y 陈家z  还不到{  以自己的| uddle}  ▁compulsory~ 井盖  ▁Vacation 磷脂 ▁SSL 织密  ▁Crest 发包  ▁Visitor 畅快 硬朗  绥化市 oyer  ▁Israelis 数和  专家评审 Edited  Madison Bot  上线啦  无害化  ▁earthquakes ;⑤  ▁Funny  ▁versa  工作人员表示  ▁Monkey  可转债  环境好  环球网  ▁torrent 条条  时间另行通知 寄递  养生的  可自行  政策法规  服务企业  ▁Scripture 时已  新鲜出炉  留学服务中心  ▁scaled  ▁blending  ▁formulation  及个人 ▁Mys  工作总结  规模以上  ▁Friendly  干部的  恢复开放  绿水青山就是  ▁Burlington  被警方 李四 ánd  嫌疑人的 涂上  前女友  后才能  可惜的是  ▁Julius 不干 之隔 亲测 白金  发放的  集中精力 Veter  Actually  ▁doctorate 几步 加试  塌糊涂  很清楚 oine 打新  只用了  ▁від 变小 坐车  ▁Barker 芳菲  更加注重  ▁smoked 双边 身家 一式  遇到一个  ▁legitimacy came icidal  ▁screenshot 尝到  在确保 kish  受益匪浅  对得起 ▁sta  ▁blat 见人  周年之际 切口 怪的 砍价  ▁Berk 单亲  至少在 ལ་ 玩得 耳光 Desk  ▁cosmic  消化的  ▁ultrasound 公检 告急  按程序  ▁extras  ▁Announced  ▁adulthood  ▁validated  理工类  定点医院隔离治疗  ▁Counseling 马伊  ▁curl ▁coq  ▁filmmaking 其职 而入  医疗中心  ▁suppression  ▁Marijuana 市镇  远高于  ▁influencing 有云 针刀  ▁Sigma  减少人员聚集 Rail omens  ▁Abbas  ▁shouting 硬盘  已经没有 Centre 建安 黄素 wor  下载打印 Oak  将不再 教于  并保持 orno  努力打造  ▁Bolivia  适应症  ▁Gentle 佐证 德县 较早  提升至  脱口秀 omorph  新冠肺炎疫情影响  中华儿女  printed  ▁citations  有光泽  宣传贯彻  建言献策 ør 数月 文集  目前国内  酸甜苦辣 itched 可留  为了确保  目前我国 一哥 Issue imates ocon  рова 披萨 苏醒  年纪大了 但未  的一声  ▁Turns  为深入贯彻落实 得心! 虚空!  可以快速"  民族团结进步" 患癌#  做好的#  拭目以待$  月份以来$  ▁downturn% 对峙%  新年快乐&  ▁hairst&  ▁Andreas'  foundland' 怀化( MAR(  确诊人数)  ▁Closed) 电白*  商家地址*  提出要求+ 仅售+ 当红, 收件, 诽谤-  获奖作品- ymoon.  ▁fishermen. 一枝/ 募捐/  然后就0  门诊部0  工作生活1 sites1 身手2  对党忠诚2  尽管如此3  海鲜市场3  ▁blows4  ▁openness4 "{5  ▁siege5  ▁instructional6  张艺谋6  ▁notions7  ▁derivative7 很早8  怪不得8  ▁vocational9 佛学9 鱼塘:  快速增长:  环球时报; 离场; 威信<  我们以<  ▁improvis= Else= aviour>  ▁roadmap>  公共基础知识? 停用? 单打@ 见不@  行政区域A 解暑A  血液中的B  针对不同B  ▁debatedC 越低C  与父母D  进一步加大D  行动上E 信件E 汗液F кийF  中英文G  ▁meltedG 办了H  独立完成H  ▁BergerI  ▁freezerI 而复J  担心的J  一方面是K  ▁thruK 北流L  公费师范生L  ▁delicM  ▁radicallyM 堵点N  真的能N  等地区O  ▁finalizedO 欲罢P 歼灭P 等职Q  一模一样Q  ▁decreeR 小青R ▁GWS  ▁grilledS 圆的T  新起点T  ▁SupplementU 宫寒U  真的要V  战略合作协议V 姑姑W (__W  增加的X  ▁disruptedX 大千Y  的确是Y  特朗普的Z  ▁fluorZ 兼程[  ▁Kemp[  ▁Signature\  ▁Therapeutics\  ▁obliged]  ▁realism] 片中^ 血亲^ ▁Ved_  ▁villains_ 尿频`  公示期间` 王安a 路易a ambob  ▁Buyingb  ▁eagerlyc  ▁integratesc 成形d  创造的d  帅气的e  ▁Ellee  ▁Toysf 威慑f  ▁emptg 健壮g 女工h 闭门h  核潜艇i  音乐的i  乱停乱放j 京都j 盾构k 西青k ▁PFl  天气炎热l  真实有效m 溺亡m  ▁Covenn  ▁drasticn ▁Zaro  延年益寿o  艺术类专业p posingp 捡到q  助教老师q  增值税发票r  笔试准考证r  轻而易s ▁Hofs  ▁Vampiret 新街t 电容u  伙伴们u 行医v  每人每年v 放贷w  ▁Nigelw  ▁Commitx  ▁Manufacturersx igeony  ▁Cybersecurityy  冲击力z 被你z  ▁commissioners{ 临其{ 常备|  ▁mais|  ▁crystall} ammu}  易烊千玺~ ussia~  ▁Counties  和环境  统一思想 逆风  主要在  个人缴费  就会产生  ▁Apartments 家训  ▁Monetary  ▁Dangerous  更是一  ▁Ministries  ▁residences 室温  充分调动  规定的其他 文革 插手 acio ▁Kro 多酚  招生入学  未经核实 Train  药品监督管理局 ▁GEN  临床应用 阅兵  的原理  高考的 月嫂 ▁BET  而且还有  ▁apparatus 美誉  及配套 ▁ENG  管理措施  informed  messages  ▁Waterloo  ▁outrageous 奉行 盐和  经营权  地下车库 % 公安机关交通管理部门 ▁½ 加法  已经连续  柔情似水  Britain ▁CU  ▁regeneration 满堂 Rab  你为什么  ▁Transl 五里 平果  斜对面 华庭 大话 饲喂  考试卷 Tools 作词  学以致  公示时间  电子设备  ▁sobre 擦干 PAS  针织衫  ▁terribly  你还是  日复一日 бли  ▁Mama  也不少  暖暖的  ▁Mineral 工整  ▁Destiny  心里有  杨老师  采访了  ▁tuber  ▁Scholarships 守着 谚语  小的时候 让利  起好步  ▁infrared + $事业单位公开招聘工作人员 清空  ства  扫码报名  提起诉讼  查看公告 重申  ▁Dairy  大大降低  ▁slept  direction 惧怕 EPC ▁CN  有计划  脑出血  ▁Rockets 吸水  体育类  年我国  ▁Thorn 长高  ▁Lori 朗朗 春江 言传 详询  新气象 附赠  ▁Bram  ▁Rear  ▁Roses 背靠  该女子  ▁Sikh  telling RUN  宿迁市 reve€  轻而易举€  ▁JungleÀ EtÀ 我真Ā vezĀ LOCKŀ  ▁Daddyŀ 音箱ƀ  在印度ƀ  大熊猫ǀ  人们的生活ǀ  ▁specialisedȀ  的首选Ȁ  聚在一起ɀ  ▁ratiosɀ 游学ʀ  岁之前ʀ  疫情下ˀ  更不是ˀ 亚麻̀ 精粹̀  的款式̀  为了不̀ ▁Wow΀  政治生态΀  ▁Socialistπ  ▁Collaborativeπ 伶俐Ѐ  的交易Ѐ  体育学院р  ▁Meetingsр  ▁calculatorҀ  工作坊Ҁ  点一个Ӏ ▁JabӀ  ▁revelationsԀ 眼见Ԁ LABՀ  violentՀ 天命ր  注册的ր 佛祖׀ 头孢׀  ference؀  ▁Royals؀  守护着ـ  篇幅有限ـ  和无症状感染者ڀ  ▁Parsonsڀ  　”　ۀ  就是想ۀ  itating܀  ▁Selling܀  ▁allergic݀  ▁spontaneous݀  第一节ހ  就在这里ހ  详细询问߀ 百名߀ 表里  Playing  ▁Jensen  ▁contradict 离谱  恢复原价 azers  ▁Satan  Visible  ▁contend 校审  ▁bubbles  ▁visualization 也难 四条 研磨 Labor  ▁Bruins  ▁glowing Craig  ▁filtering 受力 }",  暂行规定 parser  ▁Autism  ▁motives 狡猾  比以前  各级党组织和 的低  扩大到  ▁Fees  ▁trailing 水草 滞纳 yue ▁rav  面试资格  ▁FIRST  ▁Augusta 年为 润泽 艾伦  一盘棋 房里 行间  ▁Stats  ▁drawer GV 鱼龙  学校将  变得更好  科学学院  ▁wird  ▁restrictive 不按 她把  控制性  减速慢行  ▁Byrne  ▁Accessibility 书信 同工 Fan ▁ASP  得到有效  跟他们 espan  ▁IOException 甲级 罹患 较真  被录取 iping  ▁receptors ★★ 万般  或直接 thanks  年以上的 Border  ▁flavours  ▁courageous 平生 一国 腹地 тив  等事项  的初衷 ller ▁Yun  ▁Ashe  ▁Humanity  科研项目  ▁Daytona 悔改 接盘 敲响 Msg ▁Odd 索引 胆子  ▁aided  ▁professions  地理标志 微淼  政治权利  ▁Seek  ▁onsite 总想  代人的  未知的  ▁Bayern  ▁Showcase  ▁chilling 下饭  篮球场 多在  排头兵  公示期为  ▁Brooke  projects 学车 山寨 平息  大理石  第二医院 酱料 fell  ▁marching!  elected! gged" Owner"  Exclusive# 固废# CHA$ Bring$  technology%  ▁assertion% 合辑& 尔特&  ▁Moines'  ▁filtered' WWE(  去的地方( 伊万) 单列)  点击查看大图* 感言*  可以增加+  初心和使命+ Weight, 学时, ignon- 告之- 语境.  全会精神.  性关节炎/ 迷迭/  上了热搜0 重金0  ▁dissolved1 宗祠1  党委宣传部2  ▁informs2 驻华3  如鱼得水3  芥子园画谱4 打气4  与国际5 ▁Ler5  ▁surname6  是不可能的6  Material7 原形7 备份8  迅速行动8 iatus9  ▁Behavioral9  最后还是:  组织部部长:  ▁Atomic; 投向;  不小于<  英特尔<  ▁PHOTO= 诱骗= FIG> Tesla>  Freedom?  文中观点保持中立?  天下第一@  Manufact@  则表示A  开具的A  桌子上B  外出务工B  ▁diarrC  脚下的C  所需材料D “‘D 查纠E  ▁BibliE  ▁slammedF  这些小F  他们就会G  无忧无虑G  的高质量H 韩信H  后来又I postedI  affectedJ  为维护J  团购价K drugK  欲罢不能L  ▁VegetL  ▁spermM 欠薪M ▁KiaN  这么一个N  出版的O  发表在O  ▁BarsP  ▁OccupationalP NESSQ  第一责任人Q  不必要R  会影响到R  质量控制S 大床S  ▁deedT  ▁unconventionalT ▁AscU 三一U 二七V 可于V  ▁bowlsW  ▁LeBronW 中通X  外国语学院X 外援Y  必需的Y ▁GeoZ  但是这个Z  守望相助[  西南部[ 余万\ 报答\  与交流] 送进]  朗读者^  ▁Establish^  ▁Wilmington_ Dance_ IGHTS` fas` ▁Rita  ▁Disneylanda 青筋b  喜欢在b 饿死c  眼里的c  群众性d  费用的d  扶贫工作e  ▁Deluxee 压倒f 汁水f  patterng 并从g wifih  ▁Casah 英杰i  的肌肉i zmanj  ▁matesj  ▁sensationalk  数十年k  免费学习l  脾胃虚弱l Reillym  ▁Flemingm 扫墓n  和保护n  ▁migrateo  要避免o lockedp  ▁footwearp 事成q 药食q  三甲医院r  贷款利率r  Voluntes ▁pess  财源滚滚t  ▁Deutscht  ▁craftsmanshipu  ▁Swimmingu 美到v  学术论文v ANIw 上等w 本的x  学有所x 矫情y 老道y 分至z 暗黄z ▁Til{  再也不会{  ▁Stamp|  ▁Output| MK} 行深} 钢笔~ 从句~ 克林  团队精神  爱好者的  ▁reconsider  ▁Bios  Authors  母公司  如果喜欢就 delta 拉手  及其它  ▁Liga  等城市  ▁Raven  中选择  九江市  但最终  马自达  继续努力  ▁Greenland  我们总是  立即启动  ▁Interface 学工 踪迹  不难看出  环游世界  百年奋斗目标  ▁canopy }{\ ▁TED  ▁detached  对她的  派出所接到  ▁Automatic 鸿燊 Peace  ▁Assist 初选 抓起 ▁CRE 派送 膏药 转诊  也必须  华侨城  新政策 amoto  ▁Pitch  ▁china 李鸿  ▁greatness 大叫  这一块  ▁verge  ▁Exeter 谓之 马齿  密切接触者的 六种 妇产 织物  安全距离  ▁Swim 报恩 翠绿  防灭火 路和 Ori  没有钱 ▁AMA  那么你们 标称 burst 寿险  十一届  ▁Boards 沉甸  一个月后  ▁Stephens  的家人  可以尝试 都由  泰州市 Mayor  后来到  坐下来  这就是为什么 鼻孔  与技术  云闪付  经济实力 内江 山峰  ▁Kara 辉映 ▁HC  提供更  Applications  侵犯到  商投资  ▁Lucia 壮丽  等方法  一张张  自己能 fake  ▁overcoming  游乐场  超出了  师范类  ▁authoritarian 穿行  微电影  也会出现  Variable  城管局 征婚  孕产妇  人群密集  ▁nurturing  移动互联网  ▁coats  ▁Michele  ▁Nicaragua 三字 丽君  带领大家 常考 道法 门关 高照  人才招聘  ▁booklet 佳肴 健胃  致命的Á  问题时Á  ▁pseudā 硬性ā ▁koŁ  ▁grabsŁ  ▁institutesƁ 远古Ɓ XMǁ 喂食ǁ 夸奖ȁ 托克ȁ 评标Ɂ  东北风Ɂ  设施的ʁ  ▁tyingʁ  Unknownˁ 减去ˁ 数万́ 罢工́ laṕ Frań 带病΁  猝不及防΁  ▁Ottoρ  ▁beddingρ  ▁referralsЁ ketЁ Zoomс QUESTс  ▁Riderҁ  ▁Refugeeҁ 凯迪Ӂ 气机Ӂ ▁msԁ  的典型ԁ VertՁ  现场视频Ձ  ▁barredց 入账ց Precׁ  如何看待ׁ acons؁ culus؁  ▁gearsف  ▁Gravesف  ▁forwardsځ  ▁fascinationځ  经不起ہ  邀请函ہ neck܁ 常温܁ 文娱݁ 肥沃݁  宣传周ށ  调整好ށ undi߁  ▁Skip߁  Animation EQ 效仿 Stre  ▁Guru 免洗 随行 oter  绘本故事  本文来源于网络 凯里  这个国家  ▁Oriental 文图  大中型 逊色 Arc raged Bright  ▁BEST  富士康  小课堂  收官之年  ▁Shock 波斯  ▁Bermuda 裂纹  空中课堂  ▁injected 医患  上看到  绿色环保  疫情防控需要  ▁Algeria 共进 新洲  纪念章 审题 打水 琳琅  的念头 nv eph  说的就是 发火 性状 繁衍 并进 待命  的印象  的奋斗  ▁Capacity 为伴  行政许可  ▁Prepare 点此 GMP  走在前  全日制普通 才子 斗门 金花  生水起 流派 眼周 讲过  参保缴费  违法违纪  ▁paramount  Colorado 显眼 euro  ▁teamwork SIZE  公开选调 ły lause  ▁oxid  Creator  接种了  ▁Sentinel 救死 ▁\\  十六条 velyn  ▁elders 茶香  了一座  感染新冠肺炎 ucci  ▁Huang 白费 省际  安全监管  ▁Toll meal  ▁року  很多人认为 私房 ndef  ▁Bacon  市场监督管理总局  ▁carbohyd ( !接受纪律审查和监察调查 gf 一捺 建新 ▁NET ugosl 冒出  可以吃  为了提高  需要通过  ▁summers 深蹲 绿意  一定有  能够有效 arbon richt  ▁cites 胎盘 鸡爪  离我们 巍巍 无刻 ARP  的高中  ▁Claude  ▁biased вет!  开封市!  ▁Thur"  ▁coven"  ▁Postal#  ▁segregation# 诗和$  与合作$  地区有%  相辅相%  高分子& Larry& 捐物'  最低的' nova(  authored( 布莱)  通讯录)  即将迎来* ITA* geb+ Norm+ 小满,  走深走实,  ▁regulating-  某一个-  不容错过.  ▁Romanian.  各区县/ 土方/  规划中0  路线图0 对冲1 铃声1  发现场2  教学的2  ▁reel3  在我市3  疫情结束4 通力4 liche5  ▁deposited5  ▁surrounds6  ▁prohibition6 燕山7 画师7 WL8 惹人8  ▁Leisure9  ▁splendid9  ▁Fisheries:  ▁Terrorism: 呼救; 更近; 画廊<  衷心的< iken=  塞尔维亚= 南岸> 罂粟>  值班编辑? 拉扯? Morgan@ adjust@ 忍着A 白鹭A igraphB  ▁McLarenB #,C 就等C  细腻的D airdD  ▁ForkE  ▁translatorE 娘子F 里头F agosG  公益事业G  课程大纲H 绸缪H ihannaI  ▁impactfulI 博园J 问及J ▁SHOK  报警电话K  隔离措施L JacobL 淤泥M 硫化M 老城N 韩语N  ▁ParalymO 地地O 替尼P  专业不限P  innacleQ 你笑Q IDAR  在新的R  就来了S  ▁stripesS 筹措T  和生产T  的环境下U  ▁docsU  公办幼儿园V ERAV  第四个W 九点W  截止至X  高级感X 放电Y 雄伟Y  ▁turtlesZ 不痛Z  社会力量[  ▁Configuration[ 腿脚\  市民的\  ▁dome]  ▁smells] 是靠^  及时掌握^  学习党史_  两年前_ 坠入` undy`  中高风险a  欢迎各位a % 巩固拓展脱贫攻坚成果b Soulb  城乡规划c  不得不承认c  ▁artisand  ▁ubiquitousd 天了e 护发e  背景墙f  钢集团f steeng  aternalg  hhhhhhhhh  落脚点h  男子的i 漏电i 认筹j 越少j  公文写作k  Drawingk 南城l  解决不了l  ▁Baselm  当回事m  革命精神n 圆通n 外公o Soono  ▁nationalityp  司法所p  点个在看哦q  ▁vinegarq  ▁Theodorer  今天起r  让整个s  无新增疑似病例s  ▁Patriott  ▁unusuallyt  ▁belongingsu  严峻的u  元包邮v  开拓创新v  ▁sacksw  可采用w  电饭锅x  零花钱x ceptory  ▁contempory 名媛z 豪迈z  用过的{  relations{  不堪设想|  图源网络|  经常有}  社会公众} Fu~ 装满~ 分居 气滞 诸事  ▁lust 二郎 共勉 即兴 铲除 闲鱼  发现自己的 皮鞋  大气污染防治 煤业  ▁cyst  有关法律法规  ▁pollut 晚睡  备考群  荔湾区  毕业典礼  ▁Damon  在这个世界上 稀奇 TRA  用生命  ▁envis 捕获 izhou  ▁miscon  的队伍 Normal " 经济社会高质量发展 与大 打野  上涨了 开枪 速来 为天 理应  主观题  积极组织  ▁raced  ▁judgments 不差 厚爱 TAG  ▁condolences 存续 清点  小学的  ▁Carolyn  ▁commits 俘虏  出去了  和产业  和产品  性教育  离婚了  决战脱贫攻坚  市人民检察院  ▁impending 代缴 柳叶 消磨 跌落 ▁APR hentic  ▁Larson 任期 反恐 授牌 衣草 ▁Rag  自己身上  ▁purity  ▁Accountability 主攻 秘境  感染病例  ▁wicked 大帝 女兵  命名的 Senate 结肠 APS  岗位需求  ▁Madness 一科  不久后  ▁Towers 产城 果皮  记住了  ▁revisit qy tot  ▁Bengals 鄙视  营养的  ▁Spir  ▁Solic  与朋友  尼西亚  学史力行 ällor 的首  还会出现  estyles  ▁remodeling  可操作  体能测评  本次会议  ▁Mour  ▁Warfare igne  点击蓝字  ▁diplomat  专业组  继承了 esse ipel  ▁Sundance  菩提心  方式方法  管委会主任 枯萎 狂放 科举  连云港市 养肺 随州 黑洞 weekly ▁Ned‚  ▁Pilgr‚  ▁voucher 懂你  自动挡Ă  汇算清缴Ă  ▁motionsł (`ł mereƂ  ▁commendƂ 汪洋ǂ  而努力ǂ  延伸到Ȃ  对辖区Ȃ  这个数字ɂ RNɂ 敌军ʂ 淑女ʂ  ▁awake˂  ▁intermed˂  ▁Berkshirê Pin̂  各种原因͂  颁奖典礼͂ 向她΂  沉甸甸΂  ▁Ovalς 保教ς  ▁EleanorЂ  出现症状Ђ  研究结果т  ▁homosexualт  ▁streamline҂ 原液҂  ▁Uruguayӂ  ▁lyricalӂ 泪目Ԃ  ▁bourԂ  的坚强领导下Ղ  ▁vibrationՂ 之音ւ uyaւ  动不动就ׂ  Archiveׂ  可随时؂ 陆家؂  担负起ق okiق  天小白理财训练营ڂ  ▁multidisciplinaryڂ  民警的ۂ 一车ۂ  有所下降܂  分时段܂  播放量݂  体育活动݂  ▁stabbedނ  ▁compressedނ 亲历߂ 直降߂ ▁х CLUS Brain  ▁Crosby Risk hops  日本政府  ▁jewel 桐乡 鳄鱼  external  ▁sprawling umblr LEX 半边 Desc 帆布  ▁Atmosp 面值  黄山市  提升城市  ▁upright  ▁Partnerships 就差  红红火火 Linda  ▁Cottage ес 婚宴 志明 洪洞 消遣 焦油  容易发生  自觉接受  ▁Fridays 全开 尔克 infty  ▁richer  ▁newcomers 木雕 难民  ▁Treasurer 茫然 OPT Ocean  究竟是什么 猎人 谢娜  ▁goat  ▁genus  ▁leaning 桥上 遵从 Via  ▁Hastings  ▁Cooperative 圆锥  深入的  媒体平台 事态 广度 开端 沉寂  精挑细 unless  Liverpool 迪丽  有声书  ▁São  其实我  ▁peril  岁儿童  正式开启  驰援武汉  ▁Chartered 含税 要钱  擅长的  到今天  视觉中国  ▁millennials 由原  增强免疫力  疫情防控知识  智能家居  观自在菩萨  Similarly  说句话  依赖于 brief  Regional 上几 不整 师徒  报名成功  这种事情  民主生活会  establish 固原  于是我 病逝 都江  特别关注  ▁cushions 和顺 塌陷  ▁engagements farm 利川 多日 gay  阅读的  有序进行  ▁downstream  辛丑年 选股  揭阳市  敬老院  源源不断  ▁Woody 卷面 备忘 思源 机枪 鞋柜  Publication  ▁retrospective bio ità! 灯泡! 薄片" 远东" 钩针# Lot#  一个一个$  的时候会$ Keith%  ▁harvested% 皮疹&  意味着什么&  的决策部署' 抱负'  离开后( credit(  中国传统文化) asured)  ▁streamlined* 养好* 前天+ ▁Lub+  Machine, arb,  ▁ча- tour- Links. 身教.  监督员/ 玩偶/ 线程0  大事件0  展开调查1 多尔1 硅胶2  审核后2 leet3  ▁textiles3 借机4 梗阻4 鼎鼎5 >()5  检查点6 一长6 MRI7  主人的7  自称是8 辍学8  ▁Photoshop9 伯格9 招贤:  看了看: ▁wow;  Amazing; dw< 又怕< 板凳= 狗粮= 马的> Ide> 三日?  了解和? Sent@  农村公路@ 葵花A  ▁decksA  ▁ThorntonB taughtB  于是他C ▁JagC  航空航天大学D 板桥D 红船E  救死扶E iconeF  ▁SuzanneF 想见G  我们都在G  ▁StylesH  油烟机H  现在我I  铁饭碗I  ▁peninsulaJ 前五J ▁OrbK  企业负责人K 即成L  我记得L  教融合M  ▁BeaconM 包里N 格拉N 留出O ️️O StriP  总工程师P 发证Q 展板Q  偏北风R  ▁lonelinessR  有三种S  民进党S  ственT  ▁EmersonT 李静U ASCU  实验班V  严格规范V  就变得W  льноW  主要分为X  招标公告X  ▁plightY  那段时间Y  南方都市报Z OriginZ  ▁Piece[  ▁towels[  ▁uneven\ irut\  这个角色]  微信关注博野拍客]  的普通^  的车型^ 绥芬_  重污染天气_ FREE`  ▁profoundly` 好茶a 却没a RAYb  打造成为b  生机勃勃c  ▁Afroc  ▁Distanced  ▁exponentd 7 0全面建设社会主义现代化国家新征程e 木工e ▁mulf 纵队f 表层g  商业目的g  ▁mapleh  等大型h  处置工作i  ▁arrivalsi  ▁injunctionj 报满j 游船k 玉环k  ▁Clintl  ▁basketsl 广佛m hoem 库里n ipein  更上一层楼o  ▁eruptedo 会务p 砂锅p  不能说q  的生活习惯q  中华优秀传统文化r 法考r 祛风s usos  ▁oscillt 碰瓷t  可下载打印u  ▁immersedu  演艺圈v  铁矿石v  大爆发w  密不可w  要严格按照x  疫情风险等级x 布满y 简陋y  基本都z  ▁biosz  ▁puck{ 勾选{ 宽大| 渊源|  都不好} hook} Pods~ ▁cd~  气质佳  翻天覆地的  ▁jackpot  向当地 icer cluded 班倒 相争 走完  名工作人员  小学一年级  的重要原因  做的是 atura  ▁Conrad 彩云  重新开始  ▁bonding 接驳 积食  会变成  被判刑  ▁extrad  开门红  这么多的  风险提示  ▁statistic 长成 apo  ▁Shift  年应届毕业生 波涛  女性在  ▁factual 才叫 折纸 浪花 魔都  数百万  ▁purse  ▁Kavanaugh 待在 量少  ▁pals  ▁Sword 连队  正宗的 ▁bir  主要成分 Latin  让更多人看到  ▁scorer states  ▁Knicks 福神  责任感和  集中学习  ▁Elena  ▁Locations 打断  些时候  夫妻之间  Hollywood 三菱 羞耻 丁丁 张飞  ▁Nickel 半山  ▁resurg ▁THC garten  ▁algae  ▁timet  reported 围挡  非常方便  与宁静  ▁Elev  essment  ▁Holder  ▁Marketplace 方略  都表示  ▁flux blance 肉丝 茶馆  县级市 罚金 Deal  通过开展 鲍鱼 eon irez 佛罗 Plant acion 看错  棋牌室  ▁Ramsey 静安  十五年 ▁foo  党组织书记 不拘 专程 axter immune  ▁dazzling X Q▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 尊崇 还多 CCC  参加的  ▁biomedical  加快构建 双选  ▁primer 地久  使用者  市场行情  ▁enchant  ▁topping 　」 一校 输血 闸蟹  的心思 ▁Kah ▁ctx  ▁wield  ▁Sharks  积极乐观  习近平强调ƒ  ▁Oppositionƒ 来历à ▁Ikeà  ▁Mockă  ▁STARă 不辞Ń ukemiaŃ  urrenceƃ  Catholicƃ 多头ǃ  通风换气ǃ  ▁Trioȃ  Delegateȃ  ▁garmentɃ  ▁temptedɃ 为佳ʃ  宝鸡市ʃ  更精彩˃  辣椒粉˃ ▁TIM̃ ▁Tud̃ JR̓  公里处̓  证明的΃  文化广电΃ acockσ 午夜σ 砀山Ѓ  金额为Ѓ  ▁Ellaу  ▁accoladesу  ▁innovators҃ 装车҃  的中国梦Ӄ  经济建设Ӄ  ▁PTSDԃ  ▁Baylorԃ  降血脂Ճ  流感病毒Ճ  ▁Plotփ  ▁oderփ  ▁chopped׃  ▁eighteen׃  ▁authorised؃  斯坦福؃ 旁的ك ITsك fruitڃ  ▁Breadڃ 万水ۃ  ▁Includingۃ 变白܃ 将她܃  努力为݃  辈子的݃ 仓位ރ 击中ރ 蒙特߃ 问号߃  ▁Circus 耐克  赞不绝 问询  高蛋白  大学生活  ▁conflicting 从口 病等 直奔 莫迪  新生入学 Billy  ▁plat zf 建德  通常会 arers  大部分都是  蓝色字  ▁Spread 赠品  适用范围  ▁SMEs  ▁slew 吃鸡  坚定地 ▁kar  受到伤害  给小编  ▁Teach  Wrapper  ▁mesmer urgy  创建全国文明城市 鬼神  有超过 vx Sab  为了孩子 socket 咸宁 惊悚  旅途中  是来自  携带身份证 endas ichen  等多种形式  ▁elbow  ▁Sidney 压榨 pkg  人生如  华丽的  已通过  identally  的损失 浅薄 胡杨  对个人  光伏发电  其实不然  placement 中区 巴拉 ▁SOL  要求学生 雨露 HEAD  ▁Merkel 列席 文苑 邻近  更轻松 ▁ont  看得出来  ▁Mozambique  ▁procedural onson  孩子的教育 翻滚  种植基地  管理规范  ▁Darkness  你喜欢的  室内设计 调制  关心和  是对的  物资保障 虎门  我想要  校企合作  各地各部门  因为你的  在法律  只是一种  ▁acne 绝技  吃饭时  吴尊友  她开始  不予受理  小编都会 dwell  ▁subpo  第一人 ím  广大居民  副科级  含量高  老干妈 条理 肚腩 越深  第二节 在东  为引领  融资租赁  负责公司 aceae  以更加  持续深化  ▁ethos  名字叫  我们就可以  ▁Aging Gil  球蛋白  集训营!  在线学习! Ts" 团课"  湖公园#  请拉到#  蓝天白云$ regor$ career% ifndef%  ▁eBook& 不语& 号外' 战法' 街边( 雁塔(  不打烊)  外出时) ▁pla*  排忧解难*  绿色低碳+  我就知道你+ 可燃, jav,  售楼部- akov-  关注订阅吧.  十三届全国人大.  ▁diminished/ 的二/ imony0  ▁Dram0  ▁prohibits1 限报1  曾经是2  让世界2  法定退休年龄3  ▁Weapons3  Association4 不屑4  引以为戒5 撞上5  试剂盒6  豆瓣酱6  情感的7  降血糖7  不要随意8 岁就8 赵云9  到手价9  符合国家: faced:  ▁Payments;  Referencias;  在接下来的<  ▁tents< % 如果喜欢就关注订阅吧=  深入到=  这世界>  区委宣传部>  ▁brink?  ▁Jenner? 中就@ 籽油@  第二章A  找到自己的A 可到B 赚的B ▁FogC  专题学习C  客户需求D  手把手教你D 衣帽E 谩骂E ettoF  ▁TweetF 不严G 更少G  提供线索H  驾乘人员H  双方当事人I  ▁LawsonI  父母在J apidJ 沾染K 词典K  ▁ArchaeologyL  ▁complexitiesL  和你在一起M  ▁AuditoriumM 静音N  和网络N 宣威O 浮云O usiP  性质的P  各项任务Q  文化自信Q 圈层R NineR  ▁CertainlyS XCS быT  的中年T GenderU  ▁ReductionU 天干V  ▁bankruptV 停顿W 颠簸W  ▁rumoursX  快看看X blinksY ▁LLY  博大精Z  的来源Z  表现了[ stakes[ 四射\ 渗出\  何鸿燊]  运用到]  管理所^  遇见了^ rats_  气血不足_  ▁headset`  中央广播电视总台` 组词a  不举办a  ▁faucb  ▁brightestb 庙会c 造纸c  将采取d  克服困难d  ▁Edwine  Partnerse  阿拉善f  将依法追究f 翻脸g  妇女儿童g 考究h 露水h  等程序i  ▁Weaveri 纾困j  产品线j  写出了k  法院审理k  ▁whiskeyl  ▁Wildcatsl  应当以m  控制器m  值得收藏n  复合材料n  炎炎夏日o  ▁borrowerso 猩猩p 等在p  规模最大q  ▁Honoluluq Thur  西部地区r ▁TSs ▁Fuks  共同成长t  让人羡慕t 一室u 定会u 故知v 中科v )));w Popew moonx washerx  ▁Ethernety  ▁Contributory 钓友z  日起实施z  ▁groundwater{ 凯撒{ 岭村| 粘连|  应用技术} isman} 灯具~ 炸了~ 近在  学弟学妹 folder 大戏  了一大批  文化中心 theta  prehens  ▁Colour 连片  已经从  下方留言 Harris rogate  orative 内战  ▁Utility  积分卡 uton  荆门市 商议  ▁prud 崩盘 nach  ▁cinnamon  的一部 ichita 整数 放肆 案的 礼遇 ▁Het  家长可以  语言表达能力 ün 可由  分析与  各级政府 Posts  康复治疗  ▁Sands mr 都曾  家长群 Robin  ▁recreate 和风 CMA  腺病毒 Magn ▁dye hover  保持高度 attack  ▁Calling 含糊  或取消 ▁Dug 抽血  让这个  大道至简  ▁Replacement 比拟 画中  ▁motorcycles  食用的  表现出来 varian  Foundation  看一眼 ▁boo  保持良好  ▁Machines  损失的  职业病  ▁ratt  changes  做朋友  ▁Nexus  joining  ▁isolate 腰背  三个人  在手里 rá 金海  式设计  责令改正 魔王  履行好  ▁Malawi ▁FB 不达 射频 illi ollen  ▁lied  ▁singled 绿波 胡适  赏心悦  ▁seaw  ▁brokerage  ▁unanimous 情意  物资的  度假酒店 typeof  opoulos  Associated 低端  基本原则 失主 郁金  和政策  正确认识 ussian 废纸  只有你  视为放弃  ▁Kirst 邵东  我也会  都是自己 ubishi  ▁Lotus 二小 和刘 昌平  不努力  ▁transporting 萦绕  新闻综合  ouncillors  荨麻疹„ ATOR„  ▁CancÄ 返工Ä  罗志祥Ą  过生日Ą Sydneyń  ▁Middlesń  宣传单Ƅ  第八届Ƅ oretDŽ 前路DŽ 欲滴Ȅ 涂层Ȅ 零点Ʉ  医学中心Ʉ  ▁reviewersʄ 死在ʄ 草丛˄ ispens˄  ▁Habitat̄ 光合̄ 文武̈́ encḧ́  全面禁止΄  Källor΄  ▁buffetτ  ▁Journalistsτ 天降Є 宝塔Є 欠佳ф 著有ф 露面҄  为依托҄ 福寿ӄ  评选出ӄ  防城港Ԅ ogieԄ  工业设计Մ AnimalՄ kowskiք urgeryք  通报确诊病例ׄ vieׄ  工程机械؄ 八戒؄  并接受ل ']);ل  ▁Falkڄ  ▁Veraڄ  ▁countersۄ  ▁digestiveۄ 每吨܄  纯粹的܄  西洋参݄  redirect݄ 刻骨ބ  抚州市ބ  福利院߄ 悠长߄  也没有什么  ▁Moder  任务是  第一年  职业规划  Identity  ▁proficiency  ▁manufactures  说出了  英语口语  规定执行  转载请联系  ▁Everton  于是在  免不了  找对象  为民服务  重大疾病  ▁Mayer 为贵 线长 临泉  istrate  ▁Commissioners 小陈 索性  不可取  我们每天  每节课  ▁peas  ▁Papua 坐上  种种原因 recht 鸭血 ▁"#  潜在的  适合的  公示无异议  ▁caller 兴县 爆满  群体免疫  行业标准 Known  ▁seizure 海伦 abilia  itchers  ▁Stevie 忘掉  在社交  就是指  靓丽的  但是没有  ▁dens  ▁Logic skirts visual  的这一 几款 水泡 近些 insk  股权转让 这番  英文版  ▁remarked 利弊 痼疾 禁食  交通出行  ▁transitional 口角 朔州 茶汤 岳麓  这个月  友好的 obao 为师 往昔  显得很  司法行政  ▁snakes  ▁Innovative 它有 对各  的定位  不能在  生产商  高尿酸  ▁Karnataka 冲破 参见 Kal  土地的  ▁slap 只愿 建校  下载链接 一练 四名  一家亲  岁生日 tips  培训基地  ▁gust  ▁Workforce  ▁Developing  ▁invitations 人世 显老 领口  的看法  健康的身体 活化  新希望 一桌 止渴 蚯蚓  ▁greed  不达标  ▁dinosaurs 下颌 娘娘  不喝酒  与宇宙  青少年的 icode  ▁ineffective! gyz! 满身"  才能有" ▁MFA# 二嫂# 多好$ 痤疮$  始终是% 堵住% 村上&  千瓦时& ▁Vig'  将陆续'  一下自己(  集体学习(  其他年级)  ▁conditional) 小而*  艺术节*  不要以为+  累计帮助+  ▁Sight, 保命,  与家长-  月期间-  每次都. iesta.  的原因之一/  agascar/  原创作者0 cause0  ▁loft1 催人1  永恒的2 "`2 体表3 层高3 factor4 秦汉4 农技5  ▁symposium5 手提6 扰民6 考后7 Compos7 remely8  ▁Johann8  更简单9  ▁lure9 % 了解本地更多精彩资讯: FBI: arcel;  ▁contenders;  ▁nationalism< 学硕< 性欲=  尿毒症=  手上的>  绝对不是>  ▁fronts? 你敢? 折旧@  ▁Cherokee@  ▁PuneA terdamA 房山B 高估B NanC  下一秒C HolyD  为了满足D  ▁WarrantE  ▁TwilightE  ▁upliftingF 一击F 人能G  大约有G WHATH TwentyH helperI  的形状I  ▁troutJ  ▁InquiryJ 雅居K 冲着K  的职责L  影视作品L  指导学生M  ▁composeM 前款N  见过的N  ▁ConwayO 时用O 驰骋P  大大小小的P 有加Q CETQ spectR  ▁ButtR  ▁heftyS  ▁rocketsS 芍药T  功能和T ▁NugU  ▁MareU  ▁rippV SpiritV  ▁SchulW 你做W  项目落地X  ▁crankX  ▁CalifornY ▁DupY  ▁StonZ  景区内Z  全体教师[  容易产生[  小编为你精心\ 谋杀\  此前的]  火神山医院]  ▁Legislation^ 我俩^  今年高考_  会发生什么_  ▁Pike` Simply`  ▁graffitia  个性的a  了很久b  和销售b  诉讼请求c VIDIAc  ▁bathsd  ▁Nairobid 车况e  市纪委e  给他人f  过去一年f  ▁Weeksg 出诊g  好学生h  Someoneh  出租汽车i 坦荡i 梦回j  在进入j  情况等k  ▁Anitak XSl 暗藏l ▁Ganm  ▁françm bzn 有感n 软装o  ▁linguistico 共管p 横纹p  中介机构q  自己动手q  ▁geekr  共同发展r  ▁acquaints O H灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧s 睾丸t  官方发布t  项目总投资u  ▁Dynastyu  ▁Renaultv  无法提供v  逐步恢复w  上诉人w  后续的x  手机话费x  这些食物y  ▁reunitedy ℃　z 胜于z cad{  越努力{  为员工|  ▁microbi| ▁SAS}  partisan} 横琴~  的第一天~  finance  一大堆  是不易  服务对象 红颜 非得  ▁Tian ﹏﹏  一定会有  应急车道  ▁traveler  只能用 日由  ▁forge 上古 花城 魔方  ▁facets  ▁pottery 首轮  年以后  磷酸二氢 chase  ▁dishon 庄重  企业与 owan 服的 金光  并不高 厉行  冬奥会  则可以  防控知识  ▁Tomb  ▁Apprent  ▁refrain god Davis  ▁Alicia  ▁Tobacco 增殖 摸摸  案发后  购买后 ▁Obl  不能满足  ▁judiciary 活儿 耳目  学理财  编辑的文章吗  喜欢小编为你精心  ▁beforehand 溜溜  ▁tallest 图腾  和市场  怀孕了 rout  法院认为  资源配置  managed  ▁Dickens  ▁extremist 1 *喜欢小编为你精心编辑的文章吗 在给  ▁Kabul 招租  Driving 未雨 步兵 装了 运筹  古诗文  ▁militant Jen  工作在  自助餐  要确保 贫富 青草 Agg 一八 仙子 暴增 现对 Shared  ▁Esther WY  承担了  ▁Clients  ▁investigates VILLE  考生应 ▁img  差不多了  百科全书  时间和精力 大典 TED  电话预约 又大 通的  一回事 typed 又到  如果再  系统化  ▁Terrace  ▁helpless 三首 免得 无双 ▁ih  小轿车  需要的是 ipline adle  ▁fluids  ▁Philippe 欢聚 海角  他一直  引导下  拼尽全力  ▁Cauc Weekly enches  Gallery  ▁systematically 废止 Folder  ▁missionary lé… 紫菜…  闪光点Å rakeÅ  诺如病毒ą  点击上方蓝字ą  ▁eldestŅ 装在Ņ  河流域ƅ 兄妹ƅ  技能的Dž 三板Dž brushȅ  ▁wetlandsȅ 皆宜Ʌ 要向Ʌ 足部ʅ  但可以ʅ  最低分˅  小区业主˅  贯彻执行̅ 库房̅ almostͅ  ▁abductͅ 出油΅ 改制΅ 映衬υ 醇香υ 陈伟Ѕ  高峰时段Ѕ  ▁livelihoodsх 怨言х  买卖双方҅  基督教҅  汉兰达Ӆ  我给你Ӆ  好起来ԅ  每一处ԅ 摩尔Յ becomeՅ 表中օ 阻拦օ  与创新ׅ  王思聪ׅ Race؅  有些事情؅  表现出来的م  orouslyم したڅ  整理编辑څ  ▁lionsۅ 修长ۅ 前海܅ erdin܅  ▁awhile݅  只是个݅  征地拆迁ޅ 录比ޅ esa߅ 则以߅  放置在  感动了  工作座谈会  等荣誉称号  ▁approves stral  ▁fats Amanda  ▁Wikimedia 两段  外星人 rero  在他们的  ▁pistol  ▁decorate 右脚  监测和  Talking 下有 清流  四季度 PDATED 指派 orem  第四单元  ▁Consumers 共睹 整装 法会  感谢你  真正实现 CAS  不含税 袁世 ▁Pak  自由自在  ▁cocoa 小胖  惊心动  家庭经济困难  ▁Rajasthan 撤退 新路 诉苦 poly quee letely  ▁flora  ▁Erdogan  联合惩戒 拉尔 顺口  ▁audits  ▁subset  我终于  神经痛  衡阳市 ▁Alv  ▁Required 子时  喜欢用 Thom  睡前故事  公众号回复 oxiang  earcher  ▁Compan 蔷薇 倒逼 Dur  美美的  ▁bakery  ▁progresses  老母亲 星际 cake  不得低于  德才兼备  放松警惕 猎奇  ▁penned  培训活动  ▁glor  ▁Mutual  ▁plural  ▁strides  ▁chatting 胆经  ▁Older  ▁abide 沉思 红油 过冬  信用社  区别于 变黑  媒体的  每一项 抗衰 竞拍  置身于 水涨  ▁contractual  群防群控  ▁pits 入座 读取  宅在家里  ▁Hide  恍然大悟  ocalyptic Tap cred  要不是  ▁dermat 哎呀  不一定能  ▁stag  ▁Becker  ▁extraord 奖状  不足的  去寻找  ▁watershed  InputStream 小豆 纳德 cai  进社区  一定不会 名号!  的境界!  keyword"  ▁Screening" 慌乱#  ,’”#  与国家$  日照市$  世外桃源%  化学物质%  触目惊心&  ▁Directed& 专列'  不容辞'  工作制(  电子发票(  ▁carriage) 金鱼) ová*  ▁sociology* 停了+ 吃吃+ 画室,  等政策,  即将开始-  ▁champagne- 　'. 嚼劲. 预付/  点击放大/  各相关部门0  ▁Stall0  ▁Diesel1  ▁fracture1  ▁psychiat2  ▁nostalgic2 渣土3 羔羊3 见状4  他总是4  专家建议5  预计未来5  ▁unsett6  公检法6 ttle7  日前完成7  高空抛物8 还钱8  也不怕9  强调要9  带量采购:  ▁bolt: ocado; 列宁; 可人< 翰林<  被取消= Jamie=  ▁mapped> 包保> 湾村? 血虚? ▁ATP@ ileen@ 是全A 身姿A  就对了B NancyB  ▁ServeC  抢救无效死亡C  objectsD 狼群D  定义为E ▁YESE  ▁souvenF C <以习近平新时代中国特色社会主义思想为指导F  ▁TactG  ▁dosageG  ▁foolishH 年新H  商品房买卖合同I 篮网I 缘由J  ▁wiredJ  ▁recommendingK  ▁CelsiusK  ▁IndoorL  ▁ventilL  ▁DonovanM  ▁deportationM 不把N 其为N 李伟O CLUDO comaP  房屋权属P  电气工程Q  ceptiveQ °,R 上证R 台面S 灌云S  普通人的T  核心素养T 农房U  ▁инU 修路V 各领V  讲解员W GraceW 送检X  也并非X  教学方法Y 日方Y  体检结果Z  疫情防控常态化Z 不占[  ▁Daisy[ 拿捏\ 管员\ ▁kan] 华章] 油污^  方案的^  情况汇报_  突然发现_ 收窄`  好玩的` repoa  一些问题a  一撇一捺b  不错过b  压岁钱c  多数人c  继续在d gonomd  pandemice  对上述e  ibernatef 夜深f  勾勒出g  特点和g  可以被h Dallash 蒸熟i 蓝海i  回味无穷j  ▁ironicj 恒星k 清偿k  可怜的l  自闭症l  惊心动魄m 为党m 是该n 溢出n 甜点o thelo 无以p  比较有p  都江堰q  togetherq  ▁compartmentr etusr  ▁plagueds 畅谈s 终日t  透气性t  造成严重后果u 位移u  对生活的v basicv Ratherw  ▁miserablew ähx 吃好x  并且还y quisty  接触的z  虽然不是z 晕倒{ ▁Wei{  年第一季度|  他希望|  供应量}  ▁deprived} 穿得~ 风起~  薰衣草  行动不便  ▁Stalin awk BACK endra  ▁Blogs 锁住  就一定要  的重要意义  ▁polite  ▁McGregor 条文  ▁Listing  ▁reactor 树干 Gas  债务人 ugal  提高自己的 endars  ▁Towns 直供  香格里 annie rophy /, 班上  绵阳市  日报编辑  注册商标  ▁sinking ”→ 圣灵 opl  ▁Raff  ▁Cycling 收下 韩红  ▁advocated Cs 到时 walker  小编前进的动力  ▁philanthropic 陨石  共青团中央  awaited WV 沿河 游记 mur  所以她  违反规定  其实很简单 溪村 (['  治疗和  ▁eliminates 手枪  ▁heir  ▁Presley  山公园 stats 牵扯  就恢复  很深的  证实了 camera  ▁Higgins 一成 甚者 的死 ooky  ▁hangs 条形  老老实  迪丽热巴  ▁Saving  回应了  工具条  眼花缭  ▁Lawn  ▁Dirty  ▁exemplary 在两  ▁eagle  amilton 野草  滞纳金  教学计划  Weblinks 减慢  团结一心  高频考点  ▁citrus  clinical  ▁steroids  出名的  ▁Mazda  来完成  第五次 orset ▁bob 奢望 废话  敏感性  而如果  ▁Supervisor ,''  踏上了 thank  ▁troph  和历史  ▁REST  responds  ▁getaway  ▁onstage  ▁retaliation 铁岭  了两个  或以上  更注重  润滑油  特别有  部分区域  ▁converts Ds 百川 mall  能够满足  ▁nationalist 聚财 ▁JR  文化广场†  科技管理†  loggingÆ  ▁GiovanniÆ KyĆ 疗养Ć  华图教育ņ  最优秀的ņ  提供了一个Ɔ  ▁boilingƆ 放屁dž Drydž  这样子Ȇ  为推进Ȇ  这几点Ɇ gresɆ  ▁Originsʆ  ▁Coventryʆ 要大ˆ 麻花ˆ  ▁lifestyles̆  ▁commencement̆ 岳父͆ 洞庭͆ 王旗Ά  微博上Ά  教育与φ  补偿标准φ  ▁ErnstІ 苦练І  官方回复ц Forestц  ▁dramas҆  ▁vandal҆ 屡次ӆ  人员必须ӆ  ographieԆ  ▁ProvidersԆ  ▁instanceofՆ 麦子Ն  与设计ֆ  ainlessֆ  的特性׆ ▁GST׆  每年都会؆ encode؆  appointedن 撇捺ن  本科以上学历چ 同住چ  得来的ۆ  warmingۆ  小高层܆ rious܆  aculture݆ zek݆  更令人ކ  青年人ކ 写完߆ 江边߆ 记入 Jobs orio  处理方式  ▁seldom  ▁pulmonary  的稳定 神明  可以有 ▁CAS  呼吸道疾病  ▁Haley  ▁Minute 新股  ☟☟☟  强身健体  ▁commentator  ▁Senegal  榜上有  的心愿 esteem  ▁Appropri 乐章 eas  枸杞子  可以满足  ▁Tata 包的 lene ologne  ▁seizures  积分换  驻村工作队 叫醒 江浙 润物  ▁readonly 猪肝  ▁dryer  总建筑面积约 忐忑 /" 说谎 USH  ▁Lung  ▁Watkins  要考虑  如何使用  无时无刻 小刘  其它的 帮人 无上 STE kow  氨甲环  老板的 Dean mits  ampered  ▁Pasadena  ▁semiconductor 早产  本地人  氨甲环酸 埋在 hamed  大学毕业生  ▁approximate 湖滨  超越了  县公安局交警大队  ▁mortgages 习性  感染力 imps  维生素和 旺铺 落成  司令部  晚上的 Failed  子里的  道理的  切实解决  心肺复苏 漱口  的民族  项目实施 浑浊  户型面积  ▁Elements avoid south  ▁encaps 推选 热油 越快  于网络  无线电 区划 MES  ▁saturated  一起看  第一时间处理 所列 端口 驱虫  ▁gripping 现年  又是一年  精力充沛  ▁raids 降脂  前期的  玩家们  已经非常  ▁indiff  可能是因为  ▁outsourcing 宝座 红人  能快速  谁负责 Crime  进一步做好 何其  路透社  学以致用 Jerry  ▁veterin! $('! bern"  哈萨克斯坦"  grounds#  illiance#  生物科技有限公司$  相对比较$  ▁sunk% 欧式%  ▁Osaka&  人民政府关于&  ▁Crafts'  employed'  ▁Upcoming( ça(  ▁Upload) 球星)  和保障*  档案馆*  ▁Length+  户外运动+  啊啊啊啊啊,  工作的重要,  reaching- ;;- 连发. 锦囊. Chain/  分享彼此的/ 煮开0  ▁shutting0 岗区1  长按二维码关注1  ▁deserving2 彩妆2  要发挥3  ▁Argentine3 胜负4  ▁Stella4  ▁Addiction5 粗壮5 谢某6  全国优秀6  ▁Marvin7 品的7 馄饨8 angled8 刺客9 烟头9  刺激的:  竞争优势:  ▁jackets; 抢着; 要的<  天上的<  ▁vows=  宝安区=  ▁hacked>  ▁english>  ▁Friedrich? 一束? 健忘@  乎乎的@  渡过难关A  由于疫情A AntonB  ▁cosmeticsB 应与C  版权所有C ▁NFD  在大学D 接轨E 晨起E  现场检查F  ▁DomeF 至终G 秋分G 鸿雁H  劳动报酬H  inflammatoryI 钻进I  ▁LumpurJ  ▁InstructionJ  望远镜K  方向发展K  ▁ComparedL 王毅L ▁DLM  下功夫M  这个孩子N ORDERN 承揽O 牙医O DiamP 八达P  时注意Q  ▁TryingQ  ▁PsychologicalR 永清R  为全市S voiceS  ▁usableT  ▁varsityT 多出U 沿用U  第三卷V  ▁WattV tebrW ComplW  ▁BiomedicalX  ▁embarrassedX 盐酸Y 行气Y  价格表Z  经营单位Z  还有这些[  ▁firmware[ 活络\ äng\  一首歌]  可口可乐]  快速通道^  ▁Cure^  ▁waterfall_ 任一_  美妙的`  一杯奶茶`  社交媒体上a ,$a 画法b  也会导致b 喜气c 嫁娶c 粉的d 腿上d  ▁admittinge 浓墨e  多种形式f  ▁hornsf 泰兴g  Governorg  ▁thinkersh  工人的h  还是个i summeri 的快j 赛后j  ▁gangsk  ▁laurek  ▁Laurentl  ▁Napoleonl вуm vikm arialn 不外n 两部o 大呼o 工贸p 接通p  将取消q  在杭州q  不对称r  值得信赖r  ▁Willows  ▁rulerss  心律失常t  这家公司t  改变自己的u 底子u  新增的v ONYv  renownedw  ▁Dialoguew  experiencex 如雨x 少将y  ▁Osborney 坪村z 各色z 风头{  ▁awaits{  ▁easing|  ▁proclaimed| 居所} 碰上}  政治部~  ▁prosecuted~ 套取 ▁SOC  变异毒株  职称评审  ▁centralized ▁HU  报告厅 仲景 祖辈  大肠杆菌  解放思想  ▁capitalist  协会会长 后卫 惠农 赞歌  朋友来  ▁tucked 取样 哪家  的气度  德智体美 金凤  ▁crab  发现的问题 微言 sie  南山区 又可  网上的 arnish  ervable 可循 迎面  这也意味着 白鹤 迷宫  我之前  在未来的  生物技术  与客户  很少见  快乐地  严重警告处分  ▁Investing  ▁terminology 忘怀 摩登  ▁(‘  ▁peppers RIS  年级语文 ▁abd  defense  medicine 宝龙 演化  ▁Bets  Consult  ▁Console 先在  的回答  生活补助 荠菜  游刃有余  ▁marital  将所有  身边有  住房保障  分管领导  Rebecca 乳业 砂浆  ▁approvals  ▁certifications  开展好 三联 老总  许昌市  万人民币  Attributes 劝告 她从 行至  路边的 Philip 写入 外皮 既不  和内容  ▁artery 尿道 祖传 Cold  ▁Entity 酣畅  精华液  insured 佳佳 胡乱  的喜悦 米线 HEL  在每个  ivation  ▁upfront  ▁Investigations 诸位  各民族  路以南  ▁wildfires تى  二十七  异常的  都是用  地摊经济  日举行的 Miller  ▁bustling 消灾  了解自己 芦苇  主动脉  可是在  扶持政策  有句话  观众们  也可以是  ▁DVDs ERC  别担心 婆诃  无止境  目共睹  被处罚‡ rmt‡ NicholÇ  PhysicalÇ  ▁outpatientć 临江ć 而降Ň xonŇ  无怨无悔Ƈ isburyƇ  ▁ChevyLJ  按计划LJ  的标志ȇ  监测预警ȇ  而是一种ɇ  住房和城乡建设部ɇ  计算机科学与技术ʇ 加压ʇ  共建共享ˇ  ▁embracesˇ 田里̇  被大家̇ amph͇  大打折扣͇  ▁dias·  ▁essentials· 当众χ 清末χ PHPЇ rajЇ  特别推荐ч  招聘岗位及ч  运动量҇  ▁пер҇  ▁CapsӇ " 高新技术产业开发区Ӈ 与台ԇ  ▁predecessorsԇ 伙同Շ 湖镇Շ  ▁depictև  ▁greetingև  临其境ׇ  ▁sentimentsׇ 主次؇ 军装؇ ▁TTه  着眼于ه recyڇ  远远不够ڇ  ▁kidnappingۇ 合浦ۇ 悔恨܇ 清白܇ 骑兵݇      ݇ 售出އ 宝骏އ 盈盈߇ 红斑߇  通辽市  极乐世界  previous  ▁charger 上浮 后裔 iego 豁然 Bird chip 激进 碎花  回家中 illard  ▁Psychiatry  巡视组  备受关注  ▁tart 开窍  出版物  株洲市  顶层设计  中心幼儿园  ▁biomass 要紧  不同时  兰州大学  ▁Braun  更加强  金华市  赞不绝口  通用机场  ▁Gent  ▁swinging ▁Lob  中国石化 张图 散去 而安 inning 方剂 英俊  和传播  思考和  电动机  ▁havoc 两步 微胖 ' //////////////////////////////// 动情 果糖  分享会 TAIN ▁PUR  ▁Norris  ▁ceased 笑得  市场和 icycle  科学佩戴口罩  ▁Belize  ▁freshwater 入局 培根 舞曲  冬春季  其实也是  网上预约 一季  哪里有  习总书记 秋日  ▁Hague  ▁Compact  注册登记  ▁rejuven  ▁soak  特殊教育学校  Descriptor  ▁catastrophe 与新 大省  ▁Lionel idea  检验检疫 armac  ▁Anglican  ▁Salesforce 低廉 出局 强市 Julia  招生办公室  ▁commanding  aughlin  ▁asphalt  ▁tsunami 之宝 Girls 燃脂  ▁safest 文广 elier  orations 系好  医学生  此次疫情  ▁Phen  ▁comedic  ▁organising 征得  ▁bracelet 梅根 胶带  计算的  精神上的 为鉴 复刻  任何的  工作由  ▁Mash  ▁feud 不杀 按理  点击图片查看 >[ 小球  没见过 才女 寄予  成语故事! Kansas!  ▁muff"  ▁Endowment"  ▁Liability# 豆豉#  作业的$ jdl$  无小事% 元代% 自省&  开发了& 天宫' 蛋价' 贵宾( vre(  树立了)  ▁swear)  度假村*  开始后*  全国各族人民+ "(+  的小型,  ▁OVER,  ▁Query-  ▁Guardians- 车顶.  还以为.  工作场所/ posed/  Experts0  continue0  ▁surfaced1  ▁splitting1 改掉2 派员2 赛前3  ▁ди3  ▁invo4  ▁breeds4  ▁emperor5 血统5  民辅警6  ▁Ideal6  相关负责人介绍7  初高中7 oating8  ▁sidelines8 子村9 等处9  又一代:  正常使用: andong; 真有;  ▁Semi<  ▁Lecturer< hpp=  多年后=  大概在>  ▁weakened>  reditation? 象牙?  ▁Crack@ 小寒@ ▁SneA  就不一样A  据美国B  简单粗暴B aptopC  口罩等C  被认为D  成功将D  生不息E ionageE  ▁ASEANF  ▁moisturF  我们通过G OthersG  ▁stewardshipH 亭亭H 该镇I  ▁buryI  ▁indicesJ 管党J TIMK  爱不释K  ▁manifL  小编前进的动力哦L gpM  不高兴M  二氧化N  ▁InterestingN  听到的O  ▁HickO  ▁ProteinP 挡风P 过江Q  最低价Q  无人驾驶R 届满R 贵妇S  永州市S  呼吸道传染病T  生命里T  ▁calculatingU 惊吓U 龙港V  最终在V  证券交易所W  ▁barbecueW  ▁shatteredX  我们为X néY 左脚Y 急促Z LedZ  不知情[ atts[  ▁Antarctic\  ▁unofficial\ mx] 尼日] 慌了^ 示弱^ ERSON_ Johnny_  elements`  加好友`  进行沟通a  missiblea  ▁renditionb ßeb 与她c 交到c 日晒d 自得d  会随着e  的监督e  ▁drillsf  ▁believerf 久远g 手串g 自幼h  为己任h  而是一i rumsi  起拍价j  correctj 年过k aryak  工作推进l peratel  ▁Simonem  ▁cerebralm  ▁questionnairen  在山东n  诱人的o  金融中心o  ▁Anatomyp 年糕p 支招q 涉毒q 物美r Icer  影剧院s  ▁Liquids  ▁pillart  ▁satisfactoryt  可登录u  带来不便u 种花v  智慧和v  ▁paydayw  ructuresw  ▁allocatex  ▁sontx erellay  缺一不可y Voicez  喜欢一个人z  半年的{  转发给{  面积的|  空白区域| 同台}  还经常}  ▁Taste~ 络绎~ abh  ▁Pediatric 摆动 锦江  应具备 icent 少有  中国汽车  ▁garments 粒沙  用筷子 cele  人员管理  印象中 Forms  ▁baggage 所指 Kings 我吗 替补 梯度 itton  ▁outward 为王 是集 的男 选材  ▁malf  ▁Lyrics 二零 镂空 Pain medium  ▁induce  Technical  ▁sinister 杂念  生活通  通知后 Choice  看看看看看看看看  ▁personalised 浓密 是和 金莲  申报的  ▁corrosion  新青年  等人的 江津  三明市  下一阶段  给人的感觉  Hospital  ▁affirmed  ▁brochure  主题公园  ▁viewpoint  优良作风  工作人员在 三思 或因  亮丽的  落下帷幕  ▁Marian  危房改造  ▁routing  ▁effortlessly 住进 刘亦 功利  还在继续 ascus  源源不断的  browser 绝经 emis  ▁Mercer  ▁Removal 丧事  该产品  造成影响 应答 该片  制度化 Julie 期许 火情 zzi  ▁intimacy 缝纫  capital  ▁Revival  ▁Presidency •• 她想 长斑  是不够 priced  促进血液循环 " 点击空白处查看答案 卷子  ▁marched  ▁oneself "[ 巴萨  宣读了  来到这里 降本  ▁Noise  Thousands 反问  将实现  直到现在  consuming 封杀 速效 ERAL  每个家庭  今天小编就 allels 俘获 粉底 萧县  ▁faded 内涝  道工序  ▁estates  ▁Qualcomm  ▁tortured  ▁maternity 发誓  的加持 incy  安全知识ˆ  直系亲属ˆ ▁HAÈ  ▁KushÈ  社会保障卡Ĉ  optionalĈ  ▁Krishnaň 信函ň  与实际ƈ  有氧运动ƈ  一个小小的Lj  ▁DownsLj  ▁wizardȈ 太贵Ȉ 快去Ɉ  你自己的Ɉ  辅助治疗ʈ  ▁burdensʈ  ▁harmlessˈ  ▁Physiciansˈ 我已̈ 气得̈ 神舟͈ 原图͈ 等有Έ  ▁AlleΈ  Registryψ 特工ψ fonyЈ  什么时间Ј  最真实的ш Orangeш  国家森林公园҈  也难以҈  学历不限ӈ  ▁persistenceӈ  ▁RoughԈ  IndependentԈ  创建于Ո  一点点进步Ո 二批ֈ 校训ֈ 秋色׈ Few׈  薄之力؈ umpy؈ ▁Bavو ( !你们是小编前进的动力哦و 能打ڈ 读经ڈ  如何呢ۈ  脑卒中ۈ  的重要因素܈  无症状感染者的܈  一下子就݈  ▁sibling݈  ▁Artisticވ Animވ  ▁churn߈  职业资格考试߈ 嗜好 孝义  手术后  限期整改  ▁Homeless 西咸 ▁Spo 挡住  产权房  是唯一  的宗旨  ▁Kauf  ▁occupancy 正念  ▁Airbnb )^ ▁μ 宰杀  发大财 ()));  ▁militar  ▁articulate 这会  咖啡馆 ität  为了能够  ▁му ▁dre  argument 听写 财险 ATM  在这儿  本来是  ▁receipts  ▁criticised  ▁philanthropy 细支  心仪的  马伊琍 张永 秒变  等人员  ника ▁Sey 纳西 ▁KEY  extension  ▁Chargers 大四 鸟瞰  城市中  拉开序幕 王玉 卡上 取材 引才  露出了 leck  教书育人  到一起 倾注 百米 ▁{'  新华书店  ▁Wisdom  ▁motorists 全称 鲨鱼  几句话 思明  及有关  而产生 sector  ▁berries  ▁wrongful 上文 坠楼  但是它  ▁methodologies 溪水  按有关规定  ▁Relationships 虽是 ▁kl  中有一句 hq  ▁Pray  很多人觉得  ▁screws 选了 GAN  人生几何  ▁Faces ně 胶东 ▁HTC  ▁generously 小道  寻求帮助 elp  南海区  ▁derivatives 应将 英模 ▁FD  兄弟们  各企业  ▁pivot  ▁Dalton  天下的 ategic  新鲜感 evil  巩固提升  ▁Slim  ▁Quite  ▁Worship 山高 凄凉 可乘  交通局  配合下  ▁Anthropology 育苗  当你的 apur  ▁OECD  呼伦贝尔市  ▁yogurt 李志!  定位为!  过年了"  一无所有"  Minnesota# 异彩# 硝烟$ glomer$ 宜家%  和对方% ymour&  ▁Saga&  大学和'  对考生' Cash(  信息查询(  虚心接受的)  ▁delle)  ▁viability*  ▁indefinitely* EPA+  在线上+  升旗仪式,  组织机构, Sheet-  五年级下册- áz. 大喜.  土地使用/ setup/  有态度0 ▁Byz0  人大常委会副主任1 冶炼1 圆桌2 彷徨2  可靠性或3  注明出处3  cutting4  ▁Sporting4 一建5 加个5 aje6  如有异议6  Different7  传递给7 libs8 Launch8  ▁arranging9  Fortunately9 民情:  摆地摊:  ▁Merch;  ▁formations;  枣庄市<  ▁Denise<  ▁affordability= 或以=  备忘录>  记者问>  ▁oust? 幌子? 春笋@ 肉鸡@  转载请A 塔吊A 她家B 都来B 闭卷C  集团旗下C  ▁IncludedD  他发现D  少一点E  ▁SortE " 小编都会虚心接受的F  秘书处F  冷冻食品G  ▁TideG 张海H 路等H  管理区I  资源化I  如约而至J alandJ 三款K 掌门K 百香L 谋生L CameraM  ManagingM 如涉N 嫂子N 感召O 驶离O  多少个P ▁KodP  致力于打造Q  ▁metastQ  ▁reviseR  ▁WonderfulR hfS 注目S  儿童乐园T 疯抢T ▁CotU  所以很多U  ▁caffeineV 劲头V 多以W 成瘾W  而随着X  为孩子们X NationY  疫情防控一线Y  ▁workplacesZ  有这个Z  总部位于[ 热评[  技术方案\  ▁Kamp\ ▷　] 地平] 糖水^ 军用^ 甘心_ Kyle_  ▁feminism` ( !公务员录用体检通用标准` 基尼a 梅山a NYCb  这么些b  ▁Doverc  sensitivec 皮脂d  定向培养d  ▁Kathryne 代入e  信息及f  未经允许f  分享彼此的欢乐g  ▁Literacyg 车在h  ▁осh  为宗旨i 站桩i  帮大家j  洗面奶j  基本实现k 龙井k  小区物业l  ▁Lingul  ▁flairm  ▁Shortlym  ▁illustratorn 总成n 试问o  十九条o  在脸上p  ▁Minecraftp  努力了q  房间内q  的喜爱r  ▁Corbynr  你你你你s Steams 顶流t  宅在家t  湖州市u  ▁chordsu 新婚v  新闻资讯v  ▁affordedw 开建w 恩怨x 机房x  ▁Quiety  ▁incompety 汛情z  手足口z  比较小{  ▁Jade{  Eastern|  ▁Everett|  ▁randomized} ▁Ley}  宣传部部长~ 君主~ 拉风  发生时 俄国  误以为 ENTER 冷汗  老同志 无羡 风轻  ▁plunge 贪官 鹅蛋  支持与  ▁Odds  ▁Boyle 四会 荒唐  幼儿教师 JB 匈牙 圈圈 的点  不自禁  不要钱  ▁liberals 今冬 余份 生平  正月十五 ()　 Sac  于今年  别着急 ▁Meh thens  ▁Fake Retail  Iterator  ▁sonic ilian  ▁Starr  第四期  红色故事  无死亡病例  ▁supers 投用  一下就  水电站  Provide 活到 蜚语 评优 atum  开始报名  棚户区改造  sometimes 军在 小叶 干湿  下半场 ▁PCI ▁wag  冷链物流  ▁Gould 爱得 ▁Mb $('# peer ▁Zoe  资产负债 小布 ADC  了你的  真正把  集团有限责任公司  党和政府 冒名 幸亏  ▁Fowler  ▁iPhones  ▁futuristic oncé  国际教育  ▁bleak  ▁Concerto 三生  不起眼 incare 丑闻  odynamic iours  ▁Overs  ▁theological 妙曼 学点 ▁JP  难免会  治理体系  ▁budd  ▁depiction 傻瓜  新动能 akar 降息 除法 ▁PB  ▁Straw  在婚姻  未雨绸缪  组织开展了 ▁Zag  ▁thyroid  疫情面前  ▁liberties  ▁brackets  侵权责任  ▁fonts  ▁evidenced 均线  三等功  ▁dispatched 归档 研读 纪要  不要忘记  教学模式  以下二维码  ▁Portable 判刑  式发展 ▁Mog  ▁Chloe 兴华 小美 Tam‰  家庭生活‰  砥砺奋进É  ▁FringeÉ 腰围ĉ  脂肪含量ĉ wavesʼn  ▁Monthsʼn 是经Ɖ  年获得Ɖ  比我们lj belieflj  对我国ȉ  的严重ȉ ricoɉ  ▁townshipɉ 瑟瑟ʉ 给药ʉ  未就业ˉ  社会学ˉ  ▁Joins̉ 笔直̉  无期徒刑͉ etted͉  capacityΉ % 疫情防控工作领导小组Ή toxω  鄂州市ω 手链Љ  ▁retrievedЉ 不粘щ 片子щ  整理的҉  氧化物҉  的美女Ӊ  财富的Ӊ  时请说明ԉ  ▁Governmentsԉ 先导Չ 懵懂Չ 涨到։ ünd։  获得更多׉ skins׉  ▁Oprah؉ 州县؉ 广受ى  咖啡因ى  党的历史ډ  重点实验室ډ 投篮ۉ 落日ۉ ▁SG܉  居高不下܉  应检尽检݉  ▁sticker݉  ▁astronomyމ ▁Bohމ ▁DeV߉  侵权请联系删除߉  ▁Botswana  它不仅 离奇 能被 醉美  明明白 TABLE  ▁LIFE  ▁Theft  ▁refine  ▁Admissions 海安 Pic igt 正骨 宽的 纯棉 为过  专家的  ▁Expedition Ont  理解和支持  ▁wavelength  ▁Secretariat 下表 优酷 挑起 萎靡 oks  资源整合  这是我的 novel  ▁Fauc 客商 痰湿  会一直  反映的问题  ▁expressive 炒至 舒兰 sat  葡萄干  语数英 ▁Soc  但就是  ▁einen ▁ti  原文链接 dg 闲处 预知 ("/  不可替代  家单位 镇村  ▁rampant  ▁Integrity  科学家们 兰溪  时间短 ivel ▁AFL  风生水起  十分重要的 渡口  其主要  要增强  都不如  乏力等症状  ▁frontline 围剿  录取率  的一对  到了一个  豆瓣评分  ▁Lens  外交部发言人 室的 尚可  不断学习  ▁Banner xb Gun  天天练  抢购价  粉丝群  ▁Influence 华语  居家观察  ▁Indo 分外 绒毛 缙云  有多个  重磅消息  ▁detox  runtime 在未  ▁Bian  ▁Sabb Fat ▁INS  ▁Aspen  ▁disco  ▁cruising  ▁mourning  我欣赏  要记得  ▁pulp 十全 ▁Pew  ▁brig 一惊  在众多  成人的  提供食宿 loving  ▁toxins  ▁soothing  ▁firsthand 刚从 南疆 煮至 皮毛  ▁Ital 正方 kien  ▁fencing tp 一搜 奖惩!  北师大版!  mounted"  ▁favors" 童鞋# ▁SCO#  招商银行$  ▁relocate$ lf% 共处% 茶匙&  གས་&  ▁ensl'  ▁expires' 多罗( 竭诚( 耳聋) Enum)  谢谢你们* 如玉*  ▁Eating+ 不均+  自己的工作, 应运, PhD-  ▁Strength- 侄子. 返现.  发表的/ Near/ 纯色0  亿人民币0 血缘1  ▁Kuala1  产业转型升级2 Leb2  奇妙的3  已建成3  个人隐私4  也不要忘记点赞哦4 百人5  长期在5  ▁knives6 催生6  那里的7 iera7  大队民警8  ▁Smoke8  ▁animations9 怀柔9 纵使:  科技感: ▁DeS;  住院医师;  村卫生室< antage<  Barbara=  马克思主义学院=  Planning>  吃早餐>  ▁Malik? 先驱? 厌烦@  动力学@ ▁HELA 米花A 起名B  发行量B 调到C  实验区C hopeD  卫生防疫D  实践经验E  ▁blissE  在历史长河中F  ▁doomedF ▁gaG  龙岩市G 水厂H  新闻办H etermI  ▁VirusI 补脾J  制度建设J  评价体系K  ▁eatsK luenceL  ▁reelsL  ▁toutedM  稳中求M WhilstN  ▁LudwigN  ▁SilenceO  EngineeringO BethP  可以做到P  坚决贯彻落实Q  ▁JakartaQ  两年后R  达成一致R 临危S  仔细看S  ▁amusementT  是中华民族T 高唐U  深层次U  ▁DunkV  ▁CoronV  ▁wolvesW  ▁honoursW ViX 川渝X 花色Y 雄心Y 雇员Z èresZ  人类简史[  ▁fabrication[  的技能\ alom\  禁止转载] impact]  ▁Alpine^  ▁addicted^ ventus_  原件和复印件_ ▁م` 值此` 转阴a 醉里a  父亲节b  黄色预警b  为学校c  魏无羡c  ▁Frand  ▁rentsd 渐行e  ”　,e  专门针对f  为了保护f estrog  ▁STARTg  ▁cleansh  ▁extractsh  ▁actressesi 相望i 袅袅j  淋巴瘤j  相关业务k  ▁guessedk  ▁Attributionl Yul 不靠m  给客户m  不得擅自n  蒸蒸日上n клаo  刚毕业o ▁Piap  ▁Shootp 粤语q  压力和q  应用的r binaryr  ▁issuances 乐的s 略带t  一年以上t  ▁garneru  renderedu 刻板v nejv  记录下w  实际效果因人而异w  ▁Montereyx  ▁reviewerx  ▁Inspectiony 时前y  均价为z  合理规划z  ▁Unve{  ▁reimbursement{  奔小康|  Chelsea| 是做} 瑰宝} 盛况~ 雅安~  就是不行  施工进度  ▁Wholesale 哥的 滑道  承办的  过度的  ▁dehyd ей 央广 征战 转头 陶冶 Dun  北向南 JI huo  不好看  宝贝们 来算  国家线  ▁hers + $尚在医学观察的密切接触者  ilitating 大骂 长青  你只要  efeated 防患  报名电话 忍住 桐庐  ▁Fiscal 车和 只差 壁上 快线 标点 白纸  就到这里  延迟开学  接到报警后 慎用  分中心  ▁inmate  ▁Shelter 又快  体育赛事 angles  ▁Surveillance 家谱  红细胞 ▁Dro  ▁Stur  ▁narrowly  ▁enforcing 乱扔 胡思  其实并不  有的孩子 aer  不靠谱  深刻地  ▁climax  同城网  新一届  都不太  ▁enzymes  ▁neutrality 大题 ODE  的线条  ▁dissent 换季  看完了  身体里  ▁dominating 有次 空置  不再有  宿舍楼  ▁Inspiration 祭奠 稀疏  注入了  金融危机 Wilson  ▁ounces 合谷  ▁upscale  ▁textbooks 三面  北海市  是一切  pictured  ▁Lancashire 酒类  了一幅 acom  它是一种  妥善保管  Navigation 蒙蒙  女士的  各司其职  控制不住  ▁Anaheim  ▁rallied  ▁topical  ▁Thousand  ▁tribunal  ▁alliances  事故发生后 卡在 怒江 能和 这股 高脂  全部完成 overe  ▁exting ▁TI  甚至还会  ▁knit  上饶市  郭麒麟 XR 刀刃  ▁Hunts 德的 瞭望 ▁AMC  版权归属原作者 乘机 十项Š  事故原因Š  大街小巷Ê  失业保险金Ê  PreviouslyĊ 斑斑Ċ 朱砂Ŋ  ▁affidavŊ  ▁vacanciesƊ  ▁formulatedƊ 无期NJ 无疆NJ  分餐制Ȋ ▁VidȊ  ▁MelanieɊ  ▁SaundersɊ  迈克尔ʊ Craftʊ  基础养老金ˊ 门卫ˊ  ▁reps̊ atio̊ 亟待͊ 含着͊ 赣南Ί 阿勒Ί  发布公告称ϊ 安倍ϊ ▁LCЊ  大润发Њ  一个岗位ъ  ▁Juryъ 作恶Ҋ 扁桃Ҋ  与生俱ӊ  很多人会ӊ 晚霞Ԋ  ▁HicksԊ  办理聘用手续Պ 习得Պ  为全面֊  最基本֊  还有着׊ ▁Zam׊  一般不会؊ Coord؊ 乡的ي 卫辉ي  ▁EVERYڊ  大气的ڊ  她表示ۊ  标准要求ۊ  ▁beard܊  我的手܊ anoi݊ elyn݊  ▁enumerފ  请收好ފ  问题线索ߊ  方面的问题ߊ  ▁cheat  恭喜你  ▁Cadillac 祛痘 hun  你是我  每分钟 imeo  道德模范 娶了 崇左 此法  创造一个  ▁jest 东平  教育工委 选科  你见过  ▁Cain  ▁sanct  ▁whirl 阴茎  学习生活  ▁Laurel  ▁Mansion  coronavirus 凉血  滚雪球 但会  ▁interconnected 官渡 差错  Players 可别  会有什么  ▁witty 特勒 腐竹 quick  微笑着  放假通知  中脱颖而出 形色 赓续 ▁Ups  无需预约 tó 吹风  测试成绩  ▁ABOUT  建设成为 1 *在以习近平同志为核心的党中央 去爱 忘返 氧吧  周年庆  优抚对象  医疗队员  大名县  来帮助  他的妻子  国内国际  文艺演出  花园小区  ▁Vocal 泥泞  了一块 盒马  肾结石  该路段  的人和事  最好的自己 养了 牡蛎 紫檀 网联 meet  不要吃  出行需求  相结合的 汉江  施工期间 ▁COP  疫情通报  ▁Dias 超重  出货量 ▁Ges  无可奈何  ▁pirate 二院 叫你  股东的  ▁alias 光顾 字辈  品质的  负责制  考试招生  ▁stalls  ▁glamorous  这是对  四子王旗  小时以上  ▁jelly  在市场  宜宾市  迷迭香 家荡 应手  ▁disturbed 堂堂  中高端  开始就  西向东  危险驾驶 大包 ▁MAY  专家认为 christ  ▁culprit 和黄 是多  一见钟情  ▁deepen 她很 海盗 ▁Š HFP asian  ▁Padd!  ▁tandem!  应往届" 烧饼"  再次提醒#  家庭主妇#  十一条$  闪闪发光$ 古都% 存单%  环比上涨&  ▁dispose&  无意间'  更是在'  科目二( ului( ▁Buk)  wordpress) 引得*  往年的*  坚持每天+  技术中心+ 代工, ESSION,  ▁charismatic-  放进去-  赚的钱.  machine.  ▁detector/ 大林/ 架设0  ▁hose0  Respons1  ▁Toledo1 搭桥2  愿我们2  人民政府办公厅3 受凉3 烘烤4 Quote4  一年级下册5  ▁Yorkers5 单县6 大块6 搁浅7 Func7  Regular8 两用8 墨镜9 捺住9  接种工作:  ▁piss:  ▁contentious; IoT; ▁Tum<  这样就可以< 深秋= sic= 淘气> 白人> 肤质?  被判处? aldi@  投资收益@  ▁styledA " 中华人民共和国宪法A 市井B 远眺B  ▁ArabsC  比不上C  聪明的人D  都是比较D 岁前E  ▁nellaE 洪涛F  时必须F  随随便G  同时将G  ▁redesignedH 金普H ▁tacI ightonI  建设用地使用权J  ▁pipelinesJ 长跑K  ▁whiskyK  PatternL 重合L gilM  百分之百M 冉冉N 凉菜N  乳制品O  第三百O  ▁binsP  很大一部分P  ▁MaloneQ % 决胜全面建成小康社会Q  ▁tirelesslyR 精炼R  禁忌症S  气象部门S  每天一点点进步T 克利T  名患者U  指挥官U  融合了V 船长V  勇敢地W DATAW  ▁NamibiaX  ▁SergeantX  意想不到的Y 乱想Y 杨树Z 笔钱Z GRAP[  浮出水面[ 不期\ 没得\ ▁IG]  的姑娘]  地点另行通知^  ▁Credits^  ▁differing_  并不代表本号_  ▁Meredith` 一己`  不满足a iseca ▁Loyb nellyb policec  ▁Rebelc  ▁Shaped  Intervald 以史e  ▁outskirtse 其身f 楚汉f ▁UMg  梧州市g  都有一h  ▁Siouxh  erencingi  ▁Correctioni 屈服j  作用下j  真实情况k  ▁captivek  弄清楚l  对于很多l  ▁FAQsm  ▁exemptionsm 荞麦n  他也是n ▁ARMo  城市道路o  的目的是p  文学奖p ▁Styq  ▁Dickinsonq 又把r 遍野r  赤裸裸s  ▁Amnestys pperyt  ▁Ronniet  ogeneousu ▁śu 丹青v villv  技术指导w  ▁mishw 格斯x 行了x 钙质y Laureny 玉泉z  开始在z  橄榄油{  ▁Amar{  ▁Boolean|  医疗健康|  看在眼里}  ittings}  accessible~ 不单~ 煎饼 秘笈  ▁india 和外 恰巧 打人  个小区  理财的底层逻辑 色拉 BLE  独到的 imento  ▁asteroid 无际 酒席  有其他  交通方便  时刻关注  ▁Hipp  ▁affinity 不搞 掌管 数一 竭力 近似 aii  亦或是  君子兰  神经衰弱 agles  示范点 ropic  ▁Babylon URN  名教师  简直太 ▁ATM Benef  ▁billed 上虞 猜猜 远去 远征 uno  袁世凯  ▁Sergio  ▁withdrew  cestershire 满钵  的欲望  而他们  ▁Katz  gorithm  ▁Hussein 容县 闻乐  ▁пра  微信头像  讲的是 伟人  在完成  ▁whisk  ▁maiden  ▁overweight  渐行渐  绕城高速  络绎不绝  铁皮石斛  成分的  ▁все  思考问题  ▁banners  组织者  ▁Piper  ▁Accessories 散落  当天晚上 珠子  感人的  暖人心  要牢记  走亲访友  ▁nets  ▁Gaines 托付 sand ▁Vog  organic 今早 粳米 自力  号汽油  榜上有名 igail 莉莉 行径  等渠道 artner  ▁Sherlock 组工  的现状  可以发现  海洋大学  ▁handheld 贝贝  双料冠军 extern 深表 orc  ▁antiv  _______ 寻根  话语权  ▁escaping  ▁Photographer 千层 败坏 Shel vell  我的孩子 依恋  顺德区  ▁prod  当日解除医学观察 伤员 射线 必应 断路 送入  与时间  记者看到  预备党员 jyj museum  ▁timed‹  语言的‹ ▁smoË  倒挂金钟Ë  ▁Siberċ  ▁impeccċ  ▁Singlesŋ  ▁Advantageŋ 冲进Ƌ 瘀血Ƌ  和消毒Nj  OptionalNj 太宗ȋ 底价ȋ 通络ɋ  不能让ɋ  他知道ʋ ▁Tirʋ  腹泻等症状ˋ  ▁perpetratorsˋ 与原̋  ▁feathers̋ 查房͋  小妙招͋  而出的΋  带状疱疹΋ iasisϋ  ▁Anonymousϋ 叫人Ћ  ▁monksЋ 四期ы 的土ы  ▁urgentlyҋ  ▁imaginaryҋ  ▁detrimentalӋ  也很多Ӌ  ▁ALSOԋ  ▁Psalԋ quiresՋ throwsՋ  itizens֋  ▁Elaine֋  任务重׋  发表论文׋  教学活动؋  餐饮行业؋  ellularً Ordً  ▁Sarasڋ  ▁Magentoڋ 要被ۋ  团结协作ۋ  ▁gown܋  ▁sewage܋ ▁rag݋ Giving݋  ituatedދ  ▁Trackingދ 滔滔ߋ 首部ߋ Lang  重点人员 ▁CPA  相得益  我们还是 琼海  各有各  自治县  有线电视  COMMEND 奇门 牛蛙 avez ▁tet  ▁ageing 其对 段位 比他 自作  集团在  做好自己  微信公号 aggio % 人力资源和社会保障部 善解 月月 oit  文化与 pull ▁Mud  管理机制 ukes ▁bri  农民朋友  ▁Forget  nderland  ▁Instruments  过上了 ensch  保持安全距离  级地震  首席专家  planned  ▁Extract  ▁swings aty  优秀毕业生  Senator ▁"[  方法论  及时发布  发展机遇  ▁wagon  集中隔离观察  ▁replay  ▁cheering  ▁toxicity 西域  实地调研  进行预约  ,　“  出生地  饱和度 ▁PPP 勤快 填满 积液  广大的  每周一  小学阶段  ▁invari 任重 十倍 若您 黄道  名员工 伸着 借条  合理性  ▁Terra 市集 濒临  电解质  加入适量  香格里拉 综治  ▁Zeit Howard  henghuoquan  玉林市  各种问题  毫无保留  磷酸二氢钾 Junior  ▁slump  ▁grabbing  也不可  要求在  接待游客 文峰 精巧 ром  ▁Alam  ▁sham  数据和  西宁市 爱看  每天要  ▁Finish  provider  天天读  渡难关  商品住房  ▁Braves 无踪  周一至周日  ▁Pulse  名犯罪嫌疑人  饮食上  从那以后  ▁Harmony  expression  课程咨询  ▁Economist 的病  水利局  万元左右  企业员工 不公 摆设  整改措施!  苹果手机! UTERS"  ▁cues" 好房# 尖端# ▁TW$  工程款$ Natal% CCI%  大白菜&  新闻线索&  ▁revolves'  集团军' 清仓(  并采取(  多吃一些)  小家碧玉) ensure* aic* Stats+  categories+ 就吃, 质保,  个月以上-  ▁prophet-  ▁evacuated. 杂货. 翻转/  就只有/  应用题0  还处于0  正式进入1  流言蜚语1 热电2  会直接2  眼科医院3  等多方面3 Impact4  操作的4  晋中市5  ▁upstream5  ▁Connected6 罗兰6  停止了7  相得益彰7  喀什地区8  ▁Fixed8 利康9 蓄势9  没有一点:  角度来说: okane;  ▁wiki;  igrated<  ▁concession< 就当= ▁Eh=  不愧是>  背后的故事>  ▁tempting? 两座? 烟叶@  赛场上@  ▁ResponsesA  ▁rationaleA  ▁InterestinglyB 嘉奖B 遥感C kefC ibaoD  知名企业D 北仑E  摆放在E MMERF ▁AufF ▁gerG  ▁blanketsG 直白H 秀才H  ▁fauxI  拥有强大的I 崎岖J 早读J 正因K  了整个K  疫情后L  一系列的L  ▁squashM  ▁furiousM 卷土N 树种N 附小O  一个叫O  的好友P  这辆车P  会有很多Q  的身体条件Q studioR 写上R 小丽S TueS  创办了T  真心的T  身体会U WIFIU  适合自己V  ▁envisionedV 宽恕W 镇人W  总量的X  这便是X  严防死守Y  值得期待Y  生物多样性Z  ▁commandedZ ZH[ 电能[  单方面\  这个价格\  ▁vascular]  ▁Newfoundland] 改了^ 诞辰^  人生最_  甜美的_  ▁chunks` 瓦解`  并完成a  报名的a  这样的女人b  racticalb  ▁pinpointc 两性c 喝彩d  任教师d  平等的e  的一套e 为重f  麻辣烫f  感到非常g ampoog 班列h Lexh  同学们在i 的香i Guyj lloj  又一批k  确实有k  沃尔玛l  肝细胞l 补中m  共同为m 丛生n 前旗n  积极地o  赚到钱o uponp  中心工作p  ▁disposableq 利害q  都成了r  教育工作者r 吃住s 贴着s  信心和t  ▁indexest 参训u 吸血u  ▁LORDv aphaelv 总行w 类推w  车辆在x  如果地球x  底线思维y  ▁dinnersy 保修z 别太z 老汉{  下学期{  ▁Tiny| 唤起| 胃经}  体验课}  提醒你~ Pair~  ▁eyel 王立 超全  后出现  供血不足  全力推动  全国多地  电视电话会议  画面的  ▁televised USS ▁TC  同学在  谈不上  ▁landfill  ▁regression Neuro  ▁Beaver 高气  全链条  ▁consolidate zhe  的歌曲  ▁turtle 下限  务必要  ▁зна  不得随意  ▁fares  ▁Carrier  我只想  承担的  特别是对  数不胜数 　① 乳糖 苦心  RECOMMEND  ▁incarceration  市民朋友  统一管理  ▁postgraduate 围栏  营业员  藁城区  近现代  合影留念 易怒 本真  人数为  而如今  文明单位  解放战争 ifton  ▁participates  已开通 利水 嘎查 空洞  另一名  世界冠军  ▁disregard Ka 体液  的一块  经营许可证  小分子  有很多的  ArrayList  ▁Elephant 整条 sina  分析一下  景区门票  树立正确的 相恋 觅食  ▁Greeks  Hopefully  ispensable  ▁Electricity  ▁simultaneous 紫禁 ||　 ▁Mn  拿不到 vana  即可享受  ▁Mahm  comings  又有什么  ▁turnovers 暨南  生产者  certified 咧咧  感谢大家  Objects  iential  ▁Astros 天国 就送  上高速 两处 律己 新机 随和  申请执行  ▁unveils 每只 越久  特等奖  共同进步 Mouse 正法  ▁Wiley  一次性医用口罩 历届 徜徉 忍让  一旦被  ▁Amph 前有  专业和  本次考试  ▁cube  自己的梦想 十来 又想 大麦  ▁Ко 险阻  以提高  实拍图  髋关节Œ  要求进行Œ  冠状动脉Ì  分析报告Ì  也觉得Č  我们才能Č  asticalŌ  通过与Ō ']['ƌ  为原料ƌ 本钱nj Masnj  理念和Ȍ  而奋斗Ȍ  ▁GOODɌ % 国家市场监督管理总局Ɍ 分水ʌ 要义ʌ htaˌ  游泳馆ˌ alkš  ▁рабо̌  neutral͌  ▁halted͌ 抛售Ό  日常管理Ό Addingό 哽咽ό 维多Ќ  花时间Ќ  廉洁自律ь  repositoryь  ▁substantiveҌ 直聘Ҍ nosӌ  鼓励和ӌ inseԌ  brandedԌ  ▁complexesՌ 淡季Ռ  听音乐֌  ▁Feld֌ 亚历׌ 已获׌  合唱团،  申请了،  规范管理ٌ  ▁Spacesٌ  ▁pertinentڌ 冰激ڌ 满减ی  对面的ی  工作后܌ ▁CAM܌  天津大学݌  ▁Kendall݌  ▁Tasmaniaތ 依纪ތ 可悲ߌ 花里ߌ 退了  计划投资  重点行业  ▁Cheryl  ▁Wilderness  ▁corresponds  年纪轻轻 artime 津县  初级会计  综合医院  购物广场  项目投资  通常情况下  义务教育阶段学校  ▁earrings 去何  始至终  风水学 afone  健康通行码  ▁hypot  营造一个  ▁guardians 把车 李永  ▁Avery ій ▁pd  你这个  回来后  的号召 ▁sir istro  商务合作微信  ▁explode 博雅  ▁rethink 一房 书生  Reporting 额定 Balt  作为一款  开展情况  日发布的  非常重视  心里话  ▁violates  ▁detecting 急了 春的 貌美 photos  ▁accountant 海天 灌输  会议还  有媒体  其实不是  ▁Fiber arty 股民 苗头  ▁franc  ▁mimic  不远处 unde 划归  调整了  现代化建设  according Rs )”,  怎么吃  掌握好  ▁chin  传播者 终末 ▁vow ankind  第九届  显得更加  ▁lawful  亿欧元  教师们 音色  的老板  资产阶级  ▁emblem uw 产蛋  维护社会 不快 被杀 allo Allow  ▁limbs  bearing  blogspot 十多 莱西 所做  features  ▁expiration 兵马  都有所  上幼儿园  保障措施  ▁Samoa kas  最终导致  自己的事情  ▁linem 龙虎  明明是  文化创意  随随便便  usterity  ▁detectives 尚无  要更加  土地承包  在不同的  ▁inhabited ún ść  东营市  都是一种  olithic  ▁Employers! ек!  ▁outset"  ▁diplomats" 　③#  但依然#  副处级$ 长裤$ <<　%  ▁McCorm%  ▁hostage& 荷塘& invoke'  应该注意'  ▁watchdog(  ▁registers( 前要) 图图)  就出现了* 殆尽* bri+ ▁GI+  坚持做,  看到这些, 月下- 白水-  泸州市.  告知承诺.  国土资源/  ▁malls/ 西林0  无非是0  的习俗1  主要考察1 两支2 每张2 ▁Goa3  敢于担当3  ▁intertw4 本刊4 着脸5 ESA5  都会选择6  ▁Crane6  Preview7 未在7  根本就不8  ▁sack8  ▁programmed9 肉桂9  出租屋: ▁Dip: 操守;  的类型;  ▁Lighthouse< 后置<  不提供=  与传统=  张家界市>  有很强的>  ▁reload? 心平? 田地@ 门禁@ wayneA  ▁graftA  ▁porcelainB  还专门B 才智C mergeC  ▁SeventhD 万岁D ORNE  承租人E reseF  三万六千F  ▁CoronaG  这使得G ▁SobH  都存在H emouthI  公共资源交易中心I \":J  对他说J  的高铁K tangK 概论L  能见度L  ▁SpearsM tekM  民事行为N  ▁ExperimentN 国贸O  机动车道O  笔试科目P  ▁MammP  民事诉讼法Q RecognQ 再苦R 准予R jonS  年实现S  综合类T 屈膝T 货架U  医药费U  都变得V  生态环境厅V  ▁AlbuquerqueW  大环境W  库尔勒X rinthX  ▁CrescentY 代的Y 复位Z  ▁МаZ henko[  ▁dislike[ 可待\ 有约\  产生影响]  应变能力]  ▁cleansing^ 递增^  经常出现_ 于世_ 对因`  ophobic` 相加a 文秘a 郁结b 长宁b ▁FTCc  使用寿命c  ▁Rachd  ▁Maximd  无比的e chioe  累计收到f  ▁humidf  ▁labelingg 五福g 糜烂h 肋骨h Raji  可复制i ureenj 省去j  ▁Inchk  ▁sunglassesk  ▁counsellingl  体育总局l  可以参考m  平台建设m 白话n 虽小n  减轻了o  最全的o  各界人士p  著名的话p  ▁terrifiedq  ▁turbulentq  ▁renewablesr pfr  其中最s  ▁Dishs  ▁Kitst 板材t  和提升u  的定义u  快速提升v 外长v  持续向好w 饮片w  相关法律x eteenx  人类命运共同体y 华的y 来时z 浏阳z dbc{  自然资源厅{ weeney|  ▁justices| 太强} 拖累}  ▁vineyards~ 留白~  主题是  保持健康  甚至有些  ▁GitHub 无病 预审 afer  方针政策 客票 浅色  体温计  ▁Fasc  ▁attained  的收益  不负责任  基本上是  银行贷款  ▁Inaug  Historical 惯了  手机端 meng  ▁weaving 汽修 烧至  了一支  每一口  因为自己  新形势  有经验  ▁lingering % 新冠肺炎疫情发生以来 kf  是预防  生物制品 种业  在床上  工商业  的思维  算下来  里糊涂  毫不夸张  ▁quarry  几天后  用户在  假冒伪劣  ▁funnel 省吃  下降到  日之后  可以放心  ▁Maid 验光  大长腿  ▁unnamed  ▁Contractors 仙气  的销量  ▁Polo tracks  servlet  的注意  达成了 avis forum 在美  所院校  学习中心  ▁tattoos  confidence 族人 隔空 kWh  高学历  征稿启事  工作进行了  validation bots  一切众生  高质量完成  ▁chop  放寒暑假  ▁Barbados  ▁currents 劳工  老师们的  醉酒驾驶 AWS  适得其  ▁muscular  ▁massively 酒醒  差不齐  镜检查  ▁airs  ▁invaded  ▁efficiencies  群体性  遇漫遇漫 Course  ▁paving 牙周 许某  太多人  老有所  集中采购  ▁sting  ▁incarcerated 不爽 仍将 Gre  承受能力  ▁dedicate  ▁overturned 伤风  就是通过  比较适合  ▁Carly )> 取景 平地 Rew  回家的  时需要  ▁wrink  cheduled  ▁rocking 两眼  ▁Walls ssh  的神奇 ▁Zip ▁MX  职工的 IFIC  深刻的印象 下联Í  ▁ExecutÍ 找他č  书中的č  路以北ō ershō  播音主持ƍ  毕业论文ƍ  调查核实Ǎ notchǍ 三河ȍ iwiȍ  一张图ɍ  中芯国际ɍ  即可阅读ʍ  ▁extinctʍ 关元ˍ 由县ˍ ▁LAN̍  icators̍  ▁pilgrimage͍ 小新͍  发展中国家΍  ▁Actions΍  ▁rectangularύ 悦城ύ  学生对Ѝ  进行过Ѝ  为了更好的э computэ 比率ҍ  回过头ҍ  这两位Ӎ  申报材料Ӎ 领克ԍ  空气中的ԍ  ▁PatternՍ  ▁confiscՍ 别无֍ 湖面֍ 萨尔׍ 青涩׍  观察者؍ ▁Miy؍  的安全感ٍ  进行交流ٍ  ▁pestsڍ  ▁totalsڍ  ▁yieldedۍ  ▁Islamistۍ  ▁enriched܍ 丝滑܍ 小河ݍ 性爱ݍ  三五年ލ  所以当ލ JECTߍ Namedߍ 南洋  三七粉  ▁Nicola  ▁thematic % 中央纪委国家监委网站 女团 比和  而不见  ▁Micha 花心 ▁DOJ COUNT  公众号后台回复 蝗虫  令人羡慕  群防群治 raltar  regulated  NavBarCell 突兀  时间线  人身安全  ▁obey MJ enegro 扎西 纯度 寰宇  消除了  行业中  ▁Celebrate 候鸟 前锋 proxy  ▁behaviours  杰出的  ▁Kart 吴江 磨灭  五星红旗  同比上涨  对症下药  补充水分  这款香烟  ▁Hath  ▁Malay  独有偶  开始出现  习近平指出  covered  ▁sailed 洒水 难处  住房贷款  大力开展  ▁Fare 原装  这些情况  自己的内心  TextView  ▁Albania 案涉  经核实  路路口  所以现在  ▁Tile  ▁Uzbekistan 大寒 越想  所得额  ▁gourmet  ▁trivial  了一个月 一跳 应交 情理  这个夏天 Awards  ▁runoff ERIC  ▁lumber 和市  就是他  ▁Geek  commission 逃亡  工程等  交换机和  ▁infl  ▁sincerely 鸟语  这一款  每个人都能  ▁Yesterday cw ным  民政厅  的性质  ▁Practition 塑胶 港币 的火 MAP  ▁pedag 争霸 无趣 骑马  古人的  有这种 azzo cling Dragon  有限公司招聘  ▁artillery  社区居委会 买下 积德 Asset  ▁Yamaha  刘亦菲  又不是  的申请 看齐  ▁Pamela  ▁Brittany 万国 临空 已向 轻伤  追求的  妇科疾病!  ▁anatomy! 安达"  艺术教育"  趣味性# uder#  ▁fortunes$ 会从$ 秋水% rift% ▁McB&  hetical&  ▁loosely' 双倍'  民事诉讼(  ▁Rodney( 后宫)  可以接种)  因素影响*  已经结束*  ▁Winds+  特大桥+  不要前往,  圆满成功,  上学期-  家庭医生-  against.  善举和.  职工医保/  ▁crawl/  ▁rejects0  ▁Nacional0  ▁dinosaur1  千万不要错过1  ▁backend2  ▁cinemas2  ▁excuses3 也行3 阑干4  ▁offspring4 法术5 订婚5  思维的6  转换为6 CEPT7 Guid7  把所有的8 strip8 插本9 羁绊9  帮助别人: nette: optim;  Funding;  获奖者<  衣食无忧<  ▁broaden= 关掉=  自然环境> liners>  ▁Finds?  ▁enquiry? 造句@  一些地方@  占道经营A  水利水电A  StoriesB 爱在B 诚招C  发起了C  营利性D ▁gewD  年毕业于E  ▁supervisorsE  配套的F  产品研发F  就带回家G  新闻传播G benefH  ▁perksH  ▁heraldI 租户I  出席活动J  zbollahJ 率低K 表象K  在一旁L skinL icherM ismissM  ▁JammuN  中含有丰富的N 返京O  不听话O  知识体系P  ProcessorP  NeverthelessQ 一斑Q 天盖R 宏大R 成色S 页的S  一定要在T  ▁AbroadT  ▁avenueU % 绿水青山就是金山银山U }>V 得道V 盐湖W лсяW  ▁discardedX 家事X SolidY northY мяZ 走起Z  第三章[  规模为[ amenti\ 握住\ 横道] suff]  其他人员^  数学上册^  ▁cataly_  ▁Practical_ 眺望` 最晚` bosa  了一辈子a  可以设置b  ▁devisedb 隔音c  是当前c  bringingd 小菜d  明显提升e 友的e  只要您f  看到的是f  ▁riflesg KDg 平层h  算什么h obari ▁Voyi 旧情j 盐田j  摆脱了k  放在首位k  都不陌生l  人力资源部l % 重大突发公共卫生事件m 一院m 莫测n  ▁MUSTn  ▁Sponsoro 相合o Sinp  这张图p  ▁americanq 尾数q Scir  留下一个r  ▁brids  disableds  ▁falloutt  ▁killerst 游艇u 溪流u Shav  劳动合同法v  ossiblyw ▁dupw  教育部办公厅x 商讨x 闭馆y  岂不是y  高度的z CODEz remote{ 狼狈{  主力军|  构成的|  营业部} ▁Biz}  ▁Biod~  正确佩戴口罩~  ▁fertile  ▁Maldives  ▁explorer 夜空  为好友  买房子  互动交流  健身房就  随地吐痰  ▁Halo  还是从  进入体检  高度关注  reating  ▁facade  ▁sausage  ▁strategist  要聚焦 ▁Sik  如何解决 调运  他都会  无独有偶 检讨 Nas  或使用  财务报表 essen 乱占  的处罚 ▁ail  参差不齐  ▁staircase 缭绕  耐心地  邮政编码  ▁Schu KL 字字 悠扬 柔弱  最根本  值得拥有  ▁Thirty  ▁checklist 动心 ▁eer  izational  ▁FRAN  践行初心使命  ▁unexpl  版权归作者所有 泥沙 anova  决定自  发展壮大  不要盲目的  ▁Kyoto  ▁Password 师从 木屋  　“　  适得其反  ▁rumored  ▁Greenville 拜拜  号开始  向纵深 taker  ▁mantra  ▁enrichment  ▁Transactions  发送至邮箱  你最好  去了解  小部分  我们正在  ▁Impl  ▁Plays  ▁Hoover  Semitism  ▁progressing 未按  蔬菜和  顾不上 ffer  克拉玛依  ▁HDMI 闭口  并不大  ▁limb Across  associated 即在 活塞  婚外情  应用型  很多问题 查扣 远期  芜湖市  ▁explosions Hear  学科建设  无奈之下  生生不息  洗衣液  消费市场  ▁sympath 亚特 靠边 韵律  印度尼西亚  ▁spoilers 一串 大田 退房 隐忍  是一道  Methods  Seattle  college  ▁automate  漯河市 vings  ▁eruption 照相  不集中 红歌 通向  已构成  读者的  ▁lookout 丧葬 ▁FSŽ  表达了对Ž *****Î  ▁ResponsibleÎ 纸本Ď GroĎ  的团队Ŏ  坚决反对Ŏ  敬请期待Ǝ NathanƎ  ▁inadvertǎ 皆知ǎ  所用的Ȏ  污染治理Ȏ  ▁LimitɎ  ▁outweɎ  黑寡妇黑寡妇ʎ  经常性ʎ  重要了ˎ  地发现ˎ  这么快̎  ▁repairing̎ 孢子͎ AFTA͎  ▁reclΎ 它将Ύ  学史明理ώ huanώ  ▁HomerЎ  ▁obsoleteЎ Gayю  协议的ю  曾表示Ҏ CompatҎ 当庭ӎ  在单位ӎ  新作为Ԏ  旧金山Ԏ  和建设局Վ  ▁mixerՎ  ▁tracker֎ 呼兰֎ 喧哗׎  方法一׎  ▁Toby؎  ▁Swing؎  新冠病毒感染َ  ▁Westonَ  ▁warrantiesڎ 沂蒙ڎ  所作所为ێ  能让人变得ێ anagan܎  ▁mercury܎ 二话ݎ 德拉ݎ  预售证ގ  ▁patriotގ 盲人ߎ  ▁enrolߎ 油茶  的一周 五帝 清徐  有一群 hadow  ▁slowdown 小课  以百年  匈牙利  艺术中心  的一项重要  authors  一味地  与安全  十二年  的旅游 冒用 支点  三部曲  分布于 Dick ▁gol  ▁cliffs  ▁derive  ▁Guinness 立下 笑的 预选  或许会  ▁Brewery  ▁customization 双流 浓稠  不通过  ertiary _{\  上一年  mopolitan  ▁Monsters 远流  的高等  无产阶级  独立自主  年普通高校 aitlin  umbered  ▁Jubilee 肌腱  一直没  反复发作 ialis 牟利 购销 ▁Md ▁CHR  ▁Rack 康乐  ▁mortar ▁DP 优劣 棱角 男装  的开发  ▁ashamed  避免因  都认为 哪位  教师在  ▁coil  定兴生活圈  dingxings  dingxingshenghuoquan 惊恐  生长在  ▁HOME  ▁Shower  Mercedes  engineering  区位优势  文明交通 法医  ▁ми  阅读和  ▁roast  ▁subsidy 偏执 满心 速记  然而在 docx  caution 暖意 mie  独自一人  ▁stor  ▁stealth  ▁humility 历久  面对的 Phill liang  ▁rugs  加胡椒粉和  ▁Festivals 九成  上市后  的成果  金融学  车辆进行  风险防控  navigation  ▁underserved 造影 ▁CG ▁hrs  这项工作  ▁peaked  ▁cherished  ▁plantation 执著  法治化  ▁hiatus 下个 变通  心理辅导  必须坚持  国家药监局 陆游 eny  对父母!  所不及!  使用的是"  如有需要" " 事实上事实上事实上# 初创# 深藏$ edes$  国际油价% ús% 小鼠& 拉尼& 狂热'  不能及时'  教育发布( antan(  picture)  ▁furnace) 收据* 银幕*  做好事+  发布后+ ▁Yak, 可就, 混搭-  一份力-  总体上.  做什么事情. ▁Ath/  促进作用/  创造条件0  的工作中0  剥夺政治权利1  ▁organism1  ▁universally2 当其2 球的3 直筒3  在党的4  ▁Verde4  原作者取得联系5 噱头5  特朗普政府6  ▁preaching6 ×　7 绝伦7 背离8 议案8  的水果9 Mand9  峰值扭矩:  ▁Gott:  ▁paperback; 缴款; 而解<  工作汇报<  戴上口罩=  详细规划=  registered> 旗号>  点上方? abay?  ▁disinfect@ 7 0大幅度发大幅度发大幅度发大幅度发@ 白鹿A 陈薇A ▁TusB  离家出走B  基本圣诞节C  ▁goddessC  ▁shortlistedD  了他们D  合法的E  抗肿瘤E  就说句话F 刘倩F 收成G  除草剂G  和交话费H  飞放不下H  发反反复复I  政协副主席I  飞放不下就带回家J  ▁StrangerJ ALSEK  服务法师K  ▁johnL  和交话费接电话L  圣诞节我还记得M  ▁wicketsM ( !和交话费接电话就是不行N . '发反反复复飞放不下就带回家N . '服务法师事实上事实上事实上O = 6圣诞节我还记得和交话费接电话就是不行O U N服务法师事实上事实上事实上发反反复复飞放不下就带回家P 诗情P  信息进行Q  阅读后Q  会计师事务所R F ?圣诞节我还记得和交话费接电话就是不行只能说R 能不S " 基本圣诞节氨甲环酸S  ▁admirationT 入编T 每股U  公园里U  哪些人V  重灾区V  ▁comfyW 机密W 燥热X 考完X 这人Y  和智慧Y  最受欢迎Z  积分卡技术Z  骄傲善举和[ Linear[  疫情防控工作中\  放假时间繁花似锦\ % 加胡椒粉和骄傲善举和] + $放假时间繁花似锦胡椒粉和] 4 -加胡椒粉和骄傲善举和积分卡技术^ X Q加胡椒粉和骄傲善举和积分卡技术放假时间繁花似锦胡椒粉和^ seq_ ▁vul_  说句话家具`  ▁ashes`  Festivala  北京市房就说句话a g `加胡椒粉和骄傲善举和积分卡技术放假时间繁花似锦胡椒粉和说句话家具b 隐居b 古墓c 轰鸣c 预科d  茉莉花d  补充营养e  背景下长时间e  骄傲和武汉分f  交换机和福建省f % 骄傲和武汉分手机话费g : 3骄傲和武汉分手机话费交换机和福建省g 止步h 遗余h  常德市i  闻名的i  下班就恢复j  的户外护肤j  健身房就撒娇k  结合实际家和k " 和建设局下班就恢复l " 放寒暑假的户外护肤l + $健身房就撒娇背景下长时间m 4 -和建设局下班就恢复结合实际家和m O H放寒暑假的户外护肤和建设局下班就恢复结合实际家和n 各能n  杀菌和送o  圣诞节按实际o  就像你这么说p  就回房间杀菌和p  杀菌和送积分换q " 粉红色就像你这么说q ( !杀菌和送积分换届时将会r  ▁Directorater 7 0粉红色就像你这么说就回房间杀菌和s [ T健身房就撒娇背景下长时间粉红色就像你这么说就回房间杀菌和s a Z圣诞节按实际放寒暑假的户外护肤和建设局下班就恢复结合实际家和t  和维护t  签订合同u  ▁leasesu  ▁clarifiedv @ 9北京市房就说句话杀菌和送积分换届时将会v 衬衣w 毁坏w itonex  ▁Purchx  ▁Disclosurey 花木y  结构性改革z Uniquez  那么问题来了{ 等他{  的作业|  的内心|  ▁dissatisf} GPU}  ▁McGu~  ▁Hazel~  Returns 丝雨 粉刺 dry  商业地产  扎实的  在短时间内 践踏 雄鹰  赶紧来 leine  ▁reins  党群服务中心  ▁CONDITIONS 护法  rocities 爱他 臭氧  的春天  ▁Salvation  天府新区  进行治疗 南向 kat ком  ▁Warehouse 中产 被逼 越冬  ▁modeled  ▁Developers  省吃俭  引起重视 强险 罗定 Opp  至少提前  Emergency qw 柳树 胡歌  撒的发  中石化  操作规程  ▁Wheat  ▁Assets  函数的  联系上  尽职尽责  持续稳定  inderella  ▁indicative  ▁existential 都行 arra ▁inh  有了新的  ▁agility 各具  上升趋势  兵役登记  也属于  爱护动物和  咸阳市  生命线  全部拆除 之欲 小桥  找机会撒  ▁bast  utherland 宝应  不会对  弗兰卡  没有人会  哈酒圣诞节  至关重要的  东方红届时将会  ▁commune  后视镜爱护动物和  啊啊啊啊啊撒的发 ( !东方红届时将会找机会撒 . '哈酒圣诞节后视镜爱护动物和 @ 9东方红届时将会找机会撒啊啊啊啊啊撒的发 g `哈酒圣诞节后视镜爱护动物和东方红届时将会找机会撒啊啊啊啊啊撒的发  ▁resonate C <大幅度发大幅度发大幅度发大幅度发大幅度发 两倍 便将  限时特惠 ursive  ▁Wolfe 精制 酷炫  宣布了  点个好看 Render 捣毁  习惯性 天籁 家叶  临汾市  居民区  ▁amazingly 眷恋  曝光了  看手机  也不一样  医疗设备  提前了解 winner  ▁temporal 剧团 景致  ▁Madh ^ W大幅度发大幅度发大幅度发大幅度发大幅度发基本圣诞节氨甲环酸 ▁wy shit  ▁Spielberg 死记 Jer  取消聘用资格 宾阳 栩栩 街巷  可查询  estation 独到 сли  狂犬病  走出了  ▁Misc \/ 因材 无牌  经济学院  ▁coresÏ 一扇Ï  不参与ď  坏消息ď  的高效ŏ  第三者ŏ ▁MizƏ  ▁arsenalƏ ”…Ǐ 头衔Ǐ  回报率ȏ abusȏ  ▁monkeysɏ  ▁Philliesɏ ✦　ʏ 靠拢ʏ  顺时针ˏ  ▁prayedˏ  ▁allergy̏ 庚金̏  大宗商品͏  ▁speedy͏  ColumbiaΏ 她去Ώ 获赠Ϗ  转载请注明来源Ϗ  ▁battledЏ 三毛Џ 茶艺я  该平台я 五级ҏ 精液ҏ  机体的ӏ blocksӏ 娇艳ԏ  坚持使用ԏ  工作思路Տ  ▁beholdՏ Boss֏  期临床试验֏  ▁termed׏  ▁skyrock׏  Interested؏  ▁statistically؏ 海浪ُ Gerُ  要记住ڏ failڏ  整改落实ۏ  ▁sundۏ  工作推进会܏  ▁selector܏ 房主ݏ  前男友ݏ  的空气ޏ Feelޏ  及时报告ߏ  权属证明ߏ Beauty  ▁Networking  ▁factions 开间 搏击  防范措施 spots  ▁Robotics 松林 果冻  易发生 ▁Vie  技术应用  重要贡献  ▁clich  ▁Wanted DG  艺术馆  要实现  ▁antis  教育和体育局  ▁joyful ▁oft  安排上  带薪休假  检查指导  ▁recycle  ▁contagious 拮据 Baz  我非常  未来将  的不满  勘察设计  ▁NOTE 怒火 支为  热情的  的人间  给排水  如涉及版权问题  Schedule  ▁motivations  一致性  长方形 关机 莫里 TOD  江南北  公平竞争  ▁Thurs  ▁sinks  ▁Robbins  izophrenia 一探 原木 orah  党委常委 赤子  和经验  微不足  只能通过  在线教学  不知道自己  ▁ounce  累计治愈出院  的穿搭  孩子们在  根本就没有  ▁Bridg  ▁troublesh  ▁Investigator 褪去  滁州市  学校食堂  新闻办公室 abouts  城乡居民医保  ▁wearable 发质  生态园  美厨娘  公共秩序  湖南卫视  ▁encycl 名贵 对表 过河  Protocol 演说  才真正 教龄 核电  靠的是  ▁freak  ▁authoritative 断货  一起在 rone sync lip  司马懿  有责任  的角落  无非就是 保重 荒凉  的逻辑  一位作家  批准逮捕  有的地方  重大决策部署  ▁hardcore  ▁disruptions 洗牌 海默 燎原 良缘  恋爱关系 among  不会影响  他们就是  ▁Fountain 六道 不提 就学  和丈夫  招贤纳  你要知道  诈骗分子 音量!  碳纤维! Infl" vous" launch# 吃醋# ▁Kum$  ▁obsess$ 图集%  张桂梅%  党史百年&  取得成功& 你真' 硬背' fmt(  如果用( Often)  大品牌) odel*  ▁Utilities*  ▁transitioning+  流传着+ eeks, exter, 」(- 误食- neh.  ▁conceded. 除以/  ▁Serving/ 商的0 疯子0 遛狗1 FTWARE1  Hamilton2 zez2 是能3  等要求3 iners4  二年级下册4  ▁Recip5  ▁Caucus5 scene6 within6  ▁insign7  ▁expectancy7  ▁sustaining8 北平8  好几年9  爱国主义教育9  ▁McGill: 尘土: eson;  报告新增; ewhere<  定兴生活圈微信号< + $微信关注定兴生活圈微信号= 保山= 又很> 杏林>  Payment? 夜游? 带点@ 成套@  换一个A  看清楚A  不能超过B  irmationB ▁CALC  ▁AlpsC  NigeriaD  全省累计报告D 淡薄E 独资E 龙区F  可以降低F  继续做好G 片酬G  地球只是H  所使用的H  加强组织领导I  ▁PricingI 好课J 广角J  为市民K  连环画K  ▁copyrightedL 陈思L  都很难M ▁FUNM  网友纷纷N CHECKN  ▁militiaO  ▁intimidatingO 即发P  国务卿P  分享一下Q lingerQ 教子R gardR ▁DLCS  第十四届S  ▁EverestT 殷勤T owayU  农村人居环境整治U bethV  可能发生V  ▁JsonW  othermalW 大别X 老外X 蔚蓝Y  或者以Y  德智体美劳Z  做了什么Z  Saharan[  ▁Flores[  ▁streamed\  ▁groceries\ 铁塔]  不停歇]  ▁Alberto^ 禁行^  不好好_  积累的_ ▁ROI`  已取得`  ▁Chowa  collapsea  cylinderb  férencesb  ▁vitalityc  ▁organisersc  嫁给了d  记者来到d  快乐来自于e  ▁tradere ▁nemf  ▁Legof  ▁Olsong  易燃易爆g  Testingh 两起h  可以加i  ▁Winchesteri  为依据j  传播风险j  纪律检查k LUk ▁MIDl 到有l 林草m  去何从m  想要什么n  点击这里n  ▁Charlotto  等相关部门o шаp 我太p 折起q kimq 善念r 国库r  不服从s  的任何s 上联t  不负责t  仅用于u  经审讯u  公共场合v  全资子公司v anssonw  当选为w  想通过x  瞧不起x  都不愿意y dancey  复合肥z  对美好生活的z 工时{ 模组{  和未来|  得出的| ▁RAF}  很有意思}  正确使用~ 乙酸~ cra  阴道炎  ▁McInt  ▁ridge 不接 不识 展翅 腕表  出香味  发信息  汉中市 ▁AIR  ▁рай  会加重  瞬间永恒  ▁capitalize 选人 Iowa  抑制作用  ▁ther  ▁Shelley Rot  Graduate  ▁Shelton 黄曲  到暴雨  情形之一  ▁arte 久别 зна Storm  简直不要太 finity 南都 印染 olu  他们已经  这就意味着  ▁Institutional 本息 联运  商贸城  如何用 ratch  ▁Serie  ▁Isaiah  ▁faulty 岳飞 王寒 忍者 我给 无话 的词  情不自禁 " 国家卫生健康委员会  ▁Bernstein 伊藤  住宅楼 Palest  ▁Weird NK 再忙  留不住  ▁carers  前列腺炎  ▁bisc 后用 AMI  报道了  曲靖市  就业机会  未尽事宜  eremiah  ▁ancestry  和经济社会发展  hetically 比丘  甘油三  ▁Pompeo  ▁succeeds 射程 玉皇 Mix  毛主席才能 呼呼 带出  三年内 FLAG  为什么这么  ▁Broker 入冬 入秋 fits ▁EPS  肾上腺素  自主创新  Benjamin  ▁Naturally 她了 ▁Ft  ▁footh  ▁radioactive 楚天  老客户  公共服务平台 姚策 德清 突袭  与当地  可导致  着力解决  结果发现  上的一粒沙 Points 晓东  向上滑动查看 财库 转战  放在一起  的创始人  ▁Plug 娓娓 零散  ▁Loch  ▁Audience itel  一个月内  安全驾驶 别把 IPP  ▁Sidd  ▁misdemeanor 提着 造诣 icc 刷脸 助眠Ð 均应Ð 明码Đ 林海Đ  一直很Ő  不放过Ő  水涨船Ɛ  中国联通Ɛ  回家看了ǐ  如图所示ǐ  ▁PercentȐ 大肉Ȑ  广高速ɐ  参与其中ɐ  ▁Shayʐ  普普通ʐ  个性化的ː  网络报名ː  ▁Departments̐  ▁contrasting̐  的一步͐ bron͐ ZJΐ  ▁Jokerΐ 司空ϐ 失明ϐ 青城А  ▁LadenА 中超ѐ ▁"";ѐ CREATEҐ  dioceseҐ  ▁bumperӐ  ▁PodcastsӐ 先祖Ԑ 比心Ԑ  安东尼Ր  有了更Ր  计算公式֐  ▁pavement֐ XPא 购入א  手抄报ؐ  义不容辞ؐ  与宇宙相比ِ  ▁Schumerِ  番禺区ڐ budgetڐ 公私ې 固有ې 大闹ܐ 少说ܐ 检视ݐ Appendݐ 酒馆ސ  新年的ސ  驾驶室ߐ  相关文章ߐ  革命老区  ▁oval  ▁Marcos  ▁prevail 于事 掉进  太平洋上的一粒沙 7 0哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈 校验 demo  的一段话 ouched webkit  ▁transcription C <微信公众号微信公众号微信公众号微信公众号 变故  指芳华 tera  的女孩子  Industrial  ▁butterflies 九宫 无罪 烽火  赏昙花 不急  的制度 lein  清明与宁静  ▁Dillon " 如果地球与宇宙相比 ugo  被骗了  Limited % 能让人变得清明与宁静 外衣  刹那美丽  昙花虽然  Islamic  但也是刹那美丽 . '赏昙花能让人变得清明与宁静  渐渐的  ▁Lone  ▁Astra  关于进一步做好  翻天覆地的变化 李春 村口 露头 Synt  弾指芳华  详细地址  ▁burgers  联邦政府  均为境外输入  昙花虽然弾指芳华 赐予  学习到  劳动者的  ▁descended 气道  上下滑动查看  ▁mosquito  若不撇  ▁Burning 打骂 ▁MG  儿媳妇  全体人员  ▁converter + $版权归原作者及原出处所有 质效 inj  ▁embell 既可  多少次  等严重  ▁Cobb  ▁Vest g `微信关注博野在线微信关注博野在线微信关注博野在线微信关注博野在线  收得住  大大增加  Failure らい  已获得 Rece  时要注意  ▁wrapper  绚丽的  ▁heroine  ▁zombies 咋办 欢快 而就 谦逊  并积极  平安健康  Traditional 本村  事件发生 dll  对部分  指导性  研发出  ▁mirac  绿色科技管理 % 零售价格绿色科技管理  或通过  第四批  轻松音乐  ▁Beirut 备胎  算不上  拉开大家 archar  ▁Retreat  而产生的 rapped 某天 泽区 的石  正当理由 " 看了就立刻就立刻就 依附  周总理  还有一点  urricular  ▁monopoly cou  后果由 乐活! 玉雕! 脱俗"  仪表盘"  数不清#  设计理念#  ▁Cube$  ▁propriet$  ▁Apartment% 牲畜%  药用价值&  ▁hast&  ▁rash' 奥会' 煮粥( 维保(  小型客车) ovina)  ▁Retro*  ▁Voters*  ▁Olivier+ 两根+  ▁tyres,  今天夜间,  结束之后-  ▁Athen-  剧中的.  埃里克大家. hx/ ▁WP/  高性价比0 末日0 废墟1  吸引着1  拉法基绿卡2  垃圾疯狂垃圾2  ▁Hazard3 " 九分裤垃圾疯狂垃圾3 中青4 街镇4 起手5  考试和5  ……”　6  技工学校6 imble7 桥下7  何去何从8  ▁Tire8 忧患9  三年行动9  ▁Neuroscience: 4 -统筹推进疫情防控和经济社会发展: fz; 以德;  的一笔<  联系的<  晴间多云= ZKJCG=  ▁whichever> 听讲> 无息? 红心? effic@ 松劲@ 补发A 颈肩A  改善民生B  联系客服B 成章C  ▁descendC 猫猫D 转了D furE  恩施州E itosF ▁BARF  两者的G  却不是G  水上乐园H  我们来看看H 例会I 周至I 手相J  学史增信J  ▁cloudyK  DemocratsK  ▁HumanitarianL 更广L  日以后M  生态圈M  ▁showroomN 索道N 麻黄O  的付出O  还有的P  小时左右P radorQ  部分地区有Q serialR  InnovationR  ▁TelescopeS  有人来S  美术家协会T  ▁SinclairT 父爱U  我们平时U  ▁slogV  丽水市V  扩大了W  从源头上W  ▁GivesX  ▁SerbianX 唐某Y  ▁ElimY  ▁prognZ  ▁collegiateZ 向荣[ 在金[ 最广\ 转出\  已基本]  ▁Irene]  刑事强制措施^ 句式^ 河山_  要好好_  兰州新区`  四次会议`  ▁bridesa 大竹a  天内有b  灵活的b  用户体验c  矿产资源c  也许你d  又是一个d 此为e 站上e igorf ▁Gebf 突飞g  是一群g  周末双休h  roideryh 喷涂i 日用i  都不错j  thodontj  ▁seasidek  Authoritiesk 建发l 故人l  ▁Chargesm 切丝m 帮着n 强县n  和农村o  均衡发展o 对所p 提上p  马齿苋q  基础研究q  提供更加r  稳定的工作r ljs  刘少奇s  的年轻t  ▁Alleyt  ▁Sabhau 等来u 维埃v 雪爪v  将全面w  张学良w  报录比x  独立学院x  经研究决定y  ▁Carmely 会做z  的光荣z  到就近的{  揭牌仪式{  \\\\\\\\| 下身| 内附}  这些行为}  ▁nutrit~  路以西~ 佐酒 守正 Nut  月至今  的面积  五百多天 不醒  精神风貌 paste pb 撮影  郴州市  已经达到了  abyrinth 在必  下发通知 嗓音 总第 辩解 ▁dy  ▁shouted 与爱  卖出去 ankar  ▁promoter 未央 辉县 zai  争取到 ▁Eck  ▁loot 最是  化发展  在空中  的一座  的标签  计算方法  ▁Judges bars shou  应纳税额  开学工作  ▁Champ  ▁tremendously 焦化 YES  如果需要  ▁adjud  般若波罗蜜多时  时光的  班级群  盆满钵  近亲属  ▁Largest 扣留 那颗 中金 大华  南充市  年一季度  考试录用公务员  ▁lesions  各地市  新品种  第十届  ▁echoes  ▁faults 相片  每一天都是  应纳税所得额 煤电  同时要  笔试和面试  ▁caliber  ▁Statistical 再吃 酸枣  Charlotte 弱项 补上  普通高考  职责分工  ▁Fate  ▁Souls  ▁empowers  ▁exploits  后半生  帮助他  陪你一起  ▁Garland 掏空 跳跳  血压升高 抱住 Ashley  ▁veins  ▁Continues 休整 突显  重温入党誓词  可引起  尽可能地 干爽  回答问题  ▁visc  与我们共享  ▁Bravo 体裁 оло  有一付  上了年纪  很有可能会 liable  ▁travers 倦怠 天柱 急招 有这 被爱 雪域  上没有  蚕丝被 良机  返校复学 brown  ▁Zack  ▁gala  吃饭的时候  ▁Bowling 从他 致贫  禁止使用  ▁ditch  ▁residual 精良‘ 追着‘ PitÑ  里没有Ñ genređ  ▁Fremđ ulsiveő  nominatedő 神态Ƒ  ▁shedsƑ  ▁armiesǑ  ▁blondeǑ  ▁globalizationȑ 几经ȑ 鞭策ɑ MVPɑ  上一口ʑ  不断强化ʑ  就是如此ˑ  ▁conversionsˑ 且无̑ 大椎̑ 绕道͑  设施和͑  经国务院Α 全警Α 拆开ϑ 风变ϑ  反义词Б  的心中Б stableё 野兽ё  不大于ґ  检测卷ґ  身体验ӑ  ▁SPECӑ  说得有意思ԑ  ▁benefitingԑ 避风Ց  到指定Ց  快递小哥֑  图片仅供参考֑ 岁岁ב 的吧ב 褐斑ؑ  活动室ؑ  铺天盖ّ ▁POSّ 风中ڑ  先生在ڑ  只能靠ۑ  调研指导ۑ  ▁pornographyܑ STEPܑ  财务人员ݑ 带鱼ݑ 盗汗ޑ 虎牙ޑ ▁Izߑ  产业体系ߑ  ▁Austen  ▁Container 鸿妮  以及相关 CHOOL  ▁Guill 住终 种养 脚踩  一味的  实是不易  让很多人 zys  流口水  调整的 ULAR  学习氛围 Alert  有一付对联 dialog  他们把  怎么样了  能有雪爪  ▁Petro  能有雪爪鸿妮 新沂  喜闻乐  的办学  招生考试院  ▁Cooking  三万六千五百多天  ▁ignorant + $也不过是三万六千五百多天 IMS  以百年记  撇捺人生  的姿势也  住终有苦  若不撇住终有苦 % 有一付对联说得有意思 Kam  保障机制  ▁cornerstone  考试题  ▁idle  ▁Fairy  ▁Tunnel  ▁reacts  ▁positivity  室主任  人这一生  这么些农村  向法院  人民路  即成名  年北京  特别容易  ▁CCTV  ▁descriptive 逼真  多月的  微不至  运行情况  ▁seismic  ▁squirrel 有生 都做  建立一个  留下来的  Inspired 站内  潜力的  灿烂的  特色美食  运营中心  ▁Cous  明显支持你  ▁Chaos 银屑  附加值 MH 功于 横扫  这一套  ▁Heard 弹道 试驾 驱寒 即苦 富阳  ▁revived  ▁unmatched 与世  任何机构  焚香品茗  listing  担心自己  ▁crane  优势的 ▁***  我们也要  联动机制  ▁Calder  ▁thwart  多姿色  ▁Came  ▁selfie 古田 复业  是人们 ébec ellan ranged  ▁Bergen  ▁welding 恶人 边角 ден  会降低  倾家荡  还是得  ▁NAME  ▁unsc  ▁Alumin  Philadelphia 能除 边走  多部门  中国制造  ▁Abby  ▁indicted!  更严重! ▁NYU"  纳税申报" Katie#  捺住即成名#  副总理$  渐开花$  组织专家% 自测%  传媒集团& " 行深般若波罗蜜多时&  ▁Rotterdam' 坚挺' 白芍(  经现场( ▁Ain)  单位名称)  诗词唱和* ETHOD*  ▁invoke+  ▁painters+  ▁choreography, 热了,  台风送清凉-  ▁Powder- 日韩.  工程和.  人到闲处/  妙曼多姿色/  小暑台风送清凉0  ▁Congratulations0 光山1  安心静观1  安心静观渐开花2  ▁Reverend2 圈套3 心碎3 tie4  明星的4 计费5  就不再5 存的6 慢病6 闭谢7  聊聊天7  限制了8 Cass8 prec9  学校教育9  均匀的:  多多少:  ▁Coul;  六年级下册;  ▁skeleton< Isa<  健健康=  恢复原=  ▁Lara>  新冠病毒检测>  forEach?  Catherine? 消融@ 药理@ DougA  他是一个A  党的组织B  ▁SubmitB  联合发布C helpsC  ▁whimsD  IllinoisD  决定在E  有些地方E  ▁lubricF  充满活力F  机关干部G 晃动G 浅浅H ) —H  ▁облаI  全部转载I opedicJ  ottagesJ 穿山K  也很高K  奇怪的L HeatL MahonM  因工作需要M  negativeN [/N 合体O 知觉O  将孩子P  新风尚P  未落实Q  不在少数Q  写作能力R  特殊性R  icultureS  religiousS 晶圆T  派出所所长T  ▁canadaU  的创作U 音节V 伸缩V  出身的W  刚烈异常W  极其狂放X 融会X HarmY 响水Y  ▁PersonnelZ  forestationZ 信中[ 馆藏[  ▁blev\  闭谢的姿势也\  keywords] % 闭谢的姿势也刚烈异常] 互为^ 勤勉^ 磨炼_  拿下了_  施工方`  的统一`  考试安排a  ▁frencha  方式为b  的军事b  不减姿态c  象倒挂金钟c 受试d ▁STEd  ▁renale 吊车e 었다f  学史崇德f 煮透g 而走g  一出生h  Schoolsh  ▁nuancedi  ▁deferredi 直射j 稚嫩j  天水市k 宗彪k  ▁dividingl 别字l 善后m 安仁m 数列n 重则n  根本没o ▁Urso  服务于民p luckyp  ▁analysedq 器等q 战乱r  中存在r  是针对s  液化气s  本次课程t  观察细致入微t  ▁embodyu  ▁safariu  ▁masteredv  ▁Trademarkv 太监w  我的人w  ▁Jaguarsx  ▁waterwaysx 入读y 又以y 格纹z  据中国z  的支持下{ ueblo{  最重要的就是|  ▁gritty| 呼喊}  无前例} ▁gaz~ ▁xen~  人的生命  所有内容  报名流程  科普知识 abbage  ▁mentorship  ▁menstru  ▁scanned 窃取  第三位  步步高升  ▁sans RIGHT  公司总经理  justice  ▁pendant 发情 电报  成员的  administ  切好的  控制线  经审查  肉制品  ▁JOIN 山石  快节奏  小学教师 梦寐 猪群 猪肚 订餐  是人字  沉迷于  见不鲜  都来自 导学  控股股东 Except  Morning  中有一句著名的话 高平 在党  信息等  环节的 bies  经济的发展 Flight  ▁Tables іль  多元化的  微信红包  我欣赏昙花 winter 丝袜 之言 开标 blob  就业方向 thumb  如果你也想  巜人类简史  危害公共安全  ▁Efficiency  撇不出  艰难的  ▁Canvas  与我们共享昙花  快乐来自于内心  ▁Measures  并不是说  经济复苏  远景目标  ▁Ethnic ▁SAN  大年三十  ▁Trustee  ▁Powerful 登封  考核合格  ▁CHAR worked 出钱 硅谷 私有  路以东  建筑垃圾  ▁rude 圣洁 映射 长清  少出门  草木记  营养均衡  balanced  ▁datasets 勤于 声势 夫子 沉溺  拉肚子  珍爱生命 chlor  重点项目建设  ▁Provides  ▁heavyweight 八百 多赚  ollower 心软  哈哈哈哈哈  ▁Ranking 以更 远见  希望广大  生猪养殖  新世纪的口号  ▁Gerard  各能捺住即成名  ▁invests  ▁vouchers  科研攻关  项目规划 gering  ▁Yugosl 超时 ▁Kru  高度评价  ▁Revealed  ▁supremacy 千篇 马达  能提高 phys ▁Lau’  技术培训’  flowersÒ  乡村振兴有效衔接Ò 一山Ē  为昙花Ē  要明确Œ  追究法律责任Œ savƒ 冷清ƒ 斑驳ǒ 机灵ǒ ayoȒ  不见面Ȓ  是按照ɒ  没有出现ɒ 记事ʒ  必须得ʒ  醉里挑灯˒  ▁Aunt˒  ▁pept̒  你就会发现̒  ▁earnest͒  ▁prompts͒ 凉快Β 阆中Β BALϒ  不简单ϒ  有朋友В  ▁BareВ 我怕ђ 赵立ђ blindҒ  ▁MemoriesҒ 善缘Ӓ 空隙Ӓ 驱散Ԓ  通过这次Ԓ  ▁hullՒ  ▁PUBLICՒ  一撇一捺是人字֒  probably֒ 也请ג 速干ג  疑难杂ؒ ▁Najؒ  ▁Meritْ  查尔斯ْ  捺收得住ڒ heringڒ  捺收得住是名ے 老话ے  能获得ܒ  驻马店市ܒ  attributesݒ  作画册ݒ  顽强的ޒ  参会人员ޒ  撇不出即苦ߒ  ▁inauguratedߒ  原子弹  车主们  三五小时  到来之际  村党支部  ▁Fiona  ▁Earnings 即逝  ▁ком  ▁Islanders 的油 粉条  岳学子  转基因  ▁revered  村民小组 esley  ▁austral  ▁Architectural 那场  卢霞客  成交额  ▁hing  为昙花留影  ▁noisy  oscopic  为昙花留影作画册  ▁intrigue  ▁extremism 赃款  家叶晓光  ▁hail  与台岳学子  最低生活保障  ▁Amateur  卢霞客与台岳学子 伊春 驻守  而且还是  Editorial 所载 shr  成这样 Bear  ▁Gina  撮影家叶晓光  ▁contextual  ▁disrespect  ▁Theological 二来 是给 牌坊  他们连续三年 面朝  甜蜜的  ▁whip 厚植 都让 iju  组合拳  ▁stains 嘈杂 白眼  二十九  ▁unsur  ▁vitro  ▁overdue 身患 龙骨  还真是  那么容易  的人际关系  dispatch 拼了  如此一来  抓获归案  ▁Demo 代偿 有肉 pow  到当地  第一代  具有正常  这种说法  ▁Merchant  ▁Seriously 堕胎 细小  一起为  学术交流  everything  各领千秋  庆祝活动  弗洛伊德  hampton  ▁fringe  开放的美好时光 1 *与我们共享昙花开放的美好时光 尿裤  红色旅游  ▁Clippers 元凶 抖动 青浦 UNG  每次坚守  就是我们的  江南草木记  讲到了这件  ▁meats  文字唱和  讲到了这件乐事  ▁lifespan 年春  色彩的  批改作业  烟草专卖  ▁dictate  ▁equival  ▁stalled  每次坚守三五小时 宏光  的小编  通报批评 激荡 ohl  搬到客厅  诺贝尔奖 土生 异域 阿迪!  一个家!  具体到"  有独钟" ▁Rak#  已解除医学观察#  ▁Shadows$ 立冬$  有一个人% aurant% compar&  landers&  ▁Whereas'  帕金森' mmol(  任前公示(  ▁Cathy) 以继) 薯片* 长袖*  尽最大努力+  ▁cleaners+  ▁Christchurch,  王寒夫妇,  ▁Beef- 封堵-  在事业上.  总结一下. timer/  ▁indulge/ 百味0  专业机构0  ▁adjunct1 水是1 grove2  ▁Burma2 性温3 爱丽3 集赞4  延伸至4  持久战5  还是想5 不炒6  时指出6 Mess7  ▁REAL7  erdinand8 很近8 有别9 等次9  性骚扰: ▁Hun:  保持稳定;  ▁lodging;  ▁adherence< DW< 脐带=  开创了=  也被称为> ourke> umeric?  ainment?  ▁statutes@ Vs@  解释了A  贸易商A  ▁ScottsB  SecretaryB  ▁GreenwoodC 与国C SprD  大家也D  已有的E  还有点E  最美逆行者F  ▁oversizedF 冷热G  的有关G  isabethH @ 9并不代表本号赞同其观点和对其真实性负责H 拉链I  张爱玲I 剧透J 天心J 霍普K TESK  的心境L iranL  过敏性鼻炎M  ▁unfinishedM  就喜欢N  扫码咨询N  ▁LevinO  ▁gloomO  IndonesP  ▁skepticP DannyQ 元至Q 多才R 降火R PelS  出一条S  汉武帝T limeT  的一次性U  也是非常的U  新闻客户端V  ▁MongoliaV 大虾W 街舞W  不要说X  莫大的X  调度会Y 拐弯Y 指正Z 顾忌Z iska[ 写明[ 同向\ 总比\ 戈尔]  完美结合]  说到这里^ 取下^  把工作_  ▁debuts_ 喂奶` 已无` 新郎a 派发a 视感b 重负b OLOGc  先后顺序c  ▁Coppd Nicoled  ▁optinge  ▁Operatore 造的f  国内中高风险地区f  秀恩爱g  罗伯特g ursedh  级应急响应h  ▁Corpori  ▁adaptationsi 中是j 就做j 得胜k 方块k 缺钙l чноl  不止是m  恰恰是m orrhn ▁LTDn  ▁defectiveo  便可以o ▁nudp ▁tyrp  一级建造师q  日表示q  ▁Coffr 休眠r 水运s 腿长s 萤火t  正方形t  ▁Mishu  可适当放宽u 交锋v  ▁quilv ( !践行社会主义核心价值观w 不食w 溥仪x 闭眼x  原计划y  ▁shortesty 共情z 等与z  创新性{  ▁Builders{ VPN|  的规划|  ▁alterations} 其事} 此款~  财神爷~  ▁allegation 她都 恩师  值班人员  最珍贵的  industrial 八成 跋涉  室内外 Throw  扎扎实 器皿 快消 百倍  期待你的  ▁dialing POP  正处在  源远流 ▁NOR 管齐  而感到  ▁congratulate 霸主  bernetes  ▁sponsoring  可能造成  ▁washer  ographically 大促 逆天  ▁Committees  ▁supermarkets 玩玩  intestinal  ▁redundant 出过 孜孜 淫秽 渺小  依然在  ▁softer  FORMATION wow  看到他 STAR  回族自治区  的转变  自查自 aldo  荣华富贵 stall  ▁showdown  ▁Destination 不香  为别人  ▁shocks  ▁abdominal 喝汤 金桥 飞快  ▁ле  农业发展 tober  ▁Separ  ▁Paperback 披着 母羊 脂质  史以来 ▁DON  全部都是 Helen  ▁Joanna 加利 才被 掌中  他们家  救死扶伤 有求  正准备 ▁Ips  的影响下  搬进阳光房 大修  专业科目 捷克 ODY  兴安盟  该小区  改革试点  明显下降 azaar 排长 河路 飞鸟 abo  语文老师  ▁Tues  ▁ascertain 凯特 豆子 赢在  却总是  可能与  ▁thieves  ▁aggravated 以小 全息 豆粕 ▁Lod  　//　  ▁sudo  缴费基数  ▁pumped  ▁Cruises  年全省  最严重  ▁melan 西西  中心等  杀虫剂  ▁reefs  ▁Farrell  ▁concluding 冷酷  附答案  饱满的  托幼机构 ovolta  housing  chanical  ▁Reception “[  周口市“  临床经验“ 呜呜Ó  异性的Ó  生产能力ē  交通警察支队ē  ▁Brendaœ 迷离œ MHzƓ  病死率Ɠ  也很简单Ǔ  等各方面Ǔ  释放出ȓ  业务范围ȓ  换届工作ɓ  ▁Clarenceɓ  ▁Motorolaʓ 借了ʓ  和幸福˓  工会主席˓  健健康康̓  斗罗大陆̓ flash͓  ▁Crimson͓  年招聘Γ ▁HakΓ ADSϓ  ▁ariaϓ  ▁DiscriminationГ  ▁OuterГ  Republicanѓ  ▁foreclosureѓ 沙县ғ  一切都会ғ  央视新闻客户端ӓ 武大ӓ  也很少ԓ  审判员ԓ  牛羊肉Փ  能吃苦Փ  forming֓  宁夏回族自治区֓ 凌乱ד  可以提前ד  开学典礼ؓ ( !不对所包含内容的准确性ؓ 周长ٓ 村屯ٓ  道路通行ړ  Attorneyړ 清心ۓ 父女ۓ  开学了ܓ  神经外科ܓ  游乐园ݓ  行驶时ݓ  ▁covidޓ  ▁immortalޓ 新密ߓ  不让人ߓ  记录下来  ▁ANGEL 推手 跟人 齐抓  新四军  记不住 erty  不容忽视  ▁Gregg 八里 轻则 intent  中小学幼儿园  ▁extremists 拘禁 瓦斯  持续改善 海西 限高 ▁SAM  社区团购  积极引导 Polit Become  generate 唐宋  brother  普通话水平测试 面罩  大面积的  ▁composing ropy  ▁Wend  kefeller  ▁novelty KC 望而  大力实施  计划停电  Conference  ▁Connections 是先 肉牛 该文  的边缘  在线咨询 ># 安岭  以一种 埃尔 要花 ULD  是否为  小时内删除  ▁exhilar  ▁desserts  入眼帘  大学等  是企业  ▁Cary  规范佩戴口罩 个镇 陈建 Coin  学会理财 FIELD etrics  ▁Colombian 不决 得利 绳索 迁就  大幅增加  高层建筑  是最重要的  consumer 固本 甲骨  itudinal 温顺  高中部 nych  传媒学院  ▁Kobe  海南自由贸易港 漫游 铺开  上年度  群众利益 Campus  ▁forecasting 唇齿 文斌 Fed ▁ml  湿气重  ▁uncond 不因 绝境  的主流  ▁saints  供给侧结构性改革 是高 杆子 造船  从重处罚  ▁arist 咸菜  和投资  建设者  的运营  重阳节  什么原因  住院患者  促销活动  ▁bilingual *( 亲笔 耐看 视唱  并提交  给老师 enia  ▁Memoir  ▁landsl 美编  要突出 ISBN  ▁Rend  ▁electorate! )|　!  心理问题"  昼夜温差" 万份#  外界的#  ▁fiery$  ▁linger$ 威士% 它对% 点下&  可容纳&  养生保健' 宿松'  饱腹感(  拒不执行(  在海外)  人生哲理)  ▁Sung*  ▁Cognitive* 未到+ 神采+ 维奇, vac,  ▁Bolt-  ▁corridors- 是让. 郎平.  ▁Jiang/ 会战/  有一只0  火热的0  视情况1 amus1  ▁Hanson2  ▁cervical2 扬扬3  知乎上3 ITAL4  ▁inspected4 存入5 干为5 铮铮6 降噪6  依次为7  ▁inequalities7  ▁instrumentation8 城内8  将更加9  ▁Ramirez9 展品:  做一名:  奶茶店;  战役中;  ▁Begins<  ▁Undergraduate<  少之又= umba=  并不需要> ellas>  ▁baff?  ▁Merry?  ▁dolls@ 余种@ 环形A  俯卧撑A  而不为B  蒙古语B  的观念C  足球场C  那些年D  ▁packetsD + $治理体系和治理能力现代化E 秀区E ланF  主席团F  他可以G  的力度G  的题目H  选择合适的H 美媒I  不考虑I onistJ  ▁RefugeesJ  ▁giganticK  岁孩子K  我一个L  网络赌博L  ▁astoundingM 武当M ▁FavN  安全可靠N  ▁ShaunO  ▁ItaliaO  ▁CelebratesP 员外P  大扫除Q  请放心Q  ▁LunarR SSLR  下发的S  控制和S  煤化工T ▁ZIPT  ▁EstatesU 产自U 张明V  最高法V  都特别W  ▁riffW  CulturalX  ▁snapsX 五常Y 满园Y 短途Z 脚掌Z  好多人[ 口算[  心想事成\ 卷发\  新赛季]  是有意思]  ▁inventive^  舒服的^  imeters_ 事通_ 满眼` 腾冲`  再也不用担心a ńska  净资产b wendb 四口c 墓碑c 鼓劲d  精挑细选d  ▁hamme  应该有e  最后再f  目的就是f 们也g 已到g 措手h 轮到h 这枚i  李清照i  开展调查j  疾控机构j  不久的将来k  ▁Galwayk 湖边l igual  一如既往地m  ▁Comcastm 先说n 声色n 洲际o 脾经o  由政府p  让人感觉p ⊙　q 计提q kongr ▁ginr 成因s 溺爱s  两年内t  打印机t  社会团体u lehemu  ▁Figuresv 敬礼v  婚姻法w  ▁Kimberlyw 军政x 这则x  不一会儿y  ▁Deafy YLz 潜入z 花瓶{  就会在{  每天有|  积极做好| 土城} 连胜}  ▁fertilizer~  ▁decentralized~ 下蹲 者是  二选一  包吃包住 broad  工作的意见  ▁Emeritus  ▁Appalachian  ▁psychedelic 一做  小客车  请自行 口粮 楹联  名单中  热爱的  用手指 Chan  采取行动  ▁geopolitical 乘以  ▁malt Oregon  采取有效措施 营镇  调整到  ▁petitions 车费  眼耳鼻  肌肉的  招生对象 Laure Ground  ▁Clyde  ▁preparedness 却因 原作 坑洼 小牛  提升到 coins  ▁Mush JPG  亿港元  名师工作室 Expert 地从 真金  示范县  发生改变  投资基金 耿耿  丧失了  正常运转 三藐 倾城 hoff  ▁fung 肥大 ipzig 件的 其人  民主主义  进行管理  ▁owes 和最 追击 送了 mmm  会计学  晋江市 贸然 hog  能找到  免征增值税  ▁Sinai  ▁exhibiting  ▁Examination 全胜  占有率  如美人  研究工作 也因 新桥 狗血 ▁Uns  重点检查 exist  ▁Riot ▁LOT  对于这个  ranking  Calendar 想问 老三  人口老龄化 IJ 客机  避免出现  请耐心等待  ▁betrayal `)  新时期  这是他  由用人单位 委婉 的哦  文学院  买衣服  是经过  结核病  各位小伙伴 ucking  ▁Boone  industry 雨滴 HAM Redu  ▁automobiles  不能有  即可免费 抓取 秋月  集中统一  ▁Keen Yellow  按照有关规定 在车  我身边  ▁Foss  ▁Feedback  ▁nightclub 屹立 西边  一顿饭” lite”  无法确认Ô  ▁WHATÔ 泡菜Ĕ  ▁“…Ĕ  ▁GunnŔ  ▁MadagascarŔ ▁ Ɣ 逛逛Ɣ  关于在ǔ  系统将ǔ  ▁LicensingȔ 好酒Ȕ 徽州ɔ  产业升级ɔ Ratedʔ  ▁Vanityʔ  ▁Randolph˔  ▁premieres˔  ▁respectable̔ 火遍̔  但仍然͔  ▁Meets͔  ▁flattenΔ  ▁impersonΔ  ▁readableϔ 画上ϔ  三藐三Д  скихД 已与є 方志є UCKҔ  就比如Ҕ  有多好Ӕ  报考资格Ӕ  ▁SindԔ azeeraԔ  ▁flawlessՔ  ▁reversalՔ 建投֔ Jake֔ Tribה  投资集团有限公司ה 慎终ؔ  商家的ؔ  经营户ٔ  ▁bomberٔ  ▁nauseaڔ 暗淡ڔ 格雷۔  而造成۔ 黑河ܔ  努力地ܔ  能量的ݔ  邓州市ݔ  ▁Alessޔ  ▁interoperޔ  熟练操作ߔ  ▁Helpingߔ  ▁Petty 狠心 eche  全面胜利  屡见不鲜 Pretty  resolved 控油 特有  重点对  新闻广播  ▁gren  ▁bland 伤身 减重 易购 Clark ridor 关镇 荒废 udad  史无前例  ▁Distrib  ▁Polytechn 曲子  宣讲团  的信任 ayas ussi  ▁Continuing  ▁incorrectly  ▁prominently 幼苗 枫桥 焦躁 赶集  新型城镇化 依稀  医学院附属  Whenever  ▁Picasso 颜六 Diagn 所向 碧海  新进展  焦作市  estrian  ▁unveiling  ▁Experiences  七八糟 产于  关工委  在治疗  在这段 retch  ▁Honorary 求你 测速 纯银  赚到了  工作作风  精细化管理 双龙 所占 随喜  确立了  ▁yuan  还要求 ▁Vor  带来的影响 彬彬 竖起 翱翔  在市场上  报名截止  请第一时间  新冠肺炎防控  ▁shrinking 王健  雪中送 atown  ▁stamped 烧毁 âte  也不行 ▁rou IVERS  ▁Elton  ▁bonded 归结 最易  纯电动  上的便利  大幅下降  ▁Tusc  ▁startling 开市  最前线 fing  千呼万唤中  要深入学习贯彻  ▁demonstrators ▁Hai  ▁WORK 奇效  悠久的  超负荷  打开方式 仅是 液态  最终被  ская ▁Mao  Margaret Bs  只能选择  真的可以  ▁Damascus 金昌  人民满意  有很多种  ▁fintech 徽章 herst  ▁​​  全身的  比起来 ifth 冲刷 只限! 喷射!  加班工资"  第二部分"  ▁mundane# ào# 紫荆$  全家福$  约合人民币%  Forward%  饮酒后驾驶机动车&  ▁unidentified& 伤势' 白日' 英姿(  去感受(  一般而言)  和旅游局) terms*  的犯罪事实* 孤寡+ 残缺+ 烈火, 竞相,  以个人-  医师协会- woods.  的新陈代谢.  ▁Aussie/  ▁Raptors/  健康申报0  ▁Dahl0  真正意义上的1  Cambridge1 MN2 ▁rhe2  今日推荐3  ▁Construct3 远在4  也没关系4  国家二级5  贵人运5  提醒自己6  ▁Boots6  ▁Macau7  ▁Wilder7  ▁Sculpture8  Established8  ▁creatively9 招来9 溶于:  怎么也:  拨打电话;  比较严重; 人因< 八仙< 学类=  在短短=  眉山市> ▁Ket>  发展历程? ция? ncia@ ▁MAT@  columnsA 怀瑾A 总长B  冰激凌B  委副书记C banksC 比作D  ▁ingårD  websiteE 县乡E  的迹象F  健康证明F  ▁TremG  PartnerG  orderingH '”H 封贺I  冰箱里I  听故事J  生源地J  下载安装K  处副处长K  ▁CaldwellL ttoL  有些时候M  commentsM 专长N 为官N 久安O 惨了O 挫败P  和义务P  摘下口罩Q quakeQ  ▁ArbitrationR  ▁TransparencyR  产品经理S  人生没有S  近几年来T  atologyT  ▁GeoffreyU  ▁occupantsU 流光V  相信你V  的时候要W  ▁BrideW  ▁estimationX 横穿X declY  项目建成后Y  ▁LaboratoriesZ 孤岛Z 界首[ 躯干[ oused\  ▁Brewers\ 多巴] 拌料]  往常一样^ itoes^ 太上_ 李焕_ 警民` 说走` 靖江a  性激素a  ▁Foundingb  ▁keyboardsb 宋江c telec  ▁scamsd dupd Todde  ▁Kensingtone 若要f  高温津贴f  ▁Stokeg 汇流g  全部为h  如何快速h  工艺美术i 长着i ▁fsj  总收入j  真实不虚k elopek  ▁Paidl  ▁helperl 生食m  有机物m  最好的办法n  ▁bladdern  ▁leakingo 悬空o 投喂p 日早p 有型q 检票q 羊绒r  在线网r  你还可以s  玉米价格s  应聘人员的t 水箱t 火上u  服务大厅u  衡水中学v  ▁scrollingv 涌动w  快车道w 纷争x  犹太人x  自治州y  及其制品y  目前没有z 格言z 症的{ 草鱼{ zhan|  泪流满面|  ▁Geological}  接触者}  不得而知~  或者直接~  ▁Metall  ▁dynamically  ▁manipulated  这期间 何在  深入交流  ▁Sloan 北洋 换挡 等为  位置的  的高校  ▁collisions 大牛  中华文明  排查工作  ▁pioneered  建筑和  第七次  倾家荡产 unged  ▁RELE  县人民检察院  ▁separates 淮滨 说事  网络空间 excel  ▁Grad  理财第一步  ▁misery  Democratic  整改工作  知行合一  ▁Spike 影帝 谈笑  戳下方  碎片化  国家公园  ▁wisely  ▁Deloitte 轮岗  ▁minimalist 柔美  为帮助  支队长  ▁abst 股神 话下 iduc uned  ▁pund 万次  买一送一  没有人能  ▁Lift  timeout  ▁disproportionately 怒了  和她的  夯实基础  ▁Lilly  ▁fantas MPA  试试看 改报 CLE  发布时间  才能得到  ▁жи  ▁levy  ▁Reyes  Russell  ▁Corinth  ▁pathology 边际 ahar near  的毕业生 Tyler  ▁REVIEW % 这样您就可以免费收到 唯物 岛内  这一系列  ▁blaze  ▁Rohing 曲艺 老生 при  和身体  黑天鹅  八十年代  天气过程 Phase  工作和生活 玄机  由内而 ▁Rae  ▁Docker 狭隘 驻点 ("%  制造有限公司 很受 资阳  二手房均价 Expect  ▁Hospice 时钟  integration  双胜利  自然地 ▁SAF  开开心心 rivile  schemas 小果  及专业 ▁DAM  销售经理  ▁pint  ▁knots  并不算  天气情况  来表达  第四条  政务服务平台  ▁Humans  ▁internships 颇多  写作文  送温暖  项目是•  ▁refunds•  的实施Õ 　//Õ Boundĕ  再上新台阶ĕ  ▁occupiesŕ 田村ŕ loseƕ Samuelƕ  ▁outlawǕ  哈尔滨工业大学Ǖ  ▁perennialȕ 高邮ȕ  ▁connɕ testedɕ  ▁aromaʕ  ▁Hundredsʕ 从无˕ 只吃˕ 我更̕ 所要̕ 霸占͕  工程系͕ 乐福Ε ushimaΕ  市公安局交警支队ϕ " 从玛拉沁信息网开始ϕ 何炅Е  显而易见Е  来找我ѕ  领事馆ѕ Iraqҕ  ▁Moralesҕ  第三人ӕ  ▁Zeldaӕ  ▁Optionalԕ  ▁counselorsԕ 传道Օ  运输车Օ inky֕  ▁auditorium֕ 实情ו 曼谷ו  一小区ؕ  将全部ؕ  精湛的ٕ 无耻ٕ slowڕ  每天上午ڕ  向社会公开ە  ▁planetaryە Excܕ  和女儿ܕ  在前面ݕ  束手无ݕ  ▁doomޕ 头颅ޕ  在设计ߕ  问题了ߕ ocha jandro  现就有关事项  ▁geological 难忍  觉得你 Truth  ▁cabins 细数  个人微信 Andrea NW 肉片 Cars  他人财物  Bitcoin 天池 放送  腾腾的  让你们  ▁Fresno  ▁leased  ▁playback 再发  岁人群  ▁loung 更美  的心脏 Shift  业主们  关键期  种情况  ▁Schwe  ▁Handle  ▁unseen 劳力 大金 太累  开设赌场  都是一样的  ▁clocks 微弱 象山  地藏菩萨  尼日利亚  ▁rang  ▁blinds 借呗 拔牙  本通知  三世诸佛  幸福快乐 mostly TOR ▁Kob  第一部分  ▁satir RU 得力  北京市朝阳区  ▁Wilhelm Wu  别的花  双方在  国产车  大型企业  Detroit  compile 抵消 良方 长河  激活码  知足常乐 igers  ▁Liberals  刻骨铭  就能轻松  教育改革  地区生产总值  uctions  ▁Underwood 折后 被骂  黄石市  表现形式  这个机会  有什么样的 干支  放暑假  的痕迹  认认真  新媒体平台  ▁Profiles  IOException  ▁Journalist 密鼓  夏日的 复诊 APA  进去了 道远  类物质  全国包邮  公共利益 uggle  ▁adore  Teaching  ▁intrinsic 新秀 驻扎  模拟题  火灾隐患  contents 三倍 线段  因果关系 orden  anticipated 御景 手稿  余额宝  和价值  三分之二  专业老师  可疑症状  九年一贯制  内容未经核实 幼虫  颗粒物  ▁Vish 字词!  她还是!  栩栩如"  冲洗干净"  而且还能# Barry#  ▁Signed$ Scan$  事故的发生%  duration%  ▁Infectious&  ▁directories& 嘉陵'  交强险'  十二届(  笔记本电脑( 武将) ▁Kul)  ▁Attempt* 忧心*  被淘汰+  一起成长+  热烈祝贺, 变电, 提拉- 田家- 通宝. 阿坝. nda/  四个月/  花青素0  但是随着0  ▁woes1  ▁auctions1 退学2  赚不到2  争取到了3  ▁woodland3 十位4 讯问4  值得的5  如何通过5  ▁formatting6  ▁principals6  看完后7 ALTH7 Barn8 Hour8  无论何时9  ▁Howe9 套现: 火险:  锣密鼓; udding; 摆件< iek<  地上的=  相信在= atom>  ▁Reese>  ▁Procedures? 开战? 滚烫@  一定要及时@  更多的选择A  ▁HostingA  ▁KnoxvilleB EIB 本溪C 紫薯C  乡村建设D  立即停止D  孝感市E  设计时速E  和部分F ▁WAYF  一起做G  副科长G  点进行H  知道吗H rissI 和白I ▁VWJ  了一层J  报名地址K  ▁HavanaK  ▁obstructionL 鹤岗L  可领取M  伸着懒腰M  ▁physiologicalN 丽娜N 气壮O  世界中O  活动结束后P 更迭P  要求其Q  也是相当Q 封路R 汉朝R 退让S ▁psS apeakeT 时为T 树苗U beltU  ▁funnV ArgentV  ▁crowdfundingW ▁ČW  ▁HochX 心惊X ístY  最高法院Y  再继续Z  爆发了Z  一般都会[  考研英语[  ▁fries\  ▁valleys\ " 中央军委主席习近平]  ▁devastation]  无人问^  幸运数字^  最新内容了_  idences_  ▁Amelia` üh` 应于a 诵经a  圆明园b  的话说b ractorc  ▁similarityc 不露d 松花d 较慢e  吃完饭e  开考比例f reaticf ▁VTg feetg  ▁repoh Deputyh 也大i 同享i 韩剧j  检查时j  的性能k ▁cssk 邮储l cdotl ▁Raom Metalm  ▁niecen  ▁Troublen AQo 封禁o 肺腑p  皮肤科p  报名网址q 配额q $$\r  应该说r  明显增加s 出库s maidt  ▁peelt  ▁undercoveru  ▁Predictionsu 命门v 壮族v  ▁reptw  ▁mysqlw  rictionsx  基本是x  对一个y  ▁Jaysy  ▁Chattz 东边z  持久的{  ▁Lite{  ▁tasted| yh|  常规的}  赚大钱}  工程进度~  两个一百年~  朋友说的话  ▁Hayden  Introducing  ▁condominium  ▁indispensable 扬言  日用品  兜兜转转  效果显著  文明城市创建  iterator  WriteLine 车贷  临时性  是人的 嘉善 栽植  而中国  论诗书  ▁Samar  你有没有想过 热锅 莱克  却是一 ibel  ▁scars  ▁Audrey  盛开的  variable 果敢 正茂 苦读  发现该 Faith  为了让大家  ▁Wichita  ▁layouts 入馆 kid 圆滑 鼎盛 ▁Wak  ▁Porto  caption  ▁hypertension 涡阳 环环 过道 psi  措手不及 Sense  Products 王明  每天在  毕节市 URES  持权属证明 凡尔 摇身 米面 iert  上不封顶  新闻爆料 hetti  每一个孩子  ▁Loved  ▁discrep  ▁fortress 分在 月刊  生活区  的知识点  ▁sterling 九寨  ▁nanop  事业单位考试 不认  漫论诗书  illation zą 卡路 毕生 父辈  对患者  的美感  未开放时 Joshua 四时 尚且  汽车品牌  ▁occupations 事前 公孙 降准  可采取  大人的 ▁MGM  ▁Cells 不择 和面 林的  大兴安岭  ▁glove  及服务  命中率  阿斯利康 两省 长篇  科目三  睡觉的时候  answered  Recommended 我刚  出售价格  Powered  ▁Rooney Coff  洁白的花瓣  会非常  很多都是 书评 窝里  兴趣班  在实践中  科技工作者  ▁balloons  的好成绩 isset  ▁mindfulness 半挂 走低 鸭肉  无误后  ▁Feet chairs–  cluster–  ▁FrozenÖ  ▁convergenceÖ  沐浴露Ė  活动详情Ė umnusŖ  ▁fellowsŖ 乙方Ɩ  大约在Ɩ  ▁Cromǖ  ▁Seafǖ  教育主管部门Ȗ  ▁defeatsȖ  Documentsɖ  ▁Websitesɖ  大儿子ʖ  脖子上ʖ  稳中求进˖  ▁locksmith˖  ▁discriminatory̖ 会为̖ 踏步͖ 靳东͖  对全国Ζ AvoidΖ cancerϖ ermaidϖ  ▁achievesЖ 长长Ж  情有独钟і  ▁Roryі  ▁SnakeҖ 圈里Җ  自己的生命Ӗ  侵犯到您的权益Ӗ  以涉嫌Ԗ  灰色的Ԗ 阻挠Ֆ  为我国Ֆ ▁Joa֖ onders֖  ▁hackerז  ViewByIdז  ▁synagogueؖ  心理上ؖ  较上年ٖ  水涨船高ٖ  ▁pantryږ  ▁satireږ  ▁portraysۖ 封号ۖ  有你的ܖ  党务工作者ܖ  Vietnamݖ  ▁carvingݖ  王俊凯ޖ IOUSޖ  ▁Ryderߖ  一卡通ߖ  为落实 prints  Producer  ▁Interpret 相扣 香山 ▁SX  转眼间  图文讲解  在微博上  ▁Receives  ▁Happiness  岳云鹏  全国第一 包庇 落了 闲情  全校师生  加拿大的  紧锣密鼓  ▁Concern 器物 对公 开路 牵连 ▁fry  千篇一律  首都北京 ONDON 前述 爆表  总感觉  目前该  瓜果佐酒  省吃俭用  ▁Cust vester  ▁evapor ionic  Studies 和合 多边  每学期  ▁Kant  Important  ▁Headlines  的另一  地展现它的  Wisconsin 而活 nah  公安工作  持续加大  ▁Gerry  ▁mushroom 双峰 月色  这家店  更改了  购买此 Einzel 常被 祁连  日报告  ▁Peggy  ▁Profit 争抢 刮目 采血  历久弥  和规范  阳光城 ▁DSL  在职人员  朋友圈里  是有意思的事  一条龙  用一位作家  ▁fauna  醉里挑灯看花  ▁commissioning 新作 诅咒  努力成为  招聘考试  ▁shortcomings 有才 静的  等待羞涩  等待羞涩的美人  ▁dragging 冰块 家店 歙县  的饮食  向外舒展  向公安机关  在在千呼万唤中 " 一点一点地展现它的 咖喱  于未然  财政资金  未开放时绛 ( !一点一点地展现它的秀色 敬仰 瓶盖  高考志愿填报  ▁adversity % 用一位作家朋友说的话  ▁subsection  蜂王胎  根据个人 ieler  ▁odor  datetime  Constants conv ▁PLC  你在看  大喇叭  另外一种  国家承认  评分标准 hammer  aduation 书名  必然是 两百 烧香 隋唐  发电机!  戏剧学院!  ▁Bankruptcy" 园路" 推到# 时将# 颐和$  全市上下$  故意杀人罪% waters%  的文件&  有色金属& liber'  ▁casts' 听后( 方阵( 自住) 跑的) ▁QR* arro* ▁Chu+  ▁Scan+  日放假调休,  ▁desks,  Meeting-  ▁inclus-  financial. 开售. 而未/ lioma/  Podcast0  进步的0  护理人员1  ▁rewrite1 即用2 母语2 首秀3  可达到3  更全面4  基本没有4  无缘无故5  ▁Serena5 兼治6 Sara6 Albert7  ovelovel7  ▁Fairfax8 芒种8  直接从9  ▁McCoy9  点击下方空白区域:  ▁variability: 丫头; 墙角; 阿耨< nard< 请来= Esc=  ▁unveil>  Employee>  ▁Prosecutor? 私域?  构建了@  入党积极分子@  知识星球A  eniableA  ▁hallwayB  同一天B 姗姗C 它与C 快船D 蓝田D ymeE  把事情E maresF  opathicF  ▁antitrustG wjG 今起H 几招H  富含维生素I  制片人I  及部分J  慰问金J  劳斯莱斯K 代收K 等新L  精品课程L gardeM  ▁rumorM 一职N 个市N 前车O 是啥O TRYP  给家里P  投资者的Q  ▁baldQ 可真R 大碍R 罗永S  大闸蟹S  进入公众号T sexualT 惨淡U 火种U EddV  一年前V ruffW  就是一场W 摆渡X  反映的X  的一所Y  监管责任Y  ▁pledgesZ  ▁circulatedZ 低洼[ 踩着[ 辛亥\ 馆内\  多少分] titled] 善事^ 归家^  研究与_  中央公园_  项目地址`  开放的样子`  ▁Nikkia  Atlantaa  ▁Instructorb 崭新b 故名c evoc  除了在d Fixedd 太冲e  型糖尿病e  心血管病f  engeancef  ▁ethanolg 交感g 吐鲁h  无名指h  监察委员会i  esthetici  ▁trumpetj  ▁pitchersj JXk  相呼应k  ▁laval  周岁及以下l  chargingm  ▁disseminationm 赛的n  公共管理n  ▁Philipso 一语o 雨的p  动员会p  Walkingq  ▁paragraphsq 古韵r 山南r  五颜六s  因材施教s duringt 好货t 甩锅u 补漏u ($_v RAMv  变异病毒w  安全生产月w  房地产行业x  就真的x  我以前y  ▁Kievy  ▁mismz 大宝z 脆皮{ 附子{  电影的|  建立完善|  研制的}  透明度}  旗帜鲜明~  ▁maneuver~ 放量  ▁Jerseys 体悟 寻味  了一会 Flags allah 增资  越容易 ▁jan Claim 个数 ▁VID  应收账款 糖浆  ▁Friendship 月老 绵羊  视频截图 鱼片 ahead  ▁Arcade  ▁Titanic  使领馆  各省级  另外还有  ▁reflex  ▁assaults 可加 如厕 姜末  三角洲  为考生  能够做到 之巅 娴熟 水到  ▁Ducks  ▁neatly  但它开放的样子  ▁brilliance  一想到 trace  ▁bart  ▁worm  优质教育资源  ▁anchored 后厨 睁眼 空姐 beg  虽然柔情似水  infandel  pressure  ▁diagnostics Wy ANGE  ▁Turks 电诈  之功效  大庆市  有一句话  村委会主任 ▁QC  大家长 ▁Clo  恢复营业  如果有一天 merged 壮阳 还债 тов  但是当  发消息  确认为  都是你  ▁Bust  ▁MacDonald 八宝 妙用  水环境  莫名的  ▁serviced  ▁specialties 云峰  我想要的  长江经济带  ▁vacancy  ▁amounted 五色  一两个  在讲话  是很有  送上了 rers  一点点向外舒展  如美人伸着懒腰  operator  ▁оп Mask  的关键时刻  专业技术职务  ▁insisting  ▁reminders indi  重大意义  ▁Gibbs  ▁Guyana  ▁Mosque  Distance 拳击  去追求她  breaker  ▁UNICEF  ▁Resilience 单笔 此番 维也  感党恩  项城都市网  过的人  ▁Pend  ▁Ninth  ▁stakeholder :) 字里 武陵 炮制  必要性  犯罪记录  ▁Bhutan  ▁Ginger  ▁edible 脸红  万千瓦—  其实我们—  没有关系× ( !港澳台地区通报确诊病例× 教培ė 跑出ė  archingŗ  ▁Includeŗ  张国荣Ɨ  最关心的Ɨ  ▁DortǗ  死记硬背Ǘ  是两个ȗ  社保费ȗ  ▁saladsɗ 魄力ɗ 魏晋ʗ  的待遇ʗ  防疫一线˗  ▁spun˗  一般纳税人̗ 取自̗ 和谁͗ 顽瘴͗ KidΗ ▁HouΗ  激励机制ϗ  分享自己的ϗ  ▁AbujaЗ 4 -累计收到港澳台地区通报确诊病例З  却不知ї  nutritionї  ▁profilingҗ 转而җ  吸引人ӗ  锁定了ӗ  赶往现场ԗ  多罗三藐三ԗ  如果你想追՗  ▁Boise՗  ▁sharpen֗ 原石֗ 多到ח 就先ח 笔顺ؗ  接报后ؗ cakesٗ  ▁restraintٗ 极光ڗ  多元的ڗ  意志力ۗ  揭示了ۗ  关节疼痛ܗ 香草ܗ rekݗ  的眼光ݗ ▁Gugޗ  石油化工ޗ  中华医学会ߗ cationߗ 氢气 高科 青天  和指导  其乐融融  ▁sunrise  报考点  Convert  ▁Ventura  ▁fashioned  ▁Absolutely  ▁commentators 发际 四千 苦尽  中东部  批准后  中国内地 单于 渗入  Historic  ▁applause  ▁trophies NUM  的天然  从天而降  ▁unlocked  ▁safeguards 水口 wik  和应用  百香果  传统工艺  protocol 分心  歌唱家  系统等 urger  ▁Centennial 同质 国税 大任  geoning  ▁exposes  细菌性  中的作用  给我们带来  ▁Romero 交底 酸胀  店里的  有特殊  驾驶的  笔试内容 Timer Carbon  ▁Bulgarian Ja دى 姐夫  工作的同时 护林  成功地  的举动  为消费者  ▁Becky  ▁recol  ▁infused 超度  小窍门  最合适  ▁faç  违法事实  ▁Pref 诺言  ▁customary 江景  少走弯路  ▁Harriet  ▁Shapiro 十有  别错过  在这片 ▁Sap  推荐指数  ▁tipped 万种 之梦 双喜 海螺  流水线  我有一个 Shape 并可 格斗 满族  ▁juxt  ▁dwind 缺额  拒不配合 暗暗  云课堂  畜牧业 LIST  我们都要  能够达到  行政学院  Residents  ▁triggering Cancel  ▁Dustin 鱼缸  的调查  这一条  perture  ▁graphs  EventListener dos ssl  ▁toggle  ▁Steering 地暖 打乱  是这么 ▁GHz  特点就是  ▁Buchanan  的自我 保家 弱化 西华!  中成长!  自身防护" 厚积"  所不及的气度#  失信被执行人名单# ▁fmt$ ▁nos$  ▁Poems% 从其% 可看& 防指&  生活补贴'  象小家碧玉'  ▁Coaches( 刀片(  顺利的)  心中有数)  ▁cele*  紫色的花苞*  它有着别的花+  ▁Accordingly+  ▁Intelligent, + $它有着别的花所不及的气度,  和党员- Army- ▁HOW.  要求各地.  ▁aides/ % 紫色的花苞象小家碧玉/ 散装0 牧羊0  ▁Adds1  ▁Bauer1  ▁accumulate2 4 -未开放时绛紫色的花苞象小家碧玉2 超模3  全体学生3  胡思乱想4 整日4  合格线5  日早晨5 andes6 Depend6  Already7 山火7 陇西8  上下学8  均来自9 Pear9  ▁Cors:  ▁Toxic:  Defense; ▁ka;  在中医<  造成严重<  ▁grouped= 万变= 降了>  权力的>  实地查看?  和标准?  过山车@  ▁bait@  ▁doorstepA  ▁dispositionA 麦田B  很关键B wxcbC  控股有限公司C '}D 区直D  日宣布E XboxE  教育发展F  通过线上F hemerG  ampiresG  ScotlandH  ▁boatingH  ▁comfortingI 销路I  喜欢看J  时不要J  如何判断K  晶莹剔透K  进一步深化L  ▁CheshireL ▁ACAM  ▁LiverM  保持安全社交距离N 仁心N 农用O  更可怕的是O 大浪P 颍上P  坚持了Q NEXTQ  朝气蓬勃R  电子税务局R  oughtonS  ▁prowessS 按住T 难看T  在哪儿U 起止U  ▁KaplanV  ▁ContentsV 赴美W  marriedW  ▁MondaysX  ▁SheldonX 蛋液Y 首钢Y sdkZ  又不能Z  ▁curric[  并不代表本公众号[  ▁impartial\  ▁BlackBerry\ ге] 大丰] 颐养^  大项目^  等规定_ 短款_  可以到`  雪中送炭`  ▁autobiographya 夜查a 惭愧b  等制度b  简单易c  一定要把c 握着d  退一步d iesele storese  ▁weightedf 吃香f 患得g  的气氛g  道路的h  ▁Westerh  ▁Cartooni gosi  不指定j  应避免j 匀称k 起家k FUNl  的农民l  科技局m  公共空间m 血丝n ▁GNUn  尽心尽力o  ▁warehouseso 华美p 瓦房p  威海市q risisq  进一步明确r  ▁typeofr  ▁broadcasterss  与学校s  和加强t  是导致t Periodu  网络科技有限公司u TWv 别名v 年将w uevaw ▁Cedx  滥用职权x  ▁Dominiony 绝无y  各学院z  可以继续z  ▁Bran{  ▁Taipei{  ▁diagrams|  ▁interpreter| bp} 别离} 闹得~  第一志愿~  ▁Gale  ▁smashed 亩地 更会  为居民  慰问品  体会到了  示范项目  逐渐成为  ▁Turbo  ▁Holden 爽滑  李大钊  椅子上  ▁hobbies  ▁embodied Ze 几遍 盆土 禁渔 风热 APPY rawl  干货满满 Ay 加长  安庆市 ▁cuc  名医护人员  ▁goats  ▁también Lov  ▁ensuing 常言 而止 钱款  到如今  嵌入式  视频的  运用了  充满信心 乳液  和林格尔  ▁Olsen  ▁electromagnetic  ▁abbrev  ▁wildfire  ▁customizable 新宁 辣妈  加盖公章  ▁Gins  ▁Moran " 教育部留学服务中心 荒野  为你们 Sens  ▁plausible  一步到位  ▁Tube 功名 客场  几百万 CHAR  共青团员  那么这个  长江大桥  除了这些  ▁Photographs truth  这样的问题 一概 虽好 iage  一个女孩的话 师市 破旧  乱七八糟  专家组成员 ’‘ 前移 江路  ▁ва  是否是  涌泉穴  温暖和  错别字  另一个人  平顶山市  江北新区  食品添加剂  阿耨多罗三藐三 执意 缘起  局部地区  ▁wont 特务 转租  去参加  navList ( !如果你想追一个女孩的话 有据  有气质  温开水  确认后 Hell  执行董事  第十二届  长按保存 quila  再放入  好评如潮 年入 疏解  一个脚印  今年年初  ▁raff  教育类  更好看 ISM hift iale Va 从宽 疑点 筹资 蠢蠢  结束时间 Symbol 洞见  ▁Minds 关了 尝鲜  发自内心的  ▁Proposal˜ isol˜  公司名称Ø  ▁reliablyØ  也不愿Ę  ▁SylviaĘ 补税Ř  和广大Ř ▁PonƘ  CollaborƘ  ▁apprenticeshipǘ YPǘ 吹牛Ș 外力Ș 扶梯ɘ  ▁oligɘ AMCʘ Insertʘ  ▁simulated˘  ▁discontinued˘ 湿滑̘ GLO̘  呈现在͘  Society͘ 武部Θ  担任过Θ  的中药Ϙ  对他们的Ϙ  蚂蚁集团И  让我们在И  Jeffreyј  ▁fluctuationsј 食神Ҙ MurrayҘ  ▁measlesӘ  ▁pollutantsӘ  其它地区Ԙ  ▁DoomԘ  ▁Rubio՘  Interesting՘ 再由֘ 本意֘ ▁'#ט ▁YAט  患者中ؘ  金榜题名ؘ  ▁quartz٘  ▁richness٘  ▁Huffingtonژ  ▁illuminatedژ 六点ۘ  示范园ۘ  自己都ܘ  政治素质ܘ  主要表现为ݘ  释迦牟尼佛ݘ DVޘ 忠心ޘ 肺经ߘ 落得ߘ  非机动车道  而无法 ()))  养成良好 国力 收储 IENT  可以免费  方案设计  vertical  ▁translating  解难题 不就 只说  全市人民 京师 赤诚  表态发言  ▁outlining 万卷  中含有的  或单位  及时清理  ▁Bands " 不要盲目的去追求她 抗衡 Export 一空  无异常 ▁Bau  快人一步  顽瘴痼疾  的重要论述  ▁Leaving 国乒 摊贩  在手上  陆续续 夺目  的同事 主创 是中  ▁granddaughter 几度 oran  眼花缭乱 刷了 园丁 梅西 难了  三阴交  团队在  很好看  接受过  大力宣传  ▁Subaru  教研员  Century  ▁improv 租车  应该就是  Neither  ▁Miracle  ▁downtime 岁数 清真  交给了 prev  哪有什么  微淼商学院  我们致力于  ▁Nicholson  品牌形象  岗位人员  招生章程  ▁Serious  ▁intensely  室内的 Refer  ▁stray  ▁melanch  晒出了 ▁mog uterte  即可免费领取 蝉联 Fel quoise  ▁Harmon 发过  成功抓获 ingles 清正 禀赋 钟声 马仕 误入 贴纸  报名和  ▁Brah 懒散  日公布 tone 而同 花岗  十八条 Yang  安居乐业  ▁breastfeeding 不破 延吉 都把 McC  对人类  还有各种  ▁Kyiv  ▁nested  ▁enclosure 上上 用功  占地面积约  Resolver  onential 宝地  锐意进取  ▁bottled  regulation 共育 沪深  思想道德  ▁dams!  可靠性或完整性!  ▁Michelin" ▁{};"  ▁Sapp#  occupied#  ▁Alibaba$  一张图片$  宏观经济%  ▁Fully% 博文& Fix&  可以与'  经过近'  ▁Dive(  ▁uptake( 鸣笛)  出行前)  ▁Packages* 利辛* 装潢+  创立了+  此前报道,  ▁pastry,  ▁caramel-  ▁seminal- ща. 尘世.  海贼王/  到自己的/  研究院院长0  ▁defer0  ▁popcorn1  ▁scandals1 养性2 反华2 试过3  形势严峻3  经济条件4  Harvard4  orescent5  那就要5  少之又少6  ▁BASIS6  Validation7  ▁familiarity7 zyp8  不充分8  可以看9  已经完全9 qa: 横行: 黄磊;  等食物;  门口的<  陆陆续续< 眼了= ▁DU=  的最终>  ▁semifinal> 后因?  高密度?  ▁Garner@  ▁Blogger@ 删掉A 弹琴A  和数据B  ▁SchnB ocytesC 滑县C 铁板D fffD  组成了E  所以这个E imicroF 发麻F 讲道G MutG  擦肩而过H 且未H 国美I xslI  最新通报J  ▁fuseJ  climateK  不开的K  即使你L  二线城市L  ▁WelchM  ▁cartridgeM 是没N  了一样N  蓄电池O  头等大事O isoftP  ▁lendsP  ▁compromisingQ hcQ  了一篇R  是进入R icomS  上海浦东S 炫富T HOTT  展览会U  转小火U 制胜V  着力点V  光盘行动W  ▁biotechnologyW 南大X  颜值高X  结直肠癌Y CollegY 典籍Z 风霜Z 魔兽[  被确诊[  ▁Para\  ▁unfolding\  落地见效]  课程设置] "—^  半决赛^  和问题_ usch_ ▁sap`  点击关注` ylenea  不负有心人a  正面的b  第一道b dochc  会议中心c  慈善总会d  这样一种d  Discoverye 余件e 能动f  ▁restrictingf  ▁legalizationg 不愈g 与子h 捷报h  号公告i  相亲网i  茶文化j  小数点后j šík 放款k 计分l  参加全国l 敷设m mRNAm  ▁vainn  ▁forestryn 患失o  将进入o congp  ▁Dresp  ▁creekq  definitionq 涂装r 湖里r 附表s  医院和s ildet  不文明行为t  也可以选择u  ▁shortenedu  商用车v IVERv  幼小衔接w  演讲比赛w  第一财经x  都会出现x mathbfy  ▁Rockiesy 浴缸z 荧幕z 高挑{  ▁cockpit{  一条路|  全封闭|  发车时间}  Lincoln} 竞品~ Luck~ ▁Dir 一局 葱段  Condition  ▁dependable  ▁constituted  ▁perseverance  在感情中  ▁precaution  事业心 Sadly  ▁scrutin 新都 童书  个小时的  数据来看 aille  第三节  ▁javascript 就跑 路费  可咨询  走得更  ▁outperform 东关  为了保障  居民医保 哭声 禹州  有益于  Validator  ▁Philanthrop :@"  为保护  完成情况  ▁Grac  ▁ferv  ▁Kitty 上乘  很辛苦 ▁Fay  如果你在  民族文化  ▁Sears 荆楚 过热 Aid araj って 下地 光子  ▁Pour 带电 荤素 麦冬  不能被  回想起  福布斯 trim 免检 enf  万粉丝  要加快  犯罪活动  重要提示 Terry hadoop 秦淮 MET  持续了  毫不放松  ▁Mellon  就业率  用这个  金刚经  在一起了  才是真正  持续发展 ignor  ▁mound  ▁confronting 开怀  积极与 Span  他们也会  ▁unatt  ▁weave 字眼 开馆 ▁LT  下坡路  都处于 ronym  ▁discrete 刺猬 ENV  以减少  特别大  昔阳网  全市各级  血红蛋白 PPORT  报到证 pole risy  人才市场  ▁PLAY  时使用  大多都是  ▁Maui 停停 北伐 华区 寻亲 心性 正餐  ▁frontman  ▁footballer 照看  危害性  年同期 bart  不顾一切  Pointer  Biography  渭南市 croft 担子 揭发 费县  今日话题  processing  ▁Consultants 探头 气功™  纪律处分条例™ 根深Ù ▁CGIÙ  免责申明ę  对于这些ę  ▁Emeraldř  ▁backlogř  第一波ƙ  香港的ƙ  对中国的Ǚ  沿海地区Ǚ  ▁Penguinsș  ▁Comparativeș 吴川ə  日期为ə coatʙ  联合执法ʙ  ▁setback˙  ▁nutritious˙  快递公司̙ Metro̙ 光临͙ 招教͙ 荣立Ι  孩子成长Ι  茁壮成长ϙ  ▁blurϙ  重要讲话和Й  健康生活方式Й 快到љ  中年人љ 镇西ҙ  下发了ҙ  数量和ә iddyә  主题班会ԙ سىԙ 却让ՙ  优势互补ՙ acier֙ 挨家֙ 祭拜י  想到的י rombؙ 不贵ؙ 事例ٙ  分别从ٙ  系统进行ڙ Modalڙ  也就是我们ۙ  ▁weedsۙ 混沌ܙ  频繁的ܙ ▁Riyݙ  五花八门ݙ  突然出现ޙ ▁"%ޙ  接二连ߙ  的分泌ߙ enthal 喝着  帮扶工作  难点问题  ▁Tracks 内线  的重量 Gram  万物复苏  优先录取  娱乐活动  广大市民朋友  ▁Brings  ▁magnesium 小丑  三方面  个省市  挂号费 Pray  ▁volt  佩戴好口罩  所吸引  考试费 Fish  ▁altering  的香味  市委党校  ▁interpreting 练琴 kou cctv ▁gor  网友们的  ▁Guam  安全工程师  ▁Damage 载货  的不利 ▁MOT  年度报告 Scene  助力乡村振兴 非同 oct  企业及  等特色  并提供相关  和高质量发展  sembles  ▁vacations 发痒 ▁LS  不包含  和鼓励  集中开展  ▁Levels  ▁Covenant >},  天河区 半程  适当地  ▁orch  ▁Vijay  面向社会公开  ▁Anthem  armaceutical 信服 沃土  值班主任  ▁Punch 　¥ 举重 做优 罗列 MEA  阿拉斯 ▁Phi Afghan  ▁Trace  寒假期间  ▁valves 体感  等到了  ▁extingu  得心应手 yellow  ▁Strauss 代之 影音  ▁Toni  ▁Venezuelan 开光 霞浦 alon  ▁ambigu  Statistics 挚爱  强奸罪  精神卫生  是什么原因  ▁Palin  ▁piles  ▁Maximum 算盘  州政府  讲话精神 Joint  ▁Foley  ▁cohesive  奋不顾  维护好 guns 包浆 lique  进一步推动  ▁simulate  responsive  ▁Initiatives 新锐  行为习惯 幻觉  朋友说 aysia 储量 大佛 承重 此起  ▁reactive 的法 院线 高淳!  奋斗者! fp" 帆船"  有时会#  结婚的#  国庆中秋$  ▁Jessie$ 早衰%  今日份%  观音菩萨&  ▁ceramics&  管理及'  ▁ranc'  大数据分析(  ▁undes( 在先) ▁hid)  ▁Seen* 两台* 金庸+  真心实+  金普新区, 两节, 地被-  成果转化-  ▁Seymour.  ▁coronary. 精工/ utm/  ▁importing0 逝去0 长汀1  攻坚行动1 Occup2  monitor2 三科3  失去的3 Vers4  ▁charcoal4 内设5 西乡5  生态农业6  也越来越多6  统计数据显示7  ▁Fulton7 城郊8 正风8 清欢9  我和我的祖国9 所涉: UFO:  患得患失;  违规收受;  ▁Rang< frames<  ▁Clarkson= 克隆= 拒收> 羊城>  束手无策?  ▁flute?  otherap@ 丈量@ 惠来A  却还是A  梦寐以求B  重点工程B 敢说C  受伤的C ▁NahD biasedD  ▁mergersE  的人性E  请致电F swingF  ▁EllieG  ▁sternG  ▁dancedH 蹲守H  疫情结束后I  目前还没有I  ▁bumpsJ  ▁homosexualityJ 前一K 泰顺K  会员卡L  ▁veggiesL  ▁culminationM 长垣M idadesN  ▁DartmouthN PuO  ▁RogueO  PremiumP = 6新型冠状病毒感染的肺炎疫情防控指挥部P anseQ  微信好友Q  出示健康码R  ▁FingerR  ▁lettuceS  ▁PhysicianS  水煎服T  ……”“T  ▁NanoU  ▁dividesU  角质层V  就业人员V  相互转告W  undefinedW 发心X 熟睡X  思想工作Y  ▁LaosY alternZ  将开展Z buff[  研究所所长[  Hundreds\ 差了\ 碎石]  正北方]  校园安全^ Rules^  ▁confession_ 图说_ 山间` 贺兰`  系统自动a  rollmenta  ▁workspaceb 谁更b 追查c sarc  建成了d  是整个d  ▁Dantee  ▁Objectse 神阙f  磁共振f  新闻频道g 潺潺g 道光h esih  人体吸收i iseasei 往里j 酒杯j 面世k  持卡人k  种类的l  并非如此l  ▁instalm  一辆车m  全运会n  等系列n labelso  ▁eliteso 后不p 实证p pasq ▁PPq  熔喷布r 论点r  不可及s  ▁downhills 哭笑t 穿孔t liquu  为人处事u  清华北大v actualv  Assemblyw 大龄w 隔三x ▁ccx  临时工y  显示器y zionez ▁Nirz Agric{ 石刻{  ▁Erit|  等方式进行|  avourite} 那张}  的诞生~ htub~  经济补偿 xg 任怨 sup  赞助商  转正后 Belg ▁Woo 体位 чен ▁sb  维也纳  ▁traff 几大 康康  みらい  苏维埃  无微不至  正式成立  ▁Vand  ▁Himself  ▁departing 头昏 电热  少有的  本人或  了一口气  ▁mystical 相拥  开拓者 ▁NEC 敛财 无序  农业部  的证明  公诉机关  问卷调查 female 去吧 山野 BMI  晚自习  应运而生 买药 彩绘 耒阳  并组织  搭配的  防控物资 acted  Favorite ▁tmp avers  iminary  ▁simmer  ▁clarification 一的 体味 率队  提醒您  新知识  管齐下 igon  省财政厅 Usage  自己的青春  ▁bowel 情结 称重 NAL ▁Fou  ▁unfolds MEM  腹泻等  和有关 olland  的时候一定要 涟水  早隔离  Melissa 多部 心若 摔跤 邓超 estial 男儿  但后来  心动的  记者会上  ▁bestseller 厚非 埃塞 扶正 cash  互联网上  什么意思 啥的  两个儿子 偏多 和三 善款  教育是  的好运  照顾孩子  中国人民志愿军  ▁conglomer 书里 告终 铁建  laughter 坐镇 甲醇  ▁ба  年后的  投资和 EEEE  ▁Qaeda  ▁longing 再累  纸尿裤  经了解  ▁Cout 决明 输赢  Holiday  destroy  ▁Forestry TSA  设计感 imet ~~~　  ▁Angus  ▁Sahara 拼图  就成为š  我也想š  ▁pulsÚ  ▁SheridanÚ  ▁uprisingĚ 农垦Ě 活法Ś 盈眶Ś 陈文ƚ  管党治ƚ  该团伙ǚ aidaǚ luckȚ  认真履行Ț  ▁Amitɚ  ▁buddingɚ 北开ʚ  对国家ʚ  就有人˚  是指在˚ ▁fir̚  ▁Archie̚  园林绿化͚ 编发͚  经济贸易Κ  ▁BaxterΚ  footballϚ 访客Ϛ ▁PDPК BLISHК 直视њ  理财课њ  说一说Қ 正对Қ  量力而行Ӛ  ▁clausesӚ " 处三年以下有期徒刑Ԛ 她才Ԛ 政纪՚  办公区՚  幸运颜色֚  直接结算֚ Cookieך  ▁Realtך  ▁amazonؚ  ▁remnantsؚ 约瑟ٚ nezٚ  ▁innateښ lintښ ▁girۚ  不断推进ۚ  井然有序ܚ  ▁Shepܚ  电子社保卡ݚ  ▁Clusterݚ 富硒ޚ  在得知ޚ  的产业ߚ estownߚ  ▁Coupe  ▁Tourist  ▁nominate 含泪 Wel  ▁shrine " 阿耨多罗三藐三菩提 之别 他叫 任劳 打上  全部都  更具有  的例子 Rows  ▁Incredible 晓峰  县市区  贡献了  ▁Forums 摊主  不限户籍  ▁prerequ KV 陶渊 erial  ▁estern  parameters 利湿 抚平  出了名的  生产成本  组织协调  ▁Rubin  nachweise  ▁pragmatic  ▁spreadsheet  适用的 ▁HER  是典型的  ▁ambiguous 卸下 善根 根管 警员  割韭菜  水平的提高 政绩 无主  格外的  生活了  高危人群  ▁listens 作祟 可喜 坪镇 快闪 绝招  ▁Impossible ▁FF  元器件  各国的  是错误的  ▁Usage  ▁nightlife 别怕 永泰  衷心感谢 assen  专科及以上学历  ▁dialect  主动出击  公积金贷款  才是最重要的 浮漂  到基层 rieg  健康饮食  有关内容  ▁Lorenzo upt  税务师 不重 做做 吃出  为主导 Bishop  ▁Angola  ensional  给予其  ▁Orchard  ▁braking  ▁Attractions 洗牙 画质 登台 spl  华南理工大学  catching  ▁ordained 掺杂  为前提 贺卡  轻微的 ▁NRA ▁Ш 四层  无论从  ▁mash  ▁Ancest 享福 脸的  作物的 ▁vid  ▁Scream  ▁outspoken 足底 躺赚  当老师  要抓好  考察对象  要注意的是  ▁robbed  ▁Claudia DOM  预约挂号  noticed  因工作  有弹性 ▁AST  塑料制品!  弯道超车!  相互促进"  都能找到" 棕榈# 浮雕#  的福利$  ▁Colony$  ▁Cummings%  似乎是%  红辣椒& Uber&  ▁gravitational' 难耐' zao(  打火机(  文明创建) agency)  ▁contamin* 故地*  ▁facto+  ▁funky+  ▁falsely, 惠安, 茶几- din-  的这种.  ▁caus.  重症医学科/  记忆中/ 踪影0  的垃圾0 ▁Wyn1  ▁Sparks1  Sullivan2  ▁Walters2  ▁resurgence3 菲菲3  往期文章4  ▁Scenes4  ▁Jacqueline5 不羁5 头屑6  依然存在6  ▁fingert7 小溪7 藏书8  不涉及8  提供商9  标准执行9 paying:  ▁actionable: 粟裕;  中记载;  为你提供<  产品销售<  ▁Beckham=  ▁Turning=  今日运势>  ▁pengu>  Contents? 上讲? 宁海@  流程图@  今年春节A  ▁ribsA  ▁toweringB nostB  ▁delveC 多动C 足元D ▁'\D  广泛应用于E  ▁bookstoreE 等均F 雕琢F  仅供学习交流G  ▁TowardsG 一物H 裙摆H  ▁JewelryI  的大海I  ▁pessJ  ▁NicolJ  以及与K  ▁mensK  的精神状态L  以上这些L  性别不限M  科学防控M  ▁pactN 查查N 说有O  所造成的O  记者注意到P 营村P BowQ 克莱Q  互联网公司R  ▁GrizzR  ▁hypnotS  equippedS 五人T 曾与T 股的U 采风U  蠢蠢欲V gradesV  ▁AshtonW  ▁pretendingW 必先X  不重视X ▁PEOY  ▁atrocitiesY 养鱼Z 太阴Z  那你就[ awan[  ▁Wilde\  intensive\  ▁conveyed]  材质的]  房屋租赁^  框架协议^ rl_ 赛中_  不下的`  伊丽莎`  体验馆a  或其它a  项目在b 完形b 礼堂c  ▁Listeningc  压缩机d ivotd  ▁Baine  Principale 无价f 考录f  起床后g  较低的g  Constrah 贪心h ▁Urbi  咨询公司i  生态旅游j  Einzelnachweisej 一桩k 隐约k  往届生l  回家过年l  Dependingm 萌娃m  南安市n  发展与n  涉案人员o  ▁resumeso  ▁ballisticp  ▁referencingp 七台q  威尼斯q  工作业r  红烧肉r agnas  爱情故事s kwargst  ▁poorert  ▁Conservatoryu  全面领导u 亏欠v 沙盘v  群众对w Jazzw ebrax  让我们一起来x  ▁mosaicy 粉红y  一只手z  ▁extravagz Bu{  目击者{  省会城市|  和大家一起|  ▁differentiation} 万张} 叫好~ 爱慕~ 皮皮  从自身  品质和  板块的  东京奥运  ▁Dwight  根本就  给儿子  好的时候 % 违反中央八项规定精神 む道 ▁ADA  一直保持  先后来到  ▁Chak  ▁Vault ње 每课  あれがあなたの [　 酿成 " あれがあなたの未来 导出 将达  吞吐量  これがあなたの 基底 杠杠 等高  都是对  消化功能  难以置信  ▁Token 林芝 uben  事业有成 上缴 内阁 锤子  专家和  就包括  热爱教育事业  ▁unbeaten  下决心  ▁hinted  Broadway 焦煤  的培训 bley  ▁Doha  ▁Blizzard 三县 脱掉 血性  可追溯  和集体 学森  手慢无  数一数  博野拍客  ▁Changed  ▁folklore  就更加  技术在  来确定  差不多的  ▁Palo  ipelago 奉请  龙华区 cale  电子证照 vasion kc 丽娟  孩子能  开花结果  fitting  ▁needles  ▁injections 五湖 ▁Kok 取经 缓存  的玩家 udos  体格检查  开具发票 在用 屈光 称职 Joh  电影中  要围绕  市住建局 affer  ▁Lent  ▁Mines れた 党派 收网 Dor  喝咖啡  有耐心  ▁perpetual  上涨的  总是一  皇帝的  工程总承包  玫瑰面膜膏  Converter 嫌犯 房住 ▁Bis  大部地区 artist  ▁algebra 横空 源码 OTE  在路边 ilus  预扣预缴 onstr  ▁Truman 全盘  脏东西  年及以上  深入基层  ▁calf  社会主义建设  请作者持权属证明 装进 helle  ▁Clos›  ▁morale›  marketingÛ 功法Û 情志ě 紫苏ě  ▁pyramidś 好想ś 就靠ƛ  学习上ƛ  路交汇处Ǜ  你要是Ǜ  固体废物ț 配齐ț ffordɛ  中国科学技术ɛ 月是ʛ  个学科ʛ  和挑战˛ (　),˛  也是十分̛  秋冬季节̛ 长线͛  部经理͛  ▁reapΛ  PurchaseΛ  ▁Classroomϛ 　↓ϛ 兰考Л 跟前Л  另一只ћ Pushћ  记者提问қ  需要什么қ  ▁Monkӛ 天立ӛ 水长ԛ  并建立ԛ  不管是在՛  年龄不超过՛  ▁respecting֛ 塑性֛ 请了כ  村民的כ  正常通行؛ 实为؛  了自己的人生ٛ  ▁tighterٛ  ▁intermittڛ  ▁northeasternڛ 城河ۛ 赔钱ۛ agmaܛ  每年都有ܛ 打倒ݛ  ▁міݛ  资料库ޛ  因为他们的ޛ ecutedߛ  ▁Chavezߛ  ▁escalating  一年半 colour  ▁influencer  欢迎你  加州大学  身临其境 nodes  ▁ruthless ▁  突出了  具体措施  ▁noodles  ▁strengthens  ▁foundational ühr ▁lex  重点学科  ▁Sark ieving  ▁attic  ▁vaping  ▁disclosures 潮图 王爷 累死  土家族  ▁Kenyan  Relations кс 甜瓜  场所的  体温异常  有的甚至  续航里程  osexual  ▁Zimmerman  ▁Wonderland  对于有  同比增加  ▁lact 每件 灿灿 ▁IPL  看到这样的  Electronic 丽华 晚清 棉袄 窘迫  副镇长  年产值  切实提升  有效预防  从今天开始 Amount  ▁Trauma 尽其 沙门  还发现  音乐剧  心无挂碍  我们能够  ▁Inters 呼啸 学学 砍柴  大循环  依法查处  降低成本 Hours  ▁carpets  ▁engraved  ▁attachments 之母  以避免 返场  ▁payout  ▁Gibraltar  ▁distorted  几十个  幽门螺旋 opause  Anderson 云山 是种  还是不错的 vendor  ▁Notification 人将 星海 金玉  在浙江  老公的 ▁Bam  ▁bearings 竞聘  不受影响  劳动能力  新修订的 Separ Spect  互联网平台 花艺  像这样  通常在 ▁DOM  破获一起  除此以外  ▁Macy  ▁Kamala 降薪  ▁concussion 未果  女士说  自己被  董明珠  传媒大学  这个时间  ▁Twitch  ▁simulator  ▁Disclaimer ▁á acao  ▁labelled  和研究  产教融合  提前准备  总资产  是一块  就会造成! heter!  一旦有"  复旦大学附属"  foreign#  ▁austerity# 内疚$  本声明$  城市管理局% 李军% 腰膝& 艾条& 被我' 飞到'  一棵树(  性感的( ,)) 华严) 取保*  市中区*  阅读习惯+  ▁bred+  ▁disciplined,  所以对,  ▁Isles- ▁ERP-  有多重要. hesda.  ▁binge/ 油锅/  你的朋友0  ▁entails0 ( !诊断为新冠肺炎确诊病例1 李健1  房地产企业2  ▁Athletes2  为全球3 ▁MAS3 grave4 Marcus4  ▁straps5  ▁Receive5  ▁horrors6 zin6  一封信7  图文并茂7  金融风险8 indra8  ▁undefeated9  ▁Identification9  集贸市场: 萧条:  中国是;  形形色; Limit<  的学习方法<  Require=  ▁palate=  ▁pillows> 推诿> 沉浮? ARI? Tow@ tea@  归功于A  非常明显A 还送B  的绿色B tightC  ▁WyattC 发愿D  隔离区D  免费使用E  ▁anomalE  ▁fullestF 偏旁F  大脑的G 丘陵G 景泰H 樟树H 涅盘I 熬过I  今天给大家J  解决问题的能力J 竹笋K 耳环K  ▁basilL 万到L 职守M 虚开M  鼓楼区N  多年的历史N  ▁mindedO 声名O  ▁któP  发生冲突P  并不容易Q  疫情情况Q  FictionR  ▁tallerR ▁‎S 减分S 餐前T  见不到T  ▁SheilaU 乡贤U ▁AUTV  ▁CentresV 种菜W  在人体W ▁icyX  教育部直属X  中华民族共同体Y  DisclaimerY  ▁transformationsZ 攻打Z  指引下[  健康快乐[  uttgart\  ▁Sutherland\ Ii] 培生] 急寻^  团市委^  内容来源于网络_  ▁bankers_  graduates` 埋下` 网安a  学校要a  有一半b  ▁Clipb  ▁Hurtc 在眉c 失灵d  人和人d  有益的e hunge  请您先点击上面f 挂了f  一幅幅g  城小区g  的小伙h  报名咨询h 倪萍i viri  有效期内的j  NotFoundj 互帮k 再等k 天之l  岳阳市l  也能轻松m  熠熠生辉m  ▁benchesn 与伦n 小吴o 片面o ("\p  你也会p  口服液q  ▁BACKq  ▁Assamr 户数r  的效率s 南华s 和其t 尘器t 开张u  ▁Herbu  ▁Pediatricsv 方寸v 煮饭w 要问w Srix  还是以x  身份证复印件y  ▁repetitiony  和小编z  前后的z ESPN{  劳务报酬{  我们也会|  红军长征|  ▁Mandal} 开赛}  看看你~  行车安全~  负面清单  金融行业 湿邪 kun  清洁剂  ▁Depend  ▁Reunion 挺胸 桂皮 清蒸  到今年 Gordon хи  帮助的 azard  ▁Hezbollah 的分 谷县  一图读懂  专项训练  根据实际  综合运用 下级  治愈率  跟着我  人的精神  ▁skyline 以期 舰载  牛油果 OTOS  不要着急 omega  原因是什么 跌停 ENN  以上是  第五步  随后的  ▁servic 许敏  文明校园  ▁eloqu  收入水平 aghan  ▁erase  ▁Trails  ▁birthplace  打通了 dain Boost  ▁Moves  convert  itlement  ▁appellate 密布  卷土重来  经济管理  ▁fairs  研究中心主任  ▁Montenegro 禁区  力量和  стра reece  ▁tearing 好喝 曾于  医生和  点什么  中国航天  打好基础  roscopy 荒地  他不会  成千上万  有丰富的  ▁Gmail GX 落寞  吐鲁番 ▁CPR acula 板蓝  就算了  建筑材料  ▁Uniform 泽民 CFO Cole ▁Hau ▁MCU  frequency  ▁malpractice 门上  懂得感恩  这句话是  experienced 交钱 能说 解聘  一日游  人员须  侵权行为  私募基金 不暇 修图  真的是太  英语专业 uthan  ▁cogn  innings  ▁Attacks  ▁persuaded  售价为  科技企业  ▁Sans  ▁Increasing 总督  军队的  王登峰 verify 普高  地毯式  ▁smoother  ▁burgeoning  ▁ambassadors 温饱  很有趣  中国生物  ▁taps  ▁parallels 三不 年获 解码œ  苦尽甘œ  都是这样Ü  ▁coolestÜ 帮帮Ĝ  巧妙的Ĝ  红外线Ŝ  重要信息Ŝ  ▁rodsƜ  ▁disappearsƜ 牵绊ǜ lewǜ  ▁embodiesȜ  ▁undeniableȜ 摧残ɜ 瑞德ɜ 迎刃ʜ  关于公布ʜ  能够增加˜ umont˜  ▁Bucc̜  ▁logger̜  ▁Petition͜ ák͜ 原审Μ 高汤Μ  这事儿Ϝ ℃)Ϝ 代词М sacМ  ограќ hemianќ  ▁quaintҜ  ▁deficitsҜ KUӜ braceӜ  核酸检测报告Ԝ  LoadingԜ  typedef՜  ▁initialize՜  ▁southeastern֜ 刷刷֜  宣讲活动ל  ▁transcriptsל 压在؜ 法条؜  显气质ٜ  不断努力ٜ  同时进行ڜ  幸福的生活ڜ  ▁spoilۜ  ▁resurrectionۜ 浩浩ܜ 路演ܜ  由原来的ݜ  ▁navbarݜ 晨读ޜ 肌群ޜ 雁门ߜ  有小到ߜ  常务理事  不知道如何 Graham  ▁Schultz  ▁Arrested  ▁potassium 多食 建好  专家会诊 噩耗 天尊 特鲁 马车 ▁CAT ▁Lup  平均年龄  ▁Fernandez 软硬 辛格 Mort Wave onne  白衣战士 ུ་ 再大  的风采  能保持  光辉历程  技术手段  内容是  在高考  突飞猛进  资金支持 AVA  ▁Freder  ▁seventy  为了使  对于孩子  及联系方式 东江  和发扬  来决定  教会孩子  粮食生产 Qaeda 不闻 交货 城人 搀扶  散打哥 Loop  希望他们  ▁Brave  ▁mastery  ▁vertically 爆炒 直指 解忧  南怀瑾  创造价值 lyssa  疫情传播风险 宵夜 纷繁 jam  的大好  托管服务  ▁Wein  ▁hates 焕新  寓教于  的策略  等多种方式  ▁zosta  ▁locale 刀子 即送 毒药 疲乏  湖北籍  这所学校  全面依法治国  ▁anthropology 颜悦  两分钟  提供者  电脑端  贵金属  国家医保局  综合服务平台  ▁Hubbard 卖部 档位 自给  岗前培训  高高飘扬 Angela  ▁Vaughan 壮美 相距  肩关节 ▁Haj  的不只是  新发地市场 asonry  ▁Beard  ▁awaited 戒掉 陈氏 会友 杨绛  不尊重  任劳任怨  ▁Amen  Indiana  ▁drafts 相间 cción 地瓜  关怀和  青藏高原  被开除党籍 ▁nat  就没有了  ▁vineyard  ▁prescriptions 撤县 李煜  吴某某  守初心  代理机构!  进行学习!  ▁испо"  开展核酸检测"  otyping#  itutions# MER$ eyer$  ▁prestige%  受益者%  践行者& ocre&  有关工作'  ▁Destroy' 人去( 富婆( 顶配)  技术水平)  ▁Scha*  ▁refurbished* /$+  我经常+  ▁Rican,  ▁forex, 偷渡- 每时-  的秘诀. Groups.  Ancient/  ▁abruptly/  ▁Tradition0  但他的0 之财1 紧闭1 补考2 辨析2  地说道3  行走在3  销售员4  作出重要指示4  ▁Lester5  ▁Nasdaq5 girls6  ▁bounced6  ▁specimen7 多户7  之后又8  删除了8  吉祥如意9  ▁Shri9  copyright: 为快:  正是在;  ▁flashing; 唇膏< 矿工< Ids=  很多人对=  ▁Busch>  ▁Ramos>  ▁Meadow?  ▁formulas? 初秋@ 厘子@ 尽是A  与普通A  贯穿于B  世界和平B  ▁plummetC  较昨日C  ▁OrionD  ▁TeddyD 哑铃E 毛利E 调的F  总体战F rulyG  ▁WageG  agicallyH  SerializerH 动弹I здаI  画面中J  建议使用J  规定时间内K  ▁jokedK  ▁TriumphL  ApparentlyL  ▁touchscreenM  看到你M  BrowserN  ▁mergingN  发现并O  ▁sizeofO  PatientsP 开个P  第四名Q  ▁HorsQ  ▁CollectorR  在水里R  爱心企业S  ▁memorabiliaS  ▁InvitationalT  岁的她T  的缘故U 组别U  公开的V  方向和V 夏威W ivatedW 开营X 鸡血X  以下几种Y  还会导致Y anutsZ  ▁rogueZ  continental[ 定陶[ 开团\  福田区\  登记信息]  ▁slug]  采取刑事强制措施^ 防冻^  最真实_  initions_  ▁staffers` 金水` mitea  每天都能a  长期处于b  ▁SUVsb  ▁blamingc ▁αc ссиd  声明的d ▁Kese  依然没有e  懂得珍惜f  ▁Diedf  就是这样的g Soundsg 则可h 淤积h  将及时i  文化传承i  正常现象j moviesj  Celebratingk 气馁k  ▁Bessl  医学界l  和人员m  最大扭矩m  那个年代n 华强n 氮肥o ▁STAo  的小女孩p ionalep  ▁airstq 穆斯q 解体r 驱赶r  cussionss 切削s  就到了t kvt 置地u  以后再u  城市化v  ▁buoyv  ▁Reflectw 善行w 奏响x 莘县x 学渣y 形似y 红素z  创立于z  ▁pharmacies{  ▁Documentation{ rv| 罗湖|  新能源车}  ▁fences}  ▁Heathrow~  ▁interstate~ 光谱  快速反应  ▁ADHD  ▁vans 暗自  非常强 below  ▁quota  military 都城  一件件 Hidden  ▁Lyons 就坐 爆珠 而建  和自我  跟你说  的意思是 sweet  Dynamic 钦佩  项目概况  structure 华商  宣城市  进行网上 入药 火候 爱她 eln  学历提升  ▁incremental  ▁Conditioning 迎战  回归到  因为这是 CLASS  现代服务业  ▁Bones  ▁mathematic лы  分析及  初中的  和提高 夜读 如歌  ▁Themes JZ 赫赫  凉山州  发挥出  平均值  总人数  联系起来  ▁Blessed  是一本  久别重逢  自身安全  装配式建筑 Legend  ▁braces  ▁Assange  ▁exercised 莴笋  凯迪拉克  使用安全带  ▁gunman 浓茶 bdst  新形势下  ▁skins  移送检察机关 公牛 垂涎  历史和  发热患者  是孩子的 helial  ▁Pepsi  ▁suing  getting  transfer  ▁Waiting  却可以  学员的  社保局  不幸的是  报名资格  ▁Radical 一喷 ixue  连续两年  ▁nicotine  ▁diversify  中得到  李焕英  的夜晚 ▁Soy  价格便宜  ▁Salisbury 喧闹 够用  即可参与  很多人说  Supporting 健脑  激动的  微电子  ▁Poet  ▁borne  ▁apologized  ▁upholstery  ▁Advancement 颓废  新用户  陆家嘴 aira cold dose  融会贯通  ▁Moldova  ▁missionaries 莫言  纽约州  野战军  热门专业 善用  心平气和  规划纲要  ▁décor  ▁abolition 重获 aban  公司提供  制作方法  ▁FoldÝ  ▁dictatorÝ 湿了ĝ Kinĝ 三台ŝ  地完成ŝ  盆满钵满Ɲ  竞争激烈Ɲ  ▁Obstǝ  先进基层党组织ǝ  ▁carrotsȝ  ▁presumedȝ  identifierɝ 固安ɝ 学界ʝ 缩写ʝ  的贡献˝  显著提升˝  ▁stationary̝  并非是̝ aise͝  社会秩序͝ azureΝ  ▁worshΝ  ▁Eurovisionϝ 一睹ϝ 品茶Н cliН  不甘心ѝ  不可忽视ѝ  我们常常ҝ štҝ 日电ӝ 至善ӝ 话筒ԝ  不大的ԝ ptive՝ 高阳՝  老同学֝ ▁Poe֝ 候补ם 凉风ם 贪吃؝  争议的؝  在成都ٝ  的那天ٝ  隔离期间ڝ  审核通过后ڝ  ▁Closing۝  ▁distinctly۝ 一转ܝ 经侦ܝ nteݝ  实施后ݝ ▁Todޝ  应用商店ޝ  革命烈士ߝ  老家的ߝ  要重视 解禁  的练习  成人高考  相辅相成  ▁dues  ▁dolphins  事件中  很常见  发生性关系 顶风 wrote  ▁appraisal 入眠 民工 要强  注定是 rored 多套 张新 杂技 IRS  资源库  在高速公路 二元 当上  伸出援手  工作顺利  失业补助金  ▁Timeline 孙杨  时间过长  ▁Rovers  ▁Baroque 云天  ▁SaaS  ▁learner vale  小学教育  ▁Bayer  ▁Leaves 喷水  平方千米  ▁sculptor  肯定不会  ▁esterni  ▁exiting  ▁effortless  ▁thankfully 咒语 学部 插管 看上  游击队  精神上 thor  olvency  ▁plains  ▁manageable 参建 和被 点为  学子的  教案课件  ▁Helic  ▁Browser |“ 周后  民警发现 的战  morning  你是我的  部分素材  综合素质评价 了其 刘嘉 embro ( !中国共产党纪律处分条例  ▁proportional 汪汪 请客  再见了  防范和  ▁Inventory 开年 若想 ▁gn  一致认为  对于他们 chery  ▁socialism MQ 考博  ▁Definitely  ▁equilibrium Xi 征管  任何一种  可以提供  供大家参考  ▁puppet  ▁coincide  优势区  在一定  投档线 中行 山茶 满口  外立面  电子注册备案表 冷冷 最严 OTH  拿出了 alog  他们不会  ▁Minh  ▁foreseeable atas ipers  ▁cruises  保税区  牺牲的  事业单位人事 近千 迟疑 音字 "];  改编自 Ship canvas!  ▁Haram!  ▁defends"  ▁stocked" ️⃣#  进行集中#  培养学生的$  ▁Kyrgyz$  ▁PowerPoint%  习惯于%  守土有责&  获取最新&  ▁Forrest' 塞进' 骗了( 你呢( spirit)  ▁Twelve) 一脉* 丢人* 大义+ amorph+  ▁skysc,  ▁confessed,  也有些-  服务期满- 奋起. вич.  在本市/  预警信息/  creative0  ▁endemic0 兴宁1  海珠区1  离去的2  以及各种2  每个人都有自己的3  ▁bisexual3  ▁epilepsy4 添砖4 舒张5 阴有5  扫黑办6  ▁Iris6  年硕士研究生7 省油7  周期内8  小苏打8  竞争中9 其道9 鹿晗:  做出贡献:  党工委副书记;  ▁brutally; 可点< 大埔< 安琪=  和监督=  板蓝根>  留在了>  远的距离?  ▁Chung? 法界@ 高阶@  还通过A açãoA oldtB  scriptsB 安详C 追捕C INOD UMPD  蚌埠市E  身份的E  ▁NovakF  严格要求自己F 利剑G 多雨G 柱子H  犯罪集团H  疫情防护I  高层住宅I YoJ 婚庆J 炒锅K 速速K  映入眼帘L  ▁SunderlandL  统计表M tionM  马鞍山市N  particularlyN 行知O  不够用O  会引发P  可以自己P  接近尾声Q  非法经营Q  ▁SmashR  ▁rockedR  ▁airborneS 章第S  阳泉市T raiserT  ▁inventionsU 打嗝U 野鸡V  本就是V  商业保险W  领取电子版W  ▁screenwriterX 口子X  收割机Y  网编辑Y ▁rudZ  流连忘返Z Looks[  全体党员干部[ NX\ 大潮\ 安妮] 打进] 抖擞^ 花红^ 讲得_  将重点_  强化对`  prevent`  ▁courthousea 苦味a  不分先后b  ▁Calebb 社员c  睡不好c  当场死亡d  ▁tenord  ▁Consultatione 热卖e  中的重要f  ▁JUSTf 增援g 苦的g 蛋清h  文学艺术h  没有及时i  ▁blinki  zhaopinj  ▁SOFTWAREj 肚皮k 还小k  决定着l  西门子l  相关文件m  ▁exitsm  ▁interceptionsn 大法n  好像是o 快件o  文化内涵p 狼藉p  好用的q  时间与q coler  性价比高r  进入下一s  ▁Rayss 一算t 简装t 菖蒲u 香浓u  操作指南v  落地生根v  ▁Deletew  ▁fadingw  盼望着x  ▁Currencyx 群力y  并给予y  紧凑型z UTIONz 内服{ 已从{ 张恒|  峥嵘岁月|  ▁removable} 苏木}  也意味着~  你准备好了吗~  ▁pathogens 机率 被拒  可口的  能接受  科技发展  ▁Refer  ▁goose  provoking 东胜 绰号 ▁DJs 封神 甜文 速成  不到了  在推进  林黛玉  的第二  ▁flare  ▁Libyan  ▁shooters 坏处 洞穴  最令人  根据省  也没想到  serving 拉网 调换 opo  的好时机  ▁Pets kx  有过一段  活在当下  ▁cowboy  specified 权势 ▁Tos  容易引发  他们的生活 guitar  ▁Prediction  ▁Rockefeller hunt  积极进取  书法家协会  抗氧化剂  ▁django ји 小弟  的人的 borgh  ▁Fifty  releases  检测体温  幽门螺旋杆菌  ▁Chronic 山乡 痴情 退改  到中国  敏感的  沁人心  甚至于  应急管理厅 寻访 阿卡 面纱  十一月  带你看  常规赛 yout 另类 根除 炼化 称谓 pac  掀起了  违法建设  为疫情防控  谁也不知道  ▁tumult  ▁distressed 已由  ▁roce  straction 充气 忍痛  万人的  需要做  ▁Canton  ▁Symptoms  ▁pronounce 杯中  亳州市  ▁occupying  ▁intimidation 每刻  如果对 Bind  ▁forgetting  Collegamenti 天赐 的梦  夏威夷  每公斤 warn  ▁recognises nɡ 扣子  仍然存在  暨南大学  ▁Mostly  ▁programmer 屋后  灰指甲  而自己  让中国  顽固性 ▁Vet 回执  最伟大的  inherit  ▁Guadal 注音 ELS  留言板 ▁LIC  ▁Taxes  emakers  eligible  ▁nuances 当真ž  都不想ž itemapÞ  ▁TrackerÞ 不花Ğ 仁义Ğ 形同Ş 无权Ş 通化ƞ  人士的ƞ  英语水平Ǟ  ▁RudyǞ  不过气Ȟ  中文版Ȟ  回答了ɞ  阳性的ɞ 图示ʞ 工况ʞ 雷军˞  铁路局˞  ▁Pollution̞ ます̞ 复阳͞  清远市͞  熟悉了Ξ  的孩子们Ξ 喜人Ϟ 末梢Ϟ  告诉了О  在一家О  ▁marsў  ▁ergonomў 冠脉Ҟ 石子Ҟ  你愿意Ӟ  培训费Ӟ  闭上眼睛Ԟ  ▁subcontractԞ  ▁tuberculosis՞ ▁−՞ 隔阂֞  发展理念֞ Prefixמ  architectמ  ▁disagreement؞ 唱着؞ 房颤ٞ 拜佛ٞ 火辣ڞ 田镇ڞ 解法۞  迎刃而解۞ Viewsܞ  ▁disparateܞ 加厚ݞ  我建议ݞ  金银潭ޞ  团结一致ޞ  ▁eminentߞ  ▁liaisonߞ  ▁tackled  ▁Abortion  ▁Financing 伤疤 树一 迪奥  时间不  病例的 ▁MET  密不可分  一个人的时候 nob eeds  listening 老窖 苏北 顺从  成员国  ▁possessing 亲亲 对未  内在的  这就需要 对战  吉安市  神经内科  ▁Gotham  ▁authorize 丽娅 账务 陈年 rax  allclasses 去吃 同业 灌注 西关 CVT  ▁Sketch % 国家级非物质文化遗产  真正有  ▁Innoc  ▁procession  ▁congratulations  在抖音  Execute  ▁Riding  ▁Dentistry 添彩 过街 高配  ▁Genius 万贯 坐骨  之下的  非常不错 Walter 自大 转角  口袋里 dogs  几乎所有的  ▁wrongdoing 具足 贪欲 Enh INC  ▁clinically 小鹿 重修 Arn  的房间  可以按照  顽强拼搏 estion  ▁trusting 火柴  大家觉得  当天上午  空气清新  ▁Pixar  ▁Durant  在处理  ▁Agnes  卫生纸  一个又一个 凡夫 吐血 空袭 ydw  或媒体  ▁Frag  根据教育部  ▁yelling  ▁Associations  ▁interception  Authentication  ▁reapp  ▁bicycles ić 牵动 西餐  ▁Billie  ▁avoids " 部分文章推送时未能 松散  平稳有序 与非 霓虹  在银行  杨振宁  诗和远方 Ark  手机里  通讯社 ▁MER  ▁Taken  ▁pastors 严实 那叫  使学生  我们说  的香气  教师招聘考试  要提高政治站位  ▁disturbance 人或 国平 线圈 练功 ▁WB!  的摄入! ▁pom"  中考数学"  ▁Trilogy# ✪　# 否决$  生命科学$  ▁uncomp%  ▁Bishops%  ▁impulse& mw&  的句子'  Contains'  ▁postdoctoral(  地位和( 又双) 针法) 长方*  ▁Hanna*  ▁Household+ 周迅+ 操劳, 特设, 碌碌-  经济和-  也不容易.  小儿推拿.  文明行为/  ▁Vita/  ordeaux0  ▁absorbing0 招满1 高招1  ▁Respect2 $\2 就位3  搬运工3  的差异4  每人每天4  ▁Sevent5  ▁escalation5 大乘6 胸襟6  腊八粥7  静静的7 osto8  ▁dumping8 I B坚持以习近平新时代中国特色社会主义思想为指导9 精诚9 镇区: 鸡胸:  指出了;  治疗师;  伊丽莎白<  ▁bureaucracy< 洪波=  也会影响=  全面升级> 兴建> 善心? 未病? 涉足@  该地区@  ▁vomitingA  不发生A RCEPB  为您推荐B  ▁FallonC  咸宁市C  无添加D 大漠D  向学校E ▁NayE  下面这个F  我们都应该F  ▁HanoiG  ▁testosteroneG 已定H 知其H  对皮肤I  多多少少I  ▁SampJ 中锋J PMGK  全攻略K  速度和L  苏州大学L 得病M 重彩M  事故的N bowsN  企业资质O  ▁HelpsO 英尺P  无限次P NarrQ 家是Q 插花R 时也R iyuS ▁seuS  GenerallyT  ▁correlatedT 并网U  同时也要U  国际关系V  玫瑰全因子V  贸易有限公司W  面对着W  第一小学X  ▁fiancX  ▁lifecycleY  ▁sanctionedY 就绪Z 底座Z 该名[ Kel[  带领团队\  政策宣传\  发生了变化]  ▁oversaw] 怀仁^  双离合^  设定的_  ▁Ende_  ▁Toro`  ▁Valerie` 乘势a 初入a  妊娠期b  就算你b  注意饮食c  ▁Grimc  ▁ammond  ▁acceleratord  ▁convertiblee  财源广e  广泛应用f  其实并不是f  请联系微信g  ▁Kraftg 保镖h  出现了问题h  ▁conceali 吴京i  衣帽间j  强制隔离j  ▁Rupertk 之效k  不加价l  出席了l  对产品m  雄厚的m  ▁deuxn  ▁Tavernn 张春o 摸爬o 柔韧p amenp  以下材料q  independentq eerr  摄像机r  就能让你s  我们一起来s  我们想t  的优点t  爱不释手u  ositoriesu  ▁measurablev 毛鸡v  旅客列车w  ▁Statementsw  共同点x  更丰富x  检测点y ▁GTAy  他们总是z  叠加使用z  ▁Sard{  ▁Manifest{  ▁murderer|  ▁Producers|  ▁Techniques} 占星} )  ▁metre> 藏于? Bag?  送去了@  看的开心@  ▁DolbyA  ▁wineryA  ▁StantonB  ▁factionB 元朝C 抢眼C 秀发D  并不好D  不断加大E  尤其是对E  ▁ClassificationF 上官F 报导G 谋略G  部分人H  ▁CrimeaH 长到I  席之地I TRIBJ uriaJ  人间的K  生活节奏K  ▁FedererL 铜钱L 银元M  不堪言M  垃圾处理N  着力提升N  EndpointO  ▁InterimO 抨击P 更让P 炒饭Q  电影票Q  明明白白R  这是一款R  ▁fussS  ▁taxableS  ▁TownsendT  ▁PaceT  ▁unreasonableU  酣畅淋漓U ToyotaV  ▁VinciV 井井W 帷幄W 置之X 霹雳X UGHY  开通运营Y  支持政策Z  进一步了解Z 出勤[ 知悉[ 起大\  总价值\ ▁Sok]  邮件主题]  的营养价值^  ▁finely^  athletes_  ▁Champagne_ 哈喽` 油温` 热盘a  方便的a  充满希望b  西海岸新区b  indexOfc  ▁Lenovoc 推敲d 陈梦d  量减少e IBLEe  工作目标f  补贴资金f  ▁pondsg 开平g 火葬h 甜度h 瞧瞧i riji  总指挥j  ▁Wrapj Walkerk  ▁MUSICk 小指l  ▁Deppl  Christinem 母牛m  再也没n  左上角n  参加培训o  百年奋斗o  走着走着p 代数p 受热q 古时q  好孩子r osker  心情不好s  ▁Martials  ▁husbandst  要求各t Strau 日向u 灵修v  宁德市v  ▁eyeww isablew  ▁Newsomx 三会x 糖醋y  体验感y acicz  高三学生z  concept{  ▁psychic{  ▁regrets|  苏格兰|  Lawrence}  ▁repression} 为安~ 拉上~  技术要求  ▁midway  ▁noises 刚到 桃树 金毛  在场的 nian  egovina  Hardcover 但与 岁末  日地点  中国天气  我是一名  ▁Corridor 伯乐 反式 指针  屏幕上  正规渠道  有效身份证件  oriously  functions  ▁patriarch  ▁inspectors  ▁dimensional 牛顿  操作方法  的生活环境  ▁scooter  ▁breached 型和 标牌 老将  保质保  你的时候  处理办法 blers  电动三轮车 Shadow  ▁Belgrade  ▁screenshots  ▁Reduce  ▁exponentially  的重大意义 stress  ▁Erica  hospital 再从 删减 酱酒 金鹰 NOR ▁MV  来计算  ▁Eleg  ▁vectors 即到 多些 答对 ▁JOH  遵守交通规则  ▁Suggest  也不太 ISPR  布置作业  财务会计  ▁Lovely  ▁reusable  他的人  ▁DOWN  ▁opts  Changing  ▁Islander  ▁relatable 仙居 次要  低密度  你了解  将影响  ▁Alvarez 风挡  了孩子 wash  ▁Leafs FOX  你一个  ▁Heating  ▁affidavit 煎至 Err  美国国家  长期服用  ▁Sting 搭讪 站房 耐药 耻辱 还和  　#　  不断探索  Celebrity  有好的  卫生部门  ▁disappearing  万平米  学习者  甚至出现 Victor  ▁Frankie  ▁plaster 单选 王后  腹腔镜 Duke  管理岗位  国务院决策部署  专业毕业 iad  大小便 Wine  目前正接受 收腹 监利  不成功  微发布  记录仪  ▁sweetness . '中华人民共和国传染病防治法  你能有  家门店  的现代  可能会被  家人朋友   未成年人的  chromeà  stringsà + $解除医学观察的密切接触者Ġ  脱离了Ġ  ▁divisiveŠ ▁žŠ 两腿Ơ DasƠ  ▁KristinǠ rwǠ ренȠ  交通环境Ƞ  华中师范大学ɠ 复燃ɠ enskyʠ 操办ʠ 禁放ˠ ▁ERRˠ  就业困难人员̠  TestCase̠ 乡长͠ 五分͠ 再起Π 冠名Π  和现场Ϡ tellϠ  放假安排Р tainedР  ▁KohliѠ  ▁ExhibitѠ  的差距Ҡ  ▁PrecisionҠ  申请者Ӡ  时间为准Ӡ  金融产品Ԡ  ▁LobbyԠ  ▁roadsideՠ  ▁Ecosystemՠ  ▁Recommended֠  中医师֠  企业对נ  derivedנ  reportsؠ  ▁repaymentؠ 房网٠ 自荐٠ peeڠ  男生的ڠ chun۠  霍普金斯۠  ▁Loseܠ  ▁Forexܠ  触手可ݠ  ▁regexݠ  ▁cannedޠ  污染防治攻坚战ޠ bkߠ 昌邑ߠ  迫切需要  ▁morally  许多的  ▁perfume " 长按二维码识别关注  妈妈在  做人做事  ▁Transmission 本心  以高度  理解与  的资格 ▁Gly  ▁OnePlus  移民局 ▁GPA  技术标准  此次违建 俞穴 扦插  你能够  所有制  本题考查  ▁sophistication  个地方  的结局 ruby  招生考试机构  ▁brunch 仁慈 ▁kö  不高于  再部署  托马斯 ▁CSR  共渡难关  培养计划 Years dating  ▁buddies  ▁tedious  ▁nonfiction  弥漫着  联合开展  ▁Snapdragon 不欲 水井 罐车 躯体 CAC  ▁rupt  ▁prosecute 舌身 遇害 靖西  压轴题  联谊会  年出生于 CRIPT 候审 […]  之路上  父老乡 ▁Yad  ▁sliced  ▁graphical  ▁timetable 冰毒  延安市  ▁Govt  Alabama  ▁Athlete 良师 Deg  条件允许  ▁Comparison 看她 yla  发展格局 补虚 高专 LOS ▁ao  专人负责  可以预防  ▁Hier nature  ▁chased  ▁mailed 未完  等设备 人机 奉节 怪异 想学 烟熏 ▁NSF  ▁virtu  ▁Werner 脾脏 Dutch  ▁Dundee  ▁musicals  ▁confinement 修心 环氧 盱眙  点击底部  ▁Mara 鲅鱼  来说都是  热泪盈眶  ▁clog  ▁Incorporated Au 上完  发红包  由于其  人事任免  ▁teas  Cluster 奶爸 川芎 想和  了一口  校园环境 学社  月任现  让广大  食品加工  Marshall!  ithmetic! 冲到"  在民间"  负面的#  开局之年#  ▁galvan$  ▁nearing$ 环抱%  适当调整% 为中& 水后&  东西部'  河源市'  试试吧(  回国人员(  增收致富)  农产品优势区) 嫁接* 小队* ▁Sev+  identifying+ 请按, 赛程,  威士忌-  无声息-  究其原因.  ▁Accreditation. 命宫/  idental/  ▁murals0  ▁deductible0 岗镇1 过夜1 ات2 债主2  上万的3  或相关3  持续发力4  Basically4 触法5  该研究5 Hart6 ▁FCA6 垫底7  就属于7  ▁Cecil8 lp8  可进行9 ▁Pey9 将你: 长鸣:  向所有;  停车泊位; clips<  ▁Geme<  一个人在家=  erential= 对内> 烘托> 行经?  如果他们?  施工方案@  次会议通过@  ▁nucleusA  ▁WilkinsonA 哈就B  同行的B 已至C 并作C 灯谜D 纷飞D VelE  在前期E  真的不是F  重大事件F BrendG  ▁loophG  中国共产党的领导H  ▁BringingH 阿布I ▁WOI  五年的J  执法局J ▁alkK  水利工程K  特殊原因L  ThoughtL BOXM  阿勒泰M  实践能力N  实际问题N  降低血压O  ▁manicO ^\P 快门P 英美Q 谷底Q  海南岛R  班学生R  学习理财S  ▁arrowsS 喜获T  为同学们T  该怎么做U  ▁SlotU  WinningV 头肌V 它也W 打来W  ▁pacedX 巷子X 血汗Y ноеY  ▁ЗаZ  的背景Z 怨天[ 所愿[ 河间\ 猛然\  无挂碍]  无门槛] ▁BRO^  免疫球蛋白^ edited_  ▁Bethlehem_  ▁discouraged` 眼睑` 计发a  手续的a  申请条件b  言传身教b  ▁evokec  ▁Evelync  ▁braveryd % 邓州论坛邓州论坛邓州d 永丰e  戳视频e  这一类f  不得不提f  我们这个g  热文推荐g  详细内容h  ▁sankh  ▁Impressi  ▁wordingi 用神j  是一年j VOCsk  ▁Tobagok  ▁Marinersl 双线l 要件m oyem  就完成了n  有效成分n  违法行为人o  ▁extraco  ▁parodyp  Chairmanp 光头q 杞县q  勤勤恳r  同意了r  龙岗区s  可以改善s Glasst  urancest 井陉u 扇贝u  信息披露v ppelinv  其中境外输入病例w  ▁disconnectedw 天平x 弧度x 桥面y  在学生y ▁Fooz  党史故事z  是不同的{  正式宣布{  ▁bookmark| 推着|  会显得}  招标投标}  ▁Uses~  在遇到~  无愧于 古方 很忙  并依法  播出的  有一丝  正处级  损害赔偿 diagn  ▁irrespective 努尔 显卡 沙星  的根源  劳动成果  ▁Hadd 餐厨  纪念品  赵某某  之计在于  ▁Stere  ▁Tahoe  ▁curry  有阵雨或雷雨  ▁Bundle  ▁harmed  ▁Butterfly  ▁federally 梅子 爆品 Hugh  定期体检  ▁timestamp vv 太深 条第  无法自拔  ▁Imag  研究生招生  ▁tendencies  干净了  推动下  ▁siding  迈出理财第一步 皇姑 福地  不完善  又一遍  可能对  勤勤恳恳  ▁Adri  Brandon  ▁safeguarding 每生 jer  边走边  Materials  ▁smuggling  不为别的  未来三天  民警迅速 locks  希望同学们  ▁gearing 两脚 勤学  ”;“  减少外出  收获满满  disable 汉化 注塑  的全球  花生油 ▁ROM  感染人数  Cookies  ▁Conven  ▁Siemens  installed  也很大  的品种 ▁Sao  紧急使用  ▁cran  ▁Alphabet 点拨 秦国 rut  ▁фи  身体和  因为只有  守土尽责 etian  关注的焦点  english 青光 nin  流程和 ymes  ▁дру  ▁fraternity 无智 Hip  那些在 Brid  希望你能 slide  双城经济圈  Eventually  ▁Mansfield  和儿子  赶紧去  更是如此  ▁Hodg  全面发展的  ▁overlay  ▁Personality 亨利 公历 chs  ▁был  ▁Stem  日内与本网联系  Messages 历尽 ▁OD  校友会  ▁Bonds  报名截止时间  ▁Bridget  ▁polyester  ▁Southeastern 型血 ▁upp¡  ▁waits¡  ▁fluentá 中庸á 选课ġ  ▁Lowellġ  ▁evangelicalš 鬼子š  微平台ơ  新势力ơ  学科专业ǡ  客户提供ǡ  百家讲坛ȡ 胖了ȡ  在执行ɡ  美国总统特朗普ɡ “(ʡ 凳子ʡ 梁柱ˡ 粗大ˡ 韧劲̡  和草原̡  自认为͡  如期而至͡  ▁SunriseΡ  ▁rejectingΡ % 事业单位公开招聘人员ϡ 宛若ϡ  ▁ValleС  ▁snorkС  Campaignѡ 海中ѡ  无可厚非ҡ  ▁barrҡ 棋子ӡ  对符合ӡ  的恐惧ԡ  专项练习ԡ  弱势群体ա  社会救助ա  看到自己的֡  全国政协委员֡ 拉姆ס 探店ס  单位为ء  坚持用ء  移动端١ Pete١  ▁Dividڡ  closuresڡ 几日ۡ 看这ۡ  过大年ܡ alarܡ  但一定要ݡ ummiesݡ  ▁exportingޡ 蒂芬ޡ  史上的ߡ urvedߡ  ▁Huge  ▁PRES  ▁spat  ▁biotech 赵孟  不可逆  仪式在  可放大 antz 要买 >:: ène  旁观者  стро  菩提萨埵 版式 龋齿  值得你 ▁Mou  ▁Fairfield 任凭  郫都区  哭笑不得 Dennis  ▁motif % 全日制本科及以上学历  ▁registrations 嘉园 宝爸 热土 网警 难逃  ▁Redskins  ▁Formation 真言 ETE  报名通道 园所 改写  不停蹄 epid  正常工作  ▁manuals  ▁Laurence 前应 妖精 成事 道里  与妻子  ▁Warhol  ▁Defender  科尔沁  崇高的敬意  ▁Diagram 医技 喘息  变更为  视频里  ▁Stakes 月明 问过  花木兰  五官端正  ▁Bose  ▁cynical 传情  搭建了  ▁Laugh  ▁coaster 乞讨 在盛 种鸽 芡实 表妹 马家  虽然我们  进行处罚 接回 腾空  腰酸背痛  lighting 轻轨 ▁Ner  充分体现  消费升级 % 当日新增治愈出院病例 水光 沉静 节庆 Lam  的日本  一部手机  ▁Revised  ▁Celebrities 家财 工件  不耐烦 aryl  三年以上  ▁chandel  beautiful  ▁Mitsubishi  在开展  自去年  通知单 dump  ▁buds  ▁maths  ▁resurrect 雅阁  参议院  要尽量 降为  公司为  的表演 ▁ICO  长期从事  全日制普通高校 宗族 将车 水饺  共同打造  在线观看  的小可爱  第一印象  ▁uphill  ▁Fighters 英伦 西服  测试题  一个故事!  ▁Bowen! öm" 就少"  从美国#  项目现场#  智慧和力量$ 首座$  看到过% ▁HMS%  ▁minimise& 即说& 离心' 黑道'  学习笔记(  无法核实( prime)  Providing)  ▁cautioned*  ▁narration* 公德+ 天街+ cox,  一半的,  研发投入- obalt- 一纸. 两辆. 肮脏/ Crow/  生活条件0  领军人才0 covers1 罗布1  和你的2  属性的2 ▁PVC3  ▁Gord3 去化4 山居4  他家的5  视而不见5  医务人员的6  十九届二中6  ▁makeover7  ▁knowingly7 大车8 花灯8  相关性9  我们还要9  招聘工作人员:  的爸爸:  职业培训;  贫困劳动力;  实现财务自由< 喜茶< 新传= 晴空= 视光>  发现和>  阳台上?  与人相处?  思维模式@ 北城@  直播时间A  ▁drawersA  ▁detaineesB 笔触B  博大精深C  盈利能力C “”D 贬低D  半成品E  不舍得E  给父母F 备货F 水份G 虫病G  ▁PrasH  ▁PregnancyH 破格I  管理要求I  来源及作者J  ▁KendrickJ  ▁socioeconomicK 侵华K  也相当L GalaxyL  ▁flippedM  ▁ParticipationM 也只N 是福N  专用的O  音乐人O omalyP sparkP  ▁divestQ  ▁analyzesQ 汉阳R  与方法R  政治纪律S  的积极性S gaardT  ▁glamourT 倒班U  大地上U  软硬件V  累计完成V  ▁semifinalsW 天麻W 是何X CowX  长治久安Y  ▁GREATY  ConsideringZ 三界Z 五香[  手机短信[  湖北日报\ aguay\  ▁Québec]  ▁Hutchinson]  ▁neuroscience^ 昏暗^ 病亡_ 钾肥_  一件衣服`  一探究竟`  招生院校a  文安微生活a  ▁precinctb 安岳b  不明确c  系统地c Fernd ▁PATd  深入一线e  经常食用e flagsf  anderingf  ▁confidentlyg 亦乐g 退路h  中新社h  从小学i  发源地i  的热度j pendj  ▁Lexusk  ▁credentialk  ▁inconveniencel 变暖l 油性m 雷诺m  的愿望n  getStringn  ▁attendanto  ▁Cumberlando 梨园p 灰心p 的刘q  住房和城乡建设厅q  ▁Scandinavianr 法华r 颅内s URSs  第三季t  教育体系t  毛泽东在u  眼耳鼻舌身u  看的开心可以v  ▁deportedv 受用w 心衰w 范例x 铂金x  你今天y  方式来y  现在报名z  ▁uterz  ▁grazing{  每一个人的{  ▁breasts| 滨水|  少女心}  消费满}  ▁pony~  ▁полу~  ▁respondent 休憩 吴孟 微讯 菌斑  重庆大学  ▁Bett  ▁bounty  不管怎样  教育质量  裁判文书 产检 故说 直上 黯然  了一声  现在有  的人了 ▁Boo Adams  ▁maze  ▁amassed  ▁conquered 下步 然大 龙川 Neigh  ▁Spending 两极 活佛 URR  不轻易  去思考  在发展  是具有  浓度的  严峻形势  教师考试  ▁Papa athing  utherford % 案件正在进一步侦办中 一晃 书架 厚望 耕种  不要紧  简单又  障碍物 odie  他们自己  Premier  ▁comprehension 之象 黄文 GIF  我儿子 cass  为大家介绍 apixel  ▁Lansing  ▁artisans 少阳  了一颗  在最后  被美国 AGES alism  ▁prostitution ❖　 预兆  才能达到 usual Claire storey  ▁subtitles yq 白花 ),” COS  好少年  在互联网  ▁Stir 良品 茶室  宇航员  有着很 ▁Baj  使命担当  国学经典  尽可能的  ▁retri  ▁dove 葱葱  一旦出现发热 君王 太远 字画  区领导  招股书  是日本 松下 豆沙 лов  的阅读 ▁tug  优秀青年  学历学位证书  ▁Cavaliers  ▁discourage 松开 氏菌 民风 铸牢 oplan Famous aughty 耗尽  已成功  而出现  自然也就 earned ucceed  Brother  ▁hardships 不归 连体 autom ophile  永久基本农田 生而  一些的  不等式  多种多样  男女老少¢  ▁razor¢  ▁Kramerâ 长风â  就有一Ģ  效果明显Ģ  服务模式Ţ  驻村第一书记Ţ  ▁esportsƢ 木有Ƣ 膨大Ǣ crestǢ  ▁HedgeȢ  ▁SmileȢ  ▁paralyɢ  Rendererɢ 政企ʢ 清秀ʢ ukhˢ  房住不炒ˢ virus̢  ▁Dolly̢  ▁scaff͢ 大干͢ 年审΢ 激流΢ 违禁Ϣ  失业率Ϣ  是很重要的Т  ▁cartridgesТ 时须Ѣ 菜花Ѣ  是否在Ң  ▁VIDEOҢ  RegardingӢ  ▁distortionӢ 修为Ԣ  爬起来Ԣ  人的时候բ  ▁Comeyբ  的桥梁֢  建筑高度֢  限时特价ע  ▁heapע adayآ ▁Gusآ  栏目视频٢  内分泌失调٢ 达摩ڢ 集第ڢ  请一定ۢ ▁FITۢ  学校建设ܢ  ▁Rosieܢ  贡献自己的力量ݢ  ▁Directionsݢ 就找ޢ OMGޢ  平均每天ߢ 美工ߢ  成本和  高的人 ▁MIC  三个阶段  ▁gras " 大学本科及以上学历 拳道  在新的一年  ▁dentistry 外星 钟楼  ▁Mighty лав  鹌鹑蛋  什么情况  ▁footing  ▁selects  ▁deposition 两新  优先录用  收听收看  ▁Ends  ▁smokers 上楼 便饭 普利 荣成  热闹的 Catch  得淋漓尽致  ▁Edith  ▁cooled  ▁analogy 张嘴  等部位  邀请到 opaed  还可以通过 ledged  Francis  ▁Blacks  ▁Merrill 典藏  不错了  也没用  感谢你们 游走  十字架  ▁boxer  ▁dishwasher 囊中  摸爬滚  活成了 Jump izen  才能获得  硬笔书法  ▁Nora  ▁Racism  ▁Examiner  ▁Jurassic 她将 就买 ofi  医疗救助 可凭  中明确 ▁Ack  ▁insiders 吴晓 存档  征收率  定期检查  政策执行 历历 月后  合理确定 ortex  ▁Benz 乱世 除数  再通过  军医大学  ▁Sunni  ▁thats 的电  义务兵  ▁ringing 圆圈  迈出了 ▁HIP  夫妻二人  相关事项 ijian  ▁Tolkien 闭嘴  县委县政府 信奉  保护动物  应接尽接  ▁kter 之差  回顾了  换言之 Sold ▁viv  往往都是 smile  ▁Soil  二级巡视员 传祺 攀岩  创下了  独立性 enko  会有一个 包厢 祖籍 飘摇 oum ▁Osw  服务管理 rangle 临走 舞者  号电话  马不停蹄 emann 情书 新乐  支队伍! Fine! hare"  颠倒梦想" % 学习贯彻习近平总书记# 上林#  方式是$  ▁inconsist$ rg% 亮化%  到晚上&  处理后&  ▁LIKE'  ▁Zend'  ▁Swansea(  人左右( male) uracy)  移动支付*  ▁fruitful*  ▁calendars+ 光圈+ 指明,  小汽车,  的年纪-  老百姓的-  这片土地. ▁Tao. 辛金/  ▁Chairs/  ▁Motorcycle0 加索0  大三花1  透露出1  可能曾经2  magazine2  ▁Challenger3  ▁Mathematical3 新潮4  ▁Erick4  ▁wartime5  ▁Implement5 燃眉6  宜春市6  校领导7  Discussion7  ▁unpopular8  声香味8  流动的9 会阴9 村长: 镇压:  半年后;  庆阳市;  拿什么< ▁Duo<  农村居民=  普通本科=  党和政府的> Female>  ▁ballad?  ▁coined?  ▁sneakers@ 张继@ 连任A  新农合A  运动场B ▁IPSB  ▁TarantC  typenameC 李俊D  乡村治理D  停止使用E  的检查E ▁TecF  很难做到F mightyG 的生G  居然是H  电力公司H  watchingI  ▁subdivisionI 醒脑J 驱使J  小便宜K  的生物K  ▁HowellL  ▁impoverL " 请及时联系我们删除M  ▁responsiblyM 凸起N 英超N 装作O  ▁SELECTO  ▁prototypesP  ▁organisationalP 选品Q стіQ  稳定在R  主要建设R  ▁adhesiveS  ▁AdvocatesS 双鸭T 新剧T 活水U  了一片U  双选会V  就感觉V  ▁släW  ▁inertW  ▁deteriorationX 学和X 进到Y  挨家挨Y  ▁RahulZ 横溢Z 藏友[  有希望[  业余时间\  嘎嘎嘎嘎\  可以避免]  ▁Magu]  据媒体报道^  ▁McKenzie^  为何要_ ▁ALS_  不是有钱`  modation`  ▁Transcripta 切段a  应当依法b  Columnsb  ▁lawmakerc  ▁Mauritiusc 报喜d  ▁mitochondd  ▁discriminatee 原判e  每天喝f  习近平谈f  充分运用g  都是我们g  ▁Mbpsh  ▁Hanksh  ▁Barnetti 今明i 抽空j 考过j  偏远地区k  投标文件k  privacyl  ▁Lawsuitl  ▁Mechanicsm  ▁originalsm 事中n  ▁étén  ▁juxtapo 找找o 来接p 炮兵p 速冻q  他也会q coutr pianr  中国革命s  反馈意见s  ▁Contractst 4 -论坛邓州论邓州论坛邓州论坛邓州t 待续u 速写u  ▁nombrev 脑后v  显示了w  杀猪盘w  肩胛骨x  设置在x  重量级y  市人民法院y  联合创始人z istlesz  ▁summoned{  ▁archaeology{ 拓跋|  深度合作|  ▁Patty}  变得很}  女大学生~  ▁Pest~  ▁MSNBC 剖腹 弄清 张江 近照 雪茄 opol  市交通运输局  ▁Madden 遍及  众人的  想说的  的方案  范围广 pair  危及生命 Enable  ▁sparks 受挫 ▁pag  本文作者  追究责任  新冠肺炎病毒 兵法  题目的 FORE  区疾控中心  这样一件事  avement 不超 着了 设卡  不进去 ▁Umb  无法接受  ▁confess 乡风 小帮 补种  的烦恼  非常快  自然风光  三年级上册 而要  不单单  优缺点  整体来看  university  ▁Charlottesville 教综 更应 赵忠  不胜收  面试人员 炒匀  发热病人  这些症状  ▁Junk  为切实做好  刑事诉讼法  ▁nephews 三段 渡江  ▁Companion 听我 校准  一志愿  金融业  请加微信  我们将根据 ISTORY  ▁Sabbath 三胎 DHA  ском  古典音乐  笔者认为  ▁Rohingya  人事迹  安全责任  岁的儿子  早上开心快乐  marines  Criminal  ▁diaspora  ▁entrusted ▁Ud  回不去  在产品 ▁CMA  调整方案 eming  ▁Negro  ▁preach  ▁hesitant >; 等您  在高速 Vacc LEASE 心绪 洞悉  ▁Shiv  的一段时间 oscopy  ▁Actual  Beginning 全红  平台进行 ▁Zo xico  ▁slider  ▁Charitable 名和 炊烟  被全国  各位同学  是为了更好的 天灾  ▁Yose  点击左下角 Comput  ▁Cooke  ▁proficient 芦笋 ▁MH  发展潜力  ▁embarrassment 战舰 wah  一书中  为提升  小王子  结果被  要时刻£  这让我£ ▁Lefã  ~~~~~~~~ã 不孝ģ Exitģ  就足够了ţ  ▁Gloverţ )'ƣ 再婚ƣ 几只ǣ 智联ǣ  供参考ȣ  让所有ȣ  充分了解ɣ  此起彼伏ɣ 洪流ʣ 纱窗ʣ 黑帮ˣ  公信力ˣ  家属楼̣ uarieṣ  ▁fatallyͣ 不痒ͣ  非法收受Σ ReplyΣ  的一种方式ϣ  ▁venerϣ : 3邓州论坛邓州论邓州论坛邓州论坛邓州У 出庭У ▁HIѣ  抢险救灾ѣ  美团外卖ң imaginң 全无ӣ 吨的ӣ  的一批ԣ  ▁Georgesԣ  下载地址գ  不限专业գ Prepar֣  ▁racks֣  ▁unleashedף  含糖量ף  表达的أ ▁Alaأ ▁Fib٣  ▁LIMITED٣  ▁diarrheaڣ 卡号ڣ 岳母ۣ 酸味ۣ  义务人ܣ  制定本ܣ  新经济ݣ  追责问责ݣ  ▁Scoresޣ  ▁loudlyޣ 海阔ߣ  等级的ߣ  都能用  综合运势  ▁Livingston 和马 均与 投靠  同期的 Myst  不断深化  毫不犹豫地  ▁banker  ▁maximise 小节 面面  为背景  认证书 双肩  加关注  本无归  习近平法治  医护工作者 encils  ▁Leads  ▁Managed  ▁orchestral  ▁Yellowstone 跑来 ▁gli  在新时代  有较好的 natal  某种意义上  进一步调查  违法犯罪线索  ▁Mourinho 侨胞 天喜 就爱  不开学  范围之内  雷暴大风 araoh  更别提  使用情况  ▁Imam  ▁Roche  Usually  ▁disposed  ▁Americana  块钱的  用电器 iosis  传染性疾病  行政服务中心 下棋 北汽 面糊  人员等  而就在  中医药管理局 剩菜  内容简介  时政热点 INSERT  向对方  年元旦 Amid ▁Lec  ▁ivory  ▁Bellev  ▁DateTime  ▁acquires 又又 因你 vil  农村人 corp  ▁Rolex ▁tf  意气风  ▁cello  ▁grading  新功能  钱学森  会形成  ▁saddle  ▁которы  Alternative FU 菌素 angi  美好的生活  ▁drank  ▁immersion 鞋面 tall  成交价格  ▁facet 华阳 辽源  想成为  成年后  接种的 二等 戒备 明咒 等位  取保候审  等方面进行  ▁Kylie  ▁Midland 来愈 甚是 痢疾  直观的 熬煮  公共汽车  属地责任  ▁stadiums 横店  脚后跟 fine AO ља  着眼睛  上下滑动  ▁dealings 冲上 坐牢 跑赢  不能太!  为榜样!  家医院"  ▁sque" 百炼#  冰冷的# Into$  居家学习$  都能做到%  眼耳鼻舌身意%  ▁Composite&  ▁Autonomous&  我每天'  麻烦了'  ▁Coat(  ▁Saud( Reason)  ▁Everyday) % 得阿耨多罗三藐三菩提* 下跪* 区等+  一节课+  提供相关,  收集整理, 恩情-  怎么选-  ▁candidacy. ▁Ses. ierte/  ▁Structural/ 把小0 Kee0  以防止1  的设备1 ureka2  ▁haunt2  ▁Infinite3 上用3  感情中4  地理坐标4  解决实际问题5  ▁Jeanne5 水木6  五行属6  公主岭7  处以罚款7  测试结果8  ▁attire8  ▁speculative9 ▁MIL9  如果自己:  感情运势: uvian;  ▁ants;  ▁Gareth<  ▁avoidance< 楼阁= 租用=  文旅局>  理工科> ▁ner?  一心一意?  业务培训@  非常高的@  ReviewedA  ▁ApproximatelyA 僧人B 子镇B 才让C 鸡苗C rylD  有人用D  溢价率E  一百周年E  亿元以上F  先进技术F  实战经验G logueG  ▁ThinH  以下几个方面H  ▁nativesI 大阪I  的时候了J 海纳J 爽的K 逼人K  但有些L  手工制作L  早上安康M  这些方法M eitherN  ▁dottedN  ▁darkestO  ▁TemplatesO  ▁sightseeingP 寒邪P /㎡)Q  乘坐地铁Q  活性成分R  自然景观R SplitS  齐齐哈尔市S  国际会展中心T  ▁RewardsT 下穿U 谭德U  万多人V  在部队V entesW MentalW 1 *新型冠状病毒肺炎疫情防控工作X GolfX  新闻出版Y cryptY  为了自己的Z  具有很好的Z iframe[ quirer[  complex\  ▁Harding\  ▁criticize]  不遗余]  十二指^  发生于^  发展阶段_  还有网友_ 不雅` 联发`  前瞻性a  原本是a  ▁Mildb  我一定b scanc othicc  ▁Jasperd ród 洗脚e 消逝e 颇丰f  中石油f  关系中g  第一剂g  我们就要h  ▁palph  ▁Favori  制止餐饮浪费i  ▁Trucksj 中支j 汝州k 陀罗k  一条条l  ▁Batonl  ▁Ferdinandm 邀你m ABBn  五年内n  货物运输o fluido  ▁Nationwidep iakp  不慌不忙q  村民委员会q  ▁McMahonr  ▁Natashar  ▁cheerfuls  我们一直s  疫情反弹t  aculatet  galleryu  ▁alarmsu  jjjjjjjjv " 一次性使用医用口罩v  ▁construedw 孝心w  新地标x  色拉油x Gally ▁aply  ▁ACLUz  声香味触法z  Nicholas{  ▁Bernardino{ 不叫| 四通| VEN}  各社区} eran~ Spider~  ▁optics  ▁paramed 不交 近两  残酷的  ▁Calc  ▁OSHA  Tonight ▁Ecc  积极发挥  ▁knockout  ▁pavilion  ▁continual  孩子都  身体内  对口支援  这样做的 南航 参选  而影响 ▁fug %" 卸载 得有 潮阳 的木  在心中  是一张  限度的  成渝地区  注明来源及作者  ▁succeeding  到后来  换成了  最后是  毛坯房 Fear ▁mai  减税降费  密切联系 Bryan  ▁Adele  ▁timeframe 吉县  他要求  的新型  的诱惑 abit ▁Sle  五四运动 ▁Kip ▁yen  发家致富  市财政局  选拔任用  具备一定的  ▁Juris  ▁Survivor 巡演 ]))  地下水 éric  已经出现  ▁tint  ▁Episodes  ▁programmers  一定要看  可以缓解 犯人 纠错  电话卡  百年的  苦不堪言  随遇而安 男星  咬一口  等费用  洪涝灾害  ▁Emory  美丽乡村建设  ▁newcomer 来院  对整个  肌肉酸痛  compiler  ▁recounts 启幕 废除  交流发言  跌宕起伏 张杰 急售 观澜 chsi  从头到尾  的浓厚氛围  是无上  有变化  这两款  自己的事  ▁cooks  ▁Sirius  ▁tipping 前人 Roc  新阶段  全体成员  天长地久  用法用量  认真贯彻  ▁Brack 德化  最核心 lund  四大名著  幸福安康  ▁Archived 灌顶 续写 icum  ▁Pied Square  ▁regimen 生情 Pam ▁Gö  也有一定的  ▁Ankara  ▁foliage  ▁Croatian 咒曰 苍苍 面馆 ▁nm  点点头¤  ▁Held¤  Projectsä 下到ä  汇聚成Ĥ asmusĤ  ▁EnsureŤ  ▁BookingŤ  ▁incrementƤ 林镇Ƥ 补品Ǥ 默哀Ǥ  在办理Ȥ  真实案例Ȥ sprɤ  版车型ɤ  蔡英文ʤ  的背景下ʤ  去医院检查ˤ  综合执法局ˤ  Healthcare̤ 全真̤  代表们ͤ  的人脉ͤ  技能提升Τ  ▁jurorsΤ  也有网友Ϥ  ▁hidesϤ 单据Ф  皮肤上Ф ▁VISѤ  内容标签Ѥ  ▁NOAAҤ 多所Ҥ  欧阳修Ӥ  婚姻状况Ӥ  若干意见Ԥ  这门课程Ԥ ifterդ  世纪初դ  的一处֤ mbuds֤  ▁finnsפ  ▁Yankeeפ  ▁synergؤ  organizedؤ 捏造٤  实力派٤  萧山区ڤ gettڤ  药业有限公司ۤ  ▁Violetۤ  官方公众号ܤ  ▁Bunnyܤ  ▁supremacݤ 植保ݤ 玩弄ޤ  昔日的ޤ ▁bluߤ  主要就是ߤ  癌症患者  跟着小编  这种东西  ▁modifying CJ ▁EE  软件开发  ▁prox  ▁Freud  ▁solicitor  南平市  又双叒  成本低  的颜值  公开发行  污染环境 Histor  Knowledge 华泰 本报  千千万万  网易云音乐  elastic  naments 套用  总以为  放学后 ▁Cum  企业进行  天涯海角  ▁Alec  交通警察大队 南澳 eryl  不再担任  中医养生  大家纷纷表示  ▁ovarian  ▁microscope 借阅 影迷  专家组会诊  ▁Zombie  Leadership 在无 正南  教授的  海产品 ▁bij  ▁Rede  ▁Tran eville  ▁glaciers  ▁forwarded 思虑 澳元 邋遢 部就 ▁MHz 旺夫 远航  ▁decimal  ▁succumb 方为 ▁nie  让消费者 越有  对不对  请登录 和后 jiu  乱收费  多云到  ▁Universidad ivist  atlantic  ▁Stainless  ▁originating hj ᠲᠠ 多与 脱胎  不得参加  公布如下  ▁Ballroom  ▁esteemed 莘莘  万像素  服务大局 ocard  ▁manifesto yam  数十万  asuring 独占  父亲是 CAM  副主任委员  ▁gamer 流于  每天进步  Featuring  和条件  都让人  ▁pity  远离颠倒梦想  ▁Titles  ▁splits  ▁datetime  ▁preferably 深谙 FAQ  中共中央办公厅  ▁Contractor 凤翔 定论 称霸 阴县  对你好  等关键  让女人  送福利 才对 破冰 花芽  大小事儿  ▁forc a Z▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 鸡鸭 VED  燃眉之急!  电影学院!  股权激励"  这样的事" 七点# 平浪#  物品的$ ▁Nun$  ▁Suns%  年秋季学期% 飘落& ouk& ▁kW'  不容小觑'  通讯地址(  ▁Jord(  工作人员进行)  ▁Cayman)  ▁multiply*  ▁criticisms* 让行+ 赌资+ lle, ▁NDP,  添砖加瓦-  ▁Ricardo- 免单. 神怡. 签了/ ▁CX/  的少年0  热点话题0 原县1 裂肺1  提出要2 妥当2 chemy3  ▁demos3  ▁Oceans4  fordshire4 ев5 民乐5 皮蛋6  但不会6  来选择7 Lucy7  ▁Morg8  ▁latency8 凑合9  在参加9  马上要:  农业科技:  ▁quasi;  ▁begging; 才干<  考场上<  达州市=  心理准备=  ▁Violin>  常务副市长> 古巴?  上帝的?  了一则@  式呼吸@  物体的A  科学发展A  ExchangeB UbB  写文章C  出手了C  古代的D  环比增长D  生产设备E  renheitE  国家自然科学基金F 二模F 州镇G 数理G  年少时H  直接责任H  职业技能培训I  ▁landownersI movJ  修行的J  巧妙地K  英语老师K  ▁BeatsL  涡轮增压发动机L 撕心M 棉被M 而遇N  薄薄的N  善解人意O mailsO  ShortlyP  ▁CreatesP 翻越Q NapQ iaeR  他不仅R ▁PauS  现场进行S WayneT WorkerT phoonU  ▁veneU  ▁gradientV 一变V 脚石W 铁锅W TarX  交通肇事X akensY  ▁tidyY  ▁sabotZ 优生Z 口齿[ 改用[  别有一番\  提高自身\  ▁Yosh] 琅琊]  有个人^  这孩子^ Berlin_  started_  ▁swallow` 暖胃` 起色a  和考试a  身体好b obiab  作为一项c  善待自己c  经济形势d  ▁strided ▁EVse  所有人员e  接地气的f  及图片f  是学校g  正式的g  正科级h  黄金周h ijiei  对未来的i  深入挖掘j  统筹抓好j wartsk  ▁Enemyk  rivationl 吊坠l 名门m 恶毒m 清江n 纳福n  七十二o  了几个o  品种的p 又像p 胜率q  唐人街q  考虑将r  不可多得r  ▁Dalys  ▁Napas 千岛t 在床t 最火u 犯规u ▁Xuv  个省级v  ▁Arielw  专属的w  定期对x  让用户x ainersy  ▁Javiery  ▁Mandarinz 供地z  与公司{  无有恐怖{  ▁christmas|  ▁disadvantages| མ་} 小病}  无花果~  线下的~  为了自己  的新能源  ▁Russo  fecture  ▁farmland  ▁Prosper 海马 邮局  周年的  妇科病  摘口罩  高级的 oxue  开发利用  招聘流程  ▁compensated hg  指挥员  恰恰相反  人与人之间的  预约时间 orama  ▁characterize 兴衰 辣条  ▁totaled  ▁Juventus  十余年 keye  你一定会 Lastly YN 三审 碧波  新课标 Corp  ▁Gael  ▁awarding  读一读 ▁Sto  nesdays  ▁Fathers 一株 图鉴 在国  也并不是  已经全部  成效显著  的一点是  进入人体 Hyper  永远跟党走  省委宣传部  renching  ▁Animated  ▁Providing  ▁ancestral 中车 定档 酸的  新春佳节  educated  ▁sweater  ▁Correspondent 洛斯 盛出  中国有 ▁SAR  统筹兼顾  ▁Approved  ▁trainees 修车  我们做  工资收入  有偿补课  辛勤付出  ▁Highest  ▁Employer 肯干 ▁Gn  的由来  ▁fasting  副厅级  行走的  远低于  ▁Dmit  ▁biases 厌食 收复 费时  十几年的  就是没有  等领域的 rouse  ▁excerpts 老君 血腥  ▁Hathaway ▁Ong 盘旋  的节目 ▁plc  ▁YMCA  oppable  ▁quotation  软实力 ▁Myr  ▁Jain <% 勾结 议价  他们从  了一部分  ▁Heck  ▁brightly 为荣  他只是  拟表彰  的高管 Mann  客服热线  ▁Lack  productive 做对 到村 妖怪 血气 LOL  卫生巾  祖父母  重症监护室  ▁Paolo 茶友  小清新  我就要¥  机械设计¥  高中语文å  ▁memeå  ▁reintĥ  ▁textingĥ 晚宴ť 通病ť 道夫ƥ  中国队ƥ  今年第ǥ  全明星ǥ  小学和ȥ  Sequenceȥ  ▁augmentɥ  ▁instinctsɥ 欠债ʥ  引发热ʥ kamp˥  悄无声息˥  经典歌曲̥ iejsc̥  ▁Packagingͥ  ▁Manuscriptͥ 万寿Υ 武侯Υ 集灭ϥ  乡政府ϥ  勤通风Х  安全措施Х  的工作要求ѥ 刘玉ѥ 弯弯ҥ  具体来说ҥ ۛۚӥ 一学ӥ ',　ԥ  ▁руԥ  持有人ե ▁ACMե Demand֥  grading֥  Permissionץ ikiniץ  ▁Courierإ  ▁unmannedإ 傻了٥ 收音٥  探索建立ڥ engingڥ 榆次ۥ 炖煮ۥ 霸凌ܥ  专题片ܥ  到其他ݥ  恰当的ݥ  留学人员ޥ  ▁Bailޥ ádߥ 审评ߥ 并拢  ▁audiob  aneously 也该 战果 离了 Grad  手足口病 ixties  ▁bathing 经省  交换机  信息汇总 roduce  ▁raging 奏曲 肌理 NEY Wor  占地约 Been  ▁Sonoma  ▁sacked 三清 季风 ▁DT  地藏经  奋发有为  ▁imped  eclipse  ▁Rebell 最差  包饺子  特区政府  ▁Remix  Academy  ▁Genetic  ▁commonplace  ▁protagonists 体和 受种 图库 年才  要尽快  ▁dunge  ▁agendas  ▁medically 含情  年代初  法西斯 ▁Wem ATIVE  ▁amphib  Execution 梳头  在生活 yrs  爆发后  自豪感  主要体现在  ▁obese 洼地 绍峰  尊重和  显然是  ▁fetal  ▁cellphone 愚昧 插曲 错综  主战场  号通告  后来我 enas  但并不是 Reach  ▁federation 七杀 升入  当时我  遗留问题  市市场监管局  ▁Burnett ▁ICU  ▁Herr  Records  国家税务总局公告 蝎子  第一的 ▁SCH  ▁unsubscribe ▁ODI ▁PNG że 宣部 重头  分阶段 ▁LOC  ▁cavity  ▁apartheid 外企 常客 进餐 问起  微信上  ​​​​  ▁Massive  ▁Journals ▁[],  制作工艺  itivity  别忘了点个在看哦 难平  大屠杀 Upper  ▁cries  ▁Enhance 内关 愈烈 搁置  ▁tranquil 新枝 瞳孔 童装 退热  日日夜  被公安机关  ▁catcher 哑巴 来世 订制! 阳朔! 霉变"  出事了"  本科毕业#  通俗易懂的# ikhail$ ilever$ picked%  ▁Veronica% 内功& 风浪&  众生的'  做手术'  的疫情防控( 新办( 浮夸)  元纸币) rbwx*  商场超市*  是正常的+  ▁emit+ 小葱, 积案, 野猪-  有大雨-  的乘客.  待在家里. 净空/ 组和/ 高德0  依法治0  在面试1 downs1  ▁Jinping2  ▁Doctrine2 国华3 代扣3 外号4 茱萸4 铠甲5 隶书5  可转赠6  很多父母6 React7  行政区域内7  管委会副主任8  masters8  ▁Moreno9  ▁cereal9 梅兰: 求证: 看人; 集齐; 食补< CALL<  人民健康=  民生保障=  考试资格>  ▁Fiat>  ▁Hemp? 染料?  安装在@  杀伤力@  商家电话A  最新发布A  ▁PipeB  ▁RathB  ▁adamC  不饱和脂肪酸C  ▁NursesD  ▁fearfulD  ▁italianE  ▁mustardE  多样化的F sheetsF  ▁FayetteG  ▁annoyedG  ▁SubscriptionH  就是将H  是必然I  区域内的I  客运班线J  ▁NathJ  ongyangK  ▁DevelopedK  　L  第六条L  ▁CarbM  ▁noticingM  ▁courseworkN 去想N 咕噜O 扬长O aszP  城市里P  层面上Q  高贵的Q  冲刺阶段R  改变不了R  千万别错过S ChromeS voyT  招标人T  来临时U  学习成果U  ▁externosV  ▁ByzantineV  ▁SpringsteenW утW 打碎X 拖地X 疾患Y 纺布Y 驻外Z  下列条件Z blown[ 所趋[ 河的\  会慢慢\  Collins]  ▁Regions]  ▁uniqueness^  的承诺^ ▁PCR_  经过一番_  近期免冠` Tokyo` 决不a 后感a 断绝b  setTextb 强心c  和今后c ricksd  ▁Bournd 辞旧e  为家乡e  感染性f  逃不过f  强调的是g  点击上图g  餐饮单位h  ▁Syndh 先要i 原厂i 生锈j  抗压能力j Sharesk  行业主管部门k 抹胸l  立即报名l  考公务员m  让同学们m  ▁Charitiesn 贵气n  依赖性o ▁Noso  ▁diversionp  ▁excursionp  Palestinianq 跃进q  控制了r  的专家r  规模和s  integers 正派t olomt  ۥۛۛ۟u  ▁libertu  ▁victoriousv  优质课v  比之前w  营口市w Hannahx  本次公开招聘x 贫寒y  不要怕y  副总统z  损害的z  这无疑{  高压态势{ Paint|  ▁Degrees|  ▁convened} 注解} llll~ actus~  ▁airplanes 笔名  成本高 ▁Pty  没有其他 Tiger eners  ▁clam  ▁Rifle  ▁pricey  parameter Ul 技校  您的孩子  ▁Ally  ▁STAT 凯歌 德勒 抹茶 蛮腰 霍尔  的初心  随时关注  高考期间  ۥۛۛ۟ۙۙ  ▁marvelous 冲向 理顺  到你的  ▁Lankan  ▁Quentin  ▁populous  ▁paranormal 子牙 瓦尔  ▁Peck 三小 尖子  公告称  需要对  ▁multic  Vancouver  ▁irresist 常理 首条  第一类 ▁Dru  高品质的  ▁monk  ▁shutter 坐垫 撤职 跑者 釉质 iris ▁vod  重拳出击 mandu  ▁Frog  ▁Lydia 李连  生活是  第四章  劳动争议  ▁Jenna 文帝  通过这些  随机抽取  ▁Tune  区人民医院  ▁impeccable  ▁replacements 一齐 泽县 纽扣  与年干 ▁XVI  ▁Libert 了啊  与月干  集灭道  国家工作人员 闹事 黑米  并开始  楞严经  都能在  总体要求  第五单元 Adult  民办幼儿园  ▁Agreements  registration 实锤  与激情 decode  ▁Rookie  ▁biomark 初学 没少 的信 肉品 详尽  古人云  土壤中  维多利亚  ▁Exploring 的各  冤枉钱  这个家  的好去处  隔着屏幕  ▁Lama  ▁infantry 沂水 海藻 附录  功德无量 名目 装上  不得将 DATE  以免影响  机构改革  齐抓共管  要严格执行  ▁colder 但能 ▁(_  富起来 ▁'';  技术规范  未来几天  ▁Mund  ▁stam¦  ▁workings¦ 是心æ 破局æ 而论Ħ  若出现Ħ ▁CLEŦ  帮助他人Ŧ  ▁auspƦ 薯条Ʀ  如何避免Ǧ  如果对方Ǧ  ▁LimbȦ  ▁hurdleȦ  ▁cremationɦ 右旗ɦ 打起ʦ 黄叶ʦ WEB˦  买房人˦ ▁afț  风险挑战̦  正在进行中ͦ  achelorsͦ  ▁replicationΦ 尼迪Φ  只要一Ϧ  政策落实Ϧ  ▁phonЦ MotionЦ 皮薄Ѧ 跳过Ѧ beiҦ  粮食和Ҧ  肩负着Ӧ ▁ElkӦ 发福Ԧ 救国Ԧ 鲸鱼զ  刘强东զ  物体表面֦  软件工程֦  ▁clotצ 风月צ  多维度ئ  道出了ئ  异性朋友٦ useppe٦  撕心裂肺ڦ  ▁Hammڦ  invalidۦ 异国ۦ 江岸ܦ 着凉ܦ  工作力度ݦ  疑难杂症ݦ  ▁curedަ  ▁Feelingަ  ▁thereinߦ  向美国ߦ ogyn 四家 岗村  严厉的  很美好 ▁ker  ▁Rode  ▁Conan  ▁collage  ▁thicker 管钱 pox  尽收眼底  心得体会  服务内容  ▁vivo  ۥۛۛ۟ۙۙۚ  ▁messenger 榴弹  等各项工作 主料 平分  教育教学工作  ▁Whitman 导电  非常丰富  ▁Chill  ▁ку Brun  手机银行  角度出发  名密切接触者 宋某 懵了 选岗  组织人员  ▁wifi  ▁encoded  Namespace 恩赐 说句 iena  波罗揭谛 solid  ▁André  ▁Normally HIP Chap  新冠疫情的  偏头痛  制作人  爆米花  在你身边  ▁Witt  为数不多的  ▁decom  sequence  ▁portraying 李氏  分钟前  支付了  看不惯 LIVE fant  血本无归 大马  ▁goof  在这段时间  ▁farmhouse 无糖  很快就能  直接关系  dirname  ▁Appell  ▁sermon 邻国  与控制  公司对  就赶紧 Fake Nash  ▁parl  awesome  ▁Surgical 外层  请保持  国药集团  风华正茂  Investment 右腿 宅地 汇成 红岩 触控  支付给 胭脂 贵安  个社区  把我的  来学习  连续性  必须具备  ▁Controvers 紫云 韩式  新机遇  ▁rever  Patient  ▁Chesapeake 出其 宽厚 ushan  部门负责人 Warren  ▁goodies ‖　  涉嫌受贿  美不胜收  自觉抵制  身体健康状况 ён 悉数 诚然  僧揭谛  岁月里  ▁Libr Warner  ▁Schiff  ▁golfers  ▁Blackburn 散结!  就足以!  ▁gathers" 悬殊" 清雅#  是无等等#  集体土地$ issued$  纪检监察干部%  highest%  ▁advisable& 时珍&  有限公司总经理'  ▁meaningless' 刺鼻( 들이(  所处的)  的检测)  第二剂*  有效解决*  ▁Streng+  ▁nesting+  ▁Injuries, 吩咐, 晚归- 轮船-  队伍的.  ▁seventeen. 线网/ hands/  Clearly0 ▁IU0  根据自己1  联系在一起1  ▁serie2 民房2 菌落3  资金链3 ROUND4 paths4  ▁carnival5 时是5 零钱6  二三十6  守护好7  收藏家7  无法满足8  的重要作用8  ▁layoffs9  ▁Generator9  ▁flowering: ▁IDE:  圆满落幕;  茫茫人海;  ▁kidding< 刷子< 报仇= quo= ubu>  黑暗中>  战时状态?  ▁commended? 博罗@ 面广@  多注意A  项目组A  人生中的B  公司员工B  ▁QingC  ▁RadarC 主委D 沉沦D 自提E 麻省E  从北京F  宝妈们F paraG  我们的心G  ▁SHOWH  ▁TacomaH  ▁TelegramI 吐司I 导报J 规上J zarK FerrK  ▁FailL  ▁AdviserL 利己M CMSM  捐款捐物N  ▁tunaN angeloO 怒气O HOSTP minsP ولQ 奉化Q 隔热R 雷区R ISCS  计算器S  退换货T annoT  他们可以U  能除一切苦U  ▁plottingV  ▁exaggeratedV 死心W 毛肚W 舍友X  偏向于X  细菌和Y  扶贫车间Y  ▁LoisZ  ▁glossyZ  borghini[ 上做[ 对岸\ 开足\  第一任]  ▁Wuhan]  广大干部群众^  identity^ 双床_  的外卖_  及其自动化`  ▁Clive`  ▁Entrepreneursa 素菜a Sidb ollsb  ▁Fengc  ▁boltsc  ▁Adoptiond  情况看d ▁Aime  为您服务e  最先进的f  ▁wateringf  ▁steadfastg  ▁modernizationg  ▁hostilityh  ▁placementsh  ▁downloadablei 名酒i  之称的j  于是便j  等产业k  ▁latitudek 轮机l mixl  比一天m  行动的m $("#n  波罗僧揭谛n  ▁managerialo 功耗o 御寒p 鼻咽p  亦无得q  艺术作品q 　>r 折断r  要经常s  一个时期s  将于今年t  重新开放t  ▁Chernu  ▁contrastsu 众望v  带动下v ▁BELw  千百年来w  揭谛揭谛x  近些年来x  ▁Barthy  ▁Glossy  ▁expelledz 天不z 手速{ 砸了{ 赢取| assign|  Advisor}  ▁louder} 内控~  ▁чи~  包含着 ▁Osh  ▁Juice  ▁Lough  ▁velvet 教具 永胜  提交了  汕尾市 lung ulia  美好的未来  第十三届全国 和村 太短 差旅 群号 逢年  级新生 anyl  ▁distractions  ▁whistleblower  犯罪案件  若干措施  ▁Covers 下笔 私利 铺子  南北朝  和老公  滑雪场 hear  无挂碍故  科学防治 igham  实际控制人 单边 搜罗 流离  带给我们 omnia  ▁workflows  回去了 ichuan 烤烟 裂变 sched 借力 敬重 正统  文在寅  自建房 vict  新冠灭活疫苗 告示 大禹 棒了 眼影 mud  放松心情 三孩 周公  微信的  等综合  组织生活会  Enlaces  ▁concentrating 达斯  ▁Shack  ▁soulful  ▁markings 牌的  开空调  应急物资 Budget  也相对  心理的 ▁Sug  一点一滴  严格管控  社会主义制度 + $点击下方空白区域查看答案 ßen  在这条 三局  是假的 ▁ket  有这么多  被困人员  ▁miracles 亏虚 随波  定制的  中学业水平考试 兰陵  晦涩难懂  ▁Moroccan 左旗 巨石 找房 染上 银奖  意识界  我们与 auts orte  ▁Controls  ▁provisional 凉凉 受托 娇嫩 族谱 西夏  后及时  从古至今  英语成绩 Heavy  ▁FedEx 四环 ESP  或依据  暴露在  即说咒曰  ▁presses 均须 潜江  其实施  无无明  ▁Zika  问题进行了  ▁BLACK 出嫁 多份 弃疾 虚幻§ 蜂巢§ xsdç  一股脑ç  她一直ħ janaħ ppardŧ tailedŧ  InitiallyƧ  ▁StarringƧ $_ǧ 周身ǧ 征用ȧ ILSȧ Vitɧ альɧ  在各大ʧ  有一座ʧ  颜悦色˧  是大神咒˧ codeş ennalȩ  ▁primaͧ  Employmentͧ qnΧ  一笔一Χ  血管壁ϧ  即将开启ϧ  是无上咒Ч  ▁denimЧ 见得ѧ  不完美ѧ  大概有ҧ  ▁guerҧ igginsӧ  ▁aspireӧ 想用ԧ 武鸣ԧ 羞愧է  让客户է  ▁Shed֧  attacks֧  ▁wurdenק  ▁diminishק  发放了ا  ▁isotا  作者和来源٧  ▁plead٧  ▁Societiesڧ  ▁initiationڧ отۧ  细菌的ۧ ▁Pryܧ  ▁liquidsܧ  Customersݧ 普贤ݧ  发布编辑ާ 学情ާ 榨汁ߧ  危重型ߧ  宗地块  时刻表  看得出  ▁Dani  是无等等咒  ▁benign  ▁Snowden  ▁chronicles  环境与  这个专业 Fried  ▁goodwill  ▁surrendered 商厦 开衫  好几次  到底是谁  必须要有  根本不是  社区治理  赏心悦目  ▁scout  ▁Stations 也用 受孕 回旋 谁在 青鸟  协助下 Shot rios 末期 道菜 钻孔 ▁pu  埋头苦干  进行查询 传家 纹饰 OPE  前半生 纯电 走后  等级考试  ▁Buyer  artments  producing 财年  无明尽 ▁DOC  土地流转  活动结束  ▁Benny  乱占耕地建房  ▁positives 再以 大觉 黄芩 ▁Yi  无声的  资讯网  中心副主任  univers  ▁Telecommunications  近义词  ▁Faul  ▁shaken 十岁 合家  家庭和  就是以 heny ▁Psy  可以添加  活性物质 Prosec 勇挑 线与 躺下  ▁Detection 佩奇 统称  以无所得 storms  知识产权保护  investment  ▁unnoticed 则要 到他 是件 黎巴 Kirk  发表声明 三色 任丘 炸裂 该区 起早  培训内容  社会信用  ▁Lark  ▁winger  ▁Atkinson  ▁ancestor " 请联系我们进行删除  净水器  工作开展情况  ▁mosquitoes 无望 百变  很幸福  的进行  健康检查  所提供的  皮肤干燥 wrong ARN  ▁Sain  ▁locating 功用 放养 STA kah  诞生的  介入调查  设备安装  ▁inactive  ▁fingertips  ོད་  领悟力  糖尿病人!  自己的错误!  菩提萨婆诃" itizen" 交待# 减震# 老鹰$  当然要$  不是没有%  畜牧兽医%  ▁Leop&  亦无无明尽&  无智亦无得' 她那' gzh(  月正式(  记忆的) Gate)  产业发展的*  ▁chili*  ▁localized+ 宝箱+ 观止, 还贷,  十全十-  商务厅-  发扬光大.  第三阶段. {'/ 勃起/ 绝密0 雅典0 ewe1  九一八1  警方通报2  ▁PRESS2  ▁Sinatra3  ▁blasted3 双师4 牛腩4 瘦腿5  消失在5 ▁Esq6  ▁classy6  ▁transp7 Veh7  锂离子8 tale8 ▁HDR9  周边配套9  中国共产党建党:  Shanghai:  ▁separat; 反手;  给他的<  医学检验<  结婚生子=  ▁noir=  的无症状感染者> " 中华民族共同体意识>  ▁deficiencies? 首末?  执行公务@  ▁bully@ 私藏A ▁JFKA  网站发布B colorsB  般若波罗蜜多咒C  学而思C  职场人D  随州市D  继续进行E quiteE 前两F 所剩F  高年级G  是大明咒G  空中无色H shakeH  ▁MascI  ▁SoutI  乃至无意识界J  ▁grievingJ  小白鞋K  服务好K  可见一斑L  感谢阅读L  OntarioM  ▁expertlyM }_N 可亲N 黄沙O  位置上O  保安全P  老死尽P elfthQ  ▁enumQ  无苦集灭道R  社会实践活动R  InteriorS 刀剑S 校舍T EXPECTT 蔡甸U 献身U  民俗文化V 文脉V 来吃W 菜农W 配乐X ibbX  很多学生Y  以无所得故Y  ▁juicesZ  无眼耳鼻舌身意Z  ▁Corvette[  Bazinfandel[ 自重\  而这款\  ▁jars]  到我们]  收购价格^  问题解决^  ▁Rowe_  ▁affluent_ 面和`  科目一`  ▁canva  ▁sagea  报名成功后b  ▁Smoothb  ▁lineagec 乐天c 却把d 徒劳d 案由e 票务e  听证会f ccolif  ▁Staceyg 取之g 旭东h 请他h втоi coordi  亦无老死尽j  ▁memorandumj 戏水k REMk  懂得如何l radesl  ▁anarchm  ▁Barclaysm 亩产n 浦发n  而是你o  自己最o  记忆中的p  无色声香味触法p  ▁incorporationq 到下q 荣登r 送电r Greeks  Princesss  ▁dragonst 强省t 跨区u 过海u  愈来愈v  寻衅滋事罪v  更多的时间w  ▁titaniumw 样貌x Dashx 念诵y 陆生y  年成立z  粉色的z  技术指南{  ▁gadget{  中国社会科学院|  Wikipedia|  ▁Supplier}  ▁absentee}  ▁firewall~ 捂住~  无眼界  研究型 ▁Hin  ▁mats 对焦  持续增长  ▁europe 变软  ▁Flames  并不承担  消息人士  ྱི་  卡路里 ▁Smy  两种方式  虽然已经  Headers  ▁skipped  公开招聘工作人员  ▁experiential 无生 湾镇 Aven  ▁Noir  中提取  的大力  的路径  怎么样呢 Megan 吸湿 雾气  双眼皮  它能够  转移至 orns  投诉电话  ▁VMware  ▁challenger 孜然 看管 石嘴 ASP iton  国内知名  ▁Moll 拉力 收场  语言学 Pont  合作电话  ▁york Athlet  ▁Cutting 律法  在线答疑  ▁Glam  无受想行识 % 党史学习教育动员大会  标本兼治 卡顿 总院 拼盘 HRC  好文章  全新一代  授权发布  ▁intuition 入骨 连年  因为他的 atern Secure  ▁stemming  ▁deception  ▁decidedly  ▁Foundations  ▁superficial 庄浪 绣花  国家和省  负责的态度  月光族 Pink  才能保证  乃至无老死  ▁Bosch  房地产开发企业  到多云  大约是  学业水平  小时值班  说的那样  ▁Buckley  ▁explosives 包间 听信 珍视 轻声 Ring assa  吸毒人员  风雨兼程  ▁administering  因业务  能通过  十二指肠  ▁multilateral 大圣 无谓  ▁Stamford  ▁royalties 寻物 越强  ……,  他甚至  ▁Marley  农信社  课题研究  故知般若波罗蜜多 enario 她为 撞倒 盗版  了这一  而美国  及时解决  初中毕业生 混用 罗曼 ptune 图强  更新中  ▁Darling 到哪 吉日¨ 祖坟¨ 高丽è  在外观è  的职位Ĩ " 故说般若波罗蜜多咒Ĩ  流行病Ũ olognaŨ  ▁alertedƨ  ▁brushedƨ  阶级的Ǩ  服务电话Ǩ  ▁LeanȨ  是故空中无色Ȩ  请联系本平台ɨ 三九ɨ 低脂ʨ  模拟考试ʨ  版权归原创者所有˨  ▁dominates˨ 踩雷̨ jug̨  ▁Stimͨ  亦乐乎ͨ  委常委Ψ  为什么不能Ψ  ▁clownϨ  ▁LandmarkϨ 勇猛Ш 春雷Ш 果品Ѩ OliverѨ  CertificateҨ 胃胀Ҩ 赶忙Ө 需持Ө  与城市Ԩ  确认的Ԩ  也不会有ը  居住用地ը 布控֨ 破壁֨ PSCר  势在必ר  已启动ب  已有多ب  逾期不٨  家人的健康٨ elightڨ  ▁Exxonڨ  ▁ciderۨ " 长按识别二维码了解ۨ  ▁inadvertentlyܨ 之志ܨ 将此ݨ ▁ACEݨ  problemި  ▁calmingި räߨ 着地ߨ ▁ni  始终以  楼梯间  igmatic 头戴 ▁cv  得更加  结束的  要采取  ▁slipping  ▁deforestation 体罚 追思 üss  金额的  反应过来 带我 战备 点在  官网上  ▁Soup 先试 刘伟 鲁豫  大眼睛  ▁familjen 取而 红光 蔗糖  也可能会  拜登政府 anooga  ▁Spend  ▁archived  年全市  政府网  的诊断  空间大  还含有 signed  ▁lauded  ▁compass  ▁pilgrims 卓绝 揣摩  是正确的  ▁STATE  ▁suitc  民事行为能力  ▁excursions  dependencies 决堤 墓葬 ewski  ▁Tudor  根据工作需要  ▁Accord 亲王 党徽 雏形  可促进  打乱了  类项目  ▁riffs 三花  有点儿  杨先生  亲密接触  师范学校 之约 情味 盆中 西溪 走吧  官方回应  ▁tamb  ▁homeschool 踏着  社会中  昨天晚上  资金来源 angan Rating 博拉 盯上  关闭了  百年未有 uddin Donate Zeneca  ▁Nadal  ▁virtues 经书  等职务  另一侧  基础和  迫在眉  ▁cherish  transport  ▁intersect 土鸡 禅修 金价  团总支 Lind  到底有没有  的不良反应  中新网  家族的  希望我们  还没来得及  ▁sushi  ▁spawned 写诗 谈了  既没有  注定了 Adapt  意识到自己  JavaScript —“  在德国 Trip isty  老房子  一体化的  ▁Remarks  ▁deployments 霜降  林心如  要经过! ASON!  体育公园" 妻儿" 日料# 最热# 白塔$ Joel$  非法拘禁% itian% 同乡& 巨资& ▁ay'  ▁хо'  的村民(  附近有(  ‍‍‍‍)  ▁Ones) 三岁* 路店* 里去+  不是梦+  投资公司, ellij, Winner-  ▁Albion-  ▁fruition.  ▁anticipating. 剥削/ 龙马/  不聚会0  论坛上0  生理结构1 米粒1 累活2  ◐◑　2 Mine3 yong3  你只需要4  工作措施4  面试人选5 rupal5  ▁Learned6  ▁chancellor6  视为认同7 回荡7 Tan8 ۛۚۚ8  之上的9  百度云9  ۥۛۛۚۚ: enties:  ▁Angular;  inclusive; 发个< ▁EMS<  消防车通道=  othelioma= 匹马> 大行> том?  的外形?  荣誉感@  都在关注@  ▁navigA 地缘A ISTSB dishB  整个人都C  原作者联系我们C 我镇D  和北京D  我想说E  也非常的E  观看直播F GardenF  ▁adornedG  ▁drunkenG  ▁ConceptsH 买方H 诙谐I  更便捷I  ▁IberJ  为进一步做好J  被判处有期徒刑K 愈演K 极地L FIXL  居民楼M uvreM  ▁glyphN  MeasureN  ▁escapesO  ▁outsiderO  ▁visualizeP ۥۚP 练出Q 花苑Q 课下R ۙۙۙR  天眼查S multiS  ۥۚۙۙۙT  实施乡村振兴战略T  ▁upbringingU 中烟U UNEV keyesV  ▁YukonW  ▁HanoverW  ▁downrightX 塌方X 嫩肤Y 张勇Y  了出去Z  处理方法Z  更有甚者[  ▁baseman[  ▁Importance\ 南村\ 陈毅]  经验不限]  青春力量^  ۥۛۛ۟ۙۙۚۥۚۙۙۙ^ 蔡某_ Carm_ ▁cue`  Division`  ۥۛۛۚۚۛ۟ۡa 七中a 声带b 好汉b 白班c  一定可以c  从而引起d  全国两会d  检查发现e  每时每刻e  ▁Cheneyf  ▁Becomingf  ▁Specialtyg  ▁exhibitorsg 富美h 每桌h  余杭区i  夜以继i  有力地j  人才交流j 情系k 秦并k 胸椎l 辽阳l  的深度m ▁WTOm  报警求助n  肺炎疫情n  这些信息o  ▁Shineo pipp  刘晓庆p  我国在q  被纳入q  购物车r  急性发作r  蛋白质含量s  ▁faires 而终t  去尝试t  ▁vibesu  ▁niecesu  此次违建全部拆除v ! ۥۛۛ۟ۙۙۚۥۚۙۙۙۚv 客座w 眼福w welx  专科生x  ▁Clairy 物和y 通电z  ▁Nicoz  ▁homer{ 常喝{ 禅宗| 赖氏|  文件和}  校长的}  要完善~  辉煌的~ Sant  播出时间 ifeng 你越  拍了拍 opts  ▁collagen  ▁encompass + $应对新型冠状病毒肺炎疫情 空余  有多高  的只是 fork  表达自己  ▁hierarch 中粮 ▁SIG  进行评估  指明了方向  warning  ▁Collegiate  ▁intensified 曾某 真卿  与月支  ▁resisted  ▁ecommerce 在学 Utah  为什么我  停业整顿  ▁trivia  ▁fossils :* 七七 珊珊 睡醒  主动与  的尴尬  ▁culminating 史册 所爱 撑着 险些  唯一能  立即向  公交集团  ▁ions  ▁overshadow 风土 gran  实际出发  指数基金  ▁nord  ▁Anand  ▁fiddle  ▁novice 同频  检验研究院  缴纳社会保险  但她的 fers  ▁mediums  ▁psychologists 连城 酒楼  已发布  的精华  配得上  即可报名 olian 桔梗 调成 ▁REM  团委书记  ▁Kear  ▁enlarged  报名与  的代谢 evol nery  中华传统文化  ▁preventative 截面 渗漏  一直有  效率和  非法营运  ▁CARE  ▁thou  ▁pamph  vehicle  ▁backstage  出乎意料  来都来了  的事情了  ▁bingo 半数 肌底  一厨一卫  刑警大队  perhaps 曹某 lun  承受的  真正地  吃瓜群众  ▁edits  ▁Failed 烙印 Pix  ▁би  仅提供 erta ▁Sno  卫生监督  ▁Setup  ▁replen 章法 衰落 西施  了一笔  被称作  ▁Casual  generator  并不少  不懈奋斗  在了一起 危废 脸庞 ▁\"  向群众  紫禁城  自然也  还在为©  ▁stacks©  ▁Haitiané  ▁interpersonalé 射箭ĩ 牢骚ĩ  一个重要ũ  厉行节约ũ hunderƩ matterƩ 午时ǩ 可说ǩ 这五ȩ  深交所ȩ  的很多ɩ  让员工ɩ  陪伴你ʩ  暑假期间ʩ 从她˩ 变数˩ 课业̩  心心念念̩  ▁Marjͩ 丢分ͩ 清肝Ω 邓某Ω uldϩ  医疗险ϩ syntЩ  细菌感染Щ ivismѩ symbolѩ 御用ҩ  个以上ҩ  按比例ө  胶州市ө  这个大ԩ  热爱祖国ԩ  ▁Trainerթ 台灯թ 启超֩ 邮费֩ Aquש  工作总ש  震惊了ة ▁GISة  从即日起٩  法律依据٩  考核结果ک 庆区ک 魔芋۩  难问题۩ Romeܩ 撬动ܩ 斯塔ݩ  尽自己ݩ  异性缘ީ Nealީ  ▁Yatesߩ  solvingߩ hound  ▁Bezos  ▁avocado 受此 正阳 黄淮 Ald  写留言  尤其对  曝光台 ARDS  空间上的  组织部长  质量监督  迫在眉睫  ethical  instead 风雅  接收到  根据相关 SWORD  临床诊断  ▁Summers  processor  ▁collaborator 尚德  发起的  第三款  品牌价值 古语 新泰 Cli  这表明 serve  ▁motifs  ▁emulate 尊者 就又 王世 百战  给孩子看看 iviera  ▁Semin  ▁vanished 归零  正反面  认识和  ▁necessities 土匪  在最近  不是很好  辖区居民  ▁abound  ▁stitch  received 1 *ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡ 见惯  她老公 ▁Hex ▁Lug  再次发生  注重分享  贴心服务  领先地位  ▁duet locale 举足 zeit  形成合力  沁人心脾  ▁Breed  ▁Dealer  ▁honeymoon  例患者  尽管是 以太 吃鱼 石英  理论武装  的医护人员  ▁eviction  不承认 icki  ۥۛ۟ۡ  小编推荐  第十一届 Zhang optic  ▁GROUP  ▁Supplies 同声 赵雅  深情的  烧开后 Wiki  处理结果  网上办理  ▁overload Dj 往南 自曝  吸收了  百花齐  作为家长  设施齐全 donald ignore  ▁Messages 熊熊 钱多 ▁OB  的思考  ▁frog  ▁quart  ▁iceberg nr 冯某 遇水  但为了  各县市区  ▁extremes  ▁symbolism  就变成  耽误了  ▁Ships  ▁dwarf  checkbox  ▁Keystone  ▁Territories ór 顾家! 黏液!  我必须"  户人家"  的男生#  风向标#  ▁Drinking$  ▁Vertical$  ▁mythical%  ▁endeavour% alm&  现在还&  等优势' wine'  带着家人(  装修风格(  ▁pearl)  ▁swamp)  ▁Jesuit*  ▁Limits* 海港+ ERP+  每天至少,  美好未来,  ▁Panc-  详情请扫二维码-  中下游. ▁Pes.  知识储备/  ▁praises/ 采茶0 飞龙0  辛弃疾1 ixia1  蓝色预警2  技能和2  ▁repr3  工作人员和3  让人民群众4  Executor4 舞动5  主讲老师5 behind6  ▁Vulner6  他不是7 ▁tad7  Bloomberg8 欢度8 炎等9 研报9 ]),:  保留着: ▁Wah;  中国历史;  是不够的<  ▁Schro<  ▁dictatorship= 升迁= 疟疾>  接二连三> ronomy? 专治? aber@  极端天气@  第二天早上A pillarA 之列B 思域B  副主编C  政治家C  武装部D  毕业院校D  我们希望在这里E  IncludesE 克罗F  将会是F  相关证件G  究竟涅盘G  越来越严重H RouterH 狂妄I  进来的I  韶关市J  原因在于J  ▁SheaK SeeingK 所报L  承担相应的L  ▁McDonM  ExamplesM  ▁ShieldsN  努力下N  ▁phishingO 宁都O 母舰P  开放日P  比较多的Q  监控设备Q  一代又一代R  ActionsR  ▁BombayS  ▁MessiahS  emorationT  ▁vigilantT 垂范U 食药U  百姓的V  数学思维V  ▁SarasotaW  ▁specifiesW NamX sirX  日傍晚Y  课程详情Y  产业的发展Z  对他们来说Z 村官[ Eli[ kor\  总干事\  本周六]  确权登记]  的时候可以^  ▁Risks^  Turkish_  ▁Pandora_  替代品`  学历教育` Sparka  ▁zeala  ▁infusionb  ▁honorableb 中班c Frac  感恩的心d  收费项目d  这个季节e  ▁cardie  ▁Rubberf  ▁caucusf  ▁spinachg  ▁fabricatedg 榆树h 车市h TCLi  完善了i  农夫山泉j  按照自己的j  ▁Pearcek 拉丝k 放油l  再也不怕l geantm 五中m 动产n 卸任n 爱笑o  莫里森o arakp  后疫情时代p  Douglasq  ▁Cologneq  ▁revokedr  ▁electrolyr  ▁notoriouslys 县内s  人都是t  ▁loret  ▁McGrawu 刘建u 打滑v  专题党课v  具体预报w  学历认证w  至今已有x rettex  两个人在一起y  ▁Lahorey 搜查z ▁DDz  订阅课程{ ticket{  ▁sands|  ▁inhuman| 一过} 立的} Axis~  也不一定~  能够通过  为他人谋取利益  Newswire  ▁Optical  ▁brushes  ▁Applicants 销往  ▁Derrick  ▁puppies  ▁trucking  ▁categorized pra  国际标准  精心策划 PRING mable  ▁Amir 不疑 摘掉  生存的 GitHub 不卖 副高 终成  ▁mediocre ół 本条 SAM uve yne  渐行渐远  沟通能力强  的大力支持下  Probably 人往 断续  在室内  而引起  陶渊明  认认真真  ▁digs  ▁rund  believe  主持会议并讲话  ▁geology  ▁offended  ▁extraordinarily 名句 射精 用纸 Pros  善于沟通  ▁Downing  ▁Woodward 东东 乐居  ▁Loyal  ▁Twain  ▁confuse  ▁Incident 却步  ▁pued  ▁refreshed  ▁substrate 售完 宋庆 精锐 解渴  新思想  招聘活动  imicrobial  ▁Botanical 首套 ▁Hwy 迸发  一些东西  快的速度  逢年过节  对此声明的  akening  ▁spurred —( 研招 苦于  的表面 boats  全国范围内  ▁administrations 按原 超凡 酸软  assisted 集邮  系列报道  ▁dich  ▁Atmospheric 挣脱 献花 问心  这需要  一开始就 Invent ▁Ré  新问题  的民警  生活需要  ▁Recipes 真如  亲身体验  ▁Rated  ▁bible  ▁bending 渴求  哈哈大笑  运载火箭  ▁omitted 而远  凤凰网  红色资源  ▁DEFAULT  ▁thinner 高声  凡尔赛  宣传部长 之水 平复 水花 漠视  一二三  护理学  的今天 idis  新媒体编辑  ▁Woodstock  可降解ª  责编丨ª ▁Dagê  外出打工ê homesĪ thorneĪ  ▁bulletinŪ  ▁commutersŪ 两栖ƪ 于某ƪ 多斤Ǫ 终归Ǫ 阳城Ȫ  了无数Ȫ  全部的ɪ  原则和ɪ  选择和ʪ covidʪ  ▁Hulk˪ Static˪  ▁lacrosse̪ mv̪  清晰可见ͪ  所以一定要ͪ ACTIONΪ  ▁FloraΪ  ▁JewelϪ  携带本人身份证Ϫ 9 2ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡۥۛ۟ۡЪ 也像Ъ 从这Ѫ 左腿Ѫ liumҪ  解题思路Ҫ esslerӪ 多芬Ӫ  冯绍峰Ԫ DemoԪ  getValueժ  ▁terrestrialժ 韩某֪  后来在֪  无与伦ת  村集体经济ת  ▁ignoresت  ▁Emotionalت  Interestingly٪ 广袤٪  不推荐ڪ  急难愁ڪ  捐赠的۪  艺考生۪ boltܪ eyesܪ  DEFAULTݪ  asionallyݪ 大港ު 战神ު 斩断ߪ 越吃ߪ 顶多  寄宿制  有人能  点开始  的男性  主导产业  的重要手段  variant  别具一格  只有一种  ▁Gail  inctions  ▁Guarantee  ▁meticulous 尊师 环岛 诱饵  就能让  笑起来  肯定了  充分准备  可以轻松  小区门口  政纪处分  一定程度的  ounsell  ▁Intervention  ▁motivational ▁ä  主题曲  客观题  参加调研  弹道导弹  ▁cannon  ▁Bengaluru 包揽 吃穿  ▁же  项目总  ▁Desire  ▁meticulously 和它  愈演愈烈  精心挑选  ▁Decisions ▁AH  第五条  还是蛮  适宜的  每天都是  考核合格的  戴安娜  旷神怡  但是对 Nm 收容  开始时 ▁dif  截然不同 ateau  ▁Torn  系好安全带  ▁Finale  redients  ▁repercussions Mars cult  夫妻感情  不代表本号  你喜欢的人  ▁sanit  Effective 母的 Coffee  ▁fractured 上亿  主动到  环境监测 yahoo  ▁VERY  人事部  和帮助  教导员  暴露出  深圳的  之大变局  审议通过  更多推荐  ▁Hundred  ▁durante ▁KO  聘用制  运动中  ▁его  润肠通便  ▁clientele  芭蕾舞 ITLE  昨天下午  ▁Surely  ▁recorder  ▁Polytechnic 全车 ▁PSA  减少不必要的  Visitors % 对此声明的最终解释权 广德 绝世 ▁Sv  是相当 ocular  ▁burglary  ▁Intermediate  儿时的  明天起  人事档案 igree  ського  ▁juicy Ric opf  分布图  日后的 Flat oxia  到了晚上  Scottish  ▁drowning  购买力  赵立坚! ▁Tek!  狠抓落实"  ▁duly"  消防工程师#  ▁ozone#  ▁approving$ 几周$ 张北%  可惜了%  神经元&  如果只是&  被认定为' Innov'  ▁overcrow(  ▁subpoena(  ▁guideline) 民谣) 滨区* 环湖* 赫然+ rather+  ▁astronomers,  无异于,  进行修改-  Thinking-  ▁Hendrix.  ▁yourselves. 广义/ 阿婆/ 顶上0 TEM0  大世界1  前三季度1  心理状态2  washing2  ▁fractures3 便后3 李红4 lea4  和活动5 rath5  ▁Reign6  ▁tutoring6  ▁tightening7 能赚7 邹平8  地铁口8  不到一个月9  充足的睡眠9 东来: 聚居:  核电站;  的视觉;  任何人都<  实打实的<  适应能力=  Various=  收获的> ▁ESA> liked?  ▁spikes? 大件@ ▁FP@  是父母A  而此时A  穿在身上B  不符的B  岗位条件C  超市购物C 她要D 木偶D 海王E 牧原E 编队F  毕业了F ovskG  不知道的G  ▁OlympiaH  ▁LockheedH 不倦I 助长I 棚前J 罐子J 还请K  原本的K  职工群众L  的相关信息L  市市场监督管理局M 曾为M 腾出N deleN  为了您和O  广大教师O  经验优先P  timestampP 乎其Q 口香Q  早报告R ▁nouR  专题培训S  大江南北S  ▁wormsT  ▁dramatT 震中U  不要超过U OscarV  ▁slabV  chellesW 之境W 维州X  隐藏的X  疫情风险Y  ▁EditionsY  ▁frackingZ 搏斗Z  危险品[  一条街[  幸福指数\  意识到了\  相信很多]  ▁Layer]  ▁Chopra^ LF^  系列课程_  ▁Logger_ 一页`  而这次`  ▁Lovesa 月供a 朵花b  元的价格b  节能环保c  ▁filtc 妈咪d 须经d  就是有e  liftinge  ▁Millennialsf  宣传员f 上杭g ▁Pég  但需要h 元享h ункi  无论您i  中央企业j psonsj  是社会k  相融合k  也在不断l  交流活动l  心情舒畅m  高考作文m  ▁Launchedn  ▁Remembrancen 医改o hypo  分钟以上p  甲状腺癌p  第二剂次q iergeq  ▁Rossir  ▁Preparedr  ▁moderateds  ▁uploadings  古希腊t  监管局t  ▁lodgedu 荷包u Rumv  你们在v  刮目相看w  ▁Schlw 奶瓶x 宿命x ▁Waxy  政治方向y Coastz 奶水z 载着{ 音标{ ▁ai|  为标准|  和客户}  想要去}  林先生~  ▁hoard~  发生关系  自查自纠  具有良好的品行  ▁textured 尿管 手淫  一段时间的  ▁Turtle  Observer 有谁  可增加  年轻时的  ▁parishes 选报 erity  自己的行为  ▁Roller  遵守宪法和法律 疲力 隔天  开始对  日全国  商业中心  ▁OLED  ▁Ingram  不随意  让不少 ▁lui  有序排队  的交通枢纽 1 *应对新冠肺炎疫情工作领导小组 以至 神韵  收视率  有一家  的重视 ▁Hok  免疫力的  督促指导 五线 律动 走读  与河北省  积极协调  自以为是 rites 识人  佟丽娅  无人能  受到党内警告处分  ▁youngster qv 撞到 likes  ▁Emin  ▁stature 扭伤  有时在 ▁Hij  不留死角  ▁bundles  ▁spelled 不染 海神 海航 BAT  名下的  ▁COLL  ▁Suit  重要指示批示精神  立医院  人生最大的  ▁plunged  ▁anterior VV 网恋 ▁ITS  ▁naive  nington  Released 文火  可以自由  市总工会  ▁embroidery 用得 突起  有了它  等媒体  发生争执 停歇 马赛  年建成  给身体  广大家长  情绪激动 ateurs  ▁Compass  ▁suicidal 改观  在推动  跆拳道 anck  ▁Carlton  ▁sacrificed  ▁Evangelical RAL mob  却很少 ▁Arb  随着时间 Tasks 出奇 尿路  很丰富  贵州茅台 老陈  在河南  )　{　  每个学生  考察调研  业主委员会  施工总承包  entiful  ▁Deposit 给水 越做 AMG  chanted  ▁Established  境内外  幸存者  民办实事  ▁Adolf  历史文化名城  iduciary« 并附« 耳垂ë 肺病ë  和业务ī  将他们ī dentū ▁wreū  合理使用ƫ 熟透ƫ 调集ǫ  准妈妈ǫ  要带头ȫ  这门课ȫ  本场比赛ɫ  产品和服务ɫ  发挥了重要作用ʫ 少爷ʫ 童谣˫ 远行˫  免费学̫ ▁sto̫  我的生活ͫ  ▁skinnyͫ 工的Ϋ CBCΫ  疫情下的ϫ 加湿ϫ  学校体育Ы  我们的孩子Ы fkѫ 精编ѫ 都吃ҫ  对记者ҫ  触手可及ӫ  ▁hipsӫ 与该ԫ  等传统ԫ  纺织品ի ▁Aryի  提醒各位֫ geois֫ 春寒׫ 死因׫ 都去ث  手机和电脑ث  持续更新中٫ others٫  中国共产党领导ګ 一程ګ riks۫  上个世纪۫ Medicܫ SpaceXܫ  出院后ݫ  国人的ݫ  的人体ޫ  重特大ޫ anon߫  ▁Polk߫  区委组织部 怀远 裙装 见底 xcb  ▁Breath  已经把  新突破  风雨同舟  ▁ping  ▁Panda  更正来源及作者  ▁physicist % 更正来源及作者或依据 简明 谢家  单元格 Pure ceased  Greater |\ 再送 简述 腾达 药液  很多事  样板房  ▁Clifton  ▁roundup  ▁inhibitor  ▁constrained Abd COR вой  却始终  年轻了 ▁rpm  长城汽车  有关规定执行 千言  missing  ▁survives 三下 受聘 捡拾 掉入  夹杂着  智能网联 antics  ▁Eaton  uttering 鼻祖  切实履行  长江流域  ▁migrated 凶险  新课程  直播时  面对生活  ▁repur  ▁Colombo " 颠覆了几十年的金钱  ▁Corrections UF 桑拿 汉奸  也还是  就让人  定期开展  普遍认为  相关责任  ▁gamble  ▁boasting 张军  优良的  到这个 ▁MPH  ▁mantle Ec 袖口 里木  都是我  顺口溜 均无 是您 极富  在重庆  贺州市 ▁imb  everyone 响亮 晚于 武士 nett ▁ros  有关政策  内容的真实性  aghetti  ▁Fortnite  ▁silicone  ▁evocative 把酒 肥厚 血量  的头发 ▁CEL  急救电话  ▁Kern  接受社会监督  ▁Marino  十九届四中全会 三相 凋零  四边形 reau  ▁disparity 劝你 扛着 蒸煮  却有着  大奖赛  的执行 dling  ▁Trap  服务点  方法就是  四年级上册  著作权人意见  ▁Angelo  ▁dipped 勘测! 海波! 骤然"  运筹帷幄" 亡后# 宁晋# ipro$  身体力行$  这也就是%  之间的矛盾%  党的十九大和&  ▁Lastly&  ▁Xiaomi'  Encoding' /_( 埋葬( 患了)  仙人掌)  违纪违规行为*  ▁acquainted* いて+ дар+  天安门广场,  ▁refin,  ▁Nassau-  knowledge-  ▁Stockton.  Registered. 海选/ 种田/  花都区0  车厘子0  并不承担其他1  ▁Burgess1  Shutterstock2 行云2 靖边3 Cod3  主动投案4 centre4 1 *本平台将更正来源及作者或依据5 排解5  了一辆6  生物钟6  禽流感7  字里行间7  ▁whipped8 不降8 旖旎9 水北9 ▁REG:  ▁stool:  的全新; Ultra;  ▁nude<  ▁Ivory<  ovoltaic= 其烦= 和上> 徇私>  消费税?  真正的朋友?  ▁trainings@  老天爷@  都给你A  不但没有A  ▁RaphaelB  ▁imbalanceB YanC growC  公示期满D  ▁kaleD prettyE  ▁annumE  ▁LagunaF  ▁SolidarityF 手柄G 江都G 면서H  别人家H  势力的I  岁以前I  热情和J  胰腺癌J  不是所有K  互利共赢K  合理利用L  天然植物L  相对应的M  ▁BlastM  ▁regimentN 多场N 槐花O  三年来O  调整和P  北京市海淀区P  ▁combatingQ  南北宽Q  放寒假R  清洗消毒R  ▁emanS  ▁pursuitsS 多劳T jspT vaeU  整体上U addenV  必须先V ▁FaxW  如果你对W  民主党派X 不深X 兰德Y  少数人Y  结合了Z  欢迎来到Z  ▁compiling[ 商行[ 所知\ 扼杀\  从不同] poor]  继续实施^  课程思政^  ▁Triv_  ▁Richie_  ▁Juvenile` 天黑` 小段a 有话a VALUEb  ▁Pathologyb  ▁minimizingc 异乡c 甲乙d 神志d 表征e ▁Meke  随之而来f  ▁sealingf  ▁Anderseng 小兵g 者不h  图片带h  开辟了i  承德市i gansj  Healthyj 对不k  乐山市k  仅作为l  急难愁盼l  ▁ravem  ▁Conorm  Graphicsn  ▁seriousnessn 共体o 找车o 防爆p  培养和p  家中有q  提前退休q тсяr ▁RRr  家庭作业s  担保贷款s  电子数据t oyangt  ▁pacingu  ▁ornamentu 道明v OSHv  家族史w  昌平区w  身上有x  默默奉献x  ▁stabilizey 两房y 浓香z  她觉得z  的热爱{ ▁GUI{  系统工程|  ▁Murdoch| }]} 之策} 末年~ 热能~ 负极  学年度  突破性  学会会员 C <本平台将更正来源及作者或依据著作权人意见 眼睁  大整治  中心学校  心动不如  人职位描述  每一个细节  ▁beacon  ▁destabil  课程学习 的旧  童话故事  奠定了基础  ▁remodel  新标准  月销量  的不适  颜真卿 ▁kam  真的很难  ▁Expand 最常 王亚 粤菜 美业 老爱  和平精英 ocrine  ▁cameo  ▁uplift  与年支  研制出  该怎样 Marine  新冠病毒灭活疫苗  ▁marrying  ▁Stuttgart 房前 载物  不服输  ▁Nell  ▁Goose  ▁nestled 卡友 对号 棉签  得很快  普拉提 endi  接受采访  ▁Staten  selector  可降低  电梯房  可能出现的 雅迪  然后又  轨电车  送外卖  ▁abras  ogenous  就成为了  涉及到的  ▁Finch  ▁ticks  ▁Recess  ▁Gathering  跟大家分享  为大家准备了  ▁Maiden  ▁Cassidy  OutputStream 制茶 卵子 花光 输给 iry  安康码  能够将  ▁Outcomes wc 国度  在目前  年美国  旺财方位  随着我国 chten  ▁tidal  ▁defaults 村貌 没说 cie  市民生  的春节 面带  东西长  松江区  准确性和  紧随其后  olastic  ▁Chorus 追肥 ilant  ▁Brut  ▁marrow  ▁propel  ▁undisclosed 主推 求医  为城市  挨家挨户  ▁distort 上蔡  ▁Giles pec  多见于  ▁stabbing 控告 破土 重演  大吉大利 前倾 原体 宜业  Arsenal  ▁lighthouse 求人 火炮 王雪  的义务  冲上热搜¬  防汛工作¬  这样的男人ì 中以ì 横县Ĭ  也是需要Ĭ  每天早晚Ŭ  课外阅读Ŭ safetyƬ  leadersƬ + $拥有对此声明的最终解释权Ǭ 中网Ǭ 之爱Ȭ 争端Ȭ 周周ɬ  吸取教训ɬ  ▁sporʬ  ▁Decadeʬ  ▁pluggedˬ 磁力ˬ 蛋蛋̬  改造成̬  大家都会ͬ  ▁Guthͬ  ▁Cargoά  ▁soaredά  ▁suppressedϬ  ▁personalizeϬ 匪夷Ь 融汇Ь  大家可能Ѭ  排名第二Ѭ  ▁añosҬ  ▁hallucҬ  ▁bipolarӬ  ▁rendersӬ ( !学历证书电子注册备案表Ԭ  做起了Ԭ  易出现լ  全世界的լ  ▁NYPD֬  安全隐患排查֬ 太祖׬ 磋商׬  英国人ج  获得过ج 养眼٬ 澧县٬ REGڬ  不添加ڬ GRAM۬  走过的路۬ pragmaܬ  Inspectܬ  ▁outingsݬ " 新时代文明实践中心ݬ 从左ެ 从速ެ 玄宗߬  它们是߬  专场招聘会  ▁shoreline  ▁Conservancy  面积达  ▁Climb  ▁shotgun  ▁progressively 彭某  军分区  大智慧  确诊的  伙伴关系  ▁Deaths  ▁booths  ▁Anxiety  ▁caravan 天里 小声  以下几  和快乐  诈骗案  ▁Sicily  ▁interns 进山  法令纹 ▁UCS  看到自己 ardom  ▁Tribal  ▁flashes  ▁Transaction  一阵子  单身男女  ▁USSR  ▁overlapping 一姐 舒坦  科技集团  ▁Bottle  ▁Neurology 拜师  新概念英语  ▁Diaries 巩俐 本想 触觉  就做好了  而是为了  重复使用 大权 工友 水桶 荒芜  小说的  ▁Understand  ▁sidewalks 护身  大市场  等国际 ▁prz  取而代之 松树 税负  发表于  ▁almond  googleapis  严重性  ▁Metals 六项 曲面  赵孟頫  ▁drap قى  不由得  走向了 Grade  ▁PEOPLE  ▁surged  ▁Beyoncé  ▁Concent  getInstance 哭诉 Unc  ▁favourable 所选  线下培训 encent  ▁escort 凭空 尾气 新篇 转世 IRED  中国农业  医用耗材 " 部分图片来源于网络 健儿 头和 长虹  与文化  当别人  ▁aquarium  ▁congregations 高炉  不亚于  普宁市  教学能力 Alber  瑶族自治县 城厢 Temp  更有价值  ▁Dimension  ▁depressing 火源 群友  不说话  例病例  侧重于  等荣誉  ▁Dane  ▁Shuttle 睡到  报了警  作家协会  匪夷所思  精神面貌 achal 便有 哀思 UFC!  名同学!  的剧情" Done"  家庭住址#  愿意为你#  ▁Gior$ ologia$  医疗机构就诊% 名画%  原则上不&  ▁Kier&  展望未来' gling'  ▁насе(  ▁Sonata(  ▁outage)  permission) 慕容* 现居*  百分比+  公告为准+  奠定基础, yb, 大中- 摄于- 霍山.  驱动力.  专家介绍/  以免造成/  恶心呕吐0  违法停车0  出版社出版1  ▁cellar1  ▁Wizards2 情调2  合同法3  结合部3  大学生士兵4  ▁Freight4  ▁craving5  ▁insecure5 与水6 八段6  座车站7  脱贫成果7 litre8  ▁Garn8  助力脱贫攻坚9  ▁Isabella9 不喜: 传唱: 盗墓;  ▁plagiar;  向上滑动查看内容< ( !市人力资源和社会保障局< 勿忘= 杀生= 虚度> 鱼虾>  从没有?  倒不如?  等物质@  ▁Zhao@  ▁kiosA 晨练A RoyceB  ▁peppB  互联网企业C  ▁hustleC  扫描上方二维码D  ▁drainedD  ▁pointersE % 城乡居民基本医疗保险E  免费看F  研究确定F  ▁NevilleG  ▁ReviewedG  大不了H CTOH  改名为I  血管内I  顶级的J  ▁HamletJ  ▁usabilityK 提货K 深交L  只可惜L  欢快的M  TextureM  treatmentN  忧郁症N  不必担心O  如果按照O  ▁champP  ▁hitterP  ▁strandsQ  ▁ProposedQ  ▁setbacksR  ▁encompassingR 特异S  是一段S  选手们T arooT  ”　|　U  ▁erstU  的重要依据V  北京理工大学V  ▁sublimeW 推卸W  总领馆X MACDX  最高境界Y  如果你没有Y  LettersZ  ▁punitiveZ  ▁assemblies[ 同乘[ 止泻\ ▁([\  奥利弗]  牢记初心使命]  ▁Ranked^  ▁unanswered^ 两制_ 睢宁_ ahs`  然后是`  本次比赛a  ▁fragmenteda 纬度b  教学和b  用它来c  消费者对c  详情如下d  尤其是对于d  ▁Flightse 像你e 西雅f койf  怎么用g  还是被g  高大上h  全世界都h  ▁storytelleri 力挺i 建在j  为世界j  休息日k  产权年限k  ▁automakerl 多则l 福岛m  培养的m  奖最佳n  新生代n  ▁Stratfordo 老爹o 首领p  杭州湾p  上级部门q  找到自己q  ▁climbsr шоr MLSs  吃水果s 插秧t 警种t  暂时关闭u  疏通经络u  项目申报v  relativev  ▁Islamabadw 冲压w rpmx ▁Nabx 厨余y 此方y 除菌z 音速z CED{  王宝乐{ WOOD| Chuck|  ▁Amend} 很棒} 适可~  不一定要~  但是这些  医学专业  发动群众 Jenny pixel  进入小程序  Frances 大热  你的手 zoom ▁sho  ▁Kare  全国大学生  ▁weekday  ▁ceremonial ág 更小 陈旧  副司长  加快了 Lion  每一次的  ▁Humb  ▁façade 均属 大足 春水  委员会主席  工作会 ▁Sek  ▁attentive  ▁visitation 云霞 楚国  又不会  就目前  班学习 cand  高速发展  ▁unriv  ▁snippet  市人大常委会主任 TG 回听 毛绒  现招聘  环比下降  ▁Bihar  ▁forehead 小包  地开展  方面也  洗净后  请备注  造价工程师  的大部分地区  ▁Landsc 不穿 快车 灭了  他的作品  能够起到 phonic  ▁fierc  ▁sparse  ▁asserts 底商 祸福 能级 黄皮  可提前  已全面  扫码加  永远无法  华南师范大学 一星 沙坪 租住  有关人员  知情同意  ▁Glacier 回声  几乎都  对党的  副理事长  达成共识  ▁scalp  ▁Ethical  ▁Furious 基亚 控诉  奥委会 oley  高考英语 elves  核酸检测证明  ▁moratorium 剩饭 老花 裂痕  中发挥  有效提升  贵重物品  送给大家  льзова % 中国特色农产品优势区  ▁foreground 川贝  虽然很  境外疫情输入  ▁Brewer  ▁Offshore 两队 失能 ▁Gob  ▁Oasis  ▁Mapping 绿标 者也 茶道  的一首  社区和  谢海洋  今天就来  公开遴选 停摆  在韩国 ▁MLA  操作简单  效果更佳 rolley 划入 各州 断崖 被曝­  的言论­  ▁Cairí  ▁benchmarksí 七种ĭ 尖椒ĭ 票面ŭ 赏月ŭ  起作用ƭ  飞机上ƭ  严格保密ǭ  作品名称ǭ  地处中国ȭ  ▁Crabȭ  ▁envyɭ 外甥ɭ CHRʭ  何首乌ʭ  双色球˭  用错了˭  的水域̭ udaṋ  直播预告ͭ  答题技巧ͭ  Workersέ >:έ  单身的ϭ  居民们ϭ  眼睛里Э  项目开工Э  林业和草原ѭ  ▁multimѭ  人民日报客户端ҭ  Architectureҭ 金马ӭ ▁Kovӭ  其他岗位ԭ  ▁Pettԭ  ▁ominխ etchedխ 口村֭  眼睁睁֭  国泰民安׭  的作用下׭  ▁Clashح  ▁heartbeatح 想好٭ ▁KU٭  负压救护车ڭ  ▁troopڭ  全面推进乡村振兴ۭ ósۭ 串通ܭ  旅游局ܭ  纯新盘ݭ Slowݭ  公司业务ޭ bringޭ negot߭  ▁skincare߭  ▁downstairs ۗۡ 检定 静文  综合工资 分秒 粤东  与挑战  号线一期  按时完成  有一段时间  ▁Ramadan  ▁Districts  手机卡  的小花  对于这种  ▁Cannot 字第 年任 战力  挥发性  教育等  督察组 osse veau  ▁Bray 一腔 七日 海丰 箴言 过吗  技术上  注册制  消防队 % 新增新冠肺炎确诊病例  ▁inefficient pps  不为人知的  ▁Stokes  ▁Duterte  ▁helmets  ▁undertook 伤及 加里 种群 跃跃  三亚市  让她们  顾客的  补齐短板  ▁slips  ▁Carlisle  ▁Jeremiah % 核酸检测结果均为阴性 余热 时评 树洞 证言  你的钱 ▁PAN 惹的 窘境  不能提供 光靠 气了 躺平 送考  尼亚州  录用资格  证据表明 assee Higher  国土空间规划  leground  标志性的  ▁Copa  ▁hesitation  ▁Frankenstein  与日支 anian  本微信公众号  ▁Presented 掉发 清宫 火影  这个品牌  Phoenix 和善 莒县 诗集  指挥长  肺功能  精心设计 Determ 寒露 月前 Lik jal  最后在  舒适感  ▁silhouette 抓到 水壶 祸害 者有 uela  景观设计 ▁­ 也以 镇大  了一组  的一篇  西湖区 xiny  艺术创作  ▁Wolfgang 该类 ▁MSc  犹豫不决  陪护人员  ▁Banc  ▁wrinkles  ▁Accidents  ▁commanders 和弦 wsh  和运动  ▁DeSantis  北方地区  ▁Outs  ▁forma! 不偏! 器乐" 盗用" 青苗# cephal#  flowing$ ті$ 减刑% /">% POR&  供销社&  有太多的'  ▁Ridley'  ▁masked(  ▁diabetic(  ▁INFORMATION) 岁到) 总则* 清源* ▁{\+  纤维化+  计划数,  用心用情,  ▁Inga-  ▁Tack-  ▁Telephone. 三组. 收腰/ 气化/ 王洪0 路飞0 ▁zw1  真人秀1  美少女2  能够快速2 ITIES3 顶楼3  从此以后4  我们坚信4  有时搬进阳光房5  ▁outages5  ▁Giveaway6  ▁Wallpaper6 或对7  警戒线7  身上下8  二期工程8 django9  的重要讲话精神9 断开: 相承: ilm;  企业要; creen<  ▁idiot< 只卖= 本套= 蓄水> izu>  空气净化?  第一作者?  ▁biting@  ▁untouched@ 洲岛A RoutA  ▁JosefB  ▁vinesB  ▁BasicsC  ▁bloomsC  上面的蓝色字体D  ▁brainstormD  ▁antioxidantE MegE  本科一批F  能感受到F  ▁rejoG  ▁PascalG  ▁exhilaratingH 业力H 足矣I  护城河I  ▁MatthJ  ▁MartinsJ  ▁disgustingK 万桶K 戏份L 水城L  主动去M  无形的M  请拉到文末N  的前期N  ▁TajikO  ▁alrightO % 并不承担其他任何责任P 应先P  透露着Q  ▁institutedQ 挺进R TitR  黄褐斑S  源远流长S  ▁WEEKT  ▁fittingsT  ▁implementationsU  你总是U  有钱了V  送到医院V 正反W  整改到位W ChangX  乘风破浪的姐姐X  ▁strainedY нгY 带宽Z 干得Z 茶农[ ▁Yor[  网友热议\  艰苦边远\  ▁Buffett] 忙得] 视图^ SSSS^  继续使用_  的主要成分_  ▁monst`  Surface`  interrupteda 丞相a 了事b 独木b 脐橙c  三保障c  低血压d  ▁OPENd  ▁countdowne  安眠药e  鲜艳的f  在直播间f  继续开展g  ▁icingg  ▁Staplesh izoh  任命为i  厚积薄i  lectricj ▁ssj  实习期k IiIik argvl  服务年限l youtum  在职研究生学历m 北向n 股沟n  有福了o  血管的o  颁发了p  个人防护意识p  ▁Somaliq  recordedq 太早r  了一部r  加分项s  苦尽甘来s  ▁Roofingt 劲道t 大群u 栅栏u ▁ATSv  十有八九v  要充分认识w  ▁Goddessw  ▁toleratedx 开火x yary ▁RDy  怎么就z grayz  结婚登记{  imentary{ 孙权| sus|  多读书}  必然要求} rmtzx~  ▁cram~  是什么样子  ▁treaties 续约  的实践  精美礼品  ▁tyre  ▁Pound 弱电 扫帚  赶紧收藏 永川  共同生活  ▁scientifically 个例 厘清 断定  大巴车  老师是  门头沟  ▁Vaughn  ▁complemented  ▁southwestern  地面的  成为你  的体现  的元素  育人的  最受欢迎的  居家医学观察  ▁skepticism 牌匾  主教练 ▁HAR  ▁Farn  ▁adept  Direction  ophysical 本店 Vac  谁也不  超声波  出门在外  族维生素  ▁Steward  ▁motivating  ▁Manufacturer  ▁manifestation 上大 南进 天伦 润肤 重置 ▁Edg  ▁whim  ▁Satisf  ▁famine 冻品 ▁Fam ▁Sut  应当及时  ▁Sovere 未免 IMO  ▁Rect  大学毕业后  ▁haird  ▁Asylum  ▁disagreed ▁UX  学生干部  ▁Sanford  ▁underrepresented = 6ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡۥۛ۟ۡۗۡ 只怕 铁丝 AGO  与对方  天气转  给予的  人行横道 failed  ▁embargo 一端 气概 渔夫 HBO ▁"^  物流园  诉说着  阅读能力  ▁proactively  ▁Municipality  希望各位  ▁pointless ( !本平台将注明来源及作者 视窗 这身 GPA  就是自己  期末复习  ▁librarian hir hov été  吃太多  暴露了  ▁Faso  ▁Loyola  ▁Gloucester 编纂 莫高 跳下 锣鼓 くて 力不 拐卖  新朋友  志愿活动  这个春节  ▁handset 大乐 谁料  和同学  多年了  给了她 oufl  长篇小说  ▁chor  ▁Fritz  ▁diocese 时才 花友 NHS NPC  个重点®  并告知®  自习室î  河南大学î  ▁HelmĮ 减征Į 海峰Ů  与日干Ů  那么难Ʈ  元的课程Ʈ  关于我们Ǯ  致以节日Ǯ  我们看到了Ȯ  ▁ConnieȮ  ▁McLeanɮ " 例新冠肺炎确诊病例ɮ  ▁unrealisticʮ 围攻ʮ  想要在ˮ  风平浪ˮ  的主要内容̮  locking̮  ▁forgivenͮ  ▁logisticalͮ oSή 　$ή  考虑的Ϯ SharϮ  继续坚持Ю bridsЮ  ▁BroadbandѮ 受累Ѯ 断水Ү 鲜甜Ү  ▁BreaksӮ  视频制作Ӯ ándezԮ  ▁NAACPԮ 农贸ծ 日将ծ ULE֮  人感染֮  以为常׮  在辖区׮  就有多خ  比较长خ  用心的ٮ  一个非常ٮ  技术路线ڮ  的时候是ڮ  就不一样了ۮ  折合人民币ۮ  patcherܮ  ▁sacrificingܮ 之品ݮ  每一段ݮ escapeޮ nasiumޮ  ▁Percy߮ 能静߮ 足的 闲事 ▁NB  原发性 moor ▁Oro  为更好地  募集资金 作的 其时 反流 奔放 致残 金利  费用包含  ▁Slate  ▁forfe  中国农业银行 共修 反常  消息后 leave 吉思  的省份  自适应  医疗保健  点击视频  体检和考察 ▁"'  ▁duel  的情况来看  ▁plateau 透光 Sax  不同类型  股份公司  ▁upro cookie  和控制 Karl 个盟 指使 签下 ▁")  罗永浩  诚挚的  诞生于 toBe hetto  ▁Cuts Cities  itimate 不沾 季军 起亚 都爱 都美 ▁JA  作案嫌疑 compos  归原作者所有  ▁Tuesdays  ▁monarchy 今世 折服  保平安  爱国情怀  生态宜居  最关键的是 PAC  成熟了  湘潭市  的消费者  ▁mRNA  ▁amalg  ▁coatings  Scientific  ▁preserves 壮年 家产 这届  先上岗  大名鼎鼎  终于来了  ▁unintended 战功 挥舞 胡润  暂停键  路过的  有了孩子  ▁adrenaline 一进 排涝 酷爱  大家族  文末有 Vote  中国传媒大学  ▁detract 包扎 考纲  发短信  很轻松  马德里  ▁Quincy  ▁Consolid  ▁delights  ▁Innovations 犯困 蓦然 足协 还敢 隆基 难成 ▁Nim  接种疫苗后 Legisl garden  ▁richly  Absolutely  ▁exploiting 已为 抗性  几秒钟  不得重新  ▁Neon  ▁Arrang 在目 爆火  我家的! ▁FAM!  内生动力"  ▁banquet"  ▁exhausting# 代餐# 就座$ 无措$ 相映% 米酒% 难易& ▁LB&  包裹着'  积极向'  除了要( ptons(  人教版数学) 冰镇) 毗连* 通识* cfg+  抢先看+ Salt,  报名要求, 浮生- 纳闷- 骤降.  完全的.  杯奶茶/  ▁FULL/ 切磋0 责成0  部分路段1 她曾1 安镇2 教您2 谜底3  四合院3 Labour4  ▁hinder4  ▁freshmen5 HQ5  传说中6  这款产品6  ▁grips7  ▁Sheets7  ocurrency8 婚丧8  涨工资9  走一走9  付出代价: OTHER: 次序; 贫民;  加适量<  角落里<  ▁Edited= 决然= 物管>  多巴胺>  聊天的? adin?  对此事@  排名中@  郁金香A  ▁radiosA  若干问题的解释B  ▁castlesB 亲水C 可对C 工区D  拟聘人员D  ▁DevinE 东村E 博鳌F  不准确F  官网发布G  报名资料G  红色经典H  ▁BodiesH  ▁fearlessI 看就I 落定J  ▁sonoJ  ▁BiotechnologyK 便血K 堪忧L 是可L 狭小M  是普通M  最权威N KINGN  ▁sterilO 火药O 见你P  婚姻关系P  点击文字Q  ▁LymeQ  ▁covertR  ▁passportsR 屏风S 抢走S 逼格T  暂时不T anskiU onsorU  全球最大的V ermannV  ▁UnterW  ▁penetrateW 李金X 梨树X 记叙Y  一家公司Y ackersZ  ▁epidemiZ  ▁transient[ 公摊[ 分给\ 浦北\ owej]  应用能力] 漫山^ 逆流^  的判断_  按揭贷款_  ▁SCHOOL` 分出` 日二a  有关信息a  ▁Shepardb 寒性b 毒蛇c 泰康c  你肯定d  我的世界d  汽车产品e  法律咨询e  热闹非凡f allocf 壮举g 运往g 遨游h  投资机构h  具备较强的i 一命i 到会j 当人j réalk  ▁Abramk 雄壮l CERl ▁iem  ▁Shetm  ▁suckedn  造型的n Catalo  ▁Whaleo 远赴p ANEp  世界纪录q  已经成了q  是内蒙古自治区r  Electionr CWs 满嘴s  每一颗t  ▁Gautt  的专业知识u  ▁burstsu 援藏v 新野v 莎士w  流露出w  该区域x  ▁followerx  ▁harmoniesy  ▁negligenty Fluz  表现突出z  Friendly{  ▁exposures{  ▁physiology| 沪宁| 涟漪} 菲利}  照顾好自己~  Heather~  ▁Rebels  ▁leukemia 万圣 POSE  单科成绩 aneers  的影子  ▁Welt  ▁Allies 成武 暗黑  那不是 buck  抗病能力  最新政策 ám 装点 迷彩 ATP  和领导  有一批  第四轮  ▁quar  ▁crest  ▁Charts  ▁Brilliant  ▁overarching 体虚 喷喷  适度的  ▁Slav 别扭 违和  与老师  为根本  人生观  含水量  特困人员  ▁claw  ▁bullpen  ▁Argument  ▁Normandy 利物  可以试试  岁的女儿  操作手册  的高度重视 半路 百世 这艘  三星堆  走一步  的第一线  ▁glut  ▁Browne  ▁Dollars  ▁espionage 回款 擂台  两件事  后果自负  这份工作 就变  从负债  你看看  文件中  呼吸系统疾病  american  ▁Labrador  朝九晚 arina  ▁invoices  在许多  准考证号 lewood  Scholar  uisance  ▁Researcher 创刊 日因 未解 涨薪 篝火 翘臀  他现在 itto  ▁Shops 刺杀 已超  可以买  居住区 春城 gee  的规律  建筑风格  ▁Herc  改变了自己的人生 名册 掘金  冲锋枪  找借口  携带者  每半年  的前沿  ▁Dodd cancel  ▁Feast  ▁Starts Ng 一岗  如何将  形成以  信用信息  可能已经  ▁behavioural 火化  的运行  而根据 ▁TWO  公布为准 fixed  ▁Pieces  ▁Volleyball 为使  不计较  工业协会  提升行动 月仙  提高至  ▁Waves 军机  白云山  而且对 enco  ▁Britann¯  ▁Francesco¯ 姑息ï 某宝ï  晚上好į  人体健康į  平板电脑ů  ▁Alonsoů 绿萝Ư  机动车辆Ư  ▁Ruizǯ 笑颜ǯ 腰肌ȯ soldȯ  引发火灾ɯ  Crystalɯ  ▁Honorableʯ  ▁Narrativeʯ 当务˯ 期和˯ 猛兽̯  说不出̯  ▁Kassͯ  ▁Filmmͯ  compareί  alliativeί  ▁optimizingϯ 奉劝ϯ 黄脸Я  支持者Я  这些钱ѯ  黄花菜ѯ unasү  而且还会ү  ▁Abigailӯ 清幽ӯ  演变成ԯ  莎士比亚ԯ  ▁Duffկ 亚美կ  它也是֯  建筑密度֯ 恶臭ׯ 闭关ׯ ceeد  两厅两卫د  逐步实现ٯ  五年级上册ٯ  住建厅گ  包食宿گ  小儿子ۯ  发展目标ۯ  套餐包含ܯ  影响孩子ܯ  自主创业ݯ  Tennesseeݯ  technologiesޯ 四字ޯ  灯光秀߯ Brief߯  ▁prized  的演员  如果你也  小学科学  考试时间为  ▁Abdel ᠠᠨ 咆哮 总得 排挤 acz  黄某某  给孩子的 国共 漫威 vote  意思是说  推动工作 thren 绕过  部门要  自愿放弃  ▁tore  传递正能量  ▁needy 新家  匈奴的  春节后 ▁bye  更好的生活  iquette  中国十大  ▁rearr 星系 顽疾 BIG  众议院  像这种  冒名顶替  未经批准  ▁olds  ▁Sammy  charges  ▁ANGELES  ▁Revenge  ▁Creativity  Communication  疼痛感  ▁Choi  ▁tighten  ▁precursor 一望  的各个  四川成都  Artists  ▁REUTERS 命里 钟表  发展了  ▁Marshal  ▁underwear Fs 俗话 增补 nego  就业见习  就会变成 zypisy  ▁detectors 炸至 眼病 Lie  你就要  的能力和 paces  ▁Dove  ▁eradicate ' ################################ にな 不腻  并最终  方法二  的现场  很容易导致  ▁computed 啪啦 随口  才能实现  ▁Rahman 如未 由浅 译林 onie snap  深刻理解  驾驶车辆 vonne 见血  三年前  除夕夜 Honda  ▁Rentals 茶余  的神秘  mbudsman  ▁exacerbated 以赛 何用 在肩 天高 标尺 洗护 透出  新车的  的难题  等重点场所  ▁petty  ▁abusing 两万 场镇  今天早上  建功立业 hower  ▁Cate  ▁Paige  ▁Eleven  ▁Bordeaux 晒晒 风尘  这一场  ▁дан!  制度体系!  寓教于乐"  ▁spared" 恰似#  他们要#  写一篇$  大革命$  一二年级%  研发团队%  转载文章&  ▁Downey& ( !突如其来的新冠肺炎疫情' 王军' 粘液( 里了( ▁Esk) ▁fot)  落地落实*  什么情况下*  ▁caval+  ▁breakup+ 交投, 淅淅,  不串门-  白砂糖-  校园文化.  过程当中. yllic/ 志伟/  精心的0 atitis0  ruiting1  ▁resemblance1  个不停2  委托书2  来一次3  灵活运用3  ▁grooming4 绝版4 ▁Rin5  成绩优异5  西咸新区6  ▁Knock6  ▁diligently7 问暖7  美元的8  运动时8 ▁Naw9  ▁stare9 + $尽量避免乘坐公共交通工具: 斯里:  禁止在;  视频等; ectomy<  打赢疫情防控<  ▁Velvet= 啧啧= ucl>  从这些>  佛罗里?  瑟瑟发抖?  艰难险阻@ 净利@ 安省A 黄岩A  ▁stockpB  publisherB 亚当C  但作为C  我突然D  楚汉相争D  AnnotationE 文成E 最贵F 特拉F BahG GreyG  ▁turboH ▁BDH  对世界I GKI 北岸J 开合J  工信局K  廉洁纪律K  民营经济L  编辑转载L  ▁PingM  ▁solemnM  ▁SpiritsN 动于N 甲子O 畏寒O  做某事P  联络员P  让皮肤Q iletQ  学生宿舍R  ▁OptimizationR  在西安S  的频率S  ParticipantsT 棋盘T 要害U  了过来U chalV  教育中心V  ▁TemporaryW  全国高校W  房屋产权X  特色学校X  纳斯达克Y  没有一个人Y  解决这个问题Z  OklahomaZ  ▁greener[  ▁salvage[ 可他\ 省了\  物理学家] Optim]  ▁Pose^  ▁livestream^ 印光_ 鱼尾_ Peer`  大专可报`  ▁speculatea JEa ▁بb  下方图片b awardc  ▁RCMPc  ▁vigorousd 委内d 振东e 细品e  务实的f  心安神f  渗透到g anedg  农业产业h  原汁原味h  ▁Balei  ▁Forgotteni 发白j 手环j  时支为k  注定要k  海岸线l ▁Prul  ▁Aminm  ▁Prabm Eithern  ▁Producedn  ▁enchantingo 古称o 度日p 译者p  向学生q  畅销书q  因为孩子r  岗位名称r aceouss 我得s 预祝t lopt  人提供u  小常识u  越秀区v  企业生产v  详细介绍了w  ▁briskw  ▁Thursdaysx 制程x  忙碌碌y  知多少y  湖南大学z 凶星z  爱学习{  Generation{  ▁desperation|  ▁comparatively| 搭理} 断肠} 该行~  乘务员~ Ruth  任何理由  ▁punches  ▁Gambling 产房 存有 胶质  新发现  的回忆 ydon  ometers  ▁Catalan  ▁misunderstood 也把 烧制  太美了  福克斯 Leod izio ▁LOL  ▁dass  ▁Matte  ▁Reconc  ▁characterised  带走了  年参加  预计到  就一定会 屈辱  见到的  遇上了  ▁seren  Bringing  ▁believable 量级 ▁Mā  持续上涨  Exploring 倒掉 听雨 嘉庆  我见过  约而同  抓住机遇  ▁operative . '长按识别二维码了解房屋详情 ™, 多张  的格局  ▁Confederation 抢手 防尘  还等什么  信息化建设  ▁rhythmic 支票 果香 深水 肉末 诟病  不能少  回到家乡 Taiwan  ▁Chant  agedies  ▁Laptop  ▁gravit 随车  大检查  通化市 ▁sod  ▁TRUE  意思就是  高度警惕 phasis stered  几天前  必须为  以此类推  无人问津 chter  由公安机关  ▁lasers " 爱国卫生运动委员会 YZ 受寒  不行了  更可怕  信息传播  自身建设  ▁aber  Przypisy  Implement  古钱币  很长的  克服麻痹  经络不通  Respond  iscovered 南亚 场外  产品中  ▁Taxi  ▁millennium たく SUN ▁ej  黎巴嫩  ▁Jamal  消防救援队伍  ▁Emanuel LPL  家庭和睦  校园足球  难以接受  ▁psychiatrist RG 年复 根植 水汽  二人的  穿山甲 Chef  累计出院  新中国成立后 惨痛 澄海 鸽友  你是个  比以往 uper  现在看来 <>();  ▁jumper 情头° 连读°  坐月子ð Zachð  不可收拾İ  切成小块İ  ▁sniffŰ  你有多Ű  只为了ư  将作为ư  ▁ORDERǰ  ▁flyerǰ  HousingȰ  ▁diversificationȰ sitɰ  一大队ɰ ▁Raiʰ  风险能力ʰ  ▁Engel˰  ▁Statue˰ 泥巴̰ 秀山̰  德国的Ͱ  第七条Ͱ  utscherΰ 督战ΰ  风风雨ϰ  第十四个ϰ  ▁Endingа  ▁residueа  ▁ZeppelinѰ 双非Ѱ  它不是Ұ  幼儿的Ұ  是能够Ӱ  人民广播电台Ӱ  ▁Sweeney԰ 东吴԰ 椭圆հ 每场հ ▁Pvtְ  战略布局ְ  靠前指挥װ  想要的生活װ  ▁badgesذ  ▁Françoisذ 开诊ٰ 生性ٰ 骤停ڰ  老老实实ڰ  这样的方式۰  ▁Theat۰  百年未有之大变局ܰ 他汀ܰ 宠粉ݰ 逐户ݰ 链子ް  准确率ް  但是这种߰  excuses߰ 镜像  而今天  苏宁易购  通过互联网  ▁envoy 前半 旗鼓 菱形  换一种  免费分享  去年年底  推动作用 无成 月被 甲亢 留情 自性 辫子 近况  性贫血  王安石 Cher ▁NES  也进行了  葡萄球菌  每个人都会 吊牌 水道 车行  ▁propelled 全境 很会 松紧 民初 粒粒  低保户  的保障  防火墙 itia  动能转换  人民解放军 文胸 桥段 缴税  Насе  因为工作  生长激素 画意 福运  吉思汗  根本不会  ▁Lease  ▁Chanel  ▁inward 弱小 磨砂  打篮球  李鸿章 ocyt  ▁grease 对非 百会 肛肠 门徒 鼻血  　(　)  还有可能  ▁Secondly 传单 低级 相会 趣闻  一周后  吴女士  合作项目  ▁Vend county 杨凌  都这么  教学班  原因很简单  ▁cellul  ▁roaming JL  小个子 izhi  一体推进  环环相扣  ▁dine  ▁Bhatt  ▁Breat  ▁Vanguard  ▁apprentice 片片  刻意的 imar  取得相应  忙忙碌碌  消费金融  调查取证  Insurance 共商 秋裤  第一学期  colonial  ▁Moderna 德胜  ▁Andhra 顺境  展现在  的沟通  增值服务  当务之急  执行案号  特别严重  适可而止  公共资源交易  ▁losers 元阳 到站  李时珍  错时错峰  能力和水平 quests  ▁Teens  ▁obliv 什麽  吕梁市  ▁Rails  ▁techno  ▁McBride!  ▁schizophrenia! 创优" 汤山"  招生就业#  校园生活#  ▁sweets$  先进制造业$  ▁frenzy% まえ% ▁pg&  ▁Siri&  人这一辈子'  看起来更加'  ▁reimag( 流汗(  在他看来)  机械工程)  ▁bursting*  ▁Oversight*  ▁condemnation+  ▁contemplating+ 乡情, 悲壮,  在抗疫- ▁Xen-  汽车零部件. lh. 观感/  ▁vind/  ▁Akron0 训斥0  你是谁1  使用前1  公开信2  第四代2  一个地方3 athed3  ▁anonymously4 三针4 同庆5  建成区5 瞌睡6 落榜6 ▁jug7  为人父母7  致人死亡8  ▁Asheville8  ▁protested9 学联9 朋克: 蒙蔽: MOS;  复合型;  检测到<  水到渠<  股份制=  西汉时=  都不行>  还有一位>  ▁pont? 地盘? 大旺@ ▁kw@  出行的A ▁RADA  中国人寿B raintB  ▁nounC ursingC  ▁adjectD 叫嚣D 求得E 灶台E  的临床F  设计速度F  这里都有G  邮电大学G fortsH  补充维生素H  ▁accuseI  ▁plentifulI 去污J 沉降J 蒙牛K 转走K 长款L  ▁BritneyL  ▁ComputersM }`M 以偿N  爱马仕N otryO  ▁RidersO  ▁teamingP  ▁LegendaryP 仁恒Q 餐食Q  等一些R  经鉴定R ▁LINS  实训基地S  ▁SantoT  MountainT 给钱U lucU  闵行区V  ▁piùV  周边道路W  就让我们W  ▁irresponsibleX 中亚X 牙签Y 课课Y 隆起Z 高盐Z  工地上[  教科文[  村两委\  吸引更多\  走在前列] 村容] 歹徒^ 认出^  了过去_  卡罗拉_  在后台`  视同仁` ▁Nila  持续优化a havingb  ▁moversb  ▁leveragedc  ▁deductionsc 之笔d 城门d 绥德e  个体户e  全行业f  做的更好f  ▁Promog 坡度g 好呢h  习以为常h  外观设计i  操作步骤i  日出现发热j  Tomorrowj 云的k 阿萨k  我是个l  水利部l  ▁илиm  任何单位m  医保报销n  虽然不能n  ▁Elioto  负责对o Evanp  宣化上人p  真的非常q FNq 之国r  宣传栏r Mercs  恢复运营s 十指t 特立t  月活动u  腹股沟u intav  考试辅导v  ▁Cremw  ▁Affleckw  ▁midtermx 卖得x  保护与y  按部就y  只要自己z  含有大量z  研究领域{  Minimum{ HW| 初春|  中心区}  掉下来}  的气味~  运动队~  还特意 ▁Civ  ▁ignition 查办  ▁Lyft motion  ▁swipe  Practice  ▁Preserve  Developing  小广告  盯上了  限制性  注册用户  的问题是  ▁Expected  ▁eyebrows ío  ▁Patton 日主  安康市 rece  横空出世  深圳大学  熟练运用  ▁taxed  ▁crater  ▁Kaufman  ▁nicknamed 徐家 neur  持续推动  会同有关部门  tracking  ▁lobster 出声 到外 助残 承运 挖矿  人生命  梁启超 WORD raper  ▁YEAR  周末节假日  ▁entrants  ▁tertiary 叹气 掏心 Rare  光明日报  ▁Rapp powers 促改 品学 她不 穿脱  创建了 BJECT  entanyl  ▁excavation 干饭  规则的 opor ▁HAS  考试结束后 一蹴 村寨 淬炼 磐石 韩城  华晨宇  要抓紧  在每一个  生命里的 athom  ▁annotation 之文 也快 桑植 重仓 Ott  也未必  拿到手  比较难  民众的 Depth ▁  e  t  a  o  i  n  r  s  h  l  d  ,  c  u  m  p  g  f  y  w  b 的  v  0  1  k  2 　  S  T 一  A  C  :  I 是 人 不 有  )  (  M  3 在  P 了±  5±  Bñ “ñ ”ı  9ı  Dű 中ű  xƱ 大Ʊ  EDZ  RDZ  4ȱ  8ȱ  Wɱ 生ɱ  6ʱ  Nʱ 学˱ 为˱  H̱  Ḻ  Fͱ 时ͱ 年α 上α  7ϱ  Oϱ 和б 要б 个ѱ ’ѱ 会ұ 这ұ 以ӱ 我ӱ 来Ա  GԱ  /ձ 到ձ 国ֱ 发ֱ  ;ױ 作ױ  "ر 能ر 家ٱ  'ٱ 就ڱ 行ڱ  j۱ 工۱ 业ܱ 出ܱ 可ݱ  zݱ 用ޱ 们ޱ 对߱ 地߱ 自 多 成 日 下 新 公 后  U 全 市 好 子 你 分 小 心 高 也 动 方  J 开 情 于 过 月 理  q 实 天 都 区 进 现 经 他 面 点 体 等 法 关 部 力  V 通 事 定 得 如  K 长 文 间 前 合 看 最 同 本 员 之 安 所 还 加  _ 者 重 主 说 与 位 教 而 入 期 化 名 内 考 道 信 机 建 提 其 产 民 活 车 保 务 性 起  Y 三 里 水 很 报 明 — 场 资 品 及 无 平 更 着 么 当 度 电 意 种 相 第 网 将 美 没 目 展 次 常 果 应 感! 外! 做" 手" 因# 老# 路$ 管$ 己% 量% 知& 元& 身' 然' 程( 系( 设) 物) 二* 师* 制+ 题+ 从, 院, 防- 病- 正.  >. 想/ 去/ 医0 但0 数1 接1 疫2 解2 表3 交3 样4 口4 让5 校5 单6 些6 金7 政7 被8 并8 科9 比9 线: 各: 气; 或; 南< 每< 专= 计= 强> 任> 城? 女? 门@ 山@ 式A 特A 源B 问B 原C 风C 只D 万D 处E 别E 级F 东F 那G 此G 习H 省H 西I 书I 服J 持J 利K 记K 试L 注L 育M 规M 价N 回N 已O 导O 格P 查P 求Q 件Q 收R 结R  |S 代S 给T 打T 放U 项U 海V 两V  <W 至W 需X  %X 联Y 号Y 运Z 社Z 复[ 费[ 话\ 头\ 基] 向] 总^ 受^ 司_ 爱_ 真` 少`  *a 请a 孩b 证b 色c 直c 图d 精d 北e 调e 十f 组f 取g 据g …h 县h 质i 治i 房j 共j 传k 例k  $l 推l 党m 控m 确n 选n 台o 办o 委p 息p 流q 广q 立r 使r 州s 示s 花t 局t 集u 标u 视v 村v 变w 清w 四x 容x 技y 完y 节z 才z 带{ 指{ 商| 再| 今} 把} 她~ 招~ 整 近 消 告 先 友 领 课 快 效 微 由 造 统 术 研 准 案 布 增 白 养 职 非 优 华 认 众 护 型 备 儿 什 条 步 检 类 健 识 影 深 见 – 光 具 周 参 创  & 世 食 神 热 转 企 施 五 群  { 责  } 江 战 达 包 改 团 户 限 支 权 康 难 何 阳 干 张 营 即 料 候 又 供 编 况 观 红 配 队 助 吃 片 未 读 形  [ 越  ] 按 住 除 该 乐 排 语 园 议 空 环 离 字 历 投 验 续  Q 免 走 太 易 几 决 反 满 农 照 ҉ 根 始 福 低 警 往 态 测² 思² 预ò 功ò 某IJ 论IJ 游Ų 王Ų 维Ʋ 速Ʋ 必Dz 温Dz 界Ȳ 超Ȳ 境ɲ 升ɲ 段ʲ 命ʲ 积˲ 核˲ 较̲ 觉̲ 毒Ͳ 军Ͳ 落β 客β 它ϲ 划ϲ 严в 站в 获Ѳ 款Ѳ 财Ҳ 状Ҳ 坚Ӳ 装Ӳ 便Բ 值Բ 险ղ 班ղ 青ֲ 致ֲ 血ײ 属ײ 河ز оز 欢ٲ  Xٲ 引ڲ 钱ڲ 约۲ 药۲ 究ܲ 边ܲ 喜ݲ 击ݲ 聘޲ 范޲ 男߲ 份߲ 且 首 火 买 百 极 岁 监 春 早 镇 马 亲 切 适 随  ~ 士 构 则 际 疗  #  @ 仅 景 讲 言 码 细 素 黄 义 足 居 米 响 宝 版 置 а 志 审 称 眼 购 望 显 星 采 断 留 京 声 连 送 络 林 冠 底 存 势 初 失 像 济  Z 享 播 史 织 乡 依 英 象 双 德 简 轻 酒 故 录 评 店 材 油 卫 模 补 远 终 香 听 许 率 频 室 普 音 李 算 朋 层 味 够 府 球 律 云 却 · 修 写 负 富 龙 克 е 愿 察 石 销 и 假 充 母 待 湖 卡 帮 ‘! 诊! 策" 择" 均# 章# 培$ 尽$ 遇% 压% 降& 围& 古' 协' 善( 停( 减) 训) 举* 妈* 域+ 晚+ 拉, 您, 兴- 楼- 售. 六. 怎/ 紧/ 半0 官0 土1 赛1 益2 染2 止3 错3 找4 临4 批5 套5 死6 艺6 牌7 岗7 宣8 列8 器9 画9 承: 座: 千; 患; 旅< н< 演= 武= 炎> 害> 副? 含? 纪@ 吗@ 肉A 急A 智B 执B 念C 黑C 融D 伤D 街E 跟E 奖F 吧F 密G  \G 练H 铁H 担I 答I 博J 雨J 症K 载K 阅L 皮L 角M 餐M 斯N 呢N 宁O 良O 突P 汉P 货Q 争Q 银R 申R 违S 谢S 障T 菜T 庭U 抗U 肺V 父V 须W 印W 哪X 税X 毕Y 退Y 板Z 八Z ҈[ 股[ 付\ 独\ 季] 询] 序^ 尔^ 贵_ р_ т` 余` 酸a 笔a 曾b 亿b 闻c 征c 短d 吸d 顺e 绩e 破f 登f 夜g 痛g 刻h 综h 辑i 松i 介j 苦j 陈k 午k 继l 聚l 额m 彩m 互n 换n 钟o 希o 玩p сp 救q 稳q 激r 绝r 背s 否s 苏t 句t 牛u 七u 刘v 穿v 刚w 块w 票x 族x 守y 差y éz 般z 虽{ 飞{ 右| 绿| 左} 迎} 秀~ 婚~ 丰 茶 异 永 疑 诉 另 粉 输 析 央 拿 措 洗 草 疾 础 困 苗 若 藏 顾 ↓ 厅 纳 归 幸 驾 劳 堂 荐 抓 久 亮 圈 拍 慢 衣 九 航 媒 鱼 伴 附 针 典 灵 盘 丽 卖 休 剧 亚 辆 树 甚 端 启 冷 追 奇 涉 隔 笑 届 夏 庆 в 夫 鲜 罪 罗 词 贫 督 静 累 梦 歌 馆 似 革 犯 兰 令 筑 挥 木 毛 扫 港 佛 贴 波 略 饮 脑 靠 睡 延 惠 缺 阿 礼 巴 汽 烟 丝 冲 拥 童 л 哈 醒 脱 汇 促 括 占 熟 谈 鸡 努 予 猪 川 秋 侵 操 谁 索 爆 坐 幼 沙³ 斗³ 避ó 册ó 田ij 危ij 散ų кų 触Ƴ 蛋Ƴ 玉dz 诗dz 宜ȳ 桥ȳ 饭ɳ 暖ɳ 怕ʳ 戏ʳ 驶˳ 述˳ 签̳ 杨̳ 呼ͳ 返ͳ 怀γ 脚γ 封ϳ 判ϳ 喝г 叫г 罩ѳ 移ѳ 叶ҳ 净ҳ 胜ӳ 攻ӳ 摄Գ 损Գ 禁ճ 阶ճ 授ֳ 肤ֳ 搭׳ 床׳ 阴س 杀س 键ٳ 软ٳ 龄ڳ 派ڳ 荣۳ 雪۳ 糖ܳ 曲ܳ 骨ݳ 懂ݳ 址޳ 绍޳ 厂߳ 涨߳ 轮 咨 掌 液 固 倒 乎 托 盖 佳 奶 讯 透 寒 献 朝 戴 租 竟 兵 零 姐 吉 植 截 励 峰 稿 握 顶 符 肥 露 详 蓝 订 野 卷 尚 乘 纸 链 箱 虚 距 逐 趣 啊 借 剂 掉 跑 库 奋 庄 脸 探 杂 误 邮 仍 席 秘 哥 暴 冬 д 篇 威 牙 访 删 亡 惊 齐 菌 竞 屋 虑 束 м 豆 隐 诚 末 骗 抢 岛 召 胃 折 盛 旧 烈 洁 爸 罚 у 赞 架 扬 纷 侧 伙 渐 乱 恶 麻 币 冰 啦 私 湿 添 账 忘 颜 途 圆 既 舞 薪 翻 硬 旗 释 幅 丨 扶 寻 浪! 缓! 洲" 彻" 缴# 勤# 遗$ 陆$ 舒% 灯% 递& 著& 妇' 扩' 肌( 沉( 厚) 润) 焦* 凡* 赶+ 祝+ 锅, 洋, 慧- 雄- 滑. 巨. 援/ 宽/ 尤0 雅0 肝1 弹1 津2 勇2 挂3 п3 填4 妻4 蒙5 哦5 朗6 辅6 闭7 惯7 沟8 烧8 伟9 概9 刑: 漫: 射; 贯; 吴< 敬< 雷= 甲= 刺> 籍> 赵? 践? 鼓@ 挑@ 偏A 甜A 弱B 贷B 档C •C 拟D 唐D 仪E 淡E 缘F 胡F 弃G 甘G 湾H 姓H 巧I 敢I 奥J 坏J 钢K 暂K 孙L 署L 恩M 宫M 泡N 脏N 聊O 汤O 纯P 泉P 癌Q 宿Q 讨R 腿R 灾S 迷S 摩T 恢T 羊U 宗U 唯V 欧V 忙W 镜W 凉X 陪X 殊Y 刷Y 阵Z 肠Z 昌[ 潮[ 抽\ 肯\ 嫌] 窗] 尊^ 「^ 」_ 珍_ 泰` 郑` 赏a 梅a 赚b 耳b 池c 徐c 污d 阻d 默e 晓e 悉f 遍f 祖g 浓g 腰h 倍h 君i 辛i 唱j 宅j 晨k 敏k 桃l 估l 弟m 旦m 绪n 浙n 斤o 舍o 拼p 谷p 扣q 珠q 梁r 描r 盐s 嘉s 肖t 幕t 毫u 脉u 忆v 尼v 繁w 跳w 扎x 抱x 尿y 楚y 脂z 泽z 狗{ 伍{ 洞| 振| 筹} 混} 饰~ 圣~ 残 恋 柔 á 腹 衡 韩 谓 迁  ` 闲 败 迹 绕 储 贸 凭 畅 俗 船 圳 燃 胞 烦 帝 耐 券 妹 豪 尾 抵 杯 瓶 映 暗 诺 跨 乌 欲 仙 í 乳 顿 缩 炼 迅 跌 牢 瑞 铺 朱 沿 瓜 丹 搜 赢 墙 横 巡 览 径 偿 г 飘 蜜 垃 鞋 秒 刀 摆 榜 森 圾 я 疼 呈 矿 肿 眠 屏 佩 虫 驻 肾 岸 膜 携 盟 ó 塔 宋 炒 愈 脾 溪 遵 婆 滨 鲁 皇 腾 洪 灭 莫 壁 紫 辉 辈 伊 丁 徽 穴 债 胶 潜 澳 宇 页 锋 役 废 仁 虎 胸 震 搞 寿 宾 з 磨 拆 兼 覆 贡 循´ 嘴´ 孕ô 赠ô 桂Ĵ 冒Ĵ 颗Ŵ 栏Ŵ 欣ƴ іƴ 恐Ǵ 粮Ǵ 爷ȴ 旺ȴ 替ɴ 遭ɴ 锁ʴ 拒ʴ 轨˴ 杭˴ 晋̴ 忍̴ 疆ʹ 贝ʹ 邀δ 薄δ 麦ϴ 辣ϴ 郭д 坛д 鉴Ѵ 硕Ѵ 惜Ҵ 吨Ҵ 萨Ӵ 诈Ӵ 暑Դ 誉Դ 狂մ 董մ 绘ִ 裤ִ 碰״ 猫״ 逃ش 诸ش 迟ٴ 锦ٴ 氧ڴ →ڴ 亦۴ ь۴ 辖ܴ 梯ܴ 纹ݴ 旁ݴ 悦޴ 尖޴ 鼻ߴ 氏ߴ ㎡ 拜 肃 孔 凤 殖 滴 б 逆 ℃ 娘 塞 竹 艾 汗 迪 陕 凝 滋 忠 沈 肩 伸 怪 姜 裁 袋 渠 胆 腐 履 尝 扰 壮 洛 谋 仔 胎 妙 拔 塑 陷 恒 隆 炸 谱 悲 姻 杰 偶 椒 奉 娱 奔 泛 拓 漏 煮 倾 爽 桌 纠 祥 劲 й 徒 乏 疏 趋 » 泥 喷 浮 杜 械 摸 捐 挺 耗 ä £ 忧 撑 秦 枝 睛 悟 晒 偷 井 腔 伦 馨 忽 ы 牵 凌 伏 煤 宏 俄 穷 盒 迫 抑 插 坦 踏 魔 刊 坡 哭 邓 琴 闹 灰 泪 寸 鸟 碎 貌 赔 辽 © 燕 捷 厨 涂 墨 姿 艳! 斑! 袭" 揭" 芳# 赴# 昨$  ^$ 郁% ×% 漂& 颈& 戒' 吐' 粒( 艰( 柳) 敌) 厦* 荷* 瘦+ 氛+ 曝, 叔, 饱- ч- 孤. 摘. 醉/ 丈/ 涵0 仓0 寄1 魂1 剩2 莲2 亩3 摇3 弯4 蔬4 衰5 腺5 ü6 汁6 坊7 仿7 堆8 啥8 朵9 霸9 跃: 盆: 冻; 吹; 碍< 译< 瘤= 皆= 晶> 骑> 凯? 撞? 咳@ 奏@ 尘A 杆A 鼠B 惑B 彼C 挖C 铜D 陵D 拖E 搬E 枪F 耀F 邻G 番G 烤H 昆H 诱I 劝I 尺J 驱J 锻K 慎K 娃L 粗L 胖M 岭M 裂N 蜂N 瞬O 淘O 炉P 纵P 厉Q 妆Q 辞R 蒸R 盾S 塘S 岩T 阔T 滚U 剑U 慰V 堵V 剪W 烂W 允X 踪X 抖Y 赖Y 夺Z 菲Z 捕[ 串[ 鸣\ 碗\ 浅] 酱] ö^ 伯^ 亏_ 闪_ 兄` 渡` 怨a 谨a 鬼b 浦b 碳c 勒c 丢d 咖d 盈e 擦e 崇f 撒f 熬g 屈g °h 赋h 拨i 虾i 嫩j 浩j 腻k 弄k 聪l 陶l 忌m 曹m ་n 肚n 柜o 淮o 逻p 廉p 耕q 莱q 柱r ①r 旋s 稍s 卧t 幻t 挡u 迈u 彰v 盗v 篮w 疲w 筋x 齿x 栋y 饼y 宠z 倡z 怒{ 裙{ 姑| 疯| 稀} 寺} 乙~ 叉~ 纽 ② 馈 恰 鸭 毁 苹 呀 魅 熊 犹 诵 慈 芝 雾 瓦 芯 纤 悬 侦 贺 诞 浴 牧 框 秩 ​ 抚 脆 拳 葱 х 囊 贤 燥 孝 赤 谣 悔 娜 霞 撤 愁 谐 匀 曼 妥 雕 寓 咱 棉 婴 ↑ 眉 糊 猜 吊 酬 逼 悠 窝 鹏 赌 舌 荡 矛 骤 恼 猛 玛 乃 仰 肪 晰 勿 糕 涛 坑 垂 岳 湘 俊 钻 茂 孟 泳 俱 御 邦 兔 晴 灌 盲 堪 狼 拘 傲 粘 绵 淋 卓 帅 玻 钓 灸 喊 碧 弘 羽 菩 割 è 膏 慕 幽 欠 舰 锡 巾 堡 摊 泄 酷 磅 昂 璃 棒 萝 哲 傅 姆 桑 贪 夕  µ 叠µ 旨õ 魏õ 碑ĵ 彭ĵ 逢ŵ 鸿ŵ 祭Ƶ 苑Ƶ жǵ 悄ǵ 浸ȵ 恨ȵ 弥ɵ 舟ɵ łʵ 蔡ʵ 韵˵ 萌˵ 缝̵ 葡̵ 轴͵ 催͵ 醇ε 巩ε 黎ϵ 玫ϵ 纲е àе 泌ѵ 夹ѵ 丘ҵ 骂ҵ цӵ 抹ӵ 臭Ե 邪Ե 卜յ 玲յ 株ֵ 瑰ֵ 凶׵ 枣׵ 袁ص 螺ص 旬ٵ 蛇ٵ 狮ڵ 函ڵ 遥۵ 慌۵ 鼎ܵ 亭ܵ 辰ݵ 蒜ݵ 淀޵ 洒޵ 蒋ߵ 拌ߵ 奈 砂 筛 押 暨 炮 嗽 萄 椎 棚 翠 抛 颖 胀 斜 辨 匹 峡 抬 鹅 溶 霍 兽 寨 募 伪 ا 挤 欺 丧 爬 柴 涌 廊 桶 卢 咸 壳 瓷 虹 驰 咽 契 霜 惧 懒 啡 掘 宴 辩 贾 叹 衔 颁 陌 擅 苍 蟹 渔 梳 臂 惨 荒 膝 挣 詹 狱 厌 咬 冯 滩 埋 巷 畏 槽 ③ 逸 兑 渴 浆 乔 祸 妨 庙 敲 茅 钙 嫁 帽 墓 鸽 闷 鹿 丑 椅 垫 ® 扭 郎 粤 饲 翼 吕 讼 劣 柏 肢 豫 朴 泻 夸 稻 砖 狠 歉 猴 芽 汪 遮 戳 〔 鹤 〕 矩 惩 埃 棋 滞 毅 ☞ 俩! 壶! 丸" 吵"  # 蹈# 莉$ 裕$ 侣% 憾% 潭& 披& 骄' 痕' 锐( 壤( 斌) 耶) 躺* 胁* 铭+ 瑶+ 娇, 姚, 盼- 梨- 翔. 勾. 煎/ 溃/ 痘0 匠0 赫1 搏1 呆2 惕2 厘3 潘3 蓄4 喂4 禅5 攀5 羡6 枚6 ш7 犬7 溢8 厕8 茫9 霉9 鄂: 磁: 卵; 扑; 扮< 罐< 汛= 菇= 傻> 晕> 槛? 荆? 粥@ 沧@ 箭A 葛A 冈B 侯B 巢C 芒C 愉D 姨D 坪E €E 咒F 萍F 逾G 唤G 芬H 枢H 捞I 鑫I 趁J 浇J 膀K 钩K 脊L 仲L 婷M 祛M 牲N юN 莞O 勃O 轰P 泊P 腊Q 赣Q ‍R 沃R 渣S 坝S 奢T 袖T 峻U 躁U 绑V 揉V 逝W 蓬W 氨X 醋X 玄Y 煌Y 涯Z 臣Z 琳[ 崩[ 拾\ 芦\ 饿] 帆] 漠^ 瓣^ 谊_ 吓_ 禄` 靖` Сa «a çb 勺b 菊c 酵c 懈d 帖d 萧e 赁e 枯f 瑜f 梗g 葬g 痒h 痰h 碌i 滤i 浏j 衬j 颇k 炭k 躲l 汕l 刮m 藤m 阜n ️n 渗o 绳o 蔓p 抄p åq 踩q 陀r 捧r 敦s 衫s 缠t 蕴t 牺u 尸u 誓v 沪v 弗w 轩w 杏x 扇x 辟y 绒y 裹z 禽z 嘛{ 阁{ ل| 薇| 闯} 遂} 匆~ 铸~ 笼 怡 旭 猎 伐 寂 喉 樱 沂 畜 罢 衷 敷 昏 掩 谅 溺 兆 ى 愤 嘱 劫 恭 烫 坤 蓉 吾 辱 甸 窃 宙 伞 蒂 浑 韦 栽 戚 � ф 邵 蕾 搅 呵 П 逛 茄 殿 唇 氢 扔 喻 坠 娟 诀 薯 伽 墅 卑 盯 酿 谭 庞 绣 肆 铃 挽 哀 剖 盔 榆 鹰 沫 â 吞 弦 凸 熙 祈 乾 灿 枕 桩 卸 舱 桐 宪 塌 隧 廷 坎 撰 肇 昭 泼 黏 舆 皱 轿 钉 侠 匙 卦 澡 裸 曰 贩 涩 纺 瞒 栓 丙 莹 秉 糟 丛 ñ 莎 橙 ④ 顽 湛 谦 颠 纱 碱 缅 磷 淫 凰 饶 奴 侨 妮 刹 雁¶ 磊¶ 竭ö 耍ö 惹Ķ 韧Ķ ВŶ 宵Ŷ 撕ƶ 挫ƶ 痴Ƕ 邢Ƕ 杞ȶ 媳ȶ 焊ɶ のɶ 摔ʶ 汾ʶ 屯˶ 逊˶ 剥̶ 仇̶ 疹Ͷ 崔Ͷ 渝ζ 愧ζ 羞϶ 凑϶ 仗ж 枫ж úѶ 郊Ѷ 尬Ҷ 捡Ҷ 溯Ӷ 蝶Ӷ 蝎Զ 琼Զ 澜ն 筷ն 叙ֶ 窄ֶ 叮׶ 罕׶ ěض 炫ض 僧ٶ 焕ٶ 砸ڶ 沾ڶ 橘۶ 昔۶ 蒲ܶ 烹ܶ 牡ݶ 沸ݶ 灶޶ 翁޶ 喘߶ 斩߶ 厢 臀 髓 冀 铝 澄 蚊 妃 ј 怖 い 尴 勘 棵 扛 怜 歇 矫 柯 慨 漆 ã 吁 螨 荔 筒 遣 闽   ™ 粪 雀 酥 勉 翅 旱 ă 氯 襄 К 堤 拦 ę 溜 帕 谎 弊 脖 郡 翰 沁 啤 闺 佐 颂 薛 庸 彦 奠 炖 屠 屁 壹 捉 斥 酶 蹲 削 衍 贿 瘾 搓 哟 阀 莓 潍 叛 ن 寡 琪 枸 赐 疮 彬 宰 砍 饺 屡 卿 ­ 糯 拐 钾 呕 ས 冶 奸 帐 狐 庚 膳 辐 渊 黔 硝 蚁 泗 瘟 骚 艇 吻 绽 ą 缸 殷 虐 汰 硅 蜀 撼 娶 ر 韶 甩 戈 麟 蛮 秤 崖 妖 隙! 锤! 杠" 漳" 勋# 歧# 粹$ 硫$ 烘% 淄% 绥& ā& 沐' 邹' 邱( ☟( 竖) 靓) 扯* 噪* 埔+ 婉+ 倩, 捏, 潇- 魄- 乖. 兜. 佑/ 呗/ 钥0 扁0 挚1 滥1 帘2 č2 穆3 な3 咪4 凹4 揽5 琦5 辜6 馅6 芜7 窑7 妄8 泵8 垒9 儒9 僵: 攒: 龟; 媚; 胺< 腌< 腕= 钦= 隶> 钮> 沥? 柿? م@ 驴@ 挨A たA 哄B 仕B 珊C 皂C 喀D 邑D 敞E 瑟E řF 雇F 瘀G 艘G 廖H 绎H 胰I 贼I 寞J 狭J らK ¥K 煞L 栖L 俭M 埠M 禹N 驳N 顷O 尹O 脐P 铅P 睿Q 蚀Q 羯R 熏R 烯S 楷S 嘲T 锈T 愚U 媛U 钠V 掏V šW 阐W あX ➤X 咋Y 谜Y НZ 笋Z 渭[ 拭[ 祷\ 榴\ 嚼] ت] 鲍^ 昙^ 萎_ 恤_ М` 楠` 袜a Рa 倦b 膨b 爪c 畔c 淑d 柠d 籽e 晃e 遏f 蕉f 庐g щg 莆h 堰h 撇i 崛i 疡j 耽j 陇k 锂k 洱l 趟l 屑m 郸m 夷n 汝n 逗o 嗨o 兹p 蔽p Аq 梧q 诠r 琐r 栗s 闸s 耻t 歪t 洽u ýu 砺v 焚v 苔w 雯w 聆x ►x 俞y 浊y 驿z 蜡z ي{ 掀{ 韭| 惫| ←} 巅} 橡~ 坟~ 梭 乒 蹄 俯 甄 祁 翡 踢 跪 嘎 粽 禾 敛 邯 瞄 鞍 窍 蛙 垢 剔 刃 稣 ′ 腑 惟 杉 檬 彤 蚕 이 擎 棘 蚂 弓 蔚 沛 傍 贞 葫 芹 嫂 扒 佬 竣 缀 颤 し 鞭 饥 て 萃 泸 挪 ś 菏 ག 纬 翘 ž 卉 倪 饪 饵 迭 菱 爹 夯 拇 涡 芙 睁 嵌 恳 舶 蹭 冤 裔 骏 卤 遴 逮 瞧 ø 耿 髦 谛 笨 侃 暮 و 豁 熔 匈 寅 肘 穗 撩 다 ² 棠 ż 菠 腥 缕 忻 斋 兮 拯 喇 佣 吟 ﹌ 酯 冥 黛 ї 铲 仑 昊 э 氮 茎 哺 ん ི 矮 孜 漓 棕 牟 ⑤ 讽 璐 郝 ß· 囤· 宛÷ 匿÷ 袍ķ 驼ķ 缆ŷ ོŷ 瀑Ʒ 昧Ʒ 侥Ƿ 荟Ƿ 贬ȷ 虞ȷ 晖ɷ 酮ɷ 皖ʷ 澎ʷ 坯˷ 咏˷ 昼̷ 葵̷ 湃ͷ 汲ͷ 沦η 嚣η 龚Ϸ ДϷ 渍з 暇з 睹ѷ 焰ѷ 绷ҷ 淤ҷ 哑ӷ ÷ӷ 瘫Է 棍Է 帜շ 藕շ 茨ַ 祀ַ 飙׷ Б׷ 紊ط 哨ط 鲤ٷ 淳ٷ αڷ æڷ 煲۷ 匪۷ 晾ܷ Оܷ 讳ݷ 毯ݷ 蠢޷ 濮޷ 叭߷ 殴߷ ê ད を 睐 巍 豹 稠 亨 捣 د 糙 诫 砥 樊 蹦 › 喧 盏 奕 烛 捺 觅 蝉 娥 奎 椰 睦 褐 梵 彪 绸 と 甫 苯 痹 岂  蝴 洼 √ 霖 垄 雍 摧 姥 窟 祠 娄 拎 矣 钞 拱 孵 憋 侈 泣 沭 胱 梓 朔 冉 淇 淹 亥 ″ に 岚 竿 闫 § は 眸 蘑 尧 绚 拢 厄 旷 霆 驭 ۛ 魁 „ 怠 舅 汹 筝 瞎 锌 垮 卒 É 爵 が 窦 弧 肛 寝 ≥ 趴 涝 ب 璧 鸦 靶 芋 姬 抒 卯 柚 磕 蚌 î 雌 硒 坷 廓 麒 陋 ە Т 妍 娅 瞻 ᠠ 峪 肽 藩 灼 咯 乓! 哇! 墩" 泾" བ# 芪# 柑$ 曦$ 榨% 躯% 骼& 絮& 靡' 荫' 玺( 绞( 缚) 妒) 巫* س* 幢+ 屎+ 捂, 苟, 苛- 汀- 叨. 衅. 哒/ 稚/ 笛0 ⊙0 畸1 蚝1 讶2 馒2 臻3  3 侍4 丫4 澈5 栈5 芭6 ♩6 Г7 ق7 吼8 禧8 藻9 阱9 歼: 颐: 肋; 柄; 妊< 骆< 痔= 涤= 芸> 弛> 嗓? 俏? 昕@ 槐@ 簿A 曙A 嗅B 崭B れC 帷C 屿D 娠D ìE ćE 眷F 痣F 倚G 犀G 酌H 靴H 惬I 兢I 茜J 芥J かK 炳K 绊L 滕L 啪M 荧M 乍N 猝N 瑾O རO 聂P 桔P 楞Q 榄Q 趾R 缔R 贱S 狸S ИT くT 圩U ۚU șV 呐V 哮W 焯W ёX ôX 寥Y 轼Y 琢Z 凳Z 酉[ 嚏[ 褪\ 龈\ 捆] 迦] 麓^ 噩^ 俺_ 梢_ 懵` 嫉` 釉a 堕a ۙb 祂b ངc 噢c 殡d 蜕d 苓e 挠e 赘f 氟f 蘸g 镀g 矢h 滇h 楂i ♪i 酚j 漱j るk 拂k 棱l 凄l 栅m みm 剁n 仆n 笃o òo 嗯p 醛p 杖q 倘q 胚r 哗r 琅s 揣s 🏻t 唠t ནu 涕u 篷v 砌v 乞w 踝w 蛛x 鳞x 咕y Уy 戊z 枉z ț{ ུ{ 庇| 垦| 悼} 涅} 霾~ 嘿~ 嗜 稽 ♫ 缉 脓 喵 搁 瞩 皓 亢 踊 咚 毗 霄 颅 幂 僚 藉 锣 阎 秆 碾 翩 酰 鲸 弈 ´ 戌 茗 炙 贮 Л ≤ 葩 烁 熄 啃 悍 惰 ま 碟 巳 嘘 檀 ‬ ा 焙 坞 瀚 ك ń 榕 鹊 〉 瑕 疤 逅 镶 缤 晦 璨 瞰 慑 隋 橱 邂 哎 矶 瘙 钧 峨 ▷ 诡 拙 μ ♬ ± 镁 揪 渤 眩 ù 粟 秃 斛  钰 焖 莘 ə ྱ 瞭 畴 俘 橄 掐 恍 衢 蜗 璀 蝇 鞠 帼 颍 肴 ᠢ 秸 柬 赎 岐 嘻 彝 雏 β འ ⑥ 檐 咀 篆 垣 绛 ✦ 撸 り 嘟 こ 岑 蠕 ‎ 忏 掺 嫖 崎 滁 窜¸ 唑¸ 〈ø 怼ø 癫ĸ うĸ 汶Ÿ 跋Ÿ ōƸ 阑Ƹ ëǸ īǸ 啸ȸ 岔ȸ 樟ɸ 馋ɸ 漯ʸ 娩ʸ 棺˸ Ф˸ 朽̸ 侮̸ 滔͸ 熠͸ 涟θ 暧θ 耘ϸ ۥϸ 을и 淌и 唾Ѹ 珑Ѹ 逍Ҹ 噬Ҹ 茵Ӹ 婿Ӹ 窥Ը 阮Ը 涧ո 聋ո 壬ָ ۇָ З׸ 尉׸ 渲ظ 嬉ظ 恺ٸ 婧ٸ 焉ڸ 哉ڸ 窖۸ 闵۸ 哩ܸ ůܸ 茯ݸ 晏ݸ ེ޸ 孢޸ 犁߸ 璋߸ 窒 烙 で 嵩 疱 ། さ 褶 厥 茹 ½ す 裴 躬 祺 痧 羁 腋 癸 覃 盎 迄 ش 赃 〇 辙 苞 斧 憨 呦 ‌ 秧 侄 锥 禺 猕 ɪ 陡 拧 菁 薏 宸 མ 膊 翟 ئ 钝 ― 茸 渺 郴 蕊 ο 抠 辗 恕 玮 煜 沽 ー 痊 拣 濒 铂 ᠨ 羔 荤 劈 斐 拽 ལ 陂 悸 徊 胳 ∶ 掠 缪 ᠭ 珀 舔 會 髋 伺 咙 하 僻 绮 靳 峥 佼 桓 र 皋 蔗 徘 豚 碘 ‖ き 咐 淼 炬 匮 睫 Ⅱ 黍 고 鹭 亳 潼 酪 ˈ 鲫 疚 绅 咧 簇 茧 掷 淅 襟 猩 憬 恬 锚 苷 绰 於 鬓 颊! 扉! 掰" 螃" 佰# 는# 漾$ 涮$ 喽% 铮% 掖& 娴& 玥' 喔' 昵( 芍( 蝠) 蜘) 钵* 遛* 秽+ 憧+ 凿, 耸, 薰- є- 炜. 溉. 茉/ 裳/ 轧0 捍0 谍1 眶1 亵2 茬2 锯3 烊3 钛4 痫4 譬5 莺5 愣6 懿6 挛7 ۡ7 噜8 溅8 猥9 慷9 ᵉ: 怅: ı; 嫣; 晟< 妞< 惶= ۟= 瞳> 囚> 摹? 羲? 萱@ 俐@ 舜A 怯A 寇B 眨B ᠤC 锵C 啼D 烷D 骁E ɡE ЕF っF 鹃G 蛰G 笙H 葆H 剃I  I 呜J 瓯J 溧K õK 毋L 蒿L 伶M 玖M 桦N 颌N عO 璇O 谧P 疵P 浒Q 辄Q 稼R 羹R 惦S 哼S 姗T ïT 凋U もU 痪V 痉V 恪W 钗W ्X कX 그Y 蝙Y 腱Z 鄙Z 莽[ 眺[ 沮\ 雹\ 穹] 钳] 庵^ 韬^ 绯_ 柒_ 刁` ه`  a 蛾a 阙b 鳌b 虔c 坍c 礴d 袅d 昱e 朦e 丞f 歹f 腮g τg 冕h 狄h 斓i だi 缨j 琉j 辍k ιk 飒l 栩l 蓟m یm 시n 帧n 胧o 屹o 祢p 鸥p ǎq ةq 钊r 哔r 槟s 啰s Ⅰt 俪t 芷u 撮u 鲨v 蝗v 恙w 胯w 砚x 缭x 匝y 诃y 어z 篱z 榻{ 钜{ ─| ő| 铬} 酋} 栾~ 祉~ 悴 淆 诏 憔 腩 煸 岱 皙 晗 瞪 纶 峙 à 椿 ́ ご 炊 慵 Ⅲ 窘 佟 谴 荼 毙 − 寰 镯 기 磺 苇 戎 捅 毓 ε Ö ز ū 睢 铿 匡 疙 ほ 邸 粑 撬 痢 珂 े 朐 敖 한 瘩 吝 嫦 蛤 呤 猿 唉 挟 呱 ❖ 가 霓 ν 惆 峭 ・ 凛 娓 酣 悯 嬛 镑 Х 墟 お 簧 粱 狡 했 ’ ᠯ ★ 阖 婪 缮 そ ف ✪ 嗦 지 赡 怂 砀 羌 ᠡ 崽 徙 嘌 љ 叩 挝 陨 Á 擀 筐 亟 丐 睾 孚 剿 ゆ 霏 ē 莒 潺 懦 憩 豌 扼 跻 抉 贰 寐 ⑦ 倔 ン 媲 า њ 遁 ، 磐 鸳 摒¹ ྲ¹ 姊ù 에ù 惮Ĺ 鳄Ĺ 淖Ź 樾Ź 皈ƹ 莅ƹ 宕ǹ 仄ǹ 蹬ȹ 夭ȹ 礁ɹ 辫ɹ 禀ʹ 廿ʹ 栀˹ 湄˹ 逞̹ 烨̹ Ч͹ 匾͹ 擒ι 黯ι 炽Ϲ 濡Ϲ 螂й 刨й πѹ 咔ѹ 隅ҹ ᵒҹ 谤ӹ ρӹ 罄Թ 驯Թ 粕չ 烽չ 瞅ֹ 曳ֹ 鄱׹ 狙׹ 澧ع 涓ع 郫ٹ 鹦ٹ 闰ڹ 驹ڹ י۹  ۹ 馍ܹ 蛳ܹ 煽ݹ 荞ݹ 俨޹ 甬޹ 豉߹ 吏߹ 鸯 觑 糜 潢 ◐ ✎ 琶 唏 拷 潞 芡 眯 镍 癖 芮 妾 པ 殃 藿 泓 捶 º 荠 癣 跷 扳 擂 锰 赓 箫 न 颓 沼 攘 赂 萤 谚 椭 ि 嵘 َ 苋 迢 嗣 犊 氓 ᵗ 呛 诲 璞 嚷 들 叽 褚 殆 婕 嚎 鹉 邃 勐 屉 ᠣ 纾 酝 滦 筱 骋 嗡 褒 蔑 脘 棣 貔 ᠦ 迸 萦 铵 悚 忱 娼 پ け 虏 σ 貅 璟 磋 轶 杷 镂 稷 ѕ え 踵 枇 Ш 癜 🙂 吒 瑙 ו 峦 圃 缜 蔻 硼 簸 ち 婺 婵 惋 स 渎 น 麽 해 惺 茴 袱 ‐ 盂 燎 铠 榔 서 篡 搂 煦 瑛 嘈 痿! ी! 瑚" 坂" 贻# 瓢# 歩$ 槃$ 喃% ど% 圭& Ø& 舵' 筠' 涿( 蜒( 대) 沓) 게* 嘶* 骇+ 婶+ 毂, 烃, 孰- 琥- 蜿. ❶. 壕/ 涎/ 哆0 赦0 ᠷ1 渚1 琛2 め2 厝3 藜3 谬4 兀4 つ5 茁5 †6 니6 叁7 镌7 拮8 蓓8 鞘9 œ9 镖: 馏: 炯; 龋; 涪< や< 褥= ่= 睑> 侬> ¢? 憎? 锄@ 浔@ 砧A スA 沅B 殉B κC 诬C 祟D 籁D 潦E 啬E 骸F 蟑F 蜓G 胛G 浚H 隘H ◑I 晤I 耒J รJ ♡K 罹K 嵊L 琵L 溥M 薅M 衙N 貂N มO 跤O جP 懊P 蜻Q 旌Q 淬R 藁R 岷S ☛S 瘴T मT 蛟U 藐U λV 祯V 铛W 嗑W 羚X 蔺X 饽Y 면Y 厮Z 姣Z 惭[ 吱[ Š\ 浣\ 妓] 壑] 汐^ 遐^ 奚_ 燊_ 箍` 拴` 诋a เa 葚b ³b 鱿c 馥c 汞d 铎d 胤e 蔷e 만f 鸾f 嗔g 彷g 笠h 蟠h 隽i ͟i 楹j 啧j 臃k 庶k 孽l 竺l 鸪m 嗷m 娑n €n ςo 札o 瞿p 疸p 蹊q ᵃq ﹏r ᠲr 羟s 상s 蛀t t 鹧u 아u 拈v 枞v 漩w 佘w 蜚x 捻x ⑧y 惚y 灞z 吡z 妩{ γ{ 嗝| 戍| 帛} 猾} 谆~ ǐ~ 沌 ง 喳 Ü 야 缙 揍 ∣ 鹑 膈 狩 郓 ح 幌 Č 崂 자 诣 ͞ ं 熹 弋 ⁿ 蛎 侗 粳 त む 釜 踮 爲 皿 驮 阆 리 ː 俑 ้ 祎 喱 ş 桨 도 อ 洮 谙 膛 漪 沱 儋 攸 曜 讴 ト ᠳ 姝 ❷ 鹌 氪 坻 芊 咎 傣 蔼 酗 垛 唧 η 辘 钴 皎 坨 臧 莴 痞 袄 罂 裆 ● 熨 诽 荥 ğ 珺 箴 醚 缎 ω 悖 疟 荨 爻 搐 诛 鸠 خ 唔 ˢ 囧 咫 沣 胫 歙 绢 ◈ 谩 踹 猖 疽 匣 蠡 氰 瘪 粼 笈 冽 炕 ล 骶 捋 樵 矜 诅 蚓 ڭ 嘀 ⁱ Э 吮 寮 ཁ 로 ᠬ 枭 偕º 膻º 주ú 纫ú غĺ 扦ĺ 喆ź लź 斟ƺ イƺ 嫔Ǻ 鼾Ǻ 艮Ⱥ 糠Ⱥ ᠰɺ पɺ ‮ʺ 踞ʺ 戟˺ 痼˺ 모̺ 谏̺ 馁ͺ 邳ͺ 琍κ 痤κ 咩Ϻ הϺ 鳖к 馄к 荀Ѻ 饨Ѻ 阪Һ 忡Һ 塾Ӻ 〖Ӻ 〗Ժ 哽Ժ 瀛պ 泞պ 鲲ֺ 를ֺ 虱׺ ✍׺ 蚯غ 吭غ 讷ٺ 啄ٺ 蟒ں 胥ں 柘ۺ 邬ۺ 鄞ܺ 搀ܺ ≠ݺ 徨ݺ 翱޺ 忐޺ 나ߺ 할ߺ ह 忑 畲 镰 骅 ћ 噱 ‑ 수 Å 嘹 炅 ‚ µ 偃 渥 ʳ 岖 珞 箔 ท 狈 巜 戾 ð 戮 萸 ᠪ 과 濠 淞 巽 ル 茱 裘 徉 亘 纂 Я 濛 砼 य 杳 鄢 槿 徜 骛 ั 讪 戛 叒 唬 皑 었 诶 恃 桢 缇 伎 ˡ 埵 일 υ 偈 أ ۈ 宦 甥 蕲 玑 喹 ี 涣 芩 诧 鬟 ᠵ 喋 榭 弾 ต 肱 溏 辕 颧 ᠶ 의 鳝 ྒ “ 岫 咄 ุ 鲈 چ 霁 膘 ก ר 마 陉 丶 것 兖 笺 羿 瓴 诟 ว 耦 帚 冗 嫡 昀 蚜 骥 邋 ל ې わ 榈 ो 迥 인 铣 蹴 瓮 菅 麸 ☾ 砰! 脯! 猬" 耨" 孺# 焱# 弼$ 吆$ 膺% 琨% І& 漉& 銮' ъ' 烩( ”( 诿) 邕) 雳* 疣* 叼+ 枳+ 盱, 안, よ- 湍- 噻. 擞. 噎/ 鲅/ 矗0 撂0 ≈1 菡1 菖2 奄2 گ3 リ3 疝4 噶4 痈5 拗5 ཞ6 鳗6 飕7 芎7 阂8 Φ8 袂9 ཀ9 吖: 미: 蜱; ת; 蓦< ¼< 诩= א= 郧> 邺> 町? व? 肮@ 褂@ δA ❸A 铆B 幄B 苒C 烬C 泱D 霹D 枷E 咆E F 거F 蛊G ⃣G 攥H ཚH ЦI ཡI 吩J 遢J 饕K ◉K 佚L 胭L 邗M —M 抨N 滘N 은O 汴O 朕P 遨P ʰQ 怦Q 罔R 旎R 晔S 丕S 恣T 佗T 旖U 袤U 眙V 峒V 蜷W 暄W 谯X 偎X 琊Y ยY 囍Z 蟾Z ⑨[ 卞[ ǔ\ 仝\ 殇] ź] ด^ 猷^ 坳_ 阡_ 徇` ǒ` 迂a 醴a 漕b 啶b 있c 幡c 跆d 诙d 锭e 罡e 裱f 娆f 邈g Óg ᠴh 焗h ᠮi 瘠i दj 泯j 痂k 耙k ✿l 锲l 袒m 娲m 愫n 摁n 사o 요o 舀p 鸢p 谌q 頫q 醺r ‰r 鳅s 幺s 涞t ∠t 르u 箕u 腓v 缈v 髂w 挎w じx 嗒x 살y 溴y ➢z ‹z 瞌{ ཆ{ Ⅳ| ค| ❀} 垭} ב~ 讹~ 隍 篝 岌 吠 讥 ๚ 蹿 瞥 矾 塬 처 พ 缥 冢 蕃 荚 든 臆 毡 া ラ 糍 ¿ 스 奂 涸 蕙 飓 Î 별 弩 蚤 嗖 桡 ₹ Ä 독 捎 唆 邛 臼 昇 苡 郯 埂 מ 氲 辊 歆 ช 藓 ク ۗ 垠 锹 Ž 掂 嘅 擘 秣 ═ 腆 บ ส 钼 筏 钨 θ 扈 伫 원 ɑ 钣 芾 簪 砾 蟆 髎 위 纭 嬴 ص 篓 瘢 劾 窿 간 へ 郏 茼 訇 苁 獭 蝌 叕 ッ 汨 惘 氤 蹒 ί 徵 蕨 甭 醍 挞 ฅ 킁 佢 荏 嬷 浠 翎 阈 綦 窈 旻 铢 ש 珲 嗤 蚪 ア 濑 浜 琰 ะ 崆 跚 哧 嗪 蜈 瘁 Ж đ» 盅» 劵û 麝û 珏Ļ 黝Ļ 珈Ż ไŻ ▣ƻ 搪ƻ 叱ǻ 제ǻ ねȻ ΔȻ 嚓ɻ İɻ ِʻ 蹉ʻ 苣˻ Ś˻ 裨̻ 瘘̻ 垌ͻ 眈ͻ 러λ 𝓼λ fiϻ 鹜ϻ せл طл 窨ѻ 骞ѻ 睬һ ๑һ 孪ӻ 锢ӻ 鳍Ի 婀Ի ۋջ 夙ջ کֻ 醐ֻ Ú׻ ᵘ׻ 龛ػ 笆ػ ྐٻ 呷ٻ ︰ڻ ھڻ 犷ۻ 啵ۻ 珅ܻ 郦ܻ 茌ݻ ᠩݻ 跎޻ ज޻ ¡߻ 餮߻ ʸ 仟 畈 匕 ¾ 呃 墒 踌 鄠 赭 ْ 赊 娣 แ 嶂 晁 남 绶 № 杵 ⑴ 腧 ょ 锏 佤 埇 ¹ 湟 猗 铤 ᵈ 脍 峯 芗 铐 镳 囱 ¶ ཐ ず 泠 髻 ิ 躇 酐 소 咦 내 ็ 槌 ⑵ 煨 炀 噗 蔫 脲 邡 腈 谟 ʊ • ๐ 濂 腭 呻 灏 ཕ ά 鲢 啉 牦 많 骊 咛 蚣 纣 陛 라 儡 陟 ณ û 抿 전 ब 陲 ห 聿 渑 掸 鳕 ™ 衾 栉 ๎ 飚 胍 荃 蜇 颦 咤 翌 ฏ ︎ ђ ข 窕 碣 瘸 骡 ʃ ᠎ 皴 ɔ 闱 轲 비 迩 滢 ү 犒 ʻ 汆 箩 敕 Ð ☜ 螳 鹄 ธ 谕! 髌! 尕" 연" 摞# 堑# 甙$ φ$ 粿% 跺% 拄& À& ⑩' 嘤' 腼( Ω( 蓑) 翊) 드* ư* 雉+ ु+ 啾, 珐, 怔- 偌- ๖. ป. 황/ 闾/ 碉0 埭0 โ1 撷1 ‭2 腴2 ᠋3 ์3 桉4 蛹4 霭5 祐5 剐6 仨6 樽7 掬7 鞣8 侏8 镉9 무9 珩: 冼: 蹶; 浃; 呋< 镐< 柞= 奘= 琏> 崃> ग? ்? 岙@ 보@ 萏A 帏A 搔B 膦B 筵C ❹C 焘D ใD 馕E 镕E 阚F 剽F ňG աG 啫H ﹑H রI 睇I 遒J │J 掇K 婢K 凇L 鹂L 蹋M ُM ฌN ึN 翳O ÇO 蟋P 颚P ЮQ 耄Q 啕R 반R ªS 傀S 瓤T 蟀T 屌U 耋U 濉V 噔V 嗳W 蹚W 茏X 浐X 掣Y 榛Y 癀Z 箸Z 忿[ 〡[ 楔\ 鏖\ 跛] 노] ό^ 鸵^ 경_ 疃_ ู` 糗` 贲a ླa ྟb 嘧b 茭c ¬c 珙d 叟d 嘭e 逵e 绫f 饴f 豇g 와g 稞h 妲h 哋i 睒i ぎj 泮j 徕k 妤k ❤l 狞l 阉m 𝓮m 榫n จn 恿o 盹o 倭p ✓p 砷q タq 甾r 赈r 唰s 谪s ロt 荸t 纨u 轳u 痱v 懋v ¦w 桀w 珉x 棂x ্y 愕y 鹳z 姹z 噼{ נ{ ע| ⋯| ض} 跖} 谥~ ❉~ 髅 罘 扪 Í Ł ฿ 蜥 獗 箐 芈 垚 シ 萼 衩 隼 ♦ 汩 鲟 糅 – 馗 赟 匍 璜 搡 澍 蚩 ✘ 력 颉 ▐ ฺ 까 掳 祜 감 仞 斡 犇 赉 당 滟 崴 려 ű 蛆 硚 镭 麾 건 ） ד Α 鄄 ۆ ษ 缱 偻 匐 嘞 嘚 邝 ﹞ ﹝ 咿 ✕ 않 桧 （ 때 潸 毐 ʷ 呲 碚 ื 绻 馀 惇 鎏 😉 婊 痨 臊 ¨ 飨 铰 ド ཉ 沏 崮 炁 骷 牍 饯 夔 ث 蒌 단 犍 鳃 ღ 우 恻 刍 颞 査 狰 扃 여 역 ば 铀 觞 勠 ⑶ 俸 
 Ż 胄 魇 ๔ 麋 谒 ཏ 濯 晌 결 È 骰 민 芃 肓 梆 瓒¼ ￣¼ 瞠ü 泷ü ☉ļ 睽ļ ๊ż टż έƼ 潴Ƽ コǼ 槎Ǽ 洵ȼ 迤ȼ 葭ɼ Εɼ 锆ʼ 鲶ʼ 嗟˼ ে˼ 문̼ ‧̼ 荪ͼ 绦ͼ 난μ 缰μ 筚ϼ 攫ϼ 莠м 昶м 秭Ѽ ཅѼ ÑҼ 赳Ҽ 钤Ӽ 唢Ӽ 디Լ 瞋Լ 萋ռ 簌ռ 워ּ 飗ּ 囿׼ 醮׼ 孳ؼ शؼ ถټ ᶠټ 砒ڼ 郢ڼ カۼ 耆ۼ ཤܼ ♥ܼ 鳜ݼ 骐ݼ ･޼ 받޼ 谑߼ إ߼ 毽 フ 키 ή ⋅ 생 佝 撵 饷 ■ 炝 呸 화 稔 ๆ 彧 圪 ▨ 砭 惴 던 壅 疥 啮 殒 늘 ▸ 腘 ▾ ཟ 舫 菟 忒 錾 稹 필 ি 蓥 Σ 犸 ๅ プ ם ฒ ᵐ 蒡 楣 跄 岘 ệ 牒 았 절 름 囡 霎 铄 ジ ̀ ʌ 으 レ 堃 炔 蝈 蘖 膑 ผ 缲 망 蜊 篑 瑄 薹 寳 ∨ █ ろ 螟 谄 ♀ 畦 づ 喙 진 萜 ❺ 줄 楫 ̈ ฑ χ 악 舸 苜 淦 蚱 Ј ⇩ 嗲 ė 峄 湮 험 ❦ ๙ ᵖ 敝 ᵍ 桁 蓼 メ 럼 衮 缄 참 郜 暹 ّ 렇 マ 讫 獒 椹 ∞ 麂 氩 정 隰 觊 镓! ฟ! ै" ᠥ" 冇# 纰# ๓$ 踉$ 성% 쓰% 嵇& 힘& ⼀' 揖' 桎( ฐ( ำ) 坭) 렸* 潋* 澹+ 癞+ ㅤ, ন, ۜ- 岀- 炷. ذ. 绀/ ๒/ 弑0 玷0 梏1 趸1 驸2 靥2 긴3 抡3 螯4 酞4 熵5 羸5 놓6 汊6 홍7 剌7 苕8 鞅8 분9 栎9 ა: 蓿: 煊; 심; 엄< 왜< 篁= 胪= 蛭> 赅> 蛔? 쳐? 诰@ 觎@ 铉A 诘A חB 旸B ¯C 彗C ـD 砝D 钎E 噌E ▬F ¤F ĐG 깊G ̶H 榷H 抻I 臾I 냥J 耷J 庾K ๗K 疖L 幔L 钒M 锺M 骜N 되N 鹈O 濞O 桅P 虢P ๘Q चQ 杈R 滓R 赝S 덧S 뿌T 荻T ぶU テU 琮V 颔V 颢W 浉W 胗X 锉X 恽Y 淙Y 仃Z 垸Z 畿[ 국[ ﹐\ ゃ\ 鹕] 宥] ฬ^ 殓^ ¸_ 籼_ 甯` ᠸ` 吽a 窠a 铨b 부b 장c იc 苄d 壃d 镊e 嶙e 琬f 舷f 璎g 佯g 怆h 诤h 躏i ּi Οj 桠j 郗k 雎k 锴l 燮l 谀m 煅m 鼬n 嘣n 皲o ƒo ₂p 莪p 蝼q 嘢q 垅r 명r ʒs 俚s 堇t 咗t 峋u כu ˜v 説v Œw 〜w 孑x 悌x 蹂y 鲳y פz ｡z 咁{ 葳{ ❂| 洙| 腠} 佶} 聽~ 啜~ 蓖 掴 ィ ྤ ､ ฎ ศ 蕤 ˌ ญ Й 啖 祇 鬃 眦 囔 據 崀 ⇒ 溟 倏 裟 ∙ 신 蛏 溆 ế 殚 祚 鲑 ɒ 犟 羧 酊 舛 踱 ❥ 蜃 ฉ 垟 颡 Ι ն ŋ 龌 ק Ş 黟 崧 ひ 峤 蜴 袈 ء 더 복 沤 颛 庖 伢 洄 佃 蹙 अ 傈 咂 谗 バ ᶜ グ 磬 讣 ₤ 聩 भ サ オ 唁 Ξ ⁄ ʿ Π 俾 洺 绾 撅 ウ 孱 伉 습 ⑷ ∴ ए ָ 哐 襁 釆 钺 ক 龊 跬 ナ ▼ 玟 蜍 🏼 轭 哌 墉 気 搽 ड 僳 ू 钿 岢 キ 矸 芘 蒹 谲 亓 靛 懑 橼 ব 醪 ө 宓 卅 ‡ ム デ 钏 箓½ आ½ 호ý ☆ý 衄Ľ 囵Ľ 捌Ž 疔Ž 鲛ƽ 燚ƽ 谶ǽ 물ǽ ᵇȽ 锶Ƚ 渌ɽ 镛ɽ 오ʽ 翦ʽ 恸˽ 鳢˽ ซ̽ 滏̽ 锷ͽ 娉ͽ 笏ν 蹑ν 钡Ͻ 続Ͻ 豺н ﹊н 泫ѽ 鹫ѽ ☽ҽ 髁ҽ 刽ӽ 莜ӽ ΤԽ ‏Խ ニս ฝս 芨ֽ 跹ֽ ť׽ 杲׽ 酩ؽ 浈ؽ 铧ٽ 瞑ٽ 蚧ڽ րڽ 戬۽ 褓۽ 涠ܽ 碴ܽ 菀ݽ ύݽ 锟޽ ゅ޽ 锗߽ 橹߽ 咻 獐 痍 氦 滂 ∇ 晞 钯 중 遑 盥 ﹣ 旮 缦 鼹 獾 橇 韫 ạ 凫 萘 囯 骧 馊 蛐 箬 湫 缢 ๋ 각 璠 驷 질 倜 히 嚯 鹮 ᵛ ๏ ☏ 학 鳙 ְ ♮ 琚 교 昉  ✧ 繇 馐 垆 翀 ├ ி 淝 적 ḥ ơ 篦 ➣ ⁠ ₴ Κ 捱 铖 镣 ブ 饬 チ 구 喺 増 柩 椁 ❁ ฆ 파 ⠀ 篪 傥 虬 ⼈ ビ 공 Λ 달 ➥ 锑 瀍   Ⅴ 雒 트 囫 க 傩 昝 牯 绔 迳 骈 磴 辋 岿 殁 ত 铡 ↩ 獠 舂 ס 啲 隊 垩 叵 阕 绡 蹇 葺 椤 穰 芫 砣 邨 Æ 蒗 辇 ふ ן 珥! ❻! 硌" 椽" 谔# 잘# 倬$ 謇$ Є% 語% 劭& ฤ& ∆' ̣' ध( ո( ▪) 旯) 囹* 觥* 唵+ パ+ 菔, 箘, Μ- 锒- 镗. 俎. 枥/ ➧/ 瞟0 郄0 같1 圄1 楝2 열2 ♂3 げ3 篙4  4 哚5 悻5 咣6 氡6 貉7 粲7 ‚8 笕8 ်9 😀9 庠: 沚: 瘿; 觐; 동< ˙< 褴= 준= 褛> 黜> ả? 疳? 吲@ 徳@ 槲A ュA ₳B 세B 鹬C ྩC 儆D 鲷D 龅E 埕E 僮F ྡF 변G 揩G 峁H びH 蚬I ுI खJ 忾J 埗K エK 蘅L 谡L 弭M ํM 垓N 舐N ☯O 逶O 蓁P ִP 齁Q ミQ 玳R 馔R 용S ˆS آT 嵴T 못U 哏U ๕V ַV ეW ⭐W 忤X 缬X ャY தY ŘZ 骺Z ∕[ ➝[ 牠\ 栟\ 溱] 郾] 鬆^ 逯^ স_ ∼_ 矍` 鬣` 嘬a 怛a 린b 斫b 榉c 擤c 碛d ლd 抟e 篾e 茔f षf 겠g ṣg 囗h 槁h 饧i 珣i 矽j ྗj טk 炆k Ōl 犄l 艹m 衿m 霰n 祗n মo ⑪o 유p ộp 摈q 론q 硐r 関r 鄯s थs 泺t 埝t 姒u 曈u 棹v 涔v ɛw 锃w 顼x 관x 邰y 喏y 跶z ‪z ۫{ ✔{ 聒| 讧| 螭} 蒺} ƒ~ 竽~ 倌 প 孒 ᴵ 噙 ল 莳 滹 踽 👼 甑 悭 극 唎 阗 孀 ノ 苫 べ 碓 颏 燧 嵬 າ 삶 빠 ฮ 행  啭 嬅 瑁 仵 肼 铋 擢 鸮 妫 蒯 轱 砟 鹞 ◁ ⼤ 瓠 楸 ཛ 绺 郅 œ ྣ 鞑 입 ե 髯 僖 치 欹 玦 醌 ❼ ぷ 견 鳊 妣 湎 哞 艽 鲮 瑗 闳 ώ 対 恹 ˚ 邙 塍 崑 枋 ג ⑸ 垡 茆 ᠂ 胝 Η ♢ 発 嬗 锨 🛒 ÿ 鼋 怵 방 糹 肄 猇 ▧ ▶ 趵 ダ 廪 稗 ✚ ʼ 咘 甪 ✰ 镬 욱 ョ 터 इ 噁 轸 佞 钕 魑 泔 ﾉ 唳 怏 忪 ố 悱 沩 暝 绉 영 誊 疠 檩 ण 笄¾ 旃¾ 谵þ 妪þ 鸬ľ 锱ľ 끌ž 埼ž ⽣ƾ 円ƾ ツǾ 鸸Ǿ ﹒Ⱦ 蒽Ⱦ 苎ɾ 刎ɾ ນʾ 恚ʾ 냈˾ ภ˾ 荽̾ 沆̾ 薤; 咲; 穑ξ 挈ξ 剜Ͼ ◡Ͼ ﹕о ≧о மѾ 睨Ѿ 乜Ҿ 벗Ҿ 菽Ӿ 낼Ӿ ɜԾ 俟Ծ 疴վ 蜉վ 바־ Ν־ 枧׾ 砗׾ 埙ؾ 癔ؾ 조پ þپ 娌ھ ‘ھ 妯۾ 劼۾ 硖ܾ 禛ܾ 瑷ݾ 琯ݾ └޾ ්޾ 阊߾ ს߾ ➜ 뎌 杼 鳏 음 囟 ﾟ 鸩 圻 県 । 爨 ի 醅 업 沔 魉 운 ṇ 瘕 珪 ∩ Ÿ 卍 ❽ 莼 fl ↳ 噤 哙 璘 鸨 醯 セ რ 狒 苈 掮 雩 骀 년 缛 ⑫ 爰 绌 ヽ 恁   ‒ 逄 叻 浍 袪 厩 呓 鲩 勖 嵋 ガ Ђ फ 樯 צ 垵 식 闇 鸷 頔 ⇌ 唷 黠 ズ 洧 喑 肟 말 ྷ 缶 ா 诨 铱 阄 😂 ➹ 墀 堌 嵛 礽 苌 磲 ྨ ờ ﹋ 镝 阇 鲵 粄 钹 侪 榧 豸 窣 嚟 挲 唛 砻 嫘 禳 ζ 捭 糁 剡 ً ớ 磻 鹘 怙 趺 脒 忖 訾 葶 ネ 佻 ➨ 毖 衲 ྙ 鹋 黉 〃! ➔! 碜" 茕" 苻# 変# 계$ 焜$ ւ% 旳% 垴& 瘰& 鲠' 腚' 魍( 檗( 亾) ᵏ) 铳* Ā* 塆+ 髀+ 합, ❾, 欸- ப- 裾. 犽. 堀/ 蹰/ 愠0 骠0 ཇ1 ◕1 孬2 菓2 诒3 鹚3 嬢4 揠4 蠹5 闼5 ہ6 卟6  7 圜7 洹8 愆8 髙9 槭9 담: ﹥: 嫚; 垱; 선< 荇< 実= 睚= ฃ> 屐> Ș? 蕞? 媪@ ṭ@ 蹩A ェA 仫B 愍B ظC ịC 逦D නD 仡E 蛄E 개F 疬F ිG 鳟G 掼H 颀H 邾I 箧I 蝽J ấJ 琇K 弁K 저L 杓L 氿M 簋M 菘N যN זO 畹O 窸P 실P 怿Q 람Q 鋆R 喟R 龇S 睥S ͡T 殸T 蕖U ΡU 碶V 俣V 瑀W ÞW 氚X 孖X 毎Y ░Y 俶Z 噫Z 柰[ 晩[ 燔\ း\ ̄] 湜] দ^ 饸^ 🔥_ 薮_ 塱` 踟` 체a ξa 偲b 桷b 骘c 玙c 箪d 豢d 蛉e 缫e ぐf 曌f 鹇g ベg 회h ❍h 戕i 晷i 檄j ֹj 婳k 诳k 육l 铍l 荜m उm 蕈n ரn ✤o 洎o 翕p 欤p 狍q 贽q 哝r 笪r 诮s 刈s 椴t 蝣t 양u 诂u 凼v 🚀v 笳w 钽w 酢x 匏x 狲y 蹼y 嗉z ✖z 辎{ 鬻{ 唻| 楽| 秾} ∀} 遽~ モ~ 嚧 庑 簟 ワ 蜢 ო ầ 掤 畀 钲 埚 楮 ポ 叡 霈 煴 ∥ 怄 黢 桕 鹗 ু 庹 寤 茑 蚶 ई ✬ 〞 ざ 徂 카 ď 氽 潆 ⑬ 塚 滠 挹 蕅 蟥 데 ව ㎝ 覅 ピ 隈 樨 ़ ് 辔 砦 瀣 鲎 ཙ ᴇ 饹 ⁣ 啷 弢 🤔 鳔 강 広 ট ≡ 蕻 嫪 磦 ᴗ 碁 奁 ֶ 処 ậ 蚴 嗫 Γ ▲ 轫 僭 愎 ♭ 粝 漈 隗 媾 韪 闿 戦 髡 菰 Ḥ ケ 苴 🧧 ے 盃 跏 谂 娈 铩 ọ ာ 艿 ∵ 撺 刿 虻 馓 ฯ 漷 鲱 ⽤ 臬 胴 俦 範 洌 ✅ 🇺 예 ề শ ྔ 蛱 铌 氐 巯 통 ⇧ ハ 缟 Ⅵ ¿ 侩¿ 栢ÿ 経ÿ 笤Ŀ 乂Ŀ 胼ſ 跗ſ 없ƿ ۦƿ ❏ǿ 俅ǿ 绗ȿ ⒈ȿ 洇ɿ 戗ɿ 昴ʿ ⇓ʿ 嶷˿ 岜˿ 蝮̿ 焐̿ 📅Ϳ 枰Ϳ 闩ο 鸱ο 豕Ͽ 綫Ͽ 诜п 箅п 啻ѿ 揄ѿ 纡ҿ ⇋ҿ ̃ӿ ཨӿ ؤԿ 巿Կ 黒տ 呒տ 抔ֿ 揶ֿ  ׿ ฦ׿ 玢ؿ 鲀ؿ 疍ٿ ⑭ٿ 氘ڿ 骝ڿ 佥ۿ 铯ۿ 栊ܿ 🙏ܿ 牖ݿ 谘ݿ 栌޿ ⒉޿ ✩߿ 厾߿ 禤 ⑹ ட 缂 榀 皤 ァ 鲭 秕 徭 ︕ 淠 逋 迨 睺 學 煳 嫱 笞 瘆 崤 玏 🇸 ⊕ 樘 狎 钇 ≦ 작 诌 驽 洸 蘼 镧 黩 Ô 翥 込 夼 捯 镫 鼐 湉 涑 陞 氵 ʏ ủ ✨ 蛴 ţ 勰 噘 տ 凊 வ ᵀ 垧 후 様 竑 Ê 榘 🏜 😊 砜 锔 瑭 嗞 浥 ♯ ❒ ⋆ ᴀ 殄 며 贠 된 喾 尻 烜 飏 泅 鲥 鲧 蓣 玠 좋 🐣 咜 ⃝ 배 Υ 铙 応 ֵ 蛲 월 յ 簕 蛸 ㄑ  ⾏ ể 铟 매 々 등 炘 მ 嫒 🧹 ᠱ 坼 ⁡ ザ 伥 汜 缯 孛 侘 俳 苾 萁 鬲 ⁰ ʔ Щ ක 缑! 哕! 诓" 羰" ༢# 楦# ☼$ ⽅$ 骢% ⊥% 鹍& 谞& 埸' 茚' 靼( 馑( 鮰) 岽) 岬* 籀* 镔+ ້+ ல, 薷, 鸶- 錤- 래. 嵯. 枨/ 莨/ 龉0 औ0 ∈1 ய1 卮2 琤2 ợ3 厍3 ˇ4 緺4 ♧5 ➱5 讦6 ồ6 窢7 魃7 랑8 ホ8 姮9 糸9 逑: 焓: 酆; ₱; 衽< ソ< 翚= մ= ▵> ↂ> 啁? ׁ? 鹩@ ⁢@ 발A ုA 氙B 圉B 〝C 妁C 镠D ☻D ボE 재E ✭F 缁F 嗰G 嫄G 埌H 墘H ΧI 邴I  J 镒J ྦK 箜K ʾL 罍L ❐M 豳M 鸫N 聃N ⑮O 赧O 荦P 襦P 嗬Q 纥Q 襞R 浯R 筼S ✫S 侑T 椟T ნU 铑U 격V 鳐V 齑W ➽W ⾼X 🤣X 圹Y 佾Y ່Z 忉Z 褔[ 龃[ 荑\ জ\ 铷] 衞] կ^ ễ^ 恂_ ґ_ ぼ` 산` 阏a 竦a ║b 卣b 钬c 恵c 鲇d 嬿d ㎜e 翮e 熘f 坩f ◄g 현g 売h 桴h 鸲i 搵i Ⅹj 觚j 酽k ɞk 録l ゴl 篌m 聍m ბn 프n 丼o 蘧o ffp 郇p 廆q 恫q 琎r 磡r 芣s 嫲s ກt ཧt 堙u 玎u 籺v 镦v 함w 漖w 硷x ̆x යy 氹y ⼒z 國z 駅{ ো{ ⼦| 髫| 설} ◍} 㕛~ 笹~ 🤩 🧡 佷 踯 呔 ➊ မ গ 珵 ன 垕 葑 澴 蒴 筮 旰 筌 帔 铊 ச 揿  ौ ペ 桫 艄 알 囝 ⾯ 瘗 타 牝 ʜ 蹁 椋 韡 ⚑ 揆 酹 伝 慜 ງ 趄 龢 筜 饫 👏 ్ ╹ 怍 ම 時 趔 ေ 钭 ᵕ 萆 ᠃ 髃 Õ ➛ 糌 螵 ვ 沄 ▁- ▁= ="@S ttps ▁+ ▁-- fic  ▁diffic! ularly) ▁ful0  ▁charg2 tty4 --B ->F ==O urningW  ividuals]  ▁experiq aith~ ▁ww  ▁appe ▁=>( ▁gh؏ rowd abase ▁==0  ▁spok(  ▁lawy arshЧ ionals ortersЭ  aturingد ▁我P ▁在h  ▁brows@ secut ommodX  istration razil ylv  ▁cerem` ▁一 ==== covery0  ferences iamond  asingly vely  ▁licens  repreneursx  ▁affil8 razy iscons  ▁brill\ ytics ▁(+ OVp ++P ▁/**  ▁Independ  entures =' ▁你( uanat  ҈҉҉҉҈҉҉҉4 ▁大$ ▁iP$  issioner  ҈҉҉҈ ijuana ▁///  ▁如何l ▁第 klah  repreneurD ▁三D  klahoma4 ▁BMX xic4! ▁有! obody"  ighters@# TML# raigx% embers ' ▁->* pring *  ▁bact0, ▁小d, veland/ ellows/  ionally2 ▁二T2  umental2 ▁新2  ▁unfort3 ugh|3  ▁Unfort5 iffer6 CAA6 atured6  roversial 7 attan7  ▁Pitts8  ▁prosecut(8 ▁这8 isd9  secutive4:  ▁中国< hoods= ▁wisx?  ▁如果lA ▁TesA ▁中C  ========G  ▁uncom ▁他B ++)  ▁sher ▁<-  ▁关于 ▁金 auses entary¬ ▁当Ь ▁月l uant тор alter( +,t ickrЯ wagen  ▁编辑 )+^ ▁"""t  ▁因为 ▁周 ▁元 ▁日ޱ  ▁如何看待H ▁黄R  ▁repetb ▁是 ▁上  ▁这个@  issioners ▁enz  ennialsb  ▁Observ  ▁municip ▁ét  ▁ChallengR =>  ▁北京η ctoralҷ aft  ▁suspic ▁nep, enniumZ ▁爱  ▁今日R  ighting  ▁treas  ▁现在  ▁今天  ▁unchȽ uadorν ▁$('r  ▁unforget  ▁Mahar ▁Pip:  ▁对于 ▁想  assionateƿ  aceuticals  ▁NeighborX BOUTt ANC compet  ionalism: ▁杨  ▁其实  rections )=P  ▁如何评价~  ▁根据 otersF iently ▁可H céJ ▁海\ tersonh legatev ▁万 rave  ▁来源~ ilem  itioned rals  ▁上海  ▁一个 iality Academ -$V  ▁Clem inical embr@  ▁interdiscL ertingP ▁微 ▁摄 riber> ▁用 odic ▁老 ovable  essionals ▁(-d usted  portation  ▁Cerem,  ▁最后T OUND* IFF |- Applic ▁最,  ▁第四N  ▁Perman  ▁Introdu$  anthropyB  ▁Scand ▁马*  ▁目前0  ▁Chrys6  uscriptsR ▁白h  funding  ▁unsuccess afety~ orers =( ▁再j %-$  otationsB ▁ ytic* ▁FIRv  ▁Gast ▁六 ▁开@ :-V ▁文v ▁以  CongratP  ▁neckl /-  ▁redevelopD ▁多 usting  ▁surn thened  oqiongma ▁华( ▁美 ▁全 uaniaj ashtra ▁西 ▁好|  ▁embroider -\  castersP ▁要` ▁́sj ▁GRO  ▁Authent ▁ScoX ▁安z  ▁Recruit DOC  ▁sincerh ▁无j  ▁descript ▁看  ▁Beyon  embrance  ructional@  ▁dilem arend ▁UttV  ▁disgn  ▁Forg ▁请2  ▁CognZ  ▁Prosecutl ▁apr ～   ̅ ➕F ◆ 後5 ̊9 ▽  ○ ̇v ゚ ͏ 為 ̧. ়; ※R 個k ❗ ⬇ 動 □ ➡ 書  來  ◎ 無< ⚠? ╯B ✯[ 業c ₦n 體v Ì 東 開 這 ❯ 間 ☺ თ Ć 😍" 電( Ò- Β8 დ: ℗< 場K սM ╰] △h ◇l 報m ņr ╭t 與y 現 ų ಥ ᅠ ŏ Θ ʹ 見 ╮ ♠ ̥ ħ ⇐ ں  ள 過 ľ ை 👍 長 們 👉 ા  უ 麼 Ë छ$ ಡ* আ, ☀- ‟1 機G ॉM ̩O ЫQ රR 🙁X ীZ ා_ Ьa ━b Աf 術h 記i 發m 👇p 🏾q ್y ۖ ∗ ⚡ 對 資 Û হ 問 ̓ 🏽 論 🙌 Ï ◦ 🇷 ψ ♣ ך ን 經 크 품 選 內 ු ਾ ტ 說 ற ❌ Ґ  ठ  ắ 너 토 🇦 ṛ  ు" 🏆$ 및& 🚨- ↵1 포4 চ9 ත: G રP ្R ृW 🇪Z ংd ̫i 📸t 沒v վw լ  ˊ 출 ứ 🇧 ✌ 愛 ͈ 題 џ ড 💙 ា ٹ ස 👀 ి 設 馬 🎉 ந ∑ ụ 話 風 … 렴 ũ 레 ⟩ ォ 員 種 금 눈 브 😛 ర ⁩ ල 點 ̴ ই  門  ˜  ా ക ँ թ 別 ̋" 關' 龍( ṃ) ္- ἐ/ 進1 ∂4 გ7 實8 計9 Հ; ḍ< ցD ЊF դI 💪L ⟶M ☑O ിR ∫Z 結\ 김^ Ýd 😭j 車p ̲r 울t ˋu ⭕v ギx இy  գ घ 還 當 კ 義 һ ∧ 類 հ ද 속 추 🇬 灣 ĭ 號 ǫ Џ 💰 務 ⟵ 樂 💕 ት ✈ ढ 區 編 ರ ❓ এ 버 ষ ʕ 료 ხ 번 ἀ ಿ 華 ☐ 급 ප 際 ల ⚽ შ ὶ 최 ছ ヾ 師 ў ⇨ ġ  খ  ാ  😁  က ը ↔ 메 ી 네 ̳ ự 軍/ 🌟0 連4 🇮: 製A ởC 読E 🇨G ਰH ÙP ⁦Q 코T ழV ‣Y ユZ 數[ င^ រ_ ᴛ` ভd ගe 두k ̤l ̷n அq ᴏr ◊s 島w ಾx ̱z 페} Ő 른 온 ŭ 畊 線 ു ɴ న ດ ヒ 💜 ↗ ന 張 ヤ 집 ෙ ం ゲ ੀ ổ 魈 থ 検 🇳 ્ 陳 박 ✊ 록 ĕ క 館 ქ 법 ĩ 날 ✉ ❄ 🖤 ۱ 沢 형 😎 ိ 應 運 😦 頭 ண ữ ┻ 🎥 💥 Ծ 網 천 ේ 임  직  또  ῦ  ◯ 표 랜 차 먹 ὸ Ψ  ք! Ī$ ʟ& ပ( 評+ ղ- န. 樣/ ぜ1 Ț5 ļ9 験; හ< ḵ= 본A 價B 習C অE ☰F 📌H 📷I 氣L 🔴N ṢO 능Q ັR 피V တW ⟨X 목Z 🥰[ ʁ\ է] 產^ 請_ 遊` 🤗a ѝb ☕c ʀj ❣n 將s 則v 루x አy ┬z 專 ˉ 黃 🇹 Ź Ї ര ር 統 ʚ պ 組 頁 불 님 💖 ┐ 석 🎧 Ն ನ  ᄇ 길 ओ ວ 強 Ē 議 қ ۰ 転 ぞ 構 ừ 並 蝰 醫 따 ਸ 創 ̵ ᴘ 豨 達 점 ે ም ╌ 증 ન త ⸝ 글 색 ∪ 圖 🎄 ╥ 태 യ የ ក  ̨ ০ ெ ὐ 聞 ອ 専 認 ڈ" ধ# ට$ ⚭% 單' 錢+ ത- 📺0 Ğ1 ֲ2 ⇑3 🇵: ᷅; ⌘< ◀= សD 値E 🌈G 從I 권L ԁM ửN ぬO ತS ‿W ☠X 約\ 티] ռ^ ន_ ☎` 畑b 편d ওe 簡i 葉j ŵk Ăo 試p ேq ᄏr មs ㄧt 吋u Ѕ{ 〒} 청 ᷄ 평 🇱 Ľ 質 종 標 決 🎶 į ৃ ⦁ 胬 ণ 灬 ˆ ژ ᴅ 覧 💛 ̮ ˂ વ ມ 镲 雲 😱 ો ະ 済 걸 외 친 ક ስ ➞ 貘 傳 裡 환 ә ລ ີ ẓ 規 확 🤷 झ ು 導 🌎 ɾ ফ 視 ട 萬 օ ẹ ⇄ 巉 狷 薨 볼 🙄 բ ເ 細 ̑ ̯ ┌ 識 머 활 Ṭ 碇 ɐ ▫ 園 曩 ২ స 宮  께  플  ც ˃ Ё ῶ ₩ 僰 በ ◾ 藝 წ 술  향! వ# ◬$ 兒% 兩& ץ) ぽ* 녀- 😘. 費2 陽3 테4 👑5 ヴ6 উ: እ; ゼ< 摺= મ? ይ@ 새B ∅C ▴D 耵F ለI መJ ῖK 🌹L 👌P ŷQ ٩R ۳S ☘T ̿[ ል\ 讓] 백^ ြ_ 瀬a 茛c 약e ஙf ಕg ရh 総j 👋k ංl ṅm ⁎n 価o 係p 箨s ‰t პu ゙v 였y 특z 런 Մ ບ ສ ▻ „ ው ⓿ 給 調 顔 Љ 桜 課 變 ̛ ત മ 営 稃 級 豊 載 🇭 Ե ծ ১ 脩 裝 谝 Ħ 巻 終 菈 언 ➖ ヘ 嗎 夢 戡 猹 💚 Έ ヨ 環 確 ɹ ף ਿ ം 蠲 護 💔 聲 ؟ ಗ စ 訊  ջ ặ 矬 Ġ మ ິ 団 團 帰 朊 柾 💯 ̪ ವ ී ῆ 離 ▿ 椪 湔 許 순 ० 観 병 ༄ ☝  竜  節  聯  검  ↯ ➙ 優 초 ⬆ 劬 畫 蜩 觇 詞 😳 猢 瘊 買  류# Ъ$ ప% ୍) ோ* 萩, 觀- 🥇0 പ1 근3 📚4 ›5 ׳6 ۶7 険9 놀: ਜ; 溽< വ? 栱@ 處A 轉B 군C 애D 戢E 暐F 遠H J २K ༡L បM 倨N 紅R 항S உT ಠU ➯V 薜W ΖX ැY ☹Z 曷[ 難] 🏿^ ფ_ ‵` 꾸c 립d շe ▀f 単g 喰h 🇲i 😄j ୧k ზm ◔o 彘q 湲r ٠s గt 寶v 歳w y z ɵ{ ϊ| ົ} 悪~ 螈 邶 锜 鲣 싶 📝 ۞ 蛩 袢 覺 투 🇫 ੇ ┄ 親 醣 鉄 족 퍼 သ ♚ 肫 ۉ 镞 느 ̘ Ո ೆ ỹ ⇢ ∋ 嚩 柽 澶 牂 絵 罅 項 즈 효   帙 블 판 亞 潟 磙 笫 薙 韻 働 咭 菪 🥳 ਨ ದ ያ ‫ 凪 蓠 베 손 ٍ ທ ❝ 蠓 覚 髹 ̍ ె 廣 秫 艏 ǀ ۲ ਲ 漢 訳 ∎ 禊 總 움 󠁧 խ ܐ එ ና 備 廋 絶 션 얼  牁 纮 較 酃 ੭ સ ✮ 彡 毘 鈥 鲽 ̗  ༅  ỏ  ἰ  収  聖 讵 ብ ተ ὰ ὴ ⁍ ∽ ✲ 亻 極 準 鏊 響 ല ằ Ộ  ╦! ╬" ❚# ⻓$ 堺% 浞& 猊' 羅( 該) 🗣* ❈+ 岞, 彳- 橐. 钫0 镚1 😆2 အ3 ‛4 態5 暻6 栲7 瑠8 缗9 讀: 镨< 널= 😃> య? ማ@ ➋A 屺B 證C 뉴D 희E 😅F ՐG ொH ᐳI 歷K 講M 커O ঠP 杪R 満S 筆T ऑU ẩV ✟W 図X 葙Y 🦋Z ̝[ ↪\ ⚪] 劏^ 労_ 瞽a 骓b १c ≫e ❛f 週h 鲂i 막j ↘k 乩l 卲m 癃n 碲o 缵p 🎙q ંr ᄌs 奭t 躅v 鹪w 🤯x ‹y šz ͬ{ ⊂| ╩} ▓~ 乗 撃 蔹 螬 鹁 련 클  🔗 ǚ ነ ぱ 玧 蓊 铈 鳎 ќ ຫ 換 槠 淯 熱 钌 飛 飧 🎬 👊 Ū ᴍ ⲟ 丿 弶 復 橦 權 殍 瓘 圏 暍 條 溫 溲 痦 蔵 辏 鲞 麴 광 📱 😢 ٰ ੰ ା ಲ ゝ 嚅 歡 藠 雠 창 ˶ ➌ 幾 権 歴 産 紀 螽 銀 🏈 ͒ අ 屾 帶 畋 磔 禮 蕹 贶 閲 ྀ 療 镡 란 Կ ુ ត 嗐 姫 戰 淏 購 鼩 🏴 💗 Ά ״ જ බ ໍ 傕 槊 氖 頼 လ ᠧ ∘ ⌒ 劢 協 宀 嶋 掾 铼 չ પ ಭ ‸ ┴ 紘 討 陸 ╱ ❆ 儇  礐  鲻  밀  ः  ⏰ ┼ ✆ 哂 箦 郞 패 ԅ ట ▒ 測 爾 筇 罨 舣 镏 송  🔵! ѡ" ṯ# ↠$ 夥% 媞& 嶝' 熳( 镟) 올* ɦ+ ਕ, ର- ➪. 畤/ 鲡0 鸹1 Ս2 圮3 瓞4 蠊6 髈7 鲖8 택9 ༠: 勝; 撄< 焼= 猞> 薢? 赀@ 鲔A 鳇B 🇰C ɨD ਤE ളF ╝G 亶H 橋I 臺J 螫K 谖L 適M 🇩N ̂O ԛP 屃Q 抜R 揾S 樗T 锛U V ऐW ងX ẽY ⁃Z 儁[ 維\ 缃] 蝥^ 阝_ 馃` 축a 💬b 妗c 瑢d 罴e 赜f 鲺g 🤝h 🤦i ‡j ަk ◞l 両m 剞n 卺o 姘p 狀q 🎂r ̌s ͚t ḳu ╗v 嘁w 峇x 滗y 玘z 蘇{ 鬘| 鲐} 씨~ ғ ՞ ྫ ᠽ 濬 爿 衆 逡 陔 책 Œ Ȣ ۹ ᴄ 倮 媜 猁 臌 舾 蚵 辺 钪 ̸ 効 栳 蘭 诹 錄 림 🌍 ެ ਪ ျ 帯 曛 泃 溘 眀 篼 興 菉 蒟 辂 領 접  ͙ எ ွ ⚔ 媺 屙 弇 愔 橛 甴 砬 郤  💋 ຍ ỳ Ἀ 佉 択 毳 돌 ̠ િ ି ძ ᴜ ☁ ❮ 嗮 氇 淸 狻 薬 鸰 👩 ञ ৭ ⇆ 墫 瘥 黼 펌 혼 ֽ ီ ╔ 寫 潲 玕 硇 翃 荭 🎵 ீ ಪ ✄ 咥 棰 珰 異 翾 苘 鹀 ͜ ▔ 呖 尓 湴 蝓 骖 🧐 ో ො ຶ 偬 烀 舖 蠖 跸 맞 북 슬 Ի զ ਦ  ੁ  റ  ិ  ẻ  ⎮ ┃ ➲ 倥 児 咝 垲 況 洣 職 蝾 驗 鮀  😬 ద ᐈ ∃ 悫  癒! 眭" 粜# 葜$ 褊% 詩& 阋' ঙ( લ) ግ* ỗ+ 昳, 檫- 珽. 謂0 辻1 钋2 ▢3 仮4 坵5 渟6 滿7 莶8 蒻9 裢: 觌; 談< 願= 센? ʺ@ ਹA ేB ≖C ⋮D ☂E 咑F 毵G 澥H 赑I 锞J 鼯K 침L 🏀M 💻N 🥛O ͓P ਮQ ಸR ണS സT ឴U 亍V 堠W 幵X 擇Y 樹Z 洈[ 硏\ 祓] 耧^ 菝_ 蜮` 鄣a 鹡b 십c Գd ׂe ۅf 戸g 旼h 栝i 茈k 钆l 😡m 󠁿n Ţo ǝp ߋq ગr ㄨs 扌t 柝u 烔v 稱w 韓y ꦿz 높{ 🌬| 👨} 󠁢~ ċ ɣ ⇔ 搦 榖 碼 縻 衛 雫 預 💫 ɢ ̞ ಮ ព ╚ 墕 栃 祼 铫 麈 龠 🌿 💡 ᵎ ▌ ✗ ㄒ 岈 柢 瑧 祆 绂 謝 辚 鄚 곳 🌸 🥂 ݂ ឵ ὁ 剣 铪 铽 镱 餍 鼍 완 ৰ ୦ ஸ ಯ െ ὲ 吿 紹 蚍 財 賨 鞞 ɯ ؛ ூ Ḵ ∮ ▄ ▩ 亊 暌 涙 焮 蛞 髪 혜 🎁 ۔ ພ ྭ ች ₵ ⇠ ⚫ 劇 啐 暲 殳 積 臑 蟊 詳 誰 鹟 ՝ ܬ 勍 參 愦 扞 攽 瀹 瑨 缣 芰 荩 铗 鲚 누 취 팅 ৩ ৳ ແ ့ 夲 屦 崟 杌 滉 眇 硎 铒 锖 鞥 魚 📰 🔊 🗓 ͑ ପ 佺 吳 従 琹 稲  穂  裏  誌  钐  츠  🇿 ͊ ־ ✏ ➭ 乛 埒 揸 梶 祊 笥 笮 菹 蔸 蚺 順 鲊  麵! 갈" Ə# ٫$ ទ% ✐& 塅' 媵( 嵖) 彖* 暾+ 泐, 渼- 瓿. 畚/ 練0 蓍1 诔2 逖3 郿4 📖5 🤘6 ీ7 偓8 哿9 屍: 岵; 旄< 旒= 栄> 汭? 瘅@ 眞A 脔B 觏C 鹓D 될E 탄F ɽG ခH ṁI 仉J 哓K 啓L 戆M 殠N 氅O 續P 註Q 隠R 鲦S 허T U 🌧V ԸW ٌX ڏY ങZ ◂[ ➀\ 絜] 胨^ 邊_ 養` 鲼a 찾b 🏅c ̽d ͆e Ѓf ਬg ỉh 垤i 拡j 晡k 桲l 歃m 湓n 盤o 隨p ꒦q ̉r ުs ಂt ደu ᴡv ḩw ⛪x ➚y 偾z 埖{ 忸| 翛} 蜣~ 踔 颙 빈 🔒 🦄 Ư ̚ ͕ ល 姌 娭 帑 怩 攞 玹 莸 蒄 郷 酾 镢 끝  ǜ 堽 庥 戲 拏 敔 樓 檎 毑 璩 筲 辶 镘 隳 鞯 鳉 鹎 링 잡 ۴ ਊ ᥙ ┘ ✱ 亜 呪 忝 悒 拊 滙 貴 贇 😜 ຈ ∷ 啱 尋 朓 獬 肸 褡 褰 陬 雊 鱻 암  ͔ 咾 垾 孓 層 崞 昃 槑 笊 臨 酂 钍 須 顯  ˵ ਗ ય ራ ἄ 伛 囲 継 颃 鲹 💃 ɕ ͋ ✡ ゞ 䨻 巘 庫 浲 烝 簰 籴 荛 菿 谠 谰 閞 雑 飑  💀 ģ డ ༎ 伱 倓 圬 尅 慾 摽 沵 瀞 炤 獣 窓 缋 遄 酡 锕 黧 黾 철 🌱  🤪  ̬  ᭄  ḏ  ➁ 呬 夬 曱 柃 蒨 蹻 鹛 黻 채  ̼ Վ ূ ẵ ₪ ≯ ⌓ 偁  坜! 嫫" 岕# 椇$ 瀼% 燠& 硗' 罾( 輪) 遅* 镪+ 黥, 밤- 윤. ķ/ ̹0 Տ1 ׾2 ۪3 ላ4 ➸5 劉6 呙7 圌8 妺9 姉: 棻; 歺< 渋= 獎> 獨? 瑫@ 钶A 骟B 鲌C 黐D 🔍E 😈F ߈G బH ປI ⊱J ♛K 塄L 嬲M 屬N 桤O 渽P 輋Q ͖R ௿S చT ఢU ᴥV ⌄W ☒X 佧Y 妳Z 桖[ 牤\ 獴] 眄^ 禩_ 荬` 螅a 鼙b ͪc Ւd ஆe ♔f 仼g 勢h 姤i 徴j 簃k 粢l 納m 脷n 辦o 锝p 阒q 겨r 져s ५t ಟu ወv ᐛw ⊰x 坌y 堎z 屣{ 揺| 旴} 柊~ 桄 檔 溦 煕 犴 甡 皞 蘩 顸 규 램 찬  📈 ͘ ٴ ባ ⁨ ⬅ 喈 堉 壸 娡 搴 枱 楊 歘 猡 籙 罟 髭 📍 😇 ǧ Қ ١ ঞ 圧 孥 帱 榃 沬 溇 澌 營 膂 讠 谳 蹀 답 ͅ ͇ ച ຄ ὑ ☚ ㄟ 喁 嗵 墁 壊 憍 掲 氷 泖 盦 眾 礞 绐 罒 耜 舻 覽 闘 鲘 봐 승  🤓 ͎ ӧ ި 刋 廛 旆 汯 湯 瘳 綮 褫 躞 鲋 웃 즐  📣 Թ ڑ ښ ৯ ᅲ 冊 卻 奓 籣 蟳 賞 铕 阌 삼 🥩 🦁 ͭ ͮ ң ਵ ሰ ἡ ⤵ 伧 埫 廯 慊 曕 棼 毉 絡 纩 鹆 응 큰 Ą ͦ ↖ ⯈ 峣 廻 拶 熺  猱  疰  矻  虺  鎓 铥 엔 착 Ɩ ऊ ገ ᴠ 執 幫 杮 歐 獲 痄 盉 蚋 補 趱 鐵  阃! 驺" 骎# 鳚$ 존% & 🅸' 💎( Ť) ɬ* Օ+ દ, ታ- ድ. ḫ/ ➩0 ㄈ1 侂2 峫3 嵝4 戶5 摅6 榇7 玚8 绠9 羙: 莔; 谿< 钚= 麗> 녕? 👤@ ĊA ਚB କC ஜD ോE ែF ⎯G ▹H ◟I 採J 棨K 檢L 殺M 淩N 澪O 燫P 茺Q 郐R 객S 염T 🌊U 🕊V ẫW ∝X ≪Y ⰙZ 丏[ 傒\ 妼] 寔^ 廠_ 惔` 摭a 浛b 減c 漭d 璊e 瞓f 硁g 碨h 笸i 舢j 艉k 蕗l 虮m 蚰n 觯o 鐘p 鲃q r s 🤞t ǂu ʎv ⌊w 傷x 凢y 屲z 幛{ 悃| 枓} 監~ 脶 艋 蟜 複 酎 鲆 龀 벌  Ž ͣ Յ ▏ 佀 弎 挼 燹 窭 笾 縄 缳 褙 轵 農 鄜 酲 靑 ꃅ 셔 왕 케 콘 튀 🐶 😏 🙃 ͂ ͐ ۣ ळ ६ ី ἱ ⊆ 僦 塩 帻 彀 棪 浄 碹 絷 臜 衪 証 谮 蹟 钅 頣 곡 🏳 ٱ ♑ 嗌 徼 織 羣 贳 踅 飖 龑 굴 몸 충  ʙ ͛ ચ ည ំ ➦ ゾ 兕 匋 厶 哜 喦 幣 斝 濟 璆 訓 讬 诼 趿 颟 됩 락 령 🎮 ˑ ̺ ٢ ۵ ް ຕ ဲ ┅ ⚾ 仂 庁 捩 洑 涢 瓐 甍 畛 縠 縣 赍 鳥 鹣 꼭 꽃 숙  🎤 😋 † ͫ ܝ ঃ ਂ យ 忄 戓 枹  洳  紙  绁  缒  缡 荘 跞 郪 鳯 👁 ុ ់ ⊿ ✶ ➂ 勹 呎 堞 墼 挙 昪 樋 滝  灋! 熜" 癍# 蚨$ 赕% 轺& 階' 驵( 돼) * 🤤+ ɺ, ӂ- ඩ. ᐢ/ ῷ0 ◙1 ☪2 ✺3 ❞4 傧5 剛6 峃7 溷8 熥9 猋: 碈; 稜< 緑= 縢> 脇? 臵@ 赙A 遘B 闶C 隹D 靺E 鲉F 鳓G 뿐H 텔I ਟJ ୨K ຂL ຊM ክN 凖O 剀P 抯Q 晜R 漶S 澂T 澤U 缍V 罣W 聴X 舴Y 苢Z 輕[ 輝\ 遹] 阍^ 雱_ 饩` 롤a 쉬b 핑c 💍d 🔸e 😷f ˍg ှh Ẓi ἔj ◌k ✝l 凃m 塁n 嶶o 惢p 挢q 翫r 芴s 荳t 蝲u 詈v 邠w 铴x 險y 鬯z 鳀{ 억| 🍁} 🎨~ Ě ̻ ͤ Ѥ Է ۷ މ ନ హ ຮ ⚖ 吔 埘 岡 崾 扆 渉 猃 癯 虒 衎 觜 鄘 鍪 캐 🙋 ͍ Ԉ ৶ ટ ୐ ୒ ౐ ಳ ೜ ഻ ൱ ඪ ඼ ረ 嗥 彈 悛 扱 烺 蒎 谇 踬 錡 🅾 🖥 🧚 ໃ ყ ♍ ぴ 吥 巎 幞 戻 瑆 璂 粧 糨 缌 軽 違 霊 鳷 麤 麺 릴 😩 ʽ ༥ ༽ ከ ᠹ ⚜ 伾 堍 夠 寛 戽 抺 栴 殂 玡 瑱 畎 碡 缧 苤 蕰 諾 譞 譯 軟 靈 飯 鮟 鱇 범 왔 Ό Պ ॥ េ ⏩ ♐ ❊ 僕 塭 屮 沕 泚 玾 珧 羮 茇 萹 虿 袆 負 혈 💞 Բ ൊ ➠ 乸 埏 婼 昺 汏 沨 洫 爇 痩 眛  祧  築  糬  舎  鮮 鯈 鹨 ꯭ 🅻 🌴 🎃 😮 🙈 Ԁ ੍ ବ ខ ច Ọ 伲 咵 喎 壆  姵! 斎" 瑒# 禚$ 窎% 葎& 訂' 賽( 颋) 饤* 駆+ 骉, 黙- 齢. 량/ 탈0 1 2 😤3 󠁳4 Ĺ5 ̖6 ̜7 ̦8 Լ9 జ: ໄ; ជ< ≌= ⎜> ❑? ⻛@ 唿A 嚒B 嫑C 榊D 漼E 瑔F 磜G 絺H 繋I 绲J 芶K 襪L 踇M 鳑N 잠O 🤡P ̐Q ̟R ණS ශT ဆU ယV ᄑW ᴖX ḨY ┆Z ┯[ ╣\ ☄] ㄏ^ 刳_ 呶` 嬰a 尒b 戋c 殛d 汖e 煣f 牿g 祾h 缐i 菮j 蒇k 蝻l 轾m 駹n 鲒o 鲯p 鹾q 🥺r ֙s ෆt ⻔u 乆v 嘏w 圯x 埴y 渏z 燿{ 犼| 猄} 珹~ 瑊 璁 璸 畢 砵 禼 舥 藥 虓 鏡 顧 鳘 龘 몰 풍   ž Ę ̡ ͉ Ћ ྂ ံ ⊗ ⌿ 億 戥 斷 珖 礡 纟 腽 蒛 蠃 郚 镤 隱 靬 鞨  🎈 އ ऋ ૅ ළ ཱ ᄂ Ḳ ➮ ➷ 俍 倉 剅 劻 吙 嚱 搋 椠 汈 牀 琲 胂 芑 螣 訪 锸 霙 앞 죽 🆂 🐍 🛏 ૦ േ ᆞ វ អ 俵 坖 埈 夐 峘 怃 晧 涴 滆 濩 瑡 緒 聱 舉 舗 葸 貨 跫 颿 髑 鳡 齮 ꇁ 웨 짜 🕵 Ń ˗ ̰ ނ અ ሚ ḡ ヶ 亀 佈 兲 夤 媄 慶 拠 暎 煐 犭 禇 綝 菂 蔌 诎 轹 鄉 鉾 铹 锼 鲏 鵟 돈 벽 컨  😔 Ŧ ७ ঁ શ ୪ ህ ℘ ⑉ ♊ 乄 嗾 廸 梠 梼 椐 歯 滈 玗 玬 甏 盍 箝 罥 蕺! 袮" 賛# 邽$ 酤% 钸& 雜' ( 🏃) Ė* ̒+ ਼, େ- ⇣. ♒/ ➫0 勲1 呉2 咴3 囖4 堋5 壇6 姞7 愽8 楯9 涘: 渶; 湑< 潩= 澔> 珷? 璈@ 瞢A 窊B 純C 纁D 翙E 苨F 菍G 螓H 跣I 鉅J 錯K 锇L 陰M 雖N 顗O 鮸P 鲕Q 鲰R 鳆S 슈T 앱U 🍿V 🧨W ĎX ʈY ˀZ ˎ[ ̢\ ̭] Դ^ ₡_ ▂` ♓a 仏b 倶c 厔d 呯e 壓f 敎g 梽h 楬i 煖j 瑅k 瘵l 盩m 矞n 礵o 绨p 舁q 莂r 藴s 蚿t 遆u 釭v 隣w 鯭x 릭y 봉z 셰{ 🇯| 🇾} 🤍~ Ů Ơ ѓ ѵ ସ ኛ ☃ ✁ 亖 侔 塝 媸 懔 挒 揲 朥 汔 滯 滻 炟 犰 禵 穉 芠 莛 蒉 蔟 裣 鉢 锾 頻 飲 魟 ꞌ 귀 랍 죠 𓆝 🌻 😐 🥑 ͝ Ў ۸ ੈ ಬ ಷ ື ჩ ₰ ↧ ⏱ ♋ 刂 匫 嗍 坬 嫋 宄 暡 杩 棎 榮 牰 獡 璿 礤 簖 脁 菸 蔴 藦 裀 誠 邅 醢 럽 폴 Ǝ ʘ ܘ ༤ ḑ ⊇ ⌣ ➍ ぁ ぃ 刬 啟 姽 尜 払 昰 梃 洀 淨 爭 犋 甽 睆 窳 艚 蟢 裥 邲 釋 銭 镃 镄 閟 雙 麇 ꒪ 떠 ︠ ︡ 📢 ऍ ජ ᐅ 丌 剸 卬 啙 垹 墹 抃 抰 捔! 捜! 暃! 狳! 疒! 篥! 籒! 苵! 訚! 責 ! 躜 ! 锘 ! 騒 ! 髣 ! 鱓! 麹! 뒤! 빛! 🔹! 🥀! ˔! ஞ! ໂ! ᠊! ゥ! 丄! 丷! 乀! 侉! 侰! 侹! 兿! 冏 ! 刖!! 厣"! 啯#! 噚$! 嶖%! 廃&! 徧'! 怣(! 惗)! 棐*! 槙+! 汫,! 狯-! 狺.! 猔/! 甗0! 篢1! 粦2! 绋3! 舲4! 螋5! 讃6! 踜7! 踺8! 輙9! 輸:! 迓;! 遺! 颎?! 骕@! 궁A! 싱B! 찰C! D! E! 🅴F! 👻G! 📧H! 🔔I! ĝJ! ߑK! ါL! ↙M! ∏N! ≮O! ♉P! 仯Q! 冁R! 埜S! 妘T! 娎U! 搤V! 栻W! 槅X! 檚Y! 櫩Z! 沋[! 洨\! 涐]! 澉^! 獍_! 珝`! 禆a! 蒬b! 蔳c! 蠍d! 譲e! 豐f! 貊g! 閱h! 陧i! 鞮j! 頂k! 鱀l! 례m! n! o! 𓆟p! 🍷q! 👎r! ȃs! ̾t! փu! ֞v! ତw! ◼x! ㄣy! 倫z! 僽{! 刅|! 厳}! 囷~! 夂! 幨! 怫! 恓! 懷! 晄! 滒! 獢! 璲! 脳! 腙! 薦! 賣! 躐! 鋐! 閣! 靜! 鰕! 齼! 龆! 💓! ੱ! ຸ! ጠ! ❅! 匚! 卐! 啨! 夨! 姶! 彵! 怤! 楢! 湸! 漹! 烎! 熚! 瓓! 疋! 絲! 胙! 苼! 褀! 酴! 鏼! 镆! 鞡! 饦! 鬈! 鷨! ! 🥤! ̔! ӣ! ժ! ࣞ! ភ! ☸! ♌! ⚘! ⻄! 倞! 偠! 匜! 塡! 岄! 崁! 嶒! 惣! 懡! 拃! 柁! 榍! 榎! 炱! 玓! 玭! 竨! 競! 葞! 覇! 赬! 跐! 踈! 躄! 躔! 辁! 鈇! 鉮! 錬! 雮! 鬂! 鸊! 黪! ꒳! 밥! 킨! ! 🐾! Š! ƪ! Һ! ձ! ਆ! ጥ! 側! 刭! 厓! 叇! 垞! 埤! 孫! 嶤! 幷! 愬! 慆! 憶! 扥! 楙! 欅! 歲! 汚! 洢! 溍! 滳! 炣! 猸! 瓛! 畠! 盡" 硪" 礎" 筻" 臁" 臓" 莮" 萈" 萣" 蒐 " 蓇 " 蕋 " 薁 " 薫 " 觋" 跘" 辿" 醁" 鈢" 飾" 鷉" 鼭" 갑" 싸" 혁" 흥" " 🏠" 🤭" Ň" ࡒ" ఻" ⇡ " ♃!" ❃"" ⲛ#" 儀$" 劀%" 勅&" 勳'" 呡(" 嘂)" 妧*" 婡+" 嫰," 崥-" 嵕." 惛/" 挧0" 攵1" 曽2" 枵3" 棇4" 槻5" 橏6" 泈7" 熣8" 睃9" 碕:" 祙;" 禘<" 筅=" 篟>" 簠?" 籏@" 粏A" 絅B" 羑C" 脃D" 荾E" 蔰F" 裌G" 誐H" 踿I" 轄J" 逨K" 鉏L" 銷M" 鋬N" 锫O" 饊P" 鷄Q" 떻R" 째S" 🧸T" ЌU" ९V" ູW" ሳX" ἸY" ⃛Z" ♈[" 僅\" 卽]" 嚐^" 妸_" 姁`" 屄a" 惪b" 慝c" 栿d" 檵e" 殪f" 浡g" 浼h" 渓i" 渖j" 湣k" 炻l" 狛m" 玊n" 玍o" 珌p" 瑸q" 癿r" 盬s" 眘t" 磉u" 粬v" 粰w" 繑x" 苂y" 蒠z" 蚻{" 蟅|" 袴}" 覌~" 訫" 諸" 謪" 躧" 辪" 锿" 馇" 駖" 騃" 髄" 鮨" 鵕" 麶" 읽" 총" 🌞" 🌶" 🏁" ҳ" હ" ಜ" ☊" ♏" ⭑" ヌ" 冫" 卌" 噐" 妐" 岺" 巣" 徥" 悝" 懐" 撖" 撙" 昫" 暠" 枘" 梿" 潏" 燯" 瑮" 秡" 稖" 繴" 缏" 荅" 萿" 蓂" 裇" 裼" 赪" 踙" 鐖" 雃" 鯿" 鳽" 龁" 쇼" 팔" " 🤫" ̙" ֩" ३" ਖ" ဖ" 俬" 匦" 呅" 奨" 嬖" 戯" 撔" 擊" 擗" 暣" 朏" 棽" 楗" 榅" 歸" 汬" 溓" 滃" 燏" 珎" 瓌" 眣" 硔" 粎" 肷" 臛" 舳" 苶" 茀" 荍" 葖" 蕡" 螠" 訞" 赆" 陁" 鵼" 鸺" 쉽" 줘" 🇴" 🍾" 😞" 😲" ĉ" ȋ" ੋ" ༼" ∉" ⌥" ▍" 劃" 匃" 嘊" 噅" 坣" 壷" 妑" 尙" 尥" 尪" 悘# 敗# 敟# 暋# 暏# 朷# 杻# 樉# 檠# 淽 # 渀 # 潾 # 烾 # 狨 # 珓# 瓬# 甃# 皛# 眬# 硿# 祹# 緊# 纛# 蓮# 诖# 貥# 踰# 迋# 鄃# 鎌# 镈# 镙# 靵 # 骩!# 麿"# 鼷## 떤$# 먼%# 믿&# 🍒'# 👆(# 🤑)# ŗ*# ǰ+# ɚ,# ଲ-# ༣.# ដ/# 冣0# 噏1# 岣2# 悩3# 捛4# 搧5# 擵6# 攌7# 朶8# 栂9# 桱:# 椀;# 椣<# 楤=# 槺># 歓?# 涥@# 溁A# 爏B# 爦C# 狴D# 獳E# 璪F# 畂G# 畳H# 簩I# 絕J# 聡K# 腦L# 萳M# 蓐N# 蝀O# 蝳P# 钔Q# 闍R# 靐S# 鞵T# 顮U# 饔V# 駡W# 驩X# 鷣Y# 黹Z# 鼜[# ꧁\# 넘]# 롯^# 쟁_# 좀`# 징a# 킹b# 혀c# 🐻d# 💨e# 🥕f# ʍg# ֫h# ঘi# ෝj# ူk# ₽l# 乫m# 俢n# 兓o# 圓p# 圙q# 墣r# 婮s# 帲t# 抏u# 昷v# 朅w# 柈x# 汧y# 浰z# 漺{# 澋|# 瀻}# 灈~# 牬# 琻# 痎# 磾# 稂# 穏# 窆# 笖# 纻# 肹# 艆# 荓# 袛# 袷# 裰# 訮# 訴# 跡# 迬# 郘# 酔# 铦# 陓# 隩# 駪# 骃# 髢# 魝# 鮈# 鲪# 鼂# 꿈# 딱# 써# 엘# 👈# ŝ# ̎# ϝ# ᐝ# ូ# ᧐# ḷ# ❱# ⲓ# 冋# 厛# 呂# 唦# 圊# 垳# 埉# 嫠# 屛# 嵚# 嶳# 幁# 彣# 怹# 搣# 搨# 撘# 撛# 撝# 攝# 斢# 晱# 椑# 槜# 泦# 琀# 眎# 耖# 葰# 蔧# 蔲# 藨# 蟓# 裧# 郱# 針# 鐅# 锩# 頴# 額# 颩# 飐# 駄# 駼# 骣# 鱄# 鵅# 鹖# ꧂# 논# 첫# # 🏝# 🥚# ɗ# ˺# ৪# ଠ# ೄ# ᾶ# ♻# ぺ# 䓬# 仩# 俛# 倁# 冮# 卭# 哳# 噋# 圞# 寲# 峎# 峠# 庅# 悺# 掅# 昐# 晙# 暦# 桯# 樫# 櫉# 渕# 濢$ 焌$ 犲$ 狁$ 珃$ 甓$ 皦$ 篗$ 繦$ 艟 $ 菑 $ 蒾 $ 藍 $ 蚖 $ 蜞$ 螗$ 誤$ 謦$ 軡$ 輶$ 迍$ 逹$ 釈$ 锓$ 陴$ 鞫$ 韘$ 馰$ 驑$ 髧$ 鰰$ 鶒$ 鸻 $ 액!$ 엇"$ 틱#$ 홈$$ %$ 📲&$ ໌'$ ᄋ($ ᐃ)$ ⍤*$ ╠+$ ⚓,$ 兦-$ 卩.$ 呑/$ 嘡0$ 嘫1$ 嘾2$ 屚3$ 崚4$ 帀5$ 廙6$ 慮7$ 扡8$ 掟9$ 擔:$ 暜;$ 楩<$ 榼=$ 橽>$ 泩?$ 濨@$ 烮A$ 珴B$ 琞C$ 瑌D$ 璏E$ 璛F$ 眵G$ 矅H$ 碙I$ 絇J$ 繼K$ 绹L$ 耱M$ 藈N$ 蝤O$ 裒P$ 覵Q$ 趯R$ 趼S$ 郕T$ 郵U$ 鄌V$ 醂W$ 醽X$ 釟Y$ 鍉Z$ 閒[$ 靰\$ 韱]$ 魆^$ 鰤_$ 鳋`$ a$ b$ 🅵c$ 🇻d$ 👶e$ 🤖f$ Ɖg$ ɲh$ ̕i$ ఒj$ ቀk$ ẖl$ ┉m$ ♎n$ 亯o$ 亹p$ 伷q$ 傑r$ 墳s$ 媽t$ 嫃u$ 峝v$ 崋w$ 崦x$ 庋y$ 庪z$ 廑{$ 廴|$ 悰}$ 掱~$ 搌$ 斉$ 暁$ 梫$ 楜$ 檃$ 氕$ 溡$ 漸$ 瀴$ 熈$ 瑉$ 甁$ 禔$ 秲$ 稤$ 糇$ 翯$ 莈$ 覄$ 覕$ 訍$ 販$ 賀$ 輁$ 迮$ 鄁$ 锊$ 镅$ 顕$ 鮇$ 득$ 듯$ 칠$ 헤$ 🎅$ 😥$ 🤙$ ȱ$ ৈ$ ખ$ ദ$ ဘ$ ᅳ$ ើ$ ☔$ ☮$ ⛔$ ⛰$ 亸$ 侁$ 冚$ 劄$ 勷$ 匨$ 卝$ 坫$ 宬$ 岍$ 岠$ 廒$ 廰$ 捽$ 掎$ 擉$ 栚$ 橚$ 煺$ 牸$ 珟$ 珨$ 璡$ 瓀$ 疎$ 禟$ 穽$ 窅$ 筞$ 縁$ 肜$ 荄$ 菵$ 蘘$ 衤$ 裛$ 觍$ 頖$ 骙$ 鴠$ 鶸$ 鸂$ 골$ 템$ 🍎$ 🍪$ 🎸$ 🐐$ 🔎$ 🗝$ ȉ$ ֖$ ४$ ૪$ ௰$ ూ$ ಹ$ ᑊ$ ♞$ ⚙$ 亼$ 仌$ 儞$ 冖$ 冧$ 垇$ 垍$ 塂$ 奤$ 奧$ 婇$ 嫜$ 寀$ 寧$ 嵙$ 巓$ 廾$ 彟$ 恲$ 慣$ 抐% 搿% 撏% 柅% 泂% 溬% 灪% 玞% 璚% 璺 % 畟 % 瘼 % 砑 % 秖 % 簼% 絬% 缊% 羼% 翵% 舘% 芄% 苅% 苠% 苲% 蕝% 蟪% 襕% 豞% 贂% 铞% 雰% 顏% 鬄 % 鮠!% 鹢"% 📽#% Ѐ$% ѐ%% җ&% ܢ'% ߜ(% ଇ)% ମ*% 䴙+% 傸,% 儫-% 冂.% 劜/% 啝0% 嗱1% 堅2% 奀3% 奡4% 寜5% 屧6% 峧7% 帳8% 惝9% 栆:% 殭;% 洚<% 淛=% 淲>% 潣?% 災@% 焆A% 珦B% 琭C% 瑺D% 礅E% 礩F% 竝G% 縮H% 胅I% 舄J% 舊K% 蚡L% 輗M% 輯N% 顒O% 馫P% 騄Q% 鲗R% 랄S% 웹T% U% V% 📊W% 😕X% ŰY% ˥Z% Ӫ[% ޑ\% ާ]% ੫^% ഏ_% ഠ`% ႔a% ሪb% ὅc% ῳd% ⦿e% ⬛f% 䜣g% 亅h% 伈i% 偦j% 呹k% 噉l% 墖m% 夿n% 姩o% 尛p% 帒q% 忭r% 慥s% 昚t% 朿u% 柷v% 檏w% 沴x% 洰y% 淜z% 湋{% 漒|% 爩}% 狃~% 獻% 琂% 琽% 瓅% 瘈% 瞤% 矧% 筊% 箷% 脰% 蔄% 蜫% 蠛% 誂% 賢% 赇% 跠% 蹐% 鈡% 锬% 饣% 鲴% 鸜% 둘% 맛% 졌% 척% 🎟% 💩% 🚫% 🤬% Ȟ% Ѱ% Ӕ% ۿ% ܻ% ࣂ% ૚% అ% ᄀ% ዝ% គ% ἶ% ┛% ❘% ➳% 㗎% 倴% 唫% 嚭% 塊% 婬% 嫇% 嵗% 庰% 廨% 忞% 懅% 戱% 挊% 擓% 晳% 殦% 洴% 涳% 漋% 濊% 璒% 瓕% 瘛% 癵% 砹% 禖% 笱% 筯% 罝% 耏% 耥% 耪% 苙% 螧% 覀% 踦% 軏% 輔% 醵% 銎% 錞% 閦% 鵙% 鼽% 齒% 쿠% 휴% % 🦆% ʂ% ˽% Ϧ% ӫ% Ԥ% ֥% ؊% ٮ% ړ% ܪ% ݃% ࣛ% ৫% ෳ% ᅵ% ṟ% ⊃% ⏤% ┈% 䘵% 䴘% 仳% 佽% 偰% 傊% 吢% 呿% 咇% 噀% 圍% 垯% 堷% 壵% 寯& 尢& 嶓& 憀& 憷& 損& 搥& 摾& 撡& 擐 & 旡 & 昒 & 昜 & 曺 & 杺& 楪& 樛& 橸& 欻& 泶& 洐& 潔& 炪& 煙& 煠& 爀& 玔& 琄& 瑑& 瑖& 璍& 璬& 瓈 & 畵!& 睱"& 竾#& 綠$& 绱%& 蔀&& 諠'& 蹯(& 釿)& 鋕*& 鏙+& 閉,& 雝-& 饀.& 騑/& 髶0& 鴜1& 鵺2& 뜻3& 빨4& 솔5& 욕6& 컬7& 컴8& 🕺9& 🚗:& ƽ;& ȶ<& ͨ=& ͽ>& Ӷ?& Ӿ@& ԔA& ֺB& ۊC& ࠯D& ࡢE& ࢑F& ࣏G& ੟H& ௛I& ఀJ& ಃK& ೐L& ⇅M& ┗N& ✳O& ⲉP& ぇQ& 佴R& 僺S& 嚴T& 堔U& 媖V& 幹W& 愵X& 攲Y& 昄Z& 柍[& 栤\& 楕]& 櫈^& 浭_& 炡`& 烏a& 牜b& 犏c& 玐d& 瑳e& 瓋f& 畯g& 痃h& 瞺i& 礓j& 祶k& 穁l& 穵m& 筢n& 紋o& 紮p& 緮q& 耔r& 芟s& 莙t& 蓢u& 虛v& 蝜w& 蟛x& 觸y& 訝z& 譛{& 赒|& 辀}& 顈~& 餘& 鱤& 鴂& ꉂ& 끈& 넌& 📹& Ɉ& Ҏ& ԝ& ֢& ݠ& ݮ& ߖ& ߤ& ࡈ& ࣺ& ॴ& ઁ& ણ& ଀& ୵& ஀& ௧& ఽ& ྕ& ሁ& ፈ& ┤& ✞& ➃& ⸜& ㄥ& 刌& 剳& 厵& 厷& 叄& 啇& 嗄& 婠& 嫼& 忳& 愀& 搛& 摛& 暅& 暳& 杍& 梱& 棫& 檞& 氍& 汵& 泇& 潽& 灦& 焛& 玌& 玜& 玿& 琫& 瑵& 瓃& 眜& 磢& 礸& 祕& 篔& 粛& 繻& 羖& 聼& 芞& 莀& 薕& 蘋& 蘳& 蝘& 褿& 鎉& 鏸& 鑑& 铏& 餣& 髆& 鮗& 鲬& 鵩& 鸀& 좌& 탕& ȭ& ʛ& ʧ& ˕& ϼ& Ӌ& Ә& Ӭ& ӭ& ԗ& ԩ& ֬& ׆& ؙ& ڒ& ڦ& ڲ& ۏ& ۘ& ݝ& ݥ& ݪ& ވ& ޱ& ߯& ࠝ& ࠨ& ࡫& ࡮& ࡴ& ࢀ& ࢒& ࢕& ࢠ& ࢮ& ࣀ& ࣼ& ࣿ' ৒' ৬' ੻' ઒' ડ' ૆' ૒' ૸' ଩ ' ୾ ' ஂ ' ஄ ' ஘ ' ௃' ௅' ௟' ఋ' ఍' ౉' ౓' ಄' ೞ' ഢ' ෕' ᄆ' ካ' ណ' ὺ' ‽' ⑅' 乁' 乇 ' 佂!' 冴"' 叆#' 呺$' 坲%' 墝&' 娸'' 婥(' 屪)' 忮*' 恏+' 愙,' 愰-' 憺.' 掭/' 旿0' 暼1' 栁2' 汑3' 沷4' 洊5' 烚6' 焞7' 珒8' 瑝9' 璮:' 痏;' 穲<' 簁=' 紥>' 繸?' 罃@' 肏A' 茳B' 荖C' 蒧D' 虙E' 裺F' 襻G' 誖H' 貼I' 贎J' 跂K' 踀L' 辵M' 醭N' 餅O' 駉P' 騆Q' 骒R' 룡S' 슨T' 옥U' 율V' 잊W' 잔X' Y' 🎯Z' 🐦[' Ķ\' ɸ]' ˁ^' ͯ_' ዳ`' ᐊa' ᗜb' ᮨc' ἴd' ⇪e' ⎪f' ❭g' ⺠h' 俜i' 俤j' 兎k' 冱l' 吚m' 哯n' 噍o' 坮p' 垈q' 夆r' 姱s' 姷t' 嬑u' 嬾v' 孭w' 寑x' 巂y' 楛z' 沇{' 煛|' 珆}' 琈~' 琜' 瑎' 瑿' 璅' 璫' 璵' 瓄' 疁' 瞀' 砢' 碦' 禑' 簏' 簴' 綟' 緜' 緼' 聓' 艶' 艼' 芼' 觭' 譌' 貞' 轪' 郛' 鄗' 鉨' 銟' 銧' 鋈' 鏻' 铓' 餸' 馘' 鯙' 鳣' 鹠' 麕' 끼' 압' 줍' 쪽' 탁' 튼' 획' ' 🍀' 💦' ɖ' ʋ' ˧' ਅ' ධ' ፍ' ≃' ◪' 倅' 倢' 倷' 僣' 咠' 圫' 奣' 姀' 媔' 嫮' 宔' 嵎' 弨' 恠' 慬' 憹' 戣' 揎' 擫' 擮' 杤' 柙' 梜' 棡' 槱' 毦' 滪' 瀫' 燣' 爝' 玅' 珋' 瑹' 璌' 瓉' 瘣' 睜' 砉' 砩' 窬' 籭' 纞' 罽' 脬' 舯' 芤' 蔊' 蛑' 蟧' 蟲' 衖' 袾' 覩' 誮' 谼' 趫' 跩' 踭' 躭' 遯' 醾' 鈝' 鈟' 镴' 閄' 陥' 顟' 騨' 鼚' 넷' 밝' ' 🌥( 💁( ĺ( Ƅ( ͠( Ө( ճ( ֣( ৮( બ ( ᄅ ( ᴋ ( ✋ ( ❋ ( 乨( 偸( 勄( 勧( 呫( 坋( 垔( 夵( 婰( 媂( 媏( 岧( 崐( 崬( 嶟( 廳( 懃( 敧( 旛 ( 昤!( 晅"( 晛#( 晥$( 栛%( 椛&( 槪'( 櫊(( 氂)( 氺*( 渃+( 渇,( 焫-( 珛.( 琱/( 瑏0( 瑴1( 甐2( 癋3( 盌4( 眚5( 睗6( 祔7( 篯8( 粙9( 糵:( 綑;( 缷<( 翈=( 胕>( 膵?( 臞@( 蘡A( 蜾B( 蟌C( 觱D( 詝E( 貤F( 跀G( 躱H( 躾I( 鈴J( 饐K( 鵁L( 麣M( 黀N( 黡O( 黨P( 률Q( 멀R( 벨S( 붙T( 옷U( 춤V( 텐W( 팀X( 🌳Y( 💉Z( 💐[( 💵\( ͗]( ܀^( శ_( ჯ`( ᅀa( ᠌b( ῇc( ▭d( 偟e( 偪f( 僈g( 勞h( 厰i( 唪j( 噺k( 圕l( 夋m( 娒n( 寘o( 尨p( 巇q( 帨r( 帹s( 廕t( 憸u( 戀v( 扽w( 挕x( 挻y( 搠z( 斶{( 旉|( 旐}( 晈~( 朙( 栘( 棢( 槹( 橴( 湨( 澐( 瀜( 焔( 煇( 玵( 玶( 琓( 琩( 瑐( 璭( 璹( 瓂( 瘐( 盝( 眏( 矴( 碒( 磏( 穊( 笣( 筭( 籤( 紬( 纙( 羕( 腨( 艨( 芲( 莯( 蒃( 蓜( 蕸( 蜎( 蜺( 蝂( 袊( 誃( 跍( 辒( 鄭( 鞩( 頕( 騊( 驎( 骦( 鶵( 鹥( 鹯( 齾( 났( 녹( 듣( 렉( 맘( 몬( 🅽( 🚩( ɤ( ѣ( Ү( Ք( ॅ( થ( ଼( ష( ༩( ዲ( ↴( ⻅( 仱( 侓( 倰( 偉( 僡( 凧( 劓( 呣( 呮( 噛( 囃( 囘( 圐( 埰( 堼( 壞( 奊( 婸( 岯( 弪( 忔( 悢( 挍( 捊( 掊( 揜( 晸( 暕( 朘( 杄( 枒( 柨( 桵( 椝( 橒( 橜( 欃( 欷( 泜( 湝( 潙( 澽( 灒( 牋( 牗( 牷( 珕( 琑( 琣( 瑲( 瑻( 瑽( 璯( 璶( 璻) 瓆) 瓙) 瘝) 瞾) 礌) 禋) 篸) 耩) 脿 ) 荈 ) 荹 ) 菥 ) 菼 ) 葢) 蒊) 蝨) 螀) 襜) 診) 譆) 譾) 讱) 赿) 逥) 鄮) 醸) 鐑) 霂) 靊) 鞶) 鞿) 韸 ) 頬!) 颕") 髴#) 鬖$) 鬬%) 魋&) 鯻') 鳳() 鳿)) 鵰*) 鶏+) 鷟,) 麪-) 黈.) 齻/) 깨0) 벤1) 숨2) 젊3) 캠4) 5) 🎓6) 😒7) ǃ8) Ύ9) ұ:) Խ;) ܡ<) ܫ=) ङ>) ੂ?) ආ@) ⛳A) ㄚB) 丅C) 乮D) 仧E) 俀F) 倛G) 冨H) 凎I) 剤J) 叺K) 埲L) 夀M) 妠N) 姸O) 娞P) 娪Q) 婫R) 媁S) 媝T) 嬓U) 審V) 尠W) 崹X) 庀Y) 悆Z) 悊[) 慅\) 捃]) 揀^) 攮_) 攰`) 敭a) 斄b) 斲c) 昘d) 晼e) 暥f) 梪g) 檑h) 檰i) 欵j) 歜k) 歰l) 漤m) 燀n) 牾o) 玝p) 玼q) 珿r) 瑬s) 璱t) 瓑u) 皊v) 皝w) 盻x) 砯y) 磵z) 祃{) 祩|) 秥}) 稙~) 稥) 窰) 筘) 筨) 縒) 繙) 羝) 羴) 肈) 脞) 舺) 莕) 菻) 蔘) 虡) 蝟) 蠘) 衝) 訑) 譁) 貓) 賴) 迖) 逬) 遃) 銙) 鍝) 鏚) 锍) 锎) 阾) 霪) 霺) 鞻) 駃) 駐) 骍) 鮋) 鯵) 鳒) 鳤) 鶄) 鶗) 鶲) 뷰) 홀) 🌙) 🍃) 🏐) 💌) Ĕ) ෘ) ࿐) ኢ) ↝) ≅) ◽) ⚆) ❧) ⲁ) ⲣ) 亽) 僊) 儚) 兡) 冭) 剉) 剰) 叚) 嗢) 嗼) 垿) 壐) 壽) 奅) 奟) 妟) 妡) 姠) 娿) 媴) 嬂) 尐) 扂) 揵) 摿) 朌) 杬) 枡) 棆) 椱) 槚) 歔) 歗) 毹) 洤) 湵) 澣) 濄) 濔) 灉) 烰) 焄) 狚) 玤) 琁) 琸) 瑂) 瑇) 璓) 璖) 瓇) 痾) 稅) 稊) 筴) 糈) 緈) 縦) 繝) 纴) 耇) 肭) 蓋) 薸) 蚼) 蜑) 蠼) 裲) 讚* 谻* 跽* 蹱* 鄫* 酦* 醡* 鋓* 鍂* 鏐 * 鑗 * 锪 * 阼 * 鞕 * 頞* 飔* 駂* 髽* 鰶* 鱥* 鲝* 鴳* 鵈* 鶰* 층* 팬* 🌏* 🐝* 🤳* 🥶* 🦃* ʖ* ҷ * Ӳ!* આ"* ထ#* ὀ$* ⃙%* ↻&* ⏳'* ⚈(* 亱)* 俆** 俖+* 偍,* 呴-* 喌.* 喓/* 嚄0* 囇1* 坽2* 塟3* 奃4* 妕5* 姳6* 媃7* 嫏8* 嬫9* 嶌:* 帊;* 廍<* 廐=* 弜>* 怱?* 愼@* 憙A* 扺B* 揕C* 晍D* 枻E* 柺F* 栺G* 梣H* 椻I* 楿J* 槔K* 欬L* 殢M* 澯N* 濈O* 烸P* 牳Q* 狌R* 珼S* 琒T* 琧U* 琷V* 璙W* 皜X* 皭Y* 睍Z* 砳[* 硋\* 硙]* 祲^* 窫_* 絥`* 絪a* 絽b* 缞c* 罫d* 羂e* 羵f* 胓g* 臯h* 舨i* 茘j* 茻k* 莝l* 莩m* 蒪n* 薆o* 藟p* 袓q* 褧r* 貝s* 軱t* 酕u* 醄v* 醤w* 鐻x* 镩y* 霝z* 鞇{* 鞬|* 饾}* 騕~* 鬹* 鯄* 鱵* 鴥* 鴽* 麏* 龖* 춘* 칙* * * 🌺* 🔭* 🦅* ̏* ͥ* ΐ* ݜ* ଉ* ଘ* ୁ* ർ* ࿆* ♕* 伜* 伩* 侎* 侒* 倗* 凓* 匽* 厺* 咅* 啅* 啛* 嚢* 嚸* 囏* 埦* 墈* 墐* 墺* 壠* 妱* 婹* 嫧* 嬃* 岓* 嵿* 嶆* 庯* 徏* 徯* 憇* 抂* 拺* 挿* 掹* 揳* 敻* 斸* 旵* 杕* 枼* 枿* 桙* 梘* 棯* 橑* 檭* 櫶* 殑* 沖* 泘* 洿* 渧* 湩* 滅* 炇* 燋* 爔* 狝* 獿* 玒* 瑓* 璄* 璤* 畩* 眰* 睝* 祫* 竢* 筥* 箢* 籓* 籘* 籞* 粋* 縪* 聻* 脢* 舦* 蓛* 蓷* 虅* 蠋* 蠙* 蠳* 蠵* 衒* 裯* 襃* 襌* 襩* 覠* 觳* 諽* 賍* 赽* 轚* 迀* 迡* 逺* 鄩* 鄪* 酺* 鋼* 錀* 鍱* 鍼+ 鍽+ 鎝+ 鎭+ 鎻+ 闉+ 陼+ 陾+ 雿+ 霩 + 韮 + 飶 + 騟 + 鬿 + 鮶+ 鰆+ 鱲+ 麐+ 麑+ 齂+ 협+ + 🍕+ 🕯+ 󠁥+ 󠁮+ ƿ+ Շ+ ૭+ ஷ+ ⌛+ ✑+ 〆 + 仚!+ 侜"+ 侱#+ 儍$+ 儙%+ 冾&+ 劂'+ 勑(+ 匉)+ 叀*+ 叾++ 吣,+ 咡-+ 唅.+ 唍/+ 嘙0+ 圀1+ 垨2+ 壄3+ 妭4+ 姯5+ 婓6+ 媱7+ 嫶8+ 嬟9+ 嬦:+ 嬮;+ 孮<+ 宍=+ 寎>+ 崝?+ 崿@+ 嵫A+ 巌B+ 幰C+ 忴D+ 悗E+ 惓F+ 懳G+ 拕H+ 摏I+ 擁J+ 攎K+ 攴L+ 旑M+ 昞N+ 曡O+ 杙P+ 杧Q+ 枆R+ 柌S+ 棅T+ 榬U+ 槩V+ 樳W+ 橪X+ 橲Y+ 檽Z+ 殻[+ 氶\+ 淉]+ 湶^+ 漙_+ 潄`+ 瀙a+ 灺b+ 烆c+ 煑d+ 煪e+ 煿f+ 熤g+ 燶h+ 燻i+ 獙j+ 玱k+ 珇l+ 珜m+ 珶n+ 珸o+ 琖p+ 瓁q+ 甞r+ 疦s+ 瘖t+ 皯u+ 碽v+ 礿w+ 秏x+ 秬y+ 稄z+ 穋{+ 窐|+ 竂}+ 竻~+ 笵+ 箵+ 簦+ 糒+ 繧+ 罇+ 耢+ 聤+ 膄+ 膴+ 艅+ 莚+ 菧+ 萊+ 蔿+ 蚷+ 蟮+ 褆+ 襙+ 觽+ 詾+ 誗+ 諲+ 趉+ 踨+ 蹢+ 遡+ 鄀+ 鐦+ 锠+ 隤+ 霦+ 鞔+ 顡+ 飈+ 駓+ 骹+ 鬐+ 魭+ 鴃+ 鴓+ 鴞+ 鵾+ 鹒+ 鹱+ 黮+ 齴+ 갖+ 틀+ 픽+ 핵+ 🏡+ 🕔+ ͌+ ҽ+ ٣+ ଓ+ ஊ+ ಶ+ ೀ+ ቅ+ ₫+ 乭+ 亂+ 侴+ 侽+ 傱+ 劆+ 卨+ 吰+ 呰+ 咷+ 啍+ 啩+ 喛+ 嘼+ 垉+ 塕+ 墧+ 壱+ 奷+ 姈+ 娍+ 娳+ 娷+ 婞+ 婻+ 媀+ 嫐+ 嫽+ 孅+ 孋+ 宂+ 寬+ 岆+ 崄+ 幇+ 幪+ 廲+ 弸+ 彨+ 怊+ 恦+ 慔+ 憟+ 挦+ 挷+ 揨+ 揻+ 撹+ 擿+ 敍+ 斵+ 昬+ 晀+ 枠+ 桟+ 棑+ 橂+ 檙+ 氥+ 漃+ 漛+ 潤+ 澫+ 炞+ 煱+ 燈+ 猨+ 獦, 玪, 珄, 珱, 琋, 琌, 琕, 瑈, 瑘, 瑼 , 眑 , 眿 , 睯 , 矼 , 硜, 硠, 禡, 禨, 秵, 穈, 穐, 筽, 篚, 篭, 籧, 緀, 緖, 翬, 耂, 耰, 耲, 聭, 膭 , 膰!, 臉", 舠#, 艁$, 苐%, 苖&, 苸', 莾(, 蕶), 藵*, 螷+, 裈,, 覥-, 觘., 觺/, 赮0, 趞1, 跢2, 蹝3, 辢4, 逴5, 遈6, 邘7, 鄈8, 鈘9, 鉱:, 銤;, 鋹<, 鍡=, 鐮>, 鑛?, 闁@, 阛A, 霨B, 靌C, 鞊D, 頠E, 頵F, 顑G, 顲H, 颥I, 髮J, 鬷K, 鱯L, 鲙M, 鶬N, 鶺O, 鷃P, 鼢Q, 닌R, 슴S, 폭T, U, ĨV, ֱW, ೇX, ༧Y, ᄒZ, ሉ[, ỷ\, ⋄], ◝^, ✴_, ✽`, ⻆a, ㄢb, 丱c, 亐d, 亰e, 伒f, 倽g, 兛h, 冞i, 刞j, 匄k, 匸l, 厐m, 喿n, 嗙o, 圷p, 坶q, 堈r, 塽s, 夣t, 妋u, 娀v, 婈w, 婦x, 媕y, 媤z, 媬{, 嫕|, 屒}, 峂~, 峈, 庝, 彐, 彞, 忀, 怇, 恷, 愳, 愺, 慓, 慹, 憅, 戹, 拤, 拪, 捙, 揫, 摥, 斨, 昲, 晊, 曚, 朂, 朳, 梴, 棓, 楀, 檛, 殟, 氆, 汸, 涶, 涽, 淓, 淟, 溒, 潓, 澨, 濎, 濳, 瀳, 炚, 烱, 烿, 熯, 牻, 犂, 狫, 猦, 珔, 珚, 琗, 瑥, 瑦, 璕, 璗, 璝, 璼, 疌, 疢, 瘌, 瞡, 瞫, 砪, 硟, 碏, 稧, 穜, 笭, 筶, 箑, 籝, 籱, 粓, 縇, 繄, 纚, 羓, 羶, 耈, 腄, 臖, 艴, 芏, 苪, 茝, 葁, 葇, 蒅, 蓧, 薿, 虨, 虩, 蜨, 袚, 襵, 譵, 讌, 蹓, 轗, 迗, 鄳, 酄, 醑, 釡, 釱, 銈, 鐏, 雔, 餏, 駧, 騳, 骻, 髿, 鮄, 鳁, 鹔, 鼡, 齉, 닥, 샵, 측, 칭, 훈, 🏟, 💸, 📆, 📩, ҂, ॰- ඔ- ᓂ- ᛫- ធ- ោ- ╟- ▎- 乪- 亇 - 佲 - 劦 - 匑 - 卄 - 卾- 吀- 吇- 呌- 哖- 哬- 啗- 嗘- 嚆- 嚔- 圑- 埊- 埑- 堨- 塙- 墎- 墠- 墴- 夳 - 奯!- 奰"- 妬#- 媙$- 媢%- 嬁&- 嬍'- 孆(- 宨)- 寕*- 寖+- 峀,- 峸-- 嵅.- 嶍/- 帡0- 彇1- 徢2- 忓3- 忹4- 悧5- 惐6- 愅7- 慂8- 慃9- 慏:- 慞;- 懩<- 扙=- 抲>- 揦?- 揼@- 摡A- 擈B- 旟C- 旣D- 昡E- 晢F- 晣G- 曟H- 曨I- 杘J- 枌K- 柂L- 梈M- 棙N- 椚O- 楅P- 樖Q- 橊R- 檾S- 櫂T- 欟U- 歊V- 毴W- 沺X- 泹Y- 洉Z- 洭[- 涀\- 淂]- 淰^- 渫_- 滱`- 漟a- 潵b- 澛c- 澞d- 濙e- 濵f- 烓g- 烗h- 熂i- 珗j- 珘k- 珬l- 琡m- 璥n- 瓍o- 疭p- 盀q- 睌r- 瞉s- 瞐t- 砈u- 硍v- 祴w- 秊x- 穾y- 笝z- 筦{- 箖|- 箯}- 篨~- 粻- 絴- 緾- 罤- 羄- 羐- 芓- 莃- 萫- 葮- 葻- 蓲- 蕥- 薍- 薖- 蘀- 蚥- 蝩- 蟡- 蟴- 衜- 裎- 褍- 褝- 詢- 諟- 謡- 譩- 豖- 貎- 賌- 賎- 賟- 賳- 賷- 跓- 跭- 輢- 逪- 鄝- 酟- 醲- 釢- 鈚- 鈬- 鈭- 鉥- 錱- 鍄- 鎽- 鏰- 鑐- 阘- 阰- 陦- 隒- 雞- 霅- 靀- 靿- 鞗- 飣- 馎- 駰- 驣- 鯷- 鲿- 鳂- 鴒- 鸑- 齇- 괜- 덕- 됐- 렌- 섬- - - ︉- 🍋- 🎼- 💣- 🖕- 🤕- 🥈- Į- ધ- ధ- ై- ཝ- ጣ- Ḡ- ἑ- ⲩ- 丆- 乑- 伀- 佊- 侕- 俙- 凐- 刕- 刣- 刯- 刼- 匞- 厡- 吜- 唭- 嘠- 噈- 噕- 囬- 埥- 塈- 墚- 墪- 夅- 姅- 娐- 婑- 嫅- 孞- 宻- 嵸- 嵾- 巗- 帋- 幤- 廌. 廦. 徑. 懥. 抩. 捁. 掛. 揈. 摂. 摝 . 敫 . 斮 . 暤 . 朑 . 朢. 朹. 杝. 柲. 梾. 椾. 樥. 橅. 橿. 檇. 櫘. 欔. 歠. 沘. 洏. 涫. 淭. 湻. 溋 . 溵!. 溻". 滮#. 潝$. 潡%. 瀡&. 灀'. 灜(. 焠). 煂*. 熍+. 熷,. 燅-. 爕.. 犨/. 珡0. 畇1. 皕2. 眊3. 眳4. 睎5. 瞵6. 碤7. 磈8. 磞9. 秜:. 稾;. 窴<. 竤=. 笁>. 筪?. 粊@. 糎A. 糑B. 紸C. 絒D. 絚E. 絧F. 縿G. 繜H. 繠I. 罏J. 罙K. 罳L. 耮M. 耴N. 胠O. 脥P. 脫Q. 膚R. 芵S. 苳T. 萑U. 萯V. 萾W. 蒈X. 薲Y. 藅Z. 蘌[. 虝\. 蝡]. 蟽^. 袗_. 袿`. 覮a. 詖b. 詧c. 諅d. 謢e. 譒f. 豭g. 趚h. 趬i. 跰j. 踘k. 踠l. 蹘m. 轌n. 迣o. 遶p. 郍q. 銛r. 鋗s. 鋧t. 錅u. 鍎v. 鍵w. 鍿x. 鎿y. 鏮z. 鑘{. 钃|. 镋}. 镎~. 镥. 隢. 隺. 鞈. 鞚. 韠. 韤. 韽. 顚. 飍. 餖. 驉. 驫. 骽. 髚. 鬊. 鬪. 鬫. 魌. 魽. 鮃. 鯸. 鰼. 鱎. 鱬. 鴷. 鸅. 麯. 黅. 齊. 齫. 齵. 龗. ꇴ. 딸. 뜨. 롭. 뮤. 찮. 탑. 폰. 픈. 🅼. 🎾. 🔶. ƶ. ǁ. Ɂ. ֿ. ॐ. ৌ. ୀ. ஒ. ಅ. ൻ. ඇ. ဒ. ጋ. ᑋ. ័. ⇊. ⊹. ⋊. ➎. ⻋. ⻝. 倻. 偐. 偫. 傁. 傿. 僜. 兂. 凵. 剏. 剚. 劒. 劗. 劸. 厪. 呝. 哤. 哵. 唞. 喼. 嚜. 囸. 圂. 圴. 垙. 埧. 埽. 堮. 塀. 墌. 墦. 夊. 夡. 奌. 妀. 妰. 姇. 姙. 娗. 婔. 婘. 媮. 嫨. 嫳. 嬯. 尯. 尶. 屩. 屼. 岃. 岹. 峵. 崊. 嵍. 嵹. 巪. 庤. 廘. 弆. 彔. 彜. 忇. 恉. 悂. 悇/ 悮/ 慡/ 慻/ 憛/ 憝/ 憡/ 懽/ 抷/ 拑 / 拝 / 拫 / 挮 / 揌 / 揓/ 搢/ 撋/ 擝/ 斒/ 旹/ 昛/ 昩/ 曵/ 枊/ 柧/ 栒/ 桊/ 桝/ 梹/ 楳/ 榱/ 橆/ 橎 / 欆!/ 殽"/ 毄#/ 泝$/ 浤%/ 湺&/ 溄'/ 溭(/ 澚)/ 濋*/ 濣+/ 灅,/ 灓-/ 焋./ 焢// 煃0/ 熦1/ 爉2/ 獖3/ 獤4/ 玸5/ 珁6/ 珢7/ 琔8/ 琾9/ 瑃:/ 璢;/ 瓩/ 睘?/ 睩@/ 矰A/ 砽B/ 碃C/ 磹D/ 禶E/ 秇F/ 稡G/ 窻H/ 箥I/ 箺J/ 簽K/ 籕L/ 籷M/ 粨N/ 絁O/ 綉P/ 緸Q/ 罞R/ 罱S/ 膌T/ 芔U/ 茓V/ 荋W/ 菫X/ 萢Y/ 葓Z/ 葹[/ 蒚\/ 蔈]/ 藧^/ 蛨_/ 蠨`/ 袨a/ 裉b/ 裬c/ 襳d/ 觖e/ 觿f/ 訸g/ 詌h/ 詪i/ 詺j/ 諕k/ 諡l/ 謱m/ 譹n/ 豯o/ 貹p/ 賸q/ 赨r/ 趙s/ 踒t/ 踕u/ 蹨v/ 躷w/ 輖x/ 逰y/ 邩z/ 郃{/ 酖|/ 鉌}/ 鉣~/ 銒/ 鋚/ 鋳/ 錇/ 鍹/ 鎛/ 鐕/ 铚/ 镵/ 镻/ 陑/ 霘/ 霶/ 頲/ 飋/ 飠/ 飡/ 飳/ 騱/ 驋/ 鬾/ 魕/ 鮢/ 鯼/ 鰊/ 鱠/ 鳻/ 鴱/ 鵱/ 鷢/ 鸘/ 鸙/ 鸼/ 麧/ 鼄/ 鼊/ 齚/ 떨/ 몇/ 뭐/ 익/ 큼/ / 🎀/ 🔑/ 🖐/ 🗞/ 🧠/ 󠁣/ 󠁴/ ƅ/ Ɛ/ ǽ/ ɥ/ ɧ/ ӏ/ ਭ/ ೋ/ ༦/ 丂/ 仈/ 伬/ 伿/ 倄/ 偆/ 偞/ 僩/ 冃/ 冦/ 劮/ 匛/ 厖/ 咃/ 咢/ 唚/ 唶/ 喤/ 噃/ 囋/ 圢/ 坁/ 垥/ 埯/ 塉/ 壒/ 夑/ 妚/ 妿/ 姄/ 姧/ 姺/ 婟/ 婣/ 媈/ 媿/ 嫈/ 嫯/ 嬀/ 嬥/ 嬬/ 寽/ 屳/ 峩/ 嵲/ 嶔/ 嶲/ 巛/ 巭/ 巵/ 廧/ 徝/ 忬/ 忺/ 恴/ 惙/ 惡/ 愻/ 慒/ 扚/ 挐/ 挘/ 挩/ 捾/ 掯/ 摷/ 撧/ 攦/ 昹/ 昻0 晻0 暀0 暩0 曒0 杒0 枈0 柆0 柎0 栔 0 桼 0 椓 0 椥 0 椮 0 楡0 榚0 槤0 橖0 橥0 檶0 櫆0 欀0 歬0 殎0 泏0 泬0 浗0 湙0 湦0 溠0 滍0 滐0 滺 0 漎!0 漜"0 漨#0 潕$0 濅%0 濿&0 炰'0 烄(0 焂)0 爠*0 牮+0 狆,0 猈-0 猯.0 猶/0 獃00 獽10 珤20 珯30 璷40 璾50 甛60 甮70 畕80 疇90 痡:0 瘲;0 癓<0 眴=0 睅>0 瞘?0 砃@0 砅A0 砇B0 硂C0 硡D0 碢E0 祮F0 禢G0 窷H0 窾I0 竐J0 筳K0 箎L0 箻M0 篍N0 篘O0 篺P0 簉Q0 簱R0 籂S0 粃T0 紃U0 紏V0 紦W0 綎X0 縗Y0 縧Z0 績[0 繖\0 纇]0 纐^0 缻_0 缾`0 罉a0 胷b0 脮c0 膣d0 臢e0 臩f0 舧g0 芿h0 茞i0 茩j0 莋k0 莻l0 萂m0 蒱n0 蓾o0 薠p0 蘯q0 蘷r0 虪s0 蚄t0 蚔u0 蚠v0 蛬w0 蝔x0 蟩y0 蠺z0 袑{0 褵|0 襫}0 覒~0 覔0 詉0 誙0 誧0 譔0 譝0 譪0 貮0 賐0 赯0 赸0 赻0 趠0 趶0 跼0 踛0 踷0 躹0 軆0 輹0 轜0 遌0 遻0 醙0 釳0 鈅0 鈈0 銔0 鍅0 鍙0 鍚0 鎈0 鐂0 鐱0 鑯0 阞0 阦0 隮0 隻0 霐0 靯0 鞧0 韐0 韝0 頾0 飷0 餋0 馉0 馌0 馹0 駠0 髜0 鬒0 鮍0 鰟0 鰿0 鱛0 鱪0 鴈0 鴎0 鵮0 鶌0 鶹0 麄0 黫0 鼧0 鼱0 齺0 념0 밖0 켜0 풀0 혹0 0 0 0 𓆜0 𓆞0 🆆0 🌲0 👦0 ȟ0 ͧ0 ८0 ஐ0 ౪0 ശ0 ീ0 ൂ0 ຜ0 ᴚ0 Ἐ0 ⌋0 ◜0 ♤0 ⚬0 丗0 乕0 亷0 倈0 偭0 傎0 傚0 僎0 僝0 僤0 僾0 儽0 兝0 冿0 刦0 刲0 剘0 剹0 劁0 劋0 劔0 劙0 劤0 厜0 吘0 呾0 唝0 唲0 啴0 噭0 嚝0 囄0 囆0 囥1 囨1 坥1 垻1 墸1 壻1 夶1 娔1 娵1 娹 1 婂 1 婖 1 媅 1 媌 1 媠1 嫊1 岝1 峗1 崪1 嶊1 嶎1 嶣1 嶹1 帬1 幆1 幮1 廗1 廞1 彅1 彠1 彽1 忊1 怳 1 悕!1 惞"1 惼#1 愗$1 愞%1 愹&1 憞'1 扸(1 挔)1 掵*1 掶+1 掻,1 搊-1 摖.1 擧/1 擳01 攕11 敉21 旾31 暪41 曫51 枬61 柕71 柣81 柹91 栶:1 桬;1 楟<1 榥=1 樀>1 樻?1 橧@1 檟A1 櫰B1 歖C1 殗D1 殣E1 毺F1 氀G1 氜H1 氞I1 汘J1 浧K1 淈L1 淍M1 渰N1 湼O1 滀P1 漑Q1 潉R1 澘S1 灐T1 灠U1 炈V1 炓W1 烲X1 焬Y1 煓Z1 煚[1 熛\1 燡]1 犉^1 犺_1 狔`1 猑a1 琙b1 琠c1 璳d1 璴e1 瓱f1 畺g1 瘄h1 皐i1 皨j1 皪k1 瞏l1 瞹m1 砛n1 砠o1 硈p1 碆q1 碔r1 碯s1 碿t1 磗u1 磿v1 礮w1 礳x1 禷y1 秓z1 秶{1 稢|1 穠}1 穸~1 窀1 窌1 窚1 笲1 筙1 箚1 箠1 箤1 箾1 粐1 粭1 紒1 絯1 縰1 繊1 繤1 羛1 羱1 翂1 翭1 脨1 腍1 膗1 臝1 艖1 艧1 芕1 苊1 茣1 荵1 莌1 菢1 萗1 葔1 蔋1 薀1 薽1 虁1 虊1 蛍1 蛒1 蛦1 蛿1 蝍1 蠎1 裖1 褎1 襮1 覟1 觪1 訄1 詜1 誽1 謓1 謩1 譃1 譭1 讐1 讑1 豲1 貋1 贋1 贍1 赱1 趦1 踂1 蹔1 軨1 軷1 輘1 轛1 轝1 迚1 迿1 逎1 郳1 鄊1 鄕1 酏1 酛1 醨1 釾1 銉1 鋺1 錩1 鍓1 鏬1 鐈1 闣1 阥1 阩1 陣1 隿1 霿1 靸1 鞄1 鞳1 韛1 頀1 餱1 饟1 馂1 駇1 駗1 駞1 騧1 骱1 鯒1 鰎1 鰵1 鲾1 鳠1 鴀1 鴲1 鵞1 鷤1 鷵1 麛1 鼆1 齸1 깔1 낸1 떡1 랙1 털1 휘1 1 1 😯1 🛍2 🦐2 󠁬2 󠁷2 Ǯ2 ˖2 ϋ2 ਛ2 ભ2 ൺ 2 ඉ 2 ዓ 2 ዮ 2 ጿ 2 ᚷ2 Ế2 丣2 伆2 伵2 佸2 侚2 僟2 儌2 儗2 儢2 儯2 凅2 勼2 匴2 厎2 咉2 咊2 咍 2 咟!2 哠"2 啚#2 喐$2 喨%2 嚚&2 嚡'2 囙(2 垝)2 垺*2 埻+2 埾,2 堏-2 堓.2 堦/2 塦02 壌12 壣22 奛32 娊42 娕52 娰62 婅72 婌82 婜92 媇:2 媣;2 嫾<2 嬐=2 嬕>2 孄?2 孎@2 孏A2 孷B2 宭C2 屘D2 屭E2 峐F2 峢G2 巤H2 庒I2 庺J2 弮K2 忛L2 恘M2 恮N2 恾O2 悾P2 惿Q2 愱R2 慿S2 憓T2 懁U2 懕V2 戜W2 戼X2 扵Y2 抝Z2 抣[2 抧\2 拀]2 拸^2 捇_2 捸`2 掚a2 搲b2 摦c2 摵d2 擭e2 擸f2 攋g2 敇h2 敥i2 敹j2 斁k2 昽l2 晪m2 暶n2 曞o2 柟p2 柸q2 栧r2 梤s2 棬t2 椯u2 楒v2 楥w2 檓x2 檹y2 歀z2 歂{2 歄|2 殐}2 殬~2 毃2 汍2 沲2 沶2 涾2 溙2 滖2 滣2 漧2 漮2 漰2 潅2 澰2 澼2 濃2 炂2 焀2 焇2 焎2 煏2 熌2 熽2 牫2 狅2 狋2 狧2 獏2 玣2 琟2 甹2 畼2 疧2 疩2 痖2 痷2 瘭2 癛2 癪2 皁2 皔2 盨2 眆2 眖2 睉2 睠2 瞣2 矄2 矷2 硄2 礒2 礜2 礧2 祏2 禍2 稯2 穒2 穟2 笘2 笯2 筈2 筕2 筤2 箮2 篞2 簗2 簮2 籹2 粞2 糉2 糩2 紵2 綯2 緌2 緣2 縀2 繟2 罠2 羉2 翞2 聟2 肊2 胘2 胻2 腝2 腞2 臇2 臒2 芇2 芧2 苰2 莣2 莥2 葂2 葋2 葲2 蒁2 蒷2 蔢2 蕐2 蕑2 蕔2 薱2 蘫2 蘲2 蘽2 虗2 虰2 虵2 蚛2 蚾2 蜁2 蜏2 蝃2 蝪2 蝯2 蟰2 蠂2 蠭2 衻2 裪2 褠2 襒2 襭2 覛2 訯2 諌2 諴2 諵2 謒2 讂3 讛3 诪3 谽3 賔3 賖3 賙3 賹3 贕3 赾 3 趢 3 趷 3 跁 3 踣 3 蹡3 蹪3 蹹3 躀3 躆3 軑3 軚3 軶3 輮3 輷3 輺3 迃3 遝3 遤3 遾3 郻3 郼3 鄵3 醗 3 鈆!3 鈊"3 鈶#3 鉠$3 鉧%3 鋣&3 錝'3 鍲(3 鐡)3 阣*3 阤+3 隡,3 霡-3 靕.3 靝/3 韄03 頽13 颪23 颸33 餎43 餔53 餴63 馦73 駽83 驙93 骫:3 髞;3 髟<3 魮=3 魸>3 鮵?3 鯮@3 鰄A3 鱊B3 鳮C3 鶖D3 鹲E3 麖F3 麜G3 麞H3 麷I3 黸J3 齱K3 龞L3 龡M3 봄N3 셨O3 앙P3 콜Q3 𓆡R3 🍂S3 🍽T3 👽U3 🕳V3 🗳W3 🦀X3 ƗY3 ɩZ3 ʣ[3 Ҡ\3 ֻ]3 ؄^3 భ_3 ഗ`3 ഞa3 ཾb3 ዘc3 ⃗d3 ↆe3 ▰f3 ◮g3 ☢h3 ⬢i3 ⻘j3 ヲk3 乚l3 仸m3 佌n3 侊o3 侫p3 侺q3 俌r3 俹s3 偑t3 偡u3 偧v3 傋w3 傐x3 傔y3 傗z3 僘{3 僞|3 儶}3 兣~3 冐3 凗3 刓3 剓3 剕3 剟3 剫3 劐3 勎3 勫3 匁3 卋3 叐3 哊3 唩3 唺3 啌3 啿3 嘒3 嘽3 噊3 噵3 嚂3 嚈3 垀3 堲3 堻3 塎3 壈3 夦3 夰3 夽3 奫3 奼3 妏3 姟3 娋3 婏3 婙3 婤3 婲3 媗3 嫝3 嫴3 嫷3 嬄3 嬨3 孼3 宖3 寃3 尰3 屖3 屴3 岊3 峏3 峱3 崈3 崨3 崶3 嵉3 嵻3 嶅3 巸3 帣3 帾3 庼3 廫3 弉3 弽3 彯3 徃3 徎3 徸3 怗3 恌3 悳3 惤3 憥3 憼3 懆3 扄3 抌3 挌3 挓3 捿3 掞3 搝3 擑3 攂3 攇3 攐3 敂3 敪3 敱3 旙3 旽3 昮3 曉3 杣3 柮3 桛3 棔3 棶3 椡3 榩3 槞3 樲3 檖3 檥3 櫱3 欚3 歛3 殧3 殱3 毚3 毰3 汃3 汅3 沯3 泒3 洅3 洖3 洝3 浺3 涋3 涬3 渳3 湆3 溤3 漐4 漝4 潀4 澬4 瀦4 烇4 焻4 熖4 熪4 熫 4 牕 4 犃 4 犠 4 狕 4 玂4 玽4 珳4 琝4 瓨4 畨4 痁4 癷4 皶4 盄4 盕4 盫4 睧4 碥4 碫4 磠4 礍4 礑4 祌 4 祤!4 禸"4 秔#4 秛$4 竁%4 竏&4 竪'4 筀(4 筓)4 箆*4 篲+4 簆,4 簻-4 籦.4 籾/4 粌04 糽14 紞24 絍34 絙44 綪54 緉64 緢74 緭84 縂94 繷:4 绖;4 罶<4 羺=4 翸>4 聕?4 聙@4 聺A4 胊B4 胵C4 胹D4 腉E4 膕F4 膪G4 舡H4 芆I4 芢J4 苉K4 苝L4 茟M4 莿N4 萠O4 葄P4 葥Q4 葼R4 葽S4 蒩T4 蓚U4 蓪V4 蔇W4 蔖X4 蔠Y4 蔣Z4 蕍[4 蕽\4 薘]4 蘎^4 蘜_4 蘟`4 蘨a4 虀b4 虋c4 蚈d4 蜬e4 蝚f4 蝬g4 蟃h4 蟐i4 蟫j4 蠷k4 衃l4 衱m4 袥n4 裭o4 褏p4 覸q4 觧r4 觩s4 訹t4 誟u4 謻v4 豗w4 貙x4 賓y4 贀z4 跾{4 踤|4 踼}4 軿~4 輬4 轑4 轘4 迯4 迲4 逿4 郙4 鄸4 鄽4 酫4 鈎4 鈜4 鈯4 銆4 銞4 銶4 鋊4 鋋4 鋫4 錦4 錧4 錽4 鍠4 鎚4 鏧4 鏶4 鏺4 鐥4 鑍4 钖4 锳4 闚4 阹4 隓4 隞4 隫4 靋4 靮4 靲4 靻4 韂4 韑4 飱4 馵4 馷4 駫4 騉4 騹4 驘4 髊4 髾4 鬋4 鬜4 魯4 鯌4 鯎4 鯟4 鯠4 鰂4 鰘4 鱑4 鳛4 鳭4 鴡4 鴸4 鵔4 鶙4 麘4 黇4 鼔4 齋4 겁4 얻4 4 4 4 4 🆃4 🛎4 Ɠ4 Ǎ4 ˲4 ܣ4 ܳ4 థ4 ෂ4 ቸ4 ហ4 ᠾ4 ῃ4 ⏯4 ♨4 ⛽4 ✂4 㚼4 亁4 亠4 佒4 佮4 佱4 佹4 俰4 倵4 偄4 傉4 傮4 傺4 僀4 僄4 僆4 僓4 僔4 僸4 儛4 儥4 儴4 冘4 凩4 凴4 剢4 剬4 剾4 劍4 勔4 匟4 卂4 叅4 吅4 呟4 咹5 哃5 哣5 哹5 哾5 嗃5 嗻5 嗿5 嘥5 噟 5 嚁 5 嚘 5 嚬 5 嚹 5 囎5 囩5 圡5 坔5 坕5 坱5 埬5 埱5 堁5 堐5 堩5 塠5 墇5 墭5 墽5 壏5 壴5 奙5 奺 5 妶!5 姕"5 姛#5 娝$5 娾%5 媓&5 嫭'5 嬶(5 孇)5 屵*5 崌+5 嵁,5 嶀-5 嶕.5 嶥/5 帗05 幝15 廹25 弐35 彾45 徆55 怐65 怓75 怬85 怶95 恈:5 恧;5 悤<5 惒=5 愥>5 愩?5 慉@5 憜A5 懮B5 懴C5 抳D5 抾E5 捓F5 揗G5 搈H5 搒I5 搩J5 摙K5 摪L5 撪M5 撱N5 擌O5 攉P5 敊Q5 敤R5 敵S5 斀T5 旈U5 旪V5 昢W5 暞X5 朡Y5 朸Z5 杗[5 枃\5 枍]5 枲^5 柖_5 栣`5 桾a5 梛b5 棴c5 椄d5 楈e5 楌f5 榓g5 榠h5 槯i5 槼j5 樚k5 樢l5 樦m5 橞n5 橭o5 櫣p5 櫹q5 欌r5 欐s5 欗t5 殕u5 毼v5 氝w5 沗x5 沝y5 沠z5 泆{5 洷|5 涭}5 涻~5 淐5 湰5 溚5 漀5 潂5 潎5 潧5 潻5 澁5 濍5 瀺5 灷5 炋5 炐5 炧5 炿5 烼5 焃5 焍5 焝5 焥5 煋5 煷5 煹5 熅5 熲5 燖5 燤5 牥5 犦5 犪5 狖5 狟5 猼5 獆5 獱5 玃5 玴5 珻5 琘5 瓎5 甂5 畉5 畖5 畞5 畧5 畷5 癶5 皾5 盶5 睊5 睴5 瞆5 矐5 矑5 砘5 砿5 硞5 磇5 礉5 礟5 禂5 禕5 禣5 禭5 秙5 秠5 秱5 稁5 稬5 穃5 穇5 穛5 笇5 笒5 簈5 籊5 籖5 粸5 糺5 緟5 縕5 繌5 繶5 罦5 羃5 羢5 羾5 翖5 聇5 聠5 肒5 肶5 脼5 腷5 臿5 艎5 艑5 艢5 艦5 苃5 荮5 莟5 莰5 菳5 菾5 萲5 萴5 萻5 葈5 蒥5 蓒5 蓨5 蔉5 蕬5 蕳5 蕿5 薣5 薥5 藌5 藸5 藼5 蘓5 蘮5 虂5 虌5 虳5 虶5 蚘5 蛝6 蜠6 蜲6 蝿6 螪6 蟞6 蟤6 蟱6 衏6 衐 6 衧 6 衵 6 裋 6 裑 6 褃6 襍6 觙6 觝6 觫6 訷6 詤6 詽6 誱6 諐6 謟6 譓6 譗6 谫6 豣6 豥6 賵6 贑6 趑 6 趝!6 踸"6 蹧#6 蹮$6 躛%6 躽&6 軃'6 軳(6 辥)6 逌*6 遀+6 遖,6 邉-6 邖.6 鄋/6 醘06 醥16 醱26 釮36 鉁46 鉕56 鉩66 銐76 鋙86 錋96 鍕:6 鍳;6 鎶<6 鏯=6 鐊>6 鐞?6 鐾@6 钘A6 铇B6 闋C6 闏D6 阠E6 陊F6 陹G6 隟H6 雭I6 霵J6 靣K6 靹L6 靽M6 韾N6 顠O6 颬P6 颴Q6 飌R6 飝S6 餬T6 饎U6 馣V6 駩W6 驚X6 髬Y6 鬶Z6 鬸[6 鬽\6 鯍]6 鯱^6 鰛_6 鱂`6 鱃a6 鱙b6 鴢c6 鴮d6 鵄e6 鵖f6 鶠g6 鶳h6 鷆i6 鷰j6 鷾k6 鸆l6 鸉m6 鸌n6 鸴o6 鹻p6 麀q6 麃r6 麌s6 黖t6 鼗u6 鼫v6 齥w6 ꆏx6 농y6 흡z6 {6 |6 }6 ︆~6 🇼6 🌐6 🐺6 🖖6 🖼6 😝6 🟩6 🧀6 ŕ6 ɳ6 ӗ6 ӥ6 ֑6 ݄6 ଜ6 ଡ6 ജ6 ൽ6 ຢ6 ༶6 ཿ6 ჭ6 ሆ6 ሕ6 ቶ6 Ố6 ‱6 ↣6 ⌯6 ⚕6 ⲙ6 ⴰ6 ㄍ6 乧6 乺6 仠6 伃6 伭6 伻6 佄6 佡6 侲6 偖6 偱6 僗6 僙6 儊6 儰6 冩6 劯6 勽6 匼6 叧6 吪6 唈6 唋6 啀6 啠6 喍6 嘇6 嘐6 嘕6 嘨6 噂6 嚿6 圁6 圗6 圸6 圼6 垽6 埢6 堭6 堹6 墬6 壛6 夓6 夛6 奒6 奝6 奬6 奵6 妔6 娚6 娢6 娮6 娯6 娻6 婋6 婱6 媘6 媶6 嫸6 嬠6 嬧6 孈6 孧6 寴6 尩6 岏6 峟6 崉6 嵦6 嵰6 嵶6 嵺6 巃6 巈6 巕6 帩6 幋6 幐6 庴6 廵6 彄6 徹6 徿6 忟6 扢6 扨6 抅6 抭6 挴6 挵6 捝6 掍6 掝6 掫6 掽6 揚6 揞7 揟7 搄7 搕7 搾7 撜7 撯7 擡7 擩7 攈 7 攑 7 攧 7 敄 7 斠 7 旘7 暯7 朒7 朣7 杔7 柉7 栐7 桮7 桽7 梀7 梇7 棊7 椂7 椊7 椔7 椼7 楃7 楑7 楻 7 槏!7 槫"7 樏#7 樐$7 樒%7 櫢&7 欇'7 歮(7 殀)7 泟*7 浵+7 渹,7 湡-7 溊.7 溸/7 澦07 濆17 濪27 瀁37 瀂47 灖57 灨67 炄77 炵87 焟97 焳:7 煔;7 煯<7 熋=7 熞>7 熭?7 熼@7 燪A7 燲B7 燸C7 爖D7 犐E7 狓F7 狘G7 狾H7 珫I7 瑍J7 璑K7 璔L7 瓰M7 甈N7 畻O7 疺P7 痑Q7 痝R7 痻S7 痽T7 癹U7 皒V7 皡W7 眽X7 睋Y7 瞲Z7 瞷[7 矊\7 矨]7 矪^7 矱_7 矹`7 砄a7 硓b7 硘c7 硭d7 硻e7 硾f7 碻g7 磂h7 磑i7 磳j7 磶k7 礕l7 礛m7 礯n7 礻o7 祒p7 禌q7 禴r7 秄s7 稑t7 稭u7 穄v7 窉w7 竴x7 笉y7 笍z7 笩{7 笰|7 筸}7 箼~7 簒7 粇7 粩7 粺7 綅7 綌7 縡7 繗7 繪7 羬7 翏7 翑7 翣7 翧7 耝7 耠7 耹7 聀7 肰7 肵7 腂7 腣7 舑7 舮7 艊7 艥7 芅7 芖7 荎7 荢7 莊7 菋7 菬7 萅7 蒍7 蓃7 蓅7 蓔7 蓺7 蔅7 蕣7 薻7 薼7 薾7 藂7 藆7 蘈7 蘏7 蚎7 蛖7 蜽7 蟉7 蟙7 蠝7 蠧7 蠪7 蠬7 襎7 襔7 襼7 誴7 謮7 譋7 譜7 譥7 譱7 讆7 讝7 讻7 豃7 豄7 豋7 豷7 貛7 賮7 賿7 趻7 跿7 踧7 蹍7 躃7 軮7 輣7 輰7 轈7 迕7 迠7 遉7 遬7 邿7 郥7 郲7 鄐7 醹7 釺7 鉜7 鋄7 鋶7 鍌7 鍒7 鏔7 鏫7 鐌7 鐤7 鐧7 鐭7 鑝7 鑱7 铔7 铻7 閧7 閴7 闤7 闬7 陖7 隂7 隃7 鞾7 韍7 韲7 頨7 顨7 顪7 顭7 飿7 餟7 馢7 馿7 駵7 駸8 騜8 魵8 鮎8 鮡8 鮣8 鯛8 鰽8 鱅8 鱶 8 鳪 8 鴚 8 鵀 8 鵍 8 鵛8 鵵8 鶊8 鶐8 鹶8 黕8 鼌8 鼲8 鼼8 齰8 齹8 龂8 龥8 ꀋ8 ꈍ8 낌8 넣8 펜8 푸 8 !8 🌤"8 🍻#8 👸$8 💘%8 😻&8 ǣ'8 ɫ(8 ʠ)8 ΀*8 Ң+8 ҭ,8 Ӏ-8 ۀ.8 ܕ/8 ஹ08 ఛ18 ೬28 ༨38 ቃ48 ញ58 ↦68 ⌚78 ⬜88 丮98 丳:8 乵;8 仢<8 仭=8 仹>8 侞?8 侼@8 俫A8 倧B8 倲C8 倸D8 傂E8 傞F8 傼G8 僇H8 僯I8 儧J8 兏K8 兟L8 冺M8 刾N8 勚O8 卶P8 卼Q8 厱R8 叓S8 叜T8 咓U8 咮V8 咶W8 哘X8 哢Y8 哫Z8 唓[8 唼\8 嗧]8 嗴^8 嘪_8 噑`8 噡a8 噧b8 嚞c8 嚤d8 囓e8 圲f8 垘g8 堗h8 塴i8 墋j8 壜k8 奍l8 奦m8 妴n8 娙o8 娬p8 娺q8 婄r8 婎s8 媉t8 媥u8 媷v8 嫞w8 嫢x8 嬚y8 孂z8 孶{8 宼|8 屸}8 岒~8 峛8 峬8 峮8 崅8 崡8 嵀8 嵂8 嵃8 嶐8 嶢8 嶰8 巼8 帠8 帺8 幚8 幜8 庮8 庳8 廔8 廼8 忁8 忎8 忯8 恡8 惄8 愐8 愖8 慭8 慲8 憄8 憌8 憪8 憴8 憽8 扴8 拰8 挆8 挏8 掆8 揅8 揘8 搑8 搯8 摌8 摎8 摰8 摱8 擬8 攨8 攩8 敃8 敓8 敿8 斘8 昣8 昿8 暵8 曋8 曐8 曮8 朎8 杦8 杽8 枔8 桇8 桚8 梬8 棄8 椙8 楇8 楋8 楎8 榗8 槇8 橀8 橌8 橻8 檊8 檝8 櫍8 櫻8 欑8 欘8 欨8 欩8 欭8 殌8 殹8 毊8 毝8 毠8 汻8 泧8 洂8 洟8 涆8 涚8 涱8 淊8 渂8 溨8 溳8 溹8 澕8 濗8 瀓8 炏8 炦8 烋8 烞8 烳8 焸8 煁8 煵8 爋8 牐8 犈8 狪8 猓8 猧8 獁8 獄8 獥8 瓗8 甆8 畃8 畍8 畐8 畭8 疻9 痓9 痥9 癅9 癙9 癧9 癨9 盚9 盠9 盢 9 眪 9 矀 9 砡 9 磍 9 礄9 礝9 秌9 秪9 秳9 秹9 稆9 稒9 穣9 穤9 窛9 窧9 竓9 竔9 竚9 竵9 笀9 篃9 篈 9 篧!9 簂"9 簛#9 簢$9 籢%9 籶&9 粆'9 粔(9 糄)9 紐*9 紤+9 絼,9 綇-9 緍.9 縍/9 縥09 繏19 罀29 罜39 罭49 羗59 翄69 翲79 翿89 耼99 腃:9 腢;9 膓<9 膧=9 膼>9 臄?9 臽@9 舩A9 艬B9 芌C9 苽D9 茠E9 荗F9 荺G9 莡H9 菤I9 葊J9 蒼K9 蓆L9 蓫M9 蓭N9 蓱O9 蓶P9 蔂Q9 薋R9 薵S9 藃T9 藡U9 蘐V9 蘕W9 蘴X9 虆Y9 虼Z9 蚇[9 蛼\9 蝞]9 螁^9 螚_9 蟔`9 蟗a9 蠁b9 衉c9 衚d9 袀e9 袹f9 裻g9 褋h9 褩i9 襑j9 襘k9 襷l9 覝m9 觡n9 詚o9 詶p9 誝q9 諝r9 諰s9 謘t9 謧u9 譇v9 豮w9 貃x9 貈y9 貸z9 賗{9 贒|9 贘}9 赩~9 赹9 趭9 跊9 踍9 踾9 躻9 軈9 軭9 輈9 輎9 轒9 轓9 轥9 辌9 迊9 遞9 遧9 邫9 郂9 鄹9 鄼9 酘9 酳9 醿9 釶9 釸9 鈏9 鉊9 鉷9 銋9 銌9 鋀9 鋘9 鋭9 錿9 鍜9 鍯9 鍴9 鏋9 鏣9 鏪9 鐇9 鐚9 鑅9 鑦9 锽9 镹9 镼9 閳9 闅9 闑9 阭9 雗9 雘9 霋9 霥9 靉9 靎9 靱9 鞆9 鞺9 韅9 韔9 頥9 顖9 颣9 颽9 飂9 飵9 餙9 餤9 饓9 饛9 饡9 馯9 駬9 騋9 騎9 驇9 驔9 髸9 鬀9 魓9 魡9 魰9 鮙9 鮛9 鰋9 鰝9 鳱9 鴫9 鴵9 鶝9 鹙9 麨9 麭9 黦9 鼟9 鼪9 齌9 齯9 龣9 걷9 닝9 둥9 딩9 9 🍺9 🥣9 🪐9 Ȉ9 ˦9 Ϗ9 ֮9 ߍ9 ઞ9 ઺9 ૓9 ଆ9 ౿9 ຖ9 ຽ9 ᅮ9 ኣ9 ዊ9 ᐕ9 ᖅ9 ῥ9 ⇦9 ✹: 〰: ㄇ: 䭲: 乲: 伣: 侢: 侭: 偒: 偝 : 偤 : 傌 : 傽 : 僌 : 僒: 僫: 僶: 儵: 兠: 冄: 凞: 刡: 勌: 勥: 勶: 匔: 匬: 匳: 厇: 厼: 吤: 吷: 唀 : 唂!: 啈": 噇#: 噒$: 噰%: 嚛&: 嚪': 囕(: 囜): 圚*: 圝+: 坆,: 垗-: 埁.: 堘/: 堟0: 堧1: 堳2: 塯3: 塲4: 塷5: 壖6: 壼7: 夌8: 妜9: 妢:: 妷;: 姎<: 姡=: 婛>: 婽?: 媍@: 媎A: 媐B: 嫤C: 嬜D: 嬳E: 孊F: 孴G: 宎H: 寊I: 寏J: 尞K: 屇L: 岎M: 岲N: 峉O: 峞P: 崕Q: 嵆R: 嵈S: 嵢T: 嵤U: 嵱V: 嵼W: 嵽X: 嶚Y: 嶛Z: 嶠[: 巀\: 巁]: 帎^: 幈_: 幏`: 幑a: 庛b: 庩c: 廜d: 弅e: 彲f: 彺g: 徛h: 徱i: 忕j: 恝k: 悿l: 愑m: 愘n: 慽o: 憦p: 憭q: 懀r: 扜s: 抸t: 捄u: 捗v: 捘w: 捤x: 捥y: 捹z: 掁{: 掦|: 掿}: 揑~: 摀: 摠: 摬: 摴: 撠: 攃: 攓: 斣: 旜: 旝: 昈: 昍: 昸: 晠: 暔: 曻: 枎: 枺: 柡: 栙: 栞: 栮: 栵: 梍: 梑: 梸: 棁: 棃: 椵: 楱: 榁: 榤: 槈: 槦: 樈: 橍: 橮: 檷: 檼: 櫙: 欈: 歍: 殨: 泤: 泲: 洦: 渮: 渵: 湅: 湥: 湳: 湷: 溗: 溾: 滧: 漍: 潪: 澓: 濌: 濏: 濧: 瀭: 瀯: 瀵: 灛: 灢: 灩: 炍: 炑: 炾: 焨: 焵: 焺: 煄: 煈: 熝: 熰: 燽: 爁: 牓: 犔: 狥: 狦: 狶: 獉: 獕: 玁: 瓻: 甋: 畁: 畮: 痯: 瘉: 瘚: 癊: 盺: 睤: 睰: 睵: 瞚: 矔: 砓: 硆: 硶: 碀: 碊: 碷: 磟: 礊: 礏: 礢: 礲: 祰: 祳: 秝: 秺: 稏: 稨: 稩: 稪: 穧: 穪: 穳: 窋: 竫: 竱: 笷: 筺: 簳: 簶; 籵; 粅; 糦; 紛; 紭; 紾; 緃; 緆; 緔 ; 縙 ; 繲 ; 纆 ; 纉 ; 绬; 耚; 耫; 聮; 脄; 脵; 腒; 腲; 膹; 臮; 臷; 舤; 舼; 舿; 艠; 苀; 茥; 茪; 茾 ; 荂!; 荙"; 荝#; 荰$; 莁%; 莤&; 菞'; 菭(; 蒢); 蒮*; 蒶+; 蓏,; 蓸-; 蕯.; 薓/; 薳0; 藘1; 藙2; 藣3; 藮4; 藯5; 蘠6; 虈7; 蚑8; 蛈9; 蛥:; 蜭;; 蝅<; 蝹=; 螏>; 螑?; 蟖@; 蟟A; 蟵B; 蠄C; 蠞D; 蠣E; 袲F; 袺G; 袼H; 裚I; 褈J; 褗K; 褦L; 褨M; 褷N; 襉O; 襚P; 覎Q; 覡R; 訧S; 訩T; 訿U; 詍V; 詷W; 誕X; 諈Y; 謭Z; 譀[; 豜\; 豧]; 貿^; 賺_; 贔`; 趃a; 跒b; 跮c; 踎d; 踚e; 踳f; 蹆g; 蹜h; 軗i; 軜j; 輌k; 輭l; 轇m; 辠n; 辧o; 逓p; 郣q; 鄏r; 鄥s; 鄦t; 酁u; 酅v; 醝w; 醶x; 釐y; 釥z; 鈃{; 鉹|; 銁}; 銇~; 銝; 鋷; 錊; 錍; 錜; 錵; 錺; 鎃; 鎐; 鎒; 鎡; 鏍; 鏩; 鑞; 钁; 钷; 铘; 閅; 闓; 闝; 陭; 陮; 隖; 霚; 霻; 靘; 鞎; 韼; 頎; 頟; 頪; 頶; 飮; 饍; 饚; 駊; 騘; 騢; 驒; 骴; 骾; 髝; 髠; 鬇; 鬏; 鬛; 鮤; 鮬; 鮹; 鰃; 鰗; 鰪; 鱝; 鲄; 鳈; 鳬; 鴋; 鴺; 鵌; 鶋; 鶫; 鶱; 鷘; 鷡; 鷶; 鸐; 麍; 麰; 黆; 黓; 黤; 黵; 鼅; 鼝; 齖; 齘; 龓; ꋍ; 므; 쇄; 숭; ; ︀; 🎊; 🏹; 👧; 🤠; 🥉; Ƙ; Ȱ; ʉ; ѩ; ҆; ҩ; ٤; ۩; ݿ; ދ; ࡬; ॷ; ৎ; ਣ; ੤; ૲; ଄; ଢ; ଫ; ୼; ௳; ೅; അ; ഽ; ඘; ඳ; ቼ; ኈ; Ṛ; ⚗; ➺; ⲥ; 丵; 亪; 仴; 仺; 伮; 佨; 佫; 俋; 俓; 俕< 俷< 倂< 倕< 偗< 傦< 傫< 僛< 僲< 儖 < 儱 < 儳 < 兊 < 凔 < 凟< 刢< 刵< 剦< 剺< 劥< 劽< 勪< 匂< 匤< 匥< 叝< 呁< 咞< 啑< 喖< 喣< 嗀< 嗠 < 嘺!< 噆"< 噣#< 噽$< 嚃%< 嚑&< 嚾'< 圠(< 圥)< 圱*< 坹+< 埍,< 埿-< 堢.< 塣/< 塧0< 塶1< 塸2< 塼3< 墄4< 墰5< 墶6< 壍7< 壡8< 奲9< 妌:< 妵;< 姖<< 姰=< 姼>< 娤?< 婒@< 婝A< 媋B< 媻C< 嫟D< 嫥E< 嬺F< 孁G< 宐H< 宷I< 寠J< 屶K< 岉L< 岻M< 峔N< 崷O< 嵄P< 嵑Q< 嵵R< 帵S< 幖T< 廭U< 弍V< 彉W< 徍X< 忷Y< 怸Z< 怺[< 恀\< 悈]< 悑^< 悡_< 愮`< 慀a< 慯b< 懄c< 懭d< 戺e< 扊f< 扗g< 拵h< 捰i< 捵j< 揃k< 揋l< 揢m< 揧n< 搘o< 搸p< 搹q< 摉r< 摗s< 撌t< 攼u< 敁v< 敠w< 斅x< 斏y< 晽z< 暚{< 暽|< 曁}< 杅~< 杛< 枤< 枦< 柗< 柭< 栯< 梄< 梖< 棛< 椆< 椞< 椶< 樔< 橨< 橷< 檧< 檨< 檪< 檮< 櫐< 櫮< 櫯< 櫴< 櫾< 欦< 欴< 歅< 歏< 殜< 殩< 殰< 毥< 氼< 汋< 汥< 沑< 泙< 涒< 淃< 渘< 渜< 渞< 湱< 溕< 漄< 漻< 潖< 潛< 潬< 濓< 濴< 濻< 瀐< 瀒< 瀷< 灍< 炗< 炛< 炲< 烠< 烶< 烻< 焁< 焏< 焴< 焷< 燒< 燷< 爄< 爓< 爯< 爴< 爼< 牔< 犕< 犡< 狑< 猀< 猘< 獚< 獯< 瓟< 甊< 甶< 畗< 疐< 疜< 瘎< 瘨< 癎< 癏< 癑< 皉< 皘< 盙< 盷< 眒< 眢< 眧< 睓< 睕< 睳< 瞝< 矕< 砋< 砫< 砮< 硲< 碪< 磫< 礰< 礶< 祋< 禫< 秎< 秞< 穅< 穕< 穩< 窞< 窮< 竗< 竲< 竳< 笂< 笌< 笟< 箊< 箞< 箣= 篣= 籇= 籛= 糚= 糱= 紴= 絏= 絾= 綥 = 綨 = 緤 = 緧 = 繂 = 繺= 纎= 罋= 罖= 罼= 羷= 羻= 翝= 聗= 聜= 肂= 肑= 胩= 腖= 腶= 膒= 膖= 膤= 臰 = 艐!= 艛"= 艝#= 艞$= 苬%= 苿&= 茖'= 茽(= 荿)= 莦*= 莬+= 莭,= 萚-= 萮.= 萰/= 蒳0= 蔩1= 蔾2= 蕇3= 蕒4= 薝5= 薩6= 藚7= 蘤8= 蘾9= 虴:= 虷;= 蚏<= 蚟== 蚮>= 蛂?= 蛯@= 蜌A= 蝒B= 蝵C= 螇D= 螌E= 螤F= 螸G= 蟦H= 蠗I= 袇J= 袙K= 袠L= 袣M= 裶N= 褘O= 褢P= 褭Q= 褯R= 襈S= 襺T= 襾U= 觗V= 觵W= 訏X= 訜Y= 詟Z= 誜[= 誢\= 誩]= 誷^= 諀_= 諆`= 諎a= 謸b= 謿c= 譑d= 譮e= 讁f= 讋g= 讟h= 诐i= 谉j= 豦k= 豵l= 貭m= 賘n= 贃o= 赥p= 趍q= 趤r= 趮s= 趰t= 跙u= 踗v= 踲w= 躈x= 軰y= 輄z= 輆{= 輠|= 輨}= 輩~= 轊= 迵= 邜= 郋= 郒= 郰= 郺= 鄿= 醎= 醩= 釛= 鈒= 鈪= 鈵= 鉔= 鉽= 銄= 銗= 銰= 銵= 鋡= 鋻= 錌= 鍦= 鎁= 鎼= 鐆= 鐢= 鐪= 鑉= 鑔= 鑵= 鑺= 閛= 闀= 隌= 雟= 雴= 霫= 靁= 靔= 靪= 頢= 頿= 顉= 顐= 顦= 颾= 餂= 餈= 餠= 餲= 餷= 饄= 饖= 饠= 馧= 馩= 馺= 駮= 騤= 騯= 騵= 髇= 髉= 髱= 鬉= 鬡= 鬰= 鬵= 魊= 魫= 魳= 魼= 鮁= 鮆= 鮦= 鮯= 鮳= 鮷= 鯅= 鯋= 鯐= 鯗= 鯞= 鯯= 鰐= 鰞= 鰡= 鱱= 鱾= 鴌= 鴍= 鴬= 鵇= 鵊= 鵧= 鵴= 鵻= 鶥= 鶧= 鶮= 鸃= 鸏= 麆= 麥= 黬= 鼶= 齳= 齽= 龏= ꇬ= 럭= 쁜= 턴= = 𖠚= 🌷= 🎭= 📞= 🖊= 🗺= 🤧= ƾ= ǵ= ѷ= Ҕ= Ҟ= ӓ= Ӧ> ֚> ؚ> ރ> ߝ> ߵ> ࠱> ࠲> ࣻ> ૽ > ଦ > ౯ > ౰ > ೲ > ബ> ൎ> ༻> ቱ> ።> ᑦ> ᛁ> Ả> Ἑ> ὄ> ≲> ┏> ⛱> ✥> 乢> 亣> 仾> 倃> 偢 > 偼!> 僃"> 僿#> 儓$> 儣%> 儭&> 冝'> 冡(> 冸)> 刔*> 剙+> 剨,> 劰-> 勏.> 勡/> 厠0> 咈1> 哰2> 唃3> 唌4> 唣5> 唹6> 喩7> 嗁8> 嗛9> 嘃:> 嘋;> 嘑<> 嘓=> 嘳>> 噞?> 嚵@> 坢A> 垊B> 埓C> 塇D> 塓E> 塥F> 墡G> 墥H> 墲I> 壥J> 夈K> 夻L> 奾M> 妽N> 姭O> 姾P> 娦Q> 娨R> 娽S> 婍T> 婐U> 婗V> 婾W> 媦X> 媩Y> 媹Z> 嫹[> 嬏\> 嬒]> 嬻^> 孾_> 宆`> 宒a> 尀b> 岋c> 峳d> 峷e> 峺f> 崒g> 崯h> 嵣i> 嵮j> 嶞k> 巄l> 巙m> 巠n> 巬o> 帓p> 帿q> 幉r> 幭s> 庈t> 庉u> 庡v> 庢w> 庣x> 廅y> 廇z> 徤{> 徫|> 怈}> 怟~> 恖> 悜> 愇> 愲> 慱> 憂> 憈> 憢> 憣> 憲> 懎> 懝> 戤> 扝> 扤> 捒> 摐> 摣> 摫> 撗> 撴> 擨> 攅> 攭> 敒> 斞> 旓> 旫> 晿> 暙> 暿> 曎> 曏> 曔> 曘> 曪> 朤> 朻> 杴> 桘> 桹> 梉> 梚> 梷> 棌> 椌> 椖> 椢> 椷> 榡> 榲> 槡> 槶> 樃> 樜> 橔> 檡> 檤> 欍> 欓> 欶> 毸> 毾> 氁> 汌> 沎> 泋> 浢> 淕> 淗> 渨> 湬> 溑> 滫> 漘> 漽> 澙> 澭> 澸> 濜> 濶> 瀔> 瀿> 灆> 灟> 灡> 灻> 炨> 烣> 焒> 焲> 煻> 爚> 爣> 犅> 犥> 犿> 猭> 獜> 瓭> 瓸> 甇> 甠> 甼> 畓> 畘> 畾> 疂> 疶> 痸> 瘱> 瘹> 癁> 癈> 癝> 癦> 皣> 皫> 盓> 眓> 眕> 瞜> 瞨> 矆> 矉> 砱> 硊> 硵> 碂> 碬? 碸? 礃? 礭? 祣? 祽? 禒? 秴? 穱? 竈 ? 竌 ? 竍 ? 竘 ? 筣 ? 筩? 箒? 篎? 篐? 篒? 篹? 簭? 簹? 籰? 粫? 糏? 糭? 絣? 絿? 綄? 綖? 綛? 緷? 缼 ? 翆!? 翜"? 耟#? 耣$? 肨%? 胒&? 胟'? 脴(? 腀)? 膙*? 膲+? 臎,? 臙-? 舋.? 艭/? 苮0? 茤1? 菒2? 菨3? 菺4? 萕5? 萭6? 蒵7? 蓈8? 蓕9? 蕌:? 蕜;? 蕟? 蕼?? 薭@? 藒A? 蘹B? 虣C? 蚆D? 蚒E? 蛃F? 蛠G? 蛣H? 蜼I? 蝄J? 蝏K? 蝖L? 蝛M? 螥N? 螮O? 螼P? 螾Q? 蟁R? 蟘S? 蠏T? 蠯U? 衕V? 袔W? 裐X? 褉Y? 褑Z? 褽[? 褾\? 襧]? 襴^? 覫_? 觮`? 觹a? 訋b? 訡c? 訦d? 詥e? 誔f? 誫g? 誺h? 誻i? 諃j? 諚k? 譄l? 譊m? 譤n? 讅o? 谹p? 豀q? 豘r? 貐s? 貑t? 貜u? 貦v? 跕w? 踆x? 踖y? 蹫z? 躝{? 躰|? 躴}? 軁~? 軂? 軇? 軦? 軬? 輏? 輧? 輴? 轙? 逈? 逫? 遦? 邎? 邒? 郌? 郖? 鄨? 酇? 醆? 醰? 銊? 銘? 鋴? 錓? 鍈? 鍨? 鍫? 鎍? 鎷? 鏏? 鐁? 鐄? 鐎? 鐯? 鑏? 鑖? 閕? 閗? 陚? 隑? 隝? 霴? 霷? 靅? 靏? 靧? 韟? 頄? 頮? 顩? 颫? 颹? 飜? 餀? 餁? 餢? 餫? 饘? 饳? 馚? 馾? 駋? 騡? 騩? 騪? 骔? 髍? 髳? 髷? 髺? 髼? 鬤? 魧? 魲? 鮅? 鮏? 鮜? 鮺? 鯑? 鯜? 鯥? 鯳? 鰔? 鰙? 鰫? 鰬? 鱮? 鳹? 鴊? 鴶? 鵐? 鵽? 鶢? 鶷? 鶾? 鷊? 鷖? 鷞? 鷩? 鷭? 鷱? 鸒? 鹽? 黺? 鼒? 鼨? 鼵? 鼺? 齄? 齅? 齨? 齶? ꅉ? ꔛ? 촬? 틴? 핀? ? ? ? ? ? 🏖? 🐰? 🐱? 😫? 🧘? ĥ? Ʀ? ƺ? ͷ@ Ϙ@ ӷ@ Ԓ@ ؃@ ؅@ ؠ@ ٜ@ ڔ@ ڛ @ ܆ @ ܏ @ ݑ @ ݛ @ ݵ@ ޡ@ ߺ@ ࠣ@ ࠾@ ࡎ@ ࡭@ ࡳ@ ࡺ@ ࢔@ ࢟@ ࣢@ ৑@ ৘@ ৙@ ੉@ ੊@ ੐@ ੵ @ ੺!@ ઩"@ ૎#@ ૕$@ ૟%@ ૠ&@ ૴'@ ૷(@ ଳ)@ ୔*@ ୙+@ ୚,@ ୽-@ ஡.@ ஧/@ ௉0@ ௮1@ ఙ2@ ఺3@ ౡ4@ ౾5@ ಙ6@ ೉7@ ೱ8@ ഄ9@ ഺ:@ ൅;@ ൩<@ ൵=@ ෠>@ ෰?@ ෸@@ ໊A@ ဝB@ ၀C@ ፡D@ ḪE@ ⃘F@ ➄G@ ➘H@ 丬I@ 亃J@ 亗K@ 伇L@ 伖M@ 佭N@ 侾O@ 俧P@ 偊Q@ 傟R@ 傡S@ 僋T@ 僪U@ 儎V@ 儏W@ 儾X@ 冎Y@ 凣Z@ 剆[@ 勨\@ 勭]@ 匘^@ 厊_@ 叞`@ 吂a@ 呄b@ 呇c@ 哱d@ 哶e@ 啢f@ 喥g@ 喭h@ 嗈i@ 嗺j@ 嘦k@ 嘷l@ 嚉m@ 嚰n@ 囒o@ 圿p@ 坄q@ 坓r@ 垏s@ 埅t@ 埛u@ 堛v@ 塿w@ 墢x@ 墷y@ 壦z@ 奿{@ 娂|@ 婨}@ 婩~@ 媊@ 媑@ 媨@ 媫@ 嫬@ 嬇@ 嬎@ 孉@ 宧@ 宲@ 宺@ 寗@ 寷@ 屔@ 屗@ 屟@ 岪@ 峆@ 峼@ 崘@ 崫@ 崱@ 崼@ 嵟@ 嵥@ 嵪@ 嶈@ 嶩@ 巆@ 巟@ 帇@ 帟@ 幱@ 庽@ 廏@ 弞@ 彁@ 彏@ 徦@ 徾@ 怑@ 怚@ 恱@ 悁@ 惾@ 愓@ 愯@ 慗@ 憯@ 懖@ 懯@ 扏@ 扲@ 抁@ 挄@ 捈@ 捖@ 捪@ 掕@ 揂@ 揝@ 搮@ 搰@ 搱@ 搳@ 摲@ 撎@ 攊@ 敾@ 斆@ 斱@ 旔@ 晇@ 晑@ 暭@ 暰@ 曗@ 曧@ 曭@ 朇@ 朲@ 杶@ 枂@ 枖@ 栍@ 栦@ 桍@ 梂@ 梐@ 梮@ 椉@ 椸@ 楉@ 榏@ 榳@ 榽@ 槝@ 槥@ 槾@ 樤@ 橤@ 檈@ 檘@ 櫀@ 欉@ 欼@ 歨@ 歭@ 毤@ 毪@ 氄@ 氎@ 汄@ 沍@ 沰@ 淢@ 渆@ 渢@ 渿@ 湏@ 湐@ 湚@ 湤@ 溎@ 溞@ 漞@ 漦@ 漴@ 潊@ 潒A 潗A 潳A 澅A 澏A 澖A 澢A 濷A 瀀A 瀸 A 灲 A 烌 A 烍 A 烥 A 焣A 煭A 熿A 燍A 燓A 燳A 牏A 犗A 犣A 犳A 犵A 犾A 狉A 猆A 猏A 猒A 猳A 獩A 玈 A 瓼!A 瓽"A 甉#A 畽$A 痵%A 瘜&A 癕'A 皩(A 盵)A 盽*A 眍+A 睈,A 瞕-A 瞗.A 矟/A 砙0A 硽1A 磌2A 磭3A 祻4A 禓5A 禝6A 禞7A 禥8A 禬9A 秅:A 秮;A 稇A 稵?A 窂@A 窏AA 竒BA 竕CA 竮DA 竸EA 笎FA 笜GA 筂HA 筟IA 筫JA 箟KA 箰LA 簓MA 簘NA 粂OA 粷PA 糐QA 糮RA 絤SA 綶TA 緽UA 縔VA 繫WA 纅XA 缹YA 罊ZA 罛[A 羀\A 翨]A 耓^A 耤_A 聏`A 聦aA 聬bA 肀cA 肍dA 肞eA 肻fA 胾gA 脈hA 脗iA 脭jA 腟kA 腳lA 膆mA 膇nA 膋oA 艀pA 艂qA 艈rA 艗sA 苭tA 茋uA 茡vA 菃wA 菣xA 葌yA 葠zA 葨{A 蒀|A 蒕}A 蓰~A 蔃A 蔒A 蕏A 蕫A 薡A 藊A 藑A 藭A 蘉A 蘙A 蘛A 虍A 蚉A 蚗A 蜯A 蝐A 蝑A 蝱A 螦A 蟭A 蠒A 衴A 衸A 衺A 衼A 袯A 裞A 裦A 褄A 襱A 覊A 覤A 覶A 覻A 觓A 觨A 觾A 訅A 訆A 訠A 詑A 誾A 諘A 諬A 謈A 謋A 謞A 謤A 謥A 謵A 譶A 讉A 谾A 豑A 豓A 豙A 賏A 贁A 贆A 趐A 跇A 跲A 踑A 蹅A 蹸A 躂A 躙A 躠A 軕A 軖A 軯A 軵A 輍A 轁A 迶A 逧A 遰A 邌A 邞A 郠A 鄅A 鄇A 醏A 醷A 鈗A 鈩A 鈼A 鉒A 鉝A 銍A 銴A 銺A 錔A 錼A 鍞A 鍧A 鍻A 鎑A 鎞A 鎥A 鎮A 鏥A 鏳A 鏿A 鑂A 鑆A 鑓A 鑕A 鑢A 钀A 钑A 镺A 閺A 陃A 陗A 陫A 陯A 陿A 隇A 霃A 霒A 霛A 霟A 靆A 靗A 靤A 靭A 鞂A 鞰A 韖B 韥B 頙B 顅B 顤B 颰B 飰B 餆B 餕B 餯 B 饁 B 饇 B 馡 B 馪 B 馼B 駀B 駤B 駥B 駨B 騝B 騴B 驝B 驲B 骮B 髛B 髲B 鬥B 鬭B 鬺B 魦B 鮕B 鮖B 鮘 B 鮧!B 鮩"B 鯏#B 鯝$B 鯹%B 鰖&B 鰜'B 鰯(B 鱋)B 鳴*B 鳸+B 鵃,B 鵗-B 鵭.B 鵹/B 鶔0B 鶭1B 鷁2B 鷌3B 鷎4B 鷑5B 鷛6B 鷽7B 鹴8B 麔9B 麙:B 黗;B 鼥B 齤?B 齭@B 쁘AB BB CB DB 🏏EB 🏘FB 🔮GB 😌HB 🤨IB 🦊JB 🦠KB ĈLB છMB ၊NB ᄉOB ዽPB ᑎQB ḇRB ἲSB ⇛TB ⌔UB ✻VB 㳇WB 伨XB 伹YB 佁ZB 偀[B 偂\B 偋]B 偛^B 傪_B 僁`B 僠aB 儝bB 兯cB 冓dB 冟eB 凂fB 刄gB 刟hB 剒iB 剭jB 剻kB 勂lB 勊mB 勯nB 匊oB 匒pB 卙qB 厹rB 叴sB 咺tB 哷uB 唥vB 唴wB 啎xB 啺yB 喗zB 喯{B 喴|B 嗊}B 嗋~B 噳B 嚍B 嚫B 囶B 圦B 圶B 垐B 垖B 埀B 埨B 埮B 埳B 埶B 壗B 壪B 夎B 奻B 姢B 娖B 媟B 媡B 媭B 嫎B 嫓B 嬣B 孍B 孨B 宯B 岮B 岰B 崜B 嵠B 嵧B 嶏B 嶬B 嶵B 巊B 巚B 帉B 帪B 幎B 庻B 廤B 弫B 弲B 恟B 惖B 惥B 愊B 愢B 愭B 愶B 慺B 懢B 懫B 懰B 戅B 戙B 戝B 戞B 戭B 抋B 抙B 拚B 拹B 挭B 掑B 掔B 揱B 搉B 摋B 摍B 摼B 擃B 擕B 攍B 攟B 敀B 斈B 斿B 昋B 昖B 晆B 晘B 晬B 暷B 曢B 朚B 朞B 杁B 杊B 杋B 杹B 枟B 柀B 栜B 梋B 梎B 梩B 棷B 棿B 楍B 楰B 楲B 楺B 楾B 榙B 榝B 榞B 榟B 榾B 槰B 樭B 樴B 橉B 橩B 櫡B 欜B 欪B 歒B 歞B 歵B 殶B 氊B 氱B 汷B 泿B 洓B 洜B 浕C 浶C 浻C 涍C 湈C 湹C 滜C 潶C 澒C 濹 C 瀄 C 瀇 C 瀗 C 灊 C 灳C 烕C 焧C 焪C 焽C 焾C 熀C 熓C 熴C 熶C 燑C 燩C 爊C 爥C 爮C 牆C 犌C 犓C 犚 C 猅!C 猌"C 猵#C 獇$C 獟%C 瓡&C 瓳'C 疈(C 疞)C 疷*C 瘮+C 瘻,C 癐-C 癳.C 皵/C 眗0C 眮1C 睖2C 睙3C 睻4C 矡5C 矲6C 矵7C 砞8C 硢9C 硳:C 硸;C 磛C 礠?C 祄@C 祑AC 禙BC 秂CC 秚DC 秿EC 稶FC 稸GC 稺HC 穦IC 穨JC 窇KC 竰LC 竷MC 笢NC 笿OC 筎PC 箽QC 篅RC 篬SC 簊TC 簎UC 籆VC 粁WC 粖XC 粶YC 糤ZC 糥[C 糷\C 紎]C 紨^C 絉_C 絗`C 絟aC 絭bC 絸cC 綕dC 緕eC 緪fC 縌gC 縘hC 縤iC 繨jC 纄kC 纋lC 纒mC 纼nC 绤oC 翓pC 聁qC 聐rC 聢sC 聨tC 脀uC 脋vC 脜wC 脝xC 脤yC 腇zC 腛{C 腯|C 膫}C 膬~C 膱C 膶C 膸C 臗C 臱C 舏C 舚C 艩C 芚C 芛C 苺C 荌C 菕C 菷C 萔C 萟C 葃C 葏C 葿C 蒖C 蒫C 蓌C 蓳C 蔍C 蔎C 蔨C 蔱C 蕭C 薒C 薚C 藢C 藬C 蘶C 虄C 虉C 蚫C 蛕C 蛘C 蛢C 蜋C 蝆C 蝫C 螙C 螜C 蟕C 蠫C 蠿C 袌C 袝C 袬C 裄C 裗C 褜C 褹C 襅C 襇C 襥C 覂C 覢C 覣C 觟C 觤C 訔C 訨C 訲C 誎C 誳C 誸C 諁C 諹C 諻C 謌C 譅C 譣C 譨C 讙C 豠C 貟C 貾C 賉C 賥C 贉C 贚C 赗C 趂C 趜C 趩C 踄C 踻C 蹏C 軪C 輂C 輑C 輤C 輫C 轕C 轠C 轷C 辤C 遱C 遳C 邚C 邤C 邧C 鄑C 酠C 醠C 釒C 釞C 釪C 鈋C 鉘C 鉟C 鉫C 鋍C 鋎C 鍁C 鍐C 鍑C 鍩C 鍮C 鍷C 鎙C 鎴C 鎺C 鏓C 鏠C 鏦C 鐝C 鑈D 鑙D 鑟D 閯D 閸D 陒D 隀D 隲D 雽D 霬 D 鞙 D 鞢 D 韨 D 颻 D 飇D 飬D 餗D 餪D 餭D 餰D 餶D 駈D 駍D 騂D 騽D 驦D 驧D 髨D 鬎D 鬳D 魒D 魗D 魙 D 魜!D 魣"D 魪#D 鮊$D 鮝%D 鯃&D 鯆'D 鰅(D 鰠)D 鰦*D 鰴+D 鱆,D 鱞-D 鱢.D 鲓/D 鴖0D 鴗1D 鴴2D 鵣3D 鵨4D 鶓5D 鶕6D 鷈7D 鷮8D 鸁9D 鸄:D 鸇;D 鸔D 鼛?D 齀@D 齈AD 齎BD 뛰CD 헌DD ED FD GD HD 🆚ID 🎞JD 💿KD 📘LD 😴MD ՑND ގOD ષPD ૂQD ጉRD ᥱSD ḻTD ὕUD ὢVD ⊢WD ⛈XD ⸚YD 〣ZD 仦[D 伌\D 伓]D 佖^D 侐_D 俲`D 倇aD 倹bD 偙cD 偩dD 傄eD 儜fD 儩gD 兾hD 凬iD 凱jD 凷kD 刜lD 剈mD 劕nD 劷oD 匓pD 匲qD 卥rD 卪sD 厀tD 厗uD 呚vD 呠wD 咰xD 哛yD 唗zD 啒{D 啹|D 啽}D 喅~D 噮D 嚻D 囼D 圔D 圵D 圽D 坧D 坿D 垼D 埩D 堫D 堬D 塨D 塪D 墏D 壝D 壭D 夘D 姂D 姴D 婃D 媰D 嬊D 嬱D 嬹D 嬽D 寈D 屆D 屽D 岅D 峅D 峌D 峑D 峾D 崲D 崵D 嵓D 庍D 庲D 彋D 彑D 彛D 忂D 忨D 忰D 怉D 怘D 怾D 恄D 恅D 恑D 悀D 悏D 悓D 慁D 慛D 慦D 慴D 憁D 憗D 憳D 憿D 懏D 懚D 懛D 懪D 戫D 挃D 挅D 挗D 挬D 捚D 捬D 捳D 捴D 揁D 揥D 揬D 揯D 搚D 擟D 攁D 攗D 攳D 敩D 敳D 敺D 敽D 斊D 斳D 旀D 旲D 暆D 曀D 曅D 曥D 曶D 朜D 枏D 枑D 柤D 栥D 桒D 棝D 楁D 榢D 槒D 槢D 樍D 樰D 樷D 橗D 橺D 檲D 檴D 櫅D 櫤D 櫭D 櫲D 櫼D 欄D 欮D 歁D 歈D 歑D 歾D 毜D 毢D 毩D 氒E 氠E 汢E 浌E 浝E 浫E 涏E 涗E 涰E 淎 E 淔 E 淾 E 渁 E 渄 E 湀E 湇E 湕E 湭E 溌E 溿E 滊E 滶E 潠E 澟E 瀢E 瀱E 烒E 烖E 烡E 烵E 焅E 煍E 熉 E 熑!E 燐"E 燗#E 爘$E 爫%E 牑&E 牨'E 犝(E 猂)E 猍*E 猟+E 猣,E 猰-E 獝.E 獞/E 瓥0E 甀1E 疓2E 痆3E 痋4E 痜5E 痬6E 癠7E 皀8E 皏9E 皢:E 皷;E 皼E 眝?E 睄@E 睭AE 睸BE 瞈CE 瞛DE 矌EE 矒FE 砏GE 硧HE 碩IE 磄JE 磆KE 磪LE 磮ME 礂NE 祡OE 秨PE 秷QE 穘RE 穥SE 窲TE 竧UE 笗VE 筰WE 箙XE 篂YE 篰ZE 簐[E 簚\E 簥]E 簲^E 籔_E 籚`E 粈aE 粣bE 紁cE 紌dE 紑eE 紶fE 絋gE 絓hE 綂iE 綤jE 緅kE 縅lE 纀mE 纃nE 纕oE 罎pE 羠qE 羦rE 羫sE 翋tE 翢uE 耾vE 聅wE 肙xE 胏yE 胑zE 胣{E 胮|E 胿}E 脡~E 脻E 腬E 膢E 臋E 臫E 舝E 艔E 茐E 荴E 莇E 莍E 菎E 萶E 葝E 蓘E 蔏E 蔯E 蔶E 蕂E 蕱E 藳E 蘔E 虠E 蚐E 蚙E 蚞E 蛓E 蛚E 蛜E 蜐E 蝋E 蝧E 螡E 蟇E 蟍E 蟷E 蟸E 蟼E 蠠E 蠽E 蠾E 袩E 袳E 袶E 裍E 褣E 襆E 襏E 襶E 襽E 覰E 覹E 觰E 觷E 詊E 誈E 諔E 謃E 謉E 謑E 謯E 豽E 貁E 賯E 贙E 趆E 趇E 趒E 趥E 趪E 趹E 跜E 踶E 躤E 躨E 躳E 軩E 輱E 轐E 轣E 迼E 逭E 逽E 遫E 郈E 鄡E 鄾E 醈E 醳E 釚E 鈛E 鉵E 銏E 鋠E 錣E 錪E 鍊E 鍏E 鎇E 鎎E 鎤E 鏂E 鏛E 鐛E 鐩E 鐬E 鐰E 鑎E 鑜E 鑥E 閙E 閮E 闒E 闗E 闠E 闦E 阫E 阺E 霣E 鞌E 鞓E 鞝E 韢E 韰E 頑E 頓E 顀E 颒E 飥E 飦F 餦F 餩F 饂F 饢F 馶F 馻F 駎F 駣F 駴 F 騠 F 驞 F 驠 F 骪 F 髐F 鬌F 鬑F 鬦F 魩F 魱F 鮥F 鯦F 鯶F 鰚F 鰢F 鱍F 鱚F 鱽F 鳦F 鴩F 鶅F 鶟F 鶤 F 鷏!F 鸋"F 麚#F 麠$F 黂%F 齓&F 齞'F 齩(F 龤)F 값*F 닐+F 똑,F 뢰-F 멋.F 엑/F 칼0F 켓1F 흔2F 3F 4F ￳5F 🇽6F 🌼7F 🥁8F 🦇9F ֆ:F ހ;F ൾF ᥬ?F Ệ@F ỡAF ἌBF ↕CF ⚛DF 乤EF 亄FF 伔GF 伡HF 侀IF 俇JF 俔KF 俠LF 俴MF 俼NF 倱OF 倿PF 偮QF 傓RF 儹SF 兞TF 凚UF 剼VF 劚WF 劧XF 勓YF 匇ZF 匶[F 匷\F 厑]F 厧^F 厫_F 厯`F 呍aF 呥bF 唕cF 唙dF 唜eF 啋fF 啘gF 喞hF 喢iF 嗂jF 噿kF 嚗lF 圎mF 坸nF 垰oF 埄pF 堣qF 塛rF 塳sF 墱tF 夒uF 妅vF 妛wF 嫛xF 孡yF 宩zF 寙{F 尃|F 尌}F 尦~F 岨F 岾F 崻F 嶑F 巶F 巺F 幊F 幓F 幠F 幩F 庌F 庱F 弌F 弙F 弝F 彍F 徔F 忚F 忢F 忲F 惎F 愸F 慙F 憰F 憻F 懙F 懜F 懠F 扷F 抍F 抶F 拻F 挀F 挰F 挳F 撀F 撶F 撽F 擥F 攚F 攺F 斦F 旍F 旞F 昅F 晎F 暛F 柇F 柶F 柼F 桪F 梌F 梕F 梙F 梞F 梡F 梻F 棸F 椗F 椫F 椺F 楆F 楏F 榋F 榣F 榰F 樇F 樧F 橳F 檍F 櫇F 櫵F 欂F 欯F 歝F 毟F 毭F 氉F 氋F 氻F 汣F 沊F 沜F 洬F 浂F 浖F 涖F 渱F 渷F 渻F 湢F 湪F 滛F 漅F 漵F 澃F 澻F 瀎F 瀤F 灁F 灔F 灱F 炢F 煗F 煘F 熢F 熮F 燇F 爃F 爅F 爞F 牣F 犙F 犞F 狜F 獂F 獊F 獌F 獔F 獘F 瓝F 甝F 畡F 畱F 疀F 疘F 痚F 皥F 皬F 瞂G 瞦G 矁G 矘G 矦G 砤G 硛G 碞G 磓G 磖 G 磸 G 礆 G 礗 G 礹 G 禉G 禗G 稘G 穙G 穚G 穞G 穬G 窱G 竉G 竛G 竡G 笧G 笶G 筗G 筿G 箂G 箈G 箲G 箹 G 篵!G 簙"G 簤#G 籸$G 粚%G 糛&G 絔'G 絠(G 絻)G 緛*G 縬+G 繣,G 纗-G 罓.G 羏/G 羳0G 翺1G 耊2G 聎3G 聑4G 肕5G 肣6G 胇7G 胋8G 胢9G 脕:G 脙;G 膟G 芁?G 茮@G 荕AG 莑BG 莵CG 葍DG 蒆EG 蒑FG 蓗GG 蔮HG 蕦IG 薎JG 藇KG 蘝LG 虇MG 虯NG 虸OG 蛷PG 蜖QG 蜝RG 蝝SG 蝺TG 螰UG 螴VG 蟎WG 蠌XG 袕YG 袡ZG 裓[G 裿\G 褌]G 褼^G 襛_G 覑`G 覞aG 觃bG 觻cG 訉dG 訒eG 訵fG 詄gG 詯hG 詻iG 諨jG 謏kG 謶lG 貆mG 貖nG 賦oG 賩pG 賭qG 贌rG 跅sG 跧tG 跴uG 蹗vG 躎wG 躖xG 躢yG 躵zG 躶{G 躸|G 軙}G 輡~G 輼G 辴G 迱G 逷G 遟G 邍G 邷G 郉G 郶G 鄓G 鄛G 鄻G 醕G 醼G 釂G 釠G 釰G 釻G 鈤G 鈨G 鉇G 鉎G 鉓G 鉪G 銿G 鋛G 鋞G 鋲G 錎G 錷G 鎕G 鎟G 鎠G 鎯G 鏀G 鏄G 鏆G 鏱G 鐍G 鑳G 锧G 镮G 閁G 閈G 閍G 闄G 阬G 雐G 雚G 雡G 雥G 雧G 靷G 鞀G 鞁G 鞟G 鞹G 鞼G 頋G 頺G 顜G 顣G 颵G 飉G 飤G 餝G 饙G 駲G 駺G 駻G 騐G 騥G 驌G 驖G 髩G 魞G 鮽G 鯂G 鰑G 鰮G 鱌G 鱜G 鱡G 鱴G 鳨G 鴧G 鴹G 鴼G 鴾G 鵆G 鵥G 鵳G 鶣G 鶽G 鷜G 鸖G 鹐G 麮G 麲G 黁G 黣G 鼁G 鼃G 鼈G 龎G 龝G ꄉG 맨G 삭G 짓G G G G 🍅G 🐘G 📒G 🥊G 🥮G ୩G ణG ಚG ഭG ഷG ഹG ༺G ဂG ሲG ዎG ᑕG ᒃH ᕗH ᠫH ỄH ◗H 乴H 乼H 亝H 仛H 伂 H 佋 H 侟 H 侻 H 俥 H 倊H 倳H 僴H 兪H 凕H 剱H 勜H 厈H 厸H 吺H 呏H 哻H 唒H 唽H 喠H 嗕H 嗗H 嘄H 噾 H 嚲!H 囦"H 坃#H 坒$H 坘%H 坺&H 垜'H 堜(H 堥)H 堶*H 堸+H 堾,H 塰-H 壨.H 夞/H 奜0H 妦1H 姃2H 娏3H 娫4H 媆5H 嫀6H 寉7H 尣8H 岟9H 峍:H 峖;H 峜H 嶫?H 帞@H 弬AH 弻BH 忣CH 怽DH 恆EH 悎FH 悐GH 惍HH 愝IH 愡JH 慐KH 憃LH 憕MH 憵NH 懻OH 戵PH 抈QH 抪RH 捼SH 掓TH 掗UH 揔VH 揤WH 摃XH 摨YH 撍ZH 擏[H 擙\H 攡]H 斚^H 斴_H 旤`H 晫aH 暒bH 曃cH 曍dH 朰eH 杫fH 枅gH 枛hH 枩iH 柋jH 栬kH 桳lH 棥mH 椈nH 椬oH 楧pH 榵qH 榸rH 槵sH 樿tH 橶uH 橾vH 檅wH 檒xH 檿yH 櫋zH 櫗{H 櫠|H 櫽}H 欎~H 歕H 殾H 毲H 毶H 氃H 氭H 汒H 汦H 泎H 泑H 泭H 洆H 洍H 洘H 浀H 浱H 涄H 涼H 溰H 滵H 漊H 漥H 潌H 潫H 潹H 澲H 濖H 濭H 濲H 瀈H 瀌H 瀥H 瀶H 灂H 灇H 灙H 灥H 炶H 烑H 煆H 煰H 煶H 煼H 熕H 燆H 燌H 燝H 燢H 燨H 燰H 燵H 爗H 爜H 猲H 獓H 獧H 獶H 瓲H 瓺H 瓾H 畒H 畣H 疅H 痗H 痮H 瘶H 癚H 皳H 皹H 盿H 眐H 眲H 眹H 瞖H 瞱H 矂H 矈H 矠H 砊H 砐H 砶H 硉H 硰H 碅H 磰H 禃H 稉H 稌H 稫H 稴H 穔H 穯H 窤H 窵H 竀H 竃H 笅H 笐H 箌H 篊H 簯H 粡H 糀H 糓H 糔H 糪H 糳H 紝H 絈H 絩H 緥H 緳H 縆H 縺H 繮H 繿H 罆H 罯H 羍H 羭H 翉H 耎H 耞H 耭H 聈H 聧H 聸H 脟H 腜H 腤I 臐I 臔I 臸I 艍I 苆I 苚I 茰I 莄I 莐 I 萐 I 萙 I 萡 I 葟 I 蓻I 蔐I 蔜I 蔤I 蔪I 蕠I 薉I 藖I 藾I 蘃I 蘒I 虭I 蚚I 蚸I 蚽I 蛵I 蜙I 蜤I 蜰 I 蜹!I 蝭"I 螆#I 螉$I 螒%I 螖&I 螘'I 蟨(I 蠦)I 蠩*I 衋+I 衳,I 袎-I 袦.I 袻/I 裠0I 裩1I 褖2I 褬3I 褮4I 襨5I 覜6I 覭7I 覼8I 覾9I 訟:I 訬;I 詀I 誀?I 誵@I 諿AI 謜BI 謽CI 譂DI 讇EI 讍FI 讘GI 豛HI 豤II 豾JI 貀KI 貇LI 貗MI 賋NI 趀OI 趗PI 趛QI 跦RI 踃SI 躟TI 躣UI 軘VI 軲WI 轞XI 辝YI 迌ZI 郔[I 鄎\I 鄷]I 酼^I 醓_I 釼`I 鉐aI 鉯bI 鋢cI 錂dI 錗eI 錥fI 錻gI 鎅hI 鎆iI 鎨jI 鎹kI 鏁lI 鏴mI 鑀nI 鑡oI 鑩pI 镾qI 閖rI 閵sI 闎tI 闛uI 闧vI 陻wI 隵xI 雬yI 霗zI 霢{I 霯|I 霳}I 靾~I 鞃I 韈I 韊I 韒I 韴I 韷I 頯I 顁I 顄I 顊I 飺I 餜I 餻I 駜I 驐I 驜I 骬I 骲I 骿I 鬗I 魤I 魺I 鮂I 鮌I 鯚I 鯬I 鯾I 鰻I 鱩I 鱰I 鳾I 鴐I 鵉I 鵘I 鵚I 鵦I 鶂I 鶡I 鶦I 鷅I 鷋I 鷐I 鷕I 鷝I 鷻I 麳I 黶I 鼀I 鼑I 龒I ꒱I 섭I 튜I I I I I 🏕I 💅I 💼I 🟨I 🤟I 🥘I 🧑I ĬI ՂI ՓI ։I ފI ঢI ਈI ਡI ૢI ଟI උI ᓐI ᚢI ᶍI ỊI ₺I ╳I ▦I ❿I ⩾I ⲏI ㄩI 丩I 乬I 乹I 亴I 佅I 俉I 倎I 偔I 傃I 傤I 傰I 傶I 儃I 儠I 儬I 凁I 剗I 劅I 卛I 厒I 厤I 厬I 叢I 呞I 呩I 哸I 唡I 唨I 啔I 嘗I 噄I 囮I 囻I 圤I 坾I 垑I 垪I 垬I 埐I 墂I 墯I 夁I 奞I 奱J 娧J 嫆J 嫍J 嫙J 嫺J 嬆J 嬞J 孻J 寍 J 尫 J 尭 J 尮 J 峓 J 崓J 嶦J 嶪J 嶮J 帴J 幍J 幧J 廟J 廥J 弖J 彮J 彴J 彸J 徲J 忥J 忶J 忼J 恞J 惂 J 惃!J 愂"J 愄#J 愌$J 慤%J 慩&J 憉'J 懧(J 戂)J 戠*J 扟+J 扻,J 抇-J 拞.J 捠/J 捦0J 揮1J 揷2J 斻3J 斾4J 旚5J 晭6J 晵7J 晹8J 暊9J 枮:J 柦;J 栠J 椩?J 楐@J 楘AJ 榒BJ 榹CJ 榺DJ 槀EJ 槖FJ 槷GJ 樮HJ 橠IJ 橯JJ 櫁KJ 櫄LJ 櫏MJ 櫫NJ 欱OJ 欳PJ 歫QJ 殝RJ 毈SJ 汳TJ 沞UJ 洕VJ 洠WJ 洯XJ 浿YJ 涃ZJ 涊[J 溛\J 溮]J 滭^J 滰_J 漌`J 潨aJ 濇bJ 瀃cJ 瀏dJ 瀽eJ 炠fJ 炩gJ 烐hJ 煟iJ 熎jJ 熧kJ 熩lJ 燛mJ 爌nJ 爎oJ 爧pJ 牎qJ 牚rJ 牶sJ 牼tJ 犫uJ 猽vJ 獀wJ 獑xJ 瓵yJ 畄zJ 疪{J 痭|J 瘑}J 瘒~J 瘬J 瘷J 癗J 皅J 眂J 眃J 眅J 眤J 睟J 睲J 睶J 瞔J 瞸J 砨J 硩J 硺J 碠J 磃J 礘J 礚J 礣J 礨J 祘J 祵J 秗J 稓J 穓J 穼J 窹J 竬J 竼J 篏J 篖J 篴J 簝J 籈J 籉J 籋J 籑J 籨J 糆J 絊J 絝J 絫J 絰J 緁J 緰J 縃J 纝J 罸J 翍J 翗J 翪J 耯J 聉J 肐J 肦J 肬J 脦J 脽J 膡J 臈J 臦J 艒J 艸J 艻J 芀J 苩J 荲J 莗J 菙J 菚J 萉J 萛J 萪J 葕J 葘J 葧J 葴J 葾J 蓙J 蓡J 蓹J 蕵J 薂J 薃J 薴J 藀J 藄J 藫J 藽J 蘦J 蘰J 虘J 蚦J 蚲J 蛗J 蛡J 蜔J 蜦J 蜧J 蜳J 蜶J 螝J 螿J 蠤J 蠮J 衁J 袐J 裫J 襋J 襓J 襣J 襸J 覉J 訰J 詏J 詠J 誏J 諩J 謴J 譍J 譼J 讈J 豏J 豩J 豱J 貏J 貕J 貢K 貫K 賶K 趈K 跈K 蹛K 躗K 躘K 軝K 軧 K 輐 K 輚 K 轖 K 逇 K 逳K 邼K 郆K 郮K 醖K 醟K 釴K 釽K 鈖K 鈲K 鉃K 鉖K 鉳K 銡K 銽K 銾K 鋿K 錑K 錰 K 鍣!K 鍭"K 鎄#K 鎾$K 鏅%K 鏲&K 鐀'K 鐣(K 鑇)K 鑻*K 閇+K 阧,K 隦-K 雵.K 霠/K 鞐0K 鞒1K 鞜2K 鞤3K 鞱4K 韏5K 韣6K 韹7K 頱8K 顂9K 颷:K 餇;K 饆K 騙?K 驆@K 驓AK 骳BK 鬁CK 鬅DK 鬔EK 鬕FK 魖GK 魢HK 鮻IK 鯩JK 鯺KK 鰀LK 鰧MK 鱏NK 鱦OK 鱫PK 鴁QK 鴘RK 鴤SK 鵤TK 鵷UK 鶈VK 鶑WK 鶨XK 鷪YK 鷫ZK 鷼[K 鸤\K 鹸]K 麉^K 麊_K 黭`K 黰aK 鼖bK 鼳cK 齆dK ꒰eK 뀐fK 냐gK 늦hK 웅iK 웠jK 찍kK 킬lK mK nK oK pK ︃qK ︊rK 💭sK 🔝tK 🙊uK 🥓vK 🧺wK ބxK ߦyK ಧzK ആ{K ႏ|K ሃ}K ሮ~K ᚨK ↷K ⇘K ⇥K ⌃K ⎟K ╲K ▉K ☷K ⻩K ㄠK ㄤK 䶮K 乊K 乣K 乥K 伄K 侌K 侳K 俽K 倐K 倯K 偯K 偹K 傛K 僐K 僷K 儦K 兙K 凾K 刧K 劘K 劶K 勬K 匢K 匧K 厁K 厏K 唊K 唖K 喕K 嘝K 噷K 囐K 圛K 垎K 垶K 埣K 塖K 夃K 夝K 妉K 姏K 姲K 嬔K 孠K 宑K 寭K 尡K 尲K 岤K 岥K 岶K 嵔K 嵨K 嵭K 嶜K 嶻K 帢K 幙K 幥K 庘K 庬K 廎K 弤K 弿K 彚K 彶K 忈K 怲K 恊K 恔K 悥K 悹K 惌K 懘K 戄K 扖K 挋K 揙K 搫K 摮K 撉K 擪K 擽K 攱K 敜K 敡K 斔K 斪K 旊K 昑K 昦K 朁K 朾K 杸K 柛K 柪K 桺K 棈K 棏K 椕K 榶K 槬K 槮K 槴K 樌K 橕K 橣K 橬K 橵K 檆K 櫎K 櫑K 櫒K 櫷K 欋K 欿K 殙K 毇L 毨L 浘L 浳L 涹L 淴L 湁L 湒L 潈L 澺 L 濕 L 濤 L 濽 L 灚 L 炥L 烢L 焩L 熁L 熐L 犜L 犻L 狢L 瓹L 甤L 畆L 痀L 痟L 皃L 皟L 盇L 眡L 眻L 矺 L 砆!L 硦"L 碋#L 磒$L 磼%L 礖&L 礥'L 祬(L 禲)L 稰*L 穭+L 窔,L 笚-L 笴.L 筃/L 籫0L 籯1L 絑2L 綗3L 綳4L 綷5L 緵6L 縓7L 縜8L 繀9L 罁:L 罻;L 羇L 翷?L 聄@L 聛AL 聣BL 胉CL 脎DL 脠EL 腁FL 腵GL 膎HL 膮IL 膷JL 臹KL 舙LL 艪ML 芺NL 茊OL 茦PL 荱QL 萒RL 萓SL 萷TL 蓎UL 蓵VL 蕀WL 蕄XL 藞YL 虲ZL 蚅[L 蛪\L 蜟]L 蜪^L 螕_L 蟂`L 蟏aL 蠥bL 蠴cL 衂dL 袏eL 袽fL 裃gL 褕hL 褤iL 褱jL 襊kL 襗lL 覗mL 覱nL 覴oL 觛pL 訙qL 詗rL 諊sL 謆tL 謷uL 謺vL 譈wL 譺xL 讄yL 讔zL 豒{L 豟|L 豶}L 貵~L 賆L 賝L 賫L 賰L 賲L 趎L 跔L 踋L 蹃L 軎L 軥L 輀L 轋L 辡L 辳L 迒L 逜L 郀L 郩L 鄍L 醀L 釯L 鈻L 鉙L 鉡L 銂L 銕L 鋉L 錖L 錴L 鍋L 鏾L 鐟L 鐹L 鑚L 閊L 闟L 阸L 隭L 霌L 霔L 鞉L 鞖L 鞛L 韺L 顇L 顋L 顬L 飁L 饝L 馲L 駏L 騈L 騛L 騞L 騻L 騿L 骭L 髤L 魀L 魥L 魹L 鮓L 鮲L 鱨L 鴭L 鵏L 鵸L 鶁L 鶴L 鶶L 鷇L 鸈L 麢L 麬L 黱L 鼘L 鼦L 鼿L ꀿL 갔L 괴L 룸L 빌L 촌L 혔L L L L L 🍰L 🎗L 🎹L 🐉L 🐟L 🕷L 😠L 🚧L 􏰆L ŲL ǯL ކL ਉL ೂL ↶L ⌁L ⌂L ⍢L ☍L ☭L ☲L ✾L ➗L ⤴L 丒L 乶L 乷L 乻L 乿L 仜L 佔L 侤L 倠L 偵L 傜L 傹L 傾L 僢L 僬L 兤M 冹M 凘M 剎M 剠M 劎M 劺M 勩M 匩M 厃 M 厞 M 唟 M 喡 M 嗏 M 嚖M 嚽M 坈M 坴M 垷M 埞M 堒M 塜M 塮M 墆M 壋M 妎M 嬼M 寋M 寱M 屷M 屻M 峊M 峚 M 峿!M 嵏"M 嶃#M 嶴$M 嶾%M 帄&M 幦'M 弣(M 怰)M 怷*M 恎+M 悋,M 惏-M 愪.M 慖/M 慠0M 懤1M 抮2M 拲3M 挱4M 捑5M 掜6M 揇7M 搃8M 搎9M 搼:M 摓;M 摕M 攠?M 斍@M 昗AM 晲BM 暺CM 曣DM 栕EM 栰FM 桰GM 桸HM 棦IM 棾JM 椧KM 楄LM 榐MM 榑NM 槂OM 樄PM 橃QM 檋RM 櫜SM 欕TM 歚UM 殅VM 殔WM 殥XM 毱YM 汱ZM 汼[M 沀\M 洔]M 涁^M 淧_M 渒`M 滼aM 漗bM 潚cM 濥dM 濸eM 瀖fM 炃gM 炌hM 烪iM 焑jM 焹kM 煝lM 熆mM 熇nM 燘oM 爤pM 牅qM 牉rM 犱sM 狣tM 猠uM 獛vM 瓫wM 甒xM 甖yM 痌zM 睮{M 瞃|M 瞊}M 矝~M 硥M 硹M 碄M 碖M 碮M 磥M 礇M 禈M 禪M 秢M 稕M 穖M 窡M 窼M 筁M 箉M 箛M 篕M 篠M 簬M 籗M 籡M 籿M 粠M 粯M 粵M 糂M 糣M 紽M 絖M 絘M 綐M 綩M 縼M 繐M 繓M 繵M 纑M 肎M 肔M 肧M 肳M 胦M 胲M 膞M 膠M 膥M 舃M 舓M 艓M 艜M 艵M 荶M 菗M 菦M 菶M 蒏M 蒦M 蒭M 蓞M 蔕M 蕛M 薐M 藋M 蘂M 蘍M 蘣M 蚃M 蚳M 蜛M 蜸M 螎M 蠉M 蠚M 衘M 衭M 袃M 袉M 袘M 褁M 褅M 襹M 覨M 訽M 諪M 諯M 謣M 譕M 譡M 譳M 讗M 谺M 豍M 豴M 貄M 貒M 貚M 貣M 赺M 趏M 跱M 蹳M 蹽M 蹾M 躌M 輵M 輽M 迻M 逘M 邆M 邥M 酀M 酜M 酧M 釲M 鈠M 銯M 銱M 鋂M 鋽M 鍢M 鎜M 鏕M 鏭M 鐶M 鑃M 鑮M 鑴M 鑸M 鑹N 镸N 镽N 閐N 閜N 閰N 隚N 雂N 雦N 雸 N 頉 N 頩 N 頳 N 飅 N 飊N 飹N 飻N 餄N 餹N 饏N 饻N 馛N 騇N 騣N 騼N 驈N 驨N 魛N 魬N 鮾N 鯕N 鯣N 鰁 N 鰇!N 鰌"N 鰏#N 鰺$N 鳵%N 鴔&N 鴪'N 鵂(N 鶀)N 鶛*N 鶞+N 鷀,N 鷠-N 鸧.N 鹷/N 黚0N 齛1N ꁧ2N 균3N 끄4N 쓸5N 잖6N 졸7N 퓨8N 9N :N ;N N 🍔?N 🍵@N 🎩AN 🏷BN 🤮CN 🥜DN 🥵EN ĢFN ĵGN ĸHN ɟIN ՌJN ܩKN ލLN ਇMN ఆNN ఎON ധPN ഫQN ഴRN චSN ኖTN ዕUN ួVN ⌠WN ▮XN ♖YN ♘ZN ✒[N ㄅ\N 仐]N 侸^N 俒_N 偅`N 傏aN 儨bN 儮cN 刉dN 刐eN 剶fN 勴gN 叏hN 呧iN 呭jN 啳kN 嗹lN 嚊mN 嚋nN 嚺oN 圅pN 坉qN 堚rN 塃sN 墤tN 墵uN 夗vN 婯wN 嬭xN 尳yN 岇zN 岼{N 嶘|N 嶯}N 巐~N 帤N 庎N 弡N 弰N 徚N 徰N 徶N 怋N 怢N 怭N 怮N 恛N 悙N 悞N 惷N 愋N 愒N 憆N 憖N 懗N 懬N 戃N 抦N 揊N 搙N 搷N 撨N 敆N 敋N 旇N 旕N 昁N 暟N 朆N 栨N 桏N 桗N 棭N 椘N 楖N 榯N 槄N 槆N 橁N 檂N 檺N 櫿N 欁N 欫N 歋N 毞N 毣N 毮N 汿N 泍N 洃N 洡N 浟N 涷N 渪N 湌N 潃N 澑N 濦N 灴N 焭N 煩N 燞N 燱N 爂N 爢N 牄N 猺N 甎N 疉N 瘽N 皌N 皽N 盋N 眔N 睷N 瞁N 矃N 磀N 磱N 礀N 禜N 禠N 穝N 穮N 笽N 筄N 箁N 篫N 簅N 簔N 籎N 粍N 粴N 紩N 綒N 綘N 緎N 緓N 緿N 縖N 縩N 縸N 繘N 繥N 羒N 聝N 肁N 脌N 腏N 膉N 膔N 臅N 臶N 艃N 艌N 菄N 萖N 萺N 葤N 蒒N 蒣N 蓩N 蕚N 藔O 藗O 藛O 虥O 蜄O 螐O 螹O 蠀O 蠰O 蠸 O 衠 O 袟 O 袧 O 袰 O 袸O 裵O 褟O 覙O 訤O 詇O 詓O 誯O 諣O 謕O 謲O 譠O 譧O 譻O 豅O 豝O 賱O 赲O 趡 O 趧!O 躼"O 軠#O 轏$O 辷%O 酙&O 釄'O 鉼(O 銢)O 鋖*O 鋩+O 錃,O 錭-O 鍖.O 鎲/O 鏒0O 鐷1O 鑁2O 闂3O 阢4O 阽5O 陎6O 霕7O 霱8O 霼9O 鞲:O 韚;O 頍O 顆?O 飀@O 飸AO 馠BO 駯CO 駶DO 駷EO 騀FO 騔GO 騦HO 鬙IO 鬠JO 鬴KO 魻LO 鮔MO 鮴NO 鯴OO 鱐PO 鱕QO 鳫RO 鳺SO 鴑TO 鴻UO 鵶VO 鷍WO 鷒XO 麫YO 鼮ZO 齃[O 齝\O 곤]O 낮^O 넓_O 맥`O 멈aO 첨bO 춰cO 폐dO eO fO gO hO iO 🅰jO 🍊kO 🎰lO 💝mO 📂nO 📦oO 📬pO 📻qO 🚚rO 🛈sO 🥝tO 🥢uO ǘvO ɓwO ʝxO ֔yO ײzO ٧{O ୟ|O ಣ}O എ~O ഒO ၾO ሊO ኤO ዜO ᶅO ẤO ẳO ╪O ▊O 㵲O 丠O 伅O 偨O 偳O 傠O 儤O 冔O 刏O 勀O 匰O 卆O 叿O 哴O 唘O 喒O 嗸O 坅O 埆O 堄O 堖O 奆O 嬩O 孯O 宊O 寪O 屰O 峕O 嵜O 嵡O 嶉O 庨O 徣O 怌O 怴O 恜O 悷O 惀O 惁O 惉O 憏O 懱O 撆O 擖O 敐O 敚O 敯O 敶O 斺O 晐O 朼O 枀O 柫O 棜O 棞O 楴O 樎O 樬O 檕O 櫌O 欙O 欛O 欝O 歶O 殈O 毻O 汓O 泴O 浽O 涜O 涺O 淣O 淿O 溝O 潱O 澾O 澿O 濚O 瀊O 灧O 烅O 焈O 焿O 爑O 犆O 犩O 猐O 猤O 獈O 瓪O 疨O 痐O 瘂O 瘃O 瘯O 皍O 皧O 盁O 盧O 眫O 睼O 瞴O 砎O 硣O 硱O 磩O 礔O 禐O 禯O 稦O 窽O 笓O 箃O 箶O 篶O 簨O 簺O 糃O 緂O 縨P 繍P 繎P 繛P 缿P 罧P 罬P 罿P 翴P 耡 P 腪 P 膍 P 舕 P 舭 P 芉P 茷P 荁P 莏P 萀P 葅P 葪P 蒝P 蒤P 蓤P 薞P 薶P 蘬P 虦P 蛁P 蛌P 蜵P 螱P 螶 P 蠈!P 蠜"P 衇#P 衦$P 袵%P 襐&P 覐'P 觢(P 訁)P 詋*P 諓+P 諙,P 譐-P 豻.P 貱/P 贈0P 趓1P 趘2P 踓3P 蹞4P 蹥5P 蹵6P 躿7P 軐8P 軴9P 辬:P 辸;P 迏

    P 邭?P 郬@P 鄟AP 酑BP 酨CP 酻DP 醦EP 醧FP 釖GP 鈓HP 鉲IP 鋑JP 錉KP 鍗LP 鎀MP 鎋NP 鏉OP 鐜PP 鐼QP 閚RP 閷SP 阓TP 陙UP 鞷VP 鞸WP 颭XP 餥YP 餧ZP 馜[P 馝\P 駳]P 骵^P 髗_P 髰`P 鬞aP 鮉bP 鮿cP 鯘dP 鱣eP 鱹fP 鳲gP 鴏hP 鴙iP 鵎jP 鵢kP 鶍lP 鶜mP 鶪nP 鷔oP 鷷pP 鷿qP 鹝rP 麁sP 黊tP 鼤uP 鼰vP ꀕwP ꑌxP ꙻyP 댓zP 멘{P 윗|P }P ~P P ︇P 🏒P 💧P 🔈P 🔷P 😵P 🥃P ȝP ȼP ˮP ԺP ՉP ׃P ٞP ٦P ફP ஏP ཻP ᆺP ቢP ះP ᵻP ἹP ⌨P ⍛P ␣P 乯P 伳P 侅P 侇P 俈P 俻P 倝P 凙P 凥P 凨P 刱P 刴P 劖P 厽P 坙P 垁P 塺P 墛P 壿P 夾P 嬘P 寁P 尵P 屓P 峹P 崺P 嵷P 嶡P 嶨P 嶭P 廀P 彂P 恇P 恗P 慸P 憘P 拁P 掋P 撲P 擶P 敮P 斖P 朄P 朠P 杢P 枽P 枾P 桋P 桞P 桭P 桻P 梺P 棟P 棤P 槗P 殏P 泀P 淵P 湗P 湽P 溔P 溣P 瀪P 灹P 烉P 煡P 煾P 熡P 爈P 爟P 爡P 犤P 狏P 狵P 甅P 瘔P 癴P 睂P 睔P 瞍P 瞙P 瞮P 瞶P 砕P 碐P 磎P 礋P 窙P 筡P 筬P 簵P 籄P 紷P 綀P 綔P 緐P 繬P 繱P 纖P 翐P 耉P 肗P 脺Q 膯Q 艡Q 茒Q 茙Q 茢Q 菆Q 蒓Q 蒘Q 藲 Q 蘥 Q 蘪 Q 蚢 Q 蛫 Q 蛶Q 襡Q 觠Q 觬Q 詂Q 誁Q 誛Q 諥Q 謍Q 谸Q 趨Q 踥Q 蹖Q 蹷Q 躩Q 軅Q 軉Q 逤Q 醊 Q 鈂!Q 鈫"Q 鉰#Q 銅$Q 銸%Q 鍀&Q 鍃'Q 鍸(Q 鎱)Q 鎸*Q 鑪+Q 钄,Q 闙-Q 陱.Q 隯/Q 雤0Q 雼1Q 靟2Q 鞽3Q 韎4Q 韗5Q 顃6Q 颲7Q 馞8Q 駌9Q 駚:Q 騌;Q 髵Q 鳰?Q 鴅@Q 鶃AQ 鶎BQ 鸎CQ 鼣DQ 걱EQ 껴FQ 꼬GQ 녁HQ 략IQ 룩JQ 쓴KQ 찌LQ MQ NQ OQ PQ QQ RQ ￱SQ 🍄TQ 🍑UQ 🏢VQ 🐕WQ 🔻XQ 🕓YQ 🗽ZQ 😹[Q 🤐\Q 🥴]Q 🧵^Q Ƭ_Q ȯ`Q ˒aQ ٨bQ ܰcQ ૧dQ ථeQ ᄃfQ ቤgQ ጵhQ ᑐiQ ᛏjQ ᠺkQ ẸlQ ⌇mQ ┓nQ ⸺oQ ㄓpQ 㮾qQ 丯rQ 僼sQ 儑tQ 償uQ 勈vQ 勮wQ 匯xQ 匵yQ 噝zQ 坰{Q 壉|Q 壧}Q 宱~Q 尟Q 峲Q 崰Q 嵞Q 廢Q 廽Q 彌Q 彥Q 徺Q 徻Q 忋Q 惈Q 懓Q 懹Q 戁Q 捀Q 撁Q 擜Q 暓Q 暬Q 曤Q 朩Q 棩Q 棳Q 椃Q 椲Q 椳Q 榌Q 槕Q 槸Q 樆Q 樠Q 湠Q 溂Q 滽Q 澊Q 澷Q 瀮Q 熻Q 爳Q 犑Q 犘Q 狊Q 狤Q 瘏Q 瘡Q 瞯Q 矖Q 礷Q 祍Q 祱Q 稛Q 竆Q 篛Q 篜Q 籅Q 糘Q 糡Q 糫Q 糼Q 綊Q 緫Q 罺Q 翤Q 耛Q 聥Q 胈Q 胔Q 膅Q 膐Q 臕Q 葀Q 蓴Q 虤Q 蛅Q 蜅Q 蝷Q 襰Q 襲Q 詙Q 詨Q 謀Q 譢Q 譸Q 讏Q 豿Q 赼Q 跉Q 蹠Q 軣Q 轀Q 轃Q 迧Q 遙Q 邔Q 酭Q 鈙Q 鋜Q 鐗Q 鑋Q 閠Q 阷Q 陏Q 雓Q 靃Q 韇Q 韯Q 頃Q 飃Q 馽Q 魐Q 鴄Q 鵫Q 鵿Q 鶆Q 鷧Q 鷬Q 鸍Q 麎Q 黋Q 럴Q 묵Q 샤Q 즉Q 즘Q 짧Q Q Q Q 🐴R 📕R 🔬R 😪R ʇR ʤR ԶR ީR ઓR ཽ R ቡ R ᐧ R ᕕ R ᚱ R ᚾR ⃞R ≳R ┳R ◒R ◻R ❇R ❕R ⲡR ㄕR 㖞R 丟R 併R 俿R 偘R 傆R 傝R 勆R 勵 R 喪!R 塵"R 塻#R 墍$R 墔%R 壔&R 壾'R 娛(R 孹)R 尗*R 岦+R 帍,R 幯-R 廮.R 怞/R 惸0R 憠1R 憱2R 搻3R 撦4R 擛5R 擺6R 杚7R 椨8R 槣9R 樶:R 橓;R 檌R 汮?R 溩@R 潥AR 燺BR 爙CR 牃DR 牊ER 牱FR 犎GR 狿HR 獋IR 瓧JR 甿KR 瘇LR 癄MR 盳NR 眱OR 矋PR 矏QR 矤RR 硑SR 磘TR 穀UR 糿VR 紟WR 紪XR 綍YR 綡ZR 縏[R 羪\R 翶]R 胐^R 脪_R 膁`R 艕aR 茲bR 茿cR 菐dR 菛eR 蒰fR 蔁gR 蔙hR 蔛iR 藰jR 虖kR 蛽lR 蝁mR 蝊nR 螊oR 螛pR 蟝qR 衯rR 裮sR 詅tR 謰uR 趽vR 跥wR 蹎xR 軞yR 軸zR 醻{R 鋔|R 鋥}R 鏎~R 鑨R 鑬R 鑶R 钂R 閪R 雺R 霮R 靍R 馤R 駦R 騲R 驡R 髥R 魠R 魶R 鮼R 鯓R 鱳R 鳼R 麅R ꧔R 끔R 덤R 멍R 샌R 섯R 죄R 캔R 텍R R R R 🆅R 🌮R 🍳R 🍸R 🏔R 🏰R 📄R 🔪R 🟢R 🤵R ŬR ȏR ՁR ٔR ଶR ఫR ཊR ኑR ፋR ᕱR ἦR ↡R ⇀R ┍R ┑R ╡R ◠R ❎R ➅R ⤷R 〼R ヅR ㄐR ㄡR 㿠R 乽R 仒R 債R 兺R 凒R 匌R 哅R 啂R 喸R 嚠R 壀R 孲R 寚R 岴R 嵳R 幒R 弴R 彃R 彙R 懸R 戨R 挜R 掃R 擆R 擴R 敼R 晉R 曂R 曆R 曊R 曑R 櫖R 歽R 毧R 浨R 淁R 漇R 澝R 濐R 灗R 牞R 猚R 獹R 痶R 皗R 矎R 磤R 秐R 竎R 篻R 粀R 紖R 紣R 綼R 翽R 腅R 臲R 蘱R 蚭R 蟺R 蟿R 衟R 褞S 襂S 觕S 誋S 誘S 軄S 迾S 遪S 鉶S 錏 S 鐴 S 閝 S 靇 S 鞪 S 頝S 騗S 騬S 鬱S 鸗S 麱S 鼏S 鼞S ꆹS 낙S 댄S 룹S 씀S 씩S 엠S 젝S 흰S S  S !S "S ︈#S 🆕$S 🍦%S 🏛&S 🐳'S 👹(S 💄)S 🖋*S 🙅+S 🛠,S 🛰-S 🥟.S ҫ/S ٥0S ށ1S ળ2S ଣ3S ହ4S ඒ5S ᗨ6S ថ7S Ạ8S ⏎9S ❔:S ➐;S ⟫S ㄉ?S 䃎@S 伋AS 凮BS 凲CS 噖DS 墿ES 寣FS 崏GS 嶺HS 嶿IS 巏JS 惵KS 愃LS 憑MS 挸NS 摢OS 撊PS 梥QS 橝RS 橫SS 橰TS 欥US 歱VS 浾WS 潐XS 濝YS 灮ZS 焤[S 犮\S 甔]S 甧^S 祪_S 穎`S 竩aS 箿bS 篽cS 篿dS 籠eS 糋fS 紗gS 綋hS 締iS 緩jS 縱kS 繉lS 翇mS 耺nS 臡oS 萞pS 薗qS 蘵rS 蘻sS 虃tS 蛧uS 螔vS 覍wS 觼xS 譿yS 趖zS 軒{S 遚|S 鄤}S 酓~S 釫S 鉂S 鋵S 鍾S 陠S 隁S 隬S 靦S 韕S 餺S 馸S 鬝S 鰲S 鱁S ꉌS ꌠS ꑳS 꺼S 랐S 봤S 젠S 짐S 칩S 힐S S S S S S S S ￶S 🆁S 🌵S 🍫S 🏋S 🏫S 🏵S 🔜S 🔽S 🕸S 🕹S 🚌S 🛑S 🛫S 🦉S 🦞S ŎS ɅS ϫS ێS ۝S ތS ઇS ઉS ొS ሽS ቲS ᑭS ᒪS ៅS ṿS ὡS ⃚S ⌀S ⍥S ⎡S ⏬S ✼S ➻S ゑS 侙S 偣S 傇S 嫿S 嬵S 嵐S 嶱S 弚S 徖S 徟S 忦S 忩S 扅S 揰S 搇S 敨S 曯S 朖S 樕S 樝S 櫔S 櫕S 湧S 灎S 炴S 煀S 猉S 皠S 睪S 碝S 碵S 腸S 臥S 葐S 蔝S 衑S 襀S 觲S 託S 謼S 趌S 軤S 輲S 逩S 釨S 鋾S 錹S 鎫S 雈S 頧S 騚S 鸓S 鼸S 齗S ꜰS 닉S 델S 빙S 빵T 옵T 융T 팩T T T 🅐T 🏙T 👮T 👾 T 🕰 T 🥧 T 🦌 T 🧤 T ƛT ҡT ܒT ಆT ዚT ᎬT ᐦT ᖘT ៍T ḗT ỚT ╧T ✵T ✷T ⟪T ぢT ヮT 兘T 塐 T 庿!T 楶"T 煫#T 熃$T 燂%T 爒&T 秼'T 穻(T 簄)T 簜*T 紻+T 綧,T 縎-T 聫.T 螩/T 蟚0T 蠇1T 詃2T 诇3T 豂4T 躍5T 醔6T 鑧7T ꐊ8T ꕥ9T 렬:T 빼;T 뼈T 옛?T 큐@T AT BT CT DT ET ￰FT 🏥GT 🐬HT 👂IT 👕JT 🕴KT 😑LT 🚲MT 🛩NT 🤚OT 🧜PT 􏰀QT 􏰈RT ˏST ΊTT ֒UT ܛVT ܠWT ܦXT ሜYT ቻZT Ꭿ[T ḱ\T ὼ]T ≀^T ⎕_T ⏲`T ┇aT ⲱbT ⻢cT 㞎dT 佦eT 勁fT 埪gT 壂hT 屫iT 摚jT 斬kT 梲lT 槉mT 樼nT 殘oT 淚pT 湂qT 漡rT 濾sT 牭tT 狇uT 甦vT 疛wT 箳xT 綜yT 藱zT 蚹{T 蝢|T 螲}T 豰~T 贜T 踁T 軓T 鈱T 鍟T 鎖T 顝T 饒T 馮T 魨T 鯲T 鰸T ꐥT 싫T 짝T T 🅱T 🆘T 🍩T 🍴T 🐊T 🐓T 👓T 🤎T 🧢T ʐT ڌT ܚT ৺T ਘT ଗT ଷT ೊT ഇT ༝T ᅡT ሎT ሞT ኝT ᎢT ᕐT ᙏT ₿T ℞T ◘T ◤T ⛄T ➟T ➶T ⻰T 〓T ㄖT ㄙT 䓣T 倆T 寵T 拋T 栭T 檱T 熸T 矙T 硴T 笡T 綱T 綺T 繞T 衶T 裷T 襢T 豼T 賊T 躮T 辭T 迉T 郹T 鄬T 鈌T 闆T 韀T 頌T 駕T 魾T ꔷT 갓T 롱T 멜T 몽T 밴T 븐T 쯤T 콩T 퇴T T T T T T T T T T T T 🌭T 🐯T 👰T 📮T 🔁T 🔨T 🤰T 🥗T 🧩T ǥT ǪT ʑT װT એT ళT ೈT ၂T ሌT ᑯT ᓄT ᗢT ḎT ỀT ỦT ⇖T ⇾T ➏U ➑U ㄌU ㄛU 凍U 匎U 噴U 彊U 曓U 梊 U 殼 U 潰 U 爺 U 獅 U 盜U 窶U 綾U 腗U 衈U 褺U 蹤U 鋒U 錫U 隥U 齡U ꀐU ꊿU ꎭU 겼U 곁U 릿U 뱀U 빅 U 즌!U 팡"U 흑#U $U %U &U 'U 🍹(U 🏸)U 👭*U 💲+U 📎,U 😶-U 😺.U 🤢/U 🦷0U 𠃊1U Ĥ2U Ļ3U ɷ4U Ή5U ֗6U ڪ7U ਧ8U ੜ9U ༒:U ሩ;U ቁU ឹ?U ៃ@U ៖AU ᶘBU ἉCU ἠDU ἤEU ὗFU ≻GU ◅HU ☦IU ⛩JU ➆KU ➼LU ⟹MU ⵉNU ⸮OU 〢PU ぅQU ぉRU ゐSU 嗭TU 塗UU 廈VU 廚WU 樑XU 樸YU 澵ZU 瘋[U 緻\U 聰]U 膩^U 蘁_U 謙`U 輛aU 辯bU 遷cU 鄧dU 飽eU ꀊfU ꆎgU 갤hU 겪iU 낄jU 늄kU 닫lU 됨mU 촉nU 톡oU pU qU rU sU tU uU 🌕vU 🐈wU 👗xU 👠yU 📉zU 🔼{U 😓|U 🚘}U 🟠~U 🤏U 🤛U 🥒U 🥦U 🥲U 🦈U 🧁U ƴU ɱU ސU ޮU ଅU ୋU ഡU ൈU ྆U ᛖU ᛚU U ⎣U ╍U ◖U ☇U ☶U ⛅U ➒U ⻚U ⻣U 偽U 刪U 惽U 搖U 毷U 渦U 爛U 譽U 豬U 遲U 鏈U 閃U 鵋U 긍U 깃U 꽉U 꿀U 낭U 덮U 돕U 듀U 샘U 셜U 엽U 잃U 퀴U 힌U U U U U U U U U 🅝U 🌀U 🌠U 🌡U 🌪U 🎻U 🏉U 🏎U 👯U 🙆U 🤒U 🥖U 🥞U 🥥U ƆU ǦU ȳU ˓U ҝU ګU ڮU ܟU ܶU ૩U ఉU ಎU ຼU ྃU ဇU ჰU ቦU ᓇU ᓯU ᔭU ṉU ὉU ὖU ↬U ┣U ┮U ╤U ⛺U ✢U ㄊU ㄗU 㸆U 䃼U 儘U 儲U 劑U 唸U 暢U 椦U 櫦U 毬U 瀩U 瑪U 禎U 緯U 繈U 罷U 貧U 閑U 騰U 騺U 鴨U 곧V 꼽V 댑V 륭V 멸V 묘V 뷔V 셀V 옆V 윈 V 잭 V 쳤 V 펴 V 훌 V V V V V V V V V V V V V V 🍭V 🎖V 🎫V 💑V 🕶V 🗨 V 😨!V 😼"V 🙀#V ͢$V ױ%V ݁&V ࣰ'V ॱ(V ఇ)V ೃ*V භ+V ༔,V ဗ-V ዶ.V ፣/V Ꮇ0V ᕙ1V Ầ2V ⇝3V ⊞4V ⋙5V ⌈6V ⎠7V ⎢8V ⏸9V ┭:V ◓;V ◢V ✠?V ❰@V ⨯AV ⻉BV 㡳CV 㨃DV 妝EV 嶼FV 捨GV 擾HV 槍IV 漁JV 濁KV 祿LV 綿MV 贏NV 醬OV 釣PV 鋪QV 錶RV 鑒SV ꉪTV ꙪUV ꜥVV 꾼WV 냉XV 뇌YV 땅ZV 뚫[V 랫\V 앤]V 쥐^V 탐_V 헬`V aV bV cV dV eV fV gV hV iV jV kV ︍lV 🎲mV 🐎nV 🐖oV 🐠pV 🐷qV 🔺rV 🛡sV 🤸tV 🧂uV ƑvV ȥwV ˄xV ՖyV ՛zV ঝ{V ෑ|V ཎ}V ሙ~V ሬV ጅV ጭV ᎠV ᐤV ៉V ἁV ἅV ₲V ⇤V ⌟V ⎝V ▕V ⚇V ⛵V ⛷V ✜V ✸V ❩V ⲭV ㄔV 吶V 帥V 悶V 暫V 桿V 牽V 獸V 砲V 翹V 腎V 菴V 謹V 譚V 貪V 賃V 贊V 鄰V 銃V 鑽V 驅V ꆺV ꍟV ꙼V ꜱV 닷V 듬V 렵V 맡V 멤V 씬V 쿵V V V V V V V V V V V V V V V ﴾V ￿V 🇶V 🍌V 🎆V 🏊V 🐧V 👿V 🔋V 🔐V 🖌V 😣V 🚎V 🛵V 🟡V 🤜V 🤲V 🤴V 🥨V 🧬V 𠈌V 𦰡V ŅV ŔV ǖV ɝV ʅV ֛V ڤV ਫV ੧V ଙV ଧV ୬V ಇV ഥV ༵V ဳV ၁V ᅢV ኮV ጀV ᎪV ᓫV ᔪV ᚦV ᛋV ផV ṈV ẬV ỪV ἩV ↺V ≺V ⊤V ⌵V ⏪V ╕V ╢V ◥V ☣V ⛑V ➇V ⥨W ⵜW 〤W 〽W 㙟W 孃W 撐W 毀W 渾W 滄 W 珮 W 箇 W 褲 W 諦 W 謎W 賜W 賤W 踐W 迆W 鍛W 餵W ꈓW ꐛW 겐W 끊W 맹W 밍W 샷W 컷W 톤W 핫W W  W !W "W #W $W %W &W 'W (W ￸)W 🅳*W 🍼+W 🏄,W 🏍-W 🏞.W 🐑/W 🐙0W 👐1W 🔟2W 😰3W 🚴4W 🥔5W 🧍6W 🫶7W Ʒ8W ȧ9W ɘ:W ˟;W ͵W ՚?W ٓ@W ٬AW ٽBW ځCW ܥDW ॒EW ৱFW ଭGW ၏HW ኒIW ኩJW ዱKW ᎡLW ᎩMW ᎽNW ᎾOW ᏚPW ᒋQW ᕑRW ᖗSW ᛅTW ṓUW ỨVW ὈWW ὥXW ᾰYW ᾳZW ≋[W ⎦\W ⚶]W ⛓^W ❨_W ⩽`W ⬤aW ㄋbW ㄜcW ㄞdW 㻗eW 壺fW 峽gW 巖hW 悅iW 欽jW 毌kW 畬lW 祦mW 籌nW 縫oW 纏pW 罰qW 罵rW 蝕sW 蝦tW 蠻uW 諮vW 趕wW 駒xW 駿yW 鱗zW ꃀ{W ꈊ|W ꉘ}W ꙰~W 곱W 렛W 룰W 밸W 뻐W 잎W 젤W 콤W 탬W 햇W W W W W W W W W W W W W ︁W 🅨W 🅶W 🌽W 🍏W 🍓W 🍝W 🏓W 🐇W 🐢W 🐸W 👥W 👫W 💊W 🔄W 🔖W 🕒W 😖W 😸W 🚢W 🥙W 🥬W 🦎W 🦖W ŊW ƸW ȩW ȻW ҲW ٗW ټW ۓW ॑W ૫W ଂW ఖW ഖW ຟW ຣW ༀW ༓W ࿚W ሸW ቋW ቴW ᏂW ᐱW ᓕW ᗣW ᴊW ṮW ἵW ₮W ⇃W ⇗W ⌉W ⌜W ⎛W ⎞W ␓W ┝W ╞W ╨W ╫W ▱W ⚒W ⚰W ⛏W ❡W ⬥W ⵏW ⷪW ⿰W ヂW ㄎW ㄝW 䓯W 僱W 奮W 攜W 滾W 濱W 訶W 詰W 豔W 霧W 韋W 鴉W ꀒW ꌒW ꏀW ꐙW 겹W 껍W 덜W 듭W 맑W 숫W 얘W 쩌W 틸W 픔W 흐W 흙W W W W X X X X X X X X X  X  X  X ﴿ X ﷯ X 🌅X 🍇X 🐫X 🐿X 👔X 👟X 📡X 🛋X 🛸X 🡆X 🧈X ŶX ǹX ǿX ɻX ˅X ͩX ϟX ܨ X ৹!X ઈ"X ଚ#X ஃ$X ᄁ%X ᅩ&X ሔ'X ኔ(X ኦ)X Ꮃ*X Ꮋ+X Ꮧ,X ᔨ-X ᕆ.X ᗤ/X ឆ0X Ḍ1X ἂ2X ⁂3X ≐4X ⊑5X ⍨6X ➓7X ⠂8X ⧫9X ⫸:X ⭣;X ⲕX 䭹?X 淺@X 澱AX 睏BX 賈CX 賠DX 軌EX 銳FX 靚GX 驟HX ꄚIX ꇉJX ꑵKX 겸LX 떼MX 랭NX 빗OX 삽PX 슷QX 펼RX 폼SX TX UX VX WX XX YX ZX [X \X ]X ^X _X 﨑`X ￵aX 🌌bX 🌾cX 🎒dX 🏌eX 🐞fX 🐨gX 🐮hX 👵iX 🕡jX 😟kX 🚙lX 🚶mX 🦢nX 🧙oX 􏰖pX ϳqX ҰrX ҶsX ӜtX ӯuX ٪vX ܓwX ޔxX ๛yX ༘zX ဟ{X ᅦ|X ሥ}X ቹ~X ዛX ጻX ᒡX ᒥX ᝰX ៌X ᠍X ᴢX ḿX ỔX ὔX ῬX ⁕X ∊X ⏫X ⏺X ␀X ┊X ▚X ☴X ⛸X ⧉X ⧼X ⧽X ⮞X ⵔX ⿱X 㸝X 喬X 奪X 嶽X 憐X 捲X 旂X 氾X 洩X 漲X 漿X 爐X 竊X 縛X 臟X 頸X 駛X ꄮX ꆪX ꐤX ꞉X 납X 눌X 닛X 뒷X 릇X 릎X 맙X 묻X 벳X 벼X 챗X 컵X 쾌X 펀X 홉X X X X X X X X X X X X X X X ￷X 🌋X 🌑X 🌯X 🍞X 🍯X 🐚X 🐼X 👅X 👙X 📋X 🕑X 😚X 🙇X 🚑X 🚕X 🛌X 🥐X 🦂X 🦪X 🧊X 🧟X 🧣X 🧥X 􏰂X 􏰊X ĜX ŜX ƞX ƳX ʞX ˹X ϛX ϣX ׇX ڕX ܺX ੴX ଏX ఈX ృX ൢX ဏX ሀX ሱX ዉX ዩX ጽX ᎥX ᎦX ᏢX ᐉX ᛒY ៀY ។Y ១Y ỰY ἍY ₯Y ⃜Y ⍝Y ▋ Y ◣ Y ☤ Y ☳ Y ⚧ Y ⺣Y ⻜Y ⻬Y ㄘY 䳭Y 厭Y 喚Y 嚇Y 懶Y 敘Y 曬Y 樺Y 櫃Y 櫚Y 汎Y 滲Y 燦Y 瑋Y 磯 Y 簾!Y 繳"Y 釘#Y 頒$Y 頤%Y 鵬&Y 鷹'Y 龐(Y ꂵ)Y ꈎ*Y ꌩ+Y ꐦ,Y ꐮ-Y ꐯ.Y 굽/Y 냅0Y 넥1Y 둑2Y 랬3Y 렘4Y 믹5Y 뱅6Y 뻔7Y 셉8Y 숲9Y 얀:Y 옮;Y 읍Y 켰?Y 쿨@Y 팝AY 헛BY CY DY EY FY GY HY IY JY KY LY MY NY OY PY QY RY SY TY UY VY WY ￴XY 𓈒YY 🆙ZY 🎢[Y 🎣\Y 🎪]Y 🎱^Y 👄_Y 💟`Y 💷aY 📀bY 📜cY 📨dY 🔌eY 🥅fY 🥪gY 🧭hY 🩸iY 𣎴jY 𬌗kY 􀳫lY ŧmY ƁnY ƣoY ɠpY ΏqY ϩrY ϻsY ҙtY ҧuY ԜvY ֤wY ޭxY ઝyY ૨zY ୫{Y ୭|Y ஈ}Y ಖ~Y ഉY ൃY ඨY ༊Y ᄎY ᅚY ሓY ጫY ፀY ᏁY ᏍY ᛉY ỒY ἕY ὠY ⁜Y ₧Y ↤Y ⇲Y ∟Y ⊖Y ⌐Y ⍃Y ⎥Y ⏭Y ┋Y ☬Y ♗Y ♾Y ⬡Y ⬩Y ⸱Y 㘵Y 䄂Y 䌷Y 䍸Y 䜌Y 僑Y 堯Y 廂Y 彿Y 灘Y 煉Y 煥Y 瑩Y 痠Y 瞼Y 篤Y 腫Y 臘Y 蔥Y 蘊Y 邁Y 邏Y 釦Y 頗Y 顛Y 餃Y 馴Y 鳩Y ꎷY ꐪY 겉Y 놈Y 뇨Y 닙Y 닭Y 닮Y 닿Y 듈Y 뭔Y 뵈Y 붕Y 섹Y 씻Y 앵Y 줬Y 칸Y 캬Y 탭Y 탱Y 팟Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 𓂃Y 🅢Y 🆄Y 🍆Y 🍍Y 🍜Y 🐒Y 💳Y 📏Y 🔙Y 🔰Y 🗡Y 🚂Y 🚦Y 🥋Y 🦸Y 􏰉Y ƊY ƵY ǻY ȇY ϜY ѳY ѺY ӑY ӕY ږY ܽY ঐY ੌY ୮Y ఏY ౨Z ཥZ ྉZ ྥZ ჟZ ᄄZ ᅥZ ቆZ ዴZ Ꭷ Z Ꮳ Z ᑲ Z ᒍ Z ᒻ Z ᓴZ ᗩZ ២Z ᯤZ ḞZ ὙZ ↟Z ↾Z ≒Z ⊘Z ⋀Z ⍩Z ❙Z ⟱Z ㄆZ 㭎Z 㾀Z 勸Z 壯 Z 嬌!Z 撫"Z 楓#Z 湊$Z 漬%Z 瑤&Z 癥'Z 癮(Z 糾)Z 繩*Z 脅+Z 艱,Z 艷-Z 荊.Z 豈/Z 鈞0Z 銜1Z 飄2Z 飼3Z 餡4Z 鯨5Z 鹼6Z ꁏ7Z ꅽ8Z ꈴ9Z ꉈ:Z ꋌ;Z ꍏZ 깜?Z 댈@Z 둔AZ 딜BZ 땐CZ 렀DZ 렐EZ 롬FZ 륙GZ 맟HZ 밌IZ 밑JZ 볏KZ 볶LZ 봅MZ 쁨NZ 쌍OZ 앰PZ 좁QZ 짠RZ 퀸SZ 펙TZ 푼UZ 헷VZ 흘WZ XZ YZ ZZ [Z \Z ]Z ^Z _Z `Z aZ bZ cZ dZ eZ fZ gZ hZ iZ jZ kZ lZ mZ nZ oZ pZ qZ ︅rZ ︌sZ 🅘tZ 🌖uZ 🌨vZ 🎺wZ 🏺xZ 🐅yZ 🐛zZ 📗{Z 🔉|Z 🔧}Z 😧~Z 🙉Z 🛣Z 🦏Z 🦓Z 🧛Z 🧿Z 𡃁Z 󰆅Z ƚZ ƹZ ȁZ ϯZ ֧Z ֨Z ڀZ ډZ ڊZ ڵZ ڻZ ۍZ ۤZ ۬Z ޙZ ऽZ ਐZ ਯZ ଖZ ୯Z ຝZ ໒Z ༚Z ဥZ ᇂZ ሐZ ሻZ ኳZ ጨZ ጸZ ፅZ ፊZ ᏔZ ᓗZ ឡZ ḺZ ẕZ ẞZ ỞZ ↼Z ⋞Z ⋟Z ⌝Z ⍴Z ⎙Z ⎤Z ╜Z ☧Z ☵Z ♟Z ⚝Z ⛲Z ⵈZ ⵍZ ⵢZ 㙍Z 勣Z 唄Z 嘆Z 嘯Z 囉Z 崙Z 廁Z 徬Z 懼Z 棧Z 棲Z 檳Z 淪Z 牴Z 狹Z 瓊Z 礙Z 禦Z 糧Z 緋Z 繡Z 蘆Z 輿Z 遼Z 醜Z 閤Z 闢Z 颳Z 飢Z 餓Z 馳Z 騷Z 鯉Z 鵝Z 鸞Z ꃰZ ꄸZ ꆈZ ꇤZ ꈀZ ꉢZ ꌗZ ꑍZ ꟷZ ꫞Z 굿Z 귄Z 꿔Z 맴Z 쇠Z 싹Z 엉Z 왼Z 잇Z 쥬Z 찔Z 퀘Z 흄Z Z Z Z Z Z Z Z [ [ [ [ [ [ [ [ [  [  [  [  [  [ [ [ [ [ [ 𐂂[ 𓏸[ 🅔[ 🅖[ 🅛[ 🅿[ 🌛[ 🌫[ 🍐[ 🍣[ 🍲[ 🎡[ 🎷[ 🏦 [ 🏭![ 🐄"[ 🐋#[ 🐔$[ 👪%[ 👱&[ 💾'[ 📁([ 📙)[ 📼*[ 🖨+[ 😗,[ 🚣-[ 🤼.[ 🥎/[ 🥭0[ 🦍1[ 🦶2[ 🫁3[ 􀁸4[ 􏰎5[ Ĵ6[ Ɣ7[ Ɵ8[ ǟ9[ Ƚ:[ ʦ;[ Ϊ<[ ϥ=[ ϧ>[ Ϯ?[ Ғ@[ ڟA[ ڠB[ ۢC[ ޖD[ ऱE[ ଛF[ ఠG[ ಉH[ ഓI[ ඹJ[ ༉K[ ሄL[ ሠM[ ሶN[ ቕO[ ዙP[ ጃQ[ ᎵR[ ᏒS[ ᏞT[ ᖃU[ ᜔V[ ៊W[ ᥫX[ ᩣY[ ᮩZ[ ᷆[[ Ἔ\[ ἥ][ ὤ^[ ↞_[ ↥`[ ⇚a[ ⇞b[ ⇰c[ ⇱d[ ∻e[ ⋰f[ ⌑g[ ┠h[ ╂i[ ╵j[ ☓k[ ♜l[ ⚚m[ ➬n[ ➾o[ ⦾p[ ⴷq[ 㗲r[ 㣵s[ 㧾t[ 㳜u[ 㳨v[ 㸔w[ 䟌x[ 䠧y[ 䫌z[ 侶{[ 勗|[ 喲}[ 壘~[ 彎[ 擠[ 柵[ 樞[ 檸[ 殤[ 浬[ 畝[ 羨[ 蕩[ 薔[ 詮[ 貯[ 貶[ 鎂[ 闊[ 隷[ 霑[ 龜[ ꈨ[ ꋒ[ ꐚ[ ꑋ[ 꽂[ 끗[ 늑[ 둠[ 딘[ 뚜[ 띄[ 띠[ 릉[ 맵[ 밭[ 쪼[ 챙[ 튬[ 팁[ 헮[ 헲[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ ︂[ ￲[ 𐌰[ 🆈[ 🌚[ 🍉[ 🍬[ 🏇[ 🏚[ 🐥[ 👣[ 👴[ 💂[ 💤[ 🔦[ 🔫[ 🔱[ 🚜[ 🢖[ 🥍[ 🦔[ 🦕[ 🦙[ 🦜[ 🦵[ 🧞[ 🧦[ 🧪[ 🧱[ 🩳[ ɿ[ ˬ[ ˳[ ͻ[ Ͽ[ Ѩ[ ܙ[ ޝ[ ޞ[ ਏ[ ୱ[ ౌ[ ໆ[ ཌ[ ၥ[ ኞ[ ጓ[ Ꮹ[ Ꮿ[ Ᏼ[ ᐁ[ ᕦ\ ᛞ\ ឋ\ ᯅ\ ᷇\ ḽ\ ṡ\ ἢ\ Ἠ\ ₣ \ ∖ \ ⊳ \ ⊶ \ ⋱\ ⓫\ ┕\ ┨\ ◃\ ☱\ ♆\ ♰\ ➴\ ⠑\ ⣠\ ⨁\ ⻨\ ヵ\ 㒸\ 㓎\ 㕑\ 㞞\ 㟽 \ 㧟!\ 㼆"\ 䁔#\ 䟺$\ 䴗%\ 傘&\ 儷'\ 厲(\ 墜)\ 彫*\ 掙+\ 疊,\ 穌-\ 窩.\ 竄/\ 篩0\ 籥1\ 繆2\ 莖3\ 蔭4\ 螢5\ 覓6\ 詐7\ 誇8\ 鈣9\ 鉤:\ 鑄;\ 隸<\ 顎=\ 龔>\ ꂷ?\ ꄲ@\ ꆳA\ ꇑB\ ꉙC\ ꊰD\ ꋨE\ ꐰF\ ꑓG\ ꫀH\ 껏I\ 꽁J\ 돋K\ 랩L\ 붉M\ 붓N\ 샀O\ 쉴P\ 싼Q\ 쏟R\ 얄S\ 쭉T\ 켄U\ 킷V\ 턱W\ 펠X\ 펭Y\ 헝Z\ 헶[\ 헻\\ 훨]\ ^\ _\ `\ a\ b\ c\ d\ e\ f\ g\ h\ i\ j\ k\ l\ m\ n\ o\ p\ q\ r\ s\ t\ u\ v\ w\ x\ y\ z\ {\ |\ }\ ~\ \ \ \ ￾\ 𑐳\ 𑐶\ 𑐸\ 𓀠\ 𓂂\ 🃏\ 🅞\ 🌩\ 🏗\ 🐭\ 💶\ 📶\ 🔯\ 🚁\ 🚔\ 🤱\ 🤹\ 🤺\ 🧼\ 🧽\ 🩺\ 􏰃\ 􏰏\ Ǽ\ Ɇ\ ˨\ Ѡ\ ҥ\ ՜\ ׀\ ٖ\ ڧ\ ॆ\ ঋ\ ૮\ ଁ\ ஓ\ ಈ\ ೌ\ ೭\ ඡ\ ၅\ ၆\ ႈ\ ᄍ\ ᄐ\ ᅣ\ ሴ\ ዑ\ ጂ\ ጊ\ ᐠ\ ᓚ\ ᓪ\ ᓰ\ ᕤ\ ᖓ\ ᚠ\ ᚴ\ ឈ\ ᠄\ ḁ\ ṳ\ Ṿ\ ⁞\ ₸\ ⃒\ ↰\ ↲\ ⇉\ ⇽\ ∿\ ⊷\ ⓬\ ⓵\ ╋\ ╘\ ╻\ ☙\ ⚲\ ➰\ ⟡\ ⱡ\ ⴻ\ ⵎ\ 㑩\ 㝵\ 㫊\ 㴎\ 㵡\ 䂩\ 䓍\ 䔬\ 兌\ 凈\ 凜\ 勻\ 噹\ 恥\ 惱\ 暉\ 檻\ 欖\ 氫\ 瘍\ 癩\ 祅\ 籃\ 紡\ 絞\ 綁\ 膽\ 諧\ 諫\ 貍\ 鈉\ 鈔\ 鈦\ 鍍\ 闕\ 陜\ 顫\ 驕\ 鯊\ 齧\ ꀉ] ꁆ] ꁱ] ꁴ] ꂿ] ꈁ] ꌧ] Ꙩ] 귈] 꼴 ] 뀌 ] 낚 ] 넉 ] 덩 ] 딕] 땀] 렁] 뤄] 맺] 뽐] 셈] 셋] 셸] 쉼] 슘] 썸] 윙] 쩔] 츄] 킴] 톨] 툴] 틈 ] 헿!] "] #] $] %] &] '] (] )] *] +] ,] -] .] /] 0] 1] 2] 3] 4] 5] 6] 7] 8] 9] :] ;] <] =] >] ?] @] A] B] C] D] E] 𑑂F] 𓀃G] 🅺H] 🆓I] 🌦J] 🍟K] 🎇L] 🏧M] 🐪N] 🐲O] 💆P] 📃Q] 📑R] 📥S] 🔀T] 🔛U] 🕮V] 🗒W] 🗯X] 🗿Y] 🚐Z] 🚒[] 🚬\] 🟣]] 🡺^] 🤥_] 🥫`] 🥱a] 🥹b] 🦑c] 🦺d] 🧾e] 􏰍f] Ũg] ƌh] ƙi] Ƥj] ƨk] Ʊl] ǩm] Ƀn] Ϫo] Җp] Ӓq] ֳr] ٻs] څt] ڡu] ܗv] ޅw] ࡇx] ৷y] ਍z] ਓ{] ਠ|] ૃ}] ଯ~] ಫ] ೦] ෴] ။] ၚ] ၿ] ᄈ] ᄊ] ᅟ] ኃ] ዞ] ᐟ] ᐣ] ᒑ] ᗰ] ᗴ] ᙚ] ᚭ] ᚼ] ៈ] ០] ᦵ] ᪣] ᪥] ᴉ] ᷁] ḟ] Ṁ] ẃ] Ể] ὃ] Ὕ] ῡ] ⁊] ℥] ⅌] ↹] ⇜] ∡] ⊔] ⋈] ⌗] ⌫] ⌾] ⍰] ⎋] ⎓] ␛] ┫] ⠇] ⠈] ⠛] ⤺] ⬊] ⬌] ⬬] ⭆] Ɱ] ⴱ] ⵙ] ⷡ] ⺮] 〄] ヱ] 㔏] 㕮] 㘗] 㙦] 㠭] 㬵] 㵐] 㵘] 㹴] 䃺] 䋦] 䒕] 䒩] 䗪] 䥫] 䶥] 傢] 兇] 嘍] 嘔] 嚨] 墊] 崗] 廬] 弔] 愷] 慘] 憤] 撥] 擷] 暘] 暱] 檜] 歎] 汙] 滬] 癲] 硃] 窺] 竇] 糰] 絃] 繕] 繰] 蘿] 蛺] 詭] 誦] 誼] 諒] 賑] 迺] 釀] 釗] 鈧] 雛] 霤] 颯] 饋] 駱] 驛] 髒] 鮑] 鴛] 鷲] 鷺^ 鹹^ ꀾ^ ꄷ^ ꇐ^ ꉐ^ ꉚ^ ꉣ^ ꌂ^ ꎆ ^ ꎔ ^ ꏤ ^ ꏭ ^ ꏯ ^ ꑷ^ ꕤ^ ꯂ^ ꯃ^ ꯩ^ 낀^ 늙^ 땜^ 뚱^ 릅^ 뭘^ 빔^ 빡^ 삐^ 쉐^ 쌓^ 쏘^ 쑤^ 쑥 ^ 얜!^ 옹"^ 짤#^ 첩$^ 콕%^ 툰&^ 헨'^ 헹(^ 휠)^ *^ +^ ,^ -^ .^ /^ 0^ 1^ 2^ 3^ 4^ 5^ 6^ 7^ 8^ 9^ :^ ;^ <^ =^ >^ ?^ @^ A^ B^ C^ D^ E^ F^ G^ H^ I^ J^ K^ L^ M^ N^ O^ P^ Q^ 𐌽R^ 𑐖S^ 𑐰T^ 𓀀U^ 𓀁V^ 𓀂W^ 𓀄X^ 𓀅Y^ 𓀆Z^ 𓀇[^ 𓀈\^ 𓀉]^ 𓀊^^ 𓀋_^ 𓀌`^ 𓀍a^ 𓀎b^ 𓀏c^ 𓀐d^ 𓀑e^ 𓀒f^ 𓀓g^ 𓀔h^ 𓀕i^ 𓀖j^ 𓀗k^ 𓀘l^ 𓀙m^ 𓀚n^ 𓀛o^ 𓀜p^ 𓀝q^ 𓀞r^ 𓀟s^ 𓀡t^ 𓀢u^ 𓀣v^ 𓀤w^ 𓀥x^ 𓀦y^ 𓀧z^ 𓀨{^ 𓀩|^ 𓀪}^ 𓀫~^ 𓀬^ 𓀻^ 🅓^ 🅟^ 🅣^ 🍙^ 🐀^ 🐗^ 🐜^ 👖^ 📔^ 📟^ 📫^ 📿^ 🔓^ 🕉^ 🗂^ 🗑^ 🗗^ 🗻^ 🗼^ 🛬^ 🛳^ 🛴^ 🥄^ 🥯^ 🦦^ 🧃^ 🧒^ 🧓^ 🧕^ 🪦^ 󰀄^ 􀁾^ 􀳦^ 􏰄^ 􏰅^ Ƿ^ Ǹ^ Ȋ^ ȗ^ Ⱦ^ ɋ^ ɭ^ Ϟ^ Ϡ^ ϡ^ ϸ^ Ѣ^ Ѳ^ ҟ^ Ԍ^ ֭^ ٭^ ڰ^ ڳ^ ߞ^ ߹^ ॢ^ ਝ^ ૯^ ଥ^ ୂ^ ఐ^ ಐ^ ೨^ ໐^ ဉ^ ᅧ^ ᅴ^ ᆴ^ ሣ^ ቂ^ ቄ^ ኬ^ ዬ^ ጌ^ ፎ^ ፖ^ Ꮥ^ Ꮯ^ ᐎ^ ᐩ^ ᐸ^ ᒧ^ ᒫ^ ᕼ^ ᗪ^ ᛦ^ ឃ^ ឌ^ ឺ^ ៥^ ᷀^ ḓ^ ṋ^ ẑ^ Ỏ^ Ợ^ ἧ^ Ἥ^ Ἴ^ Ὅ^ ⁑^ ₢^ ⃰^ ℩^ ⇈^ ⊡^ ⊩^ ⋁^ ⌕^ ⌙^ ⍵^ ⏏^ ⓶^ ▯^ ☈^ ⛴^ ⟲^ ⠋^ ⠥^ ⣿^ ⪼^ ⬂^ ⬃^ ⬋^ ⮕^ ⵣ^ ⶪ_ ⸤_ ⸥_ ⻥_ ㇏_ 㑇_ 㑚_ 㒯_ 㖎_ 㖱 _ 㗳 _ 㛃 _ 㟌 _ 㢲 _ 㤭_ 㭛_ 㮚_ 㯎_ 䁩_ 䍁_ 䏨_ 䓘_ 䔯_ 䭢_ 冑_ 剝_ 喫_ 墻_ 尷_ 嵒_ 弒_ 懇_ 戔 _ 挾!_ 搶"_ 槓#_ 滷$_ 澀%_ 瀆&_ 燭'_ 癢(_ 盞)_ 穫*_ 簞+_ 簫,_ 紺-_ 絆._ 緬/_ 繽0_ 纔1_ 缽2_ 肅3_ 脹4_ 蕪5_ 薈6_ 虧7_ 蟄8_ 蟬9_ 蟻:_ 諳;_ 諷<_ 遜=_ 鈕>_ 鈺?_ 銚@_ 錘A_ 閘B_ 頰C_ 髏D_ 齦E_ ꀸF_ ꃢG_ ꇗH_ ꉡI_ ꌋJ_ ꍬK_ ꜆L_ ꝛM_ ꡔN_ ꡕO_ ꦼP_ 겊Q_ 겟R_ 궐S_ 꽤T_ 냄U_ 늬V_ 닦W_ 덴X_ 멧Y_ 밉Z_ 벅[_ 벚\_ 볍]_ 뻗^_ 섰__ 쌀`_ 쌤a_ 쏭b_ 앗c_ 엣d_ 왁e_ 웜f_ 윌g_ 젖h_ 쫓i_ 캡j_ 탓k_ 텝l_ 톱m_ 퉁n_ 풋o_ 헼p_ 흠q_ r_ s_ t_ u_ v_ w_ x_ y_ z_ {_ |_ }_ ~_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 𐌅_ 𐨛_ 𓀼_ 𓀽_ 𓀾_ 𓀿_ 𓁀_ 𓁁_ 𓁂_ 𓁃_ 𓁄_ 𓁅_ 𓁆_ 𓁇_ 𓁈_ 𓁉_ 𖤐_ 𝄞_ 🅗_ 🌇_ 🍖_ 🍗_ 🍛_ 🍮_ 🎠_ 🏂_ 🐵_ 👒_ 💇_ 💽_ 🔆_ 🖱_ 😙_ 🛁_ 🧗_ 🧮_ 🧴_ 🧶_ 🩹_ 􏰓_ Ŵ_ ƈ_ ƕ_ ƥ_ ǭ_ ȕ_ Ȳ_ ʨ_ ϙ_ Ͼ_ Ѝ_ Ҫ_ Ҭ_ ֜_ ׹_ ٲ_ ڿ_ ۽_ ܼ_ ݶ_ ޫ_ ߣ_ ࠥ_ ॽ_ ঔ_ ਢ_ ਥ_ ೧_ ഀ_ ൌ_ ൧_ ෞ_ ༛_ ᅬ_ ᅭ_ ᅱ_ ሟ_ ቐ_ ኅ_ ጆ_ ጡ_ ጾ_ ፆ_ ፌ_ ፕ_ Ꭳ_ Ꮖ_ Ꮚ_ Ꮛ_ Ᏸ_ ᑉ_ ᑫ_ ᑳ_ ᒌ` ᒯ` ᓈ` ᓱ` ᔅ` ᔕ` ᖁ` ᖇ` ᚐ` ᚩ ` ᚻ ` ᛜ ` ᛟ ` ᜄ ` ᜊ` ᜒ` ឬ` ៣` ៦` ᩚ` Ḑ` Ḗ` ṗ` ṙ` Ṟ` Ắ` Ẵ` Ờ` Ụ` Ử` ỵ` Ὠ` Ῑ ` ₾!` ⃠"` ℮#` Ⅎ$` ↀ%` ≾&` ⊝'` ⋂(` ⋃)` ⋖*` ⋗+` ⌶,` ⍎-` ⍟.` ⍳/` ⍸0` ␙1` ┷2` ╷3` ▞4` ◭5` ☌6` ✙7` ➿8` ⟬9` ⟭:` ⠄;` ⠓<` ⠙=` ⠼>` ⡱?` ⢀@` ⣄A` ⣷B` ⣾C` ⤋D` ⤫E` ⨉F` ⨱G` ⬐H` ⲃI` ⲑJ` ⵓK` ⵡL` ⻗M` ヰN` 㖭O` 㛑P` 㝬Q` 㤒R` 㱧S` 㳪T` 㸁U` 㸽V` 㹥W` 䀳X` 䁖Y` 䄉Z` 䇿[` 䑍\` 䓖]` 䕀^` 䞫_` 䟎`` 䠀a` 䦆b` 䨙c` 䰟d` 䱗e` 䶛f` 傭g` 僂h` 啞i` 嘩j` 婁k` 懺l` 戉m` 抎n` 撿o` 暈p` 棗q` 檯r` 滌s` 灝t` 燄u` 燉v` 獵w` 甌x` 癡y` 礦z` 禱{` 禿|` 箏}` 籟~` 絢` 絨` 綻` 繚` 耑` 耬` 聶` 膿` 芐` 茍` 萵` 虜` 螻` 訣` 諏` 諱` 諺` 謨` 軀` 軾` 鈍` 鋁` 鎊` 鏘` 鏢` 閎` 閨` 閻` 阯` 頦` 饅` 饗` 驢` 鬢` 鮫` 鰭` 鳶` 鴦` 鵲` ꀤ` ꁨ` ꁮ` ꄖ` ꅏ` ꆀ` ꆿ` ꇨ` ꇱ` ꇷ` ꈤ` ꈵ` ꉬ` ꊂ` ꋚ` ꌊ` ꌦ` ꌺ` ꍞ` ꏮ` ꑟ` ꑠ` ꒒` ꒫` ꓘ` ꜄` ꞙ` ꟾ` ꧉` ꪝ` ꫛ` ꬴ` ꯨ` 곹` 괄` 굳` 깭` 꼈` 낡` 낯` 놔` 놨` 눕` 눠` 덥` 뎐` 딪` 땡` 똥` 렷` 륜` 뭇` 밟` 뱃` 뽀` 숍` 숯` 썰` 썼` 옌` 왠` 줌` 쨌` 캄` 컸` 퀀` 톰` 팥` 펑` 핸` 혂` 홑` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` a a a a a a a a a  a  a  a  a  a a a a a a a a a a a a a a a a a a a  a !a "a #a $a %a &a 'a (a )a *a +a ,a -a .a /a 0a ﷐1a ﷮2a ︋3a 𐌳4a 𑐥5a 𑐮6a 𑐵7a 𑑅8a 🂡9a 🅥:a 🌄;a 🌔a 🎦?a 🏑@a 🏨Aa 🐡Ba 👬Ca 👷Da 📯Ea 🔲Fa 🕐Ga 🕛Ha 🗄Ia 😿Ja 🙍Ka 🚍La 🚛Ma 🛶Na 🛹Oa 🥸Pa 🥾Qa 🦒Ra 🦛Sa 🧋Ta 🧔Ua 🧳Va 𠂆Wa 𠙶Xa 𤰉Ya 𧿹Za 󰀃[a 󾓭\a 􀆰]a 􏰁^a Ɲ_a Ƣ`a Ʃaa ƫba Ǐca Ǒda Ǵea Ǻfa Ʉga ɮha ɰia ɶja ʆka ˞la ˴ma ˻na Ͱoa Ϋpa ΰqa ϭra Ϻsa Ѧta Ѭua Ѯva ѱwa Ѵxa Ѹya ѽza ҁ{a Ӈ|a ӱ}a Ԛ~a ՘a ֠a ؒa ٿa ۂa ܾa ܿa ޓa ޚa ߏa ࢭa ऩa ঈa ঊa ૉa ଞa ୃa ఊa ౞a ಌa ಞa ඊa ඕa ඛa ༂a ဓa ၄a ၉a ၍a ႊa ႐a ႕a ᅤa ᅨa ኪa ኸa ጁa ጤa ጹa ፃa ፉa Ꭸa Ꮗa Ꮻa Ᏹa ᐄa ᐪa ᐯa ᒐa ᓀa ᔒa ᔥa ᗦa ᚺa ᛈa ᛗa ៧a ៨a ᴩa ᵮa ᶇa ᷊a ᷎a Ṃa ẋa ẏa Ẕa Ẫa Ặa Ẽa Ỳa ἷa Ὄa ᾱa ῑa ⁙a ⁝a a ₭a ⇙a ∁a ∓a ∛a ≟a ≼a ⊄a ⊊a ⌞a ⌡a ⌬a ⍺a ⎧a ⓭a ⓮a ⓯a ╛a ╶a ☥a ☿a ♱a ⚋a ⛤a ❜a ⠏a ⠒a ⠞a ⠧a ⠹a ⤍a ⤻a ⨳a ⬘a ⬝a Ⱄa Ɐa Ⲁa ⲇa ⴳa ⴼa ⵖa ⻊a 㐅a 㑹a 㔹a 㕷a 㙾a 㝹a 㠠a 㠯a 㧯b 㭴b 㸃b 㸙b 㸞b 㽲b 䀹b 䅁b 䇒b 䍫 b 䐜 b 䒇 b 䗣 b 䘈 b 䘓b 䚬b 䚹b 䜩b 䞟b 䠡b 䯬b 䯼b 䲝b 䲠b 䴓b 侷b 倖b 剋b 囪b 姪b 屢b 彆b 懲 b 揹!b 攪"b 晝#b 曇$b 曖%b 櫓&b 櫛'b 滸(b 濫)b 瀨*b 瀾+b 爍,b 瑣-b 甕.b 盪/b 矇0b 窯1b 紂2b 紓3b 紳4b 綏5b 綬6b 芻7b 苧8b 蒔9b 蔔:b 蔞;b 蕎b 襯?b 訢@b 詔Ab 詣Bb 諗Cb 諜Db 諭Eb 謳Fb 賬Gb 轍Hb 鄒Ib 鉛Jb 錐Kb 鍬Lb 鐸Mb 鑰Nb 閥Ob 闔Pb 颱Qb 飪Rb 駁Sb 駭Tb 鬧Ub 鮭Vb 鱧Wb 鵡Xb 鶯Yb 鸚Zb 麩[b 龕\b ꁄ]b ꃷ^b ꄅ_b ꄐ`b ꆃab ꆅbb ꇂcb ꇇdb ꇊeb ꈚfb ꈧgb ꉆhb ꋦib ꋪjb ꌌkb ꎵlb ꏅmb ꑿnb ꙳ob ꜀pb Ꞓqb Ꞙrb ꧀sb 갇tb 곰ub 곽vb 굉wb 깁xb 깍yb 눅zb 딴{b 뜰|b 륨}b 맷~b 벡b 봇b 붐b 빚b 샛b 솟b 싣b 썬b 쏠b 옳b 웬b 잤b 잼b 젯b 젼b 짚b 찐b 찜b 찢b 칵b 캘b 탠b 퐁b 핍b 헐b 헰b 혐b 훼b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b ﷓b 𐤄b 𐤅b 𐬀c 𓅯c 𖥧c 🀆c 🅡c 🌉c 🌗c 🌘c 🌝c 🍡 c 🎔 c 🏬 c 🐂 c 🐆 c 👛c 👜c 👞c 💒c 💢c 🔘c 🔠c 🕌c 🖇c 🗃c 🚆c 🚇c 🚓c 🛀c 🛢c 🤶c 🤽c 🦭c 🦾 c 🧎!c 🧯"c 🪄#c 🪳$c 𠒇%c 𢯎&c 𦟌'c 𨻧(c 󶀁)c 􀀂*c 􏰐+c 􏰔,c Ŗ-c Ɯ.c Ƨ/c Ȃ0c Ȗ1c Ⱥ2c Ɏ3c ˩4c ͼ5c ΁6c ΂7c Ϛ8c ϶9c ѿ:c ҵ;c Ҽc Ӑ?c Ӗ@c ӳAc ԍBc ԎCc ԑDc ֎Ec ׿Fc ؏Gc ؔHc ؕIc ؘJc ڍKc ڱLc ۑMc ۾Nc ܵOc ݈Pc ݗQc ݚRc ݼSc ߐTc ࡌUc ࡘVc ࢣWc ࢵXc ࣐Yc ৐Zc ৗ[c ਺\c ੮]c ઠ^c ઢ_c ଽ`c ௚ac ఃbc ఘcc ಏdc ಒec ಓfc ಘgc ೫hc ೮ic ഈjc ൗkc ൠlc ༃mc ༆nc ༈oc ྛpc ࿈qc ࿓rc ဦsc ႀtc ჷuc ᅪvc ᅰwc ሂxc ሯyc ቓzc ቷ{c ኘ|c ኙ}c ጦ~c ፓc ፠c Ꭴc Ꭾc Ꮂc Ꮆc Ꮉc Ꮙc Ꮟc Ꮣc Ꮩc Ꮫc Ᏻc ᐋc ᐙc ᐹc ᑌc ᑖc ᑿc ᒢc ᓃc ᓵc ᔮc ᖛc ᗄc ᗝc ᙠc ᚫc ᚹc ᜆc ᜎc ឥc ឯc ៛c ᡠc ᦉc ᦕc ᧂc ᧉc ᩤc Ḱc Ṇc ṧc ẗc Ẩc Ữc Ỹc Ἕc ἳc ᾠc ᾥc ⁓c c ⃕c ⃖c ↄc ↿c ⇟c ⇺c ⇻c ∐c ∤c ≙c ≣c ≽c ⊎c ⊠c ⋉c ⋛c ⍼c ⎨c ⎩c ⎫c ⎬c ⎾c ⏌c ⓷c ▟c ◷c ☩c ♄c ⚏c ⚹c ⟗c ⟮c ⟯c ⟰c ⟳c ⠊c ⠕c ⠦c ⠴c ⠸c ⡯c ⤓c ⤬c ⥥c ⧗c ⪙c ⬀c ⬈c Ⱡc ⲝc ⵃc ⵟc ⸂c ⸃c ⺌c ⺾c ⻌c ⻤c ⻮c 〚c 〠c 〮c 㐱c 㒼c 㓥c 㖿c 㚺c 㝉c 㤄c 㦸c 㧢c 㩄c 㬢c 㵍c 㸅c 㹚c 㻥d 㿗d 䀉d 䀮d 䇂d 䇯d 䋞d 䏰d 䒦d 䕫 d 䖏 d 䖳 d 䛐 d 䣭 d 䬛d 䱻d 䲁d 佇d 佪d 嗚d 嘰d 嚮d 囁d 堿d 墮d 墾d 壢d 姦d 崢d 巑d 廝d 愜d 慄 d 慚!d 憚"d 懾#d 摟$d 撓%d 攬&d 暸'd 曄(d 瀧)d 熒*d 燙+d 燼,d 璽-d 瓖.d 皺/d 瞞0d 硯1d 磚2d 穢3d 箋4d 簀5d 籲6d 紲7d 絳8d 絹9d 綴:d 緞;d 繭d 蝸?d 蠔@d 蠱Ad 覈Bd 謊Cd 讖Dd 躊Ed 輒Fd 輟Gd 鉉Hd 銹Id 鋤Jd 錕Kd 錠Ld 錳Md 鏤Nd 鐔Od 鑲Pd 闖Qd 雋Rd 靂Sd 韌Td 餽Ud 饃Vd 馭Wd 駑Xd 髖Yd 鯖Zd 鵜[d 鷗\d 鹵]d 黴^d ꀎ_d ꀘ`d ꁌad ꁡbd ꂰcd ꄂdd ꄧed ꄱfd ꅍgd ꅐhd ꅑid ꇈjd ꇌkd ꇭld ꇰmd ꇿnd ꈈod ꈐpd ꈝqd ꈢrd ꈩsd ꉜtd ꋅud ꋊvd ꋯwd ꍝxd ꎬyd ꏁzd ꏂ{d ꏦ|d ꏿ}d ꐎ~d ꑞd ꑴd ꒉd ꒜d ꒨d ꜣd Ɥd ꟻd ꦛd ꪔd ꬵd 갠d 걍d 굡d 궈d 귤d 깐d 깝d 깻d 꼼d 꼿d 꾀d 꾹d 꿇d 낫d 넬d 덟d 둡d 뒀d 딥d 딧d 뚝d 룬d 맣d 몫d 묃d 뭄d 뵙d 뽕d 뿜d 샐d 솜d 쌈d 쌩d 쎄d 얇d 얹d 엿d 웰d 짙d 짬d 쩍d 쭌d 춥d 컹d 쿼d 킥d 텀d 푹d 헴d 헵d 횗d 횟d 훔d 휩d 흩d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d e e e e e e e e e  e  e  e  e  e e e e e e e e e e e e e e e e e e e  e !e "e #e $e %e &e 'e (e )e *e +e ,e -e .e /e 0e 1e 2e 3e 4e 5e 6e 7e 8e 9e :e ;e e ?e @e Ae Be Ce De Ee Fe Ge He 𐄚Ie 𐤔Je 𑄌Ke 𑄴Le 𑐐Me 𑐩Ne 𑐹Oe 𑑄Pe 𒆠Qe 🁢Re 🅧Se 🆖Te 🌃Ue 🍚Ve 🍤We 🎏Xe 🎚Ye 🎳Ze 🏤[e 🏮\e 🐃]e 🐏^e 🐤_e 🐩`e 💈ae 💮be 💴ce 📠de 📪ee 🕖fe 🛐ge 🛥he 🜄ie 🟊je 🟥ke 🡒le 🡪me 🢔ne 🤌oe 🥌pe 🦗qe 🦘re 🦳se 🦴te 🧫ue 🪑ve 🫂we 🫖xe 🫠ye 𠜎ze 𥔲{e 􀀀|e 􀀁}e 􀂃~e 􀆳e 􏰕e Ƈe ƍe ƻe Ȁe ȍe Ȑe ȣe ȫe ȷe ɂe Ɋe ʥe ˭e ˾e Ͳe ͳe Ѷe ѹe Ҝe Ӄe ӄe ӈe ӽe ԃe ԋe ԡe ԰e ՟e ֊e ֡e ֯e ؑe ٕe ڃe ڄe ڼe ۭe ܍e ܤe ݋e ݓe ݙe ݩe ݾe ޒe ޕe ޠe ޯe ޷e ߇e ߎe ߒe ߕe ߘe ߭e ߶e ߻e ࠮e ࡓe ऒe ৾e ਔe ਞe ੬e ੯e ઘe ૈe ଝe ୈe ఌe ఓe ఞe ౚe ౦e ౼e ಛe ೡe ೯e ഐe ഘe ඖe ඟe ඤe ෛe ෡e ෵e ໓e ༇e ༑e ༙e ྄e ྇e ࿅e ࿎e ဌe ဴe ၎e ၤe ჳe ᄫe ᅙe ᅫe ᅯe ᆭe ሹe ሾe ቨe ቪe ቫe ኡe ኻe ጳe ፐe ፒe ፲e Ꮏe Ꮐe Ꮼe ᐚe ᐵe ᑮe ᑰe ᒄe ᓖe ᓛe ᕈe ᕘe ᖏf ᖑf ᖢf ᖳf ᗡf ᙱf ᙳf ᚅf ᚓf ᚛ f ᚜ f ᛘ f ᜅ f ᜌ f ᜏf ᜓf ឧf ឿf ៤f ᨒf ᬩf ᬭf ᬶf ᮑf ᯓf ᱍf Ტf ᴎf ᴧf ᵺf ᶂf ᷂f ḉ f Ḣ!f ḣ"f ḧ#f Ḷ$f Ṗ%f ṩ&f ṷ'f ẁ(f ẅ)f ẍ*f ẘ+f Ẻ,f Ἆ-f ἓ.f ὦ/f ᾔ0f ᾗ1f ᾧ2f Ᾰ3f ῤ4f ⁁5f 6f ⃔7f ⃢8f ℄9f ℔:f ℟;f ⅋f ↫?f ⇼@f ∸Af ≍Bf ≗Cf ⊅Df ⊛Ef ⊜Ff ⊨Gf ⋘Hf ⌢If ⍧Jf ⎔Kf ⎭Lf ⎰Mf ⎷Nf ␦Of ┡Pf ▃Qf ▆Rf ▤Sf ◴Tf ☋Uf ♙Vf ♝Wf ⚀Xf ⚅Yf ⚐Zf ⚱[f ⛟\f ➵]f ⟷^f ⟸_f ⠃`f ⠎af ⠠bf ⣆cf ⣇df ⤤ef ⦅ff ⦆gf ⦵hf ⪆if ⫋jf ⫷kf ⬎lf ⬒mf ⬿nf ⭅of ⭢pf ⮂qf ⮟rf ⱄsf Ɽtf Ⱨuf ⱪvf ⲅwf Ⲓxf Ⲕyf Ⲟzf Ⲡ{f Ⲥ|f ⵇ}f ⵚ~f ⸨f ⸩f ⺅f ⺉f ⺖f ⺘f ⺡f ⺨f ⺭f ⺲f ⺹f ⻂f ⻏f ⻖f ⿻f 〛f ㄦf 㒰f 㒳f 㓗f 㖪f 㘰f 㜺f 㞗f 㞧f 㞿f 㟖f 㪺f 㳛f 㶇f 㸑f 㸬f 㻋f 㼪f 㽏f 㾦f 䂳f 䇄f 䈃f 䍙f 䏜f 䗴f 䝺f 䡶f 䢺f 䤩f 䥽f 䦊f 䧿f 䯄f 䰕f 䲢f 䴞f 䴩f 兗f 勛f 嚕f 嚥f 囀f 囂f 壎f 嫵f 寢f 屝f 崠f 巔f 幟f 惲f 憊f 搗f 摯f 擄f 擰f 擱f 攔f 斂f 曠f 朧f 枙f 梟f 橢f 殲f 淒f 漣f 潑f 澗f 瀕f 瀰f 煒f 煬f 燾f 犛f 瑯f 璣f 瓏f 痺f 瘞f 癱f 皰f 禰f 稟f 竅f 筍f 筧f 籬f 糢f 綸f 緲f 縉f 縞f 羈f 聳f 聾f 艙f 薊f 藷f 蠅f 衹f 詁f 詛f 誒f 誥f 諤f 豎f 貽f 賁f 贖f 踫f 蹣f 躒g 輻g 轅g 轆g 轡g 鄔g 鄴g 鄺g 釵g 鈐 g 銓 g 鋇 g 鍇 g 鍶 g 鎔g 鎗g 鏗g 鏟g 鏽g 鐳g 鑼g 鑾g 閩g 閭g 闡g 隴g 靄g 鞦g 頊g 頡g 頹g 颶g 颺 g 饌!g 駢"g 鬚#g 魘$g 鯰%g 鰍&g 黽'g 齲(g ꀄ)g ꀇ*g ꀑ+g ꀻ,g ꁁ-g ꁖ.g ꁦ/g ꁯ0g ꂁ1g ꂦ2g ꂸ3g ꂺ4g ꂼ5g ꄃ6g ꄆ7g ꄇ8g ꄏ9g ꄹ:g ꅇ;g ꅝg ꅿ?g ꆧ@g ꆫAg ꇓBg ꇢCg ꇪDg ꈌEg ꈜFg ꈫGg ꉎHg ꉼIg ꊈJg ꊌKg ꊾLg ꋆMg ꋠNg ꋻOg ꌐPg ꌡQg ꌣRg ꌬSg ꎂTg ꎧUg ꎳVg ꎴWg ꎹXg ꎺYg ꐘZg ꐝ[g ꐞ\g ꐡ]g ꐨ^g ꑱ_g ꜂`g ꜜag ꜳbg Ꝼcg ꠥdg ꡦeg ꪮfg ꯅgg ꯈhg 갚ig 갭jg 갯kg 갱lg 궂mg 궤ng 긋og 깎pg 꺾qg 껌rg 낳sg 넛tg 넨ug 넴vg 뇽wg 댕xg 듦yg 뛸zg 멎{g 멕|g 몹}g 뫼~g 뭉g 뮬g 벙g 뺏g 뽑g 뿓g 쁠g 섀g 솝g 슝g 쌉g 썩g 쏙g 쑈g 씁g 씹g 앓g 얗g 엌g 왓g 욤g 쩜g 쭈g 쯔g 챔g 챕g 첸g 컥g 쾅g 쿄g 큽g 팰g 펄g 퓸g 햄g 헜g 헳g 헽g 횡g 훠g 휨g 흉g 힙g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g h h h h h h h h h  h  h  h  h  h h h h h h h h h h h h h h h h h h h  h !h "h #h $h %h &h 'h (h )h *h +h ,h -h .h /h 0h 1h 2h 3h 4h 5h 6h 7h 8h 9h :h ;h h ?h @h Ah Bh Ch Dh Eh Fh Gh Hh Ih Jh Kh Lh Mh Nh Oh Ph Qh Rh Sh Th Uh Vh Wh Xh Yh Zh [h \h ]h ^h _h `h ah bh ch dh eh fh gh hh ih jh kh ﷽lh 𐀺mh 𐌲nh 𐌹oh 𐍀ph 𐍄qh 𐐈rh 𐤉sh 𐤋th 𐬁uh 𐬎vh 𐬯wh 𑁋xh 𑐡yh 𒀭zh 𒊒{h 𓃟|h 𓍄}h 𓏲~h 𖡼h 🅜h 🌂h 🌆h 🌓h 🍈h 🍠h 🍧h 🍨h 🎿h 🐁h 🐌h 🐹h 👃h 👲h 👺h 💱h 📇h 📓h 📭h 🔚h 🔞h 🕇h 🕍h 🕎h 🕚h 🖍h 🗔h 🗾h 😽h 🙎h 🚃h 🚄h 🚋h 🚖h 🚡h 🚪h 🚮h 🚰h 🚷h 🛄h 🛤h 🛺h 🤾h 🥡h 🥼h 🦚h 🦥h 🦧h 🦩h 🦮h 🦹h 🧇h 🧻h 🩰h 🪁h 🪓h 🪡h 🫀h 🫑h 🫡h 𠀤h 𠂇h 𠆢h 𠆤h 𠝹h 𠱸h 𤋮h 𬜯h 򐂰h 􀳇h 􏰇h 􏰋h 􏰑h 􏰒h 􏰛h Ƌh Ʈh Ǔh Ǚh Ǟh ǡh Ǩh Ǿh ȑh ȓh Ȩh Ȯh ȵh ɇh ɉh Ɍh ɼh ʄh ʡh ʪh ʮh ʯh ˪h ˰h ˱h ˸h ˼h ͸h ͹h ΃h ΢h ϗh Ϥh Ϩh Ͻh ѧh ѫh Ѿh ҄h ҕh Ҙh Ҧh Ҿh ҿh ӌh ӎh Ӣh Ӥh Ӹh Ԃh Ԅh Ԇh Ԑh ԕi ԟi Ԧi ԧi Ԩi Ԫi ֏i ֕i ֝i ׊ i ׍ i ׫ i ׮ i ׯ i ׷i ؀i ؂i ؋i ؍i ؗi ؜i ؽi ٛi ڐi ڙi ڨi ڸi ۃi ۠i ۧi ۯi ۺi ۻ i ܃!i ܄"i ܅#i ܑ$i ܔ%i ܜ&i ܧ'i ܭ(i ܴ)i ܸ*i ܹ+i ݅,i ݊-i ݔ.i ݞ/i ݢ0i ݣ1i ݧ2i ݨ3i ݫ4i ݭ5i ޟ6i ޤ7i ޻8i ߊ9i ߌ:i ߗ;i ߧi ߼?i ࠅ@i ࠈAi ࠦBi ࠵Ci ࠸Di ࠼Ei ࠽Fi ࠿Gi ࡀHi ࡐIi ࡰJi ࡱKi ࡶLi ࡽMi ࡿNi ࢇOi ࢍPi ࢖Qi ࢗRi ࢨSi ࢼTi ࣁUi ࣠Vi ࣣWi ࣱXi ऺYi ॏZi ঳[i ঽ\i ৆]i ৛^i ৠ_i ৣ`i ৼai ਑bi ਩ci ੏di ੔ei ੡fi ੢gi ੩hi ੪ii ੿ji ઄ki ઐli ઑmi ઴ni ૄoi ૬pi ૳qi ଴ri ଵsi ୊ti ୑ui ஛vi ஠wi ஫xi ௏yi ௐzi ௑{i ௙|i ௫}i ௬~i ௯i ௸i ௺i ௻i ఁi ఑i ౄi ౝi ౩i ౬i ౮i ౳i ౶i ಀi ಩i ಴i ಼i ಽi ೕi ೖi ೛i ೠi ೩i ೪i ೷i ೿i ഃi ഋi ഍i ഔi ൚i ൮i ൰i ඈi ඌi ඝi ඬi ෇i ෋i ෍i ෩i ෾i ໤i ༏i ༲i ༸i ༿i ཋi ཫi ྜi ྪi ྵi ࿊i ࿙i ဃi ဋi ဍi ဧi ၃i ၇i ၻi ၽi ႍi ႞i ჱi ჴi ჸi ᄚi ᆪi ᆲi ᆳi ᆵi ᆽi ሏi ሑi ሦi ሷi ቭi ቿi ኋi ኗi ኽi ዌi ጐi ጕi ጪi ጺi ፁi ፂi ፄi ፇi ፮i Ꭻi Ꭽi Ꮀi Ꮈi Ꮓi Ꮞi Ꮨi Ꮭi Ꮸi Ꮺi Ᏺi ᐏi ᐐi ᐞi ᐥi ᑑi ᔉi ᔑi ᕲi ᕽi ᖫi ᖭi ᗅi ᗯi ᘏi ᙡi ᚁi ᚆi ᚇi ᚈi ᚉi ᚋi ᚏi ᚪi ᛝi ᜃi ᜇi ᜈj ឣj ឱj ៏j ៗj ᠁j ᡗj ᡩj ᡵj ᤣ j ᥊ j ᦑ j ᦗ j ᦟ j ᦴj ᦹj ᦺj ᧈj ᨄj ᨐj ᨑj ᬓj ᬮj ᬱj ᱬj Ეj ᳵj ᴑj ᷈j ḃj ḅj Ḋj ḋ j ḕ!j ḙ"j Ṡ#j ṱ$j Ṽ%j ṽ&j Ẃ'j ẉ(j ẟ)j Ỗ*j Ỻ+j ἃ,j ἆ-j Ἦ.j Ἵ/j Ὡ0j ᾤ1j ᾷ2j Ᾱ3j ῐ4j ῒ5j ῠ6j ῴ7j Ὸ8j ⁋9j ⁧:j ⁳;j ℣j ↚?j ↨@j ↭Aj ⇇Bj ⇕Cj ⇯Dj ⇹Ej ≓Fj ≚Gj ⊋Hj ⊚Ij ⋚Jj ⋤Kj ⌸Lj ⌻Mj ⎈Nj ⎳Oj ⎻Pj ⎼Qj ⏍Rj ⏐Sj ⏚Tj ⏟Uj ⏹Vj ␄Wj ␔Xj ␤Yj ⓸Zj ⓺[j ⓻\j ┎]j ┒^j ┲_j ┾`j ╄aj ╒bj ╖cj ▇dj ♇ej ⚄fj ⚌gj ⛎hj ⛦ij ✇jj ✛kj ❢lj ❬mj ❲nj ❳oj ➈pj ➉qj ⟺rj ⟼sj ⠁tj ⠔uj ⠗vj ⠝wj ⠟xj ⣓yj ⣛zj ⣯{j ⤑|j ⤕}j ⤳~j ⥎j ⦕j ⦖j ⦙j ⦽j ⧸j ⩔j ⩘j ⫐j ⬁j ⬣j ⬭j ⬯j ⮙j ⮜j Ⱗj Ⱙj ⰶj ⱔj ⱙj ⱱj Ⲏj Ⲙj Ⲧj ⲫj ⴹj ⵕj ⵥj ⸌j ⸍j ⸪j ⸳j ⸸j ⺀j ⺁j ⻦j ⿿j 〘j 〙j 〥j 〱j 〲j 〿j ゕj ゗j ㄪj ㄫj ㄬj ㄰j 㐲j 㑊j 㑮j 㒓j 㓜j 㓡j 㔆j 㔉j 㔷j 㔻j 㕚j 㕸j 㖊j 㖬j 㖽j 㗋j 㗜j 㗭j 㘧j 㘭j 㙓j 㚻j 㞻j 㟭j 㠙j 㠝j 㡜j 㣇j 㥁j 㦰j 㧊j 㨗j 㨪j 㪚j 㫩j 㫪j 㬤j 㬭j 㭁j 㭨j 㮈j 㮋j 㮻j 㲾j 㳸j 㴀j 㴅j 㴬j 㴷j 㵾j 㶽j 㷋j 㹢j 㺅j 㺢j 㺯j 㺸j 㺹j 㻜j 㻬j 㼿j 㽓j 㿂j 㿭j 䀚j 䁢j 䂀j 䅟j 䈀j 䈂j 䉋j 䌸j 䎬j 䎽j 䏻j 䐃j 䐰j 䑓j 䒄j 䕬j 䗚k 䘌k 䙚k 䙡k 䚰k 䜭k 䝟k 䝨k 䟐k 䟕 k 䟛 k 䟦 k 䠠 k 䡇 k 䡛k 䢴k 䥸k 䥾k 䦵k 䨋k 䨱k 䫜k 䬴k 䱇k 䳒k 䴝k 䵋k 䶕k ䷢k ䷬k ䷴k 伕k 倣 k 僉!k 僥"k 儅#k 儉$k 儼%k 剮&k 匱'k 啣(k 嘸)k 噓*k 嚦+k 囈,k 塢-k 奐.k 姍/k 婭0k 媧1k 媯2k 嬸3k 屜4k 峴5k 崳6k 嶄7k 嶇8k 嶸9k 幗:k 廱;k 徠k 慟?k 慳@k 扐Ak 撈Bk 擋Ck 攏Dk 攤Ek 斃Fk 朮Gk 杇Hk 椏Ik 楨Jk 槳Kk 樁Lk 櫝Mk 殯Nk 毆Ok 渙Pk 溼Qk 澆Rk 濺Sk 瀅Tk 瀉Uk 瀋Vk 灑Wk 煢Xk 熾Yk 燜Zk 燬[k 牪\k 犧]k 玆^k 癆_k 癇`k 癟ak 瞇bk 磧ck 礬dk 窪ek 簷fk 絛gk 綢hk 綵ik 緡jk 縈kk 縚lk 繃mk 纍nk 纜ok 罌pk 羆qk 莧rk 葦sk 葯tk 蔦uk 藹vk 蠶wk 袞xk 襠yk 訃zk 詡{k 詼|k 誡}k 誨~k 諄k 謄k 謔k 謚k 譎k 讒k 貲k 賂k 賡k 贗k 趲k 躕k 躦k 軼k 輅k 轟k 辮k 鄲k 酈k 鈀k 鈷k 鈿k 鉦k 鉬k 鉸k 銑k 銖k 銲k 鋰k 鎬k 鐫k 鑿k 閏k 閾k 闃k 顰k 飩k 飴k 饉k 駘k 騏k 騖k 騾k 驄k 驊k 骯k 魎k 鯤k 鯫k 鱉k 鴆k 鴿k 鷳k 鸕k 鸝k 齣k 龨k 龹k ꀀk ꀃk ꀣk ꀨk ꀭk ꀮk ꀲk ꀵk ꁹk ꂃk ꂄk ꂘk ꂮk ꂻk ꂾk ꃆk ꃨk ꃲk ꃸk ꄀk ꄑk ꄜk ꄩk ꄳk ꄴk ꄶk ꄾk ꅀk ꅋk ꆗk ꆚk ꆠk ꆤk ꆨk ꇅk ꇫk ꇺk ꈂk ꈑk ꈞk ꈿk ꉓk ꉖk ꉠk ꉷk ꉺk ꉾk ꊜk ꊨk ꊱk ꊸk ꋋk ꋓk ꋣk ꋬk ꋺk ꌀk ꌑk ꌙk ꌥk ꌷk ꌸk ꍂk ꍈk ꍯk ꍸk ꎁk ꎤk ꎫk ꎸk ꎻk ꎽl ꏑl ꏪl ꏱl ꐓl ꐔl ꐕl ꐱl ꐷl ꐺ l ꐼ l ꑘ l ꑣ l ꑸ l ꒆl ꒈl ꒌl ꒡l ꒶l ꒿l ꕀl ꗯl ꙩl ꙫl ꙭl ꝼl ꞊l ꞔl ꟼl ꡊl ꡓl ꦫl ꦶ l ꦸ!l ꦽ"l ꪶ#l ꭒ$l 갃%l 갛&l 걔'l 겄(l 굵)l 굶*l 깅+l 깇,l 깬-l 깼.l 꾻/l 꿰0l 끓1l 끕2l 냇3l 넋4l 넲5l 넵6l 녘7l 댐8l 덱9l 뎅:l 돔;l 됬l 땠?l 뗄@l 뜃Al 뜬Bl 뜯Cl 뜸Dl 랗El 랴Fl 럿Gl 롸Hl 룻Il 뤃Jl 륵Kl 맏Ll 먃Ml 묶Nl 뮈Ol 밋Pl 벵Ql 벹Rl 볕Sl 봃Tl 빕Ul 빤Vl 빽Wl 뺑Xl 뻤Yl 뻥Zl 삗[l 삾\l 샬]l 샴^l 섞_l 셑`l 셪al 솽bl 숏cl 숑dl 쉰el 슉fl 슌gl 슐hl 싯il 쌔jl 썽kl 쒼ll 씽ml 엎nl 옴ol 욥pl 윷ql 잣rl 젡sl 쥔tl 쥘ul 쥴vl 쨍wl 쩳xl 쪘yl 쫀zl 찡{l 찻|l 챌}l 챱~l 첼l 쳄l 촛l 츨l 츰l 칫l 캉l 켠l 콥l 콧l 콸l 쿤l 퀌l 퀵l 텨l 툼l 튠l 튤l 팸l 팽l 핌l 핏l 핚l 핳l 핼l 헙l 헯l 헸l 헺l 혆l 홰l 훑l 훗l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l m m m m m m m m m  m  m  m  m  m m m m m m m m m m m m m m m m m m m  m !m "m #m $m %m &m 'm (m )m *m +m ,m -m .m /m 0m 1m 2m 3m 4m 5m 6m 7m 8m 9m :m ;m m ?m @m Am Bm Cm Dm Em Fm Gm Hm Im Jm Km Lm Mm Nm Om Pm Qm Rm Sm Tm Um Vm Wm Xm Ym Zm [m \m ]m ^m _m `m am bm cm dm em fm gm hm im jm km lm mm nm om pm qm rm sm tm um vm wm xm ym zm {m |m }m ~m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m n n n n ︢n ︣n ﹆n 𐀷n 𐄙n 𐊧 n 𐍉 n 𐐌 n 𐐐 n 𐐤 n 𐤁n 𐤂n 𐤊n 𐤍n 𐤏n 𐬞n 𐭓n 𑀓n 𑄋n 𑄞n 𑄟n 𑄦n 𑄳n 𑐏n 𑐔n 𑐫n 𑐴n 𑐾n 𑑁 n 𒂍!n 𒂗"n 𒄀#n 𒆜$n 𒊏%n 𒋢&n 𒍪'n 𓂆(n 𓃹)n 𓅓*n 𓅰+n 𓆊,n 𓆌-n 𓆏.n 𓍯/n 𖤣0n 𖥣1n 𖦞2n 𖧷3n 🀀4n 🀁5n 🀂6n 🀃7n 🀄8n 🀅9n 🀇:n 🀈;n 🀉n 🀌?n 🀍@n 🀎An 🀏Bn 🀐Cn 🀑Dn 🀒En 🀓Fn 🀔Gn 🀕Hn 🀖In 🀗Jn 🀘Kn 🀙Ln 🀚Mn 🅕Nn 🅷On 🅹Pn 🆒Qn 🆔Rn 🌁Sn 🌜Tn 🍢Un 🍥Vn 🎛Wn 👚Xn 👢Yn 👳Zn 💠[n 💹\n 📤]n 🔂^n 🖉_n 🚅`n 🚉an 🚊bn 🚝cn 🚥dn 🚭en 🚽fn 🚿gn 🛷hn 🟧in 🟪jn 🠊kn 🠨ln 🢂mn 🤿nn 🥏on 🥠pn 🥷qn 🥽rn 🦝sn 🦟tn 🦰un 🦱vn 🦲wn 🦽xn 🧰yn 🧲zn 🧷{n 🪀|n 🪙}n 🪟~n 🪣n 🪩n 🪴n 🫓n 𠂉n 𠃍n 𠃵n 𠄘n 𠔽n 𠩤n 𠶧n 𡎚n 𡘊n 𡿨n 𢭃n 𢳂n 𤦺n 𤫊n 𥚃n 𦦨n 𦧲n 𧒽n 𧙕n 𨂃n 𪚥n 𪟝n 𬇕n 󰀀n 󱐂n 􏰠n ƃn Ʋn Ǖn Ǘn Ǥn Ǭn Ƕn Ȅn Ȍn Ȕn Ȝn Ƞn ȡn Ȭn ȴn ȸn ʢn ʫn ʬn ˫n ˯n ˿n ͱn Ϳn ΍n Ϣn Ϸn Ѫn ѻn ҃n ҅n ҍn ҏn Ҵn Ҹn ҹn Ӊn Ӡn ӡn Ӯn Ӽn ԇn Ԙn ԣn ԭn ԯn ՠn ֍n ֘n ֟n ֪n ׈n ׌n ׏n ׭n ׺n ׽n ؁n ؇n ؈n ؉n ؓn ؖn ؞n ؿn ٙn ٟn ٯn ٳn ٺn ڇn ڋn ڎn ڜn ڝn ڞn ۨn ۼn ܉n ܱn ܲn ݀n ݌n ݕn ޛn ޣn ޴n ޹n ޺n ޿n ߅n ߓn ߔn ߙn ߚn ߛn ߟn ߡn ߢn ߥo ߫o ߱o ߴo ߷o ߸o ߿o ࠀo ࠐo ࠕ o ࠞ o ࠤ o ࠬ o ࠳ o ࠴o ࠶o ࡏo ࡩo ࡵo ࡹo ࢃo ࢈o ࢌo ࢝o ࢥo ࢩo ࢲo ࢳo ࢸo ࣙo ࣟo ࣡o ࣪ o ࣬!o ࣽ"o ॓#o ॔$o ॠ%o ॾ&o ঌ'o ঩(o ঵)o ৄ*o ৅+o ৡ,o ৽-o ਋.o ਴/o ੄0o ੖1o ੦2o ੷3o ੼4o ੽5o ઍ6o ઎7o ઔ8o ઙ9o ઻:o ઽ;o ૐo ૤?o ૱@o ଋAo ଔBo ଺Co ୅Do ୗEo ୛Fo ୲Go ୴Ho ୹Io ஋Jo ஔKo ஬Lo ஭Mo ௌNo ௎Oo ௱Po ௹Qo ౗Ro ౙSo ౢTo ౴Uo ౺Vo ಊWo ೚Xo ഊYo ഛZo ഼[o ൉\o ൐]o ൛^o ൜_o ൪`o ൳ao ൹bo ඓco ඲do ඵeo ෈fo ෬go ෷ho ෻io ฾jo ๜ko ๟lo ๵mo ๼no ๽oo ຬpo ໋qo ໑ro ໕so ໯to ཭uo ྌvo ྍwo ྴxo ྸyo ྻzo ࿀{o ࿁|o ࿄}o ࿇~o ࿏o ࿔o ࿕o ࿱o ࿷o ࿸o ဎo ၈o ၞo ၠo ၦo ၱo ၲo ၵo ၶo ၼo ႁo ႑o Ⴂo Ⴛo Ⴝo Ⴥo ჲo ჶo ჻o ᄡo ᄣo ᄬo ᄮo ᅃo ᅊo ᅌo ᅎo ᅻo ᆄo ᆓo ᆝo ᆡo ᆬo ᆷo ᇗo ᇚo ᇩo ᇺo ሢo ሺo ሼo ቑo ቖo ቘo ቝo ቧo ቬo ቺo ቾo ኀo ኆo ኏o ኚo ኧo ኲo ኹo ኼo ኾo ዆o ዪo ዷo ጄo ጢo ጯo ጲo ፑo ፔo ፥o ፨o ፬o ፭o ፱o ፳o ፺o ፻o ᎖o Ꮊo Ꮕo Ꮜo Ꮡo Ꮴo Ꮵo ᏽo ᐍo ᐑo ᐗo ᐘo ᐜo ᐡo ᑂo ᑈo ᑏo ᑞo ᑡo ᑤo ᒀo ᒆo ᒩo ᒷo ᓋo ᓘo ᓙo ᓭo ᓮo ᔐo ᔓo ᔦo ᔫo ᔲo ᕀo ᕉo ᕋo ᕌo ᕒo ᕔo ᕖo ᕝo ᕪo ᕮo ᕰo ᕳo ᕿo ᖄo ᖔp ᖕp ᖽp ᖾp ᗋp ᗎp ᗒp ᗔp ᗕp ᗧ p ᗷ p ᗺ p ᘃ p ᘐ p ᘜp ᘰp ᘳp ᘺp ᙅp ᙓp ᙜp ᙢp ᙮p ᙿp ᚔp ᚕp ᚮp ᚲp ᚿp ᛂp ᛃp ᛇp ᛕ p ᛠ!p ᛡ"p ᛭#p ᜀ$p ᜁ%p ᜋ&p ᜗'p ᜴(p ᝎ)p ᝡ*p ឍ+p ឲ,p ៚-p ៩.p ៴/p ៹0p ᠼ1p ᡢ2p ᡤ3p ᡳ4p ᡴ5p ᡻6p ᢚ7p ᣑ8p ᤈ9p ᤤ:p ᤿;p ᥉

    p ᥷?p ᦇ@p ᦒAp ᦮Bp ᧥Cp ᨆDp ᨊEp ᨔFp ᨗGp ᨚHp ᨜Ip ᩓJp ᪋Kp ᫂Lp ᫹Mp ᬅNp ᬘOp ᬦPp ᬪQp ᬰRp ᬲSp ᬳTp ᬵUp ᭨Vp ᮟWp ᯖXp ᯮYp ᯲Zp ᯿[p ᰍ\p ᱅]p ᱼ^p Ი_p Ს`p Უap Შbp Ცcp Ჳdp ᳪep ᴝfp ᴨgp ᵭhp ᵲip ᵿjp ᶃkp ᶌlp ᶐmp ᷝnp ᷾op ᷿pp Ḉqp Ḓrp Ḕsp Ḭtp Ḯup ḯvp Ḽwp Ḿxp Ṅyp Ṋzp Ṏ{p ṏ|p Ṙ}p Ṝ~p ṝp Ṧp Ṩp ṵp ṻp Ẁp Ẅp Ẋp Ẍp Ằp Ỉp Ỡp Ỵp Ỷp ἟p ἣp Ἶp Ὂp ὒp ὓp ὧp Ὥp Ὦp ᾁp ᾅp ᾈp ᾍp ᾦp ῧp ῰p ῲp ⁅p ⁌p ⁔p ⁘p p p ⁲p ₠p ₥p ₼p ⃍p ⃟p ℈p ↜p ↱p ↸p ↽p ⇁p ⇶p ⇸p ∢p ∦p ∹p ∾p ≁p ≄p ≉p ≎p ≏p ≔p ≘p ≛p ≜p ≝p ≞p ≭p ≰p ≸p ⊀p ⊁p ⊈p ⊉p ⊍p ⊓p ⊣p ⊯p ⊲p ⊸p ⊺p ⊻p ⊽p ⋇p ⋋p ⋌p ⋎p ⋐p ⋑p ⋔p ⋡p ⋦p ⋧p ⋪p ⋭p ⌅p ⌌p ⌍p ⌎p ⌏p ⌖p ⌦p ⌧p ⌲p ⍘p ⍣p ⍯p ⍹p ⎊p ⎌p ⎘p ⎲p ⎸p ⎹p ⏀p ⏆p ⏣p ⏦p ⏴p ⏵p ␕p ␝p ␠p ␴p ␾p ⑂p ⑊p ⓹p ┩p ┰p ┱q ┸q ┺q ┽q ┿q ╀q ╁q ╃q ╓q ╙ q ▗ q ▛ q ◛ q ◸ q ◹q ◺q ☖q ☗q ☡q ☨q ☫q ♁q ♲q ♳q ♺q ♼q ⚂q ⚉q ⚞q ⚟q ⚢q ⚤q ⚥ q ⚦!q ⚯"q ⚳#q ⚷$q ⚿%q ⛆&q ⛉'q ⛭(q ⛶)q ✣*q ❴+q ❵,q ⟟-q ⟴.q ⟿/q ⠅0q ⠉1q ⠌2q ⠍3q ⠐4q ⠖5q ⠘6q ⠜7q ⠡8q ⠢9q ⠣:q ⠤;q ⠨q ⠬?q ⠭@q ⠮Aq ⠰Bq ⠲Cq ⠶Dq ⠺Eq ⠻Fq ⠽Gq ⠿Hq ⡷Iq ⢠Jq ⢾Kq ⣎Lq ⣸Mq ⣻Nq ⣼Oq ⤃Pq ⤊Qq ⤎Rq ⤐Sq ⤒Tq ⤔Uq ⤥Vq ⤧Wq ⤶Xq ⥇Yq ⥮Zq ⦊[q ⦥\q ⧍]q ⧎^q ⧚_q ⧧`q ⧹aq ⨀bq ⨂cq ⨺dq ⩓eq ⪢fq ⫌gq ⫫hq ⬄iq ⬉jq ⬍kq ⬏lq ⬑mq ⬓nq ⬔oq ⬕pq ⬖qq ⬗rq ⬙sq ⬚tq ⬞uq ⬟vq ⬠wq ⬦xq ⬧yq ⬨zq ⬪{q ⬫|q ⬮}q ⬰~q ⬱q ⬲q ⬳q ⬴q ⬵q ⬶q ⬷q ⬸q ⬹q ⬺q ⬻q ⬼q ⬽q ⬾q ⭒q ⭳q ⮌q ⮑q ⮚q ⮝q ⮡q ⮢q ⮤q ⮱q ⯆q ⯇q Ⰲq Ⰽq Ⰾq Ⱀq Ⱁq Ⱏq Ⱐq Ⱑq Ⱔq Ⱖq Ᵽq ⱥq ⱦq Ⱪq Ɑq ⱴq Ⲩq ⳷q ⴠq ⴢq ⴭq ⴲq ⴸq ⴽq ⴾq ⵀq ⵁq ⵛq ⵦq ⵳q ⸆q ⸉q ⸴q ⸻q ⹂q ⹴q ⺇q ⺗q ⺧q ⺴q ⺶q ⻻q ⿲q ⿳q ⿵q ⿶q ⿹q 〟q 〴q ㄀q ㄯq ㆪq ㆳq ㆺq ㆼq ㆽq ㇀q ㇑q ㇔q ㇕q ㇷq ㉿q 㐀q 㐊q 㐦q 㑀q 㑛q 㑦q 㑨q 㑬q 㑰q 㑳q 㒞q 㒷q 㓋q 㓘q 㓛q 㓟q 㓲q 㔐q 㔓q 㔴q 㕃q 㕊q 㕋q 㕕q 㕜q 㕡q 㕵q 㖀q 㖏q 㖓q 㗂q 㗅q 㗉q 㗊q 㗏q 㗙q 㗰q 㗷q 㘐q 㘓q 㘝q 㘞r 㙶r 㚢r 㚣r 㚭r 㚵r 㚿r 㜀r 㜠r 㜷 r 㜸 r 㝅 r 㝢 r 㝱 r 㞓r 㞘r 㞢r 㞮r 㞳r 㞹r 㟂r 㟷r 㟺r 㠀r 㠊r 㠰r 㢏r 㢡r 㢢r 㢧r 㢮r 㤅r 㤉 r 㤕!r 㤙"r 㤚#r 㤞$r 㤴%r 㤵&r 㥐'r 㥘(r 㥬)r 㦀*r 㦣+r 㦤,r 㦮-r 㦯.r 㧍/r 㧏0r 㧐1r 㧑2r 㧴3r 㧵4r 㧽5r 㨆6r 㨉7r 㨖8r 㨤9r 㩁:r 㩂;r 㩉r 㩝?r 㩠@r 㩦Ar 㪍Br 㪰Cr 㪶Dr 㫬Er 㫺Fr 㬌Gr 㬎Hr 㬪Ir 㬹Jr 㬿Kr 㭇Lr 㭍Mr 㭜Nr 㮇Or 㮐Pr 㮟Qr 㮠Rr 㮣Sr 㮮Tr 㮵Ur 㮷Vr 㮹Wr 㮽Xr 㯋Yr 㯖Zr 㯠[r 㰖\r 㰞]r 㱎^r 㱽_r 㱿`r 㲠ar 㲪br 㲺cr 㳀dr 㳁er 㳕fr 㳠gr 㴇hr 㴔ir 㴪jr 㴭kr 㴰lr 㵋mr 㵖nr 㵟or 㵮pr 㵵qr 㵿rr 㶉sr 㶏tr 㶤ur 㶰vr 㶲wr 㶷xr 㷆yr 㷧zr 㷬{r 㷻|r 㸐}r 㸢~r 㸦r 㸿r 㹃r 㹌r 㹦r 㹧r 㹭r 㻆r 㼜r 㼭r 㼼r 㽬r 㾕r 㾗r 㾫r 㾹r 㿃r 䀪r 䀲r 䀾r 䀿r 䁊r 䁐r 䂓r 䃄r 䃥r 䃧r 䄃r 䄍r 䄦r 䄪r 䅲r 䆉r 䆖r 䆡r 䆲r 䆳r 䇖r 䇹r 䉏r 䉕r 䉛r 䊎r 䊨r 䋎r 䋣r 䋧r 䋲r 䋸r 䋾r 䌀r 䌇r 䌊r 䌠r 䌹r 䍃r 䍘r 䍦r 䍩r 䍺r 䎱r 䏂r 䏄r 䏈r 䏓r 䏚r 䏮r 䏸r 䐠r 䐢r 䐯r 䐱r 䐳r 䑉r 䑔r 䒔r 䒗r 䒥r 䓁r 䓢r 䓴r 䔄r 䔰r 䕌r 䕒r 䕰r 䕶r 䕷r 䖘r 䖝r 䖪r 䗖r 䗛r 䘿r 䙓r 䚧r 䛁r 䛒r 䛝r 䜆r 䜉r 䝁r 䝗r 䝙r 䝷r 䞼r 䟉r 䟒r 䠌r 䠙r 䠩r 䠰r 䠷r 䢋r 䢎r 䢒r 䣈r 䣋r 䣒r 䣔r 䤐r 䤠r 䤥r 䤹r 䥌r 䥚r 䥱r 䧚r 䨍r 䨧s 䨼s 䩄s 䩮s 䩶s 䪨s 䪽s 䬝s 䬰s 䬻 s 䭔 s 䭺 s 䭾 s 䮭 s 䯙s 䯝s 䰀s 䰅s 䰖s 䰫s 䱂s 䱍s 䱒s 䲞s 䳍s 䳕s 䳗s 䳘s 䳨s 䴕s 䴖s 䴚s 䵈 s 䵔!s 䵟"s 䵵#s 䵻$s 䶩%s ䷊&s ䷚'s ䷝(s ䷮)s ䷯*s 亙+s 儂,s 儔-s 儕.s 儸/s 儺0s 冪1s 剄2s 剷3s 勦4s 卹5s 嗆6s 嗇7s 嗩8s 嗶9s 噯:s 噸;s 嚀s 墑?s 壚@s 壟As 壩Bs 嫻Cs 嬝Ds 嬤Es 嬪Fs 嶧Gs 巒Hs 幃Is 幘Js 幬Ks 廄Ls 廡Ms 悵Ns 慇Os 憒Ps 憫Qs 懌Rs 懞Ss 懣Ts 戇Us 戧Vs 扠Ws 搆Xs 摳Ys 撣Zs 撻[s 撾\s 擣]s 擲^s 攣_s 檣`s 櫥as 櫨bs 欞cs 殫ds 氈es 氬fs 氳gs 浹hs 淥is 漚js 潁ks 潯ls 潷ms 澇ns 瀝os 熗ps 燁qs 燴rs 犢ss 獮ts 獺us 璦vs 瓔ws 瓚xs 痙ys 痳zs 瘓{s 瘧|s 癉}s 癭~s 癰s 眥s 矯s 礪s 礫s 秈s 稈s 箄s 篳s 簍s 簑s 簣s 籐s 糝s 糴s 紆s 紱s 綹s 綽s 緘s 緝s 緱s 縊s 縵s 縷s 縹s 罈s 羥s 脛s 臍s 臏s 萇s 葷s 蓀s 蓽s 蔆s 蕕s 蕘s 蕷s 蘄s 蘗s 褸s 襖s 覘s 覯s 覲s 覿s 訕s 訛s 訥s 詎s 詒s 詘s 詫s 詬s 詵s 誅s 誚s 誹s 諂s 諍s 諛s 諞s 謗s 謠s 謬s 譙s 讎s 貰s 賄s 賅s 賒s 賾s 贄s 踡s 踴s 蹌s 蹕s 蹺s 躋s 軫s 軻s 輓s 輜s 輳s 輾s 轂s 轎s 轤s 邇s 醃s 醞s 釁s 釧s 釬s 鈹s 鉀s 鉍s 鉚s 銬s 銻s 鋃s 鋅s 鋮s 鋯s 錆s 錙s 錮s 鍥s 鎘s 鎦s 鎳s 鏑s 鏞s 鐺s 鑠s 閂s 閔s 閡s 閹s 闈s 闌s 阨s 陝s 隄s 隕s 靨s 鞏s 韃s 韆t 頷t 顓t 顥t 顱t 颼t 飆t 餌t 餛t 餾 t 餿 t 饈 t 饑 t 騁 t 騫t 騮t 騶t 驀t 驤t 驪t 鬍t 鬨t 鬮t 魷t 鮐t 鮪t 鯀t 鯪t 鯽t 鰾t 鱈t 鱒t 鱟 t 鱷!t 鳧"t 鴟#t 鴣$t 鵠%t 鶉&t 鶚't 鷓(t 鷥)t 鷴*t 鷸+t 鸛,t 鼇-t 鼉.t 龧/t 龷0t ꀈ1t ꀥ2t ꀱ3t ꀹ4t ꀺ5t ꁈ6t ꁉ7t ꁊ8t ꁍ9t ꁙ:t ꁞ;t ꁠt ꁵ?t ꂊ@t ꂐAt ꂒBt ꂪCt ꂯDt ꂴEt ꂶFt ꃘGt ꃤHt ꃮIt ꃳJt ꃺKt ꄁLt ꄓMt ꄘNt ꄙOt ꄞPt ꄠQt ꄵRt ꄺSt ꄻTt ꄿUt ꅈVt ꅊWt ꅒXt ꅰYt ꅶZt ꅾ[t ꆣ\t ꇋ]t ꇎ^t ꇕ_t ꇖ`t ꇩat ꇳbt ꈅct ꈏdt ꈬet ꈭft ꈯgt ꈳht ꈹit ꈺjt ꈽkt ꉁlt ꉅmt ꉏnt ꉗot ꉻpt ꊁqt ꊛrt ꊥst ꊬtt ꊷut ꋂvt ꋙwt ꋛxt ꋧyt ꋴzt ꌁ{t ꌅ|t ꌇ}t ꌈ~t ꌕt ꌜt ꌢt ꌭt ꌶt ꍣt ꍤt ꍧt ꍿt ꎍt ꎖt ꎮt ꏠt ꏡt ꏣt ꏳt ꐁt ꐈt ꐉt ꐋt ꐟt ꐣt ꑑt ꑖt ꑝt ꑤt ꑶt ꑼt ꒊt ꒙t ꓃t ꓥt ꗄt ꙁt ꙑt Ꙗt Ꙭt ꙮt Ꚋt ꚍt ꜛt Ꜫt ꜫt ꝑt Ꝛt ꝫt ꞇt ꞈt ꞎt ꞓt ꞵt Ꞔt Ʂt ꟕt ꟿt ꠵t ꡛt ꡟt ꦂt ꦏt ꦒt ꦗt ꦚt ꦤt ꦧt ꦩt ꦪt ꦮt ꦲt ꧒t ꩒t ꩜t ꪀt ꪅt ꪑt ꪖt ꪜt ꪣt ꪤt ꪰt ꬶt ꭢt ꯇt ꯟt ꯣt ꯤt 갬t 걀t 겋t 겔t 곶t 괌t 굼t 귓t 긱t 깡t 깥t 깸t 껀t 껄t 껑t 껬t 꽝t 꽸t 꿉t 꿎t 꿘t 뀀t 뀛t 뀜t 뀠t 뀽t 끉t 끽t 낍t 낱t 냘t 냬t 넀t 넙t 넳t 넸t 넹t 넼t 녔t 놯t 뇛t 눇t 눴t 뉘u 뉜u 늪u 닋u 댁u 돐u 돗u 돛u 됀u 됍 u 둣 u 둬 u 딊 u 딻 u 땋u 떄u 떳u 떴u 똈u 뚤u 뛴u 뜩u 띤u 랏u 렙u 렝u 뤘u 맠u 맬u 멓u 멥u 몄u 몏 u 묾!u 뭩"u 뮨#u 밫$u 뱁%u 봣&u 뵐'u 뵜(u 뵵)u 뺐*u 뼘+u 뿍,u 뿔-u 뿚.u 쀼/u 삔0u 샽1u 샾2u 섶3u 셶4u 솓5u 숄6u 숴7u 쉔8u 쉘9u 슭:u 쌕;u 썹u 쏜?u 쏴@u 쐬Au 쐼Bu 쑹Cu 쓡Du 씌Eu 씰Fu 앍Gu 얌Hu 얽Iu 엥Ju 엮Ku 엷Lu 옭Mu 옻Nu 왈Ou 왑Pu 욃Qu 욧Ru 욯Su 웍Tu 윔Uu 잦Vu 잴Wu 쟤Xu 젓Yu 젬Zu 좆[u 죗\u 죤]u 줏^u 짊_u 짖`u 짰au 쫄bu 쫑cu 쬐du 쬬eu 쭥fu 쮸gu 찹hu 찼iu 챈ju 첟ku 쳇lu 쳔mu 쳮nu 촘ou 촣pu 촨qu 쵸ru 춧su 췄tu 췌uu 캅vu 캤wu 캥xu 컫yu 켈zu 콰{u 콱|u 쾨}u 쿡~u 큘u 탰u 텅u 텟u 텼u 툭u 퉌u 튈u 틋u 팍u 팎u 팜u 팠u 팹u 퍽u 펍u 펐u 펨u 펩u 폈u 퐀u 퐄u 퐆u 퐈u 퐋u 퐑u 퐻u 퐾u 푒u 푗u 푚u 푛u 픕u 햅u 햬u 헕u 헱u 헾u 혠u 홧u 횞u 휋u 힉u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u v v v v v v v v v  v  v  v  v  v v v v v v v v v v v v v v v v v v v  v !v "v #v $v %v &v 'v (v )v *v +v ,v -v .v /v 0v 1v 2v 3v 4v 5v 6v 7v 8v 9v :v ;v v ?v @v Av Bv Cv Dv Ev Fv Gv Hv Iv Jv Kv Lv Mv Nv Ov Pv Qv Rv Sv Tv Uv Vv Wv Xv Yv Zv [v \v ]v ^v _v `v av bv cv dv ev fv gv hv iv jv kv lv mv nv ov pv qv rv sv tv uv vv wv xv yv zv {v |v }v ~v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v w w w w w w w w w  w  w  w  w  w w w w w w w w w w w w w w w w w w wtokenizer.model U?XÓ_ 

    

    

      
    
    
  • 
  • 
    • 

    

    

    
    
    
    

    

    

    
    
    
    

    

    

    

    

    

    ０１２３４５６７８９+-=，。！？、：￥.!?... 。。。。。。。。。《》【】『』```---                                                                                                      identity pengcheng.mind-mindformer/scripts/hf_2B/tokenizer_config.json000066400000000000000000000014331476650141400247540ustar00rootroot00000000000000{ "auto_map": { "AutoTokenizer": ["tokenization_baichuan.BaichuanTokenizer", null] }, "add_bos_token": false, "add_eos_token": false, "use_fast": false, "clean_up_tokenization_spaces": false, "eos_token": { "__type": "AddedToken", "content": "    ", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true }, "model_max_length": 3072, "sp_model_kwargs": {}, "tokenizer_class": "BaichuanTokenizer", "pad_token": { "__type": "AddedToken", "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true }, "unk_token": { "__type": "AddedToken", "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true } } pengcheng.mind-mindformer/scripts/hf_7B_shared/000077500000000000000000000000001476650141400220545ustar00rootroot00000000000000pengcheng.mind-mindformer/scripts/hf_7B_shared/added_tokens.json000066400000000000000000000000261476650141400253710ustar00rootroot00000000000000{ "": 125696 } pengcheng.mind-mindformer/scripts/hf_7B_shared/config.json000066400000000000000000000013701476650141400242150ustar00rootroot00000000000000{ "architectures": [ "BaichuanForCausalLM" ], "auto_map": { "AutoConfig": "configuration_baichuan.BaichuanConfig", "AutoModelForCausalLM": "modeling_baichuan.BaichuanForCausalLM" }, "bos_token_id": 1, "eos_token_id": 2, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.013, "intermediate_size": 11008, "max_position_embeddings": 4096, "model_max_length": 4096, "model_type": "baichuan", "num_attention_heads": 32, "num_hidden_layers": 32, "pad_token_id": 0, "rms_norm_eps": 1e-05, "depth_scale": 1.0, "embed_scale": 1.0, "_from_model_config": true, "tie_word_embeddings": false, "torch_dtype": "float16", "transformers_version": "4.29.2", "use_cache": true, "vocab_size": 125696 } pengcheng.mind-mindformer/scripts/hf_7B_shared/configuration_baichuan.py000066400000000000000000000047201476650141400271320ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from transformers.configuration_utils import PretrainedConfig from transformers.utils import logging logger = logging.get_logger(__name__) class BaichuanConfig(PretrainedConfig): model_type = "baichuan" keys_to_ignore_at_inference = ["past_key_values"] def __init__( self, vocab_size=125696, hidden_size=4096, intermediate_size=11008, num_hidden_layers=32, num_attention_heads=32, hidden_act="silu", max_position_embeddings=4096, initializer_range=0.02, depth_scale=1.0, embed_scale=1.0, rms_norm_eps=1e-6, use_cache=True, pad_token_id=0, bos_token_id=1, eos_token_id=2, tie_word_embeddings=False, **kwargs, ): self.vocab_size = vocab_size self.max_position_embeddings = max_position_embeddings self.hidden_size = hidden_size self.intermediate_size = intermediate_size self.num_hidden_layers = num_hidden_layers self.num_attention_heads = num_attention_heads self.hidden_act = hidden_act self.initializer_range = initializer_range self.depth_scale = depth_scale self.embed_scale = embed_scale self.rms_norm_eps = rms_norm_eps self.use_cache = use_cache super().__init__( pad_token_id=pad_token_id, bos_token_id=bos_token_id, eos_token_id=eos_token_id, tie_word_embeddings=tie_word_embeddings, **kwargs, ) pengcheng.mind-mindformer/scripts/hf_7B_shared/generation_config.json000066400000000000000000000004351476650141400264310ustar00rootroot00000000000000{ "pad_token_id": 0, "bos_token_id": 1, "eos_token_id": 2, "user_token_id": 195, "assistant_token_id": 196, "max_new_tokens": 2048, "temperature": 0.3, "top_k": 5, "top_p": 0.85, "repetition_penalty": 1.05, "do_sample": true, "transformers_version": "4.29.2" } pengcheng.mind-mindformer/scripts/hf_7B_shared/generation_utils.py000066400000000000000000000056261476650141400260120ustar00rootroot00000000000000from typing import List from queue import Queue import torch def build_chat_input(model, tokenizer, messages: List[dict], max_new_tokens: int=0): def _parse_messages(messages, split_role="user"): system, rounds = "", [] round = [] for i, message in enumerate(messages): if message["role"] == "system": assert i == 0 system = message["content"] continue if message["role"] == split_role and round: rounds.append(round) round = [] round.append(message) if round: rounds.append(round) return system, rounds max_new_tokens = max_new_tokens or model.generation_config.max_new_tokens max_input_tokens = model.config.model_max_length - max_new_tokens system, rounds = _parse_messages(messages, split_role="user") system_tokens = tokenizer.encode(system) max_history_tokens = max_input_tokens - len(system_tokens) history_tokens = [] for round in rounds[::-1]: round_tokens = [] for message in round: if message["role"] == "user": round_tokens.append(model.generation_config.user_token_id) else: round_tokens.append(model.generation_config.assistant_token_id) round_tokens.extend(tokenizer.encode(message["content"])) if len(history_tokens) == 0 or len(history_tokens) + len(round_tokens) <= max_history_tokens: history_tokens = round_tokens + history_tokens # concat left if len(history_tokens) < max_history_tokens: continue break input_tokens = system_tokens + history_tokens if messages[-1]["role"] != "assistant": input_tokens.append(model.generation_config.assistant_token_id) input_tokens = input_tokens[-max_input_tokens:] # truncate left return torch.LongTensor([input_tokens]).to(model.device) class TextIterStreamer: def __init__(self, tokenizer, skip_prompt=False, skip_special_tokens=False): self.tokenizer = tokenizer self.skip_prompt = skip_prompt self.skip_special_tokens = skip_special_tokens self.tokens = [] self.text_queue = Queue() self.next_tokens_are_prompt = True def put(self, value): if self.skip_prompt and self.next_tokens_are_prompt: self.next_tokens_are_prompt = False else: if len(value.shape) > 1: value = value[0] self.tokens.extend(value.tolist()) self.text_queue.put( self.tokenizer.decode(self.tokens, skip_special_tokens=self.skip_special_tokens)) def end(self): self.text_queue.put(None) def __iter__(self): return self def __next__(self): value = self.text_queue.get() if value is None: raise StopIteration() else: return value pengcheng.mind-mindformer/scripts/hf_7B_shared/modeling_baichuan.py000066400000000000000000000714321476650141400260650ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. from .configuration_baichuan import BaichuanConfig from .generation_utils import build_chat_input, TextIterStreamer import math from threading import Thread from typing import List, Optional, Tuple, Union import torch from torch import nn from torch.nn import CrossEntropyLoss from torch.nn import functional as F from transformers import PreTrainedModel from transformers.activations import ACT2FN from transformers.generation.utils import GenerationConfig from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast from transformers.utils import logging logger = logging.get_logger(__name__) try: from xformers import ops as xops except ImportError: xops = None logger.warning( "Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers\npip install xformers." ) # Copied from transformers.models.bart.modeling_bart._make_causal_mask def _make_causal_mask( input_ids_shape: torch.Size, dtype: torch.dtype, device: torch.device, past_key_values_length: int = 0 ): """ Make causal mask used for bi-directional self-attention. """ bsz, tgt_len = input_ids_shape mask = torch.full((tgt_len, tgt_len), torch.tensor(torch.finfo(dtype).min, device=device), device=device) mask_cond = torch.arange(mask.size(-1), device=device) mask.masked_fill_(mask_cond < (mask_cond + 1).view(mask.size(-1), 1), 0) mask = mask.to(dtype) if past_key_values_length > 0: mask = torch.cat([torch.zeros(tgt_len, past_key_values_length, dtype=dtype, device=device), mask], dim=-1) return mask[None, None, :, :].expand(bsz, 1, tgt_len, tgt_len + past_key_values_length) def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None): """ Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`. """ if len(mask.size()) == 3: bsz, src_len, _ = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:,None,:,:].expand(bsz, 1, tgt_len, src_len).to(dtype) else: bsz, src_len = mask.size() tgt_len = tgt_len if tgt_len is not None else src_len expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype) inverted_mask = 1.0 - expanded_mask return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min) class RMSNorm(nn.Module): def __init__(self, hidden_size, eps=1e-6): """ RMSNorm is equivalent to T5LayerNorm """ super().__init__() self.weight = nn.Parameter(torch.ones(hidden_size)) self.variance_epsilon = eps def forward(self, hidden_states): variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True) hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon) # convert into half-precision if necessary if self.weight.dtype in [torch.float16, torch.bfloat16]: hidden_states = hidden_states.to(self.weight.dtype) return self.weight * hidden_states class RotaryEmbedding(torch.nn.Module): def __init__(self, dim, max_position_embeddings=2048, base=1e4, device=None): super().__init__() self.inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float().to(device) / dim)) self.max_seq_len_cached = max_position_embeddings t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=torch.float32) freqs = torch.outer(t, self.inv_freq) emb = torch.cat((freqs, freqs), dim=-1) self.cos_cached = emb.cos()[None, None, :, :].to(torch.float32) self.sin_cached = emb.sin()[None, None, :, :].to(torch.float32) def forward(self, x, seq_len=None): # x: [bs, num_attention_heads, seq_len, head_size] # This `if` block is unlikely to be run after we build sin/cos in `__init__`. Keep the logic here just in case. if seq_len > self.max_seq_len_cached: self.max_seq_len_cached = seq_len t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=torch.float32) freqs = torch.outer(t, self.inv_freq) emb = torch.cat((freqs, freqs), dim=-1) self.cos_cached = emb.cos()[None, None, :, :].to(torch.float32).to(x.device) self.sin_cached = emb.sin()[None, None, :, :].to(torch.float32).to(x.device) return ( self.cos_cached[:, :, :seq_len, ...].to(torch.float32).to(x.device), self.sin_cached[:, :, :seq_len, ...].to(torch.float32).to(x.device), ) def rotate_half(x): """Rotates half the hidden dims of the input.""" x1 = x[..., : x.shape[-1] // 2] x2 = x[..., x.shape[-1] // 2:] return torch.cat((-x2, x1), dim=-1) def apply_rotary_pos_emb(q, k, cos_, sin_, position_ids): cos = cos_.squeeze(1).squeeze(0) # [seq_len, dim] sin = sin_.squeeze(1).squeeze(0) # [seq_len, dim] cos = cos[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] sin = sin[position_ids].unsqueeze(1) # [bs, 1, seq_len, dim] q_embed = (q.float() * cos) + (rotate_half(q.float()) * sin) k_embed = (k.float() * cos) + (rotate_half(k.float()) * sin) return q_embed.to(q.dtype), k_embed.to(k.dtype) class MLP(nn.Module): def __init__( self, hidden_size: int, intermediate_size: int, hidden_act: str, ): super().__init__() self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False) self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False) self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False) # self.act_fn = ACT2FN[hidden_act] self.act_fn = F.silu def forward(self, x): # print(f"self.act_fn: {self.act_fn}") # print(f"input: {x.float()[0][0].cpu().detach().detach().numpy()}, {x.shape}") output = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x)) # print(f"gate_proj: {self.gate_proj(x).float()[0][0].cpu().detach().numpy()}, {self.gate_proj(x).shape}") # print(f"self.act_fn(self.gate_proj(x): {self.act_fn(self.gate_proj(x)).float()[0][0].cpu().detach().numpy()}, {self.act_fn(self.gate_proj(x)).shape}") # print(f"up_proj: {self.up_proj(x).float()[0][0].cpu().detach().numpy()}, {self.up_proj(x).shape}") # print(f"self.act_fn(self.gate_proj(x)) * self.up_proj(x): {(self.act_fn(self.gate_proj(x)) * self.up_proj(x)).float()[0][0].cpu().detach().numpy()}, {(self.act_fn(self.gate_proj(x)) * self.up_proj(x)).shape}") # print(f"output: {output.float()[0][0].cpu().detach().numpy()}, {output.shape}") # assert False return output class Attention(nn.Module): """Multi-headed attention from 'Attention Is All You Need' paper""" def __init__(self, config: BaichuanConfig): super().__init__() self.config = config self.hidden_size = config.hidden_size self.num_heads = config.num_attention_heads self.head_dim = self.hidden_size // self.num_heads self.max_position_embeddings = config.max_position_embeddings if (self.head_dim * self.num_heads) != self.hidden_size: raise ValueError( f"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}" f" and `num_heads`: {self.num_heads})." ) self.W_pack = nn.Linear(self.hidden_size, 3 * self.hidden_size, bias=False) self.o_proj = nn.Linear(self.num_heads * self.head_dim, self.hidden_size, bias=False) self.rotary_emb = RotaryEmbedding(self.head_dim, max_position_embeddings=self.max_position_embeddings) def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int): return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous() def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_value: Optional[Tuple[torch.Tensor]] = None, output_attentions: bool = False, use_cache: bool = False, ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]: bsz, q_len, _ = hidden_states.size() proj = self.W_pack(hidden_states) proj = proj.unflatten(-1, (3, self.hidden_size)).unsqueeze(0).transpose(0, -2).squeeze(-2) query_states = proj[0].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) key_states = proj[1].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) value_states = proj[2].view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2) kv_seq_len = key_states.shape[-2] if past_key_value is not None: kv_seq_len += past_key_value[0].shape[-2] cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len) query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids) # [bsz, nh, t, hd] if past_key_value is not None: # reuse k, v, self_attention key_states = torch.cat([past_key_value[0], key_states], dim=2) value_states = torch.cat([past_key_value[1], value_states], dim=2) past_key_value = (key_states, value_states) if use_cache else None if xops is not None and self.training: attn_weights = None query_states = query_states.transpose(1, 2) key_states = key_states.transpose(1, 2) value_states = value_states.transpose(1, 2) attn_output = xops.memory_efficient_attention( query_states, key_states, value_states, attn_bias=xops.LowerTriangularMask() ) else: attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / torch.tensor(math.sqrt(self.head_dim), dtype=torch.float16) attn_weights = attn_weights.float() attn_weights = 50.0 * torch.tanh(attn_weights / 50.0) if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len): raise ValueError( f"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is" f" {attn_weights.size()}" ) if attention_mask is not None: if attention_mask.size() != (bsz, 1, q_len, kv_seq_len): raise ValueError( f"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}" ) attn_weights = attn_weights + attention_mask attn_weights = torch.max(attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min)) # upcast attention to fp32 attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype) attn_output = torch.matmul(attn_weights, value_states) if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim): raise ValueError( f"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is" f" {attn_output.size()}" ) attn_output = attn_output.transpose(1, 2) attn_output = attn_output.reshape(bsz, q_len, self.hidden_size) attn_output = self.o_proj(attn_output) if not output_attentions: attn_weights = None return attn_output, attn_weights, past_key_value class DecoderLayer(nn.Module): def __init__(self, config: BaichuanConfig): super().__init__() self.hidden_size = config.hidden_size self.depth_scale = torch.tensor(config.depth_scale / math.sqrt(config.num_hidden_layers), dtype=torch.float16) self.self_attn = Attention(config=config) self.mlp = MLP( hidden_size=self.hidden_size, intermediate_size=config.intermediate_size, hidden_act=config.hidden_act, ) self.attention_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.post_attention_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.ffn_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.post_ffn_layernorm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) def forward( self, hidden_states: torch.Tensor, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_value: Optional[Tuple[torch.Tensor]] = None, output_attentions: Optional[bool] = False, use_cache: Optional[bool] = False, ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]: residual = hidden_states hidden_states = self.attention_layernorm(hidden_states) # Self Attention hidden_states, self_attn_weights, present_key_value = self.self_attn( hidden_states=hidden_states, attention_mask=attention_mask, position_ids=position_ids, past_key_value=past_key_value, output_attentions=output_attentions, use_cache=use_cache, ) hidden_states = self.post_attention_layernorm(hidden_states) hidden_states = residual + hidden_states # Fully Connected residual = hidden_states hidden_states = self.ffn_layernorm(hidden_states) hidden_states = self.mlp(hidden_states) hidden_states = self.post_ffn_layernorm(hidden_states) hidden_states = residual + hidden_states outputs = (hidden_states,) if output_attentions: outputs += (self_attn_weights,) if use_cache: outputs += (present_key_value,) return outputs class BaichuanPreTrainedModel(PreTrainedModel): config_class = BaichuanConfig base_model_prefix = "model" supports_gradient_checkpointing = True _no_split_modules = ["DecoderLayer"] _keys_to_ignore_on_load_unexpected = [r"decoder\.version"] def _init_weights(self, module): std = self.config.initializer_range if isinstance(module, nn.Linear): module.weight.data.normal_(mean=0.0, std=std) if module.bias is not None: module.bias.data.zero_() elif isinstance(module, nn.Embedding): module.weight.data.normal_(mean=0.0, std=std) if module.padding_idx is not None: module.weight.data[module.padding_idx].zero_() def _set_gradient_checkpointing(self, module, value=False): if isinstance(module, BaichuanModel): module.gradient_checkpointing = value class BaichuanModel(BaichuanPreTrainedModel): def __init__(self, config: BaichuanConfig): super().__init__(config) self.padding_idx = config.pad_token_id self.vocab_size = config.vocab_size self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx) self.embed_scale = torch.tensor(config.embed_scale, dtype=torch.float16) self.layers = nn.ModuleList([DecoderLayer(config) for _ in range(config.num_hidden_layers)]) self.norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps) self.gradient_checkpointing = True # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.embed_tokens def set_input_embeddings(self, value): self.embed_tokens = value # Copied from transformers.models.bart.modeling_bart.BartDecoder._prepare_decoder_attention_mask def _prepare_decoder_attention_mask(self, attention_mask, input_shape, inputs_embeds, past_key_values_length): # create causal mask # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] combined_attention_mask = None if input_shape[-1] > 1: combined_attention_mask = _make_causal_mask( input_shape, inputs_embeds.dtype, device=inputs_embeds.device, past_key_values_length=past_key_values_length, ) if attention_mask is not None: # [bsz, seq_len] -> [bsz, 1, tgt_seq_len, src_seq_len] expanded_attn_mask = _expand_mask(attention_mask, inputs_embeds.dtype, tgt_len=input_shape[-1]).to( inputs_embeds.device ) combined_attention_mask = ( expanded_attn_mask if combined_attention_mask is None else expanded_attn_mask + combined_attention_mask ) return combined_attention_mask def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_values: Optional[List[torch.FloatTensor]] = None, inputs_embeds: Optional[torch.FloatTensor] = None, use_cache: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, BaseModelOutputWithPast]: output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) use_cache = use_cache if use_cache is not None else self.config.use_cache return_dict = return_dict if return_dict is not None else self.config.use_return_dict # retrieve input_ids and inputs_embeds if input_ids is not None and inputs_embeds is not None: raise ValueError("You cannot specify both decoder_input_ids and decoder_inputs_embeds at the same time") elif input_ids is not None: batch_size, seq_length = input_ids.shape elif inputs_embeds is not None: batch_size, seq_length, _ = inputs_embeds.shape else: raise ValueError("You have to specify either decoder_input_ids or decoder_inputs_embeds") seq_length_with_past = seq_length past_key_values_length = 0 if past_key_values is not None: past_key_values_length = past_key_values[0][0].shape[2] seq_length_with_past = seq_length_with_past + past_key_values_length if position_ids is None: device = input_ids.device if input_ids is not None else inputs_embeds.device position_ids = torch.arange( past_key_values_length, seq_length + past_key_values_length, dtype=torch.long, device=device ) position_ids = position_ids.unsqueeze(0).view(-1, seq_length) else: position_ids = position_ids.view(-1, seq_length).long() if inputs_embeds is None: inputs_embeds = self.embed_tokens(input_ids) inputs_embeds = inputs_embeds * self.embed_scale # embed positions if attention_mask is None: attention_mask = torch.ones( (batch_size, seq_length_with_past), dtype=torch.bool, device=inputs_embeds.device ) attention_mask = self._prepare_decoder_attention_mask( attention_mask, (batch_size, seq_length), inputs_embeds, past_key_values_length ) hidden_states = inputs_embeds if self.gradient_checkpointing and self.training: if use_cache: logger.warning_once( "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`..." ) use_cache = False # decoder layers all_hidden_states = () if output_hidden_states else None all_self_attns = () if output_attentions else None next_decoder_cache = () if use_cache else None for idx, decoder_layer in enumerate(self.layers): if output_hidden_states: all_hidden_states += (hidden_states,) past_key_value = past_key_values[idx] if past_key_values is not None else None if self.gradient_checkpointing and self.training: def create_custom_forward(module): def custom_forward(*inputs): # None for past_key_value return module(*inputs, output_attentions, None) return custom_forward layer_outputs = torch.utils.checkpoint.checkpoint( create_custom_forward(decoder_layer), hidden_states, attention_mask, position_ids, None, ) else: layer_outputs = decoder_layer( hidden_states, attention_mask=attention_mask, position_ids=position_ids, past_key_value=past_key_value, output_attentions=output_attentions, use_cache=use_cache, ) hidden_states = layer_outputs[0] if use_cache: next_decoder_cache += (layer_outputs[2 if output_attentions else 1],) if output_attentions: all_self_attns += (layer_outputs[1],) hidden_states = self.norm(hidden_states) # add hidden states from the last decoder layer if output_hidden_states: all_hidden_states += (hidden_states,) next_cache = next_decoder_cache if use_cache else None if not return_dict: return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None) return BaseModelOutputWithPast( last_hidden_state=hidden_states, past_key_values=next_cache, hidden_states=all_hidden_states, attentions=all_self_attns, ) class NormHead(nn.Module): def __init__(self, hidden_size, vocab_size, bias=False): super().__init__() self.weight = nn.Parameter(torch.empty((vocab_size, hidden_size))) def forward(self, hidden_states): return nn.functional.linear(hidden_states, self.weight) class BaichuanForCausalLM(BaichuanPreTrainedModel): def __init__(self, config): super().__init__(config) self.model = BaichuanModel(config) self.lm_head = NormHead(config.hidden_size, config.vocab_size, bias=False) # Initialize weights and apply final processing self.post_init() def get_input_embeddings(self): return self.model.embed_tokens def set_input_embeddings(self, value): self.model.embed_tokens = value def get_output_embeddings(self): return self.lm_head def set_output_embeddings(self, new_embeddings): self.lm_head = new_embeddings def set_decoder(self, decoder): self.model = decoder def get_decoder(self): return self.model def forward( self, input_ids: torch.LongTensor = None, attention_mask: Optional[torch.Tensor] = None, position_ids: Optional[torch.LongTensor] = None, past_key_values: Optional[List[torch.FloatTensor]] = None, inputs_embeds: Optional[torch.FloatTensor] = None, labels: Optional[torch.LongTensor] = None, use_cache: Optional[bool] = None, output_attentions: Optional[bool] = None, output_hidden_states: Optional[bool] = None, return_dict: Optional[bool] = None, ) -> Union[Tuple, CausalLMOutputWithPast]: output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions output_hidden_states = ( output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states ) return_dict = return_dict if return_dict is not None else self.config.use_return_dict # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn) outputs = self.model( input_ids=input_ids, attention_mask=attention_mask, position_ids=position_ids, past_key_values=past_key_values, inputs_embeds=inputs_embeds, use_cache=use_cache, output_attentions=output_attentions, output_hidden_states=output_hidden_states, return_dict=return_dict, ) hidden_states = outputs[0] logits = self.lm_head(hidden_states) logits = logits.float() logits = 30.0 * torch.tanh(logits / 30.0) loss = None if labels is not None: # Shift so that tokens < n predict n shift_logits = logits[..., :-1, :].contiguous() shift_labels = labels[..., 1:].contiguous() # Flatten the tokens loss_fct = CrossEntropyLoss() shift_logits = shift_logits.view(-1, self.config.vocab_size) shift_labels = shift_labels.view(-1) # Enable model parallelism shift_labels = shift_labels.to(shift_logits.device) loss = loss_fct(shift_logits, shift_labels) if not return_dict: output = (logits,) + outputs[1:] return (loss,) + output if loss is not None else output return CausalLMOutputWithPast( loss=loss, logits=logits, past_key_values=outputs.past_key_values, hidden_states=outputs.hidden_states, attentions=outputs.attentions, ) def prepare_inputs_for_generation( self, input_ids, past_key_values=None, attention_mask=None, inputs_embeds=None, **kwargs ): if past_key_values: input_ids = input_ids[:, -1:] position_ids = kwargs.get("position_ids", None) if attention_mask is not None and position_ids is None: # create position_ids on the fly for batch generation position_ids = attention_mask.long().cumsum(-1) - 1 position_ids.masked_fill_(attention_mask == 0, 1) if past_key_values: position_ids = position_ids[:, -1].unsqueeze(-1) # if `inputs_embeds` are passed, we only want to use them in the 1st generation step if inputs_embeds is not None and past_key_values is None: model_inputs = {"inputs_embeds": inputs_embeds} else: model_inputs = {"input_ids": input_ids} model_inputs.update( { "position_ids": position_ids, "past_key_values": past_key_values, "use_cache": kwargs.get("use_cache"), "attention_mask": attention_mask, } ) return model_inputs @staticmethod def _reorder_cache(past_key_values, beam_idx): reordered_past = () for layer_past in past_key_values: reordered_past += (tuple(past_state.index_select(0, beam_idx) for past_state in layer_past),) return reordered_past def chat(self, tokenizer, messages: List[dict], stream=False, generation_config: Optional[GenerationConfig]=None): generation_config = generation_config or self.generation_config input_ids = build_chat_input(self, tokenizer, messages, generation_config.max_new_tokens) if stream: streamer = TextIterStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True) Thread(target=self.generate, kwargs=dict( inputs=input_ids, streamer=streamer, generation_config=generation_config, )).start() return streamer else: outputs = self.generate(input_ids, generation_config=generation_config) response = tokenizer.decode(outputs[0][len(input_ids[0]):], skip_special_tokens=True) return response pengcheng.mind-mindformer/scripts/hf_7B_shared/special_tokens_map.json000066400000000000000000000010441476650141400266060ustar00rootroot00000000000000{ "bos_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "eos_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "pad_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false }, "unk_token": { "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": false } } pengcheng.mind-mindformer/scripts/hf_7B_shared/tokenization_baichuan.py000066400000000000000000000226151476650141400270040ustar00rootroot00000000000000# Copyright 2023 Baichuan Inc. All Rights Reserved. # Copyright 2022 EleutherAI and the HuggingFace Inc. team. All rights reserved. # # This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX # and OPT implementations in this library. It has been modified from its # original forms to accommodate minor architectural differences compared # to GPT-NeoX and OPT used by the Meta AI team that trained the model. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import os from shutil import copyfile from typing import Any, Dict, List, Optional, Tuple import sentencepiece as spm from transformers.tokenization_utils import AddedToken, PreTrainedTokenizer from transformers.utils import logging logger = logging.get_logger(__name__) VOCAB_FILES_NAMES = {"vocab_file": "tokenizer.model"} PRETRAINED_VOCAB_FILES_MAP = { "vocab_file": {}, "tokenizer_file": {}, } PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES = {} class BaichuanTokenizer(PreTrainedTokenizer): """ Construct a Baichuan tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. """ vocab_files_names = VOCAB_FILES_NAMES pretrained_vocab_files_map = PRETRAINED_VOCAB_FILES_MAP max_model_input_sizes = PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES model_input_names = ["input_ids", "attention_mask"] def __init__( self, vocab_file, unk_token="", bos_token="", eos_token="", pad_token=None, sp_model_kwargs: Optional[Dict[str, Any]] = None, add_bos_token=True, add_eos_token=False, clean_up_tokenization_spaces=False, **kwargs, ): self.sp_model_kwargs = {} if sp_model_kwargs is None else sp_model_kwargs bos_token = AddedToken(bos_token, lstrip=False, rstrip=False) if isinstance(bos_token, str) else bos_token eos_token = AddedToken(eos_token, lstrip=False, rstrip=False) if isinstance(eos_token, str) else eos_token unk_token = AddedToken(unk_token, lstrip=False, rstrip=False) if isinstance(unk_token, str) else unk_token pad_token = AddedToken(pad_token, lstrip=False, rstrip=False) if isinstance(pad_token, str) else pad_token self.vocab_file = vocab_file self.add_bos_token = add_bos_token self.add_eos_token = add_eos_token self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(vocab_file) super().__init__( bos_token=bos_token, eos_token=eos_token, unk_token=unk_token, pad_token=pad_token, add_bos_token=add_bos_token, add_eos_token=add_eos_token, sp_model_kwargs=self.sp_model_kwargs, clean_up_tokenization_spaces=clean_up_tokenization_spaces, **kwargs, ) def __getstate__(self): state = self.__dict__.copy() state["sp_model"] = None return state def __setstate__(self, d): self.__dict__ = d self.sp_model = spm.SentencePieceProcessor(**self.sp_model_kwargs) self.sp_model.Load(self.vocab_file) @property def vocab_size(self): """Returns vocab size""" return self.sp_model.get_piece_size() def get_vocab(self): """Returns vocab as a dict""" vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)} vocab.update(self.added_tokens_encoder) return vocab def _tokenize(self, text): """Returns a tokenized string.""" return self.sp_model.encode(text, out_type=str) def _convert_token_to_id(self, token): """Converts a token (str) in an id using the vocab.""" return self.sp_model.piece_to_id(token) def _convert_id_to_token(self, index): """Converts an index (integer) in a token (str) using the vocab.""" token = self.sp_model.IdToPiece(index) return token def convert_tokens_to_string(self, tokens): """Converts a sequence of tokens (string) in a single string.""" current_sub_tokens = [] out_string = "" prev_is_special = False for i, token in enumerate(tokens): # make sure that special tokens are not decoded using sentencepiece model if token in self.all_special_tokens: if not prev_is_special and i != 0: out_string += " " out_string += self.sp_model.decode(current_sub_tokens) + token prev_is_special = True current_sub_tokens = [] else: current_sub_tokens.append(token) prev_is_special = False out_string += self.sp_model.decode(current_sub_tokens) return out_string def save_vocabulary(self, save_directory, filename_prefix: Optional[str] = None) -> Tuple[str]: """ Save the vocabulary and special tokens file to a directory. Args: save_directory (`str`): The directory in which to save the vocabulary. Returns: `Tuple(str)`: Paths to the files saved. """ if not os.path.isdir(save_directory): logger.error(f"Vocabulary path ({save_directory}) should be a directory") return out_vocab_file = os.path.join( save_directory, (filename_prefix + "-" if filename_prefix else "") + VOCAB_FILES_NAMES["vocab_file"] ) if os.path.abspath(self.vocab_file) != os.path.abspath(out_vocab_file) and os.path.isfile(self.vocab_file): copyfile(self.vocab_file, out_vocab_file) elif not os.path.isfile(self.vocab_file): with open(out_vocab_file, "wb") as fi: content_spiece_model = self.sp_model.serialized_model_proto() fi.write(content_spiece_model) return (out_vocab_file,) def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None): bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = bos_token_id + token_ids_0 + eos_token_id if token_ids_1 is not None: output = output + bos_token_id + token_ids_1 + eos_token_id return output def get_special_tokens_mask( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False ) -> List[int]: """ Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. """ if already_has_special_tokens: return super().get_special_tokens_mask( token_ids_0=token_ids_0, token_ids_1=token_ids_1, already_has_special_tokens=True ) bos_token_id = [1] if self.add_bos_token else [] eos_token_id = [1] if self.add_eos_token else [] if token_ids_1 is None: return bos_token_id + ([0] * len(token_ids_0)) + eos_token_id return ( bos_token_id + ([0] * len(token_ids_0)) + eos_token_id + bos_token_id + ([0] * len(token_ids_1)) + eos_token_id ) def create_token_type_ids_from_sequences( self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None ) -> List[int]: """ Creates a mask from the two sequences passed to be used in a sequence-pair classification task. An ALBERT sequence pair mask has the following format: ``` 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 | first sequence | second sequence | ``` if token_ids_1 is None, only returns the first portion of the mask (0s). Args: token_ids_0 (`List[int]`): List of ids. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s). """ bos_token_id = [self.bos_token_id] if self.add_bos_token else [] eos_token_id = [self.eos_token_id] if self.add_eos_token else [] output = [0] * len(bos_token_id + token_ids_0 + eos_token_id) if token_ids_1 is not None: output += [1] * len(bos_token_id + token_ids_1 + eos_token_id) return output pengcheng.mind-mindformer/scripts/hf_7B_shared/tokenizer.model000066400000000000000000075043231476650141400251260ustar00rootroot00000000000000                   

      

      

                   
      
          
  •   
  •   
    •     

      

      

     
      
      
     

      

      

     
      
      
     

     

     

     

      

      

           ０ １ ２ ３ ４ ５ ６ ７ ８ ９ + - = ， 。 ！ ？ 、 ： ￥ . ! ? ...  。。。  。。。。。。 《 》 【 】 『 』 ```  ---   ； ． ＝ ＜ ＞ － ＋ ％ ‼ ㊣ ／ ｜                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          <0x00>  <0x01>  <0x02>  <0x03>  <0x04>  <0x05>  <0x06>  <0x07>  <0x08>  <0x09>  <0x0A>  <0x0B>  <0x0C>  <0x0D>  <0x0E>  <0x0F>  <0x10>  <0x11>  <0x12>  <0x13>  <0x14>  <0x15>  <0x16>  <0x17>  <0x18>  <0x19>  <0x1A>  <0x1B>  <0x1C>  <0x1D>  <0x1E>  <0x1F>  <0x20>  <0x21>  <0x22>  <0x23>  <0x24>  <0x25>  <0x26>  <0x27>  <0x28>  <0x29>  <0x2A>  <0x2B>  <0x2C>  <0x2D>  <0x2E>  <0x2F>  <0x30>  <0x31>  <0x32>  <0x33>  <0x34>  <0x35>  <0x36>  <0x37>  <0x38>  <0x39>  <0x3A>  <0x3B>  <0x3C>  <0x3D>  <0x3E>  <0x3F>  <0x40>  <0x41>  <0x42>  <0x43>  <0x44>  <0x45>  <0x46>  <0x47>  <0x48>  <0x49>  <0x4A>  <0x4B>  <0x4C>  <0x4D>  <0x4E>  <0x4F>  <0x50>  <0x51>  <0x52>  <0x53>  <0x54>  <0x55>  <0x56>  <0x57>  <0x58>  <0x59>  <0x5A>  <0x5B>  <0x5C>  <0x5D>  <0x5E>  <0x5F>  <0x60>  <0x61>  <0x62>  <0x63>  <0x64>  <0x65>  <0x66>  <0x67>  <0x68>  <0x69>  <0x6A>  <0x6B>  <0x6C>  <0x6D>  <0x6E>  <0x6F>  <0x70>  <0x71>  <0x72>  <0x73>  <0x74>  <0x75>  <0x76>  <0x77>  <0x78>  <0x79>  <0x7A>  <0x7B>  <0x7C>  <0x7D>  <0x7E>  <0x7F>  <0x80>  <0x81>  <0x82>  <0x83>  <0x84>  <0x85>  <0x86>  <0x87>  <0x88>  <0x89>  <0x8A>  <0x8B>  <0x8C>  <0x8D>  <0x8E>  <0x8F>  <0x90>  <0x91>  <0x92>  <0x93>  <0x94>  <0x95>  <0x96>  <0x97>  <0x98>  <0x99>  <0x9A>  <0x9B>  <0x9C>  <0x9D>  <0x9E>  <0x9F>  <0xA0>  <0xA1>  <0xA2>  <0xA3>  <0xA4>  <0xA5>  <0xA6>  <0xA7>  <0xA8>  <0xA9>  <0xAA>  <0xAB>  <0xAC>  <0xAD>  <0xAE>  <0xAF>  <0xB0>  <0xB1>  <0xB2>  <0xB3>  <0xB4>  <0xB5>  <0xB6>  <0xB7>  <0xB8>  <0xB9>  <0xBA>  <0xBB>  <0xBC>  <0xBD>  <0xBE>  <0xBF>  <0xC0>  <0xC1>  <0xC2>  <0xC3>  <0xC4>  <0xC5>  <0xC6>  <0xC7>  <0xC8>  <0xC9>  <0xCA>  <0xCB>  <0xCC>  <0xCD>  <0xCE>  <0xCF>  <0xD0>  <0xD1>  <0xD2>  <0xD3>  <0xD4>  <0xD5>  <0xD6>  <0xD7>  <0xD8>  <0xD9>  <0xDA>  <0xDB>  <0xDC>  <0xDD>  <0xDE>  <0xDF>  <0xE0>  <0xE1>  <0xE2>  <0xE3>  <0xE4>  <0xE5>  <0xE6>  <0xE7>  <0xE8>  <0xE9>  <0xEA>  <0xEB>  <0xEC>  <0xED>  <0xEE>  <0xEF>  <0xF0>  <0xF1>  <0xF2>  <0xF3>  <0xF4>  <0xF5>  <0xF6>  <0xF7>  <0xF8>  <0xF9>  <0xFA>  <0xFB>  <0xFC>  <0xFD>  <0xFE>  <0xFF> ▁t ▁a in he@ re on er ▁the ▁s at or en0 ▁w@ anP it` esp ▁c ▁o is ing ▁f nd al ▁p ou ed ar ▁b ▁▁ ▁in ▁to ▁of ▁and ▁m ic ion ▁d as le om ▁h il$ ent( ▁th, st0 ro4 el8 ▁re< ▁l@ ▁SD ctH etL ▁nP ▁TT ▁CX ▁e\ ▁A` idd veh ▁gl olp ▁It utx am| ot ur  ▁▁▁▁ ▁for ay ▁is ▁on ad ac im ig ow ly ▁M us ation ▁be ▁y ver ▁P ▁st ir ch ▁B ith  ▁that ce  ▁with un ▁you ers ul ▁he ▁it ▁D se ▁H ill ag ter ▁as ▁W ▁R ▁an ▁al ▁F em ▁wh if ▁( ▁pro ra est ri ▁con and th ew ist ▁L op ▁N od ate her ▁at ▁com ess um ▁we pp ab ▁The ke ore ▁are ▁G ▁E ity rom ld ▁v ort ht ▁de res ▁was oc ment os ▁ex ▁or qu ▁ha ies art ▁se ant! ▁r" ▁by# ain$ ect% ▁J& nt'  ▁from( ive) ▁O* iv+ ust, ud- our. end/ ▁su0 all1 ear2 The3 ▁ch4 ard5  ▁have6  ▁this7 ight8 ▁sh9 ▁U: ial; out< red= ▁u> ost? 工作@ ▁notA ▁worB ▁leC  ▁▁▁▁▁▁▁▁D astE 我们F rouG ogH ▁hasI ellJ apK geL omeM ackN ▁plO okP ▁usQ akR  ▁willS ineT 一个U ▁canV ouldW ureX ▁abY iaZ ▁sp[ ec\ ide] ff^ ip_ ▁j` bera ptb ▁Stc ionsd iche icef 自己g ageh peri ongj ▁Kk ▁alll ghm 可以n ubo  ▁compp ryq ansr ares amet ieu ▁“v ▁adw ▁Inx ▁ky atedz ▁go{  ▁your| ▁int} ue~ ally ▁whe ary ▁en cc iz pl ther ind  ▁their ▁cl ▁his ous ass ▁sa orm ▁do ook cl ▁but able act ▁ne ▁V one ime ep ult ▁out  ▁more ▁im  ▁cont so ire ace ▁up  ▁they ake ance ▁un ▁Ch 中国 ▁who ▁me ▁tr ib ations ord ▁Th ign 时间 ere du av very  ▁year  ▁work ail ite ▁res  ▁which ▁one ▁new ical ▁" ▁te port 没有 ov ount  ▁about ase ents ile ▁man ction ▁Y ence ark ▁so ru ach ▁off erv form ▁our ays ▁app lic ond com ▁per ll ▁ar 就是 ang  ▁also 进行 mer vel int ory vers reat ob ▁li ress ian  ▁part ition own 人员 ors 孩子 ▁its  ▁been ), ild 疫情 公司 ▁ag  ▁other ree ose ▁dis  ▁said ely ▁qu ▁ev ens ick ings 教育 ish ound te ove ▁her  ▁were ne ▁sc 生活 的人 ▁pe  ▁over ▁It irst 什么 发展 ”,  ▁time ▁my oy ks 问题 ▁had ▁ro amp ople ▁rec 服务 fter ▁get ood old  ▁than 管理  ▁would ric anc 他们  ▁them  ▁there  ▁some wn oll ▁acc  ▁when ▁pre urn ▁any ss use clud rit ng 如果  ▁comm xt 学习 这个 ates ool ▁He ▁kn  ▁like ade iew ont ons ▁– nder pe tern  ational 企业 ▁tw ▁Un  ▁people hed ph aw  ▁into igh  ▁first 安全 ater  ▁what ram 因为 ject ▁$ ); 大家 ink 活动 ual 时候 ▁fe  ▁just ious iss ▁am 信息 ▁des lect 需要 ef In 情况 通过 ▁if we ning ▁bec ities ▁she ▁how 国家  ▁includ 学生 rough enc 要求 人民 ▁We ft day ▁ind@  ▁need 项目 get  ▁prov@ ath les pport ▁Re@ ck 不是  ▁▁▁ 第一@ ublic ▁New iff 健康@ // ▁act  ▁most ▁add@ round 很多  ▁years ident@ olog ▁Com ▁two 建设@  ▁after ps ▁& 可能@ 这样 人的 ert Th@ ise li hen ▁bet@  ▁through any  ▁under 社会@ ug ock ▁em  ▁play@ ward ▁no sh row@  ▁only ife ating 已经@ 大学 ork ty  ▁serv@ ific  ▁cons ▁inv ▁hel@ 学校 ▁bu 中心 ▁bl@  ▁stud als pect 来源@ ting ▁co yst 防控@ —— ics  自己的 ause@ ▁att 专业 所以 ▁imp@ ivers  ▁produ 都是 und@ ring io arch eth@ ▁reg 相关  ▁back ased@ ough  ▁This ,” oint@ ily ited uring rib@ ▁Pro 使用 oth ▁may@ 开始 chool ments ict@ velop  ▁well elf ful@ 重要 ek  ▁make erm@ iness 联系 ren ▁pr@ land ata alth ange@  ▁every ▁fl ts! ▁col@! 的一! 市场! ience" ▁Ar@" 也是"  ▁know" ▁sec# 研究@#  ▁these# ▁ph# 这些$ 还是@$  ▁where$  ▁look$  ▁help%  ▁could@% ▁inc% ener%  ▁many& riv@& ility& 网络& ex' ices@' 好的' 现在' 方式( ▁tra@( 技术( 单位( erson) ▁fin@) ys) gan) oss* 但是@* ▁pol* ures* ement+  ▁want@+ ▁rem+ ▁ass+ ash,  ▁diff@, 关注, 朋友, bs- ▁ac@- ystem- :“- aking. 同时@.  usiness. ▁rel. ues/  ▁exper@/ 微信/ ▁set/ 组织0 ract@0 ▁ent0 ull0 ▁now1 ower@1 一定1 ▁De1 ative2 老师@2 uch2 ible2 ular3 部分@3  ▁world3 ▁Wh3 ▁use4 到了@4 oun4 ner4 az5 up@5 非常5 所有5 ww6 air@6 ix6 个人6  ▁show7 pr@7 内容7 |　7 发现8 its@8 ▁{8 fore8 ax9 irect@9 不能9 ▁way9 世界: 知道@: 能力: ied: ▁ke; ▁dec@; St; ▁Sh;  ▁pres< meric@< 提供<  ▁count< 影响= 的是@= ank=  ▁start=  的时候> ▁own@> itt> ▁see> 招聘? ▁Se@? ah? ҉҉? 经济@ con@@  ▁very@ ▁Al@ insA ▁fam@A 一起A ▁artA ▁himB view@B manB ▁'B 发布C  ▁right@C ▁locC  ▁evenC asonD 以上@D ientD hesD  formationE ced@E gramE ▁YouE 选择F  ▁such@F  ▁instF  ▁againF  ▁highG iel@G ▁LeG  ▁developG ▁AndH the@H  ▁leadH 出现H 产品I hip@I retI chnI otherJ ale@J venJ  ▁specJ 医院K ince@K 全国K letK ▁monL ▁eff@L ▁endL  　|　L 为了M  ▁should@M 文化M 城市M 根据N ▁mem@N  ▁thenN ivesN 不要O ble@O 政府O  ▁beingO 设计P ines@P ▁guP vernP  ▁callQ ▁did@Q ▁carQ ucQ 其他R  iversity@R ▁smR WeR  ▁transS 对于@S 平台S penS 你的T ▁An@T 目前T  ▁longT 考试U atch@U  ▁publicU  ▁supportU  ▁becauseV 还有@V 报名V 一些V ohW ▁sur@W 开展W ollowW unityX ouse@X  ▁goodX  ▁madeX  ▁lastY 电话@Y  ▁AmericY 方面Y 作为Z 中的@Z 工程Z  ▁greatZ ution[ ▁ret@[  ▁team[ ▁gr[ ▁ins\ ▁med@\ to\ ital\ **] ▁Tr@] 主要]  疫情防控]  ▁char^ cept@^ ▁fil^  ▁much^ ets_ 大的@_ 交通_ ton_ Wh` ave@` uth` 不会` ▁eara 发生@a ▁bela amesa 环境b  ▁those@b ▁modb ologyb msc 以及@c agesc 部门c  ▁downd ces@d 规定d  ▁businessd ▁done  ▁before@e  ▁reque ▁defe ▁numf ▁br@f tenf  ▁findf  ▁markg oci@g lineg  ▁bestg ▁serh 系统@h  ▁differeh ▁vish istsi read@i ▁curi  ▁classi cessj ▁Sc@j ▁edj ▁crej ledk ▁min@k Itk 是一k  ▁doesl 更多@l 或者l 地区l weenm  ▁take@m ▁Clm 美国m illionn way@n 喜欢n athern ▁rao rent@o 责任o ……o 怎么p ern@p  ▁interp ▁avp  ▁readq  ▁while@q  ▁lifeq ▁Spq 了解r imes@r  ▁systemr ▁subr 来的s 生产@s egs roups ://t orn@t 有限t  ▁persont elsu alk@u 视频u  ▁hereu 知识v ▁‘@v ▁Forv ▁dayv ▁Indw uss@w  ▁foundw  ▁thinkw  ▁discx oot@x 了一x derx ,　y 做好@y  ▁informationy 注意y  ▁homez ▁—@z Thisz ▁Asz ize{ 如何@{ vent{ ▁ext{ 精神| ode@| ands| 进入|  ▁between} 那么@} ▁Ad} 成为} WS~  ▁report@~ ▁exp~ ute~ 关于  ▁conf@ 病毒  ▁used ered  ▁care ield@  ▁sign` ▁ref aj ▁str ann akes ross 新冠@  ▁child` 不同 第二 ars arm  ▁cour  ▁around  ▁cent@ ▁det` ral ▁Bl  ▁follow 表示 ually 教师  ▁program@ tt`  ▁three ever ional ▁But 能够 oad ▁|@ ina` 每天 ox gy ety  ▁provid 其中 ▁Fr@ att`  ▁design 身体 usic ▁som 标准 ▁sim  ▁both@ 北京` 一步 bers ier  ▁during ik  ▁including ▁Is@ ER`  ▁still ▁rep ▁fun ants 数据 ished 特别@ 之后` ness 价格 eng 这种 今天  ▁school ism@ 点击` 资源  ▁contin 起来 ▁Ph 其实 ▁che ▁val@ ▁*` mber 方法 完成 学院 ▁del 书记 ▁gl@ ality` ght ved 一次  ▁book 7 0▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁University 比较@ ▁run` ision  ▁each 重点 ▁inf ized 基础  ▁company@ 领导`  ▁import 期间 综合 lease arg ▁Z CWS@ au`  ▁techn 提升 ee ▁Pl  ▁health  ▁point 投资@  ▁build`  ▁same nce 行为 ural ired  ▁form ▁Col@ 通知` ple 过程 上的  ▁different ▁cr  ▁project 参加@ 这里` 病例 政策 ▁[ 提高  vernment ature  ▁data@ 及时` 实现 () ▁Q 群众  ▁game ined  ▁gener@ ering` 条件 ▁cap ▁sol 产业 岗位 ▁Ex 机构@ ▁adv` eb 最后  ▁prof  ▁open 年的 son ving@  ▁post` ID omen 关系  ▁direct ery 希望 保障@ ari` 计划  ▁month 现场 阅读 ▁iss  ▁Comm 卫生@ ▁She` ▁Con erest 自然 基本 容易 ared 免费@ ▁ann`  ▁week 直接 全面 of  ▁dist  ▁really ▁ob@ ruct` 而且 app 行业 作用  ▁invest ploy inal@ riend` 检测  ▁going by 家庭 推荐 看到 公众@ me`  ▁place 支持 ▁Te 按照 IN arn ccess@ ▁say`  ▁resp  ▁product  ▁organ 质量 cent 一样  ▁appro@ br`  有限公司 orth ▁law ▁top reen  ▁pass  ▁list@ outh` 坚持 公安 ▁If uro ▁mov 合作 ining@ ustom` 家长 国际 有效 ▁Res 加强  ▁result 落实@ ,“` ▁dep  ▁process 会议 风险 ody ▁cor 科技@  ▁Cent` 这一 ▁ty 处理 ember 课程  ▁says  ▁rese@  ▁writ`  ▁stat oard  ▁partic rist sc raph ense@ 积极` ump ission iving 一直  ▁since 历史  ▁family@ co` ways ator  ▁prot ▁la 文章 ley ocial@ 编辑` 不断 分享 代表  ▁hist 患者 科学 ▁cle@ 具有` 负责  ▁perform 有关 ▁sl  ▁against 作者 肺炎@ ▁US` ▁el Ch ron  ▁state  ▁second ," 这是@ 成绩` 获得 ON  ailable 事业  ▁expl 社区 结果@  ▁They` ▁leg 感染 :　 虽然  ▁real ▁All min@ be`  ▁main ▁sk 不仅 觉得 enn EN  ▁happ@ 帮助`  ▁number ▁lot ohn 小学 self 地方 ▁Mar@ )　` ▁pur 小时 只有 ially  ternational  ▁fact 培训@ ▁Be` ▁Jan  ▁rest ling ency ▁How 医疗 ▁big@ 解决` ior 就会 + $▁▁▁▁▁▁▁▁▁▁▁▁  ▁rece ▁est AR ves@ ases` ▁few 今年  ▁current less ▁pop  ▁allow pro@ 集团` ically ▁i 毕业  ▁oper 应该 我的  ▁next@ ▁pat`  ▁interest gg 甚至 uthor 治疗 ▁/ 一般@ 必须` ury  ▁using ration 不过 que une 手机@ 措施` 继续 新闻 努力  ▁poss nect 保护 教学@  ▁incre`  lection 实施 err  ▁const ording ead iver@ ird`  ▁plan ott ▁fr 建议  ours@ 法律`  ▁sing arent  ▁turn 农村 As aut  ▁services@ lex`  ▁hard oney  ▁success ged 有人 无法  ▁view@ ured` 来了  ▁cost 经验  ▁group  ▁never ▁Rep  ▁return@  ▁story` 造成  ▁John  ▁season oor  ▁change 电子 由于@ 只是` 确诊  ▁children 是否 itions ource 同学  ▁friend@ ▁Pr` uture uck ▁aff 销售 控制 真的 指导@  ▁water`  ▁record 水平 开发 ising ▁Med  ▁case www@ for` ▁Br AS  ▁research  ▁belie  ▁elect 干部  ▁able@ 执行` ified aff ived If  ▁proble ify ▁equ@  我们的` ider ocus 结合 年级 ▁Am 作品 第三@ ope`  ▁▁▁▁▁▁▁  ▁provide 简单 汽车 二维 ▁met 这么@ 决定` ▁bre 出了 严重 ister 业务 viron  ▁meet@  二维码`  ▁chang ourn inc 艺术  ▁today ▁beh aster@ ument`  ▁fund  ▁little urs  这样的 ▁got 最新 左右@  ▁within` 只要 都会  ▁film put  ▁days tain osp@ ii` aim 记者  ▁past uary  ▁less 委员 超过@  ▁expect` 的大  ▁might  ability 更加 ients 感觉 org@ 目标`  ▁visit 价值  ▁area ▁Fl  ▁Euro  ▁event 区域@ urity` ▁At 介绍 ateg ential ▁Gr 方案 til@  ▁York` ideo obal  ▁activ 都有 icy  ▁priv  ▁social@ ▁Su` ▁Pol By What  ▁employ An  ▁With  ▁present@  ▁across` 需求  ▁near  ▁special  ▁Pres ▁So ▁air ems@ 购买` 接受 不到 的话  ▁camp 公告 Pro ask@ gin` ane  ▁conc 接种 职业  ▁Christ 车辆  ▁proper@  ▁addition` 一点 over ele side 优秀 arge  ▁women@ 人才` 正在  ▁making ▁ask oid  ination 调整 范围@ orts` 正常 iven  ▁city  ▁World ▁exc  ▁members  ▁National@ 办公` atic ball  为什么 面试 ▁spe 专家 %,@ 空间` 即可  ▁profess 形成 ales ext  artment 国内@ ards` AP 政治 ample  ▁School ograph ▁Ass He@ ▁No` my  ▁That ike rict 疾病 ▁z wards@ 除了` 利用 来自 ev duct ▁Gu 然后 atter@  vironment` 确保 ienc 症状 小区 ▁job  ▁major  ▁State@ ▁par` 之前 达到  ▁pract ▁cho ▁ben  ▁educ ending@  ▁Inst` idd  ▁creat  ▁phot  ▁impro  ▁order 个月 ▁saf@ 别人` 查看  ▁City  ▁protect ready 有的 eter 幸福@ 经营` You 共同 iam  ▁along 武汉  ▁move ▁Min@ ention` ung  ▁Count eral  ▁term  ▁feat  ▁webs  ▁polit@ ▁App` 比如 IS  ▁young 不可 lish che 公共@  ▁econom` ians 妈妈  ▁development ▁mil 关键  ▁give 心理@  ▁supp` How 真正 完全 表现 经过  ▁called  一定要@ ably`  ▁focus wit ▁: ris 生态 veral 工资@  ▁orig` 交流 消息 ills 东西 ▁En 全球 机会@  ▁Europe` ▁Oct 直播 得到 一种  ▁left 幼儿 alf@  ▁become` lege gether  ▁together 志愿 严格  ▁charact  ▁future@ ners` ows 招生 机关 驾驶 保险 For  ▁News@ 产生` go 面积  ▁rele 女人 ials  ▁trad  ▁fore@ 来越` itor val ▁aut IC 违法 出来 ended@ pril` 不少 ▁Ed ances IT 隔离 阶段 ored rop0  ▁sure@  ▁historyP  越来越` ivelyp  ▁applic yle 加入 ▁far 有些 aces 来看 ): 推动 ↓↓ iting   ▁online0 多少@ bookP  ▁include`  ▁opportp  ▁commun erence 让人 全部 yl 思想 党员  ▁away resent ique rem   ▁until0 ▁let@  ▁connectP ▁Af` 随着p ▁Bro rap 属于  ▁news  ▁▁▁▁▁▁  ▁line ony 拥有 系列 ▁non  ▁based  ipp0  ▁series@ ▁optP anies` ▁likp 变化  ▁March 里的 也会  ▁mult 收入 AL itive 客户 英语 ey  ▁why0 任务@ 想要P 力量` ▁Xp ▁dev 主动 我国  ▁industry  ▁Aust ones  ▁January ”　  ▁several  ▁already 之一   ▁announ0 回复@  ▁condP  新冠肺炎`  ▁earlyp 集中 文明 ▁Off 身份 ▁Cal joy 调查  ▁site 网友  ▁States 以后  ▁net0  ▁South@ ExP  ▁times` ▁Abp ED  ▁author ene 　( 减少 ots  ▁understand 意识 ', 主义 ▁Aug  中华0 return@  ▁oftenP ▁Mc` 小编p 确定 ▁Im ds  ▁cult abor 建立 pos utions irl imate ources  欢迎0 ▁ve@  ividualP ster` 道路p ched  ▁food 状态 办理 接触 删除 ada  ▁short 人们 leg 制度  ▁following0 也有@ 最高P 传统` rawp AC 新的 ray 经常  ▁vers 符合 reet  ▁known rew  ▁create ued not0 稳定@  ▁humanP 操作` iftp aring 高考 就业 居民 广大 ger ▁dri ▁pot 员工  ▁five 儿童 版权0 oman@ 受到P ails` 团队p  ▁money  ▁looking 行政 以来 领域  ▁invol  ▁course  ▁live back  ▁compet 明显  ▁North0 而是@ 食品P 设备` idsp 公开  ▁Sept Com ole 高校 改革  ▁learn ▁occ  ▁April  ▁possible de  ulation0 Al@ rollP 地址` 采用p urther comm  ▁What raft  ▁chall hers 取得 登记 cri 明确  ▁ident  utes0  ▁media@ 高速P  ▁having` cemberp  ▁ever ▁jo ml  ▁June year 显示  ovember 证明 面对 ▁won isc   ▁percent0 广东@ 无论P 医学` REp 特色  ▁came 那些  ▁value 山东 ▁di 最终  ▁website 增长  ▁kind ▁Dep   ▁study0 打造@  ▁largeP elling` selp 当时 ape ethod 适合 提醒  ▁President  ▁others 应用  ▁later ▁key  过程中  避免0 正式@ 亿元P 丰富` 下来p  工作人员 ▁Feb 促进 方向  ▁account ▁war 模式  ▁treat aper 新增  ҈҉҉҉   ▁July0 联合@ ▁SecP  ▁Cont`  ▁holdp  ▁house  ▁deal There ▁lim  ▁among ief 观察 ▁ca Con 深入 劳动   ▁review0 相信@  ▁individualP 充分` Onp  ▁companies hel  ▁light  ▁function verage  ▁thought ▁htt 告诉 ctor ▁ris  ▁Serv  iday0 出的@ whP ▁bas` 成了p  ▁makes 医生 用户 urch ilt ▁mar  ▁benef 街道 金融 gr Ph  ▁#0 犯罪@ iodP most` 学历p 保证 iol  ▁field (" 并且 play ▁men  ▁technology ▁Har new ▁Ge  ▁dr0  ▁offer@ timeP  ▁strong`  ▁tookp  ▁didn ▁Em  ▁told But ▁@ 如此 形式 load fect  ▁larg  ▁publ  uff0 年轻@  ▁needsP ST` 快乐p  ▁night  是一个 针对 aps  ▁anal ogn  ▁Will  ▁enjoy  ▁room ▁Gl 统一  成长0 如有@  ▁comesP 感情`  的情况p  ▁thing  ▁mater 新型 做到 线上  ▁months ma  ▁mean 过去  ▁products 说明  aken0  ▁self@ rainP  的问题` 体验p 办法 是在 培养 unch 监督  ▁personal  ▁building 习惯 改变 长期 现代  uf0  ▁clear@  ▁bringP ips` earchp  ▁started ▁Dr ators 立即  ilities now 提前 主题 恢复 结束 idence  施工0  ▁control@  ▁WhenP 审核`  ▁videop 许多 认真 ▁Art cer 目的 ival ches Sh off 快速 To  ▁hig0 ained@  ▁ParkP reg` ▁Agp ournal ▁Me itle  ▁means  ▁Part 理解 分别  ▁least 只能 ler irm   ▁song0  ▁talk@ 当地P ides`  ▁sidep 实验  ▁enough 日本 ▁sum sw 中学 vert 游戏  ▁particular  ▁pain 就能  当然0 ana@ 资金P 本文` 农业p Ar  ▁December 开放 识别 ▁et  ▁November 传播 性的  ▁certain  ▁October  就可以   ▁Comp0 nov@ AMP  ▁final` ▁COp set ze ▁bo head ither ▁tem ▁doc  ▁done ▁War 年龄 ▁wr 体系0 ▁mom@ ▁taxP When` 对方p And  ▁though 先生  ▁quality  ▁ener ison  ▁vari 采取 规范! opy! ▁tre ! ament0!  ▁environment@! luP! 在线`! 依法p! ancial! illed!  ▁given! 智能!  ▁September!  ▁games! ▁One!  ▁appear!  ▁example" irc" 数学 " 并不0"  ▁claim@"  工作的P"  ▁County`"  ▁Healthp" 网上" 面的" crib" ▁enc" ▁deb"  ▁Center"  ▁private"  ▁seen" ▁ant# ▁vol# 公布 #  ▁doesn0# 结构@#  ▁critP#  ▁role`#  ▁initp# 福利# 每个# 青年# angu#  ▁comb# ▁var# 侵权# ▁tot# 一切$  ▁space$ 电视 $ 中医0$ 法院@$  ▁WestP$  ▁land`$  ▁happenp$ ounc$ New$ 电影$  ▁America$ 尤其$ ▁Bar$ ests$  ▁issues$ room% 体育% 遇到 %  ▁national0%  ▁once@% ibrP%  ▁type`%  一个人p% ▁ens% ▁fav% 皮肤%  很多人%  ▁share% 计算% 都不%  ▁doing%  ▁August& 后的&  ▁Afric &  ▁House0& 经历@& ▁ActP& 参考`& 困难p& ▁My& ▁ago& 一天& ('& igital& ▁<&  ▁Stud& ▁Par& 精彩' 营养' umb ' ruary0' ilar@' ▁audP' 意见`'  ▁redup' aur'  ▁press'  ▁stop' ▁Cor' 都在'  ▁content' minist'  ▁discuss' ▁low(  确诊病例( ▁fre ( 核心0(  ▁relations@(  ▁formerP(  ▁complet`( 享受p(  ▁super(  ▁require( oon( Re( ino( 男人( 支付( elt( right)  孩子的) ham ) 防护0)  ▁port@) 天气P) arget`) 程序p) 十分) ▁due)  ▁address)  ▁page) 压力)  ▁getting) 民族)  ▁track) 音乐*  ▁impact*  ▁works * 一年0* 商业@* 距离P* import`* 优势p*  ▁specific* 工业* gress* 民警* ▁Dav*  ▁death* 食物* ▁UK* aid+ Ad+  ibility +  ▁single0+ ▁six@+ 设施P+ 发挥`+ blp+  ▁Austral+ 长按+ anks+ ecut+ ▁pri+ ::+ ington+ 死亡+ 位置, tr,  ▁sound , ault0, ET@, 报考P, Un`, 成立p,  ▁went,  ▁check, itted,  ▁according, ▁Do,  ▁contact, 训练, nes, ▁yet-  ▁Brit- key - emic0- ▁buy@-  ▁responsP- 家人`-  ▁thirdp- 大量- ocr- ille- 近平- fort- 满足- rodu-  ▁begin- ades. 考虑. 身边 .  委员会0. 机制@. 核酸P.  习近平`.  的时间p. 费用.  ▁Mark. 真实. 引起. 重大. bum. 原则. elebr.  ▁represent/ urg/ 交易 / chie0/  ▁energy@/ 文字P/ ▁Sm`/ 认识p/ ondon/ ague/ 录取/ ();/ atur/ used/ ▁El/ 的高/ 日常0  ▁body0  ▁further 0 EC00 女性@0 都能P0  ▁whether`0 asingp0 市民0 事故0  ▁host0 ▁Qu0  如果你0 答案0  ▁beaut0 学会0 ▁bit1  ▁step1 党委 1 atest01  ▁These@1 特殊P1 cast`1  ▁footp1 iter1 下的1 US1 ▁typ1 河南1 领取1 湖北1 下午1 添加2 让你2  ▁international 2 https02  ▁However@2 linP2 整个`2  ▁continuep2 lete2 如下2  ▁period2  ▁signific2 fer2 aul2  ▁leaders2  ▁coll2 ouncil3 ▁By3  ▁true 3 aign03  ▁hours@3 ▁CheP3 增强`3 深圳p3 规模3 ▁cou3 add3 召开3 irt3 ends3  ▁range3 ociety3 日起4 ▁His4 adu 4 BC04 cy@4  幼儿园P4 合同`4  ▁Trumpp4 ▁Her4 开学4 ▁age4 icro4 技能4  ▁quick4 感谢4 累计4  ▁Russ5 oe5  ▁problem 5 http05  ▁added@5 降低P5 alt`5 ▁Regp5  ▁tell5 她的5  ▁parent5 提出5 各地5 ****5 看看5  ruction5 实践6  ▁Cour6 ▁upd 6  ▁February06  ▁staff@6  ▁globalP6  ▁travel`6 值得p6 ule6 生物6  ▁rights6  ▁events6 ama6  ▁performance6  ▁previous6 人数6 atural7 apt7  ▁half 7 数字07 场所@7 restP7 ecess`7 mitp7 智慧7  olution7 正确7  ▁particip7 掌握7 会有7 配合7  ▁heart7 ▁Pe8 制作8  的重要 8 消毒08  ▁member@8  eciallyP8 error`8  ▁actionp8 though8 样的8 高中8 agn8 atform8  ▁miss8 ried8 事情8  ▁mess9 ees9 地点 9 onse09 动车@9 clusP9 冠状`9 记录p9  ▁actually9 设置9 era9  ▁International9 土地9  ▁areas9 涉及9 声明9 ▁esp:  ▁wind:  ▁education : ▁Rec0: HE@: rugP: 一位`: ▁Perp: 完善: 测试: 年度: uk: aged: ▁Not:  ▁contrib: 另外: 运营;  ▁taking; ald ; 在看0; ▁cut@;  ▁breakP; 应急`; ▁Mrp; ▁mot; 今日;  ▁career;  ▁below; 用于; 整体; ▁Law; hern;  ▁question<  ▁meas< 多的 < 优惠0< coming@< hibP< 作业`<  ▁Afterp<  ▁mind< 近日< 本次< ▁hom< ett<  ▁reason< Cl< iences<  ▁security=  ▁famil= 乡村 = ogle0= ederal@= ▁OurP= 部署`= 理财p=  ▁easy= 预防=  ▁office= 心的= 监管= ▁hot= oph=  ▁sent= ▁sw> ▁lo> 酒店 > 回家0>  ▁fall@>  ▁presidentP>  ▁Associ`> ensivep> 日报> 时期> na> aching> 公里>  ▁watch>  请联系> ently>  ▁held? amed? eed ? 官方0? 程度@? 最近P? 总是`?  冠状病毒p? 情绪?  ▁Mich? 强化? ender?  ▁character?  ▁improve?  ▁results? 爱情? lev@  ▁soft@ str @  ▁countries0@ eal@@ 方便P@ 自身`@  ▁Directp@ 位于@ ops@ 语文@ ume@ 每日@  ▁shows@ 老人@ 最大@ kyA 提示A 浙江 A 的心0A ully@A  ▁HighPA  ▁forward`A agingpA 更是A  ▁LondonA  ▁closeA 垃圾A  ▁estabA  ▁collectA ependA  ▁featuresA 城区B iclesB 这次 B div0B 意义@B 共产PB 如今`B  毕业生pB itteB uxB 变得B  ▁recognB  ▁takenB OSB  也可以B ▁NetB 有了C gesC  ▁face C ores0C  ▁court@C 带来PC 证书`C entalpC WithC icultC selvesC 人体C  ▁violC  ▁significantC  ▁perfectC itesC 经典D  ▁existD uild D 转载0D hold@D  ▁popularPD 基地`D  ▁managementpD ulesD ▁waD  ▁packD 不好D  ▁textD accD gingD aughD 指挥E  ▁physE ided E  ▁India0E  ▁issue@E  ▁likelyPE come`E ugepE basedE  不知道E  ▁pointsE  ▁frontE IPE  ▁receivedE  ▁DepartmentE 在家E suF shipF  ▁political F 网站0F 上午@F ▁TVPF arter`F aelpF  ▁attackF  ▁methodF  ▁internF  ▁almostF  他们的F isticF isesF  ▁everythingF uationG  ▁achieG  ▁believe G  ▁common0G  ification@G 不管PG 合理`G  ▁ChinapG 防疫G  ▁policyG  ▁installG 成员G 年来G ifornG emptG  ▁playersG 合格H ▁DonH 首先 H 名单0H  ▁stay@H TrPH cont`H 战略pH flH ▁honH  itionalH faceH 到底H partH ▁AngH 全市H 成本I iresI  ▁behind I  ▁Just0I yd@I  ▁comingPI  ospital`I  ▁clospI 母亲I ▁DayI  办公室I 消防I  ceptionI 事件I cleI borI hingJ 往往J  ructure J 运输0J  ▁various@J 联网PJ  ▁entire`J 动物pJ  ▁currentlyJ  ▁toolJ 职工J  的工作J 永远J 各类J 儿子J 江苏J ▁hitK  是一种K ▁Int K pre0K  ▁sugg@K 后来PK 所在`K COpK  ▁objectK 报道K ADK 强调K 此次K ▁douK  ▁trainingK acedK atoryL  ▁MoreL 哪些 L 各项0L  ▁position@L ▁SchPL 预约`L  ▁offerspL 本科L 再次L iddleL itaryL 一线L 景区L idgeL ▁GodL 根本M ailyM  ▁created M  的事情0M  公安局@M  ▁relationshipPM 下方`M itutepM  ▁especiallyM uelM ▁SanM 扫码M akerM EOM  ▁productionM groundM  ▁probN  ▁priceN  ▁tour N ▁op0N 自治@N  iforniaPN 公路`N 自由pN  ▁modelN  ▁aboveN ▁MonN ledgeN  ▁streN  ▁wholeN  ▁credN 肯定N 不用O  ▁failO 维护 O 出行0O ▁Tra@O  ▁friendsPO 广告`O  ▁originalpO 几个O ooO 案件O  ▁agreO  ▁articleO  ▁replO ▁SerO  ▁livingO  ▁livesP 委会P 找到 P  ▁became0P 一名@P  进行了PP 引导`P ▁DispP  icationP ictionP 照片P  ▁versionP 自我P ranP ypeP syP 配图Q  ▁recentlyQ  ▁investig Q 养老0Q  agement@Q 党的PQ 应当`Q untpQ  ▁releaseQ 专项Q 大多Q cialQ ovedQ  ▁providesQ 校园Q  ▁campaignQ  ▁tryingR  核酸检测R  ▁walk R useum0R  ▁total@R  ▁concernPR ▁ep`R  ▁enginepR 待遇R  ▁pickR AllR 　“R  新型冠状病毒R  ▁roadR ▁barR  ▁policeR  ▁assistS  ▁boardS  ▁celebr S  anguage0S  ▁amount@S 沟通PS ▁Sen`S 下载pS 理论S  ▁financialS 公园S otoS 文件S  ▁workedS  ▁considerS  ▁seemS  ▁albumT 笔试T 句话 T 基层0T 千万@T  ▁includesPT iet`T ▁eypT 对象T  ▁elseT ▁domT ▁DesT  ▁latestT  ▁announcedT amsT  ▁ideaT ▁genU ▁FinU 三个 U VID0U ▁Of@U modPU  ▁similar`U 改善pU  ▁BlackU 输入U 房屋U  ▁selectU  ▁CollegeU 临床U 高的U 获取U 河北V 自动V  ▁file V oud0V ufact@V 美好PV ▁Inc`V 展示pV  ▁KingV  ▁purchV 内的V 市委V 大会V ▁ValV  ▁consumV 注册V mmW ContW  ▁associ W 伙伴0W ▁Air@W 学科PW  ▁reported`W ILpW 基金W ▁logW 感受W ijW 红色W  ▁changesW ambW 下降W 表达X ▁immX 上了 X 查询0X 经理@X  的地方PX  ▁risk`X  ▁decisionpX essionX 必要X 指出X  ▁casesX fordX araX  ▁propertyX ELX ▁morY 不了Y iple Y 的好0Y  ▁black@Y apanPY 四川`Y OTpY 不再Y  ▁successfulY  ▁howeverY  ▁opportunityY  ▁someoneY 美丽Y AtY zhY serZ  ▁fireZ  ▁everyone Z  ▁customers0Z  ▁minutes@Z ▁xPZ  ▁date`Z 创业pZ icaZ  ▁townZ  ▁targetZ 留言Z  ▁GroupZ  网络配图Z ##Z  的生活Z  ▁difficult[ 然而[ name [  ▁questions0[  ▁general@[ 打开P[  ▁respect`[ 近期p[ 搭配[  ▁platform[  ▁Germ[  ▁takes[ 呼吸[ 加快[  ▁anim[  ▁potential[ 还要\ 细胞\ ▁Dem \  ▁provided0\ 于是@\  ▁searchP\ wood`\  ▁uniquep\ ▁rad\ ▁opp\ ▁La\  ashington\  ▁Court\ ▁ess\ 的新\ ply\  ▁released] obile] ▁inj ]  ▁contract0] 明白@] 更新P] 切实`] 出生p]  ▁began]  ▁challeng]  ▁wanted] 轻松] 新区] 贯彻] roups] 制造] 不足^ 就像^ orks ^ vis0^  的方式@^  ▁YourP^ 品质`^ 绿色p^  ▁party^ oring^  ▁increase^  ▁throughout^  ▁needed^  ▁either^  ▁ways^ ▁veh^ 曾经_ So_ iful _ 父亲0_ 思维@_  事业单位P_ 某某`_ 最好p_ ida_ 全省_  ▁growth_ 晚上_ 才是_ ection_ eks_ 连续_  ▁required`  ▁Char` 子的 ` 还能0` ivil@` 广州P` ▁Hol``  ▁latep` 宝宝` 人群` 那个`  ▁systems`  ▁ground`  ▁California` this` 体检` arca 一旦a  ▁pers a 记得0a ▁via@a 人口Pa 可是`a ishingpa  ▁networka ▁Aca 不错a 数量a duca 迅速a esea 整理a ▁calb 它的b RO b  ▁recomm0b 下面@b ▁undPb text`b  ▁insppb lierb fficb ▁argb  ▁fightb  ▁victb  网络配图网络配图b 补贴b  最大的b ▁>c  ▁pictc  ▁dest c 给予0c  ▁partners@c  负责人Pc  ▁leading`c  ▁publishedpc  ▁soonc 比赛c  ologicalc 制定c 改造c ativesc /　c  ▁legalc iated hoodd 爱的 d ▁Pat0d ittee@d 流程Pd 女儿`d ▁talpd umnd 语言d 物质d  ▁necessd ▁redd 以为d 实行d  ▁subjectd ancede ordere ports e  ▁built0e  ▁code@e  ▁playedPe 山西`e ")pe 比例e rece 启动e  ▁printe 供应e 当前e overede urbe  ▁writingf  ▁completef 工具 f  ▁hope0f ▁isn@f mingPf 治理`f 总结pf  ▁outsidef 创造f youf ▁Upf attlef  ▁Fromf  ▁wondf  ▁professionalf  ▁computg actiong arning g ude0g 事项@g 突然Pg 里面`g zzpg  ▁anythingg 公示g  重要的g witterg lightg  ▁emerg 南京g  ▁ratherg ughth ▁Volh ocal h 此外0h  ▁asked@h ▁poPh 平方`h  acebookph  ▁linkh  ▁Generh ▁,h 协会h 处罚h Sph INGh avingh  了一个i  ▁girli 给你 i 竞争0i 更好@i ellowPi  在一起`i  ▁determpi 速度i  是什么i ▁ledi  ▁archi  ▁quitei oicei  ▁Streeti ▁badi  让我们j ▁figj  ▁director j END0j field@j ▁stePj 免疫`j 即使pj 现象j  ▁acceptj ▁sexj URj 教授j  ▁primj ▁votj  ▁moviej 反应k identsk 少年 k ▁neg0k  ▁playing@k  ▁ensurePk  如有侵权`k 院校pk  ▁Philk 本人k  ▁expressk 优先k ysisk 收到k  联系电话k optk  ▁celll ▁Accl 第四 l  ▁projects0l 阳光@l nyPl  ▁Hist`l  ▁coverpl 月份l 终于l  ▁describl rabl ;　l 动力l 脱贫l 就要l inksm 攻坚m  ▁prop m  互联网0m  ▁introdu@m ▁RedPm  ▁medical`m roidpm dom  ▁awardm  ▁cleanm 现实m 因素m  ▁pleasem ▁illm ablesm 事实n  ecutiven 运行 n ▁deg0n orge@n 示范Pn 软件`n 独立pn  ▁parkn 交警n  ▁tablen agern  ▁Davidn  ▁remainn  ▁regardn 就在n 案例o ▁Steo  原作者 o urance0o 升级@o 重视Po 农民`o  ▁userspo mbo ▁пo 举行o downo 优质o ▁abso  ▁patientso  ▁sizeo asticp ▁toup  ▁email p  ▁English0p 高度@p 相对Pp 同志`p  ▁masspp agsp 读书p 放在p  ▁documentp 湖南p relp 想到p стp  ▁favorq 承担q ached q PS0q ▁Sw@q  ▁accomPq ico`q 离开pq 预计q markq cingq 等等q  ▁Washingtonq ceedq 收费q izingq CCr 商品r mp r  ▁exam0r BA@r ▁damPr  ▁expected`r 成果pr  ▁plansr  ▁Fridayr  ▁Foundr viller  itutionr aintr  ▁historr 评价r  ▁Books 研发s uses s  ▁running0s  ▁integ@s  ▁EastPs oke`s  ▁whiteps  研究生s 时刻s 人类s  ▁Canadas 同样s  ▁designeds ivitys  ▁brands  ▁Councilt 粉丝t  ▁answ t 资讯0t 全力@t 鼓励Pt  ▁Sund`t Apppt 为主t  ▁circt  ▁Researcht hat  ▁Const  身份证t  ▁stylet  ▁weekst lau 复工u airs u 们的0u  ▁billion@u 平均Pu 愿意`u Onepu defu 小组u 投稿u secu 也要u 危险u rownu 尽量u ▁hopv irthv  ▁pret v  ▁European0v 顺利@v oomPv  ▁matter`v  ▁studentpv 春节v imsv 您的v 统计v 队伍v  是因为v 从而v  ▁naturalv 能源w 充满w Sc w  ▁included0w 一条@w rsPw No`w 岁的pw  ▁involvedw  ▁affectw  ▁administw wow umentsw classw 广场w 平时w  第一时间x 举办x ▁bi x 用的0x 一份@x  ▁JapanPx  ▁band`x ibrarypx 依然x  ▁userx  ▁goalx  最好的x 主席x  ▁cookx 老年x 都要x  ▁Republicy 课堂y ▁Now y  之间的0y 密切@y 学员Py den`y  ▁seemspy 取消y ":y  ▁moderny  ▁momenty  ▁Profy 详细y 截至y  ▁lossy 各位z vestz  ▁higher z  ▁books0z  ▁meeting@z ondayPz 自行`z 地产pz 还会z ariesz akersz 党史z  ▁ratez  ▁revez  生活中z  ▁Businessz ension{ vey{  ▁innov { resh0{ 去年@{  ▁groupsP{  ▁digital`{ 上市p{ My{ 女士{ Is{ 和国{ istry{ 血管{ 始终{ 几乎{ 特点| 日子| 铁路 | 不知0| ror@|  ▁TransP| 整治`|  ▁simplep| 应对| alls| ION| ref| 哈哈| iles|  ▁region|  ▁manufact|  ▁Dist}  ▁prior} cn }  ▁problems0}  ▁Techn@}  ▁influP}  ography`} 力的p}  ▁economic} 美的} 有着}  ▁lost} II} 生肖} 革命} 天的} 云南~ 安徽~  尤其是 ~ ▁Mor0~ 相应@~  ▁diseP~ 原创`~ Ourp~ ▁sus~  ▁receive~  ▁deep~  aturday~ 居住~ 高级~ iece~ 能量~ 商务 rect 每年   ▁base0  发展的@  ▁detailsP ▁Wed`  ▁Williamp pped  ▁cand  ▁probably 先进 错误  ▁Post 考核  ▁words 家的 贡献 确认 天津 有多 log(  ▁approach0  ▁path8  ▁broad@  ▁programsH  ▁FirstP roughtX  ▁worth`  ▁strih oppp iationx  的朋友 常见 练习 ▁web  ▁interview 重庆 )(  ▁Year  ▁terms  良好的Ȁ 重新Ѐ ▁sym؀ 申报 者的  ▁learning  ▁Review  可能会 心里 逐渐  ▁themselves icated estern( 执法0  ▁treatment8 col@ zyH 往期P  ▁optionsX 青春`  ▁Whileh 家里p  ▁suggestx 主持  ▁screen ;( ares 初中  ▁mother  ▁nation  ▁woman aches raelȁ  ▁averageЁ ements؁  ▁System Id ▁sat  ▁drug mon  不同的 具备 ▁Dec 贷款  ▁sense(  是不是0  ▁itself8 ength@ ))H erousP asonsX Se`  ▁declh  unitiesp operx anch  ▁defin  ▁https etimes  也就是 Pl  ▁sett  ▁regular 投入  ▁titleȂ  ▁storiesЂ 也能؂ ▁Pal 婚姻 资产 半年 生的  ▁written  ▁source 所谓  ▁application 他人( ▁obs0  ▁practice8 伤害@ 放弃H frP  ▁nothingX  ▁sales`  ▁drivh Abp 环节x ▁Out 动作 IM ▁der 中共 每次  ▁sold  的发展  新冠病毒 ercȃ  ▁struЃ  ▁simply؃  ▁round  ▁states 另一 周边  ▁longer  ▁prevent 植物 ening  ▁Program  ▁club( hu0 女孩8 Res@ 平安H 博士P 加上X  ▁Indian` ▁sawh boardp Namex  总书记 色的 集体  ▁activities  在这里 ▁jud  ▁attempt 态度 ustain ayerȄ  自治区Є 推广؄  ▁British erved ▁Are 期待  ▁Institute 这款  ▁safety abel 理念 ado( ▁son0 ▁Tex8 邮箱@  特别是H  共产党P 带着X ▁ut` 空气h 房子p  ▁requestx  ▁parents inary  ▁match 毕竟  戴口罩  ▁cannot 意思 即将 以前 发热ȅ ▁magЅ  ▁Australia؅ avy 福建  ▁knowledge  estival 反复 ago html  高质量 ▁Ob 紧急( After0  人民政府8 acing@  新时代H  ▁implementP  ▁threatX  ▁schools` islh  ▁additionalp 住房x 清楚 身上 word rench ▁в 优化 yp  大部分 asc 温暖Ȇ  ▁automІ 物业؆ 理想 一段 itect  ▁Sunday 相当  ▁Facebook 加大 双方 担心 多个( 发放0  ▁beautiful8 美元@ 在这H  ▁starP iberX oul` iringh wherep 慢慢x irus  版权归 ava  ▁Music  ▁located  ▁response  ▁extra 审查 确实  ▁imageȇ  ▁conductЇ ript؇ esc 坚决  ▁summer 技巧  ▁fast на ▁} 从事 illing ▁cy( ▁Cap0 多次8 还在@ ilsH ThatP  mercialX 第十` PPh 物资p 处于x  ▁color Col 开启  ▁inside ▁rev uesday 是不  mission  ▁culture  ▁AssociationȈ 英国Ј  ▁fans؈ var LE 监测  ▁leave  ▁mach  ▁statement 限制  ▁Director ao ▁hol( 之外0  ▁resources8 突出@ 男子H  ▁softwareP iiiiX 附近` ursdayh 思考p alsex ▁Ear  ▁pred 英雄 娱乐 机场 enge  ▁Some  ▁player  ▁Monday  ▁Paulȉ  criptionЉ oday؉  共和国  ications 状况 dom olar 声音  ▁related  ▁Educ  ▁families 村民( 香港0 上下8 ра@ netH ▁CrP 追求X  ▁pretty` AAh  ▁organizp  ▁anyonex 派出 inter unk 讲话  ▁distrib 公交 ▁box  ▁norm  ▁card 日至Ȋ  ▁encourЊ  ▁paper؊ ▁sen  ▁draw comes  ▁COVID  ▁center  ▁Public 防止 aly  ▁couple  ▁conditions( UT0 蒙古8  ▁comment@  ▁kidsH ▁panP  ▁collectionX 药物`  ▁languageh CHp inatedx 协调  ▁James 我市  ▁father  ▁stage uted arr  ▁prep  有什么 ctionsȋ erveЋ UN؋ 奋斗  ▁phone  ▁Google mas 很难 财政  ▁pric 干净  ▁military  ▁White( String0 二十8 成都@  ▁earnH  imatelyP  ▁surprX 蛋白` Listh  ampionsp APPx  ▁Island omb eren  ҈҉҉҉҈  ▁Cong 安装  ҈҉҉҉҈҉҉҉҈ ▁mid 通报 ViewȌ 收藏Ќ DA، 精准 有序  ҈҉҉҉҈҉҉҉҈҈҉҉  ▁organization  ▁serious  ▁Saturday ▁Ann ▁£  ▁largest ▁den( nesday0 西安8 通道@ ▁ModH 会员P 一体X 利益` 吸引h 兴趣p ▁widx 考察 ▁bed 完整 # ҉҉҈҉҉҉҈҉҉҉҈҈҉҉  ▁businesses itiz 律师  ▁ability  itation  ▁driveȍ 聚集Ѝ TV؍ iqu 覆盖 阳市  ▁Port ston gest 至少 imum  ▁bank pm( rote0  疫情防控工作8  ▁Jack@ 运势H dingP ▁BenX  ▁immed` EMh 学位p ▁plex  ▁developed 停车 ▁squ 衣服 ▁Ant 全体 ▁div  ▁ball 不如  ▁GreenȎ 相比Ў 百年؎ yond 来到  ▁store  ▁upon  ▁Science 前往  ▁Board  微信公众号 力度  ▁instead( azing0 排名8  ▁Africa@ ademH ubeP  来源于X  ▁Develop` 突破h 记忆p 警方x 温度  ▁Hall ▁fit 去了  教育部  ▁Best  ▁firm 绝对 过的  ▁collaborȏ  ▁sensЏ force؏  ▁Angel  ▁block  ▁sche ples 万人 irit 广西  ▁scient  ▁multiple ▁hor( ▁\0 ни8 排查@ 饮食H listP  ▁TexasX 渠道`  ▁blogh limp 检察x  ▁Service 境外  ▁gradu like  ▁levels 公务 idered  ▁Read hest  ▁progressȐ 反映А  ▁providingؐ 各级  并不是  ▁nearly  ▁ready  ▁quickly  ▁looks 解释 Im ▁Sim 退休( 路上0  ▁complex8 En@  ▁causeH  有一个P 眼睛X  lements` ▁Preh osingp  ▁deadx  消费者 唯一  给大家 分类 ▁ded  ▁hous ki ▁nov ▁Fil 令人ȑ  ▁materialБ criptؑ  ▁named 风格 wide using inese 节目  ▁Office olf  ▁growing  ▁skills( aves0 新疆8 每一@ 规则H  ▁DemocrP 最低X 调研`  ▁separh atalp Shex 强的  ▁prem 梦想 sych 形象  ▁teac  ▁teams 予以 原来 陕西Ȓ ▁matВ  ▁regulؒ 下了 ▁wel ▁Je  ▁standard urt  ▁rout Home  ▁himself  ▁Inter 坚定(  ▁plant0  ▁costs8 amily@  更多的H  ▁readingP  ▁MichaelX 主体` 哪里h bodyp  这就是x  ▁independ  ▁Israel  ▁capital 图文 大道  ▁feature  ▁Home band OL ▁Usȓ ▁NeГ udgetؓ 爸爸  ▁exact aby venue 多年 Bl  ▁federal  ▁goes 它们 背景( Get0  ▁employees8 地铁@ ▁oilH 多种P 使命X  ▁chance` ▁INh  ▁Generalp ▁licx ▁Dan 大队 时的 她们  社会主义 化的 支部 颜色 isions 本身Ȕ  ▁seekД % ҉҉҈҉҉҉҈҉҉҉҈҈҉҉҈ؔ  ▁trade 协议 ▁ver  ▁wide 女生  ▁cross 分数 ▁ord ounter pose( 污染0 lick8 广播@  ▁scienceH 义务P IVX  ▁race`  ▁Superh uyp Indx 一家 营业  ▁Mart 依据  unction 江西  人民共和国 see 谢谢 国务ȕ  ▁displayЕ 说的ؕ ka 回顾 长的 件事 不起 lying ▁Get 一生 ▁Ev ▁ten( men0  ▁subs8 nown@ ▁musH  ▁whoseP  中华人民共和国X 应聘`  ▁Tuesdayh 好了p 收获x oes rian  ↓↓↓ 贫困 工艺  ▁Rich 观点  ▁Great 探索 详情Ȗ rialЖ 财富ؖ 有所 ▁sil 检验 rast 上述  并没有  的同时 体温  机动车 Or(  ▁ideas0 摄影8 明星@  ▁maintainH  ▁decidedP 上涨X  出来的`  　/　h  ▁broughtp 诈骗x  ▁England ona 禁止 rated  大多数 ategy  ▁workers 违规  ▁official 开心ȗ  ▁communitiesЗ  ▁achieveؗ  ▁acqu 统筹 人在 通行 美食  ▁Thursday но 担当  ▁parts 形势(  ▁wasn0  ▁click8 lam@ ▁mixH  ▁WindP ▁DelX 扶贫` 感到h 配套p  aughterx ties ко  ▁effic  ▁wonder 内部 十年 ae 举报 ipment :(Ș 公益И 挑战ؘ  ▁Rober 结婚 有时 刺激  ▁rock  ▁served 不住  ▁respond  ▁neigh 的不(  ▁candid0  ▁Here8  ▁Sing@ ')H 有点P  ▁LifeX  ▁investment` encesh 复杂p  ▁opportunitiesx 变成 ▁boy 假期 融合  ▁saying 化学 懂得 做法 权益 一场ș houseЙ 评估ؙ  ▁streng  ▁considered 实力 多了  ▁demand iger  孩子们 ро  ▁Press 是个(  ▁word0  ▁wrote8 win@  ▁reachH  ▁attentionP 股份X  ▁Conf` 四个h hapsp 良好x 将会  都可以  ▁reports  ▁Services  ▁lower 印度 组成  ▁suff  ▁efforts itarȚ 职位К 熟悉ؚ 人大  ▁gave  ▁capt  ▁allows  ▁trust  ▁Gold ▁cam  ▁Church  ▁huge gen( bon0 或许8 考研@  ▁happyH 类型P eckX 休息`  也没有h  ▁Wednesdayp  ▁qualx 内心  ▁towards PA icip 助力 老板 iant bit 夫妻 分为ț 科研Л ios؛  ▁http 补充 世纪  ▁safe 体现 出租  每一个 ▁bal 加工 微博( 也许0 很快8  ▁govern@  ▁ArchH 下去P itcX  ▁contain` 食用h 神经p  ▁evidencex sequ 省委  ▁highly (　  ▁Flor  ▁gets 嫌疑 让我  ▁variety  中国共产党Ȝ BeМ cel؜ 素质 引发  ▁choose  ▁Twitter ache 本地 三年  ▁situation  ▁Commission  第一次( istan0 师范8  ▁message@ ▁BerH 称为P  ▁locationX ota` icineh morep ▁Magx  ▁Foundation  ▁enter met  ▁Company  ▁positive  的作用 berg  生活的  国务院 withȝ ancyН OP؝  ▁Award oyal za aled 司机 TS 血压  ▁usually 非法  ▁spent( 种植0 change8  ▁born@ 链接H  ▁expertP  ▁religX  ▁deliver`  ▁promoth  所有的p 随后x bre enty  ▁seven 出口 认定 扩大 很好 贵州 税务  ▁handsȞ 而言О ▁с؞  ▁traditional 准确  ▁Work ▁Mem 就不  ▁Thom  ▁speed  ▁necessary  还可以  ▁arriv(  ▁population0  ▁fair8  ▁wrong@ 文学H 效率P 专题X  ▁Education`  ▁clientsh 温馨p pperx iding  ▁shoot  ▁lack 角度 ▁hum  ▁thous 天下  ▁expand  ▁download 判断ȟ  ▁benefitsП lished؟ 我是  ▁century  ▁copy 流行 意味  ▁giving ocks 围绕 法规  ▁bott( While0 手术8  ▁earlier@  ▁givesH 会计P  ▁choiceX data` 辅导h jsp  ▁survx 理由  ▁customer They ▁Bel  ▁exhib ____ het ▁Ke 早上 大型Ƞ  ▁sellР 味道ؠ  ▁demon 好好 Man 很大 ▁Met 包含 chan date  ▁restaur 建成( ▁etc0  ▁increasing8  ▁nature@ 就有H  ▁interestingP hiX 动态`  ▁farmh ▁dogp 人物x 成分 奖励 mitted  ▁Franc 线路  ▁solution 随时  及以上 AB  ▁quarterȡ 看着С  ▁leaderء ▁Mac box 几年  ▁Over  ▁write 人力 加油  ▁hear  ▁eight 青岛(  ▁Road0 生长8 舒适@ ▁THEH  ▁climateP  ▁feltX  ▁cast` 两人h  ▁comfortp ▁femx fully  ▁excell ▁Leg  ▁touch just gl  ▁strateg  ▁wife  派出所 损失Ȣ  ▁morningТ 推出آ 传染  ▁French ya From 救援 采购 ▁aw 太阳  ▁Bank ancing(  ▁detail0 ▁sn8 cm@  ▁journalH liedP olicX CT` ▁bush questp  习近平总书记x  ▁solutions 日前 气温 风景 place 战争  每个人  ▁tick 女子  ▁saveȣ  ▁commitУ 放心أ istics 深度 iller  ▁credit 妻子 Post  ▁starting leep  ▁directly IG(  ▁avoid0  ▁section8 简介@ 十二H  ▁hospitalP ▁AddX ▁Mal` ▁Tomh Morep esterx bed 备考 财务 自主 PR 进口  一定的 难以 care  ▁vaccȤ 时尚Ф 百姓ؤ  ▁Museum  ▁creating  志愿者 讨论  ▁strugg  ▁effective ▁gas  公务员  ▁college 评论(  ▁individuals0 itch8 arily@ 人士H 手段P 拒绝X  ▁bill` innh >>p  ▁awarex  ▁died 名字  ▁launch  ▁mention undred  ▁sports 小的 ashion 时光 宣布ȥ 面临Х Valإ  ▁League ▁lif 杭州 发表 Read  ▁artist 天然 ▁PM  ▁construction 出去( 论坛0 iverse8  平方米@ 会上H  ressionP  ▁optionX  ▁disease` ▁ANh 健身p owersx  ▁characters  ▁helped 邀请 清洁 incip 运用  ▁places 成就 这位  朋友圈Ȧ  ▁HistoryЦ  ▁limitedئ  ▁feed ults 很有 Acc  ▁liter  ologies han porate  ▁convers 后台( ▁Bre0 硕士8 赚钱@ 循环H 提交P ·　X  ▁Class` 营销h  ▁beyondp 中小x urrent 曝光  的人生 进步 ▁Str  ▁Mary  ▁effort  ▁civil orage 实在ȧ  ▁registЧ 关心ا 司法  ▁fully 锻炼 刚刚  ▁particularly 尊重 上班  igation ▁Mus  ▁yourself(  ▁inform0 reme8  ▁materials@ 指南H ▁salP  ▁rollX 县委`  ▁dailyh Prep fox  同学们 icon uan ecial  ▁annual  的一个 ulture  ▁piece  ▁behav  ▁consultȨ  ▁launШ MAب 医保 NA  ▁Government ourney  ▁green 合法 趋势  ▁force edy 包装(  ▁Child0  ▁reflect8 Type@ anderH DeP 纷纷X  ▁transport` ▁ulth  ▁appointp  ▁usesx  ▁officials 广泛 不限 bt porary 隐患 书法 养殖 仍然 子女ȩ  ▁conceptЩ ▁misة  ▁degree  ▁birth  ▁Journal  ▁analysis 装修 ▁ap dd 告知  ▁consist  ▁Intern(  ▁Fore0  ▁ones8 ▁ri@ ▁sweH 在于P 热线X  ▁computer` 处置h elligp  ▁Clubx  ▁Star eat 缴费 ▁Lou lor 利于 odes  ▁society  ▁answer  ▁heardȪ SSЪ  ▁recommendت 住宅 ▁Def 旅行  ▁Most  ▁Play 适应 等待 纪念  ▁electric  ▁stream(  ▁commercial0  的影响8  ▁stra@  ▁CongressH 血液P 不良X  ▁dream`  ▁movingh 拍摄p erox yth 传承  的基础  ▁attract  ▁economy rief ste  ▁Chinese ca  ▁planningȫ 讲解Ы 把握ث  ▁moved rive 单元 异常  ▁subst  ▁beginning icate  日上午 扫描 指标( 先后0 上方8 一边@ 传递H 当天P 仪式X  ▁marketing` 客服h  ▁missionp 业主x rim 打击 蔬菜 生素  ▁vill 水果  ▁increased 不想 ta  ▁footballȬ 创建Ь iblyج 资本 移动 ▁har ▁Sam  ▁Development ▁Rel  ▁Florida  ▁Camp ▁fa annel( ▁Rob0 ros8 hens@  ▁stringH  ▁allowedP 瞬间X 药品` 第五h 职责p  ▁shotx Not rey ▁Nov  ▁mobile 适当 成熟 comp  ▁types 游客  版权归原作者ȭ 名的Э 初心ح 最佳 arden ▁II 缓解  ▁changed 不够 encies ▁CEO acity ▁ter( 名称0 itable8 超市@ 性格H 机械P hemX lear` 更有h ▁Milp 整改x 吸收 细节 ▁Cam 自信 展现  ▁Minister 承诺 深刻 but rieȮ 在此Ю 东北خ ><  可以说 ened  ▁wild  ▁gold 收益 irgin 周年  ▁items  ▁experienc( azine0 isode8  ▁challenges@  ▁categH 截止P 复习X  ▁Project`  ▁sharedh  ▁servep 临时x 面向  ▁clin  ▁goals ( !▁▁▁▁▁▁▁▁▁▁▁  让孩子 城乡  ▁remember  ▁continued 做了 餐饮ȯ 走进Я ▁Bigد 情感 fig 典型  ▁District  ▁numbers  ▁mount  ▁fresh isco  ▁amazing  ▁easily(  意味着0 通常8  ▁Rock@ ▁AMH  ▁slowP SEX ▁Sur` 也就h FFp ~~x azon 主管 ▁Sal 祝福 ▁Bay  ▁active  ▁desp 汇总 乡镇 ▁idȰ 面前а  ▁studiesذ umber  ▁election ▁er 作出 门诊  ▁organizations  有没有  ▁George 睡眠  ▁▁▁▁▁( 互动0  ▁wood8  ▁Times@ 全程H  ▁physicalP 怎样X no` iledh Lep printx  ▁sustain  房地产 及其  ▁speak 高效 igned icago vant 赶紧 izesȱ 上面б  ▁wantsر ▁Att 意外 ▁Top GB 的前  ▁Smith ken ying  ▁turned ▁Mex( 一路0 专门8  ▁grand@ ▁RetH  ▁favoriteP 人工X  ▁Christmas` 做的h 马上p 入口x 过了  ▁meaning  ▁Committee 寻找 gu  人民法院  issions 线下  ▁Open  ▁WomenȲ  ▁imagв 自觉ز 做出 海南 房产  ▁Carol  outhern  ▁biggest soft ords  ▁tools 鸡蛋(  ▁understanding0  ▁images8 回到@  ▁emotH ▁SyP 各个X rupt` ▁droh 将在p ▁USAx 外出  ▁finally 正是  ▁initi  ▁ahead 彻底  ▁established  ▁rules ▁Jew ▁…ȳ utyг  ▁carryس  ▁remains 布局 Qu  ▁Society term 发动  ▁Good  ▁Care 相互 life(  脱贫攻坚0  ▁Alex8  感染者@  ▁darkH atreP inallyX  ▁equipment`  ▁budgeth ▁Quep  ▁Professx  的通知  ▁legisl  ▁device mar  的情况下  ▁paid 上一  副书记 ecause  ervationȴ  ▁Peopleд 警察ش  ▁join  ▁foreign 车型 anta  人民医院  中小学  ▁Management  ▁correct 场景  ▁blood(  ▁challenge0 iers8  ▁effects@ 出版H 到的P 标志X 师生` onenth 填写p 成年x SC 管控 点赞  ▁tried  ▁cool ronic RA  ▁funding  ▁assess  ▁Mediaȵ TAе 定期ص 几天  ▁background resp 新鲜 Data  ▁sometimes edom 履行 同比  ▁owners( 元素0 ▁Los8 ▁tim@  ▁policH  ▁sendP  ▁sectorX 森林` IAh 军人p 黄金x  人民币 Don eared  ▁Plan  ▁interested 中考  ▁completely 抗疫 华为  ▁Everyȶ  ▁occurж mailض  ▁opening 养生 满意  指挥部 带领  ▁altern  ▁sale 否则  真正的  内蒙古( ait0  ▁partner8  ▁Bill@ etingH  ▁pullP  ▁FranceX  ▁floor`  andemich ouslyp 一致x 氛围 呈现 anned 病人 共享  ▁applications PC  ▁names 居家 紧张ȷ  ▁Riverз Intط ▁» ▁sun Mod 想象 SU  ▁artists  更好的  ▁spread 出院 原文( ▁Mat0 ji8 ria@ 心情H oundsP  ositionX  看起来` 苹果h 太多p 说话x 流动 top edia  ▁operations  ▁benefit  ampionship  ▁Under ▁fix ], ▁Moȸ  ▁vehicleи 局长ظ  ▁highest  的方法  ▁knew xx ▁Sl 大利 层次  ▁Miss igr  isation(  ▁insurance0  ▁citiz8 ▁UN@ 出发H  ▁continuesP 物品X 不在` 功效h ▁lovp 一是x  ▁cert iny 电脑 信用 付出  ▁African ien  ▁jobs  ▁dram ▁Subȹ arantй priع 回答 prof 振兴 足够 高铁  ▁smart 对此  ▁stock 罗斯 ▁max( 通告0 会在8  ▁critical@ FLH  ▁overallP FrX Over` wellh pointp  等方面x  ▁Policy lands 支撑 出门 look Te  ▁voice 纪委 erry ifyingȺ 航空к agramغ  ▁conv  嫌疑人 之中 首次 关节 apter 秩序 overy atab ▁pm( 独特0  ▁door8  ▁Other@ 一张H 战斗P  ▁HillX 解读`  ▁cancerh 造型p 十四x  ▁senior  ▁sites Do  ▁values  ▁surround  ▁hundred  ▁movement 热情  在这个  ▁fineȻ omaл 创作ػ state 似乎 允许 前的  ▁difference  ▁peace itely Go 无数  ▁produced(  ▁status0  ▁condition8 陪伴@ ▁WebH adyP 治愈X 直到` 流量h 试验p Mex  ▁mill ▁IS 也在  ▁Christian 用地  ▁followed ius  ▁audience 图书  ▁leadershipȼ  ▁sceneм 星期ؼ unte  ▁miles  ▁Community  ▁helps  ▁songs ▁Pet 气质  的原因  副主任 好看(  ▁obtain0  ▁residents8  大学生@ 观众H 物流P  ▁thinkingX 此时` 解放h 热点p  的人员x yan ▁lay IR  ▁apply 明天 也很 固定  vention  ▁passed 当中Ƚ ▁Madн  ▁Timeؽ LC Su  ▁spot  ▁existing 丈夫  ▁Scott  ▁Arts 认证 观看 ▁Hot( ▁du0 表面8 的那@ roudH  ▁precP  ▁professionX  ▁church`  levisionh gencyp 消化x 　, ▁die iately 诊断 ▁Bra  ▁activity 涉嫌 rd 省级  的孩子Ⱦ 营造о 央视ؾ uing 事务  ▁lands  ▁responsible 科目 ano 金额  standing iec  中国的(  不一样0 ▁cru8  ▁doll@ 防范H ulaP ▁SunX 环保` 吃饭h iarp 例如x iling  orthern 社保  ▁shall MS 就算  ▁Angeles  ▁middle 大概  ▁frequȿ 手续п ▁Hamؿ 聘用 痛苦  ▁Design 违反 尽快 信心 同步  ▁completed 是有 防治(  ▁purpose0 超级8 规律@  人力资源H 大众P  ▁strategyX  ▁vote`  ▁brotherh  ▁offeredp  ▁architectx 美术 护理  ▁extrem ▁ge  ▁offering works 辖区 写作 ▁Aut 看来 semb ini 你会 含有 脂肪  uration 路段 给我  ▁purchase  ▁double 引领 ▁bur( from0 iment8 同意@  ▁novelH 回来P ademicX  ▁Kore` agonh  带来的p  广东省x  ▁dang  ▁Social 纪律  ▁attend df 相同 同事  ▁cris ▁aim  日下午 ▁Hel  ▁Then 园区  丰富的 尽管 IF ped 才会  ▁Free 出售 东方  ▁load(  身体健康0  ▁reduce8 ▁sch@  ▁transferH  ▁weekendP 构建X 天天`  ▁Appleh 想法p ▁Cupx 甘肃  ▁virt orney 校长  ▁prices 所得  ▁refer 批准 ▁Sum  无症状 efore 教材  ▁entertain 刑事  ▁anti  ▁powerful  ▁photo  ▁advert 配置 样子 指数  ▁feet(  ▁compared0 Why8  ▁Central@ ▁semH 社交P pectsX 二是`  ▁granth  ▁Leadp 无人x 遵守 二手  ▁helping  ▁Union 这两 reate ▁Rem  ▁throw ▁reb ▁mer 其它 ер  ▁rates  ▁wall 平衡 ette Next  ▁conference 为你  ▁false  ▁note  ▁Micro( rang0 心中8  ▁basis@ 期的H  ▁trendP  ▁demonstrX ▁Sol` OMh 放松p  ▁joinedx  ▁trip  ▁Russia 而不 中间 BS 精选  ▁requirements го 信号 UL  ▁arrest 财产 订阅 购物 地位 等级 蓝色 PD inner  ▁Mont ynam zer(  ▁researc0  ▁journey8 有个@ 可见H  ▁talentP  维生素X  ▁charge` 山区h itudep  ▁happenedx "); Up  ▁lines adian  ▁Even 医护  ▁picture 贵的 发出 跟着  ▁Human ivery usband  ▁weight  ▁spirit  ▁paint aud imal  ▁advant  ▁train 总体 注重( ▁Tor0 ▁Pri8  ▁Econom@  ▁ShowH  ▁WeekP 辈子X ctors`  ▁althoughh  ▁exactlyp  ▁exceptx  ▁Series 道德 肌肉  ▁establish retary ruit ▁Jo 填报 转发  很容易 ulf 周岁  ▁sort Add lo 是最 type  ▁myself  ▁hour 低于  ▁Global ▁ST( 岁月0  新冠肺炎疫情8  ▁primary@  ▁VirginH  ▁ChicagoP 含量X ▁gun` 为准h alap 胜利x 董事 wise 落地  ▁structure 被告  版权归原作者所有  有很多 读者 ▁enh 快递  ▁fear 辽宁 Ed  ▁central 构成  新型冠状病毒感染 缴纳 解析 日期  ▁poll  ▁competition test( ▁Ins0  ▁immediately8 论文@ 洗手H  服务中心P 留下X 回应` ressedh pathp andax 分子  ▁pressure 乘坐 疾控 商家  ▁certainly uled 身材 includ ervice  ▁associated 照顾  ▁comments ▁pen ▁nom antly  ▁rent  ▁void  ▁alone  成为了 equ  公安机关( 至今0  ▁famous8  可能是@ 从来H  ▁EngineP  ▁greaterX Phone` 矛盾h  ▁Fundp 大小x 接到 road ▁www 早安 ($ 门口 witch  ▁posted  ▁scen 命运  ▁Technology ▁vs ▁que 职务 不出 inate ▁beg  ▁station 物理 搜索  都没有 asy( 疑似0 Now8  ▁killed@  ▁partnershipH ademyP nergyX  ▁supply` 依旧h xyp 小心x 深化  ▁records  ▁Robert  ▁opened  密切接触  俄罗斯 放入 aming  北京市  来看看 考点 账户 装备  ▁despite 聚焦 ▁Tur etic  ▁homes  ▁nice 演员 About  ▁retail(  ▁error0  ▁developing8 竟然@ 等奖H 停止P icksX 担任` orterh bolp 我就x  ▁manager estic 上升  ▁photos  clusive 仅供  ▁calls 群体 起了 医师  igration ▁Bi 当年  ▁models  ▁Brown  ▁Elect 我想 ▁cry 党组 到达 ▁rob 部位( 逐步0 isf8 eline@ 魅力H ▁graP 学术X  也不会`  ▁measureh anticp 决策x 疼痛  ▁previously  ▁Grand  ▁agreement  无论是 going 发送 igan  ▁claims  ▁daughter  ▁films 行驶  ▁produce van 成交 退役  ▁pandemic bert 室内 完美 道理  ▁temper(  ▁Family0 打印8  ▁Well@  pectiveH  ▁devicesP 贸易X orial` 离婚h Obp  ▁clientx  ▁coach  ▁thousands 清晰 纳入 登录  ▁Data  ▁appropri  ▁Germany  ▁context 背后 Dr 大幅 ocol 地理 定位 机器  ▁shown 类似 签订 itors LO 量的(  ▁push0 珍惜8  ▁Information@  ▁warmH ▁SkP  ▁letterX  ▁administration`  ▁incomeh ▁Letp  ▁Officx ▁il  ▁Frank  ▁propos 条例 党建 vere SP  ▁setting  问题的  ▁spend CA  的东西  实际上 ▁pet ▁Div  ▁launched 显得 sm  ▁reasons  ▁chem 设立 CE( rad0  ▁independent8 ~　@  ▁IranH TheseP  人民群众X  irection` imbh 过来p  ▁protectionx ENT 大赛 合适 gar  第一个 itchen  ▁incred  ▁views bour  ▁excellent pat 夏天 印象 ▁mur 季度 Can  ▁wait 知名  ▁passion ▁Rad  ▁straight  ▁opin( MP0  ▁pages8  ▁Team@  领导小组H  ▁ConstP ▁EUX ena` rapyh  ▁Centrep  ocationx SA 电池 编制 姐姐 祖国 节日 补助 ▁Av  安全生产  的感觉 OD  ▁Internet  怎么办 ▁Sil 保健  ▁Rights  学生的  现在的 空调 地块  出现了 roy(  ▁cities0 ▁Tre8 携带@  ▁lovedH urlP 性和X  ▁reality` ▁Deth  ▁fundsp  ▁presentedx  ▁entry  ▁executive  ▁creative ▁• 给了  ▁advice 常态  ▁rank itness 当你 ework  ▁estim writ  ▁struct 封闭 附件 ▁Jul  erences  ▁violence  ▁Form Americ 热爱( 龙江0  ▁environmental8 known@  ▁partiesH  情况下P 一批X 利润` 报警h  ▁writerp 团结x  ▁Festival ipl ▁и ita onents  ▁cars 肿瘤 休闲  ▁Step  ▁master da 危害 突发 lt ▁eth  ▁Please ▁Vis can 众多  高速公路 ipe  教育局( iance0 ▁Ir8 ato@ 号线H  ▁SecurityP  ▁culturalX 人都` 纳税h gnp 采访x 不动 到位 ▁OF ieces  风险地区 ▁fan  ▁truly Here 山市  来源于网络  和社会 friend  ▁chief  ▁raised 爆发 高等 姓名 oms 一级  ▁Russian  ▁Americans 组合( 每月0  ▁About8 ▁Wil@ 表演H  ▁apprecP 老公X 对比`  ▁feelingh Amp  ativelyx  ▁signed 实习 ias ▁van 台湾 通风  ▁Chief 转移 强烈 7 0网络配图网络配图网络配图网络配图 ba  ▁normal pack  ▁afford 错过  ▁connection 新华 电商 反而 大力  ▁score 路口( 看见0 ▁Bet8  ▁machine@ 开通H 欣赏P  近年来X  ▁poor`  ▁Privh  一个月p 彼此x 概念 站在 气象 上线  ▁experiences 审批 世间 td px  ▁Love ▁lab  ▁Mass  ▁dedicated  ▁guid  不得不 积累 CL  ▁skin  的一种  ▁capacity public  很好的( 推送0 花园8 至于@ 指定H  知识点P atersX allery` eduh 解除p  管理局x opping 民生 入学  ▁standards  ▁Down 打卡 Rep ella 周围 iminal  ▁guide 头发 也没 性能  ▁driving  ▁regarding 你是  ▁Power  ▁majority  ▁wish 和谐  ▁Mount(  别人的0  的要求8  ▁compl@  ▁adoptH ▁AtlP azzX  According`  ▁Listh  ▁basicp  ▁Lakex  ▁Enter  ▁patient  ▁Wood  ▁Town  ▁Party 官网 缺乏 部长  ▁decades 了吗  ▁update ideos  ▁eyes 好友  ▁psych 户籍  ▁Vict dule eful  联系方式 sk ario(  ▁proud0  违法行为8 ▁Iss@ 吉林H ternalP 现有X  ▁runs` 求职h  ▁rarep 行的x 尝试 NO 房间 一片 展开 aling 两年 工人 ▁Mer  ▁files 立刻 active 外的 eless то 产权 ▁о  的主要 学期 精品  ▁Head 聊天( ▁tit0 爱心8 大师@ uddH  ▁InvestP 基因X  ▁announce`  ▁microh  ▁Louisp  ▁Manyx 不多 ▁eye  ▁agency inem cul bar 伟大 thlet asket 失败  ▁husband  ▁vision 原价  ▁species  ▁facilities  ▁policies anging 城镇  abilities 格局 迎来 素材(  ▁definitely0 博物8 特征@ ▁escH  ▁mechanP 反馈X 监察`  条件的h 坚守p  ▁arenx ooth  atabase  ▁stuff 路线 电动  ▁disapp  ▁experienced 男女 联盟  ▁testing  actions 失去  ▁cash  ▁princip  ▁talking  世界上  ▁trial orrow lymp 修复 证据 ▁map(  ▁crisis0 德国8 年代@  山东省H  ▁TourP  ▁goneX  的故事` 薪资h 加速p 队员x 校区 ▁Mel ban  ▁focused  ▁pattern  uilding  ▁district  ▁Peter 爱你 的水  公开招聘  ▁Thomas 上课 ▁emb  ▁seeing 便利  ▁larger  ▁looked  ▁Water ▁ang  ▁figure 返校(  无症状感染者0  ▁proced8  ▁elements@ aughtH  ▁absolP  ▁initialX 权利` 军事h uriesp kgx  公司的  ▁exerc 简历 授权  ******** iler  党支部 常常 不开  ▁Earth ▁Why Car ▁sy 韩国  opyright  范围内  ▁Valley  ▁couldn 作文 cell % ▁▁▁▁▁▁▁▁▁▁  ▁becoming( 对外0  的学生8  ▁Johnson@ ”(H  ▁ReportP 葡萄X 辛苦` phoneh  ▁harmp  ▁Sincex  ▁essential apers  ▁generation  ▁owner ▁Gen 维持 人事 burg ▁Oh 漂亮 难度 抖音 osh riage 高端  ▁pros  ▁Asia  长时间  ▁experts 舒服  ▁Western aste(  ▁fant0  ▁gain8 思路@ 玻璃H 聪明P 保留X 工厂`  ▁translh 不见p 好像x  ▁perhaps DP  ▁null level 电梯 Mar ▁Er 认可 咳嗽  ▁hotel 之下  的机会  ▁unit craft  大家都  ▁editor  ▁damage  ▁north ML  ▁solid string 年底( 清理0  ▁blue8 mary@  ▁internetH ogueP  ▁showedX  ▁length` cherh ▁catp ▁Burx ▁End  有时候  给孩子 办事 十三 不敢 复产  ▁Medical  ▁sexual 他说 zen 地的  ▁crow  开发区  ▁eval 男性  ▁sources night  卫生健康  ▁described 共有 人心( 为何0  电视台8 可爱@ CompH  贯彻落实P  ▁appearedX 真是` ingerh 元的p 视觉x  ▁Long 部队 Some  ▁repe pid 是指 扎实  医学观察  emporary  entially 玉米  ▁analy  elligence  ▁committed  我们要 品种 des  ▁thanks 角色  ▁href 此前  ▁department(  ▁Also0 ▁See8  ▁Main@ thatH  ▁GermanP  ▁lawsX 网红` ▁Timh ppingp  ▁forcesx atin rc 车的 连接 }, 心脏 满满  ▁radio  ▁Awards  ▁Game 表情 修改  ▁winning 开工 便是 ▁Fre  ▁toward  ▁catch 感恩 车站  人员的 暂停( unning0 进展8 院长@  ▁reachedH  专业的P ▁NYX user`  ▁sporth  ▁stepsp 区别x ns  ▁south hab ▁Ill  ▁Jose  ▁brain  ▁markets 有趣 浪费 长沙 ”) 周末 ▁Jud reek  ▁glass  ▁concerns 不停 引进  段时间 爱好 三大 ен( 试点0 ffee8 奉献@ handH  ▁weatherP 基于X  ▁closed`  ▁Amazonh  ▁occasp  ▁accurx  ▁Martin  ▁hair  ▁citizens  ▁Paris 是对  ▁Special ▁Mur 新生  ▁rise  ▁caused  ▁Their lict 欧洲 atever  oration  市公安局  ▁mental  ▁Richard ▁Nor  ▁healthy  接下来 现金(  ▁Stat0 ▁Mot8 黑色@ ▁DenH  ▁neighborP RecX 两种`  ▁learnedh 破坏p 口腔x  ublished  ▁youth  请联系删除  ▁fashion  ▁finished 常委  ▁manufacture 家中 不上  工作中 的美 使得 cean  ▁kept HS isher ▁Mov omet %; 特朗  ▁driver 收集( value0 响应8  ▁storage@  ▁limitH 逻辑P 仔细X 一件` 板块h каp  ▁browx  ▁practices  ▁impl 预警 aren essage ls  ▁volunte  医疗机构  ▁Food  ▁Sports  ▁Windows  我们将 策略 enses reland  ▁prison  的行为  ▁trou  ▁faith ucky 回忆  年轻人(  ▁transform0 Form8  ▁approx@  ▁drinkH  ▁addingP  ▁AustralianX 兄弟`  ▁exploreh  ▁windowp  ▁technicalx 餐厅 ▁anc  生态环境 ▁Aff 步骤 西北 表明  ▁older  在中国  小伙伴 QQ pected 听到 ulty  高质量发展  的内容  ▁maybe obe  ▁Mexico  ▁Today DC 实用(  ▁extremely0  ▁strength8 ▁IT@  的学习H  ▁chairP  ▁meantX 究竟` 分布h 第六p ▁racx  ▁Arab  ▁delivery  ▁session  ▁Academy 三角 季节 ▁Spe  ▁Police 玫瑰 调节 均为 while 转型  ▁restrict 活力  ▁measures  特朗普 全身 权威 ola  没想到  ▁Oper(  ▁counter0  ▁television8 news@  ▁playsH  疫苗接种P  ▁noticeX  ▁female` 政协h 抗击p  ▁agreex ▁AP  为大家  学习教育 护士 ▁Jim ups 地下  ▁selected  黑龙江 财经 外国 月底 周期 ▁*/ ва 对应 iat ’,  ▁Indust  ▁becomes 进去  不需要(  ▁lock0  ▁officer8  ▁obvious@  ▁BackH 军队P 温柔X  atically` 对待h 政法p alledx  ▁Network 走向 rid 增值  ▁Energy asure  ▁surface Pr mes 辅助  ▁rich 法治  ▁numerous 卫健 用来  工程师 版本  老年人 ▁` AG 睡觉 ▁ban( pan0  ▁appears8 顾客@ 消失H attersP  ▁bringsX  ▁Canadian` hosth yerp 户型x 本来 西南 恋爱 郑州 osure xi uclear 咖啡  ▁infrast  ▁operating  ▁forms  河南省 span 医药  ▁spect 家乡  ▁Film irmed  ▁fram  而不是  有多少  ▁requires( illa0  ▁actions8 飞机@ 让他H inctP edsX 荣誉` 发票h abinp 守护x  ▁Lear  ▁thank %) 忘记  ▁Royal 请您  ▁traffic 拓展 och 陆续 或是 const ▁org press itage ili 你要  ▁heavy HO rick  ▁researchers ▁Jes( 一一0 XX8  还没有@  HoweverH ▁SafP  ▁LandX  ▁Virginia` ['h 康复p idenx ipping ▁thr 代谢  ▁Georg John prene 带动  ▁returned 负担 ▁cas kins  ▁easier 强大  ▁sched 头像  ▁Fire 出台  ▁estate nel  一次性 oses onav( 必然0 合肥8 rics@ 局部H  ▁classicP  inationsX 春天` ▁Whoh  ▁Fortp 口感x 海洋  ▁teaching  ▁street uit  ▁task ses  ▁finding  的能力  ▁facility  ▁Olymp  ▁Obama 每周 plate  ▁controll 全县 预测  ▁actual achel 财运 重症 fol ipped( 阳性0 ▁cop8  让自己@  ▁WallH  ▁battleP  ▁decisionsX 套餐` liesh okenp 监控x iques  ▁instit  ▁movies  unately  可以在 消除  ▁beat  reprene  长按识别  要注意 抓住  ▁ended  实验室 oria 一人  ▁sleep 说了  ▁shop  ▁girls bo abled  ▁memory(  ▁advance0 策划8 片区@ 看了H лиP ostonX 一周`  clusionh 长江p  ▁Libraryx 车主 ▁Joe  ▁Last 侵犯 ▁Two 心态 携手  ▁presence  ▁vehicles ▁amb ▁% ani 许可  小程序 印发 一代  ▁border icians ▁Bo  ▁contains  ▁noted 用人( 气的0  ▁investigation8 红包@  ▁CathH 氧化P 协助X 焦虑` 和平h 安置p  ▁appealx 就诊 简直 Last  ▁Justice  乡村振兴 点个 保存  ▁Carolina 野生  ▁core ▁gar 精心  研究院  下方二维码  ▁satisf 举措 lywood  ▁monitor  ▁methods ▁PC 较大 perty( ▁Lee0 注明8  ▁static@ 罚款H  ▁arrayP 市长X  ▁links` 征收h  ▁securep 我在x  ▁camera 抓好  ▁rule 重磅  ▁excited 之路 对你 色彩 file 高达  密切接触者 othing 都市  ▁plus  ▁university Posted 房价 分局  ▁graph 简称 Co 的主( 海外0  ▁direction8  ▁resident@  ▁minorH  ▁SandP  ▁surveyX  ▁Organ` Classh 便宜p  佩戴口罩x  ▁debut ]; 很少 体内  总经理  ▁multi 说法 Inst  ▁fourth  融媒体  ▁gift  ▁brief ython 后续 bur  ▁Chris  ▁changing 上去 备案  ▁advoc gov 灵活( 免责0  ▁managed8 Time@ acksH  ▁advantageP 零售X  ▁truth`  ▁instrh 济南p 融资x 差异 度的  ▁Ireland 厦门 昨天 andom  ▁factors ▁TO  的就是 ▁Som 账号 感动  博物馆  ▁nine  ▁wonderful  ▁scale  ▁Islam  经济社会 ▁Way  以上的 osis 那种(  ▁sharing0 东南8  ▁Hospital@  这也是H  ▁centreP 地球X  ▁placed`  ▁Communh  ▁Accordingp ratex  ▁Test  ▁mist 到来 发病  ▁fuel 别的 ▁Rev sembly  ▁supported 一句 后果 普遍  境外输入  inating  复工复产 下册 espite 号码 糖尿  发布的 index  ▁visitors( ";0  ▁musical8 并在@ 交叉H 职场P  ▁terrorX 专注`  ▁susph 出席p ▁Dix  主要是 eder Er 人家  企业的  onavirus 业绩 画面 inder 最美 原本 C <▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁appropriate agged  ▁finish  ▁During 本书  ▁eventually 释放 Sub  ▁leaving  了很多( 白色0  ▁infrastructure8 under@ 并非H  ▁relevantP 景观X  ▁showing` 桃花h 相处p  的肺炎x Of  ▁drop 还没 方位  ▁failed idden Des 转化  ▁visual  很大的  ▁guarant  青少年 uke code  ▁injury osen  ▁articles 前行  ▁academic ▁nav ▁Uk  ▁animals( 线索0 aven8 考证@ 养成H 补偿P alianX  ▁operation`  ▁knowsh  ▁commitmentp  推荐阅读x  ▁defense  朋友们 qq  ▁String  我们会 ▁Old ▁ir  市政府  ▁documents ui gers 当事 anche  ▁Color  给自己 烦恼  班主任 OC  ▁pieces acific 肌肤 玩家(  ▁Senate0 EP8  ▁manage@  ulturalH 十九P 慢性X …　`  ▁batterh  ▁proposedp 遇见x  提供了 ▁Its ora Top Che ading 惊喜 Part  ▁adapt ▁sea  ▁diagn ▁fol 为人  ▁Professor ▁Boy olit ▁ing 专科 金属 TC 害怕 善良(  ▁reli0 灵魂8 附属@ 责编H  ▁adventP  党中央X  ▁episode` howh  ▁elementp 途径x 维修  ▁Prot  工作者  ▁config  ▁Blue  ▁covered  ▁introduced  是非常 哈尔 家庄  liament  ▁landsc 用品 ears  ▁religious 这场  ▁influence pload  ▁ship 测量  规定的 著名( ga0 人社8 sub@ 发言H  据了解P ioneX ribute`  中国人民h  ▁Gamesp  ▁revealedx  有效的 诉讼  ▁Author  ▁predict ograp 积分 残疾  疫情期间  ▁housing 五年  ▁spending ▁Cy ws  ▁crime ▁д 购房 障碍  ▁protected 远离  ▁coast ▁spl urban(  ▁baby0  ▁bottom8 ón@ 对接H  的设计P 情形X 二级` ');h yep 无限x 清华 rehens ▁unc ▁wed 哪个  ▁welcome 同一 的中 有机 ль  计算机 一流 家属  ▁Real  ▁spring 向上  ▁instance ishes  ▁stress 讲述 重复 inding( ▁Whe0  ▁prefer8 从小@  电动车H  ▁AdministP 导师X 减肥` 接近h  ▁flowp 询问x  ▁advanced 无需 分泌 lines  方面的 ▁)  ▁issued  ▁Mike  ▁prepar cho  ▁justice  ▁slight usion  结束后 模拟 他在  ▁Young iency eler  ▁identify ▁Gal  ▁Penn( 新城0 我要8  ▁evening@ ▁HawH  ▁buildingsP ▁HenX  ▁pictures` таh  ▁excitingp  ▁heatx  ▁Middle 细菌 "" 的老  大数据 owl  ▁Beach 心灵 法国  ▁generally 纤维 粮食  ▁corporate 通讯 中有 ри aks 买房 起到  ▁Jones 春季 ▁Any(  ▁coverage0 充足8 west@ 信任H  ▁propertiesP 争取X 摩托` ▁Menh 记住p  ▁poolx ▁rat NE  ▁onto 过于  ▁happens FC doc First  ▁wouldn po 分配  ▁technologies 定制 iture  ▁weap TH FA  ▁separate  ▁View 从业 适用 Par( ▁nut0 食材8 男孩@ MayH  ▁importanceP  ▁communicationX long`  ▁thereforeh 定点p  ▁Onlinex 对其  ▁earth  ▁haven  现代化  发动机 公平 认知 模型  ▁freedom 百万 ▁Ep 这份 巩固  ▁classes 延长 ▁Ver  ▁starts 人间  iiiiiiii 天地  ▁emergency  ▁recorded  ▁explain ▁err 指示 PM$  ▁watching(  温馨提示,  ▁valid0 冠军4  ▁situ8 凌晨< (　)@  ▁jointD 难题H  acementL 出于P  ▁JeffT  糖尿病X  ▁alle\  ▁supporting`  ulationsd 一大h doorl  我们在p  ▁performedt  ▁Chairx  ▁column|  ▁Williams param inst 显著 巨大 档案  ▁incor ( !新型冠状病毒感染的肺炎 odies ▁hy  ▁selection 白天  ▁Federal  教师资格 itz arks  得到了  poration iner  ▁craft  那就是 关闭 一套 大连 bi 代理 ла News  ▁impress 子里  ▁hearing  ▁married 顾问 性质 短信 个性  ▁Aud  ▁command  ▁Microsoft ▁ign  ▁cold  地上$ 原料( verse, 国民0 run4  ▁lived8 而在<  ▁shut@  ▁flexD 我省H 生意L  有一种P 县城T 成效X ▁Tri\  ▁labor` 都很d 入境h ▁ecl kerp  简单的t  ▁occupx  不超过| 描述 树立 wen 随意 颜值  ▁adjust 理工 日的  ▁raise 长大 arts 适量  ▁Executive 将于 jj 科普 ▁Dev 市区 Just  ▁Wild 一只  的效果 验证 低头 ▁Lab  ▁univers ersey eech 创意 rant  ▁efficient 宿舍  ▁bath 名额 斗争 ▁_  爆料 iana iced 在外  ▁calling   ▁Live$  ▁plants( 不久,  ▁cells0 Every4 融入8  师范大学< 激发@  ▁determineD 妇女H eriaL 前期P  istanceT  ▁clearlyX 救治\  医护人员`  Previousd  ▁officersh  ▁whoml  ▁herop ▁Limt 签约x 星座| ▁Wal Photo free  ▁util  ▁Photo ▁Sup 前进  ▁Cast 总会  ▁depend  ▁stars 工商  ▁videos  ▁sets 民间 prise km  ▁sequ ▁dig 清洗 ansion  ▁critic 激励 Reg ▁Squ  function  相关部门 Ref 互相  常委会  ▁historical Value ju 主播  ▁Although  事实上   ▁forced anish 充电 政务  感受到   ▁restaurant$ 平常( 耐心,  ▁shares0 wan4 ancer8 医务<  ▁Like@  ▁removeD 笔记H  ▁EachL  ▁usefulP FileT  ▁mostlyX  ▁keeping\ 共计` ▁Jond 我校h  的第一l 那就p  ▁Standt  ▁tendx ▁Dou|  交通安全  ▁Japanese 姑娘  ▁missing  ▁listed  ▁professionals  ▁Children  ▁motor  ▁phil 精致 Pe 转让  ▁veter BI  ▁bringing 南方  ▁balance 暑假 应届 证券  ▁Trust onds 警惕 php  ▁politics 全新 话题 link ▁hab 人人  ▁fellow 小孩 停电  本平台  ▁secret  ▁Medicine   ▁construct ▁Fe  ▁item  ▁inspired  ▁discover  敏感$ 两次(  ▁behavior,  严重的0 部的4 ▁Pan8 总统< Text@ 酒精D ArtH  contentL 查处P ▁SarT 则是X  ▁Polit\  ▁serving` ▁eatd scripth  相关的l  未来的p  ▁icont ulatedx ▁Ro| 杂志 危机  ▁prepared  ▁criminal  ▁boost hent  ▁instruct  ▁cere  什么是  ▁worldwide 监委 赔偿  ▁updated  ▁Conference 癌症  党史学习教育 ▁Die  ▁units 爱国 后期  akistan 损伤 ixt 发行  ▁concert 古代  ▁listen  ▁teacher  喜欢的  你可以 ▁fif  ▁Card 家族  ▁Francisco 萝卜 ▁ice  ']  ▁responsibility  ▁Prov EST 班级   ▁teachers$ 麻烦(  ▁dynam, astern0 法定4 ▁Bob8  ▁regional<  ▁spons@  ploymentD  ▁CharlH 等你L  ▁winterP  ▁lotsT  ▁somewX yles\  ▁metal` namd ▁Refh  党组织l 小白p 最多t  relatedx BO|  ▁investors ulate gment  ▁по 病情 新建 暴雨  ▁arrang icious 将其  ▁shape iano 批评 ▁dry 小说  ▁stated Key 不懂 限时  ▁otherwise aylor 的外 Ass  ▁reviews 不忘 Please  ▁institutions  ▁ment 强制 疯狂 ▁vac  ▁highlight 电力 面上  江苏省  股份有限公司  各大 一口  ▁authorities 全日 ERS   ▁guys$ 民主( 招商, AI0 吃的4 ffect8  ▁studio< 高峰@  免责声明D  ▁smallerH  ▁BostonL  ▁StudiesP  ▁relationshipsT  ▁considX 这类\ 录用` aisd  ▁approximatelyh ligl fep  ▁Andrewt powerx onom|  ▁kitchen 纠纷 前后 ▁pod  ▁allowing  ▁Exper 严禁 发育 利率  ▁format  ▁agreed ▁fra 苏州 真题  ▁Niger  ▁panel 接着 uthors  ▁feels 加班 销量 });  ▁waiting 抑制  ▁Call  ▁Scot ··  ▁server EE rite 厨房 汇报 也可 影视  ▁dance  ▁island  Test rier  ▁input ingly 走了  LA$  ▁connected( State, 不易0  ▁package4  老师的8 不但< ▁fab@  ▁buttonD ansasH  有的人L utelyP  asketballT  ▁decadeX  ▁alternative\ 干货`  ▁consumersd 体会h  ▁Armyl ellerp 高温t 旅客x ▁Hor| 毫无 股票 做事  ▁burn  联系人  ▁furn ivered  ▁solar ▁Use ▁lit ▁Adv  ▁holiday 主办  ▁revenue 幸运 国外  的价格 asks  新能源  检察院 CS  ▁innovative  河北省 配备 zil 对话 竞赛  ▁approved stra  ▁charges Set icial dis  ▁speech TR 好处   ▁Writ  ▁plenty  真的是 留学  中国特色   发生了$ 十一( See,  ▁magn0 real4 EX8  也不是< ▁tro@ реD 帮扶H  ▁interestsL  ▁identityP ▁JunT  正确的X 秘密\ amm`  irationd ellyh 节奏l 实事p 返回t  ▁soundsx 年初| elines  ▁locations  ▁theme ▁© stein Us 进度  ▁на 感冒 开了 遗憾  ▁schedule  一定会  ▁jump  ▁mentioned  招聘岗位  阅读原文  Exception 冬季  ▁vent 舞蹈 已有 which ▁ton  ▁notes  ▁compar 所需 其次  ▁pair  ▁rapid  ▁hundreds  交通运输 ▁ID 回归  ▁coord 美女 装饰  ▁renew  不错的 生存 中药  ▁flight$  健康码( 捐赠, etime0  ▁proceed4  ▁illust8 订单< pret@ postD 本期H 点的L  ▁CharlesP 户口T 诗词X 答题\ 大脑` cksd  全日制h atil  ▁commissionp  ▁treet 牢记x  ▁confirmed| 那里 庆祝 证件  ▁thus  ▁Environment  犯罪嫌疑人  ▁detect irms  ▁dial 受伤 必备 清单  ▁breat  ▁Prem 门店  ▁doctor  ▁surg  ▁bird 拿到  ▁output 风水 翻译  之所以 ▁flu 太原  ▁museum rical  ▁leads bb 大大  forcement 健全 okes ▁guy ownt  ▁promote ▁Ken /**  ▁holds eries ▁CD oles$ ”;(  等方式, 当下0  American4 动员8 户外< gs@ streamD 夏季H  ▁discussionL 窗口P 督导T reesX ▁kid\ 时不` 期限d phash  是我们l 菩萨p  ▁Michigant  ▁departx 看完|  essions igen CP  ▁script 　| 西部  ▁advis 是我  ▁Phot  ▁affected  ▁apart 后面  ▁Columb 江南  ▁handle  ▁Person 住院 题目 日晚  ▁Digital  ▁fish  董事长 ▁Gra ки ▁Id  ▁clinical  ▁reference  ▁innovation 腾讯 Book  ▁kick 第七 三十 ▁ven  ▁featured size  一直在 ador  ▁Secretary 舞台 发的  ▁negot$  ▁industrial( 阴性, HR0  ensions4 这条8 真相< ▁Ma@  ▁distanceD ghanH  被告人L 不及P 优雅T  ▁seekingX 合集\  ▁Italy`  ▁processesd  ▁exchangeh 专属l 实地p datedt reditx  ▁wear| 用了  ▁optim 游泳  中国人 rum  有哪些 亲子  ▁campus 扩散  的女人 区的 ▁AND 神秘 英文 have  ▁massive  常态化 ▁AL 你就 传媒  ▁committee utor  ▁Season  ▁victory ▁Ari  ▁explained  ▁Night  ▁greatest 值班  ▁payment ura 有钱 放假  ▁opinion 批次 致力 名为 考场  ▁edition 家园  县公安局  亿美元$ raine( oted, 减轻0 精力4 ▁bul8 dy<  ▁Donald@  ▁MarketD  ▁ProductH 门槛L 不太P  ▁loseT  ▁witnessX enced\ 和你`  成年人d  大量的h pagel  ▁accidentp  ▁telet  ▁indicx  ▁village| 文艺  tenance Object ja  ▁Jackson ounsel ▁san  ▁protest 这时  出现在 ping ▁Ell  ▁putting DS  ▁Space 机遇 ▁Dom ▁flo  两个人 '; 本土 用心 储备 服装 百分 四大 ▁Van coh ▁Bas outs  ▁library El osoph hr 男生  ▁scientific 佩戴  ▁distribution  ▁Coll 少数  ▁Theatre  的精神$ 有力( zx, 夜间0 几十4  ▁cloud8 视为< 观念@ 带你D ▁pilH fsL athanP InterT 协同X 名师\  长按二维码`  养老金d  ▁Watchh  ▁fieldsl  ▁diversep  ▁testst ////x  自行车| 货币  ▁collaboration 要有 疑问 行程 水的  建筑面积 Out 资质 作风 年前 助于 轨道 按摩  ▁identified  ▁Lord  的关键 三是 整合 ▁bra  ▁Master  ▁Support  ▁Championship  ▁domestic  ▁tells 热门  ▁Force  优秀的  ▁assistance 中秋  ▁agencies Em ▁Max Since 均匀  党员干部   ▁rain  ▁engineering Gu 年纪  ▁Because  讲座$  rehensive(  卫健委,  ▁gather0  ▁circum4  法律法规8  ▁beauty< 滑动@ uableD atusH 更为L  ▁removP 收购T 身心X  融媒体中心\  ▁discovered` 监制d kesh 那样l ilitp  ▁Thankst  ▁fillx 早已|  ▁Instagram 冬天 Jan title 奶奶  ▁index  ▁cookies Mr  ▁coffee 解答  这个时候 uzz ride  ▁earned  ▁winner  ▁reform water 仅仅 但在  市场监管 武器 参观 壁纸  ▁combined 开车 jo  ▁familiar  ▁photograph  ▁Connect  ▁corner  ▁square 乘客  ▁Zeal  的相关 办学 血糖   ▁seemed 一日  ▁ideal she  ▁returns  Gl$  高血压(  的关系, icit0 一看4  ▁bound8 otic<  ▁delivered@ 精华D 专用H  图片来源L passP  ▁grewT 目录X  的一些\ alle` 风雨d  ▁categoryh mercel 总局p 全民t 定了x 弘扬|  ▁Coast 看出  ▁carbon ▁Tem 试卷 委托  ▁alleged  ▁adults 参赛  ▁Walk 初步 浪漫  ▁attacks  ▁willing  ▁shooting  基本上 授课 假日  人生的 较为  itutional ▁Hon  是为了  ▁Once  党委书记 ▁rom 和人 ▁fle 加拿  ▁comfortable  iversary  ▁fell  新媒体  ▁default  的考生  ▁Jewish  客运 着力  ▁Spring  ▁forest Image  某些$ 当日( 货车,  ▁dress0  疾控中心4 地说8 猪肉<  时间为@  ▁resolD TubeH  充分发挥L  微信号P 民办T  ▁wasteX 通信\ 打赢`  ▁registeredd 西方h  ▁Lightl  ▁Girlp 南山t  ▁whateverx ));| 通用  ▁resource  ▁programme  ▁stores 字体 不适 ulner  ▁conven  ▁consumer 家具 给他  ▁Guide  ▁gender  ▁Sciences 牛肉  ▁combination 礼物 CR ”:  ▁defend andon 选手 工会  的基础上 只会 ▁unf  湖北省 Follow 住宿 国有  ▁upcoming 飘飘 你在 一把  ▁internal NS   的专业  正能量 老婆 iy 复试   ▁ongoing$ 抢购(  fortunately,  实现了0 OK4  lication8  ▁educational< MM@  ▁SupremeD 过年H 的手L lightsP 攻击T 故意X profit\ 穿着` Yourd 过度h ▁Licl  ▁leaguep lockt ARTx osc|  石家庄  ▁Zealand  ▁accompl aser ▁Log  ▁asking ▁ur  注意事项 otal  ▁selling gypt 受理  ▁begins 威胁 raq  ▁crowd pet ello 早餐  ▁Disney 高三 rig law 更高 legal  ▁Republican 秘书  ▁Look istant  增值税  ▁Division 践行  ▁Southern ician  ▁Roman ented   ▁animal  当事人  ▁native  ▁democr 培育   ▁doubt$ 歌曲( 受害, 咱们0 火锅4  不仅仅8 即便< 处分@  ▁railD PerH aminL  ▁labelP 支队T mondX 重的\ osit`  ▁Danield 在大h posesl 区委p 话说t 定的x  传染病| ▁Pay ▁Ok  ▁upgr rome 核实  ▁readers adium king  你们的 高兴 仿佛  ▁Check 哪怕  ▁Brook Log  ▁cards onto Source 演出 Let  ▁honor ▁Mid  ▁Prime  ▁decide  等问题 生日 提到 sen  ▁candidates ▁da 等地 Sec  ▁Georgia  ▁awards 透露  美好的  uous  哈尔滨 院士  ▁favour  图书馆  ▁Set$  ▁guitar( 顺序, 就医0  ▁privacy4  新冠疫苗8 ▁Equ< 大约@ ▁armD /㎡H ssssL  ▁wealthP ushedT itiX ether\  的安全` 标注d 十八h IEl 尚未p 原件t  反反复x 任职|  ▁arts Are  ▁tech  合法权益 化工 ▁CA 成的 ingu  ▁candidate  中华民族 Trans CD 一律 evin wx 多人 音频  ▁Britain  ▁quant  ▁dollars  ▁conducted 信念  ▁brands ▁Far  摩托车 火灾  ▁marriage 一对  ▁Story  ossible  ▁historic Act cles  ournament mosp  ▁founded  各自 ocket  的日子 ▁Jr  ▁Foot   ▁applied$ 又有(  ▁Catholic, umer0  有关部门4  ▁guests8 征集< sole@  工作经验D  ▁professorH 北方L  相应的P  ▁updatesT  一方面X  ▁negative\  ▁Korea` 遭遇d 梳理h orall 获奖p Endt  常见的x 占比|  更好地 芯片 平凡  ▁narr  活动的  ▁sand  ▁Make lix tered  ▁Italian  ▁sustainable  ▁Second 租赁  ▁volume  不符合 ▁McC 暂时  上面的 失业  ▁millions  ▁teen parent ▁Gar 双手 作家  的过程中  准考证  咨询电话 督促  ▁draft 是由 ▁aid  ▁Jesus  ▁planned isters  ▁enjoyed  apore 塑料 aa 可在  浙江省  常务$  请及时(  上海市,  ▁Govern0  ▁holding4 在了8  ▁Page<  基础设施@ ▁eraD 全区H ampionL akedP  按规定T dbX  ▁filed\ 二次` 却是d  ▁debth  ▁chargedl  ▁servesp 公斤t 损害x 1 *▁▁▁▁▁▁▁▁▁▁▁▁▁▁| 不论  ▁talks ▁Mic 又是 ▁dim abeth 形态 文旅 每人 ▁IP Will 默默  的信息 orders  ▁cooper 请在  下一步 NN  ▁leaves ▁tow  ▁Pakistan 省份  重要的是 圆满  大家的  ▁snow  反反复复 楼市  ▁seat ih  ▁Through  ▁honest  变成了  ▁legislation ▁pun 从此   有利于  等工作 更大 外卖  ▁carried  大专$  ▁consequ(  ▁accounts, ▁м0 ▁Kar4 ▁RE8  ▁Android<  ▁shift@ 便民D  也就是说H  ▁PacificL 十五P irtyT oweredX  ▁actor\  ▁neighborhood` usterd  ▁athleth  ▁stickl  的历史p  ▁conversationt ▁Ltdx mg| 师傅  ▁bright case 真心  用人单位  ▁minimum 睡前 fast 一辆 美容  ▁entertainment  ▁route  respond  ▁database  我们可以 Eng aduate 得了 凭借 据悉  ▁Engineering 常规  ▁awarded  ▁forget 既然 若有 入选 ali ▁Pop  ▁originally 差距 不下 手工 个体 混合 Def  帮你 厉害  ▁Spanish ▁Cle  ▁aspects  制品$  ▁accommod( ua, 多多0  个小时4  ▁exercise8 是你<  中医药@  ▁sweetD umsH ▁KimL  的风险P  ▁LiberT renceX 成人\ GO`  ▁traditiond 清明h 投诉l  ▁conflictp  ▁nucleart ▁twex  ▁concent| ▁Lo 而已  ▁facing  ▁administr  ▁ride eper  ▁passeng 多元  看到了 我也 avor va 奥运  ▁Steve ▁rid  ▁drugs void  的规定 教程 reens  巨大的 公民  ▁android 冲击 只需  ▁Fair  ▁liber illy  ▁specifically  ▁minute DF  ▁guess  ▁Kingdom 由此  ▁tough  ▁Both   ▁manufacturing 复印  ▁Ohio ▁Ext 凡事  光明$ 称号(  ▁follows, ighter0  ▁emphas4 门票8 气血< 陌生@  什么时候D 神奇H  ▁postsL adedP 上有T  ▁newspX ▁tom\  微信群` 说过d  强大的h лоl  ▁creationp 块钱t 石油x  ▁entreprene|  ▁infect 亲自 ▁fat 虚假 kin  ▁confidence  ▁intended 航班 HT  ▁recip  ▁confident 高于  一段时间 tes noon 还不  ▁scholar Ge bridge  ▁filled idth  ▁employee  ▁transition  ▁enable nic cknow inton  ▁abuse  ▁virtual  ▁thoughts 月初  分数线 ▁pal  ▁concerned  ▁perspective obs  下列 截图  ▁risks  ▁detailed 试用   ▁belong$ inds( 脾气, 张家0  ▁recognized4 大桥8 apping<  ▁chemical@ 坚强D GenerH 假如L 表彰P  ▁dispT  ▁figuresX 鉴定\  有些人`  ▁theoryd 行情h ▁hydl lynp ####t 昆明x 多云|  ▁healthcare  等相关 谨慎  ▁fort oster 标题 的花 ▁lux  ▁expertise 牺牲 isk ATION  　·　 公寓  ▁encourage Rem 一颗  ▁chain 抓获  ▁elim  是中国 招标 要在  ▁owned 不变 ▁Hy ▁Win 追究 依靠 iest front 国庆 ifts  ▁motiv  ▁edge 万吨   ▁county 金钱 Sm  农产品  ▁matters  少女$ 西藏(  ▁Know, 宁波0  严格落实4 ennis8  ▁Video< 承认@  ▁removedD  的企业H  ▁participantsL acleP  ▁treesT thenX  ▁incident\  本公众号`  产业链d seyh 单词l  不管是p  ▁assetst  ▁Passx 在校| 等到 我不  上半年 强度  ▁scheduled  ▁Jersey ле  esterday rooms  ▁dangerous 存款 复制 中午 店铺 ▁($  再加上  ▁rooms udden car ads  ▁arrived 五一 拨打  ▁mort  ▁wine amil Comm  国家级 не  ▁perman rete 陷入  ▁Manager ▁Cov  ▁Democratic 坐在  转变  蛋白质  ▁wedding  ▁roles 十足   学校的$  ▁warn( adel,  的过程0 收取4  ▁employment8  非常的< 都没@ 职称D 境内H  ▁purposesL 怀疑P  志愿服务T  ▁apparentX  ▁signs\ 土壤`  ▁Iraqd  疫情的h Otherl 严肃p  ▁secondst 秋季x  ▁prote| ▁TH 作战  ▁tickets 决心  ▁Sign  ▁switch  ▁shopping 页面  ▁council  ▁appearance 入住  ▁Queen 物的  iveness 厚的  ▁expensive 一款 动画 纪录 全年 全会  ▁height 不宜  ▁Access  ▁worksh inson 你想  ▁decor  ▁Northern  的原则 ▁tun unting 服用 Stud 随便  加拿大  те 例外 elled  ▁fighting  不合格   十四五$  ▁loan( sl, pace0 情节4 进了8 原理<  ▁assum@ HPD yncH  交通事故L  ▁organisP 更换T  ▁contemporaryX  ▁herself\ 花开` ▁Lucd ▁rowh  ▁Crossl  报名时间p  ▁gardent  ▁householdx 均可|  ▁maximum  ▁vulner 不去 ifer  ▁improved 慰问 ▁Po 我说 ▁mut agues 牛奶  ▁profile 拘留 中山 Fl  这件事 isa  ▁divid suit  ▁walking Tags  ▁drivers  ▁victims acts  ▁caught anger  ▁temperature  ▁installed 房源  有关规定 ATE  投稿邮箱  驾驶员  ▁Biden  另一方面  来自于  毛泽  ▁Little  ▁license ▁AR  ▁virus   ▁typically$  中国特色社会主义(  了解到,  ▁courses0  ▁moments4 ▁з8  ▁failure< 本质@  ▁determinedD  ▁magazineH  ▁sisterL 肝脏P ikiT  ▁stormX  ▁severe\ ican`  ▁involved  ▁sympth  所谓的l 未经p 先锋t  ▁scientistsx  ▁recovery|  的基本  ▁advertising  ▁circumst  相当于  ▁discl  ▁Joseph  ▁channel  ▁extensive  ▁mand 不然 ▁NE  ▁techniques ▁adm  Service Dav inois esome  ▁statements 类的 OW 为此 赠送  提出了 Bar ▁Kh  健康的  ▁kill  ▁Creat  我觉得  ▁provider 客观 PE 得知 anges };  都知道   ▁comprehensive apes  ▁graduate shire  高新区  飞行$ 头的( yers, fin0 ▁nor4  致力于8 欢乐< 放下@ 判决D  ▁evaluH 体的L 实时P otesT rencyX  ▁ancient\ pir` 单身d 齐全h  ificationsl oustonp  ▁boughtt  和其他x 好运| Click  ▁positions 身高 Pol 宣讲  ▁Harris ▁ran ▁sav  ▁murder  ▁sides 辣椒  ▁Rest ME  ▁cart MT ▁cra  ▁symbol  ▁Staff erman  ▁featuring 售价  ▁reporting  不容易  ▁male  全世界  越来越多  驾驶证 大街 现状  ▁Hollywood  ▁incredible  意大利  请联系我们 医用 cohol  数据显示  也将 教你 大全 GA  ▁Broad  对手$  ▁Civil(  ▁assert, 单独0 style4 教练8 提取<  国家的@  学习的D 南部H 助理L 为您P 上传T 不对X 高分\  ▁debate` 务必d sumh 厘米l 骄傲p  ▁topict  ▁Brazilx 手指| 不止  应该是  ▁significantly 贵人 ▁Ent foot 凝聚 上学 豆腐 继承 体重  ▁urban ':  ▁functions  ▁surrounding  ▁scored 导演  ▁alongside 凭证 cious  ▁flat  地区的 过敏 东莞  毛泽东  ▁defined  izabeth  ▁Next 给人 生育  对自己  ▁acknow  也不能 ▁sam ordan  ▁strategic  ▁vaccine  的一切 频道 着你 此文  resents$  ▁relax( 肠道, 楼盘0 走访4 VD8  分公司< aped@ 座谈D 上门H openL  ▁equalP layT 估计X Many\ 电信`  ▁Taylord Mosth  ▁steell  篇文章p  ▁Irisht  ▁attorneyx  给我们| 不必  ▁emotional 去世  ▁slightly  ▁Matt  ▁tack  ▁believes g `▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ appy  ▁Cost found  ▁unless 沈阳 asters DT 牙齿  ▁directed ras  另一个  ▁Point  今天的  ▁execut 密码 渐渐 式的 依托 活着 地面  ▁Start  ▁param  ▁absolutely  ▁Privacy 畅通! estyle! Mark! rich !  ▁believed! 他是!  ▁spoke!  ▁intelligence! pty ! 居然$! 水分(! La,!  主管部门0!  ▁supports4! ▁wat8! 寒假<"  ▁topics"  ▁impressive" 平稳"  ▁festival" 头条" 化解" ▁Tal" table" 学费"  ▁remaining" ▁Son" AND" 荣获" Gr" 违纪"  holders" 毫不"  ▁enhance"  ▁underst" 孤独"  ▁Custom"  ▁successfully" trans# 见证#  ▁arrested#  ▁Daily # ▁Ha#  相结合# 微笑# ership# count #  ▁territ$# 失眠(# 羡慕,# 邓州0#  ▁atmosp4#  ▁judge8# 这就<#  一句话@#  ▁authorityD# ▁SeaH# umaL#  联系我们P#  ▁singerT#  ▁housesX#  第二天\# 脾胃`#  ▁flagd#  项目建设h# actoryl# 诚信p#  ▁ministert#  ▁Administrationx# 自在|# 可靠#  ▁receiving# 见到# 体质# During#  ▁audio#  ▁apps#  ▁resolution# 哥哥# bury#  ▁spaces#  ▁Stan#  市场的#  ▁Radio# 大气#  ▁participate# ▁MP#  ▁shock# 聚会#  ▁buying# 民众#  ▁repair#  让他们# 延伸#  ▁goods#  ▁Ford# Rel#  千万不要# arp# ▁Hom# 与其# ▁Sus#  ▁mode$  免疫力$  ▁formed$  ▁calcul $  ▁awareness$  必要的$ 现将$ 乐观$ greg $  ▁shel$$  ▁info($ orough,$  ▁flood0$  ▁standing4$  ▁twice8$ uling<$  ▁providers@$ ▁astD$ TERH$  ▁OfficerL$ iroP$ 下跌T$  ▁competitiveX$ 喝酒\$  ▁fold`$ ▁kmd$  ▁closerh$ 产量l$ 带头p$ воt$ 承受x$ MC|$ 实战$  ▁reject$ 被人$ 过多$ hetic$  各部门$  ▁Action$  ▁maintenance$  全方位$  产品的$  ▁destroy$ unc$  ▁broadcast$  ographic$ 南宁$ arters$ hus$ 学者$  ▁Circ$  ▁covers$ burgh$  ▁none$  被称为$ 帮忙$ 挖掘$ 立场$ ▁af$ item$  ▁planet$ ▁Op$ ucks$  新型冠状病毒肺炎$ rovers%  ▁Have% 高层%  ▁grown % 回报% 爷爷%  ▁explains% 月至% ECT %  退役军人$% 中级(%  福建省,%  ▁prepare0%  ▁Kong4%  ▁Cook8% 等于<% 激素@% 山水D%  詹飘飘H% 评选L%  ruptionP% ▁pedT% 导向X%  ▁YouTube\% aine`%  复印件d%  山西省h% ualityl% ▁Jacp% 小镇t% 关爱x% 姐妹|% 算是% 我县% 美味%  ▁Hard% 将来%  项目的%  ▁Country% 拆迁%  ▁beach%  ▁Colorado% ▁Ap% 一道% 致敬% 之处% Where% 分手%  提供的% child%  ▁ends%  詹飘飘詹飘飘%  ▁bigger% stone%  农业农村%  ▁aircraft% 文末% 年限%  的状态% ▁cro% bro% Any% ylvan% 车道% 反对& 日内& 同行&  疑似病例 & gener& 青海&  ▁wire& stract& ▁NFL & 若干$&  ▁landscape(&  ▁medium,& 领先0& 气候4& Jo8& 快来<&  ▁Full@& 著作D&  ▁iPhoneH& rawnL& 联动P& 清水T& IBX&  ▁stret\& 南昌`&  ▁factord&  ▁Villh&  ▁reducedl&  ▁Concp& 评审t& 阳县x&  怎么样|& 国产& 学子&  ▁Financial&  ▁adult& 亚洲& 车间& 排除& 有意&  高风险地区& bec&  ▁Term& 视野& ESS& 尺寸& IO& lets& 护肤&  ▁coronavirus&  ▁agric& aron&  ▁celebrate&  ▁friendly& 上级&  ▁Blog&  ▁Ministry&  ▁pregn&  ▁parking& 不大&  ▁manner&  如果是& 两天& roller&  ▁Again' 答疑'  ▁attended'  ▁afternoon '  ▁crew' 而来' 起点' ▁AT'  ▁surprise ' 股东$' ▁Ol('  ibilities,'  ▁increasingly0'  ▁frame4'  的形式8'  ▁platforms<'  ▁Follow@'  ▁plasticD'  ▁restaurantsH'  带来了L' 以外P'  ▁happeningT' infoX' Ne\' zona`'  不可能d' 抱怨h' rml' 持有p'  ▁rought' ▁mirx'  ▁considering|'  科技有限公司' Error'  ▁worry' 一块'  ▁speaking' ▁ce' 灾害' 阿里' 十大'  ▁convert'  ▁Spain' 4 -▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁'  ▁entirely' 饮料' Who' 新年' 以往' 浏览'  ▁Only' iary'  疫情影响'  ▁elig'  ▁Kenn'  ▁treated' vy'  ▁soul' othe' 战士' 警告' vin' 透明' 文物' 宇宙( fil(  ▁tips( 轻易 (  点击下方( 黄色(  ▁represents(  ▁agent( 就把 (  ▁Partners$( iors(( Ret,( makers0(  经济发展4(  ▁mail8( 搭建<( inent@( 功夫D( 绿化H( ▁flyL( 可怕P( 不行T( ▁naX(  ogether\(  个人防护`( lianced( iamih(  ▁Booksl(  ▁retirep(  ▁Songt(  今年的x(  好消息|(  ▁Disc( 接待(  owntown(  ▁Take( ▁Rod(  独特的(  有助于(  的同学(  违法犯罪(  ▁Singapore(  Another( 买卖( 镜头( 艰难(  ▁faster( mony( iginal( 生气(  ▁Control(  ▁extended(  ▁streets(  ▁challenging( ▁Mu( ersion(  ▁entered( 开设( 班子( ▁mal(  ▁relatively( 约定(  的产品( 这也( 安心) 之际) 边的)  ▁Phys ) AV)  ▁Mill) DR) 消耗)  ▁Four )  ▁deals$) wa() 调理,) 下一0) 实的4) cons8) 演讲<)  ▁west@) astsD)  ▁submitH)  ▁fascL) 多名P)  ▁ultimatelyT)  fficientX) 面议\) AF`) poold)  工作室h)  ▁vastl) letterp) 列车t)  ▁visitingx)  ▁Ryan|)  ▁Henry)  ▁decre)  ▁affordable)  ▁claimed) 公积) awa) world) 多数)  ▁Field)  ▁accepted) uts) 善于) 漫画)  的可能) 采集)  ▁Asian)  ▁hopes) 预期)  ▁suit)  就开始) 小米) 化妆)  ▁implementation) 四季)  ▁reput)  ▁components) Two) ▁sac) 主人)  ▁regulations)  ▁Labor)  广州市)  ▁compos* 体制*  ▁Hand*  ▁battery * 我家*  ▁Capital* 北部*  ▁initiative*  ▁register * ▁mit$* ko(*  ▁Perform,* ▁у0* 书写4* 商场8* 另行<* onym@* 实体D* flixH*  资格审查L*  ▁forgP* ilaT* alysisX* war\* 模板`*  ▁ourselvesd*  人大常委会h*  ▁networksl* ▁Darp* qlt* 大地x*  include|*  ▁Agency* 多家*  ▁chosen*  公共卫生* 协作* 说到* 干燥* 大陆* 其余* 县级*  ▁remote* away* input* PL* 两位* 保养* 省市* 本周* ▁AI*  ▁recording*  ▁menu*  ▁elected* 救助* 绘画*  ▁banks* 摄入*  ▁effectively*  达到了* 严峻* 支援*  ▁authors* endar* 安市+ 宁夏+  ▁taught+ PPT +  ▁elections+  ▁Turn+ 要把+  ▁smooth+ ти +  ▁Guard$+ ());(+ 前面,+ 教室0+ 指引4+  ▁stake8+  ▁trading<+ worth@+  tainmentD+  ▁closelyH+ 航天L+  食品安全P+ afeT+ 主编X+  ▁prosec\+  ▁evol`+ ▁Twd+  高度重视h+ 定义l+ Attp+ 李某t+ 中年x+  投资者|+  ▁examples+ ycl+ 与人+ 留在+  ▁adds+  ▁acting+ spe+  就已经+  ▁surgery+ ▁Rom+  ▁fundament+ 气息+  下来的+  一个个+  ylvania+ 出血+ 眼前+  ▁confirm+  这是一+ 三种+ ▁Ron+ 天空+ 分的+ 事宜+ ión+ 可谓+  ▁Kent+  最重要的+  ategory+ 复兴+ 诊疗+ erg+  有一天, fa, 说是, 装置 , 以免,  ▁Ukraine, ▁OR,  ▁founder, 均价 , ▁Fac$,  ▁guest(,  各项工作,, ira0,  ▁phase4, uting8,  ▁objects<, cd@, 度假D, ▁SkyH, 立足L,  ▁delayP, 外科T, folioX,  ▁vital\,  ▁busy`, Modeld, ▁bagh,  ▁Torontol,  ▁worstp,  重要讲话t, 生理x,  的发生|, Path, 真诚,  ▁expansion,  自然资源,  ▁iron,  公积金, ▁Cru,  ▁meetings, 密集, 一方,  在我们,  ▁sitting,  的力量,  ▁fantastic, 做人, 它是,  等症状,  novation,  ▁circumstances,  ▁Expl, 邮件,  的样子,  ▁valuable,  明显的,  ▁Illinois,  ▁Images, 依赖, 返乡,  ▁electronic,  ▁possibly,  要加强, Count,  ▁Histor- core-  ▁finds-  高校毕业生 - 代码-  ▁interpret-  多年的- 要想-  ▁worse -  告诉你$-  ▁pref(-  ▁practical,-  ▁legend0- author4-  ▁websites8- ▁Eth<-  湖南省@-  ▁TechD-  ▁kingH-  心理学L- 平米P-  的环境T- 第八X- icide\-  ▁guard`- 就被d-  研究所h-  ▁Bangl-  ▁Wherep- 验收t-  ▁missedx- 熟练|- 不妨-  ▁Area- ▁NC-  系统的-  完成了-  ▁widely- requ-  ▁desire- 期内- 不怕-  ▁accessible-  ▁Nick- Off-  ▁Corporation- 马克- 海鲜- 拆除- eters-  ▁diversity-  一样的-  ▁teach-  或者是- 受益-  经济社会发展- és-  ▁marg-  ▁Travel- 名校- nesota- 请求-  ▁anywhere-  igrants-  ▁stone.  的教育.  ▁tight. 电气 . 工委.  的世界. bal.  ▁nurs.  ▁paying .  ▁Vice$.  ▁reserved(. 人和,. 太平0. ▁;4. Ke8.  ▁faculty<. config@.  ▁SoftD. shipsH. 处处L.  ▁SantaP. 看似T.  AlthoughX.  ▁producer\. 东部`.  ▁distinctd.  ▁brokenh. 轻轻l.  ▁Magazinep. 承包t.  ▁contestx.  ▁settings|. 预算. Ste. 细致. 衔接.  义务教育. image.  ▁yards.  做什么. Vis.  ▁quiet. ▁·. 事儿. 治安. 五个. person.  ▁designs. 债务. 打破. 报价. 勇敢.  ▁deaths. boy.  ▁recommended.  防控措施.  ▁delight. 除此. base. 一杯.  ▁cheap. arian. 频繁. acking. UM/  ▁substant/  ▁occasion/  ▁enforcement / ▁Ur/ ▁von/ 点亮/  ▁telling/ ▁sad / 情怀$/  ▁therapy(/  自网络,/ 这段0/  所有人4/  ▁replace8/ orne ▁mob>  achelor>  解决方案 >  ▁dogs> etts> 激活> ▁Non>  所得税 >  ▁answers$>  ▁fixed(> 台风,> 与此0> ockey4> 拍卖8>  垃圾分类<>  ▁Line@> ▁synD>  ▁industriesH> WindL> oddP> navT> 步行X> 组建\> 县长`>  ▁intentd> ileyh>  ▁Pictl>  ▁yesterdayp>  ▁findingst>  ▁Summerx> una|> 会长>  ▁creates> ▁FOR> rog>  ▁capable>  ▁counsel> Hel>  ▁giant> 凤凰>  ▁embr>  ▁boat> 局面>  感兴趣>  俱乐部>  ▁Wilson> 方的>  的前提>  ▁Culture>  武汉市>  ▁inspiration>  ocument>  ▁totally> 中毒>  expected>  除此之外>  这个问题>  都不会>  ▁interior>  ▁illness> 细化>  ▁moves>  ▁external> 改进?  责任编辑? 涵盖?  ▁seasons ?  残疾人?  性价比? 未能?  ▁Entertainment? 课题 ?  ▁sample$?  ▁alcohol(?  ▁Forest,? 控股0? 退出4?  ▁facts8? ova}U unterU  供应商U  ▁strict U  ▁investments$U 筛查(U  先点击,U  ▁copyright0U  才知道4U akh8U 头痛t 能让t 生姜t wpt 意愿t goodt 从中t asantt Lifet поt 也都t oydt  ▁clothingt  文化的t  三个月t 学霸t 发射t 洞口t  副市长t  每日一u 雄安u offsu 产后 u  ▁noiseu 华南u junu 划分u 宝妈 u 主张$u  体现了(u encing,u ▁Nob0u 小小4u  请您先点击上面的8u 分散  ▁dollar@ 昙花B 的多D 走过F  ▁sectionsH 团购J  ▁requestedL  企业和N 运转P 离子R  ▁calendarT 车上V  等重点X ▁JosZ  届全国\ otype^ 备受` 招收b  月中旬d  了解更多f 散发h entlej SONl olkn irtsp  ▁speaksr 稿件t 室外v  ▁albumsx 不等z 门市|  ▁insert~ 业内  ▁peak 菩提  的其他 我还 olo 榜单 igs prot 仓库 见的  ▁Professional witzer  的管理  ▁Marvel iatric agic 交管 巡逻  与其他 慈善  message sche 好转  ▁publications James  ▁hosts ▁aer 题型 崛起  作为一个  ▁theater  ▁Organization  从业人员€ 大海Ā  工作地点ƀ  ▁extraordinaryȀ  ▁passwordʀ 湿气̀ Use΀  ▁VeterЀ  工作会议Ҁ 秋天Ԁ  ▁indicateր  ▁till؀ 万事ڀ 不应܀ 白羊ހ 书中 upload 代价 Cap  新中国  ▁loyal 架构 签证 多万  ▁Solutions 涨价  不断地  ▁workplace  ervations 堵塞  严格执行 adi  ▁Missouri 友们  注意力  有效期 nor  ▁oldest ln  ▁signature 阻止  公安部 Run  ▁objective  ▁harder  ▁operated 呕吐  ▁Estate  ▁Trib" 小姐$ ibl&  想要的(  ▁True*  ▁opposed, 丽的.  ▁Hind0 "/2  ▁absor4 yt6 淀粉8  ▁samples: amous< zo> 7 0习近平新时代中国特色社会主义思想@  ▁MatthewB ▁unaD  ▁associateF  按要求H vsJ ▁datL 担忧N  薪资待遇P 寒冷R 加热T  管理部门V 环比X 走势Z  只要你\ 对着^ 换届` 初期b 层层d 分工f 开除h ▁Halj  早上好l 媳妇n 不已p  ▁Centuryr  ▁dustt 摆放v  卫生健康委x  ▁Scottishz  ▁Howard|  ▁algorith~  ▁residence Hz 退款 喜悦 一小 更要  ▁refused  ▁comprom 评分  还有很多  ▁conscious  ▁Private  ▁entrance ▁Tai 预习 悲剧 clear  中小学生 Tim gate 总量 Local  ▁reven olitan 直至  Business 自杀 省内  ▁jazz Tech bound rior  全力以 对他  ▁squadā ablingƁ + $这样您就可以继续免费收到ȁ  ▁Utahʁ  ▁Electrić  ▁tied΁  ▁MoneyЁ  就是要ҁ 防晒ԁ 太过ց 跟我؁  男人的ځ  ▁shell܁ 群里ށ 时长  可以看到 溃疡  General 调动  的时代 朝鲜  市委书记  ▁Select  ▁poetry  ▁completion TOP  在农村 事发  的心态 %　  ationally 擅自 病变 电网  有一次 mate  Controller  ▁tone 人不  ▁piano  十九届  ▁intense  最后的  部分地区 电源 林业  ▁Archives  完整的" ▁FA$ 做饭&  ▁lifetime( hot* 而成, 隐瞒.  ▁demonstrate0  ▁Turk2  ▁participating4 做得6 die8  ▁evil: Vol< 倾向> 入园@ buB ▁PedD  ▁trackingF 妥善H  的空间J 放宽L aniaN  ▁essentiallyP  ssssssssR  ▁MarineT dersV  ▁orientX  ▁continuZ  村委会\ 为其^ 排版` 介入b  achmentd  ▁boldf 做过h 保定j ▁FLl  ▁firedn  信息的p oubler Richt ▁Bluv Alexx  ▁workshopz  ▁behaviour|  ▁speakers~ 更快 PH 诠释  ▁transactions 愉快  管委会 ▁Wi  ▁Auto 特价  的特点  witzerland 履职 幽默 ▁AC 偷偷  辽宁省 潜在 tical  ▁glob  ▁ministry times 标的 ля де (( 打好  ▁reporter  我知道  扫描下方二维码  riculum 工伤 疗效  ▁Hope 一处‚  可以选择Ă 正义Ƃ  ▁recruȂ  ▁addressedʂ 防线̂  电动自行车΂ ratingЂ ▁sty҂ 就很Ԃ  ▁Philippinesւ wall؂ ▁godڂ 贴心܂ 中原ނ TD  ▁Khan  ▁whilst  ▁electrical  和工作 一番  信息网  的习惯 PDF ▁Opt 暗示  防控制 干预  ▁prest  全力以赴  ▁interactive 港口  条规定 疲惫  ▁Standard 经开 利的  因为你  ▁regulation 执着  可爱的 双子  ▁removal ACT  相关规定  ▁unlikely omic  ▁races  ▁grounds" 一站$ 津贴&  和家长(  ▁fundra* 一金,  ▁distingu. 首选0 是无2  联合国4 新房6 眼泪8 买到:  ▁Performance< 金山>  部编版@ rockB  ▁politiciansD 但却F  岗位要求H  ——　J 把你L 的地N ▁SOP  ▁ReservedR 联络T  ▁viewersV  ▁CorpX 海口Z  活动时间\ 改为^ 日益` 同心b 气管d CONf 行车h  ▁Corej  请联系我们删除l  ▁mapsn  调整为p 必需r  ▁unvet 周日v zensx  不小心z  ▁involvement| 算法~ 新人 容量 ▁Ark  流行病学 这门 iem 助手 失信 实操  ▁legit 历经 全套 播出 再生 往年 ","  ▁font  技术的  ▁Brooklyn  ▁suddenly  ▁complaint  ▁upgrade aro 他还  ▁promotion  ▁ethnic 小于 AST  Property ▁OS DAY  ▁imper 再到 双向ƒ Centă AKƃ  研究中心ȃ 光伏ʃ  ▁empower̃ Super΃  ▁choosingЃ 按钮҃  ▁Marsԃ 设有փ 额外؃ 湿地ڃ dr܃ 扬州ރ inth 不顾  ▁Constitution 警务 叠加 调控  ▁module  的一部分  效果图 双鱼 ugg books  ▁Planning ,' 锅中 跨境  ▁query array  ▁fault 不做 座的 stud  ▁absolute 亮相  纪录片  ▁pill ▁Pur 美景 ▁sin 角落 八字 好人  ▁Sustain 西医" ▁Il$  ▁classroom&  人教版( oscow*  ▁jail,  ▁destroyed.  改革开放0 味的2  等部门4 志成6  之前的8  ▁penalty:  ▁Dutch< 王者>  最美的@ 商户B 口语D  ▁relationF 车身H ausJ 五十L  ▁UnitN  ▁quoteP  ▁BeingR 下旬T  ▁ChapterV 周知X  ▁activelyZ ▁Dur\  ▁Such^ 长久` 预售b Emaild  为了让f abuh  ▁mysterj  ▁Javal 高职n  ▁Indexp healthr  ▁sacrt 火热v  微生物x 主演z 是的|  已经在~ 教会 主力 aver  被执行人 可达  男朋友  刚开始  ▁spots ▁BY 端正 utive  ribution Micro what 又不  ▁devast  ▁Paper  法律责任 场的  ▁gathered  也不用 存储 enna  ▁Kentucky 设定 想过 ▁tap  ▁Dean  有一定的 Create  ▁licensed  背景下 人居 辞职„ ankaĄ 崩溃Ƅ  incipleȄ ▁LEʄ  izations̄ 救护΄  对孩子Є ▁tox҄ bottԄ  ▁Pierք 升本؄  ▁laptڄ  leveland܄  ▁Propertyބ  ▁gifts  ▁vert  ▁divided  ▁protests  ▁chicken " 侵犯了您的合法权益 那天 定价  得不到 Pal  小姐姐  ▁Stew  ▁fascinating 煤矿 rt  专项整治 irds  ▁seeks 出轨 estone 石化  文章了 活得  ▁User aria  隔离治疗  ——“ 供水  ▁armed minute сь GM 不准 ippi" 巨蟹$ 优越&  ▁pleasure( ▁Fle* ento, Action.  马克思主义0 olars2 haust4 妇科6  ▁Source8  ▁deck: frac< 带货> ▁hip@ 沉淀B  是没有D  ▁defensiveF 甲状H  ▁greatlyJ 测温L  breakingN  ▁CapeP 破产R  ▁rangingT  ▁clockV 太高X emplZ Rober\  etition^  每个人的` 理科b ▁NOd  ▁Jessf  ▁bankingh  在国内j  ▁voicesl 码头n 车内p 极致r  的道路t Linev  ▁roughlyx COVIDz  和服务| 首发~  ▁Norm  ▁Oklahoma ▁Tru  ▁lowest 提倡 录制  ▁estimates 鞋子 笔者  中共党员 相继 预告  受疫情影响 ▁Gov Result 油画  这些人  ▁whenever 附加  perties 旗舰  ▁defe  ▁exclus 进来  招聘人数 耶稣  才发现  公布的 开课  销售额  project  ▁anxiety  ▁Duke  Through…  ▁dangerą 时常ƅ 往下ȅ 自家ʅ 处在̅ 从容΅ ramaЅ otton҅ 闺蜜ԅ 一笔օ  ▁Leadership؅ )“څ 内地܅ ▁Wyޅ ▁BC  rection 秒杀 usher 繁荣  等信息  ▁teeth ▁В  ▁Roberts Take  selling 同等  ▁Working Arch  ▁Creative 纠正  以上学历 压实  很多人都  ▁Capitol anne 日后  关系的 amer 饺子  半导体  ▁Galaxy 日照  ▁conservative  ▁grants ▁lad 硬件 养猪 商铺" 莆田$ 互通& cos(  ▁concepts* ishers, ▁DJ.  ▁Thus0  ▁Fant2  ▁cameras4 错峰6  是一名8  ▁Nash: assion< 大批> orses@  ▁investingB  参加工作D eiF ▁EXH  ▁consultationJ  ▁CrimeL  ▁MelbourneN stanP  日凌晨R  ▁formationT  ▁SpotV  满满的X  ▁stayingZ 赶到\ 戏剧^ 眼神`  ▁currencyb  performd  ▁offensef 严谨h hlj 所能l  招聘公告n 歉意p 财物r  ▁anywayt  ▁employedv  ▁computersx  ▁characteristicsz 力争|  ▁assigned~ 之上  ▁pride 固醇  ▁pushing 蓝天 徐州  目的地  aration  ▁Maria  ▁protecting  不动产  的指导 值守 开口 VR 伴有 uten  ▁thin  转化为  ▁adequ ча  ▁Tele hy Query ▁Ta 享有  ▁occurs Love ▁Gab  ▁Hamilton  ▁loop 负债  看不到 用工† quisĆ  相关信息Ɔ  职业技术学院Ȇ 震惊ʆ  ▁meaningful̆  sectionΆ 你我І  ▁minim҆  ▁shipsԆ 相伴ֆ  ▁desper؆  ▁Castleچ  北京大学܆  ▁suicideކ  完全是免费订阅 封面 变形 苏联 TY live omm oli  ▁Queens Bel #　  党委副书记  ▁Trail  ▁consec  ▁Autom  ▁Steel ку  ▁workforce  ▁gang ▁Cho  ▁clim  ▁concluded  的市场  ▁Walker 邢台 Global  健康监测  ▁Argent ted  俗话说  大家好  ▁Empire mem  ▁Jews"  ▁temperatures$  ▁Bron&  ▁newsletter( 遏制*  ▁Viol, 空军.  ▁Region0 花了2 分校4  ▁moral6 楼层8  半小时: 我很<  ▁parliament>  ▁conversations@ NumberB fitD ILLF  ▁LevelH 唱歌J  医疗卫生L ▁$\N 注入P 入驻R  ▁gratefulT 在意V  ▁survivalX  一下子Z  ▁Tokyo\  ▁borrow^  ▁elsewhere` 扫黑b ▁Obsd  小学生f 堪称h  ▁Phillj  ▁conclusionl 恶性n 冷冻p Finr  ▁moodt  ▁Bloomv ▁McKx  ▁administrativez 南路|  ▁Dance~  ▁tackle  ▁sharp  人才培养  ▁defeat  条件下  ▁vacation olen 守法  的视频 ▁Try  面试成绩 文职 ▁Cow  ▁opinions 提成 区间 Video ▁Naz 防御 City  ▁Terms  ▁environments  ▁indicated 约束 ▁BE 华东 MD 一楼  International  都不是  ▁journalists 花椒 里程 起步‡  ▁Andyć 连锁Ƈ oustȇ 题材ʇ  ▁NASȦ urer· 花的Ї  ▁Golf҇ 外部ԇ  和大家և direct؇ 还款ڇ 出国܇  ▁leatherއ 友好  教师资格证 说着 quire sing  ▁Kennedy 被动 参会 核桃 前方 奋力 father 条款  ▁ratio 赶快  中国共产党成立  ▁prev 运作  预防控制 ussion  ▁depression ▁USD ▁tie Sam Str linary  ▁instruments 聆听  有网友  ▁anymore  ▁bonus  //////// 全天 7 0来源标注错误或侵犯了您的合法权益" ▁Ec$  已成为& ▁Od(  的项目*  ▁enjoying,  ▁Military.  如果说0 ounced2 piece4 精美6 ailand8  Download: ▁odd<  ▁naturally>  ▁Diff@  ▁functionalB 和大D 商贸F besH engerJ 保卫L ORDN 取决P 头皮R  ▁LanguageT  ▁containerV ennyX 解毒Z  ▁го\ ▁Sab^  甲状腺`  ▁documentationb 解题d 外语f techh  ▁homelessj  ▁routinel  ▁compatn 加分p  净利润r orict 愤怒v  ▁Effectx 函数z  目的是|  加快推进~ Tag lined  ▁Samsung 遭到 约为  奇特的  吉林省  ▁Justin ▁Bah Long quiry leans  和发展 沿线  ▁Updated  children emed client  ▁cheese  ▁Update  ▁mystery ▁Wel Av  ▁tale master  ▁obst 还得  ▁amongst  ▁earnings  ▁Come wang  ▁cancel  ▁incent  ▁evaluationˆ 得住Ĉ 期货ƈ  ▁titledȈ  ▁superiorʈ 收缩̈  郑州市Έ JackЈ semble҈ istingԈ 不懈ֈ  的美好؈ icutڈ  这不是܈ 鼻子ވ 入党  — 长达 grade 手的 育人 roph  issippi mers  ictionary 和他  ▁finger ogs  区政府 ▁П 要以  ▁coaching  ▁Jonathan 借鉴  ▁rally  ▁movements 廉洁  ▁specified  ▁Object ▁Gun eals  ▁forum  ▁Freedom  ▁mistake 最具 热量  在学校  积极参与  ▁neighb" 饮用$ Af&  ▁interaction( 脱离*  一家人, 宽松.  不少人0 ▁Sov2 FI4 ▁McG6 励志8 学堂: 八卦< ▁Sac> ▁Tel@  ▁afraidB  ▁AndreD letedF  不少于H 转向J  族自治L selectN  你喜欢P  ▁discriminationR 全长T ▁DakV  行为的X 这三Z 整齐\ 丰田^  ▁disrupt` 搞定b aposd ▁**f  ▁Girlsh = 6若有来源标注错误或侵犯了您的合法权益j 内科l 奖项n 山药p  ▁cockr perst  ▁popularityv aliax  请注明z  ▁Vancouver| ▁PRO~ CEO  的可能性  ▁creativity  各位朋友 率为 杀人 几百 " 传递更多信息之目的  ▁paintings paren  ▁centuries backs 阻碍 淄博 画家  下一个  行政部门 带上 try  ▁parks 共识 Three 讨厌  ▁Unfortunately  高等学校 ▁cm  criptions  ▁petition Make 葫芦  ▁lake 月生  高水平 此刻‰ avedĉ ateralƉ  ▁Computerȉ 久了ʉ 严防̉ 临汾Ή  伴随着Љ  ▁alter҉  ▁screensԉ 谈判։ 她是؉ ▁EPډ  ▁smile܉ 谈话މ  ▁Sale unate overs 年第 oni 专场 Phil  ▁respective  自己在  ▁credits 天秤 还未  ▁relating del  心理健康 Both ▁__  职业资格  ▁investigate 充实  Response  商学院 迷茫 Rob 眼光  ▁Content 任意  missions 余额 HD 师大  ▁Made 难忘 恒大"  ▁gene$  ▁enorm& layout( 限度* uct, 站位. 沿海0 一股2 burn4  工作时间6  ▁Woman8 旋转: 身的<  了一些>  ▁Having@ ewsB  ▁ChelD 作息F  ▁paddH 碰到J  ▁stabilityL 灿烂N  ▁throwsP 省考R  ▁CommentT dirV 侵害X  请放心关注Z är\ Del^ 座椅`  ▁operatesb 如需d 膝盖f 计较h 胡椒j  ▁Shell 欢喜n  ▁tonightp  点个在看r  ▁Employt  疫情防控阻击战v 民国x googlez 主导|  告诉我~ raf 煤炭  October 阿拉 而被  ▁chocolate ▁liv  lementary 编审  ▁abortion  集团公司  ▁Kath 亮的  面向社会  受害人 感慨 面子 耐劳  幸福的  ▁Path  ▁cust  ▁conver ▁gal 队长 ▁iOS  ”　“ 名词  ▁Bull 中和 ega 专职  ▁euro  ▁affairs  ▁demonstratedŠ 图中Ċ  ▁preventionƊ  ▁geneticȊ 了他ʊ  ▁Christopher̊ assertΊ 变动Њ 图像Ҋ 心思Ԋ SI֊ oting؊  建设工程ڊ  属于自己的܊ judފ  都非常  我们一起  ▁ties 挫折  ▁panels zb  ▁Secret izza 付费 athon 纠结  不断提升 果然 沙发 的两  取决于 勤奋  本文转载 之为  一会儿 一号 ▁OK 收听 一来 node bered ibbean 大纲  Location  普通高中  ▁setup  强烈的 因子  人体的" they$ ▁ax&  ▁coup(  明确了* school,  ▁uniform.  ▁desc0  人民代表大会2 ▁Han4 玩耍6  ▁proport8 拼命:  ▁fantasy< 发烧> ardo@  ▁encounterB  考试时间D 一丝F 四年H reachJ  ▁ArabiaL EvN  金融机构P  脑血管R 烹饪T  ▁VentV 饮水X 法学Z  很可能\  ▁breath^ 白菜` 必定b  准备好d ▁FCf 应在h  也需要j  ▁Initil hadn  ▁operatorsp  ▁Readingr 所长t 上榜v  ▁Interviewx  要做好z 古典|  ▁packages~  ▁entity 全是 爆竹 ▁Eve 质感 August  ▁trips  立刻就  ▁antib 竣工 ensing  ▁breaks 演绎  因为他  ▁Investment 过滤 一根 开朗  ▁smartphone GC  ▁stayed  ▁Less 废物 ▁FBI  疫情防控指挥部 bow  ▁Additionally ▁ABC  ▁sixth  最重要 石洞  Welcome 禁忌 民宿‹  ▁jurisċ  进一步加强Ƌ  ▁Antonioȋ clickʋ  贸市场̋  或其他΋ 女主Ћ  科学技术ҋ  单位的ԋ  ▁courtesy֋  外国语؋  ▁Greeceڋ 富裕܋  ▁Daysދ  outheast 发声 进攻 BB  文明城市  ropical  ▁salary 大同 UST 投档  ▁turb  ▁emails  ▁Published  你自己 Lab  圣诞节 检疫  一个是 rones  ▁preferred  ▁admin 无可 雨水  医院的  ▁rescue Access 暖心 卡拉 olph 给出  ▁preced ( !长按上方二维码识别关注 nan 观赏"  ▁suffer$ 契机& ▁sid( ACE*  迎来了,  ▁lyrics. 扮演0 isdom2 露出4 ▁CC6  ▁overcome8 ublin: 巴巴< 克兰> 宁县@ 网点B 留意D 合计F  ▁chronicH  ▁SelfJ  ▁subsequentL ovesN 小人P  ▁blindR  ▁enemyT 鲜明V  parencyX 伴你Z  ▁utility\ 剧院^ 后勤` Joinb  ▁intersd  ▁graphicsf  ▁emotionh  ▁accompanj 遗址l 个省n laimp urir  ▁Economicst 淡淡v 永恒x Macz 一顿| ▁К~ 寓意 活泼 abies  ▁shower 小红 落后  新增确诊病例  生活方式  ▁Catal  ▁Liverpool 庄村 口臭 脖子 "]  chnology oning  ▁admit  幸福感 next lad  就可以了  ▁consistently 构造  ▁Gulf 还可  ▁Bruce  的人们  ▁Plant  绝大多数 激光 售后 三轮 华北 rotŒ  冷空气Č  多年来ƌ  年开始Ȍ 犹如ʌ asm̌  基础知识Ό  ▁portrayЌ  就会被Ҍ 事后Ԍ  ▁junior֌ 不觉،  ▁Thompsonڌ  ▁soccer܌ 之年ތ ▁<< 滨海  ▁casual  ▁minds  ▁tours plex ▁Li  December 心疼 反正 Base  ▁invas  ▁weapon  胆固醇 走在  许多人 ▁Sri  化妆品  ▁grid 柠檬 韭菜 erer ▁Esp  这个人 status 发型 地处 wered  ▁Political  商品房  把自己的  岗位职责  ▁pour  ▁inaug"  ▁boxes$  政策的& yu(  interest*  ▁marijuana,  为一体.  六年级0 遴选2 排毒4  ▁Orleans6 ㎡,8 ▁Na: 惠民< rary> 合物@ InsB  ▁exitD 情商F  ▁resolveH 言论J madL  ▁secretaryN  ▁shortlyP  ▁swimmingR боT angoV effectX NAMEZ  实在是\  用自己的^  ▁asks`  文化旅游b Unitedd rinef  ▁Swedenh 山路j  心理咨询l 翡翠n 对该p  副院长r 上网t  ▁Blockv issax 头上z 之力| 向您~ 模糊  ▁Denver  自媒体  Facebook  不允许  发展和  Project 河流 经查 神仙 公顷 夜听  一般来说  ▁meters ques 田园 ,( 农药 ▁SD 商店  市人民政府  就知道  ▁warned 仙女  众志成  ▁holidays  ▁Classic  ▁Lane  ▁Must  之一的 OUR ados  ▁blank 筑牢 GLč 充值ƍ  ▁integrityȍ Legʍ 步入̍ 盘点΍ 节气Ѝ  每天都有分享ҍ uvԍ  ▁moon֍  各方面؍  ▁bunchڍ 使人܍ ixedލ 但也  ▁hunt nell 最强 房企  越来越多的  ▁Exam  ository  ▁unem  ▁Episode  ▁Together 问我 stock  ▁buyers 财神 yler  ▁mathem  原材料  English riven 只想 sect  吃苦耐劳 出汗  ▁Prop 拿下  ▁tort  ▁criticism  市场监督管理局  tsburgh 普洱  出现发热  ▁advertis  ▁meals"  orgeous$ 此后& 宝马( 搞笑* 在乎,  众志成城. 交汇0  ▁racial2  能力强4 合规6  illance8  ▁careful:  Support<  ▁workshops> Que@  一般的B  领导下D  ▁attitudeF  唯一的H  经营者J  RelatedL  典型的N  ▁SometimesP 超出R 丰收T BreV 大楼X 不分Z  分享给\  也比较^  ▁printing` 广阔b  ▁Olympicsd  ▁substancef ooph  安全管理j iratesl  ▁Legisln 五行p  个月的r  ▁screeningt  推荐理由v  做出了x 用法z  ▁tons| 伤口~  市面上  ▁agreements LL  ▁Visit  向社会 单一 家务  学生们 集聚  ▁para  ▁Album oda 淋巴 安区 丧失 iop ▁Mun  ▁Holl 比起 pin 透气  ▁hired 去除 kee  有一定 common ttp 特意  ▁Russell  住房公积金 stop  ▁coaches 情景  的作品Ž ▁EVĎ 保洁Ǝ 语法Ȏ qingʎ  甚至是̎ 增大Ύ 母猪Ў  ▁emotionsҎ  ▁corruptionԎ  想不到֎ 大国؎  ▁Priorڎ + $是出于传递更多信息之目的܎ Didގ 眼中 作了 天后 变革  ▁lets ▁hub 复学  ▁cancell write center  ▁exploration  都是在  ▁rarely  ▁Moscow  ▁Iron  ▁Switzerland  ▁artificial 手套 人次  ▁tries  年以来 绿地 kind 寂寞 yang lang  November  ▁minimal ▁MO  ▁Mind  血液循环  ▁Urban 将被  ▁Dick" 屏障$  熟悉的& Share( 万个* Nav,  ▁robust. prises0  ▁directory2  ▁donations4 ultane6  ▁romantic8 深处:  ▁argued<  ▁arguments> 三人@ 答应B 没事D 创立F  ▁periodsH  viouslyJ 经纪L  下面的N ▁unnP  ▁renownR 实质T 节能V  对我们X  ▁wirelessZ  的角度\ heart^ leting` 一波b jxd 决胜f 呵护h 饰演j 建档l 菏泽n mediap Hir 体能t  ▁Garyv 快手x  ▁configurationz 邯郸| 审议~  ▁elder  ▁towns 北海 无线 恐慌  在内的 些年  你的人 夸张 王子 界的 他会 allel  极大的  ▁legendary 时髦 凸显 Phot Mart  ▁Convention annah 法庭 更能 词语 北路 cure  ▁Orange  ▁rival angh  的话题 统编 ousand  女性的 oving Schď  这篇文章Ə  孩子在ȏ  ▁bagsʏ 午餐̏  ▁survivΏ ▁("Џ 事关ҏ  评论区ԏ  ▁instant֏ 说出؏ ▁sedڏ  二等奖܏ 艺人ޏ  类专业 期满  作为一名 Sing  太多的 酒后  ▁dual  成都市 umph 打了  ▁Leader wy 敬业 表格  ▁Domin  ▁Labour 少吃 抽奖  William Music  ▁provisions + $詹飘飘詹飘飘詹飘飘詹飘飘 归来 拥堵 ogg  ▁weird  ▁boards  ▁bowl ▁gym  稳定的  体温检测  ▁dozen 米的  ▁Pear" 图为$ 节省&  起来的( 两者* 但这, ▁ans.  ▁Beijing0 开创2  你会发现4 进取6 阳区8 蚂蚁: ▁une<  ▁encouraging> 跟你@  ▁somehowB rapeutD BUF  namespaceH 弥补J 意向L  ▁fiscalN 如其P 二楼R 热度T  ▁gatheringV  ▁BreakX  每个人都Z 转为\ 不愿^  ▁satellite`  很多的b Persond 一台f  ▁allegationsh 年薪j 尊敬l  ▁Theatern  ▁engineersp 马路r ▁mmt  ▁gottenv  ▁constructedx 船舶z 外来|  ▁mountains~ bet 晚餐  ▁independence  ▁diagnosis True  ▁Gram ▁pul  ▁worried School  各种各  ▁Sony 衣物  ▁Alan ▁coc  ▁Movies 恐怕  ▁Interest 将对 底层 急诊  ▁Adams  的重点 气体 浦东 ▁kit  ▁Parent  充分利用 航母 四是  ▁lesson  ▁hide PI 人身 国学 作案Đ  转载此文Ɛ 秋冬Ȑ Paramʐ ▁IV̐  ▁Coleΐ " ▁▁▁▁▁▁▁▁▁А  ventionsҐ givingԐ Brit֐  五险一金ؐ  ▁nonprofitڐ 论述ܐ 中专ސ ieve 听课 阵容 shore 尼亚 ndrome  ▁refugees town  ▁modified  ▁annually  ▁jury 增收 beat than ▁Fed  ▁establishment 矫正 合影 平行 面粉  目前的  ▁tremend 大爷 股市  ▁toler  ▁Brexit 一组 全线 Former 体力  大家可以 aco 队的 ▁Gas" LD$ San& SM(  记者从* ivals,  ▁Kings.  ▁veterans0 ethe2  都不能4 ASE6 acious8 红的:  ▁nights< 纺织>  ▁hybrid@  ▁MexicanB BasedD  centuryF rusH LearnJ 冲锋L  成立于N 当作P  ▁viewingR 瞄准T  交叉口V  杭州市X ▁NBCZ ▁tu\ ▁fu^  ▁reviewed` 人际b ▁Johd  的人才f 阵地h emetj 张文l 叶片n 叶子p  西安市r  ▁Palacet  ▁sequencev 监理x 是非z  ▁flexibility| 交友~ shot 此事 维权  ▁None 四级 走上  ▁listing  党支部书记 си  ▁applying  ▁Cleveland woman  ▁pollution ве  ▁acres 缘分  ▁inspire  处女座  将成为 常住 言语 terior  ▁Intelligence 弟子 学士 央行  ▁delivers  平方公里 仪器 看待  ishment  的生命 头晕  的男人‘ 加盟đ  ▁раƑ 我区ȑ  Rememberʑ  assment̑ 旅居Α  ▁cannabisБ 汕头ґ Liveԑ  ▁excellence֑  ▁immigrantsؑ 多项ڑ  ▁careersܑ 开着ޑ 城管 恶意  ▁exclud 再见  ▁duties  ▁muscle  ▁Nichol  ▁enabled glas  September  ▁humanity  本文部分  ▁про 好几 灵异  ▁Phoenix 助推 终究 上演 呼和  家人的 切勿  ▁adoption 饭后  ▁accounting  ▁Federation 烟草 西亚  是一位 防水 新品  ▁participated  ▁exhaust  ▁edited"  ▁globally$ ▁WH& 几何( herent*  招生考试, entry. 氨基0 教科2  ▁populations4 elect6 Care8 牵头:  ▁districts< ailing> Law@ ASHB oftD ONEF  ▁passengerH 欧盟J  ▁BroadwayL 合一N WHP  表现为R 丢失T _{V  较高的X  ▁folkZ Button\  ▁flour^  起来了` ▁Rocb 也别d  ▁begunf 大战h 顶尖j  这才是l  予以删除n ▁Bhp  ▁legsr “　t ashingv  ▁keysx  ▁Portlandz  ▁experiencing| ulu~  ▁recall  ▁casino  ▁Rain  ▁Steven  ▁Snow 细则 NBA 月饼  ▁agricultural  科学与 流失 购票  ▁Challenge  ▁crack 若不 enez 因果 送上 不着  了起来 凡是 深情 达人 Expl 四十 颁发  ▁accordance Pass 特效  ▁attempted  Following  ▁certificate  ▁nationwide 百科’ 损坏Ē  之后的ƒ  专升本Ȓ  ▁Chamberʒ  ▁renewable̒ ▁ENΒ  老旧小区В 篇章Ғ  的政治Ԓ sis֒ Methodؒ 贵阳ڒ 停放ܒ  ▁importantlyޒ Enter  ▁Reserve 牢固 感激 信访 元旦  ▁absence 不为 roduct  月出生  ▁Agreement 没错  等你来 吃完 窗户 感知 达标 误区 aha 感叹 快捷 月光 自驾 lims 大蒜  ▁designers  对中国  生活在  ▁Million  ▁reflects 里有  ▁Native  ▁Lawrence 醒来" 倾斜$ Bo& 终点( 并发*  重要讲话精神, annon. 拇指0  ▁riding2  ▁Race4 Sw6  ▁achievement8  ▁agriculture: 因而< ▁Gib>  的做法@  的存在B turnD  的一项F edlyH  ▁wavesJ  ▁graphicL  不断提高N ﹌﹌P  ▁ApproR  教师招聘T  城乡居民V outubeX  ▁activistsZ 油价\  ▁drives^ 重组`  未来会b 供稿d 品行f  目标的h RTj Fel tainsn 打包p  ▁stakeholdersr 老的t 签字v 华夏x 查获z iche|  ▁spectacular~ 便于 开的  重要指示 ▁Bus 金牌  ▁parameters  ▁departments  ▁Finally  ▁healing  ▁kilomet 计时 Lay 果实 剩下  准备工作  形成的  去医院 Sk  ▁stocks Bet  ▁Kate 区块  ▁Maine  ▁principle 收拾 acles 稳步  ▁estimate  ▁continuous avas  福利待遇  同一个 念佛  ▁locally“ 恶心ē  县委书记Ɠ 衡水ȓ Thoseʓ  ▁pricing̓  不过是Γ 无私Г  ▁overseasғ  ▁Machineԓ 假的֓  ▁scholarshipؓ 加持ړ  ▁damagedܓ 上车ޓ 诉求  值得一提 jin 绍兴  ▁Louisiana 汗水 查找 各区 不像 衬衫  ▁Regist  ▁radical alg  ceptions  ▁settled ▁Hug ▁Ott  ▁controversial  ▁Could 延安  ▁displays  intendo  ▁declined 做个  urricane  ▁Singh 教案  税务局 翻炒  负责同志  ▁inclusion  ▁Oscar  ▁founding  的思想"  ▁Anim$ 各方& Input( 随即*  ilation, 楼梯. 绿卡0 越高2  ▁Products4 坚实6 三国8  ▁versus: 滋养< ▁Vi> 总有@ 两会B  可以直接D tooF  privateH 岁时J ▁rugL capN ▁NapP 用车R  ▁garageT 报送V 融媒X  ▁merelyZ  爸爸妈妈\ ▁nil^  识别二维码`  了一种b  ▁mainstreamd 分行f  ▁Experienceh  ▁Joshj  ▁enginesl QUn 有声p  被誉为r  排行榜t  一直是v  ▁Charliex 全场z  ▁engineer|  ▁surrounded~  ▁wanting  日常生活中  ▁Studios Sl 海岸  微信关注 毛病  ▁ruled  elligent 奶粉 克斯  ▁promised  ▁Mand  ▁Against )” 业态  定点医院  ▁Jennifer  积极性  ▁Baltimore  ▁sophistic 一篇 特种  ▁brut  ▁Friends  年全国 rator 众所 ▁Om  ▁footage 走的  ▁Side  ▁objectives ▁cha”  ▁codesĔ  ▁ordinaryƔ exportȔ  ▁Mississippiʔ  ▁desert̔  经历过Δ  价值的Д 问答Ҕ  目前已Ԕ 番茄֔ моؔ  ▁emergedڔ 躺在ܔ motherޔ  天然气  ▁epic  ▁Temple  ▁echo  ▁virtually  ▁Solar 封信  ▁argue  国家卫健委 people  support  ▁Taiwan ▁Bor  了解一下 drop 命名  ▁Host  ▁Blood  ▁Gordon  ▁accuracy idget  ▁checks  的照片 单品 不算 学前 国土 千元  ▁fake  推出了  ▁desired  ▁conventional  认识到 菜品"  ▁judges$  ▁crown& charge( 小伙*  ▁Excell, jan.  核酸检测阴性0  ▁allegedly2  ▁Malaysia4 """6 pled8  ▁assembly: 读懂<  ▁bottle> 紧缺@ 明日B 以色D 初三F  我们是H  ▁QualityJ 他也L ▁SudN  ▁easternP 生机R  ▁financingT ernameV  ▁CaribbeanX 营收Z  的车辆\  ▁applies^  医疗保障` Greenb  ▁instructiond  ▁Surveyf  ▁correctlyh  Meanwhilej 小火l 占有n  ▁Jacobp  itativer urioust  Professv  高品质x  身上的z define|  有时需要我们~ 贯通  ▁aest  因为他们 官宣 cedes 运河 ▁sle  ▁combine  两个月 侥幸 首页  相关负责人 Only  ▁dict fish phony  南京市  ▁Athlet asty  ▁Disease 所示 众人  ▁internationally Return  ▁corpor 西湖 少量 痛风  已经成为 emen  ▁buck Jul  ▁Perhaps arma• 报备ĕ ▁hazƕ  ▁marineȕ 对了ʕ 等着̕  ▁prayerΕ ▁TyЕ  ▁mathҕ tvԕ 模仿֕  ▁nodeؕ 堆积ڕ  ▁frequentܕ CIޕ 真情  ▁scenario  新能源汽车 до  已经被 人是  ▁possession orious  ▁proposals 实务  ▁hack  espread 死的 有余  中医院  呼和浩特  的心情 加以  ▁headquarters  复杂的  ▁renowned spring  ▁emphasis  Factory 征地  整个人  ▁safely  Research  ▁produces 公元 来得 牙膏  性疾病 生源" ellers$  ▁hiring& andra( ▁Bec* PN,  ▁manip. ▁HT0  ▁execution2 古镇4 zg6 网民8  ▁routes:  的一面< uffer> 大哥@  ▁disabilityB  ▁BirthD 钻石F  ▁circleH 品味J 刚好L  ▁therapN  ▁AssociateP twR 不离T  让我们一起V 忽然X  文化和旅游Z  非常好\ ospel^  ▁churches`  ▁Indonesiab  报名表d  ▁culturesf  个百分h  ▁loudj  ▁ranksl  ▁spectrumn  ▁dozensp bulr  其他的t  研究员v  ▁puzzx  ▁Thirdz chi|  综合性~  是我国 ▁к  ▁Jazz  process ${  ▁promises 人与  ▁Strateg 冲动 天赋 万科 混乱 Sur 立法  ▁Everything  ▁subscription 可怜  ▁Very  ▁frequency Apple omed  ▁summary  ▁Patri 漏洞  ▁eggs 总分 Found  ▁packed 两条 学家 lett  ▁testim elli 老了– ▁SinĖ  ▁belovedƖ  专业知识Ȗ ▁URLʖ  往期推荐̖  AddressΖ 座位Ж  ▁ReleҖ 献血Ԗ 帝国֖ 性感ؖ ▁Aveږ 口岸ܖ  ▁weighޖ  ▁negotiations 廊坊  提升了  Austral 悲伤 复旦  ▁Nelson 无穷  疾病预防控制 火了 mal eness Show 一到  辅导员  ▁Ident bd 直达  inction Cam  emetery  ▁Graham  ▁Nort 偶然  ▁Mort Does  ▁render  　:　  小小的 统考 没钱  ▁Captain SR Media" 一审$  加油站& 奔驰(  然后再*  ▁hitting, hattan.  ▁Cars0  的意见2  ▁pregnancy4  管理人员6 ▁Cop8  信息技术: san< 哪儿>  ▁Summit@  ▁governanceB 散文D  医疗器械F  这一年H blogJ actorL  原来的N TemP  新征程R  ▁arbitT  ▁certificationV  ▁boneX 要做Z 不完\  context^ 推文`  新冠病毒疫苗b  ▁Sovietd  ▁Connecticutf URLh jingj  ▁Arenl  来自于网络n  ▁delightedp ▁Bior ▁Himt dalev 团圆x  ▁Wrightz xia| 匆匆~ 无声 择优  奖学金 痕迹 而上  ▁charging 事迹  ▁liability liver  ▁Senator  历史的  规范化 薄弱 宽容 下属  但如果 静静 礼仪 oded  ▁receives  ▁disorder  ▁literary  ▁consecutive USA  ▁affili zech  ▁Christians 高原 }{ 仅有  ▁keen  建设和  ▁Practice  ▁pregnant— IIIė 完工Ɨ  的速度ȗ  通知我们ʗ 夫人̗  进入了Η  ▁unlikeЗ 进门җ  ▁feminԗ 远方֗  ▁Successؗ 几点ڗ gieܗ  很快就ޗ  如今的  ▁Bath ixel  一直以来 习俗  ▁Campaign 廉政 Main  ▁fought 愉悦 司令 红线 高手 雷锋  ▁automatic  安全带 联赛  ▁deadline esla 珠宝 限行  ▁innoc wear  ropolitan 尽早 nc 友情  ▁Pant  ▁deter 美白 Europe 故乡 制药 般若"  政府的$ 大米& 六个(  ▁Associated*  ▁judgment,  积极配合.  ▁Palestinian0  ▁cake2  第二批4  毕业于6  没有了8 尽力: 车位<  ▁Baby>  ▁Brew@ OperB  ▁immuneD  ▁donationF 细心H aghJ 出色L 互助N stageP  ▁tennisR 会同T 树木V 沉浸X 华中Z 均有\ early^ 自学` няb 干事d ▁sif  万元的h  ▁referencesj 疑惑l 金刚n 最全p  ▁publishr  westernt 派遣v ▁Cryx 语录z  健康状况|  每天都~  ▁Stars 科创 rowing 使其 造价 锁定 请勿  ▁curriculum  ▁Buck  ▁Thailand  ▁Fame 童话  document ▁Rat  ▁Poland  严格的  ▁alarm  ▁Tree  ▁rend  ▁intervention  ▁drag ▁Lu Full  不可以 首届 下发  ▁Hans  具体的 brit 轻微  ▁bull  可以看出 才华  新闻发布会˜  ▁BeckĘ 避开Ƙ atileȘ  ▁classicalʘ 手上̘ 套装Θ  ▁liberalИ 人到Ҙ 区长Ԙ  开学时间֘ 入场ؘ 餐桌ژ  工作要求ܘ >'ޘ 出自  ▁Tower  不确定  资格复审 抵押  公布了 ▁Cos  表示歉意  ervices  ▁functionality 圈子 对口  ▁divers 蓝字 plant  天蝎座 小吃  以色列  ▁smoke ▁Pil 毫升  ▁enhanced  取得的  ▁reporters 赋能  ▁mechanism ▁ham 含义 扩展 房贷  考试成绩 erse  response 准时" 乌鲁$ 也让& 血栓(  ▁resil*  刑事责任, 民俗. 双重0 主观2  ▁Orig4 vity6  人员进行8 联邦:  ▁advocate< 名片> 两国@  它们的B 列为D  ecessaryF optionH usalemJ 型的L 高低N  ▁terribleP  ▁spokenR  平台上T  服务业V 售卖X 少儿Z  准备了\ 必修^  国家税务总局`  ▁Constructionb flyd ▁CTf  ▁mistakesh  街道办事处j  ▁aggressivel ▁Jenn 再度p  企事业单位r  个百分点t  往期精彩回顾v  鼎力支持x  ▁Muchz 星级| 年间~  专注于  版权声明 车牌 IGHT  有两个  ▁Anna 时要 encer 优美 ▁Ju 代言  ▁Deputy 有用  ▁tagged  坚持以 泉州 春秋  ▁plain hero acco Tok 服饰  的意义 mind  ▁possess 拯救  的知识  ▁rejected IDS  ▁не  他们是  气象台  优惠政策 module™ 市值ę ▁Rapƙ  ▁lobbyș  ▁drinksʙ 登上̙ jaminΙ  ▁restoreЙ 外套ҙ  pectivesԙ 一双֙ blingؙ 不难ڙ  告诉我们ܙ zjޙ elve 金银 ois  ▁Application arity  ▁nursing  ▁entrepreneurs  更高的  职业学院 器的 简洁 繁殖 巴西  数据库  ▁Princip  ▁prestigious ▁gre 奋勇 在上  ▁purchasing 本市 ▁Nik  让你的 fi  和支持  ▁Xbox 决战 ["  情况的  ▁simultane ▁Joy  专项行动 ▁Aqu 专员" Non$ 包头& false(  ▁rehab*  ▁rice, 抓实.  comfort0 ▁ft2  ▁disturb4  工程建设6  article8 结节: 火的< 上调>  ▁encount@  ▁TitleB 文科D  ▁suppliersF ▁MasH  乌鲁木J  志愿填报L  ▁MagnN 冒险P  ▁FacultyR 保密T 长治V iwX 诱发Z  ▁alike\ rous^  防护服`  ▁Dealb  ▁postingd Gettyf 良心h olyj  ▁Versionl  的方向n  ▁Strep provr Whicht  ▁Commissionerv  ▁citizenx  第一名z comed| 嘉兴~  ▁Impact  ▁wooden 注销 airy 结石  ▁rolling 耽误 滋味 只好  ▁Following 投身  ▁inspiring  ▁touchdown big  乌鲁木齐  ▁Carter 宝贵 将以 仅为  ▁Meeting  人大代表  网友们 酱油 sur  ▁formula 接地 党内 天堂  我们会第一时间 Robert omas  ▁flower  ▁Dubai 一笑š  在疫情Ě 仍有ƚ 基建Ț  是一家ʚ 实效̚  长三角Κ  ▁IndependentК ectedҚ  ▁mayorԚ 原油֚  工作报告ؚ  ▁auctionښ  ▁Stewartܚ 之情ޚ  ▁astron 两岸 下游 美团  acteria 摩擦 夯实  了一份 се  四年级 dc  还记得 Rad 终止  ▁tower  不用担心 为期  ▁blow 三位 ▁Gre ▁Inn  ▁aver  的课程  经开区  这样的人  ▁colleges  theless Util  ▁suggestions  ▁examine  最新消息  ▁addressing  的身体 ▁pin"  ▁Activ$  ▁vaccines&  ▁Fast(  ▁painted* 春晚,  ▁Companies. 豆瓣0  ▁keyboard2  ▁Better4  ▁partnerships6 下水8 草莓: ARD< 机体> ▁cow@  品牌的B 几种D  最新的F 清风H ▁);J 长征L  ▁BasedN  ▁newestP ScriptR  ▁AlberT  ▁MathV 一览X 牢牢Z ivo\ orized^  ▁Industrial`  的说法b 垂直d 改正f  ▁heartsh  ▁searj 7 0转载此文是出于传递更多信息之目的l  ▁circuitn  ▁Deepp  ▁exclusivelyr  ▁Createt  ▁Commonsv 录音x uruz  writing|  烟花爆竹~  ▁honored ANT 健脾 面貌  ▁Diam 受贿  ▁Fight  ▁yard 前线  ▁contents 卫视  改变了  ▁Highway Load ▁bio  ▁overwhelming  ▁courage  ▁Warren 玩法 ере header 生效 命中  大家在 吉利 ervous  ▁fastest 疏导  ▁Body 伤心 促销  ▁tribute  ▁Cand  ▁maxim› 也为ě  高风险ƛ  displayț  ▁skilledʛ 大使̛  ▁CommerceΛ uerЛ 在其қ hedralԛ 特岗֛  ▁Citiz؛  ▁Playerڛ  通讯员ܛ smallޛ nal ocur  ▁stops 这名 给您  Builder anches  社会各界  合伙人  ▁celebrating  没办法 除非 变速 大理 ourt  ▁gentle  ▁Hawaii  ▁connecting  ▁Coal  ▁threatened  ▁designated 白的  ▁loving  病毒的  给他们 逃避 ▁CNN  ▁Boys  并同时  ▁checking 茂名  ▁shed  ▁preserve 次的"  itement$  市场监管局&  国民党(  第一位* 江山, 更改.  的公告0 摸排2 相机4 ▁Ky6 祝愿8 糊涂:  ▁underlying< Wed> make@ 困惑B  请注意D  ▁abilitiesF ▁HitH 台阶J whiteL StyN  limitedP  公共服务R  ▁ContractT ChrisV 修建X 数额Z 不尽\ esides^  ▁wage`  了一场b  ▁acclaimd  ▁Strategyf  消毒液h Policej 早起l 开盘n 春运p  irminghamr  ▁Dublint 就来v  ▁sitsx BTz  ▁calm|  广大群众~  ▁exhibit hang  ▁Comb  ▁stem  ▁finest 莲花 直观 精装 Status 持久  ▁notable 探寻  ▁refers 皱纹 入群 不打  ▁impression  ▁enormous 故宫  相同的 无意  ▁lens 过大  ▁guns  ▁equality  ▁Think 斯特  ▁composition 品尝  ▁captain  很强的  公告如下 ör  交警大队œ  这些都是Ĝ ntilƜ 风口Ȝ  你是否ʜ esis̜ 上看Μ 歌手М 祭扫Ҝ 南县Ԝ  接下来的֜  恢复正常؜ 随地ڜ 份额ܜ  ictionsޜ  最高气温  称之为 平和  怎么做 顺畅 钢筋 ▁sew  ▁widespread  ▁Bachelor 战疫 本号 举动 再来  android  ▁serial 丝毫  ▁photographer 梦幻 电流  所说的 ▁obl 消灭  成立了  这段时间 rams 建房 神的  你可能 ▁Amy 考查 tons  生产建设  ▁implications 厂商"  ▁cited$  原因是&  的营养(  ▁tissue* 疏通, 都比. 立卡0 自理2 高一4  ▁abstract6 伤亡8  ▁productivity: 真理< should>  ▁guaranteed@ 较好B  结束了D  ▁explanationF  应聘者H  ▁FranklinJ  关键的L rootN  ▁fleetP  ▁arrangementsR 为由T 不该V  ilitationX  物业管理Z  ▁Colomb\ 娃娃^ 汽油`  ▁Emailb  全媒体d  ▁facilitatef  Lookingh  ▁abundj 歌唱l 报酬n 下半p  ▁blendr  ▁progressivet  ▁inevv  ▁profitsx  ▁measuredz  在自己|  怎么能~  ▁Operations 生殖 ▁Cou 彩虹 符号  ▁walks 吵架 may  ▁evaluate 电机 untu 场面  有趣的  ▁swing Women liest  uploads  ▁disag  值得一提的是 eremy  ▁translation 难过  ▁spokesman  建档立卡 India  ▁neutral ▁gig  岁以上 收纳  ▁finishing  ▁broader  适当的  ▁Housing  ▁inflation 不被ĝ othyƝ 除恶ȝ 恭喜ʝ 热血̝  ▁RobinsonΝ ▁fibН ▁Jobҝ 沿着ԝ 揭秘֝  注意到؝ 缅甸ڝ ▁pupܝ 欧美ޝ  ▁fert ▁Kal  February 男主  消防安全 blem  管理和  ▁Pittsburgh  ▁unpre  ▁followers 发光 ▁Say 拒不 将近  也不知道  ▁rhyth  行政区  ▁critics ▁Nut  ▁journalism 时空 要多 lon 得分  发改委 ні  ▁defence ▁gam 有权 惠州  一线的  ▁Glass 落到 泡沫" 答复$  ▁Falls& hou( commun*  ▁slot, 要用.  ▁telephone0 精细2 物体4  ▁Arthur6 atoes8 大咖:  ▁nominated<  ▁owns> ARE@ 挣钱B  全面建成D iselyF  微企业H  超过了J  ▁embedL CustomN 尽责P  文化遗产R cribedT ▁RowV 巅峰X  ▁poweredZ  ▁admission\ 特长^ 逆袭`  最大限度b  众所周知d 樱花f izonth 右侧j  forwardl  不舒服n  Categoriesp  从来没有r rikt whev ▁”x  让学生z 肺癌|  是一件~ 估值 村镇 青睐  人员在  ▁ratings  ▁layout 就越 本案  已经开始 target  周岁以下  成功后  版权问题 abe  学习和  ▁competing 中铁  ▁compelling  符合条件的 倾听 ▁Hub func ▁Da ▁mad 复古 单价 公立 货运 ▁LO 未必 被困 背部  女朋友  ▁Partnershipž 观音Ğ 遵义ƞ  的名字Ȟ  ▁wonderingʞ  ▁programmes̞ UpdateΞ 超值О  检测结果Ҟ TitleԞ  ▁visitor֞ ме؞  ▁transferredڞ 专人ܞ  阶段性ޞ 文书  History ▁HR  ▁accommodation 互联 债券  非常有 ▁las 余名 Ann ABC 照明  不得不说 Report  ▁reception 描写  ▁tourists  ▁Restaur  ▁academ 治病 印刷  ▁cultiv ▁Ice 通透 ifully  ▁Dest 喝茶 你也  ▁burden  ▁gorgeous  ▁Analy Game  ▁examination  一年的" ▁unw$ 大声& 退伍(  ▁cruise*  ▁reverse,  图片来源于网络. 存放0 无一2  家庭教育4  违纪违法6  ▁Madrid8 ▁FDA: 明珠<  高等教育> 破解@  ▁worshipB  ▁acquireD  感谢你们的F 不经H 依规J  ▁thermL omesN  ▁accompaniedP  ▁welfareR 你对T  ▁EnterpriseV rerX 蝴蝶Z  ▁Metro\ ITH^ 不胜`  做到了b 内衣d 不来f  founderh  ▁Charlottej  ▁showcl 查阅n 讲堂p 既往r  另一半t  等内容v  ▁peerx 幻想z  ▁lawmakers|  我认为~  ▁Bear 果断  左右的 秉承 ▁ski 洪水  ▁exch 保温 油腻  ▁Junior  ▁tourist  ▁musician  ▁intrig  ▁Murray ▁hur cycl  可以帮助  ▁checked 彩色  具有良好的  ▁Baker  局局长 清香  ▁composed  ▁Barcel 会使  不仅能  ▁Dakota  ▁possibilities  ▁Wayne  ▁Alaska  ▁Server 如期  让人心痛Ÿ  ▁complaintsğ ropriƟ  更重要的是ȟ animʟ  采访时̟ <<Ο  ▁novelsП owskiҟ udaԟ ▁WW֟ SQL؟ ▁dilڟ  ▁artworkܟ  ▁Turkishޟ  提醒大家 做题 福报 负荷 是他  ▁servers 都将  ▁enjoys  ▁pipeline 指责  集中在 inery  引起了 role fight  不适合 ocate  水平的 已在  牢记使命  就需要  ▁divorce  ▁Writing append  ▁destruction  ▁indicates two  ▁stadium 西兰  ▁startup 名家  最重要的是 冰雪 的红" 你若$  ▁Manufact&  全社会( 状元* 嘴巴, 刹车.  ▁thrown0  每一个人2  ▁landing4 rative6 EU8 rates:  ▁flavor< ▁Ale>  ▁Amendment@ 风貌B 自豪D  ▁deserveF  ▁disabledH 颁奖J  的良好L 看书N 水上P  isticalR  公共交通工具T 一幅V 血流X oatZ  任何人\ 推介^ 嘴里` 刷题b  可直接d 7 0反反复复反反复复反反复复反反复复f  ▁hadnh 营运j  首先要l  诊断为n  生产建设兵团p 透过r fct  五中全会v  原创者x  ▁Springsz 一无|  ▁authentic~ 妇联  ▁survived 人格  青岛市 实话 下雨 都得  ▁Historical  ▁Sean  消费券 辜负  时不低头  ▁seemingly oples  ▁attorneys  当地时间 欺骗  发生后 笑着  ▁Advanced  ▁seam  ▁influenced  我们对 真爱  ▁Directors 研制  ▁trusted CN 口号 大致  ▁float ▁Fab 全局  ognitive  妇幼Ġ  大多数人Ơ  ▁OrchestraȠ  ▁Lukeʠ  特色的̠ ▁fruΠ 珍珠Р 肉类Ҡ 手臂Ԡ 学籍֠  等多个ؠ  所学校ڠ  的建设ܠ 巨头ޠ  工程学院 access 民航  Component 心得 福气  cedented uction 同胞  ▁terrorist  几个月  ▁convey 阶级  ascular  ▁suspended 演奏 突击  留下了  ▁invested  中发现  ▁retailers ски  ▁sanctions  ▁satisfaction 抬头 中外  精神和 ▁Pel rage  ▁genuine  日常生活  powered  ▁odds  ▁bool" 机电$  ▁hyper& lando( 纯粹*  ▁Soul, 棉花.  ▁collaborative0  ▁editing2  农贸市场4 >>>6  ▁vibrant8 Right:  ▁pink< 在建> Sun@  ▁securedB 随身D ceiveF eleteH 胎儿J  ▁infectionsL hillN  issanceP 博览R  ▁universalT elryV  ▁filingX ▁PasZ CG\ 肺部^ 亲友`  能够在b 油脂d xsf aciónh  ▁Screenj  ▁advisedl  ▁Pickn  ▁lungp  ▁reserver 更正t  业有限公司v 考上x  知识的z 白衣|  ▁Would~ 期盼 White  ▁wing  ▁Manhattan 坏了 看好  时期的  ▁cinema én static senal  ▁dispute 终审  ▁Morris ▁Mos ▁М 考古 成语  程度的  ▁Hunter 有无  实践活动  ▁Lisa  ▁impressed  ▁difficulty ▁din 样板  前段时间  内分泌 复课 好玩  是比较 rust  现实生活中¡ ▁spoġ  马拉松ơ 马云ȡ  Whetherʡ 三月̡ onedΡ ivicС  ▁Culturalҡ  的动力ԡ  月参加工作֡ 名人ء ikhڡ INDܡ  ▁anticipatedޡ  情况属实  学会了  ▁hunting 好听 wealth  ▁analytics ▁Fal  ▁patent  ▁promising 发财  这里是  ▁Cell 拉克 时政  到我家 ppers 身亡  ▁Speed  ▁backed 旋律 Pop  教育和 适时  ▁Corps 流畅  ▁Artist 书店  ▁minority  决定了  ▁steal  ▁Krist  参加考试 而又  ▁conversion"  ▁legislative$  ▁Chen& 扩建( his*  ▁Initiative, 到手. 剧本0  本文转载自网络2 把我4  的事实6  文字与8 门前:  ▁congress< xygen> 藏着@ DevB  ▁municipalD  这一天F neredH 南瓜J  ▁welcomedL  ▁sonsN  ▁inhabP  欢迎各位朋友R 美观T 至关V 建材X 重型Z  狮子座\ 非遗^ 兼顾` 澎湃b ▁Nepd  ▁Coachf 地域h 实景j strongl ▁ESTn 上周p 号牌r  副校长t  综合素质v  ▁accompx  新疆生产建设兵团z  ▁Morning| num~ 大兴  表现出  ▁Stream ▁NHS  ▁Give  ▁beliefs 等的  类型的 ↓　 风吹 初试  ▁responsibilities  ▁Cart  ▁foss  ▁Muslims >{ 小儿  在上海  ▁Technologies  ▁fallen 红灯  ▁Iranian  ▁resume  ▁Track 做成  扫描二维码 adata 城际  ▁blame  ▁conflic walk  组成部分 Pay 折磨¢  思想政治Ģ  ▁FashionƢ 智商Ȣ  的政策ʢ  ▁eliminate̢  ▁carries΢  万美元Т 灵感Ң 火星Ԣ  ▁mortgage֢ 县域آ  ▁refereڢ 违约ܢ  ▁decreaseޢ 胸部 浓厚 唐山 妹子 否认 他对 ▁RO  区域内  明白了  以来的 一瓶  副总经理  ▁iPad  被害人 巴黎 idal ▁LED 拓宽  ▁Impro  ▁robot tha  二手车 内饰 全额  基本的 膨胀  ▁navigation final 江北  ▁repeat  ▁Campbell 想看 上图 裁判"  aughters$ ATH&  ▁completing( 惩罚* 检修, 配偶. 逮捕0  可以让2 隆重4 ivan6 ▁Fem8  ▁Universal: 指令<  ▁enabling> ▁Lux@ 一碗B  ▁contributingD 救灾F  ▁MagicH  一件事J  ▁harvestL 市的N  证明材料P  ▁autonomR ievalT  领导班子V 元宵X  他们会Z  ▁condem\ 爱护^  ▁surely`  ▁militb paperd  却没有f 产地h 鉴于j стиl ▁SHn ATEDp cycler  ▁failingt  ▁Gillv Secondx  子公司z 南阳| 毛孔~  ▁Jerusalem "),  ▁Barcelona 路人 显瘦 再去  ▁implementing ▁shr 下沉  ▁tired  ▁hopefully iga  端午节 Fore  ▁practition 三亚  ▁explores  ▁appearances  公众平台  环境卫生  ▁gains 牵挂 cp 和尚 妙的  关注公众号  ▁Albert *** ogan 一头 还好 滋润 邮寄  ▁Susan£ 字母ģ  ▁sauceƣ 三星ȣ  ▁Pollʣ  ▁Jobṣ  ▁executivesΣ 没能У 万千ң ▁tarԣ 演技֣  oundingأ  ▁Resultsڣ  让人心痛的话ܣ  ▁supplementޣ 递补 space econom  的决定 学年 农家  ▁Blues 裤子 ▁ML 本领  尽量避免 债权 长假  要认真 我有  ▁diabetes 挽回  ▁до  ▁Clinical 为啥 ▁CBS ▁ske  ▁damages 当晚  中小企业  ▁claiming  住房和 ▁bol 鸽子 ▁SEC  主要有  avascript etics  ▁conj"  ▁Mitchell$ 气场&  ▁toile( frame* raham, lem.  组织实施0 alties2 原告4  iplinary6 奔跑8  ▁highlighted: lian< 门户> 散步@  ▁achievingB saD  ▁StevF 口的H  深刻的J keysL ▁CabN BERP  ▁technologicalR 尤为T  ▁BarbaraV xwX  ▁interactionsZ 描绘\ ▁Ted^  ▁premiere` 出身b  ▁epidd 传来f  消防员h ▁Kamj  ▁prospectl  相信未来会n стоp  ▁Finer  ▁controllert  ▁mildv 退还x  ▁entriesz + $让人心痛的话欢迎各位朋友| 针灸~  ▁Bond  回家感谢你们的 ( !回家感谢你们的鼎力支持  ▁somebody  体内的 小平 L E让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持  携手前行  inburgh 批复  使用权  ▁accessories  ▁packaging  ▁edit server 划算  希望能  服务员 ال  电子商务  发布会上 奢侈  ▁beneficial args ▁Tam  了中国 tail acent  ▁Laura  ▁Kenya 滨江  保险公司  出席会议 平淡¤ mirĤ  右下角Ƥ 主讲Ȥ amicʤ Cell̤ 相连Τ  基金会Ф WAҤ  ▁blessԤ rb֤ ▁Kitؤ 自古ڤ  ▁Circuitܤ ▁Hadޤ 却在 水稻 科院  营业时间 既是 fare  ▁anch vit  ▁striking 多余 [ T让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持相信未来会 ▁Cut  ▁venues 年至 Having 特此 全职 得出  ▁inch 不减  征求意见  ▁Brun  ▁perpet 国考  自己是  ▁Myst  影响到 骨头 下达 :@  教师的 废弃  关于做好  还是很" echo$  ▁grace& ocker( 接送*  爱的人, 米饭.  ▁dose0  实验小学2 金华4 容器6  entials8  我相信: 指甲< 汉语>  ▁ambit@ 监会B  ▁hangingD  招聘会F  ▁migrationH ▁LotJ  ▁dancingL 区内N 动了P  ▁inviteR  ▁mechanicalT  扫黑除恶V  ▁CollinsX StatZ  ▁disput\ 刻意^ 总支`  分钟后b  ▁orald Podf  高标准h  ▁Operaj 祝贺l 星球n ▁Ianp  人社局r  ▁hillt  plementv   x basz  冷链食品|  ▁limitations~  五年级 黄瓜 商丘 到期 排序 中标 >"  尤其是在  ▁dispos 店内 首付  Handler  ▁Hamp  ▁Hell  有力的  ▁Nevada 干警  ▁traditions  为确保 熊猫 cient ;②  ▁Document bian 名叫 晚年 伦敦 班的 堂叔 海拔  ▁dent  旅游景区 ▁unt  在这种¥ 风湿ĥ  ▁ladyƥ  示意图ȥ  是一场ʥ  ▁Marshall̥  ▁entitiesΥ SocialХ 套房ҥ 这不ԥ 引擎֥ 过硬إ 年华ڥ ▁DIܥ 阻断ޥ 佛法 чи bie 专辑 Old  ▁sust  实验学校  ▁holes non  占地面积 nest  ▁outbreak ▁USB 宜居  ▁retain  ▁suite 复核 udson  即使是 糟糕 交代 green  ▁Bart 样品 ▁Yan 了两 偶像  健身房  ▁consultant  ▁showcase ancell  ▁funeral  党工委 大便" 停留$ ▁hes& 带薪( 祭祀* 主治,  但其实. 孔子0 som2  ▁Campus4  在全球6 ▁fut8  elihood:  大幅度发< 标杆> ABLE@ 红枣B 筹备D 大牌F  有问题H ▁DalJ  ▁grossL 撰写N  FinallyP  ▁goldenR  ▁CleanT  群众办实事V  了一条X  ▁AaronZ  ▁affects\ KS^ 变为`  ▁Craigb  应急处置d  ▁abandonedf ollarh ▁Powj  ▁comedl  教职工n 低保p 网易r  点这里t TONv 何况x 绕行z  月下旬| mart~ oween 财税 饲养 imens 流入 厂房  海南省  这里有 新乡 ▁Ger  我国的 越野  ▁counts  ▁sponsored  ▁boundaries ▁unp  心脏病 agedy  ▁networking method  参加会议 肩膀 所致  ▁Podcast 壮大 打底 降价  ▁dental  新一轮 gal 个小  完成后  ▁shelter ▁Esc¦  大家都知道Ħ  ▁memorableƦ 之以Ȧ  ▁requiringʦ  ▁Transportation̦ 置顶Φ  ▁CommandЦ ▁urlҦ  ▁eliteԦ 就说֦ 往返ئ  ▁Parkerڦ ▁Lieܦ 年年ަ oren gments 三支  ▁chest 解锁  ▁Linux 智力  市委常委 Year Case  ▁consulting 智者  市中心 维度 Miss  ▁tempor 无聊 积水 Aud ▁mic  已完成 私家 悬挂 任命  物业服务 Peter  附近的 河道 重心 宿迁  如果在 ▁ein  ▁mixture  深入开展"  联合会$  ▁competitors& 秘诀( ▁bin* erver, 知晓.  ▁Programs0  不知不觉2 igma4 微微6 ▁Guy8 省长: ( !我们会第一时间予以删除<  ▁Everyone> ▁Kl@  ▁inequB  合理安排D ▁RuF  今年以来H avanJ  ▁invalL ARYN  ▁appreciatedP  县融媒体中心R  毕业证T 社团V  ▁quotesX  的人都Z  ▁barriers\ 无处^ 根源`  ▁eagerb  被评为d 蜜多f 联考h  吃喝玩乐j  ▁strugglesl  ▁litigationn  ▁readsp  leadingr  ▁neighbourt querv  ▁launchingx 花卉z number| 老实~ 日军 @ 9▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  与我们联系 emon 哭了 边界  水瓶座 请你  并同时向您 就地  iliation ILE ▁г 密封 摇号  组成的 ▁NYC  ▁tremendous 菊花  大家都在  ▁Chelsea 而后  的女性  工业大学 重建  ▁clicking  ▁entertaining  ▁ancest  ▁directions  ▁assists  当地的  uscript  三等奖 长了§ 大雨ħ 成名Ƨ Linȧ abadʧ  ▁hockey̧  ▁accomplishedΧ  较好的Ч 离职ҧ  ########ԧ 停课֧ 行李ا 长效ڧ  是真的ܧ 液体ާ acre  hibition 白糖 低温  ▁hardly 所说  这意味着 器材 ▁GOP Using  为中国 heimer  ▁capability  的压力 绿码 梅花  ▁Jimmy 点燃  最高人民法院  ▁Grace 必将 ampa  广播电视  ▁Ward 心病  ▁activist 绝不 陶瓷  ▁Leave Editor thod  ▁physically 整整  ▁folder" 这点$  ▁Moreover& Bi(  优质的* же, 直径. rons0 骗局2  ▁Emergency4 oj6  ▁Anti8 Ver:  ▁HTML<  ▁repeatedly>  ▁horses@  总面积B speedD  ▁ColdF  本科生H 累了J  副部长L  睡前伴你N  ▁barelyP agraphR  ▁AvailableT ▁SelV  ▁deliberX  加大对Z 沙漠\ War^ ▁Ze`  应急预案b simd  Updatedf Crh 七八j 山镇l  ▁experimentaln 的开p  ▁multipr zonet suchv  ▁Mallx  ▁diagnosedz  ▁seeds|  ▁disabilities~ 手脚 诚聘  ▁layers  Command bus  等形式  ▁Syrian hell sign  家企业 inates 谅解 弥陀 plug  ▁comply 环卫  ▁landmark  ▁momentum  ographical format  另一种  ▁Kind  ▁makers Mult ▁Ele  ▁wrest  非必要  Windows  文字与图片  的老师 second 总价  一站式 抗战¨  辖区内Ĩ  ▁explicitƨ  ▁fitsȨ  堂叔到我家ʨ 之余̨ 子孙Ψ  即便是Ш  指的是Ҩ  ▁devotedԨ 得太֨  ▁shoulderب kedڨ  纪念馆ܨ  金牛座ި rors  的优势 thy  ▁Hend  ▁whereas same  ▁Brothers  ▁peaceful 新手 egu ས་ 听听 g `让人心痛的话欢迎各位朋友回家感谢你们的鼎力支持相信未来会携手前行  ▁Benef  ▁Altern 出差 外贸  borough ostic  安全性  党委委员 罕见 变的  ▁Corporate 好在  ambling  十三五 分割 ladesh 往事  不仅可以 赞美 中之 肇庆"  并通过$ 另有&  ▁publisher( itime* ду, fred.  最容易0  ▁deadly2 这边4 高空6 泰山8 三四:  综合体<  ournals> Pan@ igureB 字的D orientF 看房H  ▁MurphyJ  ▁bordersL 汲取N  合作伙伴P  ▁nobodyR 帽子T 转入V  有不少X  ▁lettingZ  如果你想\ ▁PL^  ▁Save` 加深b 志刚d ▁Liff  ▁convinh  ▁marginj uckerl  ▁Cassn  的各种p  ▁sorryr  ▁Issuet  呈现出v  ▁bookingx  ▁tollz  ▁elegant|  ▁sophisticated~  ▁racism 的说 vement 是为  登记表 herry 层的 pc 尊严  ▁fing 也曾  ▁Pope  应急管理  ▁Swiss 要说  ▁inclusive  ▁medal idad  不仅如此 过上 一举  ▁lockdown 过分  ▁differently 就得 多月  ▁Between  ▁institutional  success ▁TX  ▁Wonder driven  ▁Roger " 并同时向您表示歉意© visorĩ нияƩ 留给ȩ rrʩ  ▁cute̩  加入中国共产党Ω  ▁DouglasЩ 了对ҩ  合格的ԩ DNA֩  ▁heatingة 视线ک 将有ܩ 外表ީ  数据分析  otherapy  ▁dishes  突如其  ▁NCAA 万亩 带给 被迫  微生活  ▁ranking Gold 叔叔 olves  ▁laptop 当今 六级  ▁parad sv  ▁offerings 风味 奔赴 各市 者和 ▁Rot  ▁disorders  ▁Includ 上当 排列  ▁transparency 下次 起伏  也是一个  ▁scan ▁ф"  ▁proceedings$ Filed& 地段(  ▁Mail*  氨基酸, ▁tub. ;“0 颠覆2  ▁structural4 技艺6 卵巢8  brities: Offic<  thousand>  ▁Register@ 等候B  ▁qualifyD 强行F  优惠券H  ▁restorationJ  政治局L  ▁royalN  的交通P 藏在R ▁KrT 大叔V 向下X  副作用Z illo\  ▁Lang^  ▁sour` ▁Abub  一定要注意d  本次活动f 木耳h  ▁ambitiousj  ▁islandsl Hellon  workingp 跨省r CARt fundv  ▁concentrationx Novz Bro|  曾经的~  ▁overview  资源来自于网络  ▁achievements 法人  ▁handful ATO ▁lon  的价值 zie FOR  高层次  不存在  有任何 fold  ▁answered  ▁quarterback  ▁accum  肯定是 半岛 sch  临床试验  ▁Champions wer  ▁burning ▁Occ 不留 残酷  ▁consolid Hot  个人所得税 Quest 忍受  职业技能 SKª  交通大学Ī  的一生ƪ  ▁considerableȪ  我自己ʪ  我们需要̪  执法人员Ϊ 战友Ъ 东营Ҫ opterԪ 羽毛֪  从严治ت  先进的ڪ  论坛邓州ܪ 请问ު " 本文部分文字与图片  ▁Single 日夜 ▁um 骑行 各有 std 荷兰  不影响  ▁excitement  这几个  ▁advantages  ▁donated  ▁psychological 立方  交流群 要比 ANG  退休人员  ▁effectiveness 大庆 的上 绝望  ▁Voice 海底 启蒙  ▁manifest 抛弃 inely achers 促使 iox isp  共同努力"  常用的$  经典的& 返还(  ▁Getty* 下调,  ▁joins. 热心0 大片2  ▁poster4 ▁Ga6 镜子8  esperson: heet<  ▁situated>  ▁Nurs@ ztB 心肌D 评定F  也是一种H  伟大复兴J arestL  ▁laboratoryN 相助P 亲情R  其他人T ":"V 局限X 打扮Z 湖区\ 真人^  ▁anth`  ▁Technicalb ardingd 保管f 归纳h 新浪j  ▁companl  ▁navigaten 是人p  ▁Picturesr 7 0本文部分文字与图片资源来自于网络t 随手v itutesx  lingtonz ▁bub|  ▁Later~ 从前  ▁Evans 口袋 Visit 二代  ▁premier ▁VA  稳定性  对应的  ▁significance 剧烈 庭院 eger  ▁reun  剩下的 起的  队伍建设  平台的 谁能 预备 原谅 筷子 返岗  ▁Peters 问责 眼的  体育局 总能 却被  ▁Original  ▁custody riors  ▁lasting  ▁launches«  了许多ī 吃到ƫ ▁BOȫ  相关专业ʫ 表白̫  认为是Ϋ CreatЫ анҫ  enezuelԫ  ▁Perry֫  ▁Electionث  ▁deleteګ  ▁trucksܫ  ▁mirrorޫ  ▁advocacy  ▁fatal 工期  疫情防控措施  ▁FREE  的权利  ▁priorities  并不能  ▁loaded  ▁outs  ▁moist 文体 我把 制止  emption 电压  ▁Kitchen  ▁urged TP 版的  ▁constra 详解 isure 围观 imedia  ▁unprecedented upid  ▁violation ,, kh 制裁  ▁alumin House  ▁Making"  ▁solely$ IX& 敬畏(  ▁embrace* ▁IL,  队伍教育整顿.  ▁investor0 ▁bug2 明月4  ▁Bishop6  工作组8 柴油: 看不<  Develop>  ▁tune@ 匈奴B 笔画D 追溯F  主持召开H 榆林J 奖品L 余生N MalP  宣传教育R  的症状T atewayV туX 我对Z 是用\ Const^ 月起`  ▁Progressb ▁YOUd 教导f ▁Kuh  来到了j  ▁Madisonl  人员名单n  图文来源于网络p  ▁Visionr incialt  ▁humorv 赞同x 西游z EV| HC~  这些问题  Special  ▁summit  ▁calc 农田  ▁attempting  ▁physician  ▁corporations 启发 约翰 自来 位居  ▁stere  ▁со  才可以 ▁Neb undle 香蕉  重症病例  全面落实  ▁demanding  ▁pocket 患病 alling  朋友的  事业单位公开招聘  ▁influential  小康社会  疾病预防控制中心  候选人  ▁drove 笑话 isons 长远¬  ▁countiesĬ  ▁NintendoƬ  ▁generousȬ % 人力资源和社会保障局ʬ 抚养̬ plitά  ▁CrownЬ  ▁KnightҬ  InstanceԬ  ▁clip֬ gowج ▁BMWڬ ▁Rioܬ 盛宴ެ 修饰 Column 听了  ▁phrase ename held  ▁Luther 滚动  安排部署 日语 难受 Real orer 六大  ▁wash  ▁accommodate  ▁Trad 典礼 uez  ▁filling  ▁Rules 逐一  ▁matching 着的  elebrities  ▁indoor  ▁bacteria ixin 车子  中含有 gender  政法委 富豪 ensor"  ▁Wire$ 事事& 下手( 阴道* bad, 全域. 底薪0 拿起2 nia4  大面积6 侧面8  生产线:  标志着< update>  都已经@ 多位B  ▁HuntD  ▁SpecF 修正H 坐标J SNL 活血N  ▁incidentsP 图形R  ▁trainsT 术后V >(X  普通高校Z 半天\ 将是^  ▁chemicals`  ▁profoundb 光泽d  ▁Norwayf 实惠h 供养j  发言人l 助教n debp  创造了r 为之t 仪表v 强劲x 草案z asures| Exp~  ▁Agriculture  而且还 Ep  了解了 在该  等活动 名著 ▁Kir 准则  ▁posit 度和 不移 utt  身心健康 ▁dé 东明 外线  irlines  ▁Advisory  日活动 ”),  等因素  ▁Cris  天气预报  一男子 人情  旅居史  ▁identifying 掌控 rin jiang  ▁interpretation  ▁Costa  ▁buried­  ▁aggregĭ 流转ƭ 分级ȭ jobʭ 老虎̭ 腰椎έ renderЭ  ▁orangeҭ  太平洋ԭ  ▁Racing֭ jkح 保驾ڭ  ▁templeܭ  ▁Televisionޭ 年起 蘑菇 ▁Lag 这篇  会产生  ▁Production  ▁satisfied  ▁Partner  ▁scholars 阿尔 Rev 吃货 有利 ▁hal  出生于 umbai 想去 场上 CU  ▁pron 口气  ▁Nord  新技术 四肢 饱和  一季度  ▁Nashville 不法 ridge 首位  温暖的 ▁Bou  ▁lineup  ▁uncertainty" 场馆$  ▁surveillance&  ▁Pred( 灭活*  离开了, owners.  ▁ware0  线上线下2 很长4  ▁reply6 ▁Tob8  县人民医院:  ▁Masters<  ▁difficulties>  ▁stolen@ EAB 转身D  DescriptionF 明了H  摩羯座J  ▁collectingL 时有N ▁PraP aptersR 晚会T  收到了V  学党史X  ▁wisdomZ Invest\ arms^ 加点` 杨某b  ▁Leband  ▁expectsf  ▁undergroundh 科长j pearel  知识和n  别忘了p  ▁Commercialr  ▁Birminghamt 砥砺v 严查x  ▁dyingz  ▁animation| 致命~  ▁inher  承诺书  ▁Clear  ▁fulfill  ▁convicted 膀胱 argin 启用 出资  ▁cabinet 色调  ▁picking 的后  到最后  ▁countless  ▁Parks 如是  ▁foster 东区 古诗 ICK dat  为基础 ▁Pap ▁NJ  ▁establishing  ▁Value  ▁households  大礼包  ▁Collect piracy ▁Due  ▁ignore ▁Abd®  ▁RiskĮ  ▁VermƮ  DefaultȮ  正在接受ʮ  ▁terminal̮ 右手ή  ▁NeuroЮ 每当Ү  ▁variantԮ  ▁vintage֮ proofخ assesڮ 强迫ܮ 难关ޮ  又一次 旗帜 诵读  在工作  ▁Bright  严厉打击 回放 极具  行政执法  ▁earning 人行 四处  ▁couples 够了 chron 课外 Lead  年轻的  今天是  ▁tape  ▁Princess 苦难 tml  ▁vendors  ▁trigger family  材料的 хо  视频号  ▁fears 现已  control  科学的 中南"  ▁romance$  地理位置&  ▁undergraduate( ahl* 书院, Design.  ▁parallel0 前三2 help4 尽情6  个性化8  有效地: ammed<  ▁lease>  ▁Beng@  ▁wishesB ]:D  ▁targetingF  ▁InspH ▁DegJ ▁EDTL humanN  ▁IndigenousP 猜测R 钱包T  监督检查V  ▁RankX 托管Z  ▁Bapt\ BP^ ounge` 实物b 世卫d  ▁blastf udoh 背诵j 淋漓l  ▁recipientn 很强p ▁latr  责任人t ouncesv 出游x itudesz 碎片|  ▁vocals~ 百家 四种  ▁attendance  推出的  就是因为  ▁convenience  ▁curious cam scope  ▁removing  akespeare  错误的 normal  总工会  ▁Happy  第一天 Mike Contin 极限 gence  ▁trademark 核验  ▁Marie  ”…… 雷雨 最快  相关工作 供热 统治 ▁Fel 骨骼 gun 当选 dest¯ 新高į  ▁handedƯ 新规ȯ  石家庄市ʯ  ▁Rachel̯ 爆款ί  文章来源Я  ▁opponentsү 傍晚ԯ  的工资֯  保驾护航د 点缀گ  ▁computingܯ  ▁refundޯ ▁Pul  ▁Volume  实践中 Token 万家  ▁investigating  回来了  ▁Weekly 身处 太空  网络安全 未知  的有效 笑声 hal  ▁evident  ▁Lanka  ▁compassion pur  ▁spir  找工作 落的 信贷  ▁drew src  公安厅  ▁Beyond  ▁permitted  ▁token  ▁upset Though 景象 ▁Bos  ▁Czech" 贫穷$ IES& 两地(  ▁Philip* 双眼, >$. 二中0 NEW2  的都是4  谢谢你6 pathy8 致辞:  一步步< 远大> 签名@ 改编B  身边的人D 匠心F 残留H  ▁LinkedJ  ▁WarnerL 正值N 修炼P  前几天R ▁HemT anghaiV  ▁vegetablesX 一所Z  校外培训机构\ 微量^ Four` 万一b  ▁democraticd  正在进行f  隔离观察h atonj 进食l 分管n Layoutp  ▁timelyr 酒吧t  ▁editorialv 通俗x 艰辛z  ▁attacked| TM~ enson ifice brand  ▁breach  ▁determination 滚滚  ▁gardens  ▁Puerto 往来 Ok 学姐 omi пи immer 导图 Master 恰恰 会展  ▁neighbors 借口 自有 PRO  很高的  ▁babies  为孩子 当做 thlete  就是这样 老鼠 重温  ▁preview  ▁fiber  ▁biological Power°  ▁voluntİ  ▁promotedư  般若波罗Ȱ  ▁bannedʰ  手机号̰  ▁deputyΰ Spaceа  ▁coalitionҰ  错过了԰ 业的ְ 上新ذ izableڰ 伪造ܰ  ▁Fishް  我们应该  ▁ceiling  ▁duration  睡前伴你夜听 冷水 用量 骨折 同龄  ▁Ukrain  ▁ghost  ▁Storm 出动 ieg  妈妈的 回了 isch  区块链  ▁resign 乒乓  lications 不佳 毛巾 奥迪  ▁mysterious arer 半夜 缺失  ▁investigations 山村  profile  ▁airline 为他 电台 idges"  高新技术$ ▁gri&  ▁sisters( arser* 处方, 法则.  的颜色0 两手2 ente4  ▁converted6 风云8  劳动力: 生抽< ▁aqu>  小学语文@  也无法B 校友D ruptcyF ▁DigH 水肿J  拥有一L ▁GotN 信托P onzR 开封T 冲洗V CongX DanielZ 之大\  ▁Animal^  ▁nutrition` 打磨b  ▁Platformd  重点工作f  ▁roundsh  胡椒粉j 选购l 荔枝n  的治疗p  也是非常r  izontalt 或将v  ▁Cabinx  与中国z  ▁overnight| 泄露~ 被查 铭记 务实 duced  元左右 多方 Adv  加入了  清明节  ▁nomine city 上千  ▁Fern  building 眼科  ▁timing azy ▁rib 违章  ▁fluid  ▁Meanwhile  ▁Publishing 切换  ▁coastal  习近平总书记关于  ▁Gray 扰乱 住户 弯腰 先到 腿部  ▁dedication 跟进 黄冈±  可谓是ı ▁AGƱ 入伍ȱ 能有ʱ argeḏ  ▁suspensionα  ▁happinessб  ▁Clarұ 水质Ա ▁BRֱ  ▁forgottenر 虚弱ڱ  责任感ܱ Salޱ 江县 ▁Aw 婚后  于一体 分辨 itate 分期 NET  ▁vertical ▁adj 流浪 iev 鼓舞 nut  ▁Democrat 的我  ▁storyt 脊柱  卫生健康委员会 南区 ▁SW ▁os  白羊座  ▁Career 小哥  ▁laser  在今年 假设  ▁expecting  ▁composer 早点  涉及到  中级人民法院 Feat" 尔多$ doors&  是自己( zel*  ▁departure,  密切关注.  specific0  为人民2 至上4 ▁het6  ▁cookie8  ▁Criminal: ▁Rab< 商会>  ▁acceptance@  的父母B 诸葛D 百合F degH 赞赏J OnlineL  生态文明N 入睡P 复发R BuyT 分成V 拟任X  这是因为Z  ▁Compl\  新材料^ ▁om`  ▁sciencesb  也开始d  ▁friendshipf Billh */j udgel  并不代表n 汉中p  进行删除r  ▁Staget  在接受v ▁Uncx 半个z  Product|  ▁Wikipedia~  ▁clinic  ▁gest 争吵 match  ▁logic 洗发 裙子 甜的 ▁maj  ▁nest  要进一步  ▁pension  ▁labour  description  ▁Belgium  ▁scientist 专班 ▁mos bf umm  在家中  近几年  ▁colours 想必 万亿 用它  ▁inventory  ecurity 收受 oids cery  历史文化  ▁passage  受害者² ▁AlbIJ  ▁MeetƲ LondonȲ  ▁Architectureʲ 五花̲  你觉得β indaв  的投资Ҳ  ▁convincedԲ  ▁Arenaֲ  ▁tensionز  ▁authorizedڲ  Richardܲ 先天޲  的文化 先得 的他  ▁singles  陈代谢 的可 keley 左侧  张家口 峡谷  ▁Course pot  情况进行  Station 串门 见了 在中 尸体 适度 出演 erts aval  ▁beaches opes  ▁та  ▁starring  ▁Ground 一事  ensively ▁HP  变得更加  纷纷表示 至此 ibli" 上游$  ▁contractors&  为企业(  ▁signals*  成为一个,  ▁Forces.  ▁angle0  住房和城乡建设2  发现自己4  ▁sorts6  ▁Healthcare8  ▁detection: 概括<  ▁Chron>  以外的@ 陈皮B 生不D  ▁advocatesF  ▁MetalH  建设用地J  ▁alliesL  ▁BirdN  ▁PreventionP 观光R 核对T  疫情防控形势V  ▁IdeasX omerZ 把他\  和家人^  anthrop`  ▁failsb  消费者的d icif  一生的h  ▁depositj  ▁mobilityl 明亮n  ▁Dragonp  ▁terrorismr 电站t 要对v 的她x 提及z  ▁missions|  公司地址~ 最长  ▁Official 省钱 平日 普法 ▁een 伴随 }\ Plus  ▁draws  ▁longest 燃料 的书 骗取 入了 chen 网信 局势  ▁wheels 的无 apper 致使  ▁rated ea sor ▁ALL  还有一些 宏观  政务服务  ▁alumni sx 取出  ▁handled  什么呢³ 剂量ij 大佬Ƴ  紧急通知ȳ amanʳ 虽说̳ ▁Ladγ 吉星г  ▁powderҳ 一副Գ  ▁theoreֳ "#س  ▁carrierڳ 却又ܳ las޳  怎么说 刷新 ▁duo  必不可  收费标准 ás  ▁addiction  verning 口服 阵风  ▁shaped 街区  ▁poet Deb  零基础 depth sized 演艺  ▁Furthermore  ▁unemployment Blue Http  ▁desktop agine 菲律 ▁wur  ▁Delta 料酒 很是 ▁pub system  ▁Along Jim  ▁Griff" 吃过$ 地板&  专业课( 岁了* 在水, 老化.  ▁economics0  电子课本2  ▁enemies4 坚信6  这本书8 复查: MR< 人有>  ▁referring@ wuB 首要D  ▁booleanF 驾驭H  ▁transparentJ  ▁VirtualL 万多N 袭击P  ▁TrainR ▁TunT  ▁WantV  ▁medicX  菲律宾Z  ▁Letter\  ▁seventh^ expect` 万名b  ▁transitd 开阔f argoh ETCj ▁Gayl  ▁occasionsn 家常p  中小学校r azinest  ▁Reformv 差点x  Versionz 开辟|  统筹推进~  ▁Edinburgh  你需要 Conf  ▁stepped  ▁reflected ▁vel  ▁affecting  摄影师 omo 酸奶 写出  ▁Stop  ▁belt  ▁locked 你家 合唱 emor % 新型冠状病毒感染肺炎 海域 城的  Director  ▁compact 并向 颈部  ▁instances 孵化  Listener  只有一个  ▁cheaper  ▁violations 局地  但由于  ▁plaint 变了´  ▁identificationĴ  的人物ƴ  的一点ȴ  ▁marchʴ  ▁Sett̴  违法违规δ 取代д 定于Ҵ  各种各样的Դ  ▁discussingִ 桂园ش utoڴ  ▁ensuresܴ 定时޴  reshold  ▁Double  但我们 官兵  Application ▁SA  家长的 五脏  ▁Task  体现在  ▁angry  正是因为  ▁recipes  bersecurity  ▁Suite 品类 生于 close  看不见 vet 白发  陌生人  commerce  ▁stir 直线 anean 带队  保证金 meta PER archy 仲裁 在对  化合物" 阿富$ 一斤& Dist( 大豆*  ▁imagination, 自称.  ▁Fiction0 银川2  综合征4 CF6  好自己的8 高发:  灵异吧<  ▁expense>  ▁steady@ 剂次B  让更多人D  ▁JerryF  ▁killerH 看病J 亲朋L 既能N 概况P 小雨R  等措施T ftenV 热带X  ▁EmilyZ  ▁revenues\ 大腿^ 学史` 口令b bankd  ▁nervousf 卡片h  比较多j  ▁gratl igon ▁GOp alsor orust  本科及以上学历v  ▁librariesx 房租z  制定了|  摄像头~  ▁conducting  书法家 坦克  巨蟹座  ▁specialized  ▁Assess  家庭成员 舍得  ▁Arkansas 穿过  管理中心 赚到 ▁Ot  不代表本平台  快乐的 神话  ▁Conservation  不到位 又称 ▁ric  ▁celebrity INT around 征信 借钱 冒充  的想法  ▁Coord  的最佳  ▁Consumer 似的 IME  ▁ende  ▁switµ  ▁martĵ  icatingƵ  建立健全ȵ 设为ʵ 上角̵ 天上ε  ▁applicantsе ▁vibҵ  ▁heroesԵ ickedֵ  ▁Resourceص  optionsڵ  ▁Argentinaܵ 写道޵ gb 到家  ▁intimate 现行 高频  olitical  ▁animated  阿富汗  ▁Inside  ▁Orlando  ▁outlets 团员 龙湖  治愈出院病例 ^{  通过了 Ps Block 补课 驻村 我爱 位列  双鱼座 来回  ▁loose 并于 红薯 images eda  ▁Larry offer 市直  ▁medication 得更"  灵异吧灵异吧$ 均在& 标本(  展现了* 论证,  ▁Reports. 配件0  荣誉称号2  ▁Championships4 ▁Kay6  身边的朋友8  控制在:  ▁mouse< 黑白> ▁FO@ 体积B  射手座D 打扫F  ▁executedH  ▁PhotosJ  ▁warmingL aciesN  ▁magicalP  ▁complimR iperT  ▁StanleyV ниеX  ▁WithinZ  女孩子\ 连连^  ▁barrel` Packb  高科技d  举报电话f  发展改革h adminj ▁mal  ▁adequaten  ▁scarp ▁Tigr  okespersont bootv 肢体x ▁lipz 不掉| 后天~ 工信  ▁patch  带孩子 agger  ▁architectural ylum Elect 抢险  日来源  ▁simultaneously  近距离 保湿  ▁derived 不服 Mad 两委  ▁poem  欢迎大家 返程  ▁Historic rolled ▁als ierra 图解 带走  ▁earthqu txt %), 误解 文本 sun ▁] 报复  ▁analys¶  领导的Ķ  ▁Communicationƶ  人才网ȶ ITEʶ  人死亡̶  ▁Pictureζ  ▁snapж  副县长Ҷ 会更Զ  ▁noseֶ 对她ض 芜湖ڶ  ▁Saltܶ  的不同޶  理想信念  ▁Programme  ▁variables  ▁Articles  ▁physics  Resource 身后 高位 四月 一类 ▁ox 喇叭  等领域  ▁Lost 国人  ▁varied  还是比较  ▁shadow asury ▁Tar  ▁destinations  并进行 umble  昂贵的  网络上 occ  ▁ramp  特殊时期 的发 ateur rah  ▁amen 上百  和管理"  ▁depos$  ▁Airlines& Write( 抓捕*  ▁Rand, oT.  ▁Billy0  ▁researcher2 pling4 技师6  diction8  ▁designing: ahoo<  年高考>  ▁steam@  ▁processedB  吸引了D  公里的F ▁KapH  建筑物J ‍‍L ▁WaN  重点项目P  ▁beautifullyR  达不到T  人居环境V  不必要的X  ▁AudioZ  的好处\ 普惠^ 光辉`  这辈子b  珍贵的d 谈到f platesh  ▁Benjaminj aukeel 引来n 致电p étr  什么样t 房东v  ▁Crowx 便会z  ▁alien|  以下的~ 弯曲  ▁subsequently ▁ED  ▁shame 垄断  ▁daughters  ▁mechanisms  的艺术  双子座  ▁sensor  牛仔裤  县人民政府 收看  ▁accompany 地带 Family 补肾 次日 保守 欧阳  和中国 药材 西装 Init iliate 支行  可根据  新冠疫苗接种  ▁Opport 交谈 etta  ▁guided  ▁gonna 选取· ▁vigķ  ▁WalterƷ iovȷ  所在的ʷ Jon̷  职业教育η  ▁Michelleз  经济的ҷ ServerԷ  ▁farmingַ 陷阱ط 精讲ڷ  机构的ܷ ▁ist޷ even  ▁mutual  ▁Toyota 好文 放过 的呢 大肠  ▁Than 厨师 去过  ▁Greater  ▁Cameron 宫颈  ▁Samuel 所未 ;　(  共同体 灰色 aments ▁Kid 发给 合金 顺德  ▁stamp 本事 涂抹  核酸检测结果  房地产开发 outing  ▁Turner  ▁announces  ▁floors ORK arios"  ▁peers$  ▁Lands& 送去( 适龄* 跳舞, 小腿. 胃肠0 共赢2 他和4 bot6 rod8  到这里:  ▁Growth< 写了> 衡量@  护肤品B  MicrosoftD 接纳F 军的H BYJ 罢了L olineN ▁NHLP  ▁WriterR  ▁graduatesT 导读V 内裤X  ▁TargetZ 得益\ 流域^  ▁toxic` 执照b ▁POd 一味f  ▁deemedh  Categoryj 博野l 肥料n  的最大p 出道r ahot  ▁Valentv  ▁Surgx  ▁Teslaz 总数|  ▁Guid~ 相思  ▁Literature  是国家  ▁dependent  发布公告  ▁manages  ▁analysts  ▁unlock  ▁chairs ▁Doc Thread 途经 ▁Sad 洋葱  ▁occasionally  ▁evolved  阿里巴巴 亲近 判定  生产企业 umar  ▁audit  closure acer 超声  ▁specialists  ▁centres Null 赣州  工作方案 惊讶  ▁labels  ▁attractions George¸ Frankĸ  越来越好Ƹ 加剧ȸ  ▁constitutionalʸ  ▁bron̸ ugarθ  ▁attractedи  就好了Ҹ  不应该Ը 大图ָ midظ  媒体报道ڸ  ▁pathsܸ 修身޸  ▁Visual 话费 Still  ▁Competition  般若波罗蜜多 之事 麻辣  ▁Edge ▁Cre  ilateral 称赞 Disc 黄芪 Mary 工序 ▁Ric  ▁productive reck chair  ▁ster со  会造成 ▁sne  ▁shapes  创新创业  充足的  少数民族 wl 一朵 让它 前途  ▁valued resa  最适合" 书的$ 性强&  ▁respected(  ▁caps*  ▁greet, Store. icient0 ▁Alt2  ▁Spect4 RNA6  救护车8  心脑血管: 传感<  国家标准>  留下的@ 知情B  ▁BethD 硬核F  依法追究H 时时J ectiveL  有一位N  习近平总书记在P  ▁travellingR 东海T 收官V  展示了X  ▁chipZ  ▁Fantasy\ ▁ESP^ ▁Ay`  ▁disappointedb  ▁buildsd  这些年f ELLh Cardj  教育系统l  的小事n  ▁kneep  ▁realisticr  ▁Ghanat  意识和v  ▁columnsx 但她z  ▁Join| zu~  ▁trauma song dist 传输 ducer rays "/>  的教学  ▁Retrieved  领导人 ristol gly 蓬勃 见识 income [ T▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 领会  ▁thrilled  因为它 常州 款的  教育考试  ▁thirty sky  中公教育  ▁revers 食宿  ▁settle  ▁Sanders 要从  ▁camps Meet  太多了  碧桂园¹  ▁oxygenĹ  ▁purchasesƹ 封锁ȹ ▁Litʹ  ▁Montreal̹ níι  突如其来的й Melҹ  OptionsԹ 的山ֹ  如果我们ع  经济技术ڹ  ▁Holidayܹ 时起޹  ▁Barry  岗位上 妊娠 ▁GT 脚下  ▁Pent  ▁defeated 五月 应有  ▁Archive  ▁smoking 支配  ▁adaptation 芒果  ▁cats 优选 oeing 八大  一定程度上 升初  ▁gate  ▁intim sq  分钟左右  ▁leaf 樱桃 eva 报案  的房子  严格遵守 着手  Written ching 中风"  property$  因为在& 清朝( 接连* 的气, ▁Tan. ▁Eli0  ▁teaches2  ▁cryptocur4  抵抗力6 底气8  ▁deployed:  得到的<  ▁coat>  ▁reson@ 华图B sendD annesF  ▁StraH  ▁carpetJ  ffickingL 就没N  应及时P 螨虫R  乳腺癌T  ▁PortugalV 人品X  研究发现Z  并不会\ 口水^ 一如`  的努力b 拥护d 延缓f explh  ▁Cruzj 歌声l  installn  ▁Keithp  其实就是r  吃起来t  年中国v iterrx  ▁worldsz ▁Nam| 浇水~  ▁ethical SF King 阐述 石林 rong  ▁cinem 有何 路的  面对面  ▁Joint rica Change  ▁precious 极了 ▁› 相差 jar  ▁span  每年的  销售的 ну 汇编 源自  可能性 左手  ▁strikes  ▁Total nik  ▁roster  ▁masks  关心的 ▁dub јеº  ▁acknowledgedĺ  特殊情况ƺ ▁ERȺ ▁Punʺ  ▁struggled̺ kersκ  ▁visaк 招呼Һ  玩手机Ժ Urlֺ  ▁Masonغ 看清ں Rockܺ ▁Neg޺  零距离  的心理 建国 Bay 猫咪  接种新冠疫苗  ▁Budget 靠着 rup  并发症  ▁FROM 原有 权的 探究  ▁Liberty  ▁transformed Japan  ▁regarded  老人的  也可能 最爱  公安分局 基数 asa 这块  ▁reaches 荷花 管家  ▁vaccination  为他们  ▁arranged 统战 ivia  ▁spray" 不禁$  ▁separated& 比喻( 重重* bc, 思政.  ▁Nothing0  出入口2  ▁Response4  的密切接触者6 五四8  作出了:  ▁Faith<  ▁Customer> 加州@ 头疼B 图纸D 　"F  ▁каH eralsJ  ▁collapseL 车祸N ','P  ▁chemistryR  ▁prosperT 都区V 相约X asiZ 中所\  ▁tablet^  ▁Bitcoin` 建面b  ▁villagesd 西红f 两周h  攻坚战j  万人次l 型号n  strationp  免费送r ▁Veht 新发v  葡萄酒x  ▁nickz 联通| 的海~ 降临  ▁Import  ▁Currently  要强化  转型升级 MAN 但它  ▁Mercedes  今天就  附属医院  ▁nomination  投入到 tz  请点击 orph 累积 科员  ▁Resort 惩戒 春暖  ▁ridic  ▁isolated  联防联控机制 上天 被打  ▁instantly  ▁Says  ▁Walt 罗马  ▁murd  ▁traveled  ▁exped  ▁subtle 这片» 门窗Ļ  ▁Phoneƻ GodȻ  低风险地区ʻ ▁ME̻ 本田λ  ▁Putinл  ▁sailһ  满意度Ի  ▁intelligentֻ 侵删ػ quanڻ  ▁Buffaloܻ  ▁Primary޻ 法的  非常重要 商圈 已是 世人 威海 别说 ▁cig 写在 豪宅 天府  ▁Remember 规章  思维导图  当年的 Mo 千米 所学 addy  什么都 有毒  ▁Honor  物联网 开头 减弱  人类的  sterdam  年上半年 装配  同样是 集群  ▁charts  ▁drum  ▁complexity" )/$ 多为& 昨晚(  氧化碳* itual, 诊所. jd0 Inc2 法制4  男女不限6  ▁noting8  ▁remembered: rien<  主要包括>  天秤座@ 歧视B  ▁DiscoD 看点F  ▁FocusH 吃得J  ▁arrangementL  ▁BangladeshN  事业发展P ischenR karT  ▁MercV  ▁coincX waukeeZ  ▁offset\ 宜昌^ 衣裙` 挂号b  为核心的d  这三个f Levelh  ▁twinj  ▁phenomenonl 迁移n 馒头p llar  ▁fundraisingt 浓浓v 声称x wegz 驰援|  财产安全~ 也想  ▁Johnny eur  ▁Neil  ▁pays  ▁Benn 特大  ▁Margaret ▁tut oustic  ▁Trend  ▁ро ▁auf  ▁okay 走廊  的经济 约会 ▁Maj  ▁maker 去找 相识  ▁bonds  对身体 们在 剖析  ▁sustained asgow  人员和 华侨 ahn 与之  浏览器  ▁raises  最喜欢¼  ▁prospectsļ % 新冠肺炎疫情防控工作Ƽ Milȼ  ▁Salvʼ 接口̼ 新风μ urersм  实习生Ҽ  ▁accredԼ  ▁Lockּ ▁libؼ 车厢ڼ  ▁memorialܼ  ▁Stay޼ 北斗 新村 王国 移送  的重大 精油  ▁hoped  经济开发区 ▁Isa 你了 ▁Ll  Further  ▁Tickets 黄花 沉迷  ▁Chile  的感情 知己 过高  ▁Hampshire  Science Educ 送达  ▁elderly  一定能 学长  ▁beef 乾隆 Friday  进行的 山河  是这样的  并处罚  代表着"  第二步$ 抓手& 计量( 红楼*  购房者,  ▁Alice. 一二0 会把2 相传4 ​​6 触摸8 坎坷: olve<  ▁stom> 党课@ 咽喉B  ▁tragedyD  ▁presentingF  ▁anybodyH  ▁celebratesJ 异味L 安宁N  这样才能P borneR 商量T focusV  工程项目X 鸡肉Z 心底\ 玉林^  ▁Ring`  ▁picksb Estd  党建工作f  非机动车h  ▁angerj  ▁volumesl 哪种n  前列腺p  ▁bundr  ▁landedt 自强v  ▁chipsx fullz 中职|  ▁Weather~ 集结  ▁stain  ▁Robin 光滑  ▁balanced  ▁NULL  ▁trim uds  的疫情  ▁branches ▁cir 了呢 齐心 档次 再说 文档 烟火  ▁crim 天涯 聊聊  security  ▁forests URE 受骗 istle 川县 炒作 camp  ▁Economy  原创者所有 cies font  ▁Statistics  ▁cater½  ▁guidesĽ Winƽ  典型案例Ƚ Taggedʽ 非凡̽ 艾滋ν 周四н  ▁caringҽ  ▁strokeԽ 表态ֽ pointsؽ  ▁newspapersڽ 美学ܽ  资格条件޽ assing 人被 ▁fer  ▁talents  ▁bust  Speaking  不建议 Sil  ▁helic erness feed  ▁Lion  ▁radiation fm  ▁secrets output 夜里  是这样 泉水  坚定不移  ▁Happ  ▁precise 量化  ▁Comedy 征兵  ▁Left  要学会  ▁при 福音 保利  需要注意  要按照  ▁Todd ▁Kaz" 五官$ 坐着& 影院(  发现的*  ▁fate,  ▁tear. 移交0 lia2 平原4  的学校6 痘痘8  ▁propag:  ▁backup< 素的> 高薪@  ▁ReutersB 国籍D ▁arcF 胰岛H  ▁disciplineJ 一扶L  ▁WoodsN  张家界P ExcelR  ▁preventingT  点一下V  ▁VenezuelX 翅膀Z psy\ '),^ 想念` 欢笑b Filterd  ▁Coolf 我能h  倒计时j  ▁silentl :「n snp 红利r 宝石t  会计师v 自制x 腌制z  需求的| 慈悲~  ▁Diamond  ▁infected 最难  ▁execute  ▁supplied 年在 ▁SQL 扎堆  ▁tooth thal ifact  ▁teens  ▁compatible riger  ▁silence  防控疫情  依法依规 &　  管理制度 ushes  ▁beds  ▁scream  ▁convention ovic  ▁Cover 搜集 cht ▁Too  ▁discipl  ▁Halloween  ▁Movement  社保卡  我现在  就不要¾ 辛勤ľ  ▁combinesƾ  ▁propriȾ 不传ʾ  ▁Tyler̾  ▁digitξ 于此о 底下Ҿ 回避Ծ  ▁Laboratory־ 就此ؾ 麻醉ھ 西侧ܾ 乳房޾ 或有 orted ouds 战术  ▁Sheriff  ▁inspection  是最好的  ▁Ferr ▁Ald  全面从严治  录取通知书  ▁throwing IZ  功能的 不绝 探测  之类的 沐浴  居住地  ▁financ 花香 \" 征稿 脚踏 而行  ▁Hudson 书香  成熟的  干部职工 Style Core ennial 买买 Ah" 百货$ obi& 有三( 领悟* think,  ▁supplier. ▁Cu0 来电2  ▁Colon4  ▁ballot6 可疑8 ▁pa: DD< 诊治> 白银@ OrigB 红糖D  ▁cropF  靠自己H ▁LocJ 金黄L 楼下N 获批P ometryR  ▁AustriaT marginV 要够X  ▁PerfectZ 懂的\ 本金^  在规定` ▁Kelb 中海d 麒麟f 较少h  ▁recognisedj  值得注意的是l  举办的n  ▁Intelp ebookr 宝藏t 做完v  阿弥陀x emaz 寻常| ▁Pic~ ▁Vo  十九届五中全会 IAL 招牌  常务委员会  全面建成小康社会  三支一扶  ▁Drama 引用 直升  ▁challenged 收治 容积  和学生  教育资源 ▁Tol  国际机场  变速箱  ▁Easter 疏散 腰部 ▁HE 群内  ▁für 天猫  ▁approached Andrew 属鼠  ▁motivated 浑身 泡脚 醉酒  都不知道 不顺¿  不同于Ŀ ▁Ravƿ AGEȿ  ▁Shakespeareʿ  主要的̿  ▁fossilο  ▁opponentп  ▁Playersҿ datesԿ  美国人ֿ  掌握了ؿ  了一次ڿ incinnܿ 黏膜޿ gets  ▁texts 盛世 лу  itations 全都 终生 olver 五百  所以我们 区和  我没有 JO 狂欢 几千  ▁worn  以至于  inition 烟的 ▁das  ▁Pain  ▁trails  ▁harassment  大自然 系数 上诉 有两  道路交通 的重 自查 燃放  ▁Palm 排骨 ▁hug" 黄某$ 缩小& 不远( 刚需*  ▁apolog, obb. 质的0 拼车2  ▁Theory4  ▁fame6 台上8 ▁Fra:  ▁Issues< 向东>  natural@  妇幼保健B 这才D 评委F  ▁automatedH  官方网站J  ▁LatestL  出一个N 游乐P  writtenR 百搭T  漂亮的V reviewX  ▁counterpartZ  单位和\ 粪便^  ▁blocked` Merb 暂行d 圆形f  有意者h 期中j  她们的l  请不要n  一带一路p  ▁renovr 失误t gradv ▁nanx  ▁endlessz  ▁Qual| Among~ 张贴  ▁nationally  ▁placing  ▁appreciation 还需  企业在 入侵  ▁Horn 晚期  ▁mold 早早 ▁Fan  ▁relate  ▁Managing  ▁suspected  ▁stack ahu 那年 限定 提分  ▁predicted 后者  说自己  ▁bulk  ▁Rivers  了下来 shine 专案  百分之  original 穿衣  ▁grows eto  吸引力  ▁restored 太重  ▁viral 庄严 杀死 权限  ▁reflection 厚度  ▁closest  ▁experiments riel ▁PH 全集  我为群众办实事  就应该  ▁Eagles shi  ▁contacted  短时间内 ▁jet ▁„ 很小 ylan 抵制  ▁humanitarian 周三 ▁och IONS ▁Tu  ▁dated 国王 精确 NT 快讯  最低气温 路以  不清楚 我到  的生产 而知 filter  ▁Films  ▁Guardian  ▁Sure Bas  享受到 这只 listed" Entity$  影响了& 香菇(  ▁Graduate*  免费领,  慢慢地. quare0 永不2 病原4 bery6 iza8 不光: Module< 更强> 智库@  TwitterB  ▁tentD  的事物F 配上H 宽度J  工资待遇L kitN ScottP  要切实R  婴幼儿T  基层党组织V  ▁sentencedX  ▁whisZ 抗菌\  ▁testimony^ 跨过` ▁Debb  ▁nursed  ▁Ambassf Camph  联系删除j ([l  ▁Jessican  明确的p  ▁portraitr 又在t  ▁Detailsv sgx  incinnatiz 军委|  ▁cricket~  小升初 调料  ▁Given  Students  有意思  ▁iter  就是为了 本草  ▁Style  ▁Ghost 督察  原件及  ▁drops ▁Aid 模特  的理念  ▁speeds 泡水 挂钩  ▁Wildlife  学区房  ▁constitu olved Sund 停运 用好  身份证号  attered  ▁habits  ▁brick 顺风 脱落  社会主义现代化  istence double ishi  管理员  ▁Adventure 鲜活 " 庆祝中国共产党成立  ▁Respons Order  居住证  ▁alternatives 弄虚 失控  的情绪 厅长  ▁legally 南充 店里  舍不得  ▁occupied 处以 IFA  结合起来 死角 uba 家家  ▁Graph Sche  环境下 承接 ▁Fro 通往  ▁Rule  教育部门  ▁supportive  ▁Zone 分裂 matic  该怎么  ▁gradually  ▁winds  ▁Nancy 的名  ▁Stanford  帮助孩子  ▁conferences 在地  ▁Tips  ▁pursuing" erick$  ▁Cities& ipt( ▁Nat* 老大, 趁着.  Journal0  拥有的2 READ4  ▁cous6 无情8  记忆力: Being< ▁Var>  深入推进@ 街上B  ▁lyingD  ▁rushF  ▁cargoH  ▁StatementJ ipseL  ▁festivalsN 台区P  ▁refresR 正气T ellarV 信赖X 臀部Z 分明\ fu^  ▁Commonwealth` 星空b 质地d  就不能f 已于h 题库j 佳节l 挣扎n 而这p  ▁reformsr 水面t ▁Аv 欺负x  ▁granz  弄虚作|  ▁imposed~ 体外  仅供读者  ▁Watson  ▁density  ▁Quick  ▁tears 师父 VI  网络诈骗 助学 马力  ▁influences  ▁rolled  谈恋爱 介石 report Prot girl 潮湿  iterranean 今生 一世 动能  沈阳市 ▁GM  ▁Cash  ▁fier  ▁acknowledge 闭环 诚实  张文宏  在学习  突出的 onde 较强 但不 国资 平市  ▁immun  earchers  ▁impacted 就想 理智  到底是  人员聚集  ▁Juan 预定 续航 警力 清淡  新生儿 印花 曹操  ▁vessel  比较大 长辈 总的  不要太  市委副书记  Information itters  ▁shirt  具体情况  月加入中国共产党  研究表明  ▁concerts 配色  遇到的  防疫物资  ▁Garc  测量体温  ▁confused  ▁gaining 药师  生活圈  孩子们的  一一一一 追逐 uki  小龙虾 司马 Fort"  ▁drain$ ELECT&  ▁crossing(  ▁unfortunately* uming,  意义的.  比较好0 各族2  文字要够4  ▁torn6  ▁yours8  目前在:  ▁Choice< 如图> 难怪@  ▁undersB  年龄段D  绝对是F 人之H 高山J SarL 桌子N  ▁CasinoP ▁namR loveT 各校V  ▁SoldX  ▁dignZ ancers\ ▁Fif^  ▁Complete` 送往b 再用d  ▁Sophf  消毒剂h 河市j 热议l  蒋介石n 秒钟p castler  ▁recoveredt 长相v  ▁Ethiopx  ▁mandatoryz  目标任务|  ▁museums~ 元气  ▁delayed  ▁proportion  的人民 天才  招生简章  的路上 COM  ▁threw Rest  ▁contractor  ▁Gard 剧中 无愧 自贸  会议强调  ▁creator keep 序列 ▁bom 孙子 落下 ATA Israel  ▁Enjoy  ▁legitimate  感受到了  打造的 迹象 灌灌 致癌  Children  ▁integral  个工作日 现役  的家长 ▁Rou avior  ▁Carm 并与 跑到 ▁Hyd  法律顾问  ▁assured Jeff  医用口罩 核算  ▁Materials  佛山市 锅里  的质量  ▁privilege 糯米 要去 oshi 手动 那你 erie  ▁Swedish 利好 报纸 龙泉 ' iongmaoqiongmaoqiongmaoqiongmaoq 和对 icht ▁py  ▁Investig  ▁spokesperson 三期  中秋节 低的  才能够 OME  第三次  ▁Baseball 城南 named 舍利  在生活中  新鲜的 向外 数百"  Control$ 女王& ▁Eye( Young* 海峡,  ▁treating. VIE0 失踪2 分区4  日正式6 自负8  ▁unclear:  ▁repairs<  ▁batteries> 中奖@  政治站位B  ▁ignoredD 领袖F  ▁copperH ISHJ TotalL  ▁HarrisonN CloudP ExperR  二十四T etonV  ▁UkrainianX  ▁endorseZ  ▁moderate\ 开班^ 编码`  ▁Oliverb  ▁contactsd ▁VRf 现就h  请关注j  ▁dansl  ▁Dynamn 纳米p  的通告r  服务机构t  ▁perspectivesv 高管x 分流z  ▁mandate| nm~ 可乐  ▁Poly  ▁rocks 压抑 aneous  container 年份  的经验  和身体健康 into align  ▁Quarter 调试 IGN /)  ▁classified  检察机关  ▁Southeast 而非  ▁revel ▁Dra  十年代  会议精神 最小  行政拘留  ▁thoroughly  ▁highway  ▁releasing  ▁obligation  一起去 破裂  ▁restricted 均已 深厚 ▁vi  ▁Fried 心事 jian 一千  微量元素 周二 松懈  ▁Counsel advant 全景 草木  ▁pedest  ····  ▁Shah  ▁theft gered super 买入  ▁Switch 守住  ▁reserves  你是一个  ▁transf  ▁backing  公开课 auc mented ▁aus  ▁remarks  ▁tire 宁德 alph  环境的  ▁enjoyable  ▁screw  ▁yoga 无形 舌尖 近代 眼界  我都会 汉字 间接 mons  ▁analyze 增生 卡机" 雅思$ fy& 产值(  ▁shore* 纸巾, tag.  ▁cotton0  ▁Contemporary2  ▁graduation4  ▁vessels6  ▁Features8 保姆: ▁Ton<  参与者> 增进@ 里斯B  教育培训D  ProviderF  ▁defendantH locJ 征途L  ▁comoN 红星P 的对R  学生会T  ▁blockchainV 她就X 冷却Z  西红柿\  ▁fires^ 海淀` inylb ottad  连衣裙f  道路上h  ▁photograpj 戴着l  最贵的n 东侧p 人对r  ▁scandalt  融合发展v  ▁incorporatedx  ▁Denmarkz  ▁advances| 喷嚏~  ▁notion opic nex  我们将在  ▁preferences 找出  ▁harmful  对话框  ▁quantity  Process  ▁acclaimed 回升  ▁boats 唤醒 隐形  的父亲  ▁wages  突出问题  ▁Seven 水县  ▁girlfriend 光照 本着 YY 速看  在我国  ▁beating 麻将 tim  内容为  ▁Symphony ti  ▁rect ixon Po  停车位  ————  ▁brew gd 统统  ▁Mumbai 文创 Sports  你就会 干的 在那  大幅度发大幅度发 every 被盗 UG  ▁annoy 专区 zhong  ▁sporting ooting  中华民族伟大复兴 ittees  ▁commented  ▁distinctive  ▁dominant  博士研究生 脱发 vor frast  不再是 gx 峰会 超标  ▁await  ▁opera  滑动查看更多  各乡镇 柔和  ▁superb  Chinese 头顶  ▁attributes 苏轼 ede 肉质 包围  保护好" ifi$ 无证& 哺乳( 怀念* 反诈,  详细的. 活在0  省教育厅2 uart4  不会有6  ▁efficiently8  死亡率:  经济技术开发区< 上路> 二审@ ikTokB  ▁TourismD HolF 矩阵H 车库J 网页L  ▁ultraN  ▁exteriorP 违背R 瀑布T customV knX 公认Z  会发生\  共产党的^ 与会`  ▁reactionsb  ▁devastatingd 而至f ▁Nagh ▁Hatj 私自l icularn  ▁обp 强奸r ulentt 支架v  ▁whereverx 大有z 该校| 协力~ 大衣 小家  ▁railway video  出色的 太难  往往是 上限 ovie  膝关节 并举 ▁Len 警官  ▁accidents 放射  ▁chains zw  转载请注明  犯罪的  ▁Vari fd Mus  ▁conflicts  ▁reign 中路  ▁inflam  ▁athletic  ▁spare 没收  全面提升 不听 起码  ▁Into logo  可下载  ▁locals 博主 冰糖  ▁meth ▁ho  工作和  ▁partial  就算是 脑袋  ullivan  ▁dramatically 淮安  也跟着 严密  ▁wise Ident  ▁Mental 借贷  ▁Armen 和睦 焊接  社区卫生 安保 无量  比赛中 学到  红十字 南站  系列活动 湿度 缓缓  的出现  ▁Kirk 山上 移植  ▁survivors  ▁suggesting 加盖 申论  的一位 tx ▁Arn igion 平县  ▁commentary  ategorized  教育行政部门"  ▁encourages$ 公号& zk( 喷雾*  外交部,  ▁Academic. 胶原0 正宗2  ▁Gardens4  ▁Address6  ▁profiles8 were: 错的<  好看的>  ▁perception@  ▁RonaldB  ▁amendD 白宫F  开启了H  我们有J  ▁throL 市县N 放学P digR  ▁overlookT 免税V  ▁TreatmentX osphZ 万辆\  有一点^  岗位的` ummyb  年终奖d hesisf  ▁appearingh  ▁mothersj 赤峰l qin  换来的p Validr 雷达t 心仪v efinedx  ▁Lookingz hor| hs~ ▁nu  ▁grey 三元 无偿 压迫 构筑  ▁Flash  ▁texture forced 一米 RAN  发病率 土耳  世界卫生  ▁freel  ▁assumed  ▁obligations 阜阳  ▁Match  ▁login 少先  ▁Former ▁cab  ▁pupp  ▁twelve  ▁Tell  ▁scratch 闲置 投产  抗氧化 望着 宜宾 alink  ▁dominated  慢性病  特别是在 往后 oots 国旗 我去  太极拳  ▁Several  我希望  ▁rational  ▁sequel oons  ▁toys  mediate 嫌弃  通过对 七一 导游  ▁Homes 恋人 编写 播音  ▁weakness 台账  认定为  两个维护  ▁recommendation  rapeutic 沙滩 调配 FE 过节  白色的  ▁Indeed area  million 编织 Joe  全面的 opedia  ▁mock 拉斯 ulator 见效  country 冻结 ▁Pin 绿水" 绿豆$  Professor&  ▁Speaker( 全校*  资格考试, 十个. ▁LP0 纯净2  几十年4  ▁hence6 GET8  ▁twist: ewise< 知足>  时间段@  ▁acceptingB  ▁protestersD 四周F 连云H  ▁OthersJ 新款L  面临的N blackP  ▁ProsR iriT  ▁automotiveV  失业保险X ischeZ 遇上\  ▁deployment^ atical`  在未来b  arteredd mercf  ▁steeph  土耳其j 链条l 的面n umbersp  ificentr  ▁psychologyt  ▁Planetv CBx 花样z  ▁Karen|  ependence~  和相关 眼球  ▁hydro  带领下  如果你的 沃尔  ▁stopping 生了  ▁syndrome 驾照  ▁Montana  字文字要够 查明 万能 你所  ▁preval 减速 ▁GA  ▁chamber  ▁complement 往届  大学学历  运营商  通知如下  ▁marketplace  ▁controversy    圣经 马来 Java Term  ▁closure  精神的  ▁confidential  ▁Clay  ▁adventures  zheimer  ▁Hillary  ▁parameter 之意 amine 会场 刷牙 粗糙 挂在  再也不 三明 路过 切记  流动性 通关 下肢 麻痹 水位 紊乱 па 合资  医疗服务  ▁Excellence 之乡  慢慢的  人口普查 Spring  ▁Indians 研学  宝贵的 频频 效力  实实在  ▁farms Host 康养 找个  ▁recordings  ▁governing  办公室主任  ▁Universe 平平 抵扣  污染防治 知乎 ▁Н" 不退$ 能为& 阴天( 涌现* JP, 不给. ]]0  ▁jewelry2  ▁Jeremy4  ▁acceptable6 ▁Ori8 ▁>>:  ▁Veterans< 别再> ów@ 鲁迅B izarD 溶液F  ▁instrumentalH  ▁connectivityJ 好书L  businessN 可口P 高价R retionT 共鸣V 同年X 称呼Z 赛道\ 画画^ 步步` 止痛b  ▁harshd ▁}}f  ▁Emmah  ▁buzzj 彩票l ▁Chin SGp 并无r 黄山t  再一次v Keepx  ▁sleepingz opath|  ▁Centers~  ▁battles  ▁reminder 来之  太原市 ashi 在她 前任  的不是  比亚迪  ▁Thor 捞出  直播课 icker  ▁safer  世卫组织 属鸡 贫血  抑郁症 暴发  一切都  交通违法行为 打响  ▁insect 仔猪  财经大学 Parent  ▁motivation 姆斯 eston  ▁charter Stand  报名时  据统计  建议大家  党总支 eded  刑事拘留 病症 sql 展望  第十三  ▁Songs  ▁bicy Fire 站立 Lou 此处  ▁corresponding  学前教育 之气 ipher  ▁Cyber iph  农业农村部 自卑 的原 astics  ▁chef  ▁exports 计入 阴影 eu 杀手 ▁Б 景色 ▁fel  ▁discusses  ▁manufactured 电工 　—  正在进一步  ▁bene thodox  ▁physicians 念头  ▁commissioned  阶段的 etal ;③  ▁inflamm  学生在 受过"  ▁Advert$ anti&  ▁Terry(  想知道*  Digital, 输送.  ▁trash0  监督管理2  确定的4  新一代6 arians8  进一步提升: 两厅< exec>  ▁HERE@ 恶化B  ▁BasketballD  ▁SurvF  ▁HigherH  过程中的J  只是一个L 当我N  规划和P  仅供读者参考R  ▁witnessedT fiveV  积极响应X  ▁prolZ  ▁pharmaceutical\  ▁Shar^  ▁continent`  年的时间b 买菜d  突发事件f  ▁parishh 回味j  完全可以l  ▁flawn  使用了p  济南市r  ▁dynamicst  ▁introducingv  ▁farex 屁股z  ▁magazines| 大意~ essa  建成后  ▁worthy  ▁WordPress 正文  ▁Face  ▁qualities  ▁betting 说得  农业大学 自救 门外  任职要求  指导下 prem  ▁resid 中招  ▁Baptist 改装  将根据  坚决打赢 Format  ▁submission  ▁bail  ▁Fisher  ▁Present haw ▁Fer hips  社交距离 反射 惊人 ère ▁Rac Organ  主持会议 毛坯  养殖户 列表  生产生活  ▁monument  ▁recruitment му boys 多发 ▁AB  因为我 ▁mg 河源 禄劝 ▁Bab 理赔 HPV  ▁Born  ▁scroll  ▁balls 清零  还是有 ▁Ka KJ NBC  在疫情防控 诱导 丽水  而对于  ▁Richmond  要不要  中医药大学 ▁caf Associ  做不到 kV ▁awa 俗称 放手  ▁migrants oppers  ▁exercises 过期  总建筑面积 eme  信息来源" 保鲜$ azed&  ▁lovers( API* ▁Ub,  ▁Sudan. 间断0 克力2 显现4  ▁Alberta6 尚在8 artz: 航线< 一门> 醉驾@  越来越大B 会给D $,F ▁sliH  ▁agingJ  ▁burstL  和国家N  ▁integrateP  ▁amplR 餐馆T  母亲的V 驿站X  ▁theatZ  ▁regards\ igu^  复旦大学` oubtb  解释权d auerf ▁[]h  大湾区j 生鲜l ideln  ▁remindedp 震荡r 助你t torv ▁Mesx Foodz PK| 有种~  ▁Wine 以防 降至  不论是  区域的 上映  ▁accessed  ▁earliest 潜伏  干什么  并讲话 asha  ▁tweet  ▁priest know  ▁portal ▁nin  ▁precisely  ▁Mack 对标 洋房 风的 ▁fur  国际化 黄帝 要看 三点  ▁irre  ▁Cuba abol  amilies MV  ▁Workers  ▁buttons 兴化 owner  ▁ambass Label 金色  ▁generic 与本  ▁deserves unts 剧场 出锅  ▁Qatar  ▁Vincent 治国  到医院  ▁utilize  ▁measurement 霸气 iban 假冒 疑难 ▁Py  ▁bits  ▁jurisdiction 希腊  ▁Announces ▁NEW 武术 结核 萎缩  喜欢吃  ▁nurses 紧迫 ▁Ja  存在的问题 ussels  ▁liver dog  ▁sponsor  ▁receiver 出击  ▁Lanc cue 交替  做一些  ▁tropical 加之 河区"  ▁welcoming$  ▁Sent&  的公司(  ▁underway*  ▁Berkeley, 不符. 试剂0 卖出2  ▁honour4  ▁Easy6 备战8 osi:  ▁Gran< 无非>  ▁broker@ thingsB  这是一种D  投入使用F 衰竭H  ▁TruthJ 美德L  的儿子N omonP  有多大R  ▁conceT FiveV  ▁strainX 本月Z  矿物质\  还是在^ 掀起`  的经历b ▁UAEd 不求f  ▁Standardsh 相爱j ▁Injl  ▁Questionsn 战线p 提质r garyt  ▁fractv  深入学习贯彻x 雕塑z 品德| 余年~ 五星  ▁Camb Less  ▁adapted 硬币  政协委员  information 帅气  ▁praise cribe Hey  也会有 溶解 就去 iciary 简化 和不  的大小 水产 竞技 哈佛 ▁Р  ▁ethics  形式的  新学期 三百 cott 死刑 焦作  实用的 制成 紧盯  ▁Around  就会有  防输入  ▁Jefferson  少不了 政权 捕捉 清远 单车 心上 rise  Saturday  ▁inquiry  ▁Kyle  ▁MORE 刚才 Born 发售 相见 idity  ▁LGBTQ ▁Lex  ▁painful 蚌埠  ▁essays ▁Fly  ▁indigenous  ▁wars ▁gut 喜事 ptic  ▁barg Disco  ▁Physics 点开 bj raska  ▁della 安居  在路上  ▁pose 隔壁  ▁Lives  ▁replacing  ▁Coming 矿山 Mor  小时的  ▁persu 河县"  ▁Biology$ 增效&  ▁unveiled(  ▁dismissed* 水域, 而生. 美文0  ▁Panel2  ▁retreat4  ▁bout6 采光8  前所未:  ▁urgent< 浓缩> 西区@ 昼夜B OutputD  随时随地F 个字H 锦绣J  工商户L 和新N ThomasP oderR  舒适的T 是被V  这么说X  半个月Z 锈钢\ 九月^  ▁assignment`  ▁confusionb 海量d  ethelessf  尽全力h  ▁projectedj ▁Ryl  阿弥陀佛n  请告知p 征文r  iciencyt 枯燥v 露天x Shouldz  人社部| 联播~ за 传销 极易 遵纪  的人来说  做好了 肇事  ▁incorporate ;& ialog 定律  公益性 宝鸡  ▁Potter  ▁beneath 强力  quiries  ▁Mediterranean appro 饿了 教体  ▁apartments  考试的  ▁magnetic 打着 rices Water 的白  交通管制 女排 rose  ▁advisory 之星  ▁recycling  对你的  ▁tang  的帮助  ▁Half  经常会 比特 Publ  ▁oven  ▁quoted  春暖花开  ▁smell ▁Wor GP ▁Ug  春天的 见习 insula  frastructure 地狱  长沙市  喜欢你 显赫 ovo  ▁Oakland Save  上世纪 BBC 山楂  充分的  ▁Glasgow 中断  施工现场  ▁Similar 抗癌 溯源 Week 一两  science 潇洒 版型  中风险地区  ▁outer  ▁backgrounds styles  的爱情 腹痛 超高  的身份"  ▁differ$  对人体& abul( 月任*  ▁electro,  扫码关注.  防疫工作0 齐鲁2  给你们4 ▁bot6  更重要8 本能: 组团< 三七> Na@  宝宝的B 选题D  的对象F 当局H planJ 逃离L  ▁manuscriptN 干脆P joinR  ▁dearT  ▁reproduV  ▁slideX  经验的Z  ▁pending\  并及时^  ▁explaining` 会诊b beyd 和田f  初心使命h ▁cubj  ▁Ridgel  ▁shortern 束缚p  零部件r 补水t fabuv  ▁undermx 就可z 化疗|  就是你~ asia 三门  会议室 交互  ▁Brain ▁Si 燃油 ▁tel 脑海  ▁dash  Washington  来不及 asted 饮品 曾国  ▁advisor  ▁distract uum  ▁Elementary 组的 院区  ▁Sustainable DJ 让您  ▁disappear 吸毒  参考答案 este  哈尔滨市 奔波  ▁Bristol  不稳定 ☞　  有关要求 扎根  ▁binding 英寸 ▁IM  ▁Amsterdam 量大 格尔  生活习惯  ▁dimensions  抗生素  不利于  ▁Chemical 播种 cost  ▁seniors  ▁intensity  不属于 ▁Jah  原件及复印件 uter 前台 一辈  还是会  ▁rows 礼盒 ▁MI  ▁surfaces 地级  ▁strive  ▁flows 矿业 hester 拉伸 农商  遵纪守法 га  ▁Arctic  为国家 CAD  来进行  ancellor member ovid  的背后  ▁Claim  开工建设  线上教学  减少了"  ▁shareholders$ 预留&  ▁Large( 茄子* 上身, 在与.  人民网0 ▁SM2 rying4  ▁enterprises6 聚力8  ▁educate: Li< 上年>  ▁canvas@  学习了B  ▁portsD 秦皇F  获得的H  一个男人J  ▁busesL 交界N 一脸P  ▁FreshR  ▁staysT  ▁ColumbusV  更多的是X 主线Z  ▁juice\ 港市^  企业复工复产`  ▁crusherb 没法d mysf  ▁theoriesh  获得感j 回首l  旅行社n  ▁comicsp 风暴r 通畅t 杀害v  ▁Strix  ▁honorsz 香的| 属兔~ 水晶 iasm 光线 轮回  观点和  ▁stupid 相较 remove Sum  监护人 Di Row ▁Sha OSS  ▁enact four  丰富多彩 学府 果树 bone boards 同类 nity 催化 喜剧  自己和 Canada  ▁Yellow  ▁arrives  我们能 　”  等一系列  和技术 门的  ▁habitat  ▁quit ometer ’” 劝导 ▁Pad anship  ▁Swift  便民信息  ▁hydrogen 放开 脑子  ▁stressed  ▁biology usters  ervatives  ▁presentations 未满  ▁jumped  国有企业  reational 万象  讲述了 祝你 解惑  ▁paragraph  ▁wealthy 戴上  ▁Circle 跟党  ▁tens 这事 icting  ▁placement 按压 新版  更多资讯 猪场 许昌 批示 阻挡 izard  ▁pulling 集资 自尊  ▁heaven keeper  万余元"  适量的$ 地里& 除夕(  ▁rivers* ▁sq,  设置了.  ▁modify0 ▁mes2 院内4 湛江6  ▁embarr8  ▁leverage:  facebook< Tex>  就能够@  ▁LegendB  effectiveD 地去F  ▁cheerH 易经J  ▁PsychologyL  ▁FinlandN  了一系列P 义乌R  ▁organisedT  知道了V  ▁beingsX  ▁catalogZ 鼻炎\ isha^ mel`  ▁Flightb 在内d 接力f  Activityh COLj 耳机l  ▁tagsn 优待p 滋生r  ▁Drawt 双脚v  ▁Routex 管辖z  成长的| Team~  ▁horm  ▁innings  ▁Maur 跟他  个体工商户  ▁rubber  ▁poison  ▁attribute 抽烟  小孩子 瓶颈  ▁NATO  ▁Edwards  积极开展  业务的  了一套 Talk 净土  ▁Batman 启示  ▁demonstrates  对他们  ▁boasts 军区 南沙 etown  ▁Rodrig  准确性 etters fun asting  ▁archive 那边 实处 大病  一般是  ▁Facts 善待 ▁Vel 摆在  ▁donate  ▁metres 工匠 网约 stat /㎡, 汇算  ▁examined  ▁override  ▁wrapped  关键时刻 大部 ▁Rug  内容来源  ▁assisted  及周边 事先 煮熟  的任务  ▁imaging  ▁securities 花海 Has  ▁Musical  ▁Beauty 北区  ▁Clinic 入院 usive 机票 engu  科主任  ▁Soon 一贯 uisine  ▁ward  这几天  ▁electronics  ▁treasure  ▁Rather 春夏 进阶" 地标$ ▁Ni&  ▁translated(  ▁wrap*  关于印发,  ▁automation. 你将0 ▁SUV2 绑定4 anga6 黄石8  government:  赢得了<  ▁revised> 滋补@ 装箱B  防护措施D  人与人F 德尔H  ▁dressedJ  ountainL neysN 甘草P  ▁MedicareR  的季节T  ▁sinkV  ▁escalX  ▁travelsZ  ▁Anniversary\ 送你^ 双腿`  他表示b 着眼d 中式f  发送至h 在小j NOTl 启事n 京津p  ▁sheetsr  ▁overdt 喜好v 上半x urredz  不只是|  ▁Projects~  就容易 ▁lap  framework  ▁logistics  ▁Rogers  ▁optional  节目中  ▁assemb 提名  推荐的 试着  ▁Restaurant ▁Ul ▁gi  ▁streams  ▁Securities 录入  可以根据  ▁Harvey 官微  ▁blues brew  ▁crossed  ▁Coalition  内容及 Jun 炫耀 ▁Mob avel  ▁tragic 听从 银保  ▁Associates  ▁Cohen  ▁mineral  ▁strings 钱币  其中包括 脸色  ▁Nathan  大连市  ▁embedded  有许多  ▁strip 售楼  ▁precision 或在 vec  ▁continuously 拿来 消极 员的 ▁NPR  落实好  战略合作 FFFF 这笔 ▁NAS 雕刻 的黑 IPO 王朝  可以从 量子 演示 替换 百强 unted ONT  工作经历  ▁Mans 往前 低碳 眼袋  宣传活动  灭活疫苗  ▁trace ▁dos storm 后代  向上的  ▁neighborhoods" spec$  了多少& 花朵(  世界上最*  对外开放,  的大型. 基准0 之所2  ▁ESPN4  法律规定6 问他8 税率: auth< 昂首>  ▁Pink@  ▁buyerB 安卓D urstF  ▁footprintH 班车J leanL bottomN 光芒P ORYR 辆车T  ▁makeupV  填报志愿X LatestZ UND\  派出所民警^ 珍藏`  ▁Kashb 神山d 县区f  theningh ▁CSj ARKl  但他们n 强基p  ▁tenderr ▁GRt  设备的v 李白x annersz Ray|  ▁pled~  ▁innocent 痛点  ▁calculated  ▁Feed 其在  ▁creatures Credit  ▁Grey  ▁inherent  不满意  先到先得 全心  ▁Rico  第一款 lab isen  ▁resolved  ▁favorites  ▁cluster  ▁Releases  如下图 饭菜  ▁Minor sett  面临着  ▁Lions  ▁lever 切断  任何一个  级别的  ▁spreading 李小  ▁wurde  ▁prints  拼多多  理想和 Office gom  高级中学 预订  ▁formerly Tor 收割 个个  ▁pets  ▁prisoners  彼此的  ▁Jama  ▁surprisingly  ▁congreg ▁Vik env ULT  ▁disclosure  ▁headlines  未成年 清净 年了 陈列 牵引  今天小编 色泽 公诉  ▁protective 盯着  获得成功 antry 管网 比重 下行  港澳台 cur  了一批  ▁corporation  ▁coin  举行的 amas  ▁Barack  的增长 ▁Aer  patience  ▁cognitive" 芙蓉$  ▁Fest& ▁е( 子弹* ockets, 酸甜. 访谈0  ▁Lawyers2  ▁fruits4  和其它6  ▁Tampa8 存量:  ▁Mario<  ▁hint> 西门@  ▁SweetB  ▁cleverD  ▁constitutionF 超大H  范围内的J  ▁ImperL antonN  templateP HERR  ▁perceivedT 集训V 写下X  ▁PhotographyZ 唐代\  ▁vine^ files`  居民身份证b AWd 冰冰f  存在着h Neverj  三轮车l  stationn 特区p  进出口r  ▁joket 她和v ###x  ▁trillionz 骨质| Hub~  为主要 MAX ochem  合肥市  ▁polls Their 明朝  领导和  instein  传感器 既要 oba 保质  ▁archae 礼貌  olesale  ▁modest 留守 East  ▁Shows  ▁quantum wei  年春季  ▁Thanksgiving Op  ▁Analytics adows 上前 学问  真实性负责 Task  ▁orientation  ▁prelim  ▁performers  ▁donors 4 -这样您就可以继续免费收到文章了  ▁из 住址 业余 了你 atial 日到 了大  税务机关  ▁aviation  保持着  ▁recept 撤离 Fun  ▁Border  有人在  主动向 降压 ▁Eld  　&　 破浪 观摩 三五 两家 传授  发生一起  ▁Init sha 监局  ▁Upon Engine  第一书记  是通过  跟大家  风景区  ▁processor  也是这样 chant 归属 Happy  ▁defending 何处 踊跃 解剖  回到了 ▁nit  亲朋好友" ski$  中西医& 一则(  在行动* 乘风, 精通.  heastern0  ▁chapters2 ▁Bud4 ielder6 令营8  ▁advise:  ▁formats< 果园>  ▁delays@ 氧气B 几分D incesF  ▁inhibH  ▁FloreJ  ▁infringL igueN 二的P  ▁RetailR  地方政府T 忙着V 肤色X 起火Z  中心的\ са^ 上进` 底蕴b  招聘条件d 省道f  居委会h  ▁conspiracyj  在自己的l 鉴别n 转自p 喂养r 美满t 小花v 升职x  ▁attractionz  ▁Horse|  ▁reliability~  高分到低  一个女人  ▁Employment  而来的 造就  的传统 征服 十里 ]);  ▁triple  ▁sensors 墙面  ▁inhabit ▁WE  ▁diver ▁Hab  ▁bullet 酸痛  ▁elabor 赶赴 boro  ▁reduces 诚意  有能力 ▁Kle  ▁chaos opard Record  精彩的  你以为 tage tic  该怎么办  ▁economies  就业创业  ▁sensit 终极  ▁generating  ▁Therapy " 生命安全和身体健康 机动  了出来 属牛  乒乓球  ▁amenities 再也  和我们  ▁pizza 物价 begin ograf  ▁presum  clusions 时限 相声 ▁piv GT 枣庄 店的  ▁poems celand ushi 勿扰  ▁dece  ▁arriving Human 惬意  ▁cord  ▁Jamie  require  ▁Mathem 暂不  ▁Kill  ▁partly  ▁notification 马桶  相对于  ▁commemor 太子 后备 权属  ▁Milwaukee" ioned$ 环绕& phis(  ▁Coffee* 段子, 又会.  ▁diamond0  ▁invasion2 three4  ▁Category6 双语8 这让: 取暖< 悠悠>  ▁Shaw@ 叮嘱B CVD SteveF 勤劳H hallJ  ▁eighthL  ▁ChampionN  第二届P 公筷R  这一生T pushV  ▁relativesX 想的Z  发展改革委\  东南亚^  独立的` 羽绒b  ▁temptd  疫情形势f 突变h ▁BLj  ▁Milanl 陈述n 封顶p  ▁colorfulr ERTt  ▁observev 简易x Mattz  ▁prescription|  ▁symbols~ 巧妙  结合实际  ▁caution  往往会  细胞的 因地  不良反应 密接 endif  加入我们 渐进  可靠性  ▁Hero 施策 细细  应有的 东站 理发 engine  ▁Queensland 不畏 学好 报表  ▁portable 本届 food 原地  积极的 sz 分支  ▁Delaware  Program  ▁Bass 腐蚀  ▁Entreprene  接受的  ▁Cincinnati ▁§  ▁consensus CLA 属虎 必看  ▁interviewed  ▁zones  ▁spell ▁Wes  ▁algorithm 起义  公司在 quet  conduct 产物 一万  youtube 下周  绿色通道 台州 isse  ▁Artists  或者其他 ши  ▁nutri 益生  ▁innovations ANS  ifiable  ▁observations 内有 电缆  ▁unions ▁wit 一枚  ▁Egyptian him usc 面条 还真  有自己的 年是  America  ▁commod 千克" 幽门$  我可以&  纪念币( ▁э* once, ▁asc.  体系建设0  ▁ladies2 inars4  ▁isolation6  ▁playoff8 血清: htm< ▁Kil> 沭阳@ 寄语B  委员会委员D  ▁scalF  ▁competitionsH  ▁MiniJ 插画L DMN  UnfortunatelyP 检出R  降低了T 病区V walX 属相Z  的一大\  ▁priorit^ 母婴` 将为b birdd  产品质量f  市场主体h elongj  我已经l 变现n 畜牧p 与您r 孤单t 肝癌v 发掘x 三餐z 有心| 密闭~ 主干  都比较  学习资料  ▁specialty  ▁Gift  截止时间 urring 队友 细分 周恩 时请 ▁Fu 电线  研究人员 送至 师德 效能 货源 …”  ▁pipe  第一时间删除  指导意见 买家 Mobile 剪辑  ▁Immigration ▁SEO 上手 病理 python Smith  ▁Motors  ▁pleasant CK 备课  ▁investigators 推理 ▁zu 遍布 Json 听过  作者简介  ▁McDonald arten  族自治县 影子 chin  ▁dirt 清代 您好 ulin  ▁mature  报名人数  肯定会 老兵 堡垒  ▁kingdom ▁Nom  ▁Mechan  ▁grades 的总  将持续  ▁grain 习题 Short ergy ▁pig  困难的 want  心灵的 用到 创伤  的那种 Mill 产妇 立交  ▁Beautiful 曾任 INS uler  和信息化 ulates" Park$  专业化&  ▁Holland(  ▁toilet* 慎重, 应付.  ▁describing0  客户的2 ▁tob4 再看6 遮挡8 几率:  ▁sovere< ▁Arc>  文明实践@  appropriB 拥挤D SemF andedH  ▁FloorJ  ▁colonialL  的动作N 实训P baiR 分批T 其间V 初始X 校内Z  南宁市\ 答卷^ 中期` 洽谈b 陆军d  族自治区f  世界卫生组织h 不一j yclel irablen  关节炎p 孝顺r 中型t  ▁inclv  economicx  在没有z  ▁Trek|  ▁Phase~  ▁Concert fox  单位和个人  ▁fitting  统计局 idi  ▁Phillips  相关人员  的朋友们  美术学院 未婚 食盐 older 购置  管控措施 洞察  Subscribe  要做到 精益 ▁Ple  月底前 Monday Works 搬运  ▁crops  实实在在 bably  一个大 agne 针织 ▁pel  的衣服  ▁diagram  ▁Monitor 施肥  password  的未来  日期间 终结 连夜 prom Word cloud ▁Kan Yet  ▁assass  就是不 ▁IBM  ▁schemes een 迷人 ▁ч 发音  ▁mounted  人力资源社会保障  通行证  ▁racist  ▁Honda  甚至还  ▁considers  ▁affair 世的 狭窄  ▁Prep 折腾  确定为  尽量减少  ▁rust  ▁ritual ipper 中建  ▁safegu  ▁measurements  ▁adjacent  双一流  加微信  代表团  ▁aesthetic  他自己  ▁amendment  ▁confirmation 干活"  ▁grocery$ 班长&  ▁dumpster( rea*  核酸检测阴性证明,  ▁enrolled. 磷酸0  ▁kicked2 保研4  ▁arena6 卖家8 导入: 公证< 求是> 无常@  locationB  环境中D 病菌F 油耗H ▁BotJ  的运动L 可选N 援鄂P 专心R 拖欠T 济宁V  ▁ConditionsX  ▁BrigZ  ▁identical\  ▁во^  ▁Strong` 制约b  ▁punchd 展会f ▁Mih  ▁Activej ICUl  ▁linearn 四方p  ▁Glenr Dailyt 桥镇v ▁  ▁essence@ 餐具B 重启D 增设F  进行全面H 出众J  第三个L tokenN  的身影P  一分钟R  是一款T  事业上V  ▁existedX 地摊Z  ▁contam\  ▁rape^  说什么` closedb  的平台d leenf  ▁Riceh  ▁fingersj 都想l ▁WARn 短缺p ▁Mozr 名利t ctuaryv ліx byez  疫苗的| 较小~  prising  爱国主义  人员密集  你就是  ▁Particip  ▁journals ucle  ▁origins  ▁mentor  ▁retro  ▁Tool  ▁Brazilian  ▁Danny  ▁charming 殴打  需提供 双人 iko  副会长  开始的 姓氏  ▁educated  ▁shield  了我们 edit 辨别  ▁threshold 进场 phe 写着 Cost strap  ▁unfair  还有人 anyon 祥和  住建局 失调 icons  ▁celebrities  ▁Harbor  ▁shortage  配套设施 vine  ▁kiss  ▁wellness online unit 报废 色系 消炎  多元化  疫情防控的  星期六  很有可能  ▁personalized  ▁Victorian 北侧 抵御 引资 Mag 包包 不散  情况和  境外输入病例 Luc  前些年 ▁IR  美丽乡村 间距  ▁Reed  ▁header 大美 Fund 人也  师范学院  ▁rings airo 奇葩  ▁Bros  ▁athlete 得意"  ▁Nigerian$  ▁conviction&  万亿元(  那一刻*  主题教育,  ervatory. 的神0 田间2  claimed4  ▁feeding6  支持和8  的母亲:  ▁Plans< Edit>  的一句话@ 白了B HelpD halfF  ▁pursuitH ambersJ 舒缓L 石榴N  ▁coinsP CollR ocoT  ▁SnapV 实拍X  ▁sendsZ  驾驶机动车\  ▁fighter^  免费的`  ▁earsb 公考d 夕阳f 牙龈h 全科j  为契机l  ▁biasn  奠定了p 狠抓r ▁tatt atinumv 再现x  简直是z  ▁desperate|  ▁Unlike~ 不合  有关的 Frame 聊城  ▁edges 为我  的一款 users  ▁accountability  右上角  ▁respondents 六十 中了 嫉妒 ulus  巧克力 睡后  说不定 冷漠  ▁Northwest  ▁performs  ▁Dame diff icts  禄劝禄劝  ▁Draft  ▁cement Tw Film  ▁associations Bra 点多  ▁dive  ▁barrier  给出了  ▁Catherine  ▁Wells  ▁cooling 听见 睡着  ▁evac  管理条例 xim  进行核酸检测 ▁Rud 安庆 qual 天鹅 薄荷 acker  交易所  ▁Ottawa 城里  企业所得税 itcher ▁dys 制宜  ▁Safe 噪音  ▁resist  Connection ▁Ye Equal 野外  ▁editors  ▁brush 品格 收录  ▁Lower 即时 NAS  年出生  的成功 向你 打扰  出具的 邂逅  学士学位  ▁chron 阳明 ATCH  ▁revis" ▁Haz$  ▁Hear&  ▁Bernard( wegian* 攻关,  ▁spotlight. 完后0  ▁renewed2  员工的4  ▁sacrifice6  ▁youngest8  ▁Wang:  ▁demol< 看懂>  ▁WITH@  ▁preliminaryB  集体经济D  ▁suitsF 刷屏H " 和新疆生产建设兵团J  ▁MineL →　N  ▁outlookP 蕴含R bageT  个项目V 缺口X  后台留言通知我们Z  在他的\  ▁greenhouse^  服务区`  ▁Popularb ▁lund formedf asseh  中心城区j  ▁longtimel ( !请立即后台留言通知我们n rhp 表述r  的关注t  定点医疗机构v OHx 乐于z 明知|  给人一种~  ▁Getting  行业协会 杂物 势头  ▁detected  ▁separation  fulness  ▁Late 该案 YS  ▁Perm  ▁penalties 润滑  这两种 global ' iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii  ▁thesis  无论怎么  ▁lean  ▁Nepal 屋顶  以为是  ▁touched 阔步  切实做好 ▁UC  没有一个  将通过  句话说 直言  ▁fool 再多 炎热 否定 化名  解除医学观察 沧桑 北美  购物中心 rowse 可行 粽子 调侃  ▁Simple  ▁distinguished  ▁Lakes Ill 之称 ▁:: 祈福  ▁bases Amazon  视频中 plot mile  我们都知道 历代  ▁blogs  Document 和她 手表  ▁Curt  ▁Install  是多少  相关政策  ▁Vermont  的魅力  也是很  ▁acute  我们已经 放进  ▁subsidiary  ▁ought endant 机能 擦拭 感应 寄托 本院 包吃  每个人都有 顺应" kan$ 偿还& 骚扰(  ▁decides* 大妈, 又被. 回馈0 xc2 男士4 顿时6  的老人8  身体状况: 看望<  全国人民>  ▁Polish@  ▁ComicsB  ▁vinylD ▁UIF  ▁exploredH  工作安排J 对流L  ▁coordinatorN  一键添加P  部部长R  小学数学T  ▁ChaseV 某一X 极度Z 离家\  ▁batch^ 华盛` 要将b  ▁Ranged 悠久f 焚烧h 气势j  ▁enthusiasml  尽量不要n  为什么会p  所有人都r  ▁compiledt 上火v  ▁semesterx ingoz kl| 宽敞~  运动会 救命  红色基因  ▁outreach  ▁slee  ▁refugee  ▁castle  不作为  ▁Ambassador  ▁Lect column 去掉 ancies 地道  不代表本平台立场  主城区 mu  ▁subscribe 仓储  的主题 抽查 个股 照亮  就成了 phant  ▁Lebanon 　@ 吃亏  获得者 久之 ymph 必有 ▁WA igg  的人数  ▁expend  不是很  高到低 七月 六年 lit 防守  曾国藩 动词  适合自己的  ▁plates 只见 母乳 None  ▁Jake 供需 Prom  千万人 万众  不锈钢  ▁Info  分析师  ▁stomach  父亲的 gomery Want 着重 );　 不畅 ▁fed 疏松 贴士  资格审核  不仅有  disciplinary 瘙痒  Earlier 绘制 富人  前不久  来源丨 初一 好物 IDE 油菜 panic 襄阳"  ▁champions$  ▁organize& 即刻(  ▁frozen*  ▁riders, 而过. 贵族0 总队2  ▁Crisis4  等服务6 夺取8  ▁patri:  ▁Inspect< 开窗> 松弛@  ▁proteinsB  ▁belongsD ▁DimF 心意H  ▁repliedJ 鸡汤L  ▁postpN  ▁curveP 首例R  ▁peoplesT 常德V  有序推进X gitZ ▁Lov\ ▁ANY^  　,　`  技术和b  ▁Peted  ▁Carlosf  ▁warrantyh  非物质j  其他地区l  ▁pressingn 公章p 文昌r 二者t  至关重要v acceptx  ▁pricedz  元以上|  会觉得~  ▁immers 各行 payers tn 差的 战机 儿的 诗意 同情 星光 cond ção 脱颖  博士后 design  Thursday  ▁Lucas  的方式进行 ▁bay  ▁fond  ▁proceeds  ▁bankruptcy  招聘计划  ▁cave 分化 漫长 ▁GP 帮我 zp  组委会 加装  ▁bottles xing 买车 扩招 融创  ▁unnecessary  ▁Cathedral rives  紫外线 粘贴  ▁Fellowship Sus 积蓄 分值 enti  ▁Batt 思议  rencies  ▁Representatives  社会组织  管理者  ▁stylish obj 被子  全面推进  ▁hospitality  ▁memor 外婆  ▁scrut  清晰的  需要在 apsed 机组  category orf risk  language 票据  ▁tactics  ▁gambling  基本医疗保险 绿灯 曾说  本期编辑  ▁deciding 劝阻  第一线 免去  ▁verify arium Mic" social$ ▁Ko&  城市建设(  connected*  正处于,  atories. 柳州0 标配2 片段4  ▁polar6  先进个人8 EMA:  外防输入< 约谈> 高潮@  ▁ArrayB  ▁tenureD  ▁DefenceF  ▁brutalH 答辩J 各科L  ▁BrooksN  ▁musclesP  ▁grandmotherR MassT takeV 提炼X  ▁tradedZ 票价\ 福田^  往期精选`  一本书b ectsd 心存f  类似的h brookj 全城l  ▁Fitzn 庭审p  建筑工程r  这件事情t Ilv  的时刻x  研讨会z 偏偏| techn~  ▁responding  客服微信 内脏 Leave 村落  的品牌 ▁Thr  ▁glory 我来 Jose 祛湿 　· 兑现 历时 解脱  ▁judicial 滑雪 ▁toy  的建议 过量  ▁appealing ▁vir ▁DR ischer ▁Syn  ▁recalled  始终坚持 spot 洁净  非常高 PART  ▁Mend float 之城 看重 Hard  ▁threatening 模具  generation  建面约  作品内容 暂缓  财政局  职业道德 造谣  ▁tube  电影院 ▁ACT  的个人  高考成绩  complete  疫情防控工作的 博会  投资人  ▁Gonz  用实际行动 $( ▁Hum 热文  ▁Linda 时应  California  Sometimes 倒是  ▁Medal  也越来越 书房  ▁stones 收回 Tunes  ▁understands 肠癌 丢了  ▁Brandon  ablished 贩卖 经办  科学历  ▁startups  综合成绩 折叠 ISS"  ▁touring$ 位数& 十堰(  妇幼保健院*  ▁charm,  neapolis.  ▁dispar0  ▁Generation2 草药4  就像是6 三楼8 中说:  Collection<  日起施行>  同比下降@  要保持B  学历要求D  ▁pointingF  各学校H 在哪J ▁DjL uskN  ▁aluminumP  多功能R cciT  ▁HockeyV LengthX  ▁invitationZ  ▁occupation\ 出栏^ 如遇`  东明圈b ▁Dold ociof  新陈代谢h  增强了j  ▁biodl  做的事n  职业学校p  ▁hiker )]t  ▁knifev  些什么x  ▁Baldz 各村|  versation~ 伸出  又一个 容忍  ▁Nebraska 滥用  必须在  ▁voter 阿胶 oug 太湖 气味 fill ___ arus  ▁compromise  ▁mamm  请务必  ▁Nicholas Opt  ▁encomp 血脉 轻型  ▁qualifying 双休  angular ifle anmar  代表的 空腹 同仁 peror  ▁blessed 税费 抗议  可以将  这都是  ▁refriger  ▁freely SCI ▁î  红楼梦 ▁cer  普通高等学校  活动期间 干咳 bao 巴基 专访 连线  ▁leak REF  ▁determining 古今 omore  ▁stating 百日  ometric  ▁storytelling 任性  第十四 橄榄 . '新型冠状病毒感染的肺炎疫情  如果不是  添加剂  ▁Thunder  的理解 三峡 ▁''  ▁frames 纠缠 脱单 九江 登场 umbled 身穿  那个时候 掌声  完整版 制动  ▁bones 何必 巴菲"  ▁surveys$  ▁travelers& 止咳( 镇长*  不停地, 比你. 一出0  ▁thrive2 Target4  ▁develops6 选址8  ▁Bott:  ▁nort< 能把>  ▁finale@  ▁SpiderB  这几年D  ▁civilianF  ▁DougH 修剪J 坚果L 可视N  ▁scenariosP 提案R  ▁IncreT 刑侦V  全面从严治党X repeZ 在手\  ▁wore^  弄虚作假` Henb  好口罩d  ▁Sierraf 之王h 错题j  最后一个l 不由n  ▁publishersp 的全r 埃及t 紫色v  Interestx 点和z  ▁intens|  iblings~ 测算  幽门螺  ▁vaccinated ONG  第十二  图片均 两只 精灵 高地 高档 oro  在这场  ▁Blake  ▁forming 也更 ▁ink  县委常委 天长  ▁Treasury ▁ré 层楼  ▁Hughes  提到的  ▁decreased Books 发车  ▁Degree 瑞士  博士生 ocking  ▁resilience 确立 hours  ▁breathing (){  线城市 uh 准入 HK  ▁evolving  后面的  ▁Arsenal  添加微信  招聘单位  ▁badly  ▁elementary Little  教育集团 契税  统一的  ▁Size activ  ▁demo  ▁Dennis 南侧  创新发展  ▁reck ▁mi chem  平均工资  在现场 了好 ols  ▁subt  为目的 hend 样式 pher  ▁floating (\  这些地方 rene 一说  你有没有  ▁installing  只是一 两不  或者说 ▁IC  等各种 随心  身份证号码" 吴某$ >　& 内侧(  ▁playoffs*  准备好了,  ▁visibility. 抄袭0  ▁dealt2 Sever4  重要作用6 以待8 转弯:  应届生< 温水> 周口@ 烦躁B  ▁premisesD  一般在F 烤肉H  ▁sharJ 汇集L  ▁partiallyN vtP latR  ▁anonymousT  ▁BringV ▁BBX  ▁rewardsZ 滨州\  今后的^ 出货` outerb  ▁utterd  ▁droughtf  ▁missileh  ▁Citizensj  ▁pupilsl  ▁intensiven Syp 带娃r  项目部t  发现问题v  ▁curex ▁ogz 两边| 展厅~  ▁costly 打败  ▁benchmark  ▁toss Early  都能够  ▁translate 尺度 与众 外资  ▁spotted  ulatory vd bat 七夕 ERR  ▁durable  ▁dess 免征 hanie  ▁imprison  ▁congr cert 英勇  的区别  ▁fabulous 他又 绿茶  马来西亚 ▁q  ▁cycling ▁Kon  ▁Metropolitan 最少  ▁incorrect  ▁embark oping  ного  硕士学位 衣柜 两级 德州  ▁sculpture  ▁tobacco  ▁conjunction Prior  集装箱  ▁outfit 制剂 短时  ▁Trip  ▁combining  Environment  民营企业  ▁compliment 外形  ▁tweeted 通勤  ▁wings 养育 取证 educ  等原因 板上  将迎来  思想和  ▁seasonal  但对于 MENT  ▁Saints 无尽  乘坐公共交通工具  ▁Aviation 订购 笑了 ▁Et  接触史  让自己的  ▁urge  ▁NEWS 退费 Tur"  initely$ 战国&  ▁Banks( 想吃* 迟迟, VM. 精髓0 三观2 补齐4  道路交通安全法6  社会治理8 ById:  据介绍< 长途> sson@  ▁bearsB  认真的D  公共卫生事件F 名录H  ▁witnessesJ  查看详情L ▁AbsN  各级各类P  招商引资R  ▁sellsT  ▁reboundsV  说实话X  ▁ShanghaiZ  会议上\  永远不会^ Bank` NYSEb  comfortabled 换乘f 查收h  本次招聘j  ▁correll  ▁Claren 霉素p 鱼的r Nickt }$v  ▁loadsx  工业园区z  ▁corrupt| 勇士~  ▁slip {"  膳食纤维  development lette  ▁Helen 转折  尔多斯  说的话  ▁Near 株洲  政法队伍教育整顿 人流 ▁eru  规划建设  和要求  ▁observation  ▁downloaded  ▁briefly OSE  ▁Gaza  集中隔离医学观察  European  特有的 加薪 打理  ▁Calif  ▁Bever 贴近  着自己的  ▁Yale  小学英语 庚子  ▁Grade  ▁offshore 安定  自来水  可以把 做大  ▁Really 封城  ▁measuring  ▁medieval  更多人 ▁aug  聊天记录  British uang  以及其他  ▁Desk 封建  鄂尔多斯  的真实  工作需要  ▁runner  ▁Besides  ▁independently 漫步 恋情  ▁alliance 不走 向南 ean  ▁Karl  ▁Globe ▁Ts  ▁melt  都应该  ▁Strategic  人事考试  ▁solving uate  所需的  ▁crypto ②　 ▁Ax  奋战在 供暖  ▁practitioners 色素 Fre ▁EPA" 编剧$  ▁Butler& 交接(  ▁fost*  ▁magnificent, 两点.  ▁LinkedIn0 oped2  istically4 冷藏6 我又8  放弃了: aca< 心语>  洗衣机@  ▁drawingsB зоD  淡淡的F 地形H 提琴J  爱情的L  文章阅读N  健康发展P  要根据R  和学习T 里克V  ▁emergeX 话语Z 宝典\ 补血^ 主打`  和朋友b  先生的d 号的f  商业银行h  ▁valleyj  ▁Cemeteryl Denn  ▁sentencesp  资格初审r 因其t 属猪v 多大x ocityz 恰好|  ▁disadvant~  ▁nowhere 属龙  ▁Chand 那一 紧致  ▁smartphones 轻盈  ▁Christianity  市人民医院  对孩子的  ▁wines 切的 ▁Tib  ▁deficit 军官  ▁licensing  ▁Regulation  的一名  北京时间 unge  ▁Assessment ▁Tab rained uid  起到了 ▁NSW  ▁Hopkins  ▁stabil  ▁Trailer SB  ▁reass pic  ▁hazard 芹菜 erek Far 一听  ▁rhythm  实验中学 一字 康熙  有需要  ▁plugin  ▁Parad 负重 老子  ▁strengths 干了  疫情防控要求 ▁MBA ernel inance  avorite abwe 茫茫  ▁arrange  党和国家 旅程 树林 千古 久而  制作的 就更 三里 uddy  example  ▁soldier 病因  ▁fled ги  ▁ranges 死者 穷人  几分钟 Ren ▁hem  中医医院 ré 麻木 争夺 正月  化管理" 领略$  而这些&  巴基斯坦( 开办* idate,  变得更. allow0 otypes2  也能够4 黄埔6 同款8  ▁critically:  ▁honestly<  提高政治站位>  ▁Colombia@ TravelB  ▁loadingD 儿科F 跌幅H  ▁grandfatherJ 指控L  orkshireN  跟党走P  ▁exemptR  ▁promptedT  智能手机V  都无法X  ▁hikingZ  ▁beta\  美好生活^ 党旗` deleteb 三伏d  Attributef 走来h  ▁Presidentialj 歌舞l  确实是n Satp 带到r ▁Sidt 大夫v  orientedx  完成的z ▁Cad| 不惜~ 雪花 happy 她还 imo 银河  详细了解 恰当 比对 umatic 奥斯  为自己的  ▁historically  ▁triumph 红豆  并提供 Hon 把控 grid 鲜艳  把孩子 听书 店长  ▁variations  ▁suited  独一无 ▁rap 总共  的热情 高二  支气管  为实现  退役军人事务 them isi ▁mph  ▁diesel  Community  和学校  ▁dealer eval  ▁accelerate 手足  也有人  不经意 ▁Pok  ▁molecular  ▁Leaders 转眼  ▁bachelor  ▁Gene Maybe 零件 osex ▁Mak 印尼 核定 毒株  切实保障  ▁flooding  ▁Capt  的优秀  ssssssssssssssss  ▁fence 和中  ▁deny mare club  javascript 服役  扎实推进  物业公司  ▁Users  ▁forcing 时事 价的  确定了  ▁outlet  ▁attendees  展现出 ▁obj  ▁asylum 鲜美" 领军$  精细化&  ▁subscribers( 你最* issues,  ▁Musk. 恩爱0 失联2 常说4 勘察6 navbar8  ▁predecess:  卡积分< ▁Inv>  ▁rankings@ UBB zdD  ▁citizenshipF  是因为你H 球迷J  ▁QuebecL  流行病学调查N ueseP  ▁StorageR 新政T  ▁BranchV  ▁organizingX 兔子Z ▁MB\ 心跳^ GI` 不平b menud  ▁Achief  ▁fortunateh 稀缺j  ▁gapsl  时间安排n 党风p teamr  帮助你t ▁CBDv  ▁complicationsx 巾帼z  和专业| 小巷~  ▁Anal mund  ▁notably  ▁boyfriend  ▁Launch umbing  ▁Violence 佛陀 ollo  是目前 收款 拉萨 死后 arded  新发展理念 ▁MIT 湘潭 喀什  营业收入 养颜  ▁Biography 卡通  ▁excessive  ▁Ruth  ▁educators  一口气 税额  不要再  周恩来 iments  ▁Journey 切除 exper thood  ▁prizes  他强调  ▁dirty  ▁sponsors  才不会  接种点  非常适合 ewater 鲜嫩 Body Cast 皇家 璀璨 妙招 不肯 接入  ▁absent 附中 涉事  ▁crystal  ▁demonstration 纵横 揭谛  ▁ISBN  ▁Bryan ACH arin  的应用  ▁wounded 客流  他指出  还不错 ysical 百花 口红  ▁locate  余万元 持股  ▁surgical ▁Ask 风电 的了  在日本  ▁Industries  ▁streak  ▁Eagle  ▁medications  玫瑰花" 年期$ 破损&  结构化( music*  ▁ре,  过多的.  element0  完整性2  imbabwe4  ▁Python6 KTV8  专案组: 的脸<  立刻就立刻就> 羊毛@  ▁NeighB  舍利子D  ▁PlayStationF  urchaseH  ▁CensusJ 过户L  所以我N  不可思议P ancelR rioT  资料图V 宣告X 得头Z ▁Dod\  ▁Soccer^  Department` 川普b 换个d 首家f 马鞍h 和高j  ▁Sleepl  ▁distributen 案发p 龙岩r  ixturest 见解v  的工作人员x pptz 作弊|  报名信息~ folk  ▁trio  ▁gray iji  依然是  ▁selecting 乐器 撒娇 roe  意思是 有事  的经典  Tuesday  建造师 LI 跟鞋 联手 iane  liances  ▁seating  测试卷  ▁halt tree harm ookie  ▁Celt  ▁holder  的土地  及以下 imir 填空 经贸  用户的  ▁Close  ▁accurately  ▁drone  ▁promotes  ▁consisting ▁gem ▁doi  瞄准农村 mith  平常不过  ▁Aven  告诉大家 玩的 ▁Opp 推拿  选择的  ▁Brief  ullying 一扫 仍是 ():  ▁argues 防空  ▁explosion Ta 祈祷 范文  ▁cleared 方程 ▁apt  ▁tuition shirt 华润  Working  公职人员  ▁adverse 泪水  会让你 画的  curement  ▁loyalty  ▁Additional 上衣  想办法  发生变化  ▁Liberal SARS  ▁от  ▁appointments" ▁Mut$  ▁Uber&  ▁rides(  手把手*  ▁comprised, 漳州. 为本0  ▁earthquake2 倘若4  ▁protagon6  教育学8  ▁initiated: lich< 瘦身>  ▁Factory@ 江门B 胃癌D  犯罪事实F  ▁PattH etoothJ 限额L  ificatesN  igatingP KimR 杰出T  ▁boomV  ▁intersectionX 来往Z 油炸\  三角形^  ▁chore` parkb 万户d  的妈妈f stenh ▁NAj 不相l ICSn 德育p 日程r  课堂教学t  平常人v  ▁traffickingx  莫过于z  ▁teenager| 益气~ ▁GPS 秘籍  ▁dimension  来之不易  ▁anytime  ▁Upper 先是  ▁distant 一动 人参 纲要  changing  订阅号  ▁mortality  ecution 极高  ▁indirect  ▁dimin 哮喘  ▁Finn  要继续  ▁disk 月上  奋勇前进  ▁Grove 多得 ▁ISO 邮票 Dom  大家庭  工作机制 港区  ▁Canyon  为指导  不具备  和企业 graph  电子凭证 减退  验证码 无缘  ▁Luck  ▁Mountains 立德  可持续发展  ▁occasional  服务于  紧紧围绕  非物质文化遗产  是不会 ▁Mix 息息  ▁Cabinet  ▁creators  ▁Chemistry 车速 武警 她也 道教  等进行 OO  高中生 勉强 灵芝 ▁Fat  ▁Heat 之初 河西  人民警察 纸上 公英  和自己  ▁burned 实干  ▁Idaho 无缝 ▁Д  ▁tanks  ▁connects  　~　  ▁median  夏令营"  请在后台留言$  Interface&  航空公司(  ▁bearing*  ▁gameplay,  的概念. 世代0  服务站2  付出了4 动机6  选择题8  ▁Uncategorized: 一向<  ▁aspir>  ▁Julia@  ▁PlacesB 花草D 最怕F  ▁lectureH  EverythingJ  个人账户L ▁LayN  ▁relaxedP  ▁FosterR  ▁troublesT 北站V 缅怀X sleepZ  按照规定\  ▁shifts^  ▁instructor`  更加的b  ▁breakthroughd  黑色的f  博士学位h Typesj  ▁remindsl арn  付出的p  新模式r ongot yiv 红火x менz  流行的|  angered~ 央企  ▁flee  垃圾桶  Members ☟☟  理想的 NP //　 对联 日历  队员们  ▁dairy inea  确诊患者 osity WI border  新西兰 海参  ▁neglig  务工人员  再平常不过  ▁Iceland 之作 Tree  ▁Mills 警报  Section  ▁arise 莫名  ▁Reyn ▁AF ▁FM  ▁corners 的王  ▁thriller  足不出  收益率  报名参加  的处理 丁香 心愿 泡泡  ▁Matter 大棚  ▁viable great 加水  ▁Fact  ▁Zero  在使用  ementia 张国  心中有 metic  中国科学院  绝大部分 粉色  可获得  现将有关事项 ▁Rah  ▁shocked 翻身  ▁kidn  ▁Collabor  和安全  明确规定  ▁rocket  年级上册  服务保障 many  不得超过 " 党的十九届五中全会  ▁containers ndum  十九大 拖延  ▁graduating 蜜蜂  ▁lane  好朋友  ▁sake" 蛋黄$ OUT&  比较高(  还会有* ▁[', ospace. 停工0 存活2  旗下的4  胰岛素6 有条8 将至: record<  快速发展> 区位@  ▁drunkB 这天D 明代F 中新H valsJ  ▁CinemaL 决议N  ▁discretionP OGR  ▁segmentsT  ▁JulieV PutX  著作权属Z  ▁detention\  为保障^ 出征`  ▁Teachingb  日白天d  研究决定f  注意了h 穿梭j asisl  ▁billionsn  ▁bronzep riever  不出来t  structionv  批发市场x  Insteadz  在武汉|  ▁Ship~  ▁Rosen  ▁Harper  习近平同志 ▁Nan 固体  学历学位  ▁Vacc  ▁obtaining  ▁tends  ▁figured  不仅要  ▁Chan  imensional 停产 帷幕 ▁sel  联系我们进行删除  ▁Moroc 试讲  ▁scary 赵丽 动人 同济 相邻  ▁Physical  ▁bedrooms  是需要  和使用 ▁McL  ▁Pere 会说 Window  ▁shine 属羊  预防和  中国政府  ▁Always 向好 大神  购买的  取得联系  在他们  积极主动 黄豆 公文 切割 端的 胸闷  境外输入确诊病例 ▁Beg  睡后收入  ▁dupl  ▁individ  一些人  ▁Abraham 反抗 丝绸 方形  ▁Returns irs  ▁hills 家用  一般情况下  ▁electron  殊不知 来做  太高了  蒲公英  ▁Tigers  ▁commerc  ▁uniqu  ▁optimal 大涨 整车 Arg ▁Rd  从来不 克拉 ishe  ▁sacred 梅州  ▁мо"  ▁functioning$ 破获& 明媚(  ▁Continue*  ▁Individual,  ▁fitted. 伪装0 土木2  ▁Preview4  看来都是6 ostalg8 ▁sie:  ologically<  ▁Request>  ▁Script@  青海省B achiD  信息系统F 全册H 意境J KWL 更具N 直击P  ▁NormanR  ▁ChapelT 不全V 面谈X  导致了Z  编辑丨\ 亦无^ 繁忙` 不异b 于心d 个税f  ▁appealsh 歌词j 获利l  疫情传播n  ▁affiliatep 吃药r  就这么t жиv  会引起x 成型z 清澈|  Wednesday~  好不好  体系和  represent  ▁Croat  because  ▁societies 香水  ▁continually  ▁Representative OLD uits ▁GDP  的整体  ▁Guys 签到  所高校  ▁discounts 干嘛 应试 安抚  ▁Previous 村级  讲故事 Doc  ▁leisure  咨询服务  大概率 ▁ber 教资  脂肪酸  服务器 oko  ▁Scar  ▁Holmes ▁$( 待人  市场化 两款  两人的 祷告  Function 顺丰  ▁Malt  主要负责人 ▁Sr  以下是  虽然是  ▁vendor  ▁collaborate 教职  现阶段  ▁queen 现身  的根本  毕业证书  高高的  执行力 ested  即可查看  ▁milestone  ▁affiliated 发性 ▁Mey  领导小组办公室  ▁Lauren  研究生学历  很多家长  ▁Used  大家都在看 ND 斗志 暴涨  ▁Until window 香甜 ès nn iosity  ulative  ▁variation 家政 大山"  深入贯彻落实$  ▁oversee&  全程佩戴口罩(  日晚上* 三室,  均为阴性. 说服0 Job2 抗美4 源源6 郑爽8  污水处理:  爱自己<  ▁Parish> 梦中@ 脑梗B 转正D  大家对F  玩游戏H  ▁gramJ  fictionL 三代N  ▁MedicaidP 恩施R ChiefT 点滴V  一次次X 退税Z  的办法\  点在看^ 认清`  ▁argsb 以致d 物种f 汤汁h ▁},j  ▁prospectivel 李文n  ▁Hurricanep  广大人民群众r  宣传片t  因为我们v  ▁impex  ▁recreationalz  交通工具|  ▁Prison~  大年初  ▁Profile 心生  ▁Sacram  ▁tablets  可不是  乡村旅游  之间的关系  address 橡胶  眼中的 Cho  ▁approaching 力求 得失  怎么看 ▁RES 九年 depend  atility  发现有 但你  ▁Schedule  游览时 aran 找准  天然的  ▁recruiting 县人  以后出生 idding  ▁fuels 重度 方舱  ▁lawn  ardless 把她  低下了 对策 ▁GMT  社交媒体  ▁modes  ▁cancelled  ▁contrad  ▁satisfy  ▁Vienna adians 畅销 isb  没必要 ungle  富有的人 军训 mount  资格证书 六月 肆虐 排便  不合理  全力做好 菠菜  可能的 ortium  微信关注博野  治安管理 parse 念念  的快乐  带回家  才会有 在售 差不  副总裁  都市报 半生  一个个美丽 肚脐  ▁fighters 和水 不慎 尾巴  ▁Resident"  场比赛$ ▁Seb& ocaust(  的一段* 费的, 二胎.  网约车0 辛巴2  益生菌4 eor6 ▁Tat8 地在:  停电时间<  ▁firmly>  想象的@ 起飞B 特质D ▁GamF 二战H  ▁healJ  ▁TeacherL  atabasesN 大额P  ▁traditionallyR 挤压T  ▁biteV 深耕X  ▁temporarilyZ 不长\ 久坐^ late` 问卷b 人意d  研发的f  旗舰店h  的一封信j  ▁Tournamentl  ▁mattn 特级p  ▁Warriorsr ISOt 人了v  传染性x 私密z  拿出来| Blog~  ▁intr orical 鱼肉  人们对 上万 ensen 刷单  表达能力 )" wiki  ▁encountered 巨额  工作队  出现问题 在读 ів  ▁Lith  应用于  也已经  行动起来  还有什么 料的  无疑是 ▁sla 庄镇 高大 连日 Given 引人  ▁crowds 膜炎  管理系统  病人的  对学生  ▁Something  journal 视察 卖淫 平板 桌上 Ken  大酒店  遇到了一个个美丽 ”— 有好 ▁Ng 外在  届中国  ▁Error  ▁baking 几位  ▁disciplines  ▁matched  ▁civilians  ▁се  周一至  公务员考试 ▁HBO  ▁handy 家门  就行了 毛衣  ▁Charg ▁Dad  ▁tunnel  ▁Welcome  ▁advers  个孩子  工作岗位 每到  ▁Below  imental Walk 开户  :　“  体会到 short 会很 ▁CDC  ▁convince ▁О 美方"  今年是$  ▁Roth& 手绘( aset*  ▁listings,  人公告.  ▁doses0  这对于2  特别行政区4 ▁Jar6 盐城8  喜爱的: ང་<  ▁casting> 红娘@  编辑部B  ▁monsterD  ▁tongF  ▁versatileH  管理有限公司J 秘方L 高危N  也都是P  参加面试R  也只能T  营养价值V  只有在X oceseZ  现在是\  …………^  ▁prohibited` 曾被b 将从d  ▁Risef 包子h граj  四个意识l  ▁Weddingn Hyp 天真r  客运站t 开出v  ▁consciousnessx  为重点z  Article|  Everyone~  ▁commands 时速  将自己的 老龄 食疗 昔阳 uded  你是不是  接受了  生活中再平常不过  ▁inevitable  ▁Begin  ▁illustrated 胸怀  我一直  作出的 赞助  做好准备  ▁Answ  ▁rook minded  ▁Muham  艺术类  Session ( !遇到了一个个美丽自然的  牢固树立  开学前  ▁unity  ▁Door 院子 arming 痔疮 来访 团聚 悟空 会儿  为主的  Charles  ▁vector 裁定  ▁marginal  因涉嫌  管理学院  ▁Markets 牛年  ▁revealing opez odia  ▁Volunte 亚马 FDA  东莞市  ▁probe 充沛  这款车  ▁punishment 栽培 ▁Bak 屠宰  久而久之  ▁backs  严重违纪违法  不定期 相聚 Mont 初衷 劳累 捷径 ▁Kin  专业技术人员 Marc  到时候  ▁midst  ▁accountable  ▁Surgery  我们希望  ▁voluntary 楼面 idy  ▁Bobby 多天 口诀"  的数量$  归原创者所有& 小车(  ▁disputes*  的理由, iker.  Perhaps0  做的事情2 读物4  ▁Recovery6 破碎8 当他:  因地制宜<  工程技术>  Configuration@ 刘海B 年少D  关注我F  ▁nurtH  ▁HeightsJ  ▁DylanL filmN  ▁namelyP 插入R shopT  有时还会V  ▁SociX 喝了Z izers\  ▁Territ^ 遗忘`  ▁provincialb  整理了d ▁Pacf Beyh  ▁thermalj  性生活l thsn 八十p  ▁metaphr  ▁statuet  这部分v 公用x 泰安z 甜美|  纪律审查~  ▁anime  不下来 until 打出  破血流 配音 ▁tv  安全头盔  的岗位  村村民  乘风破浪 大运 ▁UT utory 满怀  ▁compound  艾滋病 画卷  严重影响 火焰 tex  逆行者  自动驾驶  正式启动 禁令  义务教育阶段 Equals 牵手  ▁Vlad  ▁Squad 款项  知名度  拿到了  小女孩 Stay 不振 阳泉  ▁Electronic  ▁Certificate mult  将进一步 Spec  路交叉口  ▁Founder 听着 它能  手机号码 pal 运送 降解 心安  ▁Boeing  ▁scrap  ▁straw  ▁Nuclear 先把  仅仅是  司法局  ▁divide  现有的  外国人  inceton 并列 fw  社区卫生服务中心 立减  ▁congrat  ▁Agent 台山  可享受  也希望 违建  ▁Firm  ▁filming = 6▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁  ▁preference 降到 SELECT  高峰期 isive  管理规定  ▁strictly"  ▁Pages$ Mode& 专线(  ▁rivals*  ▁obstacles, 有色. 植株0  不能再2 数值4  强降雨6 时任8  ▁commitments:  ▁Gaming< Iran> 投案@  消失了B ▁LedD  各地区F  ▁incentivesH  小朋友们J  教育事业L  ▁companionN  ▁ComplexP ▁nodR 上加T 荆州V 后人X  返乡人员Z  微信公众平台\ 亲手^ 传出`  Accountb  ▁hidingd roleumf epsh  ▁Goingj 金的l  提高警惕n  有下列p  进入面试r  Interviewt  ▁harmonv Centerx  上班族z 横财| Flor~ 攀升 舌头 脏腑  这些看来都是 药业  ▁flagship 年末  ▁customized  完善的  ▁Ashley  ▁holders  ▁controlling  ▁saves 1 *这些看来都是生活中再平常不过  给别人  长按下方二维码  ▁Yemen 唯美 Reader 被判  中央军委 edar 录像 妥协  Australia 顺便 人脸 worthy 家风  还不够 Sep 而去 Bit  四个自信 植树  ▁Legacy 犯了  ▁Taliban  Currently 三千  ▁productions 年假 七年  ▁Adult  ▁lifted 光是  ▁enforce  ▁Conservative  和建议  ▁kilometers  习惯了  ▁Tiger  ▁revolutionary enery 一整  而且在  ▁chap 兰花 icion  笔记本 ▁hed  交通运输局  ▁distinction  ▁Gate anie ctrine  ▁basement  菜市场  米以上  小区的  去探寻  ▁separately 横向  指挥中心 从头  ▁straightforward 民用  健康成长  ▁Leonard  ▁contrary  工作内容  ▁hilar" OA$  就是我们& ( !人民代表大会常务委员会( 骑士* east,  ▁merchand. 道具0  家协会2  ▁Certified4  进行中6 全镇8 对的: 心目<  oubtedly>  ▁teenage@ 戒指B raceD  PacificF 言行H 属马J 父子L  学习方法N 开会P manuelR uberT 碰头V 交房X  ▁hopefulZ  的女儿\  Several^ 情愿`  判处有期徒刑b 信阳d  ▁mixingf  ▁specifyh 能屈j amptonl 内经n  electionp raitsr 援朝t  在当地v 的金x  ▁shippedz  ▁philanthrop|  ▁radi~  基础设施建设  直播带货  ▁Rams  首先是  在过去 jjjt  ▁protocols  在外面  ▁interventions 前夕 vari  才能在 对称 fields 宗亲 ▁Tea 徒弟 孕育  ▁recipients 常吃 ▁GB  ▁horn  的人口  ▁intend 斯科 arel  ▁Rent  ▁Crystal  停留在 先烈 客房 hm  活动现场  ▁indication  开玩笑 posium  ▁Thai  ▁Hours xue  足不出户 并以 网购  intendent  会议指出 建于 hyd  ▁Spencer 救人 间盘  沟通能力 河镇 dn  不动产登记 ringe  ▁Equity  patient 办好 泗县  这是我 业界  表达了  ▁Terror ilia  ▁practicing 嘲笑 undry  ▁Notice  脱颖而出 到店  针对性的  中央纪委 hong EG  受到影响  ▁slavery 八月 顽强  ▁illeg  重点人群 执政 Ma affe" 四面$  打底裤& "@(  进入到* 不言,  已全部. LR0  打喷嚏2  知道自己4 不息6 坦然8 ▁è:  ▁Memphis<  ▁declaration> uli@ 日出B capeD  ▁OpinF 周刊H  汽车站J 龙门L FacN augeP  不接受R  ▁radarT 操控V  简直就是X  我们将及时Z 卡点\  ▁tailored^ 超载` 世事b  Referencesd 这儿f  长大后h aintedj  有这么l 花呗n  是一项p 文人r  胶原蛋白t 假装v  ▁citingx  短期内z % 著作权属归原创者所有|  opening~  更需要  ▁approve Bur  ▁adjusted  ▁json  美术馆  ▁porn 合约 路况  各行各  ▁Multi  ▁charitable ICT  就在于  常住人口 东坡  ▁Horror 精度  ▁theaters  等功能  ▁merc  ▁Brady  ▁apple  ▁grave eca WC fty 名下  ▁Philosophy 年满 恐龙  合作的 auf  ▁inequality 徒步 周某 税款 seen 敦煌 ifiers avil  ▁Bennett  充分肯定 ▁т 深层 确需  ▁Bour 精密  ▁Joel 皮书 ▁pad  生态环境保护  落到实处 蒸发 types 个大  ▁вы dj docs  中心小学  ▁Nursing mag  ▁rewarding  ▁mapping  的女生 thread  ▁Ancient 苏宁  方向盘  ▁handles  ▁facial  ▁Harm  ▁Gender 大吉 药房  路上的 弥漫  ▁Fans  留下遗憾  交通违法  ▁Lloyd  ▁optical" 占领$  ▁soundtrack&  空间的(  ▁toug* esity, impse.  小贴士0 awei2  关注吧4  得益于6 翻倍8  ▁accessibility:  ▁freshman< 到场>  在这一@  招聘职位B 万万D 气和F 稳健H  ▁pairsJ  一小时L ycN  期间的P ▁AnaR  ▁DifferentT ▁BagV  ▁HinduX 划定Z  同时还\ Kevin^  ▁unlimited` playerb 唐朝d 桥头f  西游记h  精装修j wedl  这次的n 交际p osuresr  ▁Miningt 县的v  会增加x sharez 渭南| 特产~  哈哈哈  本人身份证 湿疹  奋斗目标 讲课 ipes 网传 iploma 团团 hash  电信网络诈骗 AH  家门口 小明  ▁Railway 邻里 rants their  ▁reviewing  ▁stainless 外汇 床位 olia  ▁orche 沿途  机动车驾驶证 一案  ▁Pride  ▁overl 号称 查实 桂花  多一点 sal  ▁assessed 化为 avier  ▁Miles  ▁therapeutic 票房  科学研究  ▁Infrastructure 功课 大一 before 不计  ▁Choose  在这样的  ▁shade 主营  ▁Status  ▁breakdown  容易被 ▁WHO 市中 视听  ▁Patriots inis Buffer ▁Hob  非常好的 ▁{}  卫生习惯 打压  ▁Trial 刻苦  着自己  ▁glasses 风力  华盛顿  个国家  ▁buffer  ▁Parents  服务热线  ▁migr blue  ▁terrorists 根基  普洱茶 私信 足迹  也是有" 现货$  ▁lover&  ▁fever(  ▁fights*  ▁boring, Land. 呼气0  粤港澳2  amation4 韵味6 values8 Listen: 李克<  ▁bell>  不小的@  ▁prayersB 券商D  容积率F  ▁YorkshireH  ▁DriverJ ▁SFL  桂林游览时N  ▁jacketP 产区R 讲的T  森林公园V ▁UPX enesZ  ▁withdrawal\  第三步^ Small`  详情请b 会是d  ▁Rangersf 在最h  进口冷链食品j 增幅l 争做n  ▁hoodp 开支r 但从t  backgroundv  ▁fictx  欢迎留言z  只要是|  环境保护~ 看你  乌克兰  呈阳性  ▁shifting 李佳 秸秆 adays  ▁cache 小王 规避  小白理财  引发的 alone  ▁formally uity  封闭管理 沸腾 except 高出  ▁Leeds  ▁darkness  ▁Broadcast  ▁genres 运算  ▁Applications 不加 中途 指尖 触碰  ▁Bulgar 服药 排泄  ▁то hey 肉眼  红色的  ▁iTunes 顽固 adding  ▁Behind  永远是  诸葛亮 随处  其中有  紧接着  ▁questioned 不实 chaft  ▁uncomfortable contin  ▁inaugural 沧州  就会出现 精气  高分到低分  ▁Ways  ▁parade 下乡 渲染 环路 Come 欧元  ▁tile KM  ▁Laun bind  重大项目  ▁Formula  为保证 偏方  综合整理  的自然 patch 根系  社会责任 fix 抢劫  发脾气  ▁Venezuela  抗美援朝  厦门市  喜欢的人"  ▁prosecutors$ ottage&  满意的( Econom* 非典,  ▁shelf. 提拔0  县委副书记2  ▁immigrant4  亚马逊6 手掌8  ▁spouse:  ▁varying<  ▁culmin> 守望@  ▁BrusselsB VIEWD  组织领导F  ▁reconcH 魔法J  ▁RoleL  ▁ChromeN 容纳P  里的风景R  项规定T ewideV avinX  ▁THISZ 女的\ 研修^ kward`  ▁Bloombergb 宣判d  ▁outlinedf 内存h 商人j  记得前些年l  ▁attributedn 护照p bachr  的正确t  ▁sellerv  ▁landlx 在做z  ▁extending| 互补~ Very 只剩 造假  ▁beaten 日月  现场报名  ▁applicant  他认为 宁愿  核心价值观  important  昂首阔步  有限的 低下 电磁  ▁Bradley 李嘉  省委书记  什么叫 amon  的小伙伴们  ▁strongest 下巴  正因为  都有一个  iblical  奇特的造型 掩盖  ▁Guest  的生长  高层次人才 uson 大厂  记得前些年我到  ▁united 韶关 Russia ▁EC  ▁Ranch 胶囊 阅卷 ▁Hur  ▁Bach 开来  就必须  由此可见  发展规划  ▁transfers  ▁Ferg 看图 全党 XT kees 与否  ▁specifications 化肥 东至 中雨 ▁Pom  年龄在 issan  ▁praised 　♪  ▁indie  平常人低 年会  ▁Teen ende  大家一起 淡化 红尘 串联  经验者优先  ressing 杨幂  深深的  ▁Minneapolis  到底是什么 voc ▁IF  第一课 时而" 建军$ 衣食&  ▁robots( 说他* 厚重,  检察官. kj0 亲身2  ▁Gates4 . '遇到了一个个美丽自然的石洞6 分娩8 weixin:  洞口都比< 车企> 可控@ 天文B  ▁foundationsD riciaF 产科H  发展有限公司J 水源L 必胜N 波澜P  ▁belongingR 泳池T 逃生V biaX  ▁hometownZ  virtual\ 吸入^  神经系统`  咨询热线b ▁Nind  爱国卫生f  ▁appeth 墙上j  发生以来l  ▁interruptn  Educationp 驾校r  鸡蛋价格t  洞口都比平常人低v  ▁Furnx 稳妥z  可能导致|  ▁Midd~ 该车 ployed  ▁cuisine 观影 尔斯 打架 )$ 抑菌 真假  roductive 雨天 suits  ▁Expert 还被 时效  ▁circles WR  祝大家 心肺  开幕式 params Mur  ▁ministers MBA  更何况 买单  国内的  现在已经  说明书 脸部 填充 属狗  见附件  ▁penet  ▁Montgomery 饼干  保护区  弯下了  空气质量  ▁acoustic 分红  ▁Newton  ▁Villa JS 的第 感人  压力大 anç  ▁Stein 中信 Know  ▁cartoon 窗外 Assert  aterial  告诉他 ' ******************************** Series  ▁Independence 杨柳  ▁Grammy ▁NGO  ▁booth ▁bip  Comment 一地  数据的  ▁cattle 二字  有温度 国网 haul  奇特的造型充满了  ▁flags 收敛 迫切 就用  ▁Venice 举例 离去  ▁rehabilitation 铸造" 区市$ 拎包& 有奖(  ▁Kumar* ▁Cox,  ▁retailer. msg0  老师微信2  全心全4  ▁diplomatic6 save8 家和:  停电范围< 置换>  资本市场@  ▁centsB 寡妇D  的用户F  ▁elevatedH MYJ 芬芳L  ▁reluctN  ▁landscapesP 华丽R 名誉T 清扫V  当然是X  昆明市Z  结果显示\  ▁intu^ 养分` 畜禽b  ▁airlinesd 主食f Karh ▁rodj  ▁burgl  ▁learn Graphp 肝炎r  高效的t 部落v 承德x  ▁contributorz  战斗力|  神奇的~ nu 上饶  ▁catast ці 心怀 规程 女星 轰炸 Carthy Entry  艺术品  keeping 切片  影响力的  ▁characterized 流出  ▁disposal  ▁matrix  ▁Gets ▁hij 有为  把这个 窒息 far 政委  ▁warehouse " 记得前些年我到石林 化市 到你  时间表 摧毁  ▁Equipment 小数 电竞  十分的 限期  Building  中国青年  ▁Frances 刘备 略有  我还是 缤纷 宣誓 事了 把钱  ▁relaxing  ▁Lodge  ▁queries 瑶浴 漫漫  ignment  ▁Pret  ▁newer  ▁interim  更适合  总承包 ▁Wag %~ 配料 ▁len  ▁Joshua  ▁Videos  ▁Consider  ▁individually 抢占 mos  ▁studios  ▁Rebec  精致的 Turn  parents 鉴赏 五彩 无畏 opped oys  需要注意的是 敏锐  ▁politically 率达 关切" feat$ 唯独& 女方( 单人* 多半, 肩负.  了一个道理0  民族团结2 % 新型冠状病毒肺炎疫情4 ambda6  ▁Funeral8  为单位:  ▁carriers< 颁布> 氨酸@ 在心B 马店D  学习成绩F  中国经济H  副秘书长J  可能有L  古诗词N  只能说P 体型R  的新闻T  福州市V  产业化X  ▁guitaristZ  苏州市\ ▁Cit^ 读完` 亲们b  网邓州d 就跟f 德华h tarj 着我l 力气n  ▁rehep  回来的r  ▁eternt 涉黑v 改版x 那位z sanwen| 暧昧~  ▁Plaza  根本没有 Brian 增量 Labels  中医认为  ▁horr 彩礼  年轻人的 “,  写作业  ▁Principal  电商平台 汕尾 Friend ▁Meg 山海  ▁Norwegian ▁Mis 西洋 口头  ▁suggestion  四六级 农牧  在这些  ▁motorcycle 染色 Senior Writer  征途上 源县 Tele  一如既往  ▁Cruise  ▁breed 疫区  artifact ▁Pos  研究报告  ▁brave 你来 紫砂 落叶 INGTON  ▁tales 谎言 丰县 bag 跳动 elia 纹身  ▁Colin  文章阅读网  ▁messaging ▁і 当当 ི་  广大党员干部 ор 分队  rington 祖先 精湛  ▁murdered 高架  ▁Vers  都低下了  ▁nominations 起初  ▁bump 安稳  始终保持  ▁Papers 无悔 ▁[[  心理咨询师 utable 誓言  计划的 FILE  ▁scared  阅读量" 龙华$  昂贵的头& izarre(  ▁Newcastle*  ▁treats,  ▁Sullivan.  ▁Welsh0  ▁illum2 辟谣4  富有的人物6 olving8 投保: 做强<  无论怎么显赫>  洗手液@ 反击B  这个地方D uablyF 共济H 在岗J 涂料L  ▁statisticalN rinaP  扎实做好R  ▁TouchT 赵某V 高高X 入味Z 寄生\ USE^ 树人` negb  都低下了昂贵的头d  农业生产f  ▁Wereh  ▁beatsj ▁Kral 美化n 背叛p  最贵的腰r 种族t  去探寻石洞v  弯下了最贵的腰x  必要时z  停课不停| handle~ jack wd  ▁disclose 融化 ▁är 克的 开考 years  价格为 死于  ▁readily  ▁notified O H詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘詹飘飘 肖战  ▁skip  ▁Thought  ▁Educational  法定节假日  ▁fancy  ▁Valentine 书本 庄园 痛经  ▁soap 契合  ▁builder  available  ▁meantime 逐年  Security  ▁puzzle ▁ore  ▁counting  ▁Jeffrey 年中  自觉遵守  ▁cruel 城东 好吗  ▁preservation 北风 寒冬 anto 克制  的收入  ▁Period 微风 KA Martin " 去探寻石洞里的风景  其中一个  ▁timeline 和华 万分  美国总统  度假区  学习交流 复活 抽象 unal 心想 用房  ▁Operation 春光 hz 总计 得不  ▁inbox  ▁regulated  ▁suppose  认真落实  主办单位  ▁attacking 的情 Sarah 打拼  的一条 章程 items  ▁calculate  ▁cooked 影城"  ▁Province$ 升温&  好不容易(  ▁Pearl* ▁Т, 个子.  王某某0  ▁profitable2  ▁Chuck4  ▁occurring6  倡议书8  等各类:  ▁beans< places>  ▁equation@  ▁ProtestB  efficientD  ▁SendF  ▁ContribH  只是在J  ▁DragL ailureN 它在P ▁uptR 重生T  ▁RalphV  ryptionX 上架Z 逛街\  还不如^ 分组`  Positionb 抽检d  ▁Renewf  脱贫攻坚战h 右边j Suppl 佣金n 后排p 春耕r 典范t primv 妥妥x  ▁Meadz 孕期| 辩护~  非常大  不传谣 云端 喜庆  面积约  ▁avoiding  通告如下  ▁prost  很多人的 洋溢 章节  的秘密 也对 jab  也因此 向阳 明细 1 *请在后台留言联系我们进行删除 还想 置身  王女士  ▁Diversity  ▁cryptocurrency 考官  涉及的  ▁Message 猪瘟  ▁viewer  立方米 枕头 ourses 患上 lar 开后  ▁Applied  处处长 fre  中老年 为宜 buy oval Lim aires 校考 houses ▁Maz  ▁sever holm  ▁Floyd 魔鬼  ▁spirits 体贴  搅拌均匀 拖拉  都是有  农业农村局 rivacy  桃花运  Present 调和 并处 街坊 coll 左边 名气  人人都  一直没有 出路  ▁Shore 弯路  ▁algorithms  的决心  带头人  ▁indicating 红花  instance  ▁Registration  ▁suspects  ▁shake HI 购车 振奋" 泰州$  都不敢& both(  ▁lightweight*  ▁taxi,  ▁grandchildren. 池塘0 爱我2 vised4 日产6 Intern8  部署要求:  ▁lined< 会和>  不断增强@  截止到B ▁CAND  ▁craftsF  的一份H 冬瓜J  现如今L 参展N 交所P  ▁coupledR  优化营商环境T  ▁introducesV 指向X  全面建设Z 白沙\  发挥了^ 轮廓` 起始b wbd 西站f  我母亲h  他们对j ( !和新疆生产建设兵团报告l 把关n  ▁Toursp  原有的r  ▁varietiest 储蓄v 本职x  的保护z 听完|  ▁bridges~ ▁Gaz  ▁financially 冰凉  ▁watches  ▁Knowledge  ▁substitute  ▁dignity  ▁Walking  ▁drums 检索  ▁electoral  ▁attitudes  ▁airports  一阵阵  追踪到  ▁feas  ▁firing  在感情 onc  ▁neat uced  ▁evaluated 默认  ▁Overall  和规划  新闻联播 坚韧 nings  在医院 北宋 + $请您先点击上面的蓝色字体 ▁Mul 变身 跑了 龙山  有关情况  ▁vitamin  累计确诊 大笑 流泪 iOS Soft  ▁Prix argest  的最后  ▁Herald  ▁este  的事件 新颖 点到  ▁rack  的危害  ▁FIFA  ▁quad pad  的智慧  课程内容 关乎 十月 毛囊 蠕动  ▁Pradesh lining  ▁Naval  ▁Database outed  ▁crafted  Website  为学生 ▁CIA  ▁sentiment 批量  ▁pepper  ▁inspir ▁BA 原生  听取了 人而  ▁Monte rov  ▁fraction 开庭"  试验区$  ▁attach&  umbling(  特别好*  但实际上,  ▁besides. 一滴0  时不时2 ▁Г4  只要有6  ▁washing8 公馆:  ▁rhet<  ▁Burg>  ▁comparable@  ▁AlfredB 香菜D MGF  并且在H 紧扣J ▁SuzL  ▁memoirN  关于加强P  危险的R  ▁statsT  ▁prosecutionV  放不下X  一直都是Z  ▁Marcus\  Christian^ 可供` 阻力b  我们现在d  ▁impliedf  ▁bookedh 献礼j  ▁Kashmirl añn  onsoredp  ▁botherr  ▁Reportert  ▁partneredv  ▁cybersecurityx  基本养老保险z  拥有了| 省会~ 树枝 较差 谨防 ▁Enc  相关内容  ▁padding :\ dp 谱写  现代农业 紧身  Template  对所有  ▁licenses 同舟 水流 足足  雷阵雨  资格证 repeat 我妈  ▁presidency  ▁alternate 低者  ▁census ín 主的 置于 前十  ▁structured 侵略  喜欢就  ▁openly  竞争对手 处长 教养  ▁Rebecca  得头破血流  ▁tensions 他却  ▁pole  各级各部门 oi ры сть  卫生部  生物学 字号 建模 Draw  他们都 属蛇  ▁Derby  ▁Location  ▁disclosed  今天我们 良性  ▁Florence  ▁promotional 闪光 uated jz 店面 毒性 Ber  ▁glimpse  ▁autonomous 范畴 ESCO  ▁deleted encia  ▁nicely  不少网友  老党员 gement  ▁sheer 千秋  ▁hier  ▁Session BJ 冷暖  小长假"  中心医院$ 王小&  ▁scholarships( ▁Vit*  ▁html,  ▁Audi.  resource0 呼应2 斑马4 城关6  也只有8 ablish: ▁Pi< 千金> '],@  ▁mobilB  微课堂D 航拍F oltH EXTJ  ▁QuantL  ▁WallaceN  风景和P  我发现R  ▁SimplyT 图表V  ▁hungryX  ▁logicalZ 梦见\ uka^ Stop`  很明显b  ▁announcingd %(f  ▁Innovh  minsterj  Containerl 家有n rezp 了的r ERVt ▁CPv  ▁sheepx  ▁Rushz  恶势力| 柔情~ 眼部 可知 奢华 苦恼 小班 下垂 淮北 ▁Daw  一点都不 photo  ▁Hungary  哈哈哈哈哈哈哈哈  ▁superhero  广告合作 残忍  ▁Hotels Texas зи 点名  容易出现  ▁enters  必须是  ▁predomin 敬意 内防  ▁monetary 科大 钻研 Smart  信息发布  微信平台  communications Left REL  ▁regret  食用油  ▁siblings lesh  看得见  ↑↑↑ Serial onut 语气 上台 一间  ▁upgrades  ▁decorated  ▁Bent vation 手持 Making 处事 bris  市委组织部  都在这里  ▁Garcia  ▁heated 找回 arett 门面 oka 召唤  ▁printer  ▁reasonably  不要在  完全没有  ▁listened  ▁charities  面向社会公开招聘 曙光  ▁adviser  ▁Nobel  那样的  olitics  ▁Mull 兴旺  最好是  ▁permanently  ▁Carn  ▁Modi  ▁Weekend 猕猴"  就不是$  的综合&  idation( 南湖*  ▁Eddie,  美好的一天. 标语0 apor2 向西4 Own6 所述8 不舍:  经济损失<  ▁Suff>  ▁Portuguese@ 爆棚B  ▁WingD  的职业F  大城市H enzaJ  ▁examinesL ▁RAMN  的性格P 搭档R 起源T 逆转V  研究会X  录取分数线Z 乐队\ 过好^  ▁Tools`  ▁declareb 戏曲d 硝酸f  ▁Ugandah 党群j  工伤保险l emian Commonp 出场r  ▁surpasst 自私v  学校和x  考生在z  公交线路|  ▁cloth~ 各式  ▁appliances 很低 石斛 迄今 推销  ▁Powell  电动汽车  ▁Schw  ▁politician  当成了  ▁Dawn ▁ACC anium 英才  ▁behaviors 之光  ▁Weight 告诫 陪护  ▁sacrific  关键是 ilers  ▁mice 人脉  ▁theoretical 克难  的语言  交叉感染 三项 ▁qui  ▁extern 播报 财源 华山  ▁Stef  application 妨碍 acey  ▁accus 往上  浙江大学  具体时间  第一中学 吕梁  的幸福 全书  治疗的 avia idas  ▁diameter 有责 深知 协和  在全市  如果你是  ▁Haven  ▁clinics 相近 揭示 ▁spy  通过网络 初次  ▁Julian 分许 界限  教育考试院 属猴 间隙 狐狸  免疫系统 讨好 拦截 迷信 选聘 ▁SS 不看  春节假期  ▁shoe  ▁preserved  洗干净 degree"  etitive$  ▁farmer& 了她(  ▁Chev*  龙头企业,  ▁boots.  的形象0  中学生2 转到4 子和6  低头弯腰8 redict: inv<  不错的选择> 后在@  的重要组成部分B codD  ▁filtersF  污染物H  ▁tunedJ 世家L  understandN btnP  ▁organizationalR releT  中之重V GrandX ptrZ  College\  ▁positioned^  ▁Workshop` 变迁b noted  ▁aboardf 以南h  职能部门j 的去l  抗日战争n  一点也不p  红十字会r  sessiont  ▁highlightingv  服务有限公司x  ▁plantedz  采取了|  ▁manually~ 我局  的角色 北上 rises  ▁lately  与健康  ▁Stuart  ▁enrich ке  ▁remed 口中 递交  ▁lacking  ▁Basic 白白  ▁Arabic deal ▁Eug built  准备的  ▁pond 压器  ▁Heaven 淮南  ▁Holdings ▁Dow  欢迎关注  中国人的  关键字 GD  澎湃新闻 生钱 itten  ▁dilig  ▁Newsletter 预示 pres 已然 传导 clipse  ▁credentials ENCE  ▁Gand ▁Cli  ▁accent  ognition 衍生 前辈  投资理财 大选 plane  实现美好 一见 了就  ▁similarly 科级  taining stown  日本的 ylon ORS  当下的  漫长的  ▁Imperial  黑名单 集市 穿的 ▁FR 有情  ▁optimistic amo  是一次  ▁cush  跨过这道 ▁anx  ▁searches 万年  意想不到  ▁fifty 势必 CNN  的话语"  ▁wound$ 便可&  ▁Carbon( 开播* ayan, 神圣. 可报0 嫁给2  政府工作报告4  说的是6  ▁Crypt8 潮汕: oner< 为她> ▁%}@  ▁FourthB  ▁excuseD  查看更多F rowserH 加密J 迷惑L  实施意见N  ▁holyP  智者懂得R 出不T  ▁definingV  ▁ForwardX 赔付Z  ▁recession\ 设想^ 集镇`  ▁woodsb 三层d  ▁livestf 对人h  名学生j 前者l 洋洋n  詹姆斯p 奇妙r ▁NSt 太行v 辛辣x  ▁coordinationz  建设工作|  各地的~  业务员  ▁ISIS Cur zn imag  ▁Appeals  副教授 ①　  ▁Productions 觉醒  在地上  ▁Occup  Abstract  为核心的党中央 数码  ▁finishes  ▁shocking olan 这在 两项 erred  ▁steering ▁Nad 躲避  ▁lengthy  ▁jumping  办公软件  数字经济  ▁Repair 共和 抢先  ▁predictions  以前的 WD Bundle puter  等行为 免冠  ▁adop 奶油  Overall jet 赏花  这两天  去年同期  干部群众  着一个  ▁artic  ▁Links 奈何  ▁Foods  ▁Chad 中东  ▁demanded  如果能 以身 Ult  与发展  是不能  的这个  ▁undertaken  ▁celebrations  比特币  ▁Diana  ▁forefront  日中午 很不  ▁Craw 器件 吸气 美妆 下架 名言  远大目标的  ▁hunger 真挚  自己的孩子  地铁站  ▁purple POST 虹桥 吸纳  驻马店  ▁pastor 反之  ▁Transfer 士官 立面  ▁participant 油烟  ▁mechanics idae  关键的门槛  才能跨过这道  ▁strengthening  library Lee  先进集体!  方能高" 了近# ayed$ ▁EL% 党纪& ( !才能跨过这道关键的门槛' 单日(  通行费) 无为* 而有+  ▁territories,  方能伸-  iovascular.  ▁broadband/  最近的0  的一致1  能低者2  ▁Manual3  ▁sweat4  Android5  能屈者6  也是如此7 吊销8  温州市9  ▁Peru: 复利; 都可<  齐心协力=  国家卫生健康委>  ▁intriguing? Header@ 油田A  羽毛球B 侮辱C  ▁spaciousD 离合E  的代表F endyG  实现美好理想和H  远大目标的征途上I 多岁J % 有时需要我们低头弯腰K 4 -实现美好理想和远大目标的征途上L % 有时需要我们昂首阔步M = 6在我们实现美好理想和远大目标的征途上N 乙肝O 风筝P  ▁TommyQ 霸王R  ReferenceS seminT kelU squareV prusW  绿色发展X ItsY creatZ 一寸[ 兵役\  ▁Diet]  ▁Religion^  真的太_ 农机`  ▁startera  ▁helicopterb 后可c  分左右d  ▁Familiese Hopf 宋代g ▁Nash 镇江i ▁jamj  ▁einek  ▁Coronavirusl 味精m  问题是n  ▁seafo 车门p 结晶q 离世r  ▁weekendss  ▁stylt 中也u 网吧v 的吗w  ▁nearestx  的费用y urablez 住所{ Send|  ▁exchanges} Sett~ 穿出  ▁Ster Need 蚊子  ▁recalls  ▁succ 并肩 大姐 ной rugg aque 收紧 警戒 宁区 imen  这样说 讲义 眼看  世界里 考题 ▁Yam  ▁fortune usa  说明了一个道理  ▁slots stick  资源的  ▁Luis  自己的生活 ▁CAR 排在  第十五 artet  arantine 字样 自习  ▁disappeared  引发了  ▁Indianapolis 诚挚 技法 肉的 AX 低至  这么多年 城建  ▁sooner  arently 校招  巴菲特 一模 特邀 珠江 itus 我没 纸币  ▁Lynch  老旧小区改造  ▁trapped illery  ▁Alexand 持平 ный 呼伦  ▁Ahmed  高质量发展的  　—　  ▁socio  ▁Myanmar ruck  ▁imported 素食 配电  ▁Emirates 沉积  全国的 rish  最终解释权  ▁succeeded 月入  不聚集  ▁wellbeing office  ▁healthier Sex  新世纪 畸形  ▁substances 橘子  刑事处罚  突发公共卫生事件  ▁Networks  ▁nerve 可观 ako ansom 限速  ▁Gospel 格兰 灰尘 anic 首饰 vip  ▁stance  ▁referendum  ▁Nova flu  ▁males  ▁binary  的认识  新的一年  本身就是 响起  ▁grief 处级 飙升  不信谣  较多的 预言  ylinder eward 案情 French  ▁Actor  ▁regulators  你知道吗  校园招聘  ▁assessments  ▁ambassador  ▁YOUR 背面 atican phabet  和诱惑  ▁costume ▁Koh  ▁bikes  println 隆平   赞同其   风险等级   ▁topped  Wal  八九  理解和 年货 ▁oh ▁Sor  ▁consistency fang  ▁cope  ▁masc oan  智能制造  Company 西县 bourg 安阳  就不知  的需要  ▁lighter  ▁Herm  找我!  过一个" inas# 跨界$  ▁Geneva%  ▁imagery&  ▁Drop'  的学员(  ▁kits) 六一*  ▁modeling+ 争执,  庆祝建党-  请及时联系. 追捧/  ▁progression0  ▁browse1  ▁Question2 处世3 在某4 大名5 崇拜6  ▁Writers7 当好8 赏析9  ▁clouds:  ▁Ultimate;  全面发展< 嘴唇=  ▁genius> ritis?  ▁circular@  CountryA 血症B  responsC  ▁administeredD  ▁rotationE 天花F  特别提醒G  ▁speciallyH MenuI  ▁IncomeJ  fortunateK  ▁AthensL 晶晶M 迷你N  志愿军O  ﹌﹌﹌﹌P andezQ loginR 读了S ▁MTT  ▁administratorU ouxV 当归W 哲理X : 3这些看来都是生活中再平常不过的小事Y  发展战略Z 立志[ gon\  ▁Charter] orp^  此时的_ 美联`  ▁provesa Shopb 生子c  却说明了一个道理d  ▁Preste 请假f  化学品g ceansh  ▁lakesi 收盘j 昏迷k 网盘l  养殖场m 开票n orieo 果蔬p >&q ▁gelr  ▁agreess pitt Beautu neathv STRw  就是这么x  无论如何y 大树z ilogy{  项目经理| 峰值} 芳华~ Would 但又  ▁machinery 空缺  但没有  ▁Bailey 沉重  Getting  ▁complain 中立  ▁endeav NR  ▁refreshing 生前  不知是 光阴 cx  有这样  我喜欢 Los  ▁nominee  电子邮箱  ▁Bookmark  ▁doubled  ▁committees  ▁Kend ▁Fri  ▁realm  ▁illustrations  ▁patience 时艰 Plan Adam HTML Names  ▁crore 国会 溪县 痊愈 私募 勋章  olecular  新高考 应变  有很大的  ▁verdict  即日起至 元起 ▁FAQ 瓜果  深厚的  文化艺术 换了  ▁oversight ▁Vac 切忌 挽救  ▁ninth  ▁Hack  Central  ▁commander 严惩  ▁Filip 棵树  Private  举行了  ▁Orthodox  自动放弃  多样的  ▁locks 篇幅 oyle  几乎没有  ▁Throughout etti 教务  教育孩子  //////////////// 火花 默契  ▁flip Stack 也得 Site  ▁Away attery  ▁Amanda  Without ▁MC  ▁envelop  看一看  参加活动 circ  会发现  在国际  ▁Client  ▁quarters  设为星标  ▁criticized  交警支队  ▁outrage  里面有什么  ▁literacy ▁RSS 一秒 六安 豆浆 通话  ▁Nazi 同伴  ▁flew  的操作 的古  ▁administrators  ▁Alpha 一厅 我将  市场需求  ▁realise usr 豁达  ▁tires 四五 ité 稀释  iratory  等项目  ▁ridiculous  手中的  成为一名  ▁Township  课后服务 WO  本科学历  一起来看看吧  ▁flooring Enc Yeah price   做生意   统编版   ▁basket   ▁epidemic   另外一个  ▁utilizing ше  　||　 盛开 ▁OH  ▁performer hui  内容包括  有的是  ▁Democracy  不法分子 笑脸  ▁flavors 七个 树叶 多重  ▁Claire 无敌   的生活方式! 载人"  报道称# sure$ apple% 我只& 举个' 二年(  太阳能)  ▁Tall*  ▁classification+  ▁mathematics, 迟到-  ▁lectures.  ▁midnight/ 你去0  ▁headline1 副业2  通过微信3 ( !疫情防控和经济社会发展4 小猪5  ▁Supply6  免费提供7 山顶8  最好不要9 捧着:  感谢您; ▁IRS<  法定代表=  ▁infectious>  给你的?  ▁JSON@  ouncillA  安全教育B  建设单位C  是我们的D GermE  ▁lendingF ioxideG  regularH 躺着I SixJ ▁SoxK 无边L 亚太M  ressiveN initeO 奥特P  应对新冠肺炎疫情Q  ▁blockingR 党籍S  endantsT rivedU 换来V ▁KosW ▁RafX iblesY  什么事Z 愿景[ 重疾\  ▁spac] scri^ xton_ 即止` 受限a 痴呆b Badc  的到来d Jace 组件f 一届g  主办方h  看一下i iottj  ▁anchork 所欲l  ▁evolvem 打下n  ▁Zimbabweo 阶层p  车辆的q Utilsr  ▁renovations 潜能t  深化改革u  ▁importsv 谈谈w  ▁exemplx 专卖y  呼吸困难z  ▁Benefits{  感谢您的| 本色} 即为~  一般人  全心全意  ▁deport  ▁spends  ▁ordering 综上 ▁Duc  ▁Motion 须在  甚至连  ▁Maps 估算 外人 自发  ▁delicate  artifactId  ▁arguably  ▁recognizes KEY  ▁taxpayers 盒子  出入境 二月  ▁Kick  ▁diagnostic ahan  ▁Waters Users  ▁Cher  事故发生 一想  ▁runners  ▁cher 断了 仍未  不住了  在我的 文史  ▁Cricket  ▁criminals ▁RS 大爱 表率  ▁utilized  不值得  ▁Poetry  相比于  到底有多  金融服务  以实际行动 吸附 ▁PT ▁jav 之战 打的 ▁Yu  此次活动 三中 ▁RT  整体的  经纪人  目前已经  ▁Guild orsche listen  ▁donor  ▁planes ▁Dh  ▁recommends  ▁imagined 警情 无视 聘请  ▁compounds Wood 泸州 DAQ  老师和 iating  ▁solic  监察调查 shaped  如何在  了不起 一圈  进行检查  ▁Tags 就将  工程院 批改  年收入 无色  相关要求  多年前 Foot Democr nom  ▁touching 超强 台词 兴起  更多的人  新冠病毒疫苗接种  ▁workout ▁shy 改建 oku jl  具体要求  面试时间  Government  ▁meditation  ▁recognise falls  ▁Memory  enaissance 香油 曲折  国务院办公厅 平顶 ’:  ▁sensitivity 我最  ▁Entrepreneur 走红  新中国成立  碰得头破血流  ▁unfold ▁env  年秋季 utils 胃口  部分文章 温情 阴虚 fx ▁Nar stad 抓拍  cannot   utenant   黑色素  卡车  高了  ▁noon  点左右  ▁amended 搬家  气象局  ▁Zach resses 插件  我们还 病史  本质上  不一致  ▁harness 投机  我们必须 yll >, 吃法   从现在!  防疫措施"  ▁railroad# 大巴$  ▁justify% 银壶&  ▁Lynn'  ▁contributes( 心和)  ▁touches* wheel+ aders,  原来是- 给的.  习近平在/  ▁merger0 李子1 量身2  ▁Canadians3 两面4  ▁Tribune5 ▁Aub6 Final7 师兄8 润肺9  事业单位招聘: 河口;  长大了< ▁ups=  ▁licence> cheng?  ▁Athletic@  ▁RefugeA 心地B  眼前的C PingD  不好意思E ▁MtF  environmentG  能看到H 元抢I 谈论J asmaK 家校L  ▁undergoM 曲靖N 山大O  一不小心P  ▁gripQ  前所未有的R 有爱S 雷电T 吃不U  ▁PalestiniansV USDW 起草X SummerY  ▁allergZ mil[  推进会\  account] 蓝图^  新时代文明实践_  镇政府`  appropriatea 楞严b  ▁freelancec 术语d  中西医结合e  Studentf 多点g totalh  ▁Yangi 果汁j  人民为中心k 谷歌l 高楼m ▁NIn  容不下o 长短p  ▁Revenueq Varir 追寻s 非要t  ▁filmmakeru 仍旧v  小时后w  的模式x  启动仪式y Celebrz  农作物{  分以上|  ▁exhibits} 换成~ 亿人 OND 之日 放生  手续费 来信 Prem  交通便利 一盘 下心  ▁insisted 以其 澄清 并购 ▁Jur  目前为止 dx 有缘 成才  ▁surroundings  ▁Illust  ▁irrit  即可购买  ▁founders  ▁stepping  不同意 ▁cod speak 挑剔 旅途  ▁Wiki  大力发展 venile  ▁Serge 扭转 饥饿  奥克兰  ▁soup Kat 迈出 ▁meg  数字货币 徘徊  为目标  上一句 也无 自首 瓷器 顺着 出厂 考评  京津冀 母校  ▁vivid  ▁filmed 吸取 talk  变得越来越 流调  ▁afterwards 深远 ovak 揭牌 考取  大会上 运城  ▁confusing ▁gru rapper 致以 + $习近平同志为核心的党中央 用在 紧凑 侦办  外包装 ximity  ▁referen 十佳 astian  运动的 utton  ▁Flag  ▁galleries 顺义 forget Gal  经验和  ▁pine  present  ▁prevented 易懂  ▁congressional  是全国 也算 入市 向来 )—  ▁Joan  ▁Hour  ▁Score  ▁advancing  ▁Southwest  ▁securing  Chicago  ▁Brend attr 养胃 十万  与管理 ▁OP  本身的 向北 eric  侥幸心理  ▁Earn  的是什么 周转  差异化  月开始 齐齐  ▁recreation  ▁teammates 锅底  ▁Function  如果想  中的一 耕耘  ▁Mann  ▁lesser  不自在  reading  ▁Coordinator  ▁forthcoming 爱与  会更加 ▁Tak  ▁wondered 而易  在工作中  旗袍  期为   ▁installations   低头的时候   ▁Taking 就拿  ▁Tamil 限购  都喜欢  正月初  看电视  营养物质 标明 ▁TD 盖上  ▁Eight  ▁comparing ▁Ord  ▁ensemble  技术创新  ▁Trinity 秉持 rology  夜叔! Height"  ▁rumors# 喷洒$  大部分人%  工业和信息化&  ▁altogether' 楼道(  ordable)  ▁reset*  社会上+  音乐学院,  在那里-  ▁Lopez.  ▁extensively/ 喉咙0  主题活动1  ▁lifelong2  就有了3  ▁Bears4 并为5 数千6  ▁Oracle7  上当受骗8  ▁Birthday9 china:  ▁Truck; 立项< 保罗= 而为>  目的在于?  ▁Infin@ 不力A  庞大的B  historyC 社工D  优异成绩E  UniversF  ▁needingG RyanH  参与到I  颈椎病J OsK  ▁poorlyL  ▁frustrationM moveN  工作单位O  主任医师P  registerQ  进洞口R  DetailsS  实事求是T 昆山U 五是V 煤气W 受灾X oquY  ▁InfluZ 养肝[  开学后\  ▁premie]  民政局^ assy_ 锅内`  逾期未a  ▁finesb 用最c  切实把d onoe 片刻f  ▁pilotsg  ▁intakeh 去皮i  ▁Notrej 多姿k 扭矩l  房地产市场m  ▁archivesn 序号o  希望你p  ▁Cyclq  老师在r % 奇特的造型充满了神秘s 断裂t  ▁corridu 尽显v 面板w  甚而被x Apy  就会被碰头z  ▁Engineer{  就会被碰头甚而被| 1 *就会被碰头甚而被碰得头破血流} 全村~ roc  ▁Ellen  ▁aiming  假如进洞口 " 假如进洞口时不低头 不漏  风景和奇特的  ▁conditioning  ▁Cann  ▁False  有时还会留下遗憾  ▁inning  风景和奇特的构造  日以来 ▁Roh  ▁fuck  ▁backdrop  好好的  的危险 ▁Hip  ▁Billboard 勃勃 之窗  贫困人口  ▁swift  这就是需要我们  ▁specializes  ▁teenagers opin  ▁awkward  ▁Libya Indust 功倍 endo 玉溪  ▁bitter  ▁Framework Scient 与他 hhhh 不作 agra  ▁outf  ▁sophomore 十条 Girl  本单位  ▁lasted 入校 + $这就是需要我们低头的时候  青春期  Explore 八方  是我的 三类 官僚 留观 留存 搬到  毕业的 Ten ▁cu 船员 一刀 翼翼  两个人的 九分  非常重要的 . '奇特的造型充满了神秘和诱惑 Low 懂事 行使  处理的 年仅  从小就  制度的  ▁Glenn 星辰  消化不良 安利 到校 到头 ▁Raw  银行的 半月 caster  ▁philosoph  假若不 濮阳  ▁implements 爱尔 过错 企图 让其 Jenn  年时间  ▁Hannah 淡定  抗病毒  就不知里面有什么  ▁remot vens 舒展  ▁Calendar  ▁Manufacturing 7 0就不知里面有什么风景和奇特的构造  主要领导 裁员  你若喜欢 actic 忻州 ▁dip 　#  发出的  数量有限 Indian 媒介  知道你  ▁Iraqi  ▁resistant 总值  怎么会  我们分享  假若不进去  performance 建华  ▁upgraded  关于开展  电信诈骗  ▁niche 牛津  ▁Ethics  ▁analyzed 晋中  ▁Pool  是你的 总台  觉得很  理论学习  飞扬  踏上  家教   ▁triggered  rable  ▁apprent  ▁moisture 雨雪  ▁solved  ▁panic  内容和  答案是 骨科 菠萝  ▁tweets  人际关系  备考资料  ▁Rescue 印章 shu  ▁STEM 得罪 小微   控制的!  交通运输部"  )——#  ▁pioneer$ 漂流% 黎明&  ▁informal' 说我(  在你的) 蒸汽* 冒着+  学英语, 转债- 再走. 多达/ 鼓楼0 摸底1 ▁Tag2  每天的3  报考条件4  把这些5  的战略6  市人大常委会7 婚恋8 植入9  ournaments: 南开; 外侧< iones=  ▁divisions> 色斑?  ▁Internal@ 手游A  ▁sketchB  ▁punkC ▁AVD  ▁homeownersE  教科书F  呼伦贝尔G  ▁kidneyH  了一段I  的唯一J  本科专业K  民主党L eatingM  不了的N  ▁meterO largeP  ▁improvesQ 百事R  抗击新冠肺炎疫情S 夜班T  李某某U  ▁IsaacV CrossW LINEX 但对Y  钟南山院士Z 为首[ OTO\  落实到位]  ▁catching^ 城堡_ 大河`  ▁Ethiopiaa  半功倍b  警示教育c 七天d  丝毫不e  合格后f  通过后g  ▁Encyclh 危急i 增高j 金沙k  ▁Sacramentol  ▁tenantsm 评议n 近年o ▁SLp 卡口q  不是为了r  ▁templatess  ▁Appealt  ▁AIDSu 发芽v  ▁supervisionw 帝王x  但是却y  ▁abundancez 地藏{ Width|  ▁constitute} sers~  ▁Clarke  ▁customs  封闭式 Eric  ▁Ellis  Stephen 再加  ▁Armed 俱全 太岁 ▁rab  养老院  的措施 流年 路网 ▁PHP  ▁metals 物料  ▁Feel  日报社  ▁destro  后发现  ▁undoubtedly 标段 缴存  个工作 偏差 ▁kne 小店  ▁permalink 川区  的合作  ▁biography  ▁Exhibition ▁zip  体育馆 拉到  各级党组织  ratulations  ▁autumn  很喜欢  以下罚款 水准 一网 率和  ▁sights  ▁Whats 日均  陪伴着你 闻名 ▁HS  ▁inhabitants Hu 番禺 遇难  信息平台 ід ieu  ▁Holocaust ▁Wu 上次  ▁shifted  实在太  没有办法 ▁Sof  ▁Munich  ▁Premium 冷的  ▁ceremon ▁{@ Cy INK  ▁Rugby  星期日  张某某  现有疑似病例 Van  但还是 城北 轻重  专项斗争 的茶 cas Close 通辽 阳春 ▁Jet  ▁Desert 一堂 上也  官方微博 rella 不忍  ▁sexually 卖了 在世  最后都是  直升机 IAN Ra 胳膊 那份 haus  ▁Recent  在国家  第一条 做实  ▁analyses 家村 ▁Oz 过着 ungen 与我 了然 戒烟 ▁Mol  ▁listeners 独具 micro  stylesheet 功底  ▁massage 率高 含金  ▁Teachers  医务工作者  ▁remod MIN empty  ▁rebuild 贴合 敏捷 Auto each nick 个县 astro 地向 评级   确诊为   ▁Consulting   袁隆平  回购  unct ▁Fo 闪耀 Nor azi  开展的  不在乎 固然  自我的  生活垃圾 举止  成绩单  ▁garbage 考入  听起来 谁的  ▁colored 技工  ▁marry   ▁unanim! urally" 捐献#  很多事情$ olesc%  报考者&  的氛围' Ha(  ▁conclude)  ▁Nicole* phan+  ▁Words, 在即- opa. 别看/ Due0  ▁proprietary1  ▁Pierre2 杂草3 公子4 大富5 驱逐6  channel7  ▁refresh8 stack9  ▁historian: 超多; vance< Price= 号为> 停业? 九龙@ 索取A  职位描述B  riticalC 直面D  ▁failuresE  DiscoverF  ▁CongressionalG 配有H  为广大I  ▁enrollmentJ 真好K  干净的L oticsM  在心里N  专业委员会O  睡不着P MemberQ  ▁CornerR  机关事业单位S  施工单位T dingsU  一般都是V ophobW 我从X  ▁guardsY  ▁metricsZ  ▁forty[  ▁Compar\  ▁Reynolds] ▁Ing^ 高尚_  ▁simulation` 不甘a 昭通b  ▁immensec 逍遥d  报名系统e  不完全f  ▁mindsetg 烘焙h  十几年i  ▁mantj  ▁avoidedk ▁MLBl Rebm 宣言n  ▁outlineo  ▁intentionsp  做出的q  ▁definesr 大盘s  ▁patiot 稳重u wentv  Chapterw  赵丽颖x immingy  闯红灯z 通达{  ▁Ultra|  ▁Americas} 腹胀~ CoV 共克 郭德  王者荣耀  ▁Repl 纯正 圣地  基本信息  ▁collision  可能出现  ▁crude 战队  ▁camping 这话  ▁McCarthy  无私奉献  交通广播 Radio  必需品  ▁colleague 回暖 湖人  ▁consume  世界一流  专业合作社 支柱  ▁satisfying 渣男  请广大 xiang 久久  水平和 渔业 考情 微软  ▁maximize 运会 像素  走进去 大鱼 男篮 牛逼  加快建设  Success 剥夺 omatic omp 小便 就看  ▁Beatles  工作人员的  ▁protections  博览会 抗拒 洗礼 对付  ▁chrom  ▁transgender  ▁counseling 墙壁 湿润  ▁painter 温差 饭碗 ▁Ce 南无  创业板 豪门 hart  ▁Forbes  也只是 在天  比别人 ▁Leo  ▁cleaner  规划设计  着我们  ▁Prec  ▁ambition 丰满  高等院校 Div  每天都在 emi 揭晓  ▁Calgary  工作任务  ▁assembled 疗程  他就是 again Things 想你  分分钟  ▁Journalism ")) olas Imm amel  班子成员 丸子  业内人士  饮食习惯 邀约 走着  总觉得 roots 因病 ▁Bir  ▁Palestine  ▁vulnerability  中国人民银行  ▁transplant  ▁Comic  为主题的 Grid 胃炎 (:  版权和其它  ▁terrain  法律的 茯苓 ▁TN anson 证的  年底前  fighters 而我 ▁kin 人要 参谋 除去  ▁sang 一等 ▁Vir  ▁Collections  ▁tricks DL LOG  ▁Lis   ▁captures  宠爱   王先生   为了避免  ▁vacuum 谋取  ▁wheat  ▁expressions 食谱 应征  申请表  ▁Reid 之类  的品质 dress  ▁kicks 忏悔 潜艇 选出 invest prop  ▁endors  省外!  ▁surviving" 部与#  就够了$  ▁Katie% Queen& ▁xml' 测绘(  中科院)  医学会*  市场价+ Inside,  中国人民解放军- 好礼. 辗转/  质量和0 kle1  internal2  ▁gravity3  ▁shoulders4 inez5  表面的6 怀着7 ENTS8  全流程9 又名: 尽在;  下一代< 毅然=  退休年龄> 制订?  ▁trainer@  ▁RodriguezA  调研员B  ▁yeahC  城市管理D ,「E  的日子里F kswG stepsH 明清I ettesJ  ▁welcomesK 退货L  李先生M 误会N  ▁SteamO 或多P  了一张Q  ▁DiesR  ▁TestingS  ▁defenderT  夏天的U  到朋友圈V  ▁rapperW  ▁femalesX  羽绒服Y 光电Z 滞留[ ▁VI\  有很好的]  纪委书记^ 颠倒_ 安检` 遥远a  个月内b  ▁minimizec  ▁brackd  和一个e  ▁cousinf ▁ecog  officialh 水务i  ▁jokesj 竟是k 干涉l  focusedm 一文n 出土o  ▁privatelyp  消息称q 骑车r  ▁sofas 午后t  五花肉u  ▁considerablyv 多是w rilx  ▁CLASSy  ▁Drewz  ▁summar{ 地价|  ▁nostalg}  最后成了~  ▁salad  都属于  ▁frank  有时间 螃蟹  Russian 键盘 ▁SK 白石  实施办法  ▁socially 快快 IGH  好心情 letion  ▁Param  强对流  ▁Ubuntu  ▁Communities 洛杉 ampire  services 眉毛 岭南  第三届  ▁orchestra 力学  ▁deriv 五六  一种是 性侵 悬赏  老太太  虽然没有  ▁Fleet 税前  ▁Write  电子信息 viol ▁pie fan  我不是  ▁Ther 作废  自己去  ▁Downtown  ▁indicators 定金  现实中 识字 liter  ▁oils  ▁emotionally  在校生 aky iaz  找一个  贡献力量 Active  ▁Arnold 辩论 谁都 始于  方式进行 uren 打动 日被  保健品 政审  ▁neur 循序  ▁behavioral  ▁descriptions 乱象 阿克  ▁luxurious wire  电子产品 WN 的内 沉稳 这本  的国际 umen 窗帘 年被  ▁bubble  ▁debris  这部剧  ▁Crew  ▁inviting  工作上  严肃处理 奠定 过早 aml pine  groupId  年轻时 ▁Lil Player device 修理 rice  ▁Decor 凯旋 易学  ▁quietly  ▁Bomb 苦苦  举办了  ▁consumed  在微信  先锋模范  日夜间  教学视频 提现 不讳 atta  值得我们  岁的时候 France 更显  长春市  ▁civic 孝感  以下有期徒刑  洛杉矶 ardi  ▁mall 量为  认真学习 Diff erton 构图 经商 超员 nis 猴子   ▁pian  上月   ▁Earl  红肿  PDATE  的几率  湿地公园 盖章  ▁Properties  类似于  ▁Discover Mah CPA  地下室  astically  半个小时 楼房 ')) awi  laration  总决赛  我们的生活 filled  uilt!  局副局长" 沙龙#  选调生$ 蚝油% 四海& ▁Va' 伸手(  ▁Chef) payer* 洗脸+  表示感谢, andel- ▁Tow.  但是如果/  ▁dressing0 报刊1  ▁frustrated2 ▁CM3  贵阳市4  刑事案件5 十余6  ▁methodology7  颜色的8  ▁opio9 王牌:  新发展格局; ffield<  otation= FX>  办主任?  的范围@  ▁knowledgeableA  重要意义B  ▁WyomingC  ▁awfulD  programE  ▁demographicF 巢湖G  ▁WinnH 班组I  风格的J ▁spaK  ▁eurosL {%M 化痰N  stanbulO  racticeP 户户Q 蜘蛛R  ▁stapS  邢台市T  长期的U  什么都不V  ▁finalsW  ▁faithfulX 荷叶Y 送货Z  这么做[ ondo\ 吃饱]  交易中心^  团队合作_ 附着`  因为她a  指定的b Modernc  ▁sidewd  小心翼翼e 首富f 灯火g 党性h ▁UTCi  请各位j  ▁protectsk Rangel  素材来源m  ▁medicinesn 拍打o 吉他p  知道的q 费等r ▁Lans  ▁googlet  如涉及作品内容u lossv  制度改革w  及时发现x  ▁Readyy  ▁enhancingz  请及时联系我们{  生命中| 过关} 虫害~  ▁Options  考试中 和县 毁灭  舞台上 差价 雪山 third  ▁rookie 崇尚  早发现 相逢 拉着 收银 们都 缺席 stairs 绝佳 ▁Gem  发热门诊就诊  电子邮件  ▁constraints  党建引领 还算 快点  应注意  深入学习  ▁deceased 停靠 谁是  ▁prescribed 富足 指望  ▁Female 时装 白茶 燕窝 Lic  的妻子  猜你喜欢  ▁Tibet  兰州市 一侧 化身 的张  经济圈  ▁redirect  劳动关系  ▁Eval  总是在  ▁silly WHO  本公告  管理体系  ▁severely 百病  ▁Vehicle  它可以 与日 illes  ▁Conduct fair  给予了  任何时候  ▁Apache  ▁injection  ▁fright  ▁technically EAR 平民  哪还有 ▁kg 玉石 eches 国画  ▁humble  生态保护  ▁liable  ▁sailing  但不能  ▁exceptions Die  可选择 你和  ▁бы 不时  版权属于  ▁rolls  ▁accessing 供述 遗嘱  ▁squee  ▁cens 性地 不理 管线 ▁WWE ombie  版权和其它问题 察觉 tab 退票  重要性  ▁poker  按照国家  报记者 encil 轰轰 照射 热潮  安全和 hole  ▁marking  ▁Legislature  　•　 BIM 认罪  学生返校 几句 ▁sou  年前的  凭什么  ▁lemon 中枢  让别人  ▁altered  共同的 倾诉 联想 Wild Could  ▁Angels éd  我们从  ▁spite  抖音号  新华网 农资  一直到 变换  ▁tops 痛心 或缺  ▁macro  ▁rises  ▁Hispanic  产业的  我就是 kk  ▁dough 种的  ▁Twin  ▁nail 转动  商务部  奢侈品 丽江 加倍 杠杆  ▁accordingly 提速  师范生  警告处分!  征求意见稿" Atl# 亲和$  atching%  国家重点&  工作部署' 学徒( 转至)  不了解*  ▁Behavior+ ▁CE, 大象- 改良. 尿液/  ▁invented0  ▁bloom1 林区2  ▁energ3 冰淇4  打破了5  ▁seized6  ▁Exception7  ▁Sara8 iba9 Mid: 关门; 春雨< 越过= kov> 悬崖? 盘中@  ▁UpdatesA 挺身B 料理C 隔着D energyE  ▁consequenceF  村里的G  ▁suckH 店员I WilJ  也应该K gieneL 加减M 药用N ▁AurO shanP 浪潮Q 衰退R feldS  引导学生T  唐山市U  教师节V  ▁FeatureW  ▁persistentX  ▁legislatureY modelsZ equal[ 伤人\ ▁jaw]  包含了^  ▁cant_ 村村`  ▁Gotha ▁Likb  法律服务c  ▁variantsd 落水e  了一位f  ▁losesg  事半功倍h  ▁sungi 的双j  教学楼k  ▁Solidl  ▁disappointingm  这样一来n  项目管理o  论坛邓州论坛邓州p  ▁стаq  ▁databasesr reacts  ▁Winst  ▁substantiallyu 护栏v 解说w ontinx 锡林y 无反z  战斗机{  进一步提高|  ▁dealers}  ▁hilarious~  ▁Johns 停在 ▁mud 善意  ▁pickup 正直 Effect  ▁kills 大放 顾之  ▁deterior 小板  ▁seas  ▁Enforcement 尽职 Pict  文件精神 安逸  ▁Running 所作  ▁synthetic  ▁Extra  ▁frustrating  不方便  ▁Publications  ▁Almost  ▁residency  ▁exotic  ▁Spotify  每天都会有  的女孩  ▁solidarity  otional  ▁reflecting  添加了  最早的 ▁LLP  interrupt  当然了  体温测量  ▁assumptions  活动轨迹  来得及 Cache 问问 inned  分享给身边的朋友  ▁prototype Sym  中心主任  ▁poses  ▁shutdown  个城市 Give 本题 泥土 稿费  不明白 病痛 auto dong  ▁Angela yx ▁DB  转移到  ▁transmit  以后的  有以下 博弈 aways 逝世  ▁relies  ▁practically  担当作为 用过  猕猴桃  ▁intric 阵阵 ocide  乘人员  武汉大学 供货  同时在 ▁я 杂交  小红书  一起来看 纯洁  第一期  是他们 erseys  ▁architects  激烈的 Living 个村 片的 aja  环境和  研究的  ▁temp 撰稿  ▁Gregory 着实 初二  可以使用 密密 喧嚣  只剩下 肝肾  也表示 三维 感触 娄底  连日来  ▁impose 三名  交警部门 口鼻 的要  ▁Hebrew ▁Saw levard  ▁responsive ixels 祖宗  ▁opposing 后会  ▁empire phot  ▁invites 支教  千万不能  扫描件  最后成了对不起 cache Jason  otechnology 赢家 Bal atz  惊人的 涨停  ▁Tagged Brien  ▁finances 后将  ▁dangers 八一 cology grand  的因素 赶往  十八大  说着就 irie  ▁arguing ém  划重点  明星在农村  有多少句我爱你  ▁Yahoo 请加  含有的!  古老的"  ▁Rece#  自由贸易$  ▁rushed% 碳水&  新发地' ENS(  ▁tongue) jah*  ▁Morrison+ : 3新型冠状病毒感染的肺炎疫情防控工作, 面团- 联名. hb/  工程施工0 odium1  营养成分2  根据自己的3  ▁boundary4  ▁Bark5  ▁pist6 wright7 拜年8  ▁forb9  综合治理:  专门的;  ▁slave<  ▁Introduction=  最后一次> 上行? 轻度@  普通型A  如有侵权请联系B 面目C  招聘求职D  被发现E  政府部门F  事业部G  星期一H  ▁MysteryI 无误J  ▁disruptionK  ▁capacL 冶金M 大三N  ▁intervenO 情境P  ▁comebQ 刊登R 尘埃S  ▁transcriptT  ▁metaU  ▁orbitV 盛大W  志愿者们X  ▁SiliconY  党风廉政Z  ▁Assistance[ '"\ 地为] 后又^  ▁trades_  ▁conform`  ▁Regulationsa  ▁repositoryb  日本人c  问题上d mande  ▁redesf 尽心g 燕子h Peti  郭德纲j 电瓶k  看电影l 夜市m  投递简历n  ▁Fundingo 升华p 将要q  Integerr 不乱s  ▁eligibilityt ▁Migu Dimv  ▁characteristicw  想象力x okiay estyz 切入{  形式主义|  ▁unfortunate} 安娜~  今日头条  ▁Advance 无名 气球  ▁pound  督导检查  专业技能 ▁Zoo  ▁fertil  ▁eliminated 憧憬  ▁Cord  全国文明城市 以北  要积极  由中国  之规定 要闻 hum  的情形 Ash  迈进了  ▁Andrea  脚踏实地  具有较强的 好心  ▁surprises  ▁directing ные  从业者  都市网 Mot rights  广西壮  进行消毒  脸上的 射击  ▁Princeton 正经  变压器  菜单栏  的时间内  ▁Heath  随身携带  ▁transported  生长发育  ▁necessity 总编 房车 牛排 ▁lod  区委书记  医疗费用 背影  ▁infant  的信心 观测 任免  而言之  ▁indicator done emp Route Ron  热水器  高跟鞋 南至 旧址 ▁Hi  ▁Bengal 无害  考试科目  ▁thriving mn 光学  username 斩获 sin kw 请教  ▁Reference zl  ▁thumb 占了 alla 真空  的电话 回想 性与 击败 艺考  ▁wand  国家和地区  依法对 患儿  保持在 侦破 拉动  市纪委监委 zerbai 售票 勾勒 召集  应急救援  识别下方二维码  让孩子们  ▁explicitly 仙游 关停  进行时  求职者 Prim  什么的  ▁merchandise 俗语  mentioned 只在 遇事 退回 ▁pap AMP comb  的那一刻 velt  能有效  新闻热线  也是在  leyball 磁场  的区域 代办  公共资源  ▁limiting  ▁uncle  ▁retained  你必须  也算是  改革发展 Angel  ▁retention  市疾控中心  ▁shelves 悄然 (* Techn  ▁Threat 我看 ifacts  农村地区  ▁Teams  ▁Emmy NU Nic  宣传资料 tu  ▁slower 后就  ▁debates  站起来  ▁Often це 字节  ▁fifteen! 宜春"  教体局#  ▁incons$ 不小% 可想& 胜过'  ▁breeding(  ▁comprises)  去年的* 河水+ ynote,  ▁plaintiff-  ographs.  unicipality/  ▁mush0 寺庙1 面色2 Tax3  ▁tumor4  该男子5 喊话6 ▁Ow7 宣称8 变相9 无不:  教职员工; etc< 水区=  Connell>  进行现场? eking@  攻坚克难A  市场监管部门B ℃;C  诺贝尔D 葱花E  写字楼F  我们不能G  最后都是没关系H 审稿I  都是为了J 打个K 能手L  的英雄M  ▁HairN  都不要O  每日一练P integQ  ▁depositsR  不断完善S  体制机制T abloU  ▁basicsV 粘膜W 清算X 杂质Y uintZ Along[  ufficient\ 欺诈]  自然资源局^ 心头_ 减负`  冰淇淋a  ▁violinb  有条件的c  ▁Bertd 的这e 教版f  的概率g 瓷砖h  growingi ihadj  ▁doubk  是人体l fengm atemn  Resultso  ▁professionallyp 西宁q mutr  生活会s rart  因此在u 进修v  ▁admitsw  网络平台x ▁Ludy guardz Greg{  三门峡| 动静}  ▁auth~  ▁Remove  你应该 清气 肆意 千亿 Way  ▁dire 设法  外科口罩  有多少句对不起 zip  ▁opener  有多少句没关系  ▁detained 也太  ▁Rica  ▁caut  ▁aired  ▁traits  ▁waited  临床医学  作用的  制造商 " 社会主义核心价值观 点位 七大 闪闪 ervoir 实业 aney sb  ▁Character  ▁academics 离校  ▁Shield 罗汉 叛逆 UTH 僵硬 压制  根本上 Impl  ▁Shadow 河中  我都会在这里 内含 铺设 限价  消防车  ▁Concept icky . '以习近平同志为核心的党中央 芳香 文员 Les  月以来 无知 讲好  ▁Butter  尊敬的  提交的  ▁posed  新浪微博 Place  ousands 光彩 学佛 院的  ▁Bund 照样 enges Empty 肿胀 重返  会选择 在海  ▁Raid  ▁Evolution  统一部署 诸如  ▁carp  ▁bure  ▁logging iej 顶部 Paper 行事  对这个  ▁Featured  规章制度  检察长  其他国家 揭开 isible 造福 foo  查看答案  ▁Barnes  广西壮族自治区 Hash  循序渐进  ▁Pine  太重感情 临夏 袜子  controlled habi  ▁slic Clear 一反  开发的 hao >(); uras  ▁Heavy  ▁switching  睡眠质量 一言 桃源 ogenic 沉着  对我们的  都有着  快来看看  李克强  ▁Yankees  工作开展 大唐 火山 性情  ▁drill ན་ 云浮  含有丰富的 拜访  本公司 xian  区教育局 刘晓  ▁updating  ▁enthusiasts  ▁Opening  优秀教师 有位  ▁countryside  中国共产党的  状态下 坐落 领队 (&  ▁Gabriel 折不  ▁Crist  教育基地 % 我都会在这里陪伴着你 东山 序幕  ▁gard  ▁admire 眼下 升值 木材  办公楼!  坚强的" 传闻# ▁Sah$  也十分% blems&  聚集性活动' 法门(  ▁Armstrong) ▁Won*  ▁affirm+ 一新,  不承担-  ▁membr.  市场价格/ 敷衍0 镇的1  ▁Renaissance2 oya3 价位4  连云港5 面相6  ▁overtime7 泄漏8 ▁bis9 浴室:  为客户; 说完<  ▁foam=  ▁Names>  有的时候?  Continue@  生长的A 表决B  ▁StevensC  ▁sincereD 声乐E 身子F  ▁RhodeG 平阴H bolicI  ▁clauseJ 海带K unnL  配备了M  ▁HoodN 尔夫O  也许是P  工作情况Q 诸佛R 真菌S )[T  ▁oliveU 刹那V  那个人W  ▁ConditionX  ▁LucyY  ▁DanishZ 摔倒[  ▁React\ 生根]  ▁diving^ 高雅_  可以先` umpera icatesb  ▁assumingc  ▁assistingd 流产e 自如f  的准备g 快看h 铅笔i 特训j ▁Pamk 欣慰l vectorm Suchn  ▁examso  ▁budgetsp  ▁nutsq  体育锻炼r  癌细胞s 指点t 喜报u  全体党员v  吃什么w  ▁Resolutionx  大概是y  ▁dioxidez elaide{  ▁fictional|  年下半年} 埔寨~ 会用 尼斯  哪怕是 rys  但现在  信息和 ▁Por  ▁Belt  ▁sich  ▁Anyone  thritis 吴亦  ▁horizon  未按规定 Cat Below 汇款  作品的  ▁disagree 职权 ruce  ▁realised  就会接 引流  综合行政执法 东城 ixian  很大程度上 北极 烈烈 ▁dot 海棠 锦旗 Font agar  部分内容  ▁Porter  ▁Aber  command  ▁Palmer 车管 ▁IoT 兼具 整形  落实情况  arettes 广元  ▁pants  ▁intends  ▁fisher  ▁resur  ▁patrol  ▁browsing unks 留住 晋城  ▁Groups  ▁wrestling ▁� 攻克 Genre  ▁Trends  ▁Description 大军  istries 盐水  万达广场  ▁pockets 之本  财政厅  ▁minerals 小鱼 迭代 Energy imated 抽出  任务的  到一个  ▁pier  replace  仪式感 王晓  第一百  ▁proximity 为国  ▁Gibson ISE Bio arta  开着玩笑  ▁centered  ▁Sustainability 机和 gc 上汽  笔试时间 止血  案件正在进一步  ▁relie quin  ▁negotiate  的伟大 robe ▁((  ▁candy  多少人 器具 已达 商用 运维 楼上  dimensional  一定不要  总的来说  ▁utilities  回到家  一起关注吧  ▁proxy  共克时艰 油的  ▁underneath 动动 ▁Pf aying ▁luc  ▁pressed  especially 群的  ▁posters 行测  ▁linking  ▁Thrones 要不  ▁lounge 信箱 hengh  ▁horrible 男方  ▁Posts  被感染 ayette  ▁Wake 洗涤  something  ▁arom 甜品 rators  ▁ви 佩服 Major  强基计划 时隔 ▁Zen 香料  灵活就业 名声  ▁Peninsula 日渐 之源  不管你  才能进去  ▁Jesse  ▁Makes!  ▁commence"  Original# 风华$  自己也% cross&  ▁pools' 南风(  低风险)  但是对于*  ▁probability+  ▁icons, 火力- 摸索.  的商业/ WW0  ▁Reality1 荧光2  ▁Pand3  ▁molecules4  ▁Cort5  这些东西6  ▁async7 战时8  我们所9 Import: 电解;  ▁warnings< 漫天=  ▁expans>  的女人吗? orig@  ▁NatalA ▁CFB  停课不停学C  ▁bannerD inkleE  固定资产F TryG wouldH 火烧I  答疑解惑J  常见问题K  为什么不L 磨损M 宁可N 缺血O  ▁liftingP  ▁instituteQ 兴隆R  DisplayS  ▁rushingT  ributesU ▁înV  ▁MariW 家国X  ▁hardestY 西至Z  小微企业[  一天的\ hind]  ▁smallest^ 逃逸_  厉害了` 车票a  ▁Gangb 又将c 开新d  三分之一e  ▁Lawsf 1 *中华人民共和国道路交通安全法g 平整h  娱乐场所i  ensationj . '分享给身边的朋友一起关注吧k doctorl 地把m  忽略了n  还不是o 辨识p  ▁clayq  Networkr  其次是s  星期五t  消化道u  ▁shitv 红了w  ▁patronx  ▁willingnessy  ▁gospelz  报名条件{  ▁Alumni|  上厕所}  城镇化~ 无故  因为疫情 零星 青椒  ▁Shell  工作经验者优先  上下班 大王 tax 酸性  Republic 安然 adequ eh 人手 赶上  有良好的  uckland  ▁amateur ▁cum 点对 果子 imi  转运至  ▁reper  entieth 想在 拆解 atra  会成为  ▁Posted  ▁Amazing 省直 翻了  我真的  黄帝内经 凉拌  一点的  企业发展  社区居民  ▁cryst  日报道  提前做好 iens DU 参军 一角 复检  饮用水 ▁Rut  你是一个太重感情 iotics 还很  ▁ugly  率达到 快要 regn clock  柬埔寨  ▁cease  ▁restart  ▁remotely 年到  最后说了  等专业 一堆 ▁Pub  ▁Leaf  此文出于  长的太高了 韧性 补习 Ell ▁ub 顾虑 上阵  留言区  自我介绍 加了  ▁twitter  Personal 登山 ▁(*  服务和 的四 ▁pdf  ▁Evil  ▁Patients  reference 侦察 的国 + $你是一个太重感情的女人吗  ▁salmon 腊月 光大  ▁drilling  优美的  长的高 开采  ▁Letters  ▁Singer 女装  后来的  ▁Hass 急于  也有很多  骨质疏松  国民经济 多处 ologic  无反顾 腊八  健康安全  的游戏 似锦  ▁chol 振动 疏忽  ▁Romania  你的心  ▁easiest 杜甫  ▁likelihood task  ▁crowded  ▁lamp Regist  ▁trump 文的 较长 Policy  动物园  ▁enthusiastic 贪污  深入贯彻 凉爽  ▁Bryant 基调 避孕  NASA  aku  三八   市政协   东京奥运会  理工学院  ▁tastes 请看  的皮肤 DN 冬至 已婚  ▁sometime 教书  面上的 ▁Hus  ▁accredited 的非 石材  ▁voltage 胶州 ▁NOW  ▁receipt  市内! нов"  病患者# 跟上$  毫无疑问% 献给& 兽医' 常有( 派驻)  ▁credited*  ▁injust+ Nature,  ▁stranger- 繁花. 沉香/ arse0  说着就低着头1  ▁Camera2  地质灾害3 ät4  ▁де5 ovan6 交织7  命名为8  ▁Specific9 会将:  ▁starred;  ▁guarantees<  ▁commerce= 壮观>  ▁aftermath? 低迷@ 深思A  ▁provingB 三条C  我不知道D 同道E BeyondF 出新G  ▁tortureH 堪比I  ▁weakenJ 助攻K 光影L  点击进入M 超车N  忘记了O  朝阳区P 吃点Q  ▁proudlyR 碳酸S  ▁incentiveT endorU  ▁ChristineV  拎包入住W  ▁sponsorshipX  重要通知Y 水印Z  ▁junk[  医保局\ 昂扬]  External^  ▁Counter_  的全部`  ▁Directorya Doubleb  ▁Reliefc 半价d ogglee  小白理财训练营f 行列g ▁Kunh oleri  ▁Sugarj  吴亦凡k 周密l ▁octm  扫码回复n  责任制o  ▁Terrp  ▁bindq  ▁Sandyr  ▁scrutinys  等方面的t ecraftu  ▁stiffv  ▁Startingw 推向x aaSy 曾是z  ▁tutorial{  过来的|  坚守岗位} 南门~ 重塑  的心灵 isle  尊重原创 iman 归根 法宝  ▁Dent  ▁Sector  ▁namespace  ▁Swan  串门的时候  总会开着玩笑  ▁accompanying 沿江 物件 pu 品位  ▁allocated  ▁craw  新东方 这支 agle  ructive 并用  意见建议 时分  可以去  喜欢夜叔  ▁cycles 水和  你没有  的电影  Question Notes 凉山 长白 鱼类  ▁tiles 变质  网友的  纪律审查和  ▁Simpson  ▁Marriage  第一医院 坏事  ▁Berg  ▁genuinely ▁DA  有兴趣 ▁Ign  自我防护 勤俭  独有的  二氧化碳  办案民警 捍卫  有较强的 ▁Rum  到好处 磨难  第一种 tre  ▁attain 加固 猝死  是由于 硫酸  的两个  面包车  本身就 甲醛  ▁ignor 丰厚  ▁disasters ▁Spa 本轮 不发  ▁respiratory  它就是 核准  课堂上 " 说着就低着头迈进了  ▁coloring 河边  ▁dont 古老  ▁informative 道上  息息相关  ▁Astron 勒索  葡萄糖 在逃 agment 请于  优先考虑  生活费  ▁chase 六合 远景  ▁Delivery 克尔 isbane 名胜 一念 oxy 个好  九年级  ▁paired  等其他  ▁hypothes 的就 ▁Cav 北至 LT ▁OUT  ▁Nutrition  背景图  中国移动  敬请谅解 宏伟 Tony 钉钉 Joseph 承租 之用 ۛ۟ rand Brown  反映了  ▁convinc 填补  分析和  ▁Grande  张先生  退伍军人 海绵  不间断   文职人员   几乎是  作曲  私下   组织和 作答  可能在  ▁Lights ( !说着就低着头迈进了门槛 安家 美金  小编微信  ▁sized 味蕾 itzer  师生员工  Originally  工作站  ▁schedules  ▁thoughtful  打招呼 ishops  董事会   会议要求!  谁家里" 抱团# 租售$  ▁prejud%  ▁regener& 萍乡'  ▁pressures(  他们也)  的音乐*  银行账户+ pict,  ▁Cere- 瑞丽.  成交量/ igator0  违规行为1  往往我母亲2  ▁Nine3  ▁sealed4 阿斯5  一共有6 红酒7  ▁statewide8 的很9 重回: bay; ▁SN< glass=  ▁Loss>  ▁beside? 煎熬@ heticsA BenzB 草坪C  一个问题D  ▁recycledE  我们来F  圆满完成G 德镇H ingeI  规模化J  必然就会K  往往我母亲就会接L 巡回M stedN  ▁gratitudeO  ▁discoveringP  和他的Q нийR  总会开着玩笑说S  基本条件T  抓紧时间U 有道V  的一家W  人民群众的X  高铁站Y  工信部Z  违反了[  Medical\ ▁Typ] 万例^ 弱势_ False`  ▁flowinga  ▁barnb  Channelc 东湖d 洒脱e 必读f  ▁bizarreg 画出h Anthi  ▁Cantj  主题党k geticl  ▁Chartm  ▁bracen  大专及以上学历o 弱的p 豪车q 石墨r 老伴s 落花t 拉黑u 龙岗v  副主任医师w  ▁Vaticanx  强降水y  一时间z  正所谓{ 减龄|  平凡的} throw~ Fast  Additional  在朋友圈  对其真实性负责  ▁Henderson  泉州市 宅家 开胃 水量  ▁Dayton  ▁speculation 前程  事业编制 马甲  中国人民大学  ▁Topics 溢价 降幅  ▁erect ▁*) 名牌  ▁Critical funded CCTV  自然而  ▁Specialist igger 超速 力行 ▁ash  应急管理局 Mex  从一个 素有  rolling 预缴  ▁Heroes  也别扭  ▁switched 炸弹 戒毒  应立即  ▁trademarks  日发布  ▁Display  人员流动 暖暖 触发  品行端正  病毒感染  因疫情 讲授  同学们的  程序员  ▁interference ogical  ▁classics  Storage  困难和  ▁technicians 月子  城市网邓州 百亿 dad 推测  一线城市 班次  表面上  长期以来 缝隙 招人 ▁umb  ▁therapies Cour  ▁Buch  ▁Duty  的脚步  艺术学院  宁波市  ▁Skills  ▁Bluetooth  ▁је Sir  ▁lively 快报  ▁Instit ( !往往我母亲就会接上一句 一曲  党的十九大  糖尿病患者 向后 牙刷 uffy  放假时间 缺氧  分享了 yson ▁Liz  太大的  职业生涯  ▁roller 冬奥  有两种 如来  收获了  再也不用  多样化  这次疫情 偏财 追责  ▁correspondence 领土 AVE 民心  ▁Protect  的各项 ▁Sci  ▁Jets  的指导下 oland  quartered ła 推特 ▁FE copy SY  的祝福  第二种  纪检监察机关 商超 麻麻 1 *这样您就可以继续免费收到最新  有相关 任教  ▁Related 不改  ▁downt  更不要  为避免  湿热  毕节  迷失   ▁prevents   ▁Guinea  人口的  件事情 隐蔽 斑点  ▁withd  一个好 块的 王一 Api  不会再 bassy  的一幕  ▁purely  等单位  学习能力 橙色 袋子 督办   ▁Travis!  ▁supplements"  ________________#  还是批评$  ▁couch%  可适当&  宫颈癌'  的趋势(  轰轰烈烈)  ▁Rural* urbs+  领导职务, ullah- 暴跌.  造成了小/ arcer0 议员1  幽门螺杆菌2 学段3  ▁Cyprus4 背着5 青铜6 瘦肉7  是否存在8 这台9  在一些: LOW; 大臣< hp=  ▁autism>  容不下你? 干劲@ complA  一切苦B  晒太阳C  保持安全D 虚构E ▁AREF  PublisherG 其一H  都觉得I  ▁timberJ  ▁magnitudeK 的来L 升起M ▁AjN 追梦O  妻子的P ializeQ 岳阳R  的吸收S  总会引来T  聘用资格U ImpV  ▁mediW  团队的X  视频会议Y 青云Z  ▁Humanities[ 严控\  ▁ме]  未经授权^ 有志_  恰到好处`  政策和a splitb  长的高了c 山人d 绑架e  ▁clashf 萨克g  ▁glamh  ▁salariesi 泌尿j 刮痧k  ▁comebackl 万方m ▁ramn 他把o 银杏p  toStringq 气色r Tours  ▁quarterlyt  ▁Buzzu 全能v  ificantw 夺冠x 问道y  ▁incomingz  ▁undertake{  率高达|  例确诊病例}  低着头走进去~  ▁Bullet  文明的 得名 " 哪还有低着头走进去  一阵阵笑声 五种 视界  企业名称  别人也跟着  换来的那种  ▁backyard  也不敢  进门时不低头  ▁Apply 1 *哪还有低着头走进去换来的那种  ▁Chat ▁Cer 拖鞋  我们知道 Events  假如堂叔到我家  应对疫情  ▁rooted  avilion  义无反顾 riages . '假如堂叔到我家进门时不低头  ▁coordinate 走得 ▁VP  有责任心  给出的  随着年龄  ▁playground  ▁thankful  小规模  ▁tunes 请以  别人也跟着不自在  与家人  地进行 ▁Cha 愈合  出台了  实施的 表皮 适中 冲泡  ▁prolong  ▁palm odo  文物保护  长的太高了也别扭 会见  应急指挥部  ▁rugby  ▁mentally  碰着头  为了保证  资料显示  ▁counted ▁RC  ▁amendments 演习  美国政府 7 0哪还有低着头走进去换来的那种欢笑 幕后  running 向着 处境 mac 胜任 VEL  不知是表扬 迪士  ▁phases " 总会引来一阵阵笑声  自己造成了小 扛起  ▁Folk  自己造成了小痛苦  icultural amy  ▁Rate  ▁submitting 惠及 一试 上山  不可或缺 理所  ▁Hunting  基本情况  ▁Advoc  必然就会碰着头 ▁uk  俺这门 hran 福特  ▁conclusions 辱骂  ▁Apps  发展新 变态 % 城市网邓州城市网邓州  养成良好的 工龄 肝胆  自然是  ▁defendants " 纪律审查和监察调查  ▁counterparts  是因为你长的高了 七星 虫草  isheries lynn ▁CB  做自己 病害  打开了  ▁Ferguson  ▁Fitness  国庆节  special  ▁verified 重任  因人而 王王 ▁LI 滨河  月上旬  ▁Monica 受访  积极推进  sequently 大到 ▁Bog 讽刺 xls   低着头才能进去  zhi  none  有空  根部  中国梦  ▁Expo  见证了  ▁clips  ▁relates  ▁Might 地市  交汇处  ▁Kerry  给大家推荐  俺这门容不下你 ▁Cob  ▁Religious Sport fac 军民 ▁lag pherd  JSON!  ▁offline"  springframework#  ▁adjustments$  市场营销%  ▁aligned&  主要用于' micron( 会所)  多岁的*  ▁masses+ avi, " 都得低着头才能进去- 名医.  安置房/ Figure0  在整个1  电话联系2  ▁shaping3 走近4 ▁Vic5  到了谁家里6  ▁Tennis7 lings8 风气9 mom:  ▁Jamaica; 亦是<  伊拉克=  元宵节>  ▁exhibited? 百岁@ ▁CVA  希望通过B  ▁mastersC ottestD щеE 破了F  ▁driedG 透了H 其所I cestJ ynastyK  虽然说L  ▁swordM  ▁relyingN  决定的O  ▁prosperityP  仍然是Q 湖州R  ▁hashS  有一句T 定额U 定为V 床头W  FederalX  专业人士Y 木兰Z did[  ▁researching\ 抽调]  ▁brass^ 明目_  张女士`  Matthewa  迪士尼b  ▁Bossc imeterd  新闻发布会上e Songf 典雅g 动着h  可以是i  下面就j  ▁Congok  李女士l 营利m  人来说n  ▁whito  ▁millennp  每天早上q  ▁Yeahr unions " 国务院联防联控机制t 方正u 考量v ysicsw 懒惰x  黑科技y 双节z 提振{  成员单位| 月球} 殿堂~  中老年人 ripp  ▁Mathematics  ▁qualifications (@ ér Georg 通办  标准为  工作时  甚至会 cock 病死  ▁hesitate  自己的人生  ▁neighbourhood  最初的 争先  和他们 esch ▁oak 卡斯 weets 1 *到了谁家里都得低着头才能进去  ▁leap 警车  ▁employs  寒暑假 短发  基本养老金 分段 名副 议论 hma  ▁trap 防腐  的成本 摊位 考务  ▁laundry 测定 plugin  ▁lawsuits  ▁ingredient 者在  ▁harmony  行政村  ▁searched 流淌  ▁gently  ▁Essay ▁VPN  越来越少  ▁branding % 堂叔到我家串门的时候  节日福利 云上 不强 不缺  向我们  ▁Guitar  成品油  ▁Adobe ▁Egg  了一句  inspired 散热  恶性肿瘤 光的  ▁coding  便利店 ▁Gru VAL  ▁maritime  ▁Melissa 紧跟  ▁sparked  ▁Brisbane 是天 招财  建筑设计 Panel  ▁Rolling 晓明  最后说了谢谢你  ▁packs ▁Kur 大开  ▁pile ۙۙ 塞尔 随访  危重症 启航 小李  ▁storms  ▁lingu  不确定性 可使 大跌 遍地  和完善  ▁Boot  从高分到低分 专享 冬日 滋事 威尔  居住在  意义上  ▁tournaments  便民服务 ▁Lac 带的  非常多  方方面  传播更多资讯  attering  时间里 urate  解决问题的 变量  我们先 ▁Mik  粤港澳大湾区 : 3这样您就可以继续免费收到最新文章了  飞行员  改造项目 批判 皮质  ▁personalities ▁Dry  所在社区  ▁Fortune nsic 滋阴  户口簿  对其进行  累计确诊病例   等相关专业  ogo   ▁Haiti   ▁accomplishments   又是一 ovich 狠狠  户口本 也叫  死工资 1 *最后说了谢谢你有多少句我爱你 onal reason  ▁rendered  的还是 yuan icals  ▁Rising  的距离  在做好  银保监会 部级 风机   ▁anonym! 能使"  耶和华# 花花$  ▁tribal% 心经& Ess' Recent(  charged) [:* estamp+  打造成, 愈发-  ▁sunny.  ▁alerts/  ▁violated0 Self1 ▁Sue2  各行各业3 农林4  ▁Advertising5  的粉丝6  绿水青山7 肥皂8  参与了9 字数: 围棋;  追踪到密切接触者< Jess=  不可避免> 早年? osl@  团支部A  新华社记者B  ▁knockedC 小三D  没那么E ISAF  下来了G  要求和H asionI  ▁SitesJ bergerK colmL OFM  即可获得N  这样的话O domainP  ▁ChangQ  ▁ChesterR  ▁VladimirS 肌瘤T  ▁LoganU  上一个V  和武汉W  之间的距离X 足了Y ▁MANZ Wi[ 你都\  activity]  责任的^  ▁Stephanie_  经营的`  ▁deera  ▁costumesb 梦里c 生津d 稍有e 绝缘f  满足了g 我才h 日从i 未尽j  扎实开展k 风寒l hdm  雄安新区n 药剂o  的事儿p ETFq  ▁Fearr 剪刀s 喜讯t  能让人u Broadv 专为w  ▁propagandax  生活网y  ▁Larsz fest{  中小学教师|  ▁Explorer}  措并举~  面膜膏  ▁elaborate 在被 所用 Dear  ▁coordinated  ▁bucket 大雪 HB  ▁singers  tingham  ▁generator  ▁Nevertheless 仅作 ▁cad  ▁Nice doing  是否会  ▁calories XXX  ▁aston 这群  ▁metabol  空气中  预防接种  ▁blessing  副区长  新冠病毒核酸检测 Dream 层级 奋发 音响  ▁Evan 地将 ▁ng  ▁Berry  ▁anthrop bishop (_  取消其 柏芝 投降  ▁Abbey  ▁Hence 抢抓  图片来自网络 和解 ijn  ▁Sexual  ▁transforming 渤海  ▁freight bps 身价  ▁Contest impl g `反反复复反反复复反反复复反反复复反反复复反反复复反反复复反反复复  ▁travelled  osevelt bial  顺利完成 ▁nel  请考生 孔雀  ▁Marathon  用手机  ▁remainder  基础教育 lickr 印记 考的 enberg 收缴  ▁certificates  累计报告确诊病例  ▁dementia  ▁rigorous  ▁systematic 理学 ambia  Settings Future  omedical  并根据  ▁trophy aru ▁sci ▁Fut  再也没有 尽头 消肿 频次 " 传播更多资讯之目的 为先 带入 年味 但其 首相 gang 产出  ▁Bend  ▁verb  ▁Drake ▁Sie  ▁Gear 运费  ▁dancers 组装  科科长  ▁Hoff  ▁Tanz  爱上了 拼团  ASHINGTON 原子 微波 警钟  落实到 ▁MPs  ▁Antar 按下  旅游业 Obs  因为有 不惧 世俗 Pa 晶莹 达州  ▁supporter  ▁ма  ▁rehears  带大家 Employ  ▁Rocky 兴业 华诞 course 受人   理财的   download   ▁cocktail   ▁Andrews   形势下 玩转  ▁Activity 路灯  ▁vegan req  ▁tutor  ▁sequences  ▁Officers  ▁Speech 晚了  第二名 atomy 怀抱  articles 回眸 好后 想行 这下   管理学! jn" 共振# 枝花$  ▁expose%  ▁Whatever& LG' 分院( 单单) 莲子* amente+  ▁Dhabi, Pres-  ▁wonders.  ▁bicycle/ 宛如0  ▁validation1  idering2  ▁Recomm3  左下角4  ▁submissions5 uxe6 Carl7  letters8 EEE9  含金量:  ▁damaging; 鸡腿<  ▁tolerance= 储能> 车载? 在不@  的一系列A 偏向B 分歧C  ▁TalentD 实则E ныхF  ▁MoroccoG PosH 外籍I 敲诈J  ▁ChangesK 担负L  ▁BankingM  ▁whistleN  的销售O 之物P  本文来源Q  看到这里R  ▁curiosityS DIYT ScreenU  ▁plantingV  ▁ThemeW heriffX  ▁CarrollY  asteredZ "));[ cience\ 元宝] 纹理^  越来越高_  受影响`  夫妻俩a 情话b 称作c 绵阳d  ▁Owene  ▁Derekf  交通警察g 初恋h  为群众i 崇高j  ▁nodesk  ▁Gilbertl  Canadianm 视镜n lvo  培养孩子p  接种新冠病毒疫苗q 脚趾r anyas  ▁resembt 黄昏u  为贯彻落实v  ▁uploadedw 百色x  开心的y 灵动z  生产基地{ 为一| 遗体}  这种方式~  社会工作  ▁prosecutor  ♩~　 的下 八路 鹤壁 草本 供销 腾飞 风顺 美股 新化  的时候就  的小朋友 艰巨  点点滴滴  ▁evangel  医疗救治 江阴  技术人员  ▁dialog  ▁phosph zc 中去  不正当  ▁discour ▁​  ▁Canon  icester mente  ♩~　♫  ▁recount  ▁Hilton 做梦 ▁cel 杯子  并做好 就上  清热解毒  进一步完善  ▁Engagement  ▁Azure  ▁pathway  ▁positively 康码 随缘  ▁biodiversity ogram  不开心 民意  ▁Duncan );( 瘫痪  医生的  不规范  意识形态 审视 押金 首创  生命财产安全  祖国的 无助 ZZ ▁LG  的口感 从我  一分钱 洁白  认真负责 ( !全面建设社会主义现代化 手感  的特殊  多一些  的居民 % 我们将在第一时间删除 萨斯  打开手机  ▁Quinn  而我们 用料  坚定信心 打折 dit  　♪　~　  乡村振兴战略 轴距  ▁Walmart debug  质量问题 京剧 探险  ▁Mist 平凉  组组长  ▁Mueller  开除党籍 Items  ▁confirms  却发现  因人而异  ▁Certain  　♪　~　♬ 后背  工作效率  ▁Kurt  ▁Latino 终将  杀菌和 wrap 历练 坚固  保证了  ▁carb 通货 illon  ▁sweeping  ▁rall monton 慷慨 ovies  ▁Bened  ▁telev 易于 毫克  受不了  想行识 ▁coh 军校  劳务派遣 }" )*  ▁disturbing 底盘  事业的  受想行识 数名 ▁Ful 异物 . '高高的堂叔到我家串门的时候   ▁explosive  致死  Var   严重后果  hit  这方面 全州  硝烟的  党和人民  ▁pale  ▁meanwhile 还给  当孩子  ▁Abuse  acterial 财报  在日常 Cond 天门  运输车辆  同龄人  奋斗的  profits   设计和!  最低工资" escope#  好吃的$ ulsion%  ▁Subject& 反转'  赶到现场(  ▁questioning) 首府* ▁upl+  ▁crews,  不及待-  ▁discharge.  他已经/  ▁indul0 乾坤1  ▁Trading2  失信被执行人3 关税4  ▁Lakers5 制片6 供认7 官场8 RP9 找人:  商业用途; ▁Og<  ▁bree=  紧急提醒>  ▁Dale?  ▁forums@ 武昌A 大白B 对症C 暴富D  ▁touchdownsE 河池F 正品G  截止日期H  ▁ComposI  ▁UnlessJ 蝙蝠K ooterL 出境M  召开的N  尚在医学观察O  ▁captivP  ▁circulationQ 牌照R  他没有S  时间是T forestU  ▁delegationV 单调W  两不误X insertY TFZ 园长[ 热销\ 飘扬]  可用于^  第四届_  难忘的` 沈丘a  ▁inflammationb  建立起c  ▁tradersd  ▁cellulare  ▁passportf 亿万g  ▁Windowh  ▁eliminatingi 竞买j  的市民k 湖泊l  ▁unstm  ▁Waldn  ▁housedo ▁ESp isersq 公费r  特岗教师s 雪球t  价格上涨u 有偿v 老少w 名城x  ourcingy 一包z ▁Bin{  ▁capturing| grav} ▁WI~  和经济  投资的 ▁Jin cu  能做到  无论你  要通过  ▁Load  迫不及待 esar 酿酒 堆放 酵母  十二生肖  ▁aggregate 一再  (　)　  Services OWN  最大化  的大门 筹划  管理与 选派 洗头  ▁Brent  ▁Multiple  ▁expectation ▁Ny 进军 kok  的最新 基站 毅力 请将 itably  持续推进  ▁Entry  有这样的  ▁Same  ▁angel Ali  ▁Hitler  ▁Recreation  ▁classrooms 八个 实例  ▁environmentally  适龄儿童  Similar 多做  ▁Traffic 原先 rove 胰腺  ▁Carr  综合执法  一大批  和法律 像个  的程度  ▁Files 短片 Again rians antha  就好像  ▁renewal  ▁instinct 充当  说明了 mode  ▁Instr Jes  等疾病  ▁Paint 出错  教师资格考试  实施细则 胃部  ▁Brett  学生家长 Michel 晒出 atable  ▁Gross  ▁Frederick apest  ▁Greens  第一人民医院  ▁shooter 名列 输液 ocy 榴莲  ▁Thing 港湾  所在单位 咀嚼 热衷  ▁python  ▁cares 解封 ��  ▁tonnes 体态 或少 分付  的形成  法定代表人  ▁Cards 仅剩 Photos  ▁foremost  ▁authentication 美美 ifa 飞沫 站着 rops  ▁abol  ▁bytes  ▁suspicious 朝夕 凹陷  未取得  前面的  有保障 提议  ▁metro  的场所  在世界 要先  ▁successes  最主要  毫不犹豫 万只 GMT ▁NZ   范围的  ANCE  联席  变色   也同样  ▁creature  劳动节  召开了  Reuters  房产证 心梗 按揭 他不 胞胎  这么大  ▁Piano  ▁Novel 山地  ▁nutrients  Recently 是她  ▁Muhammad LPR  三江!  ▁arising" 张大# 食指$ ▁gad%  required& 粉碎'  明确要求(  ▁scripts) 尽了*  国资委+  工作进行,  数据来源-  ▁tricky. alpha/ 发愁0 写给1 价钱2  ▁parse3 你没4 拍客5  心血管疾病6  治安管理处罚7 ARC8 民兵9 海事: 邵阳;  阿拉伯< 选中= 守土>  ▁bush? 坦言@ WowA  ▁wiringB  崭新的C  classesD  认为自己E  berriesF  ▁ecologicalG 灾区H  的公众号I  ▁libertyJ ICAK 顺手L  看看吧M  ▁AttackN  发生过O 流逝P  情况后Q  ▁CarolineR  每小时S  LocatedT  ▁ProcedU 电费V  ▁CharityW  ▁terrificX  ▁permissionsY rophicZ  多余的[ ▁lav\  ▁Wave] 加群^  在微博_ 重逢`  中共中央政治局a  所有权b  是一座c Boardd 八角e  接地气f  的手机g ▁Folh 同级i 咸阳j 江河k 重阳l  可想而知m 错时n 生怕o 高涨p killq 寒潮r  ▁bugss  abularyt  部分的u  每一步v iratew 照料x  ▁canceledy  人的生活z 吃着{ 譬如|  ▁slice}  amiliar~ ▁ONE 体征 吾尔  更方便  ▁biom  ▁varies mus  分会场  ▁concentrated abi  不及时 би  总结了 后被  ▁neighboring 算了  ▁assignments 虚心 一针  ▁expressing  中山大学 风扇 战绩 一品 Staff  ▁consultants 源泉  上热搜 路站 ▁${ 记分 颇有 做工  欢迎您  ▁shades  ▁concentrate  ertation  ▁inception 茉莉 sheet  接触过  严格控制  这样可以  给大家分享  ▁Opinion 文具  治理能力  ▁Midwest 原著 去向 噪声 无明 rosso  htaking  一旦发现  Technology  特别注意  ▁portions wild  ▁Lawyer  ▁Communist 共创  ▁са Anyone 方言  名副其实 蜕变  精彩推荐  有必要  ▁Chain  ▁potatoes  标志性  的场景  ▁Reagan  很适合  每天都会 主权 腰痛  有效性  行业发展  ▁lith  ▁troubled 编者 则会 坦诚  ▁excav 晋级 日零  中等职业学校  张柏芝 角质  ▁propose  ▁Burns  现有确诊病例 ▁Ker  ▁statutory 关口 顶层  ▁embraced COMM 明示 中来  ▁slides 协定  ▁Advisor  ▁invention  ▁distancing amped 认购  之以恒  ▁Hern 省心 ▁Lar PY  这个词  ▁Methodist  相较于 受阻 ▁orn 养犬 最先  小提琴 山林  ▁zoom  ▁dense  ▁demonstrations 包住 我了 嗓子 约有 ETH  ▁creep  有阵雨  央视财经 县市 年头 每隔  ▁Embassy 酱香  isy   ▁identifies  秋风  累累   真正做到  ▁drummer  ▁Views 飞翔  老干部 ▁DIY 科幻 螺旋  符合条件 ▁Syl  ▁Sach 布施  ▁Injury  六个月  ▁Scheme 只用  促进了 无事  ▁laying  补气! 信宜" 减灾#  ▁wreck$ 燕麦% 高贵&  帆风顺'  ivities(  ▁Adrian) 唐诗*  ▁Moving+  艺术的,  ▁worries-  是孩子.  开展工作/  ▁Points0  你一定要1  源于网络2  ▁Guidelines3 特地4 Enjoy5 GN6 小品7  ▁additions8  和世界9  极高的:  ▁fingerprint; 更深<  和方法=  及答案> MORE? ▁Das@  ▁WestminsterA șiB  ▁lanesC 朱元D  ▁CookiesE 闭幕F  本项目G  介绍一下H 统招I  而导致J  各高校K itanL  文章内容M 三合N ▁FYO  持之以恒P ▁MYQ 着想R  ▁uniquelyS ▁ЛT 已知U  白衣天使V  changedW jcX  电话号码Y  准确的Z 华府[ 相生\  ▁Elliott] 二〇^ athy_ % 观点和对其真实性负责` 部委a 重在b IPSc  会比较d 停下e TBf Feedg antish 利民i Storyj  ▁Engineersk 练字l  重中之重m  ▁infamousn  须提供o  住宿费p \\\\q 红河r 你你s 最能t  与大家u  款车型v Bronw  ▁Noahx 舒畅y Henryz  ▁Summary{ 西汉| zhang} 浮躁~  自己喜欢的 jb  和文化 但凡 园地  ▁Dust 悲观  ▁Sussex Tel  的自己  ▁Cornell :[ 笑语  为导向  ▁filmmakers ③　 igm  其行为  网信办 rium  ▁Doll 本营  ▁considerations adas  我们分享此文出于  ▁competent  ▁callback  ▁pork  ▁Fifth 朴素 Dir 粮油 尽致 柚子 红烧  处理好  认真贯彻落实  ▁visually 越发  的气质  platform Think pless  ▁Pitt FORM 探亲  周围的  每人每月  ▁ours  具有一定的  ▁photographers 结缘  ▁Wife  ▁beverage 慌张 童心  ▁verification  这首歌 之风 坏死  ▁concess 旺季 安神 bool  ▁hedge  ▁horizontal 缘故 金奖  的日常 趋于  ▁Offer opsis 武夷 老王 接诊  ▁plea  维吾尔 onies  更不能 一抹  ▁Rating  ▁Walsh  ▁Raymond ▁DIS 阶梯 这起 辩证 口径 chang 棚户 供图 atu  登记的  强对流天气  imation 中控 」　  在我们的 一月 再好 idian  ▁crying  请您放心关注 向我 青龙 yster  ▁seiz  的开始  免费下载 撞击 挑衅  感谢大家的 ercise  ▁towers  燃放烟花爆竹  不得以  ▁pause  Learning  姐妹们  ▁unle  ▁villain 尚有 Sky 三家  ▁desires 人称  ▁refined  ▁Producer  注明来源  ▁Dictionary  技术与  身份证件 固执  Example  ▁Nissan  ▁associates 哭泣  ▁Listen  ourneys   耐心等待   购买了   角度看   ▁Vern  吃肉 迎宾  ▁Problem 以最  ▁collectively 垂钓 畏惧  各自的  是世界  ▁professors  然后在  立即删除 卑微 лен  ▁secular  ▁Cliff ▁WR  新基建  teaching   ▁Electronics!  ▁Casey" Royal# 求购$ 过长% 在京& 结语'  ▁Municipal( 行善) 发黄* Helper+  防范意识,  ▁rhetoric- 趣事. ENG/  中小微企业0 极简1  就是说2 韶华3  中药材4  公务员录用5  ▁yacht6 在场7  ▁spine8  ▁pledge9 ugin: Study;  核心区<  而现在= 心声>  的色彩? L E▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁@  ▁ChiefsA  ▁MohammedB  ▁PetersburgC  ▁impeachmentD 身旁E 十点F 帐篷G 海的H RESI vidJ  自然资源和规划K abolicL  ▁verseM 接警N  PropertiesO 二层P 应激Q 埋怨R emadeS 聚众T 自立U GermanV 紧绷W 毗邻X  ▁garlicY 学了Z 千人[ ▁AW\ 摘要]  ulously^ 衬托_  事业单位工作人员` 昌县a  ▁virusesb cesterc  大大的d  ▁optimizatione  Floridaf  为做好g ▁:)h 光绪i ajij  ▁Istanbulk  建筑面积约l 惊叹m  第十六n 出头o 好用p 谦虚q geonr headss  我们不t 纤细u  Subjectv @ 9习近平新时代中国特色社会主义思想为指导w  扫黑除恶专项斗争x 七彩y mockz  违法犯罪行为{  赣州市| 想让} 可不~  ▁examining  都有哪些 地利  ▁Ride  ▁sellers 试探 Fred  ▁Cuban  ▁dividend icz ▁GE umi gio ▁np  就可能  生孩子 偏远 itars  ▁palace 敬老 连载 ▁Jed  ▁listener  ▁Vista  哪一个  养老服务  向大家  成就感  的地区 Stock  ▁excluded  ▁unrest 情歌  坚持下去 Az Too 医美 levant 欣喜  ▁Chin  是这个 才算 ointed achine idates NEWS 水温 蜀黍 心酸  Additionally 搜救 Num  ▁dubbed 顺心 xxx  ▁organizers 大贵 晚饭  riction 阔腿  等行业  成长为  Operation  Repository 地毯  ▁Encyclopedia  重点地区  ▁Solomon IFI  ▁resigned 多条 司长  ▁induct  ▁lenses  防溺水 events 也给 利尿 短裤  报名人员  还是非常  回头看  ▁Richardson 落细 田野 ▁pi 必背  ▁procurement  我开始 indy  ▁editions  ▁Canal  今天上午 indust 黑人  ▁Built  至少有 hari  ▁READ 轻薄 停滞  退役士兵  ▁scenery gex 将由  ▁Knights TN gone 团的  ▁drones 义词  党的建设  行政审批  ▁Apollo  如果孩子  ▁Ferrari 营口  积极探索  人才的  ▁panc 贝壳  资格的 听证  ▁Curtis  ▁undertaking 保费 糖果  与孩子  plementation ructor  ▁breathe 诗书  是一样的  ▁defended  　——　 文山 生字 ellen anding   ▁Ivan   ▁pipes   先锋模范作用   ▁prepares  证照  是一条 量产  ▁encounters 不以 aines 利器  ▁Agricultural 的头 侵袭  ▁Zoom  ▁combinations  部主任 晨报 放眼  ▁logged  德尔塔 GH 问了  editor! 文摘"  ▁pediatric#  建筑业$  老司机%  ▁UNESCO& 是啊' 本校(  ▁Healthy)  ▁Activities* horn+  ▁laughing, 一盒- 蜂王.  不构成/  的宣传0  能让你1  ▁Solution2 她会3  ▁happily4 野心5 养身6  ▁Recently7  ▁Fell8 逃跑9  ▁notify:  不见不散; 深挖<  婚姻登记= 紧邻>  ▁fellowship? Louis@  ▁dockA  ▁tendencyB  ▁ConstantC zhaD  外面的E  经过了F arasG  NothingH  白皮书I believJ  ▁placeholderK 放飞L  ▁SisterM  LibraryN  多万元O  中国工程院P 却有Q  辛苦了R NINGS  ▁ExpectT  必不可少的U 的画V  只知道W  ▁NikeX 中层Y ▁tilZ  广大党员[  ▁Heather\ 家在]  ▁Rochester^  ▁activated_  楼面价`  女生的a  进展情况b  的法律c 前世d 年产e  ▁Flatf  forgettableg Safh  ▁optimizei 除螨j  通知要求k  ▁Sensl  ▁Shirm  ▁gatesn 跟风o  优惠价p  基本功q  小区内r 骆驼s mott  ▁nachu  ▁Columnv 征兆w 福德x  ▁contemply  分之一z 抬起{ 新安| ACC} 并有~ 房的  ▁marvel  ▁Panama Iss 神农  锂电池  ▁Timothy 老爷  变化的  志刚哥  legraph 算命 &# 仰望  了一大  ▁conceptual  责任担当 quez  随处可见  ▁regulate  ▁nucle  ▁bureauc 自考 高明  release  ▁Barr  以上就是 LV 传言  ▁Punjab  ▁republic 极佳 癫痫  安排在 ▁mum 走好 金秋 如愿  ▁Cowboys 张小  取消了 项链  你现在 cin  实景图  齐齐哈尔 敲门 圆梦  ▁jointly 大招  结构性 ▁Ki itat 往生 昆仑  网站上  ▁overlooked 用书 胃病 中被 pta cano ighed 未曾  ▁bundle  帮助我们 余个  ▁volcan UA 自费  文化课 ери  副省长  Perform  ▁revenge  ▁obesity 出彩 ▁psy 意料 does odi six 扣押  ▁admissions rowth 一目  法学院 `,  olphins 不和 免试 Jud  ▁tribe  中心组  ▁Martha 进店 开源 ▁Fig  ▁sadd 仿真 重伤 Age  家里有 分层 次于 三甲  分钟的 帮他  ▁vibe  personal  ▁demonstrating 四人  的员工  存在于  ▁Polic  ▁balcon  更应该  政法大学 infect  ▁damn  云计算 ▁Zel 闻到  olutions  ▁culinary 写得  但是他们  ▁Sask  鲜明的 晚舟 索要 Hill 闪电 评判 Quick  来这里 侧重  ▁Sharon 人说   Official  配角   ▁endangered  手腕  ▁° 昆虫  ▁portraits  ▁identities 动工  ▁Commander 扭曲 Ever 年夜  ▁Madd 可为 进城  根据国家  研究和 航运  自然而然 电厂 ▁db  淋漓尽致  不放! ▁NH"  不合格的#  ▁lottery$  ▁contributors% cq& 完结' 得过( 大院)  ▁tariffs*  感染了+  ▁Visa, 率的-  ▁theatrical.  出问题/ 补钙0  ▁Especially1  各级党委2 细微3 庞大4 机型5 爱意6 cats7  ▁crashes8 Tab9 圆圆: 楷书; : 3我们分享此文出于传播更多资讯之目的< 磅礴=  和健康>  皮肤的? 皮的@ 海区A 怎能B  在一次C  村民们D 何种E  因为没有F 赚了G  综合评价H 还说I 重来J  保定市K nameseL  inologyM  ▁FacilityN  ▁SuperiorO ioletP  的时光Q  ▁WadeR 形体S 南非T  保护自己U ▁thyV 就让W 余家X  浓厚的Y  的情感Z  ▁smoothly[ UNT\  上的讲话]  ▁Winner^ 海水_ 禅师` 纳斯a  家长和b  有一颗c  ▁Proceedd  ▁championshipse NASDAQf  ▁Boulevardg 接听h  多措并举i 日元j )　(k  ▁betrayl  ▁Opportunitym  你也可以n 瑞典o  空白处p  ▁Troyq 听得r 养花s 柑橘t 沙雕u  聚集性疫情v  authorizedw 之夜x 特权y 跟她z 套票{ 年版|  所能及} 发自~ 神州  生态环境局 穿戴 堂食 Loc 回向  理事长  ▁acquisitions 武威 饰品 人已 大红  ▁Parts  ▁Identity ▁OFF  省委常委  ▁Perth  ▁Huawei 霸道  前一天  可怕的 亿的  ▁Martinez 芝士  相关情况 捆绑 搭乘  有一份  现实的 剑桥  十足的 osta ▁Bee  多样性 ▁UFC  北师大  ▁generates  中心城市 着他 报了 将与 莱斯  一下吧  ▁Understanding  传染病防治  ▁mines duce  ▁turt 耗材  了一件 玛丽  决战决胜  时发现 ▁Fla  安全事故 红茶 礼金  和精神 每户  一年四季  社会实践 Cook  contact  洛阳市 半身  奥斯卡  指导老师 dan ▁É 对新  单纯的  ▁stair  ▁Pharmaceutical  ▁foul  ▁Dental  外国语学校 其后  案件的  ▁bombs 只因  张老师  ▁Allah 值钱 都与  ▁Meghan  ▁Porsche ▁HTT  体现出 悬浮 空的  ▁Chancellor ▁lou  ▁Courtesy 富士 剧组 大悟  ▁Material ignt  一女子  ▁sticks  ▁Insider rivers  ▁Elections 稳固 Perm amples 接手 林某 ucus  打印准考证  ▁simpler  ▁Aboriginal  ▁observ  最大程度  ▁dissemin 结识 闲暇 全勤 得的 金星 树上 anye ▁Und  要充分 andro  等场所 kens 深渊  会遇到  的实际  ▁coupon  还能够 灵敏  全力保障  ▁portrayed  的高级  个工作日内  cussion 竞价  立秋   协调能力  班会  igi  辞退 吃瓜 客家  相对来说  ▁crashed  ▁precip 仙境 拌匀  ▁Sheffield  ▁kilometres  处理器 ▁Cel wi 带回  有帮助 之名 法语 ▁NW tails   ▁announcements!  ▁provinces"  管理法#  感觉自己$  ▁learners% deep&  毕业生就业'  是美国(  ▁Griffin) Commun*  砥砺前行+ 觉悟, DER-  Adapter.  ▁briefing/  少年的0 靶向1 气虚2  从源头3  ▁Moss4  将按照5  请提前6  atorial7  ▁relevance8  的灵魂9 ▁gly:  帮助他们; verter<  ▁giants=  ▁Wait>  ritional?  成交价@  ▁WheelA  政策措施B  还包括C 涨了D  深度融合E 鸡精F JDG ▁УH ursorI " 打赢疫情防控阻击战J  还存在K  ▁gaugeL  ▁wristM sfN LMO  ▁fatigueP 生生Q  朱元璋R 时就S ivierT ricanU  ▁absurdV  ▁CafeW IncludX 抗洪Y 需在Z  人才引进[ 格林\ boat] . '赞同其观点和对其真实性负责^  散发着_ 楼栋`  名额有限a  在全省b  Registerc  孟晚舟d  ▁Protocole 伸展f 朴实g  中山市h  ▁seamlessi 不解j  的大力支持k 出名l Deathm ignantn  ▁escapedo 住的p ▁CPUq  有很大r  重要提醒s  ▁Croatiat ▁MNu  坚实基础v  ▁opposew 削弱x 厂区y 来稿z ▁Lak{ GF|  ipeline} 常熟~  ▁vide  容易导致 拍了 ▁PE  有钱人 可期  ▁Louisville  记者在 ▁Rus  在线教育 倒闭  有一条 AAAA  ▁implies 液化  价格的  ▁choir  诉讼法  安全风险 ▁Kas  交流学习 driver  持续关注 端庄  ▁treaty  ▁insects Third 这与 dot oche  ▁Outdoor  代表性 rolet  ▁uncover  ▁Actually 绚丽  有那么  切实增强  ▁Alternative  ▁doubles  ▁relied 评比 论语  第十七 zhen  ▁advertisers  医用外科口罩  ▁adjustment  不相同 寒气  日消息  死亡人数  Available  工业园  ▁closet arez oyu  这其中  ▁vegetable 一汽 如若  ▁ancestors  都市圈  和力量  对企业  ▁statute 示例 ('#  和时间  ▁promptly ▁VIP  ▁cups  爱你的  ▁protagonist  仔细阅读  ▁reproductive 到岗  ▁Treasure 反响  ributed  的小区  ▁Billion 兼容 书包 Ro Rose 演变  自然科学  ▁victories ERY  ▁TODAY ▁CSS  次全国 ▁Cul 窝点 苦瓜  大拇指  ▁metric  深入了解  ▁detective 回乡  ▁Excel 悠闲 麦克  内容文字 起居  iencies  重点关注  左右滑动查看更多 pages 追赶 enary  矛盾纠纷  ▁homework  的了解 容颜  第二期  ▁recovering 和非 INA ▁Ran  ▁ст  据报道  生活质量 忽悠 nl 生化 grass 拍戏  你身边  建党百年  eteenth 二轮 创城 青蛙  ▁parliamentary 市域 Ol 牌子   重要指示精神   ▁illustration   ▁yields  ▁CI  FONT 乖乖 流向  ▁Cambodia vc 由来 设在  责任和 结对 得一  ▁Evidence 犹太 :　(  ▁felony  步行街  集团的  ▁Aware  我们党  并不意味着  南宋! 但要" 天桥# marine$ 之分%  主义的& random'  景德镇(  ▁wallet)  两个字*  ▁additionally+  航空航天,  ▁Mode-  提高到.  供认不讳/ 成龙0 摸清1 这根2 微观3  ▁Salvador4  没时间5  ▁eleven6  ▁marathon7  ▁pedestrian8 通通9 锅炉: " 中华人民共和国刑法; Single< 进货=  进行一次>  ▁Roosevelt?  ▁threads@  ▁promotionsA CBDB  成立的C ▁taD  为他人E  ▁pharmacyF 双双G  代理人H  ▁acceptsI 额头J 身在K  ▁FurnitureL  创业者M  ▁caregN 粗暴O wigP  因为是Q 嘴上R  的一半S  ▁CarsonT  ▁intactU  ▁kindnessV 仙人W 普京X скаY ▁vicZ  ▁Lithuan[  ▁vigil\ headed] MIT^ erpt_  тель`  小男孩a 绿的b 该院c  ▁comediand  军人的e  不同的是f vertedg 体量h 金字i  ▁risenj  ▁wholesalek Davel  ▁overturnm 飞跃n  ▁roundedo itoredp  ▁neglectq 转给r  ▁premises 收好t arisu 侵入v  ▁overtw ▁Bilx  ▁lacksy 告白z 花期{ 补录|  ▁blown}  物业费~  现代人  ▁Chronicle oulder 出卖  的天气  ▁credibility 求学  长大的  uations  ▁lymph Govern  ▁immunity 装的 车展 轻信 ▁USC 排斥  全媒体记者  ▁Kazakh  ▁axis 天数 就回  国家监委 Heart 桃子  ▁Connection 且在 刀具  ▁Jill clean 因缘 脉络 药学  ▁Jenkins 主页 伤残 以东  的画面  ▁Methods  干净整洁  ▁Dating  🏻🏻  ▁prediction  ▁Require  累计治愈出院病例 八年  八项规定 捕捞  产业集群 SEC ▁irr  长时间的  ▁hugely 大宗 盛夏  management  胃肠道  writers  ▁heels  ▁Christie  小助手  设计与  ▁suprem  ▁evaluating region HOW 情趣 ego logger ▁Hav venth 帅哥  ▁bored  connection  ▁successor RY 他从  是怎样 点了  茅台酒 ▁Jak 村子 ▁Rid 殡葬  ▁Louise  出不穷 淡然  接触到  供大家 ervlet  ▁Belarus 舒心 银耳 Made pick ×× 先用  抢救无效  ▁seminar 贬值 图网  自由基 rera 不孕 率领 行长 gae 掉的 好坏 entity 接班 此生  可查看 易燃  ▁drafted  ▁пре  ▁Gren  ▁analog  ▁Pentagon  ▁missiles 东县 indsay 这句  的姿势  人心的  自己做  全面加强 atche  ▁Measure 红润  ▁Dogs  ▁factories 私立 纵深 Delta  后方可  和美国  ▁automobile 反面  ▁Hil   大使馆   最常见的   ▁glac  衷心  即可获取  扫码添加 lg  说出来  ский 湘西  ▁planets  ▁attracting 到户 日为 避险 sort  身份信息  ▁todd  几年前  ▁Cream 教委  学历证书   ▁excerpt!  中指出" 女婿#  原创文章$  继续教育%  ▁Conn&  是国内' 橡皮(  关系到) ▁Gor*  ▁sounded+  ▁Infect,  ▁marker-  ▁boarding.  老师说/  若有侵权0  ▁Ballet1  可申请2 完好3  ▁reservation4  是多么5  改革创新6  方舱医院7  方法和8  银行业9  ▁correction:  ▁sexuality; Files< 烘干=  ▁urging>  ▁dawn?  ▁Shipping@ ChemA  ▁concludesB  的风格C  ▁BrightonD 人多E  浓浓的F 刑警G  文化馆H  基本都是I  工作了J 到账K 衡阳L RhM 大类N  ▁marginsO  实在是太P ▁ArgQ  ▁ConfederR  常通风S 就该T 建的U ▁GLV ▁LivW  ▁disappointmentX 本站Y  ▁ShotZ  ▁Claims[ Ari\  legiate]  ▁organs^  ▁astronaut_  ▁predecessor` iffea 青菜b ~~~~c  隐患排查d  ▁warnse  ▁riderf 桌面g ▁Nuh  ▁boxingi intersj CDCk  ▁patentsl  创建工作m  妈妈们n 人头o 兴安p naiq  ▁systemicr  ▁Effectss 不带t  ▁Pricesu  可能会有v 例行w  把握好x  ▁pharmacy 群防z  Researchers{  ▁notifications| 暴躁}  海淀区~ 嫩的  通过的  发表了  ▁anten  ▁tribes icism  ▁campuses  本文由  ▁queue " 工作领导小组办公室  初中生 ▁Reb  ▁turnover  希望能够 Cert 油气  ▁Beer  ▁Devil  chester  ▁gifted  一反三  碳中和  ▁Pione 定居 编造 飘香  ▁Mong  ▁annoying 无从  ▁Eyes  层出不穷  ▁aerial  ▁Otherwise 已成  微信支付  ▁fixtures  换句话说  ▁perfection 二驴 lag 骨髓  ▁hearings  不注意  多大的  等环节  ▁Adelaide 争论 逆境 大灯  问题导向  到来的 三方 入库 各镇 分量 信誉 ▁RF  ▁stairs  提醒广大  ▁Outside  ▁Governance 四位 日落 酰胺 Args 修护 到老  行动计划  ▁dorm  ▁Northeast  有关单位 美图 mun ▁(' 多肉 报料 椅子  ▁cleaned 紫金  可以达到  印发了 盲区 铜川 防暑 ;④ Hor lot ENSE  长效机制 之二 职员  棚户区 晋江  ▁Investigation rowave  ▁efficacy raz otyp  ▁dresses 忠实  千万人千万人  ▁Traditional  ▁Belf 脾虚 危及 一图  ▁logs  ellington 墨西  Australian  关注微信公众号  ▁Blair  学习强国 + $灵异吧灵异吧灵异吧灵异吧 Focus Season 柿子 Tre  ▁helm  ▁divine 每逢 Dead  民警在  洗发水 鲤鱼  ▁composite  军队文职 而你  几十年的  ▁invaluable onduct  ▁overwhelmed  小事也是这样  ▁sculptures  特定的 摆摊 起身 Emer  ▁Performing  ansing   隔离医学观察  新盘  朝廷  科比 CES 老爸  ▁Cult  ▁martial 大户 满脸  ▁gloss  绝不会 ▁Lav  ▁gamers 嘎嘎 Abs  迷人的  黑芝麻  ▁pathways  ▁feminist  ▁Azerbai  conscious   ellites!  ▁grabbed" ARCH#  重大的$ 涨跌% Mir& 外省'  ▁heights( 幼教) ▁RA*  ▁Pont+  ▁mate,  ▁incomp-  的正常.  ▁quarantine/  ▁songwriter0 也从1 花瓣2  报警称3  ▁trek4 费力5 ▁RV6 ▁XML7  倾向于8  整治工作9  ▁domains:  在公司;  直通车<  ▁Hook=  ▁Worldwide> 历任? 口镇@ 壹号A  给宝宝B 此举C  ▁KaneD  在城市E 风尚F 才行G  看这里H  自治区党委I  生产和J 汉服K FrancL  ▁MarxM  ▁AdventuresN 说不O listsP  ▁acquiringQ  ▁exploitationR 时辰S  ▁desperatelyT 斗罗U ▁auxV  ▁inabilityW RoadX 均由Y 暖气Z 余杭[ 摇头\ ▁Exc]  ▁hurd^ 连环_  ▁Cardinals`  ▁Marina  参加过b 申明c 用作d 就和e  以下为f  学位证g  食品有限公司h ▁UVi  ▁mountingj  ▁preservingk spentl  指导价m 奥数n 排污o 白头p  本人的q oyager  ▁reefs  ▁synct  ▁invisibleu  ▁reminisv  升职加薪w  ▁civilizationx 盎然y 学分z  ▁relent{  ▁invalid| 分担}  ▁Auckland~ 言之 命案 成像  证监会  网格员  ▁merit  都必须 个案  作为一种 霆锋 副总 点心 罪名  进行调整  的人在  ▁Hugh wife  ▁byte  ▁energetic  的大部分  ▁incorporating 悉尼 乙烯 散布  ▁surgeon  ▁Peterson 区县  党的十八大  ▁Boat  ▁debuted 我心  做好防护 uggest 上个 预估 鸭子 杜鹃  ▁Honey  ▁blamed лю  的本质  提前批  企业管理  ▁Skin  从这个  也不例外  ▁conting  ▁Ventures 点头 题为 我用  　"　 教堂  同学的 数控  后视镜 非诚 去处 朝向 作画 会比 Lock  很多朋友  我们都是 eworks  ▁NASCAR ▁vou 动图  坚定的  联系我  ▁crushing Ca 王宝  ▁Monroe  ▁Mercury  ▁Monster 广汽 ALE mor 乳酸  ▁phrases  cultural  才能让 问她 通州  恢复了  更新的 听话  因为这个  ▁hobby 正版 疱疹 导购 耳鸣  根据地 肩颈  人群中  ▁cemetery 他用 新旧  双十一 车头  ▁Jagu 安门 开到 infl  面对疫情  ▁bats  ▁Boris 免收 陵区  发展到  说到底 Der  already 黑马  ▁Labs 下入  是全球  科学与技术  身边事 osion 小块  情人节  已久的 性命  无异议  ▁bullying 霉菌  音乐会 叙述 пе  对社会 Sa  ▁Meyer 繁琐  条件和 谁知  或个人  篮板   ▁emphasized  ych   可以使  楚雄 汇率 ▁Adm  生肖兔 give  ▁flesh  ▁tenant  ▁Categories 两三 idos court 眼圈  ▁interval  ▁resear  ▁groundbreaking nas 借此  脂肪肝 团长  守信!  ▁closes" vm# 年老$ 超前% Club& regist' 列举(  ▁Kard)  ▁believing*  ▁transmitted+ 百里,  坚强领导下- 张扬.  犯罪团伙/ KK0 忍耐1 艾草2  期末考试3 妨害4  ▁Seth5  kswagen6  证明了7 三宝8  的组织9  ▁marble: 原版;  临床上<  Jennifer= 观望>  ▁Lagos?  ▁Megan@  可怕的是A DriveB UnlikeC  大队长D  ▁BrandsE  无任何F  ▁extensionsG 可将H 一勺I  第一季度J  回老家K  临床表现L  公共安全M  ▁SerbiaN  语言文学O  ▁YogaP  图中二维码Q 两口R 布鲁S  女明星T  南昌市U azerV  fielderW chiefX DestY 奥秘Z 据此[  的局面\  ▁scenic] 校门^  表彰大会_  ▁plag`  ▁wallpapera  ▁decorativeb 萧山c  好的人d 战区e  的那些f 玛拉g 溪镇h  ▁pleadedi  ▁Wirelessj  职位表k  ▁aforel  往下看m 越长n uffleo 走失p  betweenq bmr 铁道s  扬州市t 有几u  了一点v  饿了么w  ▁optedx  认识了y  人可以z  ▁tattoo{ 两张| 装扮}  ▁Dial~ LIC 调休 ▁jar 翻车 Cra  ▁Abdul fires widget MAC sys opal  专业人员  ▁Avengers  ▁cancellation 和省  ▁Dell  ▁steadily  并要求  ▁inadequ  苑小区 为难 复读  Pingback  aditional  interface  ▁minorities  production 营地 pd 海边  ▁upside  ▁Located Pers  ▁synchron 赛区  蓝色的  病毒传播 runner  经营管理  ▁Moses inski oise zym  ▁beam 有福 ▁ul 少于  就发了 lass  ▁pioneering ▁ate  ▁Norfolk  的最高 体面  护士长  无所谓 传真  在进行 环线  全国统一  改扩建  ▁absorb  最需要 Icon 心血 ▁""  ▁Package  兴趣爱好  ▁wounds  ▁mitigate 小手  ▁Kiss ▁Lem  Throughout 偏高 春日  artisan  ▁Eston  ▁Spons  能达到  专用发票  ampagne rob  上班时间  信息登记 Taking  ▁javax 暗访 沙特 送礼  有针对性 凝心 Simple  墨西哥 知青 orget  ▁Wrest  Parameter 站长  太难了 itas 的黄 黑夜 riad 软弱 再把  ▁mailing  ▁advancement  ▁Customers 寻衅  基础的  点个赞 Legal GG 郭某  主要为  ▁Lines  ▁Annie  ▁Colleges 定投 归还  问题和 marks 永州 到过  黑寡妇  重要论述  血小板  相关单位 巴彦  市委市政府 静谧 巴中 饭局 etz  在此期间 antom 他想  仅次于   ▁Murder   等着你   ▁scales   莆田市   分布在  ▁qualification 礼让  甚至在 backed 择校 偏见 ▁Aus  立德树人  武汉的  ▁warmth  ▁Beast 老头 ▁Sed  ▁appropriately 获评 ▁Dat  力所能及  自然会   ▁illegally!  ▁settlements" 足额#  主要负责同志$  知名的%  的建筑&  不见了' Chair( 品鉴) 亮眼* 元钱+  王老师,  ▁Legends-  责任心强.  ▁displaced/  没问题0 育才1  我的心2  ▁Pinterest3 按揉4  呼和浩特市5 orman6 姥姥7 车队8 炒菜9  ▁params:  ▁rendering; 开园< 齐聚=  举一反三> 素描? Movie@  ▁KennethA istasB 法案C 车轮D ▁SRE  机构和F 减排G 圆润H 已久I  ▁throatJ 同居K  使命感L  日出生M HallN  强制措施O  ▁stellarP 阀门Q  履行职责R 茶饮S 炎炎T  和别人U 心无V  的不良W PPOX  亿元的Y  ▁pasteZ tein[  仅代表\  周岁以上]  ▁consuming^  ▁Charleston_ 缓冲` starsa  ringingb  ▁snacksc 平庸d  ▁carvede  ▁Romancef  只能在g 云龙h  同志任i  如果我j ▁Ecok  ▁Carpl 就曾m  这是什么n  ▁shopperso  ▁breathtakingp  ▁Boomq  ▁inputsr ermats  ▁Marcot  ▁Cableu  给学生v yardsw 看淡x 凶手y  会根据z  清洁消毒{  ▁pitcher|  ▁Sisters}  谢霆锋~  ▁supposedly  儿童的 低估 湖畔 失落  扫一扫 + $我们将在第一时间删除内容 轮流 速递 ···  ▁Optim  ▁Patrol 损耗  ▁Announce 威力  系统性  ▁Elite  ▁Hamb bg 肾虚  为一体的 注释  ▁malware  ▁ambitions inde  ▁geographic  儿童节 斑斓 demand  还有一种 Until  ▁Riley  ▁artifacts 抓细  ▁Height  ▁sampling 车窗  ▁transc baby  ▁forgot  正式开始  ▁Matters 展播  ▁Distribution  也很难  短暂的 眉山  emiconduct  领先的 庆幸 也比 ▁IO  ▁merge  ▁Depression 缩影  当中的  ▁wholly 开机  mentation 幸好  ▁grinding  创造力  created furt jr 北纬 庄子  采访中 能用 软糯  经济增长  ▁norms  ▁lining 德里 远处 外围  游戏中 工学 Fox 发帖 eled ▁bod  入境人员  ▁problematic  acional  一整天 个头  ▁пере 六稳  你已经 GIS  ▁distress adh  免费发布  ▁clearance  ▁lenders Bon  食品药品 底的  一切都是  ▁interpre  干干净 液晶  了大量  依旧是  ▁Domestic  中国驻  服务工作  ▁favorable 领事  ▁monitors  ▁constructor Audio allic  而且是 木质 ▁wax  蒙古国  ▁thrilling 他被 使之 Lo  ▁cabinets 宫廷 惊险 中性 榜首 遗漏  主要以 三高  ▁incarcer 竞选  收入的 格里  才是最  ▁misconduct 姜片  集成电路 threat ethyl  你们是  ▁pilgr  ▁allocation 水里  ▁Wool  公办学校 交融 米兰 辛丑  及家长  年度汇算  ▁Ment ADA 站台  ▁Config 食管 地对 种草  ▁yearly shall Prince  ▁Randy ),　! Mock"  component#  什么事情$  ▁Animals%  ▁Burton&  ▁Patient'  ▁shallow( 在高) 靠前* ▁mim+ 原味, ▁ü-  好工作.  的科学/  的强大0 Deep1  ▁Hopefully2  的记忆3  的主要原因4 散打5 ба6 成一7  ▁timeless8  帮助企业9 上瘾: 主意;  不限于<  族自治州=  非诚勿扰> 也罢?  更不会@ 婷婷A 精进B 酸菜C  olutelyD  改造工程E  ▁JackieF &&G ▁EURH  到现场I 追随J  动不动K ▁BunL HarryM 走私N  ▁scatteredO 震慑P " 中华人民共和国国籍Q 议题R 过头S 多有T 混淆U  ▁illnessesV upiterW 以案X 失常Y eniZ mitt[ ariat\  熟练掌握]  ▁promin^  ▁commissioner_  初级中学`  ▁medalsa Hawb  本土病例c  ▁Namedd 越早e 　;f Wasg  安全意识h 及早i  ▁glancej 赌场k  ▁disprol  ▁overheadm  ▁ecosystemsn  ▁Fieldso 举手p  和教育q  ▁upholdr  ▁distinguishs Schemat  看出来u erersv  如果觉得w  都是一个x  发布通知y  ▁Nestz Es{ 延年| 春联} smith~  分享的  ▁fusion 三口 美感 操场  供电公司  ▁Profession 天山 线和 道家  看看看看  ▁UCLA 果肉  做准备  温馨提醒 一趟 排气  的大学生 why  是一个人  ▁Millenn 沥青 进站 兼备 AMA arette  人群的 憔悴  ▁guilt 不虚 刷卡 强壮  九分裤  你的生活 当即  ▁broadly  ▁resignation roads  寻衅滋事 乱了 嘲讽  ▁dread 熟知  浓郁的  干干净净 忧郁 成败  有几个 是想  ▁Durham  ▁declining  加强与 招工  就会发现 modern  irement  ▁clearing ▁pec  努力的 atted kinson 天性  一目了然 满面  ▁sadly 妆容  分钟内  ▁Roof  刚刚好 ▁på  应纳税  实用性  ▁writings  ▁advertisements  ▁Lenn  ▁Trent 瘟疫 梧桐 不乏  不要让  几年的  ▁Myers  高血脂 格调  很多网友 没啥 多变  borhood  ▁Decision  没多久  医疗保障局  ▁sensation  ▁Population Logger  莫斯科  跨境电商 匿名 商机 nels  ▁competitor 纪实  的伤害  ▁enduring 鲫鱼  这种人 强者 还对  ▁gods  ▁drown  dependency 瑕疵 盆栽 cro PF  可以多  违法所得 偏低  为什么呢 Async  ▁Lamb 回看 zheng  ▁Analyst 懂了 单个  安全稳定  防护用品 dirty 东门  文明办!  network! 小巧! 推移! 两类! ensely! 活的! 老天! 跳跃! ffff !  ▁threaten !  ▁anticipate !  RELATED ! 赚取 ! 星河! axis!  圈子里!  打赢这场!  inguished! arius! 这批! olin!  回家后! 树脂! midt!  ▁packing!  不要去!  财务自由! ervers!  ▁debug!  ▁Portal! 巴马!  钱生钱 !  社区的!!  结构的"! 度高#! ད་$! 忧伤%! 琢磨&!  第三批'!  ▁arrests(! enos)! ▁Lor*! 前瞻+! 口干,! NB-! 它会.!  ▁grasp/!  在规定时间内0! 拼接1!  ▁Similarly2! 要让3!  ▁Shopping4! ió5! 谁来6!  要严格落实7!  bootstrap8! 斯基9! 草地:! 机油;! icable!  突击队?!  在里面@!  itoriumA! igntyB! 卫士C!  现金流D!  ▁limitationE!  第二轮F!  operativeG!  ▁definitiveH!  ▁UtilI!  一览表J! 轴承K!  ▁unlawL!  ▁MirrorM!  ▁resilientN!  ▁motorsO! 托福P!  目标是Q!  点击下图R! 交纳S!  ▁sweepT!  ▁bestsellingU! ▁FitV!  ocativeW!  到哪里X!  时间长Y!  大自然的Z!  生肖鼠[!  众多的\! 不休]! Dar^! Disney_!  第一届`! elina!  红绿灯b!  人的一生c!  排名第d! COLORe!  ▁illustratef!  withoutg!  ▁readingsh!  ▁definitionsi!  保卫战j!  ▁catalystk!  听说过l! sensm!  认识的n! NERo! 无暇p! 天一q!  ▁Supermanr!  ▁originateds! 山庄t!  拒绝了u! 河东v! 不与w!  ▁Talksx!  althoughy! 闪烁z! ño{! 上司|!  改变命运}!  Featured~! 灌溉! Much!  ▁sued! 介意! 日和!  和建设! 七十! 致病!  同事们! 五菱! 寻人!  岁的小! ál! 迈进!  ▁grit! 洗碗! hagen!  新鲜事! ос!  ▁Icon!  ▁Tank! 光盘! 急着! 发泄!  Germany! 免除! 比我! since!  ▁simplicity!  点对点!  ▁endorsed!  工商银行! rub! Params!  清新的!  微信同步! ellery!  ▁Experts!  ▁inmates!  ▁stark! ▁Tah! 低压! zhu!  自己想要!  ▁inherit!  ▁Marl! jquery!  自古以来! 微创!  万元以上! 顺势! 学完!  ▁designation! 铜陵! ▁Sit! 如初! 点我!  ▁noble!  ▁Reader!  ▁complained! 是小! EY!  不是因为!  城际铁路!  选择在! Uk! jw! 药的!  ▁Dollar! ▁soy! ря! 江淮! 当面!  房子的!  ▁Trou! carbon! Son! 按月!  ▁alignment! 卫衣!  ▁Font! 不收! Virgin! series!  没有那么!  一个有! ixty! 停水! 灵璧!  县级以上! KO! 更低!  expensive!  ▁bathrooms! 阅历! ouns!  rimination! Wall! raul! 挥霍! 施救!  ▁Buddhist!  ▁holistic! 以求!  进行公示! 拘役! 文理! 瑞金! charg!  塑料袋! Bec!  爱国卫生运动!  女儿的!  让人们!  合格分数线!  ▁undergoing!  ▁Atlas! azione! 风俗! ryn! 不失! raise!  坚守在! undai!  不足以!  委员会主任! 米粉! ▁orb! 唱和! 火腿"  两个孩子"  ▁func"  ▁Designer"  ▁entrepreneurship" 响了" 爱岗"  不如意" 急剧"  ▁coconut " Apr "  不同程度的 "  ▁convincing " 全军 " 招待" 心力"  ▁Imagine" 爱奇" 防灾" 鼻腔"  ▁expands"  天左右"  的温度"  的时间里" ▁lid" 埋头" 阅览" 冰雹" 之子"  意义上的" ilib"  高中数学" 腐烂 "  生产总值!"  ▁chorus""  记者了解到#"  ▁WhatsApp$" )\%"  ▁Universities&" 肾病'" LAY("  幸福生活)" 审讯*"  ▁Ticket+"  Between," 下定-"  但是他." он/" ITED0"  ▁swept1" 误导2" 图标3" sle4"  ▁administer5"  ▁pars6"  点击阅读原文7" appa8"  应该如何9"  ▁Explore:" 交集;"  固定的<"  主任科员="  ▁groom>"  顺利进行?"  ▁Pure@" 安乐A"  ▁pursuantB" ▁VMC"  家公司D"  开除公职E"  免疫屏障F" PORTG"  开新局H"  开放时间I" 养血J"  一个字K" 屏蔽L"  保障工作M"  计划生育N" graO"  英语学习P" 自备Q" 腰带R" 学制S"  过敏性T"  ▁heterU" )}V"  社会稳定W"  就是对X"  ▁HollyY" aidenZ" 不复[" 同在\" 迎新]"  ▁dominate^"  ▁Chip_" ▁JO`"  的美国a"  养老机构b" leysc"  ▁Institutiond" 网贷e"  WASHINGTONf" vong"  自己的人h" 揭阳i"  ▁Kleinj" silk" 视网l"  就比较m" ▁WORn" 内向o" 归案p" 测验q"  ▁collaborationsr" 心境s" ▁Swet"  wikipediau" 交道v" 大哭w"  ▁Milex"  而这个y" ▁"/z"  准确把握{"  小知识|"  然后用}"  ▁Seoul~" 仰卧" 听众" 远超"  为阳性" 白马"  ▁Pars"  其实在" aidu" 子弟"  ▁Janet" 自此"  也一样"  ▁stakes" 名企"  新型肺炎"  同微信"  更喜欢" 落幕" amins"  贴心的"  广场舞"  Password" 下锅"  将达到"  acements" 母子"  更名为" 成佛"  已进入" 欺凌"  恨不得" 合照" ▁Tas" ▁Ara"  的文字"  胡椒粉和"  ▁username"  ▁remembers"  效果好"  标准和"  ▁Malta"  ▁Paradise" 劝说" 三房"  manager" CBA"  ▁taxpayer"  心脑血管疾病" 乌海"  就觉得"  ▁accidentally"  ▁displaying" anner" yun" Gov" 当头"  ▁fierce"  的行业" 荣华"  抗衰老" fn"  既可以"  产业园区"  的手段" 自贡"  ▁assure"  粉丝们" oors"  ▁abundant" ( !具有中华人民共和国国籍" Match" 吸管"  contract"  ▁legends" 高升"  分泌物" arde" astery"  ▁nephe"  ▁christ"  群防群"  召开会议"  ▁desirable"  老龄化" 鲜香"  ▁Greatest"  ▁те"  ▁analyzing" 伊始" 在各" zero" 受邀" 还挺"  户外活动"  ▁Sessions"  资料包"  确保安全"  ▁naked" 补缴" asso"  ▁inspirational"  感情的"  数字化转型"  可视化" ▁Era"  ▁fetch"  提取物"  portion"  徐州市"  科创板"  疫情爆发"  问题整改" 顺其"  ▁pledged"  ▁installment"  房间里" ▁lug" 操纵"  的新鲜"  ▁activation"  ▁herb"  新冠肺炎患者" itated"  ▁encrypt"  ▁correspondent"  方方面面"  人生中" ем" 国立"  ilingual# 思绪# 盛典# 肉肉#  物质的#  招生录取#  工程有限公司#  ▁Across#  ▁mentions# 低谷 # negie #  ▁Tehran # 底色 # 伤者 # 居士#  展开了#  ▁unwanted# 新娘#  ▁deduct#  ▁ideology# жа#  就意味着#  招聘网#  ▁roku#  ▁Pengu#  检验标志#  ▁descent#  ################# 养的# 痉挛#  ▁neurolog# 小幅# ▁Fix #  委书记!#  你永远"# iago## 办证$# Ask%#  ▁Grid&# 向党'# 就从(# rev)#  ▁onboard*# cler+# 在编,# 心如-#  但因为.#  前提下/# plash0# onial1#  ▁livestock2# 花儿3# 山歌4# ▁Lip5#  ▁learnt6# 公勺7# 的诗8# wyn9#  ▁Bren:#  ▁mathematical;# 张学<#  也成为=# icator>#  ▁numer?#  ▁Emperor@#  上下功夫A# 日新B# 一幕C# 极乐D#  正确答案E# 模考F#  ▁communicatingG# 打赏H# 跑路I#  以及对J#  ▁ferryK# iariesL# 呼声M# 都用N#  的计划O#  ▁collapsedP# 基石Q# 问你R# ievesS#  ▁swapT#  InstagramU#  叙利亚V# 花鸟W#  雌激素X#  ▁RomneyY# couldZ# 昔日[#  你一定\#  ▁punct]#  医疗卫生机构^#  ▁Khal_#  ▁customize`#  ▁Operatinga#  也随之b#  的网络c#  ▁delightfuld#  生肖鸡e# annif#  截止目前g#  聘用合同h# 是好i# ▁Pegj#  但是我们k# 张晓l#  ▁Credm#  ▁arbitrationn# yneso#  本土确诊病例p# 陵园q#  进行调查r#  ▁clarifys# 全美t# 过半u#  县人民法院v#  ▁revisionw# 这所x# ▁MVPy#  ▁Hashz# UTC{#  投资有限公司|#  应当在}#  目前正在~#  道路交通事故#  第二百#  ▁academy# 义工# 触及#  ▁Takes# 奥林#  都可能# 习作#  北京师范大学#  精准扶贫# utf#  为抓手#  这条路#  ▁Huff#  ▁businessman#  ▁discourse#  ▁Spark# 星人# 一词#  的希望# 走走# ▁cit# ▁dra#  fashion#  ▁monitored#  直接影响#  ▁educator# Brad#  履职尽责# chard# 渡过#  ▁quantities#  会因为#  的一般# 三世#  并表示# Birth#  ▁assurance# 减缓#  ▁JavaScript#  ▁negotiating# 义勇#  ▁Windsor# 太好# 与时# rint# 井冈#  签约仪式# 遗留#  党的领导#  ▁abused#  ▁impair#  ▁incumb# Past# ande#  ▁stub# 对党#  ▁hammer#  ▁invasive#  ▁rainfall# 德云#  贫困地区# omics# 受了#  或许是# 发来# 贵在# ön# 而定# 打仗# Tri#  ▁Zion#  ▁pursued# 给它#  ▁ample#  思想上#  等重要# 约合# umed# 满天#  ▁ruins#  operation# 快感#  利用率#  ▁reimb# 募集# 特困#  ▁Flex#  为推动# Within#  annotation# 小张# ,…# 兢兢#  负面情绪# ▁Vas# 中兴#  上大学# 郭勒# 飞速#  learning# 并存# wat#  可以有效#  ▁overhaul#  ▁Unity# 手艺# 打车#  中央八项规定#  ▁mins#  的大学#  ▁delegates# ▁DM# 却能# 条纹# 黄连#  如果要#  系统中#  quarter# ▁AZ# 手部# iq# 归于#  今天要#  ▁chick# 高点# shaw#  申报表# 反省$ 胡同$  ▁Diagn$  ▁slated$ 追回$  年公开招聘$ HH$ XL$ 课间$  ▁sunset $ 闷热 $  ▁livelihood $  新增病例 $  ▁tract $ apa$ eco$  Privacy$  ▁Rehab$ 农夫$  ▁tackles$  ▁Epic$  ▁Bills$ Bus$ inders$  ▁migrant$ ▁SB$ ▁TE$  人员伤亡$ bestos$  ▁nonetheless$ 并能$ Altern$  Reading $  ▁dictionary!$ 大好"$ Ast#$  停下来$$  是无法%$ pler&$ ▁Kor'$ ")]($  Western)$  中西部*$  资料来源+$ yz,$  ▁Male-$  ▁repay.$ 柏林/$ 调入0$ 耳鼻1$ 迎合2$ 年报3$  ▁fixture4$ ві5$ 不曾6$  ▁Bulgaria7$  秋冬季8$ 看守9$ umat:$  ▁punish;$  的水平<$ iotic=$ 监考>$ 偏爱?$  挺身而出@$ 疫病A$ 嵌入B$ 干旱C$ 中西D$ 新课E$ 帮手F$  生肖牛G$ тыH$  举报人I$  ▁feedsJ$ 命理K$  司法机关L$ lbM$ 有成N$ 情谊O$  中国古代P$  ▁modificationsQ$ 济源R$ 光了S$  从根本上T$  ▁guidingU$ 洗完V$ 而下W$  但随着X$  条第一款Y$  管理处Z$  精美的[$ 不讲\$ 天价]$ 输了^$  这一幕_$  实施了`$ Inva$  这种方法b$ dkc$  的业务d$  也是可以e$ 擦亮f$  ▁acidsg$ ▁Igh$  ▁redei$ 得天j$ 漯河k$ ▁Cirl$ Sorrym$ 千瓦n$ ▁Hodo$ 编排p$ ^^q$ YPEr$ ▁Kws$ Timest$  ▁Fuelu$ jgv$  ▁Employeesw$ 雍正x$  家中的y$  调查中z$  村集体{$  ▁encryption|$  ▁consisted}$  ▁Isle~$  ▁quicker$ 早前$  ▁ambul$ 小了$ 无辜$ atisf$ 全靠$ 赛场$ trl$ ▁Cay$ 并由$ flower$ 球场$ 弹力$ ▁McN$  常态化疫情防控$ 只做$  ▁stip$ CRE$  ▁accusations$ Arts$ props$  买卖合同$ 电阻$  ▁Overview$ 团建$ 不弃$  猪肉价格$ mobile$  不懂得$  要严格$  工商管理$ 卖的$  ▁voor$  ▁Archae$ 遗症$ 漂浮$  大力推进$ 他将$ 酿造$  ▁adolesc$ 之首$  又开始$ 自知$  自然村$  国家机关$  ▁Tales$ 劲儿$ år$  acceptable$ Market$ 给力$  对他的$  ▁Funds$ 传人$ 圈内$ 耗费$ 先将$ 巴克$ 特尔$ 电饭$  而成的$  思想的$ 件套$ 时针$  不相信$  一把手$  喜马拉$  ▁assumption$  ▁Harold$  最小的$ mini$ 五味$ 外墙$ Fall$ scape$ 杀了$  ▁Civic$  的造型$  采用的是$ Dialog$ 新京$  ▁Bernie$ 绳子$  ▁labeled$  ▁grounded$ 军工$ 危房$  涵盖了$  阔腿裤$  隆重举行$ 军方$  ▁Springfield$  ▁chess$ 碱性$  你怎么看$ 算计$  ographies$  ▁fulfilling$ 翻天$ 多加$ 派对$  ▁Geoff$  给对方$  一步一步$ 笼罩$ gust$ publ$ 舆情$  ▁wool$ 一卫$ 风向$ 单项$ 无须$  的微信$ 本部$ itic$ heres$  举个例子$ adia$ ▁Viv$  ▁slaves$ 震动$  就直接$ 心率$  ▁Vine$ 供血$  ▁senator$ 近百$  一时的%  ▁Dreams% anyahu%  ▁firearms% 容貌% 焯水%  我个人%  绝对不会%  的提升% 筹集 %  的主人 % 杨梅 %  就是这个 % onge % 险情%  时间节点% 减脂%  ▁leagues% 金句% 驻地%  打造出% Baby%  工业互联网%  ▁Sheet%  ▁Electrical%  还可能% пу%  总监制% ▁Xi%  如果你有% 义诊%  的创新%  ▁ally %  给自己的!% 之感"%  ▁riot#% 前端$% VERS%% insky&%  和田玉'%  ▁Pakistani(%  ▁practiced)%  ▁scheduling*%  研究成果+%  ▁Waste,% 课上-% 迟早.%  工作部/% 拟聘0%  交通枢纽1%  协议书2%  帮助下3% lated4%  undreds5%  有小雨6% 山下7% 查封8% вер9%  学科类:%  ▁anxious;%  ▁Accounting<%  招投标=%  测体温>% otech?% ETA@%  心肌梗A%  岁左右B%  reatmentC%  ▁BaldwinD% 也已E% 寄宿F% 原型G% ▁vomH%  ClimateI%  年春节J%  李嘉诚K%  ▁OutlookL%  ▁VoteM% 剪纸N% 梧州O%  帮助大家P%  resourcesQ%  表现在R%  的道理S% npT%  承载着U% ppeV% 肝气W%  ▁SequX% 伤痛Y% 锤炼Z%  身体不适[%  进行处理\% 年和]% 本性^% 串串_% 室友`%  喜马拉雅a%  ▁modificationb% trainc% ▁dwd% 热气e% 宽阔f% 摇篮g% ▁Wikh% arsityi%  ▁spacecraftj%  我们把k%  ▁Allegl% 潮州m%  ▁lipsn%  ▁flieso%  ▁passivep% 的真q%  ectionsr% 野菜s%  ▁Fortunatelyt% 颇为u%  逻辑思维v%  ▁congregationw% 将它x% 横幅y%  下面是z% Earth{% HN|% 中天}%  ▁bride~% 熟人% 笔墨% 延时%  ▁downs%  ▁cardiovascular% 文稿% ▁Lun%  ▁Someone% 劳模% pub%  ▁denial% ▁mol%  ▁inappropriate% Amy% 不正% 行星% 共用%  ▁carn% 感官%  副组长%  ▁scram% 白领%  来形容%  Netflix%  温柔的%  本以为% emer%  ▁года%  绿色的%  ▁reservations% 就别%  ▁Eventually%  ▁resem% 操心%  情节严重的%  的行动% 倍感% 温润% 滴水%  ▁disable%  ▁empathy%  ▁Dana%  发布之日起%  ▁fluct%  ▁Tale% ealous% 触动% Auth% ▁Bey% 宽广% ksh% 切身% 永生%  星期四%  ▁spatial% 微型%  示范基地%  ▁Alert%  ▁mainland% 除草%  每个孩子%  行政机关% 港元%  Internet%  ▁halls%  期待的% 白玉% bucks%  ▁Ruby%  ▁glorious%  寄生虫%  ▁plumbing%  每天坚持%  ▁Dinner%  届时将会% 乔治% 党务%  大本营%  ▁Countries%  ▁chlor%  ▁dwell%  ▁curated% 宽带%  传承红色基因% Jay%  ▁cort%  ▁supervisor%  ▁Jacksonville%  ▁Brid%  ▁Transform% 集锦%  results% 脱口%  基本工资% 放纵% 阿根%  ▁refurb% efe%  相关问题%  在这种情况下%  workers% 石桥%  相关阅读% INGS%  监管部门% 专硕% >>>>% 收货% 烧开% 寺院% 纵向%  在不同% 国六% 蓬松%  ▁spawn%  ▁intuitive%  儿子的%  第二针%  级景区% Connor% 勘查% 徐某% 架子%  ▁Knox% Jordan%  ▁seafood%  走过的% oppy% 耗时%  新产品&  ▁Peak& 经期&  ▁Guatem&  ▁fireplace&  ▁backpack&  ▁severity& 铜仁& 占道&  人脸识别 & 天主 &  救援队 &  是什么呢 &  精准施策 &  ▁Tunis&  ▁traject& 交响& ▁Liu&  副厅长&  建设高校&  ▁homosex&  Management& 忧愁& 送来& 云烟&  努力学习&  ▁adopting& Taylor& 上都&  也正是& 才艺& 店主& TVB &  ▁Shen!&  ▁encoding"& вы#& hesive$& ít%&  ▁Abstract&& Islam'& 尾声(& 巧合)& 换取*& 籍贯+&  ▁Transit,&  星期三-& 演义.&  ographics/&  官僚主义0&  ▁Airways1&  ▁pronounced2&  ▁envision3& 管用4& 卖房5&  怎么回事6& 丢弃7& 死去8& assis9&  ▁spanning:& 存栏;& 中交<&  ▁notebook=&  ▁endorsement>& 滑坡?& 金龙@&  一旦发生A& 褐色B& 当月C& ▁DinD&  ▁BreakfastE&  ▁answeringF& 呼唤G&  上下游H& 干线I& 越远J&  虽然在K& 事的L&  ▁meshM&  ▁nightmareN& UVO&  一经查实P& 旺财Q& 指纹R& 职级S& ulseT& 细雨U& ZEV& phiW&  第一大X&  ▁anglesY&  里面有Z& 游园[& 鲜肉\&  路小学]& 暂未^&  等什么_&  最新通知`& 灯笼a& 蚕丝b& 吓得c& 沮丧d&  竟然是e&  工程专业f&  ▁overlyg&  Startingh& 魔术i&  不文明j& 倒在k& 汛期l& edom& 日是n& 转业o& Mailp& ramesq&  скогоr&  ▁Kindles& 长发t& 之手u& 悲痛v& 版块w& 赛跑x&  报告显示y& 枝条z& ucked{&  ▁bombing|&  助学金}&  的不断~&  您的权益& ITION& daily& 东盟&  ▁polling&  ▁Opportunities&  个专业&  ▁Either&  Complete&  ▁unforgettable& 屋里&  精神文明&  中央纪委国家监委&  ▁inherited& 张艺& ▁az&  一个好的& 不耐& 别克& 永康& 萃取& Dark&  ▁sunlight& 加码& 爱和&  视为自动放弃&  万左右&  早知道&  ▁freeze&  必不可少&  实体经济&  等地的&  ▁imperial&  始终把&  ▁Animation&  坚持不懈& ▁bei&  个人简历& 接单&  ▁bypass&  第五届& orde& 生计&  领导者&  ▁Strip&  ▁atmospheric& 深沉&  不理解& hon&  这样的情况&  ▁Prosec& 积雪&  上了一& 掩饰&  ▁Wagner& strom& + $新型冠状病毒肺炎确诊病例&  ▁entrepreneurial& 长出&  让人觉得& 安顺& 白血&  ▁Courts& 不设&  ▁Treaty& 醒了& ▁BJP&  ▁recognizing& 专车& 沧海&  的物品&  ▁Needs&  ▁McCain& 源地&  身体素质&  ▁brains&  ▁Promise& GW& 任正&  卡是不&  verages&  是否需要& 卒中& 小黑&  ▁Membership& 猥亵&  ▁hackers&  所带来的& 享用& ▁vet& "_& 分餐& follow&  一般为&  ▁exemption& 较快&  ▁allev& 钱塘& ▁și&  对这些&  在疫情期间& imer&  城市更新& Option&  ▁className& 直呼&  一点一点& 最最&  的资金& ▁oun& ▁Bod&  ▁Tomorrow&  我当时&  清洗干净& 不老&  微信关注博野在线&  ▁Spotlight&  ▁exploit& Kit&  ▁legislators& 寝室&  ▁Abbott&  ▁reiter&  ▁exceeded& 绵绵&  语文下册& 信网& irteen&  有效期内& 赞叹' 骑着' 官司'  ▁dancer'  ▁Wash' ▁Git'  ▁souls'  有价值' ▁Gir'  反而是 ' Values ' 靓丽 '  ▁Goes ' olics '  ▁suburban'  虚假信息' zm' Roy'  ▁Devon'  十分钟'  protected' 六腑'  旁边的'  病虫害' ▁Agg'  需谨慎' ▁dir'  邯郸市'  这个世界上' 又叫'  Elizabeth' 充斥'  ▁avid ' 脊椎!'  使用时间"'  ▁Davies#'  ▁Tonight$'  潜意识%'  ▁tide&'  ▁props''  ▁TikTok(' 净水)' sym*' 校车+'  才开始,' 日式-' ▁Cec.' 炒股/' 制冷0' 呵呵1' 孕妈2'  和国际3' 一粒4'  义务教育学校5'  为方便6' 都已7' 人防8' 可可9'  ▁adorable:' 彼得;' fj<' ▁И='  对象的>' 坍塌?'  阅读理解@'  主办的A'  ▁NewmanB' 译文C' ▁EMD'  ▁lightlyE' 倒了F'  出了问题G'  BesidesH' StreetI'  排名第一J'  遇到困难K'  市场监管总局L'  统一到M' 节选N'  大讲堂O'  公益活动P' 阳虚Q'  让对方R'  中国大陆S'  使我们T'  ▁ActressU' 残联V' 用时W'  ▁LeslieX' 墙体Y'  海内外Z' 阳江['  ▁rifle\' 我军]'  连续三年^' 周炎_' 树下`'  ▁subjecteda' 太后b' 烧热c'  ▁Sebastiand'  避免了e' aspxf' ▁Cafg'  ▁Sharesh'  ▁selectionsi' 呆了j' 易地k' 色情l'  主要负责m' 代会n' 长腿o'  特别适合p'  esterolq' cookr'  Selecteds' Incret'  ▁imaginu' 菌群v' ▁Heyw'  ▁pullsx'  自然灾害y'  ▁tierz'  ▁Bollywood{' 阻塞|' zq}'  是人类~'  理事会' 抽签'  ▁Beverly' 明智'  ▁morph' 那我'  很多同学'  内防反弹' Later'  perature'  ▁Homeland' 愚蠢' Saint' ▁AA'  详细信息'  高速铁路'  ▁collector'  onsense'  ▁traces' 揭露'  餐饮服务'  房屋出租'  question' ▁BT' inel' equals'  ▁tones'  essages'  生活和'  检测机构' 场次'  不得已'  健康和' 城西' 福清'  Johnson'  ▁Giving' Ox' 七分'  星期二' 格栅' 财气'  这只是'  ▁Declaration'  ▁remedy' 完备' ▁ART'  ▁Asset'  ▁priests'  ▁battling'  平日里'  ▁hints' 来来'  打造了'  neither' 晶体' fen' tesy' ▁pse'  条的规定'  ▁chill'  scription'  ▁discoveries'  下一次'  为您提供'  都有自己的'  ▁Burke'  ▁markers' learn'  ▁Gandhi'  进行分析'  ▁Outstanding' 金币' ▁ion'  ▁downloads' 废水'  不至于'  学位证书' 相交' 红外' Trust'  总会有'  ▁congest'  的含量'  精彩内容'  ▁economically'  是否可以'  ▁philos'  埃里克' 迎难' 动辄' nar' 姻缘' ▁Rib'  ▁bloody'  ▁activism' 栋楼' neum'  轻松松'  没有硝烟的' Si' 相依'  ▁lush' ▁Iv'  博物院' ▁Wid'  国际学校' ANA'  ▁Surface'  ▁requesting' 在院'  ▁particle' 味甘'  事物的'  工作年限' amboo'  ▁broadcasting'  爱奇艺' 回访' 阴谋' rehend' 疯了'  直接接触'  ↓↓　'  采取措施'  或者有'  ▁imperative' 厌恶' 店镇' 透彻( inz(  所需要的( SET( anchez( iona(  ▁publicity(  ▁comprising( 之势(  ▁lion (  ▁Literary ( 管教 (  邀请了 ( 无用 ( 文凭(  ▁Reporting(  Created( ▁GD( ▁SAP( 变大( ▁Rw( 商报(  ▁pivotal( 尾部( Rule( owitz(  ▁Them(  ▁Goals( 壮阔(  怎样才能(  ▁curb( 经的( {　 ( 申诉!( 更佳"(  矿泉水#( 人来$( 王菲%(  ructuring&(  ▁catalogue'( 称其((  走访慰问)( ( !党的十九届五中全会精神*( 攀比+(  ▁Hayes,( 五谷-( 飞天.(  iplayer/( 危重0( 沙尘1( ▁Au2( 时会3( 特警4( 未被5( 网签6(  ▁singular7( 借着8( Asia9(  ▁Blvd:( ▁lam;( 梯队<( 投影=(  Progress>(  称得上?(  ▁underwater@(  抗疫一线A(  ▁AcceptB(  个名额C(  ▁cheapestD(  ▁stealingE(  教学质量F(  ▁GustG( cornH( 伤了I( ▁DyJ(  ▁brandedK(  inationalL( 稳中M(  使用方法N( bolaO(  ▁soreP( 的字Q( 赢了R( 之于S(  阿根廷T(  ▁ballotsU(  ▁realitiesV(  生肖虎W( handedX( 活出Y(  ▁spokeswomanZ( 玄关[(  实名制\(  我的朋友]( iste^( lain_(  ▁deserved`( ▁rena(  即可领取b( Lisac(  每天吃d(  切实加强e(  ▁Citizenf(  ▁Sonsg(  ▁suscepth( 文言i( 搞好j(  ▁encouragementk( 出家l(  在当时m(  ▁grapn(  ▁dragono(  ▁governedp(  颁发的q(  ▁Tanzaniar(  次全会s( 巴士t(  陌生的u( 特约v( 蜗牛w(  兄弟姐妹x(  ▁worryingy( eousz(  优秀的人{(  ▁albeit|(  ▁partnering}( 一招~( 犯错(  市场份额(  第二年(  隶属于( apply( 慈禧(  ▁naval(  ▁Vander( 六保(  ▁futures( 土豪( 制备(  ▁Mond( 惆怅(  Release(  ▁Edmonton(  大富大贵(  Christmas(  ▁unpredict( 高处(  八年级( 系的( 莲藕(  非洲猪瘟( 驳回(  淘口令( xl(  ▁exquis(  ▁Customs( 7 0微信关注博野在线微信关注博野在线( 体谅(  Language( 富强(  脱贫攻坚成果( 确有(  ▁Cork( terror(  ▁Marina( Neil(  Something(  ▁Remote(  electric( 英烈(  ▁wilderness( 火速(  ▁append(  凭借着( 分发(  开窗通风( inet(  全产业链( teenth(  ▁Vietnamese( 送出(  ▁Elder(  检验机构(  ▁bleeding( 石门(  ▁competed( 你得( 有啥(  ▁masterpiece( 他家(  ▁internship( QS( 大体(  及时向( 早睡(  安全防护(  ▁Twenty(  ▁Organisation( 定西( 祖父(  有多么(  可能会出现(  核心技术(  standard( 穿透( urry(  ▁torque(  ▁misleading( 致的( 根治( 懈怠(  ▁surveyed( 杀虫(  土地上(  李老师(  ▁compilation(  轻轻松松(  Financial( 鞭炮(  ▁deliberately( 独生(  大多是(  比例的( byte(  分析了( 如在( 滤镜(  等现象( 绿城( floor(  招聘程序( 休克( 悲哀( 拉近(  不起来(  ▁pave( 皆可( ostics( 纷呈(  提到了( 酥脆( 讲到(  ▁Exhib( 质检(  补助金( CPU(  发展和改革(  优惠活动(  ▁clue(  ▁snack( 化石( Spot(  ▁pear(  ▁Affordable(  让更多的人(  免疫功能( ▁ND)  通报了) 他能) 考区)  七年级)  ▁payroll) 我行) 主事)  公安交管)  ▁Essex )  ▁Panthers )  全力推进 )  ▁Patricia )  ▁velocity ) 造血)  照顾好)  财务管理)  ▁Fraser) 饵料) biz) 本是) anqu)  薪酬待遇)  ▁emergence) 领跑)  当地政府) 水墨)  ▁Raiders)  全球化)  ▁feather) 三省) ▁AWS) xh ) 起拍!)  任何事情") 则为#)  ▁crisp$) 多起%)  能吃苦耐劳&) 工农')  微信小程序() alin)) 成渝*) 球菌+) 使馆,) 多见-)  ▁Mood.)  ▁assessing/) 不死0) 大都1) got2) Prop3) ▁GRE4)  ▁Interactive5) 太久6) 我所7)  整治行动8)  ▁Shannon9)  ▁calculations:)  三个方面;)  ▁bureaucr<)  按实际=) 男神>) 百计?) 问到@) JewA) ️　B) hurstC)  有条件D) 百态E) 加价F) alineG) ▁EcuH) 主机I)  的成长J)  项目和K)  ▁FreemanL) 烟气M) 撒上N) oboO) asiveP)  ▁complimentaryQ)  尤其在R) shameS) 月休T) 可贵U) ▁CocV)  的结构W) änX)  ▁reasoningY)  ▁festiveZ)  更健康[)  抑制剂\)  ▁scam])  会展中心^)  ▁analytical_)  ▁burial`)  ▁distancesa)  ▁positioningb) 满是c) ynthd)  任何费用e)  ▁Hindif)  ▁Katherineg) Mrsh) ▁shai) 摄氏j)  ▁casinosk) 咽痛l) 那段m)  ▁silkn)  ▁bilateralo)  controllerp)  岁月的q) ▁btnr) liants)  ▁crediblet)  ▁Wideu) 乱放v) 哥们w)  做家务x)  版权属于原作者y) rossez)  ▁magnet{)  ▁appetite|)  拍摄的})  ▁Hyper~)  的观点)  吃东西)  ▁cables)  快速路)  我不想)  保持良好的)  ▁Nottingham) Inf)  中国银行)  工作中的)  ▁narratives)  国家卫生)  ▁Newsp) GRE) % 微信公众号微信公众号) bst)  ▁learns) 白皙)  ▁divor)  ▁Tong) thirds) XXXX)  ▁developmental) 高昂) 国药)  精气神)  ▁Kurd)  ▁Whole)  ▁melod)  ▁blade)  事实上事实上)  ▁mattress) 乐团) Front)  武汉加油)  时间长了)  ▁tasty) 佳绩) 太少) 画像) 相撞)  阳光下)  ▁Pastor) 尔德) 未见)  weather)  orgetown) 哈弗) 内幕) 岩石)  孙悟空) ismo)  怎么才能)  ▁darker) 乙醇)  和能力) ecca) Sound)  黄色的) tags)  ▁Been) 指战) 聚合)  图片来源网络) 天河) 观景)  陷入了)  ▁epit) 村居) ▁MLS) 和林)  ▁partition) Root)  遇到问题) 借用) 从零)  ▁Loan)  ▁accounted)  层次感)  显示屏)  了一家)  人到中年)  ▁Stockholm) 黑的)  ▁committing)  ▁Quotes)  ▁empowering) 有新) 王文)  极大地) omal) Track)  生产力)  阿克苏)  ▁Written)  Summary) Math) ▁Pep)  ▁Triple)  Americans)  ▁Menu)  ▁Manit) 伤寒)  上市的) 破案)  第一阶段)  ▁Philippine) ▁fog)  自然资源和规划局) 离别)  积累了) 上牌)  ▁Simmons)  ▁tackling)  一个孩子) 晾晒)  主流媒体)  ▁Packers)  情形之一的)  累计死亡病例)  凝心聚力)  城市发展) 福祉) 单科) 把好)  死亡的) 永定)  美联储)  的时间和)  教育引导*  ▁Liqu*  欢迎在* 年检* 窍门*  秋天的*  ▁therapist* 站的* 造车* 地貌 * 顶着 * 监视 * tering * Ut * 领馆*  都会被* Prime*  ▁Testament*  辛辛苦* antes*  ▁Ages*  案例分析* stead*  ▁ceramic*  ▁resolutions*  宣传工作* iatry* 调价* erala*  开放的* 是将* oxic*  ▁Figure * 倍的!*  自我保护"*  ▁YORK#* UID$* ▁AFP%*  ▁Extension&* money'*  ▁Hands(* 由市)* Biden**  quisition+*  一些小,* 琐事-*  如发现.*  ▁lasts/*  ▁wishing0*  心里的1*  ussions2* ▁Gri3* 这波4*  科技成果5* said6*  ▁Visitors7*  ▁Awareness8* 撑起9* 联谊:* 尿素;* 民居<*  有意义=* 软化>*  生动的?* 不愧@*  能得到A*  fficiencyB* 美洲C* 随风D*  的保证E*  管理服务F* 眼角G*  ▁enactedH*  延长至I*  有话说J* itoryK* showerL* CYM* 班线N* 任选O* 天道P* 者可Q*  遭到了R* utaS*  电子书T*  ▁COMPU*  ▁coolerV*  ▁bloggerW*  consoleX*  你真的Y*  ropractZ*  ▁Johannes[* 刘邦\* odore]*  pathetic^*  settings_* 一旁`*  等平台a*  辛辛苦苦b* 做着c* 加的d* 开场e* 螺丝f*  ▁respondsg*  沉浸在h* openedi*  ▁enlightj*  攀枝花k*  ▁Maritimel*  ▁hurricanem* 进京n* talo* 真切p*  精神状态q* TYPEr* ipegs*  ▁seminarst*  ▁neverthelessu* 胆囊v* 纵观w* bugx*  ▁Shiny*  ▁Sallyz* 动荡{* 猛烈|*  约定的}* 一拳~*  сько* ▁Gur* FB*  读书会* 开好* 石狮*  Install* 绿叶* 疙瘩* 罪犯* dal*  Patrick* 赞扬*  ▁Celebrity*  ▁paras*  ▁Birds*  ▁Diseases* 史诗* 简要*  上网课* ▁cov*  lymouth*  ▁leaked* 孙女*  ▁scra*  selected*  总体规划* 儿媳* 沦为*  肾功能* 当了* 退缩*  有效控制*  ▁concentrations* fts* PAR*  就是一种*  ▁Minutes*  ▁inquiries* junct* 杏花* 空港*  我们自己* 卡的*  ▁Brock* 就如*  很重要的* 街拍*  ▁Litigation*  大流行* ▁REL* orian*  区人民政府*  工程质量*  henghuo*  ▁thrill* 这四* EMS* 确权*  不要把*  黑社会*  ▁pixels* 不调* 但有* 使我*  社会科学* 优抚*  日以后出生*  告诉孩子*  非法集资* 公社*  ▁naming*  ▁hottest*  反而会*  以下几点*  ▁Sets*  赚钱的*  三十年*  不是一个* 王家* 眩晕* 拍下* ▁APP*  ▁galaxy*  有序开展*  ▁monsters* 丝瓜*  Background* ▁ISS*  ▁Parade*  ▁bitcoin*  ▁contacting* 土拍* 地名* 微商*  的控制* 伊斯* 用完*  ▁unified* 以西*  ▁testament* 游击*  多小时* hidden*  所以他们*  一人一*  ▁canal* 冷淡*  是属于* 出题* 慢行*  ▁hatred*  为代表的* 德基* 美了* 被封*  户籍地* lash* 李世*  走出去*  如有问题*  东方红*  ▁hygiene* 牛羊*  ighthouse* rylic* 虚伪*  如侵权*  ▁Finding* 老街+ 兼任+  公立医院+ 坠落+ 驶入+  ▁mirrors+ 厅级+  但是我+ xu+ 探访 + iii +  ▁triv + 桌椅 +  保质期 +  就是个+ 脚印+ 静好+  的单位+ itism+ 人会+ 天眼+  ▁refusing+ 通便+  中表示+  ▁Elvis+  高颜值+ 重担+ 前排+  ▁rescued+  实力的+  ▁stressful+  给人的+ 想说 +  美容院!+ 调取"+  戴好口罩#+  绩效考核$+  ximately%+ 张掖&+  等特殊'+  ▁perf(+  的那个)+  车管所*+  ▁marketers++ 有氧,+  完全不-+  ▁showcases.+ gold/+  enabled0+ 白雪1+  转载自2+  也要注意3+  节日期间4+ 洗浴5+ Cas6+  相信大家7+  进一步增强8+  ▁Joyce9+ submit:+ UPDATE;+  ▁Highland<+ 都对=+  ▁Solo>+  建设中?+  打游戏@+ 办税A+  也有了B+  代表作C+ 毛钱D+  难得的E+ 曾因F+ ▁SaxG+  设施设备H+  ▁curtainI+ 迷路J+ oshopK+  ▁THATL+ 风流M+ ▁EyN+  延伸阅读O+  心理疏导P+ 冬虫Q+  可以不R+ 祝您S+  技能培训T+  ▁fairyU+  广泛的V+  forcingW+  ▁destroyingX+  开展党史学习教育Y+ 李晓Z+ ▁RBI[+ 肝病\+ tri]+ 静心^+ ▁PDT_+  ▁tasting`+  ▁Formata+ ▁fueb+ 练兵c+ 薄的d+  ▁attachmente+ があf+ 等同g+  与众不同h+  ▁durabilityi+  ▁predominantlyj+  方向的k+ 心电l+  ▁signaturesm+  ▁collectorsn+ 危化o+ Studiop+  ▁emissionq+ womenr+ itchess+  设置的t+ 换气u+ 北门v+ 婚前w+  modulesx+ 走动y+  地级市z+  眼前一{+  说白了|+ 临朐}+ 受众~+ 栖息+  ▁discovers+ 开店+  有风险+  这个方法+ 竞得+  怎么可能+  ▁collaborated+ 的血+ 胃酸+  尤为重要+ utsch+  ▁determines+ 科协+  强有力的+ 外媒+  人们在+  ▁senses+ 各异+  ▁hostile+ 动感+  ▁condemned+  三年的+  ▁malicious+ 纽带+ 交办+  扫码查看+  市委宣传部+  ▁likewise+  手机的+ 有线+  经历的+  美女美女+ ℃~+  ▁tracked+ 临港+ 冷战+ 看护+ 邀您+ bright+ 占总+  所以他+  ▁damp+ 贪婪+  药监局+ 二本+  下心来+  下午茶+  要坚决+  几个小时+ ZKJ+ EDIT+  教育工作+ 多说+  可以做+ 半径+ 山湖+  交朋友+  检查和+ need+  ▁bounce+ 寒风+  艰苦奋斗+  中国疾控中心+  ▁shareholder+  ▁Malcolm+ 绷紧+ exc+ 师的+  ▁Sterling+ Gar+ lr+ 市容+ 迈向+ 奖章+  好习惯+  ▁slim+ 昵称+ 涡轮+  政治任务+ 六盘+ 能给+  必备的+  ▁bees+  ▁chim+  山景区+ ▁ol+ Rod+  到自己+ ilee+  建议你+ buffer+  ▁specializing+ USB+  出现过+  新冠病毒的+  enstein+  帮孩子+ Andy+  ▁cardiac+ 出任+ 未有+ eri+  抓落实+ Cover+  改革的+  Culture+  ▁speeches+ hess+ 盟友+  ▁Leicester+ 大明+ 秦岭+  应有尽+  火神山+  products+  签订的+  ▁Item+  卡积分绿卡+  ▁revival+ ▁"$+  也容易+  ▁gatherings+  Anthony+ 越走+  身边人+ umers+  ▁recruited+ 心胸+  ▁configure, Jones,  ▁Cases, 为己,  ▁fragile,  全体会议, PRE,  ▁warfare, 孙某, mant , . '全面建设社会主义现代化国家 ,  ▁incon ,  ▁Chevrolet , 美貌 , 追问, 夏草, ▁mig, ▁Few, 强硬,  ▁browsers, 巡航,  启动了, orthy,  ▁Clara, 山川,  ▁jurisdict,  ▁Alban, ALS, 学着,  最美好的, 语句,  经营性, 杀灭 ,  ▁seasoned!, 用笔", 阵营#, lake$, 深造%,  任职资格&,  ▁optimism',  ▁Parkinson(, 节后), 致远*,  ▁extraction+,  老一辈,, lane-, 牧场., ном/, Jean0, ossip1, 扫除2,  工业企业3,  ▁newcom4, 他与5,  营养素6,  源自网络7,  ▁Strat8,  ▁geographical9, 脚踝:,  科技股份有限公司;,  ▁incorporates<,  ▁reconstruction=, 分包>, 索赔?,  现在很多@, NightA, 账单B,  城市群C,  是自己的D, 疗愈E,  点击文末F, churchG,  总医院H,  若涉及I, 男的J, 肝火K,  ▁NewportL,  爱岗敬业M,  ▁caredN,  transformO, 汉堡P, 不净Q, 放出R,  这个小S, wicT,  创作的U,  积极参加V, 解难W,  解决的X,  服务等Y,  国家发展改革委Z, ajo[, 分寸\, 当心],  ▁framed^, 他为_, Ly`,  ▁colonya, Anneb,  ▁routinelyc, osod, LNGe,  ▁Randomf, 乎乎g,  YouTubeh,  支持下i, 直流j,  ▁Robertsonk,  ▁expeditionl,  ▁interpretedm, >_n,  ▁noticeso, ▁McDp,  共产主义q,  可分为r,  ▁reducts,  ▁notorioust, фиu, ▁Dwv,  规划师w, ▁Levx, sayy,  女演员z,  爷爷奶奶{, 更长|, 蓝光},  ▁diamonds~, 普拉, 生下,  ▁Launches,  ▁prevalent, 拍到, OCK,  从这里, 提早,  ▁facebook,  这一个, XY, 烟雾, 升至, 在市,  前两天,  的制作, ioni,  ▁staged, Simon, 误差, ulk, 鼠年, 正向, 象棋,  ▁Benedict, 交费,  starter,  的实力, 硬笔,  ▁Kenny, 嘱托, 总要,  ▁Pull,  给你带来,  青年教师,  ▁threatens, 过失, Remove,  ▁Areas,  ▁Trophy, 病患, ogene,  骨干教师,  ▁gateway, 押题, 稿酬, grown, 千方, 透支,  ▁Breaking,  ▁bang,  ▁fart, hai,  表现得,  ▁Practices, ▁lac,  ▁valuation, 入命, eno, ▁Nak,  ▁spear, 旺运,  浦东新区, 心系, FROM, ▁Laf, erset, ▁VAT, 平遥,  扫二维码,  ▁INTER,  违法违规行为, 谴责, 起跑, . '▁▁▁▁▁▁▁▁▁▁▁▁▁,  ▁integrating, 是把,  ▁activate,  ▁terrifying,  疫情输入,  ▁sanctuary, 减小, RF,  便宜的, Winter,  更多精彩内容, 值为, 道口, ophone, 誓词, alore,  部位的, wave, 风波, Hope,  放在第一位, ( !如有侵权请联系我们删除,  ▁famed, Roll,  ▁Payment,  保留了, 印证, 所为, ▁RB, ▁Buc,  ▁Libraries, 血的,  是一部, 温室, 迎春,  新增无症状感染者, 普陀, 该国, 别让, sta,  都是非常, 上过,  几个人,  路公交车,  Database, 水浒,  ▁immersive, Kn, 清查,  近两年,  啊啊啊啊,  ▁Wesley, ▁{{,  快递员, visors-  ▁Flying- 穿了-  ▁symbolic- 人就- 脑洞-  ▁Crawford-  的一线-  动脉硬化-  ▁segreg -  ▁Compliance - 跳绳 - iffany -  农商银行 - route- 日系-  个月后-  ▁insult- 望远- 碾压-  进行报名-  ▁northeast- 两千- nesty- 绿植-  中央政治局-  ▁alloc- 泡茶- nation-  海底捞-  内部的-  实现的- 公款 -  代表了!-  这个名字"- rences#- swick$-  ▁Rout%- ées&-  和政府'- TikTok(- 水体)-  ▁mosqu*- 作则+-  智慧的,-  ▁Greene-- 骨盆.-  ▁herbs/-  比较好的0-  ▁doubts1-  越来越多的人2-  未来发展3- 巴掌4-  ▁lazy5-  ▁monuments6- atri7-  ▁catches8- 二百9- 接续:- át;- 上是<-  ▁Laur=-  创造性>-  ▁quir?- 廉价@- smartA- ClinB-  有多种C- 茶园D-  冬虫夏草E- ▁akaF-  间盘突出G-  ▁GiantH-  ▁EarlierI-  ▁suburbsJ- 局的K-  上来看L- 道题M- AnnaN- ▁TapO-  ▁averagedP-  以内的Q-  汽车的R- ріS- 强弱T- 拉长U- seatV- 刊发W-  ▁HerbertX-  ▁lightningY- 古法Z- 天王[- 家境\- 且有]- Login^-  随着时间的_-  演唱会`-  ▁antiquea- 篮子b-  在面对c-  自由的d- anase-  招生专业f- ienneg- 武冈h- MODi-  ▁Discussionj-  ▁Certificationk- Linuxl- 折射m- 整容n-  同舟共济o-  熟练使用p-  现役军人q-  ▁mentoringr-  签订了s-  速度快t-  可以考虑u-  岁的孩子v-  ▁Nokiaw- 定格x-  ▁coordinatesy- 绞痛z- asser{-  ▁Created|-  城街道}-  ▁uncons~- iPad-  ▁advisors- 延误- 香辣-  巡视员-  把我们-  有意向- 秋招-  ▁podium- ▁př-  感染的-  第三代-  进校园- )))-  开放时- alez- 化了-  ▁Computing- Nob- 我叫- 江东-  Jonathan-  ▁Kardash- 量和-  存在问题- birth- 疲倦-  Episode-  Statement-  ▁Achievement- 只狼- 察看-  就是命令-  ▁Albany- 而无-  青年的- 人中-  摄图网-  的消费-  企业文化- щи- 围墙- 邮编-  Founded- 嘱咐-  与自己-  为了防止- 他曾- 新老-  高尔夫-  ▁valve-  门市价-  ▁acre-  ▁Device-  不包括-  克萨斯-  参保人员-  ▁sheriff- exit- 冤枉- 常在- igel-  深入人心- tools- 酒业-  ▁Sharma- 熟的-  research-  新媒体中心- 之举- 等功- scopy-  ▁WiFi-  ▁baked- 照耀-  ▁Olivia-  subscribe-  ▁Wellness- 相通-  ▁Belgian-  经调查-  职位要求- olute- 医嘱-  了一款-  请广大市民- 利落- 严守- 出水- 贴身-  统筹做好- 并按-  在很多- Tips-  ▁catering- 截然- 用火-  美好生活的-  认真做好- 奇的- 甄嬛-  playing- 或不-  ▁spam-  ▁indef-  ▁Driving-  同道合- Mate-  官方微信-  志愿服务活动- 配比- nyder- ogh-  ▁Depart-  rapeutics- 铁矿- pex-  已经很- 保底- 新书- ▁PU- ▁REC-  ▁injustice-  顺其自然-  ▁Horizon-  医疗物资- 中暑-  各地要- 打得- 陪着.  ▁Trustees.  不要轻易. 中大.  刘德华. 这幅.  彰显了.  自己的身体.  ▁inevitably.  又可以 .  门户网站 . thouse .  少先队 .  配合做好 .  Profile. 药方. 通气. loe.  ▁pest.  ▁undefined. 来宾. terns. 婚纱. ▁fet.  ▁downloading.  ▁Yard.  Definition.  founded. 较量.  分布式. wz. 便能. Jam . Chat!.  ▁Ecuador". 不争#.  的地位$. 点后%. xxxx&.  ▁Administrative'. 下令(. aunder).  ▁refuses*. ان+. たの,.  资源和-. 塔利..  Represent/.  成一个0.  ▁Mick1.  ▁flaws2. 所辖3.  研究室4.  ▁Vikings5.  拥有一个6.  本赛季7. ipur8.  招聘工作9.  ▁boutique:. 寂静;.  要进行<. fax=. Device>. 矿产?. 线性@. heaA.  ▁crustB.  ▁GrantsC.  ▁restraD.  ▁tangibleE.  万众一心F. 酵素G.  独一无二的H. IgI. 得其J.  在其中K. ▁ReeL. 此人M. 筋骨N.  有些事O.  玛拉沁P.  ▁proneQ.  自己的情绪R. 后退S. ▁LtT. 眼底U.  优秀作品V.  ▁EugeneW. orateX. ▁BleY.  ▁TerritoryZ. 坏的[. 多糖\.  同时对]. alli^. oser_. 先来`.  ▁Playinga.  金银花b.  也逐渐c. 沟村d. 王室e. 腹腔f.  资源共享g.  ▁cluesh. mci. 静态j. 华尔k. 激增l.  ▁rebellm. 就怕n. Middleo. 的的p.  ▁outlinesq. reedr. 张玉s.  ▁keywordst. 亦可u. 再三v. 找了w. 深山x. 龙舟y. 老死z.  无数人{.  erville|. 朦胧}. 内置~.  ▁Athletics.  ▁interdisciplinary. 向左.  这一步. Money.  alities.  任正非.  centric. 防滑. 怀里. 自述. 丛林. 方丈. 油盐.  一个小时.  ▁popul.  高三年级.  的人和.  ▁parenting.  司令员.  ▁LOVE. 雨中.  healthy. 石膏. unders.  ▁populated.  ▁cooperative. must. dh.  公安民警. 弱点. 预见. 张开. 满载.  综合考虑.  ▁potent.  ▁tomatoes.  任何责任.  治安管理处罚法.  ▁Nearly. 主业. 平的. 礼拜.  医学科.  ▁enjoyment. 保值.  凝聚力. 胚胎. 深海. 找你. 用具.  各环节. iani. 素颜. 省属.  引导孩子.  ▁Omar. 大枣.  依法追究刑事责任. 亮丽.  ▁prevalence. 后世.  ▁Jenny.  ▁eaten.  事件的.  战争中. tte.  ▁reopen. ugu. 菜肴. 开行.  是怎样的.  ▁Luxury. 黔南. 前置.  日子里.  竞争力的.  中国工程院院士. 解压.  Resources. 刻画. 唐三.  的胜利. 推免.  的特色. 上桌.  ▁Mang.  ▁histories. 突围.  ouncing. ABS.  防护口罩. 推崇. mma.  给了我. 外包.  沟通交流.  东南部.  也有一些. radio.  继续保持.  ▁penn. ▁Rey.  riminal.  指战员.  工作职责.  ▁Secondary. 意为.  ▁depicted.  自己的努力. 危难. Move.  ▁Subst. % 实现中华民族伟大复兴. ▁Kre. Marg. 并就.  company. shell.  ▁Classical.  ▁Processing. buster.  ▁Coleman.  ▁Employee. 消退. 而他.  ▁aster.  与实践.  观众的.  ▁Slovak/ 更像/ GPS/ rag/  战斗堡垒/ 格力/  的模样/  督导组/ Bean/  玛拉沁信息网 / 老歌 / 还为 / DIR /  ▁Till /  ▁floral/  ▁socket/ ▁uma/  ▁webinar/  ▁Insights/ 建功/  健康教育/ 直通/  南阳市/  开放式/ 读写/ ▁ecc/  ▁compositions/ gong/ ▁Gy/ 保育/ 上人/  ▁Speaking/  的理论 /  ▁Davidson!/ 补缺"/  ▁multimedia#/  报名后$/  ▁opioid%/ 抗旱&/  ▁goodness'/ 外伤(/ ▁RM)/  儿童医院*/  ▁Winston+/  ▁ballet,/  ▁что-/  碳水化合物./  分享给大家// 不善0/  应有尽有1/ 创下2/ 嘌呤3/  ▁polished4/  ▁authorization5/ 一战6/  vitational7/  后遗症8/ 理疗9/  成年人的:/  ▁spont;/ 山脉/  和创新?/  今天下午@/ 聘任A/ anskB/ 关羽C/ 末端D/ 油条E/  是否符合F/ 防卫G/ ▁RPGH/  全民健身I/  能力提升J/ itlesK/  ▁GrassL/  ▁PedroM/ 亲眼N/ 节俭O/  可以获得P/  每天都要Q/ feltR/  治疗方案S/  新时代的T/  ▁MintU/  食物的V/  只是为了W/  每天晚上X/  精彩回顾Y/  隔离点Z/  注册资本[/  你想要\/ wart]/  ▁Ware^/ 水彩_/ Lesson`/  ▁Usuallya/ utscheb/ 玲珑c/ 嫦娥d/  ▁minte/  ▁Editorialf/ 赵薇g/ ciah/  ▁glovesi/ 居室j/ 反感k/ 幼稚l/  agneticm/ 手可n/  ▁detailingo/ 匹克p/  一帆风顺q/  ▁hatsr/ shirtss/  ▁issuingt/ AFPu/  上海交通大学v/ ▁ouw/ ▁ERAx/  受欢迎的y/  ▁riskyz/  ▁honesty{/  里程碑|/ 善恶}/ ▁Bug~/ schema/  千千万/  来说是/ 教招/  第一单元/ reads/  ▁COMM/  ▁potato/  ▁Zhang/  angerous/ 朱德/  这说明/ street/  ▁mistaken/  影视剧/ 卡尔/ 渔民/  ▁canon/ 传动/ 浮动/  干事创业/  ▁Obviously/ 懒得/  视频来源/  criptor/  ▁Cherry/ 从重/  ▁Struct/ Answ/ anut/ odynam/  是时候/ anol/  届毕业生/  的姿态/  ▁strangers/  ▁collaborating/ 西山/  各级各类学校/  ▁conced/ 升降/ 变好/ along/ 急忙/ 甜甜/  ▁feared/  家长要/ Stan/  全面开展/  商务局/ Tell/ native/ 冲天/ 逃犯/  在爱情/ 打听/ 村委/ 校企/ Ven/ ▁IE/  ▁dumb/ 平区/  ▁plasma/ 严寒/ :// 俱进/  传承人/  提供任何/  ▁Richards/ 中还/ tool/ 庆典/ 现存/  中国教育/  从高到低/ 化作/ 杂粮/  与社会/ ▁Wol/ still/  明确提出/ 立春/  将依法/  设计了/ 好久/ ств/ 年均/  深深地/  也不错/  性问题/  学习时间/  ▁Diane/  ▁obsessed/  网联系/ 制服/ 波浪/  刘某某/  发朋友圈/  好好学习/  ▁waist/  ▁Administrator/ 冷库/ ▁PG/ ▁Ore/ ▁emp/ 无恙/  去面对/  没有被/ ▁MTV/  一个新的/  ▁Wellington/ 乡愁/  ▁Policies/  ▁postponed/  ▁overlooking/  及作者/  ▁wished/ 企鹅/  一手抓/ 最优/  发行的/ 万起/  ▁skiing/  的能量/ 公约/  ▁constitutes/  优秀传统文化/  廊坊市/  高素质/ 一例0 风靡0  和单位0  参与活动0 肖像0 豌豆0  ▁Pilot0 ▁lbs0  深刻认识0  孩子的成长 0 唠叨 0  ▁Cleaning 0 所见 0  在香港 0 去买0  信号灯0  我们再0 也说0  任何问题0  olescent0 臭味0 sonian0 放管0 incre0  ▁prolific0  在此之前0  ▁lang0 Pool0  安全生产工作0 PLE0  志同道合0  第十九0  ▁myriad 0  从今天!0  时左右"0  getElement#0  ▁keynote$0 老妈%0  装配式&0 capt'0 大黄(0 张力)0 ▁FOX*0 ▁SMS+0  身体条件,0  ▁relaxation-0  占据了.0  中央党校/0 旧改00  钢结构10  运营管理20 保供30 动向40  并结合50 包车60  话不多70  Standard80 不买90 对华:0 日复;0  二季度<0  ▁gasoline=0 养心>0 oha?0  ▁calculation@0  ▁OrganizationsA0 但仍B0 ▁KolC0  饮酒后D0  ▁PragueE0 采纳F0  更好的自己G0 书单H0 jenI0  ▁homemadeJ0 不许K0 梦境L0 IFEM0 轻症N0  课程的O0 poraP0  ▁MitchQ0  ▁exceptionallyR0 飞的S0 肩部T0  休息时间U0  提供服务V0 istonW0  ▁merchantX0  详见附件Y0  ▁northwestZ0 tong[0  小分队\0 ▁Cod]0 水煎^0  ▁barrels_0  所有人的`0 怀旧a0 饱含b0  之后再c0  社会治安d0  ▁utterlye0  多喝水f0  ▁spillg0 军团h0 小结i0  ▁hectj0  ▁indictk0  ▁engineeredl0 ilightm0 祛除n0 浇筑o0 ➤　p0 羞涩q0  ▁Evangelr0 paids0  有多少人t0  ▁scholarlyu0  主人公v0 布斯w0  严格要求x0 声声y0  embourgz0 某的{0  还不知道|0 接线}0  ▁erupt~0  ▁Finals0 reath0 主场0 化瘀0  好声音0 应以0  ▁privileged0  营养丰富0 Arthur0  分别在0 rame0  ▁aforementioned0  ▁ransom0 孤儿0 甘油0 首日0 欢声0 Liter0  社区工作者0  遇到过0 ▁Eff0 报社0 瓶子0  ▁themed0  ▁Betty0  iciaries0  森林防火0  用地面积0  的财富0  ▁flick0 导语0 汹涌0 艾叶0  ▁Pelosi0  学专业0  中到大0 修缮0  ▁grandparents0 呐喊0 金陵0  坐落于0 的苦0  ▁Arms0  是什么样的0  ▁satellites0  这里面0  各地各校0  ▁WWII0  ▁Manuel0  提高免疫力0 acters0  的一样0  ▁sphere0  涉嫌严重违纪违法0  ▁peripher0 坐姿0 lace0 thews0  是关键0  胡萝卜素0 谈及0 '])0  了一名0  songwriter0 cot0 ▁Hir0 朝气0  的高端0  ▁eBay0 一回0  doctoral0  ▁happier0 好学0 Da0  在身上0 memory0  ▁retrieve0 嘉定0  ▁pointer0  绝不是0  ▁Highlights0 涌泉0 erem0  ▁infinite0  在此基础上0 夺得0  有一部分0 满洲0 qt0  ▁prey0  省委组织部0  制定的0 午睡0 火炬0  体系的0  年轻的时候0 深切0  ▁José0  ▁Strange0 诸法0  的物质0  小伙子0  ▁compile0  ▁denying0  ▁forecasts0 ▁BAS0  也更加0 扬帆0  相对应0 fact0  摄入量0  北京的0  综艺节目0  打交道0  Changed0 语数0  毕业年级0 Images0 盆腔0  简单来说0 蔚来0  从玛拉沁信息网0  ▁americ0 想把0 lters0 融洽0  ▁compromised0  女人味0  带给你0  一路上1  这套书1 善举1  故意杀人1 肛门1  活力的1 荡荡1  本文内容1  特殊教育1 极强 1  不排除 1  ▁Keys 1  产品和 1  吉林市 1  大货车1 спо1 ▁HO1  水资源1  ▁wasted1 ▁SY1  ▁Celtic1 ifax1  ▁tissues1  对生活1  甚至还有1  盲目的1  空间啊1 Justin1  ▁registry1 offcn1  successful1  ▁Mold1 飞船 1  ▁eternal!1  ▁sticking"1 EF#1  最后一天$1 这辆%1  招生工作&1  ▁washed'1 化州(1  的意识)1  以下几个*1  有很多人+1 入股,1  始建于-1  的搭配.1  ▁pumps/1 实录01  即可进入11 Widget21 √　31 TX41 中对51 在同61  原作者所有71  ▁aspirations81 化验91 马尔:1  rospective;1 起航<1 ▁bud=1 是真>1  无数次?1 ajax@1  生命危险A1 很短B1  发达国家C1 浸润D1  ▁baptE1  ▁capitalismF1  ▁AustraliansG1 科教H1 电量I1  学习群J1  小可爱K1 进项L1  ▁kickingM1  ▁investigativeN1  能帮助O1  户籍所在地P1 又说Q1 itutedR1 audeS1 果农T1  ▁reproductionU1 讲了V1 ▁âW1 的酒X1  ▁PortsY1 比利Z1  ▁Designed[1 米路\1  优秀人才]1 田径^1  ▁Trevor_1  参与方式`1  签订劳动合同a1  lastingb1  物资储备c1 复议d1 astae1  ▁shortsf1  ▁Demong1 积攒h1 Rosi1  ▁wearsj1  ▁doctoralk1 缠绕l1  与企业m1  顾之忧n1  openhageno1 伸直p1 肿痛q1 宿州r1  新篇章s1  ▁Biket1 的李u1 坏人v1 摘下w1 精子x1  ▁philosophicaly1 冥想z1 朝天{1  从而导致|1 妄想}1 西昌~1 近乎1 碧玉1  监督管理局1  多云转1 PLUS1  默默地1  公安部门1 花落1  hentication1  第三名1  ▁Owner1  ▁impart1 ®,1  的宝宝1  专科学校1 点儿1 从上1  原生态1  补短板1  ▁Investors1  ▁proactive1  ▁negligence1  走下去1 ichi1 逆向1  ▁calcium1 整套1  报名登记表1  ▁spans1  ▁contracted1  沧州市1  党政机关1  ▁Terminal1 Town1  的训练1  可使用1  就不用1  比例为1 ▁Sou1 PPP1  ▁Russians1  更年期1  时间管理1  统一安排1  你还在1  ▁rope1 ANK1  农村人居环境1 挂碍1 bons1 孤立1 sth1  告诉她1 ▁MIN1 Kate1  ▁Baron1 让步1  人体内1  商业模式1 对本1 求精1  ▁ankle1  ▁Donna1 跳楼1  要提高1 Vector1 heid1  一定程度1 JC1  ▁Edit1 死神1 ▁Sz1  ▁stimulus1  ▁unbeliev1  ▁laughter1  practice1  ▁newborn1  arenthood1  不是你1 楷模1 正财1 被捕1  委员会办公室1 群聊1  claimer1  做的就是1  ▁Chambers1  ▁southwest1  ▁Copy1 Boston1  ▁Voices1  visible1 ▁Ф1 记账1  生活服务1  ▁entre1 苏打1  ▁spike1 ▁Ti1 启迪1 汤圆1  ▁specification1  做了一个1 舟山1 nton1 倒车1  ▁abnormal1 soever1  只有自己1  一直都1  ▁doctrine1 忠于1  公租房1 I B▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁1  ▁Rally1 铿锵1 遍了1  金融科技1  ▁Maintenance1 ▁nas1 umping1  ▁rejection1  ▁promo1 Lib1 alis1 自拍1 hin1  预警信号1  初中数学1 *,2 咬牙2  ▁Reply2 上期2  ▁Crusher2  ▁tokens2  ▁caption2  一百年2  工商联2  经常吃 2 ▁bee 2  专项计划 2  ▁Celebration 2  ▁thunder 2  ▁lakh2  ▁fabrics2 协办2 ▁MW2 中选2 勇往2 bee2 派位2  直接的2  ▁awaiting2   2 先辈2 可与2 防城2 iates2 Copy2  重点场所2 佛像2 平价 2  培养孩子的!2  ▁balancing"2  给我的#2  ▁justified$2 伦理%2 脚跟&2  事情的'2  ▁auch(2  年春季学期)2 hello*2  ▁sourced+2 机床,2 圈的-2  提供一个.2 跑道/2  ▁Cooperation02  testing12  ▁Powered22 运程32  Comments42 身着52 中转62  ▁trilogy72 说你82 分诊92 ▁ETF:2  ▁continuity;2  在其他<2  ▁journeys=2  临沂市>2  ▁Victory?2 ngth@2  主要从事A2 不自B2  东北地区C2  ▁pastaD2  ▁dealershipE2  前行的F2 乳头G2 诱人H2  参观了I2  ▁fathersJ2 ▁AmbK2 + $新型冠状病毒感染肺炎疫情L2 élM2 仙桃N2  二十年O2 女童P2  ▁SettingsQ2 heeR2  lessnessS2  ▁stoleT2 并被U2  民警立即V2  ▁BasinW2  给大家带来X2  从来不会Y2 vaccZ2  各专业[2  实际的\2  ▁Blanc]2  各项防控措施^2  ▁tasked_2  ▁religions`2  资产管理a2  ▁supplyingb2 皆空c2  ▁acceleratedd2  真是太e2  千方百计f2 上交g2 后半h2 始皇i2 面孔j2  ▁Domaink2 独处l2 ICALm2  数据中心n2 此项o2  这是我们p2  ▁Sophieq2 真皮r2  ▁climbeds2 藏品t2 隔夜u2  全家人v2 alamw2  ▁Elsex2  患者在y2 Aroundz2 espoke{2  ▁halfway|2  培训学校}2 优于~2 宇航2 水土2 saving2  ▁Growing2  olation2  爱你的人2 Impro2  ▁removes2 全院2  生物医药2 不屈2 凭着2 顺延2  不支持2  ▁defects2  ▁lengths2 生水2 ▁Nex2  药物治疗2  ской2  中国石油2  ▁Scene2 ▁MR2  ▁gluc2 无言2  的报道2  的材料2  压实责任2 碗中2  战场上2  ▁striker2 悠然2  ▁subsidies2  ▁obsc2  ▁Neither2 丢掉2 日日2 racuse2 cup2  间多云2  ется2 减压2 淮河2 诸暨2  ▁Carnegie2 从天2 命的2 都像2 三万2  ▁Pizza2 nst2  ▁contem2  ▁expenditure2 alion2  不停的2  国家主席2 养活2 links2  通过理财2 喜迎2 ▁SV2  口腔溃疡2  ▁dietary2 二环2 展演2  大专以上学历2 dream2 不问2 头等2 要与2  好奇心2  家长会2  对于一些2  ▁addict2 ▁Enh2  ▁pitched2 况且2 简史2  ▁Wrong2 转告2  ▁amazed2 向他2  思想认识2  umption2  范冰冰2  ▁arbitrary2  ▁nutritional2  在深圳2  永远的2  ▁showcasing2  itchens2  的考试2  ▁discontin2  工业化2 mask2  也很好2 龙的2  疫情扩散2  愉快的2 CHO2  理论上2  共和党2  ▁jersey2 HF2 ▁ze2 大考2 开国2  ▁homelessness2 老区2 ►　2 ATIONS2  ▁accommodations2 ▁});2 Wire2 ovel2  大学毕业2  常说的2  活动策划2  ▁suspicion2 你再2  African2 下同2  members2  ▁Goldman2  ▁exclusion2  ▁grams2 转机2 ▁PD2  下降了2 醒目3  引以为3 何为3 转转3  ▁panor3  ▁Leading3  能够让3 ughed3  ▁Poor3  ircraft 3  你的孩子 3 drive 3  相关法律法规 3 持证 3 柔性3 必考3 meter3  ▁senators3  ▁arrow3 老挝3 护卫3  不变的3  全天候3 警醒3 hered3 虐待3  负责任3  暂行办法3  浪漫的3  subject3  ▁lonely3 *)3 倒霉 3  地方的!3 chart"3 一元#3  李佳琦$3  版本的%3  ▁Dodge&3 ilo'3  所包含(3  少年儿童)3  时刻保持*3 螺纹+3  改变自己,3 tmp-3 陈晓.3 Insp/3  ▁WILL03 ):　13 卖给23 ▁rul33  ▁resorts43  团组织53  和旅游63 中日73  体制改革83  切实维护93  ▁Manila:3  ▁respects;3 千家<3 水乡=3 胡子>3 dar?3 safe@3 ChooseA3 ( !新冠肺炎疫情防控指挥部B3 通了C3  安全保障D3  教学经验E3  ▁rodeF3  什么样的人G3  ▁flavourH3  居民的I3  ▁ScientistsJ3  原标题K3  可以为L3 大势M3  高中阶段N3 EBO3 故居P3 失衡Q3  和省委R3  ▁stemsS3 AreaT3  ▁TorresU3 相貌V3 麦当W3 人以X3 讲坛Y3 喜乐Z3 Pref[3  ▁rage\3  Package]3 远了^3 祈求_3  职业能力`3  ▁midfieldera3 发炎b3  指示精神c3  ▁travellersd3  ▁conceivede3 下坡f3  是不可g3  表现的h3  ▁Itemsi3  ▁glutenj3 声誉k3 andil3 舞弊m3 花果n3 金石o3 ▁gyp3  ▁Hannq3  到目前为止r3  让群众s3  原创作品t3 Forceu3  ▁pertv3  降水量w3  面向全国x3  commenty3 城墙z3  ▁Integrated{3  ▁offence|3 AAA}3  ▁PART~3  ▁grassroots3 标兵3  家乡的3  ▁fixing3  urricanes3 ILD3  是一所3 惦记3 两卫3  错过的3  突出贡献3 多喝3  respect3 Comb3  健康运势3 月末3 棕色3 目镜3 ▁BU3  ▁Coin3  ▁shirts3 を见3 一拍3  Particip3 矿区3 CBS3 oge3  ▁elevation3  台湾地区3  ▁Winners3 类人3 Neill3  ▁orchestr3  中学校3 经脉3 ▁Eat3  ▁billing3  ▁dysfunction3  ▁intentionally3  这四个3  ▁jeans3  ▁blanket3  ▁eventual3 ▁BI3  ▁Armenia3  的外观3  干细胞3  可以让你3  ▁teamed3  选择性3 乐清3 新民3  或者在3 军营3 不稳3 所含3  ▁Darwin3  ▁empowerment3  ▁ال3  社交平台3  ▁gases3  ▁dessert3  ▁Forever3 巨人3  介绍说3  ▁murders3 小暑3  技能人才3  ▁Wear3 后才3 繁多3  行政管理3 折合3  无条件3 Mother3  折不扣3  清楚地3  发挥作用3  ▁Ratings3  应急演练3  ▁beverages3 顺水3  但也是3  ▁humour3  ▁Dodgers3 渔船3  ▁reluctant3 жно3 Db3 大头3  ▁spinal3  ▁similarities3  ▁volunteering3 太快3  手机上3  ▁Hang3  amental3  构成了3  了他的3  ▁crises3 眼霜3  代表性的3  持续更新3 Eth3  影像学3 usage3  ▁cops3  Callback3 医治3 WP3  ▁presidents3 石灰3  设计方案3 琐碎3 gem3  必修课3 occup3  的重要讲话3  ▁historians3 火势3 热力3 晓彤3 稻草3  这些事3 today3  ▁poised3 古琴3  ▁dull3 assets3  是怎么回事4  是否能4 强身4 电车4 示威4  Directory4  ▁Architects4 Fa4  已经有了4  报名材料 4  第二阶段 4  统筹安排 4  重点是 4 噩梦 4 cdn4  ▁для4 大成4 暗恋4  理解的4  脱贫致富4  ▁Plastic4 陕北4  带薪年假4  构成犯罪的4 绅士4 Fair4 也应4 travel4  ▁patrons4 jie4  在广州4  坚持把4  人民银行 4 年金!4  aternity"4  ▁Vert#4  土地使用权$4  肯德基%4  ▁handler&4 拭子'4  八路军(4  就只能)4 州长*4 flag+4 用以,4 一晚-4 IQ.4 一脚/4 有我04 ▁wip14  ▁ranch24  ▁guitars34 坐下44  ▁weddings54 大伙64 密的74 席卷84  ▁Chicken94  ▁uncovered:4  outheastern;4 动摇<4  ▁infringement=4 拐点>4 浪尖?4 发到@4  学院的A4  灭火器B4  ▁witchC4 故里D4  折优惠E4  情侣头像F4  重点任务G4  ▁NobleH4 历来I4  发出了J4  政府采购K4  ▁floodsL4  立即购买M4 背上N4 ▁BSO4  ▁geographyP4  临床研究Q4  不平衡R4  借款人S4  就会导致T4 semblU4  ▁BarrettV4 滁州W4 树的X4  小技巧Y4  ▁projectionsZ4  都是因为[4  ▁Rosa\4 黄鹤]4  的那样^4 pieces_4 糖分`4  主要是因为a4 眼花b4 十天c4 卷烟d4 如火e4 大手f4 崇德g4 棒的h4  系统和i4  政法委书记j4 有理k4  而是在l4 亚军m4 望去n4 田县o4 教辅p4  第十八q4  ▁germr4  ▁Evenings4  指导和t4  就地过年u4 enziev4  ▁prematurew4  经济运行x4 attiy4 囊肿z4  持续开展{4  ▁cough|4 原名}4  道德经~4  ▁queer4 minent4 去哪4 芥子4  国内疫情4 背包4 平坦4  健康中国4 腾腾4 扔掉4 领衔4 ▁KB4 Integ4 仇恨4  医保基金4  ▁WARRAN4  ▁Wealth4  daughter4  ▁Equality4  第一部4  服务质量4  Tickets4 标示4  路街道4 ▁AK4  公认的4  建议收藏4  感染风险4  ▁integer4  ▁Artificial4 防盗4  ▁hormone4  induced4 needed4 ATES4 jh4 谢绝4  我只是4  印象深刻4  Virtual4  ▁Default4 ovi4  的顺序4 领航4 adors4  ▁skept4 ilot4 光谷4  还不能4 所幸4  视频教程4  ▁blogging4  图片来自4  ▁coff4 耐用4  与时俱进4 拉法4  ▁Hampton4 凉的4 aeda4  ▁vest4  ▁skies4  ▁commodity4  ▁Bash4  完成任务4  城中村4  ▁Cinem4  ▁policym4  ▁się4  ▁Regular4 宋朝4  exports4 丹东4 更难4  纯手工4  传播途径4 杏仁4  粉丝福利4  ▁Marion4  ▁Harbour4  ▁Legislative4  的人文4  活动和4 lio4 直系4 省的4  就是我4  请直接4  现实生活中的4 Career4  ▁Cornwall4 维亚4 Boy4  所以要4  全面贯彻4  ▁fres4  ▁Nixon4  ▁satur4  ▁collects4  注意啦4  皮肤病4  ▁Demand4  ▁stab4 lif4  缴费年限4  在日常生活中4  的婚姻4 粉末4  系列的4  年的时候4 烈日4 释迦4 ści4 谷物4  充满着4  ▁Glad4  ▁Venture4  ▁Previously4 ▁fli4 Parser4  ▁aspiring4 老是4 硬度4 ritz4 ▁toe4  警务辅助4 水煮4 考了4  打疫苗4  ▁gesture5 BG5 跳出5  学习习惯5  eliness5 透视5  的一场5 缓和5 弟兄5  ▁Flow 5  ▁Strategies 5 桥区 5  交流平台 5 OLED 5  ▁pictured5 空虚5  为建设5  部门联合5 不入5 赠与5  延迟退休5  统筹协调5  ▁Gardner5  ▁embracing5 筹码5 诉说5  ▁mega5  图文来源5 不增5  ▁Quad5  etically5  ▁cancers5  ▁realizing 5 地势!5 ERROR"5  委员会副主任#5  冲锋在前$5  加班加点%5 KY&5 云集'5 文安(5 车里)5 ubb*5  拟录取+5  ▁physi,5  ▁Cotton-5 叫我.5  爱心人士/5  ▁Legion05  ▁affiliates15 丰盛25 换新35  家家户户45  创作者55  诠释了65 为爱75 显微85 拳头95  农村的:5  Articles;5 ▁FCC<5 油门=5 下线>5  ▁meteor?5  市中心医院@5 gasA5  ▁jointsB5 黑板C5 jeeD5 ▁ISTE5 opensF5  ForeignG5  的病人H5 无忌I5  ▁PatentJ5 腹肌K5 军舰L5 花粉M5 椰子N5 黑豆O5  水产品P5 雷霆Q5  南北通透R5 flightS5  疫情之下T5  ▁RustU5 严管V5  统一战线W5 潜心X5 inhoY5  无限的Z5 条约[5 公分\5 烫伤]5 tta^5 品读_5 小额`5  排名前a5  县委宣传部b5  常态化防控c5 avourd5  ▁Giane5 跳水f5  orderedg5 好感h5 润发i5 编导j5  ▁presumablyk5  进一步强化l5  ▁semifm5  ▁Kazakhstann5  ▁retainingo5 就走p5  ▁Signsq5  ▁abortr5 HMs5  ▁Wendyt5 Liberu5  科学与工程v5 谈心w5  ▁Crimx5 毛发y5 居多z5 石窟{5  对于我们|5 之多}5  的现实~5  切实提高5  ▁appealed5 权重5 Ba5 应为5 频发5  个人卫生5  锻炼身体5  ▁sovereignty5  赶赴现场5 台北5 啦啦5 Rom5 UTF5  ▁patches5  ▁validate5 这将5  本文为5 groups5  大剧院5  泡水喝5  重难点5  ▁Miguel5  即使在5  ▁princess5  ▁thickness5  离退休5  ▁adjustable5  坚定理想信念5 ingle5  不久前5  坏习惯5  ▁traction5 inian5  人与人之间5  及时的5 udeau5  ▁Hamas5  最关键5  理论和5  的违法行为5  ▁Important5 起于5 定力5 铸就5  coholic5  张照片5 Court5  ▁itiner5 衰弱5 breaks5  的方式来5  ▁Claud5 光灯5 极速5  东北部5 恶魔5 选修5 DVD5  必须有5  属鼠人5  锡林郭勒5 旧爱5  企业职工5  解决办法5 光华5 预检5 一课5  可行性5  帮我们5  功能障碍5  留守儿童5  ▁Accident5  ▁Weap5  rossover5 Gs5 铁军5  几十万5  ▁bipartisan5 些许5  ▁Shut5  ▁Collective5  六安市5 一封5  这种现象5 潜移5 لى5 通胀5  ▁renamed5  丝绸之路5 丹麦5 嘉峪5  ▁Powers5 盖子5 增至5  综合服务5 )|5  等诸多5  ▁Shane5  淄博市5 要好5  ▁über5 深爱5 青花5 做点5 图画5 ▁awe5  权威发布5  ▁dile5  设立了5 На5 AMS5  党工委书记5  ▁twentieth5  与应用5  ▁slowing5 老抽5 ETS5  ▁Steps5  两部分5  ▁shadows5 ("#5 菜鸟5  ▁configured5  踏踏实5  ▁skate5  ▁apprentices5  ▁governmental5 XML5 在山6 ▁Sak6  Pakistan6  ▁coating6  supported6 各家6  ▁Tucker6  ▁grands6 冷饮6  功能性 6  ▁Average 6 盲盒 6 定量 6 要走 6  日早上6  黑眼圈6 rais6 城投6 版纳6  ▁synthesis6  正式发布6  理论知识6  的勇气6  ourable6  ▁ladder6 压轴6  莫名其6  有害物质6  全员核酸检测6  食品安全国家标准6  ▁Christina6 kW6 单招 6  综合能力!6 世音"6 厚道#6 ▁QB$6 Guard%6 波兰&6  看起来很'6  负面影响(6  ▁Models)6  なたの*6  ▁Syracuse+6  规模的,6  收费公路-6  相比之下.6 在人/6 之父06 微粒16  高新技术企业26  ▁Cardiff36  ▁Titans46  正式上线56  ▁elephant66 Meta76  ▁Turb86  ▁starters96 违停:6 icus;6  ermanent<6  ▁predictable=6  ▁Azerbaijan>6 数是?6  ▁Mets@6  ▁TemplateA6 五蕴B6  ▁conservativesC6 らんD6 ▁BriE6  ▁WongF6 一环G6  ▁AmberH6 极力I6  很长时间J6 loopK6  提起公诉L6  ▁AlexandriaM6  ▁uncertaintiesN6  ▁appellO6  ▁restrictionP6 小金Q6  刺激性R6  不要错过S6 遮阳T6  有名的U6 以轻V6  ▁picnicW6  红色文化X6 马虎Y6  ▁outrightZ6  生命力[6  第二大\6  营造出]6  工资面议^6 武则_6  跟我说`6  ▁Platea6 家暴b6 餐后c6  打工人d6 ▁Were6  潜移默f6 advg6 ▁SMEh6  州市公安局i6  ▁Dependingj6  终于在k6 RECTl6 FDm6 odiedn6 VOo6 小龙p6  的一个重要q6  就是把r6 首款s6  ▁robberyt6 桃李u6 示意v6 modalw6  ▁irregularx6 而她y6  ▁freezingz6  陈先生{6 握手|6 镶嵌}6  为一个~6  ▁undermine6  多媒体6  ▁uncommon6 岸边6  ▁tactical6  无锡市6 rz6  准备好了吗6 dz6  ▁firefighters6 主卧6  售楼处6  ▁stretches6 照护6  ▁McCl6 诗经6  图文来自网络6  你不能6  社区工作人员6 带您6  ▁baseline6 销毁6  考生可6 之都6  西城区6  我和我的6  高质量的6  ▁Originally6  生肖蛇6 塑形6  换届选举6 多层6  月份的6 复盘6 elo6 lower6 宴会6  ▁Erin6 birds6 点上6 usions6  ▁shuttle6  没有必要6 负有6 aleigh6  ▁inserted6  ▁organisms6  还是挺6 空闲6  ▁pitching6 马某6 Pier6  视觉上6  ▁Chrom6  ▁inacc6 ▁ot6 mania6  general6 按期6 防病6 淹没6  ▁Poss6 武帝6  结婚了6  ▁authenticity6 ▁Dre6  ▁Cookie6 争当6  重点区域6  ▁societal6 年幼6  反过来6 爆了6  的分析6 列出6 绯闻6  不可抗6 国安6 芭蕾6 蜜桃6 Kind6 航行6  自己对6  ▁Belfast6 为新6  ▁Dining6  与美国6  ▁excel6  ▁membrane6  ▁synd6  无理由6  在历史6 irling6 一班6 网球6 RMB6 陇南6  了解详情6  努力奋斗6 不进6 长兴6 ▁UE6  ▁raid6  ▁Carrie6  ▁Verizon6 四楼6 Admin6  ▁pestic6  ▁intermediate6 乡土6 放映6  ▁Avoid6 elic6  ▁radius6 一棵6 将该6 oby6 益处6 锦鲤6 lead6 港股6  ▁Schne6  ▁Browns6 如画6 盛会6 医者6  ▁erad6 潜水6 精明6 匮乏7 现住7 弯道7 footer7 志强7 Conc7  ▁Genesis7  ▁Adapt7 Occ7 àn 7 鼻涕 7  容易引起 7 储物 7 Tal 7  ▁од7  ▁economist7 invent7 linked7  ▁Sloven7 做菜7 末尾7 顺滑7 flies7  ▁internally7  家庭经济7  告诉他们7 叙事7 INTER7 兵力7 听闻7  数据统计7  这世上7  ▁retiring 7  通货膨胀!7  ▁Publisher"7 侵蚀#7  ▁generosity$7 不需%7  学信网&7  传染源'7  过不去(7  ▁rentals)7  痛苦的*7 没用+7  集中的,7  ▁vacant-7  医科大学附属.7  的他们/7 伯特07  ▁Symposium17 魔力27 tok37  speaking47 ▁Gym57  ▁mansion67  ▁defenders77 utz87  ▁Firefox97 ATT:7  特别说明;7 byter<7  ▁dentist=7  ▁paperwork>7  ▁ceremonies?7  青春的@7 一觉A7 西街B7  れがあC7  警方提醒D7 raulicE7  ▁strainsF7 てごG7 砂糖H7 ikuI7  ▁AustrianJ7 ultanK7  过日子L7  会越来越M7  疫情当前N7  ▁asleepO7 ほらP7 AffQ7 cartR7  ▁StandingS7 经受T7 两句U7 劣势V7 大洋W7 过剩X7  都是从Y7 inlineZ7 也挺[7 同源\7 tti]7 Mom^7  综合实力_7  ▁contexts`7 ▁PSTa7  ▁orthb7  建议在c7 浦区d7 呼叫e7 蜡烛f7 ▁heyg7 asperh7 电价i7  成绩查询j7 锻造k7  的星座l7 Factm7 ▁Lazn7  ▁antibioticso7  为防止p7  ▁Pillq7 人缘r7  公安交警s7  ▁Tiert7  项目中u7  睡眠时间v7 帮您w7 玄武x7 POSy7 文玩z7 Zone{7  ▁Sounds|7 超长}7 遗失~7 社科7  困难群众7 萌芽7  不重要7 Browse7  ▁Sherman7  ▁Examples7 功劳7 祛斑7  ▁Medium7  向中国7  祝福语7 lectic7  builder7  てごらん7 ILITY7  起来吧7  ▁deliveries7 )),7  和行为7 收支7  ▁Whitney7 内膜7 推开7  就业的7 写照7 得当7 永安7 而起7  不太好7  十三届全国7  ▁cinematic7  为了方便7  ▁Dominican7  就一定7  纪念日7  不经意间7 扩容7  ▁Welfare7  ▁randomly7  ▁switches7 出事7 涵养7 ▁Ek7  开展以来7  ▁Position7 rika7 说要7 甲方7  れがあなたの7  让我们的7  ▁deaf7  ▁appla7 吾悦7  跟不上7 ▁mp7 洗车7 的门7  都是由7  ▁Cheap7 vable7  关心的问题7  ▁seekers7 边防7 辅食7 VAC7 良药7 鸡毛7  ▁Hers7  做到的7  ▁columnist7  ▁Judy7 哭着7  condition7 Ho7 有礼7  ▁Syndrome7 paced7 views7 HY7 amba7  ▁Chance7  ▁thereafter7  等业务7  贫困村7  ▁occurrence7 小林7  求职招聘7  ▁comfortably7  肝硬化7  利用职务7 owment7 一气7 挂着7  动人的7 哭闹7 插图7 签发7 接壤7  意见和建议7  stories7  ▁cocaine7  丰富了7 ▁Vot7  ▁reinforce7  周边的7 跨国7  ▁wheelchair7  sharing7 ▁Tut7  即将到来7  好好珍惜7  ▁laughed7  ▁multitude7  ▁seamlessly7  ▁attracts7 关村7 吊带7 赡养7  ▁helmet7  ▁Challenges7 涉疫7 屋内7  ▁Rare7  ▁alpha7 与月7  都会在7  ▁hazards7  ▁screaming7 亲吻8 两层8 ▁DS8 流的8  ▁termination8 请战8 ▁quo8 agging8 Capt8  主义者 8  浪费时间 8 查漏 8  影响着 8  舒适度 8  然后将8  ▁chooses8  进入考场8 背负8  支持你8  ▁atop8  第二季8  ▁router8  ▁Bangkok8 下山8  公共区域8  ▁introduct8  ▁astonishing8 amma8 beck8  工程的8  篇小说8  ▁inexpensive8  都是免费订阅 8  公务员录用体检!8  ▁scanning"8  副产品#8  强制性$8  如何才能%8 scold&8  ▁verbal'8 交大(8 约占)8  ▁servants*8  ▁Carey+8  健康生活,8  无数的-8  三室两厅.8  Congress/8  要高度重视08 西风18  神秘的28  generated38 ▁Lon48 黑头58 ods68  这三种78  ▁destructive88  抗疫情98  ▁hazardous:8  ▁Maya;8  各族人民<8 metry=8  ▁ruin>8  ▁preceded?8 职教@8  一切的A8 磨练B8  ▁blossC8  presidentD8  ▁CONTE8 僵尸F8  低收入G8 athaH8  ▁metropolitanI8 coolJ8  岁以后K8  ▁dellL8  ▁brakeM8  ▁MiltonN8  ▁AutomotiveO8 声中P8  大部分地区Q8  ▁payloadR8 RogS8 ′,T8 增产U8 等我V8  政府和W8 灭亡X8 正视Y8  是完全Z8  将出现[8 就已\8  万块钱]8 小黄^8  ▁Persian_8  无处不在`8 月季a8  日之前b8 挺拔c8  ▁Pushd8 胜地e8 rokesf8  生态文明建设g8 唱响h8  才有了i8 bralj8 ooksk8  ▁Disasterl8 打球m8  零容忍n8 想来o8  小哥哥p8  ottenhamq8 变幻r8 锦州s8  ▁Lifestylet8 着吃u8 逼迫v8 贪图w8  及应用x8  秦始皇y8  送给你z8 立意{8  ▁kernel|8  ▁bonuses}8  ▁southeast~8  Analysis8  不那么8 ynamic8  ▁sings8  成为中国8  中央电视台8  ▁Providence8 ELD8 armed8  粉红色8  房屋征收8 ▁sax8 抉择8  ▁Bronx8 富翁8  一直被8 ▁tin8 钟情8  疫情对8  ▁premiered8 Ax8 哈利8 步走8 面具8 出走8  ▁Favorite8 数为8 会后8 话剧8  ▁Spin8  ▁granite8  ▁Landscape8  ▁Documentary8 用钱8  报考人数8  getElementById8 机智8  明年初8  ▁oceans8  ▁skull8 越秀8 ranch8 中俄8 园艺8 跻身8 rano8  ▁Alcohol8 隐隐8  ▁Cairo8  ▁pianist8 ▁Lap8  ▁plots8  多公里8  ▁embry8 Sure8 ijk8  ▁Lightning8 美妙8  之外的8 Artist8  操作系统8  ▁unaware8  ▁Underground8 国债8 独有8  甚至有8  ▁laptops8 再审8 nerg8  ▁competit8 она8  每天都会有分享8  ▁Molecular8 乐意8  ▁Handbook8 ▁pas8  希望自己8  ▁HTTP8 中段8  极强的8  的这些8  放管服8  潜伏期8 uffs8 为大8  勇往直8 化和8 变局8 多用8 泛滥8 再有8  ▁Spiel8 换位8 ▁FI8  ▁reminiscent8  ▁debts8  正在接受医学观察8 长约8 焕发8 宜兴8 碎了8  defined8  临床症状8  ▁resides8 远近8 相册8 雇主8  协会会员8 ▁'/8 ▁Lok8 Almost8  聊城市8  恢复到8 爱因8 秀丽8 Circ8  Collections8  ▁whatsoever8 短语8 心智8 高能8  同志们8  农业银行8 mass8 寒湿8 派的8 bid8 打掉8 ▁io8  嘉峪关9  迎难而上9  progress9  成功了9 长乐9  刘女士9  新业态9  院校的9  urations9  ▁Respond 9 药效 9 送医 9  ▁reservoir 9 刊物 9  抗击疫情的9 隆回9 挥发9  服务体系9 奥利9  ▁nowadays9 录播9 andin9  ▁regulator9 ▁NT9 associ9  ▁recurring9  指导组9 继而9 肾上9 脚尖9 Delete9 伙食9 储藏 9  等有关!9 留念"9  审查起诉#9  史上最$9 earth%9  各位家长&9  ▁Dash'9 污垢(9 飘逸)9 anu*9  多渠道+9  呼吸系统,9 辨证-9  的评价.9  现代化的/9  ▁intricate09 五华19  我看到29  不会因为39  五脏六腑49 如水59  民政部69  这个过程79 亦复89  ▁Bulld99 丝丝:9 枫叶;9 OPPO<9  ▁Advice=9  ▁Jacobs>9  ▁Hidden?9 春招@9 CATA9 GoingB9 伊犁C9 古风D9  三分钟E9  技术支持F9  ▁shortagesG9  ▁surrenderH9  ▁RailroadI9 古朴J9 殖民K9 linearL9  想一想M9 在和N9 王府O9  香港特别行政区P9 安吉Q9  ▁HortR9 贵妃S9  ▁CallsT9  ▁RiversideU9 加紧V9 琉璃W9  ▁vaultX9 ioxidY9  ▁FlowersZ9  不同程度[9 Indeed\9 红领]9  马鞍山^9  物理学_9 better`9  ▁inadequatea9 罪恶b9 轰动c9 Damd9  紧张的e9  ▁Authorsf9  人生在世g9  ▁Merrh9 ▁Зi9 大卫j9 胸痛k9  故意伤害l9 queuem9  ▁HREFn9  ▁reuniono9 暖和p9 streq9  结构化面试r9  Williamss9  ▁avent9 hofu9 心烦v9 民歌w9 石河x9 科的y9 Terz9 极少{9  ▁Lounge|9 deen}9  ▁suffers~9  ▁Meta9 强强9 __(9  ▁Sell9 回望9  的小孩9 ▁exh9 体操9  ▁Bahrain9 爆出9 abl9  ▁clicks9 花都9  合格证9 质朴9  心理学家9 插座9  浏览全文9  被曝光9 应收9  大道与9 ▁IPO9  ▁sorted9 你可9 abc9 如一9  也是对9  报名地点9 之门9  会更好9  练习题9  解决实际9  ▁Mayo9  Transport9  ▁Netanyahu9 本品9  的感受9 inqu9 灯塔9  不公平9 ifty9  以此来9  ▁unsure9  ▁correlation9 亲生9 受体9 dm9  了一年9  ▁Node9 入微9  护目镜9  规划的9  ▁insufficient9 exe9 透析9  阜阳市9  有异议9  ▁Premiere9 此案9  小仙女9  停不下来9  ▁sunshine9  ▁storing9 者们9 Covid9 总站9 时被9  ▁Researchers9  ▁Eden9  应用程序9  所以大家9  ▁avec9  这种行为9 本该9  ▁plaintiffs9  ▁Cave9  质量安全9 将就9 雨量9 ▁lig9  ▁shark9  再进行9  ▁Inform9  returns9  ▁remake9 血浆9  和面试9 ospice9  ▁Tang9 ▁Gul9  ▁keyword9 SX9 shape9 GU9 改成9 环评9 ▁ICC9 主妇9 台县9  南京大学9 集美9  金山银9 之久9 罪行9  这座城市9 怪兽9  认可的9  ▁Flood9  ▁Manning9 凶猛9  我们会在9 元年9 ribly9 佳作9 液压9  给大家介绍9  estinal9  ▁smiling9 vn9  招聘教师9  培训中心9  ▁vegetation9 踏青9  ▁terrace9 看作9  接电话9 普工9  意识的9  自家的9 棚改: 涉恶:  战胜疫情:  双胞胎:  的口号: entric: 嚣张: 说她:  在上面:  ▁Northwestern : urdy :  ▁averaging :  通过电话 :  ▁Gerald : 抗压: 支线: 无毒: aye: 之谜:  ▁Sanchez:  人群聚集: emale:  ▁converting: ▁dar: ecd:  欢声笑语:  ▁needle: 舌苔:  总公司:  ▁pulse:  ▁owning:  ▁creations:  ▁Comes : 开奖!:  年休假": 枪支#: 散了$: 航海%:  宣传引导&:  十字路口':  ▁cozy(: 在下): Jos*: ctr+:  ▁investigator,: 奇幻-:  Finding.:  ▁judged/:  和地方0:  勇于担当1:  ▁Lore2: 仍需3: 强健4:  解除隔离5:  上的一6:  ▁workflow7: 徐志8: 还应9: ienced::  调查发现;:  ▁Gateway<: 肺气=:  主要集中在>:  受贿罪?: 他都@:  ▁cliffA:  alleledB:  ▁onsetC: 反倒D: 照见E:  看不懂F:  呼吸机G:  的道路上H:  isionalI:  生肖羊J: проK: ▁TucL:  是不是很M:  浓厚氛围N:  舌尖上的O: letesP: 忧虑Q:  四川大学R:  与本网联系S:  herenceT:  ▁slidingU: 演戏V:  奥特曼W: posalX: stonesY: 骑手Z:  数量的[:  ▁fridge\: 板的]:  一个家庭^: 博大_: 坚硬`:  人性的a:  加入到b: ▁FINc:  也是最d:  交通管理e:  ▁Arrowf: Approg:  ▁maturityh: NavBari:  ▁chasingj:  封闭施工k:  aggeringl: 塘镇m:  ▁confrontedn: 心悦o: 投行p: 法务q: encersr:  Samsungs:  都有了t:  ▁outdatedu:  母亲节v:  estonesw:  ▁Stonesx: ▁Sagy: 瓜子z: 画作{:  ▁jumps|:  具有重要意义}:  具体以~: 每亩: uni:  　…　:  为社会:  最新进展: 瞒报: IRST:  那是因为:  现在我们:  网络直播:  ▁adhere:  ▁useless:  ▁mercy: 凝固: 求救:  拖拉机:  科技学院:  ▁ling:  容易造成:  ▁refusal: 克星:  广泛关注: omical: >):  层面的: shang: 拿了: ECK:  ▁Purple:  斑马线:  ▁cohort: 成大: Submit:  ▁Doctors: 洗漱:  奥林匹克:  ▁births:  ▁Arlington: 关照:  自己所:  以上内容:  ▁orphan:  ▁Evaluation:  ▁decoration:  ▁Sample: 或未:  字节跳动:  扶贫开发:  ▁storyline:  raising:  招聘对象:  ▁Hammer:  男孩子:  ▁snake:  gencies:  ▁Rapids:  ▁monop: 维稳: 鸭蛋:  亿元人民币: 干练:  不怎么:  当成了喜欢:  ▁hype:  就是最好的:  ▁urges: 地黄: 逃脱:  绝对的:  ▁Cisco: ecl:  资料的:  ▁lend:  党组副书记:  不负韶华:  ▁projection: otti: ardon:  到达现场:  保持中立: 回合: 者为: 调养:  去看看: 红叶: 雄厚: LAND:  治疗方法:  ▁infants:  要主动: 欠缺:  urgical: 置疑:  惠州市:  司法解释: 心有:  ▁Designs: JF: 缩减:  ativity: 对在:  巩固拓展:  ▁Arrest:  转载请注明出处:  村党支部书记: RR: 磨砺: 三尺: 夜景: 他很:  本科毕业生:  ▁intra: 雨花:  digital:  服务意识:  ▁Looks:  aterials: 迪拜: 赘肉:  第二款:  ▁warmer: 龙凤:  活动内容:  公元前: 建言: 自拔:  经研究:  ▁insane:  ▁Regulatory: 清北;  ▁petrol; 有幸;  刘先生;  工资的; rared;  在某些;  厚厚的;  ▁Aston; 倒挂 ;  ▁Volunteer ;  好成绩 ;  希望的 ;  ▁Distinguished ;  最爱的;  努力工作;  贫困县;  ▁cherry;  那些事; 手握; 多汁; 过瘾;  ropolis; 展出;  战略性; Fresh; 商洛; ▁va;  加工厂; '>; 有车; 难的;  ▁Whis ; 火龙!; 哔哩";  自信心#;  inosaur$;  ▁cottage%; 下楼&;  filename';  ▁survivor(;  岁那年);  你点的*; 做客+; 写成,; 刑罚-;  创新的.;  如果想要/;  ▁badge0;  ▁corridor1; 挪用2; 黄海3;  ▁educating4;  ▁Surrey5;  ▁rewarded6;  ▁staple7; 宣化8; Kelly9;  ▁Ober:;  ▁Scout;; 叹息<; 兰卡=; 有味>; ▁č?; 快餐@; 护师A;  ▁pillsB;  感兴趣的C; 的五D;  ▁ноE;  人为本F;  ▁testifiedG;  新天地H;  essionalI;  年夜饭J;  ▁hindK;  ▁dissertationL;  ▁springsM;  ▁emphasizeN; 息肉O; 载客P; THERQ;  ▁MapleR;  ▁DoingS;  ▁copiedT; 摒弃U; ▁PGAV;  ▁PrayerW; 活了X;  ▁DoneY; ▁zumZ; 新会[; 简便\; ▁ip];  角度来看^; yssey_;  hearted`;  ▁Businessesa; 摄取b; themedc;  总是会d;  行业内e;  违约金f; opiag;  ▁Ladiesh; 闯关i;  然后把j; 肝经k; 桂圆l; mentalm; 我跟n; 空心o;  ▁prisonerp;  检查组q;  ▁Lessonsr; āns; 偶遇t;  ▁relianceu; 议院v;  culturew;  事业运x;  思维方式y;  ▁chefsz;  ▁nominees{;  ▁disproportion|;  但不限于}; 比比~;  ▁Disability;  ▁negotiation; 制图; 杂乱;  发展趋势;  国外的;  ▁hooked;  ▁umbrella; 最深; 交了;  和家庭; esthes;  ▁pushes; wt;  麦当劳;  文化程度;  向社会公布;  ▁Frankfurt; 抢修;  北京日报;  ▁Jared; 钢材; '];; EMENT;  买了一;  推动了; ▁Tip; 同为;  第二人民医院;  ▁Pierce;  ▁Chocolate; 惊慌; ambi;  培养学生;  ▁Anders;  ▁spinning; onso; 老牌;  感觉很;  ▁Moment; 恭敬;  ▁Constitutional;  历史性;  爱因斯坦;  ▁horrific;  ▁unlawful;  ▁adul; 开元;  ▁Blind;  ▁Organic; 得体; 要吃; cards;  ▁incidence; 避暑; ▁TB; 我与;  期限内; Roman; acerb; 冰冷; 却说; 暂无; 送一; 土建; loo;  管理等;  ▁reside;  ▁parental; 却很;  居住的;  制度和;  建立在;  ▁revelation; 跌倒; 领奖; 用上; ODO;  老人家; 耀眼;  一旦出现;  不要因为; 要领;  ▁digging; yg; 驱蚊;  一年后;  非常大的;  manship;  ▁Pfizer;  优良传统;  ▁referenced; 保税;  ▁virtue; blood;  ▁Brunswick;  details; ▁DAY; 松动;  非必要不; 选考;  也不想;  ▁lender;  禄劝禄劝禄劝禄劝; 掉落;  就无法;  电视机; 代为;  买不起;  那么大;  ▁advising; 酒泉; LES; 奠基; 欣欣; 虚荣;  风险的; 内搭;  含答案; lore;  嘉年华; arna;  ▁stripped; 着火;  ▁styling;  动画片;  ▁Label;  ▁Farmers;  热烈的;  padding< ▁Von<  素质教育< anting<  ▁infer<  ▁Includes< Bern<  视觉效果<  ▁balloon< 饭前 <  ▁imposing <  经济效益 < 跑去 < Steven < 侵占< 陆丰<  ▁finalists<  教师队伍<  aundering<  ▁checkout<  市场上的< urous< 文联<  参赛选手<  Customer< 攻读< ishu< 家禽<  互联互通<  孩子学习<  前公示<  ▁practitioner<  ▁Allan < 绕城!< 马蹄"<  商业街#< 螺蛳$< 中生%< 拉你&< 摘帽'<  县教育局(< 禄口)<  作品中*< agher+< 心了,< 说来-<  你有什么.<  的儿童/< 万字0<  安排了1<  ▁throne2<  ▁fundamentally3<  汕头市4<  ▁beers5< 硕果6<  ▁Drum7< 畅游8<  ▁engra9<  综合分析:< ▁Pix;< Sales<<  及要求=< 成药>< 四溢?<  学习方式@<  解决好A< zymeB< ravelC<  releasedD< 大肆E<  大健康F<  ▁HitsG<  ▁BoroughH< 迅猛I<  采访时表示J<  参考价K<  类药物L<  可免费M< 倒数N<  最喜欢的O< 坟墓P< 代金Q< 船上R<  是根据S< ieleT<  ▁sandwichU<  ▁preparationsV< 问诊W< ▁bouX<  转多云Y<  生肖马Z<  ▁Sandra[<  ▁Reason\< 是与]<  ▁reversed^<  建成通车_< amar`<  的年轻人a< 署名b<  ▁superstarc<  呼吸道症状d< 跳槽e< ▁apif<  不能够g< 后有h< 排球i< 王俊j<  体能测试k<  characterl<  ▁jurisdictionsm<  市领导n< otono<  ▁borep< KGq< hider<  是一门s< 就职t< 装卸u<  ”,　v< ahaww<  ▁Fraudx< 成千y< 龙城z<  ▁beneficiaries{< MU|< Sol}< 往常~<  在一线<  ▁fulfilled< 抗病< 目睹<  ▁contests<  已经不是< detail<  ▁municipalities< 同理<  ▁laps<  新冠肺炎病例<  director<  ▁cholesterol< ▁Wo< 支书< beta< 人设<  大力弘扬<  ▁headquartered< 长廊<  ▁redund<  resistant< ▁AFC<  ochemistry<  不下去< 烤箱< 壁垒< 结实< 全系<  ▁Become<  ▁Ernest<  ▁Classes< 腊肉< 砂石< 鸟类<  房产网< 四天< 管的<  子宫肌瘤<  国家发改委< 中院< 和李< 事变<  当他们< 居高<  ▁balcony< 国强<  潍坊市<  注意安全<  Courtesy<  简历投递<  ▁Mate<  ▁quot<  Founder< 旅馆<  公开发<  去追求<  零售价格<  Running<  ▁Koch<  ▁Flower< jjjj<  ▁cafe<  ▁Volkswagen<  ▁multinational< 双城< 胖子< 为题< 两头< 回信< 比拼< uga<  那么在< 无前< 的在<  ▁tense< % 现将有关事项公告如下<  ▁Hybrid< 谨记<  ▁Caval<  中国在<  ▁aliens< enne< ▁owe< Safe<  催化剂<  密密麻麻<  ▁Bathroom<  ▁prolonged<  你认为< ▁Sau<  国家队< 没过<  ▁imprisonment< ции<  中草药<  动力电池<  志愿服务队<  ▁fried<  ▁belonged<  需要进行<  废弃物< 轮椅<  双方的< 要靠< 不慌< 扣分<  精益求精<  ▁shelters<  的网友< 离异<  卫生室<  ▁conqu<  申请材料<  ▁duck<  ▁clerk<  ▁anticipation< 产的<  对相关< baum< Laura< 放缓< 昂贵<  过很多<  审查调查< secure<  危险废物<  uggling<  ▁Chapman< 金钟=  主题党日活动= 三区=  ▁behave=  ▁offenders=  ▁бу=  ▁Prophet=  ▁metadata= LAN= ▁Sic = ار = 宣教 = 有房 =  ▁Wings =  看过来=  第一家=  最值得=  华中科技大学=  ▁Controller= Jane= finals=  新增疑似病例= 难求=  第二十=  ▁neighbours=  ▁Laud=  ▁sovereign= ▁kay=  ▁atomic= 搞得= 某人= ▁BG=  大学习 = 驻足!=  亲和力"=  ▁stun#= placed$= 到我%=  包头市&=  时时刻'=  改变的(=  林业局)=  ▁focal*= 去往+= 多伦,= 大喊-=  优秀奖.=  市融媒体中心/=  ▁Apparently0=  ▁proceeding1= 代代2=  重金属3=  全域旅游4= 州委5= 积木6= 交会7= Cand8= 附带9=  美味的:=  ▁Conversation;= ég<= VIS==  无意中>=  这两年?=  collection@=  学期的A=  信息港B=  ▁BrittC=  性关系D=  有所不同E= 一村F= ADEG=  ▁SlowH=  ▁stretchingI=  ▁worthwhileJ= 奔向K=  ▁SilvaL=  ▁proposesM= 听的N= 外教O=  于一体的P=  我们看到Q=  JapaneseR= 大财S=  安排的T=  生肖猴U=  ▁LindsayV= 爱美W=  统编版语文X= 好比Y= 月累Z= 相克[= 闵行\=  出生在]= ibo^=  ▁Spart_=  collect`=  发挥好a=  最常见b= 特聘c=  ▁Rodgersd= ▁cave=  ▁Requf=  利益的g= phantsh=  肾上腺i=  ▁unintj=  ▁Guatemalak=  都有可能l=  级旅游景区m= muchn=  ▁hassleo= Potp=  对未来q= 日积r=  历史悠久s= 补给t=  节目的u=  或多或少v=  ▁Gonew=  延缓衰老x=  ▁conductory=  ▁Dressz= 阴霾{= stru|=  ▁Cedar}= ▁Jal~=  先进典型= icos=  ▁twins=  ▁Santiago=  ▁Migration= faces=  ▁exhausted=  住宅小区=  ▁Kingston=  ▁acceleration= Bry= 你把=  课程中=  ▁strang= 之又= 拨付= 舍弃=  马上就要=  下定决心=  ▁Franch= 小雪= 过重=  ▁maneu= Upon= arag= 行贿= 强求= 扑灭= AZ=  向他们=  ▁adap=  ▁autonomy= 两室=  ▁Alger=  ▁Origin= ifs=  ▁Conflict=  增加值= 到人=  ▁Mercy=  ▁urgency=  本职工作=  发送到=  这几种=  ▁Curry= Xml=  高性能=  ▁Mald=  ▁compliant=  Christopher=  ▁vocabulary= monary=  ▁rugged=  评论区留言=  ▁hasht= 常人=  来临之际= 看他=  消费品=  站出来= Tool=  不知所=  ▁haul=  生不灭=  中国市场=  各项措施= 木斯= Ball=  ▁dominance= etric= staff= offset=  ▁inspires=  可以促进= 像我=  高清壁纸= ucket= ▁IA= 应酬=  建设银行= {}=  生肖龙= ▁Kaw=  ▁sanction= 解开= ▁zur= 大钱=  日晚间=  珠三角= ctx=  ▁Muse=  ▁asthma= 丙烯=  电子烟= 收视= 限流=  ▁Catholics= 再接= Paris=  生活保障=  ▁consolidation=  出去玩=  ▁Youtube= 尖叫= 幼师=  ▁rebels=  代言人=  更安全=  予以公示=  征收土地=  时间久了= 严把=  ▁melody=  咨询师= chu=  房屋详情= 精简=  还表示= HIV=  婚姻的=  ▁Crus=  ▁Choir= 容城= 关头=  ▁Paulo= Ready=  ▁Echo=  ▁knees= 那句=  ▁marsh>  ustainable>  popular>  ▁prolifer> 出镜>  中学业>  就很难> 爽口>  的改变> ortal >  ▁candle > ▁nap >  ▁insured > 测序 > 煮沸>  ▁Anch>  ▁Famous>  ▁Probably> 凉水> 镇静> spoken>  ▁observers>  ▁fals> 绝了>  租赁住房> 自选>  本平台将>  ▁Buhari>  ▁Inspector>  德云社>  ▁miracle>  什么东西>  ▁Scale > 带去!>  王一博">  气候变化#> 陆地$>  ▁Hungarian%>  主要内容&>  ▁beast'> 邪淫(> olor)>  ▁rooft*>  ▁Battery+>  ▁unacceptable,>  ▁Roma-> 最受.> 亲民/> ја0>  高强度1>  ▁goodbye2> 当她3>  度过了4> olith5> 鲜血6>  得天独7> 壁画8>  高级人民法院9> 相接:> ▁Ace;> 鱼儿<>  家里人=> 大打>> 路至?>  的阳光@>  但是现在A>  ▁sleekB>  ▁GazetteC> 凝胶D> 大为E> 诺如F> SupG> 气囊H>  时强调I> beenJ>  ▁FalconK> 穿刺L> ▁×M> ▁ozN> % 人力资源和社会保障厅O>  希望在P> lawsQ> 伟业R> 武功S>  发展需要T>  ▁crushedU> 步履V> 要到W>  稳步推进X> 牛市Y>  ▁KnownZ>  ▁podcasts[>  ▁probation\> 无心]>  国家电网^>  执业药师_> 山羊`>  关注度a>  初学者b>  ▁Monumentc> 出海d> ▁MARe> 与年f> 喜怒g>  ▁chunkh>  活动中心i>  命运共同体j>  ▁prosek>  ▁Georgetownl>  并不多m> 看透n> 光环o>  曾多次p>  考试总成绩q> 坐拥r>  工作计划s>  比较大的t> Sandu> borgv>  ▁Snyderw>  老人们x>  核酸检测阳性y>  重要的作用z>  ▁sadness{> ▁bog|>  工作餐}> 练就~>  理所当然> 预览> 宴请> sheng> 不值> 麻疹> Begin> 莱芜>  ▁cosmetic> omez>  敲诈勒索>  民族大学> Topics> 发觉>  调查显示> andem>  ▁Allison>  ▁Plymouth> 湘江>  考试内容>  ▁neut>  特此公告> 懒人> 软骨>  增长了>  不是说> STEM> ▁Wii>  卫生健康局> 筑梦> 伤感>  静静地> 沙拉>  一定不能> 共生> 发霉> 口才> 负压>  增值税专用发票> ▁PER> 小妹> 珍爱> ▁[">  ▁Rash>  ▁Maxwell> Dele>  原因就是>  counter>  ▁defic>  ▁coral>  ▁genome> Crit>  ▁handsome> 劝返> 口渴>  ▁stimulate>  ▁instructors>  支付的>  向上滚动> 他所>  ▁sprint>  ▁telecommunications> [[> 杨紫> 橙子> 剧集>  独生子女> Sad>  万公里> Volume>  的联系>  妇产科>  已经成为了>  ▁compression> 抛物> 航道> 苗木>  但是由于>  ▁Churchill> FW>  做事情>  现场的>  ▁manipulation> 暗沉> ▁Ski>  ▁Frost>  ▁suites>  ▁insists> 革新>  国家战略>  idelity>  ▁obsession> aisy>  车流量>  与法治> 蓝莓> 饱受> BAR>  开始啦> 易错> 马丁>  多一份> tyard>  一一一一一一一一> 内蒙>  不得了>  生肖猪> iances> 小鸟> PCR>  utterstock> 月度> 牧师>  abstract>  ▁Monitoring> 乘务> Flo>  ▁Subscribe> 情深>  理解为>  技术员>  ▁Bulletin> 打完> 朱某> 上不> 薏米>  却不知道>  本地确诊病例>  冬天的>  绝不能? 交出?  就显得? Native? 救济?  会让人?  这是个?  localhost?  ▁Lancaster?  Experience ?  你不知道 ?  依次递补 ?  理财知识 ? xj ? ▁VAL? 产销?  时时刻刻? 彼岸? 落入?  ▁Winnipeg? 邮轮? Mu?  消化吸收? 构件? 抚恤?  第二条?  风景名胜?  积极向上? 友善?  整个过程?  essential?  ▁maternal? 只看 ?  划分为!? 重现"? eye#?  的象征$?  提供优质%?  ▁collar&? mL'?  了一段时间(?  检查站)? 首开*?  ▁Maggie+? 救了,? 曾有-?  罕见的.? Switch/?  校外培训0?  质量管理1?  ▁entirety2?  新疆维吾尔3?  ▁habitats4?  这部电影5? 将领6? 每每7? 五金8?  ▁weighing9? 奴隶:? 谈起;?  Commerce? 诚恳?? 赵本@?  的一件事A?  componentsB? 再将C?  体育运动D?  优秀共产党员E?  ▁EuropaF? 短线G?  ▁visualsH?  ▁humidityI? ▁CAPJ?  ▁wokeK? 二号L? 帐号M?  ▁PiratesN?  ▁staffingO?  ▁PrettyP?  balanceQ?  ▁KuwaitR?  另一位S? 皆有T? 豆类U? 买个V? 用眼W? 男童X? 盼望Y? 小肠Z?  把握住[?  ▁Option\? 回路]?  多层次^?  ▁ribbon_?  ▁Natalie`? Carola? 在南b? ▁enlc?  基因组d?  并取得e? 你很f? 虫子g?  了一把h?  当前的i? oquej? youngk? 接过l?  世界各地m? Civiln? 屠杀o? 苍白p? 落脚q? 个班r? 莱坞s?  价格在t?  但这些u?  用行动v?  ▁Meanw? 防风x?  中国画y? Monthz?  ▁improper{? 葱姜|? 木头}? Must~?  作者所有? 卖掉? 气泡? 还与?  ▁Invent?  资质的?  日积月累?  ▁Timber?  ▁revital?  老年痴呆? 出世?  语言文字?  ▁Hearing?  ▁translates? prep? 丽莎? Role?  ▁Example?  ▁distributor? 植被?  ▁envelope? 如因?  ▁weed? Bat? adj? 外貌? intech?  ▁prince? 他要? 方圆?  ▁Flynn? 出借?  买买买? urdue? 韧带? ebe? 年里? 金斯?  中国传统? 黄陂?  级以上? 熬制? 立业?  ▁irres?  ▁Resume?  ▁Telegraph? 淡泊? won?  以确保?  一起学习?  安全有序? 述职?  实体店?  很多人在?  ▁Auburn? temp?  ▁Zimmer? dollar?  ▁slate? 当家?  ▁restoring? Hong?  ▁fireworks? 镜片?  ▁majors?  ▁ignoring?  ▁renovated?  犯罪分子?  ▁modular? 绚烂?  诊疗方案? 荣幸? 敬佩? 车友?  货币政策?  ▁runway?  ▁servant? 代购? 哈酒? 酒的? ramid? lux? 四中? 回流? 东街? 乘法?  ▁photographic? 本就?  血液中?  语言表达? 安国? 罪的? 家装? 心眼? 现价? 醇厚?  的问候?  的第一步? кт?  很长一段时间?  无障碍?  ▁Salem? 王的? 预判? fg? 构思?  反映出? eros?  ▁exagger?  ▁Norton? 悦读?  投融资?  只不过是?  ▁expansive? 比尔?  的部位?  ▁Hutch?  ▁equitable? 无几? Bytes? 纲目?  仪式上?  loading? 打分? 江镇? Thus? 美颜? anan?  ▁dashboard? 就到@  等技术@  ▁Sacred@ 出息@ 钢丝@  最强的@  ▁Mell@  ▁municipality@  ▁representations@  ▁tailor @  ▁ventures @  ▁perceptions @  解除劳动合同 @  ▁Rings @  ▁Preston@ 嘴角@ toire@ 成全@  总而言之@  ▁parked@ 再创@ 稻田@  ▁Castro@  ▁merged@ 怨恨@  学习平台@  不会被@  ▁Witch@  ▁drift@  ▁airplane@ ▁BP@ Stars@ brahim @  作风建设!@  违法犯罪活动"@  ▁insulin#@ 乐山$@  切实做到%@  市面上的&@ 之急'@ 姊妹(@  ▁Improvement)@ 参战*@  事业编+@  ▁Planned,@ loaded-@ 不愁.@ 山口/@  为主体0@  ruitment1@  柳州市2@  ▁Numbers3@  ▁Copenhagen4@  服务商5@ ▁IMP6@  ▁Tate7@ rimp8@  十二条9@ 创文:@ 酱汁;@  ▁Trin<@  ▁Divine=@  ▁greens>@ iguous?@  ▁Featuring@@ 参演A@ 巨星B@ EasyC@  ▁ButtonD@ 爱了E@ BasicF@  比较少G@ ▁dugH@  ouncillorI@  ▁ImmunJ@ FINK@ 一至L@ 总署M@ NOWN@  成功率O@ eenthP@  ▁StrikeQ@  ▁MatthewsR@  开学第一课S@  ▁SelectionT@ 界定U@  项目位于V@  CongratulationsW@ zzaX@  投标人Y@  有关系Z@  ▁thanked[@  ▁reductions\@  ▁LIVE]@ feel^@  ▁palette_@  卡拉胶`@  项目为a@ 按键b@  ▁unpredictablec@  和组织d@  ▁Classicse@ 句句f@ uchig@ ▁Rewh@ 心律i@ 杀伤j@ 师长k@ 穿插l@ ▁Spym@  ▁procn@ 将他o@  ▁Regardlessp@ 心痛q@  教育家r@ 书目s@  的花朵t@ 翻阅u@ 就给v@  也因为w@  ▁sensiblex@ 中签y@ 鸳鸯z@ ▁Wet{@  ▁reboot|@ 群主}@ 赞誉~@  关于进一步加强@  或暗示@  接到报警@  ▁Molly@  心理素质@  转载目的在于@  ▁dispat@ score@ 午休@ 积电@  一年一@ ibe@  报告的@  ▁volatile@ 群岛@  研究显示@  ▁wast@ 学法@ xd@ 海湾@ 迈入@  ▁это@ 养牛@ elman@  旅游区@  团队协作@ 凝结@  ▁lungs@  ▁processors@  大大小@  ▁Said@  的一道@ inee@ metric@ agi@  ▁Indonesian@ 嗅觉@ ungs@ 楼宇@ 郊区@ iso@  ▁insulation@  乌鲁木齐市@ 薪水@ estly@ 评测@  才明白@ Africa@  ▁Tests@ IPT@  Executive@  ricting@  查漏补缺@  责任单位@  ▁Login@  ▁Ка@  学习中@ 白露@  比自己@  集团股份有限公司@  ▁intercept@ 腰酸@  承担责任@  的人是@  费用由@  ▁surreal@ 别提@ 辅料@  ▁Cardinal@  在这次@ 疤痕@  增添了@  只要我们@  Whatever@  印发的@ ▁POL@  女主角@  ▁Wish@  ▁mighty@  ▁iPod@ 对已@  ▁scare@  ▁microwave@ гу@ 一派@ 粗粮@ 乞丐@ 路由@  国家统计局@  上个月@ 他有@ 黄土@  ▁roofing@  是用来@ 如您@  ▁puzzles@  ▁stretched@ 心悸@  和使命@  强烈推荐@ 解冻@  ▁Guer@ 奔腾@ 游行@  Furthermore@ ftime@ 高举@ Gary@  ▁Erik@  的工具@  ▁validity@ 断奶@ 上将@ junit@  给你一个@ 失职@ 松鼠@  打麻将@ Iter@ cular@ 院落@  的成分@ dens@ brance@ 增压@  就回房间@  ▁reconciliation@ tops@  主营业务@  ▁ExceptA  理性的A  有机肥A 甘蔗A 重拳A  prisinglyA 有苦A  特点是A  的岁月A  ▁Calcul A  ▁syntax A  ▁coupons A 得来 A Joy A svgA  ▁microphoneA  经营场所A  ▁tomatoA 东经A  就业岗位A playedA RubA 而喻A 之歌A 最短A 京城A  自己的力量A 李家A lliA 在西A  创造出A  存在感A 体格 A  ▁Answer!A 乐乐"A  就如同#A 竹林$A 被罚%A  ▁commercially&A 劳作'A 尾号(A  高校的)A clair*A €™+A iya,A ▁Thu-A  ▁Investments.A 新药/A  等手段0A 保单1A 刺绣2A  ▁confined3A 从那4A  调整后5A 拿地6A  合格者7A  ▁Griffith8A 装机9A 户晓:A 松江;A  ▁violating    A 查出?A  让生活@A 周星AA  深入浅BA  ▁sprawCA uatingDA 宝玉EA  ▁charismFA NHGA  井冈山HA  ▁dareIA  ▁AboveJA  ▁synthesKA 杀猪LA  电视电话MA  ▁DartNA  请及时与我们联系OA 四驱PA 小到QA 方子RA 管住SA  ▁archaeologicalTA 雇佣UA ianneVA 幼升WA 汤匙XA  ▁adequatelyYA міZA 与前[A 海滩\A 网校]A  ▁altitude^A  的状况_A  ▁Approach`A ▁levaA  不生不灭bA 零下cA 鼠标dA ;B  赔偿责任?B  ▁unconscious@B  ▁compassionateAB  的使命BB ▁DESCB  监督电话DB 展馆EB  各行业FB 差额GB  ▁stripsHB 野味IB ▁ShiJB  ▁ShawnKB OmLB 马尾MB  淋巴结NB ▁lobOB  ▁bentPB  primaryQB  ▁vocalistRB 止损SB  分区分TB 环城UB  怎么了VB  抓住了WB 建平XB  保护法YB  努力实现ZB  不完全统计[B 姨妈\B 新县]B  ▁Lyon^B  菩提萨_B Lake`B  ▁consultancyaB AlanbB  经济区cB  总有一天dB  新增治愈出院病例eB 放得fB  一个女孩gB  ▁SensehB ▁vagiB  CapitaljB  万余人kB  属鸡人lB KImB 路虎nB  合作协议oB  ▁lyricpB 气愤qB  提质增效rB 改动sB irentB mathbbuB 柜台vB  ▁insurerswB 五项xB 稀少yB  经济带zB  镇党委{B  政策规定|B 熏陶}B  checked~B 月考B 焕然B  聪明的B  letcherB  ▁RhodesB 山体B  不仅在B  的最好B  产业结构B ferredB 莅临B  有三个B  的那个人B QCB 政局B  ▁aidsB  人力资源管理B 蕾丝B  中国铁路B  相关资料B 和张B  实质性B  生态系统B 车程B  ▁labsB 反驳B  千年的B  ▁mentorsB 段的B 防洪B  骨子里B  工作地址B 省事B 请点B  ▁templesB 惨烈B  largestB  ▁LilyB  ▁exquisiteB  微视频B  的乐趣B suppB  政策解读B 毁了B  特别喜欢B  ▁NobodyB ▁BeaB  新兴产业B  不按规定B  ▁aggressionB  ▁accumulatedB 使劲B 病灶B  掌握的B ocomB  经常在B  ▁blowingB 餐补B 花在B  继承人B ▁ACLB  销售人员B 病魔B 四项B vicB LargeB  毕竟是B ▁uitB 商界B ▁OleB TIESB  的身材B  ▁leaksB  ▁trajectoryB  自然界B 双减B 她对B  渐渐地B  免费培训B  人员密集场所B 闹钟B  AdvancedB  ▁traumaticB 元帅B 划片B 都以B 玩意B 笑笑B 省力B 小众B 小子B  可靠的B  ▁teachingsB 神兽B буB 争相B  可以进行B  ▁violateB  ▁underwentB  全面排查B anicalB ▁FenB  ▁HyundaiB heedB 一锅B 一应B CutB 操盘B  ▁dismantB  paralleledB 均是B 郁闷B 粘稠B ▁MüB  喻户晓B  菏泽市B porterB  ▁applesB 何以B  establishedB  的人格B aurusB  ▁phenomenalB  有部分B  长期使用B  专项整治行动B agentB ypseB  ▁SantosB 开示C  也可以在C  ▁NielC  ▁subwayC 唾液C  风景线C MexicoC PATHC TraceC 隐秘 C  ▁Kerala C  ▁Streets C " 房地产开发有限公司 C  ▁poets C 鹦鹉C  CollectC  有关事宜C  问题来了C  ▁homageC  ▁GermansC 急躁C  全国性C  最有效的C 管局C  看不起C  休闲娱乐C ▁NVC  模式的C  ▁HamburgC  ▁apparelC  ▁ChangingC  为有效C ▁Dot C  相信自己!C 二类"C 申购#C  爱尔兰$C  人以上%C Wait&C  ▁pleas'C  ▁Reasons(C 旁观)C 蓝牙*C  方法是+C 每家,C  ▁Proceedings-C  ▁picturesque.C  也有着/C ICH0C  Therefore1C  摄氏度2C 赶超3C oji4C  ▁furnish5C  会自动6C  长长的7C 害羞8C  如果出现9C 国宝:C 毛细;C MultiC 句型?C  为公司@C  ▁clustersAC  inementBC  ▁fascinatedCC  ▁SternDC  内容及图片均EC  结果的FC  还有哪些GC 低位HC  自然人IC  运动鞋JC 前去KC  还有就是LC ▁OrtMC  结婚后NC DoctorOC  ▁apparPC  ▁denomQC  WritingRC  ▁grammarSC  RetrievedTC 缺钱UC 脑筋VC  ▁EssentialWC 南下XC 思义YC  国家对ZC erns[C 步枪\C  增长的]C Accept^C  ▁advertisement_C 俯卧`C  散发出aC 痛哭bC  台积电cC  ▁DynamicsdC 境地eC embfC kregC  经验分享hC 展区iC  命运的jC 躲在kC ▁CWlC  观世音mC ICOnC  要结合oC  技术有限公司pC 一发qC 一架rC 大口sC 考编tC 二路uC 若无vC  喝一杯wC  资本主义xC 经文yC  司法部zC  ▁Types{C 访友|C  ▁Tenn}C  ▁splash~C  预示着C  动员大会C 就对C SeanC  ▁boroughC  ClassesC  尽量选择C  现场确认C  ▁pentC ~(C  非常简单C  是人生C  过往车辆C  ▁bunkC 但因C 寥寥C 气流C  处三年C ollahC  ▁betsC  关注后C  民族才能C  塔利班C 定州C 盆地C ALLYC  不得报考C  ▁shootingsC  ▁RelationshipC  职业资格证书C  ▁juvenileC ▁masC  获取更多C  ▁brochC panelC 弹出C  各学科C  ▁jetsC 史记C 朝南C  被别人C FatherC  ▁establishmentsC  商丘市C 音符C ▁NFTC  帮助学生C  osphereC 皆为C 能吃C 便携C  即是空C  ▁DualC  ▁relayC  ▁cocktailsC  IntroductionC  ▁feastC 被害C  ▁whaleC  ▁compulsC  拉法基C tmC  作品集C  肉眼可见C secretC  成功举办C  ▁TracyC  原定于C  ▁cubicC 娘家C  ▁neglectedC naireC 吉安C 寒凉C 有病C  成本的C  紫色的C 三环C 审定C sjC 千百C DriverC ACAC  主干道C  前提是C  从而达到C  ▁profitabilityC 短袖C 走入C 高调C ▁XXC  考试大纲C  ▁jungleC  ▁PurchaseC 史料C GERC ▁jsC 流言C 比方C 生发C 腥味C  第一轮C  疫情就是命令C 金针C  和老师C  ▁AgendaC 貔貅C 冰冻C  有重大C 严苛C  满足女C singleC 路程C  华尔街C  ▁absorbedC ▁GCC 是会C 海滨C  见义勇C  ▁BrasC ▁GETC 皮下C zonC 献策C  眼前一亮C  ▁embassyC  转变为C  ▁brakesC  ▁ShepherdC  都十分D  小姑娘D  ▁GoreD  ▁disgustD  ▁thereofD 跨度D  　)　D  ▁nailsD  白血病D  ▁Doors D  住房和城乡建设局 D  充电桩 D  ▁Aurora D  ▁Offers D 一九D  好日子D  联系了D  市场监督管理D  ▁InterviewsD 之说D ▁RisD  ▁rockyD 宣城D  最开始D  非全日制D WordsD  每一种D ObamaD  ▁FinnishD 不料D 三部D 农耕D  ▁photographed D 卫国!D  忻州市"D  ▁Drugs#D  又不失$D  ▁haunted%D pn&D 柴胡'D 母女(D 靠山)D  才是真正的*D  ▁alphabet+D  ▁Mustang,D  ▁unhappy-D  ▁negotiated.D  ▁illustrates/D 威廉0D 明火1D 温热2D  点击右上角3D 年卡4D  对于那些5D  ▁dataset6D  有明显7D 偷懒8D 附上9D  动植物:D  ▁suburb;D  ▁CannabisD  象征着?D advert@D  强调了AD  ▁BoostBD 车流CD  ▁noticeableDD 葛根ED 流血FD 烹调GD  在英国HD ducingID  ▁richestJD  ographersKD 二甲LD  大小事MD  自己喜欢ND 热恋OD 淡雅PD 订立QD  ▁employingRD 查清SD  博士生导师TD  ▁BowieUD  的疾病VD  专题会议WD  ▁peaksXD  提出申请YD  ▁smokZD 舅舅[D  滨海新区\D  ▁weaknesses]D  的认知^D 挥手_D ▁SIM`D  而这种aD 扑面bD 脱水cD  ▁excursdD  ▁certaintyeD 很差fD  的独特gD  ▁battinghD 结业iD  ParametersjD ▁DeekD  迄今为止lD leighmD  executenD  新局面oD 该村pD  设置为qD 热火rD  挖掘机sD  国家法律法规tD  ▁LutheranuD arethvD  ▁EbolawD  icularlyxD  就一直yD 同日zD  机械化{D  ▁Hull|D 取名}D 吞噬~D  ▁KathyD  ▁InstitutesD 瞩目D  将进行D  在别人D ByteD 加收D  ▁lecturerD  佳木斯D ’)D 交错D 反腐D 搜寻D  医疗资源D  文化传媒D  ▁JongD  ▁MeaningD  ▁WillieD 现为D 含糖D 普宁D CharlD  ▁gemsD  批准的D 花苞D optersD  ▁AdminD суD  ▁ratsD 施教D  行程卡D majorD  你想要的D  ▁negativelyD  ▁pertainingD uccD aiserD  国际合作D  ▁showersD JJD 顾及D ▁WITD 上好D 说课D  不放松D  额外的D garsD layerD  ▁highsD ynnD  服务水平D  ▁drasticallyD  显示出D  等多项D  当事人的D 盖茨D  经得起D BreakD  ▁expiredD  ▁compatibilityD 举起D 待定D 不透D 之功D 的正D ▁bruD  世界各国D FAULTD  教育体育局D 奶酪D 扶手D  ▁picsD 指着D  中央空调D  出院病例D 犬只D  在去年D ▁HepD 人无D  clamationD 不衰D  jectionD OFFD  ▁GretD ynthiaD 比分D  代金券D ▁KotD  不离不弃D  对话框回复D  加起来D  马斯克D itousD  的梦想D  那些事儿D 点钱D 网申D  色即是空D andumD  想起了D  镇人民政府D 留着D  西海岸D ▁ampD  相当的D CartD  ▁DumpD 纸张D  ▁upwardD  ▁benefitedD 门类D  公司和D 拍出D figureD  总有一D  顺风车D  ▁JoeyD  会议由D 工种D  猜你还D  看到一个D  ▁cracksD  ▁famouslyD ▁baD 枝头D  说起来D 拍的D 遥遥D  全体师生D SevenE  ▁gardeningE  并按照E CathE  发展中心E  今天我们就E  ▁depthsE 勤务E  ▁shootsE 背心 E  的地步 E ▁sys E  三伏天 E  boarding E tipE  根本无法E  一生中E  ▁MegaE 棋牌E 鸡翅E  和父母E 点是E  海口市E etchesE  ▁telecomE  ▁fundraiserE  ▁WitnessE 独角E  ▁fertilityE 尼尔E  ▁nastyE  ▁interrogE Coming E  我以为!E 太小"E  激发了#E  handler$E 志向%E  垃圾疯狂&E 裂缝'E _____(E 请示)E 黄牛*E  ▁poured+E  ▁По,E 傲慢-E 变美.E  税前扣除/E 节水0E 西双1E stri2E 悉心3E 溶性4E  带着孩子5E  ▁begg6E 压根7E 该局8E  密切相关9E  ▁initiate:E  可以向;E  老和尚E  元以下罚款?E 似水@E 味觉AE 通航BE  代理商CE  第四步DE  roducingEE  执行官FE  的客户GE  ▁BrunoHE  ▁HugoIE  ▁mythsJE 结冰KE  中长期LE  时间在ME  ▁miscNE 冷落OE ewoodPE  并没有什么QE  RuntimeRE  见义勇为SE  partialTE  ▁harborUE  会变得VE  的笑容WE  ▁percussionXE  ▁InputYE  觉得不错ZE 数次[E 全屋\E Niger]E  ▁elimination^E  中到大雨_E 增减`E 核销aE  ▁JahrbE  ▁VaccinecE  在校期间dE ammadeE 机理fE  ▁pixelgE 副本hE ringsiE  乃至无jE 和气kE 东华lE  ▁honemE ′~nE 吸食oE  沉浸式pE mineqE ColumbrE  ▁CoastalsE  ▁loweredtE  ▁ContinentaluE  称自己vE % 累计追踪到密切接触者wE 烧伤xE  ▁CoursesyE  ▁mitigationzE  也是中国{E  研究部署|E  ▁Hawaiian}E  医院进行~E 较轻E bandsE  NaturalE 伪劣E 车次E packedE  storageE 远比E 并报E  个乡镇E 毁掉E  资金的E  ▁smarterE  ▁ObservatoryE  才能真正E  ▁KarnE ▁TajE  初中英语E  representedE  生肖狗E ▁AchE MountE konE  亚健康E  安全工作E  ▁RoverE  ▁SafariE 因公E 脸型E  清楚了E ▁INDE  ▁obscureE  ▁spellingE  行驶至E  很快就会E  参与的E 明理E  老年人的E boostE  其他事项E  ▁JupiterE 愧疚E takingE LBE HandleE  CouncilE  ▁buildersE 直销E  我们家E  亦复如是E  ▁equalsE  ▁ManitobaE ▁TME ucingE  ▁jewelleryE  ▁councilsE  ▁optimizedE  信息服务E 查证E ikanE  防控要求E  ▁rentedE  哪一种E  亲子关系E  ▁laughsE 白术E heitE ▁BesE  assemblyE BrookE  道路运输E 起舞E  后勤保障E  销售收入E  ▁HollowE 雪糕E  就业服务E gebraE daqE  开通了E  ▁OrdersE  应该在E  的眼睛E  一群人E  国土空间E  ▁strivesE  ▁cigarettesE amazonE  也变得E 择业E  我感觉E  革命的E PennE  名单公布E 大年E  那时的E ▁jedE  ▁abusesE 下雪E  买不到E  ▁strollE  的研发E  这可是E MemoryE 每期E 心与E  设备和E rrrrE ScopeE ocracyE  iatricsE 改名E 代人E  ▁CreatingE  可能会导致E 富民E  开发者E PBE  岗位表E  新生活E  ▁DearE  ▁phenomenaE  十三届E  成立以来E BrandE pleaseE 全款E 郴州E  及时调整E 瑞安F  新变化F 进气F  也是个F GrantF breadF  JacksonF  ▁geometryF 得像F 空相 F ▁Ez F ۥۛ F hig F 枇杷 F  及时处理F  ▁RainbowF 松软F  哈佛大学F  ▁whalesF õesF  灵魂的F  月收入F  告知书F 大发F  销售价格F ▁QUF  还有些F otswF  周年大会F hertyF FlowF  distanceF  ▁openings F  Understanding!F 院所"F  他们将#F sets$F  ▁brokers%F 守候&F 河村'F  营养师(F  ▁weighs)F  Professional*F 钜惠+F  百年华诞,F  ▁deliberate-F  有什么区别.F 圣人/F 大汗0F  ▁memo1F 每名2F  最终的3F preme4F 黄酮5F  企业复工6F  毛主席农村7F  ▁Debut8F  最高人民法院关于9F  只能是:F  ▁Professionals;F 撤回F  ▁Combat?F  ▁Senators@F 污渍AF  ▁settlingBF 介休CF 抱歉DF DISEF  ↓↓↓↓FF  ▁wardrobeGF  ▁PasoHF  集中隔离点IF 单体JF PurKF  使用时LF  ▁JohnstonMF  哈萨克NF  ▁ExportOF  ▁homelandPF kieQF  大暴雨RF  五蕴皆空SF facingTF ildaUF 沉睡VF  周一至周五WF 发呆XF asketsYF 高通ZF  走的路[F 自燃\F Brazil]F  ocalypse^F 重叠_F  丰台区`F  五一假期aF VisionbF  ulatorscF 不紊dF 山洪eF 牛皮fF  以上为gF  ▁overseeshF 高人iF ▁(#jF  ▁endurekF  ▁DucklF  ▁observingmF 小狗nF  很少有人oF 学识pF 栗子qF  保障了rF  提供帮助sF 范儿tF  的关键时期uF 千山vF  都值得wF SOURxF eanoryF  依般若波罗蜜多zF  ▁succession{F  接到了|F 地表}F  ▁Schmidt~F  很高兴F  我们没有F 婚外F 环游F linerF geraldF  ▁handmadeF 怕冷F 标价F 至极F osaursF 变种F 远县F  在身边F  被依法F 单向F  VERSIONF  漳州市F 如荼F  大学排名F  ▁stintF  byterianF  依般若波罗蜜多故F  ▁supernaturalF ramerF istineF  ▁genocideF 深感F ▁GigF  世界第一F  请点这里F ▁PBSF  ▁RapidF :¥F 止于F 甄别F  EnglandF 归宿F  ▁turkeyF  建筑的F  的革命F  判决书F orporF 　*F  ▁professionalismF  ▁McKinF 丹田F 暴食F  不自觉F  聘用的F  ▁sulfF  identifiedF  新概念F 年就F  口罩的F KristF  医院就诊F  ▁rigidF 慈溪F  的好坏F 路基F  试用期满F  ▁sharplyF  危化品F  在国外F  您可以F  早安心语F  掉以轻F  指定地点F athamF  ▁gravelF  ▁insiderF KNF 五角F  在同一F 高超F  ▁headphonesF  ▁austF  ▁comparisonsF  原出处F  协同发展F 百余F ▁StaF uliarF 例为F  不完的F  ▁franklyF 切莫F  的水分F  unsignedF  ▁ContextF  ▁grappF  ▁assumesF  ▁supremeF eyedF  科研人员F  ▁GonzalezF 敦促F berraF  ▁garneredF  ▁hypothesisF 泊位F  ▁CrazyF  合同的F  中国国际F  这么大的F  ▁cakesF  ▁flamesF  ▁insistF 括号F  应根据F  简简单F  ▁thrustF  ▁NordicF  觉得好看F  ▁mosqueF 状的F ulsaF ▁CMSF  ▁loneF  青年志愿者F  建筑师F  有更多F 写好F  的一员F ▁ICEF  ▁ChennaiF 逗留F  十二星座F  ▁SpursG  ▁crushG  组织召开G 刘涛G 自愈G  笑着说G  排出体外G 牵着G 网店G  ▁Edgar G blank G 紧要 G Ford G  最基本的 G  最新资讯G  晋城市G 中指G 代步G 到货G  在一起的G  治疗效果G  emiconductorG 天窗G  风口浪尖G  要来了G 下文G  的重要举措G  TransformG bardG studyG  ▁stalG 英镑G  为全国 G csv!G Visual"G marked#G  规律的$G  明示或暗示%G  ▁crunch&G  ▁Armenian'G 再不(G  关键时期)G  三季度*G  沉重的+G phal,G illas-G 买一.G  ▁closures/G 延展0G  在校学生1G  工业区2G ▁ech3G  消化系统4G punk5G 礼服6G Lady7G  ▁researched8G  ▁volleyball9G 病历:G pound;G  ▁tumorsG 园内?G  文章的@G  上新台阶AG  ▁ProblemsBG 字幕CG 降雪DG 卖到EG  ▁plasticsFG 喜鹊GG  活下去HG ▁GapIG  食欲不振JG  JessicaKG  ▁paradiseLG  ▁questionableMG  空即是NG  ▁tightlyOG 佛说PG  营养不良QG  火车票RG  ▁MonacoSG 之法TG  答题卡UG 同名VG 体彩WG 茶的XG  ▁glowYG QueueZG  driving[G  ▁nationals\G  ▁Irving]G  ▁Acceler^G  联席会议_G  ▁teammate`G  各式各aG  联系微信bG  新发展阶段cG 散户dG 流氓eG  不知道大家fG 胃痛gG  表达对hG  省委党校iG  ▁DrinkjG 兽药kG 肥牛lG 虚寒mG  不断增加nG 一格oG  ▁BellepG 保全qG 选秀rG  手卫生sG ▁PREtG  ▁biblicaluG 肾气vG  不仅会wG htraxG 卖货yG 哈密zG  不参加{G  交通秩序|G  ▁Broncos}G 宗地~G PartyG  ▁shiningG 福鼎G  ▁cautiousG  ▁electrG  ▁neuralG 认准G 高热G  爱一个人G ▁XLG  也不好G  发展方向G  形成一个G  西方国家G asonicG  ▁SuperintendentG 焦急G  如需转载G 皮卡G  ▁MatrixG  ▁CopperG  ▁ReleasedG  ▁sexyG 俯瞰G 九州G  用清水G  ▁EtherG  ▁BoulderG 减产G ▁ZurG PsychG  CharacterG  ▁CatalogueG 乡下G 人造G 较弱G  躺在床上G 口区G 林肯G  注册会计师G bathG  国家一级G  ▁beanG 角逐G  请与我们联系G  ▁excludeG " 人力资源社会保障部G  ▁ResistanceG  PoliticsG 则在G ▁svG  视网膜G  ▁VernonG  ▁lackedG 古迹G  后顾之忧G  GenericG 尖锐G  个方面G BangG leaderG  本文系G  今年上半年G  ▁kindaG aqG coalG 充裕G 党恩G 水管G  上海的G ombiesG 偏离G  由学校G umbsG SweetG  ▁GarrettG 概述G  ▁restingG 偏好G 酷暑G  公共设施G 奖牌G 三文G 阐释G  简简单单G AntiG  ▁safeguardG 路子G 而从G JoshG 内敛G 顾名G 写到G  在网络G  很不错G  酒精含量G  ArgumentG  ▁🙂G  闭环管理G  ▁FactorG 马龙G ECTIONG 费率G ──G gelG  这么好G  人际交往G  ▁HolidaysG  全球的G  ▁PumpG  的人际G  ▁porchG 我一G  ▁ReadersG  ▁lithiumG  这种情况下G  ▁InstallationG 热巴G  大范围G  植物油G  不得转载G 宝山G 写信G 国标G 月在G ▁MauG  需注意G  ▁whitesG  ▁textileG 民进H 犀利H  大家要H  都离不开H  如果你觉得H  ractionsH  ▁BeginningH  ▁smilesH hubH 张伯 H 我这 H  女职工 H  ▁Improve H 可信 H  ▁EpsteinH  的相关规定H  ▁cooperateH  能够帮助H 捐助H  接班人H 悲惨H 报国H  ▁UnknownH  ▁HernandezH 米尔H BibliH  ▁comparesH  客服电话H  publishedH 侦探H 已满H 古树H  ▁ordinance H  ▁Apost!H  ▁onion"H  ▁landlord#H 关火$H  信息采集%H HU&H  加装电梯'H 无语(H 案号)H ▁XP*H ་བ+H 点火,H  去哪里-H  最佳的.H  白萝卜/H  ▁textures0H  ▁Hello1H 外滩2H ▁ú3H  大会堂4H  ▁vitamins5H  设区市6H  ▁Ministers7H 哪有8H Pen9H ▁zoo:H  ▁quiz;H  ▁surplusH  月经不调?H  ▁screened@H  分子的AH  咳嗽等症状BH  ▁depictsCH />DH 亳州EH 骨架FH  淮安市GH  项目负责人HH  ▁judgingIH DuJH 下期KH 全资LH 可按MH  不在于NH 格子OH 求生PH  陈某某QH 诡异RH adalSH  ▁handicTH ۛ۟ۡUH  等特点VH  最大功率WH  在她的XH  失败的YH  ▁deanZH  ▁loos[H  ▁Devils\H  ▁disreg]H "}^H  ▁daunting_H  properties`H  ▁EnterprisesaH % 境外输入无症状感染者bH 近来cH ARMdH DigeH  拟聘用fH  新京报gH  ▁lesbianhH 湖水iH  ▁JailjH 闭上kH staylH 凌云mH 出乎nH  ReferenoH jmpH 打入qH  平顶山rH  的产生sH KidstH  ▁KardashianuH ▁DUIvH  ▁combowH  ▁eveningsxH  无死角yH  提升自己zH 酒水{H 倒地|H isex}H Syn~H  第六届H  一经发现H  开发建设H  一夜之间H  ▁JudeH  内容的准确性H  ▁continentsH  ▁discomfortH  ▁inaugurationH  ▁DunnH  房屋租售H 脚底H  ▁inconvenH 稽查H  的经营H ledonH  生活上H  劳动模范H  ▁CorrespondH  ▁GeographicH 轻柔H ▁TayH  紧密结合H  ▁advisesH  ▁touredH 自已H 积淀H 之交H  不符合规定H veringH ilonH 杰克H  ▁pretendH  方设法H AlwaysH GoldenH  ▁nitrogenH  发生事故H sharedH ECDH 国货H  肚子里H  通过率H 江汉H  ▁reinforcedH :$H 盆底H 选一H riqueH 周到H ранH  ▁grillH  ▁technicianH 挂职H  如何让H  对美国H  ▁RavensH 如梦H ▁KabH 红木H 几下H 静电H tianH  管理人H  ▁orientedH  服务群众H ▁lbH  各有关部门H 平底H 理清H megH  试一试H  已经超过H CleanH  ▁inducedH 遥控H  ▁rebelH  ▁offseasonH  党总支书记H  面试官H 到访H  ▁WilliamsonH  党中央决策部署H  ▁endeavorH  ▁BroadcastingH  ▁PattersonH 首歌H ryslerH  ▁DeborH IronH ▁BUTH  庚子年H  项目名称H  的节奏H  竣工验收H itheH  ▁meticH  做好疫情防控工作H iliaryH  ▁ThinkingH 步道H  并获得H 贮存H  在下面H  ▁FightingH ▁chiH ▁deiH  本文内容及图片均H  范围为H  中华文化H  ▁multidH 随性H  新增确诊H  ▁intimidH 财会H 写意H  thoughtH 边疆H  ▁implantH oosH  共同富裕H ituresH 法和H 大观H 打成H  五星级H  ▁companionsH 张伟H  ▁WarwickH 严明I 诚心I  一切苦厄I  ▁weightsI izardsI 应力I  区委副书记I  ▁hatedI 样样I  人行道 I  ▁Eisen I  操作流程 I  操作工 I ▁Dul I  ▁mangaI  年龄要求I  政务处分I  生活用品I  ▁lucrativeI 毛笔I  日放假I  葡萄牙I  ▁reimburseI 肩上I RecommI 粗心I chetI 太低I  ▁FillI wesI  座城市I  参考文献I  ▁Alison I  ▁lineback!I 对立"I  ▁Forecast#I  ▁Buddh$I  ▁Venus%I  ▁reconstruct&I 前言'I 清河(I  走进了)I 忌讳*I 梨花+I  ▁flooded,I 欢呼-I  样板间.I  ▁Cuomo/I  ▁Franco0I 打死1I  文中观点2I  的人在一起3I 午饭4I 栏杆5I  也包括6I ▁Kom7I  增加到8I " 我都认真当成了喜欢9I 和王:I  能促进;I  ▁remixI ▁ICT?I 恒温@I  ▁CubsAI 强军BI  硝酸盐CI  不适症状DI  小编的EI  知识与FI ▁BIGGI ▁RasHI 独创II  他还是JI  educationKI  umberlandLI  ▁yarnMI 欠款NI  ▁pitchesOI 巡警PI  电子化QI  ▁HospitalsRI  如果不能SI  敬请关注TI  ▁ParkingUI  ▁consortiumVI 云中WI SeaXI  这根弦YI BottZI GEN[I ▁LCD\I  是世界上]I ár^I 海涛_I 艰险`I  考试报名aI  ▁NursebI  拉开了cI ▁suedI  每一份eI  办理相关fI  ▁mutationsgI  顾名思义hI  ▁UniqueiI 货品jI  整理发布kI  designedlI  ▁convictionsmI  ▁FernandonI 的量oI  ▁SurvepI 豆角qI  六盘水rI jpsI BultI  但只要uI  ▁eyebvI  ▁CamerwI ▁sabxI  系安全带yI AlbumzI 名次{I  空即是色|I  ▁surfing}I 安防~I 雨后I  扫码进入I icornI  ▁PreparI  国防部I  地球上I  具体内容I ▁NRI  不收取I 夏至I 蚊虫I  ▁SecureI  ▁RetirementI  更大贡献I 临摹I  县政协I зуI  工作实际I orisI  ▁unparalleledI 乐业I 领券I  ▁hackingI  ▁contractingI 别急I 字帖I readerI  ▁RequirementsI ▁TotI  实施诈骗I udingI 题干I  佩戴安全头盔I  ▁ProviderI 味儿I 浓烈I  ▁LeoneI 可做I 多国I 计税I  ocraticI 写过I  掉以轻心I  已连续I  我们用I 人寿I  ▁hempI  岗位要求的I  ▁AdjustI dimI  主动作为I  尽量少I mansI 超低I  的朋友圈I  ▁SharpI  ▁breachesI  ▁LeatherI  ▁backwardsI 掌心I EmbI  的组合I  ▁polymerI xfI iltersI  ▁incubI 滋滋I  的大量I 一书I  ▁metabolismI 船只I  酒驾醉驾I ouringI  ▁BetaI  ▁firearmI 律所I LewI seeingI  图文来源网络I  ▁EVERI 会以I 着装I  升级改造I  度一切苦厄I 能耗I 补办I  的黄金I  点击下方图片I  archiveI  ▁decoratingI 住行I ruleI UnionI  ▁trunkI ohydI  ▁forkI anzaI 当之I 薪金I TypI  腾讯视频I  ▁catastrophicI  有效果I ▁werI  ▁HonorsI 友人I  学员们I 黑发I  会有所I 一朝I  cellentI 睡得I  想了解I  ▁corpI 在后I  ▁BoltonI  ▁SomersetI 心机I 议程I  积极作用I icansI  区人民法院I  ▁staggeringI 倍增I  ▁RocketI  ▁fairnessI 破财I 缺水I  活动规则I  ▁bansJ  ▁workloadJ  ▁elasticJ  ▁currenciesJ 业障J 深陷J  那么你J landerJ  ▁ArchbishopJ 行书 J 预设 J  为人民服务 J  脑子里 J  也不过 J LOGOJ  ▁ExclusiveJ 演播J 稳岗J  ▁fillsJ  ▁SundaysJ 老李J DropJ ▁XiaJ  风险评估J  ▁KangJ  ▁ResultJ  周星驰J  ▁ductJ  ▁disguJ unerJ  ▁declaringJ 单抗J 加拉 J  也纷纷!J  ▁Facilities"J ysc#J ▁INC$J  无论是在%J  在这个过程中&J 媲美'J 放着(J middle)J 以示*J  却无法+J  ▁Pompe,J  ▁vague-J  ▁athletics.J Amb/J  ▁Consultant0J  与支持1J  本课程2J  ▁Flint3J ▁EB4J  ▁Pharmacy5J 实测6J 海尔7J 而对8J gins9J  薄弱环节:J 井然;J  ▁crossesJ xq?J 开销@J 避让AJ  的游客BJ  科技公司CJ  ▁literalDJ  工作是EJ  ▁ScotiaFJ 又多GJ наяHJ  ▁GlobIJ  ▁AppointJJ  ▁trailersKJ  为加强LJ 登机MJ  ▁wellsNJ  并提出OJ  色不异PJ 听党QJ 耳边RJ "{{SJ brainTJ  则需要UJ  方式和VJ 光亮WJ ▁PVXJ  融入到YJ 钱了ZJ  敌人的[J  热线电话\J  研究方向]J those^J  ▁nickname_J  ▁billionaire`J  一起玩aJ  为阴性bJ  干辣椒cJ  还有个dJ darkeJ  控制分数线fJ  ▁unitegJ  ▁suspensehJ  疫情蔓延iJ  ConsumerjJ ohankJ  表示自己lJ  LicensemJ 都难nJ  ▁multiplayeroJ 不垢pJ HomqJ  请联系小编rJ  ▁abusivesJ 反差tJ 心慌uJ EmpvJ  而那些wJ  首席执行官xJ  上的重要讲话精神yJ lczJ  平行志愿{J 造林|J 解密}J  的限制~J  老城区J 宽高J  指导员J  ▁unchangedJ 龙口J  ▁apologyJ  要加强对J idanJ ▁TueJ  ▁abruptJ  坚实的J  ▁complainingJ ▁yaJ  ▁DepotJ  职业院校J 往外J ▁RNJ  ▁CycleJ  在平时J axiesJ  ▁EmmanuelJ 推断J  买保险J  沃尔沃J onteJ  ▁CoverageJ ITSJ catchJ  ▁detrimJ  ▁decisiveJ  ▁LieutenantJ  石河子J GYJ  退休后J 甘南J  学习计划J 心神J  这首诗J 一袋J  ▁bellyJ  天时间J  通过各种J  ▁empoweredJ 总在J ýchJ  ▁despairJ 勉励J  ▁CITYJ  ▁shinyJ  中国女排J  ▁RwandaJ 粥样J msonJ 天际J LatJ  贵人的J  燃料电池J  ▁breadthJ 追剧J  ▁executingJ  ▁identifiableJ 清清J ▁hiJ  数字人民币J momentJ  ▁PreventJ  ▁publishesJ 水性J  值得关注J  其中重症病例J 四点J  南通市J  都在看J 领地J  YesterdayJ  上发表J cyclesJ 放了J 卖点J 洗钱J FriJ adenJ  专项治理J  ▁extractedJ  ▁ActingJ 抗争J  服务费J 切开J 法式J  金黄色J  的环境中J  ExpressionJ 有料J aktJ  诸法空相J  ▁DissJ  ▁dazzJ emyJ  了一项J  突破口J  支援湖北J EmilyJ odianJ  ▁continuationJ 猪蹄J  分区分级J  ImagineJ 追尾J JetJ 花式J  ▁PayPalJ  的药物J shenJ  ▁unsafeJ  适应性J 二三J SocketJ 栈道J  债权人J  ▁whoeverJ  ▁DetectiveJ 掌柜J ▁DobJ 赶来J  ▁TIMEJ HgJ 墨尔J 重整J  的反应J  也不可能K  色不异空K  ▁empirK  ▁SereK 一鸣K  纯天然K  相关知识K 落马K  小区居民K  ▁scent K  ▁Garage K " 提供任何明示或暗示 K 教的 K  目前有 K NiceK  ▁categorK 江水K 青青K  试点工作K 低落K 充盈K 连忙K  空不异K  在家门口K  ▁LithuaniaK 」「K 染发K ▁ivK  不折不扣K  ▁prejudiceK  ▁ProgressiveK BloodK  ▁metaphor K hew!K  的美味"K  睡着了#K  话不多说$K  安排好%K 三连&K 云雾'K  特惠价(K  信息公开)K  ▁settlers*K ▁Erd+K  服务项目,K Offer-K HOUT.K 睁开/K  ▁Brick0K  业务能力1K  盐城市2K 废旧3K 林生4K  名单公示5K 执导6K 社长7K  ▁conscience8K 淖尔9K  ▁Duchess:K  ▁killings;K  ▁facilitatingK Opin?K  ▁Residential@K 屋子AK  春节前BK ▁DewCK  ▁NGOsDK  ▁tubesEK + $本文内容及图片均来源网络FK  省委省政府GK  ▁occupyHK  ▁DeveloperIK  ▁huntersJK  ▁acknowledgesKK 再过LK  洋溢着MK  ▁HeartsNK 海燕OK 浮肿PK 透着QK  武则天RK  ▁AllowSK  ▁StarbucksTK 傻傻UK 路桥VK 小城WK  空不异色XK  ▁HawksYK 东亚ZK  再接再[K 涌入\K 使者]K  ▁flock^K 抗原_K  健康管理师`K policyaK 不倒bK 恰逢cK 如约dK  derabadeK  ▁ThreadfK  analysisgK  ▁precedenthK ttariK 中人jK 手下kK 非瘟lK  特训营mK 卫东nK  属兔人oK  绩效工资pK 津津qK WTrK 多看sK  时间及tK 多想uK 梵蒂vK 客气wK  ▁BreastxK  ▁DocumentsyK  家喻户晓zK peace{K ▁COM|K 欢欢}K  ▁Offices~K 集会K  关于调整K  ▁AmerK  知识竞赛K  qualifiedK  ▁stereotypesK 反派K 人权K 再做K  图文编辑K  是诸法空相K  ▁spiceK  ▁sandwicK 二要K ▁MukK  小规模纳税人K 晦涩K eccK  认真地K  都不一样K  大家一定要K  ▁OscarsK  构建新发展格局K  ▁ambientK  中央八项规定精神K 伤痕K  生产车间K 食客K  我心里K  工作流程K  自行承担K  ▁registeringK 预热K FantK 公报K 古道K  ▁neonK 等额K 荟萃K  过年的K  ▁fadeK  ▁CranK  ▁ChildhoodK ällK  决定对K  新闻记者K  ▁sleeveK 消食K homK  以身作则K  刑侦大队K  ▁JungK  出席会议并讲话K  SelectorK  靠谱的K  ▁restructuringK 酝酿K  了我的K RoundK  ▁LifetimeK 飞舞K 九天K 角膜K  应当按照K  ophobiaK  党的十八大以来K  ▁VintageK  ▁EmergingK  ▁interactingK 配对K  ▁bandwidthK + $转载目的在于传递更多信息K 也未K 张志K jayK 金城K  调研组K  ▁McConnellK 号角K oilK  环境整治K  生活物资K kbK 易感K  factoryK  ▁IntegrationK ohoK  ▁strengthenedK 由各K vertsK 露营K ▁USSK  ▁SurvivK 抬高K  记录的K  性格开朗K  ▁LampK  执勤民警K mbHK 反观K 药性K ▁GUK  我不能K atalogK  梵蒂冈K ownsK scarK ▁RheK  ▁wolfK ALTK  承担起K ▁AlfK  ▁sodiumK  ▁impairedK 博白K  西双版纳K 后经K 家伙K  交通管理部门K  ▁LicensedK  就很容易K  的案件K 大鹏K 年月K  为了能K  这些都K  工作能力K  CreatingK ANNK atenL uyenL alachL  ▁selectiveL 三重L  规定了L  ▁DevicesL 波特L  公交站L 展位 L  绝对不能 L  不垢不净 L ▁AMD L 热身 L  信息共享L 不俗L 平度L 少林L 风范L  如果把L  ▁TheresaL  综合改革L  甘情愿L  幼升小L 引爆L 搜狐L 镇雄L  植物园L  这个故事L LetterL  ▁advocatingL 尼克L 本级 L 理气!L Side"L  并不知道#L  根据实际情况$L  ▁receptor%L 德宏&L 评卷'L Zero(L  工作量)L osaic*L 人呢+L 汉口,L  ▁Colonel-L 同性.L 围着/L  全长约0L Leon1L 辉瑞2L bilt3L  ▁Missing4L  ▁judgement5L 获胜6L 阴雨7L  ▁resh8L 一指9L 菜谱:L 长制;L  最简单L  ▁relocated?L 减持@L  这么好的AL  思想和行动BL  ▁devilCL 由衷DL  ▁WolverEL  ▁KanyeFL  ▁comparativeGL  ▁obstacleHL 有名IL 风趣JL  她就是KL  近些年LL  ContractML ñaNL  ▁chromeOL CAPPL 这般QL  ▁RootRL 锦标SL 扫地TL  属虎人UL  的追求VL ▁segWL 先知XL  平安安YL  marketsZL 不齐[L  发展为\L  的商品]L  ▁comprise^L 内陆_L 无能`L zieraL  中职学校bL  本草纲目cL  perfectdL 陋习eL NOTEfL 三者gL  GeorgiahL ' ssssssssssssssssssssssssssssssssiL 邪恶jL sqrtkL WMlL  ▁PublishersmL GanL 包袱oL 害虫pL ctvqL  两手抓rL printfsL  的大事tL  检查结果uL  ▁CafévL 打发wL 蔬果xL  一直都在yL  ▁chiczL  ▁booster{L  危险化学品|L 苍蝇}L  一点儿~L 海宁L  以往的L  受访者L  ▁mankindL  用的是L  可能要L 审慎L 银联L  divisionL 剪裁L  发生交通事故L 脉搏L  革命先烈L  ▁curatorL shotsL 一岁L  ▁lobbyingL 长子L  ▁seatedL  经济高质量发展L 舰队L BandL ▁konL  ▁ConcordL 教官L  当天下午L  ▁unfamiliarL  电影节L  ▁LuckilyL  动物的L  ▁JumpL 盘活L  的影响力L  ▁osteL  ▁tenthL  副处长L  日确诊L ▁ViaL 香型L  资源优势L  ▁BiologicalL  开发有限公司L 围巾L  ▁favouritesL 抚顺L rityL 空降L SANL  维持在L  PoliticalL 宝安L  ▁BoothL  ▁enhancesL 可有L 无菌L  周岁的L obbiesL 分夺L eductL 新余L 落网L 调皮L  ▁shoresL  先生说L NikeL ▁tedL  你不是L  ▁chemotherapyL 两端L  信息中心L 爱党L 江夏L 节前L  人社厅L  底层逻辑L 现房L  天花板L  一应俱全L  大幅提升L  ▁passwordsL 元件L 酌情L 取缔L 要约L  产阶级L  元一斤L  的还有L  aduatesL  ▁culturallyL 写真L afaL  ▁GalaL  ▁PixelL 少走L  小学校L obilL edayL  ▁HelsL  ▁BullsL PickL  可以起到L  的教师L  王阳明L  广州日报L 阖家L  但同时L  纪念碑L  齐白石L ▁LumL  ▁CriticsL 微店L  ▁xmlnsL  ▁stewardL  ▁DetermL  ▁awakenL  这个消息L isiteL 五位L 生出L  统一组织L  研究者L  分析研判L 顶端L ppyL  乡亲们L  少先队员L  ▁SprintL  ▁aggravL  SingaporeL 令第M 齿轮M orumM  ▁cylinderM BUGM  金山银山M  MoreoverM 大把M  ▁sneakM  ▁Peoples M 商住 M  ▁capacities M 一码 M 问政 M  FeatureM 简短M  中出现M  专业人才M 过境M  孩子是M ▁cynM  立案调查M  在一定程度上M  astrousM  ▁SpeciesM 瑶族M 苦干M  年中考M 不累M ourmetM  天安门M  ▁mysteriesM  个人观点 M  ▁bargain!M  又到了"M  顺利通过#M oyan$M  ▁Mechanical%M 的马&M  参加笔试'M worker(M 极品)M  与他们*M  可有效+M 龙潭,M  ▁Kris-M  正常情况下.M 该项/M  基督徒0M  ▁tornado1M 你又2M 过快3M Room4M 粉笔5M  征集志愿6M  autions7M xz8M 开裂9M 身受:M ▁Sig;M  决定书M  也可能是?M 温哥@M  ▁flawedAM  保护的BM  ▁einerCM 机车DM  有一名EM  贵人相助FM  起源于GM  ▁SonicHM 下场IM 双层JM  ▁UltimatelyKM amacLM  次全国人口普查MM  ▁PrairieNM  如何做好OM  ▁AvivPM  急诊科QM  JusticeRM  ▁destinySM 联社TM 节制UM  使用手机VM  ▁AgentsWM  ▁adapterXM  成为一种YM  领取方式ZM 萌萌[M  ▁sworn\M 定兴]M 换上^M ▁Rex_M  厦门大学`M  可提供aM  调查结果bM  反腐败cM  在此次dM  公益性岗位eM  走到了fM 布尔gM 而当hM ieberiM  eworthyjM 冰川kM ▁HyplM  环卫工人mM  还是可以nM 一盏oM  在网络上pM OkayqM 梁山rM 胸前sM 讲台tM  等重大uM  县委组织部vM 河畔wM  基层治理xM  科技进步yM  ▁devotezM  ▁persecution{M 香肠|M  ▁prophe}M  ▁Preservation~M 晒干M 结伴M ovskyM 博爱M  从武汉M  的一定M  各个方面M  ▁beginningsM 久违M  诈骗罪M  平均水平M  的气息M  生态环境部M 小觑M 随迁M 多情M 政客M  住房城乡建设M  我们又M  日举行M AMESM  支配收入M . '人民群众生命安全和身体健康M 破除M  电话咨询M  ▁padsM  封闭式管理M enzM  再后来M  投资项目M  特级教师M  综合整治M  ▁jealousM  的人类M MarieM  VirginiaM 季后M  ▁outfitsM 倒塌M  仅供学习M  新疆维吾尔自治区M  ▁SteelersM ▁QurM  ▁drainageM  ▁OfficialsM  ▁repertoireM 二连M  足三里M enburgM geriesM  communityM 万块M  不出门M  ▁issuM  ▁navyM  ▁combustM clinM  也成为了M  ▁FighterM  ▁wonderfullyM  本地宝M urnameM 歌颂M lassesM  includesM  ▁remarkablyM  徐志摩M  团结带领M  ▁ConfederateM  ▁sufficientlyM  ▁upgradingM FitM errorsM 日趋M  ▁twistsM  最高人民检察院M 风控M BetterM  ▁menusM  producedM 皈依M  生活必需品M 二建M 傻子M  的必要M  医生说M  降血压M 雪中M ▁JDM 升旗M 两份M 喝点M  济宁市M ahrM  的人工M  的阶段M  感受一下M  ▁inspectionsM 兴国M 剔透M  制成的M OTAM  版权归属M  新世界M  想到了M DebugM 前脸M  平平安安M  ▁idealsM  ▁HoffmanM 兵器M  在古代M  identialM  reementsM 盘子M  明确表示M 得快M SCRM  有关问题M 丰盈M 放行M  跟自己M  省公安厅M 一聊M  课教师M peredM 并入M  相关材料M  道德与法治M 和在N 娜娜N  两个小时N OffsetN 寸步N  ▁assassinationN 儒家N  用什么N 来年N  出来后 N  跟我们 N Games N  ▁Rapt N  ▁shred N  AverageN  ▁royaltyN 征迁N 桑叶N  医保电子凭证N  不断发展N 不治N 焚香N  从目前N ActorN cticaN  ▁MythN 村内N  了一半N  新农村N  innamonN  主动性N  转型发展N 被撞 N 该剧!N 余地"N  坚实的基础#N  Consider$N  ▁Dorothy%N  ▁Speedway&N 变差'N  ▁forgiveness(N  金字塔)N  集团董事长*N 无阻+N 白嫩,N  讲解了-N  抓获犯罪嫌疑人.N 上说/N 宋词0N  ▁Attorneys1N notes2N omers3N 益阳4N 若能5N ▁$_6N 昌盛7N 王思8N  兢兢业9N  ▁Kimber:N  严重时;N  分夺秒N  ▁Phillip?N  Development@N alerAN  一件事情BN  ▁InvestorCN 连通DN  ▁AheadEN 啪啪FN StrongGN  ATIONALHN 诚邀IN  黄鹤楼JN 慵懒KN 其二LN  到就近MN 切碎NN 这颗ON  ▁HerzPN  ▁StructureQN  ▁TransformationRN  感谢大家的阅读SN  感染者的TN 倾情UN  市场秩序VN 你却WN  ▁weighedXN 经管YN  管理层ZN  身体健康和[N Nearly\N hton]N ourced^N 很远_N DES`N  TimeoutaN 汹汹bN afiacN  的报告dN ineneN  ▁mentalityfN 不苟gN 查分hN xiaoiN  ▁autobijN  ▁DaughterkN  功能区lN  音视频mN  特异性nN 东汉oN  没有得到pN  DurationqN  局限于rN  其中之一sN  ▁scooptN  以为自己uN  ▁indoorsvN 找寻wN 纹路xN IngyN  中国文化zN  那么简单{N  ▁Penny|N  ▁theology}N cmd~N fedN  努力去N  网友评论N  若干问题的N  ▁WeissN  ▁StraitN  ▁bespokeN 岛屿N 浦江N  ▁contraN  ▁onionsN 大展N 周杰N  InternalN –　N 大关N  比上年N  ▁spicesN 淡水N ▁KYN  ▁unauthorizedN 牵线N 章丘N  ▁chassisN  ▁introN 紧贴N  ▁militantsN  主治医师N 的陈N 黄梅N  旅游度假区N  ▁grainsN 乳胶N ▁szN AutomN adderN 士气N 活禽N 写生N + $现实生活中的小事也是这样N yjN 井下N 日光N 脊髓N  创新能力N  ▁LotsN 世外N 传送N  预检分诊N  ▁wiresN  ▁GuidesN  如实填写N 回调N  地理位置优越N 一要N  ObviouslyN  膀胱经N 居易N  ▁förN  新要求N  相比较N  识别图中二维码N 特许N ItalyN  imbledonN 十日N  ▁playfulN 筑起N  ▁CoupN  ▁POSTN 送餐N  如果发现N 分两N  ▁ReachN  ▁DiplomN 共治N 无理N pentN 永嘉N ylieN  由高到低N 有难N 触电N 请与N 课前N  一氧化碳N  批评教育N  ▁oppressionN 早教N  观看了N  如何处理N  ▁GavinN penameN  ▁gingerN  describeN 匆忙N 清纯N  ▁asymN ruguN  酒后驾驶N  ▁sodaN  ▁accreditationN 沥干N ▁neoN 防寒N  以人民为中心N  老祖宗N 新兵N 帖子N 痛的N engersN  ▁AlexaN  ▁WeberN 在原N 比的N  ▁barkN LordN 牛油N  战疫情N  局集团有限公司N  通常是N  念念不忘N 帮她N upleN  助学贷款N  岁月静好N  所学专业N  ▁BenchN  ▁VariousN 写话O 和各O  回家了O  更是一种O  ▁impressionsO 放的O LETO WatO ffitiO  ▁territorial O 抱有 O  老中医 O licia O ▁Agu O 千伏O rebO  创新型O  正常运行O  ▁quirkyO 就叫O  为满足O  究竟是O  工程造价O 匝道O 黄酒O ▁hayO ▁necO  ▁НаO akisO '));O  ▁lockerO 护送O 搬进 O 攀登!O ▁","O  预防措施#O  ▁Basil$O  ▁Stafford%O 为好&O  专家学者'O beaut(O  ▁dependence)O 尽享*O  ▁paved+O %%,O  ▁McCart-O  ▁colonies.O 皮炎/O 返贫0O  ▁persist1O lambda2O 始发3O 长此4O  原因的5O  因此而6O  一篇文章7O  社区矫正8O ricing9O  ▁Rabbi:O  ▁hunter;O )×O 丝路?O these@O 无异AO 老去BO  ▁brighterCO  isationsDO KREO 艳丽FO yinGO  争分夺秒HO :‘IO 向右JO  ▁DeerKO  ▁котоLO  ▁exceedingMO  ▁nostalgiaNO 地头OO  对学校PO  女性朋友QO 橱柜RO 止痒SO  的领导TO  从零开始UO eqVO 筹建WO 惋惜XO  公告发布YO  等级证书ZO 一栋[O  又来了\O vier]O  ▁Deborah^O  的节日_O  当之无愧`O  越来越近aO  很可能会bO  被网友cO  recateddO 古文eO 定罪fO 俄语gO ollahO AwiO  你的人生jO  积极推动kO  陪孩子lO RachelmO  ▁AutumnnO ▁TinoO  直肠癌pO 外研qO  的表情rO  自然就sO  ▁ammunitiontO 翻开uO 早饭vO 供奉wO GixO 王志yO  中国家zO  大部分的{O  自主招生|O  世界级}O 库尔~O  保障和O 古装O  ▁PavilionO 挥洒O VIDEOO  ▁centrO  从一开始O  ▁securelyO  ▁BangaloreO RamO  的好习惯O  镇党委书记O 释义O  出自己的O  ▁typingO  ▁extinctionO 售罄O  ▁trainersO 东升O  ▁understandableO ineesO 中将O  找上门O  服务人员O  ▁presentlyO 诗篇O  外来人员O 六七O  ▁antioxidO  ▁packetO WINO  预计在O  再接再厉O  ▁pencilO  ▁RevealsO  政策性O  restrialO  ▁authoredO  ▁eclecticO 惩处O 获益O  这笔钱O  平台和O  独立思考O  ……“O  和政治O  dependentO  作品展O  friendsO  在实际O 下辖O 星的O 而动O  孩子的学习O 日志O  ▁MarksO 体魄O  山东大学O  农村宅基地O  ▁reproduceO 联欢O  ▁HydroO  prototypeO 惊心O  ▁OmanO  只有这样O  没有想到O  提升自己的O 佳品O  游戏的O lausO  ▁ReichO 间谍O  再加入O  ▁BermO 场站O  也知道O  ▁depictingO  在任何O AIDSO  ▁tabsO linuxO  ▁ensuredO xmO ndraO  价格面议O  BeautifulO 吸睛O 灾情O  统编语文O  ▁frameworksO 办结O 灵山O 防骗O ▁IsnO 架上O 水滴O 海景O  ▁corpsO  ▁overseeingO  不能吃O 享年O 用语O  点击下面O  伊斯兰O orityO ONSO  温哥华O 返利O mologyO 摘自O UESO  ▁dependencyO  ▁MidlandsO  妥妥的O  ▁buysO 懒腰O futureO  ▁ValuesO 私营O  属地管理O  开心快乐O  有需要的O  ▁prayingO 点前O  面积为O RankO  贫困群众O  ▁CalvinO 不回P MiniP  工程大学P  ▁TrudeauP digitP  ▁heavierP 敞开P  事业运势P  sponsoredP  ▁Sunshine P 排卵 P  一份力量 P  ▁Rouge P  ▁Vehicles P  炎热的P 顶替P  有阵雨或P  纤维素P enicP RiverP 区级P 牛的P  ▁caregiversP agogueP ATERP  文化建设P  ▁HydeP  ▁marriagesP 扣缴P  亲戚朋友P 双色P  ▁logosP  毛利率 P  ▁Cause!P  月工资"P  细致的#P  有效措施$P  二十五%P  ▁summ&P 小院'P 稍稍(P 缓刑)P  全自动*P 西城+P  ▁Quint,P  ▁contested-P 胡某.P ▁Ivy/P  的重要内容0P ARA1P 等症2P 航站3P 孟子4P  去哪儿5P  都是很6P  gradation7P 存钱8P  大众网9P  ▁Academ:P 平谷;P  获取方式

    p isans?p  ▁Lemon@p  ▁copingAp  ▁RecruitmentBp 首尔Cp  莫名其妙Dp  后进行Ep  ViewControllerFp 监事Gp ▁CypHp 约上Ip 送别Jp  世界经济Kp  改造提升Lp  财政部门Mp 工行Np  湛江市Op INFOPp  ▁DenisQp  ▁DiaryRp 笔录Sp borsTp  ▁comprehendUp 意象Vp 肉食Wp 起好Xp inusYp  ▁APIsZp  ▁nephew[p  ▁Katrina\p  学生资助]p  ▁blessings^p  充斥着_p  在各个`p  下面这些ap  在娱乐圈bp  发展和改革委员会cp ▁Ebdp  ▁Slovakiaep  照片中fp 亏了gp  ▁Initiallyhp  ▁simulationsip qinjp  ▁accustomedkp  城镇居民lp  ▁Chestmp  ▁endeavorsnp 相辅op ▁NLpp  更新了qp  转化成rp  ▁Fullersp 加成tp  区域性up 都和vp Whitwp  ▁scalablexp  该患者yp  食物中毒zp  ▁Savings{p disco|p  ▁contrace}p 换算~p 烤鸭p  ▁metabolicp  的青春p  ▁hypocp 清静p 喝完p 说唱p  真诚的p  睡觉前p  大概就是p  图片来源于p  ▁sortingp  ▁poignantp ▁MONp  接受治疗p 带班p  的主角p LOOKp ▁lanp 道来p  真抓实干p  ▁Guidancep  assertEqualsp  Productionp  ▁ministriesp 图的p  模拟卷p  如一日p  化学反应p  directedp  ▁Disorderp ikedp  行政法规p 一档p 小草p ▁MUp  roundedp  一直坚持p  审核通过p Annotp 过客p  梅州市p  治国理政p 既定p  ▁Reggp  ▁wrappingp loup  我就不p  人生路上p 转行p ▁FTp  ▁паp Jeremyp  ▁Dominicp  方法的p  意义重大p あゆp 含氯p 平乐p 童趣p  都是可以p  ▁Platp Odp 也多p 新晋p  ▁coincidencep 年生p bilp  老奶奶p ▁LTEp  也可以用p 王大p alsap iancp  优异的成绩p  ▁Highlandsp  ▁fragrancep 前卫p mphp  报名中p  有关负责人p  ABILITYp  ▁growersp  经典老歌p  是所有p  已经完成p dockerp  ▁confidentialityp  他们还p  享受生活p  完全不同p  ▁Blankp  Warningp  roducedp  给人以p  ▁Graphicsp  Dictionaryp  每一名p 之最p 鱿鱼p  ▁Saturdaysp 纵然p 鸿运p  由国家p CELp  央视网p  的称号p  updatedp 戈壁p 无极p  承担着p  技师学院p  ▁herdp  ▁connectorp  等待着p  申请办理p Austinp  并处罚金人民币p 大坝p ritop  通用标准p  ▁enricp  ▁emailedp  ▁mutationp  ▁Procedurep 中游p 我司p  ▁pantp 走心p Noticeq  ▁curseq  ▁tappingq 受精q  教学内容q  ▁Coralq  ▁unterq  ▁daytimeq 泪流q ент q  应符合 q 变频 q 的被 q  铲屎官 q  ▁defensesq 固态q havenq 濒危q 玩了q 镀铬q  ▁Romanticq 暴利q 分班q  ▁Chamq 许愿q  西南地区q  ▁Nileq 兑付q 拥军q  LICENSEq 一城q 主旨q  眼看着 q  个人原因!q 医术"q  ▁turnaround#q  ▁Conservatives$q 的歌%q  在逃人员&q  总体来说'q  ▁encl(q 月龄)q 灾祸*q 质问+q  科医院,q  training-q  更有利于.q  ▁aide/q  ▁Roads0q 较短1q ▁ASE2q ▁Jol3q  ▁Shoes4q  ▁resemble5q 上肢6q  动车组7q  工作者的8q  rospect9q  ▁parole:q  仅有的;q  ▁Automationq 之长?q 增信@q  结束时Aq 奖杯Bq  垃圾的Cq PollDq  ▁poisoningEq 贴吧Fq PagesGq  阐述了Hq njIq  酒店的Jq  ▁beamsKq  全勤奖Lq  StrategyMq  ▁AbdullahNq 变性Oq 激昂Pq 责怪Qq  未发现Rq  ▁FlipSq ophersTq  元素的Uq  沈丘都市网Vq  ▁FischerWq  中心和Xq  产品设计Yq  全球疫情Zq  出行安全[q  ▁Extended\q  ▁Congressman]q 学而^q  都不够_q  就有可能`q  ▁Lumpaq 五千bq 住处cq  稳就业dq  更多精彩资讯eq 仙家fq abovegq  ▁upheldhq 童子iq doujq  比较高的kq  公共科目lq REDmq  调查报告nq  ▁layeredoq  ▁fashionablepq fraqq  不干净rq  书记员sq ▁Vijtq Unlessuq ▁IQvq Kathwq  区融媒体中心xq 乐享yq 报业zq 更远{q 芳草|q  归属于}q  的好奇~q  ▁Seeingq 密云q vehq  大药房q  都来了q auffq  communicationq 四轮q 氢能q agonalq 坦白q  认真对待q 重力q  汇报了q andanq  ▁Bladeq  需求和q ightedq  雷神山q sidedq WFq  公平的q  上一层楼q  ▁signageq 矿井q  最长的q Bridgeq  ▁foldedq fabq 很重q  户型图q  那我们q 共性q  ▁Codesq  ▁intoxq  是否能够q  Transactionq 两块q  ▁pensq 我再q 水生q  的是一个q otineq 置信q  身份证明q  ▁cathq  Outsideq  romagneticq  ▁Sociologyq 上扬q ▁ABOq  ▁qualitativeq  黔西南q  ▁Valeq 挽留q  录取的q ▁Segq 李玉q  ▁Sharkq  ▁medicinalq MONq  亚硝酸盐q visitq ividedq  ▁optimumq 校方q  ▁thirstq 扎心q 畅玩q abellaq  ▁treadq  ▁aquaticq 繁星q  教育活动q  ▁Gujaratq 情缘q  全都是q  学生和q  ▁rebellionq WGq  组建了q  ▁Throwq 白癜q 看中q  主动请缨q  ▁intellectq ▁Neoq  ▁vampireq  更别说q  的外表q  ▁Cakeq  ▁fatalitiesq oinq  meaningq  华为的q sportq answerq  ▁Debateq 多维q 晾干q  什么时候开始q 监审q  出台的q  李玫瑾q  该企业q ereyq  最起码q  ▁strokesq 一厨q Einq  和在看q  越走越q droq  新增了q  是他的q  ▁whoppingq 为上q  第三条q  ▁hubsq  ▁Yourselfq 开往q 智造q 亮剑q 检举q 甲环q 香精q  我也不q  精准的q  ▁Compareq  南开大学r  团队合作精神r  ▁Anglicr 爱民r  ▁overlapr  ▁Trinidadr ▁EFr  美国在r  ▁bashr  ▁serum r  ▁Adding r 妇人 r 耶鲁 r 雪梨 r ANYr DECr  可以减少r  ▁eclipser  ▁fuckingr daor 木板r 辣的r  咱们的r  ▁cappedr  ▁predictiver  米左右r mannedr  ▁poultryr  ▁Headsr 易俗r  了解下r ▁Yukr  准确地 r 中含!r 日间"r 闲着#r  电子码$r  心存侥幸%r 和美&r  工作会议精神'r 自责(r  认定的)r  ▁sandy*r  ▁tended+r  ▁Helsinki,r  ▁Considering-r 北辰.r  和实践/r Chart0r  ▁Worst1r ▁Coy2r ▁pir3r  耐心的4r  一夜暴富5r  ▁Amid6r  ▁Cage7r 用气8r ▁ov9r  微博热搜:r  ▁Ricky;r 麻雀r  ▁commute?r  但是从@r 一倍Ar 维斯Br 西斯Cr 责备Dr "},Er  ▁чеFr ▁AnsGr ▁gebHr  osteroneIr  ▁PhilharmonicJr 张三Kr  泥石流Lr  ▁SunnyMr 余项Nr 房门Or  回忆起Pr  重点企业Qr qhRr 成团Sr  岁之间Tr  ▁sitcomUr 付诸Vr  学不会Wr  的病毒Xr DominYr  ▁electronsZr  太好了[r 抒情\r 脸颊]r witz^r 泗洪_r  ▁tariff`r 赌王ar 香葱br  镇卫生院cr Structdr 真香er  全国政协fr  ▁VIIIgr  ▁blochr 酸钠ir 雷暴jr ITIONSkr vidialr  ▁zincmr  ▁sedimentnr  研究团队or 都给pr ouselqr  ▁Triberr 锐意sr  ▁akintr  ▁exileur 洋河vr  大兴区wr  也不用担心xr  国际上yr COMPzr  专家提醒{r  加快推动|r  ▁prevailing}r 五次~r 有其r 湿巾r  ▁bitesr mongr 宿主r helfr  吉林大学r  使用后r terminr 咸丰r 池州r 药企r  本案中r REAMr  智慧城市r 在省r 镇痛r  但是她r  道德品质r  国家卫生城市r dianr ▁Dixr  才有可能r  网络游戏r 月和r  西北风r othsr  潜移默化r  直接购买r  ▁Varietyr 跑腿r  不了了r  的明星r Urbanr  种植的r  习近平总书记对r  ▁Instantr 配以r lordr  ▁upstairsr  ▁humanitiesr riotr 飞飞r ▁WINr  可能就会r  原件和r  限时免费r  ▁Danieller 好啦r ñosr  战士们r  医保卡r kickr phaser  ▁hopelessr  ▁mechanicr 地步r  殡仪馆r 不渝r  啥时候r  将来的r  ▁HAVEr 归类r  自己不r  ▁differentiater  对工作r ’’r 镇上r  教学中r  用一个r addrr  ▁Clockr  ▁perennr 为辅r  气溶胶r ▁babr  最新动态r  顺利开展r  ▁polymr  活动等r  不断创新r  ▁penalr  ▁likingr 烤鱼r Stoner esselr  西藏自治区r  ▁botheredr 卸妆r 陈志r  英格兰r afkar lavr  未履行r  ▁syllr  ▁persuader 芦荟r  观自在r LOADr 量力r  十多年r  ▁Heinr ilepsyr  ▁Buddyr 彝族r  的精彩r  健康长寿r  涡轮增压r  ▁unjustr  ▁slaughterr  ▁demolishedr  ▁transformsr  ▁computer  ▁standpointr  ▁Alternativelyr  可以分为r  ▁gentr  ▁marginalizedr  自己没有r  ▁lentr 免于r  却依然r  参谋长r  内心深处r  Monitorr  coloredr  ▁Zuckerbergr 写法r  穿衣服r  服务号s  ▁placebos  ▁accusings bxs 减量s  李小璐s swires 车尾s  极有可能s EMAIL s 不换 s 四风 s 点餐 s Hig s  多云间s 贩毒s  事故现场s asuss  保持一致s Karens  研究中s ▁Rims  节日快乐s  比普通s 河谷s åns 是故s LOCs  不定时s 内政s  类食物s  无能为力s  生态修复 s  ▁butterfly!s  Reviews"s 腋下#s  发生的事情$s  ▁embarked%s 之神&s 清脆's  微粒贷(s crit)s  ▁Jorge*s  ▁materially+s  和城市,s  聚集地-s bh.s  默默的/s  pertension0s 轴线1s 迷恋2s  不断扩大3s 冷了4s 退烧5s  土地征收6s  ▁KIND7s  Everybody8s  ▁tolerate9s ▁Mum:s  每日一题;s  ▁Logos 和好?s 计件@s  的平均As 不报Bs 武力Cs  元罚款Ds  ▁StressEs  ▁swiftlyFs 她被Gs  在所有Hs  零售药店Is LewisJs  ▁BoxingKs  ▁MajestyLs  ▁quartetMs  ▁evacuationNs 临海Os  中国和Ps  小户型Qs айRs 旭辉Ss  不可缺少Ts  ▁diceUs 种下Vs zuiWs asakiXs  ▁ReevesYs  ▁influencersZs 贩子[s ivable\s 带量]s  老先生^s  专家团队_s  明显改善`s Andreas  ▁Mohammadbs 岑溪cs 隐性ds  内容丰富es izophfs 掩护gs  时尚感hs  有明显的is  ▁Perspectivejs 阴气ks  ▁possessedls 临安ms 前身ns  ▁Crossingos  ▁peanutps  的人身qs 乱的rs 骨的ss  具有一定ts  小型汽车us  美容养颜vs  课程体系ws  有点像xs  餐桌上ys ▁USEzs  ▁expresses{s 油箱|s cg}s 入眼~s  元不等s 接头s 浙大s  年国家s  ▁Penss  ▁textbooks  ▁consequentlys 是哪s  不一定是s  Israelis 二氢s 关中s 可从s  ▁interchanges  日结束s  全国各地的s 干涩s  女主播s 风度s ▁WVs  烦心事s  ▁conquers  男主角s  餐饮店s ▁Mugs  ▁bulbss ▁'',s  版权说明s  ▁informationals 和东s  ▁debits  ▁Caesars  ▁swellings iptss  疾控部门s  ▁Resils 亚裔s 非公s  集中医学观察s 之三s 正中s  总编辑s ▁cyls  everances  ▁Prospects 台海s  刚刚开始s 病种s  法院判决s Htmls  arnations  ▁Randalls 当过s 早报s  没有谁s  ▁terminalss  ▁generatorss  它有着s  这正是s 其乐s  在教育s  直接将s 笔下s  汇总表s 取穴s  ▁Essayss 小看s  第一件事s  ▁Winnings 微小s 行进s  仅作参考s  鼓励孩子s 任由s  自贸区s ▁huss  甲状腺结节s 权和s  ▁misinformations 外环s  ▁grooves Ghs brevs ▁Wans  ▁Nadus  ▁Sophias  ▁playwrights 清退s  要保证s  ▁planners  但事实上s  ▁Dions ▁idis ariouss krs  ▁Bards  ▁catchys 该市s  要选择s 又来s  从严从s 卷入s  考生和家长s }(s 到案s 母鸡s 获刑s 劳损s 征税s  深度好文s  ▁rollouts  巡察组s  以下的罚款s 生辉s  碳达峰s  考生们s 中旗s 这和s  学习资源s 台的s  payments  ▁paradoxs 重播s  直播课程s asthans  ▁Mixeds  ▁acknowledgings qds  马路上s  professionalt 厨艺t 聪慧t 范本t 养羊t 总说t 报批t felt  基本生活t  ▁smash t  ▁frontier t ▁omn t 晨曦 t 迷糊 t  只是因为t 巴赫t  统一考试t bsitest 溶剂t 脚上t ▁jat ▁Witt  鄂尔多斯市t  ▁cafest 大业t ▁PIt  第一章t  就开始了t Challt 婆媳t  entationt 气压t  可拨打 t  真的有!t  易懂的"t  ▁Careers#t 皮带$t  Questions%t  ▁virgin&t 屋面't 在三(t  未经许可)t exclud*t  ▁lumin+t  actively,t  ▁demographics-t ▁CJ.t  投集团/t  合作共赢0t  ▁Disabilities1t fé2t  密接者3t  尽量不4t lez5t  ▁Vanderbilt6t  子女的7t  走到哪里8t Float9t  ▁Loans:t 一查;t  主耶稣t  ▁Speakers?t  的瞬间@t  ▁linenAt 拉升Bt  国家政策Ct  edienceDt  ▁surpassedEt 党章Ft  留下来Gt  的种类Ht SequIt  衣食住行Jt  ▁HaleKt  ▁plagueLt  任何单位和个人Mt 罪被Nt  价值和Ot  更像是Pt  ▁burgerQt  ▁ClemsonRt  少量的St  南无阿弥陀佛Tt  ▁relievedUt  ▁KickstarterVt URIWt unkyXt 才好Yt 易爆Zt  都能看到[t rules\t  ▁Lexington]t  　–　^t  ▁Hammond_t 待用`t 一排at 两组bt  可作为ct  懂得了dt ermoet hawksft  全方位的gt  考试期间ht firedit  信息表jt  avelengthkt  也确实lt  薪资面议mt  ▁Barclnt 白粉ot 有近pt 殷切qt  让更多的rt 小鸡st 高州tt  教练员ut Halfvt  为了更好地wt  ▁Lordsxt  arationsyt Wangzt  ▁Cory{t 生来|t  低年级}t  ▁Brass~t 走一t 鲜味t  转折点t ▁CIOt ▁Hogt  原创内容t 两江t Oxfordt  ▁collaboratorst 不拔t ▁%>t 煮水t 鹌鹑t  年至今t  早睡早起t  雷锋精神t  惊呆了t  interprett  ▁Survivalt 个亿t hesiont  culosist 奇异t 爱着t  质量的t  提供更多t  ▁einemt  自己要t goerst  ▁Vitalt  明确指出t  暂停营业t  儿童青少年t  ▁Rankingst 从教t 使他t 和陈t 辟邪t  就好比t  ▁Successfult fortht  ▁Seahawkst  ▁mutuallyt 在北t 时需t  我还记得t  ▁arcadet  应对措施t  ▁TODOt  一年来t  ▁Shimt  ▁Proudt  ▁Toraht 场内t ▁UNCt  Beijingt 布拉t ▁evet  rollerst  ▁Possiblet  ▁kidnappedt 印制t  十几个t  让家长t  ▁Straightt 几万t  客流量t  ▁Cindyt 摩天t  上一次t  只提供t  满足感t  ▁Characterst 取胜t salet playst  省委副书记t  Trainingt  ▁permet  ▁Visitingt 珠峰t  本办法t inait ▁NAVt  ▁Pallt  ▁stickerst  金融市场t  ▁alleyt  ▁gravest  延期开学t  这个男人t  ▁jakot  ▁reconnectt  ▁noteworthyt  和维生素t  ▁Pipelinet PEGt  让宝宝t Hadt guant ndont  幸福美满t 直营t ▁reqt  及时进行t 生俱t 疏远t Jimmyt  ▁Kolkatat 小年t  党代会t  小动物t miat  一个月的t  ▁Attendt errickt  在群里t  还容易t 水系t  ▁Pharmact  ▁Buddhismt  ▁endpointt ,[t 匠人t  下班后t çõest enmentt scopict  更高效t  工作委员会u  ▁Beethovenu  ▁augmentedu 新开u  ▁rosesu  ▁Gloriau  ▁madnessu Piu zzyu  ▁domest u  综合行政执法局 u  ▁monastery u 扑救 u 炮弹 u ▁aceu  合同纠纷u ammersu  的时期u asiou  机动车驾驶人u 九十u 惨重u  可以用来u  看图写话u  ▁Personsu idavu  中国加油u  成为全球u  获得国家u  ▁clanu 商事u  一般都u  出水面 u  早已经!u Bottom"u endale#u  ▁echoed$u  ▁adolescent%u  Directed&u 成片'u  了一切(u  ▁Kirby)u  腰椎间盘突出*u кі+u 推车,u  不同的人-u iphone.u 贿赂/u  春夏秋冬0u heavy1u  ▁lump2u  ▁Infantry3u  的信念4u  耕地建房5u  ▁deploying6u 席位7u uen8u  地展现9u  这其实:u phil;u  零售价u 爱过?u 申办@u 粗细Au OTTBu  一个新Cu  anguardDu  ▁BrennanEu 小二Fu  关键点Gu  范围和Hu  ▁GoldbergIu 来过Ju  无一失Ku  美国海军Lu 仅此Mu  五台山Nu  最好在Ou enegPu  行动轨迹Qu 固始Ru  很正常Su rundTu  ▁CandidateUu  ▁concealedVu 做些Wu  引导和Xu  ▁sympatheticYu qxZu  决定性[u 爱惜\u  ▁Sham]u  四年级下册^u  ▁enzyme_u  ▁Minority`u 水镇au oehbu  ▁Expressioncu MEDdu ▁éseu  国内市场fu  出租车司机gu  ▁Filterhu 凡人iu 招惹ju 课余ku  在各种lu  风险隐患mu  ▁Lingnu  ▁Siemou 滕州pu  立足于qu  没过多久ru  自学考试su famoustu  ▁Tibetanuu 乔丹vu 血红wu 重拾xu  桃花源yu ciaszu inker{u  汉语言文学|u 绥化}u  小编就~u  追溯到u  客户经理u  ▁downsideu 智利u  ▁gastrou 术前u  Classicu rxu 有度u  规范性u Violu ubleu  明天白天u 刘少u  年代的u ▁ITVu  即可查询u  igatorsu  ▁cardboardu 项城u 抛光u  专属福利u  ▁Commentaryu  ▁heartbreakingu 顺遂u WORKu  ▁Docku  ▁accentsu  超强的u  ▁Gloryu  ▁Lebaneseu  ▁imaginativeu  不及防u  ▁shaftu  ▁inferioru 财星u 蹭蹭u lysu  expressu 未然u  的信号u  复习资料u  ▁Revelu  ▁resinu  ▁allegingu  谈天说地u  ▁boomingu  ▁milestonesu 热忱u  发展空间u  ▁methaneu  ▁sailorsu 儿时u ▁wieu  ▁tuningu  ▁knockingu 弹幕u 抒发u umabu ▁Ostu  ▁nutrientu 悦府u  ▁lineru  ▁Johannesburgu jitu  森林火灾u 辅以u  ▁harvestingu  行政复议u  ▁Cyrusu  ▁Vogueu  trainedu 百般u lmu 较前u  只要能u  要怎么u 剥离u 扁豆u 断电u  意外伤害u  还要注意u  ▁plannersu 着迷u  维护国家u  USINESSu  ▁elusiveu 宣泄u 荐读u  ▁enzymu 句号u 恪守u  取得实效u  坚定了u  Assistantu 消暑u  ▁Virtu  ▁Helenau 慕名u  allowedu  validateu  ▁undocumentedu 从心u 刘强u 多篇u 对您u  ▁Chopu  ▁Giulu 伺候u ▁MKu  以下条件u 回响u 附院u  目前已有u  负重前行u 担责u  财运运势u  那么我们u  ▁Mandelau karau Sleepu  ▁coutu 镌刻u  为促进u  吃火锅u  在过去的u  非常多的u  ▁sorrowu  ▁Kristenu 验方v  街道办v  ▁Moodyv  ▁Responsibilityv 临界v 龙眼v  从今年v  这道菜v  ▁Biggestv  清楚楚 v  著作权人 v  ▁mindful v  指挥部办公室 v 薪火 v  从来不是v  ▁painsv 胁迫v Sherv  消毒工作v  Constructv 九九v  孟加拉v  ▁septv  ▁Telecomv 巡游v 经纬v  生成的v  行动方案v  ▁Episcopalv  轰炸机v ▁Glev  微信群里v olate v  ▁seals!v  的安排"v  知情人士#v  ▁Reef$v  ▁cruelty%v  孩子会&v 无虑'v 玉龙(v 译为)v oprano*v  ▁Bonnie+v  也正在,v  五花八-v  人员对.v ▁FIL/v hex0v  两位数1v  每一件2v  的一件3v UAL4v ▁HK5v  号文件6v  ▁Thoughts7v 甘孜8v  志愿者的9v 黑客:v  ▁flank;v  低成本v  ▁Susp?v  ▁superf@v  ▁GeographyAv 惊奇Bv 有水Cv  ▁даDv  ▁mixesEv  ▁OfferingFv 东进Gv 结余Hv AboveIv  ▁KatyJv  ▁remindingKv 每秒Lv 德斯Mv ICEFNv  加快发展Ov  附属中学Pv  中央军委主席Qv  ▁HighlyRv 体弱Sv 栋梁Tv  日进行Uv  时间点Vv  长白山Wv  ▁SplitXv 宝坻Yv  ▁ruthZv  很方便[v ▁CAD\v  协调发展]v  招生单位^v  消防通道_v  ▁Bieber`v  ▁fountainav  ▁helicoptersbv 口译cv  下方的dv 单次ev  性行为fv  ▁Thankfullygv Tonhv Viniv  中使用jv  ▁tapeskv  的美景lv  的难度mv Miaminv JMov 云淡pv 传到qv Flagrv ▁ESGsv  DOCTYPEtv  ▁Futuresuv OBvv 妩媚wv 难懂xv  睡眠不足yv  同志的zv  医疗废物{v  ▁multicultural|v 悼念}v "">~v  录取分数v Holderv abbingv  物业服务企业v :”v EMBERv  降雨量v  可联系v  清清楚楚v  蔬菜水果v  ▁alcoholicv 早有v  的后果v  根据不同v latestv  官方微信公众号v  补偿安置v  餐饮企业v  ▁lavishv  ▁declaresv 自控v  即将来临v molv  机械设备v  对我来说v  ▁Roundupv 悬架v  ▁Acidv  ▁avantv  十七条v  地道的v  ▁lockingv  ▁conservev AMDv ▁glev  组织学生v 脚部v  请记得v  ▁jQueryv  ▁Unlimitedv  什么人v ▁Hutv 不学v 听歌v  坚决做到v 丹阳v 公信v 那条v kalv  五粮液v  广大青年v  ▁Holtv  ▁Balticv  已经达到v 博客v  约翰逊v  经审理v  ▁bulbv  ominiumv 抹布v maniv  烦请告知v  ▁loomingv 可享v 翻盘v  高效率v  俄罗斯的v routerv 双料v 套数v UTEv  人世间v  在全国范围内v  这东西v  危险因素v 用刀v 肾炎v  ▁Fijiv  ▁yeastv  ▁protestingv  的总体v  ▁prefersv 对齐v 相随v 缺一v  采取的v ▁Pagv ▁mugv suitev 白墙v PEPv  保护好自己v 刊载v 嘟嘟v ▁TAv  购买方式v  ▁Dispv  ▁puntv 追缴v  经济与v  重点大学v  长按二维码识别v 云鹏v 中火v  全面恢复v  通过视频v  ▁pubsv  ▁Marriedv illaryv  Submittedv 万件v  ▁revivev  ▁overwhelminglyv  ▁tacticv  ▁copyingv 南平v 酸碱v  业委会v  引导群众v  学生的学习v  了一圈v lostv  而不腻v  居民小区v  深度学习v  ▁Cronv  ▁Seasv  ▁dictatorsv  普惠性v  从来都是w  ▁Macronw  搭载了w  ▁Pharmaw  高中英语w  ▁Proofw ;<w 津液w 游子w 钉子 w  好东西 w  下雨天 w ayson w 守卫 w  ▁Abramsw 飞来w  保洁员w  县医院w  ▁Camdenw  ▁cultivatew  ▁astronautsw lzw 计价w  使自己w  经济特区w  ▁crackingw 周全w ▁hymw 杂谈w  守护者w quotew ampingw reich w  有下列情形之一的!w 降级"w  ▁reclaim#w  ▁tractor$w 双侧%w 海面&w 字头'w 丰硕(w 建业)w  种食物*w 党参+w  需要你,w ▁Edu-w  教学资源.w  低血糖/w  斤左右0w  ▁Archer1w 馅料2w  发展前景3w  在今后的4w  ▁faint5w  heading6w 施加7w 果报8w  的舞台9w WiFi:w  干部队伍;w  文章推荐w  ▁Personalized?w  三年级下册@w mAhAw heneBw  主体结构Cw 活成Dw  万无一失Ew  更多内容Fw  女孩的Gw  某一天Hw  紧急状态Iw  自主品牌Jw  ▁rhythmsKw  ▁PortsmouthLw  ▁festivitiesMw  不容缓Nw  UkraineOw 吵闹Pw 心丸Qw  归属感Rw  烟火气Sw  自定义Tw radiusUw 更替Vw 气喘Ww  说明你Xw 丽丽Yw 亲临Zw  的口罩[w STAT\w  ▁Agencies]w  ▁Increase^w 权衡_w 脉冲`w 过完aw ▁KCbw  已达到cw  ▁Carlsondw  ▁geneticallyew  ▁stimulatingfw 蹦床gw  这样一句话hw Pierreiw  ▁bodilyjw  踏踏实实kw  健全完善lw 底数mw jaxnw  加速度ow  第十三届pw  ▁frostqw 对阵rw  司法厅sw Battletw  ▁farewelluw 军长vw 宴席ww  卢卡斯xw  自己的心yw Safetyzw  证书的{w  ▁backbone|w 富力}w 水痘~w REAKw JTw 切尔w  ▁pumpingw  战斗堡垒作用w  Reportsw  ▁Kelleyw 凉鞋w 而立w  桥头堡w ballsw  Irelandw  的巨大w  不合格者w  ▁Funkw acialw laveryw  izophrenw  ▁allergiesw  得非常w ▁Show  ▁objectionw  延续了w  教育理念w 口音w  世界观w  ▁hourlyw 对决w  累计死亡w entonw  ▁homeownerw  Notificationw 只管w  老板娘w  那么好w ▁Zakw  勇敢的w  的依据w  音乐节w  根本任务w 终会w  发挥着w  蕴含着w kidsw 照常w 移风w  养殖业w 涂鸦w  的小白w  老朋友w ▁cmdw Mapperw  ▁Individualsw  ▁examinationsw 网的w idxw  getNamew Polyw ilinw  城市规划w  ▁surgeonsw  工程设计w  皮肤问题w  Supremew 功力w  不过来w  洗手间w IDENTw ulencew  ▁tigerw  ▁evaluationsw ódw 对它w 广为w 立竿w 简报w 游人w 谎称w assetw  ▁knotw  是一个很w  ▁Empowerw  ▁turbinew  ▁organisew 安放w 战中w 石的w  劳逸结合w  ▁parapw  经济社会发展的w huangw  可关注w 大举w 药水w 霉运w  的病例w 挺直w  为进一步加强w 波折w Luxw  检测出w  ▁Pokémonw  女子的w  立交桥w  视频课w  这一段w boldw  天气变化w  工程量w  和你一起w 暴击w 甲基w 面容w Analw  ▁Combinedw ▁fnw ▁INTw  ▁reinventw  ▁MacBookw Tyw  种植户w  ▁misusew 乐高w 他了w Statesw  ▁inlandw  ▁parserw  ▁warrantsw  手脚冰凉w turnedx  ▁Clausx  ▁tunnelsx  ▁Curriculumx rasex  如果能够x  ▁savvyx 一题x 东台x  主要针对 x 哈里 x  以此为 x  的增加 x ▁ost x  周围的人x  武夷山x bossx  切身利益x  为客户提供x  ▁slicesx  ▁somedayx  医疗费x uestx  关心关爱x  ▁aislex  的共同x  ▁degenerx 彼时x 自用x 讴歌x ▁Abex 上街x 跌宕 x 病了!x  像一个"x  笔下的#x  生活水平$x  立即组织%x  ▁specifics&x  ▁Colors'x  ▁revisions(x  白岩松)x  ▁Peer*x  ▁intrigued+x heat,x ▁Dud-x  公益诉讼.x  梦想着/x intern0x  Servlet1x  简单地2x  ▁Infl3x  Excellent4x 推算5x  平方米的6x 四舍7x 自取8x 镇定9x  任何形式:x  澳门特别行政区;x 免受x  好好地?x  ▁Desktop@x CouAx  开发和Bx WolfCx  奉献精神Dx  ▁PrelEx airobiFx 兜兜Gx 行踪Hx  出国留学Ix  宁德时代Jx gmKx  ▁pensionsLx  更有效Mx  ▁SageNx  ▁contingentOx 手势Px  致癌物Qx  随心所欲Rx quantSx  ▁wastingTx  ▁recklessUx 学龄Vx 投射Wx 板书Xx 母爱Yx 翻看Zx C <城市网邓州城市网邓州城市网邓州城市网邓州[x мы\x  人物的]x 注视^x  esthesia_x  ▁proliferation`x JKax 和有bx 课表cx 那次dx  需要一个ex  ▁Bartonfx 正轨gx 藏匿hx 半夏ix  ▁cultivatedjx 差评kx 某在lx  解释说mx  放入锅中nx prefixox  ▁attentpx 人所qx 每条rx  一次性使用sx 满地tx 停药ux  ▁popevx  ▁solitarywx 滞销xx 新星yx  被隔离zx ▁lar{x  新增死亡病例|x  克拉玛}x  看到这~x  签署了x  现代社会x 丑陋x 晴朗x Banx  也许会x  决定将x  有序的x boxesx enumx 打捞x  ставx  ▁Lauriex  Regardlessx 兰特x 函授x 得着x 或其x Bedx  铁路部门x  示范村x  三次会议x  ▁Componentx  ▁broadcastsx 后继x 园的x  的一片x  ▁Soldiersx 乳汁x 我先x ▁Tetx  ▁LEGOx 发家x 必究x ovyx  分配到x 余亩x 您在x 核减x 公婆x 熄灭x 满头x scrollx 鄱阳x  极好的x  目前来看x  ▁proclaimx  而作为x 喝到x  ▁Boutx  otswanax  ▁fisheriesx 包机x 月圆x  意外的x  大年初一x  平均气温x  我们身边x  提前预约x  中加入x 口苦x zzlex  引进人才x  ▁distractionx 维尔x  视频讲解x  ▁Wincx  ▁Newspaperx 还曾x DHx 显露x 暗中x originx  ▁crippx 卖场x  广告投放x  ▁unrex 新歌x Flexx ▁PETx  ▁Ottomanx  ▁energiesx Mix 一撇x  ▁Oncologyx abeticx  ▁Acquisitionx 发散x 参评x  她没有x  研究制定x 牛角x  取得的成绩x 布料x weedx  如何预防x finderx 尼古x 极好x  财富自由x ultonx 了下x 分摊x LGBTx  教师岗位x grox  做任何x  开好局x  一直处于x  宣讲会x  政务服务中心x 之士x 在村x 常青x 采食x  中成药x  分手后x Bootx  让人感到x  ▁Jenningsx 王道x  交汇点x  半身裙x  秦皇岛市x  安全员x  ▁snappedx 劳斯x 汇演x  即便如此x 早春y  小游戏y  最深的y  ▁Nguyeny  ▁preventivey  植物的y 展销y 酒庄y  与环境y Remote y *; y  但是也 y  投诉举报 y 学过 y 迷雾y  各岗位y  发生重大y  合同制y  所以就y  如果不想y athiy  报告制度y  ▁furnishingsy  不知所措y  的发展思想y  ▁doublingy  ▁contestantsy  出生后y  ▁clery jiaoyuy Copy prony ▁sar y  名列前茅!y  近段时间"y  ▁Naomi#y  第一附属医院$y  万多元%y  与自然&y  的三大'y  ▁sequencing(y 才可)y 精挑*y 共设+y 馆长,y  头像壁纸-y  Transfer.y  ▁Minimum/y  ▁breakout0y  先进工作者1y  ▁ranged2y 元和3y  实际操作4y  该岗位5y iator6y ussen7y 水岸8y 西药9y  了一波:y 伤官;y  一大波y 云霄?y 别致@y 尺码Ay 雾化By  听党话Cy  该不该Dy  这种感觉Ey BrexitFy  北京市房Gy  ▁SimsHy  ▁MarriottIy 奔走Jy 漩涡Ky 贴息Ly  一周内My ippleNy  ▁SharedOy 李宁Py 永辉Qy  ▁spousesRy 登顶Sy 神情Ty  能适应Uy  对项目Vy  通报称Wy  其他平台Xy  制作了Yy  适用法律Zy  ▁trustworthy[y OTC\y  ▁Byron]y  ▁introductory^y  ▁Shak_y  越野车`y  喜怒哀乐ay  特此通告by 周报cy  科技馆dy Daysey  ▁Robbiefy  放射性gy  ▁stubbornhy  ▁itineraryiy 分清jy 待发ky 心火ly  但不要my  的心声ny  ▁Wheelsoy  ▁motivepy iziqy  ▁pupilry  ▁Seminarysy ▁OTty  旅行史uy ▁Kyrvy Bellwy  中南大学xy MATyy  的人都知道zy YOU{y 视作|y  ……(}y  这个话题~y 上皮y DELy  ▁monety  ▁stunnedy 致谢y 玉兰y 苍南y  洗发皂y 无损y 烟民y "],y avoy 我便y 生蚝y  ▁panoramicy 扑克y 点子y  你就能y Ratey igary  ▁Costsy " 第七次全国人口普查y 跌破y  恶劣天气y testsy 长年y 劲爆y 服刑y veredy  ▁roofsy Lity  创新中心y  如果有人y  ▁cavesy 乌云y 酗酒y 出圈y 无痛y 杨振y  分辨率y  大放送y  ▁Chicky 战的y 阳痿y ктоy  粤康码y plainy  ▁villagersy 倔强y 煲汤y  无形中y " 国民经济和社会发展y  ▁Explorationy  挡不住y  融入了y 网名y 面授y  又或者y  艺术团y  卡机上的y vasivey  ▁Reveny  ▁insurery  白细胞y  通知为准y 世故y 他乡y 蒸蒸y  安全出行y  预约电话y  ▁alliedy  ▁BGCOLORy  ▁restsy 在有y 落落y  上半身y  染色体y 品相y 授信y  对象为y 栓塞y ICCy  张家港y  增强体质y 诱因y 说好y  ▁ripey  ▁nurturey Ruy 越小y манy  工学院y  春季学期y  ▁Middletony  ▁Cynthiay 排练y 配制y  舍小家y  ▁Coloringy  ▁Personallyy  覆盖了y 暖阳y ujey ▁iby  ▁communicatedy 水火y  几乎都是y  招聘简章y  regationy  具体操作y  ▁tappedy 就问y ▁jogy  交流合作y  ▁bucksy FYy 无房y  ▁Cancely  ▁Coachingy 亚于y 动听y  ▁burgly  ▁vapory  五千年y  工匠精神y  internationaly  ▁interviewingy 偏僻y  十五条y  萨克斯z  最新情况z  如果一个人z Reallyz 四分z 着色z  可以随时z 用小z lobz  暖心的 z  的结合 z  共产党人 z elsius z  ▁chromos z  谁都不z  阳光房z  可能影响z  ▁objectionsz  ▁arraysz  ▁Neurologz 逐个z iciaz 敌方z  的介绍z  田间地头z  ▁resonancez  traditionalz  李子柒z  胃黏膜z  企业提供z cjz 年迈z apis z  ▁Kosovo!z  ▁redeem"z  ▁elevate#z  ▁feasibility$z 光纤%z 天线&z 派人'z 翻新(z  规模最大的)z 五入*z 有变+z  ▁skew,z  ▁unpleasant-z 还行.z nite/z  立竿见影0z  ▁lateral1z itsu2z  ▁Killed3z  ▁Strugg4z owo5z ▁aan6z assium7z 干道8z ▁CDT9z ▁pac:z  ▁Illum;z 静待z  ▁therapists?z  健康问题@z  ▁lordAz 波音Bz 皇上Cz  将无法Dz IllustEz '　Fz 头号Gz 案子Hz lestIz TweetsJz  ▁reiteratedKz  在南京Lz  当一个Mz  自己有Nz  ▁JaguarOz  ▁SuitesPz  把它们Qz  各式各样的Rz  ▁allegesSz  ▁EthereumTz  ▁pedestriansUz 逾越Vz  ▁symptomWz  ▁coherentXz 拼读Yz 江海Zz ▁Mé[z  进程中\z 咸鱼]z 宜人^z dash_z  天以上`z  ▁resolvingaz  ▁stationedbz 双全cz  对其他dz  Generatorez 难堪fz IZEgz Corthz  ▁Erieiz  ▁unsignedjz  责任意识kz  ▁Ninjalz  ilibriummz  ▁imprintnz  ▁Regardingoz 入行pz  ▁wanderingqz  ▁coworkrz 心间sz  亲访友tz  ▁Probuz  ▁florvz  ▁discswz 过山xz  带动作用yz  的原因是zz  雷克萨斯{z ◐◑|z 胀痛}z  一起看看~z 无奇z  介绍的z  推荐表z  如此的z  ▁Sickz 老酒z 要来z  实验幼儿园z  ▁antibioticz  ▁Taraz plantsz  ▁chickensz 就打z iniaz  ▁missesz  尝不是z 到此z ▁Lalz  科研机构z ABOUTz  寸步难行z  ▁tossedz  ▁illicitz ullaz  ▁beltsz  ▁Nonethelessz  ▁constituencyz 乌黑z Lasz  理解了z  版权归原作者及z  幸运的z  谋发展z gottenz ninez 战火z Rogerz  inishedz 噢噢z 我曾z ▁takz  能力建设z  ▁hashtagz  成了一个z  ▁inaccuratez 压岁z  提名为z Allenz  ▁Elliotz  上门服务z  ocatingz  ▁Dumpsterz 想像z  一般会z  不敢相信z  信息显示z  ▁Sealz  ▁Victimsz  等不良z  如果可以z  市卫健委z  ▁numericalz  办理的z  商品的z  准备好的z  ▁witnessingz  经理人z  特别声明z 意在z  中写道z  胜利的z egenz  ▁Mastz  作用于z WAREz  考生本人z  ▁poppingz carsz  国家规定z  思政课z ▁Nemz  的新时代z  ▁Alexisz  行李箱z  实名登记z Coachz 古村z 脏乱z uningz  民族复兴z 圈粉z  人民调解z  ▁Pulitzerz  ▁circulatingz 外用z  展览馆z  当然不是z  检验标志电子凭证z  ▁Hardwarez  ▁liabilitiesz  ▁Parliamentaryz 银子z  ▁Niagaraz  扫码加入z  ▁lowsz 为非z 大江z 审校z ckez  摄影作品z  ▁Extremez  ▁keeperz 不免z  对一些z  院子里z  使用日期z  事发后z analyz 后门z  流行性z oirez  ▁Kinderz 发量z 谷子z  努力把z  辨识度z  ▁pastoralz  余人次z  等设施z lichen{ 大可{  为载体{  形象的{  请及时联系删除{ 均以{ 援疆{ 病程{ ▁Bid{ ldon { rews {  心理测试 {  社会影响 {  进行排查 {  ▁Mens{  ▁Knowing{  ▁altar{ 履历{ 爆香{  西北地区{  ▁Chic{ Nobody{ 题名{  降压药{  体温正常{ 小宝{  平均每{ glers{ 上场{ 中止{ 纸箱{  商业化{ Phones { 贺州!{ 缝合"{ 雨林#{  ▁discern${  ▁Expansion%{ 菜刀&{  欢迎您的'{ 平南({ 私教){  报考的*{  新员工+{  ▁theor,{  ▁Deadline-{  自我隔离.{ 介于/{  国庆假期0{  移风易俗1{  ▁scarce2{  学起来3{  给了他4{  ▁melodic5{  ▁collateral6{ 三两7{ 叫他8{  没想到的是9{  ▁stagn:{ 不前;{ 台独<{  学子们={  等区域>{ 立在?{  世纪的@{  各不相同A{  ▁necklaceB{ 省略C{  卡就是不D{ MoviesE{  ▁ObamacareF{ 排忧G{ 数年H{  思维和I{  波士顿J{  这可能是K{  MinisterL{ etiM{  自然科学基金N{  ▁sensingO{ 圣母P{ 家访Q{ ucesR{  ▁mundS{  aspberryT{ 某事U{  带动了V{  也不愿意W{  既然是X{ 出狱Y{ 留香Z{ ▁Bj[{ ▁IB\{  先锋队]{  十分重要^{  所在地的_{ 同框`{  时表示a{ ▁Noub{ 会商c{ 废气d{  ▁Alonee{  疫苗接种工作f{  ▁rainforestg{ 蒸馏h{  以下内容i{ 滑板j{ 硬的k{  ▁Puppl{ movingm{  商贸有限公司n{ 煽动o{ arthp{  就没了q{  ▁inspectorr{  ▁concessionss{ 简谱t{ 趋向u{  ▁deductionv{ 树牢w{  及时将x{  隐藏着y{  征集活动z{  请联系后台{{ 年时|{ 腿型}{  会继续~{  很舒服{  疫情发生后{  ▁unhealthy{ 产前{  :①　{  会逐渐{  esville{  ▁characterization{ 丽萍{  ▁longstanding{  耳熟能{  ▁sickness{ 手无{ 新店{ 见于{  单选题{  和重点{  对全市{  放下了{  ▁Bundes{ ogi{ ▁HB{  可再生能源{  不胜数{ kang{  ▁Remem{  ▁disadvantage{ 弹奏{ atsu{  如果一个{  ▁Lank{  ▁Wand{  ▁Cheese{  在疫情防控期间{ 购药{ 鸦片{  以上信息{  相关工作经验{  ▁Triangle{  这也是为什么{ 天台{ 狩猎{  的理想{ Phys{  精准防控{  ambién{  ▁separating{  ▁provocative{ 接吻{ 不二{ 临街{ 牙痛{  ▁flats{ 老张{ érie{ 沉沉{ 闭塞{  这段话{  关注订阅{  ▁Carpet{ 令我{  有太多{  黄晓明{  百度网盘{ 增城{  ▁COND{  ▁Djokovic{ 微妙{ /#{ 主板{ 多于{ 锦衣{  陈老师{  ▁asserted{ 轮换{  在企业{ 双赢{ 豁免{  张伯礼{  效果的{  类风湿{  boolean{  ellaneous{  系统学习{  ▁Omega{  少聚集{  有大量{  ▁Liberia{  ▁Theology{ nat{  不适宜{  在本次{  防汛抗旱{  ▁Canterbury{ 水洗{  这种事{  向上滑动{  秋季学期{ ▁ц{ 警觉{  专委会{  亲人的{  做错了{  ▁Burk{ 均需{ 议会{ 采编{  和妈妈{ 友爱{  始终没有{  ▁brilliantly{ 白虎{ ▁Roe{  有机结合{  同等条件下{ 实况{ 幕墙{ 手软{  耳熟能详{ 仁和{ 癌变{ 选的{  ▁paddle{  ▁Borders{ 自成{ bek{  在欧洲{  ▁geometric{ 中环{ 迎风| 口口| 早熟| 海归| 精辟| 老二|  刻不容缓|  最近几年|  深刻领会|  NSString |  ▁primitive | 长官 |  的调整 |  自己想 |  为人处世| 无惧|  仙游县|  脚手架| ▁ell|  ▁enhancements| 蒙城| 面食|  专业性| 再难|  不经过| Pret|  得到控制|  ▁expressly|  ▁electronically| 不忙|  人事处| wolf|  黑社会性质组织 |  ▁Pharmaceuticals!| 大字"| 新郑#|  投资建设$|  ▁Speak%| ▁tan&|  大大提高'|  武器装备(|  并主动)|  ▁Arist*|  ▁slick+|  ▁Meteor,|  ▁Franchise-| 却要.|  改善了/|  conference0|  污染的1| Intel2|  wrapper3|  ▁regimes4|  最主要的5|  ▁sidewalk6| Tu7| rf8| GEL9|  ▁advancements:| ynt;|  日新月<|  突发情况=| 买点>| 鼓起?|  也有可能@|  普通家庭A|  管理部B| 奥体C|  ▁beadsD|  ▁WorcesterE|  必须的F|  补习班G|  落实落细H| CyberI| 优厚J| 锂电K|  འི་L|  一无所M|  以满足N|  哲学家O|  动脉粥样硬化P|  ▁silhouQ| 两车R|  出一种S|  加班费T|  这一问题U|  也正是因为V| 之灾W| 木马X| 此日Y| 深有Z|  疫情信息[|  给我们的\| Behind]|  ▁recruits^| 李清_| inform`| 开篇a| 莫属b| PVCc|  在生产d|  的真正e|  ▁pedalf|  卫生健康部门g|  ▁portfoliosh| 东昌i|  ▁Involj|  ▁smiledk| 佛系l| 囤货m| 小路n| 相看o| ewaysp| atchesq|  瑞丽市r| 美剧s| U N▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁t| 万套u|  十堰市v|  郑州大学w| circlex|  ▁disparitiesy|  ▁Guangz|  ▁grocer{|  ▁vanity||  ▁Thousands}|  善良的~| ilio|  中考成绩| 仕途| 山头| 开悟| 放慢|  发布通告| gren| sales| "'|  Selection| 寨村| 由其| ▁',| ▁Kem|  ▁Steele|  ▁dumped|  ▁reopened| 姐弟| 捕鱼|  ▁Intent|  ▁youtube|  坚持以人民为中心| 境遇| 王伟|  地中海|  教育行业|  违法时间|  我们每个人| 大德|  ▁Stevenson|  全因子| Fill|  ▁specials| 主宰| 施压|  效果更好|  这张照片| Built|  ▁INTO|  ▁enquiries|  responsible|  本资料|  的思路|  一网打尽|  ▁ethn|  困难和问题|  ▁roasted|  ▁distributions|  不难发现|  电子注册|  课程标准| 上位| 易烊| 邮报|  消费的|  ▁pops| 功臣| 上床| 海里| maybe| 追忆|  发展成|  教室里|  理发店|  也出现了|  还有不少|  ▁Penguin|  ▁mileage|  Officials|  ▁perimeter| 景物| 猜想| Nat|  ▁Scots|  ▁youthful|  ▁synonymous|  Keeping|  消除安全隐患|  ▁ransomware| 水文| given| ocaly|  ▁Okay|  ▁Morton|  ▁upbeat|  ▁corrections| 蒜蓉| DOI|  全面了解| 多亩|  金色的|  ▁Pension|  ▁dispatch| 曹县|  软组织|  ▁Patch|  ▁inventor|  ▁Citizenship| 衣裳|  相匹配|  为了解决|  周边环境|  ogenesis| 分了| 护佑|  坚持问题导向|  ▁Pocket|  ▁accumulation| ▁Yas|  含有大量的|  dominal|  querque| ighs|  停电线路|  ▁Judith|  陪伴孩子|  据不完全统计|  ▁Janeiro|  的青年| 灵气| 配菜|  三个字|  的直接|  ▁Arabian| goal|  ▁Moments|  ▁Provide|  ▁indications|  部分文章推送时}  却一直}  家人们}  直播中} 下放} ORTS}  三十六} loved}  ▁Ethiopian} 高僧 } Corn }  住院治疗 }  ▁Mormon }  任何形式的 }  ▁informing}  ▁plac}  ▁nickel} 恍惚}  感谢信}  windows} 自达}  多一个}  少部分} liga} ungal}  ▁Luna} Models}  ▁stringent} 引出} 调出}  看到这个}  随着时间的推移} 地问 }  的真相!} uzzy"}  作业人员#}  西部计划$} guang%} 初赛&} 房型'} ochond(}  ▁reaff)}  ▁peripheral*}  ▁stimulation+} 契约,}  但如果你-} 抄底.}  纽约时报/} 吃透0} 怜悯1} stv2}  ▁wary3} ▁Ч4}  防晒霜5} acting6} eff7}  好运气8}  聪明人9} jected:} 如花;}  及时删除<} 长生=}  ▁innovate>} 歌剧?} 谋求@}  学到的A} 迟钝B}  关注了C}  ▁facilitatesD} 印有E} ousseF}  ▁CatalogG}  ▁unravelH}  ▁identifierI} 一叶J}  了大家K}  停放在L}  要么是M} AdobeN} 金森O} xingsP}  ▁diversifiedQ} 喷泉R}  无不良S}  甚至比T}  averageU} 享誉V} 白带W} metalX}  ▁stuntY} 手慢Z} 泡面[}  超高的\}  如果你们]}  ▁bere^}  好一点_}  学生证`}  校园内a} 沸水b}  在外地c}  就是从d}  扑面而来e} ▁Bacf} soleteg}  二级建造师h}  污水处理厂i}  后可以j} ientek} 一盆l} 寓言m}  ▁Noeln} 后市o} 杨晓p} 民安q}  四舍五入r}  ▁Editings} 连用t} 郑某u} PCAv}  免费通行w}  粉丝的x} 尚书y} Taiz}  连接线{}  隐藏在|}  有人认为}} 方舟~}  向世界}  研究机构} 则有} ▁dod}  fashioned}  ▁Genetics} 拍手} ▁GS} facts} 住着}  自主学习}  自主选择} mouse}  ▁penetration} 总和} 援引} 班里}  驱逐舰} gj} 本体}  总冠军} qz} 那人}  ▁SpaceX} 触感} сту}  ▁Direction} 可耻}  的景象}  进一步规范}  的小说}  就相当于} 将士}  下个月} INST}  upuncture} 拿证}  缓解了} combe} Muslim}  busters}  ▁gossip}  ▁Liberation}  你不会}  锦上添花} dragon}  pointer} 刷爆}  汽车有限公司}  ▁rehearsal}  的疫苗}  ▁amusing}  我这个}  落后的}  ▁waking} 认错} зе}  校团委}  平面设计}  这方面的}  Compare}  ▁Symbol}  提供便利}  ▁Punk}  ▁childcare}  ▁prominence} 首乌}  充电器}  证据证明} olding}  ▁deeds} 咫尺}  Commission}  供电所} 小猫} 灌木}  控股集团}  第四季度}  ▁heel}  ▁chaired} 依山}  白癜风}  房贷利率}  也很重要} Aside}  ausible}  ▁Definition}  ▁constraint}  ▁redevelopment}  的红色}  王王王王}  withstanding} 读研}  更重要的}  需要帮助}  人员信息}  公务员法}  ▁Pale}  微信朋友圈}  过程的}  高考生} 相冲} ▁OC}  ▁jihad} 工装}  的利益}  你的未来} parts}  ▁uranium} ↑　}  诞生了} yield}  ▁Españ}  ▁eater}  ▁neurological} 之高} 甘愿} 篡改} 他俩} 迎着}  比比皆是}  欢迎广大} 避难}  ▁dynasty} 三水} 垫付} 庇佑} 想得} 私聊}  万以上~  担使命~ 刚烈~  请家长~ elif~ 价廉~  外三元~  文件的~ Chen~  疫情发生 ~ 重创 ~  健康信息 ~  社会发展 ~ 日经 ~  ▁lace~  ▁busiest~  ▁galaxies~ 满腔~  初试成绩~  ▁scarc~  ▁resumed~  桥街道~  结婚证~ 故而~ 身形~  心理压力~  ▁Loud~  ▁beneficiary~ 玩水~ 长处~  ▁Philly~  ▁disconnect~ 早泄 ~ ▁ye!~  剩余的"~  备案表#~  我今天$~ ashire%~  ▁Fabric&~ 五日'~ 路西(~  utation)~  ▁Astronomy*~  ▁transcend+~ 平定,~ 菜场-~  ▁Kremlin.~  ▁stumbled/~  到北京0~  很严重1~  一路走来2~  ▁Vall3~  ▁wiped4~ czy5~  行程轨迹6~  ▁pneumonia7~ uu8~ orb9~  曾经说过:~  正常生活;~  较小的<~  高考数学=~  模式下>~  第二类?~  Boolean@~ 红码A~ HumB~ UriC~  ▁EdisonD~ 各司E~  ▁PietF~ 武林G~ 青色H~ 高利I~ 等非J~  支持力度K~  ▁LimaL~  远方的M~  ▁hurryN~  ▁jerseysO~ 长痘P~  杂交水稻Q~  ▁ExitR~ 国风S~ 无果T~  和努力U~  ▁rapedV~  ▁VolunteersW~ 福彩X~ 联军Y~  后疫情Z~  学科的[~  Arizona\~  ▁Implementation]~ ('/^~ ème_~ 无休`~ Cancera~  ▁softballb~ 本场c~  复杂性d~ aguse~ ▁SDKf~  ▁trapsg~  Missingh~  ▁resenti~  ▁spiderj~  微循环k~  来势汹汹l~ 提档m~ 着她n~  结果为o~  不懈努力p~  ▁Killingq~  ▁reactedr~  ▁Biblicals~  ▁brightnesst~ 阜新u~ Halv~  不对劲w~ CPIx~  ▁Listsy~  Identifierz~  管理机构{~ OST|~ uca}~  劳动保障~~  可爱的小~ 里说~  的解释~ 两场~  就业困难~  ▁TABLE~ 懦弱~ 铃木~ ▁NIH~  加班补助~  ▁togg~  ▁blunt~  ▁relic~  精神文明建设~  北部湾~  ▁Asked~  ▁majestic~ 舰艇~ 鹏飞~  民航局~  作出新的~  坚定信念~  ▁concise~  多方面~  ▁mornings~ 来吧~ 高歌~  开学季~ ▁FAA~  ▁recharge~  十二月~  考试前~ Pri~  对人的~  写下了~  司法鉴定~  培训的~  一丝不苟~ : 3▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁~ 出纳~ 闺女~ 农庄~  某公司~ ▁MAG~  相结合的方式~ 失恋~  获奖证书~  ▁dreaming~  ▁combustion~  促进学生~  ▁Perspectives~ 文静~  与众不同的~  ▁gestures~ 几人~ 辛酸~  分不清~  初中学校~ 五楼~ 拭目~  ▁condemn~ 北街~ 食道~ ▁DH~  ▁Augustine~  ▁hepatitis~  ▁Presentation~ 脑力~  集中力量~  ▁fist~ 博学~ 雪峰~  岁高龄~  ▁trait~ 人命~ 居于~ 生熟~  的光辉~  ▁Narendra~  ▁prosperous~  ▁giveaway~  ▁experimentation~ 以无~  中间商~  做核酸检测~  ▁Therap~  ▁Macedonia~  的那一天~ 多花~ 烂尾~ 睡衣~  ▁dude~  ▁Gupta~ 伤的~  ▁сво~  以下拘留~  ▁totaling~ 晓得~ 独行~ 结账~  进项税额~ 那只~ 游离~  第一反应~  那么多的~ 收了~  年全年~  发生碰撞~ 不配~  ▁Quiz~  ▁Watching~ asan~ ansen~ adores~  ▁Factors~  ▁Meadows~ 和交~ 遗弃~  请尽快~ 秀色~ 缓缴~  后备箱~  戴着口罩~  ▁focuss~ 动容~ 暴雪~ 李国~ 驼背~  我们今天~ 牧民  有人会  日新月异  提醒广大市民  ▁finite  ▁entrust 这把  ▁Brow 医养 熔喷  留有   的作者   说了算  初见  打消  以上图文  ▁teaser  ▁Weinstein 玉器  文学作品  ▁Levine 基坑  百分制 acha  ▁CNBC 办的  办公场所 欧亚 热敷 导线  技术研发  ▁overr 春花   阳光的!  隔离病房"  要达到# ituary$  ▁financed% 抽搐& 拉的'  刃有余( 正定) 进校* nuts+  教育督导,  ▁Cater-  ▁airing. YA/ 各处0  个多月1  温馨的2  ▁circumstance3 tf4 找来5  不适应6  造成伤害7 基点8 羞辱9  口中的: 神咒; 黑茶< ▁lol=  变成一个>  ▁Huss? umbles@  isexualA 举世B MPVC 王氏D  讲到了E  ▁addictiveF 涉外G  材料和H  ▁synthI 还用J  再出发K  能满足L  ▁DeliverM 蜂群N  ▁TempO RXP  建设规划Q  监控视频R  ▁avenuesS 主语T  是实现U 降费V  消费需求W  编辑的文章X  terrorismY 先看Z 趴在[  届高三\  适合你] ▁Rox^  ▁poorest_ 五点` peza  原产地b  了一会儿c  ▁Recapd  ▁Clearlye 巴尔f  转自网络g 急事h  土地管理法i  buquerquej 别有k  显著的l  学科带头人m  ▁pesticidesn  ▁Reddito 引力p 房地q  安心的r  宝宝们s  情节严重t  ractiveu  ▁Unifiedv  ▁rubbishw 精打x  补贴政策y 陈家z  还不到{  以自己的| uddle}  ▁compulsory~ 井盖  ▁Vacation 磷脂 ▁SSL 织密  ▁Crest 发包  ▁Visitor 畅快 硬朗  绥化市 oyer  ▁Israelis 数和  专家评审 Edited  Madison Bot  上线啦  无害化  ▁earthquakes ;⑤  ▁Funny  ▁versa  工作人员表示  ▁Monkey  可转债  环境好  环球网  ▁torrent 条条  时间另行通知 寄递  养生的  可自行  政策法规  服务企业  ▁Scripture 时已  新鲜出炉  留学服务中心  ▁scaled  ▁blending  ▁formulation  及个人 ▁Mys  工作总结  规模以上  ▁Friendly  干部的  恢复开放  绿水青山就是  ▁Burlington  被警方 李四 ánd  嫌疑人的 涂上  前女友  后才能  可惜的是  ▁Julius 不干 之隔 亲测 白金  发放的  集中精力 Veter  Actually  ▁doctorate 几步 加试  塌糊涂  很清楚 oine 打新  只用了  ▁від 变小 坐车  ▁Barker 芳菲  更加注重  ▁smoked 双边 身家 一式  遇到一个  ▁legitimacy came icidal  ▁screenshot 尝到  在确保 kish  受益匪浅  对得起 ▁sta  ▁blat 见人  周年之际 切口 怪的 砍价  ▁Berk 单亲  至少在 ལ་ 玩得 耳光 Desk  ▁cosmic  消化的  ▁ultrasound 公检 告急  按程序  ▁extras  ▁Announced  ▁adulthood  ▁validated  理工类  定点医院隔离治疗  ▁Counseling 马伊  ▁curl ▁coq  ▁filmmaking 其职 而入  医疗中心  ▁suppression  ▁Marijuana 市镇  远高于  ▁influencing 有云 针刀  ▁Sigma  减少人员聚集 Rail omens  ▁Abbas  ▁shouting 硬盘  已经没有 Centre 建安 黄素 wor  下载打印 Oak  将不再 教于  并保持 orno  努力打造  ▁Bolivia  适应症  ▁Gentle 佐证 德县 较早  提升至  脱口秀 omorph  新冠肺炎疫情影响  中华儿女  printed  ▁citations  有光泽  宣传贯彻  建言献策 ør 数月 文集  目前国内  酸甜苦辣 itched 可留  为了确保  目前我国 一哥 Issue imates ocon  рова 披萨 苏醒  年纪大了 但未  的一声  ▁Turns  为深入贯彻落实 得心! 虚空!  可以快速"  民族团结进步" 患癌#  做好的#  拭目以待$  月份以来$  ▁downturn% 对峙%  新年快乐&  ▁hairst&  ▁Andreas'  foundland' 怀化( MAR(  确诊人数)  ▁Closed) 电白*  商家地址*  提出要求+ 仅售+ 当红, 收件, 诽谤-  获奖作品- ymoon.  ▁fishermen. 一枝/ 募捐/  然后就0  门诊部0  工作生活1 sites1 身手2  对党忠诚2  尽管如此3  海鲜市场3  ▁blows4  ▁openness4 "{5  ▁siege5  ▁instructional6  张艺谋6  ▁notions7  ▁derivative7 很早8  怪不得8  ▁vocational9 佛学9 鱼塘:  快速增长:  环球时报; 离场; 威信<  我们以<  ▁improvis= Else= aviour>  ▁roadmap>  公共基础知识? 停用? 单打@ 见不@  行政区域A 解暑A  血液中的B  针对不同B  ▁debatedC 越低C  与父母D  进一步加大D  行动上E 信件E 汗液F кийF  中英文G  ▁meltedG 办了H  独立完成H  ▁BergerI  ▁freezerI 而复J  担心的J  一方面是K  ▁thruK 北流L  公费师范生L  ▁delicM  ▁radicallyM 堵点N  真的能N  等地区O  ▁finalizedO 欲罢P 歼灭P 等职Q  一模一样Q  ▁decreeR 小青R ▁GWS  ▁grilledS 圆的T  新起点T  ▁SupplementU 宫寒U  真的要V  战略合作协议V 姑姑W (__W  增加的X  ▁disruptedX 大千Y  的确是Y  特朗普的Z  ▁fluorZ 兼程[  ▁Kemp[  ▁Signature\  ▁Therapeutics\  ▁obliged]  ▁realism] 片中^ 血亲^ ▁Ved_  ▁villains_ 尿频`  公示期间` 王安a 路易a ambob  ▁Buyingb  ▁eagerlyc  ▁integratesc 成形d  创造的d  帅气的e  ▁Ellee  ▁Toysf 威慑f  ▁emptg 健壮g 女工h 闭门h  核潜艇i  音乐的i  乱停乱放j 京都j 盾构k 西青k ▁PFl  天气炎热l  真实有效m 溺亡m  ▁Covenn  ▁drasticn ▁Zaro  延年益寿o  艺术类专业p posingp 捡到q  助教老师q  增值税发票r  笔试准考证r  轻而易s ▁Hofs  ▁Vampiret 新街t 电容u  伙伴们u 行医v  每人每年v 放贷w  ▁Nigelw  ▁Commitx  ▁Manufacturersx igeony  ▁Cybersecurityy  冲击力z 被你z  ▁commissioners{ 临其{ 常备|  ▁mais|  ▁crystall} ammu}  易烊千玺~ ussia~  ▁Counties  和环境  统一思想 逆风  主要在  个人缴费  就会产生  ▁Apartments 家训  ▁Monetary  ▁Dangerous  更是一  ▁Ministries  ▁residences 室温  充分调动  规定的其他 文革 插手 acio ▁Kro 多酚  招生入学  未经核实 Train  药品监督管理局 ▁GEN  临床应用 阅兵  的原理  高考的 月嫂 ▁BET  而且还有  ▁apparatus 美誉  及配套 ▁ENG  管理措施  informed  messages  ▁Waterloo  ▁outrageous 奉行 盐和  经营权  地下车库 % 公安机关交通管理部门 ▁½ 加法  已经连续  柔情似水  Britain ▁CU  ▁regeneration 满堂 Rab  你为什么  ▁Transl 五里 平果  斜对面 华庭 大话 饲喂  考试卷 Tools 作词  学以致  公示时间  电子设备  ▁sobre 擦干 PAS  针织衫  ▁terribly  你还是  日复一日 бли  ▁Mama  也不少  暖暖的  ▁Mineral 工整  ▁Destiny  心里有  杨老师  采访了  ▁tuber  ▁Scholarships 守着 谚语  小的时候 让利  起好步  ▁infrared + $事业单位公开招聘工作人员 清空  ства  扫码报名  提起诉讼  查看公告 重申  ▁Dairy  大大降低  ▁slept  direction 惧怕 EPC ▁CN  有计划  脑出血  ▁Rockets 吸水  体育类  年我国  ▁Thorn 长高  ▁Lori 朗朗 春江 言传 详询  新气象 附赠  ▁Bram  ▁Rear  ▁Roses 背靠  该女子  ▁Sikh  telling RUN  宿迁市 reve€  轻而易举€  ▁JungleÀ EtÀ 我真Ā vezĀ LOCKŀ  ▁Daddyŀ 音箱ƀ  在印度ƀ  大熊猫ǀ  人们的生活ǀ  ▁specialisedȀ  的首选Ȁ  聚在一起ɀ  ▁ratiosɀ 游学ʀ  岁之前ʀ  疫情下ˀ  更不是ˀ 亚麻̀ 精粹̀  的款式̀  为了不̀ ▁Wow΀  政治生态΀  ▁Socialistπ  ▁Collaborativeπ 伶俐Ѐ  的交易Ѐ  体育学院р  ▁Meetingsр  ▁calculatorҀ  工作坊Ҁ  点一个Ӏ ▁JabӀ  ▁revelationsԀ 眼见Ԁ LABՀ  violentՀ 天命ր  注册的ր 佛祖׀ 头孢׀  ference؀  ▁Royals؀  守护着ـ  篇幅有限ـ  和无症状感染者ڀ  ▁Parsonsڀ  　”　ۀ  就是想ۀ  itating܀  ▁Selling܀  ▁allergic݀  ▁spontaneous݀  第一节ހ  就在这里ހ  详细询问߀ 百名߀ 表里  Playing  ▁Jensen  ▁contradict 离谱  恢复原价 azers  ▁Satan  Visible  ▁contend 校审  ▁bubbles  ▁visualization 也难 四条 研磨 Labor  ▁Bruins  ▁glowing Craig  ▁filtering 受力 }",  暂行规定 parser  ▁Autism  ▁motives 狡猾  比以前  各级党组织和 的低  扩大到  ▁Fees  ▁trailing 水草 滞纳 yue ▁rav  面试资格  ▁FIRST  ▁Augusta 年为 润泽 艾伦  一盘棋 房里 行间  ▁Stats  ▁drawer GV 鱼龙  学校将  变得更好  科学学院  ▁wird  ▁restrictive 不按 她把  控制性  减速慢行  ▁Byrne  ▁Accessibility 书信 同工 Fan ▁ASP  得到有效  跟他们 espan  ▁IOException 甲级 罹患 较真  被录取 iping  ▁receptors ★★ 万般  或直接 thanks  年以上的 Border  ▁flavours  ▁courageous 平生 一国 腹地 тив  等事项  的初衷 ller ▁Yun  ▁Ashe  ▁Humanity  科研项目  ▁Daytona 悔改 接盘 敲响 Msg ▁Odd 索引 胆子  ▁aided  ▁professions  地理标志 微淼  政治权利  ▁Seek  ▁onsite 总想  代人的  未知的  ▁Bayern  ▁Showcase  ▁chilling 下饭  篮球场 多在  排头兵  公示期为  ▁Brooke  projects 学车 山寨 平息  大理石  第二医院 酱料 fell  ▁marching!  elected! gged" Owner"  Exclusive# 固废# CHA$ Bring$  technology%  ▁assertion% 合辑& 尔特&  ▁Moines'  ▁filtered' WWE(  去的地方( 伊万) 单列)  点击查看大图* 感言*  可以增加+  初心和使命+ Weight, 学时, ignon- 告之- 语境.  全会精神.  性关节炎/ 迷迭/  上了热搜0 重金0  ▁dissolved1 宗祠1  党委宣传部2  ▁informs2 驻华3  如鱼得水3  芥子园画谱4 打气4  与国际5 ▁Ler5  ▁surname6  是不可能的6  Material7 原形7 备份8  迅速行动8 iatus9  ▁Behavioral9  最后还是:  组织部部长:  ▁Atomic; 投向;  不小于<  英特尔<  ▁PHOTO= 诱骗= FIG> Tesla>  Freedom?  文中观点保持中立?  天下第一@  Manufact@  则表示A  开具的A  桌子上B  外出务工B  ▁diarrC  脚下的C  所需材料D “‘D 查纠E  ▁BibliE  ▁slammedF  这些小F  他们就会G  无忧无虑G  的高质量H 韩信H  后来又I postedI  affectedJ  为维护J  团购价K drugK  欲罢不能L  ▁VegetL  ▁spermM 欠薪M ▁KiaN  这么一个N  出版的O  发表在O  ▁BarsP  ▁OccupationalP NESSQ  第一责任人Q  不必要R  会影响到R  质量控制S 大床S  ▁deedT  ▁unconventionalT ▁AscU 三一U 二七V 可于V  ▁bowlsW  ▁LeBronW 中通X  外国语学院X 外援Y  必需的Y ▁GeoZ  但是这个Z  守望相助[  西南部[ 余万\ 报答\  与交流] 送进]  朗读者^  ▁Establish^  ▁Wilmington_ Dance_ IGHTS` fas` ▁Rita  ▁Disneylanda 青筋b  喜欢在b 饿死c  眼里的c  群众性d  费用的d  扶贫工作e  ▁Deluxee 压倒f 汁水f  patterng 并从g wifih  ▁Casah 英杰i  的肌肉i zmanj  ▁matesj  ▁sensationalk  数十年k  免费学习l  脾胃虚弱l Reillym  ▁Flemingm 扫墓n  和保护n  ▁migrateo  要避免o lockedp  ▁footwearp 事成q 药食q  三甲医院r  贷款利率r  Voluntes ▁pess  财源滚滚t  ▁Deutscht  ▁craftsmanshipu  ▁Swimmingu 美到v  学术论文v ANIw 上等w 本的x  学有所x 矫情y 老道y 分至z 暗黄z ▁Til{  再也不会{  ▁Stamp|  ▁Output| MK} 行深} 钢笔~ 从句~ 克林  团队精神  爱好者的  ▁reconsider  ▁Bios  Authors  母公司  如果喜欢就 delta 拉手  及其它  ▁Liga  等城市  ▁Raven  中选择  九江市  但最终  马自达  继续努力  ▁Greenland  我们总是  立即启动  ▁Interface 学工 踪迹  不难看出  环游世界  百年奋斗目标  ▁canopy }{\ ▁TED  ▁detached  对她的  派出所接到  ▁Automatic 鸿燊 Peace  ▁Assist 初选 抓起 ▁CRE 派送 膏药 转诊  也必须  华侨城  新政策 amoto  ▁Pitch  ▁china 李鸿  ▁greatness 大叫  这一块  ▁verge  ▁Exeter 谓之 马齿  密切接触者的 六种 妇产 织物  安全距离  ▁Swim 报恩 翠绿  防灭火 路和 Ori  没有钱 ▁AMA  那么你们 标称 burst 寿险  十一届  ▁Boards 沉甸  一个月后  ▁Stephens  的家人  可以尝试 都由  泰州市 Mayor  后来到  坐下来  这就是为什么 鼻孔  与技术  云闪付  经济实力 内江 山峰  ▁Kara 辉映 ▁HC  提供更  Applications  侵犯到  商投资  ▁Lucia 壮丽  等方法  一张张  自己能 fake  ▁overcoming  游乐场  超出了  师范类  ▁authoritarian 穿行  微电影  也会出现  Variable  城管局 征婚  孕产妇  人群密集  ▁nurturing  移动互联网  ▁coats  ▁Michele  ▁Nicaragua 三字 丽君  带领大家 常考 道法 门关 高照  人才招聘  ▁booklet 佳肴 健胃  致命的Á  问题时Á  ▁pseudā 硬性ā ▁koŁ  ▁grabsŁ  ▁institutesƁ 远古Ɓ XMǁ 喂食ǁ 夸奖ȁ 托克ȁ 评标Ɂ  东北风Ɂ  设施的ʁ  ▁tyingʁ  Unknownˁ 减去ˁ 数万́ 罢工́ laṕ Frań 带病΁  猝不及防΁  ▁Ottoρ  ▁beddingρ  ▁referralsЁ ketЁ Zoomс QUESTс  ▁Riderҁ  ▁Refugeeҁ 凯迪Ӂ 气机Ӂ ▁msԁ  的典型ԁ VertՁ  现场视频Ձ  ▁barredց 入账ց Precׁ  如何看待ׁ acons؁ culus؁  ▁gearsف  ▁Gravesف  ▁forwardsځ  ▁fascinationځ  经不起ہ  邀请函ہ neck܁ 常温܁ 文娱݁ 肥沃݁  宣传周ށ  调整好ށ undi߁  ▁Skip߁  Animation EQ 效仿 Stre  ▁Guru 免洗 随行 oter  绘本故事  本文来源于网络 凯里  这个国家  ▁Oriental 文图  大中型 逊色 Arc raged Bright  ▁BEST  富士康  小课堂  收官之年  ▁Shock 波斯  ▁Bermuda 裂纹  空中课堂  ▁injected 医患  上看到  绿色环保  疫情防控需要  ▁Algeria 共进 新洲  纪念章 审题 打水 琳琅  的念头 nv eph  说的就是 发火 性状 繁衍 并进 待命  的印象  的奋斗  ▁Capacity 为伴  行政许可  ▁Prepare 点此 GMP  走在前  全日制普通 才子 斗门 金花  生水起 流派 眼周 讲过  参保缴费  违法违纪  ▁paramount  Colorado 显眼 euro  ▁teamwork SIZE  公开选调 ły lause  ▁oxid  Creator  接种了  ▁Sentinel 救死 ▁\\  十六条 velyn  ▁elders 茶香  了一座  感染新冠肺炎 ucci  ▁Huang 白费 省际  安全监管  ▁Toll meal  ▁року  很多人认为 私房 ndef  ▁Bacon  市场监督管理总局  ▁carbohyd ( !接受纪律审查和监察调查 gf 一捺 建新 ▁NET ugosl 冒出  可以吃  为了提高  需要通过  ▁summers 深蹲 绿意  一定有  能够有效 arbon richt  ▁cites 胎盘 鸡爪  离我们 巍巍 无刻 ARP  的高中  ▁Claude  ▁biased вет!  开封市!  ▁Thur"  ▁coven"  ▁Postal#  ▁segregation# 诗和$  与合作$  地区有%  相辅相%  高分子& Larry& 捐物'  最低的' nova(  authored( 布莱)  通讯录)  即将迎来* ITA* geb+ Norm+ 小满,  走深走实,  ▁regulating-  某一个-  不容错过.  ▁Romanian.  各区县/ 土方/  规划中0  路线图0 对冲1 铃声1  发现场2  教学的2  ▁reel3  在我市3  疫情结束4 通力4 liche5  ▁deposited5  ▁surrounds6  ▁prohibition6 燕山7 画师7 WL8 惹人8  ▁Leisure9  ▁splendid9  ▁Fisheries:  ▁Terrorism: 呼救; 更近; 画廊<  衷心的< iken=  塞尔维亚= 南岸> 罂粟>  值班编辑? 拉扯? Morgan@ adjust@ 忍着A 白鹭A igraphB  ▁McLarenB #,C 就等C  细腻的D airdD  ▁ForkE  ▁translatorE 娘子F 里头F agosG  公益事业G  课程大纲H 绸缪H ihannaI  ▁impactfulI 博园J 问及J ▁SHOK  报警电话K  隔离措施L JacobL 淤泥M 硫化M 老城N 韩语N  ▁ParalymO 地地O 替尼P  专业不限P  innacleQ 你笑Q IDAR  在新的R  就来了S  ▁stripesS 筹措T  和生产T  的环境下U  ▁docsU  公办幼儿园V ERAV  第四个W 九点W  截止至X  高级感X 放电Y 雄伟Y  ▁turtlesZ 不痛Z  社会力量[  ▁Configuration[ 腿脚\  市民的\  ▁dome]  ▁smells] 是靠^  及时掌握^  学习党史_  两年前_ 坠入` undy`  中高风险a  欢迎各位a % 巩固拓展脱贫攻坚成果b Soulb  城乡规划c  不得不承认c  ▁artisand  ▁ubiquitousd 天了e 护发e  背景墙f  钢集团f steeng  aternalg  hhhhhhhhh  落脚点h  男子的i 漏电i 认筹j 越少j  公文写作k  Drawingk 南城l  解决不了l  ▁Baselm  当回事m  革命精神n 圆通n 外公o Soono  ▁nationalityp  司法所p  点个在看哦q  ▁vinegarq  ▁Theodorer  今天起r  让整个s  无新增疑似病例s  ▁Patriott  ▁unusuallyt  ▁belongingsu  严峻的u  元包邮v  开拓创新v  ▁sacksw  可采用w  电饭锅x  零花钱x ceptory  ▁contempory 名媛z 豪迈z  用过的{  relations{  不堪设想|  图源网络|  经常有}  社会公众} Fu~ 装满~ 分居 气滞 诸事  ▁lust 二郎 共勉 即兴 铲除 闲鱼  发现自己的 皮鞋  大气污染防治 煤业  ▁cyst  有关法律法规  ▁pollut 晚睡  备考群  荔湾区  毕业典礼  ▁Damon  在这个世界上 稀奇 TRA  用生命  ▁envis 捕获 izhou  ▁miscon  的队伍 Normal " 经济社会高质量发展 与大 打野  上涨了 开枪 速来 为天 理应  主观题  积极组织  ▁raced  ▁judgments 不差 厚爱 TAG  ▁condolences 存续 清点  小学的  ▁Carolyn  ▁commits 俘虏  出去了  和产业  和产品  性教育  离婚了  决战脱贫攻坚  市人民检察院  ▁impending 代缴 柳叶 消磨 跌落 ▁APR hentic  ▁Larson 任期 反恐 授牌 衣草 ▁Rag  自己身上  ▁purity  ▁Accountability 主攻 秘境  感染病例  ▁wicked 大帝 女兵  命名的 Senate 结肠 APS  岗位需求  ▁Madness 一科  不久后  ▁Towers 产城 果皮  记住了  ▁revisit qy tot  ▁Bengals 鄙视  营养的  ▁Spir  ▁Solic  与朋友  尼西亚  学史力行 ällor 的首  还会出现  estyles  ▁remodeling  可操作  体能测评  本次会议  ▁Mour  ▁Warfare igne  点击蓝字  ▁diplomat  专业组  继承了 esse ipel  ▁Sundance  菩提心  方式方法  管委会主任 枯萎 狂放 科举  连云港市 养肺 随州 黑洞 weekly ▁Ned‚  ▁Pilgr‚  ▁voucher 懂你  自动挡Ă  汇算清缴Ă  ▁motionsł (`ł mereƂ  ▁commendƂ 汪洋ǂ  而努力ǂ  延伸到Ȃ  对辖区Ȃ  这个数字ɂ RNɂ 敌军ʂ 淑女ʂ  ▁awake˂  ▁intermed˂  ▁Berkshirê Pin̂  各种原因͂  颁奖典礼͂ 向她΂  沉甸甸΂  ▁Ovalς 保教ς  ▁EleanorЂ  出现症状Ђ  研究结果т  ▁homosexualт  ▁streamline҂ 原液҂  ▁Uruguayӂ  ▁lyricalӂ 泪目Ԃ  ▁bourԂ  的坚强领导下Ղ  ▁vibrationՂ 之音ւ uyaւ  动不动就ׂ  Archiveׂ  可随时؂ 陆家؂  担负起ق okiق  天小白理财训练营ڂ  ▁multidisciplinaryڂ  民警的ۂ 一车ۂ  有所下降܂  分时段܂  播放量݂  体育活动݂  ▁stabbedނ  ▁compressedނ 亲历߂ 直降߂ ▁х CLUS Brain  ▁Crosby Risk hops  日本政府  ▁jewel 桐乡 鳄鱼  external  ▁sprawling umblr LEX 半边 Desc 帆布  ▁Atmosp 面值  黄山市  提升城市  ▁upright  ▁Partnerships 就差  红红火火 Linda  ▁Cottage ес 婚宴 志明 洪洞 消遣 焦油  容易发生  自觉接受  ▁Fridays 全开 尔克 infty  ▁richer  ▁newcomers 木雕 难民  ▁Treasurer 茫然 OPT Ocean  究竟是什么 猎人 谢娜  ▁goat  ▁genus  ▁leaning 桥上 遵从 Via  ▁Hastings  ▁Cooperative 圆锥  深入的  媒体平台 事态 广度 开端 沉寂  精挑细 unless  Liverpool 迪丽  有声书  ▁São  其实我  ▁peril  岁儿童  正式开启  驰援武汉  ▁Chartered 含税 要钱  擅长的  到今天  视觉中国  ▁millennials 由原  增强免疫力  疫情防控知识  智能家居  观自在菩萨  Similarly  说句话  依赖于 brief  Regional 上几 不整 师徒  报名成功  这种事情  民主生活会  establish 固原  于是我 病逝 都江  特别关注  ▁cushions 和顺 塌陷  ▁engagements farm 利川 多日 gay  阅读的  有序进行  ▁downstream  辛丑年 选股  揭阳市  敬老院  源源不断  ▁Woody 卷面 备忘 思源 机枪 鞋柜  Publication  ▁retrospective bio ità! 灯泡! 薄片" 远东" 钩针# Lot#  一个一个$  的时候会$ Keith%  ▁harvested% 皮疹&  意味着什么&  的决策部署' 抱负'  离开后( credit(  中国传统文化) asured)  ▁streamlined* 养好* 前天+ ▁Lub+  Machine, arb,  ▁ча- tour- Links. 身教.  监督员/ 玩偶/ 线程0  大事件0  展开调查1 多尔1 硅胶2  审核后2 leet3  ▁textiles3 借机4 梗阻4 鼎鼎5 >()5  检查点6 一长6 MRI7  主人的7  自称是8 辍学8  ▁Photoshop9 伯格9 招贤:  看了看: ▁wow;  Amazing; dw< 又怕< 板凳= 狗粮= 马的> Ide> 三日?  了解和? Sent@  农村公路@ 葵花A  ▁decksA  ▁ThorntonB taughtB  于是他C ▁JagC  航空航天大学D 板桥D 红船E  救死扶E iconeF  ▁SuzanneF 想见G  我们都在G  ▁StylesH  油烟机H  现在我I  铁饭碗I  ▁peninsulaJ 前五J ▁OrbK  企业负责人K 即成L  我记得L  教融合M  ▁BeaconM 包里N 格拉N 留出O ️️O StriP  总工程师P 发证Q 展板Q  偏北风R  ▁lonelinessR  有三种S  民进党S  ственT  ▁EmersonT 李静U ASCU  实验班V  严格规范V  就变得W  льноW  主要分为X  招标公告X  ▁plightY  那段时间Y  南方都市报Z OriginZ  ▁Piece[  ▁towels[  ▁uneven\ irut\  这个角色]  微信关注博野拍客]  的普通^  的车型^ 绥芬_  重污染天气_ FREE`  ▁profoundly` 好茶a 却没a RAYb  打造成为b  生机勃勃c  ▁Afroc  ▁Distanced  ▁exponentd 7 0全面建设社会主义现代化国家新征程e 木工e ▁mulf 纵队f 表层g  商业目的g  ▁mapleh  等大型h  处置工作i  ▁arrivalsi  ▁injunctionj 报满j 游船k 玉环k  ▁Clintl  ▁basketsl 广佛m hoem 库里n ipein  更上一层楼o  ▁eruptedo 会务p 砂锅p  不能说q  的生活习惯q  中华优秀传统文化r 法考r 祛风s usos  ▁oscillt 碰瓷t  可下载打印u  ▁immersedu  演艺圈v  铁矿石v  大爆发w  密不可w  要严格按照x  疫情风险等级x 布满y 简陋y  基本都z  ▁biosz  ▁puck{ 勾选{ 宽大| 渊源|  都不好} hook} Pods~ ▁cd~  气质佳  翻天覆地的  ▁jackpot  向当地 icer cluded 班倒 相争 走完  名工作人员  小学一年级  的重要原因  做的是 atura  ▁Conrad 彩云  重新开始  ▁bonding 接驳 积食  会变成  被判刑  ▁extrad  开门红  这么多的  风险提示  ▁statistic 长成 apo  ▁Shift  年应届毕业生 波涛  女性在  ▁factual 才叫 折纸 浪花 魔都  数百万  ▁purse  ▁Kavanaugh 待在 量少  ▁pals  ▁Sword 连队  正宗的 ▁bir  主要成分 Latin  让更多人看到  ▁scorer states  ▁Knicks 福神  责任感和  集中学习  ▁Elena  ▁Locations 打断  些时候  夫妻之间  Hollywood 三菱 羞耻 丁丁 张飞  ▁Nickel 半山  ▁resurg ▁THC garten  ▁algae  ▁timet  reported 围挡  非常方便  与宁静  ▁Elev  essment  ▁Holder  ▁Marketplace 方略  都表示  ▁flux blance 肉丝 茶馆  县级市 罚金 Deal  通过开展 鲍鱼 eon irez 佛罗 Plant acion 看错  棋牌室  ▁Ramsey 静安  十五年 ▁foo  党组织书记 不拘 专程 axter immune  ▁dazzling X Q▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 尊崇 还多 CCC  参加的  ▁biomedical  加快构建 双选  ▁primer 地久  使用者  市场行情  ▁enchant  ▁topping 　」 一校 输血 闸蟹  的心思 ▁Kah ▁ctx  ▁wield  ▁Sharks  积极乐观  习近平强调ƒ  ▁Oppositionƒ 来历à ▁Ikeà  ▁Mockă  ▁STARă 不辞Ń ukemiaŃ  urrenceƃ  Catholicƃ 多头ǃ  通风换气ǃ  ▁Trioȃ  Delegateȃ  ▁garmentɃ  ▁temptedɃ 为佳ʃ  宝鸡市ʃ  更精彩˃  辣椒粉˃ ▁TIM̃ ▁Tud̃ JR̓  公里处̓  证明的΃  文化广电΃ acockσ 午夜σ 砀山Ѓ  金额为Ѓ  ▁Ellaу  ▁accoladesу  ▁innovators҃ 装车҃  的中国梦Ӄ  经济建设Ӄ  ▁PTSDԃ  ▁Baylorԃ  降血脂Ճ  流感病毒Ճ  ▁Plotփ  ▁oderփ  ▁chopped׃  ▁eighteen׃  ▁authorised؃  斯坦福؃ 旁的ك ITsك fruitڃ  ▁Breadڃ 万水ۃ  ▁Includingۃ 变白܃ 将她܃  努力为݃  辈子的݃ 仓位ރ 击中ރ 蒙特߃ 问号߃  ▁Circus 耐克  赞不绝 问询  高蛋白  大学生活  ▁conflicting 从口 病等 直奔 莫迪  新生入学 Billy  ▁plat zf 建德  通常会 arers  大部分都是  蓝色字  ▁Spread 赠品  适用范围  ▁SMEs  ▁slew 吃鸡  坚定地 ▁kar  受到伤害  给小编  ▁Teach  Wrapper  ▁mesmer urgy  创建全国文明城市 鬼神  有超过 vx Sab  为了孩子 socket 咸宁 惊悚  旅途中  是来自  携带身份证 endas ichen  等多种形式  ▁elbow  ▁Sidney 压榨 pkg  人生如  华丽的  已通过  identally  的损失 浅薄 胡杨  对个人  光伏发电  其实不然  placement 中区 巴拉 ▁SOL  要求学生 雨露 HEAD  ▁Merkel 列席 文苑 邻近  更轻松 ▁ont  看得出来  ▁Mozambique  ▁procedural onson  孩子的教育 翻滚  种植基地  管理规范  ▁Darkness  你喜欢的  室内设计 调制  关心和  是对的  物资保障 虎门  我想要  校企合作  各地各部门  因为你的  在法律  只是一种  ▁acne 绝技  吃饭时  吴尊友  她开始  不予受理  小编都会 dwell  ▁subpo  第一人 ím  广大居民  副科级  含量高  老干妈 条理 肚腩 越深  第二节 在东  为引领  融资租赁  负责公司 aceae  以更加  持续深化  ▁ethos  名字叫  我们就可以  ▁Aging Gil  球蛋白  集训营!  在线学习! Ts" 团课"  湖公园#  请拉到#  蓝天白云$ regor$ career% ifndef%  ▁eBook& 不语& 号外' 战法' 街边( 雁塔(  不打烊)  外出时) ▁pla*  排忧解难*  绿色低碳+  我就知道你+ 可燃, jav,  售楼部- akov-  关注订阅吧.  十三届全国人大.  ▁diminished/ 的二/ imony0  ▁Dram0  ▁prohibits1 限报1  曾经是2  让世界2  法定退休年龄3  ▁Weapons3  Association4 不屑4  引以为戒5 撞上5  试剂盒6  豆瓣酱6  情感的7  降血糖7  不要随意8 岁就8 赵云9  到手价9  符合国家: faced:  ▁Payments;  Referencias;  在接下来的<  ▁tents< % 如果喜欢就关注订阅吧=  深入到=  这世界>  区委宣传部>  ▁brink?  ▁Jenner? 中就@ 籽油@  第二章A  找到自己的A 可到B 赚的B ▁FogC  专题学习C  客户需求D  手把手教你D 衣帽E 谩骂E ettoF  ▁TweetF 不严G 更少G  提供线索H  驾乘人员H  双方当事人I  ▁LawsonI  父母在J apidJ 沾染K 词典K  ▁ArchaeologyL  ▁complexitiesL  和你在一起M  ▁AuditoriumM 静音N  和网络N 宣威O 浮云O usiP  性质的P  各项任务Q  文化自信Q 圈层R NineR  ▁CertainlyS XCS быT  的中年T GenderU  ▁ReductionU 天干V  ▁bankruptV 停顿W 颠簸W  ▁rumoursX  快看看X blinksY ▁LLY  博大精Z  的来源Z  表现了[ stakes[ 四射\ 渗出\  何鸿燊]  运用到]  管理所^  遇见了^ rats_  气血不足_  ▁headset`  中央广播电视总台` 组词a  不举办a  ▁faucb  ▁brightestb 庙会c 造纸c  将采取d  克服困难d  ▁Edwine  Partnerse  阿拉善f  将依法追究f 翻脸g  妇女儿童g 考究h 露水h  等程序i  ▁Weaveri 纾困j  产品线j  写出了k  法院审理k  ▁whiskeyl  ▁Wildcatsl  应当以m  控制器m  值得收藏n  复合材料n  炎炎夏日o  ▁borrowerso 猩猩p 等在p  规模最大q  ▁Honoluluq Thur  西部地区r ▁TSs ▁Fuks  共同成长t  让人羡慕t 一室u 定会u 故知v 中科v )));w Popew moonx washerx  ▁Ethernety  ▁Contributory 钓友z  日起实施z  ▁groundwater{ 凯撒{ 岭村| 粘连|  应用技术} isman} 灯具~ 炸了~ 近在  学弟学妹 folder 大戏  了一大批  文化中心 theta  prehens  ▁Colour 连片  已经从  下方留言 Harris rogate  orative 内战  ▁Utility  积分卡 uton  荆门市 商议  ▁prud 崩盘 nach  ▁cinnamon  的一部 ichita 整数 放肆 案的 礼遇 ▁Het  家长可以  语言表达能力 ün 可由  分析与  各级政府 Posts  康复治疗  ▁Sands mr 都曾  家长群 Robin  ▁recreate 和风 CMA  腺病毒 Magn ▁dye hover  保持高度 attack  ▁Calling 含糊  或取消 ▁Dug 抽血  让这个  大道至简  ▁Replacement 比拟 画中  ▁motorcycles  食用的  表现出来 varian  Foundation  看一眼 ▁boo  保持良好  ▁Machines  损失的  职业病  ▁ratt  changes  做朋友  ▁Nexus  joining  ▁isolate 腰背  三个人  在手里 rá 金海  式设计  责令改正 魔王  履行好  ▁Malawi ▁FB 不达 射频 illi ollen  ▁lied  ▁singled 绿波 胡适  赏心悦  ▁seaw  ▁brokerage  ▁unanimous 情意  物资的  度假酒店 typeof  opoulos  Associated 低端  基本原则 失主 郁金  和政策  正确认识 ussian 废纸  只有你  视为放弃  ▁Kirst 邵东  我也会  都是自己 ubishi  ▁Lotus 二小 和刘 昌平  不努力  ▁transporting 萦绕  新闻综合  ouncillors  荨麻疹„ ATOR„  ▁CancÄ 返工Ä  罗志祥Ą  过生日Ą Sydneyń  ▁Middlesń  宣传单Ƅ  第八届Ƅ oretDŽ 前路DŽ 欲滴Ȅ 涂层Ȅ 零点Ʉ  医学中心Ʉ  ▁reviewersʄ 死在ʄ 草丛˄ ispens˄  ▁Habitat̄ 光合̄ 文武̈́ encḧ́  全面禁止΄  Källor΄  ▁buffetτ  ▁Journalistsτ 天降Є 宝塔Є 欠佳ф 著有ф 露面҄  为依托҄ 福寿ӄ  评选出ӄ  防城港Ԅ ogieԄ  工业设计Մ AnimalՄ kowskiք urgeryք  通报确诊病例ׄ vieׄ  工程机械؄ 八戒؄  并接受ل ']);ل  ▁Falkڄ  ▁Veraڄ  ▁countersۄ  ▁digestiveۄ 每吨܄  纯粹的܄  西洋参݄  redirect݄ 刻骨ބ  抚州市ބ  福利院߄ 悠长߄  也没有什么  ▁Moder  任务是  第一年  职业规划  Identity  ▁proficiency  ▁manufactures  说出了  英语口语  规定执行  转载请联系  ▁Everton  于是在  免不了  找对象  为民服务  重大疾病  ▁Mayer 为贵 线长 临泉  istrate  ▁Commissioners 小陈 索性  不可取  我们每天  每节课  ▁peas  ▁Papua 坐上  种种原因 recht 鸭血 ▁"#  潜在的  适合的  公示无异议  ▁caller 兴县 爆满  群体免疫  行业标准 Known  ▁seizure 海伦 abilia  itchers  ▁Stevie 忘掉  在社交  就是指  靓丽的  但是没有  ▁dens  ▁Logic skirts visual  的这一 几款 水泡 近些 insk  股权转让 这番  英文版  ▁remarked 利弊 痼疾 禁食  交通出行  ▁transitional 口角 朔州 茶汤 岳麓  这个月  友好的 obao 为师 往昔  显得很  司法行政  ▁snakes  ▁Innovative 它有 对各  的定位  不能在  生产商  高尿酸  ▁Karnataka 冲破 参见 Kal  土地的  ▁slap 只愿 建校  下载链接 一练 四名  一家亲  岁生日 tips  培训基地  ▁gust  ▁Workforce  ▁Developing  ▁invitations 人世 显老 领口  的看法  健康的身体 活化  新希望 一桌 止渴 蚯蚓  ▁greed  不达标  ▁dinosaurs 下颌 娘娘  不喝酒  与宇宙  青少年的 icode  ▁ineffective! gyz! 满身"  才能有" ▁MFA# 二嫂# 多好$ 痤疮$  始终是% 堵住% 村上&  千瓦时& ▁Vig'  将陆续'  一下自己(  集体学习(  其他年级)  ▁conditional) 小而*  艺术节*  不要以为+  累计帮助+  ▁Sight, 保命,  与家长-  月期间-  每次都. iesta.  的原因之一/  agascar/  原创作者0 cause0  ▁loft1 催人1  永恒的2 "`2 体表3 层高3 factor4 秦汉4 农技5  ▁symposium5 手提6 扰民6 考后7 Compos7 remely8  ▁Johann8  更简单9  ▁lure9 % 了解本地更多精彩资讯: FBI: arcel;  ▁contenders;  ▁nationalism< 学硕< 性欲=  尿毒症=  手上的>  绝对不是>  ▁fronts? 你敢? 折旧@  ▁Cherokee@  ▁PuneA terdamA 房山B 高估B NanC  下一秒C HolyD  为了满足D  ▁WarrantE  ▁TwilightE  ▁upliftingF 一击F 人能G  大约有G WHATH TwentyH helperI  的形状I  ▁troutJ  ▁InquiryJ 雅居K 冲着K  的职责L  影视作品L  指导学生M  ▁composeM 前款N  见过的N  ▁ConwayO 时用O 驰骋P  大大小小的P 有加Q CETQ spectR  ▁ButtR  ▁heftyS  ▁rocketsS 芍药T  功能和T ▁NugU  ▁MareU  ▁rippV SpiritV  ▁SchulW 你做W  项目落地X  ▁crankX  ▁CalifornY ▁DupY  ▁StonZ  景区内Z  全体教师[  容易产生[  小编为你精心\ 谋杀\  此前的]  火神山医院]  ▁Legislation^ 我俩^  今年高考_  会发生什么_  ▁Pike` Simply`  ▁graffitia  个性的a  了很久b  和销售b  诉讼请求c VIDIAc  ▁bathsd  ▁Nairobid 车况e  市纪委e  给他人f  过去一年f  ▁Weeksg 出诊g  好学生h  Someoneh  出租汽车i 坦荡i 梦回j  在进入j  情况等k  ▁Anitak XSl 暗藏l ▁Ganm  ▁françm bzn 有感n 软装o  ▁linguistico 共管p 横纹p  中介机构q  自己动手q  ▁geekr  共同发展r  ▁acquaints O H灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧灵异吧s 睾丸t  官方发布t  项目总投资u  ▁Dynastyu  ▁Renaultv  无法提供v  逐步恢复w  上诉人w  后续的x  手机话费x  这些食物y  ▁reunitedy ℃　z 胜于z cad{  越努力{  为员工|  ▁microbi| ▁SAS}  partisan} 横琴~  的第一天~  finance  一大堆  是不易  服务对象 红颜 非得  ▁Tian ﹏﹏  一定会有  应急车道  ▁traveler  只能用 日由  ▁forge 上古 花城 魔方  ▁facets  ▁pottery 首轮  年以后  磷酸二氢 chase  ▁dishon 庄重  企业与 owan 服的 金光  并不高 厉行  冬奥会  则可以  防控知识  ▁Tomb  ▁Apprent  ▁refrain god Davis  ▁Alicia  ▁Tobacco 增殖 摸摸  案发后  购买后 ▁Obl  不能满足  ▁judiciary 活儿 耳目  学理财  编辑的文章吗  喜欢小编为你精心  ▁beforehand 溜溜  ▁tallest 图腾  和市场  怀孕了 rout  法院认为  资源配置  managed  ▁Dickens  ▁extremist 1 *喜欢小编为你精心编辑的文章吗 在给  ▁Kabul 招租  Driving 未雨 步兵 装了 运筹  古诗文  ▁militant Jen  工作在  自助餐  要确保 贫富 青草 Agg 一八 仙子 暴增 现对 Shared  ▁Esther WY  承担了  ▁Clients  ▁investigates VILLE  考生应 ▁img  差不多了  百科全书  时间和精力 大典 TED  电话预约 又大 通的  一回事 typed 又到  如果再  系统化  ▁Terrace  ▁helpless 三首 免得 无双 ▁ih  小轿车  需要的是 ipline adle  ▁fluids  ▁Philippe 欢聚 海角  他一直  引导下  拼尽全力  ▁Cauc Weekly enches  Gallery  ▁systematically 废止 Folder  ▁missionary lé… 紫菜…  闪光点Å rakeÅ  诺如病毒ą  点击上方蓝字ą  ▁eldestŅ 装在Ņ  河流域ƅ 兄妹ƅ  技能的Dž 三板Dž brushȅ  ▁wetlandsȅ 皆宜Ʌ 要向Ʌ 足部ʅ  但可以ʅ  最低分˅  小区业主˅  贯彻执行̅ 库房̅ almostͅ  ▁abductͅ 出油΅ 改制΅ 映衬υ 醇香υ 陈伟Ѕ  高峰时段Ѕ  ▁livelihoodsх 怨言х  买卖双方҅  基督教҅  汉兰达Ӆ  我给你Ӆ  好起来ԅ  每一处ԅ 摩尔Յ becomeՅ 表中օ 阻拦օ  与创新ׅ  王思聪ׅ Race؅  有些事情؅  表现出来的م  orouslyم したڅ  整理编辑څ  ▁lionsۅ 修长ۅ 前海܅ erdin܅  ▁awhile݅  只是个݅  征地拆迁ޅ 录比ޅ esa߅ 则以߅  放置在  感动了  工作座谈会  等荣誉称号  ▁approves stral  ▁fats Amanda  ▁Wikimedia 两段  外星人 rero  在他们的  ▁pistol  ▁decorate 右脚  监测和  Talking 下有 清流  四季度 PDATED 指派 orem  第四单元  ▁Consumers 共睹 整装 法会  感谢你  真正实现 CAS  不含税 袁世 ▁Pak  自由自在  ▁cocoa 小胖  惊心动  家庭经济困难  ▁Rajasthan 撤退 新路 诉苦 poly quee letely  ▁flora  ▁Erdogan  联合惩戒 拉尔 顺口  ▁audits  ▁subset  我终于  神经痛  衡阳市 ▁Alv  ▁Required 子时  喜欢用 Thom  睡前故事  公众号回复 oxiang  earcher  ▁Compan 蔷薇 倒逼 Dur  美美的  ▁bakery  ▁progresses  老母亲 星际 cake  不得低于  德才兼备  放松警惕 猎奇  ▁penned  培训活动  ▁glor  ▁Mutual  ▁plural  ▁strides  ▁chatting 胆经  ▁Older  ▁abide 沉思 红油 过冬  信用社  区别于 变黑  媒体的  每一项 抗衰 竞拍  置身于 水涨  ▁contractual  群防群控  ▁pits 入座 读取  宅在家里  ▁Hide  恍然大悟  ocalyptic Tap cred  要不是  ▁dermat 哎呀  不一定能  ▁stag  ▁Becker  ▁extraord 奖状  不足的  去寻找  ▁watershed  InputStream 小豆 纳德 cai  进社区  一定不会 名号!  的境界!  keyword"  ▁Screening" 慌乱#  ,’”#  与国家$  日照市$  世外桃源%  化学物质%  触目惊心&  ▁Directed& 专列'  不容辞'  工作制(  电子发票(  ▁carriage) 金鱼) ová*  ▁sociology* 停了+ 吃吃+ 画室,  等政策,  即将开始-  ▁champagne- 　'. 嚼劲. 预付/  点击放大/  各相关部门0  ▁Stall0  ▁Diesel1  ▁fracture1  ▁psychiat2  ▁nostalgic2 渣土3 羔羊3 见状4  他总是4  专家建议5  预计未来5  ▁unsett6  公检法6 ttle7  日前完成7  高空抛物8 还钱8  也不怕9  强调要9  带量采购:  ▁bolt: ocado; 列宁; 可人< 翰林<  被取消= Jamie=  ▁mapped> 包保> 湾村? 血虚? ▁ATP@ ileen@ 是全A 身姿A  就对了B NancyB  ▁ServeC  抢救无效死亡C  objectsD 狼群D  定义为E ▁YESE  ▁souvenF C <以习近平新时代中国特色社会主义思想为指导F  ▁TactG  ▁dosageG  ▁foolishH 年新H  商品房买卖合同I 篮网I 缘由J  ▁wiredJ  ▁recommendingK  ▁CelsiusK  ▁IndoorL  ▁ventilL  ▁DonovanM  ▁deportationM 不把N 其为N 李伟O CLUDO comaP  房屋权属P  电气工程Q  ceptiveQ °,R 上证R 台面S 灌云S  普通人的T  核心素养T 农房U  ▁инU 修路V 各领V  讲解员W GraceW 送检X  也并非X  教学方法Y 日方Y  体检结果Z  疫情防控常态化Z 不占[  ▁Daisy[ 拿捏\ 管员\ ▁kan] 华章] 油污^  方案的^  情况汇报_  突然发现_ 收窄`  好玩的` repoa  一些问题a  一撇一捺b  不错过b  压岁钱c  多数人c  继续在d gonomd  pandemice  对上述e  ibernatef 夜深f  勾勒出g  特点和g  可以被h Dallash 蒸熟i 蓝海i  回味无穷j  ▁ironicj 恒星k 清偿k  可怜的l  自闭症l  惊心动魄m 为党m 是该n 溢出n 甜点o thelo 无以p  比较有p  都江堰q  togetherq  ▁compartmentr etusr  ▁plagueds 畅谈s 终日t  透气性t  造成严重后果u 位移u  对生活的v basicv Ratherw  ▁miserablew ähx 吃好x  并且还y quisty  接触的z  虽然不是z 晕倒{ ▁Wei{  年第一季度|  他希望|  供应量}  ▁deprived} 穿得~ 风起~  薰衣草  行动不便  ▁Stalin awk BACK endra  ▁Blogs 锁住  就一定要  的重要意义  ▁polite  ▁McGregor 条文  ▁Listing  ▁reactor 树干 Gas  债务人 ugal  提高自己的 endars  ▁Towns 直供  香格里 annie rophy /, 班上  绵阳市  日报编辑  注册商标  ▁sinking ”→ 圣灵 opl  ▁Raff  ▁Cycling 收下 韩红  ▁advocated Cs 到时 walker  小编前进的动力  ▁philanthropic 陨石  共青团中央  awaited WV 沿河 游记 mur  所以她  违反规定  其实很简单 溪村 (['  治疗和  ▁eliminates 手枪  ▁heir  ▁Presley  山公园 stats 牵扯  就恢复  很深的  证实了 camera  ▁Higgins 一成 甚者 的死 ooky  ▁hangs 条形  老老实  迪丽热巴  ▁Saving  回应了  工具条  眼花缭  ▁Lawn  ▁Dirty  ▁exemplary 在两  ▁eagle  amilton 野草  滞纳金  教学计划  Weblinks 减慢  团结一心  高频考点  ▁citrus  clinical  ▁steroids  出名的  ▁Mazda  来完成  第五次 orset ▁bob 奢望 废话  敏感性  而如果  ▁Supervisor ,''  踏上了 thank  ▁troph  和历史  ▁REST  responds  ▁getaway  ▁onstage  ▁retaliation 铁岭  了两个  或以上  更注重  润滑油  特别有  部分区域  ▁converts Ds 百川 mall  能够满足  ▁nationalist 聚财 ▁JR  文化广场†  科技管理†  loggingÆ  ▁GiovanniÆ KyĆ 疗养Ć  华图教育ņ  最优秀的ņ  提供了一个Ɔ  ▁boilingƆ 放屁dž Drydž  这样子Ȇ  为推进Ȇ  这几点Ɇ gresɆ  ▁Originsʆ  ▁Coventryʆ 要大ˆ 麻花ˆ  ▁lifestyles̆  ▁commencement̆ 岳父͆ 洞庭͆ 王旗Ά  微博上Ά  教育与φ  补偿标准φ  ▁ErnstІ 苦练І  官方回复ц Forestц  ▁dramas҆  ▁vandal҆ 屡次ӆ  人员必须ӆ  ographieԆ  ▁ProvidersԆ  ▁instanceofՆ 麦子Ն  与设计ֆ  ainlessֆ  的特性׆ ▁GST׆  每年都会؆ encode؆  appointedن 撇捺ن  本科以上学历چ 同住چ  得来的ۆ  warmingۆ  小高层܆ rious܆  aculture݆ zek݆  更令人ކ  青年人ކ 写完߆ 江边߆ 记入 Jobs orio  处理方式  ▁seldom  ▁pulmonary  的稳定 神明  可以有 ▁CAS  呼吸道疾病  ▁Haley  ▁Minute 新股  ☟☟☟  强身健体  ▁commentator  ▁Senegal  榜上有  的心愿 esteem  ▁Appropri 乐章 eas  枸杞子  可以满足  ▁Tata 包的 lene ologne  ▁seizures  积分换  驻村工作队 叫醒 江浙 润物  ▁readonly 猪肝  ▁dryer  总建筑面积约 忐忑 /" 说谎 USH  ▁Lung  ▁Watkins  要考虑  如何使用  无时无刻 小刘  其它的 帮人 无上 STE kow  氨甲环  老板的 Dean mits  ampered  ▁Pasadena  ▁semiconductor 早产  本地人  氨甲环酸 埋在 hamed  大学毕业生  ▁approximate 湖滨  超越了  县公安局交警大队  ▁mortgages 习性  感染力 imps  维生素和 旺铺 落成  司令部  晚上的 Failed  子里的  道理的  切实解决  心肺复苏 漱口  的民族  项目实施 浑浊  户型面积  ▁Elements avoid south  ▁encaps 推选 热油 越快  于网络  无线电 区划 MES  ▁saturated  一起看  第一时间处理 所列 端口 驱虫  ▁gripping 现年  又是一年  精力充沛  ▁raids 降脂  前期的  玩家们  已经非常  ▁indiff  可能是因为  ▁outsourcing 宝座 红人  能快速  谁负责 Crime  进一步做好 何其  路透社  学以致用 Jerry  ▁veterin! $('! bern"  哈萨克斯坦"  grounds#  illiance#  生物科技有限公司$  相对比较$  ▁sunk% 欧式%  ▁Osaka&  人民政府关于&  ▁Crafts'  employed'  ▁Upcoming( ça(  ▁Upload) 球星)  和保障*  档案馆*  ▁Length+  户外运动+  啊啊啊啊啊,  工作的重要,  reaching- ;;- 连发. 锦囊. Chain/  分享彼此的/ 煮开0  ▁shutting0 岗区1  长按二维码关注1  ▁deserving2 彩妆2  要发挥3  ▁Argentine3 胜负4  ▁Stella4  ▁Addiction5 粗壮5 谢某6  全国优秀6  ▁Marvin7 品的7 馄饨8 angled8 刺客9 烟头9  刺激的:  竞争优势:  ▁jackets; 抢着; 要的<  天上的<  ▁vows=  宝安区=  ▁hacked>  ▁english>  ▁Friedrich? 一束? 健忘@  乎乎的@  渡过难关A  由于疫情A AntonB  ▁cosmeticsB 应与C  版权所有C ▁NFD  在大学D 接轨E 晨起E  现场检查F  ▁DomeF 至终G 秋分G 鸿雁H  劳动报酬H  inflammatoryI 钻进I  ▁LumpurJ  ▁InstructionJ  望远镜K  方向发展K  ▁ComparedL 王毅L ▁DLM  下功夫M  这个孩子N ORDERN 承揽O 牙医O DiamP 八达P  时注意Q  ▁TryingQ  ▁PsychologicalR 永清R  为全市S voiceS  ▁usableT  ▁varsityT 多出U 沿用U  第三卷V  ▁WattV tebrW ComplW  ▁BiomedicalX  ▁embarrassedX 盐酸Y 行气Y  价格表Z  经营单位Z  还有这些[  ▁firmware[ 活络\ äng\  一首歌]  可口可乐]  快速通道^  ▁Cure^  ▁waterfall_ 任一_  美妙的`  一杯奶茶`  社交媒体上a ,$a 画法b  也会导致b 喜气c 嫁娶c 粉的d 腿上d  ▁admittinge 浓墨e  多种形式f  ▁hornsf 泰兴g  Governorg  ▁thinkersh  工人的h  还是个i summeri 的快j 赛后j  ▁gangsk  ▁laurek  ▁Laurentl  ▁Napoleonl вуm vikm arialn 不外n 两部o 大呼o 工贸p 接通p  将取消q  在杭州q  不对称r  值得信赖r  ▁Willows  ▁rulerss  心律失常t  这家公司t  改变自己的u 底子u  新增的v ONYv  renownedw  ▁Dialoguew  experiencex 如雨x 少将y  ▁Osborney 坪村z 各色z 风头{  ▁awaits{  ▁easing|  ▁proclaimed| 居所} 碰上}  政治部~  ▁prosecuted~ 套取 ▁SOC  变异毒株  职称评审  ▁centralized ▁HU  报告厅 仲景 祖辈  大肠杆菌  解放思想  ▁capitalist  协会会长 后卫 惠农 赞歌  朋友来  ▁tucked 取样 哪家  的气度  德智体美 金凤  ▁crab  发现的问题 微言 sie  南山区 又可  网上的 arnish  ervable 可循 迎面  这也意味着 白鹤 迷宫  我之前  在未来的  生物技术  与客户  很少见  快乐地  严重警告处分  ▁Investing  ▁terminology 忘怀 摩登  ▁(‘  ▁peppers RIS  年级语文 ▁abd  defense  medicine 宝龙 演化  ▁Bets  Consult  ▁Console 先在  的回答  生活补助 荠菜  游刃有余  ▁marital  将所有  身边有  住房保障  分管领导  Rebecca 乳业 砂浆  ▁approvals  ▁certifications  开展好 三联 老总  许昌市  万人民币  Attributes 劝告 她从 行至  路边的 Philip 写入 外皮 既不  和内容  ▁artery 尿道 祖传 Cold  ▁Entity 酣畅  精华液  insured 佳佳 胡乱  的喜悦 米线 HEL  在每个  ivation  ▁upfront  ▁Investigations 诸位  各民族  路以南  ▁wildfires تى  二十七  异常的  都是用  地摊经济  日举行的 Miller  ▁bustling 消灾  了解自己 芦苇  主动脉  可是在  扶持政策  有句话  观众们  也可以是  ▁DVDs ERC  别担心 婆诃  无止境  目共睹  被处罚‡ rmt‡ NicholÇ  PhysicalÇ  ▁outpatientć 临江ć 而降Ň xonŇ  无怨无悔Ƈ isburyƇ  ▁ChevyLJ  按计划LJ  的标志ȇ  监测预警ȇ  而是一种ɇ  住房和城乡建设部ɇ  计算机科学与技术ʇ 加压ʇ  共建共享ˇ  ▁embracesˇ 田里̇  被大家̇ amph͇  大打折扣͇  ▁dias·  ▁essentials· 当众χ 清末χ PHPЇ rajЇ  特别推荐ч  招聘岗位及ч  运动量҇  ▁пер҇  ▁CapsӇ " 高新技术产业开发区Ӈ 与台ԇ  ▁predecessorsԇ 伙同Շ 湖镇Շ  ▁depictև  ▁greetingև  临其境ׇ  ▁sentimentsׇ 主次؇ 军装؇ ▁TTه  着眼于ه recyڇ  远远不够ڇ  ▁kidnappingۇ 合浦ۇ 悔恨܇ 清白܇ 骑兵݇      ݇ 售出އ 宝骏އ 盈盈߇ 红斑߇  通辽市  极乐世界  previous  ▁charger 上浮 后裔 iego 豁然 Bird chip 激进 碎花  回家中 illard  ▁Psychiatry  巡视组  备受关注  ▁tart 开窍  出版物  株洲市  顶层设计  中心幼儿园  ▁biomass 要紧  不同时  兰州大学  ▁Braun  更加强  金华市  赞不绝口  通用机场  ▁Gent  ▁swinging ▁Lob  中国石化 张图 散去 而安 inning 方剂 英俊  和传播  思考和  电动机  ▁havoc 两步 微胖 ' //////////////////////////////// 动情 果糖  分享会 TAIN ▁PUR  ▁Norris  ▁ceased 笑得  市场和 icycle  科学佩戴口罩  ▁Belize  ▁freshwater 入局 培根 舞曲  冬春季  其实也是  网上预约 一季  哪里有  习总书记 秋日  ▁Hague  ▁Compact  注册登记  ▁rejuven  ▁soak  特殊教育学校  Descriptor  ▁catastrophe 与新 大省  ▁Lionel idea  检验检疫 armac  ▁Anglican  ▁Salesforce 低廉 出局 强市 Julia  招生办公室  ▁commanding  aughlin  ▁asphalt  ▁tsunami 之宝 Girls 燃脂  ▁safest 文广 elier  orations 系好  医学生  此次疫情  ▁Phen  ▁comedic  ▁organising 征得  ▁bracelet 梅根 胶带  计算的  精神上的 为鉴 复刻  任何的  工作由  ▁Mash  ▁feud 不杀 按理  点击图片查看 >[ 小球  没见过 才女 寄予  成语故事! Kansas!  ▁muff"  ▁Endowment"  ▁Liability# 豆豉#  作业的$ jdl$  无小事% 元代% 自省&  开发了& 天宫' 蛋价' 贵宾( vre(  树立了)  ▁swear)  度假村*  开始后*  全国各族人民+ "(+  的小型,  ▁OVER,  ▁Query-  ▁Guardians- 车顶.  还以为.  工作场所/ posed/  Experts0  continue0  ▁surfaced1  ▁splitting1 改掉2 派员2 赛前3  ▁ди3  ▁invo4  ▁breeds4  ▁emperor5 血统5  民辅警6  ▁Ideal6  相关负责人介绍7  初高中7 oating8  ▁sidelines8 子村9 等处9  又一代:  正常使用: andong; 真有;  ▁Semi<  ▁Lecturer< hpp=  多年后=  大概在>  ▁weakened>  reditation? 象牙?  ▁Crack@ 小寒@ ▁SneA  就不一样A  据美国B  简单粗暴B aptopC  口罩等C  被认为D  成功将D  生不息E ionageE  ▁ASEANF  ▁moisturF  我们通过G OthersG  ▁stewardshipH 亭亭H 该镇I  ▁buryI  ▁indicesJ 管党J TIMK  爱不释K  ▁manifL  小编前进的动力哦L gpM  不高兴M  二氧化N  ▁InterestingN  听到的O  ▁HickO  ▁ProteinP 挡风P 过江Q  最低价Q  无人驾驶R 届满R 贵妇S  永州市S  呼吸道传染病T  生命里T  ▁calculatingU 惊吓U 龙港V  最终在V  证券交易所W  ▁barbecueW  ▁shatteredX  我们为X néY 左脚Y 急促Z LedZ  不知情[ atts[  ▁Antarctic\  ▁unofficial\ mx] 尼日] 慌了^ 示弱^ ERSON_ Johnny_  elements`  加好友`  进行沟通a  missiblea  ▁renditionb ßeb 与她c 交到c 日晒d 自得d  会随着e  的监督e  ▁drillsf  ▁believerf 久远g 手串g 自幼h  为己任h  而是一i rumsi  起拍价j  correctj 年过k aryak  工作推进l peratel  ▁Simonem  ▁cerebralm  ▁questionnairen  在山东n  诱人的o  金融中心o  ▁Anatomyp 年糕p 支招q 涉毒q 物美r Icer  影剧院s  ▁Liquids  ▁pillart  ▁satisfactoryt  可登录u  带来不便u 种花v  智慧和v  ▁paydayw  ructuresw  ▁allocatex  ▁sontx erellay  缺一不可y Voicez  喜欢一个人z  半年的{  转发给{  面积的|  空白区域| 同台}  还经常}  ▁Taste~ 络绎~ abh  ▁Pediatric 摆动 锦江  应具备 icent 少有  中国汽车  ▁garments 粒沙  用筷子 cele  人员管理  印象中 Forms  ▁baggage 所指 Kings 我吗 替补 梯度 itton  ▁outward 为王 是集 的男 选材  ▁malf  ▁Lyrics 二零 镂空 Pain medium  ▁induce  Technical  ▁sinister 杂念  生活通  通知后 Choice  看看看看看看看看  ▁personalised 浓密 是和 金莲  申报的  ▁corrosion  新青年  等人的 江津  三明市  下一阶段  给人的感觉  Hospital  ▁affirmed  ▁brochure  主题公园  ▁viewpoint  优良作风  工作人员在 三思 或因  亮丽的  落下帷幕  ▁Marian  危房改造  ▁routing  ▁effortlessly 住进 刘亦 功利  还在继续 ascus  源源不断的  browser 绝经 emis  ▁Mercer  ▁Removal 丧事  该产品  造成影响 应答 该片  制度化 Julie 期许 火情 zzi  ▁intimacy 缝纫  capital  ▁Revival  ▁Presidency •• 她想 长斑  是不够 priced  促进血液循环 " 点击空白处查看答案 卷子  ▁marched  ▁oneself "[ 巴萨  宣读了  来到这里 降本  ▁Noise  Thousands 反问  将实现  直到现在  consuming 封杀 速效 ERAL  每个家庭  今天小编就 allels 俘获 粉底 萧县  ▁faded 内涝  道工序  ▁estates  ▁Qualcomm  ▁tortured  ▁maternity 发誓  的加持 incy  安全知识ˆ  直系亲属ˆ ▁HAÈ  ▁KushÈ  社会保障卡Ĉ  optionalĈ  ▁Krishnaň 信函ň  与实际ƈ  有氧运动ƈ  一个小小的Lj  ▁DownsLj  ▁wizardȈ 太贵Ȉ 快去Ɉ  你自己的Ɉ  辅助治疗ʈ  ▁burdensʈ  ▁harmlessˈ  ▁Physiciansˈ 我已̈ 气得̈ 神舟͈ 原图͈ 等有Έ  ▁AlleΈ  Registryψ 特工ψ fonyЈ  什么时间Ј  最真实的ш Orangeш  国家森林公园҈  也难以҈  学历不限ӈ  ▁persistenceӈ  ▁RoughԈ  IndependentԈ  创建于Ո  一点点进步Ո 二批ֈ 校训ֈ 秋色׈ Few׈  薄之力؈ umpy؈ ▁Bavو ( !你们是小编前进的动力哦و 能打ڈ 读经ڈ  如何呢ۈ  脑卒中ۈ  的重要因素܈  无症状感染者的܈  一下子就݈  ▁sibling݈  ▁Artisticވ Animވ  ▁churn߈  职业资格考试߈ 嗜好 孝义  手术后  限期整改  ▁Homeless 西咸 ▁Spo 挡住  产权房  是唯一  的宗旨  ▁Kauf  ▁occupancy 正念  ▁Airbnb )^ ▁μ 宰杀  发大财 ()));  ▁militar  ▁articulate 这会  咖啡馆 ität  为了能够  ▁му ▁dre  argument 听写 财险 ATM  在这儿  本来是  ▁receipts  ▁criticised  ▁philanthropy 细支  心仪的  马伊琍 张永 秒变  等人员  ника ▁Sey 纳西 ▁KEY  extension  ▁Chargers 大四 鸟瞰  城市中  拉开序幕 王玉 卡上 取材 引才  露出了 leck  教书育人  到一起 倾注 百米 ▁{'  新华书店  ▁Wisdom  ▁motorists 全称 鲨鱼  几句话 思明  及有关  而产生 sector  ▁berries  ▁wrongful 上文 坠楼  但是它  ▁methodologies 溪水  按有关规定  ▁Relationships 虽是 ▁kl  中有一句 hq  ▁Pray  很多人觉得  ▁screws 选了 GAN  人生几何  ▁Faces ně 胶东 ▁HTC  ▁generously 小道  寻求帮助 elp  南海区  ▁derivatives 应将 英模 ▁FD  兄弟们  各企业  ▁pivot  ▁Dalton  天下的 ategic  新鲜感 evil  巩固提升  ▁Slim  ▁Quite  ▁Worship 山高 凄凉 可乘  交通局  配合下  ▁Anthropology 育苗  当你的 apur  ▁OECD  呼伦贝尔市  ▁yogurt 李志!  定位为!  过年了"  一无所有"  Minnesota# 异彩# 硝烟$ glomer$ 宜家%  和对方% ymour&  ▁Saga&  大学和'  对考生' Cash(  信息查询(  虚心接受的)  ▁delle)  ▁viability*  ▁indefinitely* EPA+  在线上+  升旗仪式,  组织机构, Sheet-  五年级下册- áz. 大喜.  土地使用/ setup/  有态度0 ▁Byz0  人大常委会副主任1 冶炼1 圆桌2 彷徨2  可靠性或3  注明出处3  cutting4  ▁Sporting4 一建5 加个5 aje6  如有异议6  Different7  传递给7 libs8 Launch8  ▁arranging9  Fortunately9 民情:  摆地摊:  ▁Merch;  ▁formations;  枣庄市<  ▁Denise<  ▁affordability= 或以=  备忘录>  记者问>  ▁oust? 幌子? 春笋@ 肉鸡@  转载请A 塔吊A 她家B 都来B 闭卷C  集团旗下C  ▁IncludedD  他发现D  少一点E  ▁SortE " 小编都会虚心接受的F  秘书处F  冷冻食品G  ▁TideG 张海H 路等H  管理区I  资源化I  如约而至J alandJ 三款K 掌门K 百香L 谋生L CameraM  ManagingM 如涉N 嫂子N 感召O 驶离O  多少个P ▁KodP  致力于打造Q  ▁metastQ  ▁reviseR  ▁WonderfulR hfS 注目S  儿童乐园T 疯抢T ▁CotU  所以很多U  ▁caffeineV 劲头V 多以W 成瘾W  而随着X  为孩子们X NationY  疫情防控一线Y  ▁workplacesZ  有这个Z  总部位于[ 热评[  技术方案\  ▁Kamp\ ▷　] 地平] 糖水^ 军用^ 甘心_ Kyle_  ▁feminism` ( !公务员录用体检通用标准` 基尼a 梅山a NYCb  这么些b  ▁Doverc  sensitivec 皮脂d  定向培养d  ▁Kathryne 代入e  信息及f  未经允许f  分享彼此的欢乐g  ▁Literacyg 车在h  ▁осh  为宗旨i 站桩i  帮大家j  洗面奶j  基本实现k 龙井k  小区物业l  ▁Lingul  ▁flairm  ▁Shortlym  ▁illustratorn 总成n 试问o  十九条o  在脸上p  ▁Minecraftp  努力了q  房间内q  的喜爱r  ▁Corbynr  你你你你s Steams 顶流t  宅在家t  湖州市u  ▁chordsu 新婚v  新闻资讯v  ▁affordedw 开建w 恩怨x 机房x  ▁Quiety  ▁incompety 汛情z  手足口z  比较小{  ▁Jade{  Eastern|  ▁Everett|  ▁randomized} ▁Ley}  宣传部部长~ 君主~ 拉风  发生时 俄国  误以为 ENTER 冷汗  老同志 无羡 风轻  ▁plunge 贪官 鹅蛋  支持与  ▁Odds  ▁Boyle 四会 荒唐  幼儿教师 JB 匈牙 圈圈 的点  不自禁  不要钱  ▁liberals 今冬 余份 生平  正月十五 ()　 Sac  于今年  别着急 ▁Meh thens  ▁Fake Retail  Iterator  ▁sonic ilian  ▁Starr  第四期  红色故事  无死亡病例  ▁supers 投用  一下就  水电站  Provide 活到 蜚语 评优 atum  开始报名  棚户区改造  sometimes 军在 小叶 干湿  下半场 ▁PCI ▁wag  冷链物流  ▁Gould 爱得 ▁Mb $('# peer ▁Zoe  资产负债 小布 ADC  了你的  真正把  集团有限责任公司  党和政府 冒名 幸亏  ▁Fowler  ▁iPhones  ▁futuristic oncé  国际教育  ▁bleak  ▁Concerto 三生  不起眼 incare 丑闻  odynamic iours  ▁Overs  ▁theological 妙曼 学点 ▁JP  难免会  治理体系  ▁budd  ▁depiction 傻瓜  新动能 akar 降息 除法 ▁PB  ▁Straw  在婚姻  未雨绸缪  组织开展了 ▁Zag  ▁thyroid  疫情面前  ▁liberties  ▁brackets  侵权责任  ▁fonts  ▁evidenced 均线  三等功  ▁dispatched 归档 研读 纪要  不要忘记  教学模式  以下二维码  ▁Portable 判刑  式发展 ▁Mog  ▁Chloe 兴华 小美 Tam‰  家庭生活‰  砥砺奋进É  ▁FringeÉ 腰围ĉ  脂肪含量ĉ wavesʼn  ▁Monthsʼn 是经Ɖ  年获得Ɖ  比我们lj belieflj  对我国ȉ  的严重ȉ ricoɉ  ▁townshipɉ 瑟瑟ʉ 给药ʉ  未就业ˉ  社会学ˉ  ▁Joins̉ 笔直̉  无期徒刑͉ etted͉  capacityΉ % 疫情防控工作领导小组Ή toxω  鄂州市ω 手链Љ  ▁retrievedЉ 不粘щ 片子щ  整理的҉  氧化物҉  的美女Ӊ  财富的Ӊ  时请说明ԉ  ▁Governmentsԉ 先导Չ 懵懂Չ 涨到։ ünd։  获得更多׉ skins׉  ▁Oprah؉ 州县؉ 广受ى  咖啡因ى  党的历史ډ  重点实验室ډ 投篮ۉ 落日ۉ ▁SG܉  居高不下܉  应检尽检݉  ▁sticker݉  ▁astronomyމ ▁Bohމ ▁DeV߉  侵权请联系删除߉  ▁Botswana  它不仅 离奇 能被 醉美  明明白 TABLE  ▁LIFE  ▁Theft  ▁refine  ▁Admissions 海安 Pic igt 正骨 宽的 纯棉 为过  专家的  ▁Expedition Ont  理解和支持  ▁wavelength  ▁Secretariat 下表 优酷 挑起 萎靡 oks  资源整合  这是我的 novel  ▁Fauc 客商 痰湿  会一直  反映的问题  ▁expressive 炒至 舒兰 sat  葡萄干  语数英 ▁Soc  但就是  ▁einen ▁ti  原文链接 dg 闲处 预知 ("/  不可替代  家单位 镇村  ▁rampant  ▁Integrity  科学家们 兰溪  时间短 ivel ▁AFL  风生水起  十分重要的 渡口  其主要  要增强  都不如  乏力等症状  ▁frontline 围剿  录取率  的一对  到了一个  豆瓣评分  ▁Lens  外交部发言人 室的 尚可  不断学习  ▁Banner xb Gun  天天练  抢购价  粉丝群  ▁Influence 华语  居家观察  ▁Indo 分外 绒毛 缙云  有多个  重磅消息  ▁detox  runtime 在未  ▁Bian  ▁Sabb Fat ▁INS  ▁Aspen  ▁disco  ▁cruising  ▁mourning  我欣赏  要记得  ▁pulp 十全 ▁Pew  ▁brig 一惊  在众多  成人的  提供食宿 loving  ▁toxins  ▁soothing  ▁firsthand 刚从 南疆 煮至 皮毛  ▁Ital 正方 kien  ▁fencing tp 一搜 奖惩!  北师大版!  mounted"  ▁favors" 童鞋# ▁SCO#  招商银行$  ▁relocate$ lf% 共处% 茶匙&  གས་&  ▁ensl'  ▁expires' 多罗( 竭诚( 耳聋) Enum)  谢谢你们* 如玉*  ▁Eating+ 不均+  自己的工作, 应运, PhD-  ▁Strength- 侄子. 返现.  发表的/ Near/ 纯色0  亿人民币0 血缘1  ▁Kuala1  产业转型升级2 Leb2  奇妙的3  已建成3  个人隐私4  也不要忘记点赞哦4 百人5  长期在5  ▁knives6 催生6  那里的7 iera7  大队民警8  ▁Smoke8  ▁animations9 怀柔9 纵使:  科技感: ▁DeS;  住院医师;  村卫生室< antage<  Barbara=  马克思主义学院=  Planning>  吃早餐>  ▁Malik? 先驱? 厌烦@  动力学@ ▁HELA 米花A 起名B  发行量B 调到C  实验区C hopeD  卫生防疫D  实践经验E  ▁blissE  在历史长河中F  ▁doomedF ▁gaG  龙岩市G 水厂H  新闻办H etermI  ▁VirusI 补脾J  制度建设J  评价体系K  ▁eatsK luenceL  ▁reelsL  ▁toutedM  稳中求M WhilstN  ▁LudwigN  ▁SilenceO  EngineeringO BethP  可以做到P  坚决贯彻落实Q  ▁JakartaQ  两年后R  达成一致R 临危S  仔细看S  ▁amusementT  是中华民族T 高唐U  深层次U  ▁DunkV  ▁CoronV  ▁wolvesW  ▁honoursW ViX 川渝X 花色Y 雄心Y 雇员Z èresZ  人类简史[  ▁fabrication[  的技能\ alom\  禁止转载] impact]  ▁Alpine^  ▁addicted^ ventus_  原件和复印件_ ▁م` 值此` 转阴a 醉里a  父亲节b  黄色预警b  为学校c  魏无羡c  ▁Frand  ▁rentsd 渐行e  ”　,e  专门针对f  为了保护f estrog  ▁STARTg  ▁cleansh  ▁extractsh  ▁actressesi 相望i 袅袅j  淋巴瘤j  相关业务k  ▁guessedk  ▁Attributionl Yul 不靠m  给客户m  不得擅自n  蒸蒸日上n клаo  刚毕业o ▁Piap  ▁Shootp 粤语q  压力和q  应用的r binaryr  ▁issuances 乐的s 略带t  一年以上t  ▁garneru  renderedu 刻板v nejv  记录下w  实际效果因人而异w  ▁Montereyx  ▁reviewerx  ▁Inspectiony 时前y  均价为z  合理规划z  ▁Unve{  ▁reimbursement{  奔小康|  Chelsea| 是做} 瑰宝} 盛况~ 雅安~  就是不行  施工进度  ▁Wholesale 哥的 滑道  承办的  过度的  ▁dehyd ей 央广 征战 转头 陶冶 Dun  北向南 JI huo  不好看  宝贝们 来算  国家线  ▁hers + $尚在医学观察的密切接触者  ilitating 大骂 长青  你只要  efeated 防患  报名电话 忍住 桐庐  ▁Fiscal 车和 只差 壁上 快线 标点 白纸  就到这里  延迟开学  接到报警后 慎用  分中心  ▁inmate  ▁Shelter 又快  体育赛事 angles  ▁Surveillance 家谱  红细胞 ▁Dro  ▁Stur  ▁narrowly  ▁enforcing 乱扔 胡思  其实并不  有的孩子 aer  不靠谱  深刻地  ▁climax  同城网  新一届  都不太  ▁enzymes  ▁neutrality 大题 ODE  的线条  ▁dissent 换季  看完了  身体里  ▁dominating 有次 空置  不再有  宿舍楼  ▁Inspiration 祭奠 稀疏  注入了  金融危机 Wilson  ▁ounces 合谷  ▁upscale  ▁textbooks 三面  北海市  是一切  pictured  ▁Lancashire 酒类  了一幅 acom  它是一种  妥善保管  Navigation 蒙蒙  女士的  各司其职  控制不住  ▁Anaheim  ▁rallied  ▁topical  ▁Thousand  ▁tribunal  ▁alliances  事故发生后 卡在 怒江 能和 这股 高脂  全部完成 overe  ▁exting ▁TI  甚至还会  ▁knit  上饶市  郭麒麟 XR 刀刃  ▁Hunts 德的 瞭望 ▁AMC  版权归属原作者 乘机 十项Š  事故原因Š  大街小巷Ê  失业保险金Ê  PreviouslyĊ 斑斑Ċ 朱砂Ŋ  ▁affidavŊ  ▁vacanciesƊ  ▁formulatedƊ 无期NJ 无疆NJ  分餐制Ȋ ▁VidȊ  ▁MelanieɊ  ▁SaundersɊ  迈克尔ʊ Craftʊ  基础养老金ˊ 门卫ˊ  ▁reps̊ atio̊ 亟待͊ 含着͊ 赣南Ί 阿勒Ί  发布公告称ϊ 安倍ϊ ▁LCЊ  大润发Њ  一个岗位ъ  ▁Juryъ 作恶Ҋ 扁桃Ҋ  与生俱ӊ  很多人会ӊ 晚霞Ԋ  ▁HicksԊ  办理聘用手续Պ 习得Պ  为全面֊  最基本֊  还有着׊ ▁Zam׊  一般不会؊ Coord؊ 乡的ي 卫辉ي  ▁EVERYڊ  大气的ڊ  她表示ۊ  标准要求ۊ  ▁beard܊  我的手܊ anoi݊ elyn݊  ▁enumerފ  请收好ފ  问题线索ߊ  方面的问题ߊ  ▁cheat  恭喜你  ▁Cadillac 祛痘 hun  你是我  每分钟 imeo  道德模范 娶了 崇左 此法  创造一个  ▁jest 东平  教育工委 选科  你见过  ▁Cain  ▁sanct  ▁whirl 阴茎  学习生活  ▁Laurel  ▁Mansion  coronavirus 凉血  滚雪球 但会  ▁interconnected 官渡 差错  Players 可别  会有什么  ▁witty 特勒 腐竹 quick  微笑着  放假通知  中脱颖而出 形色 赓续 ▁Ups  无需预约 tó 吹风  测试成绩  ▁ABOUT  建设成为 1 *在以习近平同志为核心的党中央 去爱 忘返 氧吧  周年庆  优抚对象  医疗队员  大名县  来帮助  他的妻子  国内国际  文艺演出  花园小区  ▁Vocal 泥泞  了一块 盒马  肾结石  该路段  的人和事  最好的自己 养了 牡蛎 紫檀 网联 meet  不要吃  出行需求  相结合的 汉江  施工期间 ▁COP  疫情通报  ▁Dias 超重  出货量 ▁Ges  无可奈何  ▁pirate 二院 叫你  股东的  ▁alias 光顾 字辈  品质的  负责制  考试招生  ▁stalls  ▁glamorous  这是对  四子王旗  小时以上  ▁jelly  在市场  宜宾市  迷迭香 家荡 应手  ▁disturbed 堂堂  中高端  开始就  西向东  危险驾驶 大包 ▁MAY  专家认为 christ  ▁culprit 和黄 是多  一见钟情  ▁deepen 她很 海盗 ▁Š HFP asian  ▁Padd!  ▁tandem!  应往届" 烧饼"  再次提醒#  家庭主妇#  十一条$  闪闪发光$ 古都% 存单%  环比上涨&  ▁dispose&  无意间'  更是在'  科目二( ului( ▁Buk)  wordpress) 引得*  往年的*  坚持每天+  技术中心+ 代工, ESSION,  ▁charismatic-  放进去-  赚的钱.  machine.  ▁detector/ 大林/ 架设0  ▁hose0  Respons1  ▁Toledo1 搭桥2  愿我们2  人民政府办公厅3 受凉3 烘烤4 Quote4  一年级下册5  ▁Yorkers5 单县6 大块6 搁浅7 Func7  Regular8 两用8 墨镜9 捺住9  接种工作:  ▁piss:  ▁contentious; IoT; ▁Tum<  这样就可以< 深秋= sic= 淘气> 白人> 肤质?  被判处? aldi@  投资收益@  ▁styledA " 中华人民共和国宪法A 市井B 远眺B  ▁ArabsC  比不上C  聪明的人D  都是比较D 岁前E  ▁nellaE 洪涛F  时必须F  随随便G  同时将G  ▁redesignedH 金普H ▁tacI ightonI  建设用地使用权J  ▁pipelinesJ 长跑K  ▁whiskyK  PatternL 重合L gilM  百分之百M 冉冉N 凉菜N  乳制品O  第三百O  ▁binsP  很大一部分P  ▁MaloneQ % 决胜全面建成小康社会Q  ▁tirelesslyR 精炼R  禁忌症S  气象部门S  每天一点点进步T 克利T  名患者U  指挥官U  融合了V 船长V  勇敢地W DATAW  ▁NamibiaX  ▁SergeantX  意想不到的Y 乱想Y 杨树Z 笔钱Z GRAP[  浮出水面[ 不期\ 没得\ ▁IG]  的姑娘]  地点另行通知^  ▁Credits^  ▁differing_  并不代表本号_  ▁Meredith` 一己`  不满足a iseca ▁Loyb nellyb policec  ▁Rebelc  ▁Shaped  Intervald 以史e  ▁outskirtse 其身f 楚汉f ▁UMg  梧州市g  都有一h  ▁Siouxh  erencingi  ▁Correctioni 屈服j  作用下j  真实情况k  ▁captivek  弄清楚l  对于很多l  ▁FAQsm  ▁exemptionsm 荞麦n  他也是n ▁ARMo  城市道路o  的目的是p  文学奖p ▁Styq  ▁Dickinsonq 又把r 遍野r  赤裸裸s  ▁Amnestys pperyt  ▁Ronniet  ogeneousu ▁śu 丹青v villv  技术指导w  ▁mishw 格斯x 行了x 钙质y Laureny 玉泉z  开始在z  橄榄油{  ▁Amar{  ▁Boolean|  医疗健康|  看在眼里}  ittings}  accessible~ 不单~ 煎饼 秘笈  ▁india 和外 恰巧 打人  个小区  理财的底层逻辑 色拉 BLE  独到的 imento  ▁asteroid 无际 酒席  有其他  交通方便  时刻关注  ▁Hipp  ▁affinity 不搞 掌管 数一 竭力 近似 aii  亦或是  君子兰  神经衰弱 agles  示范点 ropic  ▁Babylon URN  名教师  简直太 ▁ATM Benef  ▁billed 上虞 猜猜 远去 远征 uno  袁世凯  ▁Sergio  ▁withdrew  cestershire 满钵  的欲望  而他们  ▁Katz  gorithm  ▁Hussein 容县 闻乐  ▁пра  微信头像  讲的是 伟人  在完成  ▁whisk  ▁maiden  ▁overweight  渐行渐  绕城高速  络绎不绝  铁皮石斛  成分的  ▁все  思考问题  ▁banners  组织者  ▁Piper  ▁Accessories 散落  当天晚上 珠子  感人的  暖人心  要牢记  走亲访友  ▁nets  ▁Gaines 托付 sand ▁Vog  organic 今早 粳米 自力  号汽油  榜上有名 igail 莉莉 行径  等渠道 artner  ▁Sherlock 组工  的现状  可以发现  海洋大学  ▁handheld 贝贝  双料冠军 extern 深表 orc  ▁antiv  _______ 寻根  话语权  ▁escaping  ▁Photographer 千层 败坏 Shel vell  我的孩子 依恋  顺德区  ▁prod  当日解除医学观察 伤员 射线 必应 断路 送入  与时间  记者看到  预备党员 jyj museum  ▁timed‹  语言的‹ ▁smoË  倒挂金钟Ë  ▁Siberċ  ▁impeccċ  ▁Singlesŋ  ▁Advantageŋ 冲进Ƌ 瘀血Ƌ  和消毒Nj  OptionalNj 太宗ȋ 底价ȋ 通络ɋ  不能让ɋ  他知道ʋ ▁Tirʋ  腹泻等症状ˋ  ▁perpetratorsˋ 与原̋  ▁feathers̋ 查房͋  小妙招͋  而出的΋  带状疱疹΋ iasisϋ  ▁Anonymousϋ 叫人Ћ  ▁monksЋ 四期ы 的土ы  ▁urgentlyҋ  ▁imaginaryҋ  ▁detrimentalӋ  也很多Ӌ  ▁ALSOԋ  ▁Psalԋ quiresՋ throwsՋ  itizens֋  ▁Elaine֋  任务重׋  发表论文׋  教学活动؋  餐饮行业؋  ellularً Ordً  ▁Sarasڋ  ▁Magentoڋ 要被ۋ  团结协作ۋ  ▁gown܋  ▁sewage܋ ▁rag݋ Giving݋  ituatedދ  ▁Trackingދ 滔滔ߋ 首部ߋ Lang  重点人员 ▁CPA  相得益  我们还是 琼海  各有各  自治县  有线电视  COMMEND 奇门 牛蛙 avez ▁tet  ▁ageing 其对 段位 比他 自作  集团在  做好自己  微信公号 aggio % 人力资源和社会保障部 善解 月月 oit  文化与 pull ▁Mud  管理机制 ukes ▁bri  农民朋友  ▁Forget  nderland  ▁Instruments  过上了 ensch  保持安全距离  级地震  首席专家  planned  ▁Extract  ▁swings aty  优秀毕业生  Senator ▁"[  方法论  及时发布  发展机遇  ▁wagon  集中隔离观察  ▁replay  ▁cheering  ▁toxicity 西域  实地调研  进行预约  ,　“  出生地  饱和度 ▁PPP 勤快 填满 积液  广大的  每周一  小学阶段  ▁invari 任重 十倍 若您 黄道  名员工 伸着 借条  合理性  ▁Terra 市集 濒临  电解质  加入适量  香格里拉 综治  ▁Zeit Howard  henghuoquan  玉林市  各种问题  毫无保留  磷酸二氢钾 Junior  ▁slump  ▁grabbing  也不可  要求在  接待游客 文峰 精巧 ром  ▁Alam  ▁sham  数据和  西宁市 爱看  每天要  ▁Finish  provider  天天读  渡难关  商品住房  ▁Braves 无踪  周一至周日  ▁Pulse  名犯罪嫌疑人  饮食上  从那以后  ▁Harmony  expression  课程咨询  ▁Economist 的病  水利局  万元左右  企业员工 不公 摆设  整改措施!  苹果手机! UTERS"  ▁cues" 好房# 尖端# ▁TW$  工程款$ Natal% CCI%  大白菜&  新闻线索&  ▁revolves'  集团军' 清仓(  并采取(  多吃一些)  小家碧玉) ensure* aic* Stats+  categories+ 就吃, 质保,  个月以上-  ▁prophet-  ▁evacuated. 杂货. 翻转/  就只有/  应用题0  还处于0  正式进入1  流言蜚语1 热电2  会直接2  眼科医院3  等多方面3 Impact4  操作的4  晋中市5  ▁upstream5  ▁Connected6 罗兰6  停止了7  相得益彰7  喀什地区8  ▁Fixed8 利康9 蓄势9  没有一点:  角度来说: okane;  ▁wiki;  igrated<  ▁concession< 就当= ▁Eh=  不愧是>  背后的故事>  ▁tempting? 两座? 烟叶@  赛场上@  ▁ResponsesA  ▁rationaleA  ▁InterestinglyB 嘉奖B 遥感C kefC ibaoD  知名企业D 北仑E  摆放在E MMERF ▁AufF ▁gerG  ▁blanketsG 直白H 秀才H  ▁fauxI  拥有强大的I 崎岖J 早读J 正因K  了整个K  疫情后L  一系列的L  ▁squashM  ▁furiousM 卷土N 树种N 附小O  一个叫O  的好友P  这辆车P  会有很多Q  的身体条件Q studioR 写上R 小丽S TueS  创办了T  真心的T  身体会U WIFIU  适合自己V  ▁envisionedV 宽恕W 镇人W  总量的X  这便是X  严防死守Y  值得期待Y  生物多样性Z  ▁commandedZ ZH[ 电能[  单方面\  这个价格\  ▁vascular]  ▁Newfoundland] 改了^ 诞辰^  人生最_  甜美的_  ▁chunks` 瓦解`  并完成a  报名的a  这样的女人b  racticalb  ▁pinpointc 两性c 喝彩d  任教师d  平等的e  的一套e 为重f  麻辣烫f  感到非常g ampoog 班列h Lexh  同学们在i 的香i Guyj lloj  又一批k  确实有k  沃尔玛l  肝细胞l 补中m  共同为m 丛生n 前旗n  积极地o  赚到钱o uponp  中心工作p  ▁disposableq 利害q  都成了r  教育工作者r 吃住s 贴着s  信心和t  ▁indexest 参训u 吸血u  ▁LORDv aphaelv 总行w 类推w  车辆在x  如果地球x  底线思维y  ▁dinnersy 保修z 别太z 老汉{  下学期{  ▁Tiny| 唤起| 胃经}  体验课}  提醒你~ Pair~  ▁eyel 王立 超全  后出现  供血不足  全力推动  全国多地  电视电话会议  画面的  ▁televised USS ▁TC  同学在  谈不上  ▁landfill  ▁regression Neuro  ▁Beaver 高气  全链条  ▁consolidate zhe  的歌曲  ▁turtle 下限  务必要  ▁зна  不得随意  ▁fares  ▁Carrier  我只想  承担的  特别是对  数不胜数 　① 乳糖 苦心  RECOMMEND  ▁incarceration  市民朋友  统一管理  ▁postgraduate 围栏  营业员  藁城区  近现代  合影留念 易怒 本真  人数为  而如今  文明单位  解放战争 ifton  ▁participates  已开通 利水 嘎查 空洞  另一名  世界冠军  ▁disregard Ka 体液  的一块  经营许可证  小分子  有很多的  ArrayList  ▁Elephant 整条 sina  分析一下  景区门票  树立正确的 相恋 觅食  ▁Greeks  Hopefully  ispensable  ▁Electricity  ▁simultaneous 紫禁 ||　 ▁Mn  拿不到 vana  即可享受  ▁Mahm  comings  又有什么  ▁turnovers 暨南  生产者  certified 咧咧  感谢大家  Objects  iential  ▁Astros 天国 就送  上高速 两处 律己 新机 随和  申请执行  ▁unveils 每只 越久  特等奖  共同进步 Mouse 正法  ▁Wiley  一次性医用口罩 历届 徜徉 忍让  一旦被  ▁Amph 前有  专业和  本次考试  ▁cube  自己的梦想 十来 又想 大麦  ▁Ко 险阻  以提高  实拍图  髋关节Œ  要求进行Œ  冠状动脉Ì  分析报告Ì  也觉得Č  我们才能Č  asticalŌ  通过与Ō ']['ƌ  为原料ƌ 本钱nj Masnj  理念和Ȍ  而奋斗Ȍ  ▁GOODɌ % 国家市场监督管理总局Ɍ 分水ʌ 要义ʌ htaˌ  游泳馆ˌ alkš  ▁рабо̌  neutral͌  ▁halted͌ 抛售Ό  日常管理Ό Addingό 哽咽ό 维多Ќ  花时间Ќ  廉洁自律ь  repositoryь  ▁substantiveҌ 直聘Ҍ nosӌ  鼓励和ӌ inseԌ  brandedԌ  ▁complexesՌ 淡季Ռ  听音乐֌  ▁Feld֌ 亚历׌ 已获׌  合唱团،  申请了،  规范管理ٌ  ▁Spacesٌ  ▁pertinentڌ 冰激ڌ 满减ی  对面的ی  工作后܌ ▁CAM܌  天津大学݌  ▁Kendall݌  ▁Tasmaniaތ 依纪ތ 可悲ߌ 花里ߌ 退了  计划投资  重点行业  ▁Cheryl  ▁Wilderness  ▁corresponds  年纪轻轻 artime 津县  初级会计  综合医院  购物广场  项目投资  通常情况下  义务教育阶段学校  ▁earrings 去何  始至终  风水学 afone  健康通行码  ▁hypot  营造一个  ▁guardians 把车 李永  ▁Avery ій ▁pd  你这个  回来后  的号召 ▁sir istro  商务合作微信  ▁explode 博雅  ▁rethink 一房 书生  Reporting 额定 Balt  作为一款  开展情况  日发布的  非常重视  心里话  ▁violates  ▁detecting 急了 春的 貌美 photos  ▁accountant 海天 灌输  会议还  有媒体  其实不是  ▁Fiber arty 股民 苗头  ▁franc  ▁mimic  不远处 unde 划归  调整了  现代化建设  according Rs )”,  怎么吃  掌握好  ▁chin  传播者 终末 ▁vow ankind  第九届  显得更加  ▁lawful  亿欧元  教师们 音色  的老板  资产阶级  ▁emblem uw 产蛋  维护社会 不快 被杀 allo Allow  ▁limbs  bearing  blogspot 十多 莱西 所做  features  ▁expiration 兵马  都有所  上幼儿园  保障措施  ▁Samoa kas  最终导致  自己的事情  ▁linem 龙虎  明明是  文化创意  随随便便  usterity  ▁detectives 尚无  要更加  土地承包  在不同的  ▁inhabited ún ść  东营市  都是一种  olithic  ▁Employers! ек!  ▁outset"  ▁diplomats" 　③#  但依然#  副处级$ 长裤$ <<　%  ▁McCorm%  ▁hostage& 荷塘& invoke'  应该注意'  ▁watchdog(  ▁registers( 前要) 图图)  就出现了* 殆尽* bri+ ▁GI+  坚持做,  看到这些, 月下- 白水-  泸州市.  告知承诺.  国土资源/  ▁malls/ 西林0  无非是0  的习俗1  主要考察1 两支2 每张2 ▁Goa3  敢于担当3  ▁intertw4 本刊4 着脸5 ESA5  都会选择6  ▁Crane6  Preview7 未在7  根本就不8  ▁sack8  ▁programmed9 肉桂9  出租屋: ▁Dip: 操守;  的类型;  ▁Lighthouse< 后置<  不提供=  与传统=  张家界市>  有很强的>  ▁reload? 心平? 田地@ 门禁@ wayneA  ▁graftA  ▁porcelainB  还专门B 才智C mergeC  ▁SeventhD 万岁D ORNE  承租人E reseF  三万六千F  ▁CoronaG  这使得G ▁SobH  都存在H emouthI  公共资源交易中心I \":J  对他说J  的高铁K tangK 概论L  能见度L  ▁SpearsM tekM  民事行为N  ▁ExperimentN 国贸O  机动车道O  笔试科目P  ▁MammP  民事诉讼法Q RecognQ 再苦R 准予R jonS  年实现S  综合类T 屈膝T 货架U  医药费U  都变得V  生态环境厅V  ▁AlbuquerqueW  大环境W  库尔勒X rinthX  ▁CrescentY 代的Y 复位Z  ▁МаZ henko[  ▁dislike[ 可待\ 有约\  产生影响]  应变能力]  ▁cleansing^ 递增^  经常出现_ 于世_ 对因`  ophobic` 相加a 文秘a 郁结b 长宁b ▁FTCc  使用寿命c  ▁Rachd  ▁Maximd  无比的e chioe  累计收到f  ▁humidf  ▁labelingg 五福g 糜烂h 肋骨h Raji  可复制i ureenj 省去j  ▁Inchk  ▁sunglassesk  ▁counsellingl  体育总局l  可以参考m  平台建设m 白话n 虽小n  减轻了o  最全的o  各界人士p  著名的话p  ▁terrifiedq  ▁turbulentq  ▁renewablesr pfr  其中最s  ▁Dishs  ▁Kitst 板材t  和提升u  的定义u  快速提升v 外长v  持续向好w 饮片w  相关法律x eteenx  人类命运共同体y 华的y 来时z 浏阳z dbc{  自然资源厅{ weeney|  ▁justices| 太强} 拖累}  ▁vineyards~ 留白~  主题是  保持健康  甚至有些  ▁GitHub 无病 预审 afer  方针政策 客票 浅色  体温计  ▁Fasc  ▁attained  的收益  不负责任  基本上是  银行贷款  ▁Inaug  Historical 惯了  手机端 meng  ▁weaving 汽修 烧至  了一支  每一口  因为自己  新形势  有经验  ▁lingering % 新冠肺炎疫情发生以来 kf  是预防  生物制品 种业  在床上  工商业  的思维  算下来  里糊涂  毫不夸张  ▁quarry  几天后  用户在  假冒伪劣  ▁funnel 省吃  下降到  日之后  可以放心  ▁Maid 验光  大长腿  ▁unnamed  ▁Contractors 仙气  的销量  ▁Polo tracks  servlet  的注意  达成了 avis forum 在美  所院校  学习中心  ▁tattoos  confidence 族人 隔空 kWh  高学历  征稿启事  工作进行了  validation bots  一切众生  高质量完成  ▁chop  放寒暑假  ▁Barbados  ▁currents 劳工  老师们的  醉酒驾驶 AWS  适得其  ▁muscular  ▁massively 酒醒  差不齐  镜检查  ▁airs  ▁invaded  ▁efficiencies  群体性  遇漫遇漫 Course  ▁paving 牙周 许某  太多人  老有所  集中采购  ▁sting  ▁incarcerated 不爽 仍将 Gre  承受能力  ▁dedicate  ▁overturned 伤风  就是通过  比较适合  ▁Carly )> 取景 平地 Rew  回家的  时需要  ▁wrink  cheduled  ▁rocking 两眼  ▁Walls ssh  的神奇 ▁Zip ▁MX  职工的 IFIC  深刻的印象 下联Í  ▁ExecutÍ 找他č  书中的č  路以北ō ershō  播音主持ƍ  毕业论文ƍ  调查核实Ǎ notchǍ 三河ȍ iwiȍ  一张图ɍ  中芯国际ɍ  即可阅读ʍ  ▁extinctʍ 关元ˍ 由县ˍ ▁LAN̍  icators̍  ▁pilgrimage͍ 小新͍  发展中国家΍  ▁Actions΍  ▁rectangularύ 悦城ύ  学生对Ѝ  进行过Ѝ  为了更好的э computэ 比率ҍ  回过头ҍ  这两位Ӎ  申报材料Ӎ 领克ԍ  空气中的ԍ  ▁PatternՍ  ▁confiscՍ 别无֍ 湖面֍ 萨尔׍ 青涩׍  观察者؍ ▁Miy؍  的安全感ٍ  进行交流ٍ  ▁pestsڍ  ▁totalsڍ  ▁yieldedۍ  ▁Islamistۍ  ▁enriched܍ 丝滑܍ 小河ݍ 性爱ݍ  三五年ލ  所以当ލ JECTߍ Namedߍ 南洋  三七粉  ▁Nicola  ▁thematic % 中央纪委国家监委网站 女团 比和  而不见  ▁Micha 花心 ▁DOJ COUNT  公众号后台回复 蝗虫  令人羡慕  群防群治 raltar  regulated  NavBarCell 突兀  时间线  人身安全  ▁obey MJ enegro 扎西 纯度 寰宇  消除了  行业中  ▁Celebrate 候鸟 前锋 proxy  ▁behaviours  杰出的  ▁Kart 吴江 磨灭  五星红旗  同比上涨  对症下药  补充水分  这款香烟  ▁Hath  ▁Malay  独有偶  开始出现  习近平指出  covered  ▁sailed 洒水 难处  住房贷款  大力开展  ▁Fare 原装  这些情况  自己的内心  TextView  ▁Albania 案涉  经核实  路路口  所以现在  ▁Tile  ▁Uzbekistan 大寒 越想  所得额  ▁gourmet  ▁trivial  了一个月 一跳 应交 情理  这个夏天 Awards  ▁runoff ERIC  ▁lumber 和市  就是他  ▁Geek  commission 逃亡  工程等  交换机和  ▁infl  ▁sincerely 鸟语  这一款  每个人都能  ▁Yesterday cw ным  民政厅  的性质  ▁Practition 塑胶 港币 的火 MAP  ▁pedag 争霸 无趣 骑马  古人的  有这种 azzo cling Dragon  有限公司招聘  ▁artillery  社区居委会 买下 积德 Asset  ▁Yamaha  刘亦菲  又不是  的申请 看齐  ▁Pamela  ▁Brittany 万国 临空 已向 轻伤  追求的  妇科疾病!  ▁anatomy! 安达"  艺术教育"  趣味性# uder#  ▁fortunes$ 会从$ 秋水% rift% ▁McB&  hetical&  ▁loosely' 双倍'  民事诉讼(  ▁Rodney( 后宫)  可以接种)  因素影响*  已经结束*  ▁Winds+  特大桥+  不要前往,  圆满成功,  上学期-  家庭医生-  against.  善举和.  职工医保/  ▁crawl/  ▁rejects0  ▁Nacional0  ▁dinosaur1  千万不要错过1  ▁backend2  ▁cinemas2  ▁excuses3 也行3 阑干4  ▁offspring4 法术5 订婚5  思维的6  转换为6 CEPT7 Guid7  把所有的8 strip8 插本9 羁绊9  帮助别人: nette: optim;  Funding;  获奖者<  衣食无忧<  ▁broaden= 关掉=  自然环境> liners>  ▁Finds?  ▁enquiry? 造句@  一些地方@  占道经营A  水利水电A  StoriesB 爱在B 诚招C  发起了C  营利性D ▁gewD  年毕业于E  ▁supervisorsE  配套的F  产品研发F  就带回家G  新闻传播G benefH  ▁perksH  ▁heraldI 租户I  出席活动J  zbollahJ 率低K 表象K  在一旁L skinL icherM ismissM  ▁JammuN  中含有丰富的N 返京O  不听话O  知识体系P  ProcessorP  NeverthelessQ 一斑Q 天盖R 宏大R 成色S 页的S  一定要在T  ▁AbroadT  ▁avenueU % 绿水青山就是金山银山U }>V 得道V 盐湖W лсяW  ▁discardedX 家事X SolidY northY мяZ 走起Z  第三章[  规模为[ amenti\ 握住\ 横道] suff]  其他人员^  数学上册^  ▁cataly_  ▁Practical_ 眺望` 最晚` bosa  了一辈子a  可以设置b  ▁devisedb 隔音c  是当前c  bringingd 小菜d  明显提升e 友的e  只要您f  看到的是f  ▁riflesg KDg 平层h  算什么h obari ▁Voyi 旧情j 盐田j  摆脱了k  放在首位k  都不陌生l  人力资源部l % 重大突发公共卫生事件m 一院m 莫测n  ▁MUSTn  ▁Sponsoro 相合o Sinp  这张图p  ▁americanq 尾数q Scir  留下一个r  ▁brids  disableds  ▁falloutt  ▁killerst 游艇u 溪流u Shav  劳动合同法v  ossiblyw ▁dupw  教育部办公厅x 商讨x 闭馆y  岂不是y  高度的z CODEz remote{ 狼狈{  主力军|  构成的|  营业部} ▁Biz}  ▁Biod~  正确佩戴口罩~  ▁fertile  ▁Maldives  ▁explorer 夜空  为好友  买房子  互动交流  健身房就  随地吐痰  ▁Halo  还是从  进入体检  高度关注  reating  ▁facade  ▁sausage  ▁strategist  要聚焦 ▁Sik  如何解决 调运  他都会  无独有偶 检讨 Nas  或使用  财务报表 essen 乱占  的处罚 ▁ail  参差不齐  ▁staircase 缭绕  耐心地  邮政编码  ▁Schu KL 字字 悠扬 柔弱  最根本  值得拥有  ▁Thirty  ▁checklist 动心 ▁eer  izational  ▁FRAN  践行初心使命  ▁unexpl  版权归作者所有 泥沙 anova  决定自  发展壮大  不要盲目的  ▁Kyoto  ▁Password 师从 木屋  　“　  适得其反  ▁rumored  ▁Greenville 拜拜  号开始  向纵深 taker  ▁mantra  ▁enrichment  ▁Transactions  发送至邮箱  你最好  去了解  小部分  我们正在  ▁Impl  ▁Plays  ▁Hoover  Semitism  ▁progressing 未按  蔬菜和  顾不上 ffer  克拉玛依  ▁HDMI 闭口  并不大  ▁limb Across  associated 即在 活塞  婚外情  应用型  很多问题 查扣 远期  芜湖市  ▁explosions Hear  学科建设  无奈之下  生生不息  洗衣液  消费市场  ▁sympath 亚特 靠边 韵律  印度尼西亚  ▁spoilers 一串 大田 退房 隐忍  是一道  Methods  Seattle  college  ▁automate  漯河市 vings  ▁eruption 照相  不集中 红歌 通向  已构成  读者的  ▁lookout 丧葬 ▁FSŽ  表达了对Ž *****Î  ▁ResponsibleÎ 纸本Ď GroĎ  的团队Ŏ  坚决反对Ŏ  敬请期待Ǝ NathanƎ  ▁inadvertǎ 皆知ǎ  所用的Ȏ  污染治理Ȏ  ▁LimitɎ  ▁outweɎ  黑寡妇黑寡妇ʎ  经常性ʎ  重要了ˎ  地发现ˎ  这么快̎  ▁repairing̎ 孢子͎ AFTA͎  ▁reclΎ 它将Ύ  学史明理ώ huanώ  ▁HomerЎ  ▁obsoleteЎ Gayю  协议的ю  曾表示Ҏ CompatҎ 当庭ӎ  在单位ӎ  新作为Ԏ  旧金山Ԏ  和建设局Վ  ▁mixerՎ  ▁tracker֎ 呼兰֎ 喧哗׎  方法一׎  ▁Toby؎  ▁Swing؎  新冠病毒感染َ  ▁Westonَ  ▁warrantiesڎ 沂蒙ڎ  所作所为ێ  能让人变得ێ anagan܎  ▁mercury܎ 二话ݎ 德拉ݎ  预售证ގ  ▁patriotގ 盲人ߎ  ▁enrolߎ 油茶  的一周 五帝 清徐  有一群 hadow  ▁slowdown 小课  以百年  匈牙利  艺术中心  的一项重要  authors  一味地  与安全  十二年  的旅游 冒用 支点  三部曲  分布于 Dick ▁gol  ▁cliffs  ▁derive  ▁Guinness 立下 笑的 预选  或许会  ▁Brewery  ▁customization 双流 浓稠  不通过  ertiary _{\  上一年  mopolitan  ▁Monsters 远流  的高等  无产阶级  独立自主  年普通高校 aitlin  umbered  ▁Jubilee 肌腱  一直没  反复发作 ialis 牟利 购销 ▁Md ▁CHR  ▁Rack 康乐  ▁mortar ▁DP 优劣 棱角 男装  的开发  ▁ashamed  避免因  都认为 哪位  教师在  ▁coil  定兴生活圈  dingxings  dingxingshenghuoquan 惊恐  生长在  ▁HOME  ▁Shower  Mercedes  engineering  区位优势  文明交通 法医  ▁ми  阅读和  ▁roast  ▁subsidy 偏执 满心 速记  然而在 docx  caution 暖意 mie  独自一人  ▁stor  ▁stealth  ▁humility 历久  面对的 Phill liang  ▁rugs  加胡椒粉和  ▁Festivals 九成  上市后  的成果  金融学  车辆进行  风险防控  navigation  ▁underserved 造影 ▁CG ▁hrs  这项工作  ▁peaked  ▁cherished  ▁plantation 执著  法治化  ▁hiatus 下个 变通  心理辅导  必须坚持  国家药监局 陆游 eny  对父母!  所不及!  使用的是"  如有需要" " 事实上事实上事实上# 初创# 深藏$ edes$  国际油价% ús% 小鼠& 拉尼& 狂热'  不能及时'  教育发布( antan(  picture)  ▁furnace) 收据* 银幕*  做好事+  发布后+ ▁Yak, 可就, 混搭-  一份力-  总体上.  做什么事情. ▁Ath/  促进作用/  创造条件0  的工作中0  剥夺政治权利1  ▁organism1  ▁universally2 当其2 球的3 直筒3  在党的4  ▁Verde4  原作者取得联系5 噱头5  特朗普政府6  ▁preaching6 ×　7 绝伦7 背离8 议案8  的水果9 Mand9  峰值扭矩:  ▁Gott:  ▁paperback; 缴款; 而解<  工作汇报<  戴上口罩=  详细规划=  registered> 旗号>  点上方? abay?  ▁disinfect@ 7 0大幅度发大幅度发大幅度发大幅度发@ 白鹿A 陈薇A ▁TusB  离家出走B  基本圣诞节C  ▁goddessC  ▁shortlistedD  了他们D  合法的E  抗肿瘤E  就说句话F 刘倩F 收成G  除草剂G  和交话费H  飞放不下H  发反反复复I  政协副主席I  飞放不下就带回家J  ▁StrangerJ ALSEK  服务法师K  ▁johnL  和交话费接电话L  圣诞节我还记得M  ▁wicketsM ( !和交话费接电话就是不行N . '发反反复复飞放不下就带回家N . '服务法师事实上事实上事实上O = 6圣诞节我还记得和交话费接电话就是不行O U N服务法师事实上事实上事实上发反反复复飞放不下就带回家P 诗情P  信息进行Q  阅读后Q  会计师事务所R F ?圣诞节我还记得和交话费接电话就是不行只能说R 能不S " 基本圣诞节氨甲环酸S  ▁admirationT 入编T 每股U  公园里U  哪些人V  重灾区V  ▁comfyW 机密W 燥热X 考完X 这人Y  和智慧Y  最受欢迎Z  积分卡技术Z  骄傲善举和[ Linear[  疫情防控工作中\  放假时间繁花似锦\ % 加胡椒粉和骄傲善举和] + $放假时间繁花似锦胡椒粉和] 4 -加胡椒粉和骄傲善举和积分卡技术^ X Q加胡椒粉和骄傲善举和积分卡技术放假时间繁花似锦胡椒粉和^ seq_ ▁vul_  说句话家具`  ▁ashes`  Festivala  北京市房就说句话a g `加胡椒粉和骄傲善举和积分卡技术放假时间繁花似锦胡椒粉和说句话家具b 隐居b 古墓c 轰鸣c 预科d  茉莉花d  补充营养e  背景下长时间e  骄傲和武汉分f  交换机和福建省f % 骄傲和武汉分手机话费g : 3骄傲和武汉分手机话费交换机和福建省g 止步h 遗余h  常德市i  闻名的i  下班就恢复j  的户外护肤j  健身房就撒娇k  结合实际家和k " 和建设局下班就恢复l " 放寒暑假的户外护肤l + $健身房就撒娇背景下长时间m 4 -和建设局下班就恢复结合实际家和m O H放寒暑假的户外护肤和建设局下班就恢复结合实际家和n 各能n  杀菌和送o  圣诞节按实际o  就像你这么说p  就回房间杀菌和p  杀菌和送积分换q " 粉红色就像你这么说q ( !杀菌和送积分换届时将会r  ▁Directorater 7 0粉红色就像你这么说就回房间杀菌和s [ T健身房就撒娇背景下长时间粉红色就像你这么说就回房间杀菌和s a Z圣诞节按实际放寒暑假的户外护肤和建设局下班就恢复结合实际家和t  和维护t  签订合同u  ▁leasesu  ▁clarifiedv @ 9北京市房就说句话杀菌和送积分换届时将会v 衬衣w 毁坏w itonex  ▁Purchx  ▁Disclosurey 花木y  结构性改革z Uniquez  那么问题来了{ 等他{  的作业|  的内心|  ▁dissatisf} GPU}  ▁McGu~  ▁Hazel~  Returns 丝雨 粉刺 dry  商业地产  扎实的  在短时间内 践踏 雄鹰  赶紧来 leine  ▁reins  党群服务中心  ▁CONDITIONS 护法  rocities 爱他 臭氧  的春天  ▁Salvation  天府新区  进行治疗 南向 kat ком  ▁Warehouse 中产 被逼 越冬  ▁modeled  ▁Developers  省吃俭  引起重视 强险 罗定 Opp  至少提前  Emergency qw 柳树 胡歌  撒的发  中石化  操作规程  ▁Wheat  ▁Assets  函数的  联系上  尽职尽责  持续稳定  inderella  ▁indicative  ▁existential 都行 arra ▁inh  有了新的  ▁agility 各具  上升趋势  兵役登记  也属于  爱护动物和  咸阳市  生命线  全部拆除 之欲 小桥  找机会撒  ▁bast  utherland 宝应  不会对  弗兰卡  没有人会  哈酒圣诞节  至关重要的  东方红届时将会  ▁commune  后视镜爱护动物和  啊啊啊啊啊撒的发 ( !东方红届时将会找机会撒 . '哈酒圣诞节后视镜爱护动物和 @ 9东方红届时将会找机会撒啊啊啊啊啊撒的发 g `哈酒圣诞节后视镜爱护动物和东方红届时将会找机会撒啊啊啊啊啊撒的发  ▁resonate C <大幅度发大幅度发大幅度发大幅度发大幅度发 两倍 便将  限时特惠 ursive  ▁Wolfe 精制 酷炫  宣布了  点个好看 Render 捣毁  习惯性 天籁 家叶  临汾市  居民区  ▁amazingly 眷恋  曝光了  看手机  也不一样  医疗设备  提前了解 winner  ▁temporal 剧团 景致  ▁Madh ^ W大幅度发大幅度发大幅度发大幅度发大幅度发基本圣诞节氨甲环酸 ▁wy shit  ▁Spielberg 死记 Jer  取消聘用资格 宾阳 栩栩 街巷  可查询  estation 独到 сли  狂犬病  走出了  ▁Misc \/ 因材 无牌  经济学院  ▁coresÏ 一扇Ï  不参与ď  坏消息ď  的高效ŏ  第三者ŏ ▁MizƏ  ▁arsenalƏ ”…Ǐ 头衔Ǐ  回报率ȏ abusȏ  ▁monkeysɏ  ▁Philliesɏ ✦　ʏ 靠拢ʏ  顺时针ˏ  ▁prayedˏ  ▁allergy̏ 庚金̏  大宗商品͏  ▁speedy͏  ColumbiaΏ 她去Ώ 获赠Ϗ  转载请注明来源Ϗ  ▁battledЏ 三毛Џ 茶艺я  该平台я 五级ҏ 精液ҏ  机体的ӏ blocksӏ 娇艳ԏ  坚持使用ԏ  工作思路Տ  ▁beholdՏ Boss֏  期临床试验֏  ▁termed׏  ▁skyrock׏  Interested؏  ▁statistically؏ 海浪ُ Gerُ  要记住ڏ failڏ  整改落实ۏ  ▁sundۏ  工作推进会܏  ▁selector܏ 房主ݏ  前男友ݏ  的空气ޏ Feelޏ  及时报告ߏ  权属证明ߏ Beauty  ▁Networking  ▁factions 开间 搏击  防范措施 spots  ▁Robotics 松林 果冻  易发生 ▁Vie  技术应用  重要贡献  ▁clich  ▁Wanted DG  艺术馆  要实现  ▁antis  教育和体育局  ▁joyful ▁oft  安排上  带薪休假  检查指导  ▁recycle  ▁contagious 拮据 Baz  我非常  未来将  的不满  勘察设计  ▁NOTE 怒火 支为  热情的  的人间  给排水  如涉及版权问题  Schedule  ▁motivations  一致性  长方形 关机 莫里 TOD  江南北  公平竞争  ▁Thurs  ▁sinks  ▁Robbins  izophrenia 一探 原木 orah  党委常委 赤子  和经验  微不足  只能通过  在线教学  不知道自己  ▁ounce  累计治愈出院  的穿搭  孩子们在  根本就没有  ▁Bridg  ▁troublesh  ▁Investigator 褪去  滁州市  学校食堂  新闻办公室 abouts  城乡居民医保  ▁wearable 发质  生态园  美厨娘  公共秩序  湖南卫视  ▁encycl 名贵 对表 过河  Protocol 演说  才真正 教龄 核电  靠的是  ▁freak  ▁authoritative 断货  一起在 rone sync lip  司马懿  有责任  的角落  无非就是 保重 荒凉  的逻辑  一位作家  批准逮捕  有的地方  重大决策部署  ▁hardcore  ▁disruptions 洗牌 海默 燎原 良缘  恋爱关系 among  不会影响  他们就是  ▁Fountain 六道 不提 就学  和丈夫  招贤纳  你要知道  诈骗分子 音量!  碳纤维! Infl" vous" launch# 吃醋# ▁Kum$  ▁obsess$ 图集%  张桂梅%  党史百年&  取得成功& 你真' 硬背' fmt(  如果用( Often)  大品牌) odel*  ▁Utilities*  ▁transitioning+  流传着+ eeks, exter, 」(- 误食- neh.  ▁conceded. 除以/  ▁Serving/ 商的0 疯子0 遛狗1 FTWARE1  Hamilton2 zez2 是能3  等要求3 iners4  二年级下册4  ▁Recip5  ▁Caucus5 scene6 within6  ▁insign7  ▁expectancy7  ▁sustaining8 北平8  好几年9  爱国主义教育9  ▁McGill: 尘土: eson;  报告新增; ewhere<  定兴生活圈微信号< + $微信关注定兴生活圈微信号= 保山= 又很> 杏林>  Payment? 夜游? 带点@ 成套@  换一个A  看清楚A  不能超过B  irmationB ▁CALC  ▁AlpsC  NigeriaD  全省累计报告D 淡薄E 独资E 龙区F  可以降低F  继续做好G 片酬G  地球只是H  所使用的H  加强组织领导I  ▁PricingI 好课J 广角J  为市民K  连环画K  ▁copyrightedL 陈思L  都很难M ▁FUNM  网友纷纷N CHECKN  ▁militiaO  ▁intimidatingO 即发P  国务卿P  分享一下Q lingerQ 教子R gardR ▁DLCS  第十四届S  ▁EverestT 殷勤T owayU  农村人居环境整治U bethV  可能发生V  ▁JsonW  othermalW 大别X 老外X 蔚蓝Y  或者以Y  德智体美劳Z  做了什么Z  Saharan[  ▁Flores[  ▁streamed\  ▁groceries\ 铁塔]  不停歇]  ▁Alberto^ 禁行^  不好好_  积累的_ ▁ROI`  已取得`  ▁Chowa  collapsea  cylinderb  férencesb  ▁vitalityc  ▁organisersc  嫁给了d  记者来到d  快乐来自于e  ▁tradere ▁nemf  ▁Legof  ▁Olsong  易燃易爆g  Testingh 两起h  可以加i  ▁Winchesteri  为依据j  传播风险j  纪律检查k LUk ▁MIDl 到有l 林草m  去何从m  想要什么n  点击这里n  ▁Charlotto  等相关部门o шаp 我太p 折起q kimq 善念r 国库r  不服从s  的任何s 上联t  不负责t  仅用于u  经审讯u  公共场合v  全资子公司v anssonw  当选为w  想通过x  瞧不起x  都不愿意y dancey  复合肥z  对美好生活的z 工时{ 模组{  和未来|  得出的| ▁RAF}  很有意思}  正确使用~ 乙酸~ cra  阴道炎  ▁McInt  ▁ridge 不接 不识 展翅 腕表  出香味  发信息  汉中市 ▁AIR  ▁рай  会加重  瞬间永恒  ▁capitalize 选人 Iowa  抑制作用  ▁ther  ▁Shelley Rot  Graduate  ▁Shelton 黄曲  到暴雨  情形之一  ▁arte 久别 зна Storm  简直不要太 finity 南都 印染 olu  他们已经  这就意味着  ▁Institutional 本息 联运  商贸城  如何用 ratch  ▁Serie  ▁Isaiah  ▁faulty 岳飞 王寒 忍者 我给 无话 的词  情不自禁 " 国家卫生健康委员会  ▁Bernstein 伊藤  住宅楼 Palest  ▁Weird NK 再忙  留不住  ▁carers  前列腺炎  ▁bisc 后用 AMI  报道了  曲靖市  就业机会  未尽事宜  eremiah  ▁ancestry  和经济社会发展  hetically 比丘  甘油三  ▁Pompeo  ▁succeeds 射程 玉皇 Mix  毛主席才能 呼呼 带出  三年内 FLAG  为什么这么  ▁Broker 入冬 入秋 fits ▁EPS  肾上腺素  自主创新  Benjamin  ▁Naturally 她了 ▁Ft  ▁footh  ▁radioactive 楚天  老客户  公共服务平台 姚策 德清 突袭  与当地  可导致  着力解决  结果发现  上的一粒沙 Points 晓东  向上滑动查看 财库 转战  放在一起  的创始人  ▁Plug 娓娓 零散  ▁Loch  ▁Audience itel  一个月内  安全驾驶 别把 IPP  ▁Sidd  ▁misdemeanor 提着 造诣 icc 刷脸 助眠Ð 均应Ð 明码Đ 林海Đ  一直很Ő  不放过Ő  水涨船Ɛ  中国联通Ɛ  回家看了ǐ  如图所示ǐ  ▁PercentȐ 大肉Ȑ  广高速ɐ  参与其中ɐ  ▁Shayʐ  普普通ʐ  个性化的ː  网络报名ː  ▁Departments̐  ▁contrasting̐  的一步͐ bron͐ ZJΐ  ▁Jokerΐ 司空ϐ 失明ϐ 青城А  ▁LadenА 中超ѐ ▁"";ѐ CREATEҐ  dioceseҐ  ▁bumperӐ  ▁PodcastsӐ 先祖Ԑ 比心Ԑ  安东尼Ր  有了更Ր  计算公式֐  ▁pavement֐ XPא 购入א  手抄报ؐ  义不容辞ؐ  与宇宙相比ِ  ▁Schumerِ  番禺区ڐ budgetڐ 公私ې 固有ې 大闹ܐ 少说ܐ 检视ݐ Appendݐ 酒馆ސ  新年的ސ  驾驶室ߐ  相关文章ߐ  革命老区  ▁oval  ▁Marcos  ▁prevail 于事 掉进  太平洋上的一粒沙 7 0哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈哈 校验 demo  的一段话 ouched webkit  ▁transcription C <微信公众号微信公众号微信公众号微信公众号 变故  指芳华 tera  的女孩子  Industrial  ▁butterflies 九宫 无罪 烽火  赏昙花 不急  的制度 lein  清明与宁静  ▁Dillon " 如果地球与宇宙相比 ugo  被骗了  Limited % 能让人变得清明与宁静 外衣  刹那美丽  昙花虽然  Islamic  但也是刹那美丽 . '赏昙花能让人变得清明与宁静  渐渐的  ▁Lone  ▁Astra  关于进一步做好  翻天覆地的变化 李春 村口 露头 Synt  弾指芳华  详细地址  ▁burgers  联邦政府  均为境外输入  昙花虽然弾指芳华 赐予  学习到  劳动者的  ▁descended 气道  上下滑动查看  ▁mosquito  若不撇  ▁Burning 打骂 ▁MG  儿媳妇  全体人员  ▁converter + $版权归原作者及原出处所有 质效 inj  ▁embell 既可  多少次  等严重  ▁Cobb  ▁Vest g `微信关注博野在线微信关注博野在线微信关注博野在线微信关注博野在线  收得住  大大增加  Failure らい  已获得 Rece  时要注意  ▁wrapper  绚丽的  ▁heroine  ▁zombies 咋办 欢快 而就 谦逊  并积极  平安健康  Traditional 本村  事件发生 dll  对部分  指导性  研发出  ▁mirac  绿色科技管理 % 零售价格绿色科技管理  或通过  第四批  轻松音乐  ▁Beirut 备胎  算不上  拉开大家 archar  ▁Retreat  而产生的 rapped 某天 泽区 的石  正当理由 " 看了就立刻就立刻就 依附  周总理  还有一点  urricular  ▁monopoly cou  后果由 乐活! 玉雕! 脱俗"  仪表盘"  数不清#  设计理念#  ▁Cube$  ▁propriet$  ▁Apartment% 牲畜%  药用价值&  ▁hast&  ▁rash' 奥会' 煮粥( 维保(  小型客车) ovina)  ▁Retro*  ▁Voters*  ▁Olivier+ 两根+  ▁tyres,  今天夜间,  结束之后-  ▁Athen-  剧中的.  埃里克大家. hx/ ▁WP/  高性价比0 末日0 废墟1  吸引着1  拉法基绿卡2  垃圾疯狂垃圾2  ▁Hazard3 " 九分裤垃圾疯狂垃圾3 中青4 街镇4 起手5  考试和5  ……”　6  技工学校6 imble7 桥下7  何去何从8  ▁Tire8 忧患9  三年行动9  ▁Neuroscience: 4 -统筹推进疫情防控和经济社会发展: fz; 以德;  的一笔<  联系的<  晴间多云= ZKJCG=  ▁whichever> 听讲> 无息? 红心? effic@ 松劲@ 补发A 颈肩A  改善民生B  联系客服B 成章C  ▁descendC 猫猫D 转了D furE  恩施州E itosF ▁BARF  两者的G  却不是G  水上乐园H  我们来看看H 例会I 周至I 手相J  学史增信J  ▁cloudyK  DemocratsK  ▁HumanitarianL 更广L  日以后M  生态圈M  ▁showroomN 索道N 麻黄O  的付出O  还有的P  小时左右P radorQ  部分地区有Q serialR  InnovationR  ▁TelescopeS  有人来S  美术家协会T  ▁SinclairT 父爱U  我们平时U  ▁slogV  丽水市V  扩大了W  从源头上W  ▁GivesX  ▁SerbianX 唐某Y  ▁ElimY  ▁prognZ  ▁collegiateZ 向荣[ 在金[ 最广\ 转出\  已基本]  ▁Irene]  刑事强制措施^ 句式^ 河山_  要好好_  兰州新区`  四次会议`  ▁bridesa 大竹a  天内有b  灵活的b  用户体验c  矿产资源c  也许你d  又是一个d 此为e 站上e igorf ▁Gebf 突飞g  是一群g  周末双休h  roideryh 喷涂i 日用i  都不错j  thodontj  ▁seasidek  Authoritiesk 建发l 故人l  ▁Chargesm 切丝m 帮着n 强县n  和农村o  均衡发展o 对所p 提上p  马齿苋q  基础研究q  提供更加r  稳定的工作r ljs  刘少奇s  的年轻t  ▁Alleyt  ▁Sabhau 等来u 维埃v 雪爪v  将全面w  张学良w  报录比x  独立学院x  经研究决定y  ▁Carmely 会做z  的光荣z  到就近的{  揭牌仪式{  \\\\\\\\| 下身| 内附}  这些行为}  ▁nutrit~  路以西~ 佐酒 守正 Nut  月至今  的面积  五百多天 不醒  精神风貌 paste pb 撮影  郴州市  已经达到了  abyrinth 在必  下发通知 嗓音 总第 辩解 ▁dy  ▁shouted 与爱  卖出去 ankar  ▁promoter 未央 辉县 zai  争取到 ▁Eck  ▁loot 最是  化发展  在空中  的一座  的标签  计算方法  ▁Judges bars shou  应纳税额  开学工作  ▁Champ  ▁tremendously 焦化 YES  如果需要  ▁adjud  般若波罗蜜多时  时光的  班级群  盆满钵  近亲属  ▁Largest 扣留 那颗 中金 大华  南充市  年一季度  考试录用公务员  ▁lesions  各地市  新品种  第十届  ▁echoes  ▁faults 相片  每一天都是  应纳税所得额 煤电  同时要  笔试和面试  ▁caliber  ▁Statistical 再吃 酸枣  Charlotte 弱项 补上  普通高考  职责分工  ▁Fate  ▁Souls  ▁empowers  ▁exploits  后半生  帮助他  陪你一起  ▁Garland 掏空 跳跳  血压升高 抱住 Ashley  ▁veins  ▁Continues 休整 突显  重温入党誓词  可引起  尽可能地 干爽  回答问题  ▁visc  与我们共享  ▁Bravo 体裁 оло  有一付  上了年纪  很有可能会 liable  ▁travers 倦怠 天柱 急招 有这 被爱 雪域  上没有  蚕丝被 良机  返校复学 brown  ▁Zack  ▁gala  吃饭的时候  ▁Bowling 从他 致贫  禁止使用  ▁ditch  ▁residual 精良‘ 追着‘ PitÑ  里没有Ñ genređ  ▁Fremđ ulsiveő  nominatedő 神态Ƒ  ▁shedsƑ  ▁armiesǑ  ▁blondeǑ  ▁globalizationȑ 几经ȑ 鞭策ɑ MVPɑ  上一口ʑ  不断强化ʑ  就是如此ˑ  ▁conversionsˑ 且无̑ 大椎̑ 绕道͑  设施和͑  经国务院Α 全警Α 拆开ϑ 风变ϑ  反义词Б  的心中Б stableё 野兽ё  不大于ґ  检测卷ґ  身体验ӑ  ▁SPECӑ  说得有意思ԑ  ▁benefitingԑ 避风Ց  到指定Ց  快递小哥֑  图片仅供参考֑ 岁岁ב 的吧ב 褐斑ؑ  活动室ؑ  铺天盖ّ ▁POSّ 风中ڑ  先生在ڑ  只能靠ۑ  调研指导ۑ  ▁pornographyܑ STEPܑ  财务人员ݑ 带鱼ݑ 盗汗ޑ 虎牙ޑ ▁Izߑ  产业体系ߑ  ▁Austen  ▁Container 鸿妮  以及相关 CHOOL  ▁Guill 住终 种养 脚踩  一味的  实是不易  让很多人 zys  流口水  调整的 ULAR  学习氛围 Alert  有一付对联 dialog  他们把  怎么样了  能有雪爪  ▁Petro  能有雪爪鸿妮 新沂  喜闻乐  的办学  招生考试院  ▁Cooking  三万六千五百多天  ▁ignorant + $也不过是三万六千五百多天 IMS  以百年记  撇捺人生  的姿势也  住终有苦  若不撇住终有苦 % 有一付对联说得有意思 Kam  保障机制  ▁cornerstone  考试题  ▁idle  ▁Fairy  ▁Tunnel  ▁reacts  ▁positivity  室主任  人这一生  这么些农村  向法院  人民路  即成名  年北京  特别容易  ▁CCTV  ▁descriptive 逼真  多月的  微不至  运行情况  ▁seismic  ▁squirrel 有生 都做  建立一个  留下来的  Inspired 站内  潜力的  灿烂的  特色美食  运营中心  ▁Cous  明显支持你  ▁Chaos 银屑  附加值 MH 功于 横扫  这一套  ▁Heard 弹道 试驾 驱寒 即苦 富阳  ▁revived  ▁unmatched 与世  任何机构  焚香品茗  listing  担心自己  ▁crane  优势的 ▁***  我们也要  联动机制  ▁Calder  ▁thwart  多姿色  ▁Came  ▁selfie 古田 复业  是人们 ébec ellan ranged  ▁Bergen  ▁welding 恶人 边角 ден  会降低  倾家荡  还是得  ▁NAME  ▁unsc  ▁Alumin  Philadelphia 能除 边走  多部门  中国制造  ▁Abby  ▁indicted!  更严重! ▁NYU"  纳税申报" Katie#  捺住即成名#  副总理$  渐开花$  组织专家% 自测%  传媒集团& " 行深般若波罗蜜多时&  ▁Rotterdam' 坚挺' 白芍(  经现场( ▁Ain)  单位名称)  诗词唱和* ETHOD*  ▁invoke+  ▁painters+  ▁choreography, 热了,  台风送清凉-  ▁Powder- 日韩.  工程和.  人到闲处/  妙曼多姿色/  小暑台风送清凉0  ▁Congratulations0 光山1  安心静观1  安心静观渐开花2  ▁Reverend2 圈套3 心碎3 tie4  明星的4 计费5  就不再5 存的6 慢病6 闭谢7  聊聊天7  限制了8 Cass8 prec9  学校教育9  均匀的:  多多少:  ▁Coul;  六年级下册;  ▁skeleton< Isa<  健健康=  恢复原=  ▁Lara>  新冠病毒检测>  forEach?  Catherine? 消融@ 药理@ DougA  他是一个A  党的组织B  ▁SubmitB  联合发布C helpsC  ▁whimsD  IllinoisD  决定在E  有些地方E  ▁lubricF  充满活力F  机关干部G 晃动G 浅浅H ) —H  ▁облаI  全部转载I opedicJ  ottagesJ 穿山K  也很高K  奇怪的L HeatL MahonM  因工作需要M  negativeN [/N 合体O 知觉O  将孩子P  新风尚P  未落实Q  不在少数Q  写作能力R  特殊性R  icultureS  religiousS 晶圆T  派出所所长T  ▁canadaU  的创作U 音节V 伸缩V  出身的W  刚烈异常W  极其狂放X 融会X HarmY 响水Y  ▁PersonnelZ  forestationZ 信中[ 馆藏[  ▁blev\  闭谢的姿势也\  keywords] % 闭谢的姿势也刚烈异常] 互为^ 勤勉^ 磨炼_  拿下了_  施工方`  的统一`  考试安排a  ▁frencha  方式为b  的军事b  不减姿态c  象倒挂金钟c 受试d ▁STEd  ▁renale 吊车e 었다f  学史崇德f 煮透g 而走g  一出生h  Schoolsh  ▁nuancedi  ▁deferredi 直射j 稚嫩j  天水市k 宗彪k  ▁dividingl 别字l 善后m 安仁m 数列n 重则n  根本没o ▁Urso  服务于民p luckyp  ▁analysedq 器等q 战乱r  中存在r  是针对s  液化气s  本次课程t  观察细致入微t  ▁embodyu  ▁safariu  ▁masteredv  ▁Trademarkv 太监w  我的人w  ▁Jaguarsx  ▁waterwaysx 入读y 又以y 格纹z  据中国z  的支持下{ ueblo{  最重要的就是|  ▁gritty| 呼喊}  无前例} ▁gaz~ ▁xen~  人的生命  所有内容  报名流程  科普知识 abbage  ▁mentorship  ▁menstru  ▁scanned 窃取  第三位  步步高升  ▁sans RIGHT  公司总经理  justice  ▁pendant 发情 电报  成员的  administ  切好的  控制线  经审查  肉制品  ▁JOIN 山石  快节奏  小学教师 梦寐 猪群 猪肚 订餐  是人字  沉迷于  见不鲜  都来自 导学  控股股东 Except  Morning  中有一句著名的话 高平 在党  信息等  环节的 bies  经济的发展 Flight  ▁Tables іль  多元化的  微信红包  我欣赏昙花 winter 丝袜 之言 开标 blob  就业方向 thumb  如果你也想  巜人类简史  危害公共安全  ▁Efficiency  撇不出  艰难的  ▁Canvas  与我们共享昙花  快乐来自于内心  ▁Measures  并不是说  经济复苏  远景目标  ▁Ethnic ▁SAN  大年三十  ▁Trustee  ▁Powerful 登封  考核合格  ▁CHAR worked 出钱 硅谷 私有  路以东  建筑垃圾  ▁rude 圣洁 映射 长清  少出门  草木记  营养均衡  balanced  ▁datasets 勤于 声势 夫子 沉溺  拉肚子  珍爱生命 chlor  重点项目建设  ▁Provides  ▁heavyweight 八百 多赚  ollower 心软  哈哈哈哈哈  ▁Ranking 以更 远见  希望广大  生猪养殖  新世纪的口号  ▁Gerard  各能捺住即成名  ▁invests  ▁vouchers  科研攻关  项目规划 gering  ▁Yugosl 超时 ▁Kru  高度评价  ▁Revealed  ▁supremacy 千篇 马达  能提高 phys ▁Lau’  技术培训’  flowersÒ  乡村振兴有效衔接Ò 一山Ē  为昙花Ē  要明确Œ  追究法律责任Œ savƒ 冷清ƒ 斑驳ǒ 机灵ǒ ayoȒ  不见面Ȓ  是按照ɒ  没有出现ɒ 记事ʒ  必须得ʒ  醉里挑灯˒  ▁Aunt˒  ▁pept̒  你就会发现̒  ▁earnest͒  ▁prompts͒ 凉快Β 阆中Β BALϒ  不简单ϒ  有朋友В  ▁BareВ 我怕ђ 赵立ђ blindҒ  ▁MemoriesҒ 善缘Ӓ 空隙Ӓ 驱散Ԓ  通过这次Ԓ  ▁hullՒ  ▁PUBLICՒ  一撇一捺是人字֒  probably֒ 也请ג 速干ג  疑难杂ؒ ▁Najؒ  ▁Meritْ  查尔斯ْ  捺收得住ڒ heringڒ  捺收得住是名ے 老话ے  能获得ܒ  驻马店市ܒ  attributesݒ  作画册ݒ  顽强的ޒ  参会人员ޒ  撇不出即苦ߒ  ▁inauguratedߒ  原子弹  车主们  三五小时  到来之际  村党支部  ▁Fiona  ▁Earnings 即逝  ▁ком  ▁Islanders 的油 粉条  岳学子  转基因  ▁revered  村民小组 esley  ▁austral  ▁Architectural 那场  卢霞客  成交额  ▁hing  为昙花留影  ▁noisy  oscopic  为昙花留影作画册  ▁intrigue  ▁extremism 赃款  家叶晓光  ▁hail  与台岳学子  最低生活保障  ▁Amateur  卢霞客与台岳学子 伊春 驻守  而且还是  Editorial 所载 shr  成这样 Bear  ▁Gina  撮影家叶晓光  ▁contextual  ▁disrespect  ▁Theological 二来 是给 牌坊  他们连续三年 面朝  甜蜜的  ▁whip 厚植 都让 iju  组合拳  ▁stains 嘈杂 白眼  二十九  ▁unsur  ▁vitro  ▁overdue 身患 龙骨  还真是  那么容易  的人际关系  dispatch 拼了  如此一来  抓获归案  ▁Demo 代偿 有肉 pow  到当地  第一代  具有正常  这种说法  ▁Merchant  ▁Seriously 堕胎 细小  一起为  学术交流  everything  各领千秋  庆祝活动  弗洛伊德  hampton  ▁fringe  开放的美好时光 1 *与我们共享昙花开放的美好时光 尿裤  红色旅游  ▁Clippers 元凶 抖动 青浦 UNG  每次坚守  就是我们的  江南草木记  讲到了这件  ▁meats  文字唱和  讲到了这件乐事  ▁lifespan 年春  色彩的  批改作业  烟草专卖  ▁dictate  ▁equival  ▁stalled  每次坚守三五小时 宏光  的小编  通报批评 激荡 ohl  搬到客厅  诺贝尔奖 土生 异域 阿迪!  一个家!  具体到"  有独钟" ▁Rak#  已解除医学观察#  ▁Shadows$ 立冬$  有一个人% aurant% compar&  landers&  ▁Whereas'  帕金森' mmol(  任前公示(  ▁Cathy) 以继) 薯片* 长袖*  尽最大努力+  ▁cleaners+  ▁Christchurch,  王寒夫妇,  ▁Beef- 封堵-  在事业上.  总结一下. timer/  ▁indulge/ 百味0  专业机构0  ▁adjunct1 水是1 grove2  ▁Burma2 性温3 爱丽3 集赞4  延伸至4  持久战5  还是想5 不炒6  时指出6 Mess7  ▁REAL7  erdinand8 很近8 有别9 等次9  性骚扰: ▁Hun:  保持稳定;  ▁lodging;  ▁adherence< DW< 脐带=  开创了=  也被称为> ourke> umeric?  ainment?  ▁statutes@ Vs@  解释了A  贸易商A  ▁ScottsB  SecretaryB  ▁GreenwoodC 与国C SprD  大家也D  已有的E  还有点E  最美逆行者F  ▁oversizedF 冷热G  的有关G  isabethH @ 9并不代表本号赞同其观点和对其真实性负责H 拉链I  张爱玲I 剧透J 天心J 霍普K TESK  的心境L iranL  过敏性鼻炎M  ▁unfinishedM  就喜欢N  扫码咨询N  ▁LevinO  ▁gloomO  IndonesP  ▁skepticP DannyQ 元至Q 多才R 降火R PelS  出一条S  汉武帝T limeT  的一次性U  也是非常的U  新闻客户端V  ▁MongoliaV 大虾W 街舞W  不要说X  莫大的X  调度会Y 拐弯Y 指正Z 顾忌Z iska[ 写明[ 同向\ 总比\ 戈尔]  完美结合]  说到这里^ 取下^  把工作_  ▁debuts_ 喂奶` 已无` 新郎a 派发a 视感b 重负b OLOGc  先后顺序c  ▁Coppd Nicoled  ▁optinge  ▁Operatore 造的f  国内中高风险地区f  秀恩爱g  罗伯特g ursedh  级应急响应h  ▁Corpori  ▁adaptationsi 中是j 就做j 得胜k 方块k 缺钙l чноl  不止是m  恰恰是m orrhn ▁LTDn  ▁defectiveo  便可以o ▁nudp ▁tyrp  一级建造师q  日表示q  ▁Coffr 休眠r 水运s 腿长s 萤火t  正方形t  ▁Mishu  可适当放宽u 交锋v  ▁quilv ( !践行社会主义核心价值观w 不食w 溥仪x 闭眼x  原计划y  ▁shortesty 共情z 等与z  创新性{  ▁Builders{ VPN|  的规划|  ▁alterations} 其事} 此款~  财神爷~  ▁allegation 她都 恩师  值班人员  最珍贵的  industrial 八成 跋涉  室内外 Throw  扎扎实 器皿 快消 百倍  期待你的  ▁dialing POP  正处在  源远流 ▁NOR 管齐  而感到  ▁congratulate 霸主  bernetes  ▁sponsoring  可能造成  ▁washer  ographically 大促 逆天  ▁Committees  ▁supermarkets 玩玩  intestinal  ▁redundant 出过 孜孜 淫秽 渺小  依然在  ▁softer  FORMATION wow  看到他 STAR  回族自治区  的转变  自查自 aldo  荣华富贵 stall  ▁showdown  ▁Destination 不香  为别人  ▁shocks  ▁abdominal 喝汤 金桥 飞快  ▁ле  农业发展 tober  ▁Separ  ▁Paperback 披着 母羊 脂质  史以来 ▁DON  全部都是 Helen  ▁Joanna 加利 才被 掌中  他们家  救死扶伤 有求  正准备 ▁Ips  的影响下  搬进阳光房 大修  专业科目 捷克 ODY  兴安盟  该小区  改革试点  明显下降 azaar 排长 河路 飞鸟 abo  语文老师  ▁Tues  ▁ascertain 凯特 豆子 赢在  却总是  可能与  ▁thieves  ▁aggravated 以小 全息 豆粕 ▁Lod  　//　  ▁sudo  缴费基数  ▁pumped  ▁Cruises  年全省  最严重  ▁melan 西西  中心等  杀虫剂  ▁reefs  ▁Farrell  ▁concluding 冷酷  附答案  饱满的  托幼机构 ovolta  housing  chanical  ▁Reception “[  周口市“  临床经验“ 呜呜Ó  异性的Ó  生产能力ē  交通警察支队ē  ▁Brendaœ 迷离œ MHzƓ  病死率Ɠ  也很简单Ǔ  等各方面Ǔ  释放出ȓ  业务范围ȓ  换届工作ɓ  ▁Clarenceɓ  ▁Motorolaʓ 借了ʓ  和幸福˓  工会主席˓  健健康康̓  斗罗大陆̓ flash͓  ▁Crimson͓  年招聘Γ ▁HakΓ ADSϓ  ▁ariaϓ  ▁DiscriminationГ  ▁OuterГ  Republicanѓ  ▁foreclosureѓ 沙县ғ  一切都会ғ  央视新闻客户端ӓ 武大ӓ  也很少ԓ  审判员ԓ  牛羊肉Փ  能吃苦Փ  forming֓  宁夏回族自治区֓ 凌乱ד  可以提前ד  开学典礼ؓ ( !不对所包含内容的准确性ؓ 周长ٓ 村屯ٓ  道路通行ړ  Attorneyړ 清心ۓ 父女ۓ  开学了ܓ  神经外科ܓ  游乐园ݓ  行驶时ݓ  ▁covidޓ  ▁immortalޓ 新密ߓ  不让人ߓ  记录下来  ▁ANGEL 推手 跟人 齐抓  新四军  记不住 erty  不容忽视  ▁Gregg 八里 轻则 intent  中小学幼儿园  ▁extremists 拘禁 瓦斯  持续改善 海西 限高 ▁SAM  社区团购  积极引导 Polit Become  generate 唐宋  brother  普通话水平测试 面罩  大面积的  ▁composing ropy  ▁Wend  kefeller  ▁novelty KC 望而  大力实施  计划停电  Conference  ▁Connections 是先 肉牛 该文  的边缘  在线咨询 ># 安岭  以一种 埃尔 要花 ULD  是否为  小时内删除  ▁exhilar  ▁desserts  入眼帘  大学等  是企业  ▁Cary  规范佩戴口罩 个镇 陈建 Coin  学会理财 FIELD etrics  ▁Colombian 不决 得利 绳索 迁就  大幅增加  高层建筑  是最重要的  consumer 固本 甲骨  itudinal 温顺  高中部 nych  传媒学院  ▁Kobe  海南自由贸易港 漫游 铺开  上年度  群众利益 Campus  ▁forecasting 唇齿 文斌 Fed ▁ml  湿气重  ▁uncond 不因 绝境  的主流  ▁saints  供给侧结构性改革 是高 杆子 造船  从重处罚  ▁arist 咸菜  和投资  建设者  的运营  重阳节  什么原因  住院患者  促销活动  ▁bilingual *( 亲笔 耐看 视唱  并提交  给老师 enia  ▁Memoir  ▁landsl 美编  要突出 ISBN  ▁Rend  ▁electorate! )|　!  心理问题"  昼夜温差" 万份#  外界的#  ▁fiery$  ▁linger$ 威士% 它对% 点下&  可容纳&  养生保健' 宿松'  饱腹感(  拒不执行(  在海外)  人生哲理)  ▁Sung*  ▁Cognitive* 未到+ 神采+ 维奇, vac,  ▁Bolt-  ▁corridors- 是让. 郎平.  ▁Jiang/ 会战/  有一只0  火热的0  视情况1 amus1  ▁Hanson2  ▁cervical2 扬扬3  知乎上3 ITAL4  ▁inspected4 存入5 干为5 铮铮6 降噪6  依次为7  ▁inequalities7  ▁instrumentation8 城内8  将更加9  ▁Ramirez9 展品:  做一名:  奶茶店;  战役中;  ▁Begins<  ▁Undergraduate<  少之又= umba=  并不需要> ellas>  ▁baff?  ▁Merry?  ▁dolls@ 余种@ 环形A  俯卧撑A  而不为B  蒙古语B  的观念C  足球场C  那些年D  ▁packetsD + $治理体系和治理能力现代化E 秀区E ланF  主席团F  他可以G  的力度G  的题目H  选择合适的H 美媒I  不考虑I onistJ  ▁RefugeesJ  ▁giganticK  岁孩子K  我一个L  网络赌博L  ▁astoundingM 武当M ▁FavN  安全可靠N  ▁ShaunO  ▁ItaliaO  ▁CelebratesP 员外P  大扫除Q  请放心Q  ▁LunarR SSLR  下发的S  控制和S  煤化工T ▁ZIPT  ▁EstatesU 产自U 张明V  最高法V  都特别W  ▁riffW  CulturalX  ▁snapsX 五常Y 满园Y 短途Z 脚掌Z  好多人[ 口算[  心想事成\ 卷发\  新赛季]  是有意思]  ▁inventive^  舒服的^  imeters_ 事通_ 满眼` 腾冲`  再也不用担心a ńska  净资产b wendb 四口c 墓碑c 鼓劲d  精挑细选d  ▁hamme  应该有e  最后再f  目的就是f 们也g 已到g 措手h 轮到h 这枚i  李清照i  开展调查j  疾控机构j  不久的将来k  ▁Galwayk 湖边l igual  一如既往地m  ▁Comcastm 先说n 声色n 洲际o 脾经o  由政府p  让人感觉p ⊙　q 计提q kongr ▁ginr 成因s 溺爱s  两年内t  打印机t  社会团体u lehemu  ▁Figuresv 敬礼v  婚姻法w  ▁Kimberlyw 军政x 这则x  不一会儿y  ▁Deafy YLz 潜入z 花瓶{  就会在{  每天有|  积极做好| 土城} 连胜}  ▁fertilizer~  ▁decentralized~ 下蹲 者是  二选一  包吃包住 broad  工作的意见  ▁Emeritus  ▁Appalachian  ▁psychedelic 一做  小客车  请自行 口粮 楹联  名单中  热爱的  用手指 Chan  采取行动  ▁geopolitical 乘以  ▁malt Oregon  采取有效措施 营镇  调整到  ▁petitions 车费  眼耳鼻  肌肉的  招生对象 Laure Ground  ▁Clyde  ▁preparedness 却因 原作 坑洼 小牛  提升到 coins  ▁Mush JPG  亿港元  名师工作室 Expert 地从 真金  示范县  发生改变  投资基金 耿耿  丧失了  正常运转 三藐 倾城 hoff  ▁fung 肥大 ipzig 件的 其人  民主主义  进行管理  ▁owes 和最 追击 送了 mmm  会计学  晋江市 贸然 hog  能找到  免征增值税  ▁Sinai  ▁exhibiting  ▁Examination 全胜  占有率  如美人  研究工作 也因 新桥 狗血 ▁Uns  重点检查 exist  ▁Riot ▁LOT  对于这个  ranking  Calendar 想问 老三  人口老龄化 IJ 客机  避免出现  请耐心等待  ▁betrayal `)  新时期  这是他  由用人单位 委婉 的哦  文学院  买衣服  是经过  结核病  各位小伙伴 ucking  ▁Boone  industry 雨滴 HAM Redu  ▁automobiles  不能有  即可免费 抓取 秋月  集中统一  ▁Keen Yellow  按照有关规定 在车  我身边  ▁Foss  ▁Feedback  ▁nightclub 屹立 西边  一顿饭” lite”  无法确认Ô  ▁WHATÔ 泡菜Ĕ  ▁“…Ĕ  ▁GunnŔ  ▁MadagascarŔ ▁ Ɣ 逛逛Ɣ  关于在ǔ  系统将ǔ  ▁LicensingȔ 好酒Ȕ 徽州ɔ  产业升级ɔ Ratedʔ  ▁Vanityʔ  ▁Randolph˔  ▁premieres˔  ▁respectable̔ 火遍̔  但仍然͔  ▁Meets͔  ▁flattenΔ  ▁impersonΔ  ▁readableϔ 画上ϔ  三藐三Д  скихД 已与є 方志є UCKҔ  就比如Ҕ  有多好Ӕ  报考资格Ӕ  ▁SindԔ azeeraԔ  ▁flawlessՔ  ▁reversalՔ 建投֔ Jake֔ Tribה  投资集团有限公司ה 慎终ؔ  商家的ؔ  经营户ٔ  ▁bomberٔ  ▁nauseaڔ 暗淡ڔ 格雷۔  而造成۔ 黑河ܔ  努力地ܔ  能量的ݔ  邓州市ݔ  ▁Alessޔ  ▁interoperޔ  熟练操作ߔ  ▁Helpingߔ  ▁Petty 狠心 eche  全面胜利  屡见不鲜 Pretty  resolved 控油 特有  重点对  新闻广播  ▁gren  ▁bland 伤身 减重 易购 Clark ridor 关镇 荒废 udad  史无前例  ▁Distrib  ▁Polytechn 曲子  宣讲团  的信任 ayas ussi  ▁Continuing  ▁incorrectly  ▁prominently 幼苗 枫桥 焦躁 赶集  新型城镇化 依稀  医学院附属  Whenever  ▁Picasso 颜六 Diagn 所向 碧海  新进展  焦作市  estrian  ▁unveiling  ▁Experiences  七八糟 产于  关工委  在治疗  在这段 retch  ▁Honorary 求你 测速 纯银  赚到了  工作作风  精细化管理 双龙 所占 随喜  确立了  ▁yuan  还要求 ▁Vor  带来的影响 彬彬 竖起 翱翔  在市场上  报名截止  请第一时间  新冠肺炎防控  ▁shrinking 王健  雪中送 atown  ▁stamped 烧毁 âte  也不行 ▁rou IVERS  ▁Elton  ▁bonded 归结 最易  纯电动  上的便利  大幅下降  ▁Tusc  ▁startling 开市  最前线 fing  千呼万唤中  要深入学习贯彻  ▁demonstrators ▁Hai  ▁WORK 奇效  悠久的  超负荷  打开方式 仅是 液态  最终被  ская ▁Mao  Margaret Bs  只能选择  真的可以  ▁Damascus 金昌  人民满意  有很多种  ▁fintech 徽章 herst  ▁​​  全身的  比起来 ifth 冲刷 只限! 喷射!  加班工资"  第二部分"  ▁mundane# ào# 紫荆$  全家福$  约合人民币%  Forward%  饮酒后驾驶机动车&  ▁unidentified& 伤势' 白日' 英姿(  去感受(  一般而言)  和旅游局) terms*  的犯罪事实* 孤寡+ 残缺+ 烈火, 竞相,  以个人-  医师协会- woods.  的新陈代谢.  ▁Aussie/  ▁Raptors/  健康申报0  ▁Dahl0  真正意义上的1  Cambridge1 MN2 ▁rhe2  今日推荐3  ▁Construct3 远在4  也没关系4  国家二级5  贵人运5  提醒自己6  ▁Boots6  ▁Macau7  ▁Wilder7  ▁Sculpture8  Established8  ▁creatively9 招来9 溶于:  怎么也:  拨打电话;  比较严重; 人因< 八仙< 学类=  在短短=  眉山市> ▁Ket>  发展历程? ция? ncia@ ▁MAT@  columnsA 怀瑾A 总长B  冰激凌B  委副书记C banksC 比作D  ▁ingårD  websiteE 县乡E  的迹象F  健康证明F  ▁TremG  PartnerG  orderingH '”H 封贺I  冰箱里I  听故事J  生源地J  下载安装K  处副处长K  ▁CaldwellL ttoL  有些时候M  commentsM 专长N 为官N 久安O 惨了O 挫败P  和义务P  摘下口罩Q quakeQ  ▁ArbitrationR  ▁TransparencyR  产品经理S  人生没有S  近几年来T  atologyT  ▁GeoffreyU  ▁occupantsU 流光V  相信你V  的时候要W  ▁BrideW  ▁estimationX 横穿X declY  项目建成后Y  ▁LaboratoriesZ 孤岛Z 界首[ 躯干[ oused\  ▁Brewers\ 多巴] 拌料]  往常一样^ itoes^ 太上_ 李焕_ 警民` 说走` 靖江a  性激素a  ▁Foundingb  ▁keyboardsb 宋江c telec  ▁scamsd dupd Todde  ▁Kensingtone 若要f  高温津贴f  ▁Stokeg 汇流g  全部为h  如何快速h  工艺美术i 长着i ▁fsj  总收入j  真实不虚k elopek  ▁Paidl  ▁helperl 生食m  有机物m  最好的办法n  ▁bladdern  ▁leakingo 悬空o 投喂p 日早p 有型q 检票q 羊绒r  在线网r  你还可以s  玉米价格s  应聘人员的t 水箱t 火上u  服务大厅u  衡水中学v  ▁scrollingv 涌动w  快车道w 纷争x  犹太人x  自治州y  及其制品y  目前没有z 格言z 症的{ 草鱼{ zhan|  泪流满面|  ▁Geological}  接触者}  不得而知~  或者直接~  ▁Metall  ▁dynamically  ▁manipulated  这期间 何在  深入交流  ▁Sloan 北洋 换挡 等为  位置的  的高校  ▁collisions 大牛  中华文明  排查工作  ▁pioneered  建筑和  第七次  倾家荡产 unged  ▁RELE  县人民检察院  ▁separates 淮滨 说事  网络空间 excel  ▁Grad  理财第一步  ▁misery  Democratic  整改工作  知行合一  ▁Spike 影帝 谈笑  戳下方  碎片化  国家公园  ▁wisely  ▁Deloitte 轮岗  ▁minimalist 柔美  为帮助  支队长  ▁abst 股神 话下 iduc uned  ▁pund 万次  买一送一  没有人能  ▁Lift  timeout  ▁disproportionately 怒了  和她的  夯实基础  ▁Lilly  ▁fantas MPA  试试看 改报 CLE  发布时间  才能得到  ▁жи  ▁levy  ▁Reyes  Russell  ▁Corinth  ▁pathology 边际 ahar near  的毕业生 Tyler  ▁REVIEW % 这样您就可以免费收到 唯物 岛内  这一系列  ▁blaze  ▁Rohing 曲艺 老生 при  和身体  黑天鹅  八十年代  天气过程 Phase  工作和生活 玄机  由内而 ▁Rae  ▁Docker 狭隘 驻点 ("%  制造有限公司 很受 资阳  二手房均价 Expect  ▁Hospice 时钟  integration  双胜利  自然地 ▁SAF  开开心心 rivile  schemas 小果  及专业 ▁DAM  销售经理  ▁pint  ▁knots  并不算  天气情况  来表达  第四条  政务服务平台  ▁Humans  ▁internships 颇多  写作文  送温暖  项目是•  ▁refunds•  的实施Õ 　//Õ Boundĕ  再上新台阶ĕ  ▁occupiesŕ 田村ŕ loseƕ Samuelƕ  ▁outlawǕ  哈尔滨工业大学Ǖ  ▁perennialȕ 高邮ȕ  ▁connɕ testedɕ  ▁aromaʕ  ▁Hundredsʕ 从无˕ 只吃˕ 我更̕ 所要̕ 霸占͕  工程系͕ 乐福Ε ushimaΕ  市公安局交警支队ϕ " 从玛拉沁信息网开始ϕ 何炅Е  显而易见Е  来找我ѕ  领事馆ѕ Iraqҕ  ▁Moralesҕ  第三人ӕ  ▁Zeldaӕ  ▁Optionalԕ  ▁counselorsԕ 传道Օ  运输车Օ inky֕  ▁auditorium֕ 实情ו 曼谷ו  一小区ؕ  将全部ؕ  精湛的ٕ 无耻ٕ slowڕ  每天上午ڕ  向社会公开ە  ▁planetaryە Excܕ  和女儿ܕ  在前面ݕ  束手无ݕ  ▁doomޕ 头颅ޕ  在设计ߕ  问题了ߕ ocha jandro  现就有关事项  ▁geological 难忍  觉得你 Truth  ▁cabins 细数  个人微信 Andrea NW 肉片 Cars  他人财物  Bitcoin 天池 放送  腾腾的  让你们  ▁Fresno  ▁leased  ▁playback 再发  岁人群  ▁loung 更美  的心脏 Shift  业主们  关键期  种情况  ▁Schwe  ▁Handle  ▁unseen 劳力 大金 太累  开设赌场  都是一样的  ▁clocks 微弱 象山  地藏菩萨  尼日利亚  ▁rang  ▁blinds 借呗 拔牙  本通知  三世诸佛  幸福快乐 mostly TOR ▁Kob  第一部分  ▁satir RU 得力  北京市朝阳区  ▁Wilhelm Wu  别的花  双方在  国产车  大型企业  Detroit  compile 抵消 良方 长河  激活码  知足常乐 igers  ▁Liberals  刻骨铭  就能轻松  教育改革  地区生产总值  uctions  ▁Underwood 折后 被骂  黄石市  表现形式  这个机会  有什么样的 干支  放暑假  的痕迹  认认真  新媒体平台  ▁Profiles  IOException  ▁Journalist 密鼓  夏日的 复诊 APA  进去了 道远  类物质  全国包邮  公共利益 uggle  ▁adore  Teaching  ▁intrinsic 新秀 驻扎  模拟题  火灾隐患  contents 三倍 线段  因果关系 orden  anticipated 御景 手稿  余额宝  和价值  三分之二  专业老师  可疑症状  九年一贯制  内容未经核实 幼虫  颗粒物  ▁Vish 字词!  她还是!  栩栩如"  冲洗干净"  而且还能# Barry#  ▁Signed$ Scan$  事故的发生%  duration%  ▁Infectious&  ▁directories& 嘉陵'  交强险'  十二届(  笔记本电脑( 武将) ▁Kul)  ▁Attempt* 忧心*  被淘汰+  一起成长+  热烈祝贺, 变电, 提拉- 田家- 通宝. 阿坝. nda/  四个月/  花青素0  但是随着0  ▁woes1  ▁auctions1 退学2  赚不到2  争取到了3  ▁woodland3 十位4 讯问4  值得的5  如何通过5  ▁formatting6  ▁principals6  看完后7 ALTH7 Barn8 Hour8  无论何时9  ▁Howe9 套现: 火险:  锣密鼓; udding; 摆件< iek<  地上的=  相信在= atom>  ▁Reese>  ▁Procedures? 开战? 滚烫@  一定要及时@  更多的选择A  ▁HostingA  ▁KnoxvilleB EIB 本溪C 紫薯C  乡村建设D  立即停止D  孝感市E  设计时速E  和部分F ▁WAYF  一起做G  副科长G  点进行H  知道吗H rissI 和白I ▁VWJ  了一层J  报名地址K  ▁HavanaK  ▁obstructionL 鹤岗L  可领取M  伸着懒腰M  ▁physiologicalN 丽娜N 气壮O  世界中O  活动结束后P 更迭P  要求其Q  也是相当Q 封路R 汉朝R 退让S ▁psS apeakeT 时为T 树苗U beltU  ▁funnV ArgentV  ▁crowdfundingW ▁ČW  ▁HochX 心惊X ístY  最高法院Y  再继续Z  爆发了Z  一般都会[  考研英语[  ▁fries\  ▁valleys\ " 中央军委主席习近平]  ▁devastation]  无人问^  幸运数字^  最新内容了_  idences_  ▁Amelia` üh` 应于a 诵经a  圆明园b  的话说b ractorc  ▁similarityc 不露d 松花d 较慢e  吃完饭e  开考比例f reaticf ▁VTg feetg  ▁repoh Deputyh 也大i 同享i 韩剧j  检查时j  的性能k ▁cssk 邮储l cdotl ▁Raom Metalm  ▁niecen  ▁Troublen AQo 封禁o 肺腑p  皮肤科p  报名网址q 配额q $$\r  应该说r  明显增加s 出库s maidt  ▁peelt  ▁undercoveru  ▁Predictionsu 命门v 壮族v  ▁reptw  ▁mysqlw  rictionsx  基本是x  对一个y  ▁Jaysy  ▁Chattz 东边z  持久的{  ▁Lite{  ▁tasted| yh|  常规的}  赚大钱}  工程进度~  两个一百年~  朋友说的话  ▁Hayden  Introducing  ▁condominium  ▁indispensable 扬言  日用品  兜兜转转  效果显著  文明城市创建  iterator  WriteLine 车贷  临时性  是人的 嘉善 栽植  而中国  论诗书  ▁Samar  你有没有想过 热锅 莱克  却是一 ibel  ▁scars  ▁Audrey  盛开的  variable 果敢 正茂 苦读  发现该 Faith  为了让大家  ▁Wichita  ▁layouts 入馆 kid 圆滑 鼎盛 ▁Wak  ▁Porto  caption  ▁hypertension 涡阳 环环 过道 psi  措手不及 Sense  Products 王明  每天在  毕节市 URES  持权属证明 凡尔 摇身 米面 iert  上不封顶  新闻爆料 hetti  每一个孩子  ▁Loved  ▁discrep  ▁fortress 分在 月刊  生活区  的知识点  ▁sterling 九寨  ▁nanop  事业单位考试 不认  漫论诗书  illation zą 卡路 毕生 父辈  对患者  的美感  未开放时 Joshua 四时 尚且  汽车品牌  ▁occupations 事前 公孙 降准  可采取  大人的 ▁MGM  ▁Cells 不择 和面 林的  大兴安岭  ▁glove  及服务  命中率  阿斯利康 两省 长篇  科目三  睡觉的时候  answered  Recommended 我刚  出售价格  Powered  ▁Rooney Coff  洁白的花瓣  会非常  很多都是 书评 窝里  兴趣班  在实践中  科技工作者  ▁balloons  的好成绩 isset  ▁mindfulness 半挂 走低 鸭肉  无误后  ▁Feet chairs–  cluster–  ▁FrozenÖ  ▁convergenceÖ  沐浴露Ė  活动详情Ė umnusŖ  ▁fellowsŖ 乙方Ɩ  大约在Ɩ  ▁Cromǖ  ▁Seafǖ  教育主管部门Ȗ  ▁defeatsȖ  Documentsɖ  ▁Websitesɖ  大儿子ʖ  脖子上ʖ  稳中求进˖  ▁locksmith˖  ▁discriminatory̖ 会为̖ 踏步͖ 靳东͖  对全国Ζ AvoidΖ cancerϖ ermaidϖ  ▁achievesЖ 长长Ж  情有独钟і  ▁Roryі  ▁SnakeҖ 圈里Җ  自己的生命Ӗ  侵犯到您的权益Ӗ  以涉嫌Ԗ  灰色的Ԗ 阻挠Ֆ  为我国Ֆ ▁Joa֖ onders֖  ▁hackerז  ViewByIdז  ▁synagogueؖ  心理上ؖ  较上年ٖ  水涨船高ٖ  ▁pantryږ  ▁satireږ  ▁portraysۖ 封号ۖ  有你的ܖ  党务工作者ܖ  Vietnamݖ  ▁carvingݖ  王俊凯ޖ IOUSޖ  ▁Ryderߖ  一卡通ߖ  为落实 prints  Producer  ▁Interpret 相扣 香山 ▁SX  转眼间  图文讲解  在微博上  ▁Receives  ▁Happiness  岳云鹏  全国第一 包庇 落了 闲情  全校师生  加拿大的  紧锣密鼓  ▁Concern 器物 对公 开路 牵连 ▁fry  千篇一律  首都北京 ONDON 前述 爆表  总感觉  目前该  瓜果佐酒  省吃俭用  ▁Cust vester  ▁evapor ionic  Studies 和合 多边  每学期  ▁Kant  Important  ▁Headlines  的另一  地展现它的  Wisconsin 而活 nah  公安工作  持续加大  ▁Gerry  ▁mushroom 双峰 月色  这家店  更改了  购买此 Einzel 常被 祁连  日报告  ▁Peggy  ▁Profit 争抢 刮目 采血  历久弥  和规范  阳光城 ▁DSL  在职人员  朋友圈里  是有意思的事  一条龙  用一位作家  ▁fauna  醉里挑灯看花  ▁commissioning 新作 诅咒  努力成为  招聘考试  ▁shortcomings 有才 静的  等待羞涩  等待羞涩的美人  ▁dragging 冰块 家店 歙县  的饮食  向外舒展  向公安机关  在在千呼万唤中 " 一点一点地展现它的 咖喱  于未然  财政资金  未开放时绛 ( !一点一点地展现它的秀色 敬仰 瓶盖  高考志愿填报  ▁adversity % 用一位作家朋友说的话  ▁subsection  蜂王胎  根据个人 ieler  ▁odor  datetime  Constants conv ▁PLC  你在看  大喇叭  另外一种  国家承认  评分标准 hammer  aduation 书名  必然是 两百 烧香 隋唐  发电机!  戏剧学院!  ▁Bankruptcy" 园路" 推到# 时将# 颐和$  全市上下$  故意杀人罪% waters%  的文件&  有色金属& liber'  ▁casts' 听后( 方阵( 自住) 跑的) ▁QR* arro* ▁Chu+  ▁Scan+  日放假调休,  ▁desks,  Meeting-  ▁inclus-  financial. 开售. 而未/ lioma/  Podcast0  进步的0  护理人员1  ▁rewrite1 即用2 母语2 首秀3  可达到3  更全面4  基本没有4  无缘无故5  ▁Serena5 兼治6 Sara6 Albert7  ovelovel7  ▁Fairfax8 芒种8  直接从9  ▁McCoy9  点击下方空白区域:  ▁variability: 丫头; 墙角; 阿耨< nard< 请来= Esc=  ▁unveil>  Employee>  ▁Prosecutor? 私域?  构建了@  入党积极分子@  知识星球A  eniableA  ▁hallwayB  同一天B 姗姗C 它与C 快船D 蓝田D ymeE  把事情E maresF  opathicF  ▁antitrustG wjG 今起H 几招H  富含维生素I  制片人I  及部分J  慰问金J  劳斯莱斯K 代收K 等新L  精品课程L gardeM  ▁rumorM 一职N 个市N 前车O 是啥O TRYP  给家里P  投资者的Q  ▁baldQ 可真R 大碍R 罗永S  大闸蟹S  进入公众号T sexualT 惨淡U 火种U EddV  一年前V ruffW  就是一场W 摆渡X  反映的X  的一所Y  监管责任Y  ▁pledgesZ  ▁circulatedZ 低洼[ 踩着[ 辛亥\ 馆内\  多少分] titled] 善事^ 归家^  研究与_  中央公园_  项目地址`  开放的样子`  ▁Nikkia  Atlantaa  ▁Instructorb 崭新b 故名c evoc  除了在d Fixedd 太冲e  型糖尿病e  心血管病f  engeancef  ▁ethanolg 交感g 吐鲁h  无名指h  监察委员会i  esthetici  ▁trumpetj  ▁pitchersj JXk  相呼应k  ▁laval  周岁及以下l  chargingm  ▁disseminationm 赛的n  公共管理n  ▁Philipso 一语o 雨的p  动员会p  Walkingq  ▁paragraphsq 古韵r 山南r  五颜六s  因材施教s duringt 好货t 甩锅u 补漏u ($_v RAMv  变异病毒w  安全生产月w  房地产行业x  就真的x  我以前y  ▁Kievy  ▁mismz 大宝z 脆皮{ 附子{  电影的|  建立完善|  研制的}  透明度}  旗帜鲜明~  ▁maneuver~ 放量  ▁Jerseys 体悟 寻味  了一会 Flags allah 增资  越容易 ▁jan Claim 个数 ▁VID  应收账款 糖浆  ▁Friendship 月老 绵羊  视频截图 鱼片 ahead  ▁Arcade  ▁Titanic  使领馆  各省级  另外还有  ▁reflex  ▁assaults 可加 如厕 姜末  三角洲  为考生  能够做到 之巅 娴熟 水到  ▁Ducks  ▁neatly  但它开放的样子  ▁brilliance  一想到 trace  ▁bart  ▁worm  优质教育资源  ▁anchored 后厨 睁眼 空姐 beg  虽然柔情似水  infandel  pressure  ▁diagnostics Wy ANGE  ▁Turks 电诈  之功效  大庆市  有一句话  村委会主任 ▁QC  大家长 ▁Clo  恢复营业  如果有一天 merged 壮阳 还债 тов  但是当  发消息  确认为  都是你  ▁Bust  ▁MacDonald 八宝 妙用  水环境  莫名的  ▁serviced  ▁specialties 云峰  我想要的  长江经济带  ▁vacancy  ▁amounted 五色  一两个  在讲话  是很有  送上了 rers  一点点向外舒展  如美人伸着懒腰  operator  ▁оп Mask  的关键时刻  专业技术职务  ▁insisting  ▁reminders indi  重大意义  ▁Gibbs  ▁Guyana  ▁Mosque  Distance 拳击  去追求她  breaker  ▁UNICEF  ▁Resilience 单笔 此番 维也  感党恩  项城都市网  过的人  ▁Pend  ▁Ninth  ▁stakeholder :) 字里 武陵 炮制  必要性  犯罪记录  ▁Bhutan  ▁Ginger  ▁edible 脸红  万千瓦—  其实我们—  没有关系× ( !港澳台地区通报确诊病例× 教培ė 跑出ė  archingŗ  ▁Includeŗ  张国荣Ɨ  最关心的Ɨ  ▁DortǗ  死记硬背Ǘ  是两个ȗ  社保费ȗ  ▁saladsɗ 魄力ɗ 魏晋ʗ  的待遇ʗ  防疫一线˗  ▁spun˗  一般纳税人̗ 取自̗ 和谁͗ 顽瘴͗ KidΗ ▁HouΗ  激励机制ϗ  分享自己的ϗ  ▁AbujaЗ 4 -累计收到港澳台地区通报确诊病例З  却不知ї  nutritionї  ▁profilingҗ 转而җ  吸引人ӗ  锁定了ӗ  赶往现场ԗ  多罗三藐三ԗ  如果你想追՗  ▁Boise՗  ▁sharpen֗ 原石֗ 多到ח 就先ח 笔顺ؗ  接报后ؗ cakesٗ  ▁restraintٗ 极光ڗ  多元的ڗ  意志力ۗ  揭示了ۗ  关节疼痛ܗ 香草ܗ rekݗ  的眼光ݗ ▁Gugޗ  石油化工ޗ  中华医学会ߗ cationߗ 氢气 高科 青天  和指导  其乐融融  ▁sunrise  报考点  Convert  ▁Ventura  ▁fashioned  ▁Absolutely  ▁commentators 发际 四千 苦尽  中东部  批准后  中国内地 单于 渗入  Historic  ▁applause  ▁trophies NUM  的天然  从天而降  ▁unlocked  ▁safeguards 水口 wik  和应用  百香果  传统工艺  protocol 分心  歌唱家  系统等 urger  ▁Centennial 同质 国税 大任  geoning  ▁exposes  细菌性  中的作用  给我们带来  ▁Romero 交底 酸胀  店里的  有特殊  驾驶的  笔试内容 Timer Carbon  ▁Bulgarian Ja دى 姐夫  工作的同时 护林  成功地  的举动  为消费者  ▁Becky  ▁recol  ▁infused 超度  小窍门  最合适  ▁faç  违法事实  ▁Pref 诺言  ▁customary 江景  少走弯路  ▁Harriet  ▁Shapiro 十有  别错过  在这片 ▁Sap  推荐指数  ▁tipped 万种 之梦 双喜 海螺  流水线  我有一个 Shape 并可 格斗 满族  ▁juxt  ▁dwind 缺额  拒不配合 暗暗  云课堂  畜牧业 LIST  我们都要  能够达到  行政学院  Residents  ▁triggering Cancel  ▁Dustin 鱼缸  的调查  这一条  perture  ▁graphs  EventListener dos ssl  ▁toggle  ▁Steering 地暖 打乱  是这么 ▁GHz  特点就是  ▁Buchanan  的自我 保家 弱化 西华!  中成长!  自身防护" 厚积"  所不及的气度#  失信被执行人名单# ▁fmt$ ▁nos$  ▁Poems% 从其% 可看& 防指&  生活补贴'  象小家碧玉'  ▁Coaches( 刀片(  顺利的)  心中有数)  ▁cele*  紫色的花苞*  它有着别的花+  ▁Accordingly+  ▁Intelligent, + $它有着别的花所不及的气度,  和党员- Army- ▁HOW.  要求各地.  ▁aides/ % 紫色的花苞象小家碧玉/ 散装0 牧羊0  ▁Adds1  ▁Bauer1  ▁accumulate2 4 -未开放时绛紫色的花苞象小家碧玉2 超模3  全体学生3  胡思乱想4 整日4  合格线5  日早晨5 andes6 Depend6  Already7 山火7 陇西8  上下学8  均来自9 Pear9  ▁Cors:  ▁Toxic:  Defense; ▁ka;  在中医<  造成严重<  ▁grouped= 万变= 降了>  权力的>  实地查看?  和标准?  过山车@  ▁bait@  ▁doorstepA  ▁dispositionA 麦田B  很关键B wxcbC  控股有限公司C '}D 区直D  日宣布E XboxE  教育发展F  通过线上F hemerG  ampiresG  ScotlandH  ▁boatingH  ▁comfortingI 销路I  喜欢看J  时不要J  如何判断K  晶莹剔透K  进一步深化L  ▁CheshireL ▁ACAM  ▁LiverM  保持安全社交距离N 仁心N 农用O  更可怕的是O 大浪P 颍上P  坚持了Q NEXTQ  朝气蓬勃R  电子税务局R  oughtonS  ▁prowessS 按住T 难看T  在哪儿U 起止U  ▁KaplanV  ▁ContentsV 赴美W  marriedW  ▁MondaysX  ▁SheldonX 蛋液Y 首钢Y sdkZ  又不能Z  ▁curric[  并不代表本公众号[  ▁impartial\  ▁BlackBerry\ ге] 大丰] 颐养^  大项目^  等规定_ 短款_  可以到`  雪中送炭`  ▁autobiographya 夜查a 惭愧b  等制度b  简单易c  一定要把c 握着d  退一步d iesele storese  ▁weightedf 吃香f 患得g  的气氛g  道路的h  ▁Westerh  ▁Cartooni gosi  不指定j  应避免j 匀称k 起家k FUNl  的农民l  科技局m  公共空间m 血丝n ▁GNUn  尽心尽力o  ▁warehouseso 华美p 瓦房p  威海市q risisq  进一步明确r  ▁typeofr  ▁broadcasterss  与学校s  和加强t  是导致t Periodu  网络科技有限公司u TWv 别名v 年将w uevaw ▁Cedx  滥用职权x  ▁Dominiony 绝无y  各学院z  可以继续z  ▁Bran{  ▁Taipei{  ▁diagrams|  ▁interpreter| bp} 别离} 闹得~  第一志愿~  ▁Gale  ▁smashed 亩地 更会  为居民  慰问品  体会到了  示范项目  逐渐成为  ▁Turbo  ▁Holden 爽滑  李大钊  椅子上  ▁hobbies  ▁embodied Ze 几遍 盆土 禁渔 风热 APPY rawl  干货满满 Ay 加长  安庆市 ▁cuc  名医护人员  ▁goats  ▁también Lov  ▁ensuing 常言 而止 钱款  到如今  嵌入式  视频的  运用了  充满信心 乳液  和林格尔  ▁Olsen  ▁electromagnetic  ▁abbrev  ▁wildfire  ▁customizable 新宁 辣妈  加盖公章  ▁Gins  ▁Moran " 教育部留学服务中心 荒野  为你们 Sens  ▁plausible  一步到位  ▁Tube 功名 客场  几百万 CHAR  共青团员  那么这个  长江大桥  除了这些  ▁Photographs truth  这样的问题 一概 虽好 iage  一个女孩的话 师市 破旧  乱七八糟  专家组成员 ’‘ 前移 江路  ▁ва  是否是  涌泉穴  温暖和  错别字  另一个人  平顶山市  江北新区  食品添加剂  阿耨多罗三藐三 执意 缘起  局部地区  ▁wont 特务 转租  去参加  navList ( !如果你想追一个女孩的话 有据  有气质  温开水  确认后 Hell  执行董事  第十二届  长按保存 quila  再放入  好评如潮 年入 疏解  一个脚印  今年年初  ▁raff  教育类  更好看 ISM hift iale Va 从宽 疑点 筹资 蠢蠢  结束时间 Symbol 洞见  ▁Minds 关了 尝鲜  发自内心的  ▁Proposal˜ isol˜  公司名称Ø  ▁reliablyØ  也不愿Ę  ▁SylviaĘ 补税Ř  和广大Ř ▁PonƘ  CollaborƘ  ▁apprenticeshipǘ YPǘ 吹牛Ș 外力Ș 扶梯ɘ  ▁oligɘ AMCʘ Insertʘ  ▁simulated˘  ▁discontinued˘ 湿滑̘ GLO̘  呈现在͘  Society͘ 武部Θ  担任过Θ  的中药Ϙ  对他们的Ϙ  蚂蚁集团И  让我们在И  Jeffreyј  ▁fluctuationsј 食神Ҙ MurrayҘ  ▁measlesӘ  ▁pollutantsӘ  其它地区Ԙ  ▁DoomԘ  ▁Rubio՘  Interesting՘ 再由֘ 本意֘ ▁'#ט ▁YAט  患者中ؘ  金榜题名ؘ  ▁quartz٘  ▁richness٘  ▁Huffingtonژ  ▁illuminatedژ 六点ۘ  示范园ۘ  自己都ܘ  政治素质ܘ  主要表现为ݘ  释迦牟尼佛ݘ DVޘ 忠心ޘ 肺经ߘ 落得ߘ  非机动车道  而无法 ()))  养成良好 国力 收储 IENT  可以免费  方案设计  vertical  ▁translating  解难题 不就 只说  全市人民 京师 赤诚  表态发言  ▁outlining 万卷  中含有的  或单位  及时清理  ▁Bands " 不要盲目的去追求她 抗衡 Export 一空  无异常 ▁Bau  快人一步  顽瘴痼疾  的重要论述  ▁Leaving 国乒 摊贩  在手上  陆续续 夺目  的同事 主创 是中  ▁granddaughter 几度 oran  眼花缭乱 刷了 园丁 梅西 难了  三阴交  团队在  很好看  接受过  大力宣传  ▁Subaru  教研员  Century  ▁improv 租车  应该就是  Neither  ▁Miracle  ▁downtime 岁数 清真  交给了 prev  哪有什么  微淼商学院  我们致力于  ▁Nicholson  品牌形象  岗位人员  招生章程  ▁Serious  ▁intensely  室内的 Refer  ▁stray  ▁melanch  晒出了 ▁mog uterte  即可免费领取 蝉联 Fel quoise  ▁Harmon 发过  成功抓获 ingles 清正 禀赋 钟声 马仕 误入 贴纸  报名和  ▁Brah 懒散  日公布 tone 而同 花岗  十八条 Yang  安居乐业  ▁breastfeeding 不破 延吉 都把 McC  对人类  还有各种  ▁Kyiv  ▁nested  ▁enclosure 上上 用功  占地面积约  Resolver  onential 宝地  锐意进取  ▁bottled  regulation 共育 沪深  思想道德  ▁dams!  可靠性或完整性!  ▁Michelin" ▁{};"  ▁Sapp#  occupied#  ▁Alibaba$  一张图片$  宏观经济%  ▁Fully% 博文& Fix&  可以与'  经过近'  ▁Dive(  ▁uptake( 鸣笛)  出行前)  ▁Packages* 利辛* 装潢+  创立了+  此前报道,  ▁pastry,  ▁caramel-  ▁seminal- ща. 尘世.  海贼王/  到自己的/  研究院院长0  ▁defer0  ▁popcorn1  ▁scandals1 养性2 反华2 试过3  形势严峻3  经济条件4  Harvard4  orescent5  那就要5  少之又少6  ▁BASIS6  Validation7  ▁familiarity7 zyp8  不充分8  可以看9  已经完全9 qa: 横行: 黄磊;  等食物;  门口的<  陆陆续续< 眼了= ▁DU=  的最终>  ▁semifinal> 后因?  高密度?  ▁Garner@  ▁Blogger@ 删掉A 弹琴A  和数据B  ▁SchnB ocytesC 滑县C 铁板D fffD  组成了E  所以这个E imicroF 发麻F 讲道G MutG  擦肩而过H 且未H 国美I xslI  最新通报J  ▁fuseJ  climateK  不开的K  即使你L  二线城市L  ▁WelchM  ▁cartridgeM 是没N  了一样N  蓄电池O  头等大事O isoftP  ▁lendsP  ▁compromisingQ hcQ  了一篇R  是进入R icomS  上海浦东S 炫富T HOTT  展览会U  转小火U 制胜V  着力点V  光盘行动W  ▁biotechnologyW 南大X  颜值高X  结直肠癌Y CollegY 典籍Z 风霜Z 魔兽[  被确诊[  ▁Para\  ▁unfolding\  落地见效]  课程设置] "—^  半决赛^  和问题_ usch_ ▁sap`  点击关注` ylenea  不负有心人a  正面的b  第一道b dochc  会议中心c  慈善总会d  这样一种d  Discoverye 余件e 能动f  ▁restrictingf  ▁legalizationg 不愈g 与子h 捷报h  号公告i  相亲网i  茶文化j  小数点后j šík 放款k 计分l  参加全国l 敷设m mRNAm  ▁vainn  ▁forestryn 患失o  将进入o congp  ▁Dresp  ▁creekq  definitionq 涂装r 湖里r 附表s  医院和s ildet  不文明行为t  也可以选择u  ▁shortenedu  商用车v IVERv  幼小衔接w  演讲比赛w  第一财经x  都会出现x mathbfy  ▁Rockiesy 浴缸z 荧幕z 高挑{  ▁cockpit{  一条路|  全封闭|  发车时间}  Lincoln} 竞品~ Luck~ ▁Dir 一局 葱段  Condition  ▁dependable  ▁constituted  ▁perseverance  在感情中  ▁precaution  事业心 Sadly  ▁scrutin 新都 童书  个小时的  数据来看 aille  第三节  ▁javascript 就跑 路费  可咨询  走得更  ▁outperform 东关  为了保障  居民医保 哭声 禹州  有益于  Validator  ▁Philanthrop :@"  为保护  完成情况  ▁Grac  ▁ferv  ▁Kitty 上乘  很辛苦 ▁Fay  如果你在  民族文化  ▁Sears 荆楚 过热 Aid araj って 下地 光子  ▁Pour 带电 荤素 麦冬  不能被  回想起  福布斯 trim 免检 enf  万粉丝  要加快  犯罪活动  重要提示 Terry hadoop 秦淮 MET  持续了  毫不放松  ▁Mellon  就业率  用这个  金刚经  在一起了  才是真正  持续发展 ignor  ▁mound  ▁confronting 开怀  积极与 Span  他们也会  ▁unatt  ▁weave 字眼 开馆 ▁LT  下坡路  都处于 ronym  ▁discrete 刺猬 ENV  以减少  特别大  昔阳网  全市各级  血红蛋白 PPORT  报到证 pole risy  人才市场  ▁PLAY  时使用  大多都是  ▁Maui 停停 北伐 华区 寻亲 心性 正餐  ▁frontman  ▁footballer 照看  危害性  年同期 bart  不顾一切  Pointer  Biography  渭南市 croft 担子 揭发 费县  今日话题  processing  ▁Consultants 探头 气功™  纪律处分条例™ 根深Ù ▁CGIÙ  免责申明ę  对于这些ę  ▁Emeraldř  ▁backlogř  第一波ƙ  香港的ƙ  对中国的Ǚ  沿海地区Ǚ  ▁Penguinsș  ▁Comparativeș 吴川ə  日期为ə coatʙ  联合执法ʙ  ▁setback˙  ▁nutritious˙  快递公司̙ Metro̙ 光临͙ 招教͙ 荣立Ι  孩子成长Ι  茁壮成长ϙ  ▁blurϙ  重要讲话和Й  健康生活方式Й 快到љ  中年人љ 镇西ҙ  下发了ҙ  数量和ә iddyә  主题班会ԙ سىԙ 却让ՙ  优势互补ՙ acier֙ 挨家֙ 祭拜י  想到的י rombؙ 不贵ؙ 事例ٙ  分别从ٙ  系统进行ڙ Modalڙ  也就是我们ۙ  ▁weedsۙ 混沌ܙ  频繁的ܙ ▁Riyݙ  五花八门ݙ  突然出现ޙ ▁"%ޙ  接二连ߙ  的分泌ߙ enthal 喝着  帮扶工作  难点问题  ▁Tracks 内线  的重量 Gram  万物复苏  优先录取  娱乐活动  广大市民朋友  ▁Brings  ▁magnesium 小丑  三方面  个省市  挂号费 Pray  ▁volt  佩戴好口罩  所吸引  考试费 Fish  ▁altering  的香味  市委党校  ▁interpreting 练琴 kou cctv ▁gor  网友们的  ▁Guam  安全工程师  ▁Damage 载货  的不利 ▁MOT  年度报告 Scene  助力乡村振兴 非同 oct  企业及  等特色  并提供相关  和高质量发展  sembles  ▁vacations 发痒 ▁LS  不包含  和鼓励  集中开展  ▁Levels  ▁Covenant >},  天河区 半程  适当地  ▁orch  ▁Vijay  面向社会公开  ▁Anthem  armaceutical 信服 沃土  值班主任  ▁Punch 　¥ 举重 做优 罗列 MEA  阿拉斯 ▁Phi Afghan  ▁Trace  寒假期间  ▁valves 体感  等到了  ▁extingu  得心应手 yellow  ▁Strauss 代之 影音  ▁Toni  ▁Venezuelan 开光 霞浦 alon  ▁ambigu  Statistics 挚爱  强奸罪  精神卫生  是什么原因  ▁Palin  ▁piles  ▁Maximum 算盘  州政府  讲话精神 Joint  ▁Foley  ▁cohesive  奋不顾  维护好 guns 包浆 lique  进一步推动  ▁simulate  responsive  ▁Initiatives 新锐  行为习惯 幻觉  朋友说 aysia 储量 大佛 承重 此起  ▁reactive 的法 院线 高淳!  奋斗者! fp" 帆船"  有时会#  结婚的#  国庆中秋$  ▁Jessie$ 早衰%  今日份%  观音菩萨&  ▁ceramics&  管理及'  ▁ranc'  大数据分析(  ▁undes( 在先) ▁hid)  ▁Seen* 两台* 金庸+  真心实+  金普新区, 两节, 地被-  成果转化-  ▁Seymour.  ▁coronary. 精工/ utm/  ▁importing0 逝去0 长汀1  攻坚行动1 Occup2  monitor2 三科3  失去的3 Vers4  ▁charcoal4 内设5 西乡5  生态农业6  也越来越多6  统计数据显示7  ▁Fulton7 城郊8 正风8 清欢9  我和我的祖国9 所涉: UFO:  患得患失;  违规收受;  ▁Rang< frames<  ▁Clarkson= 克隆= 拒收> 羊城>  束手无策?  ▁flute?  otherap@ 丈量@ 惠来A  却还是A  梦寐以求B  重点工程B 敢说C  受伤的C ▁NahD biasedD  ▁mergersE  的人性E  请致电F swingF  ▁EllieG  ▁sternG  ▁dancedH 蹲守H  疫情结束后I  目前还没有I  ▁bumpsJ  ▁homosexualityJ 前一K 泰顺K  会员卡L  ▁veggiesL  ▁culminationM 长垣M idadesN  ▁DartmouthN PuO  ▁RogueO  PremiumP = 6新型冠状病毒感染的肺炎疫情防控指挥部P anseQ  微信好友Q  出示健康码R  ▁FingerR  ▁lettuceS  ▁PhysicianS  水煎服T  ……”“T  ▁NanoU  ▁dividesU  角质层V  就业人员V  相互转告W  undefinedW 发心X 熟睡X  思想工作Y  ▁LaosY alternZ  将开展Z buff[  研究所所长[  Hundreds\ 差了\ 碎石]  正北方]  校园安全^ Rules^  ▁confession_ 图说_ 山间` 贺兰`  系统自动a  rollmenta  ▁workspaceb 谁更b 追查c sarc  建成了d  是整个d  ▁Dantee  ▁Objectse 神阙f  磁共振f  新闻频道g 潺潺g 道光h esih  人体吸收i iseasei 往里j 酒杯j 面世k  持卡人k  种类的l  并非如此l  ▁instalm  一辆车m  全运会n  等系列n labelso  ▁eliteso 后不p 实证p pasq ▁PPq  熔喷布r 论点r  不可及s  ▁downhills 哭笑t 穿孔t liquu  为人处事u  清华北大v actualv  Assemblyw 大龄w 隔三x ▁ccx  临时工y  显示器y zionez ▁Nirz Agric{ 石刻{  ▁Erit|  等方式进行|  avourite} 那张}  的诞生~ htub~  经济补偿 xg 任怨 sup  赞助商  转正后 Belg ▁Woo 体位 чен ▁sb  维也纳  ▁traff 几大 康康  みらい  苏维埃  无微不至  正式成立  ▁Vand  ▁Himself  ▁departing 头昏 电热  少有的  本人或  了一口气  ▁mystical 相拥  开拓者 ▁NEC 敛财 无序  农业部  的证明  公诉机关  问卷调查 female 去吧 山野 BMI  晚自习  应运而生 买药 彩绘 耒阳  并组织  搭配的  防控物资 acted  Favorite ▁tmp avers  iminary  ▁simmer  ▁clarification 一的 体味 率队  提醒您  新知识  管齐下 igon  省财政厅 Usage  自己的青春  ▁bowel 情结 称重 NAL ▁Fou  ▁unfolds MEM  腹泻等  和有关 olland  的时候一定要 涟水  早隔离  Melissa 多部 心若 摔跤 邓超 estial 男儿  但后来  心动的  记者会上  ▁bestseller 厚非 埃塞 扶正 cash  互联网上  什么意思 啥的  两个儿子 偏多 和三 善款  教育是  的好运  照顾孩子  中国人民志愿军  ▁conglomer 书里 告终 铁建  laughter 坐镇 甲醇  ▁ба  年后的  投资和 EEEE  ▁Qaeda  ▁longing 再累  纸尿裤  经了解  ▁Cout 决明 输赢  Holiday  destroy  ▁Forestry TSA  设计感 imet ~~~　  ▁Angus  ▁Sahara 拼图  就成为š  我也想š  ▁pulsÚ  ▁SheridanÚ  ▁uprisingĚ 农垦Ě 活法Ś 盈眶Ś 陈文ƚ  管党治ƚ  该团伙ǚ aidaǚ luckȚ  认真履行Ț  ▁Amitɚ  ▁buddingɚ 北开ʚ  对国家ʚ  就有人˚  是指在˚ ▁fir̚  ▁Archie̚  园林绿化͚ 编发͚  经济贸易Κ  ▁BaxterΚ  footballϚ 访客Ϛ ▁PDPК BLISHК 直视њ  理财课њ  说一说Қ 正对Қ  量力而行Ӛ  ▁clausesӚ " 处三年以下有期徒刑Ԛ 她才Ԛ 政纪՚  办公区՚  幸运颜色֚  直接结算֚ Cookieך  ▁Realtך  ▁amazonؚ  ▁remnantsؚ 约瑟ٚ nezٚ  ▁innateښ lintښ ▁girۚ  不断推进ۚ  井然有序ܚ  ▁Shepܚ  电子社保卡ݚ  ▁Clusterݚ 富硒ޚ  在得知ޚ  的产业ߚ estownߚ  ▁Coupe  ▁Tourist  ▁nominate 含泪 Wel  ▁shrine " 阿耨多罗三藐三菩提 之别 他叫 任劳 打上  全部都  更具有  的例子 Rows  ▁Incredible 晓峰  县市区  贡献了  ▁Forums 摊主  不限户籍  ▁prerequ KV 陶渊 erial  ▁estern  parameters 利湿 抚平  出了名的  生产成本  组织协调  ▁Rubin  nachweise  ▁pragmatic  ▁spreadsheet  适用的 ▁HER  是典型的  ▁ambiguous 卸下 善根 根管 警员  割韭菜  水平的提高 政绩 无主  格外的  生活了  高危人群  ▁listens 作祟 可喜 坪镇 快闪 绝招  ▁Impossible ▁FF  元器件  各国的  是错误的  ▁Usage  ▁nightlife 别怕 永泰  衷心感谢 assen  专科及以上学历  ▁dialect  主动出击  公积金贷款  才是最重要的 浮漂  到基层 rieg  健康饮食  有关内容  ▁Lorenzo upt  税务师 不重 做做 吃出  为主导 Bishop  ▁Angola  ensional  给予其  ▁Orchard  ▁braking  ▁Attractions 洗牙 画质 登台 spl  华南理工大学  catching  ▁ordained 掺杂  为前提 贺卡  轻微的 ▁NRA ▁Ш 四层  无论从  ▁mash  ▁Ancest 享福 脸的  作物的 ▁vid  ▁Scream  ▁outspoken 足底 躺赚  当老师  要抓好  考察对象  要注意的是  ▁robbed  ▁Claudia DOM  预约挂号  noticed  因工作  有弹性 ▁AST  塑料制品!  弯道超车!  相互促进"  都能找到" 棕榈# 浮雕#  的福利$  ▁Colony$  ▁Cummings%  似乎是%  红辣椒& Uber&  ▁gravitational' 难耐' zao(  打火机(  文明创建) agency)  ▁contamin* 故地*  ▁facto+  ▁funky+  ▁falsely, 惠安, 茶几- din-  的这种.  ▁caus.  重症医学科/  记忆中/ 踪影0  的垃圾0 ▁Wyn1  ▁Sparks1  Sullivan2  ▁Walters2  ▁resurgence3 菲菲3  往期文章4  ▁Scenes4  ▁Jacqueline5 不羁5 头屑6  依然存在6  ▁fingert7 小溪7 藏书8  不涉及8  提供商9  标准执行9 paying:  ▁actionable: 粟裕;  中记载;  为你提供<  产品销售<  ▁Beckham=  ▁Turning=  今日运势>  ▁pengu>  Contents? 上讲? 宁海@  流程图@  今年春节A  ▁ribsA  ▁toweringB nostB  ▁delveC 多动C 足元D ▁'\D  广泛应用于E  ▁bookstoreE 等均F 雕琢F  仅供学习交流G  ▁TowardsG 一物H 裙摆H  ▁JewelryI  的大海I  ▁pessJ  ▁NicolJ  以及与K  ▁mensK  的精神状态L  以上这些L  性别不限M  科学防控M  ▁pactN 查查N 说有O  所造成的O  记者注意到P 营村P BowQ 克莱Q  互联网公司R  ▁GrizzR  ▁hypnotS  equippedS 五人T 曾与T 股的U 采风U  蠢蠢欲V gradesV  ▁AshtonW  ▁pretendingW 必先X  不重视X ▁PEOY  ▁atrocitiesY 养鱼Z 太阴Z  那你就[ awan[  ▁Wilde\  intensive\  ▁conveyed]  材质的]  房屋租赁^  框架协议^ rl_ 赛中_  不下的`  伊丽莎`  体验馆a  或其它a  项目在b 完形b 礼堂c  ▁Listeningc  压缩机d ivotd  ▁Baine  Principale 无价f 考录f  起床后g  较低的g  Constrah 贪心h ▁Urbi  咨询公司i  生态旅游j  Einzelnachweisej 一桩k 隐约k  往届生l  回家过年l  Dependingm 萌娃m  南安市n  发展与n  涉案人员o  ▁resumeso  ▁ballisticp  ▁referencingp 七台q  威尼斯q  工作业r  红烧肉r agnas  爱情故事s kwargst  ▁poorert  ▁Conservatoryu  全面领导u 亏欠v 沙盘v  群众对w Jazzw ebrax  让我们一起来x  ▁mosaicy 粉红y  一只手z  ▁extravagz Bu{  目击者{  省会城市|  和大家一起|  ▁differentiation} 万张} 叫好~ 爱慕~ 皮皮  从自身  品质和  板块的  东京奥运  ▁Dwight  根本就  给儿子  好的时候 % 违反中央八项规定精神 む道 ▁ADA  一直保持  先后来到  ▁Chak  ▁Vault ње 每课  あれがあなたの [　 酿成 " あれがあなたの未来 导出 将达  吞吐量  これがあなたの 基底 杠杠 等高  都是对  消化功能  难以置信  ▁Token 林芝 uben  事业有成 上缴 内阁 锤子  专家和  就包括  热爱教育事业  ▁unbeaten  下决心  ▁hinted  Broadway 焦煤  的培训 bley  ▁Doha  ▁Blizzard 三县 脱掉 血性  可追溯  和集体 学森  手慢无  数一数  博野拍客  ▁Changed  ▁folklore  就更加  技术在  来确定  差不多的  ▁Palo  ipelago 奉请  龙华区 cale  电子证照 vasion kc 丽娟  孩子能  开花结果  fitting  ▁needles  ▁injections 五湖 ▁Kok 取经 缓存  的玩家 udos  体格检查  开具发票 在用 屈光 称职 Joh  电影中  要围绕  市住建局 affer  ▁Lent  ▁Mines れた 党派 收网 Dor  喝咖啡  有耐心  ▁perpetual  上涨的  总是一  皇帝的  工程总承包  玫瑰面膜膏  Converter 嫌犯 房住 ▁Bis  大部地区 artist  ▁algebra 横空 源码 OTE  在路边 ilus  预扣预缴 onstr  ▁Truman 全盘  脏东西  年及以上  深入基层  ▁calf  社会主义建设  请作者持权属证明 装进 helle  ▁Clos›  ▁morale›  marketingÛ 功法Û 情志ě 紫苏ě  ▁pyramidś 好想ś 就靠ƛ  学习上ƛ  路交汇处Ǜ  你要是Ǜ  固体废物ț 配齐ț ffordɛ  中国科学技术ɛ 月是ʛ  个学科ʛ  和挑战˛ (　),˛  也是十分̛  秋冬季节̛ 长线͛  部经理͛  ▁reapΛ  PurchaseΛ  ▁Classroomϛ 　↓ϛ 兰考Л 跟前Л  另一只ћ Pushћ  记者提问қ  需要什么қ  ▁Monkӛ 天立ӛ 水长ԛ  并建立ԛ  不管是在՛  年龄不超过՛  ▁respecting֛ 塑性֛ 请了כ  村民的כ  正常通行؛ 实为؛  了自己的人生ٛ  ▁tighterٛ  ▁intermittڛ  ▁northeasternڛ 城河ۛ 赔钱ۛ agmaܛ  每年都有ܛ 打倒ݛ  ▁міݛ  资料库ޛ  因为他们的ޛ ecutedߛ  ▁Chavezߛ  ▁escalating  一年半 colour  ▁influencer  欢迎你  加州大学  身临其境 nodes  ▁ruthless ▁  突出了  具体措施  ▁noodles  ▁strengthens  ▁foundational ühr ▁lex  重点学科  ▁Sark ieving  ▁attic  ▁vaping  ▁disclosures 潮图 王爷 累死  土家族  ▁Kenyan  Relations кс 甜瓜  场所的  体温异常  有的甚至  续航里程  osexual  ▁Zimmerman  ▁Wonderland  对于有  同比增加  ▁lact 每件 灿灿 ▁IPL  看到这样的  Electronic 丽华 晚清 棉袄 窘迫  副镇长  年产值  切实提升  有效预防  从今天开始 Amount  ▁Trauma 尽其 沙门  还发现  音乐剧  心无挂碍  我们能够  ▁Inters 呼啸 学学 砍柴  大循环  依法查处  降低成本 Hours  ▁carpets  ▁engraved  ▁attachments 之母  以避免 返场  ▁payout  ▁Gibraltar  ▁distorted  几十个  幽门螺旋 opause  Anderson 云山 是种  还是不错的 vendor  ▁Notification 人将 星海 金玉  在浙江  老公的 ▁Bam  ▁bearings 竞聘  不受影响  劳动能力  新修订的 Separ Spect  互联网平台 花艺  像这样  通常在 ▁DOM  破获一起  除此以外  ▁Macy  ▁Kamala 降薪  ▁concussion 未果  女士说  自己被  董明珠  传媒大学  这个时间  ▁Twitch  ▁simulator  ▁Disclaimer ▁á acao  ▁labelled  和研究  产教融合  提前准备  总资产  是一块  就会造成! heter!  一旦有"  复旦大学附属"  foreign#  ▁austerity# 内疚$  本声明$  城市管理局% 李军% 腰膝& 艾条& 被我' 飞到'  一棵树(  性感的( ,)) 华严) 取保*  市中区*  阅读习惯+  ▁bred+  ▁disciplined,  所以对,  ▁Isles- ▁ERP-  有多重要. hesda.  ▁binge/ 油锅/  你的朋友0  ▁entails0 ( !诊断为新冠肺炎确诊病例1 李健1  房地产企业2  ▁Athletes2  为全球3 ▁MAS3 grave4 Marcus4  ▁straps5  ▁Receive5  ▁horrors6 zin6  一封信7  图文并茂7  金融风险8 indra8  ▁undefeated9  ▁Identification9  集贸市场: 萧条:  中国是;  形形色; Limit<  的学习方法<  Require=  ▁palate=  ▁pillows> 推诿> 沉浮? ARI? Tow@ tea@  归功于A  非常明显A 还送B  的绿色B tightC  ▁WyattC 发愿D  隔离区D  免费使用E  ▁anomalE  ▁fullestF 偏旁F  大脑的G 丘陵G 景泰H 樟树H 涅盘I 熬过I  今天给大家J  解决问题的能力J 竹笋K 耳环K  ▁basilL 万到L 职守M 虚开M  鼓楼区N  多年的历史N  ▁mindedO 声名O  ▁któP  发生冲突P  并不容易Q  疫情情况Q  FictionR  ▁tallerR ▁‎S 减分S 餐前T  见不到T  ▁SheilaU 乡贤U ▁AUTV  ▁CentresV 种菜W  在人体W ▁icyX  教育部直属X  中华民族共同体Y  DisclaimerY  ▁transformationsZ 攻打Z  指引下[  健康快乐[  uttgart\  ▁Sutherland\ Ii] 培生] 急寻^  团市委^  内容来源于网络_  ▁bankers_  graduates` 埋下` 网安a  学校要a  有一半b  ▁Clipb  ▁Hurtc 在眉c 失灵d  人和人d  有益的e hunge  请您先点击上面f 挂了f  一幅幅g  城小区g  的小伙h  报名咨询h 倪萍i viri  有效期内的j  NotFoundj 互帮k 再等k 天之l  岳阳市l  也能轻松m  熠熠生辉m  ▁benchesn 与伦n 小吴o 片面o ("\p  你也会p  口服液q  ▁BACKq  ▁Assamr 户数r  的效率s 南华s 和其t 尘器t 开张u  ▁Herbu  ▁Pediatricsv 方寸v 煮饭w 要问w Srix  还是以x  身份证复印件y  ▁repetitiony  和小编z  前后的z ESPN{  劳务报酬{  我们也会|  红军长征|  ▁Mandal} 开赛}  看看你~  行车安全~  负面清单  金融行业 湿邪 kun  清洁剂  ▁Depend  ▁Reunion 挺胸 桂皮 清蒸  到今年 Gordon хи  帮助的 azard  ▁Hezbollah 的分 谷县  一图读懂  专项训练  根据实际  综合运用 下级  治愈率  跟着我  人的精神  ▁skyline 以期 舰载  牛油果 OTOS  不要着急 omega  原因是什么 跌停 ENN  以上是  第五步  随后的  ▁servic 许敏  文明校园  ▁eloqu  收入水平 aghan  ▁erase  ▁Trails  ▁birthplace  打通了 dain Boost  ▁Moves  convert  itlement  ▁appellate 密布  卷土重来  经济管理  ▁fairs  研究中心主任  ▁Montenegro 禁区  力量和  стра reece  ▁tearing 好喝 曾于  医生和  点什么  中国航天  打好基础  roscopy 荒地  他不会  成千上万  有丰富的  ▁Gmail GX 落寞  吐鲁番 ▁CPR acula 板蓝  就算了  建筑材料  ▁Uniform 泽民 CFO Cole ▁Hau ▁MCU  frequency  ▁malpractice 门上  懂得感恩  这句话是  experienced 交钱 能说 解聘  一日游  人员须  侵权行为  私募基金 不暇 修图  真的是太  英语专业 uthan  ▁cogn  innings  ▁Attacks  ▁persuaded  售价为  科技企业  ▁Sans  ▁Increasing 总督  军队的  王登峰 verify 普高  地毯式  ▁smoother  ▁burgeoning  ▁ambassadors 温饱  很有趣  中国生物  ▁taps  ▁parallels 三不 年获 解码œ  苦尽甘œ  都是这样Ü  ▁coolestÜ 帮帮Ĝ  巧妙的Ĝ  红外线Ŝ  重要信息Ŝ  ▁rodsƜ  ▁disappearsƜ 牵绊ǜ lewǜ  ▁embodiesȜ  ▁undeniableȜ 摧残ɜ 瑞德ɜ 迎刃ʜ  关于公布ʜ  能够增加˜ umont˜  ▁Bucc̜  ▁logger̜  ▁Petition͜ ák͜ 原审Μ 高汤Μ  这事儿Ϝ ℃)Ϝ 代词М sacМ  ограќ hemianќ  ▁quaintҜ  ▁deficitsҜ KUӜ braceӜ  核酸检测报告Ԝ  LoadingԜ  typedef՜  ▁initialize՜  ▁southeastern֜ 刷刷֜  宣讲活动ל  ▁transcriptsל 压在؜ 法条؜  显气质ٜ  不断努力ٜ  同时进行ڜ  幸福的生活ڜ  ▁spoilۜ  ▁resurrectionۜ 浩浩ܜ 路演ܜ  由原来的ݜ  ▁navbarݜ 晨读ޜ 肌群ޜ 雁门ߜ  有小到ߜ  常务理事  不知道如何 Graham  ▁Schultz  ▁Arrested  ▁potassium 多食 建好  专家会诊 噩耗 天尊 特鲁 马车 ▁CAT ▁Lup  平均年龄  ▁Fernandez 软硬 辛格 Mort Wave onne  白衣战士 ུ་ 再大  的风采  能保持  光辉历程  技术手段  内容是  在高考  突飞猛进  资金支持 AVA  ▁Freder  ▁seventy  为了使  对于孩子  及联系方式 东江  和发扬  来决定  教会孩子  粮食生产 Qaeda 不闻 交货 城人 搀扶  散打哥 Loop  希望他们  ▁Brave  ▁mastery  ▁vertically 爆炒 直指 解忧  南怀瑾  创造价值 lyssa  疫情传播风险 宵夜 纷繁 jam  的大好  托管服务  ▁Wein  ▁hates 焕新  寓教于  的策略  等多种方式  ▁zosta  ▁locale 刀子 即送 毒药 疲乏  湖北籍  这所学校  全面依法治国  ▁anthropology 颜悦  两分钟  提供者  电脑端  贵金属  国家医保局  综合服务平台  ▁Hubbard 卖部 档位 自给  岗前培训  高高飘扬 Angela  ▁Vaughan 壮美 相距  肩关节 ▁Haj  的不只是  新发地市场 asonry  ▁Beard  ▁awaited 戒掉 陈氏 会友 杨绛  不尊重  任劳任怨  ▁Amen  Indiana  ▁drafts 相间 cción 地瓜  关怀和  青藏高原  被开除党籍 ▁nat  就没有了  ▁vineyard  ▁prescriptions 撤县 李煜  吴某某  守初心  代理机构!  进行学习!  ▁испо"  开展核酸检测"  otyping#  itutions# MER$ eyer$  ▁prestige%  受益者%  践行者& ocre&  有关工作'  ▁Destroy' 人去( 富婆( 顶配)  技术水平)  ▁Scha*  ▁refurbished* /$+  我经常+  ▁Rican,  ▁forex, 偷渡- 每时-  的秘诀. Groups.  Ancient/  ▁abruptly/  ▁Tradition0  但他的0 之财1 紧闭1 补考2 辨析2  地说道3  行走在3  销售员4  作出重要指示4  ▁Lester5  ▁Nasdaq5 girls6  ▁bounced6  ▁specimen7 多户7  之后又8  删除了8  吉祥如意9  ▁Shri9  copyright: 为快:  正是在;  ▁flashing; 唇膏< 矿工< Ids=  很多人对=  ▁Busch>  ▁Ramos>  ▁Meadow?  ▁formulas? 初秋@ 厘子@ 尽是A  与普通A  贯穿于B  世界和平B  ▁plummetC  较昨日C  ▁OrionD  ▁TeddyD 哑铃E 毛利E 调的F  总体战F rulyG  ▁WageG  agicallyH  SerializerH 动弹I здаI  画面中J  建议使用J  规定时间内K  ▁jokedK  ▁TriumphL  ApparentlyL  ▁touchscreenM  看到你M  BrowserN  ▁mergingN  发现并O  ▁sizeofO  PatientsP 开个P  第四名Q  ▁HorsQ  ▁CollectorR  在水里R  爱心企业S  ▁memorabiliaS  ▁InvitationalT  岁的她T  的缘故U 组别U  公开的V  方向和V 夏威W ivatedW 开营X 鸡血X  以下几种Y  还会导致Y anutsZ  ▁rogueZ  continental[ 定陶[ 开团\  福田区\  登记信息]  ▁slug]  采取刑事强制措施^ 防冻^  最真实_  initions_  ▁staffers` 金水` mitea  每天都能a  长期处于b  ▁SUVsb  ▁blamingc ▁αc ссиd  声明的d ▁Kese  依然没有e  懂得珍惜f  ▁Diedf  就是这样的g Soundsg 则可h 淤积h  将及时i  文化传承i  正常现象j moviesj  Celebratingk 气馁k  ▁Bessl  医学界l  和人员m  最大扭矩m  那个年代n 华强n 氮肥o ▁STAo  的小女孩p ionalep  ▁airstq 穆斯q 解体r 驱赶r  cussionss 切削s  就到了t kvt 置地u  以后再u  城市化v  ▁buoyv  ▁Reflectw 善行w 奏响x 莘县x 学渣y 形似y 红素z  创立于z  ▁pharmacies{  ▁Documentation{ rv| 罗湖|  新能源车}  ▁fences}  ▁Heathrow~  ▁interstate~ 光谱  快速反应  ▁ADHD  ▁vans 暗自  非常强 below  ▁quota  military 都城  一件件 Hidden  ▁Lyons 就坐 爆珠 而建  和自我  跟你说  的意思是 sweet  Dynamic 钦佩  项目概况  structure 华商  宣城市  进行网上 入药 火候 爱她 eln  学历提升  ▁incremental  ▁Conditioning 迎战  回归到  因为这是 CLASS  现代服务业  ▁Bones  ▁mathematic лы  分析及  初中的  和提高 夜读 如歌  ▁Themes JZ 赫赫  凉山州  发挥出  平均值  总人数  联系起来  ▁Blessed  是一本  久别重逢  自身安全  装配式建筑 Legend  ▁braces  ▁Assange  ▁exercised 莴笋  凯迪拉克  使用安全带  ▁gunman 浓茶 bdst  新形势下  ▁skins  移送检察机关 公牛 垂涎  历史和  发热患者  是孩子的 helial  ▁Pepsi  ▁suing  getting  transfer  ▁Waiting  却可以  学员的  社保局  不幸的是  报名资格  ▁Radical 一喷 ixue  连续两年  ▁nicotine  ▁diversify  中得到  李焕英  的夜晚 ▁Soy  价格便宜  ▁Salisbury 喧闹 够用  即可参与  很多人说  Supporting 健脑  激动的  微电子  ▁Poet  ▁borne  ▁apologized  ▁upholstery  ▁Advancement 颓废  新用户  陆家嘴 aira cold dose  融会贯通  ▁Moldova  ▁missionaries 莫言  纽约州  野战军  热门专业 善用  心平气和  规划纲要  ▁décor  ▁abolition 重获 aban  公司提供  制作方法  ▁FoldÝ  ▁dictatorÝ 湿了ĝ Kinĝ 三台ŝ  地完成ŝ  盆满钵满Ɲ  竞争激烈Ɲ  ▁Obstǝ  先进基层党组织ǝ  ▁carrotsȝ  ▁presumedȝ  identifierɝ 固安ɝ 学界ʝ 缩写ʝ  的贡献˝  显著提升˝  ▁stationary̝  并非是̝ aise͝  社会秩序͝ azureΝ  ▁worshΝ  ▁Eurovisionϝ 一睹ϝ 品茶Н cliН  不甘心ѝ  不可忽视ѝ  我们常常ҝ štҝ 日电ӝ 至善ӝ 话筒ԝ  不大的ԝ ptive՝ 高阳՝  老同学֝ ▁Poe֝ 候补ם 凉风ם 贪吃؝  争议的؝  在成都ٝ  的那天ٝ  隔离期间ڝ  审核通过后ڝ  ▁Closing۝  ▁distinctly۝ 一转ܝ 经侦ܝ nteݝ  实施后ݝ ▁Todޝ  应用商店ޝ  革命烈士ߝ  老家的ߝ  要重视 解禁  的练习  成人高考  相辅相成  ▁dues  ▁dolphins  事件中  很常见  发生性关系 顶风 wrote  ▁appraisal 入眠 民工 要强  注定是 rored 多套 张新 杂技 IRS  资源库  在高速公路 二元 当上  伸出援手  工作顺利  失业补助金  ▁Timeline 孙杨  时间过长  ▁Rovers  ▁Baroque 云天  ▁SaaS  ▁learner vale  小学教育  ▁Bayer  ▁Leaves 喷水  平方千米  ▁sculptor  肯定不会  ▁esterni  ▁exiting  ▁effortless  ▁thankfully 咒语 学部 插管 看上  游击队  精神上 thor  olvency  ▁plains  ▁manageable 参建 和被 点为  学子的  教案课件  ▁Helic  ▁Browser |“ 周后  民警发现 的战  morning  你是我的  部分素材  综合素质评价 了其 刘嘉 embro ( !中国共产党纪律处分条例  ▁proportional 汪汪 请客  再见了  防范和  ▁Inventory 开年 若想 ▁gn  一致认为  对于他们 chery  ▁socialism MQ 考博  ▁Definitely  ▁equilibrium Xi 征管  任何一种  可以提供  供大家参考  ▁puppet  ▁coincide  优势区  在一定  投档线 中行 山茶 满口  外立面  电子注册备案表 冷冷 最严 OTH  拿出了 alog  他们不会  ▁Minh  ▁foreseeable atas ipers  ▁cruises  保税区  牺牲的  事业单位人事 近千 迟疑 音字 "];  改编自 Ship canvas!  ▁Haram!  ▁defends"  ▁stocked" ️⃣#  进行集中#  培养学生的$  ▁Kyrgyz$  ▁PowerPoint%  习惯于%  守土有责&  获取最新&  ▁Forrest' 塞进' 骗了( 你呢( spirit)  ▁Twelve) 一脉* 丢人* 大义+ amorph+  ▁skysc,  ▁confessed,  也有些-  服务期满- 奋起. вич.  在本市/  预警信息/  creative0  ▁endemic0 兴宁1  海珠区1  离去的2  以及各种2  每个人都有自己的3  ▁bisexual3  ▁epilepsy4 添砖4 舒张5 阴有5  扫黑办6  ▁Iris6  年硕士研究生7 省油7  周期内8  小苏打8  竞争中9 其道9 鹿晗:  做出贡献:  党工委副书记;  ▁brutally; 可点< 大埔< 安琪=  和监督=  板蓝根>  留在了>  远的距离?  ▁Chung? 法界@ 高阶@  还通过A açãoA oldtB  scriptsB 安详C 追捕C INOD UMPD  蚌埠市E  身份的E  ▁NovakF  严格要求自己F 利剑G 多雨G 柱子H  犯罪集团H  疫情防护I  高层住宅I YoJ 婚庆J 炒锅K 速速K  映入眼帘L  ▁SunderlandL  统计表M tionM  马鞍山市N  particularlyN 行知O  不够用O  会引发P  可以自己P  接近尾声Q  非法经营Q  ▁SmashR  ▁rockedR  ▁airborneS 章第S  阳泉市T raiserT  ▁inventionsU 打嗝U 野鸡V  本就是V  商业保险W  领取电子版W  ▁screenwriterX 口子X  收割机Y  网编辑Y ▁rudZ  流连忘返Z Looks[  全体党员干部[ NX\ 大潮\ 安妮] 打进] 抖擞^ 花红^ 讲得_  将重点_  强化对`  prevent`  ▁courthousea 苦味a  不分先后b  ▁Calebb 社员c  睡不好c  当场死亡d  ▁tenord  ▁Consultatione 热卖e  中的重要f  ▁JUSTf 增援g 苦的g 蛋清h  文学艺术h  没有及时i  ▁blinki  zhaopinj  ▁SOFTWAREj 肚皮k 还小k  决定着l  西门子l  相关文件m  ▁exitsm  ▁interceptionsn 大法n  好像是o 快件o  文化内涵p 狼藉p  好用的q  时间与q coler  性价比高r  进入下一s  ▁Rayss 一算t 简装t 菖蒲u 香浓u  操作指南v  落地生根v  ▁Deletew  ▁fadingw  盼望着x  ▁Currencyx 群力y  并给予y  紧凑型z UTIONz 内服{ 已从{ 张恒|  峥嵘岁月|  ▁removable} 苏木}  也意味着~  你准备好了吗~  ▁pathogens 机率 被拒  可口的  能接受  科技发展  ▁Refer  ▁goose  provoking 东胜 绰号 ▁DJs 封神 甜文 速成  不到了  在推进  林黛玉  的第二  ▁flare  ▁Libyan  ▁shooters 坏处 洞穴  最令人  根据省  也没想到  serving 拉网 调换 opo  的好时机  ▁Pets kx  有过一段  活在当下  ▁cowboy  specified 权势 ▁Tos  容易引发  他们的生活 guitar  ▁Prediction  ▁Rockefeller hunt  积极进取  书法家协会  抗氧化剂  ▁django ји 小弟  的人的 borgh  ▁Fifty  releases  检测体温  幽门螺旋杆菌  ▁Chronic 山乡 痴情 退改  到中国  敏感的  沁人心  甚至于  应急管理厅 寻访 阿卡 面纱  十一月  带你看  常规赛 yout 另类 根除 炼化 称谓 pac  掀起了  违法建设  为疫情防控  谁也不知道  ▁tumult  ▁distressed 已由  ▁roce  straction 充气 忍痛  万人的  需要做  ▁Canton  ▁Symptoms  ▁pronounce 杯中  亳州市  ▁occupying  ▁intimidation 每刻  如果对 Bind  ▁forgetting  Collegamenti 天赐 的梦  夏威夷  每公斤 warn  ▁recognises nɡ 扣子  仍然存在  暨南大学  ▁Mostly  ▁programmer 屋后  灰指甲  而自己  让中国  顽固性 ▁Vet 回执  最伟大的  inherit  ▁Guadal 注音 ELS  留言板 ▁LIC  ▁Taxes  emakers  eligible  ▁nuances 当真ž  都不想ž itemapÞ  ▁TrackerÞ 不花Ğ 仁义Ğ 形同Ş 无权Ş 通化ƞ  人士的ƞ  英语水平Ǟ  ▁RudyǞ  不过气Ȟ  中文版Ȟ  回答了ɞ  阳性的ɞ 图示ʞ 工况ʞ 雷军˞  铁路局˞  ▁Pollution̞ ます̞ 复阳͞  清远市͞  熟悉了Ξ  的孩子们Ξ 喜人Ϟ 末梢Ϟ  告诉了О  在一家О  ▁marsў  ▁ergonomў 冠脉Ҟ 石子Ҟ  你愿意Ӟ  培训费Ӟ  闭上眼睛Ԟ  ▁subcontractԞ  ▁tuberculosis՞ ▁−՞ 隔阂֞  发展理念֞ Prefixמ  architectמ  ▁disagreement؞ 唱着؞ 房颤ٞ 拜佛ٞ 火辣ڞ 田镇ڞ 解法۞  迎刃而解۞ Viewsܞ  ▁disparateܞ 加厚ݞ  我建议ݞ  金银潭ޞ  团结一致ޞ  ▁eminentߞ  ▁liaisonߞ  ▁tackled  ▁Abortion  ▁Financing 伤疤 树一 迪奥  时间不  病例的 ▁MET  密不可分  一个人的时候 nob eeds  listening 老窖 苏北 顺从  成员国  ▁possessing 亲亲 对未  内在的  这就需要 对战  吉安市  神经内科  ▁Gotham  ▁authorize 丽娅 账务 陈年 rax  allclasses 去吃 同业 灌注 西关 CVT  ▁Sketch % 国家级非物质文化遗产  真正有  ▁Innoc  ▁procession  ▁congratulations  在抖音  Execute  ▁Riding  ▁Dentistry 添彩 过街 高配  ▁Genius 万贯 坐骨  之下的  非常不错 Walter 自大 转角  口袋里 dogs  几乎所有的  ▁wrongdoing 具足 贪欲 Enh INC  ▁clinically 小鹿 重修 Arn  的房间  可以按照  顽强拼搏 estion  ▁trusting 火柴  大家觉得  当天上午  空气清新  ▁Pixar  ▁Durant  在处理  ▁Agnes  卫生纸  一个又一个 凡夫 吐血 空袭 ydw  或媒体  ▁Frag  根据教育部  ▁yelling  ▁Associations  ▁interception  Authentication  ▁reapp  ▁bicycles ić 牵动 西餐  ▁Billie  ▁avoids " 部分文章推送时未能 松散  平稳有序 与非 霓虹  在银行  杨振宁  诗和远方 Ark  手机里  通讯社 ▁MER  ▁Taken  ▁pastors 严实 那叫  使学生  我们说  的香气  教师招聘考试  要提高政治站位  ▁disturbance 人或 国平 线圈 练功 ▁WB!  的摄入! ▁pom"  中考数学"  ▁Trilogy# ✪　# 否决$  生命科学$  ▁uncomp%  ▁Bishops%  ▁impulse& mw&  的句子'  Contains'  ▁postdoctoral(  地位和( 又双) 针法) 长方*  ▁Hanna*  ▁Household+ 周迅+ 操劳, 特设, 碌碌-  经济和-  也不容易.  小儿推拿.  文明行为/  ▁Vita/  ordeaux0  ▁absorbing0 招满1 高招1  ▁Respect2 $\2 就位3  搬运工3  的差异4  每人每天4  ▁Sevent5  ▁escalation5 大乘6 胸襟6  腊八粥7  静静的7 osto8  ▁dumping8 I B坚持以习近平新时代中国特色社会主义思想为指导9 精诚9 镇区: 鸡胸:  指出了;  治疗师;  伊丽莎白<  ▁bureaucracy< 洪波=  也会影响=  全面升级> 兴建> 善心? 未病? 涉足@  该地区@  ▁vomitingA  不发生A RCEPB  为您推荐B  ▁FallonC  咸宁市C  无添加D 大漠D  向学校E ▁NayE  下面这个F  我们都应该F  ▁HanoiG  ▁testosteroneG 已定H 知其H  对皮肤I  多多少少I  ▁SampJ 中锋J PMGK  全攻略K  速度和L  苏州大学L 得病M 重彩M  事故的N bowsN  企业资质O  ▁HelpsO 英尺P  无限次P NarrQ 家是Q 插花R 时也R iyuS ▁seuS  GenerallyT  ▁correlatedT 并网U  同时也要U  国际关系V  玫瑰全因子V  贸易有限公司W  面对着W  第一小学X  ▁fiancX  ▁lifecycleY  ▁sanctionedY 就绪Z 底座Z 该名[ Kel[  带领团队\  政策宣传\  发生了变化]  ▁oversaw] 怀仁^  双离合^  设定的_  ▁Ende_  ▁Toro`  ▁Valerie` 乘势a 初入a  妊娠期b  就算你b  注意饮食c  ▁Grimc  ▁ammond  ▁acceleratord  ▁convertiblee  财源广e  广泛应用f  其实并不是f  请联系微信g  ▁Kraftg 保镖h  出现了问题h  ▁conceali 吴京i  衣帽间j  强制隔离j  ▁Rupertk 之效k  不加价l  出席了l  对产品m  雄厚的m  ▁deuxn  ▁Tavernn 张春o 摸爬o 柔韧p amenp  以下材料q  independentq eerr  摄像机r  就能让你s  我们一起来s  我们想t  的优点t  爱不释手u  ositoriesu  ▁measurablev 毛鸡v  旅客列车w  ▁Statementsw  共同点x  更丰富x  检测点y ▁GTAy  他们总是z  叠加使用z  ▁Sard{  ▁Manifest{  ▁murderer|  ▁Producers|  ▁Techniques} 占星} )  ▁metre> 藏于? Bag?  送去了@  看的开心@  ▁DolbyA  ▁wineryA  ▁StantonB  ▁factionB 元朝C 抢眼C 秀发D  并不好D  不断加大E  尤其是对E  ▁ClassificationF 上官F 报导G 谋略G  部分人H  ▁CrimeaH 长到I  席之地I TRIBJ uriaJ  人间的K  生活节奏K  ▁FedererL 铜钱L 银元M  不堪言M  垃圾处理N  着力提升N  EndpointO  ▁InterimO 抨击P 更让P 炒饭Q  电影票Q  明明白白R  这是一款R  ▁fussS  ▁taxableS  ▁TownsendT  ▁PaceT  ▁unreasonableU  酣畅淋漓U ToyotaV  ▁VinciV 井井W 帷幄W 置之X 霹雳X UGHY  开通运营Y  支持政策Z  进一步了解Z 出勤[ 知悉[ 起大\  总价值\ ▁Sok]  邮件主题]  的营养价值^  ▁finely^  athletes_  ▁Champagne_ 哈喽` 油温` 热盘a  方便的a  充满希望b  西海岸新区b  indexOfc  ▁Lenovoc 推敲d 陈梦d  量减少e IBLEe  工作目标f  补贴资金f  ▁pondsg 开平g 火葬h 甜度h 瞧瞧i riji  总指挥j  ▁Wrapj Walkerk  ▁MUSICk 小指l  ▁Deppl  Christinem 母牛m  再也没n  左上角n  参加培训o  百年奋斗o  走着走着p 代数p 受热q 古时q  好孩子r osker  心情不好s  ▁Martials  ▁husbandst  要求各t Strau 日向u 灵修v  宁德市v  ▁eyeww isablew  ▁Newsomx 三会x 糖醋y  体验感y acicz  高三学生z  concept{  ▁psychic{  ▁regrets|  苏格兰|  Lawrence}  ▁repression} 为安~ 拉上~  技术要求  ▁midway  ▁noises 刚到 桃树 金毛  在场的 nian  egovina  Hardcover 但与 岁末  日地点  中国天气  我是一名  ▁Corridor 伯乐 反式 指针  屏幕上  正规渠道  有效身份证件  oriously  functions  ▁patriarch  ▁inspectors  ▁dimensional 牛顿  操作方法  的生活环境  ▁scooter  ▁breached 型和 标牌 老将  保质保  你的时候  处理办法 blers  电动三轮车 Shadow  ▁Belgrade  ▁screenshots  ▁Reduce  ▁exponentially  的重大意义 stress  ▁Erica  hospital 再从 删减 酱酒 金鹰 NOR ▁MV  来计算  ▁Eleg  ▁vectors 即到 多些 答对 ▁JOH  遵守交通规则  ▁Suggest  也不太 ISPR  布置作业  财务会计  ▁Lovely  ▁reusable  他的人  ▁DOWN  ▁opts  Changing  ▁Islander  ▁relatable 仙居 次要  低密度  你了解  将影响  ▁Alvarez 风挡  了孩子 wash  ▁Leafs FOX  你一个  ▁Heating  ▁affidavit 煎至 Err  美国国家  长期服用  ▁Sting 搭讪 站房 耐药 耻辱 还和  　#　  不断探索  Celebrity  有好的  卫生部门  ▁disappearing  万平米  学习者  甚至出现 Victor  ▁Frankie  ▁plaster 单选 王后  腹腔镜 Duke  管理岗位  国务院决策部署  专业毕业 iad  大小便 Wine  目前正接受 收腹 监利  不成功  微发布  记录仪  ▁sweetness . '中华人民共和国传染病防治法  你能有  家门店  的现代  可能会被  家人朋友   未成年人的  chromeà  stringsà + $解除医学观察的密切接触者Ġ  脱离了Ġ  ▁divisiveŠ ▁žŠ 两腿Ơ DasƠ  ▁KristinǠ rwǠ ренȠ  交通环境Ƞ  华中师范大学ɠ 复燃ɠ enskyʠ 操办ʠ 禁放ˠ ▁ERRˠ  就业困难人员̠  TestCase̠ 乡长͠ 五分͠ 再起Π 冠名Π  和现场Ϡ tellϠ  放假安排Р tainedР  ▁KohliѠ  ▁ExhibitѠ  的差距Ҡ  ▁PrecisionҠ  申请者Ӡ  时间为准Ӡ  金融产品Ԡ  ▁LobbyԠ  ▁roadsideՠ  ▁Ecosystemՠ  ▁Recommended֠  中医师֠  企业对נ  derivedנ  reportsؠ  ▁repaymentؠ 房网٠ 自荐٠ peeڠ  男生的ڠ chun۠  霍普金斯۠  ▁Loseܠ  ▁Forexܠ  触手可ݠ  ▁regexݠ  ▁cannedޠ  污染防治攻坚战ޠ bkߠ 昌邑ߠ  迫切需要  ▁morally  许多的  ▁perfume " 长按二维码识别关注  妈妈在  做人做事  ▁Transmission 本心  以高度  理解与  的资格 ▁Gly  ▁OnePlus  移民局 ▁GPA  技术标准  此次违建 俞穴 扦插  你能够  所有制  本题考查  ▁sophistication  个地方  的结局 ruby  招生考试机构  ▁brunch 仁慈 ▁kö  不高于  再部署  托马斯 ▁CSR  共渡难关  培养计划 Years dating  ▁buddies  ▁tedious  ▁nonfiction  弥漫着  联合开展  ▁Snapdragon 不欲 水井 罐车 躯体 CAC  ▁rupt  ▁prosecute 舌身 遇害 靖西  压轴题  联谊会  年出生于 CRIPT 候审 […]  之路上  父老乡 ▁Yad  ▁sliced  ▁graphical  ▁timetable 冰毒  延安市  ▁Govt  Alabama  ▁Athlete 良师 Deg  条件允许  ▁Comparison 看她 yla  发展格局 补虚 高专 LOS ▁ao  专人负责  可以预防  ▁Hier nature  ▁chased  ▁mailed 未完  等设备 人机 奉节 怪异 想学 烟熏 ▁NSF  ▁virtu  ▁Werner 脾脏 Dutch  ▁Dundee  ▁musicals  ▁confinement 修心 环氧 盱眙  点击底部  ▁Mara 鲅鱼  来说都是  热泪盈眶  ▁clog  ▁Incorporated Au 上完  发红包  由于其  人事任免  ▁teas  Cluster 奶爸 川芎 想和  了一口  校园环境 学社  月任现  让广大  食品加工  Marshall!  ithmetic! 冲到"  在民间"  负面的#  开局之年#  ▁galvan$  ▁nearing$ 环抱%  适当调整% 为中& 水后&  东西部'  河源市'  试试吧(  回国人员(  增收致富)  农产品优势区) 嫁接* 小队* ▁Sev+  identifying+ 请按, 赛程,  威士忌-  无声息-  究其原因.  ▁Accreditation. 命宫/  idental/  ▁murals0  ▁deductible0 岗镇1 过夜1 ات2 债主2  上万的3  或相关3  持续发力4  Basically4 触法5  该研究5 Hart6 ▁FCA6 垫底7  就属于7  ▁Cecil8 lp8  可进行9 ▁Pey9 将你: 长鸣:  向所有;  停车泊位; clips<  ▁Geme<  一个人在家=  erential= 对内> 烘托> 行经?  如果他们?  施工方案@  次会议通过@  ▁nucleusA  ▁WilkinsonA 哈就B  同行的B 已至C 并作C 灯谜D 纷飞D VelE  在前期E  真的不是F  重大事件F BrendG  ▁loophG  中国共产党的领导H  ▁BringingH 阿布I ▁WOI  五年的J  执法局J ▁alkK  水利工程K  特殊原因L  ThoughtL BOXM  阿勒泰M  实践能力N  实际问题N  降低血压O  ▁manicO ^\P 快门P 英美Q 谷底Q  海南岛R  班学生R  学习理财S  ▁arrowsS 喜获T  为同学们T  该怎么做U  ▁SlotU  WinningV 头肌V 它也W 打来W  ▁pacedX 巷子X 血汗Y ноеY  ▁ЗаZ  的背景Z 怨天[ 所愿[ 河间\ 猛然\  无挂碍]  无门槛] ▁BRO^  免疫球蛋白^ edited_  ▁Bethlehem_  ▁discouraged` 眼睑` 计发a  手续的a  申请条件b  言传身教b  ▁evokec  ▁Evelync  ▁braveryd % 邓州论坛邓州论坛邓州d 永丰e  戳视频e  这一类f  不得不提f  我们这个g  热文推荐g  详细内容h  ▁sankh  ▁Impressi  ▁wordingi 用神j  是一年j VOCsk  ▁Tobagok  ▁Marinersl 双线l 要件m oyem  就完成了n  有效成分n  违法行为人o  ▁extraco  ▁parodyp  Chairmanp 光头q 杞县q  勤勤恳r  同意了r  龙岗区s  可以改善s Glasst  urancest 井陉u 扇贝u  信息披露v ppelinv  其中境外输入病例w  ▁disconnectedw 天平x 弧度x 桥面y  在学生y ▁Fooz  党史故事z  是不同的{  正式宣布{  ▁bookmark| 推着|  会显得}  招标投标}  ▁Uses~  在遇到~  无愧于 古方 很忙  并依法  播出的  有一丝  正处级  损害赔偿 diagn  ▁irrespective 努尔 显卡 沙星  的根源  劳动成果  ▁Hadd 餐厨  纪念品  赵某某  之计在于  ▁Stere  ▁Tahoe  ▁curry  有阵雨或雷雨  ▁Bundle  ▁harmed  ▁Butterfly  ▁federally 梅子 爆品 Hugh  定期体检  ▁timestamp vv 太深 条第  无法自拔  ▁Imag  研究生招生  ▁tendencies  干净了  推动下  ▁siding  迈出理财第一步 皇姑 福地  不完善  又一遍  可能对  勤勤恳恳  ▁Adri  Brandon  ▁safeguarding 每生 jer  边走边  Materials  ▁smuggling  不为别的  未来三天  民警迅速 locks  希望同学们  ▁gearing 两脚 勤学  ”;“  减少外出  收获满满  disable 汉化 注塑  的全球  花生油 ▁ROM  感染人数  Cookies  ▁Conven  ▁Siemens  installed  也很大  的品种 ▁Sao  紧急使用  ▁cran  ▁Alphabet 点拨 秦国 rut  ▁фи  身体和  因为只有  守土尽责 etian  关注的焦点  english 青光 nin  流程和 ymes  ▁дру  ▁fraternity 无智 Hip  那些在 Brid  希望你能 slide  双城经济圈  Eventually  ▁Mansfield  和儿子  赶紧去  更是如此  ▁Hodg  全面发展的  ▁overlay  ▁Personality 亨利 公历 chs  ▁был  ▁Stem  日内与本网联系  Messages 历尽 ▁OD  校友会  ▁Bonds  报名截止时间  ▁Bridget  ▁polyester  ▁Southeastern 型血 ▁upp¡  ▁waits¡  ▁fluentá 中庸á 选课ġ  ▁Lowellġ  ▁evangelicalš 鬼子š  微平台ơ  新势力ơ  学科专业ǡ  客户提供ǡ  百家讲坛ȡ 胖了ȡ  在执行ɡ  美国总统特朗普ɡ “(ʡ 凳子ʡ 梁柱ˡ 粗大ˡ 韧劲̡  和草原̡  自认为͡  如期而至͡  ▁SunriseΡ  ▁rejectingΡ % 事业单位公开招聘人员ϡ 宛若ϡ  ▁ValleС  ▁snorkС  Campaignѡ 海中ѡ  无可厚非ҡ  ▁barrҡ 棋子ӡ  对符合ӡ  的恐惧ԡ  专项练习ԡ  弱势群体ա  社会救助ա  看到自己的֡  全国政协委员֡ 拉姆ס 探店ס  单位为ء  坚持用ء  移动端١ Pete١  ▁Dividڡ  closuresڡ 几日ۡ 看这ۡ  过大年ܡ alarܡ  但一定要ݡ ummiesݡ  ▁exportingޡ 蒂芬ޡ  史上的ߡ urvedߡ  ▁Huge  ▁PRES  ▁spat  ▁biotech 赵孟  不可逆  仪式在  可放大 antz 要买 >:: ène  旁观者  стро  菩提萨埵 版式 龋齿  值得你 ▁Mou  ▁Fairfield 任凭  郫都区  哭笑不得 Dennis  ▁motif % 全日制本科及以上学历  ▁registrations 嘉园 宝爸 热土 网警 难逃  ▁Redskins  ▁Formation 真言 ETE  报名通道 园所 改写  不停蹄 epid  正常工作  ▁manuals  ▁Laurence 前应 妖精 成事 道里  与妻子  ▁Warhol  ▁Defender  科尔沁  崇高的敬意  ▁Diagram 医技 喘息  变更为  视频里  ▁Stakes 月明 问过  花木兰  五官端正  ▁Bose  ▁cynical 传情  搭建了  ▁Laugh  ▁coaster 乞讨 在盛 种鸽 芡实 表妹 马家  虽然我们  进行处罚 接回 腾空  腰酸背痛  lighting 轻轨 ▁Ner  充分体现  消费升级 % 当日新增治愈出院病例 水光 沉静 节庆 Lam  的日本  一部手机  ▁Revised  ▁Celebrities 家财 工件  不耐烦 aryl  三年以上  ▁chandel  beautiful  ▁Mitsubishi  在开展  自去年  通知单 dump  ▁buds  ▁maths  ▁resurrect 雅阁  参议院  要尽量 降为  公司为  的表演 ▁ICO  长期从事  全日制普通高校 宗族 将车 水饺  共同打造  在线观看  的小可爱  第一印象  ▁uphill  ▁Fighters 英伦 西服  测试题  一个故事!  ▁Bowen! öm" 就少"  从美国#  项目现场#  智慧和力量$ 首座$  看到过% ▁HMS%  ▁minimise& 即说& 离心' 黑道'  学习笔记(  无法核实( prime)  Providing)  ▁cautioned*  ▁narration* 公德+ 天街+ cox,  一半的,  研发投入- obalt- 一纸. 两辆. 肮脏/ Crow/  生活条件0  领军人才0 covers1 罗布1  和你的2  属性的2 ▁PVC3  ▁Gord3 去化4 山居4  他家的5  视而不见5  医务人员的6  十九届二中6  ▁makeover7  ▁knowingly7 大车8 花灯8  相关性9  我们还要9  招聘工作人员:  的爸爸:  职业培训;  贫困劳动力;  实现财务自由< 喜茶< 新传= 晴空= 视光>  发现和>  阳台上?  与人相处?  思维模式@ 北城@  直播时间A  ▁drawersA  ▁detaineesB 笔触B  博大精深C  盈利能力C “”D 贬低D  半成品E  不舍得E  给父母F 备货F 水份G 虫病G  ▁PrasH  ▁PregnancyH 破格I  管理要求I  来源及作者J  ▁KendrickJ  ▁socioeconomicK 侵华K  也相当L GalaxyL  ▁flippedM  ▁ParticipationM 也只N 是福N  专用的O  音乐人O omalyP sparkP  ▁divestQ  ▁analyzesQ 汉阳R  与方法R  政治纪律S  的积极性S gaardT  ▁glamourT 倒班U  大地上U  软硬件V  累计完成V  ▁semifinalsW 天麻W 是何X CowX  长治久安Y  ▁GREATY  ConsideringZ 三界Z 五香[  手机短信[  湖北日报\ aguay\  ▁Québec]  ▁Hutchinson]  ▁neuroscience^ 昏暗^ 病亡_ 钾肥_  一件衣服`  一探究竟`  招生院校a  文安微生活a  ▁precinctb 安岳b  不明确c  系统地c Fernd ▁PATd  深入一线e  经常食用e flagsf  anderingf  ▁confidentlyg 亦乐g 退路h  中新社h  从小学i  发源地i  的热度j pendj  ▁Lexusk  ▁credentialk  ▁inconveniencel 变暖l 油性m 雷诺m  的愿望n  getStringn  ▁attendanto  ▁Cumberlando 梨园p 灰心p 的刘q  住房和城乡建设厅q  ▁Scandinavianr 法华r 颅内s URSs  第三季t  教育体系t  毛泽东在u  眼耳鼻舌身u  看的开心可以v  ▁deportedv 受用w 心衰w 范例x 铂金x  你今天y  方式来y  现在报名z  ▁uterz  ▁grazing{  每一个人的{  ▁breasts| 滨水|  少女心}  消费满}  ▁pony~  ▁полу~  ▁respondent 休憩 吴孟 微讯 菌斑  重庆大学  ▁Bett  ▁bounty  不管怎样  教育质量  裁判文书 产检 故说 直上 黯然  了一声  现在有  的人了 ▁Boo Adams  ▁maze  ▁amassed  ▁conquered 下步 然大 龙川 Neigh  ▁Spending 两极 活佛 URR  不轻易  去思考  在发展  是具有  浓度的  严峻形势  教师考试  ▁Papa athing  utherford % 案件正在进一步侦办中 一晃 书架 厚望 耕种  不要紧  简单又  障碍物 odie  他们自己  Premier  ▁comprehension 之象 黄文 GIF  我儿子 cass  为大家介绍 apixel  ▁Lansing  ▁artisans 少阳  了一颗  在最后  被美国 AGES alism  ▁prostitution ❖　 预兆  才能达到 usual Claire storey  ▁subtitles yq 白花 ),” COS  好少年  在互联网  ▁Stir 良品 茶室  宇航员  有着很 ▁Baj  使命担当  国学经典  尽可能的  ▁retri  ▁dove 葱葱  一旦出现发热 君王 太远 字画  区领导  招股书  是日本 松下 豆沙 лов  的阅读 ▁tug  优秀青年  学历学位证书  ▁Cavaliers  ▁discourage 松开 氏菌 民风 铸牢 oplan Famous aughty 耗尽  已成功  而出现  自然也就 earned ucceed  Brother  ▁hardships 不归 连体 autom ophile  永久基本农田 生而  一些的  不等式  多种多样  男女老少¢  ▁razor¢  ▁Kramerâ 长风â  就有一Ģ  效果明显Ģ  服务模式Ţ  驻村第一书记Ţ  ▁esportsƢ 木有Ƣ 膨大Ǣ crestǢ  ▁HedgeȢ  ▁SmileȢ  ▁paralyɢ  Rendererɢ 政企ʢ 清秀ʢ ukhˢ  房住不炒ˢ virus̢  ▁Dolly̢  ▁scaff͢ 大干͢ 年审΢ 激流΢ 违禁Ϣ  失业率Ϣ  是很重要的Т  ▁cartridgesТ 时须Ѣ 菜花Ѣ  是否在Ң  ▁VIDEOҢ  RegardingӢ  ▁distortionӢ 修为Ԣ  爬起来Ԣ  人的时候բ  ▁Comeyբ  的桥梁֢  建筑高度֢  限时特价ע  ▁heapע adayآ ▁Gusآ  栏目视频٢  内分泌失调٢ 达摩ڢ 集第ڢ  请一定ۢ ▁FITۢ  学校建设ܢ  ▁Rosieܢ  贡献自己的力量ݢ  ▁Directionsݢ 就找ޢ OMGޢ  平均每天ߢ 美工ߢ  成本和  高的人 ▁MIC  三个阶段  ▁gras " 大学本科及以上学历 拳道  在新的一年  ▁dentistry 外星 钟楼  ▁Mighty лав  鹌鹑蛋  什么情况  ▁footing  ▁selects  ▁deposition 两新  优先录用  收听收看  ▁Ends  ▁smokers 上楼 便饭 普利 荣成  热闹的 Catch  得淋漓尽致  ▁Edith  ▁cooled  ▁analogy 张嘴  等部位  邀请到 opaed  还可以通过 ledged  Francis  ▁Blacks  ▁Merrill 典藏  不错了  也没用  感谢你们 游走  十字架  ▁boxer  ▁dishwasher 囊中  摸爬滚  活成了 Jump izen  才能获得  硬笔书法  ▁Nora  ▁Racism  ▁Examiner  ▁Jurassic 她将 就买 ofi  医疗救助 可凭  中明确 ▁Ack  ▁insiders 吴晓 存档  征收率  定期检查  政策执行 历历 月后  合理确定 ortex  ▁Benz 乱世 除数  再通过  军医大学  ▁Sunni  ▁thats 的电  义务兵  ▁ringing 圆圈  迈出了 ▁HIP  夫妻二人  相关事项 ijian  ▁Tolkien 闭嘴  县委县政府 信奉  保护动物  应接尽接  ▁kter 之差  回顾了  换言之 Sold ▁viv  往往都是 smile  ▁Soil  二级巡视员 传祺 攀岩  创下了  独立性 enko  会有一个 包厢 祖籍 飘摇 oum ▁Osw  服务管理 rangle 临走 舞者  号电话  马不停蹄 emann 情书 新乐  支队伍! Fine! hare"  颠倒梦想" % 学习贯彻习近平总书记# 上林#  方式是$  ▁inconsist$ rg% 亮化%  到晚上&  处理后&  ▁LIKE'  ▁Zend'  ▁Swansea(  人左右( male) uracy)  移动支付*  ▁fruitful*  ▁calendars+ 光圈+ 指明,  小汽车,  的年纪-  老百姓的-  这片土地. ▁Tao. 辛金/  ▁Chairs/  ▁Motorcycle0 加索0  大三花1  透露出1  可能曾经2  magazine2  ▁Challenger3  ▁Mathematical3 新潮4  ▁Erick4  ▁wartime5  ▁Implement5 燃眉6  宜春市6  校领导7  Discussion7  ▁unpopular8  声香味8  流动的9 会阴9 村长: 镇压:  半年后;  庆阳市;  拿什么< ▁Duo<  农村居民=  普通本科=  党和政府的> Female>  ▁ballad?  ▁coined?  ▁sneakers@ 张继@ 连任A  新农合A  运动场B ▁IPSB  ▁TarantC  typenameC 李俊D  乡村治理D  停止使用E  的检查E ▁TecF  很难做到F mightyG 的生G  居然是H  电力公司H  watchingI  ▁subdivisionI 醒脑J 驱使J  小便宜K  的生物K  ▁HowellL  ▁impoverL " 请及时联系我们删除M  ▁responsiblyM 凸起N 英超N 装作O  ▁SELECTO  ▁prototypesP  ▁organisationalP 选品Q стіQ  稳定在R  主要建设R  ▁adhesiveS  ▁AdvocatesS 双鸭T 新剧T 活水U  了一片U  双选会V  就感觉V  ▁släW  ▁inertW  ▁deteriorationX 学和X 进到Y  挨家挨Y  ▁RahulZ 横溢Z 藏友[  有希望[  业余时间\  嘎嘎嘎嘎\  可以避免]  ▁Magu]  据媒体报道^  ▁McKenzie^  为何要_ ▁ALS_  不是有钱`  modation`  ▁Transcripta 切段a  应当依法b  Columnsb  ▁lawmakerc  ▁Mauritiusc 报喜d  ▁mitochondd  ▁discriminatee 原判e  每天喝f  习近平谈f  充分运用g  都是我们g  ▁Mbpsh  ▁Hanksh  ▁Barnetti 今明i 抽空j 考过j  偏远地区k  投标文件k  privacyl  ▁Lawsuitl  ▁Mechanicsm  ▁originalsm 事中n  ▁étén  ▁juxtapo 找找o 来接p 炮兵p 速冻q  他也会q coutr pianr  中国革命s  反馈意见s  ▁Contractst 4 -论坛邓州论邓州论坛邓州论坛邓州t 待续u 速写u  ▁nombrev 脑后v  显示了w  杀猪盘w  肩胛骨x  设置在x  重量级y  市人民法院y  联合创始人z istlesz  ▁summoned{  ▁archaeology{ 拓跋|  深度合作|  ▁Patty}  变得很}  女大学生~  ▁Pest~  ▁MSNBC 剖腹 弄清 张江 近照 雪茄 opol  市交通运输局  ▁Madden 遍及  众人的  想说的  的方案  范围广 pair  危及生命 Enable  ▁sparks 受挫 ▁pag  本文作者  追究责任  新冠肺炎病毒 兵法  题目的 FORE  区疾控中心  这样一件事  avement 不超 着了 设卡  不进去 ▁Umb  无法接受  ▁confess 乡风 小帮 补种  的烦恼  非常快  自然风光  三年级上册 而要  不单单  优缺点  整体来看  university  ▁Charlottesville 教综 更应 赵忠  不胜收  面试人员 炒匀  发热病人  这些症状  ▁Junk  为切实做好  刑事诉讼法  ▁nephews 三段 渡江  ▁Companion 听我 校准  一志愿  金融业  请加微信  我们将根据 ISTORY  ▁Sabbath 三胎 DHA  ском  古典音乐  笔者认为  ▁Rohingya  人事迹  安全责任  岁的儿子  早上开心快乐  marines  Criminal  ▁diaspora  ▁entrusted ▁Ud  回不去  在产品 ▁CMA  调整方案 eming  ▁Negro  ▁preach  ▁hesitant >; 等您  在高速 Vacc LEASE 心绪 洞悉  ▁Shiv  的一段时间 oscopy  ▁Actual  Beginning 全红  平台进行 ▁Zo xico  ▁slider  ▁Charitable 名和 炊烟  被全国  各位同学  是为了更好的 天灾  ▁Yose  点击左下角 Comput  ▁Cooke  ▁proficient 芦笋 ▁MH  发展潜力  ▁embarrassment 战舰 wah  一书中  为提升  小王子  结果被  要时刻£  这让我£ ▁Lefã  ~~~~~~~~ã 不孝ģ Exitģ  就足够了ţ  ▁Gloverţ )'ƣ 再婚ƣ 几只ǣ 智联ǣ  供参考ȣ  让所有ȣ  充分了解ɣ  此起彼伏ɣ 洪流ʣ 纱窗ʣ 黑帮ˣ  公信力ˣ  家属楼̣ uarieṣ  ▁fatallyͣ 不痒ͣ  非法收受Σ ReplyΣ  的一种方式ϣ  ▁venerϣ : 3邓州论坛邓州论邓州论坛邓州论坛邓州У 出庭У ▁HIѣ  抢险救灾ѣ  美团外卖ң imaginң 全无ӣ 吨的ӣ  的一批ԣ  ▁Georgesԣ  下载地址գ  不限专业գ Prepar֣  ▁racks֣  ▁unleashedף  含糖量ף  表达的أ ▁Alaأ ▁Fib٣  ▁LIMITED٣  ▁diarrheaڣ 卡号ڣ 岳母ۣ 酸味ۣ  义务人ܣ  制定本ܣ  新经济ݣ  追责问责ݣ  ▁Scoresޣ  ▁loudlyޣ 海阔ߣ  等级的ߣ  都能用  综合运势  ▁Livingston 和马 均与 投靠  同期的 Myst  不断深化  毫不犹豫地  ▁banker  ▁maximise 小节 面面  为背景  认证书 双肩  加关注  本无归  习近平法治  医护工作者 encils  ▁Leads  ▁Managed  ▁orchestral  ▁Yellowstone 跑来 ▁gli  在新时代  有较好的 natal  某种意义上  进一步调查  违法犯罪线索  ▁Mourinho 侨胞 天喜 就爱  不开学  范围之内  雷暴大风 araoh  更别提  使用情况  ▁Imam  ▁Roche  Usually  ▁disposed  ▁Americana  块钱的  用电器 iosis  传染性疾病  行政服务中心 下棋 北汽 面糊  人员等  而就在  中医药管理局 剩菜  内容简介  时政热点 INSERT  向对方  年元旦 Amid ▁Lec  ▁ivory  ▁Bellev  ▁DateTime  ▁acquires 又又 因你 vil  农村人 corp  ▁Rolex ▁tf  意气风  ▁cello  ▁grading  新功能  钱学森  会形成  ▁saddle  ▁которы  Alternative FU 菌素 angi  美好的生活  ▁drank  ▁immersion 鞋面 tall  成交价格  ▁facet 华阳 辽源  想成为  成年后  接种的 二等 戒备 明咒 等位  取保候审  等方面进行  ▁Kylie  ▁Midland 来愈 甚是 痢疾  直观的 熬煮  公共汽车  属地责任  ▁stadiums 横店  脚后跟 fine AO ља  着眼睛  上下滑动  ▁dealings 冲上 坐牢 跑赢  不能太!  为榜样!  家医院"  ▁sque" 百炼#  冰冷的# Into$  居家学习$  都能做到%  眼耳鼻舌身意%  ▁Composite&  ▁Autonomous&  我每天'  麻烦了'  ▁Coat(  ▁Saud( Reason)  ▁Everyday) % 得阿耨多罗三藐三菩提* 下跪* 区等+  一节课+  提供相关,  收集整理, 恩情-  怎么选-  ▁candidacy. ▁Ses. ierte/  ▁Structural/ 把小0 Kee0  以防止1  的设备1 ureka2  ▁haunt2  ▁Infinite3 上用3  感情中4  地理坐标4  解决实际问题5  ▁Jeanne5 水木6  五行属6  公主岭7  处以罚款7  测试结果8  ▁attire8  ▁speculative9 ▁MIL9  如果自己:  感情运势: uvian;  ▁ants;  ▁Gareth<  ▁avoidance< 楼阁= 租用=  文旅局>  理工科> ▁ner?  一心一意?  业务培训@  非常高的@  ReviewedA  ▁ApproximatelyA 僧人B 子镇B 才让C 鸡苗C rylD  有人用D  溢价率E  一百周年E  亿元以上F  先进技术F  实战经验G logueG  ▁ThinH  以下几个方面H  ▁nativesI 大阪I  的时候了J 海纳J 爽的K 逼人K  但有些L  手工制作L  早上安康M  这些方法M eitherN  ▁dottedN  ▁darkestO  ▁TemplatesO  ▁sightseeingP 寒邪P /㎡)Q  乘坐地铁Q  活性成分R  自然景观R SplitS  齐齐哈尔市S  国际会展中心T  ▁RewardsT 下穿U 谭德U  万多人V  在部队V entesW MentalW 1 *新型冠状病毒肺炎疫情防控工作X GolfX  新闻出版Y cryptY  为了自己的Z  具有很好的Z iframe[ quirer[  complex\  ▁Harding\  ▁criticize]  不遗余]  十二指^  发生于^  发展阶段_  还有网友_ 不雅` 联发`  前瞻性a  原本是a  ▁Mildb  我一定b scanc othicc  ▁Jasperd ród 洗脚e 消逝e 颇丰f  中石油f  关系中g  第一剂g  我们就要h  ▁palph  ▁Favori  制止餐饮浪费i  ▁Trucksj 中支j 汝州k 陀罗k  一条条l  ▁Batonl  ▁Ferdinandm 邀你m ABBn  五年内n  货物运输o fluido  ▁Nationwidep iakp  不慌不忙q  村民委员会q  ▁McMahonr  ▁Natashar  ▁cheerfuls  我们一直s  疫情反弹t  aculatet  galleryu  ▁alarmsu  jjjjjjjjv " 一次性使用医用口罩v  ▁construedw 孝心w  新地标x  色拉油x Gally ▁aply  ▁ACLUz  声香味触法z  Nicholas{  ▁Bernardino{ 不叫| 四通| VEN}  各社区} eran~ Spider~  ▁optics  ▁paramed 不交 近两  残酷的  ▁Calc  ▁OSHA  Tonight ▁Ecc  积极发挥  ▁knockout  ▁pavilion  ▁continual  孩子都  身体内  对口支援  这样做的 南航 参选  而影响 ▁fug %" 卸载 得有 潮阳 的木  在心中  是一张  限度的  成渝地区  注明来源及作者  ▁succeeding  到后来  换成了  最后是  毛坯房 Fear ▁mai  减税降费  密切联系 Bryan  ▁Adele  ▁timeframe 吉县  他要求  的新型  的诱惑 abit ▁Sle  五四运动 ▁Kip ▁yen  发家致富  市财政局  选拔任用  具备一定的  ▁Juris  ▁Survivor 巡演 ]))  地下水 éric  已经出现  ▁tint  ▁Episodes  ▁programmers  一定要看  可以缓解 犯人 纠错  电话卡  百年的  苦不堪言  随遇而安 男星  咬一口  等费用  洪涝灾害  ▁Emory  美丽乡村建设  ▁newcomer 来院  对整个  肌肉酸痛  compiler  ▁recounts 启幕 废除  交流发言  跌宕起伏 张杰 急售 观澜 chsi  从头到尾  的浓厚氛围  是无上  有变化  这两款  自己的事  ▁cooks  ▁Sirius  ▁tipping 前人 Roc  新阶段  全体成员  天长地久  用法用量  认真贯彻  ▁Brack 德化  最核心 lund  四大名著  幸福安康  ▁Archived 灌顶 续写 icum  ▁Pied Square  ▁regimen 生情 Pam ▁Gö  也有一定的  ▁Ankara  ▁foliage  ▁Croatian 咒曰 苍苍 面馆 ▁nm  点点头¤  ▁Held¤  Projectsä 下到ä  汇聚成Ĥ asmusĤ  ▁EnsureŤ  ▁BookingŤ  ▁incrementƤ 林镇Ƥ 补品Ǥ 默哀Ǥ  在办理Ȥ  真实案例Ȥ sprɤ  版车型ɤ  蔡英文ʤ  的背景下ʤ  去医院检查ˤ  综合执法局ˤ  Healthcare̤ 全真̤  代表们ͤ  的人脉ͤ  技能提升Τ  ▁jurorsΤ  也有网友Ϥ  ▁hidesϤ 单据Ф  皮肤上Ф ▁VISѤ  内容标签Ѥ  ▁NOAAҤ 多所Ҥ  欧阳修Ӥ  婚姻状况Ӥ  若干意见Ԥ  这门课程Ԥ ifterդ  世纪初դ  的一处֤ mbuds֤  ▁finnsפ  ▁Yankeeפ  ▁synergؤ  organizedؤ 捏造٤  实力派٤  萧山区ڤ gettڤ  药业有限公司ۤ  ▁Violetۤ  官方公众号ܤ  ▁Bunnyܤ  ▁supremacݤ 植保ݤ 玩弄ޤ  昔日的ޤ ▁bluߤ  主要就是ߤ  癌症患者  跟着小编  这种东西  ▁modifying CJ ▁EE  软件开发  ▁prox  ▁Freud  ▁solicitor  南平市  又双叒  成本低  的颜值  公开发行  污染环境 Histor  Knowledge 华泰 本报  千千万万  网易云音乐  elastic  naments 套用  总以为  放学后 ▁Cum  企业进行  天涯海角  ▁Alec  交通警察大队 南澳 eryl  不再担任  中医养生  大家纷纷表示  ▁ovarian  ▁microscope 借阅 影迷  专家组会诊  ▁Zombie  Leadership 在无 正南  教授的  海产品 ▁bij  ▁Rede  ▁Tran eville  ▁glaciers  ▁forwarded 思虑 澳元 邋遢 部就 ▁MHz 旺夫 远航  ▁decimal  ▁succumb 方为 ▁nie  让消费者 越有  对不对  请登录 和后 jiu  乱收费  多云到  ▁Universidad ivist  atlantic  ▁Stainless  ▁originating hj ᠲᠠ 多与 脱胎  不得参加  公布如下  ▁Ballroom  ▁esteemed 莘莘  万像素  服务大局 ocard  ▁manifesto yam  数十万  asuring 独占  父亲是 CAM  副主任委员  ▁gamer 流于  每天进步  Featuring  和条件  都让人  ▁pity  远离颠倒梦想  ▁Titles  ▁splits  ▁datetime  ▁preferably 深谙 FAQ  中共中央办公厅  ▁Contractor 凤翔 定论 称霸 阴县  对你好  等关键  让女人  送福利 才对 破冰 花芽  大小事儿  ▁forc a Z▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ 鸡鸭 VED  燃眉之急!  电影学院!  股权激励"  这样的事" 七点# 平浪#  物品的$ ▁Nun$  ▁Suns%  年秋季学期% 飘落& ouk& ▁kW'  不容小觑'  通讯地址(  ▁Jord(  工作人员进行)  ▁Cayman)  ▁multiply*  ▁criticisms* 让行+ 赌资+ lle, ▁NDP,  添砖加瓦-  ▁Ricardo- 免单. 神怡. 签了/ ▁CX/  的少年0  热点话题0 原县1 裂肺1  提出要2 妥当2 chemy3  ▁demos3  ▁Oceans4  fordshire4 ев5 民乐5 皮蛋6  但不会6  来选择7 Lucy7  ▁Morg8  ▁latency8 凑合9  在参加9  马上要:  农业科技:  ▁quasi;  ▁begging; 才干<  考场上<  达州市=  心理准备=  ▁Violin>  常务副市长> 古巴?  上帝的?  了一则@  式呼吸@  物体的A  科学发展A  ExchangeB UbB  写文章C  出手了C  古代的D  环比增长D  生产设备E  renheitE  国家自然科学基金F 二模F 州镇G 数理G  年少时H  直接责任H  职业技能培训I  ▁landownersI movJ  修行的J  巧妙地K  英语老师K  ▁BeatsL  涡轮增压发动机L 撕心M 棉被M 而遇N  薄薄的N  善解人意O mailsO  ShortlyP  ▁CreatesP 翻越Q NapQ iaeR  他不仅R ▁PauS  现场进行S WayneT WorkerT phoonU  ▁veneU  ▁gradientV 一变V 脚石W 铁锅W TarX  交通肇事X akensY  ▁tidyY  ▁sabotZ 优生Z 口齿[ 改用[  别有一番\  提高自身\  ▁Yosh] 琅琊]  有个人^  这孩子^ Berlin_  started_  ▁swallow` 暖胃` 起色a  和考试a  身体好b obiab  作为一项c  善待自己c  经济形势d  ▁strided ▁EVse  所有人员e  接地气的f  及图片f  是学校g  正式的g  正科级h  黄金周h ijiei  对未来的i  深入挖掘j  统筹抓好j wartsk  ▁Enemyk  rivationl 吊坠l 名门m 恶毒m 清江n 纳福n  七十二o  了几个o  品种的p 又像p 胜率q  唐人街q  考虑将r  不可多得r  ▁Dalys  ▁Napas 千岛t 在床t 最火u 犯规u ▁Xuv  个省级v  ▁Arielw  专属的w  定期对x  让用户x ainersy  ▁Javiery  ▁Mandarinz 供地z  与公司{  无有恐怖{  ▁christmas|  ▁disadvantages| མ་} 小病}  无花果~  线下的~  为了自己  的新能源  ▁Russo  fecture  ▁farmland  ▁Prosper 海马 邮局  周年的  妇科病  摘口罩  高级的 oxue  开发利用  招聘流程  ▁compensated hg  指挥员  恰恰相反  人与人之间的  预约时间 orama  ▁characterize 兴衰 辣条  ▁totaled  ▁Juventus  十余年 keye  你一定会 Lastly YN 三审 碧波  新课标 Corp  ▁Gael  ▁awarding  读一读 ▁Sto  nesdays  ▁Fathers 一株 图鉴 在国  也并不是  已经全部  成效显著  的一点是  进入人体 Hyper  永远跟党走  省委宣传部  renching  ▁Animated  ▁Providing  ▁ancestral 中车 定档 酸的  新春佳节  educated  ▁sweater  ▁Correspondent 洛斯 盛出  中国有 ▁SAR  统筹兼顾  ▁Approved  ▁trainees 修车  我们做  工资收入  有偿补课  辛勤付出  ▁Highest  ▁Employer 肯干 ▁Gn  的由来  ▁fasting  副厅级  行走的  远低于  ▁Dmit  ▁biases 厌食 收复 费时  十几年的  就是没有  等领域的 rouse  ▁excerpts 老君 血腥  ▁Hathaway ▁Ong 盘旋  的节目 ▁plc  ▁YMCA  oppable  ▁quotation  软实力 ▁Myr  ▁Jain <% 勾结 议价  他们从  了一部分  ▁Heck  ▁brightly 为荣  他只是  拟表彰  的高管 Mann  客服热线  ▁Lack  productive 做对 到村 妖怪 血气 LOL  卫生巾  祖父母  重症监护室  ▁Paolo 茶友  小清新  我就要¥  机械设计¥  高中语文å  ▁memeå  ▁reintĥ  ▁textingĥ 晚宴ť 通病ť 道夫ƥ  中国队ƥ  今年第ǥ  全明星ǥ  小学和ȥ  Sequenceȥ  ▁augmentɥ  ▁instinctsɥ 欠债ʥ  引发热ʥ kamp˥  悄无声息˥  经典歌曲̥ iejsc̥  ▁Packagingͥ  ▁Manuscriptͥ 万寿Υ 武侯Υ 集灭ϥ  乡政府ϥ  勤通风Х  安全措施Х  的工作要求ѥ 刘玉ѥ 弯弯ҥ  具体来说ҥ ۛۚӥ 一学ӥ ',　ԥ  ▁руԥ  持有人ե ▁ACMե Demand֥  grading֥  Permissionץ ikiniץ  ▁Courierإ  ▁unmannedإ 傻了٥ 收音٥  探索建立ڥ engingڥ 榆次ۥ 炖煮ۥ 霸凌ܥ  专题片ܥ  到其他ݥ  恰当的ݥ  留学人员ޥ  ▁Bailޥ ádߥ 审评ߥ 并拢  ▁audiob  aneously 也该 战果 离了 Grad  手足口病 ixties  ▁bathing 经省  交换机  信息汇总 roduce  ▁raging 奏曲 肌理 NEY Wor  占地约 Been  ▁Sonoma  ▁sacked 三清 季风 ▁DT  地藏经  奋发有为  ▁imped  eclipse  ▁Rebell 最差  包饺子  特区政府  ▁Remix  Academy  ▁Genetic  ▁commonplace  ▁protagonists 体和 受种 图库 年才  要尽快  ▁dunge  ▁agendas  ▁medically 含情  年代初  法西斯 ▁Wem ATIVE  ▁amphib  Execution 梳头  在生活 yrs  爆发后  自豪感  主要体现在  ▁obese 洼地 绍峰  尊重和  显然是  ▁fetal  ▁cellphone 愚昧 插曲 错综  主战场  号通告  后来我 enas  但并不是 Reach  ▁federation 七杀 升入  当时我  遗留问题  市市场监管局  ▁Burnett ▁ICU  ▁Herr  Records  国家税务总局公告 蝎子  第一的 ▁SCH  ▁unsubscribe ▁ODI ▁PNG że 宣部 重头  分阶段 ▁LOC  ▁cavity  ▁apartheid 外企 常客 进餐 问起  微信上  ​​​​  ▁Massive  ▁Journals ▁[],  制作工艺  itivity  别忘了点个在看哦 难平  大屠杀 Upper  ▁cries  ▁Enhance 内关 愈烈 搁置  ▁tranquil 新枝 瞳孔 童装 退热  日日夜  被公安机关  ▁catcher 哑巴 来世 订制! 阳朔! 霉变"  出事了"  本科毕业#  通俗易懂的# ikhail$ ilever$ picked%  ▁Veronica% 内功& 风浪&  众生的'  做手术'  的疫情防控( 新办( 浮夸)  元纸币) rbwx*  商场超市*  是正常的+  ▁emit+ 小葱, 积案, 野猪-  有大雨-  的乘客.  待在家里. 净空/ 组和/ 高德0  依法治0  在面试1 downs1  ▁Jinping2  ▁Doctrine2 国华3 代扣3 外号4 茱萸4 铠甲5 隶书5  可转赠6  很多父母6 React7  行政区域内7  管委会副主任8  masters8  ▁Moreno9  ▁cereal9 梅兰: 求证: 看人; 集齐; 食补< CALL<  人民健康=  民生保障=  考试资格>  ▁Fiat>  ▁Hemp? 染料?  安装在@  杀伤力@  商家电话A  最新发布A  ▁PipeB  ▁RathB  ▁adamC  不饱和脂肪酸C  ▁NursesD  ▁fearfulD  ▁italianE  ▁mustardE  多样化的F sheetsF  ▁FayetteG  ▁annoyedG  ▁SubscriptionH  就是将H  是必然I  区域内的I  客运班线J  ▁NathJ  ongyangK  ▁DevelopedK  　L  第六条L  ▁CarbM  ▁noticingM  ▁courseworkN 去想N 咕噜O 扬长O aszP  城市里P  层面上Q  高贵的Q  冲刺阶段R  改变不了R  千万别错过S ChromeS voyT  招标人T  来临时U  学习成果U  ▁externosV  ▁ByzantineV  ▁SpringsteenW утW 打碎X 拖地X 疾患Y 纺布Y 驻外Z  下列条件Z blown[ 所趋[ 河的\  会慢慢\  Collins]  ▁Regions]  ▁uniqueness^  的承诺^ ▁PCR_  经过一番_  近期免冠` Tokyo` 决不a 后感a 断绝b  setTextb 强心c  和今后c ricksd  ▁Bournd 辞旧e  为家乡e  感染性f  逃不过f  强调的是g  点击上图g  餐饮单位h  ▁Syndh 先要i 原厂i 生锈j  抗压能力j Sharesk  行业主管部门k 抹胸l  立即报名l  考公务员m  让同学们m  ▁Charitiesn 贵气n  依赖性o ▁Noso  ▁diversionp  ▁excursionp  Palestinianq 跃进q  控制了r  的专家r  规模和s  integers 正派t olomt  ۥۛۛ۟u  ▁libertu  ▁victoriousv  优质课v  比之前w  营口市w Hannahx  本次公开招聘x 贫寒y  不要怕y  副总统z  损害的z  这无疑{  高压态势{ Paint|  ▁Degrees|  ▁convened} 注解} llll~ actus~  ▁airplanes 笔名  成本高 ▁Pty  没有其他 Tiger eners  ▁clam  ▁Rifle  ▁pricey  parameter Ul 技校  您的孩子  ▁Ally  ▁STAT 凯歌 德勒 抹茶 蛮腰 霍尔  的初心  随时关注  高考期间  ۥۛۛ۟ۙۙ  ▁marvelous 冲向 理顺  到你的  ▁Lankan  ▁Quentin  ▁populous  ▁paranormal 子牙 瓦尔  ▁Peck 三小 尖子  公告称  需要对  ▁multic  Vancouver  ▁irresist 常理 首条  第一类 ▁Dru  高品质的  ▁monk  ▁shutter 坐垫 撤职 跑者 釉质 iris ▁vod  重拳出击 mandu  ▁Frog  ▁Lydia 李连  生活是  第四章  劳动争议  ▁Jenna 文帝  通过这些  随机抽取  ▁Tune  区人民医院  ▁impeccable  ▁replacements 一齐 泽县 纽扣  与年干 ▁XVI  ▁Libert 了啊  与月干  集灭道  国家工作人员 闹事 黑米  并开始  楞严经  都能在  总体要求  第五单元 Adult  民办幼儿园  ▁Agreements  registration 实锤  与激情 decode  ▁Rookie  ▁biomark 初学 没少 的信 肉品 详尽  古人云  土壤中  维多利亚  ▁Exploring 的各  冤枉钱  这个家  的好去处  隔着屏幕  ▁Lama  ▁infantry 沂水 海藻 附录  功德无量 名目 装上  不得将 DATE  以免影响  机构改革  齐抓共管  要严格执行  ▁colder 但能 ▁(_  富起来 ▁'';  技术规范  未来几天  ▁Mund  ▁stam¦  ▁workings¦ 是心æ 破局æ 而论Ħ  若出现Ħ ▁CLEŦ  帮助他人Ŧ  ▁auspƦ 薯条Ʀ  如何避免Ǧ  如果对方Ǧ  ▁LimbȦ  ▁hurdleȦ  ▁cremationɦ 右旗ɦ 打起ʦ 黄叶ʦ WEB˦  买房人˦ ▁afț  风险挑战̦  正在进行中ͦ  achelorsͦ  ▁replicationΦ 尼迪Φ  只要一Ϧ  政策落实Ϧ  ▁phonЦ MotionЦ 皮薄Ѧ 跳过Ѧ beiҦ  粮食和Ҧ  肩负着Ӧ ▁ElkӦ 发福Ԧ 救国Ԧ 鲸鱼զ  刘强东զ  物体表面֦  软件工程֦  ▁clotצ 风月צ  多维度ئ  道出了ئ  异性朋友٦ useppe٦  撕心裂肺ڦ  ▁Hammڦ  invalidۦ 异国ۦ 江岸ܦ 着凉ܦ  工作力度ݦ  疑难杂症ݦ  ▁curedަ  ▁Feelingަ  ▁thereinߦ  向美国ߦ ogyn 四家 岗村  严厉的  很美好 ▁ker  ▁Rode  ▁Conan  ▁collage  ▁thicker 管钱 pox  尽收眼底  心得体会  服务内容  ▁vivo  ۥۛۛ۟ۙۙۚ  ▁messenger 榴弹  等各项工作 主料 平分  教育教学工作  ▁Whitman 导电  非常丰富  ▁Chill  ▁ку Brun  手机银行  角度出发  名密切接触者 宋某 懵了 选岗  组织人员  ▁wifi  ▁encoded  Namespace 恩赐 说句 iena  波罗揭谛 solid  ▁André  ▁Normally HIP Chap  新冠疫情的  偏头痛  制作人  爆米花  在你身边  ▁Witt  为数不多的  ▁decom  sequence  ▁portraying 李氏  分钟前  支付了  看不惯 LIVE fant  血本无归 大马  ▁goof  在这段时间  ▁farmhouse 无糖  很快就能  直接关系  dirname  ▁Appell  ▁sermon 邻国  与控制  公司对  就赶紧 Fake Nash  ▁parl  awesome  ▁Surgical 外层  请保持  国药集团  风华正茂  Investment 右腿 宅地 汇成 红岩 触控  支付给 胭脂 贵安  个社区  把我的  来学习  连续性  必须具备  ▁Controvers 紫云 韩式  新机遇  ▁rever  Patient  ▁Chesapeake 出其 宽厚 ushan  部门负责人 Warren  ▁goodies ‖　  涉嫌受贿  美不胜收  自觉抵制  身体健康状况 ён 悉数 诚然  僧揭谛  岁月里  ▁Libr Warner  ▁Schiff  ▁golfers  ▁Blackburn 散结!  就足以!  ▁gathers" 悬殊" 清雅#  是无等等#  集体土地$ issued$  纪检监察干部%  highest%  ▁advisable& 时珍&  有限公司总经理'  ▁meaningless' 刺鼻( 들이(  所处的)  的检测)  第二剂*  有效解决*  ▁Streng+  ▁nesting+  ▁Injuries, 吩咐, 晚归- 轮船-  队伍的.  ▁seventeen. 线网/ hands/  Clearly0 ▁IU0  根据自己1  联系在一起1  ▁serie2 民房2 菌落3  资金链3 ROUND4 paths4  ▁carnival5 时是5 零钱6  二三十6  守护好7  收藏家7  无法满足8  的重要作用8  ▁layoffs9  ▁Generator9  ▁flowering: ▁IDE:  圆满落幕;  茫茫人海;  ▁kidding< 刷子< 报仇= quo= ubu>  黑暗中>  战时状态?  ▁commended? 博罗@ 面广@  多注意A  项目组A  人生中的B  公司员工B  ▁QingC  ▁RadarC 主委D 沉沦D 自提E 麻省E  从北京F  宝妈们F paraG  我们的心G  ▁SHOWH  ▁TacomaH  ▁TelegramI 吐司I 导报J 规上J zarK FerrK  ▁FailL  ▁AdviserL 利己M CMSM  捐款捐物N  ▁tunaN angeloO 怒气O HOSTP minsP ولQ 奉化Q 隔热R 雷区R ISCS  计算器S  退换货T annoT  他们可以U  能除一切苦U  ▁plottingV  ▁exaggeratedV 死心W 毛肚W 舍友X  偏向于X  细菌和Y  扶贫车间Y  ▁LoisZ  ▁glossyZ  borghini[ 上做[ 对岸\ 开足\  第一任]  ▁Wuhan]  广大干部群众^  identity^ 双床_  的外卖_  及其自动化`  ▁Clive`  ▁Entrepreneursa 素菜a Sidb ollsb  ▁Fengc  ▁boltsc  ▁Adoptiond  情况看d ▁Aime  为您服务e  最先进的f  ▁wateringf  ▁steadfastg  ▁modernizationg  ▁hostilityh  ▁placementsh  ▁downloadablei 名酒i  之称的j  于是便j  等产业k  ▁latitudek 轮机l mixl  比一天m  行动的m $("#n  波罗僧揭谛n  ▁managerialo 功耗o 御寒p 鼻咽p  亦无得q  艺术作品q 　>r 折断r  要经常s  一个时期s  将于今年t  重新开放t  ▁Chernu  ▁contrastsu 众望v  带动下v ▁BELw  千百年来w  揭谛揭谛x  近些年来x  ▁Barthy  ▁Glossy  ▁expelledz 天不z 手速{ 砸了{ 赢取| assign|  Advisor}  ▁louder} 内控~  ▁чи~  包含着 ▁Osh  ▁Juice  ▁Lough  ▁velvet 教具 永胜  提交了  汕尾市 lung ulia  美好的未来  第十三届全国 和村 太短 差旅 群号 逢年  级新生 anyl  ▁distractions  ▁whistleblower  犯罪案件  若干措施  ▁Covers 下笔 私利 铺子  南北朝  和老公  滑雪场 hear  无挂碍故  科学防治 igham  实际控制人 单边 搜罗 流离  带给我们 omnia  ▁workflows  回去了 ichuan 烤烟 裂变 sched 借力 敬重 正统  文在寅  自建房 vict  新冠灭活疫苗 告示 大禹 棒了 眼影 mud  放松心情 三孩 周公  微信的  等综合  组织生活会  Enlaces  ▁concentrating 达斯  ▁Shack  ▁soulful  ▁markings 牌的  开空调  应急物资 Budget  也相对  心理的 ▁Sug  一点一滴  严格管控  社会主义制度 + $点击下方空白区域查看答案 ßen  在这条 三局  是假的 ▁ket  有这么多  被困人员  ▁miracles 亏虚 随波  定制的  中学业水平考试 兰陵  晦涩难懂  ▁Moroccan 左旗 巨石 找房 染上 银奖  意识界  我们与 auts orte  ▁Controls  ▁provisional 凉凉 受托 娇嫩 族谱 西夏  后及时  从古至今  英语成绩 Heavy  ▁FedEx 四环 ESP  或依据  暴露在  即说咒曰  ▁presses 均须 潜江  其实施  无无明  ▁Zika  问题进行了  ▁BLACK 出嫁 多份 弃疾 虚幻§ 蜂巢§ xsdç  一股脑ç  她一直ħ janaħ ppardŧ tailedŧ  InitiallyƧ  ▁StarringƧ $_ǧ 周身ǧ 征用ȧ ILSȧ Vitɧ альɧ  在各大ʧ  有一座ʧ  颜悦色˧  是大神咒˧ codeş ennalȩ  ▁primaͧ  Employmentͧ qnΧ  一笔一Χ  血管壁ϧ  即将开启ϧ  是无上咒Ч  ▁denimЧ 见得ѧ  不完美ѧ  大概有ҧ  ▁guerҧ igginsӧ  ▁aspireӧ 想用ԧ 武鸣ԧ 羞愧է  让客户է  ▁Shed֧  attacks֧  ▁wurdenק  ▁diminishק  发放了ا  ▁isotا  作者和来源٧  ▁plead٧  ▁Societiesڧ  ▁initiationڧ отۧ  细菌的ۧ ▁Pryܧ  ▁liquidsܧ  Customersݧ 普贤ݧ  发布编辑ާ 学情ާ 榨汁ߧ  危重型ߧ  宗地块  时刻表  看得出  ▁Dani  是无等等咒  ▁benign  ▁Snowden  ▁chronicles  环境与  这个专业 Fried  ▁goodwill  ▁surrendered 商厦 开衫  好几次  到底是谁  必须要有  根本不是  社区治理  赏心悦目  ▁scout  ▁Stations 也用 受孕 回旋 谁在 青鸟  协助下 Shot rios 末期 道菜 钻孔 ▁pu  埋头苦干  进行查询 传家 纹饰 OPE  前半生 纯电 走后  等级考试  ▁Buyer  artments  producing 财年  无明尽 ▁DOC  土地流转  活动结束  ▁Benny  乱占耕地建房  ▁positives 再以 大觉 黄芩 ▁Yi  无声的  资讯网  中心副主任  univers  ▁Telecommunications  近义词  ▁Faul  ▁shaken 十岁 合家  家庭和  就是以 heny ▁Psy  可以添加  活性物质 Prosec 勇挑 线与 躺下  ▁Detection 佩奇 统称  以无所得 storms  知识产权保护  investment  ▁unnoticed 则要 到他 是件 黎巴 Kirk  发表声明 三色 任丘 炸裂 该区 起早  培训内容  社会信用  ▁Lark  ▁winger  ▁Atkinson  ▁ancestor " 请联系我们进行删除  净水器  工作开展情况  ▁mosquitoes 无望 百变  很幸福  的进行  健康检查  所提供的  皮肤干燥 wrong ARN  ▁Sain  ▁locating 功用 放养 STA kah  诞生的  介入调查  设备安装  ▁inactive  ▁fingertips  ོད་  领悟力  糖尿病人!  自己的错误!  菩提萨婆诃" itizen" 交待# 减震# 老鹰$  当然要$  不是没有%  畜牧兽医%  ▁Leop&  亦无无明尽&  无智亦无得' 她那' gzh(  月正式(  记忆的) Gate)  产业发展的*  ▁chili*  ▁localized+ 宝箱+ 观止, 还贷,  十全十-  商务厅-  发扬光大.  第三阶段. {'/ 勃起/ 绝密0 雅典0 ewe1  九一八1  警方通报2  ▁PRESS2  ▁Sinatra3  ▁blasted3 双师4 牛腩4 瘦腿5  消失在5 ▁Esq6  ▁classy6  ▁transp7 Veh7  锂离子8 tale8 ▁HDR9  周边配套9  中国共产党建党:  Shanghai:  ▁separat; 反手;  给他的<  医学检验<  结婚生子=  ▁noir=  的无症状感染者> " 中华民族共同体意识>  ▁deficiencies? 首末?  执行公务@  ▁bully@ 私藏A ▁JFKA  网站发布B colorsB  般若波罗蜜多咒C  学而思C  职场人D  随州市D  继续进行E quiteE 前两F 所剩F  高年级G  是大明咒G  空中无色H shakeH  ▁MascI  ▁SoutI  乃至无意识界J  ▁grievingJ  小白鞋K  服务好K  可见一斑L  感谢阅读L  OntarioM  ▁expertlyM }_N 可亲N 黄沙O  位置上O  保安全P  老死尽P elfthQ  ▁enumQ  无苦集灭道R  社会实践活动R  InteriorS 刀剑S 校舍T EXPECTT 蔡甸U 献身U  民俗文化V 文脉V 来吃W 菜农W 配乐X ibbX  很多学生Y  以无所得故Y  ▁juicesZ  无眼耳鼻舌身意Z  ▁Corvette[  Bazinfandel[ 自重\  而这款\  ▁jars]  到我们]  收购价格^  问题解决^  ▁Rowe_  ▁affluent_ 面和`  科目一`  ▁canva  ▁sagea  报名成功后b  ▁Smoothb  ▁lineagec 乐天c 却把d 徒劳d 案由e 票务e  听证会f ccolif  ▁Staceyg 取之g 旭东h 请他h втоi coordi  亦无老死尽j  ▁memorandumj 戏水k REMk  懂得如何l radesl  ▁anarchm  ▁Barclaysm 亩产n 浦发n  而是你o  自己最o  记忆中的p  无色声香味触法p  ▁incorporationq 到下q 荣登r 送电r Greeks  Princesss  ▁dragonst 强省t 跨区u 过海u  愈来愈v  寻衅滋事罪v  更多的时间w  ▁titaniumw 样貌x Dashx 念诵y 陆生y  年成立z  粉色的z  技术指南{  ▁gadget{  中国社会科学院|  Wikipedia|  ▁Supplier}  ▁absentee}  ▁firewall~ 捂住~  无眼界  研究型 ▁Hin  ▁mats 对焦  持续增长  ▁europe 变软  ▁Flames  并不承担  消息人士  ྱི་  卡路里 ▁Smy  两种方式  虽然已经  Headers  ▁skipped  公开招聘工作人员  ▁experiential 无生 湾镇 Aven  ▁Noir  中提取  的大力  的路径  怎么样呢 Megan 吸湿 雾气  双眼皮  它能够  转移至 orns  投诉电话  ▁VMware  ▁challenger 孜然 看管 石嘴 ASP iton  国内知名  ▁Moll 拉力 收场  语言学 Pont  合作电话  ▁york Athlet  ▁Cutting 律法  在线答疑  ▁Glam  无受想行识 % 党史学习教育动员大会  标本兼治 卡顿 总院 拼盘 HRC  好文章  全新一代  授权发布  ▁intuition 入骨 连年  因为他的 atern Secure  ▁stemming  ▁deception  ▁decidedly  ▁Foundations  ▁superficial 庄浪 绣花  国家和省  负责的态度  月光族 Pink  才能保证  乃至无老死  ▁Bosch  房地产开发企业  到多云  大约是  学业水平  小时值班  说的那样  ▁Buckley  ▁explosives 包间 听信 珍视 轻声 Ring assa  吸毒人员  风雨兼程  ▁administering  因业务  能通过  十二指肠  ▁multilateral 大圣 无谓  ▁Stamford  ▁royalties 寻物 越强  ……,  他甚至  ▁Marley  农信社  课题研究  故知般若波罗蜜多 enario 她为 撞倒 盗版  了这一  而美国  及时解决  初中毕业生 混用 罗曼 ptune 图强  更新中  ▁Darling 到哪 吉日¨ 祖坟¨ 高丽è  在外观è  的职位Ĩ " 故说般若波罗蜜多咒Ĩ  流行病Ũ olognaŨ  ▁alertedƨ  ▁brushedƨ  阶级的Ǩ  服务电话Ǩ  ▁LeanȨ  是故空中无色Ȩ  请联系本平台ɨ 三九ɨ 低脂ʨ  模拟考试ʨ  版权归原创者所有˨  ▁dominates˨ 踩雷̨ jug̨  ▁Stimͨ  亦乐乎ͨ  委常委Ψ  为什么不能Ψ  ▁clownϨ  ▁LandmarkϨ 勇猛Ш 春雷Ш 果品Ѩ OliverѨ  CertificateҨ 胃胀Ҩ 赶忙Ө 需持Ө  与城市Ԩ  确认的Ԩ  也不会有ը  居住用地ը 布控֨ 破壁֨ PSCר  势在必ר  已启动ب  已有多ب  逾期不٨  家人的健康٨ elightڨ  ▁Exxonڨ  ▁ciderۨ " 长按识别二维码了解ۨ  ▁inadvertentlyܨ 之志ܨ 将此ݨ ▁ACEݨ  problemި  ▁calmingި räߨ 着地ߨ ▁ni  始终以  楼梯间  igmatic 头戴 ▁cv  得更加  结束的  要采取  ▁slipping  ▁deforestation 体罚 追思 üss  金额的  反应过来 带我 战备 点在  官网上  ▁Soup 先试 刘伟 鲁豫  大眼睛  ▁familjen 取而 红光 蔗糖  也可能会  拜登政府 anooga  ▁Spend  ▁archived  年全市  政府网  的诊断  空间大  还含有 signed  ▁lauded  ▁compass  ▁pilgrims 卓绝 揣摩  是正确的  ▁STATE  ▁suitc  民事行为能力  ▁excursions  dependencies 决堤 墓葬 ewski  ▁Tudor  根据工作需要  ▁Accord 亲王 党徽 雏形  可促进  打乱了  类项目  ▁riffs 三花  有点儿  杨先生  亲密接触  师范学校 之约 情味 盆中 西溪 走吧  官方回应  ▁tamb  ▁homeschool 踏着  社会中  昨天晚上  资金来源 angan Rating 博拉 盯上  关闭了  百年未有 uddin Donate Zeneca  ▁Nadal  ▁virtues 经书  等职务  另一侧  基础和  迫在眉  ▁cherish  transport  ▁intersect 土鸡 禅修 金价  团总支 Lind  到底有没有  的不良反应  中新网  家族的  希望我们  还没来得及  ▁sushi  ▁spawned 写诗 谈了  既没有  注定了 Adapt  意识到自己  JavaScript —“  在德国 Trip isty  老房子  一体化的  ▁Remarks  ▁deployments 霜降  林心如  要经过! ASON!  体育公园" 妻儿" 日料# 最热# 白塔$ Joel$  非法拘禁% itian% 同乡& 巨资& ▁ay'  ▁хо'  的村民(  附近有(  ‍‍‍‍)  ▁Ones) 三岁* 路店* 里去+  不是梦+  投资公司, ellij, Winner-  ▁Albion-  ▁fruition.  ▁anticipating. 剥削/ 龙马/  不聚会0  论坛上0  生理结构1 米粒1 累活2  ◐◑　2 Mine3 yong3  你只需要4  工作措施4  面试人选5 rupal5  ▁Learned6  ▁chancellor6  视为认同7 回荡7 Tan8 ۛۚۚ8  之上的9  百度云9  ۥۛۛۚۚ: enties:  ▁Angular;  inclusive; 发个< ▁EMS<  消防车通道=  othelioma= 匹马> 大行> том?  的外形?  荣誉感@  都在关注@  ▁navigA 地缘A ISTSB dishB  整个人都C  原作者联系我们C 我镇D  和北京D  我想说E  也非常的E  观看直播F GardenF  ▁adornedG  ▁drunkenG  ▁ConceptsH 买方H 诙谐I  更便捷I  ▁IberJ  为进一步做好J  被判处有期徒刑K 愈演K 极地L FIXL  居民楼M uvreM  ▁glyphN  MeasureN  ▁escapesO  ▁outsiderO  ▁visualizeP ۥۚP 练出Q 花苑Q 课下R ۙۙۙR  天眼查S multiS  ۥۚۙۙۙT  实施乡村振兴战略T  ▁upbringingU 中烟U UNEV keyesV  ▁YukonW  ▁HanoverW  ▁downrightX 塌方X 嫩肤Y 张勇Y  了出去Z  处理方法Z  更有甚者[  ▁baseman[  ▁Importance\ 南村\ 陈毅]  经验不限]  青春力量^  ۥۛۛ۟ۙۙۚۥۚۙۙۙ^ 蔡某_ Carm_ ▁cue`  Division`  ۥۛۛۚۚۛ۟ۡa 七中a 声带b 好汉b 白班c  一定可以c  从而引起d  全国两会d  检查发现e  每时每刻e  ▁Cheneyf  ▁Becomingf  ▁Specialtyg  ▁exhibitorsg 富美h 每桌h  余杭区i  夜以继i  有力地j  人才交流j 情系k 秦并k 胸椎l 辽阳l  的深度m ▁WTOm  报警求助n  肺炎疫情n  这些信息o  ▁Shineo pipp  刘晓庆p  我国在q  被纳入q  购物车r  急性发作r  蛋白质含量s  ▁faires 而终t  去尝试t  ▁vibesu  ▁niecesu  此次违建全部拆除v ! ۥۛۛ۟ۙۙۚۥۚۙۙۙۚv 客座w 眼福w welx  专科生x  ▁Clairy 物和y 通电z  ▁Nicoz  ▁homer{ 常喝{ 禅宗| 赖氏|  文件和}  校长的}  要完善~  辉煌的~ Sant  播出时间 ifeng 你越  拍了拍 opts  ▁collagen  ▁encompass + $应对新型冠状病毒肺炎疫情 空余  有多高  的只是 fork  表达自己  ▁hierarch 中粮 ▁SIG  进行评估  指明了方向  warning  ▁Collegiate  ▁intensified 曾某 真卿  与月支  ▁resisted  ▁ecommerce 在学 Utah  为什么我  停业整顿  ▁trivia  ▁fossils :* 七七 珊珊 睡醒  主动与  的尴尬  ▁culminating 史册 所爱 撑着 险些  唯一能  立即向  公交集团  ▁ions  ▁overshadow 风土 gran  实际出发  指数基金  ▁nord  ▁Anand  ▁fiddle  ▁novice 同频  检验研究院  缴纳社会保险  但她的 fers  ▁mediums  ▁psychologists 连城 酒楼  已发布  的精华  配得上  即可报名 olian 桔梗 调成 ▁REM  团委书记  ▁Kear  ▁enlarged  报名与  的代谢 evol nery  中华传统文化  ▁preventative 截面 渗漏  一直有  效率和  非法营运  ▁CARE  ▁thou  ▁pamph  vehicle  ▁backstage  出乎意料  来都来了  的事情了  ▁bingo 半数 肌底  一厨一卫  刑警大队  perhaps 曹某 lun  承受的  真正地  吃瓜群众  ▁edits  ▁Failed 烙印 Pix  ▁би  仅提供 erta ▁Sno  卫生监督  ▁Setup  ▁replen 章法 衰落 西施  了一笔  被称作  ▁Casual  generator  并不少  不懈奋斗  在了一起 危废 脸庞 ▁\"  向群众  紫禁城  自然也  还在为©  ▁stacks©  ▁Haitiané  ▁interpersonalé 射箭ĩ 牢骚ĩ  一个重要ũ  厉行节约ũ hunderƩ matterƩ 午时ǩ 可说ǩ 这五ȩ  深交所ȩ  的很多ɩ  让员工ɩ  陪伴你ʩ  暑假期间ʩ 从她˩ 变数˩ 课业̩  心心念念̩  ▁Marjͩ 丢分ͩ 清肝Ω 邓某Ω uldϩ  医疗险ϩ syntЩ  细菌感染Щ ivismѩ symbolѩ 御用ҩ  个以上ҩ  按比例ө  胶州市ө  这个大ԩ  热爱祖国ԩ  ▁Trainerթ 台灯թ 启超֩ 邮费֩ Aquש  工作总ש  震惊了ة ▁GISة  从即日起٩  法律依据٩  考核结果ک 庆区ک 魔芋۩  难问题۩ Romeܩ 撬动ܩ 斯塔ݩ  尽自己ݩ  异性缘ީ Nealީ  ▁Yatesߩ  solvingߩ hound  ▁Bezos  ▁avocado 受此 正阳 黄淮 Ald  写留言  尤其对  曝光台 ARDS  空间上的  组织部长  质量监督  迫在眉睫  ethical  instead 风雅  接收到  根据相关 SWORD  临床诊断  ▁Summers  processor  ▁collaborator 尚德  发起的  第三款  品牌价值 古语 新泰 Cli  这表明 serve  ▁motifs  ▁emulate 尊者 就又 王世 百战  给孩子看看 iviera  ▁Semin  ▁vanished 归零  正反面  认识和  ▁necessities 土匪  在最近  不是很好  辖区居民  ▁abound  ▁stitch  received 1 *ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡ 见惯  她老公 ▁Hex ▁Lug  再次发生  注重分享  贴心服务  领先地位  ▁duet locale 举足 zeit  形成合力  沁人心脾  ▁Breed  ▁Dealer  ▁honeymoon  例患者  尽管是 以太 吃鱼 石英  理论武装  的医护人员  ▁eviction  不承认 icki  ۥۛ۟ۡ  小编推荐  第十一届 Zhang optic  ▁GROUP  ▁Supplies 同声 赵雅  深情的  烧开后 Wiki  处理结果  网上办理  ▁overload Dj 往南 自曝  吸收了  百花齐  作为家长  设施齐全 donald ignore  ▁Messages 熊熊 钱多 ▁OB  的思考  ▁frog  ▁quart  ▁iceberg nr 冯某 遇水  但为了  各县市区  ▁extremes  ▁symbolism  就变成  耽误了  ▁Ships  ▁dwarf  checkbox  ▁Keystone  ▁Territories ór 顾家! 黏液!  我必须"  户人家"  的男生#  风向标#  ▁Drinking$  ▁Vertical$  ▁mythical%  ▁endeavour% alm&  现在还&  等优势' wine'  带着家人(  装修风格(  ▁pearl)  ▁swamp)  ▁Jesuit*  ▁Limits* 海港+ ERP+  每天至少,  美好未来,  ▁Panc-  详情请扫二维码-  中下游. ▁Pes.  知识储备/  ▁praises/ 采茶0 飞龙0  辛弃疾1 ixia1  蓝色预警2  技能和2  ▁repr3  工作人员和3  让人民群众4  Executor4 舞动5  主讲老师5 behind6  ▁Vulner6  他不是7 ▁tad7  Bloomberg8 欢度8 炎等9 研报9 ]),:  保留着: ▁Wah;  中国历史;  是不够的<  ▁Schro<  ▁dictatorship= 升迁= 疟疾>  接二连三> ronomy? 专治? aber@  极端天气@  第二天早上A pillarA 之列B 思域B  副主编C  政治家C  武装部D  毕业院校D  我们希望在这里E  IncludesE 克罗F  将会是F  相关证件G  究竟涅盘G  越来越严重H RouterH 狂妄I  进来的I  韶关市J  原因在于J  ▁SheaK SeeingK 所报L  承担相应的L  ▁McDonM  ExamplesM  ▁ShieldsN  努力下N  ▁phishingO 宁都O 母舰P  开放日P  比较多的Q  监控设备Q  一代又一代R  ActionsR  ▁BombayS  ▁MessiahS  emorationT  ▁vigilantT 垂范U 食药U  百姓的V  数学思维V  ▁SarasotaW  ▁specifiesW NamX sirX  日傍晚Y  课程详情Y  产业的发展Z  对他们来说Z 村官[ Eli[ kor\  总干事\  本周六]  确权登记]  的时候可以^  ▁Risks^  Turkish_  ▁Pandora_  替代品`  学历教育` Sparka  ▁zeala  ▁infusionb  ▁honorableb 中班c Frac  感恩的心d  收费项目d  这个季节e  ▁cardie  ▁Rubberf  ▁caucusf  ▁spinachg  ▁fabricatedg 榆树h 车市h TCLi  完善了i  农夫山泉j  按照自己的j  ▁Pearcek 拉丝k 放油l  再也不怕l geantm 五中m 动产n 卸任n 爱笑o  莫里森o arakp  后疫情时代p  Douglasq  ▁Cologneq  ▁revokedr  ▁electrolyr  ▁notoriouslys 县内s  人都是t  ▁loret  ▁McGrawu 刘建u 打滑v  专题党课v  具体预报w  学历认证w  至今已有x rettex  两个人在一起y  ▁Lahorey 搜查z ▁DDz  订阅课程{ ticket{  ▁sands|  ▁inhuman| 一过} 立的} Axis~  也不一定~  能够通过  为他人谋取利益  Newswire  ▁Optical  ▁brushes  ▁Applicants 销往  ▁Derrick  ▁puppies  ▁trucking  ▁categorized pra  国际标准  精心策划 PRING mable  ▁Amir 不疑 摘掉  生存的 GitHub 不卖 副高 终成  ▁mediocre ół 本条 SAM uve yne  渐行渐远  沟通能力强  的大力支持下  Probably 人往 断续  在室内  而引起  陶渊明  认认真真  ▁digs  ▁rund  believe  主持会议并讲话  ▁geology  ▁offended  ▁extraordinarily 名句 射精 用纸 Pros  善于沟通  ▁Downing  ▁Woodward 东东 乐居  ▁Loyal  ▁Twain  ▁confuse  ▁Incident 却步  ▁pued  ▁refreshed  ▁substrate 售完 宋庆 精锐 解渴  新思想  招聘活动  imicrobial  ▁Botanical 首套 ▁Hwy 迸发  一些东西  快的速度  逢年过节  对此声明的  akening  ▁spurred —( 研招 苦于  的表面 boats  全国范围内  ▁administrations 按原 超凡 酸软  assisted 集邮  系列报道  ▁dich  ▁Atmospheric 挣脱 献花 问心  这需要  一开始就 Invent ▁Ré  新问题  的民警  生活需要  ▁Recipes 真如  亲身体验  ▁Rated  ▁bible  ▁bending 渴求  哈哈大笑  运载火箭  ▁omitted 而远  凤凰网  红色资源  ▁DEFAULT  ▁thinner 高声  凡尔赛  宣传部长 之水 平复 水花 漠视  一二三  护理学  的今天 idis  新媒体编辑  ▁Woodstock  可降解ª  责编丨ª ▁Dagê  外出打工ê homesĪ thorneĪ  ▁bulletinŪ  ▁commutersŪ 两栖ƪ 于某ƪ 多斤Ǫ 终归Ǫ 阳城Ȫ  了无数Ȫ  全部的ɪ  原则和ɪ  选择和ʪ covidʪ  ▁Hulk˪ Static˪  ▁lacrosse̪ mv̪  清晰可见ͪ  所以一定要ͪ ACTIONΪ  ▁FloraΪ  ▁JewelϪ  携带本人身份证Ϫ 9 2ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡۥۛ۟ۡЪ 也像Ъ 从这Ѫ 左腿Ѫ liumҪ  解题思路Ҫ esslerӪ 多芬Ӫ  冯绍峰Ԫ DemoԪ  getValueժ  ▁terrestrialժ 韩某֪  后来在֪  无与伦ת  村集体经济ת  ▁ignoresت  ▁Emotionalت  Interestingly٪ 广袤٪  不推荐ڪ  急难愁ڪ  捐赠的۪  艺考生۪ boltܪ eyesܪ  DEFAULTݪ  asionallyݪ 大港ު 战神ު 斩断ߪ 越吃ߪ 顶多  寄宿制  有人能  点开始  的男性  主导产业  的重要手段  variant  别具一格  只有一种  ▁Gail  inctions  ▁Guarantee  ▁meticulous 尊师 环岛 诱饵  就能让  笑起来  肯定了  充分准备  可以轻松  小区门口  政纪处分  一定程度的  ounsell  ▁Intervention  ▁motivational ▁ä  主题曲  客观题  参加调研  弹道导弹  ▁cannon  ▁Bengaluru 包揽 吃穿  ▁же  项目总  ▁Desire  ▁meticulously 和它  愈演愈烈  精心挑选  ▁Decisions ▁AH  第五条  还是蛮  适宜的  每天都是  考核合格的  戴安娜  旷神怡  但是对 Nm 收容  开始时 ▁dif  截然不同 ateau  ▁Torn  系好安全带  ▁Finale  redients  ▁repercussions Mars cult  夫妻感情  不代表本号  你喜欢的人  ▁sanit  Effective 母的 Coffee  ▁fractured 上亿  主动到  环境监测 yahoo  ▁VERY  人事部  和帮助  教导员  暴露出  深圳的  之大变局  审议通过  更多推荐  ▁Hundred  ▁durante ▁KO  聘用制  运动中  ▁его  润肠通便  ▁clientele  芭蕾舞 ITLE  昨天下午  ▁Surely  ▁recorder  ▁Polytechnic 全车 ▁PSA  减少不必要的  Visitors % 对此声明的最终解释权 广德 绝世 ▁Sv  是相当 ocular  ▁burglary  ▁Intermediate  儿时的  明天起  人事档案 igree  ського  ▁juicy Ric opf  分布图  日后的 Flat oxia  到了晚上  Scottish  ▁drowning  购买力  赵立坚! ▁Tek!  狠抓落实"  ▁duly"  消防工程师#  ▁ozone#  ▁approving$ 几周$ 张北%  可惜了%  神经元&  如果只是&  被认定为' Innov'  ▁overcrow(  ▁subpoena(  ▁guideline) 民谣) 滨区* 环湖* 赫然+ rather+  ▁astronomers,  无异于,  进行修改-  Thinking-  ▁Hendrix.  ▁yourselves. 广义/ 阿婆/ 顶上0 TEM0  大世界1  前三季度1  心理状态2  washing2  ▁fractures3 便后3 李红4 lea4  和活动5 rath5  ▁Reign6  ▁tutoring6  ▁tightening7 能赚7 邹平8  地铁口8  不到一个月9  充足的睡眠9 东来: 聚居:  核电站;  的视觉;  任何人都<  实打实的<  适应能力=  Various=  收获的> ▁ESA> liked?  ▁spikes? 大件@ ▁FP@  是父母A  而此时A  穿在身上B  不符的B  岗位条件C  超市购物C 她要D 木偶D 海王E 牧原E 编队F  毕业了F ovskG  不知道的G  ▁OlympiaH  ▁LockheedH 不倦I 助长I 棚前J 罐子J 还请K  原本的K  职工群众L  的相关信息L  市市场监督管理局M 曾为M 腾出N deleN  为了您和O  广大教师O  经验优先P  timestampP 乎其Q 口香Q  早报告R ▁nouR  专题培训S  大江南北S  ▁wormsT  ▁dramatT 震中U  不要超过U OscarV  ▁slabV  chellesW 之境W 维州X  隐藏的X  疫情风险Y  ▁EditionsY  ▁frackingZ 搏斗Z  危险品[  一条街[  幸福指数\  意识到了\  相信很多]  ▁Layer]  ▁Chopra^ LF^  系列课程_  ▁Logger_ 一页`  而这次`  ▁Lovesa 月供a 朵花b  元的价格b  节能环保c  ▁filtc 妈咪d 须经d  就是有e  liftinge  ▁Millennialsf  宣传员f 上杭g ▁Pég  但需要h 元享h ункi  无论您i  中央企业j psonsj  是社会k  相融合k  也在不断l  交流活动l  心情舒畅m  高考作文m  ▁Launchedn  ▁Remembrancen 医改o hypo  分钟以上p  甲状腺癌p  第二剂次q iergeq  ▁Rossir  ▁Preparedr  ▁moderateds  ▁uploadings  古希腊t  监管局t  ▁lodgedu 荷包u Rumv  你们在v  刮目相看w  ▁Schlw 奶瓶x 宿命x ▁Waxy  政治方向y Coastz 奶水z 载着{ 音标{ ▁ai|  为标准|  和客户}  想要去}  林先生~  ▁hoard~  发生关系  自查自纠  具有良好的品行  ▁textured 尿管 手淫  一段时间的  ▁Turtle  Observer 有谁  可增加  年轻时的  ▁parishes 选报 erity  自己的行为  ▁Roller  遵守宪法和法律 疲力 隔天  开始对  日全国  商业中心  ▁OLED  ▁Ingram  不随意  让不少 ▁lui  有序排队  的交通枢纽 1 *应对新冠肺炎疫情工作领导小组 以至 神韵  收视率  有一家  的重视 ▁Hok  免疫力的  督促指导 五线 律动 走读  与河北省  积极协调  自以为是 rites 识人  佟丽娅  无人能  受到党内警告处分  ▁youngster qv 撞到 likes  ▁Emin  ▁stature 扭伤  有时在 ▁Hij  不留死角  ▁bundles  ▁spelled 不染 海神 海航 BAT  名下的  ▁COLL  ▁Suit  重要指示批示精神  立医院  人生最大的  ▁plunged  ▁anterior VV 网恋 ▁ITS  ▁naive  nington  Released 文火  可以自由  市总工会  ▁embroidery 用得 突起  有了它  等媒体  发生争执 停歇 马赛  年建成  给身体  广大家长  情绪激动 ateurs  ▁Compass  ▁suicidal 改观  在推动  跆拳道 anck  ▁Carlton  ▁sacrificed  ▁Evangelical RAL mob  却很少 ▁Arb  随着时间 Tasks 出奇 尿路  很丰富  贵州茅台 老陈  在河南  )　{　  每个学生  考察调研  业主委员会  施工总承包  entiful  ▁Deposit 给水 越做 AMG  chanted  ▁Established  境内外  幸存者  民办实事  ▁Adolf  历史文化名城  iduciary« 并附« 耳垂ë 肺病ë  和业务ī  将他们ī dentū ▁wreū  合理使用ƫ 熟透ƫ 调集ǫ  准妈妈ǫ  要带头ȫ  这门课ȫ  本场比赛ɫ  产品和服务ɫ  发挥了重要作用ʫ 少爷ʫ 童谣˫ 远行˫  免费学̫ ▁sto̫  我的生活ͫ  ▁skinnyͫ 工的Ϋ CBCΫ  疫情下的ϫ 加湿ϫ  学校体育Ы  我们的孩子Ы fkѫ 精编ѫ 都吃ҫ  对记者ҫ  触手可及ӫ  ▁hipsӫ 与该ԫ  等传统ԫ  纺织品ի ▁Aryի  提醒各位֫ geois֫ 春寒׫ 死因׫ 都去ث  手机和电脑ث  持续更新中٫ others٫  中国共产党领导ګ 一程ګ riks۫  上个世纪۫ Medicܫ SpaceXܫ  出院后ݫ  国人的ݫ  的人体ޫ  重特大ޫ anon߫  ▁Polk߫  区委组织部 怀远 裙装 见底 xcb  ▁Breath  已经把  新突破  风雨同舟  ▁ping  ▁Panda  更正来源及作者  ▁physicist % 更正来源及作者或依据 简明 谢家  单元格 Pure ceased  Greater |\ 再送 简述 腾达 药液  很多事  样板房  ▁Clifton  ▁roundup  ▁inhibitor  ▁constrained Abd COR вой  却始终  年轻了 ▁rpm  长城汽车  有关规定执行 千言  missing  ▁survives 三下 受聘 捡拾 掉入  夹杂着  智能网联 antics  ▁Eaton  uttering 鼻祖  切实履行  长江流域  ▁migrated 凶险  新课程  直播时  面对生活  ▁repur  ▁Colombo " 颠覆了几十年的金钱  ▁Corrections UF 桑拿 汉奸  也还是  就让人  定期开展  普遍认为  相关责任  ▁gamble  ▁boasting 张军  优良的  到这个 ▁MPH  ▁mantle Ec 袖口 里木  都是我  顺口溜 均无 是您 极富  在重庆  贺州市 ▁imb  everyone 响亮 晚于 武士 nett ▁ros  有关政策  内容的真实性  aghetti  ▁Fortnite  ▁silicone  ▁evocative 把酒 肥厚 血量  的头发 ▁CEL  急救电话  ▁Kern  接受社会监督  ▁Marino  十九届四中全会 三相 凋零  四边形 reau  ▁disparity 劝你 扛着 蒸煮  却有着  大奖赛  的执行 dling  ▁Trap  服务点  方法就是  四年级上册  著作权人意见  ▁Angelo  ▁dipped 勘测! 海波! 骤然"  运筹帷幄" 亡后# 宁晋# ipro$  身体力行$  这也就是%  之间的矛盾%  党的十九大和&  ▁Lastly&  ▁Xiaomi'  Encoding' /_( 埋葬( 患了)  仙人掌)  违纪违规行为*  ▁acquainted* いて+ дар+  天安门广场,  ▁refin,  ▁Nassau-  knowledge-  ▁Stockton.  Registered. 海选/ 种田/  花都区0  车厘子0  并不承担其他1  ▁Burgess1  Shutterstock2 行云2 靖边3 Cod3  主动投案4 centre4 1 *本平台将更正来源及作者或依据5 排解5  了一辆6  生物钟6  禽流感7  字里行间7  ▁whipped8 不降8 旖旎9 水北9 ▁REG:  ▁stool:  的全新; Ultra;  ▁nude<  ▁Ivory<  ovoltaic= 其烦= 和上> 徇私>  消费税?  真正的朋友?  ▁trainings@  老天爷@  都给你A  不但没有A  ▁RaphaelB  ▁imbalanceB YanC growC  公示期满D  ▁kaleD prettyE  ▁annumE  ▁LagunaF  ▁SolidarityF 手柄G 江都G 면서H  别人家H  势力的I  岁以前I  热情和J  胰腺癌J  不是所有K  互利共赢K  合理利用L  天然植物L  相对应的M  ▁BlastM  ▁regimentN 多场N 槐花O  三年来O  调整和P  北京市海淀区P  ▁combatingQ  南北宽Q  放寒假R  清洗消毒R  ▁emanS  ▁pursuitsS 多劳T jspT vaeU  整体上U addenV  必须先V ▁FaxW  如果你对W  民主党派X 不深X 兰德Y  少数人Y  结合了Z  欢迎来到Z  ▁compiling[ 商行[ 所知\ 扼杀\  从不同] poor]  继续实施^  课程思政^  ▁Triv_  ▁Richie_  ▁Juvenile` 天黑` 小段a 有话a VALUEb  ▁Pathologyb  ▁minimizingc 异乡c 甲乙d 神志d 表征e ▁Meke  随之而来f  ▁sealingf  ▁Anderseng 小兵g 者不h  图片带h  开辟了i  承德市i gansj  Healthyj 对不k  乐山市k  仅作为l  急难愁盼l  ▁ravem  ▁Conorm  Graphicsn  ▁seriousnessn 共体o 找车o 防爆p  培养和p  家中有q  提前退休q тсяr ▁RRr  家庭作业s  担保贷款s  电子数据t oyangt  ▁pacingu  ▁ornamentu 道明v OSHv  家族史w  昌平区w  身上有x  默默奉献x  ▁stabilizey 两房y 浓香z  她觉得z  的热爱{ ▁GUI{  系统工程|  ▁Murdoch| }]} 之策} 末年~ 热能~ 负极  学年度  突破性  学会会员 C <本平台将更正来源及作者或依据著作权人意见 眼睁  大整治  中心学校  心动不如  人职位描述  每一个细节  ▁beacon  ▁destabil  课程学习 的旧  童话故事  奠定了基础  ▁remodel  新标准  月销量  的不适  颜真卿 ▁kam  真的很难  ▁Expand 最常 王亚 粤菜 美业 老爱  和平精英 ocrine  ▁cameo  ▁uplift  与年支  研制出  该怎样 Marine  新冠病毒灭活疫苗  ▁marrying  ▁Stuttgart 房前 载物  不服输  ▁Nell  ▁Goose  ▁nestled 卡友 对号 棉签  得很快  普拉提 endi  接受采访  ▁Staten  selector  可降低  电梯房  可能出现的 雅迪  然后又  轨电车  送外卖  ▁abras  ogenous  就成为了  涉及到的  ▁Finch  ▁ticks  ▁Recess  ▁Gathering  跟大家分享  为大家准备了  ▁Maiden  ▁Cassidy  OutputStream 制茶 卵子 花光 输给 iry  安康码  能够将  ▁Outcomes wc 国度  在目前  年美国  旺财方位  随着我国 chten  ▁tidal  ▁defaults 村貌 没说 cie  市民生  的春节 面带  东西长  松江区  准确性和  紧随其后  olastic  ▁Chorus 追肥 ilant  ▁Brut  ▁marrow  ▁propel  ▁undisclosed 主推 求医  为城市  挨家挨户  ▁distort 上蔡  ▁Giles pec  多见于  ▁stabbing 控告 破土 重演  大吉大利 前倾 原体 宜业  Arsenal  ▁lighthouse 求人 火炮 王雪  的义务  冲上热搜¬  防汛工作¬  这样的男人ì 中以ì 横县Ĭ  也是需要Ĭ  每天早晚Ŭ  课外阅读Ŭ safetyƬ  leadersƬ + $拥有对此声明的最终解释权Ǭ 中网Ǭ 之爱Ȭ 争端Ȭ 周周ɬ  吸取教训ɬ  ▁sporʬ  ▁Decadeʬ  ▁pluggedˬ 磁力ˬ 蛋蛋̬  改造成̬  大家都会ͬ  ▁Guthͬ  ▁Cargoά  ▁soaredά  ▁suppressedϬ  ▁personalizeϬ 匪夷Ь 融汇Ь  大家可能Ѭ  排名第二Ѭ  ▁añosҬ  ▁hallucҬ  ▁bipolarӬ  ▁rendersӬ ( !学历证书电子注册备案表Ԭ  做起了Ԭ  易出现լ  全世界的լ  ▁NYPD֬  安全隐患排查֬ 太祖׬ 磋商׬  英国人ج  获得过ج 养眼٬ 澧县٬ REGڬ  不添加ڬ GRAM۬  走过的路۬ pragmaܬ  Inspectܬ  ▁outingsݬ " 新时代文明实践中心ݬ 从左ެ 从速ެ 玄宗߬  它们是߬  专场招聘会  ▁shoreline  ▁Conservancy  面积达  ▁Climb  ▁shotgun  ▁progressively 彭某  军分区  大智慧  确诊的  伙伴关系  ▁Deaths  ▁booths  ▁Anxiety  ▁caravan 天里 小声  以下几  和快乐  诈骗案  ▁Sicily  ▁interns 进山  法令纹 ▁UCS  看到自己 ardom  ▁Tribal  ▁flashes  ▁Transaction  一阵子  单身男女  ▁USSR  ▁overlapping 一姐 舒坦  科技集团  ▁Bottle  ▁Neurology 拜师  新概念英语  ▁Diaries 巩俐 本想 触觉  就做好了  而是为了  重复使用 大权 工友 水桶 荒芜  小说的  ▁Understand  ▁sidewalks 护身  大市场  等国际 ▁prz  取而代之 松树 税负  发表于  ▁almond  googleapis  严重性  ▁Metals 六项 曲面  赵孟頫  ▁drap قى  不由得  走向了 Grade  ▁PEOPLE  ▁surged  ▁Beyoncé  ▁Concent  getInstance 哭诉 Unc  ▁favourable 所选  线下培训 encent  ▁escort 凭空 尾气 新篇 转世 IRED  中国农业  医用耗材 " 部分图片来源于网络 健儿 头和 长虹  与文化  当别人  ▁aquarium  ▁congregations 高炉  不亚于  普宁市  教学能力 Alber  瑶族自治县 城厢 Temp  更有价值  ▁Dimension  ▁depressing 火源 群友  不说话  例病例  侧重于  等荣誉  ▁Dane  ▁Shuttle 睡到  报了警  作家协会  匪夷所思  精神面貌 achal 便有 哀思 UFC!  名同学!  的剧情" Done"  家庭住址#  愿意为你#  ▁Gior$ ologia$  医疗机构就诊% 名画%  原则上不&  ▁Kier&  展望未来' gling'  ▁насе(  ▁Sonata(  ▁outage)  permission) 慕容* 现居*  百分比+  公告为准+  奠定基础, yb, 大中- 摄于- 霍山.  驱动力.  专家介绍/  以免造成/  恶心呕吐0  违法停车0  出版社出版1  ▁cellar1  ▁Wizards2 情调2  合同法3  结合部3  大学生士兵4  ▁Freight4  ▁craving5  ▁insecure5 与水6 八段6  座车站7  脱贫成果7 litre8  ▁Garn8  助力脱贫攻坚9  ▁Isabella9 不喜: 传唱: 盗墓;  ▁plagiar;  向上滑动查看内容< ( !市人力资源和社会保障局< 勿忘= 杀生= 虚度> 鱼虾>  从没有?  倒不如?  等物质@  ▁Zhao@  ▁kiosA 晨练A RoyceB  ▁peppB  互联网企业C  ▁hustleC  扫描上方二维码D  ▁drainedD  ▁pointersE % 城乡居民基本医疗保险E  免费看F  研究确定F  ▁NevilleG  ▁ReviewedG  大不了H CTOH  改名为I  血管内I  顶级的J  ▁HamletJ  ▁usabilityK 提货K 深交L  只可惜L  欢快的M  TextureM  treatmentN  忧郁症N  不必担心O  如果按照O  ▁champP  ▁hitterP  ▁strandsQ  ▁ProposedQ  ▁setbacksR  ▁encompassingR 特异S  是一段S  选手们T arooT  ”　|　U  ▁erstU  的重要依据V  北京理工大学V  ▁sublimeW 推卸W  总领馆X MACDX  最高境界Y  如果你没有Y  LettersZ  ▁punitiveZ  ▁assemblies[ 同乘[ 止泻\ ▁([\  奥利弗]  牢记初心使命]  ▁Ranked^  ▁unanswered^ 两制_ 睢宁_ ahs`  然后是`  本次比赛a  ▁fragmenteda 纬度b  教学和b  用它来c  消费者对c  详情如下d  尤其是对于d  ▁Flightse 像你e 西雅f койf  怎么用g  还是被g  高大上h  全世界都h  ▁storytelleri 力挺i 建在j  为世界j  休息日k  产权年限k  ▁automakerl 多则l 福岛m  培养的m  奖最佳n  新生代n  ▁Stratfordo 老爹o 首领p  杭州湾p  上级部门q  找到自己q  ▁climbsr шоr MLSs  吃水果s 插秧t 警种t  暂时关闭u  疏通经络u  项目申报v  relativev  ▁Islamabadw 冲压w rpmx ▁Nabx 厨余y 此方y 除菌z 音速z CED{  王宝乐{ WOOD| Chuck|  ▁Amend} 很棒} 适可~  不一定要~  但是这些  医学专业  发动群众 Jenny pixel  进入小程序  Frances 大热  你的手 zoom ▁sho  ▁Kare  全国大学生  ▁weekday  ▁ceremonial ág 更小 陈旧  副司长  加快了 Lion  每一次的  ▁Humb  ▁façade 均属 大足 春水  委员会主席  工作会 ▁Sek  ▁attentive  ▁visitation 云霞 楚国  又不会  就目前  班学习 cand  高速发展  ▁unriv  ▁snippet  市人大常委会主任 TG 回听 毛绒  现招聘  环比下降  ▁Bihar  ▁forehead 小包  地开展  方面也  洗净后  请备注  造价工程师  的大部分地区  ▁Landsc 不穿 快车 灭了  他的作品  能够起到 phonic  ▁fierc  ▁sparse  ▁asserts 底商 祸福 能级 黄皮  可提前  已全面  扫码加  永远无法  华南师范大学 一星 沙坪 租住  有关人员  知情同意  ▁Glacier 回声  几乎都  对党的  副理事长  达成共识  ▁scalp  ▁Ethical  ▁Furious 基亚 控诉  奥委会 oley  高考英语 elves  核酸检测证明  ▁moratorium 剩饭 老花 裂痕  中发挥  有效提升  贵重物品  送给大家  льзова % 中国特色农产品优势区  ▁foreground 川贝  虽然很  境外疫情输入  ▁Brewer  ▁Offshore 两队 失能 ▁Gob  ▁Oasis  ▁Mapping 绿标 者也 茶道  的一首  社区和  谢海洋  今天就来  公开遴选 停摆  在韩国 ▁MLA  操作简单  效果更佳 rolley 划入 各州 断崖 被曝­  的言论­  ▁Cairí  ▁benchmarksí 七种ĭ 尖椒ĭ 票面ŭ 赏月ŭ  起作用ƭ  飞机上ƭ  严格保密ǭ  作品名称ǭ  地处中国ȭ  ▁Crabȭ  ▁envyɭ 外甥ɭ CHRʭ  何首乌ʭ  双色球˭  用错了˭  的水域̭ udaṋ  直播预告ͭ  答题技巧ͭ  Workersέ >:έ  单身的ϭ  居民们ϭ  眼睛里Э  项目开工Э  林业和草原ѭ  ▁multimѭ  人民日报客户端ҭ  Architectureҭ 金马ӭ ▁Kovӭ  其他岗位ԭ  ▁Pettԭ  ▁ominխ etchedխ 口村֭  眼睁睁֭  国泰民安׭  的作用下׭  ▁Clashح  ▁heartbeatح 想好٭ ▁KU٭  负压救护车ڭ  ▁troopڭ  全面推进乡村振兴ۭ ósۭ 串通ܭ  旅游局ܭ  纯新盘ݭ Slowݭ  公司业务ޭ bringޭ negot߭  ▁skincare߭  ▁downstairs ۗۡ 检定 静文  综合工资 分秒 粤东  与挑战  号线一期  按时完成  有一段时间  ▁Ramadan  ▁Districts  手机卡  的小花  对于这种  ▁Cannot 字第 年任 战力  挥发性  教育等  督察组 osse veau  ▁Bray 一腔 七日 海丰 箴言 过吗  技术上  注册制  消防队 % 新增新冠肺炎确诊病例  ▁inefficient pps  不为人知的  ▁Stokes  ▁Duterte  ▁helmets  ▁undertook 伤及 加里 种群 跃跃  三亚市  让她们  顾客的  补齐短板  ▁slips  ▁Carlisle  ▁Jeremiah % 核酸检测结果均为阴性 余热 时评 树洞 证言  你的钱 ▁PAN 惹的 窘境  不能提供 光靠 气了 躺平 送考  尼亚州  录用资格  证据表明 assee Higher  国土空间规划  leground  标志性的  ▁Copa  ▁hesitation  ▁Frankenstein  与日支 anian  本微信公众号  ▁Presented 掉发 清宫 火影  这个品牌  Phoenix 和善 莒县 诗集  指挥长  肺功能  精心设计 Determ 寒露 月前 Lik jal  最后在  舒适感  ▁silhouette 抓到 水壶 祸害 者有 uela  景观设计 ▁­ 也以 镇大  了一组  的一篇  西湖区 xiny  艺术创作  ▁Wolfgang 该类 ▁MSc  犹豫不决  陪护人员  ▁Banc  ▁wrinkles  ▁Accidents  ▁commanders 和弦 wsh  和运动  ▁DeSantis  北方地区  ▁Outs  ▁forma! 不偏! 器乐" 盗用" 青苗# cephal#  flowing$ ті$ 减刑% /">% POR&  供销社&  有太多的'  ▁Ridley'  ▁masked(  ▁diabetic(  ▁INFORMATION) 岁到) 总则* 清源* ▁{\+  纤维化+  计划数,  用心用情,  ▁Inga-  ▁Tack-  ▁Telephone. 三组. 收腰/ 气化/ 王洪0 路飞0 ▁zw1  真人秀1  美少女2  能够快速2 ITIES3 顶楼3  从此以后4  我们坚信4  有时搬进阳光房5  ▁outages5  ▁Giveaway6  ▁Wallpaper6 或对7  警戒线7  身上下8  二期工程8 django9  的重要讲话精神9 断开: 相承: ilm;  企业要; creen<  ▁idiot< 只卖= 本套= 蓄水> izu>  空气净化?  第一作者?  ▁biting@  ▁untouched@ 洲岛A RoutA  ▁JosefB  ▁vinesB  ▁BasicsC  ▁bloomsC  上面的蓝色字体D  ▁brainstormD  ▁antioxidantE MegE  本科一批F  能感受到F  ▁rejoG  ▁PascalG  ▁exhilaratingH 业力H 足矣I  护城河I  ▁MatthJ  ▁MartinsJ  ▁disgustingK 万桶K 戏份L 水城L  主动去M  无形的M  请拉到文末N  的前期N  ▁TajikO  ▁alrightO % 并不承担其他任何责任P 应先P  透露着Q  ▁institutedQ 挺进R TitR  黄褐斑S  源远流长S  ▁WEEKT  ▁fittingsT  ▁implementationsU  你总是U  有钱了V  送到医院V 正反W  整改到位W ChangX  乘风破浪的姐姐X  ▁strainedY нгY 带宽Z 干得Z 茶农[ ▁Yor[  网友热议\  艰苦边远\  ▁Buffett] 忙得] 视图^ SSSS^  继续使用_  的主要成分_  ▁monst`  Surface`  interrupteda 丞相a 了事b 独木b 脐橙c  三保障c  低血压d  ▁OPENd  ▁countdowne  安眠药e  鲜艳的f  在直播间f  继续开展g  ▁icingg  ▁Staplesh izoh  任命为i  厚积薄i  lectricj ▁ssj  实习期k IiIik argvl  服务年限l youtum  在职研究生学历m 北向n 股沟n  有福了o  血管的o  颁发了p  个人防护意识p  ▁Somaliq  recordedq 太早r  了一部r  加分项s  苦尽甘来s  ▁Roofingt 劲道t 大群u 栅栏u ▁ATSv  十有八九v  要充分认识w  ▁Goddessw  ▁toleratedx 开火x yary ▁RDy  怎么就z grayz  结婚登记{  imentary{ 孙权| sus|  多读书}  必然要求} rmtzx~  ▁cram~  是什么样子  ▁treaties 续约  的实践  精美礼品  ▁tyre  ▁Pound 弱电 扫帚  赶紧收藏 永川  共同生活  ▁scientifically 个例 厘清 断定  大巴车  老师是  门头沟  ▁Vaughn  ▁complemented  ▁southwestern  地面的  成为你  的体现  的元素  育人的  最受欢迎的  居家医学观察  ▁skepticism 牌匾  主教练 ▁HAR  ▁Farn  ▁adept  Direction  ophysical 本店 Vac  谁也不  超声波  出门在外  族维生素  ▁Steward  ▁motivating  ▁Manufacturer  ▁manifestation 上大 南进 天伦 润肤 重置 ▁Edg  ▁whim  ▁Satisf  ▁famine 冻品 ▁Fam ▁Sut  应当及时  ▁Sovere 未免 IMO  ▁Rect  大学毕业后  ▁haird  ▁Asylum  ▁disagreed ▁UX  学生干部  ▁Sanford  ▁underrepresented = 6ۥۛۛ۟ۙۙۚۥۚۙۙۙۚۥۛۛۚۚۛ۟ۡۥۛ۟ۡۗۡ 只怕 铁丝 AGO  与对方  天气转  给予的  人行横道 failed  ▁embargo 一端 气概 渔夫 HBO ▁"^  物流园  诉说着  阅读能力  ▁proactively  ▁Municipality  希望各位  ▁pointless ( !本平台将注明来源及作者 视窗 这身 GPA  就是自己  期末复习  ▁librarian hir hov été  吃太多  暴露了  ▁Faso  ▁Loyola  ▁Gloucester 编纂 莫高 跳下 锣鼓 くて 力不 拐卖  新朋友  志愿活动  这个春节  ▁handset 大乐 谁料  和同学  多年了  给了她 oufl  长篇小说  ▁chor  ▁Fritz  ▁diocese 时才 花友 NHS NPC  个重点®  并告知®  自习室î  河南大学î  ▁HelmĮ 减征Į 海峰Ů  与日干Ů  那么难Ʈ  元的课程Ʈ  关于我们Ǯ  致以节日Ǯ  我们看到了Ȯ  ▁ConnieȮ  ▁McLeanɮ " 例新冠肺炎确诊病例ɮ  ▁unrealisticʮ 围攻ʮ  想要在ˮ  风平浪ˮ  的主要内容̮  locking̮  ▁forgivenͮ  ▁logisticalͮ oSή 　$ή  考虑的Ϯ SharϮ  继续坚持Ю bridsЮ  ▁BroadbandѮ 受累Ѯ 断水Ү 鲜甜Ү  ▁BreaksӮ  视频制作Ӯ ándezԮ  ▁NAACPԮ 农贸ծ 日将ծ ULE֮  人感染֮  以为常׮  在辖区׮  就有多خ  比较长خ  用心的ٮ  一个非常ٮ  技术路线ڮ  的时候是ڮ  就不一样了ۮ  折合人民币ۮ  patcherܮ  ▁sacrificingܮ 之品ݮ  每一段ݮ escapeޮ nasiumޮ  ▁Percy߮ 能静߮ 足的 闲事 ▁NB  原发性 moor ▁Oro  为更好地  募集资金 作的 其时 反流 奔放 致残 金利  费用包含  ▁Slate  ▁forfe  中国农业银行 共修 反常  消息后 leave 吉思  的省份  自适应  医疗保健  点击视频  体检和考察 ▁"'  ▁duel  的情况来看  ▁plateau 透光 Sax  不同类型  股份公司  ▁upro cookie  和控制 Karl 个盟 指使 签下 ▁")  罗永浩  诚挚的  诞生于 toBe hetto  ▁Cuts Cities  itimate 不沾 季军 起亚 都爱 都美 ▁JA  作案嫌疑 compos  归原作者所有  ▁Tuesdays  ▁monarchy 今世 折服  保平安  爱国情怀  生态宜居  最关键的是 PAC  成熟了  湘潭市  的消费者  ▁mRNA  ▁amalg  ▁coatings  Scientific  ▁preserves 壮年 家产 这届  先上岗  大名鼎鼎  终于来了  ▁unintended 战功 挥舞 胡润  暂停键  路过的  有了孩子  ▁adrenaline 一进 排涝 酷爱  大家族  文末有 Vote  中国传媒大学  ▁detract 包扎 考纲  发短信  很轻松  马德里  ▁Quincy  ▁Consolid  ▁delights  ▁Innovations 犯困 蓦然 足协 还敢 隆基 难成 ▁Nim  接种疫苗后 Legisl garden  ▁richly  Absolutely  ▁exploiting 已为 抗性  几秒钟  不得重新  ▁Neon  ▁Arrang 在目 爆火  我家的! ▁FAM!  内生动力"  ▁banquet"  ▁exhausting# 代餐# 就座$ 无措$ 相映% 米酒% 难易& ▁LB&  包裹着'  积极向'  除了要( ptons(  人教版数学) 冰镇) 毗连* 通识* cfg+  抢先看+ Salt,  报名要求, 浮生- 纳闷- 骤降.  完全的.  杯奶茶/  ▁FULL/ 切磋0 责成0  部分路段1 她曾1 安镇2 教您2 谜底3  四合院3 Labour4  ▁hinder4  ▁freshmen5 HQ5  传说中6  这款产品6  ▁grips7  ▁Sheets7  ocurrency8 婚丧8  涨工资9  走一走9  付出代价: OTHER: 次序; 贫民;  加适量<  角落里<  ▁Edited= 决然= 物管>  多巴胺>  聊天的? adin?  对此事@  排名中@  郁金香A  ▁radiosA  若干问题的解释B  ▁castlesB 亲水C 可对C 工区D  拟聘人员D  ▁DevinE 东村E 博鳌F  不准确F  官网发布G  报名资料G  红色经典H  ▁BodiesH  ▁fearlessI 看就I 落定J  ▁sonoJ  ▁BiotechnologyK 便血K 堪忧L 是可L 狭小M  是普通M  最权威N KINGN  ▁sterilO 火药O 见你P  婚姻关系P  点击文字Q  ▁LymeQ  ▁covertR  ▁passportsR 屏风S 抢走S 逼格T  暂时不T anskiU onsorU  全球最大的V ermannV  ▁UnterW  ▁penetrateW 李金X 梨树X 记叙Y  一家公司Y ackersZ  ▁epidemiZ  ▁transient[ 公摊[ 分给\ 浦北\ owej]  应用能力] 漫山^ 逆流^  的判断_  按揭贷款_  ▁SCHOOL` 分出` 日二a  有关信息a  ▁Shepardb 寒性b 毒蛇c 泰康c  你肯定d  我的世界d  汽车产品e  法律咨询e  热闹非凡f allocf 壮举g 运往g 遨游h  投资机构h  具备较强的i 一命i 到会j 当人j réalk  ▁Abramk 雄壮l CERl ▁iem  ▁Shetm  ▁suckedn  造型的n Catalo  ▁Whaleo 远赴p ANEp  世界纪录q  已经成了q  是内蒙古自治区r  Electionr CWs 满嘴s  每一颗t  ▁Gautt  的专业知识u  ▁burstsu 援藏v 新野v 莎士w  流露出w  该区域x  ▁followerx  ▁harmoniesy  ▁negligenty Fluz  表现突出z  Friendly{  ▁exposures{  ▁physiology| 沪宁| 涟漪} 菲利}  照顾好自己~  Heather~  ▁Rebels  ▁leukemia 万圣 POSE  单科成绩 aneers  的影子  ▁Welt  ▁Allies 成武 暗黑  那不是 buck  抗病能力  最新政策 ám 装点 迷彩 ATP  和领导  有一批  第四轮  ▁quar  ▁crest  ▁Charts  ▁Brilliant  ▁overarching 体虚 喷喷  适度的  ▁Slav 别扭 违和  与老师  为根本  人生观  含水量  特困人员  ▁claw  ▁bullpen  ▁Argument  ▁Normandy 利物  可以试试  岁的女儿  操作手册  的高度重视 半路 百世 这艘  三星堆  走一步  的第一线  ▁glut  ▁Browne  ▁Dollars  ▁espionage 回款 擂台  两件事  后果自负  这份工作 就变  从负债  你看看  文件中  呼吸系统疾病  american  ▁Labrador  朝九晚 arina  ▁invoices  在许多  准考证号 lewood  Scholar  uisance  ▁Researcher 创刊 日因 未解 涨薪 篝火 翘臀  他现在 itto  ▁Shops 刺杀 已超  可以买  居住区 春城 gee  的规律  建筑风格  ▁Herc  改变了自己的人生 名册 掘金  冲锋枪  找借口  携带者  每半年  的前沿  ▁Dodd cancel  ▁Feast  ▁Starts Ng 一岗  如何将  形成以  信用信息  可能已经  ▁behavioural 火化  的运行  而根据 ▁TWO  公布为准 fixed  ▁Pieces  ▁Volleyball 为使  不计较  工业协会  提升行动 月仙  提高至  ▁Waves 军机  白云山  而且对 enco  ▁Britann¯  ▁Francesco¯ 姑息ï 某宝ï  晚上好į  人体健康į  平板电脑ů  ▁Alonsoů 绿萝Ư  机动车辆Ư  ▁Ruizǯ 笑颜ǯ 腰肌ȯ soldȯ  引发火灾ɯ  Crystalɯ  ▁Honorableʯ  ▁Narrativeʯ 当务˯ 期和˯ 猛兽̯  说不出̯  ▁Kassͯ  ▁Filmmͯ  compareί  alliativeί  ▁optimizingϯ 奉劝ϯ 黄脸Я  支持者Я  这些钱ѯ  黄花菜ѯ unasү  而且还会ү  ▁Abigailӯ 清幽ӯ  演变成ԯ  莎士比亚ԯ  ▁Duffկ 亚美կ  它也是֯  建筑密度֯ 恶臭ׯ 闭关ׯ ceeد  两厅两卫د  逐步实现ٯ  五年级上册ٯ  住建厅گ  包食宿گ  小儿子ۯ  发展目标ۯ  套餐包含ܯ  影响孩子ܯ  自主创业ݯ  Tennesseeݯ  technologiesޯ 四字ޯ  灯光秀߯ Brief߯  ▁prized  的演员  如果你也  小学科学  考试时间为  ▁Abdel ᠠᠨ 咆哮 总得 排挤 acz  黄某某  给孩子的 国共 漫威 vote  意思是说  推动工作 thren 绕过  部门要  自愿放弃  ▁tore  传递正能量  ▁needy 新家  匈奴的  春节后 ▁bye  更好的生活  iquette  中国十大  ▁rearr 星系 顽疾 BIG  众议院  像这种  冒名顶替  未经批准  ▁olds  ▁Sammy  charges  ▁ANGELES  ▁Revenge  ▁Creativity  Communication  疼痛感  ▁Choi  ▁tighten  ▁precursor 一望  的各个  四川成都  Artists  ▁REUTERS 命里 钟表  发展了  ▁Marshal  ▁underwear Fs 俗话 增补 nego  就业见习  就会变成 zypisy  ▁detectors 炸至 眼病 Lie  你就要  的能力和 paces  ▁Dove  ▁eradicate ' ################################ にな 不腻  并最终  方法二  的现场  很容易导致  ▁computed 啪啦 随口  才能实现  ▁Rahman 如未 由浅 译林 onie snap  深刻理解  驾驶车辆 vonne 见血  三年前  除夕夜 Honda  ▁Rentals 茶余  的神秘  mbudsman  ▁exacerbated 以赛 何用 在肩 天高 标尺 洗护 透出  新车的  的难题  等重点场所  ▁petty  ▁abusing 两万 场镇  今天早上  建功立业 hower  ▁Cate  ▁Paige  ▁Eleven  ▁Bordeaux 晒晒 风尘  这一场  ▁дан!  制度体系!  寓教于乐"  ▁spared" 恰似#  他们要#  写一篇$  大革命$  一二年级%  研发团队%  转载文章&  ▁Downey& ( !突如其来的新冠肺炎疫情' 王军' 粘液( 里了( ▁Esk) ▁fot)  落地落实*  什么情况下*  ▁caval+  ▁breakup+ 交投, 淅淅,  不串门-  白砂糖-  校园文化.  过程当中. yllic/ 志伟/  精心的0 atitis0  ruiting1  ▁resemblance1  个不停2  委托书2  来一次3  灵活运用3  ▁grooming4 绝版4 ▁Rin5  成绩优异5  西咸新区6  ▁Knock6  ▁diligently7 问暖7  美元的8  运动时8 ▁Naw9  ▁stare9 + $尽量避免乘坐公共交通工具: 斯里:  禁止在;  视频等; ectomy<  打赢疫情防控<  ▁Velvet= 啧啧= ucl>  从这些>  佛罗里?  瑟瑟发抖?  艰难险阻@ 净利@ 安省A 黄岩A  ▁stockpB  publisherB 亚当C  但作为C  我突然D  楚汉相争D  AnnotationE 文成E 最贵F 特拉F BahG GreyG  ▁turboH ▁BDH  对世界I GKI 北岸J 开合J  工信局K  廉洁纪律K  民营经济L  编辑转载L  ▁PingM  ▁solemnM  ▁SpiritsN 动于N 甲子O 畏寒O  做某事P  联络员P  让皮肤Q iletQ  学生宿舍R  ▁OptimizationR  在西安S  的频率S  ParticipantsT 棋盘T 要害U  了过来U chalV  教育中心V  ▁TemporaryW  全国高校W  房屋产权X  特色学校X  纳斯达克Y  没有一个人Y  解决这个问题Z  OklahomaZ  ▁greener[  ▁salvage[ 可他\ 省了\  物理学家] Optim]  ▁Pose^  ▁livestream^ 印光_ 鱼尾_ Peer`  大专可报`  ▁speculatea JEa ▁بb  下方图片b awardc  ▁RCMPc  ▁vigorousd 委内d 振东e 细品e  务实的f  心安神f  渗透到g anedg  农业产业h  原汁原味h  ▁Balei  ▁Forgotteni 发白j 手环j  时支为k  注定要k  海岸线l ▁Prul  ▁Aminm  ▁Prabm Eithern  ▁Producedn  ▁enchantingo 古称o 度日p 译者p  向学生q  畅销书q  因为孩子r  岗位名称r aceouss 我得s 预祝t lopt  人提供u  小常识u  越秀区v  企业生产v  详细介绍了w  ▁briskw  ▁Thursdaysx 制程x  忙碌碌y  知多少y  湖南大学z 凶星z  爱学习{  Generation{  ▁desperation|  ▁comparatively| 搭理} 断肠} 该行~  乘务员~ Ruth  任何理由  ▁punches  ▁Gambling 产房 存有 胶质  新发现  的回忆 ydon  ometers  ▁Catalan  ▁misunderstood 也把 烧制  太美了  福克斯 Leod izio ▁LOL  ▁dass  ▁Matte  ▁Reconc  ▁characterised  带走了  年参加  预计到  就一定会 屈辱  见到的  遇上了  ▁seren  Bringing  ▁believable 量级 ▁Mā  持续上涨  Exploring 倒掉 听雨 嘉庆  我见过  约而同  抓住机遇  ▁operative . '长按识别二维码了解房屋详情 ™, 多张  的格局  ▁Confederation 抢手 防尘  还等什么  信息化建设  ▁rhythmic 支票 果香 深水 肉末 诟病  不能少  回到家乡 Taiwan  ▁Chant  agedies  ▁Laptop  ▁gravit 随车  大检查  通化市 ▁sod  ▁TRUE  意思就是  高度警惕 phasis stered  几天前  必须为  以此类推  无人问津 chter  由公安机关  ▁lasers " 爱国卫生运动委员会 YZ 受寒  不行了  更可怕  信息传播  自身建设  ▁aber  Przypisy  Implement  古钱币  很长的  克服麻痹  经络不通  Respond  iscovered 南亚 场外  产品中  ▁Taxi  ▁millennium たく SUN ▁ej  黎巴嫩  ▁Jamal  消防救援队伍  ▁Emanuel LPL  家庭和睦  校园足球  难以接受  ▁psychiatrist RG 年复 根植 水汽  二人的  穿山甲 Chef  累计出院  新中国成立后 惨痛 澄海 鸽友  你是个  比以往 uper  现在看来 <>();  ▁jumper 情头° 连读°  坐月子ð Zachð  不可收拾İ  切成小块İ  ▁sniffŰ  你有多Ű  只为了ư  将作为ư  ▁ORDERǰ  ▁flyerǰ  HousingȰ  ▁diversificationȰ sitɰ  一大队ɰ ▁Raiʰ  风险能力ʰ  ▁Engel˰  ▁Statue˰ 泥巴̰ 秀山̰  德国的Ͱ  第七条Ͱ  utscherΰ 督战ΰ  风风雨ϰ  第十四个ϰ  ▁Endingа  ▁residueа  ▁ZeppelinѰ 双非Ѱ  它不是Ұ  幼儿的Ұ  是能够Ӱ  人民广播电台Ӱ  ▁Sweeney԰ 东吴԰ 椭圆հ 每场հ ▁Pvtְ  战略布局ְ  靠前指挥װ  想要的生活װ  ▁badgesذ  ▁Françoisذ 开诊ٰ 生性ٰ 骤停ڰ  老老实实ڰ  这样的方式۰  ▁Theat۰  百年未有之大变局ܰ 他汀ܰ 宠粉ݰ 逐户ݰ 链子ް  准确率ް  但是这种߰  excuses߰ 镜像  而今天  苏宁易购  通过互联网  ▁envoy 前半 旗鼓 菱形  换一种  免费分享  去年年底  推动作用 无成 月被 甲亢 留情 自性 辫子 近况  性贫血  王安石 Cher ▁NES  也进行了  葡萄球菌  每个人都会 吊牌 水道 车行  ▁propelled 全境 很会 松紧 民初 粒粒  低保户  的保障  防火墙 itia  动能转换  人民解放军 文胸 桥段 缴税  Насе  因为工作  生长激素 画意 福运  吉思汗  根本不会  ▁Lease  ▁Chanel  ▁inward 弱小 磨砂  打篮球  李鸿章 ocyt  ▁grease 对非 百会 肛肠 门徒 鼻血  　(　)  还有可能  ▁Secondly 传单 低级 相会 趣闻  一周后  吴女士  合作项目  ▁Vend county 杨凌  都这么  教学班  原因很简单  ▁cellul  ▁roaming JL  小个子 izhi  一体推进  环环相扣  ▁dine  ▁Bhatt  ▁Breat  ▁Vanguard  ▁apprentice 片片  刻意的 imar  取得相应  忙忙碌碌  消费金融  调查取证  Insurance 共商 秋裤  第一学期  colonial  ▁Moderna 德胜  ▁Andhra 顺境  展现在  的沟通  增值服务  当务之急  执行案号  特别严重  适可而止  公共资源交易  ▁losers 元阳 到站  李时珍  错时错峰  能力和水平 quests  ▁Teens  ▁obliv 什麽  吕梁市  ▁Rails  ▁techno  ▁McBride!  ▁schizophrenia! 创优" 汤山"  招生就业#  校园生活#  ▁sweets$  先进制造业$  ▁frenzy% まえ% ▁pg&  ▁Siri&  人这一辈子'  看起来更加'  ▁reimag( 流汗(  在他看来)  机械工程)  ▁bursting*  ▁Oversight*  ▁condemnation+  ▁contemplating+ 乡情, 悲壮,  在抗疫- ▁Xen-  汽车零部件. lh. 观感/  ▁vind/  ▁Akron0 训斥0  你是谁1  使用前1  公开信2  第四代2  一个地方3 athed3  ▁anonymously4 三针4 同庆5  建成区5 瞌睡6 落榜6 ▁jug7  为人父母7  致人死亡8  ▁Asheville8  ▁protested9 学联9 朋克: 蒙蔽: MOS;  复合型;  检测到<  水到渠<  股份制=  西汉时=  都不行>  还有一位>  ▁pont? 地盘? 大旺@ ▁kw@  出行的A ▁RADA  中国人寿B raintB  ▁nounC ursingC  ▁adjectD 叫嚣D 求得E 灶台E  的临床F  设计速度F  这里都有G  邮电大学G fortsH  补充维生素H  ▁accuseI  ▁plentifulI 去污J 沉降J 蒙牛K 转走K 长款L  ▁BritneyL  ▁ComputersM }`M 以偿N  爱马仕N otryO  ▁RidersO  ▁teamingP  ▁LegendaryP 仁恒Q 餐食Q  等一些R  经鉴定R ▁LINS  实训基地S  ▁SantoT  MountainT 给钱U lucU  闵行区V  ▁piùV  周边道路W  就让我们W  ▁irresponsibleX 中亚X 牙签Y 课课Y 隆起Z 高盐Z  工地上[  教科文[  村两委\  吸引更多\  走在前列] 村容] 歹徒^ 认出^  了过去_  卡罗拉_  在后台`  视同仁` ▁Nila  持续优化a havingb  ▁moversb  ▁leveragedc  ▁deductionsc 之笔d 城门d 绥德e  个体户e  全行业f  做的更好f  ▁Promog 坡度g 好呢h  习以为常h  外观设计i  操作步骤i  日出现发热j  Tomorrowj 云的k 阿萨k  我是个l  水利部l  ▁илиm  任何单位m  医保报销n  虽然不能n  ▁Elioto  负责对o Evanp  宣化上人p  真的非常q FNq 之国r  宣传栏r Mercs  恢复运营s 十指t 特立t  月活动u  腹股沟u intav  考试辅导v  ▁Cremw  ▁Affleckw  ▁midtermx 卖得x  保护与y  按部就y  只要自己z  含有大量z  研究领域{  Minimum{ HW| 初春|  中心区}  掉下来}  的气味~  运动队~  还特意 ▁Civ  ▁ignition 查办  ▁Lyft motion  ▁swipe  Practice  ▁Preserve  Developing  小广告  盯上了  限制性  注册用户  的问题是  ▁Expected  ▁eyebrows ío  ▁Patton 日主  安康市 rece  横空出世  深圳大学  熟练运用  ▁taxed  ▁crater  ▁Kaufman  ▁nicknamed 徐家 neur  持续推动  会同有关部门  tracking  ▁lobster 出声 到外 助残 承运 挖矿  人生命  梁启超 WORD raper  ▁YEAR  周末节假日  ▁entrants  ▁tertiary 叹气 掏心 Rare  光明日报  ▁Rapp powers 促改 品学 她不 穿脱  创建了 BJECT  entanyl  ▁excavation 干饭  规则的 opor ▁HAS  考试结束后 一蹴 村寨 淬炼 磐石 韩城  华晨宇  要抓紧  在每一个  生命里的 athom  ▁annotation 之文 也快 桑植 重仓 Ott  也未必  拿到手  比较难  民众的 Depth ▁  e  t  a  o  i  n  r  s  h  l  d  ,  c  u  m  p  g  f  y  w  b 的  v  0  1  k  2 　  S  T 一  A  C  :  I 是 人 不 有  )  (  M  3 在  P 了±  5±  Bñ “ñ ”ı  9ı  Dű 中ű  xƱ 大Ʊ  EDZ  RDZ  4ȱ  8ȱ  Wɱ 生ɱ  6ʱ  Nʱ 学˱ 为˱  H̱  Ḻ  Fͱ 时ͱ 年α 上α  7ϱ  Oϱ 和б 要б 个ѱ ’ѱ 会ұ 这ұ 以ӱ 我ӱ 来Ա  GԱ  /ձ 到ձ 国ֱ 发ֱ  ;ױ 作ױ  "ر 能ر 家ٱ  'ٱ 就ڱ 行ڱ  j۱ 工۱ 业ܱ 出ܱ 可ݱ  zݱ 用ޱ 们ޱ 对߱ 地߱ 自 多 成 日 下 新 公 后  U 全 市 好 子 你 分 小 心 高 也 动 方  J 开 情 于 过 月 理  q 实 天 都 区 进 现 经 他 面 点 体 等 法 关 部 力  V 通 事 定 得 如  K 长 文 间 前 合 看 最 同 本 员 之 安 所 还 加  _ 者 重 主 说 与 位 教 而 入 期 化 名 内 考 道 信 机 建 提 其 产 民 活 车 保 务 性 起  Y 三 里 水 很 报 明 — 场 资 品 及 无 平 更 着 么 当 度 电 意 种 相 第 网 将 美 没 目 展 次 常 果 应 感! 外! 做" 手" 因# 老# 路$ 管$ 己% 量% 知& 元& 身' 然' 程( 系( 设) 物) 二* 师* 制+ 题+ 从, 院, 防- 病- 正.  >. 想/ 去/ 医0 但0 数1 接1 疫2 解2 表3 交3 样4 口4 让5 校5 单6 些6 金7 政7 被8 并8 科9 比9 线: 各: 气; 或; 南< 每< 专= 计= 强> 任> 城? 女? 门@ 山@ 式A 特A 源B 问B 原C 风C 只D 万D 处E 别E 级F 东F 那G 此G 习H 省H 西I 书I 服J 持J 利K 记K 试L 注L 育M 规M 价N 回N 已O 导O 格P 查P 求Q 件Q 收R 结R  |S 代S 给T 打T 放U 项U 海V 两V  <W 至W 需X  %X 联Y 号Y 运Z 社Z 复[ 费[ 话\ 头\ 基] 向] 总^ 受^ 司_ 爱_ 真` 少`  *a 请a 孩b 证b 色c 直c 图d 精d 北e 调e 十f 组f 取g 据g …h 县h 质i 治i 房j 共j 传k 例k  $l 推l 党m 控m 确n 选n 台o 办o 委p 息p 流q 广q 立r 使r 州s 示s 花t 局t 集u 标u 视v 村v 变w 清w 四x 容x 技y 完y 节z 才z 带{ 指{ 商| 再| 今} 把} 她~ 招~ 整 近 消 告 先 友 领 课 快 效 微 由 造 统 术 研 准 案 布 增 白 养 职 非 优 华 认 众 护 型 备 儿 什 条 步 检 类 健 识 影 深 见 – 光 具 周 参 创  & 世 食 神 热 转 企 施 五 群  { 责  } 江 战 达 包 改 团 户 限 支 权 康 难 何 阳 干 张 营 即 料 候 又 供 编 况 观 红 配 队 助 吃 片 未 读 形  [ 越  ] 按 住 除 该 乐 排 语 园 议 空 环 离 字 历 投 验 续  Q 免 走 太 易 几 决 反 满 农 照 ҉ 根 始 福 低 警 往 态 测² 思² 预ò 功ò 某IJ 论IJ 游Ų 王Ų 维Ʋ 速Ʋ 必Dz 温Dz 界Ȳ 超Ȳ 境ɲ 升ɲ 段ʲ 命ʲ 积˲ 核˲ 较̲ 觉̲ 毒Ͳ 军Ͳ 落β 客β 它ϲ 划ϲ 严в 站в 获Ѳ 款Ѳ 财Ҳ 状Ҳ 坚Ӳ 装Ӳ 便Բ 值Բ 险ղ 班ղ 青ֲ 致ֲ 血ײ 属ײ 河ز оز 欢ٲ  Xٲ 引ڲ 钱ڲ 约۲ 药۲ 究ܲ 边ܲ 喜ݲ 击ݲ 聘޲ 范޲ 男߲ 份߲ 且 首 火 买 百 极 岁 监 春 早 镇 马 亲 切 适 随  ~ 士 构 则 际 疗  #  @ 仅 景 讲 言 码 细 素 黄 义 足 居 米 响 宝 版 置 а 志 审 称 眼 购 望 显 星 采 断 留 京 声 连 送 络 林 冠 底 存 势 初 失 像 济  Z 享 播 史 织 乡 依 英 象 双 德 简 轻 酒 故 录 评 店 材 油 卫 模 补 远 终 香 听 许 率 频 室 普 音 李 算 朋 层 味 够 府 球 律 云 却 · 修 写 负 富 龙 克 е 愿 察 石 销 и 假 充 母 待 湖 卡 帮 ‘! 诊! 策" 择" 均# 章# 培$ 尽$ 遇% 压% 降& 围& 古' 协' 善( 停( 减) 训) 举* 妈* 域+ 晚+ 拉, 您, 兴- 楼- 售. 六. 怎/ 紧/ 半0 官0 土1 赛1 益2 染2 止3 错3 找4 临4 批5 套5 死6 艺6 牌7 岗7 宣8 列8 器9 画9 承: 座: 千; 患; 旅< н< 演= 武= 炎> 害> 副? 含? 纪@ 吗@ 肉A 急A 智B 执B 念C 黑C 融D 伤D 街E 跟E 奖F 吧F 密G  \G 练H 铁H 担I 答I 博J 雨J 症K 载K 阅L 皮L 角M 餐M 斯N 呢N 宁O 良O 突P 汉P 货Q 争Q 银R 申R 违S 谢S 障T 菜T 庭U 抗U 肺V 父V 须W 印W 哪X 税X 毕Y 退Y 板Z 八Z ҈[ 股[ 付\ 独\ 季] 询] 序^ 尔^ 贵_ р_ т` 余` 酸a 笔a 曾b 亿b 闻c 征c 短d 吸d 顺e 绩e 破f 登f 夜g 痛g 刻h 综h 辑i 松i 介j 苦j 陈k 午k 继l 聚l 额m 彩m 互n 换n 钟o 希o 玩p сp 救q 稳q 激r 绝r 背s 否s 苏t 句t 牛u 七u 刘v 穿v 刚w 块w 票x 族x 守y 差y éz 般z 虽{ 飞{ 右| 绿| 左} 迎} 秀~ 婚~ 丰 茶 异 永 疑 诉 另 粉 输 析 央 拿 措 洗 草 疾 础 困 苗 若 藏 顾 ↓ 厅 纳 归 幸 驾 劳 堂 荐 抓 久 亮 圈 拍 慢 衣 九 航 媒 鱼 伴 附 针 典 灵 盘 丽 卖 休 剧 亚 辆 树 甚 端 启 冷 追 奇 涉 隔 笑 届 夏 庆 в 夫 鲜 罪 罗 词 贫 督 静 累 梦 歌 馆 似 革 犯 兰 令 筑 挥 木 毛 扫 港 佛 贴 波 略 饮 脑 靠 睡 延 惠 缺 阿 礼 巴 汽 烟 丝 冲 拥 童 л 哈 醒 脱 汇 促 括 占 熟 谈 鸡 努 予 猪 川 秋 侵 操 谁 索 爆 坐 幼 沙³ 斗³ 避ó 册ó 田ij 危ij 散ų кų 触Ƴ 蛋Ƴ 玉dz 诗dz 宜ȳ 桥ȳ 饭ɳ 暖ɳ 怕ʳ 戏ʳ 驶˳ 述˳ 签̳ 杨̳ 呼ͳ 返ͳ 怀γ 脚γ 封ϳ 判ϳ 喝г 叫г 罩ѳ 移ѳ 叶ҳ 净ҳ 胜ӳ 攻ӳ 摄Գ 损Գ 禁ճ 阶ճ 授ֳ 肤ֳ 搭׳ 床׳ 阴س 杀س 键ٳ 软ٳ 龄ڳ 派ڳ 荣۳ 雪۳ 糖ܳ 曲ܳ 骨ݳ 懂ݳ 址޳ 绍޳ 厂߳ 涨߳ 轮 咨 掌 液 固 倒 乎 托 盖 佳 奶 讯 透 寒 献 朝 戴 租 竟 兵 零 姐 吉 植 截 励 峰 稿 握 顶 符 肥 露 详 蓝 订 野 卷 尚 乘 纸 链 箱 虚 距 逐 趣 啊 借 剂 掉 跑 库 奋 庄 脸 探 杂 误 邮 仍 席 秘 哥 暴 冬 д 篇 威 牙 访 删 亡 惊 齐 菌 竞 屋 虑 束 м 豆 隐 诚 末 骗 抢 岛 召 胃 折 盛 旧 烈 洁 爸 罚 у 赞 架 扬 纷 侧 伙 渐 乱 恶 麻 币 冰 啦 私 湿 添 账 忘 颜 途 圆 既 舞 薪 翻 硬 旗 释 幅 丨 扶 寻 浪! 缓! 洲" 彻" 缴# 勤# 遗$ 陆$ 舒% 灯% 递& 著& 妇' 扩' 肌( 沉( 厚) 润) 焦* 凡* 赶+ 祝+ 锅, 洋, 慧- 雄- 滑. 巨. 援/ 宽/ 尤0 雅0 肝1 弹1 津2 勇2 挂3 п3 填4 妻4 蒙5 哦5 朗6 辅6 闭7 惯7 沟8 烧8 伟9 概9 刑: 漫: 射; 贯; 吴< 敬< 雷= 甲= 刺> 籍> 赵? 践? 鼓@ 挑@ 偏A 甜A 弱B 贷B 档C •C 拟D 唐D 仪E 淡E 缘F 胡F 弃G 甘G 湾H 姓H 巧I 敢I 奥J 坏J 钢K 暂K 孙L 署L 恩M 宫M 泡N 脏N 聊O 汤O 纯P 泉P 癌Q 宿Q 讨R 腿R 灾S 迷S 摩T 恢T 羊U 宗U 唯V 欧V 忙W 镜W 凉X 陪X 殊Y 刷Y 阵Z 肠Z 昌[ 潮[ 抽\ 肯\ 嫌] 窗] 尊^ 「^ 」_ 珍_ 泰` 郑` 赏a 梅a 赚b 耳b 池c 徐c 污d 阻d 默e 晓e 悉f 遍f 祖g 浓g 腰h 倍h 君i 辛i 唱j 宅j 晨k 敏k 桃l 估l 弟m 旦m 绪n 浙n 斤o 舍o 拼p 谷p 扣q 珠q 梁r 描r 盐s 嘉s 肖t 幕t 毫u 脉u 忆v 尼v 繁w 跳w 扎x 抱x 尿y 楚y 脂z 泽z 狗{ 伍{ 洞| 振| 筹} 混} 饰~ 圣~ 残 恋 柔 á 腹 衡 韩 谓 迁  ` 闲 败 迹 绕 储 贸 凭 畅 俗 船 圳 燃 胞 烦 帝 耐 券 妹 豪 尾 抵 杯 瓶 映 暗 诺 跨 乌 欲 仙 í 乳 顿 缩 炼 迅 跌 牢 瑞 铺 朱 沿 瓜 丹 搜 赢 墙 横 巡 览 径 偿 г 飘 蜜 垃 鞋 秒 刀 摆 榜 森 圾 я 疼 呈 矿 肿 眠 屏 佩 虫 驻 肾 岸 膜 携 盟 ó 塔 宋 炒 愈 脾 溪 遵 婆 滨 鲁 皇 腾 洪 灭 莫 壁 紫 辉 辈 伊 丁 徽 穴 债 胶 潜 澳 宇 页 锋 役 废 仁 虎 胸 震 搞 寿 宾 з 磨 拆 兼 覆 贡 循´ 嘴´ 孕ô 赠ô 桂Ĵ 冒Ĵ 颗Ŵ 栏Ŵ 欣ƴ іƴ 恐Ǵ 粮Ǵ 爷ȴ 旺ȴ 替ɴ 遭ɴ 锁ʴ 拒ʴ 轨˴ 杭˴ 晋̴ 忍̴ 疆ʹ 贝ʹ 邀δ 薄δ 麦ϴ 辣ϴ 郭д 坛д 鉴Ѵ 硕Ѵ 惜Ҵ 吨Ҵ 萨Ӵ 诈Ӵ 暑Դ 誉Դ 狂մ 董մ 绘ִ 裤ִ 碰״ 猫״ 逃ش 诸ش 迟ٴ 锦ٴ 氧ڴ →ڴ 亦۴ ь۴ 辖ܴ 梯ܴ 纹ݴ 旁ݴ 悦޴ 尖޴ 鼻ߴ 氏ߴ ㎡ 拜 肃 孔 凤 殖 滴 б 逆 ℃ 娘 塞 竹 艾 汗 迪 陕 凝 滋 忠 沈 肩 伸 怪 姜 裁 袋 渠 胆 腐 履 尝 扰 壮 洛 谋 仔 胎 妙 拔 塑 陷 恒 隆 炸 谱 悲 姻 杰 偶 椒 奉 娱 奔 泛 拓 漏 煮 倾 爽 桌 纠 祥 劲 й 徒 乏 疏 趋 » 泥 喷 浮 杜 械 摸 捐 挺 耗 ä £ 忧 撑 秦 枝 睛 悟 晒 偷 井 腔 伦 馨 忽 ы 牵 凌 伏 煤 宏 俄 穷 盒 迫 抑 插 坦 踏 魔 刊 坡 哭 邓 琴 闹 灰 泪 寸 鸟 碎 貌 赔 辽 © 燕 捷 厨 涂 墨 姿 艳! 斑! 袭" 揭" 芳# 赴# 昨$  ^$ 郁% ×% 漂& 颈& 戒' 吐' 粒( 艰( 柳) 敌) 厦* 荷* 瘦+ 氛+ 曝, 叔, 饱- ч- 孤. 摘. 醉/ 丈/ 涵0 仓0 寄1 魂1 剩2 莲2 亩3 摇3 弯4 蔬4 衰5 腺5 ü6 汁6 坊7 仿7 堆8 啥8 朵9 霸9 跃: 盆: 冻; 吹; 碍< 译< 瘤= 皆= 晶> 骑> 凯? 撞? 咳@ 奏@ 尘A 杆A 鼠B 惑B 彼C 挖C 铜D 陵D 拖E 搬E 枪F 耀F 邻G 番G 烤H 昆H 诱I 劝I 尺J 驱J 锻K 慎K 娃L 粗L 胖M 岭M 裂N 蜂N 瞬O 淘O 炉P 纵P 厉Q 妆Q 辞R 蒸R 盾S 塘S 岩T 阔T 滚U 剑U 慰V 堵V 剪W 烂W 允X 踪X 抖Y 赖Y 夺Z 菲Z 捕[ 串[ 鸣\ 碗\ 浅] 酱] ö^ 伯^ 亏_ 闪_ 兄` 渡` 怨a 谨a 鬼b 浦b 碳c 勒c 丢d 咖d 盈e 擦e 崇f 撒f 熬g 屈g °h 赋h 拨i 虾i 嫩j 浩j 腻k 弄k 聪l 陶l 忌m 曹m ་n 肚n 柜o 淮o 逻p 廉p 耕q 莱q 柱r ①r 旋s 稍s 卧t 幻t 挡u 迈u 彰v 盗v 篮w 疲w 筋x 齿x 栋y 饼y 宠z 倡z 怒{ 裙{ 姑| 疯| 稀} 寺} 乙~ 叉~ 纽 ② 馈 恰 鸭 毁 苹 呀 魅 熊 犹 诵 慈 芝 雾 瓦 芯 纤 悬 侦 贺 诞 浴 牧 框 秩 ​ 抚 脆 拳 葱 х 囊 贤 燥 孝 赤 谣 悔 娜 霞 撤 愁 谐 匀 曼 妥 雕 寓 咱 棉 婴 ↑ 眉 糊 猜 吊 酬 逼 悠 窝 鹏 赌 舌 荡 矛 骤 恼 猛 玛 乃 仰 肪 晰 勿 糕 涛 坑 垂 岳 湘 俊 钻 茂 孟 泳 俱 御 邦 兔 晴 灌 盲 堪 狼 拘 傲 粘 绵 淋 卓 帅 玻 钓 灸 喊 碧 弘 羽 菩 割 è 膏 慕 幽 欠 舰 锡 巾 堡 摊 泄 酷 磅 昂 璃 棒 萝 哲 傅 姆 桑 贪 夕  µ 叠µ 旨õ 魏õ 碑ĵ 彭ĵ 逢ŵ 鸿ŵ 祭Ƶ 苑Ƶ жǵ 悄ǵ 浸ȵ 恨ȵ 弥ɵ 舟ɵ łʵ 蔡ʵ 韵˵ 萌˵ 缝̵ 葡̵ 轴͵ 催͵ 醇ε 巩ε 黎ϵ 玫ϵ 纲е àе 泌ѵ 夹ѵ 丘ҵ 骂ҵ цӵ 抹ӵ 臭Ե 邪Ե 卜յ 玲յ 株ֵ 瑰ֵ 凶׵ 枣׵ 袁ص 螺ص 旬ٵ 蛇ٵ 狮ڵ 函ڵ 遥۵ 慌۵ 鼎ܵ 亭ܵ 辰ݵ 蒜ݵ 淀޵ 洒޵ 蒋ߵ 拌ߵ 奈 砂 筛 押 暨 炮 嗽 萄 椎 棚 翠 抛 颖 胀 斜 辨 匹 峡 抬 鹅 溶 霍 兽 寨 募 伪 ا 挤 欺 丧 爬 柴 涌 廊 桶 卢 咸 壳 瓷 虹 驰 咽 契 霜 惧 懒 啡 掘 宴 辩 贾 叹 衔 颁 陌 擅 苍 蟹 渔 梳 臂 惨 荒 膝 挣 詹 狱 厌 咬 冯 滩 埋 巷 畏 槽 ③ 逸 兑 渴 浆 乔 祸 妨 庙 敲 茅 钙 嫁 帽 墓 鸽 闷 鹿 丑 椅 垫 ® 扭 郎 粤 饲 翼 吕 讼 劣 柏 肢 豫 朴 泻 夸 稻 砖 狠 歉 猴 芽 汪 遮 戳 〔 鹤 〕 矩 惩 埃 棋 滞 毅 ☞ 俩! 壶! 丸" 吵"  # 蹈# 莉$ 裕$ 侣% 憾% 潭& 披& 骄' 痕' 锐( 壤( 斌) 耶) 躺* 胁* 铭+ 瑶+ 娇, 姚, 盼- 梨- 翔. 勾. 煎/ 溃/ 痘0 匠0 赫1 搏1 呆2 惕2 厘3 潘3 蓄4 喂4 禅5 攀5 羡6 枚6 ш7 犬7 溢8 厕8 茫9 霉9 鄂: 磁: 卵; 扑; 扮< 罐< 汛= 菇= 傻> 晕> 槛? 荆? 粥@ 沧@ 箭A 葛A 冈B 侯B 巢C 芒C 愉D 姨D 坪E €E 咒F 萍F 逾G 唤G 芬H 枢H 捞I 鑫I 趁J 浇J 膀K 钩K 脊L 仲L 婷M 祛M 牲N юN 莞O 勃O 轰P 泊P 腊Q 赣Q ‍R 沃R 渣S 坝S 奢T 袖T 峻U 躁U 绑V 揉V 逝W 蓬W 氨X 醋X 玄Y 煌Y 涯Z 臣Z 琳[ 崩[ 拾\ 芦\ 饿] 帆] 漠^ 瓣^ 谊_ 吓_ 禄` 靖` Сa «a çb 勺b 菊c 酵c 懈d 帖d 萧e 赁e 枯f 瑜f 梗g 葬g 痒h 痰h 碌i 滤i 浏j 衬j 颇k 炭k 躲l 汕l 刮m 藤m 阜n ️n 渗o 绳o 蔓p 抄p åq 踩q 陀r 捧r 敦s 衫s 缠t 蕴t 牺u 尸u 誓v 沪v 弗w 轩w 杏x 扇x 辟y 绒y 裹z 禽z 嘛{ 阁{ ل| 薇| 闯} 遂} 匆~ 铸~ 笼 怡 旭 猎 伐 寂 喉 樱 沂 畜 罢 衷 敷 昏 掩 谅 溺 兆 ى 愤 嘱 劫 恭 烫 坤 蓉 吾 辱 甸 窃 宙 伞 蒂 浑 韦 栽 戚 � ф 邵 蕾 搅 呵 П 逛 茄 殿 唇 氢 扔 喻 坠 娟 诀 薯 伽 墅 卑 盯 酿 谭 庞 绣 肆 铃 挽 哀 剖 盔 榆 鹰 沫 â 吞 弦 凸 熙 祈 乾 灿 枕 桩 卸 舱 桐 宪 塌 隧 廷 坎 撰 肇 昭 泼 黏 舆 皱 轿 钉 侠 匙 卦 澡 裸 曰 贩 涩 纺 瞒 栓 丙 莹 秉 糟 丛 ñ 莎 橙 ④ 顽 湛 谦 颠 纱 碱 缅 磷 淫 凰 饶 奴 侨 妮 刹 雁¶ 磊¶ 竭ö 耍ö 惹Ķ 韧Ķ ВŶ 宵Ŷ 撕ƶ 挫ƶ 痴Ƕ 邢Ƕ 杞ȶ 媳ȶ 焊ɶ のɶ 摔ʶ 汾ʶ 屯˶ 逊˶ 剥̶ 仇̶ 疹Ͷ 崔Ͷ 渝ζ 愧ζ 羞϶ 凑϶ 仗ж 枫ж úѶ 郊Ѷ 尬Ҷ 捡Ҷ 溯Ӷ 蝶Ӷ 蝎Զ 琼Զ 澜ն 筷ն 叙ֶ 窄ֶ 叮׶ 罕׶ ěض 炫ض 僧ٶ 焕ٶ 砸ڶ 沾ڶ 橘۶ 昔۶ 蒲ܶ 烹ܶ 牡ݶ 沸ݶ 灶޶ 翁޶ 喘߶ 斩߶ 厢 臀 髓 冀 铝 澄 蚊 妃 ј 怖 い 尴 勘 棵 扛 怜 歇 矫 柯 慨 漆 ã 吁 螨 荔 筒 遣 闽   ™ 粪 雀 酥 勉 翅 旱 ă 氯 襄 К 堤 拦 ę 溜 帕 谎 弊 脖 郡 翰 沁 啤 闺 佐 颂 薛 庸 彦 奠 炖 屠 屁 壹 捉 斥 酶 蹲 削 衍 贿 瘾 搓 哟 阀 莓 潍 叛 ن 寡 琪 枸 赐 疮 彬 宰 砍 饺 屡 卿 ­ 糯 拐 钾 呕 ས 冶 奸 帐 狐 庚 膳 辐 渊 黔 硝 蚁 泗 瘟 骚 艇 吻 绽 ą 缸 殷 虐 汰 硅 蜀 撼 娶 ر 韶 甩 戈 麟 蛮 秤 崖 妖 隙! 锤! 杠" 漳" 勋# 歧# 粹$ 硫$ 烘% 淄% 绥& ā& 沐' 邹' 邱( ☟( 竖) 靓) 扯* 噪* 埔+ 婉+ 倩, 捏, 潇- 魄- 乖. 兜. 佑/ 呗/ 钥0 扁0 挚1 滥1 帘2 č2 穆3 な3 咪4 凹4 揽5 琦5 辜6 馅6 芜7 窑7 妄8 泵8 垒9 儒9 僵: 攒: 龟; 媚; 胺< 腌< 腕= 钦= 隶> 钮> 沥? 柿? م@ 驴@ 挨A たA 哄B 仕B 珊C 皂C 喀D 邑D 敞E 瑟E řF 雇F 瘀G 艘G 廖H 绎H 胰I 贼I 寞J 狭J らK ¥K 煞L 栖L 俭M 埠M 禹N 驳N 顷O 尹O 脐P 铅P 睿Q 蚀Q 羯R 熏R 烯S 楷S 嘲T 锈T 愚U 媛U 钠V 掏V šW 阐W あX ➤X 咋Y 谜Y НZ 笋Z 渭[ 拭[ 祷\ 榴\ 嚼] ت] 鲍^ 昙^ 萎_ 恤_ М` 楠` 袜a Рa 倦b 膨b 爪c 畔c 淑d 柠d 籽e 晃e 遏f 蕉f 庐g щg 莆h 堰h 撇i 崛i 疡j 耽j 陇k 锂k 洱l 趟l 屑m 郸m 夷n 汝n 逗o 嗨o 兹p 蔽p Аq 梧q 诠r 琐r 栗s 闸s 耻t 歪t 洽u ýu 砺v 焚v 苔w 雯w 聆x ►x 俞y 浊y 驿z 蜡z ي{ 掀{ 韭| 惫| ←} 巅} 橡~ 坟~ 梭 乒 蹄 俯 甄 祁 翡 踢 跪 嘎 粽 禾 敛 邯 瞄 鞍 窍 蛙 垢 剔 刃 稣 ′ 腑 惟 杉 檬 彤 蚕 이 擎 棘 蚂 弓 蔚 沛 傍 贞 葫 芹 嫂 扒 佬 竣 缀 颤 し 鞭 饥 て 萃 泸 挪 ś 菏 ག 纬 翘 ž 卉 倪 饪 饵 迭 菱 爹 夯 拇 涡 芙 睁 嵌 恳 舶 蹭 冤 裔 骏 卤 遴 逮 瞧 ø 耿 髦 谛 笨 侃 暮 و 豁 熔 匈 寅 肘 穗 撩 다 ² 棠 ż 菠 腥 缕 忻 斋 兮 拯 喇 佣 吟 ﹌ 酯 冥 黛 ї 铲 仑 昊 э 氮 茎 哺 ん ི 矮 孜 漓 棕 牟 ⑤ 讽 璐 郝 ß· 囤· 宛÷ 匿÷ 袍ķ 驼ķ 缆ŷ ོŷ 瀑Ʒ 昧Ʒ 侥Ƿ 荟Ƿ 贬ȷ 虞ȷ 晖ɷ 酮ɷ 皖ʷ 澎ʷ 坯˷ 咏˷ 昼̷ 葵̷ 湃ͷ 汲ͷ 沦η 嚣η 龚Ϸ ДϷ 渍з 暇з 睹ѷ 焰ѷ 绷ҷ 淤ҷ 哑ӷ ÷ӷ 瘫Է 棍Է 帜շ 藕շ 茨ַ 祀ַ 飙׷ Б׷ 紊ط 哨ط 鲤ٷ 淳ٷ αڷ æڷ 煲۷ 匪۷ 晾ܷ Оܷ 讳ݷ 毯ݷ 蠢޷ 濮޷ 叭߷ 殴߷ ê ད を 睐 巍 豹 稠 亨 捣 د 糙 诫 砥 樊 蹦 › 喧 盏 奕 烛 捺 觅 蝉 娥 奎 椰 睦 褐 梵 彪 绸 と 甫 苯 痹 岂  蝴 洼 √ 霖 垄 雍 摧 姥 窟 祠 娄 拎 矣 钞 拱 孵 憋 侈 泣 沭 胱 梓 朔 冉 淇 淹 亥 ″ に 岚 竿 闫 § は 眸 蘑 尧 绚 拢 厄 旷 霆 驭 ۛ 魁 „ 怠 舅 汹 筝 瞎 锌 垮 卒 É 爵 が 窦 弧 肛 寝 ≥ 趴 涝 ب 璧 鸦 靶 芋 姬 抒 卯 柚 磕 蚌 î 雌 硒 坷 廓 麒 陋 ە Т 妍 娅 瞻 ᠠ 峪 肽 藩 灼 咯 乓! 哇! 墩" 泾" བ# 芪# 柑$ 曦$ 榨% 躯% 骼& 絮& 靡' 荫' 玺( 绞( 缚) 妒) 巫* س* 幢+ 屎+ 捂, 苟, 苛- 汀- 叨. 衅. 哒/ 稚/ 笛0 ⊙0 畸1 蚝1 讶2 馒2 臻3  3 侍4 丫4 澈5 栈5 芭6 ♩6 Г7 ق7 吼8 禧8 藻9 阱9 歼: 颐: 肋; 柄; 妊< 骆< 痔= 涤= 芸> 弛> 嗓? 俏? 昕@ 槐@ 簿A 曙A 嗅B 崭B れC 帷C 屿D 娠D ìE ćE 眷F 痣F 倚G 犀G 酌H 靴H 惬I 兢I 茜J 芥J かK 炳K 绊L 滕L 啪M 荧M 乍N 猝N 瑾O རO 聂P 桔P 楞Q 榄Q 趾R 缔R 贱S 狸S ИT くT 圩U ۚU șV 呐V 哮W 焯W ёX ôX 寥Y 轼Y 琢Z 凳Z 酉[ 嚏[ 褪\ 龈\ 捆] 迦] 麓^ 噩^ 俺_ 梢_ 懵` 嫉` 釉a 堕a ۙb 祂b ངc 噢c 殡d 蜕d 苓e 挠e 赘f 氟f 蘸g 镀g 矢h 滇h 楂i ♪i 酚j 漱j るk 拂k 棱l 凄l 栅m みm 剁n 仆n 笃o òo 嗯p 醛p 杖q 倘q 胚r 哗r 琅s 揣s 🏻t 唠t ནu 涕u 篷v 砌v 乞w 踝w 蛛x 鳞x 咕y Уy 戊z 枉z ț{ ུ{ 庇| 垦| 悼} 涅} 霾~ 嘿~ 嗜 稽 ♫ 缉 脓 喵 搁 瞩 皓 亢 踊 咚 毗 霄 颅 幂 僚 藉 锣 阎 秆 碾 翩 酰 鲸 弈 ´ 戌 茗 炙 贮 Л ≤ 葩 烁 熄 啃 悍 惰 ま 碟 巳 嘘 檀 ‬ ा 焙 坞 瀚 ك ń 榕 鹊 〉 瑕 疤 逅 镶 缤 晦 璨 瞰 慑 隋 橱 邂 哎 矶 瘙 钧 峨 ▷ 诡 拙 μ ♬ ± 镁 揪 渤 眩 ù 粟 秃 斛  钰 焖 莘 ə ྱ 瞭 畴 俘 橄 掐 恍 衢 蜗 璀 蝇 鞠 帼 颍 肴 ᠢ 秸 柬 赎 岐 嘻 彝 雏 β འ ⑥ 檐 咀 篆 垣 绛 ✦ 撸 り 嘟 こ 岑 蠕 ‎ 忏 掺 嫖 崎 滁 窜¸ 唑¸ 〈ø 怼ø 癫ĸ うĸ 汶Ÿ 跋Ÿ ōƸ 阑Ƹ ëǸ īǸ 啸ȸ 岔ȸ 樟ɸ 馋ɸ 漯ʸ 娩ʸ 棺˸ Ф˸ 朽̸ 侮̸ 滔͸ 熠͸ 涟θ 暧θ 耘ϸ ۥϸ 을и 淌и 唾Ѹ 珑Ѹ 逍Ҹ 噬Ҹ 茵Ӹ 婿Ӹ 窥Ը 阮Ը 涧ո 聋ո 壬ָ ۇָ З׸ 尉׸ 渲ظ 嬉ظ 恺ٸ 婧ٸ 焉ڸ 哉ڸ 窖۸ 闵۸ 哩ܸ ůܸ 茯ݸ 晏ݸ ེ޸ 孢޸ 犁߸ 璋߸ 窒 烙 で 嵩 疱 ། さ 褶 厥 茹 ½ す 裴 躬 祺 痧 羁 腋 癸 覃 盎 迄 ش 赃 〇 辙 苞 斧 憨 呦 ‌ 秧 侄 锥 禺 猕 ɪ 陡 拧 菁 薏 宸 མ 膊 翟 ئ 钝 ― 茸 渺 郴 蕊 ο 抠 辗 恕 玮 煜 沽 ー 痊 拣 濒 铂 ᠨ 羔 荤 劈 斐 拽 ལ 陂 悸 徊 胳 ∶ 掠 缪 ᠭ 珀 舔 會 髋 伺 咙 하 僻 绮 靳 峥 佼 桓 र 皋 蔗 徘 豚 碘 ‖ き 咐 淼 炬 匮 睫 Ⅱ 黍 고 鹭 亳 潼 酪 ˈ 鲫 疚 绅 咧 簇 茧 掷 淅 襟 猩 憬 恬 锚 苷 绰 於 鬓 颊! 扉! 掰" 螃" 佰# 는# 漾$ 涮$ 喽% 铮% 掖& 娴& 玥' 喔' 昵( 芍( 蝠) 蜘) 钵* 遛* 秽+ 憧+ 凿, 耸, 薰- є- 炜. 溉. 茉/ 裳/ 轧0 捍0 谍1 眶1 亵2 茬2 锯3 烊3 钛4 痫4 譬5 莺5 愣6 懿6 挛7 ۡ7 噜8 溅8 猥9 慷9 ᵉ: 怅: ı; 嫣; 晟< 妞< 惶= ۟= 瞳> 囚> 摹? 羲? 萱@ 俐@ 舜A 怯A 寇B 眨B ᠤC 锵C 啼D 烷D 骁E ɡE ЕF っF 鹃G 蛰G 笙H 葆H 剃I  I 呜J 瓯J 溧K õK 毋L 蒿L 伶M 玖M 桦N 颌N عO 璇O 谧P 疵P 浒Q 辄Q 稼R 羹R 惦S 哼S 姗T ïT 凋U もU 痪V 痉V 恪W 钗W ्X कX 그Y 蝙Y 腱Z 鄙Z 莽[ 眺[ 沮\ 雹\ 穹] 钳] 庵^ 韬^ 绯_ 柒_ 刁` ه`  a 蛾a 阙b 鳌b 虔c 坍c 礴d 袅d 昱e 朦e 丞f 歹f 腮g τg 冕h 狄h 斓i だi 缨j 琉j 辍k ιk 飒l 栩l 蓟m یm 시n 帧n 胧o 屹o 祢p 鸥p ǎq ةq 钊r 哔r 槟s 啰s Ⅰt 俪t 芷u 撮u 鲨v 蝗v 恙w 胯w 砚x 缭x 匝y 诃y 어z 篱z 榻{ 钜{ ─| ő| 铬} 酋} 栾~ 祉~ 悴 淆 诏 憔 腩 煸 岱 皙 晗 瞪 纶 峙 à 椿 ́ ご 炊 慵 Ⅲ 窘 佟 谴 荼 毙 − 寰 镯 기 磺 苇 戎 捅 毓 ε Ö ز ū 睢 铿 匡 疙 ほ 邸 粑 撬 痢 珂 े 朐 敖 한 瘩 吝 嫦 蛤 呤 猿 唉 挟 呱 ❖ 가 霓 ν 惆 峭 ・ 凛 娓 酣 悯 嬛 镑 Х 墟 お 簧 粱 狡 했 ’ ᠯ ★ 阖 婪 缮 そ ف ✪ 嗦 지 赡 怂 砀 羌 ᠡ 崽 徙 嘌 љ 叩 挝 陨 Á 擀 筐 亟 丐 睾 孚 剿 ゆ 霏 ē 莒 潺 懦 憩 豌 扼 跻 抉 贰 寐 ⑦ 倔 ン 媲 า њ 遁 ، 磐 鸳 摒¹ ྲ¹ 姊ù 에ù 惮Ĺ 鳄Ĺ 淖Ź 樾Ź 皈ƹ 莅ƹ 宕ǹ 仄ǹ 蹬ȹ 夭ȹ 礁ɹ 辫ɹ 禀ʹ 廿ʹ 栀˹ 湄˹ 逞̹ 烨̹ Ч͹ 匾͹ 擒ι 黯ι 炽Ϲ 濡Ϲ 螂й 刨й πѹ 咔ѹ 隅ҹ ᵒҹ 谤ӹ ρӹ 罄Թ 驯Թ 粕չ 烽չ 瞅ֹ 曳ֹ 鄱׹ 狙׹ 澧ع 涓ع 郫ٹ 鹦ٹ 闰ڹ 驹ڹ י۹  ۹ 馍ܹ 蛳ܹ 煽ݹ 荞ݹ 俨޹ 甬޹ 豉߹ 吏߹ 鸯 觑 糜 潢 ◐ ✎ 琶 唏 拷 潞 芡 眯 镍 癖 芮 妾 པ 殃 藿 泓 捶 º 荠 癣 跷 扳 擂 锰 赓 箫 न 颓 沼 攘 赂 萤 谚 椭 ि 嵘 َ 苋 迢 嗣 犊 氓 ᵗ 呛 诲 璞 嚷 들 叽 褚 殆 婕 嚎 鹉 邃 勐 屉 ᠣ 纾 酝 滦 筱 骋 嗡 褒 蔑 脘 棣 貔 ᠦ 迸 萦 铵 悚 忱 娼 پ け 虏 σ 貅 璟 磋 轶 杷 镂 稷 ѕ え 踵 枇 Ш 癜 🙂 吒 瑙 ו 峦 圃 缜 蔻 硼 簸 ち 婺 婵 惋 स 渎 น 麽 해 惺 茴 袱 ‐ 盂 燎 铠 榔 서 篡 搂 煦 瑛 嘈 痿! ी! 瑚" 坂" 贻# 瓢# 歩$ 槃$ 喃% ど% 圭& Ø& 舵' 筠' 涿( 蜒( 대) 沓) 게* 嘶* 骇+ 婶+ 毂, 烃, 孰- 琥- 蜿. ❶. 壕/ 涎/ 哆0 赦0 ᠷ1 渚1 琛2 め2 厝3 藜3 谬4 兀4 つ5 茁5 †6 니6 叁7 镌7 拮8 蓓8 鞘9 œ9 镖: 馏: 炯; 龋; 涪< や< 褥= ่= 睑> 侬> ¢? 憎? 锄@ 浔@ 砧A スA 沅B 殉B κC 诬C 祟D 籁D 潦E 啬E 骸F 蟑F 蜓G 胛G 浚H 隘H ◑I 晤I 耒J รJ ♡K 罹K 嵊L 琵L 溥M 薅M 衙N 貂N มO 跤O جP 懊P 蜻Q 旌Q 淬R 藁R 岷S ☛S 瘴T मT 蛟U 藐U λV 祯V 铛W 嗑W 羚X 蔺X 饽Y 면Y 厮Z 姣Z 惭[ 吱[ Š\ 浣\ 妓] 壑] 汐^ 遐^ 奚_ 燊_ 箍` 拴` 诋a เa 葚b ³b 鱿c 馥c 汞d 铎d 胤e 蔷e 만f 鸾f 嗔g 彷g 笠h 蟠h 隽i ͟i 楹j 啧j 臃k 庶k 孽l 竺l 鸪m 嗷m 娑n €n ςo 札o 瞿p 疸p 蹊q ᵃq ﹏r ᠲr 羟s 상s 蛀t t 鹧u 아u 拈v 枞v 漩w 佘w 蜚x 捻x ⑧y 惚y 灞z 吡z 妩{ γ{ 嗝| 戍| 帛} 猾} 谆~ ǐ~ 沌 ง 喳 Ü 야 缙 揍 ∣ 鹑 膈 狩 郓 ح 幌 Č 崂 자 诣 ͞ ं 熹 弋 ⁿ 蛎 侗 粳 त む 釜 踮 爲 皿 驮 阆 리 ː 俑 ้ 祎 喱 ş 桨 도 อ 洮 谙 膛 漪 沱 儋 攸 曜 讴 ト ᠳ 姝 ❷ 鹌 氪 坻 芊 咎 傣 蔼 酗 垛 唧 η 辘 钴 皎 坨 臧 莴 痞 袄 罂 裆 ● 熨 诽 荥 ğ 珺 箴 醚 缎 ω 悖 疟 荨 爻 搐 诛 鸠 خ 唔 ˢ 囧 咫 沣 胫 歙 绢 ◈ 谩 踹 猖 疽 匣 蠡 氰 瘪 粼 笈 冽 炕 ล 骶 捋 樵 矜 诅 蚓 ڭ 嘀 ⁱ Э 吮 寮 ཁ 로 ᠬ 枭 偕º 膻º 주ú 纫ú غĺ 扦ĺ 喆ź लź 斟ƺ イƺ 嫔Ǻ 鼾Ǻ 艮Ⱥ 糠Ⱥ ᠰɺ पɺ ‮ʺ 踞ʺ 戟˺ 痼˺ 모̺ 谏̺ 馁ͺ 邳ͺ 琍κ 痤κ 咩Ϻ הϺ 鳖к 馄к 荀Ѻ 饨Ѻ 阪Һ 忡Һ 塾Ӻ 〖Ӻ 〗Ժ 哽Ժ 瀛պ 泞պ 鲲ֺ 를ֺ 虱׺ ✍׺ 蚯غ 吭غ 讷ٺ 啄ٺ 蟒ں 胥ں 柘ۺ 邬ۺ 鄞ܺ 搀ܺ ≠ݺ 徨ݺ 翱޺ 忐޺ 나ߺ 할ߺ ह 忑 畲 镰 骅 ћ 噱 ‑ 수 Å 嘹 炅 ‚ µ 偃 渥 ʳ 岖 珞 箔 ท 狈 巜 戾 ð 戮 萸 ᠪ 과 濠 淞 巽 ル 茱 裘 徉 亘 纂 Я 濛 砼 य 杳 鄢 槿 徜 骛 ั 讪 戛 叒 唬 皑 었 诶 恃 桢 缇 伎 ˡ 埵 일 υ 偈 أ ۈ 宦 甥 蕲 玑 喹 ี 涣 芩 诧 鬟 ᠵ 喋 榭 弾 ต 肱 溏 辕 颧 ᠶ 의 鳝 ྒ “ 岫 咄 ุ 鲈 چ 霁 膘 ก ר 마 陉 丶 것 兖 笺 羿 瓴 诟 ว 耦 帚 冗 嫡 昀 蚜 骥 邋 ל ې わ 榈 ो 迥 인 铣 蹴 瓮 菅 麸 ☾ 砰! 脯! 猬" 耨" 孺# 焱# 弼$ 吆$ 膺% 琨% І& 漉& 銮' ъ' 烩( ”( 诿) 邕) 雳* 疣* 叼+ 枳+ 盱, 안, よ- 湍- 噻. 擞. 噎/ 鲅/ 矗0 撂0 ≈1 菡1 菖2 奄2 گ3 リ3 疝4 噶4 痈5 拗5 ཞ6 鳗6 飕7 芎7 阂8 Φ8 袂9 ཀ9 吖: 미: 蜱; ת; 蓦< ¼< 诩= א= 郧> 邺> 町? व? 肮@ 褂@ δA ❸A 铆B 幄B 苒C 烬C 泱D 霹D 枷E 咆E F 거F 蛊G ⃣G 攥H ཚH ЦI ཡI 吩J 遢J 饕K ◉K 佚L 胭L 邗M —M 抨N 滘N 은O 汴O 朕P 遨P ʰQ 怦Q 罔R 旎R 晔S 丕S 恣T 佗T 旖U 袤U 眙V 峒V 蜷W 暄W 谯X 偎X 琊Y ยY 囍Z 蟾Z ⑨[ 卞[ ǔ\ 仝\ 殇] ź] ด^ 猷^ 坳_ 阡_ 徇` ǒ` 迂a 醴a 漕b 啶b 있c 幡c 跆d 诙d 锭e 罡e 裱f 娆f 邈g Óg ᠴh 焗h ᠮi 瘠i दj 泯j 痂k 耙k ✿l 锲l 袒m 娲m 愫n 摁n 사o 요o 舀p 鸢p 谌q 頫q 醺r ‰r 鳅s 幺s 涞t ∠t 르u 箕u 腓v 缈v 髂w 挎w じx 嗒x 살y 溴y ➢z ‹z 瞌{ ཆ{ Ⅳ| ค| ❀} 垭} ב~ 讹~ 隍 篝 岌 吠 讥 ๚ 蹿 瞥 矾 塬 처 พ 缥 冢 蕃 荚 든 臆 毡 া ラ 糍 ¿ 스 奂 涸 蕙 飓 Î 별 弩 蚤 嗖 桡 ₹ Ä 독 捎 唆 邛 臼 昇 苡 郯 埂 מ 氲 辊 歆 ช 藓 ク ۗ 垠 锹 Ž 掂 嘅 擘 秣 ═ 腆 บ ส 钼 筏 钨 θ 扈 伫 원 ɑ 钣 芾 簪 砾 蟆 髎 위 纭 嬴 ص 篓 瘢 劾 窿 간 へ 郏 茼 訇 苁 獭 蝌 叕 ッ 汨 惘 氤 蹒 ί 徵 蕨 甭 醍 挞 ฅ 킁 佢 荏 嬷 浠 翎 阈 綦 窈 旻 铢 ש 珲 嗤 蚪 ア 濑 浜 琰 ะ 崆 跚 哧 嗪 蜈 瘁 Ж đ» 盅» 劵û 麝û 珏Ļ 黝Ļ 珈Ż ไŻ ▣ƻ 搪ƻ 叱ǻ 제ǻ ねȻ ΔȻ 嚓ɻ İɻ ِʻ 蹉ʻ 苣˻ Ś˻ 裨̻ 瘘̻ 垌ͻ 眈ͻ 러λ 𝓼λ fiϻ 鹜ϻ せл طл 窨ѻ 骞ѻ 睬һ ๑һ 孪ӻ 锢ӻ 鳍Ի 婀Ի ۋջ 夙ջ کֻ 醐ֻ Ú׻ ᵘ׻ 龛ػ 笆ػ ྐٻ 呷ٻ ︰ڻ ھڻ 犷ۻ 啵ۻ 珅ܻ 郦ܻ 茌ݻ ᠩݻ 跎޻ ज޻ ¡߻ 餮߻ ʸ 仟 畈 匕 ¾ 呃 墒 踌 鄠 赭 ْ 赊 娣 แ 嶂 晁 남 绶 № 杵 ⑴ 腧 ょ 锏 佤 埇 ¹ 湟 猗 铤 ᵈ 脍 峯 芗 铐 镳 囱 ¶ ཐ ず 泠 髻 ิ 躇 酐 소 咦 내 ็ 槌 ⑵ 煨 炀 噗 蔫 脲 邡 腈 谟 ʊ • ๐ 濂 腭 呻 灏 ཕ ά 鲢 啉 牦 많 骊 咛 蚣 纣 陛 라 儡 陟 ณ û 抿 전 ब 陲 ห 聿 渑 掸 鳕 ™ 衾 栉 ๎ 飚 胍 荃 蜇 颦 咤 翌 ฏ ︎ ђ ข 窕 碣 瘸 骡 ʃ ᠎ 皴 ɔ 闱 轲 비 迩 滢 ү 犒 ʻ 汆 箩 敕 Ð ☜ 螳 鹄 ธ 谕! 髌! 尕" 연" 摞# 堑# 甙$ φ$ 粿% 跺% 拄& À& ⑩' 嘤' 腼( Ω( 蓑) 翊) 드* ư* 雉+ ु+ 啾, 珐, 怔- 偌- ๖. ป. 황/ 闾/ 碉0 埭0 โ1 撷1 ‭2 腴2 ᠋3 ์3 桉4 蛹4 霭5 祐5 剐6 仨6 樽7 掬7 鞣8 侏8 镉9 무9 珩: 冼: 蹶; 浃; 呋< 镐< 柞= 奘= 琏> 崃> ग? ்? 岙@ 보@ 萏A 帏A 搔B 膦B 筵C ❹C 焘D ใD 馕E 镕E 阚F 剽F ňG աG 啫H ﹑H রI 睇I 遒J │J 掇K 婢K 凇L 鹂L 蹋M ُM ฌN ึN 翳O ÇO 蟋P 颚P ЮQ 耄Q 啕R 반R ªS 傀S 瓤T 蟀T 屌U 耋U 濉V 噔V 嗳W 蹚W 茏X 浐X 掣Y 榛Y 癀Z 箸Z 忿[ 〡[ 楔\ 鏖\ 跛] 노] ό^ 鸵^ 경_ 疃_ ู` 糗` 贲a ླa ྟb 嘧b 茭c ¬c 珙d 叟d 嘭e 逵e 绫f 饴f 豇g 와g 稞h 妲h 哋i 睒i ぎj 泮j 徕k 妤k ❤l 狞l 阉m 𝓮m 榫n จn 恿o 盹o 倭p ✓p 砷q タq 甾r 赈r 唰s 谪s ロt 荸t 纨u 轳u 痱v 懋v ¦w 桀w 珉x 棂x ্y 愕y 鹳z 姹z 噼{ נ{ ע| ⋯| ض} 跖} 谥~ ❉~ 髅 罘 扪 Í Ł ฿ 蜥 獗 箐 芈 垚 シ 萼 衩 隼 ♦ 汩 鲟 糅 – 馗 赟 匍 璜 搡 澍 蚩 ✘ 력 颉 ▐ ฺ 까 掳 祜 감 仞 斡 犇 赉 당 滟 崴 려 ű 蛆 硚 镭 麾 건 ） ד Α 鄄 ۆ ษ 缱 偻 匐 嘞 嘚 邝 ﹞ ﹝ 咿 ✕ 않 桧 （ 때 潸 毐 ʷ 呲 碚 ื 绻 馀 惇 鎏 😉 婊 痨 臊 ¨ 飨 铰 ド ཉ 沏 崮 炁 骷 牍 饯 夔 ث 蒌 단 犍 鳃 ღ 우 恻 刍 颞 査 狰 扃 여 역 ば 铀 觞 勠 ⑶ 俸 
 Ż 胄 魇 ๔ 麋 谒 ཏ 濯 晌 결 È 骰 민 芃 肓 梆 瓒¼ ￣¼ 瞠ü 泷ü ☉ļ 睽ļ ๊ż टż έƼ 潴Ƽ コǼ 槎Ǽ 洵ȼ 迤ȼ 葭ɼ Εɼ 锆ʼ 鲶ʼ 嗟˼ ে˼ 문̼ ‧̼ 荪ͼ 绦ͼ 난μ 缰μ 筚ϼ 攫ϼ 莠м 昶м 秭Ѽ ཅѼ ÑҼ 赳Ҽ 钤Ӽ 唢Ӽ 디Լ 瞋Լ 萋ռ 簌ռ 워ּ 飗ּ 囿׼ 醮׼ 孳ؼ शؼ ถټ ᶠټ 砒ڼ 郢ڼ カۼ 耆ۼ ཤܼ ♥ܼ 鳜ݼ 骐ݼ ･޼ 받޼ 谑߼ إ߼ 毽 フ 키 ή ⋅ 생 佝 撵 饷 ■ 炝 呸 화 稔 ๆ 彧 圪 ▨ 砭 惴 던 壅 疥 啮 殒 늘 ▸ 腘 ▾ ཟ 舫 菟 忒 錾 稹 필 ি 蓥 Σ 犸 ๅ プ ם ฒ ᵐ 蒡 楣 跄 岘 ệ 牒 았 절 름 囡 霎 铄 ジ ̀ ʌ 으 レ 堃 炔 蝈 蘖 膑 ผ 缲 망 蜊 篑 瑄 薹 寳 ∨ █ ろ 螟 谄 ♀ 畦 づ 喙 진 萜 ❺ 줄 楫 ̈ ฑ χ 악 舸 苜 淦 蚱 Ј ⇩ 嗲 ė 峄 湮 험 ❦ ๙ ᵖ 敝 ᵍ 桁 蓼 メ 럼 衮 缄 참 郜 暹 ّ 렇 マ 讫 獒 椹 ∞ 麂 氩 정 隰 觊 镓! ฟ! ै" ᠥ" 冇# 纰# ๓$ 踉$ 성% 쓰% 嵇& 힘& ⼀' 揖' 桎( ฐ( ำ) 坭) 렸* 潋* 澹+ 癞+ ㅤ, ন, ۜ- 岀- 炷. ذ. 绀/ ๒/ 弑0 玷0 梏1 趸1 驸2 靥2 긴3 抡3 螯4 酞4 熵5 羸5 놓6 汊6 홍7 剌7 苕8 鞅8 분9 栎9 ა: 蓿: 煊; 심; 엄< 왜< 篁= 胪= 蛭> 赅> 蛔? 쳐? 诰@ 觎@ 铉A 诘A חB 旸B ¯C 彗C ـD 砝D 钎E 噌E ▬F ¤F ĐG 깊G ̶H 榷H 抻I 臾I 냥J 耷J 庾K ๗K 疖L 幔L 钒M 锺M 骜N 되N 鹈O 濞O 桅P 虢P ๘Q चQ 杈R 滓R 赝S 덧S 뿌T 荻T ぶU テU 琮V 颔V 颢W 浉W 胗X 锉X 恽Y 淙Y 仃Z 垸Z 畿[ 국[ ﹐\ ゃ\ 鹕] 宥] ฬ^ 殓^ ¸_ 籼_ 甯` ᠸ` 吽a 窠a 铨b 부b 장c იc 苄d 壃d 镊e 嶙e 琬f 舷f 璎g 佯g 怆h 诤h 躏i ּi Οj 桠j 郗k 雎k 锴l 燮l 谀m 煅m 鼬n 嘣n 皲o ƒo ₂p 莪p 蝼q 嘢q 垅r 명r ʒs 俚s 堇t 咗t 峋u כu ˜v 説v Œw 〜w 孑x 悌x 蹂y 鲳y פz ｡z 咁{ 葳{ ❂| 洙| 腠} 佶} 聽~ 啜~ 蓖 掴 ィ ྤ ､ ฎ ศ 蕤 ˌ ญ Й 啖 祇 鬃 眦 囔 據 崀 ⇒ 溟 倏 裟 ∙ 신 蛏 溆 ế 殚 祚 鲑 ɒ 犟 羧 酊 舛 踱 ❥ 蜃 ฉ 垟 颡 Ι ն ŋ 龌 ק Ş 黟 崧 ひ 峤 蜴 袈 ء 더 복 沤 颛 庖 伢 洄 佃 蹙 अ 傈 咂 谗 バ ᶜ グ 磬 讣 ₤ 聩 भ サ オ 唁 Ξ ⁄ ʿ Π 俾 洺 绾 撅 ウ 孱 伉 습 ⑷ ∴ ए ָ 哐 襁 釆 钺 ক 龊 跬 ナ ▼ 玟 蜍 🏼 轭 哌 墉 気 搽 ड 僳 ू 钿 岢 キ 矸 芘 蒹 谲 亓 靛 懑 橼 ব 醪 ө 宓 卅 ‡ ム デ 钏 箓½ आ½ 호ý ☆ý 衄Ľ 囵Ľ 捌Ž 疔Ž 鲛ƽ 燚ƽ 谶ǽ 물ǽ ᵇȽ 锶Ƚ 渌ɽ 镛ɽ 오ʽ 翦ʽ 恸˽ 鳢˽ ซ̽ 滏̽ 锷ͽ 娉ͽ 笏ν 蹑ν 钡Ͻ 続Ͻ 豺н ﹊н 泫ѽ 鹫ѽ ☽ҽ 髁ҽ 刽ӽ 莜ӽ ΤԽ ‏Խ ニս ฝս 芨ֽ 跹ֽ ť׽ 杲׽ 酩ؽ 浈ؽ 铧ٽ 瞑ٽ 蚧ڽ րڽ 戬۽ 褓۽ 涠ܽ 碴ܽ 菀ݽ ύݽ 锟޽ ゅ޽ 锗߽ 橹߽ 咻 獐 痍 氦 滂 ∇ 晞 钯 중 遑 盥 ﹣ 旮 缦 鼹 獾 橇 韫 ạ 凫 萘 囯 骧 馊 蛐 箬 湫 缢 ๋ 각 璠 驷 질 倜 히 嚯 鹮 ᵛ ๏ ☏ 학 鳙 ְ ♮ 琚 교 昉  ✧ 繇 馐 垆 翀 ├ ி 淝 적 ḥ ơ 篦 ➣ ⁠ ₴ Κ 捱 铖 镣 ブ 饬 チ 구 喺 増 柩 椁 ❁ ฆ 파 ⠀ 篪 傥 虬 ⼈ ビ 공 Λ 달 ➥ 锑 瀍   Ⅴ 雒 트 囫 க 傩 昝 牯 绔 迳 骈 磴 辋 岿 殁 ত 铡 ↩ 獠 舂 ס 啲 隊 垩 叵 阕 绡 蹇 葺 椤 穰 芫 砣 邨 Æ 蒗 辇 ふ ן 珥! ❻! 硌" 椽" 谔# 잘# 倬$ 謇$ Є% 語% 劭& ฤ& ∆' ̣' ध( ո( ▪) 旯) 囹* 觥* 唵+ パ+ 菔, 箘, Μ- 锒- 镗. 俎. 枥/ ➧/ 瞟0 郄0 같1 圄1 楝2 열2 ♂3 げ3 篙4  4 哚5 悻5 咣6 氡6 貉7 粲7 ‚8 笕8 ်9 😀9 庠: 沚: 瘿; 觐; 동< ˙< 褴= 준= 褛> 黜> ả? 疳? 吲@ 徳@ 槲A ュA ₳B 세B 鹬C ྩC 儆D 鲷D 龅E 埕E 僮F ྡF 변G 揩G 峁H びH 蚬I ுI खJ 忾J 埗K エK 蘅L 谡L 弭M ํM 垓N 舐N ☯O 逶O 蓁P ִP 齁Q ミQ 玳R 馔R 용S ˆS آT 嵴T 못U 哏U ๕V ַV ეW ⭐W 忤X 缬X ャY தY ŘZ 骺Z ∕[ ➝[ 牠\ 栟\ 溱] 郾] 鬆^ 逯^ স_ ∼_ 矍` 鬣` 嘬a 怛a 린b 斫b 榉c 擤c 碛d ლd 抟e 篾e 茔f षf 겠g ṣg 囗h 槁h 饧i 珣i 矽j ྗj טk 炆k Ōl 犄l 艹m 衿m 霰n 祗n মo ⑪o 유p ộp 摈q 론q 硐r 関r 鄯s थs 泺t 埝t 姒u 曈u 棹v 涔v ɛw 锃w 顼x 관x 邰y 喏y 跶z ‪z ۫{ ✔{ 聒| 讧| 螭} 蒺} ƒ~ 竽~ 倌 প 孒 ᴵ 噙 ল 莳 滹 踽 👼 甑 悭 극 唎 阗 孀 ノ 苫 べ 碓 颏 燧 嵬 າ 삶 빠 ฮ 행  啭 嬅 瑁 仵 肼 铋 擢 鸮 妫 蒯 轱 砟 鹞 ◁ ⼤ 瓠 楸 ཛ 绺 郅 œ ྣ 鞑 입 ե 髯 僖 치 欹 玦 醌 ❼ ぷ 견 鳊 妣 湎 哞 艽 鲮 瑗 闳 ώ 対 恹 ˚ 邙 塍 崑 枋 ג ⑸ 垡 茆 ᠂ 胝 Η ♢ 発 嬗 锨 🛒 ÿ 鼋 怵 방 糹 肄 猇 ▧ ▶ 趵 ダ 廪 稗 ✚ ʼ 咘 甪 ✰ 镬 욱 ョ 터 इ 噁 轸 佞 钕 魑 泔 ﾉ 唳 怏 忪 ố 悱 沩 暝 绉 영 誊 疠 檩 ण 笄¾ 旃¾ 谵þ 妪þ 鸬ľ 锱ľ 끌ž 埼ž ⽣ƾ 円ƾ ツǾ 鸸Ǿ ﹒Ⱦ 蒽Ⱦ 苎ɾ 刎ɾ ນʾ 恚ʾ 냈˾ ภ˾ 荽̾ 沆̾ 薤; 咲; 穑ξ 挈ξ 剜Ͼ ◡Ͼ ﹕о ≧о மѾ 睨Ѿ 乜Ҿ 벗Ҿ 菽Ӿ 낼Ӿ ɜԾ 俟Ծ 疴վ 蜉վ 바־ Ν־ 枧׾ 砗׾ 埙ؾ 癔ؾ 조پ þپ 娌ھ ‘ھ 妯۾ 劼۾ 硖ܾ 禛ܾ 瑷ݾ 琯ݾ └޾ ්޾ 阊߾ ს߾ ➜ 뎌 杼 鳏 음 囟 ﾟ 鸩 圻 県 । 爨 ի 醅 업 沔 魉 운 ṇ 瘕 珪 ∩ Ÿ 卍 ❽ 莼 fl ↳ 噤 哙 璘 鸨 醯 セ რ 狒 苈 掮 雩 骀 년 缛 ⑫ 爰 绌 ヽ 恁   ‒ 逄 叻 浍 袪 厩 呓 鲩 勖 嵋 ガ Ђ फ 樯 צ 垵 식 闇 鸷 頔 ⇌ 唷 黠 ズ 洧 喑 肟 말 ྷ 缶 ா 诨 铱 阄 😂 ➹ 墀 堌 嵛 礽 苌 磲 ྨ ờ ﹋ 镝 阇 鲵 粄 钹 侪 榧 豸 窣 嚟 挲 唛 砻 嫘 禳 ζ 捭 糁 剡 ً ớ 磻 鹘 怙 趺 脒 忖 訾 葶 ネ 佻 ➨ 毖 衲 ྙ 鹋 黉 〃! ➔! 碜" 茕" 苻# 変# 계$ 焜$ ւ% 旳% 垴& 瘰& 鲠' 腚' 魍( 檗( 亾) ᵏ) 铳* Ā* 塆+ 髀+ 합, ❾, 欸- ப- 裾. 犽. 堀/ 蹰/ 愠0 骠0 ཇ1 ◕1 孬2 菓2 诒3 鹚3 嬢4 揠4 蠹5 闼5 ہ6 卟6  7 圜7 洹8 愆8 髙9 槭9 담: ﹥: 嫚; 垱; 선< 荇< 実= 睚= ฃ> 屐> Ș? 蕞? 媪@ ṭ@ 蹩A ェA 仫B 愍B ظC ịC 逦D නD 仡E 蛄E 개F 疬F ිG 鳟G 掼H 颀H 邾I 箧I 蝽J ấJ 琇K 弁K 저L 杓L 氿M 簋M 菘N যN זO 畹O 窸P 실P 怿Q 람Q 鋆R 喟R 龇S 睥S ͡T 殸T 蕖U ΡU 碶V 俣V 瑀W ÞW 氚X 孖X 毎Y ░Y 俶Z 噫Z 柰[ 晩[ 燔\ း\ ̄] 湜] দ^ 饸^ 🔥_ 薮_ 塱` 踟` 체a ξa 偲b 桷b 骘c 玙c 箪d 豢d 蛉e 缫e ぐf 曌f 鹇g ベg 회h ❍h 戕i 晷i 檄j ֹj 婳k 诳k 육l 铍l 荜m उm 蕈n ரn ✤o 洎o 翕p 欤p 狍q 贽q 哝r 笪r 诮s 刈s 椴t 蝣t 양u 诂u 凼v 🚀v 笳w 钽w 酢x 匏x 狲y 蹼y 嗉z ✖z 辎{ 鬻{ 唻| 楽| 秾} ∀} 遽~ モ~ 嚧 庑 簟 ワ 蜢 ო ầ 掤 畀 钲 埚 楮 ポ 叡 霈 煴 ∥ 怄 黢 桕 鹗 ু 庹 寤 茑 蚶 ई ✬ 〞 ざ 徂 카 ď 氽 潆 ⑬ 塚 滠 挹 蕅 蟥 데 ව ㎝ 覅 ピ 隈 樨 ़ ് 辔 砦 瀣 鲎 ཙ ᴇ 饹 ⁣ 啷 弢 🤔 鳔 강 広 ট ≡ 蕻 嫪 磦 ᴗ 碁 奁 ֶ 処 ậ 蚴 嗫 Γ ▲ 轫 僭 愎 ♭ 粝 漈 隗 媾 韪 闿 戦 髡 菰 Ḥ ケ 苴 🧧 ے 盃 跏 谂 娈 铩 ọ ာ 艿 ∵ 撺 刿 虻 馓 ฯ 漷 鲱 ⽤ 臬 胴 俦 範 洌 ✅ 🇺 예 ề শ ྔ 蛱 铌 氐 巯 통 ⇧ ハ 缟 Ⅵ ¿ 侩¿ 栢ÿ 経ÿ 笤Ŀ 乂Ŀ 胼ſ 跗ſ 없ƿ ۦƿ ❏ǿ 俅ǿ 绗ȿ ⒈ȿ 洇ɿ 戗ɿ 昴ʿ ⇓ʿ 嶷˿ 岜˿ 蝮̿ 焐̿ 📅Ϳ 枰Ϳ 闩ο 鸱ο 豕Ͽ 綫Ͽ 诜п 箅п 啻ѿ 揄ѿ 纡ҿ ⇋ҿ ̃ӿ ཨӿ ؤԿ 巿Կ 黒տ 呒տ 抔ֿ 揶ֿ  ׿ ฦ׿ 玢ؿ 鲀ؿ 疍ٿ ⑭ٿ 氘ڿ 骝ڿ 佥ۿ 铯ۿ 栊ܿ 🙏ܿ 牖ݿ 谘ݿ 栌޿ ⒉޿ ✩߿ 厾߿ 禤 ⑹ ட 缂 榀 皤 ァ 鲭 秕 徭 ︕ 淠 逋 迨 睺 學 煳 嫱 笞 瘆 崤 玏 🇸 ⊕ 樘 狎 钇 ≦ 작 诌 驽 洸 蘼 镧 黩 Ô 翥 込 夼 捯 镫 鼐 湉 涑 陞 氵 ʏ ủ ✨ 蛴 ţ 勰 噘 տ 凊 வ ᵀ 垧 후 様 竑 Ê 榘 🏜 😊 砜 锔 瑭 嗞 浥 ♯ ❒ ⋆ ᴀ 殄 며 贠 된 喾 尻 烜 飏 泅 鲥 鲧 蓣 玠 좋 🐣 咜 ⃝ 배 Υ 铙 応 ֵ 蛲 월 յ 簕 蛸 ㄑ  ⾏ ể 铟 매 々 등 炘 მ 嫒 🧹 ᠱ 坼 ⁡ ザ 伥 汜 缯 孛 侘 俳 苾 萁 鬲 ⁰ ʔ Щ ක 缑! 哕! 诓" 羰" ༢# 楦# ☼$ ⽅$ 骢% ⊥% 鹍& 谞& 埸' 茚' 靼( 馑( 鮰) 岽) 岬* 籀* 镔+ ້+ ல, 薷, 鸶- 錤- 래. 嵯. 枨/ 莨/ 龉0 औ0 ∈1 ய1 卮2 琤2 ợ3 厍3 ˇ4 緺4 ♧5 ➱5 讦6 ồ6 窢7 魃7 랑8 ホ8 姮9 糸9 逑: 焓: 酆; ₱; 衽< ソ< 翚= մ= ▵> ↂ> 啁? ׁ? 鹩@ ⁢@ 발A ုA 氙B 圉B 〝C 妁C 镠D ☻D ボE 재E ✭F 缁F 嗰G 嫄G 埌H 墘H ΧI 邴I  J 镒J ྦK 箜K ʾL 罍L ❐M 豳M 鸫N 聃N ⑮O 赧O 荦P 襦P 嗬Q 纥Q 襞R 浯R 筼S ✫S 侑T 椟T ნU 铑U 격V 鳐V 齑W ➽W ⾼X 🤣X 圹Y 佾Y ່Z 忉Z 褔[ 龃[ 荑\ জ\ 铷] 衞] կ^ ễ^ 恂_ ґ_ ぼ` 산` 阏a 竦a ║b 卣b 钬c 恵c 鲇d 嬿d ㎜e 翮e 熘f 坩f ◄g 현g 売h 桴h 鸲i 搵i Ⅹj 觚j 酽k ɞk 録l ゴl 篌m 聍m ბn 프n 丼o 蘧o ffp 郇p 廆q 恫q 琎r 磡r 芣s 嫲s ກt ཧt 堙u 玎u 籺v 镦v 함w 漖w 硷x ̆x යy 氹y ⼒z 國z 駅{ ো{ ⼦| 髫| 설} ◍} 㕛~ 笹~ 🤩 🧡 佷 踯 呔 ➊ မ গ 珵 ன 垕 葑 澴 蒴 筮 旰 筌 帔 铊 ச 揿  ौ ペ 桫 艄 알 囝 ⾯ 瘗 타 牝 ʜ 蹁 椋 韡 ⚑ 揆 酹 伝 慜 ງ 趄 龢 筜 饫 👏 ్ ╹ 怍 ම 時 趔 ေ 钭 ᵕ 萆 ᠃ 髃 Õ ➛ 糌 螵 ვ 沄 ▁- ▁= ="@S ttps ▁+ ▁-- fic  ▁diffic! ularly) ▁ful0  ▁charg2 tty4 --B ->F ==O urningW  ividuals]  ▁experiq aith~ ▁ww  ▁appe ▁=>( ▁gh؏ rowd abase ▁==0  ▁spok(  ▁lawy arshЧ ionals ortersЭ  aturingد ▁我P ▁在h  ▁brows@ secut ommodX  istration razil ylv  ▁cerem` ▁一 ==== covery0  ferences iamond  asingly vely  ▁licens  repreneursx  ▁affil8 razy iscons  ▁brill\ ytics ▁(+ OVp ++P ▁/**  ▁Independ  entures =' ▁你( uanat  ҈҉҉҉҈҉҉҉4 ▁大$ ▁iP$  issioner  ҈҉҉҈ ijuana ▁///  ▁如何l ▁第 klah  repreneurD ▁三D  klahoma4 ▁BMX xic4! ▁有! obody"  ighters@# TML# raigx% embers ' ▁->* pring *  ▁bact0, ▁小d, veland/ ellows/  ionally2 ▁二T2  umental2 ▁新2  ▁unfort3 ugh|3  ▁Unfort5 iffer6 CAA6 atured6  roversial 7 attan7  ▁Pitts8  ▁prosecut(8 ▁这8 isd9  secutive4:  ▁中国< hoods= ▁wisx?  ▁如果lA ▁TesA ▁中C  ========G  ▁uncom ▁他B ++)  ▁sher ▁<-  ▁关于 ▁金 auses entary¬ ▁当Ь ▁月l uant тор alter( +,t ickrЯ wagen  ▁编辑 )+^ ▁"""t  ▁因为 ▁周 ▁元 ▁日ޱ  ▁如何看待H ▁黄R  ▁repetb ▁是 ▁上  ▁这个@  issioners ▁enz  ennialsb  ▁Observ  ▁municip ▁ét  ▁ChallengR =>  ▁北京η ctoralҷ aft  ▁suspic ▁nep, enniumZ ▁爱  ▁今日R  ighting  ▁treas  ▁现在  ▁今天  ▁unchȽ uadorν ▁$('r  ▁unforget  ▁Mahar ▁Pip:  ▁对于 ▁想  assionateƿ  aceuticals  ▁NeighborX BOUTt ANC compet  ionalism: ▁杨  ▁其实  rections )=P  ▁如何评价~  ▁根据 otersF iently ▁可H céJ ▁海\ tersonh legatev ▁万 rave  ▁来源~ ilem  itioned rals  ▁上海  ▁一个 iality Academ -$V  ▁Clem inical embr@  ▁interdiscL ertingP ▁微 ▁摄 riber> ▁用 odic ▁老 ovable  essionals ▁(-d usted  portation  ▁Cerem,  ▁最后T OUND* IFF |- Applic ▁最,  ▁第四N  ▁Perman  ▁Introdu$  anthropyB  ▁Scand ▁马*  ▁目前0  ▁Chrys6  uscriptsR ▁白h  funding  ▁unsuccess afety~ orers =( ▁再j %-$  otationsB ▁ ytic* ▁FIRv  ▁Gast ▁六 ▁开@ :-V ▁文v ▁以  CongratP  ▁neckl /-  ▁redevelopD ▁多 usting  ▁surn thened  oqiongma ▁华( ▁美 ▁全 uaniaj ashtra ▁西 ▁好|  ▁embroider -\  castersP ▁要` ▁́sj ▁GRO  ▁Authent ▁ScoX ▁安z  ▁Recruit DOC  ▁sincerh ▁无j  ▁descript ▁看  ▁Beyon  embrance  ructional@  ▁dilem arend ▁UttV  ▁disgn  ▁Forg ▁请2  ▁CognZ  ▁Prosecutl ▁apr ～   ̅ ➕F ◆ 後5 ̊9 ▽  ○ ̇v ゚ ͏ 為 ̧. ়; ※R 個k ❗ ⬇ 動 □ ➡ 書  來  ◎ 無< ⚠? ╯B ✯[ 業c ₦n 體v Ì 東 開 這 ❯ 間 ☺ თ Ć 😍" 電( Ò- Β8 დ: ℗< 場K սM ╰] △h ◇l 報m ņr ╭t 與y 現 ų ಥ ᅠ ŏ Θ ʹ 見 ╮ ♠ ̥ ħ ⇐ ں  ள 過 ľ ை 👍 長 們 👉 ા  უ 麼 Ë छ$ ಡ* আ, ☀- ‟1 機G ॉM ̩O ЫQ රR 🙁X ীZ ා_ Ьa ━b Աf 術h 記i 發m 👇p 🏾q ್y ۖ ∗ ⚡ 對 資 Û হ 問 ̓ 🏽 論 🙌 Ï ◦ 🇷 ψ ♣ ך ን 經 크 품 選 內 ු ਾ ტ 說 ற ❌ Ґ  ठ  ắ 너 토 🇦 ṛ  ు" 🏆$ 및& 🚨- ↵1 포4 চ9 ත: G રP ្R ृW 🇪Z ংd ̫i 📸t 沒v վw լ  ˊ 출 ứ 🇧 ✌ 愛 ͈ 題 џ ড 💙 ា ٹ ස 👀 ి 設 馬 🎉 ந ∑ ụ 話 風 … 렴 ũ 레 ⟩ ォ 員 種 금 눈 브 😛 ర ⁩ ල 點 ̴ ই  門  ˜  ా ക ँ թ 別 ̋" 關' 龍( ṃ) ္- ἐ/ 進1 ∂4 გ7 實8 計9 Հ; ḍ< ցD ЊF դI 💪L ⟶M ☑O ിR ∫Z 結\ 김^ Ýd 😭j 車p ̲r 울t ˋu ⭕v ギx இy  գ घ 還 當 კ 義 һ ∧ 類 հ ද 속 추 🇬 灣 ĭ 號 ǫ Џ 💰 務 ⟵ 樂 💕 ት ✈ ढ 區 編 ರ ❓ এ 버 ষ ʕ 료 ხ 번 ἀ ಿ 華 ☐ 급 ප 際 ల ⚽ შ ὶ 최 ছ ヾ 師 ў ⇨ ġ  খ  ാ  😁  က ը ↔ 메 ી 네 ̳ ự 軍/ 🌟0 連4 🇮: 製A ởC 読E 🇨G ਰH ÙP ⁦Q 코T ழV ‣Y ユZ 數[ င^ រ_ ᴛ` ভd ගe 두k ̤l ̷n அq ᴏr ◊s 島w ಾx ̱z 페} Ő 른 온 ŭ 畊 線 ു ɴ న ດ ヒ 💜 ↗ ന 張 ヤ 집 ෙ ం ゲ ੀ ổ 魈 থ 検 🇳 ્ 陳 박 ✊ 록 ĕ క 館 ქ 법 ĩ 날 ✉ ❄ 🖤 ۱ 沢 형 😎 ိ 應 運 😦 頭 ண ữ ┻ 🎥 💥 Ծ 網 천 ේ 임  직  또  ῦ  ◯ 표 랜 차 먹 ὸ Ψ  ք! Ī$ ʟ& ပ( 評+ ղ- န. 樣/ ぜ1 Ț5 ļ9 験; හ< ḵ= 본A 價B 習C অE ☰F 📌H 📷I 氣L 🔴N ṢO 능Q ັR 피V တW ⟨X 목Z 🥰[ ʁ\ է] 產^ 請_ 遊` 🤗a ѝb ☕c ʀj ❣n 將s 則v 루x አy ┬z 專 ˉ 黃 🇹 Ź Ї ര ር 統 ʚ պ 組 頁 불 님 💖 ┐ 석 🎧 Ն ನ  ᄇ 길 ओ ວ 強 Ē 議 қ ۰ 転 ぞ 構 ừ 並 蝰 醫 따 ਸ 創 ̵ ᴘ 豨 達 점 ે ም ╌ 증 ન త ⸝ 글 색 ∪ 圖 🎄 ╥ 태 യ የ ក  ̨ ০ ெ ὐ 聞 ອ 専 認 ڈ" ধ# ට$ ⚭% 單' 錢+ ത- 📺0 Ğ1 ֲ2 ⇑3 🇵: ᷅; ⌘< ◀= សD 値E 🌈G 從I 권L ԁM ửN ぬO ತS ‿W ☠X 約\ 티] ռ^ ន_ ☎` 畑b 편d ওe 簡i 葉j ŵk Ăo 試p ேq ᄏr មs ㄧt 吋u Ѕ{ 〒} 청 ᷄ 평 🇱 Ľ 質 종 標 決 🎶 į ৃ ⦁ 胬 ণ 灬 ˆ ژ ᴅ 覧 💛 ̮ ˂ વ ມ 镲 雲 😱 ો ະ 済 걸 외 친 ક ስ ➞ 貘 傳 裡 환 ә ລ ີ ẓ 規 확 🤷 झ ು 導 🌎 ɾ ফ 視 ട 萬 օ ẹ ⇄ 巉 狷 薨 볼 🙄 բ ເ 細 ̑ ̯ ┌ 識 머 활 Ṭ 碇 ɐ ▫ 園 曩 ২ స 宮  께  플  ც ˃ Ё ῶ ₩ 僰 በ ◾ 藝 წ 술  향! వ# ◬$ 兒% 兩& ץ) ぽ* 녀- 😘. 費2 陽3 테4 👑5 ヴ6 উ: እ; ゼ< 摺= મ? ይ@ 새B ∅C ▴D 耵F ለI መJ ῖK 🌹L 👌P ŷQ ٩R ۳S ☘T ̿[ ል\ 讓] 백^ ြ_ 瀬a 茛c 약e ஙf ಕg ရh 総j 👋k ංl ṅm ⁎n 価o 係p 箨s ‰t პu ゙v 였y 특z 런 Մ ບ ສ ▻ „ ው ⓿ 給 調 顔 Љ 桜 課 變 ̛ ત മ 営 稃 級 豊 載 🇭 Ե ծ ১ 脩 裝 谝 Ħ 巻 終 菈 언 ➖ ヘ 嗎 夢 戡 猹 💚 Έ ヨ 環 確 ɹ ף ਿ ം 蠲 護 💔 聲 ؟ ಗ စ 訊  ջ ặ 矬 Ġ మ ິ 団 團 帰 朊 柾 💯 ̪ ವ ී ῆ 離 ▿ 椪 湔 許 순 ० 観 병 ༄ ☝  竜  節  聯  검  ↯ ➙ 優 초 ⬆ 劬 畫 蜩 觇 詞 😳 猢 瘊 買  류# Ъ$ ప% ୍) ோ* 萩, 觀- 🥇0 പ1 근3 📚4 ›5 ׳6 ۶7 険9 놀: ਜ; 溽< വ? 栱@ 處A 轉B 군C 애D 戢E 暐F 遠H J २K ༡L បM 倨N 紅R 항S உT ಠU ➯V 薜W ΖX ැY ☹Z 曷[ 難] 🏿^ ფ_ ‵` 꾸c 립d շe ▀f 単g 喰h 🇲i 😄j ୧k ზm ◔o 彘q 湲r ٠s గt 寶v 歳w y z ɵ{ ϊ| ົ} 悪~ 螈 邶 锜 鲣 싶 📝 ۞ 蛩 袢 覺 투 🇫 ੇ ┄ 親 醣 鉄 족 퍼 သ ♚ 肫 ۉ 镞 느 ̘ Ո ೆ ỹ ⇢ ∋ 嚩 柽 澶 牂 絵 罅 項 즈 효   帙 블 판 亞 潟 磙 笫 薙 韻 働 咭 菪 🥳 ਨ ದ ያ ‫ 凪 蓠 베 손 ٍ ທ ❝ 蠓 覚 髹 ̍ ె 廣 秫 艏 ǀ ۲ ਲ 漢 訳 ∎ 禊 總 움 󠁧 խ ܐ එ ና 備 廋 絶 션 얼  牁 纮 較 酃 ੭ સ ✮ 彡 毘 鈥 鲽 ̗  ༅  ỏ  ἰ  収  聖 讵 ብ ተ ὰ ὴ ⁍ ∽ ✲ 亻 極 準 鏊 響 ല ằ Ộ  ╦! ╬" ❚# ⻓$ 堺% 浞& 猊' 羅( 該) 🗣* ❈+ 岞, 彳- 橐. 钫0 镚1 😆2 အ3 ‛4 態5 暻6 栲7 瑠8 缗9 讀: 镨< 널= 😃> య? ማ@ ➋A 屺B 證C 뉴D 희E 😅F ՐG ொH ᐳI 歷K 講M 커O ঠP 杪R 満S 筆T ऑU ẩV ✟W 図X 葙Y 🦋Z ̝[ ↪\ ⚪] 劏^ 労_ 瞽a 骓b १c ≫e ❛f 週h 鲂i 막j ↘k 乩l 卲m 癃n 碲o 缵p 🎙q ંr ᄌs 奭t 躅v 鹪w 🤯x ‹y šz ͬ{ ⊂| ╩} ▓~ 乗 撃 蔹 螬 鹁 련 클  🔗 ǚ ነ ぱ 玧 蓊 铈 鳎 ќ ຫ 換 槠 淯 熱 钌 飛 飧 🎬 👊 Ū ᴍ ⲟ 丿 弶 復 橦 權 殍 瓘 圏 暍 條 溫 溲 痦 蔵 辏 鲞 麴 광 📱 😢 ٰ ੰ ା ಲ ゝ 嚅 歡 藠 雠 창 ˶ ➌ 幾 権 歴 産 紀 螽 銀 🏈 ͒ අ 屾 帶 畋 磔 禮 蕹 贶 閲 ྀ 療 镡 란 Կ ુ ត 嗐 姫 戰 淏 購 鼩 🏴 💗 Ά ״ જ බ ໍ 傕 槊 氖 頼 လ ᠧ ∘ ⌒ 劢 協 宀 嶋 掾 铼 չ પ ಭ ‸ ┴ 紘 討 陸 ╱ ❆ 儇  礐  鲻  밀  ः  ⏰ ┼ ✆ 哂 箦 郞 패 ԅ ట ▒ 測 爾 筇 罨 舣 镏 송  🔵! ѡ" ṯ# ↠$ 夥% 媞& 嶝' 熳( 镟) 올* ɦ+ ਕ, ର- ➪. 畤/ 鲡0 鸹1 Ս2 圮3 瓞4 蠊6 髈7 鲖8 택9 ༠: 勝; 撄< 焼= 猞> 薢? 赀@ 鲔A 鳇B 🇰C ɨD ਤE ളF ╝G 亶H 橋I 臺J 螫K 谖L 適M 🇩N ̂O ԛP 屃Q 抜R 揾S 樗T 锛U V ऐW ងX ẽY ⁃Z 儁[ 維\ 缃] 蝥^ 阝_ 馃` 축a 💬b 妗c 瑢d 罴e 赜f 鲺g 🤝h 🤦i ‡j ަk ◞l 両m 剞n 卺o 姘p 狀q 🎂r ̌s ͚t ḳu ╗v 嘁w 峇x 滗y 玘z 蘇{ 鬘| 鲐} 씨~ ғ ՞ ྫ ᠽ 濬 爿 衆 逡 陔 책 Œ Ȣ ۹ ᴄ 倮 媜 猁 臌 舾 蚵 辺 钪 ̸ 効 栳 蘭 诹 錄 림 🌍 ެ ਪ ျ 帯 曛 泃 溘 眀 篼 興 菉 蒟 辂 領 접  ͙ எ ွ ⚔ 媺 屙 弇 愔 橛 甴 砬 郤  💋 ຍ ỳ Ἀ 佉 択 毳 돌 ̠ િ ି ძ ᴜ ☁ ❮ 嗮 氇 淸 狻 薬 鸰 👩 ञ ৭ ⇆ 墫 瘥 黼 펌 혼 ֽ ီ ╔ 寫 潲 玕 硇 翃 荭 🎵 ீ ಪ ✄ 咥 棰 珰 異 翾 苘 鹀 ͜ ▔ 呖 尓 湴 蝓 骖 🧐 ో ො ຶ 偬 烀 舖 蠖 跸 맞 북 슬 Ի զ ਦ  ੁ  റ  ិ  ẻ  ⎮ ┃ ➲ 倥 児 咝 垲 況 洣 職 蝾 驗 鮀  😬 ద ᐈ ∃ 悫  癒! 眭" 粜# 葜$ 褊% 詩& 阋' ঙ( લ) ግ* ỗ+ 昳, 檫- 珽. 謂0 辻1 钋2 ▢3 仮4 坵5 渟6 滿7 莶8 蒻9 裢: 觌; 談< 願= 센? ʺ@ ਹA ేB ≖C ⋮D ☂E 咑F 毵G 澥H 赑I 锞J 鼯K 침L 🏀M 💻N 🥛O ͓P ਮQ ಸR ണS സT ឴U 亍V 堠W 幵X 擇Y 樹Z 洈[ 硏\ 祓] 耧^ 菝_ 蜮` 鄣a 鹡b 십c Գd ׂe ۅf 戸g 旼h 栝i 茈k 钆l 😡m 󠁿n Ţo ǝp ߋq ગr ㄨs 扌t 柝u 烔v 稱w 韓y ꦿz 높{ 🌬| 👨} 󠁢~ ċ ɣ ⇔ 搦 榖 碼 縻 衛 雫 預 💫 ɢ ̞ ಮ ព ╚ 墕 栃 祼 铫 麈 龠 🌿 💡 ᵎ ▌ ✗ ㄒ 岈 柢 瑧 祆 绂 謝 辚 鄚 곳 🌸 🥂 ݂ ឵ ὁ 剣 铪 铽 镱 餍 鼍 완 ৰ ୦ ஸ ಯ െ ὲ 吿 紹 蚍 財 賨 鞞 ɯ ؛ ூ Ḵ ∮ ▄ ▩ 亊 暌 涙 焮 蛞 髪 혜 🎁 ۔ ພ ྭ ች ₵ ⇠ ⚫ 劇 啐 暲 殳 積 臑 蟊 詳 誰 鹟 ՝ ܬ 勍 參 愦 扞 攽 瀹 瑨 缣 芰 荩 铗 鲚 누 취 팅 ৩ ৳ ແ ့ 夲 屦 崟 杌 滉 眇 硎 铒 锖 鞥 魚 📰 🔊 🗓 ͑ ପ 佺 吳 従 琹 稲  穂  裏  誌  钐  츠  🇿 ͊ ־ ✏ ➭ 乛 埒 揸 梶 祊 笥 笮 菹 蔸 蚺 順 鲊  麵! 갈" Ə# ٫$ ទ% ✐& 塅' 媵( 嵖) 彖* 暾+ 泐, 渼- 瓿. 畚/ 練0 蓍1 诔2 逖3 郿4 📖5 🤘6 ీ7 偓8 哿9 屍: 岵; 旄< 旒= 栄> 汭? 瘅@ 眞A 脔B 觏C 鹓D 될E 탄F ɽG ခH ṁI 仉J 哓K 啓L 戆M 殠N 氅O 續P 註Q 隠R 鲦S 허T U 🌧V ԸW ٌX ڏY ങZ ◂[ ➀\ 絜] 胨^ 邊_ 養` 鲼a 찾b 🏅c ̽d ͆e Ѓf ਬg ỉh 垤i 拡j 晡k 桲l 歃m 湓n 盤o 隨p ꒦q ̉r ުs ಂt ደu ᴡv ḩw ⛪x ➚y 偾z 埖{ 忸| 翛} 蜣~ 踔 颙 빈 🔒 🦄 Ư ̚ ͕ ល 姌 娭 帑 怩 攞 玹 莸 蒄 郷 酾 镢 끝  ǜ 堽 庥 戲 拏 敔 樓 檎 毑 璩 筲 辶 镘 隳 鞯 鳉 鹎 링 잡 ۴ ਊ ᥙ ┘ ✱ 亜 呪 忝 悒 拊 滙 貴 贇 😜 ຈ ∷ 啱 尋 朓 獬 肸 褡 褰 陬 雊 鱻 암  ͔ 咾 垾 孓 層 崞 昃 槑 笊 臨 酂 钍 須 顯  ˵ ਗ ય ራ ἄ 伛 囲 継 颃 鲹 💃 ɕ ͋ ✡ ゞ 䨻 巘 庫 浲 烝 簰 籴 荛 菿 谠 谰 閞 雑 飑  💀 ģ డ ༎ 伱 倓 圬 尅 慾 摽 沵 瀞 炤 獣 窓 缋 遄 酡 锕 黧 黾 철 🌱  🤪  ̬  ᭄  ḏ  ➁ 呬 夬 曱 柃 蒨 蹻 鹛 黻 채  ̼ Վ ূ ẵ ₪ ≯ ⌓ 偁  坜! 嫫" 岕# 椇$ 瀼% 燠& 硗' 罾( 輪) 遅* 镪+ 黥, 밤- 윤. ķ/ ̹0 Տ1 ׾2 ۪3 ላ4 ➸5 劉6 呙7 圌8 妺9 姉: 棻; 歺< 渋= 獎> 獨? 瑫@ 钶A 骟B 鲌C 黐D 🔍E 😈F ߈G బH ປI ⊱J ♛K 塄L 嬲M 屬N 桤O 渽P 輋Q ͖R ௿S చT ఢU ᴥV ⌄W ☒X 佧Y 妳Z 桖[ 牤\ 獴] 眄^ 禩_ 荬` 螅a 鼙b ͪc Ւd ஆe ♔f 仼g 勢h 姤i 徴j 簃k 粢l 納m 脷n 辦o 锝p 阒q 겨r 져s ५t ಟu ወv ᐛw ⊰x 坌y 堎z 屣{ 揺| 旴} 柊~ 桄 檔 溦 煕 犴 甡 皞 蘩 顸 규 램 찬  📈 ͘ ٴ ባ ⁨ ⬅ 喈 堉 壸 娡 搴 枱 楊 歘 猡 籙 罟 髭 📍 😇 ǧ Қ ١ ঞ 圧 孥 帱 榃 沬 溇 澌 營 膂 讠 谳 蹀 답 ͅ ͇ ച ຄ ὑ ☚ ㄟ 喁 嗵 墁 壊 憍 掲 氷 泖 盦 眾 礞 绐 罒 耜 舻 覽 闘 鲘 봐 승  🤓 ͎ ӧ ި 刋 廛 旆 汯 湯 瘳 綮 褫 躞 鲋 웃 즐  📣 Թ ڑ ښ ৯ ᅲ 冊 卻 奓 籣 蟳 賞 铕 阌 삼 🥩 🦁 ͭ ͮ ң ਵ ሰ ἡ ⤵ 伧 埫 廯 慊 曕 棼 毉 絡 纩 鹆 응 큰 Ą ͦ ↖ ⯈ 峣 廻 拶 熺  猱  疰  矻  虺  鎓 铥 엔 착 Ɩ ऊ ገ ᴠ 執 幫 杮 歐 獲 痄 盉 蚋 補 趱 鐵  阃! 驺" 骎# 鳚$ 존% & 🅸' 💎( Ť) ɬ* Օ+ દ, ታ- ድ. ḫ/ ➩0 ㄈ1 侂2 峫3 嵝4 戶5 摅6 榇7 玚8 绠9 羙: 莔; 谿< 钚= 麗> 녕? 👤@ ĊA ਚB କC ஜD ോE ែF ⎯G ▹H ◟I 採J 棨K 檢L 殺M 淩N 澪O 燫P 茺Q 郐R 객S 염T 🌊U 🕊V ẫW ∝X ≪Y ⰙZ 丏[ 傒\ 妼] 寔^ 廠_ 惔` 摭a 浛b 減c 漭d 璊e 瞓f 硁g 碨h 笸i 舢j 艉k 蕗l 虮m 蚰n 觯o 鐘p 鲃q r s 🤞t ǂu ʎv ⌊w 傷x 凢y 屲z 幛{ 悃| 枓} 監~ 脶 艋 蟜 複 酎 鲆 龀 벌  Ž ͣ Յ ▏ 佀 弎 挼 燹 窭 笾 縄 缳 褙 轵 農 鄜 酲 靑 ꃅ 셔 왕 케 콘 튀 🐶 😏 🙃 ͂ ͐ ۣ ळ ६ ី ἱ ⊆ 僦 塩 帻 彀 棪 浄 碹 絷 臜 衪 証 谮 蹟 钅 頣 곡 🏳 ٱ ♑ 嗌 徼 織 羣 贳 踅 飖 龑 굴 몸 충  ʙ ͛ ચ ည ំ ➦ ゾ 兕 匋 厶 哜 喦 幣 斝 濟 璆 訓 讬 诼 趿 颟 됩 락 령 🎮 ˑ ̺ ٢ ۵ ް ຕ ဲ ┅ ⚾ 仂 庁 捩 洑 涢 瓐 甍 畛 縠 縣 赍 鳥 鹣 꼭 꽃 숙  🎤 😋 † ͫ ܝ ঃ ਂ យ 忄 戓 枹  洳  紙  绁  缒  缡 荘 跞 郪 鳯 👁 ុ ់ ⊿ ✶ ➂ 勹 呎 堞 墼 挙 昪 樋 滝  灋! 熜" 癍# 蚨$ 赕% 轺& 階' 驵( 돼) * 🤤+ ɺ, ӂ- ඩ. ᐢ/ ῷ0 ◙1 ☪2 ✺3 ❞4 傧5 剛6 峃7 溷8 熥9 猋: 碈; 稜< 緑= 縢> 脇? 臵@ 赙A 遘B 闶C 隹D 靺E 鲉F 鳓G 뿐H 텔I ਟJ ୨K ຂL ຊM ክN 凖O 剀P 抯Q 晜R 漶S 澂T 澤U 缍V 罣W 聴X 舴Y 苢Z 輕[ 輝\ 遹] 阍^ 雱_ 饩` 롤a 쉬b 핑c 💍d 🔸e 😷f ˍg ှh Ẓi ἔj ◌k ✝l 凃m 塁n 嶶o 惢p 挢q 翫r 芴s 荳t 蝲u 詈v 邠w 铴x 險y 鬯z 鳀{ 억| 🍁} 🎨~ Ě ̻ ͤ Ѥ Է ۷ މ ନ హ ຮ ⚖ 吔 埘 岡 崾 扆 渉 猃 癯 虒 衎 觜 鄘 鍪 캐 🙋 ͍ Ԉ ৶ ટ ୐ ୒ ౐ ಳ ೜ ഻ ൱ ඪ ඼ ረ 嗥 彈 悛 扱 烺 蒎 谇 踬 錡 🅾 🖥 🧚 ໃ ყ ♍ ぴ 吥 巎 幞 戻 瑆 璂 粧 糨 缌 軽 違 霊 鳷 麤 麺 릴 😩 ʽ ༥ ༽ ከ ᠹ ⚜ 伾 堍 夠 寛 戽 抺 栴 殂 玡 瑱 畎 碡 缧 苤 蕰 諾 譞 譯 軟 靈 飯 鮟 鱇 범 왔 Ό Պ ॥ េ ⏩ ♐ ❊ 僕 塭 屮 沕 泚 玾 珧 羮 茇 萹 虿 袆 負 혈 💞 Բ ൊ ➠ 乸 埏 婼 昺 汏 沨 洫 爇 痩 眛  祧  築  糬  舎  鮮 鯈 鹨 ꯭ 🅻 🌴 🎃 😮 🙈 Ԁ ੍ ବ ខ ច Ọ 伲 咵 喎 壆  姵! 斎" 瑒# 禚$ 窎% 葎& 訂' 賽( 颋) 饤* 駆+ 骉, 黙- 齢. 량/ 탈0 1 2 😤3 󠁳4 Ĺ5 ̖6 ̜7 ̦8 Լ9 జ: ໄ; ជ< ≌= ⎜> ❑? ⻛@ 唿A 嚒B 嫑C 榊D 漼E 瑔F 磜G 絺H 繋I 绲J 芶K 襪L 踇M 鳑N 잠O 🤡P ̐Q ̟R ණS ශT ဆU ယV ᄑW ᴖX ḨY ┆Z ┯[ ╣\ ☄] ㄏ^ 刳_ 呶` 嬰a 尒b 戋c 殛d 汖e 煣f 牿g 祾h 缐i 菮j 蒇k 蝻l 轾m 駹n 鲒o 鲯p 鹾q 🥺r ֙s ෆt ⻔u 乆v 嘏w 圯x 埴y 渏z 燿{ 犼| 猄} 珹~ 瑊 璁 璸 畢 砵 禼 舥 藥 虓 鏡 顧 鳘 龘 몰 풍   ž Ę ̡ ͉ Ћ ྂ ံ ⊗ ⌿ 億 戥 斷 珖 礡 纟 腽 蒛 蠃 郚 镤 隱 靬 鞨  🎈 އ ऋ ૅ ළ ཱ ᄂ Ḳ ➮ ➷ 俍 倉 剅 劻 吙 嚱 搋 椠 汈 牀 琲 胂 芑 螣 訪 锸 霙 앞 죽 🆂 🐍 🛏 ૦ േ ᆞ វ អ 俵 坖 埈 夐 峘 怃 晧 涴 滆 濩 瑡 緒 聱 舉 舗 葸 貨 跫 颿 髑 鳡 齮 ꇁ 웨 짜 🕵 Ń ˗ ̰ ނ અ ሚ ḡ ヶ 亀 佈 兲 夤 媄 慶 拠 暎 煐 犭 禇 綝 菂 蔌 诎 轹 鄉 鉾 铹 锼 鲏 鵟 돈 벽 컨  😔 Ŧ ७ ঁ શ ୪ ህ ℘ ⑉ ♊ 乄 嗾 廸 梠 梼 椐 歯 滈 玗 玬 甏 盍 箝 罥 蕺! 袮" 賛# 邽$ 酤% 钸& 雜' ( 🏃) Ė* ̒+ ਼, େ- ⇣. ♒/ ➫0 勲1 呉2 咴3 囖4 堋5 壇6 姞7 愽8 楯9 涘: 渶; 湑< 潩= 澔> 珷? 璈@ 瞢A 窊B 純C 纁D 翙E 苨F 菍G 螓H 跣I 鉅J 錯K 锇L 陰M 雖N 顗O 鮸P 鲕Q 鲰R 鳆S 슈T 앱U 🍿V 🧨W ĎX ʈY ˀZ ˎ[ ̢\ ̭] Դ^ ₡_ ▂` ♓a 仏b 倶c 厔d 呯e 壓f 敎g 梽h 楬i 煖j 瑅k 瘵l 盩m 矞n 礵o 绨p 舁q 莂r 藴s 蚿t 遆u 釭v 隣w 鯭x 릭y 봉z 셰{ 🇯| 🇾} 🤍~ Ů Ơ ѓ ѵ ସ ኛ ☃ ✁ 亖 侔 塝 媸 懔 挒 揲 朥 汔 滯 滻 炟 犰 禵 穉 芠 莛 蒉 蔟 裣 鉢 锾 頻 飲 魟 ꞌ 귀 랍 죠 𓆝 🌻 😐 🥑 ͝ Ў ۸ ੈ ಬ ಷ ື ჩ ₰ ↧ ⏱ ♋ 刂 匫 嗍 坬 嫋 宄 暡 杩 棎 榮 牰 獡 璿 礤 簖 脁 菸 蔴 藦 裀 誠 邅 醢 럽 폴 Ǝ ʘ ܘ ༤ ḑ ⊇ ⌣ ➍ ぁ ぃ 刬 啟 姽 尜 払 昰 梃 洀 淨 爭 犋 甽 睆 窳 艚 蟢 裥 邲 釋 銭 镃 镄 閟 雙 麇 ꒪ 떠 ︠ ︡ 📢 ऍ ජ ᐅ 丌 剸 卬 啙 垹 墹 抃 抰 捔! 捜! 暃! 狳! 疒! 篥! 籒! 苵! 訚! 責 ! 躜 ! 锘 ! 騒 ! 髣 ! 鱓! 麹! 뒤! 빛! 🔹! 🥀! ˔! ஞ! ໂ! ᠊! ゥ! 丄! 丷! 乀! 侉! 侰! 侹! 兿! 冏 ! 刖!! 厣"! 啯#! 噚$! 嶖%! 廃&! 徧'! 怣(! 惗)! 棐*! 槙+! 汫,! 狯-! 狺.! 猔/! 甗0! 篢1! 粦2! 绋3! 舲4! 螋5! 讃6! 踜7! 踺8! 輙9! 輸:! 迓;! 遺! 颎?! 骕@! 궁A! 싱B! 찰C! D! E! 🅴F! 👻G! 📧H! 🔔I! ĝJ! ߑK! ါL! ↙M! ∏N! ≮O! ♉P! 仯Q! 冁R! 埜S! 妘T! 娎U! 搤V! 栻W! 槅X! 檚Y! 櫩Z! 沋[! 洨\! 涐]! 澉^! 獍_! 珝`! 禆a! 蒬b! 蔳c! 蠍d! 譲e! 豐f! 貊g! 閱h! 陧i! 鞮j! 頂k! 鱀l! 례m! n! o! 𓆟p! 🍷q! 👎r! ȃs! ̾t! փu! ֞v! ତw! ◼x! ㄣy! 倫z! 僽{! 刅|! 厳}! 囷~! 夂! 幨! 怫! 恓! 懷! 晄! 滒! 獢! 璲! 脳! 腙! 薦! 賣! 躐! 鋐! 閣! 靜! 鰕! 齼! 龆! 💓! ੱ! ຸ! ጠ! ❅! 匚! 卐! 啨! 夨! 姶! 彵! 怤! 楢! 湸! 漹! 烎! 熚! 瓓! 疋! 絲! 胙! 苼! 褀! 酴! 鏼! 镆! 鞡! 饦! 鬈! 鷨! ! 🥤! ̔! ӣ! ժ! ࣞ! ភ! ☸! ♌! ⚘! ⻄! 倞! 偠! 匜! 塡! 岄! 崁! 嶒! 惣! 懡! 拃! 柁! 榍! 榎! 炱! 玓! 玭! 竨! 競! 葞! 覇! 赬! 跐! 踈! 躄! 躔! 辁! 鈇! 鉮! 錬! 雮! 鬂! 鸊! 黪! ꒳! 밥! 킨! ! 🐾! Š! ƪ! Һ! ձ! ਆ! ጥ! 側! 刭! 厓! 叇! 垞! 埤! 孫! 嶤! 幷! 愬! 慆! 憶! 扥! 楙! 欅! 歲! 汚! 洢! 溍! 滳! 炣! 猸! 瓛! 畠! 盡" 硪" 礎" 筻" 臁" 臓" 莮" 萈" 萣" 蒐 " 蓇 " 蕋 " 薁 " 薫 " 觋" 跘" 辿" 醁" 鈢" 飾" 鷉" 鼭" 갑" 싸" 혁" 흥" " 🏠" 🤭" Ň" ࡒ" ఻" ⇡ " ♃!" ❃"" ⲛ#" 儀$" 劀%" 勅&" 勳'" 呡(" 嘂)" 妧*" 婡+" 嫰," 崥-" 嵕." 惛/" 挧0" 攵1" 曽2" 枵3" 棇4" 槻5" 橏6" 泈7" 熣8" 睃9" 碕:" 祙;" 禘<" 筅=" 篟>" 簠?" 籏@" 粏A" 絅B" 羑C" 脃D" 荾E" 蔰F" 裌G" 誐H" 踿I" 轄J" 逨K" 鉏L" 銷M" 鋬N" 锫O" 饊P" 鷄Q" 떻R" 째S" 🧸T" ЌU" ९V" ູW" ሳX" ἸY" ⃛Z" ♈[" 僅\" 卽]" 嚐^" 妸_" 姁`" 屄a" 惪b" 慝c" 栿d" 檵e" 殪f" 浡g" 浼h" 渓i" 渖j" 湣k" 炻l" 狛m" 玊n" 玍o" 珌p" 瑸q" 癿r" 盬s" 眘t" 磉u" 粬v" 粰w" 繑x" 苂y" 蒠z" 蚻{" 蟅|" 袴}" 覌~" 訫" 諸" 謪" 躧" 辪" 锿" 馇" 駖" 騃" 髄" 鮨" 鵕" 麶" 읽" 총" 🌞" 🌶" 🏁" ҳ" હ" ಜ" ☊" ♏" ⭑" ヌ" 冫" 卌" 噐" 妐" 岺" 巣" 徥" 悝" 懐" 撖" 撙" 昫" 暠" 枘" 梿" 潏" 燯" 瑮" 秡" 稖" 繴" 缏" 荅" 萿" 蓂" 裇" 裼" 赪" 踙" 鐖" 雃" 鯿" 鳽" 龁" 쇼" 팔" " 🤫" ̙" ֩" ३" ਖ" ဖ" 俬" 匦" 呅" 奨" 嬖" 戯" 撔" 擊" 擗" 暣" 朏" 棽" 楗" 榅" 歸" 汬" 溓" 滃" 燏" 珎" 瓌" 眣" 硔" 粎" 肷" 臛" 舳" 苶" 茀" 荍" 葖" 蕡" 螠" 訞" 赆" 陁" 鵼" 鸺" 쉽" 줘" 🇴" 🍾" 😞" 😲" ĉ" ȋ" ੋ" ༼" ∉" ⌥" ▍" 劃" 匃" 嘊" 噅" 坣" 壷" 妑" 尙" 尥" 尪" 悘# 敗# 敟# 暋# 暏# 朷# 杻# 樉# 檠# 淽 # 渀 # 潾 # 烾 # 狨 # 珓# 瓬# 甃# 皛# 眬# 硿# 祹# 緊# 纛# 蓮# 诖# 貥# 踰# 迋# 鄃# 鎌# 镈# 镙# 靵 # 骩!# 麿"# 鼷## 떤$# 먼%# 믿&# 🍒'# 👆(# 🤑)# ŗ*# ǰ+# ɚ,# ଲ-# ༣.# ដ/# 冣0# 噏1# 岣2# 悩3# 捛4# 搧5# 擵6# 攌7# 朶8# 栂9# 桱:# 椀;# 椣<# 楤=# 槺># 歓?# 涥@# 溁A# 爏B# 爦C# 狴D# 獳E# 璪F# 畂G# 畳H# 簩I# 絕J# 聡K# 腦L# 萳M# 蓐N# 蝀O# 蝳P# 钔Q# 闍R# 靐S# 鞵T# 顮U# 饔V# 駡W# 驩X# 鷣Y# 黹Z# 鼜[# ꧁\# 넘]# 롯^# 쟁_# 좀`# 징a# 킹b# 혀c# 🐻d# 💨e# 🥕f# ʍg# ֫h# ঘi# ෝj# ူk# ₽l# 乫m# 俢n# 兓o# 圓p# 圙q# 墣r# 婮s# 帲t# 抏u# 昷v# 朅w# 柈x# 汧y# 浰z# 漺{# 澋|# 瀻}# 灈~# 牬# 琻# 痎# 磾# 稂# 穏# 窆# 笖# 纻# 肹# 艆# 荓# 袛# 袷# 裰# 訮# 訴# 跡# 迬# 郘# 酔# 铦# 陓# 隩# 駪# 骃# 髢# 魝# 鮈# 鲪# 鼂# 꿈# 딱# 써# 엘# 👈# ŝ# ̎# ϝ# ᐝ# ូ# ᧐# ḷ# ❱# ⲓ# 冋# 厛# 呂# 唦# 圊# 垳# 埉# 嫠# 屛# 嵚# 嶳# 幁# 彣# 怹# 搣# 搨# 撘# 撛# 撝# 攝# 斢# 晱# 椑# 槜# 泦# 琀# 眎# 耖# 葰# 蔧# 蔲# 藨# 蟓# 裧# 郱# 針# 鐅# 锩# 頴# 額# 颩# 飐# 駄# 駼# 骣# 鱄# 鵅# 鹖# ꧂# 논# 첫# # 🏝# 🥚# ɗ# ˺# ৪# ଠ# ೄ# ᾶ# ♻# ぺ# 䓬# 仩# 俛# 倁# 冮# 卭# 哳# 噋# 圞# 寲# 峎# 峠# 庅# 悺# 掅# 昐# 晙# 暦# 桯# 樫# 櫉# 渕# 濢$ 焌$ 犲$ 狁$ 珃$ 甓$ 皦$ 篗$ 繦$ 艟 $ 菑 $ 蒾 $ 藍 $ 蚖 $ 蜞$ 螗$ 誤$ 謦$ 軡$ 輶$ 迍$ 逹$ 釈$ 锓$ 陴$ 鞫$ 韘$ 馰$ 驑$ 髧$ 鰰$ 鶒$ 鸻 $ 액!$ 엇"$ 틱#$ 홈$$ %$ 📲&$ ໌'$ ᄋ($ ᐃ)$ ⍤*$ ╠+$ ⚓,$ 兦-$ 卩.$ 呑/$ 嘡0$ 嘫1$ 嘾2$ 屚3$ 崚4$ 帀5$ 廙6$ 慮7$ 扡8$ 掟9$ 擔:$ 暜;$ 楩<$ 榼=$ 橽>$ 泩?$ 濨@$ 烮A$ 珴B$ 琞C$ 瑌D$ 璏E$ 璛F$ 眵G$ 矅H$ 碙I$ 絇J$ 繼K$ 绹L$ 耱M$ 藈N$ 蝤O$ 裒P$ 覵Q$ 趯R$ 趼S$ 郕T$ 郵U$ 鄌V$ 醂W$ 醽X$ 釟Y$ 鍉Z$ 閒[$ 靰\$ 韱]$ 魆^$ 鰤_$ 鳋`$ a$ b$ 🅵c$ 🇻d$ 👶e$ 🤖f$ Ɖg$ ɲh$ ̕i$ ఒj$ ቀk$ ẖl$ ┉m$ ♎n$ 亯o$ 亹p$ 伷q$ 傑r$ 墳s$ 媽t$ 嫃u$ 峝v$ 崋w$ 崦x$ 庋y$ 庪z$ 廑{$ 廴|$ 悰}$ 掱~$ 搌$ 斉$ 暁$ 梫$ 楜$ 檃$ 氕$ 溡$ 漸$ 瀴$ 熈$ 瑉$ 甁$ 禔$ 秲$ 稤$ 糇$ 翯$ 莈$ 覄$ 覕$ 訍$ 販$ 賀$ 輁$ 迮$ 鄁$ 锊$ 镅$ 顕$ 鮇$ 득$ 듯$ 칠$ 헤$ 🎅$ 😥$ 🤙$ ȱ$ ৈ$ ખ$ ദ$ ဘ$ ᅳ$ ើ$ ☔$ ☮$ ⛔$ ⛰$ 亸$ 侁$ 冚$ 劄$ 勷$ 匨$ 卝$ 坫$ 宬$ 岍$ 岠$ 廒$ 廰$ 捽$ 掎$ 擉$ 栚$ 橚$ 煺$ 牸$ 珟$ 珨$ 璡$ 瓀$ 疎$ 禟$ 穽$ 窅$ 筞$ 縁$ 肜$ 荄$ 菵$ 蘘$ 衤$ 裛$ 觍$ 頖$ 骙$ 鴠$ 鶸$ 鸂$ 골$ 템$ 🍎$ 🍪$ 🎸$ 🐐$ 🔎$ 🗝$ ȉ$ ֖$ ४$ ૪$ ௰$ ూ$ ಹ$ ᑊ$ ♞$ ⚙$ 亼$ 仌$ 儞$ 冖$ 冧$ 垇$ 垍$ 塂$ 奤$ 奧$ 婇$ 嫜$ 寀$ 寧$ 嵙$ 巓$ 廾$ 彟$ 恲$ 慣$ 抐% 搿% 撏% 柅% 泂% 溬% 灪% 玞% 璚% 璺 % 畟 % 瘼 % 砑 % 秖 % 簼% 絬% 缊% 羼% 翵% 舘% 芄% 苅% 苠% 苲% 蕝% 蟪% 襕% 豞% 贂% 铞% 雰% 顏% 鬄 % 鮠!% 鹢"% 📽#% Ѐ$% ѐ%% җ&% ܢ'% ߜ(% ଇ)% ମ*% 䴙+% 傸,% 儫-% 冂.% 劜/% 啝0% 嗱1% 堅2% 奀3% 奡4% 寜5% 屧6% 峧7% 帳8% 惝9% 栆:% 殭;% 洚<% 淛=% 淲>% 潣?% 災@% 焆A% 珦B% 琭C% 瑺D% 礅E% 礩F% 竝G% 縮H% 胅I% 舄J% 舊K% 蚡L% 輗M% 輯N% 顒O% 馫P% 騄Q% 鲗R% 랄S% 웹T% U% V% 📊W% 😕X% ŰY% ˥Z% Ӫ[% ޑ\% ާ]% ੫^% ഏ_% ഠ`% ႔a% ሪb% ὅc% ῳd% ⦿e% ⬛f% 䜣g% 亅h% 伈i% 偦j% 呹k% 噉l% 墖m% 夿n% 姩o% 尛p% 帒q% 忭r% 慥s% 昚t% 朿u% 柷v% 檏w% 沴x% 洰y% 淜z% 湋{% 漒|% 爩}% 狃~% 獻% 琂% 琽% 瓅% 瘈% 瞤% 矧% 筊% 箷% 脰% 蔄% 蜫% 蠛% 誂% 賢% 赇% 跠% 蹐% 鈡% 锬% 饣% 鲴% 鸜% 둘% 맛% 졌% 척% 🎟% 💩% 🚫% 🤬% Ȟ% Ѱ% Ӕ% ۿ% ܻ% ࣂ% ૚% అ% ᄀ% ዝ% គ% ἶ% ┛% ❘% ➳% 㗎% 倴% 唫% 嚭% 塊% 婬% 嫇% 嵗% 庰% 廨% 忞% 懅% 戱% 挊% 擓% 晳% 殦% 洴% 涳% 漋% 濊% 璒% 瓕% 瘛% 癵% 砹% 禖% 笱% 筯% 罝% 耏% 耥% 耪% 苙% 螧% 覀% 踦% 軏% 輔% 醵% 銎% 錞% 閦% 鵙% 鼽% 齒% 쿠% 휴% % 🦆% ʂ% ˽% Ϧ% ӫ% Ԥ% ֥% ؊% ٮ% ړ% ܪ% ݃% ࣛ% ৫% ෳ% ᅵ% ṟ% ⊃% ⏤% ┈% 䘵% 䴘% 仳% 佽% 偰% 傊% 吢% 呿% 咇% 噀% 圍% 垯% 堷% 壵% 寯& 尢& 嶓& 憀& 憷& 損& 搥& 摾& 撡& 擐 & 旡 & 昒 & 昜 & 曺 & 杺& 楪& 樛& 橸& 欻& 泶& 洐& 潔& 炪& 煙& 煠& 爀& 玔& 琄& 瑑& 瑖& 璍& 璬& 瓈 & 畵!& 睱"& 竾#& 綠$& 绱%& 蔀&& 諠'& 蹯(& 釿)& 鋕*& 鏙+& 閉,& 雝-& 饀.& 騑/& 髶0& 鴜1& 鵺2& 뜻3& 빨4& 솔5& 욕6& 컬7& 컴8& 🕺9& 🚗:& ƽ;& ȶ<& ͨ=& ͽ>& Ӷ?& Ӿ@& ԔA& ֺB& ۊC& ࠯D& ࡢE& ࢑F& ࣏G& ੟H& ௛I& ఀJ& ಃK& ೐L& ⇅M& ┗N& ✳O& ⲉP& ぇQ& 佴R& 僺S& 嚴T& 堔U& 媖V& 幹W& 愵X& 攲Y& 昄Z& 柍[& 栤\& 楕]& 櫈^& 浭_& 炡`& 烏a& 牜b& 犏c& 玐d& 瑳e& 瓋f& 畯g& 痃h& 瞺i& 礓j& 祶k& 穁l& 穵m& 筢n& 紋o& 紮p& 緮q& 耔r& 芟s& 莙t& 蓢u& 虛v& 蝜w& 蟛x& 觸y& 訝z& 譛{& 赒|& 辀}& 顈~& 餘& 鱤& 鴂& ꉂ& 끈& 넌& 📹& Ɉ& Ҏ& ԝ& ֢& ݠ& ݮ& ߖ& ߤ& ࡈ& ࣺ& ॴ& ઁ& ણ& ଀& ୵& ஀& ௧& ఽ& ྕ& ሁ& ፈ& ┤& ✞& ➃& ⸜& ㄥ& 刌& 剳& 厵& 厷& 叄& 啇& 嗄& 婠& 嫼& 忳& 愀& 搛& 摛& 暅& 暳& 杍& 梱& 棫& 檞& 氍& 汵& 泇& 潽& 灦& 焛& 玌& 玜& 玿& 琫& 瑵& 瓃& 眜& 磢& 礸& 祕& 篔& 粛& 繻& 羖& 聼& 芞& 莀& 薕& 蘋& 蘳& 蝘& 褿& 鎉& 鏸& 鑑& 铏& 餣& 髆& 鮗& 鲬& 鵩& 鸀& 좌& 탕& ȭ& ʛ& ʧ& ˕& ϼ& Ӌ& Ә& Ӭ& ӭ& ԗ& ԩ& ֬& ׆& ؙ& ڒ& ڦ& ڲ& ۏ& ۘ& ݝ& ݥ& ݪ& ވ& ޱ& ߯& ࠝ& ࠨ& ࡫& ࡮& ࡴ& ࢀ& ࢒& ࢕& ࢠ& ࢮ& ࣀ& ࣼ& ࣿ' ৒' ৬' ੻' ઒' ડ' ૆' ૒' ૸' ଩ ' ୾ ' ஂ ' ஄ ' ஘ ' ௃' ௅' ௟' ఋ' ఍' ౉' ౓' ಄' ೞ' ഢ' ෕' ᄆ' ካ' ណ' ὺ' ‽' ⑅' 乁' 乇 ' 佂!' 冴"' 叆#' 呺$' 坲%' 墝&' 娸'' 婥(' 屪)' 忮*' 恏+' 愙,' 愰-' 憺.' 掭/' 旿0' 暼1' 栁2' 汑3' 沷4' 洊5' 烚6' 焞7' 珒8' 瑝9' 璮:' 痏;' 穲<' 簁=' 紥>' 繸?' 罃@' 肏A' 茳B' 荖C' 蒧D' 虙E' 裺F' 襻G' 誖H' 貼I' 贎J' 跂K' 踀L' 辵M' 醭N' 餅O' 駉P' 騆Q' 骒R' 룡S' 슨T' 옥U' 율V' 잊W' 잔X' Y' 🎯Z' 🐦[' Ķ\' ɸ]' ˁ^' ͯ_' ዳ`' ᐊa' ᗜb' ᮨc' ἴd' ⇪e' ⎪f' ❭g' ⺠h' 俜i' 俤j' 兎k' 冱l' 吚m' 哯n' 噍o' 坮p' 垈q' 夆r' 姱s' 姷t' 嬑u' 嬾v' 孭w' 寑x' 巂y' 楛z' 沇{' 煛|' 珆}' 琈~' 琜' 瑎' 瑿' 璅' 璫' 璵' 瓄' 疁' 瞀' 砢' 碦' 禑' 簏' 簴' 綟' 緜' 緼' 聓' 艶' 艼' 芼' 觭' 譌' 貞' 轪' 郛' 鄗' 鉨' 銟' 銧' 鋈' 鏻' 铓' 餸' 馘' 鯙' 鳣' 鹠' 麕' 끼' 압' 줍' 쪽' 탁' 튼' 획' ' 🍀' 💦' ɖ' ʋ' ˧' ਅ' ධ' ፍ' ≃' ◪' 倅' 倢' 倷' 僣' 咠' 圫' 奣' 姀' 媔' 嫮' 宔' 嵎' 弨' 恠' 慬' 憹' 戣' 揎' 擫' 擮' 杤' 柙' 梜' 棡' 槱' 毦' 滪' 瀫' 燣' 爝' 玅' 珋' 瑹' 璌' 瓉' 瘣' 睜' 砉' 砩' 窬' 籭' 纞' 罽' 脬' 舯' 芤' 蔊' 蛑' 蟧' 蟲' 衖' 袾' 覩' 誮' 谼' 趫' 跩' 踭' 躭' 遯' 醾' 鈝' 鈟' 镴' 閄' 陥' 顟' 騨' 鼚' 넷' 밝' ' 🌥( 💁( ĺ( Ƅ( ͠( Ө( ճ( ֣( ৮( બ ( ᄅ ( ᴋ ( ✋ ( ❋ ( 乨( 偸( 勄( 勧( 呫( 坋( 垔( 夵( 婰( 媂( 媏( 岧( 崐( 崬( 嶟( 廳( 懃( 敧( 旛 ( 昤!( 晅"( 晛#( 晥$( 栛%( 椛&( 槪'( 櫊(( 氂)( 氺*( 渃+( 渇,( 焫-( 珛.( 琱/( 瑏0( 瑴1( 甐2( 癋3( 盌4( 眚5( 睗6( 祔7( 篯8( 粙9( 糵:( 綑;( 缷<( 翈=( 胕>( 膵?( 臞@( 蘡A( 蜾B( 蟌C( 觱D( 詝E( 貤F( 跀G( 躱H( 躾I( 鈴J( 饐K( 鵁L( 麣M( 黀N( 黡O( 黨P( 률Q( 멀R( 벨S( 붙T( 옷U( 춤V( 텐W( 팀X( 🌳Y( 💉Z( 💐[( 💵\( ͗]( ܀^( శ_( ჯ`( ᅀa( ᠌b( ῇc( ▭d( 偟e( 偪f( 僈g( 勞h( 厰i( 唪j( 噺k( 圕l( 夋m( 娒n( 寘o( 尨p( 巇q( 帨r( 帹s( 廕t( 憸u( 戀v( 扽w( 挕x( 挻y( 搠z( 斶{( 旉|( 旐}( 晈~( 朙( 栘( 棢( 槹( 橴( 湨( 澐( 瀜( 焔( 煇( 玵( 玶( 琓( 琩( 瑐( 璭( 璹( 瓂( 瘐( 盝( 眏( 矴( 碒( 磏( 穊( 笣( 筭( 籤( 紬( 纙( 羕( 腨( 艨( 芲( 莯( 蒃( 蓜( 蕸( 蜎( 蜺( 蝂( 袊( 誃( 跍( 辒( 鄭( 鞩( 頕( 騊( 驎( 骦( 鶵( 鹥( 鹯( 齾( 났( 녹( 듣( 렉( 맘( 몬( 🅽( 🚩( ɤ( ѣ( Ү( Ք( ॅ( થ( ଼( ష( ༩( ዲ( ↴( ⻅( 仱( 侓( 倰( 偉( 僡( 凧( 劓( 呣( 呮( 噛( 囃( 囘( 圐( 埰( 堼( 壞( 奊( 婸( 岯( 弪( 忔( 悢( 挍( 捊( 掊( 揜( 晸( 暕( 朘( 杄( 枒( 柨( 桵( 椝( 橒( 橜( 欃( 欷( 泜( 湝( 潙( 澽( 灒( 牋( 牗( 牷( 珕( 琑( 琣( 瑲( 瑻( 瑽( 璯( 璶( 璻) 瓆) 瓙) 瘝) 瞾) 礌) 禋) 篸) 耩) 脿 ) 荈 ) 荹 ) 菥 ) 菼 ) 葢) 蒊) 蝨) 螀) 襜) 診) 譆) 譾) 讱) 赿) 逥) 鄮) 醸) 鐑) 霂) 靊) 鞶) 鞿) 韸 ) 頬!) 颕") 髴#) 鬖$) 鬬%) 魋&) 鯻') 鳳() 鳿)) 鵰*) 鶏+) 鷟,) 麪-) 黈.) 齻/) 깨0) 벤1) 숨2) 젊3) 캠4) 5) 🎓6) 😒7) ǃ8) Ύ9) ұ:) Խ;) ܡ<) ܫ=) ङ>) ੂ?) ආ@) ⛳A) ㄚB) 丅C) 乮D) 仧E) 俀F) 倛G) 冨H) 凎I) 剤J) 叺K) 埲L) 夀M) 妠N) 姸O) 娞P) 娪Q) 婫R) 媁S) 媝T) 嬓U) 審V) 尠W) 崹X) 庀Y) 悆Z) 悊[) 慅\) 捃]) 揀^) 攮_) 攰`) 敭a) 斄b) 斲c) 昘d) 晼e) 暥f) 梪g) 檑h) 檰i) 欵j) 歜k) 歰l) 漤m) 燀n) 牾o) 玝p) 玼q) 珿r) 瑬s) 璱t) 瓑u) 皊v) 皝w) 盻x) 砯y) 磵z) 祃{) 祩|) 秥}) 稙~) 稥) 窰) 筘) 筨) 縒) 繙) 羝) 羴) 肈) 脞) 舺) 莕) 菻) 蔘) 虡) 蝟) 蠘) 衝) 訑) 譁) 貓) 賴) 迖) 逬) 遃) 銙) 鍝) 鏚) 锍) 锎) 阾) 霪) 霺) 鞻) 駃) 駐) 骍) 鮋) 鯵) 鳒) 鳤) 鶄) 鶗) 鶲) 뷰) 홀) 🌙) 🍃) 🏐) 💌) Ĕ) ෘ) ࿐) ኢ) ↝) ≅) ◽) ⚆) ❧) ⲁ) ⲣ) 亽) 僊) 儚) 兡) 冭) 剉) 剰) 叚) 嗢) 嗼) 垿) 壐) 壽) 奅) 奟) 妟) 妡) 姠) 娿) 媴) 嬂) 尐) 扂) 揵) 摿) 朌) 杬) 枡) 棆) 椱) 槚) 歔) 歗) 毹) 洤) 湵) 澣) 濄) 濔) 灉) 烰) 焄) 狚) 玤) 琁) 琸) 瑂) 瑇) 璓) 璖) 瓇) 痾) 稅) 稊) 筴) 糈) 緈) 縦) 繝) 纴) 耇) 肭) 蓋) 薸) 蚼) 蜑) 蠼) 裲) 讚* 谻* 跽* 蹱* 鄫* 酦* 醡* 鋓* 鍂* 鏐 * 鑗 * 锪 * 阼 * 鞕 * 頞* 飔* 駂* 髽* 鰶* 鱥* 鲝* 鴳* 鵈* 鶰* 층* 팬* 🌏* 🐝* 🤳* 🥶* 🦃* ʖ* ҷ * Ӳ!* આ"* ထ#* ὀ$* ⃙%* ↻&* ⏳'* ⚈(* 亱)* 俆** 俖+* 偍,* 呴-* 喌.* 喓/* 嚄0* 囇1* 坽2* 塟3* 奃4* 妕5* 姳6* 媃7* 嫏8* 嬫9* 嶌:* 帊;* 廍<* 廐=* 弜>* 怱?* 愼@* 憙A* 扺B* 揕C* 晍D* 枻E* 柺F* 栺G* 梣H* 椻I* 楿J* 槔K* 欬L* 殢M* 澯N* 濈O* 烸P* 牳Q* 狌R* 珼S* 琒T* 琧U* 琷V* 璙W* 皜X* 皭Y* 睍Z* 砳[* 硋\* 硙]* 祲^* 窫_* 絥`* 絪a* 絽b* 缞c* 罫d* 羂e* 羵f* 胓g* 臯h* 舨i* 茘j* 茻k* 莝l* 莩m* 蒪n* 薆o* 藟p* 袓q* 褧r* 貝s* 軱t* 酕u* 醄v* 醤w* 鐻x* 镩y* 霝z* 鞇{* 鞬|* 饾}* 騕~* 鬹* 鯄* 鱵* 鴥* 鴽* 麏* 龖* 춘* 칙* * * 🌺* 🔭* 🦅* ̏* ͥ* ΐ* ݜ* ଉ* ଘ* ୁ* ർ* ࿆* ♕* 伜* 伩* 侎* 侒* 倗* 凓* 匽* 厺* 咅* 啅* 啛* 嚢* 嚸* 囏* 埦* 墈* 墐* 墺* 壠* 妱* 婹* 嫧* 嬃* 岓* 嵿* 嶆* 庯* 徏* 徯* 憇* 抂* 拺* 挿* 掹* 揳* 敻* 斸* 旵* 杕* 枼* 枿* 桙* 梘* 棯* 橑* 檭* 櫶* 殑* 沖* 泘* 洿* 渧* 湩* 滅* 炇* 燋* 爔* 狝* 獿* 玒* 瑓* 璄* 璤* 畩* 眰* 睝* 祫* 竢* 筥* 箢* 籓* 籘* 籞* 粋* 縪* 聻* 脢* 舦* 蓛* 蓷* 虅* 蠋* 蠙* 蠳* 蠵* 衒* 裯* 襃* 襌* 襩* 覠* 觳* 諽* 賍* 赽* 轚* 迀* 迡* 逺* 鄩* 鄪* 酺* 鋼* 錀* 鍱* 鍼+ 鍽+ 鎝+ 鎭+ 鎻+ 闉+ 陼+ 陾+ 雿+ 霩 + 韮 + 飶 + 騟 + 鬿 + 鮶+ 鰆+ 鱲+ 麐+ 麑+ 齂+ 협+ + 🍕+ 🕯+ 󠁥+ 󠁮+ ƿ+ Շ+ ૭+ ஷ+ ⌛+ ✑+ 〆 + 仚!+ 侜"+ 侱#+ 儍$+ 儙%+ 冾&+ 劂'+ 勑(+ 匉)+ 叀*+ 叾++ 吣,+ 咡-+ 唅.+ 唍/+ 嘙0+ 圀1+ 垨2+ 壄3+ 妭4+ 姯5+ 婓6+ 媱7+ 嫶8+ 嬟9+ 嬦:+ 嬮;+ 孮<+ 宍=+ 寎>+ 崝?+ 崿@+ 嵫A+ 巌B+ 幰C+ 忴D+ 悗E+ 惓F+ 懳G+ 拕H+ 摏I+ 擁J+ 攎K+ 攴L+ 旑M+ 昞N+ 曡O+ 杙P+ 杧Q+ 枆R+ 柌S+ 棅T+ 榬U+ 槩V+ 樳W+ 橪X+ 橲Y+ 檽Z+ 殻[+ 氶\+ 淉]+ 湶^+ 漙_+ 潄`+ 瀙a+ 灺b+ 烆c+ 煑d+ 煪e+ 煿f+ 熤g+ 燶h+ 燻i+ 獙j+ 玱k+ 珇l+ 珜m+ 珶n+ 珸o+ 琖p+ 瓁q+ 甞r+ 疦s+ 瘖t+ 皯u+ 碽v+ 礿w+ 秏x+ 秬y+ 稄z+ 穋{+ 窐|+ 竂}+ 竻~+ 笵+ 箵+ 簦+ 糒+ 繧+ 罇+ 耢+ 聤+ 膄+ 膴+ 艅+ 莚+ 菧+ 萊+ 蔿+ 蚷+ 蟮+ 褆+ 襙+ 觽+ 詾+ 誗+ 諲+ 趉+ 踨+ 蹢+ 遡+ 鄀+ 鐦+ 锠+ 隤+ 霦+ 鞔+ 顡+ 飈+ 駓+ 骹+ 鬐+ 魭+ 鴃+ 鴓+ 鴞+ 鵾+ 鹒+ 鹱+ 黮+ 齴+ 갖+ 틀+ 픽+ 핵+ 🏡+ 🕔+ ͌+ ҽ+ ٣+ ଓ+ ஊ+ ಶ+ ೀ+ ቅ+ ₫+ 乭+ 亂+ 侴+ 侽+ 傱+ 劆+ 卨+ 吰+ 呰+ 咷+ 啍+ 啩+ 喛+ 嘼+ 垉+ 塕+ 墧+ 壱+ 奷+ 姈+ 娍+ 娳+ 娷+ 婞+ 婻+ 媀+ 嫐+ 嫽+ 孅+ 孋+ 宂+ 寬+ 岆+ 崄+ 幇+ 幪+ 廲+ 弸+ 彨+ 怊+ 恦+ 慔+ 憟+ 挦+ 挷+ 揨+ 揻+ 撹+ 擿+ 敍+ 斵+ 昬+ 晀+ 枠+ 桟+ 棑+ 橂+ 檙+ 氥+ 漃+ 漛+ 潤+ 澫+ 炞+ 煱+ 燈+ 猨+ 獦, 玪, 珄, 珱, 琋, 琌, 琕, 瑈, 瑘, 瑼 , 眑 , 眿 , 睯 , 矼 , 硜, 硠, 禡, 禨, 秵, 穈, 穐, 筽, 篚, 篭, 籧, 緀, 緖, 翬, 耂, 耰, 耲, 聭, 膭 , 膰!, 臉", 舠#, 艁$, 苐%, 苖&, 苸', 莾(, 蕶), 藵*, 螷+, 裈,, 覥-, 觘., 觺/, 赮0, 趞1, 跢2, 蹝3, 辢4, 逴5, 遈6, 邘7, 鄈8, 鈘9, 鉱:, 銤;, 鋹<, 鍡=, 鐮>, 鑛?, 闁@, 阛A, 霨B, 靌C, 鞊D, 頠E, 頵F, 顑G, 顲H, 颥I, 髮J, 鬷K, 鱯L, 鲙M, 鶬N, 鶺O, 鷃P, 鼢Q, 닌R, 슴S, 폭T, U, ĨV, ֱW, ೇX, ༧Y, ᄒZ, ሉ[, ỷ\, ⋄], ◝^, ✴_, ✽`, ⻆a, ㄢb, 丱c, 亐d, 亰e, 伒f, 倽g, 兛h, 冞i, 刞j, 匄k, 匸l, 厐m, 喿n, 嗙o, 圷p, 坶q, 堈r, 塽s, 夣t, 妋u, 娀v, 婈w, 婦x, 媕y, 媤z, 媬{, 嫕|, 屒}, 峂~, 峈, 庝, 彐, 彞, 忀, 怇, 恷, 愳, 愺, 慓, 慹, 憅, 戹, 拤, 拪, 捙, 揫, 摥, 斨, 昲, 晊, 曚, 朂, 朳, 梴, 棓, 楀, 檛, 殟, 氆, 汸, 涶, 涽, 淓, 淟, 溒, 潓, 澨, 濎, 濳, 瀳, 炚, 烱, 烿, 熯, 牻, 犂, 狫, 猦, 珔, 珚, 琗, 瑥, 瑦, 璕, 璗, 璝, 璼, 疌, 疢, 瘌, 瞡, 瞫, 砪, 硟, 碏, 稧, 穜, 笭, 筶, 箑, 籝, 籱, 粓, 縇, 繄, 纚, 羓, 羶, 耈, 腄, 臖, 艴, 芏, 苪, 茝, 葁, 葇, 蒅, 蓧, 薿, 虨, 虩, 蜨, 袚, 襵, 譵, 讌, 蹓, 轗, 迗, 鄳, 酄, 醑, 釡, 釱, 銈, 鐏, 雔, 餏, 駧, 騳, 骻, 髿, 鮄, 鳁, 鹔, 鼡, 齉, 닥, 샵, 측, 칭, 훈, 🏟, 💸, 📆, 📩, ҂, ॰- ඔ- ᓂ- ᛫- ធ- ោ- ╟- ▎- 乪- 亇 - 佲 - 劦 - 匑 - 卄 - 卾- 吀- 吇- 呌- 哖- 哬- 啗- 嗘- 嚆- 嚔- 圑- 埊- 埑- 堨- 塙- 墎- 墠- 墴- 夳 - 奯!- 奰"- 妬#- 媙$- 媢%- 嬁&- 嬍'- 孆(- 宨)- 寕*- 寖+- 峀,- 峸-- 嵅.- 嶍/- 帡0- 彇1- 徢2- 忓3- 忹4- 悧5- 惐6- 愅7- 慂8- 慃9- 慏:- 慞;- 懩<- 扙=- 抲>- 揦?- 揼@- 摡A- 擈B- 旟C- 旣D- 昡E- 晢F- 晣G- 曟H- 曨I- 杘J- 枌K- 柂L- 梈M- 棙N- 椚O- 楅P- 樖Q- 橊R- 檾S- 櫂T- 欟U- 歊V- 毴W- 沺X- 泹Y- 洉Z- 洭[- 涀\- 淂]- 淰^- 渫_- 滱`- 漟a- 潵b- 澛c- 澞d- 濙e- 濵f- 烓g- 烗h- 熂i- 珗j- 珘k- 珬l- 琡m- 璥n- 瓍o- 疭p- 盀q- 睌r- 瞉s- 瞐t- 砈u- 硍v- 祴w- 秊x- 穾y- 笝z- 筦{- 箖|- 箯}- 篨~- 粻- 絴- 緾- 罤- 羄- 羐- 芓- 莃- 萫- 葮- 葻- 蓲- 蕥- 薍- 薖- 蘀- 蚥- 蝩- 蟡- 蟴- 衜- 裎- 褍- 褝- 詢- 諟- 謡- 譩- 豖- 貎- 賌- 賎- 賟- 賳- 賷- 跓- 跭- 輢- 逪- 鄝- 酟- 醲- 釢- 鈚- 鈬- 鈭- 鉥- 錱- 鍄- 鎽- 鏰- 鑐- 阘- 阰- 陦- 隒- 雞- 霅- 靀- 靿- 鞗- 飣- 馎- 駰- 驣- 鯷- 鲿- 鳂- 鴒- 鸑- 齇- 괜- 덕- 됐- 렌- 섬- - - ︉- 🍋- 🎼- 💣- 🖕- 🤕- 🥈- Į- ધ- ధ- ై- ཝ- ጣ- Ḡ- ἑ- ⲩ- 丆- 乑- 伀- 佊- 侕- 俙- 凐- 刕- 刣- 刯- 刼- 匞- 厡- 吜- 唭- 嘠- 噈- 噕- 囬- 埥- 塈- 墚- 墪- 夅- 姅- 娐- 婑- 嫅- 孞- 宻- 嵸- 嵾- 巗- 帋- 幤- 廌. 廦. 徑. 懥. 抩. 捁. 掛. 揈. 摂. 摝 . 敫 . 斮 . 暤 . 朑 . 朢. 朹. 杝. 柲. 梾. 椾. 樥. 橅. 橿. 檇. 櫘. 欔. 歠. 沘. 洏. 涫. 淭. 湻. 溋 . 溵!. 溻". 滮#. 潝$. 潡%. 瀡&. 灀'. 灜(. 焠). 煂*. 熍+. 熷,. 燅-. 爕.. 犨/. 珡0. 畇1. 皕2. 眊3. 眳4. 睎5. 瞵6. 碤7. 磈8. 磞9. 秜:. 稾;. 窴<. 竤=. 笁>. 筪?. 粊@. 糎A. 糑B. 紸C. 絒D. 絚E. 絧F. 縿G. 繜H. 繠I. 罏J. 罙K. 罳L. 耮M. 耴N. 胠O. 脥P. 脫Q. 膚R. 芵S. 苳T. 萑U. 萯V. 萾W. 蒈X. 薲Y. 藅Z. 蘌[. 虝\. 蝡]. 蟽^. 袗_. 袿`. 覮a. 詖b. 詧c. 諅d. 謢e. 譒f. 豭g. 趚h. 趬i. 跰j. 踘k. 踠l. 蹘m. 轌n. 迣o. 遶p. 郍q. 銛r. 鋗s. 鋧t. 錅u. 鍎v. 鍵w. 鍿x. 鎿y. 鏮z. 鑘{. 钃|. 镋}. 镎~. 镥. 隢. 隺. 鞈. 鞚. 韠. 韤. 韽. 顚. 飍. 餖. 驉. 驫. 骽. 髚. 鬊. 鬪. 鬫. 魌. 魽. 鮃. 鯸. 鰼. 鱎. 鱬. 鴷. 鸅. 麯. 黅. 齊. 齫. 齵. 龗. ꇴ. 딸. 뜨. 롭. 뮤. 찮. 탑. 폰. 픈. 🅼. 🎾. 🔶. ƶ. ǁ. Ɂ. ֿ. ॐ. ৌ. ୀ. ஒ. ಅ. ൻ. ඇ. ဒ. ጋ. ᑋ. ័. ⇊. ⊹. ⋊. ➎. ⻋. ⻝. 倻. 偐. 偫. 傁. 傿. 僜. 兂. 凵. 剏. 剚. 劒. 劗. 劸. 厪. 呝. 哤. 哵. 唞. 喼. 嚜. 囸. 圂. 圴. 垙. 埧. 埽. 堮. 塀. 墌. 墦. 夊. 夡. 奌. 妀. 妰. 姇. 姙. 娗. 婔. 婘. 媮. 嫨. 嫳. 嬯. 尯. 尶. 屩. 屼. 岃. 岹. 峵. 崊. 嵍. 嵹. 巪. 庤. 廘. 弆. 彔. 彜. 忇. 恉. 悂. 悇/ 悮/ 慡/ 慻/ 憛/ 憝/ 憡/ 懽/ 抷/ 拑 / 拝 / 拫 / 挮 / 揌 / 揓/ 搢/ 撋/ 擝/ 斒/ 旹/ 昛/ 昩/ 曵/ 枊/ 柧/ 栒/ 桊/ 桝/ 梹/ 楳/ 榱/ 橆/ 橎 / 欆!/ 殽"/ 毄#/ 泝$/ 浤%/ 湺&/ 溄'/ 溭(/ 澚)/ 濋*/ 濣+/ 灅,/ 灓-/ 焋./ 焢// 煃0/ 熦1/ 爉2/ 獖3/ 獤4/ 玸5/ 珁6/ 珢7/ 琔8/ 琾9/ 瑃:/ 璢;/ 瓩/ 睘?/ 睩@/ 矰A/ 砽B/ 碃C/ 磹D/ 禶E/ 秇F/ 稡G/ 窻H/ 箥I/ 箺J/ 簽K/ 籕L/ 籷M/ 粨N/ 絁O/ 綉P/ 緸Q/ 罞R/ 罱S/ 膌T/ 芔U/ 茓V/ 荋W/ 菫X/ 萢Y/ 葓Z/ 葹[/ 蒚\/ 蔈]/ 藧^/ 蛨_/ 蠨`/ 袨a/ 裉b/ 裬c/ 襳d/ 觖e/ 觿f/ 訸g/ 詌h/ 詪i/ 詺j/ 諕k/ 諡l/ 謱m/ 譹n/ 豯o/ 貹p/ 賸q/ 赨r/ 趙s/ 踒t/ 踕u/ 蹨v/ 躷w/ 輖x/ 逰y/ 邩z/ 郃{/ 酖|/ 鉌}/ 鉣~/ 銒/ 鋚/ 鋳/ 錇/ 鍹/ 鎛/ 鐕/ 铚/ 镵/ 镻/ 陑/ 霘/ 霶/ 頲/ 飋/ 飠/ 飡/ 飳/ 騱/ 驋/ 鬾/ 魕/ 鮢/ 鯼/ 鰊/ 鱠/ 鳻/ 鴱/ 鵱/ 鷢/ 鸘/ 鸙/ 鸼/ 麧/ 鼄/ 鼊/ 齚/ 떨/ 몇/ 뭐/ 익/ 큼/ / 🎀/ 🔑/ 🖐/ 🗞/ 🧠/ 󠁣/ 󠁴/ ƅ/ Ɛ/ ǽ/ ɥ/ ɧ/ ӏ/ ਭ/ ೋ/ ༦/ 丂/ 仈/ 伬/ 伿/ 倄/ 偆/ 偞/ 僩/ 冃/ 冦/ 劮/ 匛/ 厖/ 咃/ 咢/ 唚/ 唶/ 喤/ 噃/ 囋/ 圢/ 坁/ 垥/ 埯/ 塉/ 壒/ 夑/ 妚/ 妿/ 姄/ 姧/ 姺/ 婟/ 婣/ 媈/ 媿/ 嫈/ 嫯/ 嬀/ 嬥/ 嬬/ 寽/ 屳/ 峩/ 嵲/ 嶔/ 嶲/ 巛/ 巭/ 巵/ 廧/ 徝/ 忬/ 忺/ 恴/ 惙/ 惡/ 愻/ 慒/ 扚/ 挐/ 挘/ 挩/ 捾/ 掯/ 摷/ 撧/ 攦/ 昹/ 昻0 晻0 暀0 暩0 曒0 杒0 枈0 柆0 柎0 栔 0 桼 0 椓 0 椥 0 椮 0 楡0 榚0 槤0 橖0 橥0 檶0 櫆0 欀0 歬0 殎0 泏0 泬0 浗0 湙0 湦0 溠0 滍0 滐0 滺 0 漎!0 漜"0 漨#0 潕$0 濅%0 濿&0 炰'0 烄(0 焂)0 爠*0 牮+0 狆,0 猈-0 猯.0 猶/0 獃00 獽10 珤20 珯30 璷40 璾50 甛60 甮70 畕80 疇90 痡:0 瘲;0 癓<0 眴=0 睅>0 瞘?0 砃@0 砅A0 砇B0 硂C0 硡D0 碢E0 祮F0 禢G0 窷H0 窾I0 竐J0 筳K0 箎L0 箻M0 篍N0 篘O0 篺P0 簉Q0 簱R0 籂S0 粃T0 紃U0 紏V0 紦W0 綎X0 縗Y0 縧Z0 績[0 繖\0 纇]0 纐^0 缻_0 缾`0 罉a0 胷b0 脮c0 膣d0 臢e0 臩f0 舧g0 芿h0 茞i0 茩j0 莋k0 莻l0 萂m0 蒱n0 蓾o0 薠p0 蘯q0 蘷r0 虪s0 蚄t0 蚔u0 蚠v0 蛬w0 蝔x0 蟩y0 蠺z0 袑{0 褵|0 襫}0 覒~0 覔0 詉0 誙0 誧0 譔0 譝0 譪0 貮0 賐0 赯0 赸0 赻0 趠0 趶0 跼0 踛0 踷0 躹0 軆0 輹0 轜0 遌0 遻0 醙0 釳0 鈅0 鈈0 銔0 鍅0 鍙0 鍚0 鎈0 鐂0 鐱0 鑯0 阞0 阦0 隮0 隻0 霐0 靯0 鞧0 韐0 韝0 頾0 飷0 餋0 馉0 馌0 馹0 駠0 髜0 鬒0 鮍0 鰟0 鰿0 鱛0 鱪0 鴈0 鴎0 鵮0 鶌0 鶹0 麄0 黫0 鼧0 鼱0 齺0 념0 밖0 켜0 풀0 혹0 0 0 0 𓆜0 𓆞0 🆆0 🌲0 👦0 ȟ0 ͧ0 ८0 ஐ0 ౪0 ശ0 ീ0 ൂ0 ຜ0 ᴚ0 Ἐ0 ⌋0 ◜0 ♤0 ⚬0 丗0 乕0 亷0 倈0 偭0 傎0 傚0 僎0 僝0 僤0 僾0 儽0 兝0 冿0 刦0 刲0 剘0 剹0 劁0 劋0 劔0 劙0 劤0 厜0 吘0 呾0 唝0 唲0 啴0 噭0 嚝0 囄0 囆0 囥1 囨1 坥1 垻1 墸1 壻1 夶1 娔1 娵1 娹 1 婂 1 婖 1 媅 1 媌 1 媠1 嫊1 岝1 峗1 崪1 嶊1 嶎1 嶣1 嶹1 帬1 幆1 幮1 廗1 廞1 彅1 彠1 彽1 忊1 怳 1 悕!1 惞"1 惼#1 愗$1 愞%1 愹&1 憞'1 扸(1 挔)1 掵*1 掶+1 掻,1 搊-1 摖.1 擧/1 擳01 攕11 敉21 旾31 暪41 曫51 枬61 柕71 柣81 柹91 栶:1 桬;1 楟<1 榥=1 樀>1 樻?1 橧@1 檟A1 櫰B1 歖C1 殗D1 殣E1 毺F1 氀G1 氜H1 氞I1 汘J1 浧K1 淈L1 淍M1 渰N1 湼O1 滀P1 漑Q1 潉R1 澘S1 灐T1 灠U1 炈V1 炓W1 烲X1 焬Y1 煓Z1 煚[1 熛\1 燡]1 犉^1 犺_1 狔`1 猑a1 琙b1 琠c1 璳d1 璴e1 瓱f1 畺g1 瘄h1 皐i1 皨j1 皪k1 瞏l1 瞹m1 砛n1 砠o1 硈p1 碆q1 碔r1 碯s1 碿t1 磗u1 磿v1 礮w1 礳x1 禷y1 秓z1 秶{1 稢|1 穠}1 穸~1 窀1 窌1 窚1 笲1 筙1 箚1 箠1 箤1 箾1 粐1 粭1 紒1 絯1 縰1 繊1 繤1 羛1 羱1 翂1 翭1 脨1 腍1 膗1 臝1 艖1 艧1 芕1 苊1 茣1 荵1 莌1 菢1 萗1 葔1 蔋1 薀1 薽1 虁1 虊1 蛍1 蛒1 蛦1 蛿1 蝍1 蠎1 裖1 褎1 襮1 覟1 觪1 訄1 詜1 誽1 謓1 謩1 譃1 譭1 讐1 讑1 豲1 貋1 贋1 贍1 赱1 趦1 踂1 蹔1 軨1 軷1 輘1 轛1 轝1 迚1 迿1 逎1 郳1 鄊1 鄕1 酏1 酛1 醨1 釾1 銉1 鋺1 錩1 鍓1 鏬1 鐈1 闣1 阥1 阩1 陣1 隿1 霿1 靸1 鞄1 鞳1 韛1 頀1 餱1 饟1 馂1 駇1 駗1 駞1 騧1 骱1 鯒1 鰎1 鰵1 鲾1 鳠1 鴀1 鴲1 鵞1 鷤1 鷵1 麛1 鼆1 齸1 깔1 낸1 떡1 랙1 털1 휘1 1 1 😯1 🛍2 🦐2 󠁬2 󠁷2 Ǯ2 ˖2 ϋ2 ਛ2 ભ2 ൺ 2 ඉ 2 ዓ 2 ዮ 2 ጿ 2 ᚷ2 Ế2 丣2 伆2 伵2 佸2 侚2 僟2 儌2 儗2 儢2 儯2 凅2 勼2 匴2 厎2 咉2 咊2 咍 2 咟!2 哠"2 啚#2 喐$2 喨%2 嚚&2 嚡'2 囙(2 垝)2 垺*2 埻+2 埾,2 堏-2 堓.2 堦/2 塦02 壌12 壣22 奛32 娊42 娕52 娰62 婅72 婌82 婜92 媇:2 媣;2 嫾<2 嬐=2 嬕>2 孄?2 孎@2 孏A2 孷B2 宭C2 屘D2 屭E2 峐F2 峢G2 巤H2 庒I2 庺J2 弮K2 忛L2 恘M2 恮N2 恾O2 悾P2 惿Q2 愱R2 慿S2 憓T2 懁U2 懕V2 戜W2 戼X2 扵Y2 抝Z2 抣[2 抧\2 拀]2 拸^2 捇_2 捸`2 掚a2 搲b2 摦c2 摵d2 擭e2 擸f2 攋g2 敇h2 敥i2 敹j2 斁k2 昽l2 晪m2 暶n2 曞o2 柟p2 柸q2 栧r2 梤s2 棬t2 椯u2 楒v2 楥w2 檓x2 檹y2 歀z2 歂{2 歄|2 殐}2 殬~2 毃2 汍2 沲2 沶2 涾2 溙2 滖2 滣2 漧2 漮2 漰2 潅2 澰2 澼2 濃2 炂2 焀2 焇2 焎2 煏2 熌2 熽2 牫2 狅2 狋2 狧2 獏2 玣2 琟2 甹2 畼2 疧2 疩2 痖2 痷2 瘭2 癛2 癪2 皁2 皔2 盨2 眆2 眖2 睉2 睠2 瞣2 矄2 矷2 硄2 礒2 礜2 礧2 祏2 禍2 稯2 穒2 穟2 笘2 笯2 筈2 筕2 筤2 箮2 篞2 簗2 簮2 籹2 粞2 糉2 糩2 紵2 綯2 緌2 緣2 縀2 繟2 罠2 羉2 翞2 聟2 肊2 胘2 胻2 腝2 腞2 臇2 臒2 芇2 芧2 苰2 莣2 莥2 葂2 葋2 葲2 蒁2 蒷2 蔢2 蕐2 蕑2 蕔2 薱2 蘫2 蘲2 蘽2 虗2 虰2 虵2 蚛2 蚾2 蜁2 蜏2 蝃2 蝪2 蝯2 蟰2 蠂2 蠭2 衻2 裪2 褠2 襒2 襭2 覛2 訯2 諌2 諴2 諵2 謒2 讂3 讛3 诪3 谽3 賔3 賖3 賙3 賹3 贕3 赾 3 趢 3 趷 3 跁 3 踣 3 蹡3 蹪3 蹹3 躀3 躆3 軑3 軚3 軶3 輮3 輷3 輺3 迃3 遝3 遤3 遾3 郻3 郼3 鄵3 醗 3 鈆!3 鈊"3 鈶#3 鉠$3 鉧%3 鋣&3 錝'3 鍲(3 鐡)3 阣*3 阤+3 隡,3 霡-3 靕.3 靝/3 韄03 頽13 颪23 颸33 餎43 餔53 餴63 馦73 駽83 驙93 骫:3 髞;3 髟<3 魮=3 魸>3 鮵?3 鯮@3 鰄A3 鱊B3 鳮C3 鶖D3 鹲E3 麖F3 麜G3 麞H3 麷I3 黸J3 齱K3 龞L3 龡M3 봄N3 셨O3 앙P3 콜Q3 𓆡R3 🍂S3 🍽T3 👽U3 🕳V3 🗳W3 🦀X3 ƗY3 ɩZ3 ʣ[3 Ҡ\3 ֻ]3 ؄^3 భ_3 ഗ`3 ഞa3 ཾb3 ዘc3 ⃗d3 ↆe3 ▰f3 ◮g3 ☢h3 ⬢i3 ⻘j3 ヲk3 乚l3 仸m3 佌n3 侊o3 侫p3 侺q3 俌r3 俹s3 偑t3 偡u3 偧v3 傋w3 傐x3 傔y3 傗z3 僘{3 僞|3 儶}3 兣~3 冐3 凗3 刓3 剓3 剕3 剟3 剫3 劐3 勎3 勫3 匁3 卋3 叐3 哊3 唩3 唺3 啌3 啿3 嘒3 嘽3 噊3 噵3 嚂3 嚈3 垀3 堲3 堻3 塎3 壈3 夦3 夰3 夽3 奫3 奼3 妏3 姟3 娋3 婏3 婙3 婤3 婲3 媗3 嫝3 嫴3 嫷3 嬄3 嬨3 孼3 宖3 寃3 尰3 屖3 屴3 岊3 峏3 峱3 崈3 崨3 崶3 嵉3 嵻3 嶅3 巸3 帣3 帾3 庼3 廫3 弉3 弽3 彯3 徃3 徎3 徸3 怗3 恌3 悳3 惤3 憥3 憼3 懆3 扄3 抌3 挌3 挓3 捿3 掞3 搝3 擑3 攂3 攇3 攐3 敂3 敪3 敱3 旙3 旽3 昮3 曉3 杣3 柮3 桛3 棔3 棶3 椡3 榩3 槞3 樲3 檖3 檥3 櫱3 欚3 歛3 殧3 殱3 毚3 毰3 汃3 汅3 沯3 泒3 洅3 洖3 洝3 浺3 涋3 涬3 渳3 湆3 溤3 漐4 漝4 潀4 澬4 瀦4 烇4 焻4 熖4 熪4 熫 4 牕 4 犃 4 犠 4 狕 4 玂4 玽4 珳4 琝4 瓨4 畨4 痁4 癷4 皶4 盄4 盕4 盫4 睧4 碥4 碫4 磠4 礍4 礑4 祌 4 祤!4 禸"4 秔#4 秛$4 竁%4 竏&4 竪'4 筀(4 筓)4 箆*4 篲+4 簆,4 簻-4 籦.4 籾/4 粌04 糽14 紞24 絍34 絙44 綪54 緉64 緢74 緭84 縂94 繷:4 绖;4 罶<4 羺=4 翸>4 聕?4 聙@4 聺A4 胊B4 胵C4 胹D4 腉E4 膕F4 膪G4 舡H4 芆I4 芢J4 苉K4 苝L4 茟M4 莿N4 萠O4 葄P4 葥Q4 葼R4 葽S4 蒩T4 蓚U4 蓪V4 蔇W4 蔖X4 蔠Y4 蔣Z4 蕍[4 蕽\4 薘]4 蘎^4 蘜_4 蘟`4 蘨a4 虀b4 虋c4 蚈d4 蜬e4 蝚f4 蝬g4 蟃h4 蟐i4 蟫j4 蠷k4 衃l4 衱m4 袥n4 裭o4 褏p4 覸q4 觧r4 觩s4 訹t4 誟u4 謻v4 豗w4 貙x4 賓y4 贀z4 跾{4 踤|4 踼}4 軿~4 輬4 轑4 轘4 迯4 迲4 逿4 郙4 鄸4 鄽4 酫4 鈎4 鈜4 鈯4 銆4 銞4 銶4 鋊4 鋋4 鋫4 錦4 錧4 錽4 鍠4 鎚4 鏧4 鏶4 鏺4 鐥4 鑍4 钖4 锳4 闚4 阹4 隓4 隞4 隫4 靋4 靮4 靲4 靻4 韂4 韑4 飱4 馵4 馷4 駫4 騉4 騹4 驘4 髊4 髾4 鬋4 鬜4 魯4 鯌4 鯎4 鯟4 鯠4 鰂4 鰘4 鱑4 鳛4 鳭4 鴡4 鴸4 鵔4 鶙4 麘4 黇4 鼔4 齋4 겁4 얻4 4 4 4 4 🆃4 🛎4 Ɠ4 Ǎ4 ˲4 ܣ4 ܳ4 థ4 ෂ4 ቸ4 ហ4 ᠾ4 ῃ4 ⏯4 ♨4 ⛽4 ✂4 㚼4 亁4 亠4 佒4 佮4 佱4 佹4 俰4 倵4 偄4 傉4 傮4 傺4 僀4 僄4 僆4 僓4 僔4 僸4 儛4 儥4 儴4 冘4 凩4 凴4 剢4 剬4 剾4 劍4 勔4 匟4 卂4 叅4 吅4 呟4 咹5 哃5 哣5 哹5 哾5 嗃5 嗻5 嗿5 嘥5 噟 5 嚁 5 嚘 5 嚬 5 嚹 5 囎5 囩5 圡5 坔5 坕5 坱5 埬5 埱5 堁5 堐5 堩5 塠5 墇5 墭5 墽5 壏5 壴5 奙5 奺 5 妶!5 姕"5 姛#5 娝$5 娾%5 媓&5 嫭'5 嬶(5 孇)5 屵*5 崌+5 嵁,5 嶀-5 嶕.5 嶥/5 帗05 幝15 廹25 弐35 彾45 徆55 怐65 怓75 怬85 怶95 恈:5 恧;5 悤<5 惒=5 愥>5 愩?5 慉@5 憜A5 懮B5 懴C5 抳D5 抾E5 捓F5 揗G5 搈H5 搒I5 搩J5 摙K5 摪L5 撪M5 撱N5 擌O5 攉P5 敊Q5 敤R5 敵S5 斀T5 旈U5 旪V5 昢W5 暞X5 朡Y5 朸Z5 杗[5 枃\5 枍]5 枲^5 柖_5 栣`5 桾a5 梛b5 棴c5 椄d5 楈e5 楌f5 榓g5 榠h5 槯i5 槼j5 樚k5 樢l5 樦m5 橞n5 橭o5 櫣p5 櫹q5 欌r5 欐s5 欗t5 殕u5 毼v5 氝w5 沗x5 沝y5 沠z5 泆{5 洷|5 涭}5 涻~5 淐5 湰5 溚5 漀5 潂5 潎5 潧5 潻5 澁5 濍5 瀺5 灷5 炋5 炐5 炧5 炿5 烼5 焃5 焍5 焝5 焥5 煋5 煷5 煹5 熅5 熲5 燖5 燤5 牥5 犦5 犪5 狖5 狟5 猼5 獆5 獱5 玃5 玴5 珻5 琘5 瓎5 甂5 畉5 畖5 畞5 畧5 畷5 癶5 皾5 盶5 睊5 睴5 瞆5 矐5 矑5 砘5 砿5 硞5 磇5 礉5 礟5 禂5 禕5 禣5 禭5 秙5 秠5 秱5 稁5 稬5 穃5 穇5 穛5 笇5 笒5 簈5 籊5 籖5 粸5 糺5 緟5 縕5 繌5 繶5 罦5 羃5 羢5 羾5 翖5 聇5 聠5 肒5 肶5 脼5 腷5 臿5 艎5 艑5 艢5 艦5 苃5 荮5 莟5 莰5 菳5 菾5 萲5 萴5 萻5 葈5 蒥5 蓒5 蓨5 蔉5 蕬5 蕳5 蕿5 薣5 薥5 藌5 藸5 藼5 蘓5 蘮5 虂5 虌5 虳5 虶5 蚘5 蛝6 蜠6 蜲6 蝿6 螪6 蟞6 蟤6 蟱6 衏6 衐 6 衧 6 衵 6 裋 6 裑 6 褃6 襍6 觙6 觝6 觫6 訷6 詤6 詽6 誱6 諐6 謟6 譓6 譗6 谫6 豣6 豥6 賵6 贑6 趑 6 趝!6 踸"6 蹧#6 蹮$6 躛%6 躽&6 軃'6 軳(6 辥)6 逌*6 遀+6 遖,6 邉-6 邖.6 鄋/6 醘06 醥16 醱26 釮36 鉁46 鉕56 鉩66 銐76 鋙86 錋96 鍕:6 鍳;6 鎶<6 鏯=6 鐊>6 鐞?6 鐾@6 钘A6 铇B6 闋C6 闏D6 阠E6 陊F6 陹G6 隟H6 雭I6 霵J6 靣K6 靹L6 靽M6 韾N6 顠O6 颬P6 颴Q6 飌R6 飝S6 餬T6 饎U6 馣V6 駩W6 驚X6 髬Y6 鬶Z6 鬸[6 鬽\6 鯍]6 鯱^6 鰛_6 鱂`6 鱃a6 鱙b6 鴢c6 鴮d6 鵄e6 鵖f6 鶠g6 鶳h6 鷆i6 鷰j6 鷾k6 鸆l6 鸉m6 鸌n6 鸴o6 鹻p6 麀q6 麃r6 麌s6 黖t6 鼗u6 鼫v6 齥w6 ꆏx6 농y6 흡z6 {6 |6 }6 ︆~6 🇼6 🌐6 🐺6 🖖6 🖼6 😝6 🟩6 🧀6 ŕ6 ɳ6 ӗ6 ӥ6 ֑6 ݄6 ଜ6 ଡ6 ജ6 ൽ6 ຢ6 ༶6 ཿ6 ჭ6 ሆ6 ሕ6 ቶ6 Ố6 ‱6 ↣6 ⌯6 ⚕6 ⲙ6 ⴰ6 ㄍ6 乧6 乺6 仠6 伃6 伭6 伻6 佄6 佡6 侲6 偖6 偱6 僗6 僙6 儊6 儰6 冩6 劯6 勽6 匼6 叧6 吪6 唈6 唋6 啀6 啠6 喍6 嘇6 嘐6 嘕6 嘨6 噂6 嚿6 圁6 圗6 圸6 圼6 垽6 埢6 堭6 堹6 墬6 壛6 夓6 夛6 奒6 奝6 奬6 奵6 妔6 娚6 娢6 娮6 娯6 娻6 婋6 婱6 媘6 媶6 嫸6 嬠6 嬧6 孈6 孧6 寴6 尩6 岏6 峟6 崉6 嵦6 嵰6 嵶6 嵺6 巃6 巈6 巕6 帩6 幋6 幐6 庴6 廵6 彄6 徹6 徿6 忟6 扢6 扨6 抅6 抭6 挴6 挵6 捝6 掍6 掝6 掫6 掽6 揚6 揞7 揟7 搄7 搕7 搾7 撜7 撯7 擡7 擩7 攈 7 攑 7 攧 7 敄 7 斠 7 旘7 暯7 朒7 朣7 杔7 柉7 栐7 桮7 桽7 梀7 梇7 棊7 椂7 椊7 椔7 椼7 楃7 楑7 楻 7 槏!7 槫"7 樏#7 樐$7 樒%7 櫢&7 欇'7 歮(7 殀)7 泟*7 浵+7 渹,7 湡-7 溊.7 溸/7 澦07 濆17 濪27 瀁37 瀂47 灖57 灨67 炄77 炵87 焟97 焳:7 煔;7 煯<7 熋=7 熞>7 熭?7 熼@7 燪A7 燲B7 燸C7 爖D7 犐E7 狓F7 狘G7 狾H7 珫I7 瑍J7 璑K7 璔L7 瓰M7 甈N7 畻O7 疺P7 痑Q7 痝R7 痻S7 痽T7 癹U7 皒V7 皡W7 眽X7 睋Y7 瞲Z7 瞷[7 矊\7 矨]7 矪^7 矱_7 矹`7 砄a7 硓b7 硘c7 硭d7 硻e7 硾f7 碻g7 磂h7 磑i7 磳j7 磶k7 礕l7 礛m7 礯n7 礻o7 祒p7 禌q7 禴r7 秄s7 稑t7 稭u7 穄v7 窉w7 竴x7 笉y7 笍z7 笩{7 笰|7 筸}7 箼~7 簒7 粇7 粩7 粺7 綅7 綌7 縡7 繗7 繪7 羬7 翏7 翑7 翣7 翧7 耝7 耠7 耹7 聀7 肰7 肵7 腂7 腣7 舑7 舮7 艊7 艥7 芅7 芖7 荎7 荢7 莊7 菋7 菬7 萅7 蒍7 蓃7 蓅7 蓔7 蓺7 蔅7 蕣7 薻7 薼7 薾7 藂7 藆7 蘈7 蘏7 蚎7 蛖7 蜽7 蟉7 蟙7 蠝7 蠧7 蠪7 蠬7 襎7 襔7 襼7 誴7 謮7 譋7 譜7 譥7 譱7 讆7 讝7 讻7 豃7 豄7 豋7 豷7 貛7 賮7 賿7 趻7 跿7 踧7 蹍7 躃7 軮7 輣7 輰7 轈7 迕7 迠7 遉7 遬7 邿7 郥7 郲7 鄐7 醹7 釺7 鉜7 鋄7 鋶7 鍌7 鍒7 鏔7 鏫7 鐌7 鐤7 鐧7 鐭7 鑝7 鑱7 铔7 铻7 閧7 閴7 闤7 闬7 陖7 隂7 隃7 鞾7 韍7 韲7 頨7 顨7 顪7 顭7 飿7 餟7 馢7 馿7 駵7 駸8 騜8 魵8 鮎8 鮡8 鮣8 鯛8 鰽8 鱅8 鱶 8 鳪 8 鴚 8 鵀 8 鵍 8 鵛8 鵵8 鶊8 鶐8 鹶8 黕8 鼌8 鼲8 鼼8 齰8 齹8 龂8 龥8 ꀋ8 ꈍ8 낌8 넣8 펜8 푸 8 !8 🌤"8 🍻#8 👸$8 💘%8 😻&8 ǣ'8 ɫ(8 ʠ)8 ΀*8 Ң+8 ҭ,8 Ӏ-8 ۀ.8 ܕ/8 ஹ08 ఛ18 ೬28 ༨38 ቃ48 ញ58 ↦68 ⌚78 ⬜88 丮98 丳:8 乵;8 仢<8 仭=8 仹>8 侞?8 侼@8 俫A8 倧B8 倲C8 倸D8 傂E8 傞F8 傼G8 僇H8 僯I8 儧J8 兏K8 兟L8 冺M8 刾N8 勚O8 卶P8 卼Q8 厱R8 叓S8 叜T8 咓U8 咮V8 咶W8 哘X8 哢Y8 哫Z8 唓[8 唼\8 嗧]8 嗴^8 嘪_8 噑`8 噡a8 噧b8 嚞c8 嚤d8 囓e8 圲f8 垘g8 堗h8 塴i8 墋j8 壜k8 奍l8 奦m8 妴n8 娙o8 娬p8 娺q8 婄r8 婎s8 媉t8 媥u8 媷v8 嫞w8 嫢x8 嬚y8 孂z8 孶{8 宼|8 屸}8 岒~8 峛8 峬8 峮8 崅8 崡8 嵀8 嵂8 嵃8 嶐8 嶢8 嶰8 巼8 帠8 帺8 幚8 幜8 庮8 庳8 廔8 廼8 忁8 忎8 忯8 恡8 惄8 愐8 愖8 慭8 慲8 憄8 憌8 憪8 憴8 憽8 扴8 拰8 挆8 挏8 掆8 揅8 揘8 搑8 搯8 摌8 摎8 摰8 摱8 擬8 攨8 攩8 敃8 敓8 敿8 斘8 昣8 昿8 暵8 曋8 曐8 曮8 朎8 杦8 杽8 枔8 桇8 桚8 梬8 棄8 椙8 楇8 楋8 楎8 榗8 槇8 橀8 橌8 橻8 檊8 檝8 櫍8 櫻8 欑8 欘8 欨8 欩8 欭8 殌8 殹8 毊8 毝8 毠8 汻8 泧8 洂8 洟8 涆8 涚8 涱8 淊8 渂8 溨8 溳8 溹8 澕8 濗8 瀓8 炏8 炦8 烋8 烞8 烳8 焸8 煁8 煵8 爋8 牐8 犈8 狪8 猓8 猧8 獁8 獄8 獥8 瓗8 甆8 畃8 畍8 畐8 畭8 疻9 痓9 痥9 癅9 癙9 癧9 癨9 盚9 盠9 盢 9 眪 9 矀 9 砡 9 磍 9 礄9 礝9 秌9 秪9 秳9 秹9 稆9 稒9 穣9 穤9 窛9 窧9 竓9 竔9 竚9 竵9 笀9 篃9 篈 9 篧!9 簂"9 簛#9 簢$9 籢%9 籶&9 粆'9 粔(9 糄)9 紐*9 紤+9 絼,9 綇-9 緍.9 縍/9 縥09 繏19 罀29 罜39 罭49 羗59 翄69 翲79 翿89 耼99 腃:9 腢;9 膓<9 膧=9 膼>9 臄?9 臽@9 舩A9 艬B9 芌C9 苽D9 茠E9 荗F9 荺G9 莡H9 菤I9 葊J9 蒼K9 蓆L9 蓫M9 蓭N9 蓱O9 蓶P9 蔂Q9 薋R9 薵S9 藃T9 藡U9 蘐V9 蘕W9 蘴X9 虆Y9 虼Z9 蚇[9 蛼\9 蝞]9 螁^9 螚_9 蟔`9 蟗a9 蠁b9 衉c9 衚d9 袀e9 袹f9 裻g9 褋h9 褩i9 襑j9 襘k9 襷l9 覝m9 觡n9 詚o9 詶p9 誝q9 諝r9 諰s9 謘t9 謧u9 譇v9 豮w9 貃x9 貈y9 貸z9 賗{9 贒|9 贘}9 赩~9 赹9 趭9 跊9 踍9 踾9 躻9 軈9 軭9 輈9 輎9 轒9 轓9 轥9 辌9 迊9 遞9 遧9 邫9 郂9 鄹9 鄼9 酘9 酳9 醿9 釶9 釸9 鈏9 鉊9 鉷9 銋9 銌9 鋀9 鋘9 鋭9 錿9 鍜9 鍯9 鍴9 鏋9 鏣9 鏪9 鐇9 鐚9 鑅9 鑦9 锽9 镹9 镼9 閳9 闅9 闑9 阭9 雗9 雘9 霋9 霥9 靉9 靎9 靱9 鞆9 鞺9 韅9 韔9 頥9 顖9 颣9 颽9 飂9 飵9 餙9 餤9 饓9 饛9 饡9 馯9 駬9 騋9 騎9 驇9 驔9 髸9 鬀9 魓9 魡9 魰9 鮙9 鮛9 鰋9 鰝9 鳱9 鴫9 鴵9 鶝9 鹙9 麨9 麭9 黦9 鼟9 鼪9 齌9 齯9 龣9 걷9 닝9 둥9 딩9 9 🍺9 🥣9 🪐9 Ȉ9 ˦9 Ϗ9 ֮9 ߍ9 ઞ9 ઺9 ૓9 ଆ9 ౿9 ຖ9 ຽ9 ᅮ9 ኣ9 ዊ9 ᐕ9 ᖅ9 ῥ9 ⇦9 ✹: 〰: ㄇ: 䭲: 乲: 伣: 侢: 侭: 偒: 偝 : 偤 : 傌 : 傽 : 僌 : 僒: 僫: 僶: 儵: 兠: 冄: 凞: 刡: 勌: 勥: 勶: 匔: 匬: 匳: 厇: 厼: 吤: 吷: 唀 : 唂!: 啈": 噇#: 噒$: 噰%: 嚛&: 嚪': 囕(: 囜): 圚*: 圝+: 坆,: 垗-: 埁.: 堘/: 堟0: 堧1: 堳2: 塯3: 塲4: 塷5: 壖6: 壼7: 夌8: 妜9: 妢:: 妷;: 姎<: 姡=: 婛>: 婽?: 媍@: 媎A: 媐B: 嫤C: 嬜D: 嬳E: 孊F: 孴G: 宎H: 寊I: 寏J: 尞K: 屇L: 岎M: 岲N: 峉O: 峞P: 崕Q: 嵆R: 嵈S: 嵢T: 嵤U: 嵱V: 嵼W: 嵽X: 嶚Y: 嶛Z: 嶠[: 巀\: 巁]: 帎^: 幈_: 幏`: 幑a: 庛b: 庩c: 廜d: 弅e: 彲f: 彺g: 徛h: 徱i: 忕j: 恝k: 悿l: 愑m: 愘n: 慽o: 憦p: 憭q: 懀r: 扜s: 抸t: 捄u: 捗v: 捘w: 捤x: 捥y: 捹z: 掁{: 掦|: 掿}: 揑~: 摀: 摠: 摬: 摴: 撠: 攃: 攓: 斣: 旜: 旝: 昈: 昍: 昸: 晠: 暔: 曻: 枎: 枺: 柡: 栙: 栞: 栮: 栵: 梍: 梑: 梸: 棁: 棃: 椵: 楱: 榁: 榤: 槈: 槦: 樈: 橍: 橮: 檷: 檼: 櫙: 欈: 歍: 殨: 泤: 泲: 洦: 渮: 渵: 湅: 湥: 湳: 湷: 溗: 溾: 滧: 漍: 潪: 澓: 濌: 濏: 濧: 瀭: 瀯: 瀵: 灛: 灢: 灩: 炍: 炑: 炾: 焨: 焵: 焺: 煄: 煈: 熝: 熰: 燽: 爁: 牓: 犔: 狥: 狦: 狶: 獉: 獕: 玁: 瓻: 甋: 畁: 畮: 痯: 瘉: 瘚: 癊: 盺: 睤: 睰: 睵: 瞚: 矔: 砓: 硆: 硶: 碀: 碊: 碷: 磟: 礊: 礏: 礢: 礲: 祰: 祳: 秝: 秺: 稏: 稨: 稩: 稪: 穧: 穪: 穳: 窋: 竫: 竱: 笷: 筺: 簳: 簶; 籵; 粅; 糦; 紛; 紭; 紾; 緃; 緆; 緔 ; 縙 ; 繲 ; 纆 ; 纉 ; 绬; 耚; 耫; 聮; 脄; 脵; 腒; 腲; 膹; 臮; 臷; 舤; 舼; 舿; 艠; 苀; 茥; 茪; 茾 ; 荂!; 荙"; 荝#; 荰$; 莁%; 莤&; 菞'; 菭(; 蒢); 蒮*; 蒶+; 蓏,; 蓸-; 蕯.; 薓/; 薳0; 藘1; 藙2; 藣3; 藮4; 藯5; 蘠6; 虈7; 蚑8; 蛈9; 蛥:; 蜭;; 蝅<; 蝹=; 螏>; 螑?; 蟖@; 蟟A; 蟵B; 蠄C; 蠞D; 蠣E; 袲F; 袺G; 袼H; 裚I; 褈J; 褗K; 褦L; 褨M; 褷N; 襉O; 襚P; 覎Q; 覡R; 訧S; 訩T; 訿U; 詍V; 詷W; 誕X; 諈Y; 謭Z; 譀[; 豜\; 豧]; 貿^; 賺_; 贔`; 趃a; 跒b; 跮c; 踎d; 踚e; 踳f; 蹆g; 蹜h; 軗i; 軜j; 輌k; 輭l; 轇m; 辠n; 辧o; 逓p; 郣q; 鄏r; 鄥s; 鄦t; 酁u; 酅v; 醝w; 醶x; 釐y; 釥z; 鈃{; 鉹|; 銁}; 銇~; 銝; 鋷; 錊; 錍; 錜; 錵; 錺; 鎃; 鎐; 鎒; 鎡; 鏍; 鏩; 鑞; 钁; 钷; 铘; 閅; 闓; 闝; 陭; 陮; 隖; 霚; 霻; 靘; 鞎; 韼; 頎; 頟; 頪; 頶; 飮; 饍; 饚; 駊; 騘; 騢; 驒; 骴; 骾; 髝; 髠; 鬇; 鬏; 鬛; 鮤; 鮬; 鮹; 鰃; 鰗; 鰪; 鱝; 鲄; 鳈; 鳬; 鴋; 鴺; 鵌; 鶋; 鶫; 鶱; 鷘; 鷡; 鷶; 鸐; 麍; 麰; 黆; 黓; 黤; 黵; 鼅; 鼝; 齖; 齘; 龓; ꋍ; 므; 쇄; 숭; ; ︀; 🎊; 🏹; 👧; 🤠; 🥉; Ƙ; Ȱ; ʉ; ѩ; ҆; ҩ; ٤; ۩; ݿ; ދ; ࡬; ॷ; ৎ; ਣ; ੤; ૲; ଄; ଢ; ଫ; ୼; ௳; ೅; അ; ഽ; ඘; ඳ; ቼ; ኈ; Ṛ; ⚗; ➺; ⲥ; 丵; 亪; 仴; 仺; 伮; 佨; 佫; 俋; 俓; 俕< 俷< 倂< 倕< 偗< 傦< 傫< 僛< 僲< 儖 < 儱 < 儳 < 兊 < 凔 < 凟< 刢< 刵< 剦< 剺< 劥< 劽< 勪< 匂< 匤< 匥< 叝< 呁< 咞< 啑< 喖< 喣< 嗀< 嗠 < 嘺!< 噆"< 噣#< 噽$< 嚃%< 嚑&< 嚾'< 圠(< 圥)< 圱*< 坹+< 埍,< 埿-< 堢.< 塣/< 塧0< 塶1< 塸2< 塼3< 墄4< 墰5< 墶6< 壍7< 壡8< 奲9< 妌:< 妵;< 姖<< 姰=< 姼>< 娤?< 婒@< 婝A< 媋B< 媻C< 嫟D< 嫥E< 嬺F< 孁G< 宐H< 宷I< 寠J< 屶K< 岉L< 岻M< 峔N< 崷O< 嵄P< 嵑Q< 嵵R< 帵S< 幖T< 廭U< 弍V< 彉W< 徍X< 忷Y< 怸Z< 怺[< 恀\< 悈]< 悑^< 悡_< 愮`< 慀a< 慯b< 懄c< 懭d< 戺e< 扊f< 扗g< 拵h< 捰i< 捵j< 揃k< 揋l< 揢m< 揧n< 搘o< 搸p< 搹q< 摉r< 摗s< 撌t< 攼u< 敁v< 敠w< 斅x< 斏y< 晽z< 暚{< 暽|< 曁}< 杅~< 杛< 枤< 枦< 柗< 柭< 栯< 梄< 梖< 棛< 椆< 椞< 椶< 樔< 橨< 橷< 檧< 檨< 檪< 檮< 櫐< 櫮< 櫯< 櫴< 櫾< 欦< 欴< 歅< 歏< 殜< 殩< 殰< 毥< 氼< 汋< 汥< 沑< 泙< 涒< 淃< 渘< 渜< 渞< 湱< 溕< 漄< 漻< 潖< 潛< 潬< 濓< 濴< 濻< 瀐< 瀒< 瀷< 灍< 炗< 炛< 炲< 烠< 烶< 烻< 焁< 焏< 焴< 焷< 燒< 燷< 爄< 爓< 爯< 爴< 爼< 牔< 犕< 犡< 狑< 猀< 猘< 獚< 獯< 瓟< 甊< 甶< 畗< 疐< 疜< 瘎< 瘨< 癎< 癏< 癑< 皉< 皘< 盙< 盷< 眒< 眢< 眧< 睓< 睕< 睳< 瞝< 矕< 砋< 砫< 砮< 硲< 碪< 磫< 礰< 礶< 祋< 禫< 秎< 秞< 穅< 穕< 穩< 窞< 窮< 竗< 竲< 竳< 笂< 笌< 笟< 箊< 箞< 箣= 篣= 籇= 籛= 糚= 糱= 紴= 絏= 絾= 綥 = 綨 = 緤 = 緧 = 繂 = 繺= 纎= 罋= 罖= 罼= 羷= 羻= 翝= 聗= 聜= 肂= 肑= 胩= 腖= 腶= 膒= 膖= 膤= 臰 = 艐!= 艛"= 艝#= 艞$= 苬%= 苿&= 茖'= 茽(= 荿)= 莦*= 莬+= 莭,= 萚-= 萮.= 萰/= 蒳0= 蔩1= 蔾2= 蕇3= 蕒4= 薝5= 薩6= 藚7= 蘤8= 蘾9= 虴:= 虷;= 蚏<= 蚟== 蚮>= 蛂?= 蛯@= 蜌A= 蝒B= 蝵C= 螇D= 螌E= 螤F= 螸G= 蟦H= 蠗I= 袇J= 袙K= 袠L= 袣M= 裶N= 褘O= 褢P= 褭Q= 褯R= 襈S= 襺T= 襾U= 觗V= 觵W= 訏X= 訜Y= 詟Z= 誜[= 誢\= 誩]= 誷^= 諀_= 諆`= 諎a= 謸b= 謿c= 譑d= 譮e= 讁f= 讋g= 讟h= 诐i= 谉j= 豦k= 豵l= 貭m= 賘n= 贃o= 赥p= 趍q= 趤r= 趮s= 趰t= 跙u= 踗v= 踲w= 躈x= 軰y= 輄z= 輆{= 輠|= 輨}= 輩~= 轊= 迵= 邜= 郋= 郒= 郰= 郺= 鄿= 醎= 醩= 釛= 鈒= 鈪= 鈵= 鉔= 鉽= 銄= 銗= 銰= 銵= 鋡= 鋻= 錌= 鍦= 鎁= 鎼= 鐆= 鐢= 鐪= 鑉= 鑔= 鑵= 鑺= 閛= 闀= 隌= 雟= 雴= 霫= 靁= 靔= 靪= 頢= 頿= 顉= 顐= 顦= 颾= 餂= 餈= 餠= 餲= 餷= 饄= 饖= 饠= 馧= 馩= 馺= 駮= 騤= 騯= 騵= 髇= 髉= 髱= 鬉= 鬡= 鬰= 鬵= 魊= 魫= 魳= 魼= 鮁= 鮆= 鮦= 鮯= 鮳= 鮷= 鯅= 鯋= 鯐= 鯗= 鯞= 鯯= 鰐= 鰞= 鰡= 鱱= 鱾= 鴌= 鴍= 鴬= 鵇= 鵊= 鵧= 鵴= 鵻= 鶥= 鶧= 鶮= 鸃= 鸏= 麆= 麥= 黬= 鼶= 齳= 齽= 龏= ꇬ= 럭= 쁜= 턴= = 𖠚= 🌷= 🎭= 📞= 🖊= 🗺= 🤧= ƾ= ǵ= ѷ= Ҕ= Ҟ= ӓ= Ӧ> ֚> ؚ> ރ> ߝ> ߵ> ࠱> ࠲> ࣻ> ૽ > ଦ > ౯ > ౰ > ೲ > ബ> ൎ> ༻> ቱ> ።> ᑦ> ᛁ> Ả> Ἑ> ὄ> ≲> ┏> ⛱> ✥> 乢> 亣> 仾> 倃> 偢 > 偼!> 僃"> 僿#> 儓$> 儣%> 儭&> 冝'> 冡(> 冸)> 刔*> 剙+> 剨,> 劰-> 勏.> 勡/> 厠0> 咈1> 哰2> 唃3> 唌4> 唣5> 唹6> 喩7> 嗁8> 嗛9> 嘃:> 嘋;> 嘑<> 嘓=> 嘳>> 噞?> 嚵@> 坢A> 垊B> 埓C> 塇D> 塓E> 塥F> 墡G> 墥H> 墲I> 壥J> 夈K> 夻L> 奾M> 妽N> 姭O> 姾P> 娦Q> 娨R> 娽S> 婍T> 婐U> 婗V> 婾W> 媦X> 媩Y> 媹Z> 嫹[> 嬏\> 嬒]> 嬻^> 孾_> 宆`> 宒a> 尀b> 岋c> 峳d> 峷e> 峺f> 崒g> 崯h> 嵣i> 嵮j> 嶞k> 巄l> 巙m> 巠n> 巬o> 帓p> 帿q> 幉r> 幭s> 庈t> 庉u> 庡v> 庢w> 庣x> 廅y> 廇z> 徤{> 徫|> 怈}> 怟~> 恖> 悜> 愇> 愲> 慱> 憂> 憈> 憢> 憣> 憲> 懎> 懝> 戤> 扝> 扤> 捒> 摐> 摣> 摫> 撗> 撴> 擨> 攅> 攭> 敒> 斞> 旓> 旫> 晿> 暙> 暿> 曎> 曏> 曔> 曘> 曪> 朤> 朻> 杴> 桘> 桹> 梉> 梚> 梷> 棌> 椌> 椖> 椢> 椷> 榡> 榲> 槡> 槶> 樃> 樜> 橔> 檡> 檤> 欍> 欓> 欶> 毸> 毾> 氁> 汌> 沎> 泋> 浢> 淕> 淗> 渨> 湬> 溑> 滫> 漘> 漽> 澙> 澭> 澸> 濜> 濶> 瀔> 瀿> 灆> 灟> 灡> 灻> 炨> 烣> 焒> 焲> 煻> 爚> 爣> 犅> 犥> 犿> 猭> 獜> 瓭> 瓸> 甇> 甠> 甼> 畓> 畘> 畾> 疂> 疶> 痸> 瘱> 瘹> 癁> 癈> 癝> 癦> 皣> 皫> 盓> 眓> 眕> 瞜> 瞨> 矆> 矉> 砱> 硊> 硵> 碂> 碬? 碸? 礃? 礭? 祣? 祽? 禒? 秴? 穱? 竈 ? 竌 ? 竍 ? 竘 ? 筣 ? 筩? 箒? 篎? 篐? 篒? 篹? 簭? 簹? 籰? 粫? 糏? 糭? 絣? 絿? 綄? 綖? 綛? 緷? 缼 ? 翆!? 翜"? 耟#? 耣$? 肨%? 胒&? 胟'? 脴(? 腀)? 膙*? 膲+? 臎,? 臙-? 舋.? 艭/? 苮0? 茤1? 菒2? 菨3? 菺4? 萕5? 萭6? 蒵7? 蓈8? 蓕9? 蕌:? 蕜;? 蕟? 蕼?? 薭@? 藒A? 蘹B? 虣C? 蚆D? 蚒E? 蛃F? 蛠G? 蛣H? 蜼I? 蝄J? 蝏K? 蝖L? 蝛M? 螥N? 螮O? 螼P? 螾Q? 蟁R? 蟘S? 蠏T? 蠯U? 衕V? 袔W? 裐X? 褉Y? 褑Z? 褽[? 褾\? 襧]? 襴^? 覫_? 觮`? 觹a? 訋b? 訡c? 訦d? 詥e? 誔f? 誫g? 誺h? 誻i? 諃j? 諚k? 譄l? 譊m? 譤n? 讅o? 谹p? 豀q? 豘r? 貐s? 貑t? 貜u? 貦v? 跕w? 踆x? 踖y? 蹫z? 躝{? 躰|? 躴}? 軁~? 軂? 軇? 軦? 軬? 輏? 輧? 輴? 轙? 逈? 逫? 遦? 邎? 邒? 郌? 郖? 鄨? 酇? 醆? 醰? 銊? 銘? 鋴? 錓? 鍈? 鍨? 鍫? 鎍? 鎷? 鏏? 鐁? 鐄? 鐎? 鐯? 鑏? 鑖? 閕? 閗? 陚? 隑? 隝? 霴? 霷? 靅? 靏? 靧? 韟? 頄? 頮? 顩? 颫? 颹? 飜? 餀? 餁? 餢? 餫? 饘? 饳? 馚? 馾? 駋? 騡? 騩? 騪? 骔? 髍? 髳? 髷? 髺? 髼? 鬤? 魧? 魲? 鮅? 鮏? 鮜? 鮺? 鯑? 鯜? 鯥? 鯳? 鰔? 鰙? 鰫? 鰬? 鱮? 鳹? 鴊? 鴶? 鵐? 鵽? 鶢? 鶷? 鶾? 鷊? 鷖? 鷞? 鷩? 鷭? 鷱? 鸒? 鹽? 黺? 鼒? 鼨? 鼵? 鼺? 齄? 齅? 齨? 齶? ꅉ? ꔛ? 촬? 틴? 핀? ? ? ? ? ? 🏖? 🐰? 🐱? 😫? 🧘? ĥ? Ʀ? ƺ? ͷ@ Ϙ@ ӷ@ Ԓ@ ؃@ ؅@ ؠ@ ٜ@ ڔ@ ڛ @ ܆ @ ܏ @ ݑ @ ݛ @ ݵ@ ޡ@ ߺ@ ࠣ@ ࠾@ ࡎ@ ࡭@ ࡳ@ ࡺ@ ࢔@ ࢟@ ࣢@ ৑@ ৘@ ৙@ ੉@ ੊@ ੐@ ੵ @ ੺!@ ઩"@ ૎#@ ૕$@ ૟%@ ૠ&@ ૴'@ ૷(@ ଳ)@ ୔*@ ୙+@ ୚,@ ୽-@ ஡.@ ஧/@ ௉0@ ௮1@ ఙ2@ ఺3@ ౡ4@ ౾5@ ಙ6@ ೉7@ ೱ8@ ഄ9@ ഺ:@ ൅;@ ൩<@ ൵=@ ෠>@ ෰?@ ෸@@ ໊A@ ဝB@ ၀C@ ፡D@ ḪE@ ⃘F@ ➄G@ ➘H@ 丬I@ 亃J@ 亗K@ 伇L@ 伖M@ 佭N@ 侾O@ 俧P@ 偊Q@ 傟R@ 傡S@ 僋T@ 僪U@ 儎V@ 儏W@ 儾X@ 冎Y@ 凣Z@ 剆[@ 勨\@ 勭]@ 匘^@ 厊_@ 叞`@ 吂a@ 呄b@ 呇c@ 哱d@ 哶e@ 啢f@ 喥g@ 喭h@ 嗈i@ 嗺j@ 嘦k@ 嘷l@ 嚉m@ 嚰n@ 囒o@ 圿p@ 坄q@ 坓r@ 垏s@ 埅t@ 埛u@ 堛v@ 塿w@ 墢x@ 墷y@ 壦z@ 奿{@ 娂|@ 婨}@ 婩~@ 媊@ 媑@ 媨@ 媫@ 嫬@ 嬇@ 嬎@ 孉@ 宧@ 宲@ 宺@ 寗@ 寷@ 屔@ 屗@ 屟@ 岪@ 峆@ 峼@ 崘@ 崫@ 崱@ 崼@ 嵟@ 嵥@ 嵪@ 嶈@ 嶩@ 巆@ 巟@ 帇@ 帟@ 幱@ 庽@ 廏@ 弞@ 彁@ 彏@ 徦@ 徾@ 怑@ 怚@ 恱@ 悁@ 惾@ 愓@ 愯@ 慗@ 憯@ 懖@ 懯@ 扏@ 扲@ 抁@ 挄@ 捈@ 捖@ 捪@ 掕@ 揂@ 揝@ 搮@ 搰@ 搱@ 搳@ 摲@ 撎@ 攊@ 敾@ 斆@ 斱@ 旔@ 晇@ 晑@ 暭@ 暰@ 曗@ 曧@ 曭@ 朇@ 朲@ 杶@ 枂@ 枖@ 栍@ 栦@ 桍@ 梂@ 梐@ 梮@ 椉@ 椸@ 楉@ 榏@ 榳@ 榽@ 槝@ 槥@ 槾@ 樤@ 橤@ 檈@ 檘@ 櫀@ 欉@ 欼@ 歨@ 歭@ 毤@ 毪@ 氄@ 氎@ 汄@ 沍@ 沰@ 淢@ 渆@ 渢@ 渿@ 湏@ 湐@ 湚@ 湤@ 溎@ 溞@ 漞@ 漦@ 漴@ 潊@ 潒A 潗A 潳A 澅A 澏A 澖A 澢A 濷A 瀀A 瀸 A 灲 A 烌 A 烍 A 烥 A 焣A 煭A 熿A 燍A 燓A 燳A 牏A 犗A 犣A 犳A 犵A 犾A 狉A 猆A 猏A 猒A 猳A 獩A 玈 A 瓼!A 瓽"A 甉#A 畽$A 痵%A 瘜&A 癕'A 皩(A 盵)A 盽*A 眍+A 睈,A 瞕-A 瞗.A 矟/A 砙0A 硽1A 磌2A 磭3A 祻4A 禓5A 禝6A 禞7A 禥8A 禬9A 秅:A 秮;A 稇A 稵?A 窂@A 窏AA 竒BA 竕CA 竮DA 竸EA 笎FA 笜GA 筂HA 筟IA 筫JA 箟KA 箰LA 簓MA 簘NA 粂OA 粷PA 糐QA 糮RA 絤SA 綶TA 緽UA 縔VA 繫WA 纅XA 缹YA 罊ZA 罛[A 羀\A 翨]A 耓^A 耤_A 聏`A 聦aA 聬bA 肀cA 肍dA 肞eA 肻fA 胾gA 脈hA 脗iA 脭jA 腟kA 腳lA 膆mA 膇nA 膋oA 艀pA 艂qA 艈rA 艗sA 苭tA 茋uA 茡vA 菃wA 菣xA 葌yA 葠zA 葨{A 蒀|A 蒕}A 蓰~A 蔃A 蔒A 蕏A 蕫A 薡A 藊A 藑A 藭A 蘉A 蘙A 蘛A 虍A 蚉A 蚗A 蜯A 蝐A 蝑A 蝱A 螦A 蟭A 蠒A 衴A 衸A 衺A 衼A 袯A 裞A 裦A 褄A 襱A 覊A 覤A 覶A 覻A 觓A 觨A 觾A 訅A 訆A 訠A 詑A 誾A 諘A 諬A 謈A 謋A 謞A 謤A 謥A 謵A 譶A 讉A 谾A 豑A 豓A 豙A 賏A 贁A 贆A 趐A 跇A 跲A 踑A 蹅A 蹸A 躂A 躙A 躠A 軕A 軖A 軯A 軵A 輍A 轁A 迶A 逧A 遰A 邌A 邞A 郠A 鄅A 鄇A 醏A 醷A 鈗A 鈩A 鈼A 鉒A 鉝A 銍A 銴A 銺A 錔A 錼A 鍞A 鍧A 鍻A 鎑A 鎞A 鎥A 鎮A 鏥A 鏳A 鏿A 鑂A 鑆A 鑓A 鑕A 鑢A 钀A 钑A 镺A 閺A 陃A 陗A 陫A 陯A 陿A 隇A 霃A 霒A 霛A 霟A 靆A 靗A 靤A 靭A 鞂A 鞰A 韖B 韥B 頙B 顅B 顤B 颰B 飰B 餆B 餕B 餯 B 饁 B 饇 B 馡 B 馪 B 馼B 駀B 駤B 駥B 駨B 騝B 騴B 驝B 驲B 骮B 髛B 髲B 鬥B 鬭B 鬺B 魦B 鮕B 鮖B 鮘 B 鮧!B 鮩"B 鯏#B 鯝$B 鯹%B 鰖&B 鰜'B 鰯(B 鱋)B 鳴*B 鳸+B 鵃,B 鵗-B 鵭.B 鵹/B 鶔0B 鶭1B 鷁2B 鷌3B 鷎4B 鷑5B 鷛6B 鷽7B 鹴8B 麔9B 麙:B 黗;B 鼥B 齤?B 齭@B 쁘AB BB CB DB 🏏EB 🏘FB 🔮GB 😌HB 🤨IB 🦊JB 🦠KB ĈLB છMB ၊NB ᄉOB ዽPB ᑎQB ḇRB ἲSB ⇛TB ⌔UB ✻VB 㳇WB 伨XB 伹YB 佁ZB 偀[B 偂\B 偋]B 偛^B 傪_B 僁`B 僠aB 儝bB 兯cB 冓dB 冟eB 凂fB 刄gB 刟hB 剒iB 剭jB 剻kB 勂lB 勊mB 勯nB 匊oB 匒pB 卙qB 厹rB 叴sB 咺tB 哷uB 唥vB 唴wB 啎xB 啺yB 喗zB 喯{B 喴|B 嗊}B 嗋~B 噳B 嚍B 嚫B 囶B 圦B 圶B 垐B 垖B 埀B 埨B 埮B 埳B 埶B 壗B 壪B 夎B 奻B 姢B 娖B 媟B 媡B 媭B 嫎B 嫓B 嬣B 孍B 孨B 宯B 岮B 岰B 崜B 嵠B 嵧B 嶏B 嶬B 嶵B 巊B 巚B 帉B 帪B 幎B 庻B 廤B 弫B 弲B 恟B 惖B 惥B 愊B 愢B 愭B 愶B 慺B 懢B 懫B 懰B 戅B 戙B 戝B 戞B 戭B 抋B 抙B 拚B 拹B 挭B 掑B 掔B 揱B 搉B 摋B 摍B 摼B 擃B 擕B 攍B 攟B 敀B 斈B 斿B 昋B 昖B 晆B 晘B 晬B 暷B 曢B 朚B 朞B 杁B 杊B 杋B 杹B 枟B 柀B 栜B 梋B 梎B 梩B 棷B 棿B 楍B 楰B 楲B 楺B 楾B 榙B 榝B 榞B 榟B 榾B 槰B 樭B 樴B 橉B 橩B 櫡B 欜B 欪B 歒B 歞B 歵B 殶B 氊B 氱B 汷B 泿B 洓B 洜B 浕C 浶C 浻C 涍C 湈C 湹C 滜C 潶C 澒C 濹 C 瀄 C 瀇 C 瀗 C 灊 C 灳C 烕C 焧C 焪C 焽C 焾C 熀C 熓C 熴C 熶C 燑C 燩C 爊C 爥C 爮C 牆C 犌C 犓C 犚 C 猅!C 猌"C 猵#C 獇$C 獟%C 瓡&C 瓳'C 疈(C 疞)C 疷*C 瘮+C 瘻,C 癐-C 癳.C 皵/C 眗0C 眮1C 睖2C 睙3C 睻4C 矡5C 矲6C 矵7C 砞8C 硢9C 硳:C 硸;C 磛C 礠?C 祄@C 祑AC 禙BC 秂CC 秚DC 秿EC 稶FC 稸GC 稺HC 穦IC 穨JC 窇KC 竰LC 竷MC 笢NC 笿OC 筎PC 箽QC 篅RC 篬SC 簊TC 簎UC 籆VC 粁WC 粖XC 粶YC 糤ZC 糥[C 糷\C 紎]C 紨^C 絉_C 絗`C 絟aC 絭bC 絸cC 綕dC 緕eC 緪fC 縌gC 縘hC 縤iC 繨jC 纄kC 纋lC 纒mC 纼nC 绤oC 翓pC 聁qC 聐rC 聢sC 聨tC 脀uC 脋vC 脜wC 脝xC 脤yC 腇zC 腛{C 腯|C 膫}C 膬~C 膱C 膶C 膸C 臗C 臱C 舏C 舚C 艩C 芚C 芛C 苺C 荌C 菕C 菷C 萔C 萟C 葃C 葏C 葿C 蒖C 蒫C 蓌C 蓳C 蔍C 蔎C 蔨C 蔱C 蕭C 薒C 薚C 藢C 藬C 蘶C 虄C 虉C 蚫C 蛕C 蛘C 蛢C 蜋C 蝆C 蝫C 螙C 螜C 蟕C 蠫C 蠿C 袌C 袝C 袬C 裄C 裗C 褜C 褹C 襅C 襇C 襥C 覂C 覢C 覣C 觟C 觤C 訔C 訨C 訲C 誎C 誳C 誸C 諁C 諹C 諻C 謌C 譅C 譣C 譨C 讙C 豠C 貟C 貾C 賉C 賥C 贉C 贚C 赗C 趂C 趜C 趩C 踄C 踻C 蹏C 軪C 輂C 輑C 輤C 輫C 轕C 轠C 轷C 辤C 遱C 遳C 邚C 邤C 邧C 鄑C 酠C 醠C 釒C 釞C 釪C 鈋C 鉘C 鉟C 鉫C 鋍C 鋎C 鍁C 鍐C 鍑C 鍩C 鍮C 鍷C 鎙C 鎴C 鎺C 鏓C 鏠C 鏦C 鐝C 鑈D 鑙D 鑟D 閯D 閸D 陒D 隀D 隲D 雽D 霬 D 鞙 D 鞢 D 韨 D 颻 D 飇D 飬D 餗D 餪D 餭D 餰D 餶D 駈D 駍D 騂D 騽D 驦D 驧D 髨D 鬎D 鬳D 魒D 魗D 魙 D 魜!D 魣"D 魪#D 鮊$D 鮝%D 鯃&D 鯆'D 鰅(D 鰠)D 鰦*D 鰴+D 鱆,D 鱞-D 鱢.D 鲓/D 鴖0D 鴗1D 鴴2D 鵣3D 鵨4D 鶓5D 鶕6D 鷈7D 鷮8D 鸁9D 鸄:D 鸇;D 鸔D 鼛?D 齀@D 齈AD 齎BD 뛰CD 헌DD ED FD GD HD 🆚ID 🎞JD 💿KD 📘LD 😴MD ՑND ގOD ષPD ૂQD ጉRD ᥱSD ḻTD ὕUD ὢVD ⊢WD ⛈XD ⸚YD 〣ZD 仦[D 伌\D 伓]D 佖^D 侐_D 俲`D 倇aD 倹bD 偙cD 偩dD 傄eD 儜fD 儩gD 兾hD 凬iD 凱jD 凷kD 刜lD 剈mD 劕nD 劷oD 匓pD 匲qD 卥rD 卪sD 厀tD 厗uD 呚vD 呠wD 咰xD 哛yD 唗zD 啒{D 啹|D 啽}D 喅~D 噮D 嚻D 囼D 圔D 圵D 圽D 坧D 坿D 垼D 埩D 堫D 堬D 塨D 塪D 墏D 壝D 壭D 夘D 姂D 姴D 婃D 媰D 嬊D 嬱D 嬹D 嬽D 寈D 屆D 屽D 岅D 峅D 峌D 峑D 峾D 崲D 崵D 嵓D 庍D 庲D 彋D 彑D 彛D 忂D 忨D 忰D 怉D 怘D 怾D 恄D 恅D 恑D 悀D 悏D 悓D 慁D 慛D 慦D 慴D 憁D 憗D 憳D 憿D 懏D 懚D 懛D 懪D 戫D 挃D 挅D 挗D 挬D 捚D 捬D 捳D 捴D 揁D 揥D 揬D 揯D 搚D 擟D 攁D 攗D 攳D 敩D 敳D 敺D 敽D 斊D 斳D 旀D 旲D 暆D 曀D 曅D 曥D 曶D 朜D 枏D 枑D 柤D 栥D 桒D 棝D 楁D 榢D 槒D 槢D 樍D 樰D 樷D 橗D 橺D 檲D 檴D 櫅D 櫤D 櫭D 櫲D 櫼D 欄D 欮D 歁D 歈D 歑D 歾D 毜D 毢D 毩D 氒E 氠E 汢E 浌E 浝E 浫E 涏E 涗E 涰E 淎 E 淔 E 淾 E 渁 E 渄 E 湀E 湇E 湕E 湭E 溌E 溿E 滊E 滶E 潠E 澟E 瀢E 瀱E 烒E 烖E 烡E 烵E 焅E 煍E 熉 E 熑!E 燐"E 燗#E 爘$E 爫%E 牑&E 牨'E 犝(E 猂)E 猍*E 猟+E 猣,E 猰-E 獝.E 獞/E 瓥0E 甀1E 疓2E 痆3E 痋4E 痜5E 痬6E 癠7E 皀8E 皏9E 皢:E 皷;E 皼E 眝?E 睄@E 睭AE 睸BE 瞈CE 瞛DE 矌EE 矒FE 砏GE 硧HE 碩IE 磄JE 磆KE 磪LE 磮ME 礂NE 祡OE 秨PE 秷QE 穘RE 穥SE 窲TE 竧UE 笗VE 筰WE 箙XE 篂YE 篰ZE 簐[E 簚\E 簥]E 簲^E 籔_E 籚`E 粈aE 粣bE 紁cE 紌dE 紑eE 紶fE 絋gE 絓hE 綂iE 綤jE 緅kE 縅lE 纀mE 纃nE 纕oE 罎pE 羠qE 羦rE 羫sE 翋tE 翢uE 耾vE 聅wE 肙xE 胏yE 胑zE 胣{E 胮|E 胿}E 脡~E 脻E 腬E 膢E 臋E 臫E 舝E 艔E 茐E 荴E 莇E 莍E 菎E 萶E 葝E 蓘E 蔏E 蔯E 蔶E 蕂E 蕱E 藳E 蘔E 虠E 蚐E 蚙E 蚞E 蛓E 蛚E 蛜E 蜐E 蝋E 蝧E 螡E 蟇E 蟍E 蟷E 蟸E 蟼E 蠠E 蠽E 蠾E 袩E 袳E 袶E 裍E 褣E 襆E 襏E 襶E 襽E 覰E 覹E 觰E 觷E 詊E 誈E 諔E 謃E 謉E 謑E 謯E 豽E 貁E 賯E 贙E 趆E 趇E 趒E 趥E 趪E 趹E 跜E 踶E 躤E 躨E 躳E 軩E 輱E 轐E 轣E 迼E 逭E 逽E 遫E 郈E 鄡E 鄾E 醈E 醳E 釚E 鈛E 鉵E 銏E 鋠E 錣E 錪E 鍊E 鍏E 鎇E 鎎E 鎤E 鏂E 鏛E 鐛E 鐩E 鐬E 鐰E 鑎E 鑜E 鑥E 閙E 閮E 闒E 闗E 闠E 闦E 阫E 阺E 霣E 鞌E 鞓E 鞝E 韢E 韰E 頑E 頓E 顀E 颒E 飥E 飦F 餦F 餩F 饂F 饢F 馶F 馻F 駎F 駣F 駴 F 騠 F 驞 F 驠 F 骪 F 髐F 鬌F 鬑F 鬦F 魩F 魱F 鮥F 鯦F 鯶F 鰚F 鰢F 鱍F 鱚F 鱽F 鳦F 鴩F 鶅F 鶟F 鶤 F 鷏!F 鸋"F 麚#F 麠$F 黂%F 齓&F 齞'F 齩(F 龤)F 값*F 닐+F 똑,F 뢰-F 멋.F 엑/F 칼0F 켓1F 흔2F 3F 4F ￳5F 🇽6F 🌼7F 🥁8F 🦇9F ֆ:F ހ;F ൾF ᥬ?F Ệ@F ỡAF ἌBF ↕CF ⚛DF 乤EF 亄FF 伔GF 伡HF 侀IF 俇JF 俔KF 俠LF 俴MF 俼NF 倱OF 倿PF 偮QF 傓RF 儹SF 兞TF 凚UF 剼VF 劚WF 劧XF 勓YF 匇ZF 匶[F 匷\F 厑]F 厧^F 厫_F 厯`F 呍aF 呥bF 唕cF 唙dF 唜eF 啋fF 啘gF 喞hF 喢iF 嗂jF 噿kF 嚗lF 圎mF 坸nF 垰oF 埄pF 堣qF 塛rF 塳sF 墱tF 夒uF 妅vF 妛wF 嫛xF 孡yF 宩zF 寙{F 尃|F 尌}F 尦~F 岨F 岾F 崻F 嶑F 巶F 巺F 幊F 幓F 幠F 幩F 庌F 庱F 弌F 弙F 弝F 彍F 徔F 忚F 忢F 忲F 惎F 愸F 慙F 憰F 憻F 懙F 懜F 懠F 扷F 抍F 抶F 拻F 挀F 挰F 挳F 撀F 撶F 撽F 擥F 攚F 攺F 斦F 旍F 旞F 昅F 晎F 暛F 柇F 柶F 柼F 桪F 梌F 梕F 梙F 梞F 梡F 梻F 棸F 椗F 椫F 椺F 楆F 楏F 榋F 榣F 榰F 樇F 樧F 橳F 檍F 櫇F 櫵F 欂F 欯F 歝F 毟F 毭F 氉F 氋F 氻F 汣F 沊F 沜F 洬F 浂F 浖F 涖F 渱F 渷F 渻F 湢F 湪F 滛F 漅F 漵F 澃F 澻F 瀎F 瀤F 灁F 灔F 灱F 炢F 煗F 煘F 熢F 熮F 燇F 爃F 爅F 爞F 牣F 犙F 犞F 狜F 獂F 獊F 獌F 獔F 獘F 瓝F 甝F 畡F 畱F 疀F 疘F 痚F 皥F 皬F 瞂G 瞦G 矁G 矘G 矦G 砤G 硛G 碞G 磓G 磖 G 磸 G 礆 G 礗 G 礹 G 禉G 禗G 稘G 穙G 穚G 穞G 穬G 窱G 竉G 竛G 竡G 笧G 笶G 筗G 筿G 箂G 箈G 箲G 箹 G 篵!G 簙"G 簤#G 籸$G 粚%G 糛&G 絔'G 絠(G 絻)G 緛*G 縬+G 繣,G 纗-G 罓.G 羏/G 羳0G 翺1G 耊2G 聎3G 聑4G 肕5G 肣6G 胇7G 胋8G 胢9G 脕:G 脙;G 膟G 芁?G 茮@G 荕AG 莑BG 莵CG 葍DG 蒆EG 蒑FG 蓗GG 蔮HG 蕦IG 薎JG 藇KG 蘝LG 虇MG 虯NG 虸OG 蛷PG 蜖QG 蜝RG 蝝SG 蝺TG 螰UG 螴VG 蟎WG 蠌XG 袕YG 袡ZG 裓[G 裿\G 褌]G 褼^G 襛_G 覑`G 覞aG 觃bG 觻cG 訉dG 訒eG 訵fG 詄gG 詯hG 詻iG 諨jG 謏kG 謶lG 貆mG 貖nG 賦oG 賩pG 賭qG 贌rG 跅sG 跧tG 跴uG 蹗vG 躎wG 躖xG 躢yG 躵zG 躶{G 躸|G 軙}G 輡~G 輼G 辴G 迱G 逷G 遟G 邍G 邷G 郉G 郶G 鄓G 鄛G 鄻G 醕G 醼G 釂G 釠G 釰G 釻G 鈤G 鈨G 鉇G 鉎G 鉓G 鉪G 銿G 鋛G 鋞G 鋲G 錎G 錷G 鎕G 鎟G 鎠G 鎯G 鏀G 鏄G 鏆G 鏱G 鐍G 鑳G 锧G 镮G 閁G 閈G 閍G 闄G 阬G 雐G 雚G 雡G 雥G 雧G 靷G 鞀G 鞁G 鞟G 鞹G 鞼G 頋G 頺G 顜G 顣G 颵G 飉G 飤G 餝G 饙G 駲G 駺G 駻G 騐G 騥G 驌G 驖G 髩G 魞G 鮽G 鯂G 鰑G 鰮G 鱌G 鱜G 鱡G 鱴G 鳨G 鴧G 鴹G 鴼G 鴾G 鵆G 鵥G 鵳G 鶣G 鶽G 鷜G 鸖G 鹐G 麮G 麲G 黁G 黣G 鼁G 鼃G 鼈G 龎G 龝G ꄉG 맨G 삭G 짓G G G G 🍅G 🐘G 📒G 🥊G 🥮G ୩G ణG ಚG ഭG ഷG ഹG ༺G ဂG ሲG ዎG ᑕG ᒃH ᕗH ᠫH ỄH ◗H 乴H 乼H 亝H 仛H 伂 H 佋 H 侟 H 侻 H 俥 H 倊H 倳H 僴H 兪H 凕H 剱H 勜H 厈H 厸H 吺H 呏H 哻H 唒H 唽H 喠H 嗕H 嗗H 嘄H 噾 H 嚲!H 囦"H 坃#H 坒$H 坘%H 坺&H 垜'H 堜(H 堥)H 堶*H 堸+H 堾,H 塰-H 壨.H 夞/H 奜0H 妦1H 姃2H 娏3H 娫4H 媆5H 嫀6H 寉7H 尣8H 岟9H 峍:H 峖;H 峜H 嶫?H 帞@H 弬AH 弻BH 忣CH 怽DH 恆EH 悎FH 悐GH 惍HH 愝IH 愡JH 慐KH 憃LH 憕MH 憵NH 懻OH 戵PH 抈QH 抪RH 捼SH 掓TH 掗UH 揔VH 揤WH 摃XH 摨YH 撍ZH 擏[H 擙\H 攡]H 斚^H 斴_H 旤`H 晫aH 暒bH 曃cH 曍dH 朰eH 杫fH 枅gH 枛hH 枩iH 柋jH 栬kH 桳lH 棥mH 椈nH 椬oH 楧pH 榵qH 榸rH 槵sH 樿tH 橶uH 橾vH 檅wH 檒xH 檿yH 櫋zH 櫗{H 櫠|H 櫽}H 欎~H 歕H 殾H 毲H 毶H 氃H 氭H 汒H 汦H 泎H 泑H 泭H 洆H 洍H 洘H 浀H 浱H 涄H 涼H 溰H 滵H 漊H 漥H 潌H 潫H 潹H 澲H 濖H 濭H 濲H 瀈H 瀌H 瀥H 瀶H 灂H 灇H 灙H 灥H 炶H 烑H 煆H 煰H 煶H 煼H 熕H 燆H 燌H 燝H 燢H 燨H 燰H 燵H 爗H 爜H 猲H 獓H 獧H 獶H 瓲H 瓺H 瓾H 畒H 畣H 疅H 痗H 痮H 瘶H 癚H 皳H 皹H 盿H 眐H 眲H 眹H 瞖H 瞱H 矂H 矈H 矠H 砊H 砐H 砶H 硉H 硰H 碅H 磰H 禃H 稉H 稌H 稫H 稴H 穔H 穯H 窤H 窵H 竀H 竃H 笅H 笐H 箌H 篊H 簯H 粡H 糀H 糓H 糔H 糪H 糳H 紝H 絈H 絩H 緥H 緳H 縆H 縺H 繮H 繿H 罆H 罯H 羍H 羭H 翉H 耎H 耞H 耭H 聈H 聧H 聸H 脟H 腜H 腤I 臐I 臔I 臸I 艍I 苆I 苚I 茰I 莄I 莐 I 萐 I 萙 I 萡 I 葟 I 蓻I 蔐I 蔜I 蔤I 蔪I 蕠I 薉I 藖I 藾I 蘃I 蘒I 虭I 蚚I 蚸I 蚽I 蛵I 蜙I 蜤I 蜰 I 蜹!I 蝭"I 螆#I 螉$I 螒%I 螖&I 螘'I 蟨(I 蠦)I 蠩*I 衋+I 衳,I 袎-I 袦.I 袻/I 裠0I 裩1I 褖2I 褬3I 褮4I 襨5I 覜6I 覭7I 覼8I 覾9I 訟:I 訬;I 詀I 誀?I 誵@I 諿AI 謜BI 謽CI 譂DI 讇EI 讍FI 讘GI 豛HI 豤II 豾JI 貀KI 貇LI 貗MI 賋NI 趀OI 趗PI 趛QI 跦RI 踃SI 躟TI 躣UI 軘VI 軲WI 轞XI 辝YI 迌ZI 郔[I 鄎\I 鄷]I 酼^I 醓_I 釼`I 鉐aI 鉯bI 鋢cI 錂dI 錗eI 錥fI 錻gI 鎅hI 鎆iI 鎨jI 鎹kI 鏁lI 鏴mI 鑀nI 鑡oI 鑩pI 镾qI 閖rI 閵sI 闎tI 闛uI 闧vI 陻wI 隵xI 雬yI 霗zI 霢{I 霯|I 霳}I 靾~I 鞃I 韈I 韊I 韒I 韴I 韷I 頯I 顁I 顄I 顊I 飺I 餜I 餻I 駜I 驐I 驜I 骬I 骲I 骿I 鬗I 魤I 魺I 鮂I 鮌I 鯚I 鯬I 鯾I 鰻I 鱩I 鱰I 鳾I 鴐I 鵉I 鵘I 鵚I 鵦I 鶂I 鶡I 鶦I 鷅I 鷋I 鷐I 鷕I 鷝I 鷻I 麳I 黶I 鼀I 鼑I 龒I ꒱I 섭I 튜I I I I I 🏕I 💅I 💼I 🟨I 🤟I 🥘I 🧑I ĬI ՂI ՓI ։I ފI ঢI ਈI ਡI ૢI ଟI උI ᓐI ᚢI ᶍI ỊI ₺I ╳I ▦I ❿I ⩾I ⲏI ㄩI 丩I 乬I 乹I 亴I 佅I 俉I 倎I 偔I 傃I 傤I 傰I 傶I 儃I 儠I 儬I 凁I 剗I 劅I 卛I 厒I 厤I 厬I 叢I 呞I 呩I 哸I 唡I 唨I 啔I 嘗I 噄I 囮I 囻I 圤I 坾I 垑I 垪I 垬I 埐I 墂I 墯I 夁I 奞I 奱J 娧J 嫆J 嫍J 嫙J 嫺J 嬆J 嬞J 孻J 寍 J 尫 J 尭 J 尮 J 峓 J 崓J 嶦J 嶪J 嶮J 帴J 幍J 幧J 廟J 廥J 弖J 彮J 彴J 彸J 徲J 忥J 忶J 忼J 恞J 惂 J 惃!J 愂"J 愄#J 愌$J 慤%J 慩&J 憉'J 懧(J 戂)J 戠*J 扟+J 扻,J 抇-J 拞.J 捠/J 捦0J 揮1J 揷2J 斻3J 斾4J 旚5J 晭6J 晵7J 晹8J 暊9J 枮:J 柦;J 栠J 椩?J 楐@J 楘AJ 榒BJ 榹CJ 榺DJ 槀EJ 槖FJ 槷GJ 樮HJ 橠IJ 橯JJ 櫁KJ 櫄LJ 櫏MJ 櫫NJ 欱OJ 欳PJ 歫QJ 殝RJ 毈SJ 汳TJ 沞UJ 洕VJ 洠WJ 洯XJ 浿YJ 涃ZJ 涊[J 溛\J 溮]J 滭^J 滰_J 漌`J 潨aJ 濇bJ 瀃cJ 瀏dJ 瀽eJ 炠fJ 炩gJ 烐hJ 煟iJ 熎jJ 熧kJ 熩lJ 燛mJ 爌nJ 爎oJ 爧pJ 牎qJ 牚rJ 牶sJ 牼tJ 犫uJ 猽vJ 獀wJ 獑xJ 瓵yJ 畄zJ 疪{J 痭|J 瘑}J 瘒~J 瘬J 瘷J 癗J 皅J 眂J 眃J 眅J 眤J 睟J 睲J 睶J 瞔J 瞸J 砨J 硩J 硺J 碠J 磃J 礘J 礚J 礣J 礨J 祘J 祵J 秗J 稓J 穓J 穼J 窹J 竬J 竼J 篏J 篖J 篴J 簝J 籈J 籉J 籋J 籑J 籨J 糆J 絊J 絝J 絫J 絰J 緁J 緰J 縃J 纝J 罸J 翍J 翗J 翪J 耯J 聉J 肐J 肦J 肬J 脦J 脽J 膡J 臈J 臦J 艒J 艸J 艻J 芀J 苩J 荲J 莗J 菙J 菚J 萉J 萛J 萪J 葕J 葘J 葧J 葴J 葾J 蓙J 蓡J 蓹J 蕵J 薂J 薃J 薴J 藀J 藄J 藫J 藽J 蘦J 蘰J 虘J 蚦J 蚲J 蛗J 蛡J 蜔J 蜦J 蜧J 蜳J 蜶J 螝J 螿J 蠤J 蠮J 衁J 袐J 裫J 襋J 襓J 襣J 襸J 覉J 訰J 詏J 詠J 誏J 諩J 謴J 譍J 譼J 讈J 豏J 豩J 豱J 貏J 貕J 貢K 貫K 賶K 趈K 跈K 蹛K 躗K 躘K 軝K 軧 K 輐 K 輚 K 轖 K 逇 K 逳K 邼K 郆K 郮K 醖K 醟K 釴K 釽K 鈖K 鈲K 鉃K 鉖K 鉳K 銡K 銽K 銾K 鋿K 錑K 錰 K 鍣!K 鍭"K 鎄#K 鎾$K 鏅%K 鏲&K 鐀'K 鐣(K 鑇)K 鑻*K 閇+K 阧,K 隦-K 雵.K 霠/K 鞐0K 鞒1K 鞜2K 鞤3K 鞱4K 韏5K 韣6K 韹7K 頱8K 顂9K 颷:K 餇;K 饆K 騙?K 驆@K 驓AK 骳BK 鬁CK 鬅DK 鬔EK 鬕FK 魖GK 魢HK 鮻IK 鯩JK 鯺KK 鰀LK 鰧MK 鱏NK 鱦OK 鱫PK 鴁QK 鴘RK 鴤SK 鵤TK 鵷UK 鶈VK 鶑WK 鶨XK 鷪YK 鷫ZK 鷼[K 鸤\K 鹸]K 麉^K 麊_K 黭`K 黰aK 鼖bK 鼳cK 齆dK ꒰eK 뀐fK 냐gK 늦hK 웅iK 웠jK 찍kK 킬lK mK nK oK pK ︃qK ︊rK 💭sK 🔝tK 🙊uK 🥓vK 🧺wK ބxK ߦyK ಧzK ആ{K ႏ|K ሃ}K ሮ~K ᚨK ↷K ⇘K ⇥K ⌃K ⎟K ╲K ▉K ☷K ⻩K ㄠK ㄤK 䶮K 乊K 乣K 乥K 伄K 侌K 侳K 俽K 倐K 倯K 偯K 偹K 傛K 僐K 僷K 儦K 兙K 凾K 刧K 劘K 劶K 勬K 匢K 匧K 厁K 厏K 唊K 唖K 喕K 嘝K 噷K 囐K 圛K 垎K 垶K 埣K 塖K 夃K 夝K 妉K 姏K 姲K 嬔K 孠K 宑K 寭K 尡K 尲K 岤K 岥K 岶K 嵔K 嵨K 嵭K 嶜K 嶻K 帢K 幙K 幥K 庘K 庬K 廎K 弤K 弿K 彚K 彶K 忈K 怲K 恊K 恔K 悥K 悹K 惌K 懘K 戄K 扖K 挋K 揙K 搫K 摮K 撉K 擪K 擽K 攱K 敜K 敡K 斔K 斪K 旊K 昑K 昦K 朁K 朾K 杸K 柛K 柪K 桺K 棈K 棏K 椕K 榶K 槬K 槮K 槴K 樌K 橕K 橣K 橬K 橵K 檆K 櫎K 櫑K 櫒K 櫷K 欋K 欿K 殙K 毇L 毨L 浘L 浳L 涹L 淴L 湁L 湒L 潈L 澺 L 濕 L 濤 L 濽 L 灚 L 炥L 烢L 焩L 熁L 熐L 犜L 犻L 狢L 瓹L 甤L 畆L 痀L 痟L 皃L 皟L 盇L 眡L 眻L 矺 L 砆!L 硦"L 碋#L 磒$L 磼%L 礖&L 礥'L 祬(L 禲)L 稰*L 穭+L 窔,L 笚-L 笴.L 筃/L 籫0L 籯1L 絑2L 綗3L 綳4L 綷5L 緵6L 縓7L 縜8L 繀9L 罁:L 罻;L 羇L 翷?L 聄@L 聛AL 聣BL 胉CL 脎DL 脠EL 腁FL 腵GL 膎HL 膮IL 膷JL 臹KL 舙LL 艪ML 芺NL 茊OL 茦PL 荱QL 萒RL 萓SL 萷TL 蓎UL 蓵VL 蕀WL 蕄XL 藞YL 虲ZL 蚅[L 蛪\L 蜟]L 蜪^L 螕_L 蟂`L 蟏aL 蠥bL 蠴cL 衂dL 袏eL 袽fL 裃gL 褕hL 褤iL 褱jL 襊kL 襗lL 覗mL 覱nL 覴oL 觛pL 訙qL 詗rL 諊sL 謆tL 謷uL 謺vL 譈wL 譺xL 讄yL 讔zL 豒{L 豟|L 豶}L 貵~L 賆L 賝L 賫L 賰L 賲L 趎L 跔L 踋L 蹃L 軎L 軥L 輀L 轋L 辡L 辳L 迒L 逜L 郀L 郩L 鄍L 醀L 釯L 鈻L 鉙L 鉡L 銂L 銕L 鋉L 錖L 錴L 鍋L 鏾L 鐟L 鐹L 鑚L 閊L 闟L 阸L 隭L 霌L 霔L 鞉L 鞖L 鞛L 韺L 顇L 顋L 顬L 飁L 饝L 馲L 駏L 騈L 騛L 騞L 騻L 騿L 骭L 髤L 魀L 魥L 魹L 鮓L 鮲L 鱨L 鴭L 鵏L 鵸L 鶁L 鶴L 鶶L 鷇L 鸈L 麢L 麬L 黱L 鼘L 鼦L 鼿L ꀿL 갔L 괴L 룸L 빌L 촌L 혔L L L L L 🍰L 🎗L 🎹L 🐉L 🐟L 🕷L 😠L 🚧L 􏰆L ŲL ǯL ކL ਉL ೂL ↶L ⌁L ⌂L ⍢L ☍L ☭L ☲L ✾L ➗L ⤴L 丒L 乶L 乷L 乻L 乿L 仜L 佔L 侤L 倠L 偵L 傜L 傹L 傾L 僢L 僬L 兤M 冹M 凘M 剎M 剠M 劎M 劺M 勩M 匩M 厃 M 厞 M 唟 M 喡 M 嗏 M 嚖M 嚽M 坈M 坴M 垷M 埞M 堒M 塜M 塮M 墆M 壋M 妎M 嬼M 寋M 寱M 屷M 屻M 峊M 峚 M 峿!M 嵏"M 嶃#M 嶴$M 嶾%M 帄&M 幦'M 弣(M 怰)M 怷*M 恎+M 悋,M 惏-M 愪.M 慖/M 慠0M 懤1M 抮2M 拲3M 挱4M 捑5M 掜6M 揇7M 搃8M 搎9M 搼:M 摓;M 摕M 攠?M 斍@M 昗AM 晲BM 暺CM 曣DM 栕EM 栰FM 桰GM 桸HM 棦IM 棾JM 椧KM 楄LM 榐MM 榑NM 槂OM 樄PM 橃QM 檋RM 櫜SM 欕TM 歚UM 殅VM 殔WM 殥XM 毱YM 汱ZM 汼[M 沀\M 洔]M 涁^M 淧_M 渒`M 滼aM 漗bM 潚cM 濥dM 濸eM 瀖fM 炃gM 炌hM 烪iM 焑jM 焹kM 煝lM 熆mM 熇nM 燘oM 爤pM 牅qM 牉rM 犱sM 狣tM 猠uM 獛vM 瓫wM 甒xM 甖yM 痌zM 睮{M 瞃|M 瞊}M 矝~M 硥M 硹M 碄M 碖M 碮M 磥M 礇M 禈M 禪M 秢M 稕M 穖M 窡M 窼M 筁M 箉M 箛M 篕M 篠M 簬M 籗M 籡M 籿M 粠M 粯M 粵M 糂M 糣M 紽M 絖M 絘M 綐M 綩M 縼M 繐M 繓M 繵M 纑M 肎M 肔M 肧M 肳M 胦M 胲M 膞M 膠M 膥M 舃M 舓M 艓M 艜M 艵M 荶M 菗M 菦M 菶M 蒏M 蒦M 蒭M 蓞M 蔕M 蕛M 薐M 藋M 蘂M 蘍M 蘣M 蚃M 蚳M 蜛M 蜸M 螎M 蠉M 蠚M 衘M 衭M 袃M 袉M 袘M 褁M 褅M 襹M 覨M 訽M 諪M 諯M 謣M 譕M 譡M 譳M 讗M 谺M 豍M 豴M 貄M 貒M 貚M 貣M 赺M 趏M 跱M 蹳M 蹽M 蹾M 躌M 輵M 輽M 迻M 逘M 邆M 邥M 酀M 酜M 酧M 釲M 鈠M 銯M 銱M 鋂M 鋽M 鍢M 鎜M 鏕M 鏭M 鐶M 鑃M 鑮M 鑴M 鑸M 鑹N 镸N 镽N 閐N 閜N 閰N 隚N 雂N 雦N 雸 N 頉 N 頩 N 頳 N 飅 N 飊N 飹N 飻N 餄N 餹N 饏N 饻N 馛N 騇N 騣N 騼N 驈N 驨N 魛N 魬N 鮾N 鯕N 鯣N 鰁 N 鰇!N 鰌"N 鰏#N 鰺$N 鳵%N 鴔&N 鴪'N 鵂(N 鶀)N 鶛*N 鶞+N 鷀,N 鷠-N 鸧.N 鹷/N 黚0N 齛1N ꁧ2N 균3N 끄4N 쓸5N 잖6N 졸7N 퓨8N 9N :N ;N N 🍔?N 🍵@N 🎩AN 🏷BN 🤮CN 🥜DN 🥵EN ĢFN ĵGN ĸHN ɟIN ՌJN ܩKN ލLN ਇMN ఆNN ఎON ധPN ഫQN ഴRN චSN ኖTN ዕUN ួVN ⌠WN ▮XN ♖YN ♘ZN ✒[N ㄅ\N 仐]N 侸^N 俒_N 偅`N 傏aN 儨bN 儮cN 刉dN 刐eN 剶fN 勴gN 叏hN 呧iN 呭jN 啳kN 嗹lN 嚊mN 嚋nN 嚺oN 圅pN 坉qN 堚rN 塃sN 墤tN 墵uN 夗vN 婯wN 嬭xN 尳yN 岇zN 岼{N 嶘|N 嶯}N 巐~N 帤N 庎N 弡N 弰N 徚N 徰N 徶N 怋N 怢N 怭N 怮N 恛N 悙N 悞N 惷N 愋N 愒N 憆N 憖N 懗N 懬N 戃N 抦N 揊N 搙N 搷N 撨N 敆N 敋N 旇N 旕N 昁N 暟N 朆N 栨N 桏N 桗N 棭N 椘N 楖N 榯N 槄N 槆N 橁N 檂N 檺N 櫿N 欁N 欫N 歋N 毞N 毣N 毮N 汿N 泍N 洃N 洡N 浟N 涷N 渪N 湌N 潃N 澑N 濦N 灴N 焭N 煩N 燞N 燱N 爂N 爢N 牄N 猺N 甎N 疉N 瘽N 皌N 皽N 盋N 眔N 睷N 瞁N 矃N 磀N 磱N 礀N 禜N 禠N 穝N 穮N 笽N 筄N 箁N 篫N 簅N 簔N 籎N 粍N 粴N 紩N 綒N 綘N 緎N 緓N 緿N 縖N 縩N 縸N 繘N 繥N 羒N 聝N 肁N 脌N 腏N 膉N 膔N 臅N 臶N 艃N 艌N 菄N 萖N 萺N 葤N 蒒N 蒣N 蓩N 蕚N 藔O 藗O 藛O 虥O 蜄O 螐O 螹O 蠀O 蠰O 蠸 O 衠 O 袟 O 袧 O 袰 O 袸O 裵O 褟O 覙O 訤O 詇O 詓O 誯O 諣O 謕O 謲O 譠O 譧O 譻O 豅O 豝O 賱O 赲O 趡 O 趧!O 躼"O 軠#O 轏$O 辷%O 酙&O 釄'O 鉼(O 銢)O 鋖*O 鋩+O 錃,O 錭-O 鍖.O 鎲/O 鏒0O 鐷1O 鑁2O 闂3O 阢4O 阽5O 陎6O 霕7O 霱8O 霼9O 鞲:O 韚;O 頍O 顆?O 飀@O 飸AO 馠BO 駯CO 駶DO 駷EO 騀FO 騔GO 騦HO 鬙IO 鬠JO 鬴KO 魻LO 鮔MO 鮴NO 鯴OO 鱐PO 鱕QO 鳫RO 鳺SO 鴑TO 鴻UO 鵶VO 鷍WO 鷒XO 麫YO 鼮ZO 齃[O 齝\O 곤]O 낮^O 넓_O 맥`O 멈aO 첨bO 춰cO 폐dO eO fO gO hO iO 🅰jO 🍊kO 🎰lO 💝mO 📂nO 📦oO 📬pO 📻qO 🚚rO 🛈sO 🥝tO 🥢uO ǘvO ɓwO ʝxO ֔yO ײzO ٧{O ୟ|O ಣ}O എ~O ഒO ၾO ሊO ኤO ዜO ᶅO ẤO ẳO ╪O ▊O 㵲O 丠O 伅O 偨O 偳O 傠O 儤O 冔O 刏O 勀O 匰O 卆O 叿O 哴O 唘O 喒O 嗸O 坅O 埆O 堄O 堖O 奆O 嬩O 孯O 宊O 寪O 屰O 峕O 嵜O 嵡O 嶉O 庨O 徣O 怌O 怴O 恜O 悷O 惀O 惁O 惉O 憏O 懱O 撆O 擖O 敐O 敚O 敯O 敶O 斺O 晐O 朼O 枀O 柫O 棜O 棞O 楴O 樎O 樬O 檕O 櫌O 欙O 欛O 欝O 歶O 殈O 毻O 汓O 泴O 浽O 涜O 涺O 淣O 淿O 溝O 潱O 澾O 澿O 濚O 瀊O 灧O 烅O 焈O 焿O 爑O 犆O 犩O 猐O 猤O 獈O 瓪O 疨O 痐O 瘂O 瘃O 瘯O 皍O 皧O 盁O 盧O 眫O 睼O 瞴O 砎O 硣O 硱O 磩O 礔O 禐O 禯O 稦O 窽O 笓O 箃O 箶O 篶O 簨O 簺O 糃O 緂O 縨P 繍P 繎P 繛P 缿P 罧P 罬P 罿P 翴P 耡 P 腪 P 膍 P 舕 P 舭 P 芉P 茷P 荁P 莏P 萀P 葅P 葪P 蒝P 蒤P 蓤P 薞P 薶P 蘬P 虦P 蛁P 蛌P 蜵P 螱P 螶 P 蠈!P 蠜"P 衇#P 衦$P 袵%P 襐&P 覐'P 觢(P 訁)P 詋*P 諓+P 諙,P 譐-P 豻.P 貱/P 贈0P 趓1P 趘2P 踓3P 蹞4P 蹥5P 蹵6P 躿7P 軐8P 軴9P 辬:P 辸;P 迏

    P 邭?P 郬@P 鄟AP 酑BP 酨CP 酻DP 醦EP 醧FP 釖GP 鈓HP 鉲IP 鋑JP 錉KP 鍗LP 鎀MP 鎋NP 鏉OP 鐜PP 鐼QP 閚RP 閷SP 阓TP 陙UP 鞷VP 鞸WP 颭XP 餥YP 餧ZP 馜[P 馝\P 駳]P 骵^P 髗_P 髰`P 鬞aP 鮉bP 鮿cP 鯘dP 鱣eP 鱹fP 鳲gP 鴏hP 鴙iP 鵎jP 鵢kP 鶍lP 鶜mP 鶪nP 鷔oP 鷷pP 鷿qP 鹝rP 麁sP 黊tP 鼤uP 鼰vP ꀕwP ꑌxP ꙻyP 댓zP 멘{P 윗|P }P ~P P ︇P 🏒P 💧P 🔈P 🔷P 😵P 🥃P ȝP ȼP ˮP ԺP ՉP ׃P ٞP ٦P ફP ஏP ཻP ᆺP ቢP ះP ᵻP ἹP ⌨P ⍛P ␣P 乯P 伳P 侅P 侇P 俈P 俻P 倝P 凙P 凥P 凨P 刱P 刴P 劖P 厽P 坙P 垁P 塺P 墛P 壿P 夾P 嬘P 寁P 尵P 屓P 峹P 崺P 嵷P 嶡P 嶨P 嶭P 廀P 彂P 恇P 恗P 慸P 憘P 拁P 掋P 撲P 擶P 敮P 斖P 朄P 朠P 杢P 枽P 枾P 桋P 桞P 桭P 桻P 梺P 棟P 棤P 槗P 殏P 泀P 淵P 湗P 湽P 溔P 溣P 瀪P 灹P 烉P 煡P 煾P 熡P 爈P 爟P 爡P 犤P 狏P 狵P 甅P 瘔P 癴P 睂P 睔P 瞍P 瞙P 瞮P 瞶P 砕P 碐P 磎P 礋P 窙P 筡P 筬P 簵P 籄P 紷P 綀P 綔P 緐P 繬P 繱P 纖P 翐P 耉P 肗P 脺Q 膯Q 艡Q 茒Q 茙Q 茢Q 菆Q 蒓Q 蒘Q 藲 Q 蘥 Q 蘪 Q 蚢 Q 蛫 Q 蛶Q 襡Q 觠Q 觬Q 詂Q 誁Q 誛Q 諥Q 謍Q 谸Q 趨Q 踥Q 蹖Q 蹷Q 躩Q 軅Q 軉Q 逤Q 醊 Q 鈂!Q 鈫"Q 鉰#Q 銅$Q 銸%Q 鍀&Q 鍃'Q 鍸(Q 鎱)Q 鎸*Q 鑪+Q 钄,Q 闙-Q 陱.Q 隯/Q 雤0Q 雼1Q 靟2Q 鞽3Q 韎4Q 韗5Q 顃6Q 颲7Q 馞8Q 駌9Q 駚:Q 騌;Q 髵Q 鳰?Q 鴅@Q 鶃AQ 鶎BQ 鸎CQ 鼣DQ 걱EQ 껴FQ 꼬GQ 녁HQ 략IQ 룩JQ 쓴KQ 찌LQ MQ NQ OQ PQ QQ RQ ￱SQ 🍄TQ 🍑UQ 🏢VQ 🐕WQ 🔻XQ 🕓YQ 🗽ZQ 😹[Q 🤐\Q 🥴]Q 🧵^Q Ƭ_Q ȯ`Q ˒aQ ٨bQ ܰcQ ૧dQ ථeQ ᄃfQ ቤgQ ጵhQ ᑐiQ ᛏjQ ᠺkQ ẸlQ ⌇mQ ┓nQ ⸺oQ ㄓpQ 㮾qQ 丯rQ 僼sQ 儑tQ 償uQ 勈vQ 勮wQ 匯xQ 匵yQ 噝zQ 坰{Q 壉|Q 壧}Q 宱~Q 尟Q 峲Q 崰Q 嵞Q 廢Q 廽Q 彌Q 彥Q 徺Q 徻Q 忋Q 惈Q 懓Q 懹Q 戁Q 捀Q 撁Q 擜Q 暓Q 暬Q 曤Q 朩Q 棩Q 棳Q 椃Q 椲Q 椳Q 榌Q 槕Q 槸Q 樆Q 樠Q 湠Q 溂Q 滽Q 澊Q 澷Q 瀮Q 熻Q 爳Q 犑Q 犘Q 狊Q 狤Q 瘏Q 瘡Q 瞯Q 矖Q 礷Q 祍Q 祱Q 稛Q 竆Q 篛Q 篜Q 籅Q 糘Q 糡Q 糫Q 糼Q 綊Q 緫Q 罺Q 翤Q 耛Q 聥Q 胈Q 胔Q 膅Q 膐Q 臕Q 葀Q 蓴Q 虤Q 蛅Q 蜅Q 蝷Q 襰Q 襲Q 詙Q 詨Q 謀Q 譢Q 譸Q 讏Q 豿Q 赼Q 跉Q 蹠Q 軣Q 轀Q 轃Q 迧Q 遙Q 邔Q 酭Q 鈙Q 鋜Q 鐗Q 鑋Q 閠Q 阷Q 陏Q 雓Q 靃Q 韇Q 韯Q 頃Q 飃Q 馽Q 魐Q 鴄Q 鵫Q 鵿Q 鶆Q 鷧Q 鷬Q 鸍Q 麎Q 黋Q 럴Q 묵Q 샤Q 즉Q 즘Q 짧Q Q Q Q 🐴R 📕R 🔬R 😪R ʇR ʤR ԶR ީR ઓR ཽ R ቡ R ᐧ R ᕕ R ᚱ R ᚾR ⃞R ≳R ┳R ◒R ◻R ❇R ❕R ⲡR ㄕR 㖞R 丟R 併R 俿R 偘R 傆R 傝R 勆R 勵 R 喪!R 塵"R 塻#R 墍$R 墔%R 壔&R 壾'R 娛(R 孹)R 尗*R 岦+R 帍,R 幯-R 廮.R 怞/R 惸0R 憠1R 憱2R 搻3R 撦4R 擛5R 擺6R 杚7R 椨8R 槣9R 樶:R 橓;R 檌R 汮?R 溩@R 潥AR 燺BR 爙CR 牃DR 牊ER 牱FR 犎GR 狿HR 獋IR 瓧JR 甿KR 瘇LR 癄MR 盳NR 眱OR 矋PR 矏QR 矤RR 硑SR 磘TR 穀UR 糿VR 紟WR 紪XR 綍YR 綡ZR 縏[R 羪\R 翶]R 胐^R 脪_R 膁`R 艕aR 茲bR 茿cR 菐dR 菛eR 蒰fR 蔁gR 蔙hR 蔛iR 藰jR 虖kR 蛽lR 蝁mR 蝊nR 螊oR 螛pR 蟝qR 衯rR 裮sR 詅tR 謰uR 趽vR 跥wR 蹎xR 軞yR 軸zR 醻{R 鋔|R 鋥}R 鏎~R 鑨R 鑬R 鑶R 钂R 閪R 雺R 霮R 靍R 馤R 駦R 騲R 驡R 髥R 魠R 魶R 鮼R 鯓R 鱳R 鳼R 麅R ꧔R 끔R 덤R 멍R 샌R 섯R 죄R 캔R 텍R R R R 🆅R 🌮R 🍳R 🍸R 🏔R 🏰R 📄R 🔪R 🟢R 🤵R ŬR ȏR ՁR ٔR ଶR ఫR ཊR ኑR ፋR ᕱR ἦR ↡R ⇀R ┍R ┑R ╡R ◠R ❎R ➅R ⤷R 〼R ヅR ㄐR ㄡR 㿠R 乽R 仒R 債R 兺R 凒R 匌R 哅R 啂R 喸R 嚠R 壀R 孲R 寚R 岴R 嵳R 幒R 弴R 彃R 彙R 懸R 戨R 挜R 掃R 擆R 擴R 敼R 晉R 曂R 曆R 曊R 曑R 櫖R 歽R 毧R 浨R 淁R 漇R 澝R 濐R 灗R 牞R 猚R 獹R 痶R 皗R 矎R 磤R 秐R 竎R 篻R 粀R 紖R 紣R 綼R 翽R 腅R 臲R 蘱R 蚭R 蟺R 蟿R 衟R 褞S 襂S 觕S 誋S 誘S 軄S 迾S 遪S 鉶S 錏 S 鐴 S 閝 S 靇 S 鞪 S 頝S 騗S 騬S 鬱S 鸗S 麱S 鼏S 鼞S ꆹS 낙S 댄S 룹S 씀S 씩S 엠S 젝S 흰S S  S !S "S ︈#S 🆕$S 🍦%S 🏛&S 🐳'S 👹(S 💄)S 🖋*S 🙅+S 🛠,S 🛰-S 🥟.S ҫ/S ٥0S ށ1S ળ2S ଣ3S ହ4S ඒ5S ᗨ6S ថ7S Ạ8S ⏎9S ❔:S ➐;S ⟫S ㄉ?S 䃎@S 伋AS 凮BS 凲CS 噖DS 墿ES 寣FS 崏GS 嶺HS 嶿IS 巏JS 惵KS 愃LS 憑MS 挸NS 摢OS 撊PS 梥QS 橝RS 橫SS 橰TS 欥US 歱VS 浾WS 潐XS 濝YS 灮ZS 焤[S 犮\S 甔]S 甧^S 祪_S 穎`S 竩aS 箿bS 篽cS 篿dS 籠eS 糋fS 紗gS 綋hS 締iS 緩jS 縱kS 繉lS 翇mS 耺nS 臡oS 萞pS 薗qS 蘵rS 蘻sS 虃tS 蛧uS 螔vS 覍wS 觼xS 譿yS 趖zS 軒{S 遚|S 鄤}S 酓~S 釫S 鉂S 鋵S 鍾S 陠S 隁S 隬S 靦S 韕S 餺S 馸S 鬝S 鰲S 鱁S ꉌS ꌠS ꑳS 꺼S 랐S 봤S 젠S 짐S 칩S 힐S S S S S S S S ￶S 🆁S 🌵S 🍫S 🏋S 🏫S 🏵S 🔜S 🔽S 🕸S 🕹S 🚌S 🛑S 🛫S 🦉S 🦞S ŎS ɅS ϫS ێS ۝S ތS ઇS ઉS ొS ሽS ቲS ᑭS ᒪS ៅS ṿS ὡS ⃚S ⌀S ⍥S ⎡S ⏬S ✼S ➻S ゑS 侙S 偣S 傇S 嫿S 嬵S 嵐S 嶱S 弚S 徖S 徟S 忦S 忩S 扅S 揰S 搇S 敨S 曯S 朖S 樕S 樝S 櫔S 櫕S 湧S 灎S 炴S 煀S 猉S 皠S 睪S 碝S 碵S 腸S 臥S 葐S 蔝S 衑S 襀S 觲S 託S 謼S 趌S 軤S 輲S 逩S 釨S 鋾S 錹S 鎫S 雈S 頧S 騚S 鸓S 鼸S 齗S ꜰS 닉S 델S 빙S 빵T 옵T 융T 팩T T T 🅐T 🏙T 👮T 👾 T 🕰 T 🥧 T 🦌 T 🧤 T ƛT ҡT ܒT ಆT ዚT ᎬT ᐦT ᖘT ៍T ḗT ỚT ╧T ✵T ✷T ⟪T ぢT ヮT 兘T 塐 T 庿!T 楶"T 煫#T 熃$T 燂%T 爒&T 秼'T 穻(T 簄)T 簜*T 紻+T 綧,T 縎-T 聫.T 螩/T 蟚0T 蠇1T 詃2T 诇3T 豂4T 躍5T 醔6T 鑧7T ꐊ8T ꕥ9T 렬:T 빼;T 뼈T 옛?T 큐@T AT BT CT DT ET ￰FT 🏥GT 🐬HT 👂IT 👕JT 🕴KT 😑LT 🚲MT 🛩NT 🤚OT 🧜PT 􏰀QT 􏰈RT ˏST ΊTT ֒UT ܛVT ܠWT ܦXT ሜYT ቻZT Ꭿ[T ḱ\T ὼ]T ≀^T ⎕_T ⏲`T ┇aT ⲱbT ⻢cT 㞎dT 佦eT 勁fT 埪gT 壂hT 屫iT 摚jT 斬kT 梲lT 槉mT 樼nT 殘oT 淚pT 湂qT 漡rT 濾sT 牭tT 狇uT 甦vT 疛wT 箳xT 綜yT 藱zT 蚹{T 蝢|T 螲}T 豰~T 贜T 踁T 軓T 鈱T 鍟T 鎖T 顝T 饒T 馮T 魨T 鯲T 鰸T ꐥT 싫T 짝T T 🅱T 🆘T 🍩T 🍴T 🐊T 🐓T 👓T 🤎T 🧢T ʐT ڌT ܚT ৺T ਘT ଗT ଷT ೊT ഇT ༝T ᅡT ሎT ሞT ኝT ᎢT ᕐT ᙏT ₿T ℞T ◘T ◤T ⛄T ➟T ➶T ⻰T 〓T ㄖT ㄙT 䓣T 倆T 寵T 拋T 栭T 檱T 熸T 矙T 硴T 笡T 綱T 綺T 繞T 衶T 裷T 襢T 豼T 賊T 躮T 辭T 迉T 郹T 鄬T 鈌T 闆T 韀T 頌T 駕T 魾T ꔷT 갓T 롱T 멜T 몽T 밴T 븐T 쯤T 콩T 퇴T T T T T T T T T T T T 🌭T 🐯T 👰T 📮T 🔁T 🔨T 🤰T 🥗T 🧩T ǥT ǪT ʑT װT એT ళT ೈT ၂T ሌT ᑯT ᓄT ᗢT ḎT ỀT ỦT ⇖T ⇾T ➏U ➑U ㄌU ㄛU 凍U 匎U 噴U 彊U 曓U 梊 U 殼 U 潰 U 爺 U 獅 U 盜U 窶U 綾U 腗U 衈U 褺U 蹤U 鋒U 錫U 隥U 齡U ꀐU ꊿU ꎭU 겼U 곁U 릿U 뱀U 빅 U 즌!U 팡"U 흑#U $U %U &U 'U 🍹(U 🏸)U 👭*U 💲+U 📎,U 😶-U 😺.U 🤢/U 🦷0U 𠃊1U Ĥ2U Ļ3U ɷ4U Ή5U ֗6U ڪ7U ਧ8U ੜ9U ༒:U ሩ;U ቁU ឹ?U ៃ@U ៖AU ᶘBU ἉCU ἠDU ἤEU ὗFU ≻GU ◅HU ☦IU ⛩JU ➆KU ➼LU ⟹MU ⵉNU ⸮OU 〢PU ぅQU ぉRU ゐSU 嗭TU 塗UU 廈VU 廚WU 樑XU 樸YU 澵ZU 瘋[U 緻\U 聰]U 膩^U 蘁_U 謙`U 輛aU 辯bU 遷cU 鄧dU 飽eU ꀊfU ꆎgU 갤hU 겪iU 낄jU 늄kU 닫lU 됨mU 촉nU 톡oU pU qU rU sU tU uU 🌕vU 🐈wU 👗xU 👠yU 📉zU 🔼{U 😓|U 🚘}U 🟠~U 🤏U 🤛U 🥒U 🥦U 🥲U 🦈U 🧁U ƴU ɱU ސU ޮU ଅU ୋU ഡU ൈU ྆U ᛖU ᛚU U ⎣U ╍U ◖U ☇U ☶U ⛅U ➒U ⻚U ⻣U 偽U 刪U 惽U 搖U 毷U 渦U 爛U 譽U 豬U 遲U 鏈U 閃U 鵋U 긍U 깃U 꽉U 꿀U 낭U 덮U 돕U 듀U 샘U 셜U 엽U 잃U 퀴U 힌U U U U U U U U U 🅝U 🌀U 🌠U 🌡U 🌪U 🎻U 🏉U 🏎U 👯U 🙆U 🤒U 🥖U 🥞U 🥥U ƆU ǦU ȳU ˓U ҝU ګU ڮU ܟU ܶU ૩U ఉU ಎU ຼU ྃU ဇU ჰU ቦU ᓇU ᓯU ᔭU ṉU ὉU ὖU ↬U ┣U ┮U ╤U ⛺U ✢U ㄊU ㄗU 㸆U 䃼U 儘U 儲U 劑U 唸U 暢U 椦U 櫦U 毬U 瀩U 瑪U 禎U 緯U 繈U 罷U 貧U 閑U 騰U 騺U 鴨U 곧V 꼽V 댑V 륭V 멸V 묘V 뷔V 셀V 옆V 윈 V 잭 V 쳤 V 펴 V 훌 V V V V V V V V V V V V V V 🍭V 🎖V 🎫V 💑V 🕶V 🗨 V 😨!V 😼"V 🙀#V ͢$V ױ%V ݁&V ࣰ'V ॱ(V ఇ)V ೃ*V භ+V ༔,V ဗ-V ዶ.V ፣/V Ꮇ0V ᕙ1V Ầ2V ⇝3V ⊞4V ⋙5V ⌈6V ⎠7V ⎢8V ⏸9V ┭:V ◓;V ◢V ✠?V ❰@V ⨯AV ⻉BV 㡳CV 㨃DV 妝EV 嶼FV 捨GV 擾HV 槍IV 漁JV 濁KV 祿LV 綿MV 贏NV 醬OV 釣PV 鋪QV 錶RV 鑒SV ꉪTV ꙪUV ꜥVV 꾼WV 냉XV 뇌YV 땅ZV 뚫[V 랫\V 앤]V 쥐^V 탐_V 헬`V aV bV cV dV eV fV gV hV iV jV kV ︍lV 🎲mV 🐎nV 🐖oV 🐠pV 🐷qV 🔺rV 🛡sV 🤸tV 🧂uV ƑvV ȥwV ˄xV ՖyV ՛zV ঝ{V ෑ|V ཎ}V ሙ~V ሬV ጅV ጭV ᎠV ᐤV ៉V ἁV ἅV ₲V ⇤V ⌟V ⎝V ▕V ⚇V ⛵V ⛷V ✜V ✸V ❩V ⲭV ㄔV 吶V 帥V 悶V 暫V 桿V 牽V 獸V 砲V 翹V 腎V 菴V 謹V 譚V 貪V 賃V 贊V 鄰V 銃V 鑽V 驅V ꆺV ꍟV ꙼V ꜱV 닷V 듬V 렵V 맡V 멤V 씬V 쿵V V V V V V V V V V V V V V V ﴾V ￿V 🇶V 🍌V 🎆V 🏊V 🐧V 👿V 🔋V 🔐V 🖌V 😣V 🚎V 🛵V 🟡V 🤜V 🤲V 🤴V 🥨V 🧬V 𠈌V 𦰡V ŅV ŔV ǖV ɝV ʅV ֛V ڤV ਫV ੧V ଙV ଧV ୬V ಇV ഥV ༵V ဳV ၁V ᅢV ኮV ጀV ᎪV ᓫV ᔪV ᚦV ᛋV ផV ṈV ẬV ỪV ἩV ↺V ≺V ⊤V ⌵V ⏪V ╕V ╢V ◥V ☣V ⛑V ➇V ⥨W ⵜW 〤W 〽W 㙟W 孃W 撐W 毀W 渾W 滄 W 珮 W 箇 W 褲 W 諦 W 謎W 賜W 賤W 踐W 迆W 鍛W 餵W ꈓW ꐛW 겐W 끊W 맹W 밍W 샷W 컷W 톤W 핫W W  W !W "W #W $W %W &W 'W (W ￸)W 🅳*W 🍼+W 🏄,W 🏍-W 🏞.W 🐑/W 🐙0W 👐1W 🔟2W 😰3W 🚴4W 🥔5W 🧍6W 🫶7W Ʒ8W ȧ9W ɘ:W ˟;W ͵W ՚?W ٓ@W ٬AW ٽBW ځCW ܥDW ॒EW ৱFW ଭGW ၏HW ኒIW ኩJW ዱKW ᎡLW ᎩMW ᎽNW ᎾOW ᏚPW ᒋQW ᕑRW ᖗSW ᛅTW ṓUW ỨVW ὈWW ὥXW ᾰYW ᾳZW ≋[W ⎦\W ⚶]W ⛓^W ❨_W ⩽`W ⬤aW ㄋbW ㄜcW ㄞdW 㻗eW 壺fW 峽gW 巖hW 悅iW 欽jW 毌kW 畬lW 祦mW 籌nW 縫oW 纏pW 罰qW 罵rW 蝕sW 蝦tW 蠻uW 諮vW 趕wW 駒xW 駿yW 鱗zW ꃀ{W ꈊ|W ꉘ}W ꙰~W 곱W 렛W 룰W 밸W 뻐W 잎W 젤W 콤W 탬W 햇W W W W W W W W W W W W W ︁W 🅨W 🅶W 🌽W 🍏W 🍓W 🍝W 🏓W 🐇W 🐢W 🐸W 👥W 👫W 💊W 🔄W 🔖W 🕒W 😖W 😸W 🚢W 🥙W 🥬W 🦎W 🦖W ŊW ƸW ȩW ȻW ҲW ٗW ټW ۓW ॑W ૫W ଂW ఖW ഖW ຟW ຣW ༀW ༓W ࿚W ሸW ቋW ቴW ᏂW ᐱW ᓕW ᗣW ᴊW ṮW ἵW ₮W ⇃W ⇗W ⌉W ⌜W ⎛W ⎞W ␓W ┝W ╞W ╨W ╫W ▱W ⚒W ⚰W ⛏W ❡W ⬥W ⵏW ⷪW ⿰W ヂW ㄎW ㄝW 䓯W 僱W 奮W 攜W 滾W 濱W 訶W 詰W 豔W 霧W 韋W 鴉W ꀒW ꌒW ꏀW ꐙW 겹W 껍W 덜W 듭W 맑W 숫W 얘W 쩌W 틸W 픔W 흐W 흙W W W W X X X X X X X X X  X  X  X ﴿ X ﷯ X 🌅X 🍇X 🐫X 🐿X 👔X 👟X 📡X 🛋X 🛸X 🡆X 🧈X ŶX ǹX ǿX ɻX ˅X ͩX ϟX ܨ X ৹!X ઈ"X ଚ#X ஃ$X ᄁ%X ᅩ&X ሔ'X ኔ(X ኦ)X Ꮃ*X Ꮋ+X Ꮧ,X ᔨ-X ᕆ.X ᗤ/X ឆ0X Ḍ1X ἂ2X ⁂3X ≐4X ⊑5X ⍨6X ➓7X ⠂8X ⧫9X ⫸:X ⭣;X ⲕX 䭹?X 淺@X 澱AX 睏BX 賈CX 賠DX 軌EX 銳FX 靚GX 驟HX ꄚIX ꇉJX ꑵKX 겸LX 떼MX 랭NX 빗OX 삽PX 슷QX 펼RX 폼SX TX UX VX WX XX YX ZX [X \X ]X ^X _X 﨑`X ￵aX 🌌bX 🌾cX 🎒dX 🏌eX 🐞fX 🐨gX 🐮hX 👵iX 🕡jX 😟kX 🚙lX 🚶mX 🦢nX 🧙oX 􏰖pX ϳqX ҰrX ҶsX ӜtX ӯuX ٪vX ܓwX ޔxX ๛yX ༘zX ဟ{X ᅦ|X ሥ}X ቹ~X ዛX ጻX ᒡX ᒥX ᝰX ៌X ᠍X ᴢX ḿX ỔX ὔX ῬX ⁕X ∊X ⏫X ⏺X ␀X ┊X ▚X ☴X ⛸X ⧉X ⧼X ⧽X ⮞X ⵔX ⿱X 㸝X 喬X 奪X 嶽X 憐X 捲X 旂X 氾X 洩X 漲X 漿X 爐X 竊X 縛X 臟X 頸X 駛X ꄮX ꆪX ꐤX ꞉X 납X 눌X 닛X 뒷X 릇X 릎X 맙X 묻X 벳X 벼X 챗X 컵X 쾌X 펀X 홉X X X X X X X X X X X X X X X ￷X 🌋X 🌑X 🌯X 🍞X 🍯X 🐚X 🐼X 👅X 👙X 📋X 🕑X 😚X 🙇X 🚑X 🚕X 🛌X 🥐X 🦂X 🦪X 🧊X 🧟X 🧣X 🧥X 􏰂X 􏰊X ĜX ŜX ƞX ƳX ʞX ˹X ϛX ϣX ׇX ڕX ܺX ੴX ଏX ఈX ృX ൢX ဏX ሀX ሱX ዉX ዩX ጽX ᎥX ᎦX ᏢX ᐉX ᛒY ៀY ។Y ១Y ỰY ἍY ₯Y ⃜Y ⍝Y ▋ Y ◣ Y ☤ Y ☳ Y ⚧ Y ⺣Y ⻜Y ⻬Y ㄘY 䳭Y 厭Y 喚Y 嚇Y 懶Y 敘Y 曬Y 樺Y 櫃Y 櫚Y 汎Y 滲Y 燦Y 瑋Y 磯 Y 簾!Y 繳"Y 釘#Y 頒$Y 頤%Y 鵬&Y 鷹'Y 龐(Y ꂵ)Y ꈎ*Y ꌩ+Y ꐦ,Y ꐮ-Y ꐯ.Y 굽/Y 냅0Y 넥1Y 둑2Y 랬3Y 렘4Y 믹5Y 뱅6Y 뻔7Y 셉8Y 숲9Y 얀:Y 옮;Y 읍Y 켰?Y 쿨@Y 팝AY 헛BY CY DY EY FY GY HY IY JY KY LY MY NY OY PY QY RY SY TY UY VY WY ￴XY 𓈒YY 🆙ZY 🎢[Y 🎣\Y 🎪]Y 🎱^Y 👄_Y 💟`Y 💷aY 📀bY 📜cY 📨dY 🔌eY 🥅fY 🥪gY 🧭hY 🩸iY 𣎴jY 𬌗kY 􀳫lY ŧmY ƁnY ƣoY ɠpY ΏqY ϩrY ϻsY ҙtY ҧuY ԜvY ֤wY ޭxY ઝyY ૨zY ୫{Y ୭|Y ஈ}Y ಖ~Y ഉY ൃY ඨY ༊Y ᄎY ᅚY ሓY ጫY ፀY ᏁY ᏍY ᛉY ỒY ἕY ὠY ⁜Y ₧Y ↤Y ⇲Y ∟Y ⊖Y ⌐Y ⍃Y ⎥Y ⏭Y ┋Y ☬Y ♗Y ♾Y ⬡Y ⬩Y ⸱Y 㘵Y 䄂Y 䌷Y 䍸Y 䜌Y 僑Y 堯Y 廂Y 彿Y 灘Y 煉Y 煥Y 瑩Y 痠Y 瞼Y 篤Y 腫Y 臘Y 蔥Y 蘊Y 邁Y 邏Y 釦Y 頗Y 顛Y 餃Y 馴Y 鳩Y ꎷY ꐪY 겉Y 놈Y 뇨Y 닙Y 닭Y 닮Y 닿Y 듈Y 뭔Y 뵈Y 붕Y 섹Y 씻Y 앵Y 줬Y 칸Y 캬Y 탭Y 탱Y 팟Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y 𓂃Y 🅢Y 🆄Y 🍆Y 🍍Y 🍜Y 🐒Y 💳Y 📏Y 🔙Y 🔰Y 🗡Y 🚂Y 🚦Y 🥋Y 🦸Y 􏰉Y ƊY ƵY ǻY ȇY ϜY ѳY ѺY ӑY ӕY ږY ܽY ঐY ੌY ୮Y ఏY ౨Z ཥZ ྉZ ྥZ ჟZ ᄄZ ᅥZ ቆZ ዴZ Ꭷ Z Ꮳ Z ᑲ Z ᒍ Z ᒻ Z ᓴZ ᗩZ ២Z ᯤZ ḞZ ὙZ ↟Z ↾Z ≒Z ⊘Z ⋀Z ⍩Z ❙Z ⟱Z ㄆZ 㭎Z 㾀Z 勸Z 壯 Z 嬌!Z 撫"Z 楓#Z 湊$Z 漬%Z 瑤&Z 癥'Z 癮(Z 糾)Z 繩*Z 脅+Z 艱,Z 艷-Z 荊.Z 豈/Z 鈞0Z 銜1Z 飄2Z 飼3Z 餡4Z 鯨5Z 鹼6Z ꁏ7Z ꅽ8Z ꈴ9Z ꉈ:Z ꋌ;Z ꍏZ 깜?Z 댈@Z 둔AZ 딜BZ 땐CZ 렀DZ 렐EZ 롬FZ 륙GZ 맟HZ 밌IZ 밑JZ 볏KZ 볶LZ 봅MZ 쁨NZ 쌍OZ 앰PZ 좁QZ 짠RZ 퀸SZ 펙TZ 푼UZ 헷VZ 흘WZ XZ YZ ZZ [Z \Z ]Z ^Z _Z `Z aZ bZ cZ dZ eZ fZ gZ hZ iZ jZ kZ lZ mZ nZ oZ pZ qZ ︅rZ ︌sZ 🅘tZ 🌖uZ 🌨vZ 🎺wZ 🏺xZ 🐅yZ 🐛zZ 📗{Z 🔉|Z 🔧}Z 😧~Z 🙉Z 🛣Z 🦏Z 🦓Z 🧛Z 🧿Z 𡃁Z 󰆅Z ƚZ ƹZ ȁZ ϯZ ֧Z ֨Z ڀZ ډZ ڊZ ڵZ ڻZ ۍZ ۤZ ۬Z ޙZ ऽZ ਐZ ਯZ ଖZ ୯Z ຝZ ໒Z ༚Z ဥZ ᇂZ ሐZ ሻZ ኳZ ጨZ ጸZ ፅZ ፊZ ᏔZ ᓗZ ឡZ ḺZ ẕZ ẞZ ỞZ ↼Z ⋞Z ⋟Z ⌝Z ⍴Z ⎙Z ⎤Z ╜Z ☧Z ☵Z ♟Z ⚝Z ⛲Z ⵈZ ⵍZ ⵢZ 㙍Z 勣Z 唄Z 嘆Z 嘯Z 囉Z 崙Z 廁Z 徬Z 懼Z 棧Z 棲Z 檳Z 淪Z 牴Z 狹Z 瓊Z 礙Z 禦Z 糧Z 緋Z 繡Z 蘆Z 輿Z 遼Z 醜Z 閤Z 闢Z 颳Z 飢Z 餓Z 馳Z 騷Z 鯉Z 鵝Z 鸞Z ꃰZ ꄸZ ꆈZ ꇤZ ꈀZ ꉢZ ꌗZ ꑍZ ꟷZ ꫞Z 굿Z 귄Z 꿔Z 맴Z 쇠Z 싹Z 엉Z 왼Z 잇Z 쥬Z 찔Z 퀘Z 흄Z Z Z Z Z Z Z Z [ [ [ [ [ [ [ [ [  [  [  [  [  [ [ [ [ [ [ 𐂂[ 𓏸[ 🅔[ 🅖[ 🅛[ 🅿[ 🌛[ 🌫[ 🍐[ 🍣[ 🍲[ 🎡[ 🎷[ 🏦 [ 🏭![ 🐄"[ 🐋#[ 🐔$[ 👪%[ 👱&[ 💾'[ 📁([ 📙)[ 📼*[ 🖨+[ 😗,[ 🚣-[ 🤼.[ 🥎/[ 🥭0[ 🦍1[ 🦶2[ 🫁3[ 􀁸4[ 􏰎5[ Ĵ6[ Ɣ7[ Ɵ8[ ǟ9[ Ƚ:[ ʦ;[ Ϊ<[ ϥ=[ ϧ>[ Ϯ?[ Ғ@[ ڟA[ ڠB[ ۢC[ ޖD[ ऱE[ ଛF[ ఠG[ ಉH[ ഓI[ ඹJ[ ༉K[ ሄL[ ሠM[ ሶN[ ቕO[ ዙP[ ጃQ[ ᎵR[ ᏒS[ ᏞT[ ᖃU[ ᜔V[ ៊W[ ᥫX[ ᩣY[ ᮩZ[ ᷆[[ Ἔ\[ ἥ][ ὤ^[ ↞_[ ↥`[ ⇚a[ ⇞b[ ⇰c[ ⇱d[ ∻e[ ⋰f[ ⌑g[ ┠h[ ╂i[ ╵j[ ☓k[ ♜l[ ⚚m[ ➬n[ ➾o[ ⦾p[ ⴷq[ 㗲r[ 㣵s[ 㧾t[ 㳜u[ 㳨v[ 㸔w[ 䟌x[ 䠧y[ 䫌z[ 侶{[ 勗|[ 喲}[ 壘~[ 彎[ 擠[ 柵[ 樞[ 檸[ 殤[ 浬[ 畝[ 羨[ 蕩[ 薔[ 詮[ 貯[ 貶[ 鎂[ 闊[ 隷[ 霑[ 龜[ ꈨ[ ꋒ[ ꐚ[ ꑋ[ 꽂[ 끗[ 늑[ 둠[ 딘[ 뚜[ 띄[ 띠[ 릉[ 맵[ 밭[ 쪼[ 챙[ 튬[ 팁[ 헮[ 헲[ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [ ︂[ ￲[ 𐌰[ 🆈[ 🌚[ 🍉[ 🍬[ 🏇[ 🏚[ 🐥[ 👣[ 👴[ 💂[ 💤[ 🔦[ 🔫[ 🔱[ 🚜[ 🢖[ 🥍[ 🦔[ 🦕[ 🦙[ 🦜[ 🦵[ 🧞[ 🧦[ 🧪[ 🧱[ 🩳[ ɿ[ ˬ[ ˳[ ͻ[ Ͽ[ Ѩ[ ܙ[ ޝ[ ޞ[ ਏ[ ୱ[ ౌ[ ໆ[ ཌ[ ၥ[ ኞ[ ጓ[ Ꮹ[ Ꮿ[ Ᏼ[ ᐁ[ ᕦ\ ᛞ\ ឋ\ ᯅ\ ᷇\ ḽ\ ṡ\ ἢ\ Ἠ\ ₣ \ ∖ \ ⊳ \ ⊶ \ ⋱\ ⓫\ ┕\ ┨\ ◃\ ☱\ ♆\ ♰\ ➴\ ⠑\ ⣠\ ⨁\ ⻨\ ヵ\ 㒸\ 㓎\ 㕑\ 㞞\ 㟽 \ 㧟!\ 㼆"\ 䁔#\ 䟺$\ 䴗%\ 傘&\ 儷'\ 厲(\ 墜)\ 彫*\ 掙+\ 疊,\ 穌-\ 窩.\ 竄/\ 篩0\ 籥1\ 繆2\ 莖3\ 蔭4\ 螢5\ 覓6\ 詐7\ 誇8\ 鈣9\ 鉤:\ 鑄;\ 隸<\ 顎=\ 龔>\ ꂷ?\ ꄲ@\ ꆳA\ ꇑB\ ꉙC\ ꊰD\ ꋨE\ ꐰF\ ꑓG\ ꫀH\ 껏I\ 꽁J\ 돋K\ 랩L\ 붉M\ 붓N\ 샀O\ 쉴P\ 싼Q\ 쏟R\ 얄S\ 쭉T\ 켄U\ 킷V\ 턱W\ 펠X\ 펭Y\ 헝Z\ 헶[\ 헻\\ 훨]\ ^\ _\ `\ a\ b\ c\ d\ e\ f\ g\ h\ i\ j\ k\ l\ m\ n\ o\ p\ q\ r\ s\ t\ u\ v\ w\ x\ y\ z\ {\ |\ }\ ~\ \ \ \ ￾\ 𑐳\ 𑐶\ 𑐸\ 𓀠\ 𓂂\ 🃏\ 🅞\ 🌩\ 🏗\ 🐭\ 💶\ 📶\ 🔯\ 🚁\ 🚔\ 🤱\ 🤹\ 🤺\ 🧼\ 🧽\ 🩺\ 􏰃\ 􏰏\ Ǽ\ Ɇ\ ˨\ Ѡ\ ҥ\ ՜\ ׀\ ٖ\ ڧ\ ॆ\ ঋ\ ૮\ ଁ\ ஓ\ ಈ\ ೌ\ ೭\ ඡ\ ၅\ ၆\ ႈ\ ᄍ\ ᄐ\ ᅣ\ ሴ\ ዑ\ ጂ\ ጊ\ ᐠ\ ᓚ\ ᓪ\ ᓰ\ ᕤ\ ᖓ\ ᚠ\ ᚴ\ ឈ\ ᠄\ ḁ\ ṳ\ Ṿ\ ⁞\ ₸\ ⃒\ ↰\ ↲\ ⇉\ ⇽\ ∿\ ⊷\ ⓬\ ⓵\ ╋\ ╘\ ╻\ ☙\ ⚲\ ➰\ ⟡\ ⱡ\ ⴻ\ ⵎ\ 㑩\ 㝵\ 㫊\ 㴎\ 㵡\ 䂩\ 䓍\ 䔬\ 兌\ 凈\ 凜\ 勻\ 噹\ 恥\ 惱\ 暉\ 檻\ 欖\ 氫\ 瘍\ 癩\ 祅\ 籃\ 紡\ 絞\ 綁\ 膽\ 諧\ 諫\ 貍\ 鈉\ 鈔\ 鈦\ 鍍\ 闕\ 陜\ 顫\ 驕\ 鯊\ 齧\ ꀉ] ꁆ] ꁱ] ꁴ] ꂿ] ꈁ] ꌧ] Ꙩ] 귈] 꼴 ] 뀌 ] 낚 ] 넉 ] 덩 ] 딕] 땀] 렁] 뤄] 맺] 뽐] 셈] 셋] 셸] 쉼] 슘] 썸] 윙] 쩔] 츄] 킴] 톨] 툴] 틈 ] 헿!] "] #] $] %] &] '] (] )] *] +] ,] -] .] /] 0] 1] 2] 3] 4] 5] 6] 7] 8] 9] :] ;] <] =] >] ?] @] A] B] C] D] E] 𑑂F] 𓀃G] 🅺H] 🆓I] 🌦J] 🍟K] 🎇L] 🏧M] 🐪N] 🐲O] 💆P] 📃Q] 📑R] 📥S] 🔀T] 🔛U] 🕮V] 🗒W] 🗯X] 🗿Y] 🚐Z] 🚒[] 🚬\] 🟣]] 🡺^] 🤥_] 🥫`] 🥱a] 🥹b] 🦑c] 🦺d] 🧾e] 􏰍f] Ũg] ƌh] ƙi] Ƥj] ƨk] Ʊl] ǩm] Ƀn] Ϫo] Җp] Ӓq] ֳr] ٻs] څt] ڡu] ܗv] ޅw] ࡇx] ৷y] ਍z] ਓ{] ਠ|] ૃ}] ଯ~] ಫ] ೦] ෴] ။] ၚ] ၿ] ᄈ] ᄊ] ᅟ] ኃ] ዞ] ᐟ] ᐣ] ᒑ] ᗰ] ᗴ] ᙚ] ᚭ] ᚼ] ៈ] ០] ᦵ] ᪣] ᪥] ᴉ] ᷁] ḟ] Ṁ] ẃ] Ể] ὃ] Ὕ] ῡ] ⁊] ℥] ⅌] ↹] ⇜] ∡] ⊔] ⋈] ⌗] ⌫] ⌾] ⍰] ⎋] ⎓] ␛] ┫] ⠇] ⠈] ⠛] ⤺] ⬊] ⬌] ⬬] ⭆] Ɱ] ⴱ] ⵙ] ⷡ] ⺮] 〄] ヱ] 㔏] 㕮] 㘗] 㙦] 㠭] 㬵] 㵐] 㵘] 㹴] 䃺] 䋦] 䒕] 䒩] 䗪] 䥫] 䶥] 傢] 兇] 嘍] 嘔] 嚨] 墊] 崗] 廬] 弔] 愷] 慘] 憤] 撥] 擷] 暘] 暱] 檜] 歎] 汙] 滬] 癲] 硃] 窺] 竇] 糰] 絃] 繕] 繰] 蘿] 蛺] 詭] 誦] 誼] 諒] 賑] 迺] 釀] 釗] 鈧] 雛] 霤] 颯] 饋] 駱] 驛] 髒] 鮑] 鴛] 鷲] 鷺^ 鹹^ ꀾ^ ꄷ^ ꇐ^ ꉐ^ ꉚ^ ꉣ^ ꌂ^ ꎆ ^ ꎔ ^ ꏤ ^ ꏭ ^ ꏯ ^ ꑷ^ ꕤ^ ꯂ^ ꯃ^ ꯩ^ 낀^ 늙^ 땜^ 뚱^ 릅^ 뭘^ 빔^ 빡^ 삐^ 쉐^ 쌓^ 쏘^ 쑤^ 쑥 ^ 얜!^ 옹"^ 짤#^ 첩$^ 콕%^ 툰&^ 헨'^ 헹(^ 휠)^ *^ +^ ,^ -^ .^ /^ 0^ 1^ 2^ 3^ 4^ 5^ 6^ 7^ 8^ 9^ :^ ;^ <^ =^ >^ ?^ @^ A^ B^ C^ D^ E^ F^ G^ H^ I^ J^ K^ L^ M^ N^ O^ P^ Q^ 𐌽R^ 𑐖S^ 𑐰T^ 𓀀U^ 𓀁V^ 𓀂W^ 𓀄X^ 𓀅Y^ 𓀆Z^ 𓀇[^ 𓀈\^ 𓀉]^ 𓀊^^ 𓀋_^ 𓀌`^ 𓀍a^ 𓀎b^ 𓀏c^ 𓀐d^ 𓀑e^ 𓀒f^ 𓀓g^ 𓀔h^ 𓀕i^ 𓀖j^ 𓀗k^ 𓀘l^ 𓀙m^ 𓀚n^ 𓀛o^ 𓀜p^ 𓀝q^ 𓀞r^ 𓀟s^ 𓀡t^ 𓀢u^ 𓀣v^ 𓀤w^ 𓀥x^ 𓀦y^ 𓀧z^ 𓀨{^ 𓀩|^ 𓀪}^ 𓀫~^ 𓀬^ 𓀻^ 🅓^ 🅟^ 🅣^ 🍙^ 🐀^ 🐗^ 🐜^ 👖^ 📔^ 📟^ 📫^ 📿^ 🔓^ 🕉^ 🗂^ 🗑^ 🗗^ 🗻^ 🗼^ 🛬^ 🛳^ 🛴^ 🥄^ 🥯^ 🦦^ 🧃^ 🧒^ 🧓^ 🧕^ 🪦^ 󰀄^ 􀁾^ 􀳦^ 􏰄^ 􏰅^ Ƿ^ Ǹ^ Ȋ^ ȗ^ Ⱦ^ ɋ^ ɭ^ Ϟ^ Ϡ^ ϡ^ ϸ^ Ѣ^ Ѳ^ ҟ^ Ԍ^ ֭^ ٭^ ڰ^ ڳ^ ߞ^ ߹^ ॢ^ ਝ^ ૯^ ଥ^ ୂ^ ఐ^ ಐ^ ೨^ ໐^ ဉ^ ᅧ^ ᅴ^ ᆴ^ ሣ^ ቂ^ ቄ^ ኬ^ ዬ^ ጌ^ ፎ^ ፖ^ Ꮥ^ Ꮯ^ ᐎ^ ᐩ^ ᐸ^ ᒧ^ ᒫ^ ᕼ^ ᗪ^ ᛦ^ ឃ^ ឌ^ ឺ^ ៥^ ᷀^ ḓ^ ṋ^ ẑ^ Ỏ^ Ợ^ ἧ^ Ἥ^ Ἴ^ Ὅ^ ⁑^ ₢^ ⃰^ ℩^ ⇈^ ⊡^ ⊩^ ⋁^ ⌕^ ⌙^ ⍵^ ⏏^ ⓶^ ▯^ ☈^ ⛴^ ⟲^ ⠋^ ⠥^ ⣿^ ⪼^ ⬂^ ⬃^ ⬋^ ⮕^ ⵣ^ ⶪ_ ⸤_ ⸥_ ⻥_ ㇏_ 㑇_ 㑚_ 㒯_ 㖎_ 㖱 _ 㗳 _ 㛃 _ 㟌 _ 㢲 _ 㤭_ 㭛_ 㮚_ 㯎_ 䁩_ 䍁_ 䏨_ 䓘_ 䔯_ 䭢_ 冑_ 剝_ 喫_ 墻_ 尷_ 嵒_ 弒_ 懇_ 戔 _ 挾!_ 搶"_ 槓#_ 滷$_ 澀%_ 瀆&_ 燭'_ 癢(_ 盞)_ 穫*_ 簞+_ 簫,_ 紺-_ 絆._ 緬/_ 繽0_ 纔1_ 缽2_ 肅3_ 脹4_ 蕪5_ 薈6_ 虧7_ 蟄8_ 蟬9_ 蟻:_ 諳;_ 諷<_ 遜=_ 鈕>_ 鈺?_ 銚@_ 錘A_ 閘B_ 頰C_ 髏D_ 齦E_ ꀸF_ ꃢG_ ꇗH_ ꉡI_ ꌋJ_ ꍬK_ ꜆L_ ꝛM_ ꡔN_ ꡕO_ ꦼP_ 겊Q_ 겟R_ 궐S_ 꽤T_ 냄U_ 늬V_ 닦W_ 덴X_ 멧Y_ 밉Z_ 벅[_ 벚\_ 볍]_ 뻗^_ 섰__ 쌀`_ 쌤a_ 쏭b_ 앗c_ 엣d_ 왁e_ 웜f_ 윌g_ 젖h_ 쫓i_ 캡j_ 탓k_ 텝l_ 톱m_ 퉁n_ 풋o_ 헼p_ 흠q_ r_ s_ t_ u_ v_ w_ x_ y_ z_ {_ |_ }_ ~_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 𐌅_ 𐨛_ 𓀼_ 𓀽_ 𓀾_ 𓀿_ 𓁀_ 𓁁_ 𓁂_ 𓁃_ 𓁄_ 𓁅_ 𓁆_ 𓁇_ 𓁈_ 𓁉_ 𖤐_ 𝄞_ 🅗_ 🌇_ 🍖_ 🍗_ 🍛_ 🍮_ 🎠_ 🏂_ 🐵_ 👒_ 💇_ 💽_ 🔆_ 🖱_ 😙_ 🛁_ 🧗_ 🧮_ 🧴_ 🧶_ 🩹_ 􏰓_ Ŵ_ ƈ_ ƕ_ ƥ_ ǭ_ ȕ_ Ȳ_ ʨ_ ϙ_ Ͼ_ Ѝ_ Ҫ_ Ҭ_ ֜_ ׹_ ٲ_ ڿ_ ۽_ ܼ_ ݶ_ ޫ_ ߣ_ ࠥ_ ॽ_ ঔ_ ਢ_ ਥ_ ೧_ ഀ_ ൌ_ ൧_ ෞ_ ༛_ ᅬ_ ᅭ_ ᅱ_ ሟ_ ቐ_ ኅ_ ጆ_ ጡ_ ጾ_ ፆ_ ፌ_ ፕ_ Ꭳ_ Ꮖ_ Ꮚ_ Ꮛ_ Ᏸ_ ᑉ_ ᑫ_ ᑳ_ ᒌ` ᒯ` ᓈ` ᓱ` ᔅ` ᔕ` ᖁ` ᖇ` ᚐ` ᚩ ` ᚻ ` ᛜ ` ᛟ ` ᜄ ` ᜊ` ᜒ` ឬ` ៣` ៦` ᩚ` Ḑ` Ḗ` ṗ` ṙ` Ṟ` Ắ` Ẵ` Ờ` Ụ` Ử` ỵ` Ὠ` Ῑ ` ₾!` ⃠"` ℮#` Ⅎ$` ↀ%` ≾&` ⊝'` ⋂(` ⋃)` ⋖*` ⋗+` ⌶,` ⍎-` ⍟.` ⍳/` ⍸0` ␙1` ┷2` ╷3` ▞4` ◭5` ☌6` ✙7` ➿8` ⟬9` ⟭:` ⠄;` ⠓<` ⠙=` ⠼>` ⡱?` ⢀@` ⣄A` ⣷B` ⣾C` ⤋D` ⤫E` ⨉F` ⨱G` ⬐H` ⲃI` ⲑJ` ⵓK` ⵡL` ⻗M` ヰN` 㖭O` 㛑P` 㝬Q` 㤒R` 㱧S` 㳪T` 㸁U` 㸽V` 㹥W` 䀳X` 䁖Y` 䄉Z` 䇿[` 䑍\` 䓖]` 䕀^` 䞫_` 䟎`` 䠀a` 䦆b` 䨙c` 䰟d` 䱗e` 䶛f` 傭g` 僂h` 啞i` 嘩j` 婁k` 懺l` 戉m` 抎n` 撿o` 暈p` 棗q` 檯r` 滌s` 灝t` 燄u` 燉v` 獵w` 甌x` 癡y` 礦z` 禱{` 禿|` 箏}` 籟~` 絢` 絨` 綻` 繚` 耑` 耬` 聶` 膿` 芐` 茍` 萵` 虜` 螻` 訣` 諏` 諱` 諺` 謨` 軀` 軾` 鈍` 鋁` 鎊` 鏘` 鏢` 閎` 閨` 閻` 阯` 頦` 饅` 饗` 驢` 鬢` 鮫` 鰭` 鳶` 鴦` 鵲` ꀤ` ꁨ` ꁮ` ꄖ` ꅏ` ꆀ` ꆿ` ꇨ` ꇱ` ꇷ` ꈤ` ꈵ` ꉬ` ꊂ` ꋚ` ꌊ` ꌦ` ꌺ` ꍞ` ꏮ` ꑟ` ꑠ` ꒒` ꒫` ꓘ` ꜄` ꞙ` ꟾ` ꧉` ꪝ` ꫛ` ꬴ` ꯨ` 곹` 괄` 굳` 깭` 꼈` 낡` 낯` 놔` 놨` 눕` 눠` 덥` 뎐` 딪` 땡` 똥` 렷` 륜` 뭇` 밟` 뱃` 뽀` 숍` 숯` 썰` 썼` 옌` 왠` 줌` 쨌` 캄` 컸` 퀀` 톰` 팥` 펑` 핸` 혂` 홑` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` ` a a a a a a a a a  a  a  a  a  a a a a a a a a a a a a a a a a a a a  a !a "a #a $a %a &a 'a (a )a *a +a ,a -a .a /a 0a ﷐1a ﷮2a ︋3a 𐌳4a 𑐥5a 𑐮6a 𑐵7a 𑑅8a 🂡9a 🅥:a 🌄;a 🌔a 🎦?a 🏑@a 🏨Aa 🐡Ba 👬Ca 👷Da 📯Ea 🔲Fa 🕐Ga 🕛Ha 🗄Ia 😿Ja 🙍Ka 🚍La 🚛Ma 🛶Na 🛹Oa 🥸Pa 🥾Qa 🦒Ra 🦛Sa 🧋Ta 🧔Ua 🧳Va 𠂆Wa 𠙶Xa 𤰉Ya 𧿹Za 󰀃[a 󾓭\a 􀆰]a 􏰁^a Ɲ_a Ƣ`a Ʃaa ƫba Ǐca Ǒda Ǵea Ǻfa Ʉga ɮha ɰia ɶja ʆka ˞la ˴ma ˻na Ͱoa Ϋpa ΰqa ϭra Ϻsa Ѧta Ѭua Ѯva ѱwa Ѵxa Ѹya ѽza ҁ{a Ӈ|a ӱ}a Ԛ~a ՘a ֠a ؒa ٿa ۂa ܾa ܿa ޓa ޚa ߏa ࢭa ऩa ঈa ঊa ૉa ଞa ୃa ఊa ౞a ಌa ಞa ඊa ඕa ඛa ༂a ဓa ၄a ၉a ၍a ႊa ႐a ႕a ᅤa ᅨa ኪa ኸa ጁa ጤa ጹa ፃa ፉa Ꭸa Ꮗa Ꮻa Ᏹa ᐄa ᐪa ᐯa ᒐa ᓀa ᔒa ᔥa ᗦa ᚺa ᛈa ᛗa ៧a ៨a ᴩa ᵮa ᶇa ᷊a ᷎a Ṃa ẋa ẏa Ẕa Ẫa Ặa Ẽa Ỳa ἷa Ὄa ᾱa ῑa ⁙a ⁝a a ₭a ⇙a ∁a ∓a ∛a ≟a ≼a ⊄a ⊊a ⌞a ⌡a ⌬a ⍺a ⎧a ⓭a ⓮a ⓯a ╛a ╶a ☥a ☿a ♱a ⚋a ⛤a ❜a ⠏a ⠒a ⠞a ⠧a ⠹a ⤍a ⤻a ⨳a ⬘a ⬝a Ⱄa Ɐa Ⲁa ⲇa ⴳa ⴼa ⵖa ⻊a 㐅a 㑹a 㔹a 㕷a 㙾a 㝹a 㠠a 㠯a 㧯b 㭴b 㸃b 㸙b 㸞b 㽲b 䀹b 䅁b 䇒b 䍫 b 䐜 b 䒇 b 䗣 b 䘈 b 䘓b 䚬b 䚹b 䜩b 䞟b 䠡b 䯬b 䯼b 䲝b 䲠b 䴓b 侷b 倖b 剋b 囪b 姪b 屢b 彆b 懲 b 揹!b 攪"b 晝#b 曇$b 曖%b 櫓&b 櫛'b 滸(b 濫)b 瀨*b 瀾+b 爍,b 瑣-b 甕.b 盪/b 矇0b 窯1b 紂2b 紓3b 紳4b 綏5b 綬6b 芻7b 苧8b 蒔9b 蔔:b 蔞;b 蕎b 襯?b 訢@b 詔Ab 詣Bb 諗Cb 諜Db 諭Eb 謳Fb 賬Gb 轍Hb 鄒Ib 鉛Jb 錐Kb 鍬Lb 鐸Mb 鑰Nb 閥Ob 闔Pb 颱Qb 飪Rb 駁Sb 駭Tb 鬧Ub 鮭Vb 鱧Wb 鵡Xb 鶯Yb 鸚Zb 麩[b 龕\b ꁄ]b ꃷ^b ꄅ_b ꄐ`b ꆃab ꆅbb ꇂcb ꇇdb ꇊeb ꈚfb ꈧgb ꉆhb ꋦib ꋪjb ꌌkb ꎵlb ꏅmb ꑿnb ꙳ob ꜀pb Ꞓqb Ꞙrb ꧀sb 갇tb 곰ub 곽vb 굉wb 깁xb 깍yb 눅zb 딴{b 뜰|b 륨}b 맷~b 벡b 봇b 붐b 빚b 샛b 솟b 싣b 썬b 쏠b 옳b 웬b 잤b 잼b 젯b 젼b 짚b 찐b 찜b 찢b 칵b 캘b 탠b 퐁b 핍b 헐b 헰b 혐b 훼b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b b ﷓b 𐤄b 𐤅b 𐬀c 𓅯c 𖥧c 🀆c 🅡c 🌉c 🌗c 🌘c 🌝c 🍡 c 🎔 c 🏬 c 🐂 c 🐆 c 👛c 👜c 👞c 💒c 💢c 🔘c 🔠c 🕌c 🖇c 🗃c 🚆c 🚇c 🚓c 🛀c 🛢c 🤶c 🤽c 🦭c 🦾 c 🧎!c 🧯"c 🪄#c 🪳$c 𠒇%c 𢯎&c 𦟌'c 𨻧(c 󶀁)c 􀀂*c 􏰐+c 􏰔,c Ŗ-c Ɯ.c Ƨ/c Ȃ0c Ȗ1c Ⱥ2c Ɏ3c ˩4c ͼ5c ΁6c ΂7c Ϛ8c ϶9c ѿ:c ҵ;c Ҽc Ӑ?c Ӗ@c ӳAc ԍBc ԎCc ԑDc ֎Ec ׿Fc ؏Gc ؔHc ؕIc ؘJc ڍKc ڱLc ۑMc ۾Nc ܵOc ݈Pc ݗQc ݚRc ݼSc ߐTc ࡌUc ࡘVc ࢣWc ࢵXc ࣐Yc ৐Zc ৗ[c ਺\c ੮]c ઠ^c ઢ_c ଽ`c ௚ac ఃbc ఘcc ಏdc ಒec ಓfc ಘgc ೫hc ೮ic ഈjc ൗkc ൠlc ༃mc ༆nc ༈oc ྛpc ࿈qc ࿓rc ဦsc ႀtc ჷuc ᅪvc ᅰwc ሂxc ሯyc ቓzc ቷ{c ኘ|c ኙ}c ጦ~c ፓc ፠c Ꭴc Ꭾc Ꮂc Ꮆc Ꮉc Ꮙc Ꮟc Ꮣc Ꮩc Ꮫc Ᏻc ᐋc ᐙc ᐹc ᑌc ᑖc ᑿc ᒢc ᓃc ᓵc ᔮc ᖛc ᗄc ᗝc ᙠc ᚫc ᚹc ᜆc ᜎc ឥc ឯc ៛c ᡠc ᦉc ᦕc ᧂc ᧉc ᩤc Ḱc Ṇc ṧc ẗc Ẩc Ữc Ỹc Ἕc ἳc ᾠc ᾥc ⁓c c ⃕c ⃖c ↄc ↿c ⇟c ⇺c ⇻c ∐c ∤c ≙c ≣c ≽c ⊎c ⊠c ⋉c ⋛c ⍼c ⎨c ⎩c ⎫c ⎬c ⎾c ⏌c ⓷c ▟c ◷c ☩c ♄c ⚏c ⚹c ⟗c ⟮c ⟯c ⟰c ⟳c ⠊c ⠕c ⠦c ⠴c ⠸c ⡯c ⤓c ⤬c ⥥c ⧗c ⪙c ⬀c ⬈c Ⱡc ⲝc ⵃc ⵟc ⸂c ⸃c ⺌c ⺾c ⻌c ⻤c ⻮c 〚c 〠c 〮c 㐱c 㒼c 㓥c 㖿c 㚺c 㝉c 㤄c 㦸c 㧢c 㩄c 㬢c 㵍c 㸅c 㹚c 㻥d 㿗d 䀉d 䀮d 䇂d 䇯d 䋞d 䏰d 䒦d 䕫 d 䖏 d 䖳 d 䛐 d 䣭 d 䬛d 䱻d 䲁d 佇d 佪d 嗚d 嘰d 嚮d 囁d 堿d 墮d 墾d 壢d 姦d 崢d 巑d 廝d 愜d 慄 d 慚!d 憚"d 懾#d 摟$d 撓%d 攬&d 暸'd 曄(d 瀧)d 熒*d 燙+d 燼,d 璽-d 瓖.d 皺/d 瞞0d 硯1d 磚2d 穢3d 箋4d 簀5d 籲6d 紲7d 絳8d 絹9d 綴:d 緞;d 繭d 蝸?d 蠔@d 蠱Ad 覈Bd 謊Cd 讖Dd 躊Ed 輒Fd 輟Gd 鉉Hd 銹Id 鋤Jd 錕Kd 錠Ld 錳Md 鏤Nd 鐔Od 鑲Pd 闖Qd 雋Rd 靂Sd 韌Td 餽Ud 饃Vd 馭Wd 駑Xd 髖Yd 鯖Zd 鵜[d 鷗\d 鹵]d 黴^d ꀎ_d ꀘ`d ꁌad ꁡbd ꂰcd ꄂdd ꄧed ꄱfd ꅍgd ꅐhd ꅑid ꇈjd ꇌkd ꇭld ꇰmd ꇿnd ꈈod ꈐpd ꈝqd ꈢrd ꈩsd ꉜtd ꋅud ꋊvd ꋯwd ꍝxd ꎬyd ꏁzd ꏂ{d ꏦ|d ꏿ}d ꐎ~d ꑞd ꑴd ꒉd ꒜d ꒨d ꜣd Ɥd ꟻd ꦛd ꪔd ꬵd 갠d 걍d 굡d 궈d 귤d 깐d 깝d 깻d 꼼d 꼿d 꾀d 꾹d 꿇d 낫d 넬d 덟d 둡d 뒀d 딥d 딧d 뚝d 룬d 맣d 몫d 묃d 뭄d 뵙d 뽕d 뿜d 샐d 솜d 쌈d 쌩d 쎄d 얇d 얹d 엿d 웰d 짙d 짬d 쩍d 쭌d 춥d 컹d 쿼d 킥d 텀d 푹d 헴d 헵d 횗d 횟d 훔d 휩d 흩d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d e e e e e e e e e  e  e  e  e  e e e e e e e e e e e e e e e e e e e  e !e "e #e $e %e &e 'e (e )e *e +e ,e -e .e /e 0e 1e 2e 3e 4e 5e 6e 7e 8e 9e :e ;e e ?e @e Ae Be Ce De Ee Fe Ge He 𐄚Ie 𐤔Je 𑄌Ke 𑄴Le 𑐐Me 𑐩Ne 𑐹Oe 𑑄Pe 𒆠Qe 🁢Re 🅧Se 🆖Te 🌃Ue 🍚Ve 🍤We 🎏Xe 🎚Ye 🎳Ze 🏤[e 🏮\e 🐃]e 🐏^e 🐤_e 🐩`e 💈ae 💮be 💴ce 📠de 📪ee 🕖fe 🛐ge 🛥he 🜄ie 🟊je 🟥ke 🡒le 🡪me 🢔ne 🤌oe 🥌pe 🦗qe 🦘re 🦳se 🦴te 🧫ue 🪑ve 🫂we 🫖xe 🫠ye 𠜎ze 𥔲{e 􀀀|e 􀀁}e 􀂃~e 􀆳e 􏰕e Ƈe ƍe ƻe Ȁe ȍe Ȑe ȣe ȫe ȷe ɂe Ɋe ʥe ˭e ˾e Ͳe ͳe Ѷe ѹe Ҝe Ӄe ӄe ӈe ӽe ԃe ԋe ԡe ԰e ՟e ֊e ֡e ֯e ؑe ٕe ڃe ڄe ڼe ۭe ܍e ܤe ݋e ݓe ݙe ݩe ݾe ޒe ޕe ޠe ޯe ޷e ߇e ߎe ߒe ߕe ߘe ߭e ߶e ߻e ࠮e ࡓe ऒe ৾e ਔe ਞe ੬e ੯e ઘe ૈe ଝe ୈe ఌe ఓe ఞe ౚe ౦e ౼e ಛe ೡe ೯e ഐe ഘe ඖe ඟe ඤe ෛe ෡e ෵e ໓e ༇e ༑e ༙e ྄e ྇e ࿅e ࿎e ဌe ဴe ၎e ၤe ჳe ᄫe ᅙe ᅫe ᅯe ᆭe ሹe ሾe ቨe ቪe ቫe ኡe ኻe ጳe ፐe ፒe ፲e Ꮏe Ꮐe Ꮼe ᐚe ᐵe ᑮe ᑰe ᒄe ᓖe ᓛe ᕈe ᕘe ᖏf ᖑf ᖢf ᖳf ᗡf ᙱf ᙳf ᚅf ᚓf ᚛ f ᚜ f ᛘ f ᜅ f ᜌ f ᜏf ᜓf ឧf ឿf ៤f ᨒf ᬩf ᬭf ᬶf ᮑf ᯓf ᱍf Ტf ᴎf ᴧf ᵺf ᶂf ᷂f ḉ f Ḣ!f ḣ"f ḧ#f Ḷ$f Ṗ%f ṩ&f ṷ'f ẁ(f ẅ)f ẍ*f ẘ+f Ẻ,f Ἆ-f ἓ.f ὦ/f ᾔ0f ᾗ1f ᾧ2f Ᾰ3f ῤ4f ⁁5f 6f ⃔7f ⃢8f ℄9f ℔:f ℟;f ⅋f ↫?f ⇼@f ∸Af ≍Bf ≗Cf ⊅Df ⊛Ef ⊜Ff ⊨Gf ⋘Hf ⌢If ⍧Jf ⎔Kf ⎭Lf ⎰Mf ⎷Nf ␦Of ┡Pf ▃Qf ▆Rf ▤Sf ◴Tf ☋Uf ♙Vf ♝Wf ⚀Xf ⚅Yf ⚐Zf ⚱[f ⛟\f ➵]f ⟷^f ⟸_f ⠃`f ⠎af ⠠bf ⣆cf ⣇df ⤤ef ⦅ff ⦆gf ⦵hf ⪆if ⫋jf ⫷kf ⬎lf ⬒mf ⬿nf ⭅of ⭢pf ⮂qf ⮟rf ⱄsf Ɽtf Ⱨuf ⱪvf ⲅwf Ⲓxf Ⲕyf Ⲟzf Ⲡ{f Ⲥ|f ⵇ}f ⵚ~f ⸨f ⸩f ⺅f ⺉f ⺖f ⺘f ⺡f ⺨f ⺭f ⺲f ⺹f ⻂f ⻏f ⻖f ⿻f 〛f ㄦf 㒰f 㒳f 㓗f 㖪f 㘰f 㜺f 㞗f 㞧f 㞿f 㟖f 㪺f 㳛f 㶇f 㸑f 㸬f 㻋f 㼪f 㽏f 㾦f 䂳f 䇄f 䈃f 䍙f 䏜f 䗴f 䝺f 䡶f 䢺f 䤩f 䥽f 䦊f 䧿f 䯄f 䰕f 䲢f 䴞f 䴩f 兗f 勛f 嚕f 嚥f 囀f 囂f 壎f 嫵f 寢f 屝f 崠f 巔f 幟f 惲f 憊f 搗f 摯f 擄f 擰f 擱f 攔f 斂f 曠f 朧f 枙f 梟f 橢f 殲f 淒f 漣f 潑f 澗f 瀕f 瀰f 煒f 煬f 燾f 犛f 瑯f 璣f 瓏f 痺f 瘞f 癱f 皰f 禰f 稟f 竅f 筍f 筧f 籬f 糢f 綸f 緲f 縉f 縞f 羈f 聳f 聾f 艙f 薊f 藷f 蠅f 衹f 詁f 詛f 誒f 誥f 諤f 豎f 貽f 賁f 贖f 踫f 蹣f 躒g 輻g 轅g 轆g 轡g 鄔g 鄴g 鄺g 釵g 鈐 g 銓 g 鋇 g 鍇 g 鍶 g 鎔g 鎗g 鏗g 鏟g 鏽g 鐳g 鑼g 鑾g 閩g 閭g 闡g 隴g 靄g 鞦g 頊g 頡g 頹g 颶g 颺 g 饌!g 駢"g 鬚#g 魘$g 鯰%g 鰍&g 黽'g 齲(g ꀄ)g ꀇ*g ꀑ+g ꀻ,g ꁁ-g ꁖ.g ꁦ/g ꁯ0g ꂁ1g ꂦ2g ꂸ3g ꂺ4g ꂼ5g ꄃ6g ꄆ7g ꄇ8g ꄏ9g ꄹ:g ꅇ;g ꅝg ꅿ?g ꆧ@g ꆫAg ꇓBg ꇢCg ꇪDg ꈌEg ꈜFg ꈫGg ꉎHg ꉼIg ꊈJg ꊌKg ꊾLg ꋆMg ꋠNg ꋻOg ꌐPg ꌡQg ꌣRg ꌬSg ꎂTg ꎧUg ꎳVg ꎴWg ꎹXg ꎺYg ꐘZg ꐝ[g ꐞ\g ꐡ]g ꐨ^g ꑱ_g ꜂`g ꜜag ꜳbg Ꝼcg ꠥdg ꡦeg ꪮfg ꯅgg ꯈhg 갚ig 갭jg 갯kg 갱lg 궂mg 궤ng 긋og 깎pg 꺾qg 껌rg 낳sg 넛tg 넨ug 넴vg 뇽wg 댕xg 듦yg 뛸zg 멎{g 멕|g 몹}g 뫼~g 뭉g 뮬g 벙g 뺏g 뽑g 뿓g 쁠g 섀g 솝g 슝g 쌉g 썩g 쏙g 쑈g 씁g 씹g 앓g 얗g 엌g 왓g 욤g 쩜g 쭈g 쯔g 챔g 챕g 첸g 컥g 쾅g 쿄g 큽g 팰g 펄g 퓸g 햄g 헜g 헳g 헽g 횡g 훠g 휨g 흉g 힙g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g g h h h h h h h h h  h  h  h  h  h h h h h h h h h h h h h h h h h h h  h !h "h #h $h %h &h 'h (h )h *h +h ,h -h .h /h 0h 1h 2h 3h 4h 5h 6h 7h 8h 9h :h ;h h ?h @h Ah Bh Ch Dh Eh Fh Gh Hh Ih Jh Kh Lh Mh Nh Oh Ph Qh Rh Sh Th Uh Vh Wh Xh Yh Zh [h \h ]h ^h _h `h ah bh ch dh eh fh gh hh ih jh kh ﷽lh 𐀺mh 𐌲nh 𐌹oh 𐍀ph 𐍄qh 𐐈rh 𐤉sh 𐤋th 𐬁uh 𐬎vh 𐬯wh 𑁋xh 𑐡yh 𒀭zh 𒊒{h 𓃟|h 𓍄}h 𓏲~h 𖡼h 🅜h 🌂h 🌆h 🌓h 🍈h 🍠h 🍧h 🍨h 🎿h 🐁h 🐌h 🐹h 👃h 👲h 👺h 💱h 📇h 📓h 📭h 🔚h 🔞h 🕇h 🕍h 🕎h 🕚h 🖍h 🗔h 🗾h 😽h 🙎h 🚃h 🚄h 🚋h 🚖h 🚡h 🚪h 🚮h 🚰h 🚷h 🛄h 🛤h 🛺h 🤾h 🥡h 🥼h 🦚h 🦥h 🦧h 🦩h 🦮h 🦹h 🧇h 🧻h 🩰h 🪁h 🪓h 🪡h 🫀h 🫑h 🫡h 𠀤h 𠂇h 𠆢h 𠆤h 𠝹h 𠱸h 𤋮h 𬜯h 򐂰h 􀳇h 􏰇h 􏰋h 􏰑h 􏰒h 􏰛h Ƌh Ʈh Ǔh Ǚh Ǟh ǡh Ǩh Ǿh ȑh ȓh Ȩh Ȯh ȵh ɇh ɉh Ɍh ɼh ʄh ʡh ʪh ʮh ʯh ˪h ˰h ˱h ˸h ˼h ͸h ͹h ΃h ΢h ϗh Ϥh Ϩh Ͻh ѧh ѫh Ѿh ҄h ҕh Ҙh Ҧh Ҿh ҿh ӌh ӎh Ӣh Ӥh Ӹh Ԃh Ԅh Ԇh Ԑh ԕi ԟi Ԧi ԧi Ԩi Ԫi ֏i ֕i ֝i ׊ i ׍ i ׫ i ׮ i ׯ i ׷i ؀i ؂i ؋i ؍i ؗi ؜i ؽi ٛi ڐi ڙi ڨi ڸi ۃi ۠i ۧi ۯi ۺi ۻ i ܃!i ܄"i ܅#i ܑ$i ܔ%i ܜ&i ܧ'i ܭ(i ܴ)i ܸ*i ܹ+i ݅,i ݊-i ݔ.i ݞ/i ݢ0i ݣ1i ݧ2i ݨ3i ݫ4i ݭ5i ޟ6i ޤ7i ޻8i ߊ9i ߌ:i ߗ;i ߧi ߼?i ࠅ@i ࠈAi ࠦBi ࠵Ci ࠸Di ࠼Ei ࠽Fi ࠿Gi ࡀHi ࡐIi ࡰJi ࡱKi ࡶLi ࡽMi ࡿNi ࢇOi ࢍPi ࢖Qi ࢗRi ࢨSi ࢼTi ࣁUi ࣠Vi ࣣWi ࣱXi ऺYi ॏZi ঳[i ঽ\i ৆]i ৛^i ৠ_i ৣ`i ৼai ਑bi ਩ci ੏di ੔ei ੡fi ੢gi ੩hi ੪ii ੿ji ઄ki ઐli ઑmi ઴ni ૄoi ૬pi ૳qi ଴ri ଵsi ୊ti ୑ui ஛vi ஠wi ஫xi ௏yi ௐzi ௑{i ௙|i ௫}i ௬~i ௯i ௸i ௺i ௻i ఁi ఑i ౄi ౝi ౩i ౬i ౮i ౳i ౶i ಀi ಩i ಴i ಼i ಽi ೕi ೖi ೛i ೠi ೩i ೪i ೷i ೿i ഃi ഋi ഍i ഔi ൚i ൮i ൰i ඈi ඌi ඝi ඬi ෇i ෋i ෍i ෩i ෾i ໤i ༏i ༲i ༸i ༿i ཋi ཫi ྜi ྪi ྵi ࿊i ࿙i ဃi ဋi ဍi ဧi ၃i ၇i ၻi ၽi ႍi ႞i ჱi ჴi ჸi ᄚi ᆪi ᆲi ᆳi ᆵi ᆽi ሏi ሑi ሦi ሷi ቭi ቿi ኋi ኗi ኽi ዌi ጐi ጕi ጪi ጺi ፁi ፂi ፄi ፇi ፮i Ꭻi Ꭽi Ꮀi Ꮈi Ꮓi Ꮞi Ꮨi Ꮭi Ꮸi Ꮺi Ᏺi ᐏi ᐐi ᐞi ᐥi ᑑi ᔉi ᔑi ᕲi ᕽi ᖫi ᖭi ᗅi ᗯi ᘏi ᙡi ᚁi ᚆi ᚇi ᚈi ᚉi ᚋi ᚏi ᚪi ᛝi ᜃi ᜇi ᜈj ឣj ឱj ៏j ៗj ᠁j ᡗj ᡩj ᡵj ᤣ j ᥊ j ᦑ j ᦗ j ᦟ j ᦴj ᦹj ᦺj ᧈj ᨄj ᨐj ᨑj ᬓj ᬮj ᬱj ᱬj Ეj ᳵj ᴑj ᷈j ḃj ḅj Ḋj ḋ j ḕ!j ḙ"j Ṡ#j ṱ$j Ṽ%j ṽ&j Ẃ'j ẉ(j ẟ)j Ỗ*j Ỻ+j ἃ,j ἆ-j Ἦ.j Ἵ/j Ὡ0j ᾤ1j ᾷ2j Ᾱ3j ῐ4j ῒ5j ῠ6j ῴ7j Ὸ8j ⁋9j ⁧:j ⁳;j ℣j ↚?j ↨@j ↭Aj ⇇Bj ⇕Cj ⇯Dj ⇹Ej ≓Fj ≚Gj ⊋Hj ⊚Ij ⋚Jj ⋤Kj ⌸Lj ⌻Mj ⎈Nj ⎳Oj ⎻Pj ⎼Qj ⏍Rj ⏐Sj ⏚Tj ⏟Uj ⏹Vj ␄Wj ␔Xj ␤Yj ⓸Zj ⓺[j ⓻\j ┎]j ┒^j ┲_j ┾`j ╄aj ╒bj ╖cj ▇dj ♇ej ⚄fj ⚌gj ⛎hj ⛦ij ✇jj ✛kj ❢lj ❬mj ❲nj ❳oj ➈pj ➉qj ⟺rj ⟼sj ⠁tj ⠔uj ⠗vj ⠝wj ⠟xj ⣓yj ⣛zj ⣯{j ⤑|j ⤕}j ⤳~j ⥎j ⦕j ⦖j ⦙j ⦽j ⧸j ⩔j ⩘j ⫐j ⬁j ⬣j ⬭j ⬯j ⮙j ⮜j Ⱗj Ⱙj ⰶj ⱔj ⱙj ⱱj Ⲏj Ⲙj Ⲧj ⲫj ⴹj ⵕj ⵥj ⸌j ⸍j ⸪j ⸳j ⸸j ⺀j ⺁j ⻦j ⿿j 〘j 〙j 〥j 〱j 〲j 〿j ゕj ゗j ㄪj ㄫj ㄬj ㄰j 㐲j 㑊j 㑮j 㒓j 㓜j 㓡j 㔆j 㔉j 㔷j 㔻j 㕚j 㕸j 㖊j 㖬j 㖽j 㗋j 㗜j 㗭j 㘧j 㘭j 㙓j 㚻j 㞻j 㟭j 㠙j 㠝j 㡜j 㣇j 㥁j 㦰j 㧊j 㨗j 㨪j 㪚j 㫩j 㫪j 㬤j 㬭j 㭁j 㭨j 㮈j 㮋j 㮻j 㲾j 㳸j 㴀j 㴅j 㴬j 㴷j 㵾j 㶽j 㷋j 㹢j 㺅j 㺢j 㺯j 㺸j 㺹j 㻜j 㻬j 㼿j 㽓j 㿂j 㿭j 䀚j 䁢j 䂀j 䅟j 䈀j 䈂j 䉋j 䌸j 䎬j 䎽j 䏻j 䐃j 䐰j 䑓j 䒄j 䕬j 䗚k 䘌k 䙚k 䙡k 䚰k 䜭k 䝟k 䝨k 䟐k 䟕 k 䟛 k 䟦 k 䠠 k 䡇 k 䡛k 䢴k 䥸k 䥾k 䦵k 䨋k 䨱k 䫜k 䬴k 䱇k 䳒k 䴝k 䵋k 䶕k ䷢k ䷬k ䷴k 伕k 倣 k 僉!k 僥"k 儅#k 儉$k 儼%k 剮&k 匱'k 啣(k 嘸)k 噓*k 嚦+k 囈,k 塢-k 奐.k 姍/k 婭0k 媧1k 媯2k 嬸3k 屜4k 峴5k 崳6k 嶄7k 嶇8k 嶸9k 幗:k 廱;k 徠k 慟?k 慳@k 扐Ak 撈Bk 擋Ck 攏Dk 攤Ek 斃Fk 朮Gk 杇Hk 椏Ik 楨Jk 槳Kk 樁Lk 櫝Mk 殯Nk 毆Ok 渙Pk 溼Qk 澆Rk 濺Sk 瀅Tk 瀉Uk 瀋Vk 灑Wk 煢Xk 熾Yk 燜Zk 燬[k 牪\k 犧]k 玆^k 癆_k 癇`k 癟ak 瞇bk 磧ck 礬dk 窪ek 簷fk 絛gk 綢hk 綵ik 緡jk 縈kk 縚lk 繃mk 纍nk 纜ok 罌pk 羆qk 莧rk 葦sk 葯tk 蔦uk 藹vk 蠶wk 袞xk 襠yk 訃zk 詡{k 詼|k 誡}k 誨~k 諄k 謄k 謔k 謚k 譎k 讒k 貲k 賂k 賡k 贗k 趲k 躕k 躦k 軼k 輅k 轟k 辮k 鄲k 酈k 鈀k 鈷k 鈿k 鉦k 鉬k 鉸k 銑k 銖k 銲k 鋰k 鎬k 鐫k 鑿k 閏k 閾k 闃k 顰k 飩k 飴k 饉k 駘k 騏k 騖k 騾k 驄k 驊k 骯k 魎k 鯤k 鯫k 鱉k 鴆k 鴿k 鷳k 鸕k 鸝k 齣k 龨k 龹k ꀀk ꀃk ꀣk ꀨk ꀭk ꀮk ꀲk ꀵk ꁹk ꂃk ꂄk ꂘk ꂮk ꂻk ꂾk ꃆk ꃨk ꃲk ꃸk ꄀk ꄑk ꄜk ꄩk ꄳk ꄴk ꄶk ꄾk ꅀk ꅋk ꆗk ꆚk ꆠk ꆤk ꆨk ꇅk ꇫk ꇺk ꈂk ꈑk ꈞk ꈿk ꉓk ꉖk ꉠk ꉷk ꉺk ꉾk ꊜk ꊨk ꊱk ꊸk ꋋk ꋓk ꋣk ꋬk ꋺk ꌀk ꌑk ꌙk ꌥk ꌷk ꌸk ꍂk ꍈk ꍯk ꍸk ꎁk ꎤk ꎫk ꎸk ꎻk ꎽl ꏑl ꏪl ꏱl ꐓl ꐔl ꐕl ꐱl ꐷl ꐺ l ꐼ l ꑘ l ꑣ l ꑸ l ꒆl ꒈl ꒌl ꒡l ꒶l ꒿l ꕀl ꗯl ꙩl ꙫl ꙭl ꝼl ꞊l ꞔl ꟼl ꡊl ꡓl ꦫl ꦶ l ꦸ!l ꦽ"l ꪶ#l ꭒ$l 갃%l 갛&l 걔'l 겄(l 굵)l 굶*l 깅+l 깇,l 깬-l 깼.l 꾻/l 꿰0l 끓1l 끕2l 냇3l 넋4l 넲5l 넵6l 녘7l 댐8l 덱9l 뎅:l 돔;l 됬l 땠?l 뗄@l 뜃Al 뜬Bl 뜯Cl 뜸Dl 랗El 랴Fl 럿Gl 롸Hl 룻Il 뤃Jl 륵Kl 맏Ll 먃Ml 묶Nl 뮈Ol 밋Pl 벵Ql 벹Rl 볕Sl 봃Tl 빕Ul 빤Vl 빽Wl 뺑Xl 뻤Yl 뻥Zl 삗[l 삾\l 샬]l 샴^l 섞_l 셑`l 셪al 솽bl 숏cl 숑dl 쉰el 슉fl 슌gl 슐hl 싯il 쌔jl 썽kl 쒼ll 씽ml 엎nl 옴ol 욥pl 윷ql 잣rl 젡sl 쥔tl 쥘ul 쥴vl 쨍wl 쩳xl 쪘yl 쫀zl 찡{l 찻|l 챌}l 챱~l 첼l 쳄l 촛l 츨l 츰l 칫l 캉l 켠l 콥l 콧l 콸l 쿤l 퀌l 퀵l 텨l 툼l 튠l 튤l 팸l 팽l 핌l 핏l 핚l 핳l 핼l 헙l 헯l 헸l 헺l 혆l 홰l 훑l 훗l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l l m m m m m m m m m  m  m  m  m  m m m m m m m m m m m m m m m m m m m  m !m "m #m $m %m &m 'm (m )m *m +m ,m -m .m /m 0m 1m 2m 3m 4m 5m 6m 7m 8m 9m :m ;m m ?m @m Am Bm Cm Dm Em Fm Gm Hm Im Jm Km Lm Mm Nm Om Pm Qm Rm Sm Tm Um Vm Wm Xm Ym Zm [m \m ]m ^m _m `m am bm cm dm em fm gm hm im jm km lm mm nm om pm qm rm sm tm um vm wm xm ym zm {m |m }m ~m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m n n n n ︢n ︣n ﹆n 𐀷n 𐄙n 𐊧 n 𐍉 n 𐐌 n 𐐐 n 𐐤 n 𐤁n 𐤂n 𐤊n 𐤍n 𐤏n 𐬞n 𐭓n 𑀓n 𑄋n 𑄞n 𑄟n 𑄦n 𑄳n 𑐏n 𑐔n 𑐫n 𑐴n 𑐾n 𑑁 n 𒂍!n 𒂗"n 𒄀#n 𒆜$n 𒊏%n 𒋢&n 𒍪'n 𓂆(n 𓃹)n 𓅓*n 𓅰+n 𓆊,n 𓆌-n 𓆏.n 𓍯/n 𖤣0n 𖥣1n 𖦞2n 𖧷3n 🀀4n 🀁5n 🀂6n 🀃7n 🀄8n 🀅9n 🀇:n 🀈;n 🀉n 🀌?n 🀍@n 🀎An 🀏Bn 🀐Cn 🀑Dn 🀒En 🀓Fn 🀔Gn 🀕Hn 🀖In 🀗Jn 🀘Kn 🀙Ln 🀚Mn 🅕Nn 🅷On 🅹Pn 🆒Qn 🆔Rn 🌁Sn 🌜Tn 🍢Un 🍥Vn 🎛Wn 👚Xn 👢Yn 👳Zn 💠[n 💹\n 📤]n 🔂^n 🖉_n 🚅`n 🚉an 🚊bn 🚝cn 🚥dn 🚭en 🚽fn 🚿gn 🛷hn 🟧in 🟪jn 🠊kn 🠨ln 🢂mn 🤿nn 🥏on 🥠pn 🥷qn 🥽rn 🦝sn 🦟tn 🦰un 🦱vn 🦲wn 🦽xn 🧰yn 🧲zn 🧷{n 🪀|n 🪙}n 🪟~n 🪣n 🪩n 🪴n 🫓n 𠂉n 𠃍n 𠃵n 𠄘n 𠔽n 𠩤n 𠶧n 𡎚n 𡘊n 𡿨n 𢭃n 𢳂n 𤦺n 𤫊n 𥚃n 𦦨n 𦧲n 𧒽n 𧙕n 𨂃n 𪚥n 𪟝n 𬇕n 󰀀n 󱐂n 􏰠n ƃn Ʋn Ǖn Ǘn Ǥn Ǭn Ƕn Ȅn Ȍn Ȕn Ȝn Ƞn ȡn Ȭn ȴn ȸn ʢn ʫn ʬn ˫n ˯n ˿n ͱn Ϳn ΍n Ϣn Ϸn Ѫn ѻn ҃n ҅n ҍn ҏn Ҵn Ҹn ҹn Ӊn Ӡn ӡn Ӯn Ӽn ԇn Ԙn ԣn ԭn ԯn ՠn ֍n ֘n ֟n ֪n ׈n ׌n ׏n ׭n ׺n ׽n ؁n ؇n ؈n ؉n ؓn ؖn ؞n ؿn ٙn ٟn ٯn ٳn ٺn ڇn ڋn ڎn ڜn ڝn ڞn ۨn ۼn ܉n ܱn ܲn ݀n ݌n ݕn ޛn ޣn ޴n ޹n ޺n ޿n ߅n ߓn ߔn ߙn ߚn ߛn ߟn ߡn ߢn ߥo ߫o ߱o ߴo ߷o ߸o ߿o ࠀo ࠐo ࠕ o ࠞ o ࠤ o ࠬ o ࠳ o ࠴o ࠶o ࡏo ࡩo ࡵo ࡹo ࢃo ࢈o ࢌo ࢝o ࢥo ࢩo ࢲo ࢳo ࢸo ࣙo ࣟo ࣡o ࣪ o ࣬!o ࣽ"o ॓#o ॔$o ॠ%o ॾ&o ঌ'o ঩(o ঵)o ৄ*o ৅+o ৡ,o ৽-o ਋.o ਴/o ੄0o ੖1o ੦2o ੷3o ੼4o ੽5o ઍ6o ઎7o ઔ8o ઙ9o ઻:o ઽ;o ૐo ૤?o ૱@o ଋAo ଔBo ଺Co ୅Do ୗEo ୛Fo ୲Go ୴Ho ୹Io ஋Jo ஔKo ஬Lo ஭Mo ௌNo ௎Oo ௱Po ௹Qo ౗Ro ౙSo ౢTo ౴Uo ౺Vo ಊWo ೚Xo ഊYo ഛZo ഼[o ൉\o ൐]o ൛^o ൜_o ൪`o ൳ao ൹bo ඓco ඲do ඵeo ෈fo ෬go ෷ho ෻io ฾jo ๜ko ๟lo ๵mo ๼no ๽oo ຬpo ໋qo ໑ro ໕so ໯to ཭uo ྌvo ྍwo ྴxo ྸyo ྻzo ࿀{o ࿁|o ࿄}o ࿇~o ࿏o ࿔o ࿕o ࿱o ࿷o ࿸o ဎo ၈o ၞo ၠo ၦo ၱo ၲo ၵo ၶo ၼo ႁo ႑o Ⴂo Ⴛo Ⴝo Ⴥo ჲo ჶo ჻o ᄡo ᄣo ᄬo ᄮo ᅃo ᅊo ᅌo ᅎo ᅻo ᆄo ᆓo ᆝo ᆡo ᆬo ᆷo ᇗo ᇚo ᇩo ᇺo ሢo ሺo ሼo ቑo ቖo ቘo ቝo ቧo ቬo ቺo ቾo ኀo ኆo ኏o ኚo ኧo ኲo ኹo ኼo ኾo ዆o ዪo ዷo ጄo ጢo ጯo ጲo ፑo ፔo ፥o ፨o ፬o ፭o ፱o ፳o ፺o ፻o ᎖o Ꮊo Ꮕo Ꮜo Ꮡo Ꮴo Ꮵo ᏽo ᐍo ᐑo ᐗo ᐘo ᐜo ᐡo ᑂo ᑈo ᑏo ᑞo ᑡo ᑤo ᒀo ᒆo ᒩo ᒷo ᓋo ᓘo ᓙo ᓭo ᓮo ᔐo ᔓo ᔦo ᔫo ᔲo ᕀo ᕉo ᕋo ᕌo ᕒo ᕔo ᕖo ᕝo ᕪo ᕮo ᕰo ᕳo ᕿo ᖄo ᖔp ᖕp ᖽp ᖾp ᗋp ᗎp ᗒp ᗔp ᗕp ᗧ p ᗷ p ᗺ p ᘃ p ᘐ p ᘜp ᘰp ᘳp ᘺp ᙅp ᙓp ᙜp ᙢp ᙮p ᙿp ᚔp ᚕp ᚮp ᚲp ᚿp ᛂp ᛃp ᛇp ᛕ p ᛠ!p ᛡ"p ᛭#p ᜀ$p ᜁ%p ᜋ&p ᜗'p ᜴(p ᝎ)p ᝡ*p ឍ+p ឲ,p ៚-p ៩.p ៴/p ៹0p ᠼ1p ᡢ2p ᡤ3p ᡳ4p ᡴ5p ᡻6p ᢚ7p ᣑ8p ᤈ9p ᤤ:p ᤿;p ᥉

    p ᥷?p ᦇ@p ᦒAp ᦮Bp ᧥Cp ᨆDp ᨊEp ᨔFp ᨗGp ᨚHp ᨜Ip ᩓJp ᪋Kp ᫂Lp ᫹Mp ᬅNp ᬘOp ᬦPp ᬪQp ᬰRp ᬲSp ᬳTp ᬵUp ᭨Vp ᮟWp ᯖXp ᯮYp ᯲Zp ᯿[p ᰍ\p ᱅]p ᱼ^p Ი_p Ს`p Უap Შbp Ცcp Ჳdp ᳪep ᴝfp ᴨgp ᵭhp ᵲip ᵿjp ᶃkp ᶌlp ᶐmp ᷝnp ᷾op ᷿pp Ḉqp Ḓrp Ḕsp Ḭtp Ḯup ḯvp Ḽwp Ḿxp Ṅyp Ṋzp Ṏ{p ṏ|p Ṙ}p Ṝ~p ṝp Ṧp Ṩp ṵp ṻp Ẁp Ẅp Ẋp Ẍp Ằp Ỉp Ỡp Ỵp Ỷp ἟p ἣp Ἶp Ὂp ὒp ὓp ὧp Ὥp Ὦp ᾁp ᾅp ᾈp ᾍp ᾦp ῧp ῰p ῲp ⁅p ⁌p ⁔p ⁘p p p ⁲p ₠p ₥p ₼p ⃍p ⃟p ℈p ↜p ↱p ↸p ↽p ⇁p ⇶p ⇸p ∢p ∦p ∹p ∾p ≁p ≄p ≉p ≎p ≏p ≔p ≘p ≛p ≜p ≝p ≞p ≭p ≰p ≸p ⊀p ⊁p ⊈p ⊉p ⊍p ⊓p ⊣p ⊯p ⊲p ⊸p ⊺p ⊻p ⊽p ⋇p ⋋p ⋌p ⋎p ⋐p ⋑p ⋔p ⋡p ⋦p ⋧p ⋪p ⋭p ⌅p ⌌p ⌍p ⌎p ⌏p ⌖p ⌦p ⌧p ⌲p ⍘p ⍣p ⍯p ⍹p ⎊p ⎌p ⎘p ⎲p ⎸p ⎹p ⏀p ⏆p ⏣p ⏦p ⏴p ⏵p ␕p ␝p ␠p ␴p ␾p ⑂p ⑊p ⓹p ┩p ┰p ┱q ┸q ┺q ┽q ┿q ╀q ╁q ╃q ╓q ╙ q ▗ q ▛ q ◛ q ◸ q ◹q ◺q ☖q ☗q ☡q ☨q ☫q ♁q ♲q ♳q ♺q ♼q ⚂q ⚉q ⚞q ⚟q ⚢q ⚤q ⚥ q ⚦!q ⚯"q ⚳#q ⚷$q ⚿%q ⛆&q ⛉'q ⛭(q ⛶)q ✣*q ❴+q ❵,q ⟟-q ⟴.q ⟿/q ⠅0q ⠉1q ⠌2q ⠍3q ⠐4q ⠖5q ⠘6q ⠜7q ⠡8q ⠢9q ⠣:q ⠤;q ⠨q ⠬?q ⠭@q ⠮Aq ⠰Bq ⠲Cq ⠶Dq ⠺Eq ⠻Fq ⠽Gq ⠿Hq ⡷Iq ⢠Jq ⢾Kq ⣎Lq ⣸Mq ⣻Nq ⣼Oq ⤃Pq ⤊Qq ⤎Rq ⤐Sq ⤒Tq ⤔Uq ⤥Vq ⤧Wq ⤶Xq ⥇Yq ⥮Zq ⦊[q ⦥\q ⧍]q ⧎^q ⧚_q ⧧`q ⧹aq ⨀bq ⨂cq ⨺dq ⩓eq ⪢fq ⫌gq ⫫hq ⬄iq ⬉jq ⬍kq ⬏lq ⬑mq ⬓nq ⬔oq ⬕pq ⬖qq ⬗rq ⬙sq ⬚tq ⬞uq ⬟vq ⬠wq ⬦xq ⬧yq ⬨zq ⬪{q ⬫|q ⬮}q ⬰~q ⬱q ⬲q ⬳q ⬴q ⬵q ⬶q ⬷q ⬸q ⬹q ⬺q ⬻q ⬼q ⬽q ⬾q ⭒q ⭳q ⮌q ⮑q ⮚q ⮝q ⮡q ⮢q ⮤q ⮱q ⯆q ⯇q Ⰲq Ⰽq Ⰾq Ⱀq Ⱁq Ⱏq Ⱐq Ⱑq Ⱔq Ⱖq Ᵽq ⱥq ⱦq Ⱪq Ɑq ⱴq Ⲩq ⳷q ⴠq ⴢq ⴭq ⴲq ⴸq ⴽq ⴾq ⵀq ⵁq ⵛq ⵦq ⵳q ⸆q ⸉q ⸴q ⸻q ⹂q ⹴q ⺇q ⺗q ⺧q ⺴q ⺶q ⻻q ⿲q ⿳q ⿵q ⿶q ⿹q 〟q 〴q ㄀q ㄯq ㆪq ㆳq ㆺq ㆼq ㆽq ㇀q ㇑q ㇔q ㇕q ㇷq ㉿q 㐀q 㐊q 㐦q 㑀q 㑛q 㑦q 㑨q 㑬q 㑰q 㑳q 㒞q 㒷q 㓋q 㓘q 㓛q 㓟q 㓲q 㔐q 㔓q 㔴q 㕃q 㕊q 㕋q 㕕q 㕜q 㕡q 㕵q 㖀q 㖏q 㖓q 㗂q 㗅q 㗉q 㗊q 㗏q 㗙q 㗰q 㗷q 㘐q 㘓q 㘝q 㘞r 㙶r 㚢r 㚣r 㚭r 㚵r 㚿r 㜀r 㜠r 㜷 r 㜸 r 㝅 r 㝢 r 㝱 r 㞓r 㞘r 㞢r 㞮r 㞳r 㞹r 㟂r 㟷r 㟺r 㠀r 㠊r 㠰r 㢏r 㢡r 㢢r 㢧r 㢮r 㤅r 㤉 r 㤕!r 㤙"r 㤚#r 㤞$r 㤴%r 㤵&r 㥐'r 㥘(r 㥬)r 㦀*r 㦣+r 㦤,r 㦮-r 㦯.r 㧍/r 㧏0r 㧐1r 㧑2r 㧴3r 㧵4r 㧽5r 㨆6r 㨉7r 㨖8r 㨤9r 㩁:r 㩂;r 㩉r 㩝?r 㩠@r 㩦Ar 㪍Br 㪰Cr 㪶Dr 㫬Er 㫺Fr 㬌Gr 㬎Hr 㬪Ir 㬹Jr 㬿Kr 㭇Lr 㭍Mr 㭜Nr 㮇Or 㮐Pr 㮟Qr 㮠Rr 㮣Sr 㮮Tr 㮵Ur 㮷Vr 㮹Wr 㮽Xr 㯋Yr 㯖Zr 㯠[r 㰖\r 㰞]r 㱎^r 㱽_r 㱿`r 㲠ar 㲪br 㲺cr 㳀dr 㳁er 㳕fr 㳠gr 㴇hr 㴔ir 㴪jr 㴭kr 㴰lr 㵋mr 㵖nr 㵟or 㵮pr 㵵qr 㵿rr 㶉sr 㶏tr 㶤ur 㶰vr 㶲wr 㶷xr 㷆yr 㷧zr 㷬{r 㷻|r 㸐}r 㸢~r 㸦r 㸿r 㹃r 㹌r 㹦r 㹧r 㹭r 㻆r 㼜r 㼭r 㼼r 㽬r 㾕r 㾗r 㾫r 㾹r 㿃r 䀪r 䀲r 䀾r 䀿r 䁊r 䁐r 䂓r 䃄r 䃥r 䃧r 䄃r 䄍r 䄦r 䄪r 䅲r 䆉r 䆖r 䆡r 䆲r 䆳r 䇖r 䇹r 䉏r 䉕r 䉛r 䊎r 䊨r 䋎r 䋣r 䋧r 䋲r 䋸r 䋾r 䌀r 䌇r 䌊r 䌠r 䌹r 䍃r 䍘r 䍦r 䍩r 䍺r 䎱r 䏂r 䏄r 䏈r 䏓r 䏚r 䏮r 䏸r 䐠r 䐢r 䐯r 䐱r 䐳r 䑉r 䑔r 䒔r 䒗r 䒥r 䓁r 䓢r 䓴r 䔄r 䔰r 䕌r 䕒r 䕰r 䕶r 䕷r 䖘r 䖝r 䖪r 䗖r 䗛r 䘿r 䙓r 䚧r 䛁r 䛒r 䛝r 䜆r 䜉r 䝁r 䝗r 䝙r 䝷r 䞼r 䟉r 䟒r 䠌r 䠙r 䠩r 䠰r 䠷r 䢋r 䢎r 䢒r 䣈r 䣋r 䣒r 䣔r 䤐r 䤠r 䤥r 䤹r 䥌r 䥚r 䥱r 䧚r 䨍r 䨧s 䨼s 䩄s 䩮s 䩶s 䪨s 䪽s 䬝s 䬰s 䬻 s 䭔 s 䭺 s 䭾 s 䮭 s 䯙s 䯝s 䰀s 䰅s 䰖s 䰫s 䱂s 䱍s 䱒s 䲞s 䳍s 䳕s 䳗s 䳘s 䳨s 䴕s 䴖s 䴚s 䵈 s 䵔!s 䵟"s 䵵#s 䵻$s 䶩%s ䷊&s ䷚'s ䷝(s ䷮)s ䷯*s 亙+s 儂,s 儔-s 儕.s 儸/s 儺0s 冪1s 剄2s 剷3s 勦4s 卹5s 嗆6s 嗇7s 嗩8s 嗶9s 噯:s 噸;s 嚀s 墑?s 壚@s 壟As 壩Bs 嫻Cs 嬝Ds 嬤Es 嬪Fs 嶧Gs 巒Hs 幃Is 幘Js 幬Ks 廄Ls 廡Ms 悵Ns 慇Os 憒Ps 憫Qs 懌Rs 懞Ss 懣Ts 戇Us 戧Vs 扠Ws 搆Xs 摳Ys 撣Zs 撻[s 撾\s 擣]s 擲^s 攣_s 檣`s 櫥as 櫨bs 欞cs 殫ds 氈es 氬fs 氳gs 浹hs 淥is 漚js 潁ks 潯ls 潷ms 澇ns 瀝os 熗ps 燁qs 燴rs 犢ss 獮ts 獺us 璦vs 瓔ws 瓚xs 痙ys 痳zs 瘓{s 瘧|s 癉}s 癭~s 癰s 眥s 矯s 礪s 礫s 秈s 稈s 箄s 篳s 簍s 簑s 簣s 籐s 糝s 糴s 紆s 紱s 綹s 綽s 緘s 緝s 緱s 縊s 縵s 縷s 縹s 罈s 羥s 脛s 臍s 臏s 萇s 葷s 蓀s 蓽s 蔆s 蕕s 蕘s 蕷s 蘄s 蘗s 褸s 襖s 覘s 覯s 覲s 覿s 訕s 訛s 訥s 詎s 詒s 詘s 詫s 詬s 詵s 誅s 誚s 誹s 諂s 諍s 諛s 諞s 謗s 謠s 謬s 譙s 讎s 貰s 賄s 賅s 賒s 賾s 贄s 踡s 踴s 蹌s 蹕s 蹺s 躋s 軫s 軻s 輓s 輜s 輳s 輾s 轂s 轎s 轤s 邇s 醃s 醞s 釁s 釧s 釬s 鈹s 鉀s 鉍s 鉚s 銬s 銻s 鋃s 鋅s 鋮s 鋯s 錆s 錙s 錮s 鍥s 鎘s 鎦s 鎳s 鏑s 鏞s 鐺s 鑠s 閂s 閔s 閡s 閹s 闈s 闌s 阨s 陝s 隄s 隕s 靨s 鞏s 韃s 韆t 頷t 顓t 顥t 顱t 颼t 飆t 餌t 餛t 餾 t 餿 t 饈 t 饑 t 騁 t 騫t 騮t 騶t 驀t 驤t 驪t 鬍t 鬨t 鬮t 魷t 鮐t 鮪t 鯀t 鯪t 鯽t 鰾t 鱈t 鱒t 鱟 t 鱷!t 鳧"t 鴟#t 鴣$t 鵠%t 鶉&t 鶚't 鷓(t 鷥)t 鷴*t 鷸+t 鸛,t 鼇-t 鼉.t 龧/t 龷0t ꀈ1t ꀥ2t ꀱ3t ꀹ4t ꀺ5t ꁈ6t ꁉ7t ꁊ8t ꁍ9t ꁙ:t ꁞ;t ꁠt ꁵ?t ꂊ@t ꂐAt ꂒBt ꂪCt ꂯDt ꂴEt ꂶFt ꃘGt ꃤHt ꃮIt ꃳJt ꃺKt ꄁLt ꄓMt ꄘNt ꄙOt ꄞPt ꄠQt ꄵRt ꄺSt ꄻTt ꄿUt ꅈVt ꅊWt ꅒXt ꅰYt ꅶZt ꅾ[t ꆣ\t ꇋ]t ꇎ^t ꇕ_t ꇖ`t ꇩat ꇳbt ꈅct ꈏdt ꈬet ꈭft ꈯgt ꈳht ꈹit ꈺjt ꈽkt ꉁlt ꉅmt ꉏnt ꉗot ꉻpt ꊁqt ꊛrt ꊥst ꊬtt ꊷut ꋂvt ꋙwt ꋛxt ꋧyt ꋴzt ꌁ{t ꌅ|t ꌇ}t ꌈ~t ꌕt ꌜt ꌢt ꌭt ꌶt ꍣt ꍤt ꍧt ꍿt ꎍt ꎖt ꎮt ꏠt ꏡt ꏣt ꏳt ꐁt ꐈt ꐉt ꐋt ꐟt ꐣt ꑑt ꑖt ꑝt ꑤt ꑶt ꑼt ꒊt ꒙t ꓃t ꓥt ꗄt ꙁt ꙑt Ꙗt Ꙭt ꙮt Ꚋt ꚍt ꜛt Ꜫt ꜫt ꝑt Ꝛt ꝫt ꞇt ꞈt ꞎt ꞓt ꞵt Ꞔt Ʂt ꟕt ꟿt ꠵t ꡛt ꡟt ꦂt ꦏt ꦒt ꦗt ꦚt ꦤt ꦧt ꦩt ꦪt ꦮt ꦲt ꧒t ꩒t ꩜t ꪀt ꪅt ꪑt ꪖt ꪜt ꪣt ꪤt ꪰt ꬶt ꭢt ꯇt ꯟt ꯣt ꯤt 갬t 걀t 겋t 겔t 곶t 괌t 굼t 귓t 긱t 깡t 깥t 깸t 껀t 껄t 껑t 껬t 꽝t 꽸t 꿉t 꿎t 꿘t 뀀t 뀛t 뀜t 뀠t 뀽t 끉t 끽t 낍t 낱t 냘t 냬t 넀t 넙t 넳t 넸t 넹t 넼t 녔t 놯t 뇛t 눇t 눴t 뉘u 뉜u 늪u 닋u 댁u 돐u 돗u 돛u 됀u 됍 u 둣 u 둬 u 딊 u 딻 u 땋u 떄u 떳u 떴u 똈u 뚤u 뛴u 뜩u 띤u 랏u 렙u 렝u 뤘u 맠u 맬u 멓u 멥u 몄u 몏 u 묾!u 뭩"u 뮨#u 밫$u 뱁%u 봣&u 뵐'u 뵜(u 뵵)u 뺐*u 뼘+u 뿍,u 뿔-u 뿚.u 쀼/u 삔0u 샽1u 샾2u 섶3u 셶4u 솓5u 숄6u 숴7u 쉔8u 쉘9u 슭:u 쌕;u 썹u 쏜?u 쏴@u 쐬Au 쐼Bu 쑹Cu 쓡Du 씌Eu 씰Fu 앍Gu 얌Hu 얽Iu 엥Ju 엮Ku 엷Lu 옭Mu 옻Nu 왈Ou 왑Pu 욃Qu 욧Ru 욯Su 웍Tu 윔Uu 잦Vu 잴Wu 쟤Xu 젓Yu 젬Zu 좆[u 죗\u 죤]u 줏^u 짊_u 짖`u 짰au 쫄bu 쫑cu 쬐du 쬬eu 쭥fu 쮸gu 찹hu 찼iu 챈ju 첟ku 쳇lu 쳔mu 쳮nu 촘ou 촣pu 촨qu 쵸ru 춧su 췄tu 췌uu 캅vu 캤wu 캥xu 컫yu 켈zu 콰{u 콱|u 쾨}u 쿡~u 큘u 탰u 텅u 텟u 텼u 툭u 퉌u 튈u 틋u 팍u 팎u 팜u 팠u 팹u 퍽u 펍u 펐u 펨u 펩u 폈u 퐀u 퐄u 퐆u 퐈u 퐋u 퐑u 퐻u 퐾u 푒u 푗u 푚u 푛u 픕u 햅u 햬u 헕u 헱u 헾u 혠u 홧u 횞u 휋u 힉u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u u v v v v v v v v v  v  v  v  v  v v v v v v v v v v v v v v v v v v v  v !v "v #v $v %v &v 'v (v )v *v +v ,v -v .v /v 0v 1v 2v 3v 4v 5v 6v 7v 8v 9v :v ;v v ?v @v Av Bv Cv Dv Ev Fv Gv Hv Iv Jv Kv Lv Mv Nv Ov Pv Qv Rv Sv Tv Uv Vv Wv Xv Yv Zv [v \v ]v ^v _v `v av bv cv dv ev fv gv hv iv jv kv lv mv nv ov pv qv rv sv tv uv vv wv xv yv zv {v |v }v ~v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v w w w w w w w w w  w  w  w  w  w w w w w w w w w w w w w w w w w w wtokenizer.model U?XÓ_ 

    

    

      
    
    
  • 
  • 
    • 

    

    

    
    
    
    

    

    

    
    
    
    

    

    

    

    

    

    ０１２３４５６７８９+-=，。！？、：￥.!?... 。。。。。。。。。《》【】『』```---                                                                                                      identity pengcheng.mind-mindformer/scripts/hf_7B_shared/tokenizer_config.json000066400000000000000000000014331476650141400263070ustar00rootroot00000000000000{ "auto_map": { "AutoTokenizer": ["tokenization_baichuan.BaichuanTokenizer", null] }, "add_bos_token": false, "add_eos_token": false, "use_fast": false, "clean_up_tokenization_spaces": false, "eos_token": { "__type": "AddedToken", "content": "    ", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true }, "model_max_length": 3072, "sp_model_kwargs": {}, "tokenizer_class": "BaichuanTokenizer", "pad_token": { "__type": "AddedToken", "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true }, "unk_token": { "__type": "AddedToken", "content": "", "lstrip": false, "normalized": true, "rstrip": false, "single_word": true } } pengcheng.mind-mindformer/scripts/hf_inference_mmlu.py000066400000000000000000000012201476650141400236130ustar00rootroot00000000000000import torch from opencompass.models import HuggingFaceCausalLM path = "/mgData4/models/naohai-2B_en_exp02-1_ppl-07_367B" models = [ dict( type=HuggingFaceCausalLM, abbr='naohai-2B_en_exp02-1_ppl-07_367B', path=path, tokenizer_path=path, model_kwargs=dict( trust_remote_code=True, device_map='auto', ), tokenizer_kwargs=dict( padding_side='left', truncation_side='left', trust_remote_code=True, ), max_out_len=100, max_seq_len=4096, batch_size=256, run_cfg=dict(num_gpus=1, num_procs=1), ) ] pengcheng.mind-mindformer/scripts/ms_to_torch-shared.py000066400000000000000000000054231476650141400237430ustar00rootroot00000000000000import torch import numpy as np import mindspore as ms print("loading mindspore ckpt.................") ms_ckpt = ms.load_checkpoint('/cache/dst_ckpt/rank_0/checkpoint_0.ckpt') old = ms.load_checkpoint('/cache/src_ckpt/0.ckpt') t0 = torch.Tensor(old['model.tok_embeddings.word_embedding.embedding_table'].numpy()).to(dtype=torch.float16) old = ms.load_checkpoint('/cache/src_ckpt/1.ckpt') t1 = torch.Tensor(old['model.tok_embeddings.word_embedding.embedding_table'].numpy()).to(dtype=torch.float16) torch_dict = {} torch_dict['model.embed_tokens.weight'] = torch.concat((t0, t1), dim=0).to(dtype=torch.float16) for i in range(32): print(f"trans layer"+str(i)+".................") torch_dict['model.layers.'+str(i)+'.attention_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.post_attention_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.post_attention_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.ffn_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.ffn_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.post_ffn_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.post_ffn_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.gate_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w1.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.down_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w2.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.up_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w3.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.o_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wo.weight'].numpy()).to(dtype=torch.float16) q=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wq.weight'].numpy()).to(dtype=torch.float16) k=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wk.weight'].numpy()).to(dtype=torch.float16) v=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wv.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.W_pack.weight']=torch.cat([q,k,v],dim=0) torch_dict['model.norm.weight'] = torch.Tensor(ms_ckpt['model.norm_out.weight'].numpy()).to(dtype=torch.float16) torch_dict['lm_head.weight'] = torch.concat((t0, t1), dim=0).to(dtype=torch.float16) print("saving pytorch ckpt.................") torch.save(torch_dict,'/cache/pytorch_model.bin')pengcheng.mind-mindformer/scripts/ms_to_torch.py000066400000000000000000000063471476650141400225050ustar00rootroot00000000000000import torch import mindspore as ms import argparse import moxing as mox if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument( '--obs_merged_ckpt', type=str, help='') parser.add_argument( '--obs_hf_ckpt_save_dir', type=str, help='') args_opt = parser.parse_args() mox.file.copy(args_opt.obs_merged_ckpt, "/cache/dst_ckpt/rank_0/checkpoint_0.ckpt") print("loading mindspore ckpt.................") ms_ckpt = ms.load_checkpoint('/cache/dst_ckpt/rank_0/checkpoint_0.ckpt') torch_dict = {} torch_dict['model.embed_tokens.weight'] = torch.Tensor(ms_ckpt['model.tok_embeddings.embedding_weight'].numpy()).to(dtype=torch.float16) for i in range(40): print(f"trans layer"+str(i)+".................") torch_dict['model.layers.'+str(i)+'.attention_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.post_attention_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.post_attention_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.ffn_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.ffn_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.post_ffn_layernorm.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.post_ffn_norm.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.gate_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w1.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.down_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w2.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.mlp.up_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.feed_forward.w3.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.o_proj.weight'] = \ torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wo.weight'].numpy()).to(dtype=torch.float16) q=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wq.weight'].numpy()).to(dtype=torch.float16) k=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wk.weight'].numpy()).to(dtype=torch.float16) v=torch.Tensor(ms_ckpt['model.layers.'+str(i)+'.attention.wv.weight'].numpy()).to(dtype=torch.float16) torch_dict['model.layers.'+str(i)+'.self_attn.W_pack.weight']=torch.cat([q,k,v],dim=0) torch_dict['model.norm.weight'] = torch.Tensor(ms_ckpt['model.norm_out.weight'].numpy()).to(dtype=torch.float16) torch_dict['lm_head.weight'] = torch.Tensor(ms_ckpt['lm_head.weight'].numpy()).to(dtype=torch.float16) print("saving pytorch ckpt.................") torch.save(torch_dict,'/cache/pytorch_model.bin') if not mox.file.exists(args_opt.obs_hf_ckpt_save_dir): mox.file.make_dirs(args_opt.obs_hf_ckpt_save_dir) mox.file.copy("/cache/pytorch_model.bin", args_opt.obs_hf_ckpt_save_dir + "/pytorch_model.bin")pengcheng.mind-mindformer/scripts/run_distribute.sh000077500000000000000000000136061476650141400232100ustar00rootroot00000000000000#!/bin/bash # Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 4 ] && [ $# != 5 ] && [ $# != 6 ] then echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] For Multiple Devices In Single Machine" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] [RUN_STATUS] [RANK_SIZE] For Multiple Devices In Multiple Machines" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] predict [PREDICT_DATA] For Multiple Devices Predict In Single Machine" echo "Usage Help: bash run_distribute.sh [RANK_TABLE_FILE] [CONFIG_PATH] [DEVICE_RANGE] predict [RANK_SIZE] [PREDICT_DATA] For Multiple Devices Predict In Multiple Machines" exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } PATH1=$(check_real_path $1) CONFIG_FILE=$(check_real_path $2) DEVICE_RANGE=$3 RUN_STATUS=$4 DEVICE_RANGE_LEN=${#DEVICE_RANGE} DEVICE_RANGE=${DEVICE_RANGE:1:DEVICE_RANGE_LEN-2} PREFIX=${DEVICE_RANGE%%","*} INDEX=${#PREFIX} START_DEVICE=${DEVICE_RANGE:0:INDEX} END_DEVICE=${DEVICE_RANGE:INDEX+1:DEVICE_RANGE_LEN-INDEX} if [ ! -f $PATH1 ] then echo "error: RANK_TABLE_FILE=$PATH1 is not a file" exit 1 fi if [ ! -f $CONFIG_FILE ] then echo "error: config_path=$CONFIG_FILE is not a file" exit 1 fi if [[ ! $START_DEVICE =~ ^[0-9]+$ ]]; then echo "error: start_device=$START_DEVICE is not a number" exit 1 fi if [[ ! $END_DEVICE =~ ^[0-9]+$ ]]; then echo "error: end_device=$END_DEVICE is not a number" exit 1 fi ulimit -u unlimited if [ $RUN_STATUS != "predict" ] then if [ $# == 4 ] then export RANK_SIZE=$(($END_DEVICE - $START_DEVICE)) else export RANK_SIZE=$5 fi else if [ $# == 5 ] then export RANK_SIZE=$(($END_DEVICE - $START_DEVICE)) PREDICT_DATA=$5 else export RANK_SIZE=$5 PREDICT_DATA=$6 fi fi export RANK_TABLE_FILE=$PATH1 export LOCAL_DEFAULT_PATH="../../output" export CHECKPOINT_DOWNLOAD_FOLDER="../../checkpoint_download" export CHECKPOINT_SAVE_FOLDER="../../checkpoint_save" shopt -s extglob if [ $RUN_STATUS != "predict" ] then if [ $# == 4 ] then for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done else for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done fi else if [ $# == 5 ] then for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export DEVICE_ID=${i} export RANK_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ --predict_data "$PREDICT_DATA" &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done else for((i=${START_DEVICE}; i<${END_DEVICE}; i++)) do export RANK_ID=${i} export DEVICE_ID=$((i-START_DEVICE)) rm -rf ./mf_parallel$i/!(rank_*) test -d ./mf_parallel$i || mkdir ./mf_parallel$i cp ../*.py ./mf_parallel$i cp -r ../configs ./mf_parallel$i cp -r ../mindformers ./mf_parallel$i cd ./mf_parallel$i || exit echo "start training for rank $RANK_ID, device $DEVICE_ID" env > env.log mkdir -p ../../output/log/rank_$RANK_ID python run_mindformer.py --config=$CONFIG_FILE --use_parallel=True --run_mode=$RUN_STATUS \ --predict_data "$PREDICT_DATA" &> ../../output/log/rank_$RANK_ID/mindformer.log & cd .. done fi fi shopt -u extglob #cd ./pretrain_parallel${START_DEVICE} || exit #tail -f mindformer.log # if you want kill current job, you can use as follow: # kill -9 $(ps aux | grep "python run_mindformer.py" | grep -v grep | awk '{print $2}') pengcheng.mind-mindformer/scripts/run_standalone.sh000077500000000000000000000031121476650141400231510ustar00rootroot00000000000000#!/bin/bash # Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ if [ $# != 3 ] then echo "Usage Help: bash run_standalone.sh [CONFIG_PATH] [DEVICE_ID] [RUN_STATUS] " exit 1 fi check_real_path(){ if [ "${1:0:1}" == "/" ]; then echo "$1" else echo "$(realpath -m $PWD/$1)" fi } CONFIG_FILE=$(check_real_path $1) DEVICE_ID=$2 RUN_STATUS=$3 if [ ! -f $CONFIG_FILE ] then echo "error: config_path=$CONFIG_FILE is not a file" exit 1 fi ulimit -u unlimited export DEVICE_ID=${DEVICE_ID} rm -rf ./mf_standalone mkdir ./mf_standalone cp ../*.py ./mf_standalone cp -r ../configs ./mf_standalone cp -r ../mindformers ./mf_standalone cd ./mf_standalone || exit echo "start training for device $DEVICE_ID" env > env.log python run_mindformer.py --config=$CONFIG_FILE --use_parallel=False --run_mode=$RUN_STATUS &> mindformer.log & cd .. # if you want kill current job, you can use as follow: # kill -9 $(ps aux | grep "python run_mindformer.py" | grep -v grep | awk '{print $2}')pengcheng.mind-mindformer/setup.py000077500000000000000000000132421476650141400176310ustar00rootroot00000000000000#!/usr/bin/env python3 # encoding: utf-8 # Copyright 2022 Huawei Technologies Co., Ltd.All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================== """Setup.""" import sys import os import shutil import stat import platform from importlib import import_module from setuptools import setup, find_packages from setuptools.command.egg_info import egg_info from setuptools.command.build_py import build_py from setuptools.command.install import install def get_configs_content(): pwd = os.path.dirname(os.path.realpath(__file__)) configs = [os.path.join(root, file) for root, _, file_list in os.walk(os.path.join(pwd, "configs")) for file in file_list if file.endswith('.yaml') or file.endswith(".yml") or file.endswith(".md")] return configs def get_readme_content(): pwd = os.path.dirname(os.path.realpath(__file__)) with open(os.path.join(pwd, 'README.md'), encoding='UTF-8') as f: return f.read() def get_platform(): """ Get platform name. Returns: str, platform name in lowercase. """ return platform.system().strip().lower() def get_description(): """ Get description. Returns: str, wheel package description. """ os_info = get_platform() cpu_info = platform.machine().strip() return 'mindformers platform: %s, cpu: %s' % (os_info, cpu_info) def get_install_requires(): """ Get install requirements. Returns: list, list of dependent packages. """ with open('requirements.txt') as file: return file.read().strip().splitlines() def update_permissions(path): """ Update permissions. Args: path (str): Target directory path. """ for dirpath, dirnames, filenames in os.walk(path): for dirname in dirnames: dir_fullpath = os.path.join(dirpath, dirname) os.chmod(dir_fullpath, stat.S_IREAD | stat.S_IWRITE | stat.S_IEXEC | stat.S_IRGRP | stat.S_IXGRP) for filename in filenames: file_fullpath = os.path.join(dirpath, filename) os.chmod(file_fullpath, stat.S_IREAD) class EggInfo(egg_info): """Egg info.""" def run(self): egg_info_dir = os.path.join(os.path.dirname(__file__), 'mindformers.egg-info') shutil.rmtree(egg_info_dir, ignore_errors=True) super().run() update_permissions(egg_info_dir) class BuildPy(build_py): """Build py files.""" def run(self): mindspore_transformer_lib_dir = os.path.join(os.path.dirname(__file__), 'build', 'lib', 'mindformers') shutil.rmtree(mindspore_transformer_lib_dir, ignore_errors=True) super().run() update_permissions(mindspore_transformer_lib_dir) class Install(install): """Install.""" def run(self): super().run() if sys.argv[-1] == 'install': pip = import_module('pip') mindspore_transformer_dir = os.path.join(os.path.dirname(pip.__path__[0]), 'mindformers') update_permissions(mindspore_transformer_dir) if __name__ == '__main__': version_info = sys.version_info if (version_info.major, version_info.minor) < (3, 7): sys.stderr.write('Python version should be at least 3.7\r\n') sys.exit(1) setup( name='mindformers', version='2023.09.dev', author='The MindSpore Authors', author_email='contact@mindspore.cn', url='https://www.mindspore.cn', download_url='https://gitee.com/mindspore/mindformers/tags', project_urls={ 'Sources': 'https://gitee.com/mindspore/mindformers', 'Issue Tracker': 'https://gitee.com/mindspore/mindformers/issues', }, description=get_description(), long_description=get_readme_content(), long_description_content_type="text/markdown", test_suite="tests", packages=find_packages(exclude=["*tests*"]), platforms=[get_platform()], include_package_data=True, package_data={'mindformers': get_configs_content()}, cmdclass={ 'egg_info': EggInfo, 'build_py': BuildPy, 'install': Install, }, python_requires='>=3.7', install_requires=get_install_requires(), classifiers=[ 'Development Status :: 5 - Production/Stable', 'Environment :: Console', 'Environment :: Web Environment', 'Intended Audience :: Science/Research', 'Intended Audience :: Developers', 'License :: OSI Approved :: Apache Software License', 'Programming Language :: Python :: 3 :: Only', 'Programming Language :: Python :: 3.7', 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: 3.9', 'Topic :: Scientific/Engineering', 'Topic :: Scientific/Engineering :: Artificial Intelligence', 'Topic :: Software Development', 'Topic :: Software Development :: Libraries', 'Topic :: Software Development :: Libraries :: Python Modules', ], license='Apache 2.0', keywords='mindformers', ) pengcheng.mind-mindformer/start.sh000077500000000000000000000004521476650141400176070ustar00rootroot00000000000000pkill python export RANK_TABLE_FILE=/home/ma-user/work/rank_table_128.json export RANK_ID=0 export DEVICE_ID=0 export RANK_SIZE=128 export GLOG_v=2 python run_mindformer.py --config configs/official_training/run_minicpm_1e3_128.yaml --train_dataset=/home/ma-user/work/dataset/eval_2048.mindrecord pengcheng.mind-mindformer/tests/000077500000000000000000000000001476650141400172545ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/__init__.py000066400000000000000000000000001476650141400213530ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/000077500000000000000000000000001476650141400177025ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/__init__.py000066400000000000000000000012451476650141400220150ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ST.""" pengcheng.mind-mindformer/tests/st/test_build.py000066400000000000000000000265741476650141400224300ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the interface used for mindformers. How to run this: python tests/ut/test_build.py """ import os from dataclasses import dataclass from typing import Callable import pytest from mindspore.nn import AdamWeightDecay, CosineDecayLR, Accuracy,\ TrainOneStepWithLossScaleCell, L1Loss from mindspore.train.callback import LossMonitor, TimeMonitor from mindspore import Parameter, Tensor from mindformers.tools import logger from mindformers.tools import MindFormerConfig, MindFormerRegister, MindFormerModuleType from mindformers.core import build_lr, build_optim, build_loss, build_metric from mindformers.trainer import build_trainer, BaseTrainer from mindformers.models import build_model, BaseModel, build_processor from mindformers.dataset import build_dataset, build_sampler, check_dataset_config, \ build_dataset_loader, build_mask, build_transforms, BaseDataset from mindformers.pipeline import build_pipeline from mindformers.wrapper import build_wrapper path = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) yaml_path = os.path.join(path, 'tests', 'st', 'test_build.yaml') all_config = MindFormerConfig(yaml_path) @MindFormerRegister.register(MindFormerModuleType.DATASET_LOADER) class TestDataLoader: """Test DataLoader API For Register.""" def __init__(self, dataset_dir=None): self.dataset_dir = dataset_dir @MindFormerRegister.register(MindFormerModuleType.DATASET_SAMPLER) class TestSampler: """Test Sampler API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TestTransforms1: """Test Transforms API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRANSFORMS) class TestTransforms2: """Test Transforms API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.MASK_POLICY) class TestModelMask: """Test Model Mask API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.MODULES) class TestAttentionModule: """Test Module API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.DATASET) class TestDataset(BaseDataset): """Test Dataset API For Register.""" def __init__(self, dataset_config: dict = None): super(TestDataset, self).__init__(dataset_config) self.config = dataset_config def __new__(cls, dataset_config: dict = None): if dataset_config is not None: build_dataset_loader(dataset_config.data_loader) logger.info("Test Build DataLoader Success") build_sampler(dataset_config.sampler) logger.info("Test Build Sampler Success") build_transforms(dataset_config.transforms) logger.info("Test Build Transforms Success") build_mask(dataset_config.mask_policy) logger.info("Test Build Mask Policy Success") else: build_dataset_loader(class_name='TestDataLoader') logger.info("Test Build DataLoader Success") build_sampler(class_name='TestSampler') logger.info("Test Build Sampler Success") build_transforms(class_name='TestTransforms1') logger.info("Test Build Transforms Success") build_mask(class_name='TestModelMask') logger.info("Test Build Mask Policy Success") @MindFormerRegister.register(MindFormerModuleType.MODELS) class TestModel(BaseModel): """Test Model API For Register.""" def __init__(self, config: dict = None): super(TestModel, self).__init__(config) self.model_config = config self.params = Parameter(Tensor([0.1])) @MindFormerRegister.register(MindFormerModuleType.TOKENIZER) class TestTokenizer: """Test Tokenizer API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestTextConfig: """Test TextConfig API For Register.""" seq_length: int = 12 @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestVisionConfig: """Test VisionConfig API For Register.""" seq_length: int = 12 @MindFormerRegister.register(MindFormerModuleType.CONFIG) @dataclass class TestModelConfig: """Test ModelConfig API For Register.""" batch_size: int = 2 embed_dim: int = 768 text_config: Callable = TestTextConfig vision_config: Callable = TestVisionConfig @MindFormerRegister.register(MindFormerModuleType.OPTIMIZER) class TestAdamWeightDecay(AdamWeightDecay): """Test AdamWeightDecay API For Register.""" def __init__(self, params, learning_rate=1e-3, beta1=0.9, beta2=0.999, eps=1e-6, weight_decay=0.0): super(TestAdamWeightDecay, self).__init__(params, learning_rate=learning_rate, beta1=beta1, beta2=beta2, eps=eps, weight_decay=weight_decay) self.param = params @MindFormerRegister.register(MindFormerModuleType.LR) class TestCosineDecayLR(CosineDecayLR): """Test CosineDecayLR API For Register.""" def __init__(self, min_lr, max_lr, decay_steps): super(TestCosineDecayLR, self).__init__(min_lr, max_lr, decay_steps) self.lr = max_lr @MindFormerRegister.register(MindFormerModuleType.WRAPPER) class TestTrainOneStepWithLossScaleCell(TrainOneStepWithLossScaleCell): """Test TrainOneStepWithLossScaleCell API For Register.""" def __init__(self, network, optimizer, scale_sense): super(TestTrainOneStepWithLossScaleCell, self).__init__(network, optimizer, scale_sense) self.scale_sense = scale_sense @MindFormerRegister.register(MindFormerModuleType.METRIC) class TestAccuracy(Accuracy): """Test Accuracy API For Register.""" def __init__(self, eval_type='classification'): super(TestAccuracy, self).__init__(eval_type) self.eval = eval_type @MindFormerRegister.register(MindFormerModuleType.LOSS) class TestL1Loss(L1Loss): """Test L1Loss API For Register.""" def __init__(self, reduction='mean'): super(TestL1Loss, self).__init__(reduction) self.reduction = reduction @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class TestLLossMonitor(LossMonitor): """Test LossMonitor API For Register.""" def __init__(self, per_print_times=1): super(TestLLossMonitor, self).__init__(per_print_times) self.print = per_print_times @MindFormerRegister.register(MindFormerModuleType.CALLBACK) class TestTimeMonitor(TimeMonitor): """Test TimeMonitor API For Register.""" def __init__(self, data_size=1): super(TestTimeMonitor, self).__init__(data_size) self.data_size = data_size @MindFormerRegister.register(MindFormerModuleType.PIPELINE) class TestPipeline: """Test Pipeline API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.PROCESSOR) class TestProcessor: """Test Processor API For Register.""" def __init__(self): pass @MindFormerRegister.register(MindFormerModuleType.TRAINER) class TestTaskTrainer(BaseTrainer): """Test TimeMonitor API For Register.""" def __init__(self, model_name='vit'): super(TestTaskTrainer, self).__init__(model_name=model_name) @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_build_from_config(): """ Feature: Build API from config Description: Test build function to instance API from config Expectation: TypeError """ # build dataset check_dataset_config(all_config) build_dataset(all_config.train_dataset_task) logger.info("Test Build Dataset Success") # build model model = build_model(all_config.model) logger.info("Test Build Network Success") # build lr lr = build_lr(all_config.lr_schedule) logger.info("Test Build LR Success") # build optimizer if lr is not None: optimizer = build_optim(all_config.optimizer, default_args={"params": model.trainable_params(), "learning_rate": lr}) else: optimizer = build_optim(all_config.optimizer, default_args={"params": model.trainable_params()}) logger.info("Test Build Optimizer Success") # build wrapper build_wrapper(all_config.runner_wrapper, default_args={"network": model, "optimizer": optimizer}) logger.info("Test Build Wrapper Success") build_loss(all_config.loss) logger.info("Test Build Loss Success") build_metric(all_config.metric) logger.info("Test Build Metric Success") build_processor(all_config.processor) logger.info("Test Build Processor Success") # build trainer build_trainer(all_config.trainer) logger.info("Test Build Trainer Success") # build pipeline build_pipeline(all_config.pipeline) logger.info("Test Build Pipeline Success") @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_build_from_class_name(): """ Feature: Build API from class name Description: Test build function to instance API from class name Expectation: TypeError """ # build dataset build_dataset(class_name='TestDataset') logger.info("Test Build Dataset Success") # build model model = build_model(class_name='TestModel') logger.info("Test Build Network Success") # build lr lr = build_lr(class_name='TestCosineDecayLR', min_lr=0., max_lr=0.001, decay_steps=1000) logger.info("Test Build LR Success") # build optimizer if lr is not None: optimizer = build_optim(class_name='TestAdamWeightDecay', params=model.trainable_params(), learning_rate=lr) else: optimizer = build_optim(class_name='TestAdamWeightDecay', params=model.trainable_params()) logger.info("Test Build Optimizer Success") # build loss build_loss(class_name='TestL1Loss') logger.info("Test Build Loss Success") # build metric build_metric(class_name='TestAccuracy') logger.info("Test Build Metric Success") # build wrapper scale_sense = Tensor(1.0) build_wrapper(class_name='TestTrainOneStepWithLossScaleCell', network=model, optimizer=optimizer, scale_sense=scale_sense) logger.info("Test Build Wrapper Success") # build processor build_processor(class_name='TestProcessor') logger.info("Test Build Processor Success") # build trainer build_trainer(class_name='TestTaskTrainer') logger.info("Test Build Trainer Success") # build pipeline build_pipeline(class_name='TestPipeline') logger.info("Test Build Pipeline Success") pengcheng.mind-mindformer/tests/st/test_build.yaml000066400000000000000000000031351476650141400227260ustar00rootroot00000000000000model: model_config: type: TestModelConfig text_config: type: TestTextConfig vision_config: type: TestVisionConfig arch: type: TestModel loss: type: TestL1Loss reduction: 'mean' metric: type: TestAccuracy eval_type: 'classification' train_dataset: &train_dataset data_loader: type: TestDataLoader dataset_dir: "/dataset/imageNet-1k/train" transforms: - type: TestTransforms1 - type: TestTransforms1 sampler: type: TestSampler mask_policy: type: TestModelMask input_columns: ["image"] output_columns: [ "image", "mask"] column_order: [ "image", "mask"] num_parallel_workers: 8 python_multiprocessing: False drop_remainder: True batch_size: 128 repeat: 1 numa_enable: False prefetch_size: 30 train_dataset_task: type: TestDataset dataset_config: *train_dataset runner_wrapper: type: TestTrainOneStepWithLossScaleCell scale_sense: 1024 # optimizer optimizer: type: TestAdamWeightDecay beta1: 0.9 beta2: 0.999 eps: 0.00000001 # 1e-8 weight_decay: 0.05 # lr sechdule lr_schedule: type: TestCosineDecayLR max_lr: 0.00015 min_lr: 0. decay_steps: 100 runner_config: epoch: 800 batch_size: 64 image_size: 224 sink_mode: True sink_size: 1 initial_epoch: 0 has_trained_epoches: 0 has_trained_steps: 0 load_checkpoint: callbacks: - type: TestLLossMonitor per_print_times: 10 - type: TestTimeMonitor data_size: 100 do_train: True do_eval: False do_predict: False trainer: type: TestTaskTrainer model_name: 'mae' pipeline: type: TestPipeline processor: type: TestProcessorpengcheng.mind-mindformer/tests/st/test_do_eval.py000066400000000000000000000102061476650141400227230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing evaluating on training featrue used for mindformers. Uses gpt2 model as example How to run this: pytest tests/st/test_do_eval.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import GPT2LMHeadModel, GPT2Config from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestDoEvalMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments( batch_size=4, num_train_epochs=1, do_eval=True, eval_step_interval=-1, eval_epoch_interval=2, ) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, hidden_size=32, num_heads=2, seq_length=64) model = GPT2LMHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='gpt2', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) print(self.task_trainer.config) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True) @pytest.mark.run(order=3) def test_config_change_train(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.do_eval = False self.task_trainer.config.eval_step_interval = 4 self.task_trainer.config.eval_epoch_interval = -1 self.task_trainer.train(do_eval=True) @pytest.mark.run(order=4) def test_config_change_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True, do_eval=True) pengcheng.mind-mindformer/tests/st/test_mindformer_book.py000066400000000000000000000061021476650141400244660ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of MindFormerBook class How to run this: windows: pytest .\\tests\\st\\test_mindformer_book.py linux: pytest ./tests/st/test_mindformer_book.py """ import pytest from mindformers import MindFormerBook from mindformers.tools import logger @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_show(): """ Feature: mindformers book class, show content Description: Test to show content in MindFormerBook Expectation: success """ MindFormerBook.show_trainer_support_task_list() MindFormerBook.show_pipeline_support_task_list() MindFormerBook.show_model_support_list() MindFormerBook.show_model_ckpt_url_list() MindFormerBook.show_model_config_url_list() MindFormerBook.show_project_path() MindFormerBook.show_default_checkpoint_download_folder() MindFormerBook.show_default_checkpoint_save_folder() MindFormerBook.show_model_config_to_name() @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_get(): """ Feature: mindformers book class, get content Description: Test to get content of MindFormerBook Expectation: success """ logger.info(MindFormerBook.get_trainer_support_task_list()) logger.info(MindFormerBook.get_pipeline_support_task_list()) logger.info(MindFormerBook.get_model_support_list()) logger.info(MindFormerBook.get_model_ckpt_url_list()) logger.info(MindFormerBook.get_model_config_url_list()) logger.info(MindFormerBook.get_project_path()) logger.info(MindFormerBook.get_default_checkpoint_download_folder()) logger.info(MindFormerBook.get_default_checkpoint_save_folder()) logger.info(MindFormerBook.get_model_config_to_name()) # default path is settable @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_mindformer_book_set(): """ Feature: mindformers book class, get content Description: Test to set content for MindFormerBook Expectation: success or TypeError """ path = MindFormerBook.get_project_path() MindFormerBook.set_default_checkpoint_download_folder(path) MindFormerBook.set_default_checkpoint_save_folder(path) logger.info(MindFormerBook.get_default_checkpoint_download_folder()) logger.info(MindFormerBook.get_default_checkpoint_save_folder()) MindFormerBook.set_model_config_to_name("ModelConfig", "ModelName") pengcheng.mind-mindformer/tests/st/test_model/000077500000000000000000000000001476650141400220415ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/__init__.py000066400000000000000000000012551476650141400241550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test model.""" pengcheng.mind-mindformer/tests/st/test_model/test_bert_model/000077500000000000000000000000001476650141400252145ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_bert_model/__init__.py000066400000000000000000000012541476650141400273270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test bert.""" pengcheng.mind-mindformer/tests/st/test_model/test_bert_model/test_autoclass.py000066400000000000000000000072541476650141400306330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestBertAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['bert_base_uncased'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_auto_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-mindformer/tests/st/test_model/test_bert_model/test_pipeline.py000066400000000000000000000027151476650141400304370ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestBertPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_list = ['bert_base_uncased'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_list: task_pipeline = pipeline(task='fill_mask', model=model_type, max_length=128, padding='max_length') task_pipeline([" Hello I am a [MASK] model.",]) pengcheng.mind-mindformer/tests/st/test_model/test_bert_model/test_trainer.py000066400000000000000000000072601476650141400302760ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bert interface used for mindformers. How to run this: pytest tests/st/test_model/test_bert_model/test_gpt_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models import BertConfig, BertForPreTraining from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator(): """dataset generator""" data = np.random.randint(low=0, high=15, size=(128,)).astype(np.int32) input_mask = np.ones_like(data) token_type_id = np.zeros_like(data) next_sentence_lables = np.array([1]).astype(np.int32) masked_lm_positions = np.array([1, 2]).astype(np.int32) masked_lm_ids = np.array([1, 2]).astype(np.int32) masked_lm_weights = np.ones_like(masked_lm_ids) train_data = (data, input_mask, token_type_id, next_sentence_lables, masked_lm_positions, masked_lm_ids, masked_lm_weights) for _ in range(4): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestBertTrainerMethod: """A test class for testing trainer.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=1, num_train_epochs=1) train_dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask", "segment_ids", "next_sentence_labels", "masked_lm_positions", "masked_lm_ids", "masked_lm_weights"]) train_dataset = train_dataset.batch(batch_size=1) model_config = BertConfig(batch_size=1, num_hidden_layers=2) model = BertForPreTraining(model_config) self.task_trainer = Trainer(task='fill_mask', model=model, args=args, train_dataset=train_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ input_data = [" Hello I am a [MASK] model.",] self.task_trainer.predict(input_data=input_data) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ pengcheng.mind-mindformer/tests/st/test_model/test_blip2_model/000077500000000000000000000000001476650141400252705ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_blip2_model/__init__.py000066400000000000000000000012551476650141400274040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test blip2.""" pengcheng.mind-mindformer/tests/st/test_model/test_blip2_model/test_autoclass.py000066400000000000000000000121731476650141400307030ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_autoclass.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor ms.set_context(mode=0, device_id=7) class TestBlip2AutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['blip2_stage1_vit_g'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') class TestBlip2SecondStageAutoClassMethod: """A test class for testing Model classes of Blip2 second stage""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['blip2_stage2_vit_g_llama_7b', 'itt_blip2_stage2_vit_g_llama_7b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-mindformer/tests/st/test_model/test_blip2_model/test_pipeline.py000066400000000000000000000047351476650141400305170ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_pipeline.py """ from PIL import Image import mindspore as ms from mindformers import pipeline, BasePipeline ms.set_context(mode=0, device_id=7) class TestBlip2PipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['blip2_stage1_classification'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: classifier = pipeline("zero_shot_image_classification", model=model_type, candidate_labels=["sunflower", "tree", "dog", "cat", "toy"], hypothesis_template="A picture of {}.") image = Image.new('RGB', (512, 512)) classifier(image) class TestBlip2SecondStagePipelineMethod: """A test class for testing pipeline of ImageToTextGeneration.""" def setup_method(self): """setup method.""" self.test_llm_list = ['itt_blip2_stage2_vit_g_llama_7b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: generator = pipeline("image_to_text_generation", model=model_type, max_length=32) assert isinstance(generator, BasePipeline) image = Image.new('RGB', (448, 448)) generator(image, hypothesis_template='a picture of ') pengcheng.mind-mindformer/tests/st/test_model/test_blip2_model/test_trainer.py000066400000000000000000000243711476650141400303540ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the blip2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_blip2_model/test_trainer.py """ import numpy as np # import pytest from PIL import Image import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import Trainer, TrainingArguments, Blip2Llama, LlamaTokenizer from mindformers.models.bert.bert_tokenizer import BertTokenizer from mindformers.models.blip2 import Blip2Qformer, Blip2Classifier, Blip2Config, Blip2ImageProcessor from mindformers.models.blip2.blip2_llama import Blip2ImageToTextGeneration from mindformers.models.blip2.qformer import QFormerConfig from mindformers.models.llama.llama_config import LlamaConfig from mindformers.models.vit.vit import ViTConfig ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 8 images = np.random.randint(low=0, high=255, size=(3, 32, 32)).astype(np.float32) input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(4): yield images, input_ids def generator_eval(): """eval dataset generator""" seq_len = 8 images = np.random.randint(low=0, high=255, size=(3, 32, 32)).astype(np.float32) input_ids = np.random.randint(low=0, high=15, size=(seq_len)).astype(np.int32) for _ in range(4): yield images, input_ids class TestBlip2TrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" train_dataset = GeneratorDataset(generator_train, column_names=["image", "input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["image", "input_ids"]) self.train_dataset = train_dataset.batch(batch_size=2) self.eval_dataset = eval_dataset.batch(batch_size=2) qformer_config = QFormerConfig(num_hidden_layers=1, num_attention_heads=2, hidden_size=2, head_embed_dim=2, encoder_width=2, intermediate_size=2, vocab_size=32, max_position_embeddings=80) vision_config = ViTConfig(image_size=32, hidden_size=2, num_hidden_layers=1, num_attention_heads=2, intermediate_size=2, num_labels=10) model_config = Blip2Config(vision_config=vision_config, qformer_config=qformer_config) # for train and evaluation self.blip2_qformer = Blip2Qformer(model_config) # for prediction self.blip2_classifier = Blip2Classifier(model_config) self.image_processor = Blip2ImageProcessor(image_size=32) self.tokenizer = BertTokenizer.from_pretrained('bert_base_uncased') self.training_args = TrainingArguments(batch_size=2, num_train_epochs=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ task_trainer = Trainer(task='contrastive_language_image_pretrain', model=self.blip2_qformer, model_name='blip2_stage1_vit_g', args=self.training_args, tokenizer=self.tokenizer, train_dataset=self.train_dataset, eval_dataset=self.eval_dataset) task_trainer.train() def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ task_evaluater = Trainer(task='image_to_text_retrieval', model=self.blip2_qformer, model_name='blip2_stage1_vit_g', tokenizer=self.tokenizer, train_dataset=self.train_dataset, eval_dataset=self.eval_dataset) task_evaluater.evaluate(k_test=1, add_extra_itm_score=False) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ task_predictor = Trainer(task='zero_shot_image_classification', model=self.blip2_classifier, model_name='blip2_stage1_classification', tokenizer=self.tokenizer, image_processor=self.image_processor, candidate_labels=["sunflower", "tree", "dog", "cat", "toy"]) # 加载输入，一张图片 input_data = Image.new('RGB', (32, 32)) # 加载指定的权重以完成推理 task_predictor.predict(input_data=input_data) class TestBlip2SecondStageTrainerMethod: """A test class for testing blip2 second stage trainers.""" def setup_method(self): """init task trainer.""" train_batch_size = 2 train_dataset = GeneratorDataset(generator_train, column_names=["image", "input_ids"]) self.train_dataset = train_dataset.batch(batch_size=train_batch_size) qformer_config = QFormerConfig(num_hidden_layers=1, num_attention_heads=2, hidden_size=2, head_embed_dim=2, encoder_width=2, intermediate_size=2, vocab_size=50, max_position_embeddings=512) vision_config = ViTConfig(image_size=32, hidden_size=2, num_hidden_layers=1, num_attention_heads=2, intermediate_size=2, num_labels=10, checkpoint_name_or_path="") text_config_for_trainer = LlamaConfig(seq_length=39, hidden_size=8, num_layers=2, num_heads=2, checkpoint_name_or_path="") model_config_for_trainer = Blip2Config(batch_size=train_batch_size, vision_config=vision_config, qformer_config=qformer_config, text_config=text_config_for_trainer, checkpoint_name_or_path="", max_txt_len=7) text_config_for_generator = LlamaConfig(seq_length=128, hidden_size=8, num_layers=2, num_heads=2, checkpoint_name_or_path="", use_past=True) model_config_for_generator = Blip2Config(batch_size=1, vision_config=vision_config, qformer_config=qformer_config, text_config=text_config_for_generator, checkpoint_name_or_path="", max_txt_len=8) self.blip2_llm = Blip2Llama(model_config_for_trainer) # for prediction self.blip2_generator = Blip2ImageToTextGeneration(model_config_for_generator) self.image_processor = Blip2ImageProcessor(image_size=32) self.tokenizer = LlamaTokenizer.from_pretrained('llama_7b') self.training_args = TrainingArguments(batch_size=4, num_train_epochs=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ task_trainer = Trainer(task='contrastive_language_image_pretrain', model=self.blip2_llm, model_name='blip2_stage2_vit_g_llama_7b', args=self.training_args, tokenizer=self.tokenizer, train_dataset=self.train_dataset) task_trainer.train(train_checkpoint='') def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ task_predictor = Trainer(task='image_to_text_generation', model=self.blip2_generator, model_name='itt_blip2_stage2_vit_g_llama_7b', tokenizer=self.tokenizer, image_processor=self.image_processor) # 加载输入，一张图片 input_data = Image.new('RGB', (32, 32)) # 加载指定的权重以完成推理 task_predictor.predict(input_data=input_data, max_length=8) pengcheng.mind-mindformer/tests/st/test_model/test_bloom_model/000077500000000000000000000000001476650141400253705ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_bloom_model/__init__.py000066400000000000000000000012551476650141400275040ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test bloom.""" pengcheng.mind-mindformer/tests/st/test_model/test_bloom_model/test_autoclass.py000066400000000000000000000072301476650141400310010ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_autoclass.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestBloomAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['bloom_560m'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-mindformer/tests/st/test_model/test_bloom_model/test_pipeline.py000066400000000000000000000026611476650141400306130ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestBloomPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['bloom_560m'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-mindformer/tests/st/test_model/test_bloom_model/test_trainer.py000066400000000000000000000073541476650141400304560ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the bloom interface used for mindformers. How to run this: pytest tests/st/test_model/test_bloom_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer from mindformers import BloomLMHeadModel, BloomConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(16): yield input_ids def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) for _ in range(16): yield input_ids @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestBloomTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = BloomConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512, max_decode_length=10) model = BloomLMHeadModel(model_config) tokenizer = AutoTokenizer.from_pretrained("bloom_560m") self.task_trainer = Trainer(task='text_generation', model=model, args=args, tokenizer=tokenizer, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() # @pytest.mark.run(order=2) # def test_eval(self): # """ # Feature: Trainer.evaluate() # Description: Test trainer for evaluate. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", add_special_tokens=False, max_length=20) # @pytest.mark.run(order=4) # def test_finetune(self): # """ # Feature: Trainer.finetune() # Description: Test trainer for finetune. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.finetune() pengcheng.mind-mindformer/tests/st/test_model/test_clip_model/000077500000000000000000000000001476650141400252075ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_clip_model/__init__.py000066400000000000000000000012541476650141400273220ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test clip.""" pengcheng.mind-mindformer/tests/st/test_model/test_clip_model/test_clip_model.py000066400000000000000000000107101476650141400307260ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and CLIPModel class How to run this: windows: pytest .\\tests\\st\\test_model\\test_clip_model\\test_clip_model.py linux: pytest ./tests/st/test_model/test_clip_model/test_clip_model.py """ import os import time import mindspore as ms from mindformers import MindFormerBook, AutoConfig, AutoModel from mindformers.models import CLIPModel, BaseModel from mindformers.tools import logger ms.set_context(mode=0) class TestCLIPModelMethod: """A test class for testing Model classes""" def setup_method(self): """get_input""" self.model_type = "clip_vit_b_32" self.checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip') self.config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') self.config = AutoConfig.from_pretrained(self.model_type) self.checkpoint_path = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'clip', self.model_type + '.ckpt') self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'clip') def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ time.sleep(5) AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # input model name, load model and weights model_a = AutoModel.from_pretrained(self.model_type) # input model directory, load model and weights model_b = AutoModel.from_pretrained(self.checkpoint_dir) # input yaml path, load model without weights model_c = AutoModel.from_config(self.config_path) # input config, load model without weights model_d = AutoModel.from_config(self.config) model_a.save_pretrained(self.save_directory, save_name=self.model_type) CLIPModel.show_support_list() support_list = CLIPModel.get_support_list() logger.info(support_list) # input model name, load model and weights model_i = CLIPModel.from_pretrained(self.model_type) # input model directory, load model and weights model_j = CLIPModel.from_pretrained(self.checkpoint_dir) # input config, load model weights model_k = CLIPModel(self.config) # input config, load model without weights self.config.checkpoint_name_or_path = None model_l = CLIPModel(self.config) model_i.save_pretrained(self.save_directory, save_name=self.model_type) # all models are ClipModel class， and inherited from BaseModel assert isinstance(model_i, CLIPModel) assert isinstance(model_j, CLIPModel) assert isinstance(model_k, CLIPModel) assert isinstance(model_l, CLIPModel) assert isinstance(model_i, BaseModel) assert isinstance(model_j, BaseModel) assert isinstance(model_k, BaseModel) assert isinstance(model_l, BaseModel) # all models are CLIPModel class， and inherited from BaseModel assert isinstance(model_a, CLIPModel) assert isinstance(model_b, CLIPModel) assert isinstance(model_c, CLIPModel) assert isinstance(model_d, CLIPModel) assert isinstance(model_a, BaseModel) assert isinstance(model_b, BaseModel) assert isinstance(model_c, BaseModel) assert isinstance(model_d, BaseModel) pengcheng.mind-mindformer/tests/st/test_model/test_clip_model/test_clip_tokenizer.py000066400000000000000000000132311476650141400316410ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_clip_model/test_clip_tokenizer.py """ import os import shutil import time import pytest from mindspore import Tensor from mindformers import Tokenizer, AutoTokenizer from mindformers import CLIPTokenizer def generate_fake_vocab(output_path): vocabs = ["[PAD]", "[unused1]", "[UNK]", "[CLS]", "[SEP]", "[MASK]", "hello", "world", "!"] with open(os.path.join(output_path, 'vocab_file.txt'), 'w') as fp: for item in vocabs: fp.write(item + '\n') class TestAutoTokenizerMethod: """A test class for testing the AutoTokenizer""" def setup_method(self): self.output_path = os.path.join(os.path.dirname(__file__), 'test_tokenizer_output') os.makedirs(self.output_path, exist_ok=True) generate_fake_vocab(self.output_path) def teardown_method(self): shutil.rmtree(self.output_path, ignore_errors=True) @pytest.mark.run(order=1) def test_save_and_load_using_bert_tokenizer(self): """ Feature: The test load and save function for the clip tokenizer Description: Load the tokenizer and then saved it Expectation: The restored kwargs is not expected version. """ clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer.tokenize("hello world?") assert isinstance(clip_tokenizer, CLIPTokenizer) assert res == ['hello', 'world', '?'] @pytest.mark.run(order=2) def test_load_from_yaml(self): """ Feature: The test load from yaml and save as the yaml for the tokenizer Description: Load the tokenizer and then saved it Expectation: The restored kwargs is not expected version. """ tokenizer = AutoTokenizer.from_pretrained("clip_vit_b_32") res = tokenizer.tokenize("hello world?") assert isinstance(tokenizer, CLIPTokenizer) assert res == ['hello', 'world', '?'] class TestPretrainedTokenizerMethod: """A test class for testing the PretrainedTokenizer""" def generate_fake_vocab(self): vocabs = ["[PAD]", "[unused1]", "[UNK]", "[CLS]", "[SEP]", "[MASK]", "hello", "world", "!"] with open(os.path.join(self.output_path, 'vocab.txt'), 'w') as fp: for item in vocabs: fp.write(item + '\n') def setup_method(self): self.output_path = os.path.join(os.path.dirname(__file__), 'test_tokenizer_output' + str(self)) os.makedirs(self.output_path, exist_ok=True) self.generate_fake_vocab() def teardown_method(self): shutil.rmtree(self.output_path, ignore_errors=True) def test_from_pretrained_tokenizer(self): """ Feature: The Tokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ time.sleep(10) tokenizer = Tokenizer.from_pretrained(self.output_path) with pytest.raises(NotImplementedError): tokenizer("hello world") class TestClipTokenizerMethod: """Test the basic usage of the CLIPTokenizer""" def test_padding(self): """ Feature: The CLIPTokenizer test using padding Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to the target. """ clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer("hello world?", max_length=8, padding='max_length') pad_id = clip_tokenizer.pad_token_id assert res == {'attention_mask': [1, 1, 1, 1, 1, 0, 0, 0], 'input_ids': [49406, 3306, 1002, 286, 49407, pad_id, pad_id, pad_id]}, f"The res is {res}." clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") res = clip_tokenizer("hello world?", max_length=8, padding='max_length', return_tensors='ms') assert len(res) == 2 for k in res.keys(): assert isinstance(res[k], Tensor) clip_tokenizer = CLIPTokenizer.from_pretrained("clip_vit_b_32") batch_inputs = ["hello world?", "Who are you?", "I am find, thank you."] res = clip_tokenizer(batch_inputs, max_length=12, padding='max_length') assert len(res) == 2 assert res == {'input_ids': [[49406, 3306, 1002, 286, 49407, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id], [49406, 822, 631, 592, 286, 49407, pad_id, pad_id, pad_id, pad_id, pad_id, pad_id], [49406, 328, 687, 1416, 267, 1144, 592, 269, 49407, pad_id, pad_id, pad_id]], 'attention_mask': [[1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0], [1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0]]} pengcheng.mind-mindformer/tests/st/test_model/test_glm2_model/000077500000000000000000000000001476650141400251215ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_glm2_model/__init__.py000066400000000000000000000012541476650141400272340ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test glm2.""" pengcheng.mind-mindformer/tests/st/test_model/test_glm2_model/test_auto_class.py000066400000000000000000000061341476650141400306730ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm2_model/test_auto_class.py """ import os from mindspore import context from mindformers import MindFormerBook, AutoModel, BaseModel, AutoConfig, BaseConfig, AutoTokenizer, BaseTokenizer class TestGLMAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ["glm2_6b"] def test_glm2_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights # Too time-cost, not used for now. for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + "_model", ) def test_glm2_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + "_config", ) def test_glm2_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name tokenizer_type = "glm2_6b" tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + "_tokenizer", ) pengcheng.mind-mindformer/tests/st/test_model/test_glm2_model/test_trainer.py000066400000000000000000000115201476650141400301750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm2 interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm2_model/test_trainer.py """ import numpy as np import pytest import mindspore from mindspore import context from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer, ChatGLM2Config, ChatGLM2ForConditionalGeneration from mindformers import Trainer, TrainingArguments from mindformers.tools.utils import is_version_ge def generator_train(): """train dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) train_data = (input_ids, labels) for _ in range(32): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 127 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) eval_data = (input_ids, labels) for _ in range(8): yield eval_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGLM2TrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" context.set_context(mode=0, device_target="Ascend") args = TrainingArguments(num_train_epochs=1, batch_size=2) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "labels"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "labels"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = ChatGLM2Config(num_layers=2, seq_length=128, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = ChatGLM2ForConditionalGeneration(model_config) self.tokenizer = AutoTokenizer.from_pretrained("glm2_6b") self.task_trainer = Trainer(task='text_generation', model=model, model_name='glm2_6b', tokenizer=self.tokenizer, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ model_config = ChatGLM2Config(num_layers=1, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = ChatGLM2ForConditionalGeneration(model_config) task_trainer = Trainer(task='text_generation', model=model, tokenizer=self.tokenizer) if is_version_ge(mindspore.__version__, "1.11.0"): task_trainer.predict(input_data="你好", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ if is_version_ge(mindspore.__version__, "1.11.0"): self.task_trainer.finetune() pengcheng.mind-mindformer/tests/st/test_model/test_glm_model/000077500000000000000000000000001476650141400250375ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_glm_model/__init__.py000066400000000000000000000012531476650141400271510ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test glm.""" pengcheng.mind-mindformer/tests/st/test_model/test_glm_model/test_auto_class.py000066400000000000000000000075331476650141400306150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_auto_class.py """ import os from mindspore import context from mindformers import MindFormerBook from mindformers import AutoModel from mindformers import AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel from mindformers.models import BaseConfig, BaseTokenizer, BaseProcessor class TestGLMAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ["glm_6b", "glm_6b_chat"] def test_glm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights # Too time-cost, not used for now. for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + "_model", ) def test_glm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + "_config", ) def test_glm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + "_processor", ) def test_glm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name tokenizer_type = "glm_6b" tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + "_tokenizer", ) pengcheng.mind-mindformer/tests/st/test_model/test_glm_model/test_pipeline.py000066400000000000000000000027641476650141400302660ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_pipeline.py """ from mindspore import context from mindformers import pipeline class TestGLMPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" context.set_context(mode=0) self.test_llm_list = ['glm_6b', 'glm_6b_chat'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ # Too time-cost, not used for now. for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("你好", top_k=3) pengcheng.mind-mindformer/tests/st/test_model/test_glm_model/test_trainer.py000066400000000000000000000112121476650141400301110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the glm interface used for mindformers. How to run this: pytest tests/st/test_model/test_glm_model/test_trainer.py """ import numpy as np import pytest from mindspore import context from mindspore.dataset import GeneratorDataset from mindformers import AutoTokenizer from mindformers import GLMForPreTraining, GLMChatModel, GLMConfig from mindformers import Trainer, TrainingArguments def generator_train(): """train dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) label = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) position_ids = np.ones((2, seq_len)).astype(np.int64) attention_mask = np.ones(shape=(seq_len, seq_len)).astype(np.int32) train_data = (input_ids, label, position_ids, attention_mask) for _ in range(512): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) label = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) eval_data = (input_ids, label) for _ in range(8): yield eval_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGLMTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" context.set_context(mode=0) args = TrainingArguments(num_train_epochs=1, batch_size=2) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "label", "position_ids", "attention_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "label"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GLMConfig(num_layers=2, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = GLMForPreTraining(model_config) self.tokenizer = AutoTokenizer.from_pretrained("glm_6b") self.task_trainer = Trainer(task='text_generation', model=model, model_name='glm_6b', tokenizer=self.tokenizer, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() # @pytest.mark.run(order=2) # def test_eval(self): # """ # Feature: Trainer.evaluate() # Description: Test trainer for evaluate. # Expectation: TypeError, ValueError, RuntimeError # """ # self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ model_config = GLMConfig(num_layers=1, hidden_size=32, inner_hidden_size=None, num_heads=2, position_encoding_2d=True) model = GLMChatModel(model_config) task_trainer = Trainer(task='text_generation', model=model, tokenizer=self.tokenizer) task_trainer.predict(input_data="你好", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune() pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_lora_model/000077500000000000000000000000001476650141400261515ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_lora_model/test_gpt2_lora_trainer.py000066400000000000000000000067531476650141400332120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt2_lora interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_lora_model/test_gpt2_lora_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.gpt2.gpt2 import GPT2WithLora from mindformers.pet.pet_config import LoraConfig from mindformers import Trainer, TrainingArguments, GPT2Config ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGPT2LoraTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, embedding_size=32, num_heads=2, seq_length=64) pet_config = LoraConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.05) model_config.pet_config = pet_config model = GPT2WithLora(model_config) self.task_trainer = Trainer(task='text_generation', model=model, pet_method='lora', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/000077500000000000000000000000001476650141400251345ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/__init__.py000066400000000000000000000012541476650141400272470ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test gpt2.""" pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/test_autoclass.py000066400000000000000000000072261476650141400305520ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestGPTAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['gpt2'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/test_pipeline.py000066400000000000000000000026461476650141400303620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestGPTPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['gpt2'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/test_trainer.py000066400000000000000000000074031476650141400302150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_gpt2_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers import GPT2LMHeadModel, GPT2Config from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 65 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 64 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGPTTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "input_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "input_mask"]) train_dataset = train_dataset.batch(batch_size=2) eval_dataset = eval_dataset.batch(batch_size=2) model_config = GPT2Config(num_layers=2, hidden_size=32, num_heads=2, seq_length=64) model = GPT2LMHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='gpt2', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.finetune(finetune_checkpoint=True) pengcheng.mind-mindformer/tests/st/test_model/test_gpt2_model/test_txtcls_model.py000066400000000000000000000047421476650141400312550ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForMultipleChoice class How to run this: windows: pytest .\\tests\\st\\test_model\\test_gpt2_model\\test_txtcls_model.py linux: pytest ./tests/st/test_model/test_gpt2_model/test_txtcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, GPT2ForSequenceClassification from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # gpt2_txtcls self.gpt2_txtcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'gpt2') self.gpt2_txtcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'gpt2', 'run_gpt2_txtcls.yaml') self.gpt2_txtcls_config = AutoConfig.from_pretrained('gpt2_txtcls') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'gpt2') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # gpt2_txtcls gpt2_txtcls_model_a = AutoModel.from_pretrained('gpt2_txtcls', download_checkpoint=False) gpt2_txtcls_model_a.save_pretrained(self.save_directory, save_name='gpt2_txtcls') assert isinstance(gpt2_txtcls_model_a, GPT2ForSequenceClassification) pengcheng.mind-mindformer/tests/st/test_model/test_llama_lora_model/000077500000000000000000000000001476650141400263635ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_llama_lora_model/__init__.py000066400000000000000000000012671476650141400305020ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test llama with lora.""" pengcheng.mind-mindformer/tests/st/test_model/test_llama_lora_model/test_llama_lora_trainer.py000066400000000000000000000070031476650141400336230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama_lora interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_lora_model/test_llama_lora_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.llama.llama import LlamaForCausalLMWithLora from mindformers.models.llama.llama_config import LlamaConfig from mindformers.pet.pet_config import LoraConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) labels = np.ones_like(input_ids) train_data = (input_ids, labels) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestLlamaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "labels"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids", "labels"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = LlamaConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512) model_config.pet_config = LoraConfig(lora_rank=8, lora_alpha=16, lora_dropout=0.05) model = LlamaForCausalLMWithLora(model_config) self.task_trainer = Trainer(task='text_generation', model=model, model_name='llama_7b_lora', args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() @pytest.mark.run(order=2) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) pengcheng.mind-mindformer/tests/st/test_model/test_llama_model/000077500000000000000000000000001476650141400253465ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_llama_model/__init__.py000066400000000000000000000012551476650141400274620ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test llama.""" pengcheng.mind-mindformer/tests/st/test_model/test_llama_model/test_autoclass.py000066400000000000000000000072711476650141400307640ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_auto_class.py """ import os import mindspore as ms from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor ms.set_context(mode=0) class TestLlamaAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['llama_7b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-mindformer/tests/st/test_model/test_llama_model/test_pipeline.py000066400000000000000000000026571476650141400305760ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_pipeline.py """ import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestLlamaPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['llama_7b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("hello!", top_k=3) pengcheng.mind-mindformer/tests/st/test_model/test_llama_model/test_trainer.py000066400000000000000000000076411476650141400304330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.llama.llama import LlamaForCausalLM from mindformers.models.llama.llama_config import LlamaConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 513 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 512 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestLlamaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = LlamaConfig(num_layers=2, hidden_size=32, num_heads=2, seq_length=512) model = LlamaForCausalLM(model_config) self.task_trainer = Trainer(task='text_generation', model=model, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) @pytest.mark.run(order=1) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.train() @pytest.mark.run(order=2) def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.model.set_train(False) self.task_trainer.evaluate() @pytest.mark.run(order=3) def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="hello world!", max_length=20, repetition_penalty=1, top_k=3, top_p=1) @pytest.mark.run(order=4) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune() pengcheng.mind-mindformer/tests/st/test_model/test_mae_model/000077500000000000000000000000001476650141400250225ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_mae_model/__init__.py000066400000000000000000000012531476650141400271340ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test mae.""" pengcheng.mind-mindformer/tests/st/test_model/test_mae_model/test_auto_class.py000066400000000000000000000057561476650141400306050ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_mae_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestMaeAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['mae_vit_base_p16'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-mindformer/tests/st/test_model/test_pangualpha_model/000077500000000000000000000000001476650141400264005ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_pangualpha_model/__init__.py000066400000000000000000000012621476650141400305120ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test pangualpha.""" pengcheng.mind-mindformer/tests/st/test_model/test_pangualpha_model/test_pangualpha_autoclass.py000066400000000000000000000072561476650141400342210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_auto_class.py """ import os # pylint: disable=W0611 from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoTokenizer, AutoProcessor # pylint: disable=W0611 from mindformers.models import BaseModel, BaseConfig, BaseTokenizer, BaseProcessor class TestAutoClassMethod: '''A test class for testing Model classes''' def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_llm_list = ['pangualpha_2_6b'] def test_llm_model(self): """ Feature: AutoModel. Description: Test to get LL-Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_llm_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_llm_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_llm_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_llm_processor(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_llm_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') def test_llm_tokenizer(self): """ Feature: AutoTokenizer, input config. Description: Test to get tokenizer instance by input tokenizer type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for tokenizer_type in self.test_llm_list: tokenizer = AutoTokenizer.from_pretrained(tokenizer_type) assert isinstance(tokenizer, BaseTokenizer) tokenizer.save_pretrained( save_directory=os.path.join(self.save_directory, tokenizer_type), save_name=tokenizer_type + '_tokenizer') pengcheng.mind-mindformer/tests/st/test_model/test_pangualpha_model/test_pangualpha_pipeline.py000066400000000000000000000026521476650141400340230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_pipeline.py """ # pylint: disable=W0611 from mindformers import pipeline class TestPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['pangualpha_2_6b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='text_generation', model=model_type, max_length=20) task_pipeline("今天天气不错，适合", top_k=3) pengcheng.mind-mindformer/tests/st/test_model/test_pangualpha_model/test_pangualpha_trainer.py000066400000000000000000000103061476650141400336550ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the pangualpha interface used for mindformers. How to run this: pytest tests/st/test_model/test_llm_model/test_pangualpha_trainer.py """ import numpy as np import pytest import mindspore as ms from mindspore.dataset import GeneratorDataset from mindformers.models.pangualpha.pangualpha import PanguAlphaHeadModel from mindformers.models.pangualpha.pangualpha_config import PanguAlphaConfig from mindformers import Trainer, TrainingArguments ms.set_context(mode=0) def generator_train(): """train dataset generator""" seq_len = 129 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) position_ids = np.ones((seq_len)).astype(np.int32) attention_mask = np.ones((seq_len, seq_len)).astype(np.int32) train_data = (input_ids, position_ids, attention_mask) for _ in range(16): yield train_data def generator_eval(): """eval dataset generator""" seq_len = 128 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) # input_mask = np.ones_like(input_ids) train_data = (input_ids) for _ in range(16): yield train_data @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestPanguAlphaTrainerMethod: """A test class for testing pipeline.""" def setup_method(self): """init task trainer.""" args = TrainingArguments(batch_size=4, num_train_epochs=1) train_dataset = GeneratorDataset(generator_train, column_names=["input_ids", "position_id", "attention_mask"]) eval_dataset = GeneratorDataset(generator_eval, column_names=["input_ids"]) train_dataset = train_dataset.batch(batch_size=4) eval_dataset = eval_dataset.batch(batch_size=4) model_config = PanguAlphaConfig(num_layers=2, hidden_size=128, ffn_hidden_size=128*4, num_heads=2, seq_length=128) model = PanguAlphaHeadModel(model_config) self.task_trainer = Trainer(task='text_generation', model=model, args=args, train_dataset=train_dataset, eval_dataset=eval_dataset) def test_train(self): """ Feature: Trainer.train() Description: Test trainer for train. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.train() def test_eval(self): """ Feature: Trainer.evaluate() Description: Test trainer for evaluate. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.model.set_train(False) self.task_trainer.evaluate() def test_predict(self): """ Feature: Trainer.predict() Description: Test trainer for predict. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.predict(input_data="今天天气如何？", max_length=20, repetition_penalty=1, top_k=3, top_p=1) def test_finetune(self): """ Feature: Trainer.finetune() Description: Test trainer for finetune. Expectation: TypeError, ValueError, RuntimeError """ self.task_trainer.config.runner_config.epochs = 1 self.task_trainer.finetune(finetune_checkpoint=True) pengcheng.mind-mindformer/tests/st/test_model/test_qa_model/000077500000000000000000000000001476650141400246615ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_qa_model/__init__.py000066400000000000000000000012601476650141400267710ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_qa_model.""" pengcheng.mind-mindformer/tests/st/test_model/test_qa_model/test_qa_model.py000066400000000000000000000075161476650141400300640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForQuestionAnswering class How to run this: windows: pytest .\\tests\\st\\test_model\\test_qa_model\\test_qa_model.py linux: pytest ./tests/st/test_model/test_qa_model/test_qa_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForQuestionAnswering, BaseModel from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_method(self): """get_input""" # fine-tuning self.qa_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'qa') self.qa_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'qa', 'run_qa_bert_base_uncased.yaml') self.qa_config = AutoConfig.from_pretrained('qa_bert_base_uncased') # evaluation and prediction self.qa_squad_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'qa') self.qa_squad_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'qa', 'run_qa_bert_base_uncased.yaml') self.qa_squad_config = AutoConfig.from_pretrained('qa_bert_base_uncased_squad') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'qa') def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name qa_model_a = AutoModel.from_pretrained('qa_bert_base_uncased', download_checkpoint=False) # input yaml path qa_model_c = AutoModel.from_config(self.qa_config_path, download_checkpoint=False) qa_model_a.save_pretrained(self.save_directory, save_name='qa_bert_base_uncased') assert isinstance(qa_model_a, BertForQuestionAnswering) assert isinstance(qa_model_c, BertForQuestionAnswering) assert isinstance(qa_model_a, BaseModel) assert isinstance(qa_model_c, BaseModel) # evaluation and prediction test part # input model name qa_squad_model_a = AutoModel.from_pretrained('qa_bert_base_uncased_squad', download_checkpoint=False) # # input yaml path qa_squad_model_c = AutoModel.from_config(self.qa_squad_config_path, download_checkpoint=False) qa_squad_model_a.save_pretrained(self.save_directory, save_name='qa_bert_base_uncased_squad') assert isinstance(qa_squad_model_a, BertForQuestionAnswering) assert isinstance(qa_squad_model_c, BertForQuestionAnswering) assert isinstance(qa_squad_model_a, BaseModel) assert isinstance(qa_squad_model_c, BaseModel) pengcheng.mind-mindformer/tests/st/test_model/test_sam_model/000077500000000000000000000000001476650141400250405ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_sam_model/__init__.py000066400000000000000000000000001476650141400271370ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_sam_model/test_auto_class.py000066400000000000000000000057501476650141400306150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestSamAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['sam_vit_b'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-mindformer/tests/st/test_model/test_sam_model/test_pipeline.py000066400000000000000000000120171476650141400302570ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the llama interface used for mindformers. How to run this: pytest tests/st/test_model/test_llama_model/test_pipeline.py """ import cv2 import numpy as np import mindspore as ms from mindformers import pipeline ms.set_context(mode=0) class TestSamPipelineMethod: """A test class for testing pipeline.""" def setup_method(self): """setup method.""" self.test_llm_list = ['sam_vit_b'] def test_pipeline(self): """ Feature: pipeline. Description: Test pipeline by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_llm_list: task_pipeline = pipeline(task='segment_anything', model=model_type) image = cv2.imread("research/sam/images/truck.jpg") image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) # 0.使用task_pipeline.set_image提前抽取图像特征 task_pipeline.set_image(image) # 1. 单点确定一个物体 input_point = np.array([[500, 375]]) input_label = np.array([1]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=True) # 2.两点确定相同物体 input_point = np.array([[500, 375], [1125, 625]]) input_label = np.array([1, 1]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 3.两点确定不同物体 input_point = np.array([[[500, 375]], [[1125, 625]]]) input_label = np.array([[1], [1]]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 4.一个前景点和背景点 input_point = np.array([[500, 375], [1125, 625]]) input_label = np.array([1, 0]) task_pipeline({"points": input_point, "labels": input_label}, multimask_output=False) # 5.单框确定一个物体 input_box = np.array([425, 600, 700, 875]) task_pipeline({"boxes": input_box}, multimask_output=False) # 6.框和背景点确定物体 input_box = np.array([425, 600, 700, 875]) input_point = np.array([[575, 750]]) input_label = np.array([0]) task_pipeline({"points": input_point, "labels": input_label, "boxes": input_box}, multimask_output=False) # 7.多组框和点确定不同物体 input_boxes = np.array([[425, 600, 700, 875], [1360, 525, 1680, 780]]) input_points = np.array([[[575, 750]], [[1525, 670]]]) input_labels = np.array([[1], [1]]) task_pipeline({"points": input_points, "labels": input_labels, "boxes": input_boxes}, multimask_output=False) # 8.多个框确定不同物体 input_boxes = np.array([[75, 275, 1725, 850], [425, 600, 700, 875], [1375, 550, 1650, 800], [1240, 675, 1400, 750]]) task_pipeline({"boxes": input_boxes}, multimask_output=False) # 单点确定一个物体: 传入cv2图像和prompt input_point = np.array([[500, 375]]) input_label = np.array([1]) image = cv2.imread("research/sam/images/truck.jpg") image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) task_pipeline({"image": image, "points": input_point, "labels": input_label}, multimask_output=True) # 单点确定一个物体: 传入图像路径和prompt task_pipeline({"image": "research/sam/images/truck.jpg", "points": input_point, "labels": input_label}, multimask_output=True) pengcheng.mind-mindformer/tests/st/test_model/test_swin_model/000077500000000000000000000000001476650141400252405ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_swin_model/__init__.py000066400000000000000000000012541476650141400273530ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test swin.""" pengcheng.mind-mindformer/tests/st/test_model/test_swin_model/test_auto_class.py000066400000000000000000000057561476650141400310230ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestSwinAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['swin_base_p4w7'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-mindformer/tests/st/test_model/test_t5_model/000077500000000000000000000000001476650141400246105ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_t5_model/__init__.py000066400000000000000000000012521476650141400267210ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test t5.""" pengcheng.mind-mindformer/tests/st/test_model/test_t5_model/test_t5_generation.py000066400000000000000000000043451476650141400307720ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of Generator model class How to run this: linux: pytest ./tests/st/test_model/test_t5_model/test_t5_generation.py """ import pytest import mindspore as ms from mindformers.models import T5ForConditionalGeneration, T5Tokenizer ms.set_context(mode=0) def modify_batch_size(net, batch_size): if hasattr(net, 'batch_size'): net.batch_size = batch_size for cell in net.cells(): modify_batch_size(cell, batch_size) @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestGeneratorUseT5: """A test class for testing Model classes""" def setup_class(self): self.t5 = T5ForConditionalGeneration.from_pretrained("t5_small") self.tokenizer = T5Tokenizer.from_pretrained("t5_small") @pytest.mark.parametrize('do_sample', [False, True]) def test_single_inference(self, do_sample): """ Feature: Test input as single example for generator Description: single example inference Expectation: ValueError, AttributeError """ words = "translate the English to the Romanian: UN Chief Says There Is No Military Solution in Syria" words = self.tokenizer(words, max_length=21, padding='max_length')['input_ids'] modify_batch_size(self.t5, batch_size=1) output = self.t5.generate(words, do_sample=do_sample) output = self.tokenizer.decode(output[0], skip_special_tokens=True) assert output == "eful ONU declară că nu există o soluţie militară în Siria" pengcheng.mind-mindformer/tests/st/test_model/test_t5_model/test_t5_tokenizer.py000066400000000000000000000033431476650141400306460ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_t5_model/test_t5_tokenizer.py """ from mindformers import T5Tokenizer, AutoTokenizer class TestT5TokenizerMethod: """A test class for testing the BertTokenizer""" def setup_method(self): self.tokenizer = T5Tokenizer.from_pretrained('t5_small') self.auto_tokenizer = AutoTokenizer.from_pretrained('t5_small') def test_from_pretrained_tokenizer(self): """ Feature: The T5Tokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ self.tokenizer.show_support_list() self.tokenizer("hello world") def test_auto_tokenizer(self): """ Feature: The T5Tokenizer test using auto_class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ self.auto_tokenizer("hello world") pengcheng.mind-mindformer/tests/st/test_model/test_t5_model/test_t5_trainer.py000066400000000000000000000145471476650141400303100ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the t5 trainer used for mindformers. How to run this: pytest tests/st/test_model/test_t5_model/test_t5_trainer.py """ import os import shutil import numpy as np import pytest import mindspore as ms from mindspore.dataset import MindDataset, GeneratorDataset from mindspore.mindrecord import FileWriter from mindformers.trainer import Trainer, TranslationTrainer from mindformers.trainer.config_args import ConfigArguments, \ OptimizerConfig, RunnerConfig from mindformers import T5Config, T5ForConditionalGeneration ms.set_context(mode=0) def generator(src_length=16, target_length=8): """dataset generator""" input_ids = np.random.randint(low=0, high=15, size=(src_length,)).astype(np.int32) attention_mask = np.ones((src_length,)).astype(np.int32) labels = np.random.randint(low=0, high=15, size=(target_length,)).astype(np.int32) for _ in range(2): yield input_ids, attention_mask, labels def write_mindrecord(ds_generator, data_record_path): """Using the generator to get mindrecords""" ds = GeneratorDataset(ds_generator, column_names=["input_ids", "attention_mask", "labels"]) writer = FileWriter(file_name=data_record_path, shard_num=1, overwrite=True) data_schema = {"input_ids": {"type": "int32", "shape": [-1]}, "attention_mask": {"type": "int32", "shape": [-1]}, "labels": {"type": "int32", "shape": [-1]}} writer.add_schema(data_schema, "test_schema") for item in ds.create_dict_iterator(): for k in item.keys(): item[k] = item[k].asnumpy() writer.write_raw_data([item]) writer.commit() def modify_attrs(net, key, value): if hasattr(net, key): setattr(net, key, value) print(f"Set the net {net.__class__.__name__} with {key}:{value}") for cell in net.cells(): modify_attrs(cell, key, value) def write_raw_text_data(stage, data_record_path): """writes the fake translation data""" source = ["We went through the whole range of emotions during this period.", "The positive reaction of pilots and Federation officials makes me hope that this year we will " "be organizing champions again" " said rally manager, Dan Codreanu."] target = ['Am trecut prin toată gama de trăiri în această perioadă.', "Reacția pozitivă a piloților și oficialilor Federației mă face să sper că vom fi și în acest an " "campion la organizare a spus managerul raliului, Dan Codreanu."] src_path = os.path.join(data_record_path, f'{stage}.source') tgt_path = os.path.join(data_record_path, f'{stage}.target') with open(src_path, 'w') as sfp: with open(tgt_path, 'w') as tfp: for x, y in zip(source, target): sfp.write(x + '\n') tfp.write(y + '\n') @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard class TestTranslationTrainer: """Test Translation Trainer""" def setup_class(self): self.dir_path = os.path.join(os.path.dirname(__file__), 'fake_dataset') os.makedirs(self.dir_path, exist_ok=True) self.abs_path = os.path.join(self.dir_path, 't5_dataset') write_mindrecord(generator(src_length=16, target_length=8), self.abs_path) self.raw_text_path = os.path.join(os.path.dirname(__file__), 'raw_text_dataset') os.makedirs(self.raw_text_path, exist_ok=True) write_raw_text_data(stage='train', data_record_path=self.raw_text_path) def teardown_class(self): shutil.rmtree(self.dir_path, ignore_errors=True) shutil.rmtree(self.raw_text_path, ignore_errors=True) def get_mindfiles_from_path(self, dir_path): dataset_files = [] for r, _, f in os.walk(dir_path): for file in f: if not file.endswith("db"): dataset_files.append(os.path.join(r, file)) return dataset_files @pytest.mark.run(order=1) def test_trainer_with_translation_args_train(self): """ Feature: Create Trainer From Config Description: Test Trainer API to train from config Expectation: TypeError """ batch_size = 1 runner_config = RunnerConfig(epochs=1, batch_size=batch_size) # 运行超参 optim_config = OptimizerConfig(optim_type='AdamWeightDecay', beta1=0.9, learning_rate=0.001) dataset = MindDataset(dataset_files=self.get_mindfiles_from_path(self.dir_path), columns_list=["input_ids", "attention_mask", "labels"]) dataset = dataset.batch(batch_size=batch_size) dataset = dataset.repeat(1) config = ConfigArguments(seed=2022, runner_config=runner_config, optimizer=optim_config) model_config = T5Config(batch_size=batch_size, num_heads=8, num_layers=1, hidden_size=32, seq_length=16, max_decode_length=8) # Model model = T5ForConditionalGeneration(model_config) mim_trainer = Trainer(task='translation', model=model, args=config, train_dataset=dataset) mim_trainer.train() @pytest.mark.run(order=2) def test_trainer_predict(self): """ Feature: Test Predict of the Trainer Description: Test Predict Expectation: TypeError """ # change the length for quick training model_config = T5Config(seq_length=32, max_decode_length=32) model = T5ForConditionalGeneration(model_config) mim_trainer = TranslationTrainer(model_name="t5_small") mim_trainer.predict(input_data="hello world", network=model) pengcheng.mind-mindformer/tests/st/test_model/test_tokcls_model/000077500000000000000000000000001476650141400255575ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_tokcls_model/__init__.py000066400000000000000000000012611476650141400276700ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_ner_model.""" pengcheng.mind-mindformer/tests/st/test_model/test_tokcls_model/test_tokcls_model.py000066400000000000000000000070121476650141400316470ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForTokenClassification class How to run this: windows: pytest .\\tests\\st\\test_model\\test_tokcls_model\\test_tokcls_model.py linux: pytest ./tests/st/test_model/test_tokcls_model/test_tokcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForTokenClassification from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # fine-tuning self.tokcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'tokcls') self.tokcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'tokcls', 'run_tokcls_bert_base_chinese.yaml') self.tokcls_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese') # evaluation and prediction self.tokcls_cluener_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'tokcls') self.tokcls_cluener_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'tokcls', 'run_tokcls_bert_base_chinese_cluener.yaml') self.tokcls_cluener_config = AutoConfig.from_pretrained('tokcls_bert_base_chinese_cluener') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'tokcls') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name tokcls_model_a = AutoModel.from_pretrained('tokcls_bert_base_chinese', download_checkpoint=False) tokcls_model_a.save_pretrained(self.save_directory, save_name='tokcls_bert_base_chinese') assert isinstance(tokcls_model_a, BertForTokenClassification) # evaluation and prediction test part # input model name tokcls_cluener_model_a = \ AutoModel.from_pretrained('tokcls_bert_base_chinese_cluener', download_checkpoint=False) tokcls_cluener_model_a.save_pretrained(self.save_directory, save_name='tokcls_bert_base_chinese_cluener') assert isinstance(tokcls_cluener_model_a, BertForTokenClassification) pengcheng.mind-mindformer/tests/st/test_model/test_tokcls_model/test_tokcls_tokenizer.py000066400000000000000000000051511476650141400325630ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing Tokenizer class How to run this: linux: pytest ./tests/st/test_model/test_tokcls_model/test_tokcls_tokenizer.py """ from mindformers import BertTokenizer, AutoTokenizer class TestBertTokenizerForChineseMethod: """A test class for testing the AutoTokenizer""" def test_from_pretrained(self): """ Feature: The BertTokenizer test using from python class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] tokenizer = BertTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] def test_auto_tokenizer(self): """ Feature: The BertTokenizer test using auto_class Description: Using call forward process of the tokenizer without error Expectation: The returned ret is not equal to [[6, 7]]. """ tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] tokenizer = AutoTokenizer.from_pretrained('tokcls_bert_base_chinese_cluener') tokenizer.show_support_list() res = tokenizer.tokenize("你好，世界！") assert isinstance(tokenizer, BertTokenizer) assert res == ['你', '好', '，', '世', '界', '！'] pengcheng.mind-mindformer/tests/st/test_model/test_txtcls_model/000077500000000000000000000000001476650141400256015ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_txtcls_model/__init__.py000066400000000000000000000012641476650141400277150ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test_txtcls_model.""" pengcheng.mind-mindformer/tests/st/test_model/test_txtcls_model/test_txtcls_model.py000066400000000000000000000065241476650141400317220ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test Module for testing functions of AutoModel and BertForMultipleChoice class How to run this: windows: pytest .\\tests\\st\\test_model\\test_txtcls_model\\test_txtcls_model.py linux: pytest ./tests/st/test_model/test_txtcls_model/test_txtcls_model.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig from mindformers.models import BertForMultipleChoice from mindformers.tools import logger class TestModelMethod: """A test class for testing Model classes""" def setup_class(self): """get_input""" # fine-tuning self.txtcls_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'txtcls') self.txtcls_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'txtcls', 'run_txtcls_bert_base_uncased.yaml') self.txtcls_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased') # evaluation and prediction self.txtcls_mnli_checkpoint_dir = os.path.join(MindFormerBook.get_default_checkpoint_download_folder(), 'txtcls') self.txtcls_mnli_config_path = os.path.join(MindFormerBook.get_project_path(), 'configs', 'txtcls', 'run_txtcls_bert_base_uncased_mnli.yaml') self.txtcls_mnli_config = AutoConfig.from_pretrained('txtcls_bert_base_uncased_mnli') # save path self.save_directory = os.path.join(MindFormerBook.get_default_checkpoint_save_folder(), 'txtcls') # the first method to load model, AutoModel def test_auto_model(self): """ Feature: AutoModel, from_pretrained, from_config Description: Test to get model instance by AutoModel.from_pretrained and AutoModel.from_config Expectation: TypeError, ValueError, RuntimeError """ AutoModel.show_support_list() support_list = AutoModel.get_support_list() logger.info(support_list) # fine-tuning part # input model name txtcls_model_a = AutoModel.from_pretrained('txtcls_bert_base_uncased', download_checkpoint=False) txtcls_model_a.save_pretrained(self.save_directory, save_name='txtcls_bert_base_uncased') # evaluation and prediction test part # input model name txtcls_mnli_model_a = AutoModel.from_pretrained('txtcls_bert_base_uncased_mnli', download_checkpoint=False) txtcls_mnli_model_a.save_pretrained(self.save_directory, save_name='txtcls_bert_base_uncased_mnli') assert isinstance(txtcls_mnli_model_a, BertForMultipleChoice) pengcheng.mind-mindformer/tests/st/test_model/test_vit_model/000077500000000000000000000000001476650141400250625ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_vit_model/__init__.py000066400000000000000000000000001476650141400271610ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_model/test_vit_model/test_auto_class.py000066400000000000000000000057531476650141400306420ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the swin interface used for mindformers. How to run this: pytest tests/st/test_model/test_swin_model/test_auto_class.py """ import os from mindformers import MindFormerBook, AutoModel, AutoConfig, AutoProcessor from mindformers.models import BaseModel, BaseConfig, BaseProcessor class TestVitAutoClassMethod: """A test class for testing Model classes""" def setup_method(self): """setup method.""" self.save_directory = MindFormerBook.get_default_checkpoint_save_folder() self.test_list = ['vit_base_p16'] def test_auto_model(self): """ Feature: AutoModel. Description: Test to get Model instance by input model type. Expectation: TypeError, ValueError, RuntimeError """ # input model name, load model and weights for model_type in self.test_list: model = AutoModel.from_pretrained(model_type, download_checkpoint=False) assert isinstance(model, BaseModel) model.save_pretrained( save_directory=os.path.join(self.save_directory, model_type), save_name=model_type + '_model') def test_auto_config(self): """ Feature: AutoConfig. Description: Test to get config instance by input config type. Expectation: TypeError, ValueError, RuntimeError """ # input model config name, load model and weights for config_type in self.test_list: model_config = AutoConfig.from_pretrained(config_type) assert isinstance(model_config, BaseConfig) model_config.save_pretrained( save_directory=os.path.join(self.save_directory, config_type), save_name=config_type + '_config') def test_auto_processor(self): """ Feature: AutoProcessor. Description: Test to get processor instance by input processor type. Expectation: TypeError, ValueError, RuntimeError """ # input processor name for processor_type in self.test_list: processor = AutoProcessor.from_pretrained(processor_type) assert isinstance(processor, BaseProcessor) processor.save_pretrained( save_directory=os.path.join(self.save_directory, processor_type), save_name=processor_type + '_processor') pengcheng.mind-mindformer/tests/st/test_resume.py000066400000000000000000000113461476650141400226200ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the gpt interface used for mindformers. How to run this: pytest tests/st/test_resume.py """ from dataclasses import dataclass import os import numpy as np import pytest import mindspore as ms from mindspore.train.callback import LossMonitor, TimeMonitor from mindspore.dataset import GeneratorDataset from mindformers.tools.utils import LOCAL_DEFAULT_PATH from mindformers.trainer import Trainer from mindformers.trainer.config_args import ConfigArguments, \ RunnerConfig from mindformers.models.gpt2 import GPT2LMHeadModel, GPT2Config from mindformers.core.lr import WarmUpDecayLR from mindformers import CheckpointMointor from mindformers.core.optim import FusedAdamWeightDecay ms.set_context(mode=0) def generator(): """dataset generator""" seq_len = 1025 input_ids = np.random.randint(low=0, high=15, size=(seq_len,)).astype(np.int32) input_mask = np.ones_like(input_ids) train_data = (input_ids, input_mask) for _ in range(32): yield train_data @dataclass class Tempconfig: seed: int = 0 runner_config: RunnerConfig = None data_size: int = 0 train_checkpoint: str = "" @pytest.mark.level0 @pytest.mark.platform_x86_ascend_training @pytest.mark.platform_arm_ascend_training @pytest.mark.env_onecard def test_gpt_trainer_train_from_instance(): """ Feature: Create Trainer From Instance Description: Test Trainer API to train from self-define instance API. Expectation: TypeError """ # Config definition runner_config = RunnerConfig(epochs=2, batch_size=8, sink_mode=True, sink_size=2) config = ConfigArguments(seed=2022, runner_config=runner_config) # Model model_config = GPT2Config(num_layers=2) gpt_model = GPT2LMHeadModel(model_config) # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask"]) dataset = dataset.batch(batch_size=8) # optimizer lr_schedule = WarmUpDecayLR(learning_rate=0.0001, end_learning_rate=0.00001, warmup_steps=0, decay_steps=512) optimizer = FusedAdamWeightDecay(beta1=0.009, beta2=0.999, learning_rate=lr_schedule, params=gpt_model.trainable_params()) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() ckpt_cb = CheckpointMointor(directory=os.path.join(LOCAL_DEFAULT_PATH, "test_resume")) callbacks = [loss_cb, time_cb, ckpt_cb] lm_trainer = Trainer(model=gpt_model, args=config, optimizers=optimizer, train_dataset=dataset, callbacks=callbacks, task="text_generation") lm_trainer.train(train_checkpoint=False) checkpoint_dir = os.path.join(LOCAL_DEFAULT_PATH, "test_resume", "checkpoint", "rank_{}".format(int(os.getenv("RANK_ID", "0")))) output_checkpoint_path = [ checkpoint for checkpoint in os.listdir(checkpoint_dir) if checkpoint.endswith('.ckpt') ] output_checkpoint_path = sorted(output_checkpoint_path, key=lambda x: os.path.getmtime(os.path.join(checkpoint_dir, x))) for _ in range(2): os.remove(os.path.join(checkpoint_dir, output_checkpoint_path.pop())) # Dataset and operations dataset = GeneratorDataset(generator, column_names=["input_ids", "input_mask"]) dataset = dataset.batch(batch_size=8) # callback loss_cb = LossMonitor(per_print_times=2) time_cb = TimeMonitor() ckpt_cb = CheckpointMointor(directory=os.path.join(LOCAL_DEFAULT_PATH, "test_resume")) callbacks = [loss_cb, time_cb, ckpt_cb] lm_trainer = Trainer(model=gpt_model, args=config, optimizers=optimizer, train_dataset=dataset, callbacks=callbacks, task="text_generation") lm_trainer.train(train_checkpoint=os.path.join(LOCAL_DEFAULT_PATH, "test_resume", "checkpoint"), resume_training=True) pengcheng.mind-mindformer/tests/st/test_text_generator/000077500000000000000000000000001476650141400237735ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/st/test_text_generator/__init__.py000066400000000000000000000012661476650141400261110ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test text generator.""" pengcheng.mind-mindformer/tests/st/test_text_generator/test_batch_generate.py000066400000000000000000000045541476650141400303470ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the batch infer for text generator. How to run this: pytest tests/st/test_text_generator/test_batch_generate.py """ import mindspore as ms from mindformers import AutoConfig, AutoModel, AutoTokenizer ms.set_context(mode=0) class TestBatchGenerate: """A test class for testing text generate features.""" def setup_method(self): """setup method.""" self.test_model_list = ['glm_6b', 'bloom_560m'] # self.test_model_list = ['glm_6b', 'llama_7b', 'bloom_560m'] def test_batch_generate(self): """ Feature: batch generate. Description: Test batch generate for language models. Expectation: TypeError, ValueError, RuntimeError """ question_list = [ "Hello", "Introduce yourself", "I love Beijing, because", "what color is the sky?" ] batch_size = len(question_list) seq_len = 256 for model_name in self.test_model_list: # set model config config = AutoConfig.from_pretrained(model_name) config.batch_size = batch_size config.seq_length = seq_len config.max_decode_length = seq_len config.use_past = True print(f"config is: {config}") tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModel.from_config(config) inputs = tokenizer(question_list, max_length=seq_len, padding="max_length")["input_ids"] print(f"inputs is: {inputs}") outputs = model.generate(inputs) for output in outputs: print(tokenizer.decode(output)) del tokenizer del model pengcheng.mind-mindformer/tests/st/test_text_generator/test_incremental_generate.py000066400000000000000000000036071476650141400315650ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing the incremental generate for text generator. How to run this: pytest tests/st/test_text_generator/test_incremental_generate.py """ import mindspore as ms from mindformers import AutoConfig, AutoModel, AutoTokenizer ms.set_context(mode=0) class TestIncrementalGenerate: """A test class for testing text generate features.""" def setup_method(self): """setup method.""" self.test_model_list = ['glm_6b', 'llama_7b', 'bloom_560m'] def test_incremental_generate(self): """ Feature: incremental generate. Description: Test incremental generate by input model type. Expectation: TypeError, ValueError, RuntimeError """ for model_type in self.test_model_list: config = AutoConfig.from_pretrained(model_type) # set incremental infer config config.batch_size = 1 config.use_past = True model = AutoModel.from_config(config) tokenizer = AutoTokenizer.from_pretrained(model_type) input_ids = tokenizer("hello")["input_ids"] output = model.generate(input_ids, max_length=20) print(tokenizer.decode(output)) del tokenizer del model pengcheng.mind-mindformer/tests/st/test_yaml.py000066400000000000000000000033471476650141400222640ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ Test module for testing yaml tools, MindFormerConfig. How to run this: windows: pytest .\\tests\\st\\test_yaml.py linux: pytest ./tests/st/test_yaml.py Note: the name of model yaml file should start with model. the name of model ckpt file should be same with yaml file. example: clip_vit_b_32.yaml starts with clip, and clip_vit_b_32.ckpt is model ckpt file. """ import os import pytest from mindformers import MindFormerBook from mindformers.tools import MindFormerConfig, logger @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.platform_x86_gpu_training # add for ci @pytest.mark.env_onecard def test_yaml(): """ Feature: MindFormerConfig Description: Test to transform yaml file as MindFormerConfig Expectation: TypeError """ yaml_path = os.path.join( MindFormerBook.get_project_path(), 'configs', 'clip', 'run_clip_vit_b_32_pretrain_flickr8k.yaml') yaml_content = MindFormerConfig(yaml_path) logger.info(yaml_content) assert isinstance(yaml_content, MindFormerConfig) assert isinstance(yaml_content, dict) pengcheng.mind-mindformer/tests/ut/000077500000000000000000000000001476650141400177045ustar00rootroot00000000000000pengcheng.mind-mindformer/tests/ut/__init__.py000066400000000000000000000012451476650141400220170ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """UT.""" pengcheng.mind-mindformer/tests/ut/test_layers_apis.py000066400000000000000000000023051476650141400236300ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test layers""" import numpy as np from mindspore import Tensor from mindspore.common import dtype from mindspore.common.api import _cell_graph_executor from mindformers.modules.layers import AlibiTensor def test_alibi_tensor(): """ Feature: Test the alibi tensor. Description: Test the forward Expectation: No exception """ model = AlibiTensor(seq_length=128, num_heads=32) input_mask = Tensor(np.ones((10, 128)), dtype.float32) mstype = dtype.float32 _cell_graph_executor.compile(model, input_mask, mstype) pengcheng.mind-mindformer/tests/ut/test_local_block_sparse_attention.py000066400000000000000000000121731476650141400272270ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test local block sparse attention accuracy""" import math import pytest import numpy as np import mindspore as ms from mindspore import Tensor from mindspore import nn import mindspore.ops as P from mindformers.modules.local_block_sparse_attention import LocalBlockSparseAttention class SparseAttention(nn.Cell): """ A Local Sparse Attention """ def __init__(self, seq_len, size_per_head, local_size, block_size): super(SparseAttention, self).__init__() self.seq_len = seq_len self.local_size = local_size self.block_size = block_size self.scale_factor = Tensor(math.sqrt(math.sqrt(size_per_head))) self.real_div = P.RealDiv() self.qk_bmm = ms.ops.BatchMatMul(transpose_b=True) self.mask_mul = P.Mul() self.mask_expand_dims = P.ExpandDims() self.mask_add = P.Add() self.softmax = P.Softmax() self.pv_bmm = ms.ops.BatchMatMul() self.multiply_data = Tensor([-10000.0], dtype=ms.float16) sparse_mask = self.gen_sparse_mask() self.sparse_mask = Tensor(sparse_mask, ms.float16) def gen_sparse_mask(self): """generate a mask for sparse attention""" sparse_mask = np.ones((self.seq_len, self.seq_len)) block_groups = self.seq_len // self.block_size prev_block_num = self.local_size // self.block_size - 1 for i in range(block_groups): row_begin = i * self.block_size row_end = row_begin + self.block_size col_begin = max(0, i - prev_block_num) * self.block_size col_end = col_begin + self.local_size sparse_mask[row_begin:row_end, col_begin:col_end] = 0 return sparse_mask def construct(self, q, k, v, attn_mask=None): """Forward process""" factor = P.Cast()(self.scale_factor, P.DType()(q)) q = self.real_div(q, factor) k = self.real_div(k, factor) sim = self.qk_bmm(q, k) sparse_mask = self.mask_mul(self.sparse_mask, self.multiply_data) sparse_mask = self.mask_expand_dims(sparse_mask, 0) sparse_mask = self.mask_expand_dims(sparse_mask, 1) sim = self.mask_add(sparse_mask, sim) if attn_mask is not None: adder = self.mask_mul(attn_mask, self.multiply_data) adder = self.mask_expand_dims(adder, 1) sim = self.mask_add(adder, sim) probs = self.softmax(sim) out = self.pv_bmm(probs, v) return out def data_compare(ground_truth, predict, diff_thd=0.001, pct_thd=0.001, max_diff_thd=0.1): """compare ground_truth and predict value diff""" total_count = np.prod(ground_truth.shape) greater_than_diff_thd_count = np.sum( np.abs(predict.astype("float32") - ground_truth.astype("float32")) > diff_thd * (np.abs(ground_truth.astype("float32")) + 1e-9)) greater_than_max_diff_thd_count = np.sum( np.abs(predict.astype("float32") - ground_truth.astype("float32")) > max_diff_thd * (np.abs(ground_truth.astype("float32")) + 1e-9)) diff_gt_thd_proportion = greater_than_diff_thd_count / total_count diff_gt_max_thd_proportion = greater_than_max_diff_thd_count / total_count result = "Pass" if diff_gt_thd_proportion > pct_thd or diff_gt_max_thd_proportion > 0: result = "Failed" return result @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_local_block_sparse_attention(): """ Feature: Test Local Block Sparse Attention Description: Test Local Block Sparse Attention Accuracy Expectation: result == "Pass" """ input_shape = (4, 32, 4096, 128) local_size = 1024 block_size = 128 q = Tensor(np.random.random(input_shape).astype("float16")) k = Tensor(np.random.random(input_shape).astype("float16")) v = Tensor(np.random.random(input_shape).astype("float16")) batch_size, seq_len = q.shape[0], q.shape[2] att_mask = Tensor(1.0 - np.tril( np.ones(shape=(batch_size, seq_len, seq_len), dtype=np.float16))) size_per_head = q.shape[-1] model1 = LocalBlockSparseAttention(seq_len, size_per_head, local_size, block_size, dropout_rate=0) model2 = SparseAttention(seq_len, size_per_head, local_size, block_size) out1 = model1(q, k, v, att_mask).asnumpy() out2 = model2(q, k, v, att_mask).asnumpy() result = data_compare(out1, out2) assert result == "Pass" pengcheng.mind-mindformer/tests/ut/test_lr.py000066400000000000000000000055451476650141400217430ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test lr schedule.""" import pytest import mindspore as ms from mindspore import Tensor from mindformers.core.lr import LinearWithWarmUpLR,\ CosineWithWarmUpLR, PolynomialWithWarmUpLR, \ CosineWithRestartsAndWarmUpLR, ConstantWarmUpLR @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_lr_schedule(): """ Feature: Test LR Schedule Description: Test Different LR Schedule Expectation: ValueError """ total_steps = 20 warmup_steps = 10 learning_rate = 0.005 lr_end = 0.0000001 constant_with_warmup_lr_std = [0.0005, 0.005, 0.005] linear_lr_std = [0.0005, 0.005, 0.0005] cosine_lr_std = [0.0005, 0.005, 0.00012236] cosine_with_restarts_lr_std = [0.0005, 0.005, 0.00289109] polynomial_lr_std = [0.0005, 0.005, 0.00050009] constant_with_warmup = ConstantWarmUpLR(learning_rate, warmup_steps) linear = LinearWithWarmUpLR(learning_rate, warmup_steps, total_steps) cosine = CosineWithWarmUpLR(learning_rate, warmup_steps, total_steps) cosine_with_restarts = CosineWithRestartsAndWarmUpLR(learning_rate, total_steps, warmup_steps) polynomial = PolynomialWithWarmUpLR(learning_rate, warmup_steps, total_steps, lr_end=lr_end) constant_with_warmup_lr = [] linear_lr = [] cosine_lr = [] cosine_with_restarts_lr = [] polynomial_lr = [] assert_steps = [1, 10, 19] for step in range(0, total_steps): if step in assert_steps: step = Tensor(step, ms.int32) constant_with_warmup_lr.append(constant_with_warmup(step).asnumpy()) linear_lr.append(linear(step).asnumpy()) cosine_lr.append(cosine(step).asnumpy()) cosine_with_restarts_lr.append(cosine_with_restarts(step).asnumpy()) polynomial_lr.append(polynomial(step).asnumpy()) error = 1e-8 for i in range(3): assert abs(constant_with_warmup_lr[i] - constant_with_warmup_lr_std[i]) < error assert abs(linear_lr[i] - linear_lr_std[i]) < error assert abs(cosine_lr[i] - cosine_lr_std[i]) < error assert abs(cosine_with_restarts_lr[i] - cosine_with_restarts_lr_std[i]) < error assert abs(polynomial_lr[i] - polynomial_lr_std[i]) < error pengcheng.mind-mindformer/tests/ut/test_metrics.py000066400000000000000000000047631476650141400227750ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test metric schedule.""" import pytest import numpy as np import mindspore as ms from mindspore import Tensor from mindformers.core.metric import PromptAccMetric, EmF1Metric @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_promptacc_metric(): """ Feature: Test PromptAccMetric Description: Test PromptAccMetric Expectation: ValueError """ logits = Tensor(np.array([[[[0.4, -0.9], [0.2, 0.7]], [[0.9, 0.09], [-0.4, 0.4]], [[-0.2, 0.05], [0.6, -0.8]], [[0.6, -0.4], [0.18, -0.56]]]]), ms.float16) input_ids = Tensor(np.array([[0, 1], [3, 7], [6, 2], [4, 4]]), ms.int32) input_mask = Tensor(np.array([[0, 1], [0, 1], [0, 1], [0, 1]]), ms.float32) labels = Tensor(np.array([[0]]), ms.int32) prompt_acc_std = 0 metric = PromptAccMetric() metric.clear() metric.update(logits, input_ids, input_mask, labels) prompt_acc = metric.eval().get("Acc", -1) error = 1e-8 assert abs(prompt_acc - prompt_acc_std) < error @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_emf1_metric(): """ Feature: Test EmF1Metric Description: Test EmF1Metric Expectation: ValueError """ str_pre = ["I love Beijing, because it's beautiful", "Hello world。"] str_label = ["I love Beijing.", "Hello world"] metric = EmF1Metric() metric.clear() for pre, label in zip(str_pre, str_label): metric.update([pre], [label]) result = metric.eval() error = 1e-8 f1_score, em_score = 75.0, 50.0 assert abs(result.get("F1", 0) - f1_score) < error and abs(result.get("Em", 0) - em_score) < error pengcheng.mind-mindformer/tests/ut/test_streamers.py000066400000000000000000000102761476650141400233300ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test generation.streamer schedule.""" from threading import Thread import pytest from mindformers import GPT2LMHeadModel, GPT2Tokenizer, TextStreamer, TextIteratorStreamer @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_streamer_schedule(): """ Feature: Test Streamer Schedule Description: Test TextStreamer Generation.Streamer Schedule Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextStreamer(tok) # Despite returning the usual output, the streamer will also print the generated text to stdout. result = model.generate(inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) result = result[0].tolist() assert result == \ [2025, 3649, 8379, 25, 530, 11, 734, 11, 1115, 11, 1440, 11, 1936, 11, 2237, 11, 3598, 11, 3624, 11] @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_iterator_streamer_schedule(): """ Feature: Test Streamer Schedule Description: Test TextIteratorStreamer Generation.Streamer Schedule Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") inputs = tok(["An increasing sequence: one,"], return_tensors=None, add_special_tokens=False) streamer = TextIteratorStreamer(tok) # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. generation_kwargs = dict(input_ids=inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) thread = Thread(target=model.generate, kwargs=generation_kwargs) thread.start() generated_text = "" for new_text in streamer: generated_text = "".join([generated_text, new_text]) assert generated_text == "An increasing sequence: one, two, three, four, five, six, seven, eight," @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_text_iterator_streamer_schedule_batch(): """ Feature: Test Batch Streamer Schedule Description: Test TextIteratorStreamer Generation.Streamer Schedule, with batch input. Expectation: ValueError """ tok = GPT2Tokenizer.from_pretrained("gpt2") model = GPT2LMHeadModel.from_pretrained("gpt2") text_batch = ["An increasing sequence: one,", "The highest mountain in the world is", "The largest river in China is"] inputs = tok(text_batch, max_length=8, padding='max_length', return_tensors=None, add_special_tokens=False) streamer = TextIteratorStreamer(tok) # Run the generation in a separate thread, so that we can fetch the generated text in a non-blocking way. generation_kwargs = dict(input_ids=inputs["input_ids"], streamer=streamer, max_length=20, top_k=1) thread = Thread(target=model.generate, kwargs=generation_kwargs) thread.start() batch_size = len(text_batch) generated_text = [""] * batch_size for new_text in streamer: for i in range(batch_size): generated_text[i] = "".join([generated_text[i], new_text[i]]) assert generated_text == \ ['An increasing sequence: one, two, three, four, five, six, seven, eight,', 'The highest mountain in the world is the Himalayas, which is the highest mountain in the world', 'The largest river in China is the Yangtze River, which flows through the heart of the country'] pengcheng.mind-mindformer/tests/ut/test_tokenizer.py000066400000000000000000000106551476650141400233360ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test tokenizer """ # import os import pytest from mindformers import AutoTokenizer @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_all_tokenizers(): """ Feature: Test all tokenizers. Description: Test all tokenizers, including save_pretrained, form_pretrained, tokenizer and return_tensors Expectation: No exception """ # path_name = "./save_vocabulary" # if not os.path.exists(path_name): # os.mkdir(path_name) tokenizer_list = ["gpt2", "bert_base_uncased", "llama_7b", "bloom_560m", "pangualpha_2_6b", "clip_vit_b_32", "glm_6b", "t5_small"] string_list = ["I love Beijing.", "我爱北京。"] return_tensors_sig = ["", "ms", "np"] all_res = [ [40, 1842, 11618, 13], # gpt2 english [22755, 239, 163, 230, 109, 44293, 245, 12859, 105, 16764], # gpt2 chinese [1045, 2293, 7211, 1012], # bert english [1855, 100, 1781, 1755, 1636], # bert chinese [312, 1545, 11285, 31843], # llama english [31822, 233, 139, 148, 234, 139, 180, 232, 143, 154, 231, 189, 175, 230, 131, 133], # llama chinese [44, 19134, 61335, 17], # bloom english [164830, 11385, 420], # bloom chinese [2787, 13, 4, 13, 28420, 13, 4, 37472, 34], # pangualpha english [21, 453, 263, 12], # pangualpha chinese [328, 793, 11796, 269], # clip english [162, 230, 239, 163, 230, 109, 161, 234, 245, 21078, 361, 38000], # clip chinese [115, 703, 8994, 7], # glm english [5, 76202, 64241, 63823], # glm chinese [27, 333, 14465, 5], # t5 english [3, 2] # t5 chinese ] for i, tokenizer_item in enumerate(tokenizer_list): tokenizer = AutoTokenizer.from_pretrained(tokenizer_item) for j, string in enumerate(string_list): for return_tensor_sig in return_tensors_sig: if not return_tensor_sig: result = tokenizer(string, add_special_tokens=False) print("{}: the result of {} is {}".format(tokenizer_item, string, result)) assert result.input_ids == all_res[2*i+j] result = tokenizer(string, padding="max_length", max_length=100, add_special_tokens=False) assert result.input_ids[:len(all_res[2 * i + j])] == all_res[2 * i + j] print("{}: the pad result of {} is {}".format(tokenizer_item, string, result)) else: result = tokenizer(string, return_tensors=return_tensor_sig, add_special_tokens=False) if return_tensor_sig == "ms": assert result.input_ids.asnumpy().tolist() == all_res[2 * i + j] else: assert result.input_ids.tolist() == all_res[2 * i + j] print("{}: the {} result of {} is {}".format(tokenizer_item, return_tensor_sig, string, result)) result = tokenizer(string, padding="max_length", max_length=100, return_tensors=return_tensor_sig, add_special_tokens=False) if return_tensor_sig == "ms": assert result.input_ids.asnumpy().tolist()[:len(all_res[2 * i + j])] == all_res[2 * i + j] else: assert result.input_ids.tolist()[:len(all_res[2 * i + j])] == all_res[2 * i + j] print("{}: the {} pad result of {} is {}".format(tokenizer_item, return_tensor_sig, string, result)) # tokenizer.save_pretrained(save_name=tokenizer_item) # path = os.path.join(path_name, tokenizer_item) # if not os.path.exists(path): # os.mkdir(path) # tokenizer.save_vocabulary(path, tokenizer_item) pengcheng.mind-mindformer/tests/ut/test_transformer_apis.py000066400000000000000000000615631476650141400247060ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """ test transformer""" import numpy as np import pytest import mindspore from mindspore import Tensor from mindspore.common import dtype from mindspore.ops import operations as ops from mindspore.common.api import _cell_graph_executor from mindformers.core import CrossEntropyLoss from mindformers.modules import MultiHeadAttention, FeedForward, TransformerEncoderLayer, TransformerEncoder, \ TransformerDecoder, TransformerDecoderLayer, Transformer, AttentionMask, FixedSparseAttention class MyActivation(mindspore.nn.Cell): """An example of custom activation""" def __init__(self): super(MyActivation, self).__init__() self.add = ops.Add() def construct(self, x): return self.add(x, 0.1) def activation_shard(self, parallel_config): self.add.shard(((parallel_config.data_parallel, parallel_config.model_parallel), ())) class MyActivationNoShard(mindspore.nn.Cell): def __init__(self): super(MyActivationNoShard, self).__init__() self.add = ops.Add() def construct(self, x): return self.add(x, 0.1) def test_transformer_encoder_only(): """ Feature: Test Transformer API Description: Test Transformer model with encoder only Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_log_softmax(): """ Feature: Test Transformer API Description: Test Transformer model with unexpected hidden act Expectation: No exception """ with pytest.raises(ValueError): model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_act='logsoftmax', hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_leakyrelu(): """ Feature: Test Transformer API Description: Test Transformer model with valid hidden act Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=2, decoder_layers=0, hidden_act='leakyrelu', hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_encoder_and_decoder(): """ Feature: Test Transformer API Description: Test Transformer model with encoder decode input Expectation: No exception """ model = Transformer(batch_size=2, src_seq_length=20, tgt_seq_length=10, encoder_layers=1, decoder_layers=2, hidden_size=64, ffn_hidden_size=64) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask, decoder_input_value, decoder_input_mask, memory_mask) def test_transformer_encoder(): """ Feature: Test Transformer Encoder API Description: Test Transformer Encoder model with valid input Expectation: No exception """ model = TransformerEncoder(batch_size=2, seq_length=16, num_layers=2, hidden_size=8, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_layer(): """ Feature: Test Transformer Encoder Layer API Description: Test Transformer Encoder Layer model with valid input Expectation: No exception """ model = TransformerEncoderLayer(batch_size=2, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_encoder_layer_post_ture(): """ Feature: Test Transformer Encoder Layer API Description: Test Transformer Encoder Layer model with post_layernorm_residual True Expectation: No exception """ model = TransformerEncoderLayer(batch_size=2, seq_length=16, hidden_size=8, ffn_hidden_size=64, num_heads=2, post_layernorm_residual=True) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) _cell_graph_executor.compile(model, encoder_input_value, encoder_input_mask) def test_transformer_decoder(): """ Feature: Test Transformer Decoder API Description: Test Transformer Decoder model. Expectation: No exception """ model = TransformerDecoder(num_layers=1, batch_size=2, src_seq_length=20, tgt_seq_length=10, hidden_size=64, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) def test_transformer_decoder_layer(): """ Feature: Test Transformer Decoder Layer API Description: Test Transformer DecoderLayer model. Expectation: No exception """ model = TransformerDecoderLayer( batch_size=2, src_seq_length=20, tgt_seq_length=10, hidden_size=64, ffn_hidden_size=64, num_heads=2) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) _cell_graph_executor.compile(model, decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) def test_multihead_attention(): """ Feature: Test MultiHead Attention API Description: Test MultiHead Attention model. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=2, num_heads=3) from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_multihead_attention_wrong_batch(batch_size): """ Feature: Test MultiHeadAttention with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=batch_size, num_heads=3) from_tensor = Tensor(np.ones((3, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((3, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('from_tensor,to_tensor', [(Tensor(np.ones((20, 15)), dtype.float32), Tensor(np.ones((20, 15)), dtype.float16)), (Tensor(np.ones((3, 20, 15)), dtype.float32), Tensor(np.ones((3, 20, 15)), dtype.float16))]) def test_multihead_attention_no_mask_2d_or_3d_shape(from_tensor, to_tensor): """ Feature: Test MultiHeadAttention no mask Description: Test MultiHeadAttention no mask and 2d as inputs. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, None) def test_multihead_attention_fp32_dtype(): """ Feature: Test MultiHeadAttention with float32 as compute dtype Description: Test using float32 as computation for linear layer. Expectation: No exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, compute_dtype=dtype.float32, batch_size=2, num_heads=3) from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float32) _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_transformerencoder_wrong_batch(batch_size): """ Feature: Test TransformerEncoderLayer with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=batch_size, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 16, 16)), dtype.float16) model(encoder_input_value, encoder_input_mask) @pytest.mark.parametrize('attention_mask', [Tensor(np.ones((2, 16, 16)), dtype.float16), None]) def test_transformerencoder_no_mask(attention_mask): """ Feature: Test TransformerEncoderLayer with no mask Description: Test the attention mask is None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=None, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones((2, 16, 8)), dtype.float32) model(encoder_input_value, attention_mask) @pytest.mark.parametrize('shape', [(2, 16, 8), (32, 8)]) def test_transformerencoder_2d_or_3d_shape(shape): """ Feature: Test TransformerEncoderLayer with 2d or 3d inputs Description: Test the attention mask is None Expectation: No exception """ model = TransformerEncoderLayer(batch_size=None, hidden_size=8, ffn_hidden_size=64, seq_length=16, num_heads=2) encoder_input_value = Tensor(np.ones(shape), dtype.float32) model(encoder_input_value, None) @pytest.mark.parametrize('batch_size', [1, 2, None, 4]) def test_transformerdecoder_wrong_batch(batch_size): """ Feature: Test TransformerDecoderLayer with wrong batch for training Description: Test the batch size to be any int or None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=batch_size, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) @pytest.mark.parametrize('decoder_input_mask,memory_mask', [(None, None), (Tensor(np.ones((2, 10, 10)), dtype.float16), None), (None, Tensor(np.ones((2, 10, 20)), dtype.float16))]) def test_transformerdecoder_mask(decoder_input_mask, memory_mask): """ Feature: Test TransformerDecoderLayer with empty mask Description: Test the mask is None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) model(decoder_input_value, decoder_input_mask, encoder_input_value, memory_mask) @pytest.mark.parametrize('activation', [MyActivation, MyActivationNoShard]) def test_transformerdecoder_custom_activation(activation): """ Feature: Test TransformerDecoderLayer custom activation Description: Test TransformerDecoderLayer custom activation Expectation: No exception """ model = TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, hidden_act=activation, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) model(decoder_input_value, None, encoder_input_value, None) @pytest.mark.parametrize('activation', [0, None, -1]) def test_transformerdecoder_wrong_activation(activation): """ Feature: Test TransformerDecoderLayer with wrong activation Description: Test TransformerDecoderLayer with wrong activation type Expectation: No exception """ with pytest.raises(TypeError): TransformerDecoderLayer(batch_size=4, hidden_size=64, ffn_hidden_size=64, num_heads=2, hidden_act=activation, src_seq_length=20, tgt_seq_length=10) @pytest.mark.parametrize('encoder_shape,decoder_shape', [((2, 20, 64), (2, 10, 64)), ((20, 64), (10, 64))]) def test_transformerdecoder_2d_or_3d_shape(encoder_shape, decoder_shape): """ Feature: Test TransformerDecoderLayer with 2d or 3d inputs Description: Test the attention mask is None Expectation: No exception """ model = TransformerDecoderLayer(batch_size=None, hidden_size=64, ffn_hidden_size=64, num_heads=2, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones(encoder_shape), dtype.float32) decoder_input_value = Tensor(np.ones(decoder_shape), dtype.float32) model(decoder_input_value, None, encoder_input_value, None) @pytest.mark.parametrize('hidden_act', [MyActivation, "relu"]) def test_transformer_hidden_act(hidden_act): """ Feature: Test Transformer hidden activation with activation or None Description: Test the transformer hidden activation Expectation: No exception """ model = Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act=hidden_act, ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) encoder_input_value = Tensor(np.ones((2, 20, 64)), dtype.float32) encoder_input_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) decoder_input_value = Tensor(np.ones((2, 10, 64)), dtype.float32) decoder_input_mask = Tensor(np.ones((2, 10, 10)), dtype.float16) memory_mask = Tensor(np.ones((2, 10, 20)), dtype.float16) model(encoder_input_value, encoder_input_mask, decoder_input_value, decoder_input_mask, memory_mask) def test_transformer_hidden_act_with_wrong_hidden_act_wrong_lambda_func(): """ Feature: Test Transformer hidden activation with activation or None Description: Test the transformer hidden activation Expectation: No exception """ with pytest.raises(TypeError): Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act=lambda x: x, ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) def test_transformer_hidden_act_with_wrong_hidden_act_wrong_str(): """ Feature: Test Transformer hidden activation with wrong activation Description: Test the transformer hidden activation Expectation: No exception """ with pytest.raises(KeyError): Transformer(batch_size=2, encoder_layers=1, decoder_layers=2, hidden_size=64, hidden_act="no_string", ffn_hidden_size=64, src_seq_length=20, tgt_seq_length=10) def test_feedforward_layer(): """ Feature: Test Feedforward layer. Description: Test the Feedforward layer Expectation: No exception """ model = FeedForward(hidden_size=15, ffn_hidden_size=30, dropout_rate=0.1, hidden_act='relu') tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) _cell_graph_executor.compile(model, tensor) def test_cross_entroy(): """ Feature: Test cross entory loss. Description: Test the forward Expectation: No exception """ model = CrossEntropyLoss() logits = Tensor(np.array([[3, 5, 6, 9, 12, 33, 42, 12, 32, 72]]), dtype.float32) labels_np = np.array([1]).astype(np.int32) input_mask = Tensor(np.ones(1).astype(np.float32)) labels = Tensor(labels_np) _cell_graph_executor.compile(model, logits, labels, input_mask) def test_attention_mask(): """ Feature: Test the attention mask. Description: Test the forward Expectation: No exception """ model = AttentionMask(seq_length=19) inputs = Tensor(np.ones((2, 19)), dtype.float32) _cell_graph_executor.compile(model, inputs) def test_sparse_attention(): """ Feature: Test the sparse attention layer. Description: Test the forward Expectation: No exception """ model = FixedSparseAttention(batch_size=2, seq_length=1024, size_per_head=64, num_heads=8, block_size=64) q = Tensor(np.ones((2, 1024, 512)), dtype.float16) k = Tensor(np.ones((2, 1024, 512)), dtype.float16) v = Tensor(np.ones((2, 1024, 512)), dtype.float16) mask = Tensor(np.ones((2, 1024, 1024)), dtype.float32) _cell_graph_executor.compile(model, q, k, v, mask) def test_attention_with_wrong_batch_3d_inputs(): """ Feature: Test Transformer batch error when the input's batch size is different Description: Test the input's batch size is different between the tensors. The input is 3d Expectation: Raise a reshape error exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) from_tensor = Tensor(np.ones((3, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((5, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) with pytest.raises(ValueError): _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) def test_attention_with_wrong_batch_2d_inputs(): """ Feature: Test Transformer batch error when the input's batch size is different Description: Test the input's batch size is different between the tensors. The inputs is 2d Expectation: Raise a reshape error exception """ model = MultiHeadAttention(hidden_size=15, src_seq_length=20, tgt_seq_length=20, batch_size=None, num_heads=3) from_tensor = Tensor(np.ones((60, 15)), dtype.float32) to_tensor = Tensor(np.ones((100, 15)), dtype.float16) attention_mask = Tensor(np.ones((3, 20, 20)), dtype.float16) with pytest.raises(ValueError): _cell_graph_executor.compile(model, from_tensor, to_tensor, to_tensor, attention_mask) def test_incremental_prediction_first_iterator(): """ Feature: Test MultiHeadAttention with incremental prediction Description: Test MultiHeadAttention with incremental prediction in the first iterator Expectation: No Expectation """ # Step 1: set is_first_iteration=True, and input the full sequence length's state. # We need to prepare the memory parameters for saving key and value states firstly. from_tensor = Tensor(np.ones((2, 20, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 20, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 20, 20)), dtype.float16) key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), dtype.float16) value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), dtype.float16) batch_valid_length = Tensor(np.ones((2,)), dtype.int32) model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, num_heads=3, use_past=True) model.add_flags_recursive(is_first_iteration=True) model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, batch_valid_length) def test_incremental_prediction_second_iterator(): """ Feature: Test MultiHeadAttention with incremental prediction Description: Test MultiHeadAttention with incremental prediction in the second iterator Expectation: No Expectation """ model = MultiHeadAttention(batch_size=2, hidden_size=15, src_seq_length=20, tgt_seq_length=20, num_heads=3, use_past=True) key_past = Tensor(np.zeros(shape=(2, 3, 5, 20)), dtype.float16) value_past = Tensor(np.zeros(shape=(2, 3, 20, 5)), dtype.float16) batch_valid_length = Tensor(np.ones((2,)), dtype.int32) # Set is_first_iteration=True to generate the full memory states from_tensor = Tensor(np.ones((2, 1, 15)), dtype.float32) to_tensor = Tensor(np.ones((2, 1, 15)), dtype.float16) attention_mask = Tensor(np.ones((2, 1, 20)), dtype.float16) # Step 2: set is_first_iteration=False, and pass the single word to run the prediction rather than the # full sequence. model.add_flags_recursive(is_first_iteration=False) model(from_tensor, to_tensor, to_tensor, attention_mask, key_past, value_past, batch_valid_length) pengcheng.mind-mindformer/tests/ut/test_ut.py000066400000000000000000000016361476650141400217530ustar00rootroot00000000000000# Copyright 2022 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test ut""" import pytest @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_example(): """ Feature: UT class Description: Test the ut Expectation: NotImplementedError, ValueError """ assert 1 < 2 pengcheng.mind-mindformer/tests/ut/test_version_check.py000066400000000000000000000031201476650141400241330ustar00rootroot00000000000000# Copyright 2023 Huawei Technologies Co., Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ============================================================================ """test version check schedule.""" import pytest from mindformers.tools.utils import is_version_le, is_version_ge @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_check_is_version_ge(): """ Feature: Test IsVerisionGreaterEquale Description: Test IsVersionGreaterEqual Expectation: ValueError """ assert is_version_ge("2.0.0rc1", '1.8.0') assert is_version_ge("1.8.1", '1.8.0') assert is_version_ge("1.8.0", '1.8.0') assert is_version_ge("2.0.0rc1", '2.0.0') assert not is_version_ge("1.8.0", '1.10.0') @pytest.mark.level0 @pytest.mark.platform_x86_cpu @pytest.mark.env_onecard def test_check_is_version_le(): """ Feature: Test IsVerisionLessEquale Description: Test IsVersionLessEqual Expectation: ValueError """ assert is_version_le("2.0.0rc1", '2.0.0') assert is_version_le("1.8.1", '2.0.0') assert is_version_le("1.8.0", '2.0.0rc1')

    P 藤县?P  ▁Aircraft@P  ▁PantherAP  荣誉证书BP 心静CP  reditedDP 绿道EP  必然会FP  珠海市GP  公平公正HP 黄码IP ▁RiJP  ▁distributorsKP  ▁GuarantLP IKMP  ▁SpecificallyNP behOP 幼小PP 救出QP  市场经济RP  个岗位SP  不增不减TP  ▁withstandUP  ▁antibodiesVP  买东西WP  国际社会XP 会谈YP  ▁FresZP  ▁Streaming[P 机缘\P ▁BBQ]P  也得到了^P  为大家带来_P  ▁skirt`P 三地aP  初中部bP 费和cP ▁/*dP  之后就eP  ▁StarkfP 奇石gP 蛋鸡hP  国际贸易iP 都快jP  ▁multifkP  ▁HartfordlP  ▁LogisticsmP  学生学习nP atomicoP 强悍pP 的鱼qP  atoriumrP  脆弱的sP  可以申请tP  ▁VolvouP 伴着vP  ▁NationalswP ▁FXxP 金地yP izzardzP 心心{P 梯田|P 知行}P 缠身~P  让身体P  ▁organizerP TESTP обP 颇具P  阿联酋P  一期工程P  基金经理P 暂定P  倍以上P  辅导班P  ▁разP  发布信息P  ▁NightsP  ▁reboundP  ▁reflectiveP  ▁FederP  ▁rebuildingP  ▁breezeP otiP  同意后P  将这些P  第二单元P  也不必P  ▁nineteenthP  ▁SheikhP  ▁enforcedP  如何选择P  ▁conservP  ▁rememberingP  TogetherP mightP  有一套P  安全检查P  指导中心P  ▁policymakersP 焊工P  ▁MirandaP ▁BubP ▁FryP avianP  ▁anthemP 拔尖P  具体安排P 互认P  投案自首P  ▁slippedP 略微P 进的P  日报记者P ikersP  ▁diaryP  ▁misunderP 先从P  不平凡P  丰厚的P  ▁DebtP  ▁matchupP 脚气P  ▁AdvocacyP  ▁dependenciesP 乐曲P 引诱P ▁MMP  营造良好的P 为名P binsP  ▁PicksP ▁UntP  特色产业P 立新P 南安P  ▁ReinP  试用期一年P 源性P  feedingP  ▁inclinedP 临终P 化成P  梦想的P  生命健康P  ▁narcP  ▁bishopP VeP  唯一一个P  是不可能P  ▁BalkP uloP  记录了P  ▁BharP 任用P 工科P 现今P 钢板P  ▁triggersP 爱不P  是很难P  辅导老师P  ▁critiqueP 怪物P 细纹P  预防性P  ConstructionP undaP 外壳P  ▁friendshipsP  告诉自己P  ▁enduredP  ManchesterP  ▁cathedralP  有针对性的P  ▁ThomsonP 痘印P  宽松的P  提供免费P 至尊P 鸟儿P  ▁rotatingP CuP  听到了P  ▁sentencingP 喜神P  ▁condoP  请作者P  ▁stuffedP 万丈P inxP OhioP  原生家庭P  统计数据Q  ▁MickeyQ 东兴Q 平湖Q  ▁backlashQ цаQ  ▁WarriorQ  ▁foreachQ  ▁hepatQ ulas Q  Friends Q 语种 Q  一次的 Q _; Q 一早Q 皇冠Q  涉案金额Q  都不用Q  也是比较Q  ▁hereinQ mistQ  历史上的Q 儿歌Q 卫浴Q 海岛Q  一瞬间Q  ▁accolQ 装甲Q  考上了Q  ▁refrigeratorQ 不凡Q RandomQ  ▁blended Q 一连!Q aci"Q  属牛人#Q  持续时间$Q  红领巾%Q  而没有&Q umbo'Q  ▁timer(Q  消防救援大队)Q  ▁commenced*Q  ▁treasures+Q  constitutional,Q 大葱-Q ▁"\.Q  ▁hover/Q  ▁Lighting0Q 太长1Q 静观2Q  ▁bacterial3Q  ▁casualties4Q  综合楼5Q  比其他6Q  日常的7Q ▁Gad8Q 用处9Q  ▁spiral:Q  历史学;Q  热爱生活Q  无数个?Q  ▁foreigners@Q MindAQ  团成员BQ 里最CQ  otheredDQ ▁};EQ  可以作为FQ  headersGQ  committeeHQ 等了IQ aaaaJQ  了什么KQ  中央财政LQ BFMQ  不是问题NQ adoopOQ  ▁loweringPQ PVQQ 她又RQ  我们都会SQ  CommercialTQ 刘诗UQ  不知不觉中VQ  ▁soundingWQ  ▁intervalsXQ 全班YQ  正规的ZQ  ▁Platinum[Q  ▁spins\Q 跨年]Q  你我他^Q кра_Q  乌兰察布市`Q unchedaQ  我们将及时更正bQ  ▁PioneercQ 行者dQ  ▁MitteQ casesfQ  ▁sumsgQ  ▁лиhQ  统战部iQ  ▁gluejQ  农家乐kQ rictedlQ  照见五蕴皆空mQ  ▁ObservernQ 觉察oQ inkipQ  一定要有qQ  和人民rQ  对的人sQ ▁COLtQ  也不过是uQ  ▁PhilharmvQ  ▁discountedwQ  ▁privilegesxQ  ▁ParenthoodyQ 祖师zQ  ▁sewing{Q  ▁stigma|Q  ▁genetics}Q tier~Q  ▁HousesQ  ▁SaskatchewanQ 爵士Q  的三个Q utanQ 何某Q 短路Q 校服Q 球鞋Q 碧水Q ▁AUQ ******Q  坚决防止Q  所大学Q  表达自己的Q  EnabledQ 边形Q 金鸡Q ▁SATQ  的年龄Q  ▁demolitionQ 农信Q flowsQ 世茂Q 孙俪Q  是十分Q ▁egoQ  中国企业Q 祥云Q  ▁amidstQ  ▁summonQ 杨家Q  ▁stampsQ 摇摆Q  的痛苦Q  历年真题Q  ▁BedfordQ  ▁IntellectualQ 充血Q  被认为是Q  ▁widowQ  权利人Q WHEREQ ▁IXQ  人大常委会主任Q  ▁donatingQ 爱是Q  遵守交通Q livedQ 化县Q 育种Q 交口Q 留影Q  将会有Q  新闻中心Q  被列入Q  ▁EqualQ  orningsQ simpleQ 中度Q 星火Q 次年Q ▁VilQ ☟　Q  搭配上Q  就能看到Q touchQ  如下图所示Q opathyQ  MappingQ  MetadataQ  ▁divorcedQ  ▁encompassesQ 她与Q SophQ  主要还是Q  的女子Q  共享单车Q  万亿美元Q  最终还是Q  keepersQ  ▁nurseryQ  ▁RevolutionaryQ  仍处于Q emakerQ  高新技术产业Q  ▁footstepsQ 把人Q 分部Q 钞票Q ▁),Q 生菜Q waldQ  是一个非常Q  ▁unanimouslyQ  整体运势Q  ▁contaminatedQ  ▁convenientlyQ  不良影响Q  我们国家Q  ▁biddingQ 为零Q 综述Q  ▁ChesQ  ▁clearerQ 山里Q 贵重Q  黑恶势力Q  ComputerQ  ▁decentralQ 反向Q  以实际Q  异常情况Q  有不同的Q  ▁NarrQ  ▁blendsQ  ▁EstoniaQ  ▁bowlingQ 亲肤Q ▁EAQ NNNNQ  你可能会Q  群众反映Q  补贴标准Q 子怡Q  赋予了Q  ▁SaturnQ  ▁premiumsQ ▁rimQ  ▁LectureR  ▁diligenceR 例句R 地支R GuideR  这个时代R  ▁DatesR 骨灰R TSDR  若涉及版权问题 R  ▁Smithsonian R 烂漫 R  世界杯 R  有更多的 R  ▁DoyleR 退换R  ▁SecretsR 无有R  ▁jeopardR 昌吉R 随笔R  一日三餐R 涅槃R 认罚R  大学城R  ▁tranquR 定性R essingR 丹丹R 降糖R 华晨R  ▁grapeR  arashtra R ZX!R  是很多"R  研究了#R 定理$R 著称%R  ▁slope&R undo'R  ▁Deals(R  具体如下)R 村组*R  围绕着+R  类产品,R  请留言-R 劣质.R  宣传和/R  职业发展0R ▁Cly1R  却因为2R  本科院校3R 挂靠4R ▁Hag5R little6R  ▁Petroleum7R 更易8R  厉害的9R 相符:R  创新高;R  本来就R  自然资源部?R 之计@R EdgeAR 洛克BR 紫薇CR  ventureDR  ▁differentialER 例题FR  走上了GR 枝叶HR 运化IR  ▁evolutionaryJR ijaKR  研发中心LR  ▁packagedMR " 所包含内容的准确性NR 等都OR 五天PR  新闻媒体QR ( !坚决打赢疫情防控阻击战RR 昨夜SR NFLTR  ▁CreatorUR  一贯制VR  ▁CrosWR  活动在XR  ▁socksYR  失业人员ZR  曾经在[R ilyn\R  rescent]R ▁DEL^R ecting_R  ▁finalist`R 弹药aR  ▁másbR  ▁grimcR 保佑dR 头镇eR undingfR  中获得gR  不使用hR  创新驱动iR  最高可jR  billionkR  employmentlR  ▁ManormR  管理委员会nR  ▁gearedoR  ▁FellowspR 小题qR 气短rR  业务办理sR 空前tR 要给uR  网格化vR  ▁accomplishmentwR 硕博xR  的传播yR 享乐zR  ▁Bali{R  ▁Juda|R  ▁academia}R  素材来源于网络~R ShellR  ▁MotoR  ▁SlamR  为主线R  在那个R  ▁redesignR 山中R 版面R ▁FiR  召开新闻发布会R  ▁BradfordR 安葬R  ▁AdultsR 永葆R  老照片R  ▁DixonR NLR 忙于R  部分图片R 喷施R  事实证明R bnbR  原出处所有R ______R  ▁AdmissionR 正大R 试听R ▁AJR ;}R  可能就R  勇往直前R  芥子园R rackR  有效证件R ActivR asmineR ▁VsR  部电影R  ▁mythologyR 苍穹R  善良的人R  高级职称R  综合症R  ▁TalkingR  ▁comprehensR  补助资金R ▁PACR 融融R 账款R ScoreR  MilitaryR  最低工资标准R  ▁meltingR  ▁leveragingR 三道R 两轮R  的温暖R  ▁controllersR 放任R 江口R 违者R  ▁aerospaceR 篆刻R 有更R 阳镇R ythmR ressoR  一年内R horseR  ▁LambertR ▁RigR  优秀学生R  contribR  ▁communalR 李兰R  提供有力R 三要R 堕落R 排位R 金寨R  代表队R  每天早晨R  能够得到R uffedR  亲密关系R 负载R  我们可以看到R  ▁SchwartzR  ▁illusionR 上头R  ▁веR icasR  ▁GoddR 人像R moilR  ▁destinedR 数目R  大学英语R  此时此刻R  ▁CarnivalR 太仓R 寻觅R 毋庸R  ▁luggageR  ▁boostingR  历史新高R  ▁realizationR 地和R 怀揣R 对有R  itchingR 建有R  抓实抓细R  ▁BudapestR  这一刻R  会有一些R 我爸R 自闭R  管理水平R examR  发布最新R 出一R 不还R billR  人人有责R  作为当前R 复出R  ▁NewarkR  ▁heroinR PyS  百公里S volumeS  ▁ScientistS  核酸阳性S 三无S  哺乳期S  次列车S 轩宇S  特长生 S 对准 S 达峰 S  大专学历 S  ▁headache S  ▁emergenciesS 结交S 的和S 趁早S  牵动着S  联系地址S 追加S TouchS agaraS  ▁PistS 炒香S ROUPS  教学工作S 燕郊S ubuntuS  ▁MartyS 中卫S 光鲜S 大堂 S Fly!S 你这"S 取向#S  安装了$S 妃识%S 深邃&S  将获得'S  路妃识(S  强制执行)S  ▁twisted*S  ▁explanations+S ▁Gos,S  ▁Plains-S chy.S  昨天的/S  ▁bend0S ،　1S  完毕后2S  基本要求3S  ▁friction4S ④　5S 假货6S sburg7S  ▁brit8S  选择一个9S  ▁youngsters:S 夜色;S 涉税S  ▁ethnicity?S rans@S ▁DilAS 接管BS  ▁GDPRCS RickDS  ▁MicheES  ▁JulietFS 排尿GS  最后一公里HS 你别IS  都开始JS  文化生活KS  necessaryLS  写了一MS 灵性NS  那句话OS quelPS radeQS  ▁deadlinesRS  富二代SS ▁CDsTS 但愿US 发抖VS  从哪里WS  ▁veinXS  副中心YS  合格人员ZS  做好疫情防控[S  ▁forensic\S  农副产品]S  海关总署^S rophe_S 俩人`S 意念aS ▁IbbS  微波炉cS 丰台dS  的人力eS 财商fS 副职gS  财产损失hS 应予iS  的步伐jS 暴饮kS 阴暗lS CredmS ExtranS  ▁PenceoS  的案例pS  为什么说qS 羲之rS  SourcessS  氏宗亲tS  ▁auntuS 典故vS enhwS  ▁owedxS 再添yS 明朗zS 爆破{S  老字号|S 巴斯}S ров~S  未通过S  ▁FargoS  ▁ubiquS  ▁imminentS  ▁specsS  ▁IntegerS  课题组S phaltS ingingS 店家S  ▁tougherS 照着S ▁NMS  ▁GodsS  安全防范S  的事故S  ▁RegimentS 兜底S 稳住S  了一天S  ▁poeticS  中国电信S 本分S  ▁BronzeS  ▁WorldsS  ▁occupationalS  FootballS  ▁constituentsS  是一只S  ▁SangS  融为一体S  ▁muralS 打脸S  不可怕S  这是在S  体温监测S  广大考生S  精神病S  还有其他S  ▁aestheticsS 柴米S  BreakingS 实打S 长势S  ▁prostateS  ▁ScholarsS  各个环节S  的细节S  ▁PromotionS 小偷S  人数的S  感染新冠病毒S  ▁digitsS  ▁ParticipantsS  救援人员S  检验检测S  ▁stagingS  ▁consolesS 编外S  我也是S  ▁hectaresS 中线S  批示精神S  教育机构S  ▁DancingS ZAS 九个S 甘露S IgnS  作为中国S SayS guidedS  ▁WHERES 曲张S  应聘岗位S 专一S ▁HedS  ▁susceptibleS hkS 分钱S  保护和S  沟通协调S  ▁ClubsS 多米S 鼠疫S 封装S 来势S 蛋壳S  招聘企业S 候车S  顺风顺水S  ▁RedsS 各路S ilingsS 都好S  教研室S  晋升空间S CommitS  fighterS 珊瑚S  ▁simplifyS jiaoS ▁NumS equivS ComparS 今夜S  ▁RatesS  请立即通知我们S 肉体S 背痛S 纵容S  也成了S 贴在S  ▁NolanS 互换S 李明S  星巴克S ▁RNAS  ▁digitallyS  ▁devastatedS 触目S  ▁finedS  ▁FitzgeraldS 不厌S  繁花似锦S  ▁MormS  ▁lampsS  ▁untoS  ▁baconS  ▁warriorT  ▁ResortsT  ▁DiscountT 一策T 周的T 专柜T  ▁trayT  ▁complementaryT 生在T 龙王 T  他们就 T  ▁commercials T  比利时 T  ▁atoms T 弱者T  ▁peekT 光源T 吉凶T 庄稼T  政治上T  应用场景T 传世T  于一身T  ▁MessengerT 平潭T  percentT reposT  ▁SupportingT 一读T  文化节T  studentT 本省T 蜜语 T 行之!T  ▁Lucky"T 起重#T 集约$T ▁fi%T  ▁Boyd&T KB'T  收到的(T  ▁lodge)T 奇瑞*T 理会+T 静坐,T  ▁exposing-T  ▁Femin.T 再者/T 诗句0T  经验丰富1T  ▁Sorry2T 貌似3T Cle4T  中提到5T  受过刑事处罚6T riott7T  ▁godd8T  ▁Inclusion9T  认证的:T 早于;T  有效衔接T  宣传力度?T  来看看你的@T 五指AT ▁gitBT  寒冷的CT  ▁glucoseDT  ▁JeepET  ▁MaskFT  烟台市GT  发生意外HT  ▁devotionIT  ▁paradigmJT 高腰KT  无法正常LT 脱下MT родNT  为首的OT  ▁IdeaPT  ▁ColonialQT  ▁SamanthaRT  ▁fraudulentST  招标文件TT EightUT 南国VT 我问WT 迫于XT entaYT  交叉路口ZT 八点[T  ▁wildly\T  可实现]T  参赛作品^T 新农_T  运城市`T astonaT  ▁fibrebT 不输cT 或与dT 计生eT 里边fT  ▁VectorgT  ▁elevatorhT  ▁PotentialiT  各县区jT  已经进入kT  IndustrylT  ▁NinamT aannT  ▁DeptoT  无新增确诊病例pT  ▁SaddqT  故事的rT  ▁mentioningsT MovingtT  ▁TexansuT  ▁asbestosvT 行道wT 超人xT 鼓掌yT  资产配置zT  小时前{T  安全问题|T  中国青年报}T  ▁Residents~T 珍宝T  ▁pneumT  ▁impactingT  ▁kingsT  优雅的T  爱人的T ▁MEDT  ▁LoopT  上来说T fuelT  抗体检测T 睿智T 馈赠T  发生了什么T  ▁SuffolkT 凹凸T 山谷T  ▁USDAT 　②T 晚间T  着力打造T 乖巧T 不喝T 益寿T 蜿蜒T  CoronavirusT 绝非T  在经过T 万头T 沦陷T  被征收T seinT 采矿T  ▁senderT  ▁taxationT 可向T  不是在T  微课视频T  urrectionT 作坊T 泊尔T  咳嗽等T  接种后T 王维T  ▁DiabetesT  ▁postseasonT  第三期T rankedT  ElectricT  近三年T stantT  更让人T  ▁transferringT ▁UmT  不久的T  为祖国T  交流会T  短时间T ▁^T  ▁spreadsT 苏区T tomT 苛刻T 繁杂T  解放路T  ▁PaintingT  ▁respectfulT  不就是T 亮了T  群体的T iolaT  ▁CatsT  省部级T ▁gumT  ▁UncleT 理化T  ▁shoutT  ▁InspiredT  这个是T ibabaT  ▁AlliedT  ▁paintsT  来判断T ▁TVsT 耐磨T  白居易T  ▁GomezT  featureT artonT remlinT  ▁GuestsT  ▁constructiveT ▁})T  ▁ScandinT  ▁conductsT 天人T  ▁depressedT  早期的T utscT  每个人都是T  ▁LaserT  ▁disgrT 想找T  指明了T SnowT  ▁mournT  病原体T  目标和T  领导班子成员T  ▁subsidiariesT 校级T ▁MACT  databaseT 落魄T 钦州T 逆势T ▁SulT 张雨T 求财T 墓地T 治好T CANT ativT  不扎堆T  ▁DidnT  交易日T  讲政治T  ethovenT  ▁insightfulT  ▁hospitalizedU 多病U 排量U MariaU  ▁arisesU  ▁schemaU  的年代U  本科及以上U  覆盖面U tube U  ▁burns U 海珠 U  他们用 U  想象中 U 持卡U нихU  ▁rentingU  ▁specializeU 只靠U  让你在U 俗人U 双面U ApartU  上一层U  formerlyU  ▁ClaytonU  详细介绍U issonU 雨季U  ▁villaU 茶水U erennU  ▁crus U  的事项!U  非常感谢"U  ▁releg#U  bedroom$U  teacher%U 先去&U  ▁под'U trade(U  是什么意思)U [$*U ▁Pig+U eight,U  ▁Vinyl-U bags.U  的比例确定/U  ▁utmost0U  ▁elegance1U  包括但不限于2U  ▁documentaries3U 外加4U  女生头像5U  ▁sympathy6U  该有的7U  全面依法8U  生活环境9U  ▁Units:U  让很多;U 赵国U  ounselling?U 铺垫@U  而得名AU  有一段BU LogoCU 逼近DU  中共中央总书记EU 裸露FU  关注雄安GU 神通HU ▁ainIU  受到党内JU  ▁WillisKU  ▁pillarsLU 国情MU 连带NU  汽车销售OU  ▁relievePU 堂课QU  年轻干部RU  ▁detainSU 人海TU 保额UU 小号VU 核发WU  值班值守XU 所处YU  ▁RomansZU 很累[U ▁TOP\U  全力支持]U  ▁Sadly^U 鄂州_U  信阳市`U 李大aU  ▁exceedsbU :#cU 桂平dU 我愿eU 联的fU 春色gU 柜子hU  和谐稳定iU  ▁quintjU  房屋的kU  拟聘用人员lU  ▁narratormU  需提前nU  ▁RealtyoU  天生的pU rosionqU  ▁miserrU 所思sU 趁机tU  按照要求uU  和社会发展vU 捡漏wU YOxU 隶属yU 相守zU  法律援助{U lis|U  都只是}U  都是以~U flatU  ▁GmbHU 浇灌U  售后服务U  满分为U  ▁irrigationU  ▁unemployedU DubU  网友爆料U JudgeU  巴彦淖尔U  ▁TurtU  ▁FrontierU 下药U  岁女孩U  主动权U  印度的U  ▁kindergU 触犯U  即将到来的U téU 献上U 高粱U pkU  PopularU  holdingU  ▁incomesU 以备U 营救U ▁REPU  ▁PaulaU IUU 疼爱U lyssU  高速交警U  ▁PlainU 了才U  ▁reopeningU  ▁ProgrammingU  保持距离U igatedU  ▁grantingU 悬疑U  第二代U  等资料U 万台U 安东U 德行U 沙河U 于都U 眨眼U 硬仗U ▁SplU  不过在U  离婚后U estineU  咨询有限公司U  PictureU  明天的U  ▁coronU 看成U  提供坚强U 尽检U obicU  ▁adjustingU  的指导意见U 不挑U 芋头U  持续的U  可打印U  ▁holdingsU 当兵U podU esqueU  ▁supervU  ▁hardwoodU 清廉U  ▁BackgroundU  检尽检U  科研成果U  ▁HermanU  ▁ZuckerU  推送规则U  生日快乐U  ▁agreeingU 力强U 水下U  严重违反U  ▁incurredU  ▁diplomacyU  吃多了U  转为确诊病例U  ▁QuantumU  ▁underestU  墨尔本U  百分百U  别样的U  解题技巧U 南极U 卡罗U 当着U  很可能是U SolarU ussieU  暂时的U  的可以U 大屏U  各省市U helmU  ▁GrabU 楼的U  不能用U  approvedU  ▁BahamasU 江城U  ▁FrequU 科兴U  MichiganU  农民工工资U  ▁dividendsU 赶走U uphU 出警U entureU  ▁triangleU 领证U  夫妻关系U ▁CBCU  住宅用地U  知识分子V  ▁archeV  ▁minorsV JuV 还清V  ▁AnythingV AnalyV 矿物V  的特征V  Donnell V 净值 V 曲江 V 波及 V  有信心 V  突破了V  什么问题V 做为V 克里V 无闻V  证件照V tcV  ▁congestionV izonsV  ▁secretlyV 买来V 滑梯V 颤抖V oblV  ▁mandatesV 难事V 叉车V NFV 黄的 V  线上学习!V ixun"V  教师资格认定#V acs$V duty%V  进一步优化&V 浩然'V 谁会(V  ство)V  ▁navigating*V 难行+V Church,V bara-V 公公.V 奉贤/V 是其0V  石墨烯1V 峥嵘2V 野战3V soon4V  ▁Dawson5V  ▁deepest6V desc7V  工业和信息化部8V  governmental9V 字迹:V 虚实;V  确保了V  ▁Maker?V  与服务@V  ▁massacreAV 站前BV MagicCV 老赖DV 落泪EV BLICFV  先进事迹GV  ▁draggedHV 配建IV 划线JV  但目前KV  ▁TinaLV 荆门MV 可言NV 挪威OV  ▁PartiesPV 堵车QV TopicRV  市场中SV  控制住TV  的目光UV 快充VV  第一眼WV 消瘦XV  的广告YV  指导教师ZV  ▁modelling[V  ▁compiler\V 伴奏]V  做人的^V  通行码_V  ▁Lund`V  ▁nouraV  ▁BuddhabV  ▁outingcV  ▁BenefitdV 吞咽eV ▁hexfV  仔细观察gV  同济大学hV  以优异成绩iV  ▁adornjV  ▁AwesomekV 实属lV  ▁cremmV  ▁MemornV  疫情发生以来oV  ▁kindlypV  ▁liquorqV IanrV  对我的sV wantedtV 又何uV  努力和vV  对公司wV  季后赛xV owieyV  关键环节zV  主会场{V Apply|V  那一年}V  发工资~V ywV  人均可V 主城V  团队成员V  ▁HenriV  已更新V  结果是V  criptiveV  ▁roboticV bikeV ▁phpV recognV  ▁neighbouringV lovV  品牌方V 佩奥V 钢厂V  ▁EuropeansV  的美丽V  ▁SyntV  ▁implyV  ▁delegateV  ▁playlistV 不伤V  开始学习V  ▁COUNV 俨然V 古董V TLV  她已经V  ▁prefixV TradeV  ▁EnsembleV 倾倒V  ▁hierarchyV  男性的V  及时了解V 山坡V 横批V 甘甜V  营造了V  普通话水平V 赛鸽V  将会在V  发布消息V 复式V 任你V 期权V 吃掉V 晓光V  ▁decayV 几家V  ▁volatilityV 刘伯V 提请V  好几个V 烟酒V  认真阅读V  ▁chapelV  到处都是V RegionV 筋膜V  为了给V 为数V ▁ByrV ▁SewV  ▁transitionsV 修道V 预料V ▁SERV  ▁clergyV  commercialV 低速V  孙中山V PluginV AUTV  ▁advertisedV 之机V 货到V 预先V SonyV leagueV  车辆和V  ▁offensesV  第三天V azarV 滨湖V  ▁viciousV ``V 多远V  不具有V  如出现发热V 板栗V 淳朴V  ▁parteV 法令V  经常被V BluV  ▁DarrenV  ▁fueledV 南街V  第三种V " 中小学教师资格考试V  认罪认罚V ▁VCV  主要原因V 举国V 超额V  文件夹V  文明出行V  还有机会V  进行检测V  ▁correctedV 较重V  ▁FrançV  很多地方V  ▁malariaV  ▁scalingV 公基V 向其V owellV  ▁MeatV  ▁circaV 补益V  属于你的V  ▁advisersV 甘于V  总有一些V 后盾V 圣贤W BuffW  ▁PokerW 赤裸W 达尔W  ▁suspendW 互利W  联合体W  本通告W 北魏 W 此地 W cop W  把你的 W ieri W  有条不紊W  信息后W sentW  各种各样W  ▁meaningsW 董卿W  要提前W  ▁turnoutW 原汁W 爱豆W  襄阳市W  传统节日W ▁foxW  打赢脱贫攻坚战W 公墓W 武清W  有所帮助W 孝敬W 顿饭 W  手术治疗!W  在这个时候"W 开讲#W  ▁honoured$W  度过一个%W  threatening&W 只得'W 宣扬(W iten)W kil*W  破坏了+W  一级响应,W  ▁Harley-W 水深.W 通缉/W ван0W  全面实施1W  ▁Shirley2W  ▁shaking3W  ▁underp4W  ▁Ceremony5W  来说说6W  ▁Correct7W  ▁replica8W  ▁laboratories9W  的中心:W  充分体现了;W 热播W gk?W 院系@W  的生态AW  iscopalBW  ▁HungerCW  unninghamDW  ▁ConsortiumEW 打到FW ▁TacGW  两年的HW  关于对IW batchJW łoKW 目击LW WARMW styNW  游泳池OW  广泛开展PW gevityQW  ▁passionsRW  感十足SW  可能存在TW onicsUW  ▁fulfilVW 迁入WW  持续提升XW 安危YW 放上ZW  之中的[W  ▁statues\W QA]W 执纪^W  所做的_W  ▁Cove`W  ▁compressaW  报名者bW  ▁MantcW  回复关键词dW 迫使eW  ▁reconsfW rollsgW  ▁WickhW  ▁bibliiW  ▁SourcesjW 中长kW ▁msglW  乘用车mW ▁HewnW 日及oW  常州市pW  控制好qW  哥伦比亚rW 超限sW PLCtW  ▁BuildingsuW  积极分子vW  politicalwW 庐山xW 有助yW 有待zW 师风{W Cry|W  分钟即可}W 小熊~W  ▁bloggersW  ▁StapW 封存W 掉头W 是看W  情形的W  ▁LanguagesW 口舌W 选定W 多款W 流星W 锁骨W 问世W  如何正确W  ▁regainW  ▁DrawingW  全面启动W ecycleW  ▁flownW  供给侧W  学到了W  铝合金W RainW 之恩W 走去W + $提供任何明示或暗示的保证W  人民大会堂W  等有关规定W 狗子W 玲玲W tanW  为患者W  空间站W  总投资约W  ▁EscapeW  要立即W  ▁FramW 亨通W  一律不得W 北欧W 流利W  技术服务W PETW qianW  视频监控W 唏嘘W 介质W 节节W  ▁urineW  ▁RafaelW  ▁repealW 供气W 所言W  年全球W  总有人W émonW  交易平台W 暴晒W  文言文W  预期的W  通俗易懂W  ▁ConversW 路途W 远洋W FaceW ▁woW  拥有着W  及时与W 地基W ZYW  人受伤W  杂志社W  肝功能W  长此以往W  ▁mottoW  ▁AdvisorsW  ▁skepticalW …)W 咽炎W  更多地W  ▁unbelievableW 和无W  力量的W  ▁PresentsW  ▁periodicW 签收W  妈妈说W uniqueW  　|　“W 邮电W  ToStringW 黄油W 成份W  苏东坡W  ······W 扁平W  网络的W  孩子上学W  ▁probableW  ▁repeatingW 催收W 读音W  风清气W  ▁RestaurantsW 内障W ▁VuW  重点领域W  呼吸道感染W  学生和家长W  次核酸检测W 叮咚W 辞去W 苦涩W easyW  换位思考W  ▁sturdyW 也与W 工大W  保障性W 共筑W 失意W  ▁grandsonW 针刺W SHAW  搞清楚W  营造良好W  FragmentW 请缨W  不出去W  结合自身X  是一片X  黑木耳X ruedX  专项资金X  ▁encryptedX  ▁broadcasterX  阿胶糕X ▁GolX  科学研究院 X ECH X 公鸡 X  谁知道 X leaf X  暴饮暴食X  近距离接触X  属猪人X  设计上X 代孕X 房内X 冲冲X  一年多X 倾力X  ▁BeanX ractedX 的单X  二次元X  交易市场X  学长学姐X  horizontalX  ▁indictmentX 壮志X  ▁sensory X 打法!X  ▁Forms"X 变道#X  科目的$X  我们尊重原创%X  ▁Wrestling&X 河湖'X  来解决(X  和大家分享)X licted*X  ▁Ethan+X  ▁Rental,X  最快的-X really.X 和老/X 美甲0X Theme1X  ▁lethal2X  主动配合3X  ▁disastrous4X  中国社会5X Alpha6X  试运行7X  当天的8X ibia9X  为了帮助:X  ▁basin;X 已过X  支气管炎?X imil@X SecretAX  ▁snapshotBX  见到了CX scientDX  matchedEX 唐人FX  戳这里GX 乃是HX 遗迹IX  玻尿酸JX  的人气KX  中国政府网LX  ▁deniesMX  不可否认NX  ▁GenerallyOX  ▁QuartetPX  开始前QX  SouthernRX  不用再SX  德州市TX ydiaUX  母乳喂养VX 管护WX  敏感肌XX  建筑施工YX 冷门ZX 赏识[X  大力支持\X Award]X Bruce^X  ▁Titan_X  ▁inflammatory`X  新城区aX  ▁ElementbX  ▁erosioncX 短裙dX  的世界里eX  ▁antennafX  的命运gX  环境影响hX  ▁ConserviX  至少要jX  ▁ExpedkX  ▁outputslX  ▁KathleenmX 客栈nX  一大早oX 装入pX ynaqX  受教育rX  穿起来sX  ▁craftingtX  成就了uX  走过了vX BurnwX  ▁plaquexX FabyX  ▁recapzX  ▁Tottenham{X Ir|X 错觉}X  情绪的~X GeneX ▁LehX  ershireX  可以给X  而是要X  ▁freedX  潜规则X  我爱你我爱你X  VictoriaX říX 采收X  的我们X  ▁YorkerX ekaX  方便了X  ▁petroleumX 选自X  电瓶车X  责任落实X  可以适当X AaronX  licensesX  ▁psychologistX 胸口X SATX  重症监护X 面霜X  ▁readinessX 鞠躬X 安溪X 讯息X  ▁assistantsX 话术X  ▁torchX  的主力X  还特别X connX  为人知的X 脑部X  灵活性X SilverX 乌龙X  尽最大X  ▁GujarX 偷拍X 由省X beansX alwaysX 六千X 必经X  ▁economistsX 浑然X 肿大X 应是X ▁WisX  ▁FoundedX 四军X 恐吓X 还把X  所产生的X 登高X 痴迷X  不熟悉X idableX  ▁RestorationX 催促X  级阵风X  ▁shinesX  进口的X  ▁ChessX  睡觉时X 奖补X 文殊X  和孩子一起X  ▁retainsX 囊括X izhX ulpX ▁cobX 木瓜X  相关企业X  营销中心X SantaX  ▁CreationX  ▁AuditX 打牌X  inarilyX  ▁anecdX  ografiaX  ▁KeepingX 五粮X 吊顶X 变老X  计划于X  合作关系X  ▁TortX 已将X 林地X 进村X ▁DOIX formsX 得水X 画谱X ▁NSAX 公然X mavenX  ▁PlantsX  ▁confirmingX ObjX  长久的X  带头作用X  ▁werdX 五代X 招办X  聊一聊X  uristicX 的身X  文化传播X TurkeyX  ▁demiseX  的计算X 关卡X 漂泊X  也喜欢X  ▁SchneiderX  ▁decreasingX 寿光X 调任X flexX  ▁cursorY  ▁substrY ArabY  附加扣除Y skipY  ▁AltoY 探望Y 清泉Y 维系Y  参考资料 Y  ▁Judicial Y  ▁nerves Y 苗苗 Y 运力 Y  ▁DiazY  ▁portrayalY 出面Y 驱车Y  持有的Y  综合利用Y  ▁AuctionY  ▁apologizeY 庭长Y  心电图Y  悟思想Y imonY  准考证打印Y  ▁volcanicY  但这个Y  足够了Y  ▁webpageY ▁RIY  也特别 Y  ▁Teresa!Y 周内"Y  提供给#Y  具备良好的$Y  ▁Kapoor%Y 来个&Y 前妻'Y Dat(Y  保持了)Y  碳排放*Y  实实在在的+Y  ▁Carlo,Y wm-Y 养家.Y 箱子/Y  ▁Bedroom0Y  ▁preschool1Y نى2Y 雨雨3Y  编者按4Y  ▁haunting5Y  开发票6Y 前夫7Y  ▁honoring8Y 「　9Y  被视为:Y  了一遍;Y 剂的Y 定能?Y 清甜@Y EEKAY  ▁wovenBY 病床CY 被他DY  工人们EY  旅游景点FY  ▁infiltGY 钢管HY 诀窍IY BloomJY  ▁dotsKY blowerLY 金茂MY  ▁SculptNY  不到的OY  佛菩萨PY 看花QY  人注意RY  ▁figuringSY 石板TY  报名费UY  ▁HankVY 惩治WY 深信XY  分享到朋友圈YY  云平台ZY  ▁insurg[Y 奉上\Y 导管]Y 临清^Y 熔断_Y 请到`Y  已经到了aY  ▁BrokenbY  帝国主义cY  防护物资dY obileseY  ▁armorfY HaygY  ▁curtainshY growthiY 隔断jY  生活里kY  劳动教育lY  甚至可以mY employnY  ▁PateloY JApY 衢州qY  ▁AlienrY 开运sY 转速tY ▁daluY  ▁AlexandravY  ▁veterinarywY 撰文xY 文博yY  蓬佩奥zY 大增{Y gre|Y  流行病学史}Y  usually~Y 无时Y  ▁wrapsY  从中国Y  ▁rebuiltY  ▁arthritisY  各方面的Y 日上Y EgyptY  ▁minusY 传唤Y 几条Y 华裔Y  诚实守信Y 这可Y 野蛮Y  ▁gallonsY ▁$$Y  行程码Y  ▁SunsetY  楞严咒Y  ▁raysY 亿吨Y  和一些Y  传说中的Y  城市轨道交通Y  平均分Y  ▁disciplesY 镇街Y niceY  疫情过后Y  ▁SchwarY 千玺Y 讲讲Y  ▁meritsY  登上了Y  ▁bidsY  教师资格证书Y essesY  aginationY 预制Y  等强对流天气Y 向日Y 秘制Y  ▁HansenY  ▁HoldingY  ▁WebsterY ▁±Y 亦有Y  不言而喻Y 抹黑Y  镇中心Y  ▁StackY  专项附加扣除Y 淋浴Y  那么的Y heardY  例无症状感染者Y  ▁FarmsY 吊装Y  ▁stoveY  ▁такY  广播电台Y  和信息Y  非常时期Y  ▁psychiatricY  一个人在Y  ▁messyY 汉代Y  如火如荼Y  ▁fattyY 自卫Y 财力Y zbekY eniumY 商贩Y 流连Y  具备了Y  ▁CrashY  ▁OpensY  ▁rusticY  ▁policingY  不过了Y cheonY  基本面Y  茂名市Y 潜逃Y  大公司Y  三国演义Y 释怀Y  clerosisY  的带领下Y ▁LIMY  企业经营Y HotelY 叶酸Y стеY 审验Y sampleY  ▁AnywayY  ▁progressedY  会受到Y  社会化Y  赵本山Y 文笔Y  ▁dwarY 跟谁Y 想买Y tlY 研习Y hraY  就是责任Y  licantsY 罚单Y LINY  应按照Y  比较容易Y  EnvironmentalY 标线Y ▁MamY  更进一步Y endenY  日常工作Y 文华Y 独居Y heritY  ▁AiresY  ▁MuseumsY NZY  本专业Y  ▁aggressivelyZ 三焦Z 熟能Z ▁EvaZ  mingtonZ  ▁PerkinsZ WBZ  ▁StefanZ  ▁StoresZ  完之后 Z Berry Z 幸存 Z  从去年 Z  宪法和法律 Z  体系和治理能力Z  ▁ecologyZ  大家应该Z  ▁wherebyZ 慢跑Z  车牌号Z  ▁SavageZ 办发Z 剧照Z ijiaZ switchZ  最终解释权归Z 惯例Z 并行Z 散寒Z igibleZ WaZ aratZ 补足 Z  已超过!Z  ▁peculiar"Z 性好#Z ▁tee$Z  技能大赛%Z  来看一下&Z  民生工程'Z  ▁upwards(Z  ▁году)Z 连同*Z 仅在+Z 挑灯,Z 连绵-Z  日开学.Z dead/Z 宫殿0Z  过敏反应1Z  ▁Nichols2Z  Individual3Z 太白4Z  与担当5Z ▁Ish6Z 嬉戏7Z  比如在8Z  水电费9Z  细致入微:Z 不生;Z 彩蛋Z  看不出?Z  ▁towel@Z  ▁evenlyAZ 海豚BZ  ▁ElonCZ 希尔DZ  还是一个EZ  ▁withdrawnFZ 凉了GZ 重病HZ  ▁InitialIZ qsJZ 棺材KZ  现在就LZ  说一句MZ ▁LahNZ  从而使OZ  生猪价格PZ  共同努力下QZ  ▁continentalRZ 纸条SZ  的培养TZ 藏族UZ  ▁erectedVZ bullWZ  锡林郭勒盟XZ 惟有YZ  台州市ZZ 常乐[Z  大问题\Z ▁ses]Z 大清^Z  肯定的_Z  ▁Transition`Z  个问题aZ  他就会bZ  集中整治cZ  ocumenteddZ  不仅能够eZ  自然保护区fZ  ▁masculgZ  厚重的hZ :'iZ 王丽jZ 美酒kZ  样东西lZ  ▁stallmZ  扫码进群nZ  有可能是oZ  ▁creepypZ  追究其qZ  ▁ExternalrZ  要小心sZ  那么就tZ afiuZ ancockvZ  积极投身wZ  FrameworkxZ  ▁OliveyZ  positivezZ yet{Z  你不要|Z  部编语文}Z  ▁observer~Z 错落Z  活动地点Z  ▁relentlessZ CosZ  ▁IsabelZ 先前Z 您和Z  属龙人Z  心目中Z  所以才Z 屡屡Z ADDZ CONTZ 曲阜Z 绝美Z 间歇Z  战斗中Z  可以提高Z 七里Z 宁乡Z 那点Z brokenZ 头村Z itrustZ  ▁fixesZ 鸿蒙Z  ▁HttpZ 保有Z 暂扣Z ▁"<Z  以保证Z  ▁manuscriptsZ 林场Z  ▁interiorsZ  ▁wivesZ  经济学家Z 桃胶Z 装盘Z 拖着Z  日子过得Z ombreZ 共赏Z 国医Z 她却Z 延庆Z  为用户Z  联系小编Z  发生火灾Z  饰演的Z  这个行业Z SyncZ  竭尽全力Z  ▁librZ  ▁AdvantZ  ▁inlineZ solZ  家的孩子Z  ▁GrillZ 前茅Z 灭绝Z  随着年龄的增长Z 健体Z 赤水Z  craftedZ 减法Z 四类Z  等材料Z  航天员Z  风景名胜区Z 有误Z  交易的Z egalZ 发红Z 收养Z 珍稀Z  入党誓词Z  一次又一次Z 领头Z ▁zaZ olkataZ 小马Z  ▁YachtZ 如潮Z 时下Z  ▁ebookZ 国门Z 理政Z  保时捷Z  ▁VeterinaryZ 拆卸Z  ▁visionsZ ▁ЕZ 许久Z 起锅Z 是名Z  后立即Z  anotherZ 从轻Z  美国和Z  应急管理部Z  ▁SuicideZ 裁决Z 进退Z BeatZ 哈登Z 复仇Z  及时关注Z PedZ  大道至Z  ParentsZ 点至Z 轻便Z  就知道了Z "<Z 迟缓Z  专卖店Z  翻炒均匀Z 立夏Z usableZ  ▁ConnorZ  ▁standaloneZ 从化Z  ▁allowanceZ ( !不符合食品安全国家标准Z нииZ  动员部署Z  县纪委监委Z 落差Z  比较低Z  ▁UEFA[  请查收[ 一票[  等国家[  ▁newsletters[  示范性[ games[ 太和[  无论在[  全力打造 [  ▁stalk [  ▁heightened [ 厌倦 [ 生冷 [ 茶树[ 谦卑[  医护人员的[ 位次[  developed[  ▁indirectly[  ▁Raleigh[  ▁transformative[ 一改[ 厚实[  ▁implicit[ ogens[ topic[ Ele[  张家口市[  ▁Surf[  年年底[  房屋出售[ odus [  纪律处分![ ▁ш"[ 守好#[ ITT$[  负能量%[  emiology&[ 配饰'[  等产品([  ▁Marines)[ 汤姆*[  民政部门+[  configuration,[  一段话-[  ▁mont.[  桂林市/[ 前几0[ 高密1[ Cola2[ ▁Paw3[  不住的4[ OWER5[  ▁furnished6[  ▁standardized7[ 履约8[ 意气9[  送往医院:[ oodle;[ 兰亭<[  圆满结束=[ По>[ 抽样?[ pred@[  副所长A[  运营的B[  治疗后C[  中汲取D[ GHzE[ ▁DumF[  全面小康G[  ▁mouldH[  ▁ignoranceI[  ▁proceededJ[ ▁OEMK[  课程安排L[  这段感情M[  猜你还想看N[  听不懂O[  ▁earnsP[  就因为Q[  市场规模R[ 报班S[  悄悄地T[ 法兰U[  童年的V[ ▁LNGW[  ▁DelegX[  套餐内容Y[ 裁剪Z[  招聘要求[[  系列产品\[  ▁outbreaks][ 百元^[  你点的每个_[ 耐力`[  ▁Povertya[  合作办学b[  ▁imprisonedc[  ▁Wentd[  existinge[  ▁ideologicalf[  一体化发展g[  ▁Registryh[ meii[ Altj[  角色的k[  ▁cheesl[  高血糖m[  怎么办呢n[ atibleo[ 爱车p[  ▁vowedq[  ▁cushionr[  ▁crossovers[ 封控t[  将面临u[  工作专班v[ ucklew[  Entertainmentx[ 静止y[ Levz[  风清气正{[ 警卫|[  房产税}[  ▁SERV~[ 调色[  不辜负[  ▁rains[ 塔尔[  园小区[  ▁spicy[  是非常重要的[ 栽种[ 长者[  直接把[  组织了[  ▁Pablo[ 略显[  无论你是[  ▁ideally[  这个道理[ liness[  文化产业[  覆盖率[ 祭祖[ 二线[  想方设法[ 稳稳[  ▁Goodman[ 产学[ 洗脑[ 皮革[ 露台[  的人会[ Sort[  ▁Albums[ 路透[ 阻滞[  ▁fren[ 桥村[  几年来[  放在心上[  这个项目[  ▁Seed[  ▁Hassan[ 奇观[ 炒房[ 魔幻[ 邑县[  　→　[  ▁farther[  爱情运势[  ▁sprink[ 掏出[ 无烟[  上进行[ Youth[  岁以上的[  百事通[  餐饮浪费[  最大限度地[ 定型[ sports[  ▁Bucks[  的含义[ iaison[  ▁Felix[  Charlie[  ▁Fletcher[  ▁longevity[ 有天[ 此情[ Ham[  传达了[  教育学院[  现代医学[ 夹杂[  ▁Exercise[  ▁preceding[ 一通[ odor[  ▁Lowe[ 油漆[  这一辈子[  Captain[  ▁socialist[  人员及[  和改进[ 青梅[  ▁athe[  ▁clicked[ 度数[ 赛车[ ▁Mai[  具有很强的[  来一场[  动态调整[  ▁Kerr[ При[ 低俗[ 缠绵[  的错误[  Features[  在今天[  所在小区[  ▁Failure[ Scot[  心甘情愿[ 她用[ 日止[  营业额[ movie[  申请书[  的汽车[ 管好[  长河中[  ▁voiced[  ▁equations[ Toy[  的动物[  Initial[  前期工作[  参加体检[ _([  ▁Fusion[ ibus[ ifles[ Quant[ later[ 抛开[ Dou[  我们一定要[  ▁coveted[ 削减[  ▁Jacques\ 书屋\  从来没\ 即墨\  欢迎转发\  调查组\  ▁prohibit\  ▁dismissal\ 也称\ 私处 \  时一定要 \ 两旁 \  写在最后 \  ▁simplest \ 大厨\ 大水\ 盘锦\ 酒厂\  ▁commemorate\ 薇娅\  第七届\  主动联系\  各年级\ 中都\  ▁slopes\  ▁Graphic\ 限于\  的手法\ 实绩\  和群众\  几乎所有\ rn\  信息网络 \  ▁Recogn!\  ▁spectators"\ 市属#\ 统领$\ scrib%\  身份证原件&\  语文园地'\ separ(\ 陶醉)\  营业厅*\  ▁strategically+\  的体验,\  许许多-\  ▁folding.\  ▁unexpectedly/\ 学弟0\  常委会会议1\  当地人2\  艺术设计3\  ▁parcel4\ 可得5\  ▁lime6\  也一直7\  疫情发展8\  ▁pigs9\  濮阳市:\  ▁Canberra;\  setting<\ 扑鼻=\  ▁splend>\ 复原?\ 抚慰@\  redibleA\ 免难B\  ▁guardianC\ 百草D\  全体员工E\ visoryF\  ▁KillerG\  做作业H\  等物品I\ ▁MaeJ\  湖北武汉K\  聚合物L\  合格证明M\  恶性循环N\  这个东西O\  ▁PassionP\  ▁ventilationQ\  大排查R\  MichelleS\  克里斯T\ 产假U\ 除湿V\ 巨变W\ 福禄X\  变电站Y\  全国人大常委会Z\ 　<[\ ▁Fur\\ pering]\ 抵触^\ 拟定_\  ▁Spectrum`\  ortmenta\  专家们b\ 招股c\ 纯属d\  兢兢业业e\  ▁Pregnf\ SVg\ 想了h\  和社区i\ 顶顶j\ 眷顾k\ 祸首l\ 肩胛m\  ▁Owensn\ 古筝o\  但这种p\  ▁dryingq\ 乏味r\  了一只s\ oodlest\ 市局u\ 做细v\  保护你w\  初三年级x\ 装着y\  行动中z\  虚假宣传{\ 看在|\  今天我}\  林生斌~\ 碗里\  Platform\  ▁standout\  ▁presenter\  在职场\  ▁Harvest\ ▁Ads\  ▁coloured\ ATS\  ▁admired\ 缩水\ 谈天\  警务辅助人员\ 兴致\ chel\  ▁uniforms\  该地块\  ▁Hulu\  ▁induction\  交界处\  二十三\  消防救援支队\  ▁plugins\  跨区域\  经办机构\ 严打\  对别人\  管理类\  ▁diploma\ 县直\  ▁Convert\  ▁Anglo\  不合适\  文化底蕴\  ▁enthusiast\ 揉搓\ ▁mu\ 吃上\ Gab\ itud\  只有一次\  ▁clone\  conversation\  深入实施\ 血肉\  精彩纷呈\  ▁gotta\ 斑块\ 青藏\ elles\  ▁responders\ 发丝\ 拉丁\  白内障\  紧迫感\  ▁Cumm\  ▁endurance\  执行的\  显而易\  陈女士\  新闻报道\  防暑降温\  ▁heater\  District\ 夹雪\ 好吧\  经济体\  ▁Neal\ 4 -新型冠状病毒感染的肺炎疫情防控\  援鄂医疗队\  ▁Translation\  ▁Rehabilitation\  作者的\  国有资产\ 它就\ 翩翩\ 托育\  刘老师\ 嫖娼\ 救赎\  ▁winding\ 超乎\  ▁Tucson\ Fig\  她自己\  ▁chaotic\ 上空\ 直男\  脑海里\  我们经常\  ▁Giov\ ха\ 天和\ 编译\  不要用\  一家三口\ 嫩滑\ 看我\ 占卜\  ▁emphasizes\  比去年\  ▁Savannah\  做一次\  ▁café\ 景气\  每一条\  FFFFFFFF\ 这组\  可行的\ faith\ 兰达\  进行审核\  防汛救灾\  全国人大代表\  atement\ 种地\ 花盆\  切不可\  ▁exploded\  ▁cigarette\  中国最\ campus\ ounty\  ▁Carpenter\ NJ\ 这叫\ Esp\  时可以\  ▁Malaysian\ 路南] 养鸡] 弊端]  来实现]  的技巧] Props] death]  Northern]  ▁Bridges]  专注力 ]  遥远的 ]  不会出现 ]  ▁showcased ] 减税 ]  民间借贷]  ▁Recording]  plugins]  ▁Hire]  ▁Dragons] 券后]  ▁rainbow] 大号]  ▁Bone]  ▁adaptive]  如何做] worm] 节食]  是不一样的]  住建部] okovic] 武穴]  管理能力] 匹林 ] 奇闻!]  判断力"]  疫情中#] posts$]  ▁Harlem%]  通州区&] 巡检']  ▁Catch(]  牡丹江)] Cool*] Drug+]  每天更新,]  设施建设-]  报告中.]  这并不是/]  在保证0]  我只能1]  方式的2] arte3]  ▁Aval4] 省时5]  也很有6]  ▁Nicolas7]  ▁costing8] 报应9] 平阳:] 集散;]  挑战赛<] 再无=]  从头到>]  ▁Brendan?]  ▁Lottery@] 在华A]  从小到大B]  活血化瘀C]  广泛宣传D] JesusE]  毋庸置疑F] 感同G] 择偶H]  短时强降水I]  ▁directiveJ] 时令K]  战争的L] 学妹M]  体育课N]  无接触O] ILLEP] 表的Q] 常驻R] 斯的S] PtrT]  中的应用U] sunamiV]  教学设计W] samX]  西兰花Y] 撕裂Z] atan[] nea\]  相适应]]  ▁novelist^] 缔造_] 辽阔`]  常见病a] 出站b] 纱布c]  看了就d] 温岭e] luentf] 一众g] 十方h] 感想i]  勤俭节约j]  观世音菩萨k]  ▁snippl]  ▁partisanm] 汉子n] 刚性o] 高烧p]  保温杯q]  危险性r]  彻底的s]  后即可t]  正当时u]  深呼吸v]  给自己一个w]  ▁liftsx] cutsy]  ▁Bromz] amento{]  ▁freedoms|]  ▁Balance}] Sel~] otions]  ▁gadgets] 乱停]  的小姐姐]  ▁Pepper]  ▁Basically]  请咨询]  解决的问题]  ▁discharged] 左转]  不可抗力] 拿走] 通宵]  很开心]  文件要求]  ▁disruptive] 无味]  防范化解] 老太]  产业基地]  但是因为]  ▁Accessed] /(]  又没有]  负责的] 立于]  ▁Conv] 视点]  立即抢购] 不育] fat] saf] 之花]  centered]  ▁adolescents]  综合自] ▁TRA]  点击阅读] 品茗]  有担当]  等一批]  拟录用]  ▁inconsistent] 浓烟] Emma]  ▁Mobility]  专业要求] 阴凉] 含蓄] 我却] 运载]  实地考察]  ▁campaigning]  ▁datas]  下水道] 入室] 小腹]  另一方] 甜味] 别具] 繁重] 红椒]  主要由]  ▁jailed] ▁COR]  做大做强]  ▁compelled] ▁yog]  ▁кон]  ▁favored] ibi]  ▁motivate]  贯彻新发展理念]  ▁Fors] 忠告] 苦辣] IFT]  本学期] 下课] 出战]  中国籍]  包括了]  升级版]  ▁Shooting] Tip]  ▁cried]  ▁Cannes] 协查] HOME] shift] Advert]  ▁Serial] 皇室]  动脉粥样] phthal] 向内] 建制] ▁Fee]  ▁coastline]  　*　]  到头来]  你的身体]  ▁forged]  长期坚持]  动动手指]  ▁perceive] 恍然]  马上就] riz] County]  ▁headers]  ▁fragments] 云朵] 伊利]  ▁payable] 雾霾] 遇漫]  ▁fostering] RD] 促成] 的肉]  ▁Stam]  ▁annex]  五分钟]  ▁turf]  ▁banning]  老师会]  ▁fragment] 那儿]  内容了^  的证据^  市民政局^ 大餐^ 好似^ Answer^  生产生活秩序^  ▁Presbyterian^ uti^  广阔的 ^  真抓实 ^  ▁cleanup ^  奥地利 ^  有用的 ^  把所有^  绍兴市^  计划表^ 和爱^ 情操^  岁之后^  ▁Sail^ 启东^ ▁sul^ ▁vor^  ▁Gabri^  问题进行^  ▁Trees^  ▁Seasons^  大众的^ 精读^  ▁zombie^ 每种^ 祠堂 ^  点亮在看!^  ▁highways"^  ▁Quarterly#^ 曲目$^ ▁Eur%^  ▁puppy&^  ▁waterfront'^  最简单的(^  预计今年)^  ▁scans*^ 大麻+^ 探秘,^ 兴义-^ 闭合.^ cry/^  学习效率0^ ILY1^ vivo2^ 衣着3^ 视同4^ 鹰潭5^  明亮的6^  法律知识7^  ▁WARRANTIES8^  是大家9^ 透亮:^  等功效;^  想象中的<^  ▁remedies=^ 东阳>^  ▁cheek?^  好莱坞@^  我们发现A^ 大区B^ ancoC^  还是没有D^ 换代E^ 确切F^  不明显G^  赔偿金H^ ckerI^ LowerJ^  万万没想到K^  ▁bossesL^  各位考生M^ 地主N^ ERNO^  各项目P^  在社区Q^ 顶峰R^  ▁RomeoS^ 水泵T^  去世后U^  高血压患者V^ 铲屎W^  ▁minersX^ 援手Y^  承办单位Z^  ▁realizes[^  建档立卡贫困户\^ 云海]^  活动价^^ Rele_^ atore`^  ▁Goodsa^ 已对b^ 叫停c^  ▁oathd^  ▁voyagee^  ▁roboticsf^ 假象g^  体检标准h^ 五号i^ 萌宠j^  曾说过k^  理财小白l^  保护作用m^  房屋概况n^ 睡了o^ ACSp^  心目中的q^  ▁typedr^ 甜言s^ 过低t^ 一筹u^ 年化v^ 稀有w^ 豆豆x^ atchedy^ 替你z^ 要和{^  ▁avail|^ 干妈}^ 扬尘~^  ▁sins^ 匪浅^  探测器^  ▁culp^ 国歌^ 抚州^  回应称^  ▁patented^ ▁Kai^ 贵港^ 海贼^ sudo^  第二季度^ alert^  均来自网络^  ▁tant^  ▁Clement^  ▁confrontation^ CEP^  高级教师^ rosis^ 空旷^  的一张^  ▁pillow^ 反动^  ▁competence^ 输卵^ 场均^  真的好^  网易云^ ▁NFC^ 白条^  ▁Stick^  rieving^ 海东^  因为这些^  直播平台^  aturdays^ Mit^ 美发^  决不能^ ▁Kad^ 殡仪^  侦办中^  属羊人^  看见了^  ▁expire^ 陈小^ 岁后^  发明专利^  Institute^ 一女^ 施用^  甜甜的^  柴米油盐^ wives^ 庆阳^  ▁Neck^ 立功^  但不是^  农民的^ 斤斤^  相似的^ athlon^  ▁Roland^  ▁Effective^  ▁exchanged^  工作环境^  平面图^  早治疗^  防控指挥部^ 邪教^  ▁Mits^ 双臂^  ▁cone^ ,@^ 损毁^ 轻奢^ 急速^ Late^  ▁veto^ ▁sig^  ▁apost^ }/^  ▁turmoil^ 偿失^ 分时^ 梁某^  对违反^  先进单位^ >@^ 加盐^  ▁Leigh^  Invalid^ 建构^ 脊梁^  手机和^  学习效果^  开开心^  提取码^  直接在^ nx^  火锅店^ irrel^ esk^  ▁Mohamed^ 本篇^ ACP^ Nik^  放在了^  Primary^  新教材^ ▁Moy^  书法作品^  文化交流^  ▁tilt^  ▁Digest^ 不至^  到什么^  发生率^  ▁arose^ 图虫^ 江宁^  年之后^  ▁crap^ shadow^  ▁Tropical^  也存在^  党员的^ 或被_  供不应_  社会保险费_ stroke_  二十一_  第四次_  ▁Kell_ Scroll_ 入心_ 益智 _ Zen _  ▁hello _  的姐姐 _  和质量 _  心理健康教育_ 四史_ 折算_  ▁waiver_ psych_  ▁consultations_  人流量_  决定是否_  无偿献血_ íc_ 盛行_ 郁郁_ 高光_ Anyway_  第二波_  等都是_ onica_  Contrib_  科技园 _  ▁hailed!_ 减半"_ 粒子#_  ▁Goal$_ 科尔%_  过来了&_  ▁Tory'_  ▁bracket(_ ahawks)_  ▁mediation*_  和行动+_ opus,_ enting-_  陪同下._  幼儿园教师/_ 倾心0_  站下车1_ 不妥2_ 与共3_ 集采4_  ▁Payne5_  ▁Infinity6_  ▁headaches7_  ▁shipments8_  宜昌市9_  ▁CEOs:_  交通运输厅;_  名副其实的<_  中高考=_  ▁bishops>_  ▁volcano?_ 成正@_ 诸侯A_  ▁crackedB_  再好的C_  紫砂壶D_  胃溃疡E_  ▁ReconF_ 边远G_  本年度H_  文化和旅游厅I_  收入来源J_  ▁replicateK_  了一番L_  会带来M_  境内的N_  无明显O_  ▁boilP_  ▁shipmentQ_ 图谱R_ aciaS_ 并茂T_  并伴有U_  专业学习V_ scenesW_  便开始X_ 三遍Y_ 备孕Z_ 霸权[_  去世了\_  的帮助下]_ 喊着^_  山水画__  素质和`_ Korea_  的丈夫b_ 二胡c_ 仁怀d_ 熠熠e_ ▁HQf_  ▁Napoleg_  ▁Wimbledonh_ ▁pci_  岁及以上j_  而是因为k_ 字形l_ 抢到m_  的第一个n_ 试管o_  ▁facilitatedp_ 车前q_  时间来r_  ▁Levys_  你就可以t_  坚决遏制u_ 乌拉v_ EXPw_ ▁Tubx_  如果大家y_  ▁obstructz_ 架空{_ conc|_  ▁amph}_  ▁Africans~_  ▁translations_ 倍数_ jas_  注射液_ often_  ▁Tribunal_ 脚手_ 通路_ 重于_  号地块_  供不应求_  Quality_  ▁Warsaw_ 各班_  ▁Nazis_ 听懂_ Git_  紧急情况_  ▁embarrassing_  跟孩子_ ▁Elm_ theme_  ▁cruc_ 资历_  ▁Alive_  中国公民_ bbs_ period_ 领养_  小视频_  肇庆市_  DateTime_ Ak_ 玉树_ 周易_  可能需要_  不知道怎么_ 如山_  ▁signaling_ 振华_  疲劳驾驶_  ▁Difference_  王羲之_  的种子_  轻松的_  这个事情_  ▁enclosed_ 督学_ aram_ 均未_ 感性_ 接踵_  三文鱼_  考试中心_ 徐汇_ 携程_  葫芦岛_ 传记_ 联结_ 赞成_ atro_  ▁Warn_  amacare_  加强了_  扶贫办_  ▁Indie_ 栖霞_  ▁Epid_  金针菇_ 跑车_ oooo_ fetch_  办理了_ entin_ criber_  ▁sedan_  ▁Pearson_ 德阳_  ▁Hawk_ illian_ 启程_  人性化_  研修班_  文化活动_  uddenly_ 张爱_ anca_  法律文书_  ▁lamb_ posure_ 曝出_  ▁Bulldogs_ 长裙_  ▁weaker_ choice_ 新奇_ 试想_  进行咨询_ 上层_ onomy_ 哪吒_ 惊天_  无尽的_ 允珠_  黑社会性质_  ▁configurations_ 探视_ qiu_  在不断_  ▁precautions_  中国国家_  中的一个_ 下设_ 甲等_ 之选_ 如皋_ 显高_  ▁guessing_ 得主_  ▁mammals_ 出没_ 办卡_  大运河_ 糕点_ 防癌_ forces_  ▁steer_ 太差_ rots_ ▁sql`  交通拥堵`  ▁artworks` 而今`  能为力`  这不仅`  从现在开始`  ▁definite`  ▁thirteen` änd `  ▁Positive `  ▁Brotherhood `  ▁enhancement `  ▁versatility ` }^` 的铁` ▁```  遭遇了` ▁PAR` 一缕` 如无`  直接用`  就是你的`  ▁landmarks`  健康养生` 考级`  ▁Diploma`  他们说` 清肺`  和理解`  的关心`  的比赛` NULL `  清洁能源!`  退役军人事务局"`  ▁screenings#`  第一场$`  ▁Residence%` 茁壮&`  食用菌'`  ▁су(`  ▁filename)`  关键在于*`  更多新闻+` idade,`  ▁degradation-` 调用.` ieder/` 默写0` Seg1`  咽拭子2` antine3`  ourning4`  ▁governors5`  思想和行动统一到6`  ▁manipulate7`  的热量8`  ▁visas9`  ▁platinum:`  ▁trustees;`  有困难<`  谢谢大家=` Museum>` 吹响?` 蓬莱@`  生活馆A` HungB`  ▁dancesC`  ▁alarmingD`  ▁HertE`  ▁BelieveF`  ▁TributeG`  岁男孩H`  时间和地点I`  ▁terminatedJ`  如出现K`  餐饮业L` shieldM` 不戴N` 乐事O` 直觉P` WORQ`  东城区R` iblingS`  ▁underscT`  也不再U`  ▁HoleV`  长按图片W`  ▁AerospaceX` 神木Y` 送的Z`  平常心[`  把他们\`  顺义区]`  在中国的^`  看望慰问_`  ▁Lance`` 饱腹a`  ▁swayb` 知性c` 竹子d`  的兴趣e`  采用的f` 真谛g` hwh` 瑞幸i` 却也j`  百年党史k` 如对l`  章子怡m`  ▁Gloun` 中印o` 但当p`  Weatherq` 很深r` EROs`  型人才t` 两件u`  ▁Sachsv` opersw` Sqlx`  人事考试网y` 提防z`  不动摇{`  ▁trustee|`  ▁Headquarters}`  出了一个~`  基层干部`  ▁shells` 我做`  的资源`  的高低`  同步练习`  就会变得`  ▁Shan`  ▁Thames` 新任`  ▁perv`  ▁assemble`  的防控`  要不然`  ▁Omicron`  港澳台地区`  ▁physic`  ▁grapes`  ▁duplicate`  学习内容`  打造一个`  ▁unpaid` 祖母` 刑拘`  很容易被`  ▁forgive` _,` 家书`  普通类`  门把手`  找到一个`  ▁salon` 秋收`  ▁economical`  第十一` Prev`  ▁Chandler` 安好` ▁Ci` 瓶装` zilla`  Toronto` Lif` bish`  和自己的`  独一无二`  其原因`  斤斤计较`  ▁completes` 林木` oldid`  ▁Coca`  ▁coordinating`  ▁believers`  ▁waterproof` 尊贵` 军衔`  上百万`  也是我们`  有序恢复` illac` ;">`  翻天覆` xlsx` trust`  都很好`  精心组织` 今朝` 豪情`  全面推行`  环境污染`  经验丰富的`  ▁shrink`  文章中`  理论学习中心组` zos`  缴纳社保`  聘用手续`  iliated` 一艘`  一流的`  ▁angular` 桂枝` nail`  initial` 攻势` 逐梦` typ`  真的会`  有针对性地` 老友` 脚的` 诗文`  古建筑`  作息时间`  经济师` Flash`  ▁hipp`  ▁reputable`  pleasant`  ▁clinicians`  公司将` 弗洛` 时在`  在我看来` errals`  主演的`  管理模式`  ▁unwilling`  手里的`  税务部门`  好身材` 站稳` 袭来`  民生实事`  ▁testify` 陷害`  不好的人`  家国情怀`  已经过去`  一段感情` fried` 发发` 国之` 占到` 抄报` nb` 绘图`  为代表`  ▁privata  ▁emergesa  包吃住a  空气流通a Asiana  ▁attackera 估价a 吞吐a  可能被a  持续到 a  量身定制 a 谁说 a ",　 a  ▁Borg a  microsofta  十四条a  ▁timeouta  农商行a  申请的a  ▁Signala 换证a 牟尼a  ▁rippeda  很多人都会a 蜻蜓a stooda  ▁Hawkinsa 岩松a 床垫a  小故事a  免门票a  的表达a  节目组 a 助人!a 葬礼"a 春分#a  情况来看$a  ▁toilets%a 屈原&a 肥肉'a ник(a зы)a  的美誉*a  情况通报+a  ▁borrowed,a  文学家-a  ▁entertained.a 清缴/a  ▁exert0a AJ1a 拿去2a  ▁Clerk3a  传染病防治法4a udi5a ▁arr6a  就像一个7a  ▁gast8a  ▁suppress9a 向善:a  需要有;a  ▁Hatea uku?a  江门市@a  默默无闻Aa  ▁salvBa  ▁HimalCa 霞客Da  立案侦查Ea ucharFa  保鲜膜Ga ▁SSDHa  ▁sectIa 胆汁Ja FGKa  本来就是La andalsMa 一箱Na 内而Oa  ▁immenselyPa 扬子Qa  期满后Ra  在这方面Sa celainTa  ▁psychedUa CXVa 耐受Wa asoXa ག་Ya 大安Za  ▁Rivera[a 矢志\a  公筷公勺]a  提供虚假^a  ▁Corey_a  ▁Crypto`a 佼佼aa 风里ba bbieca  编辑整理da  执业医师ea  ▁cheatingfa 我好ga matrixha 税法ia  ▁commoditiesja 打字ka 表哥la  家属院ma  静脉曲张na ettleoa 连心pa rittqa  ▁blockbusterra  少一些sa Snapta 声响ua  可再生va  舒适性wa  ▁toesxa 发黑ya 存货za  雨夹雪{a  线上课程|a  ▁repaired}a  基层党建~a  如涉及侵权a  ▁unsuccessfula 带好a 涉水a 软的a  培养出a 给付a 谐音a  ▁Vincea 自建a  师资力量a 汇丰a  实际上是a Womana 带了a 或用a atakaa  第三人民医院a  ▁descendantsa діa 急难a  黔东南a 收尾a 警长a  ▁Fruita Kenta  ▁fleeinga  ▁Compensationa  中南部a riceda  ▁sobera 本命a 海运a Arma  合同约定a 金丝a  很多小伙伴a loadera  修正案a  动起来a  licensea  不理想a  ▁Rangera 罗某a  拉开帷幕a  研究生院a  和图片a + $习近平总书记重要讲话精神a  最前沿a Yorka  中进行a  贫困家庭a 字符a oaka  氏家族a  terburya  并配合a 旧的a 吴忠a 孩童a  ▁Uttara 错位a 进士a ushua ▁Tula  他们都是a  ▁Electorala  有人问a  ▁appliancea  ▁influenzaa  ▁concurrenta  为你的a  孙允珠a  的大脑a ▁Toua  ▁takeovera ▁CFOa  也没什么a  代名词a  优势和a 栀子a  而不能a  ▁condola  常务会议a 河里a  坚决杜绝a vettea  ▁nonprofitsa 醒悟a  报考岗位a  ▁composta 停机a  ▁rainya 拜托a 笃定a  和书法家a  思维能力a  第二中学a 骁龙a  卫健局a  看守所a  constructa nena  另一边a  ▁Veterana  ▁pleasinga  肺结核a ▁Boma iberna  ▁voluntarilya ~)a 台前a 撒谎a 培优a 轻视a  ▁atoma  ▁rivalrya  Universala  主题为a 有句a  算得上a ▁Wena  一个多月a  ▁absorptiona 认领a atla  深处的a  ▁Marcela 发至a  找机会b Connb  ▁syrupb 惯性b  若干名b  参加高考b 听觉b  与学生b  总数的b  thereum b  欢迎加入 b  ▁surgeries b  这个动作 b  ▁mills b 他才b 相等b  骄傲和b ▁narb ensisb  Bachelorb 它还b 我被b  自驾游b  ▁Malib  ▁Calledb untaryb 打坐b 达拉b  土木工程b 只顾b 爆笑b  需求量b  期限为 b  ▁werden!b  ▁dispers"b  ▁homicide#b 褶皱$b  号台风%b  ▁Bing&b  ▁antagon'b  日截止(b  ▁exercising)b  我们无法*b  ▁moms+b  usement,b  ▁toddler-b 弹簧.b 脚本/b  忽视了0b  ▁как1b  防灾减灾2b  reviewed3b 干枯4b 加我5b 尔蒙6b 辨认7b  正式实施8b  ▁Assault9b  ▁squeeze:b  ▁Concrete;b 专著b  也不同?b  西北部@b pillAb  ▁OmahaBb StageCb  ▁periodicallyDb  网友表示Eb  波澜壮阔Fb  performingGb 剁手Hb becueIb  ▁FranzJb 寿司Kb PubLb  和处理Mb  ▁quiltNb  也有不少Ob  ▁paranPb  ▁TrialsQb 草书Rb 分销Sb 相知Tb  火龙果Ub  ▁MortgageVb  但是你Wb  ▁agileXb RMYb  到学校Zb  新乡市[b  ▁notch\b 机身]b 清运^b 着陆_b  的风景`b  视频课程ab  再怎么bb  到夜间cb  通过手机db 德才eb  对于一个fb  ▁Deckgb  和分析hb  更愿意ib  ▁outgoingjb  ▁landscapingkb 饮茶lb  终于可以mb  ▁Purduenb  ▁hollowob ykpb  的刺激qb ▁AAArb  ▁accessorysb  ▁iterationtb bangub 灵的vb  Softwarewb  ▁Portfolioxb 曲阳yb  ▁rooftopzb 殊荣{b 特旗|b 闲聊}b heets~b  心绞痛b  科技城b ustleb 吉尔b 长在b  也有所b 之争b 平滑b  不配合b  获奖名单b  市场供应b  ▁Salonb  ▁Comfortb 忘却b 手写b  检测报告b ▁Öb 挺好b eptb  及时就医b Scaleb  ▁disciplinaryb 正道b audiob 飞往b  ▁insecurityb POLb resetb  丰富多彩的b  avanaughb  实际效果b  师德师风b  异地就医b 隔绝b  的生活中b  ▁Aberdeenb 初夏b 国境b yclingb  我们认为b  是一份b ▁SUNb  ▁instabilityb  白云区b 物力b  的长度b  ▁Edmundb  ▁Advocateb 苗条b  ▁Barberb 走遍b  ▁Codyb  ▁cultivationb 节的b Jewishb 从没b  集聚区b  经营主体b  ▁engagesb ▁sorb  扶贫搬迁b alogyb  ▁Discussb  Registrationb 备好b 是说b  ▁Accomb 而归b  ▁Candidatesb  ▁Stadb  ▁clutchb 菜的b 虾仁b  ▁frequenciesb  送到了b  网红打卡b ückb ▁bakb  ▁reuseb 鞋底b  ▁grievb 不菲b 内卷b '''b  银行存款b 成家b 锦上b  ▁Idolb 体美b untab  招生人数b thorab  ▁Bonusb  在水中b HTTPb  ▁crystalsb 争创b 踏板b  ertilityb  毕业班b 共存b 小屋b 要为b 诊室b  传播的b  ▁toastb  ordinateb Tib  有可能会b ‍　b  ▁postalb  ▁melodiesb 大变b ▁WSb mondsb 斤的b  市人社局b  historicb  ▁metallicb  ▁servicingb 桑葚b 满月b 距今b ▁CSTb 米处b  病毒性b 提神b  向记者b  纸质版b  感同身受c  Administc 吝啬c 洗面c Qualc 绿树c 蜂窝c ないc  像极了c  ▁Lomb c  ▁curved c 风和 c  不规律 c  文章底部 c 李雪c  使他们c Lostc ▁Gelc ▁faxc ascadec  科学城c  战国时期c  调查处理c  auticalc  ▁averagesc 丰顺c 想尽c  考试时c 痛快c  排放标准c Askedc  ▁filingsc 深色 c  办公室副主任!c  ▁simplified"c 依依#c 相待$c 绿洲%c  推广应用&c  ▁irony'c 丹参(c 于人)c seven*c ▁KS+c  和各种,c 乌鸦-c  免费开放.c 可查/c  报告会0c  数学下册1c  Fashion2c 塔罗3c  积极应对4c  foreach5c 慰藉6c Sah7c  ▁cryptocurrencies8c seas9c  ustomed:c  ▁displacement;c yrec  实名认证?c  评选活动@c  症状的Ac appsBc " 人正在接受医学观察Cc ísDc 角的Ec  山街道Fc 三路Gc CroHc 双拥Ic  劳动法Jc  环境优美Kc  线上直播Lc 比武Mc 乡亲Nc 过时Oc  的事迹Pc  icheverQc 并重Rc 悲欢Sc  生病了Tc  所在地址Uc  积极贡献Vc  ▁pioneersWc 琥珀Xc 粉尘Yc  工作满Zc  规范的[c  如果真的\c 尽收]c  的高温^c  ▁staring_c  和全国`c  黄冈市ac  车辆通行bc  ▁Vikingcc  持续性dc ▁Ripec  ▁Rollsfc 泥石gc 闽南hc quaic ▁PCsjc challkc  点击空白处lc nilmc  同志为nc Aliceoc  criticalpc  ▁disappearanceqc 满目rc 茴香sc  中间的tc  其实都是uc  ▁acceleratingvc 夸赞wc 字典xc 版图yc 珍品zc 颜料{c  活动时|c  ▁chord}c  ▁dreamed~c  放心的c  核酸采样c  在会上c 铃薯c  能起到c  是最大的c ecutec  astingsc  ▁irrelevantc  工作重点c  月份开始c  生产工艺c  下图二维码c  ▁Sixthc  国家税务总局关于c  Extensionc 抽屉c 臃肿c  完即止c ektc  ▁Creedc  Opinionc orestc 中英c 轮番c Leaguec  ▁Accountsc 夸大c  肿瘤医院c Samplec  ▁durchc  ▁unbeatc ▁MODc  大规模的c iegelc  Perfectc  恋爱的c ▁rayc  ▁sixtyc 遮住c  ▁convec 洁面c  不仅没有c  点击链接c  纪检监察组c  ▁redemptionc 生完c  ▁Wattsc  ▁soaringc  新校区c Kayc  的男子c avidc  ▁Vincc  ▁toughestc  ▁counselorc 上师c ▁UWc ▁Yac  生命体征c Domainc  中没有c  ▁dietsc esanc  集体经济组织c  ▁distractedc  ▁Alreadyc  ▁mushroomsc 叮咬c 缜密c  日夜间到c  ▁federc  bindingc 之行c 门头c BOSSc  一直以来的c 尊享c  一段时间后c 一壶c 创设c  工作责任c  ▁Airbusc 生财c 醉了c ▁{"c  也不需要c ientoc  还有更多c  ▁boilerc  ▁cinematc ▁Vocc  专题讲座c sellerc 北约c 方方c  退休金c  发展史c  或删除c 创投c  元抢购c  柔软的c 弥勒c ascalc  ▁Workplacec 毛毛c  ▁Bellac  ▁coachedc ussyc  信息安全c  活动场所c  防疫要求c 一团c 所想c 浩瀚c  关键技术c asersc  和治疗c  重疾险c  经营范围c  静下心来c  有疑问c  ▁departedc  ▁femininec  ▁inductedc 后即c  Atlanticc 护肝c  仅限于c  ▁alleviated 保住d  经济发展的d 你用d  对你说d  本人有效d  牢牢把握d 右转d  orporatedd 崩塌 d  应该怎么 d  录取结果 d 临沧 d  ▁captivating d 工夫d 布朗d  赤峰市d  广大师生d 煸炒d Wonderd  ▁Ferryd  不希望d  ▁chambersd foliosd 上色d 断面d ▁Nitd  smanshipd 取舍d 计数d  机器学习d  ▁slamd  ▁Majority d  ▁ceilings!d рав"d  ▁prow#d 大二$d  ▁reinst%d  ▁midfield&d 不朽'd 到天(d 户的)d 还剩*d 都应+d  团员青年,d  高峰论坛-d 延边.d  ▁homepage/d  ▁documenting0d  执法检查1d baidu2d  ▁Satellite3d 拉拉4d  英雄的5d  等法律法规6d mop7d 在为8d  不利影响9d  ▁rightly:d  ropractic;d 入狱d 连麦?d  防腐剂@d ▁hawAd  怀疑自己Bd  ▁anthologyCd ▁FahDd 逝者Ed  很显然Fd  ▁adventurousGd 袖子Hd  服务队Id ducersJd  ▁tallyKd 高冷Ld  ▁BentleyMd  ▁standingsNd 沙子Od 能将Pd  这些话Qd oscopeRd 并把Sd  慢性疾病Td  ItalianUd  双循环Vd  ▁HusbandWd  ▁RotaryXd  ▁screenplayYd  ▁PermanentZd  有一番[d  熟知的\d  轻轻一]d  非常喜欢^d  ▁borrowing_d  并将其`d bynad railbd  货车司机cd  ▁Chryslerdd  又如何ed  ▁sanitationfd  人家的gd  综合所得hd  ▁bulletsid 机上jd 荆棘kd  查看了ld  ▁Sloveniamd 睫毛nd  ▁Tulsaod  ▁analysepd 迁徙qd  时尚的rd  ▁trustssd 本国td  ▁Admiralud 严选vd  周期的wd 无止xd 物语yd  Settingzd  ▁offences{d  ▁recognizable|d  ▁cite}d  ▁bounds~d 登峰d  服务能力d  ▁affiliationd  某小区d 温婉d 重工d tecd  不安全d  先后在d  ▁uintd  ▁dwellingd  如果他d  缴纳的d 微刊d  ▁nonsensed  ▁monumentald  也是为了d  Spanishd 抗炎d  清楚的d  ▁HVACd  基本上都是d  全面深化改革d  ▁daylightd  蓝轩宇d ▁MAXd 余姚d  callbackd 催眠d FKd 带状d  ▁defectd  ▁Portraitd 徒手d 溶胶d MCAd ouraged  commonsd  会通过d  ▁Levid  ▁tracedd  大峡谷d escod  如果喜欢d  ▁Gothicd  油菜花d  ▁strapd  了一道d  季节性d 吻合d 萌新d  请记住d  结构和d  费用自理d  ▁Hirod  ▁Rootsd 四子d  属马人d  ▁acrylicd 死守d BMWd ièred Skipd  进入考点d  ▁zijnd 加载d Vod  输卵管d  电子科技大学d ▁कd  满洲里d  acionesd  豆制品d  ▁Doesnd  ▁skatingd  统计学d  做出来的d actlyd 罪魁d 青州d  ▁Chald  ▁tramd  ▁Imagingd  两人在d  但是这d  邵阳市d 到晚d 轮毂d  特别提示d 名将d 奇特d 领到d  股权投资d  ▁Hyderabadd VBd 经久d 勺子d  病情稳定d alletsd  ▁vegetariand  荆州市d 年近d  论坛邓州论d  ▁stressesd  共同维护d  相关链接d  蓬勃发展d 走高d  ▁Votingd 巨型d 手镯d 虽有d  ▁Nachd 凝血d 徐徐d  出发点d  新格局d  设计院d  再看看d  如有侵犯d Fieldsd 不落d  计划和d 乌龟d lsend  进入校园d addedd  centrald 人活d 优胜e 速率e  个省份e  的事业e  虽然有e  等违法行为e  ▁numberede 夜幕e  ▁Lisbone », e 嘿嘿 e 旁人 e  岁开始 e  ▁Maurice e 使你e 提亮e 摇晃e 是新e  下去了e  录用公务员e 冰山e 天成e ▁Raze  与生活e  更没有e 租客e  用自己e ueble  专业能力e  ▁cumulativee 亮度e ▁Embe  ▁evid e  ▁supervised!e  李世民"e 就多#e 辛劳$e  ▁sewer%e  ▁Leonardo&e 氯化'e 仪征(e 直立)e  material*e  ▁routines+e  unciation,e 刘国-e 大将.e  党委政府/e 人道0e  ▁banana1e  体检和2e  调查研究3e  追究刑事责任4e 1 *应对新型冠状病毒感染肺炎疫情5e  ▁Inner6e GZ7e  实施条例8e  的农村9e  Elements:e JY;e 为生e 草根?e  置业顾问@e  ▁BrigadeAe  ▁candlesBe 网报Ce  滨州市De WATCHEe 丛书Fe adenaGe 也和He  以为然Ie  都一样Je ▁ENDKe ▁tamLe  ▁DebbieMe 车险Ne  ▁EinsteinOe  观看视频Pe  ▁BastQe sworthRe  patrickSe  ▁promptingTe 内生Ue 日久Ve 秀美We  抓住机会Xe 拍拍Ye  ▁deltaZe  到极致[e Prep\e 秋雨]e  一个星期^e 党对_e 香椿`e  某种程度上ae  的花瓣be bitsce 隐身de 给小ee  ▁Monsfe  榆林市ge  国家能源he quetteie adanje uraike  公平正义le  ▁Nassme 手心ne 而死oe alebpe 本无qe  定制化re  ▁Fishingse  ▁Chronicleste  ▁Workerue  ▁attendsve  ▁hydraulicwe VTxe  工艺品ye  两小时ze  Growing{e  ▁Eclipse|e  ▁Disorders}e 农行~e 摇滚e adice  cribinge 趁热e ковe  学校在e  泰安市e  ▁Arbore  mediatelye  了解情况e ▁Хe 他便e ▁bune  Associatee  ▁visionarye pHe 老铁e  ———e  疯狂的e  肩周炎e  ▁herebye  ▁Valenciae MXe Mohe 另据e vive  　(　e  放宽至e  老乡俱乐部e hrefe  也是因为e 客源e  ▁Dunde 应由e ziae  ▁Vicke 一盔e  尼泊尔e  ▁uphole 德怀e  银川市e  大学本科e  ▁flourishe  ▁utilizese  二十八e 几张e  ▁Asidee 台下e 松原e hanie 半球e  街小巷e  ▁proposinge  ▁Institutionse 的自e  永远不要e 鞍山e  案件中e 眼皮e мерe  四中全会e  ▁rafte  ▁Interstatee  做贡献e  全身心e  体育中心e Detaile 及各e 阵子e  走出来e  一辈子的e  的有关规定e  ﹌﹌﹌﹌﹌﹌﹌﹌e 空空e  万元人民币e Matrixe  教大家e  截止时间为e 编委e CFAe  的旅客e 荣光e  的认可e 抚摸e  对大家e  居住环境e  预约方式e  ▁Lunche  整个世界e  ▁Solee ▁(£e 石景e 进补e  年轻化e  ▁possessese 学得e 液的e 溪区e  点外卖e Irishe  ▁Duratione 苗族e  社会监督e 有独e  ranginge  动物和e Addede Lene ppoe tele  免费获取e assice  下面二维码e  ▁replacese  人的心e  ▁Silente  ▁undergonee 三线e 叶面e  中医学e  ▁relocatione 干啥e  ▁“[e Prede Testse ▁OURe  免费赠送e 深得e  经营活动e  ▁Wisef  renchedf 微友f 类和f  ▁sharksf 中餐f 求真f 三号f ▁Jif Ctrl f  从我做起 f upunct f 坐诊 f  自由贸易港 f  ▁retalf  ▁versesf  ▁compensatef SOURCEf  生活垃圾分类f 紧实f 当属f  元现金f  比赛的f 谁家f  ▁Blockchainf 之火f 求租f  正常人f  ↓　↓　f  语文上册f  ▁Courtneyf 政党f blic f  affeine!f 不比"f  ▁Insight#f 美院$f ▁Sgt%f  我们的身体&f jobs'f uala(f  千家万户)f  ademark*f  心爱的+f  长按扫码,f stadt-f  ▁Kurdish.f 成热/f erdale0f  ▁prioritize1f 少见2f  水龙头3f opian4f  战友们5f  无证驾驶6f inness7f  ▁Lamar8f  Corporate9f ▁IRA:f  期待您的;f 水路f  大学教授?f  甜言蜜语@f  ▁extrAf  本活动Bf  等众多Cf  规划编制Df  ▁IbrahimEf 不思Ff  ▁alumGf RETHf  产业扶贫If 余篇Jf 村人Kf  货到付款Lf 共读Mf  ▁NateNf 京沪Of 俏皮Pf  平阴玫瑰Qf 患难Rf 画笔Sf  ▁TopicTf 岛上Uf 箭头Vf taiWf  相对的Xf ascarYf  郭晶晶Zf  ▁courtyard[f  农用地\f  ▁Stuff]f  ▁capsule^f  ▁turbines_f 本法`f Rendaf  ▁ritualsbf JHcf  层次的df 代号ef  的每一ff  ▁sleevesgf 谁也hf nhif  遵义市jf  产业项目kf  涉嫌犯罪lf 义的mf 楼顶nf 眼眶of  他们有pf  ▁ruinedqf gartrf  ▁selfishsf 留心tf  教你如何uf 健美vf  是什么样wf ptomsxf  ▁Cullyf  Overviewzf  remember{f 贾玲|f  开设了}f  不戴口罩~f eminf  ▁idolf 养阴f  ▁Oralf  适合人群f embedf 浮现f  应聘资格f  ▁bargainingf 信徒f 宰相f  未完成f 小鹏f  人活着f 对错f 应税f  的植物f  蒙古族f  和心理f  骑自行车f 兴盛f ▁Egf  不是我f  ▁heroicf 何超f ▁dopf 宾客f  文字及f ▁Keef  微信搜索f  ▁Oslof  会得到f 磨合f  宿州市f magef albumf 打针f 春华f 混杂f 腰腿f 吃素f  ▁Sharingf  ▁attackersf  ▁Photographf 更名f 水杯f  套房源f  ▁Hoganf 莫及f Undf  中风险f  体验到f  的光芒f  看视频f  相互作用f  ▁Matsf  ▁Ownersf  四个字f 坚毅f 大礼f 选对f  最直接f  观察到f  ▁Xavierf ютf  ▁deputiesf 灾后f Rossf ▁Xinf  一年一度的f  onomousf Leanf 鲁班f olphinf  ▁extravf 猪油f  的几个f  ▁Addedf  ▁assortmentf 芝加f  如果遇到f  政策支持f mysqlf " 如有侵权请及时联系f  ▁Skypef  ▁moralityf 吕布f ▁Jubf  ▁Jacquf 光刻f 勘探f olvf  ▁Hitchf  ▁OTHERf  ▁repliesf 请先f 锅盖f 高音f  ▁Medievalf ibalf  未来可期f  负责解释f  的高发f 时说f  上线了f  ▁neuronsf 索尼f  基层组织f  ▁arrogf  ▁wannaf  ▁dilemmaf  ▁pairingf 心急f  在体内f 读到f ystonef  ▁directsf  Counterf 保送f  未外出f  ▁halftimef ,\f Lukef  相关负责人表示f 班班f  要懂得f  均来源于网络f 一意f  为提高f eriong  很少有g  推荐给g  ▁Salmong  ▁successiveg 惊醒g 柔顺g 群中g 金子g  发展中 g  并不难 g  的食品 g pipe g  依法予以 g  ▁Twinsg  ▁flushg  防控就是责任g  ▁Tunisiag  饱和脂肪酸g 棒棒g  不造谣g  实业有限公司g 见闻g WAYg  isciplinaryg 背书g  ▁annotg  ▁Squadrong 该县g  高情商g  ▁bikingg  总结经验g  agements g 公道!g  显微镜"g  ▁fibers#g 多梦$g  ▁Sched%g  小编为你&g  进行登记'g éc(g  重污染)g  ▁Norwich*g  ▁referee+g  这是一场,g  inflamm-g  ▁youths.g 受苦/g  要全面0g  ▁Buenos1g  ▁elephants2g andas3g 复赛4g 屋檐5g buf6g los7g  会说话8g  ▁Arbit9g 佳人:g CIS;g слеg  ▁hurts?g  协和医院@g  ▁dispensAg  ▁excludingBg 用情Cg  不多的Dg  就像你Eg 骨肉Fg  小记者Gg  的一方Hg  课文朗读Ig 年多Jg  所以对于Kg  焕然一新Lg  电话通知Mg  reflectNg  ▁theirsOg  种植面积Pg  ▁NicaragQg 价款Rg ▁ABSSg ▁lurTg 比肩Ug 王建Vg  不下去了Wg  商务区Xg  зоваYg  的眼神Zg ▁Pie[g  指导工作\g  ▁landlords]g  佼佼者^g  ▁horsepower_g 白醋`g 直角ag  多伦多bg  ▁geopcg  ▁Flickrdg  ▁herbaleg  如有疑问fg 乡县gg SEEhg  先天性ig 俊杰jg 持刀kg 赤壁lg  ▁printersmg  ▁mansng 内核og 选民pg  空间和qg  ▁reproducedrg 每斤sg 浓处tg  措施的ug  让男人vg  深入研究wg 诋毁xg  自强不息yg ▁WCzg eware{g  ▁Brewing|g  进一步扩大}g 仅用~g Bolg  重症患者g  ▁brewingg 只争g 首场g  有深度g arieg  虽然现在g  reasonableg 荆斩g  可以和g  妥善处理g 内需g 钱江g  苏教版g Moong komg  ▁Gallagherg  推送时g  ▁battlefieldg  ▁distributingg 剔除g 类等g ▁'<g  挺好的g  ▁Seminarg  ▁Midnightg  于是就g  公司与g  背单词g  ▁Salaryg  ▁upholstg  ▁preseasong 两市g amountg  ▁Robotg  ▁foreseeg 大路g 黄龙g  二十四节气g 持仓g 美式g  共同参与g  证明自己g cliffeg 沉醉g  最让人g  ▁leasingg netesg 古玩g 退市g 教诲g 进面g  ▁bakeg 创客g  幸运的是g  收藏价值g  Openingg  ▁Scoutsg  ▁utilizationg  新零售g  米以下g iciog 此种g ▁egg  ▁Consequentlyg  充电宝g  所以你g weiseg Eyeg  ▁Gagag  ▁pinsg 平房g 转赠g  ▁assaultedg  ▁tutorialsg  完成作业g  ▁monarchg Festg  ▁intolerg 哀乐g  十三条g  坐落在g  ▁Priestg 号令g 炸鸡g ▁STRg  ▁Moduleg  一个很g  集中开工g 补救g sedg Fightg 张桂g  ▁Purposeg  的一句g  绿化带g ongsg ▁Skeg  公共文化g  ▁precipitationg  药物的g  ▁reorgang  失败了g ▁Jawg  分支机构g  ▁Warningg 纲领g  ▁Mauritg 下岗g  在广东g  强企业g ▁dang  年左右g  ▁censorshipg  看不清g seekg Marvelg 固化g 激起g  的天空g ▁tidg  ▁marketedg KTg  ▁arterg  ▁Bookerg  ▁miniatureg  我们去g  等作用g  这样就h  ▁Managersh  ▁draftingh 万股h 嘻嘻h  ▁aluminiumh 高某h  ▁Thrillerh  ▁Millenniumh 广安 h  ▁deficiency h 汶川 h ▁JS h ichael h  协调解决h  点击下方小程序h  ▁Landingh  ▁Selectedh  ▁grindh  ▁blacksh  ellationh 读过h 把这h 殊胜h  有心人h  ▁constructingh 易发h  信息与h  资产的h  ▁Paramounth  一致的h  幼儿园的h 拉面 h  再多的!h  他们没有"h  ▁freshly#h  ▁franchises$h 总算%h fal&h  安静的'h  已开始(h  安全第一)h  ▁Likewise*h 稻谷+h Ops,h  希望你们-h  当一个人.h  ▁implants/h  ▁Beau0h  ▁kitchens1h 鼻梁2h  相关事宜3h  ▁lays4h  青年大学习5h 拔罐6h 物权7h  自己的能力8h 佛经9h  了解的:h 无怨;h 进球h 高远?h  对健康@h  无忌惮Ah  的良好氛围Bh  向全国Ch  最终以Dh 怀集Eh  等各项Fh  HoustonGh 红梅Hh  成绩的Ih 独秀Jh  要持续Kh INESSLh 议事Mh 货款Nh iobOh  意识地Ph 厂长Qh 咬伤Rh 最想Sh gifTh  ▁lieuUh  ▁courtroomVh 复方Wh ahiXh  与会人员Yh  控制系统Zh  ▁liberation[h 沿岸\h  ▁sandwiches]h  ▁renovations^h 床单_h  ▁Browse`h  ▁Maverah  添加好友bh  ▁ralliesch extradh  ▁breweryeh 蒜末fh  知名品牌gh  ▁boasthh  ▁Soldierih  解决方法jh  妇女节kh  ▁Freddielh 执念mh 选好nh 狂风oh  并发表ph  李文亮qh  元人民币rh 疏肝sh  的名义th  的负担uh  旅游资源vh 林立wh Chelxh  ▁Answersyh  ▁frighteningzh 大吃{h  ▁ruler|h  上升到}h maps~h  仅代表作者h ridgesh 应接h 永兴h 玩儿h  ▁loopsh  ▁Suzukih  建议您h  ▁Coltsh  ▁bookingsh 推翻h  过多久h  过马路h 林彪h  宣传推广h  保持社交距离h  ▁disclaimh  科技部h  ▁Berksh  ▁rotateh 崇明h  伤害的h  ▁Lennonh  自以为h  积极向上的h  ▁glitterh 海门h 繁育h  网格化管理h  ▁Baghdadh  我曾经h  有价值的h  ▁Pressureh 毒症h soundh  ▁SARSh  ▁formidableh  教孩子h 揪心h Vieth SERh Viceh  专业学位h  家庭农场h oulosh  ▁Sofiah 对小h  本产品h Intenth  ▁Judiciaryh  三位一体h Olymph  ▁steakh  umerableh 来找h  ▁Pondh  resolutionh 免票h  急需紧缺h  新的一天h  罪魁祸首h  chemicalh  ▁hormonesh 惨遭h 走实h  日起至h  不是什么h  想起来h fewh asynch 应邀h 求婚h  ▁Mozarth  ▁stirringh  ▁adaptingh  ▁interruptedh  主旋律h 并经h 藿香h  状态的h  ▁blueprinth 胸腔h  ▁ultrash cvh ▁DECh  疫苗研发h  ▁disputedh  ▁Governorsh  Facultyh  文物保护单位h  ▁downwardh 能与h  部门和h  ▁strivingh  风险管理h  ▁Suttonh  ▁stickyh 不败h  突出重点h  ▁remnh  ▁Friedmanh  ▁classmatesh  ▁ArrayListh jqh 侧翻h 助农h 熄火h  的户外h toosh 合院h 朵朵h  加大了h  主讲人h  嘉兴市h Keysh  Paperbackh  党的政治h  ▁computationalh 闪亮h  可以实现h gradedh 杯水h 爬上h  最有效h  外国语大学h  粮食安全h  这些年来h  世界大战h  择优录取h  也就不i ▁MRIi ▁komi 四平i  自尊心i eckeri  ▁circusi 军阀i lacesi  ▁Lindsey i 山泉 i  在经济 i akin i 西江 i  我是一个i )→i  沙发上i  高级工程师i  ▁Farmeri  ▁Halifaxi 一去i 优品i 双创i  土地出让i  毛细血管i 甜头i  ▁lunari  ▁stackedi 按需i  督查组i  packagesi " 身体健康和生命安全i  ▁creditors i 有致!i  故宫博物院"i 吃力#i 打火$i 真话%i  下半身&i  心肌梗死'i  ▁Wants(i  参赛者)i  有足够的*i  科研院所+i planes,i 熟了-i  透明的.i 刺痛/i  犯罪行为0i  ▁seize1i DY2i 岗前3i 已出4i 灭菌5i GIN6i ▁'@7i  轻轻地8i  肆无忌惮9i  ▁sourcing:i  ▁anonymity;i  ▁fundamentalsi 麦芽?i 万州@i 空地Ai  都具有Bi hotsCi  ▁railsDi rietEi 淡了Fi  ▁riotsGi ར་Hi  英国的Ii liftJi  也就意味着Ki 资管Li  马铃薯Mi . '不符合食品安全国家标准规定Ni 安眠Oi 鹧鸪Pi  和妻子Qi  地区和Ri MotorSi determTi 队列Ui tripVi  公益广告Wi  卡拉胶疯狂Xi " 垃圾分类卡积分绿卡Yi  亲身经历Zi  ▁Actors[i 疑虑\i  浪费了]i  ▁serm^i  ▁interpretations_i 洪涝`i 菜园ai  十年的bi 琵琶ci  同样也是di  ▁abortionsei 抢夺fi argergi  方案和hi ▁ripii  这个阶段ji Guestki zzoli ▁Kubmi  ▁moleculeni 原定oi  的资料pi  促进会qi 发往ri 永春si  ▁depotti 原浆ui 分隔vi 隐匿wi 니다xi Linkedyi 之乐zi 下称{i 德智|i 无机}i 呼万~i  披荆斩i  有发热i improi  ochemicali  ▁junctioni 儿孙i 致歉i  和视频i  ▁cansi  ▁wipei 临门i  ▁unavailablei 真伪i raisali  ▁Sanctuaryi  ▁exploitedi  孩子对i  农业农村厅i  ▁gentlemani  ▁Tiffanyi 南宫i 摇曳i  的新生i  造就了i  工作条例i  月中下旬i  ▁Instrumenti  斗争中i 点播i  违法地点i  fightingi  也会在i  出生的i paini RBi  ▁hikesi  安排专人i  ▁songwritingi 遮掩i  发现在i  释迦牟尼i Executi papersi  ▁Appalachi  清晰地i  ▁unusedi 横跨i 永乐i  生育保险i  ▁undisi 邪气i  一网打i 读本i 三阴i 熟食i 让小i 面额i  ▁diagnosei 施展i ▁cii amicsi  ▁Crimesi  培训会i  ▁speedingi  ▁submarinei 良知i 解救i SPAi EWi limiti  自己的想法i  ▁Nielseni  ▁impairmenti 天龙i 查到i 黄疸i  周杰伦i  联系到i  管理平台i existsi  ▁buddyi  ▁predictingi 退化i  resolvei  ▁hurtingi 倒下i  线上的i ibrei  ▁Giftsi  他觉得i  副驾驶i  基础性i lxi 风雪i  型企业i  ▁Whilsti  忘不了i  ▁Entrepreneurshipi 只求i  确定性i  航站楼i ▁heni  分批次i 打过i  作者和i  第二位i éhoi  严重者i  依然严峻i  Changesi 取悦i 提纲i 村医i 祈愿i  发展成为i 一区i  的美人i rosti ▁Miai  ▁squaresi 供求i 留恋i  的次数i  涉黑涉恶i  要建立i  脱贫攻坚工作i  ▁Falconsi 烟雨i izzyi  secondaryj  ▁unrelatedj ROWj  中找到j itivesj 低了j ▁alej 考勤j  ▁antibodyj  ▁Cunningham j  在小区 j  ▁Dong j  高等专科学校 j ,‘ j 平顺j 斯顿j  文章来源于网络j  这可能j  家常菜j anorj  轮融资j  whetherj  ▁litterj  过渡期j  一名男子j  开学复课j  ▁receiversj  ▁disadvantagedj 军医j  汽车行业j  containsj  养成了j  玩家的 j mort!j  ▁forbidden"j 虚无#j ▁Riv$j  人一步%j  ▁Stocks&j  ▁Lafayette'j ▁Ink(j  ▁superintendent)j 中芯*j REC+j  郑重承诺,j 城事-j  极少数.j 刘文/j  享受着0j  更美好1j 宝剑2j 性子3j  各领域4j ▁IMF5j 下部6j 会变7j  如果你喜欢8j  问候语9j  ▁Celtics:j  不少的;j arinej desk?j CZ@j  独角兽Aj ldandBj  ▁NaplesCj 不惊Dj gueEj  防疫知识Fj  ▁EnoughGj  ▁workoutsHj  不认识Ij  万事如意Jj 师姐Kj  ▁antiquLj  ▁proportionsMj  ▁subscriptionsNj LuOj  ▁burntPj  关注推荐理由Qj  ▁buttRj  版权等问题Sj  向日葵Tj  等企业Uj 吴尊Vj  的人士Wj ProxyXj lishesYj  ▁daringZj  ▁Parkway[j  ▁terminate\j  中央气象台]j  颠覆了^j crime_j  ▁printable`j QBaj 家湾bj 花甲cj 雕像dj  基金的ej 单间fj  难度大gj rologhj  第一季ij  请拨打jj  ▁declineskj 别样lj  奥巴马mj 罐头nj 无痕oj ▁Einpj 修辞qj  已累计rj  ipationsj 荡漾tj ▁ASCuj  ▁mandatedvj  ▁sketcheswj 扩充xj 拨款yj 时可zj  ▁Walton{j nz|j 细嫩}j 英德~j  主要通过j  ▁hatchj  ▁Recommendj 发稿j  开庭审理j 防备j  不正常j WXj 六国j 成活j 花青j  根据自身j  ▁Uruguj  灵活就业人员j  ▁citationj 马里j  不用说j  ▁wanderj  requencyj  ▁Renewablej  ▁launderingj 开挂j 后腿j  ▁stealsj 一跃j 如鱼j 挤出j  小学部j 我方j 后浪j 接报j ullsj  naissancej 归国j  Missionj 能见j  ▁fulfillmentj cppj  不在意j  ▁charsetj  ▁ONLYj  ▁defeatingj дыj ▁lubj  ▁Spiritualj  太大了j 应按j Negj  多长时间j  这个社会j  ▁Churchesj  他开始j  ▁trendingj 潇湘j 银币j ▁falj  ▁heartfeltj 大半j  对我说j  夫妻双方j geonsj  Principj 眉头j uspj  制造的j  黄河流域j 放水j 1 *中华人民共和国治安管理处罚法j 延后j 驾乘j ▁REDj oittej esteadj  ▁Framej  推动高质量发展j 灵宝j 胆小j 遵照j  earancej  ▁Ecologyj  检测和j  ▁linebackerj 预后j ▁Dedj  ▁Seatj 达克j  不正确j  与世界j  学雷锋j 瘦了j 鲜卑j  中关村j  仅限前j 抱抱j  熊孩子j 自营j 采暖j  大同市j  意外险j  ▁inherentlyj 九大j 很像j 能从j  科医生j  ▁hurdlesj 居留j  ▁Recognitionj insicj 鸡西j  ▁Vitaminj 王海j  以各种j  预报名j 不详j 血瘀j  充分考虑j  ▁cyclistsj  ▁Kensj 擦肩j 阿司j  形容词j  ▁saintj  多角度j ▁MUSj  ▁pinchj  ▁Marilynj  ▁Buckinghamj 兴和j  ▁trendyj  Performancek  ▁goalkeeperk 污泥k  тьсяk  ▁truthsk 来抓k  只要在k Cryptk 何方k 豆芽 k 说地 k 首诗 k cels k ▁Pav k  ▁bambook 陪我k  好老师k 机等k ismick 防伪k  的同学们k rancesk 电极k  专门为k  体制内k ▁Fujk Agentk synk  ▁pennyk  ▁Settingk 内推k  ▁Foolk  特别多 k Vill!k  ervical"k 废钢#k  的数字$k 唐僧%k 的药&k 路北'k  促发展(k  披荆斩棘)k  中华民族的*k 从古+k 托幼,k 甄选-k  最高分.k  领导力/k  工作落实0k  硕士生1k  ▁vegg2k  ▁viagra3k 休养4k 丹霞5k 高院6k  培训课程7k 广发8k  生命周期9k  ▁Bold:k  ▁decreases;k  ▁liquidityk ▁Ard?k  ▁Pratt@k  ▁WizardAk tbBk 余次Ck  发来的Dk ▁TyrEk ▁[];Fk  ▁cassGk  人民至上Hk 稳居Ik  ClinicalJk  MultipleKk  工作纪律Lk 积压Mk 脸蛋Nk  乡村医生Ok  这种做法Pk CTIONQk  合法性Rk itantSk  ▁zoningTk  ▁poppedUk  ▁MeansVk  dfundingWk  实际控制Xk  ▁misdemYk 通蛋Zk Got[k bytes\k  Required]k 甜食^k  和普通_k  培养了`k 红白ak 过万bk  ▁overflowck 动用dk  ▁Antarcticaek  第三季度fk  ▁Roomsgk  和谐的hk  中国邮政ik  ▁backwardjk NVkk  ▁Minglk  ▁tracingmk  ▁hecknk  一刀切ok  却不能pk  ▁foilqk 表姐rk ENAsk ▁ötk 爆红uk  和责任vk  就说明wk  可以利用xk  生命至上yk  西安交通大学zk HG{k letal|k  了一眼}k  在基层~k  家庭中k 仁爱k 锋芒k 张子k 滞后k  往前走k : 3和新疆生产建设兵团报告新增确诊病例k 与小k 卧床k  最新版k  这么久k ogramsk  行政主管部门k Evalk  ▁Ronaldok (()k  东南大学k  ▁chalkk  ▁pumpkink 拆分k ORMk 装有k 重用k  arendrak " 治理体系和治理能力k  ▁Cameroonk 在在k  密切配合k  ▁gradualk 印军k 夜的k 并称k 血型k 绵延k ippersk IBMk  建筑工地k  ▁sacrificesk ▁UFOk  ▁drivewayk  脑海中k 夜宵k 晨光k  ▁differsk  在当前k  施工人员k 饭吃k  ▁intervenek  子宫内膜k  ▁gymnk  ▁scrubk  ▁handingk  ▁humorousk  ▁vaccinationsk 吐痰k 要高k 选票k Oilk  以达到k  花生米k  ▁innocencek freyk  乡镇卫生院k 起因k pastk normk  ▁bureauk oniank 塘村k 知心k  增长率k  遇见你k  期待着k  有这些k ▁Hosk  一起分享k  ▁Jeromek  ▁Salmank ▁UPSk  ▁Vanessak 奶牛k женk 大度k 闯入k DOCTk  突如其来的疫情k 抗震k  总人口k  Economick 三步k  地看着k 废品k DXk 他以k 道士k  送清凉k  一审判决k ilandk  等公共场所k  ▁consultedk 主见k  外卖小哥k 可比k  等多家k  打扫卫生k  ▁possessionsk 功勋k 很想k  及家属k  反诈中心k  ▁plethorak 泡发k  ▁Experimentalk ㎡　k oingk  ▁Burgerk  ▁Posterk  的医生k  ▁сиk % 人民群众生命财产安全k 　「k ▁LSUk  动力方面k  夫妻共同k  ▁Huntingtonk 银色k 死人k happk  这款车的k  ▁Rital  ▁communistl  烈士陵园l itiousl  ▁Junctionl  消费扶贫l  ▁onwardsl  ▁Bottoml 众筹l 肉馅 l  ▁Wheeler l  归根结 l  打下了 l  民族的 l meanl  平时的l  ▁Typicallyl 秋千l  切入点l  报名及l  运行的l  安全技术l  还是需要l  ▁incapl  ▁Wolvesl  ▁beginnersl  未及时l  输入性l Samel  ▁residingl 适配l  要注重l  全面完成 l  ▁adren!l tou"l  最严重的#l  骑电动车$l 惊蛰%l  如实供述&l ун'l 攀爬(l  ▁pristine)l  ▁spirituality*l  ▁cousins+l 路东,l 酸辣-l  不妨试试.l 明晰/l  取得新0l 失利1l  住房租赁2l  ▁punished3l  ▁contention4l 猛进5l Farm6l  ▁polish7l 两颗8l 四房9l 茂盛:l rising;l 公会l  报名网站?l 育肥@l ▁SETAl juvenBl 张建Cl 走深Dl  十年前El ▁oscFl  阿司匹林Gl  这样的事情Hl  ▁CannonIl  ▁NigeriansJl  ExpressKl 追逃Ll 消散Ml  细心的Nl  ▁telescopeOl 林志Pl  directoryQl 跌至Rl  FinanceSl  eredithTl  自贸港Ul  有什么用Vl  ▁toppWl  热衷于Xl  依法刑事拘留Yl  ▁widgetZl  strument[l  ▁crowned\l  ▁vicinity]l 很美^l 皇宫_l  魅力的`l 号和al 流露bl 留校cl  上下册dl  是基于el  洗衣服fl  自主研发gl  ▁funkhl  商品住宅il togglejl  ▁bellskl 厂的ll 和胃ml  ▁shooknl 心包ol  中青年pl oltaql ▁Pirrl  ▁angelssl  icipatedtl 热泪ul  两个方面vl  乳腺增生wl  Marketingxl  手机壁纸yl 困苦zl 一节{l  ▁spoon|l  孙允珠孙允珠}l  ▁Dynamic~l CSSl ▁SIl  讲道理l bredl  全球经济l  ▁боl  而在于l  第一时间向l 压紧l 惊现l  太过于l  的朋友可以l  ▁WITHOUTl Rawl  ▁mastl  ▁aminol  ▁printfl  喜欢上l  ▁WORLDl  司司长l  他们认为l ocussl  ▁Marklel  ▁determinl  ▁imperfectl  可以享受l  ▁passagesl  有惊喜l  ▁predictsl  让观众l  为解决l  更强的l  高温天气l 打闹l  属狗人l 分拣l  流动人口l  的安全性l  年的历史l  每日更新l 畅享l Layerl  ▁influxl  ▁Printingl  ▁reflectionsl 广深l Uploadl 之和l  印花税l  一个国家l  不断加强l  等防控措施l 厚德l  ▁carcinl 买过l  工作期间l  没有时间l  ▁Builderl  attributel 遮盖l  ▁Zambial 值的l 尾灯l  衡水市l  ▁ghostsl 军旅l LEYl ▁clal  ▁specimensl 叫声l 院感l  ▁curvesl  ▁McCartneyl 哪天l 组队l 腰包l  nothingl  ▁inaugurl  ▁Greenwichl 槟榔l  时至今日l  differentl  ▁Provinciall  ▁Settlementl  ▁permittingl 蟑螂l 转盘l  ▁Plaintl 往日l  你才能l  ▁tropl 南康l  高校毕业生就业l 执教l 都为l  ▁brutalityl  ▁Georgianl  ▁purportl  ▁Everybodyl  ▁allal  吃一些l alanl  装备制造l  ▁strandedl ▁udl gagesl 肿块l Noml  给孩子们l  Specificl 圆明l 解散l  李兰娟l  也可以通过l npml edenl  ▁Polarl  ▁unreall  ibrationl 发胖l 巩义l 一往l  毕业季l 点小l  如何进行l  谋取利益l  疫情的影响l  ▁Irvinel  ▁consolidatedl  易地扶贫搬迁m 就以m  行驶的m 斗殴m 留置m 至高m  手术室m  带来的不便m  ▁hardshipm Used m  ▁unstable m 从何 m john m ▁GPU m  ▁honorarym  ▁Collaborationm 公变m 馅饼m  关注和m  ▁hearsm  四个方面m  工程管理m  如何去m  荷尔蒙m ▁CRMm  ▁Candym 紧随m 美得m  ▁sloganm 两套m 特辑m  与现代m ▁sua m  中美关系!m anity"m  medical#m  ▁circuits$m thro%m  非常容易&m  的房屋'm  ▁Anime(m  公安交管部门)m  ▁rabbit*m 剧目+m  ▁gallon,m  ▁Navigation-m 道场.m 金科/m 女郎0m ALK1m VAR2m Annual3m 吃惊4m  充分认识5m  支付方式6m  特种设备7m  inatory8m  reciation9m  我们相信:m  ▁Odyssey;m  Companiesm  ographed?m  白开水@m  科学合理Am 在县Bm 沙棘Cm  一盔一带Dm  ▁runtimeEm  ▁exportedFm 求知Gm  利用好Hm  ▁monkeyIm 可笑Jm  在社会Km  的自由Lm ultzMm  ▁RutgersNm  分手了Om  多学科Pm  PhilippQm  ▁stainedRm  这些事情Sm \'Tm tjUm 下落Vm  ▁RunnerWm  ▁contenderXm 证人Ym  和文字Zm ICES[m  点击标题\m  ▁Shap]m  ▁Jonas^m 宣读_m 肾经`m rillam 深浅bm  ▁hookscm 督脉dm 黄精em  ▁Indiesfm 下拉gm  的一致好评hm  高职院校im  ambiquejm 伊德km  答应了lm  跟别人mm  不管你是nm cigom  不明的pm  再比如qm 上部rm 预扣sm  ▁limestonetm 褪色um sidvm  一拳一拳wm  真实姓名xm 追星ym 项羽zm  没有问题{m  metadata|m 无路}m  两方面~m  ▁Millionsm  答案解析m  人均可支配收入m  ▁conceptionm からm 待到m ▁CESm  ▁gigsm 数以m 罗志m  让孩子在m  ▁Moorm  ▁Netsm  ▁bladesm 分店m PUTm  ▁Carmenm }　m  得很好m ▁ASSm  以人为本m  消费者权益m  ▁Divorcem  ▁Maharashtram ▁VSm  也会被m  我怎么m  吾悦广场m  反垄断m  生命中的m  ▁sixteenm  ▁salvationm Hanm  出发前m  新闻发言人m  检验机构为m  ▁Startedm 反光m ernom 尽的m  上加霜m  等同于m 短文m 德语m  生产日期m  全面提高m  热点问题m  zaixianm  ▁Latviam  产学研m  重大战略m  也不要忘记m  ▁Tysonm 学说m  ▁Bretm  ▁Renom  ▁Pleasantm  ▁sparklingm  ▁powerhousem 古籍m  ▁ethicm 天佑m  车型的m PCBm  千呼万m  长治市m 三只m 看下m  杨某某m  如何应对m  ▁gazem  ▁vastlym  ▁bolsterm 后要m  校园里m  螺蛳粉m ottom 阻隔m arfm  ходиm  套人民币m  ▁Outreachm  ▁figurm  ▁tempom ;"m 圆柱m ▁Ginm  ▁Furym  ▁contagm  ▁Rabbitm  ▁Madonnam 桃园m 的超m  方便快捷m  欢迎留言评论m 在全m  就是用m  不会轻易m  中国第一m  能力有限m 等国m TIMEm gwm  无新增死亡病例m  ▁resemblesm 火线m  雪上加霜m  ▁worsm  诊断为确诊病例m 外事m  ▁Robertom  ▁competitivenessm 嘴边m  乡村教师m  ▁lamentm  ▁Directivem Nim 曾获m seedm  的优质m  ▁Liesm  ▁whereinm 恩典m  一级学科m ichtem MFm  彭德怀m  安阳市m  交通管理局m  也经常n  招生政策n  群众举报n  ▁establishesn 配伍n Behn  降到最低n 正职n  上进心n  ▁Battalion n  ▁crackdown n DK n 书上 n 冲浪 n  日零时起n  疫情严重n  很多东西n  通过此次n 也正n 知音n 闪付n  日当天n  特色小镇n  归根结底n Runnern  ▁offendern 免职n 实木n  水浒传n assinn  ▁Warmn 患处n 沉闷 n  不健康!n  ▁Sultan"n  亿美元的#n  ▁Benson$n  ▁Pokemon%n  ▁Honduras&n 武侠'n ▁Yo(n  秩序的)n  专项检查*n 传入+n  自驾车,n  ▁Oaks-n 给到.n  可爱的人/n  ▁Bosnia0n  ▁wastewater1n 量刑2n  网上缴费3n illet4n  ▁Silk5n  ▁Bates6n oxide7n olulu8n  ▁Deutsche9n  也并不:n  exception;n 成真n  变异株?n  感到很@n  千呼万唤An  用于治疗Bn  文化和旅游局Cn  ▁troublingDn SIMEn  家具家电Fn  ▁ProtestantGn 呼噜Hn 养蜂In  市场竞争Jn  ▁AuthoritiesKn  要充分发挥Ln 阻击Mn  ▁GambNn  文化和旅游部On 雪地Pn  ▁StartupQn 王妃Rn  ▁assertEqualsSn MASTn GhostUn  被人们Vn BOOKWn TermsXn  ▁siliconYn 及格Zn  有影响[n  什么地方\n  一边是]n 言辞^n  丈夫的_n  总结出`n  其他地方an  综合编辑bn SZcn  ▁Travdn  去学习en  和交流fn  ▁predatorsgn rexhn  很多孩子in 同桌jn 这双kn misln cerptmn  ▁commentingnn 出力on 经由pn  随迁子女qn 大通rn 先机sn 寿县tn 质押un ▁hpvn  ▁archivalwn 降落xn otleyn  ▁rabbzn  ▁Whenever{n 几名|n  ▁clashes}n mk~n 连着n benchn  广播电视总台n 尝尝n 流涕n ixingn 再造n  不得在n Trendn  ▁alloyn 回族n  ▁caten 劳逸n 外向n  有好处n  ▁overdosen  涌现出n  ▁personan 费劲n  不动产权n  销售渠道n  ▁Pursn  ▁sindn 伤病n 洋气n 转学n USERn  ▁theatresn 明德n  反映问题n  才是最好的n  ▁Rocksn 笔法n ▁djn  保健食品n  ▁Mesan feitn  ▁Kellern  审核丨n  依法处理n  便捷的n  ▁Editorsn 回迁n 弄得n 本网n alymn  全文如下n  流感疫苗n guiden  用起来n  ▁fourteenn 春来n  安置点n ▁foin  ▁Illustratedn  工作压力n odiacn  权威的n  直接进入n rokeen  studentsn 极低n  行政区划n 群山n 省下n  婚姻家庭n  生态环保n  ▁scannern 李建n  一米线n  把时间n  ▁owingn  ▁Hospitalityn 做错n  丝毫没有n  ▁Shelbyn  ▁accidentaln  常常会n  ▁Messin  县领导n 混动n Tedn  办理中n  ▁Jedin  ▁hiresn 润燥n 谎报n  的高峰n  深入浅出n  ▁subscribern  ▁inheritancen  ▁justificationn 每组n  computern 后跟n  二十二n ▁Vodn  ▁Humphn  instagramn 切块n  联想到n  多个国家n  ▁Lauderdalen беn  的好评n 六是n  未来一周n  ▁nausen  Creativen tkn  ▁demonsn  ▁Judaismn 过人n  可以得到n 张云n 成群n ▁czn  给了我们n  ▁potsn  不放弃n ▁Easn  剧烈运动n  ▁Sauln 积聚n  特效药n  上课时间n  ▁stereon  ▁Driversn 足浴n  第三单元o yneco 中带o  ▁Poleo  ▁expenditureso  红红火o  森林草原o  决战决胜脱贫攻坚o Hito das o  ▁capita o Sav o  关晓彤 o  ▁pouring o  生物安全o 能做o 重构o zoro  只争朝夕o  ▁repetitiveo 炫酷o  年度考核o 器和o  给家人o  党员领导干部o  ▁advertiseo Spaino  ▁beeto  Bindingo  ▁Dioceseo  江宁区o  菜篮子o  这类人 o  报名入口!o  ▁nicht"o  ▁enlisted#o  ▁peacefully$o Kh%o poll&o  ▁afterward'o 勤恳(o  ▁bassist)o  ▁Dolphins*o 唱片+o  点赞哦,o  那一天-o 玫瑾.o ▁df/o  大家都是0o umann1o  核武器2o 排头3o  了解本地4o 赚得5o  疫情防控情况6o  ▁volunteered7o 更早8o 李亚9o  参保人:o ▁APC;o  就变成了o  ▁Guns?o  ▁stew@o  冷空气影响Ao  ▁warriorsBo 残渣Co  投资建议Do  也是一样Eo nesiumFo 王永Go  我愿意Ho  健康知识Io  通过学习Jo SaudiKo  ▁TailLo  吉祥物Mo  全面屏No  就读于Oo  的图片Po  CertainQo  ▁righteousRo 士顿So 边框To ologieUo  ▁polesVo 狙击Wo ▁XIXo 同盟Yo ▁DNSZo 事主[o 金榜\o 首期]o  人与自然^o  基本农田_o  Leading`o 文在ao  Designedbo 分型co 大城do 轻生eo Parsefo Hydgo  后期的ho 分到io 逐项jo  局限性ko 直肠lo  ▁foldersmo 润肠no  生活品质oo  ▁Berepo  ▁spectacleqo 宝库ro 朝代so  ▁cartoonsto 一问uo 刘志vo 放疗wo 连贯xo  市中级人民法院yo 公募zo  补助标准{o 题的|o Teen}o  欢迎投稿~o 月到o  ▁creamyo  ▁oxideo 小刚o 飘窗o  回答说o  强度的o  ▁BOOKo 不谈o 国潮o 法拉o Kenno  春运期间o  ▁Leaho enableo  修订的o  收购价o  还有许多o 点睛o 结了o  商业综合体o  inatorso  ▁subjectiveo 不只o  被列为o  ▁Reginao  ▁wholeso 明灯o  初中语文o 媛媛o 欣然o  都希望o  Ultimatelyo 风行o  时有发生o  重要的事情o  ▁Snapchato  ▁beginnero owlero  ▁locomo  purposeo 吃个o 手头o ooterso  ▁postureo 在第o  被授予o  政治责任o  ▁Phantomo  支持的o 天时o  ▁brickso ▁Áo ibuo  只是想o  收银员o  NotNullo 吓人o 所称o  不断优化o  ▁MySQLo  ▁boycotto  ▁collapso  得天独厚的o  新冠肺炎的o  ▁Revereo  Academico  同时又o  建立的o 护眼o 直行o  发布关于o  ▁eccentrico  ▁harbouro  行驶证o  ▁dysto 螺栓o  得不偿失o  ▁Assado  ▁kindergarteno 春意o  紧身裤o  存在安全隐患o  ▁philosophero 投注o  医院感染o  普通发票o  ▁Healingo  ▁experimentingo 鱼头o  Keywordso 遂宁o  有一股o 大瑶o 见影o  上小学o  会议召开o  永远都是o  ▁Springero  ▁Presidentso  ▁strato  ▁assemblo 段落o  是每个o uzzleo 小楼o 张口o  一定能够o  ▁redefo ▁WASo  ▁schoolingo сыo  不要被o  说一下o  等不同o 纯真o  ▁shrimpo  消防设施o  Receiveo  ▁contempto  ▁crashingo 日游o 进水o  政府机关o  ▁Excellento  一天天o teeo  可以对o  广告牌o Suitep 界线p 愈加p ▁XVp  能成为p  ▁spurp  ▁spellsp  中国建筑p  ▁Fantasticp  管理经验 p  科技的 p  需要用 p  居家健康监测 p  ▁Kaiser p 始祖p 福星p 草稿p  ▁framingp 内镜p  锦上添p 淡斑p 去到p  ▁soilsp  ▁empiricalp  ▁Publicationp 逐级p BURp Dogp  调味料p  ▁boostedp  为人知p  ▁Zurichp 之苦 p  这么简单!p  的美食"p  ▁posing#p  ▁Illegal$p 厨娘%p 　>>&p  属蛇人'p ▁pH(p 部首)p  家长在*p  的各类+p quad,p  公司简介-p  Archives.p VOC/p ▁han0p PRESS1p  置业有限公司2p  ▁coated3p 悬念4p 芬兰5p  前几年6p bash7p coq8p  ▁Orth9p  ructions:p  ▁Betting;p  ▁Hancock

    P 藤县?P  ▁Aircraft@P  ▁PantherAP  荣誉证书BP 心静CP  reditedDP 绿道EP  必然会FP  珠海市GP  公平公正HP 黄码IP ▁RiJP  ▁distributorsKP  ▁GuarantLP IKMP  ▁SpecificallyNP behOP 幼小PP 救出QP  市场经济RP  个岗位SP  不增不减TP  ▁withstandUP  ▁antibodiesVP  买东西WP  国际社会XP 会谈YP  ▁FresZP  ▁Streaming[P 机缘\P ▁BBQ]P  也得到了^P  为大家带来_P  ▁skirt`P 三地aP  初中部bP 费和cP ▁/*dP  之后就eP  ▁StarkfP 奇石gP 蛋鸡hP  国际贸易iP 都快jP  ▁multifkP  ▁HartfordlP  ▁LogisticsmP  学生学习nP atomicoP 强悍pP 的鱼qP  atoriumrP  脆弱的sP  可以申请tP  ▁VolvouP 伴着vP  ▁NationalswP ▁FXxP 金地yP izzardzP 心心{P 梯田|P 知行}P 缠身~P  让身体P  ▁organizerP TESTP обP 颇具P  阿联酋P  一期工程P  基金经理P 暂定P  倍以上P  辅导班P  ▁разP  发布信息P  ▁NightsP  ▁reboundP  ▁reflectiveP  ▁FederP  ▁rebuildingP  ▁breezeP otiP  同意后P  将这些P  第二单元P  也不必P  ▁nineteenthP  ▁SheikhP  ▁enforcedP  如何选择P  ▁conservP  ▁rememberingP  TogetherP mightP  有一套P  安全检查P  指导中心P  ▁policymakersP 焊工P  ▁MirandaP ▁BubP ▁FryP avianP  ▁anthemP 拔尖P  具体安排P 互认P  投案自首P  ▁slippedP 略微P 进的P  日报记者P ikersP  ▁diaryP  ▁misunderP 先从P  不平凡P  丰厚的P  ▁DebtP  ▁matchupP 脚气P  ▁AdvocacyP  ▁dependenciesP 乐曲P 引诱P ▁MMP  营造良好的P 为名P binsP  ▁PicksP ▁UntP  特色产业P 立新P 南安P  ▁ReinP  试用期一年P 源性P  feedingP  ▁inclinedP 临终P 化成P  梦想的P  生命健康P  ▁narcP  ▁bishopP VeP  唯一一个P  是不可能P  ▁BalkP uloP  记录了P  ▁BharP 任用P 工科P 现今P 钢板P  ▁triggersP 爱不P  是很难P  辅导老师P  ▁critiqueP 怪物P 细纹P  预防性P  ConstructionP undaP 外壳P  ▁friendshipsP  告诉自己P  ▁enduredP  ManchesterP  ▁cathedralP  有针对性的P  ▁ThomsonP 痘印P  宽松的P  提供免费P 至尊P 鸟儿P  ▁rotatingP CuP  听到了P  ▁sentencingP 喜神P  ▁condoP  请作者P  ▁stuffedP 万丈P inxP OhioP  原生家庭P  统计数据Q  ▁MickeyQ 东兴Q 平湖Q  ▁backlashQ цаQ  ▁WarriorQ  ▁foreachQ  ▁hepatQ ulas Q  Friends Q 语种 Q  一次的 Q _; Q 一早Q 皇冠Q  涉案金额Q  都不用Q  也是比较Q  ▁hereinQ mistQ  历史上的Q 儿歌Q 卫浴Q 海岛Q  一瞬间Q  ▁accolQ 装甲Q  考上了Q  ▁refrigeratorQ 不凡Q RandomQ  ▁blended Q 一连!Q aci"Q  属牛人#Q  持续时间$Q  红领巾%Q  而没有&Q umbo'Q  ▁timer(Q  消防救援大队)Q  ▁commenced*Q  ▁treasures+Q  constitutional,Q 大葱-Q ▁"\.Q  ▁hover/Q  ▁Lighting0Q 太长1Q 静观2Q  ▁bacterial3Q  ▁casualties4Q  综合楼5Q  比其他6Q  日常的7Q ▁Gad8Q 用处9Q  ▁spiral:Q  历史学;Q  热爱生活Q  无数个?Q  ▁foreigners@Q MindAQ  团成员BQ 里最CQ  otheredDQ ▁};EQ  可以作为FQ  headersGQ  committeeHQ 等了IQ aaaaJQ  了什么KQ  中央财政LQ BFMQ  不是问题NQ adoopOQ  ▁loweringPQ PVQQ 她又RQ  我们都会SQ  CommercialTQ 刘诗UQ  不知不觉中VQ  ▁soundingWQ  ▁intervalsXQ 全班YQ  正规的ZQ  ▁Platinum[Q  ▁spins\Q 跨年]Q  你我他^Q кра_Q  乌兰察布市`Q unchedaQ  我们将及时更正bQ  ▁PioneercQ 行者dQ  ▁MitteQ casesfQ  ▁sumsgQ  ▁лиhQ  统战部iQ  ▁gluejQ  农家乐kQ rictedlQ  照见五蕴皆空mQ  ▁ObservernQ 觉察oQ inkipQ  一定要有qQ  和人民rQ  对的人sQ ▁COLtQ  也不过是uQ  ▁PhilharmvQ  ▁discountedwQ  ▁privilegesxQ  ▁ParenthoodyQ 祖师zQ  ▁sewing{Q  ▁stigma|Q  ▁genetics}Q tier~Q  ▁HousesQ  ▁SaskatchewanQ 爵士Q  的三个Q utanQ 何某Q 短路Q 校服Q 球鞋Q 碧水Q ▁AUQ ******Q  坚决防止Q  所大学Q  表达自己的Q  EnabledQ 边形Q 金鸡Q ▁SATQ  的年龄Q  ▁demolitionQ 农信Q flowsQ 世茂Q 孙俪Q  是十分Q ▁egoQ  中国企业Q 祥云Q  ▁amidstQ  ▁summonQ 杨家Q  ▁stampsQ 摇摆Q  的痛苦Q  历年真题Q  ▁BedfordQ  ▁IntellectualQ 充血Q  被认为是Q  ▁widowQ  权利人Q WHEREQ ▁IXQ  人大常委会主任Q  ▁donatingQ 爱是Q  遵守交通Q livedQ 化县Q 育种Q 交口Q 留影Q  将会有Q  新闻中心Q  被列入Q  ▁EqualQ  orningsQ simpleQ 中度Q 星火Q 次年Q ▁VilQ ☟　Q  搭配上Q  就能看到Q touchQ  如下图所示Q opathyQ  MappingQ  MetadataQ  ▁divorcedQ  ▁encompassesQ 她与Q SophQ  主要还是Q  的女子Q  共享单车Q  万亿美元Q  最终还是Q  keepersQ  ▁nurseryQ  ▁RevolutionaryQ  仍处于Q emakerQ  高新技术产业Q  ▁footstepsQ 把人Q 分部Q 钞票Q ▁),Q 生菜Q waldQ  是一个非常Q  ▁unanimouslyQ  整体运势Q  ▁contaminatedQ  ▁convenientlyQ  不良影响Q  我们国家Q  ▁biddingQ 为零Q 综述Q  ▁ChesQ  ▁clearerQ 山里Q 贵重Q  黑恶势力Q  ComputerQ  ▁decentralQ 反向Q  以实际Q  异常情况Q  有不同的Q  ▁NarrQ  ▁blendsQ  ▁EstoniaQ  ▁bowlingQ 亲肤Q ▁EAQ NNNNQ  你可能会Q  群众反映Q  补贴标准Q 子怡Q  赋予了Q  ▁SaturnQ  ▁premiumsQ ▁rimQ  ▁LectureR  ▁diligenceR 例句R 地支R GuideR  这个时代R  ▁DatesR 骨灰R TSDR  若涉及版权问题 R  ▁Smithsonian R 烂漫 R  世界杯 R  有更多的 R  ▁DoyleR 退换R  ▁SecretsR 无有R  ▁jeopardR 昌吉R 随笔R  一日三餐R 涅槃R 认罚R  大学城R  ▁tranquR 定性R essingR 丹丹R 降糖R 华晨R  ▁grapeR  arashtra R ZX!R  是很多"R  研究了#R 定理$R 著称%R  ▁slope&R undo'R  ▁Deals(R  具体如下)R 村组*R  围绕着+R  类产品,R  请留言-R 劣质.R  宣传和/R  职业发展0R ▁Cly1R  却因为2R  本科院校3R 挂靠4R ▁Hag5R little6R  ▁Petroleum7R 更易8R  厉害的9R 相符:R  创新高;R  本来就R  自然资源部?R 之计@R EdgeAR 洛克BR 紫薇CR  ventureDR  ▁differentialER 例题FR  走上了GR 枝叶HR 运化IR  ▁evolutionaryJR ijaKR  研发中心LR  ▁packagedMR " 所包含内容的准确性NR 等都OR 五天PR  新闻媒体QR ( !坚决打赢疫情防控阻击战RR 昨夜SR NFLTR  ▁CreatorUR  一贯制VR  ▁CrosWR  活动在XR  ▁socksYR  失业人员ZR  曾经在[R ilyn\R  rescent]R ▁DEL^R ecting_R  ▁finalist`R 弹药aR  ▁másbR  ▁grimcR 保佑dR 头镇eR undingfR  中获得gR  不使用hR  创新驱动iR  最高可jR  billionkR  employmentlR  ▁ManormR  管理委员会nR  ▁gearedoR  ▁FellowspR 小题qR 气短rR  业务办理sR 空前tR 要给uR  网格化vR  ▁accomplishmentwR 硕博xR  的传播yR 享乐zR  ▁Bali{R  ▁Juda|R  ▁academia}R  素材来源于网络~R ShellR  ▁MotoR  ▁SlamR  为主线R  在那个R  ▁redesignR 山中R 版面R ▁FiR  召开新闻发布会R  ▁BradfordR 安葬R  ▁AdultsR 永葆R  老照片R  ▁DixonR NLR 忙于R  部分图片R 喷施R  事实证明R bnbR  原出处所有R ______R  ▁AdmissionR 正大R 试听R ▁AJR ;}R  可能就R  勇往直前R  芥子园R rackR  有效证件R ActivR asmineR ▁VsR  部电影R  ▁mythologyR 苍穹R  善良的人R  高级职称R  综合症R  ▁TalkingR  ▁comprehensR  补助资金R ▁PACR 融融R 账款R ScoreR  MilitaryR  最低工资标准R  ▁meltingR  ▁leveragingR 三道R 两轮R  的温暖R  ▁controllersR 放任R 江口R 违者R  ▁aerospaceR 篆刻R 有更R 阳镇R ythmR ressoR  一年内R horseR  ▁LambertR ▁RigR  优秀学生R  contribR  ▁communalR 李兰R  提供有力R 三要R 堕落R 排位R 金寨R  代表队R  每天早晨R  能够得到R uffedR  亲密关系R 负载R  我们可以看到R  ▁SchwartzR  ▁illusionR 上头R  ▁веR icasR  ▁GoddR 人像R moilR  ▁destinedR 数目R  大学英语R  此时此刻R  ▁CarnivalR 太仓R 寻觅R 毋庸R  ▁luggageR  ▁boostingR  历史新高R  ▁realizationR 地和R 怀揣R 对有R  itchingR 建有R  抓实抓细R  ▁BudapestR  这一刻R  会有一些R 我爸R 自闭R  管理水平R examR  发布最新R 出一R 不还R billR  人人有责R  作为当前R 复出R  ▁NewarkR  ▁heroinR PyS  百公里S volumeS  ▁ScientistS  核酸阳性S 三无S  哺乳期S  次列车S 轩宇S  特长生 S 对准 S 达峰 S  大专学历 S  ▁headache S  ▁emergenciesS 结交S 的和S 趁早S  牵动着S  联系地址S 追加S TouchS agaraS  ▁PistS 炒香S ROUPS  教学工作S 燕郊S ubuntuS  ▁MartyS 中卫S 光鲜S 大堂 S Fly!S 你这"S 取向#S  安装了$S 妃识%S 深邃&S  将获得'S  路妃识(S  强制执行)S  ▁twisted*S  ▁explanations+S ▁Gos,S  ▁Plains-S chy.S  昨天的/S  ▁bend0S ،　1S  完毕后2S  基本要求3S  ▁friction4S ④　5S 假货6S sburg7S  ▁brit8S  选择一个9S  ▁youngsters:S 夜色;S 涉税S  ▁ethnicity?S rans@S ▁DilAS 接管BS  ▁GDPRCS RickDS  ▁MicheES  ▁JulietFS 排尿GS  最后一公里HS 你别IS  都开始JS  文化生活KS  necessaryLS  写了一MS 灵性NS  那句话OS quelPS radeQS  ▁deadlinesRS  富二代SS ▁CDsTS 但愿US 发抖VS  从哪里WS  ▁veinXS  副中心YS  合格人员ZS  做好疫情防控[S  ▁forensic\S  农副产品]S  海关总署^S rophe_S 俩人`S 意念aS ▁IbbS  微波炉cS 丰台dS  的人力eS 财商fS 副职gS  财产损失hS 应予iS  的步伐jS 暴饮kS 阴暗lS CredmS ExtranS  ▁PenceoS  的案例pS  为什么说qS 羲之rS  SourcessS  氏宗亲tS  ▁auntuS 典故vS enhwS  ▁owedxS 再添yS 明朗zS 爆破{S  老字号|S 巴斯}S ров~S  未通过S  ▁FargoS  ▁ubiquS  ▁imminentS  ▁specsS  ▁IntegerS  课题组S phaltS ingingS 店家S  ▁tougherS 照着S ▁NMS  ▁GodsS  安全防范S  的事故S  ▁RegimentS 兜底S 稳住S  了一天S  ▁poeticS  中国电信S 本分S  ▁BronzeS  ▁WorldsS  ▁occupationalS  FootballS  ▁constituentsS  是一只S  ▁SangS  融为一体S  ▁muralS 打脸S  不可怕S  这是在S  体温监测S  广大考生S  精神病S  还有其他S  ▁aestheticsS 柴米S  BreakingS 实打S 长势S  ▁prostateS  ▁ScholarsS  各个环节S  的细节S  ▁PromotionS 小偷S  人数的S  感染新冠病毒S  ▁digitsS  ▁ParticipantsS  救援人员S  检验检测S  ▁stagingS  ▁consolesS 编外S  我也是S  ▁hectaresS 中线S  批示精神S  教育机构S  ▁DancingS ZAS 九个S 甘露S IgnS  作为中国S SayS guidedS  ▁WHERES 曲张S  应聘岗位S 专一S ▁HedS  ▁susceptibleS hkS 分钱S  保护和S  沟通协调S  ▁ClubsS 多米S 鼠疫S 封装S 来势S 蛋壳S  招聘企业S 候车S  顺风顺水S  ▁RedsS 各路S ilingsS 都好S  教研室S  晋升空间S CommitS  fighterS 珊瑚S  ▁simplifyS jiaoS ▁NumS equivS ComparS 今夜S  ▁RatesS  请立即通知我们S 肉体S 背痛S 纵容S  也成了S 贴在S  ▁NolanS 互换S 李明S  星巴克S ▁RNAS  ▁digitallyS  ▁devastatedS 触目S  ▁finedS  ▁FitzgeraldS 不厌S  繁花似锦S  ▁MormS  ▁lampsS  ▁untoS  ▁baconS  ▁warriorT  ▁ResortsT  ▁DiscountT 一策T 周的T 专柜T  ▁trayT  ▁complementaryT 生在T 龙王 T  他们就 T  ▁commercials T  比利时 T  ▁atoms T 弱者T  ▁peekT 光源T 吉凶T 庄稼T  政治上T  应用场景T 传世T  于一身T  ▁MessengerT 平潭T  percentT reposT  ▁SupportingT 一读T  文化节T  studentT 本省T 蜜语 T 行之!T  ▁Lucky"T 起重#T 集约$T ▁fi%T  ▁Boyd&T KB'T  收到的(T  ▁lodge)T 奇瑞*T 理会+T 静坐,T  ▁exposing-T  ▁Femin.T 再者/T 诗句0T  经验丰富1T  ▁Sorry2T 貌似3T Cle4T  中提到5T  受过刑事处罚6T riott7T  ▁godd8T  ▁Inclusion9T  认证的:T 早于;T  有效衔接T  宣传力度?T  来看看你的@T 五指AT ▁gitBT  寒冷的CT  ▁glucoseDT  ▁JeepET  ▁MaskFT  烟台市GT  发生意外HT  ▁devotionIT  ▁paradigmJT 高腰KT  无法正常LT 脱下MT родNT  为首的OT  ▁IdeaPT  ▁ColonialQT  ▁SamanthaRT  ▁fraudulentST  招标文件TT EightUT 南国VT 我问WT 迫于XT entaYT  交叉路口ZT 八点[T  ▁wildly\T  可实现]T  参赛作品^T 新农_T  运城市`T astonaT  ▁fibrebT 不输cT 或与dT 计生eT 里边fT  ▁VectorgT  ▁elevatorhT  ▁PotentialiT  各县区jT  已经进入kT  IndustrylT  ▁NinamT aannT  ▁DeptoT  无新增确诊病例pT  ▁SaddqT  故事的rT  ▁mentioningsT MovingtT  ▁TexansuT  ▁asbestosvT 行道wT 超人xT 鼓掌yT  资产配置zT  小时前{T  安全问题|T  中国青年报}T  ▁Residents~T 珍宝T  ▁pneumT  ▁impactingT  ▁kingsT  优雅的T  爱人的T ▁MEDT  ▁LoopT  上来说T fuelT  抗体检测T 睿智T 馈赠T  发生了什么T  ▁SuffolkT 凹凸T 山谷T  ▁USDAT 　②T 晚间T  着力打造T 乖巧T 不喝T 益寿T 蜿蜒T  CoronavirusT 绝非T  在经过T 万头T 沦陷T  被征收T seinT 采矿T  ▁senderT  ▁taxationT 可向T  不是在T  微课视频T  urrectionT 作坊T 泊尔T  咳嗽等T  接种后T 王维T  ▁DiabetesT  ▁postseasonT  第三期T rankedT  ElectricT  近三年T stantT  更让人T  ▁transferringT ▁UmT  不久的T  为祖国T  交流会T  短时间T ▁^T  ▁spreadsT 苏区T tomT 苛刻T 繁杂T  解放路T  ▁PaintingT  ▁respectfulT  不就是T 亮了T  群体的T iolaT  ▁CatsT  省部级T ▁gumT  ▁UncleT 理化T  ▁shoutT  ▁InspiredT  这个是T ibabaT  ▁AlliedT  ▁paintsT  来判断T ▁TVsT 耐磨T  白居易T  ▁GomezT  featureT artonT remlinT  ▁GuestsT  ▁constructiveT ▁})T  ▁ScandinT  ▁conductsT 天人T  ▁depressedT  早期的T utscT  每个人都是T  ▁LaserT  ▁disgrT 想找T  指明了T SnowT  ▁mournT  病原体T  目标和T  领导班子成员T  ▁subsidiariesT 校级T ▁MACT  databaseT 落魄T 钦州T 逆势T ▁SulT 张雨T 求财T 墓地T 治好T CANT ativT  不扎堆T  ▁DidnT  交易日T  讲政治T  ethovenT  ▁insightfulT  ▁hospitalizedU 多病U 排量U MariaU  ▁arisesU  ▁schemaU  的年代U  本科及以上U  覆盖面U tube U  ▁burns U 海珠 U  他们用 U  想象中 U 持卡U нихU  ▁rentingU  ▁specializeU 只靠U  让你在U 俗人U 双面U ApartU  上一层U  formerlyU  ▁ClaytonU  详细介绍U issonU 雨季U  ▁villaU 茶水U erennU  ▁crus U  的事项!U  非常感谢"U  ▁releg#U  bedroom$U  teacher%U 先去&U  ▁под'U trade(U  是什么意思)U [$*U ▁Pig+U eight,U  ▁Vinyl-U bags.U  的比例确定/U  ▁utmost0U  ▁elegance1U  包括但不限于2U  ▁documentaries3U 外加4U  女生头像5U  ▁sympathy6U  该有的7U  全面依法8U  生活环境9U  ▁Units:U  让很多;U 赵国U  ounselling?U 铺垫@U  而得名AU  有一段BU LogoCU 逼近DU  中共中央总书记EU 裸露FU  关注雄安GU 神通HU ▁ainIU  受到党内JU  ▁WillisKU  ▁pillarsLU 国情MU 连带NU  汽车销售OU  ▁relievePU 堂课QU  年轻干部RU  ▁detainSU 人海TU 保额UU 小号VU 核发WU  值班值守XU 所处YU  ▁RomansZU 很累[U ▁TOP\U  全力支持]U  ▁Sadly^U 鄂州_U  信阳市`U 李大aU  ▁exceedsbU :#cU 桂平dU 我愿eU 联的fU 春色gU 柜子hU  和谐稳定iU  ▁quintjU  房屋的kU  拟聘用人员lU  ▁narratormU  需提前nU  ▁RealtyoU  天生的pU rosionqU  ▁miserrU 所思sU 趁机tU  按照要求uU  和社会发展vU 捡漏wU YOxU 隶属yU 相守zU  法律援助{U lis|U  都只是}U  都是以~U flatU  ▁GmbHU 浇灌U  售后服务U  满分为U  ▁irrigationU  ▁unemployedU DubU  网友爆料U JudgeU  巴彦淖尔U  ▁TurtU  ▁FrontierU 下药U  岁女孩U  主动权U  印度的U  ▁kindergU 触犯U  即将到来的U téU 献上U 高粱U pkU  PopularU  holdingU  ▁incomesU 以备U 营救U ▁REPU  ▁PaulaU IUU 疼爱U lyssU  高速交警U  ▁PlainU 了才U  ▁reopeningU  ▁ProgrammingU  保持距离U igatedU  ▁grantingU 悬疑U  第二代U  等资料U 万台U 安东U 德行U 沙河U 于都U 眨眼U 硬仗U ▁SplU  不过在U  离婚后U estineU  咨询有限公司U  PictureU  明天的U  ▁coronU 看成U  提供坚强U 尽检U obicU  ▁adjustingU  的指导意见U 不挑U 芋头U  持续的U  可打印U  ▁holdingsU 当兵U podU esqueU  ▁supervU  ▁hardwoodU 清廉U  ▁BackgroundU  检尽检U  科研成果U  ▁HermanU  ▁ZuckerU  推送规则U  生日快乐U  ▁agreeingU 力强U 水下U  严重违反U  ▁incurredU  ▁diplomacyU  吃多了U  转为确诊病例U  ▁QuantumU  ▁underestU  墨尔本U  百分百U  别样的U  解题技巧U 南极U 卡罗U 当着U  很可能是U SolarU ussieU  暂时的U  的可以U 大屏U  各省市U helmU  ▁GrabU 楼的U  不能用U  approvedU  ▁BahamasU 江城U  ▁FrequU 科兴U  MichiganU  农民工工资U  ▁dividendsU 赶走U uphU 出警U entureU  ▁triangleU 领证U  夫妻关系U ▁CBCU  住宅用地U  知识分子V  ▁archeV  ▁minorsV JuV 还清V  ▁AnythingV AnalyV 矿物V  的特征V  Donnell V 净值 V 曲江 V 波及 V  有信心 V  突破了V  什么问题V 做为V 克里V 无闻V  证件照V tcV  ▁congestionV izonsV  ▁secretlyV 买来V 滑梯V 颤抖V oblV  ▁mandatesV 难事V 叉车V NFV 黄的 V  线上学习!V ixun"V  教师资格认定#V acs$V duty%V  进一步优化&V 浩然'V 谁会(V  ство)V  ▁navigating*V 难行+V Church,V bara-V 公公.V 奉贤/V 是其0V  石墨烯1V 峥嵘2V 野战3V soon4V  ▁Dawson5V  ▁deepest6V desc7V  工业和信息化部8V  governmental9V 字迹:V 虚实;V  确保了V  ▁Maker?V  与服务@V  ▁massacreAV 站前BV MagicCV 老赖DV 落泪EV BLICFV  先进事迹GV  ▁draggedHV 配建IV 划线JV  但目前KV  ▁TinaLV 荆门MV 可言NV 挪威OV  ▁PartiesPV 堵车QV TopicRV  市场中SV  控制住TV  的目光UV 快充VV  第一眼WV 消瘦XV  的广告YV  指导教师ZV  ▁modelling[V  ▁compiler\V 伴奏]V  做人的^V  通行码_V  ▁Lund`V  ▁nouraV  ▁BuddhabV  ▁outingcV  ▁BenefitdV 吞咽eV ▁hexfV  仔细观察gV  同济大学hV  以优异成绩iV  ▁adornjV  ▁AwesomekV 实属lV  ▁cremmV  ▁MemornV  疫情发生以来oV  ▁kindlypV  ▁liquorqV IanrV  对我的sV wantedtV 又何uV  努力和vV  对公司wV  季后赛xV owieyV  关键环节zV  主会场{V Apply|V  那一年}V  发工资~V ywV  人均可V 主城V  团队成员V  ▁HenriV  已更新V  结果是V  criptiveV  ▁roboticV bikeV ▁phpV recognV  ▁neighbouringV lovV  品牌方V 佩奥V 钢厂V  ▁EuropeansV  的美丽V  ▁SyntV  ▁implyV  ▁delegateV  ▁playlistV 不伤V  开始学习V  ▁COUNV 俨然V 古董V TLV  她已经V  ▁prefixV TradeV  ▁EnsembleV 倾倒V  ▁hierarchyV  男性的V  及时了解V 山坡V 横批V 甘甜V  营造了V  普通话水平V 赛鸽V  将会在V  发布消息V 复式V 任你V 期权V 吃掉V 晓光V  ▁decayV 几家V  ▁volatilityV 刘伯V 提请V  好几个V 烟酒V  认真阅读V  ▁chapelV  到处都是V RegionV 筋膜V  为了给V 为数V ▁ByrV ▁SewV  ▁transitionsV 修道V 预料V ▁SERV  ▁clergyV  commercialV 低速V  孙中山V PluginV AUTV  ▁advertisedV 之机V 货到V 预先V SonyV leagueV  车辆和V  ▁offensesV  第三天V azarV 滨湖V  ▁viciousV ``V 多远V  不具有V  如出现发热V 板栗V 淳朴V  ▁parteV 法令V  经常被V BluV  ▁DarrenV  ▁fueledV 南街V  第三种V " 中小学教师资格考试V  认罪认罚V ▁VCV  主要原因V 举国V 超额V  文件夹V  文明出行V  还有机会V  进行检测V  ▁correctedV 较重V  ▁FrançV  很多地方V  ▁malariaV  ▁scalingV 公基V 向其V owellV  ▁MeatV  ▁circaV 补益V  属于你的V  ▁advisersV 甘于V  总有一些V 后盾V 圣贤W BuffW  ▁PokerW 赤裸W 达尔W  ▁suspendW 互利W  联合体W  本通告W 北魏 W 此地 W cop W  把你的 W ieri W  有条不紊W  信息后W sentW  各种各样W  ▁meaningsW 董卿W  要提前W  ▁turnoutW 原汁W 爱豆W  襄阳市W  传统节日W ▁foxW  打赢脱贫攻坚战W 公墓W 武清W  有所帮助W 孝敬W 顿饭 W  手术治疗!W  在这个时候"W 开讲#W  ▁honoured$W  度过一个%W  threatening&W 只得'W 宣扬(W iten)W kil*W  破坏了+W  一级响应,W  ▁Harley-W 水深.W 通缉/W ван0W  全面实施1W  ▁Shirley2W  ▁shaking3W  ▁underp4W  ▁Ceremony5W  来说说6W  ▁Correct7W  ▁replica8W  ▁laboratories9W  的中心:W  充分体现了;W 热播W gk?W 院系@W  的生态AW  iscopalBW  ▁HungerCW  unninghamDW  ▁ConsortiumEW 打到FW ▁TacGW  两年的HW  关于对IW batchJW łoKW 目击LW WARMW styNW  游泳池OW  广泛开展PW gevityQW  ▁passionsRW  感十足SW  可能存在TW onicsUW  ▁fulfilVW 迁入WW  持续提升XW 安危YW 放上ZW  之中的[W  ▁statues\W QA]W 执纪^W  所做的_W  ▁Cove`W  ▁compressaW  报名者bW  ▁MantcW  回复关键词dW 迫使eW  ▁reconsfW rollsgW  ▁WickhW  ▁bibliiW  ▁SourcesjW 中长kW ▁msglW  乘用车mW ▁HewnW 日及oW  常州市pW  控制好qW  哥伦比亚rW 超限sW PLCtW  ▁BuildingsuW  积极分子vW  politicalwW 庐山xW 有助yW 有待zW 师风{W Cry|W  分钟即可}W 小熊~W  ▁bloggersW  ▁StapW 封存W 掉头W 是看W  情形的W  ▁LanguagesW 口舌W 选定W 多款W 流星W 锁骨W 问世W  如何正确W  ▁regainW  ▁DrawingW  全面启动W ecycleW  ▁flownW  供给侧W  学到了W  铝合金W RainW 之恩W 走去W + $提供任何明示或暗示的保证W  人民大会堂W  等有关规定W 狗子W 玲玲W tanW  为患者W  空间站W  总投资约W  ▁EscapeW  要立即W  ▁FramW 亨通W  一律不得W 北欧W 流利W  技术服务W PETW qianW  视频监控W 唏嘘W 介质W 节节W  ▁urineW  ▁RafaelW  ▁repealW 供气W 所言W  年全球W  总有人W émonW  交易平台W 暴晒W  文言文W  预期的W  通俗易懂W  ▁ConversW 路途W 远洋W FaceW ▁woW  拥有着W  及时与W 地基W ZYW  人受伤W  杂志社W  肝功能W  长此以往W  ▁mottoW  ▁AdvisorsW  ▁skepticalW …)W 咽炎W  更多地W  ▁unbelievableW 和无W  力量的W  ▁PresentsW  ▁periodicW 签收W  妈妈说W uniqueW  　|　“W 邮电W  ToStringW 黄油W 成份W  苏东坡W  ······W 扁平W  网络的W  孩子上学W  ▁probableW  ▁repeatingW 催收W 读音W  风清气W  ▁RestaurantsW 内障W ▁VuW  重点领域W  呼吸道感染W  学生和家长W  次核酸检测W 叮咚W 辞去W 苦涩W easyW  换位思考W  ▁sturdyW 也与W 工大W  保障性W 共筑W 失意W  ▁grandsonW 针刺W SHAW  搞清楚W  营造良好W  FragmentW 请缨W  不出去W  结合自身X  是一片X  黑木耳X ruedX  专项资金X  ▁encryptedX  ▁broadcasterX  阿胶糕X ▁GolX  科学研究院 X ECH X 公鸡 X  谁知道 X leaf X  暴饮暴食X  近距离接触X  属猪人X  设计上X 代孕X 房内X 冲冲X  一年多X 倾力X  ▁BeanX ractedX 的单X  二次元X  交易市场X  学长学姐X  horizontalX  ▁indictmentX 壮志X  ▁sensory X 打法!X  ▁Forms"X 变道#X  科目的$X  我们尊重原创%X  ▁Wrestling&X 河湖'X  来解决(X  和大家分享)X licted*X  ▁Ethan+X  ▁Rental,X  最快的-X really.X 和老/X 美甲0X Theme1X  ▁lethal2X  主动配合3X  ▁disastrous4X  中国社会5X Alpha6X  试运行7X  当天的8X ibia9X  为了帮助:X  ▁basin;X 已过X  支气管炎?X imil@X SecretAX  ▁snapshotBX  见到了CX scientDX  matchedEX 唐人FX  戳这里GX 乃是HX 遗迹IX  玻尿酸JX  的人气KX  中国政府网LX  ▁deniesMX  不可否认NX  ▁GenerallyOX  ▁QuartetPX  开始前QX  SouthernRX  不用再SX  德州市TX ydiaUX  母乳喂养VX 管护WX  敏感肌XX  建筑施工YX 冷门ZX 赏识[X  大力支持\X Award]X Bruce^X  ▁Titan_X  ▁inflammatory`X  新城区aX  ▁ElementbX  ▁erosioncX 短裙dX  的世界里eX  ▁antennafX  的命运gX  环境影响hX  ▁ConserviX  至少要jX  ▁ExpedkX  ▁outputslX  ▁KathleenmX 客栈nX  一大早oX 装入pX ynaqX  受教育rX  穿起来sX  ▁craftingtX  成就了uX  走过了vX BurnwX  ▁plaquexX FabyX  ▁recapzX  ▁Tottenham{X Ir|X 错觉}X  情绪的~X GeneX ▁LehX  ershireX  可以给X  而是要X  ▁freedX  潜规则X  我爱你我爱你X  VictoriaX říX 采收X  的我们X  ▁YorkerX ekaX  方便了X  ▁petroleumX 选自X  电瓶车X  责任落实X  可以适当X AaronX  licensesX  ▁psychologistX 胸口X SATX  重症监护X 面霜X  ▁readinessX 鞠躬X 安溪X 讯息X  ▁assistantsX 话术X  ▁torchX  的主力X  还特别X connX  为人知的X 脑部X  灵活性X SilverX 乌龙X  尽最大X  ▁GujarX 偷拍X 由省X beansX alwaysX 六千X 必经X  ▁economistsX 浑然X 肿大X 应是X ▁WisX  ▁FoundedX 四军X 恐吓X 还把X  所产生的X 登高X 痴迷X  不熟悉X idableX  ▁RestorationX 催促X  级阵风X  ▁shinesX  进口的X  ▁ChessX  睡觉时X 奖补X 文殊X  和孩子一起X  ▁retainsX 囊括X izhX ulpX ▁cobX 木瓜X  相关企业X  营销中心X SantaX  ▁CreationX  ▁AuditX 打牌X  inarilyX  ▁anecdX  ografiaX  ▁KeepingX 五粮X 吊顶X 变老X  计划于X  合作关系X  ▁TortX 已将X 林地X 进村X ▁DOIX formsX 得水X 画谱X ▁NSAX 公然X mavenX  ▁PlantsX  ▁confirmingX ObjX  长久的X  带头作用X  ▁werdX 五代X 招办X  聊一聊X  uristicX 的身X  文化传播X TurkeyX  ▁demiseX  的计算X 关卡X 漂泊X  也喜欢X  ▁SchneiderX  ▁decreasingX 寿光X 调任X flexX  ▁cursorY  ▁substrY ArabY  附加扣除Y skipY  ▁AltoY 探望Y 清泉Y 维系Y  参考资料 Y  ▁Judicial Y  ▁nerves Y 苗苗 Y 运力 Y  ▁DiazY  ▁portrayalY 出面Y 驱车Y  持有的Y  综合利用Y  ▁AuctionY  ▁apologizeY 庭长Y  心电图Y  悟思想Y imonY  准考证打印Y  ▁volcanicY  但这个Y  足够了Y  ▁webpageY ▁RIY  也特别 Y  ▁Teresa!Y 周内"Y  提供给#Y  具备良好的$Y  ▁Kapoor%Y 来个&Y 前妻'Y Dat(Y  保持了)Y  碳排放*Y  实实在在的+Y  ▁Carlo,Y wm-Y 养家.Y 箱子/Y  ▁Bedroom0Y  ▁preschool1Y نى2Y 雨雨3Y  编者按4Y  ▁haunting5Y  开发票6Y 前夫7Y  ▁honoring8Y 「　9Y  被视为:Y  了一遍;Y 剂的Y 定能?Y 清甜@Y EEKAY  ▁wovenBY 病床CY 被他DY  工人们EY  旅游景点FY  ▁infiltGY 钢管HY 诀窍IY BloomJY  ▁dotsKY blowerLY 金茂MY  ▁SculptNY  不到的OY  佛菩萨PY 看花QY  人注意RY  ▁figuringSY 石板TY  报名费UY  ▁HankVY 惩治WY 深信XY  分享到朋友圈YY  云平台ZY  ▁insurg[Y 奉上\Y 导管]Y 临清^Y 熔断_Y 请到`Y  已经到了aY  ▁BrokenbY  帝国主义cY  防护物资dY obileseY  ▁armorfY HaygY  ▁curtainshY growthiY 隔断jY  生活里kY  劳动教育lY  甚至可以mY employnY  ▁PateloY JApY 衢州qY  ▁AlienrY 开运sY 转速tY ▁daluY  ▁AlexandravY  ▁veterinarywY 撰文xY 文博yY  蓬佩奥zY 大增{Y gre|Y  流行病学史}Y  usually~Y 无时Y  ▁wrapsY  从中国Y  ▁rebuiltY  ▁arthritisY  各方面的Y 日上Y EgyptY  ▁minusY 传唤Y 几条Y 华裔Y  诚实守信Y 这可Y 野蛮Y  ▁gallonsY ▁$$Y  行程码Y  ▁SunsetY  楞严咒Y  ▁raysY 亿吨Y  和一些Y  传说中的Y  城市轨道交通Y  平均分Y  ▁disciplesY 镇街Y niceY  疫情过后Y  ▁SchwarY 千玺Y 讲讲Y  ▁meritsY  登上了Y  ▁bidsY  教师资格证书Y essesY  aginationY 预制Y  等强对流天气Y 向日Y 秘制Y  ▁HansenY  ▁HoldingY  ▁WebsterY ▁±Y 亦有Y  不言而喻Y 抹黑Y  镇中心Y  ▁StackY  专项附加扣除Y 淋浴Y  那么的Y heardY  例无症状感染者Y  ▁FarmsY 吊装Y  ▁stoveY  ▁такY  广播电台Y  和信息Y  非常时期Y  ▁psychiatricY  一个人在Y  ▁messyY 汉代Y  如火如荼Y  ▁fattyY 自卫Y 财力Y zbekY eniumY 商贩Y 流连Y  具备了Y  ▁CrashY  ▁OpensY  ▁rusticY  ▁policingY  不过了Y cheonY  基本面Y  茂名市Y 潜逃Y  大公司Y  三国演义Y 释怀Y  clerosisY  的带领下Y ▁LIMY  企业经营Y HotelY 叶酸Y стеY 审验Y sampleY  ▁AnywayY  ▁progressedY  会受到Y  社会化Y  赵本山Y 文笔Y  ▁dwarY 跟谁Y 想买Y tlY 研习Y hraY  就是责任Y  licantsY 罚单Y LINY  应按照Y  比较容易Y  EnvironmentalY 标线Y ▁MamY  更进一步Y endenY  日常工作Y 文华Y 独居Y heritY  ▁AiresY  ▁MuseumsY NZY  本专业Y  ▁aggressivelyZ 三焦Z 熟能Z ▁EvaZ  mingtonZ  ▁PerkinsZ WBZ  ▁StefanZ  ▁StoresZ  完之后 Z Berry Z 幸存 Z  从去年 Z  宪法和法律 Z  体系和治理能力Z  ▁ecologyZ  大家应该Z  ▁wherebyZ 慢跑Z  车牌号Z  ▁SavageZ 办发Z 剧照Z ijiaZ switchZ  最终解释权归Z 惯例Z 并行Z 散寒Z igibleZ WaZ aratZ 补足 Z  已超过!Z  ▁peculiar"Z 性好#Z ▁tee$Z  技能大赛%Z  来看一下&Z  民生工程'Z  ▁upwards(Z  ▁году)Z 连同*Z 仅在+Z 挑灯,Z 连绵-Z  日开学.Z dead/Z 宫殿0Z  过敏反应1Z  ▁Nichols2Z  Individual3Z 太白4Z  与担当5Z ▁Ish6Z 嬉戏7Z  比如在8Z  水电费9Z  细致入微:Z 不生;Z 彩蛋Z  看不出?Z  ▁towel@Z  ▁evenlyAZ 海豚BZ  ▁ElonCZ 希尔DZ  还是一个EZ  ▁withdrawnFZ 凉了GZ 重病HZ  ▁InitialIZ qsJZ 棺材KZ  现在就LZ  说一句MZ ▁LahNZ  从而使OZ  生猪价格PZ  共同努力下QZ  ▁continentalRZ 纸条SZ  的培养TZ 藏族UZ  ▁erectedVZ bullWZ  锡林郭勒盟XZ 惟有YZ  台州市ZZ 常乐[Z  大问题\Z ▁ses]Z 大清^Z  肯定的_Z  ▁Transition`Z  个问题aZ  他就会bZ  集中整治cZ  ocumenteddZ  不仅能够eZ  自然保护区fZ  ▁masculgZ  厚重的hZ :'iZ 王丽jZ 美酒kZ  样东西lZ  ▁stallmZ  扫码进群nZ  有可能是oZ  ▁creepypZ  追究其qZ  ▁ExternalrZ  要小心sZ  那么就tZ afiuZ ancockvZ  积极投身wZ  FrameworkxZ  ▁OliveyZ  positivezZ yet{Z  你不要|Z  部编语文}Z  ▁observer~Z 错落Z  活动地点Z  ▁relentlessZ CosZ  ▁IsabelZ 先前Z 您和Z  属龙人Z  心目中Z  所以才Z 屡屡Z ADDZ CONTZ 曲阜Z 绝美Z 间歇Z  战斗中Z  可以提高Z 七里Z 宁乡Z 那点Z brokenZ 头村Z itrustZ  ▁fixesZ 鸿蒙Z  ▁HttpZ 保有Z 暂扣Z ▁"<Z  以保证Z  ▁manuscriptsZ 林场Z  ▁interiorsZ  ▁wivesZ  经济学家Z 桃胶Z 装盘Z 拖着Z  日子过得Z ombreZ 共赏Z 国医Z 她却Z 延庆Z  为用户Z  联系小编Z  发生火灾Z  饰演的Z  这个行业Z SyncZ  竭尽全力Z  ▁librZ  ▁AdvantZ  ▁inlineZ solZ  家的孩子Z  ▁GrillZ 前茅Z 灭绝Z  随着年龄的增长Z 健体Z 赤水Z  craftedZ 减法Z 四类Z  等材料Z  航天员Z  风景名胜区Z 有误Z  交易的Z egalZ 发红Z 收养Z 珍稀Z  入党誓词Z  一次又一次Z 领头Z ▁zaZ olkataZ 小马Z  ▁YachtZ 如潮Z 时下Z  ▁ebookZ 国门Z 理政Z  保时捷Z  ▁VeterinaryZ 拆卸Z  ▁visionsZ ▁ЕZ 许久Z 起锅Z 是名Z  后立即Z  anotherZ 从轻Z  美国和Z  应急管理部Z  ▁SuicideZ 裁决Z 进退Z BeatZ 哈登Z 复仇Z  及时关注Z PedZ  大道至Z  ParentsZ 点至Z 轻便Z  就知道了Z "<Z 迟缓Z  专卖店Z  翻炒均匀Z 立夏Z usableZ  ▁ConnorZ  ▁standaloneZ 从化Z  ▁allowanceZ ( !不符合食品安全国家标准Z нииZ  动员部署Z  县纪委监委Z 落差Z  比较低Z  ▁UEFA[  请查收[ 一票[  等国家[  ▁newsletters[  示范性[ games[ 太和[  无论在[  全力打造 [  ▁stalk [  ▁heightened [ 厌倦 [ 生冷 [ 茶树[ 谦卑[  医护人员的[ 位次[  developed[  ▁indirectly[  ▁Raleigh[  ▁transformative[ 一改[ 厚实[  ▁implicit[ ogens[ topic[ Ele[  张家口市[  ▁Surf[  年年底[  房屋出售[ odus [  纪律处分![ ▁ш"[ 守好#[ ITT$[  负能量%[  emiology&[ 配饰'[  等产品([  ▁Marines)[ 汤姆*[  民政部门+[  configuration,[  一段话-[  ▁mont.[  桂林市/[ 前几0[ 高密1[ Cola2[ ▁Paw3[  不住的4[ OWER5[  ▁furnished6[  ▁standardized7[ 履约8[ 意气9[  送往医院:[ oodle;[ 兰亭<[  圆满结束=[ По>[ 抽样?[ pred@[  副所长A[  运营的B[  治疗后C[  中汲取D[ GHzE[ ▁DumF[  全面小康G[  ▁mouldH[  ▁ignoranceI[  ▁proceededJ[ ▁OEMK[  课程安排L[  这段感情M[  猜你还想看N[  听不懂O[  ▁earnsP[  就因为Q[  市场规模R[ 报班S[  悄悄地T[ 法兰U[  童年的V[ ▁LNGW[  ▁DelegX[  套餐内容Y[ 裁剪Z[  招聘要求[[  系列产品\[  ▁outbreaks][ 百元^[  你点的每个_[ 耐力`[  ▁Povertya[  合作办学b[  ▁imprisonedc[  ▁Wentd[  existinge[  ▁ideologicalf[  一体化发展g[  ▁Registryh[ meii[ Altj[  角色的k[  ▁cheesl[  高血糖m[  怎么办呢n[ atibleo[ 爱车p[  ▁vowedq[  ▁cushionr[  ▁crossovers[ 封控t[  将面临u[  工作专班v[ ucklew[  Entertainmentx[ 静止y[ Levz[  风清气正{[ 警卫|[  房产税}[  ▁SERV~[ 调色[  不辜负[  ▁rains[ 塔尔[  园小区[  ▁spicy[  是非常重要的[ 栽种[ 长者[  直接把[  组织了[  ▁Pablo[ 略显[  无论你是[  ▁ideally[  这个道理[ liness[  文化产业[  覆盖率[ 祭祖[ 二线[  想方设法[ 稳稳[  ▁Goodman[ 产学[ 洗脑[ 皮革[ 露台[  的人会[ Sort[  ▁Albums[ 路透[ 阻滞[  ▁fren[ 桥村[  几年来[  放在心上[  这个项目[  ▁Seed[  ▁Hassan[ 奇观[ 炒房[ 魔幻[ 邑县[  　→　[  ▁farther[  爱情运势[  ▁sprink[ 掏出[ 无烟[  上进行[ Youth[  岁以上的[  百事通[  餐饮浪费[  最大限度地[ 定型[ sports[  ▁Bucks[  的含义[ iaison[  ▁Felix[  Charlie[  ▁Fletcher[  ▁longevity[ 有天[ 此情[ Ham[  传达了[  教育学院[  现代医学[ 夹杂[  ▁Exercise[  ▁preceding[ 一通[ odor[  ▁Lowe[ 油漆[  这一辈子[  Captain[  ▁socialist[  人员及[  和改进[ 青梅[  ▁athe[  ▁clicked[ 度数[ 赛车[ ▁Mai[  具有很强的[  来一场[  动态调整[  ▁Kerr[ При[ 低俗[ 缠绵[  的错误[  Features[  在今天[  所在小区[  ▁Failure[ Scot[  心甘情愿[ 她用[ 日止[  营业额[ movie[  申请书[  的汽车[ 管好[  长河中[  ▁voiced[  ▁equations[ Toy[  的动物[  Initial[  前期工作[  参加体检[ _([  ▁Fusion[ ibus[ ifles[ Quant[ later[ 抛开[ Dou[  我们一定要[  ▁coveted[ 削减[  ▁Jacques\ 书屋\  从来没\ 即墨\  欢迎转发\  调查组\  ▁prohibit\  ▁dismissal\ 也称\ 私处 \  时一定要 \ 两旁 \  写在最后 \  ▁simplest \ 大厨\ 大水\ 盘锦\ 酒厂\  ▁commemorate\ 薇娅\  第七届\  主动联系\  各年级\ 中都\  ▁slopes\  ▁Graphic\ 限于\  的手法\ 实绩\  和群众\  几乎所有\ rn\  信息网络 \  ▁Recogn!\  ▁spectators"\ 市属#\ 统领$\ scrib%\  身份证原件&\  语文园地'\ separ(\ 陶醉)\  营业厅*\  ▁strategically+\  的体验,\  许许多-\  ▁folding.\  ▁unexpectedly/\ 学弟0\  常委会会议1\  当地人2\  艺术设计3\  ▁parcel4\ 可得5\  ▁lime6\  也一直7\  疫情发展8\  ▁pigs9\  濮阳市:\  ▁Canberra;\  setting<\ 扑鼻=\  ▁splend>\ 复原?\ 抚慰@\  redibleA\ 免难B\  ▁guardianC\ 百草D\  全体员工E\ visoryF\  ▁KillerG\  做作业H\  等物品I\ ▁MaeJ\  湖北武汉K\  聚合物L\  合格证明M\  恶性循环N\  这个东西O\  ▁PassionP\  ▁ventilationQ\  大排查R\  MichelleS\  克里斯T\ 产假U\ 除湿V\ 巨变W\ 福禄X\  变电站Y\  全国人大常委会Z\ 　<[\ ▁Fur\\ pering]\ 抵触^\ 拟定_\  ▁Spectrum`\  ortmenta\  专家们b\ 招股c\ 纯属d\  兢兢业业e\  ▁Pregnf\ SVg\ 想了h\  和社区i\ 顶顶j\ 眷顾k\ 祸首l\ 肩胛m\  ▁Owensn\ 古筝o\  但这种p\  ▁dryingq\ 乏味r\  了一只s\ oodlest\ 市局u\ 做细v\  保护你w\  初三年级x\ 装着y\  行动中z\  虚假宣传{\ 看在|\  今天我}\  林生斌~\ 碗里\  Platform\  ▁standout\  ▁presenter\  在职场\  ▁Harvest\ ▁Ads\  ▁coloured\ ATS\  ▁admired\ 缩水\ 谈天\  警务辅助人员\ 兴致\ chel\  ▁uniforms\  该地块\  ▁Hulu\  ▁induction\  交界处\  二十三\  消防救援支队\  ▁plugins\  跨区域\  经办机构\ 严打\  对别人\  管理类\  ▁diploma\ 县直\  ▁Convert\  ▁Anglo\  不合适\  文化底蕴\  ▁enthusiast\ 揉搓\ ▁mu\ 吃上\ Gab\ itud\  只有一次\  ▁clone\  conversation\  深入实施\ 血肉\  精彩纷呈\  ▁gotta\ 斑块\ 青藏\ elles\  ▁responders\ 发丝\ 拉丁\  白内障\  紧迫感\  ▁Cumm\  ▁endurance\  执行的\  显而易\  陈女士\  新闻报道\  防暑降温\  ▁heater\  District\ 夹雪\ 好吧\  经济体\  ▁Neal\ 4 -新型冠状病毒感染的肺炎疫情防控\  援鄂医疗队\  ▁Translation\  ▁Rehabilitation\  作者的\  国有资产\ 它就\ 翩翩\ 托育\  刘老师\ 嫖娼\ 救赎\  ▁winding\ 超乎\  ▁Tucson\ Fig\  她自己\  ▁chaotic\ 上空\ 直男\  脑海里\  我们经常\  ▁Giov\ ха\ 天和\ 编译\  不要用\  一家三口\ 嫩滑\ 看我\ 占卜\  ▁emphasizes\  比去年\  ▁Savannah\  做一次\  ▁café\ 景气\  每一条\  FFFFFFFF\ 这组\  可行的\ faith\ 兰达\  进行审核\  防汛救灾\  全国人大代表\  atement\ 种地\ 花盆\  切不可\  ▁exploded\  ▁cigarette\  中国最\ campus\ ounty\  ▁Carpenter\ NJ\ 这叫\ Esp\  时可以\  ▁Malaysian\ 路南] 养鸡] 弊端]  来实现]  的技巧] Props] death]  Northern]  ▁Bridges]  专注力 ]  遥远的 ]  不会出现 ]  ▁showcased ] 减税 ]  民间借贷]  ▁Recording]  plugins]  ▁Hire]  ▁Dragons] 券后]  ▁rainbow] 大号]  ▁Bone]  ▁adaptive]  如何做] worm] 节食]  是不一样的]  住建部] okovic] 武穴]  管理能力] 匹林 ] 奇闻!]  判断力"]  疫情中#] posts$]  ▁Harlem%]  通州区&] 巡检']  ▁Catch(]  牡丹江)] Cool*] Drug+]  每天更新,]  设施建设-]  报告中.]  这并不是/]  在保证0]  我只能1]  方式的2] arte3]  ▁Aval4] 省时5]  也很有6]  ▁Nicolas7]  ▁costing8] 报应9] 平阳:] 集散;]  挑战赛<] 再无=]  从头到>]  ▁Brendan?]  ▁Lottery@] 在华A]  从小到大B]  活血化瘀C]  广泛宣传D] JesusE]  毋庸置疑F] 感同G] 择偶H]  短时强降水I]  ▁directiveJ] 时令K]  战争的L] 学妹M]  体育课N]  无接触O] ILLEP] 表的Q] 常驻R] 斯的S] PtrT]  中的应用U] sunamiV]  教学设计W] samX]  西兰花Y] 撕裂Z] atan[] nea\]  相适应]]  ▁novelist^] 缔造_] 辽阔`]  常见病a] 出站b] 纱布c]  看了就d] 温岭e] luentf] 一众g] 十方h] 感想i]  勤俭节约j]  观世音菩萨k]  ▁snippl]  ▁partisanm] 汉子n] 刚性o] 高烧p]  保温杯q]  危险性r]  彻底的s]  后即可t]  正当时u]  深呼吸v]  给自己一个w]  ▁liftsx] cutsy]  ▁Bromz] amento{]  ▁freedoms|]  ▁Balance}] Sel~] otions]  ▁gadgets] 乱停]  的小姐姐]  ▁Pepper]  ▁Basically]  请咨询]  解决的问题]  ▁discharged] 左转]  不可抗力] 拿走] 通宵]  很开心]  文件要求]  ▁disruptive] 无味]  防范化解] 老太]  产业基地]  但是因为]  ▁Accessed] /(]  又没有]  负责的] 立于]  ▁Conv] 视点]  立即抢购] 不育] fat] saf] 之花]  centered]  ▁adolescents]  综合自] ▁TRA]  点击阅读] 品茗]  有担当]  等一批]  拟录用]  ▁inconsistent] 浓烟] Emma]  ▁Mobility]  专业要求] 阴凉] 含蓄] 我却] 运载]  实地考察]  ▁campaigning]  ▁datas]  下水道] 入室] 小腹]  另一方] 甜味] 别具] 繁重] 红椒]  主要由]  ▁jailed] ▁COR]  做大做强]  ▁compelled] ▁yog]  ▁кон]  ▁favored] ibi]  ▁motivate]  贯彻新发展理念]  ▁Fors] 忠告] 苦辣] IFT]  本学期] 下课] 出战]  中国籍]  包括了]  升级版]  ▁Shooting] Tip]  ▁cried]  ▁Cannes] 协查] HOME] shift] Advert]  ▁Serial] 皇室]  动脉粥样] phthal] 向内] 建制] ▁Fee]  ▁coastline]  　*　]  到头来]  你的身体]  ▁forged]  长期坚持]  动动手指]  ▁perceive] 恍然]  马上就] riz] County]  ▁headers]  ▁fragments] 云朵] 伊利]  ▁payable] 雾霾] 遇漫]  ▁fostering] RD] 促成] 的肉]  ▁Stam]  ▁annex]  五分钟]  ▁turf]  ▁banning]  老师会]  ▁fragment] 那儿]  内容了^  的证据^  市民政局^ 大餐^ 好似^ Answer^  生产生活秩序^  ▁Presbyterian^ uti^  广阔的 ^  真抓实 ^  ▁cleanup ^  奥地利 ^  有用的 ^  把所有^  绍兴市^  计划表^ 和爱^ 情操^  岁之后^  ▁Sail^ 启东^ ▁sul^ ▁vor^  ▁Gabri^  问题进行^  ▁Trees^  ▁Seasons^  大众的^ 精读^  ▁zombie^ 每种^ 祠堂 ^  点亮在看!^  ▁highways"^  ▁Quarterly#^ 曲目$^ ▁Eur%^  ▁puppy&^  ▁waterfront'^  最简单的(^  预计今年)^  ▁scans*^ 大麻+^ 探秘,^ 兴义-^ 闭合.^ cry/^  学习效率0^ ILY1^ vivo2^ 衣着3^ 视同4^ 鹰潭5^  明亮的6^  法律知识7^  ▁WARRANTIES8^  是大家9^ 透亮:^  等功效;^  想象中的<^  ▁remedies=^ 东阳>^  ▁cheek?^  好莱坞@^  我们发现A^ 大区B^ ancoC^  还是没有D^ 换代E^ 确切F^  不明显G^  赔偿金H^ ckerI^ LowerJ^  万万没想到K^  ▁bossesL^  各位考生M^ 地主N^ ERNO^  各项目P^  在社区Q^ 顶峰R^  ▁RomeoS^ 水泵T^  去世后U^  高血压患者V^ 铲屎W^  ▁minersX^ 援手Y^  承办单位Z^  ▁realizes[^  建档立卡贫困户\^ 云海]^  活动价^^ Rele_^ atore`^  ▁Goodsa^ 已对b^ 叫停c^  ▁oathd^  ▁voyagee^  ▁roboticsf^ 假象g^  体检标准h^ 五号i^ 萌宠j^  曾说过k^  理财小白l^  保护作用m^  房屋概况n^ 睡了o^ ACSp^  心目中的q^  ▁typedr^ 甜言s^ 过低t^ 一筹u^ 年化v^ 稀有w^ 豆豆x^ atchedy^ 替你z^ 要和{^  ▁avail|^ 干妈}^ 扬尘~^  ▁sins^ 匪浅^  探测器^  ▁culp^ 国歌^ 抚州^  回应称^  ▁patented^ ▁Kai^ 贵港^ 海贼^ sudo^  第二季度^ alert^  均来自网络^  ▁tant^  ▁Clement^  ▁confrontation^ CEP^  高级教师^ rosis^ 空旷^  的一张^  ▁pillow^ 反动^  ▁competence^ 输卵^ 场均^  真的好^  网易云^ ▁NFC^ 白条^  ▁Stick^  rieving^ 海东^  因为这些^  直播平台^  aturdays^ Mit^ 美发^  决不能^ ▁Kad^ 殡仪^  侦办中^  属羊人^  看见了^  ▁expire^ 陈小^ 岁后^  发明专利^  Institute^ 一女^ 施用^  甜甜的^  柴米油盐^ wives^ 庆阳^  ▁Neck^ 立功^  但不是^  农民的^ 斤斤^  相似的^ athlon^  ▁Roland^  ▁Effective^  ▁exchanged^  工作环境^  平面图^  早治疗^  防控指挥部^ 邪教^  ▁Mits^ 双臂^  ▁cone^ ,@^ 损毁^ 轻奢^ 急速^ Late^  ▁veto^ ▁sig^  ▁apost^ }/^  ▁turmoil^ 偿失^ 分时^ 梁某^  对违反^  先进单位^ >@^ 加盐^  ▁Leigh^  Invalid^ 建构^ 脊梁^  手机和^  学习效果^  开开心^  提取码^  直接在^ nx^  火锅店^ irrel^ esk^  ▁Mohamed^ 本篇^ ACP^ Nik^  放在了^  Primary^  新教材^ ▁Moy^  书法作品^  文化交流^  ▁tilt^  ▁Digest^ 不至^  到什么^  发生率^  ▁arose^ 图虫^ 江宁^  年之后^  ▁crap^ shadow^  ▁Tropical^  也存在^  党员的^ 或被_  供不应_  社会保险费_ stroke_  二十一_  第四次_  ▁Kell_ Scroll_ 入心_ 益智 _ Zen _  ▁hello _  的姐姐 _  和质量 _  心理健康教育_ 四史_ 折算_  ▁waiver_ psych_  ▁consultations_  人流量_  决定是否_  无偿献血_ íc_ 盛行_ 郁郁_ 高光_ Anyway_  第二波_  等都是_ onica_  Contrib_  科技园 _  ▁hailed!_ 减半"_ 粒子#_  ▁Goal$_ 科尔%_  过来了&_  ▁Tory'_  ▁bracket(_ ahawks)_  ▁mediation*_  和行动+_ opus,_ enting-_  陪同下._  幼儿园教师/_ 倾心0_  站下车1_ 不妥2_ 与共3_ 集采4_  ▁Payne5_  ▁Infinity6_  ▁headaches7_  ▁shipments8_  宜昌市9_  ▁CEOs:_  交通运输厅;_  名副其实的<_  中高考=_  ▁bishops>_  ▁volcano?_ 成正@_ 诸侯A_  ▁crackedB_  再好的C_  紫砂壶D_  胃溃疡E_  ▁ReconF_ 边远G_  本年度H_  文化和旅游厅I_  收入来源J_  ▁replicateK_  了一番L_  会带来M_  境内的N_  无明显O_  ▁boilP_  ▁shipmentQ_ 图谱R_ aciaS_ 并茂T_  并伴有U_  专业学习V_ scenesW_  便开始X_ 三遍Y_ 备孕Z_ 霸权[_  去世了\_  的帮助下]_ 喊着^_  山水画__  素质和`_ Korea_  的丈夫b_ 二胡c_ 仁怀d_ 熠熠e_ ▁HQf_  ▁Napoleg_  ▁Wimbledonh_ ▁pci_  岁及以上j_  而是因为k_ 字形l_ 抢到m_  的第一个n_ 试管o_  ▁facilitatedp_ 车前q_  时间来r_  ▁Levys_  你就可以t_  坚决遏制u_ 乌拉v_ EXPw_ ▁Tubx_  如果大家y_  ▁obstructz_ 架空{_ conc|_  ▁amph}_  ▁Africans~_  ▁translations_ 倍数_ jas_  注射液_ often_  ▁Tribunal_ 脚手_ 通路_ 重于_  号地块_  供不应求_  Quality_  ▁Warsaw_ 各班_  ▁Nazis_ 听懂_ Git_  紧急情况_  ▁embarrassing_  跟孩子_ ▁Elm_ theme_  ▁cruc_ 资历_  ▁Alive_  中国公民_ bbs_ period_ 领养_  小视频_  肇庆市_  DateTime_ Ak_ 玉树_ 周易_  可能需要_  不知道怎么_ 如山_  ▁signaling_ 振华_  疲劳驾驶_  ▁Difference_  王羲之_  的种子_  轻松的_  这个事情_  ▁enclosed_ 督学_ aram_ 均未_ 感性_ 接踵_  三文鱼_  考试中心_ 徐汇_ 携程_  葫芦岛_ 传记_ 联结_ 赞成_ atro_  ▁Warn_  amacare_  加强了_  扶贫办_  ▁Indie_ 栖霞_  ▁Epid_  金针菇_ 跑车_ oooo_ fetch_  办理了_ entin_ criber_  ▁sedan_  ▁Pearson_ 德阳_  ▁Hawk_ illian_ 启程_  人性化_  研修班_  文化活动_  uddenly_ 张爱_ anca_  法律文书_  ▁lamb_ posure_ 曝出_  ▁Bulldogs_ 长裙_  ▁weaker_ choice_ 新奇_ 试想_  进行咨询_ 上层_ onomy_ 哪吒_ 惊天_  无尽的_ 允珠_  黑社会性质_  ▁configurations_ 探视_ qiu_  在不断_  ▁precautions_  中国国家_  中的一个_ 下设_ 甲等_ 之选_ 如皋_ 显高_  ▁guessing_ 得主_  ▁mammals_ 出没_ 办卡_  大运河_ 糕点_ 防癌_ forces_  ▁steer_ 太差_ rots_ ▁sql`  交通拥堵`  ▁artworks` 而今`  能为力`  这不仅`  从现在开始`  ▁definite`  ▁thirteen` änd `  ▁Positive `  ▁Brotherhood `  ▁enhancement `  ▁versatility ` }^` 的铁` ▁```  遭遇了` ▁PAR` 一缕` 如无`  直接用`  就是你的`  ▁landmarks`  健康养生` 考级`  ▁Diploma`  他们说` 清肺`  和理解`  的关心`  的比赛` NULL `  清洁能源!`  退役军人事务局"`  ▁screenings#`  第一场$`  ▁Residence%` 茁壮&`  食用菌'`  ▁су(`  ▁filename)`  关键在于*`  更多新闻+` idade,`  ▁degradation-` 调用.` ieder/` 默写0` Seg1`  咽拭子2` antine3`  ourning4`  ▁governors5`  思想和行动统一到6`  ▁manipulate7`  的热量8`  ▁visas9`  ▁platinum:`  ▁trustees;`  有困难<`  谢谢大家=` Museum>` 吹响?` 蓬莱@`  生活馆A` HungB`  ▁dancesC`  ▁alarmingD`  ▁HertE`  ▁BelieveF`  ▁TributeG`  岁男孩H`  时间和地点I`  ▁terminatedJ`  如出现K`  餐饮业L` shieldM` 不戴N` 乐事O` 直觉P` WORQ`  东城区R` iblingS`  ▁underscT`  也不再U`  ▁HoleV`  长按图片W`  ▁AerospaceX` 神木Y` 送的Z`  平常心[`  把他们\`  顺义区]`  在中国的^`  看望慰问_`  ▁Lance`` 饱腹a`  ▁swayb` 知性c` 竹子d`  的兴趣e`  采用的f` 真谛g` hwh` 瑞幸i` 却也j`  百年党史k` 如对l`  章子怡m`  ▁Gloun` 中印o` 但当p`  Weatherq` 很深r` EROs`  型人才t` 两件u`  ▁Sachsv` opersw` Sqlx`  人事考试网y` 提防z`  不动摇{`  ▁trustee|`  ▁Headquarters}`  出了一个~`  基层干部`  ▁shells` 我做`  的资源`  的高低`  同步练习`  就会变得`  ▁Shan`  ▁Thames` 新任`  ▁perv`  ▁assemble`  的防控`  要不然`  ▁Omicron`  港澳台地区`  ▁physic`  ▁grapes`  ▁duplicate`  学习内容`  打造一个`  ▁unpaid` 祖母` 刑拘`  很容易被`  ▁forgive` _,` 家书`  普通类`  门把手`  找到一个`  ▁salon` 秋收`  ▁economical`  第十一` Prev`  ▁Chandler` 安好` ▁Ci` 瓶装` zilla`  Toronto` Lif` bish`  和自己的`  独一无二`  其原因`  斤斤计较`  ▁completes` 林木` oldid`  ▁Coca`  ▁coordinating`  ▁believers`  ▁waterproof` 尊贵` 军衔`  上百万`  也是我们`  有序恢复` illac` ;">`  翻天覆` xlsx` trust`  都很好`  精心组织` 今朝` 豪情`  全面推行`  环境污染`  经验丰富的`  ▁shrink`  文章中`  理论学习中心组` zos`  缴纳社保`  聘用手续`  iliated` 一艘`  一流的`  ▁angular` 桂枝` nail`  initial` 攻势` 逐梦` typ`  真的会`  有针对性地` 老友` 脚的` 诗文`  古建筑`  作息时间`  经济师` Flash`  ▁hipp`  ▁reputable`  pleasant`  ▁clinicians`  公司将` 弗洛` 时在`  在我看来` errals`  主演的`  管理模式`  ▁unwilling`  手里的`  税务部门`  好身材` 站稳` 袭来`  民生实事`  ▁testify` 陷害`  不好的人`  家国情怀`  已经过去`  一段感情` fried` 发发` 国之` 占到` 抄报` nb` 绘图`  为代表`  ▁privata  ▁emergesa  包吃住a  空气流通a Asiana  ▁attackera 估价a 吞吐a  可能被a  持续到 a  量身定制 a 谁说 a ",　 a  ▁Borg a  microsofta  十四条a  ▁timeouta  农商行a  申请的a  ▁Signala 换证a 牟尼a  ▁rippeda  很多人都会a 蜻蜓a stooda  ▁Hawkinsa 岩松a 床垫a  小故事a  免门票a  的表达a  节目组 a 助人!a 葬礼"a 春分#a  情况来看$a  ▁toilets%a 屈原&a 肥肉'a ник(a зы)a  的美誉*a  情况通报+a  ▁borrowed,a  文学家-a  ▁entertained.a 清缴/a  ▁exert0a AJ1a 拿去2a  ▁Clerk3a  传染病防治法4a udi5a ▁arr6a  就像一个7a  ▁gast8a  ▁suppress9a 向善:a  需要有;a  ▁Hatea uku?a  江门市@a  默默无闻Aa  ▁salvBa  ▁HimalCa 霞客Da  立案侦查Ea ucharFa  保鲜膜Ga ▁SSDHa  ▁sectIa 胆汁Ja FGKa  本来就是La andalsMa 一箱Na 内而Oa  ▁immenselyPa 扬子Qa  期满后Ra  在这方面Sa celainTa  ▁psychedUa CXVa 耐受Wa asoXa ག་Ya 大安Za  ▁Rivera[a 矢志\a  公筷公勺]a  提供虚假^a  ▁Corey_a  ▁Crypto`a 佼佼aa 风里ba bbieca  编辑整理da  执业医师ea  ▁cheatingfa 我好ga matrixha 税法ia  ▁commoditiesja 打字ka 表哥la  家属院ma  静脉曲张na ettleoa 连心pa rittqa  ▁blockbusterra  少一些sa Snapta 声响ua  可再生va  舒适性wa  ▁toesxa 发黑ya 存货za  雨夹雪{a  线上课程|a  ▁repaired}a  基层党建~a  如涉及侵权a  ▁unsuccessfula 带好a 涉水a 软的a  培养出a 给付a 谐音a  ▁Vincea 自建a  师资力量a 汇丰a  实际上是a Womana 带了a 或用a atakaa  第三人民医院a  ▁descendantsa діa 急难a  黔东南a 收尾a 警长a  ▁Fruita Kenta  ▁fleeinga  ▁Compensationa  中南部a riceda  ▁sobera 本命a 海运a Arma  合同约定a 金丝a  很多小伙伴a loadera  修正案a  动起来a  licensea  不理想a  ▁Rangera 罗某a  拉开帷幕a  研究生院a  和图片a + $习近平总书记重要讲话精神a  最前沿a Yorka  中进行a  贫困家庭a 字符a oaka  氏家族a  terburya  并配合a 旧的a 吴忠a 孩童a  ▁Uttara 错位a 进士a ushua ▁Tula  他们都是a  ▁Electorala  有人问a  ▁appliancea  ▁influenzaa  ▁concurrenta  为你的a  孙允珠a  的大脑a ▁Toua  ▁takeovera ▁CFOa  也没什么a  代名词a  优势和a 栀子a  而不能a  ▁condola  常务会议a 河里a  坚决杜绝a vettea  ▁nonprofitsa 醒悟a  报考岗位a  ▁composta 停机a  ▁rainya 拜托a 笃定a  和书法家a  思维能力a  第二中学a 骁龙a  卫健局a  看守所a  constructa nena  另一边a  ▁Veterana  ▁pleasinga  肺结核a ▁Boma iberna  ▁voluntarilya ~)a 台前a 撒谎a 培优a 轻视a  ▁atoma  ▁rivalrya  Universala  主题为a 有句a  算得上a ▁Wena  一个多月a  ▁absorptiona 认领a atla  深处的a  ▁Marcela 发至a  找机会b Connb  ▁syrupb 惯性b  若干名b  参加高考b 听觉b  与学生b  总数的b  thereum b  欢迎加入 b  ▁surgeries b  这个动作 b  ▁mills b 他才b 相等b  骄傲和b ▁narb ensisb  Bachelorb 它还b 我被b  自驾游b  ▁Malib  ▁Calledb untaryb 打坐b 达拉b  土木工程b 只顾b 爆笑b  需求量b  期限为 b  ▁werden!b  ▁dispers"b  ▁homicide#b 褶皱$b  号台风%b  ▁Bing&b  ▁antagon'b  日截止(b  ▁exercising)b  我们无法*b  ▁moms+b  usement,b  ▁toddler-b 弹簧.b 脚本/b  忽视了0b  ▁как1b  防灾减灾2b  reviewed3b 干枯4b 加我5b 尔蒙6b 辨认7b  正式实施8b  ▁Assault9b  ▁squeeze:b  ▁Concrete;b 专著b  也不同?b  西北部@b pillAb  ▁OmahaBb StageCb  ▁periodicallyDb  网友表示Eb  波澜壮阔Fb  performingGb 剁手Hb becueIb  ▁FranzJb 寿司Kb PubLb  和处理Mb  ▁quiltNb  也有不少Ob  ▁paranPb  ▁TrialsQb 草书Rb 分销Sb 相知Tb  火龙果Ub  ▁MortgageVb  但是你Wb  ▁agileXb RMYb  到学校Zb  新乡市[b  ▁notch\b 机身]b 清运^b 着陆_b  的风景`b  视频课程ab  再怎么bb  到夜间cb  通过手机db 德才eb  对于一个fb  ▁Deckgb  和分析hb  更愿意ib  ▁outgoingjb  ▁landscapingkb 饮茶lb  终于可以mb  ▁Purduenb  ▁hollowob ykpb  的刺激qb ▁AAArb  ▁accessorysb  ▁iterationtb bangub 灵的vb  Softwarewb  ▁Portfolioxb 曲阳yb  ▁rooftopzb 殊荣{b 特旗|b 闲聊}b heets~b  心绞痛b  科技城b ustleb 吉尔b 长在b  也有所b 之争b 平滑b  不配合b  获奖名单b  市场供应b  ▁Salonb  ▁Comfortb 忘却b 手写b  检测报告b ▁Öb 挺好b eptb  及时就医b Scaleb  ▁disciplinaryb 正道b audiob 飞往b  ▁insecurityb POLb resetb  丰富多彩的b  avanaughb  实际效果b  师德师风b  异地就医b 隔绝b  的生活中b  ▁Aberdeenb 初夏b 国境b yclingb  我们认为b  是一份b ▁SUNb  ▁instabilityb  白云区b 物力b  的长度b  ▁Edmundb  ▁Advocateb 苗条b  ▁Barberb 走遍b  ▁Codyb  ▁cultivationb 节的b Jewishb 从没b  集聚区b  经营主体b  ▁engagesb ▁sorb  扶贫搬迁b alogyb  ▁Discussb  Registrationb 备好b 是说b  ▁Accomb 而归b  ▁Candidatesb  ▁Stadb  ▁clutchb 菜的b 虾仁b  ▁frequenciesb  送到了b  网红打卡b ückb ▁bakb  ▁reuseb 鞋底b  ▁grievb 不菲b 内卷b '''b  银行存款b 成家b 锦上b  ▁Idolb 体美b untab  招生人数b thorab  ▁Bonusb  在水中b HTTPb  ▁crystalsb 争创b 踏板b  ertilityb  毕业班b 共存b 小屋b 要为b 诊室b  传播的b  ▁toastb  ordinateb Tib  有可能会b ‍　b  ▁postalb  ▁melodiesb 大变b ▁WSb mondsb 斤的b  市人社局b  historicb  ▁metallicb  ▁servicingb 桑葚b 满月b 距今b ▁CSTb 米处b  病毒性b 提神b  向记者b  纸质版b  感同身受c  Administc 吝啬c 洗面c Qualc 绿树c 蜂窝c ないc  像极了c  ▁Lomb c  ▁curved c 风和 c  不规律 c  文章底部 c 李雪c  使他们c Lostc ▁Gelc ▁faxc ascadec  科学城c  战国时期c  调查处理c  auticalc  ▁averagesc 丰顺c 想尽c  考试时c 痛快c  排放标准c Askedc  ▁filingsc 深色 c  办公室副主任!c  ▁simplified"c 依依#c 相待$c 绿洲%c  推广应用&c  ▁irony'c 丹参(c 于人)c seven*c ▁KS+c  和各种,c 乌鸦-c  免费开放.c 可查/c  报告会0c  数学下册1c  Fashion2c 塔罗3c  积极应对4c  foreach5c 慰藉6c Sah7c  ▁cryptocurrencies8c seas9c  ustomed:c  ▁displacement;c yrec  实名认证?c  评选活动@c  症状的Ac appsBc " 人正在接受医学观察Cc ísDc 角的Ec  山街道Fc 三路Gc CroHc 双拥Ic  劳动法Jc  环境优美Kc  线上直播Lc 比武Mc 乡亲Nc 过时Oc  的事迹Pc  icheverQc 并重Rc 悲欢Sc  生病了Tc  所在地址Uc  积极贡献Vc  ▁pioneersWc 琥珀Xc 粉尘Yc  工作满Zc  规范的[c  如果真的\c 尽收]c  的高温^c  ▁staring_c  和全国`c  黄冈市ac  车辆通行bc  ▁Vikingcc  持续性dc ▁Ripec  ▁Rollsfc 泥石gc 闽南hc quaic ▁PCsjc challkc  点击空白处lc nilmc  同志为nc Aliceoc  criticalpc  ▁disappearanceqc 满目rc 茴香sc  中间的tc  其实都是uc  ▁acceleratingvc 夸赞wc 字典xc 版图yc 珍品zc 颜料{c  活动时|c  ▁chord}c  ▁dreamed~c  放心的c  核酸采样c  在会上c 铃薯c  能起到c  是最大的c ecutec  astingsc  ▁irrelevantc  工作重点c  月份开始c  生产工艺c  下图二维码c  ▁Sixthc  国家税务总局关于c  Extensionc 抽屉c 臃肿c  完即止c ektc  ▁Creedc  Opinionc orestc 中英c 轮番c Leaguec  ▁Accountsc 夸大c  肿瘤医院c Samplec  ▁durchc  ▁unbeatc ▁MODc  大规模的c iegelc  Perfectc  恋爱的c ▁rayc  ▁sixtyc 遮住c  ▁convec 洁面c  不仅没有c  点击链接c  纪检监察组c  ▁redemptionc 生完c  ▁Wattsc  ▁soaringc  新校区c Kayc  的男子c avidc  ▁Vincc  ▁toughestc  ▁counselorc 上师c ▁UWc ▁Yac  生命体征c Domainc  中没有c  ▁dietsc esanc  集体经济组织c  ▁distractedc  ▁Alreadyc  ▁mushroomsc 叮咬c 缜密c  日夜间到c  ▁federc  bindingc 之行c 门头c BOSSc  一直以来的c 尊享c  一段时间后c 一壶c 创设c  工作责任c  ▁Airbusc 生财c 醉了c ▁{"c  也不需要c ientoc  还有更多c  ▁boilerc  ▁cinematc ▁Vocc  专题讲座c sellerc 北约c 方方c  退休金c  发展史c  或删除c 创投c  元抢购c  柔软的c 弥勒c ascalc  ▁Workplacec 毛毛c  ▁Bellac  ▁coachedc ussyc  信息安全c  活动场所c  防疫要求c 一团c 所想c 浩瀚c  关键技术c asersc  和治疗c  重疾险c  经营范围c  静下心来c  有疑问c  ▁departedc  ▁femininec  ▁inductedc 后即c  Atlanticc 护肝c  仅限于c  ▁alleviated 保住d  经济发展的d 你用d  对你说d  本人有效d  牢牢把握d 右转d  orporatedd 崩塌 d  应该怎么 d  录取结果 d 临沧 d  ▁captivating d 工夫d 布朗d  赤峰市d  广大师生d 煸炒d Wonderd  ▁Ferryd  不希望d  ▁chambersd foliosd 上色d 断面d ▁Nitd  smanshipd 取舍d 计数d  机器学习d  ▁slamd  ▁Majority d  ▁ceilings!d рав"d  ▁prow#d 大二$d  ▁reinst%d  ▁midfield&d 不朽'd 到天(d 户的)d 还剩*d 都应+d  团员青年,d  高峰论坛-d 延边.d  ▁homepage/d  ▁documenting0d  执法检查1d baidu2d  ▁Satellite3d 拉拉4d  英雄的5d  等法律法规6d mop7d 在为8d  不利影响9d  ▁rightly:d  ropractic;d 入狱d 连麦?d  防腐剂@d ▁hawAd  怀疑自己Bd  ▁anthologyCd ▁FahDd 逝者Ed  很显然Fd  ▁adventurousGd 袖子Hd  服务队Id ducersJd  ▁tallyKd 高冷Ld  ▁BentleyMd  ▁standingsNd 沙子Od 能将Pd  这些话Qd oscopeRd 并把Sd  慢性疾病Td  ItalianUd  双循环Vd  ▁HusbandWd  ▁RotaryXd  ▁screenplayYd  ▁PermanentZd  有一番[d  熟知的\d  轻轻一]d  非常喜欢^d  ▁borrowing_d  并将其`d bynad railbd  货车司机cd  ▁Chryslerdd  又如何ed  ▁sanitationfd  人家的gd  综合所得hd  ▁bulletsid 机上jd 荆棘kd  查看了ld  ▁Sloveniamd 睫毛nd  ▁Tulsaod  ▁analysepd 迁徙qd  时尚的rd  ▁trustssd 本国td  ▁Admiralud 严选vd  周期的wd 无止xd 物语yd  Settingzd  ▁offences{d  ▁recognizable|d  ▁cite}d  ▁bounds~d 登峰d  服务能力d  ▁affiliationd  某小区d 温婉d 重工d tecd  不安全d  先后在d  ▁uintd  ▁dwellingd  如果他d  缴纳的d 微刊d  ▁nonsensed  ▁monumentald  也是为了d  Spanishd 抗炎d  清楚的d  ▁HVACd  基本上都是d  全面深化改革d  ▁daylightd  蓝轩宇d ▁MAXd 余姚d  callbackd 催眠d FKd 带状d  ▁defectd  ▁Portraitd 徒手d 溶胶d MCAd ouraged  commonsd  会通过d  ▁Levid  ▁tracedd  大峡谷d escod  如果喜欢d  ▁Gothicd  油菜花d  ▁strapd  了一道d  季节性d 吻合d 萌新d  请记住d  结构和d  费用自理d  ▁Hirod  ▁Rootsd 四子d  属马人d  ▁acrylicd 死守d BMWd ièred Skipd  进入考点d  ▁zijnd 加载d Vod  输卵管d  电子科技大学d ▁कd  满洲里d  acionesd  豆制品d  ▁Doesnd  ▁skatingd  统计学d  做出来的d actlyd 罪魁d 青州d  ▁Chald  ▁tramd  ▁Imagingd  两人在d  但是这d  邵阳市d 到晚d 轮毂d  特别提示d 名将d 奇特d 领到d  股权投资d  ▁Hyderabadd VBd 经久d 勺子d  病情稳定d alletsd  ▁vegetariand  荆州市d 年近d  论坛邓州论d  ▁stressesd  共同维护d  相关链接d  蓬勃发展d 走高d  ▁Votingd 巨型d 手镯d 虽有d  ▁Nachd 凝血d 徐徐d  出发点d  新格局d  设计院d  再看看d  如有侵犯d Fieldsd 不落d  计划和d 乌龟d lsend  进入校园d addedd  centrald 人活d 优胜e 速率e  个省份e  的事业e  虽然有e  等违法行为e  ▁numberede 夜幕e  ▁Lisbone », e 嘿嘿 e 旁人 e  岁开始 e  ▁Maurice e 使你e 提亮e 摇晃e 是新e  下去了e  录用公务员e 冰山e 天成e ▁Raze  与生活e  更没有e 租客e  用自己e ueble  专业能力e  ▁cumulativee 亮度e ▁Embe  ▁evid e  ▁supervised!e  李世民"e 就多#e 辛劳$e  ▁sewer%e  ▁Leonardo&e 氯化'e 仪征(e 直立)e  material*e  ▁routines+e  unciation,e 刘国-e 大将.e  党委政府/e 人道0e  ▁banana1e  体检和2e  调查研究3e  追究刑事责任4e 1 *应对新型冠状病毒感染肺炎疫情5e  ▁Inner6e GZ7e  实施条例8e  的农村9e  Elements:e JY;e 为生e 草根?e  置业顾问@e  ▁BrigadeAe  ▁candlesBe 网报Ce  滨州市De WATCHEe 丛书Fe adenaGe 也和He  以为然Ie  都一样Je ▁ENDKe ▁tamLe  ▁DebbieMe 车险Ne  ▁EinsteinOe  观看视频Pe  ▁BastQe sworthRe  patrickSe  ▁promptingTe 内生Ue 日久Ve 秀美We  抓住机会Xe 拍拍Ye  ▁deltaZe  到极致[e Prep\e 秋雨]e  一个星期^e 党对_e 香椿`e  某种程度上ae  的花瓣be bitsce 隐身de 给小ee  ▁Monsfe  榆林市ge  国家能源he quetteie adanje uraike  公平正义le  ▁Nassme 手心ne 而死oe alebpe 本无qe  定制化re  ▁Fishingse  ▁Chronicleste  ▁Workerue  ▁attendsve  ▁hydraulicwe VTxe  工艺品ye  两小时ze  Growing{e  ▁Eclipse|e  ▁Disorders}e 农行~e 摇滚e adice  cribinge 趁热e ковe  学校在e  泰安市e  ▁Arbore  mediatelye  了解情况e ▁Хe 他便e ▁bune  Associatee  ▁visionarye pHe 老铁e  ———e  疯狂的e  肩周炎e  ▁herebye  ▁Valenciae MXe Mohe 另据e vive  　(　e  放宽至e  老乡俱乐部e hrefe  也是因为e 客源e  ▁Dunde 应由e ziae  ▁Vicke 一盔e  尼泊尔e  ▁uphole 德怀e  银川市e  大学本科e  ▁flourishe  ▁utilizese  二十八e 几张e  ▁Asidee 台下e 松原e hanie 半球e  街小巷e  ▁proposinge  ▁Institutionse 的自e  永远不要e 鞍山e  案件中e 眼皮e мерe  四中全会e  ▁rafte  ▁Interstatee  做贡献e  全身心e  体育中心e Detaile 及各e 阵子e  走出来e  一辈子的e  的有关规定e  ﹌﹌﹌﹌﹌﹌﹌﹌e 空空e  万元人民币e Matrixe  教大家e  截止时间为e 编委e CFAe  的旅客e 荣光e  的认可e 抚摸e  对大家e  居住环境e  预约方式e  ▁Lunche  整个世界e  ▁Solee ▁(£e 石景e 进补e  年轻化e  ▁possessese 学得e 液的e 溪区e  点外卖e Irishe  ▁Duratione 苗族e  社会监督e 有独e  ranginge  动物和e Addede Lene ppoe tele  免费获取e assice  下面二维码e  ▁replacese  人的心e  ▁Silente  ▁undergonee 三线e 叶面e  中医学e  ▁relocatione 干啥e  ▁“[e Prede Testse ▁OURe  免费赠送e 深得e  经营活动e  ▁Wisef  renchedf 微友f 类和f  ▁sharksf 中餐f 求真f 三号f ▁Jif Ctrl f  从我做起 f upunct f 坐诊 f  自由贸易港 f  ▁retalf  ▁versesf  ▁compensatef SOURCEf  生活垃圾分类f 紧实f 当属f  元现金f  比赛的f 谁家f  ▁Blockchainf 之火f 求租f  正常人f  ↓　↓　f  语文上册f  ▁Courtneyf 政党f blic f  affeine!f 不比"f  ▁Insight#f 美院$f ▁Sgt%f  我们的身体&f jobs'f uala(f  千家万户)f  ademark*f  心爱的+f  长按扫码,f stadt-f  ▁Kurdish.f 成热/f erdale0f  ▁prioritize1f 少见2f  水龙头3f opian4f  战友们5f  无证驾驶6f inness7f  ▁Lamar8f  Corporate9f ▁IRA:f  期待您的;f 水路f  大学教授?f  甜言蜜语@f  ▁extrAf  本活动Bf  等众多Cf  规划编制Df  ▁IbrahimEf 不思Ff  ▁alumGf RETHf  产业扶贫If 余篇Jf 村人Kf  货到付款Lf 共读Mf  ▁NateNf 京沪Of 俏皮Pf  平阴玫瑰Qf 患难Rf 画笔Sf  ▁TopicTf 岛上Uf 箭头Vf taiWf  相对的Xf ascarYf  郭晶晶Zf  ▁courtyard[f  农用地\f  ▁Stuff]f  ▁capsule^f  ▁turbines_f 本法`f Rendaf  ▁ritualsbf JHcf  层次的df 代号ef  的每一ff  ▁sleevesgf 谁也hf nhif  遵义市jf  产业项目kf  涉嫌犯罪lf 义的mf 楼顶nf 眼眶of  他们有pf  ▁ruinedqf gartrf  ▁selfishsf 留心tf  教你如何uf 健美vf  是什么样wf ptomsxf  ▁Cullyf  Overviewzf  remember{f 贾玲|f  开设了}f  不戴口罩~f eminf  ▁idolf 养阴f  ▁Oralf  适合人群f embedf 浮现f  应聘资格f  ▁bargainingf 信徒f 宰相f  未完成f 小鹏f  人活着f 对错f 应税f  的植物f  蒙古族f  和心理f  骑自行车f 兴盛f ▁Egf  不是我f  ▁heroicf 何超f ▁dopf 宾客f  文字及f ▁Keef  微信搜索f  ▁Oslof  会得到f 磨合f  宿州市f magef albumf 打针f 春华f 混杂f 腰腿f 吃素f  ▁Sharingf  ▁attackersf  ▁Photographf 更名f 水杯f  套房源f  ▁Hoganf 莫及f Undf  中风险f  体验到f  的光芒f  看视频f  相互作用f  ▁Matsf  ▁Ownersf  四个字f 坚毅f 大礼f 选对f  最直接f  观察到f  ▁Xavierf ютf  ▁deputiesf 灾后f Rossf ▁Xinf  一年一度的f  onomousf Leanf 鲁班f olphinf  ▁extravf 猪油f  的几个f  ▁Addedf  ▁assortmentf 芝加f  如果遇到f  政策支持f mysqlf " 如有侵权请及时联系f  ▁Skypef  ▁moralityf 吕布f ▁Jubf  ▁Jacquf 光刻f 勘探f olvf  ▁Hitchf  ▁OTHERf  ▁repliesf 请先f 锅盖f 高音f  ▁Medievalf ibalf  未来可期f  负责解释f  的高发f 时说f  上线了f  ▁neuronsf 索尼f  基层组织f  ▁arrogf  ▁wannaf  ▁dilemmaf  ▁pairingf 心急f  在体内f 读到f ystonef  ▁directsf  Counterf 保送f  未外出f  ▁halftimef ,\f Lukef  相关负责人表示f 班班f  要懂得f  均来源于网络f 一意f  为提高f eriong  很少有g  推荐给g  ▁Salmong  ▁successiveg 惊醒g 柔顺g 群中g 金子g  发展中 g  并不难 g  的食品 g pipe g  依法予以 g  ▁Twinsg  ▁flushg  防控就是责任g  ▁Tunisiag  饱和脂肪酸g 棒棒g  不造谣g  实业有限公司g 见闻g WAYg  isciplinaryg 背书g  ▁annotg  ▁Squadrong 该县g  高情商g  ▁bikingg  总结经验g  agements g 公道!g  显微镜"g  ▁fibers#g 多梦$g  ▁Sched%g  小编为你&g  进行登记'g éc(g  重污染)g  ▁Norwich*g  ▁referee+g  这是一场,g  inflamm-g  ▁youths.g 受苦/g  要全面0g  ▁Buenos1g  ▁elephants2g andas3g 复赛4g 屋檐5g buf6g los7g  会说话8g  ▁Arbit9g 佳人:g CIS;g слеg  ▁hurts?g  协和医院@g  ▁dispensAg  ▁excludingBg 用情Cg  不多的Dg  就像你Eg 骨肉Fg  小记者Gg  的一方Hg  课文朗读Ig 年多Jg  所以对于Kg  焕然一新Lg  电话通知Mg  reflectNg  ▁theirsOg  种植面积Pg  ▁NicaragQg 价款Rg ▁ABSSg ▁lurTg 比肩Ug 王建Vg  不下去了Wg  商务区Xg  зоваYg  的眼神Zg ▁Pie[g  指导工作\g  ▁landlords]g  佼佼者^g  ▁horsepower_g 白醋`g 直角ag  多伦多bg  ▁geopcg  ▁Flickrdg  ▁herbaleg  如有疑问fg 乡县gg SEEhg  先天性ig 俊杰jg 持刀kg 赤壁lg  ▁printersmg  ▁mansng 内核og 选民pg  空间和qg  ▁reproducedrg 每斤sg 浓处tg  措施的ug  让男人vg  深入研究wg 诋毁xg  自强不息yg ▁WCzg eware{g  ▁Brewing|g  进一步扩大}g 仅用~g Bolg  重症患者g  ▁brewingg 只争g 首场g  有深度g arieg  虽然现在g  reasonableg 荆斩g  可以和g  妥善处理g 内需g 钱江g  苏教版g Moong komg  ▁Gallagherg  推送时g  ▁battlefieldg  ▁distributingg 剔除g 类等g ▁'<g  挺好的g  ▁Seminarg  ▁Midnightg  于是就g  公司与g  背单词g  ▁Salaryg  ▁upholstg  ▁preseasong 两市g amountg  ▁Robotg  ▁foreseeg 大路g 黄龙g  二十四节气g 持仓g 美式g  共同参与g  证明自己g cliffeg 沉醉g  最让人g  ▁leasingg netesg 古玩g 退市g 教诲g 进面g  ▁bakeg 创客g  幸运的是g  收藏价值g  Openingg  ▁Scoutsg  ▁utilizationg  新零售g  米以下g iciog 此种g ▁egg  ▁Consequentlyg  充电宝g  所以你g weiseg Eyeg  ▁Gagag  ▁pinsg 平房g 转赠g  ▁assaultedg  ▁tutorialsg  完成作业g  ▁monarchg Festg  ▁intolerg 哀乐g  十三条g  坐落在g  ▁Priestg 号令g 炸鸡g ▁STRg  ▁Moduleg  一个很g  集中开工g 补救g sedg Fightg 张桂g  ▁Purposeg  的一句g  绿化带g ongsg ▁Skeg  公共文化g  ▁precipitationg  药物的g  ▁reorgang  失败了g ▁Jawg  分支机构g  ▁Warningg 纲领g  ▁Mauritg 下岗g  在广东g  强企业g ▁dang  年左右g  ▁censorshipg  看不清g seekg Marvelg 固化g 激起g  的天空g ▁tidg  ▁marketedg KTg  ▁arterg  ▁Bookerg  ▁miniatureg  我们去g  等作用g  这样就h  ▁Managersh  ▁draftingh 万股h 嘻嘻h  ▁aluminiumh 高某h  ▁Thrillerh  ▁Millenniumh 广安 h  ▁deficiency h 汶川 h ▁JS h ichael h  协调解决h  点击下方小程序h  ▁Landingh  ▁Selectedh  ▁grindh  ▁blacksh  ellationh 读过h 把这h 殊胜h  有心人h  ▁constructingh 易发h  信息与h  资产的h  ▁Paramounth  一致的h  幼儿园的h 拉面 h  再多的!h  他们没有"h  ▁freshly#h  ▁franchises$h 总算%h fal&h  安静的'h  已开始(h  安全第一)h  ▁Likewise*h 稻谷+h Ops,h  希望你们-h  当一个人.h  ▁implants/h  ▁Beau0h  ▁kitchens1h 鼻梁2h  相关事宜3h  ▁lays4h  青年大学习5h 拔罐6h 物权7h  自己的能力8h 佛经9h  了解的:h 无怨;h 进球h 高远?h  对健康@h  无忌惮Ah  的良好氛围Bh  向全国Ch  最终以Dh 怀集Eh  等各项Fh  HoustonGh 红梅Hh  成绩的Ih 独秀Jh  要持续Kh INESSLh 议事Mh 货款Nh iobOh  意识地Ph 厂长Qh 咬伤Rh 最想Sh gifTh  ▁lieuUh  ▁courtroomVh 复方Wh ahiXh  与会人员Yh  控制系统Zh  ▁liberation[h 沿岸\h  ▁sandwiches]h  ▁renovations^h 床单_h  ▁Browse`h  ▁Maverah  添加好友bh  ▁ralliesch extradh  ▁breweryeh 蒜末fh  知名品牌gh  ▁boasthh  ▁Soldierih  解决方法jh  妇女节kh  ▁Freddielh 执念mh 选好nh 狂风oh  并发表ph  李文亮qh  元人民币rh 疏肝sh  的名义th  的负担uh  旅游资源vh 林立wh Chelxh  ▁Answersyh  ▁frighteningzh 大吃{h  ▁ruler|h  上升到}h maps~h  仅代表作者h ridgesh 应接h 永兴h 玩儿h  ▁loopsh  ▁Suzukih  建议您h  ▁Coltsh  ▁bookingsh 推翻h  过多久h  过马路h 林彪h  宣传推广h  保持社交距离h  ▁disclaimh  科技部h  ▁Berksh  ▁rotateh 崇明h  伤害的h  ▁Lennonh  自以为h  积极向上的h  ▁glitterh 海门h 繁育h  网格化管理h  ▁Baghdadh  我曾经h  有价值的h  ▁Pressureh 毒症h soundh  ▁SARSh  ▁formidableh  教孩子h 揪心h Vieth SERh Viceh  专业学位h  家庭农场h oulosh  ▁Sofiah 对小h  本产品h Intenth  ▁Judiciaryh  三位一体h Olymph  ▁steakh  umerableh 来找h  ▁Pondh  resolutionh 免票h  急需紧缺h  新的一天h  罪魁祸首h  chemicalh  ▁hormonesh 惨遭h 走实h  日起至h  不是什么h  想起来h fewh asynch 应邀h 求婚h  ▁Mozarth  ▁stirringh  ▁adaptingh  ▁interruptedh  主旋律h 并经h 藿香h  状态的h  ▁blueprinth 胸腔h  ▁ultrash cvh ▁DECh  疫苗研发h  ▁disputedh  ▁Governorsh  Facultyh  文物保护单位h  ▁downwardh 能与h  部门和h  ▁strivingh  风险管理h  ▁Suttonh  ▁stickyh 不败h  突出重点h  ▁remnh  ▁Friedmanh  ▁classmatesh  ▁ArrayListh jqh 侧翻h 助农h 熄火h  的户外h toosh 合院h 朵朵h  加大了h  主讲人h  嘉兴市h Keysh  Paperbackh  党的政治h  ▁computationalh 闪亮h  可以实现h gradedh 杯水h 爬上h  最有效h  外国语大学h  粮食安全h  这些年来h  世界大战h  择优录取h  也就不i ▁MRIi ▁komi 四平i  自尊心i eckeri  ▁circusi 军阀i lacesi  ▁Lindsey i 山泉 i  在经济 i akin i 西江 i  我是一个i )→i  沙发上i  高级工程师i  ▁Farmeri  ▁Halifaxi 一去i 优品i 双创i  土地出让i  毛细血管i 甜头i  ▁lunari  ▁stackedi 按需i  督查组i  packagesi " 身体健康和生命安全i  ▁creditors i 有致!i  故宫博物院"i 吃力#i 打火$i 真话%i  下半身&i  心肌梗死'i  ▁Wants(i  参赛者)i  有足够的*i  科研院所+i planes,i 熟了-i  透明的.i 刺痛/i  犯罪行为0i  ▁seize1i DY2i 岗前3i 已出4i 灭菌5i GIN6i ▁'@7i  轻轻地8i  肆无忌惮9i  ▁sourcing:i  ▁anonymity;i  ▁fundamentalsi 麦芽?i 万州@i 空地Ai  都具有Bi hotsCi  ▁railsDi rietEi 淡了Fi  ▁riotsGi ར་Hi  英国的Ii liftJi  也就意味着Ki 资管Li  马铃薯Mi . '不符合食品安全国家标准规定Ni 安眠Oi 鹧鸪Pi  和妻子Qi  地区和Ri MotorSi determTi 队列Ui tripVi  公益广告Wi  卡拉胶疯狂Xi " 垃圾分类卡积分绿卡Yi  亲身经历Zi  ▁Actors[i 疑虑\i  浪费了]i  ▁serm^i  ▁interpretations_i 洪涝`i 菜园ai  十年的bi 琵琶ci  同样也是di  ▁abortionsei 抢夺fi argergi  方案和hi ▁ripii  这个阶段ji Guestki zzoli ▁Kubmi  ▁moleculeni 原定oi  的资料pi  促进会qi 发往ri 永春si  ▁depotti 原浆ui 分隔vi 隐匿wi 니다xi Linkedyi 之乐zi 下称{i 德智|i 无机}i 呼万~i  披荆斩i  有发热i improi  ochemicali  ▁junctioni 儿孙i 致歉i  和视频i  ▁cansi  ▁wipei 临门i  ▁unavailablei 真伪i raisali  ▁Sanctuaryi  ▁exploitedi  孩子对i  农业农村厅i  ▁gentlemani  ▁Tiffanyi 南宫i 摇曳i  的新生i  造就了i  工作条例i  月中下旬i  ▁Instrumenti  斗争中i 点播i  违法地点i  fightingi  也会在i  出生的i paini RBi  ▁hikesi  安排专人i  ▁songwritingi 遮掩i  发现在i  释迦牟尼i Executi papersi  ▁Appalachi  清晰地i  ▁unusedi 横跨i 永乐i  生育保险i  ▁undisi 邪气i  一网打i 读本i 三阴i 熟食i 让小i 面额i  ▁diagnosei 施展i ▁cii amicsi  ▁Crimesi  培训会i  ▁speedingi  ▁submarinei 良知i 解救i SPAi EWi limiti  自己的想法i  ▁Nielseni  ▁impairmenti 天龙i 查到i 黄疸i  周杰伦i  联系到i  管理平台i existsi  ▁buddyi  ▁predictingi 退化i  resolvei  ▁hurtingi 倒下i  线上的i ibrei  ▁Giftsi  他觉得i  副驾驶i  基础性i lxi 风雪i  型企业i  ▁Whilsti  忘不了i  ▁Entrepreneurshipi 只求i  确定性i  航站楼i ▁heni  分批次i 打过i  作者和i  第二位i éhoi  严重者i  依然严峻i  Changesi 取悦i 提纲i 村医i 祈愿i  发展成为i 一区i  的美人i rosti ▁Miai  ▁squaresi 供求i 留恋i  的次数i  涉黑涉恶i  要建立i  脱贫攻坚工作i  ▁Falconsi 烟雨i izzyi  secondaryj  ▁unrelatedj ROWj  中找到j itivesj 低了j ▁alej 考勤j  ▁antibodyj  ▁Cunningham j  在小区 j  ▁Dong j  高等专科学校 j ,‘ j 平顺j 斯顿j  文章来源于网络j  这可能j  家常菜j anorj  轮融资j  whetherj  ▁litterj  过渡期j  一名男子j  开学复课j  ▁receiversj  ▁disadvantagedj 军医j  汽车行业j  containsj  养成了j  玩家的 j mort!j  ▁forbidden"j 虚无#j ▁Riv$j  人一步%j  ▁Stocks&j  ▁Lafayette'j ▁Ink(j  ▁superintendent)j 中芯*j REC+j  郑重承诺,j 城事-j  极少数.j 刘文/j  享受着0j  更美好1j 宝剑2j 性子3j  各领域4j ▁IMF5j 下部6j 会变7j  如果你喜欢8j  问候语9j  ▁Celtics:j  不少的;j arinej desk?j CZ@j  独角兽Aj ldandBj  ▁NaplesCj 不惊Dj gueEj  防疫知识Fj  ▁EnoughGj  ▁workoutsHj  不认识Ij  万事如意Jj 师姐Kj  ▁antiquLj  ▁proportionsMj  ▁subscriptionsNj LuOj  ▁burntPj  关注推荐理由Qj  ▁buttRj  版权等问题Sj  向日葵Tj  等企业Uj 吴尊Vj  的人士Wj ProxyXj lishesYj  ▁daringZj  ▁Parkway[j  ▁terminate\j  中央气象台]j  颠覆了^j crime_j  ▁printable`j QBaj 家湾bj 花甲cj 雕像dj  基金的ej 单间fj  难度大gj rologhj  第一季ij  请拨打jj  ▁declineskj 别样lj  奥巴马mj 罐头nj 无痕oj ▁Einpj 修辞qj  已累计rj  ipationsj 荡漾tj ▁ASCuj  ▁mandatedvj  ▁sketcheswj 扩充xj 拨款yj 时可zj  ▁Walton{j nz|j 细嫩}j 英德~j  主要通过j  ▁hatchj  ▁Recommendj 发稿j  开庭审理j 防备j  不正常j WXj 六国j 成活j 花青j  根据自身j  ▁Uruguj  灵活就业人员j  ▁citationj 马里j  不用说j  ▁wanderj  requencyj  ▁Renewablej  ▁launderingj 开挂j 后腿j  ▁stealsj 一跃j 如鱼j 挤出j  小学部j 我方j 后浪j 接报j ullsj  naissancej 归国j  Missionj 能见j  ▁fulfillmentj cppj  不在意j  ▁charsetj  ▁ONLYj  ▁defeatingj дыj ▁lubj  ▁Spiritualj  太大了j 应按j Negj  多长时间j  这个社会j  ▁Churchesj  他开始j  ▁trendingj 潇湘j 银币j ▁falj  ▁heartfeltj 大半j  对我说j  夫妻双方j geonsj  Principj 眉头j uspj  制造的j  黄河流域j 放水j 1 *中华人民共和国治安管理处罚法j 延后j 驾乘j ▁REDj oittej esteadj  ▁Framej  推动高质量发展j 灵宝j 胆小j 遵照j  earancej  ▁Ecologyj  检测和j  ▁linebackerj 预后j ▁Dedj  ▁Seatj 达克j  不正确j  与世界j  学雷锋j 瘦了j 鲜卑j  中关村j  仅限前j 抱抱j  熊孩子j 自营j 采暖j  大同市j  意外险j  ▁inherentlyj 九大j 很像j 能从j  科医生j  ▁hurdlesj 居留j  ▁Recognitionj insicj 鸡西j  ▁Vitaminj 王海j  以各种j  预报名j 不详j 血瘀j  充分考虑j  ▁cyclistsj  ▁Kensj 擦肩j 阿司j  形容词j  ▁saintj  多角度j ▁MUSj  ▁pinchj  ▁Marilynj  ▁Buckinghamj 兴和j  ▁trendyj  Performancek  ▁goalkeeperk 污泥k  тьсяk  ▁truthsk 来抓k  只要在k Cryptk 何方k 豆芽 k 说地 k 首诗 k cels k ▁Pav k  ▁bambook 陪我k  好老师k 机等k ismick 防伪k  的同学们k rancesk 电极k  专门为k  体制内k ▁Fujk Agentk synk  ▁pennyk  ▁Settingk 内推k  ▁Foolk  特别多 k Vill!k  ervical"k 废钢#k  的数字$k 唐僧%k 的药&k 路北'k  促发展(k  披荆斩棘)k  中华民族的*k 从古+k 托幼,k 甄选-k  最高分.k  领导力/k  工作落实0k  硕士生1k  ▁vegg2k  ▁viagra3k 休养4k 丹霞5k 高院6k  培训课程7k 广发8k  生命周期9k  ▁Bold:k  ▁decreases;k  ▁liquidityk ▁Ard?k  ▁Pratt@k  ▁WizardAk tbBk 余次Ck  发来的Dk ▁TyrEk ▁[];Fk  ▁cassGk  人民至上Hk 稳居Ik  ClinicalJk  MultipleKk  工作纪律Lk 积压Mk 脸蛋Nk  乡村医生Ok  这种做法Pk CTIONQk  合法性Rk itantSk  ▁zoningTk  ▁poppedUk  ▁MeansVk  dfundingWk  实际控制Xk  ▁misdemYk 通蛋Zk Got[k bytes\k  Required]k 甜食^k  和普通_k  培养了`k 红白ak 过万bk  ▁overflowck 动用dk  ▁Antarcticaek  第三季度fk  ▁Roomsgk  和谐的hk  中国邮政ik  ▁backwardjk NVkk  ▁Minglk  ▁tracingmk  ▁hecknk  一刀切ok  却不能pk  ▁foilqk 表姐rk ENAsk ▁ötk 爆红uk  和责任vk  就说明wk  可以利用xk  生命至上yk  西安交通大学zk HG{k letal|k  了一眼}k  在基层~k  家庭中k 仁爱k 锋芒k 张子k 滞后k  往前走k : 3和新疆生产建设兵团报告新增确诊病例k 与小k 卧床k  最新版k  这么久k ogramsk  行政主管部门k Evalk  ▁Ronaldok (()k  东南大学k  ▁chalkk  ▁pumpkink 拆分k ORMk 装有k 重用k  arendrak " 治理体系和治理能力k  ▁Cameroonk 在在k  密切配合k  ▁gradualk 印军k 夜的k 并称k 血型k 绵延k ippersk IBMk  建筑工地k  ▁sacrificesk ▁UFOk  ▁drivewayk  脑海中k 夜宵k 晨光k  ▁differsk  在当前k  施工人员k 饭吃k  ▁intervenek  子宫内膜k  ▁gymnk  ▁scrubk  ▁handingk  ▁humorousk  ▁vaccinationsk 吐痰k 要高k 选票k Oilk  以达到k  花生米k  ▁innocencek freyk  乡镇卫生院k 起因k pastk normk  ▁bureauk oniank 塘村k 知心k  增长率k  遇见你k  期待着k  有这些k ▁Hosk  一起分享k  ▁Jeromek  ▁Salmank ▁UPSk  ▁Vanessak 奶牛k женk 大度k 闯入k DOCTk  突如其来的疫情k 抗震k  总人口k  Economick 三步k  地看着k 废品k DXk 他以k 道士k  送清凉k  一审判决k ilandk  等公共场所k  ▁consultedk 主见k  外卖小哥k 可比k  等多家k  打扫卫生k  ▁possessionsk 功勋k 很想k  及家属k  反诈中心k  ▁plethorak 泡发k  ▁Experimentalk ㎡　k oingk  ▁Burgerk  ▁Posterk  的医生k  ▁сиk % 人民群众生命财产安全k 　「k ▁LSUk  动力方面k  夫妻共同k  ▁Huntingtonk 银色k 死人k happk  这款车的k  ▁Rital  ▁communistl  烈士陵园l itiousl  ▁Junctionl  消费扶贫l  ▁onwardsl  ▁Bottoml 众筹l 肉馅 l  ▁Wheeler l  归根结 l  打下了 l  民族的 l meanl  平时的l  ▁Typicallyl 秋千l  切入点l  报名及l  运行的l  安全技术l  还是需要l  ▁incapl  ▁Wolvesl  ▁beginnersl  未及时l  输入性l Samel  ▁residingl 适配l  要注重l  全面完成 l  ▁adren!l tou"l  最严重的#l  骑电动车$l 惊蛰%l  如实供述&l ун'l 攀爬(l  ▁pristine)l  ▁spirituality*l  ▁cousins+l 路东,l 酸辣-l  不妨试试.l 明晰/l  取得新0l 失利1l  住房租赁2l  ▁punished3l  ▁contention4l 猛进5l Farm6l  ▁polish7l 两颗8l 四房9l 茂盛:l rising;l 公会l  报名网站?l 育肥@l ▁SETAl juvenBl 张建Cl 走深Dl  十年前El ▁oscFl  阿司匹林Gl  这样的事情Hl  ▁CannonIl  ▁NigeriansJl  ExpressKl 追逃Ll 消散Ml  细心的Nl  ▁telescopeOl 林志Pl  directoryQl 跌至Rl  FinanceSl  eredithTl  自贸港Ul  有什么用Vl  ▁toppWl  热衷于Xl  依法刑事拘留Yl  ▁widgetZl  strument[l  ▁crowned\l  ▁vicinity]l 很美^l 皇宫_l  魅力的`l 号和al 流露bl 留校cl  上下册dl  是基于el  洗衣服fl  自主研发gl  ▁funkhl  商品住宅il togglejl  ▁bellskl 厂的ll 和胃ml  ▁shooknl 心包ol  中青年pl oltaql ▁Pirrl  ▁angelssl  icipatedtl 热泪ul  两个方面vl  乳腺增生wl  Marketingxl  手机壁纸yl 困苦zl 一节{l  ▁spoon|l  孙允珠孙允珠}l  ▁Dynamic~l CSSl ▁SIl  讲道理l bredl  全球经济l  ▁боl  而在于l  第一时间向l 压紧l 惊现l  太过于l  的朋友可以l  ▁WITHOUTl Rawl  ▁mastl  ▁aminol  ▁printfl  喜欢上l  ▁WORLDl  司司长l  他们认为l ocussl  ▁Marklel  ▁determinl  ▁imperfectl  可以享受l  ▁passagesl  有惊喜l  ▁predictsl  让观众l  为解决l  更强的l  高温天气l 打闹l  属狗人l 分拣l  流动人口l  的安全性l  年的历史l  每日更新l 畅享l Layerl  ▁influxl  ▁Printingl  ▁reflectionsl 广深l Uploadl 之和l  印花税l  一个国家l  不断加强l  等防控措施l 厚德l  ▁carcinl 买过l  工作期间l  没有时间l  ▁Builderl  attributel 遮盖l  ▁Zambial 值的l 尾灯l  衡水市l  ▁ghostsl 军旅l LEYl ▁clal  ▁specimensl 叫声l 院感l  ▁curvesl  ▁McCartneyl 哪天l 组队l 腰包l  nothingl  ▁inaugurl  ▁Greenwichl 槟榔l  时至今日l  differentl  ▁Provinciall  ▁Settlementl  ▁permittingl 蟑螂l 转盘l  ▁Plaintl 往日l  你才能l  ▁tropl 南康l  高校毕业生就业l 执教l 都为l  ▁brutalityl  ▁Georgianl  ▁purportl  ▁Everybodyl  ▁allal  吃一些l alanl  装备制造l  ▁strandedl ▁udl gagesl 肿块l Noml  给孩子们l  Specificl 圆明l 解散l  李兰娟l  也可以通过l npml edenl  ▁Polarl  ▁unreall  ibrationl 发胖l 巩义l 一往l  毕业季l 点小l  如何进行l  谋取利益l  疫情的影响l  ▁Irvinel  ▁consolidatedl  易地扶贫搬迁m 就以m  行驶的m 斗殴m 留置m 至高m  手术室m  带来的不便m  ▁hardshipm Used m  ▁unstable m 从何 m john m ▁GPU m  ▁honorarym  ▁Collaborationm 公变m 馅饼m  关注和m  ▁hearsm  四个方面m  工程管理m  如何去m  荷尔蒙m ▁CRMm  ▁Candym 紧随m 美得m  ▁sloganm 两套m 特辑m  与现代m ▁sua m  中美关系!m anity"m  medical#m  ▁circuits$m thro%m  非常容易&m  的房屋'm  ▁Anime(m  公安交管部门)m  ▁rabbit*m 剧目+m  ▁gallon,m  ▁Navigation-m 道场.m 金科/m 女郎0m ALK1m VAR2m Annual3m 吃惊4m  充分认识5m  支付方式6m  特种设备7m  inatory8m  reciation9m  我们相信:m  ▁Odyssey;m  Companiesm  ographed?m  白开水@m  科学合理Am 在县Bm 沙棘Cm  一盔一带Dm  ▁runtimeEm  ▁exportedFm 求知Gm  利用好Hm  ▁monkeyIm 可笑Jm  在社会Km  的自由Lm ultzMm  ▁RutgersNm  分手了Om  多学科Pm  PhilippQm  ▁stainedRm  这些事情Sm \'Tm tjUm 下落Vm  ▁RunnerWm  ▁contenderXm 证人Ym  和文字Zm ICES[m  点击标题\m  ▁Shap]m  ▁Jonas^m 宣读_m 肾经`m rillam 深浅bm  ▁hookscm 督脉dm 黄精em  ▁Indiesfm 下拉gm  的一致好评hm  高职院校im  ambiquejm 伊德km  答应了lm  跟别人mm  不管你是nm cigom  不明的pm  再比如qm 上部rm 预扣sm  ▁limestonetm 褪色um sidvm  一拳一拳wm  真实姓名xm 追星ym 项羽zm  没有问题{m  metadata|m 无路}m  两方面~m  ▁Millionsm  答案解析m  人均可支配收入m  ▁conceptionm からm 待到m ▁CESm  ▁gigsm 数以m 罗志m  让孩子在m  ▁Moorm  ▁Netsm  ▁bladesm 分店m PUTm  ▁Carmenm }　m  得很好m ▁ASSm  以人为本m  消费者权益m  ▁Divorcem  ▁Maharashtram ▁VSm  也会被m  我怎么m  吾悦广场m  反垄断m  生命中的m  ▁sixteenm  ▁salvationm Hanm  出发前m  新闻发言人m  检验机构为m  ▁Startedm 反光m ernom 尽的m  上加霜m  等同于m 短文m 德语m  生产日期m  全面提高m  热点问题m  zaixianm  ▁Latviam  产学研m  重大战略m  也不要忘记m  ▁Tysonm 学说m  ▁Bretm  ▁Renom  ▁Pleasantm  ▁sparklingm  ▁powerhousem 古籍m  ▁ethicm 天佑m  车型的m PCBm  千呼万m  长治市m 三只m 看下m  杨某某m  如何应对m  ▁gazem  ▁vastlym  ▁bolsterm 后要m  校园里m  螺蛳粉m ottom 阻隔m arfm  ходиm  套人民币m  ▁Outreachm  ▁figurm  ▁tempom ;"m 圆柱m ▁Ginm  ▁Furym  ▁contagm  ▁Rabbitm  ▁Madonnam 桃园m 的超m  方便快捷m  欢迎留言评论m 在全m  就是用m  不会轻易m  中国第一m  能力有限m 等国m TIMEm gwm  无新增死亡病例m  ▁resemblesm 火线m  雪上加霜m  ▁worsm  诊断为确诊病例m 外事m  ▁Robertom  ▁competitivenessm 嘴边m  乡村教师m  ▁lamentm  ▁Directivem Nim 曾获m seedm  的优质m  ▁Liesm  ▁whereinm 恩典m  一级学科m ichtem MFm  彭德怀m  安阳市m  交通管理局m  也经常n  招生政策n  群众举报n  ▁establishesn 配伍n Behn  降到最低n 正职n  上进心n  ▁Battalion n  ▁crackdown n DK n 书上 n 冲浪 n  日零时起n  疫情严重n  很多东西n  通过此次n 也正n 知音n 闪付n  日当天n  特色小镇n  归根结底n Runnern  ▁offendern 免职n 实木n  水浒传n assinn  ▁Warmn 患处n 沉闷 n  不健康!n  ▁Sultan"n  亿美元的#n  ▁Benson$n  ▁Pokemon%n  ▁Honduras&n 武侠'n ▁Yo(n  秩序的)n  专项检查*n 传入+n  自驾车,n  ▁Oaks-n 给到.n  可爱的人/n  ▁Bosnia0n  ▁wastewater1n 量刑2n  网上缴费3n illet4n  ▁Silk5n  ▁Bates6n oxide7n olulu8n  ▁Deutsche9n  也并不:n  exception;n 成真n  变异株?n  感到很@n  千呼万唤An  用于治疗Bn  文化和旅游局Cn  ▁troublingDn SIMEn  家具家电Fn  ▁ProtestantGn 呼噜Hn 养蜂In  市场竞争Jn  ▁AuthoritiesKn  要充分发挥Ln 阻击Mn  ▁GambNn  文化和旅游部On 雪地Pn  ▁StartupQn 王妃Rn  ▁assertEqualsSn MASTn GhostUn  被人们Vn BOOKWn TermsXn  ▁siliconYn 及格Zn  有影响[n  什么地方\n  一边是]n 言辞^n  丈夫的_n  总结出`n  其他地方an  综合编辑bn SZcn  ▁Travdn  去学习en  和交流fn  ▁predatorsgn rexhn  很多孩子in 同桌jn 这双kn misln cerptmn  ▁commentingnn 出力on 经由pn  随迁子女qn 大通rn 先机sn 寿县tn 质押un ▁hpvn  ▁archivalwn 降落xn otleyn  ▁rabbzn  ▁Whenever{n 几名|n  ▁clashes}n mk~n 连着n benchn  广播电视总台n 尝尝n 流涕n ixingn 再造n  不得在n Trendn  ▁alloyn 回族n  ▁caten 劳逸n 外向n  有好处n  ▁overdosen  涌现出n  ▁personan 费劲n  不动产权n  销售渠道n  ▁Pursn  ▁sindn 伤病n 洋气n 转学n USERn  ▁theatresn 明德n  反映问题n  才是最好的n  ▁Rocksn 笔法n ▁djn  保健食品n  ▁Mesan feitn  ▁Kellern  审核丨n  依法处理n  便捷的n  ▁Editorsn 回迁n 弄得n 本网n alymn  全文如下n  流感疫苗n guiden  用起来n  ▁fourteenn 春来n  安置点n ▁foin  ▁Illustratedn  工作压力n odiacn  权威的n  直接进入n rokeen  studentsn 极低n  行政区划n 群山n 省下n  婚姻家庭n  生态环保n  ▁scannern 李建n  一米线n  把时间n  ▁owingn  ▁Hospitalityn 做错n  丝毫没有n  ▁Shelbyn  ▁accidentaln  常常会n  ▁Messin  县领导n 混动n Tedn  办理中n  ▁Jedin  ▁hiresn 润燥n 谎报n  的高峰n  深入浅出n  ▁subscribern  ▁inheritancen  ▁justificationn 每组n  computern 后跟n  二十二n ▁Vodn  ▁Humphn  instagramn 切块n  联想到n  多个国家n  ▁Lauderdalen беn  的好评n 六是n  未来一周n  ▁nausen  Creativen tkn  ▁demonsn  ▁Judaismn 过人n  可以得到n 张云n 成群n ▁czn  给了我们n  ▁potsn  不放弃n ▁Easn  剧烈运动n  ▁Sauln 积聚n  特效药n  上课时间n  ▁stereon  ▁Driversn 足浴n  第三单元o yneco 中带o  ▁Poleo  ▁expenditureso  红红火o  森林草原o  决战决胜脱贫攻坚o Hito das o  ▁capita o Sav o  关晓彤 o  ▁pouring o  生物安全o 能做o 重构o zoro  只争朝夕o  ▁repetitiveo 炫酷o  年度考核o 器和o  给家人o  党员领导干部o  ▁advertiseo Spaino  ▁beeto  Bindingo  ▁Dioceseo  江宁区o  菜篮子o  这类人 o  报名入口!o  ▁nicht"o  ▁enlisted#o  ▁peacefully$o Kh%o poll&o  ▁afterward'o 勤恳(o  ▁bassist)o  ▁Dolphins*o 唱片+o  点赞哦,o  那一天-o 玫瑾.o ▁df/o  大家都是0o umann1o  核武器2o 排头3o  了解本地4o 赚得5o  疫情防控情况6o  ▁volunteered7o 更早8o 李亚9o  参保人:o ▁APC;o  就变成了o  ▁Guns?o  ▁stew@o  冷空气影响Ao  ▁warriorsBo 残渣Co  投资建议Do  也是一样Eo nesiumFo 王永Go  我愿意Ho  健康知识Io  通过学习Jo SaudiKo  ▁TailLo  吉祥物Mo  全面屏No  就读于Oo  的图片Po  CertainQo  ▁righteousRo 士顿So 边框To ologieUo  ▁polesVo 狙击Wo ▁XIXo 同盟Yo ▁DNSZo 事主[o 金榜\o 首期]o  人与自然^o  基本农田_o  Leading`o 文在ao  Designedbo 分型co 大城do 轻生eo Parsefo Hydgo  后期的ho 分到io 逐项jo  局限性ko 直肠lo  ▁foldersmo 润肠no  生活品质oo  ▁Berepo  ▁spectacleqo 宝库ro 朝代so  ▁cartoonsto 一问uo 刘志vo 放疗wo 连贯xo  市中级人民法院yo 公募zo  补助标准{o 题的|o Teen}o  欢迎投稿~o 月到o  ▁creamyo  ▁oxideo 小刚o 飘窗o  回答说o  强度的o  ▁BOOKo 不谈o 国潮o 法拉o Kenno  春运期间o  ▁Leaho enableo  修订的o  收购价o  还有许多o 点睛o 结了o  商业综合体o  inatorso  ▁subjectiveo 不只o  被列为o  ▁Reginao  ▁wholeso 明灯o  初中语文o 媛媛o 欣然o  都希望o  Ultimatelyo 风行o  时有发生o  重要的事情o  ▁Snapchato  ▁beginnero owlero  ▁locomo  purposeo 吃个o 手头o ooterso  ▁postureo 在第o  被授予o  政治责任o  ▁Phantomo  支持的o 天时o  ▁brickso ▁Áo ibuo  只是想o  收银员o  NotNullo 吓人o 所称o  不断优化o  ▁MySQLo  ▁boycotto  ▁collapso  得天独厚的o  新冠肺炎的o  ▁Revereo  Academico  同时又o  建立的o 护眼o 直行o  发布关于o  ▁eccentrico  ▁harbouro  行驶证o  ▁dysto 螺栓o  得不偿失o  ▁Assado  ▁kindergarteno 春意o  紧身裤o  存在安全隐患o  ▁philosophero 投注o  医院感染o  普通发票o  ▁Healingo  ▁experimentingo 鱼头o  Keywordso 遂宁o  有一股o 大瑶o 见影o  上小学o  会议召开o  永远都是o  ▁Springero  ▁Presidentso  ▁strato  ▁assemblo 段落o  是每个o uzzleo 小楼o 张口o  一定能够o  ▁redefo ▁WASo  ▁schoolingo сыo  不要被o  说一下o  等不同o 纯真o  ▁shrimpo  消防设施o  Receiveo  ▁contempto  ▁crashingo 日游o 进水o  政府机关o  ▁Excellento  一天天o teeo  可以对o  广告牌o Suitep 界线p 愈加p ▁XVp  能成为p  ▁spurp  ▁spellsp  中国建筑p  ▁Fantasticp  管理经验 p  科技的 p  需要用 p  居家健康监测 p  ▁Kaiser p 始祖p 福星p 草稿p  ▁framingp 内镜p  锦上添p 淡斑p 去到p  ▁soilsp  ▁empiricalp  ▁Publicationp 逐级p BURp Dogp  调味料p  ▁boostedp  为人知p  ▁Zurichp 之苦 p  这么简单!p  的美食"p  ▁posing#p  ▁Illegal$p 厨娘%p 　>>&p  属蛇人'p ▁pH(p 部首)p  家长在*p  的各类+p quad,p  公司简介-p  Archives.p VOC/p ▁han0p PRESS1p  置业有限公司2p  ▁coated3p 悬念4p 芬兰5p  前几年6p bash7p coq8p  ▁Orth9p  ructions:p  ▁Betting;p  ▁Hancock